Why Your Prompt Eval 'Fix' Might Be Just Noise

Teams often tweak the worst-performing prompt variant based on weekly eval runs, then attribute any improvement to the fix. But regression to the mean means the worst variant likely improved simply because it was unlucky before. Without an untouched control variant, you can't tell if the fix actually worked.