Dev Tools · 10h ago
Eval confidence intervals are too narrow due to clustered data
Binomial confidence intervals for eval pass rates assume independent examples, but most eval sets contain clustered data from shared documents or conversations. This clustering reduces effective sample size, making intervals narrower than they should be. A cluster bootstrap can correct the bias and prevent false conclusions about model improvements.
Meridian48 take
The article highlights a subtle but critical flaw in eval methodology that can lead to overconfidence in model improvements, urging practitioners to account for data dependencies.
Read the full reporting
Your eval's confidence interval assumes independent examples. Yours are clustered. →
DEV Community
eval-methodologyconfidence-intervals