Model Drift Eval Board Distinguishes Wrong Answers From Absent Ones

Model-drift grades models weekly on a frozen suite with an exact-match grader, but a call returning no valid answer scores identically to a wrong one. A model appeared to drop from 69% to 3% overnight due to a rate limit on 34 of 35 calls. The board excludes by aggregate reliability, dropping a run's accuracy point only when reliability falls below 50%, avoiding inflation from failures correlated with difficulty.