AI · 1h ago
OpenAI drops SWE-Bench Pro over reliability concerns
OpenAI no longer recommends SWE-Bench Pro for evaluating coding models, citing unreliable scores that don't reflect real-world performance. The company found that the benchmark's metrics can be gamed and don't correlate with actual software engineering tasks. OpenAI now advocates for more robust evaluation methods like SWE-Bench Verified and internal human assessments.
Meridian48 take
This move highlights the growing skepticism around AI benchmarks, which often fail to measure practical utility and can mislead both developers and investors.
openaiai-benchmarks