Why Your AI Agent Eval Set Matters More Than Your Prompt

An eval set built from real failures outlives any prompt or model, encoding what 'working' means independently of implementation. Most eval sets are weak because they are written from imagination at the start, when you know the least about how the agent fails. The article argues that every incident should become a permanent test case, and that grading behavior rather than exact strings prevents flaky tests.