Dev Tools · 6h ago
How a flawed eval made perfect MCP servers look broken
A developer found that an LLM-powered eval for MCP servers incorrectly flagged a perfect-scoring server as failing tool selection 62% of the time. The bug: the eval generated single-step tasks for multi-step tools, penalizing correct reasoning. After fixing the task synthesis, the same server scored 100%.
Meridian48 take
This is a cautionary tale for anyone building agent benchmarks: if your eval punishes models for correctly reasoning about multi-step dependencies, your failure rates are noise.
Read the full reporting
My eval said a perfect MCP server was broken. It was the eval that was lying. →
DEV Community
mcpllm-evaluation