Dev Tools · 2h ago
Benchmark reveals most AI coding agent skills don't beat a placebo
A developer benchmarked Claude Code agent skills against a placebo prompt and found that many popular skills fail to outperform generic instructions. One skill reduced code size by 23.8% vs baseline, while a 196k-star skill barely beat no instructions. The study ran 516 tests across three arms to separate real effects from prompt suggestibility.
Meridian48 take
The placebo-controlled methodology is a wake-up call for the AI agent skill ecosystem, where star counts often substitute for rigorous measurement.
Read the full reporting
I benchmarked Claude Code skills against a placebo — and half of mine failed →
DEV Community
ai-coding-agentsbenchmarking