Developer builds small benchmark to pick best LLM for coding

A developer tested three LLMs on 14 coding problems across Python, C#, and Bash, scoring pass@k, cost, and latency. All models achieved 100% pass@3 on problems they answered, but one was content-filtered on four problems. The key differentiators were cost (1.3x spread) and latency (6.6x spread), with one model performing poorly on Bash.