How to A/B test LLM prompts without fooling yourself

A developer shares a practical guide for A/B testing LLM prompts, emphasizing the need for large sample sizes to detect small improvements. The author learned that 30 test cases are insufficient; catching a 4-point improvement required hundreds of examples. Key techniques include testing both prompts on identical inputs and reporting a confidence range instead of a single average.