AI · 2h ago
New RAG Benchmark Exposes Hidden Failure: Confident Wrong Answers
RE-call introduces a benchmark that tests both retrieval accuracy and honesty, using 14 answerable and 5 unanswerable queries. It reveals that a system can have perfect ranking metrics while confidently answering unanswerable questions. The false-confident rate (FCR) metric exposes this failure, which standard RAG evals miss.
Meridian48 take
The FCR metric is overdue; without it, RAG systems can look flawless while being dangerously unreliable.
Read the full reporting
Retrieval-Augmented Self-Recall — Part 4: Benchmarking Retrieval *and* Honesty →
DEV Community
ragbenchmarking