Dev Tools · 1h ago
RAG Eval Flakiness? Fix Non-Deterministic Retrieval
A developer found that RAG evaluation scores varied run-to-run due to non-deterministic SQL ordering. Ties in ts_rank_cd caused PostgreSQL to return rows in arbitrary order, which changed Reciprocal Rank Fusion results. Adding a deterministic tie-breaker (ORDER BY score DESC, id) made the evaluation reproducible.
Meridian48 take
The fix is trivial, but the lesson is critical: non-determinism in retrieval pipelines can silently invalidate RAG evaluations, undermining trust in metrics.
Read the full reporting
Your RAG Eval Isn't Flaky. Your Retrieval Is Non-Deterministic. →
DEV Community
ragevaluation