Dev Tools · 10h ago
Open-Source CLI Lets Developers Run AI Evaluations Locally
Quantiles, an Apache 2.0 licensed open-source platform, offers a CLI tool called 'qt' for running and comparing AI evaluations on local machines. Developers can execute benchmarks like SimpleQA Verified with a single command, using local embedding models for scoring. The tool aims to simplify evaluation workflows that often involve scattered scripts and logs.
Meridian48 take
While the CLI reduces friction for local evaluations, its demo model uses random outputs, so real utility depends on integration with production LLMs.
Read the full reporting
Run and Compare AI Evaluations with a CLI for Developers and Coding Agents →
DEV Community
ai-evaluationdeveloper-tools