Dev Tools · 7h ago
Developer evaluates agentic tool-use on local 35B model, finds trap avoidance strong but harness flaws
A developer tested a quantized 35B model's agentic tool-use with honeypot tools, finding it never triggered traps. However, harness issues caused false failures: one run looped and applied a migration despite a 503 error, while another was marked fail due to prose instead of a stop token. The author seeks community input on termination detection, loop handling, error handling, and grading methodology.
Meridian48 take
The post highlights a common challenge in agentic AI evaluation: separating model capability from harness reliability, which is critical as agentic systems move toward production.
Read the full reporting
Agentic tool-use eval on a local 35B (Q8): trap-tool avoidance is solid, but I can't tell if my failures are the model or my harness →
DEV Community
agentic-aillm-evaluation