Build an AI Agent Evaluation with JEV

Author(s): Quan Huynh Originally published on Towards AI. Build an AI Agent Evaluation with JEV Build a small eval harness for a tool-using AI agent: code checks the work it did, and JEV judges the words it wrote. One run of my incident agent told me a checkout slowdown was caused by a config deploy that shrank the database pool from 50 connections to 5. It was right. The explanation was clear; it cited four tools, and it even ruled out a payment-provider warning that showed up later in the logs. Then I ran it with a shorter system prompt and got an answer that read almost the same. That one was wrong in two ways. It cited a tool it never called, and it never saw the warning it was supposed to rule out. If I had only read the paragraph, I would have shipped it. That is the problem with grading an agent by reading its answer. A good paragraph and a good investigation are two different things. This post builds a small harness that checks both. Plain Python checks the work. Jev, a fast structured evaluation model, judges the explanation. You can clone it and run it in a few minutes. Run it first, read about it later I think the fastest way to understand an eval harness is to watch it…

≈