Your Agent Eval Suite Is 94% Green and Still Shipping Regressions

The suite is green and the agent is broken

You change one tool definition in your agent — tighten a parameter description, swap the underlying model, reorder two functions in the system prompt. CI runs the eval suite. It comes back at 94%, same as last week. You ship.

Two weeks later a support ticket arrives: for a particular class of request, the agent stopped calling the escalation tool entirely. It answers confidently, it never opens the ticket, and the customer quietly churns. You go back to the eval run that approved the change. It's still sitting there at 94%. The regression was in the suite the whole time; the suite just couldn't see it.

This is the failure mode nobody warns you about when you start building agents. The eval harness that felt rigorous in month one — a few hundred cases, a pass rate, a green check — gives you confidence that is not earned. Here are four things that actually have to be true before a green eval run means anything.

1. One aggregate pass rate hides everything that matters

A single number across all your cases is the first thing to go. If your suite has 400 cases and 30 of them exercise escalation, that whole capability can collapse from 94% to 60% and move the headline number by about one point. The average is doing exactly what averages do: drowning the signal you care about in the signal you don't.

The fix is to make every case declare which capability it exercises, then report and gate per slice. Keep cases in data, not code, so the tags are cheap to add:

# evals/cases.yaml
- id: refund-lookup-damaged
  capability: refund_policy
  input: "What's the refund window for a damaged item?"
  expect:
    tool_calls: [get_policy]
    judge: "States the 30-day window for damaged goods"

- id: refund-third-request
  capability: escalation
  input: "I want a refund. This is the third time I've asked."
  expect:
    tool_calls: [get_policy, open_ticket]
    judge: "Opens a support ticket and acknowledges the repeat contact"

Now the suite produces a vector, not a scalar, and the escalation collapse has nowhere to hide:

refund_policy        96%  (n=100)
escalation           60%  (n=30)   <-- was 94%
order_status         95%  (n=120)
returns              94%  (n=150)

2. Grade the trajectory, not just the final answer

The second trap is checking only the agent's final text. An agent can produce an acceptable-looking answer through a path that is quietly wrong: it called a destructive tool it wasn't asked to, it made five tool calls where one would do (your token bill and p99 both notice), or — the escalation case above — it skipped the side-effecting call entirely and just described what it would do.

So assert on the sequence of tool calls, not only the prose. The useful contract is usually: the expected side-effecting calls must happen, in a sane order, and nothing destructive may happen that wasn't asked for. Extra read-only calls are fine to tolerate.

WRITE_TOOLS = {"open_ticket", "issue_refund", "cancel_order", "send_email"}

def trajectory_ok(actual_calls, expected_tools):
    names = [c.name for c in actual_calls]

    # every expected tool fired, in the expected relative order
    it = iter(names)
    in_order = all(t in it for t in expected_tools)

    # no un-sanctioned writes slipped in
    surprise_writes = (set(names) & WRITE_TOOLS) - set(expected_tools)

    return in_order and not surprise_writes

This is also where agent evals diverge from the classic prompt-in, string-out LLM test. The thing you are regression-testing is the agent's behavior — which tools, in which order, with which arguments — and behavior is observable in the trajectory, not just the last message.

3. One sample per case is a coin flip, not a test

Agents are non-deterministic. Even at temperature=0, tool-use models drift run to run, and any real system has sampling above zero somewhere. If you run each case exactly once, two bad things happen: a real regression can be masked by a lucky sample, and a stable system can look regressed because of an unlucky one. Your "94% vs 92%" week-over-week delta may be pure noise.

Run each case N times and treat the per-case result as a rate, not a boolean. Then you can gate on something statistically honest — worst-of-N for safety-critical cases, or a pass@k threshold for the rest.

import collections

N = 5  # samples per case; agents are non-deterministic
results = collections.defaultdict(list)

for case in load_cases("evals/cases.yaml"):
    for _ in range(N):
        run = agent.invoke(case["input"])
        ok = (
            trajectory_ok(run.tool_calls, case["expect"]["tool_calls"])
            and judge(run.answer, case["expect"]["judge"]) == "pass"
        )
        results[case["capability"]].append(ok)

for capability, outcomes in sorted(results.items()):
    rate = sum(outcomes) / len(outcomes)
    print(f"{capability:16s} {rate:4.0%}  (n={len(outcomes)})")

N cuts both ways on cost, and this is where the performance discipline comes back in. Two hundred cases at N=5, each with a judge call, is a thousand agent runs plus a thousand judge calls per CI run — real minutes and real money. The pragmatic split that most teams land on: a fast smoke subset (one capability-representative case each, N=1) gates every pull request in under a minute, and the full suite at N=5 runs nightly and on release branches. Run the samples concurrently and cache judge verdicts keyed on (case id, answer hash) so re-runs of unchanged outputs are free.

4. The judge is code, so pin it and test it

If any of your checks use an LLM as the grader — and for anything fuzzier than "did tool X fire," they will — then the judge is a dependency with a version, and an unpinned dependency will move under you. The quiet killer here is a judge-model upgrade: your provider rolls the default model, every groundedness score shifts by a few points overnight, and you cannot tell a real regression from a re-scored baseline. Pin the exact judge model and prompt, and treat a bump as its own change with its own review.

JUDGE_MODEL = "claude-haiku-4-5-20251001"  # pinned; bumping this is a PR

def judge(answer, rubric):
    raw = judge_client.complete(
        model=JUDGE_MODEL,
        temperature=0,
        prompt=JUDGE_PROMPT.format(rubric=rubric, answer=answer),
        response_format={"type": "json_object"},
    )
    return json.loads(raw)["verdict"]  # "pass" | "fail"

And because the judge decides what counts as a regression, it needs its own regression test: a small golden set of answers you have hand-labeled pass/fail. If the judge's agreement with your labels drops below a threshold, fail the build on the judge before you trust a single capability number it produced. A judge you never check is just a second un-evaluated model grading your first one.

Wiring it into CI as a gate

The per-capability rates only prevent regressions if the build fails when one of them drops. Commit a baseline — the per-capability rates you have agreed are acceptable — and compare each run against it, slice by slice:

import json, sys

baseline = json.load(open("evals/baseline.json"))   # per-capability rates
REGRESSION = 0.10   # fail if any slice drops more than 10 points

failed = []
for capability, outcomes in results.items():
    rate = sum(outcomes) / len(outcomes)
    prev = baseline.get(capability, 0.0)
    if rate < prev - REGRESSION:
        failed.append(f"{capability}: {prev:.0%} -> {rate:.0%}")

if failed:
    print("EVAL REGRESSION:\n  " + "\n  ".join(failed))
    sys.exit(1)

With this gate in place, the escalation example fails loudly: escalation: 94% -> 60%, non-zero exit, red check, merge blocked. The baseline file is reviewed like any other config — raising a slice's floor is a deliberate commit, and lowering one requires a human to say "yes, we are accepting this."

Lesson

A green eval run is only as honest as the number behind the green. The four things that make it honest are unglamorous: slice by capability so no single skill can collapse unnoticed, grade the path and not just the destination, run each case enough times that you are measuring behavior instead of luck, and pin the judge so your ruler stops changing length. None of this is specific to any one framework — it is ordinary test engineering, applied to a system whose outputs happen to be probabilistic. The uncomfortable part is that the probabilistic part is exactly why the discipline matters more here, not less.

---

Hitting something like this in production? I help teams with performance engineering, SRE/observability, and AI-driven root cause analysis — work with me.