Lets Architect the Performance Engineering

The suite is green and the agent is broken You change one tool definition in your agent — tighten a parameter description, swap the underlying model, reorder two functions in the system prompt. CI runs the eval suite. It comes back at 94%, same as last week. You ship. Two weeks later a support ticket arrives: for a particular class of request, the agent stopped calling the escalation tool entirely. It answers confidently, it never opens the ticket, and the customer quietly churns. You go back to the eval run that approved the change. It's still sitting there at 94%. The regression was in the suite the whole time; the suite just couldn't see it. This is the failure mode nobody warns you about when you start building agents. The eval harness that felt rigorous in month one — a few hundred cases, a pass rate, a green check — gives you confidence that is not earned. Here are four things that actually have to be true before a green eval run means anything. 1. One aggregate p...

Green build, red latency graph The failure mode is quiet by design. A pull request runs the full suite, everything passes, and it merges. Two days later the p99 latency graph for a hot endpoint is 40% higher than it was on Monday, and nobody can point to the cause — because 60 PRs shipped that week and every one of them was green. The regression didn't break a test. It produced exactly the right answer, a little slower each time, and no assertion in the codebase was watching the clock. There's real momentum right now behind autonomous bug-finding harnesses — tools that fuzz, triage, reproduce, and even patch defects with an agent in the loop. That works because a memory-safety bug or a crash is a discrete event a harness can trip over. A performance regression is not that. It's a distribution shift in a number your tests never recorded. If you want to catch it before it reaches production, you have to build a gate that measures time on purpose, and — this is the part pe...