Lets Architect the Performance Engineering

The suite is green and the agent is broken You change one tool definition in your agent — tighten a parameter description, swap the underlying model, reorder two functions in the system prompt. CI runs the eval suite. It comes back at 94%, same as last week. You ship. Two weeks later a support ticket arrives: for a particular class of request, the agent stopped calling the escalation tool entirely. It answers confidently, it never opens the ticket, and the customer quietly churns. You go back to the eval run that approved the change. It's still sitting there at 94%. The regression was in the suite the whole time; the suite just couldn't see it. This is the failure mode nobody warns you about when you start building agents. The eval harness that felt rigorous in month one — a few hundred cases, a pass rate, a green check — gives you confidence that is not earned. Here are four things that actually have to be true before a green eval run means anything. 1. One aggregate p...

The green cutover with a red p99 The migration goes suspiciously well. An agent reads your EKS cluster, translates every manifest, generates the Terraform for the GKE side, and opens a change for review. You apply it. Every Deployment reports Ready , every synthetic check is green, the smoke tests pass. You cut traffic over. Then real traffic arrives and p99 latency climbs. Not by a rounding error — it roughly doubles on a couple of services and stays there. Nothing is crashing. No pod is CrashLoopBackOff . The dashboards the agent knew how to reason about are all green. The problem is that a Kubernetes manifest describes intent, and the agent proved manifest-equivalence, not behavior-equivalence. Those are different claims, and the gap between them is exactly the part of a migration that only shows up under load. What the agent is genuinely good at This is not a "don't use the tool" post. Agentic migration tooling — GKE's own EKS-to-GKE assistant, or a home-g...

You flip the read path to the new database. Within a minute, a fraction of a percent of lookups start returning 404 , and a handful return data that's a few minutes stale. The write path has been dual-writing to both stores for three weeks. Every write returned 200 . The backfill job logged "complete." By every dashboard you had, the migration was done. It wasn't. The old store and the new store had been quietly drifting apart the entire time, and the cutover is exactly the moment you discover it — when the new store becomes the source of truth for reads and its gaps become user-visible. Why "both writes succeeded" doesn't mean "both stores match" The seductive thing about dual-write is that it looks correct locally. Your write path does something like this: def save_order(order): old_db.upsert(order) # legacy Postgres new_db.upsert(order) # Spanner return 200 Both calls return, you return 200 , and the r...

Green build, red latency graph The failure mode is quiet by design. A pull request runs the full suite, everything passes, and it merges. Two days later the p99 latency graph for a hot endpoint is 40% higher than it was on Monday, and nobody can point to the cause — because 60 PRs shipped that week and every one of them was green. The regression didn't break a test. It produced exactly the right answer, a little slower each time, and no assertion in the codebase was watching the clock. There's real momentum right now behind autonomous bug-finding harnesses — tools that fuzz, triage, reproduce, and even patch defects with an agent in the loop. That works because a memory-safety bug or a crash is a discrete event a harness can trip over. A performance regression is not that. It's a distribution shift in a number your tests never recorded. If you want to catch it before it reaches production, you have to build a gate that measures time on purpose, and — this is the part pe...