The Build Was Green. p99 Climbed 40% Anyway. A Perf-Regression Gate an Agent Can Explain.

Green build, red latency graph

The failure mode is quiet by design. A pull request runs the full suite, everything passes, and it merges. Two days later the p99 latency graph for a hot endpoint is 40% higher than it was on Monday, and nobody can point to the cause — because 60 PRs shipped that week and every one of them was green. The regression didn't break a test. It produced exactly the right answer, a little slower each time, and no assertion in the codebase was watching the clock.

There's real momentum right now behind autonomous bug-finding harnesses — tools that fuzz, triage, reproduce, and even patch defects with an agent in the loop. That works because a memory-safety bug or a crash is a discrete event a harness can trip over. A performance regression is not that. It's a distribution shift in a number your tests never recorded. If you want to catch it before it reaches production, you have to build a gate that measures time on purpose, and — this is the part people get wrong — you have to be very deliberate about where an LLM belongs in that gate.

Why the pipeline didn't catch it

Three things conspire here, and it's worth naming them because they each shape the fix.

• Correctness tests don't measure time. A function that got 3x slower still returns the same value. Green means "correct," not "fast."
• Ad-hoc benchmarks are noisy, so people ignore them. Run a microbenchmark twice on a shared CI runner and you'll see 5-15% swings from nothing but neighbor load and CPU frequency scaling. A gate that fails on raw percent-change cries wolf, gets muted, and then it's decorative.
• Nobody can read 60 differential flamegraphs a week. Even teams that do profile in CI drown in the output. The signal exists; the human attention to correlate it back to a specific diff does not.

So the requirements fall out naturally. The gate has to (1) measure latency deterministically enough to fail a build honestly, (2) decide significance with statistics instead of a bare threshold, and (3) hand a human a root cause instead of a wall of profile data. The first two are a solved problem with boring, reliable tools. The third is where AI actually earns a place — as long as you keep it out of the blocking decision.

The gate: the machine decides, the agent explains

The architecture I'd reach for today has a strict division of labor. Deterministic tooling owns the pass/fail call. The LLM only ever runs after that call is made, and only to explain it. You never want a hallucination holding a merge hostage, and you never want an agent's confidence standing in for a p-value.

Step 1: benchmark both sides, decide with statistics

Run the benchmarks twice — once at the PR's merge base, once at its head — and compare them with a tool that understands variance. In Go, that pairing is the standard benchmark harness plus benchstat, which runs a Mann-Whitney U test and reports a delta only when the difference is statistically significant. The same shape applies in any ecosystem: pytest-benchmark --benchmark-compare-fail in Python, Criterion plus critcmp in Rust. The principle is what matters — repeat the measurement, and let a significance test, not a single sample, make the call.

name: perf-gate
on: pull_request

jobs:
  bench:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: actions/setup-go@v5
        with: { go-version: '1.23' }

      # Benchmark the merge base.
      - name: bench base
        run: |
          git checkout ${{ github.event.pull_request.base.sha }}
          go test -run '^$' -bench . -benchmem -count 10 \
            -cpuprofile base.prof ./internal/parser | tee base.txt

      # Benchmark the PR head.
      - name: bench head
        run: |
          git checkout ${{ github.sha }}
          go test -run '^$' -bench . -benchmem -count 10 \
            -cpuprofile pr.prof ./internal/parser | tee pr.txt

      - name: compare
        run: |
          go install golang.org/x/perf/cmd/benchstat@latest
          benchstat base.txt pr.txt | tee delta.txt
          # Fail only on a significant regression above threshold.
          python3 scripts/gate.py delta.txt --max-regression 8

A few details do the heavy lifting. -count 10 gives benchstat enough samples to separate signal from runner noise. Pinning the whole thing to a single hot package (./internal/parser) keeps profiles clean and runs fast. And the threshold — 8% here — should sit comfortably above your runner's measured noise floor, which you find by benchmarking main against itself a few times and looking at the spread. Set the threshold below the noise floor and you're back to crying wolf.

Step 2: a differential profile says where

benchstat tells you a benchmark got slower. It does not tell you which function. That's what the CPU profiles are for — and pprof can diff two of them directly:

go tool pprof -diff_base base.prof -top -nodecount 15 pr.prof
#      flat  flat%   sum%        cum   cum%
#   180ms  36.0%  36.0%      180ms  36.0%  regexp.(*Regexp).MustCompile
#    60ms  12.0%  48.0%       60ms  12.0%  runtime.growslice
#   ...

The + entries are functions that consumed more time in the PR than in the base. When regexp.MustCompile jumps to the top of a diff profile, an experienced engineer knows the shape immediately: a regex that used to be compiled once is now being compiled on every call, almost always because a MustCompile got moved inside a loop or a per-request function. But that recognition takes context and time, and it's exactly the correlation that doesn't scale to every PR.

Step 3: let the agent say why — but never let it gate

Now the LLM has a job it's genuinely good at: correlation across three artifacts it can all see at once — the benchstat delta, the differential profile, and the actual code diff. Grounded in the profile, it isn't guessing; it's tying a named hot function to a specific hunk. This step runs only after the deterministic gate has already failed the build, so nothing about the merge decision depends on the model.

import subprocess, os, anthropic

delta = open("delta.txt").read()
diff_profile = subprocess.run(
    ["go", "tool", "pprof", "-diff_base", "base.prof",
     "-top", "-nodecount", "15", "pr.prof"],
    capture_output=True, text=True).stdout
code_diff = subprocess.run(
    ["git", "diff", os.environ["BASE_SHA"], os.environ["HEAD_SHA"]],
    capture_output=True, text=True).stdout

prompt = f"""You are a performance engineer reviewing a pull request.

benchstat (base vs PR), significant deltas only:
{delta}

Differential CPU profile (pprof -diff_base; '+' = more time in the PR):
{diff_profile}

The code change:
{code_diff}

Tie the single largest profile mover to the specific hunk in the diff that
explains it, and propose the minimal fix. Cite the function and the file:line.
If the profile does not support a clear cause, say so plainly. Do not
speculate beyond what the profile and diff show."""

msg = anthropic.Anthropic().messages.create(
    model="claude-sonnet-5", max_tokens=800,
    messages=[{"role": "user", "content": prompt}])
print(msg.content[0].text)  # posted back as a PR comment

The output is the comment you actually wanted on the PR: "BenchmarkParse regressed ~38% (p=0.002). The differential profile is dominated by regexp.MustCompile, which lines up with parser.go:74, where the pattern is now compiled inside parseLine instead of at package init. Hoist it back to a package-level var." A human can accept or dismiss that in ten seconds, which is the whole point.

Where the agent earns its keep, and where it doesn't

The line to hold is that the model explains, it does not adjudicate. Give it only the number — "this got 38% slower" — with no profile, and it will happily invent a plausible-sounding cause that has nothing to do with reality; the grounding artifacts are what keep it honest. It's also poor at deciding whether a regression is worth failing over. That's a judgment about your latency budget and traffic, and it belongs in the deterministic threshold you set, not in a prompt. And it can be actively misled by a coincidental diff — a large unrelated refactor in the same PR — which is another reason the differential profile, not the diff alone, has to anchor its reasoning.

Used inside those lines, though, it closes the loop that made perf gates fail in practice. The reason teams turn off profiling-in-CI isn't that the data is wrong; it's that nobody has the hours to read it. An agent that reliably converts a diff profile into a one-paragraph root cause is what makes the gate something people keep on instead of muting.

Lesson

The autonomous-harness wave is real, but the right place for the model in a performance gate is narrow and specific: after the statistics, never instead of them. Let a significance test own pass/fail so the gate can be trusted to block a merge, and let the LLM own the explanation so the failure is actionable instead of just annoying. Split it the other way — an agent deciding whether your build is fast enough — and you've traded a flaky benchmark for a confident hallucination, which is a worse trade than the one you started with.

---

Hitting something like this in production? I help teams with performance engineering, SRE/observability, and AI-driven root cause analysis — work with me.