"It felt better" is not an agent benchmark

How setups drift

You change something in your coding-agent setup: a system prompt, a new rule file, a different model, a new way of assembling context. You try it on a task or two. It feels better. You keep it.

Repeat that for a few months and you have a setup made of changes nobody can defend, and sometimes a slow regression nobody can date. When someone says "it seems worse since we added the rules", there is no way to check, because there was never a baseline to compare against.

Why two runs prove nothing

The judgement fails in a specific, predictable way.

Agent runs are high-variance. The same setup on the same task can succeed on one run and fail on the next. With two samples you cannot tell a real improvement from ordinary noise; you will see "better" about as often as "worse" even when nothing changed.

The tasks you test on are biased. The tasks you reach for after a change are the ones you had in mind while making it, which are exactly the ones the change was designed around. They are the least likely to reveal what the change broke elsewhere.

One number hides the trade-off. A change can raise the success rate while doubling the cost or the time per task, or while producing larger diffs that are harder to review. A single "pass rate" makes those changes look like pure wins.

What a defensible comparison needs

A task set from your own repository. Closed pull requests are the best raw material: each one is a real task with a known good outcome, in your code, with your conventions. Twenty to thirty of them, chosen to cover the kinds of work you actually do, beat any public benchmark for deciding about *your* setup.

Several runs per task. Because a single run is a coin toss, each task is run several times per configuration, in isolation, with the same starting state. What you record is not just pass or fail but what it cost to get there.

More than one metric. Track success, cost, time and the size or reviewability of the change together, so a regression on one axis is visible even when another improves. Each metric can be gamed, which is a reason to watch them side by side rather than optimise one.

An interval, not a number. Report the difference between two configurations with its uncertainty. "Better by somewhere between a little and a lot" and "between a little worse and a little better" lead to different decisions, and a bare average hides which one you are in.

The part that cannot be borrowed

Most of the value is in the task set, and it is also the part nobody can give you: it has to come from your repository. Building it is a day of work. After that, every future change to the setup is a measurement rather than an opinion, and "it seems worse" becomes a question you can answer.

Putting it into practice

The AgentHarness Benchmarking Suite is this method written down: how to build the task set from your own closed pull requests, a runner design covering isolation and sampling, four metrics with how each is computed and gamed, and a scoring model that reports a confidence interval. It also lists the ways an agent benchmark most often misleads.

If the setup you are measuring includes Cursor rules, the governance side of keeping them useful is covered in your Cursor rules have a governance problem, not a rules problem.