AgentHarness Benchmarking Suite

No, deliberately, and that is the central design decision. A borrowed task suite measures a codebase that is not yours, and a public one measures performance on a public benchmark — which models may have memorised. TASK-SUITE.md is a method for building 20 to 30 tasks from your own merged pull requests, where a human already wrote the verification command.

Because a single run returns 0 or 1, so two configurations 15 points apart will look identical or reversed more often than not. Thirty tasks by five runs by two configurations is 300 runs — an afternoon on a suite of three-minute tasks. With one run per task you do not have a benchmark, you have an anecdote in a spreadsheet.

No. It is model-neutral and harness-neutral by necessity: the whole purpose is to compare them, and a benchmark that assumes one cannot. The example runner is Python, a few hundred lines, and translates directly.

The verification step restores every path listed in restore_paths — normally tests/ — from the base commit before running the check, and fails the run outright if a must_not_touch path was modified. This is the guard most home-grown benchmarks omit, and omitting it is what makes a suite reward compliance theatre. It is documented as the way success rate is gamed.