You added a rule after one bad session in March and never questioned it again. ablate runs your agent 80 times — half with the file, half without — and tells you, with a confidence interval, which lines earned their tokens and which ones you can delete tonight.
$0 · no API key · no network · 3 seconds
| task | A | B | Δ | |
|---|---|---|---|---|
| fix-null-deref | 5/5 | 4/5 | +20pp | |
| add-cli-flag | 4/5 | 4/5 | 0 | |
| refactor-parser | 2/5 | 3/5 | −20pp | |
| convention-new-handler | 5/5 | 3/5 | +40pp | |
| restraint-small-fix | 3/5 | 3/5 | 0 | |
| add-readme-badge | 5/5 | 5/5 | 0 | ⚠ no signal — all pass both arms |
| migrate-config | 0/5 | 0/5 | 0 | ⚠ no signal — none pass either arm |
| fix-race-condition | 2/5 | 4/5 | −40pp |
Your 412-line CLAUDE.md costs 4,180 tokens on every turn and does not measurably change whether tasks pass (±5pp, 95% confidence).
This does NOT mean it has no effect — it means any effect is smaller than this suite can resolve. To detect a 3pp change you would need roughly 10 trials × 40 tasks (800 runs, ~$280).
Three 2026 studies measured exactly what this tool measures. Read them together and the conclusion is uncomfortable: the pass-rate effect of config changes is single-digit percentage points, and the best-powered study found nothing at all.
LLM-generated context files reduced task success by roughly 3% across four coding agents, and raised inference cost by more than 20%. Human-written files gained roughly 4%.
No detectable effect on instruction adherence for any config-structure variable, after multiple-testing correction. The best-powered study in the literature found nothing at all.
28.6% median runtime reduction and 16.6% output-token reduction from AGENTS.md presence, with comparable task completion. The cost effect shows up at N=124.
The effect size you can actually afford to measure. Pass rate is not.
What an 8-task × 5-trial run can resolve. A 3pp regression would be invisible, and the report says so.
What it would actually cost to see the 3pp effect the literature measures. Most tools never tell you this.
This is the argument for making token cost the primary metric and pass rate the guardrail. Cost is continuous, has 5–10× the effect size, and needs an order of magnitude fewer trials for the same power.
One markdown file, any agent. No hosted service, no accounts, no telemetry — every trial is stored locally and re-analysable for free.
Every trial gets its own git worktree, the config swapped in, and the agent invoked headlessly with --bare — which skips auto-discovery of hooks, plugins, MCP servers and CLAUDE.md. That flag is load-bearing: it is the only way to guarantee the config under test is the only config in play.
$ ablate compare CLAUDE.md /dev/null --trials 5 worktree .ablate/wt/a-01 ← CLAUDE.md worktree .ablate/wt/b-01 ← (no config) worktree .ablate/wt/a-02 ← CLAUDE.md … 80 trials at --concurrency 4
ablate parses the file into sections, screens them with a randomized subset design on the cheap model, then confirms the survivors in a paired, pre-specified run on the real one.
| Section | Tokens | Δ pass rate | Δ tokens/turn | Verdict | Action |
|---|---|---|---|---|---|
| s11 Historical decisions log | 1,204 | −1pp [−7, +6] | −1,204 (−29%) | ● EQUIVALENT | CUT |
| s06 "IMPORTANT" reminders | 377 | +2pp [−5, +9] | −377 (−9%) | ● EQUIVALENT | CUT |
| s07 Error handling patterns | 311 | −13pp [−21, −5] | −311 (−7%) | ● REGRESSION | KEEP |
The KEEP row is the important one. A tool that always says “delete it” is a random number generator with good UX; this one can say no, and does.
That is the feature, not the bug. Four verdicts, four colours — and EQUIVALENT is deliberately not grey, because “the effect is small” is a finding, not an absence.
The interval is tight and sits inside the margin.
A finding, not an absence — which is why it is deliberately not grey. This is the verdict that licenses you to delete.
The interval excludes zero in the good direction.
The arm you are testing is measurably better on this metric, at this sample size, on this suite.
The interval excludes zero in the bad direction.
Something in the config is load-bearing. In CI this is what fails the PR.
The interval is wider than the margin.
Underpowered. Distinct from EQUIVALENT, and never reported as if it were. The output tells you the price of a sharper answer.
Measured pass-rate effects of config changes in the literature are 3–4pp. Detecting that needs ~2,000+ agent runs — a four-figure bill for a single comparison. Token cost effects are 16–28%, and are comfortably detectable at 200 runs. That asymmetry is the entire argument for making cost the primary metric.
| Tasks | Trials | Agent runs | MDE pass rate | MDE tokens |
|---|---|---|---|---|
| 8 | 3 | 48 | 37pp | — |
| 8 | 5 | 80 | 32pp | ~16% |
| 20 | 5 | 200 | 20pp | ~10% |
| 20 | 10 | 400 | 14pp | ~7% |
| 40 | 10 | 800 | 10pp | — |
| 100 | 10 | 2,000 | 6.4pp | — |
Minimum detectable effect at 80% power, α = 0.05
There is no subscription — ablate is a local CLI billing your own API key. The first thing anyone runs costs nothing, so the decision to spend is made after you have seen what the output looks like.
Best for deciding whether to trust the output before you spend anything.
The verdict will read INCONCLUSIVE, and the report explains why rather than quietly producing a number you might believe.
Best for the question “is this file doing anything at all”. $8 with --cheap.
Best for the question you actually have: which of these 400 lines can I delete.
This spends real money against your own API key. ablate prints the estimate, the run count and the ETA before the confirm prompt — never after — and a hard --max-spend ceiling is honoured even with --yes.
Because it probably did nothing measurable, and most tools in this category will not tell you that. Measured pass-rate effects of config changes are 3–4 percentage points; an 8-task × 5-trial run can only resolve about 32pp. ablate reports the interval instead of a point estimate, so you can see the difference between “no effect” and “no power”. The honest version of this tool spends most of its output saying this. That is the feature, not the bug.
$0 · no API key · no network · 3 seconds