🧬 Your AI's Tests Pass. Here's What They Don't Check
An agent wrote a module and its own tests: 100% line coverage, 90.67% mutation score. What the seven surviving mutants reveal about verifying AI-written code.
There's a claim doing the rounds that AI-written tests cover "about half" of the failure space. Rather than argue with the number, I measured it — and found something more useful than a percentage: a repeatable way to find out what your AI's tests are missing.
The experiment (public repo, rerun it yourself): give a coding agent a deliberately boundary-rich spec — a voucher engine with min-spend thresholds, expiry-inclusive dates, half-up rounding, discount caps and stacking rules — and ask it to implement the module and write the test suite it would ship to production. No hints about what comes next.
The numbers
The agent produced 92 lines of correct implementation and 31 passing tests in under three minutes. The suite scored 100% line coverage. Every dashboard in your org would call this done.
Then I ran StrykerJS over it. Mutation testing makes small changes to the code — flip a comparison, delete a condition, blank a string — and checks whether any test notices. A mutant that survives is a defect your suite would wave through.
Seven mutants survived. Mutation score: 90.67%.
Better than the "about half" discourse suggests — that's worth saying plainly, because the honest finding here is that modern agents write good tests. But look at where the missing 9% lives:
| Surviving mutant | The production defect it represents |
|---|---|
price < 0 → price <= 0 | Free items suddenly throw errors — nothing tested a zero price |
Error messages → "" | Exceptions checked by type only; the messages nobody verified |
| Stackable check deleted | Stackable vouchers quietly join the "best voucher" race |
| Best-discount comparison → always true | The last voucher wins instead of the best one |
> → >= | A tie between equal vouchers flips to the other code |
Boundaries. Tie-breaks. Interactions between rules. Not one of these is exotic — they're the exact anatomy of the pricing incidents that make it to your postmortems. Line coverage couldn't see any of them, because coverage measures what tests execute, not what they verify. The agent's tests walked through every line and asserted the happy outcomes it had thought of. So do most human suites — the difference is that AI generates plausible-looking suites so fast that nobody reads them with the suspicion they'd apply to a colleague's.
The feedback loop is the point
Here's where it gets practical. I fed the seven survivors back to the agent and asked it to strengthen the tests. Six targeted tests later — each named after the defect it prevents — the score hit 98.67%.
The seventh mutant is the finding I didn't expect. The agent analysed it, proved it was equivalent — the mutated code is behaviourally identical to the original for every possible input (the mutation collapses into a comparison with undefined that's always false) — verified that empirically, and declined to write a test for it, recommending a // Stryker disable annotation instead.
If you've run mutation testing on a real team, you know equivalent-mutant triage is exactly where the practice goes to die: someone burns an afternoon proving a mutant can't be killed, the team decides the tool cries wolf, the config quietly disappears. The agent did that triage unprompted, correctly, in seconds. The most tedious objection to mutation testing just became cheap.
What I'd take into your team
- Stop treating coverage as the acceptance bar for AI-written tests. It's the one metric agents saturate by default — every line executed, nothing necessarily verified.
- Make mutation score the honesty check where it counts. You don't need it everywhere; run it on the modules where a silent boundary bug costs real money. That's also where suite investment should concentrate anyway.
- Wire the loop, not the tool. Generate → mutate → feed survivors back → regenerate. Each step is a one-line prompt or a CI job. The gap between 90.67% and 98.67% took one round trip and three minutes of agent time.
- Keep a human on the verdicts. The agent proposed; the numbers and the equivalence proof were checkable. That's the shape of review worth doing — judging evidence, not reading 262 lines of generated assertions.
One module, one agent, one run — indicative, not a benchmark. But the mechanism doesn't depend on the sample size: if your AI writes your tests, something other than the AI has to check what those tests actually hold in place. Mutation testing is the cheapest something I know, and it just got a lot less tedious to operate.
This is a chapter in the wider argument about what your test suite is for in the AI era — because agents don't just write tests now, they rely on them as their feedback loop. A suite that verifies nothing teaches your agents nothing.
Everything above is reproducible: qa-gary-parker/poc-ai-code-test-coverage.