Two experiments in, the pattern is already clear

Issue 2 made a promise: no claims here without reasoning you can check against your own suite. Since then I've run two public experiments — real repos, real numbers, everything reproducible — and they point at the same conclusion from opposite directions.

AI is genuinely good at producing test artefacts. It has no opinion about whether they're the right ones. That judgment is now the job.

Experiment 1: Playwright's test agents, measured

Playwright ships official AI agents — a planner, a generator, and a healer. The demos are impressive; nobody had published a practitioner's evaluation. So: same brief to the agents and to a one-pass baseline, against a public demo app.

The generator took 43 minutes to produce 7 tests — because it verifies every locator against a live browser as it writes. That live verification is the genuinely new thing: it caught a real flake trap (stale hidden loading elements) that writing from memory misses. But it also asserted everything it saw — 9 assertions per test vs my baseline's 2.5 — and every one of those is a maintenance cost when the UI legitimately changes. Nobody has taught it what not to check.

Full numbers and the healer's very revealing behaviour: Hands-on with Playwright Test Agents.

Experiment 2: what AI-written tests actually verify

A coding agent built a module and wrote the test suite it considered production-complete. The suite scored 100% line coverage. Then StrykerJS mutation testing asked the harder question — if the code silently changed, would anything fail?

Seven mutations survived. All seven were boundary conditions and rule interactions — exactly where production incidents live. Coverage measured what the tests execute; mutation measured what they verify. The gap between those two numbers is where "our AI writes the tests now" quietly becomes a liability.

The redemptive part: fed the surviving mutants back, the agent fixed six with targeted tests and correctly proved the seventh unkillable rather than gaming the score. The loop works — but someone has to run it.

That one's here: Your AI's Tests Pass. Here's What They Don't Check.

One question to take into your week

"If a bug were quietly introduced into our code tonight, what evidence do we have that the suite would fail?"

Coverage is not that evidence — experiment 2 is a controlled demonstration of why. If you've never run mutation testing, pick your most business-critical module and run one Stryker (JS/C#) or PIT (JVM) pass over it this week. The first survivor report is uncomfortable in the most useful way.

Next issue

I'm currently mid-experiment on the loudest claim in every 2026 trend report: self-healing tests. Early data says the vendors are leaving out the most important part. Numbers when they're ready.

Stay connected

• Join the Discord community — bring your first mutation-testing survivor report
• Want this thinking applied to your team? Workshops are open for registration
• The strategy essays live in reading order at The AI-Era Quality Playbook

See you in the next issue.

Gary Parker Quality Architect & Strategic Advisor