🩹 Self-Healing Tests: What the Vendors Aren't Telling You

Every 2026 trend report leads with self-healing test automation. The pitch is always the same: your UI changes, your tests fix themselves, locator maintenance disappears from your backlog. It's the loudest claim in test tooling right now, and almost nobody selling it publishes numbers.

So I ran the experiment. Everything below comes from a public PoC repo you can rerun yourself. Spoiler: the healer performed better than I expected — and the sales pitch still doesn't survive contact with the data.

The setup

A small task-manager app in three versions, simulating two releases of realistic drift:

• v1 → v2: a redesign sprint. Every id and class renamed, two buttons become icon buttons (their accessible names change), one data-testid renamed.
• v2 → v3: a CSS-modules migration. Every class hashed, ids regenerated — but accessible names and test-ids untouched.

Behaviour is identical across all three. Only the markup drifts — the failure mode self-healing exists to solve.

Against v1, I wrote the same 7 scenarios three times, differing only in locator strategy: CSS-legacy (ids and classes, the way most real suites are written), ARIA-first (roles, labels, accessible names), and test-ids. The healer is Playwright's official test-healer agent, run headless.

What two releases of drift did

Note the CSS suite's failure mode: not degradation, annihilation. The login form's three selectors alone took out all seven tests, because every test logs in first. If your suite shares brittle fixtures — and every real suite shares fixtures — locator debt doesn't accumulate linearly. It detonates.

The healer is genuinely good

Round one: I pointed the healer at the 10 failures against v2. It repaired all 10 in 4 minutes 36 seconds, zero incorrect fixes, zero flaky repairs. It debugged each failure against the live page, found the renamed element, and updated the locator.

It even respected each suite's idiom — CSS selectors were repaired as CSS selectors, accessible names as accessible names, test-ids as test-ids. It read the suite's style and stayed in its lane. As a tool, this is impressive and I want to be fair to it: the healer did its job perfectly.

Which is exactly the problem.

The finding the demos won't show you

Then I ran the freshly-healed suite against v3 — the next release:

The healed CSS suite survived exactly one release. #username had been faithfully repaired to #field-user, which promptly became #fld_9a2b. Round two of healing took another 3 minutes 23 seconds and repaired all seven again — this time to hashed CSS-modules classnames like .Login_submit__w6f4. The ARIA and test-id suites, healed once, never needed touching again.

And here's my favourite detail of the whole experiment. The healer knew. Its own round-two report ends with a warning that the repaired classnames are "stable for this build but likely to regenerate on the next rebuild, so this legacy-selector suite remains structurally fragile." The healer diagnosed the disease, then prescribed another dose — because translation is the only move it has. No vendor demo will ever show you that sentence.

This is the mechanism the marketing leaves out: a healer restores your suite to exactly the fragility it had before. It's a locator-translation machine, not a quality-improvement machine. Feed it brittleness and you get freshly-broken brittleness back next release — forever. The vendors call this "eliminating test maintenance." It's actually a subscription: healing services your locator debt, on every release, without ever paying down the principal.

Count the repairs across two releases: the CSS suite consumed 14 healing repairs. The ARIA-first suite needed 2. The test-id suite needed 1. Locator strategy — the boring, decade-old advice — did more for maintenance cost than the AI did.

The subtler problem: healing can eat signal

Look at what broke in the ARIA suite: two buttons whose accessible names changed when they became icon buttons (Done → Complete, Add → Add task). The healer silently repaired both.

But an accessible-name change is a user-facing change. Screen-reader users experience it. A test failing on it is arguably doing its job — surfacing that the redesign altered what users perceive. A healer that auto-repairs it has quietly decided, on your behalf, that this change doesn't matter. Same for the renamed data-testid: that's a broken contract between the app and the suite, and the right fix might be in the app, not the test. Healing makes both failures disappear without anyone deciding anything.

Verdict

Three things I'd want any team to take from the numbers:

1. Buy the tool, not the pitch. Playwright's healer is fast, accurate, and honest about its work — as a proposer of fixes in a PR, it's genuinely useful. The claim that self-healing eliminates test maintenance is what fails the measurement.
2. Healing is a complement to locator strategy, not a substitute. If the healer keeps repairing the same suite every release, that's not the tool working — that's a dashboard telling you where your locator debt lives. The fix for #username isn't #field-user; it's getByLabel('Username').
3. Keep a human on the loop for why it broke. Some failures are drift. Some are the app changing what users actually experience. Only one of those should be silently repaired, and the healer can't tell them apart.

This is the same conclusion the Playwright test agents experiment reached from the other side: healed code preserves the quality level of the code that broke. AI removes the toil from test maintenance — it doesn't remove the judgment. That division of labour is the thread running through the playbook.

Everything here is reproducible: qa-gary-parker/poc-self-healing-selectors.