Survive or Notice: What Your Locators Cost and What They Miss

Two questions, not one

Issue 3 ended on mutation testing: coverage measures what tests execute, not what they verify. The two experiments since then split a similar hair for locators. Every locator debate asks which selectors survive UI change. Almost nobody asks what the suite still notices afterwards.

Both experiments use seven tests per suite and public repos you can re-run.

Experiment 3: the healer repaired everything, and that is the problem

I broke a suite with two releases of realistic drift (a redesign, then a CSS-modules migration) and pointed Playwright's healer agent at the failures.

The healer is good. It repaired all 10 failures in 4 minutes 36 seconds, with zero incorrect fixes.

Then I ran the healed suites against the next release:

Healed suite	vs the next release
CSS legacy	0/7, all broken again
ARIA-first	7/7
test-ids	7/7

Across two releases the CSS suite consumed 14 healing repairs. The ARIA-first suite needed 2. The test-id suite needed 1. The healer restores a suite to exactly the fragility it had before. Locator strategy did more for maintenance cost than the AI did.

Full numbers, including the healer's own warning about its repairs: Self-Healing Tests: What the Vendors Aren't Telling You.

Experiment 4: the suite that survived best noticed least

Four suites, and five accessibility regressions a sighted user would not see.

Regressions caught	CSS legacy	test-ids	ARIA-first	ARIA snapshots
Out of 5	1	0	4	5

The test-id suite ran against a build containing all five regressions at once and finished 7/7 green. It was also the suite that survived drift best. Stability and detection are different axes.

ARIA snapshots caught all five, and paid for it in drift: 3/7 after the redesign, second-worst. The repair was cheap, though. --update-snapshots regenerated every stale template in 31 seconds and wrote a patch for review instead of touching the test file.

The blend the numbers support: act with roles and labels, assert with scoped ARIA snapshots where semantics matter, and treat each test-id as a blind spot you chose.

That one is here: ARIA Snapshots: The Assertion Layer Your Locators Are Missing.

Four tools, each with its own page

The experiments keep pointing at the same gaps in a real suite, so I have been building tools for them. Each now has a page with the problem, how it works, requirements and an FAQ:

playwright-smart-reporter: an HTML reporter that groups failures by cause and tracks flakiness across runs. Version 4 is out.
playwright-affected: maps a PR's git diff to one --grep, and runs the full suite whenever it cannot decide.
playwright-shard-balancer: shards by duration, not test count, with no orchestration job.
tracescrub: scrubs secrets from Playwright traces, HAR files and reports before they become CI artifacts. Still a prototype.

All four are MIT-licensed and on npm.

One question to take into your week

"If a redesign quietly broke a screen-reader user's path through our login form tonight, which test would fail?"

If the answer is none, pick one form and add a single scoped toMatchAriaSnapshot() assertion to it. Then change a label and watch what happens.

Stay connected

Join the Discord community and bring the first regression your snapshot catches
Want this thinking applied to your team? Workshops are open for registration
The strategy essays and the measured experiments live in reading order at The AI-Era Quality Playbook

Gary Parker Quality Architect & Strategic Advisor