🌳 ARIA Snapshots: The Assertion Layer Your Locators Are Missing

Every locator debate — CSS vs test-ids vs accessible roles — is about the same thing: which selectors survive UI change. Last week's self-healing experiment put numbers on that. But survival is only half the job of a test suite, and optimising for it hides a question almost nobody asks: what does your suite still notice?

Playwright's ARIA snapshots — toMatchAriaSnapshot, quietly getting better every release — assert on the accessibility tree instead of individual elements. I added them as a fourth strategy to the same public testbed and measured both halves: what breaks under drift, and what gets caught when the app genuinely degrades. Everything below is reproducible from the PoC repo.

The setup

Same TaskBoard app as the self-healing experiment: v1, then a redesign (v2), then a CSS-modules migration (v3). Same 7 scenarios, now written four ways — CSS-legacy selectors, test-ids, ARIA-first locators, and a snapshot suite where actions use user-facing locators but every assertion is a scoped YAML template:

await expect(page.getByRole("list")).toMatchAriaSnapshot(`
  - list:
    - listitem:
      - text: Write report
      - button "Done"
    - listitem:
      - text: Ship release
`);

New for this experiment: five extra app builds, each seeding one user-facing accessibility regression that changes nothing visually. A heading demoted from h2 to h4 (styled to look identical). List markup flattened to divs. A button swapped for a clickable div. role="alert" deleted. A label's for attribute broken. Real regressions, the kind a refactor ships on a Friday.

Half one: drift survival — snapshots lose

Second-worst, and that's the honest headline. Two failures were shared with the ARIA-first suite (the redesign changed two accessible names, breaking the action locators both suites use). The other two were pure snapshot economics: a template asserts an entire subtree, so any drift inside it — a div becoming a nav, a button growing an icon — fails the test. Assertion breadth is a liability under drift. Exactly like the generator's nine-assertions-per-test in the test-agents experiment, except here it's a choice you control with template scope.

But now look at the repair bill. npx playwright test --update-snapshots regenerated every stale template in 31 seconds — and Playwright doesn't touch your test file. It writes test-results/rebaselines.patch and leaves the decision to you. Apply, fix two action locators (one line each), and the suite passed v2 — and then v3 with zero further changes. The healer, on the same testbed, edited code directly and had to re-heal the CSS suite from scratch every single release. Patch-for-review versus silent code edits is a philosophical difference worth an entire tooling decision.

One trap: the rebaselined templates come back full-tree, not the minimal partials you wrote. The tooling trades your curated assertion for maximum density. Trim after applying, or your next drift bill doubles.

Half two: regression detection — snapshots sweep

Five invisible accessibility regressions, four suites:

Sit with the test-ids column. I ran that suite against a build containing all five regressions at once: 7/7 green. A screen-reader user can no longer operate the login form, the task list has no list semantics, a control isn't keyboard-reachable — and the most drift-resistant suite in the comparison has no idea. It survives refactors precisely because it looks at a private contract instead of anything a user experiences.

That's the finding I'd put on a slide: survival and vigilance are opposite ends of the same dial. The suite that never breaks is the suite that never notices. When someone shows you a suite with a spotless pass-rate through heavy UI churn, ask what it's actually looking at.

The ARIA-first suite did well — 4/5 — with one instructive miss: getByRole("heading", { name: "My Tasks" }) cheerfully matches an h4. Role locators check existence, not structure. The snapshot template pins [level=2], which is why only it caught the demotion.

The agent angle

The same tree your assertions check is now the interface agents use. Playwright's AI-mode snapshot adds element refs and viewport boxes:

- textbox "New task" [ref=e6] [box=400,168,153,21]
- button "Add" [ref=e7] [box=557,165,52,27]
- list [ref=e12] [box=400,229,480,96]:
  - listitem [ref=e13]:
    - text: Write report
    - button "Done" [ref=e15] [box=832,235,48,21]

Roles and names for semantics, refs for interaction, boxes for layout — this is what an MCP-driven agent sees instead of your DOM. Which makes the accessibility tree load-bearing twice over: if your app is illegible in this format, it's broken for assistive tech and opaque to every agent you point at it. The five regressions above wouldn't just fail users — they'd degrade the snapshot every agent depends on. Accessibility just became agent infrastructure.

Verdict

Locator strategy and assertion strategy are different axes, and most teams only tune the first. The blend the numbers support:

1. Act with user-facing locators (roles, labels) — they had the best survival-to-meaning ratio across both experiments.
2. Assert user perception with scoped ARIA snapshots on the regions where semantics matter — forms, alerts, navigation, anything with structure. Keep templates partial; scope is your drift-cost dial.
3. Reserve test-ids for genuinely unstable regions — knowing each one is a blind spot you chose.
4. Treat --update-snapshots + patch review as your maintenance loop. It's faster than healing, and unlike healing, a human sees every change before it lands.

This is chapter-adjacent to the suite-economics thread in the playbook: assertions are an investment with a maintenance cost and a detection yield, and you should know where yours sit on that curve.

Everything here is reproducible: qa-gary-parker/poc-aria-snapshot-assertions.