🧪 Hands-On with Playwright Test Agents: Planner, Generator, Healer — Measured

Playwright ships official AI test agents — a planner that explores your app and writes a test plan, a generator that turns the plan into live-verified test code, and a healer that repairs broken tests against the running page. The vendor demos are impressive. What I couldn't find anywhere was a measured answer to the question that actually matters: would you keep the suite these agents produce?

So I measured it. Everything below comes from a public PoC repo you can rerun yourself.

The setup

Target: the-internet.herokuapp.com, the classic demo app. Playwright 1.62.0. Agents initialised with:

npx playwright init-agents --loop=claude

That generates three agent definitions plus a playwright-test MCP server config — the agents run inside Claude Code (VS Code, Codex and opencode loops are also supported).

The brief for both sides was identical: cover login success and failure, dynamic controls (checkbox removal, input enable), and dynamic loading (both examples). The comparison baseline is a 6-test suite written in one pass from knowledge of the app, without touching a browser — the way an engineer who knows an app writes tests. One honesty note: the baseline was authored by the orchestrating model, not a human, so treat the authoring-time comparison as indicative rather than a controlled study.

The results

The healer experiment: I gave it a login test with six deliberately stale locators — wrong field ids, wrong button label, wrong flash selector, wrong heading text — simulating a suite that drifted from its app. It repaired all six in 1m 55s, correctly.

What genuinely impressed me

Live verification catches what knowledge misses. The generator discovered that /dynamic_controls keeps stale, hidden #loading elements in the DOM — a real flake trap — and wrote around it:

// The page keeps stale hidden #loading elements around, so target the visible one
const loadingIndicator = page.getByText('Wait for it...').filter({ visible: true });

My one-pass baseline avoided that trap only by accident, because it happened to assert on different elements. This is the strongest argument for the agents: they test against the app as it is, not as you remember it.

Guardrail judgment. The plan included "attempt to fill the disabled input" — a step that would hang forever against Playwright's actionability checks. The generator recognised that and substituted toBeDisabled() plus toHaveValue('') assertions. Same intent, no flake. That's not template-filling; that's a decision.

The planner discloses scope changes. Asked for two login scenarios, it produced three — and flagged the addition instead of burying it. Small thing, big trust signal.

What the demos won't tell you

The agentic tax is real. Forty-three minutes to generate seven tests, because every locator and assertion is verified in a live browser as it's written. It's unattended time — but it scales with suite size, and if you're picturing "regenerate the suite on every sprint", do that maths first.

Nine assertions per test cuts both ways. The generated tests document the UI exhaustively: button disabled during load, loading indicator visible, message text, element gone, other button back. Thorough — and every one of those intermediate assertions is a maintenance cost when the UI legitimately changes. My baseline's ~2.5 assertions per test encode a judgment about what matters; the agent asserts everything it saw. Nobody has taught it what not to check.

The healer repairs — it does not refactor. This is the finding I'd want every team to see before wiring a healer into CI. My stale CSS ids came back as correct CSS ids, not as the semantic locators a reviewer would ask for. And it matched one button's accessible name literally — { name: ' Login' }, leading space and all. It also did something smart: it cross-referenced the known-good login test elsewhere in the repo rather than guessing. But the pattern is clear: healed code preserves the quality level of the code that broke. Feed it a suite full of brittle selectors and it will faithfully restore your brittle selectors forever.

Verdict

Use the planner and generator to scaffold coverage on apps you don't know well — the live verification genuinely produces better selectors than writing from memory, and the wall-clock cost is unattended. Keep the output under review, because assertion density is a liability you'll own later.

Treat the healer as a colleague who proposes fixes in a PR, not as a silent auto-repair step. It's fast, accurate, and completely uncritical of what it restores — which makes it exactly as good as the suite you already have.

If you're thinking about what your test suite means in a world where agents both write and consume it, that's the thread running through the playbook — start with the cost of a slow test suite, because agent workflows make suite speed matter more, not less.

Everything here is reproducible: qa-gary-parker/poc-playwright-test-agents.