playwright-smart-reporter

Playwright's HTML report covers a single run and calls a test flaky only when it fails then passes on retry. Smart Reporter groups failures by cause, shows a causal timeline for each failed test and tracks flakiness across runs.

• Failures with the same fingerprint form one cluster.
• Every failed test gets one time axis that lines up its steps, network requests and console output.
• Flakiness is scored across the runs in history: the share of runs that failed or passed only on a retry.
• AI failure analysis is off by default and runs with your own key or a local model.
• The report is a single-page app that opens over file:// with no server.

• Synthetic benchmark, 10,000 tests: 70 MB of report data (2.x: 401 MB of HTML) and 412 MB peak memory while writing (2.x: 1,116 MB).
• Synthetic benchmark, 14,000 tests: report written in 215 ms with a 478 MB peak. 2.x failed at this size.

@playwright/test 1.40 or newer.

Is it free?

Yes. Everything is MIT-licensed, with no tiers and no license keys. AI analysis is the only feature with an external cost, billed by your AI provider through your own API key. Local models have no per-request cost.

How is flakiness different from Playwright's HTML report?

Playwright marks a test flaky when it fails then passes on retry within a single run. Smart Reporter looks at this run and the runs in history, and also flags results that keep changing between pass and fail.

Can AI analysis fail my test run?

No. If the model is unreachable or runs out of time, the reporter prints one warning, skips the rest of the analysis and still writes the report.

How do I keep history in CI?

History must persist between runs. Caching test-history.json is enough for one job. For sharded runs, set runId so the shards of one CI run record one run.

Does a quality gate fail the run?

Yes. A breached quality gate in the reporter options fails the Playwright run with exit code 1.