Finding Flaky Playwright Tests That Fail Every Retry

Playwright calls a test flaky when it fails and then passes on a retry in the same run. A test that fails every attempt in a run is never flaky in that run's report, even if it passes in the next run. Retries only catch tests that recover within the run. In 11 runs of a small suite, one such test failed 4 times, and Playwright's flaky count for it was 0 every time. A reporter that keeps history marked it flaky at run 5.

The suite, the script and every result are in the PoC repo.

The setup

• @playwright/test 1.65.1 and playwright-smart-reporter 4.0.1, on Node 24 and macOS.
• 20 tests with retries: 1. The tests use no browser.
• 16 tests always pass. The other 4 read the run number, so every outcome is repeatable:

• A script runs the suite 11 times with one history file. After each run it reads stats.flaky from Playwright's JSON report, and each test's status from the smart reporter's JSON export.

The reporter part of the config (workers: 4 and the dot reporter are left out):

import { defineConfig } from '@playwright/test';
 
export default defineConfig({
  retries: 1,
  reporter: [
    ['json', { outputFile: 'out/report.json' }],
    ['playwright-smart-reporter', {
      outputFile: 'out/smart-report.html',
      historyFile: 'out/test-history.json',
      maxHistoryRuns: 10,
      exportJson: true,
      quarantine: { enabled: true, outputFile: 'quarantine.json', threshold: 0.3 },
    }],
  ],
});

What a single-run report sees

Playwright's flaky count was 1 in runs 3 and 8, and 0 in the other 9 runs. Both times it was the retry-flaky test, which failed once and passed on the retry.

The intermittent test never appeared in that count. In runs 2, 5, 7 and 10 it failed both attempts, so each of those reports lists it as failed. In the other 7 runs it passed on the first attempt, so each of those reports lists it as passed. No single report has the information to call it flaky.

I ran only retries: 1. A test that fails every attempt in a run would also fail with more retries.

What a report with history sees

The smart reporter gives each test a score and a status. The score is the share of runs that failed or passed only on a retry. The status comes from how often the result changed between pass and fail from one run to the next, plus the runs that passed only on a retry. The reporter does not count the first change.

• The intermittent test is flaky from run 5 to run 11. In runs 2 to 4 the reporter says insufficient-history: it wants 5 runs before it gives a status.
• The retry-flaky test is flaky in the two runs where it passed on a retry, and unstable in the other runs from run 5.
• The broken test is failing. It fails every run, and that is not flaky.
• The fixed test is stable from the run that fixed it, although 4 of its first 5 runs failed.

At run 11 the intermittent test and the fixed test have the same score, 0.36. Their statuses differ because the fixed test changed result once, and the reporter does not count a first change. The intermittent test changed 8 times. A failure rate on its own cannot separate a flaky test from a test that was broken and then fixed.

Quarantine

With quarantine.enabled, the reporter writes the tests that are flaky with a score of at least the threshold to a JSON file:

• The intermittent test is in the file from run 5 to run 11.
• The retry-flaky test is in the file in run 3 only. In run 8 its score was 0.25, below the threshold of 0.3.
• The broken test and the fixed test are never in the file.

The file is a record. The reporter does not skip a quarantined test: the intermittent test still ran in every run, and its failures in runs 7 and 10 still counted as failures. Every run in this suite exits with code 1 anyway, because the broken test fails every time. To act on the file, read it in a CI step.

Keep the history between runs

All of this depends on the history file. In CI, cache test-history.json between runs. For sharded runs, set runId so the shards of one CI run record one run, and keep the test-history.runs folder too.

What I'd do with it

• Keep retries for what they are good at: a test that recovers inside the run.
• Do not read a flaky count of 0 as "no flaky tests". It counts recoveries inside one run.
• Track results across runs, and judge a test by how often its result changes, not by its failure rate.
• Quarantine from the status, not from a failed run, so a test that is plainly broken stays visible as broken.

The limits: the outcomes are scripted, and real flaky tests do not fail on a schedule. The suite has no browser and no network. The numbers come from one version of each tool with the reporter's default thresholds. With maxHistoryRuns: 10 each run is scored against up to 11 runs, so a longer series would drop run 1 and change the scores. I did not test how quarantine interacts with quality gates. The reporter is my own project.

Repo: qa-gary-parker/poc-playwright-flaky-detection. The reporter is at playwright-smart-reporter. More measured experiments are in the playbook.