Playwright --update-snapshots Overwrote a Good Baseline and Passed: 226 of 226 Runs
On a page that never stops moving, --update-snapshots=changed or all wrote a random frame as the baseline and passed, on Playwright 1.40 to 1.65.1. Masking the container stopped it.
In 2024 I wrote about a CI job that updates Playwright baselines: it runs npx playwright test --update-snapshots and opens a pull request with the new images. A recent Playwright issue, #44128, says that on a page that never becomes stable, that command can replace a good baseline with a random frame and still pass.
I measured it. On a page with a JS animation or a live clock, every update run passed and overwrote the baseline: 226 of 226. Without the update flag, the same pages failed every time. It happens on every version I tested, from 1.40.0 to 1.65.1. The issue reporter first saw it on 1.59.1. Masking the moving element did not stop it for the animation. Masking its container did.
The code and every run are in the PoC repo.
The setup
One test loads a small "Build status" card and calls toHaveScreenshot with the default options: a 5 second timeout, and CSS animations turned off.
test('build status card', async ({ page }) => {
await page.setContent(html);
await expect(page).toHaveScreenshot('card.png');
});There are four versions of the card:
| Page | What moves |
|---|---|
| stable | nothing |
| css-animation | the progress bar, with a CSS animation |
| js-animation | the progress bar, moved by requestAnimationFrame |
| live-clock | an "Updated 12:00:00.000" line, updated every 20 ms |
Each run has three steps:
- Write a good baseline from the stable card.
- Run one version of the card with one update setting:
none,missing,changedorall. - Run the stable card again with no update. This is the next normal CI run.
After step 2, I compare the baseline with the good one and count the pixels that changed.
I ran 1.58.2, 1.59.0, 1.59.1 and 1.65.1 with all 4 pages and all 4 settings, 10 times each. I ran 7 older versions, back to 1.40.0, with the two moving pages, 3 times each. On the older versions I ran missing and all, plus changed from 1.50, but not none. Before 1.50 the flag took no value, so there -u stands for all.
The results
| Setting | Runs | Passed | Baseline overwritten | Next normal run failed |
|---|---|---|---|---|
none or missing | 202 | 0 | 0 | 0 |
changed or all | 226 | 226 | 226 | 226 |
These are the two moving pages, on all versions. Without an update, every run failed: from 1.49.1 with "Failed to take two consecutive stable screenshots", on 1.40.0 and 1.45.0 with "Screenshot comparison failed". With an update, every run passed with exit code 0 and wrote a new baseline. The next normal run of the unchanged card then failed, because it was compared with a frame of the animation.
This is the good baseline:

These are the baselines after one update run on each moving page:


On the animation, 692 to 1,412 pixels changed. On the clock, 114 to 231.
The stable and CSS animation pages passed in all 320 runs, and no baseline pixel changed. Playwright turns off CSS animations before it takes the screenshot, so that page was stable.
No version behaved differently. The issue reporter first saw it on 1.59.1. 1.40.0, 1.45.0, 1.49.1, 1.50.0, 1.52.0, 1.55.0, 1.57.0 and 1.58.2 did the same.
Why it happens
toHaveScreenshot takes screenshots until two in a row match. Then, except with all, it compares the last one with the baseline. If the page never gives two matching screenshots before the timeout, the matcher gets an error and the last screenshot it took.
With changed or all, the matcher writes that screenshot as the baseline whenever it gets an error and a screenshot, and passes. It does not check which kind of error it was. A page that never became stable takes the same path as a page that really changed. The same branch is in the 1.49.1 source for -u.
What stopped it
I ran two mask settings on 1.65.1, 10 times each. These are the update runs (changed or all) on the two moving pages:
| Mask | Overwritten (of 40) |
|---|---|
| none | 40 |
#bar and #clock, the moving elements | 20, all on the animation |
.track and #clock, the bar's container and the clock | 0 |
Masking the clock worked, because its box does not move. Masking the progress bar did not. The mask box moves with the bar, so the likely cause is that two screenshots still never match. I did not measure that step. Masking the track that the bar moves in worked.
await expect(page).toHaveScreenshot('card.png', {
mask: [page.locator('.track'), page.locator('#clock')],
});The limits
- One test, one small page, Chromium only, one machine.
- Each run writes its baseline just before the measured run. A real update job runs against a committed baseline, but it takes the same code path.
- The older versions ran 3 times per setting, not 10.
What I'd do with it
- Treat a green
--update-snapshotsrun as "files written", not "the page was stable". The exit code does not tell you which. - After an update, run the same tests once more without the flag before you commit the new baselines. In this experiment, a run without the flag failed on the moving pages every time.
- Mask the area that a moving element moves in, not the element itself.
- In a job that opens a pull request with new baselines, look at the images, not only the green check.
Repo: qa-gary-parker/poc-playwright-update-snapshots-unstable. The ARIA snapshots experiment measures regressions that passing suites missed. More are in the playbook.
related
QA Ark Newsletter
Test automation, AI in testing and quality engineering practice. No spam, unsubscribe any time.