← blog

Sharding Playwright in GitHub Actions With One Merged HTML Report

A Playwright suite split over 4 GitHub Actions jobs, then merged into one HTML report. With one failing test, the merge job still ran and the report showed the failure with its trace.

Sharding splits a Playwright suite over several CI jobs, so each job runs part of it. Each job then has its own report, and nobody wants to open four. This guide sets up a 4-job shard matrix in GitHub Actions and merges the results into one HTML report. I ran it green, then with one broken test: the failing shard went red, the merge job still ran, and the merged report showed the failure with its trace.

The workflow, the app and both runs are in the demo repo.

The setup

  • A small checklist app in site/, served by node site/server.mjs.
  • 17 tests in 5 spec files. Three of the export tests wait about 3.5 s for an export to finish. The other tests take under 0.4 s.
  • @playwright/test 1.65.1, Chromium only, fullyParallel: true, retries: 2 on CI.
  • The workflow follows the Playwright sharding docs, with current action versions: checkout and setup-node v7, upload-artifact v7, download-artifact v8.

The config sends results to the blob reporter on CI and to the HTML reporter locally:

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  reporter: process.env.CI ? 'blob' : 'html',
  use: {
    baseURL: 'http://127.0.0.1:3000',
    trace: 'on-first-retry',
  },
  projects: [{ name: 'chromium', use: { ...devices['Desktop Chrome'] } }],
  webServer: {
    command: 'node site/server.mjs',
    url: 'http://127.0.0.1:3000',
    reuseExistingServer: !process.env.CI,
  },
});

The workflow

name: Playwright Tests
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  workflow_dispatch:
jobs:
  playwright-tests:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: lts/*
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
      - uses: actions/upload-artifact@v7
        if: ${{ !cancelled() }}
        with:
          name: blob-report-${{ matrix.shardIndex }}
          path: blob-report
          retention-days: 1
 
  merge-reports:
    # Merge even when a shard fails, so the failure shows in the report
    if: ${{ !cancelled() }}
    needs: [playwright-tests]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: lts/*
      - run: npm ci
      - uses: actions/download-artifact@v8
        with:
          path: all-blob-reports
          pattern: blob-report-*
          merge-multiple: true
      - run: npx playwright merge-reports --reporter html ./all-blob-reports
      - uses: actions/upload-artifact@v7
        with:
          name: html-report--attempt-${{ github.run_attempt }}
          path: playwright-report
          retention-days: 14

Four lines that matter

  • fail-fast: false. By default, GitHub cancels the rest of a matrix when one job fails. Then the other shards never finish and the merged report is missing their tests. With fail-fast: false, every shard runs to the end.
  • reporter: 'blob' on CI. A blob report is a zip of raw results, not a report you read. merge-reports needs blob reports as input. Each shard uploads its blob-report folder as blob-report-<index>.
  • if: ${{ !cancelled() }}, on the upload step and on the merge job. By default, a step or job runs only if everything before it passed. Without this line, a red shard skips its upload, and a red shard also skips the merge job. That is the run where you need the report most. !cancelled() still skips both when someone cancels the run.
  • merge-multiple: true. The download step matches every blob-report-* artifact and puts all the zip files in one folder. merge-reports reads that folder.

The HTML artifact name ends with github.run_attempt, so a re-run uploads a new report instead of failing on a name that already exists.

A green run

Run 44123317088 took 1m 25s from start to end. All 5 jobs passed.

ShardTestsTest timeJob time
1 of 452.1 s35 s
2 of 4410.4 s65 s
3 of 448.6 s61 s
4 of 443.0 s43 s
merge-reports--14 s

The merged report has all 17 tests, all passed.

Playwright split the 17 tests by count and in file order: 5, 4, 4 and 4. It did not look at duration. Shard 2 got two of the slow export tests and shard 3 got the third, so their test time was 4 to 5 times that of shard 1. Most of each job's time went on setup, not tests. The Chromium install step took 17 to 33 s per shard. The test step took 3 to 11 s.

A run with one failing test

To see the failure path, I opened a pull request that breaks one assertion. The test counts open items now expects 2 open of 2 where the app shows 1 open of 2.

Run 44130118878 took 1m 18s:

JobResultTestsJob time
Shard 1 of 4passed5 passed38 s
Shard 2 of 4failed1 failed, 3 passed58 s
Shard 3 of 4passed4 passed45 s
Shard 4 of 4passed4 passed40 s
merge-reportspassed-16 s
  • Shard 2 failed and the run failed, so the pull request check is red. The green merge job does not hide the failure.
  • The other three shards ran to the end, because of fail-fast: false.
  • Shard 2 still uploaded its blob report, and the merge job still ran, because of if: ${{ !cancelled() }}.
  • The merged report shows 16 passed and 1 failed. The failed test has three attempts: the first run and two retries.
  • With trace: 'on-first-retry', the first retry recorded a trace. The trace is in the merged report. Shard 2's blob report was 58 KB, against about 6 KB for the others. The HTML report artifact was 681 KB, against 204 KB for the green run.

To open the report, download the html-report--attempt-1 artifact, unzip it and run:

npx playwright show-report html-report--attempt-1

The limits

  • One small suite of 17 tests, Chromium only, one run of each case.
  • Each shard ran its tests on 1 worker, so the shard split is the only parallelism here.
  • I did not run the workflow without fail-fast: false or without !cancelled(). The behaviour without them is GitHub's documented default.

What I'd do with it

  • Copy the four lines above together. Without fail-fast: false or !cancelled(), a red run can end with tests missing from the report, or with no report.
  • Break one test on purpose once, as here, and check that the merged report shows it. A green run does not test the failure path.
  • If one shard is much slower than the others, look at where the slow tests landed. Playwright's split counts tests, not seconds. playwright-shard-balancer, which I maintain, plans shards from recorded durations instead.

Repo: qa-gary-parker/poc-playwright-shard-merge-reports. The empty shards experiment covers what Playwright 1.65 does when a filter matches no tests across all shards. More are in the playbook.

related

QA Ark Newsletter

Test automation, AI in testing and quality engineering practice. No spam, unsubscribe any time.