← blog
· series: Measured

Playwright WebMCP: Calls Pass 50 of 50, First Read Misses Late Tools

Playwright 1.65.1 called a page's WebMCP tools in 50 of 50 runs, median 9.64 ms. Tools registered 50 ms late were missing from the first read in 50 of 50 runs, with no error.

Playwright 1.65.1 can list and call the tools a page registers through WebMCP, and the calls were reliable whenever the tools existed at the moment of the call: the same task passed in 50 of 50 runs, in a median of 9.64 ms. The failures were around the calls. page.webmcp.tools() did not wait for tools the page registered late, nothing checked the input against the schema, and a tool that threw reached the test as "Tool execution failed" with no reason.

The page, the tests and every raw result are in the PoC repo.

The setup

WebMCP lets a page register tools that an agent can call instead of clicking through the UI. Playwright exposes them as page.webmcp. I wanted to know if a test can use them reliably, what breaks, and how that compares with driving the same page through selectors.

  • Playwright 1.65.1 and Chromium 159.0.8154.6 (the Playwright build, headless), launched with --enable-features=WebMCP. Node 24.14.1 on macOS arm64.
  • A toy shop page with a normal UI (search, add to cart, total, checkout) and seven WebMCP tools backed by the same cart logic: search_products, add_to_cart, get_cart, checkout, plus three awkward ones for the failure cases (echo_qty, boom, hang).
  • No model is involved. Every figure is a scripted run, 50 runs per row, each in a fresh browser context. The median is the 26th of the 50 sorted values.

The page registers each tool with a name, an input schema and an execute function. This is add_to_cart. The text helper wraps a value in a content array.

const mc = document.modelContext || navigator.modelContext;
mc.registerTool({
  name: 'add_to_cart',
  description: 'Add a quantity of a product to the cart.',
  inputSchema: {
    type: 'object',
    properties: { sku: { type: 'string' }, qty: { type: 'integer', minimum: 1 } },
    required: ['sku', 'qty'],
  },
  execute: async ({ sku, qty }) => {
    if (!Number.isInteger(qty) || qty < 1)
      return { isError: true, content: [{ type: 'text', text: 'qty must be a positive integer' }] };
    try { return text(addToCart(sku, qty)); }
    catch (e) { return { isError: true, content: [{ type: 'text', text: e.message }] }; }
  },
});

The test side is plain Playwright. This is a condensed version of the task script. It adds the mugs, and the full script adds the lamp in the same way.

const tools = await page.webmcp.tools();
const found = JSON.parse(
  (await page.webmcp.callTool('search_products', { query: 'enamel' })).content[0].text
);
await page.webmcp.callTool('add_to_cart', { sku: found[0].sku, qty: 2 });
const cart = JSON.parse(
  (await page.webmcp.callTool('get_cart', {})).content[0].text
);

Same task, tools against test ids

The task: add 2 mugs and 1 lamp, expect a total of 54. The DOM script uses getByTestId, written once against UI variant 1. Variant 2 renames every test id and label.

UIMethodPassedStepsMedian task timep95
v1WebMCP50/5069.64 ms10.63 ms
v1DOM test ids50/508134.77 ms150.8 ms
v2, renamedWebMCP50/5069.12 ms9.93 ms
v2, renamedDOM test ids0/50n/a1001.05 ms1002.51 ms

Read this table carefully, because it flatters WebMCP.

  • The v2 failure is by construction. I renamed the ids the script depends on, so every run stopped at the 1000 ms action timeout with locator.fill: Timeout 1000ms exceeded. It shows that test ids are fragile. It does not show that DOM driving is worse in general: role and text locators would survive some renames.
  • Steps are the calls the script made. The DOM figure is low because the locators were known in advance. An agent without them would also need snapshots to find them.
  • I did not measure where the DOM time goes. The likely cause is that a DOM action does more than a tool call: Playwright waits until the element is ready and then sends real input. With a model in the loop, the model's time per step is added to both methods. I did not measure that.

The part that does carry over: the tools are not tied to the markup. Variant 2 changed only the UI, so the same WebMCP script passed on both variants. That is the case for testing the tools directly.

Late registration: tools() does not wait

The page registered the tools 0, 50, 200 and 1000 ms after its script ran. Straight after goto I read page.webmcp.tools() once, then read it again every 10 ms until the tools were there. The time in the last column runs from the first read.

Registration delayFound on immediate readFound when pollingMedian time until found
0 ms50/5050/503.15 ms
50 ms0/5050/5059.12 ms
200 ms0/5050/50209.21 ms
1000 ms0/5050/501008.5 ms

With any delay, the immediate read returned an empty list and no error, in all 150 runs. A test that reads the tool list straight after navigation sees no tools, and nothing says they are on the way. Poll until the tools you need are there:

await expect
  .poll(async () => (await page.webmcp.tools()).length, { timeout: 5000 })
  .toBe(7);

What callTool does not do for you

Each benchmark case below ran 50 times, with the same outcome every time.

  • No schema check. echo_qty declares qty as a required integer with a minimum of 1 and no extra properties. Through callTool, a string "3", the value 0, a missing qty and an extra property all reached the page's execute function unchanged and returned normally. In this Playwright and Chromium build, nothing between the test and the page checked the input, so the page has to.
  • Page errors are results. add_to_cart checks its own input. For an unknown SKU it returned isError: true with the text "Unknown sku NOPE", and callTool handed that back as a normal result. It did not throw. A test has to check result.isError.
  • Thrown errors lose the reason. A tool that throws new Error('boom') reached the test as OperationError: Tool execution failed. The word "boom" is not in it.
  • Unknown tool throws, and the message lists the available tools. A browser without the flag throws "WebMCP is not enabled". A page that registers nothing returns an empty list.
  • A tool that never resolves. With { timeout: 500 }, callTool threw Timeout 500ms exceeded.

Without a timeout, the result depends on where the code runs. I ran each of these checks once, outside the 50-run benchmark. In a plain library script, the call threw Timeout 30000ms exceeded. Under the Playwright test runner, with the test timeout raised to 50 seconds, the call was still pending after 42 seconds. With the default settings, the call failed after 30 seconds with Test timeout of 30000ms exceeded. That is the test timeout, not a timeout of the call.

Pass a timeout, and check the result:

const result = await page.webmcp.callTool(
  'add_to_cart', { sku: 'NOPE', qty: 1 }, { timeout: 500 }
);
expect(result.isError).toBe(true);

What didn't work

There are two naming traps, and neither gives an error.

  • In this Chromium build the page-side object is document.modelContext. navigator.modelContext is undefined. A page that only looks at navigator.modelContext registers nothing, and nothing says why.
  • The page-side hint for a consequential tool is consequentialHint, which Playwright reports as annotations.consequential. A tool that I registered with consequential came back from tools() with no annotations.

The limits

  • No model is involved. This says nothing about whether a model picks WebMCP tools better than DOM clicks, and I did not measure tokens.
  • I called the tools through Playwright only. I did not test what a browser's own agent does with the schema before it calls a tool.
  • One toy page on localhost, one machine, one browser build, headless. The timings are from a warm local run on macOS. Read them as orders of magnitude.
  • Not tested: Playwright MCP's own WebMCP support, Firefox, other Chromium builds, headed mode.

What I'd do with it

  • Test your page's tools with page.webmcp.callTool. Wait for registration before you read the tool list.
  • Pass a timeout on every callTool, and check isError on every result.
  • Validate input inside execute. Do not rely on the declared schema.
  • When a tool can fail, return isError with a message. Do not throw: the message of a thrown error did not reach the test.

Repo: qa-gary-parker/poc-playwright-webmcp. The MCP vs CLI experiment measures the token cost of driving a page through Playwright MCP and the CLI. More are in the playbook.

related

QA Ark Newsletter

Test automation, AI in testing and quality engineering practice. No spam, unsubscribe any time.