Computer use vs browser use vs Playwright for UI testing

How many ways can an agent see a UI?

When a coding agent changes a UI, the job is not done yet: it has to see that the change actually works. We described this loop, where the agent checks its own work, in our harness engineering post. For a UI there are three ways to make that check, and all three come ready to use in coding tools today:

With computer use, the agent sees the screen as an image and works it with the mouse and keyboard. Anthropic’s tool provides screenshots plus mouse and keyboard control. Google’s documentation also says the model “sees” the screen through images and produces actions such as clicks and keystrokes. This path reaches whatever is on the screen: Google’s tool works in the browser, on mobile and on the desktop.

With browser use, the agent looks at the page’s structure rather than its image. Anthropic’s browser use tool reads the page from the accessibility tree: each element comes with its role and name and a reference such as [ref_2], and when the agent clicks, it passes that reference instead of coordinates. Where the tree sees nothing, in a canvas for example, it falls back to screenshots and coordinates. Playwright MCP also reads the page from this tree instead of from screenshots. The open source Browser Use library that shows up in search results is a separate project; in this post, “browser use” means this method.

On the code and tests path, the agent drives the browser through a library: it writes a Playwright test or calls Playwright’s CLI. Playwright CLI does not load the page’s data into the model; the agent reads only the command’s output. OpenAI’s computer use guide also recommends the code path for GPT-6 Astra: the model writes Playwright or PyAutoGUI code that drives the UI, and the structured computer tool remains an alternative. What really sets code apart is that once it is written, it can run again without a model.

What does each one cost, and how reliable is it?

We compared the three paths on three questions: how many tokens a step takes, how long a task takes, and how often a task is finished end to end. Most of the figures come from the providers’ documentation; for benchmark results, we say who reported them.

An agent that works from the screen reads a screenshot at every step, and images are billed as tokens. The three providers count them differently: Anthropic splits the image into 28-pixel squares and OpenAI into 32-pixel squares, while Google uses a fixed number for Gemini 4 and 3, whatever the size. At the 1280×800 resolution Anthropic recommends for web apps, a single screenshot comes to:

What the chart does not show is how this adds up. The agent takes a new screenshot at every step, and the old ones stay in context. According to Anthropic, a 200K-token context window fills up long before it reaches 100 screenshots. Once a request holds more than 20 images, each image falls under a tighter size limit. That is why Anthropic recommends, for its newer models, scaling screenshots down to at most 2000 pixels per side and clearing old ones on the server side. What the agent forgets when its context fills up is in our auto compact post.

The toolset has an overhead of its own too. At Anthropic, the computer use toolset adds about 4,500 input tokens to every request, and the browser use toolset about 6,600. Because this overhead is the same on every request, prompt caching pays off here. There is no single number for the accessibility tree; it depends on the page. Anthropic’s documentation says that on an ordinary page, reading the tree usually takes fewer input tokens than a screenshot. The tree can grow large too: Playwright’s team recommends the CLI over MCP for coding agents, because the CLI does not load large tool schemas and long accessibility trees into the model’s context.

For time, the providers give no per-step figure; none of the documentation we read states seconds per action. Anthropic writes that computer use can be slow compared with a person doing the same actions by hand, and recommends it for tasks where speed does not matter, such as automated software testing. OpenAI reports time per task: by its own measurement, GPT-6 Astra finished an OSWorld 2.0 task in about 40 minutes, and GPT-5.6 Sol in about 75 minutes. These tasks are long: according to the OSWorld 2.0 site, a person takes about 1.6 hours to finish one (median), and Claude Opus 4.7 makes 318 tool calls per task on average; on the old OSWorld, that number was about 30.

For reliability, the benchmark people look at most is OSWorld. All three providers measured their new models on a newer version of OSWorld and announced the results, but each one measured on a different version or subset, and all of them report partial scores:

Source: the vendors’ own announcements and model cards, 20 October 2026; every result is as reported by the vendor. The versions and subsets differ, so the rows cannot be compared with each other. Google’s model cards give a month, not a day.

The partial score measures how many of a task’s checkpoints pass; OSWorld 2.0 scores a task with 27.25 checkpoints on average. The benchmark’s own main measure, though, is whether the task is finished end to end. The gap between the two is large. In OSWorld 2.0’s own table (15 October update, release v2.1), the best result among runs where the model works on its own is Claude Opus 5’s run at the highest effort: a partial score of 77.67%, and 44.33% of tasks finished end to end. At the lowest effort, the same model finishes 18.81%.

The table has no OpenAI model on the current release; on the previous release’s offline subset, GPT-5.6 Sol at the highest effort scored 64.13% on partial score and 28.1% on finishing end to end. Gemini models are not in the table at all. So the numbers vendors announce say that the agent does most of the work; they do not say that it finishes the work. How to read a score is in our SWE-bench post.

The providers themselves write why working from the screen does not give the same result every time. According to GitHub, a change in timing or in the state of a window can lead to a different result, and can make the agent repeat an action or get stuck. Anthropic writes that the model can make mistakes when it produces coordinates, and Google that unexpected pop-ups and layout changes can confuse the model. The accessibility tree solves part of this problem: according to Anthropic, an element reference is not affected by the layout shifts that break pixel coordinates. In a Playwright test, an element is found by its role and name, Playwright waits until the expected state is reached, and the test takes the same steps in the same order on every run.

Source: the documentation linked in this section, 20 October 2026.

What do the three providers offer for computer use and browser use?

If you build your own agent on the API, all three providers have a ready-made tool. The loop is the same in all of them: the model asks for an action, and your code runs it in your own browser or virtual machine and sends back the result. As of 20 October 2026:

Source: the three providers’ computer use, browser use and pricing documentation, 20 October 2026. “Not in its documentation”: in the pages we read that day, we found no sentence that describes it.

Anthropic made a switch recently. According to its release notes, computer use left beta on 19 August and the browser use tool arrived the same day. On Opus 5.5, released on 22 September, the old computer_20251124 tool returns a 400 error on the Claude API and Google Cloud; Sonnet 5.5 and Haiku 5.5 do not accept the old tools either. The new toolset can return several actions together in one turn; your code should run them in order and stop at the first error.

OpenAI has two paths. The guide covers both the computer tool, where the model returns actions such as clicks and typing, and the path where the model writes Playwright or PyAutoGUI code. On the code path, the model can combine several actions, a loop or a condition in a single call. The guide also asks you to check the real outcome at the end instead of trusting the model’s answer. Since 29 September, the Agents API has also offered computer use in a browser hosted by OpenAI.

Google’s tool is in preview, and Google recommends close supervision for important tasks. The model gives coordinates on a 1000×1000 grid, and your code converts them to the size of the screen. If an action is risky, the response carries require_confirmation and you have to ask the user. Prompt injection scanning of screenshots is optional and off by default.

When it comes to choosing a model, the three vendors recommend different things. Google recommends Gemini 3.8 Flash for computer use. Anthropic finds Haiku 5.5 a good fit for browser use tasks where speed matters. Codex’s documentation recommends GPT-6 Astra for hard tasks that depend on screenshots or visual judgement. The general differences between models are in our model selection post.

How do coding agents test a UI?

All five coding tools give the agent a way to use the browser or the screen, but they differ in where it runs and in their defaults. We read the table from each tool’s own documentation on 20 October 2026:

Source: each tool’s documentation and release notes, 20 October 2026. “Not in its documentation”: in the pages we read that day, we found no sentence that describes it. In Copilot, computer use entered preview on 1 October and is off by default.

If you still use Gemini CLI, its browser agent is off by default; when you turn it on, it works with the page through the accessibility tree. For free tier and Google One users, Google replaced Gemini CLI with Antigravity CLI on 18 June 2026; with paid API keys, Gemini CLI can still be used.

The documentation of three of these tools recommends the same order. Claude Code treats screen control as the broadest and slowest path: it tries an MCP server first, then Bash, then Claude in Chrome, and uses the screen only if none of them is enough. According to GitHub, if an API, an MCP server, a terminal command or a browser tool can do the job, those tools give more structured information and more predictable results. Codex’s documentation also recommends the built-in browser first for a local web app, and an MCP server for repeated tasks.

Another difference is the session. Claude in Chrome shares your browser’s login state and can get into every site you are signed in to; it stops and asks you at a login page or a CAPTCHA. In VS Code, the pages the agent opens open in a separate, temporary session. Codex’s built-in browser and Antigravity IDE use a profile separate from yours.

Why is it usually better to have the agent write a Playwright test?

The check an agent makes in the browser is gone when the session ends. Turn the same check into a Playwright test, and it stays in the repository and runs again on every pull request without a model: it spends no tokens, piles up no screenshots and takes the same steps every time. VS Code’s documentation recommends the same thing: browser tools are for checks within a session, while checks that have to run in CI every time, or catch an old bug coming back, belong in the repository as Playwright or other automated tests.

In practice, a two-step routine works well. First, the agent explores the page with a browser tool or Playwright MCP and sees what the selectors and texts really are. Then it writes that flow into a test, runs the test and shows that it passes. Playwright also provides ready-made agents for this: Playwright Test Agents are a planner that explores the app and drafts a test plan, a generator that writes tests from the plan, and a healer that repairs broken tests. npx playwright init-agents --loop=claude adds these definitions for Claude Code; there are codex and vscode options too. The generator checks selectors and assertions live while it runs the scenario. Here is a test an agent might write for a sign-up form:

// Example: a test the agent wrote; CI runs it again on every pull request.
import { test, expect } from '@playwright/test';

test('sign-up form rejects an invalid email', async ({ page }) => {
  await page.goto('/signup');
  await page.getByLabel('Email').fill('not-an-email');
  await page.getByRole('button', { name: 'Sign up' }).click();
  await expect(page.getByText('Enter a valid email address')).toBeVisible();
  await expect(page).toHaveScreenshot('signup-error.png');
});

The test makes two kinds of check. toBeVisible looks at behaviour: does the error message appear? toHaveScreenshot looks at appearance: the page is compared pixel by pixel with the reference image in the repository. According to Playwright, the operating system, the browser version and even the hardware can change the image, so generate the reference image in the environment where the test will run. If you want to compare the page’s structure through the accessibility tree instead of an image, there is toMatchAriaSnapshot.

When a test breaks, leave the agent evidence so that it can see why. The config below saves a screenshot for every failed test, and a trace on the first retry in CI; in the trace you can step back and forth through the test and see what happened at each step. The files land in the test output folder, usually test-results; tell the agent to look there first when it fixes the failure.

// Example: a config that keeps evidence from a failing test.
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: {
    baseURL: 'http://localhost:3000',
    screenshot: 'only-on-failure',
    trace: 'on-first-retry',
  },
});

Checks based on screenshots are still needed for these jobs: layout shifts, colour or alignment bugs, a chart or a map drawn on a canvas. Anthropic’s documentation also lists canvas, embedded video and remote desktop surfaces among the places the tree cannot see. To find a bug like that, it is enough for the agent to look at a screenshot once; to keep it from coming back, have the agent write a test with toHaveScreenshot.

Which path should you choose for which job?

All the evidence points to the same order: if you can do the job in code, use code; if the page is in a browser, use the tree; if there is no other way in, use the screen. For common jobs:

The reasons behind the cards are above. A test for checking a web UI, because the test runs in CI without a model and takes the same steps every time. A browser tool for a one-off check, because reading the tree is usually cheaper than a screenshot and a reference is not affected by layout shifts. Keep computer use for apps that have no API, CLI or MCP server; GitHub also presents it for apps like these.

For tasks such as payments, passwords and account changes, all three providers say the same thing: let a human approve the last step. Anthropic recommends human approval for decisions such as accepting cookies, making financial transactions or agreeing to terms of service. OpenAI counts even typing sensitive information into a form as sending data. Google’s sample instructions tell the model never to try to solve a CAPTCHA. Run the agent in a separate virtual machine or with a separate browser profile, and limit the sites it can visit with an allowlist. How text on a page can steer the agent is in our MCP security post.

diffox, an AI code review tool, reads the pull request with git; it does not open the UI or take screenshots. So a test the agent writes for the UI does two jobs in the pull request: CI shows that the UI works, and in the review, what the test checks can be read in the diff. Why a passing test does not always check anything is in our post on reviewing AI-generated code.