Computer Use and Browser Automation Patterns
Agents need to use computers. The approaches range from pixel-level screenshots to turning websites into structured APIs. The right tool depends on the task shape. Source: compiled from wiki, 2026-05-19
The Spectrum
Browser automation for agents exists on a spectrum from low-level (pixels) to high-level (structured APIs):
| Approach | How it works | Best for | Fragility |
|---|---|---|---|
| Screenshot + vision | Take screenshot, use vision model to decide clicks | Universal fallback, visual verification | High (layout changes break it) |
| DOM/accessibility tree | Parse page structure, use refs to interact | Structured interaction with known pages | Medium (selector changes) |
| CDP snapshots | Chrome DevTools Protocol, ref-based element targeting | Agent-native browsing, React introspection | Low-medium |
| Site-specific skills | Pre-researched playbooks per website | Reliable repeated tasks on known sites | Low (maintained playbooks) |
| Websites as APIs | Convert web content to structured data (Reflex, Jina Reader) | Data extraction, content parsing | Low (content-focused) |
| Web directories | Curated catalogs of site capabilities | Agent discovery of what sites can do | Very low |
Key Players
| Tool | Approach | Role |
|---|---|---|
| agent-browser | Compact browser-command CLI/MCP, ref snapshots, sessions/profiles/auth, Chrome/Lightpanda, screenshots/diffs, React/perf commands | Default agent browser substrate |
| Playwright | Programmatic browser control, E2E testing | Committed regression tests and CI gates |
| browser-harness (browser-use) | Self-healing CDP harness, screenshot + coordinate-first, real Chrome, domain skills | Specialist fallback for screenshot-first real-Chrome flows |
| Stagehand (Browserbase) | AI browser automation, natural language | Cloud browser / Browserbase workflows |
| Browse.sh (Browserbase) | Site-specific skill catalog | Persistent site playbooks |
| Lightpanda | Fast headless browser, CDP-based | agent-browser engine option |
| Jina Reader | r.jina.ai/URL turns any page to markdown |
Free, zero-config |
| Defuddle | npx defuddle parse URL --markdown local extraction |
No API, offline capable |
| Autobrowse | Persistent browser playbooks (learn site, save SKILL.md) | Amortized discovery |
| Anthropic Computer Use | Full desktop automation via screenshots + vision | General computer tasks |
| cursor-ide-browser | Cursor's built-in vision pipeline | IDE-integrated browsing |
| Chrome DevTools MCP | Inspect user's existing browser session | Live debugging |
| Hermes creative GUI loop | Desktop navigation + visual/reference iteration inside a creative app | Learning tools such as TouchDesigner with a human reviewing output |
Architecture Decision
Kevin's browser automation ranking (from tool-hierarchy.mdc):
- agent-browser: default for browsing, scraping, form filling, screenshots, logged-in Chrome, UI QA, React/perf, and compact batch command loops.
- agent-browser MCP: same substrate when the active harness is MCP-first.
- browser-harness: specialist fallback for screenshot-first self-healing real-Chrome flows.
- Chrome DevTools MCP: computed styles, box model, DOM traversal, and exact device emulation.
- Playwright: deterministic E2E in CI and committed regression tests only.
For persistent playbooks: Autobrowse / Browse.sh (learn a site once, reuse forever).
For content extraction (not interaction): Jina Reader > Defuddle > WebFetch.
The Insight
Most browser automation fails because it treats every page visit as exploration from scratch. The compounding approach: research a site once, encode the navigation into a skill or playbook, reuse on every subsequent visit. Browse.sh and Autobrowse both implement this pattern. The agent investment shifts from per-task browsing to per-site onboarding. Source: compiled from wiki, 2026-05-19
The same compounding rule applies outside browsers. Amber Shen's Hermes/TouchDesigner demo shows a GUI creative-app version: the agent navigates the desktop, discovers how to connect to TouchDesigner, reads reference images, iterates with the human, and saves output. Treat this as "learn a tool, then crystallize the workflow" rather than as permission to let a screenshot agent freestyle production creative work without artifact review. Source: X/@whosamberella, 2026-06-13; Source: local video contact sheet, 2026-07-02
Testing: Route by Test Type
The same principle applies to browser testing. Instead of ranking tools monolithically, split by test type: agent-browser for exploratory rendered proof and logged-in interaction; Playwright for deterministic CI gates after behavior is worth locking. Findings from the agent loop feed back into Playwright as codified assertions. See Complementary Browser Testing for the full framework. private page
Agent desktops with handed-over sessions (Cua Spaces)
Cua Spaces gives agents a full desktop of their own on a Mac, a machine the user owns, or the user's cloud. Its answer to sign-in is Teleport: the user's app session (tabs, profile, sign-ins) is packed into a bundle, checked by SHA-256, and moved into the Space, so the agent skips the login step. Bundles stay in a local vault, locked by default; the user unlocks only the sites and files an agent needs, and sensitive items need Touch ID. The user and the agent share the desktop with separate cursors, so the user can step in and hand the task back. Source: X/@trycua thread, 2026-10-02; Source: spaces.cua.ai
This is the delegated-session pattern: the agent borrows a scoped, revocable session instead of a password. Version checked: Cua Spaces v0.20.0-beta.1, a pre-release published 2026-11-03 in trycua/cua. Source: gh release list -R trycua/cua, 2026-11-03
ChatGPT in the user's own Chrome
ChatGPT's Chrome extension (2026-10-15) reads open tabs in Chat mode and controls them in Work mode: sending messages, filling forms, downloading files and grouping tabs, in the user's signed-in browser. The suggested tasks are comparisons across open tabs and form filling from a brief. This is the opposite end from Cua Spaces above: no separate desktop, full access to the user's live sessions. Source: X/@ChatGPT thread, 2026-10-15
Hark Pro
Kevin's review of Hark Pro (2026-10-06): computer use runs fast in an isolated cloud sandbox the user can watch, with vault-based sign-in reusable across apps (useful for Google), in a cleaner interface than the usual three-column chat. Kevin's test tasks were logging into DoorDash to suggest new restaurants from past orders and comparing travel options side by side. Source: X/@kevskgs thread, 2026-10-06
Hark's launch promotion asked users to tell the agent to order a free pizza, so the ad was the product doing a task. Kevin called it outcome-based advertising. Source: X/@kevskgs, 2026-10-17
Grok Bot
Grok Bot (beta, 2026-08-11) runs AI teammates that sign in to your tools, use them as you do, and return finished work; you can close your computer and reach them from anywhere. Early uses: negotiating with vendors in the user's voice, store support and keeping a CRM current. Source: X/@bot thread, 2026-08-11; Source: X/@bot, 2026-08-11; Source: X/@bot, 2026-08-11
Phone harness
phone-harness lets an agent drive a real iPhone with no jailbreak, Xcode or WebDriverAgent: the Mac's iPhone Mirroring window is the whole transport, read by screen capture and Vision and driven by clicks. Setup is one prompt. Source: X/@shawn_pana, 2026-08-08; Source: ShawnPana/phone-harness
neko streams a full browser from a Docker container over WebRTC so several people can watch and control one session, with audio; a self-hosted shared browser for demos and remote testing. Source: X/@oliviscusAI, 2026-07-18
Timeline
- 2026-07-02 | Added Hermes + TouchDesigner as a creative-GUI computer-use pattern: desktop navigation, tool-connection discovery, reference-image reading, human iteration, and saved output. Source: X/@whosamberella, 2026-06-13
- 2026-05-19 | Page created. Compiled from wiki tool pages, Browse.sh launch, and Kevin's browser automation stack. private page
- 2026-05-19 | Added testing section. Kevin articulated the complementary browser testing pattern: Playwright for determinism, Chrome MCP + agent loop for visual judgment. Cross-linked to Complementary Browser Testing. private page