Computer-use reference

A computer-use study puts a participant (actor.type, for example computer-use) in a browser on a hosted E2B desktop and writes an evidence bundle under gitignored .humanish/: full-fidelity screenshots by default, typed text recorded as lengths only, and provider-neutral humanish.actor-trace.v1 traces. Start with the complete own-app example; the blocks below are reference fragments.

Choose a subjectCopy Anchor Link

Hosted browser studies support these sources:

• subject.source: clone (+ execution.target: e2b-desktop + a computer-use actor): the study clones your repo into the sandbox, runs your declared serve.install/serve.build/serve.start commands (detached, with readiness probing), and drives the served app at serve.url. Subject env var names declared in subject.env are provisioned from --dotenv (provenance records names; persisted text uses known-value and pattern redaction). The bundle records provenance: repo, cloned commit, env names.
• subject.source: app-url: drive your reachable deployment with policies.allowPublicTargets: true. The computer-use route accepts one participant for this shared URL. Library callers can also provision the target through RunStudyOptions.prepareDesktop.
• subject.source: local-tree: package the study resolution directory, upload its content-pinned archive, and serve it like a clone. Declare subject.serve and use subject.localTree.keep, exclude, and maxArchiveBytes to control the archive; do not declare repos or clone. Inspect what will be included and keep secrets out of the selected tree.

subject:
  source: clone
  repos: [example-org/example-app]
  serve:
    install: pnpm install --frozen-lockfile
    build: pnpm build
    start: pnpm start
    url: http://127.0.0.1:3000/
actor:
  type: openai-computer-use
  mission: Explore the app as a first-time visitor and complete its primary flow.
execution: { target: e2b-desktop }
route: computer-use
mode: live

npx humanish run cua-browser --dry-run --json --no-open

Live runs (mode: live) need OPENAI_API_KEY + E2B_API_KEY (from humanish keys set, the environment, or --dotenv) and the optional peer dependency: npm i -D @e2b/desktop. A cloned subject is served inside the sandbox on loopback; to instead drive a deployment you own (a Vercel preview, staging), use an app-url subject with policies.allowPublicTargets: true. The actor's API key never enters the sandbox; only declared subject env names do. humanish init scaffolds an example at humanish/studies/cua-browser.yaml.

Custom desktop templatesCopy Anchor Link

Set execution.desktop.template to your E2B desktop image name or ID when the subject needs additional runtimes. Keep the Desktop SDK's xdotool input support and the C.UTF-8 locale: participants type every text through LC_ALL=C.UTF-8 xdotool type. Without that locale, typing non-ASCII characters (accents, dashes, emoji, CJK) ends with text-command. A dry-run validates the study configuration; it does not inspect the tools installed in your image.

Off-app email/SMS verification (comms)Copy Anchor Link

When a flow is gated behind an email or SMS the app itself sends (a signup verification link, a one-time code, a magic link), add a comms: block to the study. The run then redirects the app's email-API sends (via one adopter-named env var: with the official Resend SDK it is RESEND_BASE_URL, which the SDK reads on its own, so nothing in the app has to change; other providers need the app to actually pass the env through, and a run whose catch captures zero sends warns at teardown) into a catch inside the sandbox, so nothing leaves the machine. Every participant, independent or shared-world, gets a deterministic inbox address automatically (an unnamed shared-world participant is role-01, role-02 and so on), and each persona's prompt carries the full handoff: the address to sign up with, the inbox URL to open, and that waiting for an email is a next step, not a blocker. The run bundle gets a digest-only humanish.comms-thread.v1 artifact (from/to/subject/link digests + an OTP count; no raw address, link, or code persists); the readable proof a persona saw the email is its inbox-page screenshots. Hosted on the clone/local-tree computer-use route and the shared-world route (refused elsewhere), vendor-neutral (Resend/SendGrid shaped, or a custom profile). An app-url study can instead use an adopter-hosted catch via comms.email.external; it must already be wired into your app. See the study file reference for every comms field.

Screenshots are full-fidelity by defaultCopy Anchor Link

Run bundles live in gitignored .humanish/, so the Observer shows exactly what the persona saw. Set policies.redactScreenshots: true to persist blurred thumbnails at capture instead (for unowned subjects, or bundles you intend to share as-is). Raw bundles stay local in gitignored .humanish/; nothing scans the pixels, so review them before sharing anywhere. humanish export --run <id> --format bundle --redact-screenshots writes a blurred copy as a new bundle workspace and leaves the original unchanged. The frame sent to the model is always full-resolution regardless. See the capture and sharing policy. humanish verify reports raw-screenshot bundles as shareSafety.status: local_only; humanish feedback issue refuses them until the run is share-ready. A frame counts as blurred only when the trace marks it blurred and it is a thumbnail the redactor wrote (at most 128 px wide, image data only); any other frame counts as raw.

Device presetsCopy Anchor Link

execution.desktop.device picks the desktop screen size, on hosted desktops and in a local browser study: mobile (414×896), small-mobile (360×740), narrow-mobile (320×700), tablet (820×1180), desktop (1440×950, default), or wide (1920×1080). Fidelity: on the computer-use / E2B-desktop route width/height size the virtual display and browser outer window. The actual page viewport is smaller because browser chrome occupies space; Chromium-family live bundles measure it through CDP and record it separately from requested/verified screen geometry. Browsers without that measurement seam omit the viewport rather than guessing. A site's width-based responsive CSS still fires, and the model is told its device in the prompt. Without the block below there is no touch input, the device-pixel-ratio isn't rendered, and the user-agent stays desktop on this route. Set device on a participants entry to override the study preset for that participant. execution.desktop.resolution is a raw escape hatch that overrides the preset.

Mobile emulationCopy Anchor Link

execution.desktop.fidelity: { mobileEmulation: true } gives every hosted or local Chrome/Chromium computer-use participant on a mobile preset (mobile, small-mobile, narrow-mobile) a mobile-emulated browser before its first action, and leaves desktop, tablet and wide participants in the same run untouched: the participant's preset width/height become the CSS viewport (414 px for mobile, where the X screen itself cannot go below 500), the preset's device pixel ratio applies (deviceScaleFactor overrides it), touch events are on (touch: false turns them off) and the browser presents a mobile user agent (userAgent replaces the default iPhone Safari string). The run bundle records desktopGeometry.fidelity with tier: mobile-emulated, the request, the CDP methods applied, and resolved: what the page itself reported afterwards (navigator.userAgent, devicePixelRatio, innerWidth, maxTouchPoints, coarse pointer). A page without a viewport meta lays out at 980 px, as it would on a phone, and the bundle says so. Firefox cannot be emulated, so the participant fails closed instead of shipping a desktop run labelled mobile. humanish waits up to 30 seconds after launch for Chrome's DevTools port and records the wait as a timed phase; if Chrome exits or never answers, a mobile-emulated participant fails with the reason and the last line of the browser log. The emulation holder also applies the settings to new page targets. Later observations record the page's fidelity and warn if it drifts from the requested settings. A bundle without a fidelity block is a responsive-viewport study whatever its preset is called.

When touch is enabled, desktop pointer input is converted to touch. Correct viewport and touch flags do not prove equivalent gesture behavior: in two hosted conformance probes, the SDK double click produced two single clicks while direct touch opened the original TodoMVC editor in both. From humanish 0.81.0, these participants carry a run warning to confirm gesture failures with direct or native touch before attributing them to the app. The earlier phone-preset outcomes remain observations of the measured humanish input path. See the dated input-conformance receipt for the method and limits; physical-device behavior was not tested.

Desktop browser choiceCopy Anchor Link

Hosted computer-use and shared-world participants use the desktop's default URL opener unless you set execution.desktop.browser to chrome, chromium, or firefox. A concrete value means "launch this browser or fail"; it never silently falls back to whatever the image prefers. When configured, run bundles record the requested browser and the resolved in-sandbox command as desktopBrowser.

Deterministic stop conditionsCopy Anchor Link

Freeform computer-use actors can keep acting after the app has already reached the state you care about. Add stopWhen to the actor or to one participants entry to stop immediately after a deterministic browser observation matches. Conditions inside one rule are ANDed together; rules under any are ORed. A participants entry's stopWhen overrides the actor default.

actor:
  type: openai-computer-use
  mission: Complete the assigned browser task.
  stopWhen:
    any:
      - id: dashboard-visible
        urlPathEquals: /dashboard
        textIncludes: Dashboard
participants:
  - id: reviewer
    instruction: Review the item and return to the queue.
    stopWhen:
      any:
        - id: returned-to-queue
          urlPathEquals: /items
          textIncludes: Queue

A matched rule ends the session as goal_satisfied: the rule is your definition of done. The participant is never told a rule matched, so its closing report describes the session from its own mission. When the rule is narrower than the mission, for example "the first table exists" for a mission to design a whole schema, the participant can open its report with DID NOT REACH THE GOAL. while the run records the goal as met. Both are accurate: the grade says your condition was reached, and the report says where the person thought they were when the session ended.

A participant with a cameraCopy Anchor Link

execution.desktop.media.camera: { source: synthetic } gives a hosted Chrome participant a capture device: an ffmpeg test pattern generated in the sandbox (or a .y4m file of yours, uploaded). The browser's own permission dialog stays in the way by default (policies.mediaPermission: prompt), because the gate is where a real person hesitates or refuses; granted bypasses it for studies about what happens after. The bundle records the feed and the exact launch flags under desktopBrowser.media. Microphone source-file injection is unsupported: a media.microphone.source other than speech is rejected before any spend, including with a custom desktop template. source: speech requires a local-agent Codex participant and cannot be combined with a hosted camera. This does not determine whether a custom template has its own working audio devices; humanish does not deliver the declared audio file to them.

Camera-free studies behave as before. A working camera does not establish audio, conferencing-provider, TURN or reconnection support.

A declared observation windowCopy Anchor Link

Some findings are "time passed and nothing broke": a call both participants stay on, an import that finishes, a dashboard that updates. A freeform participant with nothing to do keeps acting, so dwell lets the study hold instead. Once when matches (or after the first observation, when there is no when), the harness holds the page for ms, captures a frame every everyMs (default 10 s), takes no action and requests no model turn, then hands control back (then: continue, the default) or ends the session (then: stop). The window is recorded in the trace as deliberate, and it never outlasts the session budget. A participants entry's dwell overrides the actor default.

actor:
  type: openai-computer-use
  mission: Join the room, stay a while, then leave.
  dwell:
    when:
      any:
        - id: in-room
          urlIncludes: /room/
    ms: 120000
    everyMs: 10000
    then: continue

Supported primitives are urlIncludes, urlPathEquals, textIncludes, and appStatePathEquals. Page text, full URLs and app state are runtime-only and are not persisted into the run bundle; for a matched rule the trace stores only its id and primitive names. Each observation's page origin and redacted path are recorded (see Pages a participant visited). Browser URL and text observation requires a Chrome/Chromium CDP session in the desktop. For deterministic browser-observed stops, set execution.desktop.browser: chrome or chromium.

These are distinct binary requests, not interchangeable aliases. Use chrome with the stock desktop used by the camera example; choose chromium only when your template supplies that binary. The default tries available browsers, with Chrome first. doctor checks local configuration and credentials; the remote binary is checked after desktop provisioning. An unavailable explicit browser ends setup before participant dispatch.

Pages a participant visitedCopy Anchor Link

Each screenshot item in a participant's trace carries the page that observation was on, as page: { origin, path }, and the trace's finalPage is the page of the session's last observation. turn-00-start is the page before any action and turn-01 the page after turn 1's actions, so the screenshot items with a page, in order, then finalPage, are the participant's path through the app:

jq -r '.items[] | select(.kind == "screenshot" and .page) | "\(.title) \(.page.origin)\(.page.path)"' \
  .humanish/runs/<run-id>/actor.json
jq '.streams[].actor.finalPage' .humanish/runs/<run-id>/run.json

The query string, fragment and user info are never kept. A path segment that looks like an email address, a token or id (16 or more letters, digits, - and _ with a piece mixing letters and digits), a key or one of the study's secrets becomes [REDACTED_URL_PART], and so does a token-shaped host label. A study secret that spans origin and path, such as an invite link, withholds both. A short code, an all-digit id or a name in a readable path stays, so review the bundle before sharing it, as for narration. Pages need the same Chrome or Chromium browser-state probe as URL stop conditions: a Firefox participant, or one whose probe is unavailable, has screenshots without pages. Shared-world studies and the local browser route record no pages.

Email, phone and message linksCopy Anchor Link

A participant's desktop (E2B, Modal or the local browser guest) has no email, phone or messaging app. When a participant clicks a mailto:, tel:, sms: or other link that Chrome hands to another app, the desktop writes the link down and opens no app chooser. The trace records a link opened another app notice with the link's scheme and, for an email link, the recipient addresses from its address, to, cc and bcc. The subject, body and phone number are not kept. Addresses go through the trace's redaction as narration does, so an address that is not one of the study's secrets stays; review the bundle before sharing it. The participant's next turn says the link worked, that the app is not shown and that nothing was sent.

jq -r '.items[] | select(.title == "link opened another app") | .text' \
  .humanish/runs/<run-id>/actor.json

Chrome asks before it hands a tel: or sms: link to another app. The participant's browser starts with that permission for the study's own origin, so the same link on another site still shows Chrome's prompt. Firefox participants are not covered.

Requests a participant's browser madeCopy Anchor Link

With execution.network: true, each hosted computer-use participant in Chrome or Chromium records the requests its browser makes to network/<participant id>.ndjson in the run folder: one JSON line per request hop, each with the turn it happened in.

execution:
  target: e2b-desktop
  network:
    origins: [https://api.shop.example] # optional, at most 16

The participant's target origin is always kept, and origins adds the API or asset hosts to keep beside it. Requests to any other origin are counted in the stream's network.omitted.otherOrigin and not written. A line holds the start time, the turn, the method, the origin and path, the resource type, the status and the duration, and says whether the request redirected, came from a cache, failed or was still open. It holds no bodies, cookies, headers, query strings or fragments. Paths are redacted as pages are (above), so a short code, an all-digit id or an all-letter token in a path stays.

jq -c 'select(.failed or .pending or .status >= 400)' \
  .humanish/runs/<run-id>/network/<participant id>.ndjson

Capture starts when the browser's DevTools answer, a few seconds after launch, so a desktop participant's log misses most of its first page load; streams[].network.startedAt says when it began. Requests from workers, service workers and cross-site iframes are not recorded. A browser other than Chrome or Chromium records one warning and no file. The participant inspector in the Observer shows the counts and links the file, and humanish observe serves it with the run. The terminal, scripted, shared-world and local browser routes, a desktop-cli subject and execution.desktop.browser: firefox refuse the field. The run bundle contract lists each key.

Closing reports after an automatic stopCopy Anchor Link

From humanish 0.81.0. After a stopWhen match or dwell with then: stop, a provider that supports closing reports can make one read-only request using the final observation. The OpenAI provider and a local Codex participant support this; Claude Code does not. There is no extra CLI flag. Desktop interaction has already stopped, and any actions requested by the closing response are rejected rather than executed.

The request needs at least one prior participant turn, the required observation, and remaining session/model budget. It is limited to 30 seconds or the smaller remaining deadline, without retry. Reported usage counts toward the existing model budgets; an in-flight request can cross a threshold.

The trace's debrief records completed, skipped, or failed, a reason, and usage availability. An accepted typed report contains summary and frictionReports. An empty friction list means the participant reported none; a missing or failed report does not. The same request asks for up to six impressions: what looked unclear, unfinished or untrustworthy, what the participant liked or missed, and where the screen differs from how the persona does the same task in its own work or life. The trace's impressions holds them, or says why none were collected. The report preserves the original stop reason and task outcomes. Budget and error exits do not request this extra report. When an OpenAI participant ends the session itself, the loop makes one request that asks only for its impressions; the participant's own summary, reason and outcome stay as it gave them, and the request's usage counts toward the study's cost. See reading outcomes.

Pick the participant modelCopy Anchor Link

actor.type: computer-use runs the model actor.model names, on that model's vendor, with the vendor's API key:

actor:
  type: computer-use
  model: claude-haiku-5-5 # runs on Anthropic with ANTHROPIC_API_KEY
  reasoningEffort: medium
  mission: Add this week's errands and decide whether you would keep using the app.

• A level the model does not take is refused when the study is read, for actor.reasoningEffort and for each participants[].reasoningEffort.
• With no actor.model, the participant runs gpt-6.1-sol on OpenAI, as openai-computer-use does.
• A model humanish does not list yet runs when actor.provider names its vendor, openai or anthropic. humanish has no price for it, so a caps.maxUsd or caps.maxTotalUsd is refused, and its effort is checked against the levels the vendor documents. Without actor.provider the study is refused, since nothing says which vendor serves the model.
• openai-computer-use still sends any model name to OpenAI, and refuses claude-haiku-5-5 with a pointer to computer-use.
• Each vendor's participant uses that vendor's own action set. Claude acts with Anthropic's computer toolset, which adds zoom, triple click, mouse button down and up, holding a key and reading the pointer position to the clicks, typing, keys, scrolls and waits both vendors have.
• Claude reads screenshots of at most 2,000 pixels a side and 4,784 visual tokens, one per 28 by 28 pixel tile. The desktop and wide presets fit; a larger execution.desktop.resolution is refused before any desktop exists.
• actor.serviceTier: fast is OpenAI Fast mode and is refused for Claude.
• A Claude session sends its whole history with every request, screenshots included. Past 60,000 input tokens Anthropic clears the oldest screenshots before the model reads the request, which keeps each request under the 100,000-token price tier. A session whose history reaches 30 MB, after some 60 to 300 turns, ends with an error naming that limit.

Model selection and spendCopy Anchor Link

Use the cost model and budgets. Set both a study model budget (maxTotalUsd) and a per-participant backstop (maxUsd); desktop time is additional.

Failed-participant rerunsCopy Anchor Link

Multi-participant CUA fan-out can be rerun surgically without mutating the source run:

npx humanish run cua-browser --rerun-failed-from latest --json --no-open
npx humanish run cua-browser --rerun-failed-from YOUR_RUN_ID --participants lane-02,lane-04

This creates a new linked run containing only the failed/blocked/timed-out/hollow participants (or the explicit --participants selection). --participants takes participant ids: each id in a participants list, or lane-01, lane-02, … by position when the study declares none. The new run.json records rerun.sourceRunId, selected participant ids, and previous participant statuses; the source run's verdict is left unchanged. This is intentionally not automatic retry; a passing rerun is evidence of a nondeterminism candidate and does not license erasing the original red participant.

Keep a long study going when your machine sleepsCopy Anchor Link

Each participant's desktop runs on E2B or Modal, and its decisions (the screenshot, the model call, the next action) run in a humanish process. When that process runs on your machine and the machine sleeps, the participants stop deciding and the run fails; its bundle records a host.suspended event with when and how long.

So a study that can runs that process in a sandbox on the participants' provider, the runner, by default:

npx humanish run humanish/studies/YOUR_STUDY.yaml

humanish plans the study here, so a refusal costs nothing, then creates the runner, installs this same humanish in it and starts the run there. The command first names the keys the run uses and says which ones E2B's egress proxy holds. While it runs:

• the command prints an Observer address on 127.0.0.1 that shows the run live, desktops included; the provider's own port authentication guards the runner, and the credential stays in this process;
• Ctrl-C stops the run as it stops a local one: the runner's run reclaims its participants, its evidence is copied here and the runner is stopped. After the participants finish, Ctrl-C stops only the run's analysis, and the command exits with the run's own result;
• closing the terminal leaves the run going. npx humanish observe --run YOUR_RUN_ID reconnects: it serves the live Observer again, follows the run to its end and copies it back, as the first command would have. npx humanish reclaim --run YOUR_RUN_ID stops it instead, with every sandbox it made, found by the run's tags even when the runner is gone.

When the run ends, its bundle is copied into .humanish/runs/YOUR_RUN_ID, each file checked against the runner's digest, and verify, review and analyze work on it as on a local run. The command exits with the run's own code.

The runner receives the study file, the persona files and each participants[].files entry, in your own E2B or Modal account. Its run uses only the keys its plan reads. On E2B, the OpenAI, Gemini and E2B keys stay in E2B's egress proxy, which adds each one to the runner's requests to its provider, and the run's environment holds placeholders; code in the runner can still read them back through the E2B API, as trust boundaries describes. On Modal, the keys are in the run's environment. A terminal study's runtime key (CODEX_API_KEY or OPENAI_API_KEY) enters the runner's environment on both providers, because the run in the runner places it in the shell sandbox's own egress rule or Codex command; the participant gets it as a local run would give it. It costs one more sandbox for the length of the study, and on E2B it counts against your concurrent sandboxes: the run in the runner holds its desktops to your limit less one. A study that declares no execution.concurrency runs remotely while the most participants its schedule runs at once fit under that, so 20 participants arriving over a day run remotely on a limit of 20; a declared execution.concurrency must fit under it as declared. On E2B the runner is made from E2B's stock desktop template, as a participant's desktop is, for its 8 GB of memory, so it costs about what one more participant does; no desktop starts on it. On Modal it has 2 CPU and 4 GiB.

A study runs remotely when it is a computer-use study whose participants use e2b-desktop or modal-desktop, with a model API key (not a local agent) and an app-url subject on the public internet, or a live terminal study on e2b-terminal or modal-terminal. The run must use none of --scorer, --count, --rerun-failed-from, --participants or --expose, and a terminal study's subject.product.upload must be inside the project, which the runner receives. Any other hosted study runs on this machine, and the command says why. To choose:

• --local runs this one run on this machine; execution.remote: false in the study does it for every run.
• --remote requires a remote run and refuses, with the reason, a study that cannot have one; execution.remote: true does it for every run.

For any other study, run it on a machine that stays on, such as a cloud VM with Node and your provider keys, inside tmux:

tmux new -s study
npx humanish run humanish/studies/YOUR_STUDY.yaml

Detach with Ctrl-b then d, and reattach later with tmux attach -t study. On that machine, npx humanish observe --run latest --serve --port 8732 serves the Observer on 127.0.0.1:8732; ssh -L 8732:127.0.0.1:8732 YOUR_VM lets you watch it from your laptop. On a Mac that stays plugged in, caffeinate -i npx humanish run humanish/studies/YOUR_STUDY.yaml keeps it awake for the run.

Run-owned cleanupCopy Anchor Link

npx humanish reclaim --check --run latest asks E2B whether each desktop the run created still exists, and npx humanish reclaim --run YOUR_RUN_ID stops the ones an interrupted run left. Knowing that cleanup happened explains both receipts.

Trust note: serve commands run inside the disposable sandbox with the declared subject env provisioned, the same trust class as a repo's package.json scripts. Only run study files you trust, and declare only the env names that the subject needs.

State-driven local adaptersCopy Anchor Link

The computer-use loop is provider- and substrate-agnostic. You can point a study at an already-running local dev server (subject.source: local-app) and drive it through its in-process JS contract (window.app.getState() etc.) with a custom ComputerUseExecutor (screenshot optional, appState as the progress signal) paired with a non-vision ComputerUseProvider (requiresFrame falsey), keeping personas, the Observer, the evidence bundle, redaction, and the friction loop, with no E2B desktop and no clone. Supply inProcess.executor + createProvider to runStudy (a config-only run without them fails closed with a structured error). See State-driven executor.

The complete npm example includes a synthetic loopback app, HTTP bridge, full config, provider, verification and cleanup. After installing humanish, run:

node node_modules/humanish/examples/participant/run.mjs

It uses a deterministic provider with no model calls or credentials. This checks the local-app integration; it does not measure persona behavior or app usability.