Configure hosted browser participants, device fidelity, screenshots, stop rules, dwell windows, and reruns.

A computer-use lab dispatches a registered computer-use actor (actors[0].type, resolved against the actor registry, e.g. openai-computer-use) to drive an app in a hosted E2B desktop browser and emit an evidence bundle under gitignored .humanish/ (full-fidelity screenshots by default, length-only typed text, and provider-neutral humanish.actor-trace.v1 traces). Start with the complete own-app example; the blocks below are reference fragments.

Choose a subject

Hosted browser studies support these sources:

subject.source: clone (+ execution.target: e2b-desktop + a computer-use actor): the lab clones your repo into the sandbox, runs your declared serve.install/serve.build/serve.start commands (detached, with readiness probing), and drives the served app at serve.url. Subject env var names declared in subject.env are provisioned from --env-file (provenance records names; persisted text uses known-value and pattern redaction). The bundle records provenance: repo, cloned commit, env names.
subject.source: app-url: drive your reachable deployment with policies.allowPublicTargets: true. The standard independent-world route accepts one participant for this shared URL. Library callers can also provision the target through the prepareDesktop hook.
subject.source: local-tree: package the lab resolution directory, upload its content-pinned archive, and serve it like a clone. Declare subject.serve and use subject.localTree.keep, exclude, and maxArchiveBytes to control the archive; do not declare repos or clone. Inspect what will be included and keep secrets out of the selected tree.

subject:
  source: clone
  repos: [example-org/example-app]
  serve:
    install: pnpm install --frozen-lockfile
    build: pnpm build
    start: pnpm start
    url: http://127.0.0.1:3000/
actors:
  - type: openai-computer-use
    mission: Explore the app as a first-time visitor and complete its primary flow.
execution: { target: e2b-desktop }
scenario: { mode: live }

npx humanish lab run cua-browser --dry-run --json --no-open

Live runs (scenario.mode: live) need OPENAI_API_KEY + E2B_API_KEY (pass via --env-file) and the optional peer dependency: npm i -D @e2b/desktop. A cloned subject is served inside the sandbox on loopback; to instead drive a deployment you own (a Vercel preview, staging), use an app-url subject with policies.allowPublicTargets: true. The actor's API key never enters the sandbox; only declared subject env names do. humanish init scaffolds an example at humanish/labs/cua-browser.yaml.

Off-app email/SMS verification (`comms`)

When a flow is gated behind an email or SMS the app itself sends (a signup verification link, a one-time code, a magic link), add a comms: block to the lab. Humanish redirects the app's email-API sends (via one adopter-named env var: with the official Resend SDK it is RESEND_BASE_URL, which the SDK reads on its own, so nothing in the app has to change; other providers need the app to actually pass the env through, and a run whose catch captures zero sends warns at teardown) into a catch inside the sandbox, so nothing leaves the machine. Every lane gets a deterministic inbox address automatically, and each persona's prompt carries the full handoff: the address to sign up with, the inbox URL to open, and that waiting for an email is a next step, not a blocker. The run bundle gets a digest-only humanish.comms-thread.v1 artifact (from/to/subject/link digests + an OTP count; no raw address, link, or code persists); the readable proof a persona saw the email is its inbox-page screenshots. Hosted on the clone/local-tree computer-use lanes and the concurrent shared-world route (warned inert elsewhere), vendor-neutral (Resend/SendGrid shaped, or a custom profile). An app-url study can instead use an adopter-hosted catch via comms.email.external; it must already be wired into your app. See the schema contract for the full comms shape and route limits.

Screenshots are full-fidelity by default

Run bundles live in gitignored .humanish/, so the Observer shows exactly what the persona saw. Set policies.redactScreenshots: true to persist blurred thumbnails at capture instead (for unowned subjects, or bundles you intend to share as-is). Raw bundles stay local in gitignored .humanish/; nothing scans the pixels, so review them before sharing anywhere. A redact-on-export step is planned. The frame sent to the model is always full-resolution regardless. See the capture and sharing policy. humanish verify reports raw-screenshot bundles as shareSafety.status: local_only; humanish feedback issue refuses them until the run is share-ready.

Device presets

execution.desktop.device picks the hosted desktop screen size: mobile (414×896), small-mobile (360×740), narrow-mobile (320×700), tablet (820×1180), desktop (1440×950, default), or wide (1920×1080). Honest fidelity: on the computer-use / E2B-desktop route width/height size the virtual display and browser outer window. The actual page viewport is smaller because browser chrome occupies space; Chromium-family live bundles measure it through CDP and record it separately from requested/verified screen geometry. Browsers without that measurement seam omit the viewport rather than guessing. A site's width-based responsive CSS still fires, and the model is told its device in the prompt. Without the block below there is no touch input, the device-pixel-ratio isn't rendered, and the user-agent stays desktop on this route. Set actors[0].lanes[].device to override the study preset for a participant. execution.desktop.resolution is a raw escape hatch that overrides the preset.

Mobile emulation

execution.desktop.fidelity: { mobileEmulation: true } turns every hosted Chrome/Chromium computer-use lane on a mobile preset (mobile, small-mobile, narrow-mobile) into a mobile-emulated browser before the participant arrives, and leaves desktop, tablet and wide lanes in the same run untouched: the lane's preset width/height become the CSS viewport (414 px for mobile, where the X screen itself cannot go below 500), the preset's device pixel ratio applies (deviceScaleFactor overrides it), touch events are on (touch: false turns them off) and the browser presents a mobile user agent (userAgent replaces the default iPhone Safari string). The run bundle records desktopGeometry.fidelity with tier: mobile-emulated, the request, the CDP methods applied, and resolved: what the page itself reported afterwards (navigator.userAgent, devicePixelRatio, innerWidth, maxTouchPoints, coarse pointer). A page without a viewport meta lays out at 980 px, as it would on a phone, and the bundle says so. Firefox cannot be emulated, so the lane fails closed instead of shipping a desktop run labelled mobile. The emulation holder also applies the settings to new page targets. Later observations record the page's fidelity and warn if it drifts from the requested settings. A bundle without a fidelity block is a responsive-viewport study whatever its preset is called.

When touch is enabled, desktop pointer input is converted to touch. Correct viewport and touch flags do not prove equivalent gesture behavior: in two hosted conformance probes, the SDK double click produced two single clicks while direct touch opened the original TodoMVC editor in both. From Humanish 0.81.0, these lanes carry a run warning to confirm gesture failures with direct or native touch before attributing them to the app. The earlier phone-lane outcomes remain observations of the measured Humanish input path. See the dated input-conformance receipt for the method and limits; physical-device behavior was not tested.

Desktop browser choice

Hosted computer-use lanes and shared-world actor seats use the route's historical opener unless you set execution.desktop.browser to chrome, chromium, or firefox. A concrete value means "launch this browser or fail"; it never silently falls back to whatever the image prefers. When configured, run bundles record the requested browser and the resolved in-sandbox command as desktopBrowser.

Deterministic stop conditions

Freeform computer-use actors can keep acting after the app has already reached the state you care about. Add stopWhen to the actor or a lane to stop immediately after a deterministic browser observation matches. Conditions inside one rule are ANDed together; rules under any are ORed. Lane-level stopWhen overrides the actor default.

actors:
  - type: openai-computer-use
    mission: Complete the assigned browser task.
    stopWhen:
      any:
        - id: dashboard-visible
          urlPathEquals: /dashboard
          textIncludes: Dashboard
    lanes:
      - id: reviewer
        entry: /items/123
        instruction: Review the item and return to the queue.
        stopWhen:
          any:
            - id: returned-to-queue
              urlPathEquals: /items
              textIncludes: Queue

A participant with a camera

execution.desktop.media.camera: { source: synthetic } gives a hosted Chrome lane a capture device: an ffmpeg test pattern generated in the sandbox (or a .y4m file of yours, uploaded). The browser's own permission dialog stays in the way by default (policies.mediaPermission: prompt), because the gate is where a real person hesitates or refuses; granted bypasses it for studies about what happens after. The bundle records the feed and the exact launch flags under desktopBrowser.media. A microphone needs an image with an audio stack (execution.desktop.template); the stock desktop has none, so a declared microphone without a template is refused before any spend.

A declared observation window

Some findings are "time passed and nothing broke": a call both participants stay on, an import that finishes, a dashboard that updates. A freeform participant with nothing to do keeps acting, so dwell lets the study hold instead. Once when matches (or after the first observation, when there is no when), the harness holds the page for ms, captures a frame every everyMs (default 10 s), takes no action and requests no model turn, then hands control back (then: continue, the default) or ends the session (then: stop). The window is recorded in the trace as deliberate, and it never outlasts the session budget. Lane-level dwell overrides the actor default.

actors:
  - type: openai-computer-use
    mission: Join the room, stay a while, then leave.
    dwell:
      when:
        any:
          - id: in-room
            urlIncludes: /room/
      ms: 120000
      everyMs: 10000
      then: continue

Supported primitives are urlIncludes, urlPathEquals, textIncludes, and appStatePathEquals. URL and text observations are runtime-only and are not persisted into the run bundle; the trace stores only the matched rule id and primitive names. Browser URL and text observation requires a Chrome/Chromium CDP session in the desktop. For deterministic browser-observed stops, set execution.desktop.browser: chrome or chromium.

Closing reports after an automatic stop

From Humanish 0.81.0. After a stopWhen match or dwell with then: stop, a provider that supports closing reports can make one read-only request using the final observation. The current OpenAI provider supports this; local coding-agent providers do not. There is no extra CLI flag. Desktop interaction has already stopped, and any actions requested by the closing response are rejected rather than executed.

The request needs at least one prior participant turn, the required observation, and remaining session/model budget. It is limited to 30 seconds or the smaller remaining deadline, without retry. Reported usage counts toward the existing model budgets; an in-flight request can cross a threshold.

The trace's debrief records completed, skipped, or failed, a reason, and usage availability. An accepted typed report contains summary and frictionReports. An empty friction list means the participant reported none; a missing or failed report does not. The report preserves the original stop reason and task outcomes. Natural endings and budget/error exits do not request this extra report. See reading outcomes.

Model selection and spend

Use the cost model and budgets. Set both a study model budget (maxTotalUsd) and a per-participant backstop (maxUsd); desktop time is additional.

Failed-lane reruns

Multi-lane CUA fan-out can be rerun surgically without mutating the source run:

npx humanish lab run cua-browser --rerun-failed-from latest --json --no-open
npx humanish lab run cua-browser --rerun-failed-from YOUR_RUN_ID --lanes lane-02,lane-04

This creates a new linked run containing only the failed/blocked/timed-out/hollow lanes (or the explicit --lanes selection). The new run.json records rerun.sourceRunId, selected lane ids, and previous lane statuses; the source run's verdict is left unchanged. This is intentionally not automatic retry; a passing rerun is evidence of a nondeterminism candidate and does not license erasing the original red lane.

Run-owned cleanup

Live providers can record resource evidence in run.json. Stored bundle IDs are mutable evidence and do not authorize provider mutation. The cleanup command writes a durable inspection receipt until Humanish has a verified resource-lease contract. Resources already recorded as killed become already_clean; recorded live or unknown resources become failed, which makes cleanup and verification fail closed:

npx humanish cleanup --run latest
npx humanish verify --run latest

Humanish does not enumerate or bulk-delete provider accounts from this command. Same-process teardown uses trusted in-memory provider handles. For an interrupted run, humanish reclaim --run YOUR_RUN_ID can stop the exact sandboxes recorded in that run's journal. See recovery.

Trust note: serve commands run inside the disposable sandbox with the declared subject env provisioned, the same trust class as a repo's package.json scripts. Only run lab configs you trust, and declare only the env names that the subject needs.

State-driven local adapters

The computer-use loop is provider- and substrate-agnostic. You can point a lab at an already-running local dev server (subject.source: local-app) and drive it through its in-process JS contract (window.app.getState() etc.) with a custom CuaExecutor (screenshot optional, appState as the progress signal) paired with a non-vision CuaProvider (requiresFrame falsey), keeping personas, the Observer, the evidence bundle, redaction, and the friction loop, with no E2B desktop and no clone. Supply cuaHooks.buildExecutor + buildProvider to runLab (a config-only run with no hooks fails closed with a structured error). See State-driven executor.

Computer-use reference