Known failure modes
humanish ships evidence as the product, so it owes you the error bars on that evidence. This page states four things: what the computer-use substrate cannot do yet, where simulated users diverge from real ones, what a passing humanish verify grade does not prove, and when to go recruit people instead.
The published numbers against simulation are cited here first, before a skeptic gets to use them. The first-party failures are our own runs, and the ids are quotable.
What the substrate can’t do yet
Computer-use agents are good at short bounded interface work and bad at long workflows. OSWorld 2.0 benchmarks 108 long-horizon real workflows with a median human completion time of about 1.6 hours. The best model measured, Claude Opus 4.8 with maximum thinking and batched tool calls, “completes only 20.6% of tasks at a 54.8% partial score”; GPT-5.5 “plateaus near 13%” (OSWorld 2.0, arXiv:2606.29537). A single task averaged 318 tool calls under Claude Opus 4.7 at maximum thinking.
The shape of the failure matters as much as the rate. From the same abstract, verbatim: “Rather than stumbling on basic GUI control or coding, they lose track of constraints, miss information that arrives mid-task, guess rather than ask the user, and skip verification, struggling most when a task hinges on hidden state they must recover” (arXiv:2606.29537). Clicking, typing, and reading a screen mostly work. Holding a multi-hour goal does not.
What humanish does about it
Missions are bounded and the clock is published. The Excalidraw study ran four lanes to completion in 6m 06s wall-clock at ~$1.54 estimated (run cua-2026-08-07T17-44-48-760Z-87389419, rates as of 2026-08-05). Nothing in the shipped labs asks an actor to hold an hour-plus goal, because the benchmark above says it would fail about four times in five.
A stalled lane records why it stopped and fails. humanish trips a backstop after N consecutive turns with no change to the UI state and writes the reason into the bundle verbatim. In the Excalidraw study, lane 03 sketch-shapes recorded “gave up: 8 consecutive turns with no change to the UI state”, and the study publishes as 3/4 rather than 4/4.
Our own site study lost half its lanes to this. Four cold-visitor lanes ran against the live humanish.dev on 2026-08-08:
- Study
- humanish.dev · cold visitors
- Date
- 2026-08-08
- Lanes
- 2/4 passed
- Verify
- 16/16 checks
- Est. cost
- ~$3.57 · estimated (rates as of 2026-08-05)
- Dead lanes
- ~$1.91 of ~$3.57
Both dead lanes died on our own page. Inside the scroll-pinned replay section, a scroll that doesn’t cross a step threshold changes nothing visually, so eight small scrolls produce eight identical screenshots and the staleness guard concludes the lane is spinning. The screenshots show the actor had stepped several panels before it stalled, so it was making progress in the page’s own terms. Filed as danielgwilson/humanish#393: the guard should include scroll position in the state fingerprint. The two dead lanes account for ~$1.91 of the study’s ~$3.57 estimated total.
That finding cuts against us twice. The guard is our own heuristic, and the page it broke on is our own marketing surface.
What simulated users get wrong
2.1A model’s clicks are not a population’s clicks
Twelve first-click tests drawn from real UX practice, n = 3,431 real participants: GPT produced a “significantly different distribution from real data in 53% of tasks.” Personas, chain-of-thought, and sampling changes “fail to create sensible fidelity improvements apart from inflating believability” (What Would GPT Click, arXiv:2605.18302).
That last clause is the sharpest published line against persona prompting, and it applies to humanish’s own persona blocks. Adding a persona description makes output read more plausibly without making it more accurate.
What humanish does: lanes do not predict clicks. A lane executes a declared mission in a real browser on a hosted desktop, and the bundle records the actions taken, the screenshots seen, and the outcome. humanish publishes no click-distribution claim and no population match. Our own design notes concede the harder version of this: at the kinematic level of pointer paths, timing, and motor noise, the gap between agents and people is total and trivially detectable (arXiv:2604.09574), and fidelity here is scoped to which affordances an actor used, not to behavioral realism.
2.2The result depends on which model plays the user, and it is worst for the users you can least recruit
Agent success rates vary “up to 9 percentage points across different user LLMs.” Evaluations with simulated users “exhibit systematic miscalibration, underestimating agent performance on challenging tasks and overestimating it on moderately difficult ones.” Simulated users are “a differentially effective proxy for different populations, performing worst for AAVE and Indian English speakers,” with AAVE speakers seeing “consistently worse success rates and calibration errors” than Standard American English speakers (Lost in Simulation, ACL 2026, arXiv:2601.17087).
Plainly: this lands on humanish’s own framing. The site says “user testing for the users you can’t recruit.” The populations that are hardest to recruit are the ones this literature simulates worst, which makes the hardest case for humanish also the weakest case. A humanish persona is an authored YAML description, not an interview-grounded twin of a real person, and nothing in the harness measures how close it lands.
What humanish does: the actor and model are recorded in every bundle. The four self-study lanes above ran openai-responses-cu on gpt-5.5; swapping that model would move the results by an amount we have not measured and do not claim to know. humanish makes no representativeness claim and emits no segment comparison. That last omission is deliberate. Across the General Social Survey and the World Values Survey, LLM synthetic respondents “inflate between-segment gaps two to fourfold” and “would direct a team to the wrong segment in half of U.S. and most cross-cultural cases,” and at the individual level “no LLM beats even the strongest baseline” (When Synthetic Users Fail, arXiv:2607.26348). There is no survey path in humanish, and there is no demographic-segment output.
2.3A persona’s opinion is the least trustworthy thing in the bundle
Across 11 models, LLMs “preserve user’s face 45 percentage points more than humans in general advice queries and in queries describing clear user wrongdoing,” and when given either side of a moral conflict they affirm both sides in 48% of cases (ELEPHANT, arXiv:2505.13995). A model asked whether a product is good will tend toward yes.
What humanish does: reports ship verbatim and sit next to the trace that produced them, so the words can be checked against the actions. The Excalidraw study’s three passing lanes reported, in full, “Done”, “Done.” and “Done” — worth nearly nothing on their own, which is the reason they are published beside 28 screenshots, ordered action traces, and lifecycle events rather than as a testimonial. humanish does not ask personas to rate, score, or recommend the product.
2.4A cooperative simulator will tell you your product works
Grounding simulators in 14,000+ real human-LLM conversations, RealUserSim finds that grounded simulation “acts as a realistic stress test, surfacing three failure mechanisms invisible to cooperative simulators (mean -3.2% to -3.5% task success degradation)” (arXiv:2605.20204). Realistic users make a product look worse, which means a happy-path demo is evidence of a compliant simulator.
What humanish does: failed lanes are published at the same weight as passing ones. The Excalidraw study ships as 3/4. The self-study ships as 2/4. Neither number was tuned up before publication.
What the evidence layer does and does not guarantee
What humanish verify checks
Mechanical properties of the bundle. Run storage containment, run schema and bundle shape, redaction, a public-safety scan, actor engagement, actor verdict consistency, subject state provenance, cleanup receipt, rerun lineage, and cost-estimate labeling, among others. The Excalidraw study and the self-study each passed 16/16 on their runs.
Share-safety grades fail closed: a bundle grades share_ready, local_only, or blocked, and a bundle that cannot pass every gate never grades share_ready. Both published studies graded local_only, because they hold full-fidelity screenshots. Publishing the crops on the study page was a separate, reviewed act by a human.
That gate binds the operator too, on camera
During the self-study, humanish feedback issue refused to draft from the local_only bundle with HUMANISH_FEEDBACK_SHARE_SAFETY_BLOCKED. The share-ready re-run (redactScreenshots: true, run cua-2026-08-08T21-39-07-933Z-56b68f47) graded share_ready at 16/16 and produced a draft. The draft was wrong. Its actual field read “This dry-run produced a contract-proof bundle only; no browser or product behavior was exercised” — on a live run with 15 redacted screenshots and a real action trace. The lane’s own outcome, its gave-up reason, the subject URL, and the screenshot counts appear nowhere in the draft. Filed as danielgwilson/humanish#392. The share-safety gate worked. The drafter did not. Until #392 closes, read the bundle, not the draft.
What verify does not check
Verify has no opinion on whether a finding is true, whether the mission was worth running, or whether the persona behaved like a person would. It checks that the instrument reported honestly about itself. A green lane is a lane that ran and reported, and nothing further.
Outcome-only scoring is known to be blind to how a result was reached. WebArena’s URL evaluator scores on the final page URL, so a single goto to the reference URL earns full credit (arXiv:2307.13854). τ-bench states the general form: a state-based reward is necessary but not sufficient, because it cannot see whether policy was followed (arXiv:2406.12045). And the gradient is real, since calling site APIs instead of driving the UI roughly doubles WebArena scores (arXiv:2410.16464). humanish’s answer is to record which affordance class each action used and leave the verdict open, rather than to bake one in.
Judgment is the adopter’s seam
Product semantics belong in the adopter’s repo. A lab manifest can declare review.scorer.ref (or pass --scorer <path>) to load adopter-defined scoring against the full trace. The design position, verbatim from humanish’s own actor-fidelity notes: “The harness emits facts; the adopter decides what they mean.”
The field is converging on the same premise. A 2026 survey of evidence tracing and execution provenance in LLM agents opens on the reason: “Final-answer accuracy alone cannot explain how an output was produced, which evidence supported each claim, whether tool calls were justified, how memory influenced later decisions, or where failures originated” (arXiv:2606.04990). Whether a model can write a useful UX critique is separately unsettled: across eight frontier models, “UX judging is neither saturated nor one dimensional” (UXBench, arXiv:2606.16262).
Cost numbers are estimates
Every cost in a bundle is labeled fullyEstimated with a ratesAsOf date. The self-study’s ~$3.57 and the Excalidraw study’s ~$1.54 are estimates at 2026-08-05 rates, not invoices.
When to recruit real users instead
When you can recruit real users, do that. humanish covers the runs that otherwise never happen.
That line is on the front page and it is the honest boundary. Cases where humanish is the wrong instrument:
GPT’s click distribution differed significantly from 3,431 real people in 53% of tasks (arXiv:2605.18302), and agent success moves up to 9 percentage points on the choice of user model alone (arXiv:2601.17087).
Frustration, delight, hesitation, and trust do not appear in a bundle. There is no field for them because there is no measurement behind them.
Models inflate between-segment gaps two to fourfold and pick the wrong segment in half of U.S. cases (arXiv:2607.26348). humanish emits no segment output.
The best model completes 20.6% of hour-plus workflows (arXiv:2606.29537). Scope missions accordingly.
Lanes drive a standard desktop browser at 1920×1080 and record no assistive-technology path.
What is left is real and narrower: driving the build you have, right now, on runs that would otherwise not happen, and getting back a durable bundle whose failures are visible.
Standing caveats
These numbers are current as of 2026-08-08. OSWorld 2.0 results in particular will move.
The behavioral class of simulated-user research is under-validated rather than validated. Treat humanish output as early, directional evidence and as hypothesis generation. It is not a substitute for recruiting the people it stands in for.
Anything on this page that turns out to be wrong is a bug. Open an issue with the run id.