Read results and draft feedback
Turn a participant's difficulty into a finding you can reproduce, verify, and act on.
A study is useful when you can point to what happened, where it happened, and what to investigate next. Start with the run, then follow the participant's evidence.
Open the run
npx humanish runs
npx humanish observe --run latest --open
npx humanish review --run latest --jsonreview prints the run's verdict and participant summary, then the findings of its post-run analysis: each one's plain headline and what the person experienced, then its title, impact, confidence and recovery, the participants it affected, the frames and capture files it cites, and a next step. Design findings follow: problems a designer would notice in the captures, most severe first, each with the screen, a suggestion and the capture files that show it. With --json they are under analysis. When a run has no findings yet, review says why and prints the command that gets them.
Observer shows the participant's account alongside screenshots and recorded actions. Read the final report, then inspect the moment behind each claim. When tasks have been declared, the per-task funnel helps separate reaching the goal from merely reaching the end of the session.
Use a specific run ID in place of latest when you compare studies or save a finding. latest changes after another run.
Separate a finding from a broken run
| Observation | What to investigate |
|---|---|
| Participant finishes the intended task | Check the actions and task evidence support the claim. |
| Participant abandons or reports friction | Inspect that moment in the app. This may be the study's useful finding. |
| Install, provider, or sandbox fails | Fix the study setup; the participant may never have reached the app. |
| A task was never measured | Treat it as unknown, even if the participant said they completed it. |
The review's verdict says what the participants experienced. The run's ok says whether the run worked as an execution, and .humanish/runs/<id>/status.json records it as outcome.ok, with outcome.execution.failures naming each failure. The two can disagree. A participant whose session passed still reads pass when its model provider's cleanup could not be confirmed, and the run has ok: false. A terminal run that exceeds its spend cap keeps the agent's own verdict and fails with HUMANISH_TERMINAL_CAPS_EXCEEDED. A run with ok: false exits non-zero.
Synthetic participants can miss problems or report things that do not reproduce. Confirm a finding against the app before treating it as a defect. See known failure modes for when to recruit real people.
For a worked example, read the paired Save-button study: all 12 attempts saved the task, even though half used an app with a broken Save button. The comparison also exposed false feedback candidates in humanish itself.
For a finding that led to an app change, read the TodoMVC keyboard repair: two uninterrupted keyboard sessions were blocked before a local Edit patch, and two completed after it. One provider interruption per version remains in the study counts.
Verify before sharing
npx humanish verify --run latest --jsonVerification checks the bundle's evidence and automated public-safety gates. It is not a usability score or certification that a finding is true.
shareSafety.status | Meaning |
|---|---|
share_ready | Eligible for public feedback drafts after review. |
local_only | Valid local evidence that should not be shared as-is: full-fidelity screenshots, a file the scan cannot read as text, a recording, or real email. |
blocked | The bundle failed verification or a public-safety gate. |
Full-fidelity screenshots are local-only by default. A screenshot counts as blurred only when its participant's trace marks it blurred and it is a thumbnail the redactor wrote: at most 128 px wide, holding image data only. Any other frame counts as full-fidelity, and a screenshot carrying PNG text or metadata chunks fails verification.
The scan reads every other run file by its bytes. It matches secret and local-path shapes, including after undoing JSON and JS escapes, percent-encoding, quoted-printable, HTML character references, hex and base64. A file it cannot read as UTF-8 text, such as an image or archive that is not a participant's screenshot or a registered recording, or a file larger than 32 MiB, keeps the run local_only with the reason UNSCANNED_ARTIFACT, which names the file. A participant's trace larger than 32 MiB is one: verify names it in its warnings and checks that participant without its items, the Observer shows the participant without them, and humanish analyze refuses the run. sandbox-receipts.ndjson holds the run's raw sandbox ids for reclaim, so the raw-id check skips it; its text is still scanned for secrets. No Observer server serves it or status.json, and export --format bundle leaves both out. The scan does not certify that names, personal information, or sensitive content are absent from text or pixels. Use synthetic data and review what you share. To blur frames at capture instead, set policies.redactScreenshots: true (screenshots).
Draft an issue
Feedback needs a live run. feedback draft, verify, issue and issue-url refuse a
dry run with HUMANISH_FEEDBACK_REQUIRES_LIVE_RUN before verification starts, because a dry
run tests no product behavior.
Use npx humanish feedback list --run latest to see recorded candidates and any
saved draft. With no candidates, feedback draft and feedback issue can still
create a follow-up that summarizes the recorded review. That summary is not a new
finding, so the candidate list stays empty. Listing does not create a draft or verify
that it is safe to share.
For a share_ready run:
npx humanish feedback draft --run latest
npx humanish feedback verify --run latest
npx humanish feedback issue --run latest --repo your-org/your-app --format markdownThese commands generate and check a draft. They do not publish it or call GitHub to create an issue. Read the rendered Markdown, reproduce the finding, and edit it before posting.
If verification says local_only or blocked, public feedback generation refuses the bundle. You can still inspect local evidence in Observer. For a valid run whose only sharing restriction is raw screenshots, create a redacted copy:
npx humanish export --run YOUR_RUN_ID --format bundle --redact-screenshots --out ./shared-study
npx humanish verify --cwd ./shared-study --run YOUR_RUN_ID --json
npx humanish feedback draft --cwd ./shared-study --run YOUR_RUN_ID
npx humanish feedback issue --cwd ./shared-study --run YOUR_RUN_ID --repo your-org/your-app --format markdownThe new directory contains a standalone .humanish/runs/YOUR_RUN_ID/ bundle. Your original keeps its readable screenshots; the copy contains blurred thumbnails and a derivation.json receipt. The copied study retains its identity and outcomes. It is not another participant or a new attempt in your original run history.
Export rebuilds Observer and independently verifies the copy before publishing it. It refuses an existing destination, incomplete or invalid evidence, unsupported image formats and embedded image payloads. PNG and lossless WebP screenshots are supported, and the copy holds each as a blurred PNG; the copy's text and findings still need your review before sharing. The copy drops an image that nothing in run.json cites, and export refuses a run whose feedback, adapter or stream evidence cites an image that is not a participant's screenshot. A blocked bundle must be corrected at its source. Do not change a stored grade to force it through.
Export or compare evidence
npx humanish export --run latest
npx humanish stats --jsonBy default, export writes a self-contained Observer HTML file with its screenshots. It requires share_ready; --local-only permits a watermarked local export, which still needs review before sharing. Use --format bundle --redact-screenshots for the separate redacted workspace shown above.
stats summarizes outcomes, duration, and estimated costs. Unknown cost is reported as unknown. Use --study your-app to focus on one study and --since to select a period; the CLI reference lists accepted flags.
Keep original run bundles so a later comparison can return to the same screenshots, actions, and app revision. A before/after claim is stronger when the same task succeeds for multiple participants after the change.