diff --git a/CHANGELOG.md b/CHANGELOG.md index 92512f1f0..9008155b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,18 @@ # Changelog +## [1.91.232.0] - 2026-10-26 + +**The setup-browser-cookies workflow judge now has a preregistered calibration corpus, and its structured-output calibration is recorded as held, so the judge stays on today's request (GSTA-91).** In train-4 gate 40173759210, the cookie judge (claude-sonnet-5-5, free-text transport) returned JSON with an unescaped quote. The repo's fix for that shape is the JSON-schema transport, and `test/fixtures/judge-calibration/README.md` lets a configuration switch only after a calibration lands. The cookie configuration had no corpus until now. Its comparison-1 run removed the parse errors (old 2 of 144 samples, new 0 of 72) and matched or improved every other measure. It is held on one landing condition: a new false pass on a07. a07 scored 4/4/4 on all three new samples and on all three `old-b` samples. `old-a`'s a07 panel is incomplete because one old sample returned invalid JSON (both valid `old-a` a07 samples also scored 4/4/4), and false passes are counted against `old-a` alone, so the result counts against the new request only. + +### Tests +- `test/fixtures/judge-calibration/cookie-workflow/`: 24 items. p01 is the production bundle (`buildCookieWorkflowJudgeInput`); a02 and a05 are real earlier revisions; the rest are hand-edited variants rendered by the same builder. Labels were preregistered and committed before any request with the schema was sent. A separate run (openai/gpt-6.1-sol) then labeled every item blind, and a third reviewer (google/gemini-4-argon) adjudicated the two disagreements: a01 kept pass, a08 changed to fail. `runs/` holds every sample and the result. The cost was 216 calls and $2.81, priced in `manifest.json` before dispatch as a separately funded entry. +- `scripts/judge-calibration-configs.ts`: the `cookie-workflow` configuration builds prompts with `buildWorkflowJudgePrompt(COOKIE_WORKFLOW_JUDGE, …)` and uses the eval's median floors of 4/3/4. Its new request differs from the old one only by `WORKFLOW_JUDGE_RESPONSE_SCHEMA`. +- `test/judge-calibration.test.ts` covers the new configuration and its pass rule, including the actionability floor. The wiring test now requires the cookie registration to send the schema exactly when the recorded cookie calibration lands. It doesn't land, so the registration stays unchanged and 15 workflow registrations use the schema. The cookie registration may only spread `COOKIE_WORKFLOW_JUDGE`, so the calibrated request stays the one production sends. +- Two new free tests: the calibration builds byte-for-byte the prompt the cookie eval sends, and `analyze()` reproduces the committed result from its labels, samples, stop record and request options. + +### Gate integrity +- No judge request, rubric, threshold, sample count, landing rule or case changed. The cookie judge keeps its free-text request. No gate exception is proposed. + ## [1.91.231.0] - 2026-10-26 **A fresh install is about 520 MB smaller: `./setup` installs production dependencies only.** Setup ran `bun install` without `--production`, so every user also got the test and eval toolchain from `devDependencies`, mostly the Claude Agent SDK's native binaries (both the glibc and musl builds on Linux, ~440 MB). Measured on a clean Ubuntu 24.04 container with a cold Bun cache (4 vCPU, two runs each): `node_modules` goes from 1,094 MB to 572 MB, the checkout from 1.74 GB to 1.22 GB, and the whole install with Chromium from 2.42 GB to 1.90 GB. `./setup` took 24.6 s and 20.0 s before, 21.1 s and 18.9 s after. diff --git a/TODOS.md b/TODOS.md index 8453a0b26..ed7010554 100644 --- a/TODOS.md +++ b/TODOS.md @@ -777,18 +777,22 @@ Each item was deferred with a reason during the `/claude-api prompt-audit` clean - **Judge prose and rubric cleanup (W2 deferral)** — the arm, qa health-rubric, qa anti-refusal, cross-skill and voice judges, and the default workflow judge, now send a JSON schema; their prompt-audit s11 M1-M7 prose cleanup is still - open, as is the qa anti-refusal rubric's "always … regardless" wording. Two + open, as is the qa anti-refusal rubric's "always … regardless" wording. Three prose steps were triggered and not run: the qa workflow judge (its schema comparison held: 4/24 new flips vs 2/24 old, one new false pass on a06, zero - format errors, so it stays on today's request) and the default workflow judge + format errors, so it stays on today's request), the default workflow judge (landed, but one provider refusal on corpus item f03 under the new prompt, - matched by one under the old prompt). Reuse `test/fixtures/judge-calibration` + matched by one under the old prompt) and the cookie workflow judge (GSTA-91: + held on one new false pass on a07, zero new-request format errors, so it stays + on today's request; today's request itself false-passes a07 in `old-b` and on + both valid `old-a` samples, and a08 in `old-a`, so the prose step targets those). Reuse `test/fixtures/judge-calibration` and `scripts/judge-calibration.ts`. Priority P3. Effort M. -- **Uncalibrated workflow-judge configurations** — build corpora and calibrate - schema transport for the frontier configuration (review/SKILL.md workflow) and - the cookie configuration (setup-browser-cookies/SKILL.md workflow); both stay on - today's request. After any transport change, re-run the review and cookie - workflow judges. Priority P3. Effort M. +- **Uncalibrated workflow-judge configurations** — build a corpus and calibrate + schema transport for the frontier configuration (review/SKILL.md workflow); it + stays on today's request. The cookie configuration + (setup-browser-cookies/SKILL.md workflow) has a corpus and a held comparison-1 + (GSTA-91), so it also stays on today's request. After any transport change, + re-run the review and cookie workflow judges. Priority P3. Effort M. - **Claude model auto-detection for the skill overlay** — when no `claude_overlay_model` is set, detect the Claude Code model (`ANTHROPIC_MODEL`, user and managed settings `model`) in `./setup` and render its pinned overlay. diff --git a/VERSION b/VERSION index 6b0b2b443..ca6a0eb0c 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.91.231.0 +1.91.232.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index b26059b48..d582ddc09 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.91.231.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.91.232.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/package.json b/package.json index f2cf08342..5302ec594 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.91.231", + "version": "1.91.232", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", diff --git a/scripts/judge-calibration-configs.ts b/scripts/judge-calibration-configs.ts index 426b3f8ba..926bdd415 100644 --- a/scripts/judge-calibration-configs.ts +++ b/scripts/judge-calibration-configs.ts @@ -13,6 +13,7 @@ import { judgePanelMedian, judgePanelMajority, } from '../test/helpers/llm-judge'; import { buildWorkflowJudgePrompt, WORKFLOW_JUDGE_RESPONSE_SCHEMA } from '../test/helpers/workflow-judge-input'; +import { COOKIE_WORKFLOW_JUDGE } from '../test/helpers/cookie-workflow-judge-input'; import { CLAUDE_JUDGE_EVAL_MODEL } from '../lib/eval-model'; import type { CalibrationConfig, Verdict } from './judge-calibration'; @@ -106,4 +107,11 @@ export const CALIBRATION_CONFIGS: CalibrationConfig[] = [ 'land-and-deploy/SKILL.md workflow', 'canary/SKILL.md monitoring loop', 'benchmark/SKILL.md perf collection', 'setup-deploy/SKILL.md platform setup', 'sync-gbrain/SKILL.md read-only readiness', 'retro/SKILL.md instructions', 'qa-only/SKILL.md workflow', 'gstack-upgrade/SKILL.md upgrade flow']), + { + id: 'cookie-workflow', priority: 8, cases: ['setup-browser-cookies/SKILL.md workflow'], model: COOKIE_WORKFLOW_JUDGE.model, + build: inputs => buildWorkflowJudgePrompt(COOKIE_WORKFLOW_JUDGE, { files: [], text: inputs.bundle! }), + oldOptions: { max_tokens: DEFAULT_JUDGE_MAX_TOKENS }, + newOptions: { max_tokens: DEFAULT_JUDGE_MAX_TOKENS, jsonSchema: WORKFLOW_JUDGE_RESPONSE_SCHEMA }, + ...scoreJudge(COOKIE_WORKFLOW_JUDGE.thresholds), estimatedOutputTokens: 1200, + }, ]; diff --git a/test/fixtures/judge-calibration/README.md b/test/fixtures/judge-calibration/README.md index b1944df77..c8c223cb2 100644 --- a/test/fixtures/judge-calibration/README.md +++ b/test/fixtures/judge-calibration/README.md @@ -11,7 +11,8 @@ Each `/corpus.json` holds 24 items: 8 clearly passing, 8 clearly failing has been sampled, with the same frozen requests. `expected` is the verdict a careful grader would reach under the eval's pass rule (`pass_rule`). The author labeled every item; a separate agent run then labeled every item blind (shuffled ids, no author -labels) before any schema was written. `second_review` records that verdict, and each +labels) before any schema was written (for `cookie-workflow`, before any request with the +schema was sent). `second_review` records that verdict, and each disagreement carries `author_expected` and the third reviewer's `adjudication`. `scripts/judge-calibration.ts` runs the comparison and `scripts/judge-calibration-configs.ts` @@ -21,7 +22,7 @@ and priced usage, and `comparison-1.result.json` holds the flip rates, agreement false passes, error rates, p95 latency, landing decision and the UC3 prose-step trigger. `manifest.json` prices every planned call before dispatch. -## Comparison (1) results (2026-10-03, `claude-fable-5-1`) +## Comparison (1) results (2026-10-03, `claude-fable-5-1`, unless the row says otherwise) Flips are panel-verdict differences on items where both panels completed: old-vs-old is `old-a` vs `old-b`, new-vs-old is `new` vs `old-a`. Agreement is items whose panel @@ -38,14 +39,24 @@ most 80% of `JUDGE_MS` (96 s). | `cross-skill` | 0/24 | 0/24 | 23 / 23 | 0 | 0 / 0 | 19.7 / 22.6 | 0/8, 8/8 | $19.86 | lands | | `voice` | 0/24 | 0/24 | 22 / 22 | 0 | 0 / 0 | 13.3 / 13.7 | 0/8, 8/8 | $8.09 | lands | | `workflow-default` | 2/23 | 3/23 | 21 / 21 | 0 | 1 / 1 (refusal, f03) | 29.5 / 25.8 | 0/8, 8/8 | $19.66 | lands; UC3 prose step triggered by the errored sample | +| `cookie-workflow` (2026-10-25, `claude-sonnet-5-5`) | 1/22 | 1/23 | 22 / 23 | 1 (a07) | 2 / 0 (parse, a05 and a07) | 13.6 / 9.6 | 1/7, 7/8 | $2.81 | held: one new false pass | -No sample errored under either prompt except one refusal each on `workflow-default` item +Among the 2026-10-03 configurations, no sample errored under either prompt except one refusal each on `workflow-default` item `f03` (a stubbed cookie-import workflow): the old prompt refused once in `old-b` and the new prompt once, so that item's panels are incomplete and it is excluded from the flip counts (23 compared). The refusal is content-driven rather than format-driven, but under the eng trigger any errored new sample calls for a calibrated prose step, which is deferred to TODOS. `workflow-default` was funded after the first pass with a $25 extension of the W2 reservation; its 14 cases now send `WORKFLOW_JUDGE_RESPONSE_SCHEMA` without the -compact-reasoning validator (still only on `ship`). The frontier (`review`) and cookie workflow -configurations have no corpus and stay on today's request. A $0.12 two-call pilot sized +compact-reasoning validator (still only on `ship`). The frontier (`review`) configuration has no corpus and stays on today's request. A $0.12 two-call pilot sized output tokens before the manifest was priced. Total W2 spend: $90.47 of $115. + +`cookie-workflow` (GSTA-91, separately funded) is held and stays on today's request. Its one +new false pass, a07, scored 4/4/4 on all three new samples and on all three `old-b` samples; +`old-a`'s a07 panel is incomplete because one old sample returned invalid JSON (the defect the +schema transport removes), so the comparison counts it against the new request only. Both +valid `old-a` a07 samples also scored 4/4/4. The old +request failed to parse 2 of 144 samples; the new one 0 of 72. The failed landing condition +triggers the UC3 prose step, which is deferred to TODOS. Its `manifest.json` entry was priced +separately (`priced_at`, `funding`) and is maintained by hand: the `manifest` command rewrites +the whole file, so keep that entry when regenerating. diff --git a/test/fixtures/judge-calibration/cookie-workflow/corpus.json b/test/fixtures/judge-calibration/cookie-workflow/corpus.json new file mode 100644 index 000000000..014904692 --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/corpus.json @@ -0,0 +1,420 @@ +{ + "config": "cookie-workflow", + "judge": "Cookie workflow judge: buildWorkflowJudgePrompt(COOKIE_WORKFLOW_JUDGE, {files: [], text: bundle}) in test/helpers/workflow-judge-input.ts with judgeContext and judgeGoal fixed to COOKIE_WORKFLOW_JUDGE (test/helpers/cookie-workflow-judge-input.ts), model claude-sonnet-5-5; bundles produced by buildCookieWorkflowJudgeInput (same file) from the production files, real git revisions, or hand-edited copies where noted", + "pass_rule": "panel medians clarity >= 4, completeness >= 3, actionability >= 4", + "items": [ + { + "id": "p01", + "category": "pass", + "expected": "pass", + "split": "dev", + "rationale": "Production bundle: buildCookieWorkflowJudgeInput(repo root).text, byte-identical (setup-browser-cookies/SKILL.md 146-206 plus BROWSER.md 569-617; the text is unchanged since the v1.91.150.0 rewrite). Five ordered steps: the Aside READY and Mode: cdp stops, all five decisions asked together before any $B command with defaults and consent, verification variables set before the daemon starts, a picker-vs-direct rule, a three-line receipt report that separates copied cookies from verified sign-in, plus error tables and platform limits in the referenced section. The only wrinkle is step 3.1 naming `$B stop` before step 3.2 locates `$B`. A careful grader gives 4-5 on every dimension. Note: the CHANGELOG records one earlier production sample on this text at 4/4/4, so it sits nearer the boundary than the other p items.", + "inputs": { + "bundle": { + "file": "inputs/p01.bundle.txt" + } + }, + "second_review": { + "verdict": "pass", + "confidence": "high", + "reason": "The workflow requires explicit source and scope choices, private verification configuration with an existing-daemon restart, and consent for a precisely described reset. Receipt-based reporting and bounded, platform-aware recovery preserve the distinction between imported cookies and verified identity." + } + }, + { + "id": "p02", + "category": "pass", + "expected": "pass", + "split": "dev", + "rationale": "Production with step 3 reordered: locate `$B` first, then (verification only) `$B stop` a running daemon and have the user set the two variables privately, then `$B status`, which starts the daemon with them. Every decision, command, consent gate, verification rule and recovery path is unchanged, and the one ordering wrinkle in production (`$B stop` before `$B` is found) is gone. Clarity and actionability 4-5.", + "inputs": { + "bundle": { + "file": "inputs/p02.bundle.txt" + } + }, + "second_review": { + "verdict": "pass", + "confidence": "high", + "reason": "The preparation sequence locates the executable, stops an existing daemon when verification is requested, configures the private assertions, and then starts the daemon before opening the target. Explicit source choices, scoped reset consent, receipt semantics, and platform-specific recovery supply the remaining execution requirements." + } + }, + { + "id": "p03", + "category": "pass", + "expected": "pass", + "split": "dev", + "rationale": "Production SKILL.md rewritten in plainer prose (\"Ask every question before touching $B\", \"Report what actually happened\"); same five steps, same five questions with defaults, consent and scope, same commands, verification-before-startup rule, rejection recovery, three-line report and platform boundaries. BROWSER.md section unchanged. Wording only, so a careful grader scores it like production or slightly higher for readability: 4-5.", + "inputs": { + "bundle": { + "file": "inputs/p03.bundle.txt" + } + }, + "second_review": { + "verdict": "pass", + "confidence": "high", + "reason": "The instructions connect explicit browser/profile/domain choices to target navigation, private startup verification settings, and opt-in storage clearing with a concrete scope. They also define the three receipt outcomes and safe recovery boundaries, with no contradiction that blocks the intended workflow." + } + }, + { + "id": "p04", + "category": "pass", + "expected": "pass", + "split": "dev", + "rationale": "BROWSER.md content folded into SKILL.md: the receipt-failure-reason and import-error tables (with next steps) move into a new step 4 \"Errors and recovery\" subsection next to the verification-settings rejection, and the BROWSER.md section keeps the source/profile, examples, verification, reset and platform text. Nothing is removed; recovery is now on the main path instead of behind a cross-reference. Clarity and actionability 4-5.", + "inputs": { + "bundle": { + "file": "inputs/p04.bundle.txt" + } + }, + "second_review": { + "verdict": "pass", + "confidence": "high", + "reason": "The entrypoint includes the concrete import-error and receipt-failure recovery actions alongside the source, consent, and verification steps. The reference supplies compatible examples and platform limits, so moving the recovery tables into the entrypoint creates no missing or contradictory execution step." + } + }, + { + "id": "p05", + "category": "pass", + "expected": "pass", + "split": "dev", + "rationale": "Production with non-essential commentary trimmed: the Local State/Preferences naming detail, the Windows App-Bound background sentence, the picker-verbatim note, the isolated-world reset mechanism, undetected Opera layouts, the not_installed path-listing detail and SKILL.md's duplicate whole-context sentence (still in BROWSER.md). Every decision, command, consent gate, verification rule and recovery row remains. Clarity and actionability 4-5.", + "inputs": { + "bundle": { + "file": "inputs/p05.bundle.txt" + } + }, + "second_review": { + "verdict": "pass", + "confidence": "high", + "reason": "The bundle retains the necessary authorized-source decisions, startup verification configuration, exact storage-reset scope, receipt-based reporting, and actionable recovery tables. Less detail about profile-label discovery and internal implementation does not prevent executing the documented workflow safely." + } + }, + { + "id": "p06", + "category": "pass", + "expected": "pass", + "split": "heldout", + "rationale": "Production with step 2's numbered questions recast as a decision table (decision, default, what the user gives, flag) followed by the verification rule, the exact storage-reset scope and failure caveat, and the picker-vs-direct rule. Steps 1 and 3-5, the platform boundaries and the BROWSER.md section are unchanged. Same content in a more scannable form; a careful grader gives 4-5.", + "inputs": { + "bundle": { + "file": "inputs/p06.bundle.txt" + } + }, + "second_review": { + "verdict": "pass", + "confidence": "high", + "reason": "The decision table identifies defaults, required user inputs, and corresponding flags, while the surrounding instructions define verification evidence and the exact reset scope before approval. Preparation, receipts, and recovery remain specified well enough to execute without guessing an account or equating cookie import with sign-in." + } + }, + { + "id": "p07", + "category": "pass", + "expected": "pass", + "split": "heldout", + "rationale": "Production plus two worked examples: a direct import for Chrome `Profile 2` on example.com with --verify-auth (stating that the variables were set privately before step 3 started the daemon and the target is open), and the matching three-line report (14 imported, storage not requested, sign-in verified). Placeholder data only, no identities or cookie values. Everything else is production. The examples remove guesswork about the direct-import command shape; 4-5.", + "inputs": { + "bundle": { + "file": "inputs/p07.bundle.txt" + } + }, + "second_review": { + "verdict": "pass", + "confidence": "high", + "reason": "Explicit source and scope choices, pre-start private verification configuration, scoped reset consent, and recovery boundaries form a coherent executable workflow. The worked command and receipt example further show how requested verification becomes a flag and how an identity match, rather than a cookie count, supports the final report." + } + }, + { + "id": "p08", + "category": "pass", + "expected": "pass", + "split": "heldout", + "rationale": "Production reorganized: the two no-import exits (Aside READY, Mode: cdp) collected into a \"Stop conditions\" section up front, decisions renumbered as step 1, and the verification-settings rejection, the BROWSER.md error-table pointer and the platform boundaries merged into one \"Recovery\" step before the report. Every sentence of substance is preserved and step cross-references are updated consistently. 4-5.", + "inputs": { + "bundle": { + "file": "inputs/p08.bundle.txt" + } + }, + "second_review": { + "verdict": "pass", + "confidence": "high", + "reason": "The stop conditions, user decisions, preparation, import, recovery, and receipt report are explicitly separated and correctly cross-referenced. Authorized source selection, private startup configuration, exact reset consent, and platform/privacy boundaries remain intact, so no material ambiguity blocks execution." + } + }, + { + "id": "f01", + "category": "fail", + "expected": "fail", + "split": "dev", + "rationale": "Production with source selection guessed: step 2 now says do not ask for the source, use the default Comet browser (or any installed one) and pick whichever profile has the most cookies; the \"never guess / Comet is not consent\" rule is deleted and direct import is the default path. The BROWSER.md section still says ambiguity requires a choice and that the Comet default is not a recommendation, so the two files conflict on the first decision. Violates the goal's \"without guessing an account\" and leaves contradictory authority; clarity and actionability <= 3.", + "inputs": { + "bundle": { + "file": "inputs/f01.bundle.txt" + } + }, + "second_review": { + "verdict": "fail", + "confidence": "high", + "reason": "The agent is told not to ask for the source browser or profile and instead choose a default browser and the profile with the most cookies. Cookie quantity cannot establish the authorized account, and skipping unreadable profiles conflicts with the reference's requirement to resolve ambiguity rather than guess." + } + }, + { + "id": "f02", + "category": "fail", + "expected": "fail", + "split": "dev", + "rationale": "Production with consent removed from the destructive options: --clear-storage is always passed without asking and described only as wiping \"the browser's stored site data\", --all is added without consent whenever the user is unsure, and BROWSER.md's reset scope and its \"does not clear other origins/IndexedDB/service workers\" limits are deleted. BROWSER.md still says --all cannot accompany --domain or --clear-storage, so the instructed command combination is invalid. Unscoped, unconsented reset plus a contradictory command; clarity and actionability <= 3.", + "inputs": { + "bundle": { + "file": "inputs/f02.bundle.txt" + } + }, + "second_review": { + "verdict": "fail", + "confidence": "high", + "reason": "The entrypoint mandates storage clearing without consent and authorizes --all without asking, while describing the reset only as a broad stored-data wipe. It also mandates --clear-storage even though the reference forbids combining it with --all, so neither authorized scope nor a consistently executable command is established." + } + }, + { + "id": "f03", + "category": "fail", + "expected": "fail", + "split": "dev", + "rationale": "Production with sign-in proof replaced by copied cookies: the verification question and the variable setup step are deleted, the report's sign-in line becomes \"signed in\" when cookies were imported and `$B reload` returns HTTP 200, BROWSER.md now says copied cookies mean signed in and drops the --verify-auth paragraph, and the Opera example comment says \"confirm you are signed in\". Step 4 still says to add --verify-auth \"if the user said yes in step 2\" and to return to step 3 for missing verification settings, both now dangling. Directly contradicts the goal's distinction; completeness and actionability <= 3.", + "inputs": { + "bundle": { + "file": "inputs/f03.bundle.txt" + } + }, + "second_review": { + "verdict": "fail", + "confidence": "high", + "reason": "The report declares the user signed in from copied cookies plus HTTP 200, which cannot distinguish an authenticated account from a public or login page. The verification decision and configuration instructions are also missing despite surviving references to --verify-auth, so the expected identity check is not executable." + } + }, + { + "id": "f04", + "category": "fail", + "expected": "fail", + "split": "dev", + "rationale": "Production with contradictory step order: step 1 now runs the step 4 import immediately and only then returns to step 2, whose first line still says to ask every decision before any $B command; verification variables are set after `$B status` starts the daemon with an explicit ban on restarting it, while BROWSER.md says they must be in the daemon environment before startup; the target is opened after the import, while BROWSER.md and the reset/verification rules need it open first. An agent cannot find a consistent order; clarity <= 2.", + "inputs": { + "bundle": { + "file": "inputs/f04.bundle.txt" + } + }, + "second_review": { + "verdict": "fail", + "confidence": "high", + "reason": "The entrypoint explicitly orders import before the user's decisions, delays target navigation until after import, and forbids restarting while configuring verification after daemon startup. These directives contradict the consent, captured-target, and startup requirements elsewhere in the bundle." + } + }, + { + "id": "f05", + "category": "fail", + "expected": "fail", + "split": "dev", + "rationale": "Stubbed workflow: step 1 (Aside check) is intact, but steps 2-3 and the platform boundaries are TODO/TBD, step 4 is \"Run the import. Details to be added\", step 5 is \"Let the user know how it went\", and the BROWSER.md section is a one-line draft placeholder. No decisions, commands, consent, verification or recovery. Every dimension <= 2. (Same idea as workflow-default f03 on the current template.)", + "inputs": { + "bundle": { + "file": "inputs/f05.bundle.txt" + } + }, + "second_review": { + "verdict": "fail", + "confidence": "high", + "reason": "The source-selection questions, preparation, import details, reporting requirements, and platform boundaries are TODO or TBD placeholders. The referenced cookie section is also unwritten, so the bundle does not teach an executable import workflow." + } + }, + { + "id": "f06", + "category": "fail", + "expected": "fail", + "split": "heldout", + "rationale": "Command-free workflow: the five decisions and the consent language survive in generic form, but every command, flag, environment variable, code block and error table is gone from both files (\"use gstack's cookie import, either the interactive chooser or the direct form\", \"set up the verification settings\", \"see whether the user already has a browser that is signed in\"). The agent knows what to ask but cannot execute any step or recovery; actionability <= 2, completeness <= 3.", + "inputs": { + "bundle": { + "file": "inputs/f06.bundle.txt" + } + }, + "second_review": { + "verdict": "fail", + "confidence": "high", + "reason": "The opening stop condition incorrectly treats any signed-in browser as sufficient, rather than establishing that Aside or the attached target browser has those sessions. The remaining prose omits concrete verification settings, precise origin/tab reset scope, and executable platform recovery decisions, so it is not an adequate operational workflow." + } + }, + { + "id": "f07", + "category": "fail", + "expected": "fail", + "split": "heldout", + "rationale": "Production with domain/target selection and recovery removed together: the domains-and-target question, the `$B goto` step and the --domain flag are gone, so direct import has no scope while the reset and verification rules still refer to \"the target\"; recovery becomes \"If the import fails, run it again\", the platform boundaries section is deleted, and the BROWSER.md section loses the direct-import example, Windows commands, both error tables and the manual sign-in / no-workaround sentences. Undefined scope plus no recovery; completeness and actionability <= 3.", + "inputs": { + "bundle": { + "file": "inputs/f07.bundle.txt" + } + }, + "second_review": { + "verdict": "fail", + "confidence": "high", + "reason": "The questions and direct-import instructions omit domain selection and target navigation even though verification and storage reset depend on a captured target. Recovery is reduced to running a failed import again, leaving both authorized scope and meaningful failure handling undefined." + } + }, + { + "id": "f08", + "category": "fail", + "expected": "fail", + "split": "heldout", + "rationale": "Production with privacy and platform boundaries inverted: the user is told to paste the selector and expected identity into the chat, the agent sets and echoes them, the report adds the identity, profile name and cookie names/values, Keychain prompts are retried until they pass, and Chrome 136+/App-Bound blocks are bypassed by copying the real profile directory. BROWSER.md still forbids pasting cookie values or identity into logs and says there is no real-profile-copy workaround, so the files conflict on privacy and recovery. Out-of-bounds recovery plus contradictions; clarity and actionability <= 3.", + "inputs": { + "bundle": { + "file": "inputs/f08.bundle.txt" + } + }, + "second_review": { + "verdict": "fail", + "confidence": "high", + "reason": "The entrypoint requests and echoes identity details, includes cookie values in the report, repeats Keychain attempts, and recommends copying real profiles to bypass protection. These instructions directly conflict with the reference's privacy and platform boundaries, making compliant execution impossible without rejecting substantial parts of the workflow." + } + }, + { + "id": "a01", + "category": "adjacent", + "expected": "pass", + "author_expected": "pass", + "adjudication": "Kept the author label: the reset is still default-no, needs explicit approval, is Chromium-only, excludes --all and is not rolled back, so every decision gate and command stays executable; the missing origin/tab scope weakens completeness, not the clarity 4 / actionability 4 floor.", + "split": "dev", + "rationale": "One moderate defect: the storage-reset consent no longer says what is reset. SKILL.md question 4 keeps default-no, explicit approval, Chromium-only and the no-rollback caveat but drops the localStorage/sessionStorage scope; BROWSER.md says only that it clears \"the target's stored site data\" and loses the list of what it does not clear. The decision, consent gate, flag and order are unchanged, so an agent can still execute correctly; the gap is informed-consent detail. I expect completeness about 3 with clarity and actionability 4, so pass.", + "inputs": { + "bundle": { + "file": "inputs/a01.bundle.txt" + } + }, + "second_review": { + "verdict": "fail", + "confidence": "high", + "reason": "The reset is described only as clearing the target's stored site data, without identifying exact-origin localStorage, target-tab sessionStorage, or unaffected stores and tabs. Asking for approval under that description does not establish consent for a precisely scoped mutation." + } + }, + { + "id": "a02", + "category": "adjacent", + "expected": "pass", + "split": "dev", + "rationale": "Real mixed revision: current SKILL.md with the real BROWSER.md section from v1.90.0.0 (a84b0b5b6d, lines 569-587), before the Windows Opera wave. The section lacks the receipt-failure-reason and import-error tables, the Opera commands and the App-Bound handoff, so SKILL.md's \"it has the error tables\" pointer dangles and error-specific next steps are missing. The main path, consent, verification-before-startup, the missing-settings rejection and the generic manual sign-in fallback all remain. One missing recovery layer: completeness about 3, clarity and actionability 4, so pass.", + "inputs": { + "bundle": { + "file": "inputs/a02.bundle.txt" + } + }, + "second_review": { + "verdict": "pass", + "confidence": "medium", + "reason": "Source authorization, verification startup, reset scope, and honest receipt reporting remain executable, and unrecoverable imports have a display-qualified manual-sign-in path. The advertised error tables are absent, reducing recovery detail, but the remaining instructions still provide a safe way to complete or report the workflow." + } + }, + { + "id": "a03", + "category": "adjacent", + "expected": "fail", + "split": "dev", + "rationale": "One moderate defect: optional verification has no consistent order. Question 3 lets the user decide after the import, the step 3 instruction to set the variables before the daemon starts (and `$B stop` a running one) is deleted, and step 4 now says to set them after the import and rerun with --verify-auth. BROWSER.md still says the variables must be in the daemon environment before startup and a later CLI call does not reconfigure a running daemon, and step 4 still sends missing-settings rejections back to step 3, which no longer covers them. The verification path requires guessing between conflicting orders; clarity about 3, so fail.", + "inputs": { + "bundle": { + "file": "inputs/a03.bundle.txt" + } + }, + "second_review": { + "verdict": "fail", + "confidence": "high", + "reason": "The workflow starts the daemon and performs the initial import before configuring verification, then tells the user to set shell variables and rerun without restarting. This contradicts the reference's startup-only configuration rule and mutates cookies before the requested verification configuration is ready." + } + }, + { + "id": "a04", + "category": "adjacent", + "expected": "fail", + "split": "dev", + "rationale": "One moderate defect: --profile semantics contradict the reference. SKILL.md now says --profile takes the display name (`Work`, `Personal`), not the directory, in both the decision and the direct-import instruction, while BROWSER.md says --profile takes the directory (`Default`, `Profile 2`), not the display name, and its example and profile_required recovery use directories. The account-selection command has two incompatible forms; a careful grader cites the conflict and gives clarity or actionability 3, so fail.", + "inputs": { + "bundle": { + "file": "inputs/a04.bundle.txt" + } + }, + "second_review": { + "verdict": "fail", + "confidence": "high", + "reason": "The entrypoint repeatedly tells the agent to pass a profile display name, while BROWSER.md explicitly requires the directory and demonstrates Profile 2. This unresolved contradiction affects the source-selection command, so the agent cannot reliably execute direct import as instructed." + } + }, + { + "id": "a05", + "category": "adjacent", + "expected": "fail", + "split": "dev", + "rationale": "Real pre-rewrite revision: setup-browser-cookies/SKILL.md at 6681adf7a8 (lines 153-237) with BROWSER.md at 28f1385eac (569-617), the pair the v1.91.150.0 rewrite replaced. Step 1 says to follow the Aside probe's fallback handling, then a separate SETUP block hard-codes another $B lookup and build; step 2 opens \"the known target\" before step 3 has chosen browser, profile or domains; verification must be configured before startup but SETUP's `$B status` has already started the daemon; there is no report format, and the Opera example comment says `$B reload` confirms the intended account. Several moderate order defects; clarity and actionability about 3, so fail. Provenance: the CHANGELOG records an earlier production sample on this template at 3/4/3; my label rests on the defects above.", + "inputs": { + "bundle": { + "file": "inputs/a05.bundle.txt" + } + }, + "second_review": { + "verdict": "fail", + "confidence": "high", + "reason": "The ordered setup runs $B status before the next section asks for verification settings that must exist before daemon startup. No stop-and-restart path reconciles that ordering, so the otherwise detailed workflow leaves requested verification dependent on repairing the instructions." + } + }, + { + "id": "a06", + "category": "adjacent", + "expected": "pass", + "split": "dev", + "rationale": "One moderate defect: the three-line report template is replaced by one sentence (say what was imported and from where, whether storage was reset, and the sign-in result). The receipt-not-raw-cookies rule, the not checked / not verified / verified definitions and the \"counts and HTTP 200 never prove sign-in\" rule stay, and BROWSER.md still lists the receipt outcomes. The deliverable is looser but fully determined in content; completeness 3-4, clarity and actionability 4, so pass.", + "inputs": { + "bundle": { + "file": "inputs/a06.bundle.txt" + } + }, + "second_review": { + "verdict": "pass", + "confidence": "high", + "reason": "The prose report still requires import provenance, storage-reset status, and distinct verified/not-verified/not-checked results derived from the receipt. Removing a fixed three-line format does not remove the authorized-source, verification, consent, or recovery instructions needed to produce an honest deliverable." + } + }, + { + "id": "a07", + "category": "adjacent", + "expected": "fail", + "split": "heldout", + "rationale": "One moderate defect: no target page is chosen. Question 2 asks only for domains and step 3 says \"open a page for the import\", yet verification reloads \"the target\" and must find the identity element there, storage reset is scoped to the target's origin and tab, and the report and BROWSER.md all depend on that target. The BROWSER.md example (goto the domain root) suggests a default but not one that carries the identity element. Two consent-gated features depend on an undefined input; clarity about 3, so fail.", + "inputs": { + "bundle": { + "file": "inputs/a07.bundle.txt" + } + }, + "second_review": { + "verdict": "fail", + "confidence": "medium", + "reason": "The workflow asks for cookie domains but never obtains or otherwise resolves the target URL, then requires the agent to open an unspecified page. A domain-matching example does not decide the intended scheme, host, port, and verification page, leaving the reset origin and authentication target insufficiently determined." + } + }, + { + "id": "a08", + "category": "adjacent", + "expected": "fail", + "author_expected": "pass", + "adjudication": "Changed to the reviewer label: with the $B stop step gone no restart command remains, BROWSER.md says a later call cannot reconfigure a running daemon, and Step 4's missing-settings recovery returns to Step 3 after the daemon has started, so requested verification cannot be configured (actionability below 4).", + "split": "heldout", + "rationale": "One moderate defect: the stale-daemon recovery is removed from step 3 (\"the daemon reads them only when it starts, so ... run `$B stop` first\"). BROWSER.md still says a later CLI call does not reconfigure an existing daemon and that missing configuration rejects before mutation, and step 4 still returns missing-settings rejections to step 3, so the agent knows the constraint but must infer the restart. Main path, consent and reporting are intact; completeness about 3-4, clarity and actionability 4, so pass.", + "inputs": { + "bundle": { + "file": "inputs/a08.bundle.txt" + } + }, + "second_review": { + "verdict": "fail", + "confidence": "medium", + "reason": "For an already-running daemon, the entrypoint only sets variables in the calling shell and returns to that same preparation sequence when configuration is missing. BROWSER.md warns that this cannot reconfigure the daemon, but the workflow provides no restart step, leaving that verification path incomplete." + } + } + ], + "label_review": "Expected verdicts were preregistered by the author, then labeled blind (shuffled ids, no author labels, a separate agent run on openai/gpt-6.1-sol) before any schema was sent. The two disagreements (a01, a08) were adjudicated by a third reviewer (google/gemini-4-argon); adjudicated items carry the note." +} diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/a01.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/a01.bundle.txt new file mode 100644 index 000000000..7e2dee749 --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/a01.bundle.txt @@ -0,0 +1,119 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-206; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. +2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL. +3. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately. +4. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +5. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Use the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains. + +## 3. Prepare gstack's browser + +1. If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. Never invent the values. +2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +4. Open the target URL from answer 2 and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-617; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears the target's stored site data before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/a02.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/a02.bundle.txt new file mode 100644 index 000000000..695d2d228 --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/a02.bundle.txt @@ -0,0 +1,89 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-206; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. +2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL. +3. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately. +4. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +5. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Use the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains. + +## 3. Prepare gstack's browser + +1. If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. Never invent the values. +2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +4. Open the target URL from answer 2 and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-587; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/a03.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/a03.bundle.txt new file mode 100644 index 000000000..90f0c468d --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/a03.bundle.txt @@ -0,0 +1,120 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-207; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. +2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL. +3. **Sign-in verification (`--verify-auth`)?** Default no; the user may also decide after seeing the import. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately. +4. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +5. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Use the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains. + +## 3. Prepare gstack's browser + +1. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +2. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +3. Open the target URL from answer 2 and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--clear-storage` only if the user said yes in step 2. + +If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`, then rerun the import with `--verify-auth`. Never invent the values. + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-617; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/a04.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/a04.bundle.txt new file mode 100644 index 000000000..cb2182f33 --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/a04.bundle.txt @@ -0,0 +1,119 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-206; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's display name as shown in the browser (`Work`, `Personal`), not its directory. A profile that cannot be read is unknown, not empty. +2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL. +3. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately. +4. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +5. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Use the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile name and the domains. + +## 3. Prepare gstack's browser + +1. If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. Never invent the values. +2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +4. Open the target URL from answer 2 and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile's display name (not its directory); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-617; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/a05.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/a05.bundle.txt new file mode 100644 index 000000000..49c3a900c --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/a05.bundle.txt @@ -0,0 +1,143 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 153-237; entrypoint) --- +# Setup Browser Cookies + +## 1. Choose the browser + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its probe first. On `READY`, stop importing: use Aside's sessions or ask the user to sign in there. Otherwise follow the probe's fallback handling, then continue below. + +## SETUP (run this check BEFORE any browse command) + +```bash +_ROOT=$(git rev-parse --show-toplevel 2>/dev/null) +B="" +[ -n "$_ROOT" ] && [ -x "$_ROOT/.claude/skills/gstack/browse/dist/browse" ] && B="$_ROOT/.claude/skills/gstack/browse/dist/browse" +[ -z "$B" ] && B="$HOME/.claude/skills/gstack/browse/dist/browse" +if [ -x "$B" ] && "${B%/browse/*}/bin/gstack-browser-ensure"; then + echo "READY: $B" +else + echo "NEEDS_SETUP" +fi +``` + +`gstack-browser-ensure` installs Chromium on first use when the install skipped it (lazy browser); it prints `BROWSER_OK` or `BROWSER_UNAVAILABLE ` first. + +If `NEEDS_SETUP`: +1. Tell the user: "gstack browse needs a one-time build (~10 seconds). OK to proceed?" Then STOP and wait. +2. Run: `cd && ./setup` +3. If `bun` is not installed: + ```bash + if ! command -v bun >/dev/null 2>&1; then + BUN_VERSION="1.4.2" + BUN_INSTALL_SHA="bab8acfb046aac8c72407bdcce903957665d655d7acaa3e11c7c4616beae68dd" + tmpfile=$(mktemp "${TMPDIR:-/tmp}/bun-install.XXXXXX") + curl -fsSL "https://bun.sh/install" -o "$tmpfile" + # shasum is macOS/perl; coreutils-only Linux ships sha256sum instead — + # resolve whichever exists so the verify never fails on a missing tool. + if command -v sha256sum >/dev/null 2>&1; then + actual_sha=$(sha256sum < "$tmpfile" | awk '{print $(1)}') + else + actual_sha=$(shasum -a 256 < "$tmpfile" | awk '{print $(1)}') + fi + if [ "$actual_sha" != "$BUN_INSTALL_SHA" ]; then + echo "ERROR: bun install script checksum mismatch" >&2 + echo " expected: $BUN_INSTALL_SHA" >&2 + echo " got: $actual_sha" >&2 + rm "$tmpfile"; exit 1 + fi + bash "$tmpfile" "bun-v$BUN_VERSION" + rm "$tmpfile" + fi + ``` + +```bash +$B status +``` +If status says `Mode: cdp`, stop: the real browser already has sessions. + +## 2. Confirm options before import + +Open the known target and keep its tab unchanged. Both options default off and require explicit request: + +- **`--verify-auth` / picker checkbox:** reloads the target. Have the user privately configure daemon `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` **before startup**. Never invent values or assume CLI env reconfigures an existing daemon. Missing config rejects before mutation. Require a successful same-origin response and exactly one visible element whose whitespace-normalized text exactly matches the expected identity. +- **`--clear-storage`:** for suspected stale storage, obtain explicit approval. Chromium only: clears captured-origin localStorage (shared across same-origin context tabs) and target-tab sessionStorage. Other origins, other tabs' sessionStorage, IndexedDB, and service workers stay intact. It uses an isolated world/native deadline; other engines reject reset, not imports/auth checks. Never auto-approve or claim rollback after partial failure. + +## 3. Select source and scope + +```bash +$B cookie-import-browser +``` + +Ask the user to choose browser, account/profile, and domains, then say when done. Never guess accounts or treat default Comet as consent. Unreadable profiles are unknown, not empty. Rerun for an expired five-minute one-use link. + +Direct import: pass the chosen browser and `--domain` after navigating to a matching target. `--profile` takes a directory, not a display name; omit only for an unambiguous relevant profile, otherwise use the picker. `--all` requires consent for all non-expired profile cookies; it cannot accompany `--domain` or `--clear-storage`. + +Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for examples, profile labels, supported sources and platform setup. + +## 4. Report honestly + +Report receipt/picker counts, partial/zero/error and reset outcomes, not raw `$B cookies`. Imports affect the context, not one tab. **Not checked** means no requested check; **not verified** means it failed; **verified** requires positive target evidence. Zero imports, counts, or HTTP 200 never prove login. + +Never request/publish cookie values, passwords, identity/profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-617; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # confirm the intended account +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/a06.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/a06.bundle.txt new file mode 100644 index 000000000..c1170d10e --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/a06.bundle.txt @@ -0,0 +1,113 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-200; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. +2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL. +3. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately. +4. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +5. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Use the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains. + +## 3. Prepare gstack's browser + +1. If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. Never invent the values. +2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +4. Open the target URL from answer 2 and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`: say what was imported and from where, whether storage was reset, and the sign-in result. **Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-617; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/a07.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/a07.bundle.txt new file mode 100644 index 000000000..7678c1c1d --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/a07.bundle.txt @@ -0,0 +1,119 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-206; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. +2. **Domains to import.** +3. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately. +4. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +5. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Use the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains. + +## 3. Prepare gstack's browser + +1. If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. Never invent the values. +2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +4. Open a page for the import and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-617; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/a08.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/a08.bundle.txt new file mode 100644 index 000000000..470b1a478 --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/a08.bundle.txt @@ -0,0 +1,119 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-206; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. +2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL. +3. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately. +4. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +5. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Use the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains. + +## 3. Prepare gstack's browser + +1. If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. Never invent the values. +2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +4. Open the target URL from answer 2 and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-617; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/f01.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/f01.bundle.txt new file mode 100644 index 000000000..ce2625f26 --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/f01.bundle.txt @@ -0,0 +1,119 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-206; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. + +1. **Source browser and profile.** Do not ask. Use the default browser (Comet); if it is not installed, use whichever supported browser is. For the profile, pick whichever profile has the most cookies for the domains, and skip profiles that cannot be read. +2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL. +3. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately. +4. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +5. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Use direct import with the browser and profile you picked. The picker is only needed if no profile has cookies for the domains. + +## 3. Prepare gstack's browser + +1. If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. Never invent the values. +2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +4. Open the target URL from answer 2 and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the browser name you picked, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-617; section) --- +#### Choosing a source and checking sign-in + +The source browser and account/profile can be chosen automatically. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/f02.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/f02.bundle.txt new file mode 100644 index 000000000..f5d3f31dd --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/f02.bundle.txt @@ -0,0 +1,119 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-206; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. +2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL. +3. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately. +4. **Clear stale storage (`--clear-storage`).** Do not ask; always clear it, so stale sessions cannot interfere. It wipes the browser's stored site data before the cookies are applied. +5. **Every cookie in the profile (`--all`)?** Add it whenever the user is unsure which domains the site needs; there is no need to ask. + +Use the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains. + +## 3. Prepare gstack's browser + +1. If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. Never invent the values. +2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +4. Open the target URL from answer 2 and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); always add `--clear-storage`, and add `--verify-auth` only if the user said yes in step 2. + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-617; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +`--clear-storage` resets the browser's stored data before applying cookies, so stale sessions cannot interfere. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/f03.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/f03.bundle.txt new file mode 100644 index 000000000..8244e0d1f --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/f03.bundle.txt @@ -0,0 +1,115 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-204; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. +2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL. +3. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +4. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Use the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains. + +## 3. Prepare gstack's browser + +1. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +2. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +3. Open the target URL from answer 2 and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +Report **signed in** when the receipt shows cookies imported for the target's domain and `$B reload` on the target returns HTTP 200; otherwise report **signed out**. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-615; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # confirm you are signed in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Once cookies are copied for the domain, the session is signed in; an HTTP 200 on reload confirms it. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/f04.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/f04.bundle.txt new file mode 100644 index 000000000..54fe46ffe --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/f04.bundle.txt @@ -0,0 +1,119 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-206; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`), and run the import in step 4 right away so the browser has cookies; then come back to step 2 for the user's decisions. + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. +2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL. +3. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately. +4. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +5. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Use the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains. + +## 3. Prepare gstack's browser + +1. If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`, after `$B status` below has started the daemon; do not stop or restart a running daemon. Never invent the values. +2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +4. After the import finishes, open the target URL from answer 2: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-617; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/f05.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/f05.bundle.txt new file mode 100644 index 000000000..3b5bca679 --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/f05.bundle.txt @@ -0,0 +1,41 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-173; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +TODO: list the questions to ask the user. + +## 3. Prepare gstack's browser + +TBD. + +## 4. Import + +Run the import. Details to be added; see `BROWSER.md`, **Choosing a source and checking sign-in**, once it is written. + +## 5. Report honestly + +Let the user know how it went. + +## Platform boundaries + +TBD. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-572; section) --- +#### Choosing a source and checking sign-in + +_Draft: notes on source selection, sign-in checks and storage reset to come._ + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/f06.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/f06.bundle.txt new file mode 100644 index 000000000..7b4ba3b32 --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/f06.bundle.txt @@ -0,0 +1,56 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-178; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +First see whether the user already has a browser that is signed in. If so, no import is needed; tell the user and stop. + +## 2. Get every decision from the user first + +Ask the user, before you start: + +1. Which browser and which profile to copy from. Do not guess. +2. Which sites to import, and which page to open. +3. Whether to check sign-in afterwards. Default no. +4. Whether to clear stale storage. Default no; needs explicit approval. It only clears storage for that site. +5. Whether to copy every cookie. Default no; needs explicit consent. + +## 3. Prepare gstack's browser + +Get gstack's browser running. If the user wants sign-in checking, have them set up the verification settings privately first. Open the page from step 2. + +## 4. Import + +Use gstack's cookie import, either the interactive chooser or the direct form with the chosen browser, profile and site. Turn on sign-in checking or storage clearing only if the user agreed. If something goes wrong, see `BROWSER.md`, **Choosing a source and checking sign-in**. + +## 5. Report honestly + +Tell the user how many cookies were imported, whether storage was reset, and whether sign-in was verified, not verified or not checked. Cookie counts never prove sign-in. + +Never publish cookie values, passwords or identity details. + +## Platform boundaries + +Some browsers only work on some platforms. If the import cannot recover the session, the user can sign in manually. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-582; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. Several Chromium-based browsers are supported, some only on one operating system. Profiles are identified by their directory rather than their display name. If more than one profile could apply, ask the user. + +For direct import, first open a page on the site you are importing for. + +If the import fails, the receipt explains why. When cookies cannot be read, sign in manually. + +Copied cookies do not by themselves mean the user is signed in. Sign-in checking compares an element on the page with the identity the user expects; configure it before the browser starts. + +Storage stays intact by default. With explicit approval, storage clearing removes local and session storage for the target site before cookies are applied. It is not undone if it fails partway. + +**Platform limits:** some platforms need extra approval or libraries before cookies can be read, and some newer browser encryption cannot be read at all. There is no workaround that copies the user's real profile. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/f07.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/f07.bundle.txt new file mode 100644 index 000000000..e29983188 --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/f07.bundle.txt @@ -0,0 +1,72 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-194; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. +2. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately. +3. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +4. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Use the picker when answer 1 is still open; it asks for it. Use direct import when the user has named the browser and the profile directory. + +## 3. Prepare gstack's browser + +1. If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. Never invent the values. +2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +## 4. Import + +Picker (the user confirms the browser and profile there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +If the import fails, run it again. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for profile labels and supported sources. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-582; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. Run `$B cookie-import-browser` with no flags to see which browsers were detected. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/f08.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/f08.bundle.txt new file mode 100644 index 000000000..6d997c81c --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/f08.bundle.txt @@ -0,0 +1,119 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-206; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. +2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL. +3. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, ask the user to paste the selector and their expected identity into the chat so you can configure them. +4. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +5. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Use the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains. + +## 3. Prepare gstack's browser + +1. If the user wants verification, set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` yourself from the values the user pasted, and echo them back so the user can check them. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. Never invent the values. +2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +4. Open the target URL from answer 2 and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Add the expected identity, the profile name and the first few imported cookie names and values to the report so the user can confirm the right account was copied. + +## Platform boundaries + +Dia is macOS-only. If Keychain asks for approval, keep rerunning the import until it goes through; database permission errors usually clear on retry. + +If Chrome 136+ default-directory protection blocks the import, close Chrome, copy its profile directory to a temporary folder and import from the copy. On Windows, if App-Bound Encryption blocks decryption, use the same copy. Offer headed manual sign-in only as a last resort. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-617; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/p01.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/p01.bundle.txt new file mode 100644 index 000000000..138683a4b --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/p01.bundle.txt @@ -0,0 +1,119 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-206; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. +2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL. +3. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately. +4. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +5. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Use the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains. + +## 3. Prepare gstack's browser + +1. If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. Never invent the values. +2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +4. Open the target URL from answer 2 and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-617; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/p02.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/p02.bundle.txt new file mode 100644 index 000000000..08274e8ea --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/p02.bundle.txt @@ -0,0 +1,119 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-206; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. +2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL. +3. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately. +4. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +5. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Use the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains. + +## 3. Prepare gstack's browser + +1. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +2. If the user wants verification: the browser daemon reads its settings only when it starts, so if a gstack daemon is already running, run `$B stop` first. Then have the user set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. Never invent the values. +3. Run `$B status`; this starts the daemon with those settings. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +4. Open the target URL from answer 2 and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-617; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/p03.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/p03.bundle.txt new file mode 100644 index 000000000..cabd7ef64 --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/p03.bundle.txt @@ -0,0 +1,119 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-206; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Pick the gstack root first: this checkout, if it has both `BROWSER.md` and `browse/SKILL.md`; otherwise the installed root that has `bin/gstack-skill-start`. Never use a generated host stub. In that root, read the **BROWSER SETUP** section of `browse/SKILL.md` and run its Aside probe. + +- **The probe prints `READY`:** stop here. Aside already holds the user's signed-in sessions, so nothing needs importing. Tell the user to work in Aside, or to sign in there. +- **It prints anything else:** carry on with gstack's own headless browser (`$B`). + +## 2. Ask every question before touching `$B` + +Put all five questions to the user in one go, before you run any `$B` command. Do not guess any answer, and do not read the default Comet browser as consent. + +1. **Which browser and which profile?** `--profile` wants the profile's directory (`Default`, `Profile 2`), not the name shown in the browser. If a profile cannot be read, treat it as unknown, not as empty. +2. **Which domains, and which target page?** The target is one URL on a chosen domain that the session should open. +3. **Verify sign-in (`--verify-auth`)?** The default is no. Verification reloads the target and reports **verified** only when the response is successful and same-origin and exactly one visible element, matched by a CSS selector, has whitespace-normalized text equal to the expected identity. If the user says yes, they provide the selector and the expected identity privately. +4. **Clear stale storage (`--clear-storage`)?** The default is no, and it needs the user's explicit approval. It works on Chromium only: it clears localStorage for the target's origin (which all of that origin's tabs share) and sessionStorage for the target tab, and nothing else. Other browser engines refuse the reset but still import. If a reset fails partway, some storage may already be gone, and it is not restored. +5. **Import every cookie in the profile (`--all`)?** The default is no, and it needs explicit consent. It cannot be used together with a domain or with clearing storage. + +If the browser, the profile directory or the domains (questions 1-2) are still open, use the picker, which asks for them. If the user has named all three, use direct import. + +## 3. Get gstack's browser ready + +1. If the user wants verification, ask them to set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. Never make up these values. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. +2. Find `$B` using the **Browser fallback** section of `browse/SKILL.md`, and build it if the check prints `NEEDS_SETUP`. +3. Run `$B status`. `Mode: cdp` means `$B` is attached to the user's real browser and already has their sessions: tell the user no import is needed and stop. +4. Open the target URL from question 2, and do not use that tab for anything else until the import is done: + +```bash +$B goto "" +``` + +## 4. Import + +With the picker, run the command below. The user confirms the browser, profile and domains in the picker, ticks verification if they asked for it, and tells you when they are done. The picker link works once and for five minutes; if it expires, run the command again. + +```bash +$B cookie-import-browser +``` + +With direct import, run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run, and `--profile` set to the profile directory (not the display name). Add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +If the import is rejected because verification settings are missing, nothing has changed yet: tell the user which variable is missing and go back to step 3. For any other error or receipt failure reason, read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**; it holds the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report what actually happened + +Base the report on the import receipt, not on raw `$B cookies`, and give exactly three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means nobody asked for verification. **Not verified** means verification ran and failed. **Verified** requires the identity match. A cookie count, zero imports or an HTTP 200 never proves sign-in. Imported cookies apply to the whole browser context, not to a single tab. + +Never ask for or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia runs only on macOS. Approving the Keychain prompt is the user's decision. Database retries are limited; if permission is denied, the user has to act, so do not keep prompting. + +On Windows, DPAPI-compatible cookies work, but not all App-Bound Encryption cookies do; native extraction stays off until it is qualified. Closing Chrome does not get around Chrome 136+ default-directory protection, numbered profiles included. There is no TCP fallback and no copying of real profiles. Offer manual sign-in in the headed browser only when a display is available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-617; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/p04.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/p04.bundle.txt new file mode 100644 index 000000000..32c08bfdf --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/p04.bundle.txt @@ -0,0 +1,123 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-228; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. +2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL. +3. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately. +4. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +5. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Use the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains. + +## 3. Prepare gstack's browser + +1. If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. Never invent the values. +2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +4. Open the target URL from answer 2 and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +### Errors and recovery + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Run `$B cookie-import-browser` with no flags to see which browsers were detected. + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-599; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/p05.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/p05.bundle.txt new file mode 100644 index 000000000..db090e588 --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/p05.bundle.txt @@ -0,0 +1,117 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-206; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. +2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL. +3. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately. +4. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +5. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Use the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains. + +## 3. Prepare gstack's browser + +1. If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. Never invent the values. +2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +4. Open the target URL from answer 2 and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-615; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/p06.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/p06.bundle.txt new file mode 100644 index 000000000..1eba752c1 --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/p06.bundle.txt @@ -0,0 +1,125 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-212; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask all of these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +| # | Decision | Default | What the user gives | Flag | +|---|---|---|---|---| +| 1 | Source browser and profile | none: ask | The browser name and the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. | browser argument, `--profile` | +| 2 | Domains and target | none: ask | The domains to import, and one target URL on a chosen domain that the session should open | `--domain`, one per run | +| 3 | Sign-in verification | no | Yes or no. If yes, the user supplies a CSS selector and the expected identity privately. | `--verify-auth` | +| 4 | Clear stale storage | no; explicit approval required | Yes or no, after you explain the scope below | `--clear-storage` | +| 5 | Every cookie in the profile | no; explicit consent required | Yes or no. Cannot be combined with a domain or with clearing storage. | `--all` | + +Verification reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by the selector, whose whitespace-normalized text equals the expected identity. + +Storage reset is Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. + +Use the picker when decision 1 or 2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains. + +## 3. Prepare gstack's browser + +1. If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. Never invent the values. +2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +4. Open the target URL from answer 2 and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-617; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/p07.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/p07.bundle.txt new file mode 100644 index 000000000..f3cc8e84f --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/p07.bundle.txt @@ -0,0 +1,133 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-220; entrypoint) --- +# Setup Browser Cookies + +## 1. Check for Aside + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Read that root's `browse/SKILL.md` **BROWSER SETUP** section and run its Aside probe. + +- **Aside probe prints `READY`:** no import is needed, because Aside already has the user's signed-in sessions. Tell the user to use Aside, or to sign in there, and end this skill. +- **Any other result:** continue with gstack's own headless browser (`$B`). + +## 2. Get every decision from the user first + +Ask these together, before running any `$B` command. Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. +2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL. +3. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately. +4. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +5. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Use the picker when any of answers 1-2 is still open; it asks for them. Use direct import when the user has named the browser, the profile directory and the domains. + +## 3. Prepare gstack's browser + +1. If the user wants verification, have them set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. Never invent the values. +2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +3. Run `$B status`. If it reports `Mode: cdp`, `$B` is attached to the user's real browser and already has its sessions: tell the user no import is needed and end this skill. +4. Open the target URL from answer 2 and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 4. Import + +Picker (the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done; the link works once, for five minutes, so rerun the command if it expires): + +```bash +$B cookie-import-browser +``` + +Direct import: run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name); add `--verify-auth` or `--clear-storage` only if the user said yes in step 2. + +Example: the user chose Chrome, profile directory `Profile 2`, domain `example.com`, target `https://example.com/account`, verification yes, storage reset no, every cookie no. They set both verification variables privately before step 3 started the daemon, and step 3 opened the target. Then run: + +```bash +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" --verify-auth +``` + +If the import rejects because verification settings are missing, nothing changed: tell the user which variable is missing and return to step 3. Read that same root's `BROWSER.md`, **Choosing a source and checking sign-in**, for other errors and receipt failure reasons; it has the error tables, examples, profile labels, supported sources and platform setup. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +Example, for the import above when the receipt reports 14 cookies imported and the identity matched: + +```text +Imported: 14 (imported) from Chrome for example.com +Storage reset: not requested +Sign-in: verified +``` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +## Platform boundaries + +Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. + +Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-617; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/inputs/p08.bundle.txt b/test/fixtures/judge-calibration/cookie-workflow/inputs/p08.bundle.txt new file mode 100644 index 000000000..1f81b1bf8 --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/inputs/p08.bundle.txt @@ -0,0 +1,119 @@ +SKILL.md is the entry point. BROWSER.md supplies the exact referenced cookie section, not an additional execution step. + +--- BEGIN FILE "setup-browser-cookies/SKILL.md" (lines 146-206; entrypoint) --- +# Setup Browser Cookies + +Use this checkout as the gstack root if it contains `BROWSER.md` and `browse/SKILL.md`; otherwise use the installed root containing `bin/gstack-skill-start`, never a generated host stub. Every file named below is read from that root. + +## Stop conditions: no import needed + +End this skill, and tell the user no import is needed, when either check says so: + +- **Aside is available.** Before anything else, read `browse/SKILL.md` **BROWSER SETUP** and run its Aside probe. If it prints `READY`, Aside already has the user's signed-in sessions: tell the user to use Aside, or to sign in there. On any other result, continue below with gstack's own headless browser (`$B`). +- **`$B` is attached to the real browser.** If `$B status` in step 2 reports `Mode: cdp`, `$B` already has the user's sessions. + +## 1. Decisions: ask them all together, before any `$B` command + +Never guess an answer or treat the default Comet browser as consent. + +1. **Source browser and profile.** `--profile` takes the profile's directory (`Default`, `Profile 2`), not its display name. A profile that cannot be read is unknown, not empty. +2. **Domains to import,** and the **target**: one page on a chosen domain that the session should open, by URL. +3. **Sign-in verification (`--verify-auth`)?** Default no. It reloads the target and reports **verified** only after a successful same-origin response with exactly one visible element, found by a CSS selector, whose whitespace-normalized text equals the expected identity. If yes, the user supplies the selector and the expected identity privately. +4. **Clear stale storage (`--clear-storage`)?** Default no; needs explicit approval. Chromium only. It clears localStorage for the target's origin (shared by that origin's tabs) and sessionStorage for the target tab, and nothing else; other browser engines reject the reset but still import. A failed reset may already have cleared some storage and is not rolled back. +5. **Every cookie in the profile (`--all`)?** Default no; needs explicit consent. It cannot be combined with a domain or with clearing storage. + +Then choose the import path: the **picker** when any of answers 1-2 is still open (it asks for them), **direct import** when the user has named the browser, the profile directory and the domains. + +## 2. Prepare gstack's browser + +1. Verification requested: have the user set `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the shell that runs `$B`. The browser daemon reads them only when it starts, so if a gstack daemon is already running, run `$B stop` first. Never invent the values. +2. Follow `browse/SKILL.md`'s **Browser fallback** section to find `$B`, building it if the check prints `NEEDS_SETUP`. +3. Run `$B status` and apply the `Mode: cdp` stop condition above. +4. Open the target URL from decision 2 and leave that tab alone until the import finishes: + +```bash +$B goto "" +``` + +## 3. Import + +**Picker.** Run the command; the user confirms the browser, profile and domains there, ticks verification if they asked for it, then tells you when done. The link works once, for five minutes, so rerun the command if it expires. + +```bash +$B cookie-import-browser +``` + +**Direct import.** Run `$B cookie-import-browser` with the chosen browser name, one `--domain` per run and `--profile` set to the profile directory (not its display name). Add `--verify-auth` or `--clear-storage` only if the user said yes in step 1. + +## 4. Recovery + +- **Verification settings missing:** the import rejects before changing anything. Tell the user which variable is missing and return to step 2. +- **Any other import error or receipt failure reason:** read `BROWSER.md`, **Choosing a source and checking sign-in**; it has the error tables, examples, profile labels, supported sources and platform setup. +- **Platform boundaries:** Dia is macOS-only. Keychain approval is the user's choice. Database retries are bounded; permission denial needs user action, not repeated prompts. Windows supports DPAPI-compatible cookies, not all App-Bound Encryption; native extraction stays disabled pending qualification. Closing Chrome cannot bypass Chrome 136+ default-directory protection, including numbered profiles. No TCP fallback or real-profile copies. Offer headed manual sign-in only with a display available. + +## 5. Report honestly + +Report from the import receipt, not from raw `$B cookies`, as three lines: + +- `Imported: (, as the receipt says) from for ` +- `Storage reset: ` +- `Sign-in: ` + +**Not checked** means verification was not requested; **not verified** means it ran and failed; **verified** needs the identity match. Cookie counts, zero imports and HTTP 200 never prove sign-in. Imported cookies apply to the whole browser context, not one tab. + +Never request or publish cookie values, passwords, identity or profile text, session details, or raw errors in public logs. + +--- END FILE "setup-browser-cookies/SKILL.md" --- + +--- BEGIN FILE "BROWSER.md" (lines 569-617; section) --- +#### Choosing a source and checking sign-in + +Select the source browser and account/profile explicitly. The picker recognizes Chrome, Chromium, Brave, Edge, Windows-only Opera and Opera GX, and macOS-only Comet, Arc, and Dia. It shows current profile names from `Local State`, falling back to Preferences and then the directory name, with directory labels to distinguish duplicate names. `--profile` takes that directory (`Default`, `Profile 2`), not its display name. Without it, only a sole relevant profile is selected; ambiguity or unreadable profiles require a choice. The omitted-browser default remains `comet` for CLI compatibility, not as a recommendation. The picker opening link is one-use and expires after five minutes. + +For direct import, first navigate to a page matching `--domain`. Example after choosing Chrome's `Profile 2`: + +```bash +$B goto https://example.com +$B cookie-import-browser chrome --domain example.com --profile "Profile 2" +``` + +**Windows: Opera and Opera GX.** On Windows, Chrome, Edge and Brave increasingly store App-Bound Encryption cookies that gstack cannot decrypt; Opera and Opera GX still use DPAPI-protected cookies that it can. In Git Bash, with `$B` set as in the quick start above (the Windows build is `browse/dist/browse.exe`): + +```bash +SITE=app.example.com +DOMAIN=example.com +$B goto "https://$SITE" +$B cookie-import-browser opera-gx --domain "$DOMAIN" --profile Default # or: opera +$B reload # view the page; only --verify-auth proves sign-in +``` + +Run `$B cookie-import-browser` with no flags to see which browsers were detected. `--profile` can be omitted when only one profile has cookies for the domain. If the receipt reports App-Bound Encryption, run `$B handoff`, sign in to the intended account in the window that opens, then `$B resume` (needs a display). + +**Receipt failure reasons** (printed after the message as `Failure reasons: key=count`): + +| Key | Meaning | Next step | +|---|---|---| +| `unsupported_encryption` | App-Bound Encryption (v20) cookies gstack cannot decrypt | `$B handoff`, sign in, `$B resume` | +| `decryption_failed` | The cookie could not be decrypted with the browser's key | Close the source browser and retry; otherwise sign in manually | +| `native_unrecovered` | Windows native extraction ran but could not recover these cookies | Sign in manually with `$B handoff` | + +**Import errors:** + +| Code | Meaning | Next step | +|---|---|---| +| `not_installed` | No supported cookie database for that browser or profile; the message lists every path checked and, off-platform, which OS supports the browser | Pick a browser listed as available on this OS, or pass an existing `--profile` | +| `profile_required` | Several profiles qualify, none has cookies for the domain, or a profile could not be read | Retry with `--profile ""` as the message suggests, or run `$B cookie-import-browser ` to use the picker | +| `native_unsupported_browser` | Internal guard; not expected in normal use | Sign in manually with `$B handoff` | + +The picker shows receipt messages verbatim; detailed `not_installed` and `profile_required` explanations appear in CLI output. + +`--all` explicitly selects every non-expired cookie in the chosen source profile; it cannot accompany `--domain` or `--clear-storage`. Cookies are applied to the captured browser context, not isolated to a tab. The receipt distinguishes imported, partial, empty, and failed results, plus separate storage-reset and authentication outcomes. Cookies copied with authentication `not_requested` means **not checked**, not logged in. Zero imports, cookie counts, and HTTP 200 alone never prove sign-in. + +`--verify-auth` (also an explicit picker checkbox) reloads the captured target. Configure `GSTACK_COOKIE_AUTH_SELECTOR` and `GSTACK_COOKIE_AUTH_EXPECTED_IDENTITY` privately in the **daemon environment before startup**; setting them only on a later CLI call does not reconfigure an existing daemon. Missing configuration rejects before mutation. Verification requires a successful same-origin response and exactly one visible element whose whitespace-normalized text equals the expected identity. A wrong account, login redirect, missing assertion, or changed target is not verified. Do not paste cookie values, passwords, profile/account labels, or expected identity into public logs; report only sanitized outcomes. + +Storage stays intact by default. With explicit approval on a Chromium target, `--clear-storage` clears localStorage for the captured origin (exact scheme, host, and port, shared across that origin's tabs in the context) and sessionStorage for the target tab before applying cookies. Reset runs in an isolated world with a native monotonic deadline, so the site's scripts cannot forge its timeout clock. Other target engines reject reset; ordinary imports and authentication checks remain available. It does not clear other origins, other tabs' sessionStorage, IndexedDB, or service workers. Keep the target open and unchanged. A failed reset may have cleared some storage; a later cookie-application failure does not undo it. + +**Platform limits:** macOS imports may request Keychain approval; Linux `v11` cookies may require libsecret, while `v10` uses Chromium's fallback key. The Windows Node server needs Node.js 22.13 or newer with built-in SQLite enabled for cookie database reads. DPAPI-compatible cookies remain supported, but native App-Bound Encryption extraction is disabled until the browser/runtime passes qualification. Opera and Opera GX are Windows-only and read from `%APPDATA%\Opera Software\Opera Stable` or `Opera GX Stable`, in `Default` or `Profile N` directories; legacy root-level layouts, Opera side profiles and portable or relocated installs are not detected. Opera has no native extraction, so its App-Bound cookies (if any) need manual sign-in. Chrome 136+ blocks remote debugging of its default user-data directory, including numbered profiles, over both pipe and TCP; closing Chrome does not remove that protection. There is no TCP fallback or real-profile-copy workaround. If import cannot recover the session, sign in manually in gstack's headed browser when a display is available. + + +--- END FILE "BROWSER.md" --- \ No newline at end of file diff --git a/test/fixtures/judge-calibration/cookie-workflow/runs/comparison-1.result.json b/test/fixtures/judge-calibration/cookie-workflow/runs/comparison-1.result.json new file mode 100644 index 000000000..bb1e37735 --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/runs/comparison-1.result.json @@ -0,0 +1,109 @@ +{ + "config": "cookie-workflow", + "model": "claude-sonnet-5-5", + "request_hash": { + "old": "364a2ea4aca3403216ff26da8c1670fda7c29eaa6cd4031ecad9f42d92cf714e", + "new": "a2fc989f0b60008b7ad9f3cb6a529c94391e52f1db46860ecd91be14c49b6113" + }, + "complete": true, + "stopped": null, + "calls": 216, + "cost_usd": 2.8132, + "errors": { + "old": { + "refusal": 0, + "truncation": 0, + "parse": 2, + "schema": 0, + "transport": 0, + "samples": 144, + "rate": 0.013888888888888888 + }, + "new": { + "refusal": 0, + "truncation": 0, + "parse": 0, + "schema": 0, + "transport": 0, + "samples": 72, + "rate": 0 + } + }, + "latency_p95_ms": { + "old": 13617, + "new": 9610 + }, + "all": { + "items": 24, + "old_flips": { + "flips": 1, + "compared": 22 + }, + "new_flips": { + "flips": 1, + "compared": 23 + }, + "agreement": { + "old-a": 22, + "old-b": 22, + "new": 23 + }, + "false_passes": { + "old-a": [ + "a08" + ], + "old-b": [ + "a07" + ], + "new": [ + "a07" + ] + }, + "new_false_passes": [ + "a07" + ] + }, + "heldout": { + "items": 8, + "old_flips": { + "flips": 1, + "compared": 7 + }, + "new_flips": { + "flips": 1, + "compared": 7 + }, + "agreement": { + "old-a": 6, + "old-b": 7, + "new": 7 + }, + "false_passes": { + "old-a": [ + "a08" + ], + "old-b": [ + "a07" + ], + "new": [ + "a07" + ] + }, + "new_false_passes": [ + "a07" + ] + }, + "landing": { + "lands": false, + "reasons": [ + "new false passes: a07" + ], + "strict_no_flip_increase": true + }, + "prose_step_triggered": { + "triggered": true, + "reasons": [ + "a landing condition failed" + ] + } +} diff --git a/test/fixtures/judge-calibration/cookie-workflow/runs/comparison-1.samples.jsonl b/test/fixtures/judge-calibration/cookie-workflow/runs/comparison-1.samples.jsonl new file mode 100644 index 000000000..db2107bf8 --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/runs/comparison-1.samples.jsonl @@ -0,0 +1,216 @@ +{"config":"cookie-workflow","item":"f05","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"Only Section 1 (the Aside probe) is concrete. It says which root to use, what to read, and what to do when the probe prints READY versus any other result. Every other part is a placeholder: 'TODO: list the questions', 'TBD', 'Details to be added', and 'Let the user know how it went'. BROWSER.md's referenced section is only a draft note saying content is to come. Nothing covers how to select a source browser, profile, or domain without guessing. Nothing covers optional authentication verification before mutation, explicit consent for scoped storage reset, telling copied cookies apart from positive sign-in evidence, or platform and privacy boundaries. The outputs are not defined, and the agent would have to invent the whole workflow. Clarity is slightly above the floor only because the existing Aside step is unambiguous."},"latency_ms":2854,"stop_reason":"end_turn","usage":{"input_tokens":1007,"output_tokens":300},"cost_usd":0.005014} +{"config":"cookie-workflow","item":"f01","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The workflow has serious internal contradictions and gaps. Step 2 says 'Ask these together' but item 1 says 'Do not ask' and tells the agent to pick the browser and profile itself, defaulting to Comet and choosing 'the profile with the most cookies.' That is the guessing the task says to avoid, and it conflicts with BROWSER.md, where the omitted-browser default is only for CLI compatibility, 'not a recommendation,' and ambiguity requires a choice. The default of Comet is also macOS-only, so the fallback is vague on other platforms. No mechanism is given for counting cookies per profile without a picker or import. The instruction 'Use direct import... The picker is only needed if no profile has cookies' conflicts with the earlier ordering and the picker-first description in step 4. The direct import says 'one --domain per run' with no loop guidance for multiple domains, and `--all` has no command example. The selection of the target page is ambiguous when multiple domains are requested. Verification setup is well specified, including the daemon restart, and the storage-reset consent scope is precise. The report format and the distinction between copied cookies and sign-in evidence are clear. Error tables and platform boundaries are good. However, the contradictory source-selection step, the consent-ordering issues (the Aside probe runs before asking, and the `$B` setup in step 3 depends on answers from step 2), and some missing decision paths lower the scores."},"latency_ms":5335,"stop_reason":"end_turn","usage":{"input_tokens":4509,"output_tokens":501},"cost_usd":0.014028} +{"config":"cookie-workflow","item":"p05","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered. It starts with the Aside probe, then collects all decisions up front, prepares `$B`, imports, and reports in a fixed three-line format. It states consent boundaries and defaults precisely: `--clear-storage` and `--all` need explicit approval, and the receipt separates copied cookies from verified sign-in. BROWSER.md adds error tables, platform limits, and recovery paths (`handoff`/`resume`, manual headed sign-in).\n\nSome gaps and ambiguities remain:\n- Step 2 says to ask everything before any `$B` command, but step 3 requires `$B stop` and the environment variables, which depend on the verification answer. The ordering is mostly workable, but it is slightly tangled.\n- Direct import allows one `--domain` per run, yet the target must be navigated to a page matching each domain. Multi-domain handling is unclear: whether to re-run `goto` for each domain, and what to do when the target domain differs from the import domain.\n- It does not say how `--all` is invoked, or how to proceed when `--all` conflicts with the other flags.\n- The Aside probe is delegated to another file.\n- Recovery steps after `handoff` and `resume` do not say whether to re-verify.\n- The picker flow does not say how `--verify-auth` is passed in direct mode beyond appending the flag.\n- Receipt format mapping for `partial`, `empty`, and `failed` is only loosely defined.\n- The `Mode: cdp` check comes after the user decisions, so it wastes the user's time."},"latency_ms":5576,"stop_reason":"end_turn","usage":{"input_tokens":4244,"output_tokens":554},"cost_usd":0.014028} +{"config":"cookie-workflow","item":"a03","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"The workflow is well-structured and its safety and honesty rules are strong. It asks for all decisions up front, requires explicit consent for storage reset and --all, and clearly separates copied cookies from verified sign-in. The three-line receipt format is clear. Platform limits and error tables are documented.\n\nSeveral ordering and decision gaps hurt execution.\n\n1. The env variable timing contradicts itself. SKILL.md says to set the variables 'in the shell that runs $B' and then rerun the import. BROWSER.md says they must be set in the daemon environment before startup, and that setting them on a later CLI call does not reconfigure an existing daemon. The agent is never told to restart the daemon, and step 3 has already started `$B` and opened the target. The 'return to step 3' recovery after a missing-variable rejection is also vague about restarting.\n\n2. SKILL.md says the user may decide on verification after seeing the import, but verification needs a rerun. That conflicts with the instruction to leave the target tab alone and with the one-time setup.\n\n3. The picker path never says how to pass --clear-storage, --all or --verify-auth. Direct import is described only in prose, with no full example of the flags.\n\n4. Direct import is limited to one `--domain` per run, but nothing says how to handle multiple domains. The `--all` direct path is also not spelled out.\n\n5. The Aside probe, the `$B` fallback and the CDP mode check depend on external files. That is acceptable, but the 'never a generated host stub' root selection is dense.\n\n6. The receipt is not defined, such as how to read it from CLI output. The windows/handoff offers are okay, but there is no decision for what to do when import fails beyond the error tables."},"latency_ms":5864,"stop_reason":"end_turn","usage":{"input_tokens":4502,"output_tokens":611},"cost_usd":0.015114} +{"config":"cookie-workflow","item":"f01","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The workflow conflicts with its own goal of not guessing an account. Step 2 says not to ask about source and profile, then to choose the profile with the most cookies and skip unreadable ones. BROWSER.md says ambiguity or unreadable profiles require a choice, and the picker is wanted only when no profile qualifies. Step 2 also says to ask the user everything together, yet item 1 says \"Do not ask.\" The remaining areas are detailed: verification, scoped storage reset, honest reporting and platform limits. However, the \"Windows supports DPAPI\" guidance is split across files, and no step covers --all execution or how to handle multiple domains."},"latency_ms":3441,"stop_reason":"end_turn","usage":{"input_tokens":4906,"output_tokens":247},"cost_usd":0.012282} +{"config":"cookie-workflow","item":"a01","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well sequenced: Aside check, collect all decisions up front, prepare the browser, import, then report honestly. It handles consent well. Defaults are no, `--all` and `--clear-storage` need explicit approval, and the incompatibility rules are stated. The distinction between copied cookies and verified sign-in is clear and repeated. Platform and privacy boundaries are explicit, and the error tables map to next steps.\n\nThere are gaps and tensions:\n- Step 2 says to ask before any `$B` command, but step 3 requires `$B stop` and a `$B status` check that can end the skill after the user has already answered everything. That ordering is awkward.\n- Direct import with `--all` is not shown. It is unclear whether `--all` is used with the picker or whether it needs a domain or target.\n- Failure paths for `decryption_failed` and a failed reset are only partly connected to the report format.\n- Whether to retry picker or direct import after a rejection is vague.\n- The `$B handoff` and `$B resume` recovery steps appear only in BROWSER.md, so the SKILL.md entry point only partially points to them.\n- When verification is not requested but `--all` is used, which target page to open is unclear.\n- The report says to use the receipt, but it does not say how to obtain the receipt when it is not printed.\n- The multi-domain case is handled loosely: it says one `--domain` per run, but not whether to rerun, or how to aggregate the three report lines across runs.\n\nOverall it is strong but has minor ambiguities."},"latency_ms":8993,"stop_reason":"end_turn","usage":{"input_tokens":4403,"output_tokens":545},"cost_usd":0.014256} +{"config":"cookie-workflow","item":"f03","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The flow is well ordered: Aside probe, up-front consent, browser setup, import, report. The consent rules for `--clear-storage` and `--all`, the profile-directory versus display-name distinction, and the platform limits are specific. Several gaps weaken it. (1) Verification is never configured. Step 2 does not ask whether to verify sign-in, yet step 4 mentions `--verify-auth` and 'ticks verification if they asked'. A rejection for 'missing verification settings' says to tell the user 'which variable' is missing, but no variable or setting is ever named, and neither file explains how to configure one before mutation. (2) The sign-in report contradicts the intended distinction between copied cookies and positive sign-in evidence. Both files treat copied cookies plus an HTTP 200 on reload as 'signed in', which is weak evidence and ignores the receipt's separate authentication outcome. The report offers only signed in or signed out, with no 'unverified' state. (3) `--all` has no handling in the report or in the target and navigation steps. (4) The `$B` Aside probe and the 'before running any `$B` command' rule are slightly ambiguous. (5) Recovery guidance is scattered across the two files and partly generic, for example 'close the source browser and retry' alongside the 'Chrome closing doesn't help' note."},"latency_ms":9009,"stop_reason":"end_turn","usage":{"input_tokens":3986,"output_tokens":1000},"cost_usd":0.017972} +{"config":"cookie-workflow","item":"p04","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"Strong ordering: ask all decisions first, then prepare daemon env, check CDP mode, goto target, import. Consent, scoping, and honest three-line reporting are explicit. Gaps: direct import wording says one --domain per run but not how to handle multiple domains or --all syntax in SKILL.md; picker flow's `--all` option unclear; Aside probe depends on external browse/SKILL.md. Also the BROWSER.md example 'cookie-import-browser chrome' vs picker usage is mostly clear. Missing handling when verification fails beyond reporting, and when reset fails mid-way (only reported). Overall solid and executable."},"latency_ms":3525,"stop_reason":"end_turn","usage":{"input_tokens":4936,"output_tokens":245},"cost_usd":0.012322} +{"config":"cookie-workflow","item":"p04","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is clearly ordered: Aside probe, collect all decisions up front, prepare the browser, import, then report. Consent defaults and the meaning of verified, not verified and not checked are precise. Scoped storage reset and the platform and privacy limits are well covered, and the receipt failure and import error tables give concrete recovery steps. Gaps remain. (1) The `--all` flag has no direct-import syntax and does not say how it interacts with the `--domain` requirement. (2) The direct import says one `--domain` per run, but the multi-domain flow and the aggregate report are left unclear. (3) The picker versus direct choice is ambiguous when the browser and profile are known but the domains are not. (4) Step 3 asks for the target URL before the import, yet `--all` has no domain to match. (5) Dependency on `$B status` and `cdp` mode, plus the Aside probe, rests on external files. (6) Receipt states such as partial and empty have no follow-up actions. (7) The ordering of `$B stop` and the daemon env vars is only implied. (8) The error table omits some cases, such as expired picker links and missing verification variables during a picker flow. Overall it is solid and mostly executable, with a few minor ambiguities."},"latency_ms":4360,"stop_reason":"end_turn","usage":{"input_tokens":4539,"output_tokens":442},"cost_usd":0.013498} +{"config":"cookie-workflow","item":"a06","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: ask for all decisions first, prepare the browser, import, then report from the receipt. It states consent defaults, the profile directory vs display name, and what counts as sign-in evidence. Weaknesses: the SKILL says to ask before running any `$B` command, but the Aside probe may involve browse setup. Direct-import syntax is only spelled out in BROWSER.md. Only one domain per run is allowed, with no multi-domain guidance. Recovery for `--all` and for failed verification is thin, and it is unclear how the `--all` flag combines with direct import. Overall the workflow is strong and executable."},"latency_ms":2866,"stop_reason":"end_turn","usage":{"input_tokens":4852,"output_tokens":239},"cost_usd":0.012094} +{"config":"cookie-workflow","item":"f04","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"Step 1 says to run the import in step 4 right away, then return to step 2 for decisions. That contradicts step 2's rule to ask before any `$B` command, and it would import without a chosen profile or domain. Step 3 says to set verification env vars after the daemon starts, while BROWSER.md says they must be in the daemon environment before startup. Navigation order also conflicts: step 3 opens the target after import, while BROWSER.md requires navigating before import. Failure tables and platform limits are otherwise good."},"latency_ms":3591,"stop_reason":"end_turn","usage":{"input_tokens":4935,"output_tokens":198},"cost_usd":0.01185} +{"config":"cookie-workflow","item":"f03","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The flow is mostly well structured: it probes for Aside first, gathers the browser, profile and domain choices up front, scopes `--clear-storage` precisely, and covers platform limits and error tables. Several gaps and contradictions remain. (1) Optional `--verify-auth` is never asked about in step 2. Step 4 says the user 'ticks verification if they asked', and says to add the flag 'if the user said yes in step 2', but step 2 has no such question. (2) The missing 'verification settings' error refers to an unnamed variable. Neither file says what the settings are, how to configure them, or what to tell the user. The agent would have to guess, and the loop back to step 3 is unclear. (3) Sign-in evidence is conflated with copied cookies. Both files say 'signed in' once cookies are imported for the domain and reload returns HTTP 200. A 200 does not prove authentication, the receipt's separate authentication outcome is ignored, and the report offers no 'unknown' or 'unverified' state. (4) `--all` is described in step 2 but has no direct-import instructions or report format. (5) Reporting for multiple domains, with one run per domain, is underspecified. Failure handling for receipt reasons is split across the two files, and the picker and direct-import decision rules leave some ambiguity."},"latency_ms":10066,"stop_reason":"end_turn","usage":{"input_tokens":3986,"output_tokens":1025},"cost_usd":0.018222} +{"config":"cookie-workflow","item":"p02","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered and precise. It gathers all decisions before any `$B` command, defines the verify-auth and clear-storage semantics exactly, and reports the three outcomes from the receipt. It also separates copied cookies from sign-in evidence and keeps the platform and privacy boundaries explicit. BROWSER.md adds error and failure-reason tables plus a direct-import example. Weaknesses: the SKILL.md ordering is slightly odd, because the Aside probe, which depends on `browse/SKILL.md`, runs before `$B` setup. The `--all` flow is only partly specified, since SKILL.md says nothing about it in the direct-import command. The handling of `--clear-storage` with the picker is unclear. The picker flow also does not say how to detect completion or read the receipt. The one-domain-per-run rule is stated without saying how to handle multiple domains. Rejection handling for a failed or unsupported reset is thin. The unclear ordering of `$B goto` against the picker could also cause hesitation. Overall the workflow is strong but has small gaps."},"latency_ms":4693,"stop_reason":"end_turn","usage":{"input_tokens":4538,"output_tokens":391},"cost_usd":0.012986} +{"config":"cookie-workflow","item":"p01","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: Aside probe, upfront consent, daemon preparation, import, and a three-line honest report. It separates copied cookies from sign-in evidence clearly, and the platform and privacy boundaries are explicit. Error and receipt tables cover recovery paths. Gaps remain. (1) The picker is used for unresolved answers, but the flow for `--all` is unclear: the picker is not said to support it, and direct import with `--all` has no domain or profile guidance. (2) Direct import allows one `--domain` per run, with no loop guidance for multiple domains and no specification of the target-URL-per-domain relationship. (3) `--verify-auth` and `--clear-storage` have only vague handling when they fail. (4) The step 4 \"return to step 3\" instruction is fine, but the ordering is slightly awkward: the daemon is prepared with env vars before `$B` is found. (5) The `Imported` line says \"as the receipt says\" without a mapping for the count. (6) `$B stop` is mentioned only in step 3. (7) The Windows example uses `reload` and omits the consent steps. (8) The BROWSER.md example does not show `--verify-auth` or `--clear-storage` usage. Overall it is strong but has some ambiguity."},"latency_ms":5456,"stop_reason":"end_turn","usage":{"input_tokens":4521,"output_tokens":456},"cost_usd":0.013602} +{"config":"cookie-workflow","item":"f05","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"Only step 1 (the Aside probe and its branching) is concrete. The rest are placeholders: Section 2 says \"TODO: list the questions to ask the user\" and Section 4 says \"Run the import. Details to be added\". BROWSER.md's referenced section is a draft stub with no content on source/profile/domain selection, auth verification, storage reset consent, or platform and privacy boundaries. An agent has no way to import cookies safely or report honestly."},"latency_ms":2308,"stop_reason":"end_turn","usage":{"input_tokens":1404,"output_tokens":183},"cost_usd":0.004638} +{"config":"cookie-workflow","item":"p03","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow orders steps well: probe Aside, ask all five questions up front, set up the browser, import, then report. Consent defaults and the three-line receipt format are explicit. Gaps: picker vs direct import is only partly defined (what if only the domain is missing?). It also doesn't say what to do when the user declines verification yet hits an error, or when a multi-domain direct import needs repeated runs. The `--all` flow is not shown in the commands, and the step 2 vs step 3 dependency on the verification env vars could be clearer. Recovery paths via BROWSER.md are solid."},"latency_ms":2991,"stop_reason":"end_turn","usage":{"input_tokens":5030,"output_tokens":223},"cost_usd":0.01229} +{"config":"cookie-workflow","item":"a06","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow orders steps clearly: ask the user first, prepare the browser, import, then report. Consent is scoped precisely, and the verified / not verified / not checked distinction is explicit. Gaps: the Aside probe command is deferred to another file. The step 4 reference to the 'same root' BROWSER.md is slightly indirect. The skill does not say what to do if the user declines every option, or when --all is chosen without a domain. The picker flow versus the direct-import 'target' requirement is mildly confusing. Error recovery is covered by tables, but a failed --clear-storage retry path is thin."},"latency_ms":3053,"stop_reason":"end_turn","usage":{"input_tokens":4852,"output_tokens":228},"cost_usd":0.011984} +{"config":"cookie-workflow","item":"p05","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: Aside probe, upfront consent questions, daemon preparation, import via picker or direct, and a three-line honest report. It clearly separates copied cookies from sign-in evidence and gives error tables plus platform limits. Weaknesses: (1) The `--all` and `--clear-storage` consent flags are described, but the direct-import step does not mention `--all`. (2) The step-3 order is awkward: the Aside probe and `$B status` come before environment setup, and the daemon-restart condition is implied rather than stated. (3) The `$B goto` before the picker is slightly confusing. (4) Direct import allows one domain per run, but multi-domain handling and its aggregate receipt are not explained. (5) The 'Imported' line has no format for multiple runs. (6) The exact `--verify-auth` syntax in direct mode is only implied. (7) Handling of Aside failure versus other probe results is vague. (8) The reference for what to do on a failed receipt is split across the two files. Overall it is strong but has some gaps."},"latency_ms":10866,"stop_reason":"end_turn","usage":{"input_tokens":4244,"output_tokens":380},"cost_usd":0.012288} +{"config":"cookie-workflow","item":"a01","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: consent is gathered first, verification is configured before any mutation, the reset needs explicit approval, and the report distinguishes copied cookies from verified sign-in. Error tables and platform limits are given. Gaps remain. (1) Step 1 depends on an external Aside probe. (2) The target URL is set in answer 2, while the Aside check and the `$B status` check come before navigation, so the order is slightly tangled. (3) Direct import needs the picker's browser, profile and domain, but the instructions don't say how to combine multiple domains, since there is one `--domain` per run, or what to do when the target's domain differs. (4) The `--all` and `--verify-auth` interplay isn't spelled out, and it's unclear whether verification is supported with the picker and `--all`. (5) A mismatch between the receipt format and the three report lines is not handled. (6) The user is asked to set env vars privately, but the agent cannot confirm they were set. (7) Handling of `partial` imports and of `failed` resets is under-specified. Overall it is mostly executable with a few ambiguities."},"latency_ms":11045,"stop_reason":"end_turn","usage":{"input_tokens":4403,"output_tokens":397},"cost_usd":0.012776} +{"config":"cookie-workflow","item":"f04","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The workflow contradicts itself on order. Step 1 says to run the import in step 4 right away, but step 2 forbids any `$B` command before the user's decisions are collected. Step 3 says to open the target after the import, but BROWSER.md says direct import first needs a page matching `--domain`. Verification env vars are set after the daemon starts in SKILL.md, but BROWSER.md says before daemon startup. The error, receipt, and platform coverage is otherwise strong, and the report format is clear."},"latency_ms":9493,"stop_reason":"end_turn","usage":{"input_tokens":4935,"output_tokens":195},"cost_usd":0.01182} +{"config":"cookie-workflow","item":"f01","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The workflow contradicts itself on source selection. SKILL.md says \"Do not ask\" and to pick the profile with the most cookies, while BROWSER.md says ambiguity requires a choice and the Comet default is not a recommendation. The skill also says \"Ask these together\" yet tells the agent not to ask item 1, which confuses the goal of not guessing an account. Direct import also conflicts with the picker: the skill says to use direct import and that the picker is only for when no profile has cookies. Meanwhile, `profile_required` and the BROWSER.md ambiguity rules call for user choice. Consent and receipt handling are well covered, but the source-selection contradiction undermines safe execution."},"latency_ms":2996,"stop_reason":"end_turn","usage":{"input_tokens":4906,"output_tokens":261},"cost_usd":0.012422} +{"config":"cookie-workflow","item":"f03","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"Strengths: the ask-first gate is clear. Browser, profile directory, domains and target URL are collected before any `$B` command. Consent for `--clear-storage` and `--all` is explicit, with defaults of no. The scope of the storage reset is stated precisely, as are the `--all` conflicts, the platform limits, and the error and failure-reason tables. Weaknesses: (1) Optional verification is under-specified. Step 4 mentions `--verify-auth` and a 'missing variable' rejection, but step 2 never asks about verification. Neither file names the variables or says how to set them, and 'return to step 3' does not say what to fix. (2) The sign-in evidence is weak and inconsistent. Both files treat copied cookies plus an HTTP 200 on reload as 'signed in', and a login page also returns 200. The receipt is said to carry authentication outcomes, yet the report format only offers signed in or signed out, with no 'unverified' state and no use of the verify-auth result. This undercuts the goal of separating copied cookies from positive sign-in evidence. (3) Gaps in the direct-import flow: multiple domains need one run each, but only one target page is opened. `--all` is absent from the step 4 command guidance. The wait-for-the-picker handoff and the exact receipt fields are not defined. (4) Recovery paths are split across the two files, so the agent has to stitch them together."},"latency_ms":10195,"stop_reason":"end_turn","usage":{"input_tokens":3986,"output_tokens":1058},"cost_usd":0.018552} +{"config":"cookie-workflow","item":"a01","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: Aside check, collecting all decisions up front, daemon setup, import, then a three-line honest report. Sign-in evidence is clearly separated from copied cookies, and the consent defaults are explicit. Gaps: the `--all` direct-import syntax is not shown. Handling for a Chromium-only reset on a non-Chromium target is only partly specified. SKILL.md's step 2 asks for the verification selector and identity \"privately\", but step 3 has the user set them in the shell, so the agent never sees them. One decision is missing: whether to keep the target tab open after a failure. The error and recovery tables cover the common failures."},"latency_ms":3896,"stop_reason":"end_turn","usage":{"input_tokens":4800,"output_tokens":247},"cost_usd":0.01207} +{"config":"cookie-workflow","item":"p03","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered. It asks all five questions up front, forbids guessing, and chooses between picker and direct import by what the user has already named. It gives precise definitions of verification, storage reset, and --all. It requires explicit consent for destructive options and separates copied cookies from sign-in evidence. The three-line report format is exact, and the error tables and platform boundaries are detailed. Weaknesses: (1) The ordering is slightly awkward: question 2 asks for the target page, but Step 3 says to open it only after the verification setup. The ordering of $B stop versus $B status is also a little unclear. (2) The --all consent is asked in step 2, but step 4's direct import covers only one domain per run and never says how to run --all. Multi-domain imports are not described either, and the picker's handling of --all is unspecified. (3) The 'Aside' probe depends on an external section that is not included, and the Browser fallback section is also not shown. (4) Some recovery paths are thin. Examples are expired picker links, partial imports, and what to do when the user declines manual sign-in with no display. (5) Cases where the user answers 'unknown' for the profile are covered only implicitly. (6) The CLI flags for picker versus direct import are also left unspecified. Overall it is solid and mostly executable, with small gaps."},"latency_ms":11809,"stop_reason":"end_turn","usage":{"input_tokens":4633,"output_tokens":463},"cost_usd":0.013896} +{"config":"cookie-workflow","item":"f04","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The content is detailed on flags, receipt reporting, error tables, privacy and platform limits, but the ordering contradicts itself in ways that affect safety and correctness. (1) Step 1 says to run the import in step 4 \"right away\" and only then return to step 2 for decisions. Step 2 says to ask the user before any `$B` command and never guess. This would run an import with no consent, source, or profile. (2) Step 3.1 says to set the auth env vars after `$B status` starts the daemon. BROWSER.md says they must be in the daemon environment before startup, and that setting them later does not reconfigure a running daemon. Step 3.1 also refers to `$B` before step 3.2 explains how to find it. (3) Step 3.4 says to open the target after the import. BROWSER.md says direct import first needs a `goto` to a page matching the domain. The picker flow's navigation order is also unclear. (4) Step 3's `$B status` check for `Mode: cdp` comes after step 1's instruction to import immediately. (5) The \"one `--domain` per run\" rule is not reconciled with a multi-domain answer. (6) Other gaps: the `--all` direct syntax is not shown, and the missing-verification recovery is only partly specified given the daemon-restart restriction. The reporting rules and boundaries are clear, but the contradictory ordering makes the workflow hard to execute reliably."},"latency_ms":13966,"stop_reason":"end_turn","usage":{"input_tokens":4538,"output_tokens":811},"cost_usd":0.017186} +{"config":"cookie-workflow","item":"p05","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: collect all decisions up front, probe Aside, prepare $B, import, then report three lines. It also separates not checked, not verified and verified. One contradiction: the Aside probe and the daemon-start steps come before the user questions, although the questions are meant to precede any $B command. Step 3 also says to \"Follow browse/SKILL.md\", which lives outside the bundle. The direct-import flags for --all are not shown. Failure handling for partial or failed receipts is reasonably covered by the BROWSER.md tables."},"latency_ms":2988,"stop_reason":"end_turn","usage":{"input_tokens":4641,"output_tokens":211},"cost_usd":0.011392} +{"config":"cookie-workflow","item":"p02","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered and specific. It asks for all decisions up front, treats an unreadable profile as unknown rather than empty, and keeps the default Comet browser from counting as consent. Verification settings must be set before the daemon starts, so it runs `$B stop` first. `--clear-storage` and `--all` each need explicit approval, with their constraints stated. The import receipt is clearly separated from sign-in evidence. Error and receipt-failure tables and platform boundaries are included.\n\nSome gaps and ambiguities remain. The Aside probe is delegated to another file. `--all` and `--clear-storage` appear in question 5 and in BROWSER.md but not in the direct-import step of SKILL.md. The picker-versus-direct decision is slightly unclear when a user wants `--all`. The \"one `--domain` per run\" rule is stated, but the multi-domain loop is not shown. The \"Mode: cdp\" check and the daemon-restart handling when verification is not wanted are only partly covered. Retry behavior for a partial import or `decryption_failed` is brief. The target-tab requirement and the `--profile` handling for the picker are somewhat implicit. Overall the workflow is solid and executable, with a few minor omissions."},"latency_ms":8060,"stop_reason":"end_turn","usage":{"input_tokens":4538,"output_tokens":448},"cost_usd":0.013556} +{"config":"cookie-workflow","item":"p05","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: Aside probe, then up-front consent for every decision, daemon setup, import, and a three-line honest report. It separates copied cookies from sign-in evidence and defines the storage-reset scope precisely. The error and failure-reason tables, platform limits, and recovery paths via handoff/resume are documented. Gaps and ambiguities remain. The `--all` direct-import syntax and the picker's handling of `--all` are not shown. Step 3.1 wants the verification env vars set before the daemon starts, but the daemon may only start in step 3.2 or 3.3, so the ordering is a little odd. The direct import example uses `--domain` once per run, while the user may name several domains. Step 4 says to return to step 3 on missing verification settings, but not what to do if the user declines. Handling of `decryption_failed` is only loosely tied to the platform boundaries, and the guidance on when to use `$B handoff` is split across two files. Despite these gaps, an agent can execute the workflow with only minor guesswork."},"latency_ms":3696,"stop_reason":"end_turn","usage":{"input_tokens":4244,"output_tokens":374},"cost_usd":0.012228} +{"config":"cookie-workflow","item":"a04","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":2,"completeness":4,"actionability":3,"reasoning":"The workflow is well structured: it gathers consent first, treats verification and storage reset as opt-in, and reports honestly from the receipt. But SKILL.md and BROWSER.md directly contradict each other on `--profile`. SKILL.md says to use the display name, not the directory. BROWSER.md says to use the directory (`Profile 2`), not the display name, and its examples and error tables follow that. A direct import could fail or select the wrong profile. Smaller gaps remain: `--all` is not covered in the direct-import step, there is no handling for multi-domain receipts, and the daemon/env ordering is awkward."},"latency_ms":7338,"stop_reason":"end_turn","usage":{"input_tokens":4526,"output_tokens":823},"cost_usd":0.017282} +{"config":"cookie-workflow","item":"a03","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"Strong coverage of consent, receipt reporting, and platform limits. But the verification ordering conflicts: SKILL.md says set env vars in the shell running `$B` and rerun, while BROWSER.md says they must be in the daemon environment before startup (an existing daemon is not reconfigured). The picker flow says verification can be chosen after the import, but it does not say how to restart the daemon. The \"return to step 3\" recovery is vague, and `--verify-auth` on direct import is not shown with a full command. Some error-code recovery steps are inconsistent."},"latency_ms":2757,"stop_reason":"end_turn","usage":{"input_tokens":4899,"output_tokens":216},"cost_usd":0.011958} +{"config":"cookie-workflow","item":"a05","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"Strengths: the workflow is thorough on safety. It never guesses accounts, requires explicit consent for `--clear-storage` and `--all`, requires privately configured daemon env vars before startup for `--verify-auth`, and keeps 'not checked', 'not verified' and 'verified' distinct. It also gives clear platform limits, a failure-reason table and an import-error table.\n\nClarity problems: Section 1 is confusing. It says to read browse/SKILL.md's BROWSER SETUP and run its probe first, then a SETUP block is embedded in the same file, which is redundant or contradictory. It says 'On READY, stop importing', yet the SETUP block prints READY on success, which would halt the import flow. That appears to be a major ordering contradiction. It is also unclear whether 'Aside' refers to the same browser. The `$B status` check for `Mode: cdp` comes after the setup. The 'probe's fallback handling' is undefined here. Section 2 says 'Open the known target' before the target is established. The flow from picker to direct import is dense, and 'the bundle order is not execution order' adds to the ambiguity.\n\nCompleteness gaps: No decision tree for when to use the picker versus direct import. The `--verify-auth` and `--clear-storage` command syntax is not shown in an example. It is unclear how to turn on the options in the picker flow. The final report format is not specified. The agent is not told what to do after the user chooses, such as which command to run or what the receipt looks like.\n\nActionability: The core commands are present, and the handoff and resume recovery steps are actionable. The READY contradiction and some undefined references reduce executability."},"latency_ms":6081,"stop_reason":"end_turn","usage":{"input_tokens":4929,"output_tokens":583},"cost_usd":0.015688} +{"config":"cookie-workflow","item":"f02","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":3,"reasoning":"The workflow is detailed, but it contradicts itself in places that matter. (1) Step 2.4 says 'Do not ask; always clear' storage, yet the task requires explicit consent for a precisely scoped storage reset. The reset wipes all of the browser's stored site data, not just the target domain's, and the user is never told this. (2) Step 2.5 says to add `--all` whenever the user is unsure and not to ask. BROWSER.md says `--all` cannot be combined with `--domain` or `--clear-storage`, so the SKILL's own direct-import recipe (always `--clear-storage`, one `--domain`) conflicts with it. Importing every cookie in the profile is also a privacy-expanding default that the user never approved. (3) The 'Get every decision' step says to ask before any `$B` command, but step 3 has the user configure verification variables afterwards, and the Aside probe in step 1 comes before the questions. (4) Picker vs. direct import is only partly defined: if the user wants verification and is unsure of the domains, the picker path is unclear. (5) Failure handling is incomplete. The `$B cookie-import-browser` command with no flags is ambiguous, because it opens the picker in one place and lists detected browsers in another. Several recovery paths are covered, such as the receipt failure reasons, handoff, and the platform limits. The reporting format and the distinction between copied cookies and verified sign-in are clear and good. Overall the workflow is partly executable but has serious contradictions around consent and scope."},"latency_ms":12388,"stop_reason":"end_turn","usage":{"input_tokens":4382,"output_tokens":534},"cost_usd":0.014104} +{"config":"cookie-workflow","item":"p04","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: probe Aside, gather every decision first, prepare the daemon, import, then report. The scoping rules are precise, including profile directory vs display name, one domain per run, and the `--all` constraints. Verification is explicit (exactly one visible element, same-origin response, whitespace-normalized match). Storage reset needs explicit consent and its limits are stated, and failure tables map receipt and error codes to next steps. The report format separates copied cookies from sign-in evidence, and the privacy rules are clear.\n\nSome gaps remain. The `--all` direct-import invocation is only implied. Consent for `--all` is required, but the instructions never say how to import without a domain, or whether the picker covers it. How the user finishes the picker flow, and how a target URL is chosen when `--all` is used, are not specified. The picker uses the `comet` default, yet 'Never treat default Comet as consent' leaves the no-flags case slightly ambiguous. Headed manual sign-in is only conditionally offered, and `handoff` is not explained in the visible files. The `not_installed` error says 'pick a browser available on this OS' but gives no discovery step beyond the no-flags command. Step 3.1 also asks about verification before the daemon is started, which can conflict with an already-running daemon, though `$B stop` covers that. Overall the workflow is strong and mostly executable, with a few minor edge-case holes."},"latency_ms":4907,"stop_reason":"end_turn","usage":{"input_tokens":4539,"output_tokens":502},"cost_usd":0.014098} +{"config":"cookie-workflow","item":"f03","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"Strong consent flow: gather decisions up front, no guessing, and explicit scoping of --clear-storage and --all. But there are key ambiguities. SKILL.md asks for verification settings and says a rejected import reports a 'missing variable,' yet never defines what to configure (no variable names or steps). Sign-in reporting is also contradictory: it says signed in when cookies are copied and reload returns 200, which conflicts with the stated need to distinguish copied cookies from positive sign-in evidence, and the receipt's auth outcome is ignored. The Aside probe is not spelled out, and the report format lacks a verification result."},"latency_ms":3839,"stop_reason":"end_turn","usage":{"input_tokens":4383,"output_tokens":237},"cost_usd":0.011136} +{"config":"cookie-workflow","item":"f05","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"Only step 1 (the Aside probe and root selection) is concrete. The rest is placeholders: \"TODO: list the questions to ask the user\", \"TBD\" for preparing the browser and for platform boundaries, and \"Details to be added\" for the import. BROWSER.md's referenced section is only a draft note. Nothing covers browser/profile/domain selection, auth verification, scoped storage-reset consent, or telling copied cookies apart from sign-in evidence. An agent cannot run this workflow."},"latency_ms":2442,"stop_reason":"end_turn","usage":{"input_tokens":1404,"output_tokens":201},"cost_usd":0.004818} +{"config":"cookie-workflow","item":"a04","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":2,"completeness":4,"actionability":3,"reasoning":"The workflow is detailed, but it contradicts itself on a key input. SKILL.md says `--profile` takes the display name (`Work`) and not the directory, while BROWSER.md says it takes the directory (`Default`, `Profile 2`) and not the display name, and its examples use `Profile 2`. An agent can't tell which to pass, and a wrong value could select the wrong account or fail. The ordering is also awkward: the user is asked about the profile before the picker or direct import is chosen, and the profile list is only available through the picker. Otherwise coverage is strong: consent, receipt reporting, error tables, and platform limits are all present."},"latency_ms":3178,"stop_reason":"end_turn","usage":{"input_tokens":4923,"output_tokens":240},"cost_usd":0.012246} +{"config":"cookie-workflow","item":"f05","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"Only Section 1 (the Aside probe check) is concrete. It says to pick the gstack root, read BROWSER SETUP, run the probe, and end the skill on READY. The rest is placeholders: 'TODO: list the questions', 'TBD', 'Details to be added', and 'Let the user know how it went'. BROWSER.md's referenced section is only a draft note and has no content. Nothing covers how to choose a browser, profile, or domain without guessing, how to configure optional sign-in verification before mutation, how to get scoped consent for a storage reset, how to tell copied cookies from positive sign-in evidence, or what the platform and privacy boundaries are. The reporting step has no criteria, and the import command is never named. An agent cannot run this workflow beyond the first branch and would have to invent the rest, which is exactly the guessing the task is meant to prevent."},"latency_ms":3348,"stop_reason":"end_turn","usage":{"input_tokens":1007,"output_tokens":320},"cost_usd":0.005214} +{"config":"cookie-workflow","item":"a04","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"Consent gating, verification config, storage-reset scope, honest three-line reporting and platform limits are thorough and mostly well ordered. However, SKILL.md and BROWSER.md directly contradict each other on whether `--profile` takes the display name or the directory name (BROWSER.md's example uses \"Profile 2\"), which can make direct imports fail or select the wrong account. Smaller gaps remain: the `--all` flow still needs a target domain, multi-domain reporting is unclear, and there is no handling for a partial receipt."},"latency_ms":7225,"stop_reason":"end_turn","usage":{"input_tokens":4526,"output_tokens":838},"cost_usd":0.017432} +{"config":"cookie-workflow","item":"p04","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: ask all decisions up front, probe Aside, prepare `$B`, import, then report in a fixed three-line format. It gives explicit consent gates (`--clear-storage` and `--all` default to no and need approval), the `--all` exclusivity rules, and a daemon-restart requirement for the auth env vars. It also separates not checked, not verified and verified, and has error and recovery tables plus platform and privacy limits. Weaknesses: (1) The picker is described as handling browser, profile and domains, yet step 2 says ask before any `$B` command, and the picker, direct import and `--all` paths are not clearly reconciled, including how `--all` is passed. (2) `--verify-auth` appears as both a CLI flag and a picker checkbox, and the instructions for the picker path are thin. (3) Direct import allows one `--domain` per run, with no explicit multi-domain loop or guidance on whether the target must match every domain. (4) `$B handoff` and `$B resume` are said to need a display, but there is no instruction for a headless environment beyond not offering them. (5) Some cross-references are slightly redundant, for example the `profile_required` guidance mentions `$B cookie-import-browser `. (6) The ordering of the Aside probe against reading `browse/SKILL.md` relies on external files. Overall it is strong and mostly executable, with a few gaps."},"latency_ms":4929,"stop_reason":"end_turn","usage":{"input_tokens":4539,"output_tokens":497},"cost_usd":0.014048} +{"config":"cookie-workflow","item":"a03","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"The workflow has strong structure and honest reporting rules: ask first, import, then report the receipt in three lines. It covers consent for storage reset and `--all`, separates copied cookies from verified sign-in, and states platform limits. Several ordering and decision gaps hurt execution. (1) Step 2 says the user may decide on verification after seeing the import, but the verification variables must be set in the daemon environment before startup. SKILL.md says to set them in 'the shell that runs $B' and rerun, while BROWSER.md says a later CLI call does not reconfigure an existing daemon. SKILL.md never says to restart the daemon, so a rerun may silently fail. (2) Step 4 says that if verification settings are missing, 'return to step 3', which is confusing since `$B goto` is already done and the daemon may need a restart. (3) Whether the picker or direct import should be used is only partly specified. The direct import wording ('one --domain per run') leaves multi-domain handling and the target URL unclear. (4) SKILL.md names `--all` but gives no direct-import instructions or flag-combination handling beyond a note. (5) The Aside probe and the Browser fallback are delegated to external files, which is acceptable. (6) After a failed import, recovery is split between SKILL.md and BROWSER.md: handoff and resume are mentioned only briefly, and there is no explicit guidance on what to do when `partial` or `empty` results come back. (7) Reporting the Imported line needs a count, and the receipt format is not fully specified. Overall the workflow is solid, but the verification-daemon ordering is a notable contradiction."},"latency_ms":5418,"stop_reason":"end_turn","usage":{"input_tokens":4502,"output_tokens":568},"cost_usd":0.014684} +{"config":"cookie-workflow","item":"p03","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well sequenced: the Aside probe, then all five questions up front, then setup, import, and a three-line report. Consent defaults for --verify-auth, --clear-storage and --all are explicit. Copied cookies are clearly separated from sign-in evidence. Error tables and platform boundaries are given. Weaknesses: (1) The Aside probe and the \"Browser fallback\" section live in browse/SKILL.md, which is outside the bundle (this is acceptable as an external reference). (2) Step 2 asks five questions up front, but the `--all` option is not clearly handled in the import step. It conflicts with the \"one --domain per run\" rule and with the requirement that a domain be chosen, and it is not mentioned in step 4 or in the picker flow. (3) Handling of multiple domains is thin: it says one --domain per run, but the receipt report and how to aggregate across runs are not defined. (4) Order is ambiguous in places. The selector variables are set in step 3 after the questions, and the \"go back to step 3\" instruction after a missing-variable rejection is slightly awkward. $B stop would also affect the previously opened tab. (5) The picker's \"Imported\" line is not defined for the case where no receipt is available. (6) The direct-import example in BROWSER.md omits flags, and the instruction that the target page must match the domain is only implicit. Overall the document is solid, with a few gaps."},"latency_ms":11751,"stop_reason":"end_turn","usage":{"input_tokens":4633,"output_tokens":503},"cost_usd":0.014296} +{"config":"cookie-workflow","item":"a04","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":3,"reasoning":"The workflow is detailed, but it contradicts itself on `--profile`: SKILL.md says it takes the display name (`Work`) and not the directory, while BROWSER.md says it takes the directory (`Default`, `Profile 2`) and not the display name. An agent could pass the wrong value, and the example `--profile \"Profile 2\"` confirms the directory form. Ordering is also awkward: Aside probe and `$B` setup come before the `$B status` check for CDP. Omitted-browser default and `--domain` multiplicity are only partly explained. Strengths are the consent gates, the receipt reporting format, and the platform boundaries. The profile contradiction undermines the core step of selecting an account without guessing."},"latency_ms":3118,"stop_reason":"end_turn","usage":{"input_tokens":4923,"output_tokens":268},"cost_usd":0.012526} +{"config":"cookie-workflow","item":"p02","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow orders decisions well: collect consent first, configure verification before the daemon starts, and report three honest lines. It distinguishes copied cookies from verified sign-in. Gaps: Aside probe and browser fallback live in other files. Direct import only says one --domain per run, so multiple domains and the --all command form are underspecified. Step 3 says to run `$B goto` before the picker, while the picker flow's target is unclear. The `--all` mode also conflicts with the requirement for a target URL. Recovery for a missing verification variable is clear, but retry guidance after partial failures is thinner."},"latency_ms":3107,"stop_reason":"end_turn","usage":{"input_tokens":4935,"output_tokens":227},"cost_usd":0.01214} +{"config":"cookie-workflow","item":"f03","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The flow is mostly well ordered. It runs the Aside probe, collects the user's decisions up front, prepares `$B`, imports, and reports. The scoping of storage reset is precise, and the platform limits and error tables are useful. Several gaps and contradictions weaken it.\n\n**Verification is undefined.** Step 4 says to add `--verify-auth` only if the user said yes in step 2, but step 2 never asks about verification. It also says to tell the user 'which variable is missing' when verification settings are absent, yet neither file names those variables or says how to configure them. The instruction to 'return to step 3' after such a rejection is also odd. This leaves the 'configure verification before mutation' requirement unactionable.\n\n**Sign-in evidence is weak.** Both files say that copied cookies plus an HTTP 200 on reload means 'signed in', and BROWSER.md says 'once cookies are copied... the session is signed in'. A 200 is not positive sign-in evidence. The report format also has no 'unknown' or 'unverified' state, and it does not use the receipt's authentication outcome.\n\n**Smaller omissions.** Direct import with `--all` is not described in step 4. Recovery is only partly specified: `handoff` and `resume` appear in BROWSER.md, but SKILL.md does not tie them to specific failures. The Windows and Chrome 136+ boundaries are stated but not turned into decision steps."},"latency_ms":8941,"stop_reason":"end_turn","usage":{"input_tokens":3986,"output_tokens":1023},"cost_usd":0.018202} +{"config":"cookie-workflow","item":"f04","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The workflow has several direct contradictions in ordering and configuration. (1) Step 1 says to run the step 4 import right away on a non-READY result and return to step 2 afterward. Step 2 says to ask the user everything before running any `$B` command, and the import needs those decisions. (2) SKILL.md step 3 has the user set the verification env vars after `$B status` has started the daemon, and says not to restart it. BROWSER.md says the vars must be in the daemon environment before startup, and that setting them later does not reconfigure an existing daemon. Following SKILL.md therefore makes verification fail. The recovery path ('return to step 3') also cannot work without a restart, and step 3 refers to 'below' for a status check that is item 3 of the same list. (3) SKILL.md opens the target URL after the import, but BROWSER.md requires navigating to a page matching `--domain` first for direct import. (4) The `--all` direct-import syntax is not specified, and the picker's handling of `--all` and `--clear-storage` is unclear. The skill is strong on consent, scoping, honest three-line reporting, and platform and privacy limits, and its error tables are useful. The ordering and env-var conflicts would likely derail an agent, so actionability is low."},"latency_ms":13889,"stop_reason":"end_turn","usage":{"input_tokens":4538,"output_tokens":814},"cost_usd":0.017216} +{"config":"cookie-workflow","item":"a05","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"The workflow covers source and profile selection, no guessing of accounts, optional --verify-auth and --clear-storage with explicit consent, the distinction between copied cookies and verified sign-in, and platform limits. It is fairly complete on decisions and failure handling (receipt reasons, error codes, handoff/resume). Clarity suffers from ordering problems and ambiguity. Section 1 says to read browse/SKILL.md's BROWSER SETUP and run its probe first, yet SKILL.md also embeds a SETUP block. It says 'On READY, stop importing: use Aside's sessions', while the embedded script prints READY when the browser is ready, which seems to contradict continuing. The 'Mode: cdp' stop is placed after the setup. Section 2 says 'Open the known target' before the source is chosen, and where the target comes from is not defined. How to pass --verify-auth or --clear-storage in the CLI invocation is not shown, and the exact syntax is missing. The final report format is described loosely with no template. The text is dense and compressed in places (e.g. 'Unreadable profiles are unknown, not empty'), and the bundle order is non-sequential. The agent can mostly execute the workflow, but it has to resolve the READY ambiguity and infer flag usage itself."},"latency_ms":4936,"stop_reason":"end_turn","usage":{"input_tokens":4929,"output_tokens":459},"cost_usd":0.014448} +{"config":"cookie-workflow","item":"a05","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"Strengths: the workflow covers source/profile/domain selection without guessing, privately configured verify-auth, explicit consent for a precisely scoped storage reset, and the distinction between copied cookies and positive sign-in evidence. Platform and privacy limits are documented, and BROWSER.md gives examples and error tables.\n\nWeaknesses in clarity: ordering is contradictory. Section 1 tells the agent to read browse/SKILL.md's BROWSER SETUP and run its probe, then SETUP follows with its own inline probe. The `READY` handling is ambiguous: 'stop importing: use Aside's sessions' conflicts with the inline script, which prints READY for the normal flow. The `$B status` check is placed after setup. Section 2 says to 'Open the known target' before the source is chosen, yet the target is never defined or obtained. The text is also dense and compressed (for example 'isolated world/native deadline'). Many conditionals are packed together.\n\nWeaknesses in completeness and actionability: no step says how to request --verify-auth or --clear-storage in the command or what the exact CLI syntax is. The text does not say what to do if daemon env is not set (restart the daemon?). It never says how to invoke the picker versus direct import in sequence, or how to interpret the receipt after import. Windows recovery is split across the two files. The mix of hard stops and soft options leaves some decisions to the agent's judgment."},"latency_ms":5073,"stop_reason":"end_turn","usage":{"input_tokens":4929,"output_tokens":509},"cost_usd":0.014948} +{"config":"cookie-workflow","item":"p02","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: check Aside, collect all decisions up front, prepare the browser, import, then report. Consent defaults and boundaries are explicit (--all, --clear-storage, --verify-auth), and the receipt-based three-line report plus the error and failure-reason tables make recovery concrete. Gaps remain. Step 3 says to open the target before import, but it does not say what to do if a daemon restart is needed after a goto. Verification setup is mentioned in step 3 but its ordering relative to the picker is unclear. The `--all` direct-import syntax is not shown. The Aside probe command lives in another file that is only referenced. The 'multiple domains' case (one --domain per run) is ambiguous for the target and the receipt. Receipt format and error handling for partial or empty results are only loosely defined. Windows handoff and resume steps are scattered across both files. Overall the workflow is usable with minor ambiguity."},"latency_ms":10342,"stop_reason":"end_turn","usage":{"input_tokens":4538,"output_tokens":340},"cost_usd":0.012476} +{"config":"cookie-workflow","item":"a05","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"Strong on consent, privacy, and verified vs not-checked semantics, and BROWSER.md gives failure tables and platform limits. But ordering is contradictory: Section 1 says to run the probe first and stop on READY, then a SETUP block follows with NEEDS_SETUP/Mode:cdp stops, and it is unclear whether READY means stop importing or proceed. The `--verify-auth` flow is also unclear: env vars must be set before daemon startup, yet the target is opened and the daemon is already running. Missing is how to restart the daemon safely, and how to open the target in picker mode."},"latency_ms":3081,"stop_reason":"end_turn","usage":{"input_tokens":5326,"output_tokens":229},"cost_usd":0.012942} +{"config":"cookie-workflow","item":"p03","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered and precise. It asks all five questions up front, treats the Comet default as non-consent, and sets explicit consent gates for --all and --clear-storage. The verify-auth criteria are exact, the picker and direct-import paths are clearly separated, and the three-line report format is fixed. Platform boundaries and privacy rules are stated, and BROWSER.md adds error tables and examples. Weaknesses: (1) The ordering is slightly awkward: Step 2 says to ask everything before touching $B, but Step 1 runs the Aside probe and Step 3 reads env vars, and the daemon restart is in Step 3. (2) A cross-file dependency on browse/SKILL.md sections for the Aside probe and $B discovery is not included in the bundle, which is acceptable. (3) Some decisions are missing: what to do if the user wants multiple domains (one --domain per run is stated, but the loop and the target are unclear), how to handle a partial result, and what to do when the user declines every option. (4) The Windows manual sign-in fallback has little detail, and the Step 4 error path for missing verification variables says to go back to step 3 without covering a daemon already running. (5) The 'exactly three lines' report format conflicts with the need to report failure reasons and next steps. (6) Direct import requires a target page matching the domain, which can conflict when several domains are chosen. Overall it is strong but has a few gaps."},"latency_ms":11779,"stop_reason":"end_turn","usage":{"input_tokens":4633,"output_tokens":505},"cost_usd":0.014316} +{"config":"cookie-workflow","item":"f02","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":3,"reasoning":"The workflow is mostly well structured, with a clear Aside check, a verification setup order, a receipt-based report format, and error tables. It has serious contradictions and gaps, though.\n\n1. Step 2 item 4 says not to ask about `--clear-storage` and to always clear storage. The task requires explicit consent for a precisely scoped storage reset, and BROWSER.md says the reset wipes the browser's stored data. This is a major flaw in consent handling.\n2. Item 5 says to add `--all` whenever the user is unsure, with no need to ask. But `--all` imports every cookie in the profile, which is a broad privacy action taken without consent. BROWSER.md also says `--all` cannot be combined with `--domain` or `--clear-storage`. That conflicts with the always-add `--clear-storage` rule and with the 'one --domain per run' direct import.\n3. The domain-picking logic is contradictory. The user is asked for domains, but `--all` is added when unsure. The picker is described as the way to resolve open answers, yet it is also described as needing no consent for clearing.\n4. The ordering is muddled. The Aside probe is described in a file that is not shown. SKILL.md says to ask before any `$B` command, but the Aside probe and `$B status` come earlier or between. Verification env vars must be set before daemon start, while the `$B stop` guidance is present only in part.\n5. Some details are missing. The direct-import syntax omits the `--verify-auth` flag example. How to handle multiple domains is unclear. There is no decision for partial or failed reset.\n6. The reporting rules are good: the three-line receipt, 'not checked' versus 'not verified', and privacy limits.\n\nThe consent contradictions lower all three scores."},"latency_ms":12765,"stop_reason":"end_turn","usage":{"input_tokens":4382,"output_tokens":617},"cost_usd":0.014934} +{"config":"cookie-workflow","item":"a05","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"Covers source/profile selection, verify-auth config, clear-storage consent, and honest reporting well. Ordering is contradictory, though: section 1 says to run the BROWSER SETUP probe first, but the SETUP block and bun install appear inline, and the `$B status` check follows after. Section 2 says \"Open the known target\" before the target is defined or navigated to. How the daemon is restarted with the env vars is not spelled out. There is no explicit instruction on handling a missing-config rejection."},"latency_ms":2833,"stop_reason":"end_turn","usage":{"input_tokens":5326,"output_tokens":195},"cost_usd":0.012602} +{"config":"cookie-workflow","item":"a02","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered and decision-driven. It gathers all consent items up front (browser, profile directory, domains, target, verification, storage clear, --all). It handles the Aside and CDP short-circuits and sets verification env vars before the daemon starts. The receipt-based three-line report is precise, it separates copied cookies from sign-in evidence (not checked / not verified / verified), and it spells out privacy rules and platform limits.\n\nGaps remain. The picker-versus-direct choice is slightly underspecified, since the --all and target-URL handling is unclear for the picker. The direct-import command lacks an example and exact syntax in SKILL.md, though BROWSER.md has one. The 'one --domain per run' rule conflicts a little with importing multiple domains, because it implies multiple runs and each run's receipt and target handling are unspecified. Step 5 doesn't say how to aggregate across runs. The error tables are said to be in BROWSER.md but are not actually present in the excerpt. How to recover when the receipt is failed or empty is thin, beyond the headed manual sign-in offer. The --all flow is described only as a flag, with no instruction on how it interacts with the target or the picker. There is also a minor ordering tension: the Aside probe runs before $B exists, and 'Ask before running any $B command' is consistent with it. Overall it is solid but has some ambiguities."},"latency_ms":11924,"stop_reason":"end_turn","usage":{"input_tokens":3551,"output_tokens":492},"cost_usd":0.012022} +{"config":"cookie-workflow","item":"a04","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"The workflow has strong consent gates, a clear verification model, and honest three-line reporting. But SKILL.md and BROWSER.md directly contradict each other on whether `--profile` takes the display name or the directory name, and the BROWSER.md example uses the directory. That undermines the core goal of not guessing an account. There are also minor gaps: the direct `--all` invocation is not spelled out, and the step ordering around `$B stop` and finding `$B` is awkward."},"latency_ms":7617,"stop_reason":"end_turn","usage":{"input_tokens":4526,"output_tokens":875},"cost_usd":0.017802} +{"config":"cookie-workflow","item":"p02","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"Strong decision-gathering (step 2), consent gating, and honest three-line reporting separating copied cookies from verified sign-in. Ordering is coherent: ask, prepare daemon env, status, goto, import. Gaps: the Aside probe is delegated to browse/SKILL.md, and `--all` direct usage is not shown in the SKILL flow. The picker path also doesn't say how to confirm the receipt, and 'one --domain per run' leaves multi-domain handling and the verify-auth rerun path somewhat implicit. Minor tension: the `$B goto` example precedes the import, but verification env setup before `$B status` is clearly handled."},"latency_ms":3166,"stop_reason":"end_turn","usage":{"input_tokens":4935,"output_tokens":248},"cost_usd":0.01235} +{"config":"cookie-workflow","item":"p05","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: the Aside probe comes first, then all user decisions are gathered up front, then the browser is prepared, then the import runs, then the report is written. The decisions are well scoped. Consent is required for --clear-storage and --all, verification is optional, copied cookies are kept separate from sign-in evidence, and the receipt-based three-line report is clearly defined. Error tables and platform limits are also covered. Several gaps remain. The step 3 ordering is slightly awkward: the daemon must be stopped and the env vars set before `$B status`, and it is unclear whether Aside's `READY` probe or the picker/direct choice comes first. The `--all` direct-import syntax and how it interacts with a target are underspecified. The remaining ambiguities are the picker's behavior on a missing profile, how to handle multiple domains (one `--domain` per run, with repeated runs), and whether `$B goto` must be repeated for each domain. The \"Chrome 136+\" fallback path and what the agent does after a failed import are described only briefly. The `$B stop` step, the environment-variable setup, and the exact receipt format are only partly specified. Overall the workflow is strong, with minor gaps."},"latency_ms":11215,"stop_reason":"end_turn","usage":{"input_tokens":4244,"output_tokens":430},"cost_usd":0.012788} +{"config":"cookie-workflow","item":"a03","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"Strengths: SKILL.md gives a well-ordered flow (Aside check, gather decisions, prepare, import, report). It states the consent rules clearly: no guessing, the Comet default is not consent, --clear-storage and --all need explicit approval, and the exact scope of each is spelled out. It separates copied cookies from sign-in evidence with a clear three-line report format. BROWSER.md adds error tables and platform limits.\n\nProblems:\n1. Contradictory ordering on verification env vars. SKILL.md says to set them 'in the shell that runs $B' and then rerun the import. BROWSER.md says they must be set in the daemon environment before startup, and that setting them on a later CLI call does not reconfigure an existing daemon. By step 4, step 3 has already started the daemon through `$B goto`. An agent following SKILL.md would fail or have to guess that it must restart the daemon. Nothing says how to restart it.\n2. The picker flow for verification is unclear. It is not stated whether the env vars must be set before the picker runs, or how the picker's verification checkbox gets its selector and identity.\n3. Several decisions are left open. SKILL.md allows 'decide after seeing the import' for verification but does not say how to rerun without duplicating the import. Multiple domains require one `--domain` per run, yet it is unclear whether `--clear-storage` should repeat on each run. The `--all` direct import syntax is not shown. There is no flow for reading a receipt that is partial, empty, or failed, apart from a few table rows.\n4. The 'return to step 3' instruction after a missing-verification rejection is vague, since step 3 includes the `goto` that only needs to be redone if the daemon restarts.\n5. The Aside probe and the Browser fallback section are external references, which is acceptable here.\n6. The platform boundaries are good but terse. Dia's keychain handling and the 'headed manual sign-in only with display' rule are not tied to a detection step."},"latency_ms":14060,"stop_reason":"end_turn","usage":{"input_tokens":4502,"output_tokens":690},"cost_usd":0.015904} +{"config":"cookie-workflow","item":"a01","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: it gathers decisions up front, only then runs `$B`, and sets clear defaults and consent gates. It reports three honest lines and separates copied cookies from sign-in evidence. Error tables and platform limits are given. Gaps and ambiguities: (1) The picker opens a link, but the exact command output and how the agent hands it to the user are not described. (2) The `$B status` CDP check comes after the user has already answered the questions, which is slightly wasteful but not contradictory. (3) The Aside probe depends on external files. (4) The direct import example has no `--verify-auth` or `--clear-storage` example. (5) The `--all` flow is only partly specified: it is not clear whether it needs a domain-free target, and nothing says how the target is chosen. (6) The handling for a rejected verification (restart the daemon, then return to step 3) is clear, but the retry limits for decryption failures are vague ('close source browser and retry'). (7) The rule 'one `--domain` per run' versus multiple domains is only implicitly handled by running the import repeatedly. (8) The picker's receipt output format is not specified, so the report lines depend on an unseen receipt. Overall it is solid but not fully complete."},"latency_ms":11944,"stop_reason":"end_turn","usage":{"input_tokens":4403,"output_tokens":437},"cost_usd":0.013176} +{"config":"cookie-workflow","item":"a05","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"Strengths: the workflow covers source/profile/domain selection without guessing, optional auth verification configured before mutation, explicit consent for scoped storage reset, the not-checked/not-verified/verified distinction, and platform and privacy limits. Failure-reason and import-error tables give clear next steps.\n\nClarity weaknesses: the ordering is confusing. Section 1 says to read browse/SKILL.md's BROWSER SETUP and run its probe, and then a SETUP block follows that is separate from that probe. 'On READY, stop importing' conflicts with the later `$B status`/cdp check and with the setup script's own READY output, so it is unclear which READY is meant. 'Open the known target' in section 2 comes before the source is chosen, and it is unclear what the 'known target' is. The note that `--verify-auth` must be configured before daemon startup does not say what to do if the daemon is already running. The dense, compressed prose adds ambiguity.\n\nCompleteness and actionability gaps: how to pass `--verify-auth` or `--clear-storage` in the direct command is never shown. The conditions for choosing the picker over direct import are only loosely defined. How to restart the daemon is not documented. The bun install block is a reasonable fallback, but the bun version and checksum are hard-coded. The 'Aside' reference is undefined in this bundle."},"latency_ms":4565,"stop_reason":"end_turn","usage":{"input_tokens":4929,"output_tokens":485},"cost_usd":0.014708} +{"config":"cookie-workflow","item":"p01","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: Aside check, then collecting every decision up front, then setting up the browser, importing, and reporting. Consent defaults are explicit for verification, storage reset, and --all. The receipt is clearly distinguished from raw cookies as evidence, and the three-line report format is well defined. Platform limits and error tables are covered and cross-referenced. Gaps and ambiguities:\n- The \"Aside probe\" is delegated to another file and not shown here.\n- The \"Browser fallback\" section is also external.\n- The direct-import syntax in SKILL.md is only described in prose; the example is in BROWSER.md.\n- The one-domain-per-run rule is unclear for the target's relation to multiple domains.\n- The interplay of --all with the picker is not specified.\n- Step 3.1 asks for verification env setup before the daemon check, and the daemon stop ordering is slightly awkward.\n- The 'partial' receipt handling and the next step after a failed storage reset are underspecified.\n- The handling of Windows Opera and other platform recovery is spread across two files."},"latency_ms":11088,"stop_reason":"end_turn","usage":{"input_tokens":4521,"output_tokens":399},"cost_usd":0.013032} +{"config":"cookie-workflow","item":"f02","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The workflow is detailed on sign-in verification, honest reporting and platform limits, but it has serious contradictions that block faithful execution. (1) Step 2 says to never guess and to get every decision from the user, yet it tells the agent never to ask about `--clear-storage` and to always clear storage. That is a destructive wipe of site data with no consent, and the task explicitly requires explicit, precisely scoped consent. (2) Step 2 says to add `--all` whenever the user is unsure and not to ask. BROWSER.md says `--all` cannot be combined with `--domain` or `--clear-storage`. SKILL.md also requires `--clear-storage` on every direct import, so the two flags conflict outright. `--all` also imports every cookie, which violates the domain scoping the user chose. (3) The picker is said to confirm browser, profile and domains, but it is unclear how it relates to the always-on `--clear-storage`, and the picker has no stated way to get consent for the reset. (4) Direct import allows one `--domain` per run, while step 2 asks for multiple domains with no loop guidance. (5) Step 3 says to start `$B` before the Aside probe results are fully handled, and verification env setup precedes the `$B status` check, but that ordering is workable. (6) The failure-recovery tables are decent, yet `--all` cannot be reconciled with the rest, and the \"closing the browser\" retry for `decryption_failed` is unclear against the privacy constraints. Reporting is well-defined and the distinction between copied cookies and verified sign-in is strong, but the consent and flag contradictions keep all three scores low."},"latency_ms":12502,"stop_reason":"end_turn","usage":{"input_tokens":4382,"output_tokens":568},"cost_usd":0.014444} +{"config":"cookie-workflow","item":"p04","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"Strong consent gating, explicit receipt semantics (not checked vs not verified), and error tables. Gaps: the --all flow has no direct-import command or receipt-format guidance. The Aside probe is only referenced externally. Step 2 says no $B before decisions, yet step 3 sets verification env vars. Picker versus direct mode is clear, but --verify-auth in picker mode and --clear-storage handling are slightly vague. The 'Dia macOS-only' note is scattered, and mismatched-browser recovery is thin."},"latency_ms":10141,"stop_reason":"end_turn","usage":{"input_tokens":4936,"output_tokens":207},"cost_usd":0.011942} +{"config":"cookie-workflow","item":"f02","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The workflow contradicts itself on the key safety points. SKILL.md says to always add `--clear-storage` without asking, and to add `--all` whenever the user is unsure, with no need to ask. BROWSER.md says `--all` cannot be combined with `--domain` or `--clear-storage`, so the instructed combination fails. Clearing storage with no consent also conflicts with the stated goal of explicit, scoped consent. Other gaps: the picker's `--clear-storage` handling is unspecified, and the Comet default is not clearly resolved. Receipt and platform error handling is otherwise well covered."},"latency_ms":4269,"stop_reason":"end_turn","usage":{"input_tokens":4779,"output_tokens":231},"cost_usd":0.011868} +{"config":"cookie-workflow","item":"a06","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well sequenced: probe Aside, collect all decisions up front, prepare the daemon, import, then report from the receipt. The consent defaults for --verify-auth, --clear-storage and --all are explicit. The distinction between copied cookies, not checked, not verified and verified is precise, and the platform and privacy limits are clear. Error recovery is mostly documented, with tables of failure reasons and error codes and a pointer to the right BROWSER.md section.\n\nThere are gaps and minor ambiguities. SKILL.md step 2 asks for the target URL, but --all cannot be combined with a domain, and the flow for --all (which target to open, how to run it directly or in the picker) is thin. Direct import allows only one --domain per run, but the instructions do not say how to handle multiple domains; the implied answer is to loop, with a goto for each. The ordering is slightly awkward: step 3 needs the verification env vars set before the daemon starts, yet the Aside probe in step 1 and the `$B status` check come before or after other daemon actions. The \"missing verification settings\" rejection is handled, but the exact output form is not shown. The Windows `handoff`/`resume` fallback is mentioned in BROWSER.md only. There is no explicit instruction for receipt failures such as storage-reset failure or partial results beyond reporting them. The \"Browser fallback\" and \"BROWSER SETUP\" sections are external to this bundle but are treated as documented elsewhere."},"latency_ms":9441,"stop_reason":"end_turn","usage":{"input_tokens":4455,"output_tokens":523},"cost_usd":0.01414} +{"config":"cookie-workflow","item":"a03","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"Strong on consent, receipts, and honest reporting. But ordering conflicts: SKILL.md has the user set verification env vars in the shell running `$B` and then rerun, while BROWSER.md requires them in the daemon environment before startup, so an existing daemon would ignore them. SKILL.md also leaves `--all` unused in Step 4, and it never says how to restart the daemon."},"latency_ms":2563,"stop_reason":"end_turn","usage":{"input_tokens":4899,"output_tokens":159},"cost_usd":0.011388} +{"config":"cookie-workflow","item":"p02","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: check Aside, gather every decision up front, prepare `$B` (stopping a running daemon before setting verification variables), import, then report. Consent defaults and limits for `--clear-storage`, `--all` and `--verify-auth` are explicit. The three-line report format is clear and separates copied cookies from sign-in evidence. Error tables, platform limits and recovery paths (`$B handoff`, `$B resume`, manual sign-in) are documented. Weaknesses: (1) The picker and direct-import decision rule is slightly loose. It does not say what to do when the user wants `--all` or several domains, since direct import takes one `--domain` per run and the picker's handling of `--all` is unclear. (2) The ordering of `$B goto` against the daemon's start with `$B status` is subtle, and the `Mode: cdp` case is handled only briefly. (3) Handling for receipt states like `partial` and for `decryption_failed` is thin. (4) The Aside probe is defined in an external file. (5) The `--verify-auth` flow in direct mode has no explicit failure path beyond the missing-variable case. (6) The clearing step's relation to the 'leave tab alone' instruction is implicit. Overall it is solid but has some gaps."},"latency_ms":8962,"stop_reason":"end_turn","usage":{"input_tokens":4538,"output_tokens":454},"cost_usd":0.013616} +{"config":"cookie-workflow","item":"f05","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"Only the Aside check in step 1 is concrete. The rest is placeholders: step 2 says \"TODO: list the questions to ask the user,\" and step 3 says \"TBD.\" The referenced BROWSER.md section is only a draft note, so it adds no detail. Nothing covers choosing a browser, profile, or domain, optional sign-in verification, consent for storage reset, telling copied cookies apart from sign-in evidence, or platform and privacy limits. An agent could not run the import safely."},"latency_ms":3816,"stop_reason":"end_turn","usage":{"input_tokens":1404,"output_tokens":198},"cost_usd":0.004788} +{"config":"cookie-workflow","item":"a01","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well sequenced and avoids guessing. It asks for all decisions before any `$B` command, makes defaults explicit (verification, storage reset, and `--all` all default to no and need consent), and sets rules for the picker versus direct import. The three-line report format is strict, and the text states clearly that cookie counts, HTTP 200, and 'not requested' are not sign-in evidence. Error tables, receipt failure reasons, and platform limits are specific and give next steps.\n\nThere are some gaps and tensions. The `--all` flow is only partly specified: SKILL.md says it cannot combine with a domain, yet step 3 still requires a target URL and domains, and direct-import syntax for `--all` is not shown. Direct import does not say how to run several domains beyond 'one `--domain` per run'. Picker instructions are vague about how the agent learns the result, since the user just 'tells you when done'. The `--verify-auth` flow in the picker is a checkbox, but the daemon environment must be set beforehand. The Aside probe and the browser fallback are defined elsewhere, which is acceptable. The ordering of the `$B stop` step relative to an existing daemon that may hold other state is unaddressed. Handling of `partial` or `failed` receipts beyond the failure-reason table is thin, and so is the retry or consent handling for Keychain approval. The statement that the omitted-browser default is `comet` could invite mistakes, though SKILL.md guards against it. Overall, the workflow is solid and mostly executable."},"latency_ms":6491,"stop_reason":"end_turn","usage":{"input_tokens":4403,"output_tokens":539},"cost_usd":0.014196} +{"config":"cookie-workflow","item":"f03","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The flow is mostly well ordered: Aside probe, then consent questions, then browser prep, import, and a three-line report. Profile directories versus display names, Chromium-only storage reset scope, and platform limits are stated clearly. The error tables are usable. Several decisions are missing or contradictory. (1) Authentication verification is never asked in step 2, yet step 4 says to add `--verify-auth` only if the user said yes in step 2. (2) The verification variables are never named or explained, and the instruction to return to step 3 does not say how to supply them. (3) Sign-in reporting rests on cookies imported plus a reload that returns HTTP 200. That is weak evidence, and BROWSER.md even says copied cookies mean signed in, which conflicts with the goal of separating copied cookies from positive sign-in evidence. The receipt's own authentication outcome is not used, and there is no third state such as unverified or unknown. (4) The direct-import syntax for `--all` is not shown. (5) Handling of several domains when only one target is opened is unclear. (6) The picker path does not say how `--clear-storage` is requested. (7) Partial, empty and failed receipts are not mapped to recovery steps."},"latency_ms":9044,"stop_reason":"end_turn","usage":{"input_tokens":3986,"output_tokens":1008},"cost_usd":0.018052} +{"config":"cookie-workflow","item":"f01","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The workflow contradicts its stated goal. Step 2 says to ask the user for everything first, then item 1 says not to ask and to use Comet by default and pick the profile with the most cookies. That is guessing an account, and BROWSER.md says ambiguity requires a choice. It also tells the agent to use direct import and skip the picker, and to run one `--domain` per run. Target selection, verification, storage reset and honest reporting are well covered. Mixed fallbacks remain unresolved, and the Aside probe depends on external files."},"latency_ms":2714,"stop_reason":"end_turn","usage":{"input_tokens":4906,"output_tokens":204},"cost_usd":0.011852} +{"config":"cookie-workflow","item":"a03","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The workflow is well structured: it asks every question up front, handles the Aside and CDP shortcuts, sets consent defaults, and defines a three-line honest report. It also keeps clear boundaries between copied cookies and verified sign-in. Several gaps and contradictions reduce how well an agent can follow it.\n\n1. Verification ordering: SKILL.md step 4 says to have the user set the env vars in the shell that runs `$B` and then rerun the import. BROWSER.md says they must be set in the daemon environment before startup, and that setting them on a later CLI call does not reconfigure a running daemon. Step 3 already starts the daemon with `$B status` and `goto`. The agent therefore has no instruction to stop or restart the daemon, so verification could fail or be misconfigured.\n2. The option to decide on verification after seeing the import conflicts with the need to configure the daemon before startup, and the daemon restart is never explained.\n3. The picker flow and the `--verify-auth` flow are not clearly reconciled, and the picker's verification checkbox needs the same env vars.\n4. The direct import syntax is spread across two files. SKILL.md does not say how `--verify-auth` combines with direct import, and it leaves out the `--all` invocation entirely.\n5. Failure recovery is split. The receipt failure reasons and the import errors are in BROWSER.md and are only loosely referenced from SKILL.md. Errors such as profile_required are not mapped to concrete next questions for the user.\n6. The \"Failure reasons\" table lists `decryption_failed` as \"close the source browser and retry,\" which is not reconciled with the platform boundary notes.\n7. It does not say what to do when the picker link expires, or how the agent learns that the receipt is available.\n8. The report format has no slot for partial verification cases, and there is no explicit rule for what to report when the import fails."},"latency_ms":6050,"stop_reason":"end_turn","usage":{"input_tokens":4502,"output_tokens":661},"cost_usd":0.015614} +{"config":"cookie-workflow","item":"a05","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"The content is detailed and guardrailed. It covers explicit account/profile selection, pre-startup auth config, scoped consent for --clear-storage, the distinction between not checked, not verified and verified, and platform limits with failure tables. Several problems lower the scores. (1) Ordering is contradictory or confusing. Section 1 says to run the probe from browse/SKILL.md and stop on READY, then a SETUP block follows that treats READY as the success path. Reading that file's probe is also required but not included. The NEEDS_SETUP step 2 uses an undefined . (2) Section 2 says to 'open the known target' before the target is chosen, while section 3 asks the user to choose the domain. (3) It is unclear how to invoke the flags in practice, for example the exact CLI syntax for --verify-auth and --clear-storage combined with a direct import. The daemon restart procedure is not given when config is missing. Consent wording for --all and --clear-storage is only loosely specified. (4) Terse, compressed phrasing such as 'Aside's sessions' and 'default Comet' reduces readability. (5) The bun install block is detailed but tangential. The fallback paths (handoff/resume) are clear, and the failure tables are good."},"latency_ms":4688,"stop_reason":"end_turn","usage":{"input_tokens":4929,"output_tokens":450},"cost_usd":0.014358} +{"config":"cookie-workflow","item":"f04","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The reporting format, consent defaults, and platform boundaries are strong and specific. The execution order contradicts itself in several places. (1) Step 1 says to run the step 4 import right away and then return to step 2 for decisions, but step 2 says to ask everything before any `$B` command, and step 3 requires finding `$B` and checking for `Mode: cdp` before importing. (2) SKILL.md step 3 says to open the target after the import, while BROWSER.md says direct import requires first navigating to a page matching `--domain`. (3) SKILL.md says to set the auth environment variables after `$B status` has started the daemon and not to restart it. BROWSER.md says they must be in the daemon environment before startup and that a later CLI call does not reconfigure an existing daemon. This makes the verification path unreliable, and the 'return to step 3' recovery for missing settings cannot work as written. (4) `--all` is asked about in step 2 but never given a command in the import step. (5) Multiple domains are allowed, but only one `--domain` per run, with no loop guidance. The agent would have to guess at several decision points."},"latency_ms":13463,"stop_reason":"end_turn","usage":{"input_tokens":4538,"output_tokens":791},"cost_usd":0.016986} +{"config":"cookie-workflow","item":"p05","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: probe for Aside, collect every decision up front, prepare the browser, import, then report. Defaults and consent requirements are explicit (--clear-storage and --all need approval, verification is off by default, no guessing the browser or profile). The receipt-based three-line report and the separation of copied cookies from verified sign-in are clearly defined, and BROWSER.md adds error tables and platform limits. Gaps remain. Step 2 asks about --all, but step 4 never says how to run it (no direct-import syntax, and it conflicts with the 'one --domain per run' rule). The 'target' is a page on a domain, yet multiple domains require repeated runs and the target handling for those runs is unclear. How to pass the identity or selector in the picker flow is only implied. The ordering of the Aside probe against the GSTACK env vars and daemon restart is a little tangled. Recovery for specific failures (such as an unreadable profile, an expired picker link, or a failed reset) is only partly specified, and the handoff and resume steps appear only in BROWSER.md. An agent could execute this in most cases, but a few edge decisions are left implicit."},"latency_ms":11787,"stop_reason":"end_turn","usage":{"input_tokens":4244,"output_tokens":414},"cost_usd":0.012628} +{"config":"cookie-workflow","item":"p04","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered. It asks the user for decisions before running any `$B` command, and it defines consent for `--clear-storage` and `--all`. It states precisely what verification means and what a reset clears. The three-line report separates copied cookies from sign-in evidence, and the error tables and platform boundaries are clear.\n\nSome gaps remain:\n- The `--all` path is unclear. It cannot be combined with a domain, yet step 2 still requires a domain and a target, and the direct-import command says nothing about `--all`.\n- Direct import supports only one `--domain` per run, so a multi-domain import has no defined procedure such as repeated runs.\n- The `--all` and reset flags are not shown in any example.\n- The Aside probe and the Browser fallback are defined in external files, which is acceptable.\n- Step 3 says to open the target before the import, but the picker flow does not say how the target is captured.\n- The report format has no explicit slot for partial or failed outcomes beyond the status word.\n- A failed or expired picker has little recovery guidance beyond rerunning the command.\n- The `$B stop` instruction conflicts slightly with the \"leave tab alone\" rule if a daemon is running.\n\nOverall the workflow is usable."},"latency_ms":4259,"stop_reason":"end_turn","usage":{"input_tokens":4539,"output_tokens":452},"cost_usd":0.013598} +{"config":"cookie-workflow","item":"p03","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well-structured and explicit. It asks all five questions up front, never guesses answers, treats the default Comet browser as not being consent, and requires explicit approval for storage reset and --all. It also separates copied cookies from verified sign-in, gives a three-line report format, and ties error recovery to BROWSER.md tables and platform limits. The privacy rules are clear. Weaknesses: (1) The ordering is slightly awkward. Step 2 asks about verification, but env vars are set in step 3, and the verification-missing rejection sends the agent back to step 3, which may require a daemon restart. Step 3.3 can stop on cdp mode after the user has already answered the questions. (2) It does not say what to do if the user chooses --all with direct import, or how --all fits with the one-domain-per-run rule and multiple domains. (3) The Aside probe depends on external text that is not included. (4) Whether the receipt covers partial or failed multi-domain runs is unspecified. (5) The 'not verified' versus 'not checked' mapping from receipt fields is only implied. (6) The BROWSER.md reference points to a file in the root, which is fine. (7) The macOS Keychain and Dia limits are brief. (8) The fallback when no display is available is minimal. Overall it is actionable with minor gaps."},"latency_ms":11522,"stop_reason":"end_turn","usage":{"input_tokens":4633,"output_tokens":467},"cost_usd":0.013936} +{"config":"cookie-workflow","item":"p04","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: Aside probe, decisions up front, prepare, import, honest three-line report. It clearly separates copied cookies from sign-in evidence. One gap is that the `--all` path is not shown in the import commands. Another is the Step 3 order: the user must set the env vars and run `$B stop` before the Browser fallback setup, but that setup is not detailed here. Small gaps: no instruction for when the picker's user never says done, and the receipt-level `partial` handling is not specified. Overall it is executable."},"latency_ms":2885,"stop_reason":"end_turn","usage":{"input_tokens":4936,"output_tokens":204},"cost_usd":0.011912} +{"config":"cookie-workflow","item":"f03","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"Strong consent gating and error tables, but contradictions remain. SKILL.md step 5 says signed in requires an HTTP 200 on reload, yet BROWSER.md says copied cookies mean signed in and the reload only confirms; the workflow also never requires a reload and doesn't define the verification variables that a rejected import references. Step 2 says to ask before any $B command, yet the Aside probe runs earlier. Receipt-derived sign-in treats copied cookies as proof of sign-in, which conflicts with the stated goal. Direct import omits the target-domain consistency check."},"latency_ms":2743,"stop_reason":"end_turn","usage":{"input_tokens":4383,"output_tokens":223},"cost_usd":0.010996} +{"config":"cookie-workflow","item":"p01","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well-ordered: Aside check, then collect all decisions up front, then prepare the browser, import, and report. It handles consent, defaults, and the distinction between copied cookies and verified sign-in well, and the BROWSER.md error tables cover recovery. Remaining gaps: (1) the direct-import syntax is only described in prose in SKILL.md, and the example in BROWSER.md omits the verify and clear flags. (2) The Aside probe is delegated to another file that is not shown. (3) Multiple domains require multiple runs, but the handling of the target and the receipt aggregation is vague. (4) Step 2 says to ask before any `$B` command, yet the picker is used when answers are open, which is slightly in tension with it. (5) Step 3 setup of env vars comes after the user decides, and the restart behavior is clear but `$B stop` precedes finding `$B`. (6) The `--all` flow has no explicit command example. (7) The report format for the all-domains and failed cases is only partly specified. Overall it is solid but has some minor ambiguities."},"latency_ms":11377,"stop_reason":"end_turn","usage":{"input_tokens":4521,"output_tokens":388},"cost_usd":0.012922} +{"config":"cookie-workflow","item":"a02","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered and clear. All decisions are gathered up front. Defaults and consent requirements are explicit for verification, storage reset, and --all. Copied cookies are kept distinct from sign-in evidence, and the three-line report format is defined. Platform and privacy boundaries are stated. The gaps are minor. The Aside probe and the Browser fallback section live in an external file, which is acceptable. The picker's behavior when verification is requested but variables are missing is only partly covered. There is a slight tension between running `$B goto` before direct import and the picker flow. Recovery paths such as headed manual sign-in, retries, and unknown or unreadable profiles are brief. The step 2 instruction not to run any `$B` command before asking, together with the setup step that runs `$B status`, is ordered sensibly. The Windows and Chrome 136+ failures have no concrete recovery steps beyond manual sign-in. The `--all` flow is under-specified: it needs a target, it has no example, and it is unclear whether the picker supports it. The direct-import syntax for multiple domains is only 'one --domain per run'. Overall the instructions are strong and executable."},"latency_ms":8124,"stop_reason":"end_turn","usage":{"input_tokens":3551,"output_tokens":416},"cost_usd":0.011262} +{"config":"cookie-workflow","item":"f03","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The consent flow is strong: asks all decisions up front and scopes --clear-storage and --all precisely. But there are gaps and contradictions. SKILL.md reports sign-in as 'signed in' from cookies plus a 200 reload, while BROWSER.md says copied cookies mean signed in, so positive sign-in evidence is blurred. The 'missing verification settings' error never says which variable or how to set it, and --verify-auth is not defined. The 'signed in/signed out' report has no unknown state, and the receipt-to-line mapping for partial and failed imports is thin."},"latency_ms":3591,"stop_reason":"end_turn","usage":{"input_tokens":4383,"output_tokens":222},"cost_usd":0.010986} +{"config":"cookie-workflow","item":"f04","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The content is detailed on consent, scoping, honest reporting and platform limits, but the ordering contradicts itself in ways that block safe execution. (1) Step 1 says to run the step 4 import right away and only then collect decisions, while step 2 says to ask everything before any `$B` command and never guess. (2) Step 3 puts Aside/CDP detection and `$B status` after that early import, so the CDP 'no import needed' exit comes too late. (3) The verification variables are set 'after `$B status` starts the daemon' in SKILL.md, but BROWSER.md requires them in the daemon environment before startup and says later setting does not reconfigure the daemon. SKILL.md also forbids restarting the daemon, so the verification path may be unreachable. (4) SKILL.md opens the target URL after the import, but BROWSER.md's direct import first navigates to a page matching the domain. (5) The `--all` flow is not described in the direct-import steps, and it is unclear how `--all` fits with 'one `--domain` per run' and a target. The failure tables, receipt wording and platform boundaries are good, but the contradictions leave an agent without a reliable order of steps."},"latency_ms":13495,"stop_reason":"end_turn","usage":{"input_tokens":4538,"output_tokens":764},"cost_usd":0.016716} +{"config":"cookie-workflow","item":"a04","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":3,"reasoning":"The workflow is well structured, with strong consent gating, precise reporting semantics (imported vs. verified), error tables and privacy boundaries. But SKILL.md says `--profile` takes the display name, not the directory, while BROWSER.md says it takes the directory (`Default`, `Profile 2`), not the display name, and its examples use directories. Direct import could therefore pick the wrong profile or fail, which undermines the no-guessing goal. Smaller gaps: `--all` syntax and flow, multi-domain handling, and how to read the receipt are underspecified."},"latency_ms":6577,"stop_reason":"end_turn","usage":{"input_tokens":4526,"output_tokens":770},"cost_usd":0.016752} +{"config":"cookie-workflow","item":"a02","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: it asks the user for every decision before running anything, checks Aside and CDP mode to skip the import when possible, and gives a clear receipt-based three-line report. Consent for --clear-storage and --all is explicit. The rules separating copied cookies from verified sign-in are strong, and the platform and privacy limits are documented. Some gaps remain. (1) The exact `$B cookie-import-browser` flags for `--all` and the direct-import syntax are only partly shown. SKILL.md has no `--all` example, and BROWSER.md gives a single example. (2) Step 2 says to ask before any `$B` command, yet Step 1 requires reading the browse/SKILL.md Aside probe, which is ambiguous about whether the probe counts as a `$B` command. (3) Multiple domains require one run per domain, but the multi-domain target and receipt aggregation are not defined. (4) How to recover from a failed reset or a failed verification is described only loosely, as 'read BROWSER.md', and that section has no actual error tables. SKILL.md claims it contains error tables and receipt failure reasons, but BROWSER.md does not show them. (5) The ordering of Step 3, setting env vars and stopping the daemon, against Step 4 is clear. (6) The Aside and `Mode: cdp` shortcuts are clear. Overall the workflow is solid but has a few missing details."},"latency_ms":12170,"stop_reason":"end_turn","usage":{"input_tokens":3551,"output_tokens":498},"cost_usd":0.012082} +{"config":"cookie-workflow","item":"a05","split":"dev","phase":"old-b","sample":2,"status":"error","error_kind":"parse","error":"JSON Parse error: Expected '}'","latency_ms":6080,"stop_reason":"end_turn","usage":{"input_tokens":4929,"output_tokens":553},"cost_usd":0.015388} +{"config":"cookie-workflow","item":"a03","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"The workflow is well structured and covers consent, scoping, honest reporting, and platform limits. Several points hurt it.\n\n1) Verification env-var setup contradicts itself. SKILL.md says to set the variables 'in the shell that runs $B' and rerun with --verify-auth. BROWSER.md says they must be set in the daemon environment before startup, and that setting them on a later CLI call does not reconfigure an existing daemon. SKILL.md never says to restart the daemon, and step 3 has already started it. This is a real execution-order trap.\n\n2) Step 2 asks for verification and the identity values before any `$B` command. Yet it also says the user may decide after seeing the import, and the 'rerun' path conflicts with the one-use picker link and the --clear-storage consent. A rerun with --clear-storage could clear storage a second time.\n\n3) Some steps are underspecified. It does not say how to pass --all or --verify-auth in direct mode beyond a mention. It does not say how to run the Aside probe or what the Browser fallback section contains, since those live in external files. It does not say what to do when the picker is used, since the receipt must come from the picker's output. The target-URL requirement versus --all with no domain is unclear. The domain-versus-target-origin relationship is not stated.\n\n4) The direct-import error recovery is delegated to BROWSER.md, and the instruction to 'return to step 3' after a missing-config rejection is unclear about daemon restart.\n\n5) Positives: the three-line report format is exact. The evidence rules (copied cookies, 200, and counts never prove sign-in) are clear. Privacy and platform boundaries are clear. The error tables are useful."},"latency_ms":6229,"stop_reason":"end_turn","usage":{"input_tokens":4502,"output_tokens":597},"cost_usd":0.014974} +{"config":"cookie-workflow","item":"f02","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":3,"reasoning":"The workflow is detailed on verification, receipts, and platform limits, but it contradicts itself in several places and these contradictions matter. (1) SKILL.md step 2 says to always clear storage without asking, yet the task requires explicit consent for a precisely scoped reset, and `--clear-storage` wipes stored site data for the whole browser, not just the target domain. The skill never discloses that scope. (2) SKILL.md says to add `--all` whenever the user is unsure, with no need to ask, yet BROWSER.md says `--all` cannot be combined with `--domain` or `--clear-storage`. The default instructions therefore conflict with the documented constraints, and the agent is left to resolve the conflict. Because `--all` imports every cookie in the profile, applying it without asking also breaks the scoped-consent principle. (3) The picker path never says whether `--clear-storage` applies. (4) Step 2 says to ask before running any `$B` command, but step 3 has the agent set up and restart the daemon afterward, and the Aside probe in step 1 runs before any questions. (5) The direct import requires `--domain` once per run but does not say how to handle multiple domains. Strengths are the three-line honest report, the clear separation of copied cookies from verified sign-in, and the platform and privacy boundaries. The consent defaults and the `--all` conflict are significant defects."},"latency_ms":9232,"stop_reason":"end_turn","usage":{"input_tokens":4382,"output_tokens":485},"cost_usd":0.013614} +{"config":"cookie-workflow","item":"f01","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The workflow has serious internal contradictions. Step 2 says \"Get every decision from the user first\" and \"Ask these together,\" but item 1 says \"Do not ask\" about the browser and profile and tells the agent to pick the default (Comet) and the profile with the most cookies. That conflicts with the stated goal of not guessing an account and with BROWSER.md, where ambiguity or unreadable profiles require a choice and Comet is only a CLI-compatibility default, not a recommendation. Choosing the profile with the most cookies is account guessing. It also gives no way to count cookies per profile before importing. The \"Use direct import... The picker is only needed if no profile has cookies\" instruction reinforces the guessing, and it conflicts with the picker flow, which lets the user confirm the profile.\n\nOther gaps and ordering problems:\n- The target and domain are collected in step 2, but the Aside probe in step 1 runs earlier and is fine. The verification env vars come in step 3, but a missing-variable rejection sends the agent back to step 3 without saying to restart the daemon.\n- The `--all` consent is described, but the direct command syntax for it is not shown.\n- The domain choice is \"one --domain per run,\" but how to handle multiple domains with a single target is unclear.\n- The reset consent is covered well, with its scope and non-rollback stated.\n- The three-line report is well defined, and the distinction between copied cookies and positive sign-in evidence is clear.\n- Platform boundaries and recovery (handoff/resume, a display needed) are documented but scattered, and recovery from `partial` or `empty` results is thin.\n\nThe mix of strong, precise parts and the contradictory source-selection step reduces clarity and actionability."},"latency_ms":6061,"stop_reason":"end_turn","usage":{"input_tokens":4509,"output_tokens":616},"cost_usd":0.015178} +{"config":"cookie-workflow","item":"p01","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered and explicit. It asks the user for every decision before any `$B` command, defaults conservatively, and explains each flag's scope (`--all`, `--clear-storage`, `--verify-auth`). The three-line report keeps copied cookies separate from sign-in evidence, and BROWSER.md adds error and receipt tables plus platform limits. Weaknesses: (1) The direct-import `--domain` rule conflicts with the multi-domain ask: it says one domain per run, but the target is one page on one domain, so handling several domains is unclear. (2) The Aside probe depends on an external section that is not shown. (3) The picker flow does not say how to read the receipt when the user reports done. (4) The `--all` path is undefined with respect to the target and the picker. (5) Setting up verification variables sits in step 3, after the answers are collected, and the daemon-restart ordering is easy to miss. (6) A mid-flow reset failure has no defined recovery beyond reporting it. (7) Handoff and resume are offered only in BROWSER.md. (8) The picker's receipt may show verbatim messages that could leak details."},"latency_ms":8982,"stop_reason":"end_turn","usage":{"input_tokens":4521,"output_tokens":410},"cost_usd":0.013142} +{"config":"cookie-workflow","item":"f01","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The workflow contradicts itself in ways that undermine safe execution. Section 2 opens with 'Get every decision from the user first', yet item 1 says 'Do not ask' about browser and profile and tells the agent to default to Comet and pick the profile with the most cookies. That is exactly the guessing the task says to avoid. BROWSER.md says that without `--profile`, only a sole relevant profile is selected, and that ambiguity requires a user choice. The omitted-browser default of `comet` is described as 'for CLI compatibility, not as a recommendation.' It is unclear how an agent could count cookies per profile without reading cookie data. Step 2 then says to use direct import 'with the browser and profile you picked', which conflicts with the picker flow where the user confirms the choice. The ordering is also muddled: step 2 says to ask before any `$B` command, but step 3 needs `$B` to find the daemon. The Aside probe in step 1 also depends on a browse/SKILL.md that is not shown.\n\nThe parts about verification, storage reset, and honest reporting are clear. The consent rules for `--clear-storage` and `--all`, the three-line receipt report, and the 'not checked' versus 'not verified' distinction are all well specified. Error tables and platform boundaries are good.\n\nSome gaps remain. There is no guidance on multiple domains. There is no clear handling of a failed or partial receipt beyond the error tables. The `--all` flow is not given in the direct-import steps. The 'skip profiles that cannot be read' instruction conflicts with the documented `profile_required` behavior. Overall the flow is executable in places but risky and ambiguous at the key selection decision."},"latency_ms":5554,"stop_reason":"end_turn","usage":{"input_tokens":4509,"output_tokens":590},"cost_usd":0.014918} +{"config":"cookie-workflow","item":"p03","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"Strong ordering: ask all five questions before $B, consent defaults, picker versus direct import, verification env setup before daemon start, and a fixed three-line report. Gaps: multiple domains are unclear since direct import takes one --domain per run, yet question 2 allows several, and per-domain handling or goto per domain is not specified. The --all direct path is not shown in step 4. The Aside probe and $B discovery depend on external files, which is acceptable. Reporting rules don't map receipt statuses like partial or failed storage reset when the reset is not requested, and the failure when verification is requested without the picker is implicit. Errors are well covered by BROWSER.md tables."},"latency_ms":3383,"stop_reason":"end_turn","usage":{"input_tokens":5030,"output_tokens":250},"cost_usd":0.01256} +{"config":"cookie-workflow","item":"a04","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":2,"completeness":4,"actionability":3,"reasoning":"The workflow is detailed, but the two files contradict each other on a key input. SKILL.md says `--profile` takes the display name, not the directory. BROWSER.md says it takes the directory (`Default`, `Profile 2`), not the display name, and its example uses `Profile 2`. An agent cannot tell which to pass, so a direct import could fail or pick the wrong account. Also, the default Comet browser is not treated as consent, yet the picker and direct-import conditions leave some cases ambiguous. Receipt reporting, consent gates, and platform limits are otherwise well covered."},"latency_ms":2854,"stop_reason":"end_turn","usage":{"input_tokens":4923,"output_tokens":223},"cost_usd":0.012076} +{"config":"cookie-workflow","item":"a01","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: ask all decisions first, prepare the daemon env, import, then report in three lines. It clearly separates cookie import from sign-in proof. Gaps: the picker/direct choice is unclear when the user names a browser but not the profile; the 'one --domain per run' rule conflicts with the multi-domain picker; and the target is a URL while import is domain-scoped. Missing-verification rejection says to return to step 3, but the daemon-restart step is only implied. The receipt error and failure tables are good, but the `--all` consent path and several partial/empty recovery paths are thin."},"latency_ms":4191,"stop_reason":"end_turn","usage":{"input_tokens":4800,"output_tokens":228},"cost_usd":0.01188} +{"config":"cookie-workflow","item":"p01","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: ask first, prepare, import, report. It separates copied cookies from verified sign-in, sets out consent defaults, and covers receipt failure reasons. Weaknesses: the Aside probe is delegated to browse/SKILL.md, and Step 2 says to ask before any $B command while Step 3 has the user set env vars. It does not say what to do if the picker or direct import leaves the user's answers unresolved for the target domain. It also leaves unclear how to handle multiple domains (one --domain per run) alongside a single verification target, and what to do on a partial or failed storage reset. Overall it is strong and executable, with a few gaps."},"latency_ms":3768,"stop_reason":"end_turn","usage":{"input_tokens":4918,"output_tokens":249},"cost_usd":0.012326} +{"config":"cookie-workflow","item":"f02","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":3,"reasoning":"The workflow is detailed and the sign-in evidence rules are strong, but it contradicts itself on key decisions. (1) Step 2 says to always clear storage without asking, yet the task requires explicit consent for a precisely scoped reset, and clearing wipes the browser's stored site data. (2) Step 2 says to add `--all` whenever the user is unsure and not to ask, but BROWSER.md says `--all` cannot be combined with `--domain` or `--clear-storage`, so the direct-import instruction to always add `--clear-storage` conflicts with it. `--all` also broadens cookie scope beyond what the user authorized, which contradicts the 'don't guess' principle. (3) Ordering conflicts: the user is asked to decide before any `$B` command, but the Aside probe in step 1 is itself a probe, and the target is opened with `goto` before the picker, while verification env setup comes after the decisions and may need a daemon stop. (4) The direct import line is vague: it uses one `--domain` per run, and it is unclear how multiple domains are handled. (5) Errors are only partly covered, and the Aside and CDP end-skill cases are clear. The platform boundaries and honest three-line reporting are well specified. Overall the workflow is mostly executable, but the consent and flag conflicts make it risky and ambiguous."},"latency_ms":4644,"stop_reason":"end_turn","usage":{"input_tokens":4382,"output_tokens":475},"cost_usd":0.013514} +{"config":"cookie-workflow","item":"p03","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"Strong ordering: consent questions come before any `$B` use, defaults are explicit, and the three-line report format separates copied cookies from sign-in evidence. Gaps: the first step depends on an external Aside probe; `--all` handling in the direct vs. picker path is vague (the one-domain-per-run rule contradicts multi-domain requests); and \"exactly three lines\" doesn't say how to handle failure details or errors. The error tables also omit some receipt reasons and recovery for a verification rejection mid-run. Overall it is mostly executable, with a few minor ambiguities."},"latency_ms":3013,"stop_reason":"end_turn","usage":{"input_tokens":5030,"output_tokens":224},"cost_usd":0.0123} +{"config":"cookie-workflow","item":"p02","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: Aside probe, collect all decisions up front, prepare the browser and daemon, import, then report from the receipt. Consent defaults and constraints are explicit (--all, --clear-storage, --verify-auth), and the tool distinguishes copied cookies from verified sign-in. The recovery tables and platform boundaries are clear. Gaps remain. (1) The picker versus direct import choice is slightly underspecified, for example when the browser is named but the profile is not. (2) The Aside probe command is deferred to another file. (3) Direct import with --verify-auth requires private env vars, and the order of `$B goto` versus daemon start is only implied. (4) Multi-domain imports (one --domain per run) lack guidance on how verification and storage reset behave across repeated runs. (5) Handling of the --all direct-import flow and the 'profile unreadable' answer is thin. (6) Error handling for a failed verification (not verified) offers no next step except manual sign-in. (7) The 'Mode: cdp' detection is clear, but the daemon-restart nuance for verification in the picker is not stated. Overall it is strong but has a few minor ambiguities."},"latency_ms":4738,"stop_reason":"end_turn","usage":{"input_tokens":4538,"output_tokens":424},"cost_usd":0.013316} +{"config":"cookie-workflow","item":"f04","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"Step 1 says to run the import in step 4 right away, but step 2 says to ask the user before any `$B` command, and step 3 says to open the target only after the import. BROWSER.md says direct import must first navigate to a matching page, so step 3's post-import `$B goto` contradicts it. The verification env vars must be set after `$B status` starts the daemon, yet BROWSER.md says they must be set before daemon startup. Error tables and platform limits are otherwise thorough."},"latency_ms":9610,"stop_reason":"end_turn","usage":{"input_tokens":4935,"output_tokens":187},"cost_usd":0.01174} +{"config":"cookie-workflow","item":"a06","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: the Aside probe, then up-front decisions, preparation, import, and an honest report. Consent defaults for --all, --clear-storage and --verify-auth are explicit. The distinction between copied cookies and verified sign-in is unambiguous, as are the platform boundaries and the error tables. Weaknesses: (1) The ordering is slightly contradictory. The user must set the env vars and may need `$B stop`, but the instructions don't say to check beforehand whether a daemon is running. (2) The Aside probe command is deferred to another file. (3) The direct-import syntax in SKILL.md is only described in prose; the example is in BROWSER.md. (4) The case where the user wants both --all and a domain target is only touched on briefly. (5) Behavior for `--all` is not fully specified: it needs consent but the target page requirement is unclear. (6) There is no recovery path for a missing selector beyond 'return to step 3'. (7) Some error codes, such as the failure of a partial import, are not mapped to agent actions. Overall it is solid and executable, with a few gaps."},"latency_ms":11109,"stop_reason":"end_turn","usage":{"input_tokens":4455,"output_tokens":397},"cost_usd":0.01288} +{"config":"cookie-workflow","item":"p01","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: Aside check, up-front consent questions, browser prep, import, then a three-line report that separates copied cookies from verified sign-in. BROWSER.md adds error tables and platform limits. Gaps: SKILL.md has the Aside probe run before the user questions, and it never says what to do if the answers conflict, such as --all with a domain. It also leaves unclear what happens if the picker link expires or the user abandons it. Direct import allows one domain per run but does not say how to handle several domains. Also, the section 4 pointer to \"Choosing a source and checking sign-in\" for receipt failure reasons is accurate, but \"partial\" handling is left undefined."},"latency_ms":3376,"stop_reason":"end_turn","usage":{"input_tokens":4918,"output_tokens":258},"cost_usd":0.012416} +{"config":"cookie-workflow","item":"p01","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: probe Aside, collect all decisions up front, prepare the daemon, import, then report in a fixed three-line format. Defaults and consent requirements are explicit, and so are the constraints on `--all`, `--clear-storage`, and verification. Positive sign-in evidence is clearly separated from copied cookies. BROWSER.md adds error tables, receipt failure reasons, and platform limits. There are a few gaps. Step 2 says to ask before running any `$B` command, but step 1 requires the Aside probe from browse/SKILL.md, which may itself use `$B`. The `--all` direct-import syntax is not shown in SKILL.md. Direct import with `--verify-auth` or `--clear-storage` has no example. It does not say how to handle several domains beyond running one `--domain` per run. For the picker, it does not say how to confirm completion beyond the user saying they are done. Failure handling for a `--clear-storage` rejection on a non-Chromium target says the import still proceeds, but it does not say what to report. The receipt format is clear, but the way the receipt is obtained is implicit. Recovery paths such as handoff and resume are documented, but they are scattered across the two files."},"latency_ms":4613,"stop_reason":"end_turn","usage":{"input_tokens":4521,"output_tokens":434},"cost_usd":0.013382} +{"config":"cookie-workflow","item":"p04","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered. It starts with the Aside probe, then asks for all decisions up front, prepares the browser, imports, and reports. It states consent and scope boundaries precisely: --clear-storage and --all each need explicit approval, --all cannot be combined with a domain or storage reset, and the daemon reads the env vars only at startup. It also separates copied cookies from sign-in evidence, with clear Imported/Storage reset/Sign-in report lines and a rule that counts and HTTP 200 never prove sign-in. Error tables and platform boundaries are concrete. Gaps: step 2 asks for the verification selector before any `$B` command, but the env vars are only set in step 3. The 'target' is one URL but multiple domains are allowed, and the direct import runs one domain per run, so how the target maps to multiple domains is unclear. The `--all` import has no command example. Answer 2 says to use the picker when the browser or profile is open, yet the picker also handles domains. The direct-import note about navigating to a page matching `--domain` is only in BROWSER.md. The picker's receipt flow and the timing of cookie-import-browser recovery steps are somewhat thin. The 'Aside probe' details are external. The error-table `profile_required` entry suggests `$B cookie-import-browser ` for the picker, which slightly conflicts with the picker command shown elsewhere. Overall it is strong but has a few ordering and edge-case ambiguities."},"latency_ms":19087,"stop_reason":"end_turn","usage":{"input_tokens":4539,"output_tokens":518},"cost_usd":0.014258} +{"config":"cookie-workflow","item":"a06","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered and explicit: Aside probe first, then collect all decisions up front, prepare the browser, import, and report from the receipt. Defaults and consent requirements are clear (verification no, clear-storage no with approval, --all with explicit consent). The distinctions between not checked, not verified, and verified are precise. Privacy and platform boundaries are stated, and BROWSER.md adds error tables, receipt failure reasons, and examples. Gaps and ambiguities remain. (1) Step 2 asks for decisions before any $B command, but Step 1 requires reading browse/SKILL.md, whose Aside probe may itself involve commands. (2) Step 3.2 refers to a 'Browser fallback' section that is not in the bundle. (3) Step 3 says to open the target before the import, yet --all has no target or domain, and the SKILL does not say how to handle that case. (4) Direct import allows only one --domain per run, but the picker's multi-domain behavior is not explained. (5) Handling of a failed or partial storage reset is thin: the user is told it is not rolled back, with no recovery path. (6) Cookie-import-browser with no browser defaults to comet, and the detection flow for 'no flags' is vague. (7) The verify-missing path says to return to step 3 but doesn't say whether to re-ask the user. (8) Receipt statuses like partial and empty have no prescribed actions. Overall the workflow is solid and mostly executable, with a few small holes."},"latency_ms":12260,"stop_reason":"end_turn","usage":{"input_tokens":4455,"output_tokens":526},"cost_usd":0.01417} +{"config":"cookie-workflow","item":"p02","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: Aside probe, collect all decisions up front, prepare the browser and daemon env, import, then report. Consent defaults and scoping are explicit (clear-storage, --all, verify-auth), and the three-line report distinguishes copied cookies from verified sign-in. Failure tables and platform limits are documented. Gaps and ambiguities remain. (1) Step 2 asks about --all, but step 4's direct import never explains how to run it, and --all conflicts with --domain, which step 4 says to pass per run. (2) 'One --domain per run' is unclear for multiple domains, including whether the target and cookies must match. (3) Direct-import verification relies on daemon env vars set before startup, while the picker path is less clear about whether the user supplies the selector privately. (4) The reference to the Aside probe and Browser fallback depends on external files. (5) Handling of `partial`, `empty`, or `failed` receipts is only partly covered: `decryption_failed` says to close the source browser, which sits uneasily next to the Chrome 136+ note. (6) 'Sign-in verification' when --verify-auth fails does not say what the next step is. (7) The Comet default is flagged only in BROWSER.md. These are modest, so the files are mostly executable."},"latency_ms":5080,"stop_reason":"end_turn","usage":{"input_tokens":4538,"output_tokens":456},"cost_usd":0.013636} +{"config":"cookie-workflow","item":"a02","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"Strong consent gating, a clear receipt-based three-line report, and explicit separation of copied cookies from verified sign-in. Gaps: the Aside probe command is deferred to another file, and if the user wants verification after a daemon is already running, the ordering is slightly awkward (set env, stop, then find $B). Direct import allows one domain per run but doesn't say how to handle multiple domains in the report. Chromium-only reset is stated, but the handling of a rejected non-Chromium reset is only implied. Recovery for the platform limits is concise but sufficient, e.g. headed manual sign-in only when a display is available."},"latency_ms":3162,"stop_reason":"end_turn","usage":{"input_tokens":3948,"output_tokens":235},"cost_usd":0.010246} +{"config":"cookie-workflow","item":"a02","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: ask first, never guess, prepare, import, report. Three-line receipt formatting and the distinction between copied cookies and verified sign-in are precise. Weak points: the Aside probe lives in an external file, and the Mode: cdp stop condition lacks its own recovery path. Direct import \"one --domain per run\" is unclear for multiple domains, since it doesn't say to rerun while keeping the target open. Recovery for failures is only partly specified: SKILL.md defers to BROWSER.md for \"other errors\", but BROWSER.md has no error tables. Step 2 asks for the target URL yet doesn't say what to do if it is on a different domain than the import domains."},"latency_ms":3307,"stop_reason":"end_turn","usage":{"input_tokens":3948,"output_tokens":263},"cost_usd":0.010526} +{"config":"cookie-workflow","item":"a02","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: the Aside probe comes first, then all user decisions before any `$B` command, then browser preparation, import, and an honest three-line report. It handles the key distinctions well. `--profile` takes a directory rather than a display name. Unreadable profiles count as unknown. Copied cookies are not sign-in proof. `--all` and `--clear-storage` need explicit consent. Platform and privacy limits are stated. Gaps remain. Verification env vars are set in step 3, after the user decisions, which is fine, but the picker's verification checkbox flow is only loosely tied to the daemon env setup. The exact direct-import syntax for the browser name is only shown in BROWSER.md. Step 4 doesn't say what to do when the import fails for other reasons, beyond pointing to BROWSER.md's 'error tables', which that excerpt doesn't contain. 'Use the picker when any of answers 1-2 is still open' leaves a mismatch with the target URL being needed before the picker. The sequence of asking questions, then stopping the daemon, then `$B status` is slightly tangled, because `$B stop` happens before `$B` is located. The receipt format is clear, but the mapping from receipt fields to 'Imported' and 'Storage reset' states is underspecified. Overall it is solid and mostly executable."},"latency_ms":11745,"stop_reason":"end_turn","usage":{"input_tokens":3551,"output_tokens":473},"cost_usd":0.011832} +{"config":"cookie-workflow","item":"f05","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"Only step 1 (the Aside probe and its READY/other branching) is concrete. Steps 2-5 and Platform boundaries are TODO/TBD placeholders, and the referenced BROWSER.md section is only a draft stub. Nothing explains how to select an authorized source browser, profile, or domain without guessing. Nothing covers optional authentication verification before mutation, explicit consent for a scoped storage reset, distinguishing copied cookies from positive sign-in evidence, or platform and privacy limits. The 'Report honestly' step is a vague one-liner. An agent could run the probe and branch, then would have to invent everything else, which is exactly the guessing the workflow is meant to prevent."},"latency_ms":3297,"stop_reason":"end_turn","usage":{"input_tokens":1007,"output_tokens":266},"cost_usd":0.004674} +{"config":"cookie-workflow","item":"a04","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"The workflow is thorough, with explicit consent gates, a clear receipt-based three-line report, honest sign-in semantics, and good platform and privacy boundaries. However, SKILL.md says `--profile` takes the display name, not the directory, while BROWSER.md says it takes the directory (`Profile 2`) and not the display name, and its example uses the directory. This contradiction on the core account-selection parameter risks wrong-profile or failed imports. Smaller gaps are that `--all` is missing from the direct-import steps, and handling of verification failure and multi-domain targets is thin."},"latency_ms":7469,"stop_reason":"end_turn","usage":{"input_tokens":4526,"output_tokens":887},"cost_usd":0.017922} +{"config":"cookie-workflow","item":"p05","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: probe Aside, collect all decisions up front, prepare the daemon, import, then report. Consent defaults are explicit for verification, storage clear and --all. The profile directory vs display name, the one-use picker link, and copied cookies vs sign-in evidence are all clearly distinguished. Error tables and recovery paths (handoff/resume, daemon restart for env vars) are concrete. Gaps: (1) The `--all` import syntax is never shown, and it conflicts with the 'one --domain per run' direct-import wording. (2) The decision tree for multi-domain imports is thin. For example, the target is a single page, but multiple domains may be imported. (3) Handling of `decryption_failed` ('close the source browser') sits uneasily with the Chrome 136+ note. (4) The 'Browser fallback' section of browse/SKILL.md is referenced but not included, as are the Aside probe details. (5) Whether to run `$B handoff` is not tied to the platform/display check in the main flow, and the partial/empty/failed outcomes are not mapped to actions. (6) Step 2 asks the user to be specific, but the instruction to read BROWSER.md is only 'for other errors', so it is unclear when to consult it proactively. Overall the workflow is solid but has some minor ambiguities."},"latency_ms":5268,"stop_reason":"end_turn","usage":{"input_tokens":4244,"output_tokens":472},"cost_usd":0.013208} +{"config":"cookie-workflow","item":"f05","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"Only step 1 (the Aside probe and its branch) is actionable. It says which root to use, which section to read, and what to do on READY versus any other result. Every other section is a placeholder: 'TODO: list the questions to ask', 'TBD', 'Details to be added', 'Let the user know how it went', and 'Platform boundaries: TBD'. The referenced BROWSER.md section is a draft stub with 'notes to come'. Nothing says how to select a source browser, profile or domain without guessing, how to configure authentication verification before mutation, how to get explicit consent for a scoped storage reset, how to tell copied cookies from positive sign-in evidence, or how to handle platform and privacy limits. The skill also sends the agent to BROWSER.md for details that do not exist, so the path is circular and dead-ends. An agent would have to invent the whole workflow, which risks the guessing and unconsented mutation the skill is meant to prevent."},"latency_ms":3681,"stop_reason":"end_turn","usage":{"input_tokens":1007,"output_tokens":344},"cost_usd":0.005454} +{"config":"cookie-workflow","item":"a02","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered and clear. It collects decisions up front, with explicit defaults and consent for risky options. It requires the profile directory rather than the display name, sets up verification before the daemon starts, and handles Aside and CDP modes with an early exit. The report format separates copied cookies from sign-in evidence, and it states that zero imports, cookie counts, and HTTP 200 never prove sign-in. BROWSER.md adds the error and platform context. Some gaps remain. The exact syntax for the direct import with multiple domains is not shown (one --domain per run, but how to run several is left open). The error tables that SKILL.md says BROWSER.md contains are not actually in the excerpt. How `--all` interacts with the target and the direct-import flow is unclear. The recovery steps for a failed reset or an expired picker link are only partly defined. The ordering issue of `$B goto` versus `--all` or the picker is minor. Recovery after a Chrome 136+ block is only 'sign in manually when a display is available'. The Aside probe is delegated to another file. Overall it is strong but has a few missing decisions."},"latency_ms":5537,"stop_reason":"end_turn","usage":{"input_tokens":3551,"output_tokens":401},"cost_usd":0.011112} +{"config":"cookie-workflow","item":"a05","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"Detailed on scope, consent, verification, and platform limits. But ordering is contradictory: section 1 says run the probe first and stop on READY, while the SETUP block and a later \"status Mode: cdp\" check overlap, and the probe's \"fallback handling\" is undefined. Section 2 says to \"open the known target\" before the target is chosen. Direct import lacks an explicit sequence for when to ask about options, and how --verify-auth is passed to the command is unclear. Reporting guidance is good."},"latency_ms":2801,"stop_reason":"end_turn","usage":{"input_tokens":5326,"output_tokens":199},"cost_usd":0.012642} +{"config":"cookie-workflow","item":"a03","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"Strong on consent, receipt reporting, and platform limits. But key ordering conflicts: SKILL.md says to set the verify env vars in the shell and rerun, while BROWSER.md says they must be in the daemon environment before startup, so a later CLI call won't reconfigure it. The recovery loop 'return to step 3' doesn't say to restart the daemon. Also unclear: the direct-import syntax for `--verify-auth`, and the picker/direct choice when only part of the answers is known."},"latency_ms":2358,"stop_reason":"end_turn","usage":{"input_tokens":4899,"output_tokens":195},"cost_usd":0.011748} +{"config":"cookie-workflow","item":"a06","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: probe Aside, collect every decision up front, prepare the browser, import, then report from the receipt. Consent is explicit and tightly scoped for --clear-storage and --all. Copied cookies, 'not checked', 'not verified' and 'verified' are clearly distinguished. Errors, receipt failure reasons and platform limits are tabulated, with next steps. Weaknesses: (1) The Aside probe lives in another file, browse/SKILL.md, which is documented separately. (2) The direct-import syntax for adding --verify-auth and --clear-storage is only partly shown, and --all has no example. (3) There is a minor ordering tension. Step 2 says to ask before any $B command, yet step 3 has the user set env vars and possibly run `$B stop` first, and the daemon-restart flow after a missing-variable rejection is only loosely specified. (4) The BROWSER.md example runs `$B goto` with no domain-matching guidance for multiple domains. The 'one --domain per run' rule is also unclear when several domains are requested, and so is how to handle a partial import. (5) The headless fallback after a failure is covered, but there are few concrete steps for decryption retries. (6) It is unclear how to confirm the picker is finished beyond the user telling the agent. Overall the workflow is strong, with some gaps at edge cases."},"latency_ms":4744,"stop_reason":"end_turn","usage":{"input_tokens":4455,"output_tokens":485},"cost_usd":0.01376} +{"config":"cookie-workflow","item":"a06","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow orders steps well: gather all decisions first, set up the browser, import, then report from the receipt. It separates 'not checked' from 'not verified' and 'verified'. Weaknesses: Aside probe details are external; the picker-vs-direct rule is split (\"any of answers 1-2 open\"), which leaves the domain-less `--all` path unclear; the \"one --domain per run\" rule for multiple domains is terse; and the goto-before-import step is not tied to target consent. It is unclear how the agent confirms the picker finished, and `--all` direct usage lacks an example. Recovery for failures is documented (handoff/resume, error tables)."},"latency_ms":3283,"stop_reason":"end_turn","usage":{"input_tokens":4852,"output_tokens":249},"cost_usd":0.012194} +{"config":"cookie-workflow","item":"p02","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"Strong consent gating, explicit receipt wording, and a clear daemon-restart order for verification. Gaps: the Aside probe and `$B` discovery are delegated to external files; the picker-vs-direct choice is unclear when the user supplies a browser and domain but no profile; `--all` has no direct flow; the verify-before-import ordering is only implied; and the decision tree for `partial` or `failed` receipts is thin (e.g., `decryption_failed` says to close the source browser, conflicting loosely with the Chrome 136+ note)."},"latency_ms":2773,"stop_reason":"end_turn","usage":{"input_tokens":4935,"output_tokens":212},"cost_usd":0.01199} +{"config":"cookie-workflow","item":"a03","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"The workflow is strong on consent, scoping, and honest reporting: it asks for the browser, profile directory, domains, target, verification, storage reset and --all before any mutation. It separates copied cookies from verified sign-in, and the platform and privacy boundaries are covered well. However, several points are contradictory or missing, which an agent would have to resolve by guessing.\n\n1. Verification env vars conflict. SKILL.md says to set them 'in the shell that runs $B' and then rerun, while BROWSER.md says they must be set in the daemon environment before startup, and that setting them on a later CLI call does not reconfigure an existing daemon. SKILL.md never says to restart the daemon. The missing-config recovery ('return to step 3') could therefore loop forever.\n2. Step 2 says the user may decide on verification after seeing the import. But verification requires configuration before startup, and the picker checkbox path is unclear.\n3. Domains. The direct import takes one --domain per run, while the user may name several domains, and the goto target is only one domain. How to handle several domains is not defined.\n4. Picker versus direct. The `--all` flag, the user's 'done' signal and the picker's handling of clear-storage are underspecified. SKILL.md also does not say how --all or --clear-storage work in the picker.\n5. Referenced sections. The Aside probe and the Browser fallback live in a different file that is not shown, which the bundle says is acceptable.\n6. Report and recovery. The receipt's report format is clear. Recovery steps are scattered between the two files: the handoff steps, the `decryption_failed` advice to close the source browser (which conflicts somewhat with the Chrome 136 note), and the unreadable-profile case. Some of the receipt's status values are also not mapped to the three report lines."},"latency_ms":5750,"stop_reason":"end_turn","usage":{"input_tokens":4502,"output_tokens":629},"cost_usd":0.015294} +{"config":"cookie-workflow","item":"f01","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The workflow has serious contradictions and gaps. Step 2 says 'Ask these together' but item 1 says 'Do not ask' and tells the agent to pick the browser and profile itself. It also says to pick the profile with the most cookies, while BROWSER.md says ambiguity requires a user choice and only a sole relevant profile is auto-selected. This directly conflicts with the stated goal of not guessing an account. 'Use direct import' also conflicts with the picker flow. The default is Comet, which is macOS-only, and the fallback is 'whichever supported browser is installed', which is vague. Step 2 says to ask the user for the selector and identity privately, but the agent cannot know how to determine the most cookies without running commands, and all of that must happen before any `$B` command. The ordering is circular. Step 3 starts the daemon only after the user sets the env vars, which is fine, but `$B stop` handling is unspecific. Step 4 says to run one `--domain` per run, with no guidance on multiple domains and no `--all` command syntax. The receipt-based report is clear, and the verification semantics, storage-reset scope, and platform boundaries are well specified. Error tables are helpful. Overall, the contradictions on profile selection and consent undermine safe execution, even though many details are precise."},"latency_ms":25658,"stop_reason":"end_turn","usage":{"input_tokens":4509,"output_tokens":458},"cost_usd":0.013598} +{"config":"cookie-workflow","item":"f05","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"Only Step 1 (the Aside probe and the choice of gstack root) is concrete. Every other part is a placeholder: the TODO for user questions, 'TBD' for preparing the browser, 'Details to be added' for the import, a vague 'Let the user know how it went' for reporting, and 'TBD' for platform boundaries. The BROWSER.md section it points to is only a draft note that says content is 'to come'. Nothing covers source browser, profile, or domain selection, so an agent would have to guess. Nothing covers optional authentication verification, scoped storage-reset consent, telling copied cookies apart from sign-in evidence, or platform and privacy recovery. The section reference is also circular, because SKILL.md defers to BROWSER.md and BROWSER.md defers to later work. Clarity is slightly above the floor because the first step is clear and the file structure is readable. Completeness and actionability are minimal because the workflow cannot be executed beyond the Aside check."},"latency_ms":3445,"stop_reason":"end_turn","usage":{"input_tokens":1007,"output_tokens":355},"cost_usd":0.005564} +{"config":"cookie-workflow","item":"a06","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered. It checks Aside first, collects all decisions before any `$B` command, prepares the browser, imports, and then reports honestly from the receipt. Consent defaults and constraints are explicit: `--all` cannot combine with a domain or `--clear-storage`, reset is Chromium-only and can leave partial effects, and 'not checked', 'not verified' and 'verified' are distinguished. Recovery paths are well covered by the error tables and the handoff/resume steps.\n\nThere are small gaps. SKILL.md says the daemon reads the environment variables only at startup and tells the agent to `$B stop` first, but it does not say when to do that relative to step 3's ordering. The 'Browser fallback' and Aside probe details live in an external file. The direct-import syntax for the `--all` and `--clear-storage` flags is only implied. There is slight ambiguity about 'one `--domain` per run' versus multiple domains, and about how the `--verify-auth` receipt is read in direct mode. Unreadable-profile handling and the `decryption_failed` guidance ('close the source browser') sit uneasily next to the Chrome 136+ note. Overall the workflow is strong and executable."},"latency_ms":4747,"stop_reason":"end_turn","usage":{"input_tokens":4455,"output_tokens":435},"cost_usd":0.01326} +{"config":"cookie-workflow","item":"a01","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: it probes Aside first, gathers all decisions up front, prepares the browser, imports, and reports honestly. It is consistent about consent, privacy, the three-line report, and the rule that copied cookies are not sign-in evidence. The BROWSER.md error and receipt tables cover recovery. Gaps: (1) the Aside probe lives in an external file, and `$B` discovery in `browse/SKILL.md`; the preamble says to ignore missing external tools. (2) Step 2 asks for decisions 'before running any `$B` command', yet the Aside probe may involve one, and step 3 sets env vars after step 2. The ordering is mostly workable. (3) Direct import allows one `--domain` per run, but multiple domains are not clearly handled beyond repeated runs. The report aggregation for multi-run receipts is undefined. (4) Verification is only partly specified for the picker versus direct import, e.g. when the user wants `--all`, which is not mentioned in step 4's direct import. (5) The default `comet` rule is mentioned only in passing. (6) Recovery branches like `decryption_failed` ('close the source browser') are slightly at odds with the Chrome 136+ note, but they are consistent enough. Overall it is strong but has minor ambiguities."},"latency_ms":4828,"stop_reason":"end_turn","usage":{"input_tokens":4403,"output_tokens":463},"cost_usd":0.013436} +{"config":"cookie-workflow","item":"p04","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: Aside probe, then collecting every decision up front, then preparing `$B`, importing, and reporting. Consent boundaries are explicit. `--clear-storage` and `--all` default to no and need approval, and the exact scope of the storage reset is stated. Copied cookies are kept separate from sign-in evidence: verified, not verified and not checked each have a definition, and counts and HTTP 200 never count as proof. Error and recovery tables cover the failure keys and codes, and platform and privacy limits are stated.\n\nSome gaps and tensions remain. The `--verify-auth` environment variables are set in step 3 but the daemon-restart condition is easy to miss. Step 3 has `$B goto` before import, yet the picker flow does not say whether the domain must be known in advance. The picker versus direct import choice is spelled out only briefly. The syntax for `--verify-auth` in direct mode is not shown. There is no guidance on what to do when the user declines to give a profile directory, or how to find profile directories when the user only knows display names, other than the picker. Multi-domain imports are handled by running once per domain, but the aggregate report format for them is vague. `--all` is not covered in the direct-import flow. A rejected reset or a partial result has no explicit recovery step. The Aside check depends on an external probe whose result handling is only partly specified. These gaps are minor, and the agent can mostly execute the workflow."},"latency_ms":5487,"stop_reason":"end_turn","usage":{"input_tokens":4539,"output_tokens":508},"cost_usd":0.014158} +{"config":"cookie-workflow","item":"p01","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: Aside probe, up-front consent gathering, daemon setup (env vars before start, `$B stop`), import, then a three-line honest report. It separates copied cookies from verified sign-in, and it names the receipt statuses and failure-reason tables. Guardrails are explicit: `--all` and `--clear-storage` need consent, there is no default-browser consent, and the platform limits are stated. Gaps and ambiguities remain. (1) `--all` is asked about in step 2 but never appears in the direct-import or picker instructions, and it conflicts with the 'one `--domain` per run' guidance and the need for a target URL. (2) The target and navigation requirements for `--all` are unclear. (3) The ordering is slightly awkward: the user picks verification in step 2, the env vars are set in step 3, and the picker handles verification itself. (4) The Aside probe is delegated to another file. (5) It is not stated what to do when a multi-domain import is partially rejected, or when the Chromium-only reset is rejected. (6) The mapping from receipt fields to the three report lines (storage reset, sign-in) is implied rather than spelled out. (7) The error tables omit some cases, such as an expired picker link or a verification failure. (8) The Windows fallback to `$B handoff` is only briefly tied to the main flow. Overall the workflow is strong but has some missing decisions."},"latency_ms":4984,"stop_reason":"end_turn","usage":{"input_tokens":4521,"output_tokens":496},"cost_usd":0.014002} +{"config":"cookie-workflow","item":"a06","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well sequenced: probe Aside, collect all decisions up front, prepare the daemon and environment variables, run the import, then report from the receipt. It is explicit about consent defaults, the `--profile` directory versus display name, the one-use picker link, and what counts as verified versus not checked. Receipt failure reasons and import errors come with next steps, and the platform boundaries are clearly stated.\n\nThere are small gaps:\n- The Aside probe and the `$B` fallback are delegated to `browse/SKILL.md`, which is not in the bundle, and that is acceptable.\n- The `--all` flow is never shown in the direct-import steps, though it is covered by question 5 and the BROWSER.md note.\n- The \"one `--domain` per run\" rule conflicts slightly with answer 2 asking for multiple domains. The intended behavior is probably repeated runs, but this is not stated.\n- It is not specified what to do when a failed `--clear-storage` leaves partial state. It only says the failure is not rolled back.\n- The `$B handoff` fallback is introduced in BROWSER.md without explicit consent or ordering guidance in SKILL.md.\n- Step 3 tells the agent to run `$B stop` if a daemon is already running, but not how to check for one.\n- Step 3 is also somewhat out of order: verification env setup comes before `$B` is located."},"latency_ms":6315,"stop_reason":"end_turn","usage":{"input_tokens":4455,"output_tokens":497},"cost_usd":0.01388} +{"config":"cookie-workflow","item":"p03","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: probe Aside, ask all five questions up front, set up the browser, import, then report in a fixed three-line format. Consent for --clear-storage and --all is explicit, and the sign-in outcomes (verified, not verified, not checked) are clearly separated from cookie counts. Error tables, platform limits and privacy rules are present. Gaps: the Aside probe lives in a different file (browse/SKILL.md) that is not shown. The ordering is a bit tangled, because verification env vars are set in step 3 but the daemon-restart implications depend on step 2 answers. The target URL is opened before the picker, yet the picker lets the user choose domains and the target domain may not match. The file says one --domain per run but does not say how to handle multiple domains, or what to do when --all is chosen with no target. The direct-import exact flag syntax for --verify-auth and --all is only implied. The receipt format and how to read it are only loosely specified. Handling of the Dia and Comet defaults is clear enough. Overall it is solid but has some small ambiguities and missing decision branches."},"latency_ms":11529,"stop_reason":"end_turn","usage":{"input_tokens":4633,"output_tokens":392},"cost_usd":0.013186} +{"config":"cookie-workflow","item":"p05","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: consent is gathered up front, defaults are set, receipt statuses are defined, and platform limits are given. Two gaps remain. First, the Aside probe is delegated to browse/SKILL.md, which is outside the bundle. Second, the verify-auth setup order is slightly tangled: the user's choice in step 2 comes before daemon env setup in step 3, and step 4 sends the agent back to step 3 on a missing-variable rejection. Several error paths (e.g. picker expiry mid-use, failed reset follow-up) are only thinly specified. Overall it is executable with minor ambiguity."},"latency_ms":3055,"stop_reason":"end_turn","usage":{"input_tokens":4641,"output_tokens":238},"cost_usd":0.011662} +{"config":"cookie-workflow","item":"a01","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"Strong ordering: ask all decisions first, then prepare the daemon, import, and report in a fixed three-line format. Consent defaults and the copied-cookies vs. verified distinction are explicit. Gaps: the --all flow is not fully specified (the target/goto requirement is unclear), and a missing-variable reject sends the agent back to step 3 without a clear restart rule. The Aside probe and 'Browser fallback' are referenced externally. The `profile_required` recovery suggests retrying with a profile, which could encourage guessing instead of asking the user. Overall it is well scoped and usable."},"latency_ms":3721,"stop_reason":"end_turn","usage":{"input_tokens":4800,"output_tokens":229},"cost_usd":0.01189} +{"config":"cookie-workflow","item":"p01","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow orders decisions, setup, import and reporting clearly, and it separates 'not checked', 'not verified' and 'verified'. Consent defaults and the --all and --clear-storage restrictions are explicit. Gaps: Aside probe wording is delegated to an external file. The Aside check runs before the user is asked anything. The direct import ordering is slightly contradictory: the target is opened only after answers, and the domain-matching rule is spelled out only in BROWSER.md. Receipt-based errors like decryption_failed advise 'close the source browser', which sits awkwardly next to the Chrome 136 note. Failed-reset recovery and the 'partial' receipt next step are underspecified."},"latency_ms":3052,"stop_reason":"end_turn","usage":{"input_tokens":4918,"output_tokens":262},"cost_usd":0.012456} +{"config":"cookie-workflow","item":"f02","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":3,"reasoning":"The workflow is detailed on profile selection, verification, and honest reporting, but it contradicts itself in several places that matter. (1) Step 2.4 says 'Do not ask; always clear it', which conflicts with the stated need for explicit consent for a precisely scoped storage reset. Clearing wipes the browser's stored site data, and the user is never told that or asked about it. (2) Step 2.5 says to add `--all` whenever the user is unsure, with no need to ask. BROWSER.md says `--all` cannot accompany `--domain` or `--clear-storage`, and it imports every cookie in the profile, which conflicts with the domain scoping and privacy constraints. Together, 2.4 and 2.5 produce an invalid command combination with no resolution. (3) Step 2 says to ask everything before any `$B` command, but step 1 and step 3 run probes first. The ordering is also confusing: the verification env variables are set in step 3, after the user's decision in step 2. (4) The picker path does not say how clear-storage is applied. (5) The direct import says one `--domain` per run but gives no loop for multiple domains. The error handling, receipt reporting, platform boundaries, and the distinction between copied cookies and verified sign-in are good. The failure-reason tables are useful, but some details are missing, such as what to do with a `partial` receipt."},"latency_ms":4911,"stop_reason":"end_turn","usage":{"input_tokens":4382,"output_tokens":479},"cost_usd":0.013554} +{"config":"cookie-workflow","item":"f02","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":2,"completeness":2,"actionability":2,"reasoning":"Contradictions undermine the workflow. SKILL.md says to always add --clear-storage without asking, yet the task requires explicit consent for precisely scoped storage reset. It also says to add --all whenever the user is unsure, but BROWSER.md says --all cannot accompany --clear-storage or --domain, so the two flags conflict. Additionally, the setup order is unclear: env vars are set before $B is located, and the Aside probe comes before any user questions. Handling of cookie-import errors, receipts and platform limits is otherwise good."},"latency_ms":3160,"stop_reason":"end_turn","usage":{"input_tokens":4779,"output_tokens":209},"cost_usd":0.011648} +{"config":"cookie-workflow","item":"p05","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"Strong consent gating (ask all decisions before any $B command), explicit three-line report, and clear distinction between copied cookies and verified sign-in. Gaps: when verification is wanted, the env-var step (3.1) comes after the user is asked, but the ordering of Aside probe versus $B is fine. The picker-versus-direct choice is ambiguous when only the domain or browser is missing. Multiple domains are handled by one run per domain, but receipt aggregation for the report is unspecified. The Aside probe depends on external browse/SKILL.md, which is acceptable. Error recovery is delegated to BROWSER.md tables and is mostly clear, though `decryption_failed` says to close the source browser while the platform text implies that closing may not help on Chrome 136+."},"latency_ms":3654,"stop_reason":"end_turn","usage":{"input_tokens":4641,"output_tokens":285},"cost_usd":0.012132} +{"config":"cookie-workflow","item":"p03","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: probe Aside, ask all five questions up front, set up $B, import, then give a three-line report. Consent defaults and boundaries are explicit: --clear-storage and --all each need explicit approval and default to no, and the Chromium-only limit and partial-reset risk are stated. The report separates copied cookies from sign-in evidence clearly (not checked / not verified / verified). BROWSER.md adds the error tables, platform limits, and recovery paths. There are some gaps. Step 2 asks about the target page before the domains are settled, and it never says what to do if the user wants several domains with direct import, which allows one --domain per run. It does not say whether to re-ask the questions after a rejection. The Aside probe lives in another file's BROWSER SETUP section, and the SKILL does not say what to do if that section or the probe is missing, though the bundle says external files are documented separately. The --all rule is slightly confusing: SKILL says it cannot be used 'together with a domain', but the direct import and picker steps do not show the --all command. The 'exactly three lines' report format also conflicts a little with how failure reasons and recovery are reported. The 'stop' paths on Mode: cdp and on READY are clear. Overall the workflow is strong, with minor ambiguities."},"latency_ms":8630,"stop_reason":"end_turn","usage":{"input_tokens":4633,"output_tokens":456},"cost_usd":0.013826} +{"config":"cookie-workflow","item":"a04","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"The workflow is well structured: it gathers decisions up front, sets the env vars before daemon start, requires explicit consent for destructive options, separates copied cookies from sign-in evidence, and defines a clear three-line report. But SKILL.md says `--profile` takes the display name, not the directory, while BROWSER.md says it takes the directory (`Default`, `Profile 2`), not the display name, and its examples and error table use directories. This contradiction on a core argument can cause failed or wrong-profile imports in direct mode. Smaller gaps: no guidance for multi-domain picker runs, and thin handling of partial or failed reset results."},"latency_ms":7862,"stop_reason":"end_turn","usage":{"input_tokens":4526,"output_tokens":883},"cost_usd":0.017882} +{"config":"cookie-workflow","item":"a02","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"Strong: decisions are gathered up front, defaults and consent are explicit, and the receipt-based three-line report separates copied cookies from sign-in evidence. Gaps: the 'Read that same root's BROWSER.md' step refers to error tables and examples that are not in the excerpt. Picker vs direct is only loosely defined, and the direct flow is vague when several domains are named (one `--domain` per run). The `--all` direct syntax is not shown. Also, the order of $B stop, status, and then goto is workable but implicit. Failure paths after a failed reset or an expired link are covered only briefly."},"latency_ms":2890,"stop_reason":"end_turn","usage":{"input_tokens":3948,"output_tokens":230},"cost_usd":0.010196} +{"config":"cookie-workflow","item":"f05","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"Only Section 1 (the Aside probe check) is concrete. Section 2 is a TODO, Section 3 is TBD, Section 4 says details are to be added, Section 5 is a single vague sentence, and Platform boundaries is TBD. The referenced BROWSER.md section is only a draft stub. Nothing explains how to choose a browser, profile, or domain without guessing. Nothing covers optional authentication verification before mutation, explicit consent for scoped storage reset, telling copied cookies apart from positive sign-in evidence, or recovery within platform and privacy limits. The reference loops back and forth between the two files without ever providing content. An agent could run the Aside probe and exit on READY. Otherwise it would have to guess every decision, which is exactly what the workflow aims to prevent."},"latency_ms":3314,"stop_reason":"end_turn","usage":{"input_tokens":1007,"output_tokens":297},"cost_usd":0.004984} +{"config":"cookie-workflow","item":"a02","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured. Step 2 collects every decision before any `$B` command. Defaults, consent requirements and flag constraints are explicit: `--all` cannot be combined with a domain or `--clear-storage`, reset is Chromium-only, and a failed reset is not rolled back. The three-line report separates copied cookies from verified sign-in, and the platform and privacy boundaries are stated. BROWSER.md fills in the profile directory rules, example commands, verification semantics and platform limits.\n\nThere are gaps:\n- The 'Browser fallback' section of browse/SKILL.md, the Aside probe and the error tables are external to this bundle.\n- Direct import allows one `--domain` per run, but the instructions do not say how to handle several domains. Repeated runs would need to preserve the target and the verification context.\n- They do not say which domain's target applies when the user names several domains.\n- Direct `--all` usage is not shown in SKILL.md.\n- The `--verify-auth` rejection recovery says to return to step 3, but a daemon that is already running needs `$B stop` first. This is implied rather than stated at that point.\n- Receipt failure reasons are deferred to BROWSER.md, which only mentions them and does not actually list them.\n- The 'headed manual sign-in' fallback is mentioned but not spelled out.\n- The Aside probe is checked before consent is gathered. That is acceptable but slightly out of order, because `$B status` CDP mode is checked only after the questions are asked."},"latency_ms":5015,"stop_reason":"end_turn","usage":{"input_tokens":3551,"output_tokens":538},"cost_usd":0.012482} +{"config":"cookie-workflow","item":"a01","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: it asks the user for all decisions up front, does not guess or treat the Comet default as consent, and orders the steps sensibly. It distinguishes copied cookies from verified sign-in, gives a three-line receipt format, and has clear error and failure-reason tables plus platform boundaries. Weaknesses: (1) The Aside probe and the browse/SKILL.md Browser fallback are external and not shown. (2) The ordering is slightly awkward: the picker is used when browser or profile is open, yet step 3 requires goto to the target first and the verification env vars must be set before the daemon starts. (3) Direct import allows one domain per run, but multi-domain handling is only implied. (4) Verification settings are asked for in step 2 but their absence is only caught as a rejection. (5) How to obtain the receipt when using the picker is vague, as are the Chromium-detection step before --clear-storage and what to do if the user declines every option after a failure. (6) Platform fallbacks such as handoff and resume are documented only in BROWSER.md. Overall it is strong, with minor gaps."},"latency_ms":4170,"stop_reason":"end_turn","usage":{"input_tokens":4403,"output_tokens":398},"cost_usd":0.012786} +{"config":"cookie-workflow","item":"f04","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The bundle has detailed content on scoping, consent, receipts, honest reporting and platform limits, but its ordering contradicts itself in ways that block clean execution. (1) Step 1 says to run the step 4 import \"right away\" for any non-READY result, then return to step 2 for decisions. Step 2 says to ask everything before any `$B` command, and step 3 says to find `$B` and check `$B status` for CDP mode before importing. (2) Step 3.1 has the user set the auth env vars after `$B status` starts the daemon and forbids restarting it. BROWSER.md says the variables must be in the daemon environment before startup, and that setting them later does not reconfigure a running daemon. Verification therefore cannot work as written. (3) Step 3.4 says to open the target URL after the import. BROWSER.md says direct import must first navigate to a page matching the domain. (4) The `--all` consent path has no import command in SKILL.md. The error tables and receipt reasons are useful, but these ordering conflicts leave an agent unsure of the correct sequence."},"latency_ms":13617,"stop_reason":"end_turn","usage":{"input_tokens":4538,"output_tokens":769},"cost_usd":0.016766} +{"config":"cookie-workflow","item":"f01","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The workflow has serious internal contradictions. Step 2 says 'Ask these together' and 'Get every decision from the user first', but item 1 says 'Do not ask' and tells the agent to pick the default browser (Comet) and the profile with the most cookies. That is exactly the guessing of an account that the task says to avoid, and BROWSER.md says ambiguity requires a user choice and that the Comet default is for CLI compatibility, not a recommendation. Comet is macOS-only, so defaulting to it on other platforms is also wrong. Picking the profile with the most cookies is a heuristic that can select the wrong account, and the instruction does not say how to measure cookie counts without reading cookies. 'Use direct import... The picker is only needed if no profile has cookies' conflicts with BROWSER.md's profile_required and ambiguity rules. The 'one --domain per run' rule collides with multiple domains and with --all, which is not described in the direct-import step. Consent for --clear-storage and --all is covered, and the three-line report and the 'copied cookies are not sign-in proof' distinction are clear. Errors are documented in reasonable tables. Gaps: the flow when verification settings are missing is only partly specified, how to detect the Aside probe is deferred to another file, and the ordering of $B goto against the Chromium check for the storage reset is not spelled out. The overall flow is mostly executable, but the browser/profile selection contradiction would make an agent guess an account."},"latency_ms":5358,"stop_reason":"end_turn","usage":{"input_tokens":4509,"output_tokens":518},"cost_usd":0.014198} +{"config":"cookie-workflow","item":"f02","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The workflow contradicts itself on consent and scoping. SKILL.md step 2.4 says never ask and always clear storage with `--clear-storage`, which conflicts with the requirement for explicit consent to a precisely scoped reset. Step 2.5 says to add `--all` when the user is unsure, but BROWSER.md says `--all` cannot be combined with `--domain` or `--clear-storage`, so the instructions can't be run as written. Error handling and the platform limits are well covered, but the ordering and flag conflicts make safe execution unreliable."},"latency_ms":3222,"stop_reason":"end_turn","usage":{"input_tokens":4779,"output_tokens":215},"cost_usd":0.011708} +{"config":"cookie-workflow","item":"f01","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The workflow contradicts itself on its central requirement. Step 2 says to get every decision from the user first, but its item 1 says not to ask about the source browser and profile, and tells the agent to pick the default (Comet) and the profile with the most cookies. That conflicts with the stated goal of selecting an authorized source without guessing an account. BROWSER.md says that without `--profile`, only a sole relevant profile is selected and ambiguity requires a choice. Picking the profile with the most cookies is an unauthorized account guess. Comet is also macOS-only, and the BROWSER.md note says the omitted-browser default is for CLI compatibility, not a recommendation.\n\nStep 2 also says to ask before any `$B` command, yet the instruction to pick the profile with the most cookies requires inspecting the profiles, and the instructions never say how to do that. The direct-import text says to use one `--domain` per run, but the user may supply several domains, and the multi-domain flow is not defined. `--all` has no stated import syntax beyond the flag. Direct import also needs the domain-matching page open first, which the instructions mention but do not tie to the target.\n\nThe strengths are the following:\n- Clear consent gates for `--clear-storage` and `--all`.\n- Clear verification semantics, with the receipt's three-line report format.\n- Honest distinctions between copied cookies and sign-in evidence.\n- Error tables and platform boundaries.\n\nThe weaknesses are the following:\n- Several decision points are missing, such as what to do when Aside is absent and on other platforms.\n- The `$B stop` step is placed before the `$B` setup that finds `$B`.\n- The Aside probe is delegated to another file.\n- The sequencing of the verification environment variables relative to daemon startup is awkward."},"latency_ms":5840,"stop_reason":"end_turn","usage":{"input_tokens":4509,"output_tokens":633},"cost_usd":0.015348} +{"config":"cookie-workflow","item":"f03","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The core flow is well ordered: probe for Aside, ask the user for decisions up front, prepare `$B`, import, then report. Profile-directory handling, `--clear-storage` scope and consent, `--all` restrictions and platform limits are all stated precisely. The gaps are in verification and sign-in evidence. SKILL.md tells the agent to add `--verify-auth` only if the user said yes in step 2, but step 2 never asks about verification. The 'missing verification settings' error tells the agent to name 'which variable is missing', yet neither file defines those variables or how to configure them, and the picker 'ticks verification if they asked' has the same gap. Sign-in reporting is weak: it counts cookies copied plus an HTTP 200 on reload as 'signed in', which is not positive evidence of authentication, and there is no 'unknown' outcome even though a verification result could be available. Other gaps: multi-domain direct imports are only hinted at ('one `--domain` per run'), and the picker's domain choice does not say how it relates to the target URL. Recovery paths such as `handoff` exist but are scattered across the two files."},"latency_ms":8433,"stop_reason":"end_turn","usage":{"input_tokens":3986,"output_tokens":927},"cost_usd":0.017242} +{"config":"cookie-workflow","item":"f04","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The pieces are detailed: consent items, the receipt format, the error tables and the platform limits. The ordering contradicts itself in ways that break execution. (1) Step 1 says to run the step 4 import right away and then ask the user in step 2. Step 2 says to ask everything before any `$B` command and never to guess. (2) Step 3 says to set the verification env vars after `$B status` starts the daemon and not to restart it. BROWSER.md says they must be in the daemon environment before startup, and that setting them later does not reconfigure a running daemon. Verification can therefore never be configured, and the step 4 \"return to step 3\" recovery loops. (3) Step 3 says to open the target after the import. BROWSER.md says direct import first needs navigation to a page matching the domain. (4) The Mode: cdp check in step 3 comes after step 1 has already told the agent to import. (5) The direct-import instructions do not cover `--all`, and the multiple-domain case is only partly handled. The agent cannot follow this workflow safely without resolving these conflicts itself."},"latency_ms":13984,"stop_reason":"end_turn","usage":{"input_tokens":4538,"output_tokens":740},"cost_usd":0.016476} +{"config":"cookie-workflow","item":"p06","split":"heldout","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"Strong consent gating, a clear receipt report format, and well-defined verification semantics. Gaps: Step 2 says ask all decisions before any `$B` command, but the Aside probe runs earlier and the `$B status` CDP check comes after the questions, so some answers may be wasted. Missing-variable errors return to step 3 without explaining the daemon restart. The picker-versus-direct choice is clear, but the `--all` flow is underspecified in the execution steps."},"latency_ms":2948,"stop_reason":"end_turn","usage":{"input_tokens":5015,"output_tokens":185},"cost_usd":0.01188} +{"config":"cookie-workflow","item":"p08","split":"heldout","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: stop conditions, batched decisions before any `$B` command, setup, import, recovery, and a three-line report. Consent and scoping are explicit. Gaps: picker-versus-direct routing is slightly ambiguous when `--all` is chosen. The Aside probe is referenced externally. The 'one --domain per run' instruction is unclear for multiple domains. Direct import is told to add `--verify-auth` while setup order is separate. Overall it is highly usable."},"latency_ms":2995,"stop_reason":"end_turn","usage":{"input_tokens":4984,"output_tokens":193},"cost_usd":0.011898} +{"config":"cookie-workflow","item":"f06","split":"heldout","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The overall flow is readable, but the key mechanics are vague. SKILL.md never says how to start gstack's browser, which commands or flags to use for the import, or how to configure verification settings. Step 3 has the user set up verification privately but gives no concrete instructions, and BROWSER.md says to configure it before the browser starts, which is only loosely connected to step 3. The consent scoping for \"copy every cookie\" is thin, and the instructions don't explain how to find profile directories, read the receipt, or choose between the interactive and direct forms. Error recovery amounts to \"sign in manually\"."},"latency_ms":3121,"stop_reason":"end_turn","usage":{"input_tokens":1811,"output_tokens":227},"cost_usd":0.005892} +{"config":"cookie-workflow","item":"p07","split":"heldout","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is ordered and explicit: Aside probe, gather decisions first, prepare the daemon, import, then a three-line report. It handles consent defaults, the profile-directory rule, and the copied-cookies vs. verified distinction well. Gaps: Aside and $B status are checked before the user's choices are fully used, and the 'Mode: cdp' check lands after questions were already asked. The direct-import example does not say how to run --all, and the 'one --domain per run' rule for several domains is not explained further. The partial/failed receipt next steps and the case where the user wants verification but the daemon is already running are only loosely covered. Overall it is strong and executable, with minor ambiguity."},"latency_ms":3200,"stop_reason":"end_turn","usage":{"input_tokens":5130,"output_tokens":256},"cost_usd":0.01282} +{"config":"cookie-workflow","item":"a08","split":"heldout","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"Strong decision gathering, receipt reporting and platform boundaries. But ordering conflicts hurt execution: SKILL step 3.1 says to set the verification env vars in the shell running `$B`, while BROWSER.md says they must be in the daemon environment before startup, and later CLI-call settings won't reconfigure an existing daemon. Step 3 also runs `$B status` and `$B goto` before the daemon-environment timing is addressed, and the missing-variable recovery (\"return to step 3\") doesn't say to restart the daemon. Direct import also lacks details on multiple domains and on how to confirm the receipt output format."},"latency_ms":2947,"stop_reason":"end_turn","usage":{"input_tokens":4881,"output_tokens":241},"cost_usd":0.012172} +{"config":"cookie-workflow","item":"a07","split":"heldout","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is clear: it asks for all decisions before running any `$B` command, and it keeps copied cookies separate from verified sign-in with a three-line receipt format. Consent for `--clear-storage` and `--all` is explicit. Gaps: the Aside probe lives in an external file, and ordering is slightly awkward, since the verification env vars are configured in step 3 yet the picker also offers verification. Step 2 allows direct import but never says what to do with several domains beyond \"one `--domain` per run\", and the receipt-to-report mapping for `partial` is thin. The error tables and recovery paths are otherwise concrete and consistent with the platform limits."},"latency_ms":2947,"stop_reason":"end_turn","usage":{"input_tokens":4888,"output_tokens":249},"cost_usd":0.012266} +{"config":"cookie-workflow","item":"p07","split":"heldout","phase":"old-b","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is clear and well ordered: Aside probe, collect all decisions up front, prepare the daemon, import, then report in a fixed three-line format. Consent for --clear-storage and --all is explicit, as are the defaults. The verification env-var timing and the rule that cookies are not sign-in evidence are well specified. Examples are concrete. Gaps: (1) Step 2 asks for answers before any `$B` command, but step 3 sets env vars before the daemon starts, and the entry for the optional `$B stop` is ambiguous when the user declines verification. (2) The `--all` and multi-domain handling is thin. `--domain` is 'one per run', with no loop guidance, and there is no example of the `--all` command. (3) The picker flow is vague about how the verify checkbox interacts with env vars set after the daemon starts. (4) The 'Storage reset' line has no mapping from the receipt when the import itself fails. (5) The `$B status` cdp check comes after the user chose browser and profile, which is some wasted effort. (6) Recovery for expired picker links and for a failed reset is minimal. (7) `$B goto` is described as 'leave tab alone', yet the example in BROWSER.md adds `$B reload`. Overall it is solid with a few small ambiguities."},"latency_ms":5961,"stop_reason":"end_turn","usage":{"input_tokens":4733,"output_tokens":459},"cost_usd":0.014056} +{"config":"cookie-workflow","item":"a08","split":"heldout","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":3,"reasoning":"Strong consent gating, receipt-based reporting, and error tables. But step 3.1 has agents set the auth env vars in the shell that runs `$B`, while BROWSER.md says they must be in the daemon environment before startup, and a later CLI call won't reconfigure a running daemon. Step 3.2-3.3 starts the browser after the vars, but the rejection recovery loop ('return to step 3') omits restarting the daemon. Other gaps: how to handle the `--all` flow in direct import, and multi-domain runs."},"latency_ms":4039,"stop_reason":"end_turn","usage":{"input_tokens":4881,"output_tokens":206},"cost_usd":0.011822} +{"config":"cookie-workflow","item":"f08","split":"heldout","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The workflow is well structured: it gathers decisions up front, covers the consent gates, and defines the three-line report. But it has contradictions. SKILL.md says to copy a Chrome 136+ profile to a temp folder, while BROWSER.md says closing Chrome does not help and there is no real-profile-copy workaround. SKILL.md also says to echo the expected identity, cookie names and values back, which conflicts with BROWSER.md's rule to report only sanitized outcomes. Both are privacy-sensitive. Other gaps: the Keychain advice to keep rerunning is unbounded, and the 'import rejects, return to step 3' loop has no stop condition."},"latency_ms":10537,"stop_reason":"end_turn","usage":{"input_tokens":4922,"output_tokens":252},"cost_usd":0.012364} +{"config":"cookie-workflow","item":"p08","split":"heldout","phase":"old-a","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: stop conditions, a batched decisions step, preparation, picker vs. direct import, recovery, and a three-line honest report. It clearly separates copied cookies from sign-in evidence, requires explicit consent for --clear-storage and --all, and keeps verification secrets private. Recovery points to the exact BROWSER.md section with its error tables.\n\nThere are minor gaps and tensions. (1) The ask-first rule says to ask all decisions before any $B command, yet the Aside probe and stop conditions run first and the Mode: cdp check comes later in step 2, so a user could be asked questions that turn out to be unnecessary. (2) Direct import allows one --domain per run, but the flow for multiple domains is not spelled out. (3) --all as a direct-import option is not shown in step 3, and its target-navigation requirement is unclear. (4) The ordering of the verification env setup versus `$B stop` versus the Browser fallback section is slightly awkward. (5) The picker path puts the verification checkbox in the picker, but the env vars must be set beforehand, so the user has to be told to do that. (6) The receipt mechanics for the direct CLI path are not described; it is unclear where the count and status come from. (7) The no-display handoff case is only briefly covered."},"latency_ms":11407,"stop_reason":"end_turn","usage":{"input_tokens":4587,"output_tokens":461},"cost_usd":0.013784} +{"config":"cookie-workflow","item":"a08","split":"heldout","phase":"old-a","sample":2,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"The workflow is well ordered: Aside check, then all decisions up front, then prepare, import, and report. Consent defaults are clear, and the three-line report separates copied cookies from verified sign-in. It has three gaps. (1) SKILL.md says to set the verification variables 'in the shell that runs $B', but BROWSER.md says they must be in the daemon environment before startup, and a later CLI call does not reconfigure a running daemon. SKILL.md never says to restart the daemon, and its 'return to step 3' recovery for missing settings would not work against an already-running daemon. (2) `--all` is requested in step 2 but never appears in the step 4 import instructions, and its interaction with the picker and with `--verify-auth` is unspecified. (3) 'One `--domain` per run' against a single target is unclear when several domains are chosen, because verification reloads the captured target and the direct import requires navigating to a page matching `--domain`. Error and receipt-failure handling and the platform limits are otherwise well covered."},"latency_ms":8565,"stop_reason":"end_turn","usage":{"input_tokens":4484,"output_tokens":994},"cost_usd":0.018908} +{"config":"cookie-workflow","item":"p07","split":"heldout","phase":"old-a","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered. It asks for consent and decisions before any `$B` command. It sets up verification variables before the daemon starts, and it reports from the receipt in a fixed three-line format. It distinguishes copied cookies from sign-in evidence, scopes the storage reset precisely, and describes the platform boundaries. The error tables and the worked example make it easy to follow.\n\nSome gaps remain:\n- Step 1 says to run the Aside probe from `browse/SKILL.md`, but that content is not in the bundle. The preamble says external files are documented separately, so this is only a minor concern.\n- The `--all` direct-import syntax is never shown, and the text does not say whether it needs a target or a browser/profile.\n- The text does not say how to handle several domains, other than 'one `--domain` per run'. It also does not say how to handle a target that matches only one of them.\n- There is a possible conflict about the target. In step 2 the target is a single page on a chosen domain. BROWSER.md's direct-import example navigates to a bare domain URL.\n- A failed or partial import is not tied to a decision about whether to retry, with the exception of the receipt failure reasons table.\n- The picker flow does not say what to do if the user never reports back.\n- Several decisions are left to the agent's judgement. One is when to offer a headed manual sign-in. Another is how to map receipt states such as `partial` onto the report lines. The statement that `--clear-storage` is Chromium-only is also slightly in tension with the 'other engines reject but still import' wording."},"latency_ms":5929,"stop_reason":"end_turn","usage":{"input_tokens":4733,"output_tokens":547},"cost_usd":0.014936} +{"config":"cookie-workflow","item":"p06","split":"heldout","phase":"new","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: the decisions are asked up front, defaults are stated, and the Aside and CDP short-circuits are explicit. Verification semantics, the receipt report format and the error tables are clear. Gaps: the `--all` flow has no execution step (Step 4 omits it, and it conflicts with the one-domain target requirement in Step 2). Consent wording for `--all` is not specified. 'Picker' versus 'direct' is slightly ambiguous when verification is wanted. Step 4 never tells the agent to read BROWSER.md's cookie section unless an error occurs."},"latency_ms":3062,"stop_reason":"end_turn","usage":{"input_tokens":5015,"output_tokens":226},"cost_usd":0.01229} +{"config":"cookie-workflow","item":"p07","split":"heldout","phase":"new","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well sequenced: ask the user first, prepare the browser, import, then report. Receipt-based reporting is clear, and the example commands are concrete. Weak spots: SKILL.md says to open the target before import, while the Opera example in BROWSER.md runs `$B reload`. It also leaves `--all` handling unspecified, with no command example or target-tab guidance. Consent for `--clear-storage` is clear, but the picker route does not say how to pass clearing. The `--profile` omission rules differ slightly between files. Error recovery is largely covered, including handoff and manual sign-in."},"latency_ms":3188,"stop_reason":"end_turn","usage":{"input_tokens":5130,"output_tokens":229},"cost_usd":0.01255} +{"config":"cookie-workflow","item":"p07","split":"heldout","phase":"new","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well sequenced: Aside check, consent gathering, daemon prep, import, and a three-line report. It separates copied cookies from verified sign-in, and the example command is concrete. Gaps: the Aside probe command is deferred to browse/SKILL.md, which isn't supplied. Picker versus direct import is slightly ambiguous when the user supplies only some answers. There is no explicit handling of the `--all` command syntax, or of a failed storage reset in the report. Also, the Windows picker note says to run `$B goto` first, while the SKILL step 3 already covers it. Overall it is strong but has small decision gaps."},"latency_ms":2872,"stop_reason":"end_turn","usage":{"input_tokens":5130,"output_tokens":241},"cost_usd":0.01267} +{"config":"cookie-workflow","item":"a07","split":"heldout","phase":"old-b","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered and explicit. It asks every decision up front, requires the profile directory rather than the display name, and makes verification, storage reset, and --all opt-in with defaults. Cookie copying is clearly separated from sign-in evidence, and the report is a precise three-line format. Error tables and platform boundaries are included. Weaknesses: (1) The ordering is slightly contradictory: step 2 says to ask before any `$B` command, but the Aside probe in step 1 and the `$B status` check in step 3 come around it, and the verification env setup comes after the user decisions. (2) The direct-import path is ambiguous about the 'one --domain per run' rule for multiple domains, and about what to do if the user wants no domain but not --all. (3) The case where the user says yes to --verify-auth in the picker but the env vars were not set is only partly handled. (4) The 'Comet' default is flagged but not resolved for the picker. (5) The Aside probe command is deferred to another file, and what to do when the browser is an unsupported one is not covered. (6) Recovery for the partial, failed, and empty receipt statuses is covered only through the failure-reason tables. Overall it is strong and executable, with minor gaps."},"latency_ms":4309,"stop_reason":"end_turn","usage":{"input_tokens":4491,"output_tokens":435},"cost_usd":0.013332} +{"config":"cookie-workflow","item":"f08","split":"heldout","phase":"old-a","sample":1,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":3,"reasoning":"The workflow is well structured: Aside probe, decisions gathered up front, daemon env configuration, and a three-line receipt report. Several defects undermine it, though.\n\n1. Contradictory platform guidance. SKILL.md says that when Chrome 136+ blocks the import, the agent should close Chrome, copy the profile directory to a temp folder, and import from the copy. BROWSER.md says closing Chrome does not remove the protection and that there is no real-profile-copy workaround. The same copy workaround is also offered for Windows App-Bound Encryption, where BROWSER.md says to use `$B handoff` then `$B resume`. An agent cannot tell which to follow. The copy workaround also raises privacy concerns.\n\n2. Privacy conflict. SKILL.md step 5 tells the agent to add the expected identity, profile name, and the first few cookie names and values to the report. BROWSER.md says not to paste cookie values, profile/account labels, or the expected identity into public logs, and to report only sanitized outcomes. Reporting cookie values is a security problem.\n\n3. Keychain advice. \"Keep rerunning until it goes through\" is open-ended, with no retry limit or stop condition.\n\n4. Ordering and missing decisions. Env-var setup precedes finding `$B`, and `$B stop` needs `$B` to be located first. The user is asked to paste the identity into chat, which conflicts with keeping it private. It is unclear whether the picker must be used when `--all` is chosen, and `--all` handling is missing from the direct-import steps. The direct import requires navigating to a matching domain first, which step 3 covers. A \"one --domain per run\" rule for multiple domains is not reconciled with the single target.\n\n5. Gaps. Several error and receipt cases are not covered, such as what to do on a partial import."},"latency_ms":5561,"stop_reason":"end_turn","usage":{"input_tokens":4525,"output_tokens":651},"cost_usd":0.01556} +{"config":"cookie-workflow","item":"a07","split":"heldout","phase":"new","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: Aside probe first, then a batched consent step, daemon prep, import, and a three-line receipt report. It separates copied cookies from verified sign-in, and the consent scopes for storage reset and --all are precise. Gaps: SKILL.md says to ask before any $B command, yet the Aside probe may need $B. The picker-versus-direct choice is open when only some answers are known. Receipt failure handling for partial or empty imports is thin. Running multiple domains via repeated direct imports leaves the receipt report unclear. Raw `cookies` use and the Aside probe command are only referenced externally."},"latency_ms":3185,"stop_reason":"end_turn","usage":{"input_tokens":4888,"output_tokens":237},"cost_usd":0.012146} +{"config":"cookie-workflow","item":"a08","split":"heldout","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The workflow is well structured. It gets user decisions before any `$B` command, defines the three-line report, and says that cookie counts and HTTP 200 never prove sign-in. Several gaps and conflicts would trip an agent. (1) SKILL.md step 3.1 says to set the auth env vars 'in the shell that runs `$B`', but BROWSER.md says they must be in the daemon environment before startup and that a later CLI call does not reconfigure a running daemon. The skill never says to stop or restart the daemon, and the 'return to step 3' recovery for missing settings does not fix this. (2) Step 2 says the picker asks for the domain, yet the target URL, which must be on a chosen domain, is required before step 3's `$B goto`. A user who leaves the domain open therefore has no clear path. (3) Direct import allows one `--domain` per run but never says how to handle multiple domains, such as repeated runs or a combined receipt. (4) The `--all` flow is covered only in the consent questions, not in the step 4 command guidance. (5) Recovery for receipt failures is split between the two files, and there is no guidance on retry bounds. Platform boundaries and privacy rules are clear."},"latency_ms":7744,"stop_reason":"end_turn","usage":{"input_tokens":4484,"output_tokens":854},"cost_usd":0.017508} +{"config":"cookie-workflow","item":"a07","split":"heldout","phase":"old-a","sample":0,"status":"error","error_kind":"parse","error":"JSON Parse error: Unrecognized token '<'","latency_ms":5384,"stop_reason":"end_turn","usage":{"input_tokens":4491,"output_tokens":534},"cost_usd":0.014322} +{"config":"cookie-workflow","item":"f07","split":"heldout","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"Strong on consent, scoping, and honest reporting, but there are ordering and decision gaps. Step 2 asks for domain scope only indirectly (--all 'cannot combine with a domain', yet no domain question or flag). Step 4 says 'If the import fails, run it again', which conflicts with the warning that a failed reset isn't rolled back and gives no guidance on retrying safely, on App-Bound Encryption failures, or on when to switch to Opera. Direct import also lacks an exact command syntax and an explicit way to read the receipt."},"latency_ms":5191,"stop_reason":"end_turn","usage":{"input_tokens":3705,"output_tokens":211},"cost_usd":0.00952} +{"config":"cookie-workflow","item":"f06","split":"heldout","phase":"old-b","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The workflow has a sensible shape: gather decisions up front, default to no for risky options, import, then report honestly. The privacy and consent boundaries are clear, and so is the point that cookie counts never prove sign-in. The gaps are large, though. (1) Step 1 'Check for Aside' is cryptic: 'Aside' is never defined, and it is unclear how to detect an already signed-in browser. (2) Verification setup is vague. 'Set up the verification settings privately' gives no settings names, no mechanism, and no way to identify the element or expected identity. BROWSER.md says to configure it before the browser starts, while SKILL.md step 3 says to get the browser running and then configure it, so the ordering is contradictory. (3) The tools are never named. 'gstack's cookie import', the 'interactive chooser', and the 'direct form' have no commands, flags, or parameters. (4) Supported browsers, platforms, and profile discovery are unspecified. (5) Requirement 5, copying every cookie, has no mechanism and no stated scope, while the domain requirement for the default import is only implied. (6) Storage clearing has no stated failure handling beyond 'not undone'. (7) The receipt is mentioned but its format is not described, and there is no procedure for obtaining the counts or sign-in status. (8) Recovery is limited to 'sign in manually', and the instruction to ask for the page to open is not tied to how the site is identified. An agent would have to guess at tool invocation."},"latency_ms":5209,"stop_reason":"end_turn","usage":{"input_tokens":1414,"output_tokens":523},"cost_usd":0.008058} +{"config":"cookie-workflow","item":"p06","split":"heldout","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: it gathers decisions up front, then prepares the browser, imports, and reports. It states the consent rules for --clear-storage and --all, and it clearly separates copied cookies from verified sign-in with the three-line report. Verification semantics and the storage reset scope are precise. Platform boundaries and error tables are covered.\n\nSome gaps and ambiguities remain.\n- The ask-first step requires all decisions before any `$B` command, but the Aside probe in step 1 and the `$B status` CDP check in step 3 come at different points. The probe runs before asking, which is acceptable. However, the CDP exit happens only after the user has been asked.\n- `--all` is in the decision table, but the import instructions never say how to run it. It is not described in the picker path or the direct import path, and the \"one --domain per run\" rule conflicts with the multiple domains the user may give in decision 2.\n- Which domain the target URL belongs to is not specified for multi-domain runs.\n- The SKILL.md names the section to read for errors, but the BROWSER.md section's \"Windows: Opera\" example uses `$B reload` and is only loosely linked to the workflow.\n- Only some error cases have recovery steps. Examples are a missing `$B handoff` condition, retries when the database is locked, the receipt text for a partial result, and a failed reset.\n- The step 4 rejection for missing verification settings returns to step 3, but it does not say to restart the daemon.\n- Step 2 and step 5 do not say how to report when domains span multiple runs."},"latency_ms":9504,"stop_reason":"end_turn","usage":{"input_tokens":4618,"output_tokens":549},"cost_usd":0.014726} +{"config":"cookie-workflow","item":"a08","split":"heldout","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured and its decision points are explicit: it asks for all consent items up front, with defaults. It requires an explicit profile directory, separates imported cookies from verified sign-in, and gives a three-line report format. BROWSER.md adds error tables, receipt failure reasons and platform limits, and the two files agree on the core rules. Several gaps remain. (1) BROWSER.md says the verification env vars must be set in the daemon environment before startup. SKILL.md only says to set them in the shell that runs `$B`, and does not say to restart an already-running daemon. That matters when it says to return to step 3 after a missing-variable rejection. (2) `--all` is a question in step 2 but has no import instruction in step 4, and the picker's handling of `--all` is unspecified. (3) For multiple domains, 'one `--domain` per run' is stated, but how to repeat runs against the target is not. (4) 'Ask before any `$B` command' sits awkwardly with the Aside probe and `$B status` checks, which can end the skill after the questions were already asked. (5) The mapping from receipt states to the report lines, such as a rejected non-Chromium reset, is only implied."},"latency_ms":7543,"stop_reason":"end_turn","usage":{"input_tokens":4484,"output_tokens":835},"cost_usd":0.017318} +{"config":"cookie-workflow","item":"p07","split":"heldout","phase":"old-a","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered. It asks all decisions up front, probes Aside, checks for CDP mode, sets the daemon environment before startup, opens the target, and then imports with a worked example. The receipt-based report format is exact. It clearly separates copied cookies from verified sign-in, and consent for --clear-storage and --all is explicit. BROWSER.md covers error and failure-reason tables plus platform limits.\n\nGaps and ambiguities:\n- The Aside probe and the Browser fallback section live in an external file, which is acceptable given the bundle note.\n- The picker flow does not say how to handle verification env vars, or how the picker's target gets chosen, beyond the goto in step 3.\n- Direct import allows one --domain per run, but multi-domain handling is only implied (rerun per domain).\n- The --all command syntax is not shown, and the direct-import example omits --clear-storage.\n- Step 2 says to ask before running any $B command, but step 3 requires $B stop and a status check afterward. The user's choices could also change after the Aside or CDP end-conditions, which wastes the questions.\n- Handling for 'partial' or 'empty' receipts is thin. Next steps cover only certain failure keys, with no guidance for a failed storage reset or a not-verified result beyond reporting it.\n- Windows boundaries are clear, but the handoff offer is limited to displays and appears in scattered places."},"latency_ms":5388,"stop_reason":"end_turn","usage":{"input_tokens":4733,"output_tokens":506},"cost_usd":0.014526} +{"config":"cookie-workflow","item":"f08","split":"heldout","phase":"old-a","sample":2,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":3,"reasoning":"The workflow is well structured: ask all decisions first, then prepare, import, and report. It also covers consent for storage clearing and `--all`, and it separates copied cookies from verified sign-in. However, it has serious contradictions and gaps. (1) SKILL.md tells the agent to copy the Chrome profile to a temp folder and import from the copy when Chrome 136+ blocks the import. BROWSER.md says closing Chrome does not remove the protection and that there is no real-profile-copy workaround. The agent cannot tell which is correct. (2) SKILL.md says to include the expected identity, profile name, and the first few imported cookie names and values in the report. BROWSER.md says not to expose cookie values, profile labels, or the expected identity, and to report only sanitized outcomes. This is a privacy violation. (3) SKILL.md says to keep rerunning on Keychain or permission failures, which has no retry limit. (4) The Aside probe and 'Browser fallback' are referenced but not in the bundle, which is acceptable. (5) The direct-import step does not specify the `--all` syntax. (6) Step 4 sends the agent back to step 3 for a missing variable, but the sequencing is awkward because the daemon has already started. (7) The 'Windows App-Bound: use the same copy' instruction is unsupported by BROWSER.md, which offers handoff instead. (8) The picker is described as 'tells you when done', but the agent has no way to read the receipt. The contradictions on privacy and platform recovery strongly reduce clarity and safety."},"latency_ms":5641,"stop_reason":"end_turn","usage":{"input_tokens":4525,"output_tokens":549},"cost_usd":0.01454} +{"config":"cookie-workflow","item":"p08","split":"heldout","phase":"old-b","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: stop conditions, a batched decision step, setup, import, recovery and an honest three-line report. It is strong on consent (storage reset and --all both default to no), on separating copied cookies from verified sign-in, and on privacy. The remaining gaps are small. (1) The direct import command is described in prose and never shown as a full invocation with flags. It is also unclear whether several domains mean several runs, and whether `--verify-auth` can be repeated across runs. (2) The instructions are slightly contradictory about order. Step 1 says to ask before any `$B` command, but the Aside probe at the top is a separate check, so this is only a mild tension. The Opera example runs `$B reload` after import, while step 2 says to leave the tab alone until the import finishes, which does not actually conflict. (3) The `--all` direct-import path is underspecified: no domain, so the target is unclear, and the picker-versus-direct choice does not cover it. (4) The Aside probe lives in an external file and is only referenced, which is acceptable here. (5) The recovery path for the user-unreachable case and for a partial import is only loosely defined. (6) The Chrome 136+ and App-Bound Encryption fallbacks are covered through handoff and manual sign-in, but the logic for choosing among them is spread across two files."},"latency_ms":7646,"stop_reason":"end_turn","usage":{"input_tokens":4587,"output_tokens":466},"cost_usd":0.013834} +{"config":"cookie-workflow","item":"f07","split":"heldout","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"Strengths: the skill clearly gathers decisions up front (browser, profile directory, verify-auth, clear-storage, --all), has explicit consent defaults, a sound distinction between copied cookies and verified sign-in, a three-line honest report, and BROWSER.md details platform limits and privacy. Weaknesses: (1) The domain/target is never asked for or specified, although the task includes selecting a domain and the storage reset and verification rely on a 'captured target' that is never defined or opened. (2) Ordering conflict: step 3 requires env vars and a daemon stop before `$B` is found, while step 2 says to ask before any `$B` command, and the `$B status` check comes after the preparation. Cdp mode may also make the verify-auth setup moot. (3) The instruction 'If the import fails, run it again' contradicts the documented non-rollback of a failed reset and could repeat a mutation or loop, with no decision on when to stop or how to handle partial results. (4) The picker flow is unclear on how consent for verify-auth and clear-storage maps onto its checkboxes, and how the agent learns the receipt without raw output. (5) Missing guidance on the Windows App-Bound Encryption fallback to Opera, the Chrome 136+ restriction, and what to do when a profile is unreadable or the browser is unsupported. (6) The `--all` command syntax and the `--domain` flag usage are not shown. (7) 'Aside probe' details live in another file, which is acceptable. Overall it is reasonably usable but has real ambiguity gaps."},"latency_ms":6246,"stop_reason":"end_turn","usage":{"input_tokens":3308,"output_tokens":550},"cost_usd":0.012116} +{"config":"cookie-workflow","item":"a07","split":"heldout","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: check Aside, collect all decisions up front, prepare $B, import, then report in three lines. Consent for --all, --clear-storage and --verify-auth is explicit, and the rules distinguishing copied cookies from verified sign-in are strong. The platform boundaries and the error and receipt tables give clear recovery paths. Weaknesses: (1) The SKILL.md section order is slightly awkward. The user is asked for the verification selector and identity in step 2, but the env vars are set and the daemon restarted in step 3, and a missing-variable rejection loops back to step 3. (2) The picker versus direct choice does not say what to do if --all is wanted; --all is not in the direct-import instructions or the picker flow. (3) The decision to ask about domains is clear, but the 'url' to open for goto is not specified when several domains are chosen. (4) The `$B stop` step is mentioned, but its consequences are not. (5) Recovery for 'partial' and 'failed' receipts is only loosely covered, and the 'Report' step has no explicit handling for the failure reasons. (6) The rule 'one --domain per run' with several domains is implied rather than spelled out. (7) The Aside probe lives in an external file, which is acceptable. Overall it is solid, with minor gaps."},"latency_ms":4558,"stop_reason":"end_turn","usage":{"input_tokens":4491,"output_tokens":464},"cost_usd":0.013622} +{"config":"cookie-workflow","item":"f08","split":"heldout","phase":"new","sample":1,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":2,"reasoning":"The consent flow, receipt reporting and verification setup are mostly clear. But SKILL.md contradicts BROWSER.md on Chrome 136+: it says to close Chrome and import from a copied profile, while BROWSER.md says closing does not help and there is no real-profile-copy workaround. It also tells the agent to keep rerunning Keychain approvals. The report step asks for cookie values and account details, against the privacy rule to report only sanitized outcomes. Step 3 orders the selector and identity env vars before the `$B` stop and setup steps, which is murky. Direct import is also unclear on multiple domains and on how the `--all` flag is invoked."},"latency_ms":4864,"stop_reason":"end_turn","usage":{"input_tokens":4922,"output_tokens":249},"cost_usd":0.012334} +{"config":"cookie-workflow","item":"a07","split":"heldout","phase":"new","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow orders steps well: ask all decisions first, then prepare, import and report. It gives precise semantics for verification, storage reset and --all, and the three-line report is concrete. Weaknesses: the Aside probe lives in an external section and is not shown. Step 3 says to open a page, but it does not say which URL to use or how to derive it from the domain. The ordering of verification setup (step 3.1) versus the picker's verification checkbox is slightly awkward. Direct import omits --all handling and the --profile fallback. Recovery paths like handoff and resume appear only in BROWSER.md and are not linked from the main flow. Overall it is solid but has small gaps."},"latency_ms":3536,"stop_reason":"end_turn","usage":{"input_tokens":4888,"output_tokens":251},"cost_usd":0.012286} +{"config":"cookie-workflow","item":"p06","split":"heldout","phase":"old-a","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered. It requires all decisions up front, with a default table and explicit consent for --clear-storage and --all. Scope and caveats for storage reset are stated precisely, and the three-line honest report format clearly separates copied cookies from verified sign-in. The platform limits and receipt failure reasons are covered, and the BROWSER.md recovery tables are specific. There are some gaps. The Aside probe and the `$B` location are delegated to external files, which is acceptable. Step 2 asks for the target URL, but `--all` has no domain, and there is no clear path for `--all` imports (target page, flags, report line). Multiple domains with one `--domain` per run are mentioned, but the loop and target handling are unclear. The picker and direct-import choice is a little loose. How to pass the browser name and `--profile` is clear, but how to present the verification selector privately is ambiguous. Handling of the `decryption_failed` retry (closing the source browser) conflicts a little with the Chrome 136 note, though it is not contradictory. Failure paths for a partial or failed receipt are only moderately defined in the entry file. The `--all` flag is listed in the table but never appears in the import instructions."},"latency_ms":6604,"stop_reason":"end_turn","usage":{"input_tokens":4618,"output_tokens":427},"cost_usd":0.013506} +{"config":"cookie-workflow","item":"p06","split":"heldout","phase":"old-a","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured. It asks the user for all decisions up front, scopes storage reset and `--all` consent, and defines verified, not verified and not checked precisely. The three-line report format and the receipt-first reporting rule are explicit. The privacy and platform boundaries are also stated clearly.\n\nThere are some gaps and ambiguities:\n- Decision 2 asks for \"domains\" plural but also says `--domain` is \"one per run\", so multi-domain handling is only implied (rerun per domain).\n- The `--all` decision (#5) has no direct-import instruction for it in SKILL.md, and the report format does not cover it.\n- The \"Aside probe\" and \"Browser fallback\" steps live in other files that are not shown, which is acceptable per the instructions.\n- Step 3 runs `$B goto` before the import, but the picker flow and the verification-env restart ordering are slightly tangled. The agent has to run `$B stop` before the daemon starts, and `$B status` is run after setup.\n- If the user's chosen target domain differs from the other domains, the instructions do not say how to handle it.\n- The \"Imported\" line includes a count and a status, but the format for `partial` or `failed` failure reasons is not specified.\n- Recovery paths such as `handoff`/`resume` are documented, but the decision of when to offer them versus stopping is only partly specified.\n- The import-error tables and platform boundaries are good."},"latency_ms":6519,"stop_reason":"end_turn","usage":{"input_tokens":4618,"output_tokens":513},"cost_usd":0.014366} +{"config":"cookie-workflow","item":"f06","split":"heldout","phase":"new","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The workflow has clear top-level steps and honest reporting rules, but key mechanics are missing. Step 3 says to \"set up verification settings privately\" and \"get gstack's browser running\" without naming settings, commands, or how to do either. BROWSER.md says sign-in checking must be configured before the browser starts, while SKILL.md says to start the browser first and have the user configure afterward, so the ordering contradicts. The \"direct form\" has no syntax, and the \"receipt\" is never defined. Consent for copying every cookie is asked but never handled in the import step. Step 1's \"already signed in\" check has no method."},"latency_ms":3003,"stop_reason":"end_turn","usage":{"input_tokens":1811,"output_tokens":237},"cost_usd":0.005992} +{"config":"cookie-workflow","item":"p06","split":"heldout","phase":"old-b","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well-ordered. It checks for Aside first, collects every decision together before running any `$B` command, prepares the browser, imports, and reports. Consent for clearing storage and for `--all` is explicit. The scope of storage reset is stated precisely: origin localStorage, target-tab sessionStorage, Chromium only, no rollback. The sections separate copied cookies from verified sign-in, and the three-line report format is clear. Privacy limits are stated, and the platform boundaries are documented.\n\nThere are gaps and mild tensions:\n- Step 3.1 asks the user to set the env vars before the browser prepare step, but the picker can also tick verification, so the direct and picker paths differ slightly.\n- The Aside probe and the Browser fallback sections live in other files and are only referenced.\n- Decision 5 (`--all`) has no corresponding direct-import instruction, and the report line does not account for `--all`.\n- \"Clear storage\" with the picker has no stated mechanism, since the picker description mentions only verification.\n- The picker-versus-direct choice is clear, but the direct import's one-domain-per-run limit conflicts with the multi-domain ask. Looping per domain is not described, nor how to report multiple domains and targets.\n- The `$B reload` example in BROWSER.md does not match the \"leave the tab alone\" guidance.\n- Failure recovery is partly covered: a missing verification variable sends the user back to step 3, and the failure-reason tables exist. Handoff needs a display, and some error paths such as `decryption_failed` (\"close the source browser\") only loosely fit the privacy and platform limits."},"latency_ms":7456,"stop_reason":"end_turn","usage":{"input_tokens":4618,"output_tokens":581},"cost_usd":0.015046} +{"config":"cookie-workflow","item":"f07","split":"heldout","phase":"old-a","sample":2,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The workflow is mostly well structured: ask the user for browser, profile, verification, storage-reset and --all choices before any `$B` command, then import, then report in three lines. Scope, consent, and the distinction between copied cookies and verified sign-in are handled carefully. Several gaps and ordering problems reduce usability. (1) The domain scope is not defined: step 2 never asks for the target domain or URL, though the task and `--all` text refer to one, and the commands never show a `--domain` flag. (2) Step 2 says to ask before running any `$B` command, but step 1 runs the Aside probe, which may itself use browse tooling, and step 3 only checks for CDP mode afterwards. Step 3 can end the skill after the user has already answered everything. (3) Verification setup is split: the env vars must be set before daemon start, yet `$B status` may report an already-running daemon, and the instructions do not say how to detect one before deciding on `$B stop`. (4) 'If the import fails, run it again' conflicts with `--clear-storage` semantics, since a failed reset is not rolled back and a retry may repeat the reset, and with the one-time picker link. The text also does not say how to diagnose or when to stop retrying. (5) The Windows App-Bound Encryption limits and the platform-specific browsers (Comet, Arc, Dia on macOS) are only in BROWSER.md and are reached only after a failure, so the agent may offer unsupported sources up front. (6) The exact direct-import syntax is vague: the browser name is positional or flagged, and `--all` is not shown in the direct-import step. (7) The 'verified' versus 'not verified' receipt mapping is left implicit, and the receipt format is not given."},"latency_ms":6105,"stop_reason":"end_turn","usage":{"input_tokens":3308,"output_tokens":600},"cost_usd":0.012616} +{"config":"cookie-workflow","item":"p07","split":"heldout","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered. It asks the user for every decision before running any command, then prepares the browser, imports, and reports. The consent boundaries are precise: verification is off by default, storage reset needs approval and is Chromium-only, and `--all` needs explicit consent and cannot be combined with a domain or storage reset. The workflow separates copied cookies from sign-in evidence, and it gives a three-line receipt format with an example. It also handles recovery paths: missing verification variables, expired picker links, and Windows App-Bound Encryption with a headed handoff.\n\nThere are small gaps. SKILL.md does not say how the `--all` direct-import command is written, and the Aside probe and Browser fallback details live in external files. The step 2 question of whether to use direct import or the picker when a browser is named but the profile is missing is only partly covered. The failure paths for a failed storage reset are not fully spelled out. SKILL.md also says a missing-variable rejection means returning to step 3, but it never says that `$B stop` may be needed again. Multi-domain imports are only implied, by running once per domain. Receipt format mapping for partial or failed results and for `--all` (no domains) is underspecified. The BROWSER.md direct-import example runs `goto` on a bare domain rather than the target URL, which differs slightly from SKILL.md's flow but is not contradictory."},"latency_ms":5100,"stop_reason":"end_turn","usage":{"input_tokens":4733,"output_tokens":484},"cost_usd":0.014306} +{"config":"cookie-workflow","item":"a08","split":"heldout","phase":"new","sample":2,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":3,"reasoning":"Strong consent gating, receipt reporting, and platform limits. But ordering contradicts: SKILL step 3 has the user set verification env vars \"in the shell that runs $B\", while BROWSER.md requires them in the daemon environment before startup, and step 3 starts the daemon only afterward, so an already-running daemon is left unaddressed. Direct import also omits the pre-import navigation detail and a concrete command example in SKILL. Handling for \"--all\" is unclear: it is asked about but never shown in the import commands."},"latency_ms":2945,"stop_reason":"end_turn","usage":{"input_tokens":4881,"output_tokens":206},"cost_usd":0.011822} +{"config":"cookie-workflow","item":"f07","split":"heldout","phase":"old-a","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The workflow is strong on consent and privacy: it asks for decisions up front, scopes storage reset precisely, and says copied cookies, counts, and HTTP 200 are not sign-in evidence. It also gives a clear three-line report format. Several gaps and ordering problems reduce usability. (1) The domain and target are never defined. Question 4 mentions a domain, and the intro promises to pick a domain, but no step asks for it or says how to pass `--domain`. Direct import, `--clear-storage`, and `--verify-auth` all depend on a captured target tab that is never explained or set up. (2) The ordering contradicts itself. Step 2 says to ask before running any `$B` command, but step 1's Aside probe and step 3's `$B` setup come around it. The step 1 probe is also described as a `$B`-free path, which is unclear. Step 3 says to have the user set env vars privately and run `$B stop` first, but this is only needed if the user chose verification, and it is not tied to the picker's checkbox flow. Picker verification is an unexplained branch: the user can tick it in the picker, yet the env vars must already be set before the daemon starts. (3) The failure recovery is thin. 'If the import fails, run it again' is naive. It ignores the non-rollback of a failed storage reset and the case where a retry would repeat a clearing action. It also gives no guidance for the Windows App-Bound Encryption limit, Chrome 136+, or a Keychain denial beyond pointing to BROWSER.md. (4) Some receipt details are unspecified: how to read the receipt, and what to do when the sign-in status is 'not verified'. The Comet default is flagged as non-consent, which is good. Overall the workflow is usable but has real gaps."},"latency_ms":5799,"stop_reason":"end_turn","usage":{"input_tokens":3308,"output_tokens":611},"cost_usd":0.012726} +{"config":"cookie-workflow","item":"f08","split":"heldout","phase":"old-b","sample":1,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":3,"reasoning":"The workflow is well structured in places: it asks the user for all decisions up front, requires explicit consent for storage reset and --all, separates copied cookies from verified sign-in, and has a three-line receipt report. But there are serious contradictions and gaps. (1) SKILL.md tells the agent to close Chrome and copy the profile to a temp folder to get around Chrome 136+ protection, while BROWSER.md says closing Chrome does not remove that protection and that there is no real-profile-copy workaround. (2) SKILL.md tells the agent to add the expected identity, profile name, and the first few cookie names and values to the report, while BROWSER.md says never to put cookie values, profile labels, or expected identity in public logs and to report only sanitized outcomes. This is a privacy violation. (3) SKILL.md says to keep rerunning on Keychain or permission errors, which is unbounded and risks repeated prompts. (4) The order is muddled: SKILL.md gets the verification env vars in step 3 before the import, but direct import details are thin, and the instruction to run `$B stop` first conflicts with leaving the target tab open. (5) Choosing the picker versus direct import when `--all` is wanted is not specified, and the combination of `--all` with the target and `goto` is unclear. (6) The Windows App-Bound Encryption advice differs between the two files: copy the profile versus handoff and resume. The error tables and receipt reasons are useful, but these conflicts keep clarity low, and an agent following them literally could leak data or try an unsupported workaround."},"latency_ms":5238,"stop_reason":"end_turn","usage":{"input_tokens":4525,"output_tokens":547},"cost_usd":0.01452} +{"config":"cookie-workflow","item":"f06","split":"heldout","phase":"old-a","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The high-level flow (ask first, default no, report honestly, cookie counts are not proof of sign-in) is clear, and the privacy and consent boundaries are well stated. But the instructions are too vague to execute. Step 1 ('Check for Aside') is undefined: how to detect a signed-in browser is never explained, and 'Aside' is unclear. Step 3 says to 'get gstack's browser running' with no command. The verification settings are never specified: the instructions don't say what to configure, where, which element, or how the expected identity is supplied. BROWSER.md says to configure before the browser starts, while SKILL.md step 3 says to start the browser and then have the user set up settings, so the ordering contradicts and the restart case is unhandled. The interactive chooser and direct form have no names, flags, or syntax. The supported browsers and platform restrictions are unnamed. 'Receipt' is not defined. The consent wording for copying every cookie is vague, and it's unclear whether that option exists in the import tool. The storage-clear warning says it is not undone on failure but gives no recovery guidance. The failure paths all end in 'sign in manually', with no decision tree for partial failure, an ambiguous profile, or a failed verification. Sign-in verification outcomes are listed, but the steps to produce them are not."},"latency_ms":19355,"stop_reason":"end_turn","usage":{"input_tokens":1414,"output_tokens":462},"cost_usd":0.007448} +{"config":"cookie-workflow","item":"f06","split":"heldout","phase":"old-a","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The high-level flow is readable, with good defaults, consent gates, and honest reporting. However, the instructions are too vague to execute. Step 1 ('Check for Aside') is undefined. 'gstack's browser', 'cookie import', 'interactive chooser', and 'direct form' name no commands, flags, or syntax. Verification setup is not specified: no env vars or settings, no definition of the 'element' to compare, and no way to supply the expected identity, only that it is 'private' and must be done 'before the browser starts'. This conflicts with step 3 ordering, which says to start the browser and then configure verification. The list of supported browsers and platforms is not given. Profile identification by directory is mentioned, but there is no way to enumerate profiles. Consent scoping is partly covered: storage clearing is limited to one site, but the scope of 'copy every cookie' is unclear. Failure recovery is thin: 'the receipt explains why' with no receipt format and no error cases, and the fallback is a manual sign-in. The note that storage clearing is not undone on partial failure has no stated handling. Cookie counts are correctly separated from sign-in evidence, and the privacy rule is stated, but the three reporting states (verified, not verified, not checked) are not tied to concrete outcomes."},"latency_ms":5169,"stop_reason":"end_turn","usage":{"input_tokens":1414,"output_tokens":452},"cost_usd":0.007348} +{"config":"cookie-workflow","item":"p06","split":"heldout","phase":"new","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"Decisions, flags, and receipt reporting are clear, and verification and storage-reset scope are precisely defined. Gaps: the `--all` decision (#5) is asked but never routed to a command or constrained by the one-domain-per-run rule. Multiple domains are ambiguous (repeat runs, with one target). The Aside probe depends on an external section. Failure recovery for `decryption_failed` (close the source browser) is slightly at odds with the platform boundaries."},"latency_ms":2810,"stop_reason":"end_turn","usage":{"input_tokens":5015,"output_tokens":186},"cost_usd":0.01189} +{"config":"cookie-workflow","item":"a07","split":"heldout","phase":"old-a","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered. It asks for every decision up front, covers the Aside and CDP shortcuts, and puts picker and direct import in separate paths. The reporting format is precise: three lines, with 'not checked', 'not verified' and 'verified' defined, and copied cookies kept distinct from sign-in evidence. Consent for --all and --clear-storage is explicit. Scope limits are stated, including Chromium-only reset, no rollback, and one domain per run. Platform boundaries and the receipt failure and import error tables give clear recovery paths.\n\nThere are minor gaps and tensions. The Aside probe is delegated to a file outside this bundle, which is acceptable. Step 3.1 handles verification env vars, but the daemon restart is only mentioned conditionally. Direct import needs a page matching --domain, and step 3.4 says to goto a '' without explaining how to pick it. Several decision paths are underspecified: multiple domains, what to do when the user wants both --all and verification, the Dia/Keychain flow, and the partial/empty receipt next steps. Receipt parsing is also implicit, since the receipt format is never shown. The 'Picker ... ticks verification' step is slightly inconsistent with the up-front env var setup. Overall it is strong but not fully airtight."},"latency_ms":4546,"stop_reason":"end_turn","usage":{"input_tokens":4491,"output_tokens":468},"cost_usd":0.013662} +{"config":"cookie-workflow","item":"a08","split":"heldout","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: it gathers every decision up front, defaults to no for risky options, and gives a precise three-line report format. It separates copied cookies from positive sign-in evidence, and it has error and receipt-failure tables plus clear platform limits. Gaps and tensions remain. SKILL.md tells the agent to set the verification variables 'in the shell that runs $B', but BROWSER.md says they must be in the daemon environment before startup and that a later CLI call does not reconfigure a running daemon. Nothing says how to restart an already-running daemon, and the 'return to step 3' recovery depends on that. The agent is told to ask before running any `$B` command, yet the Aside probe runs first and is not clearly exempt. One target URL is paired with multiple domains and one `--domain` per run, and the `--all` direct-import invocation is not shown. The handling of `partial` or `empty` receipts beyond reporting them is thin."},"latency_ms":7755,"stop_reason":"end_turn","usage":{"input_tokens":4484,"output_tokens":820},"cost_usd":0.017168} +{"config":"cookie-workflow","item":"p08","split":"heldout","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: stop conditions, a batched decision step, daemon-environment setup for verification, explicit consent for storage reset and --all, a three-line honest report, and recovery tables. Clarity is slightly reduced by a few things. Step 1 says 'before any $B command', yet the Aside probe comes earlier and may itself run $B-related setup. The Mode: cdp stop condition comes after the target is chosen, so the user answers questions that may turn out unnecessary. The direct-import instructions say 'with the chosen browser name' and give no concrete example, though BROWSER.md does. 'One --domain per run' conflicts somewhat with 'domains' in the plural, and there is no loop guidance. The 'Aside probe' depends on external files. Completeness gaps: the exact browser name tokens are not listed in SKILL.md, the --all flow is not shown (it appears only in decisions and cannot be combined with other options), and there is no guidance for the receipt output format when the picker is used. How to read the receipt in picker mode is not described. There is also no explicit handling when decision 2 is skipped for --all. The existing-daemon restart is covered. Actionability is good: commands, error tables, and fallback paths are concrete."},"latency_ms":5805,"stop_reason":"end_turn","usage":{"input_tokens":4587,"output_tokens":440},"cost_usd":0.013574} +{"config":"cookie-workflow","item":"a07","split":"heldout","phase":"old-b","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered: check Aside, collect all decisions up front, prepare the browser, import, then report. Consent gates are explicit for --clear-storage and --all. The three-line report format is precise, and the document separates copied cookies from verified sign-in. Error tables and platform limits are detailed.\n\nGaps and ambiguities:\n- SKILL.md says to ask the user 'before running any $B command', but step 1 runs the Aside probe, which may itself involve $B. This is a minor ordering tension.\n- Step 3.1 asks for the env vars, then step 3.2 builds `$B`. It is unclear whether a daemon started during the build or status check would miss the vars.\n- The picker path and the `--all` option do not say how `--all` is invoked in direct mode or how the picker handles it. The direct-import description also omits `--all`.\n- If the user asks for multiple domains, direct mode needs 'one --domain per run', but the interaction with verification, the target tab, and the receipt is unspecified.\n- The three-line report format is awkward for multiple runs.\n- The picker's handling of storage reset is not mentioned.\n- Some error paths are covered only by the BROWSER.md tables: `not_installed`, `profile_required`, and the retry bounds for databases. Recovery after a 'verification failed' result is not explicitly defined (for example, whether to offer a handoff).\n- Some details are mixed between files, such as the Opera Windows example and the platform detail. That is acceptable but dense."},"latency_ms":5276,"stop_reason":"end_turn","usage":{"input_tokens":4491,"output_tokens":537},"cost_usd":0.014352} +{"config":"cookie-workflow","item":"a07","split":"heldout","phase":"old-a","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well-structured: it checks Aside first, gathers all decisions before any `$B` command, prepares the browser, imports, and reports in a fixed three-line format. The consent gates for `--clear-storage` and `--all` are explicit, and the guidance on distinguishing copied cookies from verified sign-in is strong. Error and receipt tables, platform limits, and privacy rules are all covered. Weaknesses: (1) Step 2 says to ask everything before any `$B` command, but step 3 says to set up verification env vars and possibly run `$B stop`, which is a mild ordering tension. (2) Step 4 sends the agent to BROWSER.md for errors, but it does not say what to do when `--verify-auth` is requested together with the picker if env vars are missing. (3) With `--all`, the picker/direct instructions don't say how to pass it, and the omitted-browser default of comet conflicts with the 'never treat default Comet as consent' rule only loosely. (4) Direct import allows one domain per run, but the case of multiple domains isn't spelled out (presumably repeat runs, each needing a prior `goto`). (5) Recovery after partial, failed, or storage-reset-failed outcomes is thin; storage-reset failure handling is not specified beyond reporting. (6) The `$B handoff` path and Windows recovery are mentioned, but the platform boundary section's 'offer headed manual sign-in only with a display' is somewhat scattered across the two files."},"latency_ms":4834,"stop_reason":"end_turn","usage":{"input_tokens":4491,"output_tokens":523},"cost_usd":0.014212} +{"config":"cookie-workflow","item":"f07","split":"heldout","phase":"new","sample":2,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"Strong on scoping and honest reporting, but there are workflow flaws. Step 2 forbids any `$B` command before the user decisions, yet step 3 only then sets up `$B`; the picker's behavior with `--domain` is unclear because the domain choice is never asked in step 2. The failure recovery \"run it again\" is vague and ignores the one-use link, platform limits and Chrome 136+ blocks. Direct import with no domain or `--all` is undefined, and the Aside probe is only referenced."},"latency_ms":9984,"stop_reason":"end_turn","usage":{"input_tokens":3705,"output_tokens":199},"cost_usd":0.0094} +{"config":"cookie-workflow","item":"p07","split":"heldout","phase":"old-b","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered. It asks for all decisions up front, checks Aside, prepares the daemon (env vars before start, `$B stop` if one is running), imports, and reports in a fixed three-line format. It is explicit that cookie counts and HTTP 200 are not proof of sign-in, and it keeps the consent and privacy rules clear. The error and failure tables and the platform limits are specific.\n\nThere are gaps and minor ambiguities:\n- The step 2 'ask before any `$B` command' rule conflicts slightly with step 1 reading the Aside probe, but the probe comes first and is not an import decision, so this is minor.\n- The picker-versus-direct choice is underspecified for cases where only some answers are known.\n- `--all` usage and its command syntax are not shown in SKILL.md.\n- Several cases lack a handling path: a failed `--clear-storage` on a non-Chromium target; what to do after `partial` or `failed` receipts, beyond a pointer to BROWSER.md; and a `--verify-auth` failure (what to do after 'not verified').\n- The example order of `goto` and the daemon start is slightly confusing, and the daemon-restart guidance is sound but could be clearer about when `$B stop` is needed.\n- The `--domain` flag is described as one per run, which makes multi-domain imports awkward.\n- BROWSER.md's direct-import example does not show the verification env or the target-tab constraints, though SKILL.md covers them.\n\nOverall it is strong and executable, with a few missing decision branches."},"latency_ms":5679,"stop_reason":"end_turn","usage":{"input_tokens":4733,"output_tokens":554},"cost_usd":0.015006} +{"config":"cookie-workflow","item":"f07","split":"heldout","phase":"old-b","sample":2,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The workflow is mostly clear on consent, scoping, and honest reporting. It separates 'not checked' from 'not verified' from 'verified'. It also states the platform limits and privacy boundaries well. Several gaps and ordering problems reduce how well an agent can execute it.\n\n1. Ordering contradiction: Section 2 says to ask everything before running any `$B` command. Section 1 already uses the Aside probe, and Section 3 runs `$B status`. Section 3 then ends the skill if the mode is cdp, after the user has answered all the questions.\n\n2. Missing details for the direct import and `--all`/domain choices. The domain is never asked about, although the goal mentions selecting a domain. The `--domain` flag is mentioned only in passing. Direct import gives no example syntax, such as how to pass the browser name, the domain, or `--all`. The target URL or tab is never specified, though verification and storage reset depend on it ('captured target', 'keep target open'). The `--all` flag is not included in the Section 4 direct import instructions.\n\n3. Weak recovery. 'If the import fails, run it again' conflicts with the warning that a failed reset is not rolled back, and gives no guidance on diagnosing the failure. There are no fallbacks for Windows App-Bound Encryption failures, and no handling for Chrome 136+. The pointer to BROWSER.md is only for 'profile labels and supported sources'.\n\n4. Verification-flow gaps. The picker lets the user tick verification, but the environment variables must be set before the daemon starts, and the ordering of that step against the picker is only implicit. The step for stopping a running daemon is clear. The case where verification is requested but the env vars are missing is stated only as 'rejects before mutation', with no instruction on what to do next.\n\n5. Cross-file lookups: the 'Browser fallback' and 'BROWSER SETUP' sections are external and not included here."},"latency_ms":13439,"stop_reason":"end_turn","usage":{"input_tokens":3308,"output_tokens":677},"cost_usd":0.013386} +{"config":"cookie-workflow","item":"p06","split":"heldout","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured. It asks for all decisions up front, ties each to a flag, and defines verification and the storage-reset scope precisely. The three-line report is clear, and the guidance on 'not checked' versus 'verified' is strong. Platform and privacy boundaries are explicit, and the BROWSER.md error and receipt tables give concrete recovery steps.\n\nThere are some gaps:\n- Decision 5 (`--all`) is asked about, but the direct-import step never says how to pass `--all` or how it interacts with the `$B goto` requirement.\n- Whether to open a target URL for `--all` is unspecified.\n- The one-domain-per-run rule conflicts with the user supplying several domains. The skill does not say to loop, or how to report multiple runs in the single `Imported` line.\n- The `Storage reset` line has no definition for a rejected reset on a non-Chromium engine.\n- Aside and the `$B` fallback are described only by reference to external files.\n- The ordering of `$B stop` against a running daemon is mentioned but not tied to a decision point.\n- `partial` and `empty` outcomes have no explicit next step.\n- The picker's `$B goto` requirement is mentioned only for direct import.\n- The headed handoff fallback is covered only partially."},"latency_ms":4838,"stop_reason":"end_turn","usage":{"input_tokens":4618,"output_tokens":463},"cost_usd":0.013866} +{"config":"cookie-workflow","item":"f07","split":"heldout","phase":"old-b","sample":1,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":3,"reasoning":"Strengths: the skill gets all four decisions from the user before running any command. It defines verified, not verified, and not checked precisely. The scoping of storage reset is clear (origin localStorage, target-tab sessionStorage, Chromium only, no rollback). Privacy rules are explicit, and the platform limits are documented.\n\nGaps and ambiguities:\n1. The domain and target are never asked about or specified. `--domain` appears only in BROWSER.md. The flow does not say how the target URL or tab is captured, or how the user supplies a domain. This is a significant omission given the stated goal of selecting a domain.\n2. Ordering conflict: step 2 says to ask before any `$B` command, but step 3 requires setting env vars and possibly running `$B stop` first. The section 4 picker also asks for verification inside the picker, which overlaps with the step 2 questions.\n3. \"If the import fails, run it again\" is vague and risks repeating a mutation. A failed reset may already have cleared storage and is not rolled back. There is no guidance on retry limits, on which failures are retryable, or on how to handle App-Bound Encryption failures (for example, whether to suggest Opera). The reference to BROWSER.md is only for \"labels and supported sources\".\n4. The direct import command lacks exact syntax. The browser flag form and the domain flag are unspecified, and the `--all` flow is not shown in section 4.\n5. The Aside probe and `Mode: cdp` end-the-skill branches are clear, but the cdp check happens after the user was already asked questions.\n6. The three-line report format is clear, but it does not say how to report when the receipt is missing or ambiguous."},"latency_ms":8113,"stop_reason":"end_turn","usage":{"input_tokens":3308,"output_tokens":593},"cost_usd":0.012546} +{"config":"cookie-workflow","item":"p08","split":"heldout","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: stop conditions come first, then decisions gathered up front, environment prep, import paths, recovery and honest reporting. Consent boundaries are explicit (--clear-storage and --all default to no and need approval; the default comet browser is not treated as consent). The three-line receipt report cleanly separates copied cookies from sign-in evidence. Recovery maps to concrete error tables and platform limits. Weaknesses: (1) The ordering is slightly contradictory. Step 1 says to ask before any `$B` command, but the Aside probe and `$B status` come earlier or later in a way that can leave decisions asked before a stop condition applies. The Aside probe is 'before anything else' but the decision step comes after it, which is mostly fine. (2) Direct import syntax is only partly specified in SKILL.md: the browser name goes positionally, and the Windows Opera example appears only in BROWSER.md. The 'one --domain per run' rule for multiple domains is vague. (3) Recovery for the `Mode: cdp` stop is placed inside step 2, after decisions were already asked. (4) The `--all` direct path and its interaction with the target and the goto are unspecified. (5) The source of the Aside probe is external, and the command for it is not shown. (6) What to do after the picker when verification fails is unclear. (7) `$B handoff` and `resume` are mentioned in the recovery tables but not tied back to the report. Overall it is strong but has a few gaps."},"latency_ms":5261,"stop_reason":"end_turn","usage":{"input_tokens":4587,"output_tokens":523},"cost_usd":0.014404} +{"config":"cookie-workflow","item":"p07","split":"heldout","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered. The Aside probe comes first, then all decisions are gathered before any `$B` command, then daemon prep, import, and a three-line report. It covers consent defaults, the daemon-environment-before-startup constraint, and the distinction between copied cookies and verified sign-in. It also has a concrete example, error tables, and platform limits. The main gaps are:\n\n- The `--all` command syntax is never shown in SKILL.md.\n- The direct-import flow doesn't say how to handle multiple domains beyond \"one --domain per run\", and the receipt format for several runs is unclear.\n- The Aside probe command is deferred to another file.\n- There is a minor ordering tension: the target is opened after the daemon starts, while the BROWSER.md example opens it before import.\n- Step 3.1 asks the user to configure variables privately, but the agent has no way to confirm they are set before the daemon starts.\n- It does not say what to do when the picker times out or the user abandons it, or how to report a partial import with failure reasons.\n- Fallbacks such as `handoff`/`resume` are covered only briefly.\n\nOverall it is strong but has a few small ambiguities."},"latency_ms":4380,"stop_reason":"end_turn","usage":{"input_tokens":4733,"output_tokens":431},"cost_usd":0.013776} +{"config":"cookie-workflow","item":"f08","split":"heldout","phase":"new","sample":2,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The workflow has clear structure, but it contradicts itself on key points. SKILL.md says to copy a Chrome 136+ profile to a temp folder and import from the copy, while BROWSER.md says there is no real-profile-copy workaround and closing Chrome does not help. SKILL.md also tells the agent to add the expected identity, profile name and cookie values to the report, which BROWSER.md forbids; it says to report only sanitized outcomes. Other gaps: how to pass a Dia Keychain retry, and the ordering of daemon setup versus the Aside check. These conflicts make the privacy and platform guidance unsafe to follow."},"latency_ms":3466,"stop_reason":"end_turn","usage":{"input_tokens":4922,"output_tokens":243},"cost_usd":0.012274} +{"config":"cookie-workflow","item":"f07","split":"heldout","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"Strong consent gating, a precise definition of verified, honest three-line reporting, and documented platform limits. Gaps: the domain choice is never asked in step 2 even though the task requires selecting a domain; step 4 says \"If the import fails, run it again\" with no limit or diagnosis, which conflicts with a failed reset that is not rolled back; and when verification is requested, the picker vs. direct flow is not fully specified."},"latency_ms":5889,"stop_reason":"end_turn","usage":{"input_tokens":3705,"output_tokens":171},"cost_usd":0.00912} +{"config":"cookie-workflow","item":"a08","split":"heldout","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The skill is well structured: it asks every decision up front, has a clear Aside/CDP short-circuit, defines the three-line honest report, and keeps the 'copied cookies are not sign-in evidence' rule consistent. Several gaps and one contradiction weaken it. (1) Verification env vars: SKILL.md says to set them 'in the shell that runs $B', but BROWSER.md says they must be in the daemon environment before startup and that a later CLI call does not reconfigure a running daemon. SKILL.md never says to stop or restart the daemon. Its 'return to step 3' recovery after a missing-settings rejection would therefore likely fail again. (2) Multiple domains: the direct import allows one `--domain` per run, but the target is a single page on one domain. It does not say how to navigate for the other domains or how to report partial multi-run results. (3) `--all` is approved in step 2, but step 4 never shows how to run it. It is unclear whether the picker offers `--all` or `--clear-storage`. (4) Handling for an unreadable profile, the choice between the picker and direct import for partial answers, and the receipt-to-report mapping, for example how `failed` or `partial` combine across runs, are thin. (5) Recovery paths (handoff/resume, retry bounds) are scattered between the two files, and the bounded database retry count is not given."},"latency_ms":8924,"stop_reason":"end_turn","usage":{"input_tokens":4484,"output_tokens":988},"cost_usd":0.018848} +{"config":"cookie-workflow","item":"p08","split":"heldout","phase":"new","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"Strong decision gating: all choices asked up front, no guessing, explicit consent for --all and --clear-storage, and a clear three-line report distinguishing not checked, not verified and verified. Gaps: the Aside probe precedes the decisions but its command lives elsewhere. Direct import syntax is only partly given in SKILL.md (the browser name is positional, shown in BROWSER.md). There is a mild tension between asking decisions before any `$B` command and running `$B status` for the cdp stop only afterward. There is no explicit handling for multiple domains beyond 'one --domain per run' and no instruction on how to read the receipt in direct mode. Recovery pointers are good."},"latency_ms":3668,"stop_reason":"end_turn","usage":{"input_tokens":4984,"output_tokens":255},"cost_usd":0.012518} +{"config":"cookie-workflow","item":"p06","split":"heldout","phase":"old-b","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: it gathers all decisions up front, defines the flags, and requires explicit consent for clear-storage and --all. The three-line report is precise, and the rules separating copied cookies from verified sign-in are clear. Recovery paths are documented in the error and receipt-failure tables and in the platform boundaries.\n\nThere are several gaps:\n- Conflicting ordering. Step 2 says to ask everything before any `$B` command, but step 3.1 has the user set the verification env vars afterward. The Aside probe in step 1 also runs before the questions.\n- The `--all` flag is asked in the decision table but is missing from the direct-import instructions in step 4 and from the picker flow.\n- Picker versus direct import for multiple domains is unclear, since `--domain` is one per run and the receipt report spans several domains.\n- Edge cases are not covered: an expired picker link is handled, but the exact behavior when the target domain differs from the import domain, and when verification is requested through the picker, is left unclear.\n- Error handling for `decryption_failed` says to close the browser, which sits awkwardly beside the Chrome 136 statement.\n- The Aside probe is delegated to another file, and the section cross-references are fairly dense.\n\nAn agent can mostly execute the workflow."},"latency_ms":4628,"stop_reason":"end_turn","usage":{"input_tokens":4618,"output_tokens":464},"cost_usd":0.013876} +{"config":"cookie-workflow","item":"p08","split":"heldout","phase":"old-a","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well structured: stop conditions, a single batched decision step, and a clear split between the picker and direct import. Verification setup, consent for --clear-storage and --all, and the three-line report format are all precisely defined. Receipt-vs-sign-in distinctions are stated consistently across both files. Weaknesses: (1) The ordering of the Aside probe and the `$B status` cdp check relative to the decisions step is slightly awkward, since the cdp check comes after asking the user. (2) The direct-import command is described in prose with no exact syntax, and BROWSER.md only shows a partial example. (3) `--all` is allowed as a decision, but it is unclear how it fits the direct path, given that it is mutually exclusive with a domain and the flow requires a domain and target. (4) The picker path has no clear way for the agent to read the receipt, and it is unclear how `--verify-auth` or `--clear-storage` get passed in the picker beyond the user ticking boxes. Clear-storage in the picker is not mentioned. (5) Multiple domains require multiple runs, but the report format does not say how to aggregate them. (6) Some recovery paths, such as `decryption_failed` with 'close the source browser', conflict somewhat with Chrome 136+ guidance. (7) The verification-missing recovery says to return to step 2, but the daemon-restart nuance is only implied."},"latency_ms":4823,"stop_reason":"end_turn","usage":{"input_tokens":4587,"output_tokens":490},"cost_usd":0.014074} +{"config":"cookie-workflow","item":"a08","split":"heldout","phase":"old-a","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well ordered and mostly unambiguous. All decisions are collected up front, with safe defaults and explicit consent for `--clear-storage` and `--all`. The Aside and CDP short-circuits are clear. The three-line report format separates copied cookies from verified sign-in. Platform and privacy limits are stated. Gaps remain. SKILL.md says to set the verification variables 'in the shell that runs `$B`', but BROWSER.md says they must be in the daemon environment before startup. The retry path 'return to step 3' never says to restart an already-running daemon, so a retry could fail again. The picker flow does not say where the agent gets the receipt after the user reports done. Direct import does not mention the `--all` syntax. Multiple domains are handled only by 'one `--domain` per run', which leaves the repeated-run handling and aggregated report unspecified. Recovery steps for failures such as `decryption_failed` and `unsupported_encryption` are in BROWSER.md, but the entry file only points to them."},"latency_ms":8224,"stop_reason":"end_turn","usage":{"input_tokens":4484,"output_tokens":913},"cost_usd":0.018098} +{"config":"cookie-workflow","item":"f06","split":"heldout","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The high-level flow is readable and the safety rules are strong: no guessing the browser or profile, explicit consent for storage clearing and full cookie copy, and the point that cookie counts never prove sign-in. But the workflow is too vague to execute, and some instructions conflict or are missing. (1) The 'Check for Aside' step is undefined, and it is unclear how to tell whether the user already has a signed-in browser. (2) There are no concrete commands, tool names, flags, or syntax for the interactive chooser or the direct import, or for launching gstack's browser. 'Get gstack's browser running' says nothing about how. (3) The 'verification settings' are not defined: no element selector, no expected identity, no config location, and no way to keep them private. The docs also say to configure verification before the browser starts, which conflicts with SKILL.md's order of starting the browser and then configuring if wanted, so the sequence is ambiguous. (4) Question 2 asks which sites and which page, while BROWSER.md says to open a page on the site first. How multiple sites interact with a single opened page is unclear. (5) Supported browsers, platforms, and profile directory discovery are not listed. (6) The 'copy every cookie' consent is not tied to any parameter. (7) Failure handling is thin: the receipt is mentioned but its format is not, and the only fallback is manual sign-in. The partial-failure behavior of storage clearing is noted, but not how to warn the user or recover. (8) Reporting does not say how to state 'verified' versus 'not verified' or what to do when the check fails."},"latency_ms":5752,"stop_reason":"end_turn","usage":{"input_tokens":1414,"output_tokens":550},"cost_usd":0.008328} +{"config":"cookie-workflow","item":"f06","split":"heldout","phase":"new","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The workflow has sound intent: ask first, default to no verification, require consent, and report honestly. But it is vague on execution. Step 3 says to configure verification before the browser starts, yet gives no setting names, no command to start the browser, and no way to supply the expected identity. Step 4 mentions the \"interactive chooser or direct form\" with no syntax, and no way to list profiles by directory. The consent scope (\"only that site\") is stated, but the \"copy every cookie\" option is never explained in the import step. Failure recovery, \"the receipt explains why\", is thin, and the platform limits are not enumerated."},"latency_ms":3866,"stop_reason":"end_turn","usage":{"input_tokens":1811,"output_tokens":233},"cost_usd":0.005952} +{"config":"cookie-workflow","item":"f08","split":"heldout","phase":"old-b","sample":2,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":3,"reasoning":"The workflow is well structured: ask all decisions first, check Aside and CDP, configure verification before the daemon starts, get consent for the storage reset, and report three separate states. But several instructions contradict each other or are underspecified.\n\n1. The report section tells the agent to add the expected identity, profile name, and the first few imported cookie names and values. BROWSER.md says not to paste cookie values, account labels, or the expected identity into public logs, and to report only sanitized outcomes. This is a serious privacy contradiction.\n\n2. Chrome 136+ guidance conflicts. SKILL.md says to close Chrome, copy the profile to a temp folder, and import from the copy. BROWSER.md says closing Chrome does not remove the protection and that there is no real-profile-copy workaround. The same conflict applies to Windows App-Bound Encryption, where SKILL.md says to use a copy but BROWSER.md says to use handoff or manual sign-in. The agent cannot tell which is right, and following SKILL.md may violate the documented boundaries.\n\n3. The Keychain instruction says to keep rerunning until approval goes through. That is an unbounded retry, and it conflicts with the 'recover within boundaries' goal.\n\n4. The ordering is muddled. Step 2 says no `$B` commands before asking, yet it also says to use the picker when answers are open, and step 3 sets the env vars only after the user answers. The `$B stop` instruction is clear, but 'Direct import' lacks the exact flag syntax for browser and profile. The picker also needs the user to supply the selector and identity, and it is unclear how those are echoed back safely, given the privacy rule against logging the expected identity.\n\n5. Direct import is limited to one `--domain` per run, so multi-domain handling is only implied. The `--all` handling after consent is not described in the import step. The `handoff`/`resume` fallback is mentioned only in BROWSER.md, and SKILL.md's 'headed manual sign-in as a last resort' is vague."},"latency_ms":6972,"stop_reason":"end_turn","usage":{"input_tokens":4525,"output_tokens":707},"cost_usd":0.01612} +{"config":"cookie-workflow","item":"f08","split":"heldout","phase":"old-b","sample":0,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":3,"reasoning":"The workflow is well structured: it gets decisions from the user first, has an Aside check, consent gates, and a three-line report. It also has serious contradictions and unsafe or unsupported instructions. (1) SKILL.md says to copy the Chrome profile to a temp folder to get around Chrome 136+ protection, while BROWSER.md says closing Chrome does not remove the protection and that there is no real-profile-copy workaround. The agent cannot tell which to follow, and the copy approach also bypasses a documented boundary. (2) SKILL.md tells the agent to include the first few cookie names and values in the report, and to include the expected identity and profile name. BROWSER.md says never to expose cookie values, profile labels or the identity, and to report only sanitized outcomes. This is a privacy contradiction. (3) The Keychain instruction to keep rerunning until approval goes through is unbounded and conflicts with the idea of recovering within boundaries. (4) Ordering is awkward: the env vars are set in step 3 but the Aside probe and `$B` discovery come in different places. The step 4 error path returns to step 3 without saying where the values come from. (5) `--all` is asked about in step 2, but the direct-import instructions never say how to invoke it. Direct import also requires `goto` first, and the picker versus direct decision is only partly defined. (6) The mix of the user pasting an identity into chat and BROWSER.md's 'privately' is another tension. Detailed error tables, receipt reasons and platform limits are otherwise good."},"latency_ms":6308,"stop_reason":"end_turn","usage":{"input_tokens":4525,"output_tokens":539},"cost_usd":0.01444} +{"config":"cookie-workflow","item":"p08","split":"heldout","phase":"new","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"Strong decision gate: all choices are asked up front, with explicit consent for --clear-storage and --all, and a clear three-line report that separates copied cookies from verified sign-in. Gaps: the Aside probe runs before decisions, but 'before any $B command' is consistent. Direct import says 'one --domain per run' yet decision 2 allows multiple domains, with no loop or ordering guidance. The picker's handling of verification and storage reset is vague. The recovery order for failed storage resets is unclear, and the SKILL.md reference to the 'Browser fallback' section is external."},"latency_ms":3117,"stop_reason":"end_turn","usage":{"input_tokens":4984,"output_tokens":223},"cost_usd":0.012198} +{"config":"cookie-workflow","item":"p08","split":"heldout","phase":"old-b","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"Strengths: the workflow is well structured, with stop conditions, a batched decision step that forbids guessing, and a clear picker vs. direct split. Consent is scoped precisely for --clear-storage and --all. The three-line report distinguishes copied cookies from verified sign-in, and recovery tables cover errors and platform limits.\n\nWeaknesses:\n- Ordering is slightly ambiguous. The Aside probe runs before everything, but it depends on reading browse/SKILL.md and on `$B`. The `$B status` stop condition sits in step 2, after the daemon may have been stopped and env vars set.\n- Direct import allows only one domain per run, but the instructions do not say how to handle multiple domains. They also do not say how the target tab interacts with several domains.\n- Direct import does not say how --all is invoked, how it combines with the browser and profile, or that it requires no target.\n- Picker mode with verification ticked requires env vars set beforehand. The text implies this but does not state it clearly.\n- The omitted-browser default of comet is mentioned, but the instruction for when the user declines to answer is thin.\n- The \"Mode: cdp\" stop could leave the user with unclear next steps.\n- The receipt format is described only abstractly, and the \"Failure reasons\" table does not map to the report lines.\n- Profile directory discovery when the user knows only the display name is left to the picker."},"latency_ms":4869,"stop_reason":"end_turn","usage":{"input_tokens":4587,"output_tokens":499},"cost_usd":0.014164} +{"config":"cookie-workflow","item":"f07","split":"heldout","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"Strengths: SKILL.md gives a clear decision flow. It asks the user for browser, profile, verification, storage reset and --all up front. It sets safe defaults, requires explicit consent for the risky options, and defines precisely what clearing storage touches. The three-line report distinguishes copied cookies from verified sign-in, and the privacy rules are clear. BROWSER.md adds detail on profiles, Opera on Windows, and platform limits.\n\nWeaknesses:\n1. Ordering contradiction: Step 2 says to ask all questions before running any `$B` command. Step 3 then has the user set env vars and start the daemon, but the Aside probe in step 1 and the `$B status` check in step 3 can come before or after those decisions. The `$B stop` instruction also runs before step 3.2 locates `$B`.\n2. The Aside probe and Browser fallback live in external files (`browse/SKILL.md`), which is acceptable, but the exact probe command is not given.\n3. `--all` and domain handling are underspecified. The skill never says how to choose or pass the target domain for a direct import, and the question 4 wording on `--all` conflicts with the domain flag. It also does not say how to supply the target URL or tab for storage reset and verification.\n4. The direct-import command is vague: it does not show the browser-name syntax or the flag order.\n5. Failure handling is thin. \"If the import fails, run it again\" conflicts with the warnings that a failed reset is not rolled back and that the picker link is single-use. It gives no guidance on which failures are retryable, such as missing config or a Windows App-Bound Encryption failure.\n6. The picker flow does not say how to pass `--all` or `--clear-storage`.\n7. The \"Comet default is not consent\" caveat is clear, but the platform-unavailable fallback path is only implicit.\n8. Verification setup before the user answers is slightly contradictory: the env vars are set privately, yet the selector and identity are supplied by the user, with no safe way for the agent to confirm they exist."},"latency_ms":6879,"stop_reason":"end_turn","usage":{"input_tokens":3308,"output_tokens":717},"cost_usd":0.013786} +{"config":"cookie-workflow","item":"f06","split":"heldout","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The high-level flow is readable and the safety boundaries are good: ask the user rather than guess, default to no for clearing storage, copying every cookie and sign-in checking, report honestly, and never expose values. But the workflow is too vague to execute. 'Check for Aside' is undefined, and 'gstack's browser', 'cookie import', 'the interactive chooser' and 'the direct form' are given with no commands, flags or tool names. There is no way to enumerate browsers or profiles, or to map a display name to a profile directory. Verification setup is described only as 'verification settings privately', with no explanation of how to configure the element and expected identity or how that is done 'before the browser starts'. SKILL.md step 3 also puts browser startup after the verification config, but it does not say what to do if the browser is already running. The 'copy every cookie' consent in step 2 has no corresponding step 4 mechanism and no description of what a scoped import means. Storage clearing is described only as 'for that site', and the partial-failure behavior is not given a recovery path. Failure handling is thin: 'the receipt explains why' with no receipt format, and 'sign in manually' is the only fallback. Platform limits are mentioned but not listed. The report step does not say what to do when verification fails. The order is also inconsistent: SKILL.md says to open the page in step 3, while BROWSER.md says to open it before a direct import."},"latency_ms":4491,"stop_reason":"end_turn","usage":{"input_tokens":1414,"output_tokens":489},"cost_usd":0.007718} +{"config":"cookie-workflow","item":"f08","split":"heldout","phase":"old-a","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The workflow is well structured: gather decisions first, check Aside, configure verification, import, then report honestly. Several defects remain.\n\n1) Direct contradiction on Chrome 136+. SKILL.md tells the agent to close Chrome, copy the profile directory to a temp folder, and import from the copy. BROWSER.md says closing Chrome does not remove the protection and that there is no real-profile-copy workaround. An agent cannot follow both.\n\n2) Privacy conflict in reporting. SKILL.md step 5 tells the agent to include the expected identity, profile name, and the first few imported cookie names and values in the report. BROWSER.md says never to expose cookie values, profile/account labels, or expected identity, and to report only sanitized outcomes. This contradiction is serious, and the cookie-value disclosure is unsafe.\n\n3) Missing decisions and ordering problems. Step 3 has the agent open the target URL, but the verification env vars are set before the daemon starts and `$B stop` is needed. The ordering is fine, but echoing the values back may leak them. SKILL.md also says to keep rerunning the Keychain prompt \"until it goes through\", which has no retry limit. There is no guidance on handling a `--all` import (the direct-import description omits it), or on how to handle multiple domains, since the direct command takes one `--domain` per run. The \"Aside probe\" and \"Browser fallback\" are defined elsewhere.\n\n4) Other gaps. The \"headed manual sign-in only as a last resort\" fallback is covered by the handoff steps, and the error and failure tables are useful. How the picker's completion is detected is only loosely specified.\n\nOverall the structure is usable, but the two contradictions and the unsafe reporting instruction lower all three scores."},"latency_ms":6991,"stop_reason":"end_turn","usage":{"input_tokens":4525,"output_tokens":634},"cost_usd":0.01539} +{"config":"cookie-workflow","item":"f06","split":"heldout","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The high-level flow is understandable: ask the user first, default to no for verification, storage clearing and copying all cookies, and report honestly. Never guess the browser or profile is a good rule. But the instructions are too vague to execute without guessing. Missing or ambiguous items: (1) 'Check for Aside' is never defined, and neither is how to detect a signed-in browser. (2) No commands, tools, flags or parameters are given for starting gstack's browser, the interactive or direct cookie import, or the sign-in check. 'gstack's browser' and the 'cookie import' are not named concretely. (3) The sign-in verification setup is unspecified: what settings, where they go, how the user sets them privately, what the expected identity is, or which page element to compare. BROWSER.md says to configure it before the browser starts, which conflicts with SKILL step 3's order (start the browser, then set up verification). (4) The scope of 'copy every cookie' is unclear, and so is what is copied by default, such as the per-site domain selection. (5) Failure handling is thin: 'the receipt explains why' with no receipt format, and the only fallback is manual sign-in. (6) The storage-clearing consent prompt is not worded, though it is scoped to the site, and the irreversibility on partial failure is only in BROWSER.md. (7) Platform support lists are not given. (8) The report format is loosely defined, and it does not say how to report verification failure."},"latency_ms":5321,"stop_reason":"end_turn","usage":{"input_tokens":1414,"output_tokens":499},"cost_usd":0.007818} diff --git a/test/fixtures/judge-calibration/cookie-workflow/runs/comparison-1.stop.json b/test/fixtures/judge-calibration/cookie-workflow/runs/comparison-1.stop.json new file mode 100644 index 000000000..6339c6830 --- /dev/null +++ b/test/fixtures/judge-calibration/cookie-workflow/runs/comparison-1.stop.json @@ -0,0 +1 @@ +{"stopped":null} diff --git a/test/fixtures/judge-calibration/manifest.json b/test/fixtures/judge-calibration/manifest.json index 14890297b..139cfadd7 100644 --- a/test/fixtures/judge-calibration/manifest.json +++ b/test/fixtures/judge-calibration/manifest.json @@ -92,6 +92,19 @@ "estimated_cost_usd": 20.36745, "cumulative_usd": 111.8289, "within_budget": false + }, + { + "config": "cookie-workflow", + "priority": 8, + "cases": 1, + "model": "claude-sonnet-5-5", + "items": 24, + "calls": 216, + "input_tokens": 894807, + "estimated_output_tokens_per_call": 1200, + "estimated_cost_usd": 4.381614, + "priced_at": "2026-10-25T16:58:41.883Z", + "funding": "separate approval (GStack Minder, 2026-10-25; CREW 5), outside the W2 $115 reservation" } ] } diff --git a/test/judge-calibration.test.ts b/test/judge-calibration.test.ts index ac56d7dfb..9cc7f13b2 100644 --- a/test/judge-calibration.test.ts +++ b/test/judge-calibration.test.ts @@ -4,6 +4,7 @@ import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; import { callJudge, JudgeRefusalError, type CallJudgeOptions, type JudgeResponseMeta } from './helpers/llm-judge'; +import { buildCookieWorkflowJudgeInput, COOKIE_WORKFLOW_JUDGE } from './helpers/cookie-workflow-judge-input'; import { analyze, classifyError, loadCorpus, undispatched, PANEL_SAMPLES, PHASES, planCalls, priceManifest, runCalls, runSample, type CalibrationConfig, type CorpusItem, type JudgeCall, type SampleRecord, @@ -201,8 +202,8 @@ describe('judge calibration harness (free, stubbed provider)', () => { describe('committed calibration configurations and corpora', () => { test('comparison (1) changes only the JSON schema; prompts come from the eval builders', async () => { const { CALIBRATION_CONFIGS } = await import('../scripts/judge-calibration-configs'); - expect(CALIBRATION_CONFIGS.map(config => config.id)).toEqual(['arm', 'qa-workflow', 'qa-health-rubric', 'qa-anti-refusal', 'cross-skill', 'voice', 'workflow-default']); - expect(CALIBRATION_CONFIGS.map(config => config.priority)).toEqual([1, 2, 3, 4, 5, 6, 7]); + expect(CALIBRATION_CONFIGS.map(config => config.id)).toEqual(['arm', 'qa-workflow', 'qa-health-rubric', 'qa-anti-refusal', 'cross-skill', 'voice', 'workflow-default', 'cookie-workflow']); + expect(CALIBRATION_CONFIGS.map(config => config.priority)).toEqual([1, 2, 3, 4, 5, 6, 7, 8]); for (const config of CALIBRATION_CONFIGS) { const { jsonSchema, ...rest } = config.newOptions; expect(jsonSchema).toBeTruthy(); @@ -232,6 +233,18 @@ describe('committed calibration configurations and corpora', () => { } }); + // /ship Step 7 coverage audit (GSTA-91). + // Value: protects=the cookie-workflow calibration sends the eval's exact prompt for the same bundle; + // fails_when=its build drops COOKIE_WORKFLOW_JUDGE context/goal or adds a capability the eval does not send; + // why_new=the corpus test only checks the bundle is contained in the prompt; seam=none; + // no_claim=provider behavior (no request is sent; both prompts come from the live SKILL.md and BROWSER.md) + test('the cookie-workflow calibration builds the prompt the cookie eval sends', async () => { + const { CALIBRATION_CONFIGS } = await import('../scripts/judge-calibration-configs'); + const cookie = CALIBRATION_CONFIGS.find(config => config.id === 'cookie-workflow')!; + const production = buildCookieWorkflowJudgeInput(path.join(import.meta.dir, '..')); + expect(cookie.build({ bundle: production.text })).toBe(production.prompt); + }); + test('panel verdicts restate each eval pass rule', async () => { const { CALIBRATION_CONFIGS } = await import('../scripts/judge-calibration-configs'); const byId = Object.fromEntries(CALIBRATION_CONFIGS.map(config => [config.id, config])); @@ -242,6 +255,12 @@ describe('committed calibration configurations and corpora', () => { expect(byId[id]!.verdict([qa(5, 5, 4), qa(5, 5, 3), qa(5, 5, 3)])).toBe('fail'); expect(() => byId[id]!.validate({ ...qa(3, 3, 4), clarity: '3' })).toThrow(); } + expect(byId['cookie-workflow']!.verdict([qa(4, 3, 4), qa(4, 3, 4), qa(4, 3, 4)])).toBe('pass'); + expect(byId['cookie-workflow']!.verdict([qa(5, 5, 4), qa(5, 5, 4), qa(3, 2, 3)])).toBe('pass'); + expect(byId['cookie-workflow']!.verdict([qa(4, 3, 4), qa(3, 3, 4), qa(3, 3, 4)])).toBe('fail'); + expect(byId['cookie-workflow']!.verdict([qa(5, 2, 5), qa(5, 2, 5), qa(5, 3, 5)])).toBe('fail'); + expect(byId['cookie-workflow']!.verdict([qa(5, 5, 4), qa(5, 5, 3), qa(5, 5, 3)])).toBe('fail'); + expect(() => byId['cookie-workflow']!.validate({ ...qa(4, 3, 4), clarity: '4' })).toThrow(); const arm = (over_engineering: number) => ({ over_engineering, construct: over_engineering ? 'helper' : 'none', reasoning: '' }); expect(byId.arm!.verdict([arm(0), arm(1), arm(3)])).toBe('pass'); expect(byId.arm!.verdict([arm(1), arm(2), arm(2)])).toBe('fail'); @@ -277,23 +296,45 @@ describe('landed schemas match the recorded calibration', () => { expect(sent).toBe(recorded.landing.lands); }); + // /ship Step 7 coverage audit (GSTA-91). + // Value: protects=the held cookie-workflow result is what analyze() derives from its committed labels, samples, stop record and request options; + // fails_when=result.json is hand-edited, a corpus label moves after the run, a sample is altered, or the config's model/options drift; + // why_new=no test reads samples.jsonl or stop.json, and the wiring test trusts result.json's landing verbatim; seam=none; + // no_claim=that the recorded samples are genuine provider output (the committed JSONL stands in for the paid run), or that prompt + // text and bundle bytes are unchanged since the run (request_hash covers model and options only) + test('the recorded cookie-workflow result is reproducible from its committed labels, samples, stop record and request options', async () => { + const { CALIBRATION_CONFIGS } = await import('../scripts/judge-calibration-configs'); + const cookie = CALIBRATION_CONFIGS.find(config => config.id === 'cookie-workflow')!; + const runs = path.join(import.meta.dir, 'fixtures', 'judge-calibration', 'cookie-workflow', 'runs'); + const samples: SampleRecord[] = fs.readFileSync(path.join(runs, 'comparison-1.samples.jsonl'), 'utf8').trim().split('\n').map(line => JSON.parse(line)); + const { stopped } = JSON.parse(fs.readFileSync(path.join(runs, 'comparison-1.stop.json'), 'utf8')); + expect(JSON.parse(JSON.stringify(analyze(cookie, loadCorpus('cookie-workflow'), samples, stopped)))).toEqual(result('cookie-workflow')); + }); + test('armJudge sends ARM_JUDGE_SCHEMA because its calibration landed', () => { expect(result('arm').landing.lands).toBe(true); expect(judgeSource).toContain('buildArmJudgePrompt(task, diff), ARM_JUDGE_MODEL, { jsonSchema: ARM_JUDGE_SCHEMA })'); }); - test('workflow transport follows the workflow-default calibration; uncalibrated configurations stay on today\'s request', () => { + test('workflow transport follows the workflow-default and cookie-workflow calibrations; uncalibrated configurations stay on today\'s request', () => { const recorded = result('workflow-default'); expect(recorded.complete).toBe(true); + const cookie = result('cookie-workflow'); + expect(cookie.complete).toBe(true); const registrations = [...evalSource.matchAll(/await runWorkflowJudge\(\{([\s\S]*?)\n \}\);/g)].map(match => match[1]!); expect(registrations.length).toBeGreaterThan(14); for (const body of registrations) { const testName = /testName: '([^']+)'/.exec(body)![1]; const defaultConfig = !/^\s+(?:agentCapability|model|readInput|thresholds|maxTokens|stream|effort):|\.\.\.COOKIE_WORKFLOW_JUDGE/m.test(body); - const expected = testName === 'ship/SKILL.md workflow' || (defaultConfig && recorded.landing.lands); + const expected = testName === 'ship/SKILL.md workflow' || (defaultConfig && recorded.landing.lands) + || (testName === 'setup-browser-cookies/SKILL.md workflow' && cookie.landing.lands); expect(body.includes('schemaTransport: true'), testName).toBe(expected); expect(body.includes('compactReasoning: true'), testName).toBe(testName === 'ship/SKILL.md workflow'); } - expect(registrations.filter(body => body.includes('schemaTransport: true'))).toHaveLength(15); + expect(registrations.filter(body => body.includes('schemaTransport: true'))).toHaveLength(cookie.landing.lands ? 16 : 15); + const cookieBody = registrations.find(body => body.includes("testName: 'setup-browser-cookies/SKILL.md workflow'"))!; + expect(cookieBody).toContain('...COOKIE_WORKFLOW_JUDGE'); + expect(cookieBody).not.toMatch(/^\s+(?:agentCapability|model|thresholds|maxTokens|stream|effort):/m); + expect(Object.keys(COOKIE_WORKFLOW_JUDGE).sort()).toEqual(['judgeContext', 'judgeGoal', 'model', 'thresholds']); }); });