test: record qa workflow judge calibration; schema held (W2 T8)

Comparison (1), 24 items: old-vs-old 2/24 flips, new-vs-old 4/24 (above
old + 1), agreement 17/24 for both prompts, one new false pass (a06, an
adjudicated adjacent item at the actionability-4 line), no errored
samples, p95 27.7 s new vs 26.8 s old. The judge stays on today's path
(no schema). The failed landing condition triggers the UC3 prose step;
it is deferred to TODOS because the W2 reservation is spent on the
remaining uncalibrated configurations first. 216 calls, $18.94.

test/fixtures/judge-calibration/qa-workflow/runs/comparison-1.result.json

@@ -0,0 +1,98 @@
+{
+  "config": "qa-workflow",
+  "model": "claude-fable-5-1",
+  "request_hash": {
+    "old": "c00326001bb0e5511ead24b7568cfeafc3a9ae4fe4a4f94a1ed5e556fae12497",
+    "new": "07ca93d3097140d5f7b4b0dcffff84f2eb20e5ddc4de92fe7693e6e1a011da81"
+  },
+  "complete": true,
+  "stopped": null,
+  "calls": 216,
+  "cost_usd": 18.9415,
+  "errors": {
+    "old": {
+      "refusal": 0,
+      "truncation": 0,
+      "parse": 0,
+      "schema": 0,
+      "transport": 0,
+      "samples": 144,
+      "rate": 0
+    },
+    "new": {
+      "refusal": 0,
+      "truncation": 0,
+      "parse": 0,
+      "schema": 0,
+      "transport": 0,
+      "samples": 72,
+      "rate": 0
+    }
+  },
+  "latency_p95_ms": {
+    "old": 26754,
+    "new": 27693
+  },
+  "all": {
+    "items": 24,
+    "old_flips": {
+      "flips": 2,
+      "compared": 24
+    },
+    "new_flips": {
+      "flips": 4,
+      "compared": 24
+    },
+    "agreement": {
+      "old-a": 17,
+      "old-b": 17,
+      "new": 17
+    },
+    "false_passes": {
+      "old-a": [],
+      "old-b": [],
+      "new": [
+        "a06"
+      ]
+    },
+    "new_false_passes": [
+      "a06"
+    ]
+  },
+  "heldout": {
+    "items": 8,
+    "old_flips": {
+      "flips": 1,
+      "compared": 8
+    },
+    "new_flips": {
+      "flips": 1,
+      "compared": 8
+    },
+    "agreement": {
+      "old-a": 5,
+      "old-b": 6,
+      "new": 6
+    },
+    "false_passes": {
+      "old-a": [],
+      "old-b": [],
+      "new": []
+    },
+    "new_false_passes": []
+  },
+  "landing": {
+    "lands": false,
+    "reasons": [
+      "new flips 4/24 > old flips 2/24 + 1",
+      "new false passes: a06"
+    ],
+    "strict_no_flip_increase": false
+  },
+  "prose_step_triggered": {
+    "triggered": true,
+    "reasons": [
+      "a landing condition failed"
+    ]
+  }
+}

test/fixtures/judge-calibration/qa-workflow/runs/comparison-1.samples.jsonl

@@ -0,0 +1,216 @@
+{"config":"qa-workflow","item":"f01","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":2,"reasoning":"The document is internally contradictory and structurally broken. Phase 7 defines a tier table (Quick/Standard/Exhaustive) then immediately overrides it with 'Fix every issue... whatever tier was selected' plus a vague 'if time is short' escape hatch, leaving the agent with no executable triage decision. Phase 8 has two steps labeled '8a', with Commit placed before Locate/Fix, so the step order is nonsensical. Phase 9 (the 'Verify' in the title) is entirely absent — no recheck, no regression test, no health-score recompute — yet rule 13 references a nonexistent 'Phase 8e.5' and rule 14 presumes a regression-detection step that is never defined. Rule 15 cites an undefined 'WTF-likelihood heuristic'. Phase 10 'Write up what you did' specifies no report structure, evidence, or coverage deliverable. The clean-tree gate tells the agent to 'continue with setup' but no setup section exists. The learnings-refresh and TODOS.md sections are the only well-specified pieces. An agent cannot reliably produce the expected fix-and-verify deliverables from this bundle."},"latency_ms":11891,"stop_reason":"end_turn","usage":{"input_tokens":1946,"output_tokens":800},"cost_usd":0.05946}
+{"config":"qa-workflow","item":"a01","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"Within its own scope (triage → fix loop → final QA → report) the document is reasonably clear: tiers, commit format, classification states (verified/best-effort/reverted/deferred), revert rule, and regression-test structure are concrete and executable. However, evaluated as the complete workflow it is a fragment. It depends on undefined artifacts and terms: Phases 1-6 and the 'standard report template', the 'WTF-likelihood heuristic' (invoked twice, never defined), `$B`/`$REPORT_DIR`, 'detected test command', 'user declined bootstrap', and the 'top-of-skill learnings pull'. The title promises Phases 7-11 but Phase 11 is absent (the 'Additional Rules' are numbered 11-15, creating confusion with phase numbering). Health-score computation and 'affected pages' are never specified. Nothing addresses native functional or mixed surfaces, authority/isolation boundaries, native contracts, conditional browser/DX loading, or blocked paths—only browser screenshots/console are covered. The regression test is added after the fix commit rather than before repair, and 8e's regression check happens before the regression test exists. Deliverables (report paths, per-issue fields, PR summary) are well-defined, so an agent could produce output for browser-only fixes, but it would need to invent or guess substantial missing context."},"latency_ms":14101,"stop_reason":"end_turn","usage":{"input_tokens":3436,"output_tokens":906},"cost_usd":0.07966}
+{"config":"qa-workflow","item":"f01","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":2,"reasoning":"The bundle is internally contradictory and structurally broken. Phase 7 first says tiers determine which issues get fixed, then instructs to fix every issue regardless of tier (and 'only critical if time is short'), leaving the agent with no executable triage decision. Phase 8 has two steps labeled '8a', with 'Commit' placed before 'Locate source' and 'Fix', so the loop commits before any change exists. Steps 8c–8e, the referenced 'Phase 8e.5' regression-test step, and the entire Phase 9 (verify/recheck) are missing despite the title 'Fix → Verify' and a 'revert on regression' rule that has no defined regression check. Phase 10 'Write up what you did' specifies no report structure, no post-fix health score, no evidence or coverage output. Rules are numbered 11–15 and cite an undefined 'WTF-likelihood heuristic' and prior rules not in the bundle. Nothing addresses authority, isolation, native functional contracts, conditional browser/DX loading, or blocked paths. Only the clean-tree gate, commit format, minimal-fix guidance, learnings keyword rules, and TODOS.md update are clear and executable."},"latency_ms":15193,"stop_reason":"end_turn","usage":{"input_tokens":2307,"output_tokens":1040},"cost_usd":0.07507}
+{"config":"qa-workflow","item":"f05","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The bundle is structured sensibly (parameter parsing, optional context lookups, surface selection via a shared exploratory STOP, owned REPORT_DIR/REPORT_FILE resolution, conditional browser setup, charter-first probing) and defines authority/isolation/blocked-path handling in reasonable detail. However, it contains a direct, unresolved contradiction at its core: the 'Run the Selected Checks' section instructs the agent to fix critical/high bugs in source and commit them, while the header and Additional Rule 1 state NEVER fix, never edit product code, never commit. An agent cannot execute both; this undermines the report-only contract and makes the central decision ambiguous. Completeness also suffers: the index lists a reporting section for finalizing the report, but no body phase invokes it or defines when probing stops and reporting begins; the native/functional method contracts (Full/Quick/Regression semantics, probe bounds, evidence format) live entirely in external files not shown, so the agent must rely on them for most executable decisions. Minor issues: the browser-setup path lives under a different skill (qa/gstack-qa) than the caller, and mixed-regression baseline resolution is described but not operationally specified. The contradiction alone drops clarity and actionability substantially."},"latency_ms":15569,"stop_reason":"end_turn","usage":{"input_tokens":3247,"output_tokens":1026},"cost_usd":0.08377}
+{"config":"qa-workflow","item":"p05","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"The document is a tightly phased, linear browser-only workflow with named outputs, concrete thresholds (severity, scoring weights, stop conditions, fix caps) and explicit gates, so an agent can follow it with little ambiguity. Minor clarity gaps: the Mode default is undefined when no URL is given and the branch is not a feature branch; `--quick` is overloaded as both Mode and Fix tier; `<base>` in the diff command is unspecified; and issue-to-category assignment (needed for scoring) is never defined. Completeness is weaker against the stated frame: there is no treatment of native functional or mixed surfaces, no conditional section loading, no authority/isolation contracts, and no blocked-path handling—everything is browser-centric. Within its own scope it also lacks procedures for link checking, accessibility/performance probing, and what 'untested category' means. Actionability is strong: deliverables (report.md, baseline.json, screenshots/, commits) are precisely specified and the fix loop (reproduce → failing test → minimal fix → re-test → classify → commit) is executable as written."},"latency_ms":16394,"stop_reason":"end_turn","usage":{"input_tokens":2610,"output_tokens":1044},"cost_usd":0.0783}
+{"config":"qa-workflow","item":"f03","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":2,"reasoning":"The bundle has several well-formed fragments (parameter table, evidence tiers, Important Rules, SPA tip) but is riddled with hollow and contradictory sections. Phase 2 has three conditional headers ('If the user specified auth credentials:', 'If the user provided a cookie file:') with no body; Phase 3 'Get a map of the application:' and Phase 4 'At each page:' / 'Responsiveness... if relevant:' trail off with no commands; Phase 6 is one sentence. The Setup table references diff-aware and `--regression` modes 'see Modes below', yet the Modes section only says 'pick whatever depth seems right', defining neither. Tiers 'determine which issues get fixed' but no fix, regression-before-repair, recheck, or diagnosis phase exists; Rule 5 'Never read source code' conflicts with diff-aware mode and any repair work. Undefined tokens (`NEEDS_SETUP`, `$B`, `<SKILL_DIR>`, 'baseline health score') and the referenced taxonomy/template files are not provided. Relative to the stated scope—browser/native/mixed surface selection, bounded probes, native contracts, isolation, authority, conditional browser/DX loading, blocked paths, coverage reporting—nearly all of it is absent. An agent could stumble through a basic browser smoke pass but could not authenticate, run regression mode, test native surfaces, add regressions, repair, or produce the expected coverage deliverables."},"latency_ms":18561,"stop_reason":"end_turn","usage":{"input_tokens":2472,"output_tokens":1225},"cost_usd":0.08597}
+{"config":"qa-workflow","item":"a03","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The phases are sequenced and the aside repl scripts are concrete, with explicit alert thresholds, a two-consecutive-check rule, a copy-out step for screenshots, and a defined baseline manifest schema. However, several decisions are left ambiguous or undefined: the `--quick` mode is listed but never specified; Phase 6 'Health Report' has no format, required fields, or coverage/evidence structure; `<page-name>` derivation from URL is never defined; 'top 5' links in Phase 3 has no ranking rule; the `branch` field and `pwd` inside the JS REPL are unexplained; Phase 2 forward-references Phase 3's link check and baseline mode skips discovery, so baseline pages and monitored pages can diverge with no rule for pages missing from the baseline; MEDIUM/LOW alerts have no defined surfacing path (only CRITICAL/HIGH trigger AskUserQuestion); and early-stop by the user is mentioned but has no mechanism. The document is also a post-deploy monitor rather than the probe/reproduce/regression-before-repair workflow the evaluation frame describes, so authority, isolation, native contracts, and conditional loading are absent. An agent could execute the monitoring loop, but would have to improvise the quick mode, naming, and final deliverable."},"latency_ms":19387,"stop_reason":"end_turn","usage":{"input_tokens":3889,"output_tokens":1234},"cost_usd":0.10059}
+{"config":"qa-workflow","item":"f03","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":2,"reasoning":"The document is a partial, internally inconsistent stub. Several sections are headings with no body (Phase 2 'If the user specified auth credentials' / 'If the user provided a cookie file', Phase 4 'At each page:', Responsiveness item 7 ends mid-sentence). 'Modes' is referenced for diff-aware and regression behavior but only says 'pick whatever depth seems right', directly contradicting the explicit Quick/Standard/Exhaustive tier table. 'Find the browse binary' never says how; `$B`, 'health score', and the output directories are undefined. The tier table talks about which issues 'get fixed' yet no diagnosis, regression-before-repair, repair, or recheck phase exists, and Rule 5 ('never read source code') and Rule 12 ('always open the browser') conflict with diff-aware mode and with any native-functional or mixed-surface selection. No authority/isolation rules, bounded-probe limits, native contracts, conditional loading conditions, or blocked-path handling are present. An agent could roughly browse pages and take screenshots per the Phase 4 checklist and Phase 5 evidence tiers, but cannot authenticate, choose a mode deterministically, produce a defined final deliverable, or satisfy the broader workflow the bundle is supposed to drive."},"latency_ms":20905,"stop_reason":"end_turn","usage":{"input_tokens":2472,"output_tokens":1509},"cost_usd":0.10017}
+{"config":"qa-workflow","item":"p01","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser-only portion is written clearly with concrete commands, a defined phase sequence (Initialize → Authenticate → Orient → Explore → Document → Wrap Up), an explicit health-score rubric, and a baseline.json schema, so an agent can execute a browser smoke/full/regression pass and produce a report. However, measured against the stated bundle scope, large pieces are missing or contradictory: there is no native functional or mixed-surface selection, no bounded-probe budget, no reproduce→add-regression-before-repair→recheck loop (tiers 'determine which issues get fixed' but no fix phase exists and the title 'Phases 1-6' implies later phases that are absent), no authority/isolation or native contract sections, no conditional DX loading or blocked-path handling, and referenced files (report template, issue-taxonomy) are not provided. Internal inconsistencies hurt clarity: Rule 5 'Never read source code' conflicts with diff-aware mode mapping changed controllers/views to routes; 'Quick' is both a tier and a mode with different meanings; `/qa-only` is referenced but undefined; the port-detection `||`/`&&` chain has precedence issues; `snapshot -D`/`-C` semantics are unexplained. Net: executable for browser QA reporting, but incomplete and ambiguous for the full workflow described."},"latency_ms":15342,"stop_reason":"end_turn","usage":{"input_tokens":4930,"output_tokens":987},"cost_usd":0.09865}
+{"config":"qa-workflow","item":"p02","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The fix-loop phases (7–11) themselves are mostly well-sequenced with concrete commands, commit formats, classification states, and a self-regulation heuristic. However, the document is only the back half of the workflow: it depends on undefined upstream artifacts and variables ($B, $REPORT_DIR, detected test command, 'standard report template', 'user declined bootstrap', '{user}' in path, rules 1–10), and the working-tree check says 'continue with setup' without setup being present. Against the stated evaluation scope it is substantially incomplete: there is no surface selection (browser vs native functional vs mixed), no native contracts, no bounded probing/exploration, no conditional browser/DX loading, and no blocked-path handling. It also places the regression test after the fix commit (8e.5 after 8c), contradicting the expected 'regression before repair' sequencing. Ambiguities remain in the WTF heuristic ('All remaining Low severity'), in Phase 9's 're-run QA on all affected pages' for non-browser surfaces, and in how final health score is computed. An agent could execute the browser-centric fix loop reasonably well, but not the full workflow as described."},"latency_ms":15284,"stop_reason":"end_turn","usage":{"input_tokens":3776,"output_tokens":1128},"cost_usd":0.09416}
+{"config":"qa-workflow","item":"p05","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"The document is a tight, well-sequenced single-file workflow: phases are ordered, each ends with a named output, defaults and overrides are tabulated, stop-and-ask triggers are explicit, severity and scoring rules are numeric, and the fix loop (reproduce → failing test → minimal fix → re-test → classify → commit) is concrete enough to execute. Gaps hurt completeness: it covers only browser-based web QA and provides nothing for native functional or mixed surfaces, no authority/isolation model, no conditional loading of browser/DX sections, and no blocked-path handling—all of which the evaluation scope expects. Within the web scope there are also undefined decisions: how an issue is assigned to a scoring category (Functional vs UX vs Accessibility, etc.), what `<base>` is in Diff-aware mode, how 'broken links' are detected when Phase 3 only clicks primary controls, what Mode defaults to when there is no URL and no feature branch, and the deferred-issue categorization for scoring. Actionability for a web target is strong (clear artifacts, commit format, PR line), but an agent would stall or improvise on non-web surfaces and on category classification."},"latency_ms":15033,"stop_reason":"end_turn","usage":{"input_tokens":2610,"output_tokens":1012},"cost_usd":0.0767}
+{"config":"qa-workflow","item":"f04","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":2,"completeness":2,"actionability":2,"reasoning":"The browser probe scripts and screenshot copy steps are concrete and reusable, and the alert template and AskUserQuestion menus are well-structured. However the workflow contains direct contradictions: Phase 5 says 'Alert the user immediately on the first error you see' while the same phase and the Important Rules require two consecutive checks before alerting; Phase 1 says 'monitor for as long as seems necessary' while the Arguments define a 10m default and 1m–30m range. Key deliverables are underspecified: Phase 2 says 'save what you collected somewhere you can find it later' yet Phases 4 and 7 depend on a `baseline.json` whose schema and location are never defined; Phase 6's health report is a single sentence with no format, contents, or output path; `--quick` and `--duration` modes are listed but never handled in the instructions; the monitoring interval and end-of-run criteria are never stated. Severity handling is inconsistent (MEDIUM/LOW alerts have no confirmation rule, 'noticeably slower' vs the 2x rule), and the `pwd` reference in the REPL script is unexplained. An agent would have to invent the alerting policy, persistence format, and report, making the outputs non-deterministic."},"latency_ms":18242,"stop_reason":"end_turn","usage":{"input_tokens":4082,"output_tokens":1253},"cost_usd":0.10347}
+{"config":"qa-workflow","item":"p04","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":4,"reasoning":"The main file gives a well-ordered Phase 7-11 flow with concrete mechanics: severity-tiered triage, a regression-before-fix step with explicit file naming (`{name}.regression-N.test.{ext}`), attribution comments, value-card format, per-fix commit rules, a clear classification scheme (verified/best-effort/reverted), explicit self-regulation stop conditions and a hard cap, and well-specified report fields and state-root commands. Conditional loading is handled via STOP blocks with a defined fallback (report setup blocker, block affected probes, continue safe probes), and authority/isolation are addressed (caller-owned authority, synthetic identity, never request credentials). However, clarity suffers from dense, jargon-heavy phrasing ('with caller-owned authority', 'Phase 5 is the before', 'Browser `{target}` is a safe hostname') and some ambiguity: `--quick` appears under both Tier and Mode, `$REPORT_DIR` is never tied to the Output dir parameter, `{user}` in the project artifact path is undefined, the `--regression <baseline>` mode is never explained, and 8a.5 mixes in `/qa-only` instructions that don't belong to this command. Completeness is limited because Phases 1-6, the probe loop, native unit/integration/E2E rules, health-score computation, and browser verification all live in referenced sections (exploratory.md, qa-patterns.md, system-functional.md, browser-verify.md) that are not present in the bundle shown, so baseline exploration and DX/browser setup cannot be evaluated or executed from this file alone. Overall an agent can execute the fix/verify/report half reliably but must rely heavily on external files and interpret several underspecified terms."},"latency_ms":24931,"stop_reason":"end_turn","usage":{"input_tokens":4598,"output_tokens":1663},"cost_usd":0.12913}
+{"config":"qa-workflow","item":"a01","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The fix-loop phases (7-10) are laid out in a readable, sequential way with concrete git commands, commit message formats, a clear verified/best-effort/reverted/deferred classification, and a well-specified regression-test procedure (pattern matching, codepath tracing, attribution comment, naming scheme, one-retry rule). However, as a standalone bundle it is heavily dependent on undefined context: it references Phases 1-6, a 'top-of-skill learnings pull', a 'standard report template', the '$B' browser helper, '$REPORT_DIR', 'detected test command', 'user declined bootstrap', and the 'WTF-likelihood heuristic' — none of which are defined here. The title promises Phases 7-11 but no Phase 11 exists (rules are numbered 11-15, implying rules 1-10 elsewhere), and 'continue with setup' points to nothing. The regression section is internally inconsistent ('Generate unit tests' immediately after a decision table allowing integration/component tests). Health score computation and 'affected pages' for Phase 9 are unspecified. Critically for the stated evaluation scope, there is nothing on authority/isolation, native or mixed surfaces, native contracts, conditional browser/DX loading, or blocked paths — the document is browser-only and assumes a running $B session. An agent with the rest of the bundle loaded could execute this; in isolation it would stall on undefined variables and heuristics."},"latency_ms":15970,"stop_reason":"end_turn","usage":{"input_tokens":3436,"output_tokens":1087},"cost_usd":0.08871}
+{"config":"qa-workflow","item":"a06","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":4,"reasoning":"The workflow defines its authority model carefully (caller/owned, report-only limits, no state helpers, no cross-host assets), gives explicit blocked-path behavior for missing sections, and specifies modes, mixed-surface ordering, clocks/deadlines, REPORT_DIR/REPORT_FILE naming, collision handling, templates, and the project-copy destination with fallbacks. Deliverables are concrete and the final report structure (shared metadata, charters, per-surface sections, baseline/replay links, proposal-only repair fields) is well specified. Clarity suffers from dense, sometimes convoluted prose: the preparation gate is loaded in 'Select Surfaces' but charters/probes deferred to a later section with instructions like 'use the shared section already loaded above; do not restart its preparation'; references to 'exploratory §4 annotations and materialize' and 'the no-repeat rule' depend on external files not shown; the accidental-early-Read recovery wording is awkward; 'this host's installed QA directory' for templates is ambiguous between qa-only and qa; and Rule 2's preflight check is not placed explicitly in the phase sequence. An agent can execute it and produce the expected artifacts, but must resolve several ordering and cross-reference ambiguities along the way."},"latency_ms":19094,"stop_reason":"end_turn","usage":{"input_tokens":4675,"output_tokens":1253},"cost_usd":0.1094}
+{"config":"qa-workflow","item":"p04","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The fix loop (8a-8f), triage, commit conventions, classification states, and self-regulation stop rules are concrete and well-sequenced, and the STOP gates for loading exploratory.md and browser-verify.md state path-resolution and blocked-path fallbacks explicitly. However several things undermine executability: `--quick` is listed as an override for both Tier and Mode with no disambiguation; `--regression <baseline>` mode is named but never described; 'Phases 1-6' is only a pointer to shared/browser/functional sections (and a confusing note that qa-patterns.md's own 'Phases 1-6' are techniques, not phases), and the health-score computation, functional contract map, and any DX loading are absent from this bundle; 8a.5 references a '/qa-only' variant and an 'upstream card' that are undefined here; the test-only commit rule in 8e.5 sits after the commit step in 8d; and the Phase 10 artifact filename uses `{user}` without defining its source. Dense jargon ('caller-owned authority', 'healthy uncovered contract') adds interpretive load. An agent can execute the triage/fix/report portion reliably but must rely on unseen sections for baseline, scoring, and browser recheck, leaving the complete workflow only partially verifiable."},"latency_ms":25841,"stop_reason":"end_turn","usage":{"input_tokens":4237,"output_tokens":1790},"cost_usd":0.13187}
+{"config":"qa-workflow","item":"f05","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The bundle is mostly well-structured: it defines authority terms (caller/owned), a section index with conditional loads from the host's installed path, request-parameter parsing with a conflict rule for --quick/--regression, surface ordering and shared-deadline rules for mixed runs, REPORT_DIR/REPORT_FILE derivation with collision handling, and blocked-path behavior when assets are missing. However, it contains a direct, unresolved contradiction: the title, preamble and Additional Rule 1 state 'NEVER fix anything... Never commit', while the end of 'Run the Selected Checks' instructs the agent to 'fix it in the source right away and commit the fix' for critical/high bugs. For a report-only workflow this is a core-contract conflict that an agent cannot resolve from the text and that could cause unauthorized product edits and commits, badly undermining clarity and safe actionability. Lesser gaps: the preflight native-command check in Rule 2 is not placed in the phase sequence; 'diff-scoped' and the probe/reporting outputs are only defined by reference to external sections; browser-setup is loaded from a sibling skill directory with no fallback definition beyond 'blocked'. The phases and deliverables are otherwise reasonably defined, but the fix/commit contradiction is severe enough to make execution ambiguous and risky."},"latency_ms":16998,"stop_reason":"end_turn","usage":{"input_tokens":3608,"output_tokens":1001},"cost_usd":0.08613}
+{"config":"qa-workflow","item":"f03","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":2,"reasoning":"The document is a browser-only baseline fragment with large gaps: Phase 2 has four conditional headings with no instructions under them; Phase 3 says 'get a map' but gives no commands; Phase 4 says 'At each page:' then jumps to the checklist; Phase 6 is a single sentence and the 'baseline health score' is referenced but never defined; 'find the browse binary' and NEEDS_SETUP detection have no concrete commands. The Modes section is a vague one-liner while the Setup table references diff-aware and `--regression` modes that are never specified, and the tier table talks about 'which issues get fixed' with no fix/repair phase anywhere. Relative to the stated scope (browser/native/mixed surface selection, bounded probes, diagnosis, regression-before-repair, recheck, coverage reporting, authority, isolation, native contracts, conditional DX loading, blocked paths), virtually all of those decision points and deliverables are absent. The Important Rules and evidence tiers are clear and actionable on their own, which keeps clarity and actionability above the floor, but an agent could not execute the full promised workflow or produce defined outputs from this bundle."},"latency_ms":13891,"stop_reason":"end_turn","usage":{"input_tokens":2472,"output_tokens":878},"cost_usd":0.06862}
+{"config":"qa-workflow","item":"f04","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":2,"completeness":2,"actionability":3,"reasoning":"The document provides concrete, runnable Aside REPL scripts for capture and link checking, a clear argument list, and well-structured AskUserQuestion decision points with explicit options, which makes the core monitoring steps executable. However, several defects undermine clarity: Phase 5 directly contradicts itself ('alert immediately on the first error you see' vs. 'a first occurrence is pending, not yet an alert' and the rule 'only alert on patterns that persist across 2+ consecutive checks'); Phase 1 says 'monitor for as long as seems necessary' despite a defined --duration argument; 'periodically' has no interval; 'noticeably slower' is vague in-phase (only partially rescued by the 2x/1.5x rule). Completeness gaps are significant: Phase 2 says 'save what you collected somewhere you can find it later' with no filename or schema, yet Phase 4 and 7 reference a 'baseline.json' and 'the same manifest schema as Phase 2' that was never defined; Phase 6's health report is a single sentence with no format, path, coverage summary, or evidence structure; --quick and --duration modes are listed but never specified in any phase; page-name derivation is undefined; handling of MEDIUM/LOW alerts (template lists only CRITICAL/HIGH/MEDIUM) and the 'pwd' variable in the script are unclear. Cross-file references (Phase 2 invoking Phase 3's link check even when Phase 3 would not be loaded because --pages was given) create loading ambiguity. An agent could run the capture loop and surface alerts, but would have to invent the manifest schema, check cadence, and final report, making deliverables inconsistent across runs."},"latency_ms":20758,"stop_reason":"end_turn","usage":{"input_tokens":4082,"output_tokens":1529},"cost_usd":0.11727}
+{"config":"qa-workflow","item":"f02","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is a six-line checklist with no operational detail. It assumes a browser-only app ('Open the site') and gives no guidance for selecting between browser, native functional, or mixed surfaces, nor any native contract definitions. There is no authority model (who may fix what, where), no isolation rules (sandboxing, test data, avoiding production), no conditional loading of browser/DX sections, and no blocked-path handling. Exploration is unbounded ('look around', 'don't overdo it') rather than defined probes with limits. Reproduction and diagnosis steps are absent; 'fix the bugs that matter' has no severity criteria or triage rule. The critical requirement to add a regression test before repair is missing entirely. 'Check again' lacks any recheck procedure or acceptance criteria, and the report has no required evidence or coverage format. The prose is readable but relies on 'good judgment' in place of executable decisions, so an agent cannot reliably produce consistent deliverables."},"latency_ms":8173,"stop_reason":"end_turn","usage":{"input_tokens":561,"output_tokens":429},"cost_usd":0.02706}
+{"config":"qa-workflow","item":"f01","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":2,"reasoning":"The bundle has direct contradictions and structural defects that block unambiguous execution. Phase 7 Triage first states tiers determine scope (Quick = critical+high), then says 'Fix every issue in the report, including low severity, whatever tier was selected' with a vague 'if time is short' override — an agent cannot resolve which rule governs. Phase 8 has two sub-steps both labeled '8a', and the Commit step is placed before Locate/Fix, inverting the natural order (commit of what?). Phase 9 is missing entirely (8 jumps to 10), so there is no verification/recheck step, no regression-test-before-repair step, and no health-score recomputation, yet Rule 13 references a nonexistent 'Phase 8e.5' and Rule 14 demands 'revert on regression' with no defined check to detect one. Phase 10 Report is a single sentence ('Write up what you did') with no required fields, evidence, coverage, or score deltas. Rule 15 cites an undefined 'WTF-likelihood heuristic'. The stash option says 'pop the stash after' but no later phase performs the pop. The learnings-refresh instruction is reasonably specific (keyword constraints, command, worked examples), and the clean-tree gate, atomic-commit format, and TODOS.md update are concrete and actionable. But the missing verify/regression phases, the tier contradiction, duplicated step labels, and the hollow report spec mean the agent cannot produce the stated deliverables (regression before repair, recheck, evidence/coverage report) from this document alone."},"latency_ms":16634,"stop_reason":"end_turn","usage":{"input_tokens":2307,"output_tokens":1156},"cost_usd":0.08087}
+{"config":"qa-workflow","item":"f04","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":2,"completeness":2,"actionability":3,"reasoning":"The browser scripts are concrete and runnable (console-error hook, nav timing, text snapshot, screenshot copy-out, link HEAD check), and the alert/question templates are well-formed, so an agent can execute the capture steps. However the alerting policy is self-contradictory: 'Alert the user immediately on the first error you see; waiting for a second check wastes critical minutes' sits beside 'A first occurrence is pending, not yet an alert' and 'Only alert on patterns that persist across 2+ consecutive checks.' Phase 1 says 'Monitor for as long as seems necessary' despite a documented default of 10 minutes and a 1m–30m --duration flag. Key artifacts are underspecified: Phase 2 says 'save what you collected somewhere you can find it later' with no schema or filename, yet Phase 4 checks for `baseline.json` and 'the same manifest schema as Phase 2'; Phase 6 is a single sentence ('write up a report') with no format, fields, location, or coverage/evidence requirements; `--quick` is listed as an argument but never defined in any phase; the monitoring interval and check-count cadence are never stated; 'top 5' link selection has no ordering rule; 'noticeably slower' is vague (the 2x rule appears only in the footer); Phase 7 baseline update does not address renaming `<page>-<n>.jpg` to the baseline naming. `pwd` inside the repl script is unexplained. Net: executable probes, but ambiguous decision rules and missing outputs."},"latency_ms":21394,"stop_reason":"end_turn","usage":{"input_tokens":3721,"output_tokens":1549},"cost_usd":0.11466}
+{"config":"qa-workflow","item":"p03","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is dense but well-ordered: parameters with defaults and conflict resolution (--quick vs --regression), a hard git-status gate, explicit authority rules (production/unknown ownership blocks, /qa is not permission), concrete report-dir and checkpoint rules (exactly four JSON fields, no batching/editing), a defined confirmation rule (two failures = confirmed), a sentinel-based browser readiness check with a defined blocked path, regression-before-fix with naming/attribution/red-test requirements, commit format, classification states, stop rules, a fully specified health-score formula, and an enumerated report. Gaps that cost points: 'Diff-aware' is listed as a browser technique but is not a selectable mode and its trigger is never stated; Quick exploration semantics for the functional surface are undefined (only the browser Quick budget is given); ISSUE-NNN numbering and how issues map to health-score categories (UX vs Functional vs Accessibility etc.) are not defined; probe/evidence directories for single-surface runs are only implied ('REPORT_DIR/.../evidence'); the screenshots dir is never created; and the stop rule 'when only Low issues remain' mildly conflicts with the Exhaustive tier. The document also declares itself a single whole-workflow file, which does not match the stated conditional-loading bundle model, though that does not impair execution. Overall an agent could execute it and produce the stated deliverables with minor judgment calls."},"latency_ms":27693,"stop_reason":"end_turn","usage":{"input_tokens":4474,"output_tokens":2106},"cost_usd":0.15004}
+{"config":"qa-workflow","item":"p03","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is tightly structured with ordered phases, explicit gates (git status stop, isolation/permission rules, aside readiness sentinel, non-local mutation ask), concrete commands, fixed artifact names/schemas (exploration-NNN.json four fields, evidence dirs, regression test naming, commit message format), a defined fix loop with stop rules and classification, a precise health-score formula, and an enumerated report contents list. Decisions are mostly executable without interpretation. Residual gaps lower it from 5: 'Diff-aware' browser technique appears in §6 but is not a declared mode in §1 and its trigger is implicit; `<base>` for the git diff is undefined; ISSUE-NNN numbering is never specified separately from exploration NNN; the evidence path uses an ellipsis ('REPORT_DIR/.../evidence') and screenshots/ placement in mixed runs is unstated; the 30-second Browser Quick budget sits uneasily with the mandatory checkpoint-per-probe rule; per-category health scoring is slightly ambiguous for Console/Links (own formula vs. per-issue deductions); and the browser-only case when aside is unavailable has no explicit terminal path. The document also declares itself a single in-order file, which conflicts with the stated multi-file conditional-loading bundle model, but internally it remains coherent."},"latency_ms":21519,"stop_reason":"end_turn","usage":{"input_tokens":4113,"output_tokens":1526},"cost_usd":0.11743}
+{"config":"qa-workflow","item":"a01","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The fix-loop core (locate → minimal fix → atomic commit → re-test → classify → regression test → report) is clearly sequenced with concrete commands, a sensible classification scheme, a well-specified regression-test procedure (pattern matching, codepath tracing, attribution comment, auto-increment naming, bounded retry/time budget), and defined report outputs with a summary and PR line. However, as a workflow bundle it has significant gaps. The title promises Phases 7-11 but Phase 11 is absent. Key dependencies are referenced but never defined here: the WTF-likelihood heuristic (rule 15 and 8e.5 step 5), `$B` and `$REPORT_DIR`, the 'detected test command', 'user declined bootstrap', the 'standard report template', and the health-score computation. Rules are numbered 11-15 with rules 1-10 missing, and the clean-tree gate says 'continue with setup' although it sits after setup/baseline, creating ordering ambiguity. The evaluation brief asks about authority, isolation, native functional contracts, conditional browser/DX loading and blocked paths—none are addressed; re-test assumes only a browser surface, so native/mixed surfaces have no verification path. 8d lacks a 'before' screenshot capture step despite requiring a before/after pair, and 'affected pages' for Phase 9 isn't defined for non-browser surfaces. An agent could execute the browser-centric fix loop reasonably well but would need to infer or guess several undefined contracts."},"latency_ms":13991,"stop_reason":"end_turn","usage":{"input_tokens":3797,"output_tokens":878},"cost_usd":0.08187}
+{"config":"qa-workflow","item":"a06","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is tightly specified for a report-only agent: authority terms (caller/owned) are defined up front, the never-fix rule is enforced through concrete prohibited actions (shell writes, renames, edit-then-restore, commit/stash), and the section index states exactly when each external file is loaded with host-installed-path constraints and an explicit blocked-path fallback (report setup blocker, mark affected probes blocked, continue independent safe probes). Request parsing has a defaults/override table plus a conflict rule for --quick vs --regression; diff-aware behavior on feature branches is defined; optional context lookups (test plan, learnings) have clear skip conditions and a 'never run cache-mutating helpers' rule. Artifact handling is unusually precise: REPORT_DIR emptiness check, UTC run-subdirectory naming with collision suffix, {target} sanitization, mixed-target labeling, REPORT_FILE derivation, charters-before-probe-clock gating, and a project-copy path with user/branch fallbacks and no-overwrite rule. Output structure (shared metadata once, charters before findings, per-surface nesting, browser scores never merged with functional, regression baseline linking, proposals-not-edits) is well defined. Weaknesses: prose is dense and occasionally convoluted (e.g., 'Load the shared preparation gate now (the exploratory STOP just below): complete its scope and selected-method Reads', 'Order: exploratory §4 annotations and materialize', 'the no-repeat rule covers preparation Reads'), relying on terms defined only in the unseen exploratory section, so an agent must trust those files to resolve what 'selected-method Reads', Full/Quick/Regression budgets, and the native contract procedure actually are. Mode definitions, clocks and native-method specifics are deferred entirely to sections, and the accidental-early-Read recovery instruction is awkwardly worded. Browser setup depends on a sibling qa/gstack-qa skill existing, with only a blocker fallback. Overall highly executable with strong guardrails, but the forward/backward references and compressed phrasing introduce some friction."},"latency_ms":30515,"stop_reason":"end_turn","usage":{"input_tokens":4675,"output_tokens":2105},"cost_usd":0.152}
+{"config":"qa-workflow","item":"p02","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"What is present (tiers, triage, the 8a–8f fix loop, classification states, WTF-likelihood heuristic, report fields, TODOS update) is written in concrete, mostly executable steps with explicit commands and commit formats. However, the document is only Phases 7–11 of a larger bundle and depends heavily on undefined context: Phases 1–6, the baseline report template, `$B`, `$REPORT_DIR`, the 'detected test command', test-framework detection/bootstrap prompt, the 'top-of-skill learnings pull', and rules 1–10 (rules here start at 11). None of the requested authority, isolation, native-surface contracts, conditional browser/DX loading, or blocked-path handling appears; the loop is browser-only via `$B`, with no native functional or mixed-surface path. Several decisions are ambiguous: the regression test is added after the fix and commit (contradicting the stated 'regression before repair' intent and leaving no proof the test fails pre-fix); '>2 min exploration' and 'touching unrelated files' (+20%) are unmeasurable or contradict the only-related-files rule; 'delete test, defer' doesn't say whether the issue or just the test is deferred; 'Re-run QA on all affected pages' in Phase 9 points to undefined phases; WTF recomputation cadence ('every 5 fixes or after any revert') resets are unspecified. An agent could follow the fix loop in a browser-only project if the prior phases and variables exist, but cannot execute the complete workflow as described from this bundle alone."},"latency_ms":15871,"stop_reason":"end_turn","usage":{"input_tokens":3776,"output_tokens":1158},"cost_usd":0.09566}
+{"config":"qa-workflow","item":"a05","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The document is short and readable, with concrete setup, explore, fix, and report steps, and the browser-only path is easy to follow. However, it does not match the expected bundle structure at all: there is no surface selection (browser vs native functional vs mixed), no native contracts, no conditional browser/DX section loading, no authority/precedence rules, no isolation (worktree/branch guard beyond a git-status check), no bounded probes (unbounded page/button/form exploration), no blocked-paths list, and critically no 'add a regression before repair' step — fixes are verified only by manual reload and screenshot. Key terms are undefined: 'important' vs 'minor' issues, 'quick', 'many unrelated files', and the 0-100 health score has no computation rule, so two agents would produce different outputs. Coverage reporting is thin (page count and screenshot count only). An agent could execute the browser loop and emit a report, but it would miss required deliverables (regression tests, coverage evidence, native/mixed surfaces) and would have to improvise on ambiguous thresholds."},"latency_ms":13145,"stop_reason":"end_turn","usage":{"input_tokens":1081,"output_tokens":932},"cost_usd":0.05741}
+{"config":"qa-workflow","item":"p05","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":4,"completeness":2,"actionability":3,"reasoning":"What is present is written clearly: phases are ordered, each ends with a named output, severity and scoring rules are numeric, and stop/ask conditions in the fix loop are explicit. However, against the stated evaluation frame the document is a single browser-only web-app flow: there is no surface selection (browser vs native functional vs mixed), no native test contracts, no conditional browser/DX section loading or stated load conditions, no authority/isolation rules (e.g., worktree isolation, which files/commands are permitted), and no blocked-path handling beyond ad-hoc 'ask the user'. Internal gaps also hurt: Phase 3 never tells the agent how to test Links, Accessibility, Performance or how to assign an issue to one of the eight scoring categories that Phase 4 depends on; `<base>` for Diff-aware is undefined; `--quick` doubles as both a mode and a fix-tier flag; the default mode when no URL is given and not on a feature branch is unspecified; Phase 6 does not say whether recomputed scores overwrite baseline.json. An agent could execute the browser path and produce report.md/baseline.json/screenshots, but would stall on category assignment and could not handle native or mixed surfaces at all."},"latency_ms":17771,"stop_reason":"end_turn","usage":{"input_tokens":2610,"output_tokens":1202},"cost_usd":0.0862}
+{"config":"qa-workflow","item":"p05","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":4,"completeness":2,"actionability":3,"reasoning":"The document is well-structured: ordered phases, each gated by a named output, concrete defaults, explicit stop-and-ask triggers, a defined severity ladder, a numeric scoring formula, and a fix loop with test-first, smallest-change, classify, and per-issue commit rules. Clarity is high for a browser-only flow, with only minor ambiguities (undefined 'primary control', no rule for assigning an issue to a scoring category, 'untested category' never operationalized, `<base>` for diff-aware unspecified, `--quick` overloaded as both mode and fix tier, no default mode when neither URL nor feature branch applies). Completeness is weak against the stated evaluation scope: the bundle is a single web-app file with no native functional or mixed-surface selection, no native contracts, no authority/permission model beyond the git-status check and password rule, no isolation guidance (sandboxing, destructive-action or payment blocked paths, test-data handling), no conditional browser/DX loading or conditions under which sections are loaded, and no 'blocked path' handling. Internally it also scores Accessibility and Performance without ever telling the agent how to probe them in Phase 3, and Phase 6 does not specify re-running screenshots or regression tests across the whole suite. Actionability is moderate: an agent can produce report.md, baseline.json, screenshots, and commits for a web app, but gaps in category mapping and a11y/perf measurement mean scores will be inconsistently derived, and non-browser or mixed targets have no executable path at all."},"latency_ms":20270,"stop_reason":"end_turn","usage":{"input_tokens":2971,"output_tokens":1353},"cost_usd":0.09736}
+{"config":"qa-workflow","item":"p04","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The file has a sensible skeleton (parameters, tiers, STOP-gated conditional loads for exploratory.md and browser-verify.md with explicit blocked-path handling, a well-sequenced fix loop 8a→8a.5→8b→8c→8d→8e→8e.5→8f, concrete bash commands, test file naming, commit conventions and report structure). However, clarity suffers from dense, compressed prose (e.g., 8a.5's value-card byte clamping, 'Phase 5 is the before' referencing an unseen qa-patterns.md, '/qa-only' mentioned without definition), overlapping Tier/Mode flags (`--quick` appears in both; `--regression` mode is never elaborated), and Phases 1-6 being entirely delegated to shared sections whose 'ordered preparation', 'probe loop', 'causal hypothesis and minimized replay' and 'health score' are only referenced, not defined here. Completeness gaps: severity criteria, health-score computation, diff-scoping rules, and functional contract-map semantics all live elsewhere, so the document alone cannot drive the baseline; conversely triage, fix loop, self-regulation, final QA, report and TODOS phases are reasonably complete. Actionability is middling: an agent can execute Phases 7-11 and produce the stated deliverables given the referenced sections exist, but the main file's reliance on unseen assets, some ambiguous parameters, and heavy inline conditionals (e.g., 'never for a functional-only repair') create room for misexecution."},"latency_ms":19409,"stop_reason":"end_turn","usage":{"input_tokens":4237,"output_tokens":1285},"cost_usd":0.10662}
+{"config":"qa-workflow","item":"f03","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":2,"reasoning":"The document is a skeletal, partially-written baseline. Several sections are headings with no body: Phase 2's 'If the user specified auth credentials' and 'If the user provided a cookie file' have no instructions; Phase 4's 'At each page:' and 'Check mobile viewport if relevant:' trail off with no commands; 'Find the browse binary' never says how to find it or what NEEDS_SETUP is. Setup points to 'diff-aware mode (see Modes below)' but the Modes section is a single vague sentence ('Pick whatever depth seems right') that defines no modes, contradicting the structured Tier/Mode parameter table. Tiers are described as determining 'which issues get fixed', yet no phase fixes anything. Relative to the stated scope (browser/native/mixed surface selection, bounded probes, reproduce/diagnose, regression-before-repair, recheck, coverage reporting, authority/isolation/native contracts, conditional loading, blocked paths), almost none of this exists—there is no native functional path, no regression or repair phase, no recheck, no coverage output, no isolation or blocked-path guidance. Phase 6 is just 'Summarize what you found' plus a trailing instruction to record a 'baseline health score' that is never defined. Referenced files (report template, issue taxonomy) are not included. The parts that are written—evidence tiers, the Important Rules, framework detection heuristics, the Quick-mode checklist—are concrete and usable, which keeps clarity and actionability above the floor, but an agent could not execute the full workflow or produce the expected deliverables from this bundle."},"latency_ms":14749,"stop_reason":"end_turn","usage":{"input_tokens":2833,"output_tokens":969},"cost_usd":0.07678}
+{"config":"qa-workflow","item":"a04","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":4,"reasoning":"The bundle defines authority boundaries (caller owns decisions/fixes; discovery writes only reports/evidence), an ordered read gate with blocked-path handling, per-caller/mode budgets, concrete guard (G) and evidence (Q) commands with exact invocations, a strict four-field checkpoint schema distinguishing browser (Write) from functional (Q-published) checkpoints, replay-before-repair and revalidation rules, an annotations.json schema plus materialize step, and a well-specified functional contract map with outcome vocabulary and regression-baseline blocking. However, the prose is extremely dense and telegraphic ('unproven=affected', 'status requires fresh evidence', 'Keep limits/notes'), making several rules interpretable only with care; the isolation/permission preflight is referenced but only partially defined here; scope.md, qa-patterns.md, templates and any DX/browser loading details live outside the bundle; the 12-probe count for /review//ship has no enforcement mechanism like G; the fresh report subdirectory naming and 'classification' value set are unspecified; and parent-side references (Phase 8, Fix-First, ASK) assume external context. Executable overall, but the compression costs clarity and invites misreads under time pressure."},"latency_ms":24359,"stop_reason":"end_turn","usage":{"input_tokens":4381,"output_tokens":1667},"cost_usd":0.12716}
+{"config":"qa-workflow","item":"a03","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The phase structure, concrete aside repl scripts, copy-out steps, alert tiers, change-not-absolute comparison rule, and the 2-consecutive-check confirmation rule are clear and executable. However, several gaps undermine unambiguous execution: the `--quick` argument is declared but no phase describes single-pass behavior; `<page-name>` derivation from URL paths is never specified, making file naming and manifest keys ambiguous; `SLUG` is captured but never used; `branch` in the manifest has no capture step; `pwd` inside the repl script is an unexplained identifier; the Phase 2 text awkwardly forward-references Phase 3's link check and per-round repetition, blurring which phase owns that work; MEDIUM/LOW alerts have no stated notification path; and Phase 6 'Health Report' has no format, schema, or output file path, so the primary deliverable is undefined. Out-of-range `--duration` handling, baseline/page-set mismatch, and what constitutes 'content disappears' for the text snapshot are also unspecified. An agent could run the monitoring loop and raise alerts, but would improvise the report, page naming, and quick mode."},"latency_ms":20573,"stop_reason":"end_turn","usage":{"input_tokens":4250,"output_tokens":1317},"cost_usd":0.10835}
+{"config":"qa-workflow","item":"f05","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The workflow has a well-structured flow (parameters → context → surface selection → artifact prep → conditional browser setup → checks → report) with clear authority/ownership definitions, deterministic REPORT_DIR/REPORT_FILE rules, blocked-path handling for missing sections, and isolation guidance. However, it contains a direct, unresolved contradiction: the Run the Selected Checks section instructs the agent to 'fix it in the source right away and commit the fix' for critical/high bugs, while the header, preamble and Additional Rule 1 say NEVER fix, never edit product code, never commit. An agent cannot execute both; this breaks the report-only contract and makes the critical-bug path ambiguous and unsafe. Additional friction: the 'Select Surfaces and Isolation' paragraph awkwardly references the STOP block 'just below' and splits one Read across two phases with deferral language that is hard to follow; preflight framework-check ordering relative to the exploratory section is implied rather than explicit; and the regression/baseline resolution for mixed runs depends on externally loaded method definitions. Completeness is moderate since the phases and outputs exist (charters, report, evidence, blockers), but the contradictory fix instruction and reliance on unseen section files leave key decisions undefined for the agent."},"latency_ms":13417,"stop_reason":"end_turn","usage":{"input_tokens":3247,"output_tokens":764},"cost_usd":0.07067}
+{"config":"qa-workflow","item":"p03","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is a single ordered file with well-structured phases (parameters → isolation → artifacts → browser readiness → charters → probe loop → triage → fix loop → final QA → report), explicit authority/permission rules, concrete commands with sentinels (ASIDE_READY, GSTACK_STEP_OK), exact checkpoint schema, file naming, commit format, stop rules and report contents. Decision points (dirty git tree, blocked browser, non-local mutation, fix tier ordering, revert/stop rules) are mostly unambiguous. Gaps that keep it from 5s: 'Diff-aware' appears only under browser techniques and is absent from the Mode table; `<base>` for the diff is undefined; evidence/probe paths use an ellipsis and single-surface runs lack a stated probe directory (vs. `browser/` and `functional/` for mixed, plus a separate `screenshots/` dir); the health-score formula doesn't say how an issue maps to UX/Accessibility/Visual/etc. categories or how the `--regression` baseline comparison is actually performed; ISSUE-NNN assignment is unspecified; the 30-second Browser Quick budget is likely infeasible for six page probes with screenshot reads. The prompt's framing of conditionally loaded bundle sections and a DX section does not match this document, which declares itself the whole workflow, but the document as written is largely executable."},"latency_ms":22706,"stop_reason":"end_turn","usage":{"input_tokens":4113,"output_tokens":1500},"cost_usd":0.11613}
+{"config":"qa-workflow","item":"f05","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The bundle is generally well-structured: a section index with stated load conditions, parameter parsing with conflict handling (--quick vs --regression), explicit authority/ownership definitions (caller/owned), isolation rules, REPORT_DIR/REPORT_FILE derivation with collision handling, conditional browser-setup loading with blocked-path fallbacks, and mixed-surface ordering/clock rules. However, it contains a direct, unresolved contradiction that undermines executability: the 'Run the Selected Checks' phase instructs the agent to 'fix it in the source right away and commit the fix' for critical/high bugs, while the header, the caller/owned definitions, and Additional Rule 1 state NEVER fix, never edit product code, never commit. An agent cannot follow both; this sentence is placed in the core execution phase and could cause a report-only run to mutate the repo. Secondary gaps: the reporting section (`sections/reporting.md`) is only listed in the index with no corresponding STOP/finalize phase in the body, so the end-of-run step is implicit; the 'Select Surfaces' section says to 'load the shared preparation gate now (the exploratory STOP just below)' which is slightly awkward ordering; and preflight framework checks (Rule 2) are not sequenced into a named phase. Completeness is moderate since most phases and blockers are defined, but clarity and actionability are seriously hurt by the fix-and-commit contradiction and the missing explicit reporting phase."},"latency_ms":15544,"stop_reason":"end_turn","usage":{"input_tokens":3608,"output_tokens":982},"cost_usd":0.08518}
+{"config":"qa-workflow","item":"f04","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":2,"completeness":2,"actionability":2,"reasoning":"The Aside REPL scripts and directory setup are concrete, but the workflow contradicts itself on core decisions: Phase 1 says 'monitor for as long as seems necessary' while Arguments define a 10m default and 1m–30m range; Phase 5 says 'alert immediately on the first error' and then says a first occurrence is 'pending, not yet an alert' until two consecutive checks (echoed by the Transient Tolerance rule). The monitoring interval is never stated ('check each page periodically'). Phase 2 says 'save what you collected somewhere you can find it later' with no schema or path, yet Phase 4 and Phase 7 reference a 'baseline.json' and 'the same manifest schema as Phase 2' that are never defined. Phase 6 is a single sentence ('write up a report') with no format, path, or required fields, so the main deliverable is undefined. `--quick` and `--duration` modes are listed but never handled in the instructions; `<page-name>` derivation and the `pwd` reference inside the JS are unexplained; the alert list includes LOW while the template only allows CRITICAL/HIGH/MEDIUM; 'noticeably slower' in Phase 5 is only loosely reconciled by the 2x/1.5x rule at the end. An agent could run the capture scripts but could not reliably decide when to alert, how long to run, or what to write in the baseline and final report."},"latency_ms":19214,"stop_reason":"end_turn","usage":{"input_tokens":3721,"output_tokens":1406},"cost_usd":0.10751}
+{"config":"qa-workflow","item":"a04","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The bundle has a recognizable phase structure (Reads → charter/preflight → probe loop → handoff → report), explicit tool invocations (G start/status/run, Q capture/materialize with flags), a precise four-field checkpoint schema, a solid native contract map with outcome taxonomy, and well-defined blocked/pass semantics. However, it is extremely compressed and relies on undefined or dangling references that an agent cannot resolve from the text: 'R ... chosen above' (no explicit selection step), how the fresh `.gstack/qa-reports` subdirectory is named, 'no plan/server is required', 'Phase 8 regression gates', 'Fix-First', 'ASK approval', and 'the shared isolation/permission preflight' (system-functional step 1 points to a preflight that exploratory.md never concretely specifies beyond a charter field). Conditionally loaded dependencies (scope.md surface selection, qa-patterns.md for browser, functional-report-template.md) are absent, so surface selection, browser probing, and final report formatting are only partially executable. Dense phrasing such as 'QA_DEADLINE receipts are not observations' or 'invented safe-path redactions' is interpretable but risks divergent agent behavior. Functional bounded runs are the most actionable path; mixed/browser paths and report deliverables are under-specified within this bundle."},"latency_ms":27970,"stop_reason":"end_turn","usage":{"input_tokens":4381,"output_tokens":2045},"cost_usd":0.14606}
+{"config":"qa-workflow","item":"p04","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The entry file gives concrete, mostly executable guidance for Phases 7-11: severity tiers, learnings search command, value-card format with byte clamping, regression-file naming (`{name}.regression-N.test.{ext}`), attribution comment, commit conventions, classification states, self-regulation stop rules and hard cap, report paths and PR summary line. Blocked-asset handling (report setup blocker, mark affected probes blocked, continue safe probes) is stated consistently at both STOP gates, and browser vs functional separation is enforced. However, the bundle as presented omits every referenced section (exploratory.md, browser-verify.md, qa-patterns.md, system-functional.md), so Phases 1-6, the probe loop, authority/isolation rules, native unit/integration/E2E rules, health-score computation, baseline.json schema and severity definitions are all undefined here; 'Phase 5 is the before' and 'Phases 1-6 as techniques' cannot be resolved. Clarity suffers from dense jargon ('caller-owned authority', 'healthy uncovered contract', 'explicitly approved browser bootstrap'), overlapping `--quick` in both Tier and Mode, a stray '/qa-only' branch, and terse sentences that require interpretation. An agent can execute the fix/verify/report half but cannot run the baseline half from this file alone."},"latency_ms":19256,"stop_reason":"end_turn","usage":{"input_tokens":4237,"output_tokens":1193},"cost_usd":0.10202}
+{"config":"qa-workflow","item":"f03","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":2,"reasoning":"The bundle only covers a browser-only 'baseline' (Phases 1-6) and is visibly truncated or hollow in many places: the Auth phase has conditional headers ('If the user specified auth credentials', 'If the user provided a cookie file') with no instructions beneath them; Phase 4's 'At each page:' has no commands; the responsiveness item ends mid-sentence; Phase 6 is just 'Summarize what you found'; the 'Modes' section says 'pick whatever depth seems right' while Setup promises a defined diff-aware mode that never appears; 'baseline health score' is mandated but never defined; and tiers 'determine which issues get fixed' though no fix/repair phase exists. Relative to the stated scope (browser, native functional or mixed surface selection, bounded probes, reproduce/diagnose, regression-before-repair, recheck, coverage reporting, authority/isolation/native contracts, conditional loading and blocked paths), almost nothing is present — there is no native surface handling, no regression or recheck step, no authority or isolation rules, and no conditional loading conditions. The Important Rules and Phase 5 evidence tiers are concrete and usable, and framework-detection heuristics are actionable, which prevents a floor score, but an agent cannot execute the full intended workflow from this document."},"latency_ms":13072,"stop_reason":"end_turn","usage":{"input_tokens":2472,"output_tokens":757},"cost_usd":0.06257}
+{"config":"qa-workflow","item":"a04","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The bundle establishes a coherent spine: caller authority vs. discovery scope, ordered prerequisite Reads, a charter/preflight phase with explicit budgets per caller/mode, a strict one-probe-at-a-time loop with publish-before-probe checkpoints, replay-before-repair, revalidation after change, handoff rules, and a final-report step with a defined annotations.json shape and the G/Q tool invocations. The functional method adds a usable contract map with outcome vocabulary and mode definitions (Full/Quick/Regression with baseline blocking rules). However, clarity suffers from extremely compressed, telegraphic prose and heavy symbolic shorthand (R, D, G, Q, NNN, PREV) that forces the agent to reconstruct meaning; several sentences are ambiguous or near-ungrammatical ('Missing or unreadable assets, prerequisites or permission block affected probes, not independent safe checks'; 'Without a total time limit, do not start D; announce finite command timeouts'; 'unproven=affected'). The step-2 instruction to 'Create exploration-NNN.json' is then overridden for functional surfaces, and the functional checkpoint example hard-codes `--deadline D` even though unbounded runs must use `--timeout-ms`, leaving the unbounded checkpoint command underspecified. `bun Q materialize R annotations.json` is path-ambiguous relative to `R/annotations.json`. Completeness is limited by essential dependencies outside the bundle: surface selection lives in scope.md, browser probing in qa-patterns.md, report templates, and '/qa Phase 8 regression gates' are all referenced but absent, so the browser/mixed path and the hand-off gates cannot be fully evaluated; the browser-only annotations convention ('evidence [], checkpoints in limits') and the classification enum for annotations are only implied. Actionability is moderate: concrete commands, budgets, file names, and stop conditions exist, and the contract map gives testable targets, but the density and the small inconsistencies make execution error-prone, and an agent cannot complete a browser or mixed run from this bundle alone."},"latency_ms":27410,"stop_reason":"end_turn","usage":{"input_tokens":4742,"output_tokens":1846},"cost_usd":0.13972}
+{"config":"qa-workflow","item":"a03","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":3,"reasoning":"The phases are sequenced clearly with concrete aside repl scripts, a defined baseline manifest schema, explicit cadence rules (60s rounds, no overlap, deadline handling), change-based alert thresholds, a 2-consecutive-check debounce, and a well-specified alert block plus AskUserQuestion options. However, several deliverables and branches are underspecified: `--quick` is listed as an argument but never described in the instructions; Phase 6 Health Report says only 'summarize the results' with no format, fields, or output path; MEDIUM/LOW alerts are confirmed but it is unclear whether/how they are surfaced versus only CRITICAL/HIGH; page-name derivation (e.g., '/' -> 'home'), duration parsing/bounds enforcement, branch retrieval, and handling when discovered pages differ from baseline pages are left implicit. Technically, the script relies on `pwd` and `_sendToTarget` as repl globals/private APIs without stating they exist, and the Phase 2 instruction to 'run Phase 3's link check' is a forward reference that mixes baseline and discovery flows. Screenshot comparison against baselines is implied but no diff method is given, so screenshots are evidence only. An agent can execute the capture/monitor loop reliably, but the final report and quick mode require improvisation."},"latency_ms":21281,"stop_reason":"end_turn","usage":{"input_tokens":3889,"output_tokens":1428},"cost_usd":0.11029}
+{"config":"qa-workflow","item":"f02","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is a six-line sketch, not an executable workflow. It never addresses the bundle's stated structure: no section files, load conditions, authority/ownership rules, isolation requirements, native functional contracts, conditional browser/DX loading, or blocked paths. Decision points are left to 'good judgment' with no criteria: how to locate the running app (and what to do if it isn't running or must be started), what counts as a surface (browser vs native vs mixed), how probes are bounded, what qualifies as a bug 'that matters', how to reproduce/diagnose, or when to stop. The required phase of adding a regression test before repairing is entirely absent, as is any recheck protocol or report schema (evidence, coverage, defects found/fixed/deferred). Steps are readable in isolation, hence a 2 on clarity, but an agent cannot reliably produce the expected deliverables or make consistent decisions from this text."},"latency_ms":8827,"stop_reason":"end_turn","usage":{"input_tokens":561,"output_tokens":493},"cost_usd":0.03026}
+{"config":"qa-workflow","item":"p02","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"Within its own scope (Phases 7-11), the fix loop is well-sequenced: triage by tier, locate → minimal fix → atomic commit → re-test → classify → regression test → WTF-likelihood self-regulation → final QA → report → TODOS. Commit formats, classification states, attribution comment, and the stop thresholds are concrete. However, as the 'complete workflow' it is badly truncated: Phases 1-6 (surface selection, exploration probes, baseline, health score formula), the browser/DX conditional loading, native functional contracts, isolation/authority rules, and blocked paths are all absent or only referenced. Many dependencies are undefined in this file: `$B`, `$REPORT_DIR`, the 'detected test command', 'user declined bootstrap' (no bootstrap step exists), the 'standard report template', the 'top-of-skill learnings pull', and rules 1-10 (rules start at 11). The ordering also contradicts the stated contract of adding a regression *before* repair — here the test is written after fix+commit, and only for 'verified' fixes. Phase 9 says 're-run QA' and 'compute final health score' without specifying how. The clean-tree check sits before Phase 7 yet ends with 'continue with setup', suggesting misplaced content. Native (non-browser) surfaces get no re-test path at all — 8d assumes a browser. An agent could execute the fix loop for browser targets if the missing files supply the variables, but cannot execute the full workflow from this document alone."},"latency_ms":15083,"stop_reason":"end_turn","usage":{"input_tokens":3776,"output_tokens":1076},"cost_usd":0.09156}
+{"config":"qa-workflow","item":"a04","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The bundle defines a coherent flow (ordered Reads → charter/preflight → bounded probe loop with published checkpoints → replay/minimize → handoff → annotated final report) and gives concrete tool invocations (G start/status/run, Q capture/materialize, exploration-NNN.json's four fields, annotations.json shape), plus a usable functional contract map with outcome vocabulary. However, the prose is extremely compressed and often cryptic ('Missing or unreadable assets, prerequisites or permission block affected probes, not independent safe checks', 'QA_DEADLINE receipts are not observations', 'invented safe-path redactions'), and symbol-heavy naming (R, D, G, Q, NNN, PREV) raises misexecution risk. Several decisions hinge on files or concepts absent from the bundle: scope.md (surface selection), qa-patterns.md (browser method), both report templates, and 'Phase 8 regression gates'. Probe-loop step 2 mixes decide/publish/functional-vs-browser mechanics in one block, the mixed-run directory and timer rules require inference, and mode/budget interactions (caller bound vs. unlimited mode, EARLIER_UTC) are stated tersely without examples. An agent could likely execute and produce deliverables, but with notable ambiguity and dependence on unseen assets."},"latency_ms":16879,"stop_reason":"end_turn","usage":{"input_tokens":4742,"output_tokens":1102},"cost_usd":0.10252}
+{"config":"qa-workflow","item":"a05","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The document is short and readable with a linear Setup → Explore → Fix → Report flow, concrete commands (git status, port probing, screenshot naming, commit format) and clear stop conditions (20 fixes, unrelated-file sprawl). However, it is a single browser-only script rather than the described bundle: there is no native/functional surface selection, no mixed-surface decision, no conditional browser/DX section loading, no authority or isolation rules (e.g., committing directly on the user's branch without a worktree or guard), no blocked-path handling, and no 'add a regression before repair' step — fixes are verified only by manual reload and screenshot. Key terms are undefined: 'important' vs 'minor' issues, 'quick', 'main buttons', how the 0-100 health score is computed, and what 'coverage' means beyond page/screenshot counts. Diagnosis is reduced to 'find the responsible source file.' An agent could execute the browser exploration and produce the report file, but would have to improvise scoring and triage, and would not produce the regression tests, coverage evidence, or native-contract checks the workflow is supposed to deliver."},"latency_ms":12778,"stop_reason":"end_turn","usage":{"input_tokens":1081,"output_tokens":955},"cost_usd":0.05856}
+{"config":"qa-workflow","item":"a05","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The document is short and readable with concrete commands, paths, naming conventions (ISSUE-NNN, issue-NNN-after.png, fix(qa) commit format) and a 20-fix stop rule, so a basic browser-only pass is followable. But it falls well short of the stated workflow scope: there is no surface selection (browser vs native functional vs mixed) or native contract at all, no conditional section loading or authority/precedence rules, no isolation strategy (fixes are committed directly on the working branch with no worktree/branch/rollback contract), no blocked-path handling, no probe budget for exploration (only the fix count is bounded), and critically no 'add a regression test before repair' step—the Fix phase goes straight from source change to manual re-screenshot. Several decisions are left ambiguous: 'important' vs 'minor' issues, 'main buttons', the undefined 0-100 health score computation, 'many unrelated files', and the odd 'feature branch with no URL' condition. Coverage reporting is limited to page/screenshot counts with no notion of untested surfaces or test evidence. An agent could produce the report file for a simple web app, but cannot produce the regression, isolation, native, or coverage deliverables the workflow is supposed to yield."},"latency_ms":14305,"stop_reason":"end_turn","usage":{"input_tokens":1081,"output_tokens":1035},"cost_usd":0.06256}
+{"config":"qa-workflow","item":"a05","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The document is short and readable: setup, explore, fix, and report steps are sequenced plainly, issue records have a defined schema, and stop conditions (20 fixes, sprawling diffs) are explicit. But it is only a browser-only checklist and falls well short of the workflow it is supposed to describe. There is no coverage of native functional or mixed surfaces, no native contract definitions, no conditional loading of browser/DX sections, no authority or isolation rules (e.g., working in a scratch branch, fixtures, data reset), and no blocked-path handling beyond 'ask the user'. Critically, the fix phase skips the required step of adding a regression test before repairing, and 'recheck' is reduced to a reload and screenshot with no automated verification. Several decisions are left undefined: what counts as an 'important' vs 'minor' issue, how the 0-100 health score is computed, what 'main buttons' means, and how probes are bounded. An agent could execute a basic browser smoke pass and produce a report file, but it could not produce the expected deliverables (regression tests, coverage evidence, native results) or make consistent decisions where the document is silent."},"latency_ms":12541,"stop_reason":"end_turn","usage":{"input_tokens":1442,"output_tokens":795},"cost_usd":0.05417}
+{"config":"qa-workflow","item":"p02","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":3,"reasoning":"The Phases 7-11 fragment is well-structured: tiering rules, dirty-tree gate with a concrete AskUserQuestion, a per-issue fix loop (locate → minimal fix → atomic commit → re-test → classify → regression test), explicit commit message formats, a numeric WTF-likelihood stop heuristic with a hard cap, and defined report/TODOS outputs. However, several decisions depend on things not defined here or ambiguously placed: $B, $REPORT_DIR, '{detected test command}', test-framework detection/bootstrap, the health-score formula, the 'standard report template', and Additional Rules 1-10 are all referenced but absent. The learnings refresh sits in Phase 7 yet is phrased per-fix, leaving loop placement unclear. Re-test (8d) is browser-only, so native functional or mixed surfaces called for by the bundle framing have no re-verification contract, and 'regression detected' has no stated criterion. The WTF heuristic has fuzzy semantics (when 'all remaining Low' applies, cumulative vs per-check accounting) and 'Touching unrelated files +20%' conflicts with the hard rule forbidding it. Phase 10 does not say whether the final report overwrites or appends the baseline file, and {user}/{datetime} formats are unspecified; '>2 min → skip and defer' is ambiguous about deferring the test vs the issue. An agent can execute most of this with the rest of the bundle loaded, but on its own it would need to guess at several integration points."},"latency_ms":23395,"stop_reason":"end_turn","usage":{"input_tokens":4137,"output_tokens":1700},"cost_usd":0.12637}
+{"config":"qa-workflow","item":"a02","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser section is concrete where it matters: executable aside repl scripts for orientation, link checks, mobile emulation, API probes and annotated screenshots, explicit sentinels (GSTACK_STEP_OK, ASIDE_DIR), evidence-copy rules, mode definitions with time caps, and firm safety rules (no source reading during discovery, no credential handling, one mutation-confirmation question before acting). However, the prose is extremely dense and leans heavily on cross-references the agent must resolve elsewhere ('shared exploratory loop', 'BROWSER SETUP', 'caller authority', 'exact-replay rule', 'Health Score Rubric', 'Phase 6 report', issue-taxonomy.md), so several decisions (deadline clamping, checkpoint cadence, conflicting-flag resolution, what 'LOOK is not ACT' permits) are not self-contained and some phrasing is ambiguous. As a complete workflow it is incomplete: only Phases 1-5 appear while Phase 6 reporting is referenced but absent; the scoring rubric, report format, and coverage-reporting structure are not defined; and nothing here covers surface selection, isolation, native functional/mixed surfaces, native contracts, diagnosis, adding a regression before repair, recheck, DX loading or blocked paths. An agent can execute the browser probes and gather evidence, but cannot reliably produce the final scored deliverable or handle non-browser surfaces from this document alone."},"latency_ms":18287,"stop_reason":"end_turn","usage":{"input_tokens":4066,"output_tokens":1235},"cost_usd":0.10241}
+{"config":"qa-workflow","item":"p03","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is tightly sequenced (parameters → isolation → artifacts → readiness → charters → probe loop → triage → fix loop → final QA → report) with concrete commands, file naming, checkpoint schema, commit format and scoring formula, so an agent can execute most steps unambiguously. Authority and isolation rules are explicit (clean-tree gate, production permission requirements, non-local mutation prompt, secrets by env-var name), and the browser-blocked path is defined with fallthrough to functional. Gaps keep it from a 5: Diff-aware exploration is described in §6 but never listed as a selectable mode or given a budget, functional `--regression` comparison against a 'previous report' has no defined method, issue-ID assignment (ISSUE-NNN) and the evidence path (`REPORT_DIR/.../evidence/NNN/`) are left implicit, 'top 5 navigation targets' is undefined, and the file claims to be a single ordered workflow while the stated loading model is conditional per-section—minor inconsistencies rather than blocking defects."},"latency_ms":26754,"stop_reason":"end_turn","usage":{"input_tokens":4113,"output_tokens":1864},"cost_usd":0.13433}
+{"config":"qa-workflow","item":"a03","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"The workflow is well-structured with concrete, copy-pasteable aside repl scripts, explicit sentinel outputs (CONSOLE_ERRORS=, NAV=, TEXT_START/END, GSTACK_STEP_OK), a clear baseline manifest schema, a bounded monitoring loop with cadence rules, well-defined alert thresholds (change-based, 2-consecutive-check confirmation), and a prescribed alert format with AskUserQuestion options. Isolation is handled by re-opening pages each check and copying screenshots out of the session dir, and the read-only rule is explicit. Weaknesses: the `--quick` argument is listed but never specified in the phases; Phase 6 'Health Report' has no output format or location, so the key deliverable is under-defined; page-name derivation (e.g., '/' → 'home') and branch capture are unstated; Phase 2 forward-references Phase 3's link check awkwardly; `pwd` inside the JS repl is unexplained; MEDIUM/LOW alert handling and the 'content disappears' text comparison have no concrete criteria; no guidance for auth-gated sites or deadline/early-stop bookkeeping persistence. An agent can execute setup, baseline, discovery, and the monitoring loop reliably, but must improvise the final report and quick mode."},"latency_ms":19050,"stop_reason":"end_turn","usage":{"input_tokens":3889,"output_tokens":1282},"cost_usd":0.10299}
+{"config":"qa-workflow","item":"p05","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"The document is a single, linear web-only workflow with crisp phase gates, named outputs, explicit defaults, numeric scoring formulas, severity rubric, fix-tier rules, and hard stop conditions (git dirty state, 2 reverts, unrelated-file edits, 10-fix checkpoints, 30-fix cap), which makes it easy to follow and execute. However, measured against the stated bundle scope it is incomplete: there is no treatment of native functional or mixed surfaces, no native contracts, no conditional browser/DX section loading, and no explicit blocked-path list beyond 'never type a password'. Within the web flow itself there are gaps that force agent judgment: no rule for assigning an issue to one of the eight scoring categories, no procedure for testing Accessibility/Performance/Links (so 'tested categories' is undefined), `<base>` in the diff-aware command is unspecified, the Mode default is silent for a non-feature branch with no URL, and 'primary control' is undefined. Deliverables (report.md, baseline.json, screenshots/, commits, PR line) are well specified, so an agent can produce the expected outputs for the browser case, but would have to improvise on categorization and would have nothing to execute for non-browser surfaces."},"latency_ms":15853,"stop_reason":"end_turn","usage":{"input_tokens":2610,"output_tokens":1030},"cost_usd":0.0776}
+{"config":"qa-workflow","item":"a04","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The bundle defines a coherent spine: ordered Reads, charter/preflight with explicit budgets, a serialized probe loop with publish-before-dispatch checkpoints, replay-before-repair, handoff rules, and a final-report/annotations step; system-functional.md adds a solid contract map, mode definitions and evidence-retention rules. However, the prose is extremely compressed and several sentences are grammatically broken or ambiguous (e.g., 'permission block affected probes, not independent safe checks'; '/review and /ship, no plan/server is required' immediately followed by 'explicit plan checks ... remain required'; the unbounded/bounded SECONDS resolution; 'QA_DEADLINE receipts'). Key decisions depend on files not in the bundle (scope.md for surface selection, qa-patterns.md for browser method, the report templates, the 'shared isolation/permission preflight' that is referenced but never actually specified), and there is no DX loading path at all despite the evaluation prompt expecting one. The G/Q binary contracts, checkpoint schema and annotations.json shape are concrete enough that an agent with the tools installed could execute and produce charters, exploration-NNN.json/captures and a report, but severity, 'browser scores', Phase 8 gates and template content are left to external authority, so deliverables cannot be fully produced from this bundle alone."},"latency_ms":22478,"stop_reason":"end_turn","usage":{"input_tokens":4381,"output_tokens":1516},"cost_usd":0.11961}
+{"config":"qa-workflow","item":"a01","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The fix-loop mechanics (triage by tier, locate→minimal fix→atomic commit→re-test→classify→regression test) are laid out in a readable, mostly unambiguous sequence, and the dirty-tree gate, commit message format, and revert rule are concrete. However, the document is a fragment of a larger bundle: the title promises Phases 7-11 but Phase 11 is absent, rules are numbered 11-15 with 1-10 missing, and it depends on undefined context—`$B`, `REPORT_DIR`, `snapshot -D`, the 'WTF-likelihood heuristic', the 'standard report template', how a health score is computed, and how a test framework is 'detected' or the user is offered a 'bootstrap'. Nothing addresses native functional surfaces, authority/isolation boundaries, conditional browser/DX loading, or blocked paths; the re-test step assumes a browser surface only. Branching has gaps: 'best-effort' fixes have no follow-up path, 'regression detected' lacks criteria, and Phase 9's 'affected pages' and warning action are vague. An agent could execute the git/fix portion and produce the report shape, but would have to invent tooling semantics and scoring to complete Phases 8d, 8e.5 and 9, so the workflow is partially actionable and clearly incomplete as a standalone bundle."},"latency_ms":15119,"stop_reason":"end_turn","usage":{"input_tokens":3436,"output_tokens":930},"cost_usd":0.08086}
+{"config":"qa-workflow","item":"a05","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The document is concise and its four phases (Setup, Explore, Fix, Report) are easy to read, with concrete URL fallbacks, issue record fields, commit message format, and a fix cap of 20. However, it falls well short of the workflow it is supposed to encode. It covers only the browser surface: there is no selection step for native functional or mixed surfaces, no bounded probe budget or exploration limits, no reproduce/diagnose protocol beyond 'retry once', and critically no requirement to add a regression test before repairing a defect. Authority and isolation are not addressed (it edits and commits directly on the working branch after only a dirty-tree check), there are no native contracts, no conditional loading of browser/DX sections, and no blocked-path or stop conditions beyond the 20-fix heuristic. Several decision points are vague: 'important' vs 'minor' issues, 'quick' fixes, 'main buttons', 'many unrelated files', and the 0-100 health score has no computation rule, so two agents would produce different reports. Deliverables are named but coverage reporting is limited to page/screenshot counts with no mapping to changed files or untested surfaces. An agent could execute a shallow browser smoke test and produce a report, but not the complete, evidence-backed test→regression→fix→verify workflow described."},"latency_ms":13343,"stop_reason":"end_turn","usage":{"input_tokens":1442,"output_tokens":798},"cost_usd":0.05432}
+{"config":"qa-workflow","item":"f02","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is six vague imperatives rather than a workflow. It only addresses a browser surface ('open the site'), with no mechanism for selecting browser vs native functional vs mixed surfaces, no native contracts, no conditional loading of browser/DX sections, and no blocked-path handling. There is no notion of authority (who decides scope, what counts as a bug 'that matters'), no isolation of test environment or data, no bounded exploration (step 2 'look around' and 'don't overdo it' give no probe limits or stopping rules), no reproduction/diagnosis procedure, and critically no 'add a regression before repair' step — fixes are made directly with only 'check again' as verification. The report has no defined structure, evidence requirements, or coverage accounting. Steps are readable in isolation (hence clarity 2 rather than 1), but every decision point is delegated to 'good judgment', so an agent cannot execute this deterministically or produce the expected deliverables."},"latency_ms":9583,"stop_reason":"end_turn","usage":{"input_tokens":561,"output_tokens":548},"cost_usd":0.03301}
+{"config":"qa-workflow","item":"p01","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser-only Phases 1-6 are reasonably well sequenced with concrete commands, a scoring rubric, and evidence rules, but the bundle contains contradictions and undefined references: 'Never read source code' conflicts with diff-aware mode that maps controllers/views to routes; tiers 'determine which issues get fixed' yet no repair, regression-before-repair, or recheck phase exists; `<SKILL_DIR>`, `$REPORT_DIR`, and `/qa-only` are used but never defined; the report template and issue taxonomy live in files not included. Against the stated scope, the workflow is missing native/functional and mixed-surface selection, bounded-probe limits, authority and isolation rules, native contracts, conditional DX loading, and blocked-path handling entirely—the title 'Phases 1-6' implies later phases that are absent. An agent could execute a browser smoke/explore pass and produce a report plus baseline.json, but cannot execute the full diagnose→regression→repair→recheck workflow or handle non-browser surfaces from this document."},"latency_ms":15536,"stop_reason":"end_turn","usage":{"input_tokens":4930,"output_tokens":1079},"cost_usd":0.10325}
+{"config":"qa-workflow","item":"f02","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is a six-line informal checklist, not the multi-file bundle described in the evaluation brief. It is readable but almost every step is ambiguous: 'find wherever the app is running' gives no discovery procedure or fallback if nothing is running; 'see if anything seems off' and 'bugs that matter' are undefined judgments; 'check again' and 'write a report' specify no format, evidence, or coverage criteria. Nearly all expected workflow elements are missing: no authority/scope rules, no isolation or safety constraints, no native/non-browser surface contract, no conditional loading of browser or DX sections, no blocked-path handling, no bounded probe limits, no reproduction/diagnosis step, and critically no regression-test-before-repair requirement. An agent cannot produce the expected deliverables (evidence, coverage, regression tests) from this; it would have to invent the entire methodology, so actionability is minimal."},"latency_ms":7189,"stop_reason":"end_turn","usage":{"input_tokens":922,"output_tokens":336},"cost_usd":0.02602}
+{"config":"qa-workflow","item":"p02","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The fix loop (8a-8f) is well-structured with concrete commands, commit formats, classification states, a WTF-likelihood heuristic with explicit thresholds, and a hard cap, and the triage/tier mapping is unambiguous. However, as a bundle for the described workflow it is substantially incomplete and leans on undefined context: Phases 1-6 (surface selection, exploration probes, baseline health score computation) are absent yet referenced throughout; the doc is purely browser-centric ($B goto/screenshot/console) with no native-functional or mixed-surface path, no authority/isolation contracts, and no blocked-path handling; 'Additional Rules' begin at 11, implying missing rules 1-10; variables like $B, $REPORT_DIR, 'detected test command', and the 'user declined bootstrap' branch are never defined or triggered anywhere in this file; 'continue with setup' points to an undefined setup step; the health score formula needed for Phase 9's regression warning is not given. The clean-tree gate is oddly placed after the baseline report is already written. Most notably, the stated workflow calls for adding a regression test before repair, but 8e.5 puts the regression test after the fix is committed and verified, and it's skipped when classification isn't 'verified', so the test never demonstrates the failure first. Report outputs and TODOS.md updates are concretely specified, so an agent in a browser-only gstack context could produce deliverables, but it would need to infer or guess at native surfaces, missing variables, and the baseline dependencies."},"latency_ms":15633,"stop_reason":"end_turn","usage":{"input_tokens":4137,"output_tokens":1098},"cost_usd":0.09627}
+{"config":"qa-workflow","item":"a05","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"clearness_note":"","completeness":2,"actionability":3,"reasoning":"The document is short and readable: setup, explore, fix, and report phases follow a logical order with concrete commands, file paths, commit message format, and stop conditions (20 fixes, unrelated-file sprawl). However, measured against the stated scope it is badly incomplete. It is browser-only — there is no surface-selection step for native functional or mixed targets, no native contract definitions, no conditional loading of browser/DX sections, and no blocked-path handling. There is no authority or isolation model (no worktree/branch isolation, no statement of what the agent may change without asking). The 'add a regression before repair' phase is absent entirely; fixes are verified only by manual re-click and screenshot, not by a test. Exploration probes are not bounded (no per-page time/click/depth caps, no handling of auth, destructive actions, or external links). Diagnosis is reduced to 'find the responsible source file.' Several decisions are left to judgment with no criteria: 'important' vs 'minor' issues, how the 0-100 health score is computed, what 'many unrelated files' means, and how to count or report coverage beyond pages/screenshots. An agent could still execute the browser loop and produce a report file, but key deliverables (health score, regression evidence, coverage) would be underspecified or missing."},"latency_ms":12586,"stop_reason":"end_turn","usage":{"input_tokens":1081,"output_tokens":839},"cost_usd":0.05276}
+{"config":"qa-workflow","item":"a01","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The fix-loop mechanics (triage tiers, minimal fix, one-commit-per-fix, classify, regression-test attribution/naming/type decision, revert rule) are written clearly and the dirty-tree gate is well specified. However the document is titled 'Phases 7-11' yet Phase 11 is absent, and it depends on Phases 1-6 and many undefined artifacts: $B and $REPORT_DIR variables, the 'WTF-likelihood heuristic', the 'standard report template', the health-score formula, the test-framework detection and 'bootstrap' decline flow, and the 'top-of-skill learnings pull'. The bundle described in the evaluation brief (browser vs native functional vs mixed surface selection, authority/isolation, native contracts, conditional browser/DX loading, blocked paths) is entirely missing—this file is browser-only ($B goto/screenshot/console) with no native surface handling or blocked-path behavior. Decision points like 'best-effort' vs 'verified' and 'cannot be fixed from source' are left to judgment without criteria, and the '>2 min exploration' rule is unmeasurable. An agent could execute the core fix/commit/retest loop if the surrounding context exists, but cannot execute the complete workflow from this document alone."},"latency_ms":14207,"stop_reason":"end_turn","usage":{"input_tokens":3436,"output_tokens":934},"cost_usd":0.08106}
+{"config":"qa-workflow","item":"p04","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":4,"reasoning":"Phases 7-11 are concrete and mostly unambiguous: severity-tiered triage, a disciplined fix loop (diagnose → regression test before repair with value card and naming/attribution rules → minimal fix → re-test → commit → classify → record), explicit self-regulation thresholds (every 5 fixes, 2 reverts, 50-fix cap), report/artifact paths with shell commands, and TODOS.md handling. Conditional loading of sections/exploratory.md and browser-verify.md states trigger, installed-path authority, and blocked-path fallback clearly. However, the document is dense and several points are ambiguous or undefined within the provided bundle: Phases 1-6 are a stub that fully delegates to unseen files (exploratory.md, qa-patterns.md, system-functional.md), so the probe loop, contract map, 'ordered preparation', and health-score computation are not evaluable; `--quick` appears as both a Tier and a Mode override; `--regression <baseline>` mode is never explained later; `/qa-only`, 'caller-owned authority', 'explicitly approved browser bootstrap', and 'Phase 5 is the before' reference concepts not defined here; baseline.json schema and how to derive {user}/{branch}/{datetime} are unspecified; `$REPORT_DIR` vs 'Output dir' naming is inconsistent. An agent could execute the fix/verify/report portion well, but the baseline exploration and some mode handling depend on external content and leave interpretive gaps."},"latency_ms":22965,"stop_reason":"end_turn","usage":{"input_tokens":4598,"output_tokens":1564},"cost_usd":0.12418}
+{"config":"qa-workflow","item":"p02","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The fix loop (8a–8f) is sequenced and mostly concrete: minimal fix, one commit per issue, before/after screenshot, classify verified/best-effort/reverted, regression test with explicit skip conditions, attribution comment, naming scheme, WTF-likelihood thresholds and a 50-fix hard cap. The load condition ('after the Phases 1-6 baseline report is written') is stated. However, the document is weakened by several ambiguities and gaps. The clean-working-tree gate says 'then start QA' and 'continue with setup', contradicting its placement after the baseline phases; the stash option never states when to pop. Phase 9 'Re-run QA on all affected pages' and 'compute final health score' have no procedure or formula here, and 'standard report template', '{detected test command}', 'user declined bootstrap', '$B', '$REPORT_DIR', and the 'top-of-skill learnings pull' are all undefined references to other files, so the agent cannot verify these contracts from this bundle. The re-test step is browser-only ($B goto/screenshot/console) with no native-functional or mixed-surface equivalent, and nothing addresses authority boundaries, isolation, native contracts, or blocked paths beyond 'never modify CI config / existing tests'. WTF-likelihood terms ('All remaining Low severity', 'touching unrelated files', '>2 min exploration') are judgment calls without definitions; 'regression detected' in 8e has no detection criterion. Additional Rules numbered 11-15 imply rules 1-10 elsewhere. An agent in a browser project with the gstack toolchain could likely execute this and produce the report/commits, but it would need to improvise on health scoring, test-command detection, and any non-browser surface."},"latency_ms":18845,"stop_reason":"end_turn","usage":{"input_tokens":3776,"output_tokens":1338},"cost_usd":0.10466}
+{"config":"qa-workflow","item":"a06","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":4,"reasoning":"The workflow has a well-defined sequence (parameters → context → surface selection via the exploratory STOP → artifact ownership → conditional browser setup → charters/probes → reporting STOP → dual report write) with explicit blocked-path behavior for missing assets, mode conflicts, mixed regression baselines, and output-permission failures. Authority/isolation concepts (caller, owned, no repairs, no state bootstrapping) and report naming/sanitization rules are concrete enough to execute. Clarity suffers, however, from dense, legalistic phrasing and cross-references to constructs defined only in external files (the 'no-repeat rule', 'exploratory §4 annotations and materialize', each method's Full/Quick/Regression definitions), the awkward 'load now but defer' instruction around the exploratory gate, the odd recovery procedure for an accidental early reporting Read, and the switch from the qa-only directory to the qa/gstack-qa directory for browser-setup.md that an agent must notice carefully. Completeness is good for a report-only variant: all phases, decision points and deliverables (REPORT_FILE, project copy, templates per surface, charters, proposed tests) are specified, though Quick/Regression time bounds and template contents are deferred to unseen sections. Overall executable, but the agent must parse heavy prose and trust several referenced-but-absent definitions."},"latency_ms":21729,"stop_reason":"end_turn","usage":{"input_tokens":4314,"output_tokens":1563},"cost_usd":0.12129}
+{"config":"qa-workflow","item":"f03","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":2,"reasoning":"The document is a truncated, browser-only stub that does not match the described bundle. Many sections are empty headers: 'Find the browse binary' has no detection command and NEEDS_SETUP/SKILL_DIR are undefined; Phase 2 auth lists conditions ('If the user provided a cookie file:') with no actions; Phase 3 'Get a map of the application:' and Phase 4 'At each page:' / 'Check mobile viewport if relevant:' have no steps; Phase 6 is one sentence. Core concepts are referenced but never defined: diff-aware mode, regression mode (--regression baseline.json), the '$B' variable, /qa-only, and the 'baseline health score' recorded at the end with no formula or scale. Tiers 'determine which issues get fixed' yet there is no reproduce/diagnose/fix phase, no regression-before-repair, and no recheck. The Modes section ('Pick whatever depth seems right') is non-decisional. Nothing addresses native functional or mixed surfaces, authority/isolation, native contracts, conditional browser/DX loading, or blocked paths. Only Phase 5 (two evidence tiers), the framework-detection heuristics, and the Important Rules are concrete and executable; an agent could improvise browser exploration but cannot produce the stated deliverables (health score, regression, native coverage) from this text."},"latency_ms":21384,"stop_reason":"end_turn","usage":{"input_tokens":2472,"output_tokens":1448},"cost_usd":0.09712}
+{"config":"qa-workflow","item":"f05","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The bundle is generally well-structured: parameters, mode-conflict handling, test-plan/learnings lookups with explicit no-write constraints, surface selection with per-method Full/Quick/Regression semantics, mixed-run ordering and shared deadlines, REPORT_DIR/REPORT_FILE derivation with collision and preservation rules, conditional browser-setup loading with blocked-path handling, and charters written before any probe clock. However, it contains a direct, unresolved contradiction: 'Run the Selected Checks' instructs the agent to fix critical/high bugs in source immediately and commit, while the title, preamble and Additional Rule 1 state NEVER fix, never commit, report-only. An agent cannot execute both; this undermines the core authority model and makes the workflow ambiguous at its most consequential decision point. Secondary gaps: the body has no explicit finalization phase invoking sections/reporting.md (only the index mentions it), and the browser-setup path resolution ('beside this host's installed caller skill') is somewhat underspecified. Deliverables are otherwise well-defined, but the contradictory repair instruction significantly lowers clarity and actionability."},"latency_ms":15599,"stop_reason":"end_turn","usage":{"input_tokens":3608,"output_tokens":989},"cost_usd":0.08553}
+{"config":"qa-workflow","item":"f04","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":2,"completeness":2,"actionability":2,"reasoning":"The Aside repl scripts and the baseline-vs-current alert taxonomy are concrete, but the workflow contradicts itself on its central decision: Phase 5 says 'Alert the user immediately on the first error you see' while the same phase and the Important Rules say a first occurrence is only 'pending' and alerts require 2+ consecutive checks. Phase 1 says 'Monitor for as long as seems necessary' despite a stated 10m default and --duration bounds; the check interval is never specified; --quick mode is listed but never defined. Phase 2 says 'Save what you collected somewhere you can find it later' with no path or schema, yet Phase 4 and 7 depend on a `baseline.json` with 'the same manifest schema as Phase 2' that was never defined. Phase 6 ('When you are done, write up a report') gives no format, location, or evidence/coverage requirements. Performance threshold is 'noticeably slower' in Phase 5 but 2x/1.5x in the rules; page-name derivation from URLs is undefined; LOW alerts for 404s have no notification path. Instructions about comparing console messages by identity are misplaced in the baseline phase. An agent can run the probes but cannot reliably decide when to alert, how long to run, or what to deliver."},"latency_ms":19794,"stop_reason":"end_turn","usage":{"input_tokens":3721,"output_tokens":1448},"cost_usd":0.10961}
+{"config":"qa-workflow","item":"f01","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"The document is internally contradictory and structurally broken. Phase 7 defines tiers (Quick/Standard/Exhaustive) and then immediately says 'Fix every issue... whatever tier was selected' and 'if time is short, fix only the critical ones,' leaving no executable triage decision. Phase 8 has two steps both labeled '8a', and the commit step appears before the locate and fix steps, so the agent would commit before any change exists. Phase 9 is missing entirely; there is no regression-test step, no re-verify/recheck step, and no health-score recomputation, yet rule 13 references 'Phase 8e.5' which does not exist. Phase 10 ('Write up what you did') defines no report format, evidence, or coverage output. Rule 15 invokes an undefined 'WTF-likelihood heuristic,' and the Additional Rules start at 11 with rules 1-10 absent. The workflow also never addresses surface selection (browser/native/mixed), probe bounds, isolation, native contracts, conditional loading, or blocked paths that the evaluation asks for. Only the dirty-tree check, learnings-search keyword guidance, and TODOS.md update are concretely executable, which is insufficient for an agent to produce the expected fix→verify deliverables."},"latency_ms":10567,"stop_reason":"end_turn","usage":{"input_tokens":2307,"output_tokens":632},"cost_usd":0.05467}
+{"config":"qa-workflow","item":"a05","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The document is short and readable, with concrete setup steps, a simple explore loop, a fix loop with a commit convention and a 20-fix cap, and a defined report path. However, it falls well short of the intended workflow. Clarity suffers from undefined terms the agent must act on: 'important' vs 'minor' issues, 'quick' fixes, 'many unrelated files', and especially the 0-100 health score, which has no formula at all; Explore step 3 also reorders step 2 after the fact rather than being a prior decision. Completeness is weak: it is browser-only with no surface-selection step for native/functional or mixed targets, no native contracts, no bounded probe budgets for exploration (unbounded page/button/form crawling), no diagnosis phase, no 'add a regression test before repair' requirement, no conditional loading of browser/DX tooling, no handling of blocked or unreachable paths (auth walls, failing dev server mid-run, missing browser tooling), no isolation/authority rules beyond a dirty-tree prompt, and no coverage accounting beyond 'pages visited'. Actionability is moderate: an agent can run the browser crawl, record ISSUE-NNN entries with screenshots, apply fixes, and write the report file, but the health score and triage decisions are unexecutable without inventing criteria, and the lack of regression tests means the 'verify' claim rests only on manual re-click plus console check."},"latency_ms":15750,"stop_reason":"end_turn","usage":{"input_tokens":1081,"output_tokens":1077},"cost_usd":0.06466}
+{"config":"qa-workflow","item":"p05","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"The document is a single, linear, web-only workflow with well-ordered phases, named gating outputs, concrete defaults, explicit severity rubric, scoring formulas, and clear stop/ask conditions (dirty tree, mode conflict, revert limits, 30-fix cap). Steps like the fix loop (reproduce → failing test → minimal fix → re-test → classify → commit) are directly executable. However, it does not address several things the evaluation scope asks for: there is no surface selection (browser vs native functional vs mixed), no native contracts, no authority/isolation model, no conditional loading of browser/DX sections, and no handling of blocked paths — it is one file, not a bundle. Within its own scope there are also gaps: Diff-aware `<base>` is unspecified; issue-to-category assignment (Functional/UX/Accessibility/Visual/Performance/Content) has no rubric and Phase 3 never says how to test accessibility, performance, or detect broken links despite scoring them; 'tested categories' and 'primary control' are undefined; stash-pop timing for option B is not stated; the browser tooling is unnamed. These leave an agent able to run the web flow but forced to improvise on scoring and on anything beyond browser testing."},"latency_ms":18052,"stop_reason":"end_turn","usage":{"input_tokens":2610,"output_tokens":1203},"cost_usd":0.08625}
+{"config":"qa-workflow","item":"a03","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"Strengths: the phase structure, argument list, and concrete Aside REPL scripts (console-error hook, navigation timing, text snapshot, screenshot copy-out, same-origin HEAD link check) give an agent executable steps; the alert taxonomy (CRITICAL/HIGH/MEDIUM/LOW), change-vs-baseline logic, 2-consecutive-check transient tolerance, round cadence rules, and AskUserQuestion formats are well specified; isolation is handled by re-opening pages each round and never overwriting baselines. Weaknesses that reduce scores: `--quick` is listed as an argument but never defined in any phase; Phase 6 'Health Report' is a single sentence with no format, file output, coverage summary, or evidence listing despite being the primary deliverable; `<page-name>` derivation, the `pwd` global in the REPL, and how to obtain `branch` are unspecified; Phase 2 awkwardly cross-references Phase 3's link script, which is written for a single `<url>` rather than per-page; MEDIUM/LOW alerts have no defined surfacing path (only CRITICAL/HIGH trigger a prompt); the handling of user choices (Rollback vs. read-only rule, Dismiss suppression semantics, 'stop early' mechanism) is undefined; text-snapshot disappearance is retained as evidence but never mapped to an alert type. The document is browser-only and does not address native/mixed surfaces, blocked paths, or regression-before-repair, so as a complete QA workflow bundle it is a competent but partial monitor skill with a vague final report."},"latency_ms":29319,"stop_reason":"end_turn","usage":{"input_tokens":4250,"output_tokens":2046},"cost_usd":0.1448}
+{"config":"qa-workflow","item":"a03","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"Phases 1–5 are reasonably well sequenced with concrete aside REPL scripts, a baseline manifest schema, explicit alert severities, a 2-consecutive-check debounce rule, and a defined cadence/deadline loop, so an agent can get a monitoring run started. However several decisions are left undefined or inconsistent: `--quick` is listed as an argument but never specified in any phase; Phase 6 'Health Report' just says 'summarize the results' with no template, required fields, output path, or coverage/evidence summary, leaving the primary deliverable vague; `<page-name>` derivation from paths and `<current branch>` acquisition are unspecified; baseline mode only captures `--pages` or homepage while monitoring mode auto-discovers pages, so baseline and monitored sets can diverge; the text snapshot is captured 'for when content disappears' but no alert criterion uses it; MEDIUM/LOW alerts and the follow-through for Rollback/Investigate choices are not defined (and Rollback conflicts with the read-only rule); and the Phase 3 link check runs `fetch`/`pwd` in REPL scope with uncertain semantics. Relative to the stated evaluation lens (native contracts, isolation, regression-before-repair, blocked paths), the bundle addresses none of those, since it is purely a browser-monitoring skill, further limiting completeness."},"latency_ms":26130,"stop_reason":"end_turn","usage":{"input_tokens":3889,"output_tokens":1930},"cost_usd":0.13539}
+{"config":"qa-workflow","item":"p01","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser phases (Initialize → Authenticate → Orient → Explore → Document → Wrap Up) are concrete, with runnable `$B` commands, a well-specified health-score rubric, and a baseline.json schema, so an agent can execute browser QA end-to-end. However, measured against the stated bundle contract, large pieces are absent or contradictory. There is no surface-selection step (browser vs native functional vs mixed), no native-test contract, no isolation/authority rules, no blocked-path handling, no conditional DX-loading logic, and no 'add regression before repair / recheck' phase even though the Tier table says tiers 'determine which issues get fixed' — fixing is never defined in Phases 1-6. Rule 5 ('Never read source code') directly conflicts with diff-aware mode (mapping controllers/views to routes) and with any fix activity. The doc depends on external files (`qa/templates/qa-report-template.md`, `qa/references/issue-taxonomy.md`) and references `/qa-only` without defining either, so deliverable format and the per-page checklist details are not verifiable from this bundle. `categoryScores` is only partially specified ('...'), and Quick mode's instruction to skip the checklist yet still produce a health score across eight categories leaves scoring underdetermined. Net: solid as a browser-only recipe, but incomplete and internally inconsistent as the full workflow described."},"latency_ms":17090,"stop_reason":"end_turn","usage":{"input_tokens":4930,"output_tokens":1227},"cost_usd":0.11065}
+{"config":"qa-workflow","item":"p03","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is a single sequential file with ten well-ordered phases, concrete gates (git status check, --quick/--regression conflict, Aside readiness sentinel), an explicit four-field checkpoint schema, severity/tier definitions, stop rules, a fully specified health-score formula and a detailed report template, so an agent can execute it end to end and produce the stated deliverables. Remaining ambiguities keep it from a 5: 'Diff-aware' exploration appears in §6 but not in the Mode table or §1 selection logic; the 'surface's probe directory' and `REPORT_DIR/.../evidence/NNN/` paths are only implicitly defined for single-surface runs; ISSUE-NNN numbering and `{target}` filename sanitization are unspecified; mapping of browser findings into UX/Visual/Content/Accessibility score categories is left to judgment; the 30-second Quick browser budget for six pages with screenshots is likely infeasible; and the §1 requirement for a clean tree slightly conflicts with §2's instruction to include uncommitted files in the diff. Isolation, authority (permission naming target/operation/effect), blocked-path handling and separation of browser score vs functional contract outcomes are clear and actionable."},"latency_ms":26419,"stop_reason":"end_turn","usage":{"input_tokens":4113,"output_tokens":1824},"cost_usd":0.13233}
+{"config":"qa-workflow","item":"f04","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":2,"completeness":2,"actionability":2,"reasoning":"The browser probe scripts (hook injection, NAV/CONSOLE_ERRORS/TEXT markers, screenshot copy-out, HEAD link check) are concrete and executable, and the alert-on-delta principle is stated well. But the document contradicts itself on the core decision: Phase 5 says 'alert immediately on the first error you see' while the same phase and Important Rules say a first occurrence is only 'pending' and alerts require 2+ consecutive checks. Phase 1 says 'monitor for as long as seems necessary' despite a 10m default and --duration flag; no check interval is given for the loop; --quick and --duration are listed as arguments but never handled. Phase 2 says 'save what you collected somewhere you can find it later' yet Phase 4 and 7 reference a `baseline.json` and 'the same manifest schema as Phase 2' that are never defined. Phase 6 is a single sentence ('write up a report') with no format, fields, or path, so the primary deliverable is undefined. Performance regression is 'noticeably slower' in Phase 5 and only loosely resolved by the 2x/1.5x rule. Post-alert branches (A–D) have no follow-on behavior. An agent can run the probes but cannot determine when to alert, how long to run, or what the final artifacts must contain."},"latency_ms":20875,"stop_reason":"end_turn","usage":{"input_tokens":3721,"output_tokens":1551},"cost_usd":0.11476}
+{"config":"qa-workflow","item":"a02","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser section is concrete where it gives scripts: Orient/read-flow, link checking, mobile emulation, and annotated-screenshot probes are copy-ready with explicit sentinels (GSTACK_STEP_OK, ASIDE_DIR, DIFF_START/END) and clear screenshot-copy/Read rules. Mode selection (Full/Quick/Regression/Diff-aware) is defined with time bounds and sensible fallbacks (missing baseline blocks only the comparison). However, the text is extremely terse and leans on undefined external contracts: the 'shared exploratory loop' and its checkpoint/exact-replay/deadline-guard rules, BROWSER SETUP's access decision and $B states, the Health Score Rubric, qa/references/issue-taxonomy.md, $REPORT_DIR, TODOS.md, and 'Phase 6 report' are all referenced but absent, so an agent cannot resolve what a checkpoint is, how to score, or what the report must contain. Phrases like 'Clamp the shared loop's deadline guard to the caller's running deadline', 'LOOK is not ACT', and 'Diff-aware selects scope, not another pass' are cryptic without the companion files. Judged as the complete workflow, it omits native-functional and mixed-surface contracts, authority/isolation rules, diagnosis, adding a regression before repair, recheck, blocked paths, and DX loading — only Phases 1–5 of the browser path exist, and even they end before documentation outputs are specified. Probes are executable, but the end-to-end deliverables (coverage report, health score, regression test) cannot be produced from this file alone."},"latency_ms":18128,"stop_reason":"end_turn","usage":{"input_tokens":4066,"output_tokens":1226},"cost_usd":0.10196}
+{"config":"qa-workflow","item":"a02","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser methodology file itself is dense but mostly followable: modes (Full/Quick/Regression/Diff-aware) have concrete triggers and bounds, and Phases 1-5 include copy-pasteable aside repl scripts with explicit sentinels (GSTACK_STEP_OK, ASIDE_DIR, CONSOLE_ERRORS) and screenshot handling rules. However, it leans heavily on external artifacts that are not present in what was supplied: the 'shared exploratory loop' and its checkpoint/exact-replay/deadline-guard rules, BROWSER SETUP's access decision and sentinel/mutation rules, the Health Score Rubric, the Phase 6 report format, $REPORT_DIR provenance, qa/references/issue-taxonomy.md, and TODOS.md conventions. Phase 6 (report) is referenced but absent, so the deliverable schema is undefined here. Nothing in this file covers native functional/mixed surface selection, isolation, native contracts, regression-before-repair, recheck, or blocked paths, so judged as the complete workflow the bundle is substantially incomplete. Some phrasing is terse to the point of ambiguity ('Clamp the shared loop's deadline guard to the caller's running deadline', 'LOOK is not ACT', 'Diff-aware selects scope, not another pass') and relies on prior context. An agent could execute the browser probes and capture evidence, but cannot reliably produce scored reports, native-surface coverage, or regression tests from this file alone."},"latency_ms":17513,"stop_reason":"end_turn","usage":{"input_tokens":4066,"output_tokens":1197},"cost_usd":0.10051}
+{"config":"qa-workflow","item":"p04","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The fix loop (8a–8f), triage tiers, commit/classification rules, naming conventions for regression files, and blocked-path handling for missing section files are concrete and mostly unambiguous. However, the bundle delegates the entire baseline (Phases 1–6), the probe loop, native unit/integration/E2E rules, the functional contract map, and browser re-verification to external files (exploratory.md, qa-patterns.md, system-functional.md, browser-verify.md) that are not present, so the agent cannot actually execute or audit those phases from this document. Several definitions are muddled: `--quick` appears under both Tier and Mode, the `--regression <baseline>` mode is never used anywhere, `{user}` and `$REPORT_DIR` are undefined, the health score is referenced but never specified, 'Phase 5 is the before' relies on external numbering, and the 8a.5/8e.5 insertions plus dense, somewhat convoluted prose (e.g. 'with caller-owned authority') reduce readability. The per-issue and summary report fields are reasonably well listed, but the overall workflow is only partially executable as provided."},"latency_ms":22121,"stop_reason":"end_turn","usage":{"input_tokens":4237,"output_tokens":1157},"cost_usd":0.10022}
+{"config":"qa-workflow","item":"f02","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is a six-line sketch, not an executable workflow. It assumes a single browser surface ('open the site') and never addresses native functional or mixed surfaces, surface selection, bounded probes, isolation, authority/permission boundaries, native contracts, conditional browser/DX loading, or blocked paths. There is no reproduction or diagnosis step, no requirement to add a regression test before repair, no recheck protocol, and no definition of report contents (evidence, coverage). Key decisions are deferred to 'good judgment' ('bugs that matter', 'don't overdo it', 'see if anything seems off'), leaving the agent with no criteria. Steps are readable in isolation but far too vague to follow unambiguously, and an agent could not produce the expected deliverables from this."},"latency_ms":8556,"stop_reason":"end_turn","usage":{"input_tokens":561,"output_tokens":507},"cost_usd":0.03096}
+{"config":"qa-workflow","item":"f03","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":2,"reasoning":"The document is a browser-only QA baseline whose structure is readable at the heading level but riddled with hollow sections: Phase 2 has three conditional headers ('If the user specified auth credentials', 'If the user provided a cookie file') with no body, Phase 4 says 'At each page:' then lists nothing, the Responsiveness item trails off with no command, and Phase 3 never says how to actually 'get a map' (which commands to run). 'Find the browse binary' references NEEDS_SETUP without defining how it is detected, and Phase 6 is just 'Summarize what you found' plus a trailing instruction to record an undefined 'baseline health score'. The Modes section ('pick whatever depth seems right') contradicts the explicit Quick/Standard/Exhaustive tier table and the diff-aware mode promised in Setup is never described. Tiers 'determine which issues get fixed' yet there is no fix, diagnose, regression-before-repair, or recheck phase at all. Against the stated scope—native functional surfaces, mixed surfaces, authority/isolation rules, native contracts, conditional browser/DX loading, blocked paths, coverage reporting—the bundle is almost entirely absent; only the browser exploration/documentation phases exist, and even those lack concrete commands. An agent could roughly perform screenshot-based exploration using the Important Rules and evidence tiers, but could not produce the full deliverables (regression tests, repair verification, coverage report, health score) or make decisions for native or blocked paths."},"latency_ms":13497,"stop_reason":"end_turn","usage":{"input_tokens":2833,"output_tokens":850},"cost_usd":0.07083}
+{"config":"qa-workflow","item":"p04","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":4,"reasoning":"Phases 7-11 are well specified: severity-tiered triage, a detailed regression-before-repair protocol (value card, naming convention, red-then-green proof, attribution comment), minimal-fix rule, re-test requirements, commit scoping, classification states, revert handling, explicit self-regulation thresholds (every 5 fixes, 2 reverts, 50 cap), and defined report/PR/TODOS outputs. Conditional loading is handled with clear STOP gates, installed-path authority, and a defined blocked-path fallback. However, Phases 1-6 (the actual baseline/probe loop, browser health score computation, functional contract map, isolation/auth setup) are entirely delegated to sections not present in this bundle, so the main file alone cannot be executed for the baseline and the score delta is undefined here. Several ambiguities reduce clarity: `--quick` appears as both a Tier and a Mode override; 'caller-owned authority' and 'authorized codification' are undefined terms; '/qa-only' is referenced without definition; 'Phase 5 is the before' points to an external numbering; the datetime format for the project-context artifact is unspecified; and some sentences are compressed to the point of being hard to parse ('Before rechecking a reproduced browser defect after repair; never for a functional-only repair, Read...'). An agent could produce the expected deliverables given the external sections exist, but would face real decision ambiguity on flag parsing and would be blocked on scoring without the referenced files."},"latency_ms":25474,"stop_reason":"end_turn","usage":{"input_tokens":4598,"output_tokens":1717},"cost_usd":0.13183}
+{"config":"qa-workflow","item":"f03","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":2,"completeness":2,"actionability":2,"reasoning":"The document has a recognizable skeleton (parameter table, six named phases, a solid Important Rules list, and a concrete two-tier evidence procedure in Phase 5), but it is riddled with stubs and contradictions. Setup says 'Find the browse binary:' with no command or detection method for NEEDS_SETUP. Setup references 'diff-aware mode (see Modes below)', but the Modes section is a single vague sentence ('Pick whatever depth seems right') that defines neither diff-aware nor the --regression mode listed in the parameter table. Phase 2 has two conditional headings ('If the user specified auth credentials', 'If the user provided a cookie file') with empty bodies. Phase 3 says 'Get a map of the application:' with no commands. Phase 4's 'At each page:' and 'Responsiveness... if relevant:' trail off with nothing. Phase 6 is 'Summarize what you found' plus a trailing instruction to 'Record baseline health score' with no formula, scale, or output location. Tiers are defined in terms of 'which issues get fixed' yet no fix/repair, regression-before-repair, or recheck phase exists. None of the requested authority, isolation, native functional contracts, conditional browser/DX loading, or blocked-path handling appears anywhere. An agent could roughly explore pages and document screenshot-backed issues, but it cannot reliably locate the binary, authenticate, select a mode, compute the health score, or produce the coverage/regression deliverables the complete workflow is supposed to yield."},"latency_ms":20050,"stop_reason":"end_turn","usage":{"input_tokens":2833,"output_tokens":1477},"cost_usd":0.10218}
+{"config":"qa-workflow","item":"a04","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":4,"reasoning":"Authority boundaries (caller owns decisions/tests/fixes; discovery writes only reports/evidence), isolation preflight, the native contract map with explicit outcome vocabulary, blocked/not-run semantics, pass criteria, and the deadline/evidence tooling (G/Q commands, checkpoint file schema, annotations.json + materialize) are all concretely specified, so an agent can execute probes and produce the expected deliverables. Conditional loading (scope.md, system-functional.md, qa-patterns.md for browser only) and the Read-before-probe ordering are stated. However, the prose is extremely dense and symbol-heavy (R, D, G, Q, NNN, PREV, SECONDS, EARLIER_UTC), mixes bounded/unbounded and browser/functional branches inline, and relies on undefined external references (Phase 8 gates, 'surface report template', 'shared isolation/permission preflight', REPORT_DIR naming, how the 12-probe count is tracked), which creates interpretive friction. Referenced files (scope.md, qa-patterns.md, templates) are not in the bundle, so browser-side and scope-selection behavior cannot be fully verified, and no DX-loading guidance appears despite being mentioned as a concern."},"latency_ms":20638,"stop_reason":"end_turn","usage":{"input_tokens":4381,"output_tokens":1548},"cost_usd":0.12121}
+{"config":"qa-workflow","item":"a05","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The document is short and its individual steps are concrete (git status gate, URL fallback order, screenshot naming, commit message format, 20-fix cap), so the browser-only path is easy to follow. But several decisions are left undefined: what makes an issue 'important' vs 'minor' or 'quick', how the 0-100 health score is computed, what 'many unrelated files' means, and how 'main buttons' are chosen. Against the stated expected workflow it is badly incomplete: there is no surface selection (native functional or mixed surfaces are never mentioned), no bounded probe limits, no requirement to add a regression test before repairing, no authority/isolation rules for what the agent may touch or run, no native contracts, no conditional browser/DX loading or blocked-path handling, and the report has no coverage section beyond page/screenshot counts. Diagnosis is reduced to 'find the responsible source file.' An agent could produce a basic browser QA report and ad-hoc fixes from this, but it could not produce regression-backed repairs, native coverage, or the evidence/coverage deliverables the workflow is supposed to yield, and it would have to invent key scoring and triage criteria."},"latency_ms":11648,"stop_reason":"end_turn","usage":{"input_tokens":1081,"output_tokens":761},"cost_usd":0.04886}
+{"config":"qa-workflow","item":"f01","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"The workflow is internally contradictory and structurally broken. Phase 7 first says tiers decide which issues get fixed, then says 'Fix every issue in the report, including low severity, whatever tier was selected' and then 'if time is short, fix only the critical ones' — three incompatible rules. Phase 8 has two steps labeled '8a', and the commit step comes before locating or fixing the bug, so the agent would commit before making changes. Phase 9 (verification/recheck) is entirely missing, and Phase 8e.5 (regression tests) is referenced in the rules but does not exist, so the core 'add a regression before repair, recheck behavior' contract is undefined. Phase 10 ('Write up what you did') gives no format, evidence or coverage requirements. Rules are numbered 11–15 with no 1–10, and 'WTF-likelihood heuristic' is undefined. The learnings-search step is reasonably specified, and the dirty-tree gate is clear, but the agent cannot execute a coherent fix→verify loop or produce the expected evidence/coverage deliverables from this document."},"latency_ms":9415,"stop_reason":"end_turn","usage":{"input_tokens":1946,"output_tokens":560},"cost_usd":0.04746}
+{"config":"qa-workflow","item":"p01","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser-only portion (Setup, Modes, Phases 1-6, health rubric) is concretely written with executable `$B` commands, screenshot conventions, and a deterministic scoring rubric, so an agent can run the exploration/documentation half. However, measured against the described complete workflow, the bundle is badly incomplete: there is nothing on native functional or mixed surface selection, authority/isolation, native contracts, blocked paths, conditional DX loading, adding a regression before repair, performing the repair, or rechecking behavior afterward. 'Tiers determine which issues get fixed' and 'Phases 1-6' imply later fix/recheck phases that are absent, and referenced files (`qa-report-template.md`, `issue-taxonomy.md`, `/qa-only`) are not provided. Clarity suffers from internal contradictions: Rule 5 'Never read source code' conflicts directly with diff-aware mode's instruction to map controller/view/model files to routes; `$REPORT_DIR` is never defined; the port-detection chain mixes `&&`/`||` in a way that short-circuits incorrectly; Quick tier appears both as a mode and a fix-scope tier with different meanings. Actionability is adequate for producing a baseline report and baseline.json, but the agent cannot produce the full expected deliverables (regression test, repair, recheck evidence, coverage of native surfaces) from this document alone."},"latency_ms":16603,"stop_reason":"end_turn","usage":{"input_tokens":4930,"output_tokens":1154},"cost_usd":0.107}
+{"config":"qa-workflow","item":"f01","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":2,"reasoning":"The document contradicts itself at the core decision point: tier definitions say Quick/Standard/Exhaustive gate which severities get fixed, yet Phase 7 then instructs to fix every issue regardless of tier (or 'only critical if time is short'). Phase 8 has two sub-steps both labeled '8a', with 'Commit' placed before 'Locate source' and 'Fix', so the commit instruction precedes any change. Phase 9 (verification/recheck) is absent entirely despite the title 'Fix → Verify', and rules reference a non-existent 'Phase 8e.5' regression-test step and an undefined 'WTF-likelihood heuristic'. Phase 10 is a single vague sentence with no report structure, no health-score re-measurement, no evidence/coverage format. Rules are numbered 11-15 with no 1-10, implying missing context. Nothing addresses browser vs native surface selection, isolation, native contracts, or blocked paths. The clean-tree gate, keyword rules for learnings search, commit format and TODOS.md update are concrete and usable, which is why actionability isn't a 1, but an agent cannot execute a coherent fix-then-verify loop or produce the expected deliverables from this bundle."},"latency_ms":11768,"stop_reason":"end_turn","usage":{"input_tokens":1946,"output_tokens":790},"cost_usd":0.05896}
+{"config":"qa-workflow","item":"f02","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is a six-line sketch, not a workflow. Steps are readable but each hinges on undefined judgment ('seems off', 'bugs that matter', 'look better'), so two agents would execute it very differently. It covers only a browser surface and omits native functional/mixed surface selection, bounded probe limits, defect reproduction and diagnosis procedure, the add-regression-before-repair contract, isolation/authority rules, conditional browser/DX section loading, blocked-path handling, and any report schema or coverage/evidence format. Nothing here lets an agent locate the app reliably, decide what to test, know when to stop, prove a fix, or produce a checkable deliverable."},"latency_ms":8839,"stop_reason":"end_turn","usage":{"input_tokens":561,"output_tokens":472},"cost_usd":0.02921}
+{"config":"qa-workflow","item":"a03","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The document is a single-file /canary monitoring skill, not the multi-file QA bundle the evaluation frame describes: there is no surface selection (browser/native/mixed), no bounded-probe exploration, no reproduce-diagnose-regression-repair loop, no native contracts, no authority/isolation rules, and no conditional loading or blocked-path handling. Judged on its own terms, phases 1-5 are reasonably concrete (working aside repl scripts, explicit sentinel lines, 60s cadence with non-overlap rule, baseline.json schema, change-based alert thresholds, 2-consecutive-check confirmation). But there are real gaps: Phase 6 'Health Report' is a one-sentence placeholder with no schema or required fields, so the main deliverable is undefined; `--quick` is listed as an argument but never handled in any phase; MEDIUM/LOW alerts have no defined surfacing path despite appearing in the alert template; page-name derivation (e.g. '/' -> 'home'), branch detection, and the `pwd` reference inside the REPL are unspecified; the forward dependency on Phase 3's link check from Phase 2 muddles ordering; and despite being a 'Visual Monitor', screenshots are only captured, never compared to baselines. An agent could run the capture loop and raise alerts, but would have to invent the final report and several branching behaviors."},"latency_ms":21375,"stop_reason":"end_turn","usage":{"input_tokens":3889,"output_tokens":1384},"cost_usd":0.10809}
+{"config":"qa-workflow","item":"a01","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The fix-loop mechanics (locate → minimal fix → atomic commit → re-test → classify → regression test → report) are laid out in a reasonable step order with concrete commit formats, attribution comments, and a clear regression-test decision table, so an agent can follow the local steps. However, the document is only a fragment of the described workflow: it presupposes Phases 1-6, a 'standard report template', a 'WTF-likelihood heuristic', `$B`, `$REPORT_DIR`, a 'detected test command', and a 'user declined bootstrap' decision, none of which are defined or referenced to a specific file. The title promises Phases 7-11 but Phase 11 never appears (rules are numbered 11-15, which is confusing). The pre-check says 'continue with setup' but setup is undefined, and stash-pop timing is never resolved. The document is browser-only (screenshots, console, snapshot) and gives no handling for native functional or mixed surfaces, authority/isolation constraints, conditional loading, or blocked paths that the evaluation brief asks for; regression tests are added after repair rather than before, contrary to the stated workflow intent. Phase 9's 're-run QA' and 'compute final health score' are not operationalized. Phase 7's per-bug learnings refresh is placed in triage rather than inside the loop, creating ordering ambiguity. Net: locally clear and partially executable, but incomplete as a standalone bundle with many unresolved dependencies and missing surface coverage."},"latency_ms":18099,"stop_reason":"end_turn","usage":{"input_tokens":3797,"output_tokens":1274},"cost_usd":0.10167}
+{"config":"qa-workflow","item":"p03","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is tightly structured with explicit, mostly deterministic decisions: parameter defaults and conflict handling (--quick vs --regression), a hard git-status gate with enumerated choices, surface selection and isolation/authority rules (production needs named permission, unknown ownership blocks the probe), a concrete browser readiness sentinel with a defined blocked path that continues the functional surface, per-behavior charters with budgets, a checkpoint-before-every-probe loop with a fixed four-field schema and a two-failure confirmation rule, severity-ordered fix tiers, a red-before-green regression requirement with naming/attribution conventions, recheck/commit/classify/stop rules, a fully specified health-score formula, and a concrete report structure that keeps browser scores and functional contract outcomes separate. Weaknesses that keep it from 5s: directory layout is underspecified (evidence path uses `REPORT_DIR/.../evidence/NNN/`, 'surface's probe directory' and `$REPORT_DIR/screenshots/` are not reconciled with the mixed-run browser/functional split); 'Diff-aware' appears as a browser technique but is not a selectable Mode and its trigger is only implied; `<base>` for the diff is never defined; ISSUE-NNN numbering and the baseline.json comparison procedure for Regression mode are not specified; 'minimize the input' lacks a stopping criterion; and the 30-second Quick browser budget is likely infeasible in practice. The document also claims to be a single file despite the stated conditional-loading bundle framing, so conditional load conditions are not actually encoded. Still, an agent could execute this end to end and produce the stated deliverables with only minor interpretive choices."},"latency_ms":28414,"stop_reason":"end_turn","usage":{"input_tokens":4474,"output_tokens":2141},"cost_usd":0.15179}
+{"config":"qa-workflow","item":"p04","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The hub file gives a recognizable Test→Fix→Verify spine with concrete tier gates, STOP/load directives with blocked-path fallbacks, a regression-before-repair rule with a naming scheme and value card, explicit classification (verified/best-effort/reverted), self-regulation caps, commit discipline, and report/PR/TODOS outputs with shell snippets. However, as presented the bundle is only the hub: Phases 1-6 (baseline, probe loop, authority/isolation, native contract map, health score, Phase 5 'before' evidence) are entirely delegated to sections/exploratory.md, qa-patterns.md and system-functional.md that are not included, so the core exploration and scoring behavior cannot be verified. Several definitions are ambiguous or dangling: `--quick` appears under both Tier and Mode, `--regression <baseline>` mode is never operationalized, `/qa-only` is referenced without definition, `{user}-{branch}` placeholders have no derivation, `$REPORT_DIR` vs the Output dir parameter is implied not stated, and no DX loading exists despite being an expected surface. Prose is dense and jargon-laden ('caller-owned authority', 'shared loop', byte-clamping rules) which slows unambiguous execution. An agent could execute Phases 7-11 reliably given a completed baseline, but the full workflow's clarity and completeness depend on unseen sections."},"latency_ms":23485,"stop_reason":"end_turn","usage":{"input_tokens":4237,"output_tokens":1532},"cost_usd":0.11897}
+{"config":"qa-workflow","item":"a04","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The bundle defines a coherent flow (ordered Reads → charter/preflight → bounded probe loop with published checkpoints → replay-before-repair → revalidation → handoff → final report) and gives concrete tool contracts for the deadline guard (G) and evidence recorder (Q), a clear functional contract map with outcome classifications, mode definitions (Full/Quick/Regression) and the annotations.json/materialize reporting path. However the prose is extremely compressed and jargon-heavy, with undefined references (Phase 8, Fix-First, ASK approval, test_stub, 'resolved REPORT_DIR', the exact subdirectory name), cryptic rules ('invented safe-path redactions', 'identity hash unchanged', 'QA_DEADLINE receipts are not observations') and a somewhat inverted checkpoint ordering for functional probes ('publish before probing' but 'the next capture publishes it') that an agent must untangle. Completeness is undercut because the mandatory first Read (scope.md, which drives surface selection) and the report templates are not in the bundle, browser specifics are deferred entirely, and mixed-run/caller-bound interactions are only sketched. An agent with the gstack tooling installed could execute the functional path and produce charters, captures, checkpoints, annotations and a report, but would have to infer several details and depends on absent files, so each dimension lands at a middling 3."},"latency_ms":22847,"stop_reason":"end_turn","usage":{"input_tokens":4742,"output_tokens":1469},"cost_usd":0.12087}
+{"config":"qa-workflow","item":"p01","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The document gives a concrete, command-level browser QA flow (setup/binary detection, auth, orient, explore, document, wrap-up, health score rubric, baseline.json) that an agent could largely follow. However, it has internal contradictions and ambiguities: Rule 5 'Never read source code' conflicts with diff-aware mode, which requires mapping changed controller/view/model files to routes; tiers 'determine which issues get fixed' but no fix/repair phase exists in Phases 1-6; Quick mode says 'no detailed issue documentation' while Phase 5 mandates documenting every issue; the trailing 'Record baseline health score' line is orphaned; Phase 6's regression-mode instruction duplicates the Modes section with slightly different sequencing. Severity definitions, the issue taxonomy, and the report template are external files not present in the bundle, so category assignment and report structure are not self-contained. Against the stated evaluation frame (browser vs native functional vs mixed surface selection, bounded probes, reproduce/diagnose, add regression test before repair, recheck, authority/isolation, native contracts, conditional DX loading, blocked paths), the bundle covers only the browser-exploration half: there is no native/functional surface selection, no isolation or authority rules, no regression-before-repair or recheck phase, no blocked-path handling, and the 'Phases 1-6' title implies later phases that are absent. Deliverables (report, screenshots, baseline.json) are defined well enough to produce, but the baseline schema is partial ('...') and the port-detection shell chain is fragile. Overall: readable and executable for browser smoke/exploration, but incomplete and partly self-contradictory as a full QA workflow."},"latency_ms":21131,"stop_reason":"end_turn","usage":{"input_tokens":5291,"output_tokens":1408},"cost_usd":0.12331}
+{"config":"qa-workflow","item":"a06","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":4,"reasoning":"The workflow defines a coherent sequence (parameters → context → surface selection via the exploratory STOP → report-dir ownership → conditional browser setup → charters-before-probes → reporting STOP → local and state-dir report copies) with explicit authority limits, isolation rules, collision-safe artifact naming, mode conflict resolution, mixed-surface ordering/deadline rules, and well-specified blocked-path behavior for missing assets, baselines, or unwritable destinations. Completeness is strong for an orchestration bundle: inputs, defaults, outputs, templates, and report structure are enumerated, and native functional contracts (fixture ownership, proposed tests naming what they protect) are covered. Clarity suffers from dense, convoluted prose ('Load the shared preparation gate now (the exploratory STOP just below)...'), undefined references to rules living in other files ('the no-repeat rule', 'exploratory §4 annotations and materialize'), ambiguity about which installed directory holds the report templates (qa vs qa-only), and an awkward 'recover from an accidental early Read' instruction. Actionability is good overall—an agent can produce REPORT_FILE and the state copy with the stated naming—but relies heavily on correctly loaded external sections and on parsing some verbose conditional sentences without error."},"latency_ms":21821,"stop_reason":"end_turn","usage":{"input_tokens":4314,"output_tokens":1543},"cost_usd":0.12029}
+{"config":"qa-workflow","item":"f04","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":2,"completeness":2,"actionability":2,"reasoning":"The role framing, argument list, and aside repl scripts are concrete and copy-pasteable, and the alert-on-deltas principle is clear. However the core decision logic is self-contradictory: 'Be fast. Alert the user immediately on the first error you see' directly conflicts with 'A first occurrence is pending, not yet an alert' and the rule 'Only alert on patterns that persist across 2+ consecutive checks'; 'noticeably slower' conflicts with the 2x/1.5x guidance. Key execution parameters are undefined: Phase 1 says 'monitor for as long as seems necessary' despite a 10m default and --duration flag; Phase 5 gives no check interval or check-number semantics; the --quick mode is never mapped to any phase; MEDIUM/LOW alert handling is unspecified. Outputs are badly underspecified: Phase 2 says 'save what you collected somewhere you can find it later' yet Phase 4 and Phase 7 depend on 'the same manifest schema' and a `baseline.json` whose structure is never defined; Phase 6 is a single sentence ('write up a report') with no format, fields, or path, so the primary deliverable is undefined. Phase 3's 'top 5' ranking is unspecified and `pwd` inside the repl is assumed. An agent could run the scripts but would have to invent the monitoring cadence, alert threshold, manifest schema, and report format, so results would be inconsistent across runs."},"latency_ms":17932,"stop_reason":"end_turn","usage":{"input_tokens":4082,"output_tokens":1224},"cost_usd":0.10202}
+{"config":"qa-workflow","item":"p03","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is dense but well-ordered: parameters and tier/mode semantics (including the --quick/--regression conflict) are explicit, the dirty-worktree gate, isolation and permission rules, report-dir collision rules, browser readiness sentinel, charter-before-probe requirement, checkpoint JSON schema, confirm-by-replay rule, severity ladder, red-test-before-fix sequence with naming/attribution, commit scope, classification and stop rules, health-score formula and report contents are all concrete enough to execute. Gaps keep it from a 5: 'Quick exploration' is only defined for browser surfaces (functional Quick behavior is unspecified); 'Diff-aware' is a browser technique but not a mode in the parameter table, leaving ambiguity about whether Full on a feature branch means every page or mapped pages; `<base>` for the diff is never defined; 'top 5 navigation targets' and the minimization step lack criteria; evidence path uses an ellipsis (`REPORT_DIR/.../evidence/NNN/`) and screenshots go to `$REPORT_DIR/screenshots/` while mixed runs are told to keep browser probes under `REPORT_DIR/browser`; ISSUE-NNN assignment point is implicit; the stop rule 'when only Low issues remain' sits awkwardly with the Exhaustive tier; and the file claims to be the whole workflow run in order while the bundle framing says sections are condition-loaded, with only §4 actually carrying a load condition. None of these block execution but each requires the agent to make an undocumented choice."},"latency_ms":24449,"stop_reason":"end_turn","usage":{"input_tokens":4474,"output_tokens":1747},"cost_usd":0.13209}
+{"config":"qa-workflow","item":"f01","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":2,"reasoning":"The document is a fragment with internal contradictions and structural gaps. Phase 7 directly contradicts the tier table ('Fix every issue... whatever tier was selected' vs Quick/Standard/Exhaustive scoping), leaving an agent with no executable triage decision. Phase 8 has two '8a' steps with Commit placed before Locate/Fix, inverting the fix→verify→commit order; steps 8c–8e and the '8e.5' regression-test step referenced in Rule 13 don't exist, and Phase 9 (recheck/verify) is missing entirely, so 'add a regression before repair' and 'recheck behavior' have no defined procedure. Phase 10 is one sentence with no report format, health-score recomputation, or evidence/coverage requirements. Rule 15 invokes an undefined 'WTF-likelihood heuristic', rules are numbered 11–15 implying absent rules 1–10, and the learnings refresh is placed in triage though it is per-fix. None of the surface selection, authority, isolation, native contracts, conditional browser/DX loading, or blocked-path material the bundle is supposed to cover appears. Only the clean-tree gate, commit format, and TODOS.md update are cleanly executable."},"latency_ms":13962,"stop_reason":"end_turn","usage":{"input_tokens":1946,"output_tokens":940},"cost_usd":0.06646}
+{"config":"qa-workflow","item":"a06","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The workflow has a recognizable phase sequence (parse parameters → optional context → surface selection via exploratory STOP → report artifact ownership → conditional browser setup → charters/probes → reporting STOP → dual report writes) and clearly states authority limits (report-only, no fixes/commits, owned-only writes), isolation (synthetic identity, no credential requests), and blocked-path handling for missing assets, baselines, or state directories. However, clarity suffers from dense, overloaded prose and cross-referenced constraints: the exploratory section is to be 'loaded now' but half its procedure deferred; an undefined 'no-repeat rule' and 'exploratory §4 annotations and materialize' are invoked without definition; the accidental-early-Read recovery instruction is convoluted; browser setup lives under a different skill directory (qa vs qa-only) with a terse locating rule; and 'follow the sections below in order' conflicts somewhat with 'bundle order is not execution order.' Completeness is moderate: Full/Quick/Regression semantics, charter format, probe bounds, and consistency checks are all delegated to unseen section files, so the document alone cannot define several decision points. Actionability is workable for an agent given the referenced files exist—REPORT_DIR/REPORT_FILE naming, UTC run IDs, project-copy paths, template choice, and proposal-only repair fields are concrete—but the many conditional qualifiers (empty-dir ownership, mixed-mode baseline resolution, shared deadlines across surfaces) create room for inconsistent execution."},"latency_ms":19578,"stop_reason":"end_turn","usage":{"input_tokens":4314,"output_tokens":1334},"cost_usd":0.10984}
+{"config":"qa-workflow","item":"p02","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"Within its own scope (Phases 7-11) the document is reasonably clear: tiers, severity-ordered fix loop, one-commit-per-fix, classification states, WTF-likelihood thresholds, and report locations are explicit. However, it is only the back half of the workflow. Nothing covers Phases 1-6, surface selection (browser vs native functional vs mixed), bounded exploration probes, native contracts, conditional browser/DX loading, authority/isolation rules, or blocked paths—all of which the agent is expected to execute. It also contradicts the stated requirement to add a regression before repair: here the test is written only after fix+commit+verify (8e.5), and is skipped in several cases. Many references are undefined in-file: `$B`, `$REPORT_DIR`, `{detected test command}`, `{user}`, the 'top-of-skill learnings pull', 'standard report template', 'user declined bootstrap', and rules numbered 11-15 imply 1-10 live elsewhere. Smaller ambiguities: the Phase 7 learnings refresh reads as per-issue but sits in triage; the stash-pop step for option B is never scheduled; 'All remaining Low severity +10%' is vague; Phase 9 'Re-run QA' depends on the missing baseline procedure; re-test (8d) is browser-only with no native path. An agent could produce fix commits and a report with reasonable assumptions, but cannot execute the full described workflow from this bundle alone."},"latency_ms":18019,"stop_reason":"end_turn","usage":{"input_tokens":3776,"output_tokens":1293},"cost_usd":0.10241}
+{"config":"qa-workflow","item":"p01","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser-QA portion (Setup, Modes, Phases 1-6, health rubric, rules) is reasonably well specified with concrete commands, evidence tiers, and a scoring formula, so an agent could run an exploratory browser pass and produce a report/baseline. However, the bundle covers only a fraction of the described workflow: there is no surface-selection step (browser vs native functional vs mixed), no bounded-probe budget, no diagnose/repair phase, no 'add regression before repair' or recheck step, and nothing on authority, isolation, native contracts, conditional DX loading, or blocked paths. The title 'Phases 1-6' and 'Tiers determine which issues get fixed' imply later phases that are absent, and the referenced template/taxonomy files are not included. Internal contradictions hurt clarity: 'Never read source code' conflicts with diff-aware mode's file-to-route mapping; Mode default 'full' conflicts with automatic diff-aware mode; '/qa-only' is referenced but undefined; Quick mode is both a tier and a mode. Baseline.json categoryScores schema is partially elided ('...'). Deliverables for the browser subset are clear, but the complete decision flow and fix/regression outputs are not executable from this bundle."},"latency_ms":15899,"stop_reason":"end_turn","usage":{"input_tokens":4930,"output_tokens":1048},"cost_usd":0.1017}
+{"config":"qa-workflow","item":"a02","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"This is only the browser-technique file of a larger bundle, and it leans heavily on externals that are not present: the 'shared exploratory loop' (which owns execution order and checkpoint rules), BROWSER SETUP and its access decision, the Health Score Rubric, qa/references/issue-taxonomy.md, $REPORT_DIR/$B conventions, TODOS.md, and a 'Phase 6 report' that is referenced but never defined. The evaluation brief asks about native functional surfaces, mixed surfaces, regression-before-repair, recheck, and evidence/coverage reporting; none of those phases exist in this document (only Phases 1-5 for browser probing), so the complete workflow is not well-defined here. Clarity is middling: mode selection (Full/Quick/Regression/Diff-aware) and tie-break rules (ask on conflicting flags, missing baseline blocks comparison not the run) are explicit, and the aside scripts are concrete and sentinel-marked, but prose is extremely terse and jargon-dense ('Clamp the shared loop's deadline guard to the caller's running deadline', 'LOOK is not ACT', 'Diff-aware selects scope, not another pass'), and the ordering between diff-aware scoping, surface isolation, and local-app discovery is somewhat tangled. There is mild tension between 'Map diffs with source before probes' and 'Never read source code during browser discovery' that an agent must infer resolves by phase. Actionability is reasonable within scope: copy-pasteable orient/flow, link-check, mobile, and annotated-screenshot scripts, explicit screenshot copy-then-Read rule, safety rules about mutations and credentials. But an agent cannot produce the final deliverable (report, score, coverage) from this file alone, and placeholders like locator('e12') and Health Score Rubric require external context."},"latency_ms":19808,"stop_reason":"end_turn","usage":{"input_tokens":4066,"output_tokens":1394},"cost_usd":0.11036}
+{"config":"qa-workflow","item":"p02","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"What is present (Phases 7-11) is mostly readable with concrete commit formats, classification states, a regression-test recipe, and a self-regulation heuristic. But as a workflow bundle it is badly truncated: it depends on Phases 1-6, the 'top-of-skill learnings pull', rules 1-10, the 'standard report template', $B/$REPORT_DIR, `snapshot -D`, test-framework detection and the 'bootstrap' prompt, health-score computation, and {user} — none of which are defined or loaded here. None of the surface-selection, native functional contracts, isolation/authority, conditional browser/DX loading, or blocked-path material the evaluation asks about appears at all; re-test is browser-only. Internal ambiguities also hurt execution: the clean-tree gate says 'continue with setup' though this is post-baseline; 'Touching unrelated files: +20%' conflicts with the rule forbidding it; 'All remaining Low severity: +10%' is undefined; 8e.5 says 'Generate unit tests, mock everything' right after prescribing integration/component tests; Phase 9 'Re-run QA' has no procedure to follow. An agent could execute the fix/commit/revert loop but could not reliably produce the final report, health-score delta, or regression tests without the missing pieces."},"latency_ms":15963,"stop_reason":"end_turn","usage":{"input_tokens":3776,"output_tokens":1177},"cost_usd":0.09661}
+{"config":"qa-workflow","item":"p01","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The document is a reasonably well-structured browser-only baseline (Phases 1-6) with concrete `$B` commands, a defined health-score rubric whose weights sum to 100, and a clear baseline.json shape. However, as the complete QA workflow it is missing most of what the framing demands: there is no native/functional (non-browser) surface selection, no mixed-surface guidance, no reproduce/diagnose phase, no 'add regression test before repair' step, no repair or recheck phase, no isolation/authority rules, and no blocked-path handling. The 'Tiers determine which issues get fixed' table presupposes a fix phase that never appears, and 'Quick' is defined both as a Tier (`--quick`) and as a Mode with different semantics, creating a real decision ambiguity. Several referenced artifacts (`qa/templates/qa-report-template.md`, `qa/references/issue-taxonomy.md`, severity definitions, `/qa-only`) are loaded conditionally but their contracts are undefined here, and `<SKILL_DIR>` / `$REPORT_DIR` are never set. Rule 5 ('Never read source code') directly contradicts diff-aware mode, which requires reading changed controllers/views to map routes. The port-detection shell chain has precedence issues, and the console rubric overlaps at 10 errors. An agent could execute the browse-based exploration and emit a report with screenshots and baseline.json, but could not confidently produce the regression-before-repair, native coverage, or recheck deliverables, nor resolve the tier/mode and source-code contradictions without guessing."},"latency_ms":21627,"stop_reason":"end_turn","usage":{"input_tokens":5291,"output_tokens":1525},"cost_usd":0.12916}
+{"config":"qa-workflow","item":"a02","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser methodology is tightly structured with concrete aside repl scripts, explicit sentinels (GSTACK_STEP_OK, DIFF_START/END), mode selection rules, and framework hints, so an agent can run individual probes. However it is only the browser-surface file and leans heavily on undefined external contracts: the 'shared exploratory loop' and its checkpoint/exact-replay rules, BROWSER SETUP and its access decision, `$B` states, `$REPORT_DIR`, the Health Score Rubric, `qa/references/issue-taxonomy.md`, TODOS.md, and a 'Phase 6 report' that does not exist in this document (phases stop at 5). Native functional surfaces, isolation, native contracts, blocked paths, adding a regression before repair, recheck, and evidence/coverage reporting are not defined here, so the full workflow the prompt describes cannot be executed from this bundle alone. Terse phrasing ('diagnosis caller-owned', 'Clamp the shared loop's deadline guard', 'LOOK is not ACT', 'Diff-aware selects scope, not another pass') assumes prior knowledge and leaves decision edges ambiguous (e.g., what 'isolating a browser surface' entails, how depth-flag conflicts are resolved for /review and /ship). Actionability is moderate: probes, screenshots, link checks and mobile emulation are directly runnable, but final deliverables (score, report structure, coverage statement) are not specified."},"latency_ms":17468,"stop_reason":"end_turn","usage":{"input_tokens":4427,"output_tokens":1230},"cost_usd":0.10577}
+{"config":"qa-workflow","item":"a04","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":4,"reasoning":"The bundle is unusually precise on mechanics: exact guard/evidence binaries (G, Q) with argument order, a strict four-field checkpoint schema, capture-ID discipline, deadline semantics (bounded vs unbounded, --deadline vs --timeout-ms), replay-before-repair rules, outcome vocabulary (pass/fail/blocked/not run/inconclusive/not applicable), mode behaviors (Full/Quick/Regression with explicit baseline-blocking), a contract map with observable criteria, and pass/blocked semantics for /review and /ship. Blocked paths (missing assets, missing baseline, expiry, invalid D) are explicitly non-passing. These make the loop executable and the deliverables (charters, exploration-NNN.json, annotations.json, materialized report) well specified. Weaknesses: the prose is extremely dense and telegraphic, with several undefined cross-references ('Phase 8 regression gates', 'Fix-First', 'ASK approval', 'explicit plan checks', 'Do not start D'), and the functional section defers to a 'shared isolation/permission preflight' that exploratory.md never actually defines beyond listing 'isolation' as a charter field. Surface selection depends on scope.md and browser method on qa-patterns.md, neither present, and no DX loading path exists despite being asked about. The 12-probe cap for /review and /ship must be hand-counted since G only tracks time. The functional vs browser checkpoint publication split (Q publishes vs agent Writes) is resolvable but initially reads as contradictory. Overall: executable by a careful agent, but requires significant inference and lacks some referenced definitions."},"latency_ms":20551,"stop_reason":"end_turn","usage":{"input_tokens":4381,"output_tokens":1476},"cost_usd":0.11761}
+{"config":"qa-workflow","item":"a02","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"This file is a dense but mostly coherent browser-technique section: modes (Full/Quick/Regression/Diff-aware) have concrete triggers and time caps, Phases 1-5 are sequenced, and the aside repl scripts are copy-ready with sentinels (GSTACK_STEP_OK, ASIDE_DIR, DIFF_START/END), explicit screenshot copy/Read steps, safe-link filtering and a HEAD-only-locally rule. Framework hints and safety rules (no source reading during discovery, single AskUserQuestion for non-LOCAL mutations, never refuse the browser for selected surfaces) are clear. However, clarity suffers from muddled ordering (Diff-aware is 'automatic with no URL' yet the local-app discovery appears after 'selecting and isolating a browser surface' and Full is 'default with a URL'), terse phrases like 'clamp the shared loop's deadline guard' and 'Diff-aware selects scope, not another pass', and heavy reliance on undefined externals ($B, $REPORT_DIR, Health Score Rubric, shared exploratory loop checkpoint rule). Completeness is weak against the stated evaluation scope: a Phase 6 report is referenced but never defined, no health-score rubric or report/coverage format is given, and nothing here covers surface selection, isolation, authority, native functional/mixed surfaces, regression-before-repair, DX loading, or blocked paths—all delegated to other files not present. Actionability is decent for probing and evidence capture but an agent cannot produce the final deliverables (score, report, coverage summary) or resolve mode conflicts from this file alone."},"latency_ms":18828,"stop_reason":"end_turn","usage":{"input_tokens":4427,"output_tokens":1431},"cost_usd":0.11582}
+{"config":"qa-workflow","item":"p05","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"The document is a well-sequenced single-file web-only workflow: each phase has a named output gate, concrete defaults, explicit stop-and-ask triggers, reproduction-before-fix, failing-test-first, and a defined report/baseline schema. Clarity is good but not flawless: `--quick` is used for both Mode and Fix tier (collision), the Mode default is undefined when no URL is given off a feature branch, `<base>` in the diff command is never defined, and Phase 4 assigns issues to eight categories (Accessibility, Performance, UX, etc.) without any rule for categorizing an issue or any Phase 3 step that actually tests accessibility/performance, so 'tested categories' is ambiguous. Completeness is the weakest: measured against the stated bundle scope, there is no native-functional or mixed-surface path, no native contracts, no conditional browser/DX loading sections, no isolation strategy (branch/worktree) beyond a git-status gate, and blocked paths are limited to 'never type a password' and 'edit only responsible files'. Actionability is strong for the web case—an agent can produce report.md, baseline.json, screenshots and commits—but the flag collision and category-scoring ambiguity would force judgment calls."},"latency_ms":16304,"stop_reason":"end_turn","usage":{"input_tokens":2610,"output_tokens":1086},"cost_usd":0.0804}
+{"config":"qa-workflow","item":"a02","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":4,"reasoning":"The browser QA file is dense and highly concrete: modes (Full/Quick/Regression/Diff-aware) have explicit triggers and time caps, Phases 1-5 are sequenced, and every probe ships as a copy-pasteable aside repl script with sentinel output (GSTACK_STEP_OK, DIFF_START/END, ASIDE_DIR) plus an explicit screenshot copy/Read step, link-check semantics (LINK/LINK ?/ERR), framework detection, and safety rules (one AskUserQuestion for non-LOCAL mutations, no source reading during discovery, never refuse browser for selected surface). That makes execution strongly actionable for an agent. Clarity suffers from extreme compression and jargon that presumes other bundle files: 'the shared exploratory loop owns execution order', 'Clamp the shared loop's deadline guard', 'LOOK is not ACT', 'Diff-aware selects scope, not another pass', 'Resolve conflicting depth flags by asking' are terse enough to be ambiguous without the referenced sections. Completeness is limited within this file: Phase 6 (report), the Health Score Rubric, $REPORT_DIR ownership, the exact-replay rule, BROWSER SETUP's access decision, issue-taxonomy.md, and the regression-before-repair/recheck steps are all delegated elsewhere; native-functional surfaces, isolation and blocked-path handling are only alluded to (e.g., 'API/CLI/job/worker/webhook targets do not select it', missing baseline blocks comparison). Within its stated scope it is solid, but an agent cannot produce the final deliverable from this file alone."},"latency_ms":16806,"stop_reason":"end_turn","usage":{"input_tokens":4066,"output_tokens":1110},"cost_usd":0.09616}
+{"config":"qa-workflow","item":"f05","split":"dev","phase":"old-a","sample":2,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The bundle is otherwise well-structured: parameters, isolation identity, test-plan/learnings lookups with explicit no-write constraints, surface/mode resolution, REPORT_DIR/REPORT_FILE ownership rules, conditional browser-setup loading, blocked-path handling, and charter-before-probe sequencing are all defined. However, 'Run the Selected Checks' contains a direct, unresolved contradiction with the workflow's core authority: it instructs the agent to fix critical/high bugs in source and commit them so the report shows them resolved, while the header, Request intent and Additional Rule 1 state NEVER fix, never edit product code, never commit. For a report-only skill this is a fatal ambiguity—an agent following that sentence violates the governing constraint, and the document gives no precedence rule to resolve it. Secondary issues: the browser-setup section is sourced from a different skill's directory (qa/gstack-qa) with only a brief locating hint, the exploratory STOP is split across two phases with somewhat dense phrasing, and a mixed-regression baseline resolution is described but not fully operationalized. The contradiction drives clarity and actionability down sharply despite otherwise solid coverage."},"latency_ms":16666,"stop_reason":"end_turn","usage":{"input_tokens":3247,"output_tokens":932},"cost_usd":0.07907}
+{"config":"qa-workflow","item":"f05","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The document is structured with a clear section index, request parameters, isolation rules, report artifact naming, conditional browser setup and explicit blocked-path handling, and it mostly defines when shared sections load. However, 'Run the Selected Checks' contains a direct instruction to fix and commit critical/high-severity bugs in source, which flatly contradicts the title, the opening mandate ('NEVER fix anything'), and Additional Rule 1 ('Never commit, stash or bootstrap'). An agent following phases in order hits an executable command that violates the workflow's core authority limits, making the mandate ambiguous and risking destructive actions. Additional gaps: the reporting section appears only in the index with no explicit 'finalize report' phase in the body; the ordering between 'Select Surfaces' (load exploratory but defer probes) and 'Run the Selected Checks' (resume) is convoluted; and browser setup depends on a sibling skill's assets with only brief locating guidance. Outputs (REPORT_DIR/REPORT_FILE, blockers, charters) are reasonably well defined, but the contradictory repair instruction significantly undermines clarity and safe actionability."},"latency_ms":18637,"stop_reason":"end_turn","usage":{"input_tokens":3247,"output_tokens":1214},"cost_usd":0.09317}
+{"config":"qa-workflow","item":"a06","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The top-level file has strong authority/isolation framing (caller vs. owned, report-only hard limits, no state-setup helpers), explicit mode-conflict handling, concrete REPORT_DIR/REPORT_FILE naming, UTC run-dir collision rules, blocked-path handling for missing assets, and a sensible linear phase order (params → context → learnings → surface selection → artifacts → conditional browser setup → checks → output). However, the prose is extremely dense and self-referential: 'load the shared preparation gate now (the exploratory STOP just below)… defer charters, clocks and probes' and the 'accidental early Read' recovery paragraph are hard to parse on first pass. Key decision content is delegated to sections not present in the bundle (exploratory.md's Full/Quick/Regression definitions, 'selected-method Reads', '§4 annotations and materialize', the reporting 'consistency check', the 'no-repeat rule'), so an agent working from this file alone cannot define a charter, a probe budget, or the finalization steps. Ambiguities remain around which 'installed QA directory' holds templates (qa-only vs. qa), the cross-skill dependency on qa/gstack-qa for browser setup, and the duplicated STOP boilerplate that mixes 'read' and 'follow' triggers. Deliverables (local report + project state copy with fallbacks) are well specified, but execution depends heavily on unseen sections, making the workflow followable only with those loaded and with careful reading."},"latency_ms":23346,"stop_reason":"end_turn","usage":{"input_tokens":4314,"output_tokens":1628},"cost_usd":0.12454}
+{"config":"qa-workflow","item":"a05","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The document is short and readable with a plain Setup → Explore → Fix → Report flow, but it is a single browser-only script rather than the multi-file, conditionally loaded bundle the evaluation frame expects. Clarity suffers from undefined terms: 'important' vs 'minor' issues, 'main buttons', how the 0-100 health score is computed (before and after), and a contradiction between Setup (ask for URL if none respond) and Explore step 3 (feature branch 'with no URL'). Completeness is weak: there is no surface selection (browser vs native functional vs mixed), no native/CLI/API contract testing, no bounded probe budgets, no diagnosis step, no regression test added before repair, no authority/permission or isolation model beyond a git-status check, no conditional browser/DX tooling loading, and no handling of blocked paths (auth walls, missing dev server, unavailable browser). Coverage reporting is limited to page/screenshot counts. Actionability is moderate: an agent could follow it to produce screenshots, commits, and a markdown report, but it would have to invent the health score formula, the importance triage, and any regression-protection step, so the expected deliverables (regression evidence, coverage, native results) would be incomplete or ad hoc."},"latency_ms":15965,"stop_reason":"end_turn","usage":{"input_tokens":1442,"output_tokens":985},"cost_usd":0.06367}
+{"config":"qa-workflow","item":"f05","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The document is otherwise well-structured (parameter parsing, isolation, report-dir ownership, conditional browser loading, blocked-path handling), but it contains a direct, unresolved contradiction: the header and Additional Rule 1 forbid any fix, edit, or commit ('NEVER fix anything... Never commit'), yet 'Run the Selected Checks' instructs the agent to 'fix it in the source right away and commit the fix' for critical/high bugs. An agent cannot follow both, and the wrong choice violates the workflow's core report-only authority contract. Secondary gaps: the body flow never explicitly invokes the reporting.md finalization step (only the index mentions it), the mode table has odd formatting, and much of the executable detail (Full/Quick/Regression definitions, probe bounds, evidence format) lives in external sections not shown, so completeness and actionability depend heavily on them. The contradiction alone makes the workflow unsafe to execute as written."},"latency_ms":15517,"stop_reason":"end_turn","usage":{"input_tokens":3247,"output_tokens":1081},"cost_usd":0.08652}
+{"config":"qa-workflow","item":"a06","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":4,"reasoning":"The bundle is well-structured with a section index, parameter table, explicit STOP gates, consistent blocked-path handling, clear authority/isolation definitions (caller/owned), and a defined output contract (REPORT_DIR/REPORT_FILE naming, project copy, no-overwrite). Conditional browser loading and the functional-only skip are explicit, and the report-only limits are unambiguous. However, several passages are dense and convoluted (e.g., 'Load the shared preparation gate now... Defer charters, clocks and probes', the accidental-early-Read recovery text, the no-repeat rule referencing an externally defined rule), and ordering instructions like 'exploratory §4 annotations and materialize' depend on content in other files, so an agent must hold cross-file state carefully. Rule 2 references a 'preflight' that is not a named phase, and the 'infer or ask' target fallback is underspecified. Mixed-regression baseline resolution is covered but intricate. Overall executable with defined deliverables, but prose density and cross-references reduce clarity."},"latency_ms":20301,"stop_reason":"end_turn","usage":{"input_tokens":4675,"output_tokens":1215},"cost_usd":0.1075}
+{"config":"qa-workflow","item":"a03","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":3,"reasoning":"The phases are well-ordered and mostly concrete: setup paths, a reusable aside REPL read script with sentinel markers (CONSOLE_ERRORS=, NAV=, TEXT_START/END, GSTACK_STEP_OK), explicit alert tiers with relative thresholds, a 2-consecutive-check confirmation rule, non-overlapping cadence math, read-only posture, and a baseline-never-overwritten isolation rule. The AskUserQuestion templates are fully specified. However, several decision points and outputs are undefined or ambiguous: `--quick` is listed as an argument but never described in Instructions; Phase 6 Health Report is a single sentence with no format, no output file path, and no coverage/evidence table despite 'Screenshots are evidence' being a rule; MEDIUM/LOW alerts have no stated handling path (only CRITICAL/HIGH trigger the prompt); `<page-name>` derivation from URL paths and how to obtain `branch` are unspecified; `pwd` inside the aside REPL is an unexplained identifier; the `[error` line sentinel has no defined source; the text-snapshot 'content disappears' comparison has no threshold; and the case where a baseline exists but its page set differs from the discovered/requested pages is unhandled. Phase 2 also cross-references Phase 3's script, which is workable but forces the agent to assemble the combined probe itself. Relative to the stated evaluation lens (authority, native functional surfaces, mixed surfaces, conditional browser/DX loading, regression-before-repair), the document is a single browser-only monitoring skill with no native contracts, no conditional loading sections, and no regression/repair phase, so it covers only a slice of the described workflow. An agent can execute the capture and monitoring loop reliably but will have to improvise the final deliverable and several edge cases."},"latency_ms":23895,"stop_reason":"end_turn","usage":{"input_tokens":4250,"output_tokens":1627},"cost_usd":0.12385}
+{"config":"qa-workflow","item":"p02","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The Phases 7-11 fragment is well-structured for its own scope: tiers, triage rules, the per-issue fix loop (locate → minimal fix → atomic commit → re-test → classify), the WTF-likelihood self-regulation heuristic with a hard cap, report outputs with path templates, and TODOS.md updates are all concrete and mostly executable. However, several defects hurt clarity: the clean-tree check is placed before Phase 7 yet says 'then continue with setup' (it belongs before Phase 1, and running it after a baseline report already exists is incoherent); the learnings-refresh is nested under Triage but is phrased per-issue ('the bug you're about to fix'), leaving cadence ambiguous; 8e.5 says 'Generate unit tests' while the decision table calls for integration/component tests; 'Touching unrelated files: +20%' contradicts the hard rule to only touch related files; and the regression test is written after the fix commit, which conflicts with the stated workflow expectation of adding a regression before repair. Completeness is the weakest dimension relative to the stated evaluation scope: the bundle is browser-only ($B goto/screenshot/console) with no native functional contract, no mixed-surface handling, no authority/isolation guidance, and no blocked-path rules; it depends on undefined externals ($B, $REPORT_DIR, 'detected test command', 'user declined bootstrap', 'standard report template', health-score computation, rules 1-10) that are referenced but not loaded or stated as conditions; 'Re-run QA on all affected pages' and 'every 5 fixes' for a loop that also needs per-revert evaluation are underspecified. Actionability is moderate: an agent with the Phase 1-6 context could execute most steps and produce the report/commits, but would have to guess at native re-test procedure, health-score recomputation, and several environmental assumptions."},"latency_ms":19613,"stop_reason":"end_turn","usage":{"input_tokens":4137,"output_tokens":1368},"cost_usd":0.10977}
+{"config":"qa-workflow","item":"p03","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is well-ordered and mostly decision-complete: parameter resolution with explicit conflict handling (--quick vs --regression), a dirty-tree gate with enumerated choices, surface selection and isolation rules with blocking conditions, report-dir collision rules, a browser readiness probe with a concrete sentinel and blocked-path fallback, charters, budgets, a checkpointed probe loop with an exact four-field schema and a two-failure confirmation rule, severity-ordered triage, a detailed fix loop (red regression test → minimal fix → recheck → commit → classify → stop rules), a numeric health-score formula, and a specified report with coverage limits. Gaps that keep it from 5: 'Quick exploration' and '--regression' are defined only for the browser surface (no functional equivalent for Quick depth or baseline comparison); Diff-aware appears in browser techniques but not in the Mode table, leaving its trigger implicit; evidence path uses 'REPORT_DIR/.../evidence/NNN/' and 'surface's probe directory' without naming the single-surface location; ISSUE-NNN assignment and the report {target} slug are undefined; and the document's 'this file is the whole workflow' framing conflicts slightly with the stated bundle/conditional-loading model. These are resolvable by reasonable inference, so an agent can execute and produce the deliverables, but a few choices would be made without explicit guidance."},"latency_ms":23044,"stop_reason":"end_turn","usage":{"input_tokens":4113,"output_tokens":1603},"cost_usd":0.12128}
+{"config":"qa-workflow","item":"a03","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":3,"reasoning":"The phases are numbered, the aside repl scripts are concrete and copy-ready, the manifest schema, alert severities, change-vs-absolute rule, 2-consecutive-check confirmation, and round cadence (max(0,60-elapsed), no overlap, stop at deadline) are explicit, and the baseline mode has a clear STOP point. Weaknesses undermine completeness and execution: the `--quick` single-pass mode is listed in Arguments but never specified in any phase; Phase 6 'Health Report' is a single sentence with no format, output path, coverage summary, or evidence/alert listing, so the main deliverable is undefined; despite being a 'Visual Monitor' there is no screenshot/visual-diff comparison rule and no threshold for text-snapshot 'content disappears'; `<page-name>` derivation from a path is never defined (only '/'→home is implied); 'top 5' links has no ranking criterion; baseline mode captures only `--pages` or the homepage while monitoring auto-discovers pages, leaving undefined behavior when discovered pages have no baseline entry; MEDIUM/LOW alert handling and the 'Rollback' option's actual actions are unspecified (and conflict with the Read-only rule); and the broken-link/monitoring-round instructions are misplaced in Phase 2 with a forward reference to Phase 3. An agent can run baseline capture and the monitoring loop reliably, but will have to improvise the final report, quick mode, and several edge cases."},"latency_ms":28501,"stop_reason":"end_turn","usage":{"input_tokens":3889,"output_tokens":1968},"cost_usd":0.13729}
+{"config":"qa-workflow","item":"p04","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The core structure (parameters, tiers, triage, fix loop 8a→8f, final QA, report, TODOS) is ordered and the STOP/blocked-path handling for missing sections is explicit and consistent. Concrete details like regression file naming, attribution comment, commit format, classification states, and self-regulation caps are executable. However, clarity suffers from dense, jargon-heavy phrasing ('caller-owned authority', 'authorized codification', 'explicitly approved browser bootstrap', '/qa-only') that is never defined, overlapping `--quick` flags across Tier and Mode, and layered references to Phases 1-6 of qa-patterns.md and 'Phase 5 is the before' that only make sense if other files are present. Completeness is undermined because the entire baseline (Phases 1-6), probe loop, health-score computation, and browser verification are delegated to external sections not in this bundle, `--regression` mode is never described in any phase, and report filename placeholders ({user}, {datetime}) are unspecified. Actionability depends heavily on gstack binaries and the external sections existing; when present an agent could produce the deliverables, but the undefined terms and externalized phases leave meaningful decision points requiring interpretation."},"latency_ms":18909,"stop_reason":"end_turn","usage":{"input_tokens":4237,"output_tokens":1353},"cost_usd":0.11002}
+{"config":"qa-workflow","item":"a06","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow has a well-defined phase order (parameters → optional context → surface selection via the exploratory STOP → artifact/report ownership → conditional browser setup → charters-then-probes → finalization STOP → dual report writes), with strong, repeated handling of authority (report-only, no edits/commits/bootstrap), isolation (synthetic identities, owned fixtures/dirs, no state helpers), blocked paths (missing sections, baselines, state dir, output preservation), mixed-run ordering/clocks, and deterministic naming for REPORT_DIR/REPORT_FILE/project copy. Decision points such as conflicting mode flags, feature-branch diff scoping, empty-vs-nonempty output dir, collision handling, and accidental early reporting Read are explicit. Weaknesses: several sentences are dense and self-referential (e.g., the 'shared preparation gate' paragraph, 'exploratory §4 annotations and materialize' referencing unseen content), core definitions of Full/Quick/Regression, probe clocks, stop conditions, and scoring are deferred entirely to external section files not shown, so an agent's execution quality depends on those; the top-level 'Run the Selected Checks' phase is thin on its own; and native command/test-infrastructure preflight is only lightly specified. Overall executable and consistent, but with notable prose density and dependency on companion files."},"latency_ms":22473,"stop_reason":"end_turn","usage":{"input_tokens":4314,"output_tokens":1539},"cost_usd":0.12009}
+{"config":"qa-workflow","item":"a01","split":"dev","phase":"old-b","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The document covers only the Fix→Verify tail (Phases 7-10; Phase 11 is promised in the title but absent) of a larger bundle, so the surface-selection, exploration/probe, native-functional contracts, isolation/authority, conditional browser/DX loading and blocked-path material the evaluation asks about is entirely missing — it is browser-only and never defines how surfaces are chosen. Within its scope the fix loop is reasonably clear and procedural (tier triage, locate→minimal fix→commit→re-test→classify→regression test), with good concrete commit formats, attribution comments, test-type decision table, and report fields. However, it leans on many undefined externals: $B, $REPORT_DIR, 'detected test command', 'user declined bootstrap', health-score computation, the 'standard report template', and the repeatedly invoked 'WTF-likelihood heuristic'. Ordering is also questionable: re-test occurs after the commit rather than before, and the regression test is written after repair (contrary to the stated 'regression before repair' expectation), while 'reverted' handling assumes HEAD is still the fix commit. The dirty-tree gate, stash-pop timing, and the 'B' stash path's cleanup are unaddressed later. An agent could execute the fix/commit/report mechanics but would stall on missing phases, undefined tooling, and the absence of any native or mixed-surface guidance."},"latency_ms":16959,"stop_reason":"end_turn","usage":{"input_tokens":3436,"output_tokens":1093},"cost_usd":0.08901}
+{"config":"qa-workflow","item":"p01","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser baseline section (Phases 1-6) is reasonably well specified: concrete $B commands, per-page checklist, evidence tiers, health-score rubric, and baseline.json schema. However, the bundle covers only the browser surface. The stated workflow also requires selecting native functional or mixed surfaces, bounded probes, diagnosis, adding a regression before repair, recheck, and authority/isolation/native-contract/blocked-path handling—none of which appear. 'Tiers determine which issues get fixed' references a repair phase that does not exist here, and '/qa-only' is mentioned but never defined. Referenced files (qa-report-template.md, issue-taxonomy.md, setup) are not included, so report format and taxonomy are unknown. Internal inconsistencies reduce clarity: Rule 5 'Never read source code' contradicts diff-aware mode's instruction to map controllers/views/models to routes; 'Quick' is both a tier (severity filter) and a mode (smoke test); 'Full' mode vs 'Standard' tier overlap; the Links category uses 'score -15 per broken link' while other categories use severity-based deductions, and it is unclear whether broken links also count as Functional issues. Output location for baseline.json and screenshot dir creation are implied but not stated. An agent can execute the browser exploration and produce a report with screenshots and a score, but cannot execute the full described workflow (native surfaces, diagnosis, regression-before-fix, recheck) from this bundle."},"latency_ms":17982,"stop_reason":"end_turn","usage":{"input_tokens":4930,"output_tokens":1230},"cost_usd":0.1108}
+{"config":"qa-workflow","item":"p05","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"The document is a single, linear web-app workflow with well-ordered phases, named outputs, explicit defaults, concrete thresholds (severity, scoring formula, stop conditions, fix caps) and clear file/commit conventions, so an agent can follow it with little ambiguity. Weak spots in clarity: the Mode default is undefined when no URL is given and the branch is not a feature branch; `<base>` for diff-aware is never specified; issue-to-category assignment (Functional vs UX vs Accessibility, etc.) has no rules even though scoring depends on it; and Phase 3 never describes how Links, Accessibility or Performance are actually probed, yet Phase 4 scores them and asks which categories were 'tested'. Completeness is the main shortfall relative to the stated evaluation scope: there is no bundle structure, no conditional loading rules, no authority/isolation guidance, no native functional or mixed-surface contracts, and no blocked-path handling—everything is browser-only. Within that narrower scope it also omits running the existing test suite before committing, restoring a stash after option (B), and any mechanism for detecting a lower-than-baseline cause. Actionability is still strong for the web case: commands, artifacts, status taxonomy and report format are concrete enough to produce the deliverables, with the category-assignment and base-branch gaps being the main points where the agent must improvise."},"latency_ms":18372,"stop_reason":"end_turn","usage":{"input_tokens":2971,"output_tokens":1329},"cost_usd":0.09616}
+{"config":"qa-workflow","item":"a06","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":4,"actionability":4,"reasoning":"The phases (parameters → context/learnings → surface selection → artifact prep → conditional browser setup → probes → reporting → dual write) are present in a sensible order, with explicit blocker handling for missing assets, mode conflicts, output-dir collisions, and blocked state copies; naming conventions (REPORT_DIR, REPORT_FILE, {target}, project copy) and report-only authority limits are concrete and executable. However, the prose is dense and often convoluted: the preparation gate is described as 'load the shared section now but defer charters/clocks/probes to a later heading', the 'accidental early Read' recovery and no-repeat exemption are awkwardly phrased, Browser Setup points to a different skill directory (qa/gstack-qa) than the other sections while templates reference an unspecified 'installed QA directory', and references like 'exploratory §4 annotations and materialize' and Full/Quick/Regression semantics are entirely delegated to unseen section files. Clock defaults, charter format, and surface-selection criteria live outside this file, so an agent depends heavily on those sections being present and consistent. Overall executable, but requiring careful reading to avoid misordering preparation versus probing steps."},"latency_ms":21072,"stop_reason":"end_turn","usage":{"input_tokens":4314,"output_tokens":1431},"cost_usd":0.11469}
+{"config":"qa-workflow","item":"a01","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":3,"reasoning":"The fix loop (8a-8e.5) is concretely specified: tier-based triage, minimal-fix discipline, one-commit-per-fix with a fixed message format, before/after screenshot paths, a three-way classification, and a well-structured regression-test procedure with explicit skip conditions, attribution comments, naming scheme, and time/attempt bounds. Report outputs (local path, project-scoped artifact, per-issue fields, summary, PR line) are also well-defined. However, the document has notable gaps for an agent executing it as a standalone file: the title promises Phases 7-11 but Phase 11 is absent; rules are numbered 11-15 implying 1-10 live elsewhere; many referenced constructs are undefined here ($B, $REPORT_DIR, the 'WTF-likelihood heuristic', the 'standard report template', 'detected test command', the test-framework bootstrap decline, the 'top-of-skill learnings pull', and how the health score is computed). Phase 9's 'Re-run QA on all affected pages' gives no procedure or bound. The re-test step assumes a browser surface only ($B goto/screenshot/console) with no native or mixed-surface contract, and the clean-tree check's stash option never says when to pop. The 'continue with setup' handoff is ambiguous. Nothing addresses authority, isolation, conditional browser/DX loading, or blocked paths, which the evaluation scope calls out. Net: solid, executable fix-loop mechanics undermined by dangling references and a missing phase, so an agent could likely produce fix commits and a report but would have to infer or import several decision points from outside this file."},"latency_ms":21042,"stop_reason":"end_turn","usage":{"input_tokens":3797,"output_tokens":1372},"cost_usd":0.10657}
+{"config":"qa-workflow","item":"f02","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"The document is a six-line sketch, not a workflow. It omits every element the bundle is supposed to define: no authority model (who decides scope, when to stop), no isolation (test data, environment, side-effect containment), no native/functional surface selection or contracts, no conditional browser/DX loading rules, no blocked-path handling, no bounded probing, no reproduction/diagnosis steps, no regression-before-repair gate, no recheck criteria, and no evidence/coverage report format. Steps like 'find wherever the app is running', 'see if anything seems off', 'fix the bugs that matter', and 'use good judgment' are unexecutable instructions with no decision criteria or defined outputs. An agent could not produce a predictable or verifiable deliverable from this."},"latency_ms":7319,"stop_reason":"end_turn","usage":{"input_tokens":561,"output_tokens":381},"cost_usd":0.02466}
+{"config":"qa-workflow","item":"p03","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is tightly ordered (parameters → isolation → artifacts → readiness → charters → probe loop → triage → fix loop → final QA → report) with explicit STOP/ask gates, blocked-path handling (aside readiness failure, unknown ownership, sign-in walls), a precise checkpoint schema, confirm-by-replay rule, severity/tier mapping, red-before-green regression requirement, commit format, classification states and stop rules. Authority and isolation are well specified (localhost may forward to prod, /qa is not permission, secrets by env-var name only). Minor ambiguities reduce scores: the Diff-aware mode appears in §2/§6 but has no listed budget and is not a §1 Mode value; the evidence path uses an unresolved `REPORT_DIR/.../evidence/NNN/`; ISSUE-NNN numbering origin is undefined; option B (stash) does not say who pops or when; `pwd` as a JS identifier in the aside script is assumed rather than defined; baseline.json comparison and category scoring (UX, Accessibility, Performance) lack assessment criteria; and the evaluation framing mentions conditionally loaded bundle files while the document declares itself a single whole-workflow file. These are gaps an agent can mostly work around, so the workflow remains executable and the deliverables (checkpoints, evidence dirs, regression tests, commits, report, baseline.json, PR line) are concretely defined."},"latency_ms":25871,"stop_reason":"end_turn","usage":{"input_tokens":4113,"output_tokens":1778},"cost_usd":0.13003}
+{"config":"qa-workflow","item":"f02","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"The document is a six-line informal checklist with no executable decisions. It does not define how to locate the running app, what surfaces to test (browser vs native functional vs mixed), how to bound exploration probes, how to reproduce or diagnose a defect, when a bug 'matters', or what the report must contain. Every step relies on 'good judgment' rather than criteria. The evaluation brief explicitly asks about authority, isolation, native contracts, conditional browser/DX loading, blocked paths, and adding a regression test before repair — none of these exist in the document. There are no stated conditions for loading separate files, no outputs or evidence/coverage format, no verification method beyond 'make sure things look better'. An agent could not reliably produce consistent deliverables from this."},"latency_ms":6303,"stop_reason":"end_turn","usage":{"input_tokens":922,"output_tokens":279},"cost_usd":0.02317}
+{"config":"qa-workflow","item":"f04","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":2,"completeness":2,"actionability":2,"reasoning":"The Aside repl scripts, screenshot copy steps, and AskUserQuestion menus are concrete, but the workflow contradicts itself and leaves key artifacts undefined. Alerting policy is directly contradictory: 'Alert the user immediately on the first error you see' vs. 'A first occurrence is pending, not yet an alert' and the rule 'Only alert on patterns that persist across 2+ consecutive checks.' Duration handling conflicts ('10 minutes'/'1m to 30m' in Arguments vs. 'Monitor for as long as seems necessary' in Phase 1), and the check interval is never specified. Phase 2 says 'Save what you collected somewhere you can find it later' with no filename or schema, yet Phase 4 asks for 'the same manifest schema as Phase 2' and Phases 4/7 depend on a `baseline.json` whose location and structure are never defined. `--quick` mode is listed but never described in any phase. Phase 6 ('write up a report') has no format, required fields, or output path, so the primary deliverable is unspecified. Minor issues: 'top 5' links by unstated criterion, 'noticeably slower' vs. the 2x/1.5x rule, alert template omits LOW, Phase 2 forward-references Phase 3's link check, and `pwd` inside the JS repl is unexplained. An agent would make many unguided judgment calls and could not reliably produce consistent baselines or reports."},"latency_ms":20514,"stop_reason":"end_turn","usage":{"input_tokens":3721,"output_tokens":1520},"cost_usd":0.11321}
+{"config":"qa-workflow","item":"p05","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"The document is a tightly written, linear browser-only workflow: each phase has a named gating output, concrete defaults, numeric thresholds (viewport, page caps, scoring tables, stop conditions after 2 reverts/10 fixes/30 total), and an explicit test-first fix loop with commit format and status classification. Decision points such as conflicting flags, dirty working tree, and sign-in are handled with explicit ask-the-user branches. Ambiguities remain: the default mode when there is no URL and no feature branch is undefined; `<base>` for the diff is never specified; issues must be assigned to one of eight scoring categories but no rule says how to categorize or what makes a category 'tested'; Links, Accessibility, Performance and Content categories are scored but Phase 3 never describes probes for them; 'primary control' is undefined; Regression mode's interaction with the fix loop and the stash option's interaction with per-issue commits are unaddressed; and the 'edit only responsible files' rule is in tension with the 'stop after editing an unrelated file' trigger. Against the broader bundle scope the evaluator was asked to consider (native functional/mixed surfaces, authority, isolation, native contracts, conditional browser/DX loading, blocked paths), the document covers none of it—it is a single-file web-only flow with no conditional section loading, no authority/escalation model, and no isolation guidance beyond a git-status check—so completeness is middling even though the deliverables it does define (report.md, baseline.json, screenshots/, commits, PR line) are concrete enough for an agent to produce them."},"latency_ms":17743,"stop_reason":"end_turn","usage":{"input_tokens":2971,"output_tokens":1275},"cost_usd":0.09346}
+{"config":"qa-workflow","item":"p01","split":"dev","phase":"new","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The six browser phases that are present are concretely written with explicit $B commands, a health-score rubric, baseline.json schema, and firm evidence rules, so an agent can follow the orient/explore/document/wrap-up loop. However the bundle is self-labeled 'Phases 1-6' and omits everything the stated workflow requires beyond browser baselining: no native functional or mixed-surface selection, no bounded-probe limits, no reproduce/diagnose phase, no 'add regression before repair' step, no recheck after fix, and no fix phase at all even though the Tier table says tiers 'determine which issues get fixed.' Authority/isolation rules, native contracts, conditional DX loading and blocked-path handling are entirely absent. Internal inconsistencies hurt clarity: Rule 5 'Never read source code' conflicts with diff-aware mode's instruction to map changed controllers/models to routes; 'Quick' is defined both as a fix-severity tier and as a 30-second smoke mode; `/qa-only` is referenced but undefined; `<SKILL_DIR>` and `$REPORT_DIR` are never bound; 'Create output directories' is unspecified; the port-detection one-liner's `&&`/`||` chain will print spurious matches; and the referenced template and taxonomy files are not included. The result is an actionable browser smoke/baseline recipe but an incomplete, partially contradictory specification of the full QA workflow the agent is expected to execute."},"latency_ms":23475,"stop_reason":"end_turn","usage":{"input_tokens":5291,"output_tokens":1653},"cost_usd":0.13556}
+{"config":"qa-workflow","item":"a04","split":"dev","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":4,"reasoning":"The bundle gives a well-ordered functional path: explicit load order and authority limits, a deadline guard (G) and evidence recorder (Q) with exact command syntax, a strict probe loop (success-first, status check, publish-before-dispatch checkpoint with a four-field schema, exact replay before repair, revalidation), a thorough contract map with enumerated outcome states, and a concrete final-report mechanism (annotations.json schema, materialize, pass/blocked rules). Blocked paths (missing assets, missing/wrong regression baseline, expired/missing D, required failure blocking /ship) are stated as hard rules. However, clarity suffers from extreme density and heavy single-letter abbreviations, cryptic sentences ('QA_DEADLINE receipts are not observations', 'Expiry leaves confirmation/minimization incomplete'), and undefined caller-side terms (Phase 8 gates, Fix-First, test_stub/ASK approval, 'DX' surfaces never appear). Budget resolution for mixed runs where functional is unbounded but browser is bounded is ambiguous ('an unlimited mode uses the caller's bound' when no caller bound exists). Completeness gaps: system-functional.md invokes a 'shared isolation/permission preflight' that exploratory.md never defines (charter only lists 'isolation' as a field); scope.md, qa-patterns.md and the report templates are referenced but absent, so surface selection, browser scoring and template structure cannot be verified from this bundle. An agent could still execute the functional workflow and produce the stated deliverables, but would have to interpret several compressed rules."},"latency_ms":30705,"stop_reason":"end_turn","usage":{"input_tokens":4381,"output_tokens":2157},"cost_usd":0.15166}
+{"config":"qa-workflow","item":"f05","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":2,"completeness":3,"actionability":2,"reasoning":"The bundle is otherwise carefully structured (parameter parsing, test-plan/learnings lookups with explicit no-write constraints, surface selection with per-method mode/baseline rules, REPORT_DIR/REPORT_FILE ownership and collision handling, conditional browser-setup loading with blocked-path fallbacks, and black-box vs read-only discovery boundaries). However, 'Run the Selected Checks' contains a direct, unqualified instruction to fix confirmed critical/high bugs in source and commit them so the report shows them resolved, which flatly contradicts the title, the opening 'NEVER fix anything' mandate, and Additional Rule 1 ('Never commit, stash or bootstrap'). This unresolved conflict sits in the core execution phase and leaves the agent with no deterministic decision on the single most important authority boundary, undermining clarity and actionability despite the solid scaffolding. Minor gaps: the reporting phase exists only as an index row with no in-body trigger after probing stops, and 'Run the Selected Checks' ends without a transition to finalization."},"latency_ms":14281,"stop_reason":"end_turn","usage":{"input_tokens":3247,"output_tokens":787},"cost_usd":0.07182}
+{"config":"qa-workflow","item":"a02","split":"dev","phase":"new","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser section itself is concrete: modes are differentiated (Full/Quick/Regression/Diff-aware), scripts are copy-ready with explicit sentinels (GSTACK_STEP_OK, DIFF_START/END), evidence-copy steps are spelled out, and safety rules (no credentials, HEAD only locally, one mutation prompt before acting) are unambiguous. However, the document is dense with jargon and leans heavily on external authority it never defines here: the 'shared exploratory loop' and its checkpoint/exact-replay/deadline-guard rules, BROWSER SETUP's access decision and `$B` states, `$REPORT_DIR`, the Health Score Rubric, issue-taxonomy.md, and a 'Phase 6 report' that does not exist in this file (phases stop at 5). Native functional surfaces, isolation, surface-selection logic, diagnosis, the regression-before-repair step, recheck, and the coverage/evidence report are all absent, so the bundle as presented cannot be executed end-to-end. There are also small internal tensions: Diff-aware includes an API fetch probe while the rules say API targets do not select the browser surface; Quick's '30 seconds' cap sits awkwardly against per-probe checkpoint requirements; 'Resolve conflicting depth flags by asking' and 'ask if none' are not tied to a specific blocked-path output. An agent could run the browser probes and produce screenshots/issue artifacts, but would have to guess at report structure, scoring, and how this slots into the native/mixed workflow."},"latency_ms":16846,"stop_reason":"end_turn","usage":{"input_tokens":4427,"output_tokens":1185},"cost_usd":0.10352}
+{"config":"qa-workflow","item":"a02","split":"dev","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser section itself is concrete: modes are distinguished, Phases 1-5 have copy-paste aside scripts with clear placeholders, sentinel outputs (GSTACK_STEP_OK, ASIDE_DIR, CONSOLE_ERRORS) and explicit evidence-copy rules. However the prose is extremely terse and relies on undefined shorthand ('Diff-aware selects scope, not another pass', 'LOOK is not ACT', 'clamp the shared loop's deadline guard'), and it defers critical decisions to files not present here: the shared exploratory loop's checkpoint/exact-replay rules, BROWSER SETUP's access decision and `$B` states, the Health Score Rubric, issue-taxonomy.md, $REPORT_DIR ownership, and a 'Phase 6 report' that is referenced but never defined. As a 'complete workflow' it omits native/mixed surface selection, reproduce/diagnose, regression-before-repair, recheck, and report/coverage output formats entirely, so an agent could execute the probes but could not produce the final deliverables or resolve blocked/authority paths from this document alone."},"latency_ms":16017,"stop_reason":"end_turn","usage":{"input_tokens":4066,"output_tokens":1046},"cost_usd":0.09296}
+{"config":"qa-workflow","item":"f04","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":2,"completeness":2,"actionability":2,"reasoning":"The document has usable scaffolding (argument list, concrete aside repl scripts with sentinel outputs, copy-out steps, AskUserQuestion templates, a clear alert block) but is undermined by direct contradictions and gaps. Alert timing is self-contradictory: Phase 5 says 'Alert the user immediately on the first error you see' while the same phase and the Important Rules say a first occurrence is 'pending, not yet an alert' and only 2+ consecutive checks trigger alerts. Phase 1 says 'Monitor for as long as seems necessary' despite --duration/10-minute defaults, and no check interval is ever defined. Phase 2 tells the agent to 'save what you collected somewhere you can find it later' with no schema or filename, yet Phase 4 and Phase 7 reference 'baseline.json' and 'the same manifest schema as Phase 2'. Phase 6 ('write up a report') has no template, fields, or output path despite the skill's purpose being an evidence report. --quick mode is never described in any phase, 'top 5' link ranking is undefined, performance thresholds are both 'noticeably slower' and '2x/1.5x', and the alert template omits LOW. The bundle also doesn't match the stated evaluation frame (native/mixed surfaces, regression-before-repair, conditional loading) at all, so those expected decision paths are simply absent. An agent could stumble through baseline capture and checks but would have to invent the monitoring cadence, alert policy, manifest format, and final deliverable."},"latency_ms":17687,"stop_reason":"end_turn","usage":{"input_tokens":3721,"output_tokens":1307},"cost_usd":0.10256}
+{"config":"qa-workflow","item":"a01","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The fix-loop phases (7–10) are written in a clear sequential order with concrete commands, commit message formats, classification states (verified/best-effort/reverted/deferred), skip conditions for the regression test, and a well-specified report structure. However, the document is only a fragment: the header promises Phases 7–11 but Phase 11 is absent; 'Additional Rules' start at 11, implying rules 1–10 live elsewhere; and it depends on undefined externals ($B, $REPORT_DIR, 'health score' computation, the 'WTF-likelihood heuristic', 'standard report template', test-framework detection and the 'user declined bootstrap' decision, learnings pull at 'top-of-skill'). Relative to the stated evaluation scope—surface selection (browser vs native functional vs mixed), bounded probes, authority/isolation, native contracts, conditional browser/DX loading, and blocked paths—none of these are covered; the workflow is browser-only and assumes Phases 1–6 already discovered issues. Decision points like 'if WORSE than baseline: WARN' lack a defined follow-up, and the stash-pop step after choice B is never scheduled. An agent could execute the fix/commit/re-test/report loop given the upstream context, but cannot select surfaces or explore from this bundle alone."},"latency_ms":13665,"stop_reason":"end_turn","usage":{"input_tokens":3436,"output_tokens":925},"cost_usd":0.08061}
+{"config":"qa-workflow","item":"f02","split":"dev","phase":"new","sample":0,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"The document is a six-line informal checklist that lacks nearly everything the described workflow requires. It never defines how to discover the running app (ports, dev-server startup, environment), has no surface-selection logic (browser vs native functional vs mixed), no bounded exploration probes, no reproduction/diagnosis procedure, no 'add regression test before repair' step, no recheck criteria, no evidence/coverage schema for the report, and no authority, isolation, native-contract, conditional-loading, or blocked-path rules. Decision points like 'bugs that matter', 'seems off', 'things look better', and 'don't overdo it' are entirely subjective with no thresholds or stopping conditions. An agent cannot execute this deterministically or produce defined deliverables; it is a placeholder, not a workflow."},"latency_ms":6548,"stop_reason":"end_turn","usage":{"input_tokens":922,"output_tokens":296},"cost_usd":0.02402}
+{"config":"qa-workflow","item":"f01","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":2,"reasoning":"The document is internally contradictory and structurally broken. Phase 7 first says tiers determine which issues get fixed, then immediately says 'Fix every issue in the report, including low severity, whatever tier was selected' with a vague 'if time is short' fallback — an agent cannot resolve which rule governs. Phase 8 has two steps both labeled '8a', and the first ('Commit') is sequenced before 'Locate source' and 'Fix', so the agent is told to commit a fix that doesn't exist yet. Phase 9 (the 'Verify' half of the title — re-test, recheck behavior, recompute health score) is entirely absent; Rule 13 references 'Phase 8e.5' for regression-test generation, which does not exist, so the 'add a regression before repair' contract has no executable home. Phase 10 'Report' is a single sentence with no required fields (evidence, coverage, before/after health score, deferred list). The 'WTF-likelihood heuristic' and rules 1-10 are referenced but never defined. The clean-tree check, learnings-refresh keyword rules, one-commit-per-fix rule, and TODOS.md update are clear and executable, which keeps this from being a 1 on clarity/actionability, but the missing verification phase, nonexistent regression step, and self-contradicting triage make the workflow non-executable as a whole."},"latency_ms":14670,"stop_reason":"end_turn","usage":{"input_tokens":1946,"output_tokens":995},"cost_usd":0.06921}
+{"config":"qa-workflow","item":"f01","split":"dev","phase":"old-b","sample":1,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":2,"reasoning":"The document is riddled with internal contradictions and structural breaks. Phase 7 states tiers determine fix scope, then immediately says 'Fix every issue... whatever tier was selected. If time is short, fix only the critical ones' — three mutually exclusive instructions. Phase 8 has two steps labeled '8a', with 'Commit' placed before 'Locate source' and 'Fix', so the agent is told to commit before any change exists. Phase 9 (verification/recheck) is absent entirely, yet Rule 14 ('revert on regression') depends on it; Rule 13 references 'Phase 8e.5' regression-test generation, which does not exist, and the stated requirement to add a regression before repair is never operationalized. Rules are numbered 11-15 with no 1-10, and the 'WTF-likelihood heuristic' is invoked but undefined. Phase 10 'Write up what you did' specifies no report contents, no post-fix health score, no evidence/coverage format. The stash option promises to 'pop the stash after' but no step does so, and 'continue with setup' points nowhere. Nothing addresses browser vs native surface selection, isolation, native contracts, or conditional loading. The learnings-refresh guidance and TODOS.md update are the only well-specified, executable pieces; the rest cannot be followed to produce the expected deliverables."},"latency_ms":15724,"stop_reason":"end_turn","usage":{"input_tokens":1946,"output_tokens":1055},"cost_usd":0.07221}
+{"config":"qa-workflow","item":"f03","split":"dev","phase":"old-a","sample":0,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":2,"reasoning":"The document has a recognizable phase skeleton and some concrete rules (evidence tiers, screenshot-read rule, redaction), but it is riddled with truncated or empty sections: 'Find the browse binary' gives no command, Phase 2's 'If the user specified auth credentials' and 'If the user provided a cookie file' have no steps, Phase 3 'Get a map of the application' and Phase 4 'At each page' list nothing, and the Responsiveness item ends in a colon. 'Diff-aware mode' and '--regression' are referenced but the Modes section is a single vague sentence defining neither. Tiers say which issues 'get fixed' yet no fix, regression-before-repair, recheck, or coverage-report phase exists; Phase 6 is 'Summarize what you found' plus an undefined 'baseline health score' tacked on at the end. Nothing addresses native functional or mixed surfaces, native contracts, isolation, authority, conditional DX loading, or blocked paths called for by the bundle description, and Rule 12 actively forbids non-browser alternatives. External templates/taxonomy and the `$B` variable are referenced but undefined. An agent could not reliably authenticate, orient, select surfaces, or produce the expected deliverables from this text alone."},"latency_ms":17542,"stop_reason":"end_turn","usage":{"input_tokens":2472,"output_tokens":1286},"cost_usd":0.08902}
+{"config":"qa-workflow","item":"f08","split":"heldout","phase":"new","sample":0,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is a well-written philosophy essay, not a workflow. It contains no phases, no surface-selection logic (browser vs native vs mixed), no bounded probe limits, no reproduce/diagnose steps, no regression-before-repair rule, no recheck procedure, no report format, and no coverage/evidence schema. None of the expected structural elements are present: authority/precedence rules, isolation requirements, native functional contracts, conditional loading triggers for browser/DX sections, or blocked-path handling. An agent reading this would understand the intended attitude ('evidence matters', 'fix with humility') but would have to invent every operational decision, making it impossible to produce the stated deliverables consistently. Prose clarity is high, but as an executable workflow it is ambiguous by omission on every decision point; completeness and actionability are effectively nil."},"latency_ms":8676,"stop_reason":"end_turn","usage":{"input_tokens":1292,"output_tokens":492},"cost_usd":0.03752}
+{"config":"qa-workflow","item":"f06","split":"heldout","phase":"new","sample":0,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"The document is a thin shell of pointers to artifacts that are not included or defined: 'QA Handbook section 2.3', 'team wiki', 'QA-PROTOCOL-v2', 'release matrix', 'usual tracker format', 'house severity scale', 'release policy', 'fix-forward policy', 'deferral process', 'usual place', 'QA owner', 'standard evidence bundle', 'normal channel'. An agent cannot resolve any of these, so no phase is executable. None of the required workflow concerns appear: no surface selection (browser vs native vs mixed), no bounded exploration probes, no repro/diagnosis procedure, no regression-before-repair rule, no recheck step, no evidence/coverage report format, no authority or isolation rules, no conditional loading of browser/DX modules, no blocked paths. The protocol step split (P1–P9 web, P10–P14 services) references steps that are never stated. The precedence rule ('handbook wins, except where the protocol is newer') is circular and gives no way to determine recency. Outputs are entirely undefined. This reads as an internal memo for humans with tribal knowledge, not an agent-executable workflow."},"latency_ms":9662,"stop_reason":"end_turn","usage":{"input_tokens":1137,"output_tokens":622},"cost_usd":0.04247}
+{"config":"qa-workflow","item":"a08","split":"heldout","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser-testing portions are concrete: the setup script, mode selection (diff-aware vs full/quick/regression), per-page explore checklist, two-tier evidence capture, health score rubric, and the Important Rules are all specific and mostly executable. However, the document has serious gaps relative to the stated workflow. It is titled 'Phases 1-6' but jumps from Phase 1 directly to Phase 3 with no Phase 2 defined. The task frames a workflow covering browser, native functional or mixed surfaces, bounded probes, reproduce/diagnose, add-regression-before-repair, recheck, and report — yet this bundle is browser-only: there is no native/functional-test contract, no diagnose or repair phase (despite 'Tiers determine which issues get fixed'), no regression-test-before-fix step, no recheck step, no authority/isolation rules, and no conditional loading or blocked-path guidance. Several variables are undefined ($REPORT_DIR, <SKILL_DIR>), the baseline.json schema and regression diff logic are unspecified, and the report template and issue taxonomy live in unreferenced external files. Rule 5 ('Never read source code') directly conflicts with diff-aware mode, which requires reading changed controllers/views to map routes. Phase 6 is a single sentence. An agent could run the browser exploration and produce a scored report, but could not execute the full stated QA workflow without substantial improvisation."},"latency_ms":14559,"stop_reason":"end_turn","usage":{"input_tokens":4732,"output_tokens":962},"cost_usd":0.09542}
+{"config":"qa-workflow","item":"p08","split":"heldout","phase":"old-a","sample":1,"status":"ok","value":{"clarity":4,"completeness":2,"actionability":3,"reasoning":"The functional-surface file itself is crisp: target/contract/tool definitions with IDs, explicit isolation and stop-and-ask rules, a contract-type-to-probe table, bounded probe execution (60s timeout, evidence/NNN/, one re-run to confirm), well-defined outcome states (pass/fail/blocked/not run/inconclusive), and a precise report structure with a strict verdict rule. However, as the complete workflow it is largely absent: there is no surface-selection logic (what makes a target browser vs functional vs mixed), no browser or DX section, no conditional loading triggers, and no authority statement for conflicts between files. The stated lifecycle also stops short — it proposes regression tests in the report but never specifies adding the regression before repair, performing the repair, or rechecking behavior afterward; diagnosis is only implicit in the re-run step. Mixed surfaces (e.g., a CLI that triggers a web flow) are unaddressed, and 'required contract' is used in the verdict without defining how a contract is marked required. An agent could execute the functional path and produce report.md and evidence reliably, but could not execute the full select/diagnose/regress/repair/recheck workflow from this alone."},"latency_ms":15945,"stop_reason":"end_turn","usage":{"input_tokens":1779,"output_tokens":982},"cost_usd":0.06689}
+{"config":"qa-workflow","item":"p08","split":"heldout","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"The functional-surface file is well structured: four numbered steps, explicit outputs (plan post, probe list, evidence/NNN/, report.md), a clear contract-type-to-probe table, defined outcome vocabulary (pass/fail/blocked/not run/inconclusive), a concrete reproduction rule (re-run once, two failures = confirmed), isolation and permission-stop rules, and a crisp verdict rule. Minor ambiguities remain: 'required contract' is never defined versus optional ones, probe bounds (how many probes, exploration budget) are not stated, and the state-reset mechanism for concurrency/retry probes is left to the agent. Completeness is the weaker dimension relative to the full workflow described: this file only covers select-target → probe → report. It has no authority/precedence section, no mixed-surface or browser hand-off conditions beyond the opening sentence, no diagnosis phase, no 'add regression before repair' step (it only proposes regression tests), and no recheck/re-verify phase after a fix. Blocked handling is defined, but DX/conditional loading of other bundle files is not referenced. Within its own scope an agent could execute this end to end and produce the stated deliverables, so actionability is high, but the deliverable set stops at report plus proposed tests rather than implemented regressions and rechecked behavior."},"latency_ms":16455,"stop_reason":"end_turn","usage":{"input_tokens":2140,"output_tokens":1017},"cost_usd":0.07225}
+{"config":"qa-workflow","item":"a07","split":"heldout","phase":"new","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The Setup section is reasonably precise: it specifies parameter parsing with defaults, tier/mode semantics (including the --quick/--regression conflict rule), the clean-tree gate with explicit AskUserQuestion options, and detailed REPORT_DIR resolution rules (UTC run subdirectory, no overwrites, caller path precedence, output blocker reporting). The authority rule to read sections/scope.md from the installed skill path and the blocked-path fallback are stated clearly. However, as a complete workflow the bundle is thin: Phases 1-6 are collapsed into a single line ('Explore the target and document issues with evidence') with no definition of surface selection (browser vs native vs mixed), probe bounds, native contracts, isolation/synthetic identity setup, reproduction/diagnosis steps, or the regression-before-repair requirement the task describes. Conditional browser/DX loading is only alluded to ('Browser session handling lives in browser setup') with no stated load condition or file reference. Phases 7-11 are one-liners with no defined outputs: the report format, evidence/coverage structure, commit conventions in the fix loop, re-test criteria for Final QA, and TODOS.md update rules are all absent. The Parameter table also has minor ambiguity (--quick appears under both Tier and Mode; 'diff-scoped' scope and 'select the surface first' are undefined here). An agent could execute setup and artifact preparation reliably, but could not produce the core deliverables (reproduced defects, regression tests, evidence/coverage report) from this bundle alone without the unreferenced section files."},"latency_ms":16682,"stop_reason":"end_turn","usage":{"input_tokens":2017,"output_tokens":1085},"cost_usd":0.07442}
+{"config":"qa-workflow","item":"p06","split":"heldout","phase":"new","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":5,"reasoning":"The document is a tightly scoped, self-contained workflow for native functional targets (workers, jobs, webhooks) with an explicit entry condition and an explicit prohibition on browsers/new harnesses, so it fits cleanly as one conditionally loaded file in the bundle. Phases are ordered and concrete: authority gating (dirty tree stop, output-dir non-overwrite with UTC run folders, regression-mode blocking on missing/wrong-target reports), isolation (owned state only, hard STOP on any possible production/third-party destination, posted plan with per-command timeout), a contract map that forces sourcing from docs/types/tests rather than implementation and separates acceptance/enqueue/processing/retry/final effect, a probe loop with reset-record-run-compare-log discipline, five explicit outcome labels, replay-twice confirmation and input minimization, severity rubric tied to fix tiers, a one-defect-at-a-time repair loop with red-test-first, smallest fix, three-part recheck with durable-state inspection, commit format, verified/best-effort/reverted classification, hard stop triggers (two reverts, unrelated file edit, 20 fixes), a final regression run, and a fully specified report with a fixed summary line. Gaps that cost points: 'highest-risk changed edge' in quick mode is undefined (no criterion for choosing it); regression mode says to read the prior report but never states how it alters the probe loop (replay only prior failures? full map plus prior?); evidence folder numbering (NNN) and exploration.md format are implied rather than specified; cleanup of owned state is only mentioned in the report, with no explicit step performing it; and there is no in-file guidance for what to do if the target turns out to be mixed or browser-facing beyond 'do not open a browser' (routing is presumably in another bundle file, which is acceptable but leaves this file silent on the hand-off). Despite these, an agent could execute every step and produce the evidence tree, exploration log, regression tests, commits and report without inventing structure."},"latency_ms":23839,"stop_reason":"end_turn","usage":{"input_tokens":3253,"output_tokens":1506},"cost_usd":0.10783}
+{"config":"qa-workflow","item":"p07","split":"heldout","phase":"new","sample":1,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":4,"reasoning":"The browser methodology is dense but mostly well-structured: six phases, three modes with explicit time caps, concrete copy-paste aside repl scripts with sentinel output (GSTACK_STEP_OK, DIFF/TEXT markers), a fully computable Health Score Rubric with weights, dedup rules, severity deductions and a baseline.json schema. Framework-specific heuristics and safety rules (one AskUserQuestion for non-LOCAL mutations, no credentials, same-origin HEAD only locally) are concrete. Clarity suffers from heavy outbound dependencies that are referenced but not defined here—the 'shared exploratory loop' and its checkpoint/exact-replay/deadline-guard rules, BROWSER SETUP's 'Browser access decision', `$B handoff/resume`, caller 'smoke and plan bounds', 'mixed-report rules', `$REPORT_DIR`, and TODOS.md—plus terse aphorisms ('LOOK is not ACT', 'Diff-aware selects scope, not another pass', 'Aside READY, otherwise $B') that an agent must interpret. Several script primitives (`pwd` inside the repl, `annotatedScreenshot`, `_sendToTarget`, snapshot ref ids like `e12`) are assumed rather than explained. Completeness is partial relative to the requested evaluation scope: authority, isolation, native functional contracts, mixed surfaces, conditional loading and blocked paths are only gestured at (e.g., 'API/CLI/job/worker/webhook targets do not select it') and live in other unshown files, so this document alone cannot drive the full QA workflow; within its own browser remit, decision points (mode selection, missing URL, missing baseline, timeout semantics in Phase 5) are reasonably defined. Actionability is strong where it counts—an agent with the referenced tooling can execute the probes, score, and produce the report and baseline.json—but it depends on the sibling files being present and consistent."},"latency_ms":24221,"stop_reason":"end_turn","usage":{"input_tokens":5407,"output_tokens":1586},"cost_usd":0.13337}
+{"config":"qa-workflow","item":"a08","split":"heldout","phase":"old-a","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The browser-only portions that are present (setup, diff-aware mode, Orient/Explore/Document steps, health rubric) are concrete and readable, but the document is structurally broken as a complete workflow. Phase 2 is absent entirely (phases jump 1→3), and nothing covers the native functional/mixed-surface path, bounded probes, authority/isolation rules, native contracts, blocked paths, or the fix cycle implied by the tiers ('which issues get fixed')—there is no reproduce-diagnose, regression-before-repair, or recheck phase despite tiers being defined by fix scope. Key variables are never defined ($REPORT_DIR, <SKILL_DIR>, timer mechanism), referenced files (qa-report-template.md, issue-taxonomy.md) and the baseline.json schema are not supplied, and '/qa-only' is mentioned but undefined. There is an internal contradiction: Rule 5 forbids reading source code while diff-aware mode requires mapping controller/view/model files to routes. An agent could execute a rough browser smoke/explore pass and score it, but could not reliably produce the full set of expected deliverables or follow a conditional loading/isolation discipline from this bundle."},"latency_ms":16745,"stop_reason":"end_turn","usage":{"input_tokens":4371,"output_tokens":1152},"cost_usd":0.10131}
+{"config":"qa-workflow","item":"f08","split":"heldout","phase":"old-a","sample":1,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is a well-written mindset essay, not a workflow. It contains no phases, decision points, or procedures for the tasks the agent must perform: no surface selection (browser vs native vs mixed), no bounded probe definitions, no reproduction/diagnosis steps, no regression-before-repair rule, no recheck procedure, and no report/evidence/coverage format. None of the bundle concerns are addressed—authority boundaries, isolation, native contracts, conditional browser/DX loading, or blocked paths are entirely absent, as are any references to other files or load conditions. The prose itself is clear and the values (evidence, depth over breadth, small fixes, leaving a trail) are sensible, which earns minimal clarity credit, but an agent reading this alone cannot determine what to do first, when to stop exploring, how to isolate changes, or what deliverables to produce."},"latency_ms":7796,"stop_reason":"end_turn","usage":{"input_tokens":931,"output_tokens":430},"cost_usd":0.03081}
+{"config":"qa-workflow","item":"a08","split":"heldout","phase":"new","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser-testing portions are concretely written: parameter table, binary discovery script, orient/explore/document command sequences, and a numeric health-score rubric are all easy to follow. However the document has notable structural problems. Phase numbering jumps from Phase 1 to Phase 3 (no Phase 2), and the setup says tiers 'determine which issues get fixed' yet there is no reproduce/diagnose, regression-test-before-repair, fix, or recheck phase at all—the workflow ends at scoring. Rule 5 ('Never read source code') directly contradicts diff-aware mode, which requires mapping changed controllers/views/models to routes. Key referenced artifacts (qa-report-template.md, issue-taxonomy.md, baseline.json schema) are not defined here, so severity classification and regression diffing cannot be executed deterministically; $REPORT_DIR is used without being set; the port-detection one-liner's &&/|| chaining is logically fragile; and 'main' is hardcoded as the base branch. Nothing in the bundle addresses native functional/mixed surfaces, isolation, authority, native contracts, conditional DX loading, or blocked paths, which the evaluation frame says the complete workflow should cover. An agent could produce a browser QA report with screenshots and a score, but could not execute the full defect-repair-regression loop or the non-browser surfaces from this document alone."},"latency_ms":17541,"stop_reason":"end_turn","usage":{"input_tokens":4732,"output_tokens":1103},"cost_usd":0.10247}
+{"config":"qa-workflow","item":"p07","split":"heldout","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser section itself is concrete: phased steps, copy-pasteable aside repl scripts with sentinels (GSTACK_STEP_OK, ASIDE_DIR), a fully computable Health Score Rubric, and a defined baseline.json schema. However, the prose is telegraphic and leans heavily on externally defined concepts that are not present in the supplied bundle: the 'shared exploratory loop' and its checkpoint/exact-replay rules, BROWSER SETUP and its Browser access decision, $B handoff/resume, $REPORT_DIR, TODOS.md, issue-taxonomy.md, qa-report-template.md, caller 'smoke and plan bounds', and 'clamp the deadline guard to the caller's running deadline'. Native functional surfaces, mixed-surface selection, authority/isolation rules, DX loading and blocked paths—explicitly in scope for this evaluation—are referenced only by allusion and never defined here, so the complete workflow cannot be judged as fully specified. Some decision points are vague (how to select between surfaces, what 'LOOK is not ACT' operationally permits, port-scan limited to three ports, 'No identifiable pages' handling). An agent could execute the browser probes and produce a scored report if the rest of the bundle resolves these references, but as presented there are notable gaps and undefined variables that would require guesswork."},"latency_ms":17583,"stop_reason":"end_turn","usage":{"input_tokens":5046,"output_tokens":1104},"cost_usd":0.10566}
+{"config":"qa-workflow","item":"p07","split":"heldout","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":4,"reasoning":"The browser-QA section has concrete, copy-pasteable aside scripts, well-defined modes (Full/Quick/Regression/Diff-aware), a six-phase sequence, and a precisely computable Health Score Rubric with weights, deductions, deduplication rules, and rounding, plus a defined baseline.json shape and report contents. However, the prose is highly telegraphic and leans heavily on externally-defined contracts (the shared exploratory loop's checkpoint/exact-replay rules, BROWSER SETUP's access decision and sentinel rules, `$B handoff/resume`, `Aside READY`, caller-owned artifact paths and deadline guards, issue-taxonomy and report-template files) that are not visible here, so several decisions (what a checkpoint is, how Quick's 30-second cap coexists with per-probe checkpoints, what 'Open static pages directly' or 'LOOK is not ACT' operationally mean, the `e12` placeholder locator, categoryScores schema) remain ambiguous for an agent reading this file alone. Native functional/mixed surfaces, regression-test-before-repair, and diagnosis are explicitly delegated elsewhere, so this section is complete only for browser discovery/evidence, not the full workflow described in the evaluation brief. Given correct conditional loading of the referenced files, an agent could execute the probes and produce the report and baseline, but clarity and completeness suffer from the dense cross-referencing and some internal tension (diff-aware 'no URL' trigger vs. 'use the supplied URL')."},"latency_ms":25877,"stop_reason":"end_turn","usage":{"input_tokens":5046,"output_tokens":1678},"cost_usd":0.13436}
+{"config":"qa-workflow","item":"a07","split":"heldout","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The Setup section is fairly precise (parameter table, tier/mode interaction, clean-tree gate with explicit options, REPORT_DIR collision rules, blocked-path handling for missing scope.md). However, the bundle as presented contains only the SKILL.md entry point; `sections/scope.md`, browser setup, native functional contracts, DX loading conditions and isolation rules are referenced but absent, so authority/isolation/native contract decisions cannot be verified or executed. Phases 1-11 are reduced to one-line labels: Phases 1-6 are collapsed into 'explore and document issues' with no probe bounds, surface-selection procedure, reproduction/diagnosis steps, or the 'add regression before repair' requirement; Phases 7-11 give no triage criteria beyond severity tier, no fix-loop commit/re-test contract, no report schema (evidence/coverage fields), and no TODOS.md format. Minor ambiguities also remain (Tier and Mode both consume `--quick`; 'Mode full' vs 'Tier Standard' overlap; 'native contracts' never defined). An agent could complete setup but would have to improvise nearly all of the testing, repair, and reporting deliverables."},"latency_ms":12111,"stop_reason":"end_turn","usage":{"input_tokens":1656,"output_tokens":783},"cost_usd":0.05571}
+{"config":"qa-workflow","item":"p06","split":"heldout","phase":"new","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The document is a tightly scoped, self-contained workflow for native functional targets (workers, jobs, webhooks) with well-ordered phases, concrete tables, explicit stop/ask conditions, evidence layout (evidence/NNN, exploration.md, ISSUE-NNN), outcome vocabulary (pass/fail/blocked/not run/inconclusive), a test-first repair loop with classification states, and a defined report structure and summary line. Authority (dirty-tree check, production/third-party STOP rule), isolation (owned state, plan post with timeouts), and blocked-path handling (missing service never a pass, environment failures reported not fixed, regression mode blocked on missing report) are clearly stated. Weaknesses: the `--quick` flag is used for both Mode and Fix tier without saying whether one flag sets both; 'highest-risk changed edge' is undefined and left to judgment; regression mode says to read the previous report but Step 4 never specifies how the probe loop differs (replay prior evidence vs. full contract map); 'observed once' fails have no explicit triage disposition; cleanup of owned state is only mentioned in the report, not as an executed step; and since this file explicitly declares itself the whole workflow, it does not address surface selection or conditional browser/DX loading referenced in the bundle framing, so an agent must rely on the parent bundle for routing. These gaps are minor relative to an otherwise highly executable workflow."},"latency_ms":20402,"stop_reason":"end_turn","usage":{"input_tokens":3253,"output_tokens":1344},"cost_usd":0.09973}
+{"config":"qa-workflow","item":"p07","split":"heldout","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser section is well-structured with numbered phases, concrete copy-pasteable aside/CDP scripts, explicit sentinel outputs (GSTACK_STEP_OK, DIFF_START/END), a precise Health Score Rubric with dedup/weighting/rounding rules, and a defined baseline.json schema. However, it is extremely terse and leans on undefined external contracts at nearly every decision point: the 'shared exploratory loop' checkpoint/exact-replay/deadline-guard rules, BROWSER SETUP's access decision and `$B` handoff states, `$REPORT_DIR`, `qa/references/issue-taxonomy.md`, and the report template are all referenced but absent. Mode selection has gaps (what applies on main with no URL; 'ask if none' vs auto port-scan; a 30-second Quick cap that must still honor per-probe checkpoints), and phrases like 'replace action/wait for interactions' or the `e12` placeholder are cryptic. Relative to the full workflow described (browser/native/mixed surface selection, isolation, authority, native functional contracts, blocked paths, reproduce→regression-before-repair→recheck), this file covers only browser discovery and scoring; diagnosis, regression authoring, native surfaces and blocked-path handling are explicitly deferred to callers and not present. An agent could execute the browser probes and compute scores if the rest of the bundle exists, but cannot complete the end-to-end workflow from this document alone."},"latency_ms":24529,"stop_reason":"end_turn","usage":{"input_tokens":5407,"output_tokens":1590},"cost_usd":0.13357}
+{"config":"qa-workflow","item":"a08","split":"heldout","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser-side phases (Orient, Explore, Document) are concrete with specific $B commands, a defined health rubric, and clear evidence tiers. However, the document is labeled Phases 1-6 yet Phase 2 is absent entirely; `$REPORT_DIR` is never set despite being used everywhere; Rule 5 ('Never read source code') directly contradicts diff-aware mode's file-to-route analysis; and `/qa-only` is referenced but undefined. Against the stated scope, large pieces are missing: no native/functional (non-browser) surface selection or contracts, no isolation or authority model, no blocked-paths guidance, no conditional loading rules for sub-files (the report template and issue taxonomy are referenced but their load conditions and contents are absent), no baseline.json schema for regression diffing, and—although tiers 'determine which issues get fixed'—there is no reproduce→diagnose→add-regression-before-repair→recheck phase at all. An agent can run a browser smoke/exploration pass and score it, but cannot reliably produce the full deliverable set or handle non-browser surfaces from this bundle alone."},"latency_ms":14009,"stop_reason":"end_turn","usage":{"input_tokens":4371,"output_tokens":882},"cost_usd":0.08781}
+{"config":"qa-workflow","item":"p07","split":"heldout","phase":"new","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser methodology is dense but mostly followable: modes are distinguished, phases are ordered, concrete aside repl scripts with sentinel output (GSTACK_STEP_OK, DIFF_START/END) are provided, and the Health Score Rubric and baseline.json schema are fully computable. However, several decisions hinge on undefined externals: the shared exploratory loop's checkpoint/exact-replay/deadline rules, BROWSER SETUP's access decision and `$B` handoff semantics, `$REPORT_DIR`, `qa/references/issue-taxonomy.md`, and the report template are referenced but not present, so an agent cannot resolve what a 'checkpoint' or 'clamp the deadline guard' actually requires from this file alone. Mode selection is ambiguous on a feature branch without URL (Diff-aware 'selects scope, not another pass' yet Full is 'default with a URL' and the no-URL case only specifies Quick when pages are unidentifiable). Blocked-path handling (NEEDS_ASIDE/ASIDE_NOT_RUNNING) names states but gives no actions. Relative to the stated evaluation scope—native functional/mixed surfaces, regression-before-repair, DX loading, isolation and authority contracts—this bundle covers only the browser slice, leaving large parts of the workflow undefined. Scripts use placeholders like `e12` and assume helper APIs (snapshot, annotatedScreenshot, pwd) without defining them, which limits standalone actionability, though an agent with the companion files could likely execute and produce the report and baseline deliverables."},"latency_ms":21787,"stop_reason":"end_turn","usage":{"input_tokens":5407,"output_tokens":1444},"cost_usd":0.12627}
+{"config":"qa-workflow","item":"f07","split":"heldout","phase":"new","sample":0,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is short and superficially readable, but its decision rules are vague ('any file that looks related', 'if you have time', 'a batch of fixes') and several directly contradict the stated workflow contract: regression tests are written after repair rather than before, failing regression tests are deleted and failing existing tests have their expectations rewritten to match new behavior, which masks defects instead of reproducing them. Re-testing is optional, so there is no verified recheck. The 'Report' step lists issues as fixed with no evidence, coverage, or diagnosis output. It omits every other phase the bundle is supposed to define: surface selection (browser/native/mixed), bounded exploration probes, reproduction and diagnosis, authority boundaries, isolation, native contracts, conditional browser/DX loading conditions, and blocked paths. Scope is unbounded ('no limit', 'clean up nearby code', add dependencies and cross-module refactors freely), and the recovery instruction `git reset --hard HEAD~1` is destructive and can discard unrelated work. An agent following this would produce unverified, scope-creeping changes with suppressed test signal and no credible evidence, so it cannot deliver the expected outputs."},"latency_ms":11511,"stop_reason":"end_turn","usage":{"input_tokens":1148,"output_tokens":864},"cost_usd":0.05468}
+{"config":"qa-workflow","item":"p06","split":"heldout","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The single-file native workflow is well structured: eight ordered steps with concrete gates (dirty git → stop, non-owned destination → stop and ask, missing regression report → block, fail-twice → confirmed defect, stop after two reverts/unrelated edit/20 fixes), a precise contract map that separates acceptance/enqueue/processing/final effect, evidence layout (evidence/NNN, exploration.md), commit format, classification states and a detailed report schema with a fixed summary line. Gaps hold it below 5: `--quick` is overloaded across Mode and Fix tier without saying whether one flag sets both; 'highest-risk changed edge' and 'adjacent happy path' have no selection rule; regression mode says to read the previous report but Step 4 never defines how a regression run differs (which contracts to replay, how to compare); a command timeout has no mapped outcome (blocked vs inconclusive); cleanup of owned state is only mentioned in the report, not as an executed step; and the file asserts it is the whole workflow yet gives no routing or handoff for mixed targets or browser/DX loading, which the bundle-level evaluation expects. Overall an agent can execute it and produce the deliverables with few ambiguities."},"latency_ms":16314,"stop_reason":"end_turn","usage":{"input_tokens":2892,"output_tokens":1146},"cost_usd":0.08622}
+{"config":"qa-workflow","item":"a07","split":"heldout","phase":"new","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The Setup section is reasonably clear: it specifies the authority rule for loading sections/scope.md from the installed skill path, the blocked-path fallback when assets are missing, parameter parsing with defaults, tier/mode interactions (including the --quick/--regression conflict resolution), the dirty-tree AskUserQuestion branch, and REPORT_DIR creation/collision rules. However, there are ambiguities (e.g., --quick appearing in both Tier and Mode rows, 'infer or ask' for Target with no decision criteria, 'select the surface first' without any selection heuristics in this file, and 'Browser session handling lives in browser setup' referencing a file never introduced). The bundle as presented is severely incomplete: Phases 1-6 (exploration, bounded probes, reproduction, diagnosis), Phase 7 triage rules beyond tier mapping, the Phase 8 fix loop (including the regression-before-repair requirement from the stated purpose), Phase 9 recheck criteria, the Phase 10 report schema/evidence/coverage format, and the Phase 11 TODOS.md update are each a single sentence or absent. Native contracts, browser/DX conditional loading conditions, isolation mechanics, and probe bounds are referenced but not defined here, and the referenced section files are not included. An agent could execute setup and artifact-directory preparation, but could not reliably produce the report, regression tests, or coverage deliverables from this document alone; 'When in doubt, stop and ask' would be triggered constantly."},"latency_ms":15016,"stop_reason":"end_turn","usage":{"input_tokens":2017,"output_tokens":902},"cost_usd":0.06527}
+{"config":"qa-workflow","item":"a07","split":"heldout","phase":"old-a","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The Setup section is reasonably precise: parameter table, tier/mode interaction rules (including the --quick/--regression conflict), clean-tree gate with explicit options, and well-specified REPORT_DIR creation/collision rules. However, the bundle as presented is incomplete: `sections/scope.md` (surface selection, isolation, native contracts) and the browser setup section are referenced but absent, so the agent cannot actually determine how to choose browser vs native surfaces, how to bound probes, or what 'blocked' handling means beyond a one-line instruction. Phases 1–6 are collapsed into 'QA Baseline' with no steps for exploration, reproduction, diagnosis, or evidence capture; Phase 8 does not mention adding a regression test before repair despite that being a stated requirement; Phases 9–11 give no report schema, recheck criteria, or TODOS.md format. Overlapping Tier/Mode semantics for `--quick` and an undefined 'regression mode' add ambiguity. An agent could complete setup but would have to invent the entire test→fix→verify loop and deliverables."},"latency_ms":14349,"stop_reason":"end_turn","usage":{"input_tokens":1656,"output_tokens":913},"cost_usd":0.06221}
+{"config":"qa-workflow","item":"f06","split":"heldout","phase":"old-b","sample":2,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"The document is almost entirely a chain of unresolved references to external, unprovided artifacts (QA Handbook 2.3, team wiki environment profile, QA-PROTOCOL-v2 steps P1–P14, release matrix, 'usual tracker format', house S0–S4 scale, release policy, fix-forward policy, deferral process, 'usual place', 'standard evidence bundle', 'normal channel'). None of these are defined in the bundle, so an agent cannot determine what to actually do at any phase. The conflict-resolution note is itself ambiguous ('handbook wins, except where the protocol is newer' gives no way to establish recency or what 'newer' overrides). Every required workflow element is absent: no surface selection (browser vs native functional vs mixed), no bounded exploration probes, no reproduction/diagnosis procedure, no regression-before-repair rule, no recheck step, no evidence/coverage reporting format, no authority/isolation rules, no native contracts, no conditional browser/DX loading, and no blocked-path handling. The four section headings sketch a generic order but provide no executable decisions or deliverables."},"latency_ms":9996,"stop_reason":"end_turn","usage":{"input_tokens":776,"output_tokens":684},"cost_usd":0.04196}
+{"config":"qa-workflow","item":"a08","split":"heldout","phase":"old-a","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The exploration/documentation portions (Orient, Explore, Document, health rubric) are concrete with runnable $B commands and explicit scoring, but the bundle is structurally broken: Phase 2 is absent (1 jumps to 3), $REPORT_DIR is never defined, 'Create output directories' is unspecified, baseline.json has no schema, and the Auth parameter has no corresponding sign-in step. Tiers are framed as 'which issues get fixed' yet there is no repair, diagnose, regression-test-before-fix, or recheck phase at all; 'regression' only means baseline diffing. No native/functional surface selection, isolation, authority, or blocked-path guidance exists despite the stated scope. Referenced files (qa-report-template.md, issue-taxonomy.md) and the /qa-only command are external and undefined. Rule 5 ('never read source code') directly conflicts with diff-aware mode's instruction to map controller/route/model files to URLs. An agent could execute a browser smoke/exploration pass and produce a scored report, but cannot produce the full reproduce→regression→repair→recheck deliverables the workflow is supposed to cover."},"latency_ms":15509,"stop_reason":"end_turn","usage":{"input_tokens":4371,"output_tokens":1068},"cost_usd":0.09711}
+{"config":"qa-workflow","item":"f08","split":"heldout","phase":"old-a","sample":2,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"This is a philosophy essay, not a workflow. It contains no phases, no surface-selection decision (browser vs native vs mixed), no bounded-probe rules, no reproduce/diagnose/regression-before-repair sequence, no recheck step, and no report/evidence/coverage deliverable spec. None of the expected mechanics exist: no authority or isolation rules, no native contracts, no conditional loading of browser/DX sections, no blocked-path handling. The prose is readable, but an agent cannot derive a single executable decision or output format from it; 'show a screenshot or log line' and 'prefer small changes' are the only vaguely operational hints. Every dimension fails at the floor."},"latency_ms":8555,"stop_reason":"end_turn","usage":{"input_tokens":931,"output_tokens":488},"cost_usd":0.03371}
+{"config":"qa-workflow","item":"f06","split":"heldout","phase":"old-a","sample":0,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"The document is a thin shell of pointers to external artifacts that are not provided: 'QA Handbook section 2.3', 'QA-PROTOCOL-v2' steps P1–P14, a 'release matrix', the 'usual tracker format', a 'house severity scale', 'fix-forward policy', 'deferral process', 'the usual place', 'standard evidence bundle', and 'the normal channel'. None of these are defined, so an agent cannot determine what to do at any phase. It gives no criteria for selecting browser vs. native vs. mixed surfaces (only a vague web/services split by undefined protocol step numbers), no bounded probe or exploration budget, no reproduction/diagnosis steps, no regression-before-repair requirement, no recheck loop, no isolation or authority rules, no conditional loading triggers, and no blocked-path handling. The sole explicit rule ('handbook wins, except where the protocol is newer') is self-contradictory without a way to compare versions. No deliverable is specified in a form an agent could produce. Essentially unexecutable."},"latency_ms":10069,"stop_reason":"end_turn","usage":{"input_tokens":776,"output_tokens":617},"cost_usd":0.03861}
+{"config":"qa-workflow","item":"f07","split":"heldout","phase":"old-a","sample":1,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is a single 'Fix Loop' fragment, not the described bundle: there is nothing on surface selection (browser/native/mixed), bounded probes, isolation, authority, native contracts, conditional browser/DX loading, blocked paths, or evidence/coverage reporting. The steps read simply but are ambiguous and open-ended ('any file that looks related', 'clean up nearby code that looks messy', 'if you have time', 'any order', unlimited fixes, unbounded refactors/dependencies) with no scope limits, stop conditions, or decision criteria. Several instructions are actively wrong for a QA/repair workflow: the regression test is written after the fix (the brief requires it before repair), a failing regression test is deleted rather than treated as evidence the fix is incomplete, existing failing tests are rewritten to match the new behavior (masking regressions), re-verification is optional, `git reset --hard HEAD~1` destroys uncommitted work, and the final report lists issues as fixed with no evidence or coverage. An agent executing this would produce unverified claims and likely degrade the codebase, so it does not yield the expected deliverables."},"latency_ms":12724,"stop_reason":"end_turn","usage":{"input_tokens":787,"output_tokens":803},"cost_usd":0.04802}
+{"config":"qa-workflow","item":"p07","split":"heldout","phase":"old-a","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser methodology itself is reasonably structured (Modes → Phases 1-6 → rubric) with concrete, copy-pasteable aside scripts, explicit sentinels (GSTACK_STEP_OK, ASIDE_DIR), a precise scoring rubric, and a defined baseline.json schema. However, the text is extremely dense and jargon-heavy, leaning on undefined external contracts ($B, NEEDS_ASIDE, 'shared exploratory loop', 'checkpoint rule', 'BROWSER SETUP Browser access decision', caller 'artifact/mixed-report rules', issue-taxonomy.md, qa-report-template.md) that are not in the bundle, so an agent cannot resolve key decisions (checkpoint timing, deadline clamping, mutation safety gating) from this file alone. Mode selection has ordering ambiguity (Diff-aware 'no URL' vs 'find a local app if URL missing' after surface isolation; Full 'default with URL'). Most critically, the evaluation asks for the complete workflow—surface selection (browser/native/mixed), native functional contracts, isolation/authority rules, regression-before-repair, blocked paths, and coverage reporting—none of which appear here; only the browser leg is present. As a standalone browser methodology it is fairly actionable; as the complete QA workflow it is substantially incomplete and depends on missing files to be executable."},"latency_ms":18373,"stop_reason":"end_turn","usage":{"input_tokens":5046,"output_tokens":1194},"cost_usd":0.11016}
+{"config":"qa-workflow","item":"a07","split":"heldout","phase":"old-b","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The Setup section is reasonably precise: parameter table, tier/mode interactions (including the --quick/--regression conflict rule), dirty-tree handling via explicit user choice, and report-directory/run-subdirectory rules with no-overwrite and authority limits are all executable. It also handles the missing scope.md case with a blocked-probe fallback. However, the bundle as presented is largely a pointer to files not included (sections/scope.md, 'browser setup', DX/native contracts), so surface selection, isolation, synthetic identity creation, bounded probe budgets, and blocked-path handling cannot actually be followed from this text. The 'Phases' section collapses Phases 1–6 into a single sentence with no probe limits, reproduction/diagnosis steps, or evidence format; the Fix Loop omits the required 'add regression before repair' step and re-test criteria; Final QA, Report (contents, coverage/evidence schema) and TODOS.md Update are one-liners with no defined outputs. Minor ambiguities also exist (--quick listed under both Tier and Mode, Auth row referencing an unnamed file). An agent could complete setup but would have to invent most of the test→fix→verify loop and deliverables."},"latency_ms":17427,"stop_reason":"end_turn","usage":{"input_tokens":1656,"output_tokens":988},"cost_usd":0.06596}
+{"config":"qa-workflow","item":"p06","split":"heldout","phase":"old-a","sample":1,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"Within its own scope (native worker/job/webhook targets) the file is tightly structured: numbered steps, explicit stop conditions (dirty git, production/third-party destinations, blocked regression mode, two reverts, 20 fixes, unrelated file edits), a clear contract table, a confirm-twice-then-minimize replay rule, test-before-fix ordering, recheck/classify semantics, and a fully specified report layout and summary line. However, the evaluation asks for the complete bundle covering browser/native/mixed surface selection, conditional browser/DX loading and the authority layer, and this file declares itself 'the whole workflow' while assuming the native target has already been selected—no routing logic, mixed-surface handling, or DX/browser sections are present. Remaining ambiguities: how the 'highest-risk changed edge' is chosen in quick mode, what regression mode actually replays after reading the prior report, what to do when no documented local runner exists (blocked vs. ask), and how 'owned state removed' cleanup is performed. An agent could execute this file end-to-end for a native target and produce the deliverables, but the surrounding bundle contract is not satisfied."},"latency_ms":14289,"stop_reason":"end_turn","usage":{"input_tokens":2892,"output_tokens":950},"cost_usd":0.07642}
+{"config":"qa-workflow","item":"a07","split":"heldout","phase":"new","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The Setup portion is reasonably precise: it defines authority (host-installed path, never product dir), blocked-path handling for missing assets, parameter parsing, fix-tier semantics, the --quick/--regression conflict rule, dirty-tree handling via AskUserQuestion, and REPORT_DIR allocation with no-overwrite rules. However, the bundle as presented is essentially only a preamble. Phases 1-6 (surface selection, bounded probes, reproduction/diagnosis, regression-before-repair), Phase 7 triage criteria, the Phase 8 fix loop, Phase 9 recheck, Phase 10 report schema/evidence/coverage format, and Phase 11 TODOS.md update are named but never defined. There is no conditional browser/DX section loading logic, no native functional contract (how to probe API/CLI/job/worker/webhook surfaces or create the synthetic identity), no definition of exploration tiers beyond the flag mapping, and 'browser setup' is referenced but not located. Some ambiguity remains even in Setup (Tier vs Mode both keyed on --quick; 'infer or ask' for target with no inference rules). An agent could complete setup but could not execute the core test→fix→verify loop or produce the expected report deliverables from this document alone."},"latency_ms":13025,"stop_reason":"end_turn","usage":{"input_tokens":2017,"output_tokens":783},"cost_usd":0.05932}
+{"config":"qa-workflow","item":"f06","split":"heldout","phase":"new","sample":2,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"The document is almost entirely composed of references to external artifacts that are not included or defined: 'QA Handbook section 2.3', 'environment profile your team normally uses', 'team wiki', 'QA-PROTOCOL-v2', 'release matrix', 'usual tracker format', 'house severity scale', 'fix-forward policy', 'deferral process', 'usual place', 'standard evidence bundle', 'normal channel'. None of these are resolvable by an AI agent, so no phase can actually be executed. There is no guidance on surface selection (browser vs native vs mixed), bounded probes, reproduction/diagnosis, adding a regression before repair, recheck, or evidence/coverage reporting—the core workflow the agent is supposed to perform. The S0–S4 scale is named but not defined. The precedence note ('handbook wins, except where the protocol is newer') is circular and introduces ambiguity rather than resolving it. No authority, isolation, native contracts, conditional loading, or blocked-path rules are present. The structure (prep/execute/fix/sign-off) is the only coherent element, but it carries no executable content."},"latency_ms":6981,"stop_reason":"end_turn","usage":{"input_tokens":1137,"output_tokens":408},"cost_usd":0.03177}
+{"config":"qa-workflow","item":"f08","split":"heldout","phase":"new","sample":1,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"The document is a well-written mindset manifesto, not a workflow. It contains no phases, decision points, surface selection logic (browser vs. native vs. mixed), probe bounding rules, reproduction/diagnosis steps, regression-before-repair contract, recheck procedure, or evidence/coverage reporting format. None of the required structural elements—authority model, isolation requirements, native contracts, conditional browser/DX loading triggers, or blocked paths—are present or even referenced. The prose itself is lucid, but there are no step-by-step phases for an agent to follow, nothing defines outputs or deliverables, and an agent could not execute anything concrete from it beyond adopting an attitude. As a QA testing workflow for an AI coding agent it is essentially empty on all three dimensions."},"latency_ms":23699,"stop_reason":"end_turn","usage":{"input_tokens":1292,"output_tokens":512},"cost_usd":0.03852}
+{"config":"qa-workflow","item":"f07","split":"heldout","phase":"old-a","sample":2,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is only a 'Fix Loop' fragment and is readable at a surface level, but its instructions are ambiguous and internally harmful: 'any file that looks related is in scope' plus 'clean up nearby code' invites unbounded scope creep; 'commit when you have a batch' gives no boundary; and the regression step inverts the stated contract (regression-before-repair) by telling the agent to delete failing tests and rewrite existing tests' expected values to match new behavior, which erases evidence rather than producing it. Re-testing is optional and the final report declares everything fixed regardless of verification, so no evidence/coverage output exists. Completeness is minimal: none of the required phases are present — no surface selection (browser/native/mixed), no bounded probes, no reproduce/diagnose step, no authority or isolation rules, no native contracts, no conditional browser/DX loading, no blocked-path handling, no severity triage or fix budget. Actionability is poor because following it literally produces destructive outcomes (git reset --hard discards work, deleted regression tests, scope-unbounded refactors and new dependencies) and a false 'all fixed' deliverable rather than verified, evidenced results."},"latency_ms":13495,"stop_reason":"end_turn","usage":{"input_tokens":787,"output_tokens":822},"cost_usd":0.04897}
+{"config":"qa-workflow","item":"f08","split":"heldout","phase":"old-b","sample":1,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"This is a mindset essay, not an executable workflow. It contains no phases, no step sequence, no decision points for selecting browser vs. native vs. mixed surfaces, no bounded-probe rules, no reproduction/diagnosis procedure, no 'regression before repair' gate, no recheck step, and no report/evidence/coverage format. Nothing addresses authority, isolation, native contracts, conditional loading of browser/DX sections, or blocked paths. An agent reading this cannot determine what to do first, when to stop, what artifacts to produce, or how to handle any branch; the only concrete guidance is 'show a screenshot or log line' and 'prefer small changes,' which is far too vague to drive execution."},"latency_ms":7358,"stop_reason":"end_turn","usage":{"input_tokens":931,"output_tokens":368},"cost_usd":0.02771}
+{"config":"qa-workflow","item":"f07","split":"heldout","phase":"old-a","sample":0,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"This is a single unbounded 'fix loop' file, not the described bundle: there is no surface selection (browser/native/mixed), no bounded exploration probes, no reproduce/diagnose phase, no authority or isolation rules, no native contracts, no conditional browser/DX loading, and no blocked-path handling. Steps are superficially readable but vague ('any file that looks related', 'if you have time') and several instructions are actively harmful and contradict the stated goals: cleaning up 'nearby messy code' and cross-module refactors invite unbounded scope creep; the regression test is written after the fix and deleted if it fails, which inverts regression-before-repair and discards the very signal that the fix is wrong; updating existing tests' expected values to match new behavior masks regressions; re-test is optional; `git reset --hard HEAD~1` is a destructive recovery that can discard other work; and the report simply 'lists the issues as fixed' with no evidence, coverage, or verification. An agent following this would produce an unverifiable report and likely degrade the codebase."},"latency_ms":11345,"stop_reason":"end_turn","usage":{"input_tokens":787,"output_tokens":661},"cost_usd":0.04092}
+{"config":"qa-workflow","item":"f06","split":"heldout","phase":"old-a","sample":2,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"The document is a thin shell that delegates every substantive decision to external artifacts that are neither included nor linked: the QA Handbook section 2.3, QA-PROTOCOL-v2 (P1–P14 contents never defined), the release matrix, the 'usual tracker format', the S0–S4 house severity scale, the fix-forward and deferral policies, the 'standard evidence bundle', the team wiki's primary profile, the QA owner, and the 'normal channel'. An agent cannot select browser vs. native vs. mixed surfaces, has no probe bounds, no reproduction/diagnosis steps, no regression-before-repair rule, no recheck step, and no evidence/coverage report schema. The only authority rule ('handbook wins, except where the protocol is newer') is self-contradictory and undecidable without version metadata. Nothing addresses isolation, native contracts, conditional loading, or blocked paths. Every phase is a pointer to tribal knowledge, so none is executable."},"latency_ms":9185,"stop_reason":"end_turn","usage":{"input_tokens":776,"output_tokens":531},"cost_usd":0.03431}
+{"config":"qa-workflow","item":"p08","split":"heldout","phase":"old-b","sample":0,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"The functional-surface file is well structured: four ordered steps, an explicit isolation gate with a stop-and-ask rule, a contract-type-to-probe table, defined outcome states (pass/fail/blocked/not run/inconclusive), a confirmation rule for failures (re-run once from same state), evidence layout (`evidence/NNN/`), and a concrete `report.md` spec with a crisp verdict rule. Decisions are mostly executable without interpretation. Weaknesses: 'required contract' is never defined, so the pass/not-passing verdict hinges on an undefined classification; 'inconclusive' lacks criteria distinguishing it from 'not reproduced'; the 60s timeout default doesn't say what to do for long-running jobs. More significantly, as the full bundle is described (surface selection between browser/native/mixed, reproduce-and-diagnose, add regression before repair, recheck behavior, conditional browser/DX loading), this file covers only exploration and reporting: regression tests are 'proposed' rather than added before repair, there is no diagnosis or recheck phase, no mixed-surface guidance, and no loading condition or handoff back to the selector/browser files. Blocked handling is clear within the file. Strong as a standalone probe-and-report procedure, incomplete against the stated end-to-end workflow."},"latency_ms":42400,"stop_reason":"end_turn","usage":{"input_tokens":1779,"output_tokens":833},"cost_usd":0.05944}
+{"config":"qa-workflow","item":"a08","split":"heldout","phase":"new","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser-exploration portion is readable with concrete commands, a clear tier/mode table, and a precise health-score rubric. However the document labels itself Phases 1-6 yet jumps from Phase 1 to Phase 3 (Phase 2 is absent), never defines `$REPORT_DIR` or `SKILL_DIR`, references an external report template and issue taxonomy without inline fallback, and leaves `baseline.json` schema and the regression diff procedure unspecified. The Auth parameter is parsed but never used in any phase. Rule 5 ('Never read source code') conflicts with diff-aware mode's instruction to inspect controllers/models to map routes. Against the stated scope the workflow is substantially incomplete: there is no native/functional or mixed-surface selection, no bounded-probe limits, no diagnosis step, no 'add a regression test before repair' step, no repair/recheck loop despite tiers describing 'which issues get fixed', no isolation or authority rules, no coverage reporting beyond pages visited, and blocked paths (setup declined, no reachable URL, auth failure) are handled only by 'ask the user'. An agent could execute the browser-only exploration and produce a scored report, but could not produce the full set of expected deliverables (regression, repair verification, native coverage) from this bundle."},"latency_ms":19293,"stop_reason":"end_turn","usage":{"input_tokens":4732,"output_tokens":1173},"cost_usd":0.10597}
+{"config":"qa-workflow","item":"a08","split":"heldout","phase":"old-b","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The browser-testing portion is written in concrete, mostly followable steps (setup script, diff-aware mode, orient/explore/document loops, scoring rubric), but the document has real gaps and contradictions. Phase 2 is entirely missing (Phase 1 jumps to Phase 3), `$REPORT_DIR` is never defined relative to the 'Output dir' parameter, '/qa-only' is referenced but never specified, and Rule 5 ('Never read source code') directly conflicts with diff-aware mode's instruction to analyze changed controller/view/model files. Against the stated evaluation scope the bundle is badly incomplete: there is no native functional or mixed-surface selection, no authority/isolation guidance, no native contracts, no blocked-path handling, no conditional DX loading, and no repair phase at all despite 'Tiers determine which issues get fixed' — nothing covers adding a regression before repair or rechecking after a fix. Key deliverables depend on files not included (`qa-report-template.md`, `issue-taxonomy.md`) and the `baseline.json` schema consumed by regression mode is undefined, so an agent cannot reliably produce or diff that artifact. Bounded probing is implied by time estimates but no explicit probe budgets or stop conditions are given. An agent could execute a browser smoke/exploration pass, but could not execute the full described workflow or produce consistent regression/coverage outputs."},"latency_ms":15812,"stop_reason":"end_turn","usage":{"input_tokens":4371,"output_tokens":1087},"cost_usd":0.09806}
+{"config":"qa-workflow","item":"f07","split":"heldout","phase":"new","sample":2,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is a single 'Fix Loop' fragment, not the described bundle: there is no surface selection (browser/native/mixed), no bounded probing, no reproduction/diagnosis phase, no authority or isolation rules, no native contracts, no conditional browser/DX loading, and no blocked-path handling. The steps it does provide are readable but vague ('any file that looks related', 'if you have time') and actively counterproductive: unbounded scope with opportunistic cleanup and cross-module refactors, regression tests written after the fix rather than before, deleting a failing regression test instead of treating it as a signal, rewriting existing tests' expected values to match new behavior (masking regressions), making re-verification optional, batching commits and recovering with destructive `git reset --hard`, and reporting issues 'as fixed' with no evidence or coverage. An agent following this would produce unverified, potentially harmful changes and a report that does not match reality, so it fails on completeness and actionability and is only marginally clear."},"latency_ms":11334,"stop_reason":"end_turn","usage":{"input_tokens":1148,"output_tokens":748},"cost_usd":0.04888}
+{"config":"qa-workflow","item":"f07","split":"heldout","phase":"old-b","sample":2,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is short and its sentences are individually readable, but it is not an executable QA workflow. It covers only a post-report fix loop and omits every other required phase: no authority or isolation rules, no native-vs-browser surface selection, no bounded exploration probes, no conditional browser/DX loading, no blocked-path handling, no severity triage or ordering, and no evidence/coverage reporting. The steps it does define are internally harmful and contradictory to the stated goal of adding a regression before repair: it tells the agent to refactor and 'clean up nearby code' (unbounded scope creep), to delete a regression test if it fails (eliminating the only evidence of a defect), to rewrite existing tests' expected values to match new behavior (masking regressions), makes re-verification optional, prescribes a destructive 'git reset --hard HEAD~1' without checking what is lost, and ends with 'list the issues as fixed' regardless of verification. An agent following this literally would produce unverified, scope-creeping changes and a report that misrepresents state, so actionability toward correct deliverables is essentially nil."},"latency_ms":14154,"stop_reason":"end_turn","usage":{"input_tokens":787,"output_tokens":836},"cost_usd":0.04967}
+{"config":"qa-workflow","item":"p07","split":"heldout","phase":"old-b","sample":2,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":4,"reasoning":"The browser QA section itself is well-structured: six ordered phases, three modes with explicit time caps and selection rules, copy-pasteable aside repl scripts with sentinel output markers (GSTACK_STEP_OK, DIFF_START/END), a precise Health Score Rubric with dedup rules, weights and rounding, and a defined baseline.json schema. Those elements make the browser probes highly executable. However, clarity suffers from very dense, jargon-heavy phrasing that assumes insider context ('Diff-aware selects scope, not another pass', 'LOOK is not ACT', 'clamp the shared loop's deadline guard', 'Resolve conflicting depth flags by asking') and from undefined references used as if known ($REPORT_DIR, $B handoff/resume, Aside READY, the shared loop's checkpoint/exact-replay rules, pwd inside repl). Completeness is only partial relative to the full workflow being evaluated: execution order, isolation, authority, native/mixed surface contracts, regression-before-repair, blocked paths, and mixed-report rules are all delegated to other unseen files (shared exploratory loop, BROWSER SETUP, issue-taxonomy, report template), so an agent cannot resolve surface selection, native probing, or stop/blocked behavior from this file alone. Minor internal tensions (Quick 'skips the checklist' vs. 'Quick plus discovered interactions'; 'Observe expected result' without a source for expectations given the no-source-reading rule) add ambiguity. Actionability remains strong for the browser portion because every probe, screenshot copy step, link check, mobile emulation, and scoring computation is concretely specified."},"latency_ms":29655,"stop_reason":"end_turn","usage":{"input_tokens":5046,"output_tokens":1432},"cost_usd":0.12206}
+{"config":"qa-workflow","item":"p06","split":"heldout","phase":"old-b","sample":1,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The file is a tight, sequential native-only workflow: authority gate (dirty git stop, non-overwriting output dirs), isolation with an explicit production/third-party STOP, a contract map drawn from docs/tests rather than implementation, a probe loop with per-probe evidence layout and a two-run confirmation rule, severity triage, a test-first repair loop with named classifications and stop triggers, a final regression run, and a fully specified report with filename, sections and summary line. Decisions are mostly binary and executable. Gaps: Regression mode is only defined as 'read the previous report' with no instruction on which contracts to replay or how outcomes map to the new report; `--quick` is listed under both Mode and Fix tier without saying whether one flag sets both; 'highest-risk changed edge' and 'stop reason' lack a selection/vocabulary rule; 'observed once' failures have no explicit triage destination; the partial-failure probe (kill after first effect) gives no guidance on mechanism. The file declares itself the whole workflow and excludes browser/DX, so conditional browser loading and mixed surfaces are out of its scope and not evaluable here, which slightly limits judging the bundle-level routing."},"latency_ms":44105,"stop_reason":"end_turn","usage":{"input_tokens":2892,"output_tokens":1518},"cost_usd":0.10482}
+{"config":"qa-workflow","item":"a07","split":"heldout","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The Setup section is fairly clear and prescriptive (STOP gate for scope.md, parameter table, tier/mode semantics, dirty-tree AskUserQuestion with three options, REPORT_DIR creation rules), though Tier vs Mode overlap on `--quick`, 'Quick exploration' is referenced but never defined, and the diff-aware/surface-selection guidance is only asserted. The core of the workflow is essentially absent: Phases 1-6 are collapsed into a single sentence with no probe bounds, surface-selection criteria, reproduction/diagnosis steps, evidence capture rules, or regression-before-repair requirement; Phases 7-11 are one-line labels with no triage criteria, fix-loop commit/re-test procedure, final-QA scope, report schema, or TODOS.md format. Native contracts, conditional browser/DX section loading, blocked-path handling beyond the setup blocker, and the referenced `sections/scope.md` are not present in the bundle. An agent could complete setup but could not reliably execute the phases or produce the expected report/coverage deliverables from this document alone; it depends on unseen section files and the fallback is 'when in doubt, stop and ask'."},"latency_ms":13220,"stop_reason":"end_turn","usage":{"input_tokens":1656,"output_tokens":818},"cost_usd":0.05746}
+{"config":"qa-workflow","item":"p08","split":"heldout","phase":"old-b","sample":2,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"The functional-surface file itself is well structured: numbered steps, an explicit isolation/permission gate, a contract-to-probe table, defined outcome states (pass/fail/blocked/not run/inconclusive), a reproduction rule, evidence paths, and a precise report spec with a clear verdict rule. Minor ambiguities remain (what makes a contract 'required' vs optional is never defined; 'reset to planned initial state' assumes reset tooling exists; no guidance on bounding total probe count or when to stop exploring). As an evaluation of the complete workflow, however, this bundle only supplies the native/functional branch: there is no surface-selection logic for browser vs native vs mixed, no browser or DX conditional-loading section, no authority statement about which file wins on conflict, and the lifecycle stops at 'proposed regression tests' rather than adding a failing regression before repair, diagnosing root cause, and rechecking after the fix. An agent can execute this file and produce report.md plus evidence, but cannot complete the broader regression-and-recheck deliverables from what is provided."},"latency_ms":15451,"stop_reason":"end_turn","usage":{"input_tokens":1779,"output_tokens":965},"cost_usd":0.06604}
+{"config":"qa-workflow","item":"f07","split":"heldout","phase":"old-b","sample":1,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is short and superficially readable, but its instructions are incoherent and contradict the bundle's stated workflow. It covers only a post-report 'fix' phase and omits surface selection (browser/native/mixed), bounded probes, reproduction and diagnosis, isolation/authority boundaries, native contracts, conditional browser/DX loading, blocked paths, and evidence/coverage reporting. Key decisions are inverted or unsafe: the regression test is written after the fix (not before), a failing regression test is deleted rather than treated as a signal, existing tests are rewritten to match new behavior (masking regressions), scope is unbounded ('clean up nearby code', add dependencies, cross-module refactors), re-verification is optional, batch commits plus `git reset --hard HEAD~1` risk discarding work, and the final report asserts 'fixed' with no evidence or recheck. An agent following this would produce unverifiable, likely harmful changes rather than the expected deliverables."},"latency_ms":12586,"stop_reason":"end_turn","usage":{"input_tokens":787,"output_tokens":739},"cost_usd":0.04482}
+{"config":"qa-workflow","item":"a07","split":"heldout","phase":"old-a","sample":2,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"The Setup section is reasonably clear and prescriptive: parameter parsing, tier/mode semantics (including the --quick/--regression conflict rule), clean-tree gating via AskUserQuestion with three explicit options, and a well-specified REPORT_DIR allocation policy (empty-dir check, UTC run subdirectory, no-overwrite, caller-path precedence, output blocker reporting). However, the bundle is largely a skeleton beyond Setup. The STOP directive depends on `sections/scope.md`, and the browser setup, native contracts, DX loading and blocked-path sections are referenced but not present, so authority, isolation and surface selection cannot actually be executed from what is given. Phases 1-6 ('QA Baseline') are collapsed into a single sentence with no definition of exploration probes, probe bounds, reproduction, diagnosis or evidence format; Phases 7-11 are one-liners. Critical workflow requirements stated in the task—adding a regression test before repair, recheck criteria, evidence/coverage reporting, report schema, TODOS.md update format—are absent. The Tier/Mode table overlaps confusingly (both list --quick) and 'unless this invocation already established its surfaces and isolation' is undefined. An agent could complete setup but would stall at Phase 1 without the missing files, and could not produce the expected deliverables (report, regression tests, coverage) from this document alone."},"latency_ms":17162,"stop_reason":"end_turn","usage":{"input_tokens":1656,"output_tokens":1211},"cost_usd":0.07711}
+{"config":"qa-workflow","item":"p07","split":"heldout","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":3,"actionability":4,"reasoning":"The browser-QA file is internally well-structured (modes → six phases → rubric → framework hints → rules) with concrete, copy-pasteable aside repl scripts, explicit sentinels (GSTACK_STEP_OK, ASIDE_DIR), a fully computable Health Score rubric (weights sum to 100, rounding and provisional rules stated), and a defined baseline.json/report deliverable. However, the prose is extremely terse and jargon-laden ('LOOK is not ACT', '$B handoff', 'Aside READY', 'caller's smoke and plan bounds'), and many critical decisions are delegated to files not shown: the shared exploratory loop's checkpoint/exact-replay/deadline rules, BROWSER SETUP's access decision and mutation-approval rules, issue-taxonomy, the report template, and caller artifact/mixed-report rules. Nothing here covers native functional surfaces, surface selection/isolation, authority boundaries, blocked paths, or the regression-before-repair step the overall workflow promises, so as a bundle view it is incomplete. Minor ambiguities remain (placeholder locator 'e12', $REPORT_DIR origin, how Quick interacts with diff-aware when no pages map). An agent with the companion files could execute this reliably; standing alone it leaves several gating decisions underspecified."},"latency_ms":19098,"stop_reason":"end_turn","usage":{"input_tokens":5046,"output_tokens":1232},"cost_usd":0.11206}
+{"config":"qa-workflow","item":"p06","split":"heldout","phase":"new","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The document is tightly scoped (native workers/jobs/webhooks, no browser) and explicitly self-contained, so the conditional browser/DX loading question is resolved by exclusion. Steps are ordered and most decision points are binary and explicit: dirty git tree → stop and ask; regression report missing/unreadable/wrong-target → blocked, offer Full; destination possibly production/third party → STOP naming target/operation/effect; fail once vs twice → observed-once vs ISSUE-NNN; stop conditions after two reverts, unrelated edit, or 20 fixes. The contract map table, evidence layout (evidence/NNN/, exploration.md), severity ladder, classification states (verified/best-effort/reverted/deferred), commit format, regression-comment format, and report sections are concrete enough to execute and produce deliverables. Weaknesses: regression mode specifies only reading the prior report, not which contracts to re-probe or how outcomes compare; `--quick` doubles as both Mode and Fix tier flag without stating they are coupled; 'highest-risk changed edge' and 'owned state' are left to judgment with no heuristic; partial-failure injection and kill/restart mechanics are unspecified; the case where no documented local run path exists (step 2.1) is not explicitly routed to blocked; cleanup is only reported, not sequenced as a step. These gaps are minor relative to an otherwise crisp, executable workflow."},"latency_ms":24595,"stop_reason":"end_turn","usage":{"input_tokens":3253,"output_tokens":1671},"cost_usd":0.11608}
+{"config":"qa-workflow","item":"a07","split":"heldout","phase":"old-b","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":2,"reasoning":"Setup is reasonably explicit (parameter table, tier/mode semantics, clean-tree AskUserQuestion with three options, REPORT_DIR naming rules, authority limits on writes, blocked-path fallback when scope.md is missing). However the bundle as presented omits the files it depends on: sections/scope.md, browser setup, native contracts, and DX/conditional loading are referenced but not provided, so surface selection and isolation cannot actually be evaluated or executed. Phases 1-6 are collapsed into a single line ('explore and document issues') with no probe budgets, reproduction/diagnosis steps, or the 'add regression before repair' requirement stated in the task; Phases 7-11 are one-liners with no report schema, evidence/coverage format, commit conventions, or TODOS.md contract. Residual ambiguities include how to detect 'already established surfaces and isolation', the overlapping --quick in both Tier and Mode rows, undefined 'diff-aware testing', and unspecified handling of --regression baselines. An agent can complete Setup but cannot reliably execute the core test→fix→verify loop or produce the expected deliverables from this document alone."},"latency_ms":26118,"stop_reason":"end_turn","usage":{"input_tokens":1656,"output_tokens":828},"cost_usd":0.05796}
+{"config":"qa-workflow","item":"p06","split":"heldout","phase":"old-a","sample":2,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"Within its own scope (workers, jobs, webhooks) the file is well structured: eight ordered steps, explicit stop/ask gates (dirty git tree, production or third-party destinations, missing regression report, two reverts, unrelated-file edits, 20 fixes), a concrete contract table, defined probe outcomes, a twice-fails confirmation rule, test-before-fix ordering, a commit format, and a fully specified report layout with filename, evidence paths and summary line. Ambiguities remain: `--quick` is overloaded as both a Mode and a Fix-tier flag with different meanings; 'highest-risk changed edge' has no selection rule; regression mode says what to read but not what to rerun or how to compare; command timeouts are set but a timed-out probe's outcome is not classified; the 'partial failure' and ordering/concurrency probes are hard to execute generically with no guidance on how to kill mid-effect; cleanup of owned state is reported but never scheduled as a step. Relative to the stated bundle (browser, native, mixed surfaces, conditional browser/DX loading), this file asserts it is 'the whole workflow' and only addresses the native path, with no routing for mixed targets (e.g., a webhook with an admin UI) or hand-off to browser/DX sections, which limits completeness of the overall decision tree. An agent could nonetheless execute this native path and produce the expected deliverables with modest judgment calls."},"latency_ms":75829,"stop_reason":"end_turn","usage":{"input_tokens":2892,"output_tokens":1389},"cost_usd":0.09837}
+{"config":"qa-workflow","item":"p06","split":"heldout","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":4,"actionability":4,"reasoning":"The workflow is a tight, sequential native-functional path (inputs/authority → isolation → contract map → probe loop → triage → repair → final run → report) with concrete gates (dirty tree stop, production-destination stop, two-replay confirmation, red-test-before-fix, three-part recheck, revert/stop thresholds) and a well-specified report schema with evidence paths and a fixed summary line. Contract table forces observation of durable effects rather than enqueue/2xx, and outcome vocabulary (pass/fail/blocked/not run/inconclusive) is unambiguous. Weaknesses: `--quick` is overloaded to mean both a probe mode and a fix tier without saying whether one flag sets both; 'highest-risk changed edge' has no selection rule; regression mode says what to read but not how the probe loop differs (replay previous contracts only? re-map?); handling of a 'blocked' prerequisite (ask vs. continue) and inability to run the target locally at all are not specified; the run-subdir rule and the fixed report filename interact unclearly; partial-failure and concurrency probes give the acceptance criterion but no execution guidance. The file explicitly scopes out browser/mixed surfaces and conditional DX loading, so those bundle concerns are deferred elsewhere rather than covered here. Overall an agent could execute this and produce the deliverables with few judgment calls."},"latency_ms":19089,"stop_reason":"end_turn","usage":{"input_tokens":2892,"output_tokens":1238},"cost_usd":0.09082}
+{"config":"qa-workflow","item":"a08","split":"heldout","phase":"old-b","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser exploration core (Orient/Explore/Document with concrete $B commands, evidence tiers, health-score rubric) is clear and executable. But the bundle is materially incomplete and internally inconsistent: it is labeled 'Phases 1-6' yet Phase 2 is absent entirely; the Auth parameter (sign-in, cookie import) is declared but never operationalized; REPORT_DIR, SKILL_DIR and the output-dir layout are never defined; referenced files (qa-report-template.md, issue-taxonomy.md) that carry the severity definitions and report format are not included, so severity assignment (which drives both the rubric and tier-based fixing) is unspecified; baseline.json schema and the regression-diff matching rule are undefined. 'Quick' is used both as a tier (fix severity cutoff) and a mode (30-second smoke), and rule 5 'Never read source code' contradicts diff-aware mode's instruction to map controllers/views/models to routes. 'Tiers determine which issues get fixed' and 'add regression before repair' imply later diagnose/repair/recheck phases that do not exist here, and there is nothing on native/functional or mixed surfaces, conditional loading, authority/isolation, native contracts, or blocked paths. An agent can run a browser smoke/exploration and write a screenshot-backed report, but cannot reliably produce the baseline/regression artifacts or any repair deliverables from this document alone."},"latency_ms":26019,"stop_reason":"end_turn","usage":{"input_tokens":4371,"output_tokens":1832},"cost_usd":0.13531}
+{"config":"qa-workflow","item":"p08","split":"heldout","phase":"old-a","sample":0,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"This functional-surface file is well structured: four ordered steps with concrete outputs (plan, probe list, evidence/NNN/, report.md), an explicit isolation gate with a stop-and-ask rule, a clear contract-to-probe table, a defined outcome vocabulary (pass/fail/blocked/not run/inconclusive), a deterministic reproduce rule (two runs = confirmed), and a crisp verdict rule. Secrets handling, timeouts, and durable-state inspection are all actionable. Gaps: it never states its own loading condition or how 'not a web page' is decided for mixed surfaces (CLI that drives a browser, API behind a UI), so the hand-off to the browser/DX sections is implicit; 'required contract' is used in the verdict but never defined versus optional contracts; 'inconclusive' has no assignment criteria; the bundle's described regression-before-repair and recheck phases are reduced here to 'proposed regression tests' with no instruction to actually add the test or re-verify after a fix; evidence numbering, probe ordering, and what to do when multiple probes depend on shared state are left to the agent. Evaluated in isolation it is executable, but as a piece of the fuller workflow it under-specifies several cross-section decision points."},"latency_ms":15267,"stop_reason":"end_turn","usage":{"input_tokens":1779,"output_tokens":859},"cost_usd":0.06074}
+{"config":"qa-workflow","item":"f07","split":"heldout","phase":"new","sample":1,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is short and superficially readable, but its instructions are vague and unbounded ('any file that looks related', 'clean up nearby code', 'if you have time'), so an agent cannot determine scope, stopping conditions, or when a fix is actually verified. It covers only a post-report fix loop and omits every other required phase: no surface selection (browser vs native vs mixed), no bounded exploration probes, no reproduction/diagnosis step, no authority or isolation rules, no native contracts, no conditional browser/DX loading, and no blocked-path handling. Decision points that do exist are actively harmful: regression tests are written after the fix rather than before, failing tests are deleted rather than used as reproduction, existing tests are rewritten to match new behavior (masking regressions), re-testing is optional, new dependencies and cross-module refactors are permitted without limit, and `git reset --hard` is prescribed as recovery, which destroys uncommitted work. The output is just 'list the issues as fixed' with no evidence, repro confirmation, or coverage reporting. An agent following this would produce unverified, scope-creeping changes and an unsupported report rather than the expected deliverables."},"latency_ms":11361,"stop_reason":"end_turn","usage":{"input_tokens":1148,"output_tokens":886},"cost_usd":0.05578}
+{"config":"qa-workflow","item":"f08","split":"heldout","phase":"new","sample":2,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is a well-written mindset manifesto, not a workflow. The prose itself is readable, but there are no phases, no surface-selection logic (browser vs native vs mixed), no bounded-probe rules, no authority/isolation constraints, no native contracts, no conditional loading of browser/DX sections, no blocked-path handling, no regression-before-repair gate, no recheck step, and no defined report/coverage deliverable. An agent reading this cannot determine what to do first, when to stop exploring, how to reproduce or diagnose, or what evidence format to produce. Every concrete decision point the evaluation criteria ask about is absent, so it scores at the floor for completeness and actionability; clarity gets a 2 only because the text is unambiguous as philosophy, even though it offers nothing to follow as steps."},"latency_ms":9080,"stop_reason":"end_turn","usage":{"input_tokens":1292,"output_tokens":494},"cost_usd":0.03762}
+{"config":"qa-workflow","item":"p07","split":"heldout","phase":"old-a","sample":0,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser-specific portion is concrete: modes, six phases with runnable aside scripts, a computable Health Score Rubric, baseline.json schema, and framework hints. However, the document is only one file of the described bundle and constantly defers to unseen contracts ('shared exploratory loop' checkpoint/exact-replay/deadline rules, BROWSER SETUP's access decision, caller's artifact/mixed-report rules, issue-taxonomy.md, qa-report-template.md). Native functional surfaces, isolation, regression-before-repair, diagnosis, recheck-after-repair, and most blocked-path handling are absent or explicitly 'caller-owned', so an agent cannot execute the complete workflow from this text. Some terse directives ('Diff-aware selects scope, not another pass', 'LOOK is not ACT', hardcoded locator e12 in the flow script) reduce clarity. Blocked paths are only partially defined (NEEDS_ASIDE/ASIDE_NOT_RUNNING, missing regression baseline)."},"latency_ms":17673,"stop_reason":"end_turn","usage":{"input_tokens":5046,"output_tokens":1240},"cost_usd":0.11246}
+{"config":"qa-workflow","item":"f08","split":"heldout","phase":"old-b","sample":2,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"This is a mindset essay, not a workflow. It contains no phases, no step sequence, no decision points for selecting browser vs. native vs. mixed surfaces, no bounded-probe limits, no reproduction/diagnosis procedure, no regression-before-repair rule, no recheck step, and no report/coverage output format. None of the required bundle elements are present: no authority ordering, no isolation requirements, no native functional contracts, no conditional browser/DX loading conditions, and no blocked-path handling. An agent reading this would have only aspirational guidance ('think like a user', 'depth beats breadth', 'leave a trail') and no executable instructions or defined deliverables. It may be useful as a preamble, but as the complete workflow it is unfollowable and nonexecutable."},"latency_ms":7428,"stop_reason":"end_turn","usage":{"input_tokens":931,"output_tokens":358},"cost_usd":0.02721}
+{"config":"qa-workflow","item":"a08","split":"heldout","phase":"old-a","sample":1,"status":"ok","value":{"clarity":3,"completeness":2,"actionability":3,"reasoning":"The browser exploration loop (Orient → Explore → Document → Wrap Up) is concrete, with specific $B commands, a per-page checklist, two evidence tiers, and a fully specified health-score rubric, so an agent could run a browser-only QA pass. However the document has notable gaps and inconsistencies: Phase 2 is missing entirely (numbering jumps 1→3); $REPORT_DIR is never defined and 'create output directories' is unspecified; the report template and issue-taxonomy files are referenced but not included; baseline.json has no schema; the Auth and Scope parameters are parsed but never consumed; 'Tiers determine which issues get fixed' implies a repair phase that does not exist. Relative to the stated bundle scope—selecting browser vs native-functional vs mixed surfaces, authority/isolation rules, native contracts, conditional DX loading, blocked paths, adding a regression test before repair, and rechecking—almost nothing is present; this is a browser-only baseline with no native surface handling, no diagnosis/repair/recheck loop, and 'Never read source code' conflicts with the diff-aware mode that maps changed files to routes. Deliverables (report, screenshots, score) are producible, but downstream phases are undefined."},"latency_ms":16660,"stop_reason":"end_turn","usage":{"input_tokens":4371,"output_tokens":1158},"cost_usd":0.10161}
+{"config":"qa-workflow","item":"f06","split":"heldout","phase":"old-a","sample":1,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"The document is a thin shell of pointers to artifacts that are not part of the bundle: the QA Handbook §2.3 checklist, a team-wiki 'primary' environment profile, QA-PROTOCOL-v2 steps P1–P14, a release matrix, the 'usual tracker format', the house S0–S4 severity scale, a fix-forward policy, a deferral process, 'the usual place', a 'standard evidence bundle', and 'the normal channel'. None of these are defined or loaded, so an agent has no executable steps at all. There is no surface-selection logic (browser vs native vs mixed), no probe budget, no reproduction/diagnosis procedure, no regression-before-repair requirement, no recheck, and no defined evidence/coverage output format. The precedence note ('handbook wins, except where the protocol is newer') is self-undermining and gives no way to determine recency. Phase headings exist, but every decision point and deliverable is delegated to tribal knowledge, making the workflow unusable by an autonomous agent."},"latency_ms":10960,"stop_reason":"end_turn","usage":{"input_tokens":776,"output_tokens":595},"cost_usd":0.03751}
+{"config":"qa-workflow","item":"f06","split":"heldout","phase":"new","sample":1,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is a thin shell of pointers to artifacts that are not provided and not described: 'QA Handbook section 2.3', 'team wiki primary profile', 'QA-PROTOCOL-v2' steps P1–P14, 'release matrix', 'usual tracker format', 'house severity scale', 'release policy', 'fix-forward policy', 'deferral process', 'usual place', 'QA owner', 'standard evidence bundle', 'normal channel'. None are defined, so an AI agent cannot determine what a probe is, how deep to explore, what severity S0–S4 mean, what the evidence bundle contains, or where to file anything. The precedence rule ('handbook wins, except where the protocol is newer') is self-undermining and cannot be resolved without version metadata that is absent. Against the stated workflow requirements it is missing essentially everything: no surface selection (browser vs native functional vs mixed), no bounded-probe budgets, no reproduce/diagnose loop, no regression-before-repair rule, no recheck step, no coverage/evidence reporting schema, no authority or isolation rules, no native contracts, no conditional browser/DX loading, no blocked paths. The prose itself is grammatically clear (hence clarity 2 rather than 1), but every decision point defers to tribal knowledge, making it non-executable for an agent."},"latency_ms":11021,"stop_reason":"end_turn","usage":{"input_tokens":1137,"output_tokens":752},"cost_usd":0.04897}
+{"config":"qa-workflow","item":"f08","split":"heldout","phase":"old-b","sample":0,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The document is a well-written mindset essay, not a workflow. It contains no phases, no surface-selection logic (browser vs native vs mixed), no probe bounds, no reproduce/diagnose/regression-before-repair/recheck sequence, no reporting format for evidence or coverage, and nothing about authority, isolation, native contracts, conditional browser/DX loading, or blocked paths. The prose itself is unambiguous, which earns a 2 on clarity, but an agent cannot derive any executable decision or deliverable from it. Every requirement the evaluation rubric names is absent, so completeness and actionability are at the floor."},"latency_ms":7635,"stop_reason":"end_turn","usage":{"input_tokens":931,"output_tokens":439},"cost_usd":0.03126}
+{"config":"qa-workflow","item":"p08","split":"heldout","phase":"old-a","sample":2,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"The functional-surface file is well structured: numbered steps, an explicit isolation gate (check localhost/outbound targets, ask before touching production), a contract-type→probe table, a strict evidence protocol (reset, 60s timeout, save exit/stdout/stderr to evidence/NNN/, inspect durable state, five outcome labels, one re-run to confirm), and a precise report spec with a pass/not-passing rule. Minor ambiguities: 'required contract' is never defined versus optional, blocked-vs-not-run distinction is thin, and cleanup is only mentioned in the report rather than as a step. The main gap is relative to the described bundle: only this one conditional file is present. There is no authority/precedence section, no browser or mixed-surface file, no DX/conditional loading rules, no diagnosis phase, no 'add regression before repair' and recheck steps (this file only proposes regression tests), and no coverage reporting across surfaces. As a standalone functional-surface section it is highly executable; as the complete workflow it is incomplete."},"latency_ms":13716,"stop_reason":"end_turn","usage":{"input_tokens":1779,"output_tokens":839},"cost_usd":0.05974}
+{"config":"qa-workflow","item":"p08","split":"heldout","phase":"new","sample":2,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"This section is tightly written: the four steps (define target/contracts/tools and isolation, map contracts to probes via a concrete table, run probes with reset/timeout/evidence/retry-once confirmation, report with a fixed structure and a strict pass/not-passing rule) are unambiguous and directly executable, and the isolation and blocked-path rules are explicit. Minor gaps in clarity: 'required contract' is never defined, nor when to choose 'inconclusive' vs 'not run', and `report.md` is written in both Step 2 and Step 4 without saying whether to overwrite or append. As a component of the described full bundle, however, it is incomplete: there is no surface-selection logic for browser vs functional vs mixed targets, no authority/conditional-loading statement, no diagnosis phase, and regression tests are only 'proposed' rather than added before repair, with no recheck step after a fix. An agent can produce the stated deliverables (plan, probe list, evidence/NNN, report.md) for a pure functional target, but the repair/regression/recheck loop and mixed-surface handling the overall workflow promises are absent from this file."},"latency_ms":15408,"stop_reason":"end_turn","usage":{"input_tokens":2140,"output_tokens":953},"cost_usd":0.06905}
+{"config":"qa-workflow","item":"p06","split":"heldout","phase":"old-b","sample":0,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"The native worker/job/webhook workflow is well-sequenced and concrete: dirty-tree gate, owned-state isolation with explicit STOP on production/third-party destinations, a contract table with observable effects, a probe loop with per-probe evidence dirs and exploration log, double-replay confirmation, red-before-green regression tests, bounded repair stop conditions (two reverts, 20 fixes, unrelated-file edit), a final regression run, and a fully specified report schema and summary line. Blocked paths (missing regression report, missing service ≠ pass, environment failures reported not fixed) are explicit. Weaknesses: the bundle as presented is only this single file that declares itself 'the whole workflow', so browser, mixed-surface and DX conditional loading, and the surface-selection decision are absent despite the evaluation scope; `--quick` is overloaded as both a mode flag and a fix-tier flag with no disambiguation; regression mode specifies only reading the prior report, not which contracts/issues to replay or how outcomes map; 'highest-risk changed edge' and 'adjacent happy path' are left to judgment; no explicit cleanup step exists even though the report must list owned state removed; and the output filename lacks a run-id to pair with the run-timestamp subdir. Overall an agent can execute this native path and produce the deliverables, but the broader bundle decisions are undefined here."},"latency_ms":17515,"stop_reason":"end_turn","usage":{"input_tokens":2892,"output_tokens":1195},"cost_usd":0.08867}
+{"config":"qa-workflow","item":"f08","split":"heldout","phase":"old-a","sample":0,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"The document is a motivational essay on QA mindset, not a workflow. It contains no phases, no surface-selection logic (browser vs. native vs. mixed), no bounded probe limits, no reproduction/diagnosis steps, no regression-before-repair rule, no recheck procedure, no evidence or coverage reporting format, and nothing about authority, isolation, native contracts, conditional browser/DX loading, or blocked paths. The prose is readable, but an agent has zero executable decisions or deliverable definitions to act on."},"latency_ms":6902,"stop_reason":"end_turn","usage":{"input_tokens":931,"output_tokens":354},"cost_usd":0.02701}
+{"config":"qa-workflow","item":"f07","split":"heldout","phase":"old-b","sample":0,"status":"ok","value":{"clarity":2,"completeness":1,"actionability":1,"reasoning":"The bundle contains only a post-report fix loop and omits every other phase the workflow description promises: no surface selection (browser/native/mixed), no bounded exploration probes, no reproduction/diagnosis steps, no authority or isolation rules, no native contracts, no conditional browser/DX loading triggers, and no blocked-path handling. The steps that do exist are superficially numbered but vague ('any file that looks related', 'if you have time') and encode harmful decisions: unbounded scope creep ('clean up nearby code', cross-module refactors and new dependencies at will), writing the regression test after the fix rather than before, deleting a failing regression test instead of treating it as a signal, rewriting existing tests' expected values to match whatever the fix produces, making re-testing optional, batching commits and recovering with a destructive `git reset --hard`, and a report that lists every issue as 'fixed' regardless of verification. An agent could mechanically follow it, but doing so undermines the stated goals of evidence-based rechecking and coverage reporting, so it is not actionable toward the expected deliverables."},"latency_ms":11683,"stop_reason":"end_turn","usage":{"input_tokens":787,"output_tokens":731},"cost_usd":0.04442}
+{"config":"qa-workflow","item":"f06","split":"heldout","phase":"old-b","sample":1,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"The document is almost entirely indirection to unspecified external artifacts: 'QA Handbook section 2.3', 'team wiki', 'QA-PROTOCOL-v2', 'release matrix', 'usual tracker format', 'house severity scale', 'fix-forward policy', 'deferral process', 'the usual place', 'standard evidence bundle', 'normal channel'. None of these are defined or bundled, so an agent has no executable steps for selecting browser/native/mixed surfaces, bounding probes, reproducing or diagnosing defects, adding a regression before repair, rechecking, or reporting evidence/coverage. Authority, isolation, native contracts, conditional browser/DX loading, and blocked paths are entirely absent. The one stated decision rule ('handbook wins, except where the protocol is newer') is self-undermining and gives no way to determine recency. Severity thresholds (S0/S1 block) are named but the scale itself is undefined. The result is a placeholder outline, not a workflow an agent can follow or produce deliverables from."},"latency_ms":8650,"stop_reason":"end_turn","usage":{"input_tokens":776,"output_tokens":596},"cost_usd":0.03756}
+{"config":"qa-workflow","item":"f06","split":"heldout","phase":"old-b","sample":0,"status":"ok","value":{"clarity":1,"completeness":1,"actionability":1,"reasoning":"The document is a thin shell of pointers to external artifacts that are not part of the bundle: the QA Handbook §2.3 checklist, the team-wiki environment profile, QA-PROTOCOL-v2 (steps P1–P14), the release matrix, the 'usual tracker format', the S0–S4 severity scale, the release policy, the fix-forward and deferral processes, the 'usual place' for results, the QA owner, the 'standard evidence bundle', and the 'normal channel'. None of these are defined or reproduced, so an agent cannot determine what to prepare, which depth to test at, how to classify or record findings, what to fix, or what to deliver. Nothing in the stated evaluation scope is present: no surface-selection logic (browser vs. native vs. mixed), no bounded exploration probes, no repro/diagnosis procedure, no 'regression before repair' rule, no recheck step, no evidence/coverage report schema, no authority or isolation rules, no native contracts, no conditional browser/DX loading, and no handling of blocked paths. The lone precedence rule ('handbook wins, except where the protocol is newer') is itself undecidable without version metadata. Phases are labeled but produce no concrete, executable decisions or deliverables."},"latency_ms":11130,"stop_reason":"end_turn","usage":{"input_tokens":776,"output_tokens":707},"cost_usd":0.04311}
+{"config":"qa-workflow","item":"p08","split":"heldout","phase":"old-b","sample":1,"status":"ok","value":{"clarity":4,"completeness":2,"actionability":4,"reasoning":"As a standalone functional-surface file this is well structured: four ordered steps, a clear isolation gate with a stop-and-ask rule, a contract-type-to-probe table, explicit outcome states (pass/fail/blocked/not run/inconclusive), a deterministic re-run rule for confirming failures, and a precise report.md/evidence/NNN layout with a strict verdict rule. Minor ambiguities remain: 'required contract' is never defined (which contracts are optional?), 'inconclusive' has no entry criteria, and 'pick the rows that apply' is left to judgment. However, measured against the described bundle it is badly incomplete: there is no surface-selection logic (browser vs native vs mixed), no browser or DX section or the conditions under which they load, no authority/permissions section beyond the single production-reach check, and no 'add regression before repair', repair, or recheck phases—the document only *proposes* regression tests and ends at reporting. Blocked paths are handled (mark blocked, continue, report reason), but the agent cannot execute the full reproduce→regress→repair→recheck loop from what is provided. For the API/CLI/job scope it actually covers, an agent could execute it and produce the stated deliverables."},"latency_ms":16966,"stop_reason":"end_turn","usage":{"input_tokens":1779,"output_tokens":1096},"cost_usd":0.07259}
+{"config":"qa-workflow","item":"p08","split":"heldout","phase":"new","sample":0,"status":"ok","value":{"clarity":4,"completeness":3,"actionability":4,"reasoning":"The functional-surface file is well structured: four numbered steps, explicit isolation gate (check localhost/outbound before running, stop-and-ask for prod/third-party), a concrete contract-type→probe table, a per-probe loop with reset/timeout/evidence/outcome vocabulary, a one-rerun confirmation rule, and a defined report.md with verdict semantics (no health score; any fail/blocked/inconclusive required contract = not passing). Minor ambiguities: 'required' vs optional contracts is never defined, evidence/NNN numbering is not tied to contract IDs, and 'reset to the planned initial state' has no stated mechanism. On completeness, the document covers only the native functional path and stops at proposing regression tests; it has no diagnosis step, no 'add a regression before repair', no repair/recheck phase, no surface-selection logic for browser/mixed targets, and no statement of authority/precedence relative to other bundle files or how blocked paths are escalated beyond marking them. As a standalone section it is executable and yields report.md plus evidence/, but it does not deliver the full reproduce→regression→repair→recheck cycle described for the overall workflow."},"latency_ms":17347,"stop_reason":"end_turn","usage":{"input_tokens":2140,"output_tokens":1056},"cost_usd":0.0742}

test/fixtures/judge-calibration/qa-workflow/runs/comparison-1.stop.json

@@ -0,0 +1 @@
+{"stopped":null}