From 625debd7eb110d40e8a5e44cc0414e645cfe68ab Mon Sep 17 00:00:00 2001 From: garrytan Date: Thu, 1 Oct 2026 18:58:52 +0000 Subject: [PATCH] fix(qa): browser-only runs materialize an empty evidence list with checkpoints in limits, matching /qa-only --- CHANGELOG.md | 2 +- qa-only/sections/exploratory.md | 2 +- qa/sections/exploratory.md | 2 +- scripts/resolvers/qa.ts | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a84b71bd3..00fb4c165 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,7 +37,7 @@ Run `bun run typecheck` and `bun run typecheck:test` before you push; both are f #### Fixed - `$B connect --supervise` respawned with a block-scoped env that no longer existed, so every restart threw and the supervisor gave up after five tries. The headed env is one helper used by connect and respawn, and the loop has behavioral tests. - Compiled `/cso` installs called an unimported `join` when launching the assertion-witness child, breaking runtime-tested witnessing for every installed user. -- Browser-only `/qa` runs were told to materialize an evidence file that only accepts functional captures; they now skip that step and link their checkpoints. The fix loop had spent its last minute on it and timed out. +- Browser-only `/qa` runs had no stated way to build the evidence file, whose rows only accept functional captures; the shared rule now says to materialize an empty evidence list with the checkpoints named in limits, matching `/qa-only`. The fix loop had spent its last minute on it and timed out. - `/office-hours` asks its goal question unless the user already chose a mode, then reads that mode's section before its first question; skipping both produced forcing questions with an empty recommendation. `/design-consultation` asks the memorable-thing question on its own after Q1 instead of packing it into Q1's call. - Free-form eval judges (docs, outcome, posture) could return JSON broken by an unescaped quote in their reasoning; they now use structured output. - `/qa` checkpoint receipts now print the report link for their `exploration-NNN.json` file; reports had been linking `.qa-evidence/NNN` capture folders as checkpoints instead. diff --git a/qa-only/sections/exploratory.md b/qa-only/sections/exploratory.md index 561ff0e4e..10d4df26d 100644 --- a/qa-only/sections/exploratory.md +++ b/qa-only/sections/exploratory.md @@ -96,7 +96,7 @@ with their failing contract and expected assertion; never create tests or freeze ## 4. Final report Use the surface report template; link each checkpoint. Separate browser scores, functional outcomes and proposed/executed tests. -With captures, write R/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits} and before Markdown `bun Q materialize R annotations.json` (fills observed/metadata; prints reportLinks); you classify. No captures (browser): skip both. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence. +Write R/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits} (browser-only: evidence [], checkpoints in limits); before Markdown `bun Q materialize R annotations.json` (fills observed/metadata; prints reportLinks); you classify. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence. Evidence is invocation-local. Missing prerequisites/expectations/observations, timeouts and refusal never pass. Pass requires all required current-input contracts to pass with no required remainder. diff --git a/qa/sections/exploratory.md b/qa/sections/exploratory.md index 9db170c3b..7a1c330c7 100644 --- a/qa/sections/exploratory.md +++ b/qa/sections/exploratory.md @@ -81,7 +81,7 @@ Never freeze buggy output, weaken tests or delete valid red tests. ## 4. Final report Use the surface report template; link each checkpoint. Separate browser scores, functional outcomes and proposed/executed tests. -With captures, write R/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits} and before Markdown `bun Q materialize R annotations.json` (fills observed/metadata; prints reportLinks); you classify. No captures (browser): skip both. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence. +Write R/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits} (browser-only: evidence [], checkpoints in limits); before Markdown `bun Q materialize R annotations.json` (fills observed/metadata; prints reportLinks); you classify. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence. Evidence is invocation-local; /ship reruns once per invocation. Missing prerequisites/expectations/observations, timeouts and refusal never pass. Pass requires all required current-input contracts to pass with no required remainder. diff --git a/scripts/resolvers/qa.ts b/scripts/resolvers/qa.ts index 813ecb8d7..35dd75e37 100644 --- a/scripts/resolvers/qa.ts +++ b/scripts/resolvers/qa.ts @@ -143,7 +143,7 @@ Never freeze buggy output, weaken tests or delete valid red tests.`} ## 4. Final report Use the surface report template; link each checkpoint. Separate browser scores, functional outcomes and proposed/executed tests. -With captures, write R/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits} and before Markdown \`bun Q materialize R annotations.json\` (fills observed/metadata; prints reportLinks); you classify. No captures (browser): skip both. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence. +Write R/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits} (browser-only: evidence [], checkpoints in limits); before Markdown \`bun Q materialize R annotations.json\` (fills observed/metadata; prints reportLinks); you classify. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence. Evidence is invocation-local${reportOnly ? '.' : '; /ship reruns once per invocation.'} Missing prerequisites/expectations/observations, timeouts and refusal never pass. Pass requires all required current-input contracts to pass with no required remainder.