From d082eccc50d26f8fb3d225be4184f15b2efdcd39 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Thu, 8 Oct 2026 03:19:18 -0700 Subject: [PATCH] v1.91.52.0 fix(qa): annotations limits shape; qa-quick does less work in the same 300 s budget (#3081) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * test(qa-quick): run the fixture in a committed git repository /qa runs inside a repository: its clean-tree check and the evidence recorder's revision come from git. The qa-quick working directory was a bare temp dir, so all 15 CI transcripts read (runs 37374409928 to 37744583272) hit 'revision is required when git rev-parse HEAD is unavailable' on the first materialize and spent a retry (6-46 s) plus a learnings-log of the pitfall. qa-only-no-fix already commits its fixture. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(qa-quick): scope Quick to the fixture page, not its 404 links basic.html's /page1 and /page2 links 404 on the test server. With the links in scope, the 15 CI transcripts spent 55-75 s of the 30 s caller probe window and its wrap-up navigating, screenshotting and checkpointing those 404s, then triaging and reporting them as issues. The case proves /qa Quick completes end to end (fallback browser, deadline guard, checkpoint, evidence materialize, report); link-following and defect detection are covered by the qa-b6/b7/b8 and qa-fix-loop cases. The links are now listed as untested coverage. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(qa): say that annotations.json limits is a non-empty string array exploratory.md §4 gave the annotations shape as {evidence: [...], limits} with 'checkpoints in limits', but never said what type limits is. The materialize validator (lib/qa-evidence.ts) wants a non-empty array of strings. In 11 of 15 qa-quick CI transcripts the agent's first annotations.json wrote limits as an object, failed with 'need limits (non-empty string array)', and spent a retry. The section now shows the shape with a one-line example and says it is never an object. Shared by /qa and /qa-only through the QA_EXPLORATORY resolver. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * v1.91.52.0 fix(qa): annotations limits shape; qa-quick does less work in the same 300 s budget Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- CHANGELOG.md | 32 ++++++++++++++++++++++++++++++ VERSION | 2 +- agents-digest/gstack-AGENTS.md | 2 +- package.json | 2 +- qa-only/sections/exploratory.md | 2 +- qa/sections/exploratory.md | 2 +- scripts/resolvers/qa.ts | 2 +- test/skill-e2e-qa-workflow.test.ts | 9 ++++++++- 8 files changed, 46 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index de8be6655..947ef6f2e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,37 @@ # Changelog +## [1.91.52.0] - 2026-10-08 + +**`/qa` says exactly what shape `annotations.json` takes, and the `qa-quick` gate eval stops timing out at its 300 s budget.** + +Before writing its report, `/qa` records an `annotations.json` file and runs `gstack-qa-evidence materialize` on it. The section that describes the file listed its fields as `{evidence: [...], limits}` and told browser-only runs to put "checkpoints in limits", but it never said what type `limits` is. The validator only accepts a non-empty array of strings. In 11 of 15 recent `qa-quick` CI transcripts, the agent's first attempt wrote `limits` as an object, the call failed, and the agent had to retry. Anyone running `/qa` or `/qa-only` could hit the same retry. + +`qa-quick` runs on every PR that touches `browse/src`. It passed 13 of 15 recent CI runs in 206-280 s, and the other 2 hit the 300 s session timeout. Two parts of the case's own setup were adding work. Its fixture directory was not a git repository, so in all 15 runs `materialize` first failed with "revision is required when git rev-parse HEAD is unavailable". Its prompt also put the fixture's `/page1` and `/page2` links in scope. Both links return 404 on the test server, so each run spent part of its 30 s probe window, and its wrap-up, investigating and reporting them. The session budget, the thresholds and the `/qa` workflow are unchanged. + +### The numbers that matter + +Diagnostic `ship-measure` trials in the CI image (Claude Code 2.1.292, Bun 1.4.2, `--jobs 2`, `CI=true`). They never change a recorded verdict. + +| `qa-quick` | Before (CI, 15 runs) | After (10 trials) | +|---|---|---| +| Passed | 13 of 15 | 10 of 10 | +| Pass duration, median (range) | 252 s (206-280 s) | 187 s (155-256 s) | +| Mean cost per trial | $1.75 | $1.46 | +| Runs with a failed `materialize` call | 15 of 15 | 0 of 10 | + +### What this means for you + +`/qa` and `/qa-only` now get `annotations.json` right on the first try, which saves a failed tool call near the end of a run. A browse PR's `qa-quick` check should no longer time out. Its slowest measured trial (256 s) sits right at the 85% headroom line (255 s), so the weekly gate's headroom alarm may still flag the case now and then. + +### Itemized changes + +#### Fixed +- `qa/sections/exploratory.md` §4 (shared by `/qa` and `/qa-only` through the `QA_EXPLORATORY` resolver) shows `limits` as `["checkpoint 001"]` and says it is a non-empty string array, never an object. That matches what `lib/qa-evidence.ts` already enforces. + +#### For contributors +- `qa-quick` commits its fixture directory to git before the session starts, as `qa-only-no-fix` already does. `/qa`'s clean-tree check and the evidence recorder's revision now work the way they do in a real repository. +- `qa-quick` limits Quick scope to the fixture page itself: its load, console health and rendered content. The two same-origin links are listed as untested coverage. The qa-b6/b7/b8 and qa-fix-loop cases still cover link following and defect detection. + ## [1.91.47.0] - 2026-10-08 **`/careful` and `/guard` now check PowerShell commands, and `/freeze` now covers notebook edits.** diff --git a/VERSION b/VERSION index 7cf6608c2..7d15e052e 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.91.47.0 +1.91.52.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index 6450a5161..23bad11ab 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.91.47.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.91.52.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/package.json b/package.json index 56b36294d..64b1b546e 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.91.47", + "version": "1.91.52", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", diff --git a/qa-only/sections/exploratory.md b/qa-only/sections/exploratory.md index 9322d695a..f1bec7d79 100644 --- a/qa-only/sections/exploratory.md +++ b/qa-only/sections/exploratory.md @@ -100,7 +100,7 @@ with their failing contract and expected assertion; never create tests or freeze ## 4. Final report Use the surface report template; link each checkpoint. Separate browser scores, functional outcomes and proposed/executed tests. -Write PROBE_DIR/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits} (browser-only: evidence [], checkpoints in limits); before Markdown `bun EVIDENCE_TOOL materialize PROBE_DIR annotations.json` (fills observed/metadata; prints reportLinks; runs once per PROBE_DIR); you classify. Annotate every safe capture, including failures/replays: an omitted capture is withheld and keeps the verdict inconclusive. Classify a capture taken before an input change `superseded`; it closes when the same command reran on current inputs. Disclose withheld/incomplete evidence. +Write PROBE_DIR/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits: ["checkpoint 001"]} (limits: non-empty string array, not an object; browser-only: evidence []); before Markdown `bun EVIDENCE_TOOL materialize PROBE_DIR annotations.json` (fills observed/metadata; prints reportLinks; runs once per PROBE_DIR); you classify. Annotate every safe capture, including failures/replays: an omitted capture is withheld and keeps the verdict inconclusive. Classify a capture taken before an input change `superseded`; it closes when the same command reran on current inputs. Disclose withheld/incomplete evidence. Evidence is invocation-local. Missing prerequisites/expectations/observations, timeouts and refusal never pass. Pass requires all required current-input contracts to pass with no required remainder. diff --git a/qa/sections/exploratory.md b/qa/sections/exploratory.md index 0d309e6ca..41a472ba1 100644 --- a/qa/sections/exploratory.md +++ b/qa/sections/exploratory.md @@ -85,7 +85,7 @@ Never freeze buggy output, weaken tests or delete valid red tests. ## 4. Final report Use the surface report template; link each checkpoint. Separate browser scores, functional outcomes and proposed/executed tests. -Write PROBE_DIR/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits} (browser-only: evidence [], checkpoints in limits); before Markdown `bun EVIDENCE_TOOL materialize PROBE_DIR annotations.json` (fills observed/metadata; prints reportLinks; runs once per PROBE_DIR); you classify. Annotate every safe capture, including failures/replays: an omitted capture is withheld and keeps the verdict inconclusive. Classify a capture taken before an input change `superseded`; it closes when the same command reran on current inputs. Disclose withheld/incomplete evidence. +Write PROBE_DIR/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits: ["checkpoint 001"]} (limits: non-empty string array, not an object; browser-only: evidence []); before Markdown `bun EVIDENCE_TOOL materialize PROBE_DIR annotations.json` (fills observed/metadata; prints reportLinks; runs once per PROBE_DIR); you classify. Annotate every safe capture, including failures/replays: an omitted capture is withheld and keeps the verdict inconclusive. Classify a capture taken before an input change `superseded`; it closes when the same command reran on current inputs. Disclose withheld/incomplete evidence. Evidence is invocation-local; /ship reruns once per invocation. Missing prerequisites/expectations/observations, timeouts and refusal never pass. Pass requires all required current-input contracts to pass with no required remainder. diff --git a/scripts/resolvers/qa.ts b/scripts/resolvers/qa.ts index d1e0843e1..32ad4524c 100644 --- a/scripts/resolvers/qa.ts +++ b/scripts/resolvers/qa.ts @@ -147,7 +147,7 @@ Never freeze buggy output, weaken tests or delete valid red tests.`} ## 4. Final report Use the surface report template; link each checkpoint. Separate browser scores, functional outcomes and proposed/executed tests. -Write PROBE_DIR/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits} (browser-only: evidence [], checkpoints in limits); before Markdown \`bun EVIDENCE_TOOL materialize PROBE_DIR annotations.json\` (fills observed/metadata; prints reportLinks; runs once per PROBE_DIR); you classify. Annotate every safe capture, including failures/replays: an omitted capture is withheld and keeps the verdict inconclusive. Classify a capture taken before an input change \`superseded\`; it closes when the same command reran on current inputs. Disclose withheld/incomplete evidence. +Write PROBE_DIR/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits: ["checkpoint 001"]} (limits: non-empty string array, not an object; browser-only: evidence []); before Markdown \`bun EVIDENCE_TOOL materialize PROBE_DIR annotations.json\` (fills observed/metadata; prints reportLinks; runs once per PROBE_DIR); you classify. Annotate every safe capture, including failures/replays: an omitted capture is withheld and keeps the verdict inconclusive. Classify a capture taken before an input change \`superseded\`; it closes when the same command reran on current inputs. Disclose withheld/incomplete evidence. Evidence is invocation-local${reportOnly ? '.' : '; /ship reruns once per invocation.'} Missing prerequisites/expectations/observations, timeouts and refusal never pass. Pass requires all required current-input contracts to pass with no required remainder. diff --git a/test/skill-e2e-qa-workflow.test.ts b/test/skill-e2e-qa-workflow.test.ts index d2ebc644b..533963bce 100644 --- a/test/skill-e2e-qa-workflow.test.ts +++ b/test/skill-e2e-qa-workflow.test.ts @@ -49,6 +49,13 @@ describeIfSelected('QA skill E2E', ['qa-quick'], () => { // Create report directory fs.mkdirSync(path.join(qaDir, 'qa-reports'), { recursive: true }); + + for (const args of [['init', '-b', 'main'], ['config', 'user.email', 'test@test.com'], ['config', 'user.name', 'Test'], + ['add', '.'], ['commit', '-m', 'initial']]) { + const git = spawnSync('git', args, { cwd: qaDir, stdio: 'pipe', timeout: 5000 }); + if (git.error) throw git.error; + if (git.status !== 0) throw new Error(`git ${args.join(' ')} failed: ${git.stderr}`); + } }); afterAll(() => { @@ -68,7 +75,7 @@ qa is a carved skill: when SKILL.md tells you to Read ~/.claude/skills/gstack/qa Skip the preamble bash block, lake intro, telemetry, and contributor mode sections — go straight to the QA workflow. Run a Quick-depth QA test on ${testServer.url}/basic.html -This fixture is one static page with two same-origin navigation links (/page1, /page2) and one external link; Quick scope is that page and its two same-origin links. +This fixture is one static page with two same-origin navigation links (/page1, /page2) and one external link. Quick scope is that page only: its load, console health and rendered content. Do not follow or probe its links; list them as untested coverage. As the caller, I set a 30-second total probe limit for this run (shorter than Browser Quick's own budget), so start the deadline with SECONDS=30. Write concise checkpoints and a concise report, retaining required evidence, scores and untested-coverage fields; cite evidence IDs instead of repeating commands. Do NOT use AskUserQuestion — run Quick tier directly.