mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-13 03:48:06 +01:00
cso-evaluator-support
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Garry Tan
|
55fbe77195 |
v1.91.24.0 test: audit wave — PR evals keep finished work, honest ceilings, EVAL_POLICY v2, faster free suite (#3031)
* chore(scripts): retire hollow and legacy paid package scripts behind one-release stubs (W8a, DX-4, CEO-31) test:codex*, test:evals*, test:e2e*, test:gate, test:periodic, eval:bg, eval:bg:all, eval:flake-rank, eval:watch and test:audit now print their replacement and exit 1 via scripts/retired-command.ts. CONTRIBUTING.md carries the same table; test/retired-commands.test.ts pins both. Eval CLI empty-state hints point at eval:bg:pr. * chore: delete dead update-readme-throughput, eval-watch, orphan fixtures and QA_FUNCTIONAL_INPUTS (W8b, CEO-24) Retired with named behavior: - update-readme-throughput.ts + readme-throughput test: replaced a README anchor that no longer exists from a JSON file that was never produced; the script was a no-op. Guarantee withdrawn (nothing to protect). - eval-watch.ts + its dashboard tests: rendered ~/.gstack-dev/e2e-live.json and _partial-e2e.json, written only by unsharded runs, so it showed nothing for every supported (sharded) entry point. Guarantee withdrawn; live progress is the gstack-detach log or gh run watch. Manual-review rendering stays covered by eval-list and eval-summary tests. - Orphan fixtures (autoplan-pending-artifact-ae.json, autoplan-existing-app/, autoplan-sign-in-ui/, parity baselines v1.47.0.0/v1.57.7.0/v1.69.1.0): no test, touchfile or script reads them; superseded baselines stay in git history. - QA_FUNCTIONAL_INPUTS: exported, never imported. * refactor(test-free): delete dead WORKER_HOSTILE list and buildShardArgs parallel mode Neither had a production caller: full-suite mode runs shard processes, so the worker quarantine was inert and --parallel was never passed (W8b). * docs(test-value): narrow prompt-byte contract to machine-read tokens (D2) Define prompt-byte contract as machine-read tokens (markers, field names, command lines, enum values, file paths, security-boundary strings) and add a sentence-pin glossary entry: convert to structural checks or delete when a behavioral eval covers the behavior. * test(ratchet): count free tests that slice a paid test's source A free test that binds readFileSync(<paid test file>) and cuts it with .slice() tests the paid harness's code out of context even when it imports a lib/ symbol. Record the 15 existing files with reasons; new ones fail with fix text. Self-test covers positive and negative shapes. * test(browse): un-quarantine compare-board; all 16 board tests pass headless (W8d, CEO-24) The GSTACK_COMPARE_BOARD_TESTS opt-in hid a headless mismatch that no longer reproduces: all 16 tests pass on Bun 1.4.0 with the bundled headless Chromium, solo, repeated, and beside other in-process browser files. Protected behavior (the comparison-board DOM contract the plan-design-review/office-hours/design-consultation feedback loop polls) is checked on every free run again. * test: replace silent existsSync early returns with explicit skips or assertions (W8d) - browse/test/build.test.ts: server-node.mjs checks skip only when the bundle is absent and GSTACK_EXPECT_BINARIES is unset; the free CI lane and test:ubicloud build it and set the flag, so a dropped build step fails with the build command instead of passing with no assertions. - gstack-update-check: VERSION is tracked; the guard could only hide a bug. - gen-skill-docs host leakage/frontmatter checks assert the host tree exists (the module-level render always produces it). - cso-docker staged-runtime test uses test.skipIf so reports show it as skipped, not passed. * test(ship): convert ship-workflow-clarity sentence pins to structural checks Add test/helpers/prompt-structure.ts (between, expectTokens, expectAbsent, expectOrdered, expectMentions) with fix text on failure. ship-workflow-clarity now pins headings, step order and machine-read tokens (commands, fields, enum and status strings); safety rules are meaning-level keyword checks. Wording is judged by the 'ship/SKILL.md workflow' judge in skill-llm-eval. Sentence pins: 90 -> 0. * ci: add platform-qualification lane for opt-in ML, gitleaks and Swift tests (W8d, CEO-24) security-bench + security-live-playwright L4 (SECURITY_BENCH), the real gitleaks 8.30.1 memory-ingest cases (GSTACK_TEST_GITLEAKS) and the DebugBridge Swift build invariants (GSTACK_TEST_SWIFT) self-skip in every lane today. No scheduled lane has their prerequisites, so per CEO-24 they move to an owned dispatch + quarterly workflow instead of being deleted. Each job was run locally (ML 10 pass, gitleaks 130 pass; Swift needs macOS). test/platform-qualification-workflow.test.ts pins gate variable, job and file together. * fix(scripts): capture-baseline and preflight-agent-sdk no longer act on --help or typos (W8e) capture-baseline parses its flags strictly: --help prints usage, an unknown or valueless flag exits 2, and only a valid invocation writes a fixture (--help used to write test/fixtures/parity-baseline-current.json). preflight-agent-sdk gains --help, rejects unknown flags, and runs the live SDK query (API spend) only with --live. * test(evals): register the three keyless periodic paid files with case ids (W1b) skill-e2e-auq-consistency, skill-e2e-auq-verbose-vs-carved-ab and codex-e2e-recommendation-substance now carry touchfiles, a periodic tier and kind rule (one trial, unchanged), so editing one selects its own case instead of restoring the full PR gate. The verbose A/B case also registers the vendored pre-cut fixture it reads, and review-army-migration-safety registers the SQL fixture it reads. * feat(evals): narrow PR fallback for free-only inputs (W1a, CEO-13/20, ENG-13) The PR profile now exempts the duration seeds, every free-lane workflow (explicit list; inventory test fails on an unclassified workflow, .yaml and workflow_call included), scripts/ubicloud/** and fixtures declared in the explicit FREE_FIXTURES map. Each remaining unknown file carries a label and fix (needs touchfile entry / add to FREE_ONLY_PR_FILES / real unknown dependency). evals.yml itself still restores the full gate. * ci: delete ci-image.yml; the eval workflows already build and push the same content-hash tag (W8f) evals.yml, evals-periodic.yml and evals-marathon.yml each compute the identical hashFiles() tag and build/push it (with :buildcache and :latest) when missing, so ci-image's push trigger and weekly cron were no-ops (30-day median 0.5 min, 36/36 success). ci-image-tag-binding now pins the three eval workflows. * ci(evals): key periodic/marathon concurrency on ref+event; branch periodic dispatch skips the gate census (W2a, ENG-9, CEO-04) * refactor(paid): drop ~160 unused imports; leaf paid-types/paid-select modules break the paid-* import cycle (W8c) * ci(quality-gate): per-PR concurrency, free runner, drop unscoped bun audit (W2a, W8f) - Concurrency keys on pull_request.number || run_id, so back-to-back main pushes no longer cancel each other's credential scan. - ubuntu-24.04 instead of a billed 8-vCPU Ubicloud runner for a ~1 min job. - bun audit --audit-level=critical ran on every PR regardless of diff, so a newly published advisory turned unrelated PRs red, with its own ignore policy. dependency-review (lockfile PRs, fail >= high) and the weekly OSV sweep remain the dependency gates. * chore(test-free): refresh the free duration seed from the audit's Ubicloud recording Adds the 19 files merged after the last refresh, including the 115 s claude-overlay-setup that packed as ~1 s and doubled a shard (W6a). The seed is re-recorded on the final tree at the end of this wave. * ci: move tiny ubicloud-standard-2 jobs to free ubuntu-24.04 runners (W8f) actionlint, skill-docs, dependency-review, version-gate and pr-title-sync are sub-minute jobs; GitHub-hosted runners are free for this public repo. * feat(test-free): pessimistic unseeded packing, seed growth ratchet and CI health summary - Unseeded files pack at the seed's p99 instead of p75 (W6a). - --ci-plan warns in the job summary past 5 unseeded files; --ci-verify lists shards that outran their prediction by 1.5x (and 30 s) plus every flaky pass absorbed by the serial retry (W6a, W7a job summary). - test/free-seed-ratchet.test.ts fails when a seeded file over 60 s is not on scripts/free-test-seed-allowlist.json, when an entry is stale, and when the allowlist grows versus its merge-base copy unless the new entry is an annotated exception with a todo (CEO-10, DX-7). free-plan runs it with full history so the shrink check is real in the required lane. - Every message names the fix command and docs anchor (DX-6). * feat(evals): state PR coverage mode, counts, reuse and fallback files in the summary and comment (CEO-13) * test: make the tracked-tree skill-doc freshness check local-only; tidy skill-docs header (W8f) Every free CI shard regenerates the tracked SKILL.md files before running tests, so the --dry-run freshness assertion compared the regenerated tree with itself and could not fail there. It now skips under GitHub Actions; skill-docs.yml's git diff --exit-code is CI's freshness gate. Removes a stale concurrency comment in skill-docs.yml. * ci(free-tests): install zsh so the bash+zsh portability arms run (W6f) ~42 zsh-arm tests (aside-probe-shell, qa-browser-preservation, generated-docs-fences) skipped in CI because zsh is not in the runner image. All 141 tests in those files pass with zsh present. * feat(evals): executors write negative receipts at the execution boundary (CEO-25, ENG-6) A completed single-trial rule/judge shard that did not pass now writes its negative receipt beside the pass receipts in the slice artifact, so a cancelled run's FAIL still blocks an older PASS when its report never runs. Receipts merge negatives first and order by (run attempt, completedAt); a same-second FAIL/PASS tie resolves to FAIL. The runner call site is a one-line hunk in scripts/test-paid-shards.ts. * docs(cso): state that contained runtime/scanner execution is unavailable today The scanner catalog is empty while its build inputs await review and the runtime catalog holds no qualified profile until the private evaluator exists, so /cso runs static assessment only. AGENTS.md, README and docs/cso-release-qualification.md now say so; docs/cso-scanners.md already did. * test(helpers): expectMentions splits on sentence ends only Clauses joined by ';' or ':' are one rule; splitting there made a reworded rule fail for punctuation. * test(plan-ceo): convert skill-ceo-section-ordering sentence pins Safety and authority rules (never/do not/stop/only/without/wait/approval) are meaning-level expectMentions checks; other sentence pins are dropped. Headings, labels, option rows, placeholders and machine-read lines (auto-decide annotation, Mode:/approved decisions format, STATUS/SOURCE fields, CROSS-MODEL marker, gstack-ceo-mode-handoff) stay as tokens; ordering checks unchanged. Behavior is covered by the periodic PTY case in skill-e2e-plan-ceo-review-section-loading.test.ts. Sentence pins: 210 -> 23. * test(helpers): expectMentions treats wrapped lines as one sentence Templates hard-wrap prose; only sentence ends and blank lines separate rules. * test(gen-skill-docs): convert sentence pins to meaning checks or drop them Safety/authority sentences become expectMentions keyword checks; other English sentences are dropped (wording is judged by the per-skill workflow judges in skill-llm-eval and the E2E cases in touchfiles-data). Commands, config keys, markers (PLAN MODE EXCEPTION, NO UNRESOLVED DECISIONS, Gate outcome), field names and alternation-based meaning regexes stay. Removed the now-empty 'contains opt-in AskUserQuestion text' test (only pinned a sentence; the resolver's AskUserQuestion wiring is covered by the codex second-opinion tests in the same describe). Lane E's existsSync hunks are untouched. Sentence pins: 216 -> 65 (remaining lines carry machine tokens). * ci(windows): shard the Windows lane via --ci-plan/--ci-run/--ci-verify; move native campaigns to native-qualification.yml - windows-free-tests.yml: a Linux plan job curates and packs the Windows-safe files into 6 shards, one strict serial shard per windows-latest job, and a Linux aggregate verifies exact coverage and every result (8.1 min on one runner today). --windows-only now combines with the CI modes; the plan binds the curated file set (W6e). - Every Windows plan and --list run prints the curated count with its top exclusion reasons in the job summary; a free ratchet fails when the curated subset drops below its floor, with the fix (W6e). - The four Dia/cookie diagnostic dispatch campaigns move to the dispatch-only native-qualification.yml with one exclusive `mode` choice input, so their red dispatch runs no longer read as a red PR lane (W8f). Their workflow pins move to test/windows-native-workflows.test.ts; the boolean-exclusivity guard is replaced by the choice input. * feat(paid): per-slice CI job ceilings and an in-process slice deadline with not_run/hung outcomes and checkpoints (W2c, ENG-2) * fix(pass-rates): segment trial history strictly by EVAL_POLICY version Reader-compatibility floor for EVAL_POLICY v2 (ENG-14/CEO-35/DX-12). Only trials under the reader's own policy version feed labels and alarms; older policy trials stay visible but are never scored or used as a drift baseline; trials from a newer policy are ignored with a printed count and fix text. Reverting the later v2 writer commits leaves a reader that cannot pool v2 records into v1 series (the pre-PR reader at |
||
|
Garry Tan
|
df89475b17 |
v1.91.11.0 refactor: one state-root rule, browse route table, shared shard engine, PTY harness split, MECE review resolvers (#3002)
* refactor(resolvers): split review.ts into MECE resolver modules (pure move) Move every function from scripts/resolvers/review.ts, unchanged, into: - review-dashboard.ts: review dashboard, plan-file review report - plan-gates.ts: approval check, exit-plan-mode gate, plan-file discovery, plan-completion audit/gate (ship + review), plan verification exec - spec-review.ts: both spec review loops, benefits-from, anti-shortcut clause - outside-voice-steps.ts: Codex second opinion, adversarial step, Codex plan review, Codex doc review, disabled-outside record - review-scope.ts: scope drift, cross-review dedup, shared-code reuse review.ts is deleted; index.ts imports the new modules. gen-skill-docs output is byte-identical for every host (--host all). Test imports and source-path references are re-pointed; the two source-text report/gate tests in gen-skill-docs.test.ts become behavioral renders across every consuming skill and host. All 46 touchfile entries that named review.ts now name all five modules, guarded by a recorded selection golden. * test(browse): black-box auth matrix for every server route and both surfaces Drives buildFetchHandler fetchLocal/fetchTunnel with no token, wrong token, root token, scoped token and the SSE cookie for all 33 routes, plus unmatched paths and wrong methods. Denials assert today's exact status, body and content type; allowed credentials assert the handler was reached. Written against the unchanged if-chain server so the W3 route-table refactor must keep it green. * refactor(shard-engine): move scripts/test-strict-output.ts to scripts/lib/shard-engine.ts The shared shard engine grows from the existing strict-output module (runShardChild, killProcessGroup, signal forwarding, strict classifier). scripts/test-strict-output.ts stays as a re-export so existing importers, mock.module paths and the strict-output/run-shard-child tests are unchanged. The engine inherits the global touchfile entry; the free runner's CLI-routing fixture copies the new module. * refactor(resolvers): decompose the three >150-line review resolvers (output-neutral) Split generateAdversarialStep, generateCodexPlanReview and generatePlanCompletionAuditInner into per-section helpers whose template literals are copied verbatim, so every function in the new modules is at or under 150 lines. gen-skill-docs output is byte-identical for every host (--host all, compared against |
||
|
Garry Tan
|
dcaea52800 |
v1.91.7.0 feat: add functional QA and pre-publication docs checks (#2983)
* feat: add surface-aware exploratory QA and ship documentation gates * test: preserve delegated QA setup authority after main integration * fix(qa): clarify exploration order and preserve report artifacts * test(qa): follow the shared setup reference directly * refactor(ship): make verification and recovery routes explicit * test(ship): align evidence and review guards with explicit routes * fix(workflows): clarify ship recovery and functional QA evidence * fix(workflows): clarify approval recovery and full QA coverage * refactor(workflows): order review transactions and clarify ship state * fix(ship): clarify final verification and fail closed at publication * fix(evals): attribute native atomic documentation writes * fix(ship): clarify recovery and documentation lifecycle guidance * fix(test): preserve observed native placeholder styling in CI * fix(codex): report watchdog timeouts without a process-exit race * Checkpoint functional QA implementation and workflow validation repairs * Fix documentation and shared-review fixture contracts * docs: clarify judge reuse and evaluation supervision * test: align review evidence and selected case contracts * test: verify append-only documentation checkpoints and recovery * fix: qualify QA workflows and CI validation repairs * fix: launch shared-libs fixture scripts on Windows * fix: qualify QA deadlines, fixture isolation, and shard cleanup * fix: preserve qualified QA and cancellation repairs * fix: enforce functional fixture authority and share strict event decoding * fix: retain free-test evidence and explain recovery * fix: reject malformed native evidence after decoder consolidation * test: use reliable capture for telemetry privacy filters * test: refresh measured quick coverage and document validation costs * Fix native fixture receipts and preserve VM validation evidence * Align negative judge controls with upstream clarity policy * Fix report-only QA preparation and public evidence handling * Clarify QA-only preparation and current-report preservation * Stream Ship quality judgments with an explicit 64k response contract * Validate compact judge reasoning locally with supported wire schema * Align functional QA fixture instructions with evidence acceptance * Bind native browser diagnostics to execution evidence and align review verdicts * Preserve native diagnostic line boundaries * Serialize functional QA evidence from native captures * Keep large QA evidence fixture payload out of Windows argv |
||
|
Garry Tan
|
7b534d3e90 |
v1.90.2.0 perf: halve local free-suite time and preserve coverage (#2972)
* perf: remove repeated test work and preserve AUQ execution budgets * fix: validate native evaluation fixture evidence at its actual boundaries * fix: clarify deployment approval and recovery state transitions * chore: document coverage and release v1.90.2.0 * test: preserve Windows scheduling and native no-change consent * test: recognize verified reads through fixture symlinks * test: isolate alias-name installation from runtime assets |
||
|
Garry Tan
|
730a1017d1 |
v1.89.1.0 fix: remove continuous checkpoint commits (#2970)
* v1.89.1.0 fix: remove continuous checkpoint commits and repair validation blockers * fix: clarify shipping and engineering review recovery * fix: interpret native no-change review descriptions * test: separate descendant readiness from timeout delivery |
||
|
Garry TanandOpenAI Codex
|
06ed920a97 |
v1.89.0.0 feat: add shared-code extraction audit (#2925)
* feat: bind shared-code review advice to source and branch * feat: add shared-code extraction audit and scoped review checks * test: recognize complete source reads and explicit coverage legends * chore: bump version and changelog (v1.88.0.0) Co-Authored-By: OpenAI Codex <noreply@openai.com> * test: capture native review questions and retain public evidence Capture the actual first public native question with strict ownership and display matching. Preserve terminal failures and raw evidence, and retain SDK completion checks. * test: recognize verified review evidence and complete fixtures Recognize complete source and diagram evidence, concrete design and developer-experience decisions, and the complete planted scenario contracts. Preserve negative controls and grading thresholds. * fix: preserve decision brief structure in native questions Keep the required pros-and-cons heading and final Net field in native question text. Regenerate host outputs and document the release and evaluation repairs. Co-Authored-By: OpenAI Codex <noreply@openai.com> * docs: update project documentation for v1.88.0.0 Co-Authored-By: OpenAI Codex <noreply@openai.com> * fix: correct eval retry accounting and ship workflow gates * fix: capture native eval evidence and stabilize CI fixtures * fix: keep shared-code eval skips read-only Choose explicit no-change answers instead of mixed fix/preservation options. Reuse the bounded revalidation prompt for path fixtures so required review metadata is available without repeated discovery. Preserve source checks, retry limits, and failed native terminal outcomes. Add captured-question and callback regressions, plus evaluation selection coverage for the affected fixtures. --------- Co-authored-by: OpenAI Codex <noreply@openai.com> |
||
|
Garry Tanandgarrytan
|
85b8c038fc |
v1.87.3.0 fix: bind review evidence to the reviewed tree (#2875)
* fix(review): bind evidence to completed unchanged review passes * fix(review): keep unresolved Codex findings unverified * docs: update review evidence documentation for v1.87.3.0 * test(cso): let Windows integration finish within subprocess budgets * docs: update project documentation for v1.87.3.0 --------- Co-authored-by: garrytan <19957+garrytan@users.noreply.github.com> |
||
|
Garry TanandOpenAI Codex
|
71f6048e8a |
v1.84.1.0 fix: default Codex and Claude to frontier models (#2835)
* fix: default cross-model workflows to frontier models * chore: bump version and changelog (v1.82.1.0) Co-Authored-By: OpenAI Codex <noreply@openai.com> * fix: repair frontier eval budgets and workflow instructions Preserve frontier models and quality thresholds while fixing truncated judge output, ordered section expansion, consent checks, QA scoring, and ship audit gates. Add regression coverage and refresh generated docs. Co-Authored-By: OpenAI Codex <noreply@openai.com> * fix: resolve workflow gaps exposed by frontier evals Clarify plan-review ordering and fallback modes, preserve deploy readiness gates, honor configured merge methods, correct benchmark and canary contracts, and restore vendored installs on setup failure. Cover recovery with real-shell regressions. Co-Authored-By: OpenAI Codex <noreply@openai.com> * fix: use agent capture budgets for deploy evals Multi-turn deploy and benchmark sessions were incorrectly limited to the single-call judge timeout. Use the existing capture tier and leave outer-test cleanup headroom, with a free policy regression test. Keep all behavioral assertions and frontier models unchanged. Co-Authored-By: OpenAI Codex <noreply@openai.com> * fix: clarify retro workflow and evaluate compare instructions Include compare mode in the frontier judge excerpt, define metric sources and snapshot ordering, and preserve the existing prompt-size budget. Co-authored-by: OpenAI Codex <noreply@openai.com> * fix: make documentation release review and publication consistent Review before commit, clarify changelog safeguards and unavailable reviewer modes, and preserve raw PR bodies across separate shell calls. Keep title sync in one shell and add regression coverage. Co-authored-by: OpenAI Codex <noreply@openai.com> --------- Co-authored-by: OpenAI Codex <noreply@openai.com> |
||
|
Garry TanandClaude Fable 5
|
e76f65a8da |
v1.77.0.0 feat: test-infrastructure overhaul wave 1 — matrix deletion, flake telemetry, sync-spawn wedge class extinct (#2746)
* fix: pin the claude CLI to an exact version in the CI image + tripwire The image installed @anthropic-ai/claude-code UNPINNED and rebuilt weekly 'to pick up CLI updates' — while bun sat carefully pinned at 1.3.13 two RUN lines above. The PTY harness screen-scrapes this CLI's TUI, and that drift broke it three separate times (welcome-screen wedge on 2.1.233, skillify HOME discovery on 2.1.237, guard/freeze hooks on 2.1.162), each debugged as a flake first. Pin 2.1.251 (current latest), bump deliberately via a PR that runs the PTY gate, and enforce with test/ci-image-cli-pin.test.ts: any global npm install in Dockerfile.ci without an exact @X.Y.Z pin fails the free suite. The weekly ci-image cron stays as a cheap tag self-heal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: stamp the claude CLI version into every eval-store run record Three harness breakages were traced to claude-CLI TUI drift only after long flake hunts, because no run record said which CLI it actually exercised. EvalCollector now stamps claude_cli_version (claude --version, cached once per process, 'unknown' when the binary is absent) into both partial and finalized records — schema-additive optional field, no SCHEMA_VERSION bump. Correlating a flake wave with a CLI release becomes a grep over ~/.gstack/projects/<slug>/evals/ instead of archaeology. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: give the spinning-shard kill test load headroom (30s -> 90s) The test spawns and group-kills three real children (one a busy-loop burning a full core) while five sibling shard processes compete for eight vCPUs. Under full-suite load it blew bun's default 30s per-test ceiling at 30,009ms — while passing in isolation in 1.4s — and red the only required lane. Every assertion in it is event-based (statuses, group-kill proof, heartbeat lines); the sole latency claim is the <30s kill-deadline sanity bound, which stays. Explicit 90s headroom, not a weakened oracle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: green-by-skip census — skip counts in the classifier, all-skipped labeling in the paid runner bun's 'Ran N tests' line COUNTS skipped tests, so a codex/gemini shard whose every test self-skipped (binary absent on the runner — true of every CI runner today) exits 0, dodges the hollow-shard guard, and reads as coverage in the weekly census. The classifier now parses bun's ' N skip' / ' N pass' recap lines; ShardOutcome carries skippedTests; formatSummary and the fail-closed slices report label an all-skipped pass explicitly: 'all N tests SKIPPED — verified nothing'. Status stays 'passed' (external service availability is host state, not a repo regression) but the census can no longer mistake absence for coverage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: extract composite actions for eval-lane setup; surviving lanes gain the fail-fast registry verification 'Fix bun temp' x3, 'Restore deps' x5, 'Seed claude interactive config' x3, and 'Register gstack skills' x3 were byte-near-identical copies across the legacy matrix, the sliced lane, and the periodic lane — and only the MATRIX copy of register-skills carried the 19-line dangling-symlink + frontmatter fail-fast loop written after a silent 'Unknown command' + 35-min-timeout incident. Extract all four into .github/actions/ composites; the register composite carries the verification loop (generalized over the skill list), so the sliced and periodic lanes — the lanes that SURVIVE the matrix deletion — now inherit the check they had silently dropped. Matrix-job inline copies are left untouched: that job is deleted next. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: delete the legacy 17-row eval matrix — the sliced lane is the only paid lane Every PR paid twice: the hand-enumerated matrix (18 test files, 22.6 min, ~$21 API measured on run 33263204465) ran serialized AHEAD of the strictly superior sliced lane via 'needs: evals' — 35.5 min wall and ~2x paid spend for the same diff. 14 of 17 rows carried no tier:, so periodic Opus benchmarks leaked into every PR (the e2e-plan row alone: 12/12 tests, 21.7 min, $7.28 — the wall-clock bound of ALL of CI). Parity receipt (static, pre-deletion): the sliced lane's gate census (49 files, derived from the runner itself) strictly contains all 18 matrix test files, plus 31 files the matrix never ran. Pure deletion — one revert restores it. The PR comment moved into slices-report (same '## E2E Evals' upsert marker, now sourced from slice artifacts + carrying the fail-closed reconciliation verdict). plan-slices loses the needs edge; the dead workflow-level EVALS_TIER env goes with it. test/evals-workflow-matrix.test.ts (and its KNOWN_MATRIX_GAPS / KNOWN_TIER_UNSET burn-down ratchets — retired: the sliced census makes 'every gate file runs' true by construction) is rewritten as test/evals-workflow-wiring.test.ts: matrix stays deleted, planner/executor/ report tier + slice-count agreement, both surviving lanes on the shared register-skills composite with its fail-fast verification loop, PR comment survival. Expected: PR eval wall 35.5 -> ~13 min, per-PR paid spend ~halved. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: provider-runner timeouts kill the whole process GROUP; codex/gemini inherit the orphan-drain hardening All three provider runners (claude/codex/gemini) killed only the direct child on timeout: tool subprocesses the CLI spawned survived as orphans holding our pipes open and burning shared API rate (observed: a 600s timeout stretching past 1400s; a stalled run once burned a core for 15 hours). gstack-detach's watchdog had the same shape one level up — killpg SIGTERM, 5s grace, then a direct-child proc.kill() that orphaned grandchildren. Fix: spawn provider children via node:child_process with detached (own process group) and killProcessGroup(SIGKILL) in the timeout handler — runShardChild's proven pattern, EPERM/ESRCH fallbacks included. The codex and gemini copies also gain the reader.cancel() + stderr Promise.race hardening only the claude copy had (they still carried the blocked-drain hang it fixed). gstack-detach's watchdog now group-SIGKILLs after the grace. Regression net: test/session-runner-groupkill.test.ts drives the REAL runSkillTest against a fake claude shim (PATH override) that spawns a grandchild and wedges — the run must classify timeout within budget and leave neither shim nor grandchild alive — plus source pins on all three runners (detached + killProcessGroup, no bare timeout kill, no Bun.spawn reversion). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: skill-e2e-opus-47 renders SKILL.md fixtures into a mkdtemp — never the live tree mkEvalRoot ran gen-skill-docs with cwd=ROOT, regenerating every in-repo SKILL.md mid-run while concurrent paid shards copyFileSync those same files in their beforeAll (EVALS_JOBS>=4 locally, 2 per CI slice) — a sibling could capture a half-regenerated or opus-rendered SKILL.md, and a timeout before afterAll stranded the whole tree at the wrong model for every later shard. A cross-shard race that could flake ANY concurrent paid test. Render via the --out-dir flag gen-skill-docs grew for exactly this reason (mirrors the repo layout, which is all the fixture reads), read the skill heads from the render dir, delete it, and drop the afterAll restore-regen entirely. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: claude CLI version resolves in the runner parent, never on a test thread Eng-review finding: getClaudeCliVersion's fallback is a SYNCHRONOUS spawnSync on the same thread that polls concurrent PTY/session tests — the judgePtyState blocking class this overhaul kills elsewhere. The paid runner parent now resolves it once (cached) and stamps GSTACK_CLAUDE_CLI_VERSION into every shard's env; eval-store short-circuits on the env var, and the fallback spawn's budget tightens 10s -> 3s (bounded one-time stall, records 'unknown' on a slow CLI). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: wire skippedTests end-to-end through runPaidShard The census unit tests hand-built outcomes and the classifier tests parsed strings; nothing proved a real child's ' N skip' recap flows into outcome.skippedTests and the formatSummary label. A commandFor fake now prints the recap shape and the test asserts the parsed counts, the all-skipped predicate, and the 'verified nothing' label. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: make the setup composites rerun-safe (codex diff-review hardenings) restore-deps: 'cp -r SRC node_modules' with an existing node_modules NESTS the copy and leaves stale deps active — rm first. register-gstack-skills: 'ln -snf' hard-errors under set -eu when a REAL directory occupies the gstack slot — clear a non-symlink leftover first. CI workspaces are fresh today; a reusable composite must survive dirty reruns. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: sweep — every sync spawn in the test trees carries a timeout (436 sites, 157 files) spawnSync/execSync/Bun.spawnSync BLOCK the main thread, so bun's in-process per-test timeout can never fire while one waits — a hung child (stdin read, network probe, dead daemon) wedges the whole shard until the runner's external wall-clock SIGKILL. This exact class reached main: free-tests run 33262077256, test/gstack-memory-ingest.test.ts (normally 2.3s) held shard 2 at the 360s wall while its five siblings finished in ~65s. Mechanical sweep in two waves (12 + 4 fan-out agents, every edit verified against its call site): default timeout: 30_000 (matches the free runner's per-test budget), 120_000 for genuinely slow ops (installs, builds, playwright, provider CLIs), helper wrappers fixed ONCE where call sites route through them. Sites that only LOOK like calls (string fixtures, grep needles, comments) were skipped with reasons — the enforcement commit that follows marks them exempt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: sync-spawn timeout tripwire — the wedge class stays extinct Free scanner over all test trees (test/, browse/test/, design/test/, make-pdf/test/, ios-qa, browser-skills): every spawnSync/execSync/ Bun.spawnSync call site must carry a timeout within a 30-line options window, or an explicit '// tripwire-exempt: <reason>' marker. Comment lines are skipped; exemptions are counted and ratcheted shrink-only (ceiling 6 = the 6 string-fixture/grep-needle sites where the pattern is CONTENT, not a call — marked in this commit). A scan-sanity test pins that the scanner still sees >100 real call sites so it can never rot to a vacuous green. Companion to the 436-site sweep in the previous commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: paid-lane flake telemetry — record-level attempts, flaky_retries, report surfacing bun --retry leaves a retried pass INVISIBLE in its output: a fail-then-pass prints the error detail but no (fail) result line and recaps as a clean pass (probed live on 1.3.10). So attempts are recorded where they cannot lie: EvalCollector.addTest stamps a 1-based attempt on same-name re-records (a retried test runs its body again and re-records), finalized runs carry flaky_retries, printSummary warns loudly, and the fail-closed slices report lists every passed-only-on-retry test — recorded and ranked, never blocking and never silent. Cross-model confirmed (codex reached the same don't-parse -the-stream conclusion independently). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: free-lane flake ledger — retry ON in CI, flaky-passes recorded and uploaded The runner's attribution-gated flaky-retry pass (cap 5, truncation veto) was OFF in the required lane and its FLAKY-PASS evidence was console-only — so a single timing flake red the merge gate while repeat offenders stayed unenumerable. free-tests.yml now sets GSTACK_FREE_RETRY_FLAKY=1 and points GSTACK_FLAKE_LEDGER at runner.temp; every flaky-pass appends a JSONL entry (SINGLE writer: the parent runner — no concurrent-append hazard by construction; fail-open with a loud warning so a broken ledger can never red the lane) and the artifact uploads UNCONDITIONALLY — a flaky-pass run is green, which is exactly when the evidence matters. Wiring pinned by free-tests-workflow-wiring; ledger behavior unit-tested incl. the fail-open path. Matches 2026 industry practice (retry for data, quarantine out of merge-blocking but never out of logging) with the repo's own receipts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: eval:flake-rank — the flake-telemetry dial Aggregates per-test series across every finalized eval-store run (shard dirs included) plus the free flake ledger: runs, fails, RETRIED PASSES (the flake signature), avg duration — ranked retries-first. This is the readable dial behind two policies: a flaky pass never blocks a merge but is always ranked here, and the WS16 required-check promotion needs weeks of clean flake-rank, not vibes. --json for machines, --dir for downloaded CI artifacts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: two-phase session timeout — silent APIs die at the startup grace, named The single spawn-armed timer charged API queue latency to the work budget: the recurring '0 turns / $0.00 / x3 attempts' failure with four budget-bump receipts (180->300s, 240->360s, 300->420s, 90->300s). Split: startup phase (no NDJSON byte yet) kills EARLY at min(grace, timeout) with the distinct exitReason 'timeout_startup' — an availability verdict, not transcript archaeology — and the work phase arms on the first byte for the REMAINING budget, so total wall never exceeds the timeout (tier envelopes are margin-free: tests pass timeout: CAPTURE_MS and bun-budget the same tier). Local grace 90s (observed queue latency 60-90s), CI floor 300s (TODOS-filed; shared runners queue harder), both pinned by the new grace tests with fake -claude shims covering the late-first-byte and silent-API paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: census integrity — 17 phantom selection keys deleted, reverse invariant added, gitignored dep patterns replaced, local map forks derived The merge-blocking gate census counted tests that could not run. Deleted (critic-verified against both quoted-occurrence and dep-registration liveness): 7 *-prosons-format keys with no declaring test, ship-plan- completion/-verification, review-plan-completion, design-shotgun-path/ session/full, autoplan-core (dead ~10 months), e2e-harness-audit (its namesake is a FREE-suite file), plus 2 dead LLM-judge keys and 2 free-file keys (budget-regression-pty, global-discover) misplaced in the PAID maps. Census: 191 -> 174 keys, gate 86 -> 78 honest. The new reverse invariant in touchfiles.test.ts makes the class structurally impossible: every key must be quoted in a living paid test file OR registered to an existing paid test file via its dep list (the constructed- name binding the 2026-08 self-registration sweep established) — zero exceptions needed today, with a live-file check on any future exception. Also: '.agents/skills/**' dep patterns replaced with the generator (scripts/gen-skill-docs.ts) — .agents/ is gitignored, so those patterns could NEVER match a git diff and review-template edits silently stopped selecting codex/gemini tests; the codex/gemini local touchfile maps are now DERIVED from the canonical map (loud throw if a key vanishes) instead of hand-forked copies that had already drifted. ios-qa-e2e demoted gate -> periodic: its gate declaration was never executable in CI (hardware exclusion only applies at tier=periodic), so every Linux PR planned a hollow shard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: routing journeys lose their answer key and end at the routing decision The journey tests exist to catch skill-DESCRIPTION regressions (touchfiles: */SKILL.md.tmpl), but the fixture CLAUDE.md shipped an explicit prompt->skill lookup table — with the answer key in context, a badly regressed frontmatter description still routed correctly, so the tests could not fail on the exact class they select for. The fixture now carries only the generic invoke-skills nudge; the frontmatter carries the routing load. Also capped all 10 journeys at maxTurns 2 / tools [Skill, Read]: only the FIRST Skill call is asserted, so 5 turns of Read/Bash/Glob/Grep was pure spend — roughly halves each journey's cost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: retire decided A/B experiments; vendor the pre-cut fixture; ban raw-SHA fixtures Three one-shot decision experiments kept re-running weekly as N=1 stochastic comparisons — flaky by construction with near-zero remaining information: skill-e2e-auq-repetition-cut-ab (its own header: gate "passed pre-landing, approved 2026-08-25"), skill-e2e-preamble-script-ab ("demoted post-Phase-3"), and opus-47's fanout arm-vs-arm (parA >= parB across two SINGLE stochastic runs — a coin flip). Deleted, with their selection keys; the SDK overlay-harness stays as the maintained instrument for the next experiment, and opus-47 keeps its routing-precision cases. verboseSkill() now reads the VENDORED test/fixtures/auq-pre-cut-...-SKILL.md instead of `git show ab66193e^:...` — a branch-local ref that dies on branch prune and already failed on shallow clones. New free tripwire (test/git-ref-fixture-tripwire.test.ts) bans the raw-SHA fixture class outright: quoted SHA:path rev-specs and gitRef-style hex defaults in the test trees fail the suite with the vendor-instead instruction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: demote plan-ceo-review-expansion-energy to periodic Opus generator + a subjective 2-axis >=4/5 LLM-judge threshold sat in the MERGE-BLOCKING gate — the exact class its sibling posture tests were demoted for, with a receipt (a +21-line preamble change once flipped the score). CLAUDE.md's own tiering rule: Opus model test -> periodic. The weekly lane keeps the regression signal; merges stop paying a judge- temperament tax. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: paid shards get per-shard TMPDIR + CHROMIUM_PROFILE isolation and a kill-path cleanup backstop The free runner treats this isolation as MANDATORY (two concurrent shards on one Chromium profile kill each other's browser; shared tmp cross-contaminates) — the paid lane had none of it. Doubly load-bearing here: a shard that hits its 30-min wall is group-SIGKILLed, so per-test afterAll cleanup never runs; the rmSync backstop is the only thing keeping wedged runs from accumulating full git-repo workspaces in the shared tmpdir forever. This is the DAG prerequisite for raising EVALS_JOBS (next commit) — more concurrency on shared state amplifies exactly the shared-tree race class opus-47 exhibited. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: paid-runner defaults 4x4 -> 8x2 — halve the local gate worst case 39 of 75 skill-e2e files hold exactly ONE test, so within-shard concurrency was dead weight for most shards: 4 jobs x 4 concurrency yielded only ~4-6 real in-flight sessions and a 13-wave local gate worst case (~6.5h). 8 jobs x 2 gives ~10-13 in-flight — under the documented-safe ~15 — and ~7 waves (~3.3h worst case). CI lanes keep their explicit EVALS_JOBS env (2 per slice; 4 for gate-census); this changes local defaults. Rollback trigger: sustained 429 storms in the WS1 telemetry across 2 PR cycles. test/eval-detach-timeout-floor.test.ts recomputed green (the raise LOWERS the worst-case floor). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: SHA-pin every action in the secrets-bearing eval lanes evals.yml and evals-periodic.yml execute PR-authored code with three provider API keys in env, yet rode mutable action tags (@v7/@v8/@v2/@v4) — while quality-gate.yml, osv-scanner.yml, and dependency-review.yml already model the SHA-pin pattern. All 30 uses sites across both lanes now pin the exact commit (tag noted in a trailing comment); dependabot's github-actions ecosystem keeps them fresh via PRs instead of silent tag moves. Pulled forward from the plan's endgame on the CEO-review + outside- voice agreement: supply-chain pins on secret lanes go first, not last. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: sweep wave 3 — the execFileSync family gets timeouts (90 sites, 17 files) The tripwire's regex covered spawnSync/execSync/Bun.spawnSync but not execFileSync — an entire blocking sync-spawn API family that could reintroduce the shard-wedge class undetected (ship review army). Same mechanical recipe as waves 1-2: timeout: 30_000 default, 120_000 for slow ops, shared wrappers fixed once, string-needle sites skipped with reasons. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: review-army + adversarial test hardening - Tripwire scans execFileSync too (ceiling 8: two more grep-needle string exemptions); merge-introduced timeout-less spawnSync in question-preference-hook fixed — the tripwire caught a site that landed on main AFTER the sweep, on its first day. - gstack-detach gains TWO watchdog kill regression tests: TERM-immune grandchild (the killpg-after-grace escalation) and the leader-dies variant (the pgid-at-spawn fix — the case the first test cannot see). - eval-flake-rank gets its unit suite (final-attempt accounting, artifact exclusion, shard recursion, recency bound). - Groupkill/startup-grace shim markers are per-run unique (pid-suffixed sleep durations): sibling Conductor worktrees run free suites with no machine lock, and fixed markers let one run pgrep/pkill the other's shims — a cross-run flake inside the anti-flake tests. - flake-ledger test pins the project-scoped local default; stale empty section headers in touchfiles-data deleted (they invited entries under deliberately retired categories). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: adversarial-review runtime fixes across the telemetry + kill paths - session-runner: exit-labeling keys off 'exit', not 'close' — an orphan holding the pipes could relabel a REAL exit (auth failure) as 'timeout_startup' availability noise; the kill path still always group-kills and cancels the reader (labeling and unblocking are separate concerns). Work phase arms on a flag, not firstResponseMs===0 (a same-ms first byte left the startup timer live all run). The CI startup grace is now a real FLOOR (Math.max), matching its name and pinning test. - gstack-detach: pgid captured AT SPAWN (== child pid under start_new_session) — resolving it after the grace raised ESRCH once the leader died on SIGTERM, orphaning TERM-immune grandchildren forever. - test-free-shards: ledger entries carry branch + git_sha (rev-parse split: '--abbrev-ref HEAD HEAD' printed the branch twice and recorded it as the sha); local ledger default is per-PROJECT, not the machine-global tmpdir. - eval-flake-rank: per-LINE ledger parse (one torn JSONL line vanished the whole series), 60-day recency bound (transcript-bearing files are MBs), shared isFinalizedEvalResultFile predicate (the artifact-taxonomy rule lived in three places); eval-store exports the predicate and finalize stops computing flakyRetries twice; paid-shards cleanup uses async rm (a SIGKILLed shard's git-workspace teardown blocked every sibling's stream classification on the parent event loop). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: CI trust-boundary + fail-closed repairs (adversarial findings) - Token/exec separation restored: slices-report (runs PR-authored code: bun install + the reconcile runner) drops to contents:read; the PR comment moves to a NEW slices-comment job holding the write token with ZERO repo code — no checkout, no bun, only downloaded artifacts + jq/gh. $GITHUB_ENV/BASH_ENV persistence is job-scoped, so the split is the boundary. The matrix-era report job had this property; the consolidation had regressed it. Pinned by the wiring test. - Reconcile exit captured via PIPESTATUS[0] in BOTH lanes: GitHub's default run-step shell has no pipefail, so `$?` after `| tee` was tee's exit — the fail-closed gate was silently fail-open. Wiring test pins it. - PR comment: final-attempt accounting restored the dropped COST accumulation (the dial read $0 forever), flaky passes render as the warning they are (never as failures), and a malformed tests[] artifact skips that file instead of aborting the whole comment under bash -e. - Remaining mutable action tags pinned (free-tests upload-artifact, ci-image checkout/docker trio — the image publisher holds packages:write and feeds the secret-bearing lanes). restore-deps fallback installs --frozen-lockfile; register-gstack-skills validates skill names before its rm -rf. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump version and changelog (v1.77.0.0) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: update project documentation for v1.77.0.0 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: cross-model doc-review fixes — flake-ledger env knobs, CI retry-on note, stale version comment Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: correct CHANGELOG receipt numbers to measured values Gate census keys: 78 -> 77 (bun-imported E2E_TIERS count). Sweep receipt: 586 sites/176 files -> 499 sites/146 files, measured by running this branch's spawnsync-timeout-tripwire against origin/main (exit 1, 499 violations across 146 unique files; green on this branch). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: slices-comment creates the PR comment via REST — the write-token job has no git context The token/exec split gives slices-comment NO checkout by design, and gh's pr-comment subcommand resolves the repo FROM git — it died with 'not a git repository' on PR #2746's first run (the update-existing PATCH path was already explicit-repo REST and worked). Create now posts through gh api repos/.../issues/N/comments, and the wiring test pins that no git-context-requiring comment call can creep back into the job. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: startup-grace probes clear CI for local semantics; new probe pins the floor clamp The two shim probes pass explicit 2s/4s graces, but in CI the runner clamps any explicit grace up to the 300s floor (deliberate adversarial-review fix), so 'silent API killed at the grace' died at the 30s work cap instead of 2s — a deterministic red on every CI run, green locally. The probes now pin LOCAL semantics with CI cleared (same save/restore pattern as their PATH shim), and a fourth probe pins the clamp itself: CI=1 + 2s grace + 6s timeout must kill at the 6s cap, still in the startup phase — proof an explicit low grace cannot bypass the floor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Garry TanandClaude Fable 5
|
394db326f2 |
v1.71.0.0 feat: token-load reduction — preamble runtime scripts, gated onboarding, 20 skill carves, CLAUDE.md trim (#2691)
* feat(gen): strip gen-time-only frontmatter keys from Claude renders
interactive + benefits-from are read from the .tmpl by buildContext at
generation time; no runtime, host, or test reader consumes them from the
generated SKILL.md (e2e-harness-audit reads .tmpl; benefits-from tests
assert rendered prose). gbrain: stays (bin/gstack-brain-context-load reads
it from the installed render); hooks: stays (Claude Code host wires
PreToolUse from it).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(gen): regenerate SKILL.md — dead frontmatter keys removed
Mechanical regen after hosts/claude.ts stripFields change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(test): context-budget ratchet — CI ceilings on always-on + eager token ledgers
New free test grades the two ledgers nothing else guards: the full-frontmatter
always-on catalog (aggregate) and per-skill eager tokens (SKILL.md +
forced-read refs), via checkBudget from lib/context-bill.ts. Ceilings live in
test/fixtures/context-budget.json with x1.05/x1.10 headroom; regenerate with
bun test/helpers/capture-context-budget.ts. New skills fail until consciously
budgeted; removed skills fail until the fixture is refreshed; reductions
ratchet the ceilings down so wins lock in.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(todos): file output-template carve wave + plan-ceo doctrine revisit; mark preamble-carve P3 in flight
Two follow-ups deferred from the approved token-reduction program (CEO review
'NOT in scope' list), filed with full context per TODOS format. The existing
P3 preamble-carve entry gets a status update pointing at the program that
supersedes it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(test): review findings — Windows path normalization, full totals rebuild, ratchet coverage
Pre-landing review (5 specialists) found one critical: the ratchet test runs
in the curated Windows lane, where path.relative yields backslash skill names
that miss the test/ filter and mismatch every POSIX fixture key. Names are now
normalized once in buildRatchetBill (toPosixName) and the fixture filter is
tightened to test/fixtures/. All eight Bill.totals fields are rebuilt from the
filtered list (no fixture-polluted perInvocation/totalMd numbers for future
consumers). New coverage: Windows-separator normalization pins, a
captureContextBudget round-trip against tree-a (headroom math exact), a
stripFields regression pin (interactive/benefits-from absent from renders,
hooks/gbrain preserved), and the ceilings test no longer double-reports
stale-fixture entries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(test): adversarial findings — stable root key, symlink-alias dedupe, fixture-shape guard
Adversarial review (Claude subagent) verified the fixture's root-skill key was
the capture machine's checkout dirname: any non-gstack-named clone (every
Conductor worktree) failed the free suite, and the documented re-run-the-capture
recovery baked the local dirname into the committed fixture — silent corruption
through the tool's own protocol. The root skill is now pinned to ROOT_SKILL_KEY
('gstack', its frontmatter name). Symlink aliases are realpath-deduped (census
precedent): connect-chrome no longer gets its own ceiling, so Windows checkouts
that materialize the symlink as a plain file can't fail the stale-ceiling
set-equality test. New guards: fixture-shape validation (a string alwaysOnTotal
can no longer silently disable the ceiling), a mutation pin that the filter
shrinks the always-on ledger vs the raw bill, an alwaysOnTotal violation test
(the branch was load-bearing with only under-budget coverage), and an atomic
temp+rename fixture write. Fixture regenerated: 59 ceilings, alwaysOnTotal 6344.
Deferred with a TODO: anchoring transformFrontmatter's denylist strip to the
frontmatter block (latent, zero live collisions, pre-existing path).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: bump version and changelog (v1.69.1.0)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: update project documentation for v1.69.1.0
CLAUDE.md: Token ceiling section documents the context-budget ratchet as
the third guard (test file, fixture, new-skill budgeting, capture command).
CONTRIBUTING.md: Tier 1 guard list gains a Context-budget ratchet bullet;
the Adding-a-new-skill checklist gains the budget-capture step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: pin exact guard semantics for the context-budget ratchet in CLAUDE.md
Doc-review finding: "a third enforced ceiling" undercounted the guard
family (skill-size-budget floors and parity ratios also watch these
ledgers, relatively). Rephrased to match the ratchet test's own header:
absolute ceilings vs relative floors/ratios.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(changelog): heaviest-skill claim matches the fixture (land-and-deploy edges review by 0.2%)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(bin): gstack-skill-start + gstack-skill-end — the preamble runtime, consolidated
Absorbs the ~13KB of bash every tier-2+ SKILL.md inlined twice over (bootstrap
fence + artifacts-sync fence) and the skill-end telemetry/sync fences. Same
KEY: value STATUS-line contract the prose interprets, plus SKILL_START_PROTO
handshake (OV5), SESSION_ID/TEL_START echoes, GSTACK_HOME-normalized state
paths (EOV7), --parent-pid session identity (EOV5: $PPID inside the script is
the ephemeral tool-call shell), OV4 sanitization of passthrough output, and a
receipted daily artifacts pull (_receipted_git, brain-sync class, fail-closed).
Per-line || true error style throughout (F3) — a mid-script failure never drops
later STATUS lines.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(gen): preamble resolvers emit a script invocation fence instead of inline bash
generate-preamble-bash: ~6.3KB fence -> 4-line gstack-skill-start invocation
(quoted-tilde pitfall handled: leading ~ interpolates through $HOME; env-var
hosts keep $GSTACK_BIN) + degraded-mode prose (F1/EOV8: safe defaults, consent
gates deferred-never-lost; OV5: proto rule). generate-brain-sync-block: ~6.8KB
bash -> interpretation prose + the privacy stop-gate (stays inline until
Phase 2's gated emission). generate-completion-status: telemetry fence -> one
gstack-skill-end call with SESSION_ID/TEL_START handoff.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(gen): regenerate all skills + golden fixtures — inline preamble bash removed
Mechanical regen after the resolver change: −12,628 lines across 52 renders
(corpus 952K -> 806K render tokens; tier-2 skills −11-13KB each). Golden
per-host ship fixtures refreshed from the fresh claude/codex/factory renders.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: skill-start contract suite + preamble A/B eval + touchfiles registration
test/gstack-skill-start.test.ts (11 free tests): STATUS-key contract vs the
prose (F2), per-host fence resolution shapes (E1), proto-first, OV4 marker
sanitization, --parent-pid identity, headless suppression, skill-end duration
math + pending cleanup. test/skill-e2e-preamble-script-ab.test.ts (gate tier,
OV7): inline-bash render (pinned from
|
||
|
Garry TanandClaude Fable 5
|
9da6692930 |
v1.68.1.0 fix: phantom AskUserQuestion hooks — canonical-only registration + self-healing settings.json (#2631)
* fix(settings-hook): KNOWN_HOOKS identity healer — per-item ownership, mutation lock, fail-closed parse
Claude Code strips the unknown _gstack_source key when it rewrites
settings.json, so tag-based dedupe degraded to exact-command equality and
every Conductor worktree's setup appended a fresh hook entry; deleted
worktrees left dead hooks erroring on every AskUserQuestion fire.
- KNOWN_HOOKS identity table (shared JS prelude, single source of truth):
ownership is intrinsic and PER HOOK ITEM — basename + relpath suffix +
event (+ matcher where defined). Tags never claim foreign items.
- New `prune-stale [--repoint <root>] [--all]`: prune dead gstack items,
re-point survivors at the stable install (tag restore from the table),
exact-duplicate collapse, uninstall/no-team identity sweep. Explicit
plan_tune_hooks:no is honored (dead pruned, live never re-pointed).
- add-event / remove-source become item-aware: replace/remove only the owned
item; a user's co-located hook in the same entry is never collateral.
- Mutation safety: mkdir lock with owner token, ownership-checked release,
atomic stale takeover; per-process-unique tmp + backup names;
backup-on-change everywhere; fail-closed on parse failure (a corrupt
settings.json is never overwritten — previously catch{} clobbered it);
locked atomic rollback.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(gstack-config): `has <key>` — key-presence provenance through STATE_DIR resolution
`get` returns the DEFAULTS value for absent keys, so callers that need to
know whether the USER decided something (vs inherited a default) had no
correct primitive — setup's consent logic was about to grep a hardcoded
~/.gstack/config.yaml, which misclassifies under GSTACK_STATE_ROOT /
GSTACK_HOME / GSTACK_STATE_DIR overrides. `has` exits 0 iff the key is
literally present in the resolved config file, with the same C-locale key
validation as get/set.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(setup): canonical-only hook registration, heal-first, PT_EXPLICIT consent provenance
Three root causes of the phantom-AskUserQuestion-hooks class, all in the
registration path:
- Bug A: the Conductor auto-opt-in upgraded PT_DECISION "prompt" -> "yes"
even when "prompt" was dev-setup's EXPLICIT --plan-tune-hooks=prompt pin,
so every new Conductor workspace installed hooks. PT_EXPLICIT (flag/env/
config-key-presence via `gstack-config has`) now gates the auto-opt-in to
the true silent fall-through.
- Bug B: hook commands were baked from $SOURCE_GSTACK_DIR (`pwd -P` of the
running tree — ephemeral for worktrees). Registration is now CANONICAL-ONLY
via _hook_command_path (${CLAUDE_CONFIG_DIR:-$HOME/.claude}/skills/gstack);
missing canonical hook = skip + log, never a baked tree path. SessionStart
moves to schema-aware add-event under its identity source; whitespace paths
are quoted.
- Bug C: nothing ever pruned, and dead tagged entries blocked the
"already installed" guards forever. Setup now heals FIRST on every run
(prune-stale --repoint at the stable install), surfaces a one-line summary
only when something changed, surfaces the plan_tune_hooks:no-vs-live-hooks
contradiction, and --no-team tears down all three sources plus an identity
sweep for untagged strays.
dev-setup's no-mutation guarantee gains its stated repair exception (prune
dead / re-point existing, never ADD).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(uninstall): run hook cleanup BEFORE install-root deletion + full identity sweep
SETTINGS_HOOK resolves via $(dirname "$0") INSIDE the install root, but the
cleanup ran after `rm -rf ~/.claude/skills/gstack` — a real global uninstall
(running the installed copy) silently no-op'd and orphaned every hook entry.
Tests masked it by running the uninstaller from the repo checkout.
The relocated block also removes the auq-error-fallback source (registered by
setup, previously never torn down) and finishes with a prune-stale --all
identity sweep so untagged strays (Claude Code strips _gstack_source) go too.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: phantom-hooks heal coverage — incident facsimile, per-item safety, lock, canonical tripwires
- gstack-settings-hook-schema-aware: 16 new cases — identity re-point (tag
restore), foreign-basename rejection, mixed-entry per-item safety for
add-event/remove-source/--all, prune-stale modes incl. bash-prefix +
Windows-backslash + spaced-path idempotence, duplicate collapse preferring
the tagged twin, plan_tune_hooks:no split, backup-on-change no-churn,
fail-closed corrupt-JSON for every mutator, stale-lock takeover,
fresh-foreign-lock skip, two-writer concurrency smoke, and an INCIDENT
FACSIMILE replaying the exact 2026-08-17 production damage (6/3/2 entries,
mixed tags, live-ephemeral Stop) healing to 2/1/1 canonical.
- NEW setup-hook-canonical-paths: static tripwires — canonical-only resolver
(no $SOURCE_GSTACK_DIR anywhere in it), heal-before-guards ordering,
unsuppressed heal output, ${VAR:-0} counter idiom, shared-prelude
concatenation at every bun call site, KNOWN_HOOKS completeness vs setup's
registrations, uninstall cleanup-before-deletion ordering, defect-class
warning present.
- setup-plan-tune-hooks-noninteractive: PT_EXPLICIT pins + `gstack-config
has` provenance + has-subcommand behavior (env-resolution, malformed keys).
- auq-error-fallback-hook: registration + both-teardown wiring (previously
untested).
- uninstall: behavioral ordering test running the INSTALLED copy from inside
the root it deletes.
- setup-windows-fallback / gstack-config-key-locale: pins updated for the new
HOOK_CMD shape and the third C-locale validator.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(test): banner-tripwire exec used JSON.stringify as shell quoting — vacuous pass + stray artifact
JSON escaping is not shell escaping. Interpolating JSON.stringify(script)
into `bash -c ${...}` left every JSON "\n" as a literal backslash-n inside
shell double quotes, collapsing the extracted release-body tripwire block
onto one line: `then\n` parsed as the command word `thenn`, and
`>&2\nelse\n` parsed as the redirect `>&2nelsen` — so every full-suite run
littered a `2nelsen` file (containing "bash: thenn: command not found") in
the repo root, and the test's single not-contains assertion passed
VACUOUSLY because all output had been redirected into that file. The
"and it actually fires" functional check never verified anything.
Fix: pass the script as an argv element (spawnSync array form) and assert
both branches for real — ABORT case must print the leak message to stderr,
clean case must print "banner tripwire clean" to stdout.
Verified: `bun test test/binding-template-drift.test.ts` previously created
the artifact deterministically; the full free suite now runs artifact-free.
The other shell-interpolation sites (evidence, schema-aware concurrency,
empty-find-fallthrough, branch-slug-hygiene) already use correct quoting.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: regression pin for legacy remove mixed-entry filtering + ownership negatives
Coverage-audit iron rule: the rewritten legacy `remove` action filters
per-item (pre-v1.67.2 it dropped the whole entry, destroying a user's
co-located SessionStart hook) — modified existing behavior, previously
untested. Also pins two ownership negatives: an owned basename+relpath under
the WRONG matcher stays foreign, and prune-stale on an absent settings file
exits 0 with removed 0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: pre-landing review fixes — review-army findings hardened
Specialist review (testing, maintainability, security, performance,
data-migration) findings, each verified against code before fixing:
- legacy remove: preserve malformed/foreign entries (hooks absent, non-array,
or pre-existing empty) — only entries THIS pass emptied are dropped
- add-event: never tag a mixed entry (old gstack versions in sibling
worktrees treat tags as entry-level ownership and would destroy the user's
co-located items); tag only single-item entries; prune-stale drops tags
from mixed entries for the same reason
- prune-stale: within-entry twin collapse (two dead copies of one hook
re-pointed to the same canonical command no longer double-fire); command
quoting hardened via gsQuoteCmd (escapes \\ " $ backtick; gsStripWrap
unescapes so identity round-trips); NUL bytes in the dedupe key replaced
with a JSON.stringify key (bash silently dropped the NULs, degrading the
separator; the file also read as binary to tooling)
- gsIsAlive: only provable absence (ENOENT/ENOTDIR) counts as dead —
EACCES/EIO/unmounted volumes no longer prune (one-way-ratchet guard)
- gsWriteIfChanged: preserves the live settings.json mode across rewrites
(a user-tightened 0600 carrying API keys was silently broadened to 0644);
fresh files start 0600; backups rotate (keep 10)
- remove-source: command-less items default to foreign (gstack only writes
type:command items); single-item stray claim requires a command
- rollback: pointer target must be a sibling settings.json.bak.* file
- uninstall + setup --no-team + SessionStart registration: stderr stays
attached — a lock give-up or fail-closed parse during TEARDOWN must be
visible ("the next setup retries" does not apply after uninstall)
- setup: team-mode banner no longer claims an auto-update hook when
registration was skipped; heal log documents the rollback-pointer caveat;
SESSION_UPDATE_CMD quoting mirrors gsQuoteCmd; lock constants named
- list-sources: corrupt settings.json reports to stderr instead of silently
printing nothing (setup guards must not misread corrupt as no-hooks)
- tests: 10 new pins (malformed-entry preservation, mixed no-tag, twin
collapse, 0600 mode, metachar escaping round-trip, backup rotation,
rollback pointer refusal, held-lock uninstall warning, matcher-drift
tripwire, ownership negatives)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: red-team findings — verify-gate identity, single quoting authority, Windows paths
Red-team pass over the hardened diff (several findings empirically verified
by the reviewer before reporting):
- KNOWN_HOOKS gains the sixth identity: gstack-verify-gate (README-documented
opt-in Stop hook). A tag-stripped verify-gate entry previously survived
prune-stale --all and errored at the end of EVERY turn after uninstall
deleted the install root — the exact phantom-hook class this branch fixes.
Uninstall also sweeps its tagged form.
- add-event is now the single quoting authority: every registered command is
normalized through the same gsQuoteCmd/gsStripWrap round-trip the healer
uses. Pre-fix, only SessionStart got caller-side quoting — a spaced/metachar
canonical root registered broken plan-tune/AUQ/timeline hooks that the very
next heal rewrote (the codebase disagreed with its own registrations).
- Windows: MSYS-form paths (/c/Users/...) are drive-translated for fs checks
only (gsWinPath) — native bun resolved them drive-relative, so the heal
judged every LIVE Windows hook dead and pruned it. The three AskUserQuestion
hooks and the Stop hook now also get the mandatory 'bash ' prefix on
Windows (previously only SessionStart did; extensionless bash shims
otherwise hit the file-association dialog).
- CANONICAL_GSTACK_ROOT falls back to $HOME/.claude/skills/gstack when a
CLAUDE_CONFIG_DIR-derived root was never installed (the installer hardcodes
the home path — split-brain left such users permanently hookless).
- prune-stale preserves foreign entries that STARTED empty (they were
silently deleted, uncounted, on every heal).
- The timeline Stop registration and its list-sources guard join the
zero-silent-mutations contract (stderr attached).
Tests: verify-gate tag-stripped heal+sweep, started-empty preservation,
add-event quoting-authority round-trip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: bump version and changelog (v1.68.1.0)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: update project documentation for v1.68.1.0
README: document canonical-only hook registration + the prune-stale
self-heal in the setup hooks section; expand the manual-uninstall note
to cover every gstack hook identity, not just timeline-stop-hook.
CONTRIBUTING: record PT_EXPLICIT provenance (Conductor auto-opt-in
fires only on the true silent fall-through) and the heal-first repair
exception in the dev-setup paragraph.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(settings-hook): fail-loud hardening — gsMain umbrella, lock exit 5, prototype-safe ownership
bun in -e mode swallows uncaught exceptions thrown after a require() and
exits 0 (verified on 1.3.13; uncaughtException handlers never fire either),
so any runtime throw in a mutator was a SILENT SUCCESS. Every script body
now runs inside a gsMain try/catch that prints "internal error ... refusing
to mutate" and exits 4.
Also: lock give-up now exits 5 instead of 0 (callers must not report a
skipped mutation as registered); basename lookup uses hasOwnProperty so a
foreign hook named "toString"/"constructor" can't resolve to an inherited
Object.prototype member and abort the sweep; ownership-checked release also
clears an empty/missing owner file; backup rotation sorts by mtime, not
name; Windows-only backslash normalization (a legal Unix path containing a
backslash is no longer rewritten); GSTACK_SWEEP_EXCLUDE_SOURCES lets a
sweep spare named sources; lock tradeoffs documented at the lock helper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(setup): honest hook-registration reporting + verify-gate sweep exclusion
_install_plan_tune_hooks now propagates per-add-event failures (lock
contention exits 5, fail-closed settings errors exit 3) and both caller
sites branch on it: success logs the installed message, failure logs a
visible "NOT registered — re-run ./setup" warning instead of claiming
success for a mutation that never happened.
--no-team's identity sweep runs with GSTACK_SWEEP_EXCLUDE_SOURCES=
verify-gate: turning team mode off must not delete the user-registered
verify-gate opt-in whose binary still exists (uninstall still sweeps it,
correctly, because there the binary itself is being removed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: adversarial regression pins — wrong-shape fail-loud, prototype basename, sweep exclusion, lock exit 5
New pins for the fail-loud hardening: a wrong-shape hooks value (object
where an array belongs) exits 4 with "refusing to mutate" and leaves the
file byte-identical (pre-gsMain this was a silent exit-0 no-op); a foreign
hook whose basename collides with Object.prototype ("toString") survives
an --all sweep that still removes gstack rows; GSTACK_SWEEP_EXCLUDE_SOURCES
preserves the verify-gate row during --all; the fresh-foreign-lock test now
asserts the loud exit 5 instead of a quiet skip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(verify-gate): allow the --no-team sweep exclusion, keep registration banned
setup now legitimately mentions verify-gate once: the --no-team identity
sweep excludes it via GSTACK_SWEEP_EXCLUDE_SOURCES so team-mode teardown
can't delete a user-registered gate. The opt-in pin tightens from a blanket
not-contains to: every mention must be a comment or that exclusion, and no
mention may sit on an add-event line.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(settings-hook): GNU-first stat in the lock stale check — Linux abort on held locks
On Linux, BSD-style `stat -f %m` prints a multi-line FILESYSTEM block to
stdout before exiting 1, so the BSD-first || chain captured that garbage
concatenated with the real `stat -c %Y` epoch. The non-numeric mtime made
`$(( now - mtime ))` a syntax error and set -e killed the binary with
exit 1 whenever a lock dir already existed — every contention path (stale
takeover, give-up, concurrent writers) broke on CI while staying green on
macOS, where BSD stat -f succeeds cleanly.
GNU `stat -c %Y` now goes first (BSD stat rejects -c with no stdout, so
macOS falls through cleanly), and a numeric guard blanks any residual
garbage so a future platform quirk degrades to the normal give-up path
instead of an arithmetic abort. Same defect class as gstack-repo-mode's
GNU-first ordering (#2195). Verified in an oven/bun Linux container:
the four CI-failing lock tests now pass (62/62 across both files).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(uninstall): 30s budgets for the two subprocess-heavy behavioral tests
Both tests spawn the copied uninstaller, which itself runs several
settings-hook bun -e children (the lock-contention one also waits out a
300ms give-up per call). On a loaded box those cold starts blow bun's
default 5s per-test timeout, and a timeout kill reports as a bare fail
with no assertion diff — observed at 5.6-8.5s under load avg 25+.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Garry TanandClaude Fable 5
|
1cab5e1108 |
v1.66.1.0 feat: content binding — evidence ledger, wtree staleness, tracker trust envelope, fail-closed hooks (#2603)
* fix(hooks): fail-closed freeze + shared extractor + careful HIGH tier Freeze boundary hook had four verified bugs: the grep-first JSON extractor truncated at escaped quotes and failed OPEN on unparseable payloads; the deny JSON was printf-interpolated so a quote- or newline-bearing path silently no-oped the block; the freeze path read stripped INTERNAL spaces (a boundary like ~/My Project could never match); and the path resolver skipped the final component, letting an in-boundary symlink write through to an out-of-boundary target. Fixes, structurally: one shared sourced helper (careful/bin/hook-extract.sh) now owns JSON extraction and JSON-encoded decision envelopes for BOTH hooks -- the two-copy drift is how freeze kept a broken extractor after careful's was fixed. Freeze is now deny-tier fail-closed (unparseable payload denies, parsed-but-no-file_path still allows), trims only leading/trailing whitespace, and resolves symlinks through the final path component. Careful gains a HIGH tier (hard deny, simple commands only): recursive delete of /, ~, or $HOME, and force-push to the repo's default branch. Compound commands always fall through to the MEDIUM ask; --force-with-lease is never HIGH. Documented as a best-effort advisory hard-stop, not a policy boundary. Plus additive-only project patterns (~/.gstack/careful-patterns.txt + per-project file): config can only ADD warn rules, never suppress a baseline family. test/hook-scripts.test.ts: 89 tests incl. malformed-payload deny, parseable deny JSON for hostile paths, space-bearing boundaries, symlink escape, HIGH tier splits, additive invariant, invalid-regex resilience. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(review): content-addressed staleness via working-tree fingerprint Review records now bind to the content they were made on. bin/gstack-review-log stamps every appended record with commit_full, tree, dirty (informational) and wtree — a working-tree fingerprint from the new bin/gstack-wtree (temp index seeded from HEAD + git add -A + write-tree). The binding fields are computed authoritatively; caller-supplied values for those keys are ignored, so a stale rendered template or a forged field can't bind a record to content it wasn't made on. Why a working-tree fingerprint instead of HEAD^{tree}: committing identical content doesn't change it (a record made on a dirty tree stays valid after the same content is committed), untracked new source files DO change it (new code can't hide from freshness), and gitignored scratch stays out. Rebase, amend and squash with identical content grade CURRENT instead of stale. Grading: the dashboard (scripts/resolvers/review.ts) and /land-and-deploy Step 3.5a apply a content-first rule to diff-scoped review rows — wtree match with both sides clean is CURRENT, full stop. Plan-tier reviews grade a plan file, not the repo tree, so they keep the 7-day logic (optional plan_sha256 caller field noted). The rev-list fallback no longer errors when the stored commit was rebased away: it grades UNKNOWN and treats it as stale. bin/gstack-review-read emits ---WTREE---/---TREE---/---DIRTY--- so graders consume one tool output. Old records without wtree fall back to the existing heuristics; no migration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(evidence): verification-evidence ledger mechanizes /ship's IRON LAW New bin/gstack-evidence: a transparent wrapper that records every verification run as {ts, label, command, cmd_sha256, exit, duration_s, commit, tree, dirty, wtree, log_path} in ~/.gstack/projects/<slug>/<branch>-evidence.jsonl, plus a read-only `check` that grades FRESH/STALE/MISSING per label. "Tests passed" now binds to the exact working-tree content it ran on (bin/gstack-wtree fingerprint), so evidence recorded on uncommitted code stays FRESH after the exact tested content is committed — the /ship Step 5 -> Step 16 case — while an untracked new source file or any content change invalidates it. Check semantics: every named label's latest record must be green, within --max-age, matching --expect-cmd's hash when given, and fingerprint-identical (or diff confined to --allow-paths — mechanizing Step 16's existing "CHANGELOG edits don't count" carve-out). No --any mode: a green lane can never mask a red sibling. Any git failure inside check (gc'd tree object, not a repo) degrades to STALE/MISSING, never an error into the calling skill flow. Transparency invariant (load-bearing, test-pinned): the child's exit code is ALWAYS the wrapper's exit code; ledger/log/redact failures are stderr warnings. Logs are per-run (0600, exclusive-open, 2MB truncation marker, 30-day opportunistic prune) — no more shared /tmp collisions between concurrent ships. Command strings are redact-scanned before recording (HIGH credential -> stored redacted). Machine-local by design: neither ledger nor logs brain-sync. Wired: ship Step 5 lanes run wrapped (per-lane labels), ship Step 16 and land-and-deploy 3.5b check the ledger first and cite FRESH evidence instead of re-running; a failed CHECK never blocks (run live), a failed RUN does. test/evidence.test.ts: 21 tests incl. the keystone dirty-record -> commit -> FRESH case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(security): trust envelope for tracker text at every model-context ingress Web page content has had a trust envelope since v1.38; tracker text did not — PR bodies, PR/issue comment bodies, and model-judged issue titles entered agent context raw. Anyone who can comment on a PR could put instructions in front of the agent. New lib/tracker-guard.ts + bin/gstack-issue-guard: every tracker-text read now emits inside a "BEGIN UNTRUSTED TRACKER CONTENT" envelope. Content is enveloped even when clean (a pattern scan is not proof of safety); injection-shaped lines get a visible [INJECTION-PATTERN] label; NFKC + zero-width normalization runs for DETECTION only (fullwidth/invisible evasion caught, content bytes never rewritten); forged END banners are zero-width-spliced so they can't close the envelope early. Fetch failure exits non-zero with NO envelope — never a fake-trusted empty one. Issue numbers are validated and gh is spawned via argv arrays. Patterns reuse lib/jsonl-store's INJECTION_PATTERNS single copy plus a separate TRACKER_EXTRA list (kept separate so decision/learning store write-rejection semantics don't change). 8 sites wired: greptile findings + replies fetches (metadata/body split — ids and paths stay machine-raw for reply POSTs), review.ts PR-body reads x2, land-and-deploy 3.5c, document-release PR/MR body (two-artifact flow: the enveloped rendering is what the agent READS, the raw tempfile is what the pipeline mutates, and a write-side banner tripwire aborts any edit that leaked envelope markup), and spec's issue-title dedupe (titles are model-judged for similarity, so they're ingress). Title-prefix rewrites and state-routing fetches are mechanical, not ingress — deliberately not enveloped. test/tracker-guard-wiring.test.ts is the CI tripwire: raw tracker-text reads outside the guard fail the suite unless carried by a reasoned SCANNER_EXEMPT entry; exemptions are liveness-checked so a moved site forces a re-audit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(binding-wave): drift tripwire, golden fixtures, TODOS follow-ups test/binding-template-drift.test.ts pins the load-bearing prose rules in the GENERATED templates (ship Step 16 evidence check, per-lane wrapped test lanes, land-and-deploy wtree-first grading + UNKNOWN fallback, dashboard content-first rule, release-body banner tripwire, greptile guard pipes) so a template refactor can't silently drop a rule while the bins keep passing their unit tests. Golden ship fixtures re-pinned to the new intentional output (claude/codex/ factory variants). TODOS.md gains the five deferred follow-ups from the review wave: eval-run evidence records, spec-spawn outcome ledger, merge-SHA custody, default-if-silent escalations, and the paid eval case proving agents apply the staleness grading rule. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(careful): trim HIGH-tier + project-pattern docs under the size budget The new sections pushed careful/SKILL.md to 2551 -> 3879 bytes (x1.52, gate caps growth at x1.5 of the v1.47 baseline). Same content, tighter prose: 3516 bytes (x1.38). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(tests): scratch-repo fixtures never invoke the operator's gpg The evidence/review-log/hook fixtures inherited global commit.gpgsign, so fixture commits called the operator's gpg-agent — which fails with "Cannot allocate memory" under parallel shard load, breaking test SETUP (not the code under test). All fixture git invocations now pass -c commit.gpgsign=false -c tag.gpgsign=false. Hermetic repos, no pinentry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: pre-landing review fixes (27 specialist findings, 3 critical) Specialist army findings, all quote-verified before fixing: Security: careful force-push guard now catches git's plus-refspec force syntax (git push origin +main carried force with no flag — silently allowed before) and refspec-form targets (HEAD:main); default-branch matching is tokenized FIXED-STRING comparison on the full branch path (slashed defaults like release/2.0 work; no ERE interpolation), glob-safe via noglob. HIGH rm tier is tokenized too: trailing long options (--no-preserve-root) and /* are root-class. Stored evidence fingerprints are 40-hex re-validated before reaching git argv. normalizeForDetection sweeps ALL Unicode format chars (\p{Cf}: soft hyphens, bidi marks, tag chars) instead of five enumerated zero-widths. The wiring scanner gains flagless gh pr/issue view patterns. The release-body banner tripwire diffs against the fetched original so a hostile pre-existing banner string can't permanently DoS doc updates. Ship/land evidence checks now pass --expect-cmd (a green `echo ok` recorded under the label can never mint FRESH); package.json stays allow-listed with the residual documented. Performance: gstack-wtree seeds its temp index by COPYING the real index (stat cache preserved — measured 40x faster than read-tree seeding, identical hash) with read-tree fallback; evidence uses findLast and one gstack-slug spawn; the stream pump honors backpressure via drain; careful's pattern block short-circuits before slug resolution when no pattern file exists. Testing: the gh-failure envelope test was VACUOUS (killing PATH killed the bun shebang before the code under test ran) — replaced with a PATH gh shim that exercises the real branch, plus shimmed happy paths (issue/pr-body/ unparseable JSON); evidence check --all + empty ledger + non-numeric --max-age (now a usage error, was silent fail-open) covered; HIGH-tier variants pinned; hook analytics respect GSTACK_HOME so tests stop writing the operator's real skill-usage.jsonl. Maintainability: dead exit ternary removed; flagValue deduped into bin-context; sentinel defusal derived from the banner constants (no invisible literals — \u escapes only); scratch-repo git fixture extracted to test/helpers/scratch-repo.ts (one hermetic incantation, three consumers); shared gstack_hook_log_fire in hook-extract.sh; the dashboard/land diff-scoped row lists are aligned (codex-review) and drift-pinned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: red-team review fixes (9 findings, 2 critical) Red team reviewed what four specialists missed — cross-cutting and self-contradiction class: CRITICAL: the release-body banner tripwire failed OPEN on the exact leak it guards (grep -c prints 0 AND exits 1 on no-match, so a fallback echo double-emitted "0" twice and the -gt comparison fell into the clean branch) — counts now default via parameter expansion, and a functional drift test executes the rendered tripwire block against a 0->1 banner delta to prove the ABORT branch fires. CRITICAL: evidence fingerprints were captured AFTER the child exited, so a working-tree edit made DURING a long suite was certified as tested content — wtree is now captured before spawn and re-checked after; mid-run drift omits the fingerprint (grades STALE) with a warning. Also: the review-grading rule dropped its dirty-gates (they nullified the keystone dirty-record->commit->CURRENT property that evidence checks already honor — wtree equality alone proves identical content); careful's HIGH force-push tier falls back to probing origin/main|master when the origin/HEAD symbolic ref is absent (Conductor worktrees — the tier was silently inert in the primary deploy environment); quoted tokens (rm -rf "/", push "main") no longer dodge the deny; freeze fails CLOSED when its own helper file is missing (bash makes a missing source target fatal non-interactively, so an existence pre-check guards it); spec dedupe distinguishes pipeline failure from zero matches instead of silently skipping dedupe on gh/jq breakage; land 3.5b sets the cross-session --expect-cmd mismatch expectation; hook analytics JSON fields are encoder-built per this wave's own rule. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: re-pin codex/factory golden fixtures post-regeneration The suite regenerates .agents/.factory in place mid-run; the prior pin snapshotted them before the dashboard-rule regen landed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump version and changelog (v1.66.1.0) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: adversarial review fixes (Claude pass, 14 findings, 1 verified-live critical) The fresh-context adversarial pass caught a live bug in this branch's own performance fix: gstack-wtree exported GIT_INDEX_FILE BEFORE resolving the real index path, so `git rev-parse --git-path index` returned the temp index itself, the stat-cache copy self-copied and failed, and every invocation fell back to the full re-hash — the fast path was dead code (verified with bash -x). Resolution now happens before the export; measured 0.08s per call on this repo. Also fixed: careful fails to an ASK (not silence) when its own helper file is missing (same partial-install state freeze already defends against); the --source label is sanitized inside the envelope lib (newline-stripped, sentinel-defused, length-capped — it sits in trusted framing); the HIGH rm tokenizer skips redirections/backgrounding/`--` (rm -rf / 2>/dev/null now denies) and knows ${HOME}; user pattern lines starting with a dash work (grep --); greptile bodies carry per-comment id headers inside the envelope so multi-comment PRs stay attributable (ids verified against raw metadata, never trusted in-body); the release-body tripwire fails CLOSED when its input files are missing (separate-shell $$ reality); land 3.5b gets the same allow-paths as ship; the "either side dirty" fallback leftover is gone from both grading surfaces; the evidence pump races drain against error (EPIPE consumers can't hang the wrapper); an unset HOME skips bookkeeping instead of creating a literal ~ dir inside the repo; a write-failure log ends with a visible marker; freeze expands a literal leading ~ in the boundary; review-log documents its log-time binding window. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: pin golden fixtures from --host all generation `bun run gen:skill-docs` generates the claude host only; .agents/.factory regenerate when the suite's --host codex/factory tests run in place. Fixture pins must come from `gen-skill-docs --host all` output or they lag one resolver edit behind and fail the next full-suite run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: assemble the fixture PAT by concatenation (no live-format literal) The repo's own pre-push credential guard (correctly) blocked the push: the redaction test's fabricated GitHub PAT was a live-format literal in the diff. The token is now concatenated at runtime — the source carries nothing the scanner can match, the engine still receives a live-format value. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: update project documentation for v1.66.1.0 CLAUDE.md: add gstack-wtree/gstack-evidence/gstack-issue-guard to the bin/ structure line and tracker-guard.ts to the lib/ line. README.md + docs/skills.md: /careful descriptions no longer claim every warning is overridable — the HIGH tier hard-denies root/home recursive deletes and default-branch force-pushes; skills.md also documents the additive-only careful-patterns.txt warn rules. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: doc-review fixes — new bins in README table, careful claims precise README.md: add gstack-wtree, gstack-evidence, and gstack-issue-guard to the Standalone binaries table (they shipped in v1.66.1.0 with no user-facing reference outside CHANGELOG). docs/skills.md: the safety-skills intro said "no configuration files" which the optional careful-patterns.txt now contradicts, and the hard-deny description undersold the deny set (the hook also denies /*, ~/, and $HOME/ forms, not just bare / and ~). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: guard reflects the hard-deny tier; changelog stats current guard/SKILL.md claimed every destructive warning was overridable — the shared careful hook now hard-denies the catastrophic shapes. CHANGELOG numbers updated to the final measured state (0.09s fingerprint, 50 findings/6 critical across all review passes). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |