mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-09 14:38:59 +02:00
kingstown
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Garry TanandClaude Fable 5
|
e76f65a8da |
v1.77.0.0 feat: test-infrastructure overhaul wave 1 — matrix deletion, flake telemetry, sync-spawn wedge class extinct (#2746)
* fix: pin the claude CLI to an exact version in the CI image + tripwire The image installed @anthropic-ai/claude-code UNPINNED and rebuilt weekly 'to pick up CLI updates' — while bun sat carefully pinned at 1.3.13 two RUN lines above. The PTY harness screen-scrapes this CLI's TUI, and that drift broke it three separate times (welcome-screen wedge on 2.1.233, skillify HOME discovery on 2.1.237, guard/freeze hooks on 2.1.162), each debugged as a flake first. Pin 2.1.251 (current latest), bump deliberately via a PR that runs the PTY gate, and enforce with test/ci-image-cli-pin.test.ts: any global npm install in Dockerfile.ci without an exact @X.Y.Z pin fails the free suite. The weekly ci-image cron stays as a cheap tag self-heal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: stamp the claude CLI version into every eval-store run record Three harness breakages were traced to claude-CLI TUI drift only after long flake hunts, because no run record said which CLI it actually exercised. EvalCollector now stamps claude_cli_version (claude --version, cached once per process, 'unknown' when the binary is absent) into both partial and finalized records — schema-additive optional field, no SCHEMA_VERSION bump. Correlating a flake wave with a CLI release becomes a grep over ~/.gstack/projects/<slug>/evals/ instead of archaeology. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: give the spinning-shard kill test load headroom (30s -> 90s) The test spawns and group-kills three real children (one a busy-loop burning a full core) while five sibling shard processes compete for eight vCPUs. Under full-suite load it blew bun's default 30s per-test ceiling at 30,009ms — while passing in isolation in 1.4s — and red the only required lane. Every assertion in it is event-based (statuses, group-kill proof, heartbeat lines); the sole latency claim is the <30s kill-deadline sanity bound, which stays. Explicit 90s headroom, not a weakened oracle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: green-by-skip census — skip counts in the classifier, all-skipped labeling in the paid runner bun's 'Ran N tests' line COUNTS skipped tests, so a codex/gemini shard whose every test self-skipped (binary absent on the runner — true of every CI runner today) exits 0, dodges the hollow-shard guard, and reads as coverage in the weekly census. The classifier now parses bun's ' N skip' / ' N pass' recap lines; ShardOutcome carries skippedTests; formatSummary and the fail-closed slices report label an all-skipped pass explicitly: 'all N tests SKIPPED — verified nothing'. Status stays 'passed' (external service availability is host state, not a repo regression) but the census can no longer mistake absence for coverage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: extract composite actions for eval-lane setup; surviving lanes gain the fail-fast registry verification 'Fix bun temp' x3, 'Restore deps' x5, 'Seed claude interactive config' x3, and 'Register gstack skills' x3 were byte-near-identical copies across the legacy matrix, the sliced lane, and the periodic lane — and only the MATRIX copy of register-skills carried the 19-line dangling-symlink + frontmatter fail-fast loop written after a silent 'Unknown command' + 35-min-timeout incident. Extract all four into .github/actions/ composites; the register composite carries the verification loop (generalized over the skill list), so the sliced and periodic lanes — the lanes that SURVIVE the matrix deletion — now inherit the check they had silently dropped. Matrix-job inline copies are left untouched: that job is deleted next. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: delete the legacy 17-row eval matrix — the sliced lane is the only paid lane Every PR paid twice: the hand-enumerated matrix (18 test files, 22.6 min, ~$21 API measured on run 33263204465) ran serialized AHEAD of the strictly superior sliced lane via 'needs: evals' — 35.5 min wall and ~2x paid spend for the same diff. 14 of 17 rows carried no tier:, so periodic Opus benchmarks leaked into every PR (the e2e-plan row alone: 12/12 tests, 21.7 min, $7.28 — the wall-clock bound of ALL of CI). Parity receipt (static, pre-deletion): the sliced lane's gate census (49 files, derived from the runner itself) strictly contains all 18 matrix test files, plus 31 files the matrix never ran. Pure deletion — one revert restores it. The PR comment moved into slices-report (same '## E2E Evals' upsert marker, now sourced from slice artifacts + carrying the fail-closed reconciliation verdict). plan-slices loses the needs edge; the dead workflow-level EVALS_TIER env goes with it. test/evals-workflow-matrix.test.ts (and its KNOWN_MATRIX_GAPS / KNOWN_TIER_UNSET burn-down ratchets — retired: the sliced census makes 'every gate file runs' true by construction) is rewritten as test/evals-workflow-wiring.test.ts: matrix stays deleted, planner/executor/ report tier + slice-count agreement, both surviving lanes on the shared register-skills composite with its fail-fast verification loop, PR comment survival. Expected: PR eval wall 35.5 -> ~13 min, per-PR paid spend ~halved. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: provider-runner timeouts kill the whole process GROUP; codex/gemini inherit the orphan-drain hardening All three provider runners (claude/codex/gemini) killed only the direct child on timeout: tool subprocesses the CLI spawned survived as orphans holding our pipes open and burning shared API rate (observed: a 600s timeout stretching past 1400s; a stalled run once burned a core for 15 hours). gstack-detach's watchdog had the same shape one level up — killpg SIGTERM, 5s grace, then a direct-child proc.kill() that orphaned grandchildren. Fix: spawn provider children via node:child_process with detached (own process group) and killProcessGroup(SIGKILL) in the timeout handler — runShardChild's proven pattern, EPERM/ESRCH fallbacks included. The codex and gemini copies also gain the reader.cancel() + stderr Promise.race hardening only the claude copy had (they still carried the blocked-drain hang it fixed). gstack-detach's watchdog now group-SIGKILLs after the grace. Regression net: test/session-runner-groupkill.test.ts drives the REAL runSkillTest against a fake claude shim (PATH override) that spawns a grandchild and wedges — the run must classify timeout within budget and leave neither shim nor grandchild alive — plus source pins on all three runners (detached + killProcessGroup, no bare timeout kill, no Bun.spawn reversion). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: skill-e2e-opus-47 renders SKILL.md fixtures into a mkdtemp — never the live tree mkEvalRoot ran gen-skill-docs with cwd=ROOT, regenerating every in-repo SKILL.md mid-run while concurrent paid shards copyFileSync those same files in their beforeAll (EVALS_JOBS>=4 locally, 2 per CI slice) — a sibling could capture a half-regenerated or opus-rendered SKILL.md, and a timeout before afterAll stranded the whole tree at the wrong model for every later shard. A cross-shard race that could flake ANY concurrent paid test. Render via the --out-dir flag gen-skill-docs grew for exactly this reason (mirrors the repo layout, which is all the fixture reads), read the skill heads from the render dir, delete it, and drop the afterAll restore-regen entirely. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: claude CLI version resolves in the runner parent, never on a test thread Eng-review finding: getClaudeCliVersion's fallback is a SYNCHRONOUS spawnSync on the same thread that polls concurrent PTY/session tests — the judgePtyState blocking class this overhaul kills elsewhere. The paid runner parent now resolves it once (cached) and stamps GSTACK_CLAUDE_CLI_VERSION into every shard's env; eval-store short-circuits on the env var, and the fallback spawn's budget tightens 10s -> 3s (bounded one-time stall, records 'unknown' on a slow CLI). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: wire skippedTests end-to-end through runPaidShard The census unit tests hand-built outcomes and the classifier tests parsed strings; nothing proved a real child's ' N skip' recap flows into outcome.skippedTests and the formatSummary label. A commandFor fake now prints the recap shape and the test asserts the parsed counts, the all-skipped predicate, and the 'verified nothing' label. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: make the setup composites rerun-safe (codex diff-review hardenings) restore-deps: 'cp -r SRC node_modules' with an existing node_modules NESTS the copy and leaves stale deps active — rm first. register-gstack-skills: 'ln -snf' hard-errors under set -eu when a REAL directory occupies the gstack slot — clear a non-symlink leftover first. CI workspaces are fresh today; a reusable composite must survive dirty reruns. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: sweep — every sync spawn in the test trees carries a timeout (436 sites, 157 files) spawnSync/execSync/Bun.spawnSync BLOCK the main thread, so bun's in-process per-test timeout can never fire while one waits — a hung child (stdin read, network probe, dead daemon) wedges the whole shard until the runner's external wall-clock SIGKILL. This exact class reached main: free-tests run 33262077256, test/gstack-memory-ingest.test.ts (normally 2.3s) held shard 2 at the 360s wall while its five siblings finished in ~65s. Mechanical sweep in two waves (12 + 4 fan-out agents, every edit verified against its call site): default timeout: 30_000 (matches the free runner's per-test budget), 120_000 for genuinely slow ops (installs, builds, playwright, provider CLIs), helper wrappers fixed ONCE where call sites route through them. Sites that only LOOK like calls (string fixtures, grep needles, comments) were skipped with reasons — the enforcement commit that follows marks them exempt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: sync-spawn timeout tripwire — the wedge class stays extinct Free scanner over all test trees (test/, browse/test/, design/test/, make-pdf/test/, ios-qa, browser-skills): every spawnSync/execSync/ Bun.spawnSync call site must carry a timeout within a 30-line options window, or an explicit '// tripwire-exempt: <reason>' marker. Comment lines are skipped; exemptions are counted and ratcheted shrink-only (ceiling 6 = the 6 string-fixture/grep-needle sites where the pattern is CONTENT, not a call — marked in this commit). A scan-sanity test pins that the scanner still sees >100 real call sites so it can never rot to a vacuous green. Companion to the 436-site sweep in the previous commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: paid-lane flake telemetry — record-level attempts, flaky_retries, report surfacing bun --retry leaves a retried pass INVISIBLE in its output: a fail-then-pass prints the error detail but no (fail) result line and recaps as a clean pass (probed live on 1.3.10). So attempts are recorded where they cannot lie: EvalCollector.addTest stamps a 1-based attempt on same-name re-records (a retried test runs its body again and re-records), finalized runs carry flaky_retries, printSummary warns loudly, and the fail-closed slices report lists every passed-only-on-retry test — recorded and ranked, never blocking and never silent. Cross-model confirmed (codex reached the same don't-parse -the-stream conclusion independently). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: free-lane flake ledger — retry ON in CI, flaky-passes recorded and uploaded The runner's attribution-gated flaky-retry pass (cap 5, truncation veto) was OFF in the required lane and its FLAKY-PASS evidence was console-only — so a single timing flake red the merge gate while repeat offenders stayed unenumerable. free-tests.yml now sets GSTACK_FREE_RETRY_FLAKY=1 and points GSTACK_FLAKE_LEDGER at runner.temp; every flaky-pass appends a JSONL entry (SINGLE writer: the parent runner — no concurrent-append hazard by construction; fail-open with a loud warning so a broken ledger can never red the lane) and the artifact uploads UNCONDITIONALLY — a flaky-pass run is green, which is exactly when the evidence matters. Wiring pinned by free-tests-workflow-wiring; ledger behavior unit-tested incl. the fail-open path. Matches 2026 industry practice (retry for data, quarantine out of merge-blocking but never out of logging) with the repo's own receipts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: eval:flake-rank — the flake-telemetry dial Aggregates per-test series across every finalized eval-store run (shard dirs included) plus the free flake ledger: runs, fails, RETRIED PASSES (the flake signature), avg duration — ranked retries-first. This is the readable dial behind two policies: a flaky pass never blocks a merge but is always ranked here, and the WS16 required-check promotion needs weeks of clean flake-rank, not vibes. --json for machines, --dir for downloaded CI artifacts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: two-phase session timeout — silent APIs die at the startup grace, named The single spawn-armed timer charged API queue latency to the work budget: the recurring '0 turns / $0.00 / x3 attempts' failure with four budget-bump receipts (180->300s, 240->360s, 300->420s, 90->300s). Split: startup phase (no NDJSON byte yet) kills EARLY at min(grace, timeout) with the distinct exitReason 'timeout_startup' — an availability verdict, not transcript archaeology — and the work phase arms on the first byte for the REMAINING budget, so total wall never exceeds the timeout (tier envelopes are margin-free: tests pass timeout: CAPTURE_MS and bun-budget the same tier). Local grace 90s (observed queue latency 60-90s), CI floor 300s (TODOS-filed; shared runners queue harder), both pinned by the new grace tests with fake -claude shims covering the late-first-byte and silent-API paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: census integrity — 17 phantom selection keys deleted, reverse invariant added, gitignored dep patterns replaced, local map forks derived The merge-blocking gate census counted tests that could not run. Deleted (critic-verified against both quoted-occurrence and dep-registration liveness): 7 *-prosons-format keys with no declaring test, ship-plan- completion/-verification, review-plan-completion, design-shotgun-path/ session/full, autoplan-core (dead ~10 months), e2e-harness-audit (its namesake is a FREE-suite file), plus 2 dead LLM-judge keys and 2 free-file keys (budget-regression-pty, global-discover) misplaced in the PAID maps. Census: 191 -> 174 keys, gate 86 -> 78 honest. The new reverse invariant in touchfiles.test.ts makes the class structurally impossible: every key must be quoted in a living paid test file OR registered to an existing paid test file via its dep list (the constructed- name binding the 2026-08 self-registration sweep established) — zero exceptions needed today, with a live-file check on any future exception. Also: '.agents/skills/**' dep patterns replaced with the generator (scripts/gen-skill-docs.ts) — .agents/ is gitignored, so those patterns could NEVER match a git diff and review-template edits silently stopped selecting codex/gemini tests; the codex/gemini local touchfile maps are now DERIVED from the canonical map (loud throw if a key vanishes) instead of hand-forked copies that had already drifted. ios-qa-e2e demoted gate -> periodic: its gate declaration was never executable in CI (hardware exclusion only applies at tier=periodic), so every Linux PR planned a hollow shard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: routing journeys lose their answer key and end at the routing decision The journey tests exist to catch skill-DESCRIPTION regressions (touchfiles: */SKILL.md.tmpl), but the fixture CLAUDE.md shipped an explicit prompt->skill lookup table — with the answer key in context, a badly regressed frontmatter description still routed correctly, so the tests could not fail on the exact class they select for. The fixture now carries only the generic invoke-skills nudge; the frontmatter carries the routing load. Also capped all 10 journeys at maxTurns 2 / tools [Skill, Read]: only the FIRST Skill call is asserted, so 5 turns of Read/Bash/Glob/Grep was pure spend — roughly halves each journey's cost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: retire decided A/B experiments; vendor the pre-cut fixture; ban raw-SHA fixtures Three one-shot decision experiments kept re-running weekly as N=1 stochastic comparisons — flaky by construction with near-zero remaining information: skill-e2e-auq-repetition-cut-ab (its own header: gate "passed pre-landing, approved 2026-08-25"), skill-e2e-preamble-script-ab ("demoted post-Phase-3"), and opus-47's fanout arm-vs-arm (parA >= parB across two SINGLE stochastic runs — a coin flip). Deleted, with their selection keys; the SDK overlay-harness stays as the maintained instrument for the next experiment, and opus-47 keeps its routing-precision cases. verboseSkill() now reads the VENDORED test/fixtures/auq-pre-cut-...-SKILL.md instead of `git show ab66193e^:...` — a branch-local ref that dies on branch prune and already failed on shallow clones. New free tripwire (test/git-ref-fixture-tripwire.test.ts) bans the raw-SHA fixture class outright: quoted SHA:path rev-specs and gitRef-style hex defaults in the test trees fail the suite with the vendor-instead instruction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: demote plan-ceo-review-expansion-energy to periodic Opus generator + a subjective 2-axis >=4/5 LLM-judge threshold sat in the MERGE-BLOCKING gate — the exact class its sibling posture tests were demoted for, with a receipt (a +21-line preamble change once flipped the score). CLAUDE.md's own tiering rule: Opus model test -> periodic. The weekly lane keeps the regression signal; merges stop paying a judge- temperament tax. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: paid shards get per-shard TMPDIR + CHROMIUM_PROFILE isolation and a kill-path cleanup backstop The free runner treats this isolation as MANDATORY (two concurrent shards on one Chromium profile kill each other's browser; shared tmp cross-contaminates) — the paid lane had none of it. Doubly load-bearing here: a shard that hits its 30-min wall is group-SIGKILLed, so per-test afterAll cleanup never runs; the rmSync backstop is the only thing keeping wedged runs from accumulating full git-repo workspaces in the shared tmpdir forever. This is the DAG prerequisite for raising EVALS_JOBS (next commit) — more concurrency on shared state amplifies exactly the shared-tree race class opus-47 exhibited. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: paid-runner defaults 4x4 -> 8x2 — halve the local gate worst case 39 of 75 skill-e2e files hold exactly ONE test, so within-shard concurrency was dead weight for most shards: 4 jobs x 4 concurrency yielded only ~4-6 real in-flight sessions and a 13-wave local gate worst case (~6.5h). 8 jobs x 2 gives ~10-13 in-flight — under the documented-safe ~15 — and ~7 waves (~3.3h worst case). CI lanes keep their explicit EVALS_JOBS env (2 per slice; 4 for gate-census); this changes local defaults. Rollback trigger: sustained 429 storms in the WS1 telemetry across 2 PR cycles. test/eval-detach-timeout-floor.test.ts recomputed green (the raise LOWERS the worst-case floor). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: SHA-pin every action in the secrets-bearing eval lanes evals.yml and evals-periodic.yml execute PR-authored code with three provider API keys in env, yet rode mutable action tags (@v7/@v8/@v2/@v4) — while quality-gate.yml, osv-scanner.yml, and dependency-review.yml already model the SHA-pin pattern. All 30 uses sites across both lanes now pin the exact commit (tag noted in a trailing comment); dependabot's github-actions ecosystem keeps them fresh via PRs instead of silent tag moves. Pulled forward from the plan's endgame on the CEO-review + outside- voice agreement: supply-chain pins on secret lanes go first, not last. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: sweep wave 3 — the execFileSync family gets timeouts (90 sites, 17 files) The tripwire's regex covered spawnSync/execSync/Bun.spawnSync but not execFileSync — an entire blocking sync-spawn API family that could reintroduce the shard-wedge class undetected (ship review army). Same mechanical recipe as waves 1-2: timeout: 30_000 default, 120_000 for slow ops, shared wrappers fixed once, string-needle sites skipped with reasons. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: review-army + adversarial test hardening - Tripwire scans execFileSync too (ceiling 8: two more grep-needle string exemptions); merge-introduced timeout-less spawnSync in question-preference-hook fixed — the tripwire caught a site that landed on main AFTER the sweep, on its first day. - gstack-detach gains TWO watchdog kill regression tests: TERM-immune grandchild (the killpg-after-grace escalation) and the leader-dies variant (the pgid-at-spawn fix — the case the first test cannot see). - eval-flake-rank gets its unit suite (final-attempt accounting, artifact exclusion, shard recursion, recency bound). - Groupkill/startup-grace shim markers are per-run unique (pid-suffixed sleep durations): sibling Conductor worktrees run free suites with no machine lock, and fixed markers let one run pgrep/pkill the other's shims — a cross-run flake inside the anti-flake tests. - flake-ledger test pins the project-scoped local default; stale empty section headers in touchfiles-data deleted (they invited entries under deliberately retired categories). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: adversarial-review runtime fixes across the telemetry + kill paths - session-runner: exit-labeling keys off 'exit', not 'close' — an orphan holding the pipes could relabel a REAL exit (auth failure) as 'timeout_startup' availability noise; the kill path still always group-kills and cancels the reader (labeling and unblocking are separate concerns). Work phase arms on a flag, not firstResponseMs===0 (a same-ms first byte left the startup timer live all run). The CI startup grace is now a real FLOOR (Math.max), matching its name and pinning test. - gstack-detach: pgid captured AT SPAWN (== child pid under start_new_session) — resolving it after the grace raised ESRCH once the leader died on SIGTERM, orphaning TERM-immune grandchildren forever. - test-free-shards: ledger entries carry branch + git_sha (rev-parse split: '--abbrev-ref HEAD HEAD' printed the branch twice and recorded it as the sha); local ledger default is per-PROJECT, not the machine-global tmpdir. - eval-flake-rank: per-LINE ledger parse (one torn JSONL line vanished the whole series), 60-day recency bound (transcript-bearing files are MBs), shared isFinalizedEvalResultFile predicate (the artifact-taxonomy rule lived in three places); eval-store exports the predicate and finalize stops computing flakyRetries twice; paid-shards cleanup uses async rm (a SIGKILLed shard's git-workspace teardown blocked every sibling's stream classification on the parent event loop). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: CI trust-boundary + fail-closed repairs (adversarial findings) - Token/exec separation restored: slices-report (runs PR-authored code: bun install + the reconcile runner) drops to contents:read; the PR comment moves to a NEW slices-comment job holding the write token with ZERO repo code — no checkout, no bun, only downloaded artifacts + jq/gh. $GITHUB_ENV/BASH_ENV persistence is job-scoped, so the split is the boundary. The matrix-era report job had this property; the consolidation had regressed it. Pinned by the wiring test. - Reconcile exit captured via PIPESTATUS[0] in BOTH lanes: GitHub's default run-step shell has no pipefail, so `$?` after `| tee` was tee's exit — the fail-closed gate was silently fail-open. Wiring test pins it. - PR comment: final-attempt accounting restored the dropped COST accumulation (the dial read $0 forever), flaky passes render as the warning they are (never as failures), and a malformed tests[] artifact skips that file instead of aborting the whole comment under bash -e. - Remaining mutable action tags pinned (free-tests upload-artifact, ci-image checkout/docker trio — the image publisher holds packages:write and feeds the secret-bearing lanes). restore-deps fallback installs --frozen-lockfile; register-gstack-skills validates skill names before its rm -rf. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump version and changelog (v1.77.0.0) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: update project documentation for v1.77.0.0 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: cross-model doc-review fixes — flake-ledger env knobs, CI retry-on note, stale version comment Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: correct CHANGELOG receipt numbers to measured values Gate census keys: 78 -> 77 (bun-imported E2E_TIERS count). Sweep receipt: 586 sites/176 files -> 499 sites/146 files, measured by running this branch's spawnsync-timeout-tripwire against origin/main (exit 1, 499 violations across 146 unique files; green on this branch). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: slices-comment creates the PR comment via REST — the write-token job has no git context The token/exec split gives slices-comment NO checkout by design, and gh's pr-comment subcommand resolves the repo FROM git — it died with 'not a git repository' on PR #2746's first run (the update-existing PATCH path was already explicit-repo REST and worked). Create now posts through gh api repos/.../issues/N/comments, and the wiring test pins that no git-context-requiring comment call can creep back into the job. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: startup-grace probes clear CI for local semantics; new probe pins the floor clamp The two shim probes pass explicit 2s/4s graces, but in CI the runner clamps any explicit grace up to the 300s floor (deliberate adversarial-review fix), so 'silent API killed at the grace' died at the 30s work cap instead of 2s — a deterministic red on every CI run, green locally. The probes now pin LOCAL semantics with CI cleared (same save/restore pattern as their PATH shim), and a fourth probe pins the clamp itself: CI=1 + 2s grace + 6s timeout must kill at the 6s cap, still in the startup phase — proof an explicit low grace cannot bypass the floor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Garry TanandClaude Fable 5
|
07b59e396c |
v1.75.0.0 feat: ponytail import wave — simplification review lens, arm benchmark, reuse ladder, instruction-tier digest (#2722)
* feat(autoplan): eng review always runs last — the gate reviews the final amended plan Reorder the pipeline to CEO -> Design (if UI scope) -> DX (if developer-facing scope) -> Eng. The old order (CEO -> Design -> Eng -> DX) let DX findings land AFTER the required gate signed off, so eng validated a stale plan. Accept-all semantics made explicit: every AskUserQuestion resolves to the recommended option; premises no longer pause the pipeline mid-run (clearly-wrong ones queue as User-Challenge items at the single Final Approval Gate). Eng's Codex voice now sees the DX consensus summary. New free static test pins the order; the chain E2E gains DX-between and Eng-terminal assertions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(review): simplification specialist — advisory over-engineering lens with ponytail's tag vocabulary New 8th Review Army specialist (DIFF_LINES > 100, --simplification force flag) hunting unrequested STRUCTURE only: delete/stdlib/native/speculative/shrink closed tags, one-line findings, lines_removable field. speculative: replaces ponytail's yagni: tag — we import the lens, not the posture; coverage stays sacred (Completeness Gaps owns it, suppressions inlined, shrink needs >=5 lines). Advisory carve-out in the merge step: advisory findings are excluded from quality_score and the findings-count header, render with an [ADVISORY] label, and are ASK-only in Fix-First. Zero-findings case prints the lens-scoped 'Simplification: lean already — nothing to cut.' from the PARENT (the specialist keeps the exact NO FINDINGS contract); with findings, the parent prints 'net: -N lines possible' summed from lines_removable. Tests: static pins for the carve-out + early-out contract (gen-skill-docs), two periodic e2e cases with planted fixtures — activation (over-build traps: hand-rolled Intl, one-impl abstract, dead config) and false-flag precision (a lean ETHOS 'choose A' diff must yield NO FINDINGS). Inspired by dietrichgebert/ponytail's /ponytail-review. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(preamble): reuse ladder in Search Before Building — rungs 2-5 of ponytail's ladder, completeness kept Tier-3+ skills gain a per-edit reflex the section only stated as research discipline: before writing new code, stop at the first rung that holds — repo helper, stdlib, native platform feature, installed dependency — then build the COMPLETE version of what remains. The closing clause is the explicit reconciliation with Boil the Ocean: the ladder governs structure, never coverage. Rungs 1/6/7 (YAGNI / one line / minimum that works) are deliberately NOT imported. Also ports ponytail's root-cause rule: one guard in the shared function beats a guard in every caller. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(preamble): bounded-closer output rule for tier-2+ skills After completing work, skills report in a few short lines — what changed, what was skipped, what to watch — and cut any explanation that outgrows the change. Explicit exemptions protect every mandated output: decision briefs, completion-status blocks, user-requested explanations, and report-shaped skills' report formats (the report IS the work in /qa-only, /plan-*-review, /retro, /document-generate). Rationale is signal-to-noise, not tokens: ponytail's own benchmark shows terse prose alone doesn't cut cost (caveman arm: -20% LOC, +7% tokens), and independent replications found its 'skipped on purpose' essays ate the code savings. Includes a good/bad closer example pair per the model-overlay guidance that a positive example beats a 'don't be verbose' instruction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(resolvers): terse-mode savings claim matches measurement — 2.6KB, not 3-5KB Measured on the v1.71 render: --explain-level=terse saves exactly 2,611 bytes per tier-2+ skill. The old ~3-5KB claim predated the preamble restructuring. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(retro,preamble): gstack-shortcut debt ledger — accepted shortcuts leave a joined trail When the user accepts an option that is BOTH Completeness <= 7 AND a durable-scope call, the decision ledger entry (gstack-decision-log, ceiling + upgrade trigger in the rationale) is the source of truth, and the agent marks each cut corner in code with gstack-shortcut(dec-<id>): <ceiling>, upgrade when <trigger> — same edit, no follow-up question, never agent-initiated. /retro Step 11.5 harvests markers into a debt ledger (grep || true — zero matches is the healthy case; skill installs and docs excluded), joins on the decision id so nothing double-counts, tags unlinked and no-trigger rot risks, and closes with 'N markers, M with no trigger.' /review suppressions: a marker with ceiling+trigger downgrades a would-be Completeness Gaps finding to acknowledged debt. Redaction test pins that the marker ships untouched (the ledger is the point) — it does not match the TODO(owner) hygiene shape. Format from dietrichgebert/ponytail's ponytail-debt; store inverted to gstack's existing decision ledger. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: refresh golden ship baselines after preamble additions (reuse ladder + bounded closer) The golden-file regression test pins the rendered ship skill byte-for-byte; the WS3/WS7 preamble sections are deliberate changes, so the baselines re-capture per the goldens' own update protocol. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(hosts): instruction-only tier — a 2KB committed rules digest any agent host can read New agents-digest/gstack-AGENTS.md (1,765 bytes, hard 2,048-byte budget): gstack's ethos one-liners, the reuse ladder, and voice rules for hosts with no install arm — Zed, Amp, Jules, or any AGENTS.md-reading agent. Generated by scripts/gen-agents-digest.ts, auto-refreshed by gen:skill-docs, committed like llms.txt so setup's explainer arms can point at it before any toolchain exists. First line carries the gstack version as its own staleness nudge. Delivery is print-path + user-performed copy ONLY: setup never writes or overwrites a user's AGENTS.md (a test pins this — no cp/ln/mv/redirect into AGENTS.md anywhere in setup). openclaw and hermes explainer arms print the path; slate keeps routing to the full Claude install and gbrain ships from its own repo. HostConfig gains the optional install.instructionTier slot, declared by both instruction-tier hosts. README host table now matches what setup actually does. Inspired by dietrichgebert/ponytail's instruction-tier AGENTS.md fallback — one generated source, never per-host hand copies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(preamble): AskUserQuestion repetition cut — gated, passed NOT-WORSE A/B Removes the duplicate statements v1.71's compaction left in the AskUserQuestion Format section: the completeness rule restated in the prose triad, the auto-decide marker syntax stated twice, the Conductor-flakiness explanation stated twice, and the self-check's full triad restatement. Every verbosity floor and all 14 format pins stay (Layer 0 green). The gate this decision rested on ran before landing (new periodic skill-e2e-auq-repetition-cut-ab.test.ts, pre-cut ref |
||
|
|
1d41ee3ab3 |
v1.63.0.0 feat: GStack 2 fork port wave — egress receipts, context-bill, sharded gate, /health fix (#2541)
* test(helpers): shared skill-census helper with three explicit counts
physicalSkillFiles (symlinked dirs included, root router included),
authoredSkills (realpath-deduped, router excluded), registryEntries
(what ./setup registers: unique frontmatter names + _gstack-command).
One counting authority for the hermetic seeder, context-bill ground
truth, and the catalog-budget test — connect-chrome's dir symlink and
the root router otherwise produce three subtly different hand-rolled
censuses. Ported-wave foundation (C11).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(evals): stop the harness grading itself
findPreviousRun excluded only the file being written, by name, so every
suite compared against _partial-e2e.json — the current run's own
accumulator, relabelled with the current tier just before each flush.
That is why every block read '+$0.00, +0s, Stable run, no regressions.'
This harness has never been able to detect a regression, and reassuring
output that cannot fail is worse than none. In-progress runs are now
excluded by role, and a run with nothing to compare against says NO
BASELINE instead of claiming stability.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit f3140b5245221fff7fb9411c7ec07c2ca11587b5)
* refactor(evals): shared partial-run predicate + finalized-run lookup
isPartialEval(data, filename) is the one place that decides what counts
as an in-progress accumulator (the _partial flag OR a _partial-prefixed
filename), and findLatestFinalizedRun(evalDir, tier) is the one place
that finds the newest real run — scanning the eval dir plus one level of
shards/<slug>/ subdirs, where the sharded paid runner points each
shard's collector. skill-budget-regression.test.ts's hand-rolled
findLatestRun (flag-blind: a flagged-but-renamed accumulator passed its
name check) is replaced by the shared helper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit b55fcf6966366fd21a8cdc46de61aab6e1b1d100)
* feat(evals): register shipped skills for hermetic PTY children
Hermetic children get a config dir that deliberately seeds no skills —
right for children that install their own, fatal for the PTY family that
TYPES /office-hours or /plan-ceo-review: claude rejects the command as
Unknown before any model turn, so the plan-family gate smokes measure
nothing. hermeticSkillsConfigDir() is a second, opt-in config dir under
the same runRoot that mirrors ./setup's registration exactly (real dir
per registry name, SKILL.md + sections/ symlinks, frontmatter-name
resolution, _gstack-command root alias), driven by the shared
skill-census so connect-chrome's dir symlink collapses the same way
setup's idempotent overwrite does.
Ported from fork commit 03c4eca2, tree walk rewritten for the upstream
layout (top-level <skill>/SKILL.md dirs, no skills/ tree). Unit tests
are new: seed shape, census parity, symlink resolution, connect-chrome
collapse, idempotence, no-API-key seed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 93dae6107b30ce453a07c2d342b60262bba6ce0b)
* feat(evals): seedSkills opt-in for PTY slash-command tests + tripwire
Wire ClaudePtyOptions.seedSkills through launchClaudePty: when set (and
hermetic, and no per-test CLAUDE_CONFIG_DIR override), the child gets
hermeticSkillsConfigDir() so typed /skill slash commands resolve instead
of dying as Unknown command before any model turn. Opted in at the three
runPlanSkill* helpers and the four direct-launch slash-command tests
(plan-design-with-ui, plan-ceo-mode-routing, autoplan-chain,
ship-idempotency).
New static tripwire (test/pty-skill-seeding-wiring.test.ts): any test
file that sends a slash command over the PTY must route through a
runPlanSkill* helper or pass seedSkills: true — an unseeded slash-command
test spends money and measures nothing. hermetic-wiring.test.ts now
blesses the repo-tree seeding path explicitly (config dir under runRoot,
symlinks into the repo checkout, never operator ~/.claude).
The CI "Register gstack skills for PTY smoke" step keeps a keep-me note:
container cross-mount symlinks defeat the TUI scanner and HOME is not
hermeticized, so the real-file copies there must survive this change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 63c52269daaffb833b3105ea9b4b99be6df8fec7)
* refactor(evals): single shared paid-test-set module
test/helpers/paid-test-set.ts is now the one definition of which test
files are paid (the exact globs package.json's test:gate expands).
scripts/test-free-shards.ts derives its free/paid exclusion from it
instead of a private regex list, dropping the dead
browse/test/security-review-fullstack.test.ts pattern (file no longer
exists). The sharded paid runner derives its enumeration from the same
module, so a file added to one list can no longer silently miss the
other.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit a7f36479a6a1f3656452370f5883371f3cb65623)
* feat(evals): env-driven lazy eval dir + shard-aware store and tooling
Importing eval-store no longer spawns the gstack-slug subprocess: the
module-level DEFAULT_EVAL_DIR constant is now a memoized defaultEvalDir()
resolved at collector construction. Resolution order: explicit
constructor arg, then GSTACK_EVAL_DIR, then slug detection — so the
sharded paid runner can point each shard child at its own
<evalDir>/shards/<slug>/ dir with plain env, no --preload.
Runs collected under a shards/ subdir record their slug in the eval
JSON (EvalResult.shard). findPreviousRun scans one shards/<slug>/ level
and prefers same-slug priors, so each shard baselines against its own
history instead of whichever shard flushed last. eval:list,
eval:summary, and eval:compare enumerate the same one level of shard
subdirs; eval:compare's no-arg mode also stops picking an in-progress
accumulator as the after-run.
eval-watch stays flat (documented follow-up): it tails a single dir for
live progress and gains nothing from per-shard baselines until the
runner emits a merged stream.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit e1f53f7d9c7fe6b65877d843f2e25bd2e2d12ffd)
* feat(evals): sharded paid tier runner
scripts/test-paid-shards.ts runs the gate/periodic tier one Bun process
per test file, with an EXTERNAL wall-clock timeout that SIGKILLs the
shard's detached process group and an aggregate that distinguishes
passed / failed / timed-out / never-started — partial execution can no
longer read as a pass. Bun's native --shard/--isolate covers none of
this: no process-group kill (hung claude/codex PTY grandchildren
survive in-process isolation), no never-started taxonomy, no per-shard
env. Each shard child gets GSTACK_EVAL_DIR=<evalDir>/shards/<slug>/
(slug = test filename sans extension, stable across runs) so shard
baselines compare against their own prior runs.
Output classification lives in scripts/test-strict-output.ts (strict
exit-code derivation, incremental fail-line classifier, child signal
forwarding) so the runner and any future strict bun-test wrapper share
one implementation. Enumeration derives from the shared paid-test-set
module; tier exclusion fires only on an explicit whole-file
EVALS_TIER === '<other>' guard.
package.json gains test:gate:sharded / test:periodic:sharded, and
eval:bg:gate / eval:bg:periodic now run the sharded scripts with detach
timeouts sized to the worst case (gate: 49 shards x 30min / 4 jobs ~
6.2h -> 25200s; periodic: 59 -> 28800s).
test/paid-shards.test.ts pins enumeration, tier classification, and the
kill-and-continue property with a real busy-loop shard.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 5e76bd5931836257f896cedfe4e93912cb759c70)
* feat(security): hash-chained egress receipt ledger (core)
Port lib/egress-receipt from the v2 fork as TypeScript: writeReceipt
(sync, fail-closed via typed EGRESS_RECEIPT_FAILED), best-effort
writeOutcome, readLedger/listReceipts/verifyLedger, GSTACK_HOME ->
GSTACK_STATE_DIR -> ~/.gstack resolution, 0600 ledger under a 0700
security dir, and an mkdir spin lock (2.5s budget) with documented
>10s-mtime stale-lock reclaim.
Changes vs the fork:
- lastRawLine tail-reads the final 4KB instead of loading the whole
ledger, so appends stay O(1) as the file grows.
- WARN-at-size: past 25MB writeReceipt emits one self-explanatory
stderr warning per process (what the ledger is, how to inspect it,
rotation TODO); verifyLedger gains a sizeWarning field. Rotation
TODO carries the chain-genesis sketch (new generation's first record
embeds the prior file's tail hash).
bin/gstack-egress-receipt is a bun script bridging shell callers:
write|outcome subcommands, exit 3 + EGRESS_RECEIPT_FAILED on stderr on
failure; --no-payload records sha256:null for git-class ops.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 619726a3d77d987a2e50151a5727b3faaaf5fc6a)
* chore(bin): delete dead brain-consumer/reader scripts
bin/gstack-brain-consumer and bin/gstack-brain-reader are byte-identical
dead scripts that POST the repo URL + a Bearer token to a /ingest-repo
endpoint gbrain removed (docs/gbrain-sync.md already documents the
removal in past tense). No live references remain; CHANGELOG mentions
are historical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 254ddc69fc5a0270fcc973e36b6a81766d835d2d)
* feat(security): shared shell receipt helpers
bin/gstack-egress-lib.sh (sourced library, gstack-gbrain-lib.sh
precedent) provides _receipted_curl and _receipted_git: write the
egress receipt BEFORE the send via gstack-egress-receipt, hand curl the
SAME payload file via --data-binary @file so the receipt hash matches
the wire bytes exactly, then append a best-effort outcome. Per-call
fail policy: 'closed' refuses the send (return 3, problem/cause/fix
message on stderr) and 'open' warns and proceeds. Payload temp files
are consumed immediately per call — no EXIT traps, since callers like
gstack-telemetry-sync own their own EXIT trap and a sourced trap would
clobber it.
Tested end-to-end against a local Bun.serve listener: receipt sha256
equals the sha256 of the bytes the listener received, fail-closed
refusal never touches the network and carries the problem/cause/fix
stderr shape, fail-open warns and proceeds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 6d067dce2d4c8815dec98be551763c85a3671357)
* feat(security): receipt core shell sinks
Wire the three core bash egress sinks through gstack-egress-lib.sh:
- gstack-telemetry-sync: the batch POST now writes the payload to a
temp file, receipts those exact bytes fail-closed, and hands curl the
SAME file. On refusal nothing is sent and the cursor does not
advance, so the batch stays buffered for the next run. The HTTP
status is recorded as the receipt outcome.
- gstack-update-check: fail-open receipts (warn + proceed) on the
Supabase ping POST, both VERSION curls (via a local
_receipted_version_fetch helper that skips non-network schemes), and
git ls-remote. The ping receipt is written inside the backgrounded
subshell, so it can never block the script's exit.
- gstack-brain-sync: fail-closed git-class receipts. The push receipt
is written BEFORE the commit consumes the queue, so a refused receipt
leaves the queue intact and the next run retries the whole drain
(pinned by a new queue-intact-on-refusal test, including the
problem/cause/fix refusal message shape). The retry-path fetch and
retry push carry their own fail-closed receipts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 3c60f699acceaf1c92a218874711e05fc17dca5d)
* feat(security): receipt TS module sinks + tunnel
writeReceipt (fail-closed, sha256:null — a subprocess or SDK owns the
wire bytes) before every TS-module network-bearing operation:
- bin/gstack-gbrain-sync.ts: before the gbrain code walk that ships
repo content to the user's gbrain DB (may be remote Postgres). A
refused receipt fails the stage with status refused-egress-receipt.
- bin/gstack-memory-ingest.ts: before the gbrain batch import of
transcript pages. A refused receipt returns a system_error verdict
without spawning the import.
- browse/src/server.ts: before both ngrok.forward call sites (start-up
BROWSE_TUNNEL=1 path and the /tunnel/start endpoint). A receipt
failure lands in the existing catch that tears the tunnel listener
back down and refuses the start.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 5677d618a48fcd0ae2b068bf868781d90f809cb5)
* feat(design): receipted fetch for OpenAI calls
design/src/receipted-fetch.ts wraps every api.openai.com call: a
content-free egress receipt (sink design-openai, sha256 of the JSON
body — hash only, never the body) is written BEFORE the send. Polarity
is FAIL-OPEN: user-facing generation must not die because an audit log
hiccuped, so a receipt failure warns on stderr and the call proceeds.
Streams pass through untouched (response bodies returned as-is;
non-string request bodies receipted as sha256:null rather than drained
to hash).
All ten call sites converted with per-command payload classes:
generate, variants (injected fetchFn passes through), iterate (both
threaded and fresh paths), evolve (image + screenshot analysis), check,
diff, design-to-code, memory.
Unit-tested with injected fetch: receipt-before-send ordering, stream
passthrough, and fail-open on an unwritable ledger.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit c0e5ff6639414ac2fd98e8ac3affb51401746b55)
* feat(security): receipt admin scripts + user git-ops (zero exceptions)
Wire the remaining shell egress through gstack-egress-lib.sh:
- gstack-gbrain-mcp-verify: both JSON-RPC probe POSTs (initialize +
tools/list) receipted fail-closed via payload files (hash == wire
bytes). A refused receipt lands in the NETWORK class — no send.
- gstack-security-dashboard / gstack-community-dashboard: the
community-pulse GETs receipted fail-open (read-only stats must not
break over an audit hiccup).
- gstack-gbrain-supabase-provision: api_call receipted fail-closed.
Each retry attempt hands the helper a fresh copy of the body file
(the helper consumes its payload). The receipt hashes the request
body only — the PAT never reaches the ledger or any log. Refusal
exits 8 without retrying.
- git-class sha256:null receipts, fail-open: gstack-artifacts-init
(ls-remote, initial push, fetch/pull recovery, retry push),
gstack-brain-restore (staging clone, existing-repo fetch),
gstack-session-update (self-update pull).
gstack-team-init needs no wiring: every git clone in it is inside an
echoed instruction string, not an executed command.
The lib now self-locates with shell builtins only (no dirname), so
sourcing works under the whitelist-PATH test harnesses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit b8c5e2055b21ab72878b3e46f8047782ee65a11c)
* test(security): egress wiring tripwire + polarity contract
Static-grep tripwire pinning the egress-receipt wiring (threat model in
the header: the ledger is forensic observability of ATTEMPTED egress,
not an exfiltration control):
- Per-sink assertions: every wired TS module imports egress-receipt and
calls writeReceipt; every wired shell sink sources
gstack-egress-lib.sh with each network op under a receipt;
ngrok-proximity check for server.ts; every design api.openai.com call
routes through receiptedFetch.
- Absence assertions: the dead brain-consumer/reader scripts stay
deleted (lstat, so a dangling symlink also fails).
- Polarity table pinned as data (fail-closed: brain-sync,
memory-ingest, gbrain-sync, telemetry-sync, ngrok, mcp-verify,
supabase-provision; fail-open: design-openai, update-check,
dashboards, git-class user ops, context-bill --exact) plus per-file
polarity spot-checks.
- NEW-SINK SCANNER with zero KNOWN_UNWIRED: sweeps bin/, lib/,
scripts/, design/src, browse/src for curl, absolute-URL fetch(, and
git remote ops (never local rev-parse/get-url; heredoc bodies and
message strings excluded) and requires every hit to be receipted or
in a REASONED exemption list where each entry carries its why.
Preamble-generated skill prose documented out-of-scope in the header.
- Shebang tripwire: no bin/gstack-* file may carry a node shebang.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit ff69ceeafaf9c017d539b6ad77ff8f95b680b979)
* feat(cli): gstack-egress reader
bin/gstack-egress (bun) — the auditor's view of the receipts ledger:
- list: one row per receipt (what gstack ATTEMPTED to send), with
--since/--host/--sink filters and --json.
- verify: recompute the hash chain; exit 3 on tamper naming the first
broken line; prints the sizeWarning when the ledger passes 25MB.
- grants: what CAN leave, built on the upstream config keys only
(telemetry, artifacts_sync_mode, redact_repo_visibility,
redact_prepush_hook via gstack-config get) — each grant names its
file, key, and the exact revoke command.
CLI smoke tests spawn the real bin against a temp GSTACK_HOME,
including a broken-chain fixture asserting exit 3.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 9e24eca0f1069fea2ea69e7df4e9b256e93d59a3)
* feat(cli): context-bill — token bill-of-materials (stripped port)
lib/context-bill.ts, ported from the v2 fork and STRIPPED to the tiers
this repo's skills can exercise: ALWAYS-ON (per-skill frontmatter bytes
with dead-key and foreign-host-file flags), EAGER (SKILL.md + any
forced 'for every invocation' references), on-disk totals, --diff,
--budget, and --exact with the calibration table. The fork's
CONDITIONAL/TRANSITIVE/LAZY/FAST-PATH parsers understand only its
dispatcher layout and were dropped; the tier fields stay in the report
shape (empty/zero/null) so re-adding a parser is additive.
TOKEN_DIVISORS and their provenance docblock kept; --help notes
recalibration via --exact's calibration block.
Three upstream fixes over the fork:
(a) findSkillDirs treats the walk ROOT as a container — the repo root's
router SKILL.md is billed AND its children are walked (the fork
short-circuited and billed one skill); walkMd skips node_modules
and dot-directories.
(b) installed-tree layout: subdirs that are their own repo checkout
(a gstack/ clone inside ~/.claude/skills, detected by .git) are
skipped, and directory symlinks (connect-chrome) are followed with
a container-recursion cycle guard.
(c) ROUTER_KEYS widened to the upstream frontmatter contract {name,
description, version, allowed-tools, triggers, preamble-tier}.
--exact writes an egress receipt (sink context-bill-exact, host
api.anthropic.com) BEFORE any count_tokens POST; if the receipt cannot
be written the run degrades to the offline estimate with a warning —
nothing is sent unrecorded. bin/gstack-context-bill is the bun shim.
Tests: fixture-tree ledgers, the three fixes, --diff/--budget exit
codes, --exact with injected fetch (envelope subtraction, receipt
ordering, fail-open degradation), CLI smoke test, and ground truth
against THIS repo via test/helpers/skill-census.ts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 675c19876b87ec927b555f5f64c7f93130b3de90)
* test(catalog): aggregate discovery-surface budget with ratchet protocol
Every host loads every skill's frontmatter name + description at
discovery, every session. applyCatalogTrim in scripts/gen-skill-docs.ts
shapes each description and the 160KB per-file warn covers body size,
but nothing capped the aggregate frontmatter — the catalog could grow
one reasonable-looking description at a time. This test is that
enforcement layer.
Measures the catalog via test/helpers/skill-census.ts authoredSkills
(symlink-deduped, root router counted separately as the _gstack-command
alias line item): 53 skills + router = 4,420 bytes = 1,105
token-equivalents today, asserted <= 1,150 (~4% headroom). Per-skill
sub-cap of 260 bytes (largest today: design-consultation at 229), plus
a non-empty-description check.
Failure messages are self-service ratchets: they print the new total,
the delta, and the update protocol (bump the constant AND the
derivation comment in the same commit; trim instead of grow for
existing descriptions). Parser handles folded block scalars
(description: >-) for fork parity; import-free by design so it
survives generator refactors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit c106fb36f768181b80c257e5cff1cde4f435f9c0)
* fix(browse): extension token bootstrap moves to pinned-origin POST; /health carries no token
GET /health is now liveness/status only in every mode — both token
carve-outs (headed-mode disjunct AND chrome-extension:// Origin
disjunct) are removed. Token bootstrap is POST /extension-token on the
local listener: the Origin header must be exactly
chrome-extension://<GSTACK_EXTENSION_ID> and the Host header's hostname
must parse to 127.0.0.1 or localhost (parsed via new URL, never literal
equality — Host arrives as '127.0.0.1:34567'). Wrong origin/host → 403
with no detail. The tunnel surface 404s the endpoint (not in
TUNNEL_PATHS, verified by test).
The extension ID is pinned by a new "key" field (RSA public key) in
extension/manifest.json; browse/scripts/extension-id.ts reproduces the
ID derivation (first 16 bytes of SHA-256 of the DER public key, hex
mapped 0-9a-f → a-p). The private key is not committed anywhere —
unpacked/baked-in loads only need the public key.
Extension side: background.js bootstraps and refreshes the token via
POST /extension-token (403 → disconnected state); sidepanel.js direct
connect path does the same; sidepanel-terminal.js's dead /health token
fallback (read AUTH_TOKEN/authToken keys the server never sent,
hardcoded port) is replaced with the window.gstackAuthToken path.
MIGRATION NOTE: the manifest key pins the extension ID, so existing
installs' side-panel local state (saved port, snoozes) resets once —
explained in-product via a one-time notice (flag
gstack_id_migrated_v162). After upgrading the server, restart the
browser so the old service worker stops polling for a token GET /health
no longer serves.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit e9a0b6847a2d17fe6656a4686b4efd0c8380eb09)
* docs: correct stale compiled-binaries claim; file three egress/eval follow-ups
CLAUDE.md's compiled-binaries section claimed browse/dist binaries are
tracked by git and appear as modified in git status — false since
|
||
|
Garry TanandClaude Opus 4.6
|
f3ee0ee28a |
feat: QA restructure, browser ref staleness, eval efficiency metrics (v0.4.0) (#83)
* feat: browser ref staleness detection via async count() validation resolveRef() now checks element count to detect stale refs after page mutations (e.g. SPA navigation). RefEntry stores role+name metadata for better diagnostics. 3 new snapshot tests for staleness detection. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: qa-only skill, qa fix loop, plan-to-QA artifact flow Add /qa-only (report-only, Edit tool blocked), restructure /qa with find-fix-verify cycle, add {{QA_METHODOLOGY}} DRY placeholder for shared methodology. /plan-eng-review now writes test-plan artifacts to ~/.gstack/projects/<slug>/ for QA consumption. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: eval efficiency metrics — turns, duration, commentary across all surfaces Add generateCommentary() for natural-language delta interpretation, per-test turns/duration in comparison and summary output, judgePassed unit tests, 3 new E2E tests (qa-only, qa fix loop, plan artifact). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: bump version and changelog (v0.4.0) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: update ARCHITECTURE, BROWSER, CONTRIBUTING, README for v0.4.0 - ARCHITECTURE: add ref staleness detection section, update RefEntry type - BROWSER: add ref staleness paragraph to snapshot system docs - CONTRIBUTING: update eval tool descriptions with commentary feature - README: fix missing qa-only in project-local uninstall command Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: add user-facing benefit descriptions to v0.4.0 changelog Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Garry TanandClaude Opus 4.6
|
84f52f3bad |
feat: eval persistence with auto-compare against previous run
EvalCollector accumulates test results during eval runs, writes JSON to
~/.gstack-dev/evals/{version}-{branch}-{tier}-{timestamp}.json, prints
a summary table, and automatically compares against the previous run.
- EvalCollector class with addTest() / finalize() / summary table
- findPreviousRun() prefers same branch, falls back to any branch
- compareEvalResults() matches tests by name, detects improved/regressed
- extractToolSummary() counts tool types from transcript events
- formatComparison() renders delta table with per-test + aggregate diffs
- Wire into skill-e2e.test.ts (recordE2E helper) and skill-llm-eval.test.ts
- 19 unit tests for collector + comparison functions
- schema_version: 1 for forward compatibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|