mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-02 15:04:37 +01:00
bf9a417d542d976dfd9f36ec9e65e35cf790cc46
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Garry Tan
|
55fbe77195 |
v1.91.24.0 test: audit wave — PR evals keep finished work, honest ceilings, EVAL_POLICY v2, faster free suite (#3031)
* chore(scripts): retire hollow and legacy paid package scripts behind one-release stubs (W8a, DX-4, CEO-31) test:codex*, test:evals*, test:e2e*, test:gate, test:periodic, eval:bg, eval:bg:all, eval:flake-rank, eval:watch and test:audit now print their replacement and exit 1 via scripts/retired-command.ts. CONTRIBUTING.md carries the same table; test/retired-commands.test.ts pins both. Eval CLI empty-state hints point at eval:bg:pr. * chore: delete dead update-readme-throughput, eval-watch, orphan fixtures and QA_FUNCTIONAL_INPUTS (W8b, CEO-24) Retired with named behavior: - update-readme-throughput.ts + readme-throughput test: replaced a README anchor that no longer exists from a JSON file that was never produced; the script was a no-op. Guarantee withdrawn (nothing to protect). - eval-watch.ts + its dashboard tests: rendered ~/.gstack-dev/e2e-live.json and _partial-e2e.json, written only by unsharded runs, so it showed nothing for every supported (sharded) entry point. Guarantee withdrawn; live progress is the gstack-detach log or gh run watch. Manual-review rendering stays covered by eval-list and eval-summary tests. - Orphan fixtures (autoplan-pending-artifact-ae.json, autoplan-existing-app/, autoplan-sign-in-ui/, parity baselines v1.47.0.0/v1.57.7.0/v1.69.1.0): no test, touchfile or script reads them; superseded baselines stay in git history. - QA_FUNCTIONAL_INPUTS: exported, never imported. * refactor(test-free): delete dead WORKER_HOSTILE list and buildShardArgs parallel mode Neither had a production caller: full-suite mode runs shard processes, so the worker quarantine was inert and --parallel was never passed (W8b). * docs(test-value): narrow prompt-byte contract to machine-read tokens (D2) Define prompt-byte contract as machine-read tokens (markers, field names, command lines, enum values, file paths, security-boundary strings) and add a sentence-pin glossary entry: convert to structural checks or delete when a behavioral eval covers the behavior. * test(ratchet): count free tests that slice a paid test's source A free test that binds readFileSync(<paid test file>) and cuts it with .slice() tests the paid harness's code out of context even when it imports a lib/ symbol. Record the 15 existing files with reasons; new ones fail with fix text. Self-test covers positive and negative shapes. * test(browse): un-quarantine compare-board; all 16 board tests pass headless (W8d, CEO-24) The GSTACK_COMPARE_BOARD_TESTS opt-in hid a headless mismatch that no longer reproduces: all 16 tests pass on Bun 1.4.0 with the bundled headless Chromium, solo, repeated, and beside other in-process browser files. Protected behavior (the comparison-board DOM contract the plan-design-review/office-hours/design-consultation feedback loop polls) is checked on every free run again. * test: replace silent existsSync early returns with explicit skips or assertions (W8d) - browse/test/build.test.ts: server-node.mjs checks skip only when the bundle is absent and GSTACK_EXPECT_BINARIES is unset; the free CI lane and test:ubicloud build it and set the flag, so a dropped build step fails with the build command instead of passing with no assertions. - gstack-update-check: VERSION is tracked; the guard could only hide a bug. - gen-skill-docs host leakage/frontmatter checks assert the host tree exists (the module-level render always produces it). - cso-docker staged-runtime test uses test.skipIf so reports show it as skipped, not passed. * test(ship): convert ship-workflow-clarity sentence pins to structural checks Add test/helpers/prompt-structure.ts (between, expectTokens, expectAbsent, expectOrdered, expectMentions) with fix text on failure. ship-workflow-clarity now pins headings, step order and machine-read tokens (commands, fields, enum and status strings); safety rules are meaning-level keyword checks. Wording is judged by the 'ship/SKILL.md workflow' judge in skill-llm-eval. Sentence pins: 90 -> 0. * ci: add platform-qualification lane for opt-in ML, gitleaks and Swift tests (W8d, CEO-24) security-bench + security-live-playwright L4 (SECURITY_BENCH), the real gitleaks 8.30.1 memory-ingest cases (GSTACK_TEST_GITLEAKS) and the DebugBridge Swift build invariants (GSTACK_TEST_SWIFT) self-skip in every lane today. No scheduled lane has their prerequisites, so per CEO-24 they move to an owned dispatch + quarterly workflow instead of being deleted. Each job was run locally (ML 10 pass, gitleaks 130 pass; Swift needs macOS). test/platform-qualification-workflow.test.ts pins gate variable, job and file together. * fix(scripts): capture-baseline and preflight-agent-sdk no longer act on --help or typos (W8e) capture-baseline parses its flags strictly: --help prints usage, an unknown or valueless flag exits 2, and only a valid invocation writes a fixture (--help used to write test/fixtures/parity-baseline-current.json). preflight-agent-sdk gains --help, rejects unknown flags, and runs the live SDK query (API spend) only with --live. * test(evals): register the three keyless periodic paid files with case ids (W1b) skill-e2e-auq-consistency, skill-e2e-auq-verbose-vs-carved-ab and codex-e2e-recommendation-substance now carry touchfiles, a periodic tier and kind rule (one trial, unchanged), so editing one selects its own case instead of restoring the full PR gate. The verbose A/B case also registers the vendored pre-cut fixture it reads, and review-army-migration-safety registers the SQL fixture it reads. * feat(evals): narrow PR fallback for free-only inputs (W1a, CEO-13/20, ENG-13) The PR profile now exempts the duration seeds, every free-lane workflow (explicit list; inventory test fails on an unclassified workflow, .yaml and workflow_call included), scripts/ubicloud/** and fixtures declared in the explicit FREE_FIXTURES map. Each remaining unknown file carries a label and fix (needs touchfile entry / add to FREE_ONLY_PR_FILES / real unknown dependency). evals.yml itself still restores the full gate. * ci: delete ci-image.yml; the eval workflows already build and push the same content-hash tag (W8f) evals.yml, evals-periodic.yml and evals-marathon.yml each compute the identical hashFiles() tag and build/push it (with :buildcache and :latest) when missing, so ci-image's push trigger and weekly cron were no-ops (30-day median 0.5 min, 36/36 success). ci-image-tag-binding now pins the three eval workflows. * ci(evals): key periodic/marathon concurrency on ref+event; branch periodic dispatch skips the gate census (W2a, ENG-9, CEO-04) * refactor(paid): drop ~160 unused imports; leaf paid-types/paid-select modules break the paid-* import cycle (W8c) * ci(quality-gate): per-PR concurrency, free runner, drop unscoped bun audit (W2a, W8f) - Concurrency keys on pull_request.number || run_id, so back-to-back main pushes no longer cancel each other's credential scan. - ubuntu-24.04 instead of a billed 8-vCPU Ubicloud runner for a ~1 min job. - bun audit --audit-level=critical ran on every PR regardless of diff, so a newly published advisory turned unrelated PRs red, with its own ignore policy. dependency-review (lockfile PRs, fail >= high) and the weekly OSV sweep remain the dependency gates. * chore(test-free): refresh the free duration seed from the audit's Ubicloud recording Adds the 19 files merged after the last refresh, including the 115 s claude-overlay-setup that packed as ~1 s and doubled a shard (W6a). The seed is re-recorded on the final tree at the end of this wave. * ci: move tiny ubicloud-standard-2 jobs to free ubuntu-24.04 runners (W8f) actionlint, skill-docs, dependency-review, version-gate and pr-title-sync are sub-minute jobs; GitHub-hosted runners are free for this public repo. * feat(test-free): pessimistic unseeded packing, seed growth ratchet and CI health summary - Unseeded files pack at the seed's p99 instead of p75 (W6a). - --ci-plan warns in the job summary past 5 unseeded files; --ci-verify lists shards that outran their prediction by 1.5x (and 30 s) plus every flaky pass absorbed by the serial retry (W6a, W7a job summary). - test/free-seed-ratchet.test.ts fails when a seeded file over 60 s is not on scripts/free-test-seed-allowlist.json, when an entry is stale, and when the allowlist grows versus its merge-base copy unless the new entry is an annotated exception with a todo (CEO-10, DX-7). free-plan runs it with full history so the shrink check is real in the required lane. - Every message names the fix command and docs anchor (DX-6). * feat(evals): state PR coverage mode, counts, reuse and fallback files in the summary and comment (CEO-13) * test: make the tracked-tree skill-doc freshness check local-only; tidy skill-docs header (W8f) Every free CI shard regenerates the tracked SKILL.md files before running tests, so the --dry-run freshness assertion compared the regenerated tree with itself and could not fail there. It now skips under GitHub Actions; skill-docs.yml's git diff --exit-code is CI's freshness gate. Removes a stale concurrency comment in skill-docs.yml. * ci(free-tests): install zsh so the bash+zsh portability arms run (W6f) ~42 zsh-arm tests (aside-probe-shell, qa-browser-preservation, generated-docs-fences) skipped in CI because zsh is not in the runner image. All 141 tests in those files pass with zsh present. * feat(evals): executors write negative receipts at the execution boundary (CEO-25, ENG-6) A completed single-trial rule/judge shard that did not pass now writes its negative receipt beside the pass receipts in the slice artifact, so a cancelled run's FAIL still blocks an older PASS when its report never runs. Receipts merge negatives first and order by (run attempt, completedAt); a same-second FAIL/PASS tie resolves to FAIL. The runner call site is a one-line hunk in scripts/test-paid-shards.ts. * docs(cso): state that contained runtime/scanner execution is unavailable today The scanner catalog is empty while its build inputs await review and the runtime catalog holds no qualified profile until the private evaluator exists, so /cso runs static assessment only. AGENTS.md, README and docs/cso-release-qualification.md now say so; docs/cso-scanners.md already did. * test(helpers): expectMentions splits on sentence ends only Clauses joined by ';' or ':' are one rule; splitting there made a reworded rule fail for punctuation. * test(plan-ceo): convert skill-ceo-section-ordering sentence pins Safety and authority rules (never/do not/stop/only/without/wait/approval) are meaning-level expectMentions checks; other sentence pins are dropped. Headings, labels, option rows, placeholders and machine-read lines (auto-decide annotation, Mode:/approved decisions format, STATUS/SOURCE fields, CROSS-MODEL marker, gstack-ceo-mode-handoff) stay as tokens; ordering checks unchanged. Behavior is covered by the periodic PTY case in skill-e2e-plan-ceo-review-section-loading.test.ts. Sentence pins: 210 -> 23. * test(helpers): expectMentions treats wrapped lines as one sentence Templates hard-wrap prose; only sentence ends and blank lines separate rules. * test(gen-skill-docs): convert sentence pins to meaning checks or drop them Safety/authority sentences become expectMentions keyword checks; other English sentences are dropped (wording is judged by the per-skill workflow judges in skill-llm-eval and the E2E cases in touchfiles-data). Commands, config keys, markers (PLAN MODE EXCEPTION, NO UNRESOLVED DECISIONS, Gate outcome), field names and alternation-based meaning regexes stay. Removed the now-empty 'contains opt-in AskUserQuestion text' test (only pinned a sentence; the resolver's AskUserQuestion wiring is covered by the codex second-opinion tests in the same describe). Lane E's existsSync hunks are untouched. Sentence pins: 216 -> 65 (remaining lines carry machine tokens). * ci(windows): shard the Windows lane via --ci-plan/--ci-run/--ci-verify; move native campaigns to native-qualification.yml - windows-free-tests.yml: a Linux plan job curates and packs the Windows-safe files into 6 shards, one strict serial shard per windows-latest job, and a Linux aggregate verifies exact coverage and every result (8.1 min on one runner today). --windows-only now combines with the CI modes; the plan binds the curated file set (W6e). - Every Windows plan and --list run prints the curated count with its top exclusion reasons in the job summary; a free ratchet fails when the curated subset drops below its floor, with the fix (W6e). - The four Dia/cookie diagnostic dispatch campaigns move to the dispatch-only native-qualification.yml with one exclusive `mode` choice input, so their red dispatch runs no longer read as a red PR lane (W8f). Their workflow pins move to test/windows-native-workflows.test.ts; the boolean-exclusivity guard is replaced by the choice input. * feat(paid): per-slice CI job ceilings and an in-process slice deadline with not_run/hung outcomes and checkpoints (W2c, ENG-2) * fix(pass-rates): segment trial history strictly by EVAL_POLICY version Reader-compatibility floor for EVAL_POLICY v2 (ENG-14/CEO-35/DX-12). Only trials under the reader's own policy version feed labels and alarms; older policy trials stay visible but are never scored or used as a drift baseline; trials from a newer policy are ignored with a printed count and fix text. Reverting the later v2 writer commits leaves a reader that cannot pool v2 records into v1 series (the pre-PR reader at |
||
|
|
0530392821 |
v1.81.0.0 feat: Aside is the browser gstack drives first; every browsing skill, the PDF/diagram renderer, and web research; the bundled browser stays the automatic fallback (#2810)
* feat(aside): browser-driver contract, cookbook, research and fallback resolvers
{{ASIDE_SETUP}} (readiness probe + ten rules for driving the user's real browser), {{ASIDE_COOKBOOK}} (script shapes verified live against Aside CLI 1.26: one flow per aside repl script, CDP console hook before navigation, evidence lines, session-directory artifact handoff, GSTACK_STEP_OK sentinel), {{ASIDE_RESEARCH}} (research through aside exec, WebSearch when Aside is absent, knowledge otherwise) and {{BROWSE_FALLBACK}} (the fifteen-row Aside-step to $B-command table plus the rules that differ, so every browsing skill keeps working on gstack's own headless browser). test/aside-driver.test.ts pins the sentences and asserts every browsing skill carries the Aside block followed by the fallback; test/helpers/aside-available.ts is the shared live-Aside probe.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(render): Aside-first local-HTML renderer with the bundled browser as fallback
lib/aside-render.ts serves the HTML's directory on loopback (Aside refuses file:// URLs), opens it with waitUntil load, prints through CDP Page.printToPDF so tagged output, outlines, header/footer templates and page numbers survive, emulates device metrics for sized screenshots, and writes in-page evaluations to files; when Aside is absent it runs the same spec through the browse daemon (newtab, load, js, pdf, screenshot, closetab) and reports ENGINE=aside|browse. bin/gstack-render.ts is the CLI skill templates call. lib/claude-bin.ts and lib/error-handling.ts become the canonical copies (browse/src re-exports them).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(browse): /browse drives Aside first, with the $B reference behind the fallback
Contract, cookbook, mode choice (aside repl by default, aside exec for reading), report format, the fallback section, and the full command reference carved on demand.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(qa): /qa and /qa-only drive Aside, fall back to $B
QA_METHODOLOGY runs every phase as Aside scripts (orient, explore, document, re-test, mobile viewport via CDP emulation, links via HEAD fetch); the authenticate phase is 'you are already signed in'; a 13th rule requires consent before mutating actions on non-local targets; the fallback section translates each step onto $B. The qa E2E tests run on whichever engine is present.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(design): design-review, design-consultation, design-shotgun, plan-design-review, design-html drive Aside
Design-system extraction is one script printing FONTS/COLORS/HEADINGS/TOUCH_TARGETS/NAV; competitor research confirms the exact URLs before opening them in the real browser and runs on the bundled browser when Aside is absent; design-html's viewport screenshots, sketches and comparison boards render through gstack-render.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(deploy): benchmark, canary, land-and-deploy Step 7, devex-review drive Aside
One aside repl script per page prints NAV/PAINT/LCP/RESOURCES/SCRIPTS/CSS/SUMMARY (benchmark), CONSOLE_ERRORS/NAV/TEXT + screenshot (canary, re-run every 60s), and the post-deploy check reads responseStatus from the navigation entry; each carries the $B fallback.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(third-party-actions): Aside is the recommended driver; gstack's visible browser stays the fallback
The readiness probe is lifted from {{ASIDE_SETUP}} at gen time (byte-identity pinned) and rule 3 points at browse/SKILL.md for how to drive; the consent question offers Aside first and gstack's own visible browser (handoff/resume for sign-in) as the fallback, as v1.72 framed it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(scrape): /scrape reads pages through Aside; the browser-skills runtime rides the fallback
Look-then-extract scripts build the JSON inside the page and print it between JSON_START/JSON_END; aside exec for fuzzy intents; on the $B fallback the browser-skills match/prototype flow and /skillify apply as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(make-pdf): print through Aside first, the bundled browser otherwise
asideClient.ts replaces the direct $B client with one render() call per PDF (the exact option mapping the browse pdf command had: paper, margins, header/footer/page numbers, tagged, outline, printBackground, preferCSSPageSize, Paged.js wait); the diagram pre-pass, oversized-image downscale and DOCX rasters each run as one render script with per-fence try/catch; exit 4 now means no browser is available and names both remedies; $P setup reports which engine it found. The e2e gates run on whichever engine is present, so the Linux lane exercises the fallback.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(diagram): the triplet is one gstack-render call
SVG, PNG and excalidraw from one invocation over the content-addressed bundle staged under /tmp/gstack-render; every diagram type gets an excalidraw export; gstack-render picks the engine and prints ENGINE=; the diagram E2E gates on either engine.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(research): web research runs in Aside first, WebSearch second
The planning, review, design, security and investigate skills research through {{ASIDE_RESEARCH}}; WebSearch stays in allowed-tools as the fallback; testing.ts's bootstrap step follows; skeleton ceilings ratcheted for the research block.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(setup,gen-skill-docs): prune renders of skills that no longer exist
setup gains _prune_stale_generated for every host tree and the doc generator removes gstack-* output dirs it did not write, so a skill removed from the source tree can never linger in an install.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test: registries, budgets and suite reconciled for Aside-first with the $B fallback
Touchfiles + E2E tiers gain the Aside keys, coverage matrix and eval baselines updated, size budget re-baselined to parity-baseline-v1.80.0.0.json (the contract plus fallback ride in every browsing skill), parity ceilings ratcheted with measured values, LLM-judge prompts and the E2E fixtures speak Aside-first, browse-fallback.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docs: Aside first, gstack browser fallback
README, BROWSER.md, docs/, CONTRIBUTING, CLAUDE.md, ARCHITECTURE, AGENTS.md, TODOS and the root router describe the one product story: Aside is the browser gstack drives first; the bundled headless browser is the automatic fallback (Linux, Windows, app closed) where cookie import, GStack Browser, pair-agent and browser-skills still apply.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* chore: regenerate SKILL.md docs, llms.txt, agents digest, ship goldens, context-budget fixture
bun run gen:skill-docs over the templates; goldens re-rendered; context-budget ceilings recaptured.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* v1.80.0.0: Aside is the browser gstack drives first; the bundled browser is the fallback
MINOR: new capability across ten skills, the renderer and research; nothing removed. CHANGELOG release summary + itemized changes; VERSION 1.80.0.0; package.json 1.80.0.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docs(todos): file non-Claude host ownership-gate and version-heading pin follow-ups
Two follow-ups from the /plan-ceo-review + /plan-eng-review pass on merging
PR #2804 with main's v1.80.0.0 ownership gate: bring the Codex/Factory/
OpenCode/Cursor/Kiro copy loops and the stale-render prune under the
.gstack-owned marker rule, and a free test pinning that the CHANGELOG top
heading equals VERSION (the collision that git cannot see).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix: pre-landing review fixes for the Aside-first branch
Review army + adversarial passes (Claude and Codex) on the merged branch:
setup
- _prune_stale_generated scans the host dirs too (the generator already
removed the render before setup ran, so the host branch was dead), skips
symlinks in the render tree (rm -rf on a slash-terminated link empties its
target), removes a host symlink only when it resolves into gstack, cleans a
bannered real dir through _cleanup_weak_dir, recognizes frontmatter-renamed
skills, and logs through log. The always-run codex render passes every host
dir that may link to it.
- NEEDS_BUILD checks all three binaries (with $_EXE) and lib/ sources; the
browser hint and the bootstrap summary honor GSTACK_SKIP_ASIDE, treat a
requested skip as a request, and derive one skill list.
lib/aside-render.ts + bin/gstack-render.ts
- The loopback server carries a per-render secret path, checks containment on
the real path (symlink escapes are 403), and rejects malformed encoding.
- Inline eval results are one base64 line, so page text cannot forge
ASIDE_DIR= or the sentinel; the last ASIDE_DIR wins.
- runProc escalates SIGTERM to SIGKILL, bounds every wait, and clears every
timer (an uncleared one kept gstack-render alive after printing OK).
- renderTmpDir refuses a shared /tmp name owned by someone else; the work dir
and server are created inside try; goto's budget follows the render budget.
- probeAside classifies a present-but-failing CLI as ASIDE_NOT_RUNNING like
the skills' bash probe; render() retries on gstack's own browser when Aside
could not start or its private CDP bridge is gone (never on a page error
or a timeout of a running script); the CLI reports the engine that actually
rendered, exits 0 on --help, rejects non-numeric flags, documents
--wait-timeout, fences EVAL/PAGE_ERRORS as untrusted content, and names the
daemon's cookie-import JS lock remedy.
- The browse path passes --scale only when asked (a scale change rebuilds
the daemon context) and restores the viewport after a sized screenshot.
resolvers / templates
- The bash probe honors GSTACK_SKIP_ASIDE and has a perl deadline on stock
macOS; .local is no longer LOCAL (mDNS); same-origin filters compare parsed
origins; link status is HEAD-checked only on LOCAL targets; every
aside exec goes through the receipted _aside_exec prelude
({{ASIDE_EXEC_PRELUDE}}), including nine template blocks that called it
bare; the design sketch and diagram staging use private directories.
- The generator prunes only bannered renders and never a host whose
generation failed.
Docs, stale comments and dead code cleaned; goldens re-rendered; tests
updated and added for every behavior above.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test: coverage for the render CLI, setup rebuild check, make-pdf exit codes, and prose $B spans
New free tests from the ship coverage audit: test/gstack-render-cli.test.ts
(argv guards, --help, output contract with a fake daemon, failure and
serve-root paths, no-browser case, prompt exit), test/setup-needs-build.test.ts
(every binary and source set flips NEEDS_BUILD, Windows suffixes),
make-pdf/test/cli-exit-codes.test.ts and setup-smoke.test.ts (error to exit
code mapping, runSetup stages, renderPdf's engine), and prose-span cases for
extractBrowseCommands in test/skill-parser.test.ts.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docs: CHANGELOG and TODOS cover the review fixes (v1.81.0.0)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docs: sync project docs with the v1.81.0.0 review fixes
BROWSER.md, ARCHITECTURE.md, CONTRIBUTING.md, README.md, CLAUDE.md,
docs/TESTING_INTERNALS.md and docs/PROJECT_STRUCTURE.md now describe the
shipped renderer and setup: the loopback render server's per-render secret
path and real-path containment, ENGINE= naming the engine that actually
rendered (mid-run retry on gstack's own browser), EVAL/PAGE_ERRORS fenced as
untrusted content, --wait-timeout and the CLI's argv guards, the receipted
_aside_exec prelude ({{ASIDE_EXEC_PRELUDE}} in the placeholder table), the
LOCAL host rule without .local, LOCAL-only HEAD checks in the links script,
GSTACK_SKIP_ASIDE across probe/renderer/setup, the ownership-gated
retired-skill prune, the widened NEEDS_BUILD check, and the new free tests
(gstack-render-cli, setup-prune-stale-generated, setup-browser-hint,
setup-needs-build, make-pdf cli-exit-codes and setup-smoke).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docs: CHANGELOG states the precise mid-run retry rule
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(test): skill-e2e-bws slices the $B setup block from the Browser fallback section
browse/SKILL.md no longer has '## SETUP' / '## Core QA Patterns' (Aside is the
primary driver; the $B block moved under 'Browser fallback'), so the gate test
sliced an empty block and handed the agent nothing to run. Anchor on
'### Find the `$B` binary' up to the next heading. 7/7 pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(test): gate POSIX-only fixtures off Windows
windows-free-tests: the gstack-render CLI tests drive a shebang fake browse
that CreateProcess cannot exec, and two NEEDS_BUILD cases assert an execute
bit and a bare-name miss that MSYS bash does not have (test -x ignores mode
bits and resolves design -> design.exe). Those describes and cases now
self-skip on win32; argument guards, --help, the no-browser case, and every
other rebuild-check case still run there.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(render): runProc waits for the exit code until the kill deadline; newtab retries once on a cold daemon
A process whose pipes have reached EOF is exiting, but runProc gave the exit
code only five seconds to arrive and then returned null, which run() reports
as a failed command. Under CI's six-shard load one such render failed with the
artifact already written. The SIGTERM/SIGKILL timers already bound the wait,
so the exit race now runs to the kill deadline.
The first CLI call auto-starts the browse daemon; on a cold start it can
answer 'Unable to connect' once while the server is still coming up. That
single case is retried after 1.5s; every other newtab failure is not.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(aside-render): warm the daemon before live fallback cases; failures name the render error
- Live fallback cases run 'goto about:blank' up to twice before asserting and
skip (never fail) when the daemon cannot come up.
- expectOk() puts r.error and the browse transcript into the assertion so a
failed render is diagnosable from the CI log.
- The argv-contract cases dump the fake's log on a miss.
- File default timeout is 30s: the subject is the CLI contract, not latency.
- Two cases pin the cold-daemon newtab retry and that other errors are not
retried.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docs: CHANGELOG notes the cold-start tolerance of the bundled-browser renderer
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Sina <sdroid674+github@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
|
||
|
Garry TanandClaude Opus 4.7
|
33cb4715ef |
v1.39.2.0 feat: GSTACK_* env-shim for Conductor + gbrain/gstack setup docs (#1534)
* feat: GSTACK_* env-key shim for Conductor workspaces New lib/conductor-env-shim.ts promotes GSTACK_ANTHROPIC_API_KEY and GSTACK_OPENAI_API_KEY to canonical names when canonical is empty. Wired into the four TS entry points that hit paid APIs or gbrain embeddings: gstack-gbrain-sync.ts, gstack-model-benchmark, preflight-agent-sdk.ts, test/helpers/e2e-helpers.ts. Side-effect-only import, 15 lines total. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: gbrain+gstack setup, Conductor env mapping (v1.39.2.0) USING_GBRAIN_WITH_GSTACK.md: new "What you get after setup" section, Path 4 (remote MCP / split-engine), /sync-gbrain workflow stages + watermark mechanics, "Conductor + GSTACK_* env vars" section, env vars table extended, two troubleshooting entries (silent embedding failure and FILE_TOO_LARGE watermark block). CONTRIBUTING.md "Conductor workspaces": new paragraph on the GSTACK_* prefix pattern and the four entry points importing the shim. VERSION 1.39.1.0 → 1.39.2.0 and CHANGELOG entry covering the shim + docs (full release-summary format with before/after table). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: unit coverage for conductor-env-shim Refactor lib/conductor-env-shim.ts to export promoteConductorEnv() so unit tests can manipulate env and call it directly (a bare side- effect IIFE on import isn't reachable from bun:test once cached). The on-import IIFE still runs — existing four-entry-point imports keep working unchanged. test/conductor-env-shim.test.ts covers all three branches: GSTACK_FOO present + FOO empty → promotion; FOO already set → no-overwrite; nothing in env → no-op. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: Conductor strips canonical API keys (not just "doesn't inherit") The prior docs framed the GSTACK_* prefix as collision-avoidance: "Conductor exposes API keys under a GSTACK_ prefix so it never collides with whatever the host system has set." That understates the mechanism — Conductor actively strips ANTHROPIC_API_KEY and OPENAI_API_KEY from every workspace's process env, so setting them in ~/.zshrc or .env doesn't help. The fix path is to set the GSTACK_-prefixed forms in Conductor's workspace env config; Conductor passes those through untouched. Three docs updated to reflect the strip, not the polite framing: USING_GBRAIN_WITH_GSTACK.md (Conductor section), CONTRIBUTING.md (Conductor workspaces paragraph), CHANGELOG.md (release summary). README.md gains a "Running gstack in Conductor?" callout in the GBrain section pointing at the canonical doc's anchor, plus a fourth path entry (remote gbrain MCP / split-engine) that was already documented in USING_GBRAIN but missing from the README summary. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Garry TanandClaude Opus 4.7
|
0570ef93a5 |
v1.24.0.0 feat: cross-platform hardening — curated Windows lane + Bun.which resolver + path-portability helper (#1252)
* feat(paths): bin/gstack-paths helper + migrate 8 skills off inline state-root chains
New bin/gstack-paths emits GSTACK_STATE_ROOT, PLAN_ROOT, TMP_ROOT exports for
skill bash blocks to source via eval. Honors GSTACK_HOME → CLAUDE_PLUGIN_DATA →
$HOME/.gstack → .gstack (and parallel chains for plan/tmp roots) so skills work
the same in plugin installs, global installs, and CI containers without HOME.
Eight skills migrate off inline ${CLAUDE_PLUGIN_DATA:-...} or ${GSTACK_HOME:-...}
chains: careful, freeze, guard, unfreeze, investigate, context-save,
context-restore, learn, office-hours, plan-tune, codex. Resolved values are
identical, so existing tests cover correctness; the win is consolidating 11
copy-pasted fallback chains behind one helper.
codex/SKILL.md.tmpl gets a new Step 0.6 Resolve portable roots that sources
gstack-paths once, then replaces hardcoded ~/.claude/plans/*.md and
/tmp/codex-*-XXXXXX.txt with "$PLAN_ROOT"/*.md and "$TMP_ROOT/codex-*-XXXXXX.txt".
Hardening direction credited to the McGluut/gstack fork; this is upstream's
factoring of the per-skill chain the fork inlined.
Tests: test/gstack-paths.test.ts covers all three fallback chains with 8 unit
tests (HOME unset, CLAUDE_PLUGIN_DATA set, GSTACK_HOME wins, etc).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(claude-bin): Bun.which wrapper for cross-platform claude resolution
Replaces 75 LOC of fork-side reimplementation (PATH parsing, Windows PATHEXT,
case-insensitive Path/PATH, X_OK) with a thin wrapper around Bun.which() — the
runtime built-in that already does all of it. New file is ~70 LOC including
the override + arg-prefix logic the runtime doesn't cover.
Override branch fixed: GSTACK_CLAUDE_BIN=wsl now resolves through Bun.which()
just like a bare claude lookup would. The McGluut fork's claude-bin.ts only
handled absolute-path overrides; bare commands silently returned null. Passing
the override value through Bun.which fixes the documented use case for free.
Five hardcoded claude spawn sites rewired through resolveClaudeCommand:
- browse/src/security-classifier.ts:396 — version probe
- browse/src/security-classifier.ts:496 — Haiku transcript classifier
- scripts/preflight-agent-sdk.ts — preflight binary pinning
- test/helpers/providers/claude.ts — LLM judge availability + run
- test/helpers/agent-sdk-runner.ts — SDK harness binary resolver
All retain their existing degrade-on-missing semantics.
Tests: browse/test/claude-bin.test.ts has 9 unit tests including the
override-PATH-resolution case the fork's version got wrong.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs+test: AGENTS.md/docs/skills.md inventory sync + private-path leak detector
Inventory sync (codex-flagged drift):
- /debug → /investigate (skill renamed in v1.0.1.0)
- AGENTS.md grows from 21 to 40+ skills, organized by category (plan reviews,
implementation, release, operational, browser, safety)
- docs/skills.md gains 11 missing entries: /plan-devex-review, /devex-review,
/plan-tune, /context-save, /context-restore, /health, /landing-report,
/benchmark-models, /pair-agent, /setup-gbrain, /make-pdf
- Stale "<5s bun test" claim dropped — slim-preamble harness + new tests means
no realistic universal claim to make
- Adds explicit "Mac + Linux full, curated Windows lane" platform statement +
"Git Bash / MSYS today, native PowerShell future" install note
New invariants in test/skill-validation.test.ts (~80 LOC):
- Private-path leak detector scans every SKILL.md / SKILL.md.tmpl for known
maintainer-only filenames (coordination-board.md, SEEKING_LOG.md,
RATIONAL_SUBJECT.md, VALUE_SIGNAL_LOOP.md, C:\LLM Playground\go).
Adapted from the McGluut fork's skill-contract-audit.ts; we don't take
the script wholesale because most of its checks are already covered by
test/gen-skill-docs.test.ts:1668-2074 and test/skill-validation.test.ts:1419
— only the private-path scan and doc-inventory cross-check are new.
- Doc-inventory cross-check: every skill directory with a SKILL.md.tmpl must
appear in both AGENTS.md and docs/skills.md. Catches the inventory drift
this commit is fixing — without this test it would just drift again.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(windows): curated windows-free-tests CI job + test-free-shards curation
Codex's v1.18.0.0 review flagged that a windows-latest matrix entry on the
existing Linux-container evals.yml workflow can't work as a drop-in, and that
the free test suite has POSIX-bound dependencies a sharded runner doesn't fix
on its own. This commit takes McGluut's test-free-shards.ts (190 LOC), adds a
Windows-fragility scan, and runs the curated subset on a separate non-container
windows-latest job.
scripts/test-free-shards.ts:
- Enumeration + paid-eval filtering + stable-hash sharding (FNV-1a). Adapted
from McGluut/gstack fork.
- Upstream-original: --windows-only filter scans each test's content for
POSIX-bound patterns: hardcoded /bin/sh, spawn('sh', ...), bash -c, raw
/tmp/, chmod, xargs, which claude. Files matching are excluded with the
reason logged. Currently filters 25 of 128 free tests; remaining 103 run
on windows-latest.
.github/workflows/windows-free-tests.yml:
- Separate non-container job (NOT a matrix entry on evals.yml). Runs:
bun run test:windows # curated subset
bun test browse/test/claude-bin.test.ts # PATHEXT+overrides on Windows
bun test test/gstack-paths.test.ts # state-root resolution
package.json: new test:free + test:windows scripts.
Honest about scope (codex-flagged): this does NOT make the full free suite
Windows-safe. The 25 excluded tests need POSIX-only surfaces ported off shell
primitives (test/ship-version-sync.test.ts:72 hardcodes /bin/bash, etc).
Tracked as a P4 follow-up TODO. Full Windows parity is the next wave; this
release ships the curated lane.
Tests: test/test-free-shards.test.ts has 14 unit tests covering enumeration,
paid-eval filtering, Windows-fragility detection (POSIX patterns + safe code),
and stable sharding determinism.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(release): v1.20.0.0 — cross-platform hardening, curated Windows lane
Cross-platform hardening. Mac + Linux full, curated Windows lane added.
Workspace-aware queue at ship time:
- v1.17.0.0 claimed by garrytan/setup-gbrain-run (PR #1234)
- v1.19.0.0 claimed by garrytan/browserharness (PR #1233)
- This branch claims v1.20.0.0 (next available slot)
(Initially bumped to v1.18.0.0 during plan-mode implementation; rebumped to
v1.20.0.0 at /ship time when gstack-next-version detected the queue had moved.)
Headline numbers (full release-note in CHANGELOG.md):
- 2 new shared resolvers: bin/gstack-paths (61 LOC), browse/src/claude-bin.ts (73 LOC)
- 8 skills migrated off inline state-root chains
- 5 hardcoded claude spawn sites rewired through the shared resolver
- 75 LOC of fork-side reimplementation replaced by Bun.which()
- 103 of 128 free tests run on windows-latest (curated, ~80%)
- +31 new unit tests + 3 new invariants
- AGENTS.md inventory grows from 21 to 40+ skills
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(windows-ci): configure git identity + extend Windows-fragility curation
First windows-free-tests CI run surfaced 34 failures across two patterns:
1. Tests that init a temp git repo via execSync('git commit ...') — Windows
runner has no default git user.email/user.name, so the commit fails.
Fix: add a "Configure git identity" step to .github/workflows/windows-free-tests.yml
that sets a CI-only identity globally.
2. Tests that use POSIX-only APIs unconditionally:
- file-mode bitmask checks (`stat.mode & 0o600`, `mode & 0o111`) — Windows
fakes mode bits and these assertions don't compose
- hardcoded forward-slash path assertions (`file.endsWith('/tab-42.json')`)
— Windows path separators are '\\'
Fix: extend WINDOWS_FRAGILE_PATTERNS in scripts/test-free-shards.ts to
detect both. 8 additional tests now excluded from the curated Windows
subset with logged reasons:
- browse/test/security-review-flow.test.ts (file mode)
- browse/test/security-sidepanel-dom.test.ts (forward-slash path)
- browse/test/url-validation.test.ts (forward-slash path)
- test/gbrain-repo-policy.test.ts (file mode)
- test/relink.test.ts (file mode)
- test/skill-validation.test.ts (file mode — single assertion at :934)
- test/team-mode.test.ts (file mode — also kills its 30 git-init beforeEach failures)
- test/upgrade-migration-v1.test.ts (file mode)
Curated Windows subset: 103 → 95 tests (still ~74% of free suite). All
14 test-free-shards unit tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(windows-ci): enforce LF + build server-node.mjs in CI
Second round of windows-free-tests fixes after the first push. Curated subset
went from 386/34 to 58/4 fails. Remaining 4 fails + 1 error trace to two root
causes:
1. Line-ending sensitivity. Windows checkout with core.autocrlf=true converts
.md/.tmpl files to CRLF. Tests that parse YAML frontmatter with
`/^---\n([\\s\\S]+?)\n---/` then return zero matches — skill-collision-
sentinel.test.ts:120 enumerated 0 skills on Windows, cascading into 3
downstream test failures (sanity, KNOWN_COLLISIONS, /checkpoint resolved).
Fix: add .gitattributes that pins LF for .md/.tmpl/.yml/.json/.toml/.sh/
.ts/.tsx/.js/.mjs/.cjs/.bash. Root-cause fix; prevents future similar
tests from hitting the same trap. Also keeps bash scripts LF on Linux
runners (CRLF in shebangs produces "bad interpreter" errors).
2. Module-level Windows assertion in browse/src/cli.ts:82 throws if
browse/dist/server-node.mjs is missing. Any test that transitively loads
cli.ts (e.g., browse/test/tab-isolation.test.ts via shard mate imports)
then fails to even start. server-node.mjs is generated by bash
browse/scripts/build-node-server.sh, which `bun run build` calls but
`bun install` does not.
Fix: add a "Build server-node.mjs" step to .github/workflows/
windows-free-tests.yml. Calls only the node-server build script, not
full `bun run build` — we don't need the compiled binaries for tests
and the full build is slow.
Expected: skill-collision-sentinel goes 0→3 pass (sanity, KNOWN_COLLISIONS,
/checkpoint resolved). tab-isolation's "unhandled error between tests"
disappears. Remaining tests should be green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(windows-ci): platform-aware claude-bin test + curate bin/ shebang spawns
Round 3 of windows-free-tests fixes. Round 2 (LF gitattributes + server-node.mjs
build) cleared shard 1 entirely (skill-collision-sentinel and tab-isolation
green). Shard 2 surfaced two more issues:
1. browse/test/claude-bin.test.ts:50 — the "PATH-resolvable override" test
creates a fake binary 'fake-claude-cli' (no extension) and expects
Bun.which to find it. On Windows, Bun.which probes PATHEXT extensions
(.cmd, .exe, .bat) — a bare-name file is not discoverable. Production
behavior is correct; the test was Mac/Linux-shaped.
Fix: branch on process.platform. On Windows, write 'fake-claude-cli.cmd'
with a Windows batch payload instead of a POSIX shebang script.
2. test/gstack-question-log.test.ts (and 18 sibling tests) — spawn a bash
shebang script via spawnSync(BIN, args). Git Bash on Windows can run
`bash /path/to/script` but spawnSync invokes CreateProcess directly,
which doesn't parse #!/usr/bin/env bash. All these tests are
Windows-fragile and can't run as-is.
Fix: extend WINDOWS_FRAGILE_PATTERNS with `path.join(.., 'bin', ..)`
detector. Curates 19 additional tests (benchmark-cli, brain-sync,
builder-profile, explain-level-config, gbrain-*, gstack-question-*,
hook-scripts, learnings, plan-tune, review-log, secret-sink-harness,
taste-engine, telemetry, timeline, uninstall).
Curated Windows subset: 95 → 76 tests (~59% of free suite). Still
meaningful Windows coverage. The 52 excluded tests are tracked as a
follow-up TODO for full Windows parity (shebang-bin spawns + POSIX file
modes + raw /tmp/ etc).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(windows-ci): curate Playwright-launching tests
Round 4 of windows-free-tests fixes. Round 3 cleared shard 2 except for
browse/test/batch.test.ts:35 which calls `await bm.launch()` and triggers
Playwright Chromium launch. The windows-latest runner doesn't have
Chromium installed (browser bring-up is a separate concern, tracked by
PR #1238 windows-pty-bun-pty-fix).
Fix: extend WINDOWS_FRAGILE_PATTERNS with `await \\w+\\.launch\\(` matcher.
Catches batch.test.ts plus 7 sibling tests (commands, compare-board,
content-security, handoff, security-live-playwright, security-sidepanel-dom,
snapshot — most already excluded by other patterns).
Curated Windows subset: 76 → 72 tests (~56% of free suite). Net curation
across all 4 rounds: 56 of 128 free tests excluded, each with a logged
reason. The 56 excluded fall into 6 buckets — POSIX shells, raw /tmp/,
chmod/xargs, file mode bitmasks, forward-slash path assertions, bin/
shebang spawns, and Playwright launches — all tracked as a P4 follow-up
TODO for full Windows parity.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(windows-ci): catch destructured join() bin-spawns + browse server tests
Round 5 of windows-free-tests fixes. Round 4 caught Playwright launchers
but two more failure shapes appeared in shard 5:
1. test/diff-scope.test.ts uses `import { join }` (destructured) and
`join(import.meta.dir, '..', 'bin', 'gstack-diff-scope')`. My round-3
pattern only matched `path.join(...)` — the destructured form slipped
through. Tightened the pattern to match the literal `, 'bin', '<name>'`
path-segment shape regardless of whether it's `path.join` or `join`
directly.
2. browse/test/sidebar-integration.test.ts spawns the browse server via
`spawn(['bun', 'run', server.ts])` with BROWSE_HEADLESS_SKIP=1. The
Bun-run-server.ts path is the same Playwright-on-Windows broken path
that the windows-free-tests job intentionally avoids — the server-node.mjs
route only kicks in for the compiled binary, not direct Bun runs of the
TypeScript source. Added a BROWSE_HEADLESS_SKIP / spawn-bun-run pattern.
Curated Windows subset: 72 → 73 tests (~57% of free suite). Net up by 1
because the tightened bin pattern released one test that was a false
positive in the loose `path\\.join` form.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(windows-ci): broaden bin/ pattern to match path.join(ROOT, 'bin')
Round 6. Round 5 tightened the bin/ pattern to require a script-name segment
after 'bin', which inadvertently released test/brain-sync.test.ts that uses:
const BIN = path.join(ROOT, 'bin');
const full = bin.startsWith('/') ? bin : path.join(BIN, bin);
The 'bin' segment is the LAST argument to path.join — there's no literal
script name to match. The earlier looser pattern caught this; round 5
broke that.
Fix: revert to `,\\s*['"]bin['"]\\s*[,)]` which matches both forms:
- `, 'bin', 'script-name')` (path.join with name) — typical
- `, 'bin')` (path.join ending at bin) — brain-sync style
Curated subset: 73 → 66 tests (~52% of free suite). The 7 additional
exclusions are all bin-script tests that were misclassified by the round-5
tightening.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(find-browse): guard main() with import.meta.main
Round 7 of windows-free-tests fixes (and a genuine bug fix beyond Windows).
browse/src/find-browse.ts called main() unconditionally at module load.
main() calls process.exit(1) when no compiled `browse` binary exists at the
known install paths. Any test that imports `locateBinary` from this module
then exits the entire test process before any tests run.
This affected the windows-free-tests CI lane because the runner intentionally
doesn't compile the browse binary (only server-node.mjs is built — full
binary compilation is slow and not needed for the curated subset). It would
also affect any Mac/Linux contributor who runs tests in a fresh checkout
before running ./setup, though the symptom is rarer there.
Fix: wrap `main()` in `if (import.meta.main) { main() }`. The CLI invocation
(via the find-browse binary or `bun run browse/src/find-browse.ts`) still
runs main() and emits the path. Imports get only the named exports.
Verified locally:
- `bun run browse/src/find-browse.ts` still prints the binary path.
- `import { locateBinary } from '...'` no longer exits the process.
- `bun test browse/test/find-browse.test.ts` passes 4/4 (was crashing
at module load).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(windows-ci): pin LF on extensionless executables (setup, bin/*, scripts/*)
Round 8 of windows-free-tests fixes. Round 7 cleared find-browse + most
shards; one fail left in shard 7:
test/setup-codesign.test.ts > codesign shell snippet is syntactically valid
expect(received).toBeTruthy() — match was null
The test extracts a bash codesign block from the `setup` file via a
\\n-anchored regex, then syntax-checks it with `bash -n`. On Windows the
regex returned null because the `setup` file was checked out with CRLF
endings — my round-2 .gitattributes only covered files matched by extension
patterns (*.md, *.sh, *.ts) and `setup` is extensionless.
Fix: extend .gitattributes with explicit rules for extensionless executables:
setup text eol=lf
bin/* text eol=lf
**/scripts/* text eol=lf
This also LF-pins all the bash bin/ scripts (gstack-paths, gstack-slug,
gstack-codex-probe, ...) which would otherwise break with "bad interpreter"
errors on Linux if a Windows contributor accidentally committed CRLF
versions. Defense in depth.
Verified locally: `git check-attr eol setup bin/gstack-paths` reports
`eol: lf` for both. Renormalized via `git add --renormalize` so any
already-LF files in the repo stay LF after the .gitattributes change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(windows-ci): gen:skill-docs in workflow + known-bad list for env-specific tests
Round 9 of windows-free-tests fixes. Round 8 cleared shard 7; shard 8
surfaced 4 fails:
1+2. test/gen-skill-docs.test.ts golden-file regression for Codex + Factory
ship skills failed with ENOENT on `.agents/skills/gstack-ship/SKILL.md`
and `.factory/skills/gstack-ship/SKILL.md`. These are gitignored
gen-skill-docs outputs that the Mac/Linux CI workflows already
regenerate elsewhere — the windows-free-tests lane never did.
Fix: add `bun run gen:skill-docs --host all` step to
windows-free-tests.yml after `bun install`.
3. test/host-config.test.ts:377 "detect finds claude" asserts the `claude`
binary is on PATH. True when running inside Claude Code; false on a
bare CI runner.
4. browse/test/findport.test.ts:117 asserts Bun.serve.stop() is
fire-and-forget (returns undefined). Bun's Windows behavior for this
polyfill differs; the assertion is Bun-on-non-Windows-specific.
Both 3 and 4 are environment/runtime-specific failures that don't fit a
regex pattern. Added a KNOWN_WINDOWS_INCOMPATIBLE explicit list to
scripts/test-free-shards.ts so they're curated by exact path, with a
reason string. The list is for cases where pattern matching can't infer
the failure shape from the source file alone.
Curated subset: 66 → 64 tests (~50% of free suite). 14 unit tests in
test/test-free-shards.test.ts still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(windows-ci): curate pre-existing breakage from v1.14.0.0 sidebar refactor
Round 10 of windows-free-tests fixes. Round 9 cleared shards 7+8; shard 9
surfaced ENOENT for browse/src/sidebar-agent.ts. That file was DELETED in
v1.14.0.0 (sidebar REPL refactor — sidebar-agent.ts and the chat queue
path were ripped in favor of the interactive xterm.js PTY). 10 security
tests still reference it via top-level fs.readFileSync and fail on import.
Verified locally: `bun test browse/test/security-source-contracts.test.ts`
on this branch reports 0 pass, 1 fail, 1 error. Mac/Linux CI exits 0
because Bun reports module-load failures as "error" not "fail" and the
exit code is 0; Windows CI exits 1 (stricter). Same pre-existing
breakage on every platform — just only visible in shard 9 of the
Windows lane.
Fix: add WINDOWS_FRAGILE_PATTERNS entry matching `sidebar-agent.ts` /
`src/sidebar-agent` references. Curates browse/test/sidebar-ux.test.ts
(other 9 likely caught by paid-eval filter or earlier patterns).
Tracked as a follow-up TODO: update or delete the 10 security tests that
reference deleted source. Out of scope for v1.20.0.0 portability wave.
Curated subset: 64 → 63 tests (~49% of free suite).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(windows-ci): broaden sidebar-agent.ts pattern to catch all references
* fix(windows-ci): catch ./bin/<name> direct path spawns
* fix(windows-ci): scope Windows job to v1.20.0.0 new portability work
12 rounds of curation revealed that gstack has a long tail of tests with
environment-specific assumptions (POSIX paths, /tmp, mode bits, bash
spawns, deleted v1.14 sidebar refs, HOME=unset guards, Bun polyfill
specifics). Each round of pattern-matching curation caught 1-2 new
buckets but kept surfacing more.
Honest scope for v1.20.0.0: this PR delivers two new portability
primitives (bin/gstack-paths + browse/src/claude-bin.ts). The Windows
CI job should verify those primitives work on Windows. Full-suite
Windows parity is a P4 follow-up that requires touching many tests
that aren't part of this PR's scope.
Change: windows-free-tests.yml now runs:
bun test test/gstack-paths.test.ts \\
browse/test/claude-bin.test.ts \\
test/test-free-shards.test.ts
That's 31 tests targeting exactly the new code paths shipped here.
The release-note headline ("curated Windows lane added") becomes
truthful when this passes — we have a real Windows CI gate on the
new portability work, not a rebadged failure-tolerant attempt at the
full suite.
Retained: scripts/test-free-shards.ts curation logic (informational
output via `--list`, useful for future expansion of the Windows lane
when contributors port specific tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(test): invoke bin/gstack-paths via bash (Windows shebang fix)
Round 13 of windows-free-tests fixes. Round 12 (scope pivot) revealed all
8 gstack-paths tests fail on Windows because the test invokes the bash
shebang script directly:
spawnSync(BIN, []) # BIN = path.join(ROOT, 'bin', 'gstack-paths')
Windows CreateProcess can't parse `#!/usr/bin/env bash` from the file.
The script never runs on Windows via this invocation path.
Fix: change to `spawnSync('bash', [BIN], ...)`. This matches production
usage — the script is sourced from inside skill bash blocks via
`eval "$(~/.claude/skills/gstack/bin/gstack-paths)"`, where bash is
always the executor. Mac/Linux behavior is identical (bash invocation
of a bash script).
Verified locally: 8/8 tests still pass on macOS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(release): rebump v1.20.0.0 → v1.22.0.0 (queue drift)
Version-gate workflow rejected v1.20.0.0 because the queue moved during
the windows-free-tests fix loop:
v1.16.0.0 → garrytan/gbrowser-unleashed (PR #1253) [new since last bump]
v1.17.0.0 → garrytan/setup-gbrain-run (PR #1234)
v1.19.0.0 → garrytan/browserharness (PR #1233)
v1.21.1.0 → garrytan/pty-plan-mode-e2e (PR #1255) [new since last bump]
Two new sibling PRs landed slot claims while we iterated on Windows.
Next free MINOR slot is v1.22.0.0.
Updated VERSION, package.json, CHANGELOG header + body. Also pushing the
round-13 windows-fix in parallel (test invokes bin/gstack-paths via bash
to handle Windows shebang).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(test): clear USERPROFILE alongside HOME (Git Bash auto-populates HOME)
Final Windows fix. 29/31 pass; 2 fail in gstack-paths HOME-unset tests:
(fail) CWD fallback when HOME also unset (container env)
(fail) PLAN_ROOT chain: GSTACK_PLAN_DIR > CLAUDE_PLANS_DIR > HOME > CWD
Root cause: Git Bash on Windows auto-populates `HOME` from `USERPROFILE`
at shell startup if HOME is empty/unset. Passing `HOME: ''` to spawnSync
does set HOME='' for the child, but Git Bash overwrites it from
USERPROFILE during init, so the script sees `${HOME:-}` as non-empty
(C:\\Users\\runneradmin) and never reaches the CWD-fallback branch.
Fix: clear USERPROFILE='' too. On Linux/Mac it's a no-op (env var doesn't
exist in normal env); on Windows Git Bash it stops the HOME auto-populate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(test): skip HOME-unset assertions on Windows (Git Bash auto-populates)
29/31 → 31/31 expected on Windows. Final fix:
The 2 still-failing gstack-paths tests assert CWD-fallback behavior when
HOME is genuinely unset (Linux container scenario). On Windows Git Bash,
HOME gets auto-derived from USERPROFILE → HOMEDRIVE+HOMEPATH → /c/Users/<user>
during shell startup. Clearing all three of those env vars in the spawn
still results in HOME being non-empty by the time the script runs.
The bash script's CWD-fallback logic IS correct — it just isn't exercisable
through the Git Bash test surface. Skip those specific assertions on
Windows; they continue to verify on Linux/Mac.
This is the only platform-specific test guard introduced; it's narrowly
scoped to the unreachable code path, not a bypass of the real check.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Garry Tan
|
23c4d7b228 |
v1.13.0.0 feat: add Claude outside-voice skill (#1212)
* Add Claude outside-voice skill * Fix gbrain config isolation test * Restore Opus fanout overlay nudge * Warn on oversized tracked files * Release v1.13.0.0 * Fix Claude diff temp file handling * Remove Opus fanout overlay nudge |
||
|
Garry TanandClaude Opus 4.7
|
e3d7f49c74 |
feat(v1.10.1.0): overlay efficacy harness + Opus 4.7 fanout nudge removal (#1166)
* refactor: export readOverlay from model-overlay resolver Needed by the overlay-efficacy eval harness to resolve INHERIT directives without going through generateModelOverlay's full TemplateContext. * chore: add @anthropic-ai/claude-agent-sdk@0.2.117 dep Pinned exact for SDK event-shape stability. Used by the overlay-efficacy harness to drive the model through a closer-to-real Claude Code harness than `claude -p`. * feat(preflight): sanity check for agent-sdk + overlay resolver Verifies: SDK loads, claude-opus-4-7 is a live API model, SDKMessage event shape matches assumptions, readOverlay resolves INHERIT directives and includes expected content. Run with `bun run scripts/preflight-agent-sdk.ts`. PREFLIGHT OK on first run, $0.013 API spend. * feat(eval): parametric overlay-efficacy harness (runner + fixtures) `test/helpers/agent-sdk-runner.ts` wraps @anthropic-ai/claude-agent-sdk with explicit `AgentSdkResult` types, process-level API concurrency semaphore, and 3-shape 429 retry (thrown error, result-message error, mid-stream SDKRateLimitEvent). Pins the local claude binary via `pathToClaudeCodeExecutable`. `test/fixtures/overlay-nudges.ts` holds the typed registry. Two fixtures for the first measurement: `opus-4-7-fanout-toy` (3-file read) and `opus-4-7-fanout-realistic` (mixed-tool audit). Strict validator rejects duplicate ids, non-integer trials, unsafe overlay paths, non-safe id chars, and missing overlay files at module load. Adding a future overlay nudge eval = one fixture entry. * test(eval): unit tests for agent-sdk-runner (36 tests, free tier) Stub `queryProvider` feeds hand-crafted SDKMessage streams. Covers: happy-path shape, all 3 rate-limit shapes + retry, workspace reset on retry, persistent 429 -> `RateLimitExhaustedError`, non-429 propagation, process-level concurrency cap, options propagation, artifact path uniqueness, cost/turn mapping, and every validator rejection case. * test(eval): paid periodic overlay-efficacy harness `test/skill-e2e-overlay-harness.test.ts` iterates OVERLAY_FIXTURES, runs two arms per fixture (overlay-ON, overlay-OFF) at N=10 trials with bounded concurrency. Arms use SDK preset `claude_code` so both include the real Claude Code system prompt; overlay-ON appends the resolved overlay text. Saves per-trial raw event streams to `~/.gstack/projects/<slug>/transcripts/` for forensic recovery. Gated on `EVALS=1 && EVALS_TIER=periodic`. ~$3/run (40 trials). * test: register overlay harness in touchfiles (both maps) Entries for `overlay-harness-opus-4-7-fanout-toy` and `opus-4-7-fanout-realistic` in E2E_TOUCHFILES (deps: model-overlays/, fixtures file, runner, resolver) and E2E_TIERS (`periodic`). Passes `test/touchfiles.test.ts` completeness check. * fix(opus-4.7): remove "Fan out explicitly" overlay nudge Measured counterproductive under the new SDK harness. Baseline Opus 4.7 emits first-turn parallel tool_use blocks 70% of the time on a 3-file read prompt. With the custom nudge: 10%. With Anthropic's own canonical `<use_parallel_tool_calls>` block from their parallel-tool-use docs: 0%. Both overlays suppress fanout; neither improves it. On realistic multi-tool prompts (audit a project: read files + glob + summarize), Opus 4.7 never fans out in first turn regardless of overlay. Zero of 20 trials. Not a prompt problem. Keeping the other three nudges (effort-match, batch questions, literal interpretation) pending their own measurement. Harness is ready for follow-up fixtures — add one entry to `test/fixtures/overlay-nudges.ts` to measure any overlay bullet. Cost of investigation: ~$7 total across 3 eval runs. * chore: bump version and changelog (v1.6.5.0) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): extend OverlayFixture with allowedTools, maxTurns, direction Per-fixture tool allowlist unblocks measuring nudges that need Edit/Write (e.g. literal-interpretation 'fix the failing tests' needs write access). Per-fixture maxTurns lets harder prompts run longer without changing the default. `direction` is cosmetic metadata for test output labeling. Also adds reusable predicates and metrics: - lowerIsBetter20Pct / higherIsBetter20Pct — 20% lift threshold vs baseline - bashToolCallCount — count of Bash tool_use across the session - turnsToCompletion — SDK-reported num_turns at result - uniqueFilesEdited — Edit/Write/MultiEdit file_path set size test/skill-e2e-overlay-harness.test.ts now threads fixture.allowedTools and fixture.maxTurns through runArm. * test(eval): 3 more overlay fixtures to measure remaining Claude nudges Measures three overlay bullets that haven't been tested yet: - claude-dedicated-tools-vs-bash — claude.md says 'prefer Read/Edit/Write/ Glob/Grep over cat/sed/find/grep'. Fixture prompts 'list every TypeScript file under src/ and tell me what each exports' and counts Bash tool_use across the session. Overlay-ON should drop it by >=20%. - opus-4-7-effort-match-trivial — opus-4-7.md says 'simple file reads don't need deep reasoning.' Fixture uses a trivial one-file prompt (config.json lookup) and measures turns_used. Overlay-ON should be <=80% of baseline turns. - opus-4-7-literal-interpretation — opus-4-7.md says 'fix ALL failing tests, not just the obvious one.' Fixture seeds three failing test files with deliberately distinct failure modes and counts unique files edited. Overlay-ON should touch >=20% more files. Adding a fourth fixture for any remaining overlay nudge is a single entry. The harness is now proven on: fanout (deleted after measurement), dedicated tools, effort-match, and literal-interpretation. * fix(eval): handle SDK max-turns throw gracefully Some @anthropic-ai/claude-agent-sdk versions throw from the query generator when maxTurns is reached, instead of emitting a result message with subtype='error_max_turns'. The runner treated that as a non-retryable error and killed the whole periodic run on the first fixture that exceeded its turn cap. Added isMaxTurnsError() detector and a catch branch that synthesizes an AgentSdkResult from events captured before the throw, with exitReason='error_max_turns' and costUsd=0 (unknown from the thrown path). The metric function still runs against whatever assistant turns were collected, so the trial produces a usable number. Hoisted events/assistantTurns/toolCalls/assistantTextParts and the timing counters out of the inner try so the catch branch can read them. No behavior change on the success path or on rate-limit retry paths. * test(eval): bump maxTurns to 15 for claude-dedicated-tools-vs-bash The prompt 'list every TypeScript file under src/ and tell me what each exports' needs 1 turn for Glob + ~5 for Reads + 1 for summary. Default maxTurns=5 was not enough; prior run threw from the SDK on this fixture and tanked the whole periodic eval. Bumping to 15 gives headroom. The runner now also handles max-turns gracefully even if a future fixture underestimates, so this is belt and suspenders. * test(eval): Sonnet 4.6 variants of the 5 Opus-4.7 fixtures Same overlays, same prompts, same metrics, `model: 'claude-sonnet-4-6'`. Tests whether the overlays behave differently on a weaker Claude model where baseline behavior is shakier. Sonnet trials cost ~3-4x less than Opus so these 5 add ~$4.50 to a full run. Measurement result from the first paired run (100 trials total, ~$14.55): - **Sonnet + effort-match shows real overlay benefit.** With the overlay on, Sonnet takes 2.5 turns on a trivial `What's the version in config.json?` prompt. Without, it takes exactly 3.0 turns in all 10 trials. ~17% reduction, below the 20% pass threshold but the signal is clean: overlay-ON distribution [2,2,2,2,2,3,3,3,3,3] vs overlay-OFF [3,3,3,3,3,3,3,3,3,3]. - All other Sonnet dimensions flat (fanout, dedicated-tools, literal interpretation). Same as Opus on those axes. - Opus effort-match remains flat (2.60 vs 2.50, +4% slower with overlay). Implication: model-stratified. The overlay stack helps Sonnet on some axes where it does nothing on Opus. Wholesale removal would hurt Sonnet. Per-nudge per-model measurement is the right move going forward. * chore: bump version to 1.10.1.0 Updates VERSION, package.json, CHANGELOG header, and TODOS completion marker from 1.6.5.0 to 1.10.1.0. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |