mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-21 23:51:35 +01:00
v1.63.0.0 feat: GStack 2 fork port wave — egress receipts, context-bill, sharded gate, /health fix (#2541)
* test(helpers): shared skill-census helper with three explicit counts
physicalSkillFiles (symlinked dirs included, root router included),
authoredSkills (realpath-deduped, router excluded), registryEntries
(what ./setup registers: unique frontmatter names + _gstack-command).
One counting authority for the hermetic seeder, context-bill ground
truth, and the catalog-budget test — connect-chrome's dir symlink and
the root router otherwise produce three subtly different hand-rolled
censuses. Ported-wave foundation (C11).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(evals): stop the harness grading itself
findPreviousRun excluded only the file being written, by name, so every
suite compared against _partial-e2e.json — the current run's own
accumulator, relabelled with the current tier just before each flush.
That is why every block read '+$0.00, +0s, Stable run, no regressions.'
This harness has never been able to detect a regression, and reassuring
output that cannot fail is worse than none. In-progress runs are now
excluded by role, and a run with nothing to compare against says NO
BASELINE instead of claiming stability.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit f3140b5245221fff7fb9411c7ec07c2ca11587b5)
* refactor(evals): shared partial-run predicate + finalized-run lookup
isPartialEval(data, filename) is the one place that decides what counts
as an in-progress accumulator (the _partial flag OR a _partial-prefixed
filename), and findLatestFinalizedRun(evalDir, tier) is the one place
that finds the newest real run — scanning the eval dir plus one level of
shards/<slug>/ subdirs, where the sharded paid runner points each
shard's collector. skill-budget-regression.test.ts's hand-rolled
findLatestRun (flag-blind: a flagged-but-renamed accumulator passed its
name check) is replaced by the shared helper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit b55fcf6966366fd21a8cdc46de61aab6e1b1d100)
* feat(evals): register shipped skills for hermetic PTY children
Hermetic children get a config dir that deliberately seeds no skills —
right for children that install their own, fatal for the PTY family that
TYPES /office-hours or /plan-ceo-review: claude rejects the command as
Unknown before any model turn, so the plan-family gate smokes measure
nothing. hermeticSkillsConfigDir() is a second, opt-in config dir under
the same runRoot that mirrors ./setup's registration exactly (real dir
per registry name, SKILL.md + sections/ symlinks, frontmatter-name
resolution, _gstack-command root alias), driven by the shared
skill-census so connect-chrome's dir symlink collapses the same way
setup's idempotent overwrite does.
Ported from fork commit 03c4eca2, tree walk rewritten for the upstream
layout (top-level <skill>/SKILL.md dirs, no skills/ tree). Unit tests
are new: seed shape, census parity, symlink resolution, connect-chrome
collapse, idempotence, no-API-key seed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 93dae6107b30ce453a07c2d342b60262bba6ce0b)
* feat(evals): seedSkills opt-in for PTY slash-command tests + tripwire
Wire ClaudePtyOptions.seedSkills through launchClaudePty: when set (and
hermetic, and no per-test CLAUDE_CONFIG_DIR override), the child gets
hermeticSkillsConfigDir() so typed /skill slash commands resolve instead
of dying as Unknown command before any model turn. Opted in at the three
runPlanSkill* helpers and the four direct-launch slash-command tests
(plan-design-with-ui, plan-ceo-mode-routing, autoplan-chain,
ship-idempotency).
New static tripwire (test/pty-skill-seeding-wiring.test.ts): any test
file that sends a slash command over the PTY must route through a
runPlanSkill* helper or pass seedSkills: true — an unseeded slash-command
test spends money and measures nothing. hermetic-wiring.test.ts now
blesses the repo-tree seeding path explicitly (config dir under runRoot,
symlinks into the repo checkout, never operator ~/.claude).
The CI "Register gstack skills for PTY smoke" step keeps a keep-me note:
container cross-mount symlinks defeat the TUI scanner and HOME is not
hermeticized, so the real-file copies there must survive this change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 63c52269daaffb833b3105ea9b4b99be6df8fec7)
* refactor(evals): single shared paid-test-set module
test/helpers/paid-test-set.ts is now the one definition of which test
files are paid (the exact globs package.json's test:gate expands).
scripts/test-free-shards.ts derives its free/paid exclusion from it
instead of a private regex list, dropping the dead
browse/test/security-review-fullstack.test.ts pattern (file no longer
exists). The sharded paid runner derives its enumeration from the same
module, so a file added to one list can no longer silently miss the
other.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit a7f36479a6a1f3656452370f5883371f3cb65623)
* feat(evals): env-driven lazy eval dir + shard-aware store and tooling
Importing eval-store no longer spawns the gstack-slug subprocess: the
module-level DEFAULT_EVAL_DIR constant is now a memoized defaultEvalDir()
resolved at collector construction. Resolution order: explicit
constructor arg, then GSTACK_EVAL_DIR, then slug detection — so the
sharded paid runner can point each shard child at its own
<evalDir>/shards/<slug>/ dir with plain env, no --preload.
Runs collected under a shards/ subdir record their slug in the eval
JSON (EvalResult.shard). findPreviousRun scans one shards/<slug>/ level
and prefers same-slug priors, so each shard baselines against its own
history instead of whichever shard flushed last. eval:list,
eval:summary, and eval:compare enumerate the same one level of shard
subdirs; eval:compare's no-arg mode also stops picking an in-progress
accumulator as the after-run.
eval-watch stays flat (documented follow-up): it tails a single dir for
live progress and gains nothing from per-shard baselines until the
runner emits a merged stream.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit e1f53f7d9c7fe6b65877d843f2e25bd2e2d12ffd)
* feat(evals): sharded paid tier runner
scripts/test-paid-shards.ts runs the gate/periodic tier one Bun process
per test file, with an EXTERNAL wall-clock timeout that SIGKILLs the
shard's detached process group and an aggregate that distinguishes
passed / failed / timed-out / never-started — partial execution can no
longer read as a pass. Bun's native --shard/--isolate covers none of
this: no process-group kill (hung claude/codex PTY grandchildren
survive in-process isolation), no never-started taxonomy, no per-shard
env. Each shard child gets GSTACK_EVAL_DIR=<evalDir>/shards/<slug>/
(slug = test filename sans extension, stable across runs) so shard
baselines compare against their own prior runs.
Output classification lives in scripts/test-strict-output.ts (strict
exit-code derivation, incremental fail-line classifier, child signal
forwarding) so the runner and any future strict bun-test wrapper share
one implementation. Enumeration derives from the shared paid-test-set
module; tier exclusion fires only on an explicit whole-file
EVALS_TIER === '<other>' guard.
package.json gains test:gate:sharded / test:periodic:sharded, and
eval:bg:gate / eval:bg:periodic now run the sharded scripts with detach
timeouts sized to the worst case (gate: 49 shards x 30min / 4 jobs ~
6.2h -> 25200s; periodic: 59 -> 28800s).
test/paid-shards.test.ts pins enumeration, tier classification, and the
kill-and-continue property with a real busy-loop shard.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 5e76bd5931836257f896cedfe4e93912cb759c70)
* feat(security): hash-chained egress receipt ledger (core)
Port lib/egress-receipt from the v2 fork as TypeScript: writeReceipt
(sync, fail-closed via typed EGRESS_RECEIPT_FAILED), best-effort
writeOutcome, readLedger/listReceipts/verifyLedger, GSTACK_HOME ->
GSTACK_STATE_DIR -> ~/.gstack resolution, 0600 ledger under a 0700
security dir, and an mkdir spin lock (2.5s budget) with documented
>10s-mtime stale-lock reclaim.
Changes vs the fork:
- lastRawLine tail-reads the final 4KB instead of loading the whole
ledger, so appends stay O(1) as the file grows.
- WARN-at-size: past 25MB writeReceipt emits one self-explanatory
stderr warning per process (what the ledger is, how to inspect it,
rotation TODO); verifyLedger gains a sizeWarning field. Rotation
TODO carries the chain-genesis sketch (new generation's first record
embeds the prior file's tail hash).
bin/gstack-egress-receipt is a bun script bridging shell callers:
write|outcome subcommands, exit 3 + EGRESS_RECEIPT_FAILED on stderr on
failure; --no-payload records sha256:null for git-class ops.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 619726a3d77d987a2e50151a5727b3faaaf5fc6a)
* chore(bin): delete dead brain-consumer/reader scripts
bin/gstack-brain-consumer and bin/gstack-brain-reader are byte-identical
dead scripts that POST the repo URL + a Bearer token to a /ingest-repo
endpoint gbrain removed (docs/gbrain-sync.md already documents the
removal in past tense). No live references remain; CHANGELOG mentions
are historical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 254ddc69fc5a0270fcc973e36b6a81766d835d2d)
* feat(security): shared shell receipt helpers
bin/gstack-egress-lib.sh (sourced library, gstack-gbrain-lib.sh
precedent) provides _receipted_curl and _receipted_git: write the
egress receipt BEFORE the send via gstack-egress-receipt, hand curl the
SAME payload file via --data-binary @file so the receipt hash matches
the wire bytes exactly, then append a best-effort outcome. Per-call
fail policy: 'closed' refuses the send (return 3, problem/cause/fix
message on stderr) and 'open' warns and proceeds. Payload temp files
are consumed immediately per call — no EXIT traps, since callers like
gstack-telemetry-sync own their own EXIT trap and a sourced trap would
clobber it.
Tested end-to-end against a local Bun.serve listener: receipt sha256
equals the sha256 of the bytes the listener received, fail-closed
refusal never touches the network and carries the problem/cause/fix
stderr shape, fail-open warns and proceeds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 6d067dce2d4c8815dec98be551763c85a3671357)
* feat(security): receipt core shell sinks
Wire the three core bash egress sinks through gstack-egress-lib.sh:
- gstack-telemetry-sync: the batch POST now writes the payload to a
temp file, receipts those exact bytes fail-closed, and hands curl the
SAME file. On refusal nothing is sent and the cursor does not
advance, so the batch stays buffered for the next run. The HTTP
status is recorded as the receipt outcome.
- gstack-update-check: fail-open receipts (warn + proceed) on the
Supabase ping POST, both VERSION curls (via a local
_receipted_version_fetch helper that skips non-network schemes), and
git ls-remote. The ping receipt is written inside the backgrounded
subshell, so it can never block the script's exit.
- gstack-brain-sync: fail-closed git-class receipts. The push receipt
is written BEFORE the commit consumes the queue, so a refused receipt
leaves the queue intact and the next run retries the whole drain
(pinned by a new queue-intact-on-refusal test, including the
problem/cause/fix refusal message shape). The retry-path fetch and
retry push carry their own fail-closed receipts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 3c60f699acceaf1c92a218874711e05fc17dca5d)
* feat(security): receipt TS module sinks + tunnel
writeReceipt (fail-closed, sha256:null — a subprocess or SDK owns the
wire bytes) before every TS-module network-bearing operation:
- bin/gstack-gbrain-sync.ts: before the gbrain code walk that ships
repo content to the user's gbrain DB (may be remote Postgres). A
refused receipt fails the stage with status refused-egress-receipt.
- bin/gstack-memory-ingest.ts: before the gbrain batch import of
transcript pages. A refused receipt returns a system_error verdict
without spawning the import.
- browse/src/server.ts: before both ngrok.forward call sites (start-up
BROWSE_TUNNEL=1 path and the /tunnel/start endpoint). A receipt
failure lands in the existing catch that tears the tunnel listener
back down and refuses the start.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 5677d618a48fcd0ae2b068bf868781d90f809cb5)
* feat(design): receipted fetch for OpenAI calls
design/src/receipted-fetch.ts wraps every api.openai.com call: a
content-free egress receipt (sink design-openai, sha256 of the JSON
body — hash only, never the body) is written BEFORE the send. Polarity
is FAIL-OPEN: user-facing generation must not die because an audit log
hiccuped, so a receipt failure warns on stderr and the call proceeds.
Streams pass through untouched (response bodies returned as-is;
non-string request bodies receipted as sha256:null rather than drained
to hash).
All ten call sites converted with per-command payload classes:
generate, variants (injected fetchFn passes through), iterate (both
threaded and fresh paths), evolve (image + screenshot analysis), check,
diff, design-to-code, memory.
Unit-tested with injected fetch: receipt-before-send ordering, stream
passthrough, and fail-open on an unwritable ledger.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit c0e5ff6639414ac2fd98e8ac3affb51401746b55)
* feat(security): receipt admin scripts + user git-ops (zero exceptions)
Wire the remaining shell egress through gstack-egress-lib.sh:
- gstack-gbrain-mcp-verify: both JSON-RPC probe POSTs (initialize +
tools/list) receipted fail-closed via payload files (hash == wire
bytes). A refused receipt lands in the NETWORK class — no send.
- gstack-security-dashboard / gstack-community-dashboard: the
community-pulse GETs receipted fail-open (read-only stats must not
break over an audit hiccup).
- gstack-gbrain-supabase-provision: api_call receipted fail-closed.
Each retry attempt hands the helper a fresh copy of the body file
(the helper consumes its payload). The receipt hashes the request
body only — the PAT never reaches the ledger or any log. Refusal
exits 8 without retrying.
- git-class sha256:null receipts, fail-open: gstack-artifacts-init
(ls-remote, initial push, fetch/pull recovery, retry push),
gstack-brain-restore (staging clone, existing-repo fetch),
gstack-session-update (self-update pull).
gstack-team-init needs no wiring: every git clone in it is inside an
echoed instruction string, not an executed command.
The lib now self-locates with shell builtins only (no dirname), so
sourcing works under the whitelist-PATH test harnesses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit b8c5e2055b21ab72878b3e46f8047782ee65a11c)
* test(security): egress wiring tripwire + polarity contract
Static-grep tripwire pinning the egress-receipt wiring (threat model in
the header: the ledger is forensic observability of ATTEMPTED egress,
not an exfiltration control):
- Per-sink assertions: every wired TS module imports egress-receipt and
calls writeReceipt; every wired shell sink sources
gstack-egress-lib.sh with each network op under a receipt;
ngrok-proximity check for server.ts; every design api.openai.com call
routes through receiptedFetch.
- Absence assertions: the dead brain-consumer/reader scripts stay
deleted (lstat, so a dangling symlink also fails).
- Polarity table pinned as data (fail-closed: brain-sync,
memory-ingest, gbrain-sync, telemetry-sync, ngrok, mcp-verify,
supabase-provision; fail-open: design-openai, update-check,
dashboards, git-class user ops, context-bill --exact) plus per-file
polarity spot-checks.
- NEW-SINK SCANNER with zero KNOWN_UNWIRED: sweeps bin/, lib/,
scripts/, design/src, browse/src for curl, absolute-URL fetch(, and
git remote ops (never local rev-parse/get-url; heredoc bodies and
message strings excluded) and requires every hit to be receipted or
in a REASONED exemption list where each entry carries its why.
Preamble-generated skill prose documented out-of-scope in the header.
- Shebang tripwire: no bin/gstack-* file may carry a node shebang.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit ff69ceeafaf9c017d539b6ad77ff8f95b680b979)
* feat(cli): gstack-egress reader
bin/gstack-egress (bun) — the auditor's view of the receipts ledger:
- list: one row per receipt (what gstack ATTEMPTED to send), with
--since/--host/--sink filters and --json.
- verify: recompute the hash chain; exit 3 on tamper naming the first
broken line; prints the sizeWarning when the ledger passes 25MB.
- grants: what CAN leave, built on the upstream config keys only
(telemetry, artifacts_sync_mode, redact_repo_visibility,
redact_prepush_hook via gstack-config get) — each grant names its
file, key, and the exact revoke command.
CLI smoke tests spawn the real bin against a temp GSTACK_HOME,
including a broken-chain fixture asserting exit 3.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 9e24eca0f1069fea2ea69e7df4e9b256e93d59a3)
* feat(cli): context-bill — token bill-of-materials (stripped port)
lib/context-bill.ts, ported from the v2 fork and STRIPPED to the tiers
this repo's skills can exercise: ALWAYS-ON (per-skill frontmatter bytes
with dead-key and foreign-host-file flags), EAGER (SKILL.md + any
forced 'for every invocation' references), on-disk totals, --diff,
--budget, and --exact with the calibration table. The fork's
CONDITIONAL/TRANSITIVE/LAZY/FAST-PATH parsers understand only its
dispatcher layout and were dropped; the tier fields stay in the report
shape (empty/zero/null) so re-adding a parser is additive.
TOKEN_DIVISORS and their provenance docblock kept; --help notes
recalibration via --exact's calibration block.
Three upstream fixes over the fork:
(a) findSkillDirs treats the walk ROOT as a container — the repo root's
router SKILL.md is billed AND its children are walked (the fork
short-circuited and billed one skill); walkMd skips node_modules
and dot-directories.
(b) installed-tree layout: subdirs that are their own repo checkout
(a gstack/ clone inside ~/.claude/skills, detected by .git) are
skipped, and directory symlinks (connect-chrome) are followed with
a container-recursion cycle guard.
(c) ROUTER_KEYS widened to the upstream frontmatter contract {name,
description, version, allowed-tools, triggers, preamble-tier}.
--exact writes an egress receipt (sink context-bill-exact, host
api.anthropic.com) BEFORE any count_tokens POST; if the receipt cannot
be written the run degrades to the offline estimate with a warning —
nothing is sent unrecorded. bin/gstack-context-bill is the bun shim.
Tests: fixture-tree ledgers, the three fixes, --diff/--budget exit
codes, --exact with injected fetch (envelope subtraction, receipt
ordering, fail-open degradation), CLI smoke test, and ground truth
against THIS repo via test/helpers/skill-census.ts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 675c19876b87ec927b555f5f64c7f93130b3de90)
* test(catalog): aggregate discovery-surface budget with ratchet protocol
Every host loads every skill's frontmatter name + description at
discovery, every session. applyCatalogTrim in scripts/gen-skill-docs.ts
shapes each description and the 160KB per-file warn covers body size,
but nothing capped the aggregate frontmatter — the catalog could grow
one reasonable-looking description at a time. This test is that
enforcement layer.
Measures the catalog via test/helpers/skill-census.ts authoredSkills
(symlink-deduped, root router counted separately as the _gstack-command
alias line item): 53 skills + router = 4,420 bytes = 1,105
token-equivalents today, asserted <= 1,150 (~4% headroom). Per-skill
sub-cap of 260 bytes (largest today: design-consultation at 229), plus
a non-empty-description check.
Failure messages are self-service ratchets: they print the new total,
the delta, and the update protocol (bump the constant AND the
derivation comment in the same commit; trim instead of grow for
existing descriptions). Parser handles folded block scalars
(description: >-) for fork parity; import-free by design so it
survives generator refactors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit c106fb36f768181b80c257e5cff1cde4f435f9c0)
* fix(browse): extension token bootstrap moves to pinned-origin POST; /health carries no token
GET /health is now liveness/status only in every mode — both token
carve-outs (headed-mode disjunct AND chrome-extension:// Origin
disjunct) are removed. Token bootstrap is POST /extension-token on the
local listener: the Origin header must be exactly
chrome-extension://<GSTACK_EXTENSION_ID> and the Host header's hostname
must parse to 127.0.0.1 or localhost (parsed via new URL, never literal
equality — Host arrives as '127.0.0.1:34567'). Wrong origin/host → 403
with no detail. The tunnel surface 404s the endpoint (not in
TUNNEL_PATHS, verified by test).
The extension ID is pinned by a new "key" field (RSA public key) in
extension/manifest.json; browse/scripts/extension-id.ts reproduces the
ID derivation (first 16 bytes of SHA-256 of the DER public key, hex
mapped 0-9a-f → a-p). The private key is not committed anywhere —
unpacked/baked-in loads only need the public key.
Extension side: background.js bootstraps and refreshes the token via
POST /extension-token (403 → disconnected state); sidepanel.js direct
connect path does the same; sidepanel-terminal.js's dead /health token
fallback (read AUTH_TOKEN/authToken keys the server never sent,
hardcoded port) is replaced with the window.gstackAuthToken path.
MIGRATION NOTE: the manifest key pins the extension ID, so existing
installs' side-panel local state (saved port, snoozes) resets once —
explained in-product via a one-time notice (flag
gstack_id_migrated_v162). After upgrading the server, restart the
browser so the old service worker stops polling for a token GET /health
no longer serves.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit e9a0b6847a2d17fe6656a4686b4efd0c8380eb09)
* docs: correct stale compiled-binaries claim; file three egress/eval follow-ups
CLAUDE.md's compiled-binaries section claimed browse/dist binaries are
tracked by git and appear as modified in git status — false since
64d5a3e4 (v0.11.16.0) untracked them, and actively harmful: it trained
agents to ignore dist binaries in git status. The section now states
the truth (untracked + gitignored; a dist binary in git status means
someone force-added it) and covers make-pdf/dist too.
TODOS.md gains the three follow-ups filed by the v1.62 port-wave
reviews: ledger rotation with chain-genesis records, launch-nonce
token bootstrap, and eval-watch shard-awareness.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: pre-landing review fixes for the v2 port wave
Review army (checklist + 5 specialists) + coverage/plan audits on the
assembled branch. Genuine correctness/security/hygiene fixes:
- test-paid-shards: strictTestExitCode now receives expectedFiles on the
real bun path, so a shard that runs fewer files than planned (harness
crash, nothing loaded) with exit 0 is no longer recorded 'passed' — the
invisible-non-execution class the runner exists to kill. Pinned by the
new test/strict-output.test.ts (also covers the chunk-boundary classifier).
- test-paid-shards: EVALS_TIER env is validated (gate|periodic) like the
--tier flag, so a typo can't self-skip every test and exit 0 green.
- package.json: test:periodic:sharded sets EVALS_ALL=1, restoring the
full-tier semantics the pre-shard script had (CI already set it; local
eval:bg:periodic silently under-measured without it).
- brain-sync.test: run() pins HOME to the temp home so gstack-artifacts-init
stops writing/clobbering the operator's real ~/.gstack-artifacts-remote.txt
every free-suite run; afterEach now also scrubs the current filename.
- egress-receipt: cap each receipt field at 512B so a serialized line always
fits the 4KB tail-read window — a longer line would make the next append
hash a truncated prior line and verifyLedger report a permanent false
TAMPER. warnLedgerSize short-circuits before statSync once fired (append
hot path).
- gstack-egress: import.meta.dir (Windows-safe) instead of new URL().pathname
so grants doesn't silently report defaults on Windows; strip control chars
from ledger-derived fields on render so a crafted receipt can't spoof the
auditor's view.
- extension/background.js + CLAUDE.md: renumber the identity-pin migration
refs v1.62 -> v1.63 (main claimed 1.62.0.0; this wave queue-advances).
- egress-receipt-wiring: pin lib/context-bill.ts unconditionally (both land
together now); drop the dead RunShardsOptions.tier field.
All fix-affected test files green; gate failures triaged as external-env
(codex/gemini CLI drift) or pre-existing (hermetic-canary fails identically
on base). Deferred polish tracked in the PR body + decision store.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: bump version and changelog (v1.63.0.0)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: file TODO to harden plan-design-with-ui PTY detection
The v1.63 seedSkills change made this gate test execute for the first
time; it reliably times out because its terminal scraper can't parse the
(correctly-rendered) scope-gate AskUserQuestion out of a spinner-mangled
PTY buffer. Shipped skill behavior is correct — test-harness limitation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: re-slot release as v1.62.1.0 (PATCH per user)
Main claimed 1.62.0.0 while the wave was in flight; the user chose the
PATCH slot over queue-advancing MINOR. Renumbers the identity-pin
migration notice (now version-free flag name so a re-slot never orphans
an already-set flag), the CLAUDE.md /health note, the CHANGELOG heading,
and the TODOS section titles.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(test): stop hard-requiring the literal ok) case label in gbrain-refresh guards
The extractor grepped for 'ok)' but the case label grew to
ok|timeout|thin-client) (#1964, #2051), so the whole file errored on
import — the free suite's only red for months. The extractor now matches
any label starting with ok and its alternations; all 7 guard assertions
run again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(test): hermetic-canary probes with ${VAR:-} so nounset shells can't fail success
The probe echoed bare $CONDUCTOR_WORKSPACE_PATH — when scrubbing WORKS
the var is unset, and under a nounset shell the echo errors, failing the
canary exactly when isolation succeeds. Defaulted expansions assert
identically under any shell. Fails identically on base; fixed here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(evals): absorb codex/gemini CLI drift; external-service tests go periodic-tier
- codex exec gains --skip-git-repo-check: newer CLIs refuse exec in an
untrusted non-git dir (our temp skill dirs) — empirically verified.
- gemini: --skip-trust was removed in gemini-cli 0.34 (argv parse error);
dropped from the session runner and the benchmark adapter. A present-
but-unusable CLI (deprecated individual code-assist auth path) now
classifies as SKIP, not a false adapter failure; the benchmark live
smoke skips on auth/rate_limit error codes (environmental) while still
failing on timeout/unknown (the drift classes it exists to catch).
- codex-e2e, gemini-e2e, and benchmark-providers gain the canonical
whole-file EVALS_TIER === 'periodic' guard per CLAUDE.md tiering rule 3
(external service -> periodic) — the sharded gate runner now excludes
all three (gate: 45 -> 42 shards).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(evals): parse single-logical-line AskUserQuestions in the PTY runner
When the PTY reflows a boxed AUQ, ALL options land on ONE logical line
after stripAnsi — parseNumberedOptions parsed one option per line, found
only '1.', and the >=2 check failed forever while the correct question
sat on screen (plan-design-with-ui timed out this way twice, with the
rendered scope-gate AUQ visible in both failure buffers). The cursor
line is now parsed as a stream of ascending N. tokens; DEC cursor-
visibility residue is stripped before matching; plan-design-with-ui's
budgets grow to fit observed ~6min preamble+thinking latency. Pinned by
test/pty-auq-single-line.test.ts using the real failure buffers; all 142
existing parser-consumer unit tests still green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: restore v1.63.0.0 (MINOR — user-confirmed final slot)
The wave ships new capability (egress receipts + two CLIs, sharded paid
runner, hermetic skill seeding) at ~8K lines — MINOR scale per the
scale-aware bump rules. Supersedes the brief v1.62.1.0 re-slot; the
version-free migration flag means no state churn from the renumber.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: spell out AskUserQuestion in the PTY single-line fixture
Rename test/pty-auq-single-line.test.ts to
test/pty-askuserquestion-single-line.test.ts and expand the AUQ
abbreviation in identifiers and comments. House style writes
AskUserQuestion in full in filenames, identifiers, and comments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: sync every doc surface with the v1.63 release
/document-release audit (4-lane, all claims verified against branch code):
- README: gstack-egress + gstack-context-bill rows in the standalone-binaries
table; Privacy & Telemetry gains the receipted-egress bullet (attempted-
egress framing per the shipped threat model).
- ARCHITECTURE: /health is liveness-only, POST /extension-token endpoint row
+ bootstrap mechanics paragraph; new Egress receipt ledger subsection under
Security model; eval persistence covers the sharded runner, GSTACK_EVAL_DIR,
and the finalized-run baseline rule.
- CLAUDE.md: sharded test scripts in Commands; sharded semantics in the
detached-evals section; PTY skill seeding in the hermetic section; egress
invariant block beside the other server-egress invariants; catalog-budget
ceiling beside the 160KB token ceiling; project-tree entries for
lib/egress-receipt.ts, lib/context-bill.ts, scripts/test-paid-shards.ts.
- CONTRIBUTING: seedSkills + live-tree seeding in the hermetic paragraph;
sharded runner in detached runs; catalog-budget in the Tier 1 list.
- BROWSER: extension token bootstrap section, tunnel egress receipts section,
identity-pin migration note in manual install.
- REMOTE_BROWSER_ACCESS: tunnel-start receipt bullet in the security model.
- gbrain docs: /sync-gbrain + brain-sync egress-receipt behavior documented;
dead consumer-token instructions removed (consumer machinery deleted this
release); new fail-closed refusal added to the error catalog.
- CHANGELOG: measured-vs-ceiling catalog numbers, contributor notes for the
external-service tier move and the PTY single-line AskUserQuestion parser,
release date.
- TODOS: /health token-distribution TODO resolved by this release, removed;
port-wave follow-up sections re-labeled to the shipped version.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: sweep drift that predates this release
Surfaced by the /document-release audit; every fix verified against the
current binaries:
- gstack-brain-init was replaced by gstack-artifacts-init in v1.27.0.0
(hard-delete, no compat shim), but README, USING_GBRAIN_WITH_GSTACK,
docs/gbrain-sync.md, and docs/gbrain-sync-errors.md still instructed
users to run it — command-not-found on every follow. Same sweep updates
~/.gstack-brain-remote.txt to the canonical ~/.gstack-artifacts-remote.txt
(legacy name still honored on restore, noted where users copy the file).
- gbrain-sync-errors.md headings re-matched to the literal messages the
binaries print today (the doc's whole value is grep-by-exact-message):
'gstack-artifacts-init: ~/.gstack/ is already a git repo pointing at:',
'Remote not reachable via SSH:', 'Failed to create or find ...'. The
already-a-repo fix now leads with the command's own set-url suggestion.
- docs/gbrain-sync.md 'Under the hood' linked a plan file that does not
exist in the repo; replaced with the decisions themselves.
- SIDEBAR_MESSAGE_FLOW startup timeline: /pty-session responds with
{terminalPort, sessionId, attachToken, leaseExpiresAt} (v1.44 shape,
verified at browse/src/server.ts:1860), not the retired
{terminalPort, ptySessionToken} pair.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: fold the Codex accuracy review of the release docs
Six findings, all verified against source before fixing:
1. 'Every send writes a receipt' overclaimed — fail-open sinks proceed with
a stderr warning when the receipt write fails, so a fail-open send can go
unrecorded (lib/egress-receipt.ts:8-14). Descriptive prose now says so;
the receipted framing keeps 'attempted'.
2. 'Receipts hash the request body' is wrong for subprocess-owned sends —
git pushes record sha256: null (lib/egress-receipt.ts:71).
3. 'grants shows every consent in force' overclaimed — it reports the four
standing config settings (bin/gstack-egress:139-181). Reworded in
README, ARCHITECTURE, and the CHANGELOG entry.
4. 'Zero-exception scanner' vs reality: the new-sink scanner carries a
reasoned SCANNER_EXEMPT list (user-directed fetches, probes, instruction
strings, skill prose). CLAUDE.md now names it.
5. Error-catalog cause/fix for the receipt refusal: the writer mkdirs the
ledger dir itself, so 'missing' isn't a cause and bare chmod fails when
it is absent — cause reworded, fix is mkdir -p && chmod.
6. gbrain-sync first-run steps described the retired binary's behavior:
default repo is gstack-artifacts-$USER, and init PRINTS the gbrain
hookup command (never auto-executes; bin/gstack-artifacts-init:384-419).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sinabina <sinabina@Sinabinas-MacBook-Pro-4.local>
This commit is contained in:
103 files changed
+7174
-618
No files matched your search
+83
-8
@@ -34,7 +34,11 @@ function run(argv: string[], opts: { env?: Record<string, string>; input?: strin
|
||||
const bin = argv[0];
|
||||
const full = bin.startsWith('/') ? bin : path.join(BIN, bin);
|
||||
const res = spawnSync(full, argv.slice(1), {
|
||||
env: { ...process.env, GSTACK_HOME: tmpHome, ...(opts.env || {}) },
|
||||
// HOME is overridden too: gstack-artifacts-init writes
|
||||
// $HOME/.gstack-artifacts-remote.txt (plain $HOME, not GSTACK_HOME), so
|
||||
// without this every free-suite run clobbers the operator's real
|
||||
// artifacts-remote pointer. Keep it inside tmpHome, which afterEach removes.
|
||||
env: { ...process.env, HOME: tmpHome, GSTACK_HOME: tmpHome, ...(opts.env || {}) },
|
||||
encoding: 'utf-8',
|
||||
input: opts.input,
|
||||
cwd: ROOT,
|
||||
@@ -56,13 +60,18 @@ beforeEach(() => {
|
||||
afterEach(() => {
|
||||
fs.rmSync(tmpHome, { recursive: true, force: true });
|
||||
fs.rmSync(bareRemote, { recursive: true, force: true });
|
||||
// Clean up any remote-helper file init may have written.
|
||||
const remoteFile = path.join(os.homedir(), '.gstack-brain-remote.txt');
|
||||
// Only remove if it points at OUR bare remote (don't clobber a real user file).
|
||||
try {
|
||||
const contents = fs.readFileSync(remoteFile, 'utf-8').trim();
|
||||
if (contents === bareRemote) fs.unlinkSync(remoteFile);
|
||||
} catch {}
|
||||
// Clean up any remote-helper file init may have written. run() now pins
|
||||
// HOME to tmpHome so these land inside the removed temp dir, but scrub the
|
||||
// real home too as defense in depth — and cover BOTH the legacy brain-remote
|
||||
// name and the current artifacts-remote name (init writes the latter).
|
||||
for (const name of ['.gstack-brain-remote.txt', '.gstack-artifacts-remote.txt']) {
|
||||
const remoteFile = path.join(os.homedir(), name);
|
||||
// Only remove if it points at OUR bare remote (don't clobber a real user file).
|
||||
try {
|
||||
const contents = fs.readFileSync(remoteFile, 'utf-8').trim();
|
||||
if (contents === bareRemote) fs.unlinkSync(remoteFile);
|
||||
} catch {}
|
||||
}
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
@@ -330,6 +339,72 @@ describe('gstack-brain-sync secret scan', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Egress receipt gate: receipt-before-commit, queue intact on refusal
|
||||
// ---------------------------------------------------------------
|
||||
describe('gstack-brain-sync egress receipt gate', () => {
|
||||
test('refused receipt leaves the queue intact, makes no commit, and next run retries', () => {
|
||||
if (process.platform === 'win32' || process.getuid?.() === 0) return; // chmod is advisory there
|
||||
run(['gstack-artifacts-init', '--remote', bareRemote]);
|
||||
run(['gstack-config', 'set', 'artifacts_sync_mode', 'full']);
|
||||
fs.mkdirSync(path.join(tmpHome, 'projects', 'p'), { recursive: true });
|
||||
fs.writeFileSync(path.join(tmpHome, 'projects/p/learnings.jsonl'),
|
||||
'{"skill":"x","insight":"y","ts":"2026-04-22T10:00:00Z"}\n');
|
||||
run(['gstack-brain-enqueue', 'projects/p/learnings.jsonl']);
|
||||
const commitsBefore = git(['rev-list', '--count', 'HEAD']).stdout.trim();
|
||||
|
||||
// Make the receipt unwritable: security dir exists but is read-only.
|
||||
// (artifacts-init may have created it already — mkdirSync's mode is a
|
||||
// no-op on an existing dir, so chmod explicitly.)
|
||||
fs.mkdirSync(path.join(tmpHome, 'security'), { recursive: true });
|
||||
fs.chmodSync(path.join(tmpHome, 'security'), 0o500);
|
||||
try {
|
||||
const refused = run(['gstack-brain-sync', '--once']);
|
||||
expect(refused.status).toBe(1);
|
||||
// DX contract: problem + cause + fix, plain language.
|
||||
expect(refused.stderr).toContain('NOT sent');
|
||||
expect(refused.stderr).toContain('EGRESS_RECEIPT_FAILED');
|
||||
expect(refused.stderr).toContain('Fix: chmod -R u+w');
|
||||
expect(refused.stderr).toContain('ATTEMPTS to send off-machine');
|
||||
// Queue intact (receipt is written BEFORE the commit consumes it).
|
||||
const queue = fs.readFileSync(path.join(tmpHome, '.brain-queue.jsonl'), 'utf-8');
|
||||
expect(queue).toContain('projects/p/learnings.jsonl');
|
||||
// No local commit was created.
|
||||
expect(git(['rev-list', '--count', 'HEAD']).stdout.trim()).toBe(commitsBefore);
|
||||
// Nothing reached the remote.
|
||||
const remoteLog = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8' });
|
||||
expect(remoteLog.stdout).not.toMatch(/sync: 1 file/);
|
||||
const status = JSON.parse(fs.readFileSync(path.join(tmpHome, '.brain-sync-status.json'), 'utf-8'));
|
||||
expect(status.status).toBe('push_failed');
|
||||
expect(status.message).toContain('EGRESS_RECEIPT_FAILED');
|
||||
} finally {
|
||||
fs.chmodSync(path.join(tmpHome, 'security'), 0o700);
|
||||
}
|
||||
|
||||
// Next run (ledger writable again) drains the intact queue and pushes.
|
||||
const retry = run(['gstack-brain-sync', '--once']);
|
||||
expect(retry.status).toBe(0);
|
||||
const log = spawnSync('git', ['--git-dir=' + bareRemote, 'log', '--oneline'], { encoding: 'utf-8' });
|
||||
expect(log.stdout).toMatch(/sync: 1 file/);
|
||||
});
|
||||
|
||||
test('successful push writes a git-class receipt before the send', () => {
|
||||
run(['gstack-artifacts-init', '--remote', bareRemote]);
|
||||
run(['gstack-config', 'set', 'artifacts_sync_mode', 'full']);
|
||||
fs.mkdirSync(path.join(tmpHome, 'projects', 'p'), { recursive: true });
|
||||
fs.writeFileSync(path.join(tmpHome, 'projects/p/learnings.jsonl'),
|
||||
'{"skill":"x","insight":"y","ts":"2026-04-22T10:00:00Z"}\n');
|
||||
run(['gstack-brain-enqueue', 'projects/p/learnings.jsonl']);
|
||||
const r = run(['gstack-brain-sync', '--once']);
|
||||
expect(r.status).toBe(0);
|
||||
const ledger = fs.readFileSync(path.join(tmpHome, 'security', 'egress.jsonl'), 'utf-8');
|
||||
const records = ledger.trim().split('\n').map((l) => JSON.parse(l));
|
||||
const pushReceipt = records.find((rec) => rec.sink === 'brain-sync' && rec.payload_class === 'curated-memory-git-push');
|
||||
expect(pushReceipt).toBeTruthy();
|
||||
expect(pushReceipt.sha256).toBeNull(); // git owns the bytes
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Uninstall preserves user data
|
||||
// ---------------------------------------------------------------
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { skillCensus } from './helpers/skill-census';
|
||||
|
||||
/**
|
||||
* Aggregate discovery-surface budget: the sum of every skill's frontmatter
|
||||
* `name` + `description` is what EVERY host loads at discovery, every session.
|
||||
*
|
||||
* This is the missing enforcement layer over the existing catalog-trim
|
||||
* mechanism: `applyCatalogTrim` in scripts/gen-skill-docs.ts (~line 865)
|
||||
* shapes each description, and the 160KB per-file warn (~line 1015) covers
|
||||
* BODY size — neither caps the aggregate frontmatter the catalog is made of.
|
||||
*
|
||||
* Import-free by design: parses skills' SKILL.md frontmatter directly. Do not
|
||||
* import gen-skill-docs internals here — this test must survive generator
|
||||
* refactors.
|
||||
*
|
||||
* Budget derivation (re-derive it, do not trust the number):
|
||||
* ref this commit
|
||||
* method for each authored skill (test/helpers/skill-census.ts
|
||||
* authoredSkills — symlink-deduped, root router excluded) plus the
|
||||
* root router's `_gstack-command` alias frontmatter as one separate
|
||||
* line item, run parseFrontmatter() below and sum
|
||||
* Buffer.byteLength(name) + Buffer.byteLength(description);
|
||||
* token-equivalents = ceil(bytes / 4).
|
||||
* result 53 authored skills = 4,371 bytes (1,093 token-equivalents);
|
||||
* + root router alias 49 bytes = 4,420 bytes total
|
||||
* = 1,105 token-equivalents (measured 2026-08-12)
|
||||
* Ceiling is 1,150 token-equivalents (4,600 bytes), so headroom is 180 bytes
|
||||
* (~4%). Dominant skill: design-consultation at 229 bytes name+description.
|
||||
*/
|
||||
const CATALOG_BUDGET_TOKEN_EQUIVALENTS = 1_150;
|
||||
|
||||
// Largest today: design-consultation at 229 bytes. A description that needs
|
||||
// more than 260 bytes is a body paragraph, not a catalog entry.
|
||||
const PER_SKILL_BYTE_CAP = 260;
|
||||
|
||||
const RATCHET_PROTOCOL =
|
||||
'Adding a skill? Re-measure with: bun test test/catalog-budget.test.ts ' +
|
||||
'(the failure prints the new total). Update CATALOG_BUDGET_TOKEN_EQUIVALENTS ' +
|
||||
'AND the derivation comment (ref/date/value/which skill moved it) in the ' +
|
||||
'SAME commit. Growing an existing description? Trim it instead — the ' +
|
||||
'catalog is what every host loads at discovery, every session.';
|
||||
|
||||
const ROOT = join(import.meta.dir, '..');
|
||||
|
||||
function parseFrontmatter(body: string): { name: string; description: string } {
|
||||
const name = body.match(/^name:\s*(.+)$/m)?.[1]?.trim() ?? '';
|
||||
// Folded block scalar (description: >-) with two-space-indented continuation
|
||||
// lines, falling back to a single-line description.
|
||||
const folded = body.match(/^description:\s*>-?\r?\n((?: .*\r?\n)+)/m)?.[1];
|
||||
const description = folded
|
||||
? folded.split(/\r?\n/).map((line) => line.trim()).filter(Boolean).join(' ')
|
||||
: body.match(/^description:\s*(?!>-?\s*$)(.+)$/m)?.[1]?.trim() ?? '';
|
||||
return { name, description };
|
||||
}
|
||||
|
||||
interface CatalogEntry {
|
||||
skill: string;
|
||||
name: string;
|
||||
description: string;
|
||||
bytes: number;
|
||||
}
|
||||
|
||||
function catalogEntries(): CatalogEntry[] {
|
||||
const entries: CatalogEntry[] = [];
|
||||
for (const skill of skillCensus(ROOT).authoredSkills) {
|
||||
const body = readFileSync(join(ROOT, skill, 'SKILL.md'), 'utf8');
|
||||
const { name, description } = parseFrontmatter(body);
|
||||
entries.push({
|
||||
skill,
|
||||
name,
|
||||
description,
|
||||
bytes: Buffer.byteLength(name) + Buffer.byteLength(description),
|
||||
});
|
||||
}
|
||||
// The root SKILL.md is a router, registered by setup as the
|
||||
// `_gstack-command` alias — not an authored skill, but its frontmatter
|
||||
// still ships in the catalog, so it counts as one line item.
|
||||
const router = parseFrontmatter(readFileSync(join(ROOT, 'SKILL.md'), 'utf8'));
|
||||
if (router.name && router.description) {
|
||||
entries.push({
|
||||
skill: '(root router)',
|
||||
name: router.name,
|
||||
description: router.description,
|
||||
bytes: Buffer.byteLength(router.name) + Buffer.byteLength(router.description),
|
||||
});
|
||||
}
|
||||
return entries;
|
||||
}
|
||||
|
||||
describe('catalog discovery-surface budget', () => {
|
||||
test(`aggregate frontmatter stays within ${CATALOG_BUDGET_TOKEN_EQUIVALENTS} token-equivalents`, () => {
|
||||
const entries = catalogEntries();
|
||||
const totalBytes = entries.reduce((sum, e) => sum + e.bytes, 0);
|
||||
const estimatedTokens = Math.ceil(totalBytes / 4);
|
||||
const delta = estimatedTokens - CATALOG_BUDGET_TOKEN_EQUIVALENTS;
|
||||
expect(
|
||||
estimatedTokens,
|
||||
`Catalog is ${estimatedTokens} token-equivalents (${totalBytes} bytes), ` +
|
||||
`${delta} over the ${CATALOG_BUDGET_TOKEN_EQUIVALENTS} budget. ${RATCHET_PROTOCOL}`
|
||||
).toBeLessThanOrEqual(CATALOG_BUDGET_TOKEN_EQUIVALENTS);
|
||||
});
|
||||
|
||||
test(`every skill's name + description stays under ${PER_SKILL_BYTE_CAP} bytes`, () => {
|
||||
for (const entry of catalogEntries()) {
|
||||
expect(
|
||||
entry.bytes,
|
||||
`${entry.skill}: name + description is ${entry.bytes} bytes, ` +
|
||||
`${entry.bytes - PER_SKILL_BYTE_CAP} over the ${PER_SKILL_BYTE_CAP}-byte ` +
|
||||
`per-skill cap. ${RATCHET_PROTOCOL}`
|
||||
).toBeLessThanOrEqual(PER_SKILL_BYTE_CAP);
|
||||
}
|
||||
});
|
||||
|
||||
test('every skill has a non-empty description', () => {
|
||||
for (const entry of catalogEntries()) {
|
||||
expect(entry.description, `${entry.skill}: empty or missing frontmatter description`).not.toBe('');
|
||||
}
|
||||
});
|
||||
});
|
||||
+11
-2
@@ -37,15 +37,24 @@ const CODEX_AVAILABLE = (() => {
|
||||
|
||||
const evalsEnabled = !!process.env.EVALS;
|
||||
|
||||
// Skip all tests if codex is not available or EVALS is not set.
|
||||
// External-service tests are periodic-tier (CLAUDE.md tiering rule 3):
|
||||
// "Requires external service (Codex, Gemini)? -> periodic". The positive
|
||||
// form below is the canonical whole-file guard shape — the sharded runner's
|
||||
// classifyPaidTestFile greps for it to exclude this file from gate.
|
||||
const tierOk = process.env.EVALS_TIER === 'periodic';
|
||||
|
||||
// Skip all tests if codex is not available, EVALS is not set, or we're in
|
||||
// the gate tier.
|
||||
// Note: Codex uses its own auth from ~/.codex/ config — no OPENAI_API_KEY env var needed.
|
||||
const SKIP = !CODEX_AVAILABLE || !evalsEnabled;
|
||||
const SKIP = !CODEX_AVAILABLE || !evalsEnabled || !tierOk;
|
||||
|
||||
const describeCodex = SKIP ? describe.skip : describe;
|
||||
|
||||
// Log why we're skipping (helpful for debugging CI)
|
||||
if (!evalsEnabled) {
|
||||
// Silent — same as Claude E2E tests, EVALS=1 required
|
||||
} else if (!tierOk) {
|
||||
process.stderr.write('\nCodex E2E: SKIPPED — external-service test, periodic tier only (EVALS_TIER === \'periodic\')\n');
|
||||
} else if (!CODEX_AVAILABLE) {
|
||||
process.stderr.write('\nCodex E2E: SKIPPED — codex binary not found (install: npm i -g @openai/codex)\n');
|
||||
}
|
||||
|
||||
@@ -0,0 +1,679 @@
|
||||
/**
|
||||
* gstack-context-bill — token bill-of-materials for an installed skills tree.
|
||||
*
|
||||
* Free tier, no network, no API keys. Covers the STRIPPED port:
|
||||
* - ALWAYS-ON ledger: exact frontmatter byte sums, dead-key flag against
|
||||
* the upstream router-key contract, foreign-host file flag
|
||||
* - EAGER ledger: SKILL.md + forced-read refs from the "for every
|
||||
* invocation" phrase; stripped tiers stay zero/empty (shape preserved)
|
||||
* - the three upstream fixes: (a) root-as-container walking +
|
||||
* node_modules/dotdir exclusion in walkMd, (b) repo-checkout subdir skip
|
||||
* for installed trees, (c) widened ROUTER_KEYS
|
||||
* - token estimate calibration, --json shape, --diff, --budget exit codes
|
||||
* - --exact via injected fetch: envelope subtraction, measurement
|
||||
* replacement, typed failures, egress receipt-before-send + fail-open
|
||||
* - ground truth against THIS repo via test/helpers/skill-census.ts
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, afterAll } from "bun:test";
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import {
|
||||
buildBill,
|
||||
calibrationTable,
|
||||
checkBudget,
|
||||
contentClass,
|
||||
contextBillMain,
|
||||
diffBills,
|
||||
estimateTokens,
|
||||
findSkillDirs,
|
||||
measureExactTokens,
|
||||
renderBill,
|
||||
tokenDisclaimer,
|
||||
walkMd,
|
||||
ExactModeError,
|
||||
TOKEN_DIVISOR,
|
||||
TOKEN_DIVISORS,
|
||||
} from "../lib/context-bill";
|
||||
import { listReceipts, sha256Hex } from "../lib/egress-receipt";
|
||||
import { skillCensus } from "./helpers/skill-census";
|
||||
|
||||
const ROOT = path.join(import.meta.dir, "..");
|
||||
const TREE_A = path.join(import.meta.dir, "fixtures", "context-bill", "tree-a");
|
||||
|
||||
function fileBytes(...segments: string[]): number {
|
||||
return fs.statSync(path.join(...segments)).size;
|
||||
}
|
||||
|
||||
/** Frontmatter block bytes, computed independently of the implementation. */
|
||||
function frontmatterBytes(file: string): number {
|
||||
const text = fs.readFileSync(file, "utf8");
|
||||
const close = text.indexOf("\n---\n", 3);
|
||||
return Buffer.byteLength(text.slice(0, close + 5), "utf8");
|
||||
}
|
||||
|
||||
function capture() {
|
||||
let buf = "";
|
||||
return {
|
||||
stream: { write: (s: string) => ((buf += s), true) } as unknown as NodeJS.WriteStream,
|
||||
text: () => buf,
|
||||
};
|
||||
}
|
||||
|
||||
describe("always-on ledger", () => {
|
||||
const bill = buildBill(TREE_A);
|
||||
|
||||
it("sums per-skill frontmatter bytes exactly", () => {
|
||||
const alpha = bill.skills.find((s) => s.name === "alpha")!;
|
||||
const beta = bill.skills.find((s) => s.name === "beta")!;
|
||||
expect(alpha.frontmatterBytes).toBe(frontmatterBytes(path.join(TREE_A, "alpha", "SKILL.md")));
|
||||
expect(beta.frontmatterBytes).toBe(frontmatterBytes(path.join(TREE_A, "beta", "SKILL.md")));
|
||||
expect(bill.totals.alwaysOnBytes).toBe(alpha.frontmatterBytes + beta.frontmatterBytes);
|
||||
});
|
||||
|
||||
it("flags only keys outside the upstream router contract (fix c: widened ROUTER_KEYS)", () => {
|
||||
const alpha = bill.skills.find((s) => s.name === "alpha")!;
|
||||
// `triggers` is part of the upstream frontmatter contract now — flagging
|
||||
// it was the fork's contract, not this repo's.
|
||||
expect(alpha.deadKeys).toEqual(["x-dead-key"]);
|
||||
expect(bill.skills.find((s) => s.name === "beta")!.deadKeys).toEqual([]);
|
||||
});
|
||||
|
||||
it("upstream contract keys are never dead: name/description/version/allowed-tools/triggers/preamble-tier", () => {
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-keys-"));
|
||||
fs.mkdirSync(path.join(tmp, "s"));
|
||||
fs.writeFileSync(
|
||||
path.join(tmp, "s", "SKILL.md"),
|
||||
"---\nname: s\ndescription: d\nversion: 1\nallowed-tools: Bash\ntriggers: t\npreamble-tier: 2\n---\n\n# S\n",
|
||||
);
|
||||
const s = buildBill(tmp).skills[0];
|
||||
expect(s.deadKeys).toEqual([]);
|
||||
fs.rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("flags foreign-host skill-shaped files in scanner scope", () => {
|
||||
const beta = bill.skills.find((s) => s.name === "beta")!;
|
||||
expect(beta.foreignFiles.map((f) => f.path)).toEqual(["agents.md"]);
|
||||
expect(bill.skills.find((s) => s.name === "alpha")!.foreignFiles).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("eager ledger", () => {
|
||||
const bill = buildBill(TREE_A);
|
||||
const alpha = bill.skills.find((s) => s.name === "alpha")!;
|
||||
|
||||
it("eager = SKILL.md + only the forced-read references", () => {
|
||||
expect(alpha.forcedRefs.map((r) => r.path)).toEqual([
|
||||
"references/CORE.md",
|
||||
"references/POLICY.md",
|
||||
]);
|
||||
const expected =
|
||||
fileBytes(TREE_A, "alpha", "SKILL.md") +
|
||||
fileBytes(TREE_A, "alpha", "references", "CORE.md") +
|
||||
fileBytes(TREE_A, "alpha", "references", "POLICY.md");
|
||||
expect(alpha.eagerBytes).toBe(expected);
|
||||
});
|
||||
|
||||
it("a skill with no forced reads bills only its SKILL.md", () => {
|
||||
const beta = bill.skills.find((s) => s.name === "beta")!;
|
||||
expect(beta.eagerBytes).toBe(fileBytes(TREE_A, "beta", "SKILL.md"));
|
||||
});
|
||||
|
||||
it("stripped tiers stay zero/empty but keep their shape (re-adding is additive)", () => {
|
||||
// OPTIONAL.md is mandated under a condition and the mode table routes two
|
||||
// legacy modules — the fork billed those in CONDITIONAL and LAZY. The
|
||||
// stripped port must not bill them anywhere NOR lose the fields.
|
||||
expect(alpha.conditionalRefs).toEqual([]);
|
||||
expect(alpha.conditionalBytes).toBe(0);
|
||||
expect(alpha.transitiveRefs).toEqual([]);
|
||||
expect(alpha.transitiveBytes).toBe(0);
|
||||
expect(alpha.lazy).toEqual([]);
|
||||
expect(alpha.orphans).toEqual([]);
|
||||
expect(alpha.fastPath).toBeNull();
|
||||
expect(alpha.routeCeiling).toBeNull();
|
||||
// With those tiers stripped, per-invocation == eager.
|
||||
expect(alpha.perInvocationBytes).toBe(alpha.eagerBytes);
|
||||
expect(alpha.perInvocationTokens).toBe(alpha.eagerTokens);
|
||||
});
|
||||
});
|
||||
|
||||
describe("upstream fix a: root-as-container + walkMd exclusions", () => {
|
||||
it("a root with its own SKILL.md is billed AND walked into (router + children)", () => {
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-root-"));
|
||||
fs.writeFileSync(path.join(tmp, "SKILL.md"), "---\nname: router\ndescription: r\n---\n# Router\n");
|
||||
fs.mkdirSync(path.join(tmp, "qa"));
|
||||
fs.writeFileSync(path.join(tmp, "qa", "SKILL.md"), "---\nname: qa\ndescription: q\n---\n# QA\n");
|
||||
const dirs = findSkillDirs(tmp).map((d) => path.relative(fs.realpathSync(tmp), fs.realpathSync(d)) || ".");
|
||||
expect(dirs.sort()).toEqual([".", "qa"]);
|
||||
// A NON-root skill dir is still a leaf: nothing nested under qa/ counts.
|
||||
fs.mkdirSync(path.join(tmp, "qa", "nested"));
|
||||
fs.writeFileSync(path.join(tmp, "qa", "nested", "SKILL.md"), "# not a skill\n");
|
||||
expect(findSkillDirs(tmp).length).toBe(2);
|
||||
fs.rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("walkMd skips node_modules and dot-directories", () => {
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-walk-"));
|
||||
fs.writeFileSync(path.join(tmp, "real.md"), "x");
|
||||
fs.mkdirSync(path.join(tmp, "node_modules", "pkg"), { recursive: true });
|
||||
fs.writeFileSync(path.join(tmp, "node_modules", "pkg", "README.md"), "y".repeat(5000));
|
||||
fs.mkdirSync(path.join(tmp, ".git"), { recursive: true });
|
||||
fs.writeFileSync(path.join(tmp, ".git", "notes.md"), "z");
|
||||
const files = walkMd(tmp).map((f) => path.basename(f));
|
||||
expect(files).toEqual(["real.md"]);
|
||||
fs.rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
});
|
||||
|
||||
describe("upstream fix b: installed-tree layout (repo-checkout subdir skip)", () => {
|
||||
it("skips a subdir that is its own repo checkout (gstack/ inside ~/.claude/skills)", () => {
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-install-"));
|
||||
// Flat installed skill dirs.
|
||||
for (const name of ["qa", "ship"]) {
|
||||
fs.mkdirSync(path.join(tmp, name));
|
||||
fs.writeFileSync(path.join(tmp, name, "SKILL.md"), `---\nname: ${name}\ndescription: d\n---\n# ${name}\n`);
|
||||
}
|
||||
// A full repo checkout dropped into the tree: has .git and its own router
|
||||
// SKILL.md plus nested skill sources. None of it is an installed skill.
|
||||
fs.mkdirSync(path.join(tmp, "gstack", ".git"), { recursive: true });
|
||||
fs.writeFileSync(path.join(tmp, "gstack", "SKILL.md"), "---\nname: _router\ndescription: d\n---\n# router\n");
|
||||
fs.mkdirSync(path.join(tmp, "gstack", "review"));
|
||||
fs.writeFileSync(path.join(tmp, "gstack", "review", "SKILL.md"), "---\nname: review\ndescription: d\n---\n# r\n");
|
||||
|
||||
const names = buildBill(tmp).skills.map((s) => s.name).sort();
|
||||
expect(names).toEqual(["qa", "ship"]);
|
||||
fs.rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("follows a directory symlink to a sibling skill (connect-chrome shape)", () => {
|
||||
if (process.platform === "win32") return;
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-symlink-"));
|
||||
fs.mkdirSync(path.join(tmp, "open-browser"));
|
||||
fs.writeFileSync(path.join(tmp, "open-browser", "SKILL.md"), "---\nname: ob\ndescription: d\n---\n# ob\n");
|
||||
fs.symlinkSync(path.join(tmp, "open-browser"), path.join(tmp, "connect-chrome"));
|
||||
const names = buildBill(tmp).skills.map((s) => s.name).sort();
|
||||
// Both entries are real scanner load, so both are billed.
|
||||
expect(names).toEqual(["connect-chrome", "open-browser"]);
|
||||
fs.rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("prefers ./skills, then .agents/skills, then .claude/skills, project before user", async () => {
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-detect-"));
|
||||
const home = path.join(tmp, "home");
|
||||
const proj = path.join(tmp, "proj");
|
||||
const plant = (dir: string, name: string) => {
|
||||
fs.mkdirSync(path.join(dir, name), { recursive: true });
|
||||
fs.cpSync(TREE_A, path.join(dir, name), { recursive: true });
|
||||
return path.join(dir, name);
|
||||
};
|
||||
const homeClaude = plant(home, path.join(".claude", "skills"));
|
||||
const homeAgents = plant(home, path.join(".agents", "skills"));
|
||||
const projClaude = plant(proj, path.join(".claude", "skills"));
|
||||
const projAgents = plant(proj, path.join(".agents", "skills"));
|
||||
|
||||
const run = async () => {
|
||||
const out = capture();
|
||||
const code = await contextBillMain([], { cwd: proj, homeDir: home, stdout: out.stream, stderr: out.stream });
|
||||
expect(code).toBe(0);
|
||||
return out.text().split("\n")[0];
|
||||
};
|
||||
|
||||
expect(await run()).toContain(projAgents);
|
||||
fs.rmSync(projAgents, { recursive: true, force: true });
|
||||
expect(await run()).toContain(projClaude);
|
||||
fs.rmSync(projClaude, { recursive: true, force: true });
|
||||
expect(await run()).toContain(homeAgents);
|
||||
fs.rmSync(homeAgents, { recursive: true, force: true });
|
||||
expect(await run()).toContain(homeClaude);
|
||||
fs.rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("names every candidate it tried when no tree exists", async () => {
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-none-"));
|
||||
const out = capture();
|
||||
const code = await contextBillMain([], { cwd: tmp, homeDir: tmp, stdout: out.stream, stderr: out.stream });
|
||||
expect(code).toBe(2);
|
||||
expect(out.text()).toContain(path.join(tmp, ".agents", "skills"));
|
||||
expect(out.text()).toContain(path.join(tmp, ".claude", "skills"));
|
||||
fs.rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
});
|
||||
|
||||
describe("token estimate calibration", () => {
|
||||
it("uses the corpus-wide divisor for path-less callers, rounded", () => {
|
||||
expect(estimateTokens(3900)).toBe(1000);
|
||||
expect(estimateTokens(TOKEN_DIVISOR * 2)).toBe(2);
|
||||
});
|
||||
|
||||
it("charges each content class its own measured divisor", () => {
|
||||
expect(contentClass("/x/plan/SKILL.md")).toBe("skillmd");
|
||||
expect(contentClass("/x/plan/references/legacy/office.md")).toBe("legacy");
|
||||
expect(contentClass("/x/plan/references/RUNTIME.md")).toBe("reference");
|
||||
expect(contentClass("/x/plan/references/artifacts/qa/t.md")).toBe("artifact");
|
||||
expect(contentClass("/x/plan/SKILL.md#frontmatter")).toBe("frontmatter");
|
||||
expect(TOKEN_DIVISORS.legacy).toBeLessThan(TOKEN_DIVISORS.skillmd);
|
||||
const alpha = buildBill(TREE_A).skills.find((s) => s.name === "alpha")!;
|
||||
expect(alpha.skillMdTokens).toBeCloseTo(alpha.skillMdBytes / TOKEN_DIVISORS.skillmd, 6);
|
||||
expect(alpha.forcedRefs[0].tokens).toBeCloseTo(alpha.forcedRefs[0].bytes / TOKEN_DIVISORS.reference, 6);
|
||||
});
|
||||
|
||||
it("never claims more precision than it has: no divisor is a measurement", () => {
|
||||
const bill = buildBill(TREE_A);
|
||||
expect(bill.tokenEstimateErrorPct).toBeGreaterThan(0);
|
||||
expect(bill.tokenSource).toContain("estimate");
|
||||
expect(bill.calibration).toBeUndefined();
|
||||
});
|
||||
|
||||
it("names the measured error band rather than a vague 'estimates'", () => {
|
||||
const text = tokenDisclaimer(buildBill(TREE_A));
|
||||
expect(text).toContain("ESTIMATES");
|
||||
expect(text).toMatch(/worst single file 40%/);
|
||||
expect(text).toContain("--exact");
|
||||
expect(text).toContain("Bytes are always exact");
|
||||
const exactText = tokenDisclaimer({
|
||||
tokenEstimateErrorPct: 0,
|
||||
tokenSource: "count_tokens (claude-opus-4-5)",
|
||||
});
|
||||
expect(exactText).toContain("measured with count_tokens");
|
||||
expect(exactText).not.toContain("ESTIMATES");
|
||||
});
|
||||
});
|
||||
|
||||
describe("rendering", () => {
|
||||
it("text output carries the live ledgers plus flags", () => {
|
||||
const text = renderBill(buildBill(TREE_A));
|
||||
expect(text).toContain("ALWAYS-ON (every session): 2 skills");
|
||||
expect(text).toContain("EAGER (per invocation)");
|
||||
expect(text).toContain("frontmatter key(s) the router never reads: x-dead-key");
|
||||
expect(text).toContain("foreign-host file in scanner scope: agents.md");
|
||||
expect(text).toContain("TOTAL on disk:");
|
||||
expect(text).toContain("Token source: estimate");
|
||||
expect(text).toContain("Token counts are ESTIMATES");
|
||||
});
|
||||
|
||||
it("states the always-on row's exclusion of the host's per-skill wrapper", () => {
|
||||
const text = renderBill(buildBill(TREE_A));
|
||||
const alwaysOn = text.slice(text.indexOf("ALWAYS-ON"), text.indexOf("EAGER ("));
|
||||
expect(alwaysOn).toContain("excludes the host's per-skill available_skills XML wrapper");
|
||||
});
|
||||
|
||||
it("--json shape is stable (stripped tiers keep their fields)", async () => {
|
||||
const out = capture();
|
||||
const code = await contextBillMain([TREE_A, "--json"], { stdout: out.stream, stderr: out.stream });
|
||||
expect(code).toBe(0);
|
||||
const bill = JSON.parse(out.text());
|
||||
expect(bill.tokenEstimate).toEqual(TOKEN_DIVISORS);
|
||||
expect(bill.tokenEstimateErrorPct).toBe(40);
|
||||
expect(Object.keys(bill.totals).sort()).toEqual([
|
||||
"alwaysOnBytes",
|
||||
"alwaysOnTokens",
|
||||
"eagerBytesBySkill",
|
||||
"eagerTokensBySkill",
|
||||
"perInvocationBytesBySkill",
|
||||
"perInvocationTokensBySkill",
|
||||
"skillCount",
|
||||
"totalMdBytes",
|
||||
"totalMdTokens",
|
||||
]);
|
||||
const skill = bill.skills[0];
|
||||
for (const key of ["name", "frontmatterBytes", "frontmatterTokens", "deadKeys", "skillMdBytes", "skillMdTokens", "forcedRefs", "eagerBytes", "eagerTokens", "fastPath", "conditionalRefs", "conditionalBytes", "conditionalTokens", "transitiveRefs", "transitiveBytes", "transitiveTokens", "perInvocationBytes", "perInvocationTokens", "routeCeiling", "lazy", "orphans", "foreignFiles", "totalMdTokens"]) {
|
||||
expect(skill).toHaveProperty(key);
|
||||
}
|
||||
for (const r of skill.forcedRefs) expect(typeof r.tokens).toBe("number");
|
||||
});
|
||||
});
|
||||
|
||||
describe("--diff", () => {
|
||||
it("reports exactly the grown row and exits 2", async () => {
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-"));
|
||||
const treeB = path.join(tmp, "tree-b");
|
||||
fs.cpSync(TREE_A, treeB, { recursive: true });
|
||||
fs.appendFileSync(path.join(treeB, "alpha", "SKILL.md"), "x".repeat(43000));
|
||||
|
||||
const diff = diffBills(buildBill(TREE_A), buildBill(treeB));
|
||||
expect(diff.rows).toHaveLength(1);
|
||||
expect(diff.rows[0]).toMatchObject({ ledger: "eager", label: "alpha", delta: 43000 });
|
||||
expect(diff.grew).toBe(true);
|
||||
|
||||
const out = capture();
|
||||
const code = await contextBillMain(["--diff", TREE_A, treeB], { stdout: out.stream, stderr: out.stream });
|
||||
expect(code).toBe(2);
|
||||
expect(out.text()).toContain("eager");
|
||||
expect(out.text()).toContain("GREW");
|
||||
fs.rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("identical trees diff clean and exit 0", async () => {
|
||||
const out = capture();
|
||||
const code = await contextBillMain(["--diff", TREE_A, TREE_A], { stdout: out.stream, stderr: out.stream });
|
||||
expect(code).toBe(0);
|
||||
expect(out.text()).toContain("No context-cost changes");
|
||||
});
|
||||
});
|
||||
|
||||
describe("--budget", () => {
|
||||
const alphaEagerTok = Math.round(buildBill(TREE_A).skills.find((s) => s.name === "alpha")!.eagerTokens);
|
||||
|
||||
it("exits 0 under budget, 2 over budget with offending files listed", async () => {
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-budget-"));
|
||||
const okBudget = path.join(tmp, "ok.json");
|
||||
const tightBudget = path.join(tmp, "tight.json");
|
||||
fs.writeFileSync(okBudget, JSON.stringify({ alwaysOnTotal: 10000, eagerPerInvocation: { alpha: alphaEagerTok } }));
|
||||
fs.writeFileSync(tightBudget, JSON.stringify({ eagerPerInvocation: { alpha: alphaEagerTok - 1 } }));
|
||||
|
||||
const ok = capture();
|
||||
expect(await contextBillMain([TREE_A, "--budget", okBudget], { stdout: ok.stream, stderr: ok.stream })).toBe(0);
|
||||
expect(ok.text()).toContain("Within budget");
|
||||
|
||||
const over = capture();
|
||||
expect(await contextBillMain([TREE_A, "--budget", tightBudget], { stdout: over.stream, stderr: over.stream })).toBe(2);
|
||||
expect(over.text()).toContain("OVER BUDGET: eagerPerInvocation.alpha");
|
||||
expect(over.text()).toContain("alpha/SKILL.md");
|
||||
expect(over.text()).toContain("references/CORE.md");
|
||||
fs.rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("perInvocation and routeCeiling keys stay accepted (they gate the eager figure while tiers are stripped)", () => {
|
||||
const bill = buildBill(TREE_A);
|
||||
expect(checkBudget(bill, { perInvocation: { alpha: alphaEagerTok } })).toEqual([]);
|
||||
expect(checkBudget(bill, { perInvocation: { alpha: alphaEagerTok - 1 } })).toHaveLength(1);
|
||||
expect(checkBudget(bill, { routeCeiling: { alpha: alphaEagerTok } })).toEqual([]);
|
||||
});
|
||||
|
||||
it("checkBudget flags a budgeted skill missing from the tree", () => {
|
||||
const violations = checkBudget(buildBill(TREE_A), { eagerPerInvocation: { ghost: 100 } });
|
||||
expect(violations).toHaveLength(1);
|
||||
expect(violations[0].ceiling).toBe("eagerPerInvocation.ghost");
|
||||
});
|
||||
});
|
||||
|
||||
describe("--exact (opt-in measurement; offline here via an injected fetch)", () => {
|
||||
let egressHome: string;
|
||||
beforeAll(() => {
|
||||
egressHome = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-egress-"));
|
||||
});
|
||||
afterAll(() => {
|
||||
try { fs.chmodSync(path.join(egressHome, "security"), 0o700); } catch {}
|
||||
fs.rmSync(egressHome, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
const ENVELOPE = 7;
|
||||
/** Deterministic stand-in for count_tokens: 1 token per 3 chars, plus envelope. */
|
||||
function fakeFetch(calls: { body: string }[] = []) {
|
||||
return (async (_url: string, init: { body: string }) => {
|
||||
calls.push({ body: init.body });
|
||||
const { messages } = JSON.parse(init.body);
|
||||
const text: string = messages[0].content;
|
||||
return {
|
||||
ok: true,
|
||||
json: async () => ({ input_tokens: Math.ceil(text.length / 3) + ENVELOPE }),
|
||||
};
|
||||
}) as never;
|
||||
}
|
||||
|
||||
it("writes the egress receipt BEFORE the first count_tokens POST", async () => {
|
||||
const home = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-receipt-"));
|
||||
let receiptsAtFirstPost = -1;
|
||||
const fetchImpl = (async (_url: string, init: { body: string }) => {
|
||||
if (receiptsAtFirstPost === -1) receiptsAtFirstPost = listReceipts(home).length;
|
||||
const { messages } = JSON.parse(init.body);
|
||||
return { ok: true, json: async () => ({ input_tokens: Math.ceil(messages[0].content.length / 3) + ENVELOPE }) };
|
||||
}) as never;
|
||||
await measureExactTokens(TREE_A, { model: "m", apiKey: "sk-test", fetchImpl, egressHome: home });
|
||||
expect(receiptsAtFirstPost).toBe(1); // the receipt existed before any POST
|
||||
const receipts = listReceipts(home);
|
||||
expect(receipts).toHaveLength(1);
|
||||
expect(receipts[0].sink).toBe("context-bill-exact");
|
||||
expect(receipts[0].host).toBe("api.anthropic.com");
|
||||
expect(receipts[0].sha256).toBeNull();
|
||||
expect(receipts[0].bytes).toBeGreaterThan(0);
|
||||
fs.rmSync(home, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("fail-open: an unwritable ledger degrades --exact to the offline estimate, sending nothing", async () => {
|
||||
if (process.platform === "win32" || process.getuid?.() === 0) return;
|
||||
const home = fs.mkdtempSync(path.join(os.tmpdir(), "context-bill-refuse-"));
|
||||
fs.mkdirSync(path.join(home, "security"), { recursive: true, mode: 0o500 });
|
||||
let called = false;
|
||||
const stdout = capture();
|
||||
const stderr = capture();
|
||||
const code = await contextBillMain([TREE_A, "--exact", "--json"], {
|
||||
stdout: stdout.stream,
|
||||
stderr: stderr.stream,
|
||||
apiKey: "sk-test",
|
||||
fetchImpl: (() => ((called = true), Promise.reject(new Error("must not send")))) as never,
|
||||
egressHome: home,
|
||||
});
|
||||
expect(code).toBe(0); // the run still answers, with the estimate
|
||||
expect(called).toBe(false); // NOTHING was sent unrecorded
|
||||
expect(stderr.text()).toContain("exact_egress_receipt_failed");
|
||||
expect(stderr.text()).toContain("Falling back to the offline estimate");
|
||||
expect(JSON.parse(stdout.text()).tokenSource).toContain("estimate");
|
||||
fs.chmodSync(path.join(home, "security"), 0o700);
|
||||
fs.rmSync(home, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("subtracts the request envelope so small files are not overcharged", async () => {
|
||||
const calls: { body: string }[] = [];
|
||||
const measured = await measureExactTokens(TREE_A, {
|
||||
model: "test-model",
|
||||
apiKey: "sk-test",
|
||||
fetchImpl: fakeFetch(calls),
|
||||
egressHome,
|
||||
});
|
||||
expect(measured.tokenSource).toBe("count_tokens (test-model)");
|
||||
const core = path.join(TREE_A, "alpha", "references", "CORE.md");
|
||||
const text = fs.readFileSync(core, "utf8");
|
||||
expect(measured.tokensOf(core, text.length)).toBe(Math.ceil(text.length / 3));
|
||||
// One probe call for the envelope, then one per measured text.
|
||||
expect(calls.length).toBe(measured.measuredFiles + 1);
|
||||
});
|
||||
|
||||
it("measures the frontmatter block, not the whole SKILL.md, for the always-on row", async () => {
|
||||
const measured = await measureExactTokens(TREE_A, {
|
||||
model: "m", apiKey: "sk-test", fetchImpl: fakeFetch(), egressHome,
|
||||
});
|
||||
const skillMd = path.join(TREE_A, "alpha", "SKILL.md");
|
||||
const fmTokens = measured.tokensOf(`${skillMd}#frontmatter`, 0);
|
||||
const bodyTokens = measured.tokensOf(skillMd, 0);
|
||||
expect(fmTokens).toBeGreaterThan(0);
|
||||
expect(fmTokens).toBeLessThan(bodyTokens);
|
||||
});
|
||||
|
||||
it("exact tokens replace the estimate everywhere the bill prices content", async () => {
|
||||
const measured = await measureExactTokens(TREE_A, {
|
||||
model: "m", apiKey: "sk-test", fetchImpl: fakeFetch(), egressHome,
|
||||
});
|
||||
const bill = buildBill(TREE_A, { tokensOf: measured.tokensOf, tokenSource: measured.tokenSource });
|
||||
const alpha = bill.skills.find((s) => s.name === "alpha")!;
|
||||
expect(alpha.eagerTokens).toBeGreaterThanOrEqual(alpha.eagerBytes / 3);
|
||||
expect(alpha.eagerTokens).toBeLessThan(alpha.eagerBytes / 3 + 4);
|
||||
expect(alpha.eagerTokens / (alpha.eagerBytes / TOKEN_DIVISOR)).toBeGreaterThan(1.2);
|
||||
expect(alpha.eagerTokens).toBe(alpha.skillMdTokens + alpha.forcedRefs.reduce((n, r) => n + r.tokens, 0));
|
||||
expect(bill.tokenEstimateErrorPct).toBe(0);
|
||||
expect(renderBill(bill)).toContain("Token counts measured with count_tokens");
|
||||
expect(renderBill(bill)).not.toMatch(/\(~\d/);
|
||||
});
|
||||
|
||||
it("measures against a relative root too, instead of silently estimating", async () => {
|
||||
const relative = path.relative(process.cwd(), TREE_A);
|
||||
const measured = await measureExactTokens(relative, {
|
||||
model: "m", apiKey: "sk-test", fetchImpl: fakeFetch(), egressHome,
|
||||
});
|
||||
const bill = buildBill(relative, { tokensOf: measured.tokensOf, tokenSource: measured.tokenSource });
|
||||
for (const s of bill.skills) {
|
||||
expect(s.totalMdBytes / s.totalMdTokens).toBeLessThan(3.2);
|
||||
}
|
||||
expect(measured.missedKeys.size).toBe(0);
|
||||
});
|
||||
|
||||
it("counts anything it could not measure instead of passing it off as measured", async () => {
|
||||
const measured = await measureExactTokens(TREE_A, {
|
||||
model: "m", apiKey: "sk-test", fetchImpl: fakeFetch(), egressHome,
|
||||
});
|
||||
expect(measured.missedKeys.size).toBe(0);
|
||||
const ghost = path.join(TREE_A, "alpha", "nope.md");
|
||||
expect(measured.tokensOf(ghost, 4150)).toBeGreaterThan(0);
|
||||
expect(measured.missedKeys.has(ghost)).toBe(true);
|
||||
});
|
||||
|
||||
it("grades its own estimate: calibrationTable reports the residual per file", async () => {
|
||||
const measured = await measureExactTokens(TREE_A, {
|
||||
model: "m", apiKey: "sk-test", fetchImpl: fakeFetch(), egressHome,
|
||||
});
|
||||
const table = calibrationTable(measured.counts, TREE_A);
|
||||
expect(table.rows.length).toBeGreaterThan(0);
|
||||
for (const row of table.rows) {
|
||||
expect(row).toHaveProperty("contentClass");
|
||||
expect(row.estimatedTokens).toBeGreaterThan(0);
|
||||
expect(row.tokens).toBeGreaterThan(0);
|
||||
expect(row.errorPct).toBeCloseTo(((row.estimatedTokens - row.tokens) / row.tokens) * 100, 1);
|
||||
}
|
||||
expect(typeof table.worstErrorPct).toBe("number");
|
||||
expect(typeof table.biasPct).toBe("number");
|
||||
});
|
||||
|
||||
it("refuses to go to the network without a key, and says nothing was sent", async () => {
|
||||
let called = false;
|
||||
await expect(
|
||||
measureExactTokens(TREE_A, {
|
||||
model: "m",
|
||||
apiKey: "",
|
||||
fetchImpl: (() => ((called = true), Promise.reject(new Error("should not run")))) as never,
|
||||
egressHome,
|
||||
}),
|
||||
).rejects.toMatchObject({ code: "exact_missing_api_key" });
|
||||
expect(called).toBe(false);
|
||||
});
|
||||
|
||||
it("maps HTTP failures to typed codes", async () => {
|
||||
const status = (code: number) => (async () => ({ ok: false, status: code, text: async () => "nope" })) as never;
|
||||
for (const [code, expected] of [
|
||||
[401, "exact_auth_rejected"],
|
||||
[403, "exact_auth_rejected"],
|
||||
[500, "exact_request_failed"],
|
||||
] as const) {
|
||||
await expect(
|
||||
measureExactTokens(TREE_A, { model: "m", apiKey: "k", fetchImpl: status(code), egressHome }),
|
||||
).rejects.toMatchObject({ code: expected });
|
||||
}
|
||||
await expect(
|
||||
measureExactTokens(TREE_A, {
|
||||
model: "m", apiKey: "k",
|
||||
fetchImpl: (() => Promise.reject(new Error("offline"))) as never,
|
||||
egressHome,
|
||||
}),
|
||||
).rejects.toBeInstanceOf(ExactModeError);
|
||||
});
|
||||
|
||||
it("offline by default: no --exact means no network call at all", async () => {
|
||||
const out = capture();
|
||||
let called = false;
|
||||
const code = await contextBillMain([TREE_A, "--json"], {
|
||||
stdout: out.stream,
|
||||
stderr: out.stream,
|
||||
fetchImpl: (() => ((called = true), Promise.reject(new Error("no")))) as never,
|
||||
apiKey: "sk-test",
|
||||
egressHome,
|
||||
});
|
||||
expect(code).toBe(0);
|
||||
expect(called).toBe(false);
|
||||
expect(JSON.parse(out.text()).tokenSource).toContain("estimate");
|
||||
});
|
||||
|
||||
it("discloses what --exact sends before sending it", async () => {
|
||||
const stdout = capture();
|
||||
const stderr = capture();
|
||||
const code = await contextBillMain([TREE_A, "--exact", "--json"], {
|
||||
stdout: stdout.stream,
|
||||
stderr: stderr.stream,
|
||||
apiKey: "sk-test",
|
||||
fetchImpl: fakeFetch(),
|
||||
egressHome,
|
||||
});
|
||||
expect(code).toBe(0);
|
||||
expect(stderr.text()).toContain("api.anthropic.com");
|
||||
expect(stderr.text()).toMatch(/sending the content of \d+ \.md file\(s\)/);
|
||||
const bill = JSON.parse(stdout.text());
|
||||
expect(bill.tokenSource).toContain("count_tokens");
|
||||
expect(bill.calibration.rows.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("degrades to the estimate when exact mode is unavailable, naming the code", async () => {
|
||||
const stdout = capture();
|
||||
const stderr = capture();
|
||||
const code = await contextBillMain([TREE_A, "--exact", "--json"], {
|
||||
stdout: stdout.stream,
|
||||
stderr: stderr.stream,
|
||||
apiKey: "",
|
||||
fetchImpl: (() => Promise.reject(new Error("should not run"))) as never,
|
||||
egressHome,
|
||||
});
|
||||
expect(code).toBe(0);
|
||||
expect(stderr.text()).toContain("exact_missing_api_key");
|
||||
expect(JSON.parse(stdout.text()).tokenSource).toContain("estimate");
|
||||
});
|
||||
|
||||
it("--help discloses --exact's egress and recalibration", async () => {
|
||||
const out = capture();
|
||||
expect(await contextBillMain(["--help"], { stdout: out.stream, stderr: out.stream })).toBe(0);
|
||||
expect(out.text()).toContain("api.anthropic.com");
|
||||
expect(out.text()).toContain("egress receipt");
|
||||
expect(out.text()).toContain("recalibrates");
|
||||
});
|
||||
});
|
||||
|
||||
describe("ground truth against THIS repo (skill-census parity)", () => {
|
||||
const census = skillCensus(ROOT);
|
||||
const dirs = findSkillDirs(ROOT);
|
||||
const rels = dirs.map((d) => path.relative(ROOT, d) || ".");
|
||||
|
||||
it("the walker reaches every physical SKILL.md the census counts", () => {
|
||||
// physicalSkillFiles is the walker's expectation: every depth-1 skill dir
|
||||
// (symlinked dirs included) plus the root router.
|
||||
for (const rel of census.physicalSkillFiles) {
|
||||
const dirRel = rel === "SKILL.md" ? "." : path.dirname(rel);
|
||||
expect(rels, `walker missed ${rel}`).toContain(dirRel);
|
||||
}
|
||||
});
|
||||
|
||||
it("the root router SKILL.md is billed (fix a live on the real repo)", () => {
|
||||
expect(census.physicalSkillFiles).toContain("SKILL.md"); // guard the guard
|
||||
expect(rels).toContain(".");
|
||||
});
|
||||
|
||||
it("a real skill's frontmatter bytes match an independent computation", () => {
|
||||
const qaDir = dirs.find((d) => path.relative(ROOT, d) === "qa")!;
|
||||
expect(qaDir).toBeTruthy();
|
||||
const bill = buildBill(qaDir);
|
||||
expect(bill.skills).toHaveLength(1);
|
||||
expect(bill.skills[0].frontmatterBytes).toBe(frontmatterBytes(path.join(qaDir, "SKILL.md")));
|
||||
expect(bill.skills[0].frontmatterBytes).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("skill count is at least the census count (deeper trees may add more, never fewer)", () => {
|
||||
expect(dirs.length).toBeGreaterThanOrEqual(census.physicalSkillFiles.length);
|
||||
});
|
||||
|
||||
it("no skill dir is billed twice", () => {
|
||||
expect(new Set(rels).size).toBe(rels.length);
|
||||
});
|
||||
});
|
||||
|
||||
describe("CLI plumbing", () => {
|
||||
it("unknown flag and missing tree are usage errors (exit 2)", async () => {
|
||||
const a = capture();
|
||||
expect(await contextBillMain(["--nope"], { stdout: a.stream, stderr: a.stream })).toBe(2);
|
||||
const b = capture();
|
||||
expect(await contextBillMain(["--diff", TREE_A], { stdout: b.stream, stderr: b.stream })).toBe(2);
|
||||
const c = capture();
|
||||
expect(await contextBillMain([path.join(os.tmpdir(), "does-not-exist-xyz")], { stdout: c.stream, stderr: c.stream })).toBe(1);
|
||||
});
|
||||
|
||||
it("bin/gstack-context-bill runs standalone", () => {
|
||||
const result = Bun.spawnSync([path.join(ROOT, "bin", "gstack-context-bill"), TREE_A]);
|
||||
expect(result.exitCode).toBe(0);
|
||||
expect(result.stdout.toString()).toContain("ALWAYS-ON");
|
||||
expect(result.stdout.toString()).toContain("EAGER");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,185 @@
|
||||
/**
|
||||
* gstack-egress-lib.sh — shared shell receipt helpers, tested end-to-end
|
||||
* against a local listener. Free tier, loopback only.
|
||||
*
|
||||
* Pins the shell-sink contract:
|
||||
* - receipt sha256 == sha256 of the exact bytes the listener received
|
||||
* (same file is hashed and handed to curl via --data-binary @file)
|
||||
* - fail-closed refusal never touches the network and its stderr carries
|
||||
* problem + cause + fix in plain language
|
||||
* - fail-open warns on stderr and proceeds unrecorded
|
||||
* - payload temp files are consumed immediately (no EXIT traps)
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeEach, afterEach, afterAll } from 'bun:test';
|
||||
import * as fs from 'fs';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import { listReceipts, sha256Hex } from '../lib/egress-receipt';
|
||||
|
||||
const ROOT = path.resolve(new URL(import.meta.url).pathname, '..', '..');
|
||||
const LIB = path.join(ROOT, 'bin', 'gstack-egress-lib.sh');
|
||||
|
||||
const received: string[] = [];
|
||||
const server = Bun.serve({
|
||||
port: 0,
|
||||
fetch: async (req) => {
|
||||
received.push(await req.text());
|
||||
return new Response('ok');
|
||||
},
|
||||
});
|
||||
const URL_BASE = `http://127.0.0.1:${server.port}`;
|
||||
|
||||
afterAll(() => {
|
||||
server.stop(true);
|
||||
});
|
||||
|
||||
let home: string;
|
||||
|
||||
beforeEach(() => {
|
||||
home = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-egress-lib-'));
|
||||
received.length = 0;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
try { fs.chmodSync(path.join(home, 'security'), 0o700); } catch {}
|
||||
fs.rmSync(home, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// Async spawn: spawnSync would block Bun's event loop, deadlocking the
|
||||
// in-process listener the script curls against.
|
||||
async function runBash(script: string) {
|
||||
const proc = Bun.spawn(['bash', '-c', script], {
|
||||
env: { ...process.env, GSTACK_HOME: home },
|
||||
stdout: 'pipe',
|
||||
stderr: 'pipe',
|
||||
});
|
||||
const [stdout, stderr, status] = await Promise.all([
|
||||
new Response(proc.stdout).text(),
|
||||
new Response(proc.stderr).text(),
|
||||
proc.exited,
|
||||
]);
|
||||
return { status, stdout, stderr };
|
||||
}
|
||||
|
||||
describe('_receipted_curl', () => {
|
||||
test('receipt sha256 equals sha256 of the body the listener received; payload file consumed', async () => {
|
||||
const body = '[{"event":"lib-test","v":1}]';
|
||||
const result = await runBash(`
|
||||
set -uo pipefail
|
||||
. "${LIB}"
|
||||
payload="$(mktemp)"
|
||||
printf '%s' '${body}' > "$payload"
|
||||
echo "PAYLOAD_PATH=$payload"
|
||||
_receipted_curl closed lib-test 127.0.0.1:${server.port} test-events "telemetry=community" "$payload" \\
|
||||
curl -s --max-time 5 -X POST "${URL_BASE}/ingest"
|
||||
echo "CURL_EXIT=$?"
|
||||
[ -e "$payload" ] && echo "PAYLOAD_STILL_EXISTS" || echo "PAYLOAD_GONE"
|
||||
`);
|
||||
expect(result.status).toBe(0);
|
||||
expect(result.stdout).toContain('CURL_EXIT=0');
|
||||
expect(result.stdout).toContain('PAYLOAD_GONE');
|
||||
expect(received.length).toBe(1);
|
||||
expect(received[0]).toBe(body);
|
||||
const receipts = listReceipts(home);
|
||||
expect(receipts.length).toBe(1);
|
||||
expect(receipts[0].sha256).toBe(sha256Hex(received[0]));
|
||||
expect(receipts[0].bytes).toBe(Buffer.byteLength(body));
|
||||
expect(receipts[0].sink).toBe('lib-test');
|
||||
expect(receipts[0].status).toBe('exit:0'); // best-effort outcome joined
|
||||
});
|
||||
|
||||
test('fail-closed refusal never hits the network; stderr is problem + cause + fix', async () => {
|
||||
if (process.platform === 'win32' || process.getuid?.() === 0) return;
|
||||
fs.mkdirSync(path.join(home, 'security'), { recursive: true, mode: 0o500 });
|
||||
const result = await runBash(`
|
||||
set -uo pipefail
|
||||
. "${LIB}"
|
||||
payload="$(mktemp)"
|
||||
printf '%s' 'secret payload' > "$payload"
|
||||
_receipted_curl closed lib-test 127.0.0.1:${server.port} test-events "telemetry=community" "$payload" \\
|
||||
curl -s --max-time 5 -X POST "${URL_BASE}/ingest"
|
||||
status=$?
|
||||
[ -e "$payload" ] && echo "PAYLOAD_STILL_EXISTS" || echo "PAYLOAD_GONE"
|
||||
exit $status
|
||||
`);
|
||||
expect(result.status).toBe(3);
|
||||
expect(received.length).toBe(0); // the send was refused, not attempted
|
||||
expect(result.stdout).toContain('PAYLOAD_GONE'); // cleaned up on refusal too
|
||||
// Problem: what did not happen.
|
||||
expect(result.stderr).toContain('lib-test NOT sent');
|
||||
// Cause: the bridge's typed error is quoted.
|
||||
expect(result.stderr).toContain('EGRESS_RECEIPT_FAILED');
|
||||
// Fix: an actionable command.
|
||||
expect(result.stderr).toContain('Fix: chmod -R u+w');
|
||||
// What this is: the ledger explained.
|
||||
expect(result.stderr).toContain('ATTEMPTS to send off-machine');
|
||||
expect(result.stderr).toContain('gstack-egress');
|
||||
});
|
||||
|
||||
test('fail-open warns and proceeds when the receipt cannot be written', async () => {
|
||||
if (process.platform === 'win32' || process.getuid?.() === 0) return;
|
||||
fs.mkdirSync(path.join(home, 'security'), { recursive: true, mode: 0o500 });
|
||||
const result = await runBash(`
|
||||
set -uo pipefail
|
||||
. "${LIB}"
|
||||
_receipted_curl open lib-open 127.0.0.1:${server.port} version-fetch "user ran --force" --no-payload \\
|
||||
curl -s --max-time 5 "${URL_BASE}/VERSION"
|
||||
`);
|
||||
expect(result.status).toBe(0);
|
||||
expect(received.length).toBe(1); // the send proceeded unrecorded
|
||||
expect(result.stderr).toContain('sending anyway');
|
||||
expect(result.stderr).toContain('lib-open');
|
||||
});
|
||||
|
||||
test('--no-payload records sha256:null and does not append --data-binary', async () => {
|
||||
const result = await runBash(`
|
||||
set -uo pipefail
|
||||
. "${LIB}"
|
||||
_receipted_curl closed lib-get 127.0.0.1:${server.port} version-fetch "user ran --force" --no-payload \\
|
||||
curl -s --max-time 5 "${URL_BASE}/VERSION"
|
||||
`);
|
||||
expect(result.status).toBe(0);
|
||||
expect(received.length).toBe(1);
|
||||
expect(received[0]).toBe(''); // GET, no body
|
||||
const receipts = listReceipts(home);
|
||||
expect(receipts.length).toBe(1);
|
||||
expect(receipts[0].sha256).toBeNull();
|
||||
expect(receipts[0].bytes).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('_receipted_git', () => {
|
||||
test('writes a git-class sha256:null receipt then runs the command unmodified', async () => {
|
||||
const result = await runBash(`
|
||||
set -uo pipefail
|
||||
. "${LIB}"
|
||||
_receipted_git closed brain-sync github.com curated-memory-git-push "artifacts_sync_mode!=off" \\
|
||||
bash -c 'echo GIT_RAN "$@"' _
|
||||
`);
|
||||
expect(result.status).toBe(0);
|
||||
expect(result.stdout).toContain('GIT_RAN');
|
||||
expect(result.stdout).not.toContain('--data-binary');
|
||||
const receipts = listReceipts(home);
|
||||
expect(receipts.length).toBe(1);
|
||||
expect(receipts[0].sha256).toBeNull();
|
||||
expect(receipts[0].payload_class).toBe('curated-memory-git-push');
|
||||
expect(receipts[0].status).toBe('exit:0');
|
||||
});
|
||||
|
||||
test('fail-closed git refusal returns 3 without running the command', async () => {
|
||||
if (process.platform === 'win32' || process.getuid?.() === 0) return;
|
||||
fs.mkdirSync(path.join(home, 'security'), { recursive: true, mode: 0o500 });
|
||||
const marker = path.join(os.tmpdir(), `gstack-egress-git-${process.pid}`);
|
||||
fs.rmSync(marker, { force: true });
|
||||
const result = await runBash(`
|
||||
set -uo pipefail
|
||||
. "${LIB}"
|
||||
_receipted_git closed brain-sync github.com curated-memory-git-push "artifacts_sync_mode!=off" \\
|
||||
touch "${marker}"
|
||||
`);
|
||||
expect(result.status).toBe(3);
|
||||
expect(fs.existsSync(marker)).toBe(false); // command never ran
|
||||
expect(result.stderr).toContain('brain-sync NOT sent');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,380 @@
|
||||
/**
|
||||
* Static-grep tripwire for egress-receipt wiring. Free tier — no API.
|
||||
*
|
||||
* THREAT MODEL: the egress ledger is forensic observability — it records
|
||||
* ATTEMPTED egress so accidents are auditable; it is not an exfiltration
|
||||
* control. Receipts are written before send, outcomes are best-effort, and
|
||||
* fail-open classes can send unrecorded with a warning.
|
||||
*
|
||||
* Every enumerated off-machine sink must route its send through the receipt
|
||||
* ledger (lib/egress-receipt.ts), receipt BEFORE send. A future egress call
|
||||
* site added without a receipt fails CI here instead of becoming a
|
||||
* user-filed issue. The NEW-SINK SCANNER at the bottom sweeps the whole
|
||||
* tree for outbound network ops and requires every hit to be either wired
|
||||
* or in the REASONED exemption list — there is no KNOWN_UNWIRED bucket.
|
||||
*
|
||||
* Out of scope, documented here on purpose: the preamble-generated brain
|
||||
* sync block (scripts/resolvers/preamble/generate-brain-sync-block.ts)
|
||||
* renders a `git fetch` into skill PROSE that the agent executes — it is
|
||||
* agent-executed instructions, not a gstack binary, so it is covered by the
|
||||
* skill-prose exemption below rather than a receipt.
|
||||
*
|
||||
* Pattern mirrors test/hermetic-wiring.test.ts: read source files as text,
|
||||
* assert invariants on their contents. Brittle by design — renaming a
|
||||
* helper must force the author to look here.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
|
||||
const ROOT = path.resolve(new URL(import.meta.url).pathname, '..', '..');
|
||||
|
||||
function read(rel: string): string {
|
||||
return fs.readFileSync(path.join(ROOT, rel), 'utf-8');
|
||||
}
|
||||
|
||||
function exists(rel: string): boolean {
|
||||
return fs.existsSync(path.join(ROOT, rel));
|
||||
}
|
||||
|
||||
// ── POLARITY TABLE (amendments T3/C8) ──────────────────────────────────────
|
||||
// Pinned as data: which sinks refuse the send when the receipt cannot be
|
||||
// written (fail-closed) vs warn and proceed (fail-open). Changing a sink's
|
||||
// polarity is a security decision — update this table deliberately.
|
||||
const POLARITY: Record<string, 'fail-closed' | 'fail-open'> = {
|
||||
// fail-closed: gstack state leaving the machine unrecorded is worse than
|
||||
// the operation failing.
|
||||
'brain-sync': 'fail-closed',
|
||||
'memory-ingest': 'fail-closed',
|
||||
'gbrain-sync': 'fail-closed',
|
||||
'telemetry-sync': 'fail-closed',
|
||||
'browse-tunnel (ngrok)': 'fail-closed',
|
||||
'gbrain-mcp-verify': 'fail-closed',
|
||||
'supabase-provision': 'fail-closed',
|
||||
// fail-open: user-facing operations that must not die over an audit-log
|
||||
// hiccup; they warn on stderr and proceed.
|
||||
'design-openai': 'fail-open',
|
||||
'update-check': 'fail-open',
|
||||
'security-dashboard': 'fail-open',
|
||||
'community-dashboard': 'fail-open',
|
||||
'git-class user ops (artifacts-init, brain-restore, session-update)': 'fail-open',
|
||||
'context-bill --exact': 'fail-open',
|
||||
};
|
||||
|
||||
/** TS sinks: must import the canonical helper and call writeReceipt(). */
|
||||
const MODULE_SINKS = [
|
||||
'bin/gstack-gbrain-sync.ts',
|
||||
'bin/gstack-memory-ingest.ts',
|
||||
'browse/src/server.ts',
|
||||
// Unconditional: context-bill ships in the same tree as this tripwire. A
|
||||
// missing file must fail loudly (a rename/move that drops its receipt wiring
|
||||
// is exactly what this pins), not silently soften the assertion.
|
||||
'lib/context-bill.ts',
|
||||
];
|
||||
|
||||
/** Shell sinks: must source the shared lib; every network op receipted. */
|
||||
const SHELL_SINKS = [
|
||||
'bin/gstack-telemetry-sync',
|
||||
'bin/gstack-update-check',
|
||||
'bin/gstack-brain-sync',
|
||||
'bin/gstack-gbrain-mcp-verify',
|
||||
'bin/gstack-security-dashboard',
|
||||
'bin/gstack-community-dashboard',
|
||||
'bin/gstack-gbrain-supabase-provision',
|
||||
'bin/gstack-artifacts-init',
|
||||
'bin/gstack-brain-restore',
|
||||
'bin/gstack-session-update',
|
||||
];
|
||||
|
||||
/** design files that talk to api.openai.com — all must use receiptedFetch. */
|
||||
const DESIGN_SINKS = [
|
||||
'design/src/generate.ts',
|
||||
'design/src/variants.ts',
|
||||
'design/src/iterate.ts',
|
||||
'design/src/evolve.ts',
|
||||
'design/src/check.ts',
|
||||
'design/src/diff.ts',
|
||||
'design/src/design-to-code.ts',
|
||||
'design/src/memory.ts',
|
||||
];
|
||||
|
||||
// ── NEW-SINK SCANNER exemptions ────────────────────────────────────────────
|
||||
// Every entry carries its reason. An unexplained network op anywhere in the
|
||||
// swept tree fails the scanner — add real sinks to the wired lists above,
|
||||
// not here.
|
||||
const SCANNER_EXEMPT: Record<string, string> = {
|
||||
'bin/gstack-team-init':
|
||||
'every git clone is inside an echoed instruction string (install docs); the script executes no network ops',
|
||||
'bin/gstack-gbrain-install':
|
||||
'user-invoked installer: bodyless HEAD reachability probe to github.com + clone of the public gbrain repo (user-directed install; no gstack state leaves the machine)',
|
||||
'bin/gstack-next-version':
|
||||
'fetches the user\'s own repo\'s base branch for version-claim freshness — a user-repo dev-workflow op, not gstack-state egress',
|
||||
'bin/gstack-version-bump':
|
||||
'git fetch appears only in an error-message string',
|
||||
'bin/gstack-redact-prepush':
|
||||
'git push mentions are hook documentation strings (bypass instructions)',
|
||||
'browse/src/security-classifier.ts':
|
||||
'HF model download: bodyless GET of a public classifier model (variable URL)',
|
||||
'browse/src/write-commands.ts':
|
||||
'user-directed page fetch — the browser command surface fetches what the user asked for',
|
||||
'browse/src/cli.ts':
|
||||
'health probe of the user\'s own pair-agent tunnel URL (reachability probe)',
|
||||
'browse/src/commands.ts':
|
||||
'git pull appears only in an upgrade-hint message string',
|
||||
'browse/src/cookie-picker-ui.ts':
|
||||
'served-page JS talking to its own loopback server (same-origin relative fetch)',
|
||||
'design/src/compare.ts':
|
||||
'served-page JS talking to its own loopback server (relative ./api fetch)',
|
||||
// Skill prose templates: these render agent-executed instructions (the
|
||||
// agent runs git in the USER\'S repo at the user\'s direction), they are
|
||||
// not gstack binaries. Includes the preamble-generated brain-sync block —
|
||||
// see the header.
|
||||
'scripts/resolvers':
|
||||
'skill prose templates — agent-executed instructions rendered into SKILL.md, not gstack binaries',
|
||||
};
|
||||
|
||||
function isExempt(rel: string): string | undefined {
|
||||
for (const [key, reason] of Object.entries(SCANNER_EXEMPT)) {
|
||||
if (rel === key || rel.startsWith(`${key}/`)) return reason;
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
// Receipt markers that make a nearby network op "wired".
|
||||
const RECEIPT_MARKER =
|
||||
/_receipted_(curl|git|version_fetch)\b|gstack-egress-receipt["']?\s+write\b|writeReceipt\(|receiptedFetch\(/;
|
||||
|
||||
/** Was a receipt marker present on this line or the 30 preceding lines? */
|
||||
function guarded(lines: string[], i: number): boolean {
|
||||
for (let j = i; j >= Math.max(0, i - 30); j--) {
|
||||
if (RECEIPT_MARKER.test(lines[j])) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
// git as a COMMAND followed by a remote op. Local ops (rev-parse, remote
|
||||
// get-url, add, commit, merge) never match; neither does prose like
|
||||
// "curated-memory-git-push" (hyphenated) or "'git fetch'" (quoted).
|
||||
const GIT_REMOTE_OP = /(^|[;|&`($!]|\s)git(\s+-C\s+\S+)?\s+(push|pull|fetch|clone|ls-remote)\b/;
|
||||
// git spawn-array form in TS: spawn("git", ["push", ...]).
|
||||
const GIT_SPAWN_OP = /["'`]git["'`]\s*,\s*\[\s*["'`](push|pull|fetch|clone|ls-remote)/;
|
||||
// curl as a command token.
|
||||
const CURL_OP = /(^|[|&;(`]|\s|\$\()curl\s/;
|
||||
// fetch() with an absolute http(s) URL (loopback filtered separately).
|
||||
const FETCH_ABS = /(^|[^A-Za-z])fetch(Fn|Impl)?\(\s*[`'"]https?:\/\//;
|
||||
|
||||
function isTextFile(full: string): boolean {
|
||||
try {
|
||||
const buf = fs.readFileSync(full);
|
||||
return !buf.subarray(0, 1024).includes(0);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function* walk(dir: string): Generator<string> {
|
||||
for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
|
||||
if (entry.name === 'node_modules' || entry.name === 'dist' || entry.name.startsWith('.')) continue;
|
||||
const full = path.join(dir, entry.name);
|
||||
if (entry.isDirectory()) yield* walk(full);
|
||||
else if (entry.isFile()) yield full;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Collect un-receipted outbound network ops in a file. Skips comments,
|
||||
* loopback lines, `command -v` probes, message-emitting lines, and shell
|
||||
* heredoc bodies (echoed instructions are not executed ops).
|
||||
*/
|
||||
function scanFile(rel: string): string[] {
|
||||
const src = read(rel);
|
||||
const isTs = /\.(ts|js|mjs|tsx)$/.test(rel);
|
||||
const lines = src.split('\n');
|
||||
const offenders: string[] = [];
|
||||
let heredocEnd: string | null = null;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const line = lines[i];
|
||||
if (heredocEnd !== null) {
|
||||
if (line.trim() === heredocEnd) heredocEnd = null;
|
||||
continue;
|
||||
}
|
||||
if (!isTs) {
|
||||
const heredoc = line.match(/<<-?\s*['"]?([A-Za-z_][A-Za-z0-9_]*)['"]?/);
|
||||
if (heredoc) heredocEnd = heredoc[1];
|
||||
}
|
||||
const trimmed = line.trimStart();
|
||||
if (/^(#|\/\/|\*|\/\*)/.test(trimmed)) continue;
|
||||
if (line.includes('127.0.0.1') || line.includes('localhost')) continue;
|
||||
if (/command -v/.test(line)) continue;
|
||||
if (/^(echo|printf|emit|die|fail|log)\b/.test(trimmed)) continue;
|
||||
|
||||
// TS: shell-style git ops only count on lines that actually execute
|
||||
// something (spawn/exec markers) — template-literal prose does not.
|
||||
const tsExecGit =
|
||||
GIT_SPAWN_OP.test(line) ||
|
||||
(GIT_REMOTE_OP.test(line) && /\b(spawn|spawnSync|exec|execSync|execFileSync|runCommand)\b/.test(line));
|
||||
const isNetOp = isTs
|
||||
? FETCH_ABS.test(line) || tsExecGit
|
||||
: CURL_OP.test(line) || GIT_REMOTE_OP.test(line);
|
||||
if (!isNetOp) continue;
|
||||
if (guarded(lines, i)) continue;
|
||||
offenders.push(`${rel}:${i + 1}: ${line.trim().slice(0, 100)}`);
|
||||
}
|
||||
return offenders;
|
||||
}
|
||||
|
||||
describe('egress receipt wiring tripwire', () => {
|
||||
test('every TS sink imports lib/egress-receipt and calls writeReceipt()', () => {
|
||||
for (const rel of MODULE_SINKS) {
|
||||
const src = read(rel);
|
||||
expect(src.includes('egress-receipt'), `${rel}: must import lib/egress-receipt`).toBe(true);
|
||||
expect(src.includes('writeReceipt('), `${rel}: must call writeReceipt() before its send`).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('every shell sink sources gstack-egress-lib.sh', () => {
|
||||
for (const rel of SHELL_SINKS) {
|
||||
const src = read(rel);
|
||||
expect(
|
||||
src.includes('gstack-egress-lib.sh'),
|
||||
`${rel}: must source bin/gstack-egress-lib.sh for _receipted_* helpers`,
|
||||
).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('every network op in a wired shell sink sits under a receipt', () => {
|
||||
const offenders = SHELL_SINKS.flatMap((rel) => scanFile(rel));
|
||||
expect(
|
||||
offenders,
|
||||
'un-receipted network call(s) — wrap in _receipted_curl/_receipted_git or write the receipt first:\n' +
|
||||
offenders.join('\n'),
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
test('browse tunnel: every ngrok.forward() has a writeReceipt in the 30 preceding lines', () => {
|
||||
const lines = read('browse/src/server.ts').split('\n');
|
||||
const offenders: string[] = [];
|
||||
let sawForward = false;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
if (!lines[i].includes('ngrok.forward(')) continue;
|
||||
if (/^\s*(\/\/|\*)/.test(lines[i])) continue;
|
||||
sawForward = true;
|
||||
const context = lines.slice(Math.max(0, i - 30), i).join('\n');
|
||||
if (!context.includes('writeReceipt(')) offenders.push(`browse/src/server.ts:${i + 1}`);
|
||||
}
|
||||
expect(sawForward, 'expected ngrok.forward call sites in server.ts').toBe(true);
|
||||
expect(offenders, 'tunnel session opened without a receipt: ' + offenders.join(', ')).toEqual([]);
|
||||
});
|
||||
|
||||
test('design: every api.openai.com call routes through receiptedFetch', () => {
|
||||
for (const rel of DESIGN_SINKS) {
|
||||
const src = read(rel);
|
||||
expect(
|
||||
src.includes('receipted-fetch'),
|
||||
`${rel}: must import design/src/receipted-fetch`,
|
||||
).toBe(true);
|
||||
const lines = src.split('\n');
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
if (!lines[i].includes('api.openai.com')) continue;
|
||||
if (/^\s*(\/\/|\*)/.test(lines[i])) continue;
|
||||
if (/\bfetch(Fn|Impl)?\(/.test(lines[i])) {
|
||||
throw new Error(
|
||||
`${rel}:${i + 1}: raw fetch to api.openai.com — route it through receiptedFetch()`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('deprecated dead-endpoint brain consumer/reader scripts stay deleted', () => {
|
||||
// lstat (not existsSync) so a dangling symlink also fails.
|
||||
for (const rel of ['bin/gstack-brain-consumer', 'bin/gstack-brain-reader']) {
|
||||
let present = true;
|
||||
try {
|
||||
fs.lstatSync(path.join(ROOT, rel));
|
||||
} catch {
|
||||
present = false;
|
||||
}
|
||||
expect(present, `${rel} was deleted (dead /ingest-repo egress sink) — do not resurrect`).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
test('polarity table names every wired sink exactly once per polarity', () => {
|
||||
const closed = Object.entries(POLARITY).filter(([, p]) => p === 'fail-closed').map(([s]) => s);
|
||||
const open = Object.entries(POLARITY).filter(([, p]) => p === 'fail-open').map(([s]) => s);
|
||||
expect(closed.sort()).toEqual([
|
||||
'brain-sync',
|
||||
'browse-tunnel (ngrok)',
|
||||
'gbrain-mcp-verify',
|
||||
'gbrain-sync',
|
||||
'memory-ingest',
|
||||
'supabase-provision',
|
||||
'telemetry-sync',
|
||||
]);
|
||||
expect(open.sort()).toEqual([
|
||||
'community-dashboard',
|
||||
'context-bill --exact',
|
||||
'design-openai',
|
||||
'git-class user ops (artifacts-init, brain-restore, session-update)',
|
||||
'security-dashboard',
|
||||
'update-check',
|
||||
]);
|
||||
});
|
||||
|
||||
test('polarity spot-checks: closed sinks refuse, open sinks warn', () => {
|
||||
// telemetry-sync (closed): the wrapped POST uses the `closed` policy.
|
||||
expect(read('bin/gstack-telemetry-sync')).toMatch(/_receipted_curl closed telemetry-sync/);
|
||||
// brain-sync (closed): refusal exits before the commit consumes the queue.
|
||||
expect(read('bin/gstack-brain-sync')).toMatch(/gstack-egress-receipt["']? write/);
|
||||
// update-check (open).
|
||||
expect(read('bin/gstack-update-check')).toMatch(/_receipted_curl open update-check/);
|
||||
// dashboards (open).
|
||||
expect(read('bin/gstack-security-dashboard')).toMatch(/_receipted_curl open security-dashboard/);
|
||||
expect(read('bin/gstack-community-dashboard')).toMatch(/_receipted_curl open community-dashboard/);
|
||||
// mcp-verify + provision (closed).
|
||||
expect(read('bin/gstack-gbrain-mcp-verify')).toMatch(/_receipted_curl closed gbrain-mcp-verify/);
|
||||
expect(read('bin/gstack-gbrain-supabase-provision')).toMatch(/_receipted_curl closed supabase-provision/);
|
||||
// design (open): the wrapper catches receipt errors and proceeds.
|
||||
const rf = read('design/src/receipted-fetch.ts');
|
||||
expect(rf).toContain('fail-open');
|
||||
expect(rf.indexOf('writeReceipt(')).toBeLessThan(rf.indexOf('fetchImpl(url, init)'));
|
||||
});
|
||||
|
||||
test('NEW-SINK SCANNER: every outbound network op in the tree is wired or reasoned-exempt', () => {
|
||||
const SWEEP = ['bin', 'lib', 'scripts', 'design/src', 'browse/src'];
|
||||
const offenders: string[] = [];
|
||||
for (const dirRel of SWEEP) {
|
||||
const dir = path.join(ROOT, dirRel);
|
||||
if (!fs.existsSync(dir)) continue;
|
||||
for (const full of walk(dir)) {
|
||||
const rel = path.relative(ROOT, full).split(path.sep).join('/');
|
||||
if (!/\.(ts|js|mjs|sh|tsx)$/.test(rel) && !isTextFile(full)) continue;
|
||||
if (isExempt(rel)) continue;
|
||||
offenders.push(...scanFile(rel));
|
||||
}
|
||||
}
|
||||
expect(
|
||||
offenders,
|
||||
'unwired outbound network op(s). Wire each through the receipt helpers ' +
|
||||
'(_receipted_curl/_receipted_git in shell, writeReceipt/receiptedFetch in TS) ' +
|
||||
'or add a REASONED exemption with the honest why:\n' +
|
||||
offenders.join('\n'),
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
test('shebang tripwire: no bin/gstack-* file carries a node shebang (amendment 2A)', () => {
|
||||
const offenders: string[] = [];
|
||||
for (const entry of fs.readdirSync(path.join(ROOT, 'bin'))) {
|
||||
if (!entry.startsWith('gstack-')) continue;
|
||||
const full = path.join(ROOT, 'bin', entry);
|
||||
if (!fs.lstatSync(full).isFile()) continue;
|
||||
if (!isTextFile(full)) continue;
|
||||
const firstLine = fs.readFileSync(full, 'utf-8').split('\n', 1)[0];
|
||||
if (firstLine.startsWith('#!') && /\bnode\b/.test(firstLine)) {
|
||||
offenders.push(`bin/${entry}: ${firstLine}`);
|
||||
}
|
||||
}
|
||||
expect(offenders, 'node shebangs in bin/ (use #!/usr/bin/env bun): ' + offenders.join(', ')).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,258 @@
|
||||
/**
|
||||
* Egress receipts — chain, fail-closed, verify, shell bridge. Free tier, no network.
|
||||
*
|
||||
* THREAT MODEL: the egress ledger is forensic observability — it records
|
||||
* ATTEMPTED egress so accidents are auditable; it is not an exfiltration
|
||||
* control.
|
||||
*
|
||||
* Pins the auditor contract:
|
||||
* - receipt-before-send fail-closed (EGRESS_RECEIPT_FAILED, no ledger = no send)
|
||||
* - content-free lines chained by prev = sha256(previous raw line)
|
||||
* - tail-read correctness (last line found without loading the whole file)
|
||||
* - abandoned-lock reclaim (stale lock dir >10s old is removed, not fatal)
|
||||
* - WARN-at-size (one self-explanatory stderr warning per process >25MB)
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import * as fs from 'fs';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import { spawnSync } from 'child_process';
|
||||
import {
|
||||
EGRESS_RECEIPT_FAILED,
|
||||
LEDGER_WARN_BYTES,
|
||||
egressLedgerPath,
|
||||
ledgerSizeWarning,
|
||||
listReceipts,
|
||||
resetLedgerSizeWarningForTests,
|
||||
sha256Hex,
|
||||
verifyLedger,
|
||||
writeOutcome,
|
||||
writeReceipt,
|
||||
} from '../lib/egress-receipt';
|
||||
|
||||
const ROOT = path.resolve(new URL(import.meta.url).pathname, '..', '..');
|
||||
|
||||
let home: string;
|
||||
|
||||
beforeEach(() => {
|
||||
home = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-egress-'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
try { fs.chmodSync(path.join(home, 'security'), 0o700); } catch {} // undo fail-closed fixtures
|
||||
fs.rmSync(home, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('egress receipt library', () => {
|
||||
test('receipts chain: prev = sha256 of the previous raw line, "" for line 1', () => {
|
||||
writeReceipt({ home, sink: 'a', host: 'h1', payloadClass: 'c', bytes: 3, sha256: sha256Hex('abc'), consent: 'k=v' });
|
||||
writeReceipt({ home, sink: 'b', host: 'h2', payloadClass: 'c', bytes: 0, sha256: null, consent: 'k=v' });
|
||||
const lines = fs.readFileSync(egressLedgerPath(home), 'utf-8').trim().split('\n');
|
||||
expect(lines.length).toBe(2);
|
||||
const first = JSON.parse(lines[0]);
|
||||
const second = JSON.parse(lines[1]);
|
||||
expect(first.prev).toBe('');
|
||||
expect(second.prev).toBe(sha256Hex(lines[0]));
|
||||
expect(first.sha256).toBe(sha256Hex('abc'));
|
||||
expect(second.sha256).toBeNull();
|
||||
expect(verifyLedger(home)).toMatchObject({ ok: true, count: 2 });
|
||||
});
|
||||
|
||||
test('ledger is 0600 and the security dir 0700', () => {
|
||||
writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' });
|
||||
const ledger = egressLedgerPath(home);
|
||||
expect(fs.statSync(ledger).mode & 0o777).toBe(0o600);
|
||||
expect(fs.statSync(path.dirname(ledger)).mode & 0o777).toBe(0o700);
|
||||
});
|
||||
|
||||
test('fail-closed: unwritable security dir throws typed EGRESS_RECEIPT_FAILED', () => {
|
||||
if (process.platform === 'win32' || process.getuid?.() === 0) return; // chmod is advisory there
|
||||
writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' });
|
||||
fs.chmodSync(path.join(home, 'security'), 0o500);
|
||||
try {
|
||||
expect(() =>
|
||||
writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' }),
|
||||
).toThrow();
|
||||
try {
|
||||
writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' });
|
||||
} catch (err: any) {
|
||||
expect(err.code).toBe(EGRESS_RECEIPT_FAILED);
|
||||
}
|
||||
} finally {
|
||||
fs.chmodSync(path.join(home, 'security'), 0o700);
|
||||
}
|
||||
});
|
||||
|
||||
test('outcome records join back onto their receipt in listReceipts', () => {
|
||||
const { id } = writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' });
|
||||
writeOutcome({ home, receipt: id, status: 204 });
|
||||
const receipts = listReceipts(home);
|
||||
expect(receipts.length).toBe(1);
|
||||
expect(receipts[0].status).toBe('204');
|
||||
expect(verifyLedger(home)).toMatchObject({ ok: true, count: 2 });
|
||||
});
|
||||
|
||||
test('tampering with a middle line breaks verification at that line', () => {
|
||||
for (let i = 0; i < 3; i += 1) {
|
||||
writeReceipt({ home, sink: `s${i}`, host: 'h', payloadClass: 'c', consent: 'telemetry=community' });
|
||||
}
|
||||
const ledger = egressLedgerPath(home);
|
||||
const lines = fs.readFileSync(ledger, 'utf-8').trim().split('\n');
|
||||
lines[1] = lines[1].replace('community', 'communitX');
|
||||
fs.writeFileSync(ledger, `${lines.join('\n')}\n`);
|
||||
// Line 2's edited bytes no longer hash to line 3's recorded prev.
|
||||
expect(verifyLedger(home)).toMatchObject({ ok: false, brokenLine: 3 });
|
||||
});
|
||||
|
||||
test('validation rejects garbage before touching the ledger', () => {
|
||||
expect(() => writeReceipt({ home, sink: '', host: 'h', payloadClass: 'c', consent: 'k' } as any)).toThrow();
|
||||
expect(() => writeReceipt({ home, sink: 's', host: 'h', payloadClass: 'c', consent: 'k', bytes: -1 })).toThrow();
|
||||
expect(() => writeReceipt({ home, sink: 's', host: 'h', payloadClass: 'c', consent: 'k', sha256: 'nothex' })).toThrow();
|
||||
expect(fs.existsSync(egressLedgerPath(home))).toBe(false);
|
||||
});
|
||||
|
||||
test('abandoned lock: a stale lock dir (>10s-old mtime) is reclaimed by the next writer', () => {
|
||||
// First write creates the security dir so the lock path's parent exists.
|
||||
writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' });
|
||||
const lock = `${egressLedgerPath(home)}.lock`;
|
||||
fs.mkdirSync(lock);
|
||||
const old = new Date(Date.now() - 60_000);
|
||||
fs.utimesSync(lock, old, old);
|
||||
const started = Date.now();
|
||||
const { id } = writeReceipt({ home, sink: 'b', host: 'h', payloadClass: 'c', consent: 'k=v' });
|
||||
expect(id).toMatch(/^[0-9a-f]{64}$/);
|
||||
// Spin budget is 2.5s; reclaim happens right after budget exhaustion.
|
||||
expect(Date.now() - started).toBeLessThan(10_000);
|
||||
expect(fs.existsSync(lock)).toBe(false);
|
||||
expect(verifyLedger(home)).toMatchObject({ ok: true, count: 2 });
|
||||
});
|
||||
|
||||
test('a fresh (recent-mtime) lock held past the budget fails closed instead of being stolen', () => {
|
||||
writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' });
|
||||
const lock = `${egressLedgerPath(home)}.lock`;
|
||||
fs.mkdirSync(lock);
|
||||
// Keep the mtime fresh so the reclaim path never fires: refresh it in the
|
||||
// background while the writer spins out its 2.5s budget.
|
||||
const refresher = setInterval(() => {
|
||||
const now = new Date();
|
||||
try { fs.utimesSync(lock, now, now); } catch { /* test teardown race */ }
|
||||
}, 1000);
|
||||
try {
|
||||
expect(() =>
|
||||
writeReceipt({ home, sink: 'b', host: 'h', payloadClass: 'c', consent: 'k=v' }),
|
||||
).toThrow(/locked/);
|
||||
} finally {
|
||||
clearInterval(refresher);
|
||||
fs.rmdirSync(lock);
|
||||
}
|
||||
}, 15_000);
|
||||
|
||||
test('tail-read: last line is found correctly on a multi-record ledger larger than the tail window', () => {
|
||||
// 30 records ≈ 9KB > the 4KB tail window, so the append path must find
|
||||
// the true last line from a partial read.
|
||||
for (let i = 0; i < 30; i += 1) {
|
||||
writeReceipt({
|
||||
home,
|
||||
sink: `sink-${i}`,
|
||||
host: 'h',
|
||||
payloadClass: `class-${'x'.repeat(200)}-${i}`,
|
||||
consent: 'k=v',
|
||||
});
|
||||
}
|
||||
const ledger = egressLedgerPath(home);
|
||||
expect(fs.statSync(ledger).size).toBeGreaterThan(4096);
|
||||
const result = verifyLedger(home);
|
||||
expect(result).toMatchObject({ ok: true, count: 30 });
|
||||
});
|
||||
|
||||
test('chain verify stays intact across 100+ records', () => {
|
||||
for (let i = 0; i < 120; i += 1) {
|
||||
writeReceipt({ home, sink: `s${i}`, host: 'h', payloadClass: 'c', consent: 'k=v' });
|
||||
if (i % 10 === 0) writeOutcome({ home, receipt: 'f'.repeat(64), status: 200 });
|
||||
}
|
||||
const result = verifyLedger(home);
|
||||
expect(result.ok).toBe(true);
|
||||
expect(result.count).toBe(132);
|
||||
expect(listReceipts(home).length).toBe(120);
|
||||
});
|
||||
|
||||
test('WARN-at-size: >25MB ledger emits one self-explanatory stderr warning per process', () => {
|
||||
const ledger = egressLedgerPath(home);
|
||||
fs.mkdirSync(path.dirname(ledger), { recursive: true, mode: 0o700 });
|
||||
// Grow the file past the threshold with valid-looking filler; the warning
|
||||
// keys off file size only.
|
||||
const filler = `${JSON.stringify({ type: 'egress', pad: 'x'.repeat(1024) })}\n`;
|
||||
const chunk = filler.repeat(1024); // ~1MB
|
||||
const writes = Math.ceil(LEDGER_WARN_BYTES / chunk.length) + 1;
|
||||
for (let i = 0; i < writes; i += 1) fs.appendFileSync(ledger, chunk);
|
||||
expect(fs.statSync(ledger).size).toBeGreaterThan(LEDGER_WARN_BYTES);
|
||||
|
||||
resetLedgerSizeWarningForTests();
|
||||
const captured: string[] = [];
|
||||
const originalWrite = process.stderr.write.bind(process.stderr);
|
||||
(process.stderr as any).write = (chunk: string) => { captured.push(String(chunk)); return true; };
|
||||
try {
|
||||
writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' });
|
||||
writeReceipt({ home, sink: 'b', host: 'h', payloadClass: 'c', consent: 'k=v' });
|
||||
} finally {
|
||||
(process.stderr as any).write = originalWrite;
|
||||
}
|
||||
const warnings = captured.filter((c) => c.includes('egress ledger is large'));
|
||||
expect(warnings.length).toBe(1); // once per process, not per write
|
||||
// Self-explanatory shape: what the ledger is, how to inspect, what's coming.
|
||||
expect(warnings[0]).toContain('ATTEMPTS to send off-machine');
|
||||
expect(warnings[0]).toContain('gstack-egress list');
|
||||
expect(warnings[0]).toContain('rotation');
|
||||
expect(warnings[0]).toContain(ledger);
|
||||
|
||||
// verifyLedger surfaces the same warning as data.
|
||||
const message = ledgerSizeWarning(ledger, fs.statSync(ledger).size);
|
||||
expect(message).toContain('MB');
|
||||
expect(verifyLedger(home).sizeWarning).toBe(message);
|
||||
});
|
||||
});
|
||||
|
||||
describe('gstack-egress-receipt shell bridge', () => {
|
||||
const bin = path.join(ROOT, 'bin', 'gstack-egress-receipt');
|
||||
|
||||
test('write hashes the exact payload file, prints the receipt id; outcome joins', () => {
|
||||
const payload = path.join(home, 'payload.json');
|
||||
fs.writeFileSync(payload, '[{"v":1}]');
|
||||
const write = spawnSync(bin, ['write', '--sink', 'telemetry-sync', '--host', '127.0.0.1:8399',
|
||||
'--class', 'telemetry-events', '--payload-file', payload, '--consent', 'telemetry=community'],
|
||||
{ encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home } });
|
||||
expect(write.status).toBe(0);
|
||||
const id = write.stdout.trim();
|
||||
expect(id).toMatch(/^[0-9a-f]{64}$/);
|
||||
const outcome = spawnSync(bin, ['outcome', id, '204'],
|
||||
{ encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home } });
|
||||
expect(outcome.status).toBe(0);
|
||||
const receipts = listReceipts(home);
|
||||
expect(receipts.length).toBe(1);
|
||||
expect(receipts[0].bytes).toBe(9);
|
||||
expect(receipts[0].sha256).toBe(sha256Hex('[{"v":1}]'));
|
||||
expect(receipts[0].status).toBe('204');
|
||||
});
|
||||
|
||||
test('--no-payload records sha256:null (git-class: a subprocess owns the bytes)', () => {
|
||||
const write = spawnSync(bin, ['write', '--sink', 'brain-sync', '--host', 'github.com',
|
||||
'--class', 'git-push', '--no-payload', '--consent', 'artifacts_sync_mode=auto'],
|
||||
{ encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home } });
|
||||
expect(write.status).toBe(0);
|
||||
const receipts = listReceipts(home);
|
||||
expect(receipts.length).toBe(1);
|
||||
expect(receipts[0].sha256).toBeNull();
|
||||
expect(receipts[0].bytes).toBe(0);
|
||||
});
|
||||
|
||||
test('write exits 3 with EGRESS_RECEIPT_FAILED when the ledger is unwritable', () => {
|
||||
if (process.platform === 'win32' || process.getuid?.() === 0) return;
|
||||
fs.mkdirSync(path.join(home, 'security'), { recursive: true, mode: 0o500 });
|
||||
const write = spawnSync(bin, ['write', '--sink', 's', '--host', 'h', '--class', 'c', '--no-payload'],
|
||||
{ encoding: 'utf-8', env: { ...process.env, GSTACK_HOME: home } });
|
||||
expect(write.status).toBe(3);
|
||||
expect(write.stderr).toContain('EGRESS_RECEIPT_FAILED');
|
||||
fs.chmodSync(path.join(home, 'security'), 0o700);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,21 @@
|
||||
---
|
||||
name: alpha
|
||||
description: Fixture dispatcher with a mode table and forced-read references.
|
||||
triggers: a-key-the-upstream-router-DOES-read
|
||||
x-dead-key: a-key-no-router-reads
|
||||
---
|
||||
|
||||
# Alpha
|
||||
|
||||
## Dispatch protocol
|
||||
|
||||
1. Infer the mode from the request.
|
||||
2. Read `references/CORE.md` and `references/POLICY.md` for every invocation. Read `references/OPTIONAL.md` before public-web work.
|
||||
3. When the target is a repository, read `references/CORE.md` once before specialist work; it is already billed eagerly.
|
||||
|
||||
## Top-level modes
|
||||
|
||||
| Mode | Infer when | Candidate internal specialists |
|
||||
|---|---|---|
|
||||
| `Discovery` | The idea is fluid. | `references/legacy/office.md` |
|
||||
| `Full chain` | Everything at once. | `references/legacy/auto.md` |
|
||||
@@ -0,0 +1,3 @@
|
||||
# Core
|
||||
|
||||
Forced-read reference number one. Read on every invocation of alpha.
|
||||
@@ -0,0 +1,3 @@
|
||||
# Optional
|
||||
|
||||
Conditional reference. Not part of the eager bill.
|
||||
@@ -0,0 +1,3 @@
|
||||
# Policy
|
||||
|
||||
Forced-read reference number two.
|
||||
@@ -0,0 +1,8 @@
|
||||
---
|
||||
name: beta
|
||||
description: Clean fixture tool skill with no forced reads and no mode table.
|
||||
---
|
||||
|
||||
# Beta
|
||||
|
||||
One-command tool skill. Nothing eager beyond this file, nothing lazy.
|
||||
@@ -0,0 +1,3 @@
|
||||
# Foreign host file
|
||||
|
||||
A skill-shaped file another host dropped into scanner scope (#1694).
|
||||
@@ -9,13 +9,18 @@ import * as fs from 'fs';
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const SRC = fs.readFileSync(path.join(ROOT, 'bin', 'gstack-config'), 'utf-8');
|
||||
|
||||
// Pull out just the gbrain-refresh `ok)` branch so assertions can't be
|
||||
// satisfied by unrelated text elsewhere in the file.
|
||||
// Pull out just the gbrain-refresh healthy-status branch so assertions can't
|
||||
// be satisfied by unrelated text elsewhere in the file. The case label grew
|
||||
// from `ok)` to `ok|timeout|thin-client)` (#1964, #2051) and may grow again,
|
||||
// so match any label that STARTS with `ok` followed by alternations.
|
||||
function okBranch(): string {
|
||||
const start = SRC.indexOf('gbrain-refresh)');
|
||||
const ok = SRC.indexOf('ok)', start);
|
||||
if (start < 0) throw new Error('Could not locate gbrain-refresh case');
|
||||
const labelMatch = /^\s*ok(?:\|[\w-]+)*\)/m.exec(SRC.slice(start));
|
||||
if (!labelMatch) throw new Error('Could not locate gbrain-refresh ok) branch');
|
||||
const ok = start + labelMatch.index;
|
||||
const end = SRC.indexOf(';;', ok);
|
||||
if (start < 0 || ok < 0 || end < 0) throw new Error('Could not locate gbrain-refresh ok) branch');
|
||||
if (end < 0) throw new Error('Could not locate gbrain-refresh ok) branch terminator');
|
||||
return SRC.slice(ok, end);
|
||||
}
|
||||
|
||||
|
||||
+26
-2
@@ -33,18 +33,42 @@ const GEMINI_AVAILABLE = (() => {
|
||||
} catch { return false; }
|
||||
})();
|
||||
|
||||
// A binary on PATH is not enough: the CLI can be present but UNUSABLE — the
|
||||
// individual code-assist auth path was deprecated upstream ("migrate to the
|
||||
// Antigravity suite"), which fails every run before any model call, and flag
|
||||
// churn (--skip-trust removed in 0.34) errors at argv parse. Probe with a
|
||||
// bare --help: a CLI that can't even print usage is unusable, and a working
|
||||
// one is cheap to confirm. Deeper auth failures are classified per-run.
|
||||
const GEMINI_USABLE = GEMINI_AVAILABLE && (() => {
|
||||
try {
|
||||
const result = Bun.spawnSync(['gemini', '--help'], { timeout: 15_000 });
|
||||
return result.exitCode === 0;
|
||||
} catch { return false; }
|
||||
})();
|
||||
|
||||
const evalsEnabled = !!process.env.EVALS;
|
||||
|
||||
// Skip all tests if gemini is not available or EVALS is not set.
|
||||
const SKIP = !GEMINI_AVAILABLE || !evalsEnabled;
|
||||
// External-service tests are periodic-tier (CLAUDE.md tiering rule 3):
|
||||
// "Requires external service (Codex, Gemini)? -> periodic". The positive
|
||||
// form below is the canonical whole-file guard shape — the sharded runner's
|
||||
// classifyPaidTestFile greps for it to exclude this file from gate.
|
||||
const tierOk = process.env.EVALS_TIER === 'periodic';
|
||||
|
||||
// Skip all tests if gemini is not available/usable, EVALS is not set, or
|
||||
// we're in the gate tier.
|
||||
const SKIP = !GEMINI_USABLE || !evalsEnabled || !tierOk;
|
||||
|
||||
const describeGemini = SKIP ? describe.skip : describe;
|
||||
|
||||
// Log why we're skipping (helpful for debugging CI)
|
||||
if (!evalsEnabled) {
|
||||
// Silent — same as Claude E2E tests, EVALS=1 required
|
||||
} else if (!tierOk) {
|
||||
process.stderr.write('\nGemini E2E: SKIPPED — external-service test, periodic tier only (EVALS_TIER === \'periodic\')\n');
|
||||
} else if (!GEMINI_AVAILABLE) {
|
||||
process.stderr.write('\nGemini E2E: SKIPPED — gemini binary not found (install: npm i -g @google/gemini-cli)\n');
|
||||
} else if (!GEMINI_USABLE) {
|
||||
process.stderr.write('\nGemini E2E: SKIPPED — gemini CLI present but unusable (auth path deprecated upstream or CLI broken; try updating @google/gemini-cli)\n');
|
||||
}
|
||||
|
||||
// --- Diff-based test selection ---
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
/**
|
||||
* gstack-egress CLI — list | verify | grants smoke tests. Free tier.
|
||||
*
|
||||
* Spawns the real bin against a temp GSTACK_HOME: list filters, verify
|
||||
* exit-3-on-tamper (naming the first broken line), sizeWarning surfacing,
|
||||
* and grants against the upstream config keys (telemetry,
|
||||
* artifacts_sync_mode, redact_repo_visibility, redact_prepush_hook).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import * as fs from 'fs';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import { spawnSync } from 'child_process';
|
||||
import {
|
||||
LEDGER_WARN_BYTES,
|
||||
egressLedgerPath,
|
||||
sha256Hex,
|
||||
writeReceipt,
|
||||
} from '../lib/egress-receipt';
|
||||
|
||||
const ROOT = path.resolve(new URL(import.meta.url).pathname, '..', '..');
|
||||
const BIN = path.join(ROOT, 'bin', 'gstack-egress');
|
||||
|
||||
let home: string;
|
||||
|
||||
beforeEach(() => {
|
||||
home = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-egress-cli-'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
fs.rmSync(home, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function run(args: string[]) {
|
||||
const result = spawnSync(BIN, args, {
|
||||
encoding: 'utf-8',
|
||||
env: { ...process.env, GSTACK_HOME: home },
|
||||
});
|
||||
return { code: result.status ?? -1, stdout: result.stdout || '', stderr: result.stderr || '' };
|
||||
}
|
||||
|
||||
describe('gstack-egress list', () => {
|
||||
test('fresh home prints "no receipts" and the ledger path, exit 0', () => {
|
||||
const r = run(['list']);
|
||||
expect(r.code).toBe(0);
|
||||
expect(r.stdout).toContain('no receipts');
|
||||
expect(r.stdout).toContain(egressLedgerPath(home));
|
||||
});
|
||||
|
||||
test('--json returns receipts and honors --sink/--host/--since filters', () => {
|
||||
writeReceipt({ home, sink: 'telemetry-sync', host: '10.0.0.1:8399', payloadClass: 'telemetry-events', bytes: 2, sha256: sha256Hex('[]'), consent: 'telemetry=community' });
|
||||
writeReceipt({ home, sink: 'design-openai', host: 'api.openai.com', payloadClass: 'generate-image-request', consent: 'user ran design command' });
|
||||
const all = run(['list', '--json']);
|
||||
expect(all.code).toBe(0);
|
||||
expect(JSON.parse(all.stdout).length).toBe(2);
|
||||
const filtered = run(['list', '--json', '--sink', 'telemetry-sync']);
|
||||
const rows = JSON.parse(filtered.stdout);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].host).toBe('10.0.0.1:8399');
|
||||
expect(rows[0].sha256).toBe(sha256Hex('[]'));
|
||||
const none = run(['list', '--json', '--since', '2999-01-01T00:00:00Z']);
|
||||
expect(JSON.parse(none.stdout).length).toBe(0);
|
||||
});
|
||||
|
||||
test('unknown option exits 2 with usage', () => {
|
||||
const r = run(['list', '--bogus']);
|
||||
expect(r.code).toBe(2);
|
||||
expect(r.stderr).toContain('Usage');
|
||||
});
|
||||
});
|
||||
|
||||
describe('gstack-egress verify', () => {
|
||||
test('exits 0 on an intact chain and 3 naming the first broken line on tamper', () => {
|
||||
writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'telemetry=community' });
|
||||
writeReceipt({ home, sink: 'b', host: 'h', payloadClass: 'c', consent: 'telemetry=community' });
|
||||
const ok = run(['verify']);
|
||||
expect(ok.code).toBe(0);
|
||||
expect(ok.stdout).toContain('chain intact: 2');
|
||||
|
||||
const ledger = egressLedgerPath(home);
|
||||
const lines = fs.readFileSync(ledger, 'utf-8').trim().split('\n');
|
||||
lines[0] = lines[0].replace('community', 'communitX');
|
||||
fs.writeFileSync(ledger, `${lines.join('\n')}\n`);
|
||||
const tampered = run(['verify']);
|
||||
expect(tampered.code).toBe(3);
|
||||
expect(tampered.stdout).toContain('line 2');
|
||||
});
|
||||
|
||||
test('prints the sizeWarning when the ledger exceeds the threshold', () => {
|
||||
writeReceipt({ home, sink: 'a', host: 'h', payloadClass: 'c', consent: 'k=v' });
|
||||
const ledger = egressLedgerPath(home);
|
||||
// verify keys the warning off file size only — pad with a trailing
|
||||
// comment-free blank region by appending to a side channel is not
|
||||
// possible in JSONL, so grow via many valid-shaped junk lines and
|
||||
// assert on sizeWarning presence (chain will break; both surface).
|
||||
const filler = `${JSON.stringify({ type: 'junk', pad: 'x'.repeat(1024) })}\n`.repeat(1024);
|
||||
while (fs.statSync(ledger).size <= LEDGER_WARN_BYTES) fs.appendFileSync(ledger, filler);
|
||||
const r = run(['verify', '--json']);
|
||||
expect(r.code).toBe(3); // filler breaks the chain — expected
|
||||
const parsed = JSON.parse(r.stdout);
|
||||
expect(parsed.sizeWarning).toContain('egress ledger is large');
|
||||
expect(parsed.sizeWarning).toContain('gstack-egress list');
|
||||
const human = run(['verify']);
|
||||
expect(human.stdout).toContain('egress ledger is large');
|
||||
});
|
||||
});
|
||||
|
||||
describe('gstack-egress grants', () => {
|
||||
test('fresh home shows the four upstream grants off, each naming file and revoke command', () => {
|
||||
const r = run(['grants']);
|
||||
expect(r.code).toBe(0);
|
||||
for (const grant of ['telemetry', 'brain-sync', 'redact_repo_visibility', 'redact_prepush_hook']) {
|
||||
expect(r.stdout).toContain(grant);
|
||||
}
|
||||
expect(r.stdout).not.toContain('[GRANTED]');
|
||||
expect(r.stdout).toContain(path.join(home, 'config.yaml'));
|
||||
expect(r.stdout).toContain('revoke:');
|
||||
});
|
||||
|
||||
test('--json flips granted=true when telemetry and sync mode are enabled', () => {
|
||||
const config = spawnSync(path.join(ROOT, 'bin', 'gstack-config'), ['set', 'telemetry', 'community'], {
|
||||
encoding: 'utf-8',
|
||||
env: { ...process.env, GSTACK_HOME: home },
|
||||
});
|
||||
expect(config.status).toBe(0);
|
||||
spawnSync(path.join(ROOT, 'bin', 'gstack-config'), ['set', 'artifacts_sync_mode', 'full'], {
|
||||
encoding: 'utf-8',
|
||||
env: { ...process.env, GSTACK_HOME: home },
|
||||
});
|
||||
const r = run(['grants', '--json']);
|
||||
expect(r.code).toBe(0);
|
||||
const grants = JSON.parse(r.stdout);
|
||||
const telemetry = grants.find((g: any) => g.grant === 'telemetry');
|
||||
expect(telemetry.granted).toBe(true);
|
||||
expect(telemetry.revoke).toContain('telemetry off');
|
||||
const sync = grants.find((g: any) => g.grant === 'brain-sync');
|
||||
expect(sync.granted).toBe(true);
|
||||
expect(sync.value).toBe('full');
|
||||
const hook = grants.find((g: any) => g.grant === 'redact_prepush_hook');
|
||||
expect(hook.granted).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('gstack-egress usage', () => {
|
||||
test('no subcommand exits 2 with usage', () => {
|
||||
const r = run([]);
|
||||
expect(r.code).toBe(2);
|
||||
expect(r.stderr).toContain('Usage');
|
||||
});
|
||||
});
|
||||
@@ -51,13 +51,20 @@ function makeFakeCurl(opts: {
|
||||
printf 'CURL_CALL '"'"'%s'"'"' ' "$@" >> "${curlCallLog}"
|
||||
echo "" >> "${curlCallLog}"
|
||||
|
||||
# Walk argv to find -o <out> and -d <data>.
|
||||
# Walk argv to find -o <out> and the request body (-d <data> or the
|
||||
# receipted --data-binary @<payload-file> shape).
|
||||
out=""
|
||||
data=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
-o) out="$2"; shift 2 ;;
|
||||
-d) data="$2"; shift 2 ;;
|
||||
--data-binary)
|
||||
case "$2" in
|
||||
@*) data="$(cat "\${2#@}" 2>/dev/null)" ;;
|
||||
*) data="$2" ;;
|
||||
esac
|
||||
shift 2 ;;
|
||||
*) shift ;;
|
||||
esac
|
||||
done
|
||||
@@ -83,6 +90,8 @@ function runVerify(token: string, url: string): { code: number; stdout: string;
|
||||
...process.env,
|
||||
PATH: `${fakeBinDir}:${process.env.PATH}`,
|
||||
GBRAIN_MCP_TOKEN: token,
|
||||
// The probe writes egress receipts — keep them in the temp home.
|
||||
GSTACK_HOME: tmpDir,
|
||||
},
|
||||
encoding: 'utf-8',
|
||||
});
|
||||
|
||||
@@ -24,7 +24,7 @@
|
||||
import * as fs from 'fs';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import { hermeticChildEnv, isHermeticEnabled } from './hermetic-env';
|
||||
import { hermeticChildEnv, hermeticSkillsConfigDir, isHermeticEnabled } from './hermetic-env';
|
||||
|
||||
/** Strip ANSI escapes for pattern-matching against visible text. */
|
||||
export function stripAnsi(s: string): string {
|
||||
@@ -61,6 +61,11 @@ export function resolveClaudeBinary(): string | null {
|
||||
}
|
||||
|
||||
export interface ClaudePtyOptions {
|
||||
/** Register the repo's shipped skills in the child's user scope via
|
||||
* hermeticSkillsConfigDir(). Required by any test that types a /skill
|
||||
* slash command; without it hermetic claude rejects the command as
|
||||
* Unknown before any model turn. No effect when EVALS_HERMETIC=0. */
|
||||
seedSkills?: boolean;
|
||||
/**
|
||||
* Permission mode for the session.
|
||||
* - 'plan' (default) — launches with --permission-mode plan
|
||||
@@ -300,6 +305,17 @@ export function isPermissionDialogVisible(visible: string): boolean {
|
||||
}
|
||||
|
||||
/** Detect any AskUserQuestion-shaped numbered option list with cursor. */
|
||||
/**
|
||||
* Strip terminal residue that survives ANSI-stripping and can interleave
|
||||
* with AUQ text: DEC cursor-visibility fragments (`[?25l` / `[?25h` — the ESC
|
||||
* byte is gone but the bracket sequence remains) and the spinner frames
|
||||
* rendered between them. Observed in plan-design-with-ui's failure buffer,
|
||||
* where `[?25l✻Sprouting…[?25h` fragments sat inside the option lines.
|
||||
*/
|
||||
export function stripPtyResidue(visible: string): string {
|
||||
return visible.replace(/\[\?25[lh]/g, '');
|
||||
}
|
||||
|
||||
export function isNumberedOptionListVisible(visible: string): boolean {
|
||||
// ❯ cursor + at least two numbered options 1-9.
|
||||
// Matches the trust dialog AND plan-ready prompt AND skill questions.
|
||||
@@ -311,7 +327,8 @@ export function isNumberedOptionListVisible(visible: string): boolean {
|
||||
// because `t-2` is a word-to-word transition. We use the weaker
|
||||
// `[^0-9]2\.` to require a non-digit before `2` (so we don't match
|
||||
// `12.0`) without requiring whitespace.
|
||||
return /❯\s*1\./.test(visible) && /(^|[^0-9])2\./.test(visible);
|
||||
const cleaned = stripPtyResidue(visible);
|
||||
return /❯\s*1\./.test(cleaned) && /(^|[^0-9])2\./.test(cleaned);
|
||||
}
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────────
|
||||
@@ -687,6 +704,7 @@ export function isScopeGateAutoSelectVisible(visible: string): boolean {
|
||||
export function parseNumberedOptions(
|
||||
visible: string,
|
||||
): Array<{ index: number; label: string }> {
|
||||
visible = stripPtyResidue(visible);
|
||||
const tail = visible.length > 4096 ? visible.slice(-4096) : visible;
|
||||
// Split on lines, look for `❯ N.` or ` N.` patterns. Up to N=9.
|
||||
// The `\s*` after `.` (not `\s+`) is required because stripAnsi removes
|
||||
@@ -728,30 +746,41 @@ export function parseNumberedOptions(
|
||||
const seenIndices = new Set<number>();
|
||||
|
||||
// Cursor line: option 1 may be inline after box dividers + prompt header
|
||||
// (`...divider...header...❯1. label`). Use a non-anchored regex that
|
||||
// captures `❯N. label` from anywhere on the line through end-of-line.
|
||||
// Only used for the cursor line — subsequent options are parsed with the
|
||||
// start-of-line `optionRe`.
|
||||
// (`...divider...header...❯1. label`) — and, when the PTY reflows the whole
|
||||
// AUQ onto ONE logical line, options 2..N sit on the SAME line after it
|
||||
// (observed with /plan-design-review's Step-0 scope gate: `❯1.Branch diff
|
||||
// ... 2.Plan or design doc ... 5.Chat about this ... Enter to select`).
|
||||
// Parse the cursor line as a STREAM: find every `N.` token (not preceded
|
||||
// by a digit, not followed by one — excludes "12." and "1.5"), require
|
||||
// ascending indices starting from the cursor's option, and take each
|
||||
// label as the text between successive number tokens.
|
||||
const cursorLine = lines[cursorLineIdx] ?? '';
|
||||
const cursorInlineRe = /❯\s*([1-9])\.\s*(\S.*?)\s*$/;
|
||||
const inlineMatch = cursorInlineRe.exec(cursorLine);
|
||||
if (inlineMatch) {
|
||||
const idx = Number(inlineMatch[1]);
|
||||
const label = (inlineMatch[2] ?? '').trim();
|
||||
if (label.length > 0 && !seenIndices.has(idx)) {
|
||||
seenIndices.add(idx);
|
||||
found.push({ index: idx, label });
|
||||
}
|
||||
} else {
|
||||
// No inline cursor match — fall back to start-of-line regex.
|
||||
const startMatch = optionRe.exec(cursorLine);
|
||||
if (startMatch) {
|
||||
const idx = Number(startMatch[1]);
|
||||
const label = (startMatch[2] ?? '').trim();
|
||||
if (label.length > 0 && !seenIndices.has(idx)) {
|
||||
seenIndices.add(idx);
|
||||
found.push({ index: idx, label });
|
||||
}
|
||||
const cursorStart = cursorLine.indexOf('❯');
|
||||
const cursorSegment = cursorStart >= 0 ? cursorLine.slice(cursorStart) : cursorLine;
|
||||
const tokenRe = /(?:^|[^0-9])([1-9])\.(?!\d)\s*/g;
|
||||
const tokens: Array<{ idx: number; labelStart: number; matchStart: number }> = [];
|
||||
for (let m = tokenRe.exec(cursorSegment); m !== null; m = tokenRe.exec(cursorSegment)) {
|
||||
tokens.push({
|
||||
idx: Number(m[1]),
|
||||
labelStart: m.index + m[0].length,
|
||||
matchStart: m.index === 0 ? 0 : m.index + 1, // skip the [^0-9] guard char
|
||||
});
|
||||
}
|
||||
// Keep only the ascending run that starts the sequence (1, 2, 3, ...);
|
||||
// stray numbers inside labels break ascension and end the run.
|
||||
let expected = 1;
|
||||
for (let t = 0; t < tokens.length; t++) {
|
||||
const token = tokens[t]!;
|
||||
if (token.idx !== expected) continue;
|
||||
const next = tokens
|
||||
.slice(t + 1)
|
||||
.find((candidate) => candidate.idx === expected + 1 && candidate.matchStart > token.labelStart);
|
||||
const labelEnd = next ? next.matchStart : cursorSegment.length;
|
||||
const label = cursorSegment.slice(token.labelStart, labelEnd).trim();
|
||||
if (label.length > 0 && !seenIndices.has(token.idx)) {
|
||||
seenIndices.add(token.idx);
|
||||
found.push({ index: token.idx, label });
|
||||
expected += 1;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1285,6 +1314,9 @@ export async function launchClaudePty(
|
||||
// Hermetic by default (test/helpers/hermetic-env.ts): operator session
|
||||
// context never reaches the child; per-test opts.env merges last.
|
||||
const childEnv = hermeticChildEnv(opts.env);
|
||||
if (opts.seedSkills && hermetic && !opts.env?.CLAUDE_CONFIG_DIR) {
|
||||
childEnv.CLAUDE_CONFIG_DIR = hermeticSkillsConfigDir();
|
||||
}
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const proc = (Bun as any).spawn([claudePath, ...args], {
|
||||
@@ -1670,6 +1702,7 @@ export async function runPlanSkillObservation(opts: {
|
||||
extraArgs: opts.extraArgs,
|
||||
env: opts.env,
|
||||
model: opts.model,
|
||||
seedSkills: true,
|
||||
});
|
||||
|
||||
try {
|
||||
@@ -1966,6 +1999,7 @@ export async function runPlanSkillCounting(opts: {
|
||||
timeoutMs: timeoutMs + 60_000,
|
||||
env: opts.env,
|
||||
model: opts.model,
|
||||
seedSkills: true,
|
||||
});
|
||||
|
||||
const fingerprints: AskUserQuestionFingerprint[] = [];
|
||||
@@ -2199,6 +2233,7 @@ export async function runPlanSkillFloorCheck(opts: {
|
||||
timeoutMs: timeoutMs + 60_000,
|
||||
env: opts.env,
|
||||
model: opts.model,
|
||||
seedSkills: true,
|
||||
});
|
||||
|
||||
try {
|
||||
|
||||
@@ -199,8 +199,12 @@ export async function runCodexSkill(opts: {
|
||||
}
|
||||
}
|
||||
|
||||
// Build codex exec command
|
||||
const args = ['exec', prompt, '--json', '-s', sandbox];
|
||||
// Build codex exec command.
|
||||
// --skip-git-repo-check: newer codex CLIs refuse exec in an untrusted
|
||||
// non-git directory ("Not inside a trusted directory and
|
||||
// --skip-git-repo-check was not specified") — our temp skill dirs are
|
||||
// exactly that. Empirically verified against codex on this machine.
|
||||
const args = ['exec', prompt, '--json', '-s', sandbox, '--skip-git-repo-check'];
|
||||
|
||||
// Spawn codex with temp HOME so it discovers our installed skill.
|
||||
// Hermetic scrub (test/helpers/hermetic-env.ts) with codex's auth surface
|
||||
|
||||
@@ -6,6 +6,9 @@ import {
|
||||
EvalCollector,
|
||||
extractToolSummary,
|
||||
findPreviousRun,
|
||||
findLatestFinalizedRun,
|
||||
isPartialEval,
|
||||
listEvalJsonFiles,
|
||||
compareEvalResults,
|
||||
formatComparison,
|
||||
generateCommentary,
|
||||
@@ -58,6 +61,19 @@ function makeResult(overrides?: Partial<EvalResult>): EvalResult {
|
||||
};
|
||||
}
|
||||
|
||||
/** Capture everything a block writes to stderr (finalize prints there). */
|
||||
async function captureStderr(fn: () => Promise<void>): Promise<string> {
|
||||
const original = process.stderr.write.bind(process.stderr);
|
||||
let captured = '';
|
||||
(process.stderr as any).write = (chunk: any) => { captured += String(chunk); return true; };
|
||||
try {
|
||||
await fn();
|
||||
} finally {
|
||||
(process.stderr as any).write = original;
|
||||
}
|
||||
return captured;
|
||||
}
|
||||
|
||||
// --- EvalCollector tests ---
|
||||
|
||||
describe('EvalCollector', () => {
|
||||
@@ -119,6 +135,41 @@ describe('EvalCollector', () => {
|
||||
expect(fs.readdirSync(tmpDir).filter(f => f.endsWith('.json') && !f.startsWith('_partial'))).toHaveLength(1);
|
||||
});
|
||||
|
||||
test('with no completed prior run, says NO BASELINE instead of comparing against its own partial', async () => {
|
||||
// addTest writes the in-progress accumulator into the same dir. If that
|
||||
// counted as a baseline, the run would compare against itself and print a
|
||||
// reassuring all-clear forever.
|
||||
const collector = new EvalCollector('e2e', tmpDir);
|
||||
collector.addTest(makeEntry({ name: 'test-1', passed: true }));
|
||||
|
||||
const output = await captureStderr(async () => { await collector.finalize(); });
|
||||
|
||||
expect(fs.existsSync(path.join(tmpDir, '_partial-e2e.json'))).toBe(true); // the trap exists
|
||||
expect(output).toContain('NO BASELINE');
|
||||
expect(output).not.toContain('vs previous');
|
||||
expect(output).not.toContain('Stable run');
|
||||
});
|
||||
|
||||
test('with a genuine prior run, reports the real delta', async () => {
|
||||
fs.writeFileSync(
|
||||
path.join(tmpDir, '0.3.5-main-e2e-20260312-100000.json'),
|
||||
JSON.stringify(makeResult({
|
||||
timestamp: '2026-03-12T10:00:00Z',
|
||||
tests: [makeEntry({ name: 'test-1', passed: true, turns_used: 5 })],
|
||||
})),
|
||||
);
|
||||
|
||||
const collector = new EvalCollector('e2e', tmpDir);
|
||||
collector.addTest(makeEntry({ name: 'test-1', passed: false, turns_used: 5 }));
|
||||
|
||||
const output = await captureStderr(async () => { await collector.finalize(); });
|
||||
|
||||
expect(output).toContain('vs previous');
|
||||
expect(output).toContain('REGRESSION');
|
||||
expect(output).toContain('1 regressed');
|
||||
expect(output).not.toContain('NO BASELINE');
|
||||
});
|
||||
|
||||
test('empty collector writes valid file', async () => {
|
||||
const collector = new EvalCollector('llm-judge', tmpDir);
|
||||
const filepath = await collector.finalize();
|
||||
@@ -131,6 +182,59 @@ describe('EvalCollector', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// --- GSTACK_EVAL_DIR + shard slug tests ---
|
||||
|
||||
describe('EvalCollector eval-dir resolution', () => {
|
||||
const savedEnv = process.env.GSTACK_EVAL_DIR;
|
||||
|
||||
afterEach(() => {
|
||||
if (savedEnv === undefined) delete process.env.GSTACK_EVAL_DIR;
|
||||
else process.env.GSTACK_EVAL_DIR = savedEnv;
|
||||
});
|
||||
|
||||
test('honors GSTACK_EVAL_DIR set after import — no --preload needed', async () => {
|
||||
// The default eval dir must resolve lazily at construction, not at module
|
||||
// load: the sharded runner sets GSTACK_EVAL_DIR in each shard child's env
|
||||
// and shard tests import this module long before any collector exists.
|
||||
const envDir = path.join(tmpDir, 'env-dir');
|
||||
process.env.GSTACK_EVAL_DIR = envDir;
|
||||
const collector = new EvalCollector('e2e');
|
||||
collector.addTest(makeEntry());
|
||||
await captureStderr(async () => { await collector.finalize(); });
|
||||
expect(fs.readdirSync(envDir).filter(f => !f.startsWith('_partial'))).toHaveLength(1);
|
||||
});
|
||||
|
||||
test('explicit constructor arg beats GSTACK_EVAL_DIR', async () => {
|
||||
process.env.GSTACK_EVAL_DIR = path.join(tmpDir, 'env-dir');
|
||||
const explicit = path.join(tmpDir, 'explicit');
|
||||
const collector = new EvalCollector('e2e', explicit);
|
||||
collector.addTest(makeEntry());
|
||||
await captureStderr(async () => { await collector.finalize(); });
|
||||
expect(fs.existsSync(path.join(tmpDir, 'env-dir'))).toBe(false);
|
||||
expect(fs.readdirSync(explicit).length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('writes the shard slug when the eval dir is a shards/ subdir', async () => {
|
||||
const shardDir = path.join(tmpDir, 'shards', 'skill-e2e-qa');
|
||||
const collector = new EvalCollector('e2e', shardDir);
|
||||
collector.addTest(makeEntry());
|
||||
await captureStderr(async () => { await collector.finalize(); });
|
||||
|
||||
const partial = JSON.parse(fs.readFileSync(path.join(shardDir, '_partial-e2e.json'), 'utf-8'));
|
||||
expect(partial.shard).toBe('skill-e2e-qa');
|
||||
const final = fs.readdirSync(shardDir).find(f => !f.startsWith('_partial'))!;
|
||||
expect(JSON.parse(fs.readFileSync(path.join(shardDir, final), 'utf-8')).shard).toBe('skill-e2e-qa');
|
||||
});
|
||||
|
||||
test('writes no shard slug for a flat eval dir', async () => {
|
||||
const collector = new EvalCollector('e2e', tmpDir);
|
||||
collector.addTest(makeEntry());
|
||||
await captureStderr(async () => { await collector.finalize(); });
|
||||
const final = fs.readdirSync(tmpDir).find(f => f.endsWith('.json') && !f.startsWith('_partial'))!;
|
||||
expect(JSON.parse(fs.readFileSync(path.join(tmpDir, final), 'utf-8')).shard).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
// --- judgePassed tests ---
|
||||
|
||||
describe('judgePassed', () => {
|
||||
@@ -259,6 +363,34 @@ describe('findPreviousRun', () => {
|
||||
expect(result).toBeNull(); // only file is excluded
|
||||
});
|
||||
|
||||
test('never returns the in-progress accumulator as a baseline', () => {
|
||||
// The current run's own partial carries the current tier + branch and the
|
||||
// freshest timestamp. If it were a candidate, every run would compare
|
||||
// against itself and report "no regressions" forever.
|
||||
fs.writeFileSync(
|
||||
path.join(tmpDir, '_partial-e2e.json'),
|
||||
JSON.stringify(makeResult({ branch: 'main', timestamp: '2026-03-14T10:00:00Z', _partial: true })),
|
||||
);
|
||||
|
||||
const result = findPreviousRun(tmpDir, 'e2e', 'main', path.join(tmpDir, 'current.json'));
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
|
||||
test('prefers a completed run over a newer in-progress accumulator', () => {
|
||||
fs.writeFileSync(
|
||||
path.join(tmpDir, '0.3.5-main-e2e-20260312-100000.json'),
|
||||
JSON.stringify(makeResult({ branch: 'main', timestamp: '2026-03-12T10:00:00Z' })),
|
||||
);
|
||||
// Newer, same tier + branch, but in-progress — must lose to the older completed run.
|
||||
fs.writeFileSync(
|
||||
path.join(tmpDir, '_partial-e2e.json'),
|
||||
JSON.stringify(makeResult({ branch: 'main', timestamp: '2026-03-14T10:00:00Z', _partial: true })),
|
||||
);
|
||||
|
||||
const result = findPreviousRun(tmpDir, 'e2e', 'main', path.join(tmpDir, 'current.json'));
|
||||
expect(result).toContain('0.3.5-main-e2e');
|
||||
});
|
||||
|
||||
test('filters by tier', () => {
|
||||
fs.writeFileSync(
|
||||
path.join(tmpDir, '0.3.6-main-llm-judge-20260314-100000.json'),
|
||||
@@ -268,6 +400,143 @@ describe('findPreviousRun', () => {
|
||||
const result = findPreviousRun(tmpDir, 'e2e', 'main', 'current.json');
|
||||
expect(result).toBeNull(); // only llm-judge file, looking for e2e
|
||||
});
|
||||
|
||||
test('a shard run prefers its own shard history over newer other-shard or flat priors', () => {
|
||||
const mine = path.join(tmpDir, 'shards', 'skill-e2e-qa');
|
||||
const other = path.join(tmpDir, 'shards', 'codex-e2e');
|
||||
fs.mkdirSync(mine, { recursive: true });
|
||||
fs.mkdirSync(other, { recursive: true });
|
||||
// Same-shard prior — oldest of the three, must still win.
|
||||
fs.writeFileSync(
|
||||
path.join(mine, '0.3.4-main-e2e-20260311-100000.json'),
|
||||
JSON.stringify(makeResult({ timestamp: '2026-03-11T10:00:00Z' })),
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.join(other, '0.3.5-main-e2e-20260312-100000.json'),
|
||||
JSON.stringify(makeResult({ timestamp: '2026-03-12T10:00:00Z' })),
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.join(tmpDir, '0.3.6-main-e2e-20260313-100000.json'),
|
||||
JSON.stringify(makeResult({ timestamp: '2026-03-13T10:00:00Z' })),
|
||||
);
|
||||
|
||||
const result = findPreviousRun(tmpDir, 'e2e', 'main', path.join(mine, 'current.json'));
|
||||
expect(result).toContain(path.join('shards', 'skill-e2e-qa'));
|
||||
});
|
||||
|
||||
test('a flat run prefers flat history over a newer shard prior', () => {
|
||||
const shardDir = path.join(tmpDir, 'shards', 'skill-e2e-qa');
|
||||
fs.mkdirSync(shardDir, { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(shardDir, '0.3.6-main-e2e-20260314-100000.json'),
|
||||
JSON.stringify(makeResult({ timestamp: '2026-03-14T10:00:00Z' })),
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.join(tmpDir, '0.3.5-main-e2e-20260312-100000.json'),
|
||||
JSON.stringify(makeResult({ timestamp: '2026-03-12T10:00:00Z' })),
|
||||
);
|
||||
|
||||
const result = findPreviousRun(tmpDir, 'e2e', 'main', path.join(tmpDir, 'current.json'));
|
||||
expect(result).toContain('0.3.5-main-e2e');
|
||||
});
|
||||
|
||||
test('falls back to a shard prior when the flat dir has no candidate', () => {
|
||||
const shardDir = path.join(tmpDir, 'shards', 'skill-e2e-qa');
|
||||
fs.mkdirSync(shardDir, { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(shardDir, '0.3.6-main-e2e-20260314-100000.json'),
|
||||
JSON.stringify(makeResult({ timestamp: '2026-03-14T10:00:00Z' })),
|
||||
);
|
||||
|
||||
const result = findPreviousRun(tmpDir, 'e2e', 'main', path.join(tmpDir, 'current.json'));
|
||||
expect(result).toContain(path.join('shards', 'skill-e2e-qa'));
|
||||
});
|
||||
});
|
||||
|
||||
// --- isPartialEval tests ---
|
||||
|
||||
describe('isPartialEval', () => {
|
||||
test('flag set but file renamed — still partial', () => {
|
||||
expect(isPartialEval({ _partial: true }, 'renamed-to-look-final.json')).toBe(true);
|
||||
});
|
||||
|
||||
test('name matches but no flag — still partial', () => {
|
||||
expect(isPartialEval({}, '_partial-e2e.json')).toBe(true);
|
||||
expect(isPartialEval(null, path.join('/some/dir', '_partial-e2e.json'))).toBe(true);
|
||||
});
|
||||
|
||||
test('finalized run is not partial', () => {
|
||||
expect(isPartialEval(makeResult(), '0.3.6-main-e2e-20260314-100000.json')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// --- listEvalJsonFiles / findLatestFinalizedRun tests ---
|
||||
|
||||
describe('findLatestFinalizedRun', () => {
|
||||
test('listEvalJsonFiles recurses exactly one shards/*/ level', () => {
|
||||
fs.writeFileSync(path.join(tmpDir, 'flat.json'), '{}');
|
||||
const shardDir = path.join(tmpDir, 'shards', 'skill-e2e-qa');
|
||||
fs.mkdirSync(shardDir, { recursive: true });
|
||||
fs.writeFileSync(path.join(shardDir, 'sharded.json'), '{}');
|
||||
// Nested one level deeper than the contract — must NOT be picked up.
|
||||
const tooDeep = path.join(shardDir, 'shards', 'nested');
|
||||
fs.mkdirSync(tooDeep, { recursive: true });
|
||||
fs.writeFileSync(path.join(tooDeep, 'too-deep.json'), '{}');
|
||||
fs.writeFileSync(path.join(tmpDir, 'not-json.txt'), '');
|
||||
|
||||
const files = listEvalJsonFiles(tmpDir).map(f => path.basename(f)).sort();
|
||||
expect(files).toEqual(['flat.json', 'sharded.json']);
|
||||
});
|
||||
|
||||
test('finds the newest finalized run across flat dir and shard subdirs', () => {
|
||||
fs.writeFileSync(
|
||||
path.join(tmpDir, '0.3.5-main-e2e-20260312-100000.json'),
|
||||
JSON.stringify(makeResult({ timestamp: '2026-03-12T10:00:00Z' })),
|
||||
);
|
||||
const shardDir = path.join(tmpDir, 'shards', 'skill-e2e-qa');
|
||||
fs.mkdirSync(shardDir, { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(shardDir, '0.3.6-main-e2e-20260314-100000.json'),
|
||||
JSON.stringify(makeResult({ timestamp: '2026-03-14T10:00:00Z' })),
|
||||
);
|
||||
|
||||
const latest = findLatestFinalizedRun(tmpDir, 'e2e');
|
||||
expect(latest?.filepath).toContain('skill-e2e-qa');
|
||||
expect(latest?.result.timestamp).toBe('2026-03-14T10:00:00Z');
|
||||
});
|
||||
|
||||
test('skips partials by flag and by name, filters by tier', () => {
|
||||
// Newest by timestamp, but partial by flag under a shard dir.
|
||||
const shardDir = path.join(tmpDir, 'shards', 'skill-e2e-qa');
|
||||
fs.mkdirSync(shardDir, { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(shardDir, 'flagged.json'),
|
||||
JSON.stringify(makeResult({ timestamp: '2026-03-16T10:00:00Z', _partial: true })),
|
||||
);
|
||||
// Partial by name only.
|
||||
fs.writeFileSync(
|
||||
path.join(tmpDir, '_partial-e2e.json'),
|
||||
JSON.stringify(makeResult({ timestamp: '2026-03-15T10:00:00Z' })),
|
||||
);
|
||||
// Wrong tier.
|
||||
fs.writeFileSync(
|
||||
path.join(tmpDir, '0.3.6-main-llm-judge-20260317-100000.json'),
|
||||
JSON.stringify(makeResult({ tier: 'llm-judge', timestamp: '2026-03-17T10:00:00Z' })),
|
||||
);
|
||||
// The genuine baseline.
|
||||
fs.writeFileSync(
|
||||
path.join(tmpDir, '0.3.5-main-e2e-20260312-100000.json'),
|
||||
JSON.stringify(makeResult({ timestamp: '2026-03-12T10:00:00Z' })),
|
||||
);
|
||||
|
||||
const latest = findLatestFinalizedRun(tmpDir, 'e2e');
|
||||
expect(latest?.result.timestamp).toBe('2026-03-12T10:00:00Z');
|
||||
});
|
||||
|
||||
test('returns null for missing dir or no finalized runs', () => {
|
||||
expect(findLatestFinalizedRun('/nonexistent/path', 'e2e')).toBeNull();
|
||||
expect(findLatestFinalizedRun(tmpDir, 'e2e')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
// --- compareEvalResults tests ---
|
||||
@@ -524,6 +793,29 @@ describe('generateCommentary', () => {
|
||||
expect(notes.some(n => n.includes('No regressions'))).toBe(true);
|
||||
});
|
||||
|
||||
test('says NO BASELINE instead of "stable" when nothing matched the prior run', () => {
|
||||
// A baseline file existed but shares no test names (renamed/retired suite),
|
||||
// so zero tests were actually compared. Claiming stability here is a lie.
|
||||
const c: ComparisonResult = {
|
||||
before_file: 'a.json', after_file: 'b.json',
|
||||
before_branch: 'main', after_branch: 'main',
|
||||
before_timestamp: '', after_timestamp: '',
|
||||
deltas: [
|
||||
{ name: 'a', before: { passed: false, cost_usd: 0 }, after: { passed: true, cost_usd: 0.10 }, status_change: 'unchanged' },
|
||||
{ name: 'b', before: { passed: false, cost_usd: 0 }, after: { passed: true, cost_usd: 0.10 }, status_change: 'unchanged' },
|
||||
{ name: 'c', before: { passed: false, cost_usd: 0 }, after: { passed: true, cost_usd: 0.10 }, status_change: 'unchanged' },
|
||||
],
|
||||
total_cost_delta: 0.30, total_duration_delta: 0,
|
||||
improved: 0, regressed: 0, unchanged: 3,
|
||||
tool_count_before: 0, tool_count_after: 0,
|
||||
matched: 0,
|
||||
};
|
||||
|
||||
const notes = generateCommentary(c);
|
||||
expect(notes.some(n => n.includes('NO BASELINE'))).toBe(true);
|
||||
expect(notes.some(n => n.includes('Stable run'))).toBe(false);
|
||||
});
|
||||
|
||||
test('returns empty for stable run with no significant changes', () => {
|
||||
const c: ComparisonResult = {
|
||||
before_file: 'a.json', after_file: 'b.json',
|
||||
|
||||
+151
-23
@@ -39,7 +39,16 @@ export function getProjectEvalDir(): string {
|
||||
return LEGACY_EVAL_DIR;
|
||||
}
|
||||
|
||||
const DEFAULT_EVAL_DIR = getProjectEvalDir();
|
||||
/**
|
||||
* Lazy + memoized so importing this module never spawns the gstack-slug
|
||||
* subprocess. Callers that pass an explicit dir or set GSTACK_EVAL_DIR
|
||||
* (the sharded paid runner does, per shard) never pay for slug detection.
|
||||
*/
|
||||
let memoizedDefaultEvalDir: string | null = null;
|
||||
function defaultEvalDir(): string {
|
||||
if (memoizedDefaultEvalDir === null) memoizedDefaultEvalDir = getProjectEvalDir();
|
||||
return memoizedDefaultEvalDir;
|
||||
}
|
||||
|
||||
// --- Interfaces ---
|
||||
|
||||
@@ -104,6 +113,8 @@ export interface EvalResult {
|
||||
total_duration_ms: number;
|
||||
wall_clock_ms?: number; // wall-clock from collector creation to finalization (shows parallelism)
|
||||
tests: EvalTestEntry[];
|
||||
/** Shard slug when the run was collected under <evalDir>/shards/<slug>/. */
|
||||
shard?: string;
|
||||
_partial?: boolean; // true for incremental saves, absent in final
|
||||
}
|
||||
|
||||
@@ -131,10 +142,94 @@ export interface ComparisonResult {
|
||||
unchanged: number;
|
||||
tool_count_before: number;
|
||||
tool_count_after: number;
|
||||
/** After-tests that had a same-named entry in the before run. 0 = nothing was
|
||||
* actually compared, so no stability claim is warranted. */
|
||||
matched?: number;
|
||||
}
|
||||
|
||||
// --- Shared helpers ---
|
||||
|
||||
/**
|
||||
* Is this eval file an in-progress accumulator rather than a finalized run?
|
||||
*
|
||||
* True on either signal: the `_partial` flag inside the JSON (the authoritative
|
||||
* role marker) OR a filename starting with `_partial` (catches accumulators
|
||||
* whose body predates the flag, and flagged files that were renamed keep being
|
||||
* caught by the flag). Every baseline lookup must exclude these — an
|
||||
* accumulator carries the current run's tier, branch, and freshest timestamp,
|
||||
* so treating it as a baseline makes the run compare against itself.
|
||||
*/
|
||||
export function isPartialEval(data: unknown, filename: string): boolean {
|
||||
if (path.basename(filename).startsWith('_partial')) return true;
|
||||
return Boolean((data as { _partial?: unknown } | null)?._partial);
|
||||
}
|
||||
|
||||
/**
|
||||
* List eval JSON files in `evalDir` plus one level of `<evalDir>/shards/<slug>/`
|
||||
* subdirectories (where the sharded paid runner points each shard's collector).
|
||||
* Returns absolute paths. Missing dirs yield [].
|
||||
*/
|
||||
export function listEvalJsonFiles(evalDir: string): string[] {
|
||||
const jsonIn = (dir: string): string[] => {
|
||||
let names: string[];
|
||||
try {
|
||||
names = fs.readdirSync(dir);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
return names.filter(f => f.endsWith('.json')).map(f => path.join(dir, f));
|
||||
};
|
||||
|
||||
const files = jsonIn(evalDir);
|
||||
const shardsRoot = path.join(evalDir, 'shards');
|
||||
let shardDirs: fs.Dirent[];
|
||||
try {
|
||||
shardDirs = fs.readdirSync(shardsRoot, { withFileTypes: true });
|
||||
} catch {
|
||||
return files;
|
||||
}
|
||||
for (const entry of shardDirs) {
|
||||
if (!entry.isDirectory()) continue;
|
||||
files.push(...jsonIn(path.join(shardsRoot, entry.name)));
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
||||
/**
|
||||
* Shard slug for an eval dir: when the dir is directly under a `shards/`
|
||||
* directory (the sharded paid runner's per-shard GSTACK_EVAL_DIR layout),
|
||||
* the dir name is the slug; otherwise null.
|
||||
*/
|
||||
export function shardSlugOfEvalDir(evalDir: string): string | null {
|
||||
const normalized = path.resolve(evalDir);
|
||||
return path.basename(path.dirname(normalized)) === 'shards' ? path.basename(normalized) : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Find the most recent finalized (non-partial) eval file for a tier, scanning
|
||||
* `evalDir` and one level of `shards/<slug>/` subdirs. Shared by the budget
|
||||
* regression gate and any tooling that needs "the latest real run".
|
||||
*/
|
||||
export function findLatestFinalizedRun(
|
||||
evalDir: string,
|
||||
tier: 'e2e' | 'llm-judge',
|
||||
): { filepath: string; result: EvalResult } | null {
|
||||
let latest: { filepath: string; result: EvalResult; timestamp: string } | null = null;
|
||||
for (const filepath of listEvalJsonFiles(evalDir)) {
|
||||
let data: EvalResult;
|
||||
try {
|
||||
data = JSON.parse(fs.readFileSync(filepath, 'utf-8')) as EvalResult;
|
||||
} catch { continue; }
|
||||
if (isPartialEval(data, filepath)) continue;
|
||||
if (data.tier !== tier) continue;
|
||||
const timestamp = data.timestamp ?? '';
|
||||
if (!latest || timestamp.localeCompare(latest.timestamp) > 0) {
|
||||
latest = { filepath, result: data, timestamp };
|
||||
}
|
||||
}
|
||||
return latest ? { filepath: latest.filepath, result: latest.result } : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Determine if a planted-bug eval passed based on judge results vs ground truth thresholds.
|
||||
* Centralizes the pass/fail logic so all planted-bug tests use the same criteria.
|
||||
@@ -171,8 +266,16 @@ export function extractToolSummary(transcript: any[]): Record<string, number> {
|
||||
}
|
||||
|
||||
/**
|
||||
* Find the most recent prior eval file for comparison.
|
||||
* Prefers same branch, falls back to any branch.
|
||||
* Find the most recent prior COMPLETED eval file for comparison.
|
||||
* Scans the eval dir plus one level of `shards/<slug>/` subdirs. Prefers
|
||||
* same shard slug (a shard's own history over another shard's or the flat
|
||||
* dir's), then same branch, then falls back to anything.
|
||||
*
|
||||
* In-progress accumulators (`_partial: true`, written by savePartial after every
|
||||
* test) are never candidates: the current run's own partial carries the current
|
||||
* tier + branch and the freshest timestamp, so including it made every run
|
||||
* compare against itself and report "no regressions" unconditionally. The
|
||||
* exclusion is by role (the `_partial` flag), not by filename.
|
||||
*/
|
||||
export function findPreviousRun(
|
||||
evalDir: string,
|
||||
@@ -180,24 +283,22 @@ export function findPreviousRun(
|
||||
branch: string,
|
||||
excludeFile: string,
|
||||
): string | null {
|
||||
let files: string[];
|
||||
try {
|
||||
files = fs.readdirSync(evalDir).filter(f => f.endsWith('.json'));
|
||||
} catch {
|
||||
return null; // dir doesn't exist
|
||||
}
|
||||
|
||||
// Parse top-level fields from each file (cheap — no full tests array needed)
|
||||
const entries: Array<{ file: string; branch: string; timestamp: string }> = [];
|
||||
for (const file of files) {
|
||||
if (file === path.basename(excludeFile)) continue;
|
||||
const fullPath = path.join(evalDir, file);
|
||||
const entries: Array<{ file: string; branch: string; timestamp: string; shard: string | null }> = [];
|
||||
for (const fullPath of listEvalJsonFiles(evalDir)) {
|
||||
if (path.resolve(fullPath) === path.resolve(excludeFile)) continue;
|
||||
try {
|
||||
const raw = fs.readFileSync(fullPath, 'utf-8');
|
||||
// Quick parse — only grab the fields we need
|
||||
const data = JSON.parse(raw);
|
||||
if (isPartialEval(data, fullPath)) continue; // in-progress run, not a baseline
|
||||
if (data.tier !== tier) continue;
|
||||
entries.push({ file: fullPath, branch: data.branch || '', timestamp: data.timestamp || '' });
|
||||
entries.push({
|
||||
file: fullPath,
|
||||
branch: data.branch || '',
|
||||
timestamp: data.timestamp || '',
|
||||
shard: data.shard || shardSlugOfEvalDir(path.dirname(fullPath)),
|
||||
});
|
||||
} catch { continue; }
|
||||
}
|
||||
|
||||
@@ -206,11 +307,17 @@ export function findPreviousRun(
|
||||
// Sort by timestamp descending
|
||||
entries.sort((a, b) => b.timestamp.localeCompare(a.timestamp));
|
||||
|
||||
// Prefer same branch
|
||||
const sameBranch = entries.find(e => e.branch === branch);
|
||||
if (sameBranch) return sameBranch.file;
|
||||
|
||||
// Fallback: any branch
|
||||
// Prefer same shard slug (null = the flat dir), then same branch, then any.
|
||||
const targetShard = shardSlugOfEvalDir(path.dirname(excludeFile));
|
||||
const preferences: Array<(e: typeof entries[number]) => boolean> = [
|
||||
e => e.shard === targetShard && e.branch === branch,
|
||||
e => e.shard === targetShard,
|
||||
e => e.branch === branch,
|
||||
];
|
||||
for (const matches of preferences) {
|
||||
const hit = entries.find(matches);
|
||||
if (hit) return hit.file;
|
||||
}
|
||||
return entries[0].file;
|
||||
}
|
||||
|
||||
@@ -226,6 +333,7 @@ export function compareEvalResults(
|
||||
const deltas: TestDelta[] = [];
|
||||
let improved = 0, regressed = 0, unchanged = 0;
|
||||
let toolCountBefore = 0, toolCountAfter = 0;
|
||||
let matched = 0;
|
||||
|
||||
// Index before tests by name
|
||||
const beforeMap = new Map<string, EvalTestEntry>();
|
||||
@@ -246,6 +354,7 @@ export function compareEvalResults(
|
||||
|
||||
let statusChange: TestDelta['status_change'] = 'unchanged';
|
||||
if (beforeTest) {
|
||||
matched++;
|
||||
if (!beforeTest.passed && afterTest.passed) { statusChange = 'improved'; improved++; }
|
||||
else if (beforeTest.passed && !afterTest.passed) { statusChange = 'regressed'; regressed++; }
|
||||
else { unchanged++; }
|
||||
@@ -314,6 +423,7 @@ export function compareEvalResults(
|
||||
unchanged,
|
||||
tool_count_before: toolCountBefore,
|
||||
tool_count_after: toolCountAfter,
|
||||
matched,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -512,7 +622,17 @@ export function generateCommentary(c: ComparisonResult): string[] {
|
||||
}
|
||||
}
|
||||
|
||||
// 4. Overall summary
|
||||
// 4. No baseline — say so. A run with nothing to compare against must never
|
||||
// read as "stable"; silence or a false all-clear is worse than no output.
|
||||
if (c.matched === 0 && c.deltas.length > 0) {
|
||||
notes.push(
|
||||
`NO BASELINE: none of these ${c.deltas.length} test(s) appear in ${path.basename(c.before_file)}. ` +
|
||||
'Nothing was compared, so this run says nothing about regressions.',
|
||||
);
|
||||
return notes;
|
||||
}
|
||||
|
||||
// 5. Overall summary
|
||||
if (c.deltas.length >= 3 && regressions.length === 0) {
|
||||
const overallParts: string[] = [];
|
||||
|
||||
@@ -649,11 +769,13 @@ export class EvalCollector {
|
||||
private tests: EvalTestEntry[] = [];
|
||||
private finalized = false;
|
||||
private evalDir: string;
|
||||
private shard: string | null;
|
||||
private createdAt = Date.now();
|
||||
|
||||
constructor(tier: 'e2e' | 'llm-judge', evalDir?: string) {
|
||||
this.tier = tier;
|
||||
this.evalDir = evalDir || DEFAULT_EVAL_DIR;
|
||||
this.evalDir = evalDir || process.env.GSTACK_EVAL_DIR || defaultEvalDir();
|
||||
this.shard = shardSlugOfEvalDir(this.evalDir);
|
||||
}
|
||||
|
||||
addTest(entry: EvalTestEntry): void {
|
||||
@@ -684,6 +806,7 @@ export class EvalCollector {
|
||||
total_cost_usd: Math.round(totalCost * 100) / 100,
|
||||
total_duration_ms: totalDuration,
|
||||
tests: this.tests,
|
||||
...(this.shard ? { shard: this.shard } : {}),
|
||||
_partial: true,
|
||||
};
|
||||
|
||||
@@ -721,6 +844,7 @@ export class EvalCollector {
|
||||
total_duration_ms: totalDuration,
|
||||
wall_clock_ms: Date.now() - this.createdAt,
|
||||
tests: this.tests,
|
||||
...(this.shard ? { shard: this.shard } : {}),
|
||||
};
|
||||
|
||||
// Write eval file
|
||||
@@ -742,7 +866,11 @@ export class EvalCollector {
|
||||
const comparison = compareEvalResults(prevResult, result, prevFile, filepath);
|
||||
process.stderr.write(formatComparison(comparison) + '\n');
|
||||
} else {
|
||||
process.stderr.write('\nFirst run — no comparison available.\n');
|
||||
process.stderr.write(
|
||||
`\nNO BASELINE: no completed prior ${this.tier} run found in ${this.evalDir}` +
|
||||
' (the in-progress accumulator is not a baseline). Nothing compared —' +
|
||||
' this run says nothing about regressions.\n',
|
||||
);
|
||||
}
|
||||
} catch (err: any) {
|
||||
process.stderr.write(`\nCompare error: ${err.message}\n`);
|
||||
|
||||
@@ -8,8 +8,8 @@
|
||||
* Key differences from Codex session-runner:
|
||||
* - Uses `gemini -p` instead of `codex exec`
|
||||
* - Output is NDJSON with event types: init, message, tool_use, tool_result, result
|
||||
* - Uses `--output-format stream-json --yolo --skip-trust` instead of `--json -s read-only`
|
||||
* (`--skip-trust` required for headless/untrusted cwds; see gemini trusted-folders docs)
|
||||
* - Uses `--output-format stream-json --yolo` instead of `--json -s read-only`
|
||||
* (`--skip-trust` was removed in gemini-cli 0.34; folder trust is settings-driven now)
|
||||
* - No temp HOME needed — Gemini discovers skills from `.agents/skills/` in cwd
|
||||
* - Message events are streamed with `delta: true` — must concatenate
|
||||
*/
|
||||
@@ -121,10 +121,11 @@ export async function runGeminiSkill(opts: {
|
||||
};
|
||||
}
|
||||
|
||||
// Build gemini command
|
||||
// --skip-trust: headless/CI and temp cwds aren't in ~/.gemini/trustedFolders.json;
|
||||
// without it gemini exits FatalUntrustedWorkspaceError before any model call.
|
||||
const args = ['-p', prompt, '--output-format', 'stream-json', '--yolo', '--skip-trust'];
|
||||
// Build gemini command.
|
||||
// --skip-trust was REMOVED in gemini-cli 0.34 ("Unknown arguments:
|
||||
// skip-trust"); folder trust moved to settings and no longer needs a flag
|
||||
// for headless runs. --yolo still auto-approves tool actions.
|
||||
const args = ['-p', prompt, '--output-format', 'stream-json', '--yolo'];
|
||||
|
||||
// Spawn gemini — uses real HOME for auth (~/.gemini; HOME is allowlisted),
|
||||
// cwd for skill discovery. Hermetic scrub with gemini's auth surface
|
||||
@@ -198,6 +199,31 @@ export async function runGeminiSkill(opts: {
|
||||
process.stderr.write(` [gemini stderr] ${stderr.trim().slice(0, 200)}\n`);
|
||||
}
|
||||
|
||||
// Environment-unusable classification: these are Google-side conditions no
|
||||
// test assertion can act on — the deprecated individual code-assist auth
|
||||
// path ("migrate to the Antigravity suite") and argv drift on older/newer
|
||||
// CLIs. Return the same SKIP shape as binary-not-found so callers report
|
||||
// SKIPPED instead of a false FAIL.
|
||||
const unusableMarkers = [
|
||||
'no longer supported for Gemini Code Assist',
|
||||
'antigravity',
|
||||
'Unknown arguments: skip-trust',
|
||||
];
|
||||
if (exitCode !== 0 && parsed.tokens === 0) {
|
||||
const marker = unusableMarkers.find((m) => stderr.toLowerCase().includes(m.toLowerCase()));
|
||||
if (marker) {
|
||||
return {
|
||||
output: `SKIP: gemini CLI unusable (${marker})`,
|
||||
toolCalls: [],
|
||||
tokens: 0,
|
||||
exitCode: -1,
|
||||
durationMs,
|
||||
sessionId: null,
|
||||
rawLines: collectedLines,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
output: parsed.output,
|
||||
toolCalls: parsed.toolCalls,
|
||||
|
||||
@@ -36,7 +36,8 @@ import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import * as os from 'os';
|
||||
import { promotedEnv } from '../../lib/conductor-env-shim';
|
||||
import { isProcessAlive } from '../../browse/src/error-handling';
|
||||
import { isProcessAlive, safeUnlink } from '../../browse/src/error-handling';
|
||||
import { skillCensus, frontmatterName } from './skill-census';
|
||||
|
||||
/** Exact env names a hermetic child keeps. Everything not listed (or matched
|
||||
* by a prefix rule below) is dropped. */
|
||||
@@ -225,6 +226,73 @@ export function getHermeticDirs(): HermeticDirs {
|
||||
return cachedDirs;
|
||||
}
|
||||
|
||||
let cachedSkillsConfigDir: string | null = null;
|
||||
|
||||
/**
|
||||
* A hermetic CLAUDE_CONFIG_DIR with the repo's shipped skills REGISTERED in
|
||||
* user scope, mirroring ./setup's registration exactly: each discovered skill
|
||||
* gets a REAL directory `<configDir>/skills/<registryName>/` containing a
|
||||
* SYMLINK to that skill's SKILL.md (absolute path), plus a `sections/`
|
||||
* symlink when the skill has one. registryName is the frontmatter `name:`
|
||||
* (dir-name fallback), NO gstack- prefix; the root SKILL.md router registers
|
||||
* as `_gstack-command`. skillCensus().registryEntries is the authoritative
|
||||
* set of what must appear here.
|
||||
*
|
||||
* The default hermetic dir deliberately seeds no skills — correct for
|
||||
* children that install their own or probe setup behavior — but a PTY test
|
||||
* that TYPES `/office-hours` needs the slash command to exist, or claude
|
||||
* rejects it as Unknown command before any model turn and the gate measures
|
||||
* nothing. Separate dir, same runRoot: opt-in per session, never contaminates
|
||||
* the default-config children, and the existing exit teardown + pid-aware GC
|
||||
* cover it. Ends in `/.claude` for the same plan-path anchoring reason as
|
||||
* HermeticDirs.configDir.
|
||||
*
|
||||
* Two intentional non-hermetic edges:
|
||||
* - Seeding reads the LIVE repo tree BY DESIGN — the skills ARE the subject
|
||||
* under test; a snapshot would measure stale copies.
|
||||
* - HOME is not hermeticized, so the ~64 absolute
|
||||
* `~/.claude/skills/gstack/...` preamble references inside each SKILL.md
|
||||
* still resolve to the operator install (same limitation as CI).
|
||||
*/
|
||||
export function hermeticSkillsConfigDir(): string {
|
||||
if (cachedSkillsConfigDir) return cachedSkillsConfigDir;
|
||||
const { runRoot } = getHermeticDirs();
|
||||
const configDir = path.join(runRoot, 'with-skills', '.claude');
|
||||
const skillsDir = path.join(configDir, 'skills');
|
||||
fs.mkdirSync(skillsDir, { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(configDir, '.claude.json'),
|
||||
JSON.stringify(buildSeedConfig({
|
||||
apiKey: process.env.ANTHROPIC_API_KEY ?? process.env.GSTACK_ANTHROPIC_API_KEY,
|
||||
trustedDirs: [repoRoot()],
|
||||
}), null, 2),
|
||||
);
|
||||
const root = repoRoot();
|
||||
for (const rel of skillCensus(root).physicalSkillFiles) {
|
||||
const skillMd = path.join(root, rel);
|
||||
const skillDir = path.dirname(rel);
|
||||
const registryName = rel === 'SKILL.md'
|
||||
? '_gstack-command'
|
||||
: frontmatterName(skillMd) || skillDir;
|
||||
const target = path.join(skillsDir, registryName);
|
||||
// Idempotent overwrite mirrors setup's re-link: connect-chrome (a dir
|
||||
// symlink to open-gstack-browser) shares its target's frontmatter name,
|
||||
// so the two walk entries collapse to one registry dir.
|
||||
fs.mkdirSync(target, { recursive: true });
|
||||
safeUnlink(path.join(target, 'SKILL.md'));
|
||||
fs.symlinkSync(skillMd, path.join(target, 'SKILL.md'));
|
||||
if (rel !== 'SKILL.md') {
|
||||
const sections = path.join(root, skillDir, 'sections');
|
||||
if (fs.existsSync(sections)) {
|
||||
safeUnlink(path.join(target, 'sections'));
|
||||
fs.symlinkSync(sections, path.join(target, 'sections'));
|
||||
}
|
||||
}
|
||||
}
|
||||
cachedSkillsConfigDir = configDir;
|
||||
return configDir;
|
||||
}
|
||||
|
||||
/** A dir younger than this is never GC'd even if its pid looks dead — guards
|
||||
* against PID reuse deleting a freshly-created dir whose original pid exited
|
||||
* and was recycled to an unrelated live process between create and GC. */
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
/**
|
||||
* The ONE definition of which test files are paid (API spend, external
|
||||
* services, e2e harnesses). package.json's test:gate/test:evals globs, the
|
||||
* free-suite exclusion in scripts/test-free-shards.ts, and the sharded paid
|
||||
* runner in scripts/test-paid-shards.ts all derive from this list — a file
|
||||
* added to one and not the others either burns money in the free suite or
|
||||
* silently never runs in the paid tier.
|
||||
*/
|
||||
|
||||
import { matchGlob } from './touchfiles';
|
||||
|
||||
/** The exact globs package.json's `test:gate` passes to `bun test`. */
|
||||
export const PAID_TEST_GLOBS = [
|
||||
'test/skill-llm-eval.test.ts',
|
||||
'test/skill-e2e-*.test.ts',
|
||||
'test/skill-routing-e2e.test.ts',
|
||||
'test/codex-e2e.test.ts',
|
||||
'test/gemini-e2e.test.ts',
|
||||
] as const;
|
||||
|
||||
/** True when a repo-relative path (either slash style) is a paid test file. */
|
||||
export function isPaidTestFile(relativePath: string): boolean {
|
||||
const normalized = relativePath.replace(/\\/g, '/');
|
||||
return PAID_TEST_GLOBS.some((glob) => matchGlob(normalized, glob));
|
||||
}
|
||||
@@ -103,10 +103,10 @@ export function resultFromGeminiStream(
|
||||
* Headless flags always passed:
|
||||
* --output-format stream-json — NDJSON events (message/tool_use/result)
|
||||
* --yolo — auto-approve tools (non-interactive)
|
||||
* --skip-trust — trust cwd for this session; required when
|
||||
* workdir is a temp/untrusted folder (benchmarks
|
||||
* use mkdtemp). Without it headless gemini exits
|
||||
* before calling the model.
|
||||
*
|
||||
* --skip-trust is gone: gemini-cli 0.34 removed the flag ("Unknown arguments:
|
||||
* skip-trust") — folder trust is settings-driven now and headless runs no
|
||||
* longer need a flag for temp workdirs.
|
||||
*/
|
||||
export class GeminiAdapter implements ProviderAdapter {
|
||||
readonly name = 'gemini';
|
||||
@@ -136,10 +136,9 @@ export class GeminiAdapter implements ProviderAdapter {
|
||||
async run(opts: RunOpts): Promise<RunResult> {
|
||||
const start = Date.now();
|
||||
// Default to --yolo (non-interactive) and stream-json output so we can parse
|
||||
// tokens + tool calls. --skip-trust is required for headless/temp workdirs
|
||||
// (gemini CLI otherwise exits: "not running in a trusted directory").
|
||||
// Callers can override via extraArgs.
|
||||
const args = ['-p', opts.prompt, '--output-format', 'stream-json', '--yolo', '--skip-trust'];
|
||||
// tokens + tool calls. Callers can override via extraArgs. (--skip-trust was
|
||||
// removed in gemini-cli 0.34; passing it errors at argv parse.)
|
||||
const args = ['-p', opts.prompt, '--output-format', 'stream-json', '--yolo'];
|
||||
if (opts.model) args.push('--model', opts.model);
|
||||
if (opts.extraArgs) args.push(...opts.extraArgs);
|
||||
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
/**
|
||||
* Shared skill census — the ONE place that counts skills (C11).
|
||||
*
|
||||
* Three consumers (hermetic seeding, context-bill ground truth, catalog-budget
|
||||
* test) each need a DIFFERENT count of "the skills", and hand-rolled walks
|
||||
* encode the wrong one somewhere. The counts diverge because of two facts:
|
||||
*
|
||||
* 1. `connect-chrome/` is a directory SYMLINK to `open-gstack-browser/`.
|
||||
* `Dirent.isDirectory()` is false for it (scripts/discover-skills.ts
|
||||
* skips it) while setup's trailing-slash shell glob follows it.
|
||||
* 2. The root `SKILL.md` is a router, registered by `setup` under the
|
||||
* alias name `_gstack-command`, not as an authored skill.
|
||||
*
|
||||
* So: use `physicalSkillFiles` when you mean "every SKILL.md a filesystem
|
||||
* walk can reach" (context-bill's walker), `authoredSkills` when you mean
|
||||
* "distinct skills a human maintains" (catalog budget), and
|
||||
* `registryEntries` when you mean "what a host discovers after ./setup"
|
||||
* (hermetic seeding must mirror this exactly).
|
||||
*/
|
||||
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
|
||||
const SKIP = new Set(['node_modules', '.git', 'dist']);
|
||||
|
||||
export interface SkillCensus {
|
||||
/** Every reachable `<dir>/SKILL.md` (symlinked dirs INCLUDED) plus the
|
||||
* root `SKILL.md` router. Relative paths from root. */
|
||||
physicalSkillFiles: string[];
|
||||
/** Distinct authored skills: symlink-deduped by realpath, root router
|
||||
* EXCLUDED. Each entry is the canonical directory name. */
|
||||
authoredSkills: string[];
|
||||
/** What `setup` registers into `~/.claude/skills/` with prefixing off:
|
||||
* one entry per unique frontmatter `name:` (falling back to dir name),
|
||||
* plus the `_gstack-command` root alias when the root router exists.
|
||||
* Symlinked dirs collapse here because they share a frontmatter name. */
|
||||
registryEntries: string[];
|
||||
}
|
||||
|
||||
/** First `name:` from SKILL.md frontmatter, mirroring `setup`'s
|
||||
* `grep -m1 '^name:'` (whitespace stripped). Empty string if absent. */
|
||||
export function frontmatterName(skillMdPath: string): string {
|
||||
let body: string;
|
||||
try {
|
||||
body = fs.readFileSync(skillMdPath, 'utf-8');
|
||||
} catch {
|
||||
return '';
|
||||
}
|
||||
const m = body.match(/^name:\s*(.+)$/m);
|
||||
return m ? m[1].replace(/\s+/g, '') : '';
|
||||
}
|
||||
|
||||
export function skillCensus(root: string): SkillCensus {
|
||||
const physicalSkillFiles: string[] = [];
|
||||
const authoredByRealpath = new Map<string, string>();
|
||||
const registrySet = new Set<string>();
|
||||
|
||||
if (fs.existsSync(path.join(root, 'SKILL.md'))) {
|
||||
physicalSkillFiles.push('SKILL.md');
|
||||
registrySet.add('_gstack-command');
|
||||
}
|
||||
|
||||
for (const entry of fs.readdirSync(root, { withFileTypes: true })) {
|
||||
if (entry.name.startsWith('.') || SKIP.has(entry.name)) continue;
|
||||
const dirPath = path.join(root, entry.name);
|
||||
// Follow directory symlinks like setup's shell glob does; Dirent alone
|
||||
// reports symlinks as non-directories.
|
||||
let isDir = entry.isDirectory();
|
||||
if (!isDir && entry.isSymbolicLink()) {
|
||||
try {
|
||||
isDir = fs.statSync(dirPath).isDirectory();
|
||||
} catch {
|
||||
continue; // dangling symlink
|
||||
}
|
||||
}
|
||||
if (!isDir) continue;
|
||||
|
||||
const skillMd = path.join(dirPath, 'SKILL.md');
|
||||
if (!fs.existsSync(skillMd)) continue;
|
||||
|
||||
physicalSkillFiles.push(`${entry.name}/SKILL.md`);
|
||||
|
||||
const real = fs.realpathSync(dirPath);
|
||||
if (!authoredByRealpath.has(real)) {
|
||||
authoredByRealpath.set(real, path.basename(real));
|
||||
}
|
||||
|
||||
registrySet.add(frontmatterName(skillMd) || entry.name);
|
||||
}
|
||||
|
||||
return {
|
||||
physicalSkillFiles: physicalSkillFiles.sort(),
|
||||
authoredSkills: [...authoredByRealpath.values()].sort(),
|
||||
registryEntries: [...registrySet].sort(),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
/**
|
||||
* Unit tests for hermeticSkillsConfigDir() — the opt-in hermetic config dir
|
||||
* that registers the repo's shipped skills for PTY slash-command children.
|
||||
* Free tier — no API calls; exercises the real seeder against the live repo
|
||||
* tree (that's the seeder's contract: the skills ARE the subject under test).
|
||||
*
|
||||
* Pins four contracts:
|
||||
* 1. The seeded dir is a valid CLAUDE_CONFIG_DIR (.claude.json present,
|
||||
* /.claude suffix, under the hermetic runRoot).
|
||||
* 2. Registration mirrors ./setup exactly: one entry per
|
||||
* skillCensus().registryEntries, each a REAL dir with a SKILL.md symlink
|
||||
* resolving to a real file (plus sections/ when the skill has one).
|
||||
* 3. connect-chrome (dir symlink) collapses into open-gstack-browser — no
|
||||
* duplicate, no connect-chrome entry.
|
||||
* 4. Per-process idempotence: the second call returns the cached dir.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import {
|
||||
hermeticSkillsConfigDir,
|
||||
getHermeticDirs,
|
||||
buildSeedConfig,
|
||||
} from './helpers/hermetic-env';
|
||||
import { skillCensus } from './helpers/skill-census';
|
||||
|
||||
const ROOT = path.resolve(__dirname, '..');
|
||||
const configDir = hermeticSkillsConfigDir();
|
||||
const skillsDir = path.join(configDir, 'skills');
|
||||
|
||||
describe('hermeticSkillsConfigDir', () => {
|
||||
test('seeded dir contains .claude.json and ends in /.claude under runRoot', () => {
|
||||
expect(fs.existsSync(path.join(configDir, '.claude.json'))).toBe(true);
|
||||
expect(path.basename(configDir)).toBe('.claude');
|
||||
expect(configDir.startsWith(getHermeticDirs().runRoot + path.sep)).toBe(true);
|
||||
});
|
||||
|
||||
test('one registry entry per skillCensus registryEntries, nothing extra', () => {
|
||||
const seeded = fs.readdirSync(skillsDir).sort();
|
||||
expect(seeded).toEqual(skillCensus(ROOT).registryEntries);
|
||||
});
|
||||
|
||||
test('every SKILL.md is a symlink resolving to a real file', () => {
|
||||
for (const entry of fs.readdirSync(skillsDir)) {
|
||||
const link = path.join(skillsDir, entry, 'SKILL.md');
|
||||
expect(fs.lstatSync(link).isSymbolicLink()).toBe(true);
|
||||
expect(fs.statSync(link).isFile()).toBe(true); // follows the link
|
||||
}
|
||||
});
|
||||
|
||||
test('sections/ symlink registered for skills that ship one', () => {
|
||||
// ship/ is a carved skill with a sections/ dir — the registered entry
|
||||
// must expose it or runtime "Read sections/<name>.md" 404s.
|
||||
expect(fs.existsSync(path.join(ROOT, 'ship', 'sections'))).toBe(true);
|
||||
const link = path.join(skillsDir, 'ship', 'sections');
|
||||
expect(fs.lstatSync(link).isSymbolicLink()).toBe(true);
|
||||
expect(fs.statSync(link).isDirectory()).toBe(true);
|
||||
});
|
||||
|
||||
test('connect-chrome collapses into a single open-gstack-browser entry', () => {
|
||||
const seeded = fs.readdirSync(skillsDir);
|
||||
expect(seeded.filter((n) => n === 'open-gstack-browser')).toHaveLength(1);
|
||||
expect(seeded).not.toContain('connect-chrome');
|
||||
});
|
||||
|
||||
test('root router registered as _gstack-command pointing at the root SKILL.md', () => {
|
||||
const link = path.join(skillsDir, '_gstack-command', 'SKILL.md');
|
||||
expect(fs.realpathSync(link)).toBe(fs.realpathSync(path.join(ROOT, 'SKILL.md')));
|
||||
});
|
||||
|
||||
test('second call returns the cached dir', () => {
|
||||
expect(hermeticSkillsConfigDir()).toBe(configDir);
|
||||
});
|
||||
|
||||
test('buildSeedConfig with undefined apiKey omits customApiKeyResponses', () => {
|
||||
// The seeder passes process.env keys straight through; when the operator
|
||||
// has no key exported the seed must stay valid (child fails auth later,
|
||||
// not here).
|
||||
const seed = buildSeedConfig({ apiKey: undefined, trustedDirs: [ROOT] });
|
||||
expect(seed).not.toHaveProperty('customApiKeyResponses');
|
||||
expect(seed.hasCompletedOnboarding).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -16,6 +16,8 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import * as os from 'os';
|
||||
import { getHermeticDirs, hermeticSkillsConfigDir } from './helpers/hermetic-env';
|
||||
|
||||
const ROOT = path.resolve(new URL(import.meta.url).pathname, '..', '..');
|
||||
|
||||
@@ -110,4 +112,23 @@ describe('hermetic wiring tripwire', () => {
|
||||
offenders.join(', '),
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
test('skill seeding stays under runRoot and reads the live repo tree, never operator ~/.claude', () => {
|
||||
// hermeticSkillsConfigDir() is a BLESSED non-hermetic edge: it registers
|
||||
// the LIVE repo tree's skills (the skills are the subject under test).
|
||||
// What it must never do is hand children the operator's ~/.claude — the
|
||||
// seeded CLAUDE_CONFIG_DIR lives under the hermetic runRoot, and every
|
||||
// registered symlink resolves into the repo checkout.
|
||||
const configDir = hermeticSkillsConfigDir();
|
||||
const { runRoot } = getHermeticDirs();
|
||||
const operatorClaude = path.join(os.homedir(), '.claude') + path.sep;
|
||||
expect(configDir.startsWith(runRoot + path.sep)).toBe(true);
|
||||
expect(configDir.startsWith(operatorClaude)).toBe(false);
|
||||
const skillsDir = path.join(configDir, 'skills');
|
||||
for (const entry of fs.readdirSync(skillsDir)) {
|
||||
const target = fs.readlinkSync(path.join(skillsDir, entry, 'SKILL.md'));
|
||||
expect(target.startsWith(operatorClaude), `${entry}: symlink escapes to ${target}`).toBe(false);
|
||||
expect(fs.realpathSync(target).startsWith(fs.realpathSync(ROOT) + path.sep), `${entry}: symlink outside repo: ${target}`).toBe(true);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,117 @@
|
||||
/**
|
||||
* Pins the paid-tier sharded runner (scripts/test-paid-shards.ts).
|
||||
*
|
||||
* Two properties matter, and both are why `test:gate` has never finished a run:
|
||||
* 1. Enumeration + sharding — every file `test:gate`'s globs expand to gets
|
||||
* its own process, and tier exclusion only ever fires on explicit evidence.
|
||||
* 2. A spinning shard is killed externally and the run CONTINUES. The fake
|
||||
* command here is a real busy loop, so an in-process timer could not save
|
||||
* it — exactly the failure mode `sample` caught on the wedged run.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
PAID_TEST_GLOBS,
|
||||
classifyPaidTestFile,
|
||||
collectPaidTestFiles,
|
||||
isPaidTestFile,
|
||||
planPaidShards,
|
||||
runPaidShards,
|
||||
summarize,
|
||||
type ShardOutcome,
|
||||
} from '../scripts/test-paid-shards';
|
||||
|
||||
describe('paid test enumeration', () => {
|
||||
test('matches the globs package.json test:gate expands', () => {
|
||||
expect(isPaidTestFile('test/skill-e2e-qa-workflow.test.ts')).toBe(true);
|
||||
expect(isPaidTestFile('test/skill-llm-eval.test.ts')).toBe(true);
|
||||
expect(isPaidTestFile('test/codex-e2e.test.ts')).toBe(true);
|
||||
expect(isPaidTestFile('test/skill-e2e-triage-audit.test.ts')).toBe(true);
|
||||
// Outside the globs: no dash, extra suffix, or a free test.
|
||||
expect(isPaidTestFile('test/skill-e2e.test.ts')).toBe(false);
|
||||
expect(isPaidTestFile('test/codex-e2e-recommendation-substance.test.ts')).toBe(false);
|
||||
expect(isPaidTestFile('test/paid-shards.test.ts')).toBe(false);
|
||||
});
|
||||
|
||||
test('discovers files and gives each one its own shard', () => {
|
||||
const files = collectPaidTestFiles();
|
||||
expect(files.length).toBeGreaterThan(0);
|
||||
expect(files.every(isPaidTestFile)).toBe(true);
|
||||
expect(PAID_TEST_GLOBS.length).toBe(5);
|
||||
|
||||
const shards = planPaidShards(files);
|
||||
expect(shards.flat().sort()).toEqual([...files].sort());
|
||||
expect(shards.every((shard) => shard.length === 1)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('tier classification', () => {
|
||||
test('excludes only on an explicit other-tier guard', () => {
|
||||
const gateGuard = "const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'gate';";
|
||||
const periodicGuard = "const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'periodic';";
|
||||
|
||||
expect(classifyPaidTestFile(gateGuard, 'gate').included).toBe(true);
|
||||
expect(classifyPaidTestFile(periodicGuard, 'gate').included).toBe(false);
|
||||
expect(classifyPaidTestFile(gateGuard, 'periodic').included).toBe(false);
|
||||
expect(classifyPaidTestFile(periodicGuard, 'periodic').included).toBe(true);
|
||||
});
|
||||
|
||||
test('keeps files whose tier is decided per-test at runtime', () => {
|
||||
// Naming an E2E_TIERS key is not evidence — 'retro' appears in the
|
||||
// LLM-judge file, which test:gate does run.
|
||||
const noGuard = "runSkillTest('retro', async () => {});";
|
||||
expect(classifyPaidTestFile(noGuard, 'gate').included).toBe(true);
|
||||
expect(classifyPaidTestFile(noGuard, 'periodic').included).toBe(true);
|
||||
expect(classifyPaidTestFile('', 'gate').included).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('shard execution', () => {
|
||||
const BUSY_LOOP = 'const end = Date.now() + 600000; while (Date.now() < end) {}';
|
||||
|
||||
const commandFor = (files: string[]) => {
|
||||
if (files[0] === 'spin') return { command: process.execPath, args: ['-e', BUSY_LOOP] };
|
||||
if (files[0] === 'fail') return { command: process.execPath, args: ['-e', 'process.exit(3)'] };
|
||||
return { command: process.execPath, args: ['-e', 'console.log("ok")'] };
|
||||
};
|
||||
|
||||
test('a spinning shard times out, is killed, and the run continues', async () => {
|
||||
const lines: string[] = [];
|
||||
const summary = await runPaidShards([['spin'], ['fail'], ['pass']], {
|
||||
timeoutMs: 1_200,
|
||||
jobs: 1,
|
||||
commandFor,
|
||||
log: (line) => lines.push(line),
|
||||
});
|
||||
|
||||
const byName = (name: string) => summary.outcomes.find((o) => o.files[0] === name) as ShardOutcome;
|
||||
expect(byName('spin').status).toBe('timed-out');
|
||||
expect(byName('fail').status).toBe('failed');
|
||||
expect(byName('pass').status).toBe('passed');
|
||||
|
||||
// The run never aborted: every shard reports, none is 'never-started'.
|
||||
expect(summary).toMatchObject({
|
||||
total: 3, executed: 3, passed: 1, failed: 1, timedOut: 1, neverStarted: 0,
|
||||
});
|
||||
|
||||
// The spinner was killed at the deadline, not left to burn a core.
|
||||
expect(byName('spin').elapsedMs).toBeLessThan(30_000);
|
||||
expect(byName('spin').groupPid).toBeGreaterThan(0);
|
||||
if (process.platform !== 'win32') {
|
||||
expect(() => process.kill(byName('spin').groupPid as number, 0)).toThrow();
|
||||
}
|
||||
|
||||
// Heartbeat: a START and a terminal line per shard, with elapsed seconds.
|
||||
expect(lines.filter((l) => l.includes(' START ')).length).toBe(3);
|
||||
expect(lines.some((l) => /TIMED-OUT in \d+s/.test(l))).toBe(true);
|
||||
expect(lines.some((l) => /PASSED in \d+s/.test(l))).toBe(true);
|
||||
}, 30_000);
|
||||
|
||||
test('summarize reports shards that never ran', () => {
|
||||
const summary = summarize([
|
||||
{ shard: 1, files: ['a'], status: 'passed', exitCode: 0, elapsedMs: 1, groupPid: 1 },
|
||||
{ shard: 2, files: ['b'], status: 'never-started', exitCode: null, elapsedMs: 0, groupPid: null },
|
||||
]);
|
||||
expect(summary).toMatchObject({ total: 2, executed: 1, passed: 1, neverStarted: 1 });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,67 @@
|
||||
/**
|
||||
* Pins single-logical-line AskUserQuestion detection (the plan-design-with-ui
|
||||
* gate-timeout class). When the PTY reflows a boxed AskUserQuestion, ALL
|
||||
* options land on ONE logical line after stripAnsi — the per-line option
|
||||
* parser then finds only option 1 and the >= 2 check fails forever while the
|
||||
* (correct) question sits on screen. Both fixture strings below are condensed
|
||||
* from REAL observed failure buffers of
|
||||
* test/skill-e2e-plan-design-with-ui.test.ts.
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from 'bun:test';
|
||||
import {
|
||||
isNumberedOptionListVisible,
|
||||
parseNumberedOptions,
|
||||
stripPtyResidue,
|
||||
} from './helpers/claude-pty-runner';
|
||||
|
||||
// Condensed from the 2026-08-13 failure buffer: whole AskUserQuestion on one
|
||||
// logical line — dividers, cursor option, options 2-5, footer. Note the
|
||||
// missing spaces ("2.Planordesigndoc") from stripped cursor-positioning
|
||||
// escapes.
|
||||
const SINGLE_LINE_QUESTION =
|
||||
'────────────Planning: /tmp/x/.claude/plans/soft-questing-jellyfish.md──────────' +
|
||||
' ☐ Review target What should I review? <gstack-qid:plan-design-review-scope-gate>' +
|
||||
'❯1.Branch diff (current WIP) Review the design implications of what changed. ' +
|
||||
'Recommendation: A when a branch diff exists.2.Planordesigndoc Paste or point me to a plan file. ' +
|
||||
'3. Spcific page, file, r pah Name a specific file. 4. Type something.' +
|
||||
'────────────5.ChataboutthisEnter to select · ↑/↓ o navigate · Escto cancel';
|
||||
|
||||
// The same AskUserQuestion with DEC cursor-visibility residue + spinner
|
||||
// frames interleaved, as captured before stripPtyResidue existed.
|
||||
const RESIDUE_QUESTION =
|
||||
'[?25l✻Sprouting…[?25h[?25l✶[?25h' + SINGLE_LINE_QUESTION.slice(0, 200) +
|
||||
'[?25l·still thinking[?25h' + SINGLE_LINE_QUESTION.slice(200);
|
||||
|
||||
describe('single-logical-line AskUserQuestion detection', () => {
|
||||
it('isNumberedOptionListVisible matches the reflowed one-line AskUserQuestion', () => {
|
||||
expect(isNumberedOptionListVisible(SINGLE_LINE_QUESTION)).toBe(true);
|
||||
});
|
||||
|
||||
it('parseNumberedOptions finds the full ascending option run on one line', () => {
|
||||
const options = parseNumberedOptions(SINGLE_LINE_QUESTION);
|
||||
expect(options.length).toBeGreaterThanOrEqual(2);
|
||||
expect(options[0]).toEqual({ index: 1, label: expect.stringContaining('Branch diff') });
|
||||
expect(options[1]?.index).toBe(2);
|
||||
});
|
||||
|
||||
it('survives DEC residue + spinner interleave', () => {
|
||||
expect(isNumberedOptionListVisible(RESIDUE_QUESTION)).toBe(true);
|
||||
expect(parseNumberedOptions(RESIDUE_QUESTION).length).toBeGreaterThanOrEqual(2);
|
||||
});
|
||||
|
||||
it('stripPtyResidue removes cursor-visibility fragments only', () => {
|
||||
expect(stripPtyResidue('[?25la[?25hb')).toBe('ab');
|
||||
expect(stripPtyResidue('keep [?250] this')).toBe('keep [?250] this');
|
||||
});
|
||||
|
||||
it('still rejects prose numbered lists (no cursor sigil)', () => {
|
||||
expect(parseNumberedOptions('steps: 1. Read the file 2. Edit it 3. Done')).toEqual([]);
|
||||
});
|
||||
|
||||
it('still parses classic multi-line AskUserQuestions', () => {
|
||||
const multiLine = 'What should I do?\n❯ 1. First option\n 2. Second option\n 3. Third option\n';
|
||||
const options = parseNumberedOptions(multiLine);
|
||||
expect(options.map((o) => o.index)).toEqual([1, 2, 3]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,77 @@
|
||||
/**
|
||||
* Static-grep tripwire for PTY slash-command skill seeding. Free tier — no API.
|
||||
*
|
||||
* The default hermetic config dir registers NO skills, so a PTY test that
|
||||
* TYPES a /skill slash command against it gets "Unknown command" before any
|
||||
* model turn — the test still runs, still spends money, and measures nothing.
|
||||
* Every test file that sends a slash command over the PTY must therefore
|
||||
* either route through a runPlanSkill* helper (which opts in for you) or pass
|
||||
* `seedSkills: true` in its own launchClaudePty options.
|
||||
*
|
||||
* Pattern mirrors test/hermetic-wiring.test.ts: read sources as text, assert
|
||||
* invariants on their contents. Brittle by design — changing the seeding
|
||||
* wiring must force the author to look here.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
|
||||
const ROOT = path.resolve(new URL(import.meta.url).pathname, '..', '..');
|
||||
|
||||
/** A PTY send whose payload starts with a slash command (`/name`, optionally
|
||||
* followed by `\r`, whitespace, or the closing quote). A second slash right
|
||||
* after the name (a file path like '/tmp/x') does NOT match. */
|
||||
const SLASH_SEND = /\.send\(\s*(['"`])\/[a-z][a-z0-9-]*(\\r|\s|\1)/;
|
||||
|
||||
const RUN_PLAN_HELPER = /\brunPlanSkill(Observation|Counting|FloorCheck)\s*\(/;
|
||||
|
||||
function testFiles(dir: string): string[] {
|
||||
const out: string[] = [];
|
||||
for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
|
||||
const full = path.join(dir, entry.name);
|
||||
if (entry.isDirectory()) out.push(...testFiles(full));
|
||||
else if (entry.name.endsWith('.test.ts')) out.push(full);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
describe('PTY skill-seeding tripwire', () => {
|
||||
test('every slash-command PTY test seeds skills (helper or seedSkills: true)', () => {
|
||||
const offenders: string[] = [];
|
||||
for (const full of testFiles(path.join(ROOT, 'test'))) {
|
||||
if (path.basename(full) === 'pty-skill-seeding-wiring.test.ts') continue;
|
||||
const src = fs.readFileSync(full, 'utf-8');
|
||||
const lines = src.split('\n');
|
||||
const sendLine = lines.findIndex((l) => SLASH_SEND.test(l));
|
||||
if (sendLine === -1) continue;
|
||||
if (RUN_PLAN_HELPER.test(src)) continue;
|
||||
if (src.includes('seedSkills: true')) continue;
|
||||
offenders.push(`${path.relative(ROOT, full)}:${sendLine + 1}`);
|
||||
}
|
||||
expect(
|
||||
offenders,
|
||||
'These tests type a /skill slash command into a hermetic PTY child that has ' +
|
||||
'no skills registered — claude rejects it as Unknown command and the test ' +
|
||||
'measures nothing. Pass seedSkills: true to launchClaudePty (or route ' +
|
||||
'through a runPlanSkill* helper): ' + offenders.join(', '),
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
test('the runPlanSkill* helpers all opt in via seedSkills: true', () => {
|
||||
// The helper family types slash commands on behalf of ~20 test files;
|
||||
// dropping the opt-in there silently un-measures all of them at once.
|
||||
const src = fs.readFileSync(path.join(ROOT, 'test/helpers/claude-pty-runner.ts'), 'utf-8');
|
||||
const optIns = src.match(/seedSkills: true/g) ?? [];
|
||||
expect(optIns.length).toBeGreaterThanOrEqual(3);
|
||||
});
|
||||
|
||||
test('launchClaudePty wires seedSkills to hermeticSkillsConfigDir()', () => {
|
||||
const src = fs.readFileSync(path.join(ROOT, 'test/helpers/claude-pty-runner.ts'), 'utf-8');
|
||||
// Gated on hermetic (EVALS_HERMETIC=0 must keep the operator config) and
|
||||
// on the per-test env override (explicit CLAUDE_CONFIG_DIR wins).
|
||||
const wired =
|
||||
/if\s*\(opts\.seedSkills && hermetic && !opts\.env\?\.CLAUDE_CONFIG_DIR\)\s*\{\s*\n\s*childEnv\.CLAUDE_CONFIG_DIR = hermeticSkillsConfigDir\(\);/.test(src);
|
||||
expect(wired, 'launchClaudePty must set CLAUDE_CONFIG_DIR from hermeticSkillsConfigDir() when seedSkills && hermetic && no per-test override').toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -27,10 +27,10 @@
|
||||
import { describe, test } from 'bun:test';
|
||||
import { spawnSync } from 'child_process';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import {
|
||||
getProjectEvalDir,
|
||||
findPreviousRun,
|
||||
findLatestFinalizedRun,
|
||||
compareEvalResults,
|
||||
assertNoBudgetRegression,
|
||||
type EvalResult,
|
||||
@@ -72,42 +72,9 @@ function currentGitBranch(): string {
|
||||
}
|
||||
}
|
||||
|
||||
interface LatestRun {
|
||||
filepath: string;
|
||||
result: EvalResult;
|
||||
}
|
||||
|
||||
/** Find the most recent finalized (non-_partial) eval file for a tier. */
|
||||
function findLatestRun(evalDir: string, tier: 'e2e' | 'llm-judge'): LatestRun | null {
|
||||
let entries: string[];
|
||||
try {
|
||||
entries = fs.readdirSync(evalDir);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
const candidates: Array<{ filepath: string; timestamp: string }> = [];
|
||||
for (const f of entries) {
|
||||
if (!f.endsWith('.json')) continue;
|
||||
if (f.startsWith('_partial')) continue;
|
||||
const fullPath = path.join(evalDir, f);
|
||||
try {
|
||||
const data = JSON.parse(fs.readFileSync(fullPath, 'utf-8')) as EvalResult;
|
||||
if (data.tier !== tier) continue;
|
||||
candidates.push({ filepath: fullPath, timestamp: data.timestamp ?? '' });
|
||||
} catch { /* ignore corrupt */ }
|
||||
}
|
||||
if (candidates.length === 0) return null;
|
||||
candidates.sort((a, b) => b.timestamp.localeCompare(a.timestamp));
|
||||
const top = candidates[0]!;
|
||||
return {
|
||||
filepath: top.filepath,
|
||||
result: JSON.parse(fs.readFileSync(top.filepath, 'utf-8')) as EvalResult,
|
||||
};
|
||||
}
|
||||
|
||||
function checkTier(tier: 'e2e' | 'llm-judge'): void {
|
||||
const evalDir = getProjectEvalDir();
|
||||
const latest = findLatestRun(evalDir, tier);
|
||||
const latest = findLatestFinalizedRun(evalDir, tier);
|
||||
if (!latest) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`[budget-regression:${tier}] no current run in ${evalDir} — skipping`);
|
||||
@@ -165,7 +132,7 @@ function checkTier(tier: 'e2e' | 'llm-judge'): void {
|
||||
/** Enforce a hard dollar cap on per-run eval cost. */
|
||||
function checkHardCap(tier: 'e2e' | 'llm-judge'): void {
|
||||
const evalDir = getProjectEvalDir();
|
||||
const latest = findLatestRun(evalDir, tier);
|
||||
const latest = findLatestFinalizedRun(evalDir, tier);
|
||||
if (!latest) return;
|
||||
const cap = TIER_CAPS[tier];
|
||||
const cost = latest.result.total_cost_usd;
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
/**
|
||||
* Pins the three-count contract of test/helpers/skill-census.ts (C11).
|
||||
*
|
||||
* No hardcoded totals here — the catalog-budget test owns the ratchet.
|
||||
* This file pins the STRUCTURAL relationships that make the three counts
|
||||
* mean different things, using the live repo as the fixture.
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from 'bun:test';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { frontmatterName, skillCensus } from './helpers/skill-census';
|
||||
|
||||
const ROOT = path.resolve(new URL(import.meta.url).pathname, '..', '..');
|
||||
const census = skillCensus(ROOT);
|
||||
|
||||
describe('skillCensus', () => {
|
||||
it('physicalSkillFiles includes the root router and the symlinked dir', () => {
|
||||
expect(census.physicalSkillFiles).toContain('SKILL.md');
|
||||
expect(census.physicalSkillFiles).toContain('connect-chrome/SKILL.md');
|
||||
expect(census.physicalSkillFiles).toContain('open-gstack-browser/SKILL.md');
|
||||
});
|
||||
|
||||
it('authoredSkills dedupes the connect-chrome symlink and excludes the root router', () => {
|
||||
expect(census.authoredSkills).toContain('open-gstack-browser');
|
||||
expect(census.authoredSkills).not.toContain('connect-chrome');
|
||||
// Root router is not an authored skill; its dir entry would be '' anyway.
|
||||
for (const name of census.authoredSkills) expect(name.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('registryEntries carries the root alias and collapses shared frontmatter names', () => {
|
||||
expect(census.registryEntries).toContain('_gstack-command');
|
||||
expect(
|
||||
census.registryEntries.filter((n) => n === 'open-gstack-browser'),
|
||||
).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('count relationships hold: physical = authored + root + symlink dups', () => {
|
||||
const symlinkDups = census.physicalSkillFiles.length - 1 - census.authoredSkills.length;
|
||||
expect(symlinkDups).toBeGreaterThanOrEqual(1); // connect-chrome today
|
||||
// Registry = unique frontmatter names + root alias. It can only collapse
|
||||
// entries relative to physical, never invent them.
|
||||
expect(census.registryEntries.length).toBeLessThanOrEqual(census.physicalSkillFiles.length);
|
||||
expect(census.registryEntries.length).toBeGreaterThan(census.authoredSkills.length - 1);
|
||||
});
|
||||
|
||||
it('frontmatterName mirrors setup: first ^name: line, whitespace stripped', () => {
|
||||
const qa = frontmatterName(path.join(ROOT, 'qa', 'SKILL.md'));
|
||||
expect(qa).toBe('qa');
|
||||
const alias = frontmatterName(path.join(ROOT, 'connect-chrome', 'SKILL.md'));
|
||||
expect(alias).toBe('open-gstack-browser');
|
||||
expect(frontmatterName(path.join(ROOT, 'no-such-dir', 'SKILL.md'))).toBe('');
|
||||
});
|
||||
|
||||
it('every registry entry a host would see resolves back to a physical SKILL.md', () => {
|
||||
const names = new Set(
|
||||
census.physicalSkillFiles
|
||||
.filter((p) => p !== 'SKILL.md')
|
||||
.map((p) => frontmatterName(path.join(ROOT, p)) || path.dirname(p)),
|
||||
);
|
||||
for (const entry of census.registryEntries) {
|
||||
if (entry === '_gstack-command') {
|
||||
expect(fs.existsSync(path.join(ROOT, 'SKILL.md'))).toBe(true);
|
||||
} else {
|
||||
expect(names.has(entry)).toBe(true);
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -71,6 +71,7 @@ describeE2E('/autoplan chain ordering (periodic)', () => {
|
||||
permissionMode: 'plan',
|
||||
cwd: tempDir,
|
||||
timeoutMs: 1_080_000, // 18 min, slightly above test budget
|
||||
seedSkills: true,
|
||||
});
|
||||
|
||||
const hits: PhaseHit[] = [];
|
||||
|
||||
@@ -30,7 +30,15 @@ import * as os from 'os';
|
||||
// --- Prerequisites / gating ---
|
||||
|
||||
const evalsEnabled = !!process.env.EVALS;
|
||||
const describeIfEvals = evalsEnabled ? describe : describe.skip;
|
||||
// External-service tests are periodic-tier (CLAUDE.md tiering rule 3) —
|
||||
// the header above says so, but without a whole-file guard the sharded gate
|
||||
// runner still selects this file into gate. The positive form below is the
|
||||
// canonical guard shape classifyPaidTestFile greps for.
|
||||
const tierOk = process.env.EVALS_TIER === 'periodic';
|
||||
const describeIfEvals = evalsEnabled && tierOk ? describe : describe.skip;
|
||||
if (evalsEnabled && !tierOk) {
|
||||
process.stderr.write('\nbenchmark-providers: SKIPPED — external-service test, periodic tier only\n');
|
||||
}
|
||||
|
||||
const PROMPT = 'Reply with exactly this text and nothing else: ok';
|
||||
|
||||
@@ -127,6 +135,15 @@ describeIfEvals('multi-provider benchmark adapters (live)', () => {
|
||||
}
|
||||
const result = await gemini.run({ prompt: PROMPT, workdir, timeoutMs: 120_000 });
|
||||
if (result.error) {
|
||||
// auth / rate_limit are ENVIRONMENT conditions the test can't act on
|
||||
// (e.g. Google deprecated the individual code-assist auth path — the
|
||||
// adapter classifies "no longer supported" as auth). A live smoke
|
||||
// reports them as a skip, not a false adapter failure. timeout/unknown
|
||||
// still fail: those are the drift classes this test exists to catch.
|
||||
if (result.error.code === 'auth' || result.error.code === 'rate_limit') {
|
||||
process.stderr.write(`\ngemini live smoke: SKIPPED — ${result.error.code}: ${result.error.reason.slice(0, 160)}\n`);
|
||||
return;
|
||||
}
|
||||
throw new Error(`gemini errored: ${result.error.code} — ${result.error.reason}`);
|
||||
}
|
||||
// Adapter must never report empty-success (#2159). After content/stats
|
||||
|
||||
@@ -84,9 +84,13 @@ describeIfSelected('hermetic isolation canaries', ['hermetic-canary', 'hermetic-
|
||||
|
||||
try {
|
||||
const result = await runSkillTest({
|
||||
// ${VAR:-} expansion, not bare $VAR: when scrubbing WORKS the planted
|
||||
// vars are unset, and under a nounset shell (set -u in the operator's
|
||||
// shell snapshot) a bare expansion of an unset var errors the whole
|
||||
// command — making the canary fail exactly when isolation succeeds.
|
||||
prompt: 'Run exactly this bash command and then stop: ' +
|
||||
'echo "CFG=$CLAUDE_CONFIG_DIR"; echo "GH=$GSTACK_HOME"; ' +
|
||||
'echo "CW=$CONDUCTOR_WORKSPACE_PATH"; echo "GP=$GBRAIN_POISON_PROBE"',
|
||||
'echo "CFG=${CLAUDE_CONFIG_DIR:-}"; echo "GH=${GSTACK_HOME:-}"; ' +
|
||||
'echo "CW=${CONDUCTOR_WORKSPACE_PATH:-}"; echo "GP=${GBRAIN_POISON_PROBE:-}"',
|
||||
workingDirectory: workDir,
|
||||
maxTurns: 3,
|
||||
allowedTools: ['Bash'],
|
||||
|
||||
@@ -152,6 +152,7 @@ describeE2E('/plan-ceo-review mode routing (gate)', () => {
|
||||
const session = await launchClaudePty({
|
||||
permissionMode: 'plan',
|
||||
timeoutMs: 540_000,
|
||||
seedSkills: true,
|
||||
});
|
||||
try {
|
||||
await Bun.sleep(8000);
|
||||
|
||||
@@ -44,7 +44,8 @@ describeE2E('/plan-design-review with UI scope (gate)', () => {
|
||||
const session = await launchClaudePty({
|
||||
permissionMode: 'plan',
|
||||
cwd: ROOT,
|
||||
timeoutMs: 480_000,
|
||||
timeoutMs: 720_000,
|
||||
seedSkills: true,
|
||||
});
|
||||
|
||||
let outcome: 'real_question' | 'plan_ready' | 'timeout' | 'exited' = 'timeout';
|
||||
@@ -70,7 +71,11 @@ describeE2E('/plan-design-review with UI scope (gate)', () => {
|
||||
`Reference plan file: ${fixtureRelPath}\r`
|
||||
);
|
||||
|
||||
const budgetMs = 360_000;
|
||||
// 600s, not 360s: the skill preamble (update-check, session bookkeeping,
|
||||
// learnings) plus extended model thinking can take ~6 minutes before the
|
||||
// scope-gate AskUserQuestion renders — a 360s budget expired seconds
|
||||
// before the (correct) AUQ appeared in the observed failure transcript.
|
||||
const budgetMs = 600_000;
|
||||
const start = Date.now();
|
||||
let lastPermSig = '';
|
||||
while (Date.now() - start < budgetMs) {
|
||||
@@ -145,6 +150,6 @@ describeE2E('/plan-design-review with UI scope (gate)', () => {
|
||||
);
|
||||
}
|
||||
},
|
||||
540_000,
|
||||
780_000,
|
||||
);
|
||||
});
|
||||
@@ -161,6 +161,7 @@ describeE2E('/ship idempotency E2E (periodic, real-PTY)', () => {
|
||||
timeoutMs: 720_000,
|
||||
// Disable network-y pieces so the agent can't reach actual github.
|
||||
env: { GH_TOKEN: 'mock-not-real', NO_COLOR: '1' },
|
||||
seedSkills: true,
|
||||
});
|
||||
|
||||
let outcome: 'detected' | 'plan_ready' | 'attempted_mutation' | 'timeout' | 'exited' = 'timeout';
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
/**
|
||||
* Pins scripts/test-strict-output.ts — the verdict-integrity layer of the
|
||||
* sharded paid runner. Its whole reason to exist is refusing to trust a zero
|
||||
* exit when failures were printed OR fewer files ran than planned; paid-shards'
|
||||
* fake commands never emit real Bun result lines, so without this file that
|
||||
* core was exercised nowhere and a regex regression would silently revert the
|
||||
* paid tier to trusting exit codes.
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from 'bun:test';
|
||||
import { BunTestOutputClassifier, strictTestExitCode } from '../scripts/test-strict-output';
|
||||
|
||||
describe('strictTestExitCode', () => {
|
||||
it('trusts a clean zero exit when the expected file count ran', () => {
|
||||
const summary = { failedTests: 0, unhandledBetweenTests: 0, terminalFileCounts: [1] };
|
||||
expect(strictTestExitCode(0, summary, 1)).toBe(0);
|
||||
});
|
||||
|
||||
it('refuses a zero exit when fewer files ran than expected (invisible non-execution)', () => {
|
||||
const summary = { failedTests: 0, unhandledBetweenTests: 0, terminalFileCounts: [1] };
|
||||
expect(strictTestExitCode(0, summary, 2)).toBe(1);
|
||||
});
|
||||
|
||||
it('refuses a zero exit when failure lines were printed', () => {
|
||||
const summary = { failedTests: 1, unhandledBetweenTests: 0, terminalFileCounts: [1] };
|
||||
expect(strictTestExitCode(0, summary, 1)).toBe(1);
|
||||
});
|
||||
|
||||
it('refuses a zero exit on an unhandled error between tests', () => {
|
||||
const summary = { failedTests: 0, unhandledBetweenTests: 1, terminalFileCounts: [1] };
|
||||
expect(strictTestExitCode(0, summary, 1)).toBe(1);
|
||||
});
|
||||
|
||||
it('propagates a non-zero child exit regardless of expectedFiles', () => {
|
||||
const summary = { failedTests: 0, unhandledBetweenTests: 0, terminalFileCounts: [1] };
|
||||
expect(strictTestExitCode(1, summary, 1)).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('BunTestOutputClassifier', () => {
|
||||
it('counts a (fail) line split across write chunks', () => {
|
||||
const c = new BunTestOutputClassifier();
|
||||
c.write('[31m(fail) my te');
|
||||
c.write('st [3.42ms][0m\nRan 4 tests across 1 files. [2.10s]\n');
|
||||
const summary = c.end();
|
||||
expect(summary.failedTests).toBe(1);
|
||||
expect(summary.terminalFileCounts).toEqual([1]);
|
||||
// exit 0 + a printed failure must not be trusted
|
||||
expect(strictTestExitCode(0, summary, 1)).toBe(1);
|
||||
});
|
||||
|
||||
it('records the terminal file count from the summary line', () => {
|
||||
const c = new BunTestOutputClassifier();
|
||||
c.write('Ran 0 tests across 1 files. [0.01s]\n');
|
||||
const summary = c.end();
|
||||
expect(summary.terminalFileCounts).toEqual([1]);
|
||||
// a fully diff-skipped single-file shard (0 tests, 1 file loaded) still
|
||||
// passes: 1 file ran, which is what was expected
|
||||
expect(strictTestExitCode(0, summary, 1)).toBe(0);
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user