mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-19 22:54:36 +01:00
Merge remote-tracking branch 'origin/main' into capy/fw3-a3
This commit is contained in:
commit
4f2e9ed830
36 files changed
+4402
-71
No files matched your search
+126
-11
@@ -1009,17 +1009,44 @@ select nothing: docs, any non-paid `*.test.ts(x)`, `FREE_ONLY_PR_FILES` (the
|
||||
free and paid duration seeds, `scripts/ubicloud/**`, every workflow that runs no
|
||||
paid evals, and free-only tools), and the fixtures in `FREE_FIXTURES`
|
||||
(`test/helpers/free-fixtures-data.ts`, each fixture with its free consumers).
|
||||
`PAID_WORKFLOW_FILES` (`evals.yml`, `evals-periodic.yml`, `evals-marathon.yml`)
|
||||
and any other unknown file restore the full gate (`prCoverage.mode =
|
||||
full-fallback`). The job summary and PR comment list each file that caused a
|
||||
fallback with one of three labels:
|
||||
Unmapped files under `DERIVABLE_PREFIXES` (bin/, lib/, scripts/, browse/,
|
||||
design/, make-pdf/, hosts/, extension/, model-overlays/, agents/, test/helpers/,
|
||||
capy/) are not unknown: `scripts/pr-dependencies.ts` derives the cases whose
|
||||
reference closure reaches them (`prCoverage.mode = dependents`), or reports that
|
||||
no paid case reaches them. The derivation scans the imports of every code file,
|
||||
including `#!`-shebang `.ts` files and extensionless Bun or Node scripts; if any
|
||||
import scan fails, every unmapped file restores the full gate (label **import
|
||||
scan failed**). `PR_FULL_GATE_FILES`, `PAID_WORKFLOW_FILES` (`evals.yml`,
|
||||
`evals-periodic.yml`, `evals-marathon.yml`) and any other unknown file restore
|
||||
the full gate (`prCoverage.mode = full-fallback`). The job summary and PR comment
|
||||
list each unmapped file that caused a fallback with one of these labels:
|
||||
|
||||
- **needs touchfile entry**: a paid test, helper or fixture no case maps; add it
|
||||
to the owning case in `E2E_TOUCHFILES` (`test/helpers/touchfiles-data.ts`).
|
||||
- **add to FREE_ONLY_PR_FILES**: a file only free lanes read; add it to the list
|
||||
in `scripts/test-pr-profile.ts` (or a fixture to `FREE_FIXTURES`).
|
||||
- **real unknown dependency**: shared code (bin/, lib/, resolvers, hosts, setup)
|
||||
whose dependents are not mapped; the full gate is the correct result.
|
||||
- **full-gate input**: a `PR_FULL_GATE_FILES` entry every case consumes; the
|
||||
full gate is intended and no mapping narrows it.
|
||||
- **deleted but still referenced** and **import scan failed**: fix the named
|
||||
reference or parse error.
|
||||
- **real unknown dependency**: a file outside the derivable directories that no
|
||||
list classifies (for example `ETHOS.md`); map it, exempt it, or derive its
|
||||
directory.
|
||||
|
||||
The fallback rate is a label, not the bill: a dependents run whose derived
|
||||
closure reaches every gate case costs about what a full-fallback run does.
|
||||
`bun run test:health` leads with the cost per merged PR and reports
|
||||
`pr-cases-per-run` (mean selected gate cases and judges per PR run, per mode)
|
||||
and `pr-known-cost-per-run` (mean known trial cost per PR run, per mode) beside
|
||||
`pr-fallback-rate`.
|
||||
|
||||
`package.json` (beyond a version-only change) and `bun.lock` stay full-gate
|
||||
inputs. Over the 225 PR runs from 2026-10-07 to 2026-10-17, no run fell back on
|
||||
them alone, dependabot PRs never reach the paid lane (`evals.yml` skips
|
||||
`github.actor == 'dependabot[bot]'`), and every runtime package's importer
|
||||
(`test/helpers/agent-sdk-runner.ts`, `test/helpers/llm-judge.ts`, `browse/src/*`,
|
||||
`make-pdf/src/*`) reaches every gate case or is in the global set, so narrowing
|
||||
them would select the same cases.
|
||||
|
||||
`test/free-fixtures.test.ts` holds the inventory: each `test/fixtures/**` path
|
||||
belongs to a paid touchfile or `FREE_FIXTURES`, and each workflow to the paid or
|
||||
@@ -1027,6 +1054,78 @@ free-only list. Editing a paid test file runs its own gate cases on the PR when
|
||||
their Bun test is named by the case id; the file's other cases stay deferred with
|
||||
a reason that says to name the test by its case id.
|
||||
|
||||
### Exercised-path map
|
||||
|
||||
The derived closure (above) reaches almost every gate case from most bin/ and
|
||||
lib/ files, because it follows prose, string and doc-comment mentions. The
|
||||
exercised-path map is a tighter consumer set that runs in **shadow mode**: it
|
||||
is computed and reported on every PR plan, and it never changes which cases a
|
||||
PR runs. Activating it is a later decision that needs PTY command capture for
|
||||
the cases that record no commands, two weeks of shadow data with zero shadow
|
||||
misses, and Garry's call on the measured saving.
|
||||
|
||||
A case's exercised consumers (`exercisedConsumers` in
|
||||
`scripts/pr-dependencies.ts`) are the union of its touchfile matches, the
|
||||
closure of its test file and helpers, the executable shell of its loaded skill
|
||||
(template and generated SKILL.md) and of every bin, lib and script that shell
|
||||
runs, and the bins its recorded trials invoked. Shell edges are command-position
|
||||
`gstack-*` names and repo paths, `.`/`source`, `./setup`, quoted path
|
||||
literals, `path.join` pairs, data files, `bun run` scripts and compiled outfiles
|
||||
(`browse/dist/browse`, `$B`, `design/dist/design`, `$D`, `make-pdf/dist/pdf`).
|
||||
Comments, echo/printf/die messages, case patterns, bare `VAR=gstack-x` strings
|
||||
and slash-command skill names are not edges; command substitutions are kept, an
|
||||
expanding heredoc keeps its substitutions and a heredoc an interpreter reads
|
||||
stays whole. The closure stops at `PR_FULL_GATE_FILES` and the global set, which
|
||||
stay full. `test/exercised-map.test.ts` pins that this function equals or is a
|
||||
reviewed superset of the measured prototype (`test/helpers/exercised-prototype.ts`).
|
||||
|
||||
The observed half is `test/helpers/exercised-commands.json` (generated; per case
|
||||
the bins and compiled outfiles its recorded trials ran, observation counts, the
|
||||
source run ids and a mean known cost per run). Refreshes are union-only: an
|
||||
observation never removes a command and a missing or empty artifact never erases
|
||||
one; removing a command is a reviewed edit. While the map is in shadow mode it
|
||||
changes no selection, so a PR that changes only the map selects no paid case.
|
||||
|
||||
Dollar figures are lower bounds. Known cost excludes PTY trials, whose cost is
|
||||
unknown: about 39% of observed case runs in the committed map. The shadow block
|
||||
and `test:health` both say so.
|
||||
|
||||
- **Shadow block:** `prCoverage.shadow` in the paid-plan manifest lists the gate
|
||||
cases the map would select, would drop and would add, a provenance line per
|
||||
changed file ("selects N case(s): touchfile (k), import (k), skill bash via
|
||||
bin/gstack-x (k), observed bin/gstack-y (k)"), the predicted saving and the
|
||||
unknown-cost share. The coverage summary prints one line of it.
|
||||
- **Shadow misses (`bun run test:health`, metric `pr-shadow-map`):** a red case
|
||||
(more executed failed than passed trials in the run) that the shadow would
|
||||
drop is an activation miss unless the case passed in a later run of the same
|
||||
branch at the same head SHA. That later pass marks it a flaky drop, which is
|
||||
listed but not counted. The D3 activation bar counts activation misses only.
|
||||
- **MAP_GAP:** the paid report runs the same bin matcher over each executed
|
||||
case's recorded commands and prints, and writes to `map-gaps.txt`,
|
||||
`MAP_GAP <case> <command>: non-blocking for this PR; the next census refresh
|
||||
widens the map (docs/TESTING_INTERNALS.md#exercised-path-map)` for a command
|
||||
outside the case's static-plus-observed closure. It is its own result: never a
|
||||
case verdict, a lane problem or an INFRA re-dispatch input.
|
||||
- **Census refresh:** the weekly census unions its own recorded commands into the
|
||||
map (no REST artifact downloads), uploads it as `exercised-map-a<attempt>`,
|
||||
reports MAP_GAP lines and the cases that recorded no commands, and prints the
|
||||
refresh command. Run `bun run test:exercised-map --from-run <id>`, or
|
||||
`--from-dir <dir>` on a locally downloaded `exercised-map-a<attempt>` artifact
|
||||
when the download hits HTTP 403, and open the change as an ordinary PR.
|
||||
`bun run test:exercised-map --check` validates the committed file.
|
||||
- **Refresh contract:** a refresh whose input is unavailable or partial writes
|
||||
nothing, keeps the committed union and exits with a stable code that names
|
||||
what was missing: `2` usage; `3` input unavailable (a missing directory, or a
|
||||
failed or HTTP 403 download); `4` input incomplete (no `shards/`, no
|
||||
`trial-outcomes.jsonl`, an executed case without a shard artifact, or a missing
|
||||
or malformed `exercised-commands.json`). The census step reports a failed
|
||||
refresh in its job summary and uploads no map.
|
||||
- **Replay receipt:** `bun run test:exercised-map --replay` replays the recorded
|
||||
red PR runs in `test/fixtures/exercised-map-replay.json` (every executed failed
|
||||
gate trial in 225 `evals.yml` pull_request runs, 2026-10-07 to 2026-10-17)
|
||||
through today's selector and its shadow, and exits 1 on any MISS: a failed
|
||||
gate case the shadow would not run. `test/exercised-map.test.ts` pins zero.
|
||||
|
||||
### PR receipts, recovery and debounce
|
||||
|
||||
Passes and failures of single-trial shards are written as receipts at the
|
||||
@@ -1164,10 +1263,11 @@ changed. Cut or unstarted reruns are incomplete, not passes.
|
||||
### Free suite duration seed
|
||||
|
||||
`scripts/free-test-durations.json` drives shard packing; files missing from it
|
||||
pack at the seed's 99th percentile. Acceptable seeds come only from
|
||||
`bun run test:ubicloud --record-durations` (each file timed in its own child on a
|
||||
VM with the CI lane's environment; it needs `UBICLOUD_API_KEY` and copies the
|
||||
refreshed seed back into the checkout). A 4-core laptop or small-sandbox
|
||||
pack at the seed's 99th percentile. Acceptable seeds come only from a recording
|
||||
in the CI lane's environment, each file timed in its own child: the weekly
|
||||
`free-durations-record.yml` run, or `bun run test:ubicloud --record-durations`
|
||||
(it needs `UBICLOUD_API_KEY` and copies the refreshed seed back into the
|
||||
checkout). A 4-core laptop or small-sandbox
|
||||
recording is not acceptable: browser and display tests skip or slow down there,
|
||||
so the seed misstates them. Contributors without `UBICLOUD_API_KEY` leave the
|
||||
refresh to a maintainer.
|
||||
@@ -1188,6 +1288,20 @@ refresh to a maintainer.
|
||||
`{file, reason, todo}` and the test prints it. A merge-base without the
|
||||
allowlist file counts as the first allowlist. Fix a failure by splitting the
|
||||
file, then rerun `bun run test:ubicloud --record-durations`.
|
||||
- `free-durations-record.yml` (Mondays 04:00 UTC, before test-health, and on
|
||||
dispatch) records the Linux seed on `ubicloud-standard-8` with
|
||||
`scripts/ubicloud/setup-free-suite.sh` and the Windows seed through
|
||||
`windows-free-tests.yml`'s `record_durations` path. When either recording adds
|
||||
or drops a test file, it force-updates the `bot/free-duration-seeds` branch,
|
||||
opens or refreshes one seed PR whose body states the ratchet result, and
|
||||
dispatches the free lanes on it. A file that fails while recording turns its
|
||||
job red but keeps its duration, and the PR body names it; the seed PR's own
|
||||
free checks decide whether the tree is healthy. An interrupted recording
|
||||
uploads no seed, and more than 5 red files on either OS proposes nothing. The
|
||||
bot never edits the allowlist: a ratchet failure is fixed on the branch by a
|
||||
maintainer before the PR merges, and while the open seed PR holds any commit
|
||||
besides the bot's one refresh commit, later runs leave it alone. Only runs on
|
||||
`main` push the branch; a dispatch from another ref reports only.
|
||||
|
||||
### Windows free lane
|
||||
|
||||
@@ -1195,7 +1309,8 @@ refresh to a maintainer.
|
||||
runs one strict serial shard per `windows-latest` job
|
||||
(`bun run test:windows --ci-run`), and verifies every shard receipt on Linux
|
||||
(`--windows-only --ci-verify`). Packing uses Windows-measured durations from
|
||||
`scripts/free-test-durations-windows.json`; refresh them with
|
||||
`scripts/free-test-durations-windows.json`; the weekly
|
||||
`free-durations-record.yml` run refreshes them, or by hand run
|
||||
`gh workflow run windows-free-tests.yml --ref <branch> -f record_durations=true`,
|
||||
then download the `free-test-durations-windows` artifact and commit it.
|
||||
|
||||
|
||||
+3
-1
@@ -41,7 +41,9 @@ paths, at most 64 per list); commands are one line under 512 bytes.
|
||||
<paths>` for entries whose inputs or outputs intersect) in a scratch
|
||||
`git worktree` seeded from the working tree, with `node_modules` linked rather
|
||||
than copied, and reports stale outputs (`REGEN_STALE`, exit 1) without touching
|
||||
the tree. An entry with no `command` is tracked only (goldens regenerated by
|
||||
the tree. When outputs differ but `package.json` declares dependencies and
|
||||
`node_modules` is missing, the entry is `REGEN_DEPS_MISSING` (install, then
|
||||
rerun) instead of stale. An entry with no `command` is tracked only (goldens regenerated by
|
||||
another entry).
|
||||
- `gstack-regen write` regenerates in place: what `gstack-restamp` and the
|
||||
Step 3 conflict rule use. On a conflict in a registered output, take either
|
||||
|
||||
@@ -2789,6 +2789,18 @@ take either side, then regenerate.
|
||||
worktree has no `node_modules` of its own, so a command that needs them must
|
||||
reference the repository's (gstack's entries do through `bun run`).
|
||||
|
||||
<a id="regen-deps-missing"></a>
|
||||
### `REGEN_DEPS_MISSING`
|
||||
|
||||
**Meaning.** A registered command exited 0 and its outputs differ from the
|
||||
committed ones, but `package.json` declares dependencies and the repository has
|
||||
no `node_modules`, so the scratch worktree ran the generator without them. The
|
||||
difference may come from the missing dependencies rather than stale files, so
|
||||
the entry is reported as unverified, not `REGEN_STALE`.
|
||||
|
||||
**Fix.** Run the printed install command (`bun install`, or the one matching the
|
||||
repository's lockfile), then rerun `gstack-regen check` or the pre-gate.
|
||||
|
||||
<a id="pregate-config-invalid"></a>
|
||||
### `PREGATE_CONFIG_INVALID`
|
||||
|
||||
|
||||
Reference in new issue
Block a user