mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-30 00:22:12 +02:00
v1.90.2.0 perf: halve local free-suite time and preserve coverage (#2972)
* perf: remove repeated test work and preserve AUQ execution budgets * fix: validate native evaluation fixture evidence at its actual boundaries * fix: clarify deployment approval and recovery state transitions * chore: document coverage and release v1.90.2.0 * test: preserve Windows scheduling and native no-change consent * test: recognize verified reads through fixture symlinks * test: isolate alias-name installation from runtime assets
This commit is contained in:
Garry Tan
@@ -100,6 +100,14 @@ reported separately. See [Overlay benchmark contract v2](OVERLAY_BENCHMARK_CONTR
|
||||
for exact correctness requirements, retired fanout cases, immutable evidence,
|
||||
and the limits of a passing result.
|
||||
|
||||
## Coverage ownership
|
||||
|
||||
[The test portfolio audit](TEST_PORTFOLIO.md) separates deterministic harness
|
||||
checks, prompt judges, live first-question captures, completed workflows and
|
||||
platform integrations. Shared setup is reusable; evidence with different
|
||||
scenario or independent-trial requirements is not. It records the preserved
|
||||
coverage and measured component savings from the test-speed refactor.
|
||||
|
||||
## Runners: how the suites execute (2026-08 overhaul)
|
||||
|
||||
**Aside-only E2E tests self-skip without a live Aside; browser-driving tests
|
||||
@@ -153,7 +161,9 @@ load-sensitive on a busy dev box, runs only in CI or on explicit opt-in
|
||||
|
||||
**Free suite (`bun run test:free`).** `scripts/test-free-shards.ts` runs N
|
||||
concurrent shard processes (serial within each) with strict-output
|
||||
classification per shard. Full-suite shards are packed by RECORDED PER-FILE
|
||||
classification per shard. Local defaults use the available CPU affinity,
|
||||
floored at one and capped at six; `GSTACK_FREE_JOBS` remains an explicit override.
|
||||
This does not change the separate CI machine count. Full-suite shards are packed by RECORDED PER-FILE
|
||||
DURATIONS (LPT, `packShardsByDuration`) when the committed seed
|
||||
`scripts/free-test-durations.json` exists — refresh it occasionally with
|
||||
`bun run test:free --record-durations` (each file timed in its own child;
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
# Test portfolio: coverage ownership and repeated work
|
||||
|
||||
Audit source: `06ed920a974809ebedc6bcbbe402fb81f5944598`, September 24, 2026.
|
||||
This is a coverage inventory and a focused refactor, not a claim that every
|
||||
test assertion is interchangeable with another assertion about the same skill.
|
||||
|
||||
## One owner for each kind of evidence
|
||||
|
||||
Tests can share setup or inspect the same public capture. They must not count
|
||||
one capture twice when the contract requires independent trials. In particular,
|
||||
free parser tests, a prompt-quality score, and a completed live workflow prove
|
||||
different things even when they mention the same skill.
|
||||
|
||||
| Responsibility | Owner | What this evidence does not replace |
|
||||
| --- | --- | --- |
|
||||
| Selection, budgets, process ownership, cleanup, native-event parsing and recorded failure controls | Free runner and fixture regressions | A live model choosing or completing the right workflow |
|
||||
| Generated files, host parity, section manifests and complete input identity | Free generation and structural tests | An agent actually loading the required section |
|
||||
| Prompt clarity and rubric quality | The registered quality judge for that complete prompt | Tool execution, native acknowledgments or a finished report |
|
||||
| Native first-question format and substance | Live SDK/PTY question capture | An answered question or a completed workflow; receipts explicitly say `workflowCompleted: false` |
|
||||
| Stochastic consistency and verbose/carved comparison | Independent captures, with separate stability and A/B oracles | One successful sample reused as three trials, or one prompt version standing in for the other |
|
||||
| Decisions, findings and report completion | Per-skill native workflow fixtures | The first question alone, screen text without native evidence, or a generic question count |
|
||||
| Offline deployment and canary report construction | The explicitly simulated workflow fixtures | A real GitHub merge, deployment, rollback or production health check |
|
||||
| Multi-phase ordering and hand-offs | One uninterrupted Autoplan chain | Four independent successful skill sessions |
|
||||
| External reviewers, other model providers, browser engines and platform behavior | Their respective live integration fixtures | Prompt parity or a mock transport |
|
||||
|
||||
Overlay efficacy experiments retain their full fixture/model/arm/trial matrix.
|
||||
Security cases retain their source, path, socket, process and lease identities.
|
||||
These are distinct scenario dimensions, not repeated work to delete.
|
||||
|
||||
## Complete inventory, not just the fast subset
|
||||
|
||||
At the audited revision, all 1,124 tracked Bun test files partition into 1,010 free
|
||||
files and 114 paid files. The paid inventory contains 207 registered E2E
|
||||
selection IDs (84 gate and 123 periodic) and 25 main quality-judge IDs. IDs,
|
||||
files, model calls, samples, attempts and executed Bun tests are different
|
||||
counts; use the run manifest and receipts rather than substituting one for
|
||||
another.
|
||||
|
||||
The full gate and periodic manifests plan 153 file processes across 112 unique
|
||||
files. Forty-one files appear in both manifests because they contain mixed-tier
|
||||
cases; that does not establish 41 duplicated scenarios. Brain privacy and ship
|
||||
idempotency have existing explicit exclusions, iOS needs hardware, and the spec
|
||||
quality file contains a TODO. The iOS fixture's native Swift smoke test is
|
||||
outside this Bun-file census and requires its platform runtime. Unavailable
|
||||
or excluded work is not coverage.
|
||||
|
||||
Existing sources remain authoritative:
|
||||
|
||||
- `test/helpers/paid-test-set.ts` owns the free/paid file boundary.
|
||||
- `test/helpers/touchfiles-data.ts` owns registered dependency and tier data.
|
||||
- `scripts/test-pr-profile.ts` owns the deliberately partial PR profile.
|
||||
- `test/helpers/eval-budgets.ts` owns supervision and retry exceptions.
|
||||
- The free and paid shard runners own inventory, scheduling and reconciliation.
|
||||
|
||||
`test:quick` and `test:pr` are feedback lanes, not full release acceptance. This
|
||||
refactor does not remove cases, shrink samples, change tiers, lower thresholds,
|
||||
shorten production deadlines, or move checks to a later cadence.
|
||||
|
||||
## Repeated work removed
|
||||
|
||||
**Synthetic terminal startup.** The 42 fake-terminal scenarios in nine files
|
||||
emit their existing readiness marker only after installing handlers. They no
|
||||
longer spend the real CLI's eight-second grace waiting for an already-ready
|
||||
fake. Native CLI grace, terminal geometry, observation delays, scenario inputs
|
||||
and all existing assertions stay intact.
|
||||
|
||||
**Publication polling.** Invalid/unavailable-journal cases still call the real
|
||||
hook and transcript reader through all 40 polling intervals. A scoped serial
|
||||
fake clock removes wall-clock sleeping, while controls verify the full logical
|
||||
deadline, late arrival, and restoration after success and failure. The real
|
||||
delayed-journal and shell-transport cases still use real time.
|
||||
|
||||
**Native watchdog setup.** Compile the identical source once per test file and
|
||||
copy the executable into each case's isolated directory. Every watchdog still
|
||||
executes; only duplicate checks of the same compiler result are consolidated.
|
||||
Mutable work directories and cleanup ownership are never pooled.
|
||||
|
||||
**Independent live captures.** Start all three consistency captures through the
|
||||
existing three-query semaphore and both A/B arms concurrently. Await every
|
||||
settlement before cleanup, retain sibling failures, and attempt cleanup for
|
||||
every owned directory. Consistency judging remains sequential; A/B has at most
|
||||
two simultaneous judges. The free regression exercises the actual registered
|
||||
callbacks, native capture receipts, semaphore and judge request builder.
|
||||
|
||||
**Runner ownership.** Path normalization belongs to the existing shared strict
|
||||
output utility, not the free runner. The paid runner no longer imports the free
|
||||
runner to use it. Exact free-only exemptions cover the free runner and the free
|
||||
AUQ replay worker, with paid import-closure and selection controls. Mapped
|
||||
dependencies still win; unknown dependencies retain the broad fallback. No
|
||||
directory-wide exemption is introduced.
|
||||
|
||||
**Local scheduling.** Default free workers use available CPU affinity, with a
|
||||
floor of one and the existing cap of six. Each shard stays serial internally,
|
||||
and explicit `GSTACK_FREE_JOBS` overrides keep their previous meaning. The
|
||||
separate 20-machine CI plan is unchanged. Windows free-test CI explicitly retains
|
||||
its two-worker budget rather than inheriting the local default; local worker
|
||||
gains are not CI gains.
|
||||
|
||||
## Measurement contract
|
||||
|
||||
Compare original and edited code on the same machine, runtime, launch
|
||||
environment and workload. Preserve failure and retry records. Count the full
|
||||
case/sample inventory and report skips and unavailable platforms separately.
|
||||
Do not subtract failures from elapsed time or use a smaller selection as proof
|
||||
that the complete suite got faster.
|
||||
|
||||
Measured component comparisons:
|
||||
|
||||
| Workload | Before | After | Coverage retained |
|
||||
| --- | ---: | ---: | --- |
|
||||
| Nine synthetic-terminal files, serial aggregate | 165.87s | 72.98s | 81 tests, 1,593 assertions, 42 PTY scenarios |
|
||||
| Publication guard and watchdog files, serial aggregate | 56.57s | 9.63s | 245 original tests; three additional clock controls |
|
||||
| Two live periodic AUQ files, same machine and runtime | 325.09s | 136.40s | Two tests, five independent captures, all original grading rules |
|
||||
|
||||
Exact selectors for the synthetic-terminal comparison:
|
||||
|
||||
```text
|
||||
test/plan-count-fixture.test.ts
|
||||
test/plan-count-design-ui-recovery.test.ts
|
||||
test/plan-count-native-input.test.ts
|
||||
test/plan-count-empty-review.test.ts
|
||||
test/plan-count-owned-permission.test.ts
|
||||
test/plan-count-quoted-frame-ak.test.ts
|
||||
test/plan-count-truncated-question.test.ts
|
||||
test/plan-count-preview-footer.test.ts
|
||||
test/eng-test-plan-edit-approval.test.ts
|
||||
```
|
||||
|
||||
The publication/watchdog pair is `test/autoplan-publication-guard.test.ts` and
|
||||
`test/cso-watchdog.test.ts`. The live pair is
|
||||
`test/skill-e2e-auq-consistency.test.ts` and
|
||||
`test/skill-e2e-auq-verbose-vs-carved-ab.test.ts`, using the default three
|
||||
consistency samples plus the two A/B captures.
|
||||
|
||||
These are separate comparisons; do not add their percentages or call them a
|
||||
complete paid-suite result. The terminal aggregate is 56% faster, the two
|
||||
publication/watchdog files are 83% faster, and the live AUQ pair is 58% faster.
|
||||
Watchdog assertion counts fall only because identical compilation is checked
|
||||
once; all behavioral and security assertions remain.
|
||||
|
||||
The matched complete local free-suite comparison used the same four-CPU Linux
|
||||
machine, Bun 1.4.0, Node 24.18.0, built artifacts, display and isolated Git
|
||||
configuration. Neither run set a worker override: the original default used two
|
||||
workers and the changed default used four.
|
||||
|
||||
| Complete free-suite result | Original | Performance refactor |
|
||||
| --- | ---: | ---: |
|
||||
| Wall time | 887.82s | 413.78s |
|
||||
| Passing cases | 25,504 | 25,547 |
|
||||
| Failing cases | 0 | 0 |
|
||||
| Skipped cases | 60 | 60 |
|
||||
| Assertions | 199,132 | 200,351 |
|
||||
| Test files | 1,010 | 1,012 |
|
||||
|
||||
This is a 53.4% local reduction. Case-level reconciliation retained every original
|
||||
passing identity except the intentionally renamed worker-default policy test,
|
||||
added 43 cases, and retained the exact same 60 skipped identities. This comparison
|
||||
predates the additional fixture and deployment-workflow regression cases; final
|
||||
acceptance must cover those too. It is not a measurement of the separate CI
|
||||
matrix or the complete paid census.
|
||||
|
||||
The exact historical PR #2956 diff selected 84 gate IDs plus 25 judges because
|
||||
the free runner was treated as an unknown paid dependency. Replaying that diff
|
||||
after the boundary repair selects the intended 26 fast IDs plus the same 25
|
||||
judges. A free-runner-only diff selects no paid work. That is routing accuracy,
|
||||
not a fresh full-census runtime improvement.
|
||||
|
||||
## Evidence validity
|
||||
|
||||
Check the executable actually used by each SDK, print-mode and terminal launcher.
|
||||
A CLI version cached during preflight does not prove the version used by later
|
||||
sessions if PATH contents change. Use native session-init versions, terminal
|
||||
startup captures or process witnesses, and keep runtime controls effective after
|
||||
the hermetic environment is constructed.
|
||||
|
||||
Captured-event regressions prove the validator accepts valid evidence and rejects
|
||||
invalid evidence. They do not turn an old failed model run into a pass. Preserve
|
||||
every configured retry attempt and distinguish actual registrations from filtered
|
||||
or out-of-tier placeholders. A retained passing judge is reusable only when its
|
||||
complete request, rubric, parameters and relevant dependencies match; native
|
||||
behavioral acceptance follows its separate freshness contract.
|
||||
|
||||
## Remaining work, not claimed savings
|
||||
|
||||
Report final acceptance with the delivery revision and its actual case inventory.
|
||||
Complete fresh gate/periodic and cross-platform performance comparisons are not
|
||||
established by the measurements above.
|
||||
|
||||
The longest indivisible live workflow limits the benefit of extra workers.
|
||||
Historical paid-duration replay suggests better scheduling alone cannot halve
|
||||
the full lane. A follow-up should unify executable case ownership/counts before
|
||||
sharing captures between judges or splitting long files: keep each oracle,
|
||||
scenario, retry and independent-trial requirement explicit. The ordered
|
||||
Autoplan chain, host integrations and security boundary cases must not be
|
||||
replaced with cheaper look-alikes.
|
||||
+2
-2
@@ -693,9 +693,9 @@ This is my **deploy pipeline mode**.
|
||||
|
||||
`/ship` creates the PR. `/land-and-deploy` finishes the job: merge, deploy, verify.
|
||||
|
||||
It merges the PR, waits for CI, waits for the deploy to finish, then runs canary checks against production. One command from "approved" to "verified in production." If the deploy breaks, it tells you what failed and whether to rollback.
|
||||
It confirms PR readiness and your merge approval, merges, then monitors CI and deployment before checking production. If deployment breaks, it reports what failed and whether rollback is available. If the new revision's deployment cannot be confirmed, it reports that uncertainty rather than treating a healthy old page as proof.
|
||||
|
||||
First run on a new project triggers a dry-run walk-through so you can verify the pipeline before it does anything irreversible. After that, it trusts the config and runs straight through.
|
||||
The first run, or a changed deployment configuration, triggers a dry-run walk-through so you can verify the pipeline before anything irreversible happens. An unchanged, previously confirmed configuration skips that walkthrough, not readiness checks or merge approval. Approval is bound to the exact PR head and destination branch; changing either requires fresh readiness and approval.
|
||||
|
||||
### Setup
|
||||
|
||||
|
||||
Reference in New Issue
Block a user