mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-03 01:46:55 +02:00
Merge remote-tracking branch 'origin/test-audit-reduction' into capy/audit-fix-wave
This commit is contained in:
commit
44177fffc6
666 files changed
+11963
-102378
No files matched your search
@@ -267,6 +267,12 @@ Historical measurements from 2026-09-21:
|
||||
| Local complete free suite | All 993 files, six workers | 4m 35s |
|
||||
| Complete Linux CI | All 993 files, 20 isolated runners | 1m 40s across test steps; 3m 7s including setup and aggregation |
|
||||
|
||||
After the 2026-09-29 test audit ([evidence](docs/test-audit-2026-09.md)):
|
||||
|
||||
| Run | Coverage | Elapsed |
|
||||
|---|---|---|
|
||||
| Complete free suite, `bun run test:ubicloud` (standard-16) | All 857 files, 20,302 passing tests | 136 seconds on the VM; 1,738 seconds of recorded serial test time |
|
||||
|
||||
The [Linux CI run](https://github.com/garrytan/gstack/actions/runs/35642667809)
|
||||
on `25030d68` included one recorded successful retry. Its slowest test step was 77 seconds;
|
||||
staggered starts made the complete test span longer. Typical PR paid-gate timing
|
||||
@@ -275,6 +281,15 @@ changes do not select paid work; mapped dependencies take precedence, and unknow
|
||||
dependencies retain the broad fallback. See the
|
||||
[coverage boundaries](docs/TEST_PORTFOLIO.md#repeated-work-removed).
|
||||
|
||||
When a paid eval fails, fix the product or the harness and add the captured case as one row in
|
||||
the detector's owner test (the detector → owner table is in
|
||||
[TEST_PORTFOLIO.md](docs/TEST_PORTFOLIO.md#detector-owner-tests)); never add a new per-incident file.
|
||||
A row is one `describe` block or table entry next to the others, for example a new
|
||||
`describe('eng-cache-writes-at', …)` in `test/eng-first-review.test.ts` that loads its fixture and asserts
|
||||
`engFirstReviewAUQ` on the captured call. Run `bun test <owner-test>`, then
|
||||
`bun test test/test-of-test-ratchet.test.ts`: the ratchet fails on any new test file that imports only
|
||||
`test/` code and names the owner test to use instead.
|
||||
|
||||
Follow [Validation discipline in AGENTS.md](AGENTS.md#validation-discipline):
|
||||
reproduce known failures with focused checks, verify adjacent source and
|
||||
generation contracts, then run the affected and remaining required selected
|
||||
@@ -435,6 +450,16 @@ Each dimension is scored 1-5. Threshold: every dimension must score **≥ 4**. T
|
||||
- Tests live in `test/skill-llm-eval.test.ts`
|
||||
- Calls the Anthropic API directly (not `claude -p`), so it works from anywhere including inside Claude Code
|
||||
|
||||
### Paid-test touchfiles
|
||||
|
||||
`test/helpers/touchfiles-data.ts` maps each paid case to the files whose edits select it. Free
|
||||
`*.test.ts` files are never listed: editing a free test does not run paid evals. `test/touchfiles.test.ts`
|
||||
derives each paid file's static `test/helpers` / `test/fixtures` import closure, plus the fixture and helper
|
||||
paths it names in string literals, and fails when that closure is not covered by the case's key. When it
|
||||
fails, add the named path to the named key and check selection with
|
||||
`bun run scripts/test-paid-shards.ts --tier gate --profile pr --list`. The rule is a lower bound: a fixture
|
||||
path the test builds at runtime is not visible to it, so add such paths to the key by hand.
|
||||
|
||||
### CI
|
||||
|
||||
A GitHub Action (`.github/workflows/skill-docs.yml`) generates all hosts on pushes to main and on PRs, then rejects tracked differences and nonignored untracked output. Generation errors also fail the job. Optional ignored host caches are not compared against Git.
|
||||
|
||||
Reference in new issue
Block a user