Merge origin/main (v1.91.15.0, #3017) into garrytan/parallel-waves; release metadata v1.91.16.0

Conflicts resolved by hand in templates, tests, CLAUDE.md, TODOS.md,
CHANGELOG.md and bin/gstack-config; generated SKILL.md/sections, the agents
digest and test/fixtures/context-budget.json regenerated. The eval codemod
re-applied to main's new gstack-paths/gstack-slug evals. Coverage-read rule:
bare echo separators after unknown output extend the unknown span (main's
pathspec CI shape). CLAUDE.md keeps the TREE_MUTATING, catalog-ceiling and
detach-timeout facts main's instruction-facts test pins.
This commit is contained in:
garrytan committed 2026-10-03 08:53:18 +00:00
commit 6fb8ea3d10
351 files changed
+7477 -8179

No files matched your search

+2 -1
View File
@@ -19,7 +19,8 @@ or as a reference doc, (3) only compress carefully-tuned prose as a last resort
cuts to the coverage audit, review army, or voice directive have real quality cost.
A second, harder ceiling guards the DISCOVERY surface: `test/catalog-budget.test.ts`
caps the aggregate frontmatter `name` + `description` across all skills at 1,171
caps the aggregate frontmatter `name` + `description` across all skills at
`CATALOG_BUDGET_TOKEN_EQUIVALENTS` (1,194 today; each new skill ratchets it)
token-equivalents (260-byte per-skill sub-cap), counted through the shared census
in `test/helpers/skill-census.ts`. This one is enforced, not a warning — every
host loads the full catalog every session, so growth here taxes every
+4 -3
View File
@@ -7,7 +7,7 @@ blocking efficacy contract; historical failures retain their original verdicts.
The contract is defined by `OVERLAY_CONTRACT` in
`test/helpers/overlay-case-policy.ts`. Every new measurement and aggregate record
identifies version 2. Eval entry names include `contract-v2` so automated result
identifies version 3. Eval entry names include `contract-v3` so automated result
matching cannot silently compare the old and new contracts by the same name.
Selection IDs for the six retained paid cases stay unchanged.
@@ -26,7 +26,7 @@ observations are never replaced with zero or discarded to obtain a passing arm.
| Fixture family | Exact task and scope contract | Additional ON requirement | Measurement |
| --- | --- | --- | --- |
| `claude-dedicated-tools-vs-bash` | Read-only workspace; native final JSON maps all five `src/` paths to their exact exported symbol names | Zero Bash calls in every ON sample | Total Bash tool calls |
| `claude-dedicated-tools-vs-bash` | Read-only workspace; native final JSON maps all five `src/` paths to their exact exported symbol names | Correct output in every ON sample, and at least 20% fewer Bash calls than OFF (when the OFF median is 0, zero Bash in every ON sample) | Total Bash tool calls |
| `opus-4-7-effort-match-trivial` | Read-only workspace; native final JSON contains the exact version string | Correct answer in every ON sample | Native `usage.output_tokens_details.thinking_tokens` |
| `opus-4-7-literal-interpretation` | Only `src/auth.ts`, `src/billing.ts`, and `src/notifications.ts` may change; public tests stay frozen; independent behavior checks run outside the writable fixture | All three target behaviors pass in every ON sample | Correct target behaviors, 0..3 |
@@ -39,7 +39,8 @@ metric to historical counts of edited files.
OFF literal samples can validly complete 0..3 behaviors; that is the measured
control variable. OFF dedicated-tool samples can validly use Bash. ON must meet
the exact requirements above even when its comparison shows a large improvement.
the exact requirements above even when its comparison shows a large improvement;
for the dedicated-tools case the 20% Bash comparison itself is part of the gate.
Scope is checked again after the implementation oracle because importing a
repaired module can itself change files. Each oracle must emit its unique
completion marker after its assertions; an implementation exiting zero during
+10 -5
View File
@@ -689,9 +689,10 @@ shard processes, serial within each, packed by recorded per-file durations
when `scripts/free-test-durations.json` exists — refresh occasionally with
`bun run test:free --record-durations`; strict-output classification per
shard: a shard without bun's terminal summary line FAILS — silent truncation
cannot report green). The former trailing serial tree-mutating shard is
gone: `TREE_MUTATING` is empty (gen-skill-docs has a main() guard and
`--out-dir` renders every host, so tests render into mkdtemps — see
cannot report green). `TREE_MUTATING` lists the files that still run in their
own trailing serial shard (today only `test/bootstrap-retention.test.ts`, for
host-wide procfs visibility); gen-skill-docs tests no longer need it (main()
guard, and `--out-dir` renders every host into mkdtemps — see
docs/TESTING_INTERNALS.md). Never type bare `bun test` for the suite: it
walks the whole repo, loading paid eval files and missing the strict
classifier.
@@ -760,7 +761,8 @@ the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session
(stray `claude`/`codex` grandchildren included), a per-shard
`GSTACK_EVAL_DIR=<evalDir>/shards/<slug>/` honored by the `EvalCollector`
constructor, and an aggregate that separates failed vs timed-out vs
never-started shards — the detach timeouts (47340s gate / 67380s periodic;
never-started shards — the detach timeouts (the `--timeout` values on
package.json's `eval:bg:gate` / `eval:bg:periodic`;
floor enforced against the live shard census by
test/eval-detach-timeout-floor.test.ts)
are sized against worst-case shard wall clock. `EVALS_JOBS` sets the shard
@@ -773,7 +775,10 @@ the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session
- Then **poll the printed logfile** with a death-aware watcher: break on the
guaranteed `### gstack-detach EXIT=<code> ###` sentinel (success AND failure are
both marked, so silence is never mistaken for success). The detached run survives
even if your watcher gets reaped, so re-checking the log always works.
even if your watcher gets reaped, so re-checking the log always works. Keep
checking until the sentinel appears or the user tells you to stop; a long run is
expected, and a promise to check later is not a result. At each check, report
which tests passed, which are still running, and any failures so far.
- Why the lock: a shared dev box with several Conductor worktrees will rate-limit
the model API if two eval suites run at once (15-way concurrency each), which
mass-times-out E2E tests. The lock makes the second run WAIT, not collide.