mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-05 00:03:15 +01:00
Merge origin/main (v1.91.15.0, #3017) into garrytan/parallel-waves; release metadata v1.91.16.0
Conflicts resolved by hand in templates, tests, CLAUDE.md, TODOS.md, CHANGELOG.md and bin/gstack-config; generated SKILL.md/sections, the agents digest and test/fixtures/context-budget.json regenerated. The eval codemod re-applied to main's new gstack-paths/gstack-slug evals. Coverage-read rule: bare echo separators after unknown output extend the unknown span (main's pathspec CI shape). CLAUDE.md keeps the TREE_MUTATING, catalog-ceiling and detach-timeout facts main's instruction-facts test pins.
This commit is contained in:
commit
6fb8ea3d10
351 files changed
+7477
-8179
No files matched your search
@@ -19,7 +19,8 @@ or as a reference doc, (3) only compress carefully-tuned prose as a last resort
|
||||
cuts to the coverage audit, review army, or voice directive have real quality cost.
|
||||
|
||||
A second, harder ceiling guards the DISCOVERY surface: `test/catalog-budget.test.ts`
|
||||
caps the aggregate frontmatter `name` + `description` across all skills at 1,171
|
||||
caps the aggregate frontmatter `name` + `description` across all skills at
|
||||
`CATALOG_BUDGET_TOKEN_EQUIVALENTS` (1,194 today; each new skill ratchets it)
|
||||
token-equivalents (260-byte per-skill sub-cap), counted through the shared census
|
||||
in `test/helpers/skill-census.ts`. This one is enforced, not a warning — every
|
||||
host loads the full catalog every session, so growth here taxes every
|
||||
|
||||
@@ -7,7 +7,7 @@ blocking efficacy contract; historical failures retain their original verdicts.
|
||||
|
||||
The contract is defined by `OVERLAY_CONTRACT` in
|
||||
`test/helpers/overlay-case-policy.ts`. Every new measurement and aggregate record
|
||||
identifies version 2. Eval entry names include `contract-v2` so automated result
|
||||
identifies version 3. Eval entry names include `contract-v3` so automated result
|
||||
matching cannot silently compare the old and new contracts by the same name.
|
||||
Selection IDs for the six retained paid cases stay unchanged.
|
||||
|
||||
@@ -26,7 +26,7 @@ observations are never replaced with zero or discarded to obtain a passing arm.
|
||||
|
||||
| Fixture family | Exact task and scope contract | Additional ON requirement | Measurement |
|
||||
| --- | --- | --- | --- |
|
||||
| `claude-dedicated-tools-vs-bash` | Read-only workspace; native final JSON maps all five `src/` paths to their exact exported symbol names | Zero Bash calls in every ON sample | Total Bash tool calls |
|
||||
| `claude-dedicated-tools-vs-bash` | Read-only workspace; native final JSON maps all five `src/` paths to their exact exported symbol names | Correct output in every ON sample, and at least 20% fewer Bash calls than OFF (when the OFF median is 0, zero Bash in every ON sample) | Total Bash tool calls |
|
||||
| `opus-4-7-effort-match-trivial` | Read-only workspace; native final JSON contains the exact version string | Correct answer in every ON sample | Native `usage.output_tokens_details.thinking_tokens` |
|
||||
| `opus-4-7-literal-interpretation` | Only `src/auth.ts`, `src/billing.ts`, and `src/notifications.ts` may change; public tests stay frozen; independent behavior checks run outside the writable fixture | All three target behaviors pass in every ON sample | Correct target behaviors, 0..3 |
|
||||
|
||||
@@ -39,7 +39,8 @@ metric to historical counts of edited files.
|
||||
|
||||
OFF literal samples can validly complete 0..3 behaviors; that is the measured
|
||||
control variable. OFF dedicated-tool samples can validly use Bash. ON must meet
|
||||
the exact requirements above even when its comparison shows a large improvement.
|
||||
the exact requirements above even when its comparison shows a large improvement;
|
||||
for the dedicated-tools case the 20% Bash comparison itself is part of the gate.
|
||||
Scope is checked again after the implementation oracle because importing a
|
||||
repaired module can itself change files. Each oracle must emit its unique
|
||||
completion marker after its assertions; an implementation exiting zero during
|
||||
|
||||
@@ -689,9 +689,10 @@ shard processes, serial within each, packed by recorded per-file durations
|
||||
when `scripts/free-test-durations.json` exists — refresh occasionally with
|
||||
`bun run test:free --record-durations`; strict-output classification per
|
||||
shard: a shard without bun's terminal summary line FAILS — silent truncation
|
||||
cannot report green). The former trailing serial tree-mutating shard is
|
||||
gone: `TREE_MUTATING` is empty (gen-skill-docs has a main() guard and
|
||||
`--out-dir` renders every host, so tests render into mkdtemps — see
|
||||
cannot report green). `TREE_MUTATING` lists the files that still run in their
|
||||
own trailing serial shard (today only `test/bootstrap-retention.test.ts`, for
|
||||
host-wide procfs visibility); gen-skill-docs tests no longer need it (main()
|
||||
guard, and `--out-dir` renders every host into mkdtemps — see
|
||||
docs/TESTING_INTERNALS.md). Never type bare `bun test` for the suite: it
|
||||
walks the whole repo, loading paid eval files and missing the strict
|
||||
classifier.
|
||||
@@ -760,7 +761,8 @@ the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session
|
||||
(stray `claude`/`codex` grandchildren included), a per-shard
|
||||
`GSTACK_EVAL_DIR=<evalDir>/shards/<slug>/` honored by the `EvalCollector`
|
||||
constructor, and an aggregate that separates failed vs timed-out vs
|
||||
never-started shards — the detach timeouts (47340s gate / 67380s periodic;
|
||||
never-started shards — the detach timeouts (the `--timeout` values on
|
||||
package.json's `eval:bg:gate` / `eval:bg:periodic`;
|
||||
floor enforced against the live shard census by
|
||||
test/eval-detach-timeout-floor.test.ts)
|
||||
are sized against worst-case shard wall clock. `EVALS_JOBS` sets the shard
|
||||
@@ -773,7 +775,10 @@ the run can also die to idle-sleep. `gstack-detach` fixes both: a fresh session
|
||||
- Then **poll the printed logfile** with a death-aware watcher: break on the
|
||||
guaranteed `### gstack-detach EXIT=<code> ###` sentinel (success AND failure are
|
||||
both marked, so silence is never mistaken for success). The detached run survives
|
||||
even if your watcher gets reaped, so re-checking the log always works.
|
||||
even if your watcher gets reaped, so re-checking the log always works. Keep
|
||||
checking until the sentinel appears or the user tells you to stop; a long run is
|
||||
expected, and a promise to check later is not a result. At each check, report
|
||||
which tests passed, which are still running, and any failures so far.
|
||||
- Why the lock: a shared dev box with several Conductor worktrees will rate-limit
|
||||
the model API if two eval suites run at once (15-way concurrency each), which
|
||||
mass-times-out E2E tests. The lock makes the second run WAIT, not collide.
|
||||
|
||||
Reference in new issue
Block a user