mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-16 13:21:30 +01:00
docs(TODOS): close the absorbed test-infra items, file the overhaul follow-ups
Closed with receipts: the periodic coverage contract (implemented as full weekly coverage + exclusions), the eval-harness observability P1 (verified already landed: heartbeat, incremental _partial persistence, live stderr + eval-watch), and the sidebar trio (already deleted — tombstones remain). Filed: matrix deletion after parity, the required-check maintainer decision, browse /tmp-namespace hardening, PTY boot-readiness waits, the single typed test registry, bun-native LPT swap, runBin/free-runner migrations, eval-list partial exclusion, phantom key cleanup, duration-weighted slicing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b54d5eefb4
commit
683b16076a
1 file changed
+94
-3
@@ -353,7 +353,16 @@ touchfiles and re-offer pending ones on the next interactive run.
|
||||
false) permanently misses the artifacts-rename migration unless they paste the
|
||||
manual command. **Effort:** M. **Priority:** P2.
|
||||
|
||||
### P2: periodic tier — three documented-red tests need structural repair
|
||||
### P2: periodic tier — TWO documented-red tests need structural repair (was three)
|
||||
|
||||
**2026-08-29 update (test-infra overhaul):** (1) the sidebar E2E trio is
|
||||
ALREADY DELETED — no file in the tree POSTs to /sidebar-command or
|
||||
/sidebar-chat; only tombstone tests remain (browse/test/sidebar-tabs.test.ts
|
||||
asserts the endpoints STAY deleted), so part (1) closes as already-done.
|
||||
(2) skill-e2e-ship-idempotency and (3) skill-e2e-brain-privacy-gate are now
|
||||
EXCLUDED from the weekly lane with tracking
|
||||
(test/helpers/periodic-exclude-data.ts) — removing their entries re-activates
|
||||
them; the structural investigations below are the re-entry condition.
|
||||
|
||||
**What:** (1) The sidebar E2E trio (navigate, url-accuracy, css-interaction)
|
||||
POSTs to /sidebar-command and /sidebar-chat — endpoints removed on every tree
|
||||
@@ -416,6 +425,64 @@ references — include it in this fix's coverage list.
|
||||
|
||||
## Test infrastructure
|
||||
|
||||
### 2026-08-29 test-infra overhaul — follow-ups (filed at implementation)
|
||||
|
||||
The overhaul landed: green-means-green fixes (make-pdf gates in the required
|
||||
lane, zero-test eval jobs killed, 4 orphaned paid files activated + orphan
|
||||
tripwire, touchfiles self-registration + warn→fail), the serial
|
||||
tree-mutating shard dissolved (main() guard + --out-dir all hosts),
|
||||
duration-packed free shards, the sharded paid runner as the CI engine
|
||||
(planner/slices/fail-closed report, parity phase), the weekly all-periodic
|
||||
coverage contract + gate census, eval-budget timeout tiers, and the
|
||||
coverage fill. Remaining, in rough priority order:
|
||||
|
||||
- **P1 — Delete the legacy evals.yml matrix after parity.** The sliced lane
|
||||
runs alongside the 18-row matrix (`needs: evals`, so provider concurrency
|
||||
never doubles). After 1-2 PR cycles of parity (compare executed-test sets:
|
||||
intersection strict + the 8 KNOWN_MATRIX_GAPS files as expected additions;
|
||||
stochastic outcomes informational), delete the matrix as a PURE-DELETION
|
||||
commit (one revert restores it), drop the `needs: evals` edge, rewrite
|
||||
test/evals-workflow-matrix.test.ts into a runner-wiring pin, and retire
|
||||
KNOWN_MATRIX_GAPS/KNOWN_TIER_UNSET wholesale. Effort S.
|
||||
- **P1 — Maintainer decision: make `slices-report` a required check** once
|
||||
post-migration flake data exists (the Codex outside-voice's "green means
|
||||
green is not delivered while paid stays advisory" point — correct, and
|
||||
deliberately a branch-protection decision, not repo YAML). Effort S.
|
||||
- **P2 — browse daemon /tmp-namespace hardening.** Every file-path transport
|
||||
to the daemon (eval <file>, load-html --from-file, pdf output, upload,
|
||||
cookie-import) assumes client and daemon share one /tmp view; a sandboxed
|
||||
shell reusing an out-of-namespace daemon gets "File not found" on files it
|
||||
just wrote (root-caused live, reproduced with unshare). Minimal fix: the
|
||||
CLI reads a local `eval <file>` itself and sends the code as `js` (
|
||||
semantics-preserving; keep the daemon path for remote callers), plus a
|
||||
namespace hint appended to read-commands.ts:313's error. Effort S.
|
||||
- **P2 — PTY boot-readiness wait.** The PTY tests' Bun.sleep(8000) preludes
|
||||
and invokeAndObserve's 6s boot_grace_ms are blind waits; a real readiness
|
||||
waitFor needs empirical CLI 2.1.x ready-marker probing in a working
|
||||
terminal environment (this sandbox's PTY probe wedged). Effort S, needs a
|
||||
dev machine.
|
||||
- **P2 — single typed test registry.** Paid globs, tiers, touchfiles keys,
|
||||
and exclusions are still separate literal authorities synced by tripwires;
|
||||
derive them from one registry and the drift class dies structurally
|
||||
(outside-voice recommendation; the tripwires are the interim). Effort M.
|
||||
- **P2 — swap the custom LPT packer for bun-native `--timings`/`--shard`**
|
||||
at the next Bun unpin (native LPT scheduling ships ≥1.3.14; the packer is
|
||||
deliberately small and swappable — see the successor note in
|
||||
scripts/test-free-shards.ts). Effort S.
|
||||
- **P3 — runBin migration remainder** (~31 of 36 local run() duplicates;
|
||||
helper + first 3 migrated). Mechanical batches. Effort S.
|
||||
- **P3 — migrate the free runner onto runShardChild** (the shared lifecycle
|
||||
helper the paid runner now uses; designed for it). Effort S.
|
||||
- **P3 — eval-list should exclude _partial runs** (pinned as current
|
||||
behavior in test/eval-cli-family.test.ts with an improvement note).
|
||||
Effort S.
|
||||
- **P3 — codex-e2e-plan-format's testIfSelected names have no map keys**
|
||||
(run-all only today) + 15 E2E / 2 judge PHANTOM touchfiles keys select
|
||||
tests that exist nowhere — add keys or delete, one sweep. Effort S.
|
||||
- **P3 — duration-weighted slice assignment** if parity data shows slice
|
||||
walls diverging >1.5x (round-robin today; eval-store durations exist).
|
||||
Effort S.
|
||||
|
||||
### P2: /context-save worktree-identity hardening (the #2052 residual)
|
||||
|
||||
**What:** Persist a stable worktree identity (path hash or worktree name) into
|
||||
@@ -460,7 +527,19 @@ Trigger condition documented in `lib/gbrain-sources.ts` at the drift log line.
|
||||
|
||||
**Effort:** M (human ~1d, CC ~45min). **Depends on:** drift-log evidence from
|
||||
the wave's `ensureSourceRegistered` logging.
|
||||
### P2: Periodic CI matrix covers 9 of ~66 e2e files — decide the coverage contract
|
||||
### ✅ DONE (2026-08-29): Periodic CI coverage contract — implemented as option (a)
|
||||
|
||||
**Resolved by the test-infra overhaul:** evals-periodic.yml re-platformed onto
|
||||
scripts/test-paid-shards.ts — ALL periodic-tier files run weekly (EVALS_ALL,
|
||||
planner manifest → 6 slices → fail-closed report) minus the reasoned
|
||||
exclusions in test/helpers/periodic-exclude-data.ts (reason + tracking per
|
||||
entry, policy-pinned). A weekly EVALS_ALL gate census rides the same cron.
|
||||
The silent-rot class is dead: a test that runs nowhere is now either planned,
|
||||
diff-skipped, excluded-with-reason, or a failed report. Original filing kept
|
||||
below for the receipts.
|
||||
|
||||
#### Original filing (closed)
|
||||
Periodic CI matrix covers 9 of ~66 e2e files — decide the coverage contract
|
||||
|
||||
**Priority:** P2
|
||||
|
||||
@@ -498,7 +577,19 @@ in `.github/workflows/evals*.yml`. Receipts from the autoplan incident:
|
||||
`~/.gstack/projects/garrytan-gstack/e2e-runs/2026-07-10-0154/` (0-turn "Unknown command"
|
||||
transcripts).
|
||||
|
||||
### Eval harness: live progress + incremental result persistence (kill the silent hour)
|
||||
### ✅ DONE (verified 2026-08-29): Eval harness live progress + incremental persistence
|
||||
|
||||
**Verified landed** (the v1.66-era harness work delivered all three asks):
|
||||
(1) heartbeat — session-runner writes ~/.gstack-dev/e2e-live.json atomically
|
||||
per tool call (+ progress.log + per-test ndjson); (2) incremental persistence
|
||||
— EvalCollector writes _partial-e2e.json after every addTest, dual-signal
|
||||
isPartialEval keeps partials out of baselines; (3) live signal — per-tool
|
||||
stderr progress lines flush unbuffered, and scripts/eval-watch.ts dashboards
|
||||
the heartbeat. The 2026-08 overhaul added per-shard full-stream spool logs
|
||||
(path printed at START) on top. Original filing kept below for receipts.
|
||||
|
||||
#### Original filing (closed)
|
||||
Eval harness: live progress + incremental result persistence (kill the silent hour)
|
||||
|
||||
**Priority:** P1
|
||||
|
||||
|
||||
Reference in new issue
Block a user