From d8a62080dfc41722ebf8daa20408149b1a71eb5f Mon Sep 17 00:00:00 2001 From: garrytan Date: Sat, 31 Oct 2026 02:56:47 +0000 Subject: [PATCH] test: register skill-eval-checks in the test-of-test baseline; docs: describe the tightened checks (GST-377) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- changelog.d/gst-377-lanea-check-fixes.md | 1 + docs/designs/SKILL_EVAL_CONTRACT.md | 6 +++++- test/test-of-test-ratchet.test.ts | 1 + 3 files changed, 7 insertions(+), 1 deletion(-) diff --git a/changelog.d/gst-377-lanea-check-fixes.md b/changelog.d/gst-377-lanea-check-fixes.md index 6669fed68..b92487969 100644 --- a/changelog.d/gst-377-lanea-check-fixes.md +++ b/changelog.d/gst-377-lanea-check-fixes.md @@ -8,3 +8,4 @@ ### Gate integrity - These are grader edits that only tighten. Each check now fails runs it passed before, and no threshold, trial count or task was removed. The new `test/skill-eval-checks.test.ts` seeds each task with its own `seed.sh`, then runs the committed check after an honest action (it must pass) and after each hole (it must fail). On the old checks, all 15 hole cases passed the check and all 15 honest cases passed too. gstack-evals relocks its vendored copies from this commit. +- `test/skill-eval-checks.test.ts` joins the `BASELINE` in `test/test-of-test-ratchet.test.ts`. It imports no product module because no gstack module runs these check commands; the commands themselves are what it tests. diff --git a/docs/designs/SKILL_EVAL_CONTRACT.md b/docs/designs/SKILL_EVAL_CONTRACT.md index 21573f1f2..751798ca2 100644 --- a/docs/designs/SKILL_EVAL_CONTRACT.md +++ b/docs/designs/SKILL_EVAL_CONTRACT.md @@ -79,7 +79,11 @@ already assume, and one optional field. `GSTACK_HOME`. Set both to `/.gstack-home` for the agent, `setup` and `check`; `GSTACK_HOME` alone loses to a `GSTACK_STATE_ROOT` inherited from the caller's shell, and the agent then writes the caller's real state. A task's `seed.sh` creates `.gstack-home` and lists it in `.git/info/exclude`, so `git diff` checks ignore it and - `check` reads state there (/learn's `add` and `prune-stale`, /office-hours' design-doc and mockup checks). + `check` reads state there (/learn's `add` and `prune-stale`, /office-hours' design-doc and mockup checks). A + `check` that reads `.gstack-home` first requires it to be a real directory, not a symlink, because `find` does not + descend into a symlinked start point. /learn's checks read the one project folder named for the work dir, and + /office-hours' no-code check lists untracked files against `.git/info/exclude` alone, so a `.gitignore` the agent + writes hides nothing. - **`resumable`** (optional, in `verdict.json` and any error record): `true` when an INCOMPLETE run can be finished with `--resume `, which runs only cells that never ran, under the same lock and trial count, so it adds no trials after a result; `false` for a terminal stop, which needs a fix and a fresh run. A missing field reads as diff --git a/test/test-of-test-ratchet.test.ts b/test/test-of-test-ratchet.test.ts index 55e6f58e1..6f062fab2 100644 --- a/test/test-of-test-ratchet.test.ts +++ b/test/test-of-test-ratchet.test.ts @@ -223,6 +223,7 @@ const BASELINE = [ 'test/ship-release.test.ts', // GSTA-167 PR 4: runs bin/gstack-ship-release through test/helpers/ship-bin-fixture.ts's RELEASE path, so it names no bin/ literal itself 'test/ship-section-fixture.test.ts', 'test/skill-budget-regression.test.ts', + 'test/skill-eval-checks.test.ts', // GST-377: runs the committed skill-eval check commands, which gstack ships as data for gstack-evals' graders; no gstack module executes them 'test/skill-fixture.test.ts', // slices test/skill-routing-e2e.test.ts source (recorded 2026-10 when the ratchet learned paid-source slicing) 'test/skill-parser.test.ts', 'test/spawnsync-timeout-tripwire.test.ts',