Commit Graph
8 Commits
Author SHA1 Message Date
Garry Tanandcapy-ai[bot] 119ed92f9c Fix seven test and eval checks that misread correct work or raced their neighbours (GST-2131) (#4223)
* Fix seven test and eval checks that misread correct work or raced their neighbours (GST-2131)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Record the harness decision for the flaky-pass helpers (GST-2131)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Register the captured scrape fixture as free (GST-2131)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Point the whole codex-review excerpt at the checkout and stop counting refused or conditional defers (GST-2131)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Make the flaky-pass changelog entry read plainly and name lib paths (GST-2131)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-09 15:54:27 +00:00
Garry Tanandcapy-ai[bot] 6417abf7a6 Move the tools only Capy crews use into a crew/ folder (GST-1833) (#4145)
* Load the Capy crew skills only on Capy machines (GST-1833)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Add changelog entry for Capy-only crew skills (GST-1833)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Move the Capy-only crew tooling into crew/ and run its tests only when needed (GST-1833)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Test that the Capy launch file puts crew/bin on PATH

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Count any ~/.capy as a Capy machine, so a Mac running Capy keeps /captain

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Fix the tests and fixtures that broke when crew tooling moved into crew/

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Plan every crew test when a pull request's merge-base can't be read

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Add a test that checks each crew-only module left the root tree for crew/

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Run crew tests on pull requests that change bin/ commands or workflows they drive

Review-Finding: p1/specialist-batched#1 reader=testing
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Keep running every crew test on every pull request; the skip comes in a later change

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Keep copied installs and older release bisects working after the crew/ move

Setup now copies crew/lib into every host's runtime root beside bin and lib, so Windows installs (which copy instead of linking) can still start gstack-ship-receipt, gstack-pregate and the other commands that import it. The release bisect runs the train-gate script from wherever the checked-out commit has it, so a bisect over commits from before the move still gets a verdict.

Review-Finding: p1/outside-voice#1 reader=outside-voice
Review-Finding: p1/outside-voice#2 reader=outside-voice
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Say in the changelog that Windows installs keep working after the crew/ move

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Point the harness guide's merge-queue rows at crew/lib/ where those modules moved

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Point the plain-title test at the wave module's new home in crew/ (GST-1833)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-08 14:26:14 +00:00
Garry Tanandcapy-ai[bot] 52d6c46957 test(flaky): marathon census reds on 297f50c7 (gh telemetry in fixture HOME, /autoplan handoff) and two free flakes (GST-1793) (#4128)
* test: pin GH_TELEMETRY=0 in hermetic eval env; office-hours handoff accepts /autoplan (flaky pass 2026-11-07)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: release-cut slow cases get the 30 s free-suite timeout; ruled-notes grant names both shas after review_loop_grant (flaky pass 2026-11-07)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* changelog: flaky pass 2026-11-07 fragment

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: record the hermetic-env.ts GH_TELEMETRY pin as non-behavioral at HARNESS_VERSION 6 (flaky pass 2026-11-07)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: pre-landing review fixes

Review-Finding: p1/specialist-batched#1 reader=testing
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-07 15:25:35 +00:00
Garry Tanandcapy-ai[bot] 94392a61ec test: harness targets skip retired skills; record the standalone /plan-eng-review paid cases' fates (GST-1698) (#4111)
* test: harness targets skip retired skills; record each standalone /plan-eng-review paid case's fate (GST-1698)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: mark retired /plan-eng-review cases in the detector inventory and TODOS (GST-1698)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-07 06:02:49 +00:00
Garry Tanandcapy-ai[bot] 55637db835 test: cut the last two behavior-guessing LLM judges; check review-bot history facts for free (GST-1403) (#4014)
* test: cut the qa anti-refusal and voice-tone judges; check review-bot history facts for free (GST-1403)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: count assertions follow the deleted judge file (GST-1403)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-05 02:30:04 +00:00
Garry Tanandcapy-ai[bot] 91ddd3bdb1 fix(evals): the Codex removed-rule arm really removes the rule from installed skills (GST-840) (#3816)
* fix(evals): Codex removed-rule arm strips the rule from the installed skills (GST-840)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(evals): copy linked directories inside a skill before stripping the rule; record the harness decision (GST-840)

/ship review found readdirSync({ recursive: true }) follows linked
subdirectories, so a rule file under a linked directory inside a real
skill directory would have been deleted and rewritten in the checkout.
Every linked directory on the way to a rule file is now copied first,
with a failing-first test row. The harness-version manifest records
codex-host-runner.ts as non-behavioral: only the manual removed arm
on the Codex host changes.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: keep the Codex safety-arm test off Windows, and spawn its probe with bunIsolationArgs (GST-840)

windows-free-shard 3 (run 41336347706) failed every case: native Bun on
Windows cannot open --config=/dev/null, and the test builds a symlinked
~/.codex/skills install that needs Developer Mode the runners lack.
./setup --host codex copies instead of linking on Windows, so the strip
only matters on POSIX. The probe now uses bunIsolationArgs().

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-05 00:29:18 +00:00
Garry Tanandcapy-ai[bot] 781f928ccf test: remove the AI scorer that grades skill prose for clarity (freeze repair, GST-1371) (#4009)
* test: remove the skill-prose clarity LLM judges (GST-1371)

Removes the 21 eval cases where an LLM scored a skill's instruction text for
clarity, completeness and actionability, with their registrations, input
builders, response cache, calibration corpora, the cookie-workflow manual-review
approval and scripts/judge-controls.ts. The three remaining skill-llm-eval
judges keep their prompts, thresholds and samples; the file's supervision wall
shrinks to fit them. Outcome judges are untouched.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: pin the retired judge-control script and cookie manual-review record as deleted (GST-1371)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: register the manual-review fixture as free (GST-1371)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: keep the orphaned clarity-judge calibration corpora for a follow-up deletion (GST-1371)

Deleting the four corpus directories alone puts the diff over the /ship review
ceiling; they serve no configuration now and go in a follow-up.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs(todos): file the clarity-judge removal follow-ups (GST-1371)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: keep skill-bundle section paths POSIX so references dedupe on Windows (GST-1371)

windows-scoped-shard (run 41918796523) failed 'cross-skill references retain
complete bytes once': path.join gave the section entry a backslash path, so a
reference to the same section was not recognized as already bundled.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-04 22:19:08 +00:00
Garry Tanandcapy-ai[bot] a1da77e6d4 v1.94.20.0 test(review): one file per review and harness pin, so independent repins merge cleanly (GST-1084) (#3903)
* test(review): one file per review and harness pin, so independent repins merge cleanly (GST-1084)

The reviewer-brief pins move from the table in test/review-packets-removed.test.ts to
test/fixtures/review-pins/{files,replays,brief-bodies}/*.json, and the harness pins from
scripts/harness-version.json to scripts/harness-pins/<path>.json. Every entry is ported
unchanged. New tests prove a stale pin still fails and that two repins of different pins
merge cleanly with a real git merge-tree.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: pre-landing review fixes (GST-1084)

- unadmitted() and the specialist coverage use Object.hasOwn, so an inherited name such as toString is not an admitted replay
- MUST_PIN names every non-specialist reviewer source; deleting one of those pins now fails
- writeManifest writes the pins before the manifest head; pin parse errors name the file
- mergePinEdits removes its repository when setup throws, and the merge tests clean up every merge they made

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: name the changelog fragment for the branch (GST-1084)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: the harness guard pins one file per harness file (GST-1084)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(ship-release): GST-1031 re-pin route reads per-file harness pins (GST-1084)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* changelog: GST-1031 route reads per-file harness pins

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs(test-audit): harness guard pins 42 files

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(harness-version): a pin ahead of the manifest version counts as drift (GST-1084)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-03 15:04:56 +00:00