evals2 follow-ups: outside-voice-evidence helper in five case touchfiles; close the evals lane's Codex-CI, late-input, batching and brain-writeback TODOs (run 37162480720); CHANGELOG lines

CHANGELOG.md

@@ -55,6 +55,7 @@ The first two rows are the ones to care about: a review that read nothing, or fo
 - **Hand edits to installed router copies are saved to `~/.gstack/backups/skill-copies/`, then overwritten.** Setup prints where.
 - **In Conductor, setup no longer installs the AskUserQuestion preference hook** and removes one it added. To keep it: `gstack-config set plan_tune_hooks yes`, then `./setup`.
 - **Required team mode also blocks Copilot CLI's skill tool.** Re-run `gstack-team-init required` to upgrade the project hook.
+- **`gstack-qa-evidence capture` prints a `revalidate` list after your inputs change**, naming the probes to rerun on current inputs before `materialize`; `materialize` says its verdict is final for that report root, so rerun the listed probes first.
 - **/gstack-upgrade asks you to paste the absolute path Step 2 prints**; every later step checks it is inside gstack's checkout before running git.
 
 ### Security
@@ -97,7 +98,7 @@ The first two rows are the ones to care about: a review that read nothing, or fo
 - Eval children no longer receive credential env vars that carry a suffix (`GITHUB_TOKEN_1`, `GITHUB_APP_PRIVATE_KEY_BASE64`); admit one explicitly with `extraAllow` (#2949). Chunked large-push coverage for the pre-push guard (#2915).
 - The test typecheck ratchet ignores the order tsc prints an object type's members in.
 - The weekly eval report counts only cases that ran: tests deselected for another tier or case shard are listed by reason with no credit, every result is attributed to its case (0 unattributed, was 72), and no PR-lane eval shard runs over 10 minutes (the deploy eval is split per case).
-- Codex evals pass only when Codex completed a command; a review whose sandbox failed now fails instead of passing. The AUQ matrix checks the `(recommended)` marker on option labels, where the auto-decide hook reads it.
+- Codex evals run on their own CI job where Codex's sandbox can start, and pass only when Codex completed a command; a review whose sandbox failed now fails instead of passing. The `/codex review` eval passes only when Codex actually reviewed the diff. The AUQ matrix checks the `(recommended)` marker on option labels, where the auto-decide hook reads it.
 
 Contributed by @CermakM (#2843), @Haijie-Lee (#2192), @SomSamantray (#2965), @Vaughan-g-aus (#2329), @BharadwajMittapelli (#2729), @maxpetrusenkoagent (#1931, #1932, #1981), @Infiniteyieldai (#2884), @afshaker (#2778), @time-attack (#2232, #2229), @jzeisweiss (#2670), @liutiming (#2783), @agile-operators (#1437), @benjaminberes-bp (#2369), @lewispeel (#2885), @Yugz29 (#2899), @fcmerle (#2962), @cdecook23 (#2784), @mb1810 (#2913), @ntdatt812 (#2949), @pneumorea-ai (#2915), @gabrielrondon (#2811), @kevingrasso32 (#2876), @xwang4-svg (#2354), @DizzyWesterwald (#2936), @oli548 (#2771), @shofel (#1968), @ryanlaiwy (#1633), @yinanli1917-cloud (#2281), @Spooks444 (#2260), @punksterlabs (#2439), @himiaocc (#2492), @jbetala7 (#1419, #1777), @mamedov (#2891), @Lockyer228 (#2651), @topcoder1 (#2263), @jonahberg (#2707), @AlinValentin7 (#3015), @philhie (#2208), @Alchemist-DevAI (#2334), @merit-blake (#2343), @day-of-davon (#2952), @loulanyue (#2797, #2792, #2794, #2796), @Bit0ps (#2840), @szsunyuan (#2799), @frosimanuel (#2818), @JiayuuWang (#1096), @RyanAlberts (#1728), @exGeni (#2704), @Jino00 (#2812), @snig-17 (#2809) and @SholtoMc (#2037).
 

TODOS.md

@@ -75,34 +75,19 @@ Deferred from the approved severe fix wave (docs/designs/SEVERE_FIX_WAVE_2026_10
   (`GIT_OBJECT_DIRECTORY` + `GIT_ALTERNATE_OBJECT_DIRECTORIES` under
   `$TMPDIR`) for the fingerprint, and report the fetch as stale-base coverage.
   **Effort:** S. **Priority:** P2.
-- **P1: Codex paid evals cannot run commands in CI** — Docker's default
-  seccomp blocks bubblewrap's user namespace; on Ubicloud hosts
-  (`kernel.apparmor_restrict_unprivileged_userns=1`) `seccomp=unconfined` gives
-  "bwrap: Failed to make / slave", and adding `apparmor=unconfined` gives
-  "loopback: Failed RTM_NEWADDR". A stock Ubicloud VM ran `codex sandbox` only
-  with host `sysctl kernel.apparmor_restrict_unprivileged_userns=0` plus both
-  `--security-opt` options. Container jobs cannot set the host sysctl, so the
-  Codex cases need a host-run job or different isolation (in progress as the
-  wave/evals2 lane). Also audit every Codex case's validator for execution
-  evidence (shared-libs-codex-read-only and outside-voice-codex-to-claude-code
-  passed in 37158847998 with commands possibly failing). Runs 37151477069,
-  37158847998. **Effort:** M. **Priority:** P1.
-- **plan-eng-review-artifact skips the QA test plan** — red in 37151477069 and
-  37158847998: the model writes review-output.md without any Bash call, so the
-  artifact's gstack-paths/gstack-slug block never runs. Removing the "write
-  only when resolved" contradiction did not fix it. Next: check whether the
-  fixture prompt ("skip the preamble bash block", one named output) suppresses
-  the artifact; one paid rerun per change. **Priority:** P2.
-- **office-hours-brain-writeback turn budget** — 14/18 trials across six runs;
-  every failure is `error_max_turns` at 12 turns with `gbrain search` but no
-  `gbrain put`, after chunked re-reads of the 1260-line SKILL.md that Bash output
-  truncation forces. The budget must not rise: add reading guidance or decide
-  whether a smaller rendered fixture still tests the point. **Priority:** P2.
-- **ship-exploratory-late-input (gate) returns `inconclusive` in 2 of 8
-  censuses** (36920606897, 37158847998). It can redden the PR lane;
-  characterize it before relying on the lane. **Priority:** P2.
-- **plan-eng-multi-finding-batching** — red in 6 of 12 runs read (batching:
-  reviewCount 0-1, or Step 0 exhausting the 1495 s budget). **Priority:** P2.
+- **plan-eng-review-artifact skips the QA test plan (0/4 since #3017)** —
+  red in 37151477069, 37158847998, 37162480720 and one local CI-image run.
+  Every attempt reads SKILL.md, sections, plan and sources, writes the whole
+  review into review-output.md in one Write, runs no Bash, and never runs the
+  Test Plan Artifact block. A prompt destination rewording and a self-check
+  rewording each got one paid rerun and did not move it. Options: put the
+  artifact instruction back in the prompt, move the assertion into a PTY
+  full-workflow case, or quarantine with this evidence. **Priority:** P2.
+- **Closed by the wave's evals2 lane (run 37162480720):** Codex paid evals now
+  run commands in CI (host-run job; all 8 Codex cases executed real commands),
+  ship-exploratory-late-input no longer ends `inconclusive` (capture names the
+  probes to rerun after an input change), plan-eng-multi-finding-batching's
+  checker and office-hours-brain-writeback's prompt are fixed (3/3).
 - **Gate-outcome telemetry** — count `not_run`/`unavailable` outcomes per
   reason code from `lib/gate-outcomes.ts` so the most common gaps get fixed
   first. **Effort:** S. **Priority:** P3.

test/helpers/touchfiles-data.ts

@@ -624,7 +624,7 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
   'ship-base-branch': [ 'bin/gstack-version-bump','bin/gstack-state-root.sh', 'lib/state-root.ts', 'ship/**', 'bin/gstack-repo-mode', 'test/skill-e2e-review-attribution.test.ts',
     'scripts/resolvers/testing.ts'
   ],
-  'ship-local-workflow': [ 'bin/gstack-version-bump', 'ship/**', 'scripts/gen-skill-docs.ts', 'test/skill-e2e-workflow.test.ts',
+  'ship-local-workflow': [ 'test/helpers/outside-voice-evidence.ts', 'bin/gstack-version-bump', 'ship/**', 'scripts/gen-skill-docs.ts', 'test/skill-e2e-workflow.test.ts',
     'scripts/resolvers/testing.ts', 'test/fixtures/coverage-audit-fixture.ts', 'test/helpers/coverage-audit.ts', 'test/helpers/coverage-audit-evidence.ts', 'test/helpers/office-hours-attempt.ts'],
   'review-dashboard-via': [ 'ship/**', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'codex/**', 'autoplan/**', 'land-and-deploy/**', 'test/skill-e2e-review-attribution.test.ts',
     'scripts/resolvers/testing.ts'
@@ -663,10 +663,10 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
   'context-save-list-all-branches':        [ 'context-save/**', 'test/skill-e2e-context-skills.test.ts', 'bin/gstack-skill-start', 'test/helpers/shipped-skill-routing.ts'],
 
   // Document-release
-  'document-release': [ 'bin/gstack-version-bump', 'document-release/**', 'test/skill-e2e-workflow.test.ts', 'test/fixtures/coverage-audit-fixture.ts', 'test/helpers/coverage-audit.ts', 'test/helpers/coverage-audit-evidence.ts', 'test/helpers/office-hours-attempt.ts'],
+  'document-release': [ 'test/helpers/outside-voice-evidence.ts', 'bin/gstack-version-bump', 'document-release/**', 'test/skill-e2e-workflow.test.ts', 'test/fixtures/coverage-audit-fixture.ts', 'test/helpers/coverage-audit.ts', 'test/helpers/coverage-audit-evidence.ts', 'test/helpers/office-hours-attempt.ts'],
 
   // Codex (Claude E2E — tests /codex skill via Claude)
-  'codex-review': [ 'lib/outside-review-result.ts', 'lib/gate-outcomes.ts', 'codex/**', 'test/skill-e2e-workflow.test.ts', 'test/fixtures/coverage-audit-fixture.ts', 'test/helpers/coverage-audit.ts', 'test/helpers/coverage-audit-evidence.ts', 'test/helpers/office-hours-attempt.ts', 'bin/gstack-codex-probe', 'scripts/resolve-codex-generation-model.ts', 'scripts/resolvers/constants.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/outside-voice-steps.ts'],
+  'codex-review': [ 'test/helpers/outside-voice-evidence.ts', 'lib/outside-review-result.ts', 'lib/gate-outcomes.ts', 'codex/**', 'test/skill-e2e-workflow.test.ts', 'test/fixtures/coverage-audit-fixture.ts', 'test/helpers/coverage-audit.ts', 'test/helpers/coverage-audit-evidence.ts', 'test/helpers/office-hours-attempt.ts', 'bin/gstack-codex-probe', 'scripts/resolve-codex-generation-model.ts', 'scripts/resolvers/constants.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/outside-voice-steps.ts'],
 
   // Codex E2E (tests skills via Codex CLI + worktree)
   'codex-discover-skill':  [ 'lib/outside-review-result.ts', 'lib/gate-outcomes.ts', 'setup','codex/**', 'scripts/gen-skill-docs.ts', 'test/helpers/codex-session-runner.ts', 'lib/worktree.ts', 'test/codex-e2e.test.ts',
@@ -718,7 +718,7 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
 
 
   // Coverage audit (shared fixture) + triage + gates
-  'ship-coverage-audit': ['bin/gstack-state-root.sh', 'lib/state-root.ts', "test/fixtures/coverage-audit-aw.json",
+  'ship-coverage-audit': [ 'test/helpers/outside-voice-evidence.ts','bin/gstack-state-root.sh', 'lib/state-root.ts', "test/fixtures/coverage-audit-aw.json",
 
     
     "test/fixtures/coverage-checkbox-tail-av.json",
@@ -818,7 +818,7 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
   'diagram-authoring-quality':  ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'diagram/**', 'lib/diagram-render/**', 'lib/aside-render.ts', 'bin/gstack-render.ts', 'test/helpers/aside-available.ts', 'browse/src/**', 'test/helpers/llm-judge.ts', 'test/skill-e2e-diagram.test.ts'],
 
   // gstack-upgrade
-  'gstack-upgrade-happy-path': [ 'setup', 'bin/gstack-relink', 'bin/gstack-session-update', 'gstack-upgrade/**', 'test/skill-e2e-workflow.test.ts', 'test/fixtures/coverage-audit-fixture.ts', 'test/helpers/coverage-audit.ts', 'test/helpers/coverage-audit-evidence.ts', 'test/helpers/office-hours-attempt.ts'],
+  'gstack-upgrade-happy-path': [ 'test/helpers/outside-voice-evidence.ts', 'setup', 'bin/gstack-relink', 'bin/gstack-session-update', 'gstack-upgrade/**', 'test/skill-e2e-workflow.test.ts', 'test/fixtures/coverage-audit-fixture.ts', 'test/helpers/coverage-audit.ts', 'test/helpers/coverage-audit-evidence.ts', 'test/helpers/office-hours-attempt.ts'],
 
   // Deploy skills
   'land-and-deploy-workflow':      [ 'bin/gstack-version-bump', 'land-and-deploy/**', 'scripts/gen-skill-docs.ts', 'test/skill-e2e-deploy.test.ts'],