diff --git a/CHANGELOG.md b/CHANGELOG.md index 9b8d67b9f..9634eebaa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -55,6 +55,7 @@ The first two rows are the ones to care about: a review that read nothing, or fo - **Hand edits to installed router copies are saved to `~/.gstack/backups/skill-copies/`, then overwritten.** Setup prints where. - **In Conductor, setup no longer installs the AskUserQuestion preference hook** and removes one it added. To keep it: `gstack-config set plan_tune_hooks yes`, then `./setup`. - **Required team mode also blocks Copilot CLI's skill tool.** Re-run `gstack-team-init required` to upgrade the project hook. +- **`gstack-qa-evidence capture` prints a `revalidate` list after your inputs change**, naming the probes to rerun on current inputs before `materialize`; `materialize` says its verdict is final for that report root, so rerun the listed probes first. - **/gstack-upgrade asks you to paste the absolute path Step 2 prints**; every later step checks it is inside gstack's checkout before running git. ### Security @@ -97,7 +98,7 @@ The first two rows are the ones to care about: a review that read nothing, or fo - Eval children no longer receive credential env vars that carry a suffix (`GITHUB_TOKEN_1`, `GITHUB_APP_PRIVATE_KEY_BASE64`); admit one explicitly with `extraAllow` (#2949). Chunked large-push coverage for the pre-push guard (#2915). - The test typecheck ratchet ignores the order tsc prints an object type's members in. - The weekly eval report counts only cases that ran: tests deselected for another tier or case shard are listed by reason with no credit, every result is attributed to its case (0 unattributed, was 72), and no PR-lane eval shard runs over 10 minutes (the deploy eval is split per case). -- Codex evals pass only when Codex completed a command; a review whose sandbox failed now fails instead of passing. The AUQ matrix checks the `(recommended)` marker on option labels, where the auto-decide hook reads it. +- Codex evals run on their own CI job where Codex's sandbox can start, and pass only when Codex completed a command; a review whose sandbox failed now fails instead of passing. The `/codex review` eval passes only when Codex actually reviewed the diff. The AUQ matrix checks the `(recommended)` marker on option labels, where the auto-decide hook reads it. Contributed by @CermakM (#2843), @Haijie-Lee (#2192), @SomSamantray (#2965), @Vaughan-g-aus (#2329), @BharadwajMittapelli (#2729), @maxpetrusenkoagent (#1931, #1932, #1981), @Infiniteyieldai (#2884), @afshaker (#2778), @time-attack (#2232, #2229), @jzeisweiss (#2670), @liutiming (#2783), @agile-operators (#1437), @benjaminberes-bp (#2369), @lewispeel (#2885), @Yugz29 (#2899), @fcmerle (#2962), @cdecook23 (#2784), @mb1810 (#2913), @ntdatt812 (#2949), @pneumorea-ai (#2915), @gabrielrondon (#2811), @kevingrasso32 (#2876), @xwang4-svg (#2354), @DizzyWesterwald (#2936), @oli548 (#2771), @shofel (#1968), @ryanlaiwy (#1633), @yinanli1917-cloud (#2281), @Spooks444 (#2260), @punksterlabs (#2439), @himiaocc (#2492), @jbetala7 (#1419, #1777), @mamedov (#2891), @Lockyer228 (#2651), @topcoder1 (#2263), @jonahberg (#2707), @AlinValentin7 (#3015), @philhie (#2208), @Alchemist-DevAI (#2334), @merit-blake (#2343), @day-of-davon (#2952), @loulanyue (#2797, #2792, #2794, #2796), @Bit0ps (#2840), @szsunyuan (#2799), @frosimanuel (#2818), @JiayuuWang (#1096), @RyanAlberts (#1728), @exGeni (#2704), @Jino00 (#2812), @snig-17 (#2809) and @SholtoMc (#2037). diff --git a/TODOS.md b/TODOS.md index ebc56e408..89611aa5b 100644 --- a/TODOS.md +++ b/TODOS.md @@ -75,34 +75,19 @@ Deferred from the approved severe fix wave (docs/designs/SEVERE_FIX_WAVE_2026_10 (`GIT_OBJECT_DIRECTORY` + `GIT_ALTERNATE_OBJECT_DIRECTORIES` under `$TMPDIR`) for the fingerprint, and report the fetch as stale-base coverage. **Effort:** S. **Priority:** P2. -- **P1: Codex paid evals cannot run commands in CI** — Docker's default - seccomp blocks bubblewrap's user namespace; on Ubicloud hosts - (`kernel.apparmor_restrict_unprivileged_userns=1`) `seccomp=unconfined` gives - "bwrap: Failed to make / slave", and adding `apparmor=unconfined` gives - "loopback: Failed RTM_NEWADDR". A stock Ubicloud VM ran `codex sandbox` only - with host `sysctl kernel.apparmor_restrict_unprivileged_userns=0` plus both - `--security-opt` options. Container jobs cannot set the host sysctl, so the - Codex cases need a host-run job or different isolation (in progress as the - wave/evals2 lane). Also audit every Codex case's validator for execution - evidence (shared-libs-codex-read-only and outside-voice-codex-to-claude-code - passed in 37158847998 with commands possibly failing). Runs 37151477069, - 37158847998. **Effort:** M. **Priority:** P1. -- **plan-eng-review-artifact skips the QA test plan** — red in 37151477069 and - 37158847998: the model writes review-output.md without any Bash call, so the - artifact's gstack-paths/gstack-slug block never runs. Removing the "write - only when resolved" contradiction did not fix it. Next: check whether the - fixture prompt ("skip the preamble bash block", one named output) suppresses - the artifact; one paid rerun per change. **Priority:** P2. -- **office-hours-brain-writeback turn budget** — 14/18 trials across six runs; - every failure is `error_max_turns` at 12 turns with `gbrain search` but no - `gbrain put`, after chunked re-reads of the 1260-line SKILL.md that Bash output - truncation forces. The budget must not rise: add reading guidance or decide - whether a smaller rendered fixture still tests the point. **Priority:** P2. -- **ship-exploratory-late-input (gate) returns `inconclusive` in 2 of 8 - censuses** (36920606897, 37158847998). It can redden the PR lane; - characterize it before relying on the lane. **Priority:** P2. -- **plan-eng-multi-finding-batching** — red in 6 of 12 runs read (batching: - reviewCount 0-1, or Step 0 exhausting the 1495 s budget). **Priority:** P2. +- **plan-eng-review-artifact skips the QA test plan (0/4 since #3017)** — + red in 37151477069, 37158847998, 37162480720 and one local CI-image run. + Every attempt reads SKILL.md, sections, plan and sources, writes the whole + review into review-output.md in one Write, runs no Bash, and never runs the + Test Plan Artifact block. A prompt destination rewording and a self-check + rewording each got one paid rerun and did not move it. Options: put the + artifact instruction back in the prompt, move the assertion into a PTY + full-workflow case, or quarantine with this evidence. **Priority:** P2. +- **Closed by the wave's evals2 lane (run 37162480720):** Codex paid evals now + run commands in CI (host-run job; all 8 Codex cases executed real commands), + ship-exploratory-late-input no longer ends `inconclusive` (capture names the + probes to rerun after an input change), plan-eng-multi-finding-batching's + checker and office-hours-brain-writeback's prompt are fixed (3/3). - **Gate-outcome telemetry** — count `not_run`/`unavailable` outcomes per reason code from `lib/gate-outcomes.ts` so the most common gaps get fixed first. **Effort:** S. **Priority:** P3. diff --git a/test/helpers/touchfiles-data.ts b/test/helpers/touchfiles-data.ts index d4f7e287f..7bcfe5418 100644 --- a/test/helpers/touchfiles-data.ts +++ b/test/helpers/touchfiles-data.ts @@ -624,7 +624,7 @@ export const E2E_TOUCHFILES: Record = { 'ship-base-branch': [ 'bin/gstack-version-bump','bin/gstack-state-root.sh', 'lib/state-root.ts', 'ship/**', 'bin/gstack-repo-mode', 'test/skill-e2e-review-attribution.test.ts', 'scripts/resolvers/testing.ts' ], - 'ship-local-workflow': [ 'bin/gstack-version-bump', 'ship/**', 'scripts/gen-skill-docs.ts', 'test/skill-e2e-workflow.test.ts', + 'ship-local-workflow': [ 'test/helpers/outside-voice-evidence.ts', 'bin/gstack-version-bump', 'ship/**', 'scripts/gen-skill-docs.ts', 'test/skill-e2e-workflow.test.ts', 'scripts/resolvers/testing.ts', 'test/fixtures/coverage-audit-fixture.ts', 'test/helpers/coverage-audit.ts', 'test/helpers/coverage-audit-evidence.ts', 'test/helpers/office-hours-attempt.ts'], 'review-dashboard-via': [ 'ship/**', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'codex/**', 'autoplan/**', 'land-and-deploy/**', 'test/skill-e2e-review-attribution.test.ts', 'scripts/resolvers/testing.ts' @@ -663,10 +663,10 @@ export const E2E_TOUCHFILES: Record = { 'context-save-list-all-branches': [ 'context-save/**', 'test/skill-e2e-context-skills.test.ts', 'bin/gstack-skill-start', 'test/helpers/shipped-skill-routing.ts'], // Document-release - 'document-release': [ 'bin/gstack-version-bump', 'document-release/**', 'test/skill-e2e-workflow.test.ts', 'test/fixtures/coverage-audit-fixture.ts', 'test/helpers/coverage-audit.ts', 'test/helpers/coverage-audit-evidence.ts', 'test/helpers/office-hours-attempt.ts'], + 'document-release': [ 'test/helpers/outside-voice-evidence.ts', 'bin/gstack-version-bump', 'document-release/**', 'test/skill-e2e-workflow.test.ts', 'test/fixtures/coverage-audit-fixture.ts', 'test/helpers/coverage-audit.ts', 'test/helpers/coverage-audit-evidence.ts', 'test/helpers/office-hours-attempt.ts'], // Codex (Claude E2E — tests /codex skill via Claude) - 'codex-review': [ 'lib/outside-review-result.ts', 'lib/gate-outcomes.ts', 'codex/**', 'test/skill-e2e-workflow.test.ts', 'test/fixtures/coverage-audit-fixture.ts', 'test/helpers/coverage-audit.ts', 'test/helpers/coverage-audit-evidence.ts', 'test/helpers/office-hours-attempt.ts', 'bin/gstack-codex-probe', 'scripts/resolve-codex-generation-model.ts', 'scripts/resolvers/constants.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/outside-voice-steps.ts'], + 'codex-review': [ 'test/helpers/outside-voice-evidence.ts', 'lib/outside-review-result.ts', 'lib/gate-outcomes.ts', 'codex/**', 'test/skill-e2e-workflow.test.ts', 'test/fixtures/coverage-audit-fixture.ts', 'test/helpers/coverage-audit.ts', 'test/helpers/coverage-audit-evidence.ts', 'test/helpers/office-hours-attempt.ts', 'bin/gstack-codex-probe', 'scripts/resolve-codex-generation-model.ts', 'scripts/resolvers/constants.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/outside-voice-steps.ts'], // Codex E2E (tests skills via Codex CLI + worktree) 'codex-discover-skill': [ 'lib/outside-review-result.ts', 'lib/gate-outcomes.ts', 'setup','codex/**', 'scripts/gen-skill-docs.ts', 'test/helpers/codex-session-runner.ts', 'lib/worktree.ts', 'test/codex-e2e.test.ts', @@ -718,7 +718,7 @@ export const E2E_TOUCHFILES: Record = { // Coverage audit (shared fixture) + triage + gates - 'ship-coverage-audit': ['bin/gstack-state-root.sh', 'lib/state-root.ts', "test/fixtures/coverage-audit-aw.json", + 'ship-coverage-audit': [ 'test/helpers/outside-voice-evidence.ts','bin/gstack-state-root.sh', 'lib/state-root.ts', "test/fixtures/coverage-audit-aw.json", "test/fixtures/coverage-checkbox-tail-av.json", @@ -818,7 +818,7 @@ export const E2E_TOUCHFILES: Record = { 'diagram-authoring-quality': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'diagram/**', 'lib/diagram-render/**', 'lib/aside-render.ts', 'bin/gstack-render.ts', 'test/helpers/aside-available.ts', 'browse/src/**', 'test/helpers/llm-judge.ts', 'test/skill-e2e-diagram.test.ts'], // gstack-upgrade - 'gstack-upgrade-happy-path': [ 'setup', 'bin/gstack-relink', 'bin/gstack-session-update', 'gstack-upgrade/**', 'test/skill-e2e-workflow.test.ts', 'test/fixtures/coverage-audit-fixture.ts', 'test/helpers/coverage-audit.ts', 'test/helpers/coverage-audit-evidence.ts', 'test/helpers/office-hours-attempt.ts'], + 'gstack-upgrade-happy-path': [ 'test/helpers/outside-voice-evidence.ts', 'setup', 'bin/gstack-relink', 'bin/gstack-session-update', 'gstack-upgrade/**', 'test/skill-e2e-workflow.test.ts', 'test/fixtures/coverage-audit-fixture.ts', 'test/helpers/coverage-audit.ts', 'test/helpers/coverage-audit-evidence.ts', 'test/helpers/office-hours-attempt.ts'], // Deploy skills 'land-and-deploy-workflow': [ 'bin/gstack-version-bump', 'land-and-deploy/**', 'scripts/gen-skill-docs.ts', 'test/skill-e2e-deploy.test.ts'],