diff --git a/CHANGELOG.md b/CHANGELOG.md index 803f0c8bd..7c7504bcd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -98,7 +98,7 @@ The first two rows are the ones to care about: a review that read nothing, or fo - Eval children no longer receive credential env vars that carry a suffix (`GITHUB_TOKEN_1`, `GITHUB_APP_PRIVATE_KEY_BASE64`); admit one explicitly with `extraAllow` (#2949). Chunked large-push coverage for the pre-push guard (#2915). - The test typecheck ratchet ignores the order tsc prints an object type's members in. - The weekly eval report counts only cases that ran: tests deselected for another tier or case shard are listed by reason with no credit, every result is attributed to its case (0 unattributed, was 72), and no PR-lane eval shard runs over 10 minutes (the deploy eval is split per case). -- Codex evals run on their own CI job where Codex's sandbox can start, and pass only when Codex completed a command; a review whose sandbox failed now fails instead of passing. The `/codex review` eval passes only when Codex actually reviewed the diff. The AUQ matrix checks the `(recommended)` marker on option labels, where the auto-decide hook reads it. +- Codex evals run on their own CI job where Codex's sandbox can start, and pass only when Codex completed a command; a review whose sandbox failed now fails instead of passing. The `/codex review` eval passes only when Codex actually reviewed the diff. The AUQ matrix checks the `(recommended)` marker on option labels, where the auto-decide hook reads it. The `/plan-eng-review` QA test-plan eval now drives the real interactive review instead of a shortcut prompt that skipped the workflow. Contributed by @CermakM (#2843), @Haijie-Lee (#2192), @SomSamantray (#2965), @Vaughan-g-aus (#2329), @BharadwajMittapelli (#2729), @maxpetrusenkoagent (#1931, #1932, #1981), @Infiniteyieldai (#2884), @afshaker (#2778), @time-attack (#2232, #2229), @jzeisweiss (#2670), @liutiming (#2783), @agile-operators (#1437), @benjaminberes-bp (#2369), @lewispeel (#2885), @Yugz29 (#2899), @fcmerle (#2962), @cdecook23 (#2784), @mb1810 (#2913), @ntdatt812 (#2949), @pneumorea-ai (#2915), @gabrielrondon (#2811), @kevingrasso32 (#2876), @xwang4-svg (#2354), @DizzyWesterwald (#2936), @oli548 (#2771), @shofel (#1968), @ryanlaiwy (#1633), @yinanli1917-cloud (#2281), @Spooks444 (#2260), @punksterlabs (#2439), @himiaocc (#2492), @jbetala7 (#1419, #1777), @mamedov (#2891), @Lockyer228 (#2651), @topcoder1 (#2263), @jonahberg (#2707), @AlinValentin7 (#3015), @philhie (#2208), @Alchemist-DevAI (#2334), @merit-blake (#2343), @day-of-davon (#2952), @loulanyue (#2797, #2792, #2794, #2796), @Bit0ps (#2840), @szsunyuan (#2799), @frosimanuel (#2818), @JiayuuWang (#1096), @RyanAlberts (#1728), @exGeni (#2704), @Jino00 (#2812), @snig-17 (#2809) and @SholtoMc (#2037). diff --git a/TODOS.md b/TODOS.md index 89611aa5b..c29064492 100644 --- a/TODOS.md +++ b/TODOS.md @@ -75,19 +75,14 @@ Deferred from the approved severe fix wave (docs/designs/SEVERE_FIX_WAVE_2026_10 (`GIT_OBJECT_DIRECTORY` + `GIT_ALTERNATE_OBJECT_DIRECTORIES` under `$TMPDIR`) for the fingerprint, and report the fetch as stale-base coverage. **Effort:** S. **Priority:** P2. -- **plan-eng-review-artifact skips the QA test plan (0/4 since #3017)** — - red in 37151477069, 37158847998, 37162480720 and one local CI-image run. - Every attempt reads SKILL.md, sections, plan and sources, writes the whole - review into review-output.md in one Write, runs no Bash, and never runs the - Test Plan Artifact block. A prompt destination rewording and a self-check - rewording each got one paid rerun and did not move it. Options: put the - artifact instruction back in the prompt, move the assertion into a PTY - full-workflow case, or quarantine with this evidence. **Priority:** P2. - **Closed by the wave's evals2 lane (run 37162480720):** Codex paid evals now run commands in CI (host-run job; all 8 Codex cases executed real commands), ship-exploratory-late-input no longer ends `inconclusive` (capture names the probes to rerun after an input change), plan-eng-multi-finding-batching's checker and office-hours-brain-writeback's prompt are fixed (3/3). + plan-eng-review-artifact (0/4 since #3017) measured a shortcut prompt that + skipped the workflow; it now drives the real interactive /plan-eng-review in + a PTY and wrote its QA test plan at 498 s in one CI-image paid run. - **Gate-outcome telemetry** — count `not_run`/`unavailable` outcomes per reason code from `lib/gate-outcomes.ts` so the most common gaps get fixed first. **Effort:** S. **Priority:** P3.