TODOS/CHANGELOG: close plan-eng-review-artifact (real PTY review, 498 s pass)

This commit is contained in:
garrytan committed 2026-10-04 00:26:38 +00:00
1 parent 289672264a
commit 364f0f3492
2 files changed
+4 -9

No files matched your search

+1 -1
View File
@@ -98,7 +98,7 @@ The first two rows are the ones to care about: a review that read nothing, or fo
- Eval children no longer receive credential env vars that carry a suffix (`GITHUB_TOKEN_1`, `GITHUB_APP_PRIVATE_KEY_BASE64`); admit one explicitly with `extraAllow` (#2949). Chunked large-push coverage for the pre-push guard (#2915).
- The test typecheck ratchet ignores the order tsc prints an object type's members in.
- The weekly eval report counts only cases that ran: tests deselected for another tier or case shard are listed by reason with no credit, every result is attributed to its case (0 unattributed, was 72), and no PR-lane eval shard runs over 10 minutes (the deploy eval is split per case).
- Codex evals run on their own CI job where Codex's sandbox can start, and pass only when Codex completed a command; a review whose sandbox failed now fails instead of passing. The `/codex review` eval passes only when Codex actually reviewed the diff. The AUQ matrix checks the `(recommended)` marker on option labels, where the auto-decide hook reads it.
- Codex evals run on their own CI job where Codex's sandbox can start, and pass only when Codex completed a command; a review whose sandbox failed now fails instead of passing. The `/codex review` eval passes only when Codex actually reviewed the diff. The AUQ matrix checks the `(recommended)` marker on option labels, where the auto-decide hook reads it. The `/plan-eng-review` QA test-plan eval now drives the real interactive review instead of a shortcut prompt that skipped the workflow.
Contributed by @CermakM (#2843), @Haijie-Lee (#2192), @SomSamantray (#2965), @Vaughan-g-aus (#2329), @BharadwajMittapelli (#2729), @maxpetrusenkoagent (#1931, #1932, #1981), @Infiniteyieldai (#2884), @afshaker (#2778), @time-attack (#2232, #2229), @jzeisweiss (#2670), @liutiming (#2783), @agile-operators (#1437), @benjaminberes-bp (#2369), @lewispeel (#2885), @Yugz29 (#2899), @fcmerle (#2962), @cdecook23 (#2784), @mb1810 (#2913), @ntdatt812 (#2949), @pneumorea-ai (#2915), @gabrielrondon (#2811), @kevingrasso32 (#2876), @xwang4-svg (#2354), @DizzyWesterwald (#2936), @oli548 (#2771), @shofel (#1968), @ryanlaiwy (#1633), @yinanli1917-cloud (#2281), @Spooks444 (#2260), @punksterlabs (#2439), @himiaocc (#2492), @jbetala7 (#1419, #1777), @mamedov (#2891), @Lockyer228 (#2651), @topcoder1 (#2263), @jonahberg (#2707), @AlinValentin7 (#3015), @philhie (#2208), @Alchemist-DevAI (#2334), @merit-blake (#2343), @day-of-davon (#2952), @loulanyue (#2797, #2792, #2794, #2796), @Bit0ps (#2840), @szsunyuan (#2799), @frosimanuel (#2818), @JiayuuWang (#1096), @RyanAlberts (#1728), @exGeni (#2704), @Jino00 (#2812), @snig-17 (#2809) and @SholtoMc (#2037).
+3 -8
View File
@@ -75,19 +75,14 @@ Deferred from the approved severe fix wave (docs/designs/SEVERE_FIX_WAVE_2026_10
(`GIT_OBJECT_DIRECTORY` + `GIT_ALTERNATE_OBJECT_DIRECTORIES` under
`$TMPDIR`) for the fingerprint, and report the fetch as stale-base coverage.
**Effort:** S. **Priority:** P2.
- **plan-eng-review-artifact skips the QA test plan (0/4 since #3017)** —
red in 37151477069, 37158847998, 37162480720 and one local CI-image run.
Every attempt reads SKILL.md, sections, plan and sources, writes the whole
review into review-output.md in one Write, runs no Bash, and never runs the
Test Plan Artifact block. A prompt destination rewording and a self-check
rewording each got one paid rerun and did not move it. Options: put the
artifact instruction back in the prompt, move the assertion into a PTY
full-workflow case, or quarantine with this evidence. **Priority:** P2.
- **Closed by the wave's evals2 lane (run 37162480720):** Codex paid evals now
run commands in CI (host-run job; all 8 Codex cases executed real commands),
ship-exploratory-late-input no longer ends `inconclusive` (capture names the
probes to rerun after an input change), plan-eng-multi-finding-batching's
checker and office-hours-brain-writeback's prompt are fixed (3/3).
plan-eng-review-artifact (0/4 since #3017) measured a shortcut prompt that
skipped the workflow; it now drives the real interactive /plan-eng-review in
a PTY and wrote its QA test plan at 498 s in one CI-image paid run.
- **Gate-outcome telemetry** — count `not_run`/`unavailable` outcomes per
reason code from `lib/gate-outcomes.ts` so the most common gaps get fixed
first. **Effort:** S. **Priority:** P3.