From c0e08a457e67b416e5cc6bcfa5851a8328c98f8c Mon Sep 17 00:00:00 2001 From: garrytan Date: Sat, 3 Oct 2026 23:06:26 +0000 Subject: [PATCH] Evals lane follow-ups: close the skillify gate TODO (I2 green), file its open reds, CHANGELOG lines; refresh measured parity annotations --- CHANGELOG.md | 3 +++ TODOS.md | 50 ++++++++++++++++++++++++------------ test/helpers/carve-guards.ts | 8 +++--- 3 files changed, 40 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9ed0c88e0..a9834ab00 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -83,6 +83,7 @@ The first two rows are the ones to care about: a review that read nothing, or fo - **Browser.** Browse failed behind `HTTP_PROXY` (#494). Headless browse and make-pdf ignored `GSTACK_CHROMIUM_PATH` (#2771, #1968, #1633); extensions opened a window (#2281). `prettyscreenshot --hide ... out.png` wrote elsewhere (#1419). `gstack-ios-qa-daemon --help` started the daemon (#1932). Aside in `~/.local/bin` was not found (#2965). - **Windows.** /freeze, /guard and /investigate denied every edit and their hooks failed under cmd.exe (#2876, #2354). Generators crashed with EEXIST (#2329). The node-server bundle inlined sharp and socks (#2260), and a minimal install needed `browse/src/server.ts` (#2439). The /cso helper failed on the current SDK and was reported as "install Visual Studio" (#3015). `gstack-config gbrain-refresh` misread status (#1981). - **Redaction.** Diagram and vector files flagged as phone and card numbers (#2885, #2827); function calls assigned to `session`/`token` (#2899); `.local.` config filenames (#2962); four-part versions (#2784); `postgres:postgres@localhost` (#2913). +- **Plan reviews.** /plan-eng-review writes its QA test plan even while some test choices are still pending, and its findings enter only through Scope Challenge C's saves. /qa-only's test value card no longer points at /qa's record step. /plan-ceo-review says what its closing cleanup does (old CEO plans are archived only with your approval). - **Setup.** Auto-update wrote "just upgraded" after a failed setup and never retried. Conductor got the AskUserQuestion preference hook (#2208, #2207, #2719); office-hours asks its questions as `Q1...` prose there (#2729). Team mode's matcher missed Copilot's `skill` tool (#2229). A failed arm64 re-sign left a broken binary. #### Added @@ -95,6 +96,8 @@ The first two rows are the ones to care about: a review that read nothing, or fo - New invariants with free tests: gate outcomes in `lib/gate-outcomes.ts` with a troubleshooting anchor per reason code; a fresh-shell conformance test and a quote-aware bash lint over every generated block on every env-var host; a per-host 160,000-byte SKILL.md ceiling. - Eval children no longer receive credential env vars that carry a suffix (`GITHUB_TOKEN_1`, `GITHUB_APP_PRIVATE_KEY_BASE64`); admit one explicitly with `extraAllow` (#2949). Chunked large-push coverage for the pre-push guard (#2915). - The test typecheck ratchet ignores the order tsc prints an object type's members in. +- The weekly eval report counts only cases that ran: tests deselected for another tier or case shard are listed by reason with no credit, every result is attributed to its case (0 unattributed, was 72), and no PR-lane eval shard runs over 10 minutes (the deploy eval is split per case). +- Codex evals pass only when Codex completed a command; a review whose sandbox failed now fails instead of passing. The AUQ matrix checks the `(recommended)` marker on option labels, where the auto-decide hook reads it. Contributed by @CermakM (#2843), @Haijie-Lee (#2192), @SomSamantray (#2965), @Vaughan-g-aus (#2329), @BharadwajMittapelli (#2729), @maxpetrusenkoagent (#1931, #1932, #1981), @Infiniteyieldai (#2884), @afshaker (#2778), @time-attack (#2232, #2229), @jzeisweiss (#2670), @liutiming (#2783), @agile-operators (#1437), @benjaminberes-bp (#2369), @lewispeel (#2885), @Yugz29 (#2899), @fcmerle (#2962), @cdecook23 (#2784), @mb1810 (#2913), @ntdatt812 (#2949), @pneumorea-ai (#2915), @gabrielrondon (#2811), @kevingrasso32 (#2876), @xwang4-svg (#2354), @DizzyWesterwald (#2936), @oli548 (#2771), @shofel (#1968), @ryanlaiwy (#1633), @yinanli1917-cloud (#2281), @Spooks444 (#2260), @punksterlabs (#2439), @himiaocc (#2492), @jbetala7 (#1419, #1777), @mamedov (#2891), @Lockyer228 (#2651), @topcoder1 (#2263), @jonahberg (#2707), @AlinValentin7 (#3015), @philhie (#2208), @Alchemist-DevAI (#2334), @merit-blake (#2343), @mvanhorn (#1737), @day-of-davon (#2952), @loulanyue (#2797, #2792, #2794, #2796), @Bit0ps (#2840), @szsunyuan (#2799), @frosimanuel (#2818), @JiayuuWang (#1096), @RyanAlberts (#1728), @exGeni (#2704), @Jino00 (#2812), @snig-17 (#2809) and @SholtoMc (#2037). diff --git a/TODOS.md b/TODOS.md index ac49df273..75243a346 100644 --- a/TODOS.md +++ b/TODOS.md @@ -75,6 +75,34 @@ Deferred from the approved severe fix wave (docs/designs/SEVERE_FIX_WAVE_2026_10 (`GIT_OBJECT_DIRECTORY` + `GIT_ALTERNATE_OBJECT_DIRECTORIES` under `$TMPDIR`) for the fingerprint, and report the fetch as stale-base coverage. **Effort:** S. **Priority:** P2. +- **P1: Codex paid evals cannot run commands in CI** — Docker's default + seccomp blocks bubblewrap's user namespace; on Ubicloud hosts + (`kernel.apparmor_restrict_unprivileged_userns=1`) `seccomp=unconfined` gives + "bwrap: Failed to make / slave", and adding `apparmor=unconfined` gives + "loopback: Failed RTM_NEWADDR". A stock Ubicloud VM ran `codex sandbox` only + with host `sysctl kernel.apparmor_restrict_unprivileged_userns=0` plus both + `--security-opt` options. Container jobs cannot set the host sysctl, so the + Codex cases need a host-run job or different isolation (in progress as the + wave/evals2 lane). Also audit every Codex case's validator for execution + evidence (shared-libs-codex-read-only and outside-voice-codex-to-claude-code + passed in 37158847998 with commands possibly failing). Runs 37151477069, + 37158847998. **Effort:** M. **Priority:** P1. +- **plan-eng-review-artifact skips the QA test plan** — red in 37151477069 and + 37158847998: the model writes review-output.md without any Bash call, so the + artifact's gstack-paths/gstack-slug block never runs. Removing the "write + only when resolved" contradiction did not fix it. Next: check whether the + fixture prompt ("skip the preamble bash block", one named output) suppresses + the artifact; one paid rerun per change. **Priority:** P2. +- **office-hours-brain-writeback turn budget** — 14/18 trials across six runs; + every failure is `error_max_turns` at 12 turns with `gbrain search` but no + `gbrain put`, after chunked re-reads of the 1260-line SKILL.md that Bash output + truncation forces. The budget must not rise: add reading guidance or decide + whether a smaller rendered fixture still tests the point. **Priority:** P2. +- **ship-exploratory-late-input (gate) returns `inconclusive` in 2 of 8 + censuses** (36920606897, 37158847998). It can redden the PR lane; + characterize it before relying on the lane. **Priority:** P2. +- **plan-eng-multi-finding-batching** — red in 6 of 12 runs read (batching: + reviewCount 0-1, or Step 0 exhausting the 1495 s budget). **Priority:** P2. - **Gate-outcome telemetry** — count `not_run`/`unavailable` outcomes per reason code from `lib/gate-outcomes.ts` so the most common gaps get fixed first. **Effort:** S. **Priority:** P3. @@ -92,8 +120,6 @@ Deferred from the approved severe fix wave (docs/designs/SEVERE_FIX_WAVE_2026_10 `