mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 14:06:50 +01:00
Evals lane follow-ups: close the skillify gate TODO (I2 green), file its open reds, CHANGELOG lines; refresh measured parity annotations
This commit is contained in:
garrytan committed
1 parent
a71fbf9821
commit
c0e08a457e
3 files changed
+40
-21
No files matched your search
@@ -83,6 +83,7 @@ The first two rows are the ones to care about: a review that read nothing, or fo
|
||||
- **Browser.** Browse failed behind `HTTP_PROXY` (#494). Headless browse and make-pdf ignored `GSTACK_CHROMIUM_PATH` (#2771, #1968, #1633); extensions opened a window (#2281). `prettyscreenshot --hide ... out.png` wrote elsewhere (#1419). `gstack-ios-qa-daemon --help` started the daemon (#1932). Aside in `~/.local/bin` was not found (#2965).
|
||||
- **Windows.** /freeze, /guard and /investigate denied every edit and their hooks failed under cmd.exe (#2876, #2354). Generators crashed with EEXIST (#2329). The node-server bundle inlined sharp and socks (#2260), and a minimal install needed `browse/src/server.ts` (#2439). The /cso helper failed on the current SDK and was reported as "install Visual Studio" (#3015). `gstack-config gbrain-refresh` misread status (#1981).
|
||||
- **Redaction.** Diagram and vector files flagged as phone and card numbers (#2885, #2827); function calls assigned to `session`/`token` (#2899); `<name>.local.<ext>` config filenames (#2962); four-part versions (#2784); `postgres:postgres@localhost` (#2913).
|
||||
- **Plan reviews.** /plan-eng-review writes its QA test plan even while some test choices are still pending, and its findings enter only through Scope Challenge C's saves. /qa-only's test value card no longer points at /qa's record step. /plan-ceo-review says what its closing cleanup does (old CEO plans are archived only with your approval).
|
||||
- **Setup.** Auto-update wrote "just upgraded" after a failed setup and never retried. Conductor got the AskUserQuestion preference hook (#2208, #2207, #2719); office-hours asks its questions as `Q1...` prose there (#2729). Team mode's matcher missed Copilot's `skill` tool (#2229). A failed arm64 re-sign left a broken binary.
|
||||
|
||||
#### Added
|
||||
@@ -95,6 +96,8 @@ The first two rows are the ones to care about: a review that read nothing, or fo
|
||||
- New invariants with free tests: gate outcomes in `lib/gate-outcomes.ts` with a troubleshooting anchor per reason code; a fresh-shell conformance test and a quote-aware bash lint over every generated block on every env-var host; a per-host 160,000-byte SKILL.md ceiling.
|
||||
- Eval children no longer receive credential env vars that carry a suffix (`GITHUB_TOKEN_1`, `GITHUB_APP_PRIVATE_KEY_BASE64`); admit one explicitly with `extraAllow` (#2949). Chunked large-push coverage for the pre-push guard (#2915).
|
||||
- The test typecheck ratchet ignores the order tsc prints an object type's members in.
|
||||
- The weekly eval report counts only cases that ran: tests deselected for another tier or case shard are listed by reason with no credit, every result is attributed to its case (0 unattributed, was 72), and no PR-lane eval shard runs over 10 minutes (the deploy eval is split per case).
|
||||
- Codex evals pass only when Codex completed a command; a review whose sandbox failed now fails instead of passing. The AUQ matrix checks the `(recommended)` marker on option labels, where the auto-decide hook reads it.
|
||||
|
||||
Contributed by @CermakM (#2843), @Haijie-Lee (#2192), @SomSamantray (#2965), @Vaughan-g-aus (#2329), @BharadwajMittapelli (#2729), @maxpetrusenkoagent (#1931, #1932, #1981), @Infiniteyieldai (#2884), @afshaker (#2778), @time-attack (#2232, #2229), @jzeisweiss (#2670), @liutiming (#2783), @agile-operators (#1437), @benjaminberes-bp (#2369), @lewispeel (#2885), @Yugz29 (#2899), @fcmerle (#2962), @cdecook23 (#2784), @mb1810 (#2913), @ntdatt812 (#2949), @pneumorea-ai (#2915), @gabrielrondon (#2811), @kevingrasso32 (#2876), @xwang4-svg (#2354), @DizzyWesterwald (#2936), @oli548 (#2771), @shofel (#1968), @ryanlaiwy (#1633), @yinanli1917-cloud (#2281), @Spooks444 (#2260), @punksterlabs (#2439), @himiaocc (#2492), @jbetala7 (#1419, #1777), @mamedov (#2891), @Lockyer228 (#2651), @topcoder1 (#2263), @jonahberg (#2707), @AlinValentin7 (#3015), @philhie (#2208), @Alchemist-DevAI (#2334), @merit-blake (#2343), @mvanhorn (#1737), @day-of-davon (#2952), @loulanyue (#2797, #2792, #2794, #2796), @Bit0ps (#2840), @szsunyuan (#2799), @frosimanuel (#2818), @JiayuuWang (#1096), @RyanAlberts (#1728), @exGeni (#2704), @Jino00 (#2812), @snig-17 (#2809) and @SholtoMc (#2037).
|
||||
|
||||
|
||||
@@ -75,6 +75,34 @@ Deferred from the approved severe fix wave (docs/designs/SEVERE_FIX_WAVE_2026_10
|
||||
(`GIT_OBJECT_DIRECTORY` + `GIT_ALTERNATE_OBJECT_DIRECTORIES` under
|
||||
`$TMPDIR`) for the fingerprint, and report the fetch as stale-base coverage.
|
||||
**Effort:** S. **Priority:** P2.
|
||||
- **P1: Codex paid evals cannot run commands in CI** — Docker's default
|
||||
seccomp blocks bubblewrap's user namespace; on Ubicloud hosts
|
||||
(`kernel.apparmor_restrict_unprivileged_userns=1`) `seccomp=unconfined` gives
|
||||
"bwrap: Failed to make / slave", and adding `apparmor=unconfined` gives
|
||||
"loopback: Failed RTM_NEWADDR". A stock Ubicloud VM ran `codex sandbox` only
|
||||
with host `sysctl kernel.apparmor_restrict_unprivileged_userns=0` plus both
|
||||
`--security-opt` options. Container jobs cannot set the host sysctl, so the
|
||||
Codex cases need a host-run job or different isolation (in progress as the
|
||||
wave/evals2 lane). Also audit every Codex case's validator for execution
|
||||
evidence (shared-libs-codex-read-only and outside-voice-codex-to-claude-code
|
||||
passed in 37158847998 with commands possibly failing). Runs 37151477069,
|
||||
37158847998. **Effort:** M. **Priority:** P1.
|
||||
- **plan-eng-review-artifact skips the QA test plan** — red in 37151477069 and
|
||||
37158847998: the model writes review-output.md without any Bash call, so the
|
||||
artifact's gstack-paths/gstack-slug block never runs. Removing the "write
|
||||
only when resolved" contradiction did not fix it. Next: check whether the
|
||||
fixture prompt ("skip the preamble bash block", one named output) suppresses
|
||||
the artifact; one paid rerun per change. **Priority:** P2.
|
||||
- **office-hours-brain-writeback turn budget** — 14/18 trials across six runs;
|
||||
every failure is `error_max_turns` at 12 turns with `gbrain search` but no
|
||||
`gbrain put`, after chunked re-reads of the 1260-line SKILL.md that Bash output
|
||||
truncation forces. The budget must not rise: add reading guidance or decide
|
||||
whether a smaller rendered fixture still tests the point. **Priority:** P2.
|
||||
- **ship-exploratory-late-input (gate) returns `inconclusive` in 2 of 8
|
||||
censuses** (36920606897, 37158847998). It can redden the PR lane;
|
||||
characterize it before relying on the lane. **Priority:** P2.
|
||||
- **plan-eng-multi-finding-batching** — red in 6 of 12 runs read (batching:
|
||||
reviewCount 0-1, or Step 0 exhausting the 1495 s budget). **Priority:** P2.
|
||||
- **Gate-outcome telemetry** — count `not_run`/`unavailable` outcomes per
|
||||
reason code from `lib/gate-outcomes.ts` so the most common gaps get fixed
|
||||
first. **Effort:** S. **Priority:** P3.
|
||||
@@ -92,8 +120,6 @@ Deferred from the approved severe fix wave (docs/designs/SEVERE_FIX_WAVE_2026_10
|
||||
`<option>` 500 ms probe, 0600 modes on ingest state and staging files,
|
||||
gbrain-sync's held-checkpoint refusal, and the question-preference hook's
|
||||
fail-closed allowlist. **Priority:** P3 each.
|
||||
- **Close the skillify gate TODO once I2 is green** (see "skillify gate test
|
||||
red" below). **Priority:** P3.
|
||||
|
||||
### P1: paid-eval follow-ups from the v1.91.12.0 proof censuses (filed 2026-09-29)
|
||||
|
||||
@@ -1192,22 +1218,12 @@ behind the secret. **Risk:** a long-lived credential in CI.
|
||||
|
||||
**Effort:** S. **Priority:** P3.
|
||||
|
||||
### P1: skillify gate test red — HOME-override sessions never discover project skills (pre-existing)
|
||||
### ✅ DONE (2026-10-03, v1.91.19.0 severe fix wave, I2): skillify gate test red
|
||||
|
||||
**What:** `test/skill-e2e-skillify.test.ts` `skillify-provenance-refusal` fails
|
||||
on BOTH this branch and origin/main @ b5a951e6 (proven 2026-08-29: identical
|
||||
2-turn `Unknown skill: skillify` transcripts). Every test in that file passing
|
||||
`env: { HOME: workDir }` gets ZERO seeded project skills in the session init
|
||||
(claude CLI 2.1.237); the passing siblings recover by Reading the SKILL.md
|
||||
directly, the refusal test's agent stops at the Skill error. Fix the harness
|
||||
(seed skills wherever HOME-overridden discovery looks, or drop the HOME
|
||||
override and pass the write target another way), or report upstream if
|
||||
project-scope `.claude/skills` discovery genuinely keys off HOME.
|
||||
|
||||
**Why:** A gate-tier safety test that is red for environmental reasons trains
|
||||
people to ignore gate reds.
|
||||
|
||||
**Effort:** S-M (harness). **Priority:** P1 (gate hygiene).
|
||||
The gate `skill-e2e-skillify` cases (provenance-refusal, approval-reject,
|
||||
happy-path) passed in all 6 gate censuses 09-30..10-03 (latest
|
||||
https://github.com/garrytan/gstack/actions/runs/37151477069), and the periodic
|
||||
scrape-match-path and scrape-prototype-path cases passed 18/18 trials.
|
||||
|
||||
### P2: auq-verbose-vs-carved-ab PRE arm reads a branch-local ref (same fragility class the repetition-cut A/B just fixed)
|
||||
|
||||
|
||||
@@ -166,7 +166,7 @@ export const CARVE_GUARDS: Record<string, CarveGuard> = {
|
||||
// wave's headline capability) grows the union to 1.195x. Deliberate:
|
||||
// the section is on-demand (loads only for Apple store targets), so
|
||||
// per-invocation cost for non-iOS ships is one manifest line.
|
||||
maxSizeRatio: 1.427, // Shared advisory identity/dedup + critical-severity validation: 248,065 union bytes / 187,706 baseline = 1.3216 (2026-09-17). + test value bar in the lazy Step 7 section (value cards, weak paths, gate table, base control, machine checks; ~13.6KB): measured 1.396 (2026-09-29). + W1 guarded state-root resolution (`eval gstack-paths; : "${GSTACK_STATE_ROOT:?…}"`) in the Context Recovery preamble, the eureka log and each state-writing bash block; measured 1.401 (2026-09-30). + the shared QA review step's plan-check timing rule (plan checks and their revalidation run on --timeout-ms after smoke expiry); measured 1.4022 (2026-09-30). + the shared TEST_BOOTSTRAP owned-change undo rules (s03 H1-H3, M1) and the value-bar coverage wording (s03 M2/M3); measured 1.4044 (2026-10-02). + severe fix wave ship-review lane: G1 NO_VERSION dispatch, title and CHANGELOG arms (versionless repos ship without inventing a version), F1 parent-side plan binding before the audit dispatch (binding, docs/designs candidates, exact not-run line) and D3's data-not-instructions line; +2,126 union bytes, measured 1.4135 (2026-10-03). + INV-1 consumer lines (outside review `unverified`/`unavailable` is missing coverage in the readiness note and PR body); measured 1.4148 (2026-10-03). + v1.91.19.0 severe fix wave: verdict-form outside-review callers with sandbox preflight and first-use notice (INV-1/B1/Q2), P0/P1 gate and unverified wording, bound-plan discovery (F1), visible LEARNINGS failures (B5), carved external pointers; measured 267,747 / 187,706 = 1.4264 (2026-10-03).
|
||||
maxSizeRatio: 1.427, // Shared advisory identity/dedup + critical-severity validation: 248,065 union bytes / 187,706 baseline = 1.3216 (2026-09-17). + test value bar in the lazy Step 7 section (value cards, weak paths, gate table, base control, machine checks; ~13.6KB): measured 1.396 (2026-09-29). + W1 guarded state-root resolution (`eval gstack-paths; : "${GSTACK_STATE_ROOT:?…}"`) in the Context Recovery preamble, the eureka log and each state-writing bash block; measured 1.401 (2026-09-30). + the shared QA review step's plan-check timing rule (plan checks and their revalidation run on --timeout-ms after smoke expiry); measured 1.4022 (2026-09-30). + the shared TEST_BOOTSTRAP owned-change undo rules (s03 H1-H3, M1) and the value-bar coverage wording (s03 M2/M3); measured 1.4044 (2026-10-02). + severe fix wave ship-review lane: G1 NO_VERSION dispatch, title and CHANGELOG arms (versionless repos ship without inventing a version), F1 parent-side plan binding before the audit dispatch (binding, docs/designs candidates, exact not-run line) and D3's data-not-instructions line; +2,126 union bytes, measured 1.4135 (2026-10-03). + INV-1 consumer lines (outside review `unverified`/`unavailable` is missing coverage in the readiness note and PR body); measured 1.4148 (2026-10-03). + v1.91.19.0 severe fix wave: verdict-form outside-review callers with sandbox preflight and first-use notice (INV-1/B1/Q2), P0/P1 gate and unverified wording, bound-plan discovery (F1), visible LEARNINGS failures (B5), carved external pointers; measured 267,752 / 187,706 = 1.4264 (2026-10-03).
|
||||
},
|
||||
'plan-ceo-review': {
|
||||
skill: 'plan-ceo-review',
|
||||
@@ -348,7 +348,7 @@ do not launch the downstream skill or open a browser.`,
|
||||
// the #538 opt-out + D1 evidence directive — ratio 1.104 measured.
|
||||
// #2499 project-scope MCP jq in the brain-sync block grew every tier-2+
|
||||
// skeleton ~1.5KB (entry resolution emitted once per SKILL.md).
|
||||
maxSkeletonBytes: 88_300, // Office-hours + sketch outside voices include host guards and completion checks. + v1.91.19.0 wave: Q<N> open-question prose form (#2729, approved Q3), DESIGN_READY --version probe (B4), Aside ~/.local/bin fallback (E7), verdict-form outside voice (INV-1); measured 88,208 (2026-10-03).
|
||||
maxSkeletonBytes: 88_300, // Office-hours + sketch outside voices include host guards and completion checks. + v1.91.19.0 wave: Q<N> open-question prose form (#2729, approved Q3), DESIGN_READY --version probe (B4), Aside ~/.local/bin fallback (E7), verdict-form outside voice (INV-1); measured 88,212 (2026-10-03).
|
||||
minUnionBytes: 115_800, // Phase 4 wave 4; measured union 118,175
|
||||
mustContain: ['design doc', 'problem statement'],
|
||||
maxSizeRatio: 1.12,
|
||||
@@ -411,7 +411,7 @@ do not launch the downstream skill or open a browser.`,
|
||||
// eureka log, the office-hours lookup and the taste-profile read; measured
|
||||
// 1.0834 (2026-09-30). + named design-direction defaults to avoid in the native
|
||||
// subagent prompt (s02 H4a); measured 1.0911 (2026-10-02).
|
||||
maxSizeRatio: 1.097, // + v1.91.19.0 wave: design --version probe (B4), taste-profile load (F3), per-round variant paths (F4), verdict-form outside voice (INV-1); measured 99,090 / 90,375 = 1.0964 (2026-10-03).
|
||||
maxSizeRatio: 1.097, // + v1.91.19.0 wave: design --version probe (B4), taste-profile load (F3), per-round variant paths (F4), verdict-form outside voice (INV-1); measured 99,092 / 90,375 = 1.0965 (2026-10-03).
|
||||
},
|
||||
cso: {
|
||||
skill: 'cso',
|
||||
@@ -485,7 +485,7 @@ do not launch the downstream skill or open a browser.`,
|
||||
maxSkeletonBytes: 74_881, // Shared-code identity/skip/action rules + critical-severity validation; measured 74,493 (2026-09-17). + v1.91.12.0 merge of #2999 (review clarity repairs: await reads, research alongside dispatch, /review deadline and setup authority, findings sources) with #3002 (guarded state-root lines, plan-check checkpoints); each fit alone; measured 74,881 (2026-10-01).
|
||||
minUnionBytes: 89_000, // Phase 4 wave 1; measured union 93,357
|
||||
mustContain: ['confidence', 'P1', 'P2', 'Review Army', 'adversarial'],
|
||||
maxSizeRatio: 1.198, // Shared-code feature + critical-severity validation: 128,042 union bytes / 108,523 baseline = 1.1799; preserves content floors. + v1.91.12.0 merge of #2999 (above, plus plan-completion fallback intent and specialist checklist-by-path) with #3002; measured 1.1843 (2026-10-01). + v1.91.19.0 wave: verdict-form Codex callers with sandbox preflight (INV-1/B1), P0/P1 structured gate (B1b), bound-plan discovery and data-not-instructions guard (F1/D3), visible LEARNINGS failures (B5); measured 129,938 / 108,523 = 1.1973 (2026-10-03).
|
||||
maxSizeRatio: 1.198, // Shared-code feature + critical-severity validation: 128,042 union bytes / 108,523 baseline = 1.1799; preserves content floors. + v1.91.12.0 merge of #2999 (above, plus plan-completion fallback intent and specialist checklist-by-path) with #3002; measured 1.1843 (2026-10-01). + v1.91.19.0 wave: verdict-form Codex callers with sandbox preflight (INV-1/B1), P0/P1 structured gate (B1b), bound-plan discovery and data-not-instructions guard (F1/D3), visible LEARNINGS failures (B5); measured 129,941 / 108,523 = 1.1974 (2026-10-03).
|
||||
},
|
||||
codex: {
|
||||
skill: 'codex',
|
||||
|
||||
Reference in new issue
Block a user