From 84253fc5f43797d4c1c3ab1e4335bdeaf256f74b Mon Sep 17 00:00:00 2001 From: garrytan Date: Sat, 3 Oct 2026 02:08:35 +0000 Subject: [PATCH] docs: TODOS for the Opus 5.5 prompt-cleanup deferrals --- TODOS.md | 75 ++++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 70 insertions(+), 5 deletions(-) diff --git a/TODOS.md b/TODOS.md index fdd692382..92664e60b 100644 --- a/TODOS.md +++ b/TODOS.md @@ -33,6 +33,76 @@ trials per case, apply the CASE_QUARANTINE entry rule instead of chasing one run at a time. Effort S. +### P2/P3: Opus 5.5 prompt-cleanup deferrals (filed 2026-10-03) + +Each item was deferred with a reason during the `/claude-api prompt-audit` cleanup. + +- **Safety rules kept at their old volume until an eval covers them.** A rule's + wording changes only when an eval shows the model still obeys it before and + after. These keep their wording, and each needs the named eval first: + - `CODEX_BOUNDARY` (scripts/resolvers/outside-voice-steps.ts): a codex-host + /review run asserting Claude Code never reads `.claude/skills/` or `agents/`. + - /ship Step 16 "IRON LAW: NO COMPLETION CLAIMS WITHOUT FRESH VERIFICATION + EVIDENCE": a ship E2E that offers a stale-evidence completion and asserts the + step blocks. + - design-review stop conditions (risk %): an eval that reaches the stop + condition with no cap coaching, runs where `browserAvailable()` resolves, and + asserts termination plus a report of remaining findings. `design-review-fix` + self-skips without Aside and accepts `error_max_turns`. + - pair-agent "output the full instruction block": no eval exercises relaying + the block verbatim. + - codex consult-mode "embed content, don't reference path": no eval checks that + the plan content, not its path, reaches Codex. + - ios-qa "DEMO MODE OVERRIDES ALL OTHER RULES": no device-free eval covers demo + mode. + Priority P2. Effort M per eval. +- **Judge prompts still say "Respond with ONLY valid JSON"** — + `test/helpers/llm-judge.ts` (prompt-audit s11 M1-M7) and the five inline judges + in `test/skill-llm-eval.test.ts` (:223, :266, :315, :384, :823) already pass a + JSON schema or should. The change lands only with a calibration corpus: stored + judged outputs per changed judge (passing, failing, threshold-adjacent), an + old-vs-old flip rate, then new-vs-old not above it. Also move the qa + anti-refusal judge rubric (`skill-llm-eval.test.ts:291-343`) off "always … + regardless" wording in the same commit. Priority P2. Effort M. +- **Fast PR profile has no CEO E2E probe** — retiring `plan-ceo-review-benefits` + removed the only fast PR probe for /plan-ceo-review; CEO template changes now + run the CEO judge and `auq-format-gate`. Adding `plan-ceo-review-plan-mode` to + `PR_PROFILE_CASE_IDS` fails the "exactly one planned owning file" rule in + scripts/lib/paid-plan.ts because the file is planned as more than one shard. + Priority P3. Effort S. +- **Register an Opus 5.5 dedicated-tools overlay case** — the Opus 5.5 run was + measured as a one-off. Registering it as a periodic case moves the periodic + census walls, release floors and file counts pinned by + `test/paid-overlay-scheduling.test.ts`, `test/overlay-lifecycle.test.ts`, + `test/paid-retry-supervision.test.ts` and `test/eng-finding-retry-budget.test.ts`; + update those budgets in the same commit. Priority P3. Effort S. +- **Ratchet parity caps back down** — `test/helpers/carve-guards.ts` caps were + raised during the cleanup (qa 1.104, land-and-deploy 1.108, + design-consultation 1.092, ship 1.405). Re-measure on the merged tree and lower + each to its measured value. Priority P3. Effort S. +- **design-shotgun executor subagents** (prompt-audit s08 G1) — replace the N + subagents that only run a fixed script with one parallel bash batch. This + changes how variants are dispatched, not the prompt text. Priority P3. Effort S. +- **Per-record "new sessions only" transcript consent** — let users consent to + ingesting only sessions created after they answer. Needs timestamp filtering + inside both transcript parsers (Claude Code and Codex) and a mode-plus-cutoff + config write that lands as one transaction. Priority P3. Effort M. +- **Per-repo transcript scope** — `recent`/`all` cover every project on the + machine that repo policy allows. Add a way to restrict transcript ingest to + chosen repos. Priority P3. Effort M. +- **Archaeology lint** — a free check that flags issue/PR numbers, plan IDs and + incident stories in runtime skill prose (templates and resolver strings), so + they stay in commit messages and CHANGELOG. Priority P3. Effort S. +- **Claude-side model detection** — Claude Code hosts always render the generic + `claude` overlay; `./setup --model` is Codex-only. When the next Claude model + needs its own overlay, detect the model on the Claude side first, or the + overlay never reaches users. Priority P2. Trigger: the next Claude model that + needs a tuned overlay. +- **Re-run the prompt audit at each frontier-model release** — run + `/claude-api prompt-audit` over templates, resolvers, overlays, CLAUDE.md and + the tests that pin them, and review the low-confidence flags it reports. + Priority P3. Trigger: each new frontier model. + ### P2/P3: impeccable interop deferrals (filed 2026-09-08, from the CEO + eng reviews of docs/designs/IMPECCABLE_INTEROP.md) Each item was weighed during the review and deferred with a reason; none blocks @@ -116,11 +186,6 @@ reviews deliberately deferred, each with rationale: VERSION bumps without a CHANGELOG entry are a workflow the suite must tolerate (`/ship` writes both in one step, so probably not). Effort S (human ~2h / CC ~10min). Priority P3. Depends on: none. -- **Config-key reader tripwire** — `transcript_ingest_mode=off` sat unread for - months while setup-gbrain advertised it. A free test that asserts every key - in bin/gstack-config's default table is read by at least one binary (or is - explicitly listed as prose-only) makes a dead consent switch a red test. - Effort S. Priority P2. Depends on: Wave E1 landing the reader. - **"Pre-existing" failure vocabulary** — scripts/resolvers/preamble/ generate-test-failure-triage.ts classifies from `git diff --name-only` and never asks for a base-branch run. Rewrite T1 to verified/unverified with the