From ee664c1470a4aa186c3f82f9e178dc76e437a445 Mon Sep 17 00:00:00 2001 From: garrytan Date: Mon, 2 Nov 2026 06:32:31 +0000 Subject: [PATCH] fix(investigate): keep the web-search sanitize guard; update BROWSER.md and docs/skills.md (review pass 1) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- BROWSER.md | 2 +- changelog.d/gsta251-investigate-drop-steps.md | 2 +- docs/skills.md | 2 +- investigate/SKILL.md | 3 ++- investigate/SKILL.md.tmpl | 3 ++- 5 files changed, 7 insertions(+), 5 deletions(-) diff --git a/BROWSER.md b/BROWSER.md index 13aa70909..5d8e49ece 100644 --- a/BROWSER.md +++ b/BROWSER.md @@ -99,7 +99,7 @@ through CDP before `goto`, as the cookbook does), and no `file://` navigation. | `/scrape` | Prototype the extraction with `aside repl`, hand back the table, list, or prices as structured data. Read-only. | | `/devex-review` | Walk the real onboarding flow and time it, carrying the cookbook inline. | | `/ship`, `/spec`, `/land-and-deploy`, `/setup-deploy`, `/office-hours` | Third-party web actions (vendor dashboards, API keys, webhooks) offered as an Aside drive across your real sessions, with the one-question consent gate for anything mutating. | -| `/plan-ceo-review`, `/plan-eng-review`, `/plan-devex-review`, `/design-consultation`, `/review`, `/investigate`, `/cso`, `/office-hours` | Web research runs through `aside exec` in your real browser (`{{ASIDE_RESEARCH}}`), one read-only request per question, answers treated as untrusted content. No Aside → the same queries go to the host's WebSearch tool; no WebSearch either → "Search unavailable" once and the skill proceeds on in-distribution knowledge. | +| `/plan-ceo-review`, `/plan-eng-review`, `/plan-devex-review`, `/design-consultation`, `/review`, `/cso`, `/office-hours` | Web research runs through `aside exec` in your real browser (`{{ASIDE_RESEARCH}}`), one read-only request per question, answers treated as untrusted content. No Aside → the same queries go to the host's WebSearch tool; no WebSearch either → "Search unavailable" once and the skill proceeds on in-distribution knowledge. | ### Local-HTML rendering diff --git a/changelog.d/gsta251-investigate-drop-steps.md b/changelog.d/gsta251-investigate-drop-steps.md index fd8393d91..b54df962f 100644 --- a/changelog.d/gsta251-investigate-drop-steps.md +++ b/changelog.d/gsta251-investigate-drop-steps.md @@ -1,2 +1,2 @@ -- **`/investigate` drops its five-phase step procedure** (GST-938). Harness Evals watched Opus 5.5 run `/investigate` on 46 bug-fix sessions: the procedure loaded every time and the model followed none of its steps (0 of 30 with Claude Code's real system prompt), while a gstack session cost $0.46 against $0.22 without gstack (HARN-14, gn-hard-follow-2026-11). The skill keeps the parts that still do a job. Those are the Iron Law, the arguments table, a short "How to work" with the three lessons that cost real time before (read call sites before calling two paths duplicates, refresh the git index before timing, test a shared premise before a third fix), the 3-strike and more-than-5-files asks, the Scope Lock with its freeze hooks, the recipes and `--backlog`, the prior-learnings search and the DEBUG REPORT. The generated Claude skill shrinks from 29.8 KB to 20.1 KB. Gone: the Phase 1-5 walkthrough, the pattern table, the web-research block and the hypothesis-keyed learnings re-pull. +- **`/investigate` drops its five-phase step procedure** (GST-938). Harness Evals watched Opus 5.5 run `/investigate` on 46 bug-fix sessions: the procedure loaded every time and the model followed none of its steps (0 of 30 with Claude Code's real system prompt), while a gstack session cost $0.46 against $0.22 without gstack (HARN-14, gn-hard-follow-2026-11). The skill keeps the parts that still do a job. Those are the Iron Law, the arguments table, a short "How to work" with the three lessons that cost real time before (read call sites before calling two paths duplicates, refresh the git index before timing, test a shared premise before a third fix), the 3-strike and more-than-5-files asks, the Scope Lock with its freeze hooks, the recipes and `--backlog`, the prior-learnings search and the DEBUG REPORT. The generated Claude skill shrinks from 29.8 KB to 20.1 KB. Gone: the Phase 1-5 walkthrough, the pattern table, the web-research block and the hypothesis-keyed learnings re-pull. The research block's rule stays as a guard: a web search names the error class and library, never raw error text, hostnames, paths or secrets. BROWSER.md and docs/skills.md now describe the short skill. - Behavior change: the model is no longer told to work through Phases 1-5 in order, and `/investigate` no longer carries the Aside web-research block. Hook-enforced versions of the two steps that could change results (reproduce first, verify after) are planned as an opt-in Claude Code candidate that ships only if a preregistered eval shows it helps. diff --git a/docs/skills.md b/docs/skills.md index c121fc89d..cb34377d1 100644 --- a/docs/skills.md +++ b/docs/skills.md @@ -656,7 +656,7 @@ I want the model imagining the production incident before it happens. When something is broken and you don't know why, `/investigate` is your systematic debugger. It follows the Iron Law: **no fixes without root cause investigation first.** -Instead of guessing and patching, it traces data flow, matches against known bug patterns, and tests hypotheses one at a time. If three fix attempts fail, it stops and questions the architecture instead of thrashing. This prevents the "let me try one more thing" spiral that wastes hours. +It leaves the order and tools to the model, keeps a few guards (stop and ask after three failed hypotheses, ask before a fix touches more than five files, lock edits to the affected module), reads a recipe for flakes, order-dependent failures, races and eval regressions, and ends with a structured debug report. It used to walk a five-phase procedure; Opus 5.5 followed none of it in 46 recorded sessions, so the procedure was dropped (GST-938). --- diff --git a/investigate/SKILL.md b/investigate/SKILL.md index d28c93c2b..fbb91557d 100644 --- a/investigate/SKILL.md +++ b/investigate/SKILL.md @@ -219,10 +219,11 @@ Find the root cause, fix it with the smallest change, prove the fix, and finish - Before timing anything that spawns git in a benchmark or bisect worktree, run `git update-index --refresh` there: a stale index makes the first `git status`/`git diff` pay a full refresh and the measurement lies. - Two fixes that shared one premise failed the same gate — write the premise down in one sentence and test it before a third fix. The evidence already points away from it. -Two guards: +Three guards: - **3-strike rule:** if 3 hypotheses fail, **STOP** and use AskUserQuestion: A) continue with a new hypothesis (describe it), B) escalate for human review, C) add logging and wait. - **Blast radius:** if the fix touches more than 5 files, use AskUserQuestion: A) proceed, the root cause spans these files, B) split, fixing the critical path now, C) rethink for a more targeted approach. +- **Web searches:** search the generic error class, library and version, never the raw error text, hostnames, IPs, file paths, SQL, customer data or secrets. --- diff --git a/investigate/SKILL.md.tmpl b/investigate/SKILL.md.tmpl index 6784a71fc..c7469537d 100644 --- a/investigate/SKILL.md.tmpl +++ b/investigate/SKILL.md.tmpl @@ -96,10 +96,11 @@ Find the root cause, fix it with the smallest change, prove the fix, and finish - Before timing anything that spawns git in a benchmark or bisect worktree, run `git update-index --refresh` there: a stale index makes the first `git status`/`git diff` pay a full refresh and the measurement lies. - Two fixes that shared one premise failed the same gate — write the premise down in one sentence and test it before a third fix. The evidence already points away from it. -Two guards: +Three guards: - **3-strike rule:** if 3 hypotheses fail, **STOP** and use AskUserQuestion: A) continue with a new hypothesis (describe it), B) escalate for human review, C) add logging and wait. - **Blast radius:** if the fix touches more than 5 files, use AskUserQuestion: A) proceed, the root cause spans these files, B) split, fixing the critical path now, C) rethink for a more targeted approach. +- **Web searches:** search the generic error class, library and version, never the raw error text, hostnames, IPs, file paths, SQL, customer data or secrets. ---