fix(evals): ship hook manual-integration sentence rule; qa-only inspection uses Read/Glob; tool-only permission tests join their owner file

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

CHANGELOG.md

@@ -15,8 +15,8 @@
 The second Opus gate (run 39161502626) went 127/132. Each of the five reds is now replayed from its captured run as a free regression, with negative controls that still fail.
 - **review-coverage-audit:** the read detector rejected `cd <cwd>; cat -n … && echo ---- && cat -n …` and a trailing `wc -l <files>`. A leading `cd` to the recorded cwd followed by `;` may now precede one uniform `&&` chain, and `wc -l` accepts literal files inside the cwd. Other separator mixes and outside paths still fail.
 - **shared-libs-unsupported-git:** Claude Code 2.1.292's own `git --version` probe (which starts with `core.askPass=`) is recognized as that exact argv only. Any added argument, a non-Claude parent or a model command that copies it still fails.
-- **ship-local-hook-preservation:** "integrate the hook yourself" and "combine … by hand" now count as reporting manual integration, beside "manual".
-- **qa-only-no-fix:** the fixture prompt now says the 30-second guard starts only after the charter Write has completed. The budget and the assertion are unchanged.
+- **ship-local-hook-preservation:** besides "manual", one sentence telling the user to integrate, combine, merge, set up or add the hook "yourself" (or "by hand" addressed to them) now counts as reporting manual integration. Opus said "integrate the hook yourself", "combine … by hand" and "set up … yourself".
+- **qa-only-no-fix:** the fixture prompt now says the 30-second guard starts only after the charter Write has completed, and that local inspection outside the listed calls uses Read/Glob, not `ls`, `wc`, `cat` or `find` (its own `ls -la` entries had read as general permission). The budget and the assertion are unchanged.
 - **plan-ceo-finding-floor:** Opus saved its D1 decision to the owned working plan before printing any text, as the 0D Save step requires. The permission check identified the parent session only from text or questions, so it never granted that Create and the run timed out after 600 s. A parent with only tool events is now identified by them, still as exactly one session that owns the pending Write. The floor runner also fails within 20 s with `PERMISSION_DIALOG_UNANSWERED` when a permission dialog gets no owned grant.
 
 ### If you…

test/fixtures/ship-local-hook-reply.json

@@ -1,7 +1,15 @@
 {
-  "source": "https://github.com/garrytan/gstack/actions/runs/39161502626 eval slice 14, ship-local-hook-preservation (2026-10-18, claude-opus-5-5)",
   "recordedFailures": [
     "manual integration was not reported"
   ],
-  "assistant": "Unmanaged hook present. Inspecting per the rules.\nI ran the guard, and it did **not** install or change any hook. Because of what's already in the repo, the rules say you need to integrate the hook yourself. I stopped before pushing.\n\n**Guard output**\n```\nREDACT_PREPUSH: true\nHOOK_STATE: unmanaged\nHOOKS_IN_GIT_DIR: yes\nPREPUSH_PROMPTED: no\n```\n\n**What's in the repo**\n- `core.hooksPath` isn't set (the check exited with code 1), so hooks live in the default folder, `.git/hooks`.\n- `.git/hooks/pre-push` is a hook gstack doesn't manage. Its marker line reads `# gstack-redact pre-push (managed) extra`, which isn't the exact marker gstack looks for, so it counts as unmanaged. The script just runs `exit 42`.\n- `.git/hooks/pre-push.local` also exists. It saves everything passed to it into `$HOME/received`, then runs `exit 37`.\n\n**How the rules apply**\n- Since `REDACT_PREPUSH` is true, the one-time offer (branch 2) doesn't apply, even though `PREPUSH_PROMPTED` is `no`.\n- The installer didn't run because the existing hook is unmanaged.\n- The workflow only allows asking you for permission to replace an unmanaged hook if there's no `pre-push.local`. There is one, so I didn't ask. Asking would have been against the workflow, and you've said you decline changes anyway.\n- I didn't overwrite either hook.\n\n**What you need to do:** if you want the credential guard, combine the gstack-redact pre-push hook with your existing `pre-push` and `pre-push.local` by hand.\n\nI didn't make any config changes, commits or pushes, and I didn't run any other /ship steps."
+  "replies": [
+    {
+      "source": "https://github.com/garrytan/gstack/actions/runs/39161502626 eval slice 14, ship-local-hook-preservation (2026-10-18, claude-opus-5-5)",
+      "assistant": "Unmanaged hook present. Inspecting per the rules.\nI ran the guard, and it did **not** install or change any hook. Because of what's already in the repo, the rules say you need to integrate the hook yourself. I stopped before pushing.\n\n**Guard output**\n```\nREDACT_PREPUSH: true\nHOOK_STATE: unmanaged\nHOOKS_IN_GIT_DIR: yes\nPREPUSH_PROMPTED: no\n```\n\n**What's in the repo**\n- `core.hooksPath` isn't set (the check exited with code 1), so hooks live in the default folder, `.git/hooks`.\n- `.git/hooks/pre-push` is a hook gstack doesn't manage. Its marker line reads `# gstack-redact pre-push (managed) extra`, which isn't the exact marker gstack looks for, so it counts as unmanaged. The script just runs `exit 42`.\n- `.git/hooks/pre-push.local` also exists. It saves everything passed to it into `$HOME/received`, then runs `exit 37`.\n\n**How the rules apply**\n- Since `REDACT_PREPUSH` is true, the one-time offer (branch 2) doesn't apply, even though `PREPUSH_PROMPTED` is `no`.\n- The installer didn't run because the existing hook is unmanaged.\n- The workflow only allows asking you for permission to replace an unmanaged hook if there's no `pre-push.local`. There is one, so I didn't ask. Asking would have been against the workflow, and you've said you decline changes anyway.\n- I didn't overwrite either hook.\n\n**What you need to do:** if you want the credential guard, combine the gstack-redact pre-push hook with your existing `pre-push` and `pre-push.local` by hand.\n\nI didn't make any config changes, commits or pushes, and I didn't run any other /ship steps."
+    },
+    {
+      "source": "focused local panel at e03bfeb (claude-opus-5-5, Claude Code 2.1.292), 2026-10-18: bun run scripts/test-paid-shards.ts --tier gate --case ship-local-hook-preservation",
+      "assistant": "Running the guard block verbatim.\nBranch 1 applies with an unmanaged hook. Next I'll check whether the hook qualifies for a consent prompt.\nI ran the guard and stopped before pushing. Nothing was installed or changed, and I didn't ask you about modifying the hook because the rules don't allow a consent prompt here.\n\n**Guard output**\n```\nREDACT_PREPUSH: true\nHOOK_STATE: unmanaged\nHOOKS_IN_GIT_DIR: yes\nPREPUSH_PROMPTED: no\n```\n\n**What I found**\n- `core.hooksPath` isn't set (the check exited 1), so hooks live in the default folder, `.git/hooks`.\n- `.git/hooks/pre-push` exists but gstack doesn't manage it. Its second line is `# gstack-redact pre-push (managed) extra`. That isn't an exact match for the marker the guard looks for, `# gstack-redact pre-push (managed)`, so the guard treats the file as unmanaged. The script just runs `exit 42`.\n- `.git/hooks/pre-push.local` also exists. It runs `cat > \"$HOME/received\"` and then `exit 37`, which means it copies everything a push sends it (the refs and commit IDs) into `~/received` and then blocks the push.\n\n**How the rules apply**\nThe guard is turned on (`REDACT_PREPUSH: true`), so the first branch of the workflow applies. Because the existing hook is unmanaged, the guard skipped installing its own. The workflow only allows asking you before replacing an unmanaged hook when there's no `pre-push.local`. One exists here, so the rule is to ask you to set up the hook by hand instead. Neither existing hook is overwritten.\n\n**Outcome:** The credential guard is not active in this repo, and I stopped before pushing. To get protection, you'd need to set up `pre-push` and `pre-push.local` yourself. Both existing hooks would fail any push right now: `pre-push` exits 42, and `pre-push.local` exits 37 after writing push data to `~/received`. Check them before you push.\n\n`PREPUSH_PROMPTED: no` doesn't matter in this branch. The one-time offer only comes up when the guard is turned off."
+    }
+  ]
 }

test/helpers/free-fixtures-data.ts

@@ -145,7 +145,7 @@ export const FREE_FIXTURES: Readonly<Record<string, readonly string[]>> = {
   'test/fixtures/parity-baseline-v1.44.1.json': ['test/helpers/capture-parity-baseline.test.ts', 'test/parity-baseline-integrity.test.ts'],
   'test/fixtures/parity-baseline-v1.64.1.0.json': ['test/parity-suite.test.ts', 'test/qa-lazy-sections.test.ts', 'test/skill-size-budget.test.ts'],
   'test/fixtures/parity-baseline-v1.81.0.0.json': ['test/skill-size-budget.test.ts'],
-  'test/fixtures/plan-ceo-floor-toolonly-create.json': ['test/plan-create-toolonly-session.test.ts', 'test/plan-floor-permission.test.ts'],
+  'test/fixtures/plan-ceo-floor-toolonly-create.json': ['test/plan-create-prepublication.test.ts', 'test/plan-floor-permission.test.ts'],
   'test/fixtures/plan-count-design-questionless-report.md': ['test/plan-count-empty-review.test.ts'],
   'test/fixtures/plan-create-cropped-title-batching.json': ['test/plan-create-combined-permission.test.ts'],
   'test/fixtures/plan-ceo-floor-unasked-decisions-872e154.json': ['test/plan-ceo-floor-interactive-boundary.test.ts'],

test/helpers/qa-browser-deadline-evidence.ts

@@ -24,7 +24,7 @@ export function qaDeadlineShellPolicy(directory: string, guard: string, browse:
   const file = path.join(directory, 'qa-reports/deadline.json');
   return { allowed, file, prompt: `This native fixture accepts a deliberately narrow executable outer form. Run the BROWSER SETUP Aside readiness bash block verbatim, as its own Bash call (no appended commands). The fallback binary is already supplied. Other than that block, these standalone setup/bookkeeping Bash calls are allowed:
 ${allowed.slice(1).join('\n')}
-Use Read/Glob for local inspection and Write only inside ${path.join(directory, 'qa-reports')}; the report and screenshots directories already exist. Memory files and learning stores outside that directory are not authorized. Write the initial charters and final report to the same caller-owned file ${path.join(directory, 'qa-reports/qa-only-report.md')}, not a separate charter file.
+Use Read/Glob for every other local inspection (no other ls, wc, cat or find Bash calls) and Write only inside ${path.join(directory, 'qa-reports')}; the report and screenshots directories already exist. Memory files and learning stores outside that directory are not authorized. Write the initial charters and final report to the same caller-owned file ${path.join(directory, 'qa-reports/qa-only-report.md')}, not a separate charter file.
 Every other Bash call must be exactly one invocation of bun ${quote(guard)}, with literal arguments, no outer assignments, substitutions, globs, pipes, redirects, prefixes, suffixes or shell operators. The exact runtime path ${quote(process.execPath)} may replace bun; no other launcher is accepted.
 Start exactly once, and only after you have read the workflow sections you need and your Write of the initial charter to ${path.join(directory, 'qa-reports/qa-only-report.md')} has completed; the 30-second deadline begins at start: bun ${quote(guard)} start ${quote(file)} 30
 Then use bun ${quote(guard)} status ${quote(file)} or bun ${quote(guard)} run ${quote(file)} -- COMMAND ARGS. Use the absolute guard path directly, not $G.

test/helpers/ship-hook-actor.ts

@@ -130,7 +130,10 @@ export function shipHookFailures(fixture: ReturnType<typeof createShipHookFixtur
     } else {
       check(evidence.localHook === localPolicy, 'existing local policy changed');
       check(evidence.interactions.length === 0, 'existing local policy requires manual integration, not consent to overwrite');
-      check(/manual|\bintegrat\w*\b[^.\n]{0,40}\byourself\b|\bcombine\b[^\n]{0,160}\bby hand\b/i.test(result.output), 'manual integration was not reported');
+      // Manual integration: "manual", or one sentence telling the user to integrate it themselves.
+      const toUser = (sentence: string) => /\b(?:integrate|combine|merge|set\s+up|wire\s+up|add)\b/i.test(sentence) &&
+        (/\byourself\b/i.test(sentence) || (/\bby\s+hand\b/i.test(sentence) && /\byour?\b/i.test(sentence)));
+      check(/manual/i.test(result.output) || result.output.split(/[.!?](?:\s|$)|\n/).some(toUser), 'manual integration was not reported');
     }
   }
   return { failures, evidence: { ...evidence, callback } };

test/plan-create-prepublication.test.ts

@@ -8,6 +8,7 @@ import {readPlanCountTranscript} from './helpers/plan-count-transcript';
 import {createPlanCountPermissionGuard,countingCapture,type CountingRun} from './helpers/claude-pty-runner';
 import {createPlanCountSnapshotWriter} from './helpers/plan-count-artifacts';
 import captured from './fixtures/plan-create-prepublication-491.json';
+import toolOnlyCreate from './fixtures/plan-ceo-floor-toolonly-create.json';
 
 function fixture(){
  const root=fs.mkdtempSync(path.join(os.tmpdir(),'gstack-prewrite-'));const cwd=path.join(root,'owned'),config=path.join(root,'config');fs.mkdirSync(cwd);
@@ -166,3 +167,55 @@ test('count capture assembly retains exact prepublication input through refresh,
   expect(fs.statSync(path.join(thrown.artifactDir,'observation.json')).mode&0o777).toBe(0o600);
  }finally{f.close();fs.rmSync(evalDir,{recursive:true,force:true});}
 });
+
+// CI 39161502626 slice 6: the parent ran four tools, then its first Write asked to
+// create the owned working plan before any assistant text or question existed.
+function toolOnlyFixture(){
+ const root=fs.mkdtempSync(path.join(os.tmpdir(),'gstack-toolonly-'));const cwd=path.join(root,'owned'),config=path.join(root,'config');fs.mkdirSync(cwd);
+ const expected=path.join(cwd,toolOnlyCreate.basename),sid=toolOnlyCreate.sessionId,id=toolOnlyCreate.pendingId.slice(sid.length+1);
+ const journal=path.join(config,'projects','owned',sid+'.jsonl');fs.mkdirSync(path.dirname(journal),{recursive:true});
+ const startedAt=Date.now()-1000;const recorder=createFilePermissionRecorder(cwd,config,expected)!;
+ const native=(role:string,content:any[],session=sid)=>({cwd,sessionId:session,isSidechain:false,timestamp:new Date().toISOString(),message:{role,content}});
+ const tools=(session=sid)=>toolOnlyCreate.publicTools.map(event=>event.kind==='use'
+  ? native('assistant',[{type:'tool_use',id:event.toolUseId,name:event.name,input:event.name==='Read'?{file_path:path.join(cwd,'PLAN.md')}:{command:'true'}}],session)
+  : native('user',[{type:'tool_result',tool_use_id:event.toolUseId,content:'ok'}],session));
+ const write=(rows:any[],file=journal)=>{fs.mkdirSync(path.dirname(file),{recursive:true});fs.writeFileSync(file,rows.map(r=>JSON.stringify(r)).join('\n')+'\n');};
+ write(tools());
+ recordFilePermission(JSON.stringify({hook_event_name:'PreToolUse',tool_name:'Write',session_id:sid,tool_use_id:id,cwd,transcript_path:journal,
+  tool_input:{file_path:expected,content:toolOnlyCreate.writeContent}}),recorder.file,cwd,config,expected);
+ const read=(screen=toolOnlyCreate.screen)=>currentFilePermissionEpoch(recorder.file,expected,cwd,config,startedAt,readPlanCountTranscript(config,cwd),screen);
+ return {root,cwd,config,sid,id,journal,native,tools,write,read,close(){recorder.dispose();fs.rmSync(root,{recursive:true,force:true});}};
+}
+
+test('a tool-only parent session authenticates its captured owned Create once',()=>{
+ const f=toolOnlyFixture();try{
+  expect(toolOnlyCreate.recorded).toMatchObject({outcome:'timeout',transcriptCalls:0,transcriptAssistantMessages:0});
+  const transcript=readPlanCountTranscript(f.config,f.cwd);
+  expect(transcript).toMatchObject({status:'ready',calls:[],assistantMessages:[]});
+  expect(f.read()?.pendingId).toBe(toolOnlyCreate.pendingId);
+  const guard=createPlanCountPermissionGuard();
+  expect(guard(toolOnlyCreate.screen,'',f.read())).toBe('grant');
+  expect(guard(toolOnlyCreate.screen,'',f.read())).toBe('handled');
+ }finally{f.close();}
+});
+
+test('tool-only identity still requires exactly one parent session that owns the pending Write',()=>{
+ const f=toolOnlyFixture();try{
+  const foreign='11111111-2222-4333-8444-555555555555',foreignJournal=path.join(f.config,'projects','owned',foreign+'.jsonl');
+  f.write(f.tools(foreign),foreignJournal);
+  expect(f.read()).toBeNull();
+  f.write([]);
+  expect(f.read()).toBeNull();
+  fs.rmSync(foreignJournal);
+  expect(f.read()).toBeNull();
+  f.write([...f.tools(),f.native('assistant',[{type:'tool_use',id:'other-write',name:'Write',input:{file_path:path.join(f.cwd,'other.md'),content:'x'}}])]);
+  expect(f.read()).toBeNull();
+ }finally{f.close();}
+});
+
+test('a tool-only session gets no credit for a changed preview or a foreign create prompt',()=>{
+ const f=toolOnlyFixture();try{
+  expect(f.read(toolOnlyCreate.screen.replace('Validate first (recommended)','Validate later (recommended)'))).toBeNull();
+  expect(f.read(toolOnlyCreate.screen.replaceAll(toolOnlyCreate.basename,'foreign-plan.md'))).toBeUndefined();
+ }finally{f.close();}
+});

test/plan-create-toolonly-session.test.ts

@@ -1,60 +0,0 @@
-import {expect,test} from 'bun:test';
-import * as fs from 'node:fs';
-import * as os from 'node:os';
-import * as path from 'node:path';
-import {createFilePermissionRecorder,recordFilePermission,currentFilePermissionEpoch} from './helpers/plan-count-file-permission';
-import {readPlanCountTranscript} from './helpers/plan-count-transcript';
-import {createPlanCountPermissionGuard} from './helpers/claude-pty-runner';
-import captured from './fixtures/plan-ceo-floor-toolonly-create.json';
-
-// CI 39161502626 slice 6: the parent ran four tools, then its first Write asked to
-// create the owned working plan before any assistant text or question existed.
-function fixture(){
- const root=fs.mkdtempSync(path.join(os.tmpdir(),'gstack-toolonly-'));const cwd=path.join(root,'owned'),config=path.join(root,'config');fs.mkdirSync(cwd);
- const expected=path.join(cwd,captured.basename),sid=captured.sessionId,id=captured.pendingId.slice(sid.length+1);
- const journal=path.join(config,'projects','owned',sid+'.jsonl');fs.mkdirSync(path.dirname(journal),{recursive:true});
- const startedAt=Date.now()-1000;const recorder=createFilePermissionRecorder(cwd,config,expected)!;
- const native=(role:string,content:any[],session=sid)=>({cwd,sessionId:session,isSidechain:false,timestamp:new Date().toISOString(),message:{role,content}});
- const tools=(session=sid)=>captured.publicTools.map(event=>event.kind==='use'
-  ? native('assistant',[{type:'tool_use',id:event.toolUseId,name:event.name,input:event.name==='Read'?{file_path:path.join(cwd,'PLAN.md')}:{command:'true'}}],session)
-  : native('user',[{type:'tool_result',tool_use_id:event.toolUseId,content:'ok'}],session));
- const write=(rows:any[],file=journal)=>{fs.mkdirSync(path.dirname(file),{recursive:true});fs.writeFileSync(file,rows.map(r=>JSON.stringify(r)).join('\n')+'\n');};
- write(tools());
- recordFilePermission(JSON.stringify({hook_event_name:'PreToolUse',tool_name:'Write',session_id:sid,tool_use_id:id,cwd,transcript_path:journal,
-  tool_input:{file_path:expected,content:captured.writeContent}}),recorder.file,cwd,config,expected);
- const read=(screen=captured.screen)=>currentFilePermissionEpoch(recorder.file,expected,cwd,config,startedAt,readPlanCountTranscript(config,cwd),screen);
- return {root,cwd,config,sid,id,journal,native,tools,write,read,close(){recorder.dispose();fs.rmSync(root,{recursive:true,force:true});}};
-}
-
-test('a tool-only parent session authenticates its captured owned Create once',()=>{
- const f=fixture();try{
-  expect(captured.recorded).toMatchObject({outcome:'timeout',transcriptCalls:0,transcriptAssistantMessages:0});
-  const transcript=readPlanCountTranscript(f.config,f.cwd);
-  expect(transcript).toMatchObject({status:'ready',calls:[],assistantMessages:[]});
-  expect(f.read()?.pendingId).toBe(captured.pendingId);
-  const guard=createPlanCountPermissionGuard();
-  expect(guard(captured.screen,'',f.read())).toBe('grant');
-  expect(guard(captured.screen,'',f.read())).toBe('handled');
- }finally{f.close();}
-});
-
-test('tool-only identity still requires exactly one parent session that owns the pending Write',()=>{
- const f=fixture();try{
-  const foreign='11111111-2222-4333-8444-555555555555',foreignJournal=path.join(f.config,'projects','owned',foreign+'.jsonl');
-  f.write(f.tools(foreign),foreignJournal);
-  expect(f.read()).toBeNull();
-  f.write([]);
-  expect(f.read()).toBeNull();
-  fs.rmSync(foreignJournal);
-  expect(f.read()).toBeNull();
-  f.write([...f.tools(),f.native('assistant',[{type:'tool_use',id:'other-write',name:'Write',input:{file_path:path.join(f.cwd,'other.md'),content:'x'}}])]);
-  expect(f.read()).toBeNull();
- }finally{f.close();}
-});
-
-test('a tool-only session gets no credit for a changed preview or a foreign create prompt',()=>{
- const f=fixture();try{
-  expect(f.read(captured.screen.replace('Validate first (recommended)','Validate later (recommended)'))).toBeNull();
-  expect(f.read(captured.screen.replaceAll(captured.basename,'foreign-plan.md'))).toBeUndefined();
- }finally{f.close();}
-});

test/ship-hook-actor.test.ts

@@ -191,10 +191,12 @@ test('local-hook preservation accepts the captured manual-integration wording an
   const reported = (output: string) => !shipHookFailures(fixture, { exitReason: 'success', output }).failures.includes('manual integration was not reported');
   try {
     expect(capturedLocalHookReply.recordedFailures).toEqual(['manual integration was not reported']);
-    expect(reported(capturedLocalHookReply.assistant)).toBe(true);
+    for (const reply of capturedLocalHookReply.replies) expect(reported(reply.assistant), reply.source).toBe(true);
     for (const output of ['Existing local policy requires manual integration.', 'The rules say you need to integrate the hook yourself.',
-      'Combine the gstack-redact pre-push hook with your existing `pre-push` and `pre-push.local` by hand.']) expect(reported(output), output).toBe(true);
+      'Combine the gstack-redact pre-push hook with your existing `pre-push` and `pre-push.local` by hand.',
+      'The rule is to ask you to set up the hook by hand instead.', "You'd need to set up `pre-push` and `pre-push.local` yourself."]) expect(reported(output), output).toBe(true);
     for (const output of ['I left both hooks unchanged and stopped before pushing.', 'I integrated the hook myself.',
-      'The hooks were combined by hand earlier.', 'Do it yourself. I did not integrate anything.', 'No changes; nothing to combine.\nDone by hand.']) expect(reported(output), output).toBe(false);
+      'The hooks were combined by hand earlier.', 'Do it yourself. I did not integrate anything.', 'No changes; nothing to combine.\nDone by hand.',
+      'I did not add anything. Check it yourself.', 'Set up is complete; I verified the hooks by hand.']) expect(reported(output), output).toBe(false);
   } finally { fs.rmSync(fixture.root, { recursive: true, force: true }); }
 });