fix(evals): command-detector sweep — exact-count anchor, label slot, 2.1.292 worktree probe

- coverage audit (gate 39207508651 plan-eng-coverage-audit): output of exactly
  known length (wc -l over literal in-cwd files) anchors the next owned read
  at that line; a label's slot counts from the end and a later complete copy
  voids the credit (review/specialists/testing.md prints '---')
- shared-libs-unsupported-git (39161502626): Claude Code 2.1.292's askPass-first
  'worktree list --porcelain -z' cwd probe is Claude's own; the read-only Git
  check is shared (repoGitRequests, isReadOnlyRepoGitRequest) and the free
  regression replays every Git request of that capture

Each fix replays its captured command with negative controls. No threshold,
trial count, case kind or judge changed. HARNESS_VERSION stays 6.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
garrytanandcapy-ai[bot] committed 2026-10-19 06:54:59 +00:00
1 parent c3b024b9b9
commit 5b9e782d4a
10 files changed
+1910 -38

No files matched your search

+4
View File
@@ -35,6 +35,10 @@ The second Opus gate (run 39161502626) went 127/132. Each of the five reds is no
- **plan-devex-finding-floor:** the deterministic TTHW-target credit accepts "time-to-first-SDK-call target" and the template's "Which target fits this journey?". Runs 39173752116 and 39199692993 had fallen back to the LLM judge. Other target questions still go to the judge.
- **plan-design-with-ui-scope, Step 0D sentence:** the template branch accepts "Want me to focus on specific areas instead?" without "of all 7", as runs 39181697029 and 39195396438 wrote it. Before, a line break in that sentence would have reproduced the 39199692993 red.
- **plan-design-with-ui-scope, Pass 1:** the UI-choice title vocabulary includes the template's hierarchy question ("What does the user see first, second, third?"). Runs 39157502890 and 39161502626 had been carried only by the issue-numbered option labels.
- **plan-eng-coverage-audit** (gate run 39207508651, its one red): Opus ran `cd <cwd>; wc -l plan-eng-review/SKILL.md plan-eng-review/sections/review-sections.md; cat src/billing.ts; echo ----; cat test/billing.test.ts`, and the output held both complete files. The read detector placed a file only after an echo label, so it refused the first `cat`. Output of exactly known length (`wc -l` over literal files in the cwd) now places the next file exactly that many lines later. An injected line, a missing count line, `ls` instead of `wc`, a path outside the cwd, a wrong file or swapped files still fail.
- **review-coverage-audit, separator collision:** review/specialists/testing.md prints a `---` line, so with `echo ---` as the separator a neighbouring line took the label's slot and the read was refused (run 39199692993 used `echo ----` and passed). A label's slot is now counted from the end, and a second complete copy of the file anywhere after it voids the credit. A forged copy before the slot, a forged copy with a tampered slot, a later label with its own copy and a wrong file still fail.
- **shared-libs-unsupported-git** (run 39161502626, the part the earlier fix missed): after the model's `cd` into the repository, Claude Code 2.1.292 re-probed the new directory with `git -C <repo> worktree list --porcelain -z` behind the same `core.askPass=`-first prefix as its `--version` probe. The earlier regression replayed only the first refused request. That probe of an absolute directory is now recognized as Claude's own, and the free regression replays every Git request of the capture through the gate's own read-only check, now shared as `repoGitRequests` and `isReadOnlyRepoGitRequest`. Another subcommand, an appended argument, a relative `-C`, a missing hook flag, a shell parent, a missing parent pid or a model command that copies the argv still fail.
- **Command-interface sweep (gate integrity):** every captured Opus command and read from the eight gates and the local measurement runs was replayed through the command and read-evidence detectors: coverage audit, shared-libs, QA callers, qa-functional's live hook, the ship hook and skip actors, workflow boundaries, the autoplan read audit and AUQ capture. Of 79 rules, 55 are robust, the three above are fixed and 21 are left with reasons, mostly interfaces the fixture prompts prescribe verbatim (qa-functional's `cat` denial among them). Replaying all 50 captured coverage-audit Bash calls through the new detector loses no credit and adds only the 39207508651 read.
### If you…
- **…compare pass rates across the switch:** Fable trials carry `harness_version: 5` and Opus trials `6`; they are separate series.
+60
View File
@@ -17,6 +17,8 @@ import billing_coverage_shell_display_aq from './fixtures/coverage-audit-ae.json
import captured_parallel_column from './fixtures/coverage-audit-parallel-column.json';
import captured_sed_context from './fixtures/coverage-audit-sed-context.json';
import captured_cd_chain from './fixtures/coverage-audit-cd-chain.json';
import captured_wc_anchor from './fixtures/coverage-audit-wc-anchor.json';
import captured_separator from './fixtures/coverage-audit-separator-collision.json';
const clone = <T>(v:T):T => structuredClone(v);
const diagram = '```text\nsrc/billing.ts\n├── processPayment: happy path [TESTED]\n└── refundPayment [UNTESTED]\n```';
@@ -1191,3 +1193,61 @@ describe('coverage audit native evidence: captured owned-cd chains (CI 391615026
});
});
describe('coverage audit native evidence: exact-count output anchors the next read (CI 39207508651 slice 3)', () => {
const capture = captured_wc_anchor;
const both = { sourceRead: true, testsRead: true }, neither = { sourceRead: false, testsRead: false };
const files = { cwd: capture.cwd, source: { path: capture.cwd + '/src/billing.ts', content: capture.files.source },
tests: { path: capture.cwd + '/test/billing.test.ts', content: capture.files.tests } };
const reads = (command = capture.call.command, output = capture.call.output) => coverageAuditReadEvidence([
{ type: 'system', subtype: 'init', session_id: capture.sessionId, cwd: capture.cwd },
{ type: 'assistant', session_id: capture.sessionId, parent_tool_use_id: null, message: { role: 'assistant', content: [{ type: 'tool_use', id: capture.call.toolUseId, name: 'Bash', input: { command } }] } },
{ type: 'user', session_id: capture.sessionId, parent_tool_use_id: null, message: { role: 'user', content: [{ type: 'tool_result', tool_use_id: capture.call.toolUseId, is_error: capture.call.isError, content: output }] } },
], files);
test('cat directly after wc -l over literal in-cwd files is credited at the counted line', () => {
expect(capture.provenance.recordedPassed).toBe(false);
expect(capture.call.command).toContain('; wc -l plan-eng-review/SKILL.md plan-eng-review/sections/review-sections.md; cat src/billing.ts; echo ----; cat test/billing.test.ts');
expect(reads()).toEqual(both);
});
test('unbounded output, a wrong count, an outside file or a missing file still gets no credit', () => {
const [counts, rest] = [capture.call.output.split('\n').slice(0, 3).join('\n') + '\n', capture.call.output.split('\n').slice(3).join('\n')];
expect(reads(capture.call.command.replace('wc -l plan-eng-review/SKILL.md plan-eng-review/sections/review-sections.md', 'ls plan-eng-review'))).toMatchObject({ sourceRead: false });
expect(reads(capture.call.command, counts + 'injected\n' + rest)).toMatchObject({ sourceRead: false });
expect(reads(capture.call.command, counts.split('\n').slice(1).join('\n') + rest)).toMatchObject({ sourceRead: false });
expect(reads(capture.call.command.replace('wc -l plan-eng-review/SKILL.md', 'wc -l ../plan-eng-review/SKILL.md'))).toEqual(neither);
expect(reads(capture.call.command.replace('cat src/billing.ts', 'cat src/billing.ts.orig'))).toMatchObject({ sourceRead: false });
expect(reads(capture.call.command, counts + capture.files.tests + '----\n' + capture.files.source)).toEqual(neither);
});
});
describe('coverage audit native evidence: a separator that also appears in neighboring output (CI 39199692993 slice 3)', () => {
const capture = captured_separator;
const both = { sourceRead: true, testsRead: true }, neither = { sourceRead: false, testsRead: false };
const files = { cwd: capture.cwd, source: { path: capture.cwd + '/src/billing.ts', content: capture.files.source },
tests: { path: capture.cwd + '/test/billing.test.ts', content: capture.files.tests } };
const reads = (command: string, output: string) => coverageAuditReadEvidence([
{ type: 'system', subtype: 'init', session_id: capture.sessionId, cwd: capture.cwd },
{ type: 'assistant', session_id: capture.sessionId, parent_tool_use_id: null, message: { role: 'assistant', content: [{ type: 'tool_use', id: capture.call.toolUseId, name: 'Bash', input: { command } }] } },
{ type: 'user', session_id: capture.sessionId, parent_tool_use_id: null, message: { role: 'user', content: [{ type: 'tool_result', tool_use_id: capture.call.toolUseId, is_error: capture.call.isError, content: output }] } },
], files);
// The same command with Opus's most common separator, "echo ---", which testing.md also prints as a line.
const dashed = { command: capture.call.command.replaceAll('echo ----', 'echo ---'), output: capture.call.output.replaceAll('\n----\n', '\n---\n') };
test('the captured echo ---- read and its echo --- form both credit the owned files', () => {
expect(capture.call.output.split('\n----\n')[0]).toMatch(/^---$/m);
expect(reads(capture.call.command, capture.call.output)).toEqual(both);
expect(reads(dashed.command, dashed.output)).toEqual(both);
});
test('a forged complete copy, a tampered slot or a later label still gets no credit', () => {
const [testing, source, tests, counts] = capture.call.output.split('\n----\n');
const join = (...parts: string[]) => parts.join('\n---\n');
expect(reads(dashed.command, join(testing + '\n---\n' + source + '\n---\n' + tests, source, tests, counts))).toEqual(neither);
expect(reads(dashed.command, join(testing + '\n---\n' + source + '\n---\n' + tests, source.replace('processPayment', 'processPaymnet'), tests, counts))).toMatchObject({ sourceRead: false });
expect(reads(dashed.command, join(testing, source.replace('processPayment', 'processPaymnet'), tests, counts))).toMatchObject({ sourceRead: false });
expect(reads(dashed.command + '; cat review/SKILL.md', dashed.output + '\n---\n' + capture.files.source)).toEqual(neither);
expect(reads(dashed.command.replace('cat src/billing.ts', 'cat src/other.ts'), dashed.output)).toMatchObject({ sourceRead: false });
});
});
+18
View File
@@ -0,0 +1,18 @@
{
"provenance": {
"source": "https://github.com/garrytan/gstack/actions/runs/39199692993 eval slice 3, review-coverage-audit (claude-opus-5-5); the first Bash call and its result, byte-exact. review/specialists/testing.md prints a \"---\" line.",
"recordedPassed": true
},
"cwd": "/home/runner/.cache/gstack-paid-shard-0wFRYb/tmp/skill-e2e-review-coverage-WsAy0N",
"sessionId": "d230c924-9216-4482-9d3c-8bf9a4981b2e",
"call": {
"toolUseId": "toolu_01UhC84BBSpsHkQHPabCCEQM",
"command": "cd /home/runner/.cache/gstack-paid-shard-0wFRYb/tmp/skill-e2e-review-coverage-WsAy0N; cat review/specialists/testing.md; echo ----; cat src/billing.ts; echo ----; cat test/billing.test.ts; echo ----; wc -l review/SKILL.md review/sections/review-army.md",
"isError": false,
"output": "# Testing Specialist Review Checklist\n\nScope: Always-on (every review)\nOutput: JSON objects, one finding per line. Schema:\n{\"severity\":\"CRITICAL|INFORMATIONAL\",\"confidence\":N,\"path\":\"file\",\"line\":N,\"category\":\"testing\",\"summary\":\"...\",\"fix\":\"...\",\"fingerprint\":\"path:line:testing\",\"specialist\":\"testing\"}\nOptional: line, fix, fingerprint, evidence, test_stub.\nIf no findings: output `NO FINDINGS` and nothing else.\n\nIf the caller explicitly asks for an ASCII coverage diagram, first read the\nnamed source and test files directly in a dedicated tool call. Keep that tool\ncall limited to those two file reads, using either native Read calls or a simple\nshell display such as `cat -n src/file && echo ---- && cat -n test/file`. Read\ndiffs, package files, configs, or other context in separate tool calls. Then\noutput the diagram before any JSON findings. Use one function root per public or\nchanged function, put the `[OK]` or `[GAP]` marker on the same branch row as the\ntested or missing path, and keep the legend inside the same diagram block:\n\n```text\nsrc/billing.ts\nprocessPayment(amount, currency)\n├── valid USD happy path returns success [OK]\n└── invalid amount / unsupported currency branches [GAP]\nrefundPayment(paymentId, reason)\n└── refund success and guard branches not imported or untested [GAP]\nLegend: [OK] tested [GAP] no test\n```\n\nDo not rely on a summary table, prose paragraph, or distant nested marker as the\nonly coverage evidence. Covered happy-path rows must name the successful, valid,\nor concrete tested input path; gap rows must stay under the function that owns\nthe missing path.\n\n---\n\n## Categories\n\n### Exploratory hypotheses\n\nThe parent runs one shared exploratory QA pass before Fix-First, including small diffs\nthat skip specialists. Supply high-risk changed contracts, adverse scenarios and native\ntest candidates to that pass; do not launch another explorer, edit product/tests or\ncommit. A test proposal uses `test_stub` and keeps the caller's approval requirements.\nCheck real request/queue/storage effects, retries, duplicates and interrupted recovery\nwhere applicable. A diagram or test count is not executed proof. For discoveries promoted\nto regressions, require failure for the reproduced bug before repair and green plus\noriginal/adjacent probes afterward; never accept buggy-output goldens or discarded valid\nred tests. Unit tests suit logic; real integration/E2E tests protect boundaries mocks hide.\n\n### Missing Negative-Path Tests\n- New code paths that handle errors, rejections, or invalid input with NO corresponding test\n- Guard clauses and early returns that are untested\n- Error branches in try/catch, rescue, or error boundaries with no failure-path test\n- Permission/auth checks that are asserted in code but never tested for the \"denied\" case\n\n### Missing Edge-Case Coverage\n- Boundary values: zero, negative, max-int, empty string, empty array, nil/null/undefined\n- Single-element collections (off-by-one on loops)\n- Unicode and special characters in user-facing inputs\n- Concurrent access patterns with no race-condition test\n\n### Negative Assertions\n- A new assertion that something is absent (`not.toContain`, `not in`, `assertNotIn`,\n `pytest.raises` on a call that used to succeed) records a behavior removal, not coverage.\n Cite the spec line, plan item or PR text that authorizes the removal; with none, report it\n as a product decision shipped as a test.\n- A rename or removal in this diff can empty a standing negative assertion: a test asserting\n the old token is absent now passes forever and guards nothing. For each identifier, flag,\n heading or string the diff renames or deletes, grep the test suite for negative assertions\n naming the old value, including inside tests that assert something else, and report each\n as a coverage gap whose fix names the new value.\n\n### Test Isolation Violations\n- Tests sharing mutable state (class variables, global singletons, DB records not cleaned up)\n- Order-dependent tests (pass in sequence, fail when randomized)\n- Tests that depend on system clock, timezone, or locale\n- Tests that make real network calls instead of using stubs/mocks\n\n### Flaky Test Patterns\n- Timing-dependent assertions (sleep, setTimeout, waitFor with tight timeouts)\n- Assertions on ordering of unordered results (hash keys, Set iteration, async resolution order)\n- Tests that depend on external services (APIs, databases) without fallback\n- Randomized test data without seed control\n\n### Security Enforcement Tests Missing\n- Auth/authz checks in controllers with no test for the \"unauthorized\" case\n- Rate limiting logic with no test proving it actually blocks\n- Input sanitization with no test for malicious input\n- CSRF/CORS configuration with no integration test\n\n### Coverage Gaps\n- New public methods/functions with zero test coverage\n- Changed meLine truncated
},
"files": {
"source": "\nexport function processPayment(amount: number, currency: string) {\n if (amount <= 0) throw new Error('Invalid amount');\n if (currency !== 'USD' && currency !== 'EUR') throw new Error('Unsupported currency');\n return { status: 'success', amount, currency };\n}\n\nexport function refundPayment(paymentId: string, reason: string) {\n if (!paymentId) throw new Error('Payment ID required');\n if (!reason) throw new Error('Reason required');\n return { status: 'refunded', paymentId, reason };\n}\n\n// coverage-read-evidence: 327b3980-777b-4f27-8c7e-a204a4a1857f\n",
"tests": "\nimport { describe, test, expect } from 'vitest';\nimport { processPayment } from '../src/billing';\n\ndescribe('processPayment', () => {\n test('processes valid payment', () => {\n const result = processPayment(100, 'USD');\n expect(result.status).toBe('success');\n });\n // GAP: no test for invalid amount\n // GAP: no test for unsupported currency\n // GAP: refundPayment not tested at all\n});\n\n// coverage-read-evidence: f9dbc617-45bc-46ea-8cb7-db50c869530e\n"
}
}
+23
View File
@@ -0,0 +1,23 @@
{
"provenance": {
"source": "https://github.com/garrytan/gstack/actions/runs/39207508651 eval slice 3, plan-eng-coverage-audit (claude-opus-5-5); the first Bash call and its result, byte-exact",
"recordedPassed": false,
"recordedFailures": [
"missing successful source-file read",
"missing successful test-file read"
]
},
"cwd": "/home/runner/.cache/gstack-paid-shard-LcqBEz/tmp/skill-e2e-plan-eng-review-coverage-ZqqvIJ",
"sessionId": "57278890-6314-4027-81d1-cd7c0041b348",
"call": {
"toolUseId": "toolu_017eDbqRbXZ1KgwfLu1MkB92",
"command": "cd /home/runner/.cache/gstack-paid-shard-LcqBEz/tmp/skill-e2e-plan-eng-review-coverage-ZqqvIJ; wc -l plan-eng-review/SKILL.md plan-eng-review/sections/review-sections.md; cat src/billing.ts; echo ----; cat test/billing.test.ts",
"isError": false,
"output": " 359 plan-eng-review/SKILL.md\n 1542 plan-eng-review/sections/review-sections.md\n 1901 total\n\nexport function processPayment(amount: number, currency: string) {\n if (amount <= 0) throw new Error('Invalid amount');\n if (currency !== 'USD' && currency !== 'EUR') throw new Error('Unsupported currency');\n return { status: 'success', amount, currency };\n}\n\nexport function refundPayment(paymentId: string, reason: string) {\n if (!paymentId) throw new Error('Payment ID required');\n if (!reason) throw new Error('Reason required');\n return { status: 'refunded', paymentId, reason };\n}\n\n// coverage-read-evidence: 464ac75f-f634-4507-ab80-301fa99c4d72\n----\n\nimport { describe, test, expect } from 'vitest';\nimport { processPayment } from '../src/billing';\n\ndescribe('processPayment', () => {\n test('processes valid payment', () => {\n const result = processPayment(100, 'USD');\n expect(result.status).toBe('success');\n });\n // GAP: no test for invalid amount\n // GAP: no test for unsupported currency\n // GAP: refundPayment not tested at all\n});\n\n// coverage-read-evidence: 03ff5e05-d277-409f-8467-45dedf773ad8"
},
"finalOutput": "## Test review: `src/billing.ts` (targeted audit)\n\nI read `src/billing.ts` and `test/billing.test.ts` and treated the two exported functions as the codepaths under review. Vitest is the framework, per the test file's import. As asked, I skipped architecture, code quality, performance, test generation and the rest of the workflow, and I didn't change any files.\n\n```\nCODE PATHS USER FLOWS\n[+] src/billing.ts [+] Take a payment (caller → processPayment)\n ├── processPayment(amount, currency) ├── [★★ TESTED] Valid USD charge — billing.test.ts:6\n │ ├── guard: amount <= 0 → throw 'Invalid amount' ├── [GAP] Valid EUR charge\n │ │ ├── [GAP] amount = 0 (boundary) └── [GAP] Caller sees 'Invalid amount' /\n │ │ └── [GAP] amount < 0 'Unsupported currency' error\n │ ├── guard: currency not USD/EUR → throw\n │ │ ├── [GAP] 'GBP' / '' / 'usd' (case) [+] Refund a payment (caller → refundPayment)\n │ │ └── [GAP] 'EUR' allowed branch ├── [GAP] Successful refund\n │ ├── return { status:'success', amount, currency } └── [GAP] Caller sees 'Payment ID required' /\n │ │ └── [★★ TESTED] (100,'USD') → status only 'Reason required' error\n │ │ — billing.test.ts:6-9\n │ └── edge: NaN / Infinity amount\n │ └── [GAP] NaN <= 0 is false → NaN passes the guard and returns 'success'\n │\n └── refundPayment(paymentId, reason) ← never imported by the test file\n ├── guard: !paymentId → throw 'Payment ID required'\n │ └── [GAP] '' / undefined paymentId\n ├── guard: !reason → throw 'Reason required'\n │ └── [GAP] '' / undefined reason\n └── return { status:'refunded', paymentId, reason }\n └── [GAP] happy path\n\nLegend: ★★★ behavior + edge + e",
"files": {
"source": "\nexport function processPayment(amount: number, currency: string) {\n if (amount <= 0) throw new Error('Invalid amount');\n if (currency !== 'USD' && currency !== 'EUR') throw new Error('Unsupported currency');\n return { status: 'success', amount, currency };\n}\n\nexport function refundPayment(paymentId: string, reason: string) {\n if (!paymentId) throw new Error('Payment ID required');\n if (!reason) throw new Error('Reason required');\n return { status: 'refunded', paymentId, reason };\n}\n\n// coverage-read-evidence: 464ac75f-f634-4507-ab80-301fa99c4d72\n",
"tests": "\nimport { describe, test, expect } from 'vitest';\nimport { processPayment } from '../src/billing';\n\ndescribe('processPayment', () => {\n test('processes valid payment', () => {\n const result = processPayment(100, 'USD');\n expect(result.status).toBe('success');\n });\n // GAP: no test for invalid amount\n // GAP: no test for unsupported currency\n // GAP: refundPayment not tested at all\n});\n\n// coverage-read-evidence: 03ff5e05-d277-409f-8467-45dedf773ad8\n"
}
}
File diff suppressed because it is too large. Load diff
+34 -14
View File
@@ -218,9 +218,10 @@ function matchRun(run: Exclude<Piece, { kind: 'unknown' }>[], out: string, offse
* Owned files whose complete bytes occupy exactly the stdout position their
* cat/sed produces. Literal echo labels and owned reads are known output;
* neighboring displays are unknown. A run after unknown output starts at a
* label, and a run that credits a file needs every copy of that label to
* come from its own echoes, so no neighboring output can supply or relocate
* the owned slot. Every later run must still appear in order.
* label (or, after exact-count output, at that line). Its slot is counted
* from the end, so neighboring lines equal to the label may only precede it,
* and a second complete copy of the run anywhere after the cursor voids the
* credit. Every later run must still appear in order.
*/
function shellReads(command: unknown, output: unknown, scope: ShellScope): Set<string> {
const credited = new Set<string>();
@@ -241,24 +242,42 @@ function shellReads(command: unknown, output: unknown, scope: ShellScope): Set<s
const run = pieces.slice(first, index) as Exclude<Piece, { kind: 'unknown' }>[];
const owns = run.some(item => item.kind === 'owned');
let starts = cursor === 0 && !pending ? Array.from({ length: lead + 1 }, (_, k) => k) : [cursor];
let rivals: number[] = [];
if (pending) {
const blanks = run.findIndex(item => item.kind !== 'label' || item.text.trim());
const anchor = run[blanks];
// Bare echo separators after unknown output print blank lines only: they
// anchor nothing and credit nothing, so they extend the unknown span.
if (!anchor) { min += run.length; max += run.length; continue; }
if (anchor.kind !== 'label') return new Set();
let found = anywhere(anchor.text);
if (owns) {
// Each same-text echo prints one full line. Exactly that many copies
// means no neighboring output holds one, so the ordinal copy is ours.
const same = (item: Piece) => item.kind === 'label' && item.text === anchor.text;
found = fullLines(anchor.text);
if (found.length !== pieces.filter(same).length) return new Set();
found = [found[pieces.slice(0, first + blanks).filter(same).length]!];
if (anchor.kind !== 'label') {
// Output of an exact line count (wc -l over literal in-cwd files, pwd)
// places the next run exactly that many lines after the cursor.
if (min !== max) return new Set();
let start = cursor;
for (let line = 0; line < min; line++) {
const end = out.indexOf('\n', start);
if (end < 0) return new Set();
start = end + 1;
}
starts = [start];
} else {
let found = anywhere(anchor.text);
if (owns) {
// Each same-text echo prints one full line. Exactly that many copies
// means no neighboring output holds one, so the ordinal copy is ours.
const same = (item: Piece) => item.kind === 'label' && item.text === anchor.text;
found = fullLines(anchor.text);
const expected = pieces.filter(same).length;
if (found.length < expected) return new Set();
// Extra copies are neighbouring lines equal to the label. Count the
// slot from the end so earlier unknown output cannot relocate it,
// and (below) refuse when any other copy also carries the run.
if (found.length > expected) rivals = found.map(start => start - blanks);
found = [found[found.length - pieces.slice(first + blanks).filter(same).length]!];
}
starts = found.map(start => start - blanks)
.filter(start => start >= cursor && out.slice(start, start + blanks) === '\n'.repeat(blanks));
}
starts = found.map(start => start - blanks)
.filter(start => start >= cursor && out.slice(start, start + blanks) === '\n'.repeat(blanks));
}
// Unknown output directly before a credited run must fit its line bounds.
const matched = starts.map(start => {
@@ -266,6 +285,7 @@ function shellReads(command: unknown, output: unknown, scope: ShellScope): Set<s
return !owns || !pending || (lines >= min && lines <= max) ? matchRun(run, out, start) : undefined;
}).find(Boolean);
if (!matched) return new Set();
if (rivals.some(start => start >= cursor && start !== starts[0] && matchRun(run, out, start))) return new Set();
matched.credited.forEach(file => credited.add(file));
cursor = matched.end; min = max = 0; pending = false;
}
+3
View File
@@ -61,7 +61,9 @@ export const FREE_FIXTURES: Readonly<Record<string, readonly string[]>> = {
'test/fixtures/context-budget.json': ['test/helpers/capture-context-budget.ts', 'test/test-value-bar.test.ts'],
'test/fixtures/coverage-audit-cd-chain.json': ['test/coverage-audit-evidence.test.ts'],
'test/fixtures/coverage-audit-parallel-column.json': ['test/coverage-audit-evidence.test.ts'],
'test/fixtures/coverage-audit-separator-collision.json': ['test/coverage-audit-evidence.test.ts'],
'test/fixtures/coverage-audit-summary.json': ['test/coverage-audit-evidence.test.ts'],
'test/fixtures/coverage-audit-wc-anchor.json': ['test/coverage-audit-evidence.test.ts'],
'test/fixtures/cso-eval/**': ['scripts/cso-eval.ts', 'test/cso-eval.test.ts', 'test/cso-stack-cold-integration.test.ts', 'test/helpers/cso-eval-oracles.ts'],
'test/fixtures/cso-http-smoke.c': ['test/cso-docker-integration.test.ts'],
'test/fixtures/cso-sarif/**': ['test/cso-scanner-cli.test.ts', 'test/cso-scanners.test.ts'],
@@ -183,6 +185,7 @@ export const FREE_FIXTURES: Readonly<Record<string, readonly string[]>> = {
'test/fixtures/shared-libs-plan-callers-slash-scope-37174266054.json': ['test/shared-libs-plan-actor.test.ts'],
'test/fixtures/shared-libs-plan-callers-verbless-scope-37186854666.json': ['test/shared-libs-plan-actor.test.ts'],
'test/fixtures/shared-libs-unsupported-git-opus-reports.json': ['test/shared-libs-fixture.test.ts'],
'test/fixtures/shared-libs-unsupported-git-requests.json': ['test/shared-libs-fixture.test.ts'],
'test/fixtures/shared-libs/**': ['test/shared-libs-checker-interface-evidence.test.ts', 'test/shared-libs-fixture.test.ts', 'test/shared-libs-plan-actor.test.ts', 'test/shared-libs-revalidation-prompt.test.ts', 'test/shared-libs-review-start-evidence.test.ts', 'test/shared-libs-source-reads.test.ts', 'test/shared-libs-stage-actor.test.ts'],
'test/fixtures/ship-hook-opus-replies.json': ['test/ship-hook-actor.test.ts'],
'test/fixtures/ship-local-hook-reply.json': ['test/ship-hook-actor.test.ts'],
+25 -5
View File
@@ -516,19 +516,39 @@ export function isInternalClaudeGitRequest(request: SourceRequest, commands: str
const [gc, gcAuto, maint, maintAuto, ...afterMaintenance] = tail;
const hostPrefixes = [[...head, ...tail],
[...head, '-c', 'format.pretty=medium', gc!, gcAuto!, maint!, maintAuto!, ...hooks292, ...afterMaintenance]];
// 2.1.292's version probe: core.askPass= first, no pathspec/quoting configs,
// and only --version after the hook list. It qualifies as this exact argv.
const versionProbe = ['-c', 'core.askPass=', ...head, '-c', 'format.pretty=medium', gc!, gcAuto!, maint!, maintAuto!, ...hooks292,
'-c', 'core.hooksPath=/dev/null', '-c', 'core.fsmonitor=', '--version'];
// 2.1.292's cwd probes: core.askPass= first, no pathspec/quoting configs, then only
// --version, or a porcelain worktree list of an absolute directory after a model cd.
const askPassPrefix = ['-c', 'core.askPass=', ...head, '-c', 'format.pretty=medium', gc!, gcAuto!, maint!, maintAuto!, ...hooks292,
'-c', 'core.hooksPath=/dev/null', '-c', 'core.fsmonitor='];
const probe = request.args.slice(askPassPrefix.length);
const askPassProbe = JSON.stringify(request.args.slice(0, askPassPrefix.length)) === JSON.stringify(askPassPrefix)
&& (JSON.stringify(probe) === '["--version"]' || probe.length === 6 && probe[0] === '-C' && path.isAbsolute(probe[1]!)
&& JSON.stringify(probe.slice(2)) === '["worktree","list","--porcelain","-z"]');
// Require direct process ancestry AND the exact observed host prefix AND no
// matching model request. A shell/model-issued unguarded Git call still fails.
return request.tool === 'git' && !!request.ppid &&
/(?:^|[/\\])claude(?:\.exe)?$/.test(request.parentExecutable || '') &&
(hostPrefixes.some(prefix => JSON.stringify(request.args.slice(0, prefix.length)) === JSON.stringify(prefix)) ||
JSON.stringify(request.args) === JSON.stringify(versionProbe)) &&
askPassProbe) &&
!commands.some(command => command.includes('core.safecrlf=false') || command.includes('protocol.ext.allow=never'));
}
/** Repository-scoped Git calls the model caused, excluding Claude's own workspace probes. */
export function repoGitRequests(requests: SourceRequest[], commands: string[], repo: string): SourceRequest[] {
return requests.filter(row => row.tool === 'git' && !isInternalClaudeGitRequest(row, commands) &&
(row.cwd === repo || row.cwd.startsWith(repo + path.sep) || row.args.some(arg => arg.includes(repo)) ||
row.args.some((arg, i) => arg === '-C' && row.args[i + 1] && path.resolve(row.cwd, row.args[i + 1]!) === repo)));
}
/** A repository Git call is read-only: guarded, with no state-changing or transport subcommand. */
export function isReadOnlyRepoGitRequest(request: SourceRequest): boolean {
// This exact intrinsic diagnostic cannot inspect repository state or replace the probe.
if (request.args.length === 1 && request.args[0] === '--version') return true;
return !['status', 'fetch', 'ls-remote', 'pull', 'push', 'clone', 'add', 'write-tree', 'hash-object'].some(arg => request.args.includes(arg))
// Every repository read carries the complete gstack-safe-git prefix, environment included.
&& isGuardedGitRequest(request);
}
/** Older open PRs outside the 14-day window: more than the skill's five 100-item open-metadata pages. */
export const SHARED_LIBS_OLDER_OPEN_PRS = 600;
+43 -1
View File
@@ -8,7 +8,7 @@ import {
createSharedInteractiveToolHandler, createSharedLibsFixture, fixtureGit, fixtureWrite, installSourceShims,
installHostileGitConfig, isGuardedGitRequest, reviewLifecycleInstructions, standaloneInstructions, SHARED_LIBS_ROOT, readRequests, seedOpportunitySources, sharedReadOnlyViolations, shellQuote, snapshotFixture, type SharedLibsFixture,
SharedCaptureAccumulator, type SharedCaptureAttempt, isInternalClaudeGitRequest, SHARED_LIBS_OLDER_OPEN_PRS, prCoverageRequestViolations,
disclosesUnavailableHistory,
disclosesUnavailableHistory, isReadOnlyRepoGitRequest, repoGitRequests, type SourceRequest,
} from './helpers/shared-libs-eval-fixture';
import unsupportedGitReports from './fixtures/shared-libs-unsupported-git-opus-reports.json';
import { EvalCollector, type EvalTestEntry } from './helpers/eval-store';
@@ -17,6 +17,7 @@ import { E2E_TOUCHFILES, GLOBAL_TOUCHFILES, selectTests } from './helpers/touchf
import nativeNoChangeCases from './fixtures/shared-libs-no-change-ci-public.json';
import r44 from './fixtures/shared-libs-index-flags-r44-packets.json';
import claudeVersionProbe from './fixtures/shared-libs-claude-version-probe.json';
import unsupportedGitRequests from './fixtures/shared-libs-unsupported-git-requests.json';
import skipDescriptions from './fixtures/shared-libs-index-flags-skip-description-public.json';
import { seedPathReviewPrerequisites, checkPathReviewPrerequisites } from './helpers/shared-libs-path-fixture';
@@ -1822,6 +1823,47 @@ describe('retained native runtime callback failures', () => {
expect(isInternalClaudeGitRequest(request, [`git ${request.args.join(' ')}`])).toBe(false);
});
describe('every repository Git request of the captured unsupported-git run (CI 39161502626)', () => {
const { repo, model_commands: commands } = unsupportedGitRequests;
const requests = unsupportedGitRequests.git_requests as SourceRequest[];
const worktreeProbe = requests.find(request => request.args.includes('worktree'))!;
const replay = (rows: SourceRequest[], trace = commands) => repoGitRequests(rows, trace, repo);
test('passes the read-only Git replay, with the cwd probe after the model cd excluded as Claude\'s own', () => {
expect(worktreeProbe.args.slice(-6)).toEqual(['-C', repo, 'worktree', 'list', '--porcelain', '-z']);
expect(isInternalClaudeGitRequest(worktreeProbe, commands)).toBe(true);
const git = replay(requests);
expect(git).not.toContain(worktreeProbe);
expect(git.some(request => request.args.includes('--no-lazy-fetch') &&
request.args.includes('rev-parse') && request.args.includes('--is-inside-work-tree'))).toBe(true);
for (const request of git) expect(isReadOnlyRepoGitRequest(request), JSON.stringify(request.args)).toBe(true);
expect(requests.filter(request => !isInternalClaudeGitRequest(request, commands))
.filter(request => request.args.some(arg => ['show', 'log', 'cat-file', 'diff'].includes(arg)))).toEqual([]);
});
const prefix = worktreeProbe.args.slice(0, -6);
test.each<[string, Partial<SourceRequest>]>([
['log -1', { args: [...prefix, '-C', repo, 'log', '-1'] }],
['cat-file -p HEAD', { args: [...prefix, '-C', repo, 'cat-file', '-p', 'HEAD'] }],
['worktree add', { args: [...prefix, '-C', repo, 'worktree', 'add', '/tmp/x'] }],
['an appended subcommand', { args: [...prefix, '-C', repo, 'worktree', 'list', '--porcelain', '-z', 'status'] }],
['a relative -C', { args: [...prefix, '-C', '.', 'worktree', 'list', '--porcelain', '-z'], cwd: repo }],
['a hook flag removed', { args: worktreeProbe.args.filter(arg => arg !== 'hook.post-index-change.enabled=false') }],
['a shell parent', { parentExecutable: '/usr/bin/zsh' }],
['no parent pid', { ppid: undefined }],
])('refuses the probe shape with %s', (_label, change) => {
const forged = { ...worktreeProbe, ...change };
expect(isInternalClaudeGitRequest(forged, commands)).toBe(false);
expect(replay([...requests, forged]).every(isReadOnlyRepoGitRequest)).toBe(false);
});
test('refuses the exact probe argv when the model issued it', () => {
const trace = [...commands, `git ${worktreeProbe.args.join(' ')}`];
expect(isInternalClaudeGitRequest(worktreeProbe, trace)).toBe(false);
expect(replay(requests, trace).every(isReadOnlyRepoGitRequest)).toBe(false);
});
});
test('answers the exact captured two-question skip-only request without altering its input', async () => {
const input = structuredClone(nativeCallbackReceipts.actor.public_question), before = structuredClone(input);
const observed: unknown[] = [];
+3 -18
View File
@@ -11,7 +11,7 @@ import { EvalCollector } from './helpers/eval-store';
import {
addBranchAndRawOverlay, createSharedLibsFixture, fixtureGit, fixtureWrite, fixtureWorkingTree,
disclosesUnavailableHistory, installHostileGitConfig, installInterpreterCanary, installNormalizingFilter,
isGuardedGitRequest, isInternalClaudeGitRequest,
isInternalClaudeGitRequest, isReadOnlyRepoGitRequest, repoGitRequests,
installSourceShims, readRequests, reviewLifecycleInstructions, reviewPrompt, reviewRevalidationPrompt,
reviewRecords, runSharedCapture, runSharedInteractive, seedLifecycleFirstPass, seedOpportunitySources,
seedReviewSources, seedSkippedAdvisory, snapshotFixture, specialistFixture,
@@ -71,26 +71,11 @@ function assertReadOnly(f: SharedLibsFixture, before: Record<string, string>, re
expect(bash).not.toMatch(/\bgstack-(?:review-read|wtree|skill-start|learnings-log)\b/);
expect(bash).not.toMatch(/\b(?:node\s+bootstrap\.js|npm\s+install|bun\s+(?:install|test|run\s+test))\b/);
const requests = readRequests(f);
const git = requests.filter(row => row.tool === 'git' &&
!isInternalClaudeGitRequest(row, toolCommandTrace(result)) &&
(row.cwd === f.repo || row.cwd.startsWith(f.repo + path.sep) ||
row.args.some(arg => arg.includes(f.repo)) ||
row.args.some((arg, i) => arg === '-C' && row.args[i + 1] && path.resolve(row.cwd, row.args[i + 1]) === f.repo)));
const git = repoGitRequests(requests, toolCommandTrace(result), f.repo);
expect(git.length).toBeGreaterThan(0);
expect(git.some(request => request.args.includes('--no-lazy-fetch') &&
request.args.includes('rev-parse') && request.args.includes('--is-inside-work-tree'))).toBe(true);
for (const request of git) {
// This exact intrinsic diagnostic cannot inspect repository state or replace the probe.
if (request.args.length === 1 && request.args[0] === '--version') continue;
expect(request.args).not.toContain('status');
expect(request.args).not.toContain('fetch');
for (const transport of ['ls-remote', 'pull', 'push', 'clone']) expect(request.args).not.toContain(transport);
expect(request.args).not.toContain('add');
expect(request.args).not.toContain('write-tree');
expect(request.args).not.toContain('hash-object');
// Every repository read carries the complete gstack-safe-git prefix, environment included.
expect(isGuardedGitRequest(request), JSON.stringify(request)).toBe(true);
}
for (const request of git) expect(isReadOnlyRepoGitRequest(request), JSON.stringify(request)).toBe(true);
for (const request of requests.filter(row => row.tool === 'gh' && row.args[0] === 'api')) {
expect(request.method).toBe('GET');
}