mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-01 22:45:02 +01:00
v1.92.33.0 fix(flaky): 2026-10-31 flaky pass — paid detector gaps, Windows cold first launches, fixture races, ubicloud pack index copy (GSTA-199) (#3697)
* fix(flaky): 2026-10-31 flaky pass: paid detector gaps, Windows cold first launches, cleanup and settlement races, ubicloud pack index copy (GSTA-199) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(ship-budget): name the child-list probe timeout so the literals check reads the spawn timeout unchanged Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes (Step 11): linear passive-pitch clause scan, tokenized grep operands, harness diagnostic reset and early stderr Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: changelog fragment for the 2026-10-31 flaky pass Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore(harness): non-behavioral HARNESS_VERSION decision for the llm-judge byte counter Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: changelog fragment path and count Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(testing): llm-judge-aborted lists bytesReceived and lastByteMs Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * revert(flaky): leave the cso-snapshot SIGCONT fix for after the SBOM generator re-pin (GST-495) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed
1 parent
84c5019e23
commit
1cac25cb8e
32 files changed
+670
-61
No files matched your search
@@ -1,5 +1,24 @@
|
||||
# Changelog
|
||||
|
||||
## [1.92.33.0] - 2026-11-01
|
||||
|
||||
**Eleven flaky tests from the 2026-10-31 flaky pass stop failing correct runs, and six more now name their cause when they fail.** Three paid detectors missed correct replies, one product script copied a file git was rewriting, and the free-test reds came from a binary's slow first launch on a fresh Windows runner, a fixture written in two steps and a cleanup that raced the runner's own scanner.
|
||||
|
||||
### Fixed
|
||||
- **`scripts/ubicloud/ubi-runner.sh`**: `pack` copies the live `.git/index` with `cat` instead of `cp`. Git renames `index.lock` over the index, and GNU cp 9.4 then fails with "replaced while being copied" (Free Tests 41073582974). A loop that swaps the index during 300 packs failed 10 times before and 0 after.
|
||||
- **Windows cold first launch** (`test/helpers/warm-first-exec.ts`): `test/land-and-deploy-flow.test.ts`, `test/skill-positional-literals.test.ts`, `test/team-init-hook-shell-neutral.test.ts` and `browse/test/cookie-import-node.test.ts` launch jq, awk or PowerShell once, unasserted, before the timed calls. On 12 fresh windows-latest VMs (run 41205346554) a first launch cost jq 60-250 ms, awk 190-290 ms and PowerShell 1.7-3.1 s more than a warm one, and in every red the file's first launch was the call that hit its kill (Windows Free Tests 41195098239, 41196095371, 41175329114, 41174352235, 41175308095, 41187420609, 41152107410). Every timed call keeps its kill.
|
||||
- **`browse/test/terminal-agent-publication-lock.test.ts`**: the preload writes its `ready` file to a temporary name and renames it, so the kill hook never parses a half-written file (Free Tests 41156375412).
|
||||
- **`test/qa-only-browser-probe.test.ts`**: on a Windows EBUSY the cleanup asks Restart Manager who holds browse.exe, waits up to 5 s only when the holder is outside the test (the runner's `provjobd.exe`), then removes once more; a test-host holder or a second failure throws with the owners named (Windows Free Tests 41068546881, 41172879765).
|
||||
- **`test/strict-output-settlement.test.ts`**: the missing-close case gives its child a 2 s wall instead of 150 ms. It asserts no elapsed time, and a child on a loaded Windows runner took over 150 ms to exit (Windows Free Tests 41164892775).
|
||||
|
||||
### Gate integrity
|
||||
- **plan-design-with-ui-scope** (`test/skill-e2e-plan-design-with-ui.test.ts`): the focus question also counts when its "All 7" option carries a letter, as in "A) All 7 passes" (paid gates 41178131219 and 41186376825). Both questions are saved as fixtures; the phrase not leading the label, a 4/7 rating and a label without "focus" still count zero.
|
||||
- **review-coverage-audit** (`test/helpers/coverage-audit-evidence.ts`): a grep display over two or more literal files after the reads no longer voids the whole call (gate 41149166998). An option, a substitution, a redirect or a tampered read still gets no credit, and a quoted file name with a space stays one operand.
|
||||
- **tpa-absent** (`test/helpers/third-party-actions.ts`): the passive pitch's negation check covers only the clause that holds "can be downloaded", so "Aside is not installed but can be downloaded at aside.com" counts once; "cannot", "never", "not something that" and "is not available to download" still count zero. The clause scan is linear: 112 KB of model text takes about 0.2 s, not 7 s.
|
||||
|
||||
### Changed
|
||||
- Seven tests and helpers gain diagnostics so the next red names its cause. The workflow judge records response bytes and the time of the last byte, so a silent socket and a model thinking on a live connection look different (gate 41185637598); `test/gstack-ship-queue.test.ts` keeps the refusing call's stderr; the ship-budget harness names a call that ended without an exit code; `test/helpers/plan-review-board-feedback.ts`, `test/team-init-hook-shell-neutral.test.ts` and `test/ship-hook-windows-paths.test.ts` print each step's or spawn's time; the ios-qa loopback tests use a listener that drops every connection and a 10 s proxy timeout, so a stuck hop fails as a named 504 instead of a bare 30 s test timeout.
|
||||
|
||||
## [1.92.32.0] - 2026-11-01
|
||||
|
||||
**Four flakes from the 2026-10-29 flaky pass stop failing correct runs.** Three free-tier tests failed on harness timing (a half-written marker file, a Windows kill that left a process tree behind, a hook kill set at 5 s), and the plan-devex-finding-floor paid case waited out 600 s after a provider error. Each repair has a free test that fails without it.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# gstack digest v1.92.32.0 — regenerate/re-copy after upgrading gstack
|
||||
# gstack digest v1.92.33.0 — regenerate/re-copy after upgrading gstack
|
||||
|
||||
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
|
||||
for agent hosts without a full skill install. The full skills add workflows,
|
||||
|
||||
@@ -45,6 +45,16 @@ beforeAll(() => {
|
||||
writePlain(path.join(root, 'AppData/Local/Opera Software/Opera GX Stable/Default/Network'), '.gx.fixture', 'gx-cookie', 'local-gx-decoy');
|
||||
});
|
||||
|
||||
// Windows Free Tests 41152107410 (shard 2): the shard's first powershell.exe took over 30 s and hit the protect call's
|
||||
// kill. That first call is a cold start: 2.9-3.3 s on 3 fresh runners where later calls take 0.24-0.38 s (GSTA-199
|
||||
// scratch run 41205291567), and the first-in-shard DPAPI case ran 13.6, 15.4 and 24.5 s in 3 of 40 lane runs (2.5-5.1 s
|
||||
// in the rest). Pay it once here, unasserted; the protect call keeps its 30 s kill.
|
||||
beforeAll(() => {
|
||||
if (process.platform !== 'win32') return;
|
||||
spawnSync(path.win32.join(process.env.SystemRoot || 'C:\\Windows', 'System32', 'WindowsPowerShell', 'v1.0', 'powershell.exe'),
|
||||
['-NoProfile', '-NonInteractive', '-Command', 'Add-Type -AssemblyName System.Security'], { encoding: 'utf8', timeout: 25_000, windowsHide: true });
|
||||
}, 30_000);
|
||||
|
||||
afterAll(() => rmSync(root, { recursive: true, force: true }));
|
||||
|
||||
describe('actual Node importer runtime', () => {
|
||||
|
||||
@@ -272,7 +272,8 @@ describe('owned terminal-agent publication lock recovery', () => {
|
||||
if (agent && String(to).endsWith('/terminal-agent-pid.lock') && !fs.existsSync(ready)) {
|
||||
const owner = JSON.parse(fs.readFileSync(to, 'utf8'));
|
||||
if (owner.kind !== 'agent-publication-v1' || owner.pid !== process.pid) throw new Error('Publication metadata was not atomic');
|
||||
fs.writeFileSync(ready, JSON.stringify({ pid: process.pid, owner }));
|
||||
fs.writeFileSync(ready + '.tmp', JSON.stringify({ pid: process.pid, owner }));
|
||||
fs.renameSync(ready + '.tmp', ready);
|
||||
Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 3000);
|
||||
throw new Error('Expected the original fixture to kill the publication holder');
|
||||
}
|
||||
|
||||
@@ -795,7 +795,9 @@ When a judge's abort signal cancels its request, `callJudge`
|
||||
(`test/helpers/llm-judge.ts`) prints one `llm-judge-aborted` JSON line to
|
||||
stderr: elapsed time, wrapper `attempts` and `retryWaitsMs` (429 waits),
|
||||
`httpRequests` and `httpStatuses` (so SDK retries show), `connectedMs`,
|
||||
`firstEventMs` and `lastEventMs`, the open `block` type, `thinkingChars` and
|
||||
`firstEventMs` and `lastEventMs`, `bytesReceived` and `lastByteMs` (every body
|
||||
byte, pings included, so a silent socket differs from a model thinking on a live
|
||||
connection), the open `block` type, `thinkingChars` and
|
||||
`textChars`, `messageId`, `requestId` and the `reason` (at most 200
|
||||
characters). It holds counts and ids only, never content, and changes no
|
||||
request, score or verdict.
|
||||
|
||||
@@ -12,6 +12,7 @@
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { createServer } from 'http';
|
||||
import { createServer as createNetServer } from 'net';
|
||||
import type { Server, IncomingMessage } from 'http';
|
||||
import { mkdtempSync, rmSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
@@ -75,6 +76,22 @@ function startStubStateServer(opts: { onUnauthorized?: (reply: () => void) => vo
|
||||
});
|
||||
}
|
||||
|
||||
const STUCK_HOP_MS = 10_000;
|
||||
|
||||
function startResettingRoute(): Promise<{ port: number; close: () => Promise<void> }> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const route = createNetServer((socket) => socket.destroy());
|
||||
route.once('error', reject);
|
||||
route.listen(0, '127.0.0.1', () => {
|
||||
const address = route.address();
|
||||
resolve({
|
||||
port: typeof address === 'object' && address ? address.port : 0,
|
||||
close: () => new Promise<void>((done, fail) => route.close((err) => err ? fail(err) : done())),
|
||||
});
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
async function fetchWith(method: string, url: string, init: { headers?: Record<string, string>; body?: string } = {}): Promise<{ status: number; bodyText: string }> {
|
||||
const res = await fetch(url, { method, headers: init.headers, body: init.body });
|
||||
return { status: res.status, bodyText: await res.text() };
|
||||
@@ -380,27 +397,20 @@ describe('daemon — loopback listener', () => {
|
||||
});
|
||||
|
||||
test('connection failure after redeploy reboots the tunnel once and keeps the replacement cached', async () => {
|
||||
const deadPort = await new Promise<number>((resolve, reject) => {
|
||||
const reservation = createServer();
|
||||
reservation.once('error', reject);
|
||||
reservation.listen(0, '127.0.0.1', () => {
|
||||
const address = reservation.address();
|
||||
const port = typeof address === 'object' && address ? address.port : 0;
|
||||
reservation.close((err) => err ? reject(err) : resolve(port));
|
||||
});
|
||||
});
|
||||
const staleRoute = await startResettingRoute();
|
||||
let bootstraps = 0;
|
||||
const d = await startDaemon({
|
||||
loopbackPort: 0,
|
||||
tailnetEnabled: false,
|
||||
pidfilePath: join(workDir, 'daemon-redeploy-refresh.pid'),
|
||||
proxyTimeoutMs: STUCK_HOP_MS,
|
||||
tunnelProvider: async () => {
|
||||
bootstraps++;
|
||||
if (bootstraps === 1) {
|
||||
return {
|
||||
udid: 'STUB-UDID',
|
||||
ipv6Addr: '127.0.0.1',
|
||||
port: deadPort,
|
||||
port: staleRoute.port,
|
||||
bootTokenRotated: 'old-deploy-token',
|
||||
};
|
||||
}
|
||||
@@ -420,40 +430,34 @@ describe('daemon — loopback listener', () => {
|
||||
try {
|
||||
const base = `http://127.0.0.1:${d.loopbackPort}`;
|
||||
const recovered = await fetchWith('GET', `${base}/screenshot`);
|
||||
expect(recovered.status).toBe(200);
|
||||
expect(recovered.status, recovered.bodyText).toBe(200);
|
||||
expect(JSON.parse(recovered.bodyText)).toEqual({ png_base64: 'abc=' });
|
||||
expect(bootstraps).toBe(2);
|
||||
|
||||
const healthyReuse = await fetchWith('GET', `${base}/screenshot`);
|
||||
expect(healthyReuse.status).toBe(200);
|
||||
expect(healthyReuse.status, healthyReuse.bodyText).toBe(200);
|
||||
expect(bootstraps).toBe(2);
|
||||
} finally {
|
||||
await d.close();
|
||||
await staleRoute.close();
|
||||
}
|
||||
});
|
||||
|
||||
test('connection failure refreshes but never replays an ambiguous tap', async () => {
|
||||
const deadPort = await new Promise<number>((resolve, reject) => {
|
||||
const reservation = createServer();
|
||||
reservation.once('error', reject);
|
||||
reservation.listen(0, '127.0.0.1', () => {
|
||||
const address = reservation.address();
|
||||
const port = typeof address === 'object' && address ? address.port : 0;
|
||||
reservation.close((err) => err ? reject(err) : resolve(port));
|
||||
});
|
||||
});
|
||||
const staleRoute = await startResettingRoute();
|
||||
let bootstraps = 0;
|
||||
const d = await startDaemon({
|
||||
loopbackPort: 0,
|
||||
tailnetEnabled: false,
|
||||
pidfilePath: join(workDir, 'daemon-mutation-no-replay.pid'),
|
||||
proxyTimeoutMs: STUCK_HOP_MS,
|
||||
tunnelProvider: async () => {
|
||||
bootstraps++;
|
||||
return bootstraps === 1
|
||||
? {
|
||||
udid: 'STUB-UDID',
|
||||
ipv6Addr: '127.0.0.1',
|
||||
port: deadPort,
|
||||
port: staleRoute.port,
|
||||
bootTokenRotated: 'old-deploy-token',
|
||||
}
|
||||
: {
|
||||
@@ -473,7 +477,7 @@ describe('daemon — loopback listener', () => {
|
||||
headers: { 'x-session-id': 'old-session', 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ x: 10, y: 20 }),
|
||||
});
|
||||
expect(ambiguous.status).toBe(503);
|
||||
expect(ambiguous.status, ambiguous.bodyText).toBe(503);
|
||||
expect(bootstraps).toBe(2);
|
||||
expect(stub.receivedRequests.filter((request) => request.path === '/tap')).toHaveLength(beforeTaps);
|
||||
|
||||
@@ -481,11 +485,12 @@ describe('daemon — loopback listener', () => {
|
||||
headers: { 'x-session-id': 'new-session', 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ x: 10, y: 20 }),
|
||||
});
|
||||
expect(explicitRetry.status).toBe(200);
|
||||
expect(explicitRetry.status, explicitRetry.bodyText).toBe(200);
|
||||
expect(stub.receivedRequests.filter((request) => request.path === '/tap')).toHaveLength(beforeTaps + 1);
|
||||
expect(bootstraps).toBe(2);
|
||||
} finally {
|
||||
await d.close();
|
||||
await staleRoute.close();
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gstack",
|
||||
"version": "1.92.32",
|
||||
"version": "1.92.33",
|
||||
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
|
||||
@@ -29,7 +29,7 @@
|
||||
"test/helpers/eval-budgets.ts": {"blob":"667d0e09b20fe5d7366f57b77b5fb9b04baa37ef","since":6},
|
||||
"test/helpers/eval-store.ts": {"blob":"a50ea2c28f508ad9f35ea9353dd2f502ee0dcfc2","since":6,"nonBehavioral":"credential rejections are classed infra instead of assertion: no case's pass or fail changes; only failed trials whose provider rejected the run's key move class, so no rerun spends on a rejected key"},
|
||||
"test/helpers/hermetic-env.ts": {"blob":"cc072895461dc52ea317d1beff72121342420d30","since":6},
|
||||
"test/helpers/llm-judge.ts": {"blob":"095f9c85dd0ae29da8ba10fed91510313a4442a4","since":6,"nonBehavioral":"callJudge emits one llm-judge-aborted diagnostic line (wrapper attempts, HTTP request and status counts seen through a pass-through fetch, 429 waits, connect and event timings, block kind, character counts, message and request ids, a capped reason; no content) when its caller's signal aborts it; observers swallow their own errors, and the request, retries, parsing, scores and the thrown abort reason are unchanged, so no case's pass or fail changes"},
|
||||
"test/helpers/llm-judge.ts": {"blob":"f14ca0db6504480698acd662351b6f0d6ef31a7b","since":6,"nonBehavioral":"llm-judge.ts adds a pass-through byte counter (bytesReceived, lastByteMs) to the aborted-judge diagnostic; the request, response bytes, deadline and scoring are unchanged"},
|
||||
"test/helpers/paid-test-set.ts": {"blob":"ff9bfe43c43ec2bdd11114f7b8ec7c5736116f75","since":6},
|
||||
"test/helpers/pty-screen.ts": {"blob":"4f77569f9ce11f9cee9f347216e468cd7c956820","since":6},
|
||||
"test/helpers/pty/auq.ts": {"blob":"16e5ea2bb5b9f28f44808a1a09024dbd3e6f28c4","since":6},
|
||||
|
||||
@@ -1105,9 +1105,10 @@ cmd_pack() {
|
||||
# index's blobs that HEAD's tree lacks (staged, intent-to-add, the stages
|
||||
# of an unmerged entry) are behind no ref, so they travel as a pack of
|
||||
# their own; without them the VM's write-tree (gstack-wtree) cannot read a
|
||||
# staged file.
|
||||
# staged file. cat, not cp: git renames index.lock over the index, which
|
||||
# cp reports as "replaced while being copied"; one open() reads one whole index.
|
||||
if [ "$(git -C "$src" config --bool core.splitIndex 2>/dev/null)" != true ]; then
|
||||
cp "$(cd "$src" && cd "$(dirname "$(git rev-parse --git-path index)")" && pwd)/index" "$tmp/.git/index"
|
||||
cat "$(cd "$src" && cd "$(dirname "$(git rev-parse --git-path index)")" && pwd)/index" > "$tmp/.git/index"
|
||||
# Read the copy, not the live index, so the pack matches it; a gitlink's
|
||||
# sha (mode 160000) is a submodule commit this object store lacks.
|
||||
GIT_INDEX_FILE="$tmp/.git/index" git -C "$src" ls-files -s | awk '$1 != "160000" {print $2}' | sort -u \
|
||||
|
||||
@@ -19,6 +19,7 @@ import captured_sed_context from './fixtures/coverage-audit-sed-context.json';
|
||||
import captured_cd_chain from './fixtures/coverage-audit-cd-chain.json';
|
||||
import captured_wc_anchor from './fixtures/coverage-audit-wc-anchor.json';
|
||||
import captured_separator from './fixtures/coverage-audit-separator-collision.json';
|
||||
import captured_two_file_grep from './fixtures/coverage-audit-two-file-grep.json';
|
||||
|
||||
const clone = <T>(v:T):T => structuredClone(v);
|
||||
const diagram = '```text\nsrc/billing.ts\n├── processPayment: happy path [TESTED]\n└── refundPayment [UNTESTED]\n```';
|
||||
@@ -1251,3 +1252,40 @@ describe('coverage audit native evidence: a separator that also appears in neigh
|
||||
expect(reads(dashed.command.replace('cat src/billing.ts', 'cat src/other.ts'), dashed.output)).toMatchObject({ sourceRead: false });
|
||||
});
|
||||
});
|
||||
|
||||
describe('coverage audit native evidence: an escaped grep over two literal files after the reads (CI 41149166998 slice 5)', () => {
|
||||
const capture = captured_two_file_grep;
|
||||
const both = { sourceRead: true, testsRead: true }, neither = { sourceRead: false, testsRead: false };
|
||||
const files = { cwd: capture.cwd, source: { path: capture.cwd + '/src/billing.ts', content: capture.files.source },
|
||||
tests: { path: capture.cwd + '/test/billing.test.ts', content: capture.files.tests } };
|
||||
const transcript = (command: string, output: string) => [
|
||||
{ type: 'system', subtype: 'init', session_id: capture.sessionId, cwd: capture.cwd },
|
||||
{ type: 'assistant', session_id: capture.sessionId, parent_tool_use_id: null, message: { role: 'assistant', content: [{ type: 'tool_use', id: capture.call.toolUseId, name: 'Bash', input: { command } }] } },
|
||||
{ type: 'user', session_id: capture.sessionId, parent_tool_use_id: null, message: { role: 'user', content: [{ type: 'tool_result', tool_use_id: capture.call.toolUseId, is_error: capture.call.isError, content: output }] } },
|
||||
];
|
||||
const reads = (command = capture.call.command, output = capture.call.output) => coverageAuditReadEvidence(transcript(command, output), files);
|
||||
const grep = 'grep -n -i -A3 "coverage diagram\\|ASCII" review/SKILL.md review/sections/review-army.md';
|
||||
|
||||
test('both complete reads before a two-file escaped grep display are credited and the recorded run passes', () => {
|
||||
expect(capture.provenance.recordedPassed).toBe(false);
|
||||
expect(capture.call.command).toContain('; cat -n src/billing.ts; echo ----; cat -n test/billing.test.ts; echo ----; ' + grep + ' | head -80');
|
||||
expect(reads()).toEqual(both);
|
||||
expect(coverageAuditVerdict({ exitReason: 'success', browseErrors: [], output: capture.finalOutput,
|
||||
transcript: transcript(capture.call.command, capture.call.output) } as any, files)).toEqual({ ...both, diagram: true, passed: true, failures: [] });
|
||||
});
|
||||
|
||||
test('an option, substitution or redirect among the grep operands, or a tampered read, still gets no credit', () => {
|
||||
for (const operands of ['review/SKILL.md -f/tmp/foreign', 'review/SKILL.md --include=*', '-r review/SKILL.md', 'review/SKILL.md "$(touch x)"', 'review/SKILL.md > out.txt']) {
|
||||
expect(reads(capture.call.command.replace('review/SKILL.md review/sections/review-army.md', operands)), operands).toEqual(neither);
|
||||
}
|
||||
const nonce = capture.files.source.match(/coverage-read-evidence: (\S+)/)![1]!;
|
||||
expect(reads(capture.call.command, capture.call.output.replace(nonce, nonce.slice(1) + '0'))).toMatchObject({ sourceRead: false });
|
||||
expect(reads(capture.call.command.replace('cat -n test/billing.test.ts', 'cat -n test/other.test.ts'))).toMatchObject({ testsRead: false });
|
||||
});
|
||||
|
||||
test('a quoted grep operand with a space stays one literal file operand', () => {
|
||||
expect(reads(capture.call.command.replace('review/SKILL.md review/sections/review-army.md', 'review/SKILL.md "docs/design notes.md"'))).toEqual(both);
|
||||
expect(reads(capture.call.command.replace('review/SKILL.md review/sections/review-army.md', '"docs/design notes.md" -f/tmp/foreign'))).toEqual(neither);
|
||||
});
|
||||
});
|
||||
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"provenance": {
|
||||
"source": "https://github.com/garrytan/gstack/actions/runs/41149166998 eval slice 5 (train gate, tested cab8ef12), review-coverage-audit (claude-opus-5-5); the second parent Bash call and its result, byte-exact",
|
||||
"recordedPassed": false,
|
||||
"recordedFailures": [
|
||||
"missing successful source-file read",
|
||||
"missing successful test-file read"
|
||||
],
|
||||
"note": "The first call printed 34 KB and the native result kept only a 2 KB preview, so it delivered neither file. The second call read both files in a ; chain that ended in a grep display with an escaped alternation over two literal in-cwd files; the escaped-grep grammar accepted one file operand only, so the whole chain got no credit."
|
||||
},
|
||||
"cwd": "/home/runner/.cache/gstack-paid-shard-O49Xex/tmp/skill-e2e-review-coverage-AWwx64",
|
||||
"sessionId": "2301d71e-8faf-4efd-8137-56549a57bf86",
|
||||
"call": {
|
||||
"toolUseId": "toolu_01NRrh5xY6cAXURo6ZJ6Ucoy",
|
||||
"command": "cd /home/runner/.cache/gstack-paid-shard-O49Xex/tmp/skill-e2e-review-coverage-AWwx64; cat review/specialists/testing.md; echo ----; cat -n src/billing.ts; echo ----; cat -n test/billing.test.ts; echo ----; grep -n -i -A3 \"coverage diagram\\|ASCII\" review/SKILL.md review/sections/review-army.md | head -80",
|
||||
"isError": false,
|
||||
"output": "# Testing Specialist Review Checklist\n\nScope: Always-on (every review)\nOutput: JSON objects, one finding per line. Schema:\n{\"severity\":\"CRITICAL|INFORMATIONAL\",\"confidence\":N,\"path\":\"file\",\"line\":N,\"category\":\"testing\",\"summary\":\"...\",\"fix\":\"...\",\"fingerprint\":\"path:line:testing\",\"specialist\":\"testing\"}\nOptional: line, fix, fingerprint, evidence, test_stub.\nIf no findings: output `NO FINDINGS` and nothing else.\n\nIf the caller explicitly asks for an ASCII coverage diagram, first read the\nnamed source and test files directly in a dedicated tool call. Keep that tool\ncall limited to those two file reads, using either native Read calls or a simple\nshell display such as `cat -n src/file && echo ---- && cat -n test/file`. Read\ndiffs, package files, configs, or other context in separate tool calls. Then\noutput the diagram before any JSON findings. Use one function root per public or\nchanged function, put the `[OK]` or `[GAP]` marker on the same branch row as the\ntested or missing path, and keep the legend inside the same diagram block:\n\n```text\nsrc/billing.ts\nprocessPayment(amount, currency)\n├── valid USD happy path returns success [OK]\n└── invalid amount / unsupported currency branches [GAP]\nrefundPayment(paymentId, reason)\n└── refund success and guard branches not imported or untested [GAP]\nLegend: [OK] tested [GAP] no test\n```\n\nDo not rely on a summary table, prose paragraph, or distant nested marker as the\nonly coverage evidence. Covered happy-path rows must name the successful, valid,\nor concrete tested input path; gap rows must stay under the function that owns\nthe missing path.\n\n---\n\n## Categories\n\n### Exploratory hypotheses\n\nThe parent runs one shared exploratory QA pass before Fix-First, including small diffs\nthat skip specialists. Supply high-risk changed contracts, adverse scenarios and native\ntest candidates to that pass; do not launch another explorer, edit product/tests or\ncommit. A test proposal uses `test_stub` and keeps the caller's approval requirements.\nCheck real request/queue/storage effects, retries, duplicates and interrupted recovery\nwhere applicable. A diagram or test count is not executed proof. For discoveries promoted\nto regressions, require failure for the reproduced bug before repair and green plus\noriginal/adjacent probes afterward; never accept buggy-output goldens or discarded valid\nred tests. Unit tests suit logic; real integration/E2E tests protect boundaries mocks hide.\n\n### Missing Negative-Path Tests\n- New code paths that handle errors, rejections, or invalid input with NO corresponding test\n- Guard clauses and early returns that are untested\n- Error branches in try/catch, rescue, or error boundaries with no failure-path test\n- Permission/auth checks that are asserted in code but never tested for the \"denied\" case\n\n### Missing Edge-Case Coverage\n- Boundary values: zero, negative, max-int, empty string, empty array, nil/null/undefined\n- Single-element collections (off-by-one on loops)\n- Unicode and special characters in user-facing inputs\n- Concurrent access patterns with no race-condition test\n\n### Negative Assertions\n- A new assertion that something is absent (`not.toContain`, `not in`, `assertNotIn`,\n `pytest.raises` on a call that used to succeed) records a behavior removal, not coverage.\n Cite the spec line, plan item or PR text that authorizes the removal; with none, report it\n as a product decision shipped as a test.\n- A rename or removal in this diff can empty a standing negative assertion: a test asserting\n the old token is absent now passes forever and guards nothing. For each identifier, flag,\n heading or string the diff renames or deletes, grep the test suite for negative assertions\n naming the old value, including inside tests that assert something else, and report each\n as a coverage gap whose fix names the new value.\n\n### Test Isolation Violations\n- Tests sharing mutable state (class variables, global singletons, DB records not cleaned up)\n- Order-dependent tests (pass in sequence, fail when randomized)\n- Tests that depend on system clock, timezone, or locale\n- Tests that make real network calls instead of using stubs/mocks\n\n### Flaky Test Patterns\n- Timing-dependent assertions (sleep, setTimeout, waitFor with tight timeouts)\n- Assertions on ordering of unordered results (hash keys, Set iteration, async resolution order)\n- Tests that depend on external services (APIs, databases) without fallback\n- Randomized test data without seed control\n\n### Security Enforcement Tests Missing\n- Auth/authz checks in controllers with no test for the \"unauthorized\" case\n- Rate limiting logic with no test proving it actually blocks\n- Input sanitization with no test for malicious input\n- CSRF/CORS configuration with no integration test\n\n### Coverage Gaps\n- New public methods/functions with zero test coverage\n- Changed Line truncated
|
||||
},
|
||||
"files": {
|
||||
"source": "\nexport function processPayment(amount: number, currency: string) {\n if (amount <= 0) throw new Error('Invalid amount');\n if (currency !== 'USD' && currency !== 'EUR') throw new Error('Unsupported currency');\n return { status: 'success', amount, currency };\n}\n\nexport function refundPayment(paymentId: string, reason: string) {\n if (!paymentId) throw new Error('Payment ID required');\n if (!reason) throw new Error('Reason required');\n return { status: 'refunded', paymentId, reason };\n}\n\n// coverage-read-evidence: 70e4e799-f2a2-455e-bd90-5104f81dd28f\n",
|
||||
"tests": "\nimport { describe, test, expect } from 'vitest';\nimport { processPayment } from '../src/billing';\n\ndescribe('processPayment', () => {\n test('processes valid payment', () => {\n const result = processPayment(100, 'USD');\n expect(result.status).toBe('success');\n });\n // GAP: no test for invalid amount\n // GAP: no test for unsupported currency\n // GAP: refundPayment not tested at all\n});\n\n// coverage-read-evidence: 5751c192-a564-47ec-8ef8-86757f97a26a\n"
|
||||
},
|
||||
"finalOutput": "I applied the testing specialist checklist on its own, with no other specialists and no fixes. I read the two files directly: `src/billing.ts` and `test/billing.test.ts`.\n\n```text\nsrc/billing.ts\nprocessPayment(amount, currency)\n├── valid USD happy path, processPayment(100, 'USD') returns status 'success' [OK]\n├── valid EUR happy path (second allowed currency, line 4) [GAP]\n├── returned amount/currency echo values (line 5, only status asserted) [GAP]\n├── amount <= 0 guard throws 'Invalid amount' (line 3, incl. 0 and negative boundary) [GAP]\n└── unsupported currency guard throws 'Unsupported currency' (line 4) [GAP]\nrefundPayment(paymentId, reason)\n├── refund success path returns status 'refunded' (line 11) [GAP]\n├── empty paymentId guard throws 'Payment ID required' (line 9) [GAP]\n└── empty reason guard throws 'Reason required' (line 10) [GAP]\nLegend: [OK] tested [GAP] no test\n```\n\n**Coverage summary**\n- `processPayment`: 1 of 5 paths covered. Only the valid USD call is tested, and it checks only `status`.\n- `refundPayment`: 0 of 3 paths covered. The test file doesn't import it.\n- Overall: 1 of 8 paths covered (about 12%). One function is partly tested and one has no tests. None of the three guard clauses that throw errors has a test.\n\n**Findings**\n{\"severity\":\"CRITICAL\",\"confidence\":10,\"path\":\"src/billing.ts\",\"line\":8,\"category\":\"testing\",\"summary\":\"refundPayment has zero test coverage: it is not imported in test/billing.test.ts, so its success path and both guard clauses (empty paymentId, empty reason) are untested\",\"fix\":\"Import refundPayment and add tests: refundPayment('pay_1','dup') returns {status:'refunded',paymentId:'pay_1',reason:'dup'}; refundPayment('', 'x') throws 'Payment ID required'; refundPayment('pay_1','') throws 'Reason required'\",\"fingerprint\":\"src/billing.ts:8:testing\",\"specialist\":\"testing\"}\n{\"severity\":\"CRITICAL\",\"confidence\":10,\"path\":\"src/billing.ts\",\"line\":3,\"category\":\"testing\",\"summary\":\"Invalid-amount guard (amount <= 0) has no negative-"
|
||||
}
|
||||
@@ -0,0 +1,161 @@
|
||||
{
|
||||
"provenance": {
|
||||
"kind": "public native AskUserQuestion calls and answers only (observation.json fingerprints[].nativeCall, byte-exact)",
|
||||
"runId": 41178131219,
|
||||
"artifact": "paid-slice-6-a1",
|
||||
"prNumber": 3662,
|
||||
"headSha": "e18cccbcc482d2a8212cc41993fee052a4701e26",
|
||||
"testedSha": "a77fae2837695ffc3e7937bb8f3e9dc5eb337fc4",
|
||||
"recordedOutcome": "ceiling_reached",
|
||||
"recordedReviewCount": 1,
|
||||
"recordedStep0Count": 3,
|
||||
"recordedPassed": false,
|
||||
"note": "Step 0D titled only 'Review focus', with the 4/10 design-completeness rating in the ELI10 body, and lettered its options: 'A) All 7 passes (recommended)'. The runner reached the review ceiling on the issue-labeled D4 finding, but the focus predicate accepted an all-seven option only when 'All 7' led the label, so the letter prefix hid Step 0D and the gate failed."
|
||||
},
|
||||
"calls": [
|
||||
{
|
||||
"preReview": true,
|
||||
"observedAtMs": 25355,
|
||||
"call": {
|
||||
"sessionId": "4b0b022d-3033-4d11-b220-4c86f38f9712",
|
||||
"toolUseId": "toolu_014Adzj8rsKq2ftZ6cFLdA7h",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D1 — What should I review? The project instructions name two targets: PLAN.md as the current plan, and a request to review the UI plan in review-input.md. They may hold the same \"User Dashboard Page\" plan, but I can't check before you choose. If I review the wrong file, the design fixes go into the wrong plan. Recommendation: B, review-input.md, because the request explicitly says to read it before choosing scope. <gstack-qid:plan-design-review-scope-target>",
|
||||
"header": "Scope",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "B) review-input.md (recommended)",
|
||||
"description": "✅ Matches the explicit request to review the supplied UI plan in review-input.md. ✅ Audit, Step 0 and mockups all anchor on that file. ❌ If PLAN.md has diverged, any edits made only there won't be reviewed."
|
||||
},
|
||||
{
|
||||
"label": "B) PLAN.md",
|
||||
"description": "✅ Matches the line in the project instructions that names PLAN.md as the current plan. ✅ Same full review: audit, Step 0, mockups and 7 passes. ❌ Ignores review-input.md if that file has newer content."
|
||||
},
|
||||
{
|
||||
"label": "A) Current branch diff",
|
||||
"description": "✅ Reviews the work in progress on main against the base branch. ✅ Good fit if UI code has already landed. ❌ The tree is clean with one seed commit, so there's probably nothing to review."
|
||||
},
|
||||
{
|
||||
"label": "C) A specific page/file/path",
|
||||
"description": "✅ Lets you point at one specific file or screen to review. ✅ Narrower and faster than a whole-plan review. ❌ You need to say which path, and the rest of the dashboard plan goes unreviewed."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D1 — What should I review? The project instructions name two targets: PLAN.md as the current plan, and a request to review the UI plan in review-input.md. They may hold the same \"User Dashboard Page\" plan, but I can't check before you choose. If I review the wrong file, the design fixes go into the wrong plan. Recommendation: B, review-input.md, because the request explicitly says to read it before choosing scope. <gstack-qid:plan-design-review-scope-target>": "B) review-input.md (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-10-31T06:52:50.945Z"
|
||||
}
|
||||
},
|
||||
{
|
||||
"preReview": true,
|
||||
"observedAtMs": 76985,
|
||||
"call": {
|
||||
"sessionId": "4b0b022d-3033-4d11-b220-4c86f38f9712",
|
||||
"toolUseId": "toolu_01FkeNMEVef5iKQZApVURMDs",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D2 — Review focus. Project/branch/task: plan-design-review of the User Dashboard plan in review-input.md on main.\nELI10: I rated this plan 4/10 on design completeness. The biggest gaps: the plan never says which panel matters most, how the layout changes per screen size, or what the modal, toasts and partial-failure states look like and how they behave. Next I'll generate visual mockups, then review all 7 design dimensions, unless you want me to narrow the focus.\nStakes if we pick wrong: if I narrow the focus, gaps in the skipped areas (like accessibility or responsive layout) go to implementation undecided.\nRecommendation: A, all 7 passes, because this is a brand-new landing page with no design system, so every dimension has open gaps.\nCompleteness: A=10/10, B=6/10, C=5/10\nPros / cons:\nA) All 7 passes (recommended)\n ✅ Covers hierarchy, states, journey, AI-slop risk, design system, responsive and accessibility, and unresolved decisions\n ✅ With no DESIGN.md, this pins down tokens and patterns before any code is written\n ❌ More decisions to work through, roughly one question per gap found\nB) Hierarchy + states + responsive only\n ✅ Concentrates on what users notice first on the post-login landing page\n ✅ Fewer questions; gets to an improved plan faster\n ❌ Leaves modal/toast accessibility and design tokens unspecified, so the implementer guesses\nC) Accessibility + interaction states only\n ✅ Locks down the keyboard, screen-reader, modal and toast contracts the plan only gestures at\n ✅ Matches the plan's existing keyboard-navigation test intent\n ❌ Skips layout hierarchy, so the page can ship as a generic three-card grid\nNet: full coverage of a new landing page versus fewer decisions now.",
|
||||
"header": "Focus",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "A) All 7 passes (recommended)",
|
||||
"description": "✅ Covers hierarchy, states, journey, AI-slop risk, design system, responsive/accessibility, and unresolved decisions. ✅ Pins down tokens before code with no DESIGN.md. ❌ More decisions, roughly one question per gap found."
|
||||
},
|
||||
{
|
||||
"label": "B) Hierarchy+states+responsive",
|
||||
"description": "✅ Concentrates on what users notice first on the post-login landing page. ✅ Fewer questions, faster improved plan. ❌ Modal/toast accessibility and tokens stay unspecified, so the implementer guesses."
|
||||
},
|
||||
{
|
||||
"label": "C) Accessibility+interaction only",
|
||||
"description": "✅ Locks down the keyboard, screen-reader, modal and toast contracts the plan only gestures at. ✅ Matches the plan's existing keyboard test intent. ❌ Skips hierarchy, so the page can ship as a generic three-card grid."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D2 — Review focus. Project/branch/task: plan-design-review of the User Dashboard plan in review-input.md on main.\nELI10: I rated this plan 4/10 on design completeness. The biggest gaps: the plan never says which panel matters most, how the layout changes per screen size, or what the modal, toasts and partial-failure states look like and how they behave. Next I'll generate visual mockups, then review all 7 design dimensions, unless you want me to narrow the focus.\nStakes if we pick wrong: if I narrow the focus, gaps in the skipped areas (like accessibility or responsive layout) go to implementation undecided.\nRecommendation: A, all 7 passes, because this is a brand-new landing page with no design system, so every dimension has open gaps.\nCompleteness: A=10/10, B=6/10, C=5/10\nPros / cons:\nA) All 7 passes (recommended)\n ✅ Covers hierarchy, states, journey, AI-slop risk, design system, responsive and accessibility, and unresolved decisions\n ✅ With no DESIGN.md, this pins down tokens and patterns before any code is written\n ❌ More decisions to work through, roughly one question per gap found\nB) Hierarchy + states + responsive only\n ✅ Concentrates on what users notice first on the post-login landing page\n ✅ Fewer questions; gets to an improved plan faster\n ❌ Leaves modal/toast accessibility and design tokens unspecified, so the implementer guesses\nC) Accessibility + interaction states only\n ✅ Locks down the keyboard, screen-reader, modal and toast contracts the plan only gestures at\n ✅ Matches the plan's existing keyboard-navigation test intent\n ❌ Skips layout hierarchy, so the page can ship as a generic three-card grid\nNet: full coverage of a new landing page versus fewer decisions now.": "A) All 7 passes (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-10-31T06:53:41.125Z"
|
||||
}
|
||||
},
|
||||
{
|
||||
"preReview": true,
|
||||
"observedAtMs": 107434,
|
||||
"call": {
|
||||
"sessionId": "4b0b022d-3033-4d11-b220-4c86f38f9712",
|
||||
"toolUseId": "toolu_01CaKQSWyyBPxTVGn1UfseQM",
|
||||
"questions": [
|
||||
{
|
||||
"question": "Want outside design voices before the detailed review? D3. Project/branch/task: User Dashboard plan review on main. ELI10: two outside reviewers can critique the plan before my 7 passes. Codex checks it against hard design rules and a litmus scorecard, and a separate Claude reviewer does a fresh completeness read. Stakes if we pick wrong: without them, every finding comes from one reviewer; with them, the review takes a few minutes longer. Recommendation: B, proceed without, because no mockup was generated and the 7 passes cover the same ground. Note: options differ in kind, not coverage, so there's no completeness score. Net: a cross-model check versus fewer steps.",
|
||||
"header": "Voices",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Yes, run outside design voices",
|
||||
"description": "✅ Codex hard-rule + litmus scorecard plus an independent Claude completeness read. ✅ Disagreements surface with both perspectives in the relevant pass. ❌ Adds a few minutes; Codex may be missing here, in which case only the Claude voice runs and it's tagged single-model."
|
||||
},
|
||||
{
|
||||
"label": "No, proceed without (recommended)",
|
||||
"description": "✅ Straight into the 7 passes with the approved mockup as anchor. ✅ Fewest steps to a fixed plan. ❌ Findings rest on one reviewer; no cross-model check on slop risk or hierarchy."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"Want outside design voices before the detailed review? D3. Project/branch/task: User Dashboard plan review on main. ELI10: two outside reviewers can critique the plan before my 7 passes. Codex checks it against hard design rules and a litmus scorecard, and a separate Claude reviewer does a fresh completeness read. Stakes if we pick wrong: without them, every finding comes from one reviewer; with them, the review takes a few minutes longer. Recommendation: B, proceed without, because no mockup was generated and the 7 passes cover the same ground. Note: options differ in kind, not coverage, so there's no completeness score. Net: a cross-model check versus fewer steps.": "No, proceed without (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-10-31T06:54:12.032Z"
|
||||
}
|
||||
},
|
||||
{
|
||||
"preReview": false,
|
||||
"observedAtMs": 176665,
|
||||
"call": {
|
||||
"sessionId": "4b0b022d-3033-4d11-b220-4c86f38f9712",
|
||||
"toolUseId": "toolu_01N9Au3JQo4gsvMyrb2bbzck",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D4 — Issue 1: panel hierarchy on /dashboard. Project/branch/task: User Dashboard plan review on main.\nELI10: Users land here right after login and scan the page. The plan has three panels but doesn't say which one they see first. Without that, the panels get equal weight and nothing tells the user what to do next. This violates \"every screen has a hierarchy\" and Krug's billboard rule.\nStakes if we pick wrong: users hunt for their next step on every login, and unread notifications compete with ambient activity.\nRecommendation: 1A, because quick actions are the most common next step and unread notifications are the only actionable content, so they come first and activity is supporting context.\nNote: options differ in kind, not coverage — no completeness score.\nPros / cons:\n1A) Actions in header, Notifications primary, Activity secondary (recommended)\n ✅ Page header holds the greeting plus quick actions, so the next step is in the first viewport\n ✅ Notifications get the wide column (8/12 at lg), and activity sits in a narrower 4/12 rail\n ❌ Users who mainly come back for the activity history scroll or look to the side for it\n1B) Actions in header, Activity primary, Notifications secondary\n ✅ Activity gets the wide column: suits teams that use the dashboard as a history log\n ✅ Notifications become a compact rail with an unread badge\n ❌ The only action on the page (mark all as read) sits in the secondary rail\n1C) Three equal panels\n ✅ Simplest to build, with no ranking decision needed\n ✅ Every panel looks equally important, matching the plan text as written\n ❌ Exactly the generic three-card mosaic; no single obvious next step\nNet: an opinionated, scannable landing page versus neutral equal weighting.",
|
||||
"header": "Hierarchy",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "1A) Notifs primary (recommended)",
|
||||
"description": "✅ Header holds greeting plus quick actions, so the next step is in the first viewport. ✅ Notifications get 8/12 at lg, activity a 4/12 rail. ❌ Users who come back mainly for activity look to the side or scroll."
|
||||
},
|
||||
{
|
||||
"label": "1B) Activity primary",
|
||||
"description": "✅ Activity gets the wide column, suiting teams that use the dashboard as a history log. ✅ Notifications become a compact rail with an unread badge. ❌ The page's only action (mark all read) sits in the secondary rail."
|
||||
},
|
||||
{
|
||||
"label": "1C) Three equal panels",
|
||||
"description": "✅ Simplest to build, with no ranking decision needed. ✅ Matches the plan text exactly as currently written. ❌ The generic three-card mosaic, with no single obvious next step for the user."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D4 — Issue 1: panel hierarchy on /dashboard. Project/branch/task: User Dashboard plan review on main.\nELI10: Users land here right after login and scan the page. The plan has three panels but doesn't say which one they see first. Without that, the panels get equal weight and nothing tells the user what to do next. This violates \"every screen has a hierarchy\" and Krug's billboard rule.\nStakes if we pick wrong: users hunt for their next step on every login, and unread notifications compete with ambient activity.\nRecommendation: 1A, because quick actions are the most common next step and unread notifications are the only actionable content, so they come first and activity is supporting context.\nNote: options differ in kind, not coverage — no completeness score.\nPros / cons:\n1A) Actions in header, Notifications primary, Activity secondary (recommended)\n ✅ Page header holds the greeting plus quick actions, so the next step is in the first viewport\n ✅ Notifications get the wide column (8/12 at lg), and activity sits in a narrower 4/12 rail\n ❌ Users who mainly come back for the activity history scroll or look to the side for it\n1B) Actions in header, Activity primary, Notifications secondary\n ✅ Activity gets the wide column: suits teams that use the dashboard as a history log\n ✅ Notifications become a compact rail with an unread badge\n ❌ The only action on the page (mark all as read) sits in the secondary rail\n1C) Three equal panels\n ✅ Simplest to build, with no ranking decision needed\n ✅ Every panel looks equally important, matching the plan text as written\n ❌ Exactly the generic three-card mosaic; no single obvious next step\nNet: an opinionated, scannable landing page versus neutral equal weighting.": "1A) Notifs primary (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-10-31T06:55:20.477Z"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,153 @@
|
||||
{
|
||||
"provenance": {
|
||||
"kind": "public native AskUserQuestion calls and answers only (observation.json fingerprints[].nativeCall, byte-exact)",
|
||||
"runId": 41186376825,
|
||||
"artifact": "paid-slice-6-a1",
|
||||
"prNumber": 3665,
|
||||
"headSha": "9e6a4693f6dbfc4c5254057732fe428daa35810f",
|
||||
"testedSha": "047dad78ba757ebd6f29267f68a4695a7b002cbd",
|
||||
"recordedOutcome": "ceiling_reached",
|
||||
"recordedReviewCount": 1,
|
||||
"recordedStep0Count": 3,
|
||||
"recordedPassed": false,
|
||||
"note": "Step 0D titled only 'Review focus', with the 4/10 design-completeness rating in the ELI10 body, and lettered its options: 'A) All 7 dimensions (recommended)'. The runner reached the review ceiling on the issue-labeled D4 finding, but the focus predicate accepted an all-seven option only when 'All 7' led the label, so the letter prefix hid Step 0D and the gate failed."
|
||||
},
|
||||
"calls": [
|
||||
{
|
||||
"preReview": true,
|
||||
"observedAtMs": 25664,
|
||||
"call": {
|
||||
"sessionId": "cf524a80-d9e6-49b2-bdd7-11c9a579366e",
|
||||
"toolUseId": "toolu_01SPReG7GjZpJDfP8tF9Hv4h",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D1 — What should I review? Plan mode is on, but there's no host plan file yet. Your request names review-input.md, while CLAUDE.md says PLAN.md is the current plan. Both look like they hold the same User Dashboard plan, but I haven't opened either file yet and won't until you pick one. Why it matters: every rating, mockup and plan edit gets anchored to the file you choose. Recommendation: B, review-input.md, because it's the file your request names. Completeness: all three options lead to a full 7-pass review of whichever target you pick.",
|
||||
"header": "Scope",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "B) review-input.md (recommended)",
|
||||
"description": "✅ It's the file your request names, so it matches what you asked for. ✅ The dashboard UI plan with its implementation contracts gets the full Step 0, mockups and 7 passes. ❌ If PLAN.md has drifted from it, those differences won't be reviewed."
|
||||
},
|
||||
{
|
||||
"label": "B) PLAN.md",
|
||||
"description": "✅ It's the file CLAUDE.md names as the current plan for this review fixture. ✅ It gets the same full review: audit, mockups and 7 passes. ❌ If the file you meant was review-input.md, the review targets a different document."
|
||||
},
|
||||
{
|
||||
"label": "A) Current branch diff",
|
||||
"description": "✅ Reviews the work in progress on main against its base. ❌ The repo is clean with one seed commit, so there's probably no UI diff to review. ❌ The dashboard plan document itself wouldn't be reviewed."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D1 — What should I review? Plan mode is on, but there's no host plan file yet. Your request names review-input.md, while CLAUDE.md says PLAN.md is the current plan. Both look like they hold the same User Dashboard plan, but I haven't opened either file yet and won't until you pick one. Why it matters: every rating, mockup and plan edit gets anchored to the file you choose. Recommendation: B, review-input.md, because it's the file your request names. Completeness: all three options lead to a full 7-pass review of whichever target you pick.": "B) review-input.md (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-10-31T08:56:07.469Z"
|
||||
}
|
||||
},
|
||||
{
|
||||
"preReview": true,
|
||||
"observedAtMs": 83634,
|
||||
"call": {
|
||||
"sessionId": "cf524a80-d9e6-49b2-bdd7-11c9a579366e",
|
||||
"toolUseId": "toolu_01Q1eo6eZkH5femF9ZSvJL7Y",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D2 — Review focus\nProject/branch/task: /plan-design-review of review-input.md (User Dashboard) on main.\nELI10: I rated this plan 4/10 on design completeness. The biggest gaps: no defined order of attention after login, panel states with no copy or actions, and no visual tokens or per-viewport layout. Next I'll generate mockups, then go through all 7 design dimensions. You can narrow that to a few areas.\nStakes if we pick wrong: narrowing can leave modal/toast accessibility or mobile layout unspecified, and implementers will guess.\nRecommendation: A because the gaps span every dimension and the review is cheap compared with rework after it ships.\nCompleteness: A=10/10, B=6/10\nPros / cons:\nA) All 7 dimensions (recommended)\n ✅ Covers hierarchy, states, journey, AI-slop risk, design system, responsive/a11y and open decisions\n ✅ Every gap gets its own decision and a fix written into the plan\n ❌ More questions: roughly one per real gap across all seven passes\nB) Focus on specific areas\n ✅ Faster, with fewer decisions centered on what you care about most\n ✅ Lets you skip dimensions you plan to handle elsewhere, like a later design system\n ❌ Gaps in the skipped dimensions stay unresolved in the plan\nNet: full coverage now versus fewer questions.",
|
||||
"header": "Focus",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "A) All 7 dimensions (recommended)",
|
||||
"description": "✅ Full coverage of hierarchy, states, journey, slop risk, system, responsive/a11y, decisions. ✅ Each gap fixed in the plan. ❌ More decision questions to answer."
|
||||
},
|
||||
{
|
||||
"label": "B) Focus on specific areas",
|
||||
"description": "✅ Faster, centered on what you care about. ✅ Skips dimensions handled elsewhere. ❌ Skipped dimensions stay unresolved in the plan. (Name the areas in Other.)"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D2 — Review focus\nProject/branch/task: /plan-design-review of review-input.md (User Dashboard) on main.\nELI10: I rated this plan 4/10 on design completeness. The biggest gaps: no defined order of attention after login, panel states with no copy or actions, and no visual tokens or per-viewport layout. Next I'll generate mockups, then go through all 7 design dimensions. You can narrow that to a few areas.\nStakes if we pick wrong: narrowing can leave modal/toast accessibility or mobile layout unspecified, and implementers will guess.\nRecommendation: A because the gaps span every dimension and the review is cheap compared with rework after it ships.\nCompleteness: A=10/10, B=6/10\nPros / cons:\nA) All 7 dimensions (recommended)\n ✅ Covers hierarchy, states, journey, AI-slop risk, design system, responsive/a11y and open decisions\n ✅ Every gap gets its own decision and a fix written into the plan\n ❌ More questions: roughly one per real gap across all seven passes\nB) Focus on specific areas\n ✅ Faster, with fewer decisions centered on what you care about most\n ✅ Lets you skip dimensions you plan to handle elsewhere, like a later design system\n ❌ Gaps in the skipped dimensions stay unresolved in the plan\nNet: full coverage now versus fewer questions.": "A) All 7 dimensions (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-10-31T08:57:05.424Z"
|
||||
}
|
||||
},
|
||||
{
|
||||
"preReview": true,
|
||||
"observedAtMs": 114012,
|
||||
"call": {
|
||||
"sessionId": "cf524a80-d9e6-49b2-bdd7-11c9a579366e",
|
||||
"toolUseId": "toolu_013N2EdnqoiSH6Z5ip7ZHYTE",
|
||||
"questions": [
|
||||
{
|
||||
"question": "Want outside design voices before the detailed review?\nD3 — Outside design voices\nProject/branch/task: /plan-design-review of review-input.md on main.\nELI10: This optional step brings in two more reviewers before my 7 passes. Codex checks the plan against hard design rules and a litmus scorecard. A separate Claude agent does an independent read for missing pieces. It's a second opinion on hierarchy and on generic-looking \"AI slop\" layouts.\nStakes if we pick wrong: skipping it means every finding comes from one reviewer; running it costs a few minutes.\nRecommendation: B because the gaps here are clear and concrete, and the 7 passes cover them directly.\nNote: options differ in kind, not coverage — no completeness score.\nPros / cons: see options.\nNet: a cross-model check versus the fastest path to a fixed plan.",
|
||||
"header": "Voices",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Yes, run outside design voices",
|
||||
"description": "✅ Codex hard-rule + litmus scorecard plus an independent Claude completeness read. ✅ Disagreements surface with both perspectives in the relevant pass. ❌ Adds a few minutes; Codex may be missing here, in which case only the Claude voice runs and it's tagged single-model."
|
||||
},
|
||||
{
|
||||
"label": "No, proceed without (recommended)",
|
||||
"description": "✅ Straight into the 7 passes with the approved mockup as anchor. ✅ Fewest steps to a fixed plan. ❌ Findings rest on one reviewer; no cross-model check on slop risk or hierarchy."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"Want outside design voices before the detailed review?\nD3 — Outside design voices\nProject/branch/task: /plan-design-review of review-input.md on main.\nELI10: This optional step brings in two more reviewers before my 7 passes. Codex checks the plan against hard design rules and a litmus scorecard. A separate Claude agent does an independent read for missing pieces. It's a second opinion on hierarchy and on generic-looking \"AI slop\" layouts.\nStakes if we pick wrong: skipping it means every finding comes from one reviewer; running it costs a few minutes.\nRecommendation: B because the gaps here are clear and concrete, and the 7 passes cover them directly.\nNote: options differ in kind, not coverage — no completeness score.\nPros / cons: see options.\nNet: a cross-model check versus the fastest path to a fixed plan.": "No, proceed without (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-10-31T08:57:34.355Z"
|
||||
}
|
||||
},
|
||||
{
|
||||
"preReview": false,
|
||||
"observedAtMs": 146460,
|
||||
"call": {
|
||||
"sessionId": "cf524a80-d9e6-49b2-bdd7-11c9a579366e",
|
||||
"toolUseId": "toolu_01HzjkF4h3fnqfV15mQeCBAS",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D4 — Issue 1.1: Order of attention on /dashboard\nProject/branch/task: design review of review-input.md (User Dashboard) on main.\nELI10: The plan never says what the user sees first after login. Without that, each panel gets equal weight, nothing stands out, and people scan without finding their next step. We need to pick the anchor and the order.\nStakes if we pick wrong: users land on a wall of equal boxes, miss unread notifications, and click around to get oriented.\nRecommendation: 1A because unread notifications are the only content that asks for action, and Open workspace is the main reason to leave the page (principle: every screen has a hierarchy).\nNote: options differ in kind, not coverage — no completeness score.\nPros / cons:\n1A) Notifications anchor + quick actions in header (recommended)\n ✅ Unread items get the widest column and the first look, so attention goes where action is needed\n ✅ Open workspace sits in the page header and is visible right away at every width\n ❌ Activity becomes secondary; users who mostly check activity scroll on mobile\n1B) Activity feed anchor, notifications in a side rail\n ✅ Familiar GitHub-style layout with the feed as the main column\n ✅ Suits users who come to see what changed rather than to clear alerts\n ❌ Unread notifications get pushed into a narrow rail and are easier to ignore\n1C) Single column at every width (actions, notifications, activity)\n ✅ Simplest layout to build and test, with one reading order everywhere\n ✅ No column-ratio decisions or breakpoint-specific layouts to maintain\n ❌ Wastes desktop width and pushes activity far below the fold at lg\nNet: pick what the user sees first: things to act on (A), recent changes (B), or a plain list (C).",
|
||||
"header": "1.1 Hierarchy",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "1A) Notifications anchor (recommended)",
|
||||
"description": "✅ Unread items get the first look and widest column. ✅ Open workspace in header, visible right away. ❌ Activity is secondary; more scroll on mobile."
|
||||
},
|
||||
{
|
||||
"label": "1B) Activity anchor + side rail",
|
||||
"description": "✅ Familiar feed-first layout. ✅ Suits users checking recent changes. ❌ Unread notifications pushed into a narrow rail, easier to ignore."
|
||||
},
|
||||
{
|
||||
"label": "1C) Single column everywhere",
|
||||
"description": "✅ Simplest to build and test, one reading order. ✅ No breakpoint layouts. ❌ Wastes desktop width; activity far below fold."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D4 — Issue 1.1: Order of attention on /dashboard\nProject/branch/task: design review of review-input.md (User Dashboard) on main.\nELI10: The plan never says what the user sees first after login. Without that, each panel gets equal weight, nothing stands out, and people scan without finding their next step. We need to pick the anchor and the order.\nStakes if we pick wrong: users land on a wall of equal boxes, miss unread notifications, and click around to get oriented.\nRecommendation: 1A because unread notifications are the only content that asks for action, and Open workspace is the main reason to leave the page (principle: every screen has a hierarchy).\nNote: options differ in kind, not coverage — no completeness score.\nPros / cons:\n1A) Notifications anchor + quick actions in header (recommended)\n ✅ Unread items get the widest column and the first look, so attention goes where action is needed\n ✅ Open workspace sits in the page header and is visible right away at every width\n ❌ Activity becomes secondary; users who mostly check activity scroll on mobile\n1B) Activity feed anchor, notifications in a side rail\n ✅ Familiar GitHub-style layout with the feed as the main column\n ✅ Suits users who come to see what changed rather than to clear alerts\n ❌ Unread notifications get pushed into a narrow rail and are easier to ignore\n1C) Single column at every width (actions, notifications, activity)\n ✅ Simplest layout to build and test, with one reading order everywhere\n ✅ No column-ratio decisions or breakpoint-specific layouts to maintain\n ❌ Wastes desktop width and pushes activity far below the fold at lg\nNet: pick what the user sees first: things to act on (A), recent changes (B), or a plain list (C).": "1A) Notifications anchor (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-10-31T08:58:08.249Z"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -193,12 +193,13 @@ describe('gate-ahead, then stamp and verify from a separate clone', () => {
|
||||
}, 240_000);
|
||||
});
|
||||
|
||||
function runAsync(repo: FixtureRepo, args: string[]): Promise<{ status: number | null; out: string }> {
|
||||
function runAsync(repo: FixtureRepo, args: string[]): Promise<{ status: number | null; out: string; err: string }> {
|
||||
return new Promise(resolve => {
|
||||
const child = spawn(BIN('gstack-ship-queue'), [...args, '--base', 'main'], { cwd: repo.work, env: { ...process.env, ...repo.env }, timeout: 120_000 });
|
||||
let out = '';
|
||||
let out = '', err = '';
|
||||
child.stdout.on('data', d => { out += d; });
|
||||
child.on('close', status => resolve({ status, out }));
|
||||
child.stderr.on('data', d => { err += d; });
|
||||
child.on('close', status => resolve({ status, out, err }));
|
||||
});
|
||||
}
|
||||
|
||||
@@ -207,7 +208,7 @@ describe('gate-ahead attempts', () => {
|
||||
const { repo } = twoLanes();
|
||||
const args = ['gate-ahead', '--after', '1', '--pr', '2', '--predecessor-ref', 'pr-a', '--allow-repo-commands'];
|
||||
const [a, b] = await Promise.all([runAsync(repo, args), runAsync(repo, args)]);
|
||||
expect([a.status, b.status]).toEqual([EXIT.ok, EXIT.ok]);
|
||||
expect([a.status, b.status], `${a.err}\n${b.err}`).toEqual([EXIT.ok, EXIT.ok]);
|
||||
const ids = [a, b].map(r => field(line(r.out, 'GATE_AHEAD:'), 'attempt'));
|
||||
expect(new Set(ids).size).toBe(2);
|
||||
const trees = [a, b].map(r => line(r.out, 'worktree: ').slice(10));
|
||||
|
||||
@@ -80,7 +80,9 @@ function stagePieces(stage: string[], scope: ShellScope, alone: boolean): Piece[
|
||||
if (!name || (discarded && name === 'echo')) return undefined;
|
||||
// Backslashes are data only in these closed grep/awk display patterns.
|
||||
const grepRange = /^grep\s+-n(?:\s+-i)?(?:\s+-B\d{1,4})?(?:\s+-A\d{1,4})?\s+(?:"(?:[^"\\$`]|\\[|.])*"|'(?:[^'\\$`]|\\[|.])*')\s+(.+)$/.exec(text);
|
||||
const displayGrep = Boolean(grepRange && literal(grepRange[1]!) && !literal(grepRange[1]!)!.startsWith('-'));
|
||||
// One or more literal file operands (the tokens after the quoted pattern), none of them an option.
|
||||
const grepOperands = grepRange ? words.slice(words.findIndex((word, index) => index > 0 && /^["']/.test(word)) + 1) : [];
|
||||
const displayGrep = grepOperands.length > 0 && grepOperands.every(operand => literal(operand) !== undefined && !literal(operand)!.startsWith('-'));
|
||||
const awkRange = /^awk\s+'\/(?:[^/\\]|\\[./|])*\/,\/(?:[^/\\]|\\[./|])*\/'\s+(.+)$/.exec(text)
|
||||
?? /^awk\s+'\/(?:[^/\\]|\\[./|])*\/\{f=1\} f&&\/(?:[^/\\]|\\[./|])*\/\{exit\} f'\s+(.+)$/.exec(text);
|
||||
const displayAwk = Boolean(awkRange && literal(awkRange[1]!) && !literal(awkRange[1]!)!.startsWith('-'));
|
||||
|
||||
@@ -68,6 +68,7 @@ export const FREE_FIXTURES: Readonly<Record<string, readonly string[]>> = {
|
||||
'test/fixtures/coverage-audit-parallel-column.json': ['test/coverage-audit-evidence.test.ts'],
|
||||
'test/fixtures/coverage-audit-separator-collision.json': ['test/coverage-audit-evidence.test.ts'],
|
||||
'test/fixtures/coverage-audit-summary.json': ['test/coverage-audit-evidence.test.ts'],
|
||||
'test/fixtures/coverage-audit-two-file-grep.json': ['test/coverage-audit-evidence.test.ts'],
|
||||
'test/fixtures/coverage-audit-wc-anchor.json': ['test/coverage-audit-evidence.test.ts'],
|
||||
'test/fixtures/credential-rejection-39696775546.json': ['test/credential-rejection.test.ts'],
|
||||
'test/fixtures/cso-eval/**': ['scripts/cso-eval.ts', 'test/cso-eval.test.ts', 'test/cso-stack-cold-integration.test.ts', 'test/helpers/cso-eval-oracles.ts'],
|
||||
@@ -88,6 +89,8 @@ export const FREE_FIXTURES: Readonly<Record<string, readonly string[]>> = {
|
||||
'test/fixtures/design-md-legacy.md': ['test/design-md.test.ts'],
|
||||
'test/fixtures/design-ui-focus-capture-39199692993.json': ['test/plan-design-with-ui-fixture.test.ts'],
|
||||
'test/fixtures/design-ui-focus-capture-40664336141.json': ['test/plan-design-with-ui-fixture.test.ts'],
|
||||
'test/fixtures/design-ui-focus-capture-41178131219.json': ['test/plan-design-with-ui-fixture.test.ts'],
|
||||
'test/fixtures/design-ui-focus-capture-41186376825.json': ['test/plan-design-with-ui-fixture.test.ts'],
|
||||
'test/fixtures/design-ui-focus-capture.json': ['test/plan-design-with-ui-fixture.test.ts'],
|
||||
'test/fixtures/detector-corpora/auto-decide-preserved/**': ['test/detector-corpus-auto-decide-preserved.test.ts'],
|
||||
'test/fixtures/detector-corpora/plan-ceo-section-loading/**': ['test/detector-corpus-plan-ceo-section-loading.test.ts'],
|
||||
|
||||
@@ -138,17 +138,27 @@ export async function callJudge<T>(
|
||||
// What a cancelled request had received, so a judge deadline names rate-limit
|
||||
// waits, SDK retries, a stream with no events, or a long thinking phase. Counts
|
||||
// only, never content; an observer that throws never changes the judge outcome.
|
||||
// The SDK drops SSE pings before streamEvent, so lastEventMs alone cannot tell a
|
||||
// live connection with a silent thinking phase from a dead one (run 41185637598):
|
||||
// lastByteMs counts every body byte, pings included.
|
||||
const started = performance.now();
|
||||
const elapsedMs = () => Math.round(performance.now() - started);
|
||||
const progress = { httpRequests: 0, httpStatuses: [] as number[], retryWaitsMs: [] as number[], connectedMs: null as number | null,
|
||||
firstEventMs: null as number | null, lastEventMs: null as number | null, block: null as string | null, thinkingChars: 0,
|
||||
textChars: 0, messageId: null as string | null, requestId: null as string | null };
|
||||
textChars: 0, messageId: null as string | null, requestId: null as string | null,
|
||||
bytesReceived: 0, lastByteMs: null as number | null };
|
||||
const observe = (record: () => void) => { try { record(); } catch {} };
|
||||
const client = new Anthropic({ fetch: async (...args: Parameters<typeof fetch>) => {
|
||||
progress.httpRequests += 1;
|
||||
const response = await globalThis.fetch(...args);
|
||||
progress.httpStatuses.push(response.status);
|
||||
return response;
|
||||
if (!response.body) return response;
|
||||
const counted = new Response(response.body.pipeThrough(new TransformStream<Uint8Array, Uint8Array>({ transform(chunk, stream) {
|
||||
observe(() => { progress.bytesReceived += chunk.byteLength; progress.lastByteMs = elapsedMs(); });
|
||||
stream.enqueue(chunk);
|
||||
} })), { status: response.status, statusText: response.statusText, headers: response.headers });
|
||||
Object.defineProperty(counted, 'url', { value: response.url });
|
||||
return counted;
|
||||
} });
|
||||
|
||||
const request = {
|
||||
|
||||
@@ -60,7 +60,11 @@ export const DESIGN_BOARD_ACTOR_PROTOCOL = [
|
||||
const SUBMIT = `
|
||||
import { readStateFile, verifyIdentity, CMDLINE_MARKER } from ${JSON.stringify(path.resolve(import.meta.dir, '../../design/src/daemon-state.ts'))};
|
||||
import fs from 'node:fs';
|
||||
// Windows Free Tests 40815257995, 41185352916: the parent killed this child at its 2 s budget with no stderr. Each
|
||||
// phase is written synchronously, so a kill's stderr ends at the step that stalled (start-up, CIM query, or HTTP).
|
||||
const input = JSON.parse(await Bun.stdin.text());
|
||||
const phase = (name) => fs.writeSync(2, name + ' with ' + (input.deadlineAt - Date.now()) + ' ms left\\n');
|
||||
phase('child started');
|
||||
const state = readStateFile(input.stateFile);
|
||||
if (!fs.lstatSync(input.stateFile).isFile() || !state
|
||||
|| !Number.isSafeInteger(state.pid) || state.pid <= 0
|
||||
@@ -70,7 +74,9 @@ if (!fs.lstatSync(input.stateFile).isFile() || !state
|
||||
// Leave time to reap a timed-out native query before the parent kills us.
|
||||
const identityBudget = input.deadlineAt - Date.now() - 100;
|
||||
if (identityBudget <= 0) throw new Error('Design feedback deadline exhausted');
|
||||
phase('identity query');
|
||||
if (!verifyIdentity(state.pid, CMDLINE_MARKER, identityBudget)) throw new Error('No matching owned design daemon');
|
||||
phase('identity verified');
|
||||
const remaining = input.deadlineAt - Date.now();
|
||||
if (remaining <= 0) throw new Error('Design feedback deadline exhausted');
|
||||
if (input.alreadySubmitted) {
|
||||
@@ -85,6 +91,7 @@ if (input.alreadySubmitted) {
|
||||
}
|
||||
const requestBudget = input.deadlineAt - Date.now();
|
||||
if (requestBudget <= 0) throw new Error('Design feedback deadline exhausted');
|
||||
phase('feedback POST');
|
||||
const response = await fetch(input.url + 'api/feedback', {
|
||||
method: 'POST', redirect: 'error', signal: AbortSignal.timeout(requestBudget),
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
|
||||
@@ -60,10 +60,28 @@ export function budgetHarness(): Harness {
|
||||
git('checkout', '-q', '-b', 'feature');
|
||||
fs.writeFileSync(path.join(repo, 'app.ts'), 'export const x = 2;\n');
|
||||
git('commit', '-qam', 'change');
|
||||
// Windows Free Tests 41131257934 (shard 6): two consecutive cases hit bun's 30 s test timeout inside one bin call each,
|
||||
// with empty output and no cause. A run that ends without an exit code now names its command, how it ended, its
|
||||
// elapsed time and, on Windows, the children it left (bash, git or a nested bin), so the next red names the stuck stage.
|
||||
let abnormal = '';
|
||||
const childListMs = 10_000;
|
||||
const run = (bin: string, args: string[], extraEnv: NodeJS.ProcessEnv = {}): Run => {
|
||||
const argv = bin === REVIEW_LOG_BIN ? ['bash', bin] : [process.execPath, ...bunIsolationArgs(), bin];
|
||||
const started = Date.now();
|
||||
abnormal = '';
|
||||
const r = spawnSync(argv[0], [...argv.slice(1), ...args], { cwd: repo, env: { ...env, ...extraEnv }, encoding: 'utf8', timeout: 60_000 });
|
||||
return { code: r.status ?? -1, out: r.stdout ?? '', err: r.stderr ?? '' };
|
||||
const result = { code: r.status ?? -1, out: r.stdout ?? '', err: r.stderr ?? '' };
|
||||
if (r.status !== null) return result;
|
||||
abnormal = `${path.basename(bin)} ${args[0]} ended without an exit code (${r.signal ?? r.error?.message ?? 'no signal'}) after ${Date.now() - started} ms`;
|
||||
if (process.platform === 'win32' && r.pid) {
|
||||
const children = spawnSync('powershell.exe', ['-NoProfile', '-NonInteractive', '-Command',
|
||||
`Get-CimInstance Win32_Process -Filter 'ParentProcessId=${r.pid}' | ForEach-Object { '{0} {1}' -f $_.ProcessId, $_.CommandLine }`,
|
||||
], { encoding: 'utf8', timeout: childListMs, windowsHide: true });
|
||||
abnormal += `; children of ${r.pid}: ${children.stdout?.trim() || (children.error ? `unavailable (${children.error.message})` : 'none')}`;
|
||||
}
|
||||
result.err += `\n[harness] ${abnormal}`;
|
||||
process.stderr.write(`[harness] ${abnormal}\n`);
|
||||
return result;
|
||||
};
|
||||
let records = 0;
|
||||
return {
|
||||
@@ -79,7 +97,7 @@ export function budgetHarness(): Harness {
|
||||
field(out, line, key) {
|
||||
const row = out.split('\n').find(l => l.startsWith(line + ':'));
|
||||
const m = row ? new RegExp(`(?:^|\\s)${key}=(\\S+)`).exec(row) : null;
|
||||
if (!m) throw new Error(`no ${key}= on a ${line}: line in:\n${out}`);
|
||||
if (!m) throw new Error(`no ${key}= on a ${line}: line in:\n${out}${abnormal ? `\n[harness] last run without an exit code: ${abnormal}` : ''}`);
|
||||
return m[1]!;
|
||||
},
|
||||
ledger: file => JSON.parse(fs.readFileSync(file, 'utf8')),
|
||||
|
||||
@@ -6,9 +6,11 @@
|
||||
* ending the previous line) never count. The canonical phrase may name Aside as "it" once the macOS 15+ floor follows
|
||||
* ("If you want to use Aside: download it (macOS 15+) at aside.com", gate 39315041557). The short forms without the
|
||||
* floor count only at the start of a sentence, bullet or quote. The passive form ("Aside runs on macOS 15+ and can be
|
||||
* downloaded at aside.com", gate 40987391676) counts when its clause starts with Aside or "it" and holds no negation.
|
||||
* downloaded at aside.com", gate 40987391676) counts when its clause starts with Aside or "it" and the download
|
||||
* predicate's own coordinate clause holds no negation: "Aside is not installed but can be downloaded at aside.com"
|
||||
* counts, "Aside is not something that can be downloaded" and "... but cannot be downloaded" never do.
|
||||
*/
|
||||
const ASIDE_DOWNLOAD_PITCH = String.raw`(?:^|[.!?;:]\s*|\n)\s*(?:[>*-]\s*)*["\u201C]?download (?:it|Aside) at aside\.com\b|(?:^|[.!?;:,\u2014\u2013\n])[\s>*_-]*(?<!(?:\b(?:not|never|no|cannot)|n['\u2019]t)\s+)(?:(?:then|please|just|so|first|now)\s+)?(?:you\s+(?:can|could|may|should|need\s+to)\s+)?download (?:Aside|it) \(macOS 15\+\) at aside\.com\b|(?:^|[.!?;:,\u2014\u2013\n])[\s>*_-]*(?:Aside|it)\b(?:(?!\b(?:not|never|no)\b|n['\u2019]t\b)[^.!?;:,\n])*?\b(?:can|may)\s+be\s+downloaded\s+(?:at|from)\s+aside\.com\b`;
|
||||
const ASIDE_DOWNLOAD_PITCH = String.raw`(?:^|[.!?;:]\s*|\n)\s*(?:[>*-]\s*)*["\u201C]?download (?:it|Aside) at aside\.com\b|(?:^|[.!?;:,\u2014\u2013\n])[\s>*_-]*(?<!(?:\b(?:not|never|no|cannot)|n['\u2019]t)\s+)(?:(?:then|please|just|so|first|now)\s+)?(?:you\s+(?:can|could|may|should|need\s+to)\s+)?download (?:Aside|it) \(macOS 15\+\) at aside\.com\b|(?:^|[.!?;:,\u2014\u2013\n])[\s>*_-]*(?:Aside|it)\b(?:(?=(?<clause>(?:(?!\b(?:can|may)\s+be\s+downloaded\b)[^.!?;:,\n])*\b(?:but|and|yet)\b))\k<clause>)?(?:(?!\b(?:not|never|no)\b|n['\u2019]t\b)[^.!?;:,\n])*?\b(?:can|may)\s+be\s+downloaded\s+(?:at|from)\s+aside\.com\b`;
|
||||
export const asideDownloadPitch = new RegExp(ASIDE_DOWNLOAD_PITCH, 'im');
|
||||
export const asideDownloadPitchCount = (text: string): number => [...text.matchAll(new RegExp(ASIDE_DOWNLOAD_PITCH, 'gim'))].length;
|
||||
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
import { spawnSync } from 'node:child_process';
|
||||
|
||||
// A Windows runner pays a one-time cost the first time it executes a binary, and
|
||||
// all of it lands on whichever test spawns that binary first. On 12 fresh
|
||||
// windows-latest VMs (probe run 41205346554) the first launch cost jq +60-250 ms,
|
||||
// awk +190-290 ms and powershell.exe +1.7-3.1 s over a warm launch; under shard
|
||||
// load the same first jq or awk launch passed a 5 s per-call kill (Windows Free
|
||||
// Tests 41195098239, 41174352235) while every later call took about 0.3 s. Pay it
|
||||
// once here, unasserted, so a per-call kill times the code under test.
|
||||
export function warmFirstExec(command: string, args: string[]): void {
|
||||
spawnSync(command, args, { stdio: 'ignore', timeout: 25_000 });
|
||||
}
|
||||
@@ -1,9 +1,10 @@
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { beforeAll, describe, expect, test } from 'bun:test';
|
||||
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { expectMentions } from './helpers/prompt-structure';
|
||||
import { warmFirstExec } from './helpers/warm-first-exec';
|
||||
|
||||
const root = join(import.meta.dir, '..');
|
||||
const source = (name: string) => readFileSync(join(root, 'land-and-deploy', name), 'utf8');
|
||||
@@ -65,6 +66,9 @@ describe('land-and-deploy decision ordering', () => {
|
||||
});
|
||||
|
||||
describe('land-and-deploy native readback dispatcher', () => {
|
||||
// Windows Free Tests 40592848146, 41195098239, 41196095371 (first dispatch) and 41175329114 (first five) hit the 5 s
|
||||
// kill, then calls settled at 0.3-0.5 s; serial retries took 0.3 s. The first jq launch is a runner cost.
|
||||
beforeAll(() => warmFirstExec('bash', ['-c', "printf '{}' | jq -e . >/dev/null"]), 30_000);
|
||||
const shell = source('sections/merge-and-deploy.md.tmpl').match(/```bash\n(READBACK=[\s\S]*?)\n```/)?.[1];
|
||||
const head = 'a'.repeat(40);
|
||||
const pr = (extra = {}) => ({
|
||||
|
||||
@@ -223,3 +223,43 @@ test('an abort reason is retained at most 200 characters long', async () => {
|
||||
await rejected;
|
||||
expect(JSON.parse(diagnostics.mock.calls[0]![0]).reason).toBe('x'.repeat(200));
|
||||
});
|
||||
|
||||
test('pings after the last SDK event show a live connection; a silent socket shows none (run 41185637598)', async () => {
|
||||
const encoder = new TextEncoder();
|
||||
const frame = (event: Record<string, unknown>) => encoder.encode(`event: ${event.type}\ndata: ${JSON.stringify(event)}\n\n`);
|
||||
const opening = [
|
||||
frame({ type: 'message_start', message: { id: 'msg_quiet', type: 'message', role: 'assistant', model: 'claude-fable-5-1',
|
||||
content: [], stop_reason: null, stop_sequence: null, usage: { input_tokens: 99023, output_tokens: 1 } } }),
|
||||
frame({ type: 'content_block_start', index: 0, content_block: { type: 'thinking', thinking: '', signature: '' } }),
|
||||
];
|
||||
const ping = frame({ type: 'ping' });
|
||||
const opened = opening.reduce((sum, chunk) => sum + chunk.byteLength, 0);
|
||||
for (const pings of [0, 3]) {
|
||||
diagnostics.mockClear();
|
||||
let delivered!: () => void;
|
||||
const received = new Promise<void>(resolve => { delivered = resolve; });
|
||||
const chunks = [...opening, ...Array.from({ length: pings }, () => ping)];
|
||||
transport.mockImplementation(async (_url: string, init: RequestInit) => new Response(new ReadableStream({
|
||||
start(stream) { init.signal!.addEventListener('abort', () => stream.error(init.signal!.reason), { once: true }); },
|
||||
pull(stream) {
|
||||
const chunk = chunks.shift();
|
||||
if (chunk) stream.enqueue(chunk);
|
||||
else delivered();
|
||||
},
|
||||
}, { highWaterMark: 0 }), { headers: { 'content-type': 'text/event-stream', 'request-id': 'req_quiet' } }));
|
||||
const controller = new AbortController();
|
||||
const rejected = callJudge('score the whole bundle', 'claude-fable-5-1', {
|
||||
max_tokens: 65_536, stream: true, jsonSchema: WORKFLOW_JUDGE_RESPONSE_SCHEMA, signal: controller.signal,
|
||||
}).catch(error => error);
|
||||
await received;
|
||||
const failure = new Error('review/SKILL.md workflow exceeded its workflow judge work deadline');
|
||||
controller.abort(failure);
|
||||
expect(await rejected).toBe(failure);
|
||||
expect(diagnostics).toHaveBeenCalledTimes(1);
|
||||
const event = JSON.parse(diagnostics.mock.calls[0]![0]);
|
||||
expect(event).toMatchObject({ type: 'llm-judge-aborted', httpStatuses: [200], block: 'thinking', thinkingChars: 0, textChars: 0,
|
||||
messageId: 'msg_quiet', bytesReceived: opened + pings * ping.byteLength });
|
||||
if (pings) expect(event.lastByteMs).toBeGreaterThanOrEqual(event.lastEventMs); // gstack-timing-ok: ordering of two recorded offsets, not an elapsed-time bound
|
||||
else expect(event.lastByteMs).toBeLessThanOrEqual(event.lastEventMs); // gstack-timing-ok: the last byte is counted before the SDK parses it
|
||||
}
|
||||
});
|
||||
@@ -99,6 +99,10 @@ const captured39852003387 = JSON.parse(fs.readFileSync(${JSON.stringify(path.joi
|
||||
// Should I review all 7 areas or focus on a few?" with lettered options ("A) Review all 7 areas").
|
||||
const captured40664336141 = JSON.parse(fs.readFileSync(${JSON.stringify(path.join(ROOT, 'test/fixtures/design-ui-focus-capture-40664336141.json'))}, 'utf8'))
|
||||
.calls.map(({ preReview, observedAtMs, call }) => nativePlanCallFingerprint(call, observedAtMs, preReview));
|
||||
// Runs 41178131219 and 41186376825 slice 6 (a77fae28, 047dad78): Step 0D titled only "Review focus",
|
||||
// rated in its ELI10 body, and lettered its options ("A) All 7 passes", "A) All 7 dimensions").
|
||||
const [captured41178131219, captured41186376825] = ['design-ui-focus-capture-41178131219.json', 'design-ui-focus-capture-41186376825.json'].map(name => JSON.parse(fs.readFileSync(${JSON.stringify(ROOT)} + '/test/fixtures/' + name, 'utf8'))
|
||||
.calls.map(({ preReview, observedAtMs, call }) => nativePlanCallFingerprint(call, observedAtMs, preReview)));
|
||||
const unnumberedFinding = fp('unnumbered-finding', "Which region is the dashboard's primary anchor, and how is the page composed?", false);
|
||||
unnumberedFinding.nativeCall.questions[0].options = nativeFinding.nativeCall.questions[0].options;
|
||||
const finding = fp('finding', 'Which loading feedback should Save show?', false);
|
||||
@@ -231,6 +235,24 @@ mock.module(${JSON.stringify(path.join(ROOT, 'test/helpers/claude-pty-runner.ts'
|
||||
expect(opts.isReviewAUQ(captured40664336141[1])).toBe(false);
|
||||
expect([0, 2].some(index => opts.isLastStep0AUQ(captured40664336141[index]))).toBe(false);
|
||||
expect(opts.isReviewAUQ(captured40664336141[3])).toBe(true);
|
||||
for (const capture of [captured41178131219, captured41186376825]) {
|
||||
expect(opts.isLastStep0AUQ(capture[1])).toBe(true);
|
||||
expect(opts.isReviewAUQ(capture[1])).toBe(false);
|
||||
expect([0, 2].some(index => opts.isLastStep0AUQ(capture[index]) || opts.isReviewAUQ(capture[index]))).toBe(false);
|
||||
expect(opts.isReviewAUQ(capture[3])).toBe(true);
|
||||
}
|
||||
const letteredFocus = 'D2 — Review focus\\nProject/branch/task: main.\\nELI10: I rated this plan 4/10 on design completeness.';
|
||||
for (const labels of [['A) All 7 dimensions (recommended)', 'B) Focus on specific areas'], ['A. All seven passes (recommended)', 'B. Hierarchy+states only'],
|
||||
['A: All 7 areas (recommended)', 'B: Only specific areas']]) {
|
||||
expect(opts.isLastStep0AUQ(labeled('lettered-focus', letteredFocus, labels)), labels[0]).toBe(true);
|
||||
}
|
||||
for (const [label, question, labels] of [
|
||||
['lettered-not-leading', letteredFocus, ['A) Review all 7 dimensions (recommended)', 'B) Focus on specific areas']],
|
||||
['lettered-bare', letteredFocus, ['A) All 7', 'B) Focus on specific areas']],
|
||||
['lettered-word-prefix', letteredFocus, ['AB) All 7 dimensions (recommended)', 'B) Focus on specific areas']],
|
||||
['lettered-out-of-seven', letteredFocus.replace('4/10', '4/7'), ['A) All 7 dimensions (recommended)', 'B) Focus on specific areas']],
|
||||
['lettered-without-focus', 'D2 — Which plan should I review?\\nELI10: I rated this plan 4/10 on design completeness.', ['A) All 7 dimensions (recommended)', 'B) PLAN.md']],
|
||||
]) expect(opts.isLastStep0AUQ(labeled(label, question, labels)), label).toBe(false);
|
||||
const lettered = ['A) Review all 7 areas (recommended)', 'B) Focus on hierarchy/states/responsive'];
|
||||
for (const [label, question, labels] of [
|
||||
['title-offer-unrated', 'D2: Should I review all 7 areas or focus on a few?\\nELI10: I will review every area unless you narrow it.', lettered],
|
||||
@@ -318,6 +340,10 @@ mock.module(${JSON.stringify(path.join(ROOT, 'test/helpers/claude-pty-runner.ts'
|
||||
: mode === 'captured-39852003387-no-focus' ? captured39852003387.filter((_, index) => index !== 0)
|
||||
: mode === 'captured-40664336141' ? captured40664336141
|
||||
: mode === 'captured-40664336141-no-focus' ? captured40664336141.filter((_, index) => index !== 1)
|
||||
: mode === 'captured-41178131219' ? captured41178131219
|
||||
: mode === 'captured-41178131219-no-focus' ? captured41178131219.filter((_, index) => index !== 1)
|
||||
: mode === 'captured-41186376825' ? captured41186376825
|
||||
: mode === 'captured-41186376825-no-focus' ? captured41186376825.filter((_, index) => index !== 1)
|
||||
: mode === 'captured-focus-after-finding' ? [...captured.filter((_, index) => index !== 1), captured[1]]
|
||||
: mode.startsWith('target-menu') ? [target, fp('other', 'Which artifact should I inspect?', false)]
|
||||
: mode === 'early-exit' ? [] : mode === 'focus-only' ? [chosenFocus]
|
||||
@@ -362,12 +388,12 @@ await import(${JSON.stringify(path.join(ROOT, 'test/skill-e2e-plan-design-with-u
|
||||
} finally { fs.rmSync(directory, { recursive: true, force: true }); }
|
||||
}
|
||||
|
||||
test.each(['source', 'paraphrase', 'native-sequence', 'native-unnumbered', 'native-passes', 'captured', 'captured-rated', 'captured-39298269902', 'captured-39363705692', 'captured-39852003387', 'captured-40664336141'])('UI gate seeds the exact target and accepts Design progress with an unselected no-UI alternative (%s)', mode => {
|
||||
test.each(['source', 'paraphrase', 'native-sequence', 'native-unnumbered', 'native-passes', 'captured', 'captured-rated', 'captured-39298269902', 'captured-39363705692', 'captured-39852003387', 'captured-40664336141', 'captured-41178131219', 'captured-41186376825'])('UI gate seeds the exact target and accepts Design progress with an unselected no-UI alternative (%s)', mode => {
|
||||
const result = exercise(mode);
|
||||
expect(result.code, result.output).toBe(0);
|
||||
}, 20_000);
|
||||
|
||||
test.each(['target-menu', 'target-menu-design-system', 'focus-only', 'captured-no-focus', 'captured-focus-after-finding', 'captured-rated-no-focus', 'captured-39298269902-no-focus', 'captured-39298269902-focus-only', 'captured-39363705692-no-focus', 'captured-39852003387-no-focus', 'captured-40664336141-no-focus', 'native-setup-only', 'native-voices-only', 'early-exit', 'timeout', 'exited', 'unanswered-finding', 'failed-finding', 'no-native-finding'])('UI gate rejects %s and removes its fixture', mode => {
|
||||
test.each(['target-menu', 'target-menu-design-system', 'focus-only', 'captured-no-focus', 'captured-focus-after-finding', 'captured-rated-no-focus', 'captured-39298269902-no-focus', 'captured-39298269902-focus-only', 'captured-39363705692-no-focus', 'captured-39852003387-no-focus', 'captured-40664336141-no-focus', 'captured-41178131219-no-focus', 'captured-41186376825-no-focus', 'native-setup-only', 'native-voices-only', 'early-exit', 'timeout', 'exited', 'unanswered-finding', 'failed-finding', 'no-native-finding'])('UI gate rejects %s and removes its fixture', mode => {
|
||||
const result = exercise(mode);
|
||||
expect(result.code, result.output).toBe(1);
|
||||
expect(result.output).toContain('plan-design-review with UI scope FAILED');
|
||||
|
||||
@@ -4,9 +4,36 @@ import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { startTestServer } from '../browse/test/test-server';
|
||||
import { bunIsolationArgs } from '../lib/bun-isolation';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const PROBE = path.join(ROOT, 'test/fixtures/qa-only-browser-probe.ts');
|
||||
const FILE_OWNERS = path.join(ROOT, 'browse/test/fixtures/native-cookie-file-owners.ts');
|
||||
|
||||
// Windows Free Tests 40624015029, 40842574997, 41068546881 and 41172879765: rm of qa-probe-* hit EBUSY after every child
|
||||
// had exited. Restart Manager names the holder: a process outside the test (the runner's %TEMP%\provjobd.exe agent)
|
||||
// opens each newly run browse.exe and lets go 1-52 ms later (GSTA-199 scratch runs 41205291567, 41205903081). Wait only for an
|
||||
// outside holder of browse.exe, then remove once more; a test-host owner, a holder past 5 s or a second failure throws
|
||||
// with the owners named.
|
||||
function removeProbeDirectory(directory: string, browse: string) {
|
||||
try { fs.rmSync(directory, { recursive: true, force: true }); return; }
|
||||
catch (error) {
|
||||
if (process.platform !== 'win32' || (error as NodeJS.ErrnoException).code !== 'EBUSY' || !fs.existsSync(browse)) throw error;
|
||||
}
|
||||
const deadline = Date.now() + 5_000;
|
||||
let receipt: { available?: boolean; owners?: { isTestHost: boolean }[] };
|
||||
for (;;) {
|
||||
const result = spawnSync(process.execPath, [...bunIsolationArgs(), FILE_OWNERS,
|
||||
Buffer.from(JSON.stringify({ root: directory, file: browse, testPid: process.pid })).toString('base64')],
|
||||
{ encoding: 'utf8', timeout: 5_000, windowsHide: true });
|
||||
try { receipt = JSON.parse(result.stdout); } catch { receipt = { available: false }; }
|
||||
const outside = receipt.available === true && !!receipt.owners?.length && !receipt.owners.some(owner => owner.isTestHost);
|
||||
if (!outside || Date.now() > deadline) break;
|
||||
Bun.sleepSync(25);
|
||||
}
|
||||
try { fs.rmSync(directory, { recursive: true, force: true }); }
|
||||
catch (error) { throw new Error(`${(error as Error).message}; browse.exe owners: ${JSON.stringify(receipt)}`); }
|
||||
}
|
||||
|
||||
test('the report-only fixture serves exactly one bounded, observable defect', async () => {
|
||||
const { server, url } = startTestServer();
|
||||
@@ -78,6 +105,6 @@ process.stderr.write(args[0]+' diagnostic\\n');
|
||||
}
|
||||
expect(fs.readFileSync(screenshot, 'utf8')).toBe('fixture image');
|
||||
} finally {
|
||||
fs.rmSync(directory, { recursive: true, force: true });
|
||||
removeProbeDirectory(directory, browse);
|
||||
}
|
||||
});
|
||||
@@ -26,15 +26,24 @@ const guard = block
|
||||
.replaceAll(`${installed}gstack-redact`, '"$GSTACK_TEST_BUN" "$GSTACK_TEST_HELPERS/gstack-redact"');
|
||||
|
||||
let root = '', main = '', worktree = '', env: NodeJS.ProcessEnv = {};
|
||||
const run = (command: string, args: string[], cwd: string) =>
|
||||
spawnSync(command, args, { cwd, env, encoding: 'utf8', timeout: 30_000 });
|
||||
// Windows Free Tests 41190552162: the file's first setup passed the 30 s hook budget (the next took 5 s, the retry
|
||||
// 0.4 s), and a killed child reads only as status null, so each failure names how long every setup spawn took.
|
||||
let spawns: string[] = [];
|
||||
const run = (command: string, args: string[], cwd: string) => {
|
||||
const started = performance.now();
|
||||
const result = spawnSync(command, args, { cwd, env, encoding: 'utf8', timeout: 30_000 });
|
||||
spawns.push(`${command}: ${Math.round(performance.now() - started)} ms`);
|
||||
return result;
|
||||
};
|
||||
const spawnLog = (result: { stderr: string }) => `${result.stderr}\nspawns: ${spawns.join(', ')}`;
|
||||
const git = (cwd: string, ...args: string[]) => {
|
||||
const result = run('git', args, cwd);
|
||||
expect(result.status, result.stderr).toBe(0);
|
||||
expect(result.status, spawnLog(result)).toBe(0);
|
||||
return result.stdout.trim();
|
||||
};
|
||||
|
||||
beforeEach(() => {
|
||||
spawns = [];
|
||||
root = realpathSync(mkdtempSync(join(tmpdir(), 'ship-hook-paths-')));
|
||||
main = join(root, 'main');
|
||||
worktree = join(root, 'linked worktree');
|
||||
@@ -51,7 +60,7 @@ beforeEach(() => {
|
||||
git(main, '-c', 'user.email=fixture@example.test', '-c', 'user.name=Fixture', '-c', 'commit.gpgsign=false', 'commit', '--allow-empty', '-qm', 'fixture');
|
||||
git(main, 'worktree', 'add', '-q', '--detach', worktree);
|
||||
const opted = run('bash', [join(HELPERS, 'gstack-config'), 'set', 'redact_prepush_hook', 'true'], main);
|
||||
expect(opted.status, opted.stderr).toBe(0);
|
||||
expect(opted.status, spawnLog(opted)).toBe(0);
|
||||
});
|
||||
|
||||
afterEach(() => rmSync(root, { recursive: true, force: true }));
|
||||
|
||||
@@ -44,13 +44,14 @@ const designFocusBoundary = (fp: AskUserQuestionFingerprint): boolean =>
|
||||
// The option leads with all seven, whatever it calls them ("All 7 areas", run 39363705692).
|
||||
// Run 40664336141 put the offer in the rated title instead ("Should I review all 7 areas
|
||||
// or focus on a few?") and lettered its options ("A) Review all 7 areas"). Either place
|
||||
// carries the offer; issue-numbered options are a finding, never Step 0D.
|
||||
// carries the offer; issue-numbered options are a finding, never Step 0D. Runs 41178131219
|
||||
// and 41186376825 lettered the leading option itself ("A) All 7 passes (recommended)").
|
||||
const lead = text.split(/\r?\n|\s+(?=Project\/branch\/task:)/, 1)[0]!;
|
||||
return /^I(?:['’]ve| have) rated this plan (?:10(?:\.0+)?|[0-9](?:\.\d+)?)\/10 on design completeness\.[\s\S]*\bWant me to focus on specific areas instead(?: of all (?:7|seven))?\?/i.test(text)
|
||||
|| (/\ball (?:7|seven) (?:design )?(?:dimensions|passes)\b/i.test(title) && /\bfocus\b/i.test(title)
|
||||
&& (/\bdesign\b/i.test(title) || rated(text.slice(title.length))) && /\?\s*$/.test(title))
|
||||
|| (/\bfocus\b/i.test(lead) && rated(text) && options.length >= 2 && !issueLabeled(options)
|
||||
&& (/\ball (?:7|seven)\s+[a-z]/i.test(lead) || options.some(({ label }) => /^\s*all (?:7|seven)\s+[a-z]/i.test(label))));
|
||||
&& (/\ball (?:7|seven)\s+[a-z]/i.test(lead) || options.some(({ label }) => /^\s*(?:[A-Z][).:]\s*)?all (?:7|seven)\s+[a-z]/i.test(label))));
|
||||
});
|
||||
|
||||
// Require a choice about the supplied UI, not a workflow offer after focus.
|
||||
|
||||
@@ -5,6 +5,7 @@ import { dirname, join } from 'node:path';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { runGeneration } from '../scripts/gen-skill-docs';
|
||||
import { warmFirstExec } from './helpers/warm-first-exec';
|
||||
|
||||
const root = mkdtempSync(join(tmpdir(), 'skill-positional-'));
|
||||
const rendered = join(root, 'rendered');
|
||||
@@ -31,6 +32,9 @@ beforeAll(async () => {
|
||||
expect((await runGeneration({ host, outputRoot: rendered })).exitCode).toBe(0);
|
||||
}
|
||||
}, 120000);
|
||||
// Windows Free Tests 40700602081, 41174352235, 41175308095: only the file's first checksum run hit the 5 s kill
|
||||
// and serial retries took 0.3 s. The first awk launch is a runner cost, not the rendered literal.
|
||||
beforeAll(() => warmFirstExec('bash', ['-c', "printf x | sha256sum | awk '{print $(1)}'"]), 30_000);
|
||||
afterAll(() => rmSync(root, { recursive: true, force: true }));
|
||||
|
||||
function substitute(text: string, args: string[]) {
|
||||
|
||||
@@ -177,8 +177,12 @@ test('an exhausted owner deadline never launches another child', async () => {
|
||||
expect(result).toMatchObject({ exitCode: null, timedOut: true, groupPid: null, incompleteCapture: { childClosed: false } });
|
||||
}, 30_000);
|
||||
|
||||
// The exit must land before the wall for its code to survive. In Windows Free Tests 41164892775 the child took over
|
||||
// 150 ms to start and exit, so the wall won (exitCode null). The same child exits in 45-102 ms across 39 lane runs of the
|
||||
// normal-exit case and in up to 371 ms on a CPU-saturated runner (GSTA-199 scratch run 41205291567). This case asserts no
|
||||
// elapsed bound, so its wall only has to come after the exit.
|
||||
test('missing close preserves a nonzero exit while explicitly refusing completeness', async () => {
|
||||
const result = await runShardChild({ ...options, args: ['-e', 'process.exit(7)'],
|
||||
const result = await runShardChild({ ...options, timeoutMs: 2_000, args: ['-e', 'process.exit(7)'],
|
||||
hookStreams: child => {
|
||||
const emit = child.emit.bind(child);
|
||||
child.emit = ((event: string | symbol, ...args: unknown[]) => event === 'close' ? true : emit(event, ...args)) as typeof child.emit;
|
||||
|
||||
@@ -10,11 +10,12 @@
|
||||
* platform's default shell (`shell: true`: /bin/sh here, cmd.exe on Windows),
|
||||
* plus PowerShell when one is installed, from a project path with a space.
|
||||
*/
|
||||
import { afterAll, describe, expect, test } from 'bun:test';
|
||||
import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { warmFirstExec } from './helpers/warm-first-exec';
|
||||
|
||||
const TEAM_INIT = fs.readFileSync(new URL('../bin/gstack-team-init', import.meta.url), 'utf-8');
|
||||
|
||||
@@ -45,6 +46,10 @@ const viaPowerShell: Runner = (env) =>
|
||||
spawnSync(powershell!, ['-NoProfile', '-NonInteractive', '-Command', command], { cwd: elsewhere, env, encoding: 'utf-8', timeout: 30_000 });
|
||||
|
||||
describe('required team hook: shell-neutral registration', () => {
|
||||
// The first powershell.exe launch on a runner costs 1.7-3.1 s more than a warm one (probe 41205346554), inside the
|
||||
// three-launch test's shared 30 s budget (41187420609 lost its first launch). Pay it once here, unasserted.
|
||||
if (powershell) beforeAll(() => warmFirstExec(powershell, ['-NoProfile', '-NonInteractive', '-Command', 'exit 0']), 30_000);
|
||||
|
||||
test('the extracted command is a node launcher with no shell variable syntax', () => {
|
||||
expect(hookSource).toContain("'use strict';");
|
||||
expect(command).toStartWith('node -e "');
|
||||
@@ -74,19 +79,28 @@ describe('required team hook: shell-neutral registration', () => {
|
||||
const home = path.join(base, `home-${name.replace(/\W+/g, '-')}`);
|
||||
fs.mkdirSync(home, { recursive: true });
|
||||
|
||||
const denied = runner(hookEnv({ HOME: home, USERPROFILE: home }));
|
||||
expect(denied.status).toBe(denyStatus);
|
||||
// A launch killed by the 30 s test budget reads as status null; the launch times name the one that stalled.
|
||||
const launches: string[] = [];
|
||||
const timed: Runner = (env) => {
|
||||
const started = performance.now();
|
||||
const result = runner(env);
|
||||
launches.push(`${Math.round(performance.now() - started)} ms`);
|
||||
return result;
|
||||
};
|
||||
|
||||
const denied = timed(hookEnv({ HOME: home, USERPROFILE: home }));
|
||||
expect(denied.status, `launches: ${launches.join(', ')}`).toBe(denyStatus);
|
||||
expect(JSON.parse(denied.stdout.trim()).hookSpecificOutput).toMatchObject({ hookEventName: 'PreToolUse', permissionDecision: 'deny' });
|
||||
expect(denied.stderr).toContain('BLOCKED: gstack is not installed globally.');
|
||||
|
||||
fs.mkdirSync(path.join(home, '.claude/skills/gstack/bin'), { recursive: true });
|
||||
const allowed = runner(hookEnv({ HOME: home, USERPROFILE: home }));
|
||||
const allowed = timed(hookEnv({ HOME: home, USERPROFILE: home }));
|
||||
expect(allowed.stderr).toBe('');
|
||||
expect(allowed.stdout.trim()).toBe('{}');
|
||||
expect(allowed.stdout.trim(), `launches: ${launches.join(', ')}`).toBe('{}');
|
||||
expect(allowed.status).toBe(0);
|
||||
|
||||
const viaProfile = runner(hookEnv({ USERPROFILE: home }));
|
||||
expect(viaProfile.stdout.trim()).toBe('{}');
|
||||
const viaProfile = timed(hookEnv({ USERPROFILE: home }));
|
||||
expect(viaProfile.stdout.trim(), `launches: ${launches.join(', ')}`).toBe('{}');
|
||||
expect(viaProfile.status).toBe(0);
|
||||
});
|
||||
}
|
||||
|
||||
@@ -454,6 +454,17 @@ describe("NEEDS_ASIDE: Darwin download pitch detector", () => {
|
||||
expect(asideDownloadPitchCount('It may be downloaded from aside.com (macOS 15+); open, sign in, re-run.')).toBe(1);
|
||||
});
|
||||
|
||||
test("a negation counts against the passive pitch only inside the download predicate's own coordinate clause", () => {
|
||||
for (const text of ['Aside is not installed but can be downloaded at aside.com.', 'Probe: Aside is not running yet and can be downloaded from aside.com.',
|
||||
'Aside is not installed but it can be downloaded at aside.com (macOS 15+).', 'It is not on this Mac yet but can be downloaded at aside.com.'])
|
||||
expect(asideDownloadPitchCount(text), text).toBe(1);
|
||||
for (const text of ['Aside is not installed but cannot be downloaded at aside.com.', 'Aside is not installed and is not available to download at aside.com.',
|
||||
"Aside isn't installed and can't be downloaded at aside.com.", 'Aside is not installed but it never can be downloaded at aside.com.',
|
||||
'Aside is not installed but is not something that can be downloaded at aside.com.', 'Aside is not installed but can not be downloaded at aside.com.',
|
||||
'Aside is not installed; the policy says it can be downloaded at aside.com.'])
|
||||
expect(asideDownloadPitchCount(text), text).toBe(0);
|
||||
});
|
||||
|
||||
test("short forms count only at a sentence start; negated canonical phrases never count", () => {
|
||||
expect('Download it (macOS 15+) at aside.com; open, sign in, re-run.').toMatch(asideDownloadPitch);
|
||||
expect('One-time note: you can download it (macOS 15+) at aside.com.').toMatch(asideDownloadPitch);
|
||||
|
||||
Reference in new issue
Block a user