mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-04 15:57:33 +01:00
v1.94.34.0 test(flaky): autoplan-guard-pty dispatch token relay; plan-design Voices menu accepts both bullet spellings (#3953)
* test: autoplan-guard-pty dispatch relay and plan-design Voices bullet spellings (GSTA-262) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: relay normalizes CRLF across chunk boundaries (outside voice) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(changelog): name 41247097050 as a PR screen Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
18d557d30c
commit
bf2cab34c5
10 files changed
+306
-37
No files matched your search
@@ -1,5 +1,12 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
|
## [1.94.34.0] - 2026-11-03
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **The autoplan-guard-pty eval no longer fails when its test actor mistypes a hash.** The live guard test has a Haiku model act out an /autoplan phase boundary, and it had to copy each reviewer dispatch, including two 64-character hashes, into its Agent call. Every so often it got a hash wrong (in PR screen 41247097050 it made up a tail for the Design dispatch's hash), and the guard rightly refused the dispatch. Now the actor sends a short declared token, `DISPATCH ceo` or `DISPATCH design`, and a loopback relay in the test driver puts that phase's exact dispatch in place of the token before Claude Code sees the tool call. The relay also puts the token back in the history the actor sees, because in the first panel's trial 1 the actor copied the substituted CEO dispatch for its Design call. The guard, the hash in the dispatch, the Haiku actor and every assertion are unchanged. The first preregistered panel stopped at that trial 1 failure. After the history fix, the second panel passed 3 of 3 (4 of 4 guarded allows in each trial, no permission cards).
|
||||||
|
- **The plan-design-with-ui-scope eval accepts both ways the skill writes the outside-voices menu.** The question format says option descriptions use literal ✅/❌ bullets, and models read that two ways: earlier captures kept the bullets in the descriptions, while train gate 41470212877 moved them into the question text and dropped them from the descriptions, so the test's fixture user refused the menu. The fixture now declares both spellings, and the picker accepts exactly the declared description, or that description with every bullet removed. The red question is saved as a fixture. Restoring only some bullets, flipping one, swapping or rewording a description still fail. A focused paid run passed.
|
||||||
|
|
||||||
## [1.94.33.0] - 2026-11-03
|
## [1.94.33.0] - 2026-11-03
|
||||||
|
|
||||||
Follow-ups to the skill routing and contract hooks (GST-1088). `gstack-ship-queue ready` no longer misreads files a PR adds as deleted when the installed gstack is older than the checkout: the paid-case selector now derives dependencies from the checkout under review. Each of the two hook switches now has one default, `SWITCH_DEFAULT` in `lib/skill-route.ts` (both still off), which setup, both hooks, `gstack-config get|list|defaults` and the new `gstack-skill-route --default <switch>` all read, so changing a default is a one-line edit; `gstack-config list` and `defaults` now show both switches, an unrecognized value takes the default in the hooks as it already did in setup, and a config file that exists but cannot be read turns both hooks off in setup and in the hooks whatever the default. The contract checker no longer counts installs, builds or version probes (`bun install`, `go build`, `node --version`, `make clean`) as a test run after an edit, while option values (`make -C web test`, `pnpm --filter app test`) and `node --test` still count, and the Stop hook lets a final stop through while Agent dispatches are still running, recording the run as `deferred`, then checks the contract once the same agents have been running for 10 minutes; `gstack-skill-contract report` re-checks deferred runs.
|
Follow-ups to the skill routing and contract hooks (GST-1088). `gstack-ship-queue ready` no longer misreads files a PR adds as deleted when the installed gstack is older than the checkout: the paid-case selector now derives dependencies from the checkout under review. Each of the two hook switches now has one default, `SWITCH_DEFAULT` in `lib/skill-route.ts` (both still off), which setup, both hooks, `gstack-config get|list|defaults` and the new `gstack-skill-route --default <switch>` all read, so changing a default is a one-line edit; `gstack-config list` and `defaults` now show both switches, an unrecognized value takes the default in the hooks as it already did in setup, and a config file that exists but cannot be read turns both hooks off in setup and in the hooks whatever the default. The contract checker no longer counts installs, builds or version probes (`bun install`, `go build`, `node --version`, `make clean`) as a test run after an edit, while option values (`make -C web test`, `pnpm --filter app test`) and `node --test` still count, and the Stop hook lets a final stop through while Agent dispatches are still running, recording the run as `deferred`, then checks the contract once the same agents have been running for 10 minutes; `gstack-skill-contract report` re-checks deferred runs.
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
# gstack digest v1.94.33.0 — regenerate/re-copy after upgrading gstack
|
# gstack digest v1.94.34.0 — regenerate/re-copy after upgrading gstack
|
||||||
|
|
||||||
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
|
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
|
||||||
for agent hosts without a full skill install. The full skills add workflows,
|
for agent hosts without a full skill install. The full skills add workflows,
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "gstack",
|
"name": "gstack",
|
||||||
"version": "1.94.33",
|
"version": "1.94.34",
|
||||||
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
|
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
|
||||||
"license": "MIT",
|
"license": "MIT",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
|
|||||||
@@ -1,7 +1,9 @@
|
|||||||
import { expect, test } from 'bun:test';
|
import { expect, test } from 'bun:test';
|
||||||
import * as fs from 'node:fs';
|
import * as fs from 'node:fs';
|
||||||
|
import * as os from 'node:os';
|
||||||
import * as path from 'node:path';
|
import * as path from 'node:path';
|
||||||
import { closePacketReadAfterNotice, dispatchPromptDelta, prepareGuardSession, stoppedAtPhaseBoundary } from './helpers/autoplan-guard-pty';
|
import { closePacketReadAfterNotice, dispatchEventRewriter, dispatchPromptDelta, dispatchToken, prepareGuardSession, startDispatchRelay, stoppedAtPhaseBoundary } from './helpers/autoplan-guard-pty';
|
||||||
|
import { codeOf, guardFixture } from './helpers/autoplan-guard-fixture';
|
||||||
|
|
||||||
const base = '/home/runner/.cache/gstack-paid-shard-x/tmp/ap-guard-pty-y';
|
const base = '/home/runner/.cache/gstack-paid-shard-x/tmp/ap-guard-pty-y';
|
||||||
|
|
||||||
@@ -27,10 +29,10 @@ test('the scripted actor ends its turn after a background CEO dispatch and reads
|
|||||||
try {
|
try {
|
||||||
const skill = fs.readFileSync(path.join(s.project, '.claude', 'skills', fs.readdirSync(path.join(s.project, '.claude', 'skills'))[0]!, 'SKILL.md'), 'utf8');
|
const skill = fs.readFileSync(path.join(s.project, '.claude', 'skills', fs.readdirSync(path.join(s.project, '.claude', 'skills'))[0]!, 'SKILL.md'), 'utf8');
|
||||||
const step = (n: number) => skill.slice(skill.indexOf(`\n${n}. `) + 1, skill.indexOf(`\n${n + 1}. `));
|
const step = (n: number) => skill.slice(skill.indexOf(`\n${n}. `) + 1, skill.indexOf(`\n${n + 1}. `));
|
||||||
expect(step(5)).toContain('If its result says the agent is running in the background, end your turn at once with no further tool call; the completion notification starts your next turn.');
|
expect(step(4)).toContain('If its result says the agent is running in the background, end your turn at once with no further tool call; the completion notification starts your next turn.');
|
||||||
expect(step(5)).not.toContain('If the agent runs in the background, end your turn and wait');
|
expect(step(4)).not.toContain('If the agent runs in the background, end your turn and wait');
|
||||||
expect(step(6).startsWith('6. Only after the CEO review agent has finished (in the turn its completion notification starts, when it ran in the background), Read the file whose path step 3 printed')).toBe(true);
|
expect(step(5).startsWith('5. Only after the CEO review agent has finished (in the turn its completion notification starts, when it ran in the background), Read the file whose path step 3 printed')).toBe(true);
|
||||||
expect(skill.indexOf('\n6. ')).toBeGreaterThan(skill.indexOf('\n5. '));
|
expect(skill.indexOf('\n5. ')).toBeGreaterThan(skill.indexOf('\n4. '));
|
||||||
} finally { s.cleanup(); }
|
} finally { s.cleanup(); }
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -43,25 +45,25 @@ test('the captured phase-boundary stop is named; a run still working or already
|
|||||||
expect(stoppedAtPhaseBoundary(STOP_SCREEN)).toBe(true);
|
expect(stoppedAtPhaseBoundary(STOP_SCREEN)).toBe(true);
|
||||||
const working = STOP_SCREEN.slice(0, STOP_SCREEN.indexOf('✻ Brewed'));
|
const working = STOP_SCREEN.slice(0, STOP_SCREEN.indexOf('✻ Brewed'));
|
||||||
expect(stoppedAtPhaseBoundary(working)).toBe(false);
|
expect(stoppedAtPhaseBoundary(working)).toBe(false);
|
||||||
const continued = working + '●Step8:ReadingtheDesignphasefile\n\nRead 1 file\n\n●Step9:ReadingtheDesigndispatchfile\n\n✻ Brewed for 80s · done 5:17 AM\n❯\n';
|
const continued = working + '●Step7:ReadingtheDesignphasefile\n\nRead 1 file\n\n●Step8:DispatchingtheDesignreview\n\n✻ Brewed for 80s · done 5:17 AM\n❯\n';
|
||||||
expect(stoppedAtPhaseBoundary(continued)).toBe(false);
|
expect(stoppedAtPhaseBoundary(continued)).toBe(false);
|
||||||
expect(stoppedAtPhaseBoundary('✻ Brewed for 10s · done 5:00 AM\n❯\n')).toBe(false);
|
expect(stoppedAtPhaseBoundary('✻ Brewed for 10s · done 5:00 AM\n❯\n')).toBe(false);
|
||||||
});
|
});
|
||||||
|
|
||||||
test('the scripted actor keeps going through the phase boundary: nothing between step 7 and step 11 lets it end its turn there', () => {
|
test('the scripted actor keeps going through the phase boundary: nothing between step 6 and step 9 lets it end its turn there', () => {
|
||||||
const s = prepareGuardSession(undefined);
|
const s = prepareGuardSession(undefined);
|
||||||
try {
|
try {
|
||||||
const skill = fs.readFileSync(path.join(s.project, '.claude', 'skills', fs.readdirSync(path.join(s.project, '.claude', 'skills'))[0]!, 'SKILL.md'), 'utf8');
|
const skill = fs.readFileSync(path.join(s.project, '.claude', 'skills', fs.readdirSync(path.join(s.project, '.claude', 'skills'))[0]!, 'SKILL.md'), 'utf8');
|
||||||
const step = (n: number) => skill.slice(skill.indexOf(`\n${n}. `) + 1, skill.indexOf(`\n${n + 1}. `));
|
const step = (n: number) => skill.slice(skill.indexOf(`\n${n}. `) + 1, skill.indexOf(`\n${n + 1}. `));
|
||||||
const intro = skill.slice(skill.indexOf('\n---\n', 4) + 5, skill.indexOf('\n1. '));
|
const intro = skill.slice(skill.indexOf('\n---\n', 4) + 5, skill.indexOf('\n1. '));
|
||||||
expect(intro).toContain('from step 1 through step 11 in one continuous run');
|
expect(intro).toContain('from step 1 through step 9 in one continuous run');
|
||||||
expect(intro).toContain('End your turn before step 11 only where step 5 or step 10 says to wait for a background agent.');
|
expect(intro).toContain('End your turn before step 9 only where step 4 or step 8 says to wait for a background agent.');
|
||||||
expect(intro).toContain('**Phase 1 complete.** is a phase boundary, not the end of the task.');
|
expect(intro).toContain('**Phase 1 complete.** is a phase boundary, not the end of the task.');
|
||||||
expect(step(7)).toContain('Do not end your turn after it: when its result arrives, go straight on to step 8.');
|
expect(step(6)).toContain('Do not end your turn after it: when its result arrives, go straight on to step 7.');
|
||||||
expect(step(8).startsWith("8. As your next assistant message, in the same turn as step 7's result (no user input is coming), Read the file")).toBe(true);
|
expect(step(7).startsWith("7. As your next assistant message, in the same turn as step 6's result (no user input is coming), Read the file")).toBe(true);
|
||||||
// The publication and the Phase 2 entry stay in separate messages: the guard's boundary is unchanged.
|
// The publication and the Phase 2 entry stay in separate messages: the guard's boundary is unchanged.
|
||||||
expect(step(7)).toContain('make exactly one tool call in that same message: the Bash command `true autoplan-published ceo`');
|
expect(step(6)).toContain('make exactly one tool call in that same message: the Bash command `true autoplan-published ceo`');
|
||||||
for (const n of [6, 7, 8, 9]) expect(step(n)).not.toMatch(/end your turn(?! after it)|wait for (its|a) completion/i);
|
for (const n of [5, 6, 7]) expect(step(n)).not.toMatch(/end your turn(?! after it)|wait for (its|a) completion/i);
|
||||||
} finally { s.cleanup(); }
|
} finally { s.cleanup(); }
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -84,3 +86,93 @@ test('a dispatch_prompt deny names the bytes the actor changed, read from the jo
|
|||||||
expect(dispatchPromptDelta(design, design)).toBe(`identical (${design.length} chars)`);
|
expect(dispatchPromptDelta(design, design)).toBe(`identical (${design.length} chars)`);
|
||||||
} finally { s.cleanup(); }
|
} finally { s.cleanup(); }
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// PR gate 41247097050 (paid-slice-1-a1), #3646's delta: "Design review: expected 977 chars, sent 993; first difference at
|
||||||
|
// offset 875": the Haiku actor kept the dispatch's contentHash to "…1c8f8c" and invented an 80-hex tail. The actor now sends
|
||||||
|
// a declared token and the driver's relay puts the exact dispatch in its place before Claude Code sees the tool call.
|
||||||
|
test('the dispatch relay hands the guard the exact dispatch for the declared token, and passes a retyped dispatch through to its denial', async () => {
|
||||||
|
const f = guardFixture('ceo'), retypedFixture = guardFixture('ceo');
|
||||||
|
const artifacts = fs.mkdtempSync(path.join(os.tmpdir(), 'ap-guard-relay-'));
|
||||||
|
const exact = f.snapshot.nativeDispatchPrompt, hash = retypedFixture.snapshot.sha256;
|
||||||
|
const retyped = retypedFixture.snapshot.nativeDispatchPrompt.replace(`INPUT: ceo ${hash}`, `INPUT: ceo ${hash.slice(0, 46)}65b428cec49f80716ab6f3015701${'0a'.repeat(3)}`);
|
||||||
|
fs.writeFileSync(path.join(artifacts, 'dispatch-ceo.txt'), exact);
|
||||||
|
const agent = (id: string, prompt: string) => ({ description: 'CEO review', subagent_type: 'general-purpose', prompt, run_in_background: false, _id: id });
|
||||||
|
const blocks = [agent('toolu_token', dispatchToken('ceo')), agent('toolu_retyped', retyped), { _id: 'toolu_bash', command: dispatchToken('ceo') }];
|
||||||
|
const sse = (data: Record<string, unknown>) => `event: ${data.type}\ndata: ${JSON.stringify(data)}\n\n`;
|
||||||
|
const stream = [sse({ type: 'message_start', message: { id: 'msg_1', role: 'assistant', content: [] } }),
|
||||||
|
...blocks.flatMap(({ _id, ...input }, index) => {
|
||||||
|
const json = JSON.stringify(input), name = 'command' in input ? 'Bash' : 'Agent';
|
||||||
|
return [sse({ type: 'content_block_start', index, content_block: { type: 'tool_use', id: _id, name, input: {} } }),
|
||||||
|
...Array.from({ length: Math.ceil(json.length / 37) }, (_, i) => sse({ type: 'content_block_delta', index,
|
||||||
|
delta: { type: 'input_json_delta', partial_json: json.slice(i * 37, i * 37 + 37) } })),
|
||||||
|
sse({ type: 'content_block_stop', index })];
|
||||||
|
}),
|
||||||
|
sse({ type: 'message_delta', delta: { stop_reason: 'tool_use' } }), sse({ type: 'message_stop' })].join('');
|
||||||
|
const seen: Array<{ path: string; key: string | null; host: string | null; body: string }> = [];
|
||||||
|
const upstream = Bun.serve({ hostname: '127.0.0.1', port: 0, async fetch(req) {
|
||||||
|
const body = await req.text(), url = new URL(req.url);
|
||||||
|
seen.push({ path: url.pathname + url.search, key: req.headers.get('x-api-key'), host: req.headers.get('host'), body });
|
||||||
|
if (!JSON.parse(body).stream) return Response.json({ id: 'msg_2', content: [{ type: 'tool_use', id: 'toolu_json', name: 'Agent', input: agent('toolu_json', dispatchToken('ceo')) }] });
|
||||||
|
const bytes = new TextEncoder().encode(stream);
|
||||||
|
return new Response(new ReadableStream({ start(sink) { for (let i = 0; i < bytes.length; i += 7) sink.enqueue(bytes.slice(i, i + 7)); sink.close(); } }),
|
||||||
|
{ headers: { 'content-type': 'text/event-stream' } });
|
||||||
|
} });
|
||||||
|
const relay = startDispatchRelay(artifacts, `http://127.0.0.1:${upstream.port}/`);
|
||||||
|
try {
|
||||||
|
const send = (stream: boolean) => fetch(`${relay.url}/v1/messages?beta=true`, { method: 'POST',
|
||||||
|
headers: { 'content-type': 'application/json', 'x-api-key': 'fixture-key' }, body: JSON.stringify({ model: 'm', stream, messages: history }) });
|
||||||
|
// The actor's history after its CEO dispatch, and a reviewer's first turn, whose dispatch is user text.
|
||||||
|
const history = [{ role: 'assistant', content: [{ type: 'tool_use', id: 'toolu_ceo', name: 'Agent', input: { description: 'CEO review', prompt: exact } },
|
||||||
|
{ type: 'tool_use', id: 'toolu_other', name: 'Agent', input: { description: 'CEO review', prompt: retyped } }] },
|
||||||
|
{ role: 'user', content: [{ type: 'text', text: exact }] }];
|
||||||
|
const text = await (await send(true)).text();
|
||||||
|
const events = text.split('\n\n').filter(Boolean).map(e => JSON.parse(/^data: (.*)$/m.exec(e)![1]!));
|
||||||
|
const inputs = blocks.map((_, index) => JSON.parse(events.filter(e => e.index === index && e.delta?.type === 'input_json_delta').map(e => e.delta.partial_json).join('')));
|
||||||
|
expect(events.map(e => e.type).at(-1)).toBe('message_stop');
|
||||||
|
expect(inputs[0]).toEqual({ description: 'CEO review', subagent_type: 'general-purpose', prompt: exact, run_in_background: false });
|
||||||
|
expect(inputs[1].prompt).toBe(retyped);
|
||||||
|
expect(inputs[2]).toEqual({ command: dispatchToken('ceo') });
|
||||||
|
expect((await (await send(false)).json() as any).content[0].input.prompt).toBe(exact);
|
||||||
|
expect(relay.substituted).toEqual(['ceo', 'ceo']);
|
||||||
|
expect(seen.map(s => [s.path, s.key, s.host === new URL(relay.url).host])).toEqual(Array(2).fill(['/v1/messages?beta=true', 'fixture-key', false]));
|
||||||
|
// Trial 1 of the first relay panel: shown the substituted CEO dispatch as its own past call, the actor imitated it for
|
||||||
|
// Design. Upstream sees the actor's own token in its history; any other prompt and the reviewer's dispatch are kept.
|
||||||
|
const upstreamHistory = JSON.parse(seen[0]!.body).messages;
|
||||||
|
expect(upstreamHistory[0].content.map((b: any) => b.input.prompt)).toEqual([dispatchToken('ceo'), retyped]);
|
||||||
|
expect(upstreamHistory[1].content[0].text).toBe(exact);
|
||||||
|
|
||||||
|
// The real guard on each relayed dispatch: the token's substitute is allowed; the retyped one is still denied.
|
||||||
|
const { run_in_background: _drop, ...payload } = inputs[0];
|
||||||
|
f.use('review', 'Agent', inputs[0]); f.journal();
|
||||||
|
expect(await f.hook(f.input('review', 'Agent', payload))).toEqual({});
|
||||||
|
expect(f.log().at(-1)).toMatchObject({ decision: 'allow', disposition: 'allow' });
|
||||||
|
const { run_in_background: _keep, ...retypedPayload } = inputs[1];
|
||||||
|
retypedFixture.use('review', 'Agent', inputs[1]); retypedFixture.journal();
|
||||||
|
expect(codeOf(await retypedFixture.hook(retypedFixture.input('review', 'Agent', retypedPayload)))).toBe('dispatch_prompt');
|
||||||
|
expect(dispatchPromptDelta(retypedFixture.snapshot.nativeDispatchPrompt, retyped)).toContain(`first difference at offset ${retyped.indexOf(`INPUT: ceo ${hash.slice(0, 46)}`) + 'INPUT: ceo '.length + 46}`);
|
||||||
|
} finally {
|
||||||
|
relay.stop(); upstream.stop(true);
|
||||||
|
f.cleanup(); retypedFixture.cleanup(); fs.rmSync(artifacts, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// /ship outside voice on PR #3953: a CRLF stream whose \r and \n land in separate chunks kept the token unsubstituted.
|
||||||
|
test('the event rewriter substitutes the token however LF or CRLF delimiters are split across chunks', () => {
|
||||||
|
const artifacts = fs.mkdtempSync(path.join(os.tmpdir(), 'ap-guard-crlf-'));
|
||||||
|
try {
|
||||||
|
fs.writeFileSync(path.join(artifacts, 'dispatch-ceo.txt'), 'exact dispatch');
|
||||||
|
const json = JSON.stringify({ description: 'CEO review', prompt: dispatchToken('ceo') });
|
||||||
|
const lf = [{ type: 'content_block_start', index: 0, content_block: { type: 'tool_use', id: 't', name: 'Agent', input: {} } },
|
||||||
|
{ type: 'content_block_delta', index: 0, delta: { type: 'input_json_delta', partial_json: json } },
|
||||||
|
{ type: 'content_block_stop', index: 0 }].map(e => `event: ${e.type}\ndata: ${JSON.stringify(e)}\n\n`).join('');
|
||||||
|
for (const stream of [lf, lf.replace(/\n/g, '\r\n')]) for (const size of [1, 2, 3, 7, stream.length]) {
|
||||||
|
const substituted: string[] = [];
|
||||||
|
const rewrite = dispatchEventRewriter(artifacts, phase => substituted.push(phase));
|
||||||
|
let out = '';
|
||||||
|
for (let i = 0; i < stream.length; i += size) out += rewrite.push(stream.slice(i, i + size));
|
||||||
|
out += rewrite.end();
|
||||||
|
expect({ crlf: stream !== lf, size, substituted }).toEqual({ crlf: stream !== lf, size, substituted: ['ceo'] });
|
||||||
|
expect(out).toContain(JSON.stringify(JSON.stringify({ description: 'CEO review', prompt: 'exact dispatch' })).slice(1, -1));
|
||||||
|
}
|
||||||
|
} finally { fs.rmSync(artifacts, { recursive: true, force: true }); }
|
||||||
|
});
|
||||||
@@ -41,5 +41,26 @@
|
|||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
},
|
||||||
|
"bulletFreeDescriptions41470212877": {
|
||||||
|
"source": "E2E Evals 41470212877 (train gate at 86bb621f), paid-slice-6-a1, observation.json pendingQuestion",
|
||||||
|
"cliVersion": "2.1.292",
|
||||||
|
"toolUseId": "toolu_01MsiErLVUJGvMQ5DFQf7j2T",
|
||||||
|
"note": "The native question carries the Pros / cons bullets in its text and drops every ✅/❌ bullet from the option descriptions; the declared menu otherwise matches byte for byte.",
|
||||||
|
"question": {
|
||||||
|
"question": "D3 — Want outside design voices before the detailed review?\nProject/branch/task: plan-design-review of review-input.md (User Dashboard) on main.\nELI10: Before I walk the 7 passes, I can get second opinions. Codex scores the plan against hard design rules and litmus checks; a separate Claude subagent does a fresh completeness read without seeing my review. Disagreements get surfaced in the relevant pass.\nStakes if we pick wrong: skipping means findings rest on one reviewer; running adds a few minutes and may only get one voice.\nRecommendation: B because mockups were unavailable and the plan's gaps are clear enough that one rigorous pass will catch them — this is close to a taste call.\nNote: options differ in kind, not coverage — no completeness score.\nPros / cons:\nA) Yes, run outside design voices\n ✅ Codex hard-rule + litmus scorecard plus an independent Claude completeness read.\n ✅ Disagreements surface with both perspectives in the relevant pass.\n ❌ Adds a few minutes; Codex may be missing here, in which case only the Claude voice runs and it's tagged single-model.\nB) No, proceed without (recommended)\n ✅ Straight into the 7 passes with the approved mockup as anchor.\n ✅ Fewest steps to a fixed plan.\n ❌ Findings rest on one reviewer; no cross-model check on slop risk or hierarchy.\nNet: a cross-model check versus the fastest path into the passes.",
|
||||||
|
"header": "Voices",
|
||||||
|
"options": [
|
||||||
|
{
|
||||||
|
"label": "Yes, run outside design voices",
|
||||||
|
"description": "Codex hard-rule + litmus scorecard plus an independent Claude completeness read. Disagreements surface with both perspectives in the relevant pass. Adds a few minutes; Codex may be missing here, in which case only the Claude voice runs and it's tagged single-model."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "No, proceed without (recommended)",
|
||||||
|
"description": "Straight into the 7 passes with the approved mockup as anchor. Fewest steps to a fixed plan. Findings rest on one reviewer; no cross-model check on slop risk or hierarchy."
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"multiSelect": false
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -5,7 +5,8 @@
|
|||||||
* carries /autoplan's own PreToolUse hook block and allowed tools. Its body is
|
* carries /autoplan's own PreToolUse hook block and allowed tools. Its body is
|
||||||
* a minimal scripted phase boundary: init, Phase 1 entry, a prepared close
|
* a minimal scripted phase boundary: init, Phase 1 entry, a prepared close
|
||||||
* packet, the report published with the `true autoplan-published` no-op, then the
|
* packet, the report published with the `true autoplan-published` no-op, then the
|
||||||
* Phase 2 entry in a later message and the Phase 2 reviewer. The restore point
|
* Phase 2 entry in a later message and the Phase 2 reviewer. The actor sends each reviewer dispatch as a declared
|
||||||
|
* token that startDispatchRelay replaces with the dispatch's exact bytes, so no actor retypes it. The restore point
|
||||||
* and phase artifacts live in the project's `.gstack/tmp/autoplan/`, as /autoplan
|
* and phase artifacts live in the project's `.gstack/tmp/autoplan/`, as /autoplan
|
||||||
* Step 1 places them. Outcomes come from the guard decision log and the journals.
|
* Step 1 places them. Outcomes come from the guard decision log and the journals.
|
||||||
*/
|
*/
|
||||||
@@ -32,7 +33,7 @@ export interface GuardSession {
|
|||||||
readLog(): Array<Record<string, any>>;
|
readLog(): Array<Record<string, any>>;
|
||||||
/** Content-bounded journal summary for a failure message: tool names, inputs' first keys, result heads. */
|
/** Content-bounded journal summary for a failure message: tool names, inputs' first keys, result heads. */
|
||||||
journalSummary(): string[];
|
journalSummary(): string[];
|
||||||
/** Each parent Agent dispatch compared with the dispatch-<phase>.txt the actor had to copy, read before cleanup. */
|
/** Each parent Agent dispatch compared with the dispatch-<phase>.txt the relay substitutes, read before cleanup. */
|
||||||
dispatchPromptDeltas(): string[];
|
dispatchPromptDeltas(): string[];
|
||||||
/** Parent tool calls from the first background completion notice on, the notice count, and reviewers' successful Read paths. */
|
/** Parent tool calls from the first background completion notice on, the notice count, and reviewers' successful Read paths. */
|
||||||
noticeEvidence(): { notices: number; afterNotice: string[]; reviewerReads: string[] };
|
noticeEvidence(): { notices: number; afterNotice: string[]; reviewerReads: string[] };
|
||||||
@@ -50,7 +51,7 @@ export function stoppedAtPhaseBoundary(screen: string): boolean {
|
|||||||
if (!marks.length) return false;
|
if (!marks.length) return false;
|
||||||
const last = marks[marks.length - 1]!;
|
const last = marks[marks.length - 1]!;
|
||||||
const after = screen.slice(last.index! + last[0].length);
|
const after = screen.slice(last.index! + last[0].length);
|
||||||
return /✻[^\n]*\bdone\b/.test(after) && !/Step\s*8|design-phase|Design\s*review/i.test(after);
|
return /✻[^\n]*\bdone\b/.test(after) && !/Step\s*7|design-phase|Design\s*review/i.test(after);
|
||||||
}
|
}
|
||||||
|
|
||||||
/** The close packet Read (step 6) came after the CEO reviewer's completion notice, so it ran in the notice's new turn. */
|
/** The close packet Read (step 6) came after the CEO reviewer's completion notice, so it ran in the notice's new turn. */
|
||||||
@@ -68,6 +69,122 @@ export function dispatchPromptDelta(expected: string, actual: string): string {
|
|||||||
return `expected ${expected.length} chars, sent ${actual.length}; first difference at offset ${at}\n expected: ${around(expected)}\n sent: ${around(actual)}`;
|
return `expected ${expected.length} chars, sent ${actual.length}; first difference at offset ${at}\n expected: ${around(expected)}\n sent: ${around(actual)}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type DispatchPhase = 'ceo' | 'design';
|
||||||
|
/** The actor's declared dispatch interface: its Agent prompt is exactly this token, never a retyped dispatch. */
|
||||||
|
export const dispatchToken = (phase: DispatchPhase): string => `DISPATCH ${phase}`;
|
||||||
|
const TOKEN_NOTE = "the test driver puts this phase's exact reviewer dispatch in place of that text before the tool runs; " +
|
||||||
|
'send it as written and never read, copy or retype the dispatch yourself';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* PR gate 41247097050 (#3646's delta): the Haiku actor retyped the Design dispatch's 64-hex contentHash and the guard
|
||||||
|
* rightly denied it. The actor is a fixture, so it never copies the dispatch: an Agent tool call whose prompt is exactly
|
||||||
|
* a declared token gets dispatch-<phase>.txt's bytes in its place. Anything else passes unchanged for the guard to judge.
|
||||||
|
*/
|
||||||
|
export function substituteDispatch(block: any, artifacts: string): DispatchPhase | undefined {
|
||||||
|
if (block?.type !== 'tool_use' || block.name !== 'Agent') return;
|
||||||
|
const phase = (['ceo', 'design'] as const).find(p => block.input?.prompt === dispatchToken(p));
|
||||||
|
if (!phase || !fs.existsSync(path.join(artifacts, `dispatch-${phase}.txt`))) return;
|
||||||
|
block.input.prompt = fs.readFileSync(path.join(artifacts, `dispatch-${phase}.txt`), 'utf8');
|
||||||
|
return phase;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The request direction (trial 1 of GSTA-262's first panel): with the substituted CEO dispatch in its own history, the
|
||||||
|
* actor imitated it for Design instead of sending its token. Each request shows the actor its own declared call instead.
|
||||||
|
* A reviewer's dispatch arrives as user text, not an Agent tool call, so it is untouched.
|
||||||
|
*/
|
||||||
|
export function restoreDispatchTokens(request: any, artifacts: string): void {
|
||||||
|
const sent = new Map((['ceo', 'design'] as const).flatMap(phase => {
|
||||||
|
const file = path.join(artifacts, `dispatch-${phase}.txt`);
|
||||||
|
return fs.existsSync(file) ? [[fs.readFileSync(file, 'utf8'), dispatchToken(phase)] as const] : [];
|
||||||
|
}));
|
||||||
|
for (const message of Array.isArray(request?.messages) ? request.messages : [])
|
||||||
|
for (const block of Array.isArray(message?.content) ? message.content : [])
|
||||||
|
if (block?.type === 'tool_use' && block.name === 'Agent' && sent.has(block.input?.prompt)) block.input.prompt = sent.get(block.input.prompt);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Rewrites one Messages API event stream: an Agent tool_use block is held until its stop, then sent as one input delta
|
||||||
|
* carrying the substituted input, or as its original events when no token matched.
|
||||||
|
*/
|
||||||
|
export function dispatchEventRewriter(artifacts: string, substituted: (phase: DispatchPhase) => void) {
|
||||||
|
let pending = '';
|
||||||
|
let held: { index: number; block: any; json: string; raw: string[] } | null = null;
|
||||||
|
const frame = (data: Record<string, unknown>) => `event: ${data.type}\ndata: ${JSON.stringify(data)}\n\n`;
|
||||||
|
const take = (raw: string): string => {
|
||||||
|
const data = /^data: (.*)$/m.exec(raw)?.[1];
|
||||||
|
const event = data ? JSON.parse(data) : undefined;
|
||||||
|
if (event?.type === 'content_block_start' && event.content_block?.type === 'tool_use' && event.content_block.name === 'Agent') {
|
||||||
|
held = { index: event.index, block: event.content_block, json: '', raw: [raw] };
|
||||||
|
return '';
|
||||||
|
}
|
||||||
|
if (!held || event?.index !== held.index) return raw;
|
||||||
|
held.raw.push(raw);
|
||||||
|
if (event.type === 'content_block_delta' && event.delta?.type === 'input_json_delta') held.json += event.delta.partial_json;
|
||||||
|
if (event.type !== 'content_block_stop') return '';
|
||||||
|
const { index, block, json, raw: original } = held;
|
||||||
|
held = null;
|
||||||
|
let input: unknown;
|
||||||
|
try { input = JSON.parse(json || '{}'); } catch { return original.join(''); }
|
||||||
|
const rewritten = { ...block, input };
|
||||||
|
const phase = substituteDispatch(rewritten, artifacts);
|
||||||
|
if (!phase) return original.join('');
|
||||||
|
substituted(phase);
|
||||||
|
return frame({ type: 'content_block_start', index, content_block: { ...block, input: {} } }) +
|
||||||
|
frame({ type: 'content_block_delta', index, delta: { type: 'input_json_delta', partial_json: JSON.stringify(rewritten.input) } }) + raw;
|
||||||
|
};
|
||||||
|
return {
|
||||||
|
push(chunk: string): string {
|
||||||
|
pending = (pending + chunk).replace(/\r\n/g, '\n');
|
||||||
|
const events = pending.split('\n\n');
|
||||||
|
pending = events.pop()!;
|
||||||
|
return events.map(e => take(e + '\n\n')).join('');
|
||||||
|
},
|
||||||
|
end: (): string => { const rest = (held ? held.raw.join('') : '') + pending; held = null; pending = ''; return rest; },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DispatchRelay { url: string; substituted: DispatchPhase[]; stop(): void }
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A loopback relay to the Messages API for the live case (point ANTHROPIC_BASE_URL at url). Headers pass through and are
|
||||||
|
* never logged; only an Agent tool call carrying a dispatch token, and that call in later requests' history, is rewritten.
|
||||||
|
*/
|
||||||
|
export function startDispatchRelay(artifacts: string, upstream = process.env.ANTHROPIC_BASE_URL || 'https://api.anthropic.com'): DispatchRelay {
|
||||||
|
const substituted: DispatchPhase[] = [];
|
||||||
|
const server = Bun.serve({
|
||||||
|
hostname: '127.0.0.1', port: 0, idleTimeout: 0,
|
||||||
|
async fetch(req) {
|
||||||
|
const url = new URL(req.url), headers = new Headers(req.headers);
|
||||||
|
headers.delete('host');
|
||||||
|
headers.set('accept-encoding', 'identity');
|
||||||
|
let body = req.method === 'GET' || req.method === 'HEAD' ? undefined : await req.text();
|
||||||
|
if (body && url.pathname.endsWith('/messages')) {
|
||||||
|
try { const request = JSON.parse(body); restoreDispatchTokens(request, artifacts); body = JSON.stringify(request); } catch { /* not JSON: forward as sent */ }
|
||||||
|
}
|
||||||
|
const res = await fetch(upstream.replace(/\/+$/, '') + url.pathname + url.search, { method: req.method, headers, body });
|
||||||
|
const out = new Headers(res.headers);
|
||||||
|
out.delete('content-encoding');
|
||||||
|
out.delete('content-length');
|
||||||
|
const type = res.headers.get('content-type') ?? '';
|
||||||
|
if (!url.pathname.endsWith('/messages') || !res.body) return new Response(res.body, { status: res.status, headers: out });
|
||||||
|
if (type.includes('application/json')) {
|
||||||
|
const message = await res.json() as { content?: unknown[] };
|
||||||
|
for (const block of message.content ?? []) { const phase = substituteDispatch(block, artifacts); if (phase) substituted.push(phase); }
|
||||||
|
return new Response(JSON.stringify(message), { status: res.status, headers: out });
|
||||||
|
}
|
||||||
|
if (!type.includes('text/event-stream')) return new Response(res.body, { status: res.status, headers: out });
|
||||||
|
const rewrite = dispatchEventRewriter(artifacts, phase => substituted.push(phase));
|
||||||
|
const decoder = new TextDecoder(), encoder = new TextEncoder();
|
||||||
|
return new Response(res.body.pipeThrough(new TransformStream<Uint8Array, Uint8Array>({
|
||||||
|
transform(chunk, sink) { const text = rewrite.push(decoder.decode(chunk, { stream: true })); if (text) sink.enqueue(encoder.encode(text)); },
|
||||||
|
flush(sink) { const text = rewrite.push(decoder.decode()) + rewrite.end(); if (text) sink.enqueue(encoder.encode(text)); },
|
||||||
|
})), { status: res.status, headers: out });
|
||||||
|
},
|
||||||
|
});
|
||||||
|
return { url: `http://127.0.0.1:${server.port}`, substituted, stop: () => { server.stop(true); } };
|
||||||
|
}
|
||||||
|
|
||||||
export function prepareGuardSession(apiKey: string | undefined): GuardSession {
|
export function prepareGuardSession(apiKey: string | undefined): GuardSession {
|
||||||
const base = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'ap-guard-pty-')));
|
const base = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'ap-guard-pty-')));
|
||||||
const home = path.join(base, 'home'), config = path.join(home, '.claude'), project = path.join(base, 'project');
|
const home = path.join(base, 'home'), config = path.join(home, '.claude'), project = path.join(base, 'project');
|
||||||
@@ -102,28 +219,26 @@ console.log(close);
|
|||||||
const tool = path.join(ROOT, 'bin', 'gstack-autoplan-snapshot.ts');
|
const tool = path.join(ROOT, 'bin', 'gstack-autoplan-snapshot.ts');
|
||||||
const sections = path.join(config, 'skills', 'gstack', 'autoplan', 'sections');
|
const sections = path.join(config, 'skills', 'gstack', 'autoplan', 'sections');
|
||||||
const body = [
|
const body = [
|
||||||
'Run this scripted phase boundary exactly, from step 1 through step 11 in one continuous run. Each numbered step is its own ' +
|
'Run this scripted phase boundary exactly, from step 1 through step 9 in one continuous run. Each numbered step is its own ' +
|
||||||
'assistant message; do nothing else. End your turn before step 11 only where step 5 or step 10 says to wait for a ' +
|
'assistant message; do nothing else. End your turn before step 9 only where step 4 or step 8 says to wait for a ' +
|
||||||
'background agent. **Phase 1 complete.** is a phase boundary, not the end of the task.',
|
'background agent. **Phase 1 complete.** is a phase boundary, not the end of the task.',
|
||||||
`1. Run this Bash command alone: bun "${tool}" init "${source}" "${active}" "${restore}"`,
|
`1. Run this Bash command alone: bun "${tool}" init "${source}" "${active}" "${restore}"`,
|
||||||
`2. Read the file ${path.join(sections, 'ceo-phase.md')} with offset 1 and limit 20.`,
|
`2. Read the file ${path.join(sections, 'ceo-phase.md')} with offset 1 and limit 20.`,
|
||||||
`3. Run this Bash command alone: bun "${review}"`,
|
`3. Run this Bash command alone: bun "${review}"`,
|
||||||
`4. Run this Bash command alone: cat "${path.join(artifacts, 'dispatch-ceo.txt')}"`,
|
`4. Call the Agent tool once with these inputs: prompt = exactly the text \`${dispatchToken('ceo')}\` (${TOKEN_NOTE}), ` +
|
||||||
'5. Call the Agent tool once with these inputs: prompt = the exact, complete output of step 4 (every line, unchanged), ' +
|
|
||||||
'description "CEO review", subagent_type "general-purpose", and run_in_background false. Include run_in_background even ' +
|
'description "CEO review", subagent_type "general-purpose", and run_in_background false. Include run_in_background even ' +
|
||||||
'if the tool schema lacks it. If its result says the agent is running in the background, end your turn at once with no ' +
|
'if the tool schema lacks it. If its result says the agent is running in the background, end your turn at once with no ' +
|
||||||
'further tool call; the completion notification starts your next turn.',
|
'further tool call; the completion notification starts your next turn.',
|
||||||
'6. Only after the CEO review agent has finished (in the turn its completion notification starts, when it ran in the ' +
|
'5. Only after the CEO review agent has finished (in the turn its completion notification starts, when it ran in the ' +
|
||||||
'background), Read the file whose path step 3 printed, completely (no offset or limit).',
|
'background), Read the file whose path step 3 printed, completely (no offset or limit).',
|
||||||
'7. In one message, write exactly this line as your own text: **Phase 1 complete.** and make exactly one tool call in that ' +
|
'6. In one message, write exactly this line as your own text: **Phase 1 complete.** and make exactly one tool call in that ' +
|
||||||
'same message: the Bash command `true autoplan-published ceo`. Do not end your turn after it: when its result arrives, go ' +
|
'same message: the Bash command `true autoplan-published ceo`. Do not end your turn after it: when its result arrives, go ' +
|
||||||
'straight on to step 8.',
|
'straight on to step 7.',
|
||||||
`8. As your next assistant message, in the same turn as step 7's result (no user input is coming), Read the file ${path.join(sections, 'design-phase.md')} with offset 1 and limit 20.`,
|
`7. As your next assistant message, in the same turn as step 6's result (no user input is coming), Read the file ${path.join(sections, 'design-phase.md')} with offset 1 and limit 20.`,
|
||||||
`9. Run this Bash command alone: cat "${path.join(artifacts, 'dispatch-design.txt')}"`,
|
`8. Call the Agent tool once with these inputs: prompt = exactly the text \`${dispatchToken('design')}\` (${TOKEN_NOTE}), ` +
|
||||||
'10. Call the Agent tool once with these inputs: prompt = the exact, complete output of step 9 (every line, unchanged), ' +
|
|
||||||
'description "Design review", subagent_type "general-purpose". If the agent runs in the background, end your turn and wait ' +
|
'description "Design review", subagent_type "general-purpose". If the agent runs in the background, end your turn and wait ' +
|
||||||
'for its completion notification.',
|
'for its completion notification.',
|
||||||
'11. Reply with the single word DONE. If any tool call is denied, quote the denial text and stop.',
|
'9. Reply with the single word DONE. If any tool call is denied, quote the denial text and stop.',
|
||||||
].join('\n');
|
].join('\n');
|
||||||
const skillDir = path.join(project, '.claude', 'skills', GUARD_SKILL);
|
const skillDir = path.join(project, '.claude', 'skills', GUARD_SKILL);
|
||||||
fs.mkdirSync(skillDir, { recursive: true });
|
fs.mkdirSync(skillDir, { recursive: true });
|
||||||
|
|||||||
@@ -35,7 +35,8 @@ export const DESIGN_BOARD_ACTOR_PROTOCOL = [
|
|||||||
'If offering that optional step, ask one single-select question with header "Voices",',
|
'If offering that optional step, ask one single-select question with header "Voices",',
|
||||||
'beginning "Want outside design voices before the detailed review?" and exactly:',
|
'beginning "Want outside design voices before the detailed review?" and exactly:',
|
||||||
...DESIGN_OUTSIDE_VOICE_OPTIONS.map(option => `- ${option.label}: ${option.description}`),
|
...DESIGN_OUTSIDE_VOICE_OPTIONS.map(option => `- ${option.label}: ${option.description}`),
|
||||||
'Option order, a letter prefix, and a trailing (recommended) marker may vary.',
|
'Option order, a letter prefix, a trailing (recommended) marker, and',
|
||||||
|
'keeping or dropping every ✅/❌ bullet in a description may vary.',
|
||||||
'Keep this choice separate from other approvals; no previews or extra actions.',
|
'Keep this choice separate from other approvals; no previews or extra actions.',
|
||||||
'The fixture user chooses No, proceed without. Observe the actual native',
|
'The fixture user chooses No, proceed without. Observe the actual native',
|
||||||
'answer before continuing when this question is offered.',
|
'answer before continuing when this question is offered.',
|
||||||
@@ -119,7 +120,8 @@ export function createDesignReviewPicker({ cwd, deadlineAt }: { cwd: string; dea
|
|||||||
|| labels.includes(DESIGN_OUTSIDE_VOICE_OPTIONS[0].label);
|
|| labels.includes(DESIGN_OUTSIDE_VOICE_OPTIONS[0].label);
|
||||||
if (claimsVoices) {
|
if (claimsVoices) {
|
||||||
const voiceActions = question.options.map((option, index) => DESIGN_OUTSIDE_VOICE_OPTIONS.findIndex(expected =>
|
const voiceActions = question.options.map((option, index) => DESIGN_OUTSIDE_VOICE_OPTIONS.findIndex(expected =>
|
||||||
labels[index] === expected.label && option.description === expected.description && option.preview === undefined));
|
labels[index] === expected.label && option.preview === undefined
|
||||||
|
&& [expected.description, expected.description.replace(/(^| )[✅❌] /g, '$1')].includes(option.description)));
|
||||||
if (question.header.trim() !== 'Voices'
|
if (question.header.trim() !== 'Voices'
|
||||||
|| !/^(?:Want outside design voices before the detailed review|Run outside design voices before the 7 passes)\?(?:\s|$)/i.test(lead)
|
|| !/^(?:Want outside design voices before the detailed review|Run outside design voices before the 7 passes)\?(?:\s|$)/i.test(lead)
|
||||||
|| question.multiSelect || voiceActions.length !== 2 || voiceActions.includes(-1)
|
|| question.multiSelect || voiceActions.length !== 2 || voiceActions.includes(-1)
|
||||||
|
|||||||
@@ -115,6 +115,33 @@ test('the native-only actor declares its scope and declines the complete capture
|
|||||||
await expectUnsubmitted(published);
|
await expectUnsubmitted(published);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Train gate 41470212877 (86bb621f, paid-slice-6-a1, passed on the in-run rerun): the native question put the Pros / cons
|
||||||
|
// bullets in its text and dropped every ✅/❌ bullet from the option descriptions; the 2.1.251 and #77 captures kept them.
|
||||||
|
// Both are readings of the format's "descriptions use literal ✅/❌ bullets", so the protocol declares both spellings.
|
||||||
|
test('the native-only actor declines the declared menu with every ✅/❌ bullet or none, and refuses anything in between', async () => {
|
||||||
|
const published = await board();
|
||||||
|
const bulletFree = structuredClone(outsideVoices.bulletFreeDescriptions41470212877.question) as NativeQuestion;
|
||||||
|
expect(bulletFree.options.map(option => option.description).join(' ')).not.toMatch(/[✅❌]/);
|
||||||
|
expect(picker()(bulletFree)).toBe(2);
|
||||||
|
expect(DESIGN_BOARD_ACTOR_PROTOCOL).toContain('keeping or dropping every ✅/❌ bullet in a description may vary');
|
||||||
|
const changes: ((menu: NativeQuestion) => void)[] = [
|
||||||
|
menu => { menu.options[0]!.description = '✅ ' + menu.options[0]!.description; },
|
||||||
|
menu => { menu.options[1]!.description = menu.options[1]!.description.replace('Findings rest', '❌ Findings rest'); },
|
||||||
|
menu => { [menu.options[0]!.description, menu.options[1]!.description] = [menu.options[1]!.description, menu.options[0]!.description]; },
|
||||||
|
menu => { menu.options[1]!.description = menu.options[1]!.description.replace('Fewest steps', 'Fewer steps'); },
|
||||||
|
menu => { menu.options[1]!.description += ' Also approve the plan.'; },
|
||||||
|
];
|
||||||
|
for (const change of changes) {
|
||||||
|
const menu = structuredClone(bulletFree);
|
||||||
|
change(menu);
|
||||||
|
expect(() => picker()(menu)).toThrow('no unambiguous declared native-only action');
|
||||||
|
}
|
||||||
|
const flipped = structuredClone(outsideVoices.question) as NativeQuestion;
|
||||||
|
flipped.options[1]!.description = flipped.options[1]!.description.replace('❌', '✅');
|
||||||
|
expect(() => picker()(flipped)).toThrow('no unambiguous declared native-only action');
|
||||||
|
await expectUnsubmitted(published);
|
||||||
|
});
|
||||||
|
|
||||||
test('the native-only actor accepts the declared question with a same-line explanation', async () => {
|
test('the native-only actor accepts the declared question with a same-line explanation', async () => {
|
||||||
const published = await board();
|
const published = await board();
|
||||||
const retained = structuredClone(outsideVoices.sameLineExplanation77.question) as NativeQuestion;
|
const retained = structuredClone(outsideVoices.sameLineExplanation77.question) as NativeQuestion;
|
||||||
|
|||||||
@@ -3,7 +3,8 @@
|
|||||||
*
|
*
|
||||||
* On the pinned Claude Code, in an interactive PTY session with default
|
* On the pinned Claude Code, in an interactive PTY session with default
|
||||||
* settings, a project skill carrying /autoplan's own hook block enters Phase 1,
|
* settings, a project skill carrying /autoplan's own hook block enters Phase 1,
|
||||||
* dispatches the CEO reviewer, reads a prepared close packet, publishes the Phase 1 report in a message
|
* dispatches the CEO reviewer (each dispatch is a declared token the driver's relay replaces with the exact
|
||||||
|
* dispatch, so the actor never retypes it), reads a prepared close packet, publishes the Phase 1 report in a message
|
||||||
* whose only tool call is `true autoplan-published ceo`, enters Phase 2 in a
|
* whose only tool call is `true autoplan-published ceo`, enters Phase 2 in a
|
||||||
* later message and dispatches the Phase 2 reviewer. The reviewer dispatch sends run_in_background, which the
|
* later message and dispatches the Phase 2 reviewer. The reviewer dispatch sends run_in_background, which the
|
||||||
* 2.1.29x default fork-subagent schema strips. Claude Code 2.1.29x can write a tool call to the journal only
|
* 2.1.29x default fork-subagent schema strips. Claude Code 2.1.29x can write a tool call to the journal only
|
||||||
@@ -18,9 +19,10 @@
|
|||||||
* must still raise no permission card.
|
* must still raise no permission card.
|
||||||
*/
|
*/
|
||||||
import { expect } from 'bun:test';
|
import { expect } from 'bun:test';
|
||||||
|
import * as path from 'node:path';
|
||||||
import { describeIfSelected, testIfSelected } from './helpers/e2e-helpers';
|
import { describeIfSelected, testIfSelected } from './helpers/e2e-helpers';
|
||||||
import { launchClaudePty, selectPtyNumberedOption } from './helpers/claude-pty-runner';
|
import { launchClaudePty, selectPtyNumberedOption } from './helpers/claude-pty-runner';
|
||||||
import { closePacketReadAfterNotice, prepareGuardSession, stoppedAtPhaseBoundary } from './helpers/autoplan-guard-pty';
|
import { closePacketReadAfterNotice, prepareGuardSession, startDispatchRelay, stoppedAtPhaseBoundary } from './helpers/autoplan-guard-pty';
|
||||||
|
|
||||||
const MODEL = 'claude-haiku-4-5-20251001';
|
const MODEL = 'claude-haiku-4-5-20251001';
|
||||||
|
|
||||||
@@ -28,7 +30,9 @@ describeIfSelected('autoplan publication guard live PTY', ['autoplan-guard-pty']
|
|||||||
testIfSelected('autoplan-guard-pty', async () => {
|
testIfSelected('autoplan-guard-pty', async () => {
|
||||||
if (!process.env.ANTHROPIC_API_KEY) throw new Error('autoplan-guard-pty requires ANTHROPIC_API_KEY; refusing to skip');
|
if (!process.env.ANTHROPIC_API_KEY) throw new Error('autoplan-guard-pty requires ANTHROPIC_API_KEY; refusing to skip');
|
||||||
const s = prepareGuardSession(process.env.ANTHROPIC_API_KEY);
|
const s = prepareGuardSession(process.env.ANTHROPIC_API_KEY);
|
||||||
const session = await launchClaudePty({ permissionMode: null, model: MODEL, cwd: s.project, env: s.env, timeoutMs: 540_000, observeScreen: true });
|
const relay = startDispatchRelay(path.join(s.project, '.gstack', 'tmp', 'autoplan'));
|
||||||
|
const session = await launchClaudePty({ permissionMode: null, model: MODEL, cwd: s.project, env: { ...s.env, ANTHROPIC_BASE_URL: relay.url },
|
||||||
|
timeoutMs: 540_000, observeScreen: true }).catch(error => { relay.stop(); throw error; });
|
||||||
const started = performance.now();
|
const started = performance.now();
|
||||||
const prompts: string[] = [];
|
const prompts: string[] = [];
|
||||||
let phase1Ms: number | null = null;
|
let phase1Ms: number | null = null;
|
||||||
@@ -54,10 +58,10 @@ describeIfSelected('autoplan publication guard live PTY', ['autoplan-guard-pty']
|
|||||||
}
|
}
|
||||||
await Bun.sleep(3_000);
|
await Bun.sleep(3_000);
|
||||||
const log = s.readLog(), evidence = s.noticeEvidence();
|
const log = s.readLog(), evidence = s.noticeEvidence();
|
||||||
const summary = { prompts: prompts.length, phase1Ms, totalMs: Math.round(performance.now() - started), evidence,
|
const summary = { prompts: prompts.length, substituted: relay.substituted, phase1Ms, totalMs: Math.round(performance.now() - started), evidence,
|
||||||
decisions: log.map(e => ({ decision: e.decision, code: e.code, path: e.path, version: e.claude_code_version })) };
|
decisions: log.map(e => ({ decision: e.decision, code: e.code, path: e.path, version: e.claude_code_version })) };
|
||||||
console.log(`[autoplan-guard-pty] ${JSON.stringify(summary)}`);
|
console.log(`[autoplan-guard-pty] ${JSON.stringify(summary)}`);
|
||||||
if (log.length < 4 && stoppedAtPhaseBoundary(session.visibleText())) console.log('[autoplan-guard-pty] PHASE_BOUNDARY_STOP: the actor ended its turn after "Phase 1 complete." instead of going on to step 8');
|
if (log.length < 4 && stoppedAtPhaseBoundary(session.visibleText())) console.log('[autoplan-guard-pty] PHASE_BOUNDARY_STOP: the actor ended its turn after "Phase 1 complete." instead of going on to step 7');
|
||||||
if (log.length !== 4 || log.some(e => e.decision === 'deny') || prompts.length)
|
if (log.length !== 4 || log.some(e => e.decision === 'deny') || prompts.length)
|
||||||
console.log(`[autoplan-guard-pty] journal:\n${s.journalSummary().join('\n')}\n[autoplan-guard-pty] screen:\n${session.visibleText().slice(-2_000)}\n[autoplan-guard-pty] last non-blank screen:\n${session.lastScreen?.() ?? ''}`);
|
console.log(`[autoplan-guard-pty] journal:\n${s.journalSummary().join('\n')}\n[autoplan-guard-pty] screen:\n${session.visibleText().slice(-2_000)}\n[autoplan-guard-pty] last non-blank screen:\n${session.lastScreen?.() ?? ''}`);
|
||||||
if (log.some(e => e.code === 'dispatch_prompt')) console.log(`[autoplan-guard-pty] dispatch prompt deltas:\n${s.dispatchPromptDeltas().join('\n')}`);
|
if (log.some(e => e.code === 'dispatch_prompt')) console.log(`[autoplan-guard-pty] dispatch prompt deltas:\n${s.dispatchPromptDeltas().join('\n')}`);
|
||||||
@@ -74,6 +78,7 @@ describeIfSelected('autoplan publication guard live PTY', ['autoplan-guard-pty']
|
|||||||
expect(prompts).toEqual([]);
|
expect(prompts).toEqual([]);
|
||||||
} finally {
|
} finally {
|
||||||
await session.close();
|
await session.close();
|
||||||
|
relay.stop();
|
||||||
s.cleanup();
|
s.cleanup();
|
||||||
}
|
}
|
||||||
}, 600_000);
|
}, 600_000);
|
||||||
|
|||||||
Reference in new issue
Block a user