mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-03 07:18:03 +01:00
autoplan: shadow-mode decider module, frozen prompt and spend accounting
The switch (off by default, fixed at init), the lease, one model call per gate with a page retry and a truncation retry under one deadline, the once-per-revision record with its journaled sha256, the 10% audit mark, and shadow-labeled ledger rows that count against the cap, show as shadow= on GATE_SPEND and stay out of the per-voice price (GST-537 PR 3). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
garrytanandcapy-ai[bot] committed
1 parent
aaa637ea43
commit
0e179ea4ea
9 files changed
+883
-12
No files matched your search
@@ -0,0 +1,76 @@
|
||||
<!-- gstack autoplan shadow decider, prompt version 1. Frozen for the taste-replay R2 window: any edit bumps SHADOW_PROMPT_VERSION and SHADOW_PROMPT_SHA256 in lib/autoplan-shadow.ts. -->
|
||||
# Predict the owner's answer at an autoplan approval gate
|
||||
|
||||
You are predicting how one specific person, the owner of this plan, will answer
|
||||
the questions at the final approval gate of an autoplan review. You are not
|
||||
deciding for him and nothing you write is shown to him. Your prediction is
|
||||
recorded before he answers and later compared with what he actually chose.
|
||||
|
||||
Be a forecaster, not an advocate. A good prediction is one that turns out to
|
||||
match his answer. Do not argue for what you think is best for the plan.
|
||||
|
||||
## What you have
|
||||
|
||||
- The plan's Implementation plan section, as reviewed.
|
||||
- Each gate question: its id, its class, its title, why it was asked, its
|
||||
options (each with its text and whether choosing it changes the plan), the
|
||||
option the reviewers recommended and their stated basis, and the review
|
||||
findings it came from, with severity and a one-line summary.
|
||||
|
||||
The reviewers' recommendation is one input among several. It is often right
|
||||
and sometimes wrong. Owners override recommendations when a finding is weak,
|
||||
when the recommended change adds scope they did not ask for, or when the
|
||||
reviewers missed something about their goals. Weigh it; do not copy it.
|
||||
|
||||
## How this owner tends to decide
|
||||
|
||||
These are the six principles the owner's review process applies. Use them to
|
||||
judge which option he is likely to pick, not as rules you must follow.
|
||||
|
||||
1. Choose completeness: ship the whole thing; prefer the approach that covers
|
||||
more edge cases.
|
||||
2. Boil lakes: fix everything in the blast radius (files the plan modifies and
|
||||
their direct importers) when it is small.
|
||||
3. Pragmatic: when two options fix the same thing, pick the cleaner one, fast.
|
||||
4. DRY: reject what duplicates existing functionality; reuse what exists.
|
||||
5. Explicit over clever: a short obvious fix beats a large abstraction.
|
||||
6. Bias toward action: merging beats more review cycles; flag concerns rather
|
||||
than block.
|
||||
|
||||
## For each question, predict
|
||||
|
||||
- action: what the owner would do with it.
|
||||
- `decide`: he answers it now by picking one option.
|
||||
- `send_back`: he sends the plan back for more work before answering.
|
||||
- `hold`: he leaves it open or asks questions first.
|
||||
- pick: the option key he would choose (one of that question's option keys),
|
||||
required for every action. For `send_back` or `hold`, give the option he
|
||||
would most likely settle on afterwards.
|
||||
- confidence: a number from 0 to 1, the probability that he picks exactly that
|
||||
option.
|
||||
- cites: the finding ids from that question's findings that most shaped your
|
||||
prediction (an empty list when none did). Cite only ids listed for that
|
||||
question.
|
||||
- reason: one or two sentences, at most 400 characters.
|
||||
|
||||
## The plan
|
||||
|
||||
{{PLAN}}
|
||||
|
||||
## The questions
|
||||
|
||||
{{QUESTIONS}}
|
||||
|
||||
## Output contract
|
||||
|
||||
Reply with exactly one fenced `json` block holding one JSON array and nothing
|
||||
else. The array has exactly one object per question id listed below, in any
|
||||
order, with no duplicates and no other ids:
|
||||
|
||||
{{IDS}}
|
||||
|
||||
Each object has exactly these fields:
|
||||
|
||||
```json
|
||||
{"id": "<question id>", "action": "decide | send_back | hold", "pick": "<option key>", "confidence": 0.0, "cites": ["<finding id>"], "reason": "<at most 400 characters>"}
|
||||
```
|
||||
@@ -143,10 +143,13 @@ export function changeBudget(state: RunState, o: BudgetChange): string[] {
|
||||
return lines;
|
||||
}
|
||||
|
||||
/** Reservations with no dispatch journaled (a crash between reserve and journal) are proven never dispatched, so released. */
|
||||
/**
|
||||
* Reservations with no dispatch journaled (a crash between reserve and journal) are proven never dispatched, so released.
|
||||
* A shadow call journals `shadow_dispatched` before its send: it counts as dispatched and is never released here.
|
||||
*/
|
||||
export function releaseUndispatched(state: RunState): string[] {
|
||||
const file = path.join(state.out, FILES.spend);
|
||||
const dispatched = new Set(readJournal(state.out).filter(r => r.event === 'dispatched').map(r => r.attempt));
|
||||
const dispatched = new Set(readJournal(state.out).filter(r => r.event === 'dispatched' || r.event === 'shadow_dispatched').map(r => r.attempt));
|
||||
return readLedger(file).reserved.filter(r => !dispatched.has(r.attempt)).map(r => {
|
||||
release(file, r.attempt);
|
||||
journal(state.out, { event: 'released_undispatched', attempt: r.attempt, estimate_usd: r.estimate_usd });
|
||||
|
||||
@@ -18,8 +18,10 @@ import { readLedger, summarize, type SpendSummary } from './spend-ledger';
|
||||
|
||||
export interface GateCost {
|
||||
summary: SpendSummary;
|
||||
/** Attempts the ledger has seen: settled plus still reserved. */
|
||||
/** Reviewer attempts the ledger has seen: settled plus still reserved, shadow calls excluded. */
|
||||
attempts: number;
|
||||
/** What shadow calls (ledger label `shadow`, lib/autoplan-shadow.ts) spent or hold: in summary and the cap, never in the per-voice price. */
|
||||
shadow_usd: number;
|
||||
/** Phases a `rec` reply would reopen through its plan-changing recommendations, in run order; empty when none changes the plan. */
|
||||
reopens: string[];
|
||||
voices: number;
|
||||
@@ -72,11 +74,14 @@ export function priorChoices(outDir: string, gateRev: number): Record<string, st
|
||||
export function gateCost(outDir: string, list: GateList, run: GateCostRun, o: { defaultPerVoiceUsd?: number } = {}): GateCost {
|
||||
const ledger = readLedger(path.join(outDir, 'spend.json'), run.spend_cap_usd);
|
||||
const summary = summarize(ledger);
|
||||
const seen = [...ledger.settled.map(s => s.reserved_usd), ...ledger.reserved.map(r => r.estimate_usd)].filter(n => n > 0);
|
||||
const voice = (r: { label?: string }) => r.label !== 'shadow';
|
||||
const settled = ledger.settled.filter(voice), reserved = ledger.reserved.filter(voice);
|
||||
const seen = [...settled.map(s => s.reserved_usd), ...reserved.map(r => r.estimate_usd)].filter(n => n > 0);
|
||||
const per_voice_usd = seen.length ? seen[seen.length - 1]! : o.defaultPerVoiceUsd ?? 0;
|
||||
const reopens = recReopens(list, run, priorChoices(outDir, list.gate_rev));
|
||||
const voices = reopens.length * (run.outside_enabled ? 2 : 1);
|
||||
return { summary, attempts: ledger.settled.length + ledger.reserved.length, reopens, voices, per_voice_usd, rec_more_usd: Math.round(voices * per_voice_usd * 100) / 100 };
|
||||
const shadow_usd = Math.round([...ledger.settled.filter(s => !voice(s)).map(s => (s.usd === 'unknown' ? s.reserved_usd : s.usd)), ...ledger.reserved.filter(r => !voice(r)).map(r => r.estimate_usd)].reduce((a, b) => a + b, 0) * 1e6) / 1e6;
|
||||
return { summary, attempts: settled.length + reserved.length, shadow_usd, reopens, voices, per_voice_usd, rec_more_usd: Math.round(voices * per_voice_usd * 100) / 100 };
|
||||
}
|
||||
|
||||
/** The two printed lines: the ledger at this gate, and the price of `rec`. */
|
||||
@@ -87,7 +92,7 @@ export function renderGateCost(c: GateCost): string[] {
|
||||
const unknown = s.unknown_count ? `; ${s.unknown_count} host-subagent attempt(s) priced unknown, counted at their reservation (read the actual figure in the host's cost view)` : '';
|
||||
const after = s.headroom_usd === null ? '' : ` → headroom ${usd(s.headroom_usd - c.rec_more_usd)}${s.headroom_usd - c.rec_more_usd < 0 ? ' (over the cap: raise it with gstack-autoplan budget before answering rec)' : ''}`;
|
||||
return [
|
||||
`GATE_SPEND: spent=${usd(s.spent_usd)} worst_case=${usd(s.worst_case_usd)} cap=${cap} headroom=${headroom} attempts=${c.attempts}${unknown}`,
|
||||
`GATE_SPEND: spent=${usd(s.spent_usd)} worst_case=${usd(s.worst_case_usd)} cap=${cap} headroom=${headroom} attempts=${c.attempts}${c.shadow_usd ? ` shadow=${c.shadow_usd.toFixed(4)}` : ''}${unknown}`,
|
||||
c.reopens.length
|
||||
? `GATE_REC_COST: rec reopens up to ${c.reopens.join(',')} ≈ ${c.voices} voice(s) × ${usd(c.per_voice_usd)} = +${usd(c.rec_more_usd)} at most (the run's per-voice reservation)${after}`
|
||||
: 'GATE_REC_COST: rec reopens nothing (no recommended option changes the plan): +0.00',
|
||||
@@ -97,7 +102,7 @@ export function renderGateCost(c: GateCost): string[] {
|
||||
/** The JSON payload beside the printed lines. */
|
||||
export function gateCostPayload(c: GateCost): Record<string, unknown> {
|
||||
const s = c.summary;
|
||||
return { spent_usd: s.spent_usd, worst_case_usd: s.worst_case_usd, cap_usd: s.cap_usd, headroom_usd: s.headroom_usd, attempts: c.attempts, unknown_count: s.unknown_count, rec_reopens: c.reopens, rec_voices: c.voices, per_voice_usd: c.per_voice_usd, rec_more_usd: c.rec_more_usd };
|
||||
return { spent_usd: s.spent_usd, worst_case_usd: s.worst_case_usd, cap_usd: s.cap_usd, headroom_usd: s.headroom_usd, attempts: c.attempts, ...(c.shadow_usd ? { shadow_usd: c.shadow_usd } : {}), unknown_count: s.unknown_count, rec_reopens: c.reopens, rec_voices: c.voices, per_voice_usd: c.per_voice_usd, rec_more_usd: c.rec_more_usd };
|
||||
}
|
||||
|
||||
/** One sentence for the owner brief. */
|
||||
|
||||
+11
-5
@@ -21,6 +21,7 @@ import * as path from 'node:path';
|
||||
import { canonicalRows, parseReceipt, parseRecordFindings, parseReviewResult, type CanonicalFinding } from './autoplan-reconcile';
|
||||
import { amendmentSteps, applyGuidance, closeDeferred, closePhase } from './autoplan-gate';
|
||||
import { finishRun } from './autoplan-ceo-only';
|
||||
import type { ShadowSwitch } from './autoplan-shadow';
|
||||
import { GSTACK_VERSION } from './autoplan-migrate';
|
||||
import { driftLine, normalizeFindingFile, reviewHash, scaffold } from './autoplan-plan-text';
|
||||
import { PHASES, SNAPSHOT_TOOL, attemptFile, bindPromptResult, detectScope, driverFile, extractHeading, phaseOrder, reviewSkillFile, scopeLines, scopeSourceLabel, sectionLoadFor, snapshotTool, writePrompts, type Phase, type Scope, type ScopeFlags } from './autoplan-prompts';
|
||||
@@ -38,8 +39,12 @@ import { annotateConfirm, confirmBlock } from './autoplan-confirm';
|
||||
/** An unreadable run.lock younger than this is a lock some other gstack is still writing, never a stale one. */
|
||||
const LOCK_WRITE_GRACE_MS = 30_000;
|
||||
|
||||
/** Exclusive ownership for one command: a lock whose pid is alive is never stolen; a dead pid's lock is reclaimed. */
|
||||
export function withRunLock<T>(out: string, fn: () => T, opts: { steal?: boolean } = {}): T {
|
||||
/**
|
||||
* Exclusive ownership for one command: a lock whose pid is alive is never stolen; a dead pid's lock is reclaimed.
|
||||
* `waitMs`: retry a live owner's lock for that long before RUN_LOCKED (the shadow's publishing step, lib/autoplan-shadow.ts).
|
||||
*/
|
||||
export function withRunLock<T>(out: string, fn: () => T, opts: { steal?: boolean; waitMs?: number } = {}): T {
|
||||
const waitUntil = Date.now() + (opts.waitMs ?? 0);
|
||||
fs.mkdirSync(out, { recursive: true });
|
||||
const lock = path.join(out, FILES.lock);
|
||||
const mine = JSON.stringify({ pid: process.pid, host: os.hostname(), at: now() });
|
||||
@@ -61,6 +66,7 @@ export function withRunLock<T>(out: string, fn: () => T, opts: { steal?: boolean
|
||||
} else if (owner.host === os.hostname() && typeof owner.pid === 'number') {
|
||||
try { process.kill(owner.pid, 0); alive = true; } catch (k: any) { alive = k.code === 'EPERM'; }
|
||||
} else if (owner.host && owner.host !== os.hostname()) alive = !opts.steal;
|
||||
if (alive && Date.now() < waitUntil) { Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 100); continue; }
|
||||
if (alive) { fs.rmSync(staged, { force: true }); throw new RunError('RUN_LOCKED', unreadable ? `${lock} is being written by another command` : `pid ${owner.pid} on ${owner.host} since ${owner.at}`, 3); }
|
||||
try { fs.unlinkSync(lock); } catch { /* raced with its owner's exit */ }
|
||||
if (i > 50) { fs.rmSync(staged, { force: true }); throw new RunError('RUN_LOCKED', `could not reclaim ${lock}`, 3); }
|
||||
@@ -72,7 +78,7 @@ export function withRunLock<T>(out: string, fn: () => T, opts: { steal?: boolean
|
||||
// ---------------------------------------------------------------------------
|
||||
// init
|
||||
// ---------------------------------------------------------------------------
|
||||
export interface InitOptions { plan: string; out: string; cwd: string; scope?: ScopeFlags; ceoOnly?: boolean; planHeading?: string; outsideEnabled?: boolean; spendCap?: number | null; sessionKind?: string; deadlineMin?: number; light?: boolean; runId?: string; scaffold?: boolean; maxRounds?: number; carryForward?: boolean; carryCheckCapUsd?: number }
|
||||
export interface InitOptions { plan: string; out: string; cwd: string; scope?: ScopeFlags; ceoOnly?: boolean; shadow?: ShadowSwitch; planHeading?: string; outsideEnabled?: boolean; spendCap?: number | null; sessionKind?: string; deadlineMin?: number; light?: boolean; runId?: string; scaffold?: boolean; maxRounds?: number; carryForward?: boolean; carryCheckCapUsd?: number }
|
||||
|
||||
/** `lines` receives what init itself reports (SCAFFOLDED:). A plan with one section heading missing refuses, or with `scaffold` is reviewed from a scaffolded copy. */
|
||||
export function initRun(o: InitOptions, lines: string[] = []): RunState {
|
||||
@@ -120,12 +126,12 @@ export function initRun(o: InitOptions, lines: string[] = []): RunState {
|
||||
deviations: ['autoplan guard: not enforced by this host; publication order is unverified (GUARD_NOT_INSTALLED)'],
|
||||
...(o.deadlineMin ? { deadline_at: new Date(Date.now() + o.deadlineMin * 60_000).toISOString() } : {}),
|
||||
...(o.light ? { light: true } : {}),
|
||||
...(o.ceoOnly ? { ceo_only: true } : {}), gstack_version: GSTACK_VERSION, plan_input: input, scope_flags: { ...(o.scope ?? {}) },
|
||||
...(o.ceoOnly ? { ceo_only: true } : {}), ...(o.shadow ? { shadow: o.shadow } : {}), gstack_version: GSTACK_VERSION, plan_input: input, scope_flags: { ...(o.scope ?? {}) },
|
||||
...(o.planHeading ? { plan_heading: o.planHeading } : {}), ...(scaffolded ? { scaffolded: true } : {}),
|
||||
...(o.carryForward === false ? { carry_forward: false } : {}), ...(o.carryCheckCapUsd !== undefined ? { carry_check_cap_usd: o.carryCheckCapUsd } : {}),
|
||||
};
|
||||
fs.writeFileSync(path.join(out, 'timing.json'), JSON.stringify({ schema_version: 1, run, phases: [], started_at: now(), session_kind: state.session_kind }, null, 2) + '\n');
|
||||
journal(out, { event: 'init', run, plan_source: source, scope, phase_order: order, ...(o.ceoOnly ? { ceo_only: true } : {}), ...(consentMissing ? { outside: 'not_run', reason: 'consent_missing' } : {}) });
|
||||
journal(out, { event: 'init', run, plan_source: source, scope, phase_order: order, ...(o.ceoOnly ? { ceo_only: true } : {}), ...(o.shadow ? { shadow: o.shadow } : {}), ...(consentMissing ? { outside: 'not_run', reason: 'consent_missing' } : {}) });
|
||||
writeState(state);
|
||||
return state;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,442 @@
|
||||
/**
|
||||
* autoplan-shadow — shadow mode for the CEO-only decider (GST-537 PR 3,
|
||||
* docs/unattended.md#shadow-mode). In a driver run that turned it on at init
|
||||
* (`autoplan_shadow on` or GSTACK_AUTOPLAN_SHADOW=1; off by default), every
|
||||
* gate the driver publishes gets one hidden model prediction per question,
|
||||
* written before anyone answers: action (decide, send_back, hold), pick,
|
||||
* confidence, cites and reason, beside the free keep-direction rule's pick.
|
||||
* Nothing routes differently and a shadow failure never blocks a gate or an
|
||||
* answer. The decider prompt is the frozen, hand-written
|
||||
* autoplan/shadow-decider.md (its sha256 is pinned below and on every row).
|
||||
*
|
||||
* The publication is two-phase so the run lock is never held during the call:
|
||||
* `claimShadow` runs under the run lock of the command that wrote the gate and
|
||||
* takes a lease (`<run>/shadow/r<rev>.pending`: pid, host, deadline; journaled
|
||||
* `shadow_pending`); `runShadow` calls the model with no lock held (each call
|
||||
* reserved under its own ledger id labeled `shadow` and journaled
|
||||
* `shadow_dispatched` before the send), then retakes the lock with a bounded
|
||||
* wait and publishes `<run>/shadow/ceo-only.r<rev>.jsonl` with link (a second
|
||||
* writer never overwrites) plus `shadow_recorded` or `shadow_failed`. A live
|
||||
* lease makes every other writer pay nothing, and `answer` waits for it
|
||||
* (`shadowBeforeAnswer`). Only a dead lease (its pid gone, or past its
|
||||
* deadline) is a crash: its dispatched calls settle `unknown`, never released,
|
||||
* and `answer` then records the revision with every row `retrospective`.
|
||||
* This module is the only one that names the shadow folder's paths; the reader
|
||||
* for gstack-evals and the private validate check are in
|
||||
* lib/autoplan-shadow-reader.ts.
|
||||
*/
|
||||
import * as crypto from 'node:crypto';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { extractHeading } from './autoplan-prompts';
|
||||
import { GSTACK_VERSION } from './autoplan-migrate';
|
||||
import { withRunLock } from './autoplan-run';
|
||||
import { FILES, RunError, journal, now, readJournal, readState, voiceRows, type RunState } from './autoplan-state';
|
||||
import { decisionId, recHash, ruleKept, type GateItem, type GateList } from './gate-list';
|
||||
import type { FindingRow } from './headless-artifacts';
|
||||
import { readJsonl } from './headless-artifacts';
|
||||
import { apiKeyFor, callModel, type ApiCallResult } from './outside-voice-api';
|
||||
import { modelFamily } from './outside-voice-runner';
|
||||
import { estimateCostUsd, hasPricing } from './pricing';
|
||||
import { readLedger, release, reserve, settle, summarize } from './spend-ledger';
|
||||
import { resolveSourceRoot } from './source-root';
|
||||
import { readConfigKey } from './state-root';
|
||||
|
||||
export const SHADOW_PROMPT_VERSION = 1;
|
||||
/** sha256 of autoplan/shadow-decider.md at SHADOW_PROMPT_VERSION; a prompt edit bumps both (test/autoplan-shadow.test.ts). */
|
||||
export const SHADOW_PROMPT_SHA256 = 'e4857932ae831b7a6d933c42c182ccd7711c68338fec6901799264f645589344';
|
||||
export const DEFAULT_SHADOW_MODEL = 'claude-opus-5-5';
|
||||
/** One deadline for the call and its one retry. */
|
||||
export const SHADOW_DEADLINE_MS = 90_000;
|
||||
export const SHADOW_MAX_TOKENS = 4_000;
|
||||
/** A reply cut at the output limit is asked once more at this limit, inside the same deadline. */
|
||||
export const SHADOW_RAISED_MAX_TOKENS = 8_000;
|
||||
export const SHADOW_SINK = 'autoplan-shadow';
|
||||
export const SHADOW_CONSENT = 'autoplan_shadow on';
|
||||
const PLAN_CHARS = 40_000;
|
||||
const REASON_CHARS = 400;
|
||||
const RELOCK_MS = 30_000;
|
||||
const LEASE_GRACE_MS = 30_000;
|
||||
export const SHADOW_ACTIONS = ['decide', 'send_back', 'hold'] as const;
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
|
||||
export const shadowPromptFile = () => path.join(resolveSourceRoot(ROOT), 'autoplan', 'shadow-decider.md');
|
||||
export const shadowDir = (out: string) => path.join(out, 'shadow');
|
||||
export const recordFile = (out: string, rev: number) => path.join(shadowDir(out), `ceo-only.r${rev}.jsonl`);
|
||||
export const RECORD_RE = /^ceo-only\.r(\d+)\.jsonl$/;
|
||||
const leaseFile = (out: string, rev: number) => path.join(shadowDir(out), `r${rev}.pending`);
|
||||
const saltFile = (out: string) => path.join(shadowDir(out), 'salt');
|
||||
const sha256 = (data: string | Buffer) => crypto.createHash('sha256').update(data).digest('hex');
|
||||
const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms));
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// the switch, fixed at init
|
||||
// ---------------------------------------------------------------------------
|
||||
export interface ShadowSwitch { enabled: boolean; model?: string; reason?: string; source: 'env' | 'config' | 'default' }
|
||||
|
||||
/**
|
||||
* GSTACK_AUTOPLAN_SHADOW (1|on|0|off), else gstack-config autoplan_shadow (default off). On also needs a priced model
|
||||
* (autoplan_shadow_model, default claude-opus-5-5), its family's API key, and a run that is neither --no-outside nor
|
||||
* CEO-only. Stored in run.json at init and never re-read. The line is printed only when shadow was asked for.
|
||||
*/
|
||||
export function shadowSwitch(o: { outsideEnabled: boolean; ceoOnly: boolean }, env: NodeJS.ProcessEnv = process.env): { shadow: ShadowSwitch; line?: string } {
|
||||
const fromEnv = !!env.GSTACK_AUTOPLAN_SHADOW?.trim();
|
||||
const value = (fromEnv ? env.GSTACK_AUTOPLAN_SHADOW! : readConfigKey('autoplan_shadow', env) ?? '').trim().toLowerCase();
|
||||
const source = fromEnv ? 'env' as const : value ? 'config' as const : 'default' as const;
|
||||
const label = fromEnv ? `GSTACK_AUTOPLAN_SHADOW=${value}` : `autoplan_shadow ${value}`;
|
||||
const off = (reason: string) => ({ shadow: { enabled: false, reason, source }, line: `SHADOW: off (${reason})` });
|
||||
if (!value || value === 'off' || value === '0') return { shadow: { enabled: false, reason: source === 'default' ? 'off by default' : label, source } };
|
||||
if (value !== 'on' && value !== '1') return off(`${label} is not on or off`);
|
||||
if (o.ceoOnly) return off('a CEO-only run writes no gate');
|
||||
if (!o.outsideEnabled) return off('--no-outside');
|
||||
const model = (readConfigKey('autoplan_shadow_model', env) ?? '').trim() || DEFAULT_SHADOW_MODEL;
|
||||
const family = modelFamily(model);
|
||||
const keyName = family === 'openai' ? 'OPENAI_API_KEY' : family === 'anthropic' ? 'ANTHROPIC_API_KEY' : `no API transport for family ${family}`;
|
||||
if (!apiKeyFor(family, env)) return off(`no API key for ${model} (${keyName})`);
|
||||
if (!hasPricing(model)) return off(`${model} has no price in lib/pricing.ts, so --spend-cap cannot bound it`);
|
||||
return { shadow: { enabled: true, model, source }, line: `SHADOW: on model=${model} (${label}); one hidden model call per gate page, counted against --spend-cap` };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// the lease
|
||||
// ---------------------------------------------------------------------------
|
||||
interface Lease { gate_rev: number; pid: number; host: string; started_at: string; deadline: string; retrospective: boolean }
|
||||
export interface ShadowClaim { gate_rev: number; retrospective: boolean }
|
||||
|
||||
function readLease(out: string, rev: number): Lease | undefined {
|
||||
try { return JSON.parse(fs.readFileSync(leaseFile(out, rev), 'utf8')) as Lease; } catch { return undefined; }
|
||||
}
|
||||
/** Live: before its deadline plus a grace, and (on this host) its pid still running. */
|
||||
function leaseLive(l: Lease, at = Date.now()): boolean {
|
||||
if (!(at <= Date.parse(l.deadline) + LEASE_GRACE_MS)) return false;
|
||||
if (l.host !== os.hostname()) return true;
|
||||
try { process.kill(l.pid, 0); return true; } catch (e: any) { return e.code === 'EPERM'; }
|
||||
}
|
||||
function dropLease(out: string, rev: number): void {
|
||||
const l = readLease(out, rev);
|
||||
if (l && l.pid === process.pid && l.host === os.hostname()) fs.rmSync(leaseFile(out, rev), { force: true });
|
||||
}
|
||||
const resultRow = (rows: Array<Record<string, any>>, rev: number) => rows.find(r => (r.event === 'shadow_recorded' || r.event === 'shadow_failed') && r.gate_rev === rev);
|
||||
const callIds = (rows: Array<Record<string, any>>, rev: number) => rows.filter(r => r.event === 'shadow_dispatched' && r.gate_rev === rev).map(r => String(r.attempt));
|
||||
|
||||
/** The shadow folder: created on first use; a symlink or a non-directory is refused, never followed. */
|
||||
function safeDir(out: string): string | undefined {
|
||||
const dir = shadowDir(out);
|
||||
try {
|
||||
const st = fs.lstatSync(dir);
|
||||
if (st.isSymbolicLink()) return `${dir} is a symlink`;
|
||||
if (!st.isDirectory()) return `${dir} is not a directory`;
|
||||
} catch (e: any) {
|
||||
if (e.code !== 'ENOENT') return `${dir}: ${e.code ?? e.message}`;
|
||||
fs.mkdirSync(dir, { mode: 0o700 });
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
/** A dead lease: its dispatched calls settle unknown at their reservation (they may have been billed), the rest are released. */
|
||||
function expireLease(out: string, rev: number, rows: Array<Record<string, any>>): void {
|
||||
const file = path.join(out, FILES.spend);
|
||||
const sent = new Set(callIds(rows, rev));
|
||||
const open = readLedger(file).reserved.filter(r => r.attempt.startsWith(`shadow-r${rev}-`));
|
||||
for (const r of open) if (sent.has(r.attempt)) settle(file, r.attempt, 'unknown'); else release(file, r.attempt);
|
||||
fs.rmSync(leaseFile(out, rev), { force: true });
|
||||
journal(out, { event: 'shadow_expired', gate_rev: rev, settled_unknown: open.filter(r => sent.has(r.attempt)).map(r => r.attempt), released: open.filter(r => !sent.has(r.attempt)).map(r => r.attempt) });
|
||||
}
|
||||
|
||||
/**
|
||||
* Under the run lock: claim the current gate revision's shadow, or say why not (nothing printed when shadow is off, the
|
||||
* run is not at a gate, or the revision already has its result). A live lease means another writer is paying: this one
|
||||
* pays nothing. `retrospective` (answer's crash path) claims only a revision whose shadow_pending has no result.
|
||||
*/
|
||||
export function claimShadow(out: string, o: { retrospective?: boolean } = {}): { claim?: ShadowClaim; line?: string } {
|
||||
const state = readState(out);
|
||||
const rev = state.gate_rev;
|
||||
if (!state.shadow?.enabled || state.ceo_only || state.status !== 'gate_pending' || !rev) return {};
|
||||
const rows = readJournal(out);
|
||||
if (resultRow(rows, rev) || fs.existsSync(recordFile(out, rev))) return {};
|
||||
const pending = rows.some(r => r.event === 'shadow_pending' && r.gate_rev === rev);
|
||||
if (o.retrospective && !pending) return {};
|
||||
const unsafe = safeDir(out);
|
||||
if (unsafe) {
|
||||
journal(out, { event: 'shadow_failed', gate_rev: rev, code: 'unsafe_dir', detail: unsafe });
|
||||
return { line: `SHADOW: failed unsafe_dir (${unsafe})` };
|
||||
}
|
||||
const held = readLease(out, rev);
|
||||
if (held && leaseLive(held)) return { line: `SHADOW: pending (gate_rev ${rev} is being recorded by pid ${held.pid} on ${held.host} until ${held.deadline}; this command pays nothing)` };
|
||||
if (held || pending) expireLease(out, rev, rows);
|
||||
const started = Date.now();
|
||||
const lease: Lease = { gate_rev: rev, pid: process.pid, host: os.hostname(), started_at: new Date(started).toISOString(), deadline: new Date(started + SHADOW_DEADLINE_MS + RELOCK_MS).toISOString(), retrospective: !!o.retrospective };
|
||||
const staged = `${leaseFile(out, rev)}.${process.pid}.${crypto.randomBytes(4).toString('hex')}.tmp`;
|
||||
fs.writeFileSync(staged, JSON.stringify(lease));
|
||||
try { fs.linkSync(staged, leaseFile(out, rev)); } catch (e: any) { if (e.code !== 'EEXIST') throw e; return { line: `SHADOW: pending (gate_rev ${rev} was claimed by another writer)` }; } finally { fs.rmSync(staged, { force: true }); }
|
||||
journal(out, { event: 'shadow_pending', ...lease });
|
||||
return { claim: { gate_rev: rev, retrospective: lease.retrospective } };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// the page: one prompt per gate
|
||||
// ---------------------------------------------------------------------------
|
||||
interface PageFinding { id: string; severity: string; title: string }
|
||||
export interface PageItem { item: GateItem; findings: PageFinding[]; frozen: Record<string, unknown> }
|
||||
export interface Page { prompt: string; items: PageItem[]; plan_sha256: string; prompt_sha256: string; request_sha256: string }
|
||||
|
||||
/** Every finding the gate items can name: both voices of every phase, plus findings a deferred close carried. */
|
||||
function findingIndex(state: RunState): Map<string, FindingRow> {
|
||||
const rows: FindingRow[] = [];
|
||||
for (const phase of state.phase_order) {
|
||||
rows.push(...voiceRows(state, phase, 'native'), ...voiceRows(state, phase, 'outside'));
|
||||
const carried = state.phases[phase]?.deferred?.carried_file;
|
||||
if (carried && fs.existsSync(path.join(state.out, carried))) rows.push(...readJsonl(path.join(state.out, carried)).rows as unknown as FindingRow[]);
|
||||
}
|
||||
return new Map(rows.map(r => [r.id, r]));
|
||||
}
|
||||
|
||||
/** The question as frozen in the record and sent in the prompt; p1 lists every finding the other items name. */
|
||||
export function buildPage(state: RunState, list: GateList, deciderText: string): Page {
|
||||
const index = findingIndex(state);
|
||||
const own = (item: GateItem) => (item.origin?.split('|')[1] ?? '').split('+').filter(id => index.has(id));
|
||||
const named = [...new Set(list.items.filter(i => i.id !== 'p1').flatMap(own))];
|
||||
const items = list.items.map(item => {
|
||||
const findings = (item.id === 'p1' ? named : own(item)).map(id => { const f = index.get(id)!; return { id, severity: f.severity, title: f.title.split('\n')[0]!.slice(0, 200) }; });
|
||||
const frozen = { id: item.id, item_class: item.item_class ?? null, kind: item.kind, title: item.title, why: item.why ?? null, phase: item.phase ?? null,
|
||||
options: item.options.map(o => ({ key: o.key, text: o.text, changes_plan: o.changes_plan ?? null })), recommended: item.recommended, recommended_basis: item.recommended_basis ?? null, findings };
|
||||
return { item, findings, frozen };
|
||||
});
|
||||
const planText = fs.existsSync(state.active_plan) ? fs.readFileSync(state.active_plan, 'utf8') : '';
|
||||
const plan = (extractHeading(planText, 'Implementation plan') ?? planText).slice(0, PLAN_CHARS);
|
||||
const prompt = deciderText.replace('{{PLAN}}', () => plan).replace('{{QUESTIONS}}', () => '```json\n' + JSON.stringify(items.map(i => i.frozen), null, 2) + '\n```')
|
||||
.replace('{{IDS}}', () => items.map(i => i.item.id).join(', '));
|
||||
return { prompt, items, plan_sha256: sha256(planText), prompt_sha256: sha256(deciderText), request_sha256: sha256(prompt) };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// the reply contract
|
||||
// ---------------------------------------------------------------------------
|
||||
export type Verdict = { ok: true; action: typeof SHADOW_ACTIONS[number]; pick: string; confidence: number; cites: string[]; reason: string } | { ok: false; why: string };
|
||||
export interface ParsedPage { byId: Map<string, Verdict>; foreign: string[]; clean: boolean }
|
||||
|
||||
function jsonArray(text: string): unknown {
|
||||
const fenced = /```(?:json)?\s*\n([\s\S]*?)```/.exec(text)?.[1];
|
||||
for (const candidate of [fenced, text.trim(), text.slice(text.indexOf('['), text.lastIndexOf(']') + 1)]) {
|
||||
if (!candidate) continue;
|
||||
try { return JSON.parse(candidate); } catch { /* next candidate */ }
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
/** Exactly one valid object per requested id; a duplicate, omission or invalid field marks that item malformed, a foreign id the page. */
|
||||
export function parsePage(text: string, items: PageItem[]): ParsedPage {
|
||||
const byId = new Map<string, Verdict>();
|
||||
const parsed = jsonArray(text);
|
||||
if (!Array.isArray(parsed)) {
|
||||
for (const i of items) byId.set(i.item.id, { ok: false, why: 'no JSON array in the reply' });
|
||||
return { byId, foreign: [], clean: false };
|
||||
}
|
||||
const want = new Map(items.map(i => [i.item.id, i]));
|
||||
const seen = new Map<string, unknown[]>();
|
||||
const foreign: string[] = [];
|
||||
for (const obj of parsed) {
|
||||
const id = (obj as { id?: unknown })?.id;
|
||||
if (typeof id === 'string' && want.has(id)) (seen.get(id) ?? seen.set(id, []).get(id)!).push(obj);
|
||||
else foreign.push(typeof id === 'string' ? id : JSON.stringify(id ?? null));
|
||||
}
|
||||
for (const [id, pi] of want) {
|
||||
const objs = seen.get(id) ?? [];
|
||||
byId.set(id, objs.length === 0 ? { ok: false, why: 'missing' } : objs.length > 1 ? { ok: false, why: 'duplicate' } : verdict(objs[0] as Record<string, unknown>, pi));
|
||||
}
|
||||
return { byId, foreign, clean: foreign.length === 0 && [...byId.values()].every(v => v.ok) };
|
||||
}
|
||||
|
||||
function verdict(o: Record<string, unknown>, pi: PageItem): Verdict {
|
||||
const cites = Array.isArray(o.cites) ? o.cites : null;
|
||||
const allowed = new Set(pi.findings.map(f => f.id));
|
||||
if (!SHADOW_ACTIONS.includes(o.action as typeof SHADOW_ACTIONS[number])) return { ok: false, why: `action ${JSON.stringify(o.action ?? null)} is not decide, send_back or hold` };
|
||||
if (typeof o.pick !== 'string' || !pi.item.options.some(op => op.key === o.pick)) return { ok: false, why: `pick ${JSON.stringify(o.pick ?? null)} is not an option of ${pi.item.id}` };
|
||||
if (typeof o.confidence !== 'number' || !Number.isFinite(o.confidence) || o.confidence < 0 || o.confidence > 1) return { ok: false, why: `confidence ${JSON.stringify(o.confidence ?? null)} is not a number from 0 to 1` };
|
||||
if (!cites || cites.some(c => typeof c !== 'string' || !allowed.has(c))) return { ok: false, why: `cites ${JSON.stringify(o.cites ?? null)} name a finding outside ${pi.item.id}'s origin` };
|
||||
if (typeof o.reason !== 'string' || o.reason.length > REASON_CHARS) return { ok: false, why: `reason is not a string of at most ${REASON_CHARS} characters` };
|
||||
return { ok: true, action: o.action as typeof SHADOW_ACTIONS[number], pick: o.pick, confidence: o.confidence, cites: cites as string[], reason: o.reason };
|
||||
}
|
||||
|
||||
/** The 10% audit mark: sha256(salt + decision id) mod 1000 < 100. */
|
||||
export function auditMark(salt: string, decision: string): boolean {
|
||||
return BigInt('0x' + sha256(salt + decision)) % 1000n < 100n;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// the call and the record
|
||||
// ---------------------------------------------------------------------------
|
||||
export interface ShadowOutcome { status: 'recorded' | 'failed' | 'dropped'; code?: string; model?: string; usd?: number; malformed?: number; line: string }
|
||||
export interface RunShadowOptions { env?: NodeJS.ProcessEnv; deadlineMs?: number }
|
||||
class ShadowFailure extends Error { constructor(public code: string, detail: string) { super(detail); } }
|
||||
interface Reply { result: ApiCallResult; usd: number | 'unknown'; attempt: string }
|
||||
|
||||
/** Classify a callModel error: no key, an HTTP status (not billed, settles 0), or a timeout or network loss (may be billed, settles unknown). */
|
||||
function failureOf(e: any): { code: string; usd: number | 'unknown' } {
|
||||
const msg = String(e?.message ?? e);
|
||||
if (/^no API key/.test(msg)) return { code: 'no_key', usd: 0 };
|
||||
if (/^(anthropic|openai) \d{3}:/.test(msg)) return { code: 'http', usd: 0 };
|
||||
return { code: /timeout|abort/i.test(msg) || e?.name === 'AbortError' || e?.name === 'TimeoutError' ? 'timeout' : 'network', usd: 'unknown' };
|
||||
}
|
||||
|
||||
/**
|
||||
* Outside any lock: the page is asked once; a reply cut at the output limit is asked once more at 8,000 tokens, and a
|
||||
* malformed one once more as a fresh request; one 90 s deadline covers both. Each call is reserved under its own
|
||||
* ledger id (label shadow), journaled shadow_dispatched before the send and settled at its actual usage. Then the
|
||||
* lock is retaken with a bounded wait and the record published. Never throws: every failure is an outcome.
|
||||
*/
|
||||
export async function runShadow(out: string, claim: ShadowClaim, o: RunShadowOptions = {}): Promise<ShadowOutcome> {
|
||||
const env = o.env ?? process.env;
|
||||
const deadline = Date.now() + (o.deadlineMs ?? SHADOW_DEADLINE_MS);
|
||||
const rev = claim.gate_rev;
|
||||
const replies: Reply[] = [];
|
||||
try {
|
||||
const state = readState(out);
|
||||
const list = JSON.parse(fs.readFileSync(path.join(out, 'gate.json'), 'utf8')) as GateList;
|
||||
if (list.gate_rev !== rev) throw new ShadowFailure('stale_gate', `gate.json is gate_rev ${list.gate_rev}, the claim is ${rev}`);
|
||||
const model = state.shadow!.model!;
|
||||
const page = buildPage(state, list, fs.readFileSync(shadowPromptFile(), 'utf8'));
|
||||
writeAtomic(path.join(shadowDir(out), `r${rev}.prompt.md`), page.prompt);
|
||||
const inputTokens = Math.ceil(Buffer.byteLength(page.prompt) / 3);
|
||||
const estimate = (maxTokens: number) => estimateCostUsd({ input: inputTokens, output: maxTokens }, model);
|
||||
const ledgerFile = path.join(out, FILES.spend);
|
||||
const headroom = summarize(readLedger(ledgerFile, state.spend_cap_usd)).headroom_usd;
|
||||
const worst = estimate(SHADOW_MAX_TOKENS) + estimate(SHADOW_RAISED_MAX_TOKENS);
|
||||
if (headroom !== null && headroom < worst) throw new ShadowFailure('budget', `headroom ${headroom.toFixed(4)} < ${worst.toFixed(4)} (one call plus its retry)`);
|
||||
const ask = async (maxTokens: number): Promise<Reply> => {
|
||||
const remaining = deadline - Date.now();
|
||||
if (remaining <= 0) throw new ShadowFailure('timeout', `the ${(o.deadlineMs ?? SHADOW_DEADLINE_MS) / 1000} s deadline passed before the call`);
|
||||
const attempt = `shadow-r${rev}-${callIds(readJournal(out), rev).length + 1}`;
|
||||
try { reserve(ledgerFile, attempt, estimate(maxTokens), { cap: state.spend_cap_usd, label: 'shadow' }); }
|
||||
catch (e: any) { throw new ShadowFailure('budget', e.message); }
|
||||
journal(out, { event: 'shadow_dispatched', gate_rev: rev, attempt, model, max_tokens: maxTokens, estimate_usd: estimate(maxTokens), retrospective: claim.retrospective });
|
||||
let result: ApiCallResult;
|
||||
try {
|
||||
result = await callModel({ model, prompt: page.prompt, timeoutMs: remaining, env, maxTokens, receipt: { sink: SHADOW_SINK, consent: SHADOW_CONSENT, payloadClass: 'plan-and-gate' } });
|
||||
} catch (e: any) {
|
||||
const f = failureOf(e);
|
||||
settle(ledgerFile, attempt, f.usd);
|
||||
throw new ShadowFailure(f.code, String(e?.message ?? e));
|
||||
}
|
||||
const { input_tokens: inTok, output_tokens: outTok } = result.usage;
|
||||
const usd: number | 'unknown' = typeof inTok === 'number' && typeof outTok === 'number' ? estimateCostUsd({ input: inTok, output: outTok }, model) : 'unknown';
|
||||
settle(ledgerFile, attempt, usd);
|
||||
fs.writeFileSync(path.join(shadowDir(out), `${attempt}.reply.txt`), result.text);
|
||||
const reply = { result, usd, attempt };
|
||||
replies.push(reply);
|
||||
return reply;
|
||||
};
|
||||
const truncated = (r: Reply) => r.result.stop_reason === 'max_tokens' || r.result.stop_reason === 'length';
|
||||
const first = await ask(SHADOW_MAX_TOKENS);
|
||||
let parsed: ParsedPage[];
|
||||
if (truncated(first)) {
|
||||
const raised = await ask(SHADOW_RAISED_MAX_TOKENS);
|
||||
if (truncated(raised)) throw new ShadowFailure('truncated', `the reply was cut at ${SHADOW_RAISED_MAX_TOKENS} output tokens`);
|
||||
parsed = [parsePage(raised.result.text, page.items)];
|
||||
} else {
|
||||
parsed = [parsePage(first.result.text, page.items)];
|
||||
if (!parsed[0]!.clean) {
|
||||
try {
|
||||
const retry = await ask(SHADOW_MAX_TOKENS);
|
||||
if (!truncated(retry)) parsed.push(parsePage(retry.result.text, page.items));
|
||||
} catch (e) { if (!(e instanceof ShadowFailure)) throw e; }
|
||||
}
|
||||
}
|
||||
return publish(out, claim, { rows: shadowRows(out, state, list, page, parsed, replies), model, replies });
|
||||
} catch (e: any) {
|
||||
const failure = e instanceof ShadowFailure ? e : new ShadowFailure('internal', String(e?.message ?? e));
|
||||
return publish(out, claim, { failure, replies });
|
||||
}
|
||||
}
|
||||
|
||||
const usdOf = (replies: Reply[]) => (replies.some(r => r.usd === 'unknown') ? 'unknown' as const : Math.round(replies.reduce((s, r) => s + (r.usd as number), 0) * 1e6) / 1e6);
|
||||
|
||||
/** One row per gate item: the frozen question, what the model predicted (the retry's verdict when valid, else the first's), the free rule's pick and the audit mark. */
|
||||
function shadowRows(out: string, state: RunState, list: GateList, page: Page, parsed: ParsedPage[], replies: Reply[]): Array<Record<string, unknown>> {
|
||||
const at = now();
|
||||
const usage = replies.reduce((u, r) => ({ input_tokens: u.input_tokens + (r.result.usage.input_tokens ?? 0), output_tokens: u.output_tokens + (r.result.usage.output_tokens ?? 0) }), { input_tokens: 0, output_tokens: 0 });
|
||||
return page.items.map(({ item, frozen }) => {
|
||||
const verdicts = parsed.map(p => p.byId.get(item.id)!);
|
||||
const k = verdicts.map((v, i) => [v, i] as const).reverse().find(([v]) => v.ok);
|
||||
const decision = decisionId(list.run, list.gate_rev, item.id);
|
||||
const base = {
|
||||
schema_version: 1, run: list.run, gate_rev: list.gate_rev, decision, item: item.id, item_class: item.item_class ?? null, phase: item.phase ?? null,
|
||||
ceo_eligible: item.id === 'p1' || item.phase === 'ceo', item_frozen: frozen, rec_hash: recHash(list), rec_rev: list.rec_rev ?? 0,
|
||||
plan_sha256: page.plan_sha256, prompt_version: SHADOW_PROMPT_VERSION, prompt_sha256: page.prompt_sha256, request_sha256: page.request_sha256,
|
||||
model: replies[replies.length - 1]?.result.model ?? state.shadow?.model, gstack_version: GSTACK_VERSION, rule_pick: ruleKept(item),
|
||||
retrospective: false, page_usage: usage, page_usd: usdOf(replies), recorded_at: at,
|
||||
};
|
||||
if (!k) return { ...base, status: 'malformed', malformed: (verdicts[verdicts.length - 1] as { why: string }).why, action: null, pick: null, confidence: null, cites: [], reason: null, audit: false };
|
||||
const v = k[0] as Extract<Verdict, { ok: true }>;
|
||||
return { ...base, status: 'recorded', action: v.action, pick: v.pick, confidence: v.confidence, cites: v.cites, reason: v.reason, reply: k[1] + 1,
|
||||
audit: v.action === 'decide' && auditMark(saltOf(out), decision) };
|
||||
});
|
||||
}
|
||||
|
||||
/** The per-run audit salt, drawn once into the shadow folder (never run.json). */
|
||||
function saltOf(out: string): string {
|
||||
try { return fs.readFileSync(saltFile(out), 'utf8').trim(); } catch { /* first use */ }
|
||||
try { fs.writeFileSync(saltFile(out), crypto.randomBytes(16).toString('hex') + '\n', { flag: 'wx', mode: 0o600 }); } catch (e: any) { if (e.code !== 'EEXIST') throw e; }
|
||||
return fs.readFileSync(saltFile(out), 'utf8').trim();
|
||||
}
|
||||
|
||||
function writeAtomic(file: string, text: string): void {
|
||||
const tmp = `${file}.${process.pid}.${crypto.randomBytes(4).toString('hex')}.tmp`;
|
||||
fs.writeFileSync(tmp, text);
|
||||
fs.renameSync(tmp, file);
|
||||
}
|
||||
|
||||
/** Under the run lock (a bounded wait, never RUN_LOCKED at once): link the record, journal the result, drop the lease. */
|
||||
function publish(out: string, claim: ShadowClaim, r: { rows?: Array<Record<string, unknown>>; model?: string; failure?: ShadowFailure; replies: Reply[] }): ShadowOutcome {
|
||||
const rev = claim.gate_rev;
|
||||
const usd = usdOf(r.replies);
|
||||
const attempts = r.replies.map(x => x.attempt);
|
||||
try {
|
||||
return withRunLock(out, () => {
|
||||
const rows = readJournal(out);
|
||||
if (fs.existsSync(recordFile(out, rev)) || resultRow(rows, rev)) { dropLease(out, rev); return { status: 'dropped' as const, line: `SHADOW: dropped (gate_rev ${rev} already has its result; this call's cost stays in the ledger)` }; }
|
||||
if (r.failure) {
|
||||
journal(out, { event: 'shadow_failed', gate_rev: rev, code: r.failure.code, detail: r.failure.message.slice(0, 500), attempts, usd, retrospective: claim.retrospective });
|
||||
dropLease(out, rev);
|
||||
return { status: 'failed' as const, code: r.failure.code, line: `SHADOW: failed ${r.failure.code}` };
|
||||
}
|
||||
const salted: Array<Record<string, unknown>> = r.rows!.map(row => ({ ...row, retrospective: claim.retrospective }));
|
||||
const text = salted.map(row => JSON.stringify(row)).join('\n') + '\n';
|
||||
const final = recordFile(out, rev);
|
||||
for (const e of fs.readdirSync(shadowDir(out))) if (e.startsWith(`ceo-only.r${rev}.jsonl.`) && e.endsWith('.tmp')) fs.rmSync(path.join(shadowDir(out), e), { force: true });
|
||||
const tmp = `${final}.${process.pid}.${crypto.randomBytes(4).toString('hex')}.tmp`;
|
||||
fs.writeFileSync(tmp, text, { flag: 'wx' });
|
||||
try { fs.linkSync(tmp, final); } catch (e: any) { if (e.code !== 'EEXIST') throw e; dropLease(out, rev); return { status: 'dropped' as const, line: `SHADOW: dropped (gate_rev ${rev} was recorded by another writer)` }; } finally { fs.rmSync(tmp, { force: true }); }
|
||||
const malformed = salted.filter(row => row.status === 'malformed').length;
|
||||
journal(out, { event: 'shadow_recorded', gate_rev: rev, sha256: sha256(text), model: r.model, usd, items: salted.length, malformed, attempts, retrospective: claim.retrospective });
|
||||
dropLease(out, rev);
|
||||
const usdText = usd === 'unknown' ? 'unknown' : usd.toFixed(4);
|
||||
return { status: 'recorded' as const, model: r.model, usd: usd === 'unknown' ? undefined : usd, malformed,
|
||||
line: `SHADOW: recorded model=${r.model} usd=${usdText}${malformed ? ` malformed=${malformed}/${salted.length}` : ''}${claim.retrospective ? ' retrospective' : ''}` };
|
||||
}, { waitMs: RELOCK_MS });
|
||||
} catch (e: any) {
|
||||
const code = e instanceof RunError && e.code === 'RUN_LOCKED' ? 'locked' : 'internal';
|
||||
return { status: 'failed', code, line: `SHADOW: failed ${code} (${e?.message ?? e}); the gate stands` };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Before `answer` takes the lock: wait for a live lease to finish (up to its deadline), then, when the revision has a
|
||||
* shadow_pending and still no result (a crash), record it first with every row retrospective. Never throws.
|
||||
*/
|
||||
export async function shadowBeforeAnswer(out: string, o: RunShadowOptions = {}): Promise<string[]> {
|
||||
try {
|
||||
const state = readState(out);
|
||||
const rev = state.gate_rev;
|
||||
if (!state.shadow?.enabled || state.status !== 'gate_pending' || !rev) return [];
|
||||
for (let l = readLease(out, rev); l && leaseLive(l); l = readLease(out, rev)) await sleep(200);
|
||||
const rows = readJournal(out);
|
||||
if (resultRow(rows, rev) || fs.existsSync(recordFile(out, rev)) || !rows.some(r => r.event === 'shadow_pending' && r.gate_rev === rev)) return [];
|
||||
const { claim, line } = withRunLock(out, () => claimShadow(out, { retrospective: true }), { waitMs: RELOCK_MS });
|
||||
if (!claim) return line ? [line] : [];
|
||||
return [(await runShadow(out, claim, o)).line];
|
||||
} catch (e: any) {
|
||||
return [`SHADOW: failed internal (${e?.message ?? e}); the answer proceeds`];
|
||||
}
|
||||
}
|
||||
@@ -74,6 +74,8 @@ export interface RunState extends RunManifest {
|
||||
carry_forward?: boolean; carry_check_cap_usd?: number; carry?: import('./autoplan-carry').CarryRecord;
|
||||
/** `next --ceo-only` (lib/autoplan-ceo-only.ts): the CEO phase alone, no gate, ends `reviewed`; never set by a scope marker. */
|
||||
ceo_only?: boolean;
|
||||
/** Shadow mode (lib/autoplan-shadow.ts): fixed at init from GSTACK_AUTOPLAN_SHADOW or autoplan_shadow and never re-read; absent reads off. */
|
||||
shadow?: import('./autoplan-shadow').ShadowSwitch;
|
||||
}
|
||||
|
||||
export const TERMINAL_VOICE_STATUSES = ['completed', 'disabled', 'unavailable', 'failed', 'skipped'] as const;
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
/**
|
||||
* GST-537 PR 3, shadow mode for the CEO-only decider (lib/autoplan-shadow.ts):
|
||||
* the frozen decider prompt, the switch, the reply contract, the 10% audit
|
||||
* mark, the call (truncation, the OpenAI output bound, the 90 s deadline, the
|
||||
* shadow's own egress receipt), and the gate-cost exclusion of shadow spend.
|
||||
* Model calls go to test/fixtures/shadow-fake-model.ts through
|
||||
* ANTHROPIC_BASE_URL and OPENAI_BASE_URL: never a real network call.
|
||||
*/
|
||||
import { afterAll, describe, expect, test } from 'bun:test';
|
||||
import * as crypto from 'node:crypto';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { SHADOW_PROMPT_SHA256, SHADOW_PROMPT_VERSION, auditMark, buildPage, claimShadow, parsePage, recordFile, runShadow, shadowPromptFile, shadowSwitch } from '../lib/autoplan-shadow';
|
||||
import { callModel } from '../lib/outside-voice-api';
|
||||
import { gateCost, renderGateCost } from '../lib/autoplan-gate-cost';
|
||||
import { readState } from '../lib/autoplan-state';
|
||||
import { reserve, settle, readLedger } from '../lib/spend-ledger';
|
||||
import type { GateList } from '../lib/gate-list';
|
||||
import { ROOT, readRun, runPhase, startSmallRun } from './helpers/autoplan-run-fixture';
|
||||
|
||||
const TMP = fs.mkdtempSync(path.join(os.tmpdir(), 'ap-shadow-'));
|
||||
const FAKE = path.join(ROOT, 'test', 'fixtures', 'shadow-fake-model.ts');
|
||||
const servers: Array<{ kill: () => void }> = [];
|
||||
afterAll(() => { for (const s of servers) s.kill(); fs.rmSync(TMP, { recursive: true, force: true }); });
|
||||
const rows = (file: string) => (fs.existsSync(file) ? fs.readFileSync(file, 'utf8').split('\n').filter(Boolean).map(l => JSON.parse(l)) : []);
|
||||
|
||||
/** A fake model server in its own process, answering step n of its script for the n-th request. */
|
||||
async function fake(name: string, script: Array<Record<string, unknown>>) {
|
||||
const dir = path.join(TMP, `fake-${name}`);
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
fs.writeFileSync(path.join(dir, 'script.json'), JSON.stringify(script));
|
||||
const proc = Bun.spawn(['bun', FAKE, dir], { stdout: 'ignore', stderr: 'inherit' });
|
||||
servers.push(proc);
|
||||
for (let i = 0; i < 200 && !fs.existsSync(path.join(dir, 'port')); i++) await Bun.sleep(25);
|
||||
const port = Number(fs.readFileSync(path.join(dir, 'port'), 'utf8'));
|
||||
const state = path.join(dir, 'state');
|
||||
fs.mkdirSync(state, { recursive: true });
|
||||
const env = { GSTACK_STATE_ROOT: state, ANTHROPIC_API_KEY: 'sk-fake-anthropic', OPENAI_API_KEY: 'sk-fake-openai', ANTHROPIC_BASE_URL: `http://127.0.0.1:${port}`, OPENAI_BASE_URL: `http://127.0.0.1:${port}/v1` };
|
||||
return { dir, env, requests: () => rows(path.join(dir, 'requests.jsonl')), receipts: () => rows(path.join(state, 'security', 'egress.jsonl')) };
|
||||
}
|
||||
|
||||
// One run at its gate (uc1 from CEO, uc2 from Eng, p1), shadow off; each case copies it and turns shadow on in run.json.
|
||||
const BASE = startSmallRun(path.join(TMP, 'base'));
|
||||
runPhase(BASE, 'ceo');
|
||||
runPhase(BASE, 'eng');
|
||||
let copies = 0;
|
||||
function gateRun(model = 'claude-opus-5-5'): string {
|
||||
const out = path.join(TMP, `run-${++copies}`);
|
||||
fs.cpSync(BASE, out, { recursive: true });
|
||||
const run = readRun(out);
|
||||
run.shadow = { enabled: true, model, source: 'env' };
|
||||
fs.writeFileSync(path.join(out, 'run.json'), JSON.stringify(run, null, 2) + '\n');
|
||||
return out;
|
||||
}
|
||||
const gateOf = (out: string) => JSON.parse(fs.readFileSync(path.join(out, 'gate.json'), 'utf8')) as GateList;
|
||||
const decider = () => fs.readFileSync(shadowPromptFile(), 'utf8');
|
||||
|
||||
describe('the decider prompt is frozen and predicts; it imports no mandate', () => {
|
||||
test('its sha256 is pinned beside its version, and every row stores it', () => {
|
||||
expect(SHADOW_PROMPT_VERSION).toBe(1);
|
||||
expect(crypto.createHash('sha256').update(fs.readFileSync(shadowPromptFile())).digest('hex')).toBe(SHADOW_PROMPT_SHA256);
|
||||
});
|
||||
test('no take-the-recommendation or keep-the-direction rule; the recommendation is a labeled input', () => {
|
||||
const text = decider();
|
||||
expect(text).not.toMatch(/\(recommended\)/i);
|
||||
expect(text).not.toMatch(/keep (the|your|its) (plan's )?direction/i);
|
||||
expect(text).not.toMatch(/sections\/ceo-only\.md/);
|
||||
expect(text).toContain("The reviewers' recommendation is one input among several");
|
||||
for (const slot of ['{{PLAN}}', '{{QUESTIONS}}', '{{IDS}}']) expect(text.split(slot).length).toBe(2);
|
||||
expect(text.trimEnd().endsWith('```')).toBe(true);
|
||||
});
|
||||
test('the CEO-only driver module does not name the CEO-only skill section', () => {
|
||||
expect(fs.readFileSync(path.join(ROOT, 'lib', 'autoplan-ceo-only.ts'), 'utf8')).not.toContain('sections/ceo-only.md');
|
||||
});
|
||||
});
|
||||
|
||||
describe('the switch is off by default and fixed at init', () => {
|
||||
const state = fs.mkdtempSync(path.join(TMP, 'cfg-'));
|
||||
const keys = { GSTACK_STATE_ROOT: state, ANTHROPIC_API_KEY: 'sk-fake', OPENAI_API_KEY: '' };
|
||||
const on = { outsideEnabled: true, ceoOnly: false };
|
||||
test('off by default, with no line', () => {
|
||||
expect(shadowSwitch(on, keys)).toEqual({ shadow: { enabled: false, reason: 'off by default', source: 'default' } });
|
||||
});
|
||||
test('config on turns it on with the default model; the env var wins over config', () => {
|
||||
fs.writeFileSync(path.join(state, 'config.yaml'), 'autoplan_shadow: on\n');
|
||||
const r = shadowSwitch(on, keys);
|
||||
expect(r.shadow).toEqual({ enabled: true, model: 'claude-opus-5-5', source: 'config' });
|
||||
expect(r.line).toStartWith('SHADOW: on model=claude-opus-5-5');
|
||||
expect(shadowSwitch(on, { ...keys, GSTACK_AUTOPLAN_SHADOW: '0' }).shadow).toMatchObject({ enabled: false, source: 'env' });
|
||||
fs.rmSync(path.join(state, 'config.yaml'));
|
||||
expect(shadowSwitch(on, { ...keys, GSTACK_AUTOPLAN_SHADOW: '1' }).shadow).toMatchObject({ enabled: true, source: 'env' });
|
||||
});
|
||||
test('no key, --no-outside, a CEO-only run, an unpriced model or a bad value give off with the reason', () => {
|
||||
const env = { ...keys, GSTACK_AUTOPLAN_SHADOW: 'on' };
|
||||
expect(shadowSwitch(on, { ...env, ANTHROPIC_API_KEY: '' }).line).toBe('SHADOW: off (no API key for claude-opus-5-5 (ANTHROPIC_API_KEY))');
|
||||
expect(shadowSwitch({ ...on, outsideEnabled: false }, env).line).toBe('SHADOW: off (--no-outside)');
|
||||
expect(shadowSwitch({ ...on, ceoOnly: true }, env).line).toBe('SHADOW: off (a CEO-only run writes no gate)');
|
||||
fs.writeFileSync(path.join(state, 'config.yaml'), 'autoplan_shadow_model: claude-unpriced-9\n');
|
||||
expect(shadowSwitch(on, env).line).toBe('SHADOW: off (claude-unpriced-9 has no price in lib/pricing.ts, so --spend-cap cannot bound it)');
|
||||
fs.rmSync(path.join(state, 'config.yaml'));
|
||||
expect(shadowSwitch(on, { ...keys, GSTACK_AUTOPLAN_SHADOW: 'yes' }).line).toBe('SHADOW: off (GSTACK_AUTOPLAN_SHADOW=yes is not on or off)');
|
||||
});
|
||||
test('a run.json without the field reads off: nothing is claimed', () => {
|
||||
const out = path.join(TMP, 'no-field');
|
||||
fs.cpSync(BASE, out, { recursive: true });
|
||||
const run = readRun(out);
|
||||
delete run.shadow;
|
||||
fs.writeFileSync(path.join(out, 'run.json'), JSON.stringify(run, null, 2) + '\n');
|
||||
expect(claimShadow(out)).toEqual({});
|
||||
expect(fs.existsSync(path.join(out, 'shadow'))).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('the reply contract: exactly one valid object per requested id', () => {
|
||||
const out = gateRun();
|
||||
const page = buildPage(readState(out), gateOf(out), decider());
|
||||
const valid = page.items.map(i => ({ id: i.item.id, action: 'decide', pick: 'a', confidence: 0.6, cites: i.findings.slice(0, 1).map(f => f.id), reason: 'r' }));
|
||||
const reply = (objs: unknown[]) => '```json\n' + JSON.stringify(objs) + '\n```';
|
||||
test('the page carries each item as rendered, with its findings, and p1 lists every finding the others name', () => {
|
||||
expect(page.items.map(i => i.item.id)).toEqual(['uc1', 'uc2', 'p1']);
|
||||
expect(page.items[0]!.frozen).toMatchObject({ item_class: 'challenge2', kind: 'approval', recommended: 'a' });
|
||||
expect(page.items[0]!.findings.map(f => f.severity).every(Boolean)).toBe(true);
|
||||
expect(page.items[2]!.findings.map(f => f.id).sort()).toEqual([...page.items[0]!.findings, ...page.items[1]!.findings].map(f => f.id).sort());
|
||||
expect(page.prompt).toContain('"id": "uc1"');
|
||||
expect(page.prompt).not.toContain('{{');
|
||||
});
|
||||
test('a good reply is clean', () => {
|
||||
expect(parsePage(reply(valid), page.items).clean).toBe(true);
|
||||
});
|
||||
const cases: Array<[string, unknown[] | string, string]> = [
|
||||
['missing', valid.slice(0, 2), 'missing'],
|
||||
['duplicate', [...valid, valid[2]], 'duplicate'],
|
||||
['unknown option', valid.map((v, i) => (i === 2 ? { ...v, pick: 'z' } : v)), 'pick "z" is not an option of p1'],
|
||||
['foreign cite', valid.map((v, i) => (i === 2 ? { ...v, cites: ['nope'] } : v)), 'cites ["nope"] name a finding outside p1\'s origin'],
|
||||
['confidence above 1', valid.map((v, i) => (i === 2 ? { ...v, confidence: 1.2 } : v)), 'confidence 1.2 is not a number from 0 to 1'],
|
||||
['bad action', valid.map((v, i) => (i === 2 ? { ...v, action: 'approve' } : v)), 'action "approve" is not decide, send_back or hold'],
|
||||
['long reason', valid.map((v, i) => (i === 2 ? { ...v, reason: 'x'.repeat(401) } : v)), 'reason is not a string of at most 400 characters'],
|
||||
['no JSON', 'I would approve.', 'no JSON array in the reply'],
|
||||
];
|
||||
for (const [name, objs, why] of cases) test(`${name} marks that item malformed`, () => {
|
||||
const p = parsePage(typeof objs === 'string' ? objs : reply(objs), page.items);
|
||||
expect(p.clean).toBe(false);
|
||||
expect(p.byId.get('p1')).toEqual({ ok: false, why });
|
||||
});
|
||||
test('a foreign id leaves the items valid but the page unclean, so it is asked again', () => {
|
||||
const p = parsePage(reply([...valid, { ...valid[0], id: 'zz9' }]), page.items);
|
||||
expect(p.foreign).toEqual(['zz9']);
|
||||
expect([...p.byId.values()].every(v => v.ok)).toBe(true);
|
||||
expect(p.clean).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('the 10% audit mark', () => {
|
||||
test('on a fixed 2,000-row corpus the marked share is what the hash gives', () => {
|
||||
const marked = Array.from({ length: 2000 }, (_, i) => auditMark('fixed-salt-0001', `run-fixture-r${1 + (i % 3)}-d${i}`)).filter(Boolean).length;
|
||||
expect(marked).toBe(Array.from({ length: 2000 }, (_, i) => BigInt('0x' + crypto.createHash('sha256').update(`fixed-salt-0001run-fixture-r${1 + (i % 3)}-d${i}`).digest('hex')) % 1000n < 100n).filter(Boolean).length);
|
||||
expect(marked / 2000).toBeGreaterThanOrEqual(0.08);
|
||||
expect(marked / 2000).toBeLessThanOrEqual(0.12);
|
||||
});
|
||||
});
|
||||
|
||||
describe('callModel: the shadow receipt, stop_reason and the OpenAI output bound (existing callers unchanged)', async () => {
|
||||
const f = await fake('transport', [{ mode: 'valid' }, { mode: 'truncated' }, { mode: 'valid' }, { mode: 'truncated' }]);
|
||||
test('a receipt override writes exactly one receipt with its sink and consent; the default stays the outside-voice receipt', async () => {
|
||||
const saved = process.env.GSTACK_STATE_ROOT;
|
||||
process.env.GSTACK_STATE_ROOT = f.env.GSTACK_STATE_ROOT;
|
||||
afterAll(() => { if (saved === undefined) delete process.env.GSTACK_STATE_ROOT; else process.env.GSTACK_STATE_ROOT = saved; });
|
||||
await callModel({ model: 'claude-opus-5-5', prompt: 'p', timeoutMs: 5000, env: f.env, maxTokens: 10, receipt: { sink: 'autoplan-shadow', consent: 'autoplan_shadow on', payloadClass: 'plan-and-gate' } });
|
||||
expect(f.receipts().map(r => [r.sink, r.consent, r.payload_class ?? r.payloadClass])).toEqual([['autoplan-shadow', 'autoplan_shadow on', 'plan-and-gate']]);
|
||||
const cut = await callModel({ model: 'claude-opus-5-5', prompt: 'p', timeoutMs: 5000, env: f.env });
|
||||
expect(cut.stop_reason).toBe('max_tokens');
|
||||
expect(f.receipts().at(-1)!.sink).toBe('outside-voice-api');
|
||||
expect(f.receipts().length).toBe(2);
|
||||
});
|
||||
test('OpenAI gets max_completion_tokens only when the caller set an output limit, and its finish_reason comes back', async () => {
|
||||
const plain = await callModel({ model: 'gpt-6.1-sol', prompt: 'p', timeoutMs: 5000, env: f.env });
|
||||
const bounded = await callModel({ model: 'gpt-6.1-sol', prompt: 'p', timeoutMs: 5000, env: f.env, maxTokens: 4000 });
|
||||
const [a, b] = f.requests().slice(2).map(r => r.body);
|
||||
expect(Object.keys(a)).toEqual(['model', 'messages']);
|
||||
expect(b.max_completion_tokens).toBe(4000);
|
||||
expect(plain.stop_reason).toBe('stop');
|
||||
expect(bounded.stop_reason).toBe('length');
|
||||
});
|
||||
});
|
||||
|
||||
describe('runShadow: retry, truncation, failures and the deadline', async () => {
|
||||
const ids = (out: string) => rows(recordFile(out, 1));
|
||||
const shadowed = async (name: string, script: Array<Record<string, unknown>>, model?: string, deadlineMs?: number) => {
|
||||
const f = await fake(name, script);
|
||||
const out = gateRun(model);
|
||||
const { claim } = claimShadow(out);
|
||||
expect(claim).toEqual({ gate_rev: 1, retrospective: false });
|
||||
const started = Date.now();
|
||||
const outcome = await runShadow(out, claim!, { env: { ...process.env, ...f.env }, deadlineMs });
|
||||
return { f, out, outcome, ms: Date.now() - started };
|
||||
};
|
||||
test('a good reply records every item, keeps the prompt and the raw reply, and settles one shadow-labeled call', async () => {
|
||||
const { f, out, outcome } = await shadowed('good', [{ mode: 'valid' }]);
|
||||
expect(outcome.status).toBe('recorded');
|
||||
expect(outcome.line).toMatch(/^SHADOW: recorded model=claude-opus-5-5 usd=0\.0140$/);
|
||||
expect(ids(out).map(r => [r.item, r.status, r.pick, r.rule_pick, r.prompt_sha256 === SHADOW_PROMPT_SHA256])).toEqual([['uc1', 'recorded', 'a', 'a', true], ['uc2', 'recorded', 'a', 'a', true], ['p1', 'recorded', 'a', 'a', true]]);
|
||||
expect(f.requests().map(r => r.body.max_tokens)).toEqual([4000]);
|
||||
expect(fs.existsSync(path.join(out, 'shadow', 'r1.prompt.md'))).toBe(true);
|
||||
expect(fs.readFileSync(path.join(out, 'shadow', 'shadow-r1-1.reply.txt'), 'utf8')).toContain('fake prediction');
|
||||
expect(readLedger(path.join(out, 'spend.json')).settled.filter(s => s.label === 'shadow').map(s => [s.attempt, s.usd])).toEqual([['shadow-r1-1', 0.014]]);
|
||||
expect(fs.existsSync(path.join(out, 'shadow', 'r1.pending'))).toBe(false);
|
||||
});
|
||||
test('a malformed page is asked once more; items valid in either reply record, the rest record malformed', async () => {
|
||||
const fixed = await shadowed('missing-then-valid', [{ mode: 'missing' }, { mode: 'valid' }]);
|
||||
expect(fixed.f.requests().length).toBe(2);
|
||||
expect(ids(fixed.out).every(r => r.status === 'recorded')).toBe(true);
|
||||
const still = await shadowed('bad-twice', [{ mode: 'bad_option' }, { mode: 'bad_option' }]);
|
||||
expect(still.f.requests().length).toBe(2);
|
||||
expect(ids(still.out).map(r => r.status)).toEqual(['malformed', 'recorded', 'recorded']);
|
||||
expect(ids(still.out)[0]).toMatchObject({ pick: null, audit: false, malformed: 'pick "z" is not an option of uc1' });
|
||||
expect(still.outcome.line).toContain('malformed=1/3');
|
||||
});
|
||||
test('a truncated reply raises max_tokens once to 8,000 (no page retry); truncated twice fails truncated', async () => {
|
||||
const raised = await shadowed('truncated-once', [{ mode: 'truncated' }, { mode: 'valid' }]);
|
||||
expect(raised.f.requests().map(r => r.body.max_tokens)).toEqual([4000, 8000]);
|
||||
expect(raised.outcome.status).toBe('recorded');
|
||||
const twice = await shadowed('truncated-twice', [{ mode: 'truncated' }]);
|
||||
expect(twice.f.requests().length).toBe(2);
|
||||
expect(twice.outcome.line).toBe('SHADOW: failed truncated');
|
||||
expect(fs.existsSync(recordFile(twice.out, 1))).toBe(false);
|
||||
});
|
||||
test('an OpenAI shadow model sends max_completion_tokens and treats finish_reason length as truncated', async () => {
|
||||
const r = await shadowed('openai', [{ mode: 'truncated' }, { mode: 'valid' }], 'gpt-6.1-sol');
|
||||
expect(r.f.requests().map(x => [x.path, x.body.max_completion_tokens])).toEqual([['/v1/chat/completions', 4000], ['/v1/chat/completions', 8000]]);
|
||||
expect(r.outcome.status).toBe('recorded');
|
||||
});
|
||||
test('a 500 fails http and settles the call at 0; the journal names the code', async () => {
|
||||
const r = await shadowed('http500', [{ mode: 'http500' }]);
|
||||
expect(r.outcome.line).toBe('SHADOW: failed http');
|
||||
expect(readLedger(path.join(r.out, 'spend.json')).settled.filter(s => s.label === 'shadow').map(s => s.usd)).toEqual([0]);
|
||||
});
|
||||
test('one deadline covers the call and its retry: a slow malformed reply leaves the retry no time', async () => {
|
||||
const r = await shadowed('deadline-retry', [{ mode: 'missing', delay_ms: 1200 }, { mode: 'valid', delay_ms: 1200 }], undefined, 1500);
|
||||
expect(r.ms).toBeLessThan(2400);
|
||||
expect(r.outcome.status).toBe('recorded');
|
||||
expect(ids(r.out).map(x => x.status)).toEqual(['recorded', 'recorded', 'malformed']);
|
||||
const slow = await shadowed('deadline-call', [{ mode: 'valid', delay_ms: 3000 }], undefined, 800);
|
||||
expect(slow.outcome.line).toBe('SHADOW: failed timeout');
|
||||
expect(slow.ms).toBeLessThan(2500);
|
||||
expect(readLedger(path.join(slow.out, 'spend.json')).settled.filter(s => s.label === 'shadow').map(s => s.usd)).toEqual(['unknown']);
|
||||
});
|
||||
test('a record already published makes a later claim a no-op: no second call is paid', async () => {
|
||||
const { f, out } = await shadowed('noop', [{ mode: 'valid' }]);
|
||||
const before = fs.readFileSync(recordFile(out, 1));
|
||||
expect(claimShadow(out)).toEqual({});
|
||||
expect(f.requests().length).toBe(1);
|
||||
expect(fs.readFileSync(recordFile(out, 1)).equals(before)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('gate cost: shadow spend counts against the cap and shows in GATE_SPEND, never in the per-voice price', () => {
|
||||
test('a $5 voice settle followed by a shadow settle keeps GATE_REC_COST at x 5.00', () => {
|
||||
const out = path.join(TMP, 'cost');
|
||||
fs.mkdirSync(out, { recursive: true });
|
||||
const file = path.join(out, 'spend.json');
|
||||
reserve(file, 'r-a1', 5, { cap: 5.2, label: 'ceo/native' }); settle(file, 'r-a1', 'unknown');
|
||||
reserve(file, 'shadow-r1-1', 0.15, { label: 'shadow' }); settle(file, 'shadow-r1-1', 0.014);
|
||||
const list: GateList = { schema_version: 1, run: 'r', gate_rev: 1, items: [
|
||||
{ id: 'd1', title: 't', kind: 'auto', recommended: 'b', phase: 'ceo', options: [{ key: 'a', text: 'keep', changes_plan: false }, { key: 'b', text: 'change', changes_plan: true }] },
|
||||
{ id: 'p1', title: 'Plan approval', kind: 'approval', recommended: 'b', options: [{ key: 'a', text: 'x' }, { key: 'b', text: 'y' }] }] };
|
||||
const c = gateCost(out, list, { phase_order: ['ceo', 'eng'], outside_enabled: true, spend_cap_usd: 5.2 });
|
||||
expect(c.per_voice_usd).toBe(5);
|
||||
expect(c.attempts).toBe(1);
|
||||
expect(c.shadow_usd).toBe(0.014);
|
||||
expect(c.summary.worst_case_usd).toBe(5.014);
|
||||
const [spend, rec] = renderGateCost(c);
|
||||
expect(spend).toContain('attempts=1 shadow=0.0140');
|
||||
expect(rec).toContain('× 5.00');
|
||||
expect(rec).toContain('(over the cap: raise it with gstack-autoplan budget before answering rec)');
|
||||
});
|
||||
});
|
||||
+1
@@ -128,6 +128,7 @@
|
||||
{ "file": "lib/autoplan-run.ts", "movedFrom": "(new: next/submit/resume over lib/autoplan-state.ts, B2)" },
|
||||
{ "file": "lib/autoplan-gate.ts", "movedFrom": "(new: phase close, gate build, answer and reopen, B2/B9)" },
|
||||
{ "file": "lib/autoplan-ceo-only.ts", "movedFrom": "(new: the CEO-only driver run's finish, report and refusals, GST-537)" },
|
||||
{ "file": "lib/autoplan-shadow.ts", "movedFrom": "(new: shadow mode for the CEO-only decider: switch, lease, call and record, GST-537 PR 3)" },
|
||||
{ "file": "lib/autoplan-export.ts", "movedFrom": "(new: the unattended export file set, B2/B12)" },
|
||||
{ "file": "lib/spec-review-cap.ts", "movedFrom": "(new: the spec-review loop at the cap: numbered gaps and deterministic fix verification, B8)" },
|
||||
{ "file": "bin/gstack-autoplan", "movedFrom": "(new: thin entrypoint over lib/autoplan-run.ts, B2)" },
|
||||
|
||||
Vendored
+59
@@ -0,0 +1,59 @@
|
||||
#!/usr/bin/env -S bun --no-env-file --config=/dev/null
|
||||
/**
|
||||
* A local stand-in for the Anthropic Messages and OpenAI Chat Completions APIs,
|
||||
* so the shadow-mode tests (test/autoplan-shadow*.test.ts) never make a real
|
||||
* network call: the tests point ANTHROPIC_BASE_URL and OPENAI_BASE_URL at it.
|
||||
* Usage: bun test/fixtures/shadow-fake-model.ts <dir>. It writes <dir>/port,
|
||||
* appends every request to <dir>/requests.jsonl, and answers the n-th request
|
||||
* with step n of <dir>/script.json (the last step repeats; default valid):
|
||||
* { "mode": valid | missing | duplicate | foreign | bad_option | bad_cite | bad_conf | garbage | truncated | send_back | http500,
|
||||
* "delay_ms"?: n }. A valid reply picks each question's first option with the
|
||||
* question's first finding as its cite; usage is 1,000 input and 500 output tokens.
|
||||
*/
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
|
||||
const dir = process.argv[2]!;
|
||||
interface Step { mode: string; delay_ms?: number }
|
||||
interface Question { id: string; options: Array<{ key: string }>; findings: Array<{ id: string }> }
|
||||
let served = 0;
|
||||
|
||||
function questions(prompt: string): Question[] {
|
||||
const at = prompt.indexOf('## The questions');
|
||||
const block = /```json\n([\s\S]*?)\n```/.exec(prompt.slice(at))?.[1];
|
||||
return block ? JSON.parse(block) : [];
|
||||
}
|
||||
|
||||
function answer(mode: string, qs: Question[]): string {
|
||||
const rows: Array<Record<string, unknown>> = qs.map(q => ({ id: q.id, action: mode === 'send_back' ? 'send_back' : 'decide', pick: q.options[0]!.key, confidence: 0.75, cites: q.findings.slice(0, 1).map(f => f.id), reason: 'fake prediction' }));
|
||||
if (mode === 'missing') rows.pop();
|
||||
if (mode === 'duplicate') rows.push({ ...rows[0]! });
|
||||
if (mode === 'foreign') rows.push({ ...rows[0]!, id: 'zz9' });
|
||||
if (mode === 'bad_option') rows[0]!.pick = 'z';
|
||||
if (mode === 'bad_cite') rows[0]!.cites = ['not-a-finding'];
|
||||
if (mode === 'bad_conf') rows[0]!.confidence = 1.5;
|
||||
if (mode === 'garbage') return 'I would approve it.';
|
||||
if (mode === 'truncated') return '```json\n[{"id": "' + (qs[0]?.id ?? 'p1') + '", "act';
|
||||
return 'Here is my prediction.\n```json\n' + JSON.stringify(rows, null, 2) + '\n```\n';
|
||||
}
|
||||
|
||||
const server = Bun.serve({
|
||||
port: 0, hostname: '127.0.0.1',
|
||||
async fetch(req) {
|
||||
const url = new URL(req.url);
|
||||
const body = await req.json() as Record<string, any>;
|
||||
let steps: Step[] = [{ mode: 'valid' }];
|
||||
try { steps = JSON.parse(fs.readFileSync(path.join(dir, 'script.json'), 'utf8')); } catch { /* default */ }
|
||||
const step = steps[Math.min(served, steps.length - 1)]!;
|
||||
served++;
|
||||
fs.appendFileSync(path.join(dir, 'requests.jsonl'), JSON.stringify({ n: served, path: url.pathname, mode: step.mode, body, at: Date.now() }) + '\n');
|
||||
if (step.delay_ms) await Bun.sleep(step.delay_ms);
|
||||
if (step.mode === 'http500') return Response.json({ error: { message: 'fake overloaded' } }, { status: 500 });
|
||||
const text = answer(step.mode, questions(String(body.messages?.[0]?.content ?? '')));
|
||||
const cut = step.mode === 'truncated';
|
||||
if (url.pathname.endsWith('/v1/messages')) return Response.json({ model: body.model, content: [{ type: 'text', text }], stop_reason: cut ? 'max_tokens' : 'end_turn', usage: { input_tokens: 1000, output_tokens: 500 } });
|
||||
return Response.json({ model: body.model, choices: [{ message: { content: text }, finish_reason: cut ? 'length' : 'stop' }], usage: { prompt_tokens: 1000, completion_tokens: 500 } });
|
||||
},
|
||||
});
|
||||
fs.writeFileSync(path.join(dir, 'port.tmp'), String(server.port));
|
||||
fs.renameSync(path.join(dir, 'port.tmp'), path.join(dir, 'port'));
|
||||
Reference in new issue
Block a user