mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 22:14:25 +01:00
* feat(ship-policy): changelog_fragments key; restamp folds fragments; tree receipt accepts the fold Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(ship-queue): land, the per-PR merge step; Version Gate enforces fragments Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(ship-queue): land --repair after drift; hook base sha; docs and CHANGELOG Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: timeout on the land gh shim's git spawn Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(ship): /ship writes a CHANGELOG fragment under changelog_fragments; land checks the depth-checked receipt (#3567) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(land): keeps the base's four-part package.json (#3557) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(ship-queue): land waits for CI through runScriptSync (script-invocation tripwire) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: land, fragment and Version Gate cases from /ship's coverage audit Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(windows): leave test/ship-queue-land.test.ts out of the Windows-safe subset (gh shim cannot launch there) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(ship-policy): rollback lands open fragment PRs first; restamp folds only with the key on Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(land): close pre-landing review findings land without --no-wait now compares the pushed branch head and polls the stamp commit's checks, refuses a PR whose base is not the stamped base, releases the lock on a pre-push failure, keeps --break-lock off drift locks, resumes from a lost journal via the stamp commit's pin, honours history merge-only, and refuses a fragment whose text is already released on the base. Version Gate stops applying open claims to a fragments stamp. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(land): close adversarial and outside-voice findings land reconciles a merge GitHub completed behind an error reply, refuses --break-lock against a holder that just entered a push or a merge, keeps a repair takeover on the lock itself (any checkout can land the repair PR), resumes --repair after a failed PR create, releases the lock on QUEUE_LAND_BASE_BUSY, never writes through a symlinked release file, keeps a package.json version source's other edits, and refuses an empty object id instead of pushing a deletion. A carried-back fragment is matched on whole released lines. Under changelog_fragments Version Gate accepts a stamped head only as a [pin, base] merge-step commit with valid fragments. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(land): refuse a pin without a READY receipt; symlinked release files refuse before restamp Under changelog_fragments, gstack-ship-queue ready records a READY receipt (refs/notes/gstack-trains) once its pregate, full_e2e and /ship receipt checks pass, and land refuses a pin without one (fix: run gstack-ship-queue ready --pr N), --dry-run included (GStack Minder ruling, 2026-10-29). The merged tree's release files may not be symlinks, so restamp never writes outside the scratch worktree. --repair adopts an existing branch only when it holds land's repair commit. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(pregate): clear's stamp-only substitution applies the changelog_fragments fold; docs match land and ready gstack-pregate clear and gstack-tree-receipt now classify through one classifyFiles, so under changelog_fragments a receipt for the fragment head clears the stamped head when the stamp is exactly the fold. The ship-policy reference names ready's receipt push, land's own check polling, the key-off refusal and the /ship changelog_fragments line. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(pregate, ship-policy): the fragment fold's two receipt rules, and clear applies them too Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(ship-queue): stamped train prefixes S_i with the self-check (PR B) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(ship-queue): train build stamps S_i under fragments; train land publishes S_i exactly; backstop covers exact trees (PR B) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: stamped trains and train land; QUEUE_TRAIN_STAMP_MISMATCH anchor Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * captain templates: fragments instead of [Unreleased]; no slot call for stamped trains Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: timeouts on the train-land gh wrapper's sync spawns Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(ship-queue): train land checks the depth-checked receipt (#3567); regenerate the doctor anchors Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(train-stamp): S_i keeps the base's four-part package.json (#3557) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(windows): leave test/ship-queue-train-land.test.ts out of the Windows-safe subset (same gh shim as the train tests) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(train-land): close pre-landing review and outside-voice findings - train land checks commit-tree and stamps with a fixed identity and date, journals before the push, and adopts its own stamp head on a rerun; a failed commit-tree never pushes a branch deletion - a symlinked release file in the merged tree refuses before the train stamp writes - mergeStep merges a conflicted manifest three-way with the predecessor's version (mergeManifest, shared with land) - train land refuses a PR retargeted off the base and a failed base refresh - Version Gate skips a released fragment carried in by a stacked pin instead of counting its bump - bisect of a stamped train tests T0 and S_1..S_k Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(train-land): a rerun whose head is still the pin re-enters the stamp step after a journaled, unpushed stamp Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(train-land): prove the record's S_i is on the gated chain before stamping; a merge main does not show yet pends Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(train-land): pin S_i to the gated head's exact prefix; an out-of-order slot refuses without freezing; stamps never sign Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(train-land): pin S_i to slot position; a failed push or unreadable PR never reads as a move; reruns of landed slots confirm; T0 in the out-of-order refusal Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(train-land): verify the whole stamped chain from git before landing; require changelog_fragments.required_checks green on the stamp commit Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(ship-policy, troubleshooting, harness): train land refusals, resume and required checks; stamped bisect; three-way manifest merges Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * ship: write the Steps 12-16 range with an ASCII hyphen so the skeleton fits maxSkeletonBytes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
341 lines
25 KiB
TypeScript
341 lines
25 KiB
TypeScript
/**
|
||
* Train verdict, bisect and the one preregistered rebuild
|
||
* (docs/ship-policy.md#run-a-merge-train; plan GSTA-49 lane T1, P3).
|
||
*
|
||
* The verdict reads the train's own gate run and the record artifact that run
|
||
* publishes (with the `gstack/full-gate` check run) on the train head; it
|
||
* only reads, never writes a check. Green needs a complete census from the
|
||
* run's own plan manifest. A red train stays red: the bisect runs only its
|
||
* red cases, one trial on every prefix T0..Tk, and attributes by pattern.
|
||
* Rebuild-once (Garry, 2026-10-18; to be recorded in EVAL_POLICY beside
|
||
* infraRedispatch): an all-green bisect earns one unchanged rebuild.
|
||
*/
|
||
import * as path from 'node:path';
|
||
import { randomBytes } from 'node:crypto';
|
||
import { DISPATCH_INPUTS, listDispatchRuns, makeNonce, runForNonce } from '../scripts/eval-bg';
|
||
import type { QueueContext } from './ship-queue';
|
||
import { git } from './ship-queue-gate';
|
||
import {
|
||
checkRun, downloadJson, execFor, asRun, remoteBranchSha, repoSlug, runView, type CheckInfo, type RunInfo,
|
||
} from './ship-queue-train-github';
|
||
import {
|
||
attemptsOf, loadBisect, nowIso, persist, trainDir, trainFile,
|
||
type AttemptRecord, type BisectRecord, type CasePanel, type GateOutcome, type PrefixResult, type TrainRecord,
|
||
} from './ship-queue-train-record';
|
||
import {
|
||
dispatchAttempt, effect, findTrain, gateDispatch, nextFor, outFailed, outOk, outPending, outRefused, prsArgs, save, trainEnv,
|
||
type TrainEnv, type TrainOut,
|
||
} from './ship-queue-train';
|
||
|
||
/** EVAL_POLICY.infraRedispatch (test/helpers/periodic-exclude-data.ts): one re-dispatch of an INFRA/INCOMPLETE-only census. */
|
||
export const INFRA_REDISPATCH = 1;
|
||
/** Rebuild-once (Garry, 2026-10-18): an all-green bisect earns exactly one unchanged rebuild; pinned to EVAL_POLICY by its test. */
|
||
export const REBUILD_ONCE = 1;
|
||
|
||
const Q = 'gstack-ship-queue';
|
||
|
||
/** The record artifact the train's own gate run publishes (plan T1b), read by verdict, verify and the backstop. */
|
||
export interface TrainGateRecord {
|
||
schema: 1; kind: 'train-gate'; nonce: string; head_sha: string; tree: string; verdict: 'GREEN' | 'RED';
|
||
census: { complete: boolean; planned: { rule: number; judge: number }; executed: { rule: number; judge: number };
|
||
unbound: number; partial_shards: number; cancelled_shards: number; killed_before_upload: number };
|
||
panels: CasePanel[];
|
||
}
|
||
/** One bisect prefix's artifact: the red cases on that prefix, one trial each. */
|
||
export interface BisectPrefixRecord { schema: 1; kind: 'train-bisect-prefix'; nonce: string; prefix_sha: string; checked_out_sha: string; panels: CasePanel[] }
|
||
|
||
export const bisectArtifact = (recordArtifact: string, sha: string) => `${recordArtifact}-bisect-${sha.slice(0, 12)}`;
|
||
|
||
export interface GateClass { outcome: GateOutcome; detail: string; red: string[]; panels: CasePanel[]; infraEligible: boolean; cost: number | null }
|
||
|
||
/** Every paid-report class mapped to one train outcome; green only for a complete, bound, published census. */
|
||
export function classifyGate(raw: unknown, expect: { nonce: string; head: string; tree: string }, run: Pick<RunInfo, 'status' | 'conclusion'>, check: CheckInfo | null): GateClass {
|
||
const base = { red: [] as string[], panels: [] as CasePanel[], infraEligible: false, cost: null as number | null };
|
||
if (run.status !== 'completed') return { ...base, outcome: 'PENDING', detail: `run is ${run.status}` };
|
||
const rec = raw as TrainGateRecord | null;
|
||
if (!rec || rec.schema !== 1 || rec.kind !== 'train-gate' || !Array.isArray(rec.panels) || !rec.census) {
|
||
return { ...base, outcome: 'INCOMPLETE', infraEligible: true, detail: `no train record artifact (run concluded ${run.conclusion ?? 'unknown'}; a slice killed before its upload step counts here)` };
|
||
}
|
||
const panels = rec.panels;
|
||
const known = panels.map(p => p.cost_usd).filter((c): c is number => typeof c === 'number');
|
||
const cost = known.length ? Math.round(known.reduce((a, b) => a + b, 0) * 100) / 100 : null;
|
||
const withPanels = { ...base, panels, cost };
|
||
if (rec.nonce !== expect.nonce || rec.head_sha !== expect.head || rec.tree !== expect.tree) {
|
||
return { ...withPanels, outcome: 'INCOMPLETE', detail: `the record is bound to nonce ${rec.nonce} head ${String(rec.head_sha).slice(0, 12)} tree ${String(rec.tree).slice(0, 12)}, not this train's attempt` };
|
||
}
|
||
const red = panels.filter(p => p.status === 'FAIL' && p.red_class === 'VERDICT').map(p => p.case).sort();
|
||
if (red.length) return { ...withPanels, outcome: 'RED', red, detail: `${red.length} red case(s): ${red.join(', ')}` };
|
||
const c = rec.census;
|
||
const gaps = [
|
||
...(c.planned.rule + c.planned.judge <= 0 ? ['the plan manifest lists no cases'] : []),
|
||
...(c.executed.rule !== c.planned.rule ? [`rule ${c.executed.rule}/${c.planned.rule} executed`] : []),
|
||
...(c.executed.judge !== c.planned.judge ? [`judge ${c.executed.judge}/${c.planned.judge} executed`] : []),
|
||
...(c.unbound ? [`${c.unbound} unbound result`] : []), ...(c.partial_shards ? [`${c.partial_shards} partial shard`] : []),
|
||
...(c.cancelled_shards ? [`${c.cancelled_shards} cancelled shard`] : []), ...(c.killed_before_upload ? [`${c.killed_before_upload} slice killed before upload`] : []),
|
||
...(!c.complete ? ['census marked incomplete'] : []),
|
||
...panels.filter(p => p.status !== 'PASS').map(p => `${p.case} ${p.status}${p.red_class ? ` (${p.red_class})` : ''}`),
|
||
];
|
||
if (gaps.length) return { ...withPanels, outcome: 'INCOMPLETE', infraEligible: !panels.some(p => p.status === 'SKIPPED'), detail: gaps.join('; ') };
|
||
if (run.conclusion !== 'success' || rec.verdict !== 'GREEN') return { ...withPanels, outcome: 'INCOMPLETE', detail: `run concluded ${run.conclusion ?? 'unknown'} with record verdict ${rec.verdict}` };
|
||
if (!check || check.conclusion !== 'success') return { ...withPanels, outcome: 'INCOMPLETE', detail: `the run did not publish a successful check run on the train head (${check ? `${check.status}/${check.conclusion}` : 'missing'})` };
|
||
return { ...withPanels, outcome: 'GREEN', detail: `rule ${c.executed.rule}/${c.planned.rule}, judge ${c.executed.judge}/${c.planned.judge}` };
|
||
}
|
||
|
||
const gateAttempts = (t: TrainRecord) => attemptsOf(t).filter(a => a.purpose !== 'bisect');
|
||
const costText = (a: AttemptRecord) => {
|
||
if (a.cost_usd === null) return '';
|
||
const unknown = a.panels.filter(p => p.cost_usd === null).length;
|
||
return ` cost=$${a.cost_usd.toFixed(2)}${unknown ? ` known + ${unknown} of ${a.panels.length} case(s) cost unknown` : ''}`;
|
||
};
|
||
const attemptLine = (a: AttemptRecord) => `ATTEMPT: ${a.id} ${a.purpose} run ${a.run?.id ?? '-'} ${a.outcome ?? a.state}${a.red_cases.length ? ` red=${a.red_cases.join(',')}` : ''}${costText(a)}${a.detail ? ` (${a.detail})` : ''}`;
|
||
|
||
function casePrice(env: TrainEnv, red: AttemptRecord, id: string): number {
|
||
return red.panels.find(p => p.case === id)?.cost_usd ?? env.train.trial_reservation_usd;
|
||
}
|
||
|
||
/** The whole-bisect reservation for a red attempt: every red case on every prefix. */
|
||
export function bisectReservation(env: TrainEnv, t: TrainRecord, red: AttemptRecord): number {
|
||
const perPrefix = red.red_cases.reduce((sum, id) => sum + casePrice(env, red, id), 0);
|
||
return Math.ceil(perPrefix * t.prefixes.length * 100) / 100;
|
||
}
|
||
|
||
function unknownDispatch(t: TrainRecord, a: AttemptRecord): TrainOut {
|
||
return outPending({ code: 'QUEUE_TRAIN_DISPATCH_UNKNOWN', message: `the ${a.purpose} dispatch ${a.id} (nonce ${a.nonce}) is unconfirmed` }, [attemptLine(a)], `${Q} train status --train ${t.nonce}`, trainFile(t.nonce));
|
||
}
|
||
|
||
/** `train verdict`: read the current gate attempt's run and record; print every attempt, so a rebuild reports both runs. */
|
||
export function trainVerdict(ctx: QueueContext, o: { train: string }): TrainOut {
|
||
const env = trainEnv(ctx);
|
||
if ('code' in env) return outRefused(env, [], 'none', 'none');
|
||
const t = findTrain(env, o.train);
|
||
if (!t) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `no train record ${o.train} here or in refs/notes/gstack-trains` }, [], `${Q} train status`, 'none');
|
||
const run = trainFile(t.nonce);
|
||
const a = gateAttempts(t).at(-1);
|
||
if (!a) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `train ${t.nonce} has no gate dispatch yet` }, [], nextFor(t), run);
|
||
if (a.state === 'dispatching' || a.state === 'unknown') return unknownDispatch(t, a);
|
||
if (a.state === 'absent') return outRefused({ code: 'QUEUE_TRAIN_DISPATCH_UNKNOWN', message: `the dispatch ${a.nonce} never started (train status found no run)` }, [attemptLine(a)], `${Q} train build --train ${t.nonce}`, run);
|
||
const slug = repoSlug(ctx);
|
||
if (!a.outcome || a.outcome === 'PENDING') {
|
||
const view = slug && a.run ? runView(ctx, slug, a.run.id) : null;
|
||
if (!slug || !view) return outPending({ code: 'QUEUE_TRAIN_PENDING', message: `cannot read run ${a.run?.id} through gh` }, [attemptLine(a)], `${Q} train verdict --train ${t.nonce}`, run);
|
||
const record = view.status === 'completed' ? downloadJson(ctx, slug, a.run!.id, env.train.record_artifact, path.join(trainDir(t.nonce), 'scratch')) : null;
|
||
const check = view.status === 'completed' ? checkRun(ctx, t.head, env.train.full_gate_check) : null;
|
||
const c = classifyGate(record, { nonce: a.nonce, head: t.head, tree: t.tree }, view, check);
|
||
Object.assign(a, { outcome: c.outcome, detail: c.detail, red_cases: c.red, panels: c.panels, infra_eligible: c.infraEligible, cost_usd: c.cost, state: view.status === 'completed' ? 'completed' : a.state });
|
||
if (c.outcome !== 'PENDING') {
|
||
t.state = c.outcome === 'GREEN' ? 'green' : c.outcome === 'RED' ? (a.purpose === 'rebuild' ? 'unattributed' : 'red') : 'incomplete';
|
||
effect(t, `verdict ${a.id}: ${c.outcome}`);
|
||
}
|
||
save(env, t, a);
|
||
}
|
||
const lines = [...gateAttempts(t).map(attemptLine), `VERDICT: ${a.outcome} train ${t.nonce} run ${a.run?.id ?? '-'} head ${t.head.slice(0, 12)} tree ${t.tree.slice(0, 12)}`];
|
||
if (a.outcome === 'PENDING') return outPending({ code: 'QUEUE_TRAIN_PENDING', message: `the gate run ${a.run?.id} has not finished (${a.detail})` }, lines, `${Q} train verdict --train ${t.nonce}`, run);
|
||
if (a.outcome === 'GREEN') return outOk(lines, nextFor(t), run);
|
||
if (a.outcome === 'RED') {
|
||
const next = a.purpose === 'rebuild'
|
||
? `second red stays red: route ${a.red_cases.join(',')} to a repair lane (GStack Minder), then ${Q} train build --prs <pins without the PRs held>`
|
||
: `${Q} train bisect --train ${t.nonce} --spend-cap ${bisectReservation(env, t, a).toFixed(2)}`;
|
||
return outFailed({ code: 'QUEUE_TRAIN_RED', message: `train ${t.nonce} is red on ${a.red_cases.join(', ')}` }, lines, next, run);
|
||
}
|
||
const used = gateAttempts(t).filter(x => x.purpose === 'infra-redispatch').length;
|
||
const next = a.infra_eligible && used < INFRA_REDISPATCH
|
||
? `${Q} train rebuild --train ${t.nonce} --infra (the single re-dispatch EVAL_POLICY.infraRedispatch allows; report both runs)`
|
||
: `diagnose run ${a.run?.url ?? a.run?.id}; the train stays incomplete (GStack Minder routes it)`;
|
||
return outFailed({ code: 'QUEUE_TRAIN_INCOMPLETE', message: `train ${t.nonce} gate is incomplete: ${a.detail}` }, lines, next, run);
|
||
}
|
||
|
||
export interface BisectClass { kind: 'attributed' | 'red-on-main' | 'unattributed'; all_green: boolean; prs: string[]; patterns: Record<string, string> }
|
||
|
||
/**
|
||
* Pattern attribution over T0..Tk (one trial each, so only shapes count):
|
||
* monotone green-then-red from Ti blames slot i's PR; red everywhere,
|
||
* T0 included, is main; all green, or anything else, is unattributed.
|
||
*/
|
||
export function classifyBisect(cases: string[], results: Array<Record<string, PrefixResult>>, slots: Array<{ pr: string }>): BisectClass {
|
||
const patterns: Record<string, string> = {};
|
||
const blamed = new Set<string>();
|
||
let main = false; let other = false; let green = 0;
|
||
for (const c of cases) {
|
||
const seq = results.map(r => (r[c] === 'red' ? 'R' : 'G')).join('');
|
||
patterns[c] = seq;
|
||
const first = seq.indexOf('R');
|
||
if (first < 0) green++;
|
||
else if (first === 0 && !seq.includes('G')) main = true;
|
||
else if (first > 0 && !seq.slice(first).includes('G')) blamed.add(slots[first - 1]!.pr);
|
||
else other = true;
|
||
}
|
||
if (main) return { kind: 'red-on-main', all_green: false, prs: [], patterns };
|
||
if (other || !blamed.size) return { kind: 'unattributed', all_green: green === cases.length, prs: [], patterns };
|
||
return { kind: 'attributed', all_green: false, prs: [...blamed].sort((a, b) => Number(a) - Number(b)), patterns };
|
||
}
|
||
|
||
function prefixResults(raw: unknown, sha: string, nonce: string, cases: string[]): { results: Record<string, PrefixResult>; cost: number } | null {
|
||
const rec = raw as BisectPrefixRecord | null;
|
||
if (!rec || rec.schema !== 1 || rec.kind !== 'train-bisect-prefix' || rec.nonce !== nonce || rec.prefix_sha !== sha || rec.checked_out_sha !== sha || !Array.isArray(rec.panels)) return null;
|
||
const results: Record<string, PrefixResult> = {};
|
||
let cost = 0;
|
||
for (const p of rec.panels) {
|
||
if (!cases.includes(p.case)) continue;
|
||
cost += typeof p.cost_usd === 'number' ? p.cost_usd : 0;
|
||
if (p.status === 'PASS') results[p.case] = 'green';
|
||
else if (p.status === 'FAIL' && p.red_class === 'VERDICT') results[p.case] = 'red';
|
||
}
|
||
return { results, cost };
|
||
}
|
||
|
||
/** Collect a finished bisect dispatch's prefix artifacts into the ledger; a still-running one returns its run state. */
|
||
function collect(env: TrainEnv, slug: string, t: TrainRecord, b: BisectRecord, a: AttemptRecord): string | null {
|
||
const view = a.run ? runView(env.ctx, slug, a.run.id) : null;
|
||
if (!view) return 'unreadable through gh';
|
||
if (view.status !== 'completed') return view.status;
|
||
let got = 0;
|
||
for (const index of a.prefixes) {
|
||
const p = b.prefixes[index]!;
|
||
const r = prefixResults(downloadJson(env.ctx, slug, a.run!.id, bisectArtifact(env.train.record_artifact, p.sha), path.join(trainDir(t.nonce), 'scratch')), p.sha, a.nonce, b.cases);
|
||
if (!r) continue;
|
||
Object.assign(p.results, r.results);
|
||
b.spent_usd = Math.round((b.spent_usd + r.cost) * 100) / 100;
|
||
got++;
|
||
}
|
||
Object.assign(a, { state: 'completed', detail: `collected ${got} of ${a.prefixes.length} prefix result(s)` });
|
||
return null;
|
||
}
|
||
|
||
function bisectOutcome(env: TrainEnv, t: TrainRecord, b: BisectRecord): TrainOut {
|
||
const c = classifyBisect(b.cases, b.prefixes.map(p => p.results), t.slots);
|
||
b.outcome = { ...c, at: nowIso() };
|
||
t.state = c.kind;
|
||
effect(t, `bisect: ${c.kind}${c.prs.length ? ' #' + c.prs.join(',#') : ''}`);
|
||
b.updated_at = nowIso();
|
||
persist(env.ctx.repoRoot, [b], { remote: env.remote });
|
||
save(env, t);
|
||
const run = trainFile(t.nonce);
|
||
const lines = [...Object.entries(c.patterns).map(([id, s]) => `PATTERN: ${id} ${s.split('').map((x, i) => `T${i}=${x}`).join(' ')}`), `BISECT: spent $${b.spent_usd.toFixed(2)} of cap $${b.spend_cap.toFixed(2)}`];
|
||
if (c.kind === 'attributed') return outOk([...lines, `BISECT: ATTRIBUTED ${c.prs.map(p => '#' + p).join(',')}`], nextFor(t), run);
|
||
if (c.kind === 'red-on-main') return outFailed({ code: 'QUEUE_TRAIN_RED_ON_MAIN', message: `${b.cases.join(', ')} red on every prefix, T0 (main) included` }, [...lines, 'BISECT: RED_ON_MAIN'], nextFor(t), run);
|
||
return outFailed({ code: 'QUEUE_TRAIN_UNATTRIBUTED', message: c.all_green ? 'every prefix was green: the red is unattributed; recovery: the one preregistered rebuild' : 'the pattern is not monotone: recovery: a repair lane, then a changed train' },
|
||
[...lines, `BISECT: UNATTRIBUTED (${c.all_green ? 'all green' : 'not monotone'})`], nextFor(t), run);
|
||
}
|
||
|
||
/** The commits a bisect tests: T0 and the plain prefixes, or for a stamped train T0 and S_1..S_k, the chain its gate ran. */
|
||
const bisectPrefixes = (t: TrainRecord) => t.stamped ? [t.prefixes[0]!, ...t.stamped] : t.prefixes;
|
||
|
||
/** Validate bisect inputs: red cases from the gate's own plan, every prefix an ancestor of train/<nonce>'s head. */
|
||
function bisectInputs(env: TrainEnv, t: TrainRecord, red: AttemptRecord): string | null {
|
||
for (const id of red.red_cases) {
|
||
if (!/^[A-Za-z0-9_.:/-]{1,128}$/.test(id) || !red.panels.some(p => p.case === id)) return `red case ${id} is not in the gate's plan`;
|
||
}
|
||
const ref = `refs/gstack/${t.branch}`;
|
||
const f = git(env.ctx.repoRoot, ['fetch', '--no-tags', '-q', env.remote, `+refs/heads/${t.branch}:${ref}`]);
|
||
if (!f.ok) return `cannot fetch ${t.branch}: ${f.err.slice(0, 120)}`;
|
||
const tip = git(env.ctx.repoRoot, ['rev-parse', ref]).out;
|
||
if (tip !== t.head) return `${t.branch} is at ${tip.slice(0, 12)}, not the recorded train head ${t.head.slice(0, 12)}`;
|
||
for (const p of bisectPrefixes(t)) {
|
||
if (!git(env.ctx.repoRoot, ['merge-base', '--is-ancestor', p.commit, tip]).ok) return `prefix T${p.index} ${p.commit.slice(0, 12)} is not an ancestor of ${t.branch}`;
|
||
}
|
||
return null;
|
||
}
|
||
|
||
/**
|
||
* `train bisect`: resumable and spend-capped. Each run reconciles the earlier
|
||
* dispatch by nonce, records every prefix result it published, then reserves
|
||
* the remaining case×prefix trials at their recorded price (unknown at the
|
||
* policy reservation) and launches only the prefixes that fit under the cap.
|
||
*/
|
||
export async function trainBisect(ctx: QueueContext, o: { train: string; spendCap: number; dryRun: boolean }): Promise<TrainOut> {
|
||
const env = trainEnv(ctx);
|
||
if ('code' in env) return outRefused(env, [], 'none', 'none');
|
||
const t = findTrain(env, o.train);
|
||
if (!t) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `no train record ${o.train} here or in refs/notes/gstack-trains` }, [], `${Q} train status`, 'none');
|
||
const run = trainFile(t.nonce);
|
||
const red = gateAttempts(t).at(-1);
|
||
if (!red || red.outcome !== 'RED') return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `bisect needs a red verdict; the train's last gate is ${red?.outcome ?? 'missing'}` }, [], nextFor(t), run);
|
||
if (red.purpose === 'rebuild') return outRefused({ code: 'QUEUE_PROOF_MISSING', message: 'a rebuild\'s second red is not bisected again' }, [], nextFor(t), run);
|
||
const prior = loadBisect(t.nonce);
|
||
if (prior?.outcome) return bisectOutcome(env, t, prior);
|
||
const bad = bisectInputs(env, t, red);
|
||
if (bad) return outRefused({ code: 'QUEUE_TRAIN_PREFIX_INVALID', message: bad }, [], `${Q} train status --train ${t.nonce}`, run);
|
||
const b: BisectRecord = prior ?? { schema: 1, kind: 'train-bisect', train: t.nonce, cases: red.red_cases, spend_cap: o.spendCap, spent_usd: 0,
|
||
prefixes: bisectPrefixes(t).map(p => ({ index: p.index, sha: p.commit, results: {}, attempts: [] })), outcome: null, updated_at: nowIso() };
|
||
b.spend_cap = o.spendCap;
|
||
const slug = repoSlug(ctx);
|
||
if (!slug) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: 'gh repo view could not name the repository' }, [], `${Q} train bisect --train ${t.nonce} --spend-cap ${o.spendCap}`, run);
|
||
for (const a of attemptsOf(t).filter(x => x.purpose === 'bisect' && x.state !== 'completed' && x.state !== 'absent')) {
|
||
if (a.state === 'dispatching' || a.state === 'unknown') {
|
||
const hit = runForNonce(listDispatchRuns(a.workflow, execFor(ctx), slug, 100) ?? [], a.nonce);
|
||
if (!hit) return unknownDispatch(t, a);
|
||
Object.assign(a, { state: 'dispatched', run: asRun(hit) });
|
||
}
|
||
const waiting = collect(env, slug, t, b, a);
|
||
save(env, t, a);
|
||
persist(ctx.repoRoot, [b], { remote: env.remote });
|
||
if (waiting) return outPending({ code: 'QUEUE_TRAIN_PENDING', message: `bisect run ${a.run?.id} is ${waiting}` }, [attemptLine(a)], `${Q} train bisect --train ${t.nonce} --spend-cap ${o.spendCap}`, run);
|
||
}
|
||
const pending = b.prefixes.filter(p => b.cases.some(c => !p.results[c]));
|
||
if (!pending.length) return bisectOutcome(env, t, b);
|
||
const perPrefix = b.cases.reduce((sum, id) => sum + casePrice(env, red, id), 0);
|
||
const admitted: typeof pending = [];
|
||
let reserved = b.spent_usd;
|
||
for (const p of pending) if (reserved + perPrefix <= o.spendCap + 1e-9) { admitted.push(p); reserved += perPrefix; }
|
||
const higher = Math.ceil((b.spent_usd + perPrefix * pending.length) * 100) / 100;
|
||
const lines = [`BISECT: cases ${b.cases.join(',')} prefixes ${pending.map(p => 'T' + p.index).join(',')} need results; $${perPrefix.toFixed(2)} reserved per prefix; spent $${b.spent_usd.toFixed(2)}, cap $${o.spendCap.toFixed(2)}`,
|
||
`BISECT: launch ${admitted.map(p => 'T' + p.index).join(',') || '(none)'}${admitted.length < pending.length ? `; ${pending.length - admitted.length} prefix(es) wait for a higher cap` : ''}`];
|
||
if (o.dryRun) return outOk([...lines, 'DRY-RUN: nothing dispatched'], `${Q} train bisect --train ${t.nonce} --spend-cap ${o.spendCap}`, run);
|
||
persist(ctx.repoRoot, [b], { remote: env.remote });
|
||
if (!admitted.length) {
|
||
return outPending({ code: 'QUEUE_TRAIN_BISECT_CAPPED', message: `the next prefix needs $${perPrefix.toFixed(2)}; $${(o.spendCap - b.spent_usd).toFixed(2)} of the cap is left; the outcome stays unattributed` }, lines,
|
||
`${Q} train bisect --train ${t.nonce} --spend-cap ${higher.toFixed(2)}`, run);
|
||
}
|
||
const nonce = makeNonce('gate', t.head, t.base.sha, randomBytes(3).toString('hex'), { purpose: 'bisect' });
|
||
const a = await dispatchAttempt(env, slug, t, 'bisect', { workflow: env.train.gate_workflow, inputs: {
|
||
[DISPATCH_INPUTS.cases]: b.cases.join(','), [DISPATCH_INPUTS.prefixShas]: admitted.map(p => p.sha).join(','),
|
||
[DISPATCH_INPUTS.expectedSha]: t.head, [DISPATCH_INPUTS.nonce]: nonce,
|
||
} }, admitted.map(p => p.index));
|
||
for (const p of admitted) p.attempts.push(a.id);
|
||
b.updated_at = nowIso();
|
||
persist(ctx.repoRoot, [b], { remote: env.remote });
|
||
if (a.state !== 'dispatched') return unknownDispatch(t, a);
|
||
lines.push(attemptLine(a));
|
||
if (admitted.length < pending.length) {
|
||
return outPending({ code: 'QUEUE_TRAIN_BISECT_CAPPED', message: `${pending.length - admitted.length} prefix(es) are past the cap; completed prefixes are kept and the outcome stays unattributed` }, lines,
|
||
`${Q} train bisect --train ${t.nonce} --spend-cap ${higher.toFixed(2)}`, run);
|
||
}
|
||
return outOk(lines, `${Q} train bisect --train ${t.nonce} --spend-cap ${o.spendCap} (collects run ${a.run?.id} once it finishes)`, run);
|
||
}
|
||
|
||
/**
|
||
* `train rebuild`: the unchanged train dispatched once more. Without
|
||
* `--infra`, only after an all-green bisect and only once (rebuild-once);
|
||
* with `--infra`, only after an INFRA/INCOMPLETE-only census and at most
|
||
* EVAL_POLICY.infraRedispatch times. Both runs stay in the record.
|
||
*/
|
||
export async function trainRebuild(ctx: QueueContext, o: { train: string; infra: boolean }): Promise<TrainOut> {
|
||
const env = trainEnv(ctx);
|
||
if ('code' in env) return outRefused(env, [], 'none', 'none');
|
||
const t = findTrain(env, o.train);
|
||
if (!t) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `no train record ${o.train} here or in refs/notes/gstack-trains` }, [], `${Q} train status`, 'none');
|
||
const run = trainFile(t.nonce);
|
||
const gates = gateAttempts(t);
|
||
const last = gates.at(-1);
|
||
const repair = `route ${last?.red_cases.join(',') || 'the red cases'} to a repair lane (GStack Minder); the train is rebuilt only once the repair merges: ${Q} train build ${prsArgs(t.slots, t.explicit_order)}`;
|
||
const refuse = (reason: string) => outRefused({ code: 'QUEUE_TRAIN_REBUILD_REFUSED', message: `rebuild refused: ${reason}` }, gates.map(attemptLine), repair, run);
|
||
if (o.infra) {
|
||
if (!last || last.outcome !== 'INCOMPLETE' || !last.infra_eligible) return refuse(`no INFRA/INCOMPLETE-only record (the last gate is ${last?.outcome ?? 'missing'})`);
|
||
if (gates.filter(a => a.purpose === 'infra-redispatch').length >= INFRA_REDISPATCH) return refuse('already re-dispatched once (EVAL_POLICY.infraRedispatch)');
|
||
} else {
|
||
const b = loadBisect(t.nonce);
|
||
if (!b?.outcome) return refuse('no bisect record');
|
||
if (gates.filter(a => a.purpose === 'rebuild').length >= REBUILD_ONCE) return refuse('already rebuilt once');
|
||
if (b.outcome.kind !== 'unattributed' || !b.outcome.all_green) return refuse(`the bisect was not all green (${Object.entries(b.outcome.patterns).map(([c, s]) => `${c}=${s}`).join(' ')})`);
|
||
}
|
||
const tip = remoteBranchSha(args => git(ctx.repoRoot, args), env.remote, t.branch);
|
||
if (tip !== t.head) return outRefused({ code: 'QUEUE_TRAIN_MOVED', message: `${t.branch} is at ${tip?.slice(0, 12) ?? 'nothing'}, not the recorded head ${t.head.slice(0, 12)}` }, [], `${Q} train build ${prsArgs(t.slots, t.explicit_order)}`, run);
|
||
const slug = repoSlug(ctx);
|
||
if (!slug) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: 'gh repo view could not name the repository' }, [], `${Q} train rebuild --train ${t.nonce}${o.infra ? ' --infra' : ''}`, run);
|
||
const a = await dispatchAttempt(env, slug, t, o.infra ? 'infra-redispatch' : 'rebuild', gateDispatch(env, t, `${t.nonce}.${gates.length + 1}`));
|
||
t.state = a.state === 'dispatched' ? 'gate-pending' : 'dispatch-unknown';
|
||
save(env, t);
|
||
if (a.state !== 'dispatched') return unknownDispatch(t, a);
|
||
return outOk([...gateAttempts(t).map(attemptLine), `REBUILD: ${a.purpose} of the unchanged train ${t.nonce} (head ${t.head.slice(0, 12)}); both runs are reported`], nextFor(t), run);
|
||
}
|