CalvinBackup/gstack
1
0 Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-10-03 01:46:55 +02:00
Code Issues Packages Projects Releases Wiki Activity

v1.91.8.0 test: smaller suite, derived paid selection, retired never-green evals (#2994)

Browse Source 
* test: delete test-infrastructure dead code (G)

- exit-propagation drives the runner's real strict verdict
  (BunTestOutputClassifier + strictTestExitCode); delete the unused
  shardRunLooksTruncated predicate.
- delete skill-coverage-matrix registry + its gate (nothing reads it; the
  floor already iterates skillCensus()).
- delete touchfiles-facade export-parity tests (Bun fails missing imports
  at link time) and the duplicated E2E_TIERS tier-value test.
- delete brain-cache-spec TRANSPORT_DEFAULT_POLICY, SKILL_RUN_RETENTION_DAYS
  and the now-unused BrainTrustPolicy type with their literal tests.
  AUTOPLAN_PREFLIGHT_BUDGET_BYTES stays: skill-preflight-budget enforces it
  against real resolver output.
- delete audit-compliance's JSDoc-comment grep.

* test: replace product tests that fake the product with real-boundary tests (F)

- design: serve.test.ts drove an inline mirror server; now two tests run the
  real serve() on an ephemeral port (reload confinement, submit exit 0).
- setup-gbrain: rollback + voyage tests execute the template-extracted init
  blocks (3 sites) instead of drifted local bash copies.
- terminal-agent: internalHandler source greps replaced by a behavioral
  /internal/grant + /internal/revoke auth matrix (no/wrong/valid token).
- /health: server-security-surface and the server-auth / security-audit-r2 /
  sidebar-tabs source greps fold into one liveness-only check on the real
  body; the L4 sidecar wiring gets a behavioral /pty-inject-scan test.
- delete tautologies (browser-manager onDisconnect, memory-command #12),
  ios swiftui tap fixture self-check, memory-ingest put_page grep, detach
  source greps, sidebar-agent absence pins, dead-CSS pins + the dead CSS,
  security-audit-r2 Task 1 + the test-only meta-commands re-export,
  duplicate generated-SKILL.md checks.
- make-pdf coverage-gaps cases move into their owner test files.

* test: delete tests of dead eval code (A)

- A1: the retired Eng lexical oracle (evaluateEngSeedCoverage,
  isEngSeedDecisionAUQ), the completion-handoff detector and the retained
  corpus had no paid caller since v1.87.6; delete their 26 replay files,
  ~2.6k helper LOC and fixtures, and the dead blocks in 8 mixed files
  (live hasNativePlanTerminal / batching assertions stay).
- A2: dead viewport approvers in autoplan-artifact-permission and their 11
  replay files + fixtures; recorder/launcher cases stay.
- A3: never-wired oracles and seeders (autoplan-phase-order,
  eng-finding-fixture, ceo-paired-fixture, design-ui-scope,
  plan-skill-completion, pty-current-screen, required-reads,
  transcript-section-logger); plan-seed-submission now decodes through the
  production createPtyScreen; section manifests name their actual guard.
- A4: zero-reference helper exports, plus execGit and invokeAndObserve
  found by the reachability pass.
- 52 fixtures orphaned by the deletions; touchfile and selection-table
  entries for every deleted path.

* test: clean up the paid eval lane (B1-B4, B6, B7)

- B1: delete paid files that assert nothing or cannot pass meaningfully:
  skill-llm-eval-spec and skill-e2e-spec-execute (test.todo), gemini-e2e
  (+ gemini-session-runner; no gemini CLI in CI), ship-idempotency (red
  since v1.63), the two opus-4-7 *-sonnet overlay wrappers, conductor-prose
  (+ its source-evaluation replay), codex-e2e-plan-format; drop their keys,
  scripts and census rows.
- B2: skill-llm-eval grades browse/sections/command-list.md with one union
  judge that also carries the baseline score pin; regression-vs-baseline
  deleted (paid run: pass, c4/c4/a4).
- B3: memory-pipeline, ios-qa, ios-qa-swift-build and plan-tune-cathedral
  make no model calls; renamed out of the paid glob so they run on every
  PR. Swift builds need GSTACK_TEST_SWIFT=1; device stub deleted.
- B4: codex-e2e*, outside-voice, aside and ios-device cannot run in the CI
  image; excluded from the weekly lane with a tracked re-entry condition.
- B6: fold opus-47's negative routing controls into skill-routing-e2e
  journey-negatives (paid run: 3/3 unrouted) and delete the file.
- B7: delete the never-green brain-privacy-gate eval; a free
  gstack-skill-start test now proves consent precedes artifacts egress.

* test: retire the finding-count cluster and trim its helpers (C)

- C0/C1: the five never-green evals (skill-e2e-autoplan-chain and
  skill-e2e-plan-{ceo,eng,design,devex}-finding-count) failed on harness and
  budget, never on skill behavior; delete them, their touchfile/tier ids,
  AUTOPLAN_CHAIN_BUDGET and the dedicated eighth periodic slice (--slices 7).
- C2: delete the helper groups whose only paid consumers were those files
  (11 modules), trim claude-pty-runner and eng-seeded-coverage to the paid
  closure, and delete the free replay tests whose assertions exercised only
  that dead code (89 files, 135 orphaned fixtures). Blocks that used dead code
  only as input for a live subject keep their assertions: the multiSelect
  default moved to plan-review-decisions, runner PTY tests use inline caller
  policies, and the timer-safe budget checks moved to eng-finding-retry-budget.
- The eight production-touching files stay except ceo-current-decision-record
  (its template read only feeds the retired counter).
- CARVE_GUARDS.autoplan is behavioral 'none'; TODOS records the lost chain
  and per-finding cadence coverage with their re-entry tests.

* test: fold per-incident replay series into their detector owners (D)

Twelve detector families move into one owner test each: 73 incident files
become describe blocks in ceo-section-loading-fixture (stale-fill race),
model-overlays, coverage-audit-evidence, autoplan-phase-observer,
native-auto-decide, outside-voice-evidence, eng-first-review,
plan-count-completion, plan-count-file-permission, ceo-mode-option,
plan-scope-selection and plan-count-prerequisite. Each block keeps its original
code and fixture, so every case still runs; only tests asserting the incident
file's own touchfile registration are dropped (41). Touchfile lists that named
an incident now name its owner.

* test: start the plan-count history PTY on its readiness marker (H)

The fake CLI prints a startup marker and the runner waits for it instead of the
fixed 8 s startup sleep (8.6 s -> 0.9 s locally). eng-semantic-terminal's
sleeping registration cases went with C; plan-count-timeout keeps the fixed wait
because it asserts deadline behavior.

* test: derive paid touchfiles from each eval's static closure (E)

touchfiles.test.ts now checks, per key, that the paid file's static
test/helpers and test/fixtures closure (plus fixture paths it names in string
literals) is covered, and names the file, path, chain and key to fix when it is
not. Free *.test.ts files are no longer touchfiles, so editing a free replay
test stops selecting paid evals: 950 entries removed, 653 real closure paths
added. The hand-copied inventories go: periodic-fixture-selection,
fake-impeccable-touchfiles and 45 per-file selection examples. Selection for
the sample edits (plan-eng-review template, claude-pty-runner,
plan-count-fixture, gstack-config) loses no case under either profile.
CONTRIBUTING documents the rule and its lower bound.

* test: skip hollow tier shards and census judges in the paid planner (B5)

A paid file is now skipped for a tier lane only when every E2E id it registers
is known statically and none has that tier; ids come from the touchfile
registrations and literal testName/*IfSelected arguments, so a comment or
skill path that quotes another id cannot unschedule it, and computed names
keep today's scheduling. --list and the manifest show each skip as
"skipped: no E2E_TIERS id has tier <tier>". The weekly gate census drops the
LLM judges (--skip-judges); they still run in the periodic census and PR gate
lanes. Gate lane 52 -> 42 files, census 41; periodic 77 -> 69.

* test: run seven paid evals on the current default capture model (B8)

skill-e2e-{auq-matrix,plan-format,qa-bugs,retro,workflow} pinned
claude-opus-4-7 and skill-e2e-office-hours plus -brain-writeback pinned
claude-sonnet-4-6; none tests a historical model, so they now capture with
resolveEvalModel('capture'), and the free harness tests that execute these
registrations receive the same resolver. The paid re-pin run passed all of
them. skill-e2e-{design,office-hours-phase4,plan-prosons,plan} keep
claude-opus-4-7: six of their cases failed on the default model (three
timeouts, a missing report file, a format miss and a posture score of 3), so
per the plan's fallback they keep their pins with a TODOS entry. The pre-spend
estimate and drop threshold are in docs/test-audit-2026-09.md.

* test: guard the reduced suite against new test-of-test files

- test/test-of-test-ratchet.test.ts records the 228 free tests that import only
  test/ code and fails on a new one, naming the owner test to extend instead;
  a stale baseline entry fails with the remove instruction.
- test/helpers/resolve-repo-path.ts is the one specifier/literal resolver for
  the ratchet and the touchfile closure invariant, with its own unit tests.
- CONTRIBUTING "Test tiers" describes the paid-failure workflow (fix, then one
  row in the detector's owner test) and the ratchet; TEST_PORTFOLIO gains the
  detector -> owner-test table and no longer claims an Autoplan chain eval.
- TODOS: automatic exclusion policy for chronically red periodic files (P3),
  the deferred native-completion table collapse, the unused CEO payment
  seeder; the PTY readiness item is narrowed to the paid runner.
- docs/test-audit-2026-09.md collects the triage, security mapping, inventories,
  selection proof, behavior-commit decisions and retained false positives.

* v1.91.8.0 test: smaller suite, derived paid selection, retired never-green evals

Release metadata for the test-reduction branch: VERSION 1.91.8.0 (1.91.7.0 is
claimed by #2983), CHANGELOG with the measured before/after table and a
contributor section, durations re-recorded on Ubicloud standard-16 (857 files,
0 failures), the agents digest, CONTRIBUTING's after-measurement row, the B8
fallback TODOS entry, and the after metrics, kept-vs-plan notes, B8 run and
census estimate in docs/test-audit-2026-09.md.

* fix(ubicloud): skip retrieval globs that match nothing instead of reporting a failed pull

* test: count issue-numbered Design findings in the UI-scope gate eval
This commit is contained in:
Garry Tan authored and GitHub committed 2026-09-29 08:58:49 -07:00
1 parent dcaea52800
commit 943105f109
668 files changed
+11998 -102381
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/brain-cache-spec.ts
-26
View File
@@ -162,12 +162,6 @@ export const SKILL_CALIBRATION_WEIGHTS: Record<string, number> = {
*/
export const CACHE_REFRESH_LOCK_TIMEOUT_MS = 5 * 60_000;
/**
* Retention policy: gstack/skill-run pages auto-archive after this many days.
* Calibration takes (kind=bet) NEVER archive (long-term scorecard needs them).
*/
export const SKILL_RUN_RETENTION_DAYS = 90;
/**
* Schema pack identity. Bumped when adding/removing/renaming page types.
* On mismatch with the version recorded in _meta.json, the cache layer
@@ -176,26 +170,6 @@ export const SKILL_RUN_RETENTION_DAYS = 90;
export const GSTACK_SCHEMA_PACK_NAME = 'gstack-core';
export const GSTACK_SCHEMA_PACK_VERSION = '1.0.0';
/**
* Trust policy values. Drives auto-push of artifacts, calibration write-back
* eligibility, and user-namespacing strategy.
*/
export type BrainTrustPolicy = 'personal' | 'shared' | 'unset';
/**
* Per-transport default policy. Local engines auto-set to personal (single-tenant
* by construction). Remote endpoints are inferred based on sources_list shape:
* exactly one source + whoami matches → personal default; multiple sources or
* federation → ask the policy question.
*/
export const TRANSPORT_DEFAULT_POLICY: Record<string, BrainTrustPolicy | 'infer'> = {
'local-pglite': 'personal',
'local-stdio': 'personal',
'remote-http-single-tenant': 'personal',
'remote-http-ambiguous': 'unset',
unknown: 'unset',
};
/**
* User-slug fallback chain (D4 A3 defensive default). Resolved once per endpoint
* and persisted via `gstack-config set user_slug_at_<endpoint-hash> <slug>`.
scripts/free-test-durations.json
+903 -1106
View File
File diff suppressed because it is too large. Load diff
scripts/paid-test-durations.json
-2
View File
@@ -15,14 +15,12 @@
"test/skill-e2e-investigate-owned-termination.test.ts": 49000,
"test/skill-e2e-learnings.test.ts": 32000,
"test/skill-e2e-office-hours-auto-mode.test.ts": 61000,
"test/skill-e2e-opus-47.test.ts": 20000,
"test/skill-e2e-plan-ceo-finding-floor.test.ts": 233000,
"test/skill-e2e-plan-ceo-plan-mode.test.ts": 35000,
"test/skill-e2e-plan-design-with-ui.test.ts": 435000,
"test/skill-e2e-plan-devex-finding-floor.test.ts": 187000,
"test/skill-e2e-plan-devex-plan-mode.test.ts": 103000,
"test/skill-e2e-plan-mode-no-op.test.ts": 206000,
"test/skill-e2e-plan-tune-cathedral.test.ts": 1000,
"test/skill-e2e-plan-tune.test.ts": 58000,
"test/skill-e2e-plan.test.ts": 312000,
"test/skill-e2e-qa-workflow.test.ts": 437000,
scripts/test-free-shards.ts
+1 -18
View File
@@ -837,7 +837,7 @@ function hasCompleteCiSummary(outcome: FreeShardOutcome): boolean {
export const QUICK_CORE = [
'test/strict-output.test.ts', 'test/gen-skill-docs.test.ts',
'test/skill-check-driver.test.ts', 'test/ceo-native-ledger-replay.test.ts',
'test/skill-check-driver.test.ts',
'test/skill-ceo-section-ordering.test.ts',
'test/qa-functional-observer.test.ts', 'test/qa-checkpoint-evidence.test.ts',
'test/test-free-shards-capture.test.ts',
@@ -960,23 +960,6 @@ function formatShardSummary(shards: string[][]): string[] {
});
}
/**
* True when a shard's output shows the run ended WITHOUT bun's final summary
* ("Ran N tests across ..."). A process.exit() fired mid-suite skips the
* summary AND hands back whatever code the caller passed — historically 0,
* which made a truncated shard indistinguishable from a green one. Exit code
* alone is therefore not evidence of completion; the summary line is.
*
* The runner itself now enforces this (and more) through
* scripts/test-strict-output.ts inside runFreeShard; this predicate remains
* the minimal documented primitive that test/exit-propagation.test.ts drives
* with genuine truncated and genuine complete bun runs.
*/
export function shardRunLooksTruncated(status: number | null, output: string): boolean {
if (status !== 0) return false; // already failing — not the silent case
return !/Ran \d+ tests? across \d+ files?/.test(output);
}
// ---------------------------------------------------------------------------
// Output contract: console filtering + per-file failure attribution.
//
scripts/test-paid-shards.ts
+63 -72
View File
@@ -67,7 +67,7 @@ import {
} from './test-strict-output';
import { PAID_TEST_GLOBS, isPaidTestFile } from '../test/helpers/paid-test-set';
import { PERIODIC_CI_EXCLUDE } from '../test/helpers/periodic-exclude-data';
import { AUTOPLAN_CHAIN_BUDGET, FILE_RETRY_BUDGETS, STRICT_RETRY_CASE_BUDGETS } from '../test/helpers/eval-budgets';
import { FILE_RETRY_BUDGETS, STRICT_RETRY_CASE_BUDGETS } from '../test/helpers/eval-budgets';
import { getProjectEvalDir, getClaudeCliVersion, isFinalizedEvalResultFile, evalEntryOutcome } from '../test/helpers/eval-store';
import { manualReviewProblem } from '../test/helpers/cookie-workflow-manual-review';
import { preflightAnthropicApi } from '../test/helpers/anthropic-preflight';
@@ -167,6 +167,39 @@ export function classifyPaidTestFile(source: string, tier: PaidTier): TierClassi
return { included: true, reason: 'no whole-file tier guard — runtime E2E_TIERS filter decides' };
}
/**
* A file is skipped for a tier lane only when its registered E2E ids are fully
* known and none of them has that tier. Ids are the touchfile registrations that
* list the file plus literal registration arguments (testName, *IfSelected);
* quoted strings elsewhere (comments, skill paths) never count. Any computed
* registration, an id missing from the file's touchfile registration, or no id at
* all keeps today's scheduling (the child's runtime filter decides).
*/
export function tierSkipReason(
file: string, source: string, tier: PaidTier,
touchfiles: Record<string, string[]> = E2E_TOUCHFILES,
tiers: Record<string, string> = E2E_TIERS,
): string | null {
const rel = normalizeRelativePath(file);
const registered = Object.keys(touchfiles).filter(key => touchfiles[key]!.includes(rel));
if (!registered.length) return null;
const computed = /testName\s*:\s*(?:`[^`]*\$\{|[A-Za-z_$])/.test(source)
|| /\btest(?:Concurrent)?IfSelected\s*\(\s*(?:`[^`]*\$\{|[A-Za-z_$])/.test(source)
|| /\bdescribeIfSelected\s*\([^,]*,(?!\s*\[)/.test(source)
|| [...source.matchAll(/\bdescribeIfSelected\s*\([^,]*,\s*\[([^\]]*)\]/g)].some(m => m[1]!.split(',')
.map(item => item.trim()).some(item => item && !/^(['"`])[^'"`$]*\1$/.test(item)));
if (computed) return null;
const literal = [
...[...source.matchAll(/testName\s*:\s*(['"`])([^'"`]+)\1/g)].map(m => m[2]!),
...[...source.matchAll(/\btest(?:Concurrent)?IfSelected\s*\(\s*(['"`])([^'"`]+)\1/g)].map(m => m[2]!),
...[...source.matchAll(/\bdescribeIfSelected\s*\([^,]*,\s*\[([^\]]*)\]/g)]
.flatMap(m => [...m[1]!.matchAll(/(['"`])([^'"`]+)\1/g)].map(n => n[2]!)),
].filter(id => id in tiers);
if (literal.some(id => !registered.includes(id))) return null;
if (registered.some(id => tiers[id] === tier)) return null;
return `skipped: no E2E_TIERS id has tier ${tier}`;
}
export interface TierSelection {
selected: string[];
excluded: Array<{ file: string; reason: string }>;
@@ -200,8 +233,9 @@ export function selectPaidTestFiles(files: string[], tier: PaidTier, rootDir = R
}
const source = fs.readFileSync(path.join(rootDir, file), 'utf8');
const classification = classifyPaidTestFile(source, tier);
if (classification.included) selected.push(file);
else excluded.push({ file, reason: classification.reason });
const skip = classification.included ? tierSkipReason(file, source, tier) : null;
if (classification.included && !skip) selected.push(file);
else excluded.push({ file, reason: skip ?? classification.reason });
}
return { selected, excluded };
}
@@ -395,7 +429,7 @@ export interface DiffSkipOptions {
* than literal.
*
* FAIL-OPEN by construction: run-all selection, non-skill-e2e paid files
* (llm-judge / codex-e2e / gemini-e2e / routing, keyed off other maps),
* (llm-judge / codex-e2e / routing, keyed off other maps),
* unreadable sources, and files with zero mapped names all KEEP their shard —
* the child's self-skip stays authoritative. A parent bug may only run
* extra work, never drop it.
@@ -466,7 +500,7 @@ export function planPaidShards(
const shards: string[][] = [];
let pending: string[] = [];
for (const file of unique) {
if (isOverlayTestFile(file) || file === AUTOPLAN_CHAIN_BUDGET.file || FILE_RETRY_BUDGETS.some(budget => budget.file === file)) {
if (isOverlayTestFile(file) || FILE_RETRY_BUDGETS.some(budget => budget.file === file)) {
if (pending.length) shards.push(pending);
pending = [];
shards.push([file]);
@@ -487,8 +521,6 @@ export interface PaidShardBudget {
/** Explicit caller limits win; registered supervision preserves existing attempts. */
export function resolvePaidShardBudget(files: string[], overrideMs?: number): PaidShardBudget {
const autoplan = files.map(normalizeRelativePath).includes(AUTOPLAN_CHAIN_BUDGET.file);
if (autoplan && files.length !== 1) throw new Error('Autoplan budget requires its own shard');
const finding = FILE_RETRY_BUDGETS.find(budget => files.map(normalizeRelativePath).includes(budget.file));
if (finding && files.length !== 1) throw new Error('Registered retry budget requires its own shard');
if (overrideMs !== undefined && (!Number.isSafeInteger(overrideMs) || overrideMs <= 0 || overrideMs > 2_147_483_647)) {
@@ -500,9 +532,9 @@ export function resolvePaidShardBudget(files: string[], overrideMs?: number): Pa
throw new Error(`Overlay shard requires at least ${OVERLAY_MIN_FILE_WALL_MS}ms; explicit wall ${overrideMs}ms cannot preserve its work and finalization budget`);
}
return {
timeoutMs: overrideMs ?? (autoplan ? AUTOPLAN_CHAIN_BUDGET.shardMs : finding ? finding.shardMs : overlay ? OVERLAY_MIN_FILE_WALL_MS : DEFAULT_SHARD_TIMEOUT_MS),
source: overrideMs !== undefined ? 'explicit' : autoplan || finding ? 'registered' : 'default',
policyId: autoplan ? AUTOPLAN_CHAIN_BUDGET.id : finding?.id ?? null,
timeoutMs: overrideMs ?? (finding ? finding.shardMs : overlay ? OVERLAY_MIN_FILE_WALL_MS : DEFAULT_SHARD_TIMEOUT_MS),
source: overrideMs !== undefined ? 'explicit' : finding ? 'registered' : 'default',
policyId: finding?.id ?? null,
};
}
@@ -604,8 +636,6 @@ export function paidShardWallUpperBoundMs(files: string[], jobs: number, overrid
export interface RunShardsOptions {
timeoutMs?: number;
/** Legacy Autoplan allocation; callers may supply registered per-file allocations. */
autoplanBudget?: PaidShardBudget;
registeredBudgets?: Record<string, PaidShardBudget>;
jobs?: number;
/** bun --max-concurrency inside each shard (EVALS_CONCURRENCY). */
@@ -662,8 +692,7 @@ export async function runPaidShard(
): Promise<ShardOutcome> {
if (files.length === 0) throw new Error('Cannot run an empty paid-test shard.');
const rootDir = options.rootDir ?? ROOT;
const planned = options.registeredBudgets?.[normalizeRelativePath(files[0]!)] ??
(files.map(normalizeRelativePath).includes(AUTOPLAN_CHAIN_BUDGET.file) ? options.autoplanBudget : undefined);
const planned = options.registeredBudgets?.[normalizeRelativePath(files[0]!)];
const budget = resolvePaidShardBudget(files, options.timeoutMs ??
(planned?.source === 'explicit' ? planned.timeoutMs : undefined));
const timeoutMs = budget.timeoutMs;
@@ -1043,7 +1072,7 @@ export interface ManifestEntry {
slice: number;
status: 'planned' | 'skipped-by-diff' | 'excluded';
reason?: string;
/** Required when the registered Autoplan workflow is planned. */
/** Required when a registered retry-budget file is planned. */
budget?: PaidShardBudget;
}
@@ -1057,8 +1086,6 @@ export interface PaidRunManifest {
profile?: PaidProfile;
selection?: PaidCaseSelection;
prCoverage?: PrProfileSelection;
/** Dedicated last slice; preceding slices retain ordinary round-robin work. */
autoplanSlice?: number;
entries: ManifestEntry[];
}
@@ -1125,7 +1152,6 @@ export function buildRunManifest(opts: {
profile?: PaidProfile;
sliceCount: number;
evalsAll: boolean;
dedicatedAutoplanSlice?: boolean;
timeoutMs?: number;
discovered?: string[];
env?: NodeJS.ProcessEnv;
@@ -1133,19 +1159,22 @@ export function buildRunManifest(opts: {
changedFiles?: string[];
/** Recorded per-file durations; defaults to the committed seed under rootDir. */
durations?: Record<string, number>;
/** Weekly gate census only: LLM judges already run in the periodic census and PR gate lanes. */
skipJudges?: boolean;
}): PaidRunManifest {
if (!Number.isInteger(opts.sliceCount) || opts.sliceCount <= 0) {
throw new Error(`--slices needs a positive integer. Received: ${opts.sliceCount}`);
}
if (opts.dedicatedAutoplanSlice && (opts.tier !== 'periodic' || opts.sliceCount < 2)) {
throw new Error('Dedicated Autoplan slice requires periodic tier and at least two total slices');
}
const rootDir = opts.rootDir ?? ROOT;
const env = opts.env ?? process.env;
const profile = opts.profile ?? validatedProfile(env.EVALS_PROFILE, 'EVALS_PROFILE');
if (profile === 'pr' && opts.tier !== 'gate') throw new Error('PR profile requires gate tier; use --profile full for periodic coverage');
const discovered = opts.discovered ?? collectPaidTestFiles(rootDir);
const { selected, excluded } = selectPaidTestFiles(discovered, opts.tier, rootDir, env);
const tierSelection = selectPaidTestFiles(discovered, opts.tier, rootDir, env);
const judge = (file: string) => /^test\/skill-llm-eval[^/]*\.test\.ts$/.test(normalizeRelativePath(file));
const selected = opts.skipJudges ? tierSelection.selected.filter(file => !judge(file)) : tierSelection.selected;
const excluded = [...tierSelection.excluded, ...(opts.skipJudges ? tierSelection.selected.filter(judge)
.map(file => ({ file, reason: 'skipped: LLM judges run in the periodic census and PR gate lanes' })) : [])];
const shards = planPaidShards(selected, { maxFilesPerShard: 1 });
const cases = computePaidCaseSelection({ profile, env, rootDir, changedFiles: opts.changedFiles });
const fast = cases.coverage?.mode === 'pr';
@@ -1157,16 +1186,14 @@ export function buildRunManifest(opts: {
}
const entries: ManifestEntry[] = [];
const overlaySlice = opts.sliceCount - (opts.dedicatedAutoplanSlice ? 1 : 0);
const overlaySlice = opts.sliceCount;
const reserveOverlaySlice = overlaySlice > 1 && runnable.some(files => files.some(isOverlayTestFile));
const ordinarySlices = overlaySlice - Number(reserveOverlaySlice);
// Spread registered long files by supervised load. Keep one ordinary-only
// lane when possible, so every lane does not inherit a long-workflow tail.
// Reserved overlay and dedicated Autoplan slices retain their ownership.
const ordinary = runnable.filter(files => !files.some(isOverlayTestFile) &&
!(opts.dedicatedAutoplanSlice && files[0] === AUTOPLAN_CHAIN_BUDGET.file));
const registered = ordinary.filter(files => files[0] === AUTOPLAN_CHAIN_BUDGET.file ||
FILE_RETRY_BUDGETS.some(budget => budget.file === files[0]));
// The reserved overlay slice retains its ownership.
const ordinary = runnable.filter(files => !files.some(isOverlayTestFile));
const registered = ordinary.filter(files => FILE_RETRY_BUDGETS.some(budget => budget.file === files[0]));
const allocations = new Map<string, number>();
if (registered.length && ordinarySlices > 1) {
const loads = Array<number>(ordinarySlices).fill(0);
@@ -1235,12 +1262,9 @@ export function buildRunManifest(opts: {
return new Map(lanes.flatMap((files, lane) => files.map(file => [file, lane + 1] as const)));
}
runnable.forEach((files) => {
const autoplan = files[0] === AUTOPLAN_CHAIN_BUDGET.file;
const slice = opts.dedicatedAutoplanSlice && autoplan ? opts.sliceCount
: files.some(isOverlayTestFile) ? overlaySlice
: (packed ?? allocations).get(files[0])!;
const slice = files.some(isOverlayTestFile) ? overlaySlice : (packed ?? allocations).get(files[0])!;
entries.push({ file: files[0], slice, status: 'planned',
...(autoplan || FILE_RETRY_BUDGETS.some(budget => budget.file === files[0])
...(FILE_RETRY_BUDGETS.some(budget => budget.file === files[0])
? { budget: resolvePaidShardBudget(files, opts.timeoutMs) } : {}) });
});
for (const s of skipped) entries.push({ file: s.files[0], slice: 0, status: 'skipped-by-diff', reason: s.reason });
@@ -1256,7 +1280,6 @@ export function buildRunManifest(opts: {
profile,
selection: cases.selection,
...(cases.coverage ? { prCoverage: cases.coverage } : {}),
...(opts.dedicatedAutoplanSlice ? { autoplanSlice: opts.sliceCount } : {}),
entries,
};
return parseRunManifest(JSON.stringify(manifest));
@@ -1316,7 +1339,7 @@ export function parseRunManifest(raw: string): PaidRunManifest {
}
}
}
const overlaySlice = parsed.sliceCount - (parsed.autoplanSlice !== undefined ? 1 : 0);
const overlaySlice = parsed.sliceCount;
const plannedOverlays = parsed.entries.filter(entry => entry.status === 'planned' && isOverlayTestFile(entry.file));
if (plannedOverlays.some(entry => entry.slice !== overlaySlice)) {
throw new Error('Overlay manifest entries must share the final ordinary slice to preserve one-process API admission');
@@ -1325,23 +1348,6 @@ export function parseRunManifest(raw: string): PaidRunManifest {
entry.status === 'planned' && !isOverlayTestFile(entry.file) && entry.slice === overlaySlice)) {
throw new Error('The final ordinary manifest slice is reserved for overlay files');
}
const autoplan = parsed.entries.filter(entry => normalizeRelativePath(entry.file) === AUTOPLAN_CHAIN_BUDGET.file);
if (autoplan.length > 1) throw new Error('Duplicate Autoplan manifest entry');
if (parsed.autoplanSlice !== undefined) {
if (parsed.tier !== 'periodic' || parsed.autoplanSlice !== parsed.sliceCount || parsed.sliceCount < 2 || autoplan.length !== 1 || autoplan[0].status !== 'planned') {
throw new Error('Dedicated Autoplan slice is missing or malformed');
}
for (const entry of parsed.entries.filter(entry => entry.status === 'planned')) {
if ((entry.file === AUTOPLAN_CHAIN_BUDGET.file) !== (entry.slice === parsed.autoplanSlice)) {
throw new Error('Dedicated Autoplan slice contains missing or unrelated work');
}
}
}
for (const entry of autoplan.filter(entry => entry.status === 'planned')) {
if (!entry.budget) throw new Error('Autoplan manifest needs an explicit budget record; emit a fresh plan');
const expected = resolvePaidShardBudget([entry.file], entry.budget.source === 'explicit' ? entry.budget.timeoutMs : undefined);
if (!sameBudget(entry.budget, expected)) throw new Error('Autoplan manifest budget differs from declared policy');
}
for (const budget of FILE_RETRY_BUDGETS) {
const entries = parsed.entries.filter(entry => normalizeRelativePath(entry.file) === budget.file);
if (entries.length > 1) throw new Error(`Duplicate registered manifest entry: ${budget.file}`);
@@ -1396,9 +1402,6 @@ export function verifySliceResults(
const reported = new Map<string, { slice: number; status: ShardStatus }>();
for (const result of results) {
for (const outcome of result.outcomes) {
if (outcome.files.map(normalizeRelativePath).includes(AUTOPLAN_CHAIN_BUDGET.file) && outcome.files.length !== 1) {
problems.push('Autoplan result must report its own shard');
}
if (outcome.files.some(file => FILE_RETRY_BUDGETS.some(budget => budget.file === normalizeRelativePath(file))) && outcome.files.length !== 1) {
problems.push('Registered result must report its own shard');
}
@@ -1432,17 +1435,6 @@ export function verifySliceResults(
if (!sameBudget(outcome.budget, expected)) problems.push(`Registered effective result budget differs from its planned/explicit allocation: ${file}`);
} catch { problems.push(`Invalid registered effective result budget: ${file}`); }
}
if (file === AUTOPLAN_CHAIN_BUDGET.file) {
if (outcome.exitCode !== 0 || outcome.executedTests !== 1 || outcome.skippedTests !== 0) {
problems.push('Autoplan must execute exactly one unskipped case with exit zero');
}
try {
const planned = manifest.entries.find(entry => entry.file === file)?.budget;
const expected = resolvePaidShardBudget([file], result.timeoutOverrideMs ??
(planned?.source === 'explicit' ? planned.timeoutMs : undefined));
if (!sameBudget(outcome.budget, expected)) problems.push('Autoplan effective result budget differs from its planned/explicit allocation');
} catch { problems.push('Invalid Autoplan effective result budget'); }
}
}
}
for (const entry of manifest.entries) {
@@ -1492,9 +1484,9 @@ type CliOptions = {
profile: PaidProfile;
profileExplicit: boolean;
listOnly: boolean;
skipJudges: boolean;
timeoutMs: number;
timeoutExplicit: boolean;
dedicatedAutoplanSlice: boolean;
jobs: number;
withinShardConcurrency: number;
maxFilesPerShard: number;
@@ -1542,8 +1534,8 @@ export function parseCliOptions(argv: string[], env: NodeJS.ProcessEnv = process
profile: validatedProfile(env.EVALS_PROFILE, 'EVALS_PROFILE'),
profileExplicit: !!env.EVALS_PROFILE,
listOnly: false,
skipJudges: false,
timeoutExplicit: !!env.EVALS_SHARD_TIMEOUT_MS,
dedicatedAutoplanSlice: false,
timeoutMs: env.EVALS_SHARD_TIMEOUT_MS
? parsePositiveInt(env.EVALS_SHARD_TIMEOUT_MS, 'EVALS_SHARD_TIMEOUT_MS')
: DEFAULT_SHARD_TIMEOUT_MS,
@@ -1579,7 +1571,6 @@ export function parseCliOptions(argv: string[], env: NodeJS.ProcessEnv = process
options.profile = validatedProfile(value, '--profile'); options.profileExplicit = true; continue;
}
if (arg === '--timeout') { options.timeoutMs = parsePositiveInt(argv[index += 1], '--timeout') * 1000; options.timeoutExplicit = true; continue; }
if (arg === '--autoplan-slice') { options.dedicatedAutoplanSlice = true; continue; }
if (arg === '--jobs') { options.jobs = parsePositiveInt(argv[index += 1], '--jobs'); continue; }
if (arg === '--files-per-shard') { options.maxFilesPerShard = parsePositiveInt(argv[index += 1], '--files-per-shard'); continue; }
if (arg === '--emit-plan') {
@@ -1587,6 +1578,7 @@ export function parseCliOptions(argv: string[], env: NodeJS.ProcessEnv = process
if (!value) throw new Error('--emit-plan needs a file path');
options.emitPlanPath = value; continue;
}
if (arg === '--skip-judges') { options.skipJudges = true; continue; }
if (arg === '--slices') { options.slices = parsePositiveInt(argv[index += 1], '--slices'); continue; }
if (arg === '--plan') {
const value = argv[index += 1];
@@ -1603,7 +1595,7 @@ export function parseCliOptions(argv: string[], env: NodeJS.ProcessEnv = process
throw new Error(`Unknown argument: ${arg}`);
}
if (options.writeDurations && !options.reportDir) throw new Error('--write-durations requires --report');
if (options.dedicatedAutoplanSlice && !options.emitPlanPath) throw new Error('--autoplan-slice requires --emit-plan');
if (options.skipJudges && (!options.emitPlanPath || options.tier !== 'gate')) throw new Error('--skip-judges applies only to an emitted gate census plan');
if (options.profile === 'pr' && options.tier !== 'gate') throw new Error('PR profile requires gate tier');
if (options.profile === 'pr' && options.maxFilesPerShard !== 1) throw new Error('PR profile requires one file per shard to preserve case accounting');
return options;
@@ -1619,9 +1611,9 @@ async function main(): Promise<number> {
tier: options.tier,
profile: options.profile,
sliceCount: options.slices,
dedicatedAutoplanSlice: options.dedicatedAutoplanSlice,
timeoutMs: options.timeoutExplicit ? options.timeoutMs : undefined,
evalsAll: process.env.EVALS_ALL === '1',
skipJudges: options.skipJudges,
});
fs.mkdirSync(path.dirname(path.resolve(options.emitPlanPath)), { recursive: true });
fs.writeFileSync(options.emitPlanPath, `${JSON.stringify(manifest, null, 2)}\n`);
@@ -1794,7 +1786,6 @@ async function main(): Promise<number> {
timeoutMs: options.timeoutExplicit ? options.timeoutMs : undefined,
jobs: options.jobs,
withinShardConcurrency: options.withinShardConcurrency,
autoplanBudget: mine.find(entry => entry.file === AUTOPLAN_CHAIN_BUDGET.file)?.budget,
registeredBudgets: Object.fromEntries(mine.filter(entry => entry.budget).map(entry => [normalizeRelativePath(entry.file), entry.budget!])),
...(manifest.prCoverage?.mode === 'pr' ? {
expectedCases: Object.fromEntries(mine.map(entry => [entry.file, expectedPrCaseCount(entry.file, manifest.selection!)])),
scripts/ubicloud/ubi-runner.sh
+9 -3
View File
@@ -178,11 +178,17 @@ cmd_sync() {
}
# pull NAME REMOTE_GLOB LOCAL_DIR: copy matching remote entries into LOCAL_DIR.
# Relative remote paths start at /home/ubi.
# Relative remote paths start at /home/ubi. A glob that matches nothing (for
# example an optional flake ledger) is reported and skipped, not a failure.
cmd_pull() {
local name=$1 from=$2 to=$3
local name=$1 from=$2 to=$3 dir
dir=$(printf %q "$(dirname "$from")")
if ! remote "$name" "cd $dir 2>/dev/null && ls -d -- $(basename "$from") >/dev/null 2>&1"; then
log "pull: nothing matches $from"
return 0
fi
mkdir -p "$to"
remote "$name" "cd $(printf %q "$(dirname "$from")") && tar -czf - $(basename "$from")" | tar -xzf - -C "$to"
remote "$name" "cd $dir && tar -czf - $(basename "$from")" | tar -xzf - -C "$to"
}
cmd_run() {