llm-judge-recommendation is a judge case: each fixture now draws a
3-sample judgePanel, gates reason_substance on the panel mean and the
present/commits/has_because checks on a 2-of-3 majority, thresholds
unchanged. armJudge no longer re-asks on a malformed verdict; it is a
failed sample, as the judge policy requires.
shared-libs-opportunity-judgment and review-design-lite are behavior
cases: their recommendation and checklist judgments may vary, but the
read-only invariant (commands, provider requests, fixture bytes, hooks,
state) and the deterministic fake-engine detector rows are contracts.
Both now go through expectContract, so any failure vetoes the panel.
The targeted batching rerun on Claude Code 2.1.284 left its first report
Write unanswered for 1,372 s and timed out: the viewport began at the
pane's relative file row and rule, with the 'Create file' title cropped
above, so the preview parser rejected the file row as foreign. That row
must now resolve to the owned path and is skipped before the unchanged
line-by-line preview match. Replay controls reject another file, another
directory and an edited preview row.
bun run scripts/test-paid-shards.ts --case <id> [--trials N] runs N
independent trials of one case through the CI panel runner (trial shards,
TRIAL_ENV identity, name-pattern isolation) and prints its panelVerdict();
N defaults to the case's policy panel and CI never reads it. The local
sharded path (test:gate:sharded, test:periodic:sharded) now plans the same
trial shards and exclusions as CI and exits on execution completeness plus
panel verdicts.
The planner job restores this PR's receipt store once and ships a single
filtered set with the plan: a pass or panel receipt with a same-or-newer
FAIL for its input identity is dropped, and a panel receipt ships only as
a whole PASS panel (re-verified with panelVerdict) from one run. Executors
read only that set (no per-slice cache restore or save), so every trial of
a panel sees the same receipts; a trial reuses its own record from the
panel receipt, keeping a split PASS's failed trial.
Trial identities drop the trial index (run-scoped) and bind the panel
policy. Executed shards carry their input identity; the report turns a
whole fresh PASS panel into a panel receipt and a FAIL panel or failed rule
shard into a negative receipt, and marks a panel that mixes reused and
fresh trials INCOMPLETE. The report job merges plan, slice and report
receipts (newest per file) and saves one store per run.
Also fixes two TS2352 casts in browse/test/dia-macos-qualification.test.ts
whose diagnostic text drifted with program order (baseline locked, fix only).
- Slice, census and marathon artifacts carry -a<run_attempt>; reports
download them per artifact (no merge), so records never overwrite and a
re-run never replaces the first attempt's verdict.
- Planners pass --max-parallel for the capacity preflight (24/16 unchanged:
the refreshed periodic plan needs 24 slices, the gate census 12).
- PR comment: jq-only job reads collector-outcomes v2 (headline, sanitized
failure block); the group_by(.name)|last recomputation is gone.
- Reports stamp series identities, upload trial-outcomes-* for history, and
shard logs upload always (a failed trial no longer reds its runner).
- Weekly report: headline + failure block of both lanes in the issue body,
the eval:pass-rates --gate step (fails closed without history), close the
issue on a green run, and UC-E1: when every red is machine-classified
INFRA/INCOMPLETE, one re-dispatch as a new run in its own concurrency
group (redispatch_of), both runs reported.
- scripts/eval-trial-series.ts stamps series_identity (eval-flake-rank's
caseSeriesIdentities) on a report's trial-outcomes JSONL as its own step,
keeping the history tool out of the paid runner's closure;
TrialOutcomeRecord gains the optional series_identity field.
- Slice-count plans let a registered trial spill into an ordinary lane when
its siblings hold every long lane, so panels never share a runner.
- Re-audited test-selection.ts (Stream B added the E2E_KINDS/BEHAVIOR_WHY
map-diff; no new module loading) and repinned its hash.
- Detach and release floors now count trial shards (66 periodic trials in
22 panels): periodic floor 33,821s, still under eval:bg:periodic's 67,380s.
- Coordination fixtures supply the executor's trial records.
Both tiers, merged in run order (the later run wins). Notable: split-overflow
1332s -> 504s, section-loading 604s -> 342s, mode-routing 575s -> 444s;
multi-finding-batching 734s -> 1318s (its red path in run 36606688266).
Planner: behavior and quarantined cases become panels of isolated trial
shards (<file>#<id>~t<N>) bound by EVALS_SELECTION_JSON=[id] and the exact
test name; the file shard excludes them by name. Trials of one case never
share a slice, result slugs are unique, panels are validated whole, unknown
registrations throw, and the planner prints a capacity preflight.
Executor: each trial shard gets its TRIAL_ENV identity and a trial record
(outcome, failure class, cause, cost); every shard writes a JUnit report.
The slice exit now means execution completeness: a failed rule shard or a
trial without a record reds the runner, a failed trial does not.
Report: panelVerdict() decides every panel of the first run attempt (later
attempts are reported, never replacing it); rule shards keep the unchanged
fail-closed checks; collector records all count (no last-attempt wins);
census runs enforce the quarantine cap and expiry. It writes
collector-outcomes v2, trial-outcomes.jsonl (trials plus JUnit rule/judge
cases), report-summary.md, and one headline + failure block with rerun
commands, and flags INFRA/INCOMPLETE-only reds for the one re-dispatch.
The fail-open suite gains the panel cases: behavior 1/3 red, 2/3 green
with its failed trial shown, missing trial INCOMPLETE, contract at 2/3 red,
quarantined 1/3 green, 0/3 and contract red, missing slice red, and a later
attempt never replacing the first.
AGENTS.md replaces the retry rule with the approved policy text (no retries;
kind fixes trials; no added trials, samples or dispatches after a result;
quarantine by CASE_QUARANTINE only; one INFRA/INCOMPLETE re-dispatch) and
notes that a pre-registered fixed panel is not rejudging. CONTRIBUTING gains
the kind rules, the judge panel, eval:pass-rates and an 'Add a paid eval'
checklist. TESTING_INTERNALS describes verdicts, quarantine, history and the
arithmetic, including the rule term: 1 trial vs 2-of-3 red rates at
p = 0.99/0.95/0.90/0.70/0.30 and lane all-green probabilities for the
current 191 rule / 22 behavior / 25 judge registry.
Run 36606688266 bundled routing, learnings and the mode choice into one
native call. Its review panel was taller than the terminal, so the tab
bar scrolled off, ceoModeSubmissionInput returned null for 240 s and HOLD
SCOPE was never submitted ('no posture match'). With no bar on screen the
viewport must still end at the focused Submit prompt, and the accumulated
screen text supplies the one complete panel, authenticated exactly as
before. Replay controls reject another mode, an unoffered answer, an
altered question, a quoted panel, trailing output, a moved cursor and an
answered or changed call.
Run 36606688266's opportunity audit read sixteen sources one per turn and
stopped at error_max_turns; the passing run 36597762183 read the same
files in three batched commands. The skill now says turns are bounded and
asks for parallel reads or one read-only command per step.
Both proof runs (36597762183, 36606688266) printed DESIGN_READY, hit
'No OpenAI API key found' on the first $D variants call, then hand-built
HTML/CSS wireframes, screenshots and a comparison board for ~195-245 s
before the first review question; the second run timed out at 600 s.
A failed first generation now takes the existing text-only path, and the
skill forbids substituting hand-built mockups.
Run 36606688266 asked ten separate native review questions (D1-D9 bound
to ledger records R1-R9) and failed reviewCount=0 < FLOOR=3: its briefs
named the plan by title instead of citing PLAN.md, its report declared
'Review target (fixed): PLAN.md' under '# Engineering review: <plan>', and
it kept an unfenced copy of the plan's own H1. The named-source route now
accepts those spellings and non-inline ledger briefs. The same replay
rejects a foreign, mixed, duplicate or missing target, another plan's
title or copied H1, a brief naming another plan or file, a mismatched
saved brief, and re-asks. The run-36597762183 capture still counts 3.
A completed GSTACK REVIEW REPORT ends the review, so the review-question
count is final there. Run 36606688266 wrote its report at 1,248 s and
closed the session at 1,318 s; the case now stops collection and applies
the unchanged floor at the report instead of waiting out the session.
No budget changes.
2.1.251 logs [claude-code:unrecognized_model] for claude-fable-5-1, the
eval capture/judge default. 2.1.284 does not. The gate PTY smoke subset
(plan-ceo/plan-devex plan-mode, plan-mode-no-op) parses on the new TUI;
plan-design-review-plan-mode passed at 293 s on 2.1.284 and timed out at
300 s on 2.1.251 on the same tree.
scripts/eval-flake-rank.ts becomes eval:pass-rates (eval:flake-rank stays an
alias, and the legacy aggregate stays exported). It reads eval-store's
trial-outcomes JSONL from the last N completed evals-periodic runs on this
branch and main (gh, downloading only the trial-outcomes artifact, cached and
size-capped, parsed as data), plus local eval dirs, and prints per-case
per-trial pass rates with 95% Wilson intervals.
A series is a case's own touchfiles minus GLOBAL_TOUCHFILES
(caseSeriesIdentities, for the report job to stamp), per model, CLI version
and policy version. Labels: INCONCLUSIVE, BROKEN, FLAKY, FAILING, PASSING.
--backfill imports legacy slice artifacts as pre-policy trials (first
attempt only, attributed by registry id, never guessed) for display only.
--gate fails with ACTION REQUIRED on post-policy evidence only: drift below
the quarantine entry rule, a rule case behaving like behavior, a one-sided
Fisher drop against the previous identity (Holm-controlled), and quarantine
entries that met their exit rule, expired after 8 weekly runs, broke the
10% tier cap or are invalid. CASE_QUARANTINE entries now carry a
failureClass (detector, harness or model-latency); a product defect has no
class and is never quarantined. The policy test pins EVAL_POLICY's approved
constants.
E2E_KINDS: rule by default (191 E2E ids), 22 behavior cases whose verdict is
a live model choice with an acceptable sub-100% per-trial rate, each with a
BEHAVIOR_WHY tolerance, and 25 judge entries (the 24 workflow judges plus the
fixed-fixture llm-judge-recommendation rubric check). Contract-shaped cases
(ask-before-decide, plan-mode no-writes, mandated steps, secrets, the batching
floor) stay rule. Behavior requires a known literal registration and an exact
Bun test name so the case runs as its own trial shard.
Map-diff selection now diffs E2E_KINDS and BEHAVIOR_WHY per key, and a base
revision without them selects every key, so a kind flip runs the panel it
introduces. test/eval-kinds.test.ts enforces coverage, tolerances,
isolatability and the reviewed counts, printing the literal to add.
Each of the 24 skill-llm-eval judges now draws EVAL_POLICY.judge.samples
independent samples of the same prompt concurrently inside the unchanged
JUDGE_MS budget. Numeric dimensions gate on the per-dimension panel mean
against the unchanged threshold; booleans (would_browse, consistent) on a
strict majority. An erroring sample fails the whole panel and is never
resampled; a refusal is an unscored panel only when every sample refused.
callJudge's 429 backoff stays: it is transport before any model output.
The workflow-judge cache stores and validates only complete panels, and its
identity now records the panel and zero file retries. Harness tests that
pinned one provider call per case now pin the panel size.
Paid evals never retry (approved 2026-09-29): delete SHORT_CASE_RETRY_FILES
and retriesWithinCaseCap, drop the retry fields from the registered wall rows
(walls now cover one run plus reserve), make retriesForFiles return 0, pass
--retry 0 explicitly, and drop --retry 1 from the package.json paid scripts.
Add the eval:pass-rates alias. Tests that pinned the old retry allowance are
updated as a policy change; review-finalization-budget now proves late-result
recording under the production zero-retry arguments.
Synthetic slice artifacts for rule fail, timeout, missing slice, unreported
entry, hollow, never-started, collector failure and wrong-slice reports all
exit red before the panel-verdict gate change lands.
EvalTestEntry gains case_id, kind, trial, panel, failure_class and
policy_version, stamped from the runner's TRIAL_ENV on isolated trial
shards. panelVerdict() is the single verdict function (INCOMPLETE on
missing or duplicate trials, contract veto at any count, quarantine
hard-break rule, INFRA/INCOMPLETE machine classification). expectContract()
records failure_class 'contract' on the collector entry and a sidecar
before throwing. trial-outcomes JSONL has a fail-closed writer and a
data-only reader.
Every E2E_TIERS and LLM_JUDGE_TOUCHFILES key starts as 'rule'; BEHAVIOR_WHY
and CASE_QUARANTINE start empty. EVAL_POLICY pre-registers the approved
panel (3, majority 2), quarantine entry 0.95/10 and exit 0.97/10, 10% cap,
8-weekly-run expiry, Fisher drift alarm and one INFRA re-dispatch.
qa-b6-static timed out on claude-fable-5-1 in census 36597762183 and in one
of two targeted reruns. Both times the stream stopped mid-message with no
pending tool, right after the model found the disabled submit button, and
stayed silent until the 300 s deadline. Per the B8 fallback, re-pin with a
TODOS entry; budgets and retries are unchanged. A rerun on opus-4-7 passed
(125 s, 5/5 detected).
HOLD Defer/Keep briefs must use 'Note: options differ in kind' (preamble),
but the answered-HOLD path demanded a Completeness score, rejected a
one-line Net with a semicolon, and read posture only from ELI10. The rerun's
brief applied HOLD SCOPE in its Recommendation reason. Revert the
ineffective 'always'/'handoff chat' wording: two runs still skipped the
mode handoff.
The actor declares 'Design: review all seven dimensions', but its picker
reused designReviewSetupAUQ, which only matches already-answered calls
(and a narrower header/label set), so the pending D1 focus menu was never
answered and the case waited out its 609 s deadline. The skill's Step 0D
requires asking; the fixture now answers it.
The census review traced the seeded race as 'R1 miss -> R1 store read (v1)
-> W commit v2 -> W cache.delete -> W fulfills -> R1 cache.set(v1) -> R2
(begun after W) hits v1', but the in-flight gate only accepted race
vocabulary or fixed sentence shapes. Order, actor, version and dismissal
mutations still fail.
The repair rerun named the seeded record by its ISO second
(2026-09-29T16:58:52Z vs .727Z) and said 'I did not write'; both were
misread as a foreign timestamp and a current write.
The parent obeyed the off switch and twice named the seeded completed
record as pre-existing, once with the quotation after its owner and once
with slash separators; the order-specific stripper counted both as current
completion. Timestamp, location, current-claim and value-match controls
still reject.
Census 36597762183: both mode-routing runs logged provenance and moved on
without the mandated handoff chat; the EXPANSION run asked an unauthorized
batch/narrow pacing menu instead of the first per-addition question; the
expansion-energy proposals led with the spec because v1.87.6.0 dropped
'lead with the felt experience'. The HOLD review detector also rejected a
decision whose grounding line named no plan file although the owned source
Read binds it.
design-review-fix drives the Aside browser and registers test.skip on Linux
runners, so its case shard executed zero cases and failed the exact-one-case
check in proof census 36597762183 (eval-slices 6). CASE_CI_EXCLUDE (reason +
tracking, beside PERIODIC_CI_EXCLUDE) now turns such cases into excluded
manifest entries that --list and the manifest surface; every planned case
shard still must execute exactly its case.
ship-docsync ran the same fixture and prompt as ship-docsync-completion and
asserted a subset of it. The file now runs one case per process, so its lane
wall is its longest case instead of half the sum of thirteen.
- Planner budget mode (--slice-budget S --jobs J): recorded per-tier wall
times pack into as many ~9-minute executors as the work needs; the plan
records per-slice estimates and the CI job timeout (supervised worst case
+ 20 min). evals.yml and evals-periodic.yml derive matrix size and
timeout-minutes from it; max-parallel covers every slice at once.
- Case shards: plan/design/review-army/shared-libs(-paths) run one registered
case per process (<file>#<case id>, exact name pattern, exactly one case).
- Retry rule: a timed-out attempt is a verdict. Only files whose every case
budget is CAPTURE tier or shorter keep one retry; walls shrink to match.
- Marathon tier: positive selection, excluded from gate/periodic planners,
run by the new evals-marathon.yml (weekly + dispatch, fresh, own report).
- PR-lane E2E reuse of verified first-attempt passes on identical inputs;
the report rejects reuse outside the fast PR profile.
- Duration seed from census run 36385945043, per tier and per case shard.
sourceDependencyClosure moves from the workflow-judge adapter into
scripts/eval-input-cache.ts unchanged, so judge keys stay byte-identical.
scripts/e2e-shard-reuse.ts builds the consumed-input identity of one PR-lane
E2E shard (test import closure, every registered case's touchfiles, globals,
runner/workflow/setup actions, child env pins, CI image, Claude CLI) and fails
closed on anything unknown. Marathon joins the always-fresh purposes.
Both census read-ready attempts spent turns reading the helper source to
resolve <user-args>, inspecting fixture internals kept inside the repo,
and reconciling 'Read + Edit' with the tmp+mv atomic write, then hit
max turns before the verdict.
Both census builder-wildness attempts answered a direct request for
adjacent unlocks without reading phase-2b-builder-brainstorm.md, whose
trigger read as applying only to the generative questions.
The census review workflow judge scored clarity/actionability 3 on both
attempts: smoke-clock limits appeared to forbid post-repair revalidation,
the caller deadline was undefined, 'ask for setup' conflicted with the
report-only browser rule, fallback-sourced HIGH discrepancies had no gate
decision, and the Step 5.8 record omitted adversarial findings.
* test: delete test-infrastructure dead code (G)
- exit-propagation drives the runner's real strict verdict
(BunTestOutputClassifier + strictTestExitCode); delete the unused
shardRunLooksTruncated predicate.
- delete skill-coverage-matrix registry + its gate (nothing reads it; the
floor already iterates skillCensus()).
- delete touchfiles-facade export-parity tests (Bun fails missing imports
at link time) and the duplicated E2E_TIERS tier-value test.
- delete brain-cache-spec TRANSPORT_DEFAULT_POLICY, SKILL_RUN_RETENTION_DAYS
and the now-unused BrainTrustPolicy type with their literal tests.
AUTOPLAN_PREFLIGHT_BUDGET_BYTES stays: skill-preflight-budget enforces it
against real resolver output.
- delete audit-compliance's JSDoc-comment grep.
* test: replace product tests that fake the product with real-boundary tests (F)
- design: serve.test.ts drove an inline mirror server; now two tests run the
real serve() on an ephemeral port (reload confinement, submit exit 0).
- setup-gbrain: rollback + voyage tests execute the template-extracted init
blocks (3 sites) instead of drifted local bash copies.
- terminal-agent: internalHandler source greps replaced by a behavioral
/internal/grant + /internal/revoke auth matrix (no/wrong/valid token).
- /health: server-security-surface and the server-auth / security-audit-r2 /
sidebar-tabs source greps fold into one liveness-only check on the real
body; the L4 sidecar wiring gets a behavioral /pty-inject-scan test.
- delete tautologies (browser-manager onDisconnect, memory-command #12),
ios swiftui tap fixture self-check, memory-ingest put_page grep, detach
source greps, sidebar-agent absence pins, dead-CSS pins + the dead CSS,
security-audit-r2 Task 1 + the test-only meta-commands re-export,
duplicate generated-SKILL.md checks.
- make-pdf coverage-gaps cases move into their owner test files.
* test: delete tests of dead eval code (A)
- A1: the retired Eng lexical oracle (evaluateEngSeedCoverage,
isEngSeedDecisionAUQ), the completion-handoff detector and the retained
corpus had no paid caller since v1.87.6; delete their 26 replay files,
~2.6k helper LOC and fixtures, and the dead blocks in 8 mixed files
(live hasNativePlanTerminal / batching assertions stay).
- A2: dead viewport approvers in autoplan-artifact-permission and their 11
replay files + fixtures; recorder/launcher cases stay.
- A3: never-wired oracles and seeders (autoplan-phase-order,
eng-finding-fixture, ceo-paired-fixture, design-ui-scope,
plan-skill-completion, pty-current-screen, required-reads,
transcript-section-logger); plan-seed-submission now decodes through the
production createPtyScreen; section manifests name their actual guard.
- A4: zero-reference helper exports, plus execGit and invokeAndObserve
found by the reachability pass.
- 52 fixtures orphaned by the deletions; touchfile and selection-table
entries for every deleted path.
* test: clean up the paid eval lane (B1-B4, B6, B7)
- B1: delete paid files that assert nothing or cannot pass meaningfully:
skill-llm-eval-spec and skill-e2e-spec-execute (test.todo), gemini-e2e
(+ gemini-session-runner; no gemini CLI in CI), ship-idempotency (red
since v1.63), the two opus-4-7 *-sonnet overlay wrappers, conductor-prose
(+ its source-evaluation replay), codex-e2e-plan-format; drop their keys,
scripts and census rows.
- B2: skill-llm-eval grades browse/sections/command-list.md with one union
judge that also carries the baseline score pin; regression-vs-baseline
deleted (paid run: pass, c4/c4/a4).
- B3: memory-pipeline, ios-qa, ios-qa-swift-build and plan-tune-cathedral
make no model calls; renamed out of the paid glob so they run on every
PR. Swift builds need GSTACK_TEST_SWIFT=1; device stub deleted.
- B4: codex-e2e*, outside-voice, aside and ios-device cannot run in the CI
image; excluded from the weekly lane with a tracked re-entry condition.
- B6: fold opus-47's negative routing controls into skill-routing-e2e
journey-negatives (paid run: 3/3 unrouted) and delete the file.
- B7: delete the never-green brain-privacy-gate eval; a free
gstack-skill-start test now proves consent precedes artifacts egress.
* test: retire the finding-count cluster and trim its helpers (C)
- C0/C1: the five never-green evals (skill-e2e-autoplan-chain and
skill-e2e-plan-{ceo,eng,design,devex}-finding-count) failed on harness and
budget, never on skill behavior; delete them, their touchfile/tier ids,
AUTOPLAN_CHAIN_BUDGET and the dedicated eighth periodic slice (--slices 7).
- C2: delete the helper groups whose only paid consumers were those files
(11 modules), trim claude-pty-runner and eng-seeded-coverage to the paid
closure, and delete the free replay tests whose assertions exercised only
that dead code (89 files, 135 orphaned fixtures). Blocks that used dead code
only as input for a live subject keep their assertions: the multiSelect
default moved to plan-review-decisions, runner PTY tests use inline caller
policies, and the timer-safe budget checks moved to eng-finding-retry-budget.
- The eight production-touching files stay except ceo-current-decision-record
(its template read only feeds the retired counter).
- CARVE_GUARDS.autoplan is behavioral 'none'; TODOS records the lost chain
and per-finding cadence coverage with their re-entry tests.
* test: fold per-incident replay series into their detector owners (D)
Twelve detector families move into one owner test each: 73 incident files
become describe blocks in ceo-section-loading-fixture (stale-fill race),
model-overlays, coverage-audit-evidence, autoplan-phase-observer,
native-auto-decide, outside-voice-evidence, eng-first-review,
plan-count-completion, plan-count-file-permission, ceo-mode-option,
plan-scope-selection and plan-count-prerequisite. Each block keeps its original
code and fixture, so every case still runs; only tests asserting the incident
file's own touchfile registration are dropped (41). Touchfile lists that named
an incident now name its owner.
* test: start the plan-count history PTY on its readiness marker (H)
The fake CLI prints a startup marker and the runner waits for it instead of the
fixed 8 s startup sleep (8.6 s -> 0.9 s locally). eng-semantic-terminal's
sleeping registration cases went with C; plan-count-timeout keeps the fixed wait
because it asserts deadline behavior.
* test: derive paid touchfiles from each eval's static closure (E)
touchfiles.test.ts now checks, per key, that the paid file's static
test/helpers and test/fixtures closure (plus fixture paths it names in string
literals) is covered, and names the file, path, chain and key to fix when it is
not. Free *.test.ts files are no longer touchfiles, so editing a free replay
test stops selecting paid evals: 950 entries removed, 653 real closure paths
added. The hand-copied inventories go: periodic-fixture-selection,
fake-impeccable-touchfiles and 45 per-file selection examples. Selection for
the sample edits (plan-eng-review template, claude-pty-runner,
plan-count-fixture, gstack-config) loses no case under either profile.
CONTRIBUTING documents the rule and its lower bound.
* test: skip hollow tier shards and census judges in the paid planner (B5)
A paid file is now skipped for a tier lane only when every E2E id it registers
is known statically and none has that tier; ids come from the touchfile
registrations and literal testName/*IfSelected arguments, so a comment or
skill path that quotes another id cannot unschedule it, and computed names
keep today's scheduling. --list and the manifest show each skip as
"skipped: no E2E_TIERS id has tier <tier>". The weekly gate census drops the
LLM judges (--skip-judges); they still run in the periodic census and PR gate
lanes. Gate lane 52 -> 42 files, census 41; periodic 77 -> 69.
* test: run seven paid evals on the current default capture model (B8)
skill-e2e-{auq-matrix,plan-format,qa-bugs,retro,workflow} pinned
claude-opus-4-7 and skill-e2e-office-hours plus -brain-writeback pinned
claude-sonnet-4-6; none tests a historical model, so they now capture with
resolveEvalModel('capture'), and the free harness tests that execute these
registrations receive the same resolver. The paid re-pin run passed all of
them. skill-e2e-{design,office-hours-phase4,plan-prosons,plan} keep
claude-opus-4-7: six of their cases failed on the default model (three
timeouts, a missing report file, a format miss and a posture score of 3), so
per the plan's fallback they keep their pins with a TODOS entry. The pre-spend
estimate and drop threshold are in docs/test-audit-2026-09.md.
* test: guard the reduced suite against new test-of-test files
- test/test-of-test-ratchet.test.ts records the 228 free tests that import only
test/ code and fails on a new one, naming the owner test to extend instead;
a stale baseline entry fails with the remove instruction.
- test/helpers/resolve-repo-path.ts is the one specifier/literal resolver for
the ratchet and the touchfile closure invariant, with its own unit tests.
- CONTRIBUTING "Test tiers" describes the paid-failure workflow (fix, then one
row in the detector's owner test) and the ratchet; TEST_PORTFOLIO gains the
detector -> owner-test table and no longer claims an Autoplan chain eval.
- TODOS: automatic exclusion policy for chronically red periodic files (P3),
the deferred native-completion table collapse, the unused CEO payment
seeder; the PTY readiness item is narrowed to the paid runner.
- docs/test-audit-2026-09.md collects the triage, security mapping, inventories,
selection proof, behavior-commit decisions and retained false positives.
* v1.91.8.0 test: smaller suite, derived paid selection, retired never-green evals
Release metadata for the test-reduction branch: VERSION 1.91.8.0 (1.91.7.0 is
claimed by #2983), CHANGELOG with the measured before/after table and a
contributor section, durations re-recorded on Ubicloud standard-16 (857 files,
0 failures), the agents digest, CONTRIBUTING's after-measurement row, the B8
fallback TODOS entry, and the after metrics, kept-vs-plan notes, B8 run and
census estimate in docs/test-audit-2026-09.md.
* fix(ubicloud): skip retrieval globs that match nothing instead of reporting a failed pull
* test: count issue-numbered Design findings in the UI-scope gate eval
The discovered paid-file census grows by one (skill-e2e-office-hours-design-draft).
Full-fallback PR selection defers every non-gate id; the shared-input pins now
expect periodic and marathon ids there.
Routing setup and cross-project learnings (D1/D2 in run 36385945043) are
never counted and are not what the case measures. The registration now uses
the runner's existing preconfiguredReviewActor so the attempt starts at the
review; engSetupAUQ still vetoes any late setup question. The registration
test pins the option.
The split actor always answered 0E's mode question with HOLD SCOPE. The
skill skips that question on an explicit choice, so the fixture now states
it and the attempt starts at the five candidate decisions (about 1.5 min
earlier in run 36385945043). Candidates, actor policy, floor and semantic
evaluation are unchanged; the fixture test pins the supplied choice.
The full startup workflow runs 1–3 real spec-review rounds (~280s each) and
hit its 1200s capture in run 36385945043 at finalize. Review depth is the
product's loop, so the case cannot fit a blocking lane without cutting
rounds. It is now marathon tier with every assertion unchanged.
skill-e2e-office-hours-design-draft.test.ts (periodic) runs the same fixed
interview only through the Write that creates the design (269s in that run)
and applies the full validator's design-draft checks, the required section
reads and the launch/foreign-skill-read guards. validateOfficeHoursDesignDraft
is extracted from validateOfficeHoursCompletion, which still applies it.
Selection: office-hours-design-draft is registered periodic; the marathon-only
file is already excluded from the gate and periodic plans by the B5 planner
rule. Tier-alignment regexes and the valid-tier check accept 'marathon'.
A type-only cast in plan-scope-selection.test.ts removes a diagnostic whose
union print order made the ratchet identity unstable; baseline tightened.
Full start-to-finish flows move out of the blocking lanes. E2E_TIERS and
E2ETier gain 'marathon'; describeE2ETier('marathon') is enabled only when
EVALS_TIER=marathon, so the gate/PR and periodic lanes (and the gate census)
never run those cases. The PR profile accepts marathon ids as scheduled
elsewhere and defers them with their own reason, even on full fallback.
The case's only verdict is reviewCount >= FLOOR (3). Run 36385945043 had
three distinct acknowledged review decisions at 6m41s but kept answering
until the ceiling (7) at 12m13s. The registration now passes the runner's
existing isCollectionComplete stop once FLOOR non-setup, non-administrative
review decisions are acknowledged; the floor check, ceiling, budget and
counter are unchanged. A child-process registration test proves the stop
predicate and that below-floor and timeout outcomes still fail.
Run 36385945043's split-overflow case asked all five candidate decisions by
8m55s, but the live candidate check required the question to open with
"E1:" and every option to be a known disposition. The skill cited ledger
row IDs ("D2.1 — R-E1: …") and offered "Hold, discuss first", so no
candidate was recognized and the attempt ran the whole review (1302s).
Identity now comes from the native header; the question must open with that
candidate's ledger reference, name only that candidate, and offer exactly one
include, defer and cut disposition. The selected answer must still be one of
those three. The semantic evaluator and every existing negative control are
unchanged; a trimmed capture from the run adds the positive case and four
row-ID negative controls.
#2994 deletes the plan-*-finding-count evals, ceo-payment-findings.ts and
design-count-review.ts. Drop the CEO throw diagnostics and Design boundary
work with them, and drop the structured completion predicate, stopReason,
review-log binding and plan/review-log evidence copy: no surviving
runPlanSkillCounting caller passes expectedPlanPath, so they would be dead
code. Keep idleFor in timeout summaries (every counting caller can time
out), asserted in the existing timeout test. W7 and W8 are unchanged.
Replaying run 36385945043's FAN-1 and ERR-1 throws (ledger rows
reconstructed from rendered diffs) through ceoPaymentFinding: the email
obligation's row, subject, option and proposal predicates pass and the
ELI10 explanation-defect predicate fails first ('lets that exception fly
out', 'the error bubbles up').
Binding the defect to the named ledger row instead (the planned fix) was
tried and reverted: scoped to the email seed it flips 30+ existing cf74
still-rejects replays, which require a vocabulary-free, ledger-bound email
question to earn credit only through a complete saved comparison. With
FAN-1's rendered currentDecision payload reconstructed, the recorded-
decision path counts it, so the real saved plan (not uploaded) must have
differed; failure artifacts now retain it.
The classifier stays fail-closed and unchanged. Its throw now prints the
header, the first 200 question characters and each obligation's predicate
results. Free regressions with provenance and negative controls: an
unrelated question, an email question whose row says it is already
rescued, and a ledger ID whose row belongs to another seed.
Replaying run 36385945043 through the Design count predicates: routing,
focus and learnings setup was not recognized as setup, Issue 1 was counted
pre-review in both attempts (the boundary fired on it), and attempt 2
counted the Font TODO proposal as a finding (review=4 and review=5 for five
issues). The paid caller now starts review at the first answered native
decision that is not setup (recognized packet, or setup header/question ID),
a completion handoff, artifact rendering or a TODO proposal (the review's
Add to TODOS.md / Skip / Build it now menu). TODO proposals are recorded as
administrative extra decisions. The replay asserts each counted call: both
attempts review=5 (Issues 1-5). isDesignCountFirstReview and its controls
are unchanged.
Replaying run 36385945043's two Design attempts showed the existing routes
rejected correct endings: attempt 1 at the typed-completion path field
('- Reviewed plan written to …' is not a 'Plan written to' line), attempt 2
at the leading-fence veto (its final message opens with the dashboard).
nativePlanTerminalPreconditions is the structural prefix of
hasNativePlanTerminal (behavior unchanged). structuredPlanCompletion adds,
inside the existing nativeSummary branch: a complete report (Design
binding for Design), a completed review-log row for the expected skill
appended during this attempt under the child's GSTACK_HOME/project slug
(resolved with bin/gstack-slug) and stamped with the fixture commit, timed
between the report/last answer (second resolution) and the final native
message, a final message with stop_reason end_turn (now carried on public
transcript messages), and no visible question or permission prompt.
Timeout summaries add idleFor and lastTerminalCandidate. Terminal and throw
captures copy the plan file and review-log rows into the artifact
directory; copies are best-effort and recorded in evidence-copy.json.
Free regressions: both captured Design endings (trimmed fixture with
provenance; report, row and end_turn reconstructed and labelled), the
negative controls, and real-PTY completion/timeout runs through the real
review logger.
computePaidCaseSelection read the version-only exemption from git even when
changed files were injected, so the shared-input test failed on main and on
version-only branches. The exemption is now an optional input; the test pins
a real package.json change and covers the version-only case.
Adds tsconfig.json (strict) over product code, fixes its remaining 90
diagnostics (type-only, interface corrections, and explicit narrowing),
and adds a typecheck job to the required free-tests aggregate running
bun run typecheck, the test-code ratchet (identity -> count baseline, fails
on new, repeated, or unlocked fixed diagnostics), and the lib/cso format
check. Reuses fixes from #2447 where they still applied.
The shared-libs shim served 2 PRs for pulls?state=all and endless full pages
for state=open. gh pr list, pulls?state=open|all|closed (per_page/page,
short last page, direction) and search/issues now page one deterministic
table: PR 7, 600 older open PRs, PR 42 and 3 closed PRs, so five 100-item
open-metadata pages still leave older open PRs unchecked. The Contents API
lists pinned directories (the captured attempt got 404 for contents/ and
contents/src while files resolved, then fell back to a raw host), unknown
endpoints return 404 instead of repo metadata, and the read-only detector
is unchanged. Free tests cover view agreement, the budget bound, gh/curl
agreement and the empty world.
Dual-voice outside-voice failures now report probeToolUseId, probeMode and
the canonical-match result with the reason the probe output was rejected.
The supervisor respawned with a block-scoped env that no longer existed, so
every attempt threw and the loop gave up after five tries. The headed env is
now one pure helper used by connect and respawn, the loop is an injectable
runHeadedSupervisor with behavioral tests, failures name the daemon log and
relaunch command, and connect's usage advertises --supervise.
Compiled installs always take the non-Bun branch, which called an unimported
join and threw before any runtime-tested assertion could be witnessed. The
child command selection is now a pure, platform-aware function; a missing
sibling launcher fails with its expected path.
Mechanical reformat only. Minified transpile output is byte-identical for
21 of 22 files; witness.ts differs only in three regex flag orders
(/mi -> /im), which JavaScript canonicalizes. Source-text assertions over
lib/cso now compare whitespace-insensitively with the same tokens.
Both EVALS_HERMETIC branches of buildHermeticEnv now carry
DISABLE_AUTOUPDATER=1 (the allowlist scrubbed the workflow's copy, so every
PTY screen showed the updater's npm-prefix failure). Per-test overrides
still win. The corrupt durations-seed test now captures its expected
warning and restores the console spy.
Keep both intents: v1.91.7.0's functional QA, docsync and exploratory
paid cases and their free owners stay; this branch's deletions stay
deleted. main's new paid keys follow the derived-closure touchfile rule
(free *.test.ts paths dropped, static helper/fixture closure added), its
new helper-only tests join the ratchet baseline, and its free selection
examples that named free test files now assert the derived selection.
Periodic CI keeps seven slices without the retired Autoplan slice; the
gate census keeps seven single-worker slices with --skip-judges. Wall
and census literals are recomputed from the merged planner, durations
are re-recorded on Ubicloud, and VERSION stays 1.91.8.0 above 1.91.7.0.
Release metadata for the test-reduction branch: VERSION 1.91.8.0 (1.91.7.0 is
claimed by #2983), CHANGELOG with the measured before/after table and a
contributor section, durations re-recorded on Ubicloud standard-16 (857 files,
0 failures), the agents digest, CONTRIBUTING's after-measurement row, the B8
fallback TODOS entry, and the after metrics, kept-vs-plan notes, B8 run and
census estimate in docs/test-audit-2026-09.md.
- test/test-of-test-ratchet.test.ts records the 228 free tests that import only
test/ code and fails on a new one, naming the owner test to extend instead;
a stale baseline entry fails with the remove instruction.
- test/helpers/resolve-repo-path.ts is the one specifier/literal resolver for
the ratchet and the touchfile closure invariant, with its own unit tests.
- CONTRIBUTING "Test tiers" describes the paid-failure workflow (fix, then one
row in the detector's owner test) and the ratchet; TEST_PORTFOLIO gains the
detector -> owner-test table and no longer claims an Autoplan chain eval.
- TODOS: automatic exclusion policy for chronically red periodic files (P3),
the deferred native-completion table collapse, the unused CEO payment
seeder; the PTY readiness item is narrowed to the paid runner.
- docs/test-audit-2026-09.md collects the triage, security mapping, inventories,
selection proof, behavior-commit decisions and retained false positives.
skill-e2e-{auq-matrix,plan-format,qa-bugs,retro,workflow} pinned
claude-opus-4-7 and skill-e2e-office-hours plus -brain-writeback pinned
claude-sonnet-4-6; none tests a historical model, so they now capture with
resolveEvalModel('capture'), and the free harness tests that execute these
registrations receive the same resolver. The paid re-pin run passed all of
them. skill-e2e-{design,office-hours-phase4,plan-prosons,plan} keep
claude-opus-4-7: six of their cases failed on the default model (three
timeouts, a missing report file, a format miss and a posture score of 3), so
per the plan's fallback they keep their pins with a TODOS entry. The pre-spend
estimate and drop threshold are in docs/test-audit-2026-09.md.
A paid file is now skipped for a tier lane only when every E2E id it registers
is known statically and none has that tier; ids come from the touchfile
registrations and literal testName/*IfSelected arguments, so a comment or
skill path that quotes another id cannot unschedule it, and computed names
keep today's scheduling. --list and the manifest show each skip as
"skipped: no E2E_TIERS id has tier <tier>". The weekly gate census drops the
LLM judges (--skip-judges); they still run in the periodic census and PR gate
lanes. Gate lane 52 -> 42 files, census 41; periodic 77 -> 69.
touchfiles.test.ts now checks, per key, that the paid file's static
test/helpers and test/fixtures closure (plus fixture paths it names in string
literals) is covered, and names the file, path, chain and key to fix when it is
not. Free *.test.ts files are no longer touchfiles, so editing a free replay
test stops selecting paid evals: 950 entries removed, 653 real closure paths
added. The hand-copied inventories go: periodic-fixture-selection,
fake-impeccable-touchfiles and 45 per-file selection examples. Selection for
the sample edits (plan-eng-review template, claude-pty-runner,
plan-count-fixture, gstack-config) loses no case under either profile.
CONTRIBUTING documents the rule and its lower bound.
The fake CLI prints a startup marker and the runner waits for it instead of the
fixed 8 s startup sleep (8.6 s -> 0.9 s locally). eng-semantic-terminal's
sleeping registration cases went with C; plan-count-timeout keeps the fixed wait
because it asserts deadline behavior.
Twelve detector families move into one owner test each: 73 incident files
become describe blocks in ceo-section-loading-fixture (stale-fill race),
model-overlays, coverage-audit-evidence, autoplan-phase-observer,
native-auto-decide, outside-voice-evidence, eng-first-review,
plan-count-completion, plan-count-file-permission, ceo-mode-option,
plan-scope-selection and plan-count-prerequisite. Each block keeps its original
code and fixture, so every case still runs; only tests asserting the incident
file's own touchfile registration are dropped (41). Touchfile lists that named
an incident now name its owner.
- C0/C1: the five never-green evals (skill-e2e-autoplan-chain and
skill-e2e-plan-{ceo,eng,design,devex}-finding-count) failed on harness and
budget, never on skill behavior; delete them, their touchfile/tier ids,
AUTOPLAN_CHAIN_BUDGET and the dedicated eighth periodic slice (--slices 7).
- C2: delete the helper groups whose only paid consumers were those files
(11 modules), trim claude-pty-runner and eng-seeded-coverage to the paid
closure, and delete the free replay tests whose assertions exercised only
that dead code (89 files, 135 orphaned fixtures). Blocks that used dead code
only as input for a live subject keep their assertions: the multiSelect
default moved to plan-review-decisions, runner PTY tests use inline caller
policies, and the timer-safe budget checks moved to eng-finding-retry-budget.
- The eight production-touching files stay except ceo-current-decision-record
(its template read only feeds the retired counter).
- CARVE_GUARDS.autoplan is behavioral 'none'; TODOS records the lost chain
and per-finding cadence coverage with their re-entry tests.
- B1: delete paid files that assert nothing or cannot pass meaningfully:
skill-llm-eval-spec and skill-e2e-spec-execute (test.todo), gemini-e2e
(+ gemini-session-runner; no gemini CLI in CI), ship-idempotency (red
since v1.63), the two opus-4-7 *-sonnet overlay wrappers, conductor-prose
(+ its source-evaluation replay), codex-e2e-plan-format; drop their keys,
scripts and census rows.
- B2: skill-llm-eval grades browse/sections/command-list.md with one union
judge that also carries the baseline score pin; regression-vs-baseline
deleted (paid run: pass, c4/c4/a4).
- B3: memory-pipeline, ios-qa, ios-qa-swift-build and plan-tune-cathedral
make no model calls; renamed out of the paid glob so they run on every
PR. Swift builds need GSTACK_TEST_SWIFT=1; device stub deleted.
- B4: codex-e2e*, outside-voice, aside and ios-device cannot run in the CI
image; excluded from the weekly lane with a tracked re-entry condition.
- B6: fold opus-47's negative routing controls into skill-routing-e2e
journey-negatives (paid run: 3/3 unrouted) and delete the file.
- B7: delete the never-green brain-privacy-gate eval; a free
gstack-skill-start test now proves consent precedes artifacts egress.
- A1: the retired Eng lexical oracle (evaluateEngSeedCoverage,
isEngSeedDecisionAUQ), the completion-handoff detector and the retained
corpus had no paid caller since v1.87.6; delete their 26 replay files,
~2.6k helper LOC and fixtures, and the dead blocks in 8 mixed files
(live hasNativePlanTerminal / batching assertions stay).
- A2: dead viewport approvers in autoplan-artifact-permission and their 11
replay files + fixtures; recorder/launcher cases stay.
- A3: never-wired oracles and seeders (autoplan-phase-order,
eng-finding-fixture, ceo-paired-fixture, design-ui-scope,
plan-skill-completion, pty-current-screen, required-reads,
transcript-section-logger); plan-seed-submission now decodes through the
production createPtyScreen; section manifests name their actual guard.
- A4: zero-reference helper exports, plus execGit and invokeAndObserve
found by the reachability pass.
- 52 fixtures orphaned by the deletions; touchfile and selection-table
entries for every deleted path.
- design: serve.test.ts drove an inline mirror server; now two tests run the
real serve() on an ephemeral port (reload confinement, submit exit 0).
- setup-gbrain: rollback + voyage tests execute the template-extracted init
blocks (3 sites) instead of drifted local bash copies.
- terminal-agent: internalHandler source greps replaced by a behavioral
/internal/grant + /internal/revoke auth matrix (no/wrong/valid token).
- /health: server-security-surface and the server-auth / security-audit-r2 /
sidebar-tabs source greps fold into one liveness-only check on the real
body; the L4 sidecar wiring gets a behavioral /pty-inject-scan test.
- delete tautologies (browser-manager onDisconnect, memory-command #12),
ios swiftui tap fixture self-check, memory-ingest put_page grep, detach
source greps, sidebar-agent absence pins, dead-CSS pins + the dead CSS,
security-audit-r2 Task 1 + the test-only meta-commands re-export,
duplicate generated-SKILL.md checks.
- make-pdf coverage-gaps cases move into their owner test files.
- exit-propagation drives the runner's real strict verdict
(BunTestOutputClassifier + strictTestExitCode); delete the unused
shardRunLooksTruncated predicate.
- delete skill-coverage-matrix registry + its gate (nothing reads it; the
floor already iterates skillCensus()).
- delete touchfiles-facade export-parity tests (Bun fails missing imports
at link time) and the duplicated E2E_TIERS tier-value test.
- delete brain-cache-spec TRANSPORT_DEFAULT_POLICY, SKILL_RUN_RETENTION_DAYS
and the now-unused BrainTrustPolicy type with their literal tests.
AUTOPLAN_PREFLIGHT_BUDGET_BYTES stays: skill-preflight-budget enforces it
against real resolver output.
- delete audit-compliance's JSDoc-comment grep.
* v1.91.5.0 feat: balanced free-suite shards, 16-way Linux runs, and bun run test:ubicloud
* chore: regenerate agents digest for v1.91.5.0
* ci: serial flaky retry and flake ledger for the Windows free lane
* ci: cancel superseded eval runs; relax LLM-judge clarity bar to 3
* fix: support Windows Opera and Opera GX cookie imports
Fixes#2957
* fix: repair Windows cookie decryption and make Opera import failures actionable
- strip the SHA-256(host_key) prefix Chromium adds to v10 values (DB meta v24+) on Windows
- keep receipts and name `$B handoff` recovery for App-Bound rows in browsers without native extraction
- explain missing browsers, missing profiles and ambiguous profile selection with next steps
- add windowsNative/resolveBrowserInfo, the operagx alias and sorted failure reasons in CLI output
- cover the Node server runtime with a real-DPAPI Windows test
* docs: document Windows Opera cookie import and guard browser lists against drift
* chore: file cookie-import follow-ups from the Opera fix wave review
* test: keep Opera receipt tests independent of the shared key cache
* test: declare generated gstack/llms.txt as a command-reference input for PR selection
* chore: release v1.91.4.0
* test: reconstruct the historical cookie-workflow approval after the Opera BROWSER.md additions
* test: run the real-DPAPI Windows check with the runner's environment
PowerShell launched with a stripped environment took ~18-21s on the Windows
runner (measured on a throwaway diagnostics run), past dpapiDecrypt's 10s
deadline; with the full environment it returns in ~0.3s. Only APPDATA is
redirected to the fixture's Opera root.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
* fix(memory-ingest): --scan-secrets scans the rendered page and fails closed
--scan-secrets ran gitleaks on the raw transcript .jsonl, then imported a
page rendered from it. gitleaks' assignment rules don't match across a
JSON-escaped quote (KEY=\"v\" on disk), so a secret the rendered page
shows as KEY="v" was imported unflagged. And the gate skipped a file only
on scanner "gitleaks" with findings, so a scan that errored (non-zero
exit, 16MB maxBuffer overflow on a file with many findings, unparseable
report) or could not run (gitleaks missing, slow-probe cooldown) imported
the file unscanned.
Scan the rendered page body, the exact bytes writeStaged() writes, via a
new secretScanText() helper, and skip the file whenever the scan did not
complete. Skipped files stay out of the state file, so the next run
retries them. Reword the helper warnings and setup-gbrain/memory.md,
which described the fail-open as intended.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(test): reconcile Bun failure markers and footer counts
* fix(sync-gbrain): verify source-scoped reads without mutation
* fix(test): recognize grounded TTHW target choices structurally
* fix(aside): make the readiness probe work under zsh and report why it failed
The probe built its deadline into `_T` and expanded it unquoted, so
`$_T aside repl …` only worked in a shell that word-splits. zsh does not: it
looked for a command literally named "gtimeout 30", the probe answered
ASIDE_NOT_RUNNING with Aside installed and ready, and every browsing skill
fell back to the bundled Chromium in silence. zsh is the macOS default and
Aside is macOS-only, so on a stock Mac the probe could never report READY.
The deadline becomes a function, `_gs_d`. It receives the command as "$@",
already split, so sh, bash and zsh all behave the same, and the gtimeout →
timeout → perl alarm chain is unchanged. A 4th arm runs the call unbounded
when none of the three is present, which is what the empty `_T` did before.
Not `eval`: it re-parses the string, so the parens and `;` of the perl arm
become syntax and that arm dies in bash *and* zsh — on a stock Mac, the arm
that actually runs.
On failure the probe now prints the CLI's reason after ASIDE_NOT_RUNNING:,
the shape gstack-render already uses: the first line that starts with a
capital letter, i.e. the CLI's own sentence or Node's `Error:` line below its
loader frame. "Not running" covers states with different fixes — no window
open for the profile, a NODE_OPTIONS preload that kills the CLI — and a bare
verdict sent all of them to "open the Aside app". The BROWSER SETUP prose
quotes that reason before asking the user to open the app.
The text pin asserted the broken invocation verbatim, so it now pins the
function and asserts neither `$_T aside repl` nor an eval form comes back. A
second test executes the rendered probe in sh, bash and zsh on each of the
four deadline arms with stubbed binaries on a narrowed PATH, plus two failing
CLIs: one that prints its own sentence, one that crashes like Node with the
useful line below the frame.
The deadline function costs zero bytes against the lines it replaces; the
reason costs 53 per copy of the probe (44 where the reworded BROWSER SETUP
line gives 9 back). That moves four guards by the measured amount:
plan-devex-review's skeleton cap to 68,550 (measured 68,544), plan-ceo-review's
skeleton cap to 80,150 (measured 80,111) and union ratio to 1.081 (measured
1.0803), and plan-eng-review's union ratio to 1.151 (measured 1.1504).
Fixes#2842, #2941.
* Clarify engineering review startup and decision flow
* Fix Windows readiness fixture PATH and command shim
* fix(test): recognize grounded TTHW target choices structurally
* Clarify engineering review startup and decision flow
* fix(test): restrict QA-only fixture tools to its no-Edit contract
* v1.90.0.0 fix(sync-gbrain): guard readiness verdicts and refresh metadata
* fix(browse): validate canonical upload targets
* fix(gbrain): classify structured PGLite busy response
* fix(browse): preserve native extension runtime APIs
* Fix displayless browser handoff ownership
* Accept unique installed autoplan methodology aliases
* fix(skills): preserve positional literals during installation
* fix(browse): checksum installer contents through stdin
* fix(test): normalize Windows checksum fixture paths
* test: emulate unavailable shasum in Windows checksum fixture
* fix(investigate): preserve owned freeze lifecycle
* fix(review): preserve N+1 retry and Red Team completion
* fix: bound Aside readiness and preserve safe fallback
* test: exercise setup and Chromium on native ARM
* fix: preserve install ownership and ARM browser selection
* Fix gbrain ingest scan boundaries and seed observation
* Refresh managed ship hooks and supervise expanded paid census
* Reject resumed gbrain pages excluded by current policy
* Recover zombie agent locks safely and enable CI Python venv
* Repair paid actor declarations and Aside pitch assertions
* Bump consolidated wave to next free minor release
* Clarify CEO review admin choices and option tradeoffs
* Preserve CEO mode handoff anchors in clarified workflow
* Make Windows portability fixtures use shell-native paths
* Restore ARM Bun alias and clarify ship review gates
* Refresh ship workflow golden snapshots
* Fix Windows DX documentation controls without piped stdin
* Decode Codex child pipes without Bun's encoded-stream stall
* Bound DX pre-review audit before product questions
* Clarify trusted review-start read in paid revalidation
* Bump consolidated wave to next free minor release
* Clarify CEO review admin choices and option tradeoffs
* Preserve CEO mode handoff anchors in clarified workflow
* Make Windows portability fixtures use shell-native paths
* Restore ARM Bun alias and clarify ship review gates
* Refresh ship workflow golden snapshots
* Fix Windows DX documentation controls without piped stdin
* Decode Codex child pipes without Bun's encoded-stream stall
* Bound DX pre-review audit before product questions
* Clarify trusted review-start read in paid revalidation
* Reconcile new main planning flow and paid judge census
* fix: reconcile rebased planning and source-bound validation
* test: pin cookie workflow judge to scored Sonnet model
* fix: keep terminal agent boot out of module imports
* fix: preserve pending-question uncertainty in engineering review
* fix: stabilize Windows reliability-wave fixtures
* fix: clarify design consultation research workflow
* fix: preserve independent design consultation inputs
* fix: resolve design taste scope and browser research guidance
* fix: make consultation opt-in preflight unambiguous
* test: await native Edge owner readiness or terminal result
---------
Co-authored-by: Bruce Krysiak <brucek@alum.mit.edu>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Antonio Vitalic <antoninte99@gmail.com>
* fix(design-detect): find impeccable installed as a Claude Code plugin
The design-detector probe only ever checked <root>/<SKILL_ROOTS>/skills/impeccable/,
never the Claude Code plugin-cache layout
(<root>/.claude/plugins/cache/<marketplace>/<plugin>/<version>/skills/impeccable/).
A plugin-installed impeccable was therefore invisible: IMPECCABLE_SKILL stayed
absent, the launcher was never found (so the NOT_CACHED run hint never fired),
and its sibling engine was never considered.
Add a plugin-cache walk alongside the existing SKILL_ROOTS walk, sharing the
same presence/launcher/repo-local-exclusion/sibling-engine logic via an
extracted checkSkillDir() helper so both paths stay behaviorally identical.
Fixes#2838
* refactor(design-detect): consolidate newestSemverDir onto safeReaddir
Both did the identical try/catch-around-readdirSync; newestSemverDir now
reuses the new safeReaddir helper instead of duplicating it.
* fix: harden Impeccable plugin discovery and regression fixtures
* test: supply eval mode to the integrated detector callback adapter
---------
Co-authored-by: Som Samantray <som.samantray@gmail.com>
* perf: remove repeated test work and preserve AUQ execution budgets
* fix: validate native evaluation fixture evidence at its actual boundaries
* fix: clarify deployment approval and recovery state transitions
* chore: document coverage and release v1.90.2.0
* test: preserve Windows scheduling and native no-change consent
* test: recognize verified reads through fixture symlinks
* test: isolate alias-name installation from runtime assets
* fix(browse): prepare reliable cookie import wave for validation
* ci: sequence quality and behavior for validation branch
* fix(browse): isolate Windows qualification and preserve native diagnostics
* test(browse): cover cookie workflow quality and isolate Windows user paths
* test(browse): trace native member startup and initialize fresh folders
* fix(browse): keep Windows member stdin alive through EOF
* fix(browse): latch native timeouts and compare contained Edge startup
* test(browse): verify native version metadata and actual Windows argv
* test(browse): qualify Dia import on isolated macOS CI
* fix(browse): require picker origin for session mutations
* fix(browse): bound credential reads through stream completion
* test(browse): inspect owned Windows process arguments natively
* test(evals): preserve passing coverage during cookie repair reruns
* test(browse): isolate Dia qualification in a fresh macOS account
* test(browse): pass bounded integer timeouts to native Mac probes
* test(browse): distinguish Windows profile initialization from containment
* test(browse): await descendant pipe readiness before parent exit
* test(browse): initialize and restore isolated macOS Keychain state
* test(browse): initialize Windows fixture folders before qualification
* test(ci): pin the same Node runtime across Windows checks
* test(browse): distinguish native macOS browser preflight stages
* test(browse): isolate Windows descendant console lifetime
* test(browse): preserve native receipts and identify fixture lock holders
* test(browse): prepare dependency resolution before native Mac worker startup
* test(ci): include lock and close checks in native diagnostics
* test(browse): preserve native owner probe stages and subprocess deadlines
* fix(browse): classify Chromium profile-in-use exit precisely
* test(browse): retain Mac qualification evidence through cleanup failures
* test(browse): bound Mac fixture paths and retire its owned user domain
* test(browse): accept vanished fixture entries without weakening cleanup
* test(browse): identify probe-created macOS user domains safely
* test(browse): observe Mac user domains without targeting them first
* test(browse): use passive fresh-user ownership throughout Mac qualification
* test(browse): distinguish profile and registered-home Keychain lookups
* test(browse): qualify Dia under one registered account home
* test(browse): identify Dia startup and owned process-group failures
* test(browse): classify bounded Dia startup diagnostics without leaking output
* fix(test): preserve native Mac sandboxing and reap owned browser children
* fix(browse): preserve Chromium sandboxing for native profile imports
* test(browse): inspect signed Mach-O architecture without launching Xcode tools
* test(browse): sample pending Dia startup and reap on all cleanup paths
* test(browse): compare protected Dia launches in fresh Bun and Node accounts
* test(browse): inspect isolated Mac GUI readiness without browser access
* v1.90.0.0 fix: bind cookie picker actions to their document
* test: validate cookie guards and fit nested launch fixtures
* ci: configure the bundled Chromium sandbox helper
* fix(browse): classify Playwright authentication timeouts
* test: retain bounded Windows lifecycle diagnostics
* test(cso): reuse bounded NTFS precision candidates
* test(review): handle explicit preservation choices safely
* test(browse): remove owned fixture directories with explicit primitives
* test(review): distinguish descriptive reuse from edit commitments
* test: admit only the approved unscored cookie workflow refusal
* test: keep the Office Hours judge mock export-complete
* fix: keep dependency-free CI planners independent of the model SDK
* test: observe the exact holder after a native fixture unlink failure
* fix: start seeded PTY observations at owned readiness
* test: acquire identity-bound Windows deletion admission before profile resets
* test: preserve qualified Git index bits without authorizing mutations
* feat: bind shared-code review advice to source and branch
* feat: add shared-code extraction audit and scoped review checks
* test: recognize complete source reads and explicit coverage legends
* chore: bump version and changelog (v1.88.0.0)
Co-Authored-By: OpenAI Codex <noreply@openai.com>
* test: capture native review questions and retain public evidence
Capture the actual first public native question with strict ownership and display matching. Preserve terminal failures and raw evidence, and retain SDK completion checks.
* test: recognize verified review evidence and complete fixtures
Recognize complete source and diagram evidence, concrete design and developer-experience decisions, and the complete planted scenario contracts. Preserve negative controls and grading thresholds.
* fix: preserve decision brief structure in native questions
Keep the required pros-and-cons heading and final Net field in native question text. Regenerate host outputs and document the release and evaluation repairs.
Co-Authored-By: OpenAI Codex <noreply@openai.com>
* docs: update project documentation for v1.88.0.0
Co-Authored-By: OpenAI Codex <noreply@openai.com>
* fix: correct eval retry accounting and ship workflow gates
* fix: capture native eval evidence and stabilize CI fixtures
* fix: keep shared-code eval skips read-only
Choose explicit no-change answers instead of mixed fix/preservation options.
Reuse the bounded revalidation prompt for path fixtures so required review
metadata is available without repeated discovery. Preserve source checks,
retry limits, and failed native terminal outcomes.
Add captured-question and callback regressions, plus evaluation selection
coverage for the affected fixtures.
---------
Co-authored-by: OpenAI Codex <noreply@openai.com>
* fix(settings): preserve symlinked settings targets
Resolve the selected target for locking, mutation, backup, and rollback; refuse target changes and preserve private modes. Addresses #2830.
* fix(redact): bind masking to original detected spans
Inspired by #2929's anchored-span diagnosis; independently implemented using normalization offsets. Addresses #2930 and the relocation portion of #2912 without changing detection sensitivity.
* fix(evals): exclude operator credentials from prefix admission
Adapts the credential-suffix screen proposed in #2636, with real launched-child regression coverage and deliberate provider-auth exceptions.
* fix(artifacts): retain custom allowlist rules on reinitialization
Preserve the exact user-owned suffix and publish only a successfully assembled replacement. Independently implements the repair reported in #2907.
* test(cso): verify exact masked reads and unmaskable payload refusal
* fix(cso): preserve exact filesystem identities through lease recovery
Preserve 64-bit device/inode identity and nanosecond race checks. Add native NTFS lifecycle coverage for #2927; retain ambiguous legacy-state refusal without claiming Windows PID-reuse recovery is resolved.
* fix(redact): bind pre-push scans to destination and preserve seam context
Uses #2935 (bd07318) as source evidence for push-target range and slice-overlap defects. Independently implemented; no cherry-pick or release metadata adoption.
* test(ci): gate native agent ownership and settings links on macOS
* fix(browse): bind agent lifetimes and cleanup to owned generations
Uses #2931 by Chris Hutton / Claude Fable 5.1 as attributed design input; independently implemented without broad sweeps or copied code. Keep uncertain children and locks rather than deleting foreign state.
* test(ci): include concurrent shutdown controls in the native macOS gate
* v1.88.1.0 fix: harden credential boundaries and owned state
* fix(redact): preserve target provenance and scan boundary semantics
* test(artifacts): read managed rules from atomic allowlist assembly
* fix: preserve native exit observations and fixture prerequisites
* fix: preserve UTF-16 offsets through redaction normalization
* fix: acknowledge seeded plans before invoking review skills
* fix: distinguish current plan input from conversation history
* fix: keep hermetic plan reviews on manual permissions
* fix: distinguish tool discovery from file permission ownership
* fix: preserve initial plan mode in observation tests
* fix: wait for scope decisions before writing review findings
* fix: carry autoplan decisions consistently into review artifacts
* test: retain native failure context in periodic assertions
* fix: advance active file permissions before queued questions
* fix: finish red-team attempts before retry and cleanup
* fix: finalize plan format captures and judges before retry
* fix: cancel setup-gbrain SDK attempts before fixture cleanup
* test: select periodic consumers of the bounded attempt helper
* fix native Bash permission cards and queued questions
* fix: preserve independent decisions and review scope
Keep CEO approach, engineering scope and outside-review choices from approving independent remedies together. Carry declared contracts through DX polish and resolve new gaps before editing the plan. Regenerate every host and retain existing stop boundaries.
Validation: 654 focused tests passed across nine files; all-host generation passed. Full free and periodic validation pending.
Co-Authored-By: OpenAI Codex <noreply@openai.com>
* fix: require approval before design plan amendments
Align the Design review philosophy and rating recipe with its section protocol: resolve one proposed fix, then apply only that approved decision and retain honest scores for declined fixes.
Validation: 469 focused tests passed across four files; all-host generation passed.
Co-Authored-By: OpenAI Codex <noreply@openai.com>
* fix: observe native question completion before transcript persistence
Match owned completion hooks to submitted choices, reject conflicting or late answers, and retain bounded failure evidence.
* test: recognize review posture in acknowledged native questions
Require the selected mode acknowledgement, a completed follow-up question, and its current decoded display while preserving existing posture assertions.
* fix: preserve settled CEO choices and isolate pending remedies
Resolve established approach gates with cited authority and keep independent fixes out of unrelated option commitments and plan amendments.
* fix: carry approved DX choices through later review steps
Choose documentation approaches within the accepted scope and map resolved confusion points without reopening them through a bulk menu.
* test: handle native settings-file edit prompts
Keep one-time owned-file approvals and retain the actual sampled Autoplan permission frame with its matching barrier state.
* test: accept standard CEO reply directives with tuning footers
Recognize the exact trailing preference footer and letter-list directive while preserving current-display and exact acknowledgement checks.
* test: scope split reviewers to their generated plan artifacts
* test: observe native Bash permissions and invocation results
* test: handle owned Bash prompts during mode preference checks
* test: preserve synchronous subprocess rejection in Codex fixture
* Fix periodic review handoff navigation
Recognize review-first and explicit manual-next-step labels while preserving exact action families, manual preference, and ambiguous-menu rejection.
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Bind pending file permissions to distinct current targets
Allow one captured file request to own the complete current dialog while unrelated file work is pending. Preserve same-path ambiguity, exact input ownership, and one-time grant checks.
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Make paired CEO verification choices genuinely unresolved
Start the positive control with proposed manual checks so its unchanged oracle measures two new coverage decisions. Preserve runtime contracts, targets, count bounds, and all assertions.
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Keep CEO review options and verification within approved scope
Audit every offered option for independent add-ons and keep new verification depth pending until accepted. Preserve already requested coverage and trace plan changes to the actual decision.
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Assemble DX review artifacts before appending the final report
Keep early DX evidence above decisions, update artifact sections in place, and append the report using the actual current file suffix. Re-read after deleting an existing report before choosing the append anchor.
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Keep outside plan reviews exclusive and invocation-owned
Follow one preflight-selected backend, terminate failed Codex work before fallback, and allocate extra prompt/output files uniquely. Consume only the current invocation’s completed output.
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Select periodic completion evaluations for report writer changes
Register the shared review resolver for eight missing consumers and regress selection for all nine completion cases without changing their IDs or tiers.
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Keep permission ambiguity fixtures on the same normalized target
Use distinct raw spellings of one target in the four negative fixtures so they exercise the normalized duplicate-owner guard after exact current-file disambiguation. Preserve the existing exception, no-input, diagnostic and cleanup assertions.
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Clarify preserved contracts in engineering review fixture
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Recognize the offered DX follow-up handoff
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Check independent commitments before presenting review options
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Keep Codex review output and status in one shell invocation
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Distinguish seeded plans from reports written by a test attempt
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Recover clipped Autoplan file approvals with bounded viewport resizing
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Recover clipped Bash approvals before binding the complete command
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Isolate setup message tests from the shared checkout
Run the real installer in a temporary payload with private config, require successful completion, and guard source and binary contents and mtimes.
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Fix periodic native permission and report completion handling
Match the pinned CLI's soft wraps and clipped headings without granting from incomplete frames. Retire completed file requests, retain mode annotations, and ask section captures for a short final acknowledgement after their full report is saved.
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Preserve review approvals and validate DX comparison artifacts
Keep independent remedies and approved amendments explicit. Give the synthetic DX review its existing documentation and validate peer comparison as required analysis alongside four native decisions. Add positive and negative semantic calibrations while preserving review counts, model budgets and prompt size limits.
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Make the five-finding CEO fixture's application boundary explicit
Materialize the request adapter and service composition used by the synthetic payment application. Explicitly declare the revised unregistered-event and mail-telemetry assumptions while preserving uncaught handler errors, the original invoice path and all five unresolved findings.
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Keep CEO state-path checks scoped to directory preparation
Co-authored-by: OpenAI Codex <noreply@openai.com>
* Use checked ports and bounded cleanup in pair-agent tests
Discover the daemon port from its owned state file, retain startup diagnostics, and await failed-start cleanup. Add occupied-port, early-exit, deadline, and foreign-state regressions while preserving the existing HTTP assertions and hook budgets.
Co-authored-by: Codex <noreply@openai.com>
* Preserve queued edit identity and recover clipped Bash permissions
Distinguish separately queued unfinished edits from mutation of one native tool ID. Keep grants bound to an exact owned request and reject reused IDs, ambiguous inputs, and competing owners.
Support the pinned renderer's literal em dash and request a repaint when only the Bash card's top rule is clipped. Grants still require the complete fresh card and an exact native acknowledgment.
Validation: 413 integrated parser/event tests passed; private repaint controls and joint source review passed. Full canonical suite and native periodic rerun remain pending.
Co-authored-by: Codex <noreply@openai.com>
* Keep periodic reviews within their approved contracts and deliverables
Carry exact approvals through engineering review, preserve declared contracts when amending CEO plans, and keep prioritization at the requested decision level. Materialize the revised synthetic SDK reference contract while retaining the five original documentation gaps.
Accept the observed semicolon in the finite DX handoff menu and register the direct source dependencies used by the engineering cases. Regenerate canonical review documents without changing model budgets, retries, count bands, or native completion assertions.
Validation: all-host generation and 275 review, fixture, selection and parity tests passed. Full free-suite and native periodic validation remain pending.
Co-authored-by: Codex <noreply@openai.com>
* Keep Eng approval cadence and independence guards explicit
* Accept ordinary punctuation in manual review handoffs
* Recover file permissions alongside queued Bash calls
* Carry approved DX work through later review findings
* Clarify the synthetic auth internal failure decision
* Bound the periodic DX fixture to onboarding changes
* Recognize native Design review handoff labels
* Hold scope in the integration-choice review fixture
* Carry approved Design decisions through review evidence
* Capture listener state when feedback reload fails
* Exclude workspace caches before checking deprecated flags
* Verify Design UI scope against a seeded review plan
* Clarify plan review decisions and outside-voice approval flow
* Reject setup menus in the Design UI gate
* docs: require focused repair validation before final acceptance
* fix: separate review commitments within existing prompt budgets
* docs: align generation and contributor validation guidance
* fix: advance native review prompts and count acknowledged findings
* chore: bump version and changelog (v1.87.1.0)
Co-Authored-By: OpenAI Codex <noreply@openai.com>
* chore: enforce cheap checks and side-effect-free validation previews
* fix: handle owned Fetch permissions and oversized native cards
* test: ground review fixtures in independent executable contracts
* fix: preserve review decisions and verify reports before completion
* test: construct the synthetic credential URL without a scanner false positive
* test: materialize DX examples and verify their actual local behavior
* fix: clarify CEO review decisions and execution order
* fix: clarify review workflow ordering and select Design quality checks
* Fix review decision gates and incomplete evaluation fixtures
Persist CEO and engineering commitment ledgers before menus, preserve exact
approvals, and distinguish implementation structure from feature scope.
Route Autoplan through the canonical CEO Step 0 ordering. Classify DX findings
before requesting approval and ground runtime claims in actual evidence.
Complete neutral non-target fixture contracts and accept the captured Design
handoff purpose without relaxing its ownership or acknowledgment checks.
Record runtime-capability verification in AGENTS.md validation discipline.
Validation: 1,335 focused tests passed across 21 files; build, all-host freshness,
skill validation (647 artifacts / 107 tracked), and credential checks passed.
Prior paid failures are preserved; behavioral acceptance remains pending.
* Fix review decision boundaries and owned Read prompts
Preserve exact approvals across review options, compare consistent DX milestones,
and keep proposed implementation separate from review evidence. Bind modern
Read prompts to one immutable native request and wait for its result.
Retain captured regression verdicts, correct fixture error names, improve import
probe diagnostics, and record focused-first validation discipline in AGENTS.md.
* Clarify CEO and engineering review decisions
Use explicit decision steps, one engineering ledger, and clear scope/write transitions. Preserve exact approvals and distinguish pending test requirements. Keep unrelated generated content unchanged.
* Fix review decision ordering and native evaluation interactions
* Clarify engineering decisions and test artifact order
* Clarify pending choices and approvals in CEO reviews
* Make CEO review phases sequential and clarify completion
* Fix Design board submission intent matching
* Seed an existing browser test baseline for Autoplan
* Document decision-log payloads before state initialization
* Preserve exact review scope and decide one change before drafting options
* Require input identity before repeating passing model judges
* Honor permitted storage throughout CEO review completion
* Match complete native permission text within the pinned renderer contract
* Align review approvals, independent choices, and bounded validation
* fix: preserve reopened approvals and declare fixture interfaces
* fix: isolate review artifacts and audit complete questions
* fix: match detector artifact permissions to configured storage
* fix: complete native permissions and review fixture workflows
* fix: order CEO review work and separate engineering guarantees
* fix: preserve native validation and separate review choices
* fix: clarify review decisions and judge complete report context
* fix: constrain review judgments and retain parse failures
* fix: compare each affected value before review decisions
* fix: make engineering review decisions and completion order explicit
* fix: give the complete Autoplan evaluation a bounded chain budget
* fix(cso): diagnose forbidden Docker endpoints before tool lookup
* fix(reviews): reconcile workflow contracts and generated artifacts after main integration
* fix(evals): migrate retained regressions to the native review harness
* fix(tests): close native harness and workflow integration regressions
* fix(evals): preserve complete permission context and native menu contracts
* fix(tests): capture synchronous command output without pipe drain stalls
* fix(reviews): clarify decision and completion ordering
* fix(reviews): separate decision readiness from final completion checks
* refactor(reviews): consolidate decision rules and completion branches
* fix(plan-eng-review): order preparation and clarify decision routing
* fix(plan-eng-review): restore size and question-format guard parity
* fix(plan-eng-review): clarify scope phases and blocked completion
* fix(plan-eng-review): unify review flow and report destination
* fix(plan-eng-review): define bootstrap and question stage ownership
* fix(plan-eng-review): clarify review structure and design lookup
* fix(plan-eng-review): render report examples and show saved decisions
* fix: consolidate Eng review decisions and select their evaluations
* test: cover overlapping terminal attachments and clean merged runner type
* fix: preserve Office Hours relationship closings during review updates
* fix: retain pasted review targets across slash invocations
* docs: preserve validation traces and correct release scope
* test: cover pasted targets in both review skills
* fix: validate report artifacts before recording success
* fix: redact source roots at CSO report boundaries
* fix: bind native Design questions before answering
* test: select report privacy and native recovery regressions
* test: bind rejection predicate in extracted observers
* fix: bind complete boxed native questions
* test: keep the Design UI fixture on native review
* fix: preserve review decisions and evaluation completion outcomes
* fix: clarify CEO approval and report completion order
* fix: align native review evaluation ownership and completion
* fix: bind review evaluators to native decisions and owned artifacts
* fix: validate review decisions against native outcomes
* fix: preserve review evidence and Autoplan phase handoffs
* test: bind review evidence to owned decisions and completion
* fix: retain owned native history across compaction
* fix(evals): validate current review decisions and setup choices
* fix: bind Autoplan reviews and phase completion to current amended input
* fix: reconcile native review evidence and close Autoplan phases
* test: recognize owned whole-candidate complexity decisions
* test: preserve report freshness for approved investigation handoffs
* fix: recognize scoped review findings and isolate dual voice fixtures
* fix: make review handoffs and question dispatch self-contained
* test: recognize complete CEO decisions and procedural pauses
* fix: bind current CEO comparison options and risk intervals
* test: bind engineering decisions and completion to owned evidence
* fix: publish Autoplan phase reports before continuing tools
* test: verify actual Autoplan dual-review dispatch evidence
* test: select dual review when shared evidence fixtures change
* fix: clarify plan review decisions and completion gates
* fix: make CEO review decisions and return paths explicit
* test: keep Autoplan prompt files inside attempt state
* test: preserve source whitespace across permission dialog wraps
* fix: publish Autoplan phase reports before continuing
* test: recognize current CEO comparisons and reject inactive records
* fix: reconcile engineering decision states before completion
* test: recognize complete Design decisions and reports
* test: verify current engineering decisions before navigation
* Recognize source-owned component reduction choices
* fix: recognize current CEO ledger and commitment grids
* test: supply RequestPolicy context to Eng count fixture
* fix: save complete engineering decisions before asking
* fix: bind Autoplan publication to the complete phase readback
* chore: prepare 1.87.5.0 reliability release
* fix: clarify engineering review completion and preserve log failures
* fix: bind CEO saved choices and current section ancestry
* fix(evals): bind review execution and completion evidence
* fix(plan-ceo-review): verify complete decisions before asking
* fix(evals): preserve complete engineering choice records
* fix(evals): preserve complete review outcomes and bounded fixtures
* fix(autoplan): publish phase reports before advancing
* fix(plan-ceo-review): validate option fields before asking
* fix(plan-eng-review): verify current decisions after answers
* fix(evals): bind review decisions and bound fixture scope
* fix(plan-ceo-review): verify decision rows and edit saved checkpoints
* fix(evals): bind review evidence and scope document lookup
* fix(plan-eng-review): update resolution state with its answer
* fix(reviews): preserve complete questions through dispatch
* fix(evals): recognize completed mode declarations
* fix(evals): define cache consistency at wrapper completion
* fix(evals): validate owned initial scope and completed review handoffs
* fix: assemble complete CEO decision fields before saving
* fix: authenticate automatic mode decisions without guessing selectors
* fix: bind engineering coverage to approved regression contracts
* fix(evals): supply review helpers to native Eng capture
* fix(plan-eng-review): preserve the full selected option scope
* fix(evals): recognize owned engineering seed and regression evidence
* fix(evals): bind engineering retry reports to native approvals
* docs: clarify release guarantees (v1.87.5.0)
Co-Authored-By: OpenAI Codex <noreply@openai.com>
* fix(evals): recognize owned engineering decisions and handoffs
* fix(evals): bind engineering decisions and completion evidence
* fix(tests): align review contracts and selection fixtures
* fix(skills): restore review prompt size limits
* fix(plan-eng-review): clarify review execution and completion
* fix(evals): preserve configured retries through all supervision layers
* Clarify Engineering decisions and report completion
* Keep native decision assertions within their source boundary
* fix: recognize owned engineering decisions and completed navigation
* fix: bind completed auto decisions to their current review
* fix: recognize explicit CEO source attribution
* fix: dispatch verified CEO decisions without recomposing fields
* test: expose existing execution deadlines to review actors
* fix: distinguish CEO decision records from incidental headings
* test: bind split-scope choices to the registered native actor
* test: connect reviewed regressions to required evaluation coverage
* Clarify CEO decision routing and completion stages
* test: expose existing section review deadlines to fixture actors
* test: recognize complete native CEO pacing inventories
* test: exclude answered history from current CEO payloads
* test: detect phase entry through owned skill HOME aliases
* test: validate native review completion and owned report permissions
* fix: make Autoplan close packets carry the parent handoff steps
* test: assess source-bound HOLD decisions within the existing deadline
* fix: keep CEO native decision fields under one formatting authority
* test: register integrated review and permission dependencies
* test: align native review adapters and finding coverage
Preserve explicit AUTO decisions, apply native single-select defaults, and bind complete cropped questions and report permissions to their owned requests. Require seeded review findings instead of crediting setup menus.
Keep captured failure controls and additive selection dependencies. The integrated candidate passed 3,099 focused tests across 65 files; affected paid validation remains required before publication.
* fix(autoplan): require phase reports before advancing
* fix(evals): bind setup and evidence to complete attempts
* fix(evals): bind native answers and pending writes to fixture scope
Preserve complete option rows when native descriptions wrap, retain current
owned Write arguments before journal publication, and keep engineering and
DX answers within their declared fixture interfaces. Add captured free
regressions without increasing model budgets or relaxing completion checks.
* fix(autoplan): verify phase reports across native tool paths
Guard owned methodology reads and reviewer dispatches, detect complete driver
loads through Bash, and distinguish report-only edits from implementation
changes. Follow authenticated native UUID ancestry when journal writes arrive
out of order and verify earlier native content for cached phase reads.
Keep current close acknowledgment and parent publication in order, require CEO
entry before later phases, and register captured failure regressions.
* fix(evals): honor native input and collection lifecycles
Match complete native Edit panes and truncated question borders, reject stderr close before EOF, and stop the CEO split fixture once its acknowledged scope decisions are collected. Keep semantic validation, process failures, report requirements, and absolute deadlines authoritative.
Add captured-event and real-process regressions with selection dependencies. Focused checks pass; final integrated paid and full-suite acceptance remain pending.
* fix(autoplan): retain native session ownership across directory changes
Recover missed native UUID ancestry through the existing strict graph while preserving ordinary event order and legacy scoping. Bind publication hooks to Claude's original project directory while retaining current cwd for requested file paths.
Captured public-event regressions, existing caller checks, and a pinned native CLI loopback verify both fixes. Preserve failed attempts and require fresh paid and final full-suite acceptance.
* docs: align evaluation limits and completion version
* fix(autoplan): allow authenticated phase reads during journal streaming
* fix(evals): bind clipped native questions and owned edit dialogs
* fix: preserve overlay retries and bounded cleanup
* fix: recognize owned planning preludes in native questions
* docs: explain overlay scheduling and cleanup guarantees
* fix: require fresh publication after Autoplan phase reruns
* Release gstack 1.87.6
* fix: preserve CI paths, process identity, and test deadlines
* fix: keep informational setup commands independent of install probes
* fix: clarify plan review decisions and bound source audit reports
* Fix remaining Windows identity and native path CI failures
* Clarify CEO review decision and reviewer-result routing
* test: accept no-install planner in retry supervision
* fix(ceo-review): make review decisions and report completion explicit
* perf(test): add fast PR gates, input-keyed judge reuse and isolated free shards
* fix(test): start isolated CEO smoke from its existing project plan
* fix(test): repair CI fixture races and preserve retry evidence
* fix(ceo-review): clarify approvals, depth and saved completion
---------
Co-authored-by: OpenAI Codex <noreply@openai.com>
* fix: report health failures and coverage accurately
* chore: prepare health reporting release 1.87.4.0
* test: restrict routing evaluations to installed project skills
* test: stabilize health selection and terminal fixtures
garrytan