Commit Graph
563 Commits
Author SHA1 Message Date
garrytan 554d84aecb Regenerate generated output after merging wave/evals 2026-10-03 23:06:27 +00:00
garrytan c0e08a457e Evals lane follow-ups: close the skillify gate TODO (I2 green), file its open reds, CHANGELOG lines; refresh measured parity annotations 2026-10-03 23:06:26 +00:00
garrytan a71fbf9821 Merge lane wave/evals 2026-10-03 23:05:37 +00:00
garrytan dd9e4e75c9 I1: a Codex review eval needs a completed command, not just review-shaped text
Run 37158847998 recorded codex-review-findings as passed again: the agent
wrote 'I couldn't run the review because the execution sandbox failed
(bwrap: Failed to make / slave)', the failure never reached stderr or a
command event, and no phrase rule matched. validateCodexReview now also
requires at least one command_execution event that completed with exit 0.
Fixtures gain that event; the CI output is a new negative control.
2026-10-03 23:02:10 +00:00
garrytan 7ae7776e50 I1: revert the seccomp-only eval container option; it does not let Codex's sandbox start
Run 37158847998 on this branch and a stock Ubicloud VM both show that
seccomp=unconfined only moves bubblewrap's failure from 'No permissions to
create a new namespace' to 'Failed to make / slave: Permission denied'.
Codex's sandbox started only with the host sysctl
kernel.apparmor_restrict_unprivileged_userns=0 plus seccomp and AppArmor
unconfined on the container, which a container job cannot set from its
steps. The relaxation bought nothing, so it goes; the fix is filed as a
TODO for the integrator.

This reverts commit ec97187.
2026-10-03 23:00:22 +00:00
garrytan f9c416fb49 Regenerate generated output after the outside-voice prelude repair 2026-10-03 22:55:25 +00:00
garrytan 091f48a6f7 Free-suite repairs from the first integrated Ubicloud run
Product:
- Outside-voice fences carry runtimeRootPrelude(ctx) explicitly again, so the
  resolver output runs on its own (the post-render pass skips fences that
  already assign GSTACK_ROOT); outsideVoiceRuntime stays deleted.
- D5: an explicit CHROMIUM_PROFILE (gbrowser's per-workspace gbd) is never
  seeded from the shared profile or registered for `browse profiles prune`;
  this was also the free suite's write to the real ~/.gstack/browse-profiles.json.
- gstack-ios-qa-daemon: BASH_COMPAT=50 for its >512-byte --help heredoc.

Tests migrated to the wave's deliberate contract changes:
- B1 heading admits `unverified`; sandbox flag is ${_GSTACK_CODEX_SANDBOX:?};
  verdict-form output (VERDICT/FINDINGS/REASON); fake codex follows the
  preflight + stdin + -o form; Q2 notice marked shown in a private state root.
- C1: an exported root needs bin/ and lib/; a lone GSTACK_BIN is not honored.
- C4: plan-ceo-review is carved on every host (carrier, discovery and
  methodology fixtures read sections/ when the render has it).
- E7: READY prints the resolved Aside path.
- INV-3: the cookie-approval reconstruction reverses the bun-install mktemp template.
- F1's `Plan:` binding read gets a reasoned tracker-guard exemption.
- Browse stubs stop their server instead of scheduling process.exit; the
  headless launch check only counts browsers its own launch started.
2026-10-03 22:55:18 +00:00
garrytan cebc517964 TODOS: /review fingerprint under Codex's read-only .git sandbox (found by the live Codex check) 2026-10-03 22:40:16 +00:00
garrytan 239d9c8982 I5: split the deploy file per case; guard PR-lane shard walls at 600 s
test/skill-e2e-deploy.test.ts ran 668 s as one gate shard in the 2026-10-03
census (six sequential cases, 51-201 s each). It is now case-sharded, so each
case is its own shard with its own (smaller) supervision budget; no timeout
or budget was raised.

- CASE_SHARDED_FILES gains the deploy file (names already equal ids).
- Gate duration seeds for the six case keys from the census JUnit walls, so
  packing stays at 12 gate-census slices (CI cap 230m -> 200m).
- shardDurationViolations + PR_LANE_SHARD_LIMIT_MS: a free test fails when a
  planned gate shard's recorded wall (committed seed, and the last gate census
  in test/fixtures/paid-report-census/gate-census-durations.json) exceeds 600 s.
- Tests that pinned deploy as one shard follow the split (cookie planner keys,
  reuse fixture key, release/PR floors).
2026-10-03 22:37:26 +00:00
garrytan 7eb85e0e3c I4: attribute every census testcase; deselections never count as SKIPPED
The paid report counted every skipped JUnit testcase as SKIPPED and every
testcase whose name was not a registry id as unattributed. On the 2026-10-03
census (run 37151477069) that was SKIPPED 82 / unattributed 72 (periodic) and
SKIPPED 282 / unattributed 24 (gate), and every case-shard sibling also wrote a
skipped history record that reset its pass-rate series.

- attributeJUnitCase: a name is its registry id or CASE_TEST_NAMES label, else
  the only id its file registers, else the keyless file itself. Skipped
  other-tier cases, case-shard siblings, isolated or unselected cases and Bun's
  '(unnamed)' hook placeholders are zero-credit deselections with a reason and
  write no history record. Anything else is listed by name as unattributed.
- One result and one history record per case per shard; a genuinely skipped
  selected case counts as SKIPPED and is listed with its reason, also in
  report-summary.md and collector-outcomes.json. SKIPPED panels show their
  reason and skipped trials are no longer called assertions.
- CASE_TEST_NAMES labels for the three /cso cases.
- tierSkipReason: a computed registration whose lane-tier ids are all
  registered to other test files is excluded at plan time, so the four gate
  and two periodic files that ran with every test skipped no longer take a
  runner.

Census rerun: periodic SKIPPED 0 / unattributed 0, gate SKIPPED 0 /
unattributed 0; verdicts unchanged.
2026-10-03 22:37:26 +00:00
garrytan 76f15815d3 I1: DX floor actor strips the editor hint from the footer line only (from #2971)
The custom-field reply normalization removed 'ctrl+g to edit in <editor>'
from every line, so question text that resembled the menu footer could be
rewritten and still authenticate. Only the last non-empty line is the
footer. Ported from Garry's #2971 with its negative control.
2026-10-03 22:36:46 +00:00
garrytan bc8e2181be test-of-test ratchet: a repo script or setup read by path counts as product code
codex-invocation-sites (runs scripts/gen-skill-docs.ts), gstack-config-gbrain-refresh
(copies bin/<name> through a variable) and setup-cso-windows-probe (reads setup and
scripts/build-cso-windows.ps1) exercise product code the detector could not see.
2026-10-03 22:33:09 +00:00
garrytan 5299cdad0c I3: /qa-only's value-card rule names only its own card location
qa-only rendered the /qa line 'Put it in the 8e.5 record (/qa) or under each
proposed test (/qa-only)'; judge samples for 'qa-only/SKILL.md workflow'
(actionability at the 4.0 gate in run 37151477069, 3.67 in 36907895813)
flag the /qa-only-irrelevant 8e.5 record. The qa-mode bar now renders the
invoking skill's location; both skills get shorter.
2026-10-03 22:31:38 +00:00
garrytan 6c0fe8517d I3: plan-ceo-review defines the closing sequence's cleanup step
Two of three judge samples for 'plan-ceo-review/SKILL.md modes' in periodic
run 37151477069 (actionability at the 4.0 gate) flag 'perform permitted
cleanup' as never defined. The only cleanup the skill has is archiving old
CEO plans, which needs approval; step 5 now says so.
2026-10-03 22:31:38 +00:00
garrytan c7a24ec9c1 I3: plan-eng-review states when findings first enter the report and that the scope menu's recommendation needs no repo probe
Judge rationales for 'plan-eng-review/SKILL.md sections' (actionability 3.67
in periodic run 37151477069, clarity 3 in 36385945043 and 35567915613) cite
two concrete tensions: 'do not add findings before Scope Challenge C' versus
the ledger saves C itself performs, and 'Recommendation: A when a branch diff
exists' versus 'do not probe for session state'. The write policy now says
findings first enter through Scope Challenge C's ledger saves; the selector
says its Recommendation line advises the user and needs no repository
inspection. The pinned sentence becomes the new meaning-level regex.
2026-10-03 22:31:38 +00:00
garrytan 303cdd02c3 I1: plan-eng-review writes the Test Plan Artifact even while test choices are pending
Periodic run 37151477069 (plan-eng-review-artifact): a non-interactive review
recorded every Test review choice as pending and never wrote the QA test
plan, because Step 5 said to write the artifact 'when these test and eval
choices are resolved' while the artifact section says to list unresolved
choices as pending. Both now say to write it after the decision gate, with
unanswered choices under Pending Decisions. The pinned sentence becomes a
meaning-level check.
2026-10-03 22:26:55 +00:00
garrytan c43326b638 Regenerate generated output for v1.91.19.0 (gen:skill-docs --host all, goldens, llms.txt, context-budget fixture) 2026-10-03 22:25:46 +00:00
garrytan 406d01256d Parity caps: measured v1.91.19.0 union/skeleton sizes for ship, review, design-consultation, office-hours and design-shotgun, with the reason beside each 2026-10-03 22:25:45 +00:00
garrytan ed27163db4 I1: shared-libs plan actor reads one exclusion grammar for scope lists
Periodic run 37151477069 refused 'Existing-caller migration and helper
hardening stay out of scope' as a hardening proposal: the actor had two
partial exclusion rules (unanchored is/are/remains outside, and an anchored
list that only stays unchanged), the third refusal at this boundary after
36633323521 and 36776104571. One anchored predicate now covers a bare list
of scope nouns (hyphenated compounds included) with is/are/stays/remains and
unchanged/untouched/excluded/out of scope/outside/not part; any other verb
in the clause is still read as a proposal. Adds the real capture and two
adversarial controls.
2026-10-03 22:25:32 +00:00
garrytan ec97187d96 I1: run paid eval containers with seccomp unconfined so Codex's sandbox can start
Docker's default seccomp profile blocks the user namespaces bubblewrap needs,
so every Codex command in a paid eval failed with 'bwrap: No permissions to
create a new namespace' (codex-review-findings, run 37151477069). Reproduced
in the CI image locally: default profile fails, seccomp=unconfined runs the
command. The runner stays non-root.
2026-10-03 22:23:57 +00:00
garrytan 3a13b6d0c0 I1: Codex evals fail a review whose commands never ran and stop reading tool errors as skill-load problems
validateCodexReview accepted codex-review-findings in periodic run 37151477069
although its only command failed in bwrap ('I couldn't run the review'); it
now requires execution evidence from classifyOutsideReview (B1's INV-1
classifier). codex-sol-scope-termination failed on 'apply_patch verification
failed: invalid hunk' because the skill-load check matched any 'invalid';
it now matches Codex 0.160's skill-loader diagnostics only.
2026-10-03 22:23:57 +00:00
garrytan ed5260a254 I1: give hermetic Codex runs a temp dir beside CODEX_HOME so Codex can create its PATH aliases
Codex 0.160 refuses to create its helper aliases (codex-linux-sandbox among
them) when CODEX_HOME sits under its temp dir, and the runner put the
hermetic home under the shard TMPDIR. Periodic run 37151477069 shows the
warning and 'Unable to spawn codex-linux-sandbox ... No viable candidates
found in PATH' for every exec_command in codex-sol-scope-termination.
2026-10-03 22:23:57 +00:00
garrytan 94c164515d v1.91.19.0 docs and metadata: CHANGELOG, TODOS, README troubleshooting/upgrade/type --selector, BROWSER.md type and profiles, AGENTS.md conventions, CONTRIBUTING #2949, touchfiles for the wave's new lib/bin files 2026-10-03 22:23:54 +00:00
garrytan 3b03df807d Release v1.91.19.0 (next free patch in the queue: #3022 holds 1.91.17.0, #3023 holds 1.91.18.0); migration renamed; engines.bun >=1.4.0 to match the CI pin (#2815) 2026-10-03 22:19:03 +00:00
garrytan c545d4e733 INV-1: gate outcome rows and troubleshooting sections for every lane's new messages 2026-10-03 22:18:16 +00:00
garrytan d98b7c1e0f Q1: type the proxy-transport test doubles (typecheck:test ratchet) 2026-10-03 22:15:53 +00:00
garrytan aab8009b68 I1: grade the AUQ matrix's (recommended) marker on option labels, the field the auto-decide hook reads
The decision-brief format puts a Pros / cons block with 'A) ... (recommended)'
in the question text and the same marker on the option label. The matrix
counted both and failed all six skills in periodic run 37151477069 with
'expected exactly one (recommended) option, found 2'. The question-preference
hook reads labels only, so the gate now checks the native question: a
Recommendation line in the question text and exactly one label ending in
(recommended). Fixtures move to the native question shape and add the real
spec capture plus a marker-only-in-text negative control.
2026-10-03 22:15:44 +00:00
garrytan b2c3e15a1d Q1 (#2971 salvage): bound iOS proxy responses
One wall-clock deadline covers headers and body (stalled and dribbling bodies
time out), the whole body is capped at 64 MiB (upstream_response_too_large),
every terminal path clears its timer, removes listeners and destroys the
request, and the daemon threads the deadline through both listeners. Ambiguous
mutations are still never replayed after a timeout or abort. Ported from
#2971's proxy-transport work plus the size bound.
2026-10-03 22:15:20 +00:00
garrytan f7efb26663 Integrator: C8 router omits disabled skills on every render (relink renders the served Claude router in place); Windows-safe E4/E6 entries; investigate hooks under the cmd.exe check; browse_extension_id reader; order-insensitive test typecheck identities
tsc printed one unchanged ceo-mode-option diagnostic's object members in a
different order (content?/isError?) once lib/fs-utils joined the generator's
import graph; identities now sort object-type members, and old baselines
normalize on read.
2026-10-03 22:11:25 +00:00
garrytan f76974be5a Integrator: outside-voice fences use the shared runtime prelude; TMPDIR mktemp and per-round variant paths in design templates 2026-10-03 22:05:04 +00:00
garrytan e6096a59b9 Merge lane wave/generator 2026-10-03 22:03:44 +00:00
garrytan ac8e30cce6 C4: rewrite section refs on hosts whose root stays a literal ~/<globalRoot>
Kiro, OpenCode, Cursor and Copilot render ~/.claude/skills/gstack as their own
literal root, not $GSTACK_ROOT, so plan-eng-review's self-check pointer
escaped the rewrite (found by the setup lane's installed-pointer check).
2026-10-03 21:51:27 +00:00
garrytan 424214ace7 C1: skillify finds the browse SDK under its own host's install root
The template hardcoded ~/.claude/skills/gstack for the canonical
browse-client.ts, so external hosts read another host's install or nothing.
A new INSTALLED_ROOT resolver (sibling of INSTALLED_BIN_DIR) names the host's
global root, or the per-install root when setup renders for one.
2026-10-03 21:48:59 +00:00
garrytan ea0b4841ba C4: no external render points at a section file that is not installed
design-html's detector install offer and plan-eng-review's self-check pointed
at $GSTACK_ROOT/<skill>/sections/<file> on every external host, where neither
skill is carved and no section tree exists. design-html now inlines the offer
off Claude, and a skill's pointer to its own {{SECTION}}-inlined section
names the inlined text.
2026-10-03 21:48:06 +00:00
garrytan d3640f28f8 B4: design --version prints the version, and the probe uses it
The design CLI treated --version as an unknown command (exit 1), so the
probe launched the binary with no arguments instead. Add --version (prints
the gstack version, exit 0, before auth or network) and probe with it, so
messages read '<path> --version exited <code>' as planned.
2026-10-03 21:46:16 +00:00
garrytan b94288db0b C9: Step 2 falls back to the install registry's source checkout (DX-7)
Codex users who installed from the README's ~/gstack clone got 'ERROR:
gstack not found' from /gstack-upgrade: Step 2 only looked under host skill
directories. It now falls back to the source checkout the install registry
recorded for the current host.
2026-10-03 21:44:22 +00:00
garrytan b8d08f1f4d G2/G3/INV-1: skill-start former-bucket notice and visible learnings failure; TMPDIR mktemp in gbrain bins; best-effort reasons 2026-10-03 21:44:06 +00:00
garrytan 8857089e85 Merge lane wave/setup 2026-10-03 21:43:08 +00:00
garrytan c05c796bc9 Merge lane wave/redaction 2026-10-03 21:43:08 +00:00
garrytan 67d5685ee3 Merge lane wave/ship-review 2026-10-03 21:43:08 +00:00
garrytan 5302d7a1bd Merge lane wave/browse-platform 2026-10-03 21:43:08 +00:00
garrytan 44e03d52aa Merge lane wave/memory 2026-10-03 21:43:08 +00:00
garrytan 93742a1ae2 Merge lane wave/inv1-codex 2026-10-03 21:43:07 +00:00
garrytan dc7a95a4b4 C8: a render given disabled skills drops them from the router (#3018)
The root router named every skill and ends 'When in doubt, invoke the skill',
so it routed to skills the user disabled. gen-skill-docs takes
--disabled-skills (runGeneration disabledSkills): the router drops their
routing rules, keeps the other side of an 'A or B' rule, and names them as
disabled. The router and gstack-upgrade cannot be disabled. With no disabled
skills the render is byte-identical.
2026-10-03 21:42:40 +00:00
garrytan 8d1b6dc470 A2: treat db-unreachable like timeout in gbrain-refresh
The memory lane's new db-unreachable status (network or DNS failure, config
intact) is transient, so gbrain-refresh keeps rendering brain-aware blocks
for it exactly as it does for timeout instead of suppressing them.
2026-10-03 21:41:25 +00:00
garrytan 3f3655b925 A2: render db-unreachable like timeout
gbrain's new db-unreachable status is a transient network failure; a setup
run while offline must not strip brain-aware blocks from every skill.
2026-10-03 21:40:42 +00:00
garrytanandBharadwaj Mittapelli 5526fca45c Q3: office-hours asks open-ended questions in a Q<N> prose form (#2719)
In Conductor, and whenever AskUserQuestion fails, prose is the only channel,
and the prose fallback defined only the D<N> decision brief. Phase 2A/2B
diagnostic questions have no option set, so the model dropped the phase.
Office-hours' preamble now defines the Q<N> open-question form (question,
why I'm asking, what a strong answer sounds like) and routes free-text
replies to it; the diagnostic sections name it. Scoped to office-hours so
every other skill's preamble is unchanged. Ported from PR #2729.

Co-authored-by: Bharadwaj Mittapelli <bharadwajmittapelli@gmail.com>
2026-10-03 21:40:12 +00:00
garrytan 4527c9b93f C3: guard Kiro's router copies on a present source; give the QA pointer adapter the copy and dist helpers
A copy fails where a symlink dangled, so the Kiro router and upgrade copies
are written only when their render exists (as every other runtime root
already does). test/qa-lazy-sections' extracted Kiro installer now supplies
_link_runtime_dists (C2) and the SKILL.md copy helpers (C3).
2026-10-03 21:38:56 +00:00
garrytan 8ae92b2bd0 C7: keep the required-mode hook's exit-2 deny
Copilot CLI treats a preToolUse exit 2 as deny (its hooks reference), and
#2413 established exit 2 plus hookSpecificOutput as the shape Claude Code
enforces, so the deny output stays as on main; only the matcher and the
in-place upgrade of older 'Skill' entries change.
2026-10-03 21:38:56 +00:00
garrytanandVaughan a1969d9287 E4: generators tolerate Bun's spurious EEXIST on Windows (#2329)
Bun on Windows throws EEXIST from a recursive mkdir of an existing directory,
so gen:skill-docs and the llms.txt writer crashed once a second file landed
in a directory. Both now use lib/fs-utils' mkdirpSync (ported from PR #2329
onto the existing helper instead of a new scripts/fs-utils.ts).

Co-authored-by: Vaughan <vaughan@cairnsitguy.com.au>
2026-10-03 21:38:00 +00:00