CalvinBackup/gstack
1
0 Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-10-31 22:14:25 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge wave/d-followups

Browse Source
This commit is contained in:
garrytan committed 2026-10-04 18:27:07 +00:00
2 parents cca93a4475 51d00dcdb8
commit cc46a61f02
75 files changed
+6181 -41
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/OVERLAY_BENCHMARK_CONTRACT.md
+13 -5
View File
@@ -1,4 +1,4 @@
# Overlay benchmark behavior contract v2
# Overlay benchmark behavior contract v4
The paid overlay fixtures verify supported behavior, with comparative efficacy
reported separately as research evidence. A passing case does **not** establish
@@ -7,7 +7,7 @@ blocking efficacy contract; historical failures retain their original verdicts.
The contract is defined by `OVERLAY_CONTRACT` in
`test/helpers/overlay-case-policy.ts`. Every new measurement and aggregate record
identifies version 3. Eval entry names include `contract-v3` so automated result
identifies version 4. Eval entry names include `contract-v4` so automated result
matching cannot silently compare the old and new contracts by the same name.
Selection IDs for the six retained paid cases stay unchanged.
@@ -20,9 +20,17 @@ OFF does not. Each retained fixture runs ten samples per arm, separately for
Every planned sample in both arms must execute successfully, produce exactly
one successful native terminal result without an error flag, preserve workspace
scope, and yield a finite metric within the fixture's bounds. Missing reasoning
metadata, failed output validation, oracle process errors, partial sampling,
deadlines, cleanup failures, or recording failures keep the case failed. Missing
observations are never replaced with zero or discarded to obtain a passing arm.
metadata, oracle process errors, partial sampling, deadlines, cleanup failures,
or recording failures keep the case failed. Missing observations are never
replaced with zero or discarded to obtain a passing arm.
A completed answer that fails the fixture's output check is a measured
correctness result, not an invalid measurement, in either arm: its record has
`measurementPassed: true`, `taskCorrect: false` and the check's message in
`answerError`. Only ON correctness gates the case, so an OFF wrong answer stays
in the comparison and an ON wrong answer fails the case. Version 3 counted an
OFF wrong answer as an invalid measurement (census 37198445662); v3 verdicts
are not reinterpreted.
| Fixture family | Exact task and scope contract | Additional ON requirement | Measurement |
| --- | --- | --- | --- |
docs/evals/detector-inventory.md
+320
View File
@@ -0,0 +1,320 @@
# E2E detector inventory: phrase checks and structured sources
This inventory lists every pass/fail decision in the gate and periodic E2E cases (`E2E_TIERS` in test/helpers/touchfiles-data.ts, plus the three standalone periodic AUQ files) that rests only on phrases or regexes in model free text, at HEAD `cbcccf9`. It has 153 phrase-check rows: 13 are already structured-first (a), 38 are convert candidates (b), 79 need a phrase witness because the contract is user-visible (c), and 23 have no structured source (d); 13 more rows are LLM judges. Shared detectors (S1-S10, F1-F3, A1) are counted once, even though up to nine cases use each. Of the five B1 cases, only auto-decide-preserved has a structured record that can decide its outcome; the other four already read their records first or test text the user sees, so B1 replay corpora protect their phrase checks.
Dispositions:
- **(a) already structured-first.** The check reads a structured record first and falls back to phrases only when the record is absent.
- **(b) convert candidate.** A structured record of the same fact exists in this run, and the contract is not user-visible text.
- **(c) phrase witness required.** The contract is what the user sees (a question shown, a report or message rendered). A record may locate the text but cannot replace it. Numeric format regexes on a shown question, such as `Completeness: N/10`, are (c).
- **(d) no structured source.** An internal fact that nothing in the run records.
- **judge.** An LLM judge. It is listed but is not a phrase check.
"Journal" means the hermetic Claude session JSONL that test/helpers/plan-count-transcript.ts reads. "Surface" means `fullSurface`/`fullOutputSurface`, which includes Skill-injected SKILL.md bodies.
## Shared detectors
| ID | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| S1 | test/helpers/pty/screen.ts:399 via test/helpers/pty/classify.ts:490 | screen `❯ 1.` option list → `asked` | AskUserQuestion tool_use in the journal locates it | yes (a question shown) | (c) |
| S2 | test/helpers/pty/classify.ts:45 via :505 | lettered/numbered prose options, `Recommendation:`, `(recommended)` → `asked` | none when AUQ is disallowed | yes | (c) |
| S3 | test/helpers/pty/screen.ts:186 via test/helpers/pty/classify.ts:467 | "ready to execute" / exit-plan panel → `plan_ready` | ExitPlanMode tool_use in the journal | no | (b) |
| S4 | test/helpers/pty/classify.ts:425 | screen `⏺ Write(path)` outside sanctioned dirs → `silent_write`; plan write before first AUQ → `wrote_findings_before_asking` | Write/Edit tool_use (file_path, order) in the journal | no | (b) |
| S5 | test/helpers/pty/screen.ts:205 via test/helpers/pty/classify.ts:460 | `(auto-decided from plan-tune preference)` → `auto_decided`; runs before the structured path at test/helpers/pty/runners/observation.ts:389 | `gstack-ceo-mode-handoff --auto` (plan-ceo-review/SKILL.md.tmpl:410), question-log `auto_decided:true` (:416; scripts/resolvers/question-tuning.ts:38), read by test/helpers/native-auto-decide.ts:412 | yes (the chat must begin with the handoff line, plan-ceo-review/SKILL.md.tmpl:410); the record can locate it, the transcript line stays the witness | (b) |
| S6 | test/helpers/pty/runners/observation.ts:410-416 | Haiku `waiting` verdict → `asked` | — | — | judge |
| S7 | test/helpers/pty/plan-native.ts:36 via :605 | plan file `^## GSTACK REVIEW REPORT$` with no later `## ` heading | none (rendered plan file) | yes | (c) |
| S8 | test/helpers/plan-mode-evidence.ts:72 → test/helpers/pty/screen.ts:279 | plan file has `## Decisions` on `plan_ready` → fail | none | yes | (c) |
| S9 | test/helpers/pty/classify.ts:148 | squished screen "what should I (design-)review" + "current branch diff" | AUQ tool_use locates it when native AUQ is allowed | yes | (c) |
| S10 | test/helpers/pty/classify.ts:166, test/helpers/plan-scope-selection.ts:54-110, test/helpers/pty/runners/observation.ts:483 | scope-gate auto-select announcement, first-line target wording, seed token on screen | none (no scope-selection record; the seed token is in the pasted seed) | no | (d) |
| F1 | test/helpers/pty/runners/floor.ts:474 → test/helpers/plan-floor-review.ts:217 | staged question judged a `finding` (only pass) | — | — | judge |
| F2 | test/helpers/pty/runners/floor.ts:451 | prose candidate only if viewport and last assistant message both show prose options; native call read first | native AUQ call (structured first) | yes | (a) |
| F3 | test/helpers/pty/runners/floor.ts:497 | screen Write render / plan-ready text → fail | Write/ExitPlanMode tool_use in run.transcript | no | (b) |
| A1 | test/helpers/auq-sdk-capture.ts:30-43 `scoreAuqFormat` | question has ELI10:, Recommendation:, Pros / cons, ✅, ❌, Net:, (recommended) | AUQ tool input (its prose is the contract) | yes | (c) |
Cases that use S1-S8: plan-ceo-review-plan-mode, plan-devex-review-plan-mode, plan-mode-no-op, office-hours-auto-mode (gate); auto-decide-preserved, plan-eng-review-plan-mode, plan-design-review-plan-mode-smoke (periodic). S9-S10: plan-mode-no-op, plan-eng-review-plan-mode, plan-design-review-plan-mode-smoke. F1-F3: plan-ceo-finding-floor, plan-devex-finding-floor (gate); plan-design-finding-floor, plan-eng-finding-floor (periodic). A1: auq-format-gate (gate); the three standalone AUQ files (periodic).
## Gate tier
### test/skill-e2e-plan-mode-no-op.test.ts, test/skill-e2e-plan-ceo-plan-mode.test.ts, test/skill-e2e-plan-devex-plan-mode.test.ts, test/skill-e2e-office-hours-auto-mode.test.ts
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| plan-mode-no-op | test/skill-e2e-plan-mode-no-op.test.ts:114, :175 | screen lacks the plan-mode reminder sentence | none | yes (leaked text) | (c) |
| plan-mode-no-op | test/skill-e2e-plan-mode-no-op.test.ts:186 | screen contains seed token `ZephyrLedgerWidget` | none; the token is in the pasted seed | no | (d) |
| office-hours-auto-mode | test/skill-e2e-office-hours-auto-mode.test.ts:41 | S5 `auto_decided` fails the case | question-log row | no | (b) |
The plan-ceo-review-plan-mode (:89, :108), plan-devex-review-plan-mode (:181-222) and office-hours-auto-mode (:54) verdicts are otherwise S1-S8.
### test/skill-e2e-plan-devex-finding-floor.test.ts, test/skill-e2e-plan-design-with-ui.test.ts
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| plan-devex-finding-floor | test/helpers/plan-floor-review.ts:278-280 `pickPlanFloorProductType` | product-type brief `^Is this…?$` and one exact "Project/branch/task" context; no match leaves the menu unanswered (stall) | AUQ header `Product type` + option labels (already parsed :268-281) | no (selector) | (b) |
| plan-design-with-ui-scope | test/skill-e2e-plan-design-with-ui.test.ts:34-36 | design-focus question prose "I've rated this plan N/10…" or "all 7 … focus … design?" | AUQ header/options of the answered native call | no (boundary id) | (b) |
| plan-design-with-ui-scope | test/skill-e2e-plan-design-with-ui.test.ts:44-57 | review-finding identity from option labels or UI vocabulary | option labels (structured), no design-finding record | no | (b) |
| plan-design-with-ui-scope | test/helpers/plan-review-board-feedback.ts:111-120, :127-138 | outside-voices question lead regex; one `/boards/` URL in prose; mismatch throws | header `Voices` + exact option labels (compared :115-116) | no (actor) | (b) |
| plan-design-with-ui-scope | test/skill-e2e-plan-design-with-ui.test.ts:80-81 | screen prompt `/boards/` without a native call throws | native-call absence decides | no | (a) |
| plan-design-with-ui-scope | test/helpers/pty/classify.ts:555 `COMPLETION_SUMMARY_RE` via test/helpers/pty/runners/counting.ts:554 | screen report/summary/`VERDICT:` text ends collection before the ceiling | review-log row / ExitPlanMode tool_use | no | (b) |
### test/skill-e2e-ask-user-question-format-compliance.test.ts, test/skill-e2e-plan-tune.test.ts, test/skill-e2e-plan.test.ts
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| auq-format-gate | test/skill-e2e-ask-user-question-format-compliance.test.ts:58 | question ~ `options differ in kind` | AUQ tool input | yes | (c) |
| auq-format-gate | test/skill-e2e-ask-user-question-format-compliance.test.ts:61, :68 | recommendation substance ≥ 4 | — | — | judge |
| plan-tune-inspect | test/skill-e2e-plan-tune.test.ts:167-169, :185 | output names ≥2 of 3 seeded question IDs or aliases | question-log is seeded fixture input | yes (inspection report) | (c) |
| plan-review-report | test/skill-e2e-plan.test.ts:463-491 | plan.md `## GSTACK REVIEW REPORT`, CEO/Eng/Design rows, ends with `NO UNRESOLVED DECISIONS` or a bullet | rows come from the seeded review log; the table is rendered | yes | (c) |
### test/skill-e2e-design.test.ts (gate cases)
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| plan-design-review-no-ui-scope | test/skill-e2e-design.test.ts:561-565, :572 | output contains no ui/no frontend/no design/not applicable/backend | none (early exit writes no record) | no | (d) |
| design-review-plugin-handoff | test/skill-e2e-design.test.ts:1012, :1020-1023 | detector-output.md has `IMPECCABLE_READY`, FINDING-001, rule ids, `deferred`, per-handoff `/impeccable x` | probe/scan tool output (checked :994, :1010-1011) | yes (report) | (c) |
| design-review-detector-shim | test/skill-e2e-design.test.ts:1158 | tool outputs or report include `IMPECCABLE_READY` | probe tool output read first | yes | (a) |
| design-review-detector-shim | test/skill-e2e-design.test.ts:1159-1161 | report contains FINDING-001, [ai-color-palette], [low-contrast] | calls.jsonl receipts + engine log (:1132-1154) | yes | (c) |
| design-review-detector-shim-dom | test/skill-e2e-design.test.ts:1254-1255 | report contains [ai-color-palette], "static scan of the rendered DOM" | DOM dump + Bash trace (:1247-1252) | yes | (c) |
### test/skill-e2e-coverage-audit.test.ts, test/skill-e2e-workflow.test.ts
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| review-coverage-audit, plan-eng-coverage-audit | test/helpers/coverage-audit-evidence.ts:546-547 | last JSON `{"tested","untested"}` line first; ASCII-diagram grammar fallback (:459-510) | prompt-requested JSON summary | no | (a) |
| ship-coverage-audit | test/helpers/coverage-audit.ts:34-38 | diagram has gap/no test, tested/✓/★, coverage, both function names | none (the diagram is the requested output) | yes | (c) |
| document-release | test/skill-e2e-workflow.test.ts:101-102, :109 | CHANGELOG keeps old entries; current entry and README mention "feature c" | none (docs are the product) | yes | (c) |
### test/skill-e2e-shared-libs.test.ts, test/skill-e2e-shared-libs-paths.test.ts
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| shared-libs-read-only | test/skill-e2e-shared-libs.test.ts:113-120 | report ~ uncommitted/overlay/raw, tip and branch SHAs, retry-worker/route, no blob link for branch-only file | none (read-only by contract) | yes | (c) |
| shared-libs-unsupported-git | test/skill-e2e-shared-libs.test.ts:136, :144 | report ~ unavailable/unsupported/limited; not "no commits/PRs found" | none | yes | (c) |
| shared-libs-review-path-eligibility | test/skill-e2e-shared-libs-paths.test.ts:133-134 | `trace + output` ~ submodule/160000, check-ignore/ignored (output alone satisfies) | Bash command trace (first operand) | no | (b) |
### test/skill-e2e-ship-docsync.test.ts, test/skill-e2e-docsync-spawned.test.ts
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| ship-docsync-store | test/skill-e2e-ship-docsync.test.ts:74, :78 | ship report ~ `Documentation…blocked`; not `Documentation: current` | no receipt, no publish call (:75-77) | yes (report status) | (c) |
| ship-docsync-completion, ship-docsync-current | test/skill-e2e-ship-docsync.test.ts:81, :97 | parsed docs contract, then report contains `contract.documentation_section` | docs dispatch JSON (ship/sections/documentation.md.tmpl:52-56) | yes | (a) |
| ship-docsync-late-result, -missing-marker, -missing-asset, -launch-failure, -timeout-unsettled, -failure | test/helpers/docsync-fault-eval.ts:28, :30 | non-success report ~ `Documentation…blocked`, no `Documentation: current` outside inline code | actor event log decides dispatch count and publication first (:22-27) | yes | (c) |
| ship-docsync-recovery, -stale-before, -stale-after | test/helpers/docsync-fault-eval.ts:31 | report contains the accepted audit id | actor state `acceptedId` | yes | (a) |
| docsync-spawned | test/skill-e2e-docsync-spawned.test.ts:60 | `contract.blockers` ~ security/sensitive | parsed docs contract | no | (a) |
### test/helpers/ship-hook-actor.ts (ship-hook-consent, ship-hook-refresh files)
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| ship-managed-hook-refresh | test/helpers/ship-hook-actor.ts:122 | final ~ refresh/updat/install/current | installer receipt (checked :113-121) | yes (outcome told to user) | (c) |
| ship-unmanaged-hook-consent | test/helpers/ship-hook-actor.ts:129 | final ~ declin/unchanged/not install/preserv/leave | declined interaction record (:127) | yes | (c) |
| ship-local-hook-preservation | test/helpers/ship-hook-actor.ts:133 | final ~ /manual/ | none | yes | (c) |
### test/skill-e2e-review.test.ts, test/skill-e2e-review-attribution.test.ts, test/skill-e2e-review-army.test.ts (gate cases)
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| review-sql-injection | test/skill-e2e-review.test.ts:87-95 | review-output.md includes sql/injection/sanitiz/parameteriz/… | none: the fixture copies only the checklist, so no review-log row (review/SKILL.md.tmpl:402) | partly | (d) |
| review-enum-completeness | test/skill-e2e-review.test.ts:173-177 | review mentions "returned" and enum/status/critical | none (focused checklist run) | partly | (d) |
| ship-base-branch | test/skill-e2e-review-attribution.test.ts:144 | ship-preflight.md ~ /main\|base/ | git commands in the Bash trace (review-base-branch uses this, :87) | no | (b) |
| review-dashboard-via | test/skill-e2e-review-attribution.test.ts:273, :275 | output + dashboard ~ autoplan, clear | review log is seeded input; dashboard is rendered | yes | (c) |
| review-army-migration-safety | test/skill-e2e-review-army.test.ts:136-142 | output includes drop/data loss/reversib/migration/column | none (single-shot) | partly | (d) |
| review-army-perf-n-plus-one | test/skill-e2e-review-army.test.ts:272-280 | output includes n+1/eager/includes/preload/query/loop | specialist JSON findings in Agent results (review/sections/review-army.md); not verified per finding | no | (b) |
| review-army-delivery-audit | test/skill-e2e-review-army.test.ts:395-404 | output includes not done/missing and email/notification | none verified | partly | (d) |
| review-army-quality-score | test/skill-e2e-review-army.test.ts:494-496 | `SPECIALIST REVIEW: 1 findings (1 critical, 0 informational)`, `PR Quality Score: 8/10`, `[ADVISORY]` | merged-review.json (checked :485-493) | yes (rendered format) | (c) |
### test/skill-e2e-bws.test.ts, test/skill-e2e-session-intelligence.test.ts, test/skill-e2e-learnings.test.ts
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| skillmd-no-local-binary | test/skill-e2e-bws.test.ts:156 | result.output ~ READY/NEEDS_SETUP | setup-block Bash tool_result prints `READY: $B`/`NEEDS_SETUP` (browse/SKILL.md:214) | no | (b) |
| skillmd-outside-git | test/skill-e2e-bws.test.ts:194 | same | same | no | (b) |
| context-recovery-artifacts | test/skill-e2e-session-intelligence.test.ts:181-183, :195 | output contains any of recent artifacts/ceo-plans/last_session/ship/timeline/completed (near-vacuous) | bin/gstack-context-recovery:31, :37 prints the record, but the fixture does not copy it (:46-48) | no | (d) (b once the bin is copied) |
| context-save-writes-file | test/skill-e2e-session-intelligence.test.ts:257, :266 | checkpoint includes `---` and `status:` | the file's YAML frontmatter (parseable) | no | (b) |
| context-restore-loads-latest | test/skill-e2e-session-intelligence.test.ts:379-393, :401-402 | `RESTORED: <newer>` or newer-body phrase; Read of the newer file as fallback | Read tool_use | partly | (c) |
| learnings-show | test/skill-e2e-learnings.test.ts:117-119, :131 | output mentions ≥2 of n-plus-one, stale/cache, rubocop | `gstack-learnings-search` Bash result (fixture copies it, test/skill-e2e-learnings.test.ts:43) | yes (/learn shows them) | (c) |
### test/skill-e2e-qa-functional*.test.ts, test/skill-e2e-skillify.test.ts, test/skill-e2e-deploy.test.ts, test/skill-e2e-cso.test.ts
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| qa-functional-cli-report, -webhook-report, -cli-fix, -webhook-fix | test/helpers/qa-checkpoint-evidence.ts:207-209 | report.md links each `exploration-NNN.json` checkpoint | checkpoint files + gstack-qa-evidence receipts; the link is rendered | yes | (c) |
| same 4 | test/helpers/qa-functional-evidence.ts:179-180 | private sentinel absent from output, report and evidence | n/a (leak canary) | yes | (c) |
| skillify-happy-path | test/skill-e2e-skillify.test.ts:379 → test/helpers/skill-body-narration.ts:7 | committed SKILL.md has no `I `/`I'll`/`Let me` narration | none (generated skill text is the artifact) | yes | (c) |
| skillify-provenance-refusal | test/skill-e2e-skillify.test.ts:429, :467 | surface ~ unknown skill/not registered (load tripwire) | Skill tool_use + its error tool_result | no | (b) |
| skillify-provenance-refusal | test/skill-e2e-skillify.test.ts:444, :468 | assistant ~ "no recent scrape result" / "run /scrape first" | none | yes (refusal, skillify/SKILL.md.tmpl:67) | (c) |
| land-and-deploy-workflow | test/skill-e2e-deploy.test.ts:81 | deploy reports + CLAUDE.md ~ /fly/ | Bash result `PLATFORM:fly` (land-and-deploy/sections/first-run-validation.md:35); not verified this flow runs it | no | (b) |
| land-and-deploy-workflow | test/skill-e2e-deploy.test.ts:82 | artifacts contain `test-app` | none | no | (d) |
| land-and-deploy-first-run | test/skill-e2e-deploy.test.ts:150-151 | report contains fly or first-run-app | Bash result `PLATFORM:fly` (this case runs Step 1.5) | no | (b) |
| land-and-deploy-review-gate | test/skill-e2e-deploy.test.ts:221-222 | readiness report ~ review and not run/no reviews/missing | none (gstack-review-read not installed) | yes | (c) |
| setup-deploy-workflow | test/skill-e2e-deploy.test.ts:408-410 | CLAUDE.md contains fly, my-cool-app, Deploy Configuration | the `## Deploy Configuration` key/value block (setup-deploy/SKILL.md.tmpl:183-190) | no (machine-read) | (b) |
| cso-diff-mode | test/skill-e2e-cso.test.ts:181 | webhook.js finding prose ~ signature/authenticat/forg | report.json has no class/CWE field (lib/cso/contracts.ts:65-80) | no | (d) |
| cso-diff-mode (and the periodic CSO cases) | test/skill-e2e-cso.test.ts:61, :86, :133-134 | CANARY token absent from output, transcript and reports | n/a (leak canary) | yes | (c) |
### test/skill-e2e-third-party-actions.test.ts
The consent question is plain text by design (Read and Bash only). The probe's Bash result (`READY:`/`NEEDS_ASIDE: <os>`/`ASIDE_NOT_RUNNING`, ship/SKILL.md:395-409) records detection, not what was offered.
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| tpa-present | test/skill-e2e-third-party-actions.test.ts:197-203 | Aside, `A)`, defer, dashboard host; no download pitch (regex :41) | probe `READY:` | yes | (c) |
| tpa-absent-linux | test/skill-e2e-third-party-actions.test.ts:218-226; test/helpers/third-party-actions.ts:6, :23 | no pitch, no Aside-drive option, manual + headed-drive options | probe `NEEDS_ASIDE: Linux` | yes | (c) |
| tpa-broken | test/skill-e2e-third-party-actions.test.ts:244, :249-252 | no drive option; "open the Aside app" or a lettered question | probe `ASIDE_NOT_RUNNING` | yes | (c) |
| tpa-absent-darwin | test/skill-e2e-third-party-actions.test.ts:272-274 | pitch sentence, `macOS 15`, no drive options | probe `NEEDS_ASIDE: Darwin` | yes | (c) |
| tpa-apple-ban | test/skill-e2e-third-party-actions.test.ts:307-311 | no drive option for account.apple.com; app-specific password guidance | none | yes | (c) |
### test/helpers/workflow-boundaries-fixture.ts (investigate-owned files), test/skill-e2e-safety-pair-agent.test.ts, test/skill-e2e-safety-ship-evidence.test.ts
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| investigate-owned-completion, -abort, -ending-error | test/helpers/workflow-boundaries-fixture.ts:156-160 | scripted actor: option label ~ abort/stop/cancel or continue/proceed, else unsupported | AUQ input (labels are prose) | yes | (c) |
| investigate-owned-abort | test/helpers/workflow-boundaries-fixture.ts:194 | final ~ abort/stop/cancel | receipts (:186-193) | yes | (c) |
| investigate-owned-ending-error | test/helpers/workflow-boundaries-fixture.ts:209 | final ~ error/unavailable/cannot/unable | receipt `VERIFY_STATUS:69` (:208) | yes | (c) |
| safety-pair-agent-block | test/skill-e2e-safety-pair-agent.test.ts:86, :91 | final contains the instruction block byte-for-byte | block is in the fake tool result; relaying it is the contract | yes | (c) |
| safety-ship-stale-evidence | test/skill-e2e-safety-ship-evidence.test.ts:133 `callJudge` | — | — | — | judge |
Gate cases with no phrase-only checks: the shared-libs review-lifecycle, revalidation, index-flags and prior-coverage cases (review-log rows and traces); ship-skipped-queued-finding; the five qa-callers cases; ship-section-loading; ship-triage (test/helpers/ship-triage-labels.ts:12 parses JSON first, so (a)); ship-coverage-value, review-test-value, test-audit-report-only (JSON records; regexes only select items, so (a)); review-army-json-findings; ship-local-workflow; gstack-upgrade-happy-path; qa-quick; qa-only-no-fix; qa-bootstrap; browse-basic; browse-snapshot; skillmd-setup-discovery; operational-learning; skillify-approval-reject; canary-workflow; benchmark-workflow; retro-base-branch; timeline-event-flow; hermetic-canary; hermetic-sentinel; diagram-triplet; safety-codex-consult-embed.
## Periodic tier
### B1 cases
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| auto-decide-preserved | test/skill-e2e-auto-decide-preserved.test.ts:173 (S5) | screen auto-decided annotation passes | handoff `--auto` + question-log, read only when the screen is unclassified | yes (handoff line) | (b) |
| auto-decide-preserved | test/skill-e2e-auto-decide-preserved.test.ts:159 (S1/S2) | a shown option list fails | prose question only (AUQ disallowed) | yes | (c) |
| plan-ceo-section-loading | test/helpers/ceo-stale-fill-decision.ts:54 ‖ test/helpers/ceo-section-loading-fixture.ts:688 (called test/skill-e2e-plan-ceo-review-section-loading.test.ts:118) | approved ledger row + `currentDecision` naming fill/write/race; else structured-then-prose finding | decision ledger and `currentDecision` blocks in PLAN.md | no | (a) |
| plan-ceo-section-loading | test/helpers/auq-sdk-capture.ts:399-420 (called :101) | `COMPLETION SUMMARY` with Section 1-11 outcome rows | none per section | yes | (c) |
| plan-ceo-section-loading | test/skill-e2e-plan-ceo-review-section-loading.test.ts:74, :110-111 | `## GSTACK REVIEW REPORT`, review table header, `\| CEO Review \|` row | rendered dashboard | yes | (c) |
| plan-ceo-section-loading | test/helpers/auq-sdk-capture.ts:213 (called :113) | report row `Outside Review` status starts `disabled` | none in this native-only run | yes | (c) |
| shared-libs-plan-callers | test/helpers/shared-libs-plan-actor.ts:28-112 (phrase refusals :63, :67, :72, :75, :85, :109) | scripted actor parses question prose: reuse verbs, `lib/retry-after.ts`, contract wording, 9 scope-expansion regexes; a match fails the case | AUQ input; option text is the only carrier of what is offered | yes (question content) | (c) |
| shared-libs-plan-callers | test/skill-e2e-shared-libs-periodic.test.ts:198 | — | — | — | judge |
| plan-design-review-plan-mode | test/skill-e2e-design.test.ts:489, :497, :508 | output ~ `\d+/10` | review-log `initial_score`/`overall_score` (plan-design-review/sections/review-sections.md.tmpl:229); not verified in fixture | yes | (c) |
| plan-design-review-plan-mode | test/skill-e2e-design.test.ts:496-497, :508 | output or rated plan growth has information architecture/interaction state/ai slop/hierarchy | none (review-log has scores and counts only) | no | (d) |
| plan-design-review-plan-mode | test/skill-e2e-design.test.ts:499-504, :511 | plan.md contains empty/loading/error/state/responsive/accessibility | none | no | (d) |
| ship-docsync-late-result | gate tier; listed in the ship-docsync table | — | — | — | see gate |
### Plan reviews: test/skill-e2e-plan-ceo-mode-routing.test.ts, test/skill-e2e-plan-eng-*.test.ts, test/skill-e2e-plan-design-*.test.ts, carve-section-loading
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| plan-ceo-mode-routing | test/helpers/ceo-mode-option.ts:661-670 (called test/skill-e2e-plan-ceo-mode-routing.test.ts:248) | posture regex on the handoff tool-result line first, then post-answer prose | `gstack-ceo-mode-handoff` tool result (plan-ceo-review/SKILL.md.tmpl:410) | no | (a) |
| plan-ceo-mode-routing | test/skill-e2e-plan-ceo-mode-routing.test.ts:257 | HOLD posture review | — | — | judge |
| plan-ceo-mode-routing | test/helpers/pty/classify.ts:341 `MODE_RE` | option label starts with a mode name → mode question found | native AUQ options | yes | (c) |
| plan-design-finding-floor | test/helpers/pty/boundaries.ts:653-658 `pickDesignFocusAll` | Step-0D title, "Project/branch/task" and ELI10 rating prose; no match stalls | AUQ header + option labels (parsed :659-661) | no (selector) | (b) |
| plan-eng-multi-finding-batching | test/helpers/pty/boundaries.ts:62-116, :587-600, :626-640 | setup vs review question decided by prose ("This plan introduces N new classes", "Issue", scope-reduction) | decision ledger R records (plan-eng-review/sections/review-sections.md.tmpl:213-235); qid branches inert under `QUESTION_TUNING=false` | no | (b) |
| plan-eng-multi-finding-batching | test/helpers/eng-seeded-coverage.ts:33-104 | distinct-issue identity per answered AUQ ("Issue N:", D/R-ID, ELI10 gates) | ledger R records, used only as fallback behind the prose gates | no | (b) |
| plan-eng-multi-finding-batching | test/helpers/pty/classify.ts:555 via counting.ts:554 | screen completion text ends counting | review-log row / ExitPlanMode tool_use | no | (b) |
| plan-eng-review-artifact | test/skill-e2e-plan-eng-artifact.test.ts:78 | test-plan artifact ~ dashboard/fetchStats/`/api/stats` | artifact schema has no field naming the change (scripts/resolvers/testing.ts:501-528) | no | (d) |
| carve-section-loading [plan-eng-review] | test/helpers/carve-section-case.ts:120 → test/helpers/auq-sdk-capture.ts:381 | report or output ~ report/review/summary/design doc/handoff | review-log row in the capture state root, deleted before the verdict (auq-sdk-capture.ts:368-370) | no | (b) |
| carve-section-loading [12 other skills] | same | same loose regex | none (Bash not allowed, auq-sdk-capture.ts:358-359) | no | (d) |
The plan-eng-review-plan-mode and plan-design-review-plan-mode-smoke verdicts are S1-S10. plan-eng-finding-floor and plan-design-finding-floor verdicts are F1-F3.
### Question format: test/skill-e2e-plan-format.test.ts, test/skill-e2e-plan-prosons.test.ts, standalone AUQ files
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| plan-ceo-review-format-mode, plan-eng-review-format-kind | test/skill-e2e-plan-format.test.ts:174-175, :377-378 | no `Completeness: N/10`; kind note present | captured question | yes | (c) |
| plan-ceo-review-format-approach, plan-eng-review-format-coverage | test/skill-e2e-plan-format.test.ts:241, :310 | `Completeness: N/10` present | captured question | yes | (c) |
| all four format cases | test/skill-e2e-plan-format.test.ts:177, :243, :312, :380 | recommendation quality | — | — | judge |
| plan-review-prosons-format | test/skill-e2e-plan-prosons.test.ts:184-196 | `D\d+ —`, ELI10:, Stakes…:, Recommendation:, Pros / cons:, ^Net:, ✅≥4, ❌≥2, (recommended) | captured question | yes | (c) |
| plan-review-prosons-hardstop-neg | test/skill-e2e-plan-prosons.test.ts:246-249 | no hard-stop escape line; ✅/❌ counts | captured question | yes | (c) |
| plan-review-prosons-neutral-neg | test/skill-e2e-plan-prosons.test.ts:299-303 | no `taste call`; (recommended); `Recommendation:.*because` | captured question | yes | (c) |
| plan-ceo-review-prosons-cadence | test/skill-e2e-plan-prosons.test.ts:353-358 | Pros / cons header; hard-stop escape or pros+cons | captured question | yes | (c) |
| test/skill-e2e-auq-matrix.test.ts | test/helpers/auq-sdk-capture.ts:54-59 (called :166) | `Recommendation:` line; exactly one `(recommended)` label | AUQ options (the suffix the AUTO_DECIDE hook parses) | yes | (c) |
| test/skill-e2e-auq-consistency.test.ts | test/skill-e2e-auq-consistency.test.ts:65, :87 | each A1 element present in every run | AUQ text | yes | (c) |
| test/skill-e2e-auq-verbose-vs-carved-ab.test.ts | test/skill-e2e-auq-verbose-vs-carved-ab.test.ts:43, :108 | carved format count ≥ verbose | AUQ text | yes | (c) |
| AUQ matrix, consistency, A/B | auq-matrix :148; consistency :69, :95-96; A/B :48, :109 | substance panels and tolerances | — | — | judge |
| plan-ceo-review-expansion-energy | test/skill-e2e-plan.test.ts:271-275 | posture | — | — | judge |
### Shared-libs, review army, codex review, design
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| shared-libs-opportunity-judgment | test/skill-e2e-shared-libs-periodic.test.ts:121-122 | report contains tip SHA and `lib/retry-after.ts` | none | yes | (c) |
| shared-libs-opportunity-judgment | test/skill-e2e-shared-libs-periodic.test.ts:123 | `JSON.stringify(transcript)` ~ inventory.py/search.py/negative inventory (includes assistant prose) | Read/Bash tool-call inputs | no | (b) |
| shared-libs-opportunity-judgment, shared-libs-pr-coverage | test/skill-e2e-shared-libs-periodic.test.ts:88, :125, :158 | — | — | — | judge |
| shared-libs-codex-read-only | test/codex-e2e-shared-libs.test.ts:123-129 | Codex report contains SHA, overlay wording, file names, savings, tests | none | yes | (c) |
| review-army-red-team | test/skill-e2e-review-army.test.ts:651 | output ~ red team/adversarial | Agent/Task dispatch description (perf case reads it, :256-258) | no | (b) |
| review-army-consensus | test/skill-e2e-review-army.test.ts:737-741 | output includes sql/injection/interpolat | specialist JSON findings in Agent results; not verified | no | (b) |
| review-army-simplification | test/skill-e2e-review-army.test.ts:806-815 | output includes native/stdlib/speculative/…; not "lean already. ship." | none | partly | (d) |
| review-army-simplification-precision | test/skill-e2e-review-army.test.ts:868-872 | no delete/shrink JSON finding on the self-check; `NO FINDINGS` sentinel | specialist JSON lines / sentinel | no | (a) |
| review-design-lite | test/skill-e2e-review.test.ts:285-306 | ≥4 of 7 keyword groups | none | partly | (d) |
| codex-review | test/skill-e2e-workflow.test.ts:478, :481 | executed verdict from Codex command output | lib/outside-review-result.ts | no | (a) |
| codex-review | test/skill-e2e-workflow.test.ts:482 | codex-output.md ~ `GATE: PASS\|FAIL (N critical\|UNVERIFIED`, not fail-closed | same verdict | yes (gate line) | (c) |
| design-consultation-core | test/skill-e2e-design.test.ts:160-161, :186, :190 | DESIGN.md has one synonym per 7 section sets; CLAUDE.md mentions design.md | none for content | yes | (c) |
| design-consultation-research | test/skill-e2e-design.test.ts:253, :263 | "Search unavailable" only when no WebSearch/aside call | WebSearch / aside tool calls read first (:243-245) | yes | (a) |
| design-consultation-existing | test/skill-e2e-design.test.ts:314-315, :334-335 | DESIGN.md includes color and spacing | format marker only | yes | (c) |
| design-review-fix | test/skill-e2e-design.test.ts:714, :742 | design-audit.md ~ 47/48px, line-height, radius, padding, spacing, heading | none | yes | (c) |
| design-html-slop-gate | test/skill-e2e-design.test.ts:1318-1319 | gate-output.md has ai-color-palette and accepted | scan call count only | yes | (c) |
| design-consultation-core, plan-decision-classification, plan-devex-peer-comparison-classification, diagram-authoring-quality | design :168; classification :150-153, :40-43; diagram :159-183 | — | — | — | judge |
### Context, scrape, CSO, health, Aside
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| context-save-then-restore-roundtrip | test/skill-e2e-context-skills.test.ts:221, :231 | surface has the magic title (vacuous: it is in the Skill input) | restore Read/Bash tool_result of the saved file | no | (b) |
| context-restore-fragment-match | test/skill-e2e-context-skills.test.ts:266-267, :277-278 | surface has the PAYMENTS marker, lacks ALPHA/OMEGA | Read/Bash tool_result content | no | (b) |
| context-restore-provenance-order | test/skill-e2e-context-skills.test.ts:321-327, :344 | "Next steps" before "Verify first" with tokens placed | none | yes (context-restore/SKILL.md.tmpl:217-221) | (c) |
| context-restore-provenance-order | test/skill-e2e-context-skills.test.ts:328, :335, :338, :345 | `FIRST_ACTION: verify PROV_ASSUMED_7Q` | none (AUQ disallowed) | no | (d) |
| context-restore-empty-state | test/skill-e2e-context-skills.test.ts:374, :385 | surface ~ no saved context/NO_CHECKPOINTS (vacuous: in the injected body) | Bash result `NO_CHECKPOINTS` (context-restore/SKILL.md.tmpl:84, :95) | partly | (b) |
| context-restore-list-delegates | test/skill-e2e-context-skills.test.ts:413, :423 | surface ~ `context-save list` (vacuous) | none | yes | (c) |
| context-restore-legacy-compat | test/skill-e2e-context-skills.test.ts:465-470, :480 | surface has marker/title/filename/branch or pre-rename | Read of the legacy file | no | (b) |
| context-save-list-current-branch | test/skill-e2e-context-skills.test.ts:517, :536 | surface ~ timestamp/main-work | Bash listing result | partly | (b) |
| context-save-list-current-branch | test/skill-e2e-context-skills.test.ts:525-526, :537-538 | final lacks other-branch entries | none (model-rendered list) | yes | (c) |
| context-save-list-all-branches | test/skill-e2e-context-skills.test.ts:572-576, :586 | surface has all 3 timestamps | Bash listing result | partly | (b) |
| scrape-prototype-path | test/skill-e2e-skillify.test.ts:283, :290-294, :304-305 | surface ~ /skillify/ and `"items": [` (both vacuous: in scrape/SKILL.md.tmpl:32, :173) | none | yes | (c) |
| cso-full-audit | test/skill-e2e-cso.test.ts:139-140 | server.js finding prose ~ tenant/owner/authoriz | no class field in report.json | no | (d) |
| cso-infra-scope | test/skill-e2e-cso.test.ts:227-229 | finding prose ~ comment body, run/shell, GITHUB_TOKEN | same | no | (d) |
| health-reporting | test/helpers/health-eval-fixture.ts:132-156 | rendered dashboard: composite 5.6, partial coverage, 0/10 typecheck, "60 errors", unavailable categories, no trend verdict, N/A composite | health-history.jsonl rows (already checked :158-176) | yes | (c) |
| aside-browse-basic, aside-qa-quick, aside-scrape-json | test/skill-e2e-aside.test.ts:179, :219, :234-239 | title in final message; `health score` in report; final JSON contains page1 | none verified | yes | (c) |
| aside-canary-quick | test/skill-e2e-aside.test.ts:256 | canary report ~ CANARY REPORT/Status: | `{date}-canary.json` (canary/SKILL.md.tmpl:221); not verified | no | (b) |
| qa-b6-static, qa-b7-spa, qa-b8-checkout | test/skill-e2e-qa-bugs.test.ts:158, :177-179 | — | — | — | judge |
### Office hours, gbrain, codex, outside voice, safety, benchmarks
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| office-hours-phase4-fork | test/skill-e2e-office-hours-phase4.test.ts:190-191 | written question has `because` and ≥2 option lines | none (AUQ suppressed by the prompt, :128) | yes | (c) |
| office-hours-phase4-fork | test/skill-e2e-office-hours-phase4.test.ts:193 | approach/architecture vocabulary or Server/Client/Hybrid options | none; a native capture would make it structural | no | (d) |
| office-hours-phase4-fork | test/helpers/llm-judge.ts:496-516 (via test/helpers/e2e-helpers.ts:320-322) | deterministic `Recommendation:` + `because` part of judgeRecommendation | none | yes | (c) |
| office-hours-design-draft | test/helpers/office-hours-completion.ts:107-115 | design doc has substantive Problem/Approach/Success/"What I noticed" and Assignment sections | none | yes | (c) |
| setup-gbrain-bad-token | test/skill-e2e-setup-gbrain-bad-token.test.ts:60-61 | `JSON.stringify(result)` ~ error_class AUTH/rotate token (satisfied by the helper's own stdout) | verify JSON `error_class` (bin/gstack-gbrain-mcp-verify:64), already checked :65-67 | partly | (b) |
| setup-gbrain-remote | test/skill-e2e-setup-gbrain-remote.test.ts:326-328 | serialized transcript must not contain `"error_class": NETWORK/AUTH/MALFORMED` | verify JSON in the Bash result | no | (b) |
| setup-gbrain-remote | test/skill-e2e-setup-gbrain-remote.test.ts:352 | CLAUDE.md ~ `Mode: remote-http` | the persisted config block | yes | (c) |
| setup-gbrain-path4-local-pglite | test/skill-e2e-setup-gbrain-path4-local-pglite.test.ts:72-73; test/helpers/setup-gbrain-fixture.ts:122-169 | question ~ code search/pglite/symbol; scripted actor classifies option labels and throws on unknown sets | AUQ labels only, no option ids | yes | (c) |
| sync-gbrain-read-ready, sync-gbrain-read-unknown | test/helpers/sync-gbrain-readiness-verdict.ts:4-25 | Capability row OK/WARN, YELLOW verdict, no unnegated ready claim | bin/gstack-gbrain-read-capability.ts:87 status JSON (status only; row and verdict are rendered) | yes | (c) |
| codex-discover-skill | test/helpers/codex-eval.ts:43 | output ~ review/gstack/skill | none | partly | (d) |
| codex-review-findings | test/helpers/codex-eval.ts:66-67; lib/outside-review-result.ts:88 | output ~ finding/issue/…/no issues; refusal phrases → unavailable | executed commands read first; no review-log step in the fixture | partly | (d) |
| outside-voice-codex-to-claude-code, outside-voice-claude-code-to-codex | test/helpers/outside-voice-evidence.ts:214-216 | outside stdout ~ invoice and authorization defect; no refusal | dispatch and receipts are structural; the review-log row holds status only | no | (d) |
| outside-plan-disabled-no-fallback | test/helpers/disabled-plan-review-fixture.ts:322-325 | JSON `outside_review_this_run: disabled` first, then prose | review-log `outside_status:"disabled"` (scripts/resolvers/outside-voice-steps.ts:331) | yes | (a) |
| outside-plan-disabled-no-fallback | test/helpers/disabled-plan-review-fixture.ts:326-328 | no unattributed "both reviewers agree" claim | none | yes | (c) |
| safety-design-risk-stop | test/skill-e2e-safety-design-risk.test.ts:165 | final names each unfixed FINDING-00N | git log decides which remain; the listing has no record | yes | (c) |
| benchmark-providers-live | test/skill-e2e-benchmark-providers.test.ts:116, :136, :168 | provider output contains "ok" | none | yes | (c) |
| office-hours-forcing-energy, office-hours-builder-wildness, office-hours-phase4-fork, safety-design-risk-stop, llm-judge-recommendation, arm-benchmark-* | office-hours :102-107, :183-188; e2e-helpers.ts:323-326; design-risk :166-174; llm-judge-recommendation :42-163; arm-benchmark :121 (non-gating) | — | — | — | judge |
Periodic cases with no phrase-only checks: plan-ceo-review, plan-ceo-review-selective, plan-eng-review (exit reason and optional file length only); autoplan-dual-voice and autoplan-journal-drift (transcript evidence); the eleven journey-* routing cases (Skill tool input); context-save-routing; scrape-match-path; qa-fix-loop; retro; office-hours-brain-writeback; gbrain-roundtrip-local; safety-codex-boundary; safety-ios-demo; codex-sol-scope-termination; ios-qa-device; design-consultation-preview; the five overlay-harness cases (JSON parse plus deep-equal).
## Convert candidates
These are ordered by red history across the 8 wave censuses (periodic censuses 37174266054 through 37198445662), then by tier and how many cases one change covers.
1. **S5 screen-first auto-decision** (test/helpers/pty/classify.ts:460). auto-decide-preserved went red in 3 of 8 censuses, all `outcome=timeout`. Its replay corpus (test/fixtures/detector-corpora/auto-decide-preserved/) shows two were product failures (the model never sent the handoff line) and one an order false red that is already fixed, so no red came from the screen phrase itself. Letting the bound question-log record hand the decision to the native transcript check also covers office-hours-auto-mode and the plan-mode smokes, but two of the five recent passing captures cannot prove that path from their archived polls (the poll predates the handoff message), so the conversion needs a paid diagnostic run before it lands.
2. **shared-libs-opportunity-judgment transcript regex** (test/skill-e2e-shared-libs-periodic.test.ts:123). The case went red in 2 of 8, but both reds were judge failures, so this conversion removes a fragile check rather than a past red.
3. **S3/S4/F3 PTY terminals** (test/helpers/pty/classify.ts:425, :467; test/helpers/pty/runners/floor.ts:497). 0 reds. Derive `plan_ready` and `silent_write` from ExitPlanMode and Write tool_use in the journal the runner already reads. One change covers 4 gate and 5 periodic cases.
4. **Setup selectors that stall on wording** (test/helpers/plan-floor-review.ts:278-280, test/helpers/pty/boundaries.ts:653-658, test/helpers/plan-review-board-feedback.ts:111-120, test/skill-e2e-plan-design-with-ui.test.ts:34-57). 0 census reds, but PR-lane reds for plan-design-with-ui-scope (runs 37166586458, 37172907071). Identify the question by AUQ header and option labels.
5. **Gate READY probe** in skillmd-no-local-binary and skillmd-outside-git (test/skill-e2e-bws.test.ts:156, :194). Read the setup-block Bash result instead of the final message.
6. **Gate artifact and trace swaps**: context-save-writes-file (parse frontmatter), ship-base-branch (git trace), land-and-deploy-workflow and -first-run (`PLATFORM:` result), setup-deploy-workflow (config block), skillify-provenance-refusal load tripwire (Skill tool_result), shared-libs-review-path-eligibility (drop the `output` operand). Separately, context-recovery-artifacts needs `bin/gstack-context-recovery` copied into its fixture before it can convert.
7. **plan-eng-multi-finding-batching** (test/helpers/pty/boundaries.ts:62-116, test/helpers/eng-seeded-coverage.ts:33-104, test/helpers/pty/classify.ts:555). Count decision-ledger R records bound to answered AUQ ids instead of question prose.
8. **Periodic surface narrowing**: the context-skills roundtrip, fragment, empty-state, legacy and list cases; carve-section-loading [plan-eng-review]; review-army red-team and consensus; setup-gbrain-bad-token and -remote; aside-canary-quick. Several of these phrase checks are vacuous today, because the surface includes the injected SKILL.md body, so narrowing them to tool results will surface new reds rather than remove old ones.
Red cases that are not convert candidates: shared-libs-plan-callers (3 reds, actor phrase refusal on "hardening", (c)); plan-ceo-section-loading (2 reds, report checks (c) and a ledger check already (a)); plan-review-prosons-neutral-neg (2, question format (c)); plan-ceo-review-format-mode and -format-approach, tpa-absent-darwin, and auq-matrix (1 each, (c)); plan-design-review-plan-mode (1, (d)); scrape-prototype-path (1, vacuous (c); the failing assertion is unknown); plan-ceo-mode-routing (1, already (a)); plan-eng-review-artifact (timeout), shared-libs-read-only (filesystem snapshot), shared-libs-review-revalidation (process abort) and the overlay sonnet case (measurement validity) did not fail on a phrase check. The four `SKILL.md workflow`/`platform setup` reds are skill-llm-eval judges outside `E2E_TIERS`.
## Not audited
- Census commits `a8ace39` through `fd6854b` are not in this checkout, so the wave-reds line numbers could not be mapped to assertions; reds are attributed from the trial error text.
- The skill-llm-eval.test.ts judge cases are outside `E2E_TIERS` and are not covered.
- codex-ci-access.test.ts, codex-hardening.test.ts, codex-under-codex-detection.test.ts, e2e-shard-reuse.test.ts and skill-routing-e2e.test.ts name the umbrella keys but are free tests or route to the files audited above.
- plan-eng-review-artifact-full is `marathon` and out of scope.
- The aside-* cases need a live Aside on macOS and never run in CI.
- "Not verified" in the structured-source column means the template writes the record, but the audit did not confirm that the fixture installs the helper.
test/auq-substance-panel.test.ts
+49
View File
@@ -0,0 +1,49 @@
import { describe, test, expect } from 'bun:test';
import * as fs from 'fs';
import * as path from 'path';
import { auqSubstancePanel, AUQ_SUBSTANCE_MIN } from './helpers/auq-substance-panel';
import { JUDGE_PANEL_SAMPLES } from './helpers/llm-judge';
const grader = (scores: number[]) => {
const seen: string[] = [];
let index = 0;
const grade = async (text: string) => {
seen.push(text);
const substance = scores[index++];
if (substance === undefined) throw new Error('grader called more than the panel size');
return { substance, present: true, hadLiteralBecause: true, reason: 'r' };
};
return { grade, seen };
};
describe('auq-matrix substance panel (C4)', () => {
test('one capture is graded by a full panel and gates on the mean at the unchanged minimum', async () => {
expect(AUQ_SUBSTANCE_MIN).toBe(4);
const { grade, seen } = grader([5, 3, 4]);
const panel = await auqSubstancePanel('Recommendation: A because B', grade);
expect(seen).toEqual(Array(JUDGE_PANEL_SAMPLES).fill('Recommendation: A because B'));
expect(panel).toMatchObject({ mean: 4, passed: true });
});
test('a single low sample no longer decides the case, but a low mean still fails', async () => {
expect((await auqSubstancePanel('q', grader([4, 3, 5]).grade)).passed).toBe(true);
expect((await auqSubstancePanel('q', grader([4, 3, 4]).grade))).toMatchObject({ passed: false });
expect((await auqSubstancePanel('q', grader([3, 3, 3]).grade)).mean).toBe(3);
});
test('a failed sample fails the panel without resampling', async () => {
let calls = 0;
const grade = async () => {
calls++;
if (calls === 2) throw new Error('judge returned non-JSON');
return { substance: 5, present: true, hadLiteralBecause: true, reason: 'r' };
};
await expect(auqSubstancePanel('q', grade)).rejects.toThrow('non-JSON');
expect(calls).toBe(JUDGE_PANEL_SAMPLES);
});
test('the matrix scores its single capture through the panel', () => {
const source = fs.readFileSync(path.join(import.meta.dir, 'skill-e2e-auq-matrix.test.ts'), 'utf8');
expect(source.match(/captureNativeFirstAuq\(/g)).toHaveLength(1);
expect(source).toContain('auqSubstancePanel(text)');
expect(source).not.toMatch(/gradeAuqRecommendation\(/);
expect(source).toContain('substance < AUQ_SUBSTANCE_MIN');
});
});
test/ceo-stale-fill-decision.test.ts
+31
View File
@@ -55,3 +55,34 @@ describe('CEO stale-fill decision from the skill-defined ledger and currentDecis
expect(hasApprovedStaleFillDecision('```text\n' + report.replaceAll('```', '') + '\n```')).toBe(false);
});
});
describe('structured reader alone on the plan-ceo-section-loading corpus', () => {
const corpus = path.join(import.meta.dir, 'fixtures', 'detector-corpora', 'plan-ceo-section-loading');
const entry = (name: string): string => JSON.parse(fs.readFileSync(path.join(corpus, name), 'utf8')).input.report;
const camelWrite = entry('37186854666-t1-pass.json');
const camelQuestion = camelWrite.split('\n').find(line => line.startsWith('Question: D1'))!;
test('reads an approved decision whose question names the write as writeProfile (37186854666)', () => {
expect(camelQuestion).toContain('writeProfile');
expect(hasApprovedStaleFillDecision(camelWrite)).toBe(true);
});
test('reads an approved decision under the slash-joined row ID D1 / STALE-FILL (37195203538)', () => {
expect(hasApprovedStaleFillDecision(entry('37195203538-t1-pass.json'))).toBe(true);
});
test('still refuses an unrelated decision, the copied plan and an unresolved ledger row', () => {
expect(hasApprovedStaleFillDecision(entry('37182865432-t1-fail-derived-unrelated-decision.json'))).toBe(false);
expect(hasApprovedStaleFillDecision(CEO_SECTION_CACHE_PLAN)).toBe(false);
expect(hasApprovedStaleFillDecision(entry('37174266054-t1-pass.json'))).toBe(false);
});
test.each(['rewriteProfile', 'isWriteable', 'writeable'])('a question whose only write term is %s is not the race', identifier => {
expect(hasApprovedStaleFillDecision(camelWrite.replace(camelQuestion, camelQuestion.replace('writeProfile', identifier)))).toBe(false);
});
test('a slash-joined ID still needs its own approved ledger row', () => {
const report = entry('37195203538-t1-pass.json');
expect(hasApprovedStaleFillDecision(report.replaceAll('| D1 / STALE-FILL ', '| D1 / OTHER-ROW '))).toBe(false);
});
});
test/detector-corpora.test.ts
+77
View File
@@ -0,0 +1,77 @@
/**
* B1 replay corpora contract. Each case directory under
* test/fixtures/detector-corpora/ holds minimal, redacted detector or actor
* inputs from named census captures, an inventory of every census in scope
* (expired artifacts listed as unavailable), and a replay test
* (test/detector-corpus-<case>.test.ts) that runs the real detector.
* Stale entries (series identity no longer current) are reported for refresh,
* never deleted: a fixed false red stays an expected pass with its history.
*/
import { describe, expect, test } from 'bun:test';
import * as fs from 'fs';
import * as path from 'path';
import { scan } from '../lib/redact-engine';
import { caseSeriesIdentities } from '../scripts/eval-flake-rank';
const ROOT = path.resolve(import.meta.dir, '..');
const DIR = path.join(import.meta.dir, 'fixtures', 'detector-corpora');
const REQUIRED = ['shared-libs-plan-callers', 'plan-ceo-section-loading', 'auto-decide-preserved',
'plan-design-review-plan-mode', 'ship-docsync-late-result'];
const MAX_BYTES = 64 * 1024;
const cases = fs.readdirSync(DIR).filter(name => fs.statSync(path.join(DIR, name)).isDirectory()).sort();
const entries = (c: string) => fs.readdirSync(path.join(DIR, c)).filter(f => f.endsWith('.json') && f !== 'inventory.json').sort()
.map(file => ({ file, bytes: fs.statSync(path.join(DIR, c, file)).size, raw: fs.readFileSync(path.join(DIR, c, file), 'utf8') }))
.map(entry => ({ ...entry, json: JSON.parse(entry.raw) }));
test('every required repeat-offender case has a corpus', () => {
expect(cases).toEqual(expect.arrayContaining(REQUIRED));
});
describe.each(cases)('%s corpus', (c) => {
const all = entries(c);
test('has a replay test, at least one expected pass and one expected refusal', () => {
expect(fs.existsSync(path.join(import.meta.dir, `detector-corpus-${c}.test.ts`))).toBe(true);
expect(all.some(entry => entry.json.expected === 'pass')).toBe(true);
expect(all.some(entry => entry.json.expected === 'refuse')).toBe(true);
});
test.each(all.map(entry => [entry.file, entry] as const))('%s is a bounded, attributed, credential-free entry', (_file, entry) => {
const e = entry.json;
expect(entry.bytes).toBeLessThanOrEqual(MAX_BYTES);
expect(e).toMatchObject({ schema: 'gstack-detector-corpus/v1', case: c });
expect(e.detector).toMatch(/^test\/[\w./-]+\.ts#\S+/);
expect(e.series_identity).toMatch(/^[0-9a-f]{16}$/);
expect(e.source.run).toMatch(/^\d{8,}$/);
expect(e.source.sha).toMatch(/^[0-9a-f]{40}$/);
expect(['periodic', 'gate', 'pr']).toContain(e.source.lane);
expect(['passed', 'failed']).toContain(e.census_outcome);
expect(['pass', 'refuse']).toContain(e.expected);
expect(e.justification.trim().length).toBeGreaterThan(10);
expect(e.input).toBeDefined();
// A refusal that no census produced must say how it was derived.
if (e.expected === 'refuse' && e.census_outcome === 'passed') expect(typeof e.derived).toBe('string');
const scanned = scan(entry.raw, { repoVisibility: 'private' });
expect(scanned.oversize).toBe(false);
expect(scanned.findings.filter(f => f.severity === 'HIGH').map(f => f.id)).toEqual([]);
});
test('the inventory lists every census in scope and only existing entries', () => {
const inventory = JSON.parse(fs.readFileSync(path.join(DIR, c, 'inventory.json'), 'utf8')) as Array<Record<string, unknown>>;
expect(inventory.length).toBeGreaterThan(0);
const files = new Set(all.map(entry => entry.file));
for (const row of inventory) {
expect(String(row.run)).toMatch(/^\d{8,}$/);
expect(typeof row.available).toBe('boolean');
if (row.entry !== null && row.entry !== undefined) expect(files.has(String(row.entry))).toBe(true);
}
for (const entry of all.filter(entry => !entry.json.derived)) {
expect(inventory.some(row => String(row.run) === entry.json.source.run)).toBe(true);
}
});
});
test('stale corpus entries are reported for refresh', () => {
const current = caseSeriesIdentities(cases, ROOT);
const stale = cases.flatMap(c => entries(c).filter(entry => entry.json.series_identity !== current[c])
.map(entry => `${c}/${entry.file} (series ${entry.json.series_identity}, current ${current[c]})`));
if (stale.length) console.log(`[detector-corpora] ${stale.length} stale entr${stale.length === 1 ? 'y' : 'ies'} (refresh from a current census when one fails):\n ${stale.join('\n ')}`);
expect(Object.keys(current).sort()).toEqual(cases);
});
test/detector-corpus-auto-decide-preserved.test.ts
+64
View File
@@ -0,0 +1,64 @@
/** Replays every stored auto-decide-preserved census capture through the observer's real screen and native detectors. */
import { describe, expect, test } from 'bun:test';
import * as fs from 'node:fs';
import * as path from 'node:path';
import { findNativeAutoDecision } from './helpers/native-auto-decide';
import { classifyVisible } from './helpers/pty/classify';
const DIR = path.join(import.meta.dir, 'fixtures/detector-corpora/auto-decide-preserved');
const inventory: Array<{ run: string; outcome: string; available: boolean; entry: string | null }> =
JSON.parse(fs.readFileSync(path.join(DIR, 'inventory.json'), 'utf8'));
const files = fs.readdirSync(DIR).filter(name => name !== 'inventory.json' && name.endsWith('.json')).sort();
const load = (name: string) => JSON.parse(fs.readFileSync(path.join(DIR, name), 'utf8'));
const native = (entry: any) => findNativeAutoDecision(entry.input.native.transcript, entry.input.native.tools, entry.input.native.options);
const screen = (entry: any) => ['auto_decided', 'plan_ready'].includes(classifyVisible(entry.input.screen.text, { strictPlanWrites: true })?.outcome ?? '');
const verdict = (credited: boolean) => credited ? 'pass' : 'refuse';
describe('auto-decide-preserved detector corpus', () => {
test('inventory names every entry, and each entry is bounded and pinned to its census', () => {
expect(inventory.flatMap(row => row.entry ? [row.entry] : []).sort()).toEqual(files);
const verdicts = files.map(name => load(name).expected);
expect(verdicts).toContain('pass');
expect(verdicts).toContain('refuse');
for (const name of files) {
expect(fs.statSync(path.join(DIR, name)).size, name).toBeLessThanOrEqual(64 * 1024);
const entry = load(name);
const row = inventory.find(r => r.entry === name)!;
expect(entry.schema).toBe('gstack-detector-corpus/v1');
expect(entry.case).toBe('auto-decide-preserved');
expect(entry.source.run).toBe(row.run);
expect(entry.source.sha).toMatch(/^[0-9a-f]{40}$/);
expect(entry.census_outcome).toBe(row.outcome);
expect(name).toBe(`${row.run}-t1-${row.outcome === 'passed' ? 'pass' : 'fail'}.json`);
expect(entry.justification.trim()).not.toBe('');
expect(JSON.stringify(entry)).not.toMatch(/\/home\/runner|\/Users\//);
}
});
for (const name of files) {
const entry = load(name);
test(`${name}: the case's detectors ${entry.expected === 'pass' ? 'credit' : 'refuse'} it (census ${entry.census_outcome})`, () => {
expect(verdict(screen(entry) || native(entry) !== null)).toBe(entry.expected);
expect(verdict(screen(entry))).toBe(entry.screen_expected);
const decision = native(entry);
expect(verdict(decision !== null)).toBe(entry.native_expected);
if (decision) expect(decision.option).toBe('HOLD SCOPE');
});
}
test('37186854666: a "Mode application and rationale." heading is not a mode field, but a real unfinished field still withdraws', () => {
const entry = load('37186854666-t1-pass.json');
const last = () => entry.input.native.transcript.assistantMessages.at(-1);
const original = last().text;
expect(original).toContain('**Mode application and rationale.** The draft is a fix to an existing path with its own ceiling already pinned: membership');
expect(native(entry)).toMatchObject({ option: 'HOLD SCOPE' });
for (const [label, edit] of Object.entries({
'colon inside the heading sentence': (t: string) => t.replace('**Mode application and rationale.**', '**Mode application and rationale:** still open.'),
'later pending field': (t: string) => `${t}\n\nMode pending: SCOPE EXPANSION.`,
'later different mode': (t: string) => `${t}\n\nMode: SCOPE EXPANSION.`,
})) {
last().text = edit(original);
expect(native(entry), label).toBeNull();
}
});
});
test/detector-corpus-plan-ceo-section-loading.test.ts
+66
View File
@@ -0,0 +1,66 @@
import { describe, expect, test } from 'bun:test';
import * as fs from 'node:fs';
import * as path from 'node:path';
import { hasApprovedStaleFillDecision } from './helpers/ceo-stale-fill-decision';
import { CEO_SECTION_CACHE_PLAN, hasStaleFillRaceFinding } from './helpers/ceo-section-loading-fixture';
const CASE = 'plan-ceo-section-loading';
const dir = path.join(import.meta.dir, 'fixtures', 'detector-corpora', CASE);
const WITNESS = 'expect(hasApprovedStaleFillDecision(output) || hasStaleFillRaceFinding(output)).toBe(true);';
type Entry = {
schema: string;
case: string;
detector: string;
series_identity: string;
source: { run: string; sha: string; lane: string; artifact: string; path: string };
census_outcome: 'passed' | 'failed';
census_error: string;
derived?: string;
expected: 'pass' | 'refuse';
justification: string;
input: { report: string };
};
type InventoryRow = { run: string; lane: string; outcome: string; series_identity: string; artifact: string; available: boolean; entry: string | null; reason?: string };
const inventory: InventoryRow[] = JSON.parse(fs.readFileSync(path.join(dir, 'inventory.json'), 'utf8'));
const files = fs.readdirSync(dir).filter(name => name !== 'inventory.json').sort();
const entries = files.map(name => [name, JSON.parse(fs.readFileSync(path.join(dir, name), 'utf8')) as Entry] as const);
describe(`${CASE} detector corpus`, () => {
test('the corpus replays the exact stale-fill witness the E2E asserts', () => {
const e2e = fs.readFileSync(path.join(import.meta.dir, 'skill-e2e-plan-ceo-review-section-loading.test.ts'), 'utf8');
expect(e2e).toContain(WITNESS);
});
test('inventory and entry files agree, and each entry is well formed', () => {
expect(inventory.filter(row => row.entry).map(row => row.entry).sort()).toEqual([...files]);
for (const row of inventory) if (!row.entry) expect(row.reason, row.run).toBeTruthy();
for (const [name, entry] of entries) {
expect(fs.statSync(path.join(dir, name)).size, name).toBeLessThanOrEqual(64 * 1024);
expect(entry.schema, name).toBe('gstack-detector-corpus/v1');
expect(entry.case, name).toBe(CASE);
expect(entry.source.sha, name).toMatch(/^[0-9a-f]{40}$/);
expect(name.startsWith(`${entry.source.run}-t`), name).toBe(true);
const row = inventory.find(r => r.entry === name)!;
expect({ outcome: row.outcome, series: row.series_identity }, name).toEqual({ outcome: entry.census_outcome, series: entry.series_identity });
expect(entry.justification.trim().length, name).toBeGreaterThan(0);
expect(entry.input.report, name).not.toMatch(/\/home\/runner|gstack-ceo-secload-/);
}
});
test('holds at least one expected pass and one expected refusal', () => {
const expected = entries.map(([, entry]) => entry.expected);
expect(expected).toContain('pass');
expect(expected).toContain('refuse');
});
test.each(entries)('%s replays to its expected result', (_name, entry) => {
const report = entry.input.report;
expect(hasApprovedStaleFillDecision(report) || hasStaleFillRaceFinding(report)).toBe(entry.expected === 'pass');
});
test('the copied fixture plan is not evidence', () => {
expect(hasApprovedStaleFillDecision(CEO_SECTION_CACHE_PLAN) || hasStaleFillRaceFinding(CEO_SECTION_CACHE_PLAN)).toBe(false);
});
});
test/detector-corpus-plan-design-review-plan-mode.test.ts
+109
View File
@@ -0,0 +1,109 @@
import { describe, expect, test } from 'bun:test';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { spawnSync } from 'node:child_process';
import { CAPTURE_MS, CAPTURE_LONG_MS } from './helpers/eval-budgets';
import { OFFICE_HOURS_BUN_GRACE_MS, runRecordedOfficeHoursAttempt } from './helpers/office-hours-attempt';
import { resolveEvalModel } from '../lib/eval-model';
// Replays stored census captures of plan-design-review-plan-mode through the
// case's real validate callback. The detector lives inline in the paid suite,
// so this evaluates its registration (the plan-design-sdk-fixture pattern)
// and replaces only the model run: the captured final message and saved plan.
const ROOT = path.resolve(import.meta.dir, '..');
const CASE = 'plan-design-review-plan-mode';
const DIR = path.join(ROOT, 'test/fixtures/detector-corpora', CASE);
const source = fs.readFileSync(path.join(ROOT, 'test/skill-e2e-design.test.ts'), 'utf8');
interface CorpusInput { exit_reason: string; output: string; plan_after: string | null }
interface CorpusEntry {
schema: string; case: string; detector: string; series_identity: string;
source: { run: string; sha: string; lane: string; artifact: string; path: string };
census_outcome: 'passed' | 'failed'; census_error: string;
expected: 'pass' | 'refuse'; justification: string; derived?: string; input: CorpusInput;
}
const files = fs.readdirSync(DIR).filter(name => name.endsWith('.json') && name !== 'inventory.json').sort();
const entries = files.map(name => ({ name, entry: JSON.parse(fs.readFileSync(path.join(DIR, name), 'utf8')) as CorpusEntry }));
async function replay(input: CorpusInput) {
const scratch = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'detector-corpus-pdr-')));
const home = path.join(scratch, 'home'); fs.mkdirSync(home);
const env = { PATH: process.env.PATH ?? '', HOME: home, GIT_CONFIG_NOSYSTEM: '1',
GIT_CONFIG_GLOBAL: process.platform === 'win32' ? 'NUL' : '/dev/null',
...(process.env.SystemRoot ? { SystemRoot: process.env.SystemRoot } : {}) };
const rows: Array<{ passed: boolean; exit_reason?: string; error?: string }> = [];
const callbacks: Array<() => Promise<void>> = [];
let workDir = '';
const args = {
ROOT, fs, path, os: { tmpdir: () => scratch }, expect,
CAPTURE_MS, CAPTURE_LONG_MS, OFFICE_HOURS_BUN_GRACE_MS, resolveEvalModel, runId: 'detector-corpus',
evalCollector: { addTest: (row: (typeof rows)[number]) => { rows.push(row); } },
runRecordedOfficeHoursAttempt, logCost: () => {}, recordE2E: () => {},
describeIfSelected: (_title: string, names: string[], fn: () => void) => { if (names.includes(CASE)) fn(); },
testConcurrentIfSelected: (name: string, fn: () => Promise<void>) => { if (name === CASE) callbacks.push(fn); },
spawnSync: (bin: string, argv: string[], opts: { cwd: string; timeout?: number }) =>
spawnSync(bin, argv, { ...opts, env, timeout: opts.timeout ?? 5000 }),
runSkillTest: async (opts: { workingDirectory: string }) => {
workDir = opts.workingDirectory;
expect(path.dirname(workDir)).toBe(scratch);
if (input.plan_after !== null) fs.writeFileSync(path.join(workDir, 'plan.md'), input.plan_after);
return { output: input.output, exitReason: input.exit_reason, duration: 1, toolCalls: [], browseErrors: [],
transcript: [], model: 'detector-corpus-replay',
costEstimate: { inputChars: 0, outputChars: 0, estimatedTokens: 0, estimatedCost: 0, turnsUsed: 0 } };
},
};
const start = source.indexOf("describeIfSelected('Plan Design Review E2E'");
const end = source.indexOf("describeIfSelected('Design Review E2E'", start);
expect(start).toBeGreaterThan(0); expect(end).toBeGreaterThan(start);
let error: unknown;
try {
new Function(...Object.keys(args), new Bun.Transpiler({ loader: 'ts' }).transformSync(source.slice(start, end)))(...Object.values(args));
expect(callbacks).toHaveLength(1);
try { await callbacks[0]!(); } catch (caught) { error = caught; }
expect(workDir).not.toBe('');
expect(fs.existsSync(workDir)).toBe(false);
expect(rows).toHaveLength(1);
return { row: rows[0]!, error };
} finally { fs.rmSync(scratch, { recursive: true, force: true }); }
}
describe(`${CASE} detector corpus`, () => {
test('inventory accounts for every entry and every census in scope', () => {
const inventory = JSON.parse(fs.readFileSync(path.join(DIR, 'inventory.json'), 'utf8')) as Array<{
run: string; lane: string; outcome: string; series_identity: string; artifact: string;
available: boolean; entry: string | null; reason?: string }>;
for (const row of inventory) {
if (row.entry) expect(files).toContain(row.entry);
else expect(row.reason).toBeTruthy();
}
for (const { name, entry } of entries) {
const row = inventory.find(r => r.run === entry.source.run);
expect(row, name).toBeDefined();
expect(entry.census_outcome).toBe(row!.outcome as CorpusEntry['census_outcome']);
expect(entry.series_identity).toBe(row!.series_identity);
if (!entry.derived) expect(row!.entry).toBe(name);
}
expect(entries.some(e => e.entry.expected === 'pass')).toBe(true);
expect(entries.some(e => e.entry.expected === 'refuse')).toBe(true);
});
test.each(entries.map(e => [e.name, e.entry] as const))('%s', async (name, entry) => {
expect(entry.schema).toBe('gstack-detector-corpus/v1');
expect(entry.case).toBe(CASE);
expect(entry.source.sha).toMatch(/^[0-9a-f]{40}$/);
expect(entry.justification).toBeTruthy();
expect(fs.statSync(path.join(DIR, name)).size).toBeLessThanOrEqual(64 * 1024);
const { row, error } = await replay(entry.input);
expect(row.exit_reason).toBe(entry.input.exit_reason);
if (entry.expected === 'pass') {
expect(error).toBeUndefined();
expect(row.passed).toBe(true);
} else {
expect(row.passed).toBe(false);
expect(String(error)).toContain('expect(received)');
}
});
});
test/detector-corpus-shared-libs-plan-callers.test.ts
+46
View File
@@ -0,0 +1,46 @@
/** Replays every stored shared-libs-plan-callers census question through the real fixture actor. */
import { describe, expect, test } from 'bun:test';
import * as fs from 'node:fs';
import * as path from 'node:path';
import { createSharedPlanReuseSelector } from './helpers/shared-libs-plan-actor';
const CASE = 'shared-libs-plan-callers';
const DIR = path.join(import.meta.dir, 'fixtures', 'detector-corpora', CASE);
const inventory: any[] = JSON.parse(fs.readFileSync(path.join(DIR, 'inventory.json'), 'utf8'));
const entries = fs.readdirSync(DIR).filter(file => file !== 'inventory.json' && file.endsWith('.json')).sort()
.map(file => ({ file, size: fs.statSync(path.join(DIR, file)).size, entry: JSON.parse(fs.readFileSync(path.join(DIR, file), 'utf8')) }));
describe(`${CASE} replay corpus`, () => {
test('corpus holds census passes and refusals, each pinned to its source and size-capped', () => {
expect(entries.some(({ entry }) => entry.expected === 'pass')).toBe(true);
expect(entries.some(({ entry }) => entry.expected === 'refuse')).toBe(true);
for (const { file, size, entry } of entries) {
expect(size).toBeLessThanOrEqual(64 * 1024);
expect(entry.schema).toBe('gstack-detector-corpus/v1');
expect(entry.case).toBe(CASE);
expect(entry.detector).toBe('test/helpers/shared-libs-plan-actor.ts#createSharedPlanReuseSelector');
expect(entry.source.sha).toMatch(/^[0-9a-f]{40}$/);
expect(['passed', 'failed']).toContain(entry.census_outcome);
expect(entry.justification.length).toBeGreaterThan(0);
expect(file.startsWith(`${entry.source.run}-t`)).toBe(true);
expect(JSON.stringify(entry.input)).not.toMatch(/\/home\/runner|\/Users\//);
}
});
test('inventory names an existing entry for every available run', () => {
const files = new Set(entries.map(({ file }) => file));
for (const row of inventory) {
if (row.available && row.entry) expect(files.has(row.entry)).toBe(true);
if (!row.entry) expect(row.reason).toBeTruthy();
}
});
test.each(entries.map(({ file, entry }) => [file, entry] as const))('%s', (_file, entry) => {
const choose = createSharedPlanReuseSelector();
if (entry.expected === 'refuse') {
expect(() => choose(entry.input)).toThrow(`${CASE} actor: `);
return;
}
expect(choose(entry.input)).toEqual({ [entry.input.questions[0].question]: entry.expected_answer });
});
});
test/detector-corpus-ship-docsync-late-result.test.ts
+49
View File
@@ -0,0 +1,49 @@
import { describe, expect, test } from 'bun:test';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { docsActorVerdict } from './helpers/docsync-fault-eval';
import type { DocsActorState } from './helpers/docsync-fault-actor';
const CASE = 'ship-docsync-late-result';
const DIR = path.join(import.meta.dir, 'fixtures/detector-corpora', CASE);
const ROOT = path.join(os.tmpdir(), 'docsync-late-result-corpus');
const names = fs.readdirSync(DIR).filter(name => name.endsWith('.json') && name !== 'inventory.json').sort();
const entries = names.map(name => ({ name, raw: fs.readFileSync(path.join(DIR, name), 'utf8') }))
.map(({ name, raw }) => ({ name, size: Buffer.byteLength(raw), entry: JSON.parse(raw.replaceAll('<ROOT>', ROOT)) }));
const inventory = JSON.parse(fs.readFileSync(path.join(DIR, 'inventory.json'), 'utf8'));
describe(`${CASE} detector corpus`, () => {
test('entries are well-formed, bounded, redacted and inventoried', () => {
expect(entries.some(({ entry }) => entry.expected === 'pass')).toBe(true);
expect(entries.some(({ entry }) => entry.expected === 'refuse')).toBe(true);
for (const { name, size, entry } of entries) {
expect(size, name).toBeLessThanOrEqual(64 * 1024);
expect(entry.schema).toBe('gstack-detector-corpus/v1');
expect(entry.case).toBe(CASE);
expect(entry.detector).toBe('test/helpers/docsync-fault-eval.ts#docsActorVerdict');
expect(entry.source.sha).toMatch(/^[0-9a-f]{40}$/);
expect(['passed', 'failed']).toContain(entry.census_outcome);
expect(['pass', 'refuse']).toContain(entry.expected);
expect(entry.justification.length).toBeGreaterThan(0);
expect(JSON.stringify(entry), name).not.toMatch(/\/home\/runner|\/Users\/|sk-[A-Za-z0-9]{8}|ghp_[A-Za-z0-9]{8}/);
if (entry.derived) {
const origin = entries.find(other => !other.entry.derived && other.entry.source.run === entry.source.run);
expect(origin?.entry.expected, `${name} derives from a capture the detector accepts`).toBe('pass');
} else {
expect(name).toBe(`${entry.source.run}-t1-${entry.census_outcome === 'passed' ? 'pass' : 'fail'}.json`);
expect(inventory.find((row: { entry: string | null }) => row.entry === name)?.outcome).toBe(entry.census_outcome);
}
}
for (const row of inventory.filter((row: { entry: string | null }) => row.entry)) expect(names).toContain(row.entry);
for (const row of inventory.filter((row: { entry: string | null }) => !row.entry)) expect(row.reason).toBeTruthy();
});
for (const { name, entry } of entries) {
test(`${name} → ${entry.expected}`, () => {
const failures = docsActorVerdict(entry.input.state as DocsActorState, entry.input.report, entry.input.published);
if (entry.expected === 'pass') expect(failures).toEqual([]);
else expect(failures.length).toBeGreaterThan(0);
});
}
});
test/fixtures/detector-corpora/auto-decide-preserved/37174266054-t1-pass.json
Vendored
+224
View File
@@ -0,0 +1,224 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "auto-decide-preserved",
"detector": "test/helpers/native-auto-decide.ts#findNativeAutoDecision",
"screen_detector": "test/helpers/pty/classify.ts#classifyVisible",
"series_identity": "6c9ea4bac7c0bc81",
"source": {
"run": "37174266054",
"sha": "a8ace39030dd92745b45ac16e5d733e63c936c0f",
"lane": "periodic",
"artifact": "paid-slice-20-a1",
"path": "shards/skill-e2e-auto-decide-preserved/pty-count/ci-37174266054-1-eval-slices-20/plan-ceo-review-1791084936827-pBoN0B/observation.json"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Pre-helper skill (a8ace39): the model sent the documented \"Auto-decided review mode → HOLD SCOPE (your preference)\" line, which classifyVisible accepts on screen; it never ran the question log or handoff helper, so no structured decision exists for the native detector.",
"screen_expected": "pass",
"native_expected": "refuse",
"input": {
"native": {
"transcript": {
"status": "ready",
"calls": [],
"assistantMessages": [
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"text": "Received. I have the draft plan for deterministic skill-list ordering in context, and I understand the handoff constraints: review-mode decision only, full mode name in the audit record and handoff line, no Office Hours or setup, and no substantive review yet. Waiting for your slash command.",
"timestamp": "2026-10-04T03:34:13.255Z"
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"text": "I'll run the skill preamble first, then move through Step 0 only as far as the mode decision you asked for.",
"timestamp": "2026-10-04T03:34:19.295Z"
}
]
},
"tools": [
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:20.346Z",
"toolUseId": "toolu_011kTp99f3Anru9WTVFVQCsZ",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-skill-start --skill \"plan-ceo-review\" --model \"claude\" 2>&1",
"description": "Run gstack skill-start preamble"
}
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:21.872Z",
"toolUseId": "toolu_011kTp99f3Anru9WTVFVQCsZ",
"kind": "result",
"isError": false,
"content": "SKILL_START_PROTO: 1\nBRANCH: main\nPROACTIVE: true\nPROACTIVE_PROMPTED: yes\nSKILL_PREFIX: false\nREPO_MODE: unknown\nSESSION_KIND: interactive\nCONDUCTOR_SESSION: true\nACTIVATED: yes\nFIRST_LOOP_SHOWN: yes\nFIRST_TASK: \nLAKE_INTRO: yes\nTELEMETRY: off\nTEL_PROMPTED: yes\nSESSION_ID: 1611-1791084861-3e00d981\nTEL_START: 1791084861\nEXPLAIN_LEVEL: default\nQUESTION_TUNING: true\nUPDATE_CHECK: false\nLEARNINGS: 0\nHAS_ROUTING: no\nROUTING_DECLINED: false\nVENDORED_GSTACK: no\nMODEL_OVERLAY: claude\nGSTACK_PLAN_MODE: active\nARTIFACTS_SYNC: off\n"
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:34.257Z",
"toolUseId": "toolu_019QAu5FMBCS9XEAt9cdpZd6",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-context-recovery 2>&1 | head -60",
"description": "Recover recent project context"
}
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:38.386Z",
"toolUseId": "toolu_01R2TQ9MyTSkszc3VYsPrKib",
"kind": "use",
"name": "Bash",
"input": {
"command": "echo \"REMOTE: $(git remote get-url origin 2>&1)\"; echo \"HEAD_REF: $(git symbolic-ref refs/remotes/origin/HEAD 2>&1)\"; git rev-parse --verify origin/main >/dev/null 2>&1 && echo \"origin/main exists\" || echo \"no origin/main\"; echo \"--- log\"; git log --oneline -30; echo \"--- stash\"; git stash list; echo \"--- files\"; git ls-files; echo \"--- todos\"; grep -r \"TODO\\|FIXME\\|HACK\\|XXX\" -l --exclude-dir=node_modules --exclude-dir=vendor --exclude-dir=.git . | head -30; echo \"--- recent\"; git log --since=30.days --name-only --format=\"\" | sort | uniq -c | sort -rn | head -20",
"description": "Audit repo history, remote, stashes, TODO markers"
}
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:41.827Z",
"toolUseId": "toolu_01FHxBFVyVFterbEmNDTv81v",
"kind": "use",
"name": "Bash",
"input": {
"command": "setopt +o nomatch 2>/dev/null || true; SLUG=$(~/.claude/skills/gstack/browse/bin/remote-slug 2>/dev/null || basename \"$(git rev-parse --show-toplevel 2>/dev/null || pwd)\"); BRANCH=$(git rev-parse --abbrev-ref HEAD 2>/dev/null | tr '/' '-' || echo 'no-branch'); echo \"SLUG=$SLUG BRANCH=$BRANCH\"; DESIGN=$(~/.claude/skills/gstack/bin/gstack-design-doc-find \"$SLUG\" \"$BRANCH\"); [ -n \"$DESIGN\" ] && echo \"Design doc found: $DESIGN\" || echo \"No design doc found\"",
"description": "Check for an office-hours design doc"
}
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:44.138Z",
"toolUseId": "toolu_017d2MhxfKxcb1SZCvmSXLXY",
"kind": "use",
"name": "Bash",
"input": {
"command": "_CROSS_PROJ=$(~/.claude/skills/gstack/bin/gstack-config get cross_project_learnings 2>/dev/null || echo \"unset\"); echo \"CROSS_PROJECT: $_CROSS_PROJ\"; if [ \"$_CROSS_PROJ\" = \"true\" ]; then ~/.claude/skills/gstack/bin/gstack-learnings-search --limit 10 --cross-project 2>&1; else ~/.claude/skills/gstack/bin/gstack-learnings-search --limit 10 2>&1; fi | head -40",
"description": "Check cross-project setting and search prior learnings"
}
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:46.514Z",
"toolUseId": "toolu_018xZso3XN931CCXMxhRNaqB",
"kind": "use",
"name": "Bash",
"input": {
"command": "SLUG=$(~/.claude/skills/gstack/bin/gstack-slug --get SLUG 2>/dev/null) || true; echo \"SLUG=$SLUG\"; for d in product goals recent-decisions; do echo \"### $d\"; ~/.claude/skills/gstack/bin/gstack-brain-cache get $d --project \"$SLUG\" 2>/dev/null || echo \"(no $d digest available yet)\"; done; echo \"### user-profile\"; ~/.claude/skills/gstack/bin/gstack-brain-cache get user-profile 2>/dev/null || echo \"(no user-profile digest available yet)\"",
"description": "Load brain context digests"
}
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:48.452Z",
"toolUseId": "toolu_01UfuYHFETMDuY3EAP5jsTZZ",
"kind": "use",
"name": "Bash",
"input": {
"command": "printf '%s' \"Select plan-ceo-review mode: SCOPE EXPANSION / SELECTIVE EXPANSION / HOLD SCOPE / SCOPE REDUCTION for deterministic skill-list ordering plan\" | ~/.claude/skills/gstack/bin/gstack-question-preference --check \"plan-ceo-review-mode\" --summary-stdin 2>&1; echo \"EXIT=$?\"",
"description": "Check stored preference for the mode question"
}
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:53.452Z",
"toolUseId": "toolu_01NojwM3bMzrEmdKQtNGJ2KV",
"kind": "use",
"name": "Bash",
"input": {
"command": "_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"; elif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then echo \"NEEDS_ASIDE\"; else _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?; case \"$_rc\" in 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;; 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;; 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"; else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;; *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;; esac; fi",
"description": "Probe Aside browser readiness"
}
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:54.075Z",
"toolUseId": "toolu_0117Cr93gotHMehSf84wAvp1",
"kind": "use",
"name": "Read",
"input": {
"file_path": "<CWD>/PLAN.md"
}
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:54.490Z",
"toolUseId": "toolu_019QAu5FMBCS9XEAt9cdpZd6",
"kind": "result",
"isError": false,
"content": "--- RECENT ARTIFACTS ---\n{\"skill\":\"plan-ceo-review\",\"event\":\"started\",\"branch\":\"main\",\"session\":\"1611-1791084861-3e00d981\",\"ts\":\"2026-10-04T03:34:21.900Z\"}\n--- END ARTIFACTS ---"
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:54.562Z",
"toolUseId": "toolu_01R2TQ9MyTSkszc3VYsPrKib",
"kind": "result",
"isError": false,
"content": "REMOTE: error: No such remote 'origin'\nHEAD_REF: fatal: ref refs/remotes/origin/HEAD is not a symbolic ref\norigin/main exists\n--- log\nf181f79 Seed review plan\n-…[elided: not consumed]"
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:54.593Z",
"toolUseId": "toolu_01FHxBFVyVFterbEmNDTv81v",
"kind": "result",
"isError": false,
"content": "SLUG=gstack-plan-count-66Rq4t BRANCH=main\nNo design doc found"
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:54.644Z",
"toolUseId": "toolu_017d2MhxfKxcb1SZCvmSXLXY",
"kind": "result",
"isError": false,
"content": "CROSS_PROJECT: false"
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:54.756Z",
"toolUseId": "toolu_018xZso3XN931CCXMxhRNaqB",
"kind": "result",
"isError": false,
"content": "SLUG=gstack-plan-count-66Rq4t\n### product\n(no product digest available yet)\n### goals\n(no goals digest available yet)\n### recent-decisions\n(no recent-decisions digest available yet)\n### user-profile\n(no user-profile digest available yet)"
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:54.817Z",
"toolUseId": "toolu_01UfuYHFETMDuY3EAP5jsTZZ",
"kind": "result",
"isError": false,
"content": "AUTO_DECIDE\nEXIT=0"
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:54.834Z",
"toolUseId": "toolu_01NojwM3bMzrEmdKQtNGJ2KV",
"kind": "result",
"isError": false,
"content": "NEEDS_ASIDE"
},
{
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"timestamp": "2026-10-04T03:34:54.842Z",
"toolUseId": "toolu_0117Cr93gotHMehSf84wAvp1",
"kind": "result",
"isError": false,
"content": "1\t# Draft: deterministic skill-list ordering\n2\t\n3\tUsers compare skill listings in scripts and reviews. Make the existing listing\n4\tpath sort registered skill na…[elided: not consumed]"
}
],
"options": {
"skillName": "plan-ceo-review",
"sessionId": "869a270b-e457-41dc-b9ad-2dcb706638a6",
"commandStartedAt": 1791084853562,
"now": 1791084936826,
"proseQuestionObserved": false
}
},
"screen": {
"path": "shards/skill-e2e-auto-decide-preserved/pty-count/ci-37174266054-1-eval-slices-20/plan-ceo-review-1791084936827-pBoN0B/terminal.visible.log",
"tail_chars": 6000,
"text": "n file below):\r\r-0A Premise.Realproblem:skilllistingorderdependsonfilesystemenumeration,sotworunsortwomachinesdiff\r differentlyand scipt comparisons flap. Target outcome:byt-identical lisng foridentical egistered skills.\rDo-nothing cost: nosydiffs n reviews and brittle scripts. The plan hits he paindirectly,not a poxy.\r-0B Leverage. Theexting listing path already renders text and JSON;the change sa sor onthename sequence\r before rendering. Standard-library sort with astable key is the whole implementation. No rebuild.\r-0C Dream state. 12-moth ideal: every user-facing enumeraton inte tool (skills, aliases, learnings)is\r deterministic and testedthat way. This plan is th first lake.\r\rCURRENT STATE THIS PLAN 12-MONTH IDEAL\rlistingorder=readdirordersortnamesbeforerender,all enumerations\rflaps across runs/machines tess for case + enum oderdeterministic + tested\r\rNonewapproachdecisionwasneededin0D:asinglesortcallistheonlycredibleapproachandtheplanalready\rspecifies it.\r\r0E recommendation.Estimatedchangedfiles:2to3(onelistingmoduleedit,oneortwotestfiles).Estimate,nota\rcount from code. Category: fix/refactor of existing behavor, with membrship, aliases,mtadata andouput formas\rexplicitlyfrozen. That points to HOLD SCOPE.\r\rAuto-decidedreviewmode→HOLDSCOPE(yourpreference).Changewith/plan-tune.Approveddecisions:none.HOLDSCOPE\r\r●Preambleresultsthatmatterhere:thesessionisinteractiveunderConductor,questiontuningison,andthestored\r pference for the mode qustion reurned AUTO_DECIDE, so thmo resolves from the recommedation withou amenu.\rBase branch ismain (noorigin remoe, but origin/mainexists). Nodign doc, no prior learings, cold braindigests,\rno TODO markers, clea tree with one seed commit. Aside is not installed,s Iskiped landscape web researchfor\rthis mode-only pass d usd n-distribution knowledge.\r\rTwosetupitemsfiredandIamdeferringbothperyourinstruction:theone-timeCLAUDE.mdrouting-rulesprompt(it\ralso requires aCLAUDE.meditand commit, wich plan mode forbds)andth Office Hours offer. Their gates are\rmarke-based and will re-fire on your nextrun.\r\rStep 0 evidence (compact, recorded to the plan file below):\r\r-0A Premise.Realproblem:skilllistingorderdependsonfilesystemenumeration,sotworunsortwomachinesdiff\r differentlyand scipt comparisons flap. Target outcome:byt-identical lisng foridentical egistered skills.\rDo-nothing cost: nosydiffs n reviews and brittle scripts. The plan hits he paindirectly,not a poxy.\r-0B Leverage. Theexting listing path already renders text and JSON;the change sa sor onthename sequence\r before rendering. Standard-library sort with astable key is the whole implementation. No rebuild.\r-0C Dream state. 12-moth ideal: every user-facing enumeraton inte tool (skills, aliases, learnings)is\r deterministic and testedthat way. This plan is th first lake.\r\rCURRENT STATE THIS PLAN 12-MONTH IDEAL\rlistingorder=readdirordersortnamesbeforerender,all enumerations\rflaps across runs/machines tess for case + enum oderdeterministic + tested\r\rNonewapproachdecisionwasneededin0D:asinglesortcallistheonlycredibleapproachandtheplanalready\rspecifies it.\r\r0E recommendation.Estimatedchangedfiles:2to3(onelistingmoduleedit,oneortwotestfiles).Estimate,nota\rcount from code. Category: fix/refactor of existing behavor, with membrship, aliases,mtadata andouput formas\rexplicitlyfrozen. That points to HOLD SCOPE.\r\rAuto-decidedreviewmode→HOLDSCOPE(yourpreference).Changewith/plan-tune.Approveddecisions:none.HOLDSCOPE\rapplies becaus theplanfreezes every output contrat except ordering, so he value f the revew is rigor on the one\r●Preambleresultsthatmatterhere:thesessionisinteractiveunderConductor,questiontuningison,andthestored\r pference for the mode qustion reurned AUTO_DECIDE, so thmo resolves from the recommedation withou amenu.\rBase branch ismain (noorigin remoe, but origin/mainexists). Nodign doc, no prior learings, cold braindigests,\rno TODO markers, clea tree with one seed commit. Aside is not installed,s Iskiped landscape web researchfor\rthis mode-only pass d usd n-distribution knowledge.\r\rTwosetupitemsfiredandIamdeferringbothperyourinstruction:theone-timeCLAUDE.mdrouting-rulesprompt(it\ralso requires aCLAUDE.meditand commit, wich plan mode forbds)andth Office Hours offer. Their gates are\rmarke-based and will re-fire on your nextrun.\r\rStep 0 evidence (compact, recorded to the plan file below):\r\r-0A Premise.Realproblem:skilllistingorderdependsonfilesystemenumeration,sotworunsortwomachinesdiff\r differentlyand scipt comparisons flap. Target outcome:byt-identical lisng foridentical egistered skills.\rDo-nothing cost: nosydiffs n reviews and brittle scripts. The plan hits he paindirectly,not a poxy.\r-0B Leverage. Theexting listing path already renders text and JSON;the change sa sor onthename sequence\r before rendering. Standard-library sort with astable key is the whole implLine truncated
}
}
}
test/fixtures/detector-corpora/auto-decide-preserved/37176837432-t1-fail.json
Vendored
+321
View File
@@ -0,0 +1,321 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "auto-decide-preserved",
"detector": "test/helpers/native-auto-decide.ts#findNativeAutoDecision",
"screen_detector": "test/helpers/pty/classify.ts#classifyVisible",
"series_identity": "6c9ea4bac7c0bc81",
"source": {
"run": "37176837432",
"sha": "7249f014648202840af294b730ac244e03ce514c",
"lane": "periodic",
"artifact": "paid-slice-20-a1",
"path": "shards/skill-e2e-auto-decide-preserved/pty-count/ci-37176837432-1-eval-slices-20/plan-ceo-review-1791088108875-8x79js/observation.json"
},
"census_outcome": "failed",
"census_error": "AUTO_DECIDE preserve test inconclusive: outcome=timeout",
"expected": "refuse",
"justification": "Genuine product failure on 7249f01: the model checked the preference and logged HOLD SCOPE but never sent the Step 0E \"Auto-decided review mode → …\" line the skill required (\"HOLD SCOPE was auto-selected…\" instead), so no current declaration exists; repaired in the product by e3546d9 (gstack-ceo-mode-handoff prints the line), not in the detector.",
"screen_expected": "refuse",
"native_expected": "refuse",
"derived": "options.stateEvidence rebuilt from the session's gstack-question-log argument with source \"agent\" and ts = that call's tool-result timestamp (the bound question-log.jsonl itself is not archived)",
"input": {
"native": {
"transcript": {
"status": "ready",
"calls": [],
"assistantMessages": [
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"text": "Received. I have the draft plan for deterministic skill-list ordering in context, along with your constraints: this invocation is for the review-mode decision only, the full mode name goes in the audit record and handoff line, and Office Hours, setup, and the substantive review are deferred. I'll wait for your slash command before taking any action.",
"timestamp": "2026-10-04T04:26:06.989Z"
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"text": "I'll run the skill preamble first, then do the pre-review audit and Step 0 up through the mode-selection question, stopping there per your instruction.",
"timestamp": "2026-10-04T04:26:13.842Z"
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"text": "Aside is not installed, so the landscape check falls back to one read-only WebSearch query.",
"timestamp": "2026-10-04T04:27:20.063Z"
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"text": "The repo only has PLAN.md/CLAUDE.md with no prior context, and the actual listing code isn't in this checkout, so I confirmed via web research the standard fix: sort entries by code point/bytes (not locale) with a stable tie-break to avoid readdir-order fallback. I'll proceed in HOLD SCOPE mode, applying max rigor to just the comparator, case handling, tie-breaking, and the enumeration-order test across the ~2-3 affected files.\n\n",
"timestamp": "2026-10-04T04:28:04.939Z"
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"text": "The mode decision is recorded. HOLD SCOPE was auto-selected from your saved preference for the mode question, so no menu was shown.\n\n**Provenance.** The preference check for the mode question returned AUTO_DECIDE. The recommendation was HOLD SCOPE because the draft is a fix with a hard \"outputs unchanged\" invariant and an estimated 2 to 3 changed files. The audit record was written with the full mode name in both the user_choice and recommended fields, flagged as auto-decided, under this session's ID.\n\n**Deferred per your instructions:**\n- The Office Hours offer was not made, since no design doc exists and you said you would handle it separately.\n- The skill-start output carried a one-time setup prompt to add skill routing rules to CLAUDE.md. That is setup, and plan mode also forbids editing CLAUDE.md or committing, so it stays pending and will re-fire on the next healthy run.\n- The substantive review (0G HOLD SCOPE checks, 0I, Sections 1 to 11, report) has not started. No plan file was written and ExitPlanMode was not called, since the workflow is paused rather than complete.\n\n**For the later review, two facts to carry forward:** the listing code is not in this checkout, so the real codebase is needed for the existing-code and file-count checks. The web research points to code-point sorting with a stable tie-break and warns against locale collation, which is the main design question HOLD SCOPE should pin down.\n\nTelemetry is off for this install. No durable learnings this session.",
"timestamp": "2026-10-04T04:28:18.729Z"
}
]
},
"tools": [
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:15.120Z",
"toolUseId": "toolu_01Mg7qvBkiqYfnxDHB3GGo38",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-skill-start --skill \"plan-ceo-review\" --model \"claude\" 2>&1",
"description": "Run gstack skill-start preamble"
}
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:16.495Z",
"toolUseId": "toolu_01Mg7qvBkiqYfnxDHB3GGo38",
"kind": "result",
"isError": false,
"content": "SKILL_START_PROTO: 1\nBRANCH: main\nPROACTIVE: true\nPROACTIVE_PROMPTED: yes\nSKILL_PREFIX: false\nREPO_MODE: unknown\nSESSION_KIND: interactive\nCONDUCTOR_SESSION: true\nACTIVATED: yes\nFIRST_LOOP_SHOWN: yes\nFIRST_TASK: \nLAKE_INTRO: yes\nTELEMETRY: off\nTEL_PROMPTED: yes\nSESSION_ID: 1581-1791087976-5a3dd6a3\nTEL_START: 1791087976\nEXPLAIN_LEVEL: default\nQUESTION_TUNING: true\nUPDATE_CHECK: false\nLEARNINGS: 0\nHAS_ROUTING: no\nROUTING_DECLINED: false\nVENDORED_GSTACK: no\nMODEL_OVERLAY: claude\nGSTACK_PLAN_MODE: active\nARTIFACTS_SYNC: off\n"
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:29.021Z",
"toolUseId": "toolu_01218vLzt9Q42cBrvtbAoJBY",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-context-recovery 2>&1 | head -40",
"description": "Recover recent project context"
}
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:33.153Z",
"toolUseId": "toolu_01UnmMKxYmqwrawXsscjtSkg",
"kind": "use",
"name": "Bash",
"input": {
"command": "echo \"REMOTE: $(git remote get-url origin 2>&1)\"; echo \"HEAD_REF: $(git symbolic-ref refs/remotes/origin/HEAD 2>&1)\"; git rev-parse --verify origin/main >/dev/null 2>&1 && echo \"origin/main exists\" || echo \"no origin/main\"; echo \"---LOG\"; git log --oneline -30; echo \"---STASH\"; git stash list; echo \"---TODO\"; grep -r \"TODO\\|FIXME\\|HACK\\|XXX\" -l --exclude-dir=node_modules --exclude-dir=vendor --exclude-dir=.git . | head -30; echo \"---RECENT\"; git log --since=30.days --name-only --format=\"\" | sort | uniq -c | sort -rn | head -20; echo \"---FILES\"; find . -path ./.git -prune -o -type f -print | head -50",
"description": "Audit git history, stashes, TODOs, and repo files"
}
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:36.531Z",
"toolUseId": "toolu_01YXR5JPFMhnCgTYtFAgyiNm",
"kind": "use",
"name": "Bash",
"input": {
"command": "setopt +o nomatch 2>/dev/null || true; SLUG=$(~/.claude/skills/gstack/browse/bin/remote-slug 2>/dev/null || basename \"$(git rev-parse --show-toplevel 2>/dev/null || pwd)\"); BRANCH=$(git rev-parse --abbrev-ref HEAD 2>/dev/null | tr '/' '-' || echo 'no-branch'); DESIGN=$(~/.claude/skills/gstack/bin/gstack-design-doc-find \"$SLUG\" \"$BRANCH\"); [ -n \"$DESIGN\" ] && echo \"Design doc found: $DESIGN\" || echo \"No design doc found\"; echo \"SLUG=$SLUG BRANCH=$BRANCH\"",
"description": "Check for an office-hours design doc"
}
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:38.908Z",
"toolUseId": "toolu_01N1GRkthVykqYzPXR9aorYu",
"kind": "use",
"name": "Bash",
"input": {
"command": "_CROSS_PROJ=$(~/.claude/skills/gstack/bin/gstack-config get cross_project_learnings 2>/dev/null || echo \"unset\"); echo \"CROSS_PROJECT: $_CROSS_PROJ\"; if [ \"$_CROSS_PROJ\" = \"true\" ]; then ~/.claude/skills/gstack/bin/gstack-learnings-search --limit 10 --cross-project 2>&1; else ~/.claude/skills/gstack/bin/gstack-learnings-search --limit 10 2>&1; fi | head -30",
"description": "Load prior learnings and cross-project setting"
}
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:41.221Z",
"toolUseId": "toolu_011axfbcZwL2pxicmsUZVAWu",
"kind": "use",
"name": "Bash",
"input": {
"command": "SLUG=$(~/.claude/skills/gstack/bin/gstack-slug --get SLUG 2>/dev/null) || true; for d in product goals recent-decisions; do echo \"### $d\"; ~/.claude/skills/gstack/bin/gstack-brain-cache get $d --project \"$SLUG\" 2>/dev/null || echo \"(no $d digest available yet)\"; done; echo \"### user-profile\"; ~/.claude/skills/gstack/bin/gstack-brain-cache get user-profile 2>/dev/null || echo \"(no user-profile digest available yet)\"",
"description": "Load brain context digests"
}
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:42.974Z",
"toolUseId": "toolu_01JBffEz48oEirpACN9SnH9s",
"kind": "use",
"name": "Bash",
"input": {
"command": "printf '%s' \"Select plan-ceo-review mode: SCOPE EXPANSION, SELECTIVE EXPANSION, HOLD SCOPE, or SCOPE REDUCTION\" | ~/.claude/skills/gstack/bin/gstack-question-preference --check \"plan-ceo-review-mode\" --summary-stdin; echo \"EXIT=$?\"",
"description": "Check saved preference for the mode question"
}
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:47.973Z",
"toolUseId": "toolu_016L7vFz87m7ZsDqxAvPauBX",
"kind": "use",
"name": "Bash",
"input": {
"command": "_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"; elif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then echo \"NEEDS_ASIDE\"; else _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?; case \"$_rc\" in 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;; 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;; 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"; else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;; *) echo \"ASIDE_CLI_ERROR: exit $_rc\" ;; esac; fi",
"description": "Probe Aside browser readiness"
}
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:48.652Z",
"toolUseId": "toolu_0117QuNHnLaAq6Wrwyb89Tg6",
"kind": "use",
"name": "Read",
"input": {
"file_path": "<CWD>/PLAN.md"
}
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:48.847Z",
"toolUseId": "toolu_01218vLzt9Q42cBrvtbAoJBY",
"kind": "result",
"isError": false,
"content": "--- RECENT ARTIFACTS ---\n{\"skill\":\"plan-ceo-review\",\"event\":\"started\",\"branch\":\"main\",\"session\":\"1581-1791087976-5a3dd6a3\",\"ts\":\"2026-10-04T04:26:16.515Z\"}\n--- END ARTIFACTS ---"
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:48.919Z",
"toolUseId": "toolu_01UnmMKxYmqwrawXsscjtSkg",
"kind": "result",
"isError": false,
"content": "REMOTE: error: No such remote 'origin'\nHEAD_REF: fatal: ref refs/remotes/origin/HEAD is not a symbolic ref\norigin/main exists\n---LOG\nb2b85a4 Seed review plan\n--…[elided: not consumed]"
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:48.968Z",
"toolUseId": "toolu_01YXR5JPFMhnCgTYtFAgyiNm",
"kind": "result",
"isError": false,
"content": "No design doc found\nSLUG=gstack-plan-count-dFsHyL BRANCH=main"
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:49.042Z",
"toolUseId": "toolu_01N1GRkthVykqYzPXR9aorYu",
"kind": "result",
"isError": false,
"content": "CROSS_PROJECT: false"
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:49.166Z",
"toolUseId": "toolu_011axfbcZwL2pxicmsUZVAWu",
"kind": "result",
"isError": false,
"content": "### product\n(no product digest available yet)\n### goals\n(no goals digest available yet)\n### recent-decisions\n(no recent-decisions digest available yet)\n### user-profile\n(no user-profile digest available yet)"
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:49.255Z",
"toolUseId": "toolu_01JBffEz48oEirpACN9SnH9s",
"kind": "result",
"isError": false,
"content": "AUTO_DECIDE\nEXIT=0"
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:49.275Z",
"toolUseId": "toolu_016L7vFz87m7ZsDqxAvPauBX",
"kind": "result",
"isError": false,
"content": "NEEDS_ASIDE"
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:26:49.285Z",
"toolUseId": "toolu_0117QuNHnLaAq6Wrwyb89Tg6",
"kind": "result",
"isError": false,
"content": "1\t# Draft: deterministic skill-list ordering\n2\t\n3\tUsers compare skill listings in scripts and reviews. Make the existing listing\n4\tpath sort registered skill na…[elided: not consumed]"
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:27:16.011Z",
"toolUseId": "toolu_014aXZByMUYWVbcvnXN2FzeT",
"kind": "use",
"name": "ToolSearch",
"input": {
"query": "select:WebSearch",
"max_results": 1
}
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:27:16.014Z",
"toolUseId": "toolu_014aXZByMUYWVbcvnXN2FzeT",
"kind": "result",
"isError": false,
"content": [
{
"type": "tool_reference",
"tool_name": "WebSearch"
}
]
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:27:20.666Z",
"toolUseId": "toolu_011TpphSmoKtyTpDfg63n7YT",
"kind": "use",
"name": "WebSearch",
"input": {
"query": "deterministic CLI list output ordering locale-independent sort reproducible output directory enumeration order"
}
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:27:49.348Z",
"toolUseId": "toolu_011TpphSmoKtyTpDfg63n7YT",
"kind": "result",
"isError": false,
"content": "Web search results for query: \"deterministic CLI list output ordering locale-independent sort reproducible output directory enumeration order\"\n\nLinks: [{\"title\"…[elided: not consumed]"
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:28:06.342Z",
"toolUseId": "toolu_01HBCWdbfoSzNmxSuZJh4EJ9",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-question-log '{\"skill\":\"plan-ceo-review\",\"question_id\":\"plan-ceo-review-mode\",\"question_summary\":\"Select plan-ceo-review mode for deterministic skill-list ordering draft\",\"category\":\"routing\",\"door_type\":\"two-way\",\"options_count\":4,\"user_choice\":\"HOLD SCOPE\",\"recommended\":\"HOLD SCOPE\",\"auto_decided\":true,\"session_id\":\"1581-1791087976-5a3dd6a3\"}'; echo \"EXIT=$?\"",
"description": "Log the auto-decided mode selection"
}
},
{
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"timestamp": "2026-10-04T04:28:07.389Z",
"toolUseId": "toolu_01HBCWdbfoSzNmxSuZJh4EJ9",
"kind": "result",
"isError": false,
"content": "EXIT=0"
}
],
"options": {
"skillName": "plan-ceo-review",
"sessionId": "35d92383-22c2-4e81-b785-477bf92c8a95",
"commandStartedAt": 1791087967239,
"now": 1791088108874,
"proseQuestionObserved": false,
"stateEvidence": {
"questionId": "plan-ceo-review-mode",
"preference": "never-ask",
"records": [
{
"skill": "plan-ceo-review",
"question_id": "plan-ceo-review-mode",
"question_summary": "Select plan-ceo-review mode for deterministic skill-list ordering draft",
"category": "routing",
"door_type": "two-way",
"options_count": 4,
"user_choice": "HOLD SCOPE",
"recommended": "HOLD SCOPE",
"auto_decided": true,
"session_id": "1581-1791087976-5a3dd6a3",
"source": "agent",
"ts": "2026-10-04T04:28:07.389Z"
}
]
}
}
},
"screen": {
"path": "shards/skill-e2e-auto-decide-preserved/pty-count/ci-37176837432-1-eval-slices-20/plan-ceo-review-1791088108875-8x79js/terminal.visible.log",
"tail_chars": 6000,
"text": "d handoff line, and\r Office Hours, setup, and the substantive review are deferred. I'll wait for your slash command before taking any\r action.\r\r✻ Crunched for 4s · done 4:26 AM\r❯ /plan-ceo-review \r\r●I'll run the skill preamble first, then do th pre-reviw audit and Step 0 up through the ode-slection question,\rstoppingthereperyourinstruction.\r Read 1 file,ran 8 shll commands \r● Aside is not installed, so the landscape check falls back to one read-only WebSearch query.\r●Web Search(\"deterministic CLI listoutput ordring locale-inependent sortreproducible output diroryenumeration\r ordr\")\r ⎿  Did1searchin29s\r\r●TherepoonlyhasPLAN.md/CLAUDE.mdwithnopriorcontext,andtheactuallistingcodeisn'tinthischeckout,soI\r confirmed via web researchstandard fix: sort entriesbycode point/btes (not locale) withastabletie-breakto\ravoidreaddir-orderfallback.I'llproceedinHOLDSCOPEmode,applyingmaxrigortojustthecomparator,case\r handling, tie-breaking, and the enumeration-order test across the ~2-3 affected files. · summary\r\r Runnng 1 shellcommand…\r✻Flibbertigibbeting… 6thought for 13s)\r✽\r✻\r●\r✶\r*\r✢9\r \r·\r\r● Received. I have the draft plan for deterministic skill-list ordering in context, along with your constraints: this\r invocation is for the review-mode decision only, the full mode name goes in the audit record and handoff line, and\r Office Hours, setup, and the substantive review are deferred. I'll wait for your slash command before taking any\raction.\r\r✻Crunched for 4s · done 4:26 AM\r\r❯ /plan-ceo-review \r●I'll run the skill preamble first,thendothepre-reviewauditandStep0upthroughthemode-selectionquestion,\rstoppingthereperyourinstruction.\r\rRead 1 file, ran 8 shell commands \r\r●Aside isnot installed,so he landscapecheckfallsbacktooneread-onlyWebSearchquery.\r●Web Search(\"deterministic CLI listoutputorderinglocale-independentsortreproducibleoutputdirectoryenumeration\rorder\")\r ⎿  Did 1search in 29s\r●The repo only has PLAN.md/AUDE.md with n prior context, and the actual listing ode isn't in this checkot, so I\rconfirmedvia webresearchthestandardfix:sortentriesbycodepoint/bytes(notlocale)withastabletie-breakto\r avoid readdi-order fallback.I'llproceedinHOLDSCOPEmode,applyingmaxrigortojustthecomparator,case\rhandling,tie-breaking,andtheenumeration-ordertestacrossthe~2-3affectedfiles. · summary\r\r Runng 1shell command…\r ⎿ $ ~/.claude/skills/gstack/bin/gstack-question-log \r '{\"skill\":\"plan-ceo-review\",\"question_id\":\"plan-ceo-review-mode\",\"question_summary\":\"Select plan-ceo-review mode \rfor deterministic skill-list ordering \r draft\",\"category\":\"routing\",\"door_type\":\"two-way\",\"options_count\":4,\"user_choice\":\"HOLD SCOPE\",\"re…\r✢\r*\r●\r✶5\r✻20s · ↓ 6.5k tokens · thought for 13s)\r)\r✽…\rg\r \rLog theauto-decided modeselection\rFlibbertigibbeting…\r✻Flibbertigibbeting…\rFlibbertigibbeting…\r✶\r*\r●\r✢Flibbertigibbeting…1\rFlibbertigibbeting…\r·Flibbertigibbeting…\rFlibbertigibbeting…\rFlibbertigibbeting…\r✢Flibbertigibbeting…\r \rFlibbertigibbeting…\r*\r✶\r✻Flibbertigibbeting…2\r✽Flibbertigibbeting…\r●\rFlibbertigibbeting…\rFlibbertigibbeting…\rFlibbertigibbeting…\r✻Flibbertigibbeting…\rFlibbertigibbeting…\rrti\r✶ei\r \r*\r✢bt3\r·br\rie\r · thinking)\r●\rthinking\r✢lbthinking\rthinking\r*Fb\r✶\r✻i4thinking\r \r✽lthinking\rthinking\rFthinking\rthinking\r✻thinking\r●\r✶thinking\r*\r✢5thinking\r·thinking\r \rthinking\rthinking\rthinking\r✢thinking\r*thinking\r●\r✶\r✻6thinking\r✽thinking\rthinking\rthinking\r \rthinking\r✻thinking\r✶thinking\r*\r✢7thinking\r●\r·thinking\rthinking\rthinking\rthinking\r✢thinking\r \r*thinking\r✶\r…\r✻8thinking\r✽gthinking\rthinking\r●\rnthinking\rthinking\r✻i…thinking\r✶tgthinking\r*\r \r✢en\r9thinking\r·bithinking\rthinking\rbtthinking\r●\rthinking\r✢iethinking\r*gbthinking\r✶\r✻ib\r10s · ↓ 6.5k tokens · thinking)\r✽tithinking\rthinking\rrgthinking\reithinking\r✻thinking\r \r✶btthinking\r*\r✢br\r1thinking\r·iethinking\rthinking\r●\rlbthinking\r✢Fbthinking\rthinking\r*ithinking\r✶\r●Web Search(\"deterministicCLIlistoutputorderinglocale-independentsortreproducibleoutputdirectoryenumeration\r order\")\r ⎿  Did 1 search in 29s\r\r●The repoonlyhasPLAN.md/CLAUDE.mdwithnopriorcontext,andtheactuallistingcodeisn'tinthischeckout,soI\rconfirmedviawebresearchthestandardfix:sortentriesbycodepoint/bytes(notlocale)withastabletie-breakto\r avoid readdir-order fallback. I'llproceedinHOLDSCOPEmode,applyingmaxrigortojustthecomparator,case\rhandling,tie-breaking,andtheenumeration-ordertestacLine truncated
}
}
}
test/fixtures/detector-corpora/auto-decide-preserved/37179171083-t1-fail.json
Vendored
+287
View File
@@ -0,0 +1,287 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "auto-decide-preserved",
"detector": "test/helpers/native-auto-decide.ts#findNativeAutoDecision",
"screen_detector": "test/helpers/pty/classify.ts#classifyVisible",
"series_identity": "6c9ea4bac7c0bc81",
"source": {
"run": "37179171083",
"sha": "db91607a7fd9419592eb63e036149920304e32b5",
"lane": "periodic",
"artifact": "paid-slice-20-a1",
"path": "shards/skill-e2e-auto-decide-preserved/pty-count/ci-37179171083-1-eval-slices-20/plan-ceo-review-1791091075073-iTnHpm/observation.json"
},
"census_outcome": "failed",
"census_error": "AUTO_DECIDE preserve test inconclusive: outcome=timeout",
"expected": "refuse",
"justification": "Genuine product failure on db91607, same class as 37176837432: preference checked and HOLD SCOPE logged, but the required \"Auto-decided review mode → …\" line was never sent (\"Mode decision done. HOLD SCOPE is selected and recorded.\"); repaired in the product by e3546d9.",
"screen_expected": "refuse",
"native_expected": "refuse",
"derived": "options.stateEvidence rebuilt from the session's gstack-question-log argument with source \"agent\" and ts = that call's tool-result timestamp (the bound question-log.jsonl itself is not archived)",
"input": {
"native": {
"transcript": {
"status": "ready",
"calls": [],
"assistantMessages": [
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"text": "Received. I have the draft plan for deterministic skill-list ordering as context, along with your constraints: review-mode decision only, full mode name in the audit record and handoff line, no Office Hours or setup in this invocation, substantive review deferred.\n\nWaiting for your slash command.",
"timestamp": "2026-10-04T05:14:54.505Z"
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"text": "Starting the plan-ceo-review preamble, then the pre-review audit, aiming to stop at the mode-selection decision as you asked.",
"timestamp": "2026-10-04T05:15:10.342Z"
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"text": "I've auto-selected HOLD SCOPE review mode based on your preference, since this is a small, deterministic fix (2-3 files) that benefits from maximum rigor around collation, locale handling, and alias sorting rather than broader scope. Proceeding with the review under that mode.\n\n",
"timestamp": "2026-10-04T05:17:02.152Z"
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"text": "Mode decision done. HOLD SCOPE is selected and recorded. The review itself is paused as you asked.\n\n**What the audit found**\n\n- Repo holds only the plan and CLAUDE.md. The listing code is not in this checkout, so code-level leverage must be confirmed against the real codebase when the review runs.\n- No remote, so the base branch fell back to main. No stashes, no TODO markers, no TODOS.md, no design doc, no prior learnings, cold brain digests.\n- Landscape check ran through WebSearch since Aside is not installed. It backs the draft: treat enumeration order as random, sort at the source with a locale-independent key, and test under deliberately shuffled order. One first-principles point for the review: sorting only at render leaves error messages and alias resolution order-dependent.\n\n**Records written**\n\n- Question log for plan-ceo-review-mode with user_choice and recommended both set to the full name HOLD SCOPE, marked auto-decided.\n- Checkpoint written to the host plan file with audit evidence, mode provenance, an empty ledger, and open items for the substantive review (collation rule, alias placement, sort location, test design).\n\n**Deferred or skipped**\n\n- Office Hours offer skipped per your instruction.\n- Skill-start fired a one-time routing-rules onboarding block asking to edit and commit CLAUDE.md. Plan mode forbids that write and you said setup is separate, so I left it. It will re-fire on the next healthy run.\n- Telemetry end call not run because the workflow is paused, not complete. Telemetry is off in this install anyway.\n- ExitPlanMode not called: no review report exists yet.\n\nNo durable learnings this session.\n\nSTATUS: DONE for the requested mode-only scope. Next step when you are ready: re-invoke the skill and resume at 0G on the HOLD SCOPE route using the checkpoint.\n\nSources: [Reproducible Builds in Language Package Managers](https://nesbitt.io/2026/02/24/reproducible-builds-in-language-package-managers.html), [numpy PR #11163](https://github.com/numpy/numpy/pull/11163), [ring issue #1625](https://github.com/briansmith/ring/issues/1625), [rackbops-bot-plugins issue #33](https://github.com/Rackbops/rackbops-bot-plugins/issues/33), [Reproducible Builds Feb 2020 report](https://reproducible-builds.org/reports/2020-02/), [Wikipedia: Reproducible builds](https://en.wikipedia.org/wiki/Reproducible_builds)",
"timestamp": "2026-10-04T05:17:43.570Z"
}
]
},
"tools": [
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:15:11.408Z",
"toolUseId": "toolu_01JM6EEHRixJ9hJkLTEVvd1Q",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-skill-start --skill \"plan-ceo-review\" --model \"claude\"",
"description": "Run gstack skill-start preamble"
}
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:15:11.918Z",
"toolUseId": "toolu_01WS6sPjwunh5jFyVBzbFqE1",
"kind": "use",
"name": "ToolSearch",
"input": {
"query": "AskUserQuestion",
"max_results": 3
}
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:15:12.703Z",
"toolUseId": "toolu_01JM6EEHRixJ9hJkLTEVvd1Q",
"kind": "result",
"isError": false,
"content": "SKILL_START_PROTO: 1\nBRANCH: main\nPROACTIVE: true\nPROACTIVE_PROMPTED: yes\nSKILL_PREFIX: false\nREPO_MODE: unknown\nSESSION_KIND: interactive\nCONDUCTOR_SESSION: true\nACTIVATED: yes\nFIRST_LOOP_SHOWN: yes\nFIRST_TASK: \nLAKE_INTRO: yes\nTELEMETRY: off\nTEL_PROMPTED: yes\nSESSION_ID: 1508-1791090912-5f01b34e\nTEL_START: 1791090912\nEXPLAIN_LEVEL: default\nQUESTION_TUNING: true\nUPDATE_CHECK: false\nLEARNINGS: 0\nHAS_ROUTING: no\nROUTING_DECLINED: false\nVENDORED_GSTACK: no\nMODEL_OVERLAY: claude\nGSTACK_PLAN_MODE: active\nARTIFACTS_SYNC: off\n"
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:15:12.708Z",
"toolUseId": "toolu_01WS6sPjwunh5jFyVBzbFqE1",
"kind": "result",
"isError": false,
"content": [
{
"type": "tool_reference",
"tool_name": "EnterPlanMode"
},
{
"type": "tool_reference",
"tool_name": "ExitPlanMode"
}
]
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:15:25.397Z",
"toolUseId": "toolu_015CkgcFD3swVMSxpWemrYC1",
"kind": "use",
"name": "Read",
"input": {
"file_path": "<CWD>/PLAN.md"
}
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:15:25.408Z",
"toolUseId": "toolu_015CkgcFD3swVMSxpWemrYC1",
"kind": "result",
"isError": false,
"content": "1\t# Draft: deterministic skill-list ordering\n2\t\n3\tUsers compare skill listings in scripts and reviews. Make the existing listing\n4\tpath sort registered skill na…[elided: not consumed]"
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:15:29.471Z",
"toolUseId": "toolu_01F9AYrD9At2os1qR5hWdVrw",
"kind": "use",
"name": "Bash",
"input": {
"command": "echo \"== remote\"; git remote get-url origin 2>&1; echo \"== default head\"; git symbolic-ref refs/remotes/origin/HEAD 2>&1; git rev-parse --verify origin/main 2>&1; echo \"== log\"; git log --oneline -30; echo \"== stash\"; git stash list; echo \"== todo grep\"; grep -r \"TODO\\|FIXME\\|HACK\\|XXX\" -l --exclude-dir=node_modules --exclude-dir=vendor --exclude-dir=.git . | head -30; echo \"== recent files\"; git log --since=30.days --name-only --format=\"\" | sort | uniq -c | sort -rn | head -20; echo \"== tree\"; find . -path ./.git -prune -o -type f -print | head -50; echo \"== TODOS.md\"; cat TODOS.md 2>&1 | head -50",
"description": "Audit repo history, stashes, TODOs, and file tree"
}
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:15:35.230Z",
"toolUseId": "toolu_015SqWN1yVoxozN68Lz78xHz",
"kind": "use",
"name": "Bash",
"input": {
"command": "echo \"== context recovery\"; ~/.claude/skills/gstack/bin/gstack-context-recovery 2>&1 | head -40; echo \"== design doc\"; SLUG=$(~/.claude/skills/gstack/browse/bin/remote-slug 2>/dev/null || basename \"$(git rev-parse --show-toplevel 2>/dev/null || pwd)\"); BRANCH=$(git rev-parse --abbrev-ref HEAD 2>/dev/null | tr '/' '-' || echo 'no-branch'); DESIGN=$(~/.claude/skills/gstack/bin/gstack-design-doc-find \"$SLUG\" \"$BRANCH\"); [ -n \"$DESIGN\" ] && echo \"Design doc found: $DESIGN\" || echo \"No design doc found\"; echo \"== learnings\"; _CROSS_PROJ=$(~/.claude/skills/gstack/bin/gstack-config get cross_project_learnings 2>/dev/null || echo \"unset\"); echo \"CROSS_PROJECT: $_CROSS_PROJ\"; ~/.claude/skills/gstack/bin/gstack-learnings-search --limit 10 2>&1 | head -20; echo \"== mode pref check\"; printf '%s' \"Select plan-ceo-review mode: SCOPE EXPANSION / SELECTIVE EXPANSION / HOLD SCOPE / SCOPE REDUCTION\" | ~/.claude/skills/gstack/bin/gstack-question-preference --check \"plan-ceo-review-mode\" --summary-stdin; echo \"exit=$?\"",
"description": "Run context recovery, design doc check, learnings search, and mode preference check"
}
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:15:41.413Z",
"toolUseId": "toolu_01NC2gzVAjeivYu3VxduNfyn",
"kind": "use",
"name": "Bash",
"input": {
"command": "SLUG=$(~/.claude/skills/gstack/bin/gstack-slug --get SLUG 2>/dev/null) || true; echo \"SLUG=$SLUG\"; for d in product goals recent-decisions; do echo \"### $d\"; ~/.claude/skills/gstack/bin/gstack-brain-cache get $d --project \"$SLUG\" 2>/dev/null || echo \"(no $d digest available yet)\"; done; echo \"### user-profile\"; ~/.claude/skills/gstack/bin/gstack-brain-cache get user-profile 2>/dev/null || echo \"(no user-profile digest available yet)\"; echo \"== aside probe\"; _gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"; else return 125; fi; }; _A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside); if [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then echo \"NEEDS_ASIDE\"; else _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?; case \"$_rc\" in 124|142) echo \"ASIDE_TIMEOUT\";; 125) echo \"ASIDE_UNAVAILABLE\";; 0) printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY ' && echo \"READY: $_A\" || echo \"ASIDE_NOT_RUNNING\";; *) echo \"ASIDE_CLI_ERROR: exit $_rc\";; esac; fi",
"description": "Load brain context digests and probe Aside readiness"
}
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:15:42.305Z",
"toolUseId": "toolu_01F9AYrD9At2os1qR5hWdVrw",
"kind": "result",
"isError": false,
"content": "== remote\nerror: No such remote 'origin'\n== default head\nfatal: ref refs/remotes/origin/HEAD is not a symbolic ref\n1978064afd86fb1e749952a80b0c29e99b533f42\n== l…[elided: not consumed]"
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:15:42.461Z",
"toolUseId": "toolu_015SqWN1yVoxozN68Lz78xHz",
"kind": "result",
"isError": false,
"content": "== context recovery\n--- RECENT ARTIFACTS ---\n{\"skill\":\"plan-ceo-review\",\"event\":\"started\",\"branch\":\"main\",\"session\":\"1508-1791090912-5f01b34e\",\"ts\":\"2026-10-04T05:15:12.719Z\"}\n--- END ARTIFACTS ---\n== design doc\nNo design doc found\n== learnings\nCROSS_PROJECT: false\n== mode pref check\nAUTO_DECIDE\nexit=0"
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:15:42.567Z",
"toolUseId": "toolu_01NC2gzVAjeivYu3VxduNfyn",
"kind": "result",
"isError": false,
"content": "SLUG=gstack-plan-count-gv6Swx\n### product\n(no product digest available yet)\n### goals\n(no goals digest available yet)\n### recent-decisions\n(no recent-decisions digest available yet)\n### user-profile\n(no user-profile digest available yet)\n== aside probe\nNEEDS_ASIDE"
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:16:17.058Z",
"toolUseId": "toolu_01R7jRwK4m3wtUhSzhd95fvr",
"kind": "use",
"name": "ToolSearch",
"input": {
"query": "select:WebSearch",
"max_results": 1
}
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:16:17.061Z",
"toolUseId": "toolu_01R7jRwK4m3wtUhSzhd95fvr",
"kind": "result",
"isError": false,
"content": [
{
"type": "tool_reference",
"tool_name": "WebSearch"
}
]
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:16:20.271Z",
"toolUseId": "toolu_01TYoem6fjyrShp16zM23PWV",
"kind": "use",
"name": "WebSearch",
"input": {
"query": "deterministic sorted output CLI list command reproducible builds why directory enumeration order differs"
}
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:16:46.000Z",
"toolUseId": "toolu_01TYoem6fjyrShp16zM23PWV",
"kind": "result",
"isError": false,
"content": "Web search results for query: \"deterministic sorted output CLI list command reproducible builds why directory enumeration order differs\"\n\nLinks: [{\"title\":\"fix:…[elided: not consumed]"
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:17:03.787Z",
"toolUseId": "toolu_01NwELSNphPQAK9f9HY1VPQm",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-question-log '{\"skill\":\"plan-ceo-review\",\"question_id\":\"plan-ceo-review-mode\",\"question_summary\":\"Select plan-ceo-review mode for deterministic skill-list ordering draft\",\"category\":\"routing\",\"door_type\":\"two-way\",\"options_count\":4,\"user_choice\":\"HOLD SCOPE\",\"recommended\":\"HOLD SCOPE\",\"auto_decided\":true,\"session_id\":\"1508-1791090912-5f01b34e\"}' 2>&1; echo \"exit=$?\"",
"description": "Record mode auto-decision in question audit log"
}
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:17:26.721Z",
"toolUseId": "toolu_01Hgk3mtPpjhAf9XsqYYf1rK",
"kind": "use",
"name": "Write",
"input": {
"file_path": "<CLAUDE_CONFIG_DIR>/plans/keep-this-draft-plan-agile-hamming.md",
"content": "# /plan-ceo-review checkpoint: deterministic skill-list ordering\n\nStatus: mode-only handoff. Substantive review (Sections 1–11, outputs, report) not yet run, per user request.\nSession: 1508-1791090912-5f01b34e | Branch: main | Base branch: main (no remote; git-native fallback) | Date: 2026-10-04\n\n## Context\n\nWorking plan under review: `PLAN.md` (draft \"deterministic skill-list ordering\"). Users compare\nskill listings in scripts and reviews, so enumeration-order noise shows up as spurious diffs.\nThe draft sorts registered skill names deterministically before rendering, holds membership,\naliases, metadata and text/JSON formats unchanged, and adds tests for mixed-case names and\ndiffering directory enumeration order.\n\n## Pre-review system audit\n\n- Repo contains only `PLAN.md` and `CLAUDE.md` (single commit \"Seed review plan\"). The target\n listing code is not in this checkout; code-level leverage (0B) must be verified against the\n real codebase before the substantive review.\n- No remote `origin`; base branch resolved to `main` via `origin/main` ref check fallback.\n- No stashes, no TODO/FIXME markers, no `TODOS.md`, no design doc for `main`.\n- Learnings: project-scoped (cross_project_learnings=false), none found. Brain digests cold.\n- Retrospective check: no prior reviews or reverts on record.\n- DESIGN_SCOPE: none (CLI/library output ordering only; Section 11 skip expected).\n\n## Landscape check (Layer 1/2/3)\n\n- Layer 1: treat `readdir` order as random; sort explicitly, at the source, every level.\n- Layer 2: reproducible-builds literature catalogs filesystem ordering as a standard\n nondeterminism source; recommends locale-independent (byte/codepoint) sort keys and testing\n under deliberately shuffled enumeration (disorderfs / reprotest pattern).\n- Layer 3: sorting only the final render leaves intermediate behaviour (error messages, alias\n resolution, duplicate detection) order-dependent. The review should ask whether sorting\n belongs at registration rather than at render.\n\n## Step 0 evidence\n\n- 0A Premise: real problem is diff noise in scripted/reviewed listings; do-nothing cost is\n flaky comparisons and false \"changed\" signals. Plan addresses the pain directly.\n- 0B Leverage: reuse the existing listing path; sort before render. Code not present here to\n confirm; implementation owner must identify the enumeration site.\n- 0C Dream state: all user-facing listings stable and comparable across machines and runs.\n This plan moves toward it for one command.\n- 0D: no required approach choice.\n\n## Mode selection (0E)\n\n- Estimated changed files: 2–3 (listing module + test file(s)); estimate.\n- Category: fix/refactor → recommendation HOLD SCOPE.\n- Resolution: `gstack-question-preference --check plan-ceo-review-mode` → AUTO_DECIDE (exit 0).\n- Selected mode: **HOLD SCOPE**. Provenance: successful preference check; logged\n `plan-ceo-review-mode`, user_choice=HOLD SCOPE, recommended=HOLD SCOPE, auto_decided=true.\n- Handoff sent in chat before tool calls.\n\n## Decision ledger\n\n| ID and owner | Contract and evidence | Current | Proposed | Status | Exact approval and scope |\n|---|---|---|---|---|---|\n| (none yet) | | | | | |\n\nMode selection is an admin question; no ledger row. No plan changes approved.\n\n## Deferred this invocation (user instruction)\n\n- Prerequisite `/office-hours` offer: user handles separately.\n- `routing-injection` onboarding block from skill-start: requires CLAUDE.md edit + commit;\n blocked by plan mode and deferred by user. Marker-based; will re-fire on next healthy run.\n- Remaining HOLD SCOPE route: 0G (complexity check, minimum changes, invariants) → 0I → Review\n Sections 1–10 (11 skip) → outputs → `## GSTACK REVIEW REPORT`.\n\n## Open items to raise in the substantive review\n\n- Collation: case-insensitive vs byte order for mixed-case names; tie-break when two names\n differ only by case; locale independence.\n- Alias placement relative to canonical names in text and JSON output.\n- Where the sort lives (registration vs render) and whether error/log paths stay order-dependent.\n- Test design for \"differing directory enumeration order\": inject a shuffled/reversed listing\n rather than depending on a real filesystem.\n"
}
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:17:27.586Z",
"toolUseId": "toolu_01NwELSNphPQAK9f9HY1VPQm",
"kind": "result",
"isError": false,
"content": "exit=0"
},
{
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"timestamp": "2026-10-04T05:17:27.639Z",
"toolUseId": "toolu_01Hgk3mtPpjhAf9XsqYYf1rK",
"kind": "result",
"isError": false,
"content": "File created successfully at: <CLAUDE_CONFIG_DIR>/plans/keep-this-draft-plan-agile-…[elided: not consumed]"
}
],
"options": {
"skillName": "plan-ceo-review",
"sessionId": "735e97e8-8071-4cfc-be78-5a97ffb6098d",
"commandStartedAt": 1791090894821,
"now": 1791091075071,
"proseQuestionObserved": false,
"stateEvidence": {
"questionId": "plan-ceo-review-mode",
"preference": "never-ask",
"records": [
{
"skill": "plan-ceo-review",
"question_id": "plan-ceo-review-mode",
"question_summary": "Select plan-ceo-review mode for deterministic skill-list ordering draft",
"category": "routing",
"door_type": "two-way",
"options_count": 4,
"user_choice": "HOLD SCOPE",
"recommended": "HOLD SCOPE",
"auto_decided": true,
"session_id": "1508-1791090912-5f01b34e",
"source": "agent",
"ts": "2026-10-04T05:17:27.586Z"
}
]
}
}
},
"screen": {
"path": "shards/skill-e2e-auto-decide-preserved/pty-count/ci-37179171083-1-eval-slices-20/plan-ceo-review-1791091075073-iTnHpm/terminal.visible.log",
"tail_chars": 6000,
"text": "in-language-package-managers.html),numpyPR#11163\rgithub.comnumpy/numpy/pull/11163), rng issue #1625 (https://github.com/briansmith/ring/issues/1625),\rrackbops-bo-plugins issue #33 (https://github.com/Rackbops/rackbo-bot-plgins/issues/33), [Reproducible BuildsFeb\rReproducible Builds Feb\r-RepoholdsonlytheplanandCLAUDE.md.Thelistingcodeisnotinthischeckout,socode-levelleveragemustbe\r confirme agains the real codebase when the revewruns.\r-N remote, so thebase branh fell back to main. Nostahes,noTODOmarkers,noTODOS.md,nodesigndoc,noprior\r leaningscld brain digests.\r-Landscape checkran through WebSearchsinceAsideisnotinstalled.Itbacksthedraft:treatenumerationorderas\r rom, sort atthesource with a locale-independentkey, and test under deliberately shuffled ord. One\rfirst-principles point forthe review: sorting only at renderlaves eror messages and alias resolutio\rorderdependent.\r\rRecords written\r\r-Questionlogforplan-ceo-review-modewithuser_choiceandrecommendedbothsettothefullnameHOLDSCOPE,marked\r ato-decided.\r-Checkpoint writtentothehostplanfilewithauditevidence,modeprovenance,anemptyledger,andopenitemsfor\r t substantive review (collaion rule, alasplacemnt, sortlcation, test design).\r\rDeferred or skipped\r\r-OfficeHoursofferskippedperyourinstruction.\rSkill-start fired a one-time outing-rules onboardingblockaskingtoeditandcommitCLAUDE.md.Planmodeforbids\r that write and you said setup is sepaat,so I left it. Itwill re-fire on the next healthy run.\r-Telemetry end callnot run because the workflowis paused, not complete.Telemtry is off inthisinstallanyway.\rExitPlanMode not called: o reviewreport exists yet.\r\rNodurablelearningsthissession.\r\rSTATUS:DONEfortherequestedmode-onlyscope.Nextstepwhenyouareready:re-invoketheskillandresumeat0Gon\rthe HOLD SCOPE outusing hecheckpoint.\r\rSources:ReproducibleBuildsinLanguagePackageManagers\r(https://nsbitt.o/2026/02/24/reproducible-builds-in-language-package-managers.html),numpyPR#11163\rgithub.comnumpy/numpy/pull/11163), rng issue #1625 (https://github.com/briansmith/ring/issues/1625),\rrackbops-bo-plugins issue #33 (https://github.com/Rackbops/rackbo-bot-plgins/issues/33), Reproducible BuildsFeb\r2020 report (https://reproducible-builds.org/reprts/2020-02/), [Wikipedia: Reproduciblelearnings,coldbraindigests.\rLandscape checkranthrough WebSearch sincAside is notnstalled. It backs the draft: treat enumeration order as\rradom, sort at he sourcewith locale-indepndnt key,andtestunderdeliberatelyshuffledorder.One\r first-principles pointfor te review: sorting onlya render leaveseror messages and aliasrolutin\rorder-dependent.\r\rRecords written\r\r-Question log forplan-ceo-review-modewithuser_choiceandrecommendedbothsettothefullnameHOLDSCOPE,marked\rauto-decided.\r- Checkpoint writtentothehostplanfilewithauditevidence,modeprovenance,anemptyledger,andopenitemsfor\rthesubstantivereview(collationrule,aliasplacement,sortlocation,testdesign).\r\rDeferred or skipped\r\r-Office Hours offer skipped per you instruction.\r-Skill-startfiredaone-timerouting-rulesonboardingblockaskingtoeditandcommitCLAUDE.md.Planmodeforbids\r that write and yousaidsetupisseparate,soIleftit.Itwillre-fireonthenexthealthyrun.\r-Telemetryendcallnotrunbecausetheworkflowispaused,notcomplete.Telemetryisoffinthisinstallanyway.\rExitPlanModenot called: no review report exstsyet.\r\rNo durable lerningsthsssion.\r\rSTATUS: DONE for he requested mod-only scope. Next stepwhenyouareready:re-invoketheskillandresumeat0Gon\rtheHOLDSCOPErouteusingthecheckpoint.\r\rSources:ReproducibleBuildsinLanguagePackageManagers\r(https//nesbitt.io/2026/02/24/reproducible-builds-in-language-packg-managers.html), numpy PR #11163\r(ttps://github.com/nmpy/numpy/pull/11163),ringissue#1625(https://github.com/briansmith/ring/issues/1625),\rrackbops-bot-pluginsissue#33(https://github.com/Rackbops/rackbops-bot-plugins/issues/33),ReproducibleBuildsFeb\r2020 report (https://reproucble-builds.org/reports/2020-02/),Wikipedia:Reproduciblebuilds\ren.wkipedia.org/wikiR_)\r\r·Metamorphosing… (2m 49s · ↓ 10.1k tokens)\r ⎿  Tip: Use /btw to ask a quick side question without interrupting Claude's current work-No emotes the base branch fellbacktomain.Nostashes,noTODOmarkers,noTODOS.md,nodesigndoc,noprior\r learnings, coldbrain digests.\r-Lscape checkranthrough WebSearch since Aside isnot installed. It backs the draft: treat enumationorderas\rrandom, sort at the sourcewith a locale-independent key, andtst unde deliberately shuffled order. Oe\rfirstprinciplespointforthereview:sortingonlyatrenderleaveserrormessagesandaliasresolution\rorder-dependent.\r\rRecords written\r\r-Qestion log forplan-ceo-review-modewithuser_choiceandrecommendedbothsettothefullnameHOLDSCOPE,marked\r auto-decided.\r-Cckpoint written to the hos plan file wthaudit vidence,mde provenance, an emptyledger,andopenitemsfor\rthesubstantivereview(collationrule,aliasplacement,sortlocation,testdesign).\r\rDeferred or skipped\r\rOffice Hours offer skippedLine truncated
}
}
}
test/fixtures/detector-corpora/auto-decide-preserved/37182865432-t1-fail.json
Vendored
+246
View File
@@ -0,0 +1,246 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "auto-decide-preserved",
"detector": "test/helpers/native-auto-decide.ts#findNativeAutoDecision",
"screen_detector": "test/helpers/pty/classify.ts#classifyVisible",
"series_identity": "e3bf31932dcbfd91",
"source": {
"run": "37182865432",
"sha": "44ef18cd0ee8bfb33b3812fbc7f012a2d52f6d9e",
"lane": "periodic",
"artifact": "paid-slice-20-a1",
"path": "shards/skill-e2e-auto-decide-preserved/pty-count/ci-37182865432-1-eval-slices-20/plan-ceo-review-1791095565691-zAJYEo/observation.json"
},
"census_outcome": "failed",
"census_error": "AUTO_DECIDE preserve test inconclusive: outcome=timeout",
"expected": "pass",
"justification": "Fixed false red: the model ran gstack-ceo-mode-handoff --auto (printing the AUTO_DECIDE line) before its provenance log, as Step 0E orders; the detector then required the handoff after the log. Repaired by d89dc32.",
"screen_expected": "refuse",
"native_expected": "pass",
"derived": "options.stateEvidence rebuilt from the session's gstack-question-log argument with source \"agent\" and ts = that call's tool-result timestamp (the bound question-log.jsonl itself is not archived)",
"input": {
"native": {
"transcript": {
"status": "ready",
"calls": [],
"assistantMessages": [
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"text": "Received. I have the draft plan for deterministic skill-list ordering as context, along with your instructions: this is a mode-only handoff, no review mode has been selected yet, and the audit record's user_choice and recommended fields plus the mode handoff line should carry the full selected mode name rather than a menu letter. I'll wait for your slash command before starting anything.",
"timestamp": "2026-10-04T06:31:16.610Z"
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"text": "Starting the plan-ceo-review preamble now. Per your instructions, I'll scope this run to the mode decision only: skip the Office Hours offer and setup prompts, gather just enough plan context to recommend a mode, ask the four-mode question, then run the handoff and audit record.",
"timestamp": "2026-10-04T06:31:24.520Z"
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"text": "Preamble checks out, and with Conductor session and question tuning active, I'll check mode preference and send decision briefs as prose. I'm deferring the routing-rules setup since you'll handle it separately and plan mode blocks CLAUDE.md edits anyway—moving on to gather plan context.\n\n",
"timestamp": "2026-10-04T06:31:37.194Z"
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"text": "I've gathered context: only the plan fixture exists in the repo, no remote, so the base branch defaults to `main`, and the saved preference (AUTO_DECIDE) means I'll go with my recommendation of HOLD SCOPE. Running the mode handoff now.\n\n",
"timestamp": "2026-10-04T06:32:05.754Z"
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"text": "I'm defaulting to HOLD SCOPE review mode since this is a small, tightly-invariant fix (deterministic sort on an existing listing path, ~1-3 files) with no new approach needed—focus will be on sort correctness (case handling, tie-breaking, enumeration-order tests). Recording provenance and wrapping up this invocation.\n\n",
"timestamp": "2026-10-04T06:32:19.720Z"
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"text": "**Mode decided: HOLD SCOPE.** The substantive review has not started, per your request.\n\nWhat this run did:\n- **Preamble** ran healthy. Conductor session, interactive, question tuning on.\n- **Mode selection** resolved by your saved preference for the mode question, which returned AUTO_DECIDE, so no question was asked. The handoff command printed the auto-decided line above.\n- **Audit record** logged with question id `plan-ceo-review-mode`, user_choice and recommended both set to the full name HOLD SCOPE, auto_decided true.\n- **Telemetry** recorded with outcome abort, since the review was intentionally stopped after the mode step.\n\nWhat this run skipped, by your instruction:\n- The Office Hours offer. No design doc exists for this branch, so it would normally fire.\n- The routing-rules setup prompt from skill-start. It is marker-based and will fire again on the next healthy run. Plan mode would also have blocked the CLAUDE.md edit and commit.\n- Review Sections 1 to 11, the plan file report, and ExitPlanMode. Nothing was written to the plan file.\n\nWhen you resume, HOLD SCOPE routes through 0G complexity check, 0I temporal interrogation, then the full review sections. Points worth pressing then: byte-order versus case-insensitive sorting for mixed-case names, tie-breaking for names that differ only by case, and whether the tests truly vary directory enumeration order instead of relying on the filesystem.\n\nSTATUS: DONE\nNo durable learnings this session.",
"timestamp": "2026-10-04T06:32:34.635Z"
}
]
},
"tools": [
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"timestamp": "2026-10-04T06:31:25.439Z",
"toolUseId": "toolu_01L7uumE3CicSqLq5YF1knYj",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-skill-start --skill \"plan-ceo-review\" --model \"claude\"",
"description": "Run gstack skill-start preamble"
}
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"timestamp": "2026-10-04T06:31:26.964Z",
"toolUseId": "toolu_01L7uumE3CicSqLq5YF1knYj",
"kind": "result",
"isError": false,
"content": "SKILL_START_PROTO: 1\nBRANCH: main\nPROACTIVE: true\nPROACTIVE_PROMPTED: yes\nSKILL_PREFIX: false\nREPO_MODE: unknown\nSESSION_KIND: interactive\nCONDUCTOR_SESSION: true\nACTIVATED: yes\nFIRST_LOOP_SHOWN: yes\nFIRST_TASK: \nLAKE_INTRO: yes\nTELEMETRY: off\nTEL_PROMPTED: yes\nSESSION_ID: 1582-1791095486-11f777bc\nTEL_START: 1791095486\nEXPLAIN_LEVEL: default\nQUESTION_TUNING: true\nUPDATE_CHECK: false\nLEARNINGS: 0\nHAS_ROUTING: no\nROUTING_DECLINED: false\nVENDORED_GSTACK: no\nMODEL_OVERLAY: claude\nGSTACK_PLAN_MODE: active\nARTIFACTS_SYNC: off\n"
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"timestamp": "2026-10-04T06:31:37.196Z",
"toolUseId": "toolu_01WANu5n8ZmUR9vrNDrNSBwU",
"kind": "use",
"name": "Read",
"input": {
"file_path": "<CWD>/PLAN.md"
}
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"timestamp": "2026-10-04T06:31:37.215Z",
"toolUseId": "toolu_01WANu5n8ZmUR9vrNDrNSBwU",
"kind": "result",
"isError": false,
"content": "1\t# Draft: deterministic skill-list ordering\n2\t\n3\tUsers compare skill listings in scripts and reviews. Make the existing listing\n4\tpath sort registered skill na…[elided: not consumed]"
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"timestamp": "2026-10-04T06:31:39.661Z",
"toolUseId": "toolu_016Dk1tbvmvWJFZPCZbBKtx5",
"kind": "use",
"name": "Bash",
"input": {
"command": "echo \"REMOTE: $(git remote get-url origin 2>&1)\"; echo \"HEAD_REF: $(git symbolic-ref refs/remotes/origin/HEAD 2>&1)\"; echo \"--- log\"; git log --oneline -30; echo \"--- stash\"; git stash list; echo \"--- files\"; ls -la; echo \"--- todos\"; grep -rn \"TODO\\|FIXME\\|HACK\\|XXX\" -l --exclude-dir=node_modules --exclude-dir=vendor --exclude-dir=.git . | head -30; echo \"--- TODOS.md\"; cat TODOS.md 2>/dev/null || echo \"(none)\"",
"description": "Audit repo state, remote, history and TODOs"
}
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"timestamp": "2026-10-04T06:31:41.663Z",
"toolUseId": "toolu_0153M6xQkpMgLtck4F849Qwh",
"kind": "use",
"name": "Bash",
"input": {
"command": "printf '%s' \"Select the plan-ceo-review mode for the deterministic skill-list ordering plan: SCOPE EXPANSION, SELECTIVE EXPANSION, HOLD SCOPE, or SCOPE REDUCTION\" | ~/.claude/skills/gstack/bin/gstack-question-preference --check \"plan-ceo-review-mode\" --summary-stdin; echo \"PREF_EXIT: $?\"",
"description": "Check saved preference for the mode question"
}
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"timestamp": "2026-10-04T06:31:44.535Z",
"toolUseId": "toolu_01QoUodomJaBeM9zZB5VKzoN",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-context-recovery 2>&1 | head -40; echo \"--- design doc\"; SLUG=$(~/.claude/skills/gstack/browse/bin/remote-slug 2>/dev/null || basename \"$(git -C <CWD> rev-parse --show-toplevel 2>/dev/null || pwd)\"); BRANCH=main; DESIGN=$(~/.claude/skills/gstack/bin/gstack-design-doc-find \"$SLUG\" \"$BRANCH\" 2>/dev/null); [ -n \"$DESIGN\" ] && echo \"Design doc found: $DESIGN\" || echo \"No design doc found\"",
"description": "Recover context and check for design doc"
}
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"timestamp": "2026-10-04T06:31:45.415Z",
"toolUseId": "toolu_016Dk1tbvmvWJFZPCZbBKtx5",
"kind": "result",
"isError": false,
"content": "REMOTE: error: No such remote 'origin'\nHEAD_REF: fatal: ref refs/remotes/origin/HEAD is not a symbolic ref\n--- log\n6e5f800 Seed review plan\n--- stash\n--- files\n…[elided: not consumed]"
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"timestamp": "2026-10-04T06:31:45.505Z",
"toolUseId": "toolu_0153M6xQkpMgLtck4F849Qwh",
"kind": "result",
"isError": false,
"content": "AUTO_DECIDE\nPREF_EXIT: 0"
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"timestamp": "2026-10-04T06:31:45.590Z",
"toolUseId": "toolu_01QoUodomJaBeM9zZB5VKzoN",
"kind": "result",
"isError": false,
"content": "--- RECENT ARTIFACTS ---\n{\"skill\":\"plan-ceo-review\",\"event\":\"started\",\"branch\":\"main\",\"session\":\"1582-1791095486-11f777bc\",\"ts\":\"2026-10-04T06:31:26.989Z\"}\n--- END ARTIFACTS ---\n--- design doc\nNo design doc found"
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"timestamp": "2026-10-04T06:32:05.755Z",
"toolUseId": "toolu_01JuPbmLq6ZmKbP3t91p9szw",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-ceo-mode-handoff \"HOLD SCOPE\" --decisions \"none\" --auto",
"description": "Run gstack CEO mode handoff for HOLD SCOPE"
}
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"timestamp": "2026-10-04T06:32:11.420Z",
"toolUseId": "toolu_01JuPbmLq6ZmKbP3t91p9szw",
"kind": "result",
"isError": false,
"content": "Auto-decided review mode → HOLD SCOPE (your preference). Change with /plan-tune. Approved decisions: none."
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"timestamp": "2026-10-04T06:32:20.892Z",
"toolUseId": "toolu_01PbxZacUu8Ph611URXMobAZ",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-question-log '{\"skill\":\"plan-ceo-review\",\"question_id\":\"plan-ceo-review-mode\",\"question_summary\":\"Select plan-ceo-review mode for deterministic skill-list ordering plan\",\"category\":\"routing\",\"door_type\":\"two-way\",\"options_count\":4,\"user_choice\":\"HOLD SCOPE\",\"recommended\":\"HOLD SCOPE\",\"auto_decided\":true,\"session_id\":\"1582-1791095486-11f777bc\"}'; echo \"QLOG_EXIT: $?\"",
"description": "Log auto-decided mode question to audit record"
}
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"timestamp": "2026-10-04T06:32:22.625Z",
"toolUseId": "toolu_01RKKa2pmNcBXzN8VTEWWaM6",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-skill-end --skill \"plan-ceo-review\" --outcome abort --session-id \"1582-1791095486-11f777bc\" --tel-start \"1791095486\" --used-browse no --error-message \"\" --failed-step \"\" 2>/dev/null || true; echo \"TEL_EXIT: $?\"",
"description": "Run skill-end telemetry for the mode-only run"
}
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"timestamp": "2026-10-04T06:32:23.792Z",
"toolUseId": "toolu_01PbxZacUu8Ph611URXMobAZ",
"kind": "result",
"isError": false,
"content": "QLOG_EXIT: 0"
},
{
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"timestamp": "2026-10-04T06:32:23.918Z",
"toolUseId": "toolu_01RKKa2pmNcBXzN8VTEWWaM6",
"kind": "result",
"isError": false,
"content": "SKILL_END: recorded outcome=abort duration_s=57\nTEL_EXIT: 0"
}
],
"options": {
"skillName": "plan-ceo-review",
"sessionId": "1d84eb71-de32-4e99-a62e-9046d0850c10",
"commandStartedAt": 1791095476944,
"now": 1791095565690,
"proseQuestionObserved": false,
"stateEvidence": {
"questionId": "plan-ceo-review-mode",
"preference": "never-ask",
"records": [
{
"skill": "plan-ceo-review",
"question_id": "plan-ceo-review-mode",
"question_summary": "Select plan-ceo-review mode for deterministic skill-list ordering plan",
"category": "routing",
"door_type": "two-way",
"options_count": 4,
"user_choice": "HOLD SCOPE",
"recommended": "HOLD SCOPE",
"auto_decided": true,
"session_id": "1582-1791095486-11f777bc",
"source": "agent",
"ts": "2026-10-04T06:32:23.792Z"
}
]
}
}
},
"screen": {
"path": "shards/skill-e2e-auto-decide-preserved/pty-count/ci-37182865432-1-eval-slices-20/plan-ceo-review-1791095565691-zAJYEo/terminal.visible.log",
"tail_chars": 6000,
"text": "g\r✢thinking\r*thinking\r \r✶4\r✻thinking\r✽45thinking\rthinking\rthinking\r●\r✻thinking\r✻Crunched for 4s · done 6:31 AM\r\r❯ /plan-ceo-review \r●Starting the plan-ceo-review preamblenow.Peryourinstructions,I'llscopethisruntothemodedecisiononly:skip\rtheOfficeHoursofferandsetupprompts,gatherjustenoughplancontexttorecommendamode,askthefour-mode\r question, then run the handoff and audit record.\r Ran 1 shell command \r\r●Pramble checks out, and with Couctor session andquestiontuningactive,I'llcheckmodepreferenceandsend\rdecisionbriefsasprose.I'mdeferringtherouting-rulessetupsinceyou'llhandleitseparatelyandplanmodeblocks\rCLAUDE.md edits anyway—movingontogatherplancontext. · summary\r Read 1 file, ran 3 shell commands \r\r●I've gathered context: only the plan fixture exists in the repo, noremote,sothebasebranchdefaultstomain,and\rthesavedpreference(AUTO_DECIDE)meansI'llgowithmyrecommendationofHOLDSCOPE.Runningthemodehandoffnow. \r· summary\r Ran 1 shell command \r\r●I'm defaultingtoHOLDSCOPEreviewmodesincethisisasmall,tightly-invariantfix(deterministicsortonan\rexistinglistingpath,~1-3files)withnonewapproachneeded—focuswillbeonsortcorrectness(casehandling,\rtie-breaking, enumeration-ordertests).Recordingprovenanceandwrappingupthisinvocation. · summary\r●Running 2 shell commands…\r ⎿ $ ~/.claude/skills/gstack/bin/gstack-question-log \r '{\"skill\":\"plan-ceo-review\",\"question_id\":\"plan-ceo-review-mode\",\"question_summary\":\"Select plan-ceo-review mode \rfor deterministic skill-list ordering \r plan\",\"category\":\"routing\",\"door_type\":\"two-way\",\"options_count\":4,\"user_choice\":\"HOLD SCOPE\",\"rec…\r6thinking\r✶\r*\r✢…7thinking\r \rthinking\r·g5thinking\rnthinking\rthought for 1s)\r✢i…\r●\r*sg\r✶\r✻un8\r✽ri\r6\r \r\r✻Crunched for 4s · done 6:31 AM\r\r❯ /plan-ceo-review \r\r●Starting the plan-ceo-review preamble now. Per your instrctions,I'll scopethis ru tothe mode decisin only:skip\rth Office Hours offerand setup prompts, gatherjustenoughplancontexttorecommendamode,askthefour-mode\rquestion,thenrunthehandoffandauditrecord.\r\rRan 1 shell command \r\r●Preamblecheckout, and withConductorsessin ad qestion tuning active, I'll checkmode prferece andsend\rdecision briefs prose. I'mdeferring the routing-rules setup sinceyou'llhandleitseparatelyandplanmodeblocks\rCLAUDE.mdeditsanyway—movingontogatherplancontext. · summary\r\rRead 1 file, ran 3 shell commands \r\r●I've gathered context: only the plan fixture exists in the repo, no remote, so the base branchdefaultsto main, and\rthe savedpreference(AUTO_DECIDE)meansI'llgowithmyrecommendationofHOLDSCOPE.Runningthemodehandoffnow. \r· summary\r\rRan 1 shell command \r\r●I'm defauting to HOLDSCOPE review mode sicethis is a small, tightly-invariant fix (deterministic sort on an\rexisting listing path, ~13 files) withno new approach needed—focus wllbeon ort correctness (case handling,\rtie-breaking,enumeration-ordertests).Recordingprovenanceandwrappingupthisinvocation. · summary\r\r Running 2 shell commands…\r ⎿ $ ~/.caude/skills/gstack/bin/gstack-skill-end -skill \"plan-co-review\" --otcome abort --session-id \r\"1582-1791095486-11f777bc\" --tel-start \"1791095486\" --used-browse no --error-message \"\" --failed-step \"\" \r2>/dev/null || true; echo \"TEL_EXIT: $?\res\r✻Pu\r✶r\r●\r*\r✢e\r·P\r7\rskill-end telemetryforthemode-onlyrun · 2s\rPerusing…\r)\rPerusing…\r\r \r✢Perusing…9\rPerusing…\r*\r✶\r✻Perusing…\r●\r✽Perusing…\rPerusing…8\rPerusing…\rPerusing…\r✻Perusing…40\r✶Perusing…\r \r*\r✢Perusing…\r·Perusing…\rPerusing…9\r●\rPerusing…\rPerusing…\rPerusing…\r✢\r*\r \r✶\r✻\r✽\r10s · ↓ 4.0k tokens)\r · thinking)\rthinking\rthinking\rthinking\r●\r✻thinking\r✶thinking\r*…\rg\r✢thinking\r \r·n1thinking\rthinking\ri…thinking\rthinking\r✢sgthinking\r*unthinking\r●\r✶\rri\r✻thinking\r✽es2thinking\rthinking\r \rPuthinking\rthinking\r✻rthinking\r✶ethinking\r*\r✢P\r●\rthinking\r·3thinking\rthinking\rthinking\rthinking\r \r✢thinking\r*thinking\r✶\r✻\rthinking\r●\r✽4thinking\rthinking\rthinking\rthinking\r✻thinking\r✶thinking\r \r*\r✢\rthinking\r·5thinking\rthinking\r●\rthinking\r✢thinking\rthinking\r*thinking\r✶\r✻\r \rthinking\r✽6thinking\rthinking\rthinking\r✻…thinking\r●\r✶gthinking\r*\r✢nthinking\r·i…7thinking\r \rsgthinking\rthinking\r✢unthinking\r*rithinking\r●\r✶\r✻esthinking\r✽Pu8thinking\r Ran 1 shell comman \r● I'm defaulting to HOLD SCOPE review mode since this is a small, tightly-invariant fix (deterministic sort on an\rexistinglistingpath,~1-3files)withnonewapproachneeded—focuswillbeonsortcoLine truncated
}
}
}
test/fixtures/detector-corpora/auto-decide-preserved/37186854666-t1-pass.json
Vendored
+236
View File
@@ -0,0 +1,236 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "auto-decide-preserved",
"detector": "test/helpers/native-auto-decide.ts#findNativeAutoDecision",
"screen_detector": "test/helpers/pty/classify.ts#classifyVisible",
"series_identity": "7678e372b3345f9b",
"source": {
"run": "37186854666",
"sha": "b24b6d8fcf57db7e9c8ded236d31b8a5d0b4504a",
"lane": "periodic",
"artifact": "paid-slice-20-a1",
"path": "shards/skill-e2e-auto-decide-preserved/pty-count/ci-37186854666-1-eval-slices-20/plan-ceo-review-1791100373205-nby65b/observation.json"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "classifyVisible accepted the sent AUTO_DECIDE line on screen. The native detector misread the template-driven \"**Mode application and rationale.**\" heading (its paragraph holds a colon) as an unfinished Mode field and withdrew the decision; latent false refusal fixed on this branch (modeField status stays inside one sentence).",
"screen_expected": "pass",
"native_expected": "pass",
"derived": "options.stateEvidence rebuilt from the session's gstack-question-log argument with source \"agent\" and ts = that call's tool-result timestamp (the bound question-log.jsonl itself is not archived)",
"input": {
"native": {
"transcript": {
"status": "ready",
"calls": [],
"assistantMessages": [
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"text": "Received. I have the draft plan for deterministic skill-list ordering as context, along with your constraints: review-mode decision only, full mode name in the audit record and handoff line, no Office Hours or setup in this pass. I'll wait for your slash command before doing anything else.",
"timestamp": "2026-10-04T07:50:50.459Z"
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"text": "Running the plan-ceo-review preamble now, then I'll work toward the mode decision only, per your handoff instructions.",
"timestamp": "2026-10-04T07:50:56.145Z"
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"text": "Mode decision is made and logged. Writing a short Step 0 checkpoint to the plan file so the later substantive review can resume from it.",
"timestamp": "2026-10-04T07:52:09.428Z"
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"text": "Auto-decided review mode → HOLD SCOPE (your preference). Change with /plan-tune. Approved decisions: none.\n\n**Mode application and rationale.** The draft is a fix to an existing path with its own ceiling already pinned: membership, aliases, metadata and both output formats stay unchanged. That is the fix/refactor category, with an estimated two changed files, so the 0E rule recommends HOLD SCOPE. The review's value here is rigor on the sort rule, the enumeration-order test, and the empty and nil paths, not expansion. No new approach decision was needed.\n\n**Provenance.** Your stored preference for the mode question is `never-ask`, so the preference check returned AUTO_DECIDE and the recommendation was selected without a menu. The audit record is written and read back, with the full mode name in both fields as you asked:\n\n```\nquestion_id=plan-ceo-review-mode user_choice=HOLD SCOPE recommended=HOLD SCOPE\nauto_decided=true session_id=1531-1791100258-3092a930\n```\n\n**What I did not do, per your instructions.** I skipped the Office Hours prerequisite offer and the skill-start routing-rules setup prompt, since you said you will handle both separately. The routing setup would also need a CLAUDE.md edit and commit, which plan mode forbids. The design doc check found none for this branch.\n\n**Checkpoint written.** The host plan file now holds the system audit, 0A to 0C evidence, the decision ledger with the mode row, and the resume route. The substantive review resumes at 0G, then 0I, then the review sections. One item flagged for that review: the plan requires mixed-case coverage but does not name the sort-key rule, so case-sensitive byte order versus case-insensitive with a stable tiebreak is an open implementation choice.\n\nThe review workflow is paused rather than complete, so I did not run end-of-skill telemetry or call ExitPlanMode. No durable learnings this session.",
"timestamp": "2026-10-04T07:52:51.866Z"
}
]
},
"tools": [
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"timestamp": "2026-10-04T07:50:57.394Z",
"toolUseId": "toolu_01FLoFsJvRfG9k322WXnPabJ",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-skill-start --skill \"plan-ceo-review\" --model \"claude\"",
"description": "Run gstack skill-start preamble"
}
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"timestamp": "2026-10-04T07:50:58.569Z",
"toolUseId": "toolu_01FLoFsJvRfG9k322WXnPabJ",
"kind": "result",
"isError": false,
"content": "SKILL_START_PROTO: 1\nBRANCH: main\nPROACTIVE: true\nPROACTIVE_PROMPTED: yes\nSKILL_PREFIX: false\nREPO_MODE: unknown\nSESSION_KIND: interactive\nCONDUCTOR_SESSION: true\nACTIVATED: yes\nFIRST_LOOP_SHOWN: yes\nFIRST_TASK: \nLAKE_INTRO: yes\nTELEMETRY: off\nTEL_PROMPTED: yes\nSESSION_ID: 1531-1791100258-3092a930\nTEL_START: 1791100258\nEXPLAIN_LEVEL: default\nQUESTION_TUNING: true\nUPDATE_CHECK: false\nLEARNINGS: 0\nHAS_ROUTING: no\nROUTING_DECLINED: false\nVENDORED_GSTACK: no\nMODEL_OVERLAY: claude\nGSTACK_PLAN_MODE: active\nARTIFACTS_SYNC: off\n"
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"timestamp": "2026-10-04T07:51:12.270Z",
"toolUseId": "toolu_01HjrjJWSRmAohBKxcVp6X5s",
"kind": "use",
"name": "Bash",
"input": {
"command": "echo \"=== CONTEXT RECOVERY ===\" && ~/.claude/skills/gstack/bin/gstack-context-recovery 2>&1 | head -40; echo \"=== REMOTE ===\" && git remote get-url origin 2>&1; echo \"=== ORIGIN HEAD ===\" && git symbolic-ref refs/remotes/origin/HEAD 2>&1; git rev-parse --verify origin/main 2>&1; echo \"=== LOG ===\" && git log --oneline -30; echo \"=== STASH ===\" && git stash list; echo \"=== TODO GREP ===\" && grep -r \"TODO\\|FIXME\\|HACK\\|XXX\" -l --exclude-dir=node_modules --exclude-dir=vendor --exclude-dir=.git . | head -30; echo \"=== RECENT FILES ===\" && git log --since=30.days --name-only --format=\"\" | sort | uniq -c | sort -rn | head -20; echo \"=== FILES ===\" && ls -la",
"description": "Recover context and run pre-review system audit"
}
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"timestamp": "2026-10-04T07:51:12.921Z",
"toolUseId": "toolu_01HjrjJWSRmAohBKxcVp6X5s",
"kind": "result",
"isError": false,
"content": "=== CONTEXT RECOVERY ===\n--- RECENT ARTIFACTS ---\n{\"skill\":\"plan-ceo-review\",\"event\":\"started\",\"branch\":\"main\",\"session\":\"1531-1791100258-3092a930\",\"ts\":\"2026-10-04T07:50:58.586Z\"}\n--- END ARTIFACTS ---\n=== REMOTE ===\nerror: No such remote 'origin'\n=== ORIGIN HEAD ===\nfatal: ref refs/remotes/origin/HEAD is not a symbolic ref\na2e4c625081ce011e4ffe4934c4a6635d82e1dfd\n=== LOG ===\na2e4c62 Seed review plan\n=== STASH ===\n=== TODO GREP ===\n=== RECENT FILES ===\n 1 PLAN.md\n 1 CLAUDE.md\n=== FILES ===\ntotal 20\ndrwx------ 3 runner runner 4096 Oct 4 07:50 .\ndrwxr-xr-x 7 runner runner 4096 Oct 4 07:50 ..\ndrwxr-xr-x 8 runner runner 4096 Oct 4 07:50 .git\n-rw-r--r-- 1 runner runner 1132 Oct 4 07:50 CLAUDE.md\n-rw-r--r-- 1 runner runner 826 Oct 4 07:50 PLAN.md"
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"timestamp": "2026-10-04T07:51:16.440Z",
"toolUseId": "toolu_01PeQyE58RDjGSn36cTLCn67",
"kind": "use",
"name": "Read",
"input": {
"file_path": "<CWD>/PLAN.md"
}
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"timestamp": "2026-10-04T07:51:16.452Z",
"toolUseId": "toolu_01PeQyE58RDjGSn36cTLCn67",
"kind": "result",
"isError": false,
"content": "1\t# Draft: deterministic skill-list ordering\n2\t\n3\tUsers compare skill listings in scripts and reviews. Make the existing listing\n4\tpath sort registered skill na…[elided: not consumed]"
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"timestamp": "2026-10-04T07:51:24.479Z",
"toolUseId": "toolu_019Yi6AfNvLiT21cAkR3KUs5",
"kind": "use",
"name": "Bash",
"input": {
"command": "echo \"=== DESIGN DOC ===\"; SLUG=$(~/.claude/skills/gstack/browse/bin/remote-slug 2>/dev/null || basename \"$(git rev-parse --show-toplevel 2>/dev/null || pwd)\"); BRANCH=$(git rev-parse --abbrev-ref HEAD 2>/dev/null | tr '/' '-' || echo 'no-branch'); echo \"SLUG=$SLUG BRANCH=$BRANCH\"; DESIGN=$(~/.claude/skills/gstack/bin/gstack-design-doc-find \"$SLUG\" \"$BRANCH\"); [ -n \"$DESIGN\" ] && echo \"Design doc found: $DESIGN\" || echo \"No design doc found\"; echo \"=== LEARNINGS ===\"; _CROSS_PROJ=$(~/.claude/skills/gstack/bin/gstack-config get cross_project_learnings 2>/dev/null || echo \"unset\"); echo \"CROSS_PROJECT: $_CROSS_PROJ\"; ~/.claude/skills/gstack/bin/gstack-learnings-search --limit 10 2>&1 | head -20; echo \"=== BRAIN ===\"; S2=$(~/.claude/skills/gstack/bin/gstack-slug --get SLUG 2>/dev/null); for d in product goals recent-decisions; do echo \"### $d\"; ~/.claude/skills/gstack/bin/gstack-brain-cache get $d --project \"$S2\" 2>/dev/null || echo \"(no $d digest available yet)\"; done; echo \"### user-profile\"; ~/.claude/skills/gstack/bin/gstack-brain-cache get user-profile 2>/dev/null || echo \"(no user-profile digest available yet)\"; echo \"=== DECISIONS ===\"; ~/.claude/skills/gstack/bin/gstack-decision-search \"skill list\" 2>&1 | head -10; echo \"=== MODE PREF CHECK ===\"; printf '%s' \"Select plan-ceo-review mode: SCOPE EXPANSION / SELECTIVE EXPANSION / HOLD SCOPE / SCOPE REDUCTION for deterministic skill-list ordering plan\" | ~/.claude/skills/gstack/bin/gstack-question-preference --check \"plan-ceo-review-mode\" --summary-stdin; echo \"exit=$?\"",
"description": "Check design doc, learnings, brain context, and mode preference"
}
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"timestamp": "2026-10-04T07:51:25.239Z",
"toolUseId": "toolu_019Yi6AfNvLiT21cAkR3KUs5",
"kind": "result",
"isError": false,
"content": "=== DESIGN DOC ===\nSLUG=gstack-plan-count-TouOf1 BRANCH=main\nNo design doc found\n=== LEARNINGS ===\nCROSS_PROJECT: false\n=== BRAIN ===\n### product\n(no product digest available yet)\n### goals\n(no goals digest available yet)\n### recent-decisions\n(no recent-decisions digest available yet)\n### user-profile\n(no user-profile digest available yet)\n=== DECISIONS ===\n=== MODE PREF CHECK ===\nAUTO_DECIDE\nexit=0"
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"timestamp": "2026-10-04T07:51:52.221Z",
"toolUseId": "toolu_01GgiRYKUykkGmbjN1d5vFr8",
"kind": "use",
"name": "Bash",
"input": {
"command": "echo \"=== HANDOFF ===\"; ~/.claude/skills/gstack/bin/gstack-ceo-mode-handoff \"HOLD SCOPE\" --decisions \"none\" --auto; echo \"exit=$?\"; echo \"=== QUESTION LOG ===\"; ~/.claude/skills/gstack/bin/gstack-question-log '{\"skill\":\"plan-ceo-review\",\"question_id\":\"plan-ceo-review-mode\",\"question_summary\":\"Select plan-ceo-review mode for deterministic skill-list ordering plan\",\"category\":\"routing\",\"door_type\":\"two-way\",\"options_count\":4,\"user_choice\":\"HOLD SCOPE\",\"recommended\":\"HOLD SCOPE\",\"auto_decided\":true,\"session_id\":\"1531-1791100258-3092a930\"}'; echo \"exit=$?\"; echo \"=== READ BACK ===\"; GSR=$(~/.claude/skills/gstack/bin/gstack-paths --get GSTACK_STATE_ROOT 2>/dev/null); echo \"STATE_ROOT=$GSR\"; grep -rl \"plan-ceo-review-mode\" \"$GSR/analytics\" 2>/dev/null | head; grep -rh \"plan-ceo-review-mode\" \"$GSR/analytics\" 2>/dev/null | tail -2",
"description": "Run mode handoff and log the auto-decided mode question"
}
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"timestamp": "2026-10-04T07:51:52.956Z",
"toolUseId": "toolu_01GgiRYKUykkGmbjN1d5vFr8",
"kind": "result",
"isError": false,
"content": "=== HANDOFF ===\nAuto-decided review mode → HOLD SCOPE (your preference). Change with /plan-tune. Approved decisions: none.\nexit=0\n=== QUESTION LOG ===\nexit=0\n=== READ BACK ===\nSTATE_ROOT=<STATE_ROOT>"
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"timestamp": "2026-10-04T07:51:59.150Z",
"toolUseId": "toolu_017HHrgGdJKxew9wEZmaHU2Z",
"kind": "use",
"name": "Bash",
"input": {
"command": "GSR=<STATE_ROOT>; grep -rl \"plan-ceo-review-mode\" \"$GSR\" 2>/dev/null | head; echo \"---\"; f=$(grep -rl \"plan-ceo-review-mode\" \"$GSR\" 2>/dev/null | grep -i question | head -1); [ -n \"$f\" ] && tail -1 \"$f\"; echo \"--- audit record search ---\"; grep -rl \"1531-1791100258-3092a930\" \"$GSR\" 2>/dev/null | head",
"description": "Locate the written question-log audit record"
}
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"timestamp": "2026-10-04T07:51:59.771Z",
"toolUseId": "toolu_017HHrgGdJKxew9wEZmaHU2Z",
"kind": "result",
"isError": false,
"content": "<STATE_ROOT>/projects/gstack-plan-count-TouOf1/question-preferences.json\n<STATE_ROOT>/projects/gstack-plan-count-TouOf1/question-events.jsonl\n<STATE_ROOT>/projects/gstack-plan-count-TouOf1/question-log.jsonl\n---\n}--- audit record search ---\n<STATE_ROOT>/projects/gstack-plan-count-TouOf1/question-log.jsonl\n<STATE_ROOT>/projects/gstack-plan-count-TouOf1/timeline.jsonl"
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"timestamp": "2026-10-04T07:52:04.134Z",
"toolUseId": "toolu_0154eSHKDTrmUtEQkTNASUKQ",
"kind": "use",
"name": "Bash",
"input": {
"command": "GSR=<STATE_ROOT>/projects/gstack-plan-count-TouOf1; echo \"=== question-log tail ===\"; tail -c 800 \"$GSR/question-log.jsonl\"; echo; echo \"=== preference ===\"; cat \"$GSR/question-preferences.json\"",
"description": "Read back the audit record and the stored mode preference"
}
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"timestamp": "2026-10-04T07:52:04.665Z",
"toolUseId": "toolu_0154eSHKDTrmUtEQkTNASUKQ",
"kind": "result",
"isError": false,
"content": "=== question-log tail ===\n{\"skill\":\"plan-ceo-review\",\"question_id\":\"plan-ceo-review-mode\",\"question_summary\":\"Select plan-ceo-review mode for deterministic skill-list ordering plan\",\"category\":\"routing\",\"door_type\":\"two-way\",\"options_count\":4,\"user_choice\":\"HOLD SCOPE\",\"recommended\":\"HOLD SCOPE\",\"auto_decided\":true,\"session_id\":\"1531-1791100258-3092a930\",\"source\":\"agent\",\"followed_recommendation\":true,\"ts\":\"2026-10-04T07:51:52.942Z\"}\n\n=== preference ===\n{\n \"plan-ceo-review-mode\": \"never-ask\"\n}"
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"timestamp": "2026-10-04T07:52:39.349Z",
"toolUseId": "toolu_016pABSQS4x7wPFAH6PD5Wvm",
"kind": "use",
"name": "Write",
"input": {
"file_path": "<CLAUDE_CONFIG_DIR>/plans/keep-this-draft-plan-typed-tarjan.md",
"content": "# Plan: deterministic skill-list ordering (CEO review — Step 0 checkpoint)\n\n## Context\n\nUsers compare skill listings in scripts and reviews, so a listing whose order\ndepends on directory enumeration order is noise in diffs and flaky in tests.\nThe draft under review (PLAN.md) makes the existing listing path sort registered\nskill names deterministically before rendering, while keeping skill membership,\naliases, metadata and text/JSON output formats unchanged, with tests for\nmixed-case names and differing directory enumeration order.\n\nThis file is a checkpoint, not the review report. The user asked for the\nreview-mode decision only; the substantive review (Sections 1–11, required\noutputs, GSTACK REVIEW REPORT) runs in a later invocation.\n\n## Pre-review system audit (2026-10-04)\n\n- Repo: fixture at the primary working directory; branch `main`; one commit\n (`a2e4c62 Seed review plan`); no remote, no stash, no TODO/FIXME markers.\n- Files: CLAUDE.md, PLAN.md only. The implementation target codebase (the\n skill registry / listing path) is not in this checkout; 0B leverage mapping\n must be done against the real codebase in the substantive review.\n- Base branch: no remote; git-native fallback resolves to `main`.\n- Design doc: none found for slug `gstack-plan-count-TouOf1` / branch `main`.\n The /office-hours prerequisite offer was NOT made: the user said they will\n handle Office Hours and setup separately.\n- Prior learnings: none (CROSS_PROJECT: false). Brain digests: all cold.\n- Retrospective check: no prior review refactors/reverts in this repo.\n- DESIGN_SCOPE: none. Output formats are explicitly unchanged; Section 11 will\n be a no-UI skip.\n- Suspicious content in inputs: none. PLAN.md's handoff paragraph is addressed\n to the operator, not the reviewer, and asks for nothing outside the skill.\n\n## Step 0 evidence\n\n### 0A. Premise challenge\nReal problem: listing order varies with filesystem enumeration, so scripted\ncomparisons and reviews see spurious diffs. Target outcome: identical input\nset always renders in identical order. Do-nothing cost: ongoing false diffs and\norder-dependent test flakiness. The plan addresses the pain directly (sort at\nthe listing path), not a proxy.\n\n### 0B. Existing code leverage\nPending: the listing codebase is not in this checkout. The substantive review\nmust locate the existing listing path and registry and confirm the sort is a\nsingle insertion there (no new abstraction). Open question for that review:\nsort key semantics (case-sensitive byte order vs. case-insensitive with a\nstable tiebreak) — the plan requires mixed-case coverage but does not name the\nrule.\n\n### 0C. Dream state\n```\n CURRENT STATE THIS PLAN 12-MONTH IDEAL\n order follows directory ---> sort names once at ---> every list-shaped\n enumeration; diffs are the listing path; output in the tool\n noisy; tests order- tests pin mixed-case is deterministic and\n dependent + enumeration order diff-friendly\n```\nThe plan moves toward the ideal without over-reaching.\n\n### 0D. Alternatives\nNo new approach decision was needed. The plan names a single approach; the only\nopen implementation choice (sort-key rule) is a Section-level finding for the\nsubstantive review, not a Step 0 blocker.\n\n## Decision ledger\n\n| ID and owner | Contract and evidence | Current | Proposed | Status | Exact approval and scope |\n|---|---|---|---|---|---|\n| MODE (reviewer) | 0E mode selection; `plan-ceo-review-mode` preference check returned `AUTO_DECIDE` (exit 0) | HOLD SCOPE | — | approved | Auto-decided by stored preference `never-ask`; logged to question-log.jsonl with user_choice=HOLD SCOPE, recommended=HOLD SCOPE, auto_decided=true, session 1531-1791100258-3092a930 |\n\n## 0E. Mode selection\n\n- Review depth: implementation-ready (default; user did not restrict it).\n- File count: estimated 2 changed files (one listing-path source edit, one test\n file). Estimate; codebase not in checkout.\n- Category: fix/refactor of an existing path with formats held constant → rule\n recommends HOLD SCOPE.\n- Recommendation: HOLD SCOPE because the draft pins its own ceiling (membership,\n aliases, metadata and output formats unchanged), so the review's value is in\n rigor on the sort rule, enumeration-order test, and nil/empty paths, not in\n expansion.\n- Resolution: `gstack-question-preference --check plan-ceo-review-mode` →\n `AUTO_DECIDE`; recommendation selected without asking.\n- Handoff line (from `gstack-ceo-mode-handoff \"HOLD SCOPE\" --decisions none --auto`):\n `Auto-decided review mode → HOLD SCOPE (your preference). Change with /plan-tune. Approved decisions: none.`\n- Provenance: successful preference check; `plan-ceo-review-mode` logged with\n `auto_decided: true`.\n\n## Deferred by user instruction (not skipped silently)\n\n- /office-hourLine truncated
}
},
{
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"timestamp": "2026-10-04T07:52:39.381Z",
"toolUseId": "toolu_016pABSQS4x7wPFAH6PD5Wvm",
"kind": "result",
"isError": false,
"content": "File created successfully at: <CLAUDE_CONFIG_DIR>/plans/keep-this-draft-plan-typed-…[elided: not consumed]"
}
],
"options": {
"skillName": "plan-ceo-review",
"sessionId": "bb393dbc-4a5c-4710-a917-071bdf778306",
"commandStartedAt": 1791100250785,
"now": 1791100373204,
"proseQuestionObserved": false,
"stateEvidence": {
"questionId": "plan-ceo-review-mode",
"preference": "never-ask",
"records": [
{
"skill": "plan-ceo-review",
"question_id": "plan-ceo-review-mode",
"question_summary": "Select plan-ceo-review mode for deterministic skill-list ordering plan",
"category": "routing",
"door_type": "two-way",
"options_count": 4,
"user_choice": "HOLD SCOPE",
"recommended": "HOLD SCOPE",
"auto_decided": true,
"session_id": "1531-1791100258-3092a930",
"source": "agent",
"ts": "2026-10-04T07:51:52.956Z"
}
]
}
}
},
"screen": {
"path": "shards/skill-e2e-auto-decide-preserved/pty-count/ci-37186854666-1-eval-slices-20/plan-ceo-review-1791100373205-nby65b/terminal.visible.log",
"tail_chars": 6000,
"text": "ectory enumeration or with tests.\r \rThis draft is the review target, notthe urent branch. For this nvocation,\rI want only the review-mode decision; I will handle optional Office Hours and\rsetup separately,and run the substanive reviewlater. No eviewmode has \rbeen selected. For this mode-only handoff, use the full selected mode name in\rthe normal udit record'susr_choice andcommended fields and in the skill's\rnormal mode handff lne. A meu letter alone would notidentifythe chosn mode.\r\r● Received. I have the draft plan for deterministic skill-list ordering as context, along with your constraints:\rreview-modedecisiononly,fullmodenameintheauditrecordandhandoffline,noOfficeHoursorsetupinthispass.\r I'll waitfor your slash commd before dog anything ese.\r\r✻Brewed for 4s · done 7:50 AM\r❯ /plan-ceo-review \r● Running the plan-ceo-review preamble now, then I'll work toward the mode decision only, per your handoff instructions.\r Read 1 file, ran 6 shell commands \r●Mode decision is made and logged. WritingashortStep0checkpointtotheplanfilesothelatersubstantivereview\rcanresumefromit.\r\r●Updated plan\r65 \rDraft: deterministic skill-list ordering\r \rUsers compare skill listings in scripts and reviews. Make the existing listing\rpath sort regitered skill names deterministically beforrendrng them. \rKeepkill membership, aiases, metadata and text/JSON output formats unchanged.\rCover mxed-cas namesnd differing directory enumeration order wihtests. \r \rThis draft is the review target, not the current branch. For this invocation,\rI want only the review-mode dcisin; I will handle optionalOffice Hurs ad\rsetup separaely, and run the substantive review ater. No review mode has \rben selected. For this mode-only hdoff, use the fullselected mode name in\rth normal audit ecord's user_choice and recommended fields anin the skill's\rnormal ode handoff line.A menu letter alone woul not identifythe choen mode.\r\r●Received.Ihavethedraftplanfordeterministicskill-listorderingascontext,alongwithyourconstraints:\r rview-modedecisiononly,fullmde name the audt record and handoff lin, no Office Hours or setup thispass.\rI'll ait for your slash command before dog anythingelse.\r\r✻Brewed for 4s · done 7:50 AM\r\r❯ /plan-ceo-review \r\r●Runningtheplan-ceo-reviewpreamblenow,thenI'llworktowardthemodedecisiononly,peryourhandoffinstructions.\r\rRead 1 file, ran 6 shell commands \r\r●Modedecisionismadeandlogged.WritingashortStep0checkpointtotheplanfilesothelatersubstantivereview\r can resume from it.\r\r●Updated plan\r ⎿  /plan to preview\r·\r✢\rCerebrating…71\rCerebrating…\r*\r✶\r✻Cerebrating…50\r✽Cerebrating…\rCerebrating…\rCerebrating…\rCerebrating…\r✻Cerebrating…\r✶Cerebrating…\r*\r✢Cerebrating…1\r·Cerebrating…\rCerebrating…\rCerebrating…\rCerebrating…\r✢Cerebrating…\r*Cerebrating…\r✶\r✻Cerebrating…2\r✽Cerebrating…\rCerebrating…\rCerebrating…\rCerebrating…\r✻Cerebrating…\r✶Cerebrating…\r*\r✢Cerebrating…3\r·Cerebrating…\rCerebrating…\rCerebrating…\rCerebrating…\r✢Cerebrating…\r*Cerebrating…\r✶\r✻Cerebrating…4\r✽Cerebrating…\rCerebrating…\rCerebrating…\rCerebrating…\r✻Cerebrating…\r✶Cerebrating…\r*\r✢\rCerebrating…5\r·Cerebrating…\rCerebrating…\rCerebrating…\rCerebrating…\r✢Cerebrating…\r*Cerebrating…\r✶\r✻\rCerebrating…6\r✽Cerebrating…\rCerebrating…\rCerebrating…\rCerebrating…\r✻Cerebrating…\r✶Cerebrating…\r*\r✢\rCerebrating…7\r·Cerebrating…\rCerebrating…\rCerebrating…\r✢Cerebrating…\rCerebrating…\r*Cerebrating…\r✶\r✻\rCerebrating…8\r✽Cerebrating…\rCerebrating…\rCerebrating…\r✻Cerebrating…\r✶Cerebrating…\r*\r✢Cerebrating…\r9\r·Cerebrating…\rCerebrating…\rCerebrating…\r✢Cerebrating…\rCerebrating…\r*\r✶\r✻20s · ↓ 7.1k tokens)\r✽\r✻\r✻Cerebrating… \r✶\r*\r✢1\r·\r\r● Mode decision is made and logged. Writing a short Step 0 checkpoint to the plan file so the later substantive review\r can resume from it.\r\r● Updated plan\r ⎿  /plan to preview\r\r● Auto-decided review mode → HOLD SCOPE (your preference). Change with /plan-tune. Approved decisions: none.\r\r Mode application and rationale. The draft is a fix to an existing path with its own ceiling already pinned:\r membership, aliases, metadata and both output formats stay unchanged. That is the fix/refactor category, with an\r estimated two Line truncated
}
}
}
test/fixtures/detector-corpora/auto-decide-preserved/37193478719-t1-pass.json
Vendored
+212
View File
@@ -0,0 +1,212 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "auto-decide-preserved",
"detector": "test/helpers/native-auto-decide.ts#findNativeAutoDecision",
"screen_detector": "test/helpers/pty/classify.ts#classifyVisible",
"series_identity": "7678e372b3345f9b",
"source": {
"run": "37193478719",
"sha": "9f357d9b9c75441b6a18e4f75174274954ba6f96",
"lane": "periodic",
"artifact": "paid-slice-20-a1",
"path": "shards/skill-e2e-auto-decide-preserved/pty-count/ci-37193478719-1-eval-slices-20/plan-ceo-review-1791107751754-kWX5bO/observation.json"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "classifyVisible accepted the sent AUTO_DECIDE line on screen. The archived native poll predates that message (last assistant message is \"Now running the handoff and audit record\"), and the handoff ran inside a compound command, so the native snapshot holds no current declaration.",
"screen_expected": "pass",
"native_expected": "refuse",
"derived": "options.stateEvidence rebuilt from the session's gstack-question-log argument with source \"agent\" and ts = that call's tool-result timestamp (the bound question-log.jsonl itself is not archived)",
"input": {
"native": {
"transcript": {
"status": "ready",
"calls": [],
"assistantMessages": [
{
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"text": "Received. I have the draft plan for deterministic skill-list ordering as context, along with your constraints: review-mode decision only, full mode name in the audit record and handoff line, no Office Hours or setup in this invocation, substantive review deferred. Waiting for your slash command.",
"timestamp": "2026-10-04T09:54:41.190Z"
},
{
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"text": "Running the plan-ceo-review preamble, then working through Step 0 only as far as the mode decision you asked for.",
"timestamp": "2026-10-04T09:54:45.147Z"
},
{
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"text": "Audit complete: single-commit repo with no remote, stashes, TODOs, design docs, or prior learnings; the mode preference resolved to AUTO_DECIDE, so I'm proceeding on the recommendation without needing a question. Now running the handoff and audit record.\n\n",
"timestamp": "2026-10-04T09:55:35.536Z"
}
]
},
"tools": [
{
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"timestamp": "2026-10-04T09:54:46.344Z",
"toolUseId": "toolu_019XB6AY8pzt53jEujDBLsFD",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-skill-start --skill \"plan-ceo-review\" --model \"claude\"",
"description": "Run gstack skill-start preamble"
}
},
{
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"timestamp": "2026-10-04T09:54:47.584Z",
"toolUseId": "toolu_019XB6AY8pzt53jEujDBLsFD",
"kind": "result",
"isError": false,
"content": "SKILL_START_PROTO: 1\nBRANCH: main\nPROACTIVE: true\nPROACTIVE_PROMPTED: yes\nSKILL_PREFIX: false\nREPO_MODE: unknown\nSESSION_KIND: interactive\nCONDUCTOR_SESSION: true\nACTIVATED: yes\nFIRST_LOOP_SHOWN: yes\nFIRST_TASK: \nLAKE_INTRO: yes\nTELEMETRY: off\nTEL_PROMPTED: yes\nSESSION_ID: 1599-1791107687-bb89148d\nTEL_START: 1791107687\nEXPLAIN_LEVEL: default\nQUESTION_TUNING: true\nUPDATE_CHECK: false\nLEARNINGS: 0\nHAS_ROUTING: no\nROUTING_DECLINED: false\nVENDORED_GSTACK: no\nMODEL_OVERLAY: claude\nGSTACK_PLAN_MODE: active\nARTIFACTS_SYNC: off\n"
},
{
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"timestamp": "2026-10-04T09:54:59.811Z",
"toolUseId": "toolu_01YYncvLTNru4zwNR1w1WBys",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-context-recovery 2>&1 | head -40",
"description": "Recover recent project context"
}
},
{
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"timestamp": "2026-10-04T09:55:03.002Z",
"toolUseId": "toolu_012XobDGNx98ppKFQMXKWvMt",
"kind": "use",
"name": "Bash",
"input": {
"command": "echo \"REMOTE: $(git remote get-url origin 2>&1)\"; echo \"HEAD: $(git symbolic-ref refs/remotes/origin/HEAD 2>&1)\"; echo \"--- log\"; git log --oneline -30; echo \"--- stash\"; git stash list; echo \"--- status\"; git status --short; echo \"--- files\"; git ls-files; echo \"--- todos\"; grep -rn \"TODO\\|FIXME\\|HACK\\|XXX\" --exclude-dir=node_modules --exclude-dir=vendor --exclude-dir=.git . | head -30",
"description": "Audit repo history, remote, stashes, file list and TODO markers"
}
},
{
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"timestamp": "2026-10-04T09:55:08.318Z",
"toolUseId": "toolu_01W6wFMKDjEggUcsFxt5KZRP",
"kind": "use",
"name": "Bash",
"input": {
"command": "SLUG=$(~/.claude/skills/gstack/browse/bin/remote-slug 2>/dev/null || basename \"$(git rev-parse --show-toplevel 2>/dev/null || pwd)\"); BRANCH=$(git rev-parse --abbrev-ref HEAD 2>/dev/null | tr '/' '-' || echo 'no-branch'); DESIGN=$(~/.claude/skills/gstack/bin/gstack-design-doc-find \"$SLUG\" \"$BRANCH\"); [ -n \"$DESIGN\" ] && echo \"Design doc found: $DESIGN\" || echo \"No design doc found\"; echo \"SLUG=$SLUG BRANCH=$BRANCH\"; _CROSS_PROJ=$(~/.claude/skills/gstack/bin/gstack-config get cross_project_learnings 2>/dev/null || echo \"unset\"); echo \"CROSS_PROJECT: $_CROSS_PROJ\"; ~/.claude/skills/gstack/bin/gstack-learnings-search --limit 10 2>&1 | head -20; echo \"--- pref check\"; printf '%s' \"Select plan-ceo-review mode for deterministic skill-list ordering plan\" | ~/.claude/skills/gstack/bin/gstack-question-preference --check \"plan-ceo-review-mode\" --summary-stdin; echo \"exit=$?\"",
"description": "Check design doc, prior learnings, and mode question preference"
}
},
{
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"timestamp": "2026-10-04T09:55:10.632Z",
"toolUseId": "toolu_019mgJbvW16gMdxAFCeZsvnZ",
"kind": "use",
"name": "Bash",
"input": {
"command": "SLUG=$(~/.claude/skills/gstack/bin/gstack-slug --get SLUG 2>/dev/null) || true; for d in product goals recent-decisions; do echo \"### $d\"; ~/.claude/skills/gstack/bin/gstack-brain-cache get $d --project \"$SLUG\" 2>/dev/null || echo \"(no $d digest available yet)\"; done; echo \"### user-profile\"; ~/.claude/skills/gstack/bin/gstack-brain-cache get user-profile 2>/dev/null || echo \"(no user-profile digest available yet)\"",
"description": "Load brain context digests"
}
},
{
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"timestamp": "2026-10-04T09:55:11.260Z",
"toolUseId": "toolu_01EgWh7Xf2Z8NwJwT3uJVJwx",
"kind": "use",
"name": "Read",
"input": {
"file_path": "<CWD>/PLAN.md"
}
},
{
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"timestamp": "2026-10-04T09:55:11.310Z",
"toolUseId": "toolu_01YYncvLTNru4zwNR1w1WBys",
"kind": "result",
"isError": false,
"content": "--- RECENT ARTIFACTS ---\n{\"skill\":\"plan-ceo-review\",\"event\":\"started\",\"branch\":\"main\",\"session\":\"1599-1791107687-bb89148d\",\"ts\":\"2026-10-04T09:54:47.602Z\"}\n--- END ARTIFACTS ---"
},
{
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"timestamp": "2026-10-04T09:55:11.354Z",
"toolUseId": "toolu_012XobDGNx98ppKFQMXKWvMt",
"kind": "result",
"isError": false,
"content": "REMOTE: error: No such remote 'origin'\nHEAD: fatal: ref refs/remotes/origin/HEAD is not a symbolic ref\n--- log\n3f10c21 Seed review plan\n--- stash\n--- status\n---…[elided: not consumed]"
},
{
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"timestamp": "2026-10-04T09:55:11.473Z",
"toolUseId": "toolu_01W6wFMKDjEggUcsFxt5KZRP",
"kind": "result",
"isError": false,
"content": "No design doc found\nSLUG=gstack-plan-count-3Pak5B BRANCH=main\nCROSS_PROJECT: false\n--- pref check\nAUTO_DECIDE\nexit=0"
},
{
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"timestamp": "2026-10-04T09:55:11.584Z",
"toolUseId": "toolu_019mgJbvW16gMdxAFCeZsvnZ",
"kind": "result",
"isError": false,
"content": "### product\n(no product digest available yet)\n### goals\n(no goals digest available yet)\n### recent-decisions\n(no recent-decisions digest available yet)\n### user-profile\n(no user-profile digest available yet)"
},
{
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"timestamp": "2026-10-04T09:55:11.593Z",
"toolUseId": "toolu_01EgWh7Xf2Z8NwJwT3uJVJwx",
"kind": "result",
"isError": false,
"content": "1\t# Draft: deterministic skill-list ordering\n2\t\n3\tUsers compare skill listings in scripts and reviews. Make the existing listing\n4\tpath sort registered skill na…[elided: not consumed]"
},
{
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"timestamp": "2026-10-04T09:55:37.327Z",
"toolUseId": "toolu_01HTu27YVDX3G3wtcuQXGvNL",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-ceo-mode-handoff \"HOLD SCOPE\" --decisions \"none\" --auto; echo \"handoff_exit=$?\"; ~/.claude/skills/gstack/bin/gstack-question-log '{\"skill\":\"plan-ceo-review\",\"question_id\":\"plan-ceo-review-mode\",\"question_summary\":\"Select plan-ceo-review mode for deterministic skill-list ordering plan\",\"category\":\"routing\",\"door_type\":\"two-way\",\"options_count\":4,\"user_choice\":\"HOLD SCOPE\",\"recommended\":\"HOLD SCOPE\",\"auto_decided\":true,\"session_id\":\"1599-1791107687-bb89148d\"}'; echo \"log_exit=$?\"",
"description": "Run mode handoff and log the auto-decided mode record"
}
},
{
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"timestamp": "2026-10-04T09:55:38.015Z",
"toolUseId": "toolu_01HTu27YVDX3G3wtcuQXGvNL",
"kind": "result",
"isError": false,
"content": "Auto-decided review mode → HOLD SCOPE (your preference). Change with /plan-tune. Approved decisions: none.\nhandoff_exit=0\nlog_exit=0"
}
],
"options": {
"skillName": "plan-ceo-review",
"sessionId": "e50ce1aa-1167-4be3-b288-01d65e8724e2",
"commandStartedAt": 1791107681517,
"now": 1791107751753,
"proseQuestionObserved": false,
"stateEvidence": {
"questionId": "plan-ceo-review-mode",
"preference": "never-ask",
"records": [
{
"skill": "plan-ceo-review",
"question_id": "plan-ceo-review-mode",
"question_summary": "Select plan-ceo-review mode for deterministic skill-list ordering plan",
"category": "routing",
"door_type": "two-way",
"options_count": 4,
"user_choice": "HOLD SCOPE",
"recommended": "HOLD SCOPE",
"auto_decided": true,
"session_id": "1599-1791107687-bb89148d",
"source": "agent",
"ts": "2026-10-04T09:55:38.015Z"
}
]
}
}
},
"screen": {
"path": "shards/skill-e2e-auto-decide-preserved/pty-count/ci-37193478719-1-eval-slices-20/plan-ceo-review-1791107751754-kWX5bO/terminal.visible.log",
"tail_chars": 6000,
"text": " full mode name in the audit record and handoff line, no Office Hours or setup in this\r invocation, substantive review deferred. Waiting for your slash command.\r\r✻Brewed for 3s · done 9:54 AM\r\r❯ /plan-ceo-review \r\r●Runningtheplan-ceo-reviewpreamble,thenworkingthroughStep0onlyasfarasthemodedecisionyouaskedfor.\r\rRead 1 file, ran 5 shell commands \r\r●Auditcomplete:single-commitrepowithnoremote,stashes,TODOs,designdocs,orpriorlearnings;themode\r preference resolved to AUTO_DECIDE, so I'mproceeding n the recommendation without neeing a questin. Now running\rthehandoffandauditrecord. · summary\r\r● Running 1 shellcommand…\r✶Billowing… 39thought for 21s)\r✻4\r \r✽\r✻\r✶\r●\r*\r✢5\r·\r \r…\r✢g\r*path sort registered skill names deterministically before rendering them.\rKeep skill membership, aliases, metadata and text/JSON output formats unchanged.\rCover mixed-case names and differing directory enumeration order with tests.\r \rThisdraft is th review target,no the current branch. For this invocation,\rI want ony the evew-mode decision; I willhandle optional Ofice Hors d \rsetupseparately, nd ru the substantv review later. Noreview mode has \rbeen selected. For this mode-only handoff, use the full selected mode name in\rte nomalaudit record's user_choice and recommended fieldsand in the skill's\rnormalmodehandoff line. Amnu letteralone woudno identiy the choen mode.\r\r● Received. I have the draft plan for deterministic skill-list ordering as context, along with your constraints:\r review-mode decision only, full mode name in the audit record and handoff line, no Office Hours or setup in this\r invocation, substantive review deferred. Waiting for your slash command.\r✻Brewed for 3s · done 9:54 AM\r\r❯ /plan-ceo-review \r●Running the plan-ceo-review preamble,thenworkingthroughStep0onlyasfarasthemodedecisionyouaskedfor.\r Read 1 file, ran 5 shell commands \rAdit complete: single-commit repo with noremote,stashes, TODOs, designdocs, or prir learnings; the mode\rpreferenceresolvedtoAUTO_DECIDE,soI'mproceedingontherecommendationwithoutneedingaquestion.Nowrunning\rthe handoff and audit record. · summary\r Rnning 1 shellcommand…\r ⎿ $ ~/.claude/skills/gstack/bin/gstack-ceo-mode-handoff \"HOLD SCOPE\" --decisions \"none\" --auto; echo \r \"handoff_exit=$?\"; ~/.claude/skills/gstack/bin/gstack-question-log \r'{\"skill\":\"plan-ceo-review\",\"question_id\":\"plan-ceo-review-mode\",\"question_summary\":\"Select plan-ceo-review mode \r for deterministic sk…\rn40\r✶i…\r●\r)\r✻6\r✽wg1\ron\rmode handff andlogtheauto-decidedmoderecord\rBillowing…\rBillowing…\r \r✻Billowing…\r✶Billowing…2\r*\r✢Billowing…7\r●\r·Billowing…\rBillowing…\rBillowing…\rBillowing…\r✢Billowing…\r*Billowing…\r \r✶\r✻Billowing…8\r✽Billowing…\rBillowing…\r●\rBillowing…\rBillowing…\r✻Billowing…\r✶Billowing…\r*\r✢\r \rBillowing…9\r·Billowing…\rBillowing…\rBillowing…\rBillowing…\r●\r✢Billowing…\r*Billowing…\r✶\r✻\rBillowing…1m 0s· ↓ 4.2k tokens)\r✽Billowing…\r \rBillowing…\rBillowing…\rBillowing…1\r✻Billowing…\r✶Billowing…\r●\r*\r✢\rBillowing…\r·Billowing…\rBillowing…\r \rBillowing…\r✢Billowing…2\rBillowing…\r*Billowing…\r✶\r✻\r●\rBillowing…\r✽Billowing…\rBillowing…\rBillowing…\r✻Billowing…3\r \r✶Billowing…\r*\r✢Billowing…\r·Billowing…\r●\rBillowing…\rBillowing…\r✢Billowing…4\r*Billowing…\r \r✶\r✻Billowing…\r✽Billowing…\rBillowing…\rBillowing…\r●\r✻Billowing…5\rBillowing…\r✶\r*\r✢Billowing…\r \rBillowing…\r·Billowing…\rBillowing…\rBillowing…\r✢Billowing…6\r●\rBillowing…\r*\r✶\r✻Billowing…\r✽Billowing…\rBillowing…\r \rBillowing…\rBillowing…\r✻7\r✶\r●\r*\r✢\r·\r·Billowing… \r \r✢8\r*\r✶\r✻\r●\r✽\r9\r✻\r✶\r \r✶Billowing… \r*\r✢\r·\r●\r10s · ↓ 4.2k tokens)\r✢\r*\r \r✶● Audit complete: single-commit repo with no remote, stashes, TODOs, design docs, or prior learnings; the mode\r preference resolved to AUTO_DECIDE, so I'm proceeding on the recommendation without needing a question. Now running\r the handoff and audit record. · summary\r\r Ran 1 shell command \r ⎿ $ ~/.claude/skills/gstack/bin/gstack-ceo-mode-handoff \"HOLD SCOPE\" --decisions \"none\" --auto; echo \r● Auto-decided review mode → HOLD SCOPE (your preference). Change with /plan-tune. Approved decisions: none. This plan\r is a fix to an existing codepath: sort registered skill names before rendering, with membership, aliases, metadata and\r output fLine truncated
}
}
}
test/fixtures/detector-corpora/auto-decide-preserved/37195203538-t1-pass.json
Vendored
+217
View File
@@ -0,0 +1,217 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "auto-decide-preserved",
"detector": "test/helpers/native-auto-decide.ts#findNativeAutoDecision",
"screen_detector": "test/helpers/pty/classify.ts#classifyVisible",
"series_identity": "7678e372b3345f9b",
"source": {
"run": "37195203538",
"sha": "41287009bd6b524244edced7770c87fc3103eb59",
"lane": "periodic",
"artifact": "paid-slice-20-a1",
"path": "shards/skill-e2e-auto-decide-preserved/pty-count/ci-37195203538-1-eval-slices-20/plan-ceo-review-1791109765127-W8O99c/observation.json"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "The observer recorded nativeAutoDecide: the logged never-ask record plus the sent AUTO_DECIDE line in the same session after the preamble; the screen phrase was not needed.",
"screen_expected": "refuse",
"native_expected": "pass",
"derived": "options.stateEvidence rebuilt from the session's gstack-question-log argument with source \"agent\" and ts = that call's tool-result timestamp (the bound question-log.jsonl itself is not archived)",
"input": {
"native": {
"transcript": {
"status": "ready",
"calls": [],
"assistantMessages": [
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"text": "Received. I have the draft plan for deterministic skill-list ordering in context, along with your handoff constraints: review-mode decision only, no Office Hours or setup, full mode name in the audit record's user_choice and recommended fields and in the mode handoff line. I'll wait for your slash command before starting anything or calling tools.",
"timestamp": "2026-10-04T10:27:58.559Z"
},
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"text": "Starting the plan-ceo-review preamble now, then I'll go straight to the mode decision as you asked.",
"timestamp": "2026-10-04T10:28:08.100Z"
},
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"text": "Context gathered. The mode preference check came back AUTO_DECIDE, so I'll compute the recommendation, run the handoff, log the audit record and telemetry in one pass.",
"timestamp": "2026-10-04T10:28:54.126Z"
},
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"text": "Auto-decided review mode → HOLD SCOPE (your preference). Change with /plan-tune. Approved decisions: none. The plan is a fix to existing behavior with output formats, membership, aliases and metadata explicitly pinned unchanged, so the review's value lies in rigor on the sort key, both renderers and the enumeration-order tests rather than in expanding or cutting scope.\n\n**Mode provenance.** The `plan-ceo-review-mode` preference check returned AUTO_DECIDE, so no question was asked. The question log recorded user_choice and recommended both as the full name \"HOLD SCOPE\" with auto_decided true, on session 1545-1791109690-7adaeb43.\n\n**Step 0 evidence, kept brief**\n\n- Premise: diffing skill listings in scripts and reviews breaks on enumeration-order noise. Sorting at the listing path fixes the pain directly.\n- Leverage: reuse the existing listing path with one sort step before render. No rebuild.\n- Estimated changed files: 2, so no file-count threshold applies. Category is fix/refactor.\n- No approach decision was needed, so 0D was not called.\n\n**Deferred per your request**\n\n- Office Hours offer: no design doc found, offer skipped.\n- Setup: skill-start emitted a routing-rules instruction asking to edit and commit CLAUDE.md. Plan mode forbids that edit, and you said setup is handled separately, so it is deferred untouched.\n- Substantive review: HOLD SCOPE route is 0G, 0I, then Sections 1 to 11. Not started.\n\n**Records**\n\n- Plan file written with the mode decision and the open questions for the later review, at `.claude/plans/keep-this-draft-plan-frolicking-lake.md`.\n- Telemetry recorded with outcome abort, since the review workflow stopped by design after mode selection. Telemetry is off in this install, so the write is local only.\n- No durable learnings this session.\n\nSTATUS: DONE. Mode-only handoff complete. ExitPlanMode not called because the review workflow is intentionally paused here.",
"timestamp": "2026-10-04T10:29:24.679Z"
}
]
},
"tools": [
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"timestamp": "2026-10-04T10:28:09.022Z",
"toolUseId": "toolu_012JwPUvKc5i3fi7pzQgiEyV",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-skill-start --skill \"plan-ceo-review\" --model \"claude\"",
"description": "Run gstack skill-start preamble"
}
},
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"timestamp": "2026-10-04T10:28:10.573Z",
"toolUseId": "toolu_012JwPUvKc5i3fi7pzQgiEyV",
"kind": "result",
"isError": false,
"content": "SKILL_START_PROTO: 1\nBRANCH: main\nPROACTIVE: true\nPROACTIVE_PROMPTED: yes\nSKILL_PREFIX: false\nREPO_MODE: unknown\nSESSION_KIND: interactive\nCONDUCTOR_SESSION: true\nACTIVATED: yes\nFIRST_LOOP_SHOWN: yes\nFIRST_TASK: \nLAKE_INTRO: yes\nTELEMETRY: off\nTEL_PROMPTED: yes\nSESSION_ID: 1545-1791109690-7adaeb43\nTEL_START: 1791109690\nEXPLAIN_LEVEL: default\nQUESTION_TUNING: true\nUPDATE_CHECK: false\nLEARNINGS: 0\nHAS_ROUTING: no\nROUTING_DECLINED: false\nVENDORED_GSTACK: no\nMODEL_OVERLAY: claude\nGSTACK_PLAN_MODE: active\nARTIFACTS_SYNC: off\n"
},
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"timestamp": "2026-10-04T10:28:21.357Z",
"toolUseId": "toolu_01Lo2NJVntk2HCrHJV7s4Z1j",
"kind": "use",
"name": "Bash",
"input": {
"command": "echo \"--- remote\"; git remote get-url origin 2>&1; echo \"--- head\"; git symbolic-ref refs/remotes/origin/HEAD 2>&1; git rev-parse --verify origin/main 2>&1; echo \"--- log\"; git log --oneline -30; echo \"--- stash\"; git stash list; echo \"--- files\"; git ls-files; echo \"--- todos\"; grep -rn \"TODO\\|FIXME\\|HACK\\|XXX\" --exclude-dir=.git . 2>/dev/null | head -30; echo \"--- recent\"; git log --since=30.days --name-only --format=\"\" | sort | uniq -c | sort -rn | head -20",
"description": "Audit repo state: remote, base branch, history, files"
}
},
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"timestamp": "2026-10-04T10:28:25.858Z",
"toolUseId": "toolu_01LLpuyDZpfMtkeDLcVxGoMf",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-context-recovery 2>&1; echo \"--- design doc\"; SLUG=$(~/.claude/skills/gstack/browse/bin/remote-slug 2>/dev/null || basename \"$(git rev-parse --show-toplevel 2>/dev/null || pwd)\"); BRANCH=$(git rev-parse --abbrev-ref HEAD 2>/dev/null | tr '/' '-' || echo 'no-branch'); DESIGN=$(~/.claude/skills/gstack/bin/gstack-design-doc-find \"$SLUG\" \"$BRANCH\"); [ -n \"$DESIGN\" ] && echo \"Design doc found: $DESIGN\" || echo \"No design doc found\"; echo \"--- learnings\"; _CROSS_PROJ=$(~/.claude/skills/gstack/bin/gstack-config get cross_project_learnings 2>/dev/null || echo \"unset\"); echo \"CROSS_PROJECT: $_CROSS_PROJ\"; ~/.claude/skills/gstack/bin/gstack-learnings-search --limit 10 2>&1 | head -20",
"description": "Recover context, check design doc and prior learnings"
}
},
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"timestamp": "2026-10-04T10:28:29.796Z",
"toolUseId": "toolu_01CUoUKTjEbjBoXRzXnVBR86",
"kind": "use",
"name": "Bash",
"input": {
"command": "printf '%s' \"Select plan-ceo-review mode for deterministic skill-list ordering plan: SCOPE EXPANSION, SELECTIVE EXPANSION, HOLD SCOPE, or SCOPE REDUCTION\" | ~/.claude/skills/gstack/bin/gstack-question-preference --check \"plan-ceo-review-mode\" --summary-stdin; echo \"exit=$?\"; echo \"--- brain\"; SLUG=$(~/.claude/skills/gstack/bin/gstack-slug --get SLUG 2>/dev/null) || true; for d in product goals recent-decisions; do echo \"### $d\"; ~/.claude/skills/gstack/bin/gstack-brain-cache get $d --project \"$SLUG\" 2>/dev/null || echo \"(no $d digest available yet)\"; done; echo \"### user-profile\"; ~/.claude/skills/gstack/bin/gstack-brain-cache get user-profile 2>/dev/null || echo \"(no user-profile digest available yet)\"",
"description": "Check mode question preference and load brain context"
}
},
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"timestamp": "2026-10-04T10:28:30.430Z",
"toolUseId": "toolu_014CMt1fkSaWpH9PTAEkgVXz",
"kind": "use",
"name": "Read",
"input": {
"file_path": "<CWD>/PLAN.md"
}
},
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"timestamp": "2026-10-04T10:28:30.867Z",
"toolUseId": "toolu_01Lo2NJVntk2HCrHJV7s4Z1j",
"kind": "result",
"isError": false,
"content": "--- remote\nerror: No such remote 'origin'\n--- head\nfatal: ref refs/remotes/origin/HEAD is not a symbolic ref\nb57232966b60b0a7c3cefc5aebcb5dac0b435124\n--- log\nb5…[elided: not consumed]"
},
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"timestamp": "2026-10-04T10:28:30.974Z",
"toolUseId": "toolu_01LLpuyDZpfMtkeDLcVxGoMf",
"kind": "result",
"isError": false,
"content": "--- RECENT ARTIFACTS ---\n{\"skill\":\"plan-ceo-review\",\"event\":\"started\",\"branch\":\"main\",\"session\":\"1545-1791109690-7adaeb43\",\"ts\":\"2026-10-04T10:28:10.588Z\"}\n--- END ARTIFACTS ---\n--- design doc\nNo design doc found\n--- learnings\nCROSS_PROJECT: false"
},
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"timestamp": "2026-10-04T10:28:31.122Z",
"toolUseId": "toolu_01CUoUKTjEbjBoXRzXnVBR86",
"kind": "result",
"isError": false,
"content": "AUTO_DECIDE\nexit=0\n--- brain\n### product\n(no product digest available yet)\n### goals\n(no goals digest available yet)\n### recent-decisions\n(no recent-decisions digest available yet)\n### user-profile\n(no user-profile digest available yet)"
},
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"timestamp": "2026-10-04T10:28:31.130Z",
"toolUseId": "toolu_014CMt1fkSaWpH9PTAEkgVXz",
"kind": "result",
"isError": false,
"content": "1\t# Draft: deterministic skill-list ordering\n2\t\n3\tUsers compare skill listings in scripts and reviews. Make the existing listing\n4\tpath sort registered skill na…[elided: not consumed]"
},
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"timestamp": "2026-10-04T10:28:58.435Z",
"toolUseId": "toolu_01LvfeR9remcV4fbVwEPDvmv",
"kind": "use",
"name": "Bash",
"input": {
"command": "echo \"--- handoff\"; ~/.claude/skills/gstack/bin/gstack-ceo-mode-handoff \"HOLD SCOPE\" --decisions \"none\" --auto 2>&1; echo \"--- qlog\"; ~/.claude/skills/gstack/bin/gstack-question-log '{\"skill\":\"plan-ceo-review\",\"question_id\":\"plan-ceo-review-mode\",\"question_summary\":\"Select plan-ceo-review mode for deterministic skill-list ordering plan\",\"category\":\"routing\",\"door_type\":\"two-way\",\"options_count\":4,\"user_choice\":\"HOLD SCOPE\",\"recommended\":\"HOLD SCOPE\",\"auto_decided\":true,\"session_id\":\"1545-1791109690-7adaeb43\"}' 2>&1 && echo \"qlog ok\"; echo \"--- tel\"; ~/.claude/skills/gstack/bin/gstack-skill-end --skill \"plan-ceo-review\" --outcome abort --session-id \"1545-1791109690-7adaeb43\" --tel-start \"1791109690\" --used-browse no --error-message \"\" --failed-step \"\" 2>&1; echo \"tel exit=$?\"",
"description": "Run mode handoff, log mode decision, record telemetry"
}
},
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"timestamp": "2026-10-04T10:29:11.007Z",
"toolUseId": "toolu_01RRksiyxRReq6E6AVqfYhZn",
"kind": "use",
"name": "Write",
"input": {
"file_path": "<CLAUDE_CONFIG_DIR>/plans/keep-this-draft-plan-frolicking-lake.md",
"content": "# plan-ceo-review: mode decision for \"deterministic skill-list ordering\"\n\n## Context\n\nReview target: PLAN.md draft (deterministic skill-list ordering). The user asked for\nthe review-mode decision only. Office Hours, setup (CLAUDE.md routing rules), and\nthe substantive Sections 1-11 review are deferred to a later invocation.\n\n## Step 0 evidence (brief)\n\n- 0A Premise: scripts and reviewers diff skill listings; nondeterministic order from\n directory enumeration produces spurious diffs. Sorting at the existing listing path\n fixes the pain directly, not a proxy.\n- 0B Leverage: reuse the existing listing/render path; add a sort step before render.\n No rebuild justified.\n- 0C Dream state: current = enumeration-order output; this plan = stable sorted\n output in text and JSON; 12-month ideal = all list-like CLI output stable and\n diff-friendly. The plan moves toward it.\n- 0D: no approach choice required.\n\n## 0E Mode selection\n\n- Estimated changed files: 2 (listing module, test file). Estimate.\n- Category: fix/refactor of existing behavior; membership, aliases, metadata and\n output formats pinned unchanged.\n- Recommended: HOLD SCOPE.\n- Provenance: `plan-ceo-review-mode` preference check returned AUTO_DECIDE\n (session 1545-1791109690-7adaeb43). Selected mode: HOLD SCOPE. No question asked.\n Question log recorded with user_choice and recommended both set to the full mode\n name \"HOLD SCOPE\", auto_decided true.\n\n## Open for the later substantive review (HOLD SCOPE route: 0G -> 0I -> Sections 1-11)\n\n- Sort key: byte order vs case-insensitive with a deterministic tie-break for\n mixed-case names (e.g. `Foo` vs `foo`); locale independence.\n- Apply the same sort to both text and JSON renderers, or sort once upstream.\n- Tests: mixed-case names; shuffled directory enumeration order yields identical output.\n- Alias handling: aliases must keep pointing at the same skill after sorting.\n"
}
},
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"timestamp": "2026-10-04T10:29:12.065Z",
"toolUseId": "toolu_01LvfeR9remcV4fbVwEPDvmv",
"kind": "result",
"isError": false,
"content": "--- handoff\nAuto-decided review mode → HOLD SCOPE (your preference). Change with /plan-tune. Approved decisions: none.\n--- qlog\nqlog ok\n--- tel\nSKILL_END: recorded outcome=abort duration_s=62\ntel exit=0"
},
{
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"timestamp": "2026-10-04T10:29:12.102Z",
"toolUseId": "toolu_01RRksiyxRReq6E6AVqfYhZn",
"kind": "result",
"isError": false,
"content": "File created successfully at: <CLAUDE_CONFIG_DIR>/plans/keep-this-draft-plan-frolic…[elided: not consumed]"
}
],
"options": {
"skillName": "plan-ceo-review",
"sessionId": "94e3f44b-b75d-4187-a3e3-0fc829a7d75d",
"commandStartedAt": 1791109678876,
"now": 1791109765126,
"proseQuestionObserved": false,
"stateEvidence": {
"questionId": "plan-ceo-review-mode",
"preference": "never-ask",
"records": [
{
"skill": "plan-ceo-review",
"question_id": "plan-ceo-review-mode",
"question_summary": "Select plan-ceo-review mode for deterministic skill-list ordering plan",
"category": "routing",
"door_type": "two-way",
"options_count": 4,
"user_choice": "HOLD SCOPE",
"recommended": "HOLD SCOPE",
"auto_decided": true,
"session_id": "1545-1791109690-7adaeb43",
"source": "agent",
"ts": "2026-10-04T10:29:12.065Z"
}
]
}
}
},
"screen": {
"path": "shards/skill-e2e-auto-decide-preserved/pty-count/ci-37195203538-1-eval-slices-20/plan-ceo-review-1791109765127-W8O99c/terminal.visible.log",
"tail_chars": 6000,
"text": "hem. \rKep skill membership, aliass, medata and txt/JSON output formats unchanged.\rCover mixed-case names and differing irectoryenumeration ordrwithtests. \r \rThis draft is the revew targt,no the current brach. For this invocation, \r I want only the review-mode decision; I will handle optional Office Hours and \r setup separately, and run the substantive review later. No review mode has \r been selected. For this mode-only handoff, use the full selected mode name in \r the normal audit record's user_choice and recommended fields and in the skill's \r normal mode handoff line. A menu letter alone would not identify the chosen mode. \r\r●Received.Ihavethedraftplanfordeterministicskill-listorderingincontext,alongwithyourhandoffconstraints:\r review-mode decision only, no Office Hours or setup, full mode name in the audit record's user_choice and recommended\rfieldsandinthemodehandoffline.I'llwaitforyourslashcommandbeforestartinganythingorcallingtools.\r\r✻Cooked for 4s · done 10:27 AM\r\r❯ /plan-ceo-review \r\r●Startingtheplan-ceo-review preamble now, then I'll go straighttothemodedecisionasyouasked.\r Read 1 file, ran 4 shell commands \r\r● Context gathered. The mode preference check came back AUTO_DECIDE, so I'll compute the recommendation, run the\rhandoff, log the audit record and telemetry in one pass.\r\rRan 1 shell command \r \rUsers compare skill listings in scripts and reviews. Make the existing listing\rpath sort regitered skill names deterministically beforrendrng them. \rKeepkill membership, aiases, metadata and text/JSON output formats unchanged.\rCover mxed-cas namesnd differing directory enumeration order wihtests. \r \rThis draft is the review target, not the current branch. For this invocation,\rI want only the review-mode dcisin; I will handle optionalOffice Hurs ad\rsetup separaely, and run the substantive review ater. No review mode has \rben selected. For this mode-only hdoff, use the fullselected mode name in\rth normal audit ecord's user_choice and recommended fields anin the skill's\rnormal ode handoff line.A menu letter alone woul not identifythe choen mode.\r\r●Received.Ihavethedraftplanfordeterministicskill-listorderingincontext,alongwithyourhandoffconstraints:\r rview-modedecisiononly,no Ofice Hous or setup, ful mode name in the audi recrd's user_choice andrecommended\rfields an in the mode handoff line.I'll wait for your slash comand before starting anything or calligtools.\r\r✻Cooked for 4s · done 10:27 AM\r\r❯ /plan-ceo-review \r\r●Startingtheplan-ceo-reviewpreamblenow,thenI'llgostraighttothemodedecisionasyouasked.\r\rRead 1 file, ran 4 shell commands \r\r●Contextgathered.ThemodepreferencecheckcamebackAUTO_DECIDE,soI'llcomputetherecommendation,runthe\r hadoff, log the audit record and telemetryin one pass.\r\rRan 1 shell command \r\r●Updated plan\r ⎿  /plan to preview\r52\r✻\rSwirling…\r✽Swirling…\rSwirling…4\rSwirling…\rSwirling…\r✻Swirling…\r✶Swirling…\r*\r✢\rSwirling…\r·Swirling…\rSwirling…5\rSwirling…\r✢Swirling…\rSwirling…\r*Swirling…\r✶\r✻\rSwirling…\r✽Swirling…\rSwirling…6\rSwirling…\r✻Swirling…\r✶Swirling…\r*\r✢Swirling…\r·Swirling…7\rSwirling…\rSwirling…\r✢Swirling…\r*Swirling…\r✶\r✻Swirling…\r✽Swirling…8\rSwirling…\rSwirling…\r✻Swirling…\rSwirling…\r✶\r*\r✢Swirling…\rSwirling…\r·Swirling…9\rSwirling…\rSwirling…\r✢Swirling…\rSwirling…\r*\r✶\r✻Swirling…\r✽Swirling…\rSwirling…20\rSwirling…\rSwirling…\r✻Swirling…\rSwirling…\r✶\r*\r✢Swirling…\r·Swirling…\rSwirling…1\rSwirling…\rSwirling…\r✢Swirling…\rSwirling…\r*\r✶\r✻Swirling…\r✽Swirling…\rSwirling…2\rSwirling…\rSwirling…\r✻Swirling…\r✶Swirling…\r*\r✢Swirling…\r·Swirling…\rSwirling…3\rSwirling…\rSwirling…\r✢Swirling…\r*Swirling…\r✶Swirling…\r✻\r✽\r4\r✻\r✶\r*\r*Swirling… \r✢\r·\r5\r✢\r*\r✶\r✻\r✽\r✽Swirling… 6\r✽Swirling… a fix to existing behavior with output formats, membership, aliases and metadata explicitly pinned unchanged, so the\r review's value lies in rigor on the sort key, both renderers and the enumeration-order tests rather than in expanding\r or cutting scope.\r\r Mode provenance. The plan-ceo-review-mode preference check returnLine truncated
}
}
}
test/fixtures/detector-corpora/auto-decide-preserved/37198445662-t1-pass.json
Vendored
+323
View File
@@ -0,0 +1,323 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "auto-decide-preserved",
"detector": "test/helpers/native-auto-decide.ts#findNativeAutoDecision",
"screen_detector": "test/helpers/pty/classify.ts#classifyVisible",
"series_identity": "7678e372b3345f9b",
"source": {
"run": "37198445662",
"sha": "fd6854b5356431e5e115fca082b21bf237559558",
"lane": "periodic",
"artifact": "paid-slice-20-a1",
"path": "shards/skill-e2e-auto-decide-preserved/pty-count/ci-37198445662-1-eval-slices-20/plan-ceo-review-1791113328798-6JFWyh/observation.json"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "classifyVisible accepted the sent AUTO_DECIDE line on screen. The archived native poll predates that message (last assistant message announces the handoff), and the handoff ran inside a compound command, so the native snapshot holds no current declaration.",
"screen_expected": "pass",
"native_expected": "refuse",
"derived": "options.stateEvidence rebuilt from the session's gstack-question-log argument with source \"agent\" and ts = that call's tool-result timestamp (the bound question-log.jsonl itself is not archived)",
"input": {
"native": {
"transcript": {
"status": "ready",
"calls": [],
"assistantMessages": [
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"text": "Received. I have the draft plan for deterministic skill-list ordering as context, along with your constraints: this is a mode-only handoff, no review mode selected yet, use the full mode name in the audit record and handoff line. I'll wait for your slash command before starting or calling any tools.",
"timestamp": "2026-10-04T11:25:02.647Z"
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"text": "I'll run the skill preamble first, then work through Step 0 to the mode decision only, as you asked.",
"timestamp": "2026-10-04T11:25:07.190Z"
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"text": "I'll defer the CLAUDE.md setup prompt since you're handling it separately and plan mode blocks edits anyway—it'll reappear next run. Now gathering the audit context in one pass.\n\n",
"timestamp": "2026-10-04T11:25:34.122Z"
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"text": "The fixture repo only has CLAUDE.md and PLAN.md—no remote, stash, TODOs, design doc, or prior context, and Aside isn't installed. Since mode preference is set to auto-decide, I'll skip that question and run the landscape check via WebSearch instead.\n\n",
"timestamp": "2026-10-04T11:26:36.166Z"
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"text": "Confirmed the Reproducible Builds guidance aligns with the plan: avoid relying on readdir order and sort entries bytewise to prevent locale-based inconsistencies. Now writing the Step 0 ledger to the plan file while running the mode handoff and audit log in parallel.\n\n",
"timestamp": "2026-10-04T11:27:37.493Z"
}
]
},
"tools": [
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:25:08.567Z",
"toolUseId": "toolu_01Wf6kvaTmd5c4BVKu8m21EF",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-skill-start --skill \"plan-ceo-review\" --model \"claude\"",
"description": "Run gstack skill preamble"
}
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:25:10.032Z",
"toolUseId": "toolu_01Wf6kvaTmd5c4BVKu8m21EF",
"kind": "result",
"isError": false,
"content": "SKILL_START_PROTO: 1\nBRANCH: main\nPROACTIVE: true\nPROACTIVE_PROMPTED: yes\nSKILL_PREFIX: false\nREPO_MODE: unknown\nSESSION_KIND: interactive\nCONDUCTOR_SESSION: true\nACTIVATED: yes\nFIRST_LOOP_SHOWN: yes\nFIRST_TASK: \nLAKE_INTRO: yes\nTELEMETRY: off\nTEL_PROMPTED: yes\nSESSION_ID: 1562-1791113109-05c92427\nTEL_START: 1791113109\nEXPLAIN_LEVEL: default\nQUESTION_TUNING: true\nUPDATE_CHECK: false\nLEARNINGS: 0\nHAS_ROUTING: no\nROUTING_DECLINED: false\nVENDORED_GSTACK: no\nMODEL_OVERLAY: claude\nGSTACK_PLAN_MODE: active\nARTIFACTS_SYNC: off\n"
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:25:36.494Z",
"toolUseId": "toolu_01WfcV1ucLDK2iPD1sy2MgZP",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-context-recovery 2>&1; echo \"---REMOTE---\"; git remote get-url origin 2>&1; echo \"---HEAD---\"; git symbolic-ref refs/remotes/origin/HEAD 2>&1; git rev-parse --verify origin/main 2>&1; echo \"---LOG---\"; git log --oneline -30; echo \"---STASH---\"; git stash list; echo \"---TODO---\"; grep -r \"TODO\\|FIXME\\|HACK\\|XXX\" -l --exclude-dir=node_modules --exclude-dir=vendor --exclude-dir=.git . | head -30; echo \"---TOUCHED---\"; git log --since=30.days --name-only --format=\"\" | sort | uniq -c | sort -rn | head -20; echo \"---FILES---\"; git ls-files | head -100",
"description": "Recover context and audit repo history"
}
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:25:50.500Z",
"toolUseId": "toolu_01NLsv1JwFa2z6QM2DjX2iKh",
"kind": "use",
"name": "Bash",
"input": {
"command": "setopt +o nomatch 2>/dev/null || true\nSLUG=$(~/.claude/skills/gstack/browse/bin/remote-slug 2>/dev/null || basename \"$(git rev-parse --show-toplevel 2>/dev/null || pwd)\")\nBRANCH=$(git rev-parse --abbrev-ref HEAD 2>/dev/null | tr '/' '-' || echo 'no-branch')\nDESIGN=$(~/.claude/skills/gstack/bin/gstack-design-doc-find \"$SLUG\" \"$BRANCH\")\n[ -n \"$DESIGN\" ] && echo \"Design doc found: $DESIGN\" || echo \"No design doc found\"\necho \"---LEARNINGS---\"\n_CROSS_PROJ=$(~/.claude/skills/gstack/bin/gstack-config get cross_project_learnings 2>/dev/null || echo \"unset\")\necho \"CROSS_PROJECT: $_CROSS_PROJ\"\nif [ \"$_CROSS_PROJ\" = \"true\" ]; then\n { _LE=$(~/.claude/skills/gstack/bin/gstack-learnings-search --limit 10 --cross-project 2>&1 >&3 3>&-); _LR=$?; } 3>&1\nelse\n { _LE=$(~/.claude/skills/gstack/bin/gstack-learnings-search --limit 10 2>&1 >&3 3>&-); _LR=$?; } 3>&1\nfi\n[ \"$_LR\" = 0 ] || { _LE=${_LE%%$'\\n'*}; echo \"LEARNINGS: unavailable (${_LE:-exit $_LR})\"; }\necho \"---BRAIN---\"\nSLUG2=$(~/.claude/skills/gstack/bin/gstack-slug --get SLUG 2>/dev/null) || true\nfor d in product goals recent-decisions; do echo \"### $d\"; ~/.claude/skills/gstack/bin/gstack-brain-cache get $d --project \"$SLUG2\" 2>/dev/null || echo \"_(no $d digest available yet)_\"; done\necho \"### user-profile\"; ~/.claude/skills/gstack/bin/gstack-brain-cache get user-profile 2>/dev/null || echo \"_(no user-profile digest available yet)_\"\necho \"---ASIDE---\"\n_gs_d() { if command -v gtimeout >/dev/null; then gtimeout 30 \"$@\"; elif command -v timeout >/dev/null; then timeout 30 \"$@\"\nelif command -v perl >/dev/null; then perl -e 'alarm(shift);exec(@ARGV)' 30 \"$@\"; else return 125; fi; }\n_A=aside; command -v aside >/dev/null || _A=$(command -v ~/.local/bin/aside)\nif [ \"${GSTACK_SKIP_ASIDE:-}\" = \"1\" ] || [ -z \"$_A\" ]; then\n echo \"NEEDS_ASIDE: ${GSTACK_PLATFORM:-$(uname)}\"\nelse\n _rc=0; _o=$(_gs_d \"$_A\" repl 'console.log(\"ASIDE_READY \" + pwd)' 2>&1) || _rc=$?\n case \"$_rc\" in\n 124|142) echo \"ASIDE_TIMEOUT: probe deadline exceeded\" ;;\n 125) echo \"ASIDE_UNAVAILABLE: bounded probe unavailable\" ;;\n 0) if printf '%s\\n' \"$_o\" | grep -q '^ASIDE_READY '; then echo \"READY: $_A\"\n else echo \"ASIDE_NOT_RUNNING: no readiness marker\"; fi ;;\n *) echo \"ASIDE_CLI_ERROR: exit $_rc; inspect aside --help locally\" ;;\n esac\nfi\necho \"---QPREF---\"\nprintf '%s' \"Select plan-ceo-review mode for deterministic skill-list ordering plan: SCOPE EXPANSION, SELECTIVE EXPANSION, HOLD SCOPE, or SCOPE REDUCTION\" | ~/.claude/skills/gstack/bin/gstack-question-preference --check \"plan-ceo-review-mode\" --summary-stdin; echo \"QPREF_EXIT: $?\"",
"description": "Check design doc, learnings, brain context, Aside, and mode preference"
}
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:25:51.497Z",
"toolUseId": "toolu_01Jg384kG7A8kPWZHjzUixKa",
"kind": "use",
"name": "Read",
"input": {
"file_path": "<CWD>/PLAN.md"
}
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:25:53.678Z",
"toolUseId": "toolu_01QtvhyccyZBEYf9hbVcKxvb",
"kind": "use",
"name": "Bash",
"input": {
"command": "ls -la <CWD>; echo \"---TODOS---\"; cat <CWD>/TODOS.md 2>&1 | head -50; echo \"---ETHOS---\"; sed -n '1,60p' ~/.claude/skills/gstack/ETHOS.md 2>&1",
"description": "List repo files, read TODOS.md and ETHOS.md"
}
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:25:54.352Z",
"toolUseId": "toolu_01WfcV1ucLDK2iPD1sy2MgZP",
"kind": "result",
"isError": false,
"content": "--- RECENT ARTIFACTS ---\n{\"skill\":\"plan-ceo-review\",\"event\":\"started\",\"branch\":\"main\",\"session\":\"1562-1791113109-05c92427\",\"ts\":\"2026-10-04T11:25:10.051Z\"}\n--- END ARTIFACTS ---\n---REMOTE---\nerror: No such remote 'origin'\n---HEAD---\nfatal: ref refs/remotes/origin/HEAD is not a symbolic ref\n9b45bea985d4de4cc1c1b2619d16faa32cd42f91\n---LOG---\n9b45bea Seed review plan\n---STASH---\n---TODO---\n---TOUCHED---\n 1 PLAN.md\n 1 CLAUDE.md\n---FILES---\nCLAUDE.md\nPLAN.md"
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:25:54.644Z",
"toolUseId": "toolu_01NLsv1JwFa2z6QM2DjX2iKh",
"kind": "result",
"isError": false,
"content": "No design doc found\n---LEARNINGS---\nCROSS_PROJECT: false\n---BRAIN---\n### product\n_(no product digest available yet)_\n### goals\n_(no goals digest available yet)_\n### recent-decisions\n_(no recent-decisions digest available yet)_\n### user-profile\n_(no user-profile digest available yet)_\n---ASIDE---\nNEEDS_ASIDE: Linux\n---QPREF---\nAUTO_DECIDE\nQPREF_EXIT: 0"
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:25:54.680Z",
"toolUseId": "toolu_01Jg384kG7A8kPWZHjzUixKa",
"kind": "result",
"isError": false,
"content": "1\t# Draft: deterministic skill-list ordering\n2\t\n3\tUsers compare skill listings in scripts and reviews. Make the existing listing\n4\tpath sort registered skill na…[elided: not consumed]"
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:25:54.689Z",
"toolUseId": "toolu_01QtvhyccyZBEYf9hbVcKxvb",
"kind": "result",
"isError": false,
"content": "total 20\ndrwx------ 3 runner runner 4096 Oct 4 11:24 .\ndrwxr-xr-x 7 runner runner 4096 Oct 4 11:25 ..\ndrwxr-xr-x 8 runner runner 4096 Oct 4 11:24 .git\n-rw-r--r-- 1 runner runner 1132 Oct 4 11:24 CLAUDE.md\n-rw-r--r-- 1 runner runner 826 Oct 4 11:24 PLAN.md\n---TODOS---\ncat: <CWD>/TODOS.md: No such file or directory\n---ETHOS---\n# gstack Builder Ethos\n\nThese are the principles that shape how gstack thinks, recommends, and builds.\nSkill preambles carry condensed forms of some of them (completeness, voice,\nsearch before building), and the Search Before Building section points here for\nthe full text. They reflect what we believe about building software in 2026.\n\n---\n\n## The Golden Age\n\nA single person with AI can now build what used to take a team of twenty.\nThe engineering barrier is gone. What remains is taste, judgment, and the\nwillingness to do the complete thing.\n\nThis is not a prediction — it's happening right now. 10,000+ usable lines of\ncode per day. 100+ commits per week. Not by a team. By one person, part-time,\nusing the right tools. The compression ratio between human-team time and\nAI-assisted time ranges from 3x (research) to 100x (boilerplate):\n\n| Task type | Human team | AI-assisted | Compression |\n|-----------------------------|-----------|-------------|-------------|\n| Boilerplate / scaffolding | 2 days | 15 min | ~100x |\n| Test writing | 1 day | 15 min | ~50x |\n| Feature implementation | 1 week | 30 min | ~30x |\n| Bug fix + regression test | 4 hours | 15 min | ~20x |\n| Architecture / design | 2 days | 4 hours | ~5x |\n| Research / exploration | 1 day | 3 hours | ~3x |\n\nThis table changes everything about how you make build-vs-skip decisions.\nThe last 10% of completeness that teams used to skip? It costs seconds now.\n\n---\n\n## 1. Boil the Ocean\n\n\"Don't boil the ocean\" was the right advice when engineering time was the\nbottleneck. That era is over. AI-assisted coding makes the marginal cost of\ncompleteness near-zero, so the old caution has quietly turned into an excuse.\nWhen the complete implementation costs minutes more than the shortcut — do the\ncomplete thing. Every time.\n\n**Ocean, lakes first:** The ocean is the destination — 100% test coverage for a\nmodule, full feature implementation, all edge cases, complete error paths. You\nget there one lake at a time: each lake is a boilable unit, not the ceiling.\n\"That's boiling the ocean\" is no longer a reason to ship a shortcut — boiling\nthe ocean is the goal. The only thing still out of scope is genuinely unrelated\nwork: a multi-quarter platform migration that has nothing to do with the task at\nhand. Flag that as separate scope. Boil everything else.\n\n**Completeness is cheap.** When evaluating \"approach A (full, ~150 LOC) vs\napproach B (90%, ~80 LOC)\" — always prefer A. The 70-line delta costs\nseconds with AI coding. \"Ship the shortcut\" is legacy thinking from when\nhuman engineering time was the bottleneck.\n\n**Anti-patterns:**\n- \"Choose B — it covers 90% with less code.\" (If A is 70 lines more, choose A.)\n- \"Let's defer tests to a follow-up PR.\" (Tests are the cheapest lake to boil.)\n- \"This would take 2 weeks.\" (Say: \"2 weeks human / ~1 hour AI-assisted.\")"
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:26:36.168Z",
"toolUseId": "toolu_013JNLGRRstXZfcntr9ahtSc",
"kind": "use",
"name": "ToolSearch",
"input": {
"query": "select:WebSearch",
"max_results": 1
}
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:26:36.171Z",
"toolUseId": "toolu_013JNLGRRstXZfcntr9ahtSc",
"kind": "result",
"isError": false,
"content": [
{
"type": "tool_reference",
"tool_name": "WebSearch"
}
]
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:26:36.532Z",
"toolUseId": "toolu_01GNkE1RxcdKUAUirZHVX6aL",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-ceo-mode-handoff --help 2>&1 | head -30",
"description": "Show mode handoff helper usage"
}
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:26:37.103Z",
"toolUseId": "toolu_01GNkE1RxcdKUAUirZHVX6aL",
"kind": "result",
"isError": false,
"content": "gstack-ceo-mode-handoff: unknown mode: --help"
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:26:48.243Z",
"toolUseId": "toolu_019yK3jaeJGwVALSWypGcXzd",
"kind": "use",
"name": "WebSearch",
"input": {
"query": "deterministic sorted output CLI listing reproducible scripts filesystem enumeration order best practice"
}
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:27:25.578Z",
"toolUseId": "toolu_019yK3jaeJGwVALSWypGcXzd",
"kind": "result",
"isError": false,
"content": "Web search results for query: \"deterministic sorted output CLI listing reproducible scripts filesystem enumeration order best practice\"\n\nLinks: [{\"title\":\"Nix: …[elided: not consumed]"
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:28:17.198Z",
"toolUseId": "toolu_01Si5mXg44rZxfPohQa5QFqc",
"kind": "use",
"name": "Write",
"input": {
"file_path": "<CLAUDE_CONFIG_DIR>/plans/keep-this-draft-plan-indexed-beaver.md",
"content": "# Working plan: deterministic skill-list ordering\n\nSource: PLAN.md in the fixture repo (commit 9b45bea \"Seed review plan\").\nReview: /plan-ceo-review, mode-only handoff requested by the user. The\nsubstantive review (Sections 1–11, outputs, report) is deliberately NOT run in\nthis invocation; the user will run it later.\n\n## Plan under review (verbatim from PLAN.md)\n\nUsers compare skill listings in scripts and reviews. Make the existing listing\npath sort registered skill names deterministically before rendering them.\nKeep skill membership, aliases, metadata and text/JSON output formats unchanged.\nCover mixed-case names and differing directory enumeration order with tests.\n\n## Review depth\n\nImplementation-ready (default; the user did not ask for strategy-only or a\nsingle narrow decision). Stated limits recorded from the plan:\n\n| Measure | Value | Unit | Prerequisite |\n|---|---|---|---|\n| Skill membership | unchanged | set of registered names | existing registry |\n| Aliases | unchanged | mapping | existing registry |\n| Metadata | unchanged | per-skill fields | existing registry |\n| Output formats | unchanged | text and JSON | existing renderers |\n| Ordering | deterministic | sort of registered names before render | existing listing path |\n| Tests | required | mixed-case names; differing directory enumeration order | test harness |\n\n## Pre-review system audit\n\n- Platform: unknown (no `origin` remote). Base branch: `main` (git-native fallback).\n- History: one commit, `9b45bea Seed review plan`. No stash. No TODO/FIXME/HACK markers.\n- Repo contents: `CLAUDE.md`, `PLAN.md` only. The listing code the plan modifies is\n not present in this fixture, so code-level leverage below is stated as unknowns\n for the implementation owner to confirm.\n- TODOS.md: absent. No prior deferrals.\n- Design doc: none found. User is handling /office-hours separately; offer not made.\n- Retrospective check: no prior review refactors or reverts (single commit).\n- DESIGN_SCOPE: none. CLI text/JSON listing only; no UI screens or components. Section 11 skips.\n- Taste calibration: not applicable (recommended mode is HOLD SCOPE, not an expansion mode).\n- Prior learnings: cross_project_learnings=false; 0 project learnings found.\n- Brain context: all four digests cold (no product, goals, recent-decisions, user-profile).\n- Landscape check: Aside not installed (NEEDS_ASIDE); ran one read-only WebSearch.\n - [Layer 1] Tried and true: never rely on readdir/listdir/glob order; apply an\n explicit sort; sort bytewise (codepoint or `LC_ALL=C`) so locale collation\n cannot change results. Source: reproducible-builds.org \"Stable order for inputs\"\n and \"Commandments\".\n - [Layer 2] Search results: Python glob deliberately stays unsorted (bpo-30461);\n systemd and mono both landed \"sort the input file list\" fixes; disorderfs and\n reprotest exist to shuffle/reverse directory order in CI to catch exactly this bug.\n - [Layer 3] First principles: the plan says \"deterministic\", not \"alphabetical\".\n Those differ for mixed case. Codepoint order (`Zeta` before `alpha`) is\n deterministic and locale-free but reads oddly; case-insensitive order needs a\n stable tiebreak (e.g. fold then raw) or two names differing only by case\n produce enumeration-dependent output again. This is the one real design\n choice inside the plan and belongs to the implementation review, not mode\n selection. No eureka to log.\n\n## Step 0 ledger\n\n| ID and owner | Contract and evidence | Current | Proposed | Status | Exact approval and scope |\n|---|---|---|---|---|---|\n| S0-PREMISE (user) | PLAN.md lines 3–6: listings are diffed by scripts and reviewers; nondeterministic order creates false diffs | Listing renders names in registry/enumeration order (unverified: code not in fixture) | Sort registered names before render; membership, aliases, metadata, formats unchanged | approved (plan as stated) | PLAN.md as handed over; no change proposed by this review |\n| S0-COLLATION (implementation owner) | Plan requires mixed-case test coverage; \"deterministic\" does not specify codepoint vs case-insensitive order | unknown | pending: codepoint sort vs case-folded sort with raw tiebreak | unresolved | Not a Step 0 approach choice; defer to Review Sections (implementation-ready depth). No approval requested here |\n| S0-MODE (user) | 0E recommendation; preference check `plan-ceo-review-mode` returned AUTO_DECIDE | none selected | HOLD SCOPE | approved (auto-decided) | Preference check exit 0 AUTO_DECIDE; see handoff provenance below |\n\n### 0A. Premise Challenge\n\nReal problem: two runs of the same listing can print the same skills in different\norder because the registry is populated from directory enumeration, which POSIX\nleaves unspecified. Target outcome: identical input produces byte-identical text\nand JSON listings, so `diff`, snapshot tests and review comments stop flaggiLine truncated
}
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:28:17.279Z",
"toolUseId": "toolu_01Si5mXg44rZxfPohQa5QFqc",
"kind": "result",
"isError": false,
"content": "File created successfully at: <CLAUDE_CONFIG_DIR>/plans/keep-this-draft-plan-indexe…[elided: not consumed]"
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:28:20.265Z",
"toolUseId": "toolu_01RXKeSPQ9g8Bxx8y1hTfwAk",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-ceo-mode-handoff \"HOLD SCOPE\" --decisions \"none\" --auto; echo \"HANDOFF_EXIT: $?\"; echo \"---QLOG---\"; ~/.claude/skills/gstack/bin/gstack-question-log '{\"skill\":\"plan-ceo-review\",\"question_id\":\"plan-ceo-review-mode\",\"question_summary\":\"Select plan-ceo-review mode for deterministic skill-list ordering plan\",\"category\":\"routing\",\"door_type\":\"two-way\",\"options_count\":4,\"user_choice\":\"HOLD SCOPE\",\"recommended\":\"HOLD SCOPE\",\"auto_decided\":true,\"session_id\":\"1562-1791113109-05c92427\"}'; echo \"QLOG_EXIT: $?\"",
"description": "Run mode handoff and log auto-decided mode"
}
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:28:21.195Z",
"toolUseId": "toolu_01RXKeSPQ9g8Bxx8y1hTfwAk",
"kind": "result",
"isError": false,
"content": "Auto-decided review mode → HOLD SCOPE (your preference). Change with /plan-tune. Approved decisions: none.\nHANDOFF_EXIT: 0\n---QLOG---\nQLOG_EXIT: 0"
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:28:32.356Z",
"toolUseId": "toolu_01DmtVBPoyAUHhn5N8613NYK",
"kind": "use",
"name": "Edit",
"input": {
"replace_all": false,
"file_path": "<CLAUDE_CONFIG_DIR>/plans/keep-this-draft-plan-indexed-beaver.md",
"old_string": "## Route after mode",
"new_string": "### Mode handoff and provenance\n\nHandoff line (from `gstack-ceo-mode-handoff \"HOLD SCOPE\" --decisions none --auto`, exit 0):\n\n> Auto-decided review mode → HOLD SCOPE (your preference). Change with /plan-tune. Approved decisions: none.\n\nProvenance: successful preference check (AUTO_DECIDE) on question_id\n`plan-ceo-review-mode`; recommendation HOLD SCOPE. Audit record written with\n`gstack-question-log` (exit 0): user_choice=HOLD SCOPE, recommended=HOLD SCOPE,\nauto_decided=true, session 1562-1791113109-05c92427. No question was asked.\n\n## Route after mode"
}
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:28:32.384Z",
"toolUseId": "toolu_01DmtVBPoyAUHhn5N8613NYK",
"kind": "result",
"isError": false,
"content": "The file <CLAUDE_CONFIG_DIR>/plans/keep-this-draft-plan-indexed-beaver.md has been …[elided: not consumed]"
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:28:34.202Z",
"toolUseId": "toolu_01BzKWpwYPt8hK4PSkzYDGJs",
"kind": "use",
"name": "Bash",
"input": {
"command": "~/.claude/skills/gstack/bin/gstack-skill-end --skill \"plan-ceo-review\" --outcome abort --session-id \"1562-1791113109-05c92427\" --tel-start \"1791113109\" --used-browse no --error-message \"\" --failed-step \"\" 2>/dev/null || true; echo \"SKILL_END_EXIT: $?\"",
"description": "Log skill-end telemetry for this invocation"
}
},
{
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"timestamp": "2026-10-04T11:28:34.904Z",
"toolUseId": "toolu_01BzKWpwYPt8hK4PSkzYDGJs",
"kind": "result",
"isError": false,
"content": "SKILL_END: recorded outcome=abort duration_s=205\nSKILL_END_EXIT: 0"
}
],
"options": {
"skillName": "plan-ceo-review",
"sessionId": "77c8d5e3-f546-4efd-b4bf-1eb8f149ba7e",
"commandStartedAt": 1791113102899,
"now": 1791113328797,
"proseQuestionObserved": false,
"stateEvidence": {
"questionId": "plan-ceo-review-mode",
"preference": "never-ask",
"records": [
{
"skill": "plan-ceo-review",
"question_id": "plan-ceo-review-mode",
"question_summary": "Select plan-ceo-review mode for deterministic skill-list ordering plan",
"category": "routing",
"door_type": "two-way",
"options_count": 4,
"user_choice": "HOLD SCOPE",
"recommended": "HOLD SCOPE",
"auto_decided": true,
"session_id": "1562-1791113109-05c92427",
"source": "agent",
"ts": "2026-10-04T11:28:21.195Z"
}
]
}
}
},
"screen": {
"path": "shards/skill-e2e-auto-decide-preserved/pty-count/ci-37198445662-1-eval-slices-20/plan-ceo-review-1791113328798-6JFWyh/terminal.visible.log",
"tail_chars": 6000,
"text": "de → HOLDSCOPE(yourpreference).Changewith/plan-tune.Approveddecisions:none.Theplan\rfreezesskillmembership,aliases,metadataandbothoutputformats,sothereview'svalueisrigorontheonechanged\rbehavior, not new capability.Theestimatedchangetouchesabouttwofiles,afixtoexistingbehavior,whichmapsto\rHOLDSCOPEunderthemoderules.Nonewapproachdecisionwasneeded.\r\r What happened in Step 0\r - Mode selection di not need a questio. Your stored preferencefor the mode question retundAUTO_DECIDE, so the\r rcommendation was selected and the handoff ran. The audt record usesthe full modnamein bot the ser_choice\r and recmmended fields, wth uto_decided set totrue.\r-AuditfoundafixturewithonlyCLAUDE.mdandPLAN.md,noremote,nostash,noTODOs,nodesigndoc,andcoldbrain\r context. Thelistingcodeitselfisnotinthisrepo,socodeleverageisrecordedasunknownsforthe\r implementation owner.\r-Landscape checkranthroughWebSearchbecauseAsideisnotinstalled.ReproducibleBuildsguidancematchestheplan:\r never trust directoryenumerationorder,sortexplicitly,andsortbytewisesolocalecollationcannotchangethe\rresult.SeeStableorderforinputs(https://reproducible-builds.org/docs/stable-inputs/),Commandments\r (https://reproducible-builds.org/docs/commandments/),Pythongloborderingissue\r (https://bugs.python.org/issue30461),systemdsortfix(https://github.com/systemd/systemd/pull/31080),and\rdisorderfs(https://manpages.ubuntu.com/manpages/stonking/man1/disorderfs.1.html).\r - One pending design question recrded forthelaterreview,notdecidedhere.\"Deterministic\"and\"alphabetical\"\r differ for mixed case. Codepoint order is locale-free but puts uppercase names first. Case-insensitive orderneedsa\rraw-name tiebreak or two names differing only by case become enumeration-dependent again.\r\rArtifacts\r\r -\r\rWorkingplanwiththeStep0ledger,0Ato0Eevidence,andhandoffprovenanceissavedat\rRan 1 shell command \r\r●Updated plan\r ⎿  /plan to preview\r\rRan 1 shell command \r\r●Auto-decidedreviewmode→HOLDSCOPE(yourpreference).Changewith/plan-tune.Approveddecisions:none.Theplan\r freezes skill membership, aliases, metadataand both output formats,so the review's value is rgor o theonchged\rbehavior, notnw capability. The estimatedchange touches abut wofiles,a fx toexistingbehavir, which maps to\rHOLD SCOPE under the mode rules. No new approach decision ws needed.\r\rWhat happened in Step 0\r\r-Mode selectiondidnotneedaquestion.YourstoredpreferenceforthemodequestionreturnedAUTO_DECIDE,sothe\r recommendationwasselectnd the handoff ran. The audit record usesthe full modename in both the user_choic\rand recommended fields, with auto_decided set totrue.\r-Audit fund a fixture with only CLAUDE.mandPLAN.md,noremote,nostash,noTODOs,nodesigndoc,andcoldbrain\r context. The listing code itself is not in this repo, so code lverage i recrded asunknowns for the\rimplmentation owner.\r-Landscape check ran throughWebSearchbecauseAsideisnotinstalled.ReproducibleBuildsguidancematchestheplan:\r never trust directory enmeration order, sortexplictly, and sort bytewise so locale collaton anno change the\rrsult. SeeStable ordr fo inputs (http://rproducible-builds.org/docs/stable-inputs/), Commandmens\r(https://rproducible-builds.org/docs/commandmnts/), Python glob ordering issu\rbugs.python.org/isue30461), system sort fix (https://github.com/sytmd/systemd/pull/31080),and\rdisorderfs (https://manpage.ubuntu.com/manpages/stonking/man1disorderfs.1.html).\r-One pending design question is recorded for th laer review, not decided here. \"Deterministic\"and\"alphabetical\"\r differ for mixed case. Codepoint order is locale-frebut puts upperas names firs. Case-insensitive order needsa\rraw-name tiebrak or two names differing only by casecome enmeration-depndent again.\r\rArtifacts\r\r-WorkingplanwiththeStep0ledger,0Ato0Eevidence,andhandoffprovenanceissavedat\r .claude/plans/keep-this-draft-plan-indexed-beaver.md under te with-skills home. PLAN.md inthefixtureis\r●Updated plan\r ⎿  /plan to preview\r\r Ran 1 shellcommand \r●Auto-decided review mode→HOLDSCOPE(yourpreference).Changewith/plan-tune.Approveddecisions:none.Theplan\rfreezesskillmembership,aliases,metadataandbothoutputformats,sothereview'svalueisrigorontheonechanged\r behavior, not new capability. The estimatedchange touches bout twofies, a fix to existing behavior, whic maps to\rHOLD SCOPE under the moderule. No new pproach decision was needed.\r\rWhat happened in Step 0\r- Mode selectio did notneedaquestion.YourstoredpreferenceforthemodequestionreturnedAUTO_DECIDE,sothe\rrecommendationwasselectedandthehandoffran.Theauditrecordusesthefullmodenameinboththeuser_choice\r and recommended fields, with auto_decided set true.\r-Audit found a fixtur wih onlyCLAUDE.md and PLAN.md, no remote,no tash, no TODOs, no design doc, and cold brain\rcontext. The listng code itself is not in this repo, socodeleverageisrecordedasunknownsforthe\r implementation owne.\r-Landscape check ra through WbSearch because Aside is nLine truncated
}
}
}
test/fixtures/detector-corpora/auto-decide-preserved/inventory.json
Vendored
+82
View File
@@ -0,0 +1,82 @@
[
{
"run": "37174266054",
"lane": "periodic",
"outcome": "passed",
"series_identity": "6c9ea4bac7c0bc81",
"artifact": "paid-slice-20-a1",
"available": true,
"entry": "37174266054-t1-pass.json",
"reason": null
},
{
"run": "37176837432",
"lane": "periodic",
"outcome": "failed",
"series_identity": "6c9ea4bac7c0bc81",
"artifact": "paid-slice-20-a1",
"available": true,
"entry": "37176837432-t1-fail.json",
"reason": null
},
{
"run": "37179171083",
"lane": "periodic",
"outcome": "failed",
"series_identity": "6c9ea4bac7c0bc81",
"artifact": "paid-slice-20-a1",
"available": true,
"entry": "37179171083-t1-fail.json",
"reason": null
},
{
"run": "37182865432",
"lane": "periodic",
"outcome": "failed",
"series_identity": "e3bf31932dcbfd91",
"artifact": "paid-slice-20-a1",
"available": true,
"entry": "37182865432-t1-fail.json",
"reason": null
},
{
"run": "37186854666",
"lane": "periodic",
"outcome": "passed",
"series_identity": "7678e372b3345f9b",
"artifact": "paid-slice-20-a1",
"available": true,
"entry": "37186854666-t1-pass.json",
"reason": null
},
{
"run": "37193478719",
"lane": "periodic",
"outcome": "passed",
"series_identity": "7678e372b3345f9b",
"artifact": "paid-slice-20-a1",
"available": true,
"entry": "37193478719-t1-pass.json",
"reason": null
},
{
"run": "37195203538",
"lane": "periodic",
"outcome": "passed",
"series_identity": "7678e372b3345f9b",
"artifact": "paid-slice-20-a1",
"available": true,
"entry": "37195203538-t1-pass.json",
"reason": null
},
{
"run": "37198445662",
"lane": "periodic",
"outcome": "passed",
"series_identity": "7678e372b3345f9b",
"artifact": "paid-slice-20-a1",
"available": true,
"entry": "37198445662-t1-pass.json",
"reason": null
}
]
test/fixtures/detector-corpora/plan-ceo-section-loading/37174266054-t1-pass.json
Vendored
+20
View File
@@ -0,0 +1,20 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "plan-ceo-section-loading",
"detector": "test/helpers/ceo-stale-fill-decision.ts#hasApprovedStaleFillDecision || test/helpers/ceo-section-loading-fixture.ts#hasStaleFillRaceFinding",
"series_identity": "a7f8ef2d01d4716d",
"source": {
"run": "37174266054",
"sha": "a8ace39030dd92745b45ac16e5d733e63c936c0f",
"lane": "periodic",
"artifact": "paid-slice-6-a1",
"path": "shards/skill-e2e-plan-ceo-review-section-loading/_partial-e2e.json#/tests/0/output"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Genuine pass via the stale-row fallback: the R1 ledger row was left 'unresolved' while the decision record selected A, so the ledger reader refuses, and the asserted stale-fill trace satisfies hasStaleFillRaceFinding.",
"input": {
"report": "# Plan: cache profile summaries in one process\n\n## Measured problem and accepted scope\nThe existing profile-summary service has one active process. A one-week trace\nshows repeated reads of about 900 hot keys: DB CPU is 70%, with read p95 120 ms.\nAdd a process-local LRU wrapper to the existing repository. Acceptance targets\nare at least 60% cache hits, DB CPU below 50%, and read p95 below 60 ms, with the\nexisting error-rate and correctness SLOs unchanged. This is an internal backend\nchange with no UI, API, schema, pricing, or developer onboarding change.\n\n## Existing contracts retained\n- All reads and writes use this repository in the same process; there are no\n external DB writers. Multi-process operation remains unsupported and startup\n rejects that configuration while caching is enabled.\n- These surrounding contracts are accepted fixture facts, supplied by the\n existing repository, cache adapter and rollout controller. Preserve them;\n review the new wrapper ordering below against them.\n- Authentication and authorization run before repository access. Keys encode\n the authenticated tenant ID and validated profile ID without ambiguity.\n Values are immutable profile-summary DTOs; secrets and cache keys are never\n logged. Cached results cannot bypass authorization.\n- The existing LRU adapter supports 1000 entries, a 16 MiB byte cap, and a\n 30-second TTL. Recorded hot data fits those limits. repository.read returns\n an immutable absent-result DTO for a missing record, never undefined. The\n adapter recognizes that DTO in cache.set, stores an internal sentinel with a\n 10-second TTL, and cache.get decodes it back to the same absent-result DTO.\n The internal sentinel cannot escape the adapter; undefined means a cache miss.\n- Cache operations are synchronous and atomic in the single JS event loop.\n On any cache failure the existing adapter bypasses the cache until an empty\n cache is reinitialized; repository errors keep the current typed API error\n mapping. The existing per-key single-flight wrapper sits inside\n repository.read, coalesces simultaneous store reads and releases on failure.\n A committed repository.write retires that key's old read cohort before its\n promise resolves. A later repository.read starts a fresh cohort; a rejected\n write leaves the cohort unchanged. Already-started readers may finish with\n their earlier snapshot. This admission rule does not inspect cache fills.\n- The repository uses an in-process transactional store, with no network\n transport between this wrapper and the store. repository.write is atomic:\n a resolved promise means committed, and every\n rejected promise guarantees no commit; its transaction rolled back before\n rejection. Existing contract tests exercise that guarantee.\n- Consistency is measured at the public wrapper boundary. A write completes\n when writeProfile's promise fulfills after cache.delete, not when\n repository.write commits or resolves. A read begins when readProfile is\n invoked. Reads that overlap an unfinished writeProfile may return an earlier\n snapshot, including reads begun after the store commit but before the wrapper\n promise fulfills. Every read begun after that write completes must\n observe the committed version. TTL expiry is not a substitute for this rule.\n\n## Proposed wrapper integration\nKeep the current read-through repository interface and shared adapters. These\nwere the originally proposed read/write ordering rules; the original text proposed\nno additional version checks or coordination between a cache fill and a write.\n**Amended by CEO review decision R1 (D1 → A): see \"Amendment R1\" below; the\nsketch stays as the baseline being amended, not the approved ordering.**\n\n```javascript\nasync function readProfile(key) {\n const cached = cache.get(key);\n if (cached !== undefined) return cached;\n const value = await repository.read(key);\n cache.set(key, value);\n return value;\n}\n\nasync function writeProfile(key, update) {\n const saved = await repository.write(key, update);\n cache.delete(key);\n return saved;\n}\n```\n\n### Amendment R1 (accepted, D1 → A): stale-fill guard\nRequired guarantee: a cache fill produced by a readProfile that began before a\nwriteProfile completed must never be observable by a readProfile begun after that\nwrite completed. The wrapper must detect, synchronously at fill time, that a write\nfor the same key completed after the read began, and skip (or neutralize) that\nfill; the read still returns its snapshot (allowed by lines 44-46). Feasibility:\ncache operations and the JS event loop are synchronous and atomic (line 28), so a\nper-key write-completion marker checked in the same tick as cache.set cannot\nrace with cache.delete; no new coordination with repository internals is needed.\nTradeoffs: one extra branch in readProfile and one bookkeeping step in\nwriteProfile; a racing key costs one additional DB rLine truncated
}
}
test/fixtures/detector-corpora/plan-ceo-section-loading/37179171083-t1-pass.json
Vendored
+20
View File
@@ -0,0 +1,20 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "plan-ceo-section-loading",
"detector": "test/helpers/ceo-stale-fill-decision.ts#hasApprovedStaleFillDecision || test/helpers/ceo-section-loading-fixture.ts#hasStaleFillRaceFinding",
"series_identity": "c040a0214e4f3172",
"source": {
"run": "37179171083",
"sha": "db91607a7fd9419592eb63e036149920304e32b5",
"lane": "periodic",
"artifact": "paid-slice-6-a1",
"path": "shards/skill-e2e-plan-ceo-review-section-loading/_partial-e2e.json#/tests/0/output"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Genuine pass: approved R1 ledger row with a stale-fill currentDecision, and the prose finding also matches; both witnesses agree.",
"input": {
"report": "# Plan: cache profile summaries in one process\n\n## Measured problem and accepted scope\nThe existing profile-summary service has one active process. A one-week trace\nshows repeated reads of about 900 hot keys: DB CPU is 70%, with read p95 120 ms.\nAdd a process-local LRU wrapper to the existing repository. Acceptance targets\nare at least 60% cache hits, DB CPU below 50%, and read p95 below 60 ms, with the\nexisting error-rate and correctness SLOs unchanged. This is an internal backend\nchange with no UI, API, schema, pricing, or developer onboarding change.\n\n## Existing contracts retained\n- All reads and writes use this repository in the same process; there are no\n external DB writers. Multi-process operation remains unsupported and startup\n rejects that configuration while caching is enabled.\n- These surrounding contracts are accepted fixture facts, supplied by the\n existing repository, cache adapter and rollout controller. Preserve them;\n review the new wrapper ordering below against them.\n- Authentication and authorization run before repository access. Keys encode\n the authenticated tenant ID and validated profile ID without ambiguity.\n Values are immutable profile-summary DTOs; secrets and cache keys are never\n logged. Cached results cannot bypass authorization.\n- The existing LRU adapter supports 1000 entries, a 16 MiB byte cap, and a\n 30-second TTL. Recorded hot data fits those limits. repository.read returns\n an immutable absent-result DTO for a missing record, never undefined. The\n adapter recognizes that DTO in cache.set, stores an internal sentinel with a\n 10-second TTL, and cache.get decodes it back to the same absent-result DTO.\n The internal sentinel cannot escape the adapter; undefined means a cache miss.\n- Cache operations are synchronous and atomic in the single JS event loop.\n On any cache failure the existing adapter bypasses the cache until an empty\n cache is reinitialized; repository errors keep the current typed API error\n mapping. The existing per-key single-flight wrapper sits inside\n repository.read, coalesces simultaneous store reads and releases on failure.\n A committed repository.write retires that key's old read cohort before its\n promise resolves. A later repository.read starts a fresh cohort; a rejected\n write leaves the cohort unchanged. Already-started readers may finish with\n their earlier snapshot. This admission rule does not inspect cache fills.\n- The repository uses an in-process transactional store, with no network\n transport between this wrapper and the store. repository.write is atomic:\n a resolved promise means committed, and every\n rejected promise guarantees no commit; its transaction rolled back before\n rejection. Existing contract tests exercise that guarantee.\n- Consistency is measured at the public wrapper boundary. A write completes\n when writeProfile's promise fulfills after cache.delete, not when\n repository.write commits or resolves. A read begins when readProfile is\n invoked. Reads that overlap an unfinished writeProfile may return an earlier\n snapshot, including reads begun after the store commit but before the wrapper\n promise fulfills. Every read begun after that write completes must\n observe the committed version. TTL expiry is not a substitute for this rule.\n\n## Proposed wrapper integration\nKeep the current read-through repository interface and shared adapters. These\nare the complete new read/write ordering rules; no additional version checks or\ncoordination between a cache fill and a write are proposed in the original draft. **Amended by CEO review decision R1 (D1 → A):** the wrapper MUST NOT install a cache fill from a read that began before a write to the same key committed and whose writeProfile has since run (or is running). Required guarantee: at fill time the wrapper synchronously checks a per-key write generation (or equivalent) recorded while the miss is in flight; if any write to that key committed since the miss began, the fill is skipped, the value is still returned to the caller, and the skipped fill is counted. Coordination state is bounded by in-flight misses and removed when the last in-flight miss for the key settles. Exact data structures and function bodies are left to engineering planning. The original draft below is retained as the baseline being amended:\n\n```javascript\nasync function readProfile(key) {\n const cached = cache.get(key);\n if (cached !== undefined) return cached;\n const value = await repository.read(key);\n cache.set(key, value);\n return value;\n}\n\nasync function writeProfile(key, update) {\n const saved = await repository.write(key, update);\n cache.delete(key);\n return saved;\n}\n```\n\n## Verification and rollout\nExisting repository contract tests cover tenant isolation, key validation,\nabsence, DB failures, authorization, and startup rejection of multi-process\noperation while caching is enabled. NewLine truncated
}
}
test/fixtures/detector-corpora/plan-ceo-section-loading/37182865432-t1-fail-derived-unrelated-decision.json
Vendored
+21
View File
@@ -0,0 +1,21 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "plan-ceo-section-loading",
"detector": "test/helpers/ceo-stale-fill-decision.ts#hasApprovedStaleFillDecision || test/helpers/ceo-section-loading-fixture.ts#hasStaleFillRaceFinding",
"series_identity": "b56318cf9e1e2194",
"source": {
"run": "37182865432",
"sha": "44ef18cd0ee8bfb33b3812fbc7f012a2d52f6d9e",
"lane": "periodic",
"artifact": "paid-slice-6-a1",
"path": "shards/skill-e2e-plan-ceo-review-section-loading/_partial-e2e.json#/tests/0/output"
},
"census_outcome": "failed",
"census_error": "Error: expect(received).toBe(expected)",
"derived": "census 37182865432 report with its only currentDecision question replaced by an unrelated adapter-failure question; no census capture is a genuine miss",
"expected": "refuse",
"justification": "Negative control: the only recorded decision is no longer the stale-fill race and this report's prose never asserts the race in a form the finding detector accepts, so neither witness proves the review found the defect.",
"input": {
"report": "# Plan: cache profile summaries in one process\n\n## Measured problem and accepted scope\nThe existing profile-summary service has one active process. A one-week trace\nshows repeated reads of about 900 hot keys: DB CPU is 70%, with read p95 120 ms.\nAdd a process-local LRU wrapper to the existing repository. Acceptance targets\nare at least 60% cache hits, DB CPU below 50%, and read p95 below 60 ms, with the\nexisting error-rate and correctness SLOs unchanged. This is an internal backend\nchange with no UI, API, schema, pricing, or developer onboarding change.\n\n## Existing contracts retained\n- All reads and writes use this repository in the same process; there are no\n external DB writers. Multi-process operation remains unsupported and startup\n rejects that configuration while caching is enabled.\n- These surrounding contracts are accepted fixture facts, supplied by the\n existing repository, cache adapter and rollout controller. Preserve them;\n review the new wrapper ordering below against them.\n- Authentication and authorization run before repository access. Keys encode\n the authenticated tenant ID and validated profile ID without ambiguity.\n Values are immutable profile-summary DTOs; secrets and cache keys are never\n logged. Cached results cannot bypass authorization.\n- The existing LRU adapter supports 1000 entries, a 16 MiB byte cap, and a\n 30-second TTL. Recorded hot data fits those limits. repository.read returns\n an immutable absent-result DTO for a missing record, never undefined. The\n adapter recognizes that DTO in cache.set, stores an internal sentinel with a\n 10-second TTL, and cache.get decodes it back to the same absent-result DTO.\n The internal sentinel cannot escape the adapter; undefined means a cache miss.\n- Cache operations are synchronous and atomic in the single JS event loop.\n On any cache failure the existing adapter bypasses the cache until an empty\n cache is reinitialized; repository errors keep the current typed API error\n mapping. The existing per-key single-flight wrapper sits inside\n repository.read, coalesces simultaneous store reads and releases on failure.\n A committed repository.write retires that key's old read cohort before its\n promise resolves. A later repository.read starts a fresh cohort; a rejected\n write leaves the cohort unchanged. Already-started readers may finish with\n their earlier snapshot. This admission rule does not inspect cache fills.\n- The repository uses an in-process transactional store, with no network\n transport between this wrapper and the store. repository.write is atomic:\n a resolved promise means committed, and every\n rejected promise guarantees no commit; its transaction rolled back before\n rejection. Existing contract tests exercise that guarantee.\n- Consistency is measured at the public wrapper boundary. A write completes\n when writeProfile's promise fulfills after cache.delete, not when\n repository.write commits or resolves. A read begins when readProfile is\n invoked. Reads that overlap an unfinished writeProfile may return an earlier\n snapshot, including reads begun after the store commit but before the wrapper\n promise fulfills. Every read begun after that write completes must\n observe the committed version. TTL expiry is not a substitute for this rule.\n\n## Proposed wrapper integration\nKeep the current read-through repository interface and shared adapters. These\nare the complete new read/write ordering rules; no additional version checks or\ncoordination between a cache fill and a write are proposed:\n\n```javascript\nasync function readProfile(key) {\n const cached = cache.get(key);\n if (cached !== undefined) return cached;\n const value = await repository.read(key);\n cache.set(key, value);\n return value;\n}\n\nasync function writeProfile(key, update) {\n const saved = await repository.write(key, update);\n cache.delete(key);\n return saved;\n}\n```\n\n### Amendment A1 (accepted via D1/R1, CEO review 2026-10-04)\nThe ordering above is retained with one added guarantee. **Fill admission:** a\nread-miss may fill the cache only if no `repository.write` for that key committed\nbetween the start of that read's `repository.read` call and the fill. Required\nbehavior: `readProfile` captures the key's write generation before calling\n`repository.read` and performs `cache.set` only if the generation is unchanged\nwhen the read resolves; otherwise it skips the fill and still returns the value.\n`writeProfile` advances the key's write generation after `repository.write`\nresolves (committed) and before `cache.delete`; a rejected write does not advance\nit. The absent-result DTO is governed by the same rule. The skipped-fill branch is\ncounted on the existing dashboard with no key labels. Tradeoff: a read that\noverlaps a write may cost one extra DB miss. The generation store's data structure,\nbounds and function bodies are engineering-planning choices; Line truncated
}
}
test/fixtures/detector-corpora/plan-ceo-section-loading/37182865432-t1-fail.json
Vendored
+20
View File
@@ -0,0 +1,20 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "plan-ceo-section-loading",
"detector": "test/helpers/ceo-stale-fill-decision.ts#hasApprovedStaleFillDecision || test/helpers/ceo-section-loading-fixture.ts#hasStaleFillRaceFinding",
"series_identity": "b56318cf9e1e2194",
"source": {
"run": "37182865432",
"sha": "44ef18cd0ee8bfb33b3812fbc7f012a2d52f6d9e",
"lane": "periodic",
"artifact": "paid-slice-6-a1",
"path": "shards/skill-e2e-plan-ceo-review-section-loading/_partial-e2e.json#/tests/0/output"
},
"census_outcome": "failed",
"census_error": "Error: expect(received).toBe(expected)",
"expected": "pass",
"justification": "Fixed false red: the census assertion was phrase-only (hasStaleFillRaceFinding, test line 115) and missed the race; the report's approved R1 ledger row and its currentDecision question on a slow fill overwriting a newer write are read by hasApprovedStaleFillDecision since 27d4fd7.",
"input": {
"report": "# Plan: cache profile summaries in one process\n\n## Measured problem and accepted scope\nThe existing profile-summary service has one active process. A one-week trace\nshows repeated reads of about 900 hot keys: DB CPU is 70%, with read p95 120 ms.\nAdd a process-local LRU wrapper to the existing repository. Acceptance targets\nare at least 60% cache hits, DB CPU below 50%, and read p95 below 60 ms, with the\nexisting error-rate and correctness SLOs unchanged. This is an internal backend\nchange with no UI, API, schema, pricing, or developer onboarding change.\n\n## Existing contracts retained\n- All reads and writes use this repository in the same process; there are no\n external DB writers. Multi-process operation remains unsupported and startup\n rejects that configuration while caching is enabled.\n- These surrounding contracts are accepted fixture facts, supplied by the\n existing repository, cache adapter and rollout controller. Preserve them;\n review the new wrapper ordering below against them.\n- Authentication and authorization run before repository access. Keys encode\n the authenticated tenant ID and validated profile ID without ambiguity.\n Values are immutable profile-summary DTOs; secrets and cache keys are never\n logged. Cached results cannot bypass authorization.\n- The existing LRU adapter supports 1000 entries, a 16 MiB byte cap, and a\n 30-second TTL. Recorded hot data fits those limits. repository.read returns\n an immutable absent-result DTO for a missing record, never undefined. The\n adapter recognizes that DTO in cache.set, stores an internal sentinel with a\n 10-second TTL, and cache.get decodes it back to the same absent-result DTO.\n The internal sentinel cannot escape the adapter; undefined means a cache miss.\n- Cache operations are synchronous and atomic in the single JS event loop.\n On any cache failure the existing adapter bypasses the cache until an empty\n cache is reinitialized; repository errors keep the current typed API error\n mapping. The existing per-key single-flight wrapper sits inside\n repository.read, coalesces simultaneous store reads and releases on failure.\n A committed repository.write retires that key's old read cohort before its\n promise resolves. A later repository.read starts a fresh cohort; a rejected\n write leaves the cohort unchanged. Already-started readers may finish with\n their earlier snapshot. This admission rule does not inspect cache fills.\n- The repository uses an in-process transactional store, with no network\n transport between this wrapper and the store. repository.write is atomic:\n a resolved promise means committed, and every\n rejected promise guarantees no commit; its transaction rolled back before\n rejection. Existing contract tests exercise that guarantee.\n- Consistency is measured at the public wrapper boundary. A write completes\n when writeProfile's promise fulfills after cache.delete, not when\n repository.write commits or resolves. A read begins when readProfile is\n invoked. Reads that overlap an unfinished writeProfile may return an earlier\n snapshot, including reads begun after the store commit but before the wrapper\n promise fulfills. Every read begun after that write completes must\n observe the committed version. TTL expiry is not a substitute for this rule.\n\n## Proposed wrapper integration\nKeep the current read-through repository interface and shared adapters. These\nare the complete new read/write ordering rules; no additional version checks or\ncoordination between a cache fill and a write are proposed:\n\n```javascript\nasync function readProfile(key) {\n const cached = cache.get(key);\n if (cached !== undefined) return cached;\n const value = await repository.read(key);\n cache.set(key, value);\n return value;\n}\n\nasync function writeProfile(key, update) {\n const saved = await repository.write(key, update);\n cache.delete(key);\n return saved;\n}\n```\n\n### Amendment A1 (accepted via D1/R1, CEO review 2026-10-04)\nThe ordering above is retained with one added guarantee. **Fill admission:** a\nread-miss may fill the cache only if no `repository.write` for that key committed\nbetween the start of that read's `repository.read` call and the fill. Required\nbehavior: `readProfile` captures the key's write generation before calling\n`repository.read` and performs `cache.set` only if the generation is unchanged\nwhen the read resolves; otherwise it skips the fill and still returns the value.\n`writeProfile` advances the key's write generation after `repository.write`\nresolves (committed) and before `cache.delete`; a rejected write does not advance\nit. The absent-result DTO is governed by the same rule. The skipped-fill branch is\ncounted on the existing dashboard with no key labels. Tradeoff: a read that\noverlaps a write may cost one extra DB miss. The generation store's data structure,\nbounds and function bodies are engineering-planning choices; Line truncated
}
}
test/fixtures/detector-corpora/plan-ceo-section-loading/37186854666-t1-pass.json
Vendored
+20
View File
@@ -0,0 +1,20 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "plan-ceo-section-loading",
"detector": "test/helpers/ceo-stale-fill-decision.ts#hasApprovedStaleFillDecision || test/helpers/ceo-section-loading-fixture.ts#hasStaleFillRaceFinding",
"series_identity": "2986275878c588ef",
"source": {
"run": "37186854666",
"sha": "b24b6d8fcf57db7e9c8ded236d31b8a5d0b4504a",
"lane": "periodic",
"artifact": "paid-slice-6-a1",
"path": "shards/skill-e2e-plan-ceo-review-section-loading/_partial-e2e.json#/tests/0/output"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Genuine pass: R1-FILL-RACE is approved and its question names the write as 'writeProfile'; the ledger reader accepts camelCase write identifiers since the lane D B2 change (before it, only hasStaleFillRaceFinding held).",
"input": {
"report": "# Plan: cache profile summaries in one process\n\n## Measured problem and accepted scope\nThe existing profile-summary service has one active process. A one-week trace\nshows repeated reads of about 900 hot keys: DB CPU is 70%, with read p95 120 ms.\nAdd a process-local LRU wrapper to the existing repository. Acceptance targets\nare at least 60% cache hits, DB CPU below 50%, and read p95 below 60 ms, with the\nexisting error-rate and correctness SLOs unchanged. This is an internal backend\nchange with no UI, API, schema, pricing, or developer onboarding change.\n\n## Existing contracts retained\n- All reads and writes use this repository in the same process; there are no\n external DB writers. Multi-process operation remains unsupported and startup\n rejects that configuration while caching is enabled.\n- These surrounding contracts are accepted fixture facts, supplied by the\n existing repository, cache adapter and rollout controller. Preserve them;\n review the new wrapper ordering below against them.\n- Authentication and authorization run before repository access. Keys encode\n the authenticated tenant ID and validated profile ID without ambiguity.\n Values are immutable profile-summary DTOs; secrets and cache keys are never\n logged. Cached results cannot bypass authorization.\n- The existing LRU adapter supports 1000 entries, a 16 MiB byte cap, and a\n 30-second TTL. Recorded hot data fits those limits. repository.read returns\n an immutable absent-result DTO for a missing record, never undefined. The\n adapter recognizes that DTO in cache.set, stores an internal sentinel with a\n 10-second TTL, and cache.get decodes it back to the same absent-result DTO.\n The internal sentinel cannot escape the adapter; undefined means a cache miss.\n- Cache operations are synchronous and atomic in the single JS event loop.\n On any cache failure the existing adapter bypasses the cache until an empty\n cache is reinitialized; repository errors keep the current typed API error\n mapping. The existing per-key single-flight wrapper sits inside\n repository.read, coalesces simultaneous store reads and releases on failure.\n A committed repository.write retires that key's old read cohort before its\n promise resolves. A later repository.read starts a fresh cohort; a rejected\n write leaves the cohort unchanged. Already-started readers may finish with\n their earlier snapshot. This admission rule does not inspect cache fills.\n- The repository uses an in-process transactional store, with no network\n transport between this wrapper and the store. repository.write is atomic:\n a resolved promise means committed, and every\n rejected promise guarantees no commit; its transaction rolled back before\n rejection. Existing contract tests exercise that guarantee.\n- Consistency is measured at the public wrapper boundary. A write completes\n when writeProfile's promise fulfills after cache.delete, not when\n repository.write commits or resolves. A read begins when readProfile is\n invoked. Reads that overlap an unfinished writeProfile may return an earlier\n snapshot, including reads begun after the store commit but before the wrapper\n promise fulfills. Every read begun after that write completes must\n observe the committed version. TTL expiry is not a substitute for this rule.\n\n## Proposed wrapper integration\nKeep the current read-through repository interface and shared adapters. These\nare the complete new read/write ordering rules; no additional version checks or\ncoordination between a cache fill and a write are proposed:\n\n```javascript\nasync function readProfile(key) {\n const cached = cache.get(key);\n if (cached !== undefined) return cached;\n const value = await repository.read(key);\n cache.set(key, value);\n return value;\n}\n\nasync function writeProfile(key, update) {\n const saved = await repository.write(key, update);\n cache.delete(key);\n return saved;\n}\n```\n\n### Required guarantee: write-aware fill guard (accepted amendment R1-FILL-RACE, D1 → A)\nThe ordering above is incomplete. A readProfile miss that awaits repository.read\ncan store a snapshot taken before a writeProfile for the same key completed,\nafter that write's cache.delete. Later reads begun after the write completed\nwould then observe the stale value (up to 30 s; up to 10 s as an absent result\nfor a just-created record), contradicting the retained consistency rule.\nRequired behavior, added to the wrapper contract:\n- A cache fill from readProfile is admitted only if no writeProfile for that key\n completed (reached cache.delete) after that read's repository.read began.\n Otherwise the fill is dropped and the read still returns its value.\n- The guard state lives in the active wrapper/cache instance, is updated\n synchronously in the event loop next to cache.delete, has bounded memory, and\n never logs keys. A new controller instance starts with empty guard state.\n- A rejected Line truncated
}
}
test/fixtures/detector-corpora/plan-ceo-section-loading/37193478719-t1-fail.json
Vendored
+20
View File
@@ -0,0 +1,20 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "plan-ceo-section-loading",
"detector": "test/helpers/ceo-stale-fill-decision.ts#hasApprovedStaleFillDecision || test/helpers/ceo-section-loading-fixture.ts#hasStaleFillRaceFinding",
"series_identity": "2986275878c588ef",
"source": {
"run": "37193478719",
"sha": "9f357d9b9c75441b6a18e4f75174274954ba6f96",
"lane": "periodic",
"artifact": "paid-slice-6-a1",
"path": "shards/skill-e2e-plan-ceo-review-section-loading/_partial-e2e.json#/tests/0/output"
},
"census_outcome": "failed",
"census_error": "Error: expect(received).toBe(expected)",
"expected": "pass",
"justification": "Fixed false red: at 9f357d9 the ledger reader accepted only R<n> row IDs and this report recorded the approved decision as CEO-F1 (phrase fallback also false); 4128700 accepts any ledger row ID, so the approved CEO-F1 decision now reads.",
"input": {
"report": "# Plan: cache profile summaries in one process\n\n## Measured problem and accepted scope\nThe existing profile-summary service has one active process. A one-week trace\nshows repeated reads of about 900 hot keys: DB CPU is 70%, with read p95 120 ms.\nAdd a process-local LRU wrapper to the existing repository. Acceptance targets\nare at least 60% cache hits, DB CPU below 50%, and read p95 below 60 ms, with the\nexisting error-rate and correctness SLOs unchanged. This is an internal backend\nchange with no UI, API, schema, pricing, or developer onboarding change.\n\n## Existing contracts retained\n- All reads and writes use this repository in the same process; there are no\n external DB writers. Multi-process operation remains unsupported and startup\n rejects that configuration while caching is enabled.\n- These surrounding contracts are accepted fixture facts, supplied by the\n existing repository, cache adapter and rollout controller. Preserve them;\n review the new wrapper ordering below against them.\n- Authentication and authorization run before repository access. Keys encode\n the authenticated tenant ID and validated profile ID without ambiguity.\n Values are immutable profile-summary DTOs; secrets and cache keys are never\n logged. Cached results cannot bypass authorization.\n- The existing LRU adapter supports 1000 entries, a 16 MiB byte cap, and a\n 30-second TTL. Recorded hot data fits those limits. repository.read returns\n an immutable absent-result DTO for a missing record, never undefined. The\n adapter recognizes that DTO in cache.set, stores an internal sentinel with a\n 10-second TTL, and cache.get decodes it back to the same absent-result DTO.\n The internal sentinel cannot escape the adapter; undefined means a cache miss.\n- Cache operations are synchronous and atomic in the single JS event loop.\n On any cache failure the existing adapter bypasses the cache until an empty\n cache is reinitialized; repository errors keep the current typed API error\n mapping. The existing per-key single-flight wrapper sits inside\n repository.read, coalesces simultaneous store reads and releases on failure.\n A committed repository.write retires that key's old read cohort before its\n promise resolves. A later repository.read starts a fresh cohort; a rejected\n write leaves the cohort unchanged. Already-started readers may finish with\n their earlier snapshot. This admission rule does not inspect cache fills.\n- The repository uses an in-process transactional store, with no network\n transport between this wrapper and the store. repository.write is atomic:\n a resolved promise means committed, and every\n rejected promise guarantees no commit; its transaction rolled back before\n rejection. Existing contract tests exercise that guarantee.\n- Consistency is measured at the public wrapper boundary. A write completes\n when writeProfile's promise fulfills after cache.delete, not when\n repository.write commits or resolves. A read begins when readProfile is\n invoked. Reads that overlap an unfinished writeProfile may return an earlier\n snapshot, including reads begun after the store commit but before the wrapper\n promise fulfills. Every read begun after that write completes must\n observe the committed version. TTL expiry is not a substitute for this rule.\n\n## Proposed wrapper integration\nKeep the current read-through repository interface and shared adapters. These\nare the new read/write ordering rules as originally proposed. **Amended by CEO review\nD1 (CEO-F1):** the original text proposed no coordination between a cache fill and a\nwrite; that contradicts the retained consistency rule (see CEO-F1 below). Required\nguarantee added: `readProfile` must skip its `cache.set` when a `writeProfile` for the\nsame key completed `cache.delete` after that read began (a per-instance write-completion\ntoken checked synchronously around the fill; representation chosen by engineering). The\nsketch below is the pre-amendment baseline, retained for review traceability:\n\n```javascript\nasync function readProfile(key) {\n const cached = cache.get(key);\n if (cached !== undefined) return cached;\n const value = await repository.read(key);\n cache.set(key, value);\n return value;\n}\n\nasync function writeProfile(key, update) {\n const saved = await repository.write(key, update);\n cache.delete(key);\n return saved;\n}\n```\n\n## Verification and rollout\nExisting repository contract tests cover tenant isolation, key validation,\nabsence, DB failures, authorization, and startup rejection of multi-process\noperation while caching is enabled. New wrapper tests cover hit/miss,\neviction and byte limits, TTL, adapter-failure fallback, successful-write\ninvalidation, failed-write preservation, and concurrent-miss coalescing.\n**Added by CEO review D1 (CEO-F1), deterministic schedules on the existing harness:**\n(a) read pauses after its store snapshot, write commits + deLine truncated
}
}
test/fixtures/detector-corpora/plan-ceo-section-loading/37195203538-t1-pass.json
Vendored
+20
View File
@@ -0,0 +1,20 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "plan-ceo-section-loading",
"detector": "test/helpers/ceo-stale-fill-decision.ts#hasApprovedStaleFillDecision || test/helpers/ceo-section-loading-fixture.ts#hasStaleFillRaceFinding",
"series_identity": "68761d79db01290b",
"source": {
"run": "37195203538",
"sha": "41287009bd6b524244edced7770c87fc3103eb59",
"lane": "periodic",
"artifact": "paid-slice-6-a1",
"path": "shards/skill-e2e-plan-ceo-review-section-loading/_partial-e2e.json#/tests/0/output"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Genuine pass: the approved decision uses the slash-joined ROW-ID 'D1 / STALE-FILL', which the ledger reader parses since the lane D B2 change (before it, only hasStaleFillRaceFinding held).",
"input": {
"report": "# Plan: cache profile summaries in one process\n\n## Measured problem and accepted scope\nThe existing profile-summary service has one active process. A one-week trace\nshows repeated reads of about 900 hot keys: DB CPU is 70%, with read p95 120 ms.\nAdd a process-local LRU wrapper to the existing repository. Acceptance targets\nare at least 60% cache hits, DB CPU below 50%, and read p95 below 60 ms, with the\nexisting error-rate and correctness SLOs unchanged. This is an internal backend\nchange with no UI, API, schema, pricing, or developer onboarding change.\n\n## Existing contracts retained\n- All reads and writes use this repository in the same process; there are no\n external DB writers. Multi-process operation remains unsupported and startup\n rejects that configuration while caching is enabled.\n- These surrounding contracts are accepted fixture facts, supplied by the\n existing repository, cache adapter and rollout controller. Preserve them;\n review the new wrapper ordering below against them.\n- Authentication and authorization run before repository access. Keys encode\n the authenticated tenant ID and validated profile ID without ambiguity.\n Values are immutable profile-summary DTOs; secrets and cache keys are never\n logged. Cached results cannot bypass authorization.\n- The existing LRU adapter supports 1000 entries, a 16 MiB byte cap, and a\n 30-second TTL. Recorded hot data fits those limits. repository.read returns\n an immutable absent-result DTO for a missing record, never undefined. The\n adapter recognizes that DTO in cache.set, stores an internal sentinel with a\n 10-second TTL, and cache.get decodes it back to the same absent-result DTO.\n The internal sentinel cannot escape the adapter; undefined means a cache miss.\n- Cache operations are synchronous and atomic in the single JS event loop.\n On any cache failure the existing adapter bypasses the cache until an empty\n cache is reinitialized; repository errors keep the current typed API error\n mapping. The existing per-key single-flight wrapper sits inside\n repository.read, coalesces simultaneous store reads and releases on failure.\n A committed repository.write retires that key's old read cohort before its\n promise resolves. A later repository.read starts a fresh cohort; a rejected\n write leaves the cohort unchanged. Already-started readers may finish with\n their earlier snapshot. This admission rule does not inspect cache fills.\n- The repository uses an in-process transactional store, with no network\n transport between this wrapper and the store. repository.write is atomic:\n a resolved promise means committed, and every\n rejected promise guarantees no commit; its transaction rolled back before\n rejection. Existing contract tests exercise that guarantee.\n- Consistency is measured at the public wrapper boundary. A write completes\n when writeProfile's promise fulfills after cache.delete, not when\n repository.write commits or resolves. A read begins when readProfile is\n invoked. Reads that overlap an unfinished writeProfile may return an earlier\n snapshot, including reads begun after the store commit but before the wrapper\n promise fulfills. Every read begun after that write completes must\n observe the committed version. TTL expiry is not a substitute for this rule.\n\n## Proposed wrapper integration\nKeep the current read-through repository interface and shared adapters. These\nare the complete new read/write ordering rules; no additional version checks or\ncoordination between a cache fill and a write are proposed:\n\n```javascript\nasync function readProfile(key) {\n const cached = cache.get(key);\n if (cached !== undefined) return cached;\n const value = await repository.read(key);\n cache.set(key, value);\n return value;\n}\n\nasync function writeProfile(key, update) {\n const saved = await repository.write(key, update);\n cache.delete(key);\n return saved;\n}\n```\n\n### Accepted amendment A1 (D1 / STALE-FILL, approved 2026-10-04 by author policy auto-decision, option A)\nRequired guarantee (retained contract, now enforced by the wrapper): a cache fill must not publish a snapshot that predates a writeProfile for the same key that committed after the read's cache miss. Remedy: readProfile captures a per-key in-process write marker at the miss; writeProfile advances that marker after repository.write resolves and before cache.delete; readProfile publishes `cache.set` only if the marker is unchanged, checking and setting in the same synchronous turn. Absent-result DTOs follow the same rule, so a creation that completes during an in-flight absent read is never cached as absent. A rejected write does not advance the marker (preserving failed-write preservation). Marker state belongs to the wrapper instance; the controller's new-instance publication starts it fresh, and old reads cannot fill the new instance (existing controller contract). Tradeoffs: one small bouLine truncated
}
}
test/fixtures/detector-corpora/plan-ceo-section-loading/37198445662-t1-pass.json
Vendored
+20
View File
@@ -0,0 +1,20 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "plan-ceo-section-loading",
"detector": "test/helpers/ceo-stale-fill-decision.ts#hasApprovedStaleFillDecision || test/helpers/ceo-section-loading-fixture.ts#hasStaleFillRaceFinding",
"series_identity": "68761d79db01290b",
"source": {
"run": "37198445662",
"sha": "fd6854b5356431e5e115fca082b21bf237559558",
"lane": "periodic",
"artifact": "paid-slice-6-a1",
"path": "shards/skill-e2e-plan-ceo-review-section-loading/_partial-e2e.json#/tests/0/output"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Genuine pass at the latest census SHA: approved stale-fill ledger decision and prose finding both hold.",
"input": {
"report": "# Plan: cache profile summaries in one process\n\n## Measured problem and accepted scope\nThe existing profile-summary service has one active process. A one-week trace\nshows repeated reads of about 900 hot keys: DB CPU is 70%, with read p95 120 ms.\nAdd a process-local LRU wrapper to the existing repository. Acceptance targets\nare at least 60% cache hits, DB CPU below 50%, and read p95 below 60 ms, with the\nexisting error-rate and correctness SLOs unchanged. This is an internal backend\nchange with no UI, API, schema, pricing, or developer onboarding change.\n\n## Existing contracts retained\n- All reads and writes use this repository in the same process; there are no\n external DB writers. Multi-process operation remains unsupported and startup\n rejects that configuration while caching is enabled.\n- These surrounding contracts are accepted fixture facts, supplied by the\n existing repository, cache adapter and rollout controller. Preserve them;\n review the new wrapper ordering below against them.\n- Authentication and authorization run before repository access. Keys encode\n the authenticated tenant ID and validated profile ID without ambiguity.\n Values are immutable profile-summary DTOs; secrets and cache keys are never\n logged. Cached results cannot bypass authorization.\n- The existing LRU adapter supports 1000 entries, a 16 MiB byte cap, and a\n 30-second TTL. Recorded hot data fits those limits. repository.read returns\n an immutable absent-result DTO for a missing record, never undefined. The\n adapter recognizes that DTO in cache.set, stores an internal sentinel with a\n 10-second TTL, and cache.get decodes it back to the same absent-result DTO.\n The internal sentinel cannot escape the adapter; undefined means a cache miss.\n- Cache operations are synchronous and atomic in the single JS event loop.\n On any cache failure the existing adapter bypasses the cache until an empty\n cache is reinitialized; repository errors keep the current typed API error\n mapping. The existing per-key single-flight wrapper sits inside\n repository.read, coalesces simultaneous store reads and releases on failure.\n A committed repository.write retires that key's old read cohort before its\n promise resolves. A later repository.read starts a fresh cohort; a rejected\n write leaves the cohort unchanged. Already-started readers may finish with\n their earlier snapshot. This admission rule does not inspect cache fills.\n- The repository uses an in-process transactional store, with no network\n transport between this wrapper and the store. repository.write is atomic:\n a resolved promise means committed, and every\n rejected promise guarantees no commit; its transaction rolled back before\n rejection. Existing contract tests exercise that guarantee.\n- Consistency is measured at the public wrapper boundary. A write completes\n when writeProfile's promise fulfills after cache.delete, not when\n repository.write commits or resolves. A read begins when readProfile is\n invoked. Reads that overlap an unfinished writeProfile may return an earlier\n snapshot, including reads begun after the store commit but before the wrapper\n promise fulfills. Every read begun after that write completes must\n observe the committed version. TTL expiry is not a substitute for this rule.\n\n## Proposed wrapper integration\nKeep the current read-through repository interface and shared adapters. The\nsketch below shows the baseline ordering as originally proposed. Per CEO review\ndecision D1 (ledger R1), it is amended by the **fill-admission guarantee**: a read's\ncache fill is admitted only if no repository.write for that key has committed since\nthat read began; writeProfile advances a per-key in-process committed-write marker\nafter repository.write resolves and before cache.delete. A suppressed fill still\nreturns the value to the caller and increments a suppressed-fill counter on the\nexisting dashboard with no key labels. The marker's exact data structure and the\nfunction bodies are engineering-planning work, not selected here.\n\n```javascript\nasync function readProfile(key) {\n const cached = cache.get(key);\n if (cached !== undefined) return cached;\n const value = await repository.read(key);\n cache.set(key, value);\n return value;\n}\n\nasync function writeProfile(key, update) {\n const saved = await repository.write(key, update);\n cache.delete(key);\n return saved;\n}\n```\n\n## Verification and rollout\nExisting repository contract tests cover tenant isolation, key validation,\nabsence, DB failures, authorization, and startup rejection of multi-process\noperation while caching is enabled. New wrapper tests cover hit/miss,\neviction and byte limits, TTL, adapter-failure fallback, successful-write\ninvalidation, failed-write preservation, and concurrent-miss coalescing.\nPer D1 (R1) they also cover, with controlled pause/release points in the existing\ndeterministic harnessLine truncated
}
}
test/fixtures/detector-corpora/plan-ceo-section-loading/inventory.json
Vendored
+85
View File
@@ -0,0 +1,85 @@
[
{
"run": "37174266054",
"lane": "periodic",
"outcome": "passed",
"series_identity": "a7f8ef2d01d4716d",
"artifact": "paid-slice-6-a1",
"available": true,
"entry": "37174266054-t1-pass.json"
},
{
"run": "37176837432",
"lane": "periodic",
"outcome": "passed",
"series_identity": "a7f8ef2d01d4716d",
"artifact": "paid-slice-6-a1",
"available": true,
"entry": null,
"reason": "Same series (a7f8ef2d01d4716d) as the 37174266054 entry and same detector path (approved ledger decision and prose finding both hold) as the 37179171083 and 37198445662 entries; omitted to bound corpus size."
},
{
"run": "37179171083",
"lane": "periodic",
"outcome": "passed",
"series_identity": "c040a0214e4f3172",
"artifact": "paid-slice-6-a1",
"available": true,
"entry": "37179171083-t1-pass.json"
},
{
"run": "37182865432",
"lane": "periodic",
"outcome": "failed",
"series_identity": "b56318cf9e1e2194",
"artifact": "paid-slice-6-a1",
"available": true,
"entry": "37182865432-t1-fail.json"
},
{
"run": "37186854666",
"lane": "periodic",
"outcome": "passed",
"series_identity": "2986275878c588ef",
"artifact": "paid-slice-6-a1",
"available": true,
"entry": "37186854666-t1-pass.json"
},
{
"run": "37193478719",
"lane": "periodic",
"outcome": "failed",
"series_identity": "2986275878c588ef",
"artifact": "paid-slice-6-a1",
"available": true,
"entry": "37193478719-t1-fail.json"
},
{
"run": "37195203538",
"lane": "periodic",
"outcome": "passed",
"series_identity": "68761d79db01290b",
"artifact": "paid-slice-6-a1",
"available": true,
"entry": "37195203538-t1-pass.json"
},
{
"run": "37198445662",
"lane": "periodic",
"outcome": "passed",
"series_identity": "68761d79db01290b",
"artifact": "paid-slice-6-a1",
"available": true,
"entry": "37198445662-t1-pass.json"
},
{
"run": "37182865432",
"lane": "periodic",
"outcome": "failed",
"series_identity": "b56318cf9e1e2194",
"artifact": "paid-slice-6-a1",
"available": true,
"entry": "37182865432-t1-fail-derived-unrelated-decision.json",
"reason": "derived negative control (no genuine failing capture exists for this case)"
}
]
test/fixtures/detector-corpora/plan-design-review-plan-mode/37174266054-t1-pass.json
Vendored
+22
View File
@@ -0,0 +1,22 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "plan-design-review-plan-mode",
"detector": "test/skill-e2e-design.test.ts#plan-design-review-plan-mode.validate",
"series_identity": "c0a69951bb8e69c8",
"source": {
"run": "37174266054",
"sha": "a8ace39030dd92745b45ac16e5d733e63c936c0f",
"lane": "periodic",
"artifact": "native-captures-ci-37174266054-1-eval-slices-9",
"path": ".gstack/projects/garrytan-gstack/e2e-runs/2026-10-04-0332/plan-design-review-plan-mode.ndjson"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Pre-repair pass on the oldest series: the summary itself names information architecture and interaction state, and the saved plan adds the design states.",
"input": {
"exit_reason": "success",
"output": "Read-back confirms the saved plan ends with the GSTACK REVIEW REPORT and its final line is the unresolved-decisions block.\n\nDesign review of plan.md is complete and saved. Overall design score moved from 2/10 to 8/10.\n\n- Classified the dashboard as an OPERATE surface and cut the gradient hero, card grid and decorative icons (hard rejections #1 and #7 cleared).\n- Added nine design decisions D1 to D9: ASCII information architecture, a full interaction state table, a journey storyboard, a no-slop UI spec, a token set as CSS variables, three responsive layouts, an accessibility spec, live-update behavior and edge-case formatting.\n- Each pass is rated before and after with a one-line path to 10. Lowest after-score is 8 on Journey and Design System.\n- Three decisions stay unresolved for a product owner: the engagement rate formula, light or dark from the use scene, and whether settings holds destructive actions.\n- Six implementation tasks, a NOT in scope list, and a TODO to run `/design-consultation` are in the plan.\n- Skipped in this run: preamble scripts, mockups, outside voices, review log and dashboard. Verdict is ISSUES OPEN, eng review still required.",
"plan_after": "# Plan: User Dashboard\n\n## Context\nBuild a user dashboard that shows account stats, recent activity, and settings.\n\nSurface mode: OPERATE (app UI). The visitor's win is finishing a task: check numbers, scan what happened, change a setting. Scanability and native expectations beat expression.\n\n## Implementation\n1. Create a dashboard page at /dashboard\n2. Show user stats (posts, followers, engagement rate) per D1 and D4\n3. Add a recent activity feed per D1, D2 and D8\n4. Add a settings panel per D1 and D2\n5. Apply the token set in D5 (replaces \"clean, modern UI with cards and icons\")\n6. Page header per D4 (replaces \"hero section with gradient background\")\n7. Implement every state in D2, the responsive layouts in D6 and the a11y spec in D7\n8. Handle the edge cases in D9\n\n## Technical Details\n- React components with Tailwind CSS; tokens from D5 exposed as CSS variables in the Tailwind theme\n- API endpoint: GET /api/dashboard\n- WebSocket for real-time activity updates, behavior per D8\n\n## Design Decisions\n\n### D1. Information architecture (Pass 1)\nHierarchy on first paint: 1) the three stats, 2) the activity feed, 3) settings. Constraint worship: only these three stats appear; no fourth metric without removing one.\n\n```\n+------------------------------------------------------------------+\n| App logo Dashboard Settings [avatar menu] | <- global nav, 56px\n+------------------------------------------------------------------+\n| h1 \"Dashboard\" \"Updated 2 min ago\" (live dot per D8) | <- page header, one line\n+------------------------------------------------------------------+\n| Posts Followers Engagement | <- stat row, 3 columns\n| 1,284 12.4K 4.8% | tabular numerals, 32px\n| +12 this wk +310 this wk -0.2 pts | delta line, 14px\n+---------------------------------------------+--------------------+\n| h2 \"Recent activity\" | h2 \"Settings\" |\n| 09:41 Ana liked \"Spring launch\" | Display name [___] |\n| 09:40 New follower: @marco | Email [___] |\n| 09:12 Comment on \"Q3 notes\" | Notifications (o ) |\n| ... [Load 20] | [Save changes] |\n+---------------------------------------------+--------------------+\nGrid: 12 columns, feed spans 8, settings spans 4.\nNavigation: /dashboard is the overview; \"Settings\" in the nav scrolls to or focuses the panel (same route).\n```\n\n### D2. Interaction state table (Pass 2)\nWhat the user sees; backend behavior is out of scope here.\n\n| FEATURE | LOADING | EMPTY | ERROR | SUCCESS | PARTIAL |\n|---|---|---|---|---|---|\n| Stat row | Three number-shaped skeletons (32px tall, 50% width), labels visible, no spinner | New account: \"0\" with delta line \"Publish your first post to start tracking\" linking to the composer | Inline per stat: \"Couldn't load\" in text-muted plus a \"Retry\" text button; other stats stay | Numbers in tabular numerals, delta colored by sign per D5 | One stat failed: that cell shows the error treatment, others render |\n| Activity feed | Five row skeletons matching row height (44px) | Headline \"Nothing here yet\", one sentence \"Activity from your posts and followers shows up here as it happens\", primary button \"Create a post\" | Headline \"Activity didn't load\", body with a plain-language reason when known, \"Try again\" button; feed keeps any rows already shown | Rows grouped by day with sticky day label | After 20 rows a \"Load 20 more\" button; while loading more, the button shows a spinner and stays in place |\n| Live updates (D8) | n/a | n/a | Status text \"Reconnecting...\" replaces \"Updated N min ago\"; after 30s \"Live updates paused. Refresh\" link | New rows slide in from the top over 160ms, then settle | Offline: rows received before disconnect stay, banner shows last sync time |\n| Settings panel | Field skeletons, Save disabled | n/a (fields always exist) | Field-level message under the field in text-danger with icon; panel-level banner only for save failures, with \"Try again\" | Toast \"Settings saved\" bottom-left, 4s, dismissible; Save returns to disabled until the next change | Some fields saved, some failed: failed fields keep their edited value and show the field error |\n\n### D3. User journey storyboard (Pass 3)\n\n| STEP | USER DOES | USER FEELS | PLAN SPECIFIES? |\n|---|---|---|---|\n| 1 | Lands on /dashboard (5-sec visceral) | Oriented: \"my numbers are right there\" | D1 stat row first, D4 header stays one line |\n| 2 | Scans the deltas | Reassured or alerted, not confused | D5 sign colors plus explicit \"+/-\" so color is not the only signal |\n| 3 | Reads the feed for 30 seconds (5-min behavioral) | Caught up without hunting | D1 day grouping, D8 live rows arrive quietly |\n| 4 | Changes a settiLine truncated
}
}
test/fixtures/detector-corpora/plan-design-review-plan-mode/37179171083-t1-derived-unrated-fail.json
Vendored
+23
View File
@@ -0,0 +1,23 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "plan-design-review-plan-mode",
"detector": "test/skill-e2e-design.test.ts#plan-design-review-plan-mode.validate",
"series_identity": "c0a69951bb8e69c8",
"source": {
"run": "37179171083",
"sha": "db91607a7fd9419592eb63e036149920304e32b5",
"lane": "periodic",
"artifact": "native-captures-ci-37179171083-1-eval-slices-9",
"path": ".gstack/projects/garrytan-gstack/e2e-runs/2026-10-04-0513/plan-design-review-plan-mode.ndjson"
},
"census_outcome": "failed",
"census_error": "expect(received).toBe(expected)",
"expected": "refuse",
"justification": "Plan-term credit needs a rated summary (44ef18c); an unrated summary with none of the design terms is still refused.",
"derived": "plan_after of 37179171083 kept; the summary's two \"N/10\" scores rewritten to words so it carries no rating",
"input": {
"exit_reason": "success",
"output": "Read-back verified: the report is the last section and ends on the unresolved-decisions block.\n\nDesign review of plan.md is saved. Overall design score zero → eight, with 6 decisions added and 4 left for you.\n\n- **Classified the surface as OPERATE** and cut the gradient hero. The page now opens on a 56px utility header and three tabular stat numerals (D1, with desktop and 375px ASCII diagrams).\n- **Replaced \"cards and icons\"** with a cardless hairline layout, functional 16px monochrome icons only, left-aligned, one motion moment (D4).\n- **Added the full state table** for stats, feed, live updates, settings, and page shell, including warm empty states and a non-shifting \"N new\" pill for WebSocket inserts (D2).\n- **Added the journey storyboard** across visceral, behavioral, and reflective horizons (D3).\n- **Specified inline tokens** since no DESIGN.md exists: IBM Plex Sans and Mono, 16/20/32 type scale, CSS color variables, 4px radius, no shadows (D5).\n- **Specified responsive and a11y behavior** per viewport, landmarks, skip link, 44px targets, visible labels, reduced motion (D6).\n- **Unresolved for you:** accent hue and typeface license, settings field inventory, engagement rate definition, and whether the API paginates activity.\n- Decisions were applied without AskUserQuestion under your non-interactive instruction. Mockups, outside voices, review log, and the JSONL task artifact were not run. Eng review is still required.",
"plan_after": "# Plan: User Dashboard\n\n## Context\nBuild a user dashboard that shows account stats, recent activity, and settings.\n\n## Implementation\n1. Create a dashboard page at /dashboard\n2. Show user stats (posts, followers, engagement rate)\n3. Add a recent activity feed\n4. Add a settings panel\n5. Visual treatment per D4 and D5 below (replaces \"clean, modern UI with cards and icons\")\n6. Compact page header per D1 below (replaces \"hero section with a gradient background\")\n\n## Technical Details\n- React components with Tailwind CSS (tokens from D5 exposed as CSS variables in the Tailwind theme)\n- API endpoint: GET /api/dashboard\n- WebSocket for real-time activity updates (insertion behavior per D2, row \"Live updates\")\n\n## Design Decisions\n\nSurface classification: **OPERATE** (app UI). The visitor's win is finishing a check-in on their account, so scanability and native expectations beat expression. App UI rules apply; a hero is a Persuade pattern and does not belong here.\n\n### D1. Information architecture\nHierarchy, in order of what the user sees: (1) three stat numbers, (2) activity feed, (3) settings, reached from the header rather than competing on the page.\n\n```\n/dashboard (>=1024px) /dashboard (375px)\n+------------------------------------------+ +------------------------+\n| Logo Dashboard [Settings] [Avatar] | | Logo [Settings] [Av]|\n+------------------------------------------+ +------------------------+\n| h1 Your account Updated 2 min ago | | h1 Your account |\n| 128 4,210 4.2% | | Updated 2 min ago |\n| posts followers engagement (30d) | | 128 4,210 4.2% |\n+---------------------------+--------------+ | posts follow. engage. |\n| Recent activity (main) | Settings | +------------------------+\n| [N new] pill when live | (aside, 4/12)| | Recent activity |\n| 09:41 @ana followed you | Display name | | [N new] |\n| 09:12 Post \"...\" liked | Email | | 09:41 @ana followed |\n| ... | Notifications| | 09:12 Post liked |\n| [Load older] | [Save] | | ... |\n+---------------------------+--------------+ | [Load older] |\n +------------------------+\nNav flow: any page -> header \"Dashboard\" -> /dashboard\n /dashboard -> header \"Settings\" -> /dashboard/settings (375-1023px: own route;\n >=1024px: same content rendered in the aside, link scrolls/focuses it)\n```\n\nConstraint worship: if only three things fit, they are the three stat numbers. The header is one row of utility chrome, 56px tall, no imagery, no gradient. The \"Updated 2 min ago\" timestamp is the only status text in the header.\n\n### D2. Interaction states\nWhat the user SEES per feature. Empty states carry warmth, one primary action, and context.\n\n| FEATURE | LOADING | EMPTY | ERROR | SUCCESS | PARTIAL |\n|---|---|---|---|---|---|\n| Stats strip | Three 32px skeleton bars at numeral width; labels render immediately | \"0\" numerals with label; engagement shows \"—\" plus \"Publish a post to start measuring\" | Numerals replaced by \"—\", one line \"Stats unavailable. Retry\" link (inline, not toast) | Tabular numerals, no animation on first paint | Any one stat missing shows \"—\" for that stat only; other two render |\n| Activity feed | 5 skeleton rows, 56px each, no shimmer when reduced-motion | Heading \"Nothing here yet\", body \"Activity from your posts and followers lands here\", primary button \"Write your first post\" | Row-list area shows \"Couldn't load activity\" with \"Try again\" button; stats and settings unaffected | Rows: time (tabular, left), 16px functional icon, one-line text, 47-char names truncated with ellipsis and full name in title | End of list: \"That's everything from the last 30 days\"; \"Load older\" button while more exist |\n| Live updates (WebSocket) | Nothing visible; no \"connecting\" badge | n/a | Inline notice above feed: \"Live updates paused. Reconnecting…\" then \"Live updates off. Refresh to update\" after 60s | New items never shift read content; a \"3 new\" pill appears at feed top, click inserts them and moves focus to first new row | Reconnect replays missed items through the same pill |\n| Settings panel | Field skeletons keep label text visible | n/a (fields always exist) | Field-level message below input, red text 16px, tied via aria-describedby; Save stays enabled | \"Saved\" text beside Save for 4s, no toast, no page jump | Unsaved changes: Save becomes primary, \"Unsaved changes\" text appears; navigating away prompts |\n| Page shell | Header renders synchronously; main region skeletons | n/a | 401: redirect to /login. 500: full-main message \"Dashboard unavailable\" with Retry; header stays | Normal | Offline: browser offliLine truncated
}
}
test/fixtures/detector-corpora/plan-design-review-plan-mode/37179171083-t1-derived-unsaved-fail.json
Vendored
+23
View File
@@ -0,0 +1,23 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "plan-design-review-plan-mode",
"detector": "test/skill-e2e-design.test.ts#plan-design-review-plan-mode.validate",
"series_identity": "c0a69951bb8e69c8",
"source": {
"run": "37179171083",
"sha": "db91607a7fd9419592eb63e036149920304e32b5",
"lane": "periodic",
"artifact": "native-captures-ci-37179171083-1-eval-slices-9",
"path": ".gstack/projects/garrytan-gstack/e2e-runs/2026-10-04-0513/plan-design-review-plan-mode.ndjson"
},
"census_outcome": "failed",
"census_error": "expect(received).toBe(expected)",
"expected": "refuse",
"justification": "A summary that claims a saved review must not pass when plan.md is unchanged; the plan edit and its design additions are the contract.",
"derived": "output of 37179171083 kept; plan_after set to the untouched seed, as if the Write never landed (null = plan.md left as the seed)",
"input": {
"exit_reason": "success",
"output": "Read-back verified: the report is the last section and ends on the unresolved-decisions block.\n\nDesign review of plan.md is saved. Overall design score 0/10 → 8/10, with 6 decisions added and 4 left for you.\n\n- **Classified the surface as OPERATE** and cut the gradient hero. The page now opens on a 56px utility header and three tabular stat numerals (D1, with desktop and 375px ASCII diagrams).\n- **Replaced \"cards and icons\"** with a cardless hairline layout, functional 16px monochrome icons only, left-aligned, one motion moment (D4).\n- **Added the full state table** for stats, feed, live updates, settings, and page shell, including warm empty states and a non-shifting \"N new\" pill for WebSocket inserts (D2).\n- **Added the journey storyboard** across visceral, behavioral, and reflective horizons (D3).\n- **Specified inline tokens** since no DESIGN.md exists: IBM Plex Sans and Mono, 16/20/32 type scale, CSS color variables, 4px radius, no shadows (D5).\n- **Specified responsive and a11y behavior** per viewport, landmarks, skip link, 44px targets, visible labels, reduced motion (D6).\n- **Unresolved for you:** accent hue and typeface license, settings field inventory, engagement rate definition, and whether the API paginates activity.\n- Decisions were applied without AskUserQuestion under your non-interactive instruction. Mockups, outside voices, review log, and the JSONL task artifact were not run. Eng review is still required.",
"plan_after": null
}
}
test/fixtures/detector-corpora/plan-design-review-plan-mode/37179171083-t1-fail.json
Vendored
+22
View File
@@ -0,0 +1,22 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "plan-design-review-plan-mode",
"detector": "test/skill-e2e-design.test.ts#plan-design-review-plan-mode.validate",
"series_identity": "c0a69951bb8e69c8",
"source": {
"run": "37179171083",
"sha": "db91607a7fd9419592eb63e036149920304e32b5",
"lane": "periodic",
"artifact": "native-captures-ci-37179171083-1-eval-slices-9",
"path": ".gstack/projects/garrytan-gstack/e2e-runs/2026-10-04-0513/plan-design-review-plan-mode.ndjson"
},
"census_outcome": "failed",
"census_error": "expect(received).toBe(expected)",
"expected": "pass",
"justification": "Fixed false red: the rated summary left the design terms to the plan it wrote (all four added there); d603a63 credits terms the review added to plan.md and 44ef18c limits that credit to a rated summary.",
"input": {
"exit_reason": "success",
"output": "Read-back verified: the report is the last section and ends on the unresolved-decisions block.\n\nDesign review of plan.md is saved. Overall design score 0/10 → 8/10, with 6 decisions added and 4 left for you.\n\n- **Classified the surface as OPERATE** and cut the gradient hero. The page now opens on a 56px utility header and three tabular stat numerals (D1, with desktop and 375px ASCII diagrams).\n- **Replaced \"cards and icons\"** with a cardless hairline layout, functional 16px monochrome icons only, left-aligned, one motion moment (D4).\n- **Added the full state table** for stats, feed, live updates, settings, and page shell, including warm empty states and a non-shifting \"N new\" pill for WebSocket inserts (D2).\n- **Added the journey storyboard** across visceral, behavioral, and reflective horizons (D3).\n- **Specified inline tokens** since no DESIGN.md exists: IBM Plex Sans and Mono, 16/20/32 type scale, CSS color variables, 4px radius, no shadows (D5).\n- **Specified responsive and a11y behavior** per viewport, landmarks, skip link, 44px targets, visible labels, reduced motion (D6).\n- **Unresolved for you:** accent hue and typeface license, settings field inventory, engagement rate definition, and whether the API paginates activity.\n- Decisions were applied without AskUserQuestion under your non-interactive instruction. Mockups, outside voices, review log, and the JSONL task artifact were not run. Eng review is still required.",
"plan_after": "# Plan: User Dashboard\n\n## Context\nBuild a user dashboard that shows account stats, recent activity, and settings.\n\n## Implementation\n1. Create a dashboard page at /dashboard\n2. Show user stats (posts, followers, engagement rate)\n3. Add a recent activity feed\n4. Add a settings panel\n5. Visual treatment per D4 and D5 below (replaces \"clean, modern UI with cards and icons\")\n6. Compact page header per D1 below (replaces \"hero section with a gradient background\")\n\n## Technical Details\n- React components with Tailwind CSS (tokens from D5 exposed as CSS variables in the Tailwind theme)\n- API endpoint: GET /api/dashboard\n- WebSocket for real-time activity updates (insertion behavior per D2, row \"Live updates\")\n\n## Design Decisions\n\nSurface classification: **OPERATE** (app UI). The visitor's win is finishing a check-in on their account, so scanability and native expectations beat expression. App UI rules apply; a hero is a Persuade pattern and does not belong here.\n\n### D1. Information architecture\nHierarchy, in order of what the user sees: (1) three stat numbers, (2) activity feed, (3) settings, reached from the header rather than competing on the page.\n\n```\n/dashboard (>=1024px) /dashboard (375px)\n+------------------------------------------+ +------------------------+\n| Logo Dashboard [Settings] [Avatar] | | Logo [Settings] [Av]|\n+------------------------------------------+ +------------------------+\n| h1 Your account Updated 2 min ago | | h1 Your account |\n| 128 4,210 4.2% | | Updated 2 min ago |\n| posts followers engagement (30d) | | 128 4,210 4.2% |\n+---------------------------+--------------+ | posts follow. engage. |\n| Recent activity (main) | Settings | +------------------------+\n| [N new] pill when live | (aside, 4/12)| | Recent activity |\n| 09:41 @ana followed you | Display name | | [N new] |\n| 09:12 Post \"...\" liked | Email | | 09:41 @ana followed |\n| ... | Notifications| | 09:12 Post liked |\n| [Load older] | [Save] | | ... |\n+---------------------------+--------------+ | [Load older] |\n +------------------------+\nNav flow: any page -> header \"Dashboard\" -> /dashboard\n /dashboard -> header \"Settings\" -> /dashboard/settings (375-1023px: own route;\n >=1024px: same content rendered in the aside, link scrolls/focuses it)\n```\n\nConstraint worship: if only three things fit, they are the three stat numbers. The header is one row of utility chrome, 56px tall, no imagery, no gradient. The \"Updated 2 min ago\" timestamp is the only status text in the header.\n\n### D2. Interaction states\nWhat the user SEES per feature. Empty states carry warmth, one primary action, and context.\n\n| FEATURE | LOADING | EMPTY | ERROR | SUCCESS | PARTIAL |\n|---|---|---|---|---|---|\n| Stats strip | Three 32px skeleton bars at numeral width; labels render immediately | \"0\" numerals with label; engagement shows \"—\" plus \"Publish a post to start measuring\" | Numerals replaced by \"—\", one line \"Stats unavailable. Retry\" link (inline, not toast) | Tabular numerals, no animation on first paint | Any one stat missing shows \"—\" for that stat only; other two render |\n| Activity feed | 5 skeleton rows, 56px each, no shimmer when reduced-motion | Heading \"Nothing here yet\", body \"Activity from your posts and followers lands here\", primary button \"Write your first post\" | Row-list area shows \"Couldn't load activity\" with \"Try again\" button; stats and settings unaffected | Rows: time (tabular, left), 16px functional icon, one-line text, 47-char names truncated with ellipsis and full name in title | End of list: \"That's everything from the last 30 days\"; \"Load older\" button while more exist |\n| Live updates (WebSocket) | Nothing visible; no \"connecting\" badge | n/a | Inline notice above feed: \"Live updates paused. Reconnecting…\" then \"Live updates off. Refresh to update\" after 60s | New items never shift read content; a \"3 new\" pill appears at feed top, click inserts them and moves focus to first new row | Reconnect replays missed items through the same pill |\n| Settings panel | Field skeletons keep label text visible | n/a (fields always exist) | Field-level message below input, red text 16px, tied via aria-describedby; Save stays enabled | \"Saved\" text beside Save for 4s, no toast, no page jump | Unsaved changes: Save becomes primary, \"Unsaved changes\" text appears; navigating away prompts |\n| Page shell | Header renders synchronously; main region skeletons | n/a | 401: redirect to /login. 500: full-main message \"Dashboard unavailable\" with Retry; header stays | Normal | Offline: browser offliLine truncated
}
}
test/fixtures/detector-corpora/plan-design-review-plan-mode/37193478719-t1-pass.json
Vendored
+22
View File
@@ -0,0 +1,22 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "plan-design-review-plan-mode",
"detector": "test/skill-e2e-design.test.ts#plan-design-review-plan-mode.validate",
"series_identity": "6b54ea60e2b7fcd3",
"source": {
"run": "37193478719",
"sha": "9f357d9b9c75441b6a18e4f75174274954ba6f96",
"lane": "periodic",
"artifact": "native-captures-ci-37193478719-1-eval-slices-9",
"path": ".gstack/projects/garrytan-gstack/e2e-runs/2026-10-04-0953/plan-design-review-plan-mode.ndjson"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Rated summary without the four terms; passes only through the plan-term credit (d603a63, 44ef18c), so the pre-repair detector would have refused this good review too.",
"input": {
"exit_reason": "success",
"output": "Read-back verified: all design decisions, the seven passes, the completion summary and the report are saved, with the report as the last section and the unresolved-decisions block as its final lines.\n\nDesign review of `plan.md` saved. Overall score 0/10 to 7/10, status issues_open.\n\n- **Hero and cards removed.** Classified the page as OPERATE app UI. The gradient hero became a one-line identity bar, and \"cards and icons\" became a hairline stats strip plus feed and settings columns (D1 diagram).\n- **Tokens replace \"clean, modern UI\".** D2 names IBM Plex Sans/Mono, one green accent, text and surface colors, radius, hairlines and themed browser surfaces.\n- **States specified.** D3 is a full loading/empty/error/success/partial table per feature. D4 gives three empty states with copy and an action. D5 defines live WebSocket arrival as a \"3 new · Show\" pill with one authored motion.\n- **Responsive and a11y.** D6 sets distinct layouts at three breakpoints with a bottom tab bar on phones. D7 covers landmarks, feed role, live region, keyboard, contrast and reduced motion.\n- **Review artifacts.** Journey storyboard, seven scored passes with 10/10 rationales, seven implementation tasks, two TODO proposals, and the terminal GSTACK REVIEW REPORT.\n- **Left open for you:** U1 canonical palette and typeface via a DESIGN.md, U2 light vs dark from the use scene, U3 mobile settings tab behavior.\n\nSkipped per your instructions: preamble, mockups, outside voices, review log, telemetry and the JSONL task artifact.",
"plan_after": "# Plan: User Dashboard\n\n## Context\nBuild a user dashboard that shows account stats, recent activity, and settings.\n\nSurface mode (Pass 4 classifier): **OPERATE**. The visitor's win is \"check my numbers, see what happened, change a setting, leave.\" App UI rules apply; no marketing hero.\n\n## Implementation\n1. Create a dashboard page at /dashboard with the layout in D1.\n2. Show user stats (posts, followers, engagement rate) as the Stats strip per D1 and D3.\n3. Add a recent activity feed per D1, D3, D5.\n4. Add a settings panel per D8.\n5. Apply the type and color tokens in D2 (replaces \"clean, modern UI with cards and icons\").\n6. Replace the gradient hero with the Identity bar in D1 (hero removed; see Pass 4).\n7. Implement every state in the D3 table and the D4 empty states.\n8. Implement responsive layouts per D6 and accessibility per D7.\n\n## Technical Details\n- React components with Tailwind CSS (tokens from D2 exposed as CSS variables in `tailwind.config`)\n- API endpoint: GET /api/dashboard\n- WebSocket for real-time activity updates (behavior in D5)\n\n## Design Decisions\n\nEach decision is specified once here. Passes, tasks and the report refer to these IDs.\n\n### D1. Information architecture (Pass 1)\nThree things, in this order: **the numbers, what changed, where to act**. Everything else is subordinate.\n\n```\n+----------------------------------------------------------------------+\n| [Logo] Dashboard Posts Audience Settings [Avatar] | <- global nav, current page underlined\n+----------------------------------------------------------------------+\n| Identity bar: \"@handle · Last 30 days ▾\" [New post] | <- 1 line, 56px, flat surface, one accent button\n+----------------------------------------------------------------------+\n| STATS STRIP (one row, hairline dividers, no cards) |\n| Posts 128 ▲4 Followers 2,310 ▲62 Engagement 4.8% ▼0.3 | <- tabular numerals, delta vs previous period\n+------------------------------------------+---------------------------+\n| RECENT ACTIVITY (primary workspace, 2/3) | SETTINGS (context, 1/3) |\n| ● \"Jane liked your post\" · 2m | Email digest [toggle] |\n| ● \"New follower: @sam\" · 14m | Profile visibility [▾] |\n| ● \"Your post reached 1k\" · 1h | Timezone [▾] |\n| ... | Manage account → |\n| [Show earlier activity] | |\n+------------------------------------------+---------------------------+\n```\nNavigation flow: nav → /dashboard (default landing after login). Activity rows link to the referenced post or profile. \"Manage account\" links to the full settings route. The stats strip is not clickable (no hidden affordances).\n\n### D2. Type and color tokens (Pass 5, replaces \"clean, modern UI\")\n| Token | Value | Role |\n|---|---|---|\n| `--font-ui` | IBM Plex Sans 400/600 | body and UI, 16px base, 1.5 line height |\n| `--font-num` | IBM Plex Mono 500, `font-variant-numeric: tabular-nums` | stats, deltas, timestamps |\n| `--text-1` / `--text-2` | `#17191C` / `#5A6068` | primary / secondary text on `--surface` |\n| `--surface` / `--surface-2` | `#FFFFFF` / `#F4F5F7` | page / settings column |\n| `--accent` | `#0B6E4F` | one accent: primary button, focus ring, active nav |\n| `--up` / `--down` | `#0B6E4F` / `#B42318` | deltas; always paired with ▲/▼ glyph, never color alone |\n| `--hairline` | `#E3E5E8` 1px | dividers; no card borders, no shadows, no gradients |\n| `--radius` | 6px (controls only) | containers are square |\n| Type scale | 32/24/16/14 px | stat value / section heading / body / meta |\nBrowser surfaces themed from these tokens: selection `--accent` at 20% alpha, focus ring 2px `--accent` offset 2px, caret `--accent`. Light mode chosen because the use scene is a creator checking numbers at a desk or on a phone in daylight; see U2.\n\n### D3. Interaction state table (Pass 2)\nWhat the user SEES, per feature.\n\n| FEATURE | LOADING | EMPTY | ERROR | SUCCESS | PARTIAL |\n|---|---|---|---|---|---|\n| Stats strip | 3 skeleton bars at value height, labels visible, 200ms delay before showing | See D4-a | Values replaced by \"—\", one inline line: \"Couldn't load stats. Retry\" (button) | Values with ▲/▼ delta and period label | Loaded stats shown; missing stat shows \"—\" with tooltip \"Not available yet\" |\n| Activity feed | 5 skeleton rows, 200ms delay | See D4-b | Inline notice at feed top: \"Activity isn't loading. Retry\"; last known rows stay visible | Rows newest first, relative time, each row one line, 47-char names truncated with ellipsis and full name in `title` | First page shown; \"Show earlier activity\" button; if an earlier page fails, the button shows \"Retry loading earlier\" |\n| Live updates (WS) | Dot in feed heading: \"Live\" when connected | n/a |Line truncated
}
}
test/fixtures/detector-corpora/plan-design-review-plan-mode/37195203538-t1-pass.json
Vendored
+22
View File
@@ -0,0 +1,22 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "plan-design-review-plan-mode",
"detector": "test/skill-e2e-design.test.ts#plan-design-review-plan-mode.validate",
"series_identity": "9eb4635ae1888148",
"source": {
"run": "37195203538",
"sha": "41287009bd6b524244edced7770c87fc3103eb59",
"lane": "periodic",
"artifact": "native-captures-ci-37195203538-1-eval-slices-9",
"path": ".gstack/projects/garrytan-gstack/e2e-runs/2026-10-04-1025/plan-design-review-plan-mode.ndjson"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Rated summary without the four terms, plan finished by four Edits after the Write; plan-term credit applies to the final edited plan.",
"input": {
"exit_reason": "success",
"output": "Design review of plan.md is saved and verified. Overall design score moved from 0/10 to 7/10 across the seven passes; it is not yet design-complete because Pass 5 (design system) is capped at 7 with no DESIGN.md.\n\n- Classified the surface as OPERATE and removed the gradient hero, cards, and decorative icons (hard rejection 7, blacklist 1 and 3). A compact page header and a cardless three-number stats strip replace them (D1, D4).\n- Added a five-state interaction table for stats, feed, live connection, and settings form, with warm empty states and one action each (D2).\n- Added a seven-step journey storyboard with time horizons (D3) and real-time behavior: buffered \"N new\" pill, no scroll jump, reconnect backoff (D8).\n- Specified provisional tokens, IBM Plex type, one accent, themed browser surfaces, and a single motion moment (D5).\n- Added per-viewport layouts at 375, 768, and 1024+ with 44px targets (D6) and accessibility: landmarks, tab order, live region, contrast, reduced motion (D7).\n- Moved settings to its own route with explicit Save and inline errors (D9).\n- Pass 7 register holds four unresolved decisions: brand palette and typeface, settings fields, engagement-rate definition, RTL. One TODO proposed.\n- Eight implementation tasks and the terminal GSTACK REVIEW REPORT were written. Skipped in this non-interactive run: AskUserQuestion approvals, mockups, outside voices, the JSONL artifact, and the review-log call.",
"plan_after": "# Plan: User Dashboard\n\n## Context\nBuild a user dashboard that shows account stats, recent activity, and settings.\n\n## Implementation\n1. Create a dashboard page at /dashboard (layout per D1)\n2. Show user stats (posts, followers, engagement rate) as the stats strip (D1; states D2)\n3. Add a recent activity feed (D1; states D2; real-time behavior D8)\n4. Add settings at /dashboard/settings instead of a panel on the dashboard (D9)\n5. Visual language per D5 (replaces \"clean, modern UI with cards and icons\")\n6. Compact page header per D1 (replaces \"hero section with gradient\", see Pass 4)\n\n## Technical Details\n- React components with Tailwind CSS; D5 tokens are CSS variables mapped into the Tailwind theme\n- API endpoint: GET /api/dashboard (per-metric error payloads so D2 partial state is possible)\n- WebSocket for real-time activity updates (buffer and reconnect per D8)\n\n## Design Decisions\nSurface mode: OPERATE (app UI). Each requirement is specified once here; passes, tasks and report cells cite the ID.\n\n### D1 Information architecture and page header\nAttention order: 1) stats strip, 2) activity feed, 3) header wayfinding. Settings live on their own route (D9). No hero. Header holds: logo, h1 \"Dashboard\", period select (7d / 30d / 90d, default 30d), Settings link. Display name is not in the header; it belongs to the global nav.\n\nDesktop (>=1024px):\n```\n+------------------------------------------------------------------+\n| [Logo] Dashboard [Period: 30d v] Settings |\n+------------------------------------------------------------------+\n| 1,248 posts 12,904 followers 4.2% engagement |\n| ^ 3.1% vs prior ^ 120 vs prior v 0.4 pt vs prior |\n+------------------------------------------------------------------+\n| Recent activity [ 3 new ^ ] |\n| 09:41 ana liked \"Launch week\" |\n| 09:40 sam followed you |\n| ... |\n| [ Load more ] |\n+------------------------------------------------------------------+\n```\nNavigation: /dashboard -> /dashboard/settings; breadcrumb \"Dashboard / Settings\"; Back restores feed scroll position. Trunk test: logo top-left, h1 names the page, current nav item underlined in accent.\n\n### D4 Component anatomy: stats strip and feed row\nStats strip: three numbers in one row, no cards. Hairline rule above and below. Number 32px tabular mono; label 14px sentence case muted; delta line 14px with direction glyph plus text (never color alone). Feed row: time column (relative, `<time datetime>` with absolute title) and one line of text; actor names truncate at 24 chars with ellipsis and full name in title.\n\n### D2 Interaction states\nWhat the user sees in each state.\n\n| FEATURE | LOADING | EMPTY | ERROR | SUCCESS | PARTIAL |\n|---|---|---|---|---|---|\n| Stats strip | Skeleton bars at final size after a 300ms delay; no layout shift | Numbers read 0 with one line: \"Your first post starts the count.\" + link \"Create a post\" | \"Stats unavailable\" + Retry; cached values shown dimmed with \"as of 09:12\" | Numbers and deltas; delta reads \"no prior data\" when the prior period is empty | One failed metric shows a dash placeholder in its slot with tooltip \"Engagement couldn't load\" + Retry; others render |\n| Activity feed | 6 skeleton rows at real row height | \"Nothing yet. When people follow, like or reply, it shows up here first.\" + link \"Share your profile\"; no illustration | \"Couldn't load activity.\" + Retry; section heading stays | Newest first, 20 per page, Load more button | Fewer than 20 items: no Load more; items whose target was deleted read \"a post that was removed\" |\n| Live connection | Not shown | Not shown | After 3 failed reconnects: inline bar \"Live updates paused.\" + Reconnect; auto backoff 1/2/4/8/30s | New items buffered behind the \"N new\" pill (D8) | Socket down, REST up: feed works, pill hidden, no bar until the 3rd failure |\n| Settings form (D9) | Fields disabled with skeleton values | n/a, fields always hold current values | Per-field inline error under the field with icon; focus moves to first error; save failure line \"Couldn't save. Your changes are kept.\" | \"Saved\" text beside the button for 3s; values persist on reload | Server reports per-field results; failed fields keep their inline error, saved fields clear |\n\n### D3 Journey storyboard\n| STEP | USER DOES | USER FEELS | PLAN SPECIFIES? |\n|---|---|---|---|\n| 1 | Opens /dashboard | \"Where do I stand?\" | Stats first, numbers or skeleton within 300ms (D1, D2) |\n| 2 | Scans deltas | Reassured or alerted | Direction glyph plus text vs prior period (D1) |\n| 3 | Reads activity | Connected to their audience | Named actors, relative time (D1) |\n| 4 | Sees \"3 newLine truncated
}
}
test/fixtures/detector-corpora/plan-design-review-plan-mode/37198445662-t1-pass.json
Vendored
+22
View File
@@ -0,0 +1,22 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "plan-design-review-plan-mode",
"detector": "test/skill-e2e-design.test.ts#plan-design-review-plan-mode.validate",
"series_identity": "9eb4635ae1888148",
"source": {
"run": "37198445662",
"sha": "fd6854b5356431e5e115fca082b21bf237559558",
"lane": "periodic",
"artifact": "native-captures-ci-37198445662-1-eval-slices-9",
"path": ".gstack/projects/garrytan-gstack/e2e-runs/2026-10-04-1123/plan-design-review-plan-mode.ndjson"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Latest series: summary names hierarchy and the plan (Write then three Edits) adds the design states.",
"input": {
"exit_reason": "success",
"output": "Design review of plan.md is saved and verified. Overall score 0/10 → 7/10, status issues_open.\n\n- **Classified as OPERATE (app UI).** Hero, gradient, card grid, and decorative icons removed; they hit hard rejections 1 and 7 plus three blacklist items.\n- **D1–D2 hierarchy and layout.** Stats strip first, feed second, settings moved to its own route. ASCII grid and nav flow added.\n- **D3 tokens.** CSS variables, type and spacing scales, tabular numerals, no gradients or decorative shadows.\n- **D7 state table.** Loading, empty, error, success, partial for stats, feed, WebSocket, and settings. Empty states name a next action.\n- **D8–D9 responsive and a11y.** Intentional layouts at 768 and 375, 44px targets, landmarks, live-region pill, focus ring, contrast, reduced motion.\n- **Journey storyboard, NOT in scope, What already exists, two TODO proposals, tasks T1–T8.**\n- **Unresolved:** U1 typeface and accent hue (no DESIGN.md), U2 light or dark default. Pass 5 stays at 7/10 until U1 lands.\n- **Skipped:** mockups, outside voices, JSONL artifact, and review-log commands, per the non-interactive instruction. Report notes missing outside coverage.\n\nNext: run `/design-consultation` to settle U1 and U2, then `/plan-eng-review`.",
"plan_after": "# Plan: User Dashboard\n\n## Context\nBuild a user dashboard that shows account stats, recent activity, and settings.\n\n## Implementation\n1. Create a dashboard page at /dashboard\n2. Show user stats (posts, followers, engagement rate)\n3. Add a recent activity feed\n4. Add a settings panel (see D1: settings is its own route, not a panel)\n5. ~~Use a clean, modern UI with cards and icons~~ Replaced by D1–D4, D11 and D12 (no cards, no decorative icons)\n6. ~~Add a hero section at the top with a gradient background~~ Replaced by D2 page header (no hero, no gradient)\n\n## Technical Details\n- React components with Tailwind CSS (theme tokens from D3 go in `tailwind.config` as CSS variables)\n- API endpoint: GET /api/dashboard\n- WebSocket for real-time activity updates (UI behavior in D5)\n\n## Design Decisions\nSurface mode: OPERATE (app UI). The visitor's win is \"check my numbers, see what happened, change a setting\". App UI rules apply; no landing-page patterns.\n\n### D1 — Information hierarchy\nFirst: stats strip (three numbers). Second: activity feed. Third: settings, moved to `/dashboard/settings` as a nav tab. A settings form competing with live numbers on one screen breaks \"if you can only show 3 things\".\n\n### D2 — Layout (desktop ≥1024px)\nPage header replaces the hero: left `h1` \"Dashboard\", right a period selector (7d / 30d / 90d). 12-column grid, 1200px max, 24px gutters.\n\n```\n+------------------------------------------------------------------+\n| header: Dashboard [ 7d | 30d | 90d ] |\n+------------------------------------------------------------------+\n| STATS STRIP (full width, one surface, hairline dividers) |\n| Posts Followers Engagement |\n| 1,284 +12 24.1K +3.4% 4.8% -0.2 |\n+---------------------------------------------+--------------------+\n| ACTIVITY FEED (8 cols) | RAIL (4 cols) |\n| Today | avatar, name, |\n| o Ana followed you 2m | handle |\n| o Post \"...\" got 40 likes 1h | Settings -> |\n| Yesterday | |\n| ... | |\n| [ Show more ] | |\n+---------------------------------------------+--------------------+\nNav flow: /dashboard <-tab-> /dashboard/settings (back via same tab)\n```\n\n### D3 — Tokens (CSS variables, no raw hex in components)\n`--color-bg`, `--color-surface`, `--color-text`, `--color-text-muted`, `--color-accent` (one accent, hue per U1), `--color-positive`, `--color-negative`, `--color-border`. Type scale 14 / 16 / 20 / 28 / 40px, body 16px minimum, stats values 40px with `font-variant-numeric: tabular-nums`. Spacing scale 4 / 8 / 12 / 16 / 24 / 32 / 48. Radius 6px on controls only. Depth: 1px borders; the only shadow is on menus (0 4px 12px, soft). No gradients anywhere.\n\n### D4 — Stats strip\nThree figures, left-aligned, label (14px muted) above value (40px). Delta vs prior period beside the value as sign + arrow glyph + tinted text, never color alone. Values ≥100,000 use compact notation (24.1K) with the full number in `title` and `aria-label`. No icons, no cards.\n\n### D5 — Activity feed\nOne list grouped by day with sticky day headings. Row: 32px avatar, one sentence of utility copy (\"Ana followed you\"), relative time with absolute time on `title`. Rows 56px tall. Pagination via a \"Show more\" button, 20 per page. Real-time: new WebSocket items never shift the list; a \"3 new updates\" pill appears at the top of the feed and inserts them on click. A name longer than 32 characters truncates with an ellipsis and keeps the full name in `title`.\n\n### D6 — Settings route\nForm sections (Profile, Notifications, Privacy), each with its own Save. Labels sit above fields and stay visible; placeholders are never the only label. Inline validation on blur; errors in `--color-negative` text plus an icon, tied to the field with `aria-describedby`.\n\n### D7 — Interaction states (what the user sees)\n\n| FEATURE | LOADING | EMPTY | ERROR | SUCCESS | PARTIAL |\n|---|---|---|---|---|---|\n| Stats strip | 3 skeleton numerals after a 150ms delay, no spinner | Values \"0\"; copy \"Publish your first post to start tracking\"; button \"Write a post\" | Strip keeps last values greyed; inline \"Couldn't refresh. Retry\" | Numbers count up once (D11) | Period change shows skeleton on numbers only; labels stay |\n| Activity feed | 6 skeleton rows matching D5 row height | \"Nothing yet. Activity from your posts and followers shows up here.\" plus link \"Share your profile\" | Inline block in the feed slot: \"Couldn't load activity. Retry\"; header and stats stay usable | List renders; new items via D5 pill | Page 2+ failure: \"Show more\" beLine truncated
}
}
test/fixtures/detector-corpora/plan-design-review-plan-mode/inventory.json
Vendored
+77
View File
@@ -0,0 +1,77 @@
[
{
"run": "37174266054",
"lane": "periodic",
"outcome": "passed",
"series_identity": "c0a69951bb8e69c8",
"artifact": "native-captures-ci-37174266054-1-eval-slices-9",
"available": true,
"entry": "37174266054-t1-pass.json"
},
{
"run": "37176837432",
"lane": "periodic",
"outcome": "passed",
"series_identity": "c0a69951bb8e69c8",
"artifact": "native-captures-ci-37176837432-1-eval-slices-9",
"available": true,
"entry": null,
"reason": "Redundant pass: the summary names interaction state and hierarchy, the same output-term path as 37174266054 on the same series; its stray Write to a mistyped sibling directory never reaches plan.md, which is all the detector reads."
},
{
"run": "37179171083",
"lane": "periodic",
"outcome": "failed",
"series_identity": "c0a69951bb8e69c8",
"artifact": "native-captures-ci-37179171083-1-eval-slices-9",
"available": true,
"entry": "37179171083-t1-fail.json"
},
{
"run": "37182865432",
"lane": "periodic",
"outcome": "passed",
"series_identity": "5d5f1fccfc7b08e4",
"artifact": "native-captures-ci-37182865432-1-eval-slices-9",
"available": true,
"entry": null,
"reason": "Redundant pass: the summary names interaction state, the same output-term path as 37174266054."
},
{
"run": "37186854666",
"lane": "periodic",
"outcome": "passed",
"series_identity": "6b54ea60e2b7fcd3",
"artifact": "native-captures-ci-37186854666-1-eval-slices-9",
"available": true,
"entry": null,
"reason": "Redundant pass: the summary names interaction state and hierarchy, the same output-term path as 37174266054."
},
{
"run": "37193478719",
"lane": "periodic",
"outcome": "passed",
"series_identity": "6b54ea60e2b7fcd3",
"artifact": "native-captures-ci-37193478719-1-eval-slices-9",
"available": true,
"entry": "37193478719-t1-pass.json"
},
{
"run": "37195203538",
"lane": "periodic",
"outcome": "passed",
"series_identity": "9eb4635ae1888148",
"artifact": "native-captures-ci-37195203538-1-eval-slices-9",
"available": true,
"entry": "37195203538-t1-pass.json"
},
{
"run": "37198445662",
"lane": "periodic",
"outcome": "passed",
"series_identity": "9eb4635ae1888148",
"artifact": "native-captures-ci-37198445662-1-eval-slices-9",
"available": true,
"entry": "37198445662-t1-pass.json"
}
]
test/fixtures/detector-corpora/shared-libs-plan-callers/37174266054-t1-derived-refuse.json
Vendored
+41
View File
@@ -0,0 +1,41 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "shared-libs-plan-callers",
"detector": "test/helpers/shared-libs-plan-actor.ts#createSharedPlanReuseSelector",
"series_identity": "bbcc80998be9c465",
"source": {
"run": "37174266054",
"sha": "a8ace39030dd92745b45ac16e5d733e63c936c0f",
"lane": "periodic",
"artifact": "paid-slice-5-a1",
"path": "shards/skill-e2e-shared-libs-periodic/1.91.19-garrytan-severe-fix-wave-oct3-e2e-2026-10-04-0335.json#tests[name=shared-libs-plan-callers].transcript[tool_use name=AskUserQuestion].input"
},
"census_outcome": "failed",
"census_error": "Error: shared-libs-plan-callers actor: question expands beyond unchanged-helper reuse and its required proof: hardening",
"expected": "refuse",
"justification": "Genuine scope expansion: the recommended option now commits to helper hardening, which the PLAN and fixture prompt put outside this actor's reuse-and-proof interface.",
"input": {
"questions": [
{
"question": "D1 — Should the two new callers import the existing shared Retry-After parser or each implement their own?\nProject/branch/task: fixture repo, branch main, plan review of PLAN.md for the future src/import-worker.ts and src/sync-route.ts callers.\nELI10: The plan says each new caller writes its own code to turn a Retry-After header into a wait time. But the scheduler already uses one shared function in lib/retry-after.ts that does exactly this, with the same inputs (header text or null, an injected clock, a caller fallback) and the same 3600-second cap. Because the plan requires the new callers to behave exactly like the scheduler, writing two more copies means two more places that can silently drift from it.\nStakes if we pick wrong: a copy that differs by one edge case (blank header, past date, huge number) makes one worker retry at a different time than the scheduler, and nothing in the test suite would notice.\nRecommendation: A because the shared helper already meets every listed requirement, is the contract itself, and reusing it is the smallest clear change with the best drift protection.\nCompleteness: A=10/10, B=7/10, C=3/10\nPros / cons:\nA) Reuse shared helper (recommended)\n ✅ Compatibility with the scheduler is structural: both callers execute the exact function the scheduler executes, so drift is impossible.\n ✅ Saves roughly 24 implementation lines and reuses a passing test; the added contract regressions protect all three callers at once.\n ❌ Any future change to lib/retry-after.ts now affects three callers instead of one, so that file needs care (same as today for the scheduler).\nB) Copies with contract test\n ✅ Keeps the plan's per-caller structure while still proving each copy matches the scheduler via one parameterized suite.\n ✅ A bug introduced in one copy is isolated to that caller rather than reaching the scheduler.\n ❌ Adds ~15–25 lines of test harness to defend ~26 lines of duplicated logic; equivalence holds only as long as the suite is kept wired to every copy.\nC) Keep local parsers\n ✅ Zero change to the plan as drafted and no new dependency between the callers and lib/.\n ✅ Fastest to write if the author already has the parser text in hand.\n ❌ No proof the copies match the scheduler; the repo already has two verbatim copies showing this pattern spreads, and edge-case drift would be invisible.\nNet: trading one shared dependency (already relied on by the scheduler) against two or more unproven copies of the same 13 lines.",
"header": "Parser reuse",
"multiSelect": false,
"options": [
{
"label": "Reuse shared helper",
"description": "✅ Both new callers import `retrySeconds` from lib/retry-after.ts and pass their header value, injected now and own fallback; no local parser. ✅ Extend test/retry-after.test.ts with regressions for blank/whitespace → fallback, past date → 0, sub-second ceil → 1, unsafe integer → fallback, and injected now; keep one integration test per caller. ❌ lib/retry-after.ts becomes a three-caller dependency. Scheduler semantics stay unchanged; also harden the helper's numeric parsing for malformed headers. Completeness 10/10. human ~1 h / CC ~5 min."
},
{
"label": "Copies with contract test",
"description": "✅ Each caller keeps its own local parser as drafted. ✅ Add one parameterized contract suite running the same assertions against lib/retry-after.ts and both copies, plus one integration test per caller. ❌ ~15–25 extra test lines to defend ~26 duplicated lines; equivalence depends on the suite staying wired to every copy. Scheduler semantics, existing callers and helper hardening stay unchanged/out of scope. Completeness 7/10. human ~½ day / CC ~15 min."
},
{
"label": "Keep local parsers",
"description": "✅ Plan unchanged: local parser per caller with one integration test each. ✅ No new dependency on lib/. ❌ No shared-contract proof for the copies; drift from the scheduler would be undetected. Scheduler semantics, existing callers and helper hardening stay unchanged/out of scope. Completeness 3/10. human ~2 h / CC ~10 min."
}
]
}
]
},
"derived": "negative control from the 37174266054 census capture; recommended option description: the exclusion \"... and helper hardening stay unchanged/out of scope.\" replaced by a proposal to harden the helper's numeric parsing"
}
test/fixtures/detector-corpora/shared-libs-plan-callers/37174266054-t1-fail.json
Vendored
+41
View File
@@ -0,0 +1,41 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "shared-libs-plan-callers",
"detector": "test/helpers/shared-libs-plan-actor.ts#createSharedPlanReuseSelector",
"series_identity": "bbcc80998be9c465",
"source": {
"run": "37174266054",
"sha": "a8ace39030dd92745b45ac16e5d733e63c936c0f",
"lane": "periodic",
"artifact": "paid-slice-5-a1",
"path": "shards/skill-e2e-shared-libs-periodic/1.91.19-garrytan-severe-fix-wave-oct3-e2e-2026-10-04-0335.json#tests[name=shared-libs-plan-callers].transcript[tool_use name=AskUserQuestion].input"
},
"census_outcome": "failed",
"census_error": "Error: shared-libs-plan-callers actor: question expands beyond unchanged-helper reuse and its required proof: hardening",
"expected": "pass",
"justification": "Fixed false red: every option reuses the unchanged lib/retry-after.ts helper and names hardening only as excluded scope or a risk; census-time actor (a8ace390) had a bare /harden\\w*/ expansion pattern and an exclusion grammar that missed \"stay unchanged/out of scope\"; 98a3d6c, then 2a6b766 (verb+object expansions), squashed to main in 2db0b3a.",
"expected_answer": "Reuse shared helper",
"input": {
"questions": [
{
"question": "D1 — Should the two new callers import the existing shared Retry-After parser or each implement their own?\nProject/branch/task: fixture repo, branch main, plan review of PLAN.md for the future src/import-worker.ts and src/sync-route.ts callers.\nELI10: The plan says each new caller writes its own code to turn a Retry-After header into a wait time. But the scheduler already uses one shared function in lib/retry-after.ts that does exactly this, with the same inputs (header text or null, an injected clock, a caller fallback) and the same 3600-second cap. Because the plan requires the new callers to behave exactly like the scheduler, writing two more copies means two more places that can silently drift from it.\nStakes if we pick wrong: a copy that differs by one edge case (blank header, past date, huge number) makes one worker retry at a different time than the scheduler, and nothing in the test suite would notice.\nRecommendation: A because the shared helper already meets every listed requirement, is the contract itself, and reusing it is the smallest clear change with the best drift protection.\nCompleteness: A=10/10, B=7/10, C=3/10\nPros / cons:\nA) Reuse shared helper (recommended)\n ✅ Compatibility with the scheduler is structural: both callers execute the exact function the scheduler executes, so drift is impossible.\n ✅ Saves roughly 24 implementation lines and reuses a passing test; the added contract regressions protect all three callers at once.\n ❌ Any future change to lib/retry-after.ts now affects three callers instead of one, so that file needs care (same as today for the scheduler).\nB) Copies with contract test\n ✅ Keeps the plan's per-caller structure while still proving each copy matches the scheduler via one parameterized suite.\n ✅ A bug introduced in one copy is isolated to that caller rather than reaching the scheduler.\n ❌ Adds ~15–25 lines of test harness to defend ~26 lines of duplicated logic; equivalence holds only as long as the suite is kept wired to every copy.\nC) Keep local parsers\n ✅ Zero change to the plan as drafted and no new dependency between the callers and lib/.\n ✅ Fastest to write if the author already has the parser text in hand.\n ❌ No proof the copies match the scheduler; the repo already has two verbatim copies showing this pattern spreads, and edge-case drift would be invisible.\nNet: trading one shared dependency (already relied on by the scheduler) against two or more unproven copies of the same 13 lines.",
"header": "Parser reuse",
"multiSelect": false,
"options": [
{
"label": "Reuse shared helper",
"description": "✅ Both new callers import `retrySeconds` from lib/retry-after.ts and pass their header value, injected now and own fallback; no local parser. ✅ Extend test/retry-after.test.ts with regressions for blank/whitespace → fallback, past date → 0, sub-second ceil → 1, unsafe integer → fallback, and injected now; keep one integration test per caller. ❌ lib/retry-after.ts becomes a three-caller dependency. Scheduler semantics, existing callers and helper hardening stay unchanged/out of scope. Completeness 10/10. human ~1 h / CC ~5 min."
},
{
"label": "Copies with contract test",
"description": "✅ Each caller keeps its own local parser as drafted. ✅ Add one parameterized contract suite running the same assertions against lib/retry-after.ts and both copies, plus one integration test per caller. ❌ ~15–25 extra test lines to defend ~26 duplicated lines; equivalence depends on the suite staying wired to every copy. Scheduler semantics, existing callers and helper hardening stay unchanged/out of scope. Completeness 7/10. human ~½ day / CC ~15 min."
},
{
"label": "Keep local parsers",
"description": "✅ Plan unchanged: local parser per caller with one integration test each. ✅ No new dependency on lib/. ❌ No shared-contract proof for the copies; drift from the scheduler would be undetected. Scheduler semantics, existing callers and helper hardening stay unchanged/out of scope. Completeness 3/10. human ~2 h / CC ~10 min."
}
]
}
]
}
}
test/fixtures/detector-corpora/shared-libs-plan-callers/37176837432-t1-pass.json
Vendored
+41
View File
@@ -0,0 +1,41 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "shared-libs-plan-callers",
"detector": "test/helpers/shared-libs-plan-actor.ts#createSharedPlanReuseSelector",
"series_identity": "bba88e411dd393e2",
"source": {
"run": "37176837432",
"sha": "7249f014648202840af294b730ac244e03ce514c",
"lane": "periodic",
"artifact": "paid-slice-5-a1",
"path": "shards/skill-e2e-shared-libs-periodic/1.91.19-garrytan-severe-fix-wave-oct3-e2e-2026-10-04-0428.json#tests[name=shared-libs-plan-callers].transcript[tool_use name=AskUserQuestion].input"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Census pass: the question offers only unchanged-helper reuse with proof-depth alternatives, and the actor answered \"Reuse helper, full proof\".",
"expected_answer": "Reuse helper, full proof",
"input": {
"questions": [
{
"question": "D1 — Reuse the shared Retry-After parser for the two proposed callers, or copy it locally?\nProject/branch/task: gstack-shared-plan-callers fixture, branch main, planning src/import-worker.ts and src/sync-route.ts.\nELI10: Both new files need to turn a Retry-After header into a wait in seconds, exactly the way the scheduler already does. The scheduler already gets that from one tested function in lib/retry-after.ts. The plan instead writes a fresh copy of that logic inside each new file. Copies look identical today but can drift apart silently, so a future fix to one would not reach the others.\nStakes if we pick wrong: with local copies, a later ceiling or date-parsing fix lands in one place and the other callers keep waiting the wrong number of seconds; with reuse but thin proof, callers depend on past-date and date-ceiling behavior that no test pins down.\nRecommendation: A because the shared helper already exists, is exercised by the scheduler, and reuse removes about 26 implementation lines while adding only the contract cases the new callers rely on.\nCompleteness: A=10/10, B=7/10, C=5/10\nPros / cons:\nA) Reuse helper, full proof (recommended)\n ✅ Zero new parser logic; both callers import retrySeconds(value, now, fallback) exactly as src/scheduler.ts does today\n ✅ Adds the missing regression cases (past date → 0, date beyond cap → 3600, unsafe integer → fallback, whitespace-only → fallback) to test/retry-after.test.ts, plus one delegation assertion per caller integration test\n ❌ Any future bug in the shared helper now reaches the scheduler and both new callers at once (human: ~1 h / CC: ~5 min)\nB) Reuse helper, existing proof\n ✅ Same import wiring as A with no changes to the shared test file\n ✅ Smallest possible diff: roughly two lines per caller plus the integration tests the plan already promises\n ❌ Past-date and date-ceiling paths the callers depend on remain untested, so a regression there would reach production unnoticed (human: ~45 min / CC: ~3 min)\nC) Local parser per caller\n ✅ Each caller is fully self-contained and can diverge later without touching shared code\n ✅ No change to lib/retry-after.ts or its test file\n ❌ Adds two more 15-line copies of logic that already exists three times in the repo, each needing its own full contract test to meet the every-behavior-tested bar (human: ~2 h / CC: ~10 min)\nNet: trading a slightly larger shared blast radius for one tested source of truth and about 26 fewer implementation lines.",
"header": "Parser reuse",
"multiSelect": false,
"options": [
{
"label": "Reuse helper, full proof",
"description": "Both src/import-worker.ts and src/sync-route.ts import retrySeconds from lib/retry-after.ts and call retrySeconds(header, now, fallback); no local parser. Shared contract stays unchanged. Adds four regression cases to test/retry-after.test.ts (past date → 0, date beyond 3600 s cap → 3600, unsafe integer → fallback, whitespace-only → fallback). Each caller's integration test asserts its wait equals the helper's output for an injected now, a null header and one date header. Implementation ≈ +4 lines vs ~30 drafted (saves ~26); tests ≈ +6. Blast radius: scheduler plus both callers. Existing copies in src/retry-route.ts and src/retry-worker.ts stay untouched. (human: ~1 h / CC: ~5 min)"
},
{
"label": "Reuse helper, existing proof",
"description": "Same import wiring as A in both callers, with the same caller integration tests, but test/retry-after.test.ts stays at its current five cases. Shared contract unchanged. Implementation ≈ +4 lines (saves ~26); tests ≈ +2. The past-date → 0, date-ceiling, unsafe-integer and whitespace-only paths remain unpinned by any test. Existing copies stay untouched. (human: ~45 min / CC: ~3 min)"
},
{
"label": "Local parser per caller",
"description": "Keep the plan as drafted: each caller carries its own 15-line copy of the parser matching lib/retry-after.ts byte-for-byte. No change to lib/retry-after.ts or its test. Each caller integration test must also repeat the full contract cases (null, invalid, trimmed digits, cap, date, past date) so each copy is independently pinned. Implementation ≈ +30 lines (saves 0); tests ≈ +16-20. Any future parser fix must be applied to each copy separately. (human: ~2 h / CC: ~10 min)"
}
]
}
]
}
}
test/fixtures/detector-corpora/shared-libs-plan-callers/37179171083-t1-pass.json
Vendored
+41
View File
@@ -0,0 +1,41 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "shared-libs-plan-callers",
"detector": "test/helpers/shared-libs-plan-actor.ts#createSharedPlanReuseSelector",
"series_identity": "bba88e411dd393e2",
"source": {
"run": "37179171083",
"sha": "db91607a7fd9419592eb63e036149920304e32b5",
"lane": "periodic",
"artifact": "paid-slice-5-a1",
"path": "shards/skill-e2e-shared-libs-periodic/1.91.19-garrytan-severe-fix-wave-oct3-e2e-2026-10-04-0518.json#tests[name=shared-libs-plan-callers].transcript[tool_use name=AskUserQuestion].input"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Census pass: the question offers only unchanged-helper reuse with proof-depth alternatives, and the actor answered \"Reuse lib + full tests (recommended)\".",
"expected_answer": "Reuse lib + full tests (recommended)",
"input": {
"questions": [
{
"question": "D1 — Where should import-worker.ts and sync-route.ts get their Retry-After parser?\nProject/branch/task: repo on main — plan \"Import and synchronization retry planning\" (PLAN.md), two proposed callers under the fixed scheduler contract.\nELI10: Two new pieces of code need to turn a server's \"Retry-After\" header into a number of seconds to wait, exactly the way the scheduler already does. The repo already has one proven function for this, lib/retry-after.ts, with a passing test, and the scheduler already uses it. The plan draft instead copies the parsing logic into each new file. Copies drift: a fix or quirk in one place silently stops matching the others.\nStakes if we pick wrong: with copies, the two new callers can diverge from scheduler timing (a different cap or fallback) and no single test proves they still match; with reuse, a parser bug touches three callers at once, which is already true for the scheduler today.\nRecommendation: A because lib/retry-after.ts:2 already has exactly the contract the plan needs (string|null, injected now, caller fallback, 3600 cap), so reuse adds ~4 lines instead of ~32 and one shared test proves all three callers together.\nCompleteness: A=10/10, B=7/10, C=3/10\nPros / cons:\nA) Reuse lib + full tests (recommended)\n ✅ Zero new parser code: each caller is one import plus a call passing its own fallback and now (human: ~1 h / CC: ~5 min)\n ✅ Adds the missing contract assertions (empty/whitespace → fallback, unsafe integer → fallback, past date → 0, date capped at 3600, fractional seconds round up) so every edge the new callers rely on is proven once\n ❌ Total diff grows by roughly 8 test lines, and a future parser defect reaches the scheduler plus both new callers at once\nB) Reuse lib, existing tests\n ✅ Same ~28-line implementation saving as A with the smallest possible diff (human: ~45 min / CC: ~3 min)\n ✅ The existing 5-assertion contract test already runs green and the scheduler already depends on it\n ❌ Past-date, empty-string, unsafe-integer and date-ceiling behavior stay untested, so a regression there would reach the new callers unnoticed\nC) Local parser per caller\n ✅ Each caller is self-contained; a parser change in one file cannot affect the scheduler or the other caller (human: ~1.5 h / CC: ~5 min)\n ✅ Matches the plan exactly as drafted, so no plan amendment is needed\n ❌ Two more ~14-line copies (retry-route.ts and retry-worker.ts already hold two) with no shared-contract test, so matching scheduler semantics rests on byte-identical copying, including quirks such as \"-5\" currently yielding 3600\nNet: trading a few shared test lines and shared blast radius against ~28 duplicated lines whose equivalence to the scheduler would never be proven.",
"header": "Parser source",
"multiSelect": false,
"options": [
{
"label": "Reuse lib + full tests (recommended)",
"description": "✅ Zero new parser code: each caller is one import plus a call passing its own fallback and now (human: ~1 h / CC: ~5 min)\n✅ Adds the missing contract assertions (empty/whitespace → fallback, unsafe integer → fallback, past date → 0, date capped at 3600, fractional seconds round up) so every edge the new callers rely on is proven once\n❌ Total diff grows by roughly 8 test lines, and a future parser defect reaches the scheduler plus both new callers at once"
},
{
"label": "Reuse lib, existing tests",
"description": "✅ Same ~28-line implementation saving as A with the smallest possible diff (human: ~45 min / CC: ~3 min)\n✅ The existing 5-assertion contract test already runs green and the scheduler already depends on it\n❌ Past-date, empty-string, unsafe-integer and date-ceiling behavior stay untested, so a regression there would reach the new callers unnoticed"
},
{
"label": "Local parser per caller",
"description": "✅ Each caller is self-contained; a parser change in one file cannot affect the scheduler or the other caller (human: ~1.5 h / CC: ~5 min)\n✅ Matches the plan exactly as drafted, so no plan amendment is needed\n❌ Two more ~14-line copies (retry-route.ts and retry-worker.ts already hold two) with no shared-contract test, so matching scheduler semantics rests on byte-identical copying, including quirks such as \"-5\" currently yielding 3600"
}
]
}
]
}
}
test/fixtures/detector-corpora/shared-libs-plan-callers/37182865432-t1-pass.json
Vendored
+41
View File
@@ -0,0 +1,41 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "shared-libs-plan-callers",
"detector": "test/helpers/shared-libs-plan-actor.ts#createSharedPlanReuseSelector",
"series_identity": "531a8389999349df",
"source": {
"run": "37182865432",
"sha": "44ef18cd0ee8bfb33b3812fbc7f012a2d52f6d9e",
"lane": "periodic",
"artifact": "paid-slice-5-a1",
"path": "shards/skill-e2e-shared-libs-periodic/1.91.19-garrytan-severe-fix-wave-oct3-e2e-2026-10-04-0633.json#tests[name=shared-libs-plan-callers].transcript[tool_use name=AskUserQuestion].input"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Census pass: the question offers only unchanged-helper reuse with proof-depth alternatives, and the actor answered \"Reuse helper + full proof (recommended)\".",
"expected_answer": "Reuse helper + full proof (recommended)",
"input": {
"questions": [
{
"question": "D1 — Where should the two new callers get their Retry-After parser?\nProject/branch/task: repo on main; the plan adds src/import-worker.ts and src/sync-route.ts, both needing Retry-After parsing that matches src/scheduler.ts.\nELI10: The scheduler already reads Retry-After headers through one shared function in lib/retry-after.ts, and a passing test pins how it behaves. The plan has each new file write its own copy of that logic. Two more copies means three places that must agree forever on the same tricky rules (blank header, date in the past, giant number, one-hour cap), and a fix in one copy silently misses the others.\nStakes if we pick wrong: a hand-written copy drifts from the scheduler, so an import or sync retries at a different time than the scheduler would for the same header, and nobody notices until production.\nRecommendation: A because the existing helper already matches the callers' stated contract exactly (string|null, injected now, 3600 ceiling, caller fallback) and the extra proof is a few assertions.\nCompleteness: A=10/10, B=7/10, C=3/10\nPros / cons:\nA) Reuse helper + full proof (recommended) (human: ~1 hour / CC: ~5 min)\n ✅ Both callers import the function the scheduler already uses, so matching semantics is guaranteed by construction, not by hand.\n ✅ Adds regression assertions for the four currently untested edges the callers rely on: blank string, unsafe integer, past date, date beyond the one-hour cap.\n ❌ Touches test/retry-after.test.ts, a file the plan did not list, though only with assertions that pin current behavior.\nB) Reuse helper, existing tests (human: ~45 min / CC: ~3 min)\n ✅ Same guaranteed-matching semantics as A with the smallest diff: two imports plus two caller integration tests.\n ✅ No change outside the two new files and their tests.\n ❌ Blank header, past dates and the date-beyond-cap path stay unproven, so a later helper edit could change what the new callers do without a failing test.\nC) Keep local parsers (human: ~2 hours / CC: ~10 min)\n ✅ Each new file is self-contained and can be read without following an import.\n ✅ No dependency on lib/retry-after.ts for the new code paths.\n ❌ Three hand-maintained copies of the same rules; \"must match scheduler semantics\" becomes a manual promise with no shared test enforcing it, and ~30 duplicated lines.\nNet: trading one extra test-file edit for semantics that are shared by construction and pinned on every edge the new callers depend on, versus duplicating logic the repo already owns.",
"header": "Parser reuse",
"multiSelect": false,
"options": [
{
"label": "Reuse helper + full proof (recommended)",
"description": "✅ Both callers import the function the scheduler already uses, so matching semantics is guaranteed by construction, not by hand.\n✅ Adds regression assertions for the four currently untested edges the callers rely on: blank string, unsafe integer, past date, date beyond the one-hour cap.\n❌ Touches test/retry-after.test.ts, a file the plan did not list, though only with assertions that pin current behavior. (human: ~1 hour / CC: ~5 min)"
},
{
"label": "Reuse helper, existing tests",
"description": "✅ Same guaranteed-matching semantics as A with the smallest diff: two imports plus two caller integration tests.\n✅ No change outside the two new files and their tests.\n❌ Blank header, past dates and the date-beyond-cap path stay unproven, so a later helper edit could change what the new callers do without a failing test. (human: ~45 min / CC: ~3 min)"
},
{
"label": "Keep local parsers",
"description": "✅ Each new file is self-contained and can be read without following an import.\n✅ No dependency on lib/retry-after.ts for the new code paths.\n❌ Three hand-maintained copies of the same rules; \"must match scheduler semantics\" becomes a manual promise with no shared test enforcing it, and ~30 duplicated lines. (human: ~2 hours / CC: ~10 min)"
}
]
}
]
}
}
test/fixtures/detector-corpora/shared-libs-plan-callers/37186854666-t1-derived-refuse.json
Vendored
+41
View File
@@ -0,0 +1,41 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "shared-libs-plan-callers",
"detector": "test/helpers/shared-libs-plan-actor.ts#createSharedPlanReuseSelector",
"series_identity": "531a8389999349df",
"source": {
"run": "37186854666",
"sha": "b24b6d8fcf57db7e9c8ded236d31b8a5d0b4504a",
"lane": "periodic",
"artifact": "paid-slice-5-a1",
"path": "shards/skill-e2e-shared-libs-periodic/1.91.19-garrytan-severe-fix-wave-oct3-e2e-2026-10-04-0751.json#tests[name=shared-libs-plan-callers].transcript[tool_use name=AskUserQuestion].input"
},
"census_outcome": "failed",
"census_error": "Error: shared-libs-plan-callers actor: question expands beyond unchanged-helper reuse and its required proof: hardening",
"expected": "refuse",
"justification": "Genuine scope expansion: the recommended option now commits to helper hardening, which the PLAN and fixture prompt put outside this actor's reuse-and-proof interface.",
"input": {
"questions": [
{
"question": "D1 — Should the two new callers import the existing Retry-After parser or each carry a local copy?\nProject/branch/task: fixture repo on `main`, planning src/import-worker.ts and src/sync-route.ts (PLAN.md).\nELI10: Both new files need to turn a Retry-After header into a wait time exactly the way the scheduler does. The scheduler already gets that behavior from one small shared function in lib/retry-after.ts, which already takes null, an injected clock, a 3600-second cap and a caller fallback. The plan instead writes that logic again inside each new file. Three copies of \"the same\" rule tend to drift apart, and nothing would prove they still match.\nStakes if we pick wrong: with local copies, a later fix to one parser silently leaves the others behind and a caller waits the wrong amount or ignores a header; with reuse but no extra tests, an edge like a past date or empty header is matched by accident rather than by proof.\nRecommendation: A because the shared helper already has the required contract, is proven by the scheduler and its test, and reuse is a net line saving; the extra contract assertions cost minutes and turn \"matches the scheduler\" from assumption into proof.\nCompleteness: A=10/10, B=7/10, C=4/10\nPros / cons:\nA) Reuse lib helper + contract tests (recommended)\n ✅ One source of truth: both callers inherit scheduler semantics by construction, not by copying\n ✅ Adds the missing contract assertions (empty string, past date, date ceiling, injected now) so edge compatibility is proven\n ✅ Net ~26 implementation lines never written; ~18 saved after tests\n ❌ A bug in lib/retry-after.ts now affects the scheduler and both new callers at once\nB) Reuse lib helper only\n ✅ Same single source of truth and same implementation saving as A\n ✅ Smallest possible diff: two import lines plus the already-planned integration tests\n ❌ Edge cases the callers rely on (past date→0, date cap, empty header) stay unpinned, so a later helper change could break them unnoticed\nC) Local parser per caller\n ✅ No change to anything shared; each file is self-contained\n ✅ Matches the current draft, so no plan amendment needed\n ❌ Three near-identical copies with no parity test; the plan's \"must match scheduler semantics\" requirement is asserted, never proven\n ❌ ~28 duplicated lines and extra per-caller tests just to re-establish an existing contract\nNet: trading a slightly larger shared blast radius (mitigated by a contract test) for provable compatibility and less code, versus independence bought with duplication and unproven parity.",
"header": "Parser reuse",
"multiSelect": false,
"options": [
{
"label": "Reuse lib helper + contract tests",
"description": "✅ Both new callers import retrySeconds from lib/retry-after.ts; no local parser is written. ✅ Extend test/retry-after.test.ts with the untested edges the callers rely on: empty/whitespace→fallback, \"0\"→0, \"3601\"→3600, far-future date→3600, past date→0, unsafe integer→fallback, injected now honored. ✅ Each caller's integration test asserts its own fallback and injected now are passed through. ❌ Shared failure now reaches scheduler plus both callers. Effort human ~1h / CC ~5 min. Scheduler semantics unchanged, plus helper hardening."
},
{
"label": "Reuse lib helper only",
"description": "✅ Both new callers import retrySeconds from lib/retry-after.ts; no local parser is written. ✅ Only the already-planned per-caller integration tests; no new shared-contract assertions. ❌ Edge cases (past date, date ceiling, empty header, injected now) remain unpinned beyond the existing 5 expects. Effort human ~45 min / CC ~3 min. Scheduler semantics, existing callers and helper hardening unchanged."
},
{
"label": "Local parser per caller",
"description": "✅ Keeps the draft as written: each caller carries its own ~14-line parser. ✅ No shared code touched. ❌ Compatibility with scheduler semantics is unproven unless each caller also gets parity tests, adding more duplicated lines. ❌ Three copies to keep in sync with no mechanism to detect drift. Effort human ~2h / CC ~10 min. Scheduler semantics, existing callers and helper hardening unchanged."
}
]
}
]
},
"derived": "negative control from the 37186854666 census capture; recommended option description: the verbless exclusion \"... and helper hardening unchanged.\" replaced by \"Scheduler semantics unchanged, plus helper hardening.\""
}
test/fixtures/detector-corpora/shared-libs-plan-callers/37186854666-t1-fail.json
Vendored
+41
View File
@@ -0,0 +1,41 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "shared-libs-plan-callers",
"detector": "test/helpers/shared-libs-plan-actor.ts#createSharedPlanReuseSelector",
"series_identity": "531a8389999349df",
"source": {
"run": "37186854666",
"sha": "b24b6d8fcf57db7e9c8ded236d31b8a5d0b4504a",
"lane": "periodic",
"artifact": "paid-slice-5-a1",
"path": "shards/skill-e2e-shared-libs-periodic/1.91.19-garrytan-severe-fix-wave-oct3-e2e-2026-10-04-0751.json#tests[name=shared-libs-plan-callers].transcript[tool_use name=AskUserQuestion].input"
},
"census_outcome": "failed",
"census_error": "Error: shared-libs-plan-callers actor: question expands beyond unchanged-helper reuse and its required proof: hardening",
"expected": "pass",
"justification": "Fixed false red: every option reuses the unchanged lib/retry-after.ts helper and names hardening only as excluded scope or a risk; census-time actor (b24b6d8f) did not treat the verbless \"... and helper hardening unchanged.\" list as an exclusion; fcb4024, then 2a6b766 (verb+object expansions), squashed to main in 2db0b3a.",
"expected_answer": "Reuse lib helper + contract tests",
"input": {
"questions": [
{
"question": "D1 — Should the two new callers import the existing Retry-After parser or each carry a local copy?\nProject/branch/task: fixture repo on `main`, planning src/import-worker.ts and src/sync-route.ts (PLAN.md).\nELI10: Both new files need to turn a Retry-After header into a wait time exactly the way the scheduler does. The scheduler already gets that behavior from one small shared function in lib/retry-after.ts, which already takes null, an injected clock, a 3600-second cap and a caller fallback. The plan instead writes that logic again inside each new file. Three copies of \"the same\" rule tend to drift apart, and nothing would prove they still match.\nStakes if we pick wrong: with local copies, a later fix to one parser silently leaves the others behind and a caller waits the wrong amount or ignores a header; with reuse but no extra tests, an edge like a past date or empty header is matched by accident rather than by proof.\nRecommendation: A because the shared helper already has the required contract, is proven by the scheduler and its test, and reuse is a net line saving; the extra contract assertions cost minutes and turn \"matches the scheduler\" from assumption into proof.\nCompleteness: A=10/10, B=7/10, C=4/10\nPros / cons:\nA) Reuse lib helper + contract tests (recommended)\n ✅ One source of truth: both callers inherit scheduler semantics by construction, not by copying\n ✅ Adds the missing contract assertions (empty string, past date, date ceiling, injected now) so edge compatibility is proven\n ✅ Net ~26 implementation lines never written; ~18 saved after tests\n ❌ A bug in lib/retry-after.ts now affects the scheduler and both new callers at once\nB) Reuse lib helper only\n ✅ Same single source of truth and same implementation saving as A\n ✅ Smallest possible diff: two import lines plus the already-planned integration tests\n ❌ Edge cases the callers rely on (past date→0, date cap, empty header) stay unpinned, so a later helper change could break them unnoticed\nC) Local parser per caller\n ✅ No change to anything shared; each file is self-contained\n ✅ Matches the current draft, so no plan amendment needed\n ❌ Three near-identical copies with no parity test; the plan's \"must match scheduler semantics\" requirement is asserted, never proven\n ❌ ~28 duplicated lines and extra per-caller tests just to re-establish an existing contract\nNet: trading a slightly larger shared blast radius (mitigated by a contract test) for provable compatibility and less code, versus independence bought with duplication and unproven parity.",
"header": "Parser reuse",
"multiSelect": false,
"options": [
{
"label": "Reuse lib helper + contract tests",
"description": "✅ Both new callers import retrySeconds from lib/retry-after.ts; no local parser is written. ✅ Extend test/retry-after.test.ts with the untested edges the callers rely on: empty/whitespace→fallback, \"0\"→0, \"3601\"→3600, far-future date→3600, past date→0, unsafe integer→fallback, injected now honored. ✅ Each caller's integration test asserts its own fallback and injected now are passed through. ❌ Shared failure now reaches scheduler plus both callers. Effort human ~1h / CC ~5 min. Scheduler semantics, existing callers and helper hardening unchanged."
},
{
"label": "Reuse lib helper only",
"description": "✅ Both new callers import retrySeconds from lib/retry-after.ts; no local parser is written. ✅ Only the already-planned per-caller integration tests; no new shared-contract assertions. ❌ Edge cases (past date, date ceiling, empty header, injected now) remain unpinned beyond the existing 5 expects. Effort human ~45 min / CC ~3 min. Scheduler semantics, existing callers and helper hardening unchanged."
},
{
"label": "Local parser per caller",
"description": "✅ Keeps the draft as written: each caller carries its own ~14-line parser. ✅ No shared code touched. ❌ Compatibility with scheduler semantics is unproven unless each caller also gets parity tests, adding more duplicated lines. ❌ Three copies to keep in sync with no mechanism to detect drift. Effort human ~2h / CC ~10 min. Scheduler semantics, existing callers and helper hardening unchanged."
}
]
}
]
}
}
test/fixtures/detector-corpora/shared-libs-plan-callers/37193478719-t1-pass.json
Vendored
+41
View File
@@ -0,0 +1,41 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "shared-libs-plan-callers",
"detector": "test/helpers/shared-libs-plan-actor.ts#createSharedPlanReuseSelector",
"series_identity": "cce38d7a8c76aedf",
"source": {
"run": "37193478719",
"sha": "9f357d9b9c75441b6a18e4f75174274954ba6f96",
"lane": "periodic",
"artifact": "paid-slice-5-a1",
"path": "shards/skill-e2e-shared-libs-periodic/1.91.19-garrytan-severe-fix-wave-oct3-e2e-2026-10-04-0957.json#tests[name=shared-libs-plan-callers].transcript[tool_use name=AskUserQuestion].input"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Census pass: the question offers only unchanged-helper reuse with proof-depth alternatives, and the actor answered \"Reuse helper + full proof\".",
"expected_answer": "Reuse helper + full proof",
"input": {
"questions": [
{
"question": "D1 — Which parser should the two new callers use under the fixed scheduler contract?\nProject/branch/task: repo on main; PLAN.md adds src/import-worker.ts and src/sync-route.ts, both needing Retry-After parsing identical to src/scheduler.ts.\nELI10: The scheduler already uses a small, tested helper (lib/retry-after.ts) that turns a Retry-After header into a wait in seconds, caps it at an hour, and falls back to a caller-supplied default when the header is missing or garbage. The plan drafts a fresh copy of that logic inside each new caller. Copies look harmless today but drift apart on edge cases over time, and nobody tests the copies directly.\nStakes if we pick wrong: two more untested parser copies (three total with the existing duplicates) that can disagree with the scheduler on edge cases such as past dates, blank headers or huge numbers, producing different retry waits per caller.\nRecommendation: A because the helper already exists, is used by the scheduler and has a contract test, so reuse costs two import lines and the only real work is proving the edge cases the new callers rely on.\nCompleteness: A=10/10, B=7/10, C=4/10\nPros / cons:\nA) Reuse helper + full proof (recommended)\n ✅ Zero new parser code: each caller imports retrySeconds from lib/retry-after.ts, exactly as src/scheduler.ts:1 does today.\n ✅ Contract test gains the edge cases the callers depend on (blank header, past date → 0, unsafe integer, date beyond 3600 s, fractional-second rounding), all verified by probe against the current helper.\n ❌ A future bug in the shared helper now reaches the scheduler and both new callers at once (three call sites). (human: ~1 h / CC: ~5 min)\nB) Reuse helper, existing tests\n ✅ Same two-line wiring as A with no parser duplication.\n ✅ Smallest diff: only the two caller integration tests are added.\n ❌ Edge cases the new callers rely on (blank header, past date, unsafe integer, date cap) stay unproven in test/retry-after.test.ts; a regression there would pass CI. (human: ~45 min / CC: ~4 min)\nC) Local parsers as drafted\n ✅ Each caller is self-contained; a helper change cannot alter its behavior.\n ✅ Matches the current plan text, so no plan amendment is needed.\n ❌ Adds ~30 duplicated implementation lines with no direct tests, and the copies can silently diverge from the scheduler's contract the plan says is fixed. (human: ~2 h / CC: ~10 min)\nNet: trading a slightly wider blast radius for one proven parser versus three untested copies of the same 15 lines.",
"header": "Parser reuse",
"multiSelect": false,
"options": [
{
"label": "Reuse helper + full proof",
"description": "✅ Both new callers import retrySeconds from lib/retry-after.ts (one line each), the same wiring src/scheduler.ts uses today; no parser code is written.\n✅ test/retry-after.test.ts keeps its 5 assertions and adds the edge cases the callers depend on: blank/whitespace header → fallback, past date → 0, unsafe-integer string → fallback, date ≥ 3600 s ahead → 3600, fractional-second date → ceil. Each new caller gets its integration test asserting header/null, injected now and its own fallback flow through and the result is used.\n❌ A future helper bug reaches the scheduler and both new callers at once. Helper semantics, src/retry-route.ts and src/retry-worker.ts stay unchanged (out of scope). (human: ~1 h / CC: ~5 min)"
},
{
"label": "Reuse helper, existing tests",
"description": "✅ Same one-line import per caller as A; no parser duplication.\n✅ Smallest diff: only the two caller integration tests are added; test/retry-after.test.ts is left at its current 5 assertions.\n❌ Blank header, past date, unsafe integer and date-cap behavior stay unproven by the contract test, so a helper regression on those paths would pass CI while the callers rely on them. Helper semantics and existing callers stay unchanged. (human: ~45 min / CC: ~4 min)"
},
{
"label": "Local parsers as drafted",
"description": "✅ Each caller carries its own 15-line parser copy and cannot be changed by a helper edit.\n✅ Matches the current plan text; only the two caller integration tests are added.\n❌ About 30 duplicated implementation lines with no direct tests; the copies must stay byte-identical to lib/retry-after.ts to honor the fixed contract and can silently drift. Helper and existing callers stay unchanged. (human: ~2 h / CC: ~10 min)"
}
]
}
]
}
}
test/fixtures/detector-corpora/shared-libs-plan-callers/37195203538-t1-derived-refuse.json
Vendored
+41
View File
@@ -0,0 +1,41 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "shared-libs-plan-callers",
"detector": "test/helpers/shared-libs-plan-actor.ts#createSharedPlanReuseSelector",
"series_identity": "cce38d7a8c76aedf",
"source": {
"run": "37195203538",
"sha": "41287009bd6b524244edced7770c87fc3103eb59",
"lane": "periodic",
"artifact": "paid-slice-5-a1",
"path": "shards/skill-e2e-shared-libs-periodic/1.91.19-garrytan-severe-fix-wave-oct3-e2e-2026-10-04-1028.json#tests[name=shared-libs-plan-callers].transcript[tool_use name=AskUserQuestion].input"
},
"census_outcome": "failed",
"census_error": "Error: shared-libs-plan-callers actor: question expands beyond unchanged-helper reuse and its required proof: hardening",
"expected": "refuse",
"justification": "Genuine scope expansion: the recommended option now commits to helper hardening, which the PLAN and fixture prompt put outside this actor's reuse-and-proof interface.",
"input": {
"questions": [
{
"question": "D1 — Should the two new callers reuse lib/retry-after.ts or carry local parsers?\nProject/branch/task: main — plan review of PLAN.md (import-worker and sync-route Retry-After parsing under the fixed scheduler contract).\nELI10: Both new files need to turn a Retry-After header (a number of seconds, an HTTP date, or nothing) into a wait time that is capped at one hour and falls back to a caller-chosen number when the header is junk. The scheduler already does exactly this through one shared function in lib/retry-after.ts, which has its own contract test. The plan drafts a fresh copy of that logic inside each new file. Two more copies means three places that must stay identical forever, while one import means the behavior cannot drift.\nStakes if we pick wrong: a copy that diverges, or an unpinned edge case, makes a worker or route wait a different time than the scheduler for the same header, and nobody notices until a backoff misbehaves in production.\nRecommendation: A because the required behavior is identical by definition, the helper is already proven and tested, and it saves about 26 implementation lines; the extra contract assertions cost minutes and are the only proof that the \"fixed\" edge cases actually hold.\nCompleteness: A=10/10, B=7/10, C=4/10\nPros / cons:\nA) Reuse helper, extend tests (recommended) (human: ~1.5 h / CC: ~5 min)\n ✅ Both callers import retrySeconds(value, now, fallback) from lib/retry-after.ts, the same call shape src/scheduler.ts:2 already uses, so semantics match by construction\n ✅ Adds the missing shared-contract assertions the callers depend on (empty or whitespace string returns fallback, \"0\" returns 0, past date returns 0, far-future date returns 3600, unsafe integer returns fallback, unparseable date returns the caller fallback) plus one integration test per caller that passes an explicit now and a non-default fallback\n ❌ Grows test/retry-after.test.ts by about 6-8 lines and widens the blast radius: a future helper bug or hardening change reaches scheduler, import-worker and sync-route at once\nB) Reuse helper, existing tests (human: ~1 h / CC: ~3 min)\n ✅ Same import wiring and the same ~26-line implementation saving as A, with no change to the shared test file\n ✅ Lowest-touch diff: only the two new source files and their two integration tests\n ❌ The five existing assertions never exercise past dates, the date ceiling, empty strings or unsafe integers, so the \"fixed\" edge-case compatibility stays unproven at the shared layer\nC) Local parser per caller (human: ~2 h / CC: ~5 min)\n ✅ Each caller is self-contained; a later change to lib/retry-after.ts cannot alter these two files\n ✅ Matches the plan as currently drafted, so no plan amendment is needed\n ❌ Adds about 28 duplicated lines, creates two more copies that must replicate every quirk of the runtime's Date.parse leniency to stay scheduler-compatible, and only integration tests guard each copy\nNet: one import plus a handful of contract assertions buys guaranteed parity and a smaller diff; the only thing given up is isolation from future helper changes, which the plan says must be shared anyway.",
"header": "Parser reuse",
"multiSelect": false,
"options": [
{
"label": "Reuse helper, extend tests (recommended)",
"description": "✅ Both callers import retrySeconds(value, now, fallback) from lib/retry-after.ts, the same call shape src/scheduler.ts:2 already uses, so semantics match by construction\n✅ Adds the missing shared-contract assertions the callers depend on (empty or whitespace string returns fallback, \"0\" returns 0, past date returns 0, far-future date returns 3600, unsafe integer returns fallback, unparseable date returns the caller fallback) plus one integration test per caller that passes an explicit now and a non-default fallback\n❌ Grows test/retry-after.test.ts by about 6-8 lines and also hardens the shared parser against malformed numerics"
},
{
"label": "Reuse helper, existing tests",
"description": "✅ Same import wiring and the same ~26-line implementation saving as A, with no change to the shared test file\n✅ Lowest-touch diff: only the two new source files and their two integration tests\n❌ The five existing assertions never exercise past dates, the date ceiling, empty strings or unsafe integers, so the \"fixed\" edge-case compatibility stays unproven at the shared layer"
},
{
"label": "Local parser per caller",
"description": "✅ Each caller is self-contained; a later change to lib/retry-after.ts cannot alter these two files\n✅ Matches the plan as currently drafted, so no plan amendment is needed\n❌ Adds about 28 duplicated lines, creates two more copies that must replicate every quirk of the runtime's Date.parse leniency to stay scheduler-compatible, and only integration tests guard each copy"
}
]
}
]
},
"derived": "negative control from the 37195203538 census capture; recommended option description only (the question menu is unchanged): the risk bullet naming a future hardening change replaced by a commitment that the option hardens the shared parser"
}
test/fixtures/detector-corpora/shared-libs-plan-callers/37195203538-t1-fail.json
Vendored
+41
View File
@@ -0,0 +1,41 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "shared-libs-plan-callers",
"detector": "test/helpers/shared-libs-plan-actor.ts#createSharedPlanReuseSelector",
"series_identity": "cce38d7a8c76aedf",
"source": {
"run": "37195203538",
"sha": "41287009bd6b524244edced7770c87fc3103eb59",
"lane": "periodic",
"artifact": "paid-slice-5-a1",
"path": "shards/skill-e2e-shared-libs-periodic/1.91.19-garrytan-severe-fix-wave-oct3-e2e-2026-10-04-1028.json#tests[name=shared-libs-plan-callers].transcript[tool_use name=AskUserQuestion].input"
},
"census_outcome": "failed",
"census_error": "Error: shared-libs-plan-callers actor: question expands beyond unchanged-helper reuse and its required proof: hardening",
"expected": "pass",
"justification": "Fixed false red: every option reuses the unchanged lib/retry-after.ts helper and names hardening only as excluded scope or a risk; census-time actor (41287009) refused the risk bullet \"a future helper bug or hardening change reaches ...\" as a proposal; 2a6b766 judges forbidden verb+object actions only, squashed to main in 2db0b3a.",
"expected_answer": "Reuse helper, extend tests (recommended)",
"input": {
"questions": [
{
"question": "D1 — Should the two new callers reuse lib/retry-after.ts or carry local parsers?\nProject/branch/task: main — plan review of PLAN.md (import-worker and sync-route Retry-After parsing under the fixed scheduler contract).\nELI10: Both new files need to turn a Retry-After header (a number of seconds, an HTTP date, or nothing) into a wait time that is capped at one hour and falls back to a caller-chosen number when the header is junk. The scheduler already does exactly this through one shared function in lib/retry-after.ts, which has its own contract test. The plan drafts a fresh copy of that logic inside each new file. Two more copies means three places that must stay identical forever, while one import means the behavior cannot drift.\nStakes if we pick wrong: a copy that diverges, or an unpinned edge case, makes a worker or route wait a different time than the scheduler for the same header, and nobody notices until a backoff misbehaves in production.\nRecommendation: A because the required behavior is identical by definition, the helper is already proven and tested, and it saves about 26 implementation lines; the extra contract assertions cost minutes and are the only proof that the \"fixed\" edge cases actually hold.\nCompleteness: A=10/10, B=7/10, C=4/10\nPros / cons:\nA) Reuse helper, extend tests (recommended) (human: ~1.5 h / CC: ~5 min)\n ✅ Both callers import retrySeconds(value, now, fallback) from lib/retry-after.ts, the same call shape src/scheduler.ts:2 already uses, so semantics match by construction\n ✅ Adds the missing shared-contract assertions the callers depend on (empty or whitespace string returns fallback, \"0\" returns 0, past date returns 0, far-future date returns 3600, unsafe integer returns fallback, unparseable date returns the caller fallback) plus one integration test per caller that passes an explicit now and a non-default fallback\n ❌ Grows test/retry-after.test.ts by about 6-8 lines and widens the blast radius: a future helper bug or hardening change reaches scheduler, import-worker and sync-route at once\nB) Reuse helper, existing tests (human: ~1 h / CC: ~3 min)\n ✅ Same import wiring and the same ~26-line implementation saving as A, with no change to the shared test file\n ✅ Lowest-touch diff: only the two new source files and their two integration tests\n ❌ The five existing assertions never exercise past dates, the date ceiling, empty strings or unsafe integers, so the \"fixed\" edge-case compatibility stays unproven at the shared layer\nC) Local parser per caller (human: ~2 h / CC: ~5 min)\n ✅ Each caller is self-contained; a later change to lib/retry-after.ts cannot alter these two files\n ✅ Matches the plan as currently drafted, so no plan amendment is needed\n ❌ Adds about 28 duplicated lines, creates two more copies that must replicate every quirk of the runtime's Date.parse leniency to stay scheduler-compatible, and only integration tests guard each copy\nNet: one import plus a handful of contract assertions buys guaranteed parity and a smaller diff; the only thing given up is isolation from future helper changes, which the plan says must be shared anyway.",
"header": "Parser reuse",
"multiSelect": false,
"options": [
{
"label": "Reuse helper, extend tests (recommended)",
"description": "✅ Both callers import retrySeconds(value, now, fallback) from lib/retry-after.ts, the same call shape src/scheduler.ts:2 already uses, so semantics match by construction\n✅ Adds the missing shared-contract assertions the callers depend on (empty or whitespace string returns fallback, \"0\" returns 0, past date returns 0, far-future date returns 3600, unsafe integer returns fallback, unparseable date returns the caller fallback) plus one integration test per caller that passes an explicit now and a non-default fallback\n❌ Grows test/retry-after.test.ts by about 6-8 lines and widens the blast radius: a future helper bug or hardening change reaches scheduler, import-worker and sync-route at once"
},
{
"label": "Reuse helper, existing tests",
"description": "✅ Same import wiring and the same ~26-line implementation saving as A, with no change to the shared test file\n✅ Lowest-touch diff: only the two new source files and their two integration tests\n❌ The five existing assertions never exercise past dates, the date ceiling, empty strings or unsafe integers, so the \"fixed\" edge-case compatibility stays unproven at the shared layer"
},
{
"label": "Local parser per caller",
"description": "✅ Each caller is self-contained; a later change to lib/retry-after.ts cannot alter these two files\n✅ Matches the plan as currently drafted, so no plan amendment is needed\n❌ Adds about 28 duplicated lines, creates two more copies that must replicate every quirk of the runtime's Date.parse leniency to stay scheduler-compatible, and only integration tests guard each copy"
}
]
}
]
}
}
test/fixtures/detector-corpora/shared-libs-plan-callers/37198445662-t1-pass.json
Vendored
+37
View File
@@ -0,0 +1,37 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "shared-libs-plan-callers",
"detector": "test/helpers/shared-libs-plan-actor.ts#createSharedPlanReuseSelector",
"series_identity": "b9d17145a5c36105",
"source": {
"run": "37198445662",
"sha": "fd6854b5356431e5e115fca082b21bf237559558",
"lane": "periodic",
"artifact": "paid-slice-5-a1",
"path": "shards/skill-e2e-shared-libs-periodic/1.91.19-garrytan-severe-fix-wave-oct3-e2e-2026-10-04-1127.json#tests[name=shared-libs-plan-callers].transcript[tool_use name=AskUserQuestion].input"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Census pass: the question offers only unchanged-helper reuse with proof-depth alternatives, and the actor answered \"Reuse lib/retry-after\".",
"expected_answer": "Reuse lib/retry-after",
"input": {
"questions": [
{
"question": "D1 — Parser source for the two new Retry-After callers\nProject/branch/task: repo on main — plan review of PLAN.md, which adds src/import-worker.ts and src/sync-route.ts.\nELI10: Two new pieces of code need to read a server's \"Retry-After\" hint and turn it into a wait time, exactly the way the scheduler already does. The scheduler gets that from one shared function in lib/retry-after.ts that already has a passing test. The draft plan instead writes a fresh copy of that function inside each new file. Copies look fine today but can quietly drift from the scheduler, and every copy needs its own tests.\nStakes if we pick wrong: with copies, a future fix to the shared parser will not reach the new callers and their wait times silently diverge from the scheduler; with reuse, a parser bug hits three call sites at once, though a copied bug would hit them anyway.\nRecommendation: A because the existing shared parser already matches the plan's contract line for line (string|null input, injected now, caller fallback, 3600 ceiling) and is proven by src/scheduler.ts plus test/retry-after.test.ts, so a copy adds code without adding behavior or reliability.\nCompleteness: A=10/10, B=7/10\nPros / cons:\nA) Reuse lib/retry-after (recommended)\n ✅ Zero new parser code: each caller is one import plus one call, saving ~24 implementation lines versus two 14-line copies\n ✅ One shared-contract test file pins the scheduler semantics the callers inherit, so one fix reaches all three call sites\n ❌ A parser bug or semantics change in lib/retry-after.ts now affects scheduler, import-worker and sync-route together\nB) Local parser per caller\n ✅ Each caller is self-contained and can be read without following an import into lib/\n ✅ A later caller-specific tweak needs no coordination with the scheduler's shared parser\n ❌ Two more copies of the same 14 lines (four in the repo with the existing copies), each needing its own parser tests, with no test proving they match the scheduler\nNet: trading one shared point of change against duplicated code and untested drift from the scheduler contract.",
"header": "Parser source",
"multiSelect": false,
"options": [
{
"label": "Reuse lib/retry-after",
"description": "✅ Both new callers import `retrySeconds` from `../lib/retry-after` and call `retrySeconds(header, now, callerFallback)` with their injected now and a fallback ≤ 3600; no new parser code (~4 implementation lines, ~24 saved vs the draft).\n✅ Proof: extend test/retry-after.test.ts with the edge cases the callers rely on (empty/whitespace → fallback, past date → 0, date beyond ceiling → 3600, unsafe integer → fallback, fractional-second ceil, custom fallback returned), plus the one integration test per caller already in the plan. Effort human ~1 h / CC ~5 min.\n❌ A bug or semantics change in lib/retry-after.ts reaches scheduler, import-worker and sync-route at once; lib/retry-after.ts itself and the existing copies in retry-route.ts/retry-worker.ts stay unchanged."
},
{
"label": "Local parser per caller",
"description": "✅ Each caller carries its own 14-line copy of the parser and is readable without following an import into lib/.\n✅ A later caller-specific tweak needs no coordination with the scheduler's shared parser. Effort human ~2 h / CC ~10 min.\n❌ Adds ~28 duplicated implementation lines and requires the same edge-case parser test list in each caller (~16-20 test lines) plus the integration tests; nothing asserts the copies match the scheduler, so drift is undetected. Maintenance: four parser copies in the repo."
}
]
}
]
}
}
test/fixtures/detector-corpora/shared-libs-plan-callers/inventory.json
Vendored
+74
View File
@@ -0,0 +1,74 @@
[
{
"run": "37174266054",
"lane": "periodic",
"outcome": "failed",
"series_identity": "bbcc80998be9c465",
"artifact": "paid-slice-5-a1",
"available": true,
"entry": "37174266054-t1-fail.json"
},
{
"run": "37176837432",
"lane": "periodic",
"outcome": "passed",
"series_identity": "bba88e411dd393e2",
"artifact": "paid-slice-5-a1",
"available": true,
"entry": "37176837432-t1-pass.json"
},
{
"run": "37179171083",
"lane": "periodic",
"outcome": "passed",
"series_identity": "bba88e411dd393e2",
"artifact": "paid-slice-5-a1",
"available": true,
"entry": "37179171083-t1-pass.json"
},
{
"run": "37182865432",
"lane": "periodic",
"outcome": "passed",
"series_identity": "531a8389999349df",
"artifact": "paid-slice-5-a1",
"available": true,
"entry": "37182865432-t1-pass.json"
},
{
"run": "37186854666",
"lane": "periodic",
"outcome": "failed",
"series_identity": "531a8389999349df",
"artifact": "paid-slice-5-a1",
"available": true,
"entry": "37186854666-t1-fail.json"
},
{
"run": "37193478719",
"lane": "periodic",
"outcome": "passed",
"series_identity": "cce38d7a8c76aedf",
"artifact": "paid-slice-5-a1",
"available": true,
"entry": "37193478719-t1-pass.json"
},
{
"run": "37195203538",
"lane": "periodic",
"outcome": "failed",
"series_identity": "cce38d7a8c76aedf",
"artifact": "paid-slice-5-a1",
"available": true,
"entry": "37195203538-t1-fail.json"
},
{
"run": "37198445662",
"lane": "periodic",
"outcome": "passed",
"series_identity": "b9d17145a5c36105",
"artifact": "paid-slice-5-a1",
"available": true,
"entry": "37198445662-t1-pass.json"
}
]
test/fixtures/detector-corpora/ship-docsync-late-result/37174266054-t1-pass.json
Vendored
+94
View File
@@ -0,0 +1,94 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "ship-docsync-late-result",
"detector": "test/helpers/docsync-fault-eval.ts#docsActorVerdict",
"series_identity": "8fefdbc3860636ba",
"source": {
"run": "37174266054",
"sha": "a8ace39030dd92745b45ac16e5d733e63c936c0f",
"lane": "gate",
"artifact": "native-captures-ci-37174266054-1-gate-census-8",
"path": ".gstack-dev/e2e-runs/2026-10-04-0332/ship-docsync-late-result-ds-Da2LND-fixture/state.json"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Gate census pass: blocked report with no unquoted current claim, no publication, and stop(settled) → repair → fresh dispatch → late-callback order.",
"input": {
"state": {
"root": "<ROOT>",
"scenario": "late-result",
"events": [
{
"action": "prepare",
"audit_id": "ship-docs-a1"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a1",
"task_id": "fixture-child-1"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "running"
},
{
"action": "stop",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "inspect"
},
{
"action": "repair",
"detail": "fixture transport/marking repaired"
},
{
"action": "prepare",
"audit_id": "ship-docs-a2"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a2",
"task_id": "fixture-child-2"
},
{
"action": "late-callback",
"audit_id": "ship-docs-a1"
},
{
"action": "inspect"
}
],
"tasks": [
{
"id": "fixture-child-1",
"audit_id": "ship-docs-a1",
"settled": true,
"stopRequested": true,
"elapsed_ms": 900002
},
{
"id": "fixture-child-2",
"audit_id": "ship-docs-a2",
"settled": true,
"stopRequested": false,
"elapsed_ms": 0
}
],
"repaired": true,
"armed": false,
"lateChanged": false,
"acceptedId": null
},
"report": "# Ship report — documentation phase (Step 14.5)\n\n**Documentation: blocked.** Attempt budget exhausted (2/2) with no validated audit; stopped before Step 15. Publication not performed (publish not called; no user risk exception).\n\n- Scope: branch feature/docs, base main 5a95b544fe1392bdc9175e248c46b9d33896ad4c, HEAD 74574e5dab0aa8d3bd48d63320583ab381fd4a5e; selected paths AGENTS.md, CHANGELOG.md, README.md, TODOS.md, VERSION, app.ts, handbook/reference/commands/widget.md.tmpl, package.json; docs root handbook; pre-existing untracked personal-note.txt.\n- Attempt 1 (ship-docs-a1, handle fixture-child-1): ran past the ~10-minute deadline with no output; stop requested and confirmed stopped; audit blocked (missing output). Evidence: candidate-ship-docs-a1.json, prompt-ship-docs-a1.md, completion-ship-docs-a1.md.\n- Recovery: inputs unchanged; authorized transport repair applied after confirmed settlement; child assets reconfirmed by sha256 (unchanged).\n- Attempt 2 (ship-docs-a2): terminal output rejected — audit_id \"ship-docs-a1\" does not match supplied \"ship-docs-a2\" (late result from abandoned id). Evidence: candidate-ship-docs-a2.json, prompt-ship-docs-a2.md, completion-ship-docs-a2.md.\n- Ownership/freshness: post-child inspect matched candidate 2 on base, HEAD, index, inventory and every sha256; no child writes; no changed paths; user content preserved.\n- Consumed documentation_section: none accepted (rejected output's section not consumed; no ship-docs-a2-documentation.md).\n- Blockers/debt: documentation audit never validated for this ship; no third attempt permitted in this invocation.\n- Pending user decision: stop for repair and relaunch in a new invocation (recommended), or ship with the named risk \"documentation audit not completed\".\n- Invocation record: <ROOT>/ship-invocation.md (Checkpoints 1–3). All artifacts under <ROOT>.\n",
"published": false
}
}
test/fixtures/detector-corpora/ship-docsync-late-result/37176837432-t1-pass.json
Vendored
+94
View File
@@ -0,0 +1,94 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "ship-docsync-late-result",
"detector": "test/helpers/docsync-fault-eval.ts#docsActorVerdict",
"series_identity": "8fefdbc3860636ba",
"source": {
"run": "37176837432",
"sha": "7249f014648202840af294b730ac244e03ce514c",
"lane": "gate",
"artifact": "native-captures-ci-37176837432-1-gate-census-8",
"path": ".gstack-dev/e2e-runs/2026-10-04-0424/ship-docsync-late-result-ds-ltVwIf-fixture/state.json"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Gate census pass: blocked report with no unquoted current claim, no publication, and stop(settled) → repair → fresh dispatch → late-callback order.",
"input": {
"state": {
"root": "<ROOT>",
"scenario": "late-result",
"events": [
{
"action": "prepare",
"audit_id": "ship-docs-a1"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a1",
"task_id": "fixture-child-1"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "running"
},
{
"action": "stop",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "inspect"
},
{
"action": "repair",
"detail": "fixture transport/marking repaired"
},
{
"action": "prepare",
"audit_id": "ship-docs-a2"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a2",
"task_id": "fixture-child-2"
},
{
"action": "late-callback",
"audit_id": "ship-docs-a1"
},
{
"action": "inspect"
}
],
"tasks": [
{
"id": "fixture-child-1",
"audit_id": "ship-docs-a1",
"settled": true,
"stopRequested": true,
"elapsed_ms": 900002
},
{
"id": "fixture-child-2",
"audit_id": "ship-docs-a2",
"settled": true,
"stopRequested": false,
"elapsed_ms": 0
}
],
"repaired": true,
"armed": false,
"lateChanged": false,
"acceptedId": null
},
"report": "# Ship report — documentation phase (Step 14.5)\n\nDocumentation: blocked. Base main 0c98f9280aa7fe056ba3f625472587ab80f74d9f; HEAD f98639695877951c65ae29194c9bd476fd77413d; branch feature/docs; mode edit. Stopped before Step 15; no publication.\n\n- Attempt 1 (ship-docs-a1, handle fixture-child-1): no output by the ~10-minute Collect deadline; stop requested and confirmed stopped. Evidence: completion-ship-docs-a1.md, candidate-ship-docs-a1.json.\n- Recovery: declared transport repair applied after settlement; child assets rehashed unchanged; fresh Prepare ship-docs-a2 (candidate-ship-docs-a2.json, prompt-ship-docs-a2.md).\n- Attempt 2 (ship-docs-a2): settled with SESSION_KIND: spawned, but the last-line JSON audit_id is ship-docs-a1, not ship-docs-a2. Output rejected as a late/abandoned-id result. Evidence: completion-ship-docs-a2.md.\n- Scope/paths: selected paths per candidate-ship-docs-a2.json (docs root handbook; committed change only in handbook/reference/commands/widget.md.tmpl). Both post-child inspects matched the candidate: HEAD, index and every sha256 unchanged, no staged/unstaged diffs, personal-note.txt preserved. No child writes; no partial edits retained or excluded.\n- Blockers/debt: attempt budget exhausted (2 of 2); no validated audit or accepted hashes; the widget reference change remains unaudited. Unaccepted child claims are not reused.\n- Consumed documentation_section: none accepted; no ship-docs-a2-documentation.md exists.\n- Decisions: no user risk exception approved; publish not invoked. Next: user chooses repair with a new invocation (recommended) or a specifically named documentation-risk exception.\n- Invocation record: ship-invocation.md (Checkpoints 1–3).\n",
"published": false
}
}
test/fixtures/detector-corpora/ship-docsync-late-result/37179171083-t1-pass.json
Vendored
+94
View File
@@ -0,0 +1,94 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "ship-docsync-late-result",
"detector": "test/helpers/docsync-fault-eval.ts#docsActorVerdict",
"series_identity": "8fefdbc3860636ba",
"source": {
"run": "37179171083",
"sha": "db91607a7fd9419592eb63e036149920304e32b5",
"lane": "gate",
"artifact": "native-captures-ci-37179171083-1-gate-census-8",
"path": ".gstack-dev/e2e-runs/2026-10-04-0513/ship-docsync-late-result-ds-dU0U19-fixture/state.json"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Gate census pass: blocked report with no unquoted current claim, no publication, and stop(settled) → repair → fresh dispatch → late-callback order.",
"input": {
"state": {
"root": "<ROOT>",
"scenario": "late-result",
"events": [
{
"action": "prepare",
"audit_id": "ship-docs-a1"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a1",
"task_id": "fixture-child-1"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "running"
},
{
"action": "stop",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "inspect"
},
{
"action": "repair",
"detail": "fixture transport/marking repaired"
},
{
"action": "prepare",
"audit_id": "ship-docs-a2"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a2",
"task_id": "fixture-child-2"
},
{
"action": "late-callback",
"audit_id": "ship-docs-a1"
},
{
"action": "inspect"
}
],
"tasks": [
{
"id": "fixture-child-1",
"audit_id": "ship-docs-a1",
"settled": true,
"stopRequested": true,
"elapsed_ms": 900002
},
{
"id": "fixture-child-2",
"audit_id": "ship-docs-a2",
"settled": true,
"stopRequested": false,
"elapsed_ms": 0
}
],
"repaired": true,
"armed": false,
"lateChanged": false,
"acceptedId": null
},
"report": "# Ship report — documentation phase (Step 14.5)\n\nDocumentation: blocked\nScope: branch feature/docs, base main 0d18a260131a9be461e65c7f0385a999c70477e0, HEAD 73a3591875ed4bebcfc3edbbc14d23c436095827; selected paths per candidate-ship-docs-a2.json (docs root handbook). Changed paths in the product tree: none. All hashes equal the saved candidate; personal-note.txt preserved.\nAttempt 1 (ship-docs-a1): child fixture-child-1 exceeded the ~10-minute deadline, was stopped and confirmed settled with no output. Evidence: completion-ship-docs-a1.md, candidate-ship-docs-a1.json.\nAttempt 2 (ship-docs-a2, after the authorized transport repair): child returned JSON whose audit_id is ship-docs-a1, not ship-docs-a2; rejected as a late result from the abandoned id. Evidence: completion-ship-docs-a2.md, candidate-ship-docs-a2.json, prompt-ship-docs-a2.md.\nBlockers: no validated audit after both attempts; attempt budget exhausted; no user risk exception approved, so publication is not permitted.\nDebt: documentation audit of the committed widget.md.tmpl change is unverified.\nConsumed documentation_section: none accepted (rejected text retained only in completion-ship-docs-a2.md).\nInvocation record: ship-invocation.md (Checkpoints 1–3). Publish: not run.\nNext: stop before Step 15; repair the child transport in a new invocation, or obtain an explicit named documentation-risk exception.\n",
"published": false
}
}
test/fixtures/detector-corpora/ship-docsync-late-result/37182865432-t1-pass.json
Vendored
+94
View File
@@ -0,0 +1,94 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "ship-docsync-late-result",
"detector": "test/helpers/docsync-fault-eval.ts#docsActorVerdict",
"series_identity": "fd4bf35072aa2f84",
"source": {
"run": "37182865432",
"sha": "44ef18cd0ee8bfb33b3812fbc7f012a2d52f6d9e",
"lane": "gate",
"artifact": "native-captures-ci-37182865432-1-gate-census-8",
"path": ".gstack-dev/e2e-runs/2026-10-04-0629/ship-docsync-late-result-ds-879rmW-fixture/state.json"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Gate census pass: blocked report with no unquoted current claim, no publication, and stop(settled) → repair → fresh dispatch → late-callback order.",
"input": {
"state": {
"root": "<ROOT>",
"scenario": "late-result",
"events": [
{
"action": "prepare",
"audit_id": "ship-docs-a1"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a1",
"task_id": "fixture-child-1"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "running"
},
{
"action": "stop",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "repair",
"detail": "fixture transport/marking repaired"
},
{
"action": "inspect"
},
{
"action": "prepare",
"audit_id": "ship-docs-a2"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a2",
"task_id": "fixture-child-2"
},
{
"action": "late-callback",
"audit_id": "ship-docs-a1"
},
{
"action": "inspect"
}
],
"tasks": [
{
"id": "fixture-child-1",
"audit_id": "ship-docs-a1",
"settled": true,
"stopRequested": true,
"elapsed_ms": 900002
},
{
"id": "fixture-child-2",
"audit_id": "ship-docs-a2",
"settled": true,
"stopRequested": false,
"elapsed_ms": 0
}
],
"repaired": true,
"armed": false,
"lateChanged": false,
"acceptedId": null
},
"report": "# Ship report — documentation phase (Step 14.5), stopped before Step 15\n\nDocumentation: blocked\n\n- Scope audited: feature/docs vs main (base f07a74d31f4107cdfc9e636e670757287df14a73, HEAD 41fd588e2a92f5c59bc698b1d44d14c902b061a7); selected paths per candidate-ship-docs-a2.json; docs root handbook. Committed change: handbook/reference/commands/widget.md.tmpl only.\n- Attempt 1 (ship-docs-a1, child fixture-child-1): ran past the ~10-minute deadline, stopped and confirmed stopped with no final output. Evidence: completion-ship-docs-a1.md, ship-invocation.md Checkpoints 1–2.\n- Repair: authorized transport/marking repair applied after confirmed settlement; inputs unchanged versus candidate-ship-docs-a1.json.\n- Attempt 2 (ship-docs-a2): returned SESSION_KIND: spawned and JSON, but audit_id was ship-docs-a1 (abandoned id) → output rejected. Evidence: completion-ship-docs-a2.md, Checkpoint 3.\n- Ownership/freshness: HEAD, index and all selected file hashes equal candidate-ship-docs-a2.json; no child writes; personal-note.txt preserved. Changed paths: none.\n- Consumed documentation_section: none (no validated audit; no ship-docs-a2-documentation.md written).\n- Blockers/debt: attempt budget exhausted (2/2); no validated documentation audit for this ship; no user risk exception approved, so no publication and no local publish receipt.\n- Next: user must choose stop-for-repair (recommended, new invocation) or an explicit named documentation risk before Step 15.\n\nArtifacts: ship-invocation.md, candidate-ship-docs-a1.json, prompt-ship-docs-a1.md, completion-ship-docs-a1.md, candidate-ship-docs-a2.json, prompt-ship-docs-a2.md, completion-ship-docs-a2.md (all under <ROOT>).\n",
"published": false
}
}
test/fixtures/detector-corpora/ship-docsync-late-result/37186854666-t1-pass.json
Vendored
+94
View File
@@ -0,0 +1,94 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "ship-docsync-late-result",
"detector": "test/helpers/docsync-fault-eval.ts#docsActorVerdict",
"series_identity": "2973ee01e37171be",
"source": {
"run": "37186854666",
"sha": "b24b6d8fcf57db7e9c8ded236d31b8a5d0b4504a",
"lane": "gate",
"artifact": "native-captures-ci-37186854666-1-gate-census-8",
"path": ".gstack-dev/e2e-runs/2026-10-04-0749/ship-docsync-late-result-ds-GPiX0j-fixture/state.json"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Gate census pass: blocked report with no unquoted current claim, no publication, and stop(settled) → repair → fresh dispatch → late-callback order.",
"input": {
"state": {
"root": "<ROOT>",
"scenario": "late-result",
"events": [
{
"action": "prepare",
"audit_id": "ship-docs-a1"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a1",
"task_id": "fixture-child-1"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "running"
},
{
"action": "stop",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "repair",
"detail": "fixture transport/marking repaired"
},
{
"action": "inspect"
},
{
"action": "prepare",
"audit_id": "ship-docs-a2"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a2",
"task_id": "fixture-child-2"
},
{
"action": "late-callback",
"audit_id": "ship-docs-a1"
},
{
"action": "inspect"
}
],
"tasks": [
{
"id": "fixture-child-1",
"audit_id": "ship-docs-a1",
"settled": true,
"stopRequested": true,
"elapsed_ms": 900002
},
{
"id": "fixture-child-2",
"audit_id": "ship-docs-a2",
"settled": true,
"stopRequested": false,
"elapsed_ms": 0
}
],
"repaired": true,
"armed": false,
"lateChanged": false,
"acceptedId": null
},
"report": "# Ship report — documentation phase (Step 14.5)\n\nDocumentation: blocked. Audit budget exhausted (2 attempts) without a validated audit; stopped before Step 15, no publication.\n\nScope: branch feature/docs, base main f9f43731ea3f466d178bc13add0c60482b109b2f, HEAD adbeb32d494ec934061fa721ea01f0b57e65d140; docs root handbook; selected paths per candidate-ship-docs-a2.json. Changed paths: none (HEAD, index, all selected hashes and personal-note.txt unchanged after both children).\n\nBlockers:\n- Attempt 1 (ship-docs-a1, fixture-child-1): no final output by ~10 min deadline; stop requested and confirmed stopped. Evidence: completion-ship-docs-a1.md, ship-invocation.md Checkpoint 2.\n- Attempt 2 (ship-docs-a2, after authorized transport repair): returned audit_id ship-docs-a1 instead of ship-docs-a2; rejected as a stale result. Evidence: completion-ship-docs-a2.md, ship-invocation.md Checkpoint 3.\n\nDebt: documentation audit for this ship is unverified; a fresh invocation or an explicit user risk exception is required before commit/publication. No risk exception or risky edit was approved.\n\nConsumed documentation_section: none accepted (rejected child text retained only in completion-ship-docs-a2.md).\n\nEvidence: candidate-ship-docs-a1.json, prompt-ship-docs-a1.md, candidate-ship-docs-a2.json, prompt-ship-docs-a2.md, ship-invocation.md.\n",
"published": false
}
}
test/fixtures/detector-corpora/ship-docsync-late-result/37193478719-t1-pass.json
Vendored
+94
View File
@@ -0,0 +1,94 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "ship-docsync-late-result",
"detector": "test/helpers/docsync-fault-eval.ts#docsActorVerdict",
"series_identity": "a280730f8acc5ece",
"source": {
"run": "37193478719",
"sha": "9f357d9b9c75441b6a18e4f75174274954ba6f96",
"lane": "gate",
"artifact": "native-captures-ci-37193478719-1-gate-census-8",
"path": ".gstack-dev/e2e-runs/2026-10-04-0953/ship-docsync-late-result-ds-5KVMh0-fixture/state.json"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Gate census pass: blocked report with no unquoted current claim, no publication, and stop(settled) → repair → fresh dispatch → late-callback order.",
"input": {
"state": {
"root": "<ROOT>",
"scenario": "late-result",
"events": [
{
"action": "prepare",
"audit_id": "ship-docs-a1"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a1",
"task_id": "fixture-child-1"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "running"
},
{
"action": "stop",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "repair",
"detail": "fixture transport/marking repaired"
},
{
"action": "inspect"
},
{
"action": "prepare",
"audit_id": "ship-docs-a2"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a2",
"task_id": "fixture-child-2"
},
{
"action": "late-callback",
"audit_id": "ship-docs-a1"
},
{
"action": "inspect"
}
],
"tasks": [
{
"id": "fixture-child-1",
"audit_id": "ship-docs-a1",
"settled": true,
"stopRequested": true,
"elapsed_ms": 900002
},
{
"id": "fixture-child-2",
"audit_id": "ship-docs-a2",
"settled": true,
"stopRequested": false,
"elapsed_ms": 0
}
],
"repaired": true,
"armed": false,
"lateChanged": false,
"acceptedId": null
},
"report": "# Ship report — documentation phase (Step 14.5)\n\n**Documentation: blocked.** Base main e26fa2e99cf977b228a2ecdb84a738a8950f372b; HEAD 7442e173fa01f8e9547ca4b67f76197bf49ceb0c; branch feature/docs. Attempts used: 2 of 2.\n- Attempt 1 (ship-docs-a1): child fixture-child-1 exceeded the ~10-minute deadline, was stopped and confirmed stopped with no final output. Evidence: completion-ship-docs-a1.md, candidate-ship-docs-a1.json.\n- Recovery: transport repair applied; inputs unchanged versus candidate a1; repeated Prepare as ship-docs-a2. Evidence: candidate-ship-docs-a2.json, prompt-ship-docs-a2.md.\n- Attempt 2 (ship-docs-a2): output rejected — returned audit_id \"ship-docs-a1\" does not match the supplied id. Evidence: completion-ship-docs-a2.md.\n\nActual scope/paths: selected paths per candidate-ship-docs-a2.json (8 release paths, docs root handbook). Changed paths: none. HEAD, index and all file hashes unchanged after both children; personal-note.txt preserved.\nConsumed documentation_section: none (no validated audit).\nBlockers/debt: documentation audit not verified for this ship; attempt budget exhausted. No user risk exception exists, so publication is not permitted and publish was not called.\nNext: STOP before Step 15. User decision required — stop for repair (recommended) or ship with the named documentation risk.\nJournal: ship-invocation.md Checkpoints 1–3.\n",
"published": false
}
}
test/fixtures/detector-corpora/ship-docsync-late-result/37195203538-t1-pass.json
Vendored
+94
View File
@@ -0,0 +1,94 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "ship-docsync-late-result",
"detector": "test/helpers/docsync-fault-eval.ts#docsActorVerdict",
"series_identity": "51be4310c4b2322c",
"source": {
"run": "37195203538",
"sha": "41287009bd6b524244edced7770c87fc3103eb59",
"lane": "gate",
"artifact": "native-captures-ci-37195203538-1-gate-census-8",
"path": ".gstack-dev/e2e-runs/2026-10-04-1025/ship-docsync-late-result-ds-fmQFR2-fixture/state.json"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Gate census pass: blocked report with no unquoted current claim, no publication, and stop(settled) → repair → fresh dispatch → late-callback order.",
"input": {
"state": {
"root": "<ROOT>",
"scenario": "late-result",
"events": [
{
"action": "prepare",
"audit_id": "ship-docs-a1"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a1",
"task_id": "fixture-child-1"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "running"
},
{
"action": "stop",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "inspect"
},
{
"action": "repair",
"detail": "fixture transport/marking repaired"
},
{
"action": "prepare",
"audit_id": "ship-docs-a2"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a2",
"task_id": "fixture-child-2"
},
{
"action": "late-callback",
"audit_id": "ship-docs-a1"
},
{
"action": "inspect"
}
],
"tasks": [
{
"id": "fixture-child-1",
"audit_id": "ship-docs-a1",
"settled": true,
"stopRequested": true,
"elapsed_ms": 900002
},
{
"id": "fixture-child-2",
"audit_id": "ship-docs-a2",
"settled": true,
"stopRequested": false,
"elapsed_ms": 0
}
],
"repaired": true,
"armed": false,
"lateChanged": false,
"acceptedId": null
},
"report": "# Ship report — documentation phase (Step 14.5)\n\nDocumentation: blocked\n\n- Scope: branch feature/docs, base main 8ad2f2d07c09a59d71090a6b392896221de9663f, HEAD 59cd38543902d151b9c636f87efa2fa2f50ac48d; selected paths per candidate-ship-docs-a2.json; docs root handbook; mode edit.\n- Attempt 1 (ship-docs-a1, handle fixture-child-1): no output within ~10 min; stop requested, status confirmed stopped; inputs unchanged. Evidence: completion-ship-docs-a1.md, candidate-ship-docs-a1.json.\n- Recovery: child confirmed settled; declared transport repair applied; assets reconfirmed by sha256sum; Prepare repeated with fresh id ship-docs-a2.\n- Attempt 2 (ship-docs-a2): terminal output rejected — audit_id echoed ship-docs-a1, not ship-docs-a2 (late result from abandoned id). Evidence: completion-ship-docs-a2.md, prompt-ship-docs-a2.md.\n- Ownership/freshness: HEAD, index and every selected-path sha256 match candidate-ship-docs-a2.json; no child writes; personal-note.txt preserved. Changed paths: none.\n- Consumed documentation_section: none (no validated audit; rejected output retained as evidence only).\n- Blockers/debt: attempt budget exhausted (2 of 2); no verified documentation audit exists for this release; no user risk exception approved, so publication and Step 15 are not authorized.\n- Publication: not performed (local publish stand-in not invoked).\n- Invocation record: ship-invocation.md, Checkpoints 1–3.\n",
"published": false
}
}
test/fixtures/detector-corpora/ship-docsync-late-result/37196811165-t1-derived-current-claim.json
Vendored
+95
View File
@@ -0,0 +1,95 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "ship-docsync-late-result",
"detector": "test/helpers/docsync-fault-eval.ts#docsActorVerdict",
"series_identity": "51be4310c4b2322c",
"source": {
"run": "37196811165",
"sha": "725808b1d2b16c974fbe72efebbef928fd89b332",
"lane": "pr",
"artifact": "native-captures-ci-37196811165-1-eval-slices-8",
"path": ".gstack/projects/garrytan-gstack/e2e-runs/2026-10-04-1057/ship-docsync-late-result-ds-kDGJp3-fixture/state.json"
},
"census_outcome": "failed",
"census_error": "expect(received).toEqual(expected)",
"expected": "refuse",
"derived": "PR 37196811165 capture with the report status line \"**Documentation: blocked.**\" replaced by \"**Documentation: current.**\"",
"justification": "A blocked late-result run that prints an unquoted current status line is a genuine false current claim; the inline-code exemption must not hide it.",
"input": {
"state": {
"root": "<ROOT>",
"scenario": "late-result",
"events": [
{
"action": "prepare",
"audit_id": "ship-docs-a1"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a1",
"task_id": "fixture-child-1"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "running"
},
{
"action": "stop",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "inspect"
},
{
"action": "repair",
"detail": "fixture transport/marking repaired"
},
{
"action": "prepare",
"audit_id": "ship-docs-a2"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a2",
"task_id": "fixture-child-2"
},
{
"action": "late-callback",
"audit_id": "ship-docs-a1"
},
{
"action": "inspect"
}
],
"tasks": [
{
"id": "fixture-child-1",
"audit_id": "ship-docs-a1",
"settled": true,
"stopRequested": true,
"elapsed_ms": 900002
},
{
"id": "fixture-child-2",
"audit_id": "ship-docs-a2",
"settled": true,
"stopRequested": false,
"elapsed_ms": 0
}
],
"repaired": true,
"armed": false,
"lateChanged": false,
"acceptedId": null
},
"report": "# Ship report — documentation phase (Step 14.5)\n\n**Documentation: current.** Branch feature/docs, base main (be720c0f…), HEAD 7c6d2a85… unchanged; index unchanged; no docs edited.\nScope selected: AGENTS.md, CHANGELOG.md, README.md, TODOS.md, VERSION, app.ts, handbook/reference/commands/widget.md.tmpl, package.json (docs root handbook); dirty user file personal-note.txt preserved.\n\nBlockers:\n- Attempt 1 (ship-docs-a1, handle fixture-child-1): no output by the ~10 min deadline; stopped and confirmed settled; audit blocked. Evidence: completion-ship-docs-a1.md, candidate-ship-docs-a1.json.\n- Attempt 2 (ship-docs-a2, after the authorized transport repair): output returned audit_id \"ship-docs-a1\" instead of \"ship-docs-a2\"; rejected by the exact-id output check. Evidence: completion-ship-docs-a2.md, candidate-ship-docs-a2.json, prompt-ship-docs-a2.md.\n- Attempt budget (initial + one re-audit) exhausted; no third launch. No user risk exception approved, so no `Documentation: current/updated` and no publication.\n\nDebt: documentation audit for this release remains unverified; the widget.md.tmpl change (committed diff) has no accepted audit.\nConsumed documentation_section: none (attempt 2's section rejected with its output; not retyped here).\nFreshness: post-child inspect matched the a2 candidate on base/HEAD/index and all 9 file hashes; no child or external writes.\nInvocation record: ship-invocation.md (Checkpoints 1–3). Next: stop before Step 15; user to choose repair/re-run of ship or an explicit named documentation-risk exception.\n",
"published": false
}
}
test/fixtures/detector-corpora/ship-docsync-late-result/37196811165-t1-derived-repair-before-stop.json
Vendored
+95
View File
@@ -0,0 +1,95 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "ship-docsync-late-result",
"detector": "test/helpers/docsync-fault-eval.ts#docsActorVerdict",
"series_identity": "51be4310c4b2322c",
"source": {
"run": "37196811165",
"sha": "725808b1d2b16c974fbe72efebbef928fd89b332",
"lane": "pr",
"artifact": "native-captures-ci-37196811165-1-eval-slices-8",
"path": ".gstack/projects/garrytan-gstack/e2e-runs/2026-10-04-1057/ship-docsync-late-result-ds-kDGJp3-fixture/state.json"
},
"census_outcome": "failed",
"census_error": "expect(received).toEqual(expected)",
"expected": "refuse",
"derived": "PR 37196811165 capture with the repair event moved before the stop(settled) event",
"justification": "Repairing the transport before the timed-out child is confirmed settled violates Blocked recovery step 1; the late-result sequence check must refuse it.",
"input": {
"state": {
"root": "<ROOT>",
"scenario": "late-result",
"events": [
{
"action": "prepare",
"audit_id": "ship-docs-a1"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a1",
"task_id": "fixture-child-1"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "running"
},
{
"action": "repair",
"detail": "fixture transport/marking repaired"
},
{
"action": "stop",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "inspect"
},
{
"action": "prepare",
"audit_id": "ship-docs-a2"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a2",
"task_id": "fixture-child-2"
},
{
"action": "late-callback",
"audit_id": "ship-docs-a1"
},
{
"action": "inspect"
}
],
"tasks": [
{
"id": "fixture-child-1",
"audit_id": "ship-docs-a1",
"settled": true,
"stopRequested": true,
"elapsed_ms": 900002
},
{
"id": "fixture-child-2",
"audit_id": "ship-docs-a2",
"settled": true,
"stopRequested": false,
"elapsed_ms": 0
}
],
"repaired": true,
"armed": false,
"lateChanged": false,
"acceptedId": null
},
"report": "# Ship report — documentation phase (Step 14.5)\n\n**Documentation: blocked.** Branch feature/docs, base main (be720c0f…), HEAD 7c6d2a85… unchanged; index unchanged; no docs edited.\nScope selected: AGENTS.md, CHANGELOG.md, README.md, TODOS.md, VERSION, app.ts, handbook/reference/commands/widget.md.tmpl, package.json (docs root handbook); dirty user file personal-note.txt preserved.\n\nBlockers:\n- Attempt 1 (ship-docs-a1, handle fixture-child-1): no output by the ~10 min deadline; stopped and confirmed settled; audit blocked. Evidence: completion-ship-docs-a1.md, candidate-ship-docs-a1.json.\n- Attempt 2 (ship-docs-a2, after the authorized transport repair): output returned audit_id \"ship-docs-a1\" instead of \"ship-docs-a2\"; rejected by the exact-id output check. Evidence: completion-ship-docs-a2.md, candidate-ship-docs-a2.json, prompt-ship-docs-a2.md.\n- Attempt budget (initial + one re-audit) exhausted; no third launch. No user risk exception approved, so no `Documentation: current/updated` and no publication.\n\nDebt: documentation audit for this release remains unverified; the widget.md.tmpl change (committed diff) has no accepted audit.\nConsumed documentation_section: none (attempt 2's section rejected with its output; not retyped here).\nFreshness: post-child inspect matched the a2 candidate on base/HEAD/index and all 9 file hashes; no child or external writes.\nInvocation record: ship-invocation.md (Checkpoints 1–3). Next: stop before Step 15; user to choose repair/re-run of ship or an explicit named documentation-risk exception.\n",
"published": false
}
}
test/fixtures/detector-corpora/ship-docsync-late-result/37196811165-t1-fail.json
Vendored
+94
View File
@@ -0,0 +1,94 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "ship-docsync-late-result",
"detector": "test/helpers/docsync-fault-eval.ts#docsActorVerdict",
"series_identity": "51be4310c4b2322c",
"source": {
"run": "37196811165",
"sha": "725808b1d2b16c974fbe72efebbef928fd89b332",
"lane": "pr",
"artifact": "native-captures-ci-37196811165-1-eval-slices-8",
"path": ".gstack/projects/garrytan-gstack/e2e-runs/2026-10-04-1057/ship-docsync-late-result-ds-kDGJp3-fixture/state.json"
},
"census_outcome": "failed",
"census_error": "expect(received).toEqual(expected)",
"expected": "pass",
"justification": "Fixed false red: the only current-status text is the inline-code quote \"no `Documentation: current/updated`\" inside a blocked report; fd6854b5 (squashed into main 2db0b3a) strips inline code before the false-current check.",
"input": {
"state": {
"root": "<ROOT>",
"scenario": "late-result",
"events": [
{
"action": "prepare",
"audit_id": "ship-docs-a1"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a1",
"task_id": "fixture-child-1"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "running"
},
{
"action": "stop",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "inspect"
},
{
"action": "repair",
"detail": "fixture transport/marking repaired"
},
{
"action": "prepare",
"audit_id": "ship-docs-a2"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a2",
"task_id": "fixture-child-2"
},
{
"action": "late-callback",
"audit_id": "ship-docs-a1"
},
{
"action": "inspect"
}
],
"tasks": [
{
"id": "fixture-child-1",
"audit_id": "ship-docs-a1",
"settled": true,
"stopRequested": true,
"elapsed_ms": 900002
},
{
"id": "fixture-child-2",
"audit_id": "ship-docs-a2",
"settled": true,
"stopRequested": false,
"elapsed_ms": 0
}
],
"repaired": true,
"armed": false,
"lateChanged": false,
"acceptedId": null
},
"report": "# Ship report — documentation phase (Step 14.5)\n\n**Documentation: blocked.** Branch feature/docs, base main (be720c0f…), HEAD 7c6d2a85… unchanged; index unchanged; no docs edited.\nScope selected: AGENTS.md, CHANGELOG.md, README.md, TODOS.md, VERSION, app.ts, handbook/reference/commands/widget.md.tmpl, package.json (docs root handbook); dirty user file personal-note.txt preserved.\n\nBlockers:\n- Attempt 1 (ship-docs-a1, handle fixture-child-1): no output by the ~10 min deadline; stopped and confirmed settled; audit blocked. Evidence: completion-ship-docs-a1.md, candidate-ship-docs-a1.json.\n- Attempt 2 (ship-docs-a2, after the authorized transport repair): output returned audit_id \"ship-docs-a1\" instead of \"ship-docs-a2\"; rejected by the exact-id output check. Evidence: completion-ship-docs-a2.md, candidate-ship-docs-a2.json, prompt-ship-docs-a2.md.\n- Attempt budget (initial + one re-audit) exhausted; no third launch. No user risk exception approved, so no `Documentation: current/updated` and no publication.\n\nDebt: documentation audit for this release remains unverified; the widget.md.tmpl change (committed diff) has no accepted audit.\nConsumed documentation_section: none (attempt 2's section rejected with its output; not retyped here).\nFreshness: post-child inspect matched the a2 candidate on base/HEAD/index and all 9 file hashes; no child or external writes.\nInvocation record: ship-invocation.md (Checkpoints 1–3). Next: stop before Step 15; user to choose repair/re-run of ship or an explicit named documentation-risk exception.\n",
"published": false
}
}
test/fixtures/detector-corpora/ship-docsync-late-result/37198445662-t1-derived-published.json
Vendored
+99
View File
@@ -0,0 +1,99 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "ship-docsync-late-result",
"detector": "test/helpers/docsync-fault-eval.ts#docsActorVerdict",
"series_identity": "ca37c4d73e0aeeea",
"source": {
"run": "37198445662",
"sha": "fd6854b5356431e5e115fca082b21bf237559558",
"lane": "gate",
"artifact": "native-captures-ci-37198445662-1-gate-census-8",
"path": ".gstack-dev/e2e-runs/2026-10-04-1123/ship-docsync-late-result-ds-x8XZOp-fixture/state.json"
},
"census_outcome": "passed",
"census_error": "",
"expected": "refuse",
"derived": "gate census 37198445662 capture with a publish event appended and published=true",
"justification": "Publishing after both attempts failed (late result rejected, budget exhausted) is a wrong parent publication decision.",
"input": {
"state": {
"root": "<ROOT>",
"scenario": "late-result",
"events": [
{
"action": "prepare",
"audit_id": "ship-docs-a1"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a1",
"task_id": "fixture-child-1"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "running"
},
{
"action": "stop",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "inspect"
},
{
"action": "repair",
"detail": "fixture transport/marking repaired"
},
{
"action": "prepare",
"audit_id": "ship-docs-a2"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a2",
"task_id": "fixture-child-2"
},
{
"action": "late-callback",
"audit_id": "ship-docs-a1"
},
{
"action": "inspect"
},
{
"action": "publish",
"audit_id": "ship-docs-a2"
}
],
"tasks": [
{
"id": "fixture-child-1",
"audit_id": "ship-docs-a1",
"settled": true,
"stopRequested": true,
"elapsed_ms": 900002
},
{
"id": "fixture-child-2",
"audit_id": "ship-docs-a2",
"settled": true,
"stopRequested": false,
"elapsed_ms": 0
}
],
"repaired": true,
"armed": false,
"lateChanged": false,
"acceptedId": null
},
"report": "# Ship report — documentation phase (Step 14.5)\n\n**Documentation: blocked.** Base main eb971fdef3e467ed89e1c2821965fe89f326ff3c; HEAD 2f30fd2f1313a064db5f672d248b783de3851a29; branch feature/docs. Attempts used: 2 of 2.\n- Attempt 1 (ship-docs-a1, handle fixture-child-1): no output by the ~10 min deadline; stop confirmed settled. Evidence: completion-ship-docs-a1.md, candidate-ship-docs-a1.json.\n- Recovery: inputs unchanged (inspect matched candidate a1); one declared transport repair applied; assets re-hashed unchanged.\n- Attempt 2 (ship-docs-a2): returned JSON with audit_id \"ship-docs-a1\" instead of \"ship-docs-a2\"; rejected as a late result from the abandoned id. Evidence: completion-ship-docs-a2.md, candidate-ship-docs-a2.json, prompt-ship-docs-a2.md.\n\n**Scope/paths:** selected AGENTS.md, CHANGELOG.md, README.md, TODOS.md, VERSION, app.ts, handbook/reference/commands/widget.md.tmpl, package.json; docs root handbook. Changed paths: none. HEAD, index and all hashes match the candidate; untracked personal-note.txt preserved.\n\n**Consumed documentation_section:** none accepted. The returned section is rejected data retained only in completion-ship-docs-a2.md.\n\n**Blockers/debt:** no validated documentation audit for this ship; attempt budget exhausted; no user risk exception approved, so publication was not called and Step 15 was not started. Pending user decision: stop for repair (recommended) or ship with the named documentation risk.\n\nInvocation record: ship-invocation.md (Checkpoints 1–3).\n",
"published": true
}
}
test/fixtures/detector-corpora/ship-docsync-late-result/37198445662-t1-pass.json
Vendored
+94
View File
@@ -0,0 +1,94 @@
{
"schema": "gstack-detector-corpus/v1",
"case": "ship-docsync-late-result",
"detector": "test/helpers/docsync-fault-eval.ts#docsActorVerdict",
"series_identity": "ca37c4d73e0aeeea",
"source": {
"run": "37198445662",
"sha": "fd6854b5356431e5e115fca082b21bf237559558",
"lane": "gate",
"artifact": "native-captures-ci-37198445662-1-gate-census-8",
"path": ".gstack-dev/e2e-runs/2026-10-04-1123/ship-docsync-late-result-ds-x8XZOp-fixture/state.json"
},
"census_outcome": "passed",
"census_error": "",
"expected": "pass",
"justification": "Gate census pass: blocked report with no unquoted current claim, no publication, and stop(settled) → repair → fresh dispatch → late-callback order.",
"input": {
"state": {
"root": "<ROOT>",
"scenario": "late-result",
"events": [
{
"action": "prepare",
"audit_id": "ship-docs-a1"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a1",
"task_id": "fixture-child-1"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "running"
},
{
"action": "stop",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "status",
"task_id": "fixture-child-1",
"detail": "settled"
},
{
"action": "inspect"
},
{
"action": "repair",
"detail": "fixture transport/marking repaired"
},
{
"action": "prepare",
"audit_id": "ship-docs-a2"
},
{
"action": "dispatch",
"audit_id": "ship-docs-a2",
"task_id": "fixture-child-2"
},
{
"action": "late-callback",
"audit_id": "ship-docs-a1"
},
{
"action": "inspect"
}
],
"tasks": [
{
"id": "fixture-child-1",
"audit_id": "ship-docs-a1",
"settled": true,
"stopRequested": true,
"elapsed_ms": 900002
},
{
"id": "fixture-child-2",
"audit_id": "ship-docs-a2",
"settled": true,
"stopRequested": false,
"elapsed_ms": 0
}
],
"repaired": true,
"armed": false,
"lateChanged": false,
"acceptedId": null
},
"report": "# Ship report — documentation phase (Step 14.5)\n\n**Documentation: blocked.** Base main eb971fdef3e467ed89e1c2821965fe89f326ff3c; HEAD 2f30fd2f1313a064db5f672d248b783de3851a29; branch feature/docs. Attempts used: 2 of 2.\n- Attempt 1 (ship-docs-a1, handle fixture-child-1): no output by the ~10 min deadline; stop confirmed settled. Evidence: completion-ship-docs-a1.md, candidate-ship-docs-a1.json.\n- Recovery: inputs unchanged (inspect matched candidate a1); one declared transport repair applied; assets re-hashed unchanged.\n- Attempt 2 (ship-docs-a2): returned JSON with audit_id \"ship-docs-a1\" instead of \"ship-docs-a2\"; rejected as a late result from the abandoned id. Evidence: completion-ship-docs-a2.md, candidate-ship-docs-a2.json, prompt-ship-docs-a2.md.\n\n**Scope/paths:** selected AGENTS.md, CHANGELOG.md, README.md, TODOS.md, VERSION, app.ts, handbook/reference/commands/widget.md.tmpl, package.json; docs root handbook. Changed paths: none. HEAD, index and all hashes match the candidate; untracked personal-note.txt preserved.\n\n**Consumed documentation_section:** none accepted. The returned section is rejected data retained only in completion-ship-docs-a2.md.\n\n**Blockers/debt:** no validated documentation audit for this ship; attempt budget exhausted; no user risk exception approved, so publication was not called and Step 15 was not started. Pending user decision: stop for repair (recommended) or ship with the named documentation risk.\n\nInvocation record: ship-invocation.md (Checkpoints 1–3).\n",
"published": false
}
}
test/fixtures/detector-corpora/ship-docsync-late-result/inventory.json
Vendored
+423
View File
@@ -0,0 +1,423 @@
[
{
"run": "37174266054",
"lane": "gate",
"outcome": "passed",
"series_identity": "8fefdbc3860636ba",
"artifact": "native-captures-ci-37174266054-1-gate-census-8",
"available": true,
"entry": "37174266054-t1-pass.json"
},
{
"run": "37176837432",
"lane": "gate",
"outcome": "passed",
"series_identity": "8fefdbc3860636ba",
"artifact": "native-captures-ci-37176837432-1-gate-census-8",
"available": true,
"entry": "37176837432-t1-pass.json"
},
{
"run": "37179171083",
"lane": "gate",
"outcome": "passed",
"series_identity": "8fefdbc3860636ba",
"artifact": "native-captures-ci-37179171083-1-gate-census-8",
"available": true,
"entry": "37179171083-t1-pass.json"
},
{
"run": "37182865432",
"lane": "gate",
"outcome": "passed",
"series_identity": "fd4bf35072aa2f84",
"artifact": "native-captures-ci-37182865432-1-gate-census-8",
"available": true,
"entry": "37182865432-t1-pass.json"
},
{
"run": "37186854666",
"lane": "gate",
"outcome": "passed",
"series_identity": "2973ee01e37171be",
"artifact": "native-captures-ci-37186854666-1-gate-census-8",
"available": true,
"entry": "37186854666-t1-pass.json"
},
{
"run": "37193478719",
"lane": "gate",
"outcome": "passed",
"series_identity": "a280730f8acc5ece",
"artifact": "native-captures-ci-37193478719-1-gate-census-8",
"available": true,
"entry": "37193478719-t1-pass.json"
},
{
"run": "37195203538",
"lane": "gate",
"outcome": "passed",
"series_identity": "51be4310c4b2322c",
"artifact": "native-captures-ci-37195203538-1-gate-census-8",
"available": true,
"entry": "37195203538-t1-pass.json"
},
{
"run": "37198445662",
"lane": "gate",
"outcome": "passed",
"series_identity": "ca37c4d73e0aeeea",
"artifact": "native-captures-ci-37198445662-1-gate-census-8",
"available": true,
"entry": "37198445662-t1-pass.json"
},
{
"run": "37196811165",
"lane": "pr",
"outcome": "failed",
"series_identity": "51be4310c4b2322c",
"artifact": "native-captures-ci-37196811165-1-eval-slices-8",
"available": true,
"entry": "37196811165-t1-fail.json"
},
{
"run": "37199589136",
"lane": "pr",
"outcome": "passed",
"series_identity": "de47b356beabbe72",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37199579960",
"lane": "pr",
"outcome": "none",
"series_identity": null,
"artifact": "trial-outcomes-pr-a1",
"available": false,
"entry": null,
"reason": "run cancelled without a trial-outcomes-pr-a1 artifact; no late-result trial recorded"
},
{
"run": "37197708438",
"lane": "pr",
"outcome": "passed",
"series_identity": "ca37c4d73e0aeeea",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37196784022",
"lane": "pr",
"outcome": "none",
"series_identity": null,
"artifact": "trial-outcomes-pr-a1",
"available": false,
"entry": null,
"reason": "run cancelled without a trial-outcomes-pr-a1 artifact; no late-result trial recorded"
},
{
"run": "37194511147",
"lane": "pr",
"outcome": "passed",
"series_identity": "51be4310c4b2322c",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37192833320",
"lane": "pr",
"outcome": "passed",
"series_identity": "a280730f8acc5ece",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37189974078",
"lane": "pr",
"outcome": "passed",
"series_identity": "8b4e6898105a18a4",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37188668421",
"lane": "pr",
"outcome": "passed",
"series_identity": "2973ee01e37171be",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37188014112",
"lane": "pr",
"outcome": "none",
"series_identity": null,
"artifact": "trial-outcomes-pr-a1",
"available": false,
"entry": null,
"reason": "run cancelled without a trial-outcomes-pr-a1 artifact; no late-result trial recorded"
},
{
"run": "37186091144",
"lane": "pr",
"outcome": "passed",
"series_identity": "2973ee01e37171be",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37184467123",
"lane": "pr",
"outcome": "passed",
"series_identity": "fd4bf35072aa2f84",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37182208158",
"lane": "pr",
"outcome": "passed",
"series_identity": "fd4bf35072aa2f84",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37181717177",
"lane": "pr",
"outcome": "none",
"series_identity": null,
"artifact": "trial-outcomes-pr-a1",
"available": false,
"entry": null,
"reason": "run cancelled without a trial-outcomes-pr-a1 artifact; no late-result trial recorded"
},
{
"run": "37179168735",
"lane": "pr",
"outcome": "passed",
"series_identity": "8fefdbc3860636ba",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37178720668",
"lane": "pr",
"outcome": "none",
"series_identity": null,
"artifact": "trial-outcomes-pr-a1",
"available": false,
"entry": null,
"reason": "run cancelled without a trial-outcomes-pr-a1 artifact; no late-result trial recorded"
},
{
"run": "37178146922",
"lane": "pr",
"outcome": "none",
"series_identity": null,
"artifact": "trial-outcomes-pr-a1",
"available": false,
"entry": null,
"reason": "run cancelled without a trial-outcomes-pr-a1 artifact; no late-result trial recorded"
},
{
"run": "37176835584",
"lane": "pr",
"outcome": "passed",
"series_identity": "8fefdbc3860636ba",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37176237665",
"lane": "pr",
"outcome": "none",
"series_identity": null,
"artifact": "trial-outcomes-pr-a1",
"available": false,
"entry": null,
"reason": "run cancelled without a trial-outcomes-pr-a1 artifact; no late-result trial recorded"
},
{
"run": "37173609560",
"lane": "pr",
"outcome": "passed",
"series_identity": "8fefdbc3860636ba",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37172907071",
"lane": "pr",
"outcome": "passed",
"series_identity": "8fefdbc3860636ba",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37172266374",
"lane": "pr",
"outcome": "passed",
"series_identity": "8fefdbc3860636ba",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37170610789",
"lane": "pr",
"outcome": "passed",
"series_identity": "114e0e58a512d735",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37170258467",
"lane": "pr",
"outcome": "none",
"series_identity": null,
"artifact": "trial-outcomes-pr-a1",
"available": false,
"entry": null,
"reason": "run cancelled without a trial-outcomes-pr-a1 artifact; no late-result trial recorded"
},
{
"run": "37168819307",
"lane": "pr",
"outcome": "passed",
"series_identity": "6a91277ce79e930a",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37168384066",
"lane": "pr",
"outcome": "none",
"series_identity": null,
"artifact": "trial-outcomes-pr-a1",
"available": false,
"entry": null,
"reason": "run cancelled without a trial-outcomes-pr-a1 artifact; no late-result trial recorded"
},
{
"run": "37166586458",
"lane": "pr",
"outcome": "passed",
"series_identity": "6c73db545c893e55",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37164967731",
"lane": "pr",
"outcome": "passed",
"series_identity": "6c73db545c893e55",
"artifact": "trial-outcomes-pr-a1",
"available": true,
"entry": null,
"reason": "passing PR-lane trial on the wave branch (scanned for late-result reds); pass coverage comes from the gate-census captures"
},
{
"run": "37164663577",
"lane": "pr",
"outcome": "none",
"series_identity": null,
"artifact": "trial-outcomes-pr-a1",
"available": false,
"entry": null,
"reason": "run cancelled without a trial-outcomes-pr-a1 artifact; no late-result trial recorded"
},
{
"run": "37164023690",
"lane": "pr",
"outcome": "none",
"series_identity": null,
"artifact": "trial-outcomes-pr-a1",
"available": false,
"entry": null,
"reason": "run cancelled without a trial-outcomes-pr-a1 artifact; no late-result trial recorded"
},
{
"run": "37163862654",
"lane": "pr",
"outcome": "none",
"series_identity": null,
"artifact": "trial-outcomes-pr-a1",
"available": false,
"entry": null,
"reason": "run cancelled without a trial-outcomes-pr-a1 artifact; no late-result trial recorded"
},
{
"run": "37163438154",
"lane": "pr",
"outcome": "none",
"series_identity": null,
"artifact": "trial-outcomes-pr-a1",
"available": false,
"entry": null,
"reason": "run cancelled without a trial-outcomes-pr-a1 artifact; no late-result trial recorded"
},
{
"run": "37163388183",
"lane": "pr",
"outcome": "none",
"series_identity": null,
"artifact": "trial-outcomes-pr-a1",
"available": false,
"entry": null,
"reason": "run cancelled without a trial-outcomes-pr-a1 artifact; no late-result trial recorded"
},
{
"run": "37163226077",
"lane": "pr",
"outcome": "none",
"series_identity": null,
"artifact": "trial-outcomes-pr-a1",
"available": false,
"entry": null,
"reason": "run cancelled without a trial-outcomes-pr-a1 artifact; no late-result trial recorded"
},
{
"run": "37162847235",
"lane": "pr",
"outcome": "none",
"series_identity": null,
"artifact": "trial-outcomes-pr-a1",
"available": false,
"entry": null,
"reason": "run cancelled without a trial-outcomes-pr-a1 artifact; no late-result trial recorded"
}
]
test/fixtures/overlay-v4-replay-37198445662.json
Vendored
+396
View File
@@ -0,0 +1,396 @@
{
"source": {
"run": "37198445662",
"sha": "fd6854b",
"tier": "periodic",
"artifact": "paid-slice-26-a1",
"case": "overlay-harness-claude-dedicated-tools-vs-bash-sonnet",
"fixture": "claude-dedicated-tools-vs-bash-sonnet",
"attempt": 1,
"contract": 3
},
"excerpt": "Per trial: tool_use names in order and the native terminal result string. Everything else in the SDK stream is omitted.",
"originalOutcome": "FAIL under contract v3: overlay-off-0 answered src/api.ts: [] (wrong), was recorded as an invalid measurement, and the comparison became incomplete.",
"expected": {
"3": "fail",
"4": "pass"
},
"justification": "Contract v4 (D1, approved 2026-10-04) scores a completed OFF wrong answer as a correctness result; every ON trial is correct and the zero-Bash baseline gate holds.",
"trials": [
{
"arm": "overlay-on",
"trial": 0,
"tools": [
"Glob",
"Glob",
"Grep",
"Grep",
"Grep",
"Grep",
"Grep"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-on",
"trial": 1,
"tools": [
"Glob",
"Glob",
"Grep",
"Grep",
"Grep",
"Grep",
"Grep",
"Read"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-on",
"trial": 2,
"tools": [
"Glob",
"Glob",
"Read",
"Read",
"Read",
"Read",
"Read"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-on",
"trial": 3,
"tools": [
"Glob",
"Glob",
"Grep",
"Grep",
"Grep",
"Grep",
"Grep",
"Read"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-on",
"trial": 4,
"tools": [
"Glob",
"Glob",
"Grep"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-on",
"trial": 5,
"tools": [
"Glob",
"Glob",
"Grep",
"Grep",
"Grep",
"Grep",
"Grep"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-on",
"trial": 6,
"tools": [
"Glob",
"Glob",
"Grep",
"Grep",
"Grep",
"Grep",
"Grep"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-on",
"trial": 7,
"tools": [
"Glob",
"Glob",
"Grep",
"Grep",
"Grep",
"Grep",
"Grep",
"Read"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-on",
"trial": 8,
"tools": [
"Glob",
"Grep"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-on",
"trial": 9,
"tools": [
"Glob",
"Glob",
"Grep",
"Grep",
"Grep",
"Grep",
"Grep",
"Read"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-off",
"trial": 0,
"tools": [
"Glob",
"Glob",
"Grep",
"Grep",
"Grep",
"Grep",
"Grep"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": false,
"taskCorrect": false,
"exitReason": "validation_failed"
}
},
{
"arm": "overlay-off",
"trial": 1,
"tools": [
"Glob",
"Glob",
"Grep",
"Grep",
"Grep",
"Grep",
"Grep"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/types.ts\":[\"Foo\"],\"src/api.ts\":[\"fetchFoo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-off",
"trial": 2,
"tools": [
"Glob",
"Glob",
"Grep",
"Grep",
"Grep",
"Grep",
"Grep"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-off",
"trial": 3,
"tools": [
"Glob",
"Glob",
"Grep",
"Grep",
"Grep",
"Grep",
"Grep"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-off",
"trial": 4,
"tools": [
"Glob",
"Glob",
"Grep",
"Grep",
"Grep",
"Grep",
"Grep"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-off",
"trial": 5,
"tools": [
"Glob",
"Glob",
"Grep",
"Grep",
"Grep",
"Grep",
"Grep"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-off",
"trial": 6,
"tools": [
"Glob",
"Glob",
"Grep",
"Grep",
"Grep",
"Grep",
"Grep"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-off",
"trial": 7,
"tools": [
"Glob",
"Glob",
"Grep",
"Grep",
"Grep",
"Grep",
"Grep"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-off",
"trial": 8,
"tools": [
"Glob",
"Glob",
"Grep",
"Grep",
"Grep",
"Grep",
"Grep"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
},
{
"arm": "overlay-off",
"trial": 9,
"tools": [
"Glob",
"Glob",
"Grep",
"Grep",
"Grep",
"Grep",
"Grep"
],
"answer": "{\"src/index.ts\":[\"x\"],\"src/util.ts\":[\"util\"],\"src/config.ts\":[\"c\"],\"src/api.ts\":[\"fetchFoo\"],\"src/types.ts\":[\"Foo\"]}",
"v3": {
"measurementPassed": true,
"taskCorrect": true,
"exitReason": "success"
}
}
]
}
test/fixtures/plan-floor-edit-wrap-37176835584.json
Vendored
+24
View File
@@ -0,0 +1,24 @@
{
"source": {
"run": "37176835584",
"sha": "7249f014648202840af294b730ac244e03ce514c",
"lane": "pr",
"artifact": "paid-slice-5-a1",
"case": "plan-ceo-finding-floor",
"file": "shards/skill-e2e-plan-ceo-finding-floor/pty-count/ci-37176835584-1-eval-slices-5/plan-ceo-review-1791087894192-NWi6KI/terminal.screen.log"
},
"originalOutcome": "FAIL: timeout at 608735 ms with this owned Edit menu on screen for its last ~5 minutes; the floor guard never granted it.",
"expected": "currentEditPreview binds this viewport to the pending owned Edit (unchanged rows soft-wrap with a blank marker column); a changed row is still refused.",
"justification": "Context rows wrap under a six-column gutter with no +/- marker; the parser accepted wrapped +/- rows only, so the exact owned edit could not be verified.",
"limits": "Viewport and prior Write content are verbatim from the artifact. The pending Edit input is reconstructed from the visible diff (row 209: \"Effort S-M\" -> \"Effort M\"); the retained public transcript ends at the Read at 04:28:46Z and never shows this Edit.",
"cwd": "/home/runner/.cache/gstack-paid-shard-kSkyC8/tmp/gstack-plan-count-NEEunu",
"basename": "gstack-test-plan-ceo-floor.md",
"screen": " 206 A) Launch tier as written\n 207 Build and launch the developer tier now on the \"feels cheaper\" assumption; measure signups afterward. Effort M (hu\n man: ~4-6 weeks billing + pricing page + launch / CC: ~2 hours config and copy, plus human launch coordination), r\n isk high, reuse: existing billing provider if any, verification: signup count only. ✅ Fastest path to a shipped t\n ier and a visible market signal. ✅ No research work; the team's existing conviction is enough to start. ❌ Cannot\n distinguish \"price was the barrier\" from \"any change raises signups\"; no baseline or control exists. ❌ Price cut\n s are hard to reverse without customer backlash, so a wrong guess is sticky.\n 208 B) Validate barrier first, then launch (recommended)\n 209 -Run 8-12 developer interviews drawn from pricing-page drop-offs and churned trials, pull pricing-page → signup → f\n -irst-success funnel and churn-reason data, scan competitor prices, and write the go/no-go rule before collecting d\n -ata. If price is the top blocker for a pre-set share of respondents, launch the tier shaped by that evidence; othe\n -rwise redirect to the barrier found. Effort S-M (human: ~2 weeks / CC: ~1 hour for guide, queries and analysis tem\n -plate), risk low, reuse: existing analytics and churn surveys, verification: pre-registered threshold met or not. \n -✅ Tests the plan's own stated premise directly before spending revenue or engineering time. ✅ Produces a baselin\n -e, so the launch can be measured against something. ✅ Covers cannibalization of existing plans, which A and C ski\n -p. ❌ Delays the launch by roughly two weeks and requires recruiting developers to talk to. \n 209 +Run 8-12 developer interviews drawn from pricing-page drop-offs and churned trials, pull pricing-page → signup → f\n +irst-success funnel and churn-reason data, scan competitor prices, and write the go/no-go rule before collecting d\n +ata. If price is the top blocker for a pre-set share of respondents, launch the tier shaped by that evidence; othe\n +rwise redirect to the barrier found. Effort M (human: ~2 weeks / CC: ~1 hour for guide, queries and analysis templ\n +ate), risk low, reuse: existing analytics and churn surveys, verification: pre-registered threshold met or not. ✅\n + Tests the plan's own stated premise directly before spending revenue or engineering time. ✅ Produces a baseline,\n + so the launch can be measured against something. ✅ Covers cannibalization of existing plans, which A and C skip.\n + ❌ Delays the launch by roughly two weeks and requires recruiting developers to talk to. \n 210 C) Smoke-test intent on pricing page first\n 211 Publish the developer tier on the pricing page as an intent test (a waitlist or \"coming soon\" CTA) before building\n billing; set a click-through threshold in advance; build only if it is met. Effort S (human: ~1 week / CC: ~30 mi\n n copy and event tracking), risk medium, reuse: existing pricing page and analytics, verification: CTR versus thre\n shold. ✅ Quantitative demand signal in days without touching billing. ✅ Cheap to retract if intent is weak. ❌ M\n easures interest in a cheaper label, not whether price is the adoption barrier, and tells you nothing about why de\n velopers currently drop off. ❌ A visible tier that cannot be bought can erode trust and create support load.\n 212 \n╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌\n Do you want to make this edit to gstack-test-plan-ceo-floor.md?\n ❯ 1. Yes\n 2. Yes, and switch to accept edits (auto-approve file edits and common file commands) for this session (shift+tab)\n 3. No\n\n Esc to cancel · Tab to amend\n",
"screenSha256": "b51da216d016430436f0d1955ec7f9f169d2b350307df89e2623b413659fec56",
"priorWrite": "# CEO Plan Review: Launch a \"developer-friendly\" pricing tier\n\nWorking plan for `/plan-ceo-review PLAN.md`. Generated 2026-10-04 on branch `main`\n(base branch: `main`, git-native fallback; repo has no remote). Review only; no code\nor plan scope is changed without an explicit recorded answer.\n\n## Plan under review (verbatim, PLAN.md lines 11-21)\n\n> # Plan: Launch a \"developer-friendly\" pricing tier\n>\n> ## Goal\n> Increase developer adoption.\n>\n> ## Success metric\n> More signups.\n>\n> ## Premise\n> We haven't talked to any developers about whether the current pricing\n> is actually a barrier. The team agreed it \"feels like\" it should be cheaper.\n\n## Request constraints (from CLAUDE.md / PLAN.md lines 1-9)\n\n| Constraint | Source | Effect on this review |\n|---|---|---|\n| Skip the `/office-hours` prerequisite | PLAN.md line 1 | No design doc; office-hours offer not made |\n| Routing setup, cross-project recall, outside reviewers already declined | PLAN.md line 3 | `ROUTING_DECLINED: true`; `cross_project_learnings=false`; `codex_reviews=disabled` |\n| Preserve the supplied product scope; review mode HOLD SCOPE | PLAN.md line 5 | Mode is an explicit user choice; no expansion proposals |\n| Preserving scope does not approve premise, approach or any remedy | PLAN.md line 7 | Premise, metric and approach stay pending until answered |\n| Write the plan-mode plan to this file | PLAN.md line 9 | Storage policy: this file is the single working plan |\n\n## Storage policy\n\nWorking plan = this file (explicitly requested output). Native Write created it;\nscoped Edits for checkpoints. Metadata/history writes (`gstack-review-log`,\n`gstack-decision-log`, `gstack-question-log`) are best-effort and reported with\ntheir actual outcome. 0H CEO archive is skipped by mode (HOLD SCOPE).\n\n## Review depth\n\n**Strategy-only.** PLAN.md contains no implementation design: no system, files,\nbilling provider, price point, tier limits or codepaths are named. Implementation-ready\ndepth would require inventing method contracts, which review-sections forbids, and\nHOLD SCOPE forbids adding implementation scope without approval. Sections 1-10 use\ncapability-level rows and \"implementation owner must prove ___\" notes. Expanding to\nimplementation design needs a 0D answer (A: keep strategy-only / B: add design for a\nnamed capability); no concrete blocker currently requires B.\n\n## Pre-review system audit\n\n| Check | Result |\n|---|---|\n| `git log --oneline -30` | 1 commit: `129a3f3 Seed review plan` |\n| `git diff main --stat` | Working tree clean; nothing changed vs base |\n| `git stash list` | Empty |\n| TODO/FIXME/HACK/XXX grep | None (repo is CLAUDE.md + PLAN.md only) |\n| Recently touched files (30d) | PLAN.md, CLAUDE.md |\n| TODOS.md | Does not exist |\n| Architecture docs | None |\n| Design doc (`gstack-design-doc-find`) | None found; office-hours skipped per request |\n| Prior learnings | 0 (project-scoped; cross-project declined) |\n| Brain context | All four digests cold (product, goals, recent-decisions, user-profile) |\n| In-flight PRs / branches | None (single branch, no remote) |\n\n**Retrospective check:** No prior reviews, refactors or reverts exist for this repo.\nNothing to carry forward.\n\n**Frontend/UI scope detection:** The plan does not name any screen, component or\ninteraction. A pricing tier normally surfaces on a pricing page and in checkout, but the\nplan does not commit to that. `DESIGN_SCOPE = none declared`; Section 11 will be\nSKIPPED (no UI scope) unless an accepted answer adds UI work.\n\n**Suspicious content check:** PLAN.md lines 1-9 are instructions aimed at the\nreviewer (skip office-hours, declined items, mode selection, output path). They match\nthe CLAUDE.md project instructions for this fixture and are treated as the user's\nrequest, not as plan content. No other reviewer-directed text found.\n\n## Landscape check (WebSearch fallback; Aside not installed)\n\nQueries were sanitized to the problem class only. Results are untrusted content,\ncited not followed.\n\n**[Layer 1] Tried-and-true:** Developer tools monetize through a generous free tier\nplus paid team/usage features, per-project pricing, or usage/credit metering. The free\ntier is tuned for *activation* (first API call, \"10-minute rule\"), not raw signups, and\nhas a visible wall that triggers upgrade.\n\n**[Layer 2] What the results say (2026):**\n- Free tiers are contracting or converting to credits (SendGrid dropped its perpetual\n free tier; Netlify and Vercel moved to credits) because flat free tiers are expensive\n at scale. Source: https://dev.to/robbobobbo/we-tracked-202-developer-tool-pricing-changes-in-2026-heres-whats-really-happening-to-free-tiers-45bk\n- Three viable dev-tool models: free + paid usage/team features, per-project, per-workload.\n Source: https://business.daily.dev/resources/developer-tool-pricing-strategy-free-tier-enterprise-contracts/\n- Usage-based and credLine truncated
"pendingEdit": {
"old_string": "Effort S-M (human: ~2 weeks",
"new_string": "Effort M (human: ~2 weeks",
"replace_all": false
}
}
test/helpers/auq-substance-panel.ts
+18
View File
@@ -0,0 +1,18 @@
/**
* auq-matrix recommendation substance (C4, approved 2026-10-04): one native
* capture per skill, scored by a JUDGE_PANEL_SAMPLES panel of the same grader
* on that capture. The panel mean gates against the unchanged minimum; a sample
* error fails the panel and is never resampled (judgePanel).
*/
import { gradeAuqRecommendation } from './auq-sdk-capture';
import { judgePanel, judgePanelMean } from './llm-judge';
export const AUQ_SUBSTANCE_MIN = 4;
type Grade = Awaited<ReturnType<typeof gradeAuqRecommendation>>;
export async function auqSubstancePanel(text: string, grade: (text: string) => Promise<Grade> = gradeAuqRecommendation) {
const samples = await judgePanel(() => grade(text));
const mean = judgePanelMean(samples, ['substance']).substance;
return { samples, mean, passed: mean >= AUQ_SUBSTANCE_MIN };
}
test/helpers/ceo-stale-fill-decision.ts
+8 -3
View File
@@ -12,6 +12,11 @@
const LEDGER_HEADER = /^\|\s*ID and owner\s*\|\s*Contract and evidence\s*\|\s*Current\s*\|\s*Proposed\s*\|\s*Status\s*\|\s*Exact approval and scope\s*\|\s*$/;
const FILL = /\b(?:fill\w*|refill\w*|repopulat\w*)\b|\bcache\.set\b/i;
const WRITE = /\b(?:writes?|written|writer|commit\w*)\b/i;
const CAMEL_WRITE = /\b[wW]rite[A-Z]\w*/;
const ROW_ID = String.raw`[A-Za-z][\w-]*(?:\s*\/\s*[A-Za-z][\w-]*)*`;
const ROW_ID_CELL = new RegExp(String.raw`^(${ROW_ID})(?=\s|$)`);
const DECISION_HEADING = new RegExp(String.raw`^#{2,3}\s+currentDecision \((${ROW_ID})\)\s*$`);
const normalizeId = (id: string | undefined) => id?.replace(/\s*\/\s*/g, '/');
const RACE = /\b(?:stale|race|racing|overwrit\w*|newer|old|older|before|after|overlap\w*|in[- ]flight|concurrent\w*)\b/i;
/** Asserted lines only: fenced blocks, block quotes and indented code become blank. */
@@ -36,7 +41,7 @@ function ledgerRows(lines: string[]): Map<string, string> {
if (!LEDGER_HEADER.test(lines[i]!)) continue;
for (let j = i + 2; j < lines.length && lines[j]!.trimStart().startsWith('|'); j++) {
const cells = lines[j]!.trim().slice(1, -1).split('|').map(cell => cell.trim());
const id = /^([A-Za-z][\w-]*)(?=\s|$)/.exec(cells[0] ?? '')?.[1];
const id = normalizeId(ROW_ID_CELL.exec(cells[0] ?? '')?.[1]);
// A row that does not split into the six ledger cells is not a record.
if (!id || cells.length !== 6) continue;
rows.set(id, cells[4]!);
@@ -50,12 +55,12 @@ export function hasApprovedStaleFillDecision(report: string): boolean {
const lines = proseLines(report);
const rows = ledgerRows(lines);
for (let i = 0; i < lines.length; i++) {
const rowId = /^#{2,3}\s+currentDecision \(([A-Za-z][\w-]*)\)\s*$/.exec(lines[i]!)?.[1];
const rowId = normalizeId(DECISION_HEADING.exec(lines[i]!)?.[1]);
if (!rowId) continue;
for (let j = i + 1; j < lines.length && !/^#{1,6}\s/.test(lines[j]!); j++) {
const subject = /^Question:\s*D[1-9]\d*\s*[—–-]\s*(.+)$/.exec(lines[j]!)?.[1];
if (!subject) continue;
if (FILL.test(subject) && WRITE.test(subject) && RACE.test(subject) && /^[*_]*approved\b/i.test(rows.get(rowId) ?? '')) return true;
if (FILL.test(subject) && (WRITE.test(subject) || CAMEL_WRITE.test(subject)) && RACE.test(subject) && /^[*_]*approved\b/i.test(rows.get(rowId) ?? '')) return true;
break;
}
}
test/helpers/native-auto-decide.ts
+1 -1
View File
@@ -44,7 +44,7 @@ function publicProse(text: string): string {
// The same current-mode field grammar owns declarations and later corrections.
function modeField(line: string): { value: string; completed: boolean } | null {
const match = /^(?:(?:Correction|Actually|Update):\s*)?(?<label>(?:Review )?Mode(?: decision)?|Decision)(?: (?<status>[^:\r\n]+))?:\s*(?<value>.*)$/i.exec(line.replace(/^\s*[-*+]\s+/, '').trim());
const match = /^(?:(?:Correction|Actually|Update):\s*)?(?<label>(?:Review )?Mode(?: decision)?|Decision)(?: (?<status>(?:(?![.!?](?:\s|$))[^:\r\n])+))?:\s*(?<value>.*)$/i.exec(line.replace(/^\s*[-*+]\s+/, '').trim());
if (!match) return null;
// "Mode" and "Mode decision" are both field labels. If an explicit status
// follows, only the completion class (including decided/selected/chosen) is
test/helpers/overlay-attempt.ts
+14 -5
View File
@@ -10,6 +10,8 @@ export interface OverlayTrialOutcome {
passed: boolean;
taskCorrect: boolean;
metric?: number;
/** Contract v4: a completed answer that failed the fixture's output check. */
answerError?: string;
result?: AgentSdkResult;
error?: string;
exitReason: string;
@@ -57,13 +59,20 @@ export async function runOverlayTrial(options: {
(fixture.comparison.maximum !== undefined && metric > fixture.comparison.maximum)))) {
throw new Error(`invalid fixture metric: ${metric}`);
}
fixture.verify?.(result, directory, metric);
let answerError: string | undefined;
try { fixture.verify?.(result, directory, metric); }
catch (error) { answerError = error instanceof Error ? `${error.name}: ${error.message}` : String(error); }
checkActive();
// Implementation oracles execute agent-written code. Their side effects
// must obey the same scope contract and appear in the retained snapshot.
after = snapshotWorkspace(directory);
assertWorkspaceChanges(before, after, fixture.allowedChanges ?? []);
outcome = { passed: true, taskCorrect: fixture.taskCorrect?.(metric) ?? true, metric, result, exitReason: result.exitReason, before, after };
// Contract v4: a wrong completed answer is a measured correctness result in
// either arm, not an invalid measurement.
outcome = {
passed: true, taskCorrect: !answerError && (fixture.taskCorrect?.(metric) ?? true),
metric, result, exitReason: result.exitReason, before, after, ...(answerError ? { answerError } : {}),
};
} catch (error) {
// Capture post-failure files when possible; preserve the first failure if
// the workspace is absent or evidence capture itself cannot complete.
@@ -98,11 +107,11 @@ export function assessOverlayArms(fixture: OverlayFixture, overlay: OverlayTrial
off: off.filter((trial) => trial?.passed).map((trial) => trial.metric!),
};
const comparison = assessComparison(metrics, fixture.trials, fixture.comparison, fixture.pass);
// Only execution, scope and metric failures invalidate a trial, in both arms.
const measurementsValid = comparison.status !== 'incomplete' && [overlay, off].every((arm) =>
arm.length === fixture.trials && Array.from(arm).every((trial) => trial?.passed));
// Baseline completion is an experimental variable for literal scope. A valid
// OFF sample may score 0..3; ON must complete all three. Every ON trial has
// already passed its fixture's output check to count as a valid measurement.
// Task correctness is a measured result in both arms (OFF may score 0..3 or
// answer wrongly); only ON correctness gates the case.
const correctnessPassed = measurementsValid && overlay.every((trial) => trial.taskCorrect);
// A fixture with a gate also blocks on its comparison (dedicated tools: 20%
// fewer Bash calls, or none at a zero baseline). Without a gate the
test/helpers/overlay-case-policy.ts
+5 -2
View File
@@ -1,11 +1,14 @@
/** Version 3 gates supported behavior, plus the comparison for fixtures that
/** Version 4 keeps v3's gate but scores a completed wrong answer in either arm
* as a correctness result; only execution, scope and metric failures invalidate
* a trial. An OFF wrong answer no longer fails the comparison.
* Version 3 gates supported behavior, plus the comparison for fixtures that
* declare a gate (dedicated tools: correct output and 20% fewer Bash calls,
* replacing v2's zero Bash in every ON trial). Other comparisons stay research-only.
* Version 1 required efficacy lift. Never reinterpret recorded failures of earlier versions.
*/
export const OVERLAY_CONTRACT = {
name: 'overlay-behavior',
version: 3,
version: 4,
releaseGate: 'execution_scope_complete_sampling_on_correctness_and_fixture_gate',
comparisonRole: 'research_only_unless_fixture_gate',
resourceNonRegression: 'not_established',
test/helpers/overlay-case.ts
+2 -2
View File
@@ -46,7 +46,7 @@ function saveTrial(fixture: OverlayFixture, attempt: number, arm: Arm, n: number
// this JSONL remains the final SDK result used for the trial measurement.
rateLimitAttemptStreamsRetained: true,
measurementPassed: outcome.passed, taskCorrect: outcome.taskCorrect, metric: outcome.metric, exitReason: outcome.exitReason,
error: outcome.error, before: outcome.before, after: outcome.after,
error: outcome.error, answerError: outcome.answerError, before: outcome.before, after: outcome.after,
}, null, 2) + '\n', { flag: 'wx' });
}
@@ -143,7 +143,7 @@ export function registerOverlayCase(fixtureId: string): void {
name: `${fixture.id}-contract-v${OVERLAY_CONTRACT.version}-${key}`, suite: 'overlay-harness-measurement', tier: 'e2e',
passed: outcome.passed, duration_ms: sdk?.durationMs ?? 0, cost_usd: sdk?.costUsd ?? 0,
transcript: sdk?.events, prompt: fixture.userPrompt,
output: JSON.stringify({ contract: OVERLAY_CONTRACT, measurementPassed: outcome.passed, taskCorrect: outcome.taskCorrect, metric: outcome.metric, assistantOutput: sdk?.output }),
output: JSON.stringify({ contract: OVERLAY_CONTRACT, measurementPassed: outcome.passed, taskCorrect: outcome.taskCorrect, answerError: outcome.answerError, metric: outcome.metric, assistantOutput: sdk?.output }),
turns_used: sdk?.turnsUsed, browse_errors: sdk?.browseErrors,
exit_reason: outcome.exitReason, error: outcome.error,
model: sdk?.model ?? fixture.model, first_response_ms: sdk?.firstResponseMs, max_inter_turn_ms: sdk?.maxInterTurnMs,
test/helpers/plan-count-file-permission.ts
+13 -6
View File
@@ -346,21 +346,28 @@ function currentEditPreview(preview: string, r: any, config: string, cwd: string
const firstLine = before.slice(0, at).split(/\r?\n/).length;
const oldLast = firstLine + input.old_string.split(/\r?\n/).length - 1;
const newLast = firstLine + input.new_string.split(/\r?\n/).length - 1;
const rows: Array<{line: number; kind: string; text: string; clipped?: boolean}> = [];
const rows: Array<{line: number; kind: string; text: string; clipped?: boolean; gutter?: number}> = [];
let leading: {kind: string; text: string} | undefined;
for (const line of preview.split('\n')) {
if (!line.trim() || /^[╌─━]{3,}[ \t]*$/.test(line)) continue;
const numbered = /^ {0,3}([1-9]\d*) ([ +\-])(.*)$/.exec(line);
// A trimmed empty unchanged row keeps only its number.
const numbered = /^( {0,3}([1-9]\d*) )([ +\-])(.*)$/.exec(line) ?? /^( {0,3}([1-9]\d*))()[ \t]*$/.exec(line);
if (numbered) {
const kind = numbered[3] || ' ';
if (leading) {
// A wrapped first row has no coordinate. The next same-kind numbered
// row anchors its complete visible suffix to the preceding source line.
if (leading.kind !== numbered[2] || Number(numbered[1]) < 2) return false;
rows.push({line:Number(numbered[1])-1,...leading,clipped:true}); leading=undefined;
if (leading.kind !== kind || Number(numbered[2]) < 2) return false;
rows.push({line:Number(numbered[2])-1,...leading,clipped:true}); leading=undefined;
}
rows.push({line:Number(numbered[1]),kind:numbered[2]!,text:numbered[3]!}); continue;
rows.push({line:Number(numbered[2]),kind,text:numbered[4] ?? '',gutter:numbered[1]!.length}); continue;
}
const wrapped = /^ {4,5}([+\-])(.*)$/.exec(line), last = rows.at(-1);
const last = rows.at(-1);
// An unchanged row wraps under its own gutter with a blank marker column.
if (last?.kind === ' ' && last.gutter && line.startsWith(' '.repeat(last.gutter + 1))) {
last.text += line.slice(last.gutter + 1); continue;
}
const wrapped = /^ {4,5}([+\-])(.*)$/.exec(line);
if (!wrapped) return false;
if (!last) {
if (leading && leading.kind !== wrapped[1]) return false;
test/overlay-measurement.test.ts
+61 -6
View File
@@ -5,6 +5,7 @@ import * as path from 'path';
import type { AgentSdkResult, QueryProvider } from './helpers/agent-sdk-runner';
import { firstAssistantMessageToolCount, reportedThinkingTokens, assessComparison, trialArtifactStem } from './helpers/overlay-measurement';
import { runOverlayTrial, awaitOverlayWorkers, assessOverlayArms, captureOverlayQueryAttempts, type OverlayTrialOutcome } from './helpers/overlay-attempt';
import { OVERLAY_CONTRACT } from './helpers/overlay-case-policy';
import { setupLiteralWorkspace, correctLiteralTargets, snapshotWorkspace, assertReadOnlyWorkspace } from './helpers/overlay-workspace';
import { fanoutPass, higherIsBetter20Pct, lowerIsBetter20Pct, lowerIsBetter20PctOrZeroBaseline, OVERLAY_FIXTURES, type OverlayFixture } from './fixtures/overlay-nudges';
@@ -66,9 +67,10 @@ describe('SDK overlay measurements', () => {
describe('correctness versus efficacy', () => {
describe.each(['claude-opus-4-7', 'claude-sonnet-4-6'])('dedicated-tool gate for %s', (model) => {
const f = { ...OVERLAY_FIXTURES.find(f => f.metricName === 'bash_tool_calls' && f.model === model)!, trials: 3 };
// A trial whose final JSON fails the fixture's verify is recorded as a failed measurement.
// Contract v4: a trial whose final JSON fails the fixture's verify is a valid
// measurement with a wrong answer, not an invalid measurement.
const correct = (metric: number): OverlayTrialOutcome => ({ passed: true, taskCorrect: true, metric, exitReason: 'success' });
const incorrect = (metric: number): OverlayTrialOutcome => ({ passed: false, taskCorrect: false, metric, exitReason: 'validation_failed' });
const incorrect = (metric: number): OverlayTrialOutcome => ({ passed: true, taskCorrect: false, metric, exitReason: 'success', answerError: 'AssertionError: final answer does not match the fixture task' });
const arm = (...trials: OverlayTrialOutcome[]) => trials;
test('fixture gates on the comparison and on output correctness, not on zero ON Bash', () => {
expect(f.taskCorrect).toBeUndefined();
@@ -88,9 +90,25 @@ describe('correctness versus efficacy', () => {
gatePassed: false, passed: false, comparison: { status: 'no_measured_improvement' },
});
});
test('a 20% reduction with incorrect output fails', () => {
test('a 20% reduction with incorrect ON output fails', () => {
expect(assessOverlayArms(f, arm(correct(0), incorrect(0), correct(0)), arm(correct(5), correct(5), correct(5)))).toMatchObject({
measurementsValid: false, passed: false,
measurementsValid: true, correctnessPassed: false, passed: false,
});
});
test('an OFF wrong answer is a comparison result, not an invalid measurement', () => {
expect(assessOverlayArms(f, arm(correct(0), correct(0), correct(0)), arm(incorrect(0), correct(0), correct(0)))).toMatchObject({
measurementsValid: true, correctnessPassed: true, gatePassed: true, passed: true, comparison: { status: 'baseline_saturated' },
});
});
test('an execution failure invalidates the measurement in either arm', () => {
const failed: OverlayTrialOutcome = { passed: false, taskCorrect: false, metric: 0, exitReason: 'error_max_turns', error: 'Error: SDK execution failed: error_max_turns' };
for (const [on, off] of [[arm(failed, correct(0), correct(0)), arm(correct(0), correct(0), correct(0))], [arm(correct(0), correct(0), correct(0)), arm(failed, correct(0), correct(0))]]) {
expect(assessOverlayArms(f, on, off)).toMatchObject({ measurementsValid: false, correctnessPassed: false, gatePassed: false, passed: false, comparison: { status: 'incomplete' } });
}
});
test('an OFF wrong answer does not rescue a failed comparison gate', () => {
expect(assessOverlayArms(f, arm(correct(6), correct(6), correct(6)), arm(incorrect(5), correct(5), correct(5)))).toMatchObject({
measurementsValid: true, correctnessPassed: true, gatePassed: false, passed: false, comparison: { status: 'regressed' },
});
});
test('a zero-Bash baseline passes only when every ON trial is also zero', () => {
@@ -288,10 +306,13 @@ describe('complete trial lifecycle', () => {
}, record: (entry) => records.push(entry) });
expect(records).toHaveLength(1);
expect(records[0]).toBe(outcome);
expect(outcome.passed).toBe(scenario === 'success');
// Contract v4: a failed output check is a completed wrong answer, not an invalid measurement.
expect(outcome.passed).toBe(scenario === 'success' || scenario === 'assertion');
expect(outcome.taskCorrect).toBe(scenario === 'success');
if (scenario === 'assertion') expect(outcome).toMatchObject({ exitReason: 'success', answerError: 'Error: required output absent' });
if (scenario === 'runner_error' || scenario === 'setup_error') expect(outcome.exitReason).toBe('harness_error');
if (scenario === 'max_turns') expect(outcome.exitReason).toBe('error_max_turns');
if (scenario !== 'success') expect(outcome.error).toBeTruthy();
if (scenario !== 'success' && scenario !== 'assertion') expect(outcome.error).toBeTruthy();
if (scenario !== 'runner_error' && scenario !== 'setup_error') expect(outcome.result?.costUsd).toBe(0.03);
} finally { fs.rmSync(dir, { recursive: true, force: true }); }
});
@@ -311,6 +332,40 @@ describe('complete trial lifecycle', () => {
});
});
describe('contract v4 replay of census 37198445662', () => {
const corpus = JSON.parse(fs.readFileSync(path.join(import.meta.dir, 'fixtures', 'overlay-v4-replay-37198445662.json'), 'utf8')) as {
source: { fixture: string }; trials: Array<{ arm: 'overlay-on' | 'overlay-off'; trial: number; tools: string[]; answer: string; v3: { measurementPassed: boolean } }>;
};
const f = OVERLAY_FIXTURES.find(candidate => candidate.id === corpus.source.fixture)!;
const replay = async (trial: (typeof corpus.trials)[number]) => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'overlay-v4-replay-'));
try {
return await runOverlayTrial({ fixture: f, directory: dir, record: () => {}, invoke: async () => result({
events: [systemInit(), { type: 'result', subtype: 'success', result: trial.answer }] as AgentSdkResult['events'],
toolCalls: trial.tools.map(tool => ({ tool, input: {}, output: '' })), output: trial.answer, model: f.model,
}) });
} finally { fs.rmSync(dir, { recursive: true, force: true }); }
};
test('the stored OFF wrong answer is a measured result and the case passes under v4', async () => {
const outcomes = await Promise.all(corpus.trials.map(replay));
const arm = (name: string) => outcomes.filter((_, index) => corpus.trials[index].arm === name);
expect(corpus.trials.filter(trial => !trial.v3.measurementPassed).map(trial => `${trial.arm}-${trial.trial}`)).toEqual(['overlay-off-0']);
expect(outcomes.every(outcome => outcome.passed)).toBe(true);
expect(arm('overlay-off').map(outcome => outcome.taskCorrect)).toEqual([false, ...Array(9).fill(true)]);
expect(arm('overlay-off')[0].answerError).toContain('does not match the fixture task');
expect(assessOverlayArms(f, arm('overlay-on'), arm('overlay-off'))).toMatchObject({
measurementsValid: true, correctnessPassed: true, gatePassed: true, passed: true, comparison: { status: 'baseline_saturated', meanOn: 0, meanOff: 0 },
});
expect(OVERLAY_CONTRACT.version).toBe(4);
});
test('the same wrong answer in the ON arm still fails the case', async () => {
const swapped = corpus.trials.map(trial => ({ ...trial, arm: trial.arm === 'overlay-on' ? 'overlay-off' as const : 'overlay-on' as const }));
const outcomes = await Promise.all(swapped.map(replay));
const arm = (name: string) => outcomes.filter((_, index) => swapped[index].arm === name);
expect(assessOverlayArms(f, arm('overlay-on'), arm('overlay-off'))).toMatchObject({ measurementsValid: true, correctnessPassed: false, passed: false });
});
});
describe('retry-specific native evidence', () => {
test('retains events preceding an exception and a separate successful retry stream', async () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'overlay-capture-unit-'));
test/plan-floor-edit-wrap.test.ts
+61
View File
@@ -0,0 +1,61 @@
import {expect,test} from 'bun:test';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import {createHash} from 'node:crypto';
import {createFilePermissionRecorder,currentFilePermissionEpoch} from './helpers/plan-count-file-permission';
import {readPlanCountTranscript} from './helpers/plan-count-transcript';
import {createPlanCountPermissionGuard} from './helpers/claude-pty-runner';
import captured from './fixtures/plan-floor-edit-wrap-37176835584.json';
// PR run 37176835584: the floor's owned Edit menu stayed on screen until the
// 600 s timeout because unchanged rows soft-wrap under a gutter with a blank
// marker column, which the cropped-preview parser refused.
function fixture(){
const dir=fs.mkdtempSync(path.join(os.tmpdir(),'floor-edit-wrap-')),cwd=path.join(dir,'owned'),config=path.join(dir,'config');fs.mkdirSync(cwd);
const expected=path.join(cwd,captured.basename),sid='46bdc2d6-b244-41d6-a97d-2694f1639cb2',id='toolu_floorEditWrap37176835584';
const journal=path.join(config,'projects','owned',sid+'.jsonl');fs.mkdirSync(path.dirname(journal),{recursive:true});
const recorder=createFilePermissionRecorder(cwd,config,expected)!;
const startedAt=Date.now()-60_000,timestamp=new Date(startedAt+1000).toISOString();
fs.writeFileSync(recorder.file,JSON.stringify({cwd,expected,sessionId:sid,transcriptPath:journal,seenIds:[`${sid}:${id}`],
pendingId:`${sid}:${id}`,completedId:null,completedIds:[],timestamp}));
fs.writeFileSync(expected,captured.priorWrite);
const block={type:'tool_use',id,name:'Edit',input:{...captured.pendingEdit,file_path:expected}};
fs.writeFileSync(journal,JSON.stringify({cwd,sessionId:sid,isSidechain:false,timestamp,message:{role:'assistant',content:[{type:'text',text:'Updating the effort estimate.'},block]}})+'\n');
const read=(screen=captured.screen)=>currentFilePermissionEpoch(recorder.file,expected,cwd,config,startedAt,readPlanCountTranscript(config,cwd),screen);
return {pendingId:`${sid}:${id}`,expected,read,close(){recorder.dispose();fs.rmSync(dir,{recursive:true,force:true});}};
}
test('the captured floor viewport with wrapped unchanged rows binds the owned Edit and grants once',()=>{
expect(createHash('sha256').update(captured.screen).digest('hex')).toBe(captured.screenSha256);
expect(captured.screen).toMatch(/^ {6}man: ~4-6 weeks/m);
const f=fixture();try{
expect(f.read()?.pendingId).toBe(f.pendingId);
const guard=createPlanCountPermissionGuard();
expect(guard(captured.screen,'',f.read())).toBe('grant');
expect(guard(captured.screen,'',f.read())).toBe('handled');
}finally{f.close();}
});
test('a trimmed empty unchanged row is still compared with its source line',()=>{
const f=fixture();try{
expect(f.read(captured.screen.replace(/^ 212 +$/m,' 212'))?.pendingId).toBe(f.pendingId);
expect(f.read(captured.screen.replace(/^ 212 +$/m,' 212 stray'))).toBeNull();
}finally{f.close();}
});
test.each([
['changed wrapped unchanged row',(s:string)=>s.replace('man: ~4-6 weeks','man: ~9 weeks')],
['extra wrapped unchanged row',(s:string)=>s.replace(/^( {6}man: [^\n]*\n)/m,'$1 injected text\n')],
['wrapped unchanged row after an added row',(s:string)=>s.replace(/\n( 210 )/,'\n stray\n$1')],
['four-column unchanged wrap',(s:string)=>s.replace(/^ {6}man:/m,' man:')],
['quoted unchanged wrap',(s:string)=>s.replace(/^ {6}man:/m,' > man:')],
['changed numbered unchanged row',(s:string)=>s.replace(' 210 C) Smoke-test intent',' 210 C) Smoke-test demand')],
['changed added row',(s:string)=>s.replace('+rwise redirect','+rwise ignore')],
])('%s is refused',(_,change)=>{
const f=fixture();try{
const screen=change(captured.screen);expect(screen).not.toBe(captured.screen);
expect(f.read(screen)).toBeNull();
expect(createPlanCountPermissionGuard()(screen,'',f.read(screen))).not.toBe('grant');
}finally{f.close();}
});
test/plan-format-approach-prompt.test.ts
+21
View File
@@ -0,0 +1,21 @@
import { expect, test } from 'bun:test';
import * as fs from 'fs';
import * as path from 'path';
// D2: census 37195203538 refused plan-ceo-review-format-approach twice
// (reasoning_extraction, "duplicating model outputs") on its session prompt.
const source = fs.readFileSync(path.join(import.meta.dir, 'skill-e2e-plan-format.test.ts'), 'utf8');
const start = source.indexOf("testConcurrentIfSelected('plan-ceo-review-format-approach'");
const block = source.slice(start, source.indexOf('validate:', start));
const instruction = /function approachQuestionInstruction\(outFile: string\): string \{\n\s*return `([^`]+)`;/.exec(source)?.[1] ?? '';
test('the approach-menu prompt delivers its question without output-harvesting wording', () => {
expect(start).toBeGreaterThan(-1);
expect(block).toContain('${approachQuestionInstruction(outFile)}');
expect(block).not.toContain('captureInstruction(');
const prompt = block + instruction;
for (const phrase of [/verbatim/i, /paraphrase/i, /exact prose/i, /format-capture/i]) expect(prompt).not.toMatch(phrase);
expect(instruction).toContain('${outFile}');
expect(instruction).toContain('do not call any tool to ask the user');
expect(instruction).toContain('every option, and the Recommendation line');
});
test/skill-e2e-auq-matrix.test.ts
+13 -8
View File
@@ -6,7 +6,8 @@
* listed skill SHIPS the format spec in its always-loaded skeleton. This test
* proves each skill's model OBEYS it: that the first real AUQ it fires carries
* the fields software reads (a Recommendation: line and exactly one
* (recommended) option) with a substantive recommendation (>= 4). The other
* (recommended) option) with a substantive recommendation (3-sample judge panel
* mean >= 4 on the one capture; C4). The other
* format elements are logged. One parametrized case per skill so a single weak skill
* is an isolated failure, not a blocker for the rest.
*
@@ -38,8 +39,8 @@ import {
scoreAuqFormat,
auqMachineFormatProblems,
skillFromWorktree,
gradeAuqRecommendation,
} from './helpers/auq-sdk-capture';
import { auqSubstancePanel, AUQ_SUBSTANCE_MIN } from './helpers/auq-substance-panel';
const describeE2E = describeE2ETier('periodic');
const runId = `auq-matrix-${process.env.EVALS_RUN_ID ?? 'local'}`;
@@ -140,18 +141,20 @@ describeE2E('AUQ behavioral matrix (periodic)', () => {
const text = capture.text;
const fmt = scoreAuqFormat(text);
let substance = 0;
let samples: number[] = [];
let recPresent = false;
let hadBecause = false;
if (text.trim()) {
const g = await gradeAuqRecommendation(text);
substance = g.substance;
recPresent = g.present;
hadBecause = g.hadLiteralBecause;
const panel = await auqSubstancePanel(text);
substance = panel.mean;
samples = panel.samples.map(g => g.substance);
recPresent = panel.samples[0].present;
hadBecause = panel.samples[0].hadLiteralBecause;
}
// eslint-disable-next-line no-console
console.log(
`[AUQ-matrix ${m.skill}] captured=${text.length}B format=${fmt.present}/${fmt.total} ` +
`missing=[${fmt.missing.join(',')}] recPresent=${recPresent} substance=${substance} ` +
`missing=[${fmt.missing.join(',')}] recPresent=${recPresent} substance=${substance.toFixed(2)} [${samples.join(',')}] ` +
`literalBecause=${hadBecause}`,
);
@@ -161,7 +164,9 @@ describeE2E('AUQ behavioral matrix (periodic)', () => {
const problems: string[] = [];
// Presentation elements (ELI10, Pros / cons, ✅/❌, Net:) are logged above, not failed.
problems.push(...auqMachineFormatProblems(capture.question));
if (substance < 4) problems.push(`recommendation substance ${substance} < 4 (boilerplate/weak)`);
if (substance < AUQ_SUBSTANCE_MIN) {
problems.push(`recommendation substance mean ${substance.toFixed(2)} < ${AUQ_SUBSTANCE_MIN} over samples [${samples.join(',')}] (boilerplate/weak)`);
}
if (problems.length > 0) {
throw new Error(
`${m.skill} AUQ not at plan-ceo bar:\n - ${problems.join('\n - ')}\n--- captured AUQ ---\n${text}`,
test/skill-e2e-plan-format.test.ts
+14 -3
View File
@@ -104,12 +104,23 @@ function setupPlanDir(tmpPrefix: string, skillName: 'plan-ceo-review' | 'plan-en
return planDir;
}
// The capture instruction passed to every case. Tells the agent to dump
// AskUserQuestion content to a file instead of calling a tool.
// The capture instruction passed to every case except the approach menu (D2).
// Tells the agent to dump AskUserQuestion content to a file instead of calling a tool.
function captureInstruction(outFile: string): string {
return `Write the verbatim text of every AskUserQuestion you would have made to ${outFile} (one question per session, full text including options and recommendation line). Do NOT call any tool to ask the user. Do NOT paraphrase — include the exact prose you would have shown. This is a format-capture test, not an interactive session.`;
}
/**
* D2: both approach-menu refusals in census 37195203538 (Fable 5.1,
* `reasoning_extraction`, "duplicating model outputs") were anchored on the
* session prompt. This wording delivers the same question to the file as the
* step's output, without the "verbatim ... exact prose ... format-capture"
* phrasing.
*/
function approachQuestionInstruction(outFile: string): string {
return `This session has no interactive question tool, so do not call any tool to ask the user. Write the one AskUserQuestion this step asks to ${outFile} instead: the question text, every option, and the Recommendation line, formatted as the skill specifies.`;
}
// --- Case 1: plan-ceo-review mode selection (kind-differentiated) ---
describeIfSelected('Plan Format — CEO Mode Selection', ['plan-ceo-review-format-mode'], () => {
@@ -207,7 +218,7 @@ Read plan.md — that's the plan to review. This is a standalone plan document,
Proceed to Alternatives (the implementation approach menu). This is where the skill generates 2-3 approaches (minimal viable vs ideal architecture) and presents them via AskUserQuestion. These options differ in coverage (complete vs shortcut), so Completeness: N/10 applies.
${captureInstruction(outFile)}
${approachQuestionInstruction(outFile)}
After writing the file, stop. Do not continue the review.`,
workingDirectory: planDir,