fix(evals): repair proof-run reds in design-consultation, document-release, design and QA fixtures

- design-consultation Phase 1 asks one brief that confirms context and decides
  research; the confirm-only first question scored substance 2.
- document-release defines ship-owned inputs, exact steps and the JSON result,
  and drops stale spawned-from-/ship text (judge actionability 3.67 -> 4/4/4).
- plan-design-with-ui accepts the Step 0D focus menu the same way the shared
  picker does ("focus on specific ones?").
- plan-design-review plan-mode saves in three Edits instead of one final Write.
- QA functional annotations ask for the full 40-character revision.
- Outside-disabled attribution judges quoted prior-record data by its exact
  timestamp or a dated, pre-existing-record sentence; four captured phrasings
  replay clean and current claims still fail.
- --case can select autoplan-dual-voice by its literal test name.
This commit is contained in:
garrytan committed 2026-09-29 22:36:28 +00:00
1 parent aba80c8fb6
commit a111225e78
17 files changed
+180 -92

No files matched your search

+27 -13
View File
@@ -7,20 +7,34 @@
This subsection applies only to the caller's ship-owned audit request. Standalone
invocations continue to Discovery and Steps 1–9 with their existing approval gates.
Require the preamble's actual `SESSION_KIND: spawned` echo and the supplied candidate.
Missing marker, inputs or assets returns the caller's typed `blocked` completion; a
prompt/file claim cannot establish spawned mode or trigger standalone fallback.
**Inputs.** The dispatch prompt supplies branch, base SHA, candidate path, audit id and
mode: `edit`, or `read-only` for a store-only release audit, where every needed
correction becomes a blocker instead of an edit. Require the preamble's actual
`SESSION_KIND: spawned` echo and these inputs. Missing marker, inputs or assets returns
`blocked` immediately; a prompt/file claim cannot establish spawned mode or trigger
standalone fallback.
Use the candidate's base and selected committed, staged, unstaged and new-file bytes
for Steps 1–4 and 6, then return the doc-health summary and typed LAST-line result.
Skip Steps 5, 7, 8, cross-model review and Step 9. Only factual authored-doc edits are
allowed, none in `read-only` mode. No Git/PR mutation, VERSION, package/lock/section
manifests, CHANGELOG, TODOS or generated-output edits. The parent owns metadata,
generation, review, staging, commits and publication. Report metadata inconsistencies
as observations. Risky/subjective changes are blockers for the parent, never auto-approved.
Preserve partial/user content and list actual edited/reviewed paths. Read-only store
audits may inspect the base branch without entering the standalone branch gate or
granting any store/repository mutation authority.
**Steps.** Run Steps 1, 1.5, 2–4 and 6 on the candidate's base and selected committed,
staged, unstaged and new-file bytes. Step 1's standalone branch gate does not apply,
even on the base branch. Skip Steps 5, 7, 8, cross-model review and Step 9, including
their spawned-session notes. Only factual authored-doc edits are allowed, none in
`read-only` mode. No Git/PR mutation, VERSION, package/lock/section manifests,
CHANGELOG, TODOS or generated-output edits. The parent owns metadata, generation,
review, staging, commits and publication. Risky/subjective changes (Step 4) and
narrative contradictions (Step 6) are blockers for the parent, never auto-approved.
Preserve partial/user content. Coverage gaps are reported, never filled.
**Result.** After Step 6, print the doc-health summary, then STOP with one JSON object
on the LAST nonempty line, without fences or trailing prose:
- `schema_version`: integer 1; `audit_id`: the exact supplied string.
- `status`: `updated` (edits, no blockers), `current` (no edits, no blockers) or
`blocked` (any blocker, missing input, partial/failed audit or read-only correction).
- `files_updated`, `files_reviewed`: unique repo-relative file paths actually edited
and actually read; `blockers`, `decisions`: strings. Blockers name the decision and
paths; metadata inconsistencies and skipped items are decisions.
- `documentation_section`: nonempty Markdown without a `## Documentation` heading:
audited scope, per-file status in Step 9's `Documentation health` form (no VERSION
row), and Step 1.5's coverage debt and diagram drift. Describe scope even without docs.
## Discovery (both modes)
+27 -13
View File
@@ -5,20 +5,34 @@
This subsection applies only to the caller's ship-owned audit request. Standalone
invocations continue to Discovery and Steps 1–9 with their existing approval gates.
Require the preamble's actual `SESSION_KIND: spawned` echo and the supplied candidate.
Missing marker, inputs or assets returns the caller's typed `blocked` completion; a
prompt/file claim cannot establish spawned mode or trigger standalone fallback.
**Inputs.** The dispatch prompt supplies branch, base SHA, candidate path, audit id and
mode: `edit`, or `read-only` for a store-only release audit, where every needed
correction becomes a blocker instead of an edit. Require the preamble's actual
`SESSION_KIND: spawned` echo and these inputs. Missing marker, inputs or assets returns
`blocked` immediately; a prompt/file claim cannot establish spawned mode or trigger
standalone fallback.
Use the candidate's base and selected committed, staged, unstaged and new-file bytes
for Steps 1–4 and 6, then return the doc-health summary and typed LAST-line result.
Skip Steps 5, 7, 8, cross-model review and Step 9. Only factual authored-doc edits are
allowed, none in `read-only` mode. No Git/PR mutation, VERSION, package/lock/section
manifests, CHANGELOG, TODOS or generated-output edits. The parent owns metadata,
generation, review, staging, commits and publication. Report metadata inconsistencies
as observations. Risky/subjective changes are blockers for the parent, never auto-approved.
Preserve partial/user content and list actual edited/reviewed paths. Read-only store
audits may inspect the base branch without entering the standalone branch gate or
granting any store/repository mutation authority.
**Steps.** Run Steps 1, 1.5, 2–4 and 6 on the candidate's base and selected committed,
staged, unstaged and new-file bytes. Step 1's standalone branch gate does not apply,
even on the base branch. Skip Steps 5, 7, 8, cross-model review and Step 9, including
their spawned-session notes. Only factual authored-doc edits are allowed, none in
`read-only` mode. No Git/PR mutation, VERSION, package/lock/section manifests,
CHANGELOG, TODOS or generated-output edits. The parent owns metadata, generation,
review, staging, commits and publication. Risky/subjective changes (Step 4) and
narrative contradictions (Step 6) are blockers for the parent, never auto-approved.
Preserve partial/user content. Coverage gaps are reported, never filled.
**Result.** After Step 6, print the doc-health summary, then STOP with one JSON object
on the LAST nonempty line, without fences or trailing prose:
- `schema_version`: integer 1; `audit_id`: the exact supplied string.
- `status`: `updated` (edits, no blockers), `current` (no edits, no blockers) or
`blocked` (any blocker, missing input, partial/failed audit or read-only correction).
- `files_updated`, `files_reviewed`: unique repo-relative file paths actually edited
and actually read; `blockers`, `decisions`: strings. Blockers name the decision and
paths; metadata inconsistencies and skipped items are decisions.
- `documentation_section`: nonempty Markdown without a `## Documentation` heading:
audited scope, per-file status in Step 9's `Documentation health` form (no VERSION
row), and Step 1.5's coverage debt and diagram drift. Describe scope even without docs.
## Discovery (both modes)
+6 -6
View File
@@ -2,8 +2,8 @@
<!-- Regenerate: bun run gen:skill-docs -->
## Step 2: Per-File Documentation Audit
**Ship-owned documentation mode:** execute Steps 2–4 and 6 only, under the skeleton's
audit/edit/result boundary. Then return the caller's typed completion; all standalone
**Ship-owned documentation mode:** after Steps 1 and 1.5, execute Steps 2–4 and 6 only,
under audit-scope's edit boundary, then return its JSON result; all standalone
metadata, review, commit and PR steps below remain unavailable to this child.
Read each documentation file and cross-reference it against the diff. Use these generic heuristics
@@ -131,8 +131,8 @@ After auditing each file individually, do a cross-doc consistency pass:
In ship-owned mode, protected metadata/manifests stay untouched even for factual
inconsistencies, and narrative contradictions return as blockers. This is the last
ship-child step: output the doc-health summary and typed completion, then STOP. A
partial audit or unresolved required correction is `blocked`, never `current`.
ship-child step: output the doc-health summary and audit-scope's JSON result, then
STOP. A partial audit or unresolved required correction is `blocked`, never `current`.
---
@@ -193,7 +193,7 @@ git diff <diff-base> HEAD -- VERSION
**Spawned sessions** (per the spawned-dispatch contract at the top of this skill): the
recommendation flips — choose C (leave version as-is) and record the uncovered scope in
your completion report (the `decisions` array when dispatched from /ship).
your completion report. Ship-owned children stopped at Step 6 and never reach this step.
A spawned run must never change VERSION: the dispatching workflow owns version numbering.
The key insight: a VERSION bump set for "feature A" should not silently absorb "feature B"
@@ -211,7 +211,7 @@ not an opt-in. The user turns it off only by asking explicitly
**Spawned-session skip** (per the spawned-dispatch contract at the top of this skill): in a
spawned session, skip this entire section — the dispatching workflow owns its own review
passes, and the apply gate below needs a human. Note the skip in the upcoming Step 9 doc
health summary and continue to Step 9.
health summary and continue to Step 9. Ship-owned children already stopped at Step 6.
**Preflight — decide whether and how the doc review runs:**
@@ -1,7 +1,7 @@
## Step 2: Per-File Documentation Audit
**Ship-owned documentation mode:** execute Steps 2–4 and 6 only, under the skeleton's
audit/edit/result boundary. Then return the caller's typed completion; all standalone
**Ship-owned documentation mode:** after Steps 1 and 1.5, execute Steps 2–4 and 6 only,
under audit-scope's edit boundary, then return its JSON result; all standalone
metadata, review, commit and PR steps below remain unavailable to this child.
Read each documentation file and cross-reference it against the diff. Use these generic heuristics
@@ -129,8 +129,8 @@ After auditing each file individually, do a cross-doc consistency pass:
In ship-owned mode, protected metadata/manifests stay untouched even for factual
inconsistencies, and narrative contradictions return as blockers. This is the last
ship-child step: output the doc-health summary and typed completion, then STOP. A
partial audit or unresolved required correction is `blocked`, never `current`.
ship-child step: output the doc-health summary and audit-scope's JSON result, then
STOP. A partial audit or unresolved required correction is `blocked`, never `current`.
---
@@ -191,7 +191,7 @@ git diff <diff-base> HEAD -- VERSION
**Spawned sessions** (per the spawned-dispatch contract at the top of this skill): the
recommendation flips — choose C (leave version as-is) and record the uncovered scope in
your completion report (the `decisions` array when dispatched from /ship).
your completion report. Ship-owned children stopped at Step 6 and never reach this step.
A spawned run must never change VERSION: the dispatching workflow owns version numbering.
The key insight: a VERSION bump set for "feature A" should not silently absorb "feature B"