mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-02 17:40:02 +02:00
fix(evals): repair proof-run reds in design-consultation, document-release, design and QA fixtures
- design-consultation Phase 1 asks one brief that confirms context and decides
research; the confirm-only first question scored substance 2.
- document-release defines ship-owned inputs, exact steps and the JSON result,
and drops stale spawned-from-/ship text (judge actionability 3.67 -> 4/4/4).
- plan-design-with-ui accepts the Step 0D focus menu the same way the shared
picker does ("focus on specific ones?").
- plan-design-review plan-mode saves in three Edits instead of one final Write.
- QA functional annotations ask for the full 40-character revision.
- Outside-disabled attribution judges quoted prior-record data by its exact
timestamp or a dated, pre-existing-record sentence; four captured phrasings
replay clean and current claims still fail.
- --case can select autoplan-dual-voice by its literal test name.
This commit is contained in:
garrytan committed
1 parent
aba80c8fb6
commit
a111225e78
17 files changed
+180
-92
No files matched your search
+10
-10
@@ -425,7 +425,7 @@ Make factual updates directly; ask about risky or subjective decisions in standa
|
||||
|
||||
## Ship-owned documentation mode
|
||||
|
||||
With a ship candidate, require the actual spawned marker and audit-scope rules below.
|
||||
With a ship candidate, follow audit-scope's inputs, steps and JSON result below.
|
||||
Missing marking/inputs/assets returns `blocked`, never standalone execution. Ship
|
||||
authority overrides generic spawned recommendations and standalone steps.
|
||||
|
||||
@@ -438,8 +438,8 @@ authority overrides generic spawned recommendations and standalone steps.
|
||||
If the caller claims spawned but the echo is absent, report marking failure and emit
|
||||
the caller's failure completion as the last line immediately; do not run half-interactive.
|
||||
Otherwise stay interactive without the marker. Outside ship-owned mode, spawned gates
|
||||
auto-choose the RECOMMENDED option, record it in the completion report, and continue:
|
||||
never call AskUserQuestion or stop for a prose answer. The NEVER-do invariants below do
|
||||
auto-choose the RECOMMENDED option, record it in the completion report, and continue
|
||||
through Step 9: never call AskUserQuestion or stop for a prose answer. The NEVER-do invariants below do
|
||||
not relax: skip any recommendation that rewrites CHANGELOG or changes VERSION and
|
||||
record why. Step 8 and cross-model review refer to this rule; narrower caller scope wins.
|
||||
|
||||
@@ -488,10 +488,10 @@ DOC_DIFF_BASE=$(git merge-base origin/<base> HEAD 2>/dev/null || git merge-base
|
||||
echo "DOC_DIFF_BASE: $DOC_DIFF_BASE"
|
||||
```
|
||||
|
||||
1. Check the current branch. In standalone mode, if on the base branch, **abort**: "You're on the base branch. Run from a feature branch." A ship-owned read-only store audit uses its supplied source scope instead.
|
||||
1. Check the current branch. In standalone mode, if on the base branch, **abort**: "You're on the base branch. Run from a feature branch." Ship-owned mode skips this gate.
|
||||
|
||||
2. Gather the diff. In ship-owned mode, also read `git diff --cached`, `git diff`,
|
||||
and selected new-file content against the supplied base, not HEAD alone.
|
||||
2. Gather the diff. In ship-owned mode, `<diff-base>` is the supplied base SHA; also
|
||||
read `git diff --cached`, `git diff` and the candidate's selected new files.
|
||||
|
||||
```bash
|
||||
git diff <diff-base> HEAD --stat
|
||||
@@ -546,16 +546,16 @@ Use these definitions:
|
||||
- **Tutorial** — learning-oriented: step-by-step walkthrough for newcomers (getting started guides)
|
||||
- **Explanation** — understanding-oriented: "why this works this way" (ARCHITECTURE decisions, design rationale)
|
||||
|
||||
3. **Output the coverage map.** Items with zero coverage are **critical gaps** — flag them for
|
||||
Step 3. Items with reference-only coverage are **common gaps** — note them for the PR body.
|
||||
3. **Output the coverage map.** Items with zero coverage are **critical gaps**; items with
|
||||
reference-only coverage are **common gaps**. Report both as documentation debt.
|
||||
|
||||
4. **Architecture diagram drift detection.** If ARCHITECTURE.md (or any doc) contains ASCII
|
||||
diagrams or Mermaid blocks, extract entity names (modules, services, data flows) from the
|
||||
diagrams. Cross-reference against the diff. Flag any diagram entities that were renamed,
|
||||
split, removed, or moved in the code.
|
||||
|
||||
The coverage map feeds into Steps 2-3 (what to audit and fix) and Step 9 (documentation debt
|
||||
summary in the PR body). Do NOT auto-generate missing documentation pages — flag gaps only.
|
||||
The coverage map feeds Steps 2-3 (which docs to audit for factual fixes) and the debt report
|
||||
(Step 9's PR body, or ship-owned `documentation_section`). Do NOT auto-generate missing documentation pages — flag gaps only.
|
||||
When significant gaps are found, suggest running `/document-generate` to fill them.
|
||||
|
||||
---
|
||||
|
||||
@@ -38,7 +38,7 @@ Make factual updates directly; ask about risky or subjective decisions in standa
|
||||
|
||||
## Ship-owned documentation mode
|
||||
|
||||
With a ship candidate, require the actual spawned marker and audit-scope rules below.
|
||||
With a ship candidate, follow audit-scope's inputs, steps and JSON result below.
|
||||
Missing marking/inputs/assets returns `blocked`, never standalone execution. Ship
|
||||
authority overrides generic spawned recommendations and standalone steps.
|
||||
|
||||
@@ -50,8 +50,8 @@ authority overrides generic spawned recommendations and standalone steps.
|
||||
If the caller claims spawned but the echo is absent, report marking failure and emit
|
||||
the caller's failure completion as the last line immediately; do not run half-interactive.
|
||||
Otherwise stay interactive without the marker. Outside ship-owned mode, spawned gates
|
||||
auto-choose the RECOMMENDED option, record it in the completion report, and continue:
|
||||
never call AskUserQuestion or stop for a prose answer. The NEVER-do invariants below do
|
||||
auto-choose the RECOMMENDED option, record it in the completion report, and continue
|
||||
through Step 9: never call AskUserQuestion or stop for a prose answer. The NEVER-do invariants below do
|
||||
not relax: skip any recommendation that rewrites CHANGELOG or changes VERSION and
|
||||
record why. Step 8 and cross-model review refer to this rule; narrower caller scope wins.
|
||||
|
||||
@@ -92,10 +92,10 @@ DOC_DIFF_BASE=$(git merge-base origin/<base> HEAD 2>/dev/null || git merge-base
|
||||
echo "DOC_DIFF_BASE: $DOC_DIFF_BASE"
|
||||
```
|
||||
|
||||
1. Check the current branch. In standalone mode, if on the base branch, **abort**: "You're on the base branch. Run from a feature branch." A ship-owned read-only store audit uses its supplied source scope instead.
|
||||
1. Check the current branch. In standalone mode, if on the base branch, **abort**: "You're on the base branch. Run from a feature branch." Ship-owned mode skips this gate.
|
||||
|
||||
2. Gather the diff. In ship-owned mode, also read `git diff --cached`, `git diff`,
|
||||
and selected new-file content against the supplied base, not HEAD alone.
|
||||
2. Gather the diff. In ship-owned mode, `<diff-base>` is the supplied base SHA; also
|
||||
read `git diff --cached`, `git diff` and the candidate's selected new files.
|
||||
|
||||
```bash
|
||||
git diff <diff-base> HEAD --stat
|
||||
@@ -150,16 +150,16 @@ Use these definitions:
|
||||
- **Tutorial** — learning-oriented: step-by-step walkthrough for newcomers (getting started guides)
|
||||
- **Explanation** — understanding-oriented: "why this works this way" (ARCHITECTURE decisions, design rationale)
|
||||
|
||||
3. **Output the coverage map.** Items with zero coverage are **critical gaps** — flag them for
|
||||
Step 3. Items with reference-only coverage are **common gaps** — note them for the PR body.
|
||||
3. **Output the coverage map.** Items with zero coverage are **critical gaps**; items with
|
||||
reference-only coverage are **common gaps**. Report both as documentation debt.
|
||||
|
||||
4. **Architecture diagram drift detection.** If ARCHITECTURE.md (or any doc) contains ASCII
|
||||
diagrams or Mermaid blocks, extract entity names (modules, services, data flows) from the
|
||||
diagrams. Cross-reference against the diff. Flag any diagram entities that were renamed,
|
||||
split, removed, or moved in the code.
|
||||
|
||||
The coverage map feeds into Steps 2-3 (what to audit and fix) and Step 9 (documentation debt
|
||||
summary in the PR body). Do NOT auto-generate missing documentation pages — flag gaps only.
|
||||
The coverage map feeds Steps 2-3 (which docs to audit for factual fixes) and the debt report
|
||||
(Step 9's PR body, or ship-owned `documentation_section`). Do NOT auto-generate missing documentation pages — flag gaps only.
|
||||
When significant gaps are found, suggest running `/document-generate` to fill them.
|
||||
|
||||
---
|
||||
|
||||
@@ -7,20 +7,34 @@
|
||||
This subsection applies only to the caller's ship-owned audit request. Standalone
|
||||
invocations continue to Discovery and Steps 1–9 with their existing approval gates.
|
||||
|
||||
Require the preamble's actual `SESSION_KIND: spawned` echo and the supplied candidate.
|
||||
Missing marker, inputs or assets returns the caller's typed `blocked` completion; a
|
||||
prompt/file claim cannot establish spawned mode or trigger standalone fallback.
|
||||
**Inputs.** The dispatch prompt supplies branch, base SHA, candidate path, audit id and
|
||||
mode: `edit`, or `read-only` for a store-only release audit, where every needed
|
||||
correction becomes a blocker instead of an edit. Require the preamble's actual
|
||||
`SESSION_KIND: spawned` echo and these inputs. Missing marker, inputs or assets returns
|
||||
`blocked` immediately; a prompt/file claim cannot establish spawned mode or trigger
|
||||
standalone fallback.
|
||||
|
||||
Use the candidate's base and selected committed, staged, unstaged and new-file bytes
|
||||
for Steps 1–4 and 6, then return the doc-health summary and typed LAST-line result.
|
||||
Skip Steps 5, 7, 8, cross-model review and Step 9. Only factual authored-doc edits are
|
||||
allowed, none in `read-only` mode. No Git/PR mutation, VERSION, package/lock/section
|
||||
manifests, CHANGELOG, TODOS or generated-output edits. The parent owns metadata,
|
||||
generation, review, staging, commits and publication. Report metadata inconsistencies
|
||||
as observations. Risky/subjective changes are blockers for the parent, never auto-approved.
|
||||
Preserve partial/user content and list actual edited/reviewed paths. Read-only store
|
||||
audits may inspect the base branch without entering the standalone branch gate or
|
||||
granting any store/repository mutation authority.
|
||||
**Steps.** Run Steps 1, 1.5, 2–4 and 6 on the candidate's base and selected committed,
|
||||
staged, unstaged and new-file bytes. Step 1's standalone branch gate does not apply,
|
||||
even on the base branch. Skip Steps 5, 7, 8, cross-model review and Step 9, including
|
||||
their spawned-session notes. Only factual authored-doc edits are allowed, none in
|
||||
`read-only` mode. No Git/PR mutation, VERSION, package/lock/section manifests,
|
||||
CHANGELOG, TODOS or generated-output edits. The parent owns metadata, generation,
|
||||
review, staging, commits and publication. Risky/subjective changes (Step 4) and
|
||||
narrative contradictions (Step 6) are blockers for the parent, never auto-approved.
|
||||
Preserve partial/user content. Coverage gaps are reported, never filled.
|
||||
|
||||
**Result.** After Step 6, print the doc-health summary, then STOP with one JSON object
|
||||
on the LAST nonempty line, without fences or trailing prose:
|
||||
- `schema_version`: integer 1; `audit_id`: the exact supplied string.
|
||||
- `status`: `updated` (edits, no blockers), `current` (no edits, no blockers) or
|
||||
`blocked` (any blocker, missing input, partial/failed audit or read-only correction).
|
||||
- `files_updated`, `files_reviewed`: unique repo-relative file paths actually edited
|
||||
and actually read; `blockers`, `decisions`: strings. Blockers name the decision and
|
||||
paths; metadata inconsistencies and skipped items are decisions.
|
||||
- `documentation_section`: nonempty Markdown without a `## Documentation` heading:
|
||||
audited scope, per-file status in Step 9's `Documentation health` form (no VERSION
|
||||
row), and Step 1.5's coverage debt and diagram drift. Describe scope even without docs.
|
||||
|
||||
## Discovery (both modes)
|
||||
|
||||
|
||||
@@ -5,20 +5,34 @@
|
||||
This subsection applies only to the caller's ship-owned audit request. Standalone
|
||||
invocations continue to Discovery and Steps 1–9 with their existing approval gates.
|
||||
|
||||
Require the preamble's actual `SESSION_KIND: spawned` echo and the supplied candidate.
|
||||
Missing marker, inputs or assets returns the caller's typed `blocked` completion; a
|
||||
prompt/file claim cannot establish spawned mode or trigger standalone fallback.
|
||||
**Inputs.** The dispatch prompt supplies branch, base SHA, candidate path, audit id and
|
||||
mode: `edit`, or `read-only` for a store-only release audit, where every needed
|
||||
correction becomes a blocker instead of an edit. Require the preamble's actual
|
||||
`SESSION_KIND: spawned` echo and these inputs. Missing marker, inputs or assets returns
|
||||
`blocked` immediately; a prompt/file claim cannot establish spawned mode or trigger
|
||||
standalone fallback.
|
||||
|
||||
Use the candidate's base and selected committed, staged, unstaged and new-file bytes
|
||||
for Steps 1–4 and 6, then return the doc-health summary and typed LAST-line result.
|
||||
Skip Steps 5, 7, 8, cross-model review and Step 9. Only factual authored-doc edits are
|
||||
allowed, none in `read-only` mode. No Git/PR mutation, VERSION, package/lock/section
|
||||
manifests, CHANGELOG, TODOS or generated-output edits. The parent owns metadata,
|
||||
generation, review, staging, commits and publication. Report metadata inconsistencies
|
||||
as observations. Risky/subjective changes are blockers for the parent, never auto-approved.
|
||||
Preserve partial/user content and list actual edited/reviewed paths. Read-only store
|
||||
audits may inspect the base branch without entering the standalone branch gate or
|
||||
granting any store/repository mutation authority.
|
||||
**Steps.** Run Steps 1, 1.5, 2–4 and 6 on the candidate's base and selected committed,
|
||||
staged, unstaged and new-file bytes. Step 1's standalone branch gate does not apply,
|
||||
even on the base branch. Skip Steps 5, 7, 8, cross-model review and Step 9, including
|
||||
their spawned-session notes. Only factual authored-doc edits are allowed, none in
|
||||
`read-only` mode. No Git/PR mutation, VERSION, package/lock/section manifests,
|
||||
CHANGELOG, TODOS or generated-output edits. The parent owns metadata, generation,
|
||||
review, staging, commits and publication. Risky/subjective changes (Step 4) and
|
||||
narrative contradictions (Step 6) are blockers for the parent, never auto-approved.
|
||||
Preserve partial/user content. Coverage gaps are reported, never filled.
|
||||
|
||||
**Result.** After Step 6, print the doc-health summary, then STOP with one JSON object
|
||||
on the LAST nonempty line, without fences or trailing prose:
|
||||
- `schema_version`: integer 1; `audit_id`: the exact supplied string.
|
||||
- `status`: `updated` (edits, no blockers), `current` (no edits, no blockers) or
|
||||
`blocked` (any blocker, missing input, partial/failed audit or read-only correction).
|
||||
- `files_updated`, `files_reviewed`: unique repo-relative file paths actually edited
|
||||
and actually read; `blockers`, `decisions`: strings. Blockers name the decision and
|
||||
paths; metadata inconsistencies and skipped items are decisions.
|
||||
- `documentation_section`: nonempty Markdown without a `## Documentation` heading:
|
||||
audited scope, per-file status in Step 9's `Documentation health` form (no VERSION
|
||||
row), and Step 1.5's coverage debt and diagram drift. Describe scope even without docs.
|
||||
|
||||
## Discovery (both modes)
|
||||
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
<!-- Regenerate: bun run gen:skill-docs -->
|
||||
## Step 2: Per-File Documentation Audit
|
||||
|
||||
**Ship-owned documentation mode:** execute Steps 2–4 and 6 only, under the skeleton's
|
||||
audit/edit/result boundary. Then return the caller's typed completion; all standalone
|
||||
**Ship-owned documentation mode:** after Steps 1 and 1.5, execute Steps 2–4 and 6 only,
|
||||
under audit-scope's edit boundary, then return its JSON result; all standalone
|
||||
metadata, review, commit and PR steps below remain unavailable to this child.
|
||||
|
||||
Read each documentation file and cross-reference it against the diff. Use these generic heuristics
|
||||
@@ -131,8 +131,8 @@ After auditing each file individually, do a cross-doc consistency pass:
|
||||
|
||||
In ship-owned mode, protected metadata/manifests stay untouched even for factual
|
||||
inconsistencies, and narrative contradictions return as blockers. This is the last
|
||||
ship-child step: output the doc-health summary and typed completion, then STOP. A
|
||||
partial audit or unresolved required correction is `blocked`, never `current`.
|
||||
ship-child step: output the doc-health summary and audit-scope's JSON result, then
|
||||
STOP. A partial audit or unresolved required correction is `blocked`, never `current`.
|
||||
|
||||
---
|
||||
|
||||
@@ -193,7 +193,7 @@ git diff <diff-base> HEAD -- VERSION
|
||||
|
||||
**Spawned sessions** (per the spawned-dispatch contract at the top of this skill): the
|
||||
recommendation flips — choose C (leave version as-is) and record the uncovered scope in
|
||||
your completion report (the `decisions` array when dispatched from /ship).
|
||||
your completion report. Ship-owned children stopped at Step 6 and never reach this step.
|
||||
A spawned run must never change VERSION: the dispatching workflow owns version numbering.
|
||||
|
||||
The key insight: a VERSION bump set for "feature A" should not silently absorb "feature B"
|
||||
@@ -211,7 +211,7 @@ not an opt-in. The user turns it off only by asking explicitly
|
||||
**Spawned-session skip** (per the spawned-dispatch contract at the top of this skill): in a
|
||||
spawned session, skip this entire section — the dispatching workflow owns its own review
|
||||
passes, and the apply gate below needs a human. Note the skip in the upcoming Step 9 doc
|
||||
health summary and continue to Step 9.
|
||||
health summary and continue to Step 9. Ship-owned children already stopped at Step 6.
|
||||
|
||||
**Preflight — decide whether and how the doc review runs:**
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
## Step 2: Per-File Documentation Audit
|
||||
|
||||
**Ship-owned documentation mode:** execute Steps 2–4 and 6 only, under the skeleton's
|
||||
audit/edit/result boundary. Then return the caller's typed completion; all standalone
|
||||
**Ship-owned documentation mode:** after Steps 1 and 1.5, execute Steps 2–4 and 6 only,
|
||||
under audit-scope's edit boundary, then return its JSON result; all standalone
|
||||
metadata, review, commit and PR steps below remain unavailable to this child.
|
||||
|
||||
Read each documentation file and cross-reference it against the diff. Use these generic heuristics
|
||||
@@ -129,8 +129,8 @@ After auditing each file individually, do a cross-doc consistency pass:
|
||||
|
||||
In ship-owned mode, protected metadata/manifests stay untouched even for factual
|
||||
inconsistencies, and narrative contradictions return as blockers. This is the last
|
||||
ship-child step: output the doc-health summary and typed completion, then STOP. A
|
||||
partial audit or unresolved required correction is `blocked`, never `current`.
|
||||
ship-child step: output the doc-health summary and audit-scope's JSON result, then
|
||||
STOP. A partial audit or unresolved required correction is `blocked`, never `current`.
|
||||
|
||||
---
|
||||
|
||||
@@ -191,7 +191,7 @@ git diff <diff-base> HEAD -- VERSION
|
||||
|
||||
**Spawned sessions** (per the spawned-dispatch contract at the top of this skill): the
|
||||
recommendation flips — choose C (leave version as-is) and record the uncovered scope in
|
||||
your completion report (the `decisions` array when dispatched from /ship).
|
||||
your completion report. Ship-owned children stopped at Step 6 and never reach this step.
|
||||
A spawned run must never change VERSION: the dispatching workflow owns version numbering.
|
||||
|
||||
The key insight: a VERSION bump set for "feature A" should not silently absorb "feature B"
|
||||
|
||||
Reference in new issue
Block a user