fix(flaky): give the judged review and Eng prose one rule each, and name what a cancelled judge received (GSTA-139 S3)

- review: Step 5.8 records an unresolved non-gate defect as open, and a repeat
  pass's reruns are revalidation that run after smoke expiry in the same
  PROBE_DIR with materialize once on the final pass (every sample of #3527's
  red panel 40617200857 and its rerun named one of these). Nearby prose trimmed
  so union, skeleton and host pins do not grow.
- plan-eng-review: drop the Eng-only em-dash Voice variant so the shared Voice
  hoists; the Scope gate's Format precedence names em dashes (5 of 6 samples of
  40634787316 named the near-copy Voice). Host size pins lowered.
- llm-judge: a cancelled judge request prints llm-judge-aborted with timing,
  429 waits and stream event counts (no content), so the next workflow-judge
  deadline (train 35 gate 40660626349) names its cause.
- benchmark: the Phase 6 example no longer lazy-loads the hero image.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
garrytanandcapy-ai[bot] committed 2026-10-28 13:42:21 +00:00
1 parent 4b8c03bea9
commit 1bdc4ce582
14 files changed
+181 -70

No files matched your search

+1 -1
View File
@@ -412,7 +412,7 @@ TOP 10 SLOWEST RESOURCES
RECOMMENDATIONS: RECOMMENDATIONS:
- vendor.chunk.js: code-split — 320KB on initial load - vendor.chunk.js: code-split — 320KB on initial load
- analytics.js: load async/defer — blocks rendering for 250ms - analytics.js: load async/defer — blocks rendering for 250ms
- hero-image.webp: add width/height (CLS), lazy-load - hero-image.webp: add width/height (CLS)
``` ```
Third-party = any host but the page's origin. Each recommendation names a resource here or in a Phase 5 regression, the metric it moves and one remedy (split, defer, compress, lazy-load, cache). Third-party = any host but the page's origin. Each recommendation names a resource here or in a Phase 5 regression, the metric it moves and one remedy (split, defer, compress, lazy-load, cache).
+1 -1
View File
@@ -201,7 +201,7 @@ TOP 10 SLOWEST RESOURCES
RECOMMENDATIONS: RECOMMENDATIONS:
- vendor.chunk.js: code-split — 320KB on initial load - vendor.chunk.js: code-split — 320KB on initial load
- analytics.js: load async/defer — blocks rendering for 250ms - analytics.js: load async/defer — blocks rendering for 250ms
- hero-image.webp: add width/height (CLS), lazy-load - hero-image.webp: add width/height (CLS)
``` ```
Third-party = any host but the page's origin. Each recommendation names a resource here or in a Phase 5 regression, the metric it moves and one remedy (split, defer, compress, lazy-load, cache). Third-party = any host but the page's origin. Each recommendation names a resource here or in a Phase 5 regression, the metric it moves and one remedy (split, defer, compress, lazy-load, cache).
+1 -23
View File
@@ -50,7 +50,7 @@ Recommendation: A when a branch diff exists, otherwise B. Reply with A, B, or C.
After target selection, use the preamble's full decision brief, transport and continuous D-numbering. Setup questions approve no engineering remedies. After target selection, use the preamble's full decision brief, transport and continuous D-numbering. Setup questions approve no engineering remedies.
**Format precedence:** Copy required command, output and question formats exactly. Apply Voice to newly composed prose. **Format precedence:** Copy required command, output and question formats exactly, em dashes included. Apply Voice to newly composed prose.
**Startup sequence** (after target selection): **Startup sequence** (after target selection):
1. Run the Preamble command and its startup instructions (Context Recovery and the preamble's own one-time setup prompts; the Prior Learnings cross-project question waits for Step 0). Defer Operational Self-Improvement, Telemetry and Plan Status Footer to finish; format/transport rules apply throughout. 1. Run the Preamble command and its startup instructions (Context Recovery and the preamble's own one-time setup prompts; the Prior Learnings cross-project question waits for Step 0). Defer Operational Self-Improvement, Telemetry and Plan Status Footer to finish; format/transport rules apply throughout.
@@ -187,28 +187,6 @@ Before emitting a tool or prose decision brief, verify:
- [ ] Follow Tool resolution: tool call unless Conductor or documented prose fallback; prose includes the mandatory triad + explicit reply selectors, then STOP. Spawned sessions follow their auto-choice rule. - [ ] Follow Tool resolution: tool call unless Conductor or documented prose fallback; prose includes the mandatory triad + explicit reply selectors, then STOP. Spawned sessions follow their auto-choice rule.
- [ ] Write non-ASCII directly, not \u-escaped. For 5+ options, split/batch into ≤4 without dropping; check dependencies and stop the chain immediately on Hold. - [ ] Write non-ASCII directly, not \u-escaped. For 5+ options, split/batch into ≤4 without dropping; check dependencies and stop the chain immediately on Hold.
## Voice
GStack voice: Garry-shaped product and engineering judgment.
- Lead with the point. Say what it does, why it matters, and what changes for the builder.
- Be concrete. Name files, functions, line numbers, commands, outputs, evals, and real numbers.
- Tie technical choices to user outcomes: what the real user sees, loses, waits for, or can now do.
- Be direct about quality. Bugs matter. Edge cases matter. Fix the whole thing, not the demo path.
- Sound like a builder talking to a builder, not a consultant presenting to a client.
- Never corporate, academic, PR, or hype. Avoid filler, throat-clearing, generic optimism, and founder cosplay.
- Do not add em dashes in prose you compose during the review. Existing templates, quoted text, command output, and required copied labels may contain them. No AI vocabulary: delve, crucial, robust, comprehensive, nuanced, multifaceted, furthermore, moreover, additionally, pivotal, landscape, tapestry, underscore, foster, showcase, intricate, vibrant, fundamental, significant, load-bearing.
- Reply in the language of the user's latest message unless asked otherwise. Code, commands, paths, identifiers, quoted output and question markers (`D<N>`, option letters, `(recommended)`) stay verbatim.
- The user has context you do not: domain knowledge, timing, relationships, taste. Cross-model agreement is a recommendation, not a decision. The user decides.
Good: "auth.ts:47 returns undefined when the session cookie expires. Users hit a white screen. Fix: add a null check and redirect to /login. Two lines."
Bad: "I've identified a potential issue in the authentication flow that may cause problems under certain conditions."
**Bounded closer.** After completing work, report in at most a few short lines: what changed, what was skipped, what to watch. No feature tours or unrequested design notes. Exempt: decision briefs, completion-status blocks, requested explanations, and a skill's mandated report (/qa-only, /plan-*-review, /retro, /document-generate). The rule limits prose around the deliverable, never the deliverable.
Good closer: "Renamed the flag in 3 files, regenerated docs, tests green. Skipped the CLI alias (unused since v1.2); watch the Windows job."
Bad closer: a tour of every edit, a restatement of the plan, and three paragraphs justifying choices nobody questioned.
## Question Tuning (skip entirely if `QUESTION_TUNING: false`) ## Question Tuning (skip entirely if `QUESTION_TUNING: false`)
Before each decision brief (AskUserQuestion or Conductor/fallback prose), choose `question_id` from `~/.claude/skills/gstack/scripts/question-registry.ts` or `{skill}-{slug}`, then run `~/.claude/skills/gstack/bin/gstack-question-preference --check "<id>"`; for an unregistered id, write the question summary to `.gstack/tmp/qt.txt` (file-write tool) and append `--summary-file .gstack/tmp/qt.txt` (one-way keyword check). `AUTO_DECIDE` means choose the recommended option and say "Auto-decided [summary] → [option] (your preference). Change with /plan-tune." `ASK_NORMALLY` means ask. Before each decision brief (AskUserQuestion or Conductor/fallback prose), choose `question_id` from `~/.claude/skills/gstack/scripts/question-registry.ts` or `{skill}-{slug}`, then run `~/.claude/skills/gstack/bin/gstack-question-preference --check "<id>"`; for an unregistered id, write the question summary to `.gstack/tmp/qt.txt` (file-write tool) and append `--summary-file .gstack/tmp/qt.txt` (one-way keyword check). `AUTO_DECIDE` means choose the recommended option and say "Auto-decided [summary] → [option] (your preference). Change with /plan-tune." `ASK_NORMALLY` means ask.
+1 -1
View File
@@ -60,7 +60,7 @@ Recommendation: A when a branch diff exists, otherwise B. Reply with A, B, or C.
After target selection, use the preamble's full decision brief, transport and continuous D-numbering. Setup questions approve no engineering remedies. After target selection, use the preamble's full decision brief, transport and continuous D-numbering. Setup questions approve no engineering remedies.
**Format precedence:** Copy required command, output and question formats exactly. Apply Voice to newly composed prose. **Format precedence:** Copy required command, output and question formats exactly, em dashes included. Apply Voice to newly composed prose.
**Startup sequence** (after target selection): **Startup sequence** (after target selection):
1. Run the Preamble command and its startup instructions (Context Recovery and the preamble's own one-time setup prompts; the Prior Learnings cross-project question waits for Step 0). Defer Operational Self-Improvement, Telemetry and Plan Status Footer to finish; format/transport rules apply throughout. 1. Run the Preamble command and its startup instructions (Context Recovery and the preamble's own one-time setup prompts; the Prior Learnings cross-project question waits for Step 0). Defer Operational Self-Improvement, Telemetry and Plan Status Footer to finish; format/transport rules apply throughout.
+9 -8
View File
@@ -806,7 +806,7 @@ After outputting your own findings, if Greptile comments were classified in Step
Before replying to any comment, run greptile-triage.md's **Escalation Detection** to choose Tier 1 (friendly) or Tier 2 (firm) reply templates. Before replying to any comment, run greptile-triage.md's **Escalation Detection** to choose Tier 1 (friendly) or Tier 2 (firm) reply templates.
1. **VALID & ACTIONABLE comments:** Use their Step 5a–5d disposition; do not ask a second fix question. Step 5c alone supplies A) Fix / B) Skip for ASK items. After a completed fix, use the **Fix reply template** with diff and explanation; cite the current diff if uncommitted, never invent a commit SHA. A Skip leaves the defect unresolved and grants no new fix permission. If evidence disproves the finding, reclassify it below. 1. **VALID & ACTIONABLE comments:** Use their Step 5a–5d disposition; do not ask a second fix question. After a completed fix, use the **Fix reply template** with diff and explanation; cite the current diff if uncommitted, never invent a commit SHA. A Skip leaves the defect unresolved and grants no new fix permission. If evidence disproves the finding, reclassify it below.
2. **FALSE POSITIVE comments:** These are reply decisions, not code approval. Show file:line (or [top-level]), summary, permalink and evidence, then ask: 2. **FALSE POSITIVE comments:** These are reply decisions, not code approval. Show file:line (or [top-level]), summary, permalink and evidence, then ask:
- A) Reply explaining why this is incorrect (recommended if clearly wrong) - A) Reply explaining why this is incorrect (recommended if clearly wrong)
@@ -835,6 +835,8 @@ Before replying to any comment, run greptile-triage.md's **Escalation Detection*
final: repeat Steps 3–5 with a full start. final: repeat Steps 3–5 with a full start.
2. On a repeat, execute Steps 3–5 in order. At Step 4.7, an edited pass supersedes 2. On a repeat, execute Steps 3–5 in order. At Step 4.7, an edited pass supersedes
every earlier QA capture (exploratory §4): rerun each probe the verdict relies on. every earlier QA capture (exploratory §4): rerun each probe the verdict relies on.
These reruns are revalidation, so they run even after smoke expiry; keep the
PROBE_DIR and run its once-only `materialize` on the final pass.
A zero-edit pass reuses only this invocation's unchanged-input QA evidence, A zero-edit pass reuses only this invocation's unchanged-input QA evidence,
rerunning any probe whose inputs changed or are uncertain. Reusing never skips a review step. rerunning any probe whose inputs changed or are uncertain. Reusing never skips a review step.
3. **Verify completed actions.** On the final zero-edit pass, reconcile this 3. **Verify completed actions.** On the final zero-edit pass, reconcile this
@@ -846,9 +848,8 @@ Before replying to any comment, run greptile-triage.md's **Escalation Detection*
defects and unresolved counts; earlier fixes do not suppress them. defects and unresolved counts; earlier fixes do not suppress them.
4. **Recheck skipped advice.** Re-read its final-snapshot supporting source and 4. **Recheck skipped advice.** Re-read its final-snapshot supporting source and
reconfirm the decision; otherwise report its history without a reusable skip. reconfirm the decision; otherwise report its history without a reusable skip.
The logger computes `snapshot_covered_paths` from eligible paths whose raw bytes The logger computes `snapshot_covered_paths` itself; never carry prior-cycle,
equal the bound snapshot blobs (`[]` if none). Never carry prior-cycle, supplied supplied or prior-record coverage forward or build this proof yourself. Fixed advice
or prior-record coverage forward or build this proof yourself. Fixed advice
needs no skip coverage. needs no skip coverage.
### 2. Fill the record ### 2. Fill the record
@@ -878,7 +879,8 @@ pass: report it with its reason.
`helper_target`. Recheck source after fixes. The logger uses `sharedLibsFingerprint`, `helper_target`. Recheck source after fixes. The logger uses `sharedLibsFingerprint`,
never supplied/model hashes. never supplied/model hashes.
Actions: `auto-fixed` (Step 5b), `fixed` (approved **and completed** in Step 5d), Actions: `auto-fixed` (Step 5b), `fixed` (approved **and completed** in Step 5d),
`skipped` (explicit Skip in Step 5c). Advice is never `auto-fixed`; pending `skipped` (explicit Skip in Step 5c),
`open` (any other defect, e.g. unanswered). Advice is never `auto-fixed`; pending
advice stays in the response, not the record. Exclude prior Step 5.0 advice stays in the response, not the record. Exclude prior Step 5.0
suppressions; include this invocation's revalidated decisions. suppressions; include this invocation's revalidated decisions.
- Gate findings additionally carry `gate` (the RH tag, e.g. `"RH-15"`), `gate_id` - Gate findings additionally carry `gate` (the RH tag, e.g. `"RH-15"`), `gate_id`
@@ -896,9 +898,8 @@ pass: report it with its reason.
`TIMESTAMP` is ISO 8601; `COMMIT` is `git rev-parse --short HEAD`. `TIMESTAMP` is ISO 8601; `COMMIT` is `git rev-parse --short HEAD`.
`quality_score` is Step 4.6's specialist score (`10.0` when small-diff specialists `quality_score` is Step 4.6's specialist score (`10.0` when small-diff specialists
were skipped or this host omits Review Army). This default is not completion evidence; were skipped or this host omits Review Army). This default is not completion evidence
unresolved non-advisory core defects still count in `issues_found`, and never lowers defect counts. The logger builds trusted `review_binding` from the
`critical`, `informational`. The logger builds trusted `review_binding` from the
validated captured branch digest, discarding caller bindings. Never invent a binding validated captured branch digest, discarding caller bindings. Never invent a binding
or replace REVIEW_START at log time; finish only this pass's own token. or replace REVIEW_START at log time; finish only this pass's own token.
+9 -8
View File
@@ -357,7 +357,7 @@ After outputting your own findings, if Greptile comments were classified in Step
Before replying to any comment, run greptile-triage.md's **Escalation Detection** to choose Tier 1 (friendly) or Tier 2 (firm) reply templates. Before replying to any comment, run greptile-triage.md's **Escalation Detection** to choose Tier 1 (friendly) or Tier 2 (firm) reply templates.
1. **VALID & ACTIONABLE comments:** Use their Step 5a–5d disposition; do not ask a second fix question. Step 5c alone supplies A) Fix / B) Skip for ASK items. After a completed fix, use the **Fix reply template** with diff and explanation; cite the current diff if uncommitted, never invent a commit SHA. A Skip leaves the defect unresolved and grants no new fix permission. If evidence disproves the finding, reclassify it below. 1. **VALID & ACTIONABLE comments:** Use their Step 5a–5d disposition; do not ask a second fix question. After a completed fix, use the **Fix reply template** with diff and explanation; cite the current diff if uncommitted, never invent a commit SHA. A Skip leaves the defect unresolved and grants no new fix permission. If evidence disproves the finding, reclassify it below.
2. **FALSE POSITIVE comments:** These are reply decisions, not code approval. Show file:line (or [top-level]), summary, permalink and evidence, then ask: 2. **FALSE POSITIVE comments:** These are reply decisions, not code approval. Show file:line (or [top-level]), summary, permalink and evidence, then ask:
- A) Reply explaining why this is incorrect (recommended if clearly wrong) - A) Reply explaining why this is incorrect (recommended if clearly wrong)
@@ -386,6 +386,8 @@ Before replying to any comment, run greptile-triage.md's **Escalation Detection*
final: repeat Steps 3–5 with a full start. final: repeat Steps 3–5 with a full start.
2. On a repeat, execute Steps 3–5 in order. At Step 4.7, an edited pass supersedes 2. On a repeat, execute Steps 3–5 in order. At Step 4.7, an edited pass supersedes
every earlier QA capture (exploratory §4): rerun each probe the verdict relies on. every earlier QA capture (exploratory §4): rerun each probe the verdict relies on.
These reruns are revalidation, so they run even after smoke expiry; keep the
PROBE_DIR and run its once-only `materialize` on the final pass.
A zero-edit pass reuses only this invocation's unchanged-input QA evidence, A zero-edit pass reuses only this invocation's unchanged-input QA evidence,
rerunning any probe whose inputs changed or are uncertain. Reusing never skips a review step. rerunning any probe whose inputs changed or are uncertain. Reusing never skips a review step.
3. **Verify completed actions.** On the final zero-edit pass, reconcile this 3. **Verify completed actions.** On the final zero-edit pass, reconcile this
@@ -397,9 +399,8 @@ Before replying to any comment, run greptile-triage.md's **Escalation Detection*
defects and unresolved counts; earlier fixes do not suppress them. defects and unresolved counts; earlier fixes do not suppress them.
4. **Recheck skipped advice.** Re-read its final-snapshot supporting source and 4. **Recheck skipped advice.** Re-read its final-snapshot supporting source and
reconfirm the decision; otherwise report its history without a reusable skip. reconfirm the decision; otherwise report its history without a reusable skip.
The logger computes `snapshot_covered_paths` from eligible paths whose raw bytes The logger computes `snapshot_covered_paths` itself; never carry prior-cycle,
equal the bound snapshot blobs (`[]` if none). Never carry prior-cycle, supplied supplied or prior-record coverage forward or build this proof yourself. Fixed advice
or prior-record coverage forward or build this proof yourself. Fixed advice
needs no skip coverage. needs no skip coverage.
### 2. Fill the record ### 2. Fill the record
@@ -429,7 +430,8 @@ pass: report it with its reason.
`helper_target`. Recheck source after fixes. The logger uses `sharedLibsFingerprint`, `helper_target`. Recheck source after fixes. The logger uses `sharedLibsFingerprint`,
never supplied/model hashes. never supplied/model hashes.
Actions: `auto-fixed` (Step 5b), `fixed` (approved **and completed** in Step 5d), Actions: `auto-fixed` (Step 5b), `fixed` (approved **and completed** in Step 5d),
`skipped` (explicit Skip in Step 5c). Advice is never `auto-fixed`; pending `skipped` (explicit Skip in Step 5c),
`open` (any other defect, e.g. unanswered). Advice is never `auto-fixed`; pending
advice stays in the response, not the record. Exclude prior Step 5.0 advice stays in the response, not the record. Exclude prior Step 5.0
suppressions; include this invocation's revalidated decisions. suppressions; include this invocation's revalidated decisions.
- Gate findings additionally carry `gate` (the RH tag, e.g. `"RH-15"`), `gate_id` - Gate findings additionally carry `gate` (the RH tag, e.g. `"RH-15"`), `gate_id`
@@ -447,9 +449,8 @@ pass: report it with its reason.
`TIMESTAMP` is ISO 8601; `COMMIT` is `git rev-parse --short HEAD`. `TIMESTAMP` is ISO 8601; `COMMIT` is `git rev-parse --short HEAD`.
`quality_score` is Step 4.6's specialist score (`10.0` when small-diff specialists `quality_score` is Step 4.6's specialist score (`10.0` when small-diff specialists
were skipped or this host omits Review Army). This default is not completion evidence; were skipped or this host omits Review Army). This default is not completion evidence
unresolved non-advisory core defects still count in `issues_found`, and never lowers defect counts. The logger builds trusted `review_binding` from the
`critical`, `informational`. The logger builds trusted `review_binding` from the
validated captured branch digest, discarding caller bindings. Never invent a binding validated captured branch digest, discarding caller bindings. Never invent a binding
or replace REVIEW_START at log time; finish only this pass's own token. or replace REVIEW_START at log time; finish only this pass's own token.
@@ -2,9 +2,7 @@
import type { TemplateContext } from '../types'; import type { TemplateContext } from '../types';
export function generateVoiceDirective(ctx: TemplateContext, tier: number): string { export function generateVoiceDirective(ctx: TemplateContext, tier: number): string {
const emDashRule = ctx.skillName === 'plan-eng-review' const emDashRule = 'No em dashes.';
? 'Do not add em dashes in prose you compose during the review. Existing templates, quoted text, command output, and required copied labels may contain them.'
: 'No em dashes.';
if (tier <= 1) { if (tier <= 1) {
return `## Voice return `## Voice
+13
View File
@@ -163,3 +163,16 @@ test('calibration keeps its future hook but no shipped source enables the absent
} }
expect(mentions).toEqual(['scripts/resolvers/gbrain.ts']); expect(mentions).toEqual(['scripts/resolvers/gbrain.ts']);
}); });
test('the judged Eng bundle carries the shared Voice once and states its em-dash precedence in the Scope gate (paid screen 40634787316)', () => {
// Five of six samples (red panel and rerun) named Eng's near-copy of the shared Voice, whose only difference was
// its em-dash clause, as a rule the agent must reconcile; the Scope gate's Format precedence already owned it.
// fails_when=a second Voice section returns to the judged bundle or the precedence stops naming em dashes.
const input = readWorkflowJudgeInput({root: path.join(import.meta.dir, '..'), skillPath: 'plan-eng-review/SKILL.md',
startMarker: '# Plan Review Mode', endMarker: null});
expect(input.text.match(/^## Voice$/gm)).toHaveLength(1);
expect(input.text).not.toContain('Do not add em dashes in prose you compose');
expect(scope).toContain('**Format precedence:** Copy required command, output and question formats exactly, em dashes included. Apply Voice to newly composed prose.');
const startup = input.files[0]!.content;
expect(startup.indexOf('**Format precedence:**')).toBeLessThan(startup.indexOf('## Shared preamble (load first)'));
});
+19 -19
View File
@@ -5,50 +5,50 @@
"agy/gstack-autoplan": 109810, "agy/gstack-autoplan": 109810,
"agy/gstack-plan-design-review": 110044, "agy/gstack-plan-design-review": 110044,
"agy/gstack-plan-devex-review": 116496, "agy/gstack-plan-devex-review": 116496,
"agy/gstack-plan-eng-review": 147241, "agy/gstack-plan-eng-review": 144907,
"agy/gstack-review": 103755, "agy/gstack-review": 103750,
"codex/gstack-autoplan": 108553, "codex/gstack-autoplan": 108553,
"codex/gstack-land-and-deploy": 100284, "codex/gstack-land-and-deploy": 100284,
"codex/gstack-plan-design-review": 111116, "codex/gstack-plan-design-review": 111116,
"codex/gstack-plan-devex-review": 118004, "codex/gstack-plan-devex-review": 118004,
"codex/gstack-plan-eng-review": 148406, "codex/gstack-plan-eng-review": 146072,
"codex/gstack-review": 103278, "codex/gstack-review": 103273,
"copilot/gstack-autoplan": 109253, "copilot/gstack-autoplan": 109253,
"copilot/gstack-plan-design-review": 110355, "copilot/gstack-plan-design-review": 110355,
"copilot/gstack-plan-devex-review": 115958, "copilot/gstack-plan-devex-review": 115958,
"copilot/gstack-plan-eng-review": 147406, "copilot/gstack-plan-eng-review": 145072,
"copilot/gstack-review": 103336, "copilot/gstack-review": 103331,
"cursor/gstack-autoplan": 108485, "cursor/gstack-autoplan": 108485,
"cursor/gstack-plan-design-review": 108900, "cursor/gstack-plan-design-review": 108900,
"cursor/gstack-plan-devex-review": 115228, "cursor/gstack-plan-devex-review": 115228,
"cursor/gstack-plan-eng-review": 145996, "cursor/gstack-plan-eng-review": 143662,
"cursor/gstack-review": 121626, "cursor/gstack-review": 121621,
"factory/gstack-autoplan": 110763, "factory/gstack-autoplan": 110763,
"factory/gstack-land-and-deploy": 100105, "factory/gstack-land-and-deploy": 100105,
"factory/gstack-plan-design-review": 111675, "factory/gstack-plan-design-review": 111675,
"factory/gstack-plan-devex-review": 118591, "factory/gstack-plan-devex-review": 118591,
"factory/gstack-plan-eng-review": 149104, "factory/gstack-plan-eng-review": 146770,
"factory/gstack-review": 125399, "factory/gstack-review": 125394,
"gbrain/gstack-autoplan": 108892, "gbrain/gstack-autoplan": 108892,
"gbrain/gstack-plan-eng-review": 128623, "gbrain/gstack-plan-eng-review": 126289,
"hermes/gstack-autoplan": 108982, "hermes/gstack-autoplan": 108982,
"hermes/gstack-plan-eng-review": 128708, "hermes/gstack-plan-eng-review": 126374,
"kiro/gstack-autoplan": 108259, "kiro/gstack-autoplan": 108259,
"kiro/gstack-plan-design-review": 108752, "kiro/gstack-plan-design-review": 108752,
"kiro/gstack-plan-devex-review": 115080, "kiro/gstack-plan-devex-review": 115080,
"kiro/gstack-plan-eng-review": 145858, "kiro/gstack-plan-eng-review": 143524,
"kiro/gstack-review": 121450, "kiro/gstack-review": 121445,
"openclaw/gstack-autoplan": 108715, "openclaw/gstack-autoplan": 108715,
"openclaw/gstack-plan-eng-review": 127138, "openclaw/gstack-plan-eng-review": 124804,
"opencode/gstack-autoplan": 109234, "opencode/gstack-autoplan": 109234,
"opencode/gstack-plan-design-review": 110187, "opencode/gstack-plan-design-review": 110187,
"opencode/gstack-plan-devex-review": 115836, "opencode/gstack-plan-devex-review": 115836,
"opencode/gstack-plan-eng-review": 147265, "opencode/gstack-plan-eng-review": 144931,
"opencode/gstack-review": 122248, "opencode/gstack-review": 122243,
"slate/gstack-autoplan": 108372, "slate/gstack-autoplan": 108372,
"slate/gstack-plan-design-review": 108826, "slate/gstack-plan-design-review": 108826,
"slate/gstack-plan-devex-review": 115154, "slate/gstack-plan-devex-review": 115154,
"slate/gstack-plan-eng-review": 145927, "slate/gstack-plan-eng-review": 143593,
"slate/gstack-review": 121538 "slate/gstack-review": 121533
} }
} }
+34 -4
View File
@@ -146,15 +146,33 @@ export async function callJudge<T>(
...(opts?.effort === undefined ? {} : { effort: opts.effort }) } }), ...(opts?.effort === undefined ? {} : { effort: opts.effort }) } }),
messages: [{ role: 'user' as const, content: prompt }], messages: [{ role: 'user' as const, content: prompt }],
}; };
const makeRequest = () => opts?.stream // What a cancelled request had received, so a judge deadline names rate-limit
? client.messages.stream(request, signal ? { signal } : undefined).finalMessage() // waits, a stream with no events, or a long thinking phase. Counts only, never content.
: client.messages.create(request, signal ? { signal } : undefined); const started = performance.now();
const elapsedMs = () => Math.round(performance.now() - started);
const progress = { retryWaitsMs: [] as number[], firstEventMs: null as number | null, lastEventMs: null as number | null,
block: null as string | null, thinkingChars: 0, textChars: 0, messageId: null as string | null, requestId: null as string | null };
const makeRequest = () => {
if (!opts?.stream) return client.messages.create(request, signal ? { signal } : undefined);
const stream = client.messages.stream(request, signal ? { signal } : undefined);
stream.on('streamEvent', event => {
progress.firstEventMs ??= elapsedMs();
progress.lastEventMs = elapsedMs();
progress.requestId ??= stream.request_id ?? null;
if (event.type === 'message_start') progress.messageId = event.message.id;
if (event.type === 'content_block_start') progress.block = event.content_block.type;
if (event.type === 'content_block_delta' && event.delta.type === 'thinking_delta') progress.thinkingChars += event.delta.thinking.length;
if (event.type === 'content_block_delta' && event.delta.type === 'text_delta') progress.textChars += event.delta.text.length;
});
return stream.finalMessage();
};
// 429s under CI concurrency: jittered exponential backoff over 3 retries // 429s under CI concurrency: jittered exponential backoff over 3 retries
// (~1s/4s/16s + jitter), honoring the server's retry-after when present. // (~1s/4s/16s + jitter), honoring the server's retry-after when present.
// The old single fixed 1s retry lost races reliably at 40-way concurrency. // The old single fixed 1s retry lost races reliably at 40-way concurrency.
let response; let response;
let attempt = 0; let attempt = 0;
try {
for (;;) { for (;;) {
try { try {
signal?.throwIfAborted(); signal?.throwIfAborted();
@@ -168,13 +186,25 @@ export async function callJudge<T>(
const baseMs = Number.isFinite(retryAfterSecs) && retryAfterSecs > 0 const baseMs = Number.isFinite(retryAfterSecs) && retryAfterSecs > 0
? retryAfterSecs * 1000 ? retryAfterSecs * 1000
: 1000 * 4 ** attempt; : 1000 * 4 ** attempt;
await delay(baseMs + Math.random() * 500, undefined, { signal }).catch(error => { const waitMs = baseMs + Math.random() * 500;
progress.retryWaitsMs.push(Math.round(waitMs));
await delay(waitMs, undefined, { signal }).catch(error => {
signal?.throwIfAborted(); signal?.throwIfAborted();
throw error; throw error;
}); });
attempt += 1; attempt += 1;
} }
} }
} catch (error) {
if (signal?.aborted) {
console.error(JSON.stringify({
type: 'llm-judge-aborted', model: resolvedModel, stream: opts?.stream === true, elapsedMs: elapsedMs(),
requests: attempt + 1, ...progress,
reason: signal.reason instanceof Error ? signal.reason.message : String(signal.reason),
}));
}
throw error;
}
opts?.onResponse?.({ opts?.onResponse?.({
id: typeof response.id === 'string' ? response.id : null, id: typeof response.id === 'string' ? response.id : null,
+19
View File
@@ -85,6 +85,25 @@ describe('callJudge cancellation', () => {
expect(create).toHaveBeenCalledTimes(1); expect(create).toHaveBeenCalledTimes(1);
}, 3000); }, 3000);
test('an abort during a 429 wait retains the wait it interrupted', async () => {
const diagnostics = spyOn(console, 'error').mockImplementation(() => {});
try {
const controller = new AbortController();
create.mockRejectedValue({ status: 429, headers: { 'retry-after': '30' } });
const pending = callJudge('Rate-limited request.', 'test-judge-model', { signal: controller.signal });
await new Promise<void>(resolve => setImmediate(resolve));
controller.abort(new Error('Expired during rate-limit backoff.'));
await expect(pending).rejects.toThrow('Expired during rate-limit backoff.');
expect(diagnostics).toHaveBeenCalledTimes(1);
expect(JSON.parse(diagnostics.mock.calls[0]![0])).toMatchObject({ type: 'llm-judge-aborted', model: 'test-judge-model',
stream: false, requests: 1, retryWaitsMs: [30000], firstEventMs: null, reason: 'Expired during rate-limit backoff.' });
create.mockResolvedValue(response as any);
diagnostics.mockClear();
expect(await callJudge('Judge this.', undefined, { signal: new AbortController().signal })).toEqual({ passed: true });
expect(diagnostics).not.toHaveBeenCalled();
} finally { diagnostics.mockRestore(); }
}, 3000);
test('keeps three 429 retries for callers without a signal', async () => { test('keeps three 429 retries for callers without a signal', async () => {
const rateLimit = { status: 429, headers: { 'retry-after': '0.001' } }; const rateLimit = { status: 429, headers: { 'retry-after': '0.001' } };
create.mockRejectedValue(rateLimit); create.mockRejectedValue(rateLimit);
+56
View File
@@ -103,3 +103,59 @@ test('streaming retains the caller abort signal instead of extending its deadlin
expect(await rejected).toBe(failure); expect(await rejected).toBe(failure);
expect(transport).toHaveBeenCalledTimes(1); expect(transport).toHaveBeenCalledTimes(1);
}); });
test('a judge aborted mid-stream retains what it had received, as counts without content', async () => {
const encoder = new TextEncoder();
const partial = [
{ type: 'message_start', message: { id: 'msg_stalled', type: 'message', role: 'assistant', model: 'claude-fable-5-1',
content: [], stop_reason: null, stop_sequence: null, usage: { input_tokens: 99023, output_tokens: 1 } } },
{ type: 'content_block_start', index: 0, content_block: { type: 'thinking', thinking: '', signature: '' } },
{ type: 'content_block_delta', index: 0, delta: { type: 'thinking_delta', thinking: 'PRIVATE_THINKING' } },
];
let delivered!: () => void;
const received = new Promise<void>(resolve => { delivered = resolve; });
transport.mockImplementation(async (_url: string, init: RequestInit) => new Response(new ReadableStream({
start(stream) {
for (const event of partial) stream.enqueue(encoder.encode(`event: ${event.type}\ndata: ${JSON.stringify(event)}\n\n`));
init.signal!.addEventListener('abort', () => stream.error(init.signal!.reason), { once: true });
setTimeout(delivered, 20);
},
}), { headers: { 'content-type': 'text/event-stream', 'request-id': 'req_stalled' } }));
const controller = new AbortController();
const rejected = callJudge('score the whole bundle', 'claude-fable-5-1', {
max_tokens: 65_536, stream: true, jsonSchema: WORKFLOW_JUDGE_RESPONSE_SCHEMA, signal: controller.signal,
}).catch(error => error);
await received;
const failure = new Error('ship/SKILL.md workflow exceeded its workflow judge work deadline');
controller.abort(failure);
expect(await rejected).toBe(failure);
expect(diagnostics).toHaveBeenCalledTimes(1);
const diagnostic = diagnostics.mock.calls[0]![0];
expect(diagnostic).not.toContain('PRIVATE_');
const event = JSON.parse(diagnostic);
expect(event).toMatchObject({ type: 'llm-judge-aborted', model: 'claude-fable-5-1', stream: true, requests: 1,
retryWaitsMs: [], block: 'thinking', thinkingChars: 'PRIVATE_THINKING'.length, textChars: 0,
messageId: 'msg_stalled', requestId: 'req_stalled', reason: failure.message });
expect(event.firstEventMs).toBeGreaterThanOrEqual(0);
expect(event.lastEventMs).toBeGreaterThanOrEqual(event.firstEventMs);
expect(event.elapsedMs).toBeGreaterThanOrEqual(event.lastEventMs);
});
test('a stream that never sent an event is retained as such, and provider errors without an abort are not', async () => {
transport.mockImplementation((_url: string, init: RequestInit) => new Promise((_resolve, reject) => {
init.signal!.addEventListener('abort', () => reject(init.signal!.reason), { once: true });
}));
const controller = new AbortController();
const rejected = callJudge('score', 'claude-fable-5-1', { max_tokens: 65_536, stream: true, signal: controller.signal }).catch(error => error);
await new Promise(resolve => setTimeout(resolve, 10));
controller.abort(new Error('deadline'));
await rejected;
expect(JSON.parse(diagnostics.mock.calls[0]![0])).toMatchObject({ type: 'llm-judge-aborted', firstEventMs: null,
lastEventMs: null, block: null, thinkingChars: 0, messageId: null, requestId: null, reason: 'deadline' });
diagnostics.mockClear();
transport.mockResolvedValue(new Response(JSON.stringify({ type: 'error', error: { type: 'api_error', message: 'boom' } }),
{ status: 500, headers: { 'content-type': 'application/json', 'x-should-retry': 'false' } }));
await expect(callJudge('score', 'claude-fable-5-1', { max_tokens: 65_536, stream: true, signal: new AbortController().signal })).rejects.toThrow();
expect(diagnostics).not.toHaveBeenCalled();
});
+15
View File
@@ -684,3 +684,18 @@ test('review suppressions allow a clean result and skip common false positives b
expectTokens(suppressions, ['"Function too long"', '"Possible null dereference"', '"N+1 query"', '"Hardcoded value"', '`Math.random()`'], 'suppressions'); expectTokens(suppressions, ['"Function too long"', '"Possible null dereference"', '"N+1 query"', '"Hardcoded value"', '`Math.random()`'], 'suppressions');
expectAbsent(suppressions, [/missing input validation/i, /missing await/i], 'suppressions'); expectAbsent(suppressions, [/missing input validation/i, /missing await/i], 'suppressions');
}); });
test('judged repeat-pass QA reruns and unresolved record actions each have one rule (paid screen 40617200857)', () => {
// Every sample of the red panel and its rerun named one of these: "post-expiry smoke rechecks are not-run" vs
// Step 5.8's rerun of superseded probes, materialize's once-per-PROBE_DIR limit vs reruns into the same report,
// and no record action for an unresolved non-gate defect (an unanswered question or an unproven repair).
// fails_when=the rerun stops being revalidation, materialize may run per pass, or such a defect has no action.
const flat = skill.replace(/\s+/g, ' ');
const repeat = flat.slice(flat.indexOf('2. On a repeat, execute Steps 3–5 in order.'), flat.indexOf('3. **Verify completed actions.**'));
expect(repeat).toContain('These reruns are revalidation, so they run even after smoke expiry; keep the PROBE_DIR and run its once-only `materialize` on the final pass.');
expect(repeat.indexOf('rerun each probe the verdict relies on')).toBeLessThan(repeat.indexOf('These reruns are revalidation'));
const actions = flat.slice(flat.indexOf('Actions: `auto-fixed` (Step 5b)'), flat.indexOf('Advice is never `auto-fixed`'));
expect(actions).toContain('`fixed` (approved **and completed** in Step 5d)');
expect(actions).toContain('`skipped` (explicit Skip in Step 5c), `open` (any other defect, e.g. unanswered).');
expect(flat).toContain('Do not record an unanswered question as skipped');
});
+2 -2
View File
@@ -410,7 +410,7 @@ test('Ship sends its authorized 64k cap and compact response contract through th
expect(WORKFLOW_JUDGE_RESPONSE_SCHEMA.properties.reasoning).not.toHaveProperty('maxLength'); expect(WORKFLOW_JUDGE_RESPONSE_SCHEMA.properties.reasoning).not.toHaveProperty('maxLength');
expect(source.match(/compactReasoning: true/g)).toHaveLength(1); expect(source.match(/compactReasoning: true/g)).toHaveLength(1);
const f = fixture(); const f = fixture();
const stream = spyOn(Messages.prototype, 'stream').mockReturnValue({ finalMessage: async () => ({ const stream = spyOn(Messages.prototype, 'stream').mockReturnValue({ on() { return this; }, finalMessage: async () => ({
stop_reason: 'end_turn', content: [{ type: 'text', text: JSON.stringify(scores) }], stop_reason: 'end_turn', content: [{ type: 'text', text: JSON.stringify(scores) }],
}) } as any); }) } as any);
try { try {
@@ -516,7 +516,7 @@ test('the structured callback rejects incomplete, schema-invalid and below-thres
try { try {
for (const response of responses) { for (const response of responses) {
const f = fixture(); const f = fixture();
stream.mockReturnValue({ finalMessage: async () => ({ stop_reason: response.stop_reason, stream.mockReturnValue({ on() { return this; }, finalMessage: async () => ({ stop_reason: response.stop_reason,
content: [{ type: 'text', text: JSON.stringify(response.value) }] }) } as any); content: [{ type: 'text', text: JSON.stringify(response.value) }] }) } as any);
const h = actualCallback(f, { judge: (prompt, model, request) => callJudge<typeof scores>(prompt, model, request) }); const h = actualCallback(f, { judge: (prompt, model, request) => callJudge<typeof scores>(prompt, model, request) });
await expect(h.run({ ...h.options, schemaTransport: true, compactReasoning: true, maxTokens: 65_536, stream: true })).rejects.toThrow(); await expect(h.run({ ...h.options, schemaTransport: true, compactReasoning: true, maxTokens: 65_536, stream: true })).rejects.toThrow();