mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-01 06:24:41 +01:00
v1.91.309.0 fix(flaky): one rule each in the judged review and Eng prose; a cancelled judge names what it was waiting on (GSTA-139 S3) (#3544)
* fix(flaky): give the judged review and Eng prose one rule each, and name what a cancelled judge received (GSTA-139 S3)
- review: Step 5.8 records an unresolved non-gate defect as open, and a repeat
pass's reruns are revalidation that run after smoke expiry in the same
PROBE_DIR with materialize once on the final pass (every sample of #3527's
red panel 40617200857 and its rerun named one of these). Nearby prose trimmed
so union, skeleton and host pins do not grow.
- plan-eng-review: drop the Eng-only em-dash Voice variant so the shared Voice
hoists; the Scope gate's Format precedence names em dashes (5 of 6 samples of
40634787316 named the near-copy Voice). Host size pins lowered.
- llm-judge: a cancelled judge request prints llm-judge-aborted with timing,
429 waits and stream event counts (no content), so the next workflow-judge
deadline (train 35 gate 40660626349) names its cause.
- benchmark: the Phase 6 example no longer lazy-loads the hero image.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(review): earlier passes link checkpoints; only the final pass materializes (all 3 samples of the focused judge run at 1bdc4ce)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs(changelog): Unreleased entry for the GSTA-139 S3 judge prose repairs
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test: record llm-judge abort diagnostic as non-behavioral; abort tests wait on events, not sleeps
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: pre-landing review fixes
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: name the llm-judge-aborted line in TESTING_INTERNALS
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
---------
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
256874a109
commit
043feff8b3
20 files changed
+318
-103
No files matched your search
@@ -1,5 +1,25 @@
|
||||
# Changelog
|
||||
|
||||
## [1.91.309.0] - 2026-10-29
|
||||
|
||||
**The judged review and Eng prose each give one rule where the paid judges kept finding two, and a judge that runs out of time now says what it was waiting on.** Every sample of two red panels on #3527's screens (review 40617200857, plan-eng-review 40634787316), attempt and rerun, named the same contradictions; a third red (ship, train 35 gate 40660626349) hit the 120 s judge deadline with nothing logged but "No completed model response".
|
||||
|
||||
### Fixed
|
||||
- **`/review` Step 5.8**: an unresolved non-gate defect (an unanswered question) is recorded as `open`, so every finding has a record action. A repeat pass's reruns of superseded probes are revalidation, so they run after smoke expiry, in the same probe directory; earlier provisional QA sections link checkpoints and only the final pass runs the once-only `materialize`.
|
||||
- **`/plan-eng-review`**: the shared Voice now loads once. Eng no longer carries a near-copy whose only difference was its em-dash clause; the Scope gate's format precedence says required formats keep their em dashes.
|
||||
- **`/benchmark` Phase 6 example**: the hero image recommendation no longer suggests lazy-loading (that delays the largest paint) and names one remedy, as the rule under it asks.
|
||||
|
||||
### Changed
|
||||
- **Judge cancellations leave evidence.** When a judge request is cancelled (the workflow judge deadline), `callJudge` prints one `llm-judge-aborted` line: elapsed time, wrapper attempts and 429 waits, HTTP requests and statuses (so the SDK's own retries show), connection, first and last stream event times, the open block type, thinking and text character counts, message and request ids, and the reason (at most 200 characters). Counts only, never content; an observer that throws cannot change a judge result. Deadlines, effort, samples and thresholds are unchanged.
|
||||
|
||||
### Tests
|
||||
- `test/review-workflow-clarity.test.ts` and `test/eng-scope-entry-ap.test.ts` pin the repaired rules; both fail with the prose change stashed.
|
||||
- `test/llm-judge-stream.test.ts` and `test/llm-judge-abort.test.ts` replay a stream cancelled mid-thinking, a stream with no events, one cancelled after headers, an SDK-retried 429, a cancelled 429 wait, an event the observer cannot read and an overlong reason, with no-abort controls; each fails with the helper change stashed. `test/workflow-judge-cache.test.ts`'s SDK stream stand-ins gain the real stream's `on()`.
|
||||
- Focused paid judge runs: review 4/4/4 at 1bdc4ce and again at 3463250, plan-eng-review 3/5/4 at 1bdc4ce ($0.18).
|
||||
|
||||
### Gate integrity
|
||||
- No threshold, sample, trial, deadline or case-kind change. `scripts/harness-version.json` records the `llm-judge.ts` change as non-behavioral (no series reset). Generated SKILL.md size pins for review and plan-eng-review were lowered to their new sizes on every host.
|
||||
|
||||
## [1.91.308.0] - 2026-10-29
|
||||
|
||||
**`gstack-restamp` no longer cuts a four-part package.json version to three parts in a repo whose convention is four.** On garrytan/gbrain#7878, `/ship` restamped gbrain's `package.json` to `"0.60.483"` beside `VERSION` `0.60.483.0`; gbrain requires the two to match and its CI version gate fails when they drift, so the stamp had to be reverted by hand. The version stage called `gstack-version-bump write`, which always stores the npm-valid three-part form in the manifest (decision 11, which is why gstack's own `package.json` reads `1.91.298`).
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# gstack digest v1.91.308.0 — regenerate/re-copy after upgrading gstack
|
||||
# gstack digest v1.91.309.0 — regenerate/re-copy after upgrading gstack
|
||||
|
||||
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
|
||||
for agent hosts without a full skill install. The full skills add workflows,
|
||||
|
||||
+1
-1
@@ -412,7 +412,7 @@ TOP 10 SLOWEST RESOURCES
|
||||
RECOMMENDATIONS:
|
||||
- vendor.chunk.js: code-split — 320KB on initial load
|
||||
- analytics.js: load async/defer — blocks rendering for 250ms
|
||||
- hero-image.webp: add width/height (CLS), lazy-load
|
||||
- hero-image.webp: add width/height (CLS)
|
||||
```
|
||||
|
||||
Third-party = any host but the page's origin. Each recommendation names a resource here or in a Phase 5 regression, the metric it moves and one remedy (split, defer, compress, lazy-load, cache).
|
||||
|
||||
@@ -201,7 +201,7 @@ TOP 10 SLOWEST RESOURCES
|
||||
RECOMMENDATIONS:
|
||||
- vendor.chunk.js: code-split — 320KB on initial load
|
||||
- analytics.js: load async/defer — blocks rendering for 250ms
|
||||
- hero-image.webp: add width/height (CLS), lazy-load
|
||||
- hero-image.webp: add width/height (CLS)
|
||||
```
|
||||
|
||||
Third-party = any host but the page's origin. Each recommendation names a resource here or in a Phase 5 regression, the metric it moves and one remedy (split, defer, compress, lazy-load, cache).
|
||||
|
||||
@@ -783,6 +783,14 @@ once, plus cleanup. Each still has 120 seconds of model work. Its 17 workflow
|
||||
judges own their deadline and abort signal, with five seconds for terminal
|
||||
recording inside a ten-second Bun grace; the other 11 retain their existing
|
||||
120-second Bun timeout. Late responses cannot create records or cache passes.
|
||||
When a judge's abort signal cancels its request, `callJudge`
|
||||
(`test/helpers/llm-judge.ts`) prints one `llm-judge-aborted` JSON line to
|
||||
stderr: elapsed time, wrapper `attempts` and `retryWaitsMs` (429 waits),
|
||||
`httpRequests` and `httpStatuses` (so SDK retries show), `connectedMs`,
|
||||
`firstEventMs` and `lastEventMs`, the open `block` type, `thinkingChars` and
|
||||
`textChars`, `messageId`, `requestId` and the `reason` (at most 200
|
||||
characters). It holds counts and ids only, never content, and changes no
|
||||
request, score or verdict.
|
||||
|
||||
The ship documentation file reserves 4,920 seconds for four 600-second cases and
|
||||
eight 300-second fault cases, run once, plus cleanup; in CI each case runs as its
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gstack",
|
||||
"version": "1.91.308",
|
||||
"version": "1.91.309",
|
||||
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
|
||||
@@ -50,7 +50,7 @@ Recommendation: A when a branch diff exists, otherwise B. Reply with A, B, or C.
|
||||
|
||||
After target selection, use the preamble's full decision brief, transport and continuous D-numbering. Setup questions approve no engineering remedies.
|
||||
|
||||
**Format precedence:** Copy required command, output and question formats exactly. Apply Voice to newly composed prose.
|
||||
**Format precedence:** Copy required command, output and question formats exactly, em dashes included. Apply Voice to newly composed prose.
|
||||
|
||||
**Startup sequence** (after target selection):
|
||||
1. Run the Preamble command and its startup instructions (Context Recovery and the preamble's own one-time setup prompts; the Prior Learnings cross-project question waits for Step 0). Defer Operational Self-Improvement, Telemetry and Plan Status Footer to finish; format/transport rules apply throughout.
|
||||
@@ -187,28 +187,6 @@ Before emitting a tool or prose decision brief, verify:
|
||||
- [ ] Follow Tool resolution: tool call unless Conductor or documented prose fallback; prose includes the mandatory triad + explicit reply selectors, then STOP. Spawned sessions follow their auto-choice rule.
|
||||
- [ ] Write non-ASCII directly, not \u-escaped. For 5+ options, split/batch into ≤4 without dropping; check dependencies and stop the chain immediately on Hold.
|
||||
|
||||
## Voice
|
||||
|
||||
GStack voice: Garry-shaped product and engineering judgment.
|
||||
|
||||
- Lead with the point. Say what it does, why it matters, and what changes for the builder.
|
||||
- Be concrete. Name files, functions, line numbers, commands, outputs, evals, and real numbers.
|
||||
- Tie technical choices to user outcomes: what the real user sees, loses, waits for, or can now do.
|
||||
- Be direct about quality. Bugs matter. Edge cases matter. Fix the whole thing, not the demo path.
|
||||
- Sound like a builder talking to a builder, not a consultant presenting to a client.
|
||||
- Never corporate, academic, PR, or hype. Avoid filler, throat-clearing, generic optimism, and founder cosplay.
|
||||
- Do not add em dashes in prose you compose during the review. Existing templates, quoted text, command output, and required copied labels may contain them. No AI vocabulary: delve, crucial, robust, comprehensive, nuanced, multifaceted, furthermore, moreover, additionally, pivotal, landscape, tapestry, underscore, foster, showcase, intricate, vibrant, fundamental, significant, load-bearing.
|
||||
- Reply in the language of the user's latest message unless asked otherwise. Code, commands, paths, identifiers, quoted output and question markers (`D<N>`, option letters, `(recommended)`) stay verbatim.
|
||||
- The user has context you do not: domain knowledge, timing, relationships, taste. Cross-model agreement is a recommendation, not a decision. The user decides.
|
||||
|
||||
Good: "auth.ts:47 returns undefined when the session cookie expires. Users hit a white screen. Fix: add a null check and redirect to /login. Two lines."
|
||||
Bad: "I've identified a potential issue in the authentication flow that may cause problems under certain conditions."
|
||||
|
||||
**Bounded closer.** After completing work, report in at most a few short lines: what changed, what was skipped, what to watch. No feature tours or unrequested design notes. Exempt: decision briefs, completion-status blocks, requested explanations, and a skill's mandated report (/qa-only, /plan-*-review, /retro, /document-generate). The rule limits prose around the deliverable, never the deliverable.
|
||||
|
||||
Good closer: "Renamed the flag in 3 files, regenerated docs, tests green. Skipped the CLI alias (unused since v1.2); watch the Windows job."
|
||||
Bad closer: a tour of every edit, a restatement of the plan, and three paragraphs justifying choices nobody questioned.
|
||||
|
||||
## Question Tuning (skip entirely if `QUESTION_TUNING: false`)
|
||||
|
||||
Before each decision brief (AskUserQuestion or Conductor/fallback prose), choose `question_id` from `~/.claude/skills/gstack/scripts/question-registry.ts` or `{skill}-{slug}`, then run `~/.claude/skills/gstack/bin/gstack-question-preference --check "<id>"`; for an unregistered id, write the question summary to `.gstack/tmp/qt.txt` (file-write tool) and append `--summary-file .gstack/tmp/qt.txt` (one-way keyword check). `AUTO_DECIDE` means choose the recommended option and say "Auto-decided [summary] → [option] (your preference). Change with /plan-tune." `ASK_NORMALLY` means ask.
|
||||
|
||||
@@ -60,7 +60,7 @@ Recommendation: A when a branch diff exists, otherwise B. Reply with A, B, or C.
|
||||
|
||||
After target selection, use the preamble's full decision brief, transport and continuous D-numbering. Setup questions approve no engineering remedies.
|
||||
|
||||
**Format precedence:** Copy required command, output and question formats exactly. Apply Voice to newly composed prose.
|
||||
**Format precedence:** Copy required command, output and question formats exactly, em dashes included. Apply Voice to newly composed prose.
|
||||
|
||||
**Startup sequence** (after target selection):
|
||||
1. Run the Preamble command and its startup instructions (Context Recovery and the preamble's own one-time setup prompts; the Prior Learnings cross-project question waits for Step 0). Defer Operational Self-Improvement, Telemetry and Plan Status Footer to finish; format/transport rules apply throughout.
|
||||
|
||||
+14
-12
@@ -650,9 +650,9 @@ findings and decides completion.
|
||||
|
||||
## Step 5: Fix-First Review
|
||||
|
||||
Before edits, confirm every dispatched reader has returned or is confirmed stopped.
|
||||
For an active or unknown reader/writer, wait or confirm it is stopped; if that
|
||||
cannot be confirmed, persist incomplete at Step 5.8 and STOP without edits.
|
||||
Before edits, every dispatched reader/writer must have returned or be confirmed
|
||||
stopped: wait, and if that cannot be confirmed, persist incomplete at Step 5.8 and
|
||||
STOP without edits.
|
||||
Terminal failure does not block fixes from independent evidence. Missing required
|
||||
output still makes the pass incomplete, even after the reader is stopped.
|
||||
|
||||
@@ -806,7 +806,7 @@ After outputting your own findings, if Greptile comments were classified in Step
|
||||
|
||||
Before replying to any comment, run greptile-triage.md's **Escalation Detection** to choose Tier 1 (friendly) or Tier 2 (firm) reply templates.
|
||||
|
||||
1. **VALID & ACTIONABLE comments:** Use their Step 5a–5d disposition; do not ask a second fix question. Step 5c alone supplies A) Fix / B) Skip for ASK items. After a completed fix, use the **Fix reply template** with diff and explanation; cite the current diff if uncommitted, never invent a commit SHA. A Skip leaves the defect unresolved and grants no new fix permission. If evidence disproves the finding, reclassify it below.
|
||||
1. **VALID & ACTIONABLE comments:** Use their Step 5a–5d disposition; do not ask a second fix question. After a completed fix, use the **Fix reply template** with diff and explanation; cite the current diff if uncommitted, never invent a commit SHA. A Skip leaves the defect unresolved and grants no new fix permission. If evidence disproves the finding, reclassify it below.
|
||||
|
||||
2. **FALSE POSITIVE comments:** These are reply decisions, not code approval. Show file:line (or [top-level]), summary, permalink and evidence, then ask:
|
||||
- A) Reply explaining why this is incorrect (recommended if clearly wrong)
|
||||
@@ -835,6 +835,9 @@ Before replying to any comment, run greptile-triage.md's **Escalation Detection*
|
||||
final: repeat Steps 3–5 with a full start.
|
||||
2. On a repeat, execute Steps 3–5 in order. At Step 4.7, an edited pass supersedes
|
||||
every earlier QA capture (exploratory §4): rerun each probe the verdict relies on.
|
||||
These reruns are revalidation, so they run even after smoke expiry; keep the
|
||||
PROBE_DIR. Earlier provisional sections link checkpoints; only the final pass
|
||||
runs its once-only `materialize`.
|
||||
A zero-edit pass reuses only this invocation's unchanged-input QA evidence,
|
||||
rerunning any probe whose inputs changed or are uncertain. Reusing never skips a review step.
|
||||
3. **Verify completed actions.** On the final zero-edit pass, reconcile this
|
||||
@@ -846,9 +849,8 @@ Before replying to any comment, run greptile-triage.md's **Escalation Detection*
|
||||
defects and unresolved counts; earlier fixes do not suppress them.
|
||||
4. **Recheck skipped advice.** Re-read its final-snapshot supporting source and
|
||||
reconfirm the decision; otherwise report its history without a reusable skip.
|
||||
The logger computes `snapshot_covered_paths` from eligible paths whose raw bytes
|
||||
equal the bound snapshot blobs (`[]` if none). Never carry prior-cycle, supplied
|
||||
or prior-record coverage forward or build this proof yourself. Fixed advice
|
||||
The logger computes `snapshot_covered_paths` itself; never carry prior-cycle,
|
||||
supplied or prior-record coverage forward or build this proof yourself. Fixed advice
|
||||
needs no skip coverage.
|
||||
|
||||
### 2. Fill the record
|
||||
@@ -878,14 +880,14 @@ pass: report it with its reason.
|
||||
`helper_target`. Recheck source after fixes. The logger uses `sharedLibsFingerprint`,
|
||||
never supplied/model hashes.
|
||||
Actions: `auto-fixed` (Step 5b), `fixed` (approved **and completed** in Step 5d),
|
||||
`skipped` (explicit Skip in Step 5c). Advice is never `auto-fixed`; pending
|
||||
`skipped` (explicit Skip in Step 5c),
|
||||
`open` (any other defect, e.g. unanswered). Advice is never `auto-fixed`; pending
|
||||
advice stays in the response, not the record. Exclude prior Step 5.0
|
||||
suppressions; include this invocation's revalidated decisions.
|
||||
- Gate findings additionally carry `gate` (the RH tag, e.g. `"RH-15"`), `gate_id`
|
||||
(the tool's hunk identity), `reason` (for `kept`) and one of the actions
|
||||
`kept`, `restored` or `open`. Never store hunk text, test names or commit
|
||||
messages in the record (it is brain-synced). `open` findings count in
|
||||
`issues_found` and in `critical`/`informational` by severity. The logger stamps
|
||||
messages in the record (it is brain-synced). The logger stamps
|
||||
`actor` from `gstack-session-kind` itself and discards any `actor` you supply;
|
||||
for a non-interactive session it also forces `action: "open"` on every gate
|
||||
finding. Dispositions are per invocation: `/ship`'s own review pass asks again.
|
||||
@@ -897,8 +899,8 @@ pass: report it with its reason.
|
||||
`TIMESTAMP` is ISO 8601; `COMMIT` is `git rev-parse --short HEAD`.
|
||||
`quality_score` is Step 4.6's specialist score (`10.0` when small-diff specialists
|
||||
were skipped or this host omits Review Army). This default is not completion evidence;
|
||||
unresolved non-advisory core defects still count in `issues_found`,
|
||||
`critical`, `informational`. The logger builds trusted `review_binding` from the
|
||||
unresolved defects, `open` ones included, still count in `issues_found` and
|
||||
by severity in `critical`/`informational`. The logger builds trusted `review_binding` from the
|
||||
validated captured branch digest, discarding caller bindings. Never invent a binding
|
||||
or replace REVIEW_START at log time; finish only this pass's own token.
|
||||
|
||||
|
||||
+14
-12
@@ -251,9 +251,9 @@ critical finding or another writer during collection. Skip silently if no docs e
|
||||
|
||||
## Step 5: Fix-First Review
|
||||
|
||||
Before edits, confirm every dispatched reader has returned or is confirmed stopped.
|
||||
For an active or unknown reader/writer, wait or confirm it is stopped; if that
|
||||
cannot be confirmed, persist incomplete at Step 5.8 and STOP without edits.
|
||||
Before edits, every dispatched reader/writer must have returned or be confirmed
|
||||
stopped: wait, and if that cannot be confirmed, persist incomplete at Step 5.8 and
|
||||
STOP without edits.
|
||||
Terminal failure does not block fixes from independent evidence. Missing required
|
||||
output still makes the pass incomplete, even after the reader is stopped.
|
||||
|
||||
@@ -357,7 +357,7 @@ After outputting your own findings, if Greptile comments were classified in Step
|
||||
|
||||
Before replying to any comment, run greptile-triage.md's **Escalation Detection** to choose Tier 1 (friendly) or Tier 2 (firm) reply templates.
|
||||
|
||||
1. **VALID & ACTIONABLE comments:** Use their Step 5a–5d disposition; do not ask a second fix question. Step 5c alone supplies A) Fix / B) Skip for ASK items. After a completed fix, use the **Fix reply template** with diff and explanation; cite the current diff if uncommitted, never invent a commit SHA. A Skip leaves the defect unresolved and grants no new fix permission. If evidence disproves the finding, reclassify it below.
|
||||
1. **VALID & ACTIONABLE comments:** Use their Step 5a–5d disposition; do not ask a second fix question. After a completed fix, use the **Fix reply template** with diff and explanation; cite the current diff if uncommitted, never invent a commit SHA. A Skip leaves the defect unresolved and grants no new fix permission. If evidence disproves the finding, reclassify it below.
|
||||
|
||||
2. **FALSE POSITIVE comments:** These are reply decisions, not code approval. Show file:line (or [top-level]), summary, permalink and evidence, then ask:
|
||||
- A) Reply explaining why this is incorrect (recommended if clearly wrong)
|
||||
@@ -386,6 +386,9 @@ Before replying to any comment, run greptile-triage.md's **Escalation Detection*
|
||||
final: repeat Steps 3–5 with a full start.
|
||||
2. On a repeat, execute Steps 3–5 in order. At Step 4.7, an edited pass supersedes
|
||||
every earlier QA capture (exploratory §4): rerun each probe the verdict relies on.
|
||||
These reruns are revalidation, so they run even after smoke expiry; keep the
|
||||
PROBE_DIR. Earlier provisional sections link checkpoints; only the final pass
|
||||
runs its once-only `materialize`.
|
||||
A zero-edit pass reuses only this invocation's unchanged-input QA evidence,
|
||||
rerunning any probe whose inputs changed or are uncertain. Reusing never skips a review step.
|
||||
3. **Verify completed actions.** On the final zero-edit pass, reconcile this
|
||||
@@ -397,9 +400,8 @@ Before replying to any comment, run greptile-triage.md's **Escalation Detection*
|
||||
defects and unresolved counts; earlier fixes do not suppress them.
|
||||
4. **Recheck skipped advice.** Re-read its final-snapshot supporting source and
|
||||
reconfirm the decision; otherwise report its history without a reusable skip.
|
||||
The logger computes `snapshot_covered_paths` from eligible paths whose raw bytes
|
||||
equal the bound snapshot blobs (`[]` if none). Never carry prior-cycle, supplied
|
||||
or prior-record coverage forward or build this proof yourself. Fixed advice
|
||||
The logger computes `snapshot_covered_paths` itself; never carry prior-cycle,
|
||||
supplied or prior-record coverage forward or build this proof yourself. Fixed advice
|
||||
needs no skip coverage.
|
||||
|
||||
### 2. Fill the record
|
||||
@@ -429,14 +431,14 @@ pass: report it with its reason.
|
||||
`helper_target`. Recheck source after fixes. The logger uses `sharedLibsFingerprint`,
|
||||
never supplied/model hashes.
|
||||
Actions: `auto-fixed` (Step 5b), `fixed` (approved **and completed** in Step 5d),
|
||||
`skipped` (explicit Skip in Step 5c). Advice is never `auto-fixed`; pending
|
||||
`skipped` (explicit Skip in Step 5c),
|
||||
`open` (any other defect, e.g. unanswered). Advice is never `auto-fixed`; pending
|
||||
advice stays in the response, not the record. Exclude prior Step 5.0
|
||||
suppressions; include this invocation's revalidated decisions.
|
||||
- Gate findings additionally carry `gate` (the RH tag, e.g. `"RH-15"`), `gate_id`
|
||||
(the tool's hunk identity), `reason` (for `kept`) and one of the actions
|
||||
`kept`, `restored` or `open`. Never store hunk text, test names or commit
|
||||
messages in the record (it is brain-synced). `open` findings count in
|
||||
`issues_found` and in `critical`/`informational` by severity. The logger stamps
|
||||
messages in the record (it is brain-synced). The logger stamps
|
||||
`actor` from `gstack-session-kind` itself and discards any `actor` you supply;
|
||||
for a non-interactive session it also forces `action: "open"` on every gate
|
||||
finding. Dispositions are per invocation: `/ship`'s own review pass asks again.
|
||||
@@ -448,8 +450,8 @@ pass: report it with its reason.
|
||||
`TIMESTAMP` is ISO 8601; `COMMIT` is `git rev-parse --short HEAD`.
|
||||
`quality_score` is Step 4.6's specialist score (`10.0` when small-diff specialists
|
||||
were skipped or this host omits Review Army). This default is not completion evidence;
|
||||
unresolved non-advisory core defects still count in `issues_found`,
|
||||
`critical`, `informational`. The logger builds trusted `review_binding` from the
|
||||
unresolved defects, `open` ones included, still count in `issues_found` and
|
||||
by severity in `critical`/`informational`. The logger builds trusted `review_binding` from the
|
||||
validated captured branch digest, discarding caller bindings. Never invent a binding
|
||||
or replace REVIEW_START at log time; finish only this pass's own token.
|
||||
|
||||
|
||||
@@ -29,7 +29,7 @@
|
||||
"test/helpers/eval-budgets.ts": {"blob":"667d0e09b20fe5d7366f57b77b5fb9b04baa37ef","since":6},
|
||||
"test/helpers/eval-store.ts": {"blob":"a50ea2c28f508ad9f35ea9353dd2f502ee0dcfc2","since":6,"nonBehavioral":"credential rejections are classed infra instead of assertion: no case's pass or fail changes; only failed trials whose provider rejected the run's key move class, so no rerun spends on a rejected key"},
|
||||
"test/helpers/hermetic-env.ts": {"blob":"cc072895461dc52ea317d1beff72121342420d30","since":6},
|
||||
"test/helpers/llm-judge.ts": {"blob":"a3177665b662932ce485ccc976c2b36a1531a0bb","since":6,"nonBehavioral":"callJudge rewrites only the message of a JSON parse failure (names the judge output, stop reason, length and the retained llm-judge-response-parse-error event); it still throws a SyntaxError before any score is parsed, the request is unchanged, and no case's pass or fail changes"},
|
||||
"test/helpers/llm-judge.ts": {"blob":"095f9c85dd0ae29da8ba10fed91510313a4442a4","since":6,"nonBehavioral":"callJudge emits one llm-judge-aborted diagnostic line (wrapper attempts, HTTP request and status counts seen through a pass-through fetch, 429 waits, connect and event timings, block kind, character counts, message and request ids, a capped reason; no content) when its caller's signal aborts it; observers swallow their own errors, and the request, retries, parsing, scores and the thrown abort reason are unchanged, so no case's pass or fail changes"},
|
||||
"test/helpers/paid-test-set.ts": {"blob":"ff9bfe43c43ec2bdd11114f7b8ec7c5736116f75","since":6},
|
||||
"test/helpers/pty-screen.ts": {"blob":"4f77569f9ce11f9cee9f347216e468cd7c956820","since":6},
|
||||
"test/helpers/pty/auq.ts": {"blob":"16e5ea2bb5b9f28f44808a1a09024dbd3e6f28c4","since":6},
|
||||
|
||||
@@ -1,17 +1,13 @@
|
||||
|
||||
import type { TemplateContext } from '../types';
|
||||
|
||||
export function generateVoiceDirective(ctx: TemplateContext, tier: number): string {
|
||||
const emDashRule = ctx.skillName === 'plan-eng-review'
|
||||
? 'Do not add em dashes in prose you compose during the review. Existing templates, quoted text, command output, and required copied labels may contain them.'
|
||||
: 'No em dashes.';
|
||||
|
||||
export function generateVoiceDirective(_ctx: TemplateContext, tier: number): string {
|
||||
if (tier <= 1) {
|
||||
return `## Voice
|
||||
|
||||
Direct, concrete, builder-to-builder. Name the file, function, command, and user-visible impact. No filler.
|
||||
|
||||
${emDashRule} No AI vocabulary: delve, crucial, robust, comprehensive, nuanced, multifaceted, load-bearing. Never corporate or academic. Short paragraphs. End with what to do.
|
||||
No em dashes. No AI vocabulary: delve, crucial, robust, comprehensive, nuanced, multifaceted, load-bearing. Never corporate or academic. Short paragraphs. End with what to do.
|
||||
|
||||
Reply in the language of the user's latest message unless asked otherwise. Code, commands, paths, identifiers and quoted output stay verbatim.
|
||||
|
||||
@@ -28,7 +24,7 @@ GStack voice: Garry-shaped product and engineering judgment.
|
||||
- Be direct about quality. Bugs matter. Edge cases matter. Fix the whole thing, not the demo path.
|
||||
- Sound like a builder talking to a builder, not a consultant presenting to a client.
|
||||
- Never corporate, academic, PR, or hype. Avoid filler, throat-clearing, generic optimism, and founder cosplay.
|
||||
- ${emDashRule} No AI vocabulary: delve, crucial, robust, comprehensive, nuanced, multifaceted, furthermore, moreover, additionally, pivotal, landscape, tapestry, underscore, foster, showcase, intricate, vibrant, fundamental, significant, load-bearing.
|
||||
- No em dashes. No AI vocabulary: delve, crucial, robust, comprehensive, nuanced, multifaceted, furthermore, moreover, additionally, pivotal, landscape, tapestry, underscore, foster, showcase, intricate, vibrant, fundamental, significant, load-bearing.
|
||||
- Reply in the language of the user's latest message unless asked otherwise. Code, commands, paths, identifiers, quoted output and question markers (\`D<N>\`, option letters, \`(recommended)\`) stay verbatim.
|
||||
- The user has context you do not: domain knowledge, timing, relationships, taste. Cross-model agreement is a recommendation, not a decision. The user decides.
|
||||
|
||||
|
||||
@@ -163,3 +163,16 @@ test('calibration keeps its future hook but no shipped source enables the absent
|
||||
}
|
||||
expect(mentions).toEqual(['scripts/resolvers/gbrain.ts']);
|
||||
});
|
||||
|
||||
test('the judged Eng bundle carries the shared Voice once and states its em-dash precedence in the Scope gate (paid screen 40634787316)', () => {
|
||||
// Five of six samples (red panel and rerun) named Eng's near-copy of the shared Voice, whose only difference was
|
||||
// its em-dash clause, as a rule the agent must reconcile; the Scope gate's Format precedence already owned it.
|
||||
// fails_when=a second Voice section returns to the judged bundle or the precedence stops naming em dashes.
|
||||
const input = readWorkflowJudgeInput({root: path.join(import.meta.dir, '..'), skillPath: 'plan-eng-review/SKILL.md',
|
||||
startMarker: '# Plan Review Mode', endMarker: null});
|
||||
expect(input.text.match(/^## Voice$/gm)).toHaveLength(1);
|
||||
expect(input.text).not.toContain('Do not add em dashes in prose you compose');
|
||||
expect(scope).toContain('**Format precedence:** Copy required command, output and question formats exactly, em dashes included. Apply Voice to newly composed prose.');
|
||||
const startup = input.files[0]!.content;
|
||||
expect(startup.indexOf('**Format precedence:**')).toBeLessThan(startup.indexOf('## Shared preamble (load first)'));
|
||||
});
|
||||
+19
-19
@@ -5,50 +5,50 @@
|
||||
"agy/gstack-autoplan": 109810,
|
||||
"agy/gstack-plan-design-review": 110044,
|
||||
"agy/gstack-plan-devex-review": 116496,
|
||||
"agy/gstack-plan-eng-review": 147241,
|
||||
"agy/gstack-review": 103755,
|
||||
"agy/gstack-plan-eng-review": 144907,
|
||||
"agy/gstack-review": 103743,
|
||||
"codex/gstack-autoplan": 108553,
|
||||
"codex/gstack-land-and-deploy": 100284,
|
||||
"codex/gstack-plan-design-review": 111116,
|
||||
"codex/gstack-plan-devex-review": 118004,
|
||||
"codex/gstack-plan-eng-review": 148406,
|
||||
"codex/gstack-review": 103278,
|
||||
"codex/gstack-plan-eng-review": 146072,
|
||||
"codex/gstack-review": 103266,
|
||||
"copilot/gstack-autoplan": 109253,
|
||||
"copilot/gstack-plan-design-review": 110355,
|
||||
"copilot/gstack-plan-devex-review": 115958,
|
||||
"copilot/gstack-plan-eng-review": 147406,
|
||||
"copilot/gstack-review": 103336,
|
||||
"copilot/gstack-plan-eng-review": 145072,
|
||||
"copilot/gstack-review": 103324,
|
||||
"cursor/gstack-autoplan": 108485,
|
||||
"cursor/gstack-plan-design-review": 108900,
|
||||
"cursor/gstack-plan-devex-review": 115228,
|
||||
"cursor/gstack-plan-eng-review": 145996,
|
||||
"cursor/gstack-review": 121626,
|
||||
"cursor/gstack-plan-eng-review": 143662,
|
||||
"cursor/gstack-review": 121614,
|
||||
"factory/gstack-autoplan": 110763,
|
||||
"factory/gstack-land-and-deploy": 100105,
|
||||
"factory/gstack-plan-design-review": 111675,
|
||||
"factory/gstack-plan-devex-review": 118591,
|
||||
"factory/gstack-plan-eng-review": 149104,
|
||||
"factory/gstack-review": 125399,
|
||||
"factory/gstack-plan-eng-review": 146770,
|
||||
"factory/gstack-review": 125387,
|
||||
"gbrain/gstack-autoplan": 108892,
|
||||
"gbrain/gstack-plan-eng-review": 128623,
|
||||
"gbrain/gstack-plan-eng-review": 126289,
|
||||
"hermes/gstack-autoplan": 108982,
|
||||
"hermes/gstack-plan-eng-review": 128708,
|
||||
"hermes/gstack-plan-eng-review": 126374,
|
||||
"kiro/gstack-autoplan": 108259,
|
||||
"kiro/gstack-plan-design-review": 108752,
|
||||
"kiro/gstack-plan-devex-review": 115080,
|
||||
"kiro/gstack-plan-eng-review": 145858,
|
||||
"kiro/gstack-review": 121450,
|
||||
"kiro/gstack-plan-eng-review": 143524,
|
||||
"kiro/gstack-review": 121438,
|
||||
"openclaw/gstack-autoplan": 108715,
|
||||
"openclaw/gstack-plan-eng-review": 127138,
|
||||
"openclaw/gstack-plan-eng-review": 124804,
|
||||
"opencode/gstack-autoplan": 109234,
|
||||
"opencode/gstack-plan-design-review": 110187,
|
||||
"opencode/gstack-plan-devex-review": 115836,
|
||||
"opencode/gstack-plan-eng-review": 147265,
|
||||
"opencode/gstack-review": 122248,
|
||||
"opencode/gstack-plan-eng-review": 144931,
|
||||
"opencode/gstack-review": 122236,
|
||||
"slate/gstack-autoplan": 108372,
|
||||
"slate/gstack-plan-design-review": 108826,
|
||||
"slate/gstack-plan-devex-review": 115154,
|
||||
"slate/gstack-plan-eng-review": 145927,
|
||||
"slate/gstack-review": 121538
|
||||
"slate/gstack-plan-eng-review": 143593,
|
||||
"slate/gstack-review": 121526
|
||||
}
|
||||
}
|
||||
+63
-21
@@ -135,7 +135,21 @@ export async function callJudge<T>(
|
||||
// could be exhausted before a frontier judge emitted any JSON.
|
||||
const resolvedModel = resolveEvalModel('judge', model);
|
||||
const maxTokens = opts?.max_tokens ?? DEFAULT_JUDGE_MAX_TOKENS;
|
||||
const client = new Anthropic();
|
||||
// What a cancelled request had received, so a judge deadline names rate-limit
|
||||
// waits, SDK retries, a stream with no events, or a long thinking phase. Counts
|
||||
// only, never content; an observer that throws never changes the judge outcome.
|
||||
const started = performance.now();
|
||||
const elapsedMs = () => Math.round(performance.now() - started);
|
||||
const progress = { httpRequests: 0, httpStatuses: [] as number[], retryWaitsMs: [] as number[], connectedMs: null as number | null,
|
||||
firstEventMs: null as number | null, lastEventMs: null as number | null, block: null as string | null, thinkingChars: 0,
|
||||
textChars: 0, messageId: null as string | null, requestId: null as string | null };
|
||||
const observe = (record: () => void) => { try { record(); } catch {} };
|
||||
const client = new Anthropic({ fetch: async (...args: Parameters<typeof fetch>) => {
|
||||
progress.httpRequests += 1;
|
||||
const response = await globalThis.fetch(...args);
|
||||
progress.httpStatuses.push(response.status);
|
||||
return response;
|
||||
} });
|
||||
|
||||
const request = {
|
||||
model: resolvedModel,
|
||||
@@ -146,34 +160,62 @@ export async function callJudge<T>(
|
||||
...(opts?.effort === undefined ? {} : { effort: opts.effort }) } }),
|
||||
messages: [{ role: 'user' as const, content: prompt }],
|
||||
};
|
||||
const makeRequest = () => opts?.stream
|
||||
? client.messages.stream(request, signal ? { signal } : undefined).finalMessage()
|
||||
: client.messages.create(request, signal ? { signal } : undefined);
|
||||
const makeRequest = () => {
|
||||
if (!opts?.stream) return client.messages.create(request, signal ? { signal } : undefined);
|
||||
const stream = client.messages.stream(request, signal ? { signal } : undefined);
|
||||
stream.on('connect', () => observe(() => {
|
||||
progress.connectedMs ??= elapsedMs();
|
||||
progress.requestId ??= stream.request_id ?? null;
|
||||
}));
|
||||
stream.on('streamEvent', event => observe(() => {
|
||||
progress.firstEventMs ??= elapsedMs();
|
||||
progress.lastEventMs = elapsedMs();
|
||||
progress.requestId ??= stream.request_id ?? null;
|
||||
if (event.type === 'message_start') progress.messageId = event.message.id;
|
||||
if (event.type === 'content_block_start') progress.block = event.content_block.type;
|
||||
if (event.type === 'content_block_delta' && event.delta.type === 'thinking_delta') progress.thinkingChars += event.delta.thinking.length;
|
||||
if (event.type === 'content_block_delta' && event.delta.type === 'text_delta') progress.textChars += event.delta.text.length;
|
||||
}));
|
||||
return stream.finalMessage();
|
||||
};
|
||||
|
||||
// 429s under CI concurrency: jittered exponential backoff over 3 retries
|
||||
// (~1s/4s/16s + jitter), honoring the server's retry-after when present.
|
||||
// The old single fixed 1s retry lost races reliably at 40-way concurrency.
|
||||
let response;
|
||||
let attempt = 0;
|
||||
for (;;) {
|
||||
try {
|
||||
signal?.throwIfAborted();
|
||||
response = await makeRequest();
|
||||
signal?.throwIfAborted();
|
||||
break;
|
||||
} catch (err: any) {
|
||||
signal?.throwIfAborted();
|
||||
if (err?.status !== 429 || attempt >= 3) throw err;
|
||||
const retryAfterSecs = Number(err?.headers?.['retry-after']);
|
||||
const baseMs = Number.isFinite(retryAfterSecs) && retryAfterSecs > 0
|
||||
? retryAfterSecs * 1000
|
||||
: 1000 * 4 ** attempt;
|
||||
await delay(baseMs + Math.random() * 500, undefined, { signal }).catch(error => {
|
||||
try {
|
||||
for (;;) {
|
||||
try {
|
||||
signal?.throwIfAborted();
|
||||
throw error;
|
||||
});
|
||||
attempt += 1;
|
||||
response = await makeRequest();
|
||||
signal?.throwIfAborted();
|
||||
break;
|
||||
} catch (err: any) {
|
||||
signal?.throwIfAborted();
|
||||
if (err?.status !== 429 || attempt >= 3) throw err;
|
||||
const retryAfterSecs = Number(err?.headers?.['retry-after']);
|
||||
const baseMs = Number.isFinite(retryAfterSecs) && retryAfterSecs > 0
|
||||
? retryAfterSecs * 1000
|
||||
: 1000 * 4 ** attempt;
|
||||
const waitMs = baseMs + Math.random() * 500;
|
||||
progress.retryWaitsMs.push(Math.round(waitMs));
|
||||
await delay(waitMs, undefined, { signal }).catch(error => {
|
||||
signal?.throwIfAborted();
|
||||
throw error;
|
||||
});
|
||||
attempt += 1;
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
if (signal?.aborted) {
|
||||
console.error(JSON.stringify({
|
||||
type: 'llm-judge-aborted', model: resolvedModel, stream: opts?.stream === true, elapsedMs: elapsedMs(),
|
||||
attempts: attempt + 1, ...progress,
|
||||
reason: (signal.reason instanceof Error ? signal.reason.message : String(signal.reason)).slice(0, 200),
|
||||
}));
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
|
||||
opts?.onResponse?.({
|
||||
|
||||
@@ -85,6 +85,25 @@ describe('callJudge cancellation', () => {
|
||||
expect(create).toHaveBeenCalledTimes(1);
|
||||
}, 3000);
|
||||
|
||||
test('an abort during a 429 wait retains the wait it interrupted', async () => {
|
||||
const diagnostics = spyOn(console, 'error').mockImplementation(() => {});
|
||||
try {
|
||||
const controller = new AbortController();
|
||||
create.mockRejectedValue({ status: 429, headers: { 'retry-after': '30' } });
|
||||
const pending = callJudge('Rate-limited request.', 'test-judge-model', { signal: controller.signal });
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
controller.abort(new Error('Expired during rate-limit backoff.'));
|
||||
await expect(pending).rejects.toThrow('Expired during rate-limit backoff.');
|
||||
expect(diagnostics).toHaveBeenCalledTimes(1);
|
||||
expect(JSON.parse(diagnostics.mock.calls[0]![0])).toMatchObject({ type: 'llm-judge-aborted', model: 'test-judge-model',
|
||||
stream: false, attempts: 1, httpRequests: 0, retryWaitsMs: [30000], firstEventMs: null, reason: 'Expired during rate-limit backoff.' });
|
||||
create.mockResolvedValue(response as any);
|
||||
diagnostics.mockClear();
|
||||
expect(await callJudge('Judge this.', undefined, { signal: new AbortController().signal })).toEqual({ passed: true });
|
||||
expect(diagnostics).not.toHaveBeenCalled();
|
||||
} finally { diagnostics.mockRestore(); }
|
||||
}, 3000);
|
||||
|
||||
test('keeps three 429 retries for callers without a signal', async () => {
|
||||
const rateLimit = { status: 429, headers: { 'retry-after': '0.001' } };
|
||||
create.mockRejectedValue(rateLimit);
|
||||
|
||||
@@ -103,3 +103,123 @@ test('streaming retains the caller abort signal instead of extending its deadlin
|
||||
expect(await rejected).toBe(failure);
|
||||
expect(transport).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
test('a judge aborted mid-stream retains what it had received, as counts without content', async () => {
|
||||
const encoder = new TextEncoder();
|
||||
const partial = [
|
||||
{ type: 'message_start', message: { id: 'msg_stalled', type: 'message', role: 'assistant', model: 'claude-fable-5-1',
|
||||
content: [], stop_reason: null, stop_sequence: null, usage: { input_tokens: 99023, output_tokens: 1 } } },
|
||||
{ type: 'content_block_start', index: 0, content_block: { type: 'thinking', thinking: '', signature: '' } },
|
||||
{ type: 'content_block_delta', index: 0, delta: { type: 'thinking_delta', thinking: 'PRIVATE_THINKING' } },
|
||||
];
|
||||
let delivered!: () => void;
|
||||
const received = new Promise<void>(resolve => { delivered = resolve; });
|
||||
transport.mockImplementation(async (_url: string, init: RequestInit) => {
|
||||
const chunks = partial.map(event => encoder.encode(`event: ${event.type}\ndata: ${JSON.stringify(event)}\n\n`));
|
||||
return new Response(new ReadableStream({
|
||||
start(stream) {
|
||||
init.signal!.addEventListener('abort', () => stream.error(init.signal!.reason), { once: true });
|
||||
},
|
||||
pull(stream) {
|
||||
const chunk = chunks.shift();
|
||||
if (chunk) stream.enqueue(chunk);
|
||||
else delivered();
|
||||
},
|
||||
}, { highWaterMark: 0 }), { headers: { 'content-type': 'text/event-stream', 'request-id': 'req_stalled' } });
|
||||
});
|
||||
const controller = new AbortController();
|
||||
const rejected = callJudge('score the whole bundle', 'claude-fable-5-1', {
|
||||
max_tokens: 65_536, stream: true, jsonSchema: WORKFLOW_JUDGE_RESPONSE_SCHEMA, signal: controller.signal,
|
||||
}).catch(error => error);
|
||||
await received;
|
||||
const failure = new Error('ship/SKILL.md workflow exceeded its workflow judge work deadline');
|
||||
controller.abort(failure);
|
||||
expect(await rejected).toBe(failure);
|
||||
expect(diagnostics).toHaveBeenCalledTimes(1);
|
||||
const diagnostic = diagnostics.mock.calls[0]![0];
|
||||
expect(diagnostic).not.toContain('PRIVATE_');
|
||||
const event = JSON.parse(diagnostic);
|
||||
expect(event).toMatchObject({ type: 'llm-judge-aborted', model: 'claude-fable-5-1', stream: true, attempts: 1,
|
||||
httpRequests: 1, httpStatuses: [200], retryWaitsMs: [], block: 'thinking', thinkingChars: 'PRIVATE_THINKING'.length, textChars: 0,
|
||||
messageId: 'msg_stalled', requestId: 'req_stalled', reason: failure.message });
|
||||
expect(event.firstEventMs).toBeGreaterThanOrEqual(0);
|
||||
expect(event.lastEventMs).toBeGreaterThanOrEqual(event.firstEventMs); // gstack-timing-ok: ordering of two recorded offsets, not an elapsed-time bound
|
||||
expect(event.elapsedMs).toBeGreaterThanOrEqual(event.lastEventMs); // gstack-timing-ok: ordering of two recorded offsets, not an elapsed-time bound
|
||||
});
|
||||
|
||||
test('a stream that never sent an event is retained as such, and provider errors without an abort are not', async () => {
|
||||
let sent!: () => void;
|
||||
const requested = new Promise<void>(resolve => { sent = resolve; });
|
||||
transport.mockImplementation((_url: string, init: RequestInit) => new Promise((_resolve, reject) => {
|
||||
init.signal!.addEventListener('abort', () => reject(init.signal!.reason), { once: true });
|
||||
sent();
|
||||
}));
|
||||
const controller = new AbortController();
|
||||
const rejected = callJudge('score', 'claude-fable-5-1', { max_tokens: 65_536, stream: true, signal: controller.signal }).catch(error => error);
|
||||
await requested;
|
||||
controller.abort(new Error('deadline'));
|
||||
await rejected;
|
||||
expect(JSON.parse(diagnostics.mock.calls[0]![0])).toMatchObject({ type: 'llm-judge-aborted', httpRequests: 1,
|
||||
httpStatuses: [], connectedMs: null, firstEventMs: null, lastEventMs: null, block: null, thinkingChars: 0, messageId: null, requestId: null, reason: 'deadline' });
|
||||
|
||||
diagnostics.mockClear();
|
||||
transport.mockResolvedValue(new Response(JSON.stringify({ type: 'error', error: { type: 'api_error', message: 'boom' } }),
|
||||
{ status: 500, headers: { 'content-type': 'application/json', 'x-should-retry': 'false' } }));
|
||||
await expect(callJudge('score', 'claude-fable-5-1', { max_tokens: 65_536, stream: true, signal: new AbortController().signal })).rejects.toThrow();
|
||||
expect(diagnostics).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('a deadline after headers but before any event retains the connection and its request id', async () => {
|
||||
let reading!: () => void;
|
||||
const connected = new Promise<void>(resolve => { reading = resolve; });
|
||||
transport.mockImplementation(async (_url: string, init: RequestInit) => new Response(new ReadableStream({
|
||||
start(stream) { init.signal!.addEventListener('abort', () => stream.error(init.signal!.reason), { once: true }); },
|
||||
pull() { reading(); return new Promise<void>(() => {}); },
|
||||
}, { highWaterMark: 0 }), { headers: { 'content-type': 'text/event-stream', 'request-id': 'req_quiet' } }));
|
||||
const controller = new AbortController();
|
||||
const rejected = callJudge('score', 'claude-fable-5-1', { max_tokens: 65_536, stream: true, signal: controller.signal }).catch(error => error);
|
||||
await connected;
|
||||
controller.abort(new Error('deadline'));
|
||||
await rejected;
|
||||
const event = JSON.parse(diagnostics.mock.calls[0]![0]);
|
||||
expect(event).toMatchObject({ httpRequests: 1, httpStatuses: [200], requestId: 'req_quiet', firstEventMs: null, block: null });
|
||||
expect(event.connectedMs).toBeGreaterThanOrEqual(0);
|
||||
});
|
||||
|
||||
test('SDK-internal 429 retries are counted, so throttling never reads as a silent stream', async () => {
|
||||
let resent!: () => void;
|
||||
const retried = new Promise<void>(resolve => { resent = resolve; });
|
||||
transport.mockResolvedValueOnce(new Response(JSON.stringify({ type: 'error', error: { type: 'rate_limit_error', message: 'slow down' } }),
|
||||
{ status: 429, headers: { 'content-type': 'application/json', 'retry-after-ms': '1' } }));
|
||||
transport.mockImplementationOnce((_url: string, init: RequestInit) => new Promise((_resolve, reject) => {
|
||||
init.signal!.addEventListener('abort', () => reject(init.signal!.reason), { once: true });
|
||||
resent();
|
||||
}));
|
||||
const controller = new AbortController();
|
||||
const rejected = callJudge('score', 'claude-fable-5-1', { max_tokens: 65_536, stream: true, signal: controller.signal }).catch(error => error);
|
||||
await retried;
|
||||
controller.abort(new Error('deadline'));
|
||||
await rejected;
|
||||
expect(JSON.parse(diagnostics.mock.calls[0]![0])).toMatchObject({ attempts: 1, httpRequests: 2, httpStatuses: [429],
|
||||
retryWaitsMs: [], connectedMs: null, firstEventMs: null });
|
||||
});
|
||||
|
||||
test('an event the diagnostic cannot read never changes the judge result', async () => {
|
||||
const unreadable = response();
|
||||
const body = (await unreadable.text()).replace('"thinking":"PRIVATE_THINKING"', '"thinking":null');
|
||||
transport.mockResolvedValue(new Response(body, { headers: { 'content-type': 'text/event-stream' } }));
|
||||
expect(await callJudge('score the whole bundle', 'claude-fable-5-1', {
|
||||
max_tokens: 65_536, stream: true, jsonSchema: WORKFLOW_JUDGE_RESPONSE_SCHEMA,
|
||||
})).toEqual(scores);
|
||||
});
|
||||
|
||||
test('an abort reason is retained at most 200 characters long', async () => {
|
||||
transport.mockImplementation((_url: string, init: RequestInit) => new Promise((_resolve, reject) => {
|
||||
init.signal!.addEventListener('abort', () => reject(init.signal!.reason), { once: true });
|
||||
}));
|
||||
const controller = new AbortController();
|
||||
const rejected = callJudge('score', 'claude-fable-5-1', { max_tokens: 65_536, stream: true, signal: controller.signal }).catch(error => error);
|
||||
controller.abort(new Error('x'.repeat(500)));
|
||||
await rejected;
|
||||
expect(JSON.parse(diagnostics.mock.calls[0]![0]).reason).toBe('x'.repeat(200));
|
||||
});
|
||||
@@ -684,3 +684,18 @@ test('review suppressions allow a clean result and skip common false positives b
|
||||
expectTokens(suppressions, ['"Function too long"', '"Possible null dereference"', '"N+1 query"', '"Hardcoded value"', '`Math.random()`'], 'suppressions');
|
||||
expectAbsent(suppressions, [/missing input validation/i, /missing await/i], 'suppressions');
|
||||
});
|
||||
|
||||
test('judged repeat-pass QA reruns and unresolved record actions each have one rule (paid screen 40617200857)', () => {
|
||||
// Every sample of the red panel and its rerun named one of these: "post-expiry smoke rechecks are not-run" vs
|
||||
// Step 5.8's rerun of superseded probes, materialize's once-per-PROBE_DIR limit vs reruns into the same report,
|
||||
// and no record action for an unresolved non-gate defect (an unanswered question or an unproven repair).
|
||||
// fails_when=the rerun stops being revalidation, materialize may run per pass, or such a defect has no action.
|
||||
const flat = skill.replace(/\s+/g, ' ');
|
||||
const repeat = flat.slice(flat.indexOf('2. On a repeat, execute Steps 3–5 in order.'), flat.indexOf('3. **Verify completed actions.**'));
|
||||
expect(repeat).toContain('These reruns are revalidation, so they run even after smoke expiry; keep the PROBE_DIR. Earlier provisional sections link checkpoints; only the final pass runs its once-only `materialize`.');
|
||||
expect(repeat.indexOf('rerun each probe the verdict relies on')).toBeLessThan(repeat.indexOf('These reruns are revalidation'));
|
||||
const actions = flat.slice(flat.indexOf('Actions: `auto-fixed` (Step 5b)'), flat.indexOf('Advice is never `auto-fixed`'));
|
||||
expect(actions).toContain('`fixed` (approved **and completed** in Step 5d)');
|
||||
expect(actions).toContain('`skipped` (explicit Skip in Step 5c), `open` (any other defect, e.g. unanswered).');
|
||||
expect(flat).toContain('Do not record an unanswered question as skipped');
|
||||
});
|
||||
@@ -410,7 +410,7 @@ test('Ship sends its authorized 64k cap and compact response contract through th
|
||||
expect(WORKFLOW_JUDGE_RESPONSE_SCHEMA.properties.reasoning).not.toHaveProperty('maxLength');
|
||||
expect(source.match(/compactReasoning: true/g)).toHaveLength(1);
|
||||
const f = fixture();
|
||||
const stream = spyOn(Messages.prototype, 'stream').mockReturnValue({ finalMessage: async () => ({
|
||||
const stream = spyOn(Messages.prototype, 'stream').mockReturnValue({ on() { return this; }, finalMessage: async () => ({
|
||||
stop_reason: 'end_turn', content: [{ type: 'text', text: JSON.stringify(scores) }],
|
||||
}) } as any);
|
||||
try {
|
||||
@@ -516,7 +516,7 @@ test('the structured callback rejects incomplete, schema-invalid and below-thres
|
||||
try {
|
||||
for (const response of responses) {
|
||||
const f = fixture();
|
||||
stream.mockReturnValue({ finalMessage: async () => ({ stop_reason: response.stop_reason,
|
||||
stream.mockReturnValue({ on() { return this; }, finalMessage: async () => ({ stop_reason: response.stop_reason,
|
||||
content: [{ type: 'text', text: JSON.stringify(response.value) }] }) } as any);
|
||||
const h = actualCallback(f, { judge: (prompt, model, request) => callJudge<typeof scores>(prompt, model, request) });
|
||||
await expect(h.run({ ...h.options, schemaTransport: true, compactReasoning: true, maxTokens: 65_536, stream: true })).rejects.toThrow();
|
||||
|
||||
Reference in new issue
Block a user