mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-18 02:42:25 +02:00
fix: red-team review fixes
Five informational findings: TODOS shard-census arithmetic corrected (census is 67 with one free ungated slot; the SECOND ungated file trips the floor) and version pointer fixed (v0.18.2.0, not v0.18.1.0); the free tripwire now pins the two dispatch-matcher marker strings so a pr-body prompt reword fails the free suite instead of surfacing as a paid-tier mystery; the E2E matcher gains a section-paste exclusion (scaffold strings disqualify) — verified against all recorded runs; the E2E header documents the tierless test:evals invisibility tradeoff. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
14ee421fb1
commit
595f6e60bd
@@ -2329,7 +2329,7 @@ Shipped as v0.5.0 on main. Includes `/plan-design-review` (report-only design au
|
|||||||
|
|
||||||
### Auto-invoke /document-release from /ship — SHIPPED
|
### Auto-invoke /document-release from /ship — SHIPPED
|
||||||
|
|
||||||
Shipped in v0.8.4; redesigned twice since. Current design (v0.18.1.0+, carved in
|
Shipped in v0.8.4; redesigned twice since. Current design (v0.18.2.0+, carved in
|
||||||
v1.54.0.0): `/ship` Step 18 (`ship/sections/pr-body.md`) dispatches
|
v1.54.0.0): `/ship` Step 18 (`ship/sections/pr-body.md`) dispatches
|
||||||
`/document-release` as a general-purpose subagent AFTER Step 17 (push) and
|
`/document-release` as a general-purpose subagent AFTER Step 17 (push) and
|
||||||
BEFORE Step 19 (PR creation); the subagent's JSON contract (`files_updated`,
|
BEFORE Step 19 (PR creation); the subagent's JSON contract (`files_updated`,
|
||||||
@@ -2380,16 +2380,18 @@ pin) and `test/skill-e2e-ship-docsync.test.ts` (dispatch E2E, gate tier).
|
|||||||
|
|
||||||
### Periodic paid-test shard census is one ungated file from the detach-timeout floor
|
### Periodic paid-test shard census is one ungated file from the detach-timeout floor
|
||||||
|
|
||||||
**What:** The periodic tier's shard census sits at exactly the 17×4 ceiling
|
**What:** The periodic tier's shard census is 67 files — one ungated slot below
|
||||||
(68-file boundary). The next paid `skill-e2e-*` file WITHOUT a whole-file
|
the 68-file (17×4) ceiling. The next paid `skill-e2e-*` file WITHOUT a
|
||||||
`describeE2ETier` self-gate counts in BOTH tier censuses, trips periodic to 18
|
whole-file `describeE2ETier` self-gate lands at 68 (still 17 waves, floor
|
||||||
shards → 34,020s floor > the 32,400s configured detach timeout, and
|
32,130s ≤ 32,400s — passes); the SECOND ungated file trips 18 waves → 34,020s
|
||||||
|
floor > the 32,400s configured detach timeout, and
|
||||||
`test/eval-detach-timeout-floor.test.ts` fails with a confusing message.
|
`test/eval-detach-timeout-floor.test.ts` fails with a confusing message.
|
||||||
|
|
||||||
**Why:** Whoever adds the next periodic E2E gets a floor failure unrelated to
|
**Why:** Whoever adds the second ungated periodic E2E gets a floor failure
|
||||||
their change. Fix options: raise the periodic detach timeout, or enforce
|
unrelated to their change. Fix options: raise the periodic detach timeout, or
|
||||||
whole-file tier self-gates on all paid files (upgrades them from the
|
enforce whole-file tier self-gates on all paid files (upgrades them from the
|
||||||
tier-alignment warn-only bucket to the hard invariant at the same time).
|
tier-alignment warn-only bucket to the hard invariant, and — bonus — restores
|
||||||
|
tierless `bun run test:evals` coverage decisions to diff selection alone).
|
||||||
|
|
||||||
**Context:** `scripts/test-paid-shards.ts` `classifyPaidTestFile` counts
|
**Context:** `scripts/test-paid-shards.ts` `classifyPaidTestFile` counts
|
||||||
ungated files in both tiers; `ship-docsync` composed `describeE2ETier('gate')`
|
ungated files in both tiers; `ship-docsync` composed `describeE2ETier('gate')`
|
||||||
|
|||||||
@@ -59,6 +59,12 @@ describe('/ship Step 18 dispatches /document-release (carve visibility)', () =>
|
|||||||
expect(content).toContain('"documentation_section"');
|
expect(content).toContain('"documentation_section"');
|
||||||
// Deliberate design: docs sync never holds a ship hostage.
|
// Deliberate design: docs sync never holds a ship hostage.
|
||||||
expect(content).toContain('Do not block /ship on subagent failure');
|
expect(content).toContain('Do not block /ship on subagent failure');
|
||||||
|
// These two strings are the ship-docsync E2E's dispatch-matcher markers
|
||||||
|
// (test/skill-e2e-ship-docsync.test.ts) — they come verbatim from the
|
||||||
|
// dictated Step 18 subagent prompt. Rewording them in pr-body.md.tmpl
|
||||||
|
// silently decouples the paid matcher; update both in lockstep.
|
||||||
|
expect(content).toContain('You are executing the /document-release workflow');
|
||||||
|
expect(content).toContain('.claude/skills/gstack/document-release/SKILL.md');
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
@@ -8,8 +8,11 @@
|
|||||||
*
|
*
|
||||||
* Gating: whole-file gate-tier self-gate (describeE2ETier) COMPOSED with
|
* Gating: whole-file gate-tier self-gate (describeE2ETier) COMPOSED with
|
||||||
* diff-based selection (describeIfSelected). The self-gate keeps this file
|
* diff-based selection (describeIfSelected). The self-gate keeps this file
|
||||||
* out of the periodic shard census (which sits at its ceiling) and under
|
* out of the periodic shard census (near its ceiling) and under the hard
|
||||||
* the hard tier-alignment invariant. Run locally with:
|
* tier-alignment invariant. DELIBERATE TRADEOFF: tierless runs (`bun run
|
||||||
|
* test:evals` / `test:e2e`) skip every tier-gated file, so this test does
|
||||||
|
* NOT run there even when ship/** changed — use the gate lane locally:
|
||||||
|
* EVALS_TIER=gate bun run test:evals # diff-selected gate lane
|
||||||
* EVALS=1 EVALS_TIER=gate EVALS_ALL=1 bun test test/skill-e2e-ship-docsync.test.ts
|
* EVALS=1 EVALS_TIER=gate EVALS_ALL=1 bun test test/skill-e2e-ship-docsync.test.ts
|
||||||
*
|
*
|
||||||
* Fixture layout (non-obvious — fake HOME + planted skill tree):
|
* Fixture layout (non-obvious — fake HOME + planted skill tree):
|
||||||
@@ -208,13 +211,19 @@ describeE2E('Ship doc-sync dispatch E2E (gate)', () => {
|
|||||||
// (e.g. a PR-body drafter) must NOT count — that false-pass would mask
|
// (e.g. a PR-body drafter) must NOT count — that false-pass would mask
|
||||||
// the exact regression this test exists to catch. Verified against
|
// the exact regression this test exists to catch. Verified against
|
||||||
// recorded burn-in transcripts: real dispatch inputs carry both markers.
|
// recorded burn-in transcripts: real dispatch inputs carry both markers.
|
||||||
const dispatchIdx = calls.findIndex(
|
// Section-paste exclusion: a subagent handed the WHOLE pr-body.md as
|
||||||
(tc) =>
|
// context carries the markers too. The dictated Step 18 prompt never
|
||||||
(tc.tool === 'Agent' || tc.tool === 'Task') &&
|
// contains the section's scaffolding, so its presence disqualifies.
|
||||||
/document-release\/SKILL\.md|executing the \/document-release workflow/i.test(
|
// Verified across all recorded runs: real dispatches match markers,
|
||||||
JSON.stringify(tc.input ?? {})
|
// zero contain scaffold strings.
|
||||||
)
|
const dispatchIdx = calls.findIndex((tc) => {
|
||||||
);
|
if (tc.tool !== 'Agent' && tc.tool !== 'Task') return false;
|
||||||
|
const input = JSON.stringify(tc.input ?? {});
|
||||||
|
return (
|
||||||
|
/document-release\/SKILL\.md|executing the \/document-release workflow/i.test(input) &&
|
||||||
|
!/## Step 19: Create PR\/MR|Parent processing:/.test(input)
|
||||||
|
);
|
||||||
|
});
|
||||||
const prCreateIdx = calls.findIndex(
|
const prCreateIdx = calls.findIndex(
|
||||||
(tc) =>
|
(tc) =>
|
||||||
tc.tool === 'Bash' &&
|
tc.tool === 'Bash' &&
|
||||||
|
|||||||
Reference in New Issue
Block a user