mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-16 13:21:30 +01:00
feat: retire /investigate after a no-gain eval (GST-1717) (#4110)
* feat: retire /investigate after a no-gain eval (GST-1717)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(skill-start): core routing offer drops every non-core rule line (GST-1717)
The retired /investigate rule became 'Bugs/errors → fix them directly', which no
longer matches '→ invoke /', so proactive=core leaked it into the offered section.
Match any '- … → ' rule line.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* revert(review): keep review/checklist.md at its no-loss-pinned bytes (GST-1717)
The pin names replay GSTA-224 H1; a reviewer checklist edit merges only after a
passing no-loss replay (crew rule 15). The /investigate flake line stays until a
replayed repin.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test: repair free-suite fallout from retiring /investigate (GST-1717)
CI run 42340956269 at 83a8a9a: drop fixtures, ledger rows, allowlist entries and
counts for the retired skill and its deleted paid files; record the harness
change as non-behavioral; the test-of-test detector accepts path.join(ROOT, 'bin');
uncovered-prompt fixtures use a tracked prompt, since an untracked path now
reads as a deleted one.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(relink): back up a retired skill's real SKILL.md before cleaning it (GST-1717)
With the skill's source gone, the customized-file check never ran, so a
banner-carrying local copy of investigate/SKILL.md was deleted instead of moved
to the backup root (red-team review, pass 1).
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(qm): restore the review checklist's flake-line rewrite (GST-1717)
review/checklist.md keeps its pinned /investigate line, so the QM edition needs
its replaceText rule back; qm/ stays fresh against the generator.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs(changelog): name the setup flag break, the relink backup and the pinned checklist line (GST-1717)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(setup,relink): back up a retired skill's real SKILL.md instead of deleting it (GST-1717)
Scoped fix round under grant a-360dd62bd46a (outside-voice blocking finding and
adversarial #1, #4). setup's prune compares a copied host entry with the render
before removing that render, and backs up a copy that differs or whose render is
gone; relink no longer takes the marker's rm -rf for a skill whose source is
gone, so a real SKILL.md is backed up and the user's files stay. setup reports a
failed remove-source of the retired gate hooks on stderr.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(pr-profile): a changed prompt counts as deleted only when it is gone from disk (GST-1717)
An untracked prompt file on disk still needs prompt validation (targeted review
finding); harness change recorded as non-behavioral.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs(changelog): exact backup wording, the hook-removal warning and GST-1744 (GST-1717)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(setup): one backup folder per host for a retired skill's copies (GST-1717)
Under grant a-c95576b8b1f9 (fix-check#1): every copy host moved its
gstack-investigate/SKILL.md to the same backup path, so a later host's copy
overwrote an earlier host's edits. A sandboxed ./setup --host factory run showed
gen-skill-docs removes the retired render before the prune, so every real copy
is backed up; each host now gets its own folder.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs(changelog): say exactly what setup backs up for a retired skill (GST-1717)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs(changelog): end the two colon sentences prose-lint flagged (GST-1717)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* test(retired-skills): name plan-eng-review once behind a pregate-literal waiver (GST-1717)
pregate's literals check read the GSTACK_RETIRED_SKILLS change as an old
value still in the tests. plan-eng-review is still retired; investigate was
appended. The fixture names it once, with the waiver.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
---------
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
5862765a20
commit
ef99eb5a55
168 files changed
+601
-5245
No files matched your search
@@ -10,14 +10,15 @@ import type { HostConfig } from './host-config';
|
||||
const SKIP = new Set(['node_modules', '.git', 'dist']);
|
||||
|
||||
/**
|
||||
* Retired skills: the source stays because another skill reads it, but no host
|
||||
* registers it. plan-eng-review is /autoplan's eng-phase methodology (GST-1698).
|
||||
* External hosts get that methodology beside gstack-autoplan/SKILL.md as
|
||||
* <name>.md, since Codex and Copilot register every nested SKILL.md. setup and
|
||||
* gstack-relink read the same list from GSTACK_RETIRED_SKILLS in
|
||||
* bin/gstack-install-registry.sh.
|
||||
* Retired skills: no host registers them, and setup removes an earlier
|
||||
* install's gstack-owned entry. plan-eng-review's source stays as /autoplan's
|
||||
* eng-phase methodology (GST-1698); external hosts get it beside
|
||||
* gstack-autoplan/SKILL.md as <name>.md, since Codex and Copilot register every
|
||||
* nested SKILL.md. investigate was cut after a no-gain eval and its source is
|
||||
* gone (GST-1717). setup and gstack-relink read the same list from
|
||||
* GSTACK_RETIRED_SKILLS in bin/gstack-install-registry.sh.
|
||||
*/
|
||||
export const RETIRED_SKILLS = ['plan-eng-review'];
|
||||
export const RETIRED_SKILLS = ['plan-eng-review', 'investigate'];
|
||||
|
||||
/** The generator and its coverage checks use the same include-minus-skip rule. */
|
||||
export function includesSkill(host: HostConfig, skillDir: string): boolean {
|
||||
|
||||
@@ -475,8 +475,6 @@
|
||||
"test/hook-scripts.test.ts": 252101,
|
||||
"test/host-renders-openai-yaml-proof.test.ts": 141,
|
||||
"test/instruction-facts.test.ts": 168,
|
||||
"test/investigate-freeze-path.test.ts": 29609,
|
||||
"test/investigate-recipes.test.ts": 74,
|
||||
"test/ios-debug-bridge-release-guard.test.ts": 75,
|
||||
"test/ios-qa-daemon-cli.test.ts": 196,
|
||||
"test/ios-qa-stateserver-hardening.test.ts": 66,
|
||||
@@ -768,7 +766,6 @@
|
||||
"test/skill-preflight-budget.test.ts": 71,
|
||||
"test/skill-triggers.test.ts": 75,
|
||||
"test/skillify-host-paths.test.ts": 4396,
|
||||
"test/sol-skill-fixture.test.ts": 5142,
|
||||
"test/spawnsync-timeout-tripwire.test.ts": 644,
|
||||
"test/spec-template-invariants.test.ts": 83,
|
||||
"test/spend-ledger.test.ts": 345,
|
||||
@@ -813,7 +810,6 @@
|
||||
"test/windows-free-selection-autoplan-guard.test.ts": 82,
|
||||
"test/windows-native-workflows.test.ts": 696,
|
||||
"test/workflow-action-pins.test.ts": 84,
|
||||
"test/workflow-boundaries-fixture.test.ts": 25430,
|
||||
"test/workflow-concurrency.test.ts": 72,
|
||||
"test/workflow-excerpt.test.ts": 328,
|
||||
"test/workflow-gh-token.test.ts": 73,
|
||||
|
||||
@@ -826,8 +826,6 @@
|
||||
"test/install-fixture.test.ts": 796,
|
||||
"test/install-verdict.test.ts": 1471,
|
||||
"test/instruction-facts.test.ts": 67,
|
||||
"test/investigate-freeze-path.test.ts": 4902,
|
||||
"test/investigate-recipes.test.ts": 22,
|
||||
"test/ios-debug-bridge-release-guard.test.ts": 11,
|
||||
"test/ios-qa-daemon-cli.test.ts": 18,
|
||||
"test/ios-qa-regen.test.ts": 1730,
|
||||
@@ -1364,7 +1362,6 @@
|
||||
"test/skill-validation.test.ts": 953,
|
||||
"test/skillify-host-paths.test.ts": 1220,
|
||||
"test/slop-diff-cli.test.ts": 285,
|
||||
"test/sol-skill-fixture.test.ts": 811,
|
||||
"test/spawned-consent-rule.test.ts": 22,
|
||||
"test/spawnsync-timeout-tripwire.test.ts": 211,
|
||||
"test/spec-quality-gate-secret-sink.test.ts": 1884,
|
||||
@@ -1459,7 +1456,6 @@
|
||||
"test/windows-free-selection-autoplan-guard.test.ts": 22,
|
||||
"test/windows-native-workflows.test.ts": 233,
|
||||
"test/workflow-action-pins.test.ts": 29,
|
||||
"test/workflow-boundaries-fixture.test.ts": 3119,
|
||||
"test/workflow-concurrency.test.ts": 17,
|
||||
"test/workflow-excerpt.test.ts": 88,
|
||||
"test/workflow-gh-token.test.ts": 22,
|
||||
|
||||
@@ -1 +1 @@
|
||||
{"blob":"c984ba618be367f2201362a9592da7a8631accb5","since":6,"nonBehavioral":"Retired standalone /plan-eng-review cases (GST-1698): removed their budgets and paid-case rows; remaining files' walls shrink with their case counts, and no remaining case's budget or verdict changes"}
|
||||
{"blob":"b88614aba91757dc9cd096e2333533de3bd57efe","since":6,"nonBehavioral":"Merge of main (GST-1698 plan-eng-review case retirement) with GST-1717: paid-cases.ts drops both retired skills' files and test-pr-profile.ts stops requiring full validation for a deleted prompt; no remaining case's prompt, runner or grading changes"}
|
||||
@@ -1 +1 @@
|
||||
{"blob":"812d84b94328409328020a842696e1a34b22b095","since":6,"nonBehavioral":"narrow screens derive setup and loadouts.json is derived; case selection only, no verdict can change (GST-1575)"}
|
||||
{"blob":"507e219536d4cbd4e28b0d76a5f9ee471e0a2176","since":6,"nonBehavioral":"GST-1717 scoped fix: a changed prompt file counts as deleted only when it is absent from disk, so an untracked prompt on disk still needs full validation (stricter selection); no case's prompt, runner or grading changes, so no verdict can change"}
|
||||
@@ -271,7 +271,6 @@ export const CODEX_CI_ENV = { binDir: 'GSTACK_CI_CODEX_BIN_DIR', home: 'GSTACK_C
|
||||
/** Paid files whose every case exercises the real Codex CLI. */
|
||||
export const CODEX_CI_FILES: readonly string[] = [
|
||||
'test/codex-e2e.test.ts',
|
||||
'test/codex-e2e-sol-scope.test.ts',
|
||||
'test/codex-e2e-shared-libs.test.ts',
|
||||
'test/codex-e2e-recommendation-substance.test.ts',
|
||||
'test/skill-e2e-outside-voice.test.ts',
|
||||
|
||||
@@ -25,8 +25,6 @@
|
||||
"test/skill-e2e-docsync-spawned.test.ts": 108078,
|
||||
"test/skill-e2e-gbrain-roundtrip-local.test.ts": 1000,
|
||||
"test/skill-e2e-hermetic-canary.test.ts": 4951,
|
||||
"test/skill-e2e-investigate-owned-completion.test.ts": 36179,
|
||||
"test/skill-e2e-investigate-owned-termination.test.ts": 48428,
|
||||
"test/skill-e2e-ios-device.test.ts": 1000,
|
||||
"test/skill-e2e-learnings.test.ts": 33590,
|
||||
"test/skill-e2e-office-hours-auto-mode.test.ts": 78399,
|
||||
@@ -124,7 +122,6 @@
|
||||
"test/codex-e2e-multiblock-live.test.ts": 180000,
|
||||
"test/codex-e2e-recommendation-substance.test.ts": 14617,
|
||||
"test/codex-e2e-shared-libs.test.ts": 49563,
|
||||
"test/codex-e2e-sol-scope.test.ts": 32228,
|
||||
"test/codex-e2e.test.ts": 110677,
|
||||
"test/llm-judge-recommendation.test.ts": 21562,
|
||||
"test/skill-e2e-arm-benchmark.test.ts": 72714,
|
||||
|
||||
@@ -199,7 +199,7 @@ export const SIGNAL_MAP: Record<string, Record<string, DimensionDelta[]>> = {
|
||||
// common option keys for "I'll review first" vs "go ahead":
|
||||
'review-first': [{ dim: 'autonomy', delta: -0.05 }],
|
||||
proceed: [{ dim: 'autonomy', delta: +0.05 }],
|
||||
// /investigate-style: "agent applies fix" vs "show me the diff first"
|
||||
// debugging-style: "agent applies fix" vs "show me the diff first"
|
||||
'apply-fix': [{ dim: 'autonomy', delta: +0.04 }],
|
||||
'show-diff': [{ dim: 'autonomy', delta: -0.04 }],
|
||||
},
|
||||
|
||||
@@ -17,7 +17,7 @@
|
||||
*
|
||||
* v1 coverage target: the ~30-50 most-common recurring question categories across
|
||||
* ship, review, office-hours, plan-ceo-review, plan-eng-review, plan-design-review,
|
||||
* plan-devex-review, qa, investigate, and land-and-deploy. One-way doors 100%.
|
||||
* plan-devex-review, qa, and land-and-deploy. One-way doors 100%.
|
||||
*
|
||||
* Adding a new entry
|
||||
* ------------------
|
||||
@@ -56,7 +56,6 @@ export type StandardOption =
|
||||
| 'reject'
|
||||
| 'defer'
|
||||
| 'skip'
|
||||
| 'investigate'
|
||||
| 'approve'
|
||||
| 'deny'
|
||||
| 'expand'
|
||||
@@ -442,26 +441,6 @@ export const QUESTIONS = {
|
||||
description: "QA tier: quick (critical/high only), standard (+medium), or deep (+low)?",
|
||||
},
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// /investigate — root-cause debugging
|
||||
// -----------------------------------------------------------------------
|
||||
'investigate-hypothesis-confirm': {
|
||||
id: 'investigate-hypothesis-confirm',
|
||||
skill: 'investigate',
|
||||
category: 'approval',
|
||||
door_type: 'two-way',
|
||||
options: ['accept', 'reject', 'refine'],
|
||||
description: "Root-cause hypothesis — accept, reject, or refine before proceeding to fix?",
|
||||
},
|
||||
'investigate-fix-apply': {
|
||||
id: 'investigate-fix-apply',
|
||||
skill: 'investigate',
|
||||
category: 'approval',
|
||||
door_type: 'one-way',
|
||||
options: ['accept', 'reject'],
|
||||
description: "Apply the proposed fix?",
|
||||
},
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// /land-and-deploy — merge + deploy + verify
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
@@ -41,7 +41,6 @@ interface SkillSaveMeta {
|
||||
|
||||
const skillSaveMap: Record<string, SkillSaveMeta> = {
|
||||
'office-hours': { slugPrefix: 'office-hours', title: 'Office Hours', tag: 'design-doc' },
|
||||
'investigate': { slugPrefix: 'investigations', title: 'Investigation', tag: 'investigation' },
|
||||
'plan-ceo-review': { slugPrefix: 'ceo-plans', title: 'CEO Plan', tag: 'ceo-plan' },
|
||||
'plan-eng-review': { slugPrefix: 'eng-reviews', title: 'Eng Review', tag: 'eng-review' },
|
||||
'plan-design-review': { slugPrefix: 'design-reviews', title: 'Design Review', tag: 'design-review' },
|
||||
@@ -52,8 +51,8 @@ const skillSaveMap: Record<string, SkillSaveMeta> = {
|
||||
'design-consultation': { slugPrefix: 'design-systems', title: 'Design System', tag: 'design-system' },
|
||||
};
|
||||
|
||||
export function generateGBrainContextLoad(ctx: TemplateContext): string {
|
||||
let base = `## Brain Context Load
|
||||
export function generateGBrainContextLoad(_ctx: TemplateContext): string {
|
||||
const base = `## Brain Context Load
|
||||
|
||||
**Skip this entire section if \`gbrain\` is not on PATH.**
|
||||
|
||||
@@ -65,10 +64,6 @@ If \`gbrain search\` returns no results or any non-zero exit, proceed
|
||||
without brain context. Full search/read protocol + examples:
|
||||
see \`docs/gbrain-write-surfaces.md\` §Context Load.`;
|
||||
|
||||
if (ctx.skillName === 'investigate') {
|
||||
base += `\n\nFor structured-data extraction requests ("track this", "extract from emails", "build a tracker"), route to GBrain's data-research skill instead: \`gbrain call data-research\`.`;
|
||||
}
|
||||
|
||||
return base;
|
||||
}
|
||||
|
||||
|
||||
@@ -18,6 +18,6 @@ If you are looping on the same diagnostic, same file, or failed fix variants, ST
|
||||
//
|
||||
// Skills by tier:
|
||||
// T1: browse, setup-cookies, benchmark
|
||||
// T2: investigate, cso, retro, doc-release, setup-deploy, canary, checkpoint, health
|
||||
// T2: cso, retro, doc-release, setup-deploy, canary, checkpoint, health
|
||||
// T3: autoplan, codex, design-consult, office-hours, ceo/design/eng-review
|
||||
// T4: ship, review, qa, qa-only, design-review, land-deploy
|
||||
@@ -2,10 +2,10 @@
|
||||
import { FREE_TEXT_WRITE_RULE, freeTextFileBash, freeTextFileUse } from '../free-text-file';
|
||||
|
||||
export function generateFailureRouting(): string {
|
||||
return `**Failures: check the cause, then route.** A failing test, error or stack trace shows where a failure surfaces, not its root cause.
|
||||
return `**Failures: check the cause, then fix it test-first.** A failing test, error or stack trace shows where a failure surfaces, not its root cause.
|
||||
|
||||
- **Cause supplied** (the user names a specific cause and a narrow fix): make one read-only check of the evidence it rests on, such as the named file, test or log. If that confirms the cause, make the fix and run the focused test that covers it. If it does not, invoke \`/investigate\` before any further tracing or edit.
|
||||
- **Cause unknown:** make one read-only triage pass over the supplied evidence and the directly implicated test and code. Do not reproduce, test hypotheses or search wider. If that pass establishes the root cause and a narrow fix, make the fix and run the focused test that covers it. If the cause, scope or intended behavior is still unclear, invoke \`/investigate\` before any broader diagnosis or edit.`;
|
||||
- **Cause supplied** (the user names a specific cause and a narrow fix): make one read-only check of the evidence it rests on, such as the named file, test or log. If that confirms the cause, make the fix and run the focused test that covers it. If it does not, leave the suggested fix alone: reproduce the failure with a focused failing test, trace it to the root cause, fix that and rerun the test.
|
||||
- **Cause unknown:** make one read-only triage pass over the supplied evidence and the directly implicated test and code. Do not reproduce, test hypotheses or search wider. If that pass establishes the root cause and a narrow fix, make the fix and run the focused test that covers it. If the cause or scope is still unclear, write or run a focused test that fails on the bug before any edit, trace the failure to its root cause, fix that and rerun the test. If the intended behavior is unclear, ask the user.`;
|
||||
}
|
||||
|
||||
export function generateTestFailureTriage(): string {
|
||||
@@ -55,7 +55,7 @@ Ask with AskUserQuestion in the AskUserQuestion Format. List each failure the sa
|
||||
### Step T4: Execute the chosen action
|
||||
|
||||
**If "Investigate and fix now":**
|
||||
- One read-only root-cause pass, then \`/investigate\` if unclear.
|
||||
- One read-only root-cause pass. If the cause is still unclear, reproduce it with a focused failing test and trace it to the root cause before the fix.
|
||||
- Fix the pre-existing failure.
|
||||
- Commit the fix separately from the branch's changes: \`git commit -m "fix: pre-existing test failure in <test-file>"\`
|
||||
- Continue with the workflow.
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
/**
|
||||
* Sideways review (multi-agent wave D2, shared with D3's flake recipe).
|
||||
* Sideways review (multi-agent wave D2).
|
||||
*
|
||||
* A finding is a defect class, not a line: the same pattern usually sits at
|
||||
* another ingress, command or call site, and the severity is the worst
|
||||
@@ -8,8 +8,6 @@
|
||||
* {{SIDEWAYS_SWEEP}} review Step 4: sweep + the forced-CRITICAL list
|
||||
* {{SIDEWAYS_SWEEP:cso}} /cso Phase 12: the sibling sweep only (D2: nothing else)
|
||||
* {{FLAKE_FIX_NUMBER:review}} a test-only flake fix needs P(fail | regression) before and after
|
||||
* {{FLAKE_FIX_NUMBER:investigate}}
|
||||
* the same number, as the flake recipe computes it
|
||||
*
|
||||
* `undeclaredBehaviorLines()` is the Scope Check addition the plan-completion
|
||||
* resolver (scripts/resolvers/plan-gates.ts, review mode) renders: diff
|
||||
@@ -33,15 +31,7 @@ export function generateSidewaysSweep(_ctx: TemplateContext, args?: string[]): s
|
||||
return `**Sideways sweep (every finding; reads code OUTSIDE the diff).** Grep the finding's pattern at every other ingress, command and call site, Read each match and rate severity on the worst sibling; a new enum value, status, tier or type constant is traced through every consumer the same way (checklist: Sideways sweep). ${forcedCriticalPhrase().replace(/^d/, 'D')} is forced CRITICAL: no lane, specialist, dedup or saved decision downgrades it. Keep findings anchored to changed code.`;
|
||||
}
|
||||
|
||||
export function generateFlakeFixNumber(_ctx: TemplateContext, args?: string[]): string {
|
||||
if (args?.[0] === 'investigate') {
|
||||
return `**The flake-fix number.** A fix that touches only the test (a wider timeout, a retry,
|
||||
a looser assertion, a skip) must state \`P(fail | regression)\`: the probability the test
|
||||
still fails when the behavior it guards regresses, before and after the change. Derive
|
||||
it from the forced probe (break the behavior on purpose, run the test N times, count
|
||||
failures) and write both numbers in \`## Flake evidence\`. A number that drops is a gate
|
||||
relaxation (review tag RH-15) and needs the owner's decision, not a merge.`;
|
||||
}
|
||||
export function generateFlakeFixNumber(_ctx: TemplateContext, _args?: string[]): string {
|
||||
return `A test-only flake fix (timeout, retry, looser assertion, skip) states \`P(fail | regression)\` before and after, from a forced probe; without both numbers it is RH-15.`;
|
||||
}
|
||||
|
||||
|
||||
@@ -6,8 +6,7 @@
|
||||
* stand-in for the Anthropic Messages API that plays a scripted session, so
|
||||
* no model is called and nothing is spent. Then it checks the GSTACK_GATE
|
||||
* markers in the session record, the hook log and the transcript JSONL.
|
||||
* Three scenarios share the server: the /investigate gate (Edit and sed -i
|
||||
* blocked until a root cause), the /plan-ceo-review gate (Stop blocked
|
||||
* Two scenarios share the server: the /plan-ceo-review gate (Stop blocked
|
||||
* until the output file ranks the top risks, written through Bash) and the
|
||||
* /autoplan gate (Stop blocked after Phase 1 until Eng publishes and the
|
||||
* Final Approval Gate is written).
|
||||
@@ -29,8 +28,6 @@ const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const args = process.argv.slice(2);
|
||||
const keep = args.includes('--keep');
|
||||
const work = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-gates-smoke-')));
|
||||
const SRC = 'export const total = (a: number, b: number) => a - b;\n';
|
||||
const FIXED = 'export const total = (a: number, b: number) => a + b;\n';
|
||||
const PLAN = '# Plan: one-click checkout\n\nShip one-click checkout behind a flag.\n';
|
||||
const AP_PLAN = '# Plan: retry the fetch helper\n\n## Implementation plan\n\nAdd one retry with backoff to src/fetch.ts.\n\n## Review record\n\nNone yet.\n';
|
||||
const FINDINGS = '\n## Findings\n\n1. Payment retries can double-charge when the provider times out.\n2. No rollback path for the new orders schema.\n';
|
||||
@@ -56,36 +53,6 @@ interface Seen { code: number; transcript: string; record: Record<string, any> |
|
||||
interface Scenario { name: string; hook: string; flag: string; skill: string; prompt: string; seed: (repo: string) => void; script: (repo: string) => Block[][]; checks: (s: Seen) => Array<[string, boolean]> }
|
||||
|
||||
const SCENARIOS: Scenario[] = [
|
||||
{
|
||||
name: 'investigate', hook: 'investigate-gate', flag: '--investigate', skill: 'investigate', prompt: 'total() returns the wrong value; investigate and fix it',
|
||||
seed: repo => { fs.mkdirSync(path.join(repo, 'src'), { recursive: true }); fs.writeFileSync(path.join(repo, 'src/total.ts'), SRC); },
|
||||
script: repo => [
|
||||
[{ type: 'text', text: 'Loading /investigate.' }, { type: 'tool_use', id: 'toolu_s1', name: 'Skill', input: { skill: 'investigate' } }],
|
||||
[{ type: 'tool_use', id: 'toolu_r1', name: 'Read', input: { file_path: path.join(repo, 'src/total.ts') } }],
|
||||
[{ type: 'tool_use', id: 'toolu_e1', name: 'Edit', input: { file_path: path.join(repo, 'src/total.ts'), old_string: 'a - b', new_string: 'a + b' } }],
|
||||
[{ type: 'tool_use', id: 'toolu_b1', name: 'Bash', input: { command: "sed -i 's/a - b/a + b/' src/total.ts", description: 'Patch through the shell' } }],
|
||||
[{ type: 'text', text: 'Root cause: total() subtracts b instead of adding it (src/total.ts:1).' }, { type: 'tool_use', id: 'toolu_r2', name: 'Read', input: { file_path: path.join(repo, 'src/total.ts') } }],
|
||||
[{ type: 'tool_use', id: 'toolu_e2', name: 'Edit', input: { file_path: path.join(repo, 'src/total.ts'), old_string: 'a - b', new_string: 'a + b' } }],
|
||||
[{ type: 'text', text: 'Fixed total().' }],
|
||||
],
|
||||
checks: s => {
|
||||
const run0 = s.record ? Object.values(s.record.hooks['investigate-gate']?.runs ?? {})[0] as Record<string, unknown> | undefined : undefined;
|
||||
return [
|
||||
['claude -p exited 0', s.code === 0],
|
||||
['record: one run, fired', !!run0 && typeof run0.fired_at === 'string'],
|
||||
['record: blocked twice (Edit, then Bash sed -i)', run0?.blocked === 2],
|
||||
['record: released after the Root cause line', typeof run0?.released_at === 'string'],
|
||||
['record: no hook errors', (s.record?.hooks['investigate-gate']?.errors ?? []).length === 0],
|
||||
['log: fired, blocked via=edit, blocked via=bash, released', /event=fired/.test(s.log) && /event=blocked .*via=edit/.test(s.log) && /event=blocked .*via=bash/.test(s.log) && /event=released/.test(s.log)],
|
||||
['transcript: deny reason carries the blocked via=edit marker (tool_result)', /"tool_result"[^\n]*GSTACK_GATE investigate-gate blocked [^"]*via=edit/.test(s.transcript)],
|
||||
['transcript: PostToolUse block carries the blocked via=bash marker (hook_blocking_error)', /hook_blocking_error[^\n]*GSTACK_GATE investigate-gate blocked [^"]*via=bash/.test(s.transcript)],
|
||||
['transcript: fired and released ride hook_system_message', /hook_system_message[^\n]*GSTACK_GATE investigate-gate fired/.test(s.transcript) && /hook_system_message[^\n]*GSTACK_GATE investigate-gate released/.test(s.transcript)],
|
||||
['requests: fired/released systemMessage markers never reach the model', !/investigate-gate (fired|released)/.test(s.requests)],
|
||||
['requests: the model saw both block reasons', /investigate-gate blocked [^"]*via=edit/.test(s.requests) && /investigate-gate blocked [^"]*via=bash/.test(s.requests)],
|
||||
['repo: the sed -i write was put back, then the released Edit applied', fs.readFileSync(path.join(s.repo, 'src/total.ts'), 'utf8') === FIXED],
|
||||
];
|
||||
},
|
||||
},
|
||||
{
|
||||
name: 'plan-ceo-review', hook: 'ceo-review-gate', flag: '--ceo-review', skill: 'plan-ceo-review', prompt: 'review plan.md as a CEO and write the review into it',
|
||||
seed: repo => fs.writeFileSync(path.join(repo, 'README.md'), '# smoke\n'),
|
||||
|
||||
@@ -30,7 +30,6 @@ export const PR_PROFILE_CASE_IDS = [
|
||||
'setup-deploy-workflow', 'context-restore-loads-latest', 'plan-tune-inspect',
|
||||
'skillify-provenance-refusal', 'diagram-triplet', 'learnings-show',
|
||||
'gstack-upgrade-happy-path',
|
||||
'investigate-owned-completion', 'investigate-owned-abort', 'investigate-owned-ending-error',
|
||||
'ship-coverage-value', 'review-test-value', 'test-audit-report-only',
|
||||
'office-hours-auto-mode', 'plan-ceo-review-plan-mode',
|
||||
] as const;
|
||||
@@ -39,8 +38,6 @@ export const PR_PROFILE_CASE_IDS = [
|
||||
export const PR_PROFILE_FILES: Record<string, readonly string[]> = {
|
||||
'test/skill-e2e-office-hours-auto-mode.test.ts': ['office-hours-auto-mode'],
|
||||
'test/skill-e2e-plan-ceo-plan-mode.test.ts': ['plan-ceo-review-plan-mode'],
|
||||
'test/skill-e2e-investigate-owned-completion.test.ts': ['investigate-owned-completion'],
|
||||
'test/skill-e2e-investigate-owned-termination.test.ts': ['investigate-owned-abort', 'investigate-owned-ending-error'],
|
||||
'test/skill-e2e-hermetic-canary.test.ts': ['hermetic-canary', 'hermetic-sentinel'],
|
||||
'test/skill-e2e-bws.test.ts': ['browse-basic', 'browse-snapshot', 'skillmd-setup-discovery'],
|
||||
'test/skill-e2e-qa-workflow.test.ts': ['qa-bootstrap'],
|
||||
@@ -244,6 +241,7 @@ export const FREE_ONLY_PR_FILES = [
|
||||
'scripts/free-test-seed-allowlist.json', // Read only by the free seed ratchet (test/free-seed-ratchet.test.ts).
|
||||
'scripts/paid-test-durations.json',
|
||||
'scripts/ubicloud/**',
|
||||
'careful/bin/hook-extract.sh', // The careful/freeze hooks' path extractor: free test/hook-scripts.test.ts covers it; no paid case reads it since the /investigate cases retired (GST-1717).
|
||||
'qm/**', // The QM edition (scripts/gen-qm-edition.ts): shipped to QM as a deployment layer, read by no paid case; free test/qm-edition.test.ts covers it (GST-1492).
|
||||
'capy/skill/**', // The vendored Capy skill (Drive bootstrapper): run by Capy agents, read by no paid case; test/capy-skill-wrapper.test.ts covers it.
|
||||
'USING_GBRAIN_WITH_GSTACK.md', // The /setup-gbrain guide: README, docs/gbrain-quick-tour.md and free tests read it; no skill prompt or paid fixture does (GST-1027).
|
||||
@@ -498,7 +496,7 @@ export function selectPrProfile(options: {
|
||||
? `Its test file changed, but the PR lane cannot select it by name; name its Bun test '${id}' (testIfSelected) to run it on PRs`
|
||||
: 'Broad gate census/release coverage; outside the fast PR profile',
|
||||
}));
|
||||
const noQuickCoverage = files.filter(file => isPromptFile(file)
|
||||
const noQuickCoverage = files.filter(file => isPromptFile(file) && (!derived || fs.existsSync(path.join(ROOT, file)))
|
||||
&& !(depends(file, maps.globalTouchfiles) && (e2e.length > 0 || judges.length > 0))
|
||||
&& !e2e.some(id => depends(file, maps.e2eTouchfiles[id]))
|
||||
&& !judges.some(id => depends(file, maps.judgeTouchfiles[id])));
|
||||
|
||||
Reference in new issue
Block a user