mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-02 09:29:49 +02:00
fix(ship,qa,document-release): repair proof-run regressions and fixture gaps
- ship-docsync-completion: yesterday's audit-scope result dropped the section's status, so /ship spliced one in; the section now opens with **Status:**. - ship-docsync-missing-asset: a missing section or old Ship-owned mode blocks before launch. - ship-docsync-late-result: the invocation record says prepare already saves the candidate selection (no extra Read; budget unchanged). - qa exploratory: await the method Reads before the first probe. - qa-callers fixture: quote the real review-log record template; allow the git log command plan-completion prescribes. - qa functional observer: a receipt caught mid-link(2) is checked at stop instead of failing with ENOENT (reproduced from CI). Each repaired case passed a focused paid run.
This commit is contained in:
1 parent
4a87fa9d59
commit
77cce3bec4
16 files changed
+136
-21
No files matched your search
@@ -32,9 +32,10 @@ on the LAST nonempty line, without fences or trailing prose:
|
|||||||
- `files_updated`, `files_reviewed`: unique repo-relative file paths actually edited
|
- `files_updated`, `files_reviewed`: unique repo-relative file paths actually edited
|
||||||
and actually read; `blockers`, `decisions`: strings. Blockers name the decision and
|
and actually read; `blockers`, `decisions`: strings. Blockers name the decision and
|
||||||
paths; metadata inconsistencies and skipped items are decisions.
|
paths; metadata inconsistencies and skipped items are decisions.
|
||||||
- `documentation_section`: nonempty Markdown without a `## Documentation` heading:
|
- `documentation_section`: nonempty Markdown without a `## Documentation` heading,
|
||||||
audited scope, per-file status in Step 9's `Documentation health` form (no VERSION
|
complete for verbatim embedding: a first `**Status:**` line with `status` and the
|
||||||
row), and Step 1.5's coverage debt and diagram drift. Describe scope even without docs.
|
result, audited scope, per-file status in Step 9's `Documentation health` form (no
|
||||||
|
VERSION row), and Step 1.5's coverage debt and diagram drift. Describe scope even without docs.
|
||||||
|
|
||||||
## Discovery (both modes)
|
## Discovery (both modes)
|
||||||
|
|
||||||
|
|||||||
@@ -30,9 +30,10 @@ on the LAST nonempty line, without fences or trailing prose:
|
|||||||
- `files_updated`, `files_reviewed`: unique repo-relative file paths actually edited
|
- `files_updated`, `files_reviewed`: unique repo-relative file paths actually edited
|
||||||
and actually read; `blockers`, `decisions`: strings. Blockers name the decision and
|
and actually read; `blockers`, `decisions`: strings. Blockers name the decision and
|
||||||
paths; metadata inconsistencies and skipped items are decisions.
|
paths; metadata inconsistencies and skipped items are decisions.
|
||||||
- `documentation_section`: nonempty Markdown without a `## Documentation` heading:
|
- `documentation_section`: nonempty Markdown without a `## Documentation` heading,
|
||||||
audited scope, per-file status in Step 9's `Documentation health` form (no VERSION
|
complete for verbatim embedding: a first `**Status:**` line with `status` and the
|
||||||
row), and Step 1.5's coverage debt and diagram drift. Describe scope even without docs.
|
result, audited scope, per-file status in Step 9's `Documentation health` form (no
|
||||||
|
VERSION row), and Step 1.5's coverage debt and diagram drift. Describe scope even without docs.
|
||||||
|
|
||||||
## Discovery (both modes)
|
## Discovery (both modes)
|
||||||
|
|
||||||
|
|||||||
@@ -5,7 +5,7 @@
|
|||||||
The **caller** (/qa, /qa-only, /review or /ship) owns decisions, tests, fixes and publication. Discovery writes only reports/evidence
|
The **caller** (/qa, /qa-only, /review or /ship) owns decisions, tests, fixes and publication. Discovery writes only reports/evidence
|
||||||
and owned fixture state; no workflows, framework installs or publication.
|
and owned fixture state; no workflows, framework installs or publication.
|
||||||
|
|
||||||
Complete these Reads in order before writing charters or probing. Do not repeat a Read already completed in this invocation.
|
Complete these Reads in order before writing charters or probing. Await their results before the first probe, never in the same response. Do not repeat a Read already completed in this invocation.
|
||||||
1. Read `sections/scope.md` relative to the installed `qa`/`gstack-qa` SKILL.md directory in full and select the surfaces.
|
1. Read `sections/scope.md` relative to the installed `qa`/`gstack-qa` SKILL.md directory in full and select the surfaces.
|
||||||
2. Read the selected surface methods below in full.
|
2. Read the selected surface methods below in full.
|
||||||
|
|
||||||
|
|||||||
@@ -52,7 +52,7 @@ and owned fixture state; no workflows, framework installs or publication.
|
|||||||
|
|
||||||
${reportOnly ? `## 0. Preparation gate
|
${reportOnly ? `## 0. Preparation gate
|
||||||
|
|
||||||
Complete these Reads in order before writing charters or probing:` : 'Complete these Reads in order before writing charters or probing. Do not repeat a Read already completed in this invocation.'}
|
Complete these Reads in order before writing charters or probing:` : 'Complete these Reads in order before writing charters or probing. Await their results before the first probe, never in the same response. Do not repeat a Read already completed in this invocation.'}
|
||||||
1. Read ${sectionPath(ctx, 'qa', 'scope')} in full and select the surfaces.
|
1. Read ${sectionPath(ctx, 'qa', 'scope')} in full and select the surfaces.
|
||||||
2. Read the selected surface methods below in full.
|
2. Read the selected surface methods below in full.
|
||||||
|
|
||||||
|
|||||||
@@ -19,8 +19,8 @@ Reentry never resets the count or authorizes a launch.
|
|||||||
## Prepare the candidate
|
## Prepare the candidate
|
||||||
|
|
||||||
1. Read installed document-release SKILL.md and its full audit-scope/release-body
|
1. Read installed document-release SKILL.md and its full audit-scope/release-body
|
||||||
content, linked as sections or inlined for external hosts. Missing/old
|
content, linked as sections or inlined for external hosts. A missing section
|
||||||
`Ship-owned documentation mode` blocks; never substitute.
|
or old `Ship-owned documentation mode` blocks before launch; never substitute.
|
||||||
2. Select release paths and base SHA. Inspect committed changes (`git diff <diff-base> HEAD`),
|
2. Select release paths and base SHA. Inspect committed changes (`git diff <diff-base> HEAD`),
|
||||||
staged (`git diff --cached`), unstaged (`git diff`) and selected new files
|
staged (`git diff --cached`), unstaged (`git diff`) and selected new files
|
||||||
(`git ls-files --others --exclude-standard`; read contents). Store-only audits
|
(`git ls-files --others --exclude-standard`; read contents). Store-only audits
|
||||||
|
|||||||
@@ -17,8 +17,8 @@ Reentry never resets the count or authorizes a launch.
|
|||||||
## Prepare the candidate
|
## Prepare the candidate
|
||||||
|
|
||||||
1. Read installed document-release SKILL.md and its full audit-scope/release-body
|
1. Read installed document-release SKILL.md and its full audit-scope/release-body
|
||||||
content, linked as sections or inlined for external hosts. Missing/old
|
content, linked as sections or inlined for external hosts. A missing section
|
||||||
`Ship-owned documentation mode` blocks; never substitute.
|
or old `Ship-owned documentation mode` blocks before launch; never substitute.
|
||||||
2. Select release paths and base SHA. Inspect committed changes (`git diff <diff-base> HEAD`),
|
2. Select release paths and base SHA. Inspect committed changes (`git diff <diff-base> HEAD`),
|
||||||
staged (`git diff --cached`), unstaged (`git diff`) and selected new files
|
staged (`git diff --cached`), unstaged (`git diff`) and selected new files
|
||||||
(`git ls-files --others --exclude-standard`; read contents). Store-only audits
|
(`git ls-files --others --exclude-standard`; read contents). Store-only audits
|
||||||
|
|||||||
@@ -19,6 +19,9 @@ test('prepare copies the exact generated prompt and snapshots actual inputs with
|
|||||||
const prepared = JSON.parse(response.text);
|
const prepared = JSON.parse(response.text);
|
||||||
const candidate = JSON.parse(fs.readFileSync(prepared.candidate, 'utf8'));
|
const candidate = JSON.parse(fs.readFileSync(prepared.candidate, 'utf8'));
|
||||||
expect(candidate).toEqual(docsCandidate(fixture.repo, 'first', 'edit', candidate.base_sha));
|
expect(candidate).toEqual(docsCandidate(fixture.repo, 'first', 'edit', candidate.base_sha));
|
||||||
|
const supplied = JSON.parse(fs.readFileSync(path.join(fixture.home, 'candidate.json'), 'utf8'));
|
||||||
|
expect(candidate.selected_paths).toEqual(supplied.selected_paths);
|
||||||
|
expect(fs.readFileSync(fixture.invocation, 'utf8')).toContain('prepare saves the same selection with current hashes, so it needs no separate Read.');
|
||||||
const source = extractDocsDispatch(fs.readFileSync(path.join(fixture.skills, 'ship/sections/documentation.md'), 'utf8'));
|
const source = extractDocsDispatch(fs.readFileSync(path.join(fixture.skills, 'ship/sections/documentation.md'), 'utf8'));
|
||||||
expect(fs.readFileSync(prepared.prompt, 'utf8')).toBe(source.replaceAll('${HOME}', fixture.home)
|
expect(fs.readFileSync(prepared.prompt, 'utf8')).toBe(source.replaceAll('${HOME}', fixture.home)
|
||||||
.replaceAll('<branch>', 'feature/docs').replaceAll('<base>', 'main')
|
.replaceAll('<branch>', 'feature/docs').replaceAll('<base>', 'main')
|
||||||
|
|||||||
+2
-2
@@ -2903,8 +2903,8 @@ Reentry never resets the count or authorizes a launch.
|
|||||||
## Prepare the candidate
|
## Prepare the candidate
|
||||||
|
|
||||||
1. Read installed document-release SKILL.md and its full audit-scope/release-body
|
1. Read installed document-release SKILL.md and its full audit-scope/release-body
|
||||||
content, linked as sections or inlined for external hosts. Missing/old
|
content, linked as sections or inlined for external hosts. A missing section
|
||||||
`Ship-owned documentation mode` blocks; never substitute.
|
or old `Ship-owned documentation mode` blocks before launch; never substitute.
|
||||||
2. Select release paths and base SHA. Inspect committed changes (`git diff <diff-base> HEAD`),
|
2. Select release paths and base SHA. Inspect committed changes (`git diff <diff-base> HEAD`),
|
||||||
staged (`git diff --cached`), unstaged (`git diff`) and selected new files
|
staged (`git diff --cached`), unstaged (`git diff`) and selected new files
|
||||||
(`git ls-files --others --exclude-standard`; read contents). Store-only audits
|
(`git ls-files --others --exclude-standard`; read contents). Store-only audits
|
||||||
|
|||||||
+2
-2
@@ -3167,8 +3167,8 @@ Reentry never resets the count or authorizes a launch.
|
|||||||
## Prepare the candidate
|
## Prepare the candidate
|
||||||
|
|
||||||
1. Read installed document-release SKILL.md and its full audit-scope/release-body
|
1. Read installed document-release SKILL.md and its full audit-scope/release-body
|
||||||
content, linked as sections or inlined for external hosts. Missing/old
|
content, linked as sections or inlined for external hosts. A missing section
|
||||||
`Ship-owned documentation mode` blocks; never substitute.
|
or old `Ship-owned documentation mode` blocks before launch; never substitute.
|
||||||
2. Select release paths and base SHA. Inspect committed changes (`git diff <diff-base> HEAD`),
|
2. Select release paths and base SHA. Inspect committed changes (`git diff <diff-base> HEAD`),
|
||||||
staged (`git diff --cached`), unstaged (`git diff`) and selected new files
|
staged (`git diff --cached`), unstaged (`git diff`) and selected new files
|
||||||
(`git ls-files --others --exclude-standard`; read contents). Store-only audits
|
(`git ls-files --others --exclude-standard`; read contents). Store-only audits
|
||||||
|
|||||||
@@ -232,7 +232,7 @@ Earlier review stages are synthetic and outside this fixture. No live review han
|
|||||||
## Checks
|
## Checks
|
||||||
Earlier check stages are synthetic and outside this fixture. No test receipts are asserted.
|
Earlier check stages are synthetic and outside this fixture. No test receipts are asserted.
|
||||||
## Initial documentation state
|
## Initial documentation state
|
||||||
Attempts used: 0. No accepted audit, hashes, exception or child handle. The supplied candidate.json is initial fixture input, not an accepted audit.
|
Attempts used: 0. No accepted audit, hashes, exception or child handle. The supplied candidate.json is initial fixture input, not an accepted audit; prepare saves the same selection with current hashes, so it needs no separate Read.
|
||||||
## Initial next steps
|
## Initial next steps
|
||||||
1. CURRENT: documentation phase (Step 14.5, or store documentation preflight).
|
1. CURRENT: documentation phase (Step 14.5, or store documentation preflight).
|
||||||
2. Save the result and optionally execute the authorized local publication stand-in if the actual documentation gate permits it.
|
2. Save the result and optionally execute the authorized local publication stand-in if the actual documentation gate permits it.
|
||||||
|
|||||||
@@ -243,7 +243,7 @@ export function qaCallerCommandAllowed(command: string, workflowCommands: string
|
|||||||
}
|
}
|
||||||
if (/[\n\r;&|<>`$\\(){}]/.test(text)) return false;
|
if (/[\n\r;&|<>`$\\(){}]/.test(text)) return false;
|
||||||
return /^(?:pwd|ls(?: -la)?|bun --version|date -u \+%Y-%m-%dT%H:%M:%SZ|bun (?:run test|test(?: cli\.test\.ts)?))$/.test(text)
|
return /^(?:pwd|ls(?: -la)?|bun --version|date -u \+%Y-%m-%dT%H:%M:%SZ|bun (?:run test|test(?: cli\.test\.ts)?))$/.test(text)
|
||||||
|| /^git (?:status --(?:short|porcelain)|branch --show-current|rev-parse (?:--short )?HEAD|merge-base origin\/main HEAD|ls-files(?: --others --exclude-standard)?)$/.test(text)
|
|| /^git (?:status --(?:short|porcelain)|branch --show-current|rev-parse (?:--short )?HEAD|merge-base origin\/main HEAD|log origin\/main\.\.HEAD --oneline|ls-files(?: --others --exclude-standard)?)$/.test(text)
|
||||||
|| /^\/?[\w./-]+\/bin\/gstack-review-log --start (?:review|adversarial-review)$/.test(text)
|
|| /^\/?[\w./-]+\/bin\/gstack-review-log --start (?:review|adversarial-review)$/.test(text)
|
||||||
|| /^\/?[\w./-]+\/bin\/gstack-(?:review-read|specialist-stats)$/.test(text);
|
|| /^\/?[\w./-]+\/bin\/gstack-(?:review-read|specialist-stats)$/.test(text);
|
||||||
}
|
}
|
||||||
@@ -638,9 +638,16 @@ process.exit(exit ?? 127);
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function callerReviewRecordTemplate(fixture: Pick<QaCallerFixture, 'caller' | 'runtime'>): string {
|
||||||
|
const source = fs.readFileSync(path.join(QA_CALLER_ROOT, fixture.caller === 'review' ? 'review/SKILL.md' : 'ship/sections/review-army.md'), 'utf8');
|
||||||
|
const templates = source.match(/^~\/\.claude\/skills\/gstack\/bin\/gstack-review-log '\{"skill":"review",.*$/gm) ?? [];
|
||||||
|
if (templates.length !== 1) throw new Error(`Expected one installed /${fixture.caller} review record template, found ${templates.length}`);
|
||||||
|
return templates[0].replace('~/.claude/skills/gstack', fixture.runtime);
|
||||||
|
}
|
||||||
|
|
||||||
export function qaCallerSessionOptions(fixture: QaCallerFixture, runId: string): Parameters<typeof runSkillTest>[0] {
|
export function qaCallerSessionOptions(fixture: QaCallerFixture, runId: string): Parameters<typeof runSkillTest>[0] {
|
||||||
return {
|
return {
|
||||||
prompt: `Load gstack's /${fixture.caller} supplied parent phase from caller-${fixture.caller}.md and resume it on the selected working-tree diff against origin/main. This excerpt comes from ${fixture.runtime}/${fixture.caller}/SKILL.md; resolve installed-relative references there, not from the excerpt file or product directory. That path identifies the asset base, not another entrypoint: do not read or invoke the full parent SKILL.md or rerun its preamble. Earlier preamble/branch/base setup is complete; use the existing local origin/main ref without fetch. Earlier-phase asset locators are ${fixture.runtime}/review/checklist.md and ${fixture.runtime}/qa/templates/functional-report-template.md. Read those files directly when referenced; recursive Glob does not follow the installed asset symlinks. Cross-project learnings are configured off in this owned fixture. ${fixture.reviewStart ? `The actual review-start helper already captured REVIEW_START=${fixture.reviewStart} for this unchanged core pass; retain that token. ` : ''} This fixture evaluates only the supplied parent phase, not later publication stages. Read README.md for the project contract and commands. Use diagnostic-client commands such as \`bun scripts/probe.ts <literal>\` for exploratory discoveries and their checkpoint evidence. A required \`bun run test\` is separate suite verification: report it as verification, never as a diagnostic observation or checkpoint anchor/target. Use the production evidence helper to publish each diagnostic checkpoint as \`reports/exploration-NNN.json\`, not inside a nested directory; do not transcribe its observed payload. ${fixture.caseId === 'ship-exploratory-plan-checks' ? 'The previously discovered plan is PLAN.md.' : 'No plan file was found.'} There is no remote service and no release publication is authorized. There is no interactive approver; do not invent answers or permission. Keep normal parent decision gates. Before every completion report or bookkeeping log, read HANDOFF.md and reports/HANDOFF.md if present for any concurrent collaborator update, await the results, and compare evidence with current inputs. A gstack-review-log completed:true record is a completion, not preliminary bookkeeping; a later handoff read cannot validate an earlier completion.\n\nDeadline bookkeeping additionally permits \`bun ${fixture.runtime}/bin/gstack-qa-deadline start ${fixture.cwd}/reports/deadline.json SECONDS [EARLIER_UTC]\`, \`bun ${fixture.runtime}/bin/gstack-qa-deadline status ${fixture.cwd}/reports/deadline.json\`, and \`bun ${fixture.runtime}/bin/gstack-qa-deadline run ${fixture.cwd}/reports/deadline.json -- bun scripts/probe.ts [literal]\`. These are closed literal forms: SECONDS must be positive and at most 300, EARLIER_UTC is the optional caller absolute deadline: use the section clock's Hard deadline UTC, never its Runner entry UTC, reserve-start time or a clock-read time. The child is only the existing diagnostic client with zero or one literal argument. Resolve these exact helper and state paths; do not use variables, another helper, another state file, nested wrappers, scripts, operators or substitutions. Only this helper may create or change reports/deadline.json and its .qa-deadline- temporary files; never use Write/Edit/MultiEdit on those paths. Record the full outer run command in checkpoints and evidence; keep the unchanged child JSON as observed, separate from prefixed guard diagnostics. A completed expired guard-run is not a probe or a pass: retain its unused checkpoint, report not-run coverage and do not restart the deadline. Keep the 12-probe smoke limit. Required suites and explicit plan checks are outside the bounded smoke budget, not permission to reset it.\n\nFunctional evidence uses the same production helper and existing diagnostic client: \`bun ${fixture.runtime}/bin/gstack-qa-evidence capture ${fixture.cwd}/reports NNN --public --deadline ${fixture.cwd}/reports/deadline.json -- bun scripts/probe.ts [literal]\`. These diagnostic receipts are declared public/synthetic, so --public is approved; a fresh three-digit ID is required each time. Explicit plan probes outside the smoke budget may replace --deadline with --timeout-ms 10000; this does not reset or bypass the smoke deadline. Publish causal intent with \`bun ${fixture.runtime}/bin/gstack-qa-evidence checkpoint ${fixture.cwd}/reports NNN CAPTURE_ID 'full prior capture command' 'causal hypothesis' 'full next capture command'\`; quote arguments literally. For complex quoting, Write only capture, observationCommand, hypothesis and nextCommand to reports/intent.json; publish with the same helper: checkpoint REPORT_ROOT NNN intent.json. Materialize is supported when evidence.json is required. Sources stay inside reports. Decide to execute the next probe before publishing its checkpoint, then await successful publication and dispatch that exact probe. If you defer an optional idea or stop exploration, do not publish a checkpoint for it; descri
Line truncated
|
prompt: `Load gstack's /${fixture.caller} supplied parent phase from caller-${fixture.caller}.md and resume it on the selected working-tree diff against origin/main. This excerpt comes from ${fixture.runtime}/${fixture.caller}/SKILL.md; resolve installed-relative references there, not from the excerpt file or product directory. That path identifies the asset base, not another entrypoint: do not read or invoke the full parent SKILL.md or rerun its preamble. Earlier preamble/branch/base setup is complete; use the existing local origin/main ref without fetch. Earlier-phase asset locators are ${fixture.runtime}/review/checklist.md and ${fixture.runtime}/qa/templates/functional-report-template.md. Read those files directly when referenced; recursive Glob does not follow the installed asset symlinks. Cross-project learnings are configured off in this owned fixture. ${fixture.reviewStart ? `The actual review-start helper already captured REVIEW_START=${fixture.reviewStart} for this unchanged core pass; retain that token. ` : ''} This fixture evaluates only the supplied parent phase, not later publication stages. Read README.md for the project contract and commands. Use diagnostic-client commands such as \`bun scripts/probe.ts <literal>\` for exploratory discoveries and their checkpoint evidence. A required \`bun run test\` is separate suite verification: report it as verification, never as a diagnostic observation or checkpoint anchor/target. Use the production evidence helper to publish each diagnostic checkpoint as \`reports/exploration-NNN.json\`, not inside a nested directory; do not transcribe its observed payload. ${fixture.caseId === 'ship-exploratory-plan-checks' ? 'The previously discovered plan is PLAN.md.' : 'No plan file was found.'} There is no remote service and no release publication is authorized. There is no interactive approver; do not invent answers or permission. Keep normal parent decision gates. Before every completion report or bookkeeping log, read HANDOFF.md and reports/HANDOFF.md if present for any concurrent collaborator update, await the results, and compare evidence with current inputs. A gstack-review-log completed:true record is a completion, not preliminary bookkeeping; a later handoff read cannot validate an earlier completion.\n\nDeadline bookkeeping additionally permits \`bun ${fixture.runtime}/bin/gstack-qa-deadline start ${fixture.cwd}/reports/deadline.json SECONDS [EARLIER_UTC]\`, \`bun ${fixture.runtime}/bin/gstack-qa-deadline status ${fixture.cwd}/reports/deadline.json\`, and \`bun ${fixture.runtime}/bin/gstack-qa-deadline run ${fixture.cwd}/reports/deadline.json -- bun scripts/probe.ts [literal]\`. These are closed literal forms: SECONDS must be positive and at most 300, EARLIER_UTC is the optional caller absolute deadline: use the section clock's Hard deadline UTC, never its Runner entry UTC, reserve-start time or a clock-read time. The child is only the existing diagnostic client with zero or one literal argument. Resolve these exact helper and state paths; do not use variables, another helper, another state file, nested wrappers, scripts, operators or substitutions. Only this helper may create or change reports/deadline.json and its .qa-deadline- temporary files; never use Write/Edit/MultiEdit on those paths. Record the full outer run command in checkpoints and evidence; keep the unchanged child JSON as observed, separate from prefixed guard diagnostics. A completed expired guard-run is not a probe or a pass: retain its unused checkpoint, report not-run coverage and do not restart the deadline. Keep the 12-probe smoke limit. Required suites and explicit plan checks are outside the bounded smoke budget, not permission to reset it.\n\nFunctional evidence uses the same production helper and existing diagnostic client: \`bun ${fixture.runtime}/bin/gstack-qa-evidence capture ${fixture.cwd}/reports NNN --public --deadline ${fixture.cwd}/reports/deadline.json -- bun scripts/probe.ts [literal]\`. These diagnostic receipts are declared public/synthetic, so --public is approved; a fresh three-digit ID is required each time. Explicit plan probes outside the smoke budget may replace --deadline with --timeout-ms 10000; this does not reset or bypass the smoke deadline. Publish causal intent with \`bun ${fixture.runtime}/bin/gstack-qa-evidence checkpoint ${fixture.cwd}/reports NNN CAPTURE_ID 'full prior capture command' 'causal hypothesis' 'full next capture command'\`; quote arguments literally. For complex quoting, Write only capture, observationCommand, hypothesis and nextCommand to reports/intent.json; publish with the same helper: checkpoint REPORT_ROOT NNN intent.json. Materialize is supported when evidence.json is required. Sources stay inside reports. Decide to execute the next probe before publishing its checkpoint, then await successful publication and dispatch that exact probe. If you defer an optional idea or stop exploration, do not publish a checkpoint for it; descri
Line truncated
|
||||||
appendSystemPrompt: `Caller execution scheduling (fixture contract):
|
appendSystemPrompt: `Caller execution scheduling (fixture contract):
|
||||||
This session has at most 25 assistant turns, including required verification and final artifacts. The command boundary applies to each Bash call, not to the number of independent tool calls in an assistant turn.
|
This session has at most 25 assistant turns, including required verification and final artifacts. The command boundary applies to each Bash call, not to the number of independent tool calls in an assistant turn.
|
||||||
After required clock and approval prerequisites settle, issue independent source Reads and read-only discovery together as separate native tool calls once their paths and inputs are known. Wait for their results before decisions that depend on them.
|
After required clock and approval prerequisites settle, issue independent source Reads and read-only discovery together as separate native tool calls once their paths and inputs are known. Wait for their results before decisions that depend on them.
|
||||||
|
|||||||
@@ -95,6 +95,7 @@ export async function observeQAWrites(root: string, options: { reportDirectory?:
|
|||||||
const events: QAWriteObservation['events'] = [];
|
const events: QAWriteObservation['events'] = [];
|
||||||
const failures: string[] = [];
|
const failures: string[] = [];
|
||||||
const publications = new Map<string, { temporary: string; dev: number; ino: number; bytes: string; parentDev: number; parentIno: number; mode: number }>();
|
const publications = new Map<string, { temporary: string; dev: number; ino: number; bytes: string; parentDev: number; parentIno: number; mode: number }>();
|
||||||
|
const pendingLinks = new Map<string, { target: string; dev: number; ino: number; mode: number; bytes: string }>();
|
||||||
let stopped = false;
|
let stopped = false;
|
||||||
const observedPath = (relative: string, knownPair = false): string => {
|
const observedPath = (relative: string, knownPair = false): string => {
|
||||||
try {
|
try {
|
||||||
@@ -150,6 +151,18 @@ export async function observeQAWrites(root: string, options: { reportDirectory?:
|
|||||||
publications.set(relativeTarget, publication);
|
publications.set(relativeTarget, publication);
|
||||||
return target;
|
return target;
|
||||||
} catch (publicationError) {
|
} catch (publicationError) {
|
||||||
|
if (receipt === undefined && (publicationError as NodeJS.ErrnoException).code === 'ENOENT' && temporaryName.test(basename)) {
|
||||||
|
let linking: number | undefined;
|
||||||
|
try { linking = fs.openSync(path.join(parent, basename), fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | fs.constants.O_NONBLOCK); }
|
||||||
|
catch (openError) { if ((openError as NodeJS.ErrnoException).code === 'ENOENT') return path.join(parent, basename); }
|
||||||
|
if (linking !== undefined) try {
|
||||||
|
const entry = fs.fstatSync(linking);
|
||||||
|
if (entry.isFile() && entry.nlink === 2 && entry.uid === parentStat.uid && (entry.mode & 0o777) === mode) {
|
||||||
|
pendingLinks.set(relative, { target: path.relative(root, target), dev: entry.dev, ino: entry.ino, mode, bytes: fs.readFileSync(linking, 'utf8') });
|
||||||
|
return path.join(parent, basename);
|
||||||
|
}
|
||||||
|
} finally { fs.closeSync(linking); }
|
||||||
|
}
|
||||||
try { return ownedPath(root, relative); } catch { throw publicationError; }
|
try { return ownedPath(root, relative); } catch { throw publicationError; }
|
||||||
} finally { if (receipt !== undefined) fs.closeSync(receipt); }
|
} finally { if (receipt !== undefined) fs.closeSync(receipt); }
|
||||||
}
|
}
|
||||||
@@ -241,6 +254,13 @@ export async function observeQAWrites(root: string, options: { reportDirectory?:
|
|||||||
|| fs.readFileSync(target, 'utf8') !== publication.bytes) throw new Error('Evidence publication did not settle unchanged');
|
|| fs.readFileSync(target, 'utf8') !== publication.bytes) throw new Error('Evidence publication did not settle unchanged');
|
||||||
} catch (error) { failures.push(String(pathFailure(root, relative, error))); }
|
} catch (error) { failures.push(String(pathFailure(root, relative, error))); }
|
||||||
}
|
}
|
||||||
|
for (const [temporary, pending] of pendingLinks) {
|
||||||
|
try {
|
||||||
|
const entry = fs.lstatSync(ownedPath(root, pending.target));
|
||||||
|
if (fs.existsSync(ownedPath(root, temporary)) || entry.dev !== pending.dev || entry.ino !== pending.ino || entry.nlink !== 1
|
||||||
|
|| (entry.mode & 0o777) !== pending.mode || fs.readFileSync(ownedPath(root, pending.target), 'utf8') !== pending.bytes) throw new Error('Evidence publication did not settle unchanged');
|
||||||
|
} catch (error) { failures.push(String(pathFailure(root, temporary, error))); }
|
||||||
|
}
|
||||||
let after: Record<string, string> = {};
|
let after: Record<string, string> = {};
|
||||||
try { after = qaTreeSnapshot(root); } catch (error) { failures.push(String(error)); }
|
try { after = qaTreeSnapshot(root); } catch (error) { failures.push(String(error)); }
|
||||||
fs.closeSync(fd);
|
fs.closeSync(fd);
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ import * as path from 'node:path';
|
|||||||
import * as os from 'node:os';
|
import * as os from 'node:os';
|
||||||
import { spawnSync } from 'node:child_process';
|
import { spawnSync } from 'node:child_process';
|
||||||
import {
|
import {
|
||||||
callerExcerpt, callerSnapshot, callerTools, createQaCallerFixture, qaCallerInstructions,
|
callerExcerpt, callerReviewRecordTemplate, callerSnapshot, callerTools, createQaCallerFixture, qaCallerInstructions,
|
||||||
QA_CALLER_CASES, QA_CALLER_TEST_MS,
|
QA_CALLER_CASES, QA_CALLER_TEST_MS,
|
||||||
qaCallerSessionOptions, qaCallerCommandAllowed, readCallerReceipt, retainQaCallerEvidence, runQaCaller, validateCallerEvidence,
|
qaCallerSessionOptions, qaCallerCommandAllowed, readCallerReceipt, retainQaCallerEvidence, runQaCaller, validateCallerEvidence,
|
||||||
type CallerProbe, type CallerReceipt, type QaCallerFixture,
|
type CallerProbe, type CallerReceipt, type QaCallerFixture,
|
||||||
@@ -136,11 +136,48 @@ describe('caller native-event observer controls', () => {
|
|||||||
const fixture = createQaCallerFixture(id, { installRuntime: false });
|
const fixture = createQaCallerFixture(id, { installRuntime: false });
|
||||||
try { return qaCallerSessionOptions(fixture, 'free-control').prompt; } finally { fs.rmSync(fixture.root, { recursive: true, force: true }); }
|
try { return qaCallerSessionOptions(fixture, 'free-control').prompt; } finally { fs.rmSync(fixture.root, { recursive: true, force: true }); }
|
||||||
};
|
};
|
||||||
expect(prompt('review-exploratory-small-cli')).toContain("/bin/gstack-review-log '<JSON>' --finish <token>`, never through bun");
|
expect(prompt('review-exploratory-small-cli')).toContain('never through bun or another interpreter. A review record fills this installed template, keeping its keys and adding none: `');
|
||||||
expect(prompt('review-exploratory-small-cli')).toContain('otherwise issues_found; a review stopped at a gate records completed:false');
|
expect(prompt('review-exploratory-small-cli')).toContain('otherwise issues_found; a review stopped at a gate records completed:false');
|
||||||
expect(prompt('ship-exploratory-small-cli')).toContain('otherwise issues_found (unavailable for missing dispatched reviewer output)');
|
expect(prompt('ship-exploratory-small-cli')).toContain('otherwise issues_found (unavailable for missing dispatched reviewer output)');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('captured PR-lane review record: an invented shape without the installed template is rejected; the template is quoted', () => {
|
||||||
|
// ci-36641824710-1-eval-slices-6 review-exploratory-small-cli, native event 8jewsM (fixture paths shortened).
|
||||||
|
const invented = `/runtime/bin/gstack-review-log '{"timestamp":"'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'","commit":"'"$(git rev-parse --short HEAD)"'","branch":"caller-change","status":"issues_found","completed":false,"gate":"fix-first-ask","findings":[{"path":"scale.ts","line":3,"category":"functional-contract","severity":"CRITICAL","fingerprint":"scale.ts:3:functional-contract","confidence":10,"action":"ask","summary":"new !n guard rejects documented lower bound 0 (exit 2 instead of 0)"}],"qa":{"probes":["probe-167de79c-2645-4b60-b06b-3d0d6eebf13f"],"checkpoints":["exploration-001.json","exploration-002.json"],"suite":"bun run test 1 pass 0 fail"},"remaining":["fix-first-ask-approval"]}' --finish 37b567b3-c4bf-4469-9bad-29096740c234`;
|
||||||
|
const errorsFor = (command: string) => {
|
||||||
|
const observed = evidence();
|
||||||
|
observed.result.transcript.push(...nativeCall('record', 'Bash', { command }, ''));
|
||||||
|
return validateCallerEvidence(observed);
|
||||||
|
};
|
||||||
|
expect(errorsFor(invented)).toEqual(['command outside declared caller observation interface', 'QA checkpoint: Unsupported checkpoint Bash interaction']);
|
||||||
|
for (const caller of ['review', 'ship'] as const) {
|
||||||
|
const template = callerReviewRecordTemplate({ caller, runtime: '/runtime' });
|
||||||
|
expect(template.startsWith(`/runtime/bin/gstack-review-log '{"skill":"review","timestamp":`)).toBe(true);
|
||||||
|
expect(template).toEndWith(`}' --finish REVIEW_START`);
|
||||||
|
const filled = template.replace('"timestamp":"TIMESTAMP"', `"timestamp":"'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'"`)
|
||||||
|
.replace('"commit":"COMMIT"', `"commit":"'"$(git rev-parse --short HEAD)"'"`)
|
||||||
|
.replace('"STATUS"', '"issues_found"').replace(/"issues_found":N/, '"issues_found":1').replace('"critical":N', '"critical":1').replace('"informational":N', '"informational":0')
|
||||||
|
.replace('SCORE', '10.0').replace('SPECIALISTS_JSON', '{}').replace('FINDINGS_JSON', '[{"fingerprint":"scale.ts:3:functional-contract","severity":"CRITICAL","action":"ask-pending"}]')
|
||||||
|
.replace('COMPLETED', 'false').replace('CONVERGED', 'false').replace('CYCLES', '0').replace('REVIEW_START', 'native-token');
|
||||||
|
expect(errorsFor(filled)).toEqual([]);
|
||||||
|
const fixture = createQaCallerFixture(caller === 'review' ? 'review-exploratory-small-cli' : 'ship-exploratory-small-cli', { installRuntime: false });
|
||||||
|
try { expect(qaCallerSessionOptions(fixture, 'free-control').prompt).toContain(callerReviewRecordTemplate(fixture)); } finally { fs.rmSync(fixture.root, { recursive: true, force: true }); }
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('captured PR-lane plan-completion git log is in the caller interface; other log forms stay outside', () => {
|
||||||
|
// ci-36641824710-1-eval-slices-7 ship-exploratory-plan-checks, native event 3Pvdmu: ship/sections/plan-completion.md requires this read.
|
||||||
|
expect(fs.readFileSync(path.join(import.meta.dir, '../ship/sections/plan-completion.md'), 'utf8')).toContain('`git log origin/<base>..HEAD --oneline`');
|
||||||
|
expect(qaCallerCommandAllowed('git log origin/main..HEAD --oneline')).toBe(true);
|
||||||
|
for (const command of ['git log', 'git log --oneline', 'git log origin/main..HEAD', 'git log origin/main..HEAD --oneline -p',
|
||||||
|
'git log origin/main..HEAD --oneline --output=/tmp/x', 'git log --all --oneline', 'git log origin/main..HEAD --oneline; git push',
|
||||||
|
'git log origin/main..HEAD --oneline && git commit -am x', 'git -c core.pager=x log origin/main..HEAD --oneline', 'git log origin/main...HEAD --oneline']) {
|
||||||
|
expect(qaCallerCommandAllowed(command)).toBe(false);
|
||||||
|
}
|
||||||
|
const fixture = createQaCallerFixture('ship-exploratory-plan-checks', { installRuntime: false });
|
||||||
|
try { expect(qaCallerSessionOptions(fixture, 'free-control').prompt).toContain("git log origin/main..HEAD --oneline, git diff"); } finally { fs.rmSync(fixture.root, { recursive: true, force: true }); }
|
||||||
|
});
|
||||||
|
|
||||||
test('a later unchanged handoff reread does not invalidate an already completed freshness decision', () => {
|
test('a later unchanged handoff reread does not invalidate an already completed freshness decision', () => {
|
||||||
const observed = evidence();
|
const observed = evidence();
|
||||||
observed.result.transcript.push(
|
observed.result.transcript.push(
|
||||||
|
|||||||
@@ -110,6 +110,33 @@ describe('QA command-observation boundary', () => {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ci-36709485593-1-eval-slices-6 qa-functional-cli-fix: link(2) raised the temporary receipt's nlink to 2 before the
|
||||||
|
// receipt.json name resolved, so the observer's open failed with ENOENT during an ordinary atomic publication.
|
||||||
|
for (const variant of ['published', 'foreign-link-kept', 'foreign-link-dropped', 'replaced-target'] as const) {
|
||||||
|
test(`evidence publication observed mid-link: ${variant}`, async () => {
|
||||||
|
const fixture = createQAFunctionalFixture('cli');
|
||||||
|
const outside = fs.mkdtempSync(path.join(path.dirname(fixture.root), 'qa-link-'));
|
||||||
|
const observer = await observeQAWrites(fixture.root, { evidenceProducer: true });
|
||||||
|
try {
|
||||||
|
const directory = path.join(fixture.root, 'qa-reports/.qa-evidence/002');
|
||||||
|
fs.mkdirSync(directory, { recursive: true, mode: 0o700 });
|
||||||
|
observer.drain();
|
||||||
|
const temporary = path.join(directory, 'receipt.json.tmp.2282.fd4f6b4d');
|
||||||
|
const target = path.join(directory, 'receipt.json');
|
||||||
|
const foreign = path.join(outside, 'second-name');
|
||||||
|
fs.writeFileSync(temporary, JSON.stringify({ version: 1, id: '002', status: 'complete' }), { mode: 0o600 });
|
||||||
|
fs.linkSync(temporary, foreign);
|
||||||
|
observer.drain();
|
||||||
|
if (variant === 'published') { fs.unlinkSync(foreign); fs.linkSync(temporary, target); observer.drain(); fs.unlinkSync(temporary); }
|
||||||
|
if (variant === 'foreign-link-dropped') fs.unlinkSync(temporary);
|
||||||
|
if (variant === 'replaced-target') { fs.unlinkSync(foreign); fs.unlinkSync(temporary); fs.writeFileSync(target, '{}', { mode: 0o600 }); }
|
||||||
|
const verdict = qaWriteVerdict(observer.stop(), 'qa-only');
|
||||||
|
if (variant === 'published') expect(verdict).toEqual([]);
|
||||||
|
else expect(verdict).toContain('incomplete write observation');
|
||||||
|
} finally { fixture.cleanup(); fs.rmSync(outside, { recursive: true, force: true }); }
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
test('lost directory watches and allowed-directory link substitutions fail closed', async () => {
|
test('lost directory watches and allowed-directory link substitutions fail closed', async () => {
|
||||||
const fixture = createQAFunctionalFixture('cli');
|
const fixture = createQAFunctionalFixture('cli');
|
||||||
const observer = await observeQAWrites(fixture.root);
|
const observer = await observeQAWrites(fixture.root);
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import { HOST_PATHS } from '../scripts/resolvers/types';
|
|||||||
|
|
||||||
function assertPreparation(text: string) {
|
function assertPreparation(text: string) {
|
||||||
expect(text).toContain('Complete these Reads in order before writing charters or probing');
|
expect(text).toContain('Complete these Reads in order before writing charters or probing');
|
||||||
|
expect(text).toMatch(/Await their results before the first probe, never in the same response\.|Await each successful Read result before continuing\./);
|
||||||
expect(text).toContain('Do not repeat a Read already completed in this invocation');
|
expect(text).toContain('Do not repeat a Read already completed in this invocation');
|
||||||
const stages = ['1. Read `sections/scope.md`', 'in full and select the surfaces',
|
const stages = ['1. Read `sections/scope.md`', 'in full and select the surfaces',
|
||||||
'2. Read the selected surface methods below in full', '**Functional surfaces:**',
|
'2. Read the selected surface methods below in full', '**Functional surfaces:**',
|
||||||
@@ -240,6 +241,8 @@ describe('QA probe entry and checkpoint gates', () => {
|
|||||||
text.replace(method, method + '\n' + method),
|
text.replace(method, method + '\n' + method),
|
||||||
'Write a **charter**\n' + text,
|
'Write a **charter**\n' + text,
|
||||||
text.replace('Do not repeat a Read already completed in this invocation', 'Repeat all Reads'),
|
text.replace('Do not repeat a Read already completed in this invocation', 'Repeat all Reads'),
|
||||||
|
// ci-36641820398-1-gate-census-7 ship-exploratory-small-cli dispatched its first probe with the resource Reads.
|
||||||
|
text.replace(' Await their results before the first probe, never in the same response.', ''),
|
||||||
]) expect(() => assertPreparation(changed)).toThrow();
|
]) expect(() => assertPreparation(changed)).toThrow();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
@@ -138,6 +138,22 @@ describe('pre-publication documentation lifecycle', () => {
|
|||||||
expect(read('ship/sections/apple-release.md.tmpl')).toContain('ship/sections/documentation.md');
|
expect(read('ship/sections/apple-release.md.tmpl')).toContain('ship/sections/documentation.md');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('ship-owned documentation_section carries its status so /ship embeds it unchanged', () => {
|
||||||
|
// ci-36641820398-1-gate-census-3 ship-docsync-completion: the section had scope, health and debt but no result,
|
||||||
|
// so the parent spliced a Status line into it and the report no longer contained the returned section.
|
||||||
|
const scope = read('document-release/sections/audit-scope.md.tmpl').replace(/\s+/g, ' ');
|
||||||
|
expect(scope).toContain('complete for verbatim embedding: a first `**Status:**` line with `status` and the result, audited scope');
|
||||||
|
expect(read('ship/sections/documentation.md.tmpl')).toContain('nonempty Markdown with scope, result and debt');
|
||||||
|
expect(read('ship/sections/pr-body.md.tmpl')).toContain("Embed Step 14.5's vetted nonempty `documentation_section`");
|
||||||
|
});
|
||||||
|
|
||||||
|
test('a missing installed document-release section blocks before launch', () => {
|
||||||
|
// ci-36709485593-1-eval-slices-2 ship-docsync-missing-asset: audit-scope.md was absent, but the parent saw the
|
||||||
|
// SKILL.md "Ship-owned documentation mode" heading, read the gate as satisfied and dispatched.
|
||||||
|
const gate = read('ship/sections/documentation.md.tmpl').replace(/\s+/g, ' ');
|
||||||
|
expect(gate).toContain('full audit-scope/release-body content, linked as sections or inlined for external hosts. A missing section or old `Ship-owned documentation mode` blocks before launch; never substitute.');
|
||||||
|
});
|
||||||
|
|
||||||
test('nested authored discovery and standalone protections survive', () => {
|
test('nested authored discovery and standalone protections survive', () => {
|
||||||
const skill = read('document-release/SKILL.md.tmpl') + read('document-release/sections/audit-scope.md.tmpl');
|
const skill = read('document-release/SKILL.md.tmpl') + read('document-release/sections/audit-scope.md.tmpl');
|
||||||
expect(skill).not.toContain('find . -maxdepth 2');
|
expect(skill).not.toContain('find . -maxdepth 2');
|
||||||
|
|||||||
Reference in new issue
Block a user