CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-06-17 07:10:12 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix: workflow judge re-appends body-carved sections after the marker slice

Browse Source 
runWorkflowJudge appended sections/*.md before slicing startMarker..endMarker.
That handles skills that moved their MARKERS into sections (plan-eng,
plan-design) but not document-release, which keeps its markers in the
skeleton and carved the workflow BODY (Steps 2-9 -> sections/release-body.md)
AFTER the endMarker — so the slice dropped it and the judge scored
completeness 2 ('Steps 2-9 are in an external file'). Now any carved section
the marker window excluded is re-appended, so the judge sees the full
workflow the agent executes. document-release: completeness 2->5, clarity
3->4. ship/plan-ceo/plan-eng/plan-design judges unchanged (their section
content is already inside the slice, so the head-dedup skips re-append).

Pre-existing since the v1.57.0.0 carve (#1907); surfaced now because
hermetic-env.ts is a global touchfile that selects every llm-judge test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-06-13 10:48:35 -07:00
parent 6d2274e0dc
commit c15f78924e
1 changed files with 15 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/skill-llm-eval.test.ts
+15 -1
View File
@@ -546,10 +546,13 @@ async function runWorkflowJudge(opts: {
// slice markers vanish from the skeleton and the judge scores empty content.
let content = fs.readFileSync(path.join(ROOT, opts.skillPath), 'utf-8');
const secDir = path.join(ROOT, path.dirname(opts.skillPath), 'sections');
const sectionBodies: string[] = [];
if (fs.existsSync(secDir)) {
for (const f of fs.readdirSync(secDir).sort()) {
if (f.endsWith('.md') && !f.endsWith('.md.tmpl')) {
content += '\n' + fs.readFileSync(path.join(secDir, f), 'utf-8');
const body = fs.readFileSync(path.join(secDir, f), 'utf-8');
sectionBodies.push(body);
content += '\n' + body;
}
}
}
@@ -565,6 +568,17 @@ async function runWorkflowJudge(opts: {
section = content.slice(startIdx);
}
// Two carve shapes exist. plan-eng/plan-design moved the MARKERS into the
// section files, so the slice above already reaches the carved content.
// document-release instead keeps its markers in the skeleton and carves the
// workflow BODY (Steps 2-9 → sections/release-body.md) AFTER the endMarker,
// so the marker slice drops it. Re-append any carved section the window
// excluded, so the judge always sees the full workflow the agent executes.
for (const body of sectionBodies) {
const head = body.trim().slice(0, 120);
if (head && !section.includes(head)) section += '\n' + body;
}
const scores = await callJudge<JudgeScore>(`You are evaluating the quality of ${opts.judgeContext} for an AI coding agent.
The agent reads this document to learn ${opts.judgeGoal}. It references external tools and files