CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-01 02:40:47 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix(test): the two expect(true) paid stubs become test.todo

Browse Source 
skill-e2e-spec-execute (600s budget) and skill-llm-eval-spec (300s)
reported PASS on every periodic run while asserting nothing. Deleting
them would remove the periodic-tier selector surface they exist to
register (diff-based selection for spec/ changes), so they become
test.todo — reported as todo/skip, never pass — with the v1.1
implementation specs kept in-file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-29 04:49:33 +00:00
co-authored by Claude Fable 5
parent e9643e131f
commit 9eaf15564c
2 changed files with 24 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/skill-llm-eval-spec.test.ts
+14 -26
View File
@@ -13,35 +13,23 @@
* Phase 3 fallback path).
*/
import { describe, test, expect } from 'bun:test';
import * as fs from 'fs';
import * as path from 'path';
import { describe, test } from 'bun:test';
const evalsEnabled = !!process.env.EVALS;
const describeEval = evalsEnabled ? describe : describe.skip;
const ROOT = path.resolve(import.meta.dir, '..');
describeEval('/spec LLM-judge eval (periodic)', () => {
test('spec body scores >= 8/10 against 14-standard rubric on fixture request', async () => {
// Sanity: required files exist for the eval.
expect(fs.existsSync(path.join(ROOT, 'spec', 'SKILL.md.tmpl'))).toBe(true);
// Full LLM-judge run lives in a follow-up. This file registers the
// periodic-tier surface so the diff-based selector picks it up when
// spec/ changes. Deterministic invariants are gate-tier; the LLM-judge
// is for measuring authored-spec quality, which is non-deterministic
// by nature.
//
// Expected v1.1 implementation:
// 1. Pick fixture prompt from test/fixtures/spec/vague-bug.md
// 2. Spawn `claude -p` with /spec loaded, send the prompt + role-play
// five Phase 1 answers (from test/fixtures/spec/vague-bug-answers.json)
// 3. Capture final spec body
// 4. Dispatch to Claude judge with prompt encoding the 14 Quality
// Standards from spec/SKILL.md.tmpl
// 5. Assert numeric score >= 8
expect(true).toBe(true);
}, 300_000);
// test.todo, not expect(true): the placeholder reported PASS on every
// run while asserting nothing — a lying green with a 300s budget. The
// file stays as the periodic-tier selector surface for spec/ changes.
//
// Expected v1.1 implementation:
// 1. Pick fixture prompt from test/fixtures/spec/vague-bug.md
// 2. Spawn `claude -p` with /spec loaded, send the prompt + role-play
// five Phase 1 answers (from test/fixtures/spec/vague-bug-answers.json)
// 3. Capture final spec body
// 4. Dispatch to Claude judge with prompt encoding the 14 Quality
// Standards from spec/SKILL.md.tmpl
// 5. Assert numeric score >= 8
test.todo('spec body scores >= 8/10 against 14-standard rubric on fixture request');
});