CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-07-12 18:56:35 +02:00
Code Issues Packages Projects Releases Wiki Activity

test: add AskUserQuestion format regression eval for plan reviews

Browse Source 
Four-case periodic-tier eval that captures the verbatim AskUserQuestion
text /plan-ceo-review and /plan-eng-review produce, then asserts the
format rule is honored: RECOMMENDATION always, Completeness: N/10 only
on coverage-differentiated options, and an explicit "options differ in
kind" note on kind-differentiated options.

Cases:
- plan-ceo-review mode selection (kind-differentiated)
- plan-ceo-review approach menu (coverage-differentiated)
- plan-eng-review per-issue coverage decision
- plan-eng-review per-issue architectural choice (kind-differentiated)

Classified periodic because behavior depends on Opus non-determinism —
gate-tier would flake and block merges.

Test harness instructs the agent to write its would-be AskUserQuestion
text to $OUT_FILE rather than invoke a real tool (MCP AskUserQuestion
isn't wired in the test subprocess). Regex predicates then validate
the captured content.

Cost: ~$2 per full run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-04-22 01:10:35 -07:00
parent e23ff280a1
commit 6a46b9099f
3 changed files with 314 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/touchfiles.test.ts
+4 -2
View File
@@ -83,8 +83,10 @@ describe('selectTests', () => {
expect(result.selected).toContain('plan-ceo-review-expansion-energy');
expect(result.selected).toContain('autoplan-core');
expect(result.selected).toContain('codex-offered-ceo-review');
expect(result.selected.length).toBe(6);
expect(result.skipped.length).toBe(Object.keys(E2E_TOUCHFILES).length - 6);
expect(result.selected).toContain('plan-ceo-review-format-mode');
expect(result.selected).toContain('plan-ceo-review-format-approach');
expect(result.selected.length).toBe(8);
expect(result.skipped.length).toBe(Object.keys(E2E_TOUCHFILES).length - 8);
});
test('global touchfile triggers ALL tests', () => {