CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-13 16:38:56 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat: hermetic Codex runner hardening + Sol scope-termination E2E

Browse Source 
The Codex E2E runner copies auth.json only (operator plugins, MCP servers,
rules, and skills no longer leak into hermetic evals), pins CODEX_HOME to
the temp dir, and supports per-run model, TOML overrides, and
--ignore-user-config. New periodic E2E installs the FULL generated
investigate skill on gpt-5.6-sol against a planted one-line bug with decoy
TODOs: the fix must land inside the boundary (untracked files counted via
git status --porcelain), decoys stay byte-identical, the regression oracle
survives unweakened, nothing gets committed, all within 30 tool calls.
The shared .agents tree is snapshotted and restored exactly in beforeAll;
fixture commits disable gpg signing. Wired into the periodic CI matrix,
paid-shard globs, eval scripts, touchfiles/E2E_TIERS
(codex-sol-scope-termination), and diff-based selection. Real-file
periodic-tier classification pins both codex E2Es out of the gate tier.
Free-tier test proves an explicit --model overrides the host default
through the real generation CLI.
This commit is contained in:
Garry Tan
2026-08-18 13:11:15 -07:00
parent 39a60588ed
commit 8229fdc489
10 changed files with 346 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/gen-skill-docs.test.ts
+30
View File
@@ -2094,6 +2094,36 @@ describe('Codex generation (--host codex)', () => {
const codexContent = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-ship', 'SKILL.md'), 'utf-8');
expect(codexContent).not.toContain('Codex design voice');
});
// ─── Explicit --model override wins over the host default ────
// Without --model the codex host renders its defaultModel (gpt) — pinned by
// the golden test. This pins the OTHER direction through the real CLI:
// `./setup --host codex --model <id>` depends on it. Runs last in this
// describe and restores the host-default render before finishing.
test('explicit --model overrides the codex host default', () => {
try {
const override = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'codex', '--model', 'claude'], {
cwd: ROOT,
stdout: 'pipe',
stderr: 'pipe',
});
expect(override.exitCode).toBe(0);
const content = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-ship', 'SKILL.md'), 'utf-8');
expect(content).toContain('Model-Specific Behavioral Patch (claude)');
expect(content).toContain('MODEL_OVERLAY: claude');
} finally {
// Restore the host-default render — later tests and the host-config
// golden read this tree.
const restore = Bun.spawnSync(['bun', 'run', 'scripts/gen-skill-docs.ts', '--host', 'codex'], {
cwd: ROOT,
stdout: 'pipe',
stderr: 'pipe',
});
expect(restore.exitCode).toBe(0);
}
const restored = fs.readFileSync(path.join(AGENTS_DIR, 'gstack-ship', 'SKILL.md'), 'utf-8');
expect(restored).toContain('Model-Specific Behavioral Patch (gpt)');
});
});
// ─── Factory generation tests ────────────────────────────────