mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-11 07:29:00 +02:00
Garry Tan
8229fdc489
The Codex E2E runner copies auth.json only (operator plugins, MCP servers, rules, and skills no longer leak into hermetic evals), pins CODEX_HOME to the temp dir, and supports per-run model, TOML overrides, and --ignore-user-config. New periodic E2E installs the FULL generated investigate skill on gpt-5.6-sol against a planted one-line bug with decoy TODOs: the fix must land inside the boundary (untracked files counted via git status --porcelain), decoys stay byte-identical, the regression oracle survives unweakened, nothing gets committed, all within 30 tool calls. The shared .agents tree is snapshotted and restored exactly in beforeAll; fixture commits disable gpg signing. Wired into the periodic CI matrix, paid-shard globs, eval scripts, touchfiles/E2E_TIERS (codex-sol-scope-termination), and diff-based selection. Real-file periodic-tier classification pins both codex E2Es out of the gate tier. Free-tier test proves an explicit --model overrides the host default through the real generation CLI.
27 lines
1.0 KiB
TypeScript
27 lines
1.0 KiB
TypeScript
/**
|
|
* The ONE definition of which test files are paid (API spend, external
|
|
* services, e2e harnesses). package.json's test:gate/test:evals globs, the
|
|
* free-suite exclusion in scripts/test-free-shards.ts, and the sharded paid
|
|
* runner in scripts/test-paid-shards.ts all derive from this list — a file
|
|
* added to one and not the others either burns money in the free suite or
|
|
* silently never runs in the paid tier.
|
|
*/
|
|
|
|
import { matchGlob } from './touchfiles';
|
|
|
|
/** The exact globs package.json's `test:gate` passes to `bun test`. */
|
|
export const PAID_TEST_GLOBS = [
|
|
'test/skill-llm-eval.test.ts',
|
|
'test/skill-e2e-*.test.ts',
|
|
'test/skill-routing-e2e.test.ts',
|
|
'test/codex-e2e.test.ts',
|
|
'test/codex-e2e-sol-scope.test.ts',
|
|
'test/gemini-e2e.test.ts',
|
|
] as const;
|
|
|
|
/** True when a repo-relative path (either slash style) is a paid test file. */
|
|
export function isPaidTestFile(relativePath: string): boolean {
|
|
const normalized = relativePath.replace(/\\/g, '/');
|
|
return PAID_TEST_GLOBS.some((glob) => matchGlob(normalized, glob));
|
|
}
|