Files
gstack/test/helpers/placeholder-allowlist.ts
Garry Tanandcapy-ai[bot] 4eff47c5cf v1.92.2.0 evals: no-gstack arms for the model panel, arm-benchmark, safety and routing evals (#3708)
* feat(eval-plan): results --check refuses a results table without its no-gstack rows

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(model-panel): a <model>#no-gstack arm per model, identity-keyed cells, one campaign, results.md (GST-468)

Each panel model also runs as <model>#no-gstack: the same fixture, ticket,
tools, limits, route and reservation, with no skill folder and no routing
line. The arm identity keys the ledger attempt, test name, row and summary;
candidate folders are named after the fixture project and the seed commit
has a neutral author. summarize reads one campaign (run id and PREREG.md
sha256) and refuses a duplicate cell; suggestionLine reads the skill arms
only, so the preregistered particle-1 line is unchanged. renderResults writes
results.md, which passes gstack-eval-plan results --check against the new
test/fixtures/model-panel/PREREG.md (supersedes GSTA-173 for later runs).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(arm-benchmark): no-gstack arm on a pinned model, 3 paired trials, both-order paired judge, spend ledger (GST-469)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(evals): no-gstack arm for the safety-rule and router evals (GST-470)

GSTACK_SAFETY_ARM and GSTACK_ROUTER_ARM take a third value, none: the same
fixture, tools and user request with no skill excerpt, rule text, step file
or installed skill. Under none every behavioral check is recorded as an
observation (none-arm-observations.jsonl) and never fails the case or writes
contract-violations.jsonl; harness preconditions still assert.

The four safety prompt builders (ship-evidence, codex-boundary,
codex-consult, design-risk) and the router fixtures move into helpers; the
rule, removed, current and before prompts stay byte-identical, proven
against reference copies of main's code. pair-agent and ios-demo refuse
none with the reason (NO_NONE_ARM). GSTACK_ROUTER_ARM now throws on an
unknown value instead of silently running current.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* chore: changelog fragment for the no-gstack arms in existing evals

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(arm-benchmark): arm-pair paired-judge calibration run (bar met 6/6, $7.22)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(paid-shards): keep loop-named panel and arm-benchmark cases selectable by --case

The lanes moved each cell's testName into helpers, so the static detector no
longer saw a computed registration and B4 failed for all eight cases. Same
runtime test names; the paid runs are unaffected.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(tests): keep FREE_FIXTURES and the placeholder allowlist current for the no-gstack arms

scripts/judge-calibration.ts is now an arm-benchmark paid touchfile, so it can no
longer be a free consumer of test/fixtures/judge-calibration/** (CI free-suite 8).
The eval-plan Step 5 fence names <prereg-path> and <results-path>; allowlist both
as PATH placeholders (CI free-suite 19).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(eval-plan): cover a complete table after a partial one, --json output and the results usage error

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(eval-plan): keep the Step 7 coverage test as reviewed

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* chore: name the changelog fragment after the branch slug

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: arm-benchmark project folder names and the arm-pair calibration corpus

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: keep the judge-calibration fixture README as reviewed

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs(skills): describe eval-plan results --check

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-10-31 22:41:36 +00:00

209 lines
16 KiB
TypeScript

/**
* Reviewed identifier placeholders for generated bash (CEO-12, DX-5, ENG-8).
*
* A `<placeholder>` the model fills in may appear in a command only when it is
* listed here. Free text (bodies, titles, messages, briefs, prompts, error
* output, reviewer or diff text) is never listed: it reaches a command only as
* the contents of a `mktemp` file the agent writes with its file-write tool.
*
* Each entry names the grammar a value must match before the skill uses it; a
* value that fails is not used. `quoted` entries may appear only inside single
* or double quotes (or a quoted heredoc), and their grammar excludes every
* character that can end or expand those quotes: `'`, `"`, backtick, `$`, `\`
* and newline. Every other entry may appear unquoted, so its grammar also
* excludes `;|&()<>?[]{}!#` and, unless it is a `list`, whitespace and `*`. test/generated-bash-placeholders
* .test.ts proves each grammar rejects those characters.
*/
/** `list`: a space-separated list where word splitting is intended (spaces, and `*` globs, allowed). */
export interface IdentifierGrammar { grammar: RegExp; what: string; quoted?: true; list?: true }
const NUM = /^[0-9]+$/;
const ID = /^[A-Za-z0-9][A-Za-z0-9._-]*$/;
/** Shell-safe subset of git ref names; the posting skills also run `git check-ref-format --branch`. */
const REF = /^[\p{L}\p{N}_][\p{L}\p{N}._/@+-]*$/u;
const PATH = /^[\p{L}\p{N}._/@+:,%=~-]+$/u;
/** Space-separated path lists (word splitting intended); `*` may glob. */
const PATHS = /^[\p{L}\p{N}._/@+:,%=~* -]+$/u;
/** Inside quotes only: anything that cannot end or expand the quotes. */
const QUOTED = /^[^'"`$\\\n]+$/;
const DATE = /^[0-9]{4}-[0-9]{2}-[0-9]{2}$/;
const NUMBER = /^-?[0-9]+(?:\.[0-9]+)?$/;
const URL = /^https?:\/\/[^'"`$\\\s<>]+$/;
const oneOf = (...alts: string[]) => new RegExp(`^(?:${alts.map(a => a.replace(/[.*+?^${}()|[\]\\/-]/g, '\\$&')).join('|')})$`);
export const IDENTIFIER_PLACEHOLDERS: Record<string, IdentifierGrammar> = {
// Enumerations: the value is one of the listed literals.
'<approval|clarification|routing|cherry-pick|feedback-loop>': { grammar: oneOf('approval', 'clarification', 'routing', 'cherry-pick', 'feedback-loop'), what: 'question category' },
'<one-way|two-way>': { grammar: oneOf('one-way', 'two-way'), what: 'door type' },
'<never-ask|always-ask|ask-only-for-one-way>': { grammar: oneOf('never-ask', 'always-ask', 'ask-only-for-one-way'), what: 'question preference' },
'<pref>': { grammar: oneOf('never-ask', 'always-ask', 'ask-only-for-one-way'), what: 'question preference' },
'<clean|flagged>': { grammar: oneOf('clean', 'flagged'), what: 'semantic review outcome' },
'<N|null>': { grammar: /^(?:[0-9]+(?:\.[0-9]+)?|null)$/, what: 'health score or null' },
'<platform>': { grammar: oneOf('github', 'gitlab', 'unknown'), what: 'detected git platform' },
'<effort-level>': { grammar: /^[a-z]*$/, what: "the request's /codex --effort level, or empty for none (the probe validates the level)", quoted: true },
// Ids, keys and numbers.
'<id>': { grammar: ID, what: 'question or record id (kebab-case)' },
'<key>': { grammar: ID, what: 'option key' },
'<summary-slug>': { grammar: ID, what: 'question summary as a kebab-case slug' },
'<comment-id>': { grammar: NUM, what: 'GitHub comment id from the fetched JSON' },
'<opened-at>': { grammar: NUM, what: 'EARLY_PR_OPENED_AT epoch seconds printed by gstack-greptile-early open' },
'<pr-number>': { grammar: NUM, what: 'PR/MR number from the open-PR lookup or the early-PR create output' },
'<new-version>': { grammar: /^[0-9]+(?:\.[0-9]+)*$/, what: 'NEW_VERSION from the version bump (dot-separated digits)' },
'<issue-number>': { grammar: /^[0-9]*$/, what: 'filed issue number, or empty when none was filed' },
'<check-number>': { grammar: NUM, what: 'canary check number' },
// /eval-plan (plan E1): the pilot's ledger inputs.
'<cap-usd>': { grammar: NUMBER, what: 'spend cap in USD' },
'<total-items>': { grammar: NUM, what: 'items in the full eval run' },
'<workers>': { grammar: NUM, what: 'concurrent eval workers' },
'<attempt-id>': { grammar: ID, what: 'PILOT_ATTEMPT id printed by pilot start' },
'<usd|unknown>': { grammar: /^(?:[0-9]+(?:\.[0-9]+)?|unknown)$/, what: 'actual cost in USD from the provider usage field, or unknown' },
'<run-id>': { grammar: NUM, what: 'GitHub Actions run id' },
'<PID>': { grammar: NUM, what: 'process id printed by an earlier block' },
'<retained-owner-token>': { grammar: ID, what: 'freeze owner token printed by acquire' },
'<RUN_ID>': { grammar: ID, what: 'run id printed by Setup' },
'<RUN_ID from Setup>': { grammar: ID, what: 'run id printed by Setup' },
'<PHASE>': { grammar: oneOf('ceo', 'design', 'dx', 'eng'), what: 'autoplan phase' },
'<VARIANT>': { grammar: /^[A-Z]$/, what: 'variant letter' },
'<SCREEN>': { grammar: ID, what: 'screen name slug' },
'<screen-name>': { grammar: ID, what: 'screen name slug' },
'<page-name>': { grammar: ID, what: 'page name slug' },
'<name>': { grammar: ID, what: 'slug' },
'<slug>': { grammar: ID, what: 'slug' },
'<dimension>': { grammar: ID, what: 'design dimension slug' },
'<dim>': { grammar: ID, what: 'profile dimension name' },
// /design-motion (GSTA-92 plan item 15): capture and export arguments the skill's plan fixes.
'<frame-size>': { grammar: /^[0-9]+x[0-9]+$/, what: 'design frame size in CSS pixels, WxH' },
'<cycle-seconds>': { grammar: NUMBER, what: 'loop cycle length in seconds' },
'<app|cards>': { grammar: oneOf('app', 'cards'), what: 'motion layout: joined app surfaces or separate cards' },
'<hex-color>': { grammar: /^#(?:[0-9A-Fa-f]{6}|[0-9A-Fa-f]{8})$/, what: 'page background colour as hex', quoted: true },
'<figma-file-key>': { grammar: ID, what: 'Figma file key from the link' },
'<figma-node-id>': { grammar: /^[0-9]+[:-][0-9]+$/, what: 'Figma node id from the link' },
'<new_value>': { grammar: NUMBER, what: 'profile dimension value' },
'<Q1_VALUE>': { grammar: NUMBER, what: 'profile answer value' },
'<Q2_VALUE>': { grammar: NUMBER, what: 'profile answer value' },
'<Q3_VALUE>': { grammar: NUMBER, what: 'profile answer value' },
'<Q4_VALUE>': { grammar: NUMBER, what: 'profile answer value' },
'<Q5_VALUE>': { grammar: NUMBER, what: 'profile answer value' },
'<actual-harness>': { grammar: oneOf('claude', 'codex'), what: 'harness name' },
'<suite>': { grammar: ID, what: 'eval suite name' },
'<picked-models>': { grammar: /^[A-Za-z0-9._:,-]+$/, what: 'comma-separated model ids' },
'<github-username>': { grammar: /^[A-Za-z0-9](?:[A-Za-z0-9-]{0,38})$/, what: 'GitHub login' },
'<gitlab-username>': { grammar: /^[A-Za-z0-9_][A-Za-z0-9_.-]*$/, what: 'GitLab username' },
// Dates.
'<today>': { grammar: DATE, what: 'YYYY-MM-DD' },
'<start_date>': { grammar: DATE, what: 'YYYY-MM-DD' },
'<since>': { grammar: /^[0-9]+[dhwm]$|^[0-9]{4}-[0-9]{2}-[0-9]{2}$/, what: 'retro window (7d, 24h, 2w) or date' },
'<window>': { grammar: /^[0-9]+[dhwm]$/, what: 'retro window (7d, 24h, 2w)' },
'<window-days>': { grammar: NUM, what: 'retro window in whole days' },
// Git refs. Branch names from remote data travel as data (ENG-8), never through this list.
'<base>': { grammar: REF, what: 'base branch detected by the skill' },
'<default>': { grammar: REF, what: 'default branch detected by the skill' },
'<diff-base>': { grammar: REF, what: 'diff base ref detected by the skill' },
'<detected-base-branch>': { grammar: REF, what: 'base branch detected by the skill' },
'<branch-name>': { grammar: REF, what: 'current branch (git branch --show-current)' },
'<prior-review-commit>': { grammar: /^[0-9a-f]{7,40}$/, what: 'commit sha' },
// Paths printed by earlier gstack blocks or chosen by the skill.
'<body-file-name>': { grammar: ID, what: 'basename of the mktemp body file printed by the previous block' },
'<title-file-name>': { grammar: ID, what: 'basename of the mktemp title file printed by the previous block' },
'<gate-file-name>': { grammar: ID, what: 'basename of the mktemp gate-command file printed by the previous block (/eval-plan price-gate)' },
'<reply-file-name>': { grammar: ID, what: 'basename of the mktemp reply file printed by the previous block' },
'<redact-file-name>': { grammar: ID, what: 'basename of the mktemp draft file printed by the free-text block' },
'<approach-file-name>': { grammar: ID, what: 'basename of the mktemp approach file printed by the free-text block' },
'<greptile-dir>': { grammar: PATH, what: 'mktemp directory printed by the Greptile fetch block' },
'<SNAPSHOT_TOOL>': { grammar: QUOTED, what: 'snapshot tool path printed by autoplan', quoted: true },
'<SOURCE_PLAN>': { grammar: QUOTED, what: 'plan path', quoted: true },
'<ACTIVE_PLAN>': { grammar: QUOTED, what: 'plan path', quoted: true },
'<RESTORE_PATH>': { grammar: QUOTED, what: 'restore path printed by autoplan', quoted: true },
'<methodologyPath>': { grammar: QUOTED, what: 'methodology path printed by autoplan', quoted: true },
'<CEO_STEP0_CHECKPOINT>': { grammar: QUOTED, what: 'checkpoint path printed by autoplan', quoted: true },
'<AMENDMENT_CHECKPOINT>': { grammar: QUOTED, what: 'checkpoint path printed by autoplan', quoted: true },
'<prepared-prompt-file>': { grammar: QUOTED, what: 'prompt file path printed by the prepare step', quoted: true },
'<run-dir>': { grammar: QUOTED, what: 'run directory printed by an earlier block', quoted: true },
'<ASIDE_DIR>': { grammar: QUOTED, what: 'Aside output directory printed by an earlier block', quoted: true },
'<ASIDE_DIR or $_TMP>': { grammar: QUOTED, what: 'Aside output directory printed by an earlier block', quoted: true },
'<REPORT_DIR from Setup>': { grammar: QUOTED, what: 'report directory printed by Setup', quoted: true },
'<the FILE path printed above>': { grammar: QUOTED, what: 'file path printed by the previous block', quoted: true },
'<install dir from Step 2>': { grammar: QUOTED, what: 'install directory printed by Step 2', quoted: true },
'<APPROVED path>': { grammar: QUOTED, what: 'approved design path', quoted: true },
'<first path from the printed saved list>': { grammar: QUOTED, what: 'image path printed by the design tool', quoted: true },
'<design-path>': { grammar: QUOTED, what: 'design doc path', quoted: true },
'<review-directory>': { grammar: QUOTED, what: 'review directory printed by mktemp', quoted: true },
'<round-1.json>': { grammar: QUOTED, what: 'round file path', quoted: true },
'<round-1.json if present>': { grammar: QUOTED, what: 'round file path', quoted: true },
'<round-2.json if present>': { grammar: QUOTED, what: 'round file path', quoted: true },
'<round-3.json if present>': { grammar: QUOTED, what: 'round file path', quoted: true },
'<source-dir>': { grammar: QUOTED, what: 'app source directory', quoted: true },
'<user-provided-path>': { grammar: QUOTED, what: 'directory the user named', quoted: true },
'<detected-directory>': { grammar: QUOTED, what: 'directory detected by the skill', quoted: true },
'<staged>': { grammar: QUOTED, what: 'staged HTML path', quoted: true },
'<stagedDir>': { grammar: QUOTED, what: 'staged directory printed by an earlier block', quoted: true },
'<outdir>': { grammar: PATH, what: 'output directory' },
'<path>': { grammar: PATH, what: 'repository path from discovery' },
'<plan-path>': { grammar: PATH, what: 'bound plan file path' },
'<prereg-path>': { grammar: PATH, what: 'preregistration file path' },
'<results-path>': { grammar: PATH, what: 'results table file path' },
'<sketch-dir>': { grammar: PATH, what: 'sketch directory printed by an earlier block' },
'<path-to-finalized.html>': { grammar: PATH, what: 'finalized HTML path' },
'<path-to-page.html>': { grammar: PATH, what: 'saved page HTML path' },
'<finalized.html>': { grammar: PATH, what: 'finalized HTML path' },
'<motion-dir>': { grammar: PATH, what: 'run directory printed by the /design-motion Step 0 MOTION_DIR line' },
'<approved-variant.png>': { grammar: PATH, what: 'approved variant image path' },
'<failing-test-file>': { grammar: PATH, what: 'test file path' },
'<source-file-under-test>': { grammar: PATH, what: 'source file path' },
'<affected-files>': { grammar: PATHS, what: 'space-separated file paths', list: true },
'<only-changed-files>': { grammar: PATHS, what: 'space-separated file paths', list: true },
'<only-verified-source-and-regression-files>': { grammar: PATHS, what: 'space-separated file paths', list: true },
'<scope>': { grammar: PATHS, what: 'path or glob', list: true },
'<project eval command>': { grammar: /^[A-Za-z0-9._/@+:,%=~ -]+$/, what: 'the eval command the project documents (words and flags only)', list: true },
// Quoted-only values: URLs, selectors, search words and commands.
'<url>': { grammar: URL, what: 'http(s) URL', quoted: true },
'<page-url>': { grammar: URL, what: 'http(s) URL', quoted: true },
'<base-url>': { grammar: URL, what: 'http(s) URL', quoted: true },
'<target-url>': { grammar: URL, what: 'http(s) URL', quoted: true },
'<TARGET_URL>': { grammar: URL, what: 'http(s) URL', quoted: true },
'<site>': { grammar: URL, what: 'http(s) URL', quoted: true },
'<selector>': { grammar: QUOTED, what: 'CSS selector', quoted: true },
'<row-selector>': { grammar: QUOTED, what: 'CSS selector', quoted: true },
'<title-selector>': { grammar: QUOTED, what: 'CSS selector', quoted: true },
'<your-keyword>': { grammar: QUOTED, what: 'search words', quoted: true },
'<keywords>': { grammar: QUOTED, what: 'search words', quoted: true },
'<keyword1>': { grammar: QUOTED, what: 'search word', quoted: true },
'<keyword2>': { grammar: QUOTED, what: 'search word', quoted: true },
'<keyword3>': { grammar: QUOTED, what: 'search word', quoted: true },
'<tests>': { grammar: QUOTED, what: 'test command', quoted: true },
'<vitest>': { grammar: QUOTED, what: 'test command', quoted: true },
// External-host renders: Claude Code outside voice and the gbrain page save.
'<gstack-runtime-root>': { grammar: QUOTED, what: 'installed runtime path', quoted: true },
'<fresh-or-resume>': { grammar: oneOf('fresh', 'resume'), what: 'Claude Code session mode' },
'<feature-slug>': { grammar: ID, what: 'feature slug for the gbrain page' },
'<page-file>': { grammar: QUOTED, what: 'mktemp page file path printed by the previous command', quoted: true },
// A3 group 1 (Aside prompts, design briefs, design approval feedback): mktemp basenames.
'<prompt-file-name>': { grammar: ID, what: 'basename of the mktemp prompt file (Aside, Codex, benchmark) printed by the previous block' },
'<brief-file-name>': { grammar: ID, what: 'basename of the mktemp design brief file printed by the free-text block' },
'<feedback-file-name>': { grammar: ID, what: 'basename of the mktemp design feedback file printed by the previous block' },
// A3 group 2 (Codex prompts and misc): mktemp basenames, mode switches,
// the benchmark prompt path and the sync-gbrain orchestrator flags.
'<focus-file-name>': { grammar: ID, what: 'basename of the mktemp Codex review focus file printed by the free-text block' },
'<receipt line>': { grammar: /^OFFICE_HOURS_VERDICT round=[1-3] sha256=[0-9a-f]{64} path=\/[^'"`$\\\n]+$/, what: 'office-hours reviewer verdict receipt (one line)', quoted: true },
'<body-top-file-name>': { grammar: ID, what: 'basename of the mktemp PR body draft (through ## Documentation) printed by the free-text block' },
'<body-rest-file-name>': { grammar: ID, what: 'basename of the mktemp PR body draft (after the documentation section) printed by the free-text block' },
'<rejected-file-name>': { grammar: ID, what: 'basename of the mktemp rejected-test list printed by the free-text block' },
'<new|resume>': { grammar: oneOf('new', 'resume'), what: 'Codex consult session mode' },
'<prompt-path>': { grammar: QUOTED, what: 'benchmark prompt file path', quoted: true },
'<user-args>': {
grammar: /^(?:(?:--(?:incremental|full|dry-run|quiet|no-code|no-memory|no-brain-sync|code-only|dream|no-dream|allow-reclone|prune-gone-worktrees)|--sources [a-z][a-z0-9_,-]*)(?: (?=-)|$))*$/,
what: 'gstack-gbrain-sync flags (space-separated; --sources takes one comma-separated type list)',
list: true,
},
};