mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-02 15:04:37 +01:00
* feat(eval-plan): results --check refuses a results table without its no-gstack rows Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(model-panel): a <model>#no-gstack arm per model, identity-keyed cells, one campaign, results.md (GST-468) Each panel model also runs as <model>#no-gstack: the same fixture, ticket, tools, limits, route and reservation, with no skill folder and no routing line. The arm identity keys the ledger attempt, test name, row and summary; candidate folders are named after the fixture project and the seed commit has a neutral author. summarize reads one campaign (run id and PREREG.md sha256) and refuses a duplicate cell; suggestionLine reads the skill arms only, so the preregistered particle-1 line is unchanged. renderResults writes results.md, which passes gstack-eval-plan results --check against the new test/fixtures/model-panel/PREREG.md (supersedes GSTA-173 for later runs). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(arm-benchmark): no-gstack arm on a pinned model, 3 paired trials, both-order paired judge, spend ledger (GST-469) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(evals): no-gstack arm for the safety-rule and router evals (GST-470) GSTACK_SAFETY_ARM and GSTACK_ROUTER_ARM take a third value, none: the same fixture, tools and user request with no skill excerpt, rule text, step file or installed skill. Under none every behavioral check is recorded as an observation (none-arm-observations.jsonl) and never fails the case or writes contract-violations.jsonl; harness preconditions still assert. The four safety prompt builders (ship-evidence, codex-boundary, codex-consult, design-risk) and the router fixtures move into helpers; the rule, removed, current and before prompts stay byte-identical, proven against reference copies of main's code. pair-agent and ios-demo refuse none with the reason (NO_NONE_ARM). GSTACK_ROUTER_ARM now throws on an unknown value instead of silently running current. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore: changelog fragment for the no-gstack arms in existing evals Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(arm-benchmark): arm-pair paired-judge calibration run (bar met 6/6, $7.22) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(paid-shards): keep loop-named panel and arm-benchmark cases selectable by --case The lanes moved each cell's testName into helpers, so the static detector no longer saw a computed registration and B4 failed for all eight cases. Same runtime test names; the paid runs are unaffected. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(tests): keep FREE_FIXTURES and the placeholder allowlist current for the no-gstack arms scripts/judge-calibration.ts is now an arm-benchmark paid touchfile, so it can no longer be a free consumer of test/fixtures/judge-calibration/** (CI free-suite 8). The eval-plan Step 5 fence names <prereg-path> and <results-path>; allowlist both as PATH placeholders (CI free-suite 19). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(eval-plan): cover a complete table after a partial one, --json output and the results usage error Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(eval-plan): keep the Step 7 coverage test as reviewed Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore: name the changelog fragment after the branch slug Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: arm-benchmark project folder names and the arm-pair calibration corpus Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: keep the judge-calibration fixture README as reviewed Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(skills): describe eval-plan results --check Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
209 lines
16 KiB
TypeScript
209 lines
16 KiB
TypeScript
/**
|
|
* Reviewed identifier placeholders for generated bash (CEO-12, DX-5, ENG-8).
|
|
*
|
|
* A `<placeholder>` the model fills in may appear in a command only when it is
|
|
* listed here. Free text (bodies, titles, messages, briefs, prompts, error
|
|
* output, reviewer or diff text) is never listed: it reaches a command only as
|
|
* the contents of a `mktemp` file the agent writes with its file-write tool.
|
|
*
|
|
* Each entry names the grammar a value must match before the skill uses it; a
|
|
* value that fails is not used. `quoted` entries may appear only inside single
|
|
* or double quotes (or a quoted heredoc), and their grammar excludes every
|
|
* character that can end or expand those quotes: `'`, `"`, backtick, `$`, `\`
|
|
* and newline. Every other entry may appear unquoted, so its grammar also
|
|
* excludes `;|&()<>?[]{}!#` and, unless it is a `list`, whitespace and `*`. test/generated-bash-placeholders
|
|
* .test.ts proves each grammar rejects those characters.
|
|
*/
|
|
|
|
/** `list`: a space-separated list where word splitting is intended (spaces, and `*` globs, allowed). */
|
|
export interface IdentifierGrammar { grammar: RegExp; what: string; quoted?: true; list?: true }
|
|
|
|
const NUM = /^[0-9]+$/;
|
|
const ID = /^[A-Za-z0-9][A-Za-z0-9._-]*$/;
|
|
/** Shell-safe subset of git ref names; the posting skills also run `git check-ref-format --branch`. */
|
|
const REF = /^[\p{L}\p{N}_][\p{L}\p{N}._/@+-]*$/u;
|
|
const PATH = /^[\p{L}\p{N}._/@+:,%=~-]+$/u;
|
|
/** Space-separated path lists (word splitting intended); `*` may glob. */
|
|
const PATHS = /^[\p{L}\p{N}._/@+:,%=~* -]+$/u;
|
|
/** Inside quotes only: anything that cannot end or expand the quotes. */
|
|
const QUOTED = /^[^'"`$\\\n]+$/;
|
|
const DATE = /^[0-9]{4}-[0-9]{2}-[0-9]{2}$/;
|
|
const NUMBER = /^-?[0-9]+(?:\.[0-9]+)?$/;
|
|
const URL = /^https?:\/\/[^'"`$\\\s<>]+$/;
|
|
const oneOf = (...alts: string[]) => new RegExp(`^(?:${alts.map(a => a.replace(/[.*+?^${}()|[\]\\/-]/g, '\\$&')).join('|')})$`);
|
|
|
|
export const IDENTIFIER_PLACEHOLDERS: Record<string, IdentifierGrammar> = {
|
|
// Enumerations: the value is one of the listed literals.
|
|
'<approval|clarification|routing|cherry-pick|feedback-loop>': { grammar: oneOf('approval', 'clarification', 'routing', 'cherry-pick', 'feedback-loop'), what: 'question category' },
|
|
'<one-way|two-way>': { grammar: oneOf('one-way', 'two-way'), what: 'door type' },
|
|
'<never-ask|always-ask|ask-only-for-one-way>': { grammar: oneOf('never-ask', 'always-ask', 'ask-only-for-one-way'), what: 'question preference' },
|
|
'<pref>': { grammar: oneOf('never-ask', 'always-ask', 'ask-only-for-one-way'), what: 'question preference' },
|
|
'<clean|flagged>': { grammar: oneOf('clean', 'flagged'), what: 'semantic review outcome' },
|
|
'<N|null>': { grammar: /^(?:[0-9]+(?:\.[0-9]+)?|null)$/, what: 'health score or null' },
|
|
'<platform>': { grammar: oneOf('github', 'gitlab', 'unknown'), what: 'detected git platform' },
|
|
'<effort-level>': { grammar: /^[a-z]*$/, what: "the request's /codex --effort level, or empty for none (the probe validates the level)", quoted: true },
|
|
|
|
// Ids, keys and numbers.
|
|
'<id>': { grammar: ID, what: 'question or record id (kebab-case)' },
|
|
'<key>': { grammar: ID, what: 'option key' },
|
|
'<summary-slug>': { grammar: ID, what: 'question summary as a kebab-case slug' },
|
|
'<comment-id>': { grammar: NUM, what: 'GitHub comment id from the fetched JSON' },
|
|
'<opened-at>': { grammar: NUM, what: 'EARLY_PR_OPENED_AT epoch seconds printed by gstack-greptile-early open' },
|
|
'<pr-number>': { grammar: NUM, what: 'PR/MR number from the open-PR lookup or the early-PR create output' },
|
|
'<new-version>': { grammar: /^[0-9]+(?:\.[0-9]+)*$/, what: 'NEW_VERSION from the version bump (dot-separated digits)' },
|
|
'<issue-number>': { grammar: /^[0-9]*$/, what: 'filed issue number, or empty when none was filed' },
|
|
'<check-number>': { grammar: NUM, what: 'canary check number' },
|
|
// /eval-plan (plan E1): the pilot's ledger inputs.
|
|
'<cap-usd>': { grammar: NUMBER, what: 'spend cap in USD' },
|
|
'<total-items>': { grammar: NUM, what: 'items in the full eval run' },
|
|
'<workers>': { grammar: NUM, what: 'concurrent eval workers' },
|
|
'<attempt-id>': { grammar: ID, what: 'PILOT_ATTEMPT id printed by pilot start' },
|
|
'<usd|unknown>': { grammar: /^(?:[0-9]+(?:\.[0-9]+)?|unknown)$/, what: 'actual cost in USD from the provider usage field, or unknown' },
|
|
'<run-id>': { grammar: NUM, what: 'GitHub Actions run id' },
|
|
'<PID>': { grammar: NUM, what: 'process id printed by an earlier block' },
|
|
'<retained-owner-token>': { grammar: ID, what: 'freeze owner token printed by acquire' },
|
|
'<RUN_ID>': { grammar: ID, what: 'run id printed by Setup' },
|
|
'<RUN_ID from Setup>': { grammar: ID, what: 'run id printed by Setup' },
|
|
'<PHASE>': { grammar: oneOf('ceo', 'design', 'dx', 'eng'), what: 'autoplan phase' },
|
|
'<VARIANT>': { grammar: /^[A-Z]$/, what: 'variant letter' },
|
|
'<SCREEN>': { grammar: ID, what: 'screen name slug' },
|
|
'<screen-name>': { grammar: ID, what: 'screen name slug' },
|
|
'<page-name>': { grammar: ID, what: 'page name slug' },
|
|
'<name>': { grammar: ID, what: 'slug' },
|
|
'<slug>': { grammar: ID, what: 'slug' },
|
|
'<dimension>': { grammar: ID, what: 'design dimension slug' },
|
|
'<dim>': { grammar: ID, what: 'profile dimension name' },
|
|
// /design-motion (GSTA-92 plan item 15): capture and export arguments the skill's plan fixes.
|
|
'<frame-size>': { grammar: /^[0-9]+x[0-9]+$/, what: 'design frame size in CSS pixels, WxH' },
|
|
'<cycle-seconds>': { grammar: NUMBER, what: 'loop cycle length in seconds' },
|
|
'<app|cards>': { grammar: oneOf('app', 'cards'), what: 'motion layout: joined app surfaces or separate cards' },
|
|
'<hex-color>': { grammar: /^#(?:[0-9A-Fa-f]{6}|[0-9A-Fa-f]{8})$/, what: 'page background colour as hex', quoted: true },
|
|
'<figma-file-key>': { grammar: ID, what: 'Figma file key from the link' },
|
|
'<figma-node-id>': { grammar: /^[0-9]+[:-][0-9]+$/, what: 'Figma node id from the link' },
|
|
'<new_value>': { grammar: NUMBER, what: 'profile dimension value' },
|
|
'<Q1_VALUE>': { grammar: NUMBER, what: 'profile answer value' },
|
|
'<Q2_VALUE>': { grammar: NUMBER, what: 'profile answer value' },
|
|
'<Q3_VALUE>': { grammar: NUMBER, what: 'profile answer value' },
|
|
'<Q4_VALUE>': { grammar: NUMBER, what: 'profile answer value' },
|
|
'<Q5_VALUE>': { grammar: NUMBER, what: 'profile answer value' },
|
|
'<actual-harness>': { grammar: oneOf('claude', 'codex'), what: 'harness name' },
|
|
'<suite>': { grammar: ID, what: 'eval suite name' },
|
|
'<picked-models>': { grammar: /^[A-Za-z0-9._:,-]+$/, what: 'comma-separated model ids' },
|
|
'<github-username>': { grammar: /^[A-Za-z0-9](?:[A-Za-z0-9-]{0,38})$/, what: 'GitHub login' },
|
|
'<gitlab-username>': { grammar: /^[A-Za-z0-9_][A-Za-z0-9_.-]*$/, what: 'GitLab username' },
|
|
|
|
// Dates.
|
|
'<today>': { grammar: DATE, what: 'YYYY-MM-DD' },
|
|
'<start_date>': { grammar: DATE, what: 'YYYY-MM-DD' },
|
|
'<since>': { grammar: /^[0-9]+[dhwm]$|^[0-9]{4}-[0-9]{2}-[0-9]{2}$/, what: 'retro window (7d, 24h, 2w) or date' },
|
|
'<window>': { grammar: /^[0-9]+[dhwm]$/, what: 'retro window (7d, 24h, 2w)' },
|
|
'<window-days>': { grammar: NUM, what: 'retro window in whole days' },
|
|
|
|
// Git refs. Branch names from remote data travel as data (ENG-8), never through this list.
|
|
'<base>': { grammar: REF, what: 'base branch detected by the skill' },
|
|
'<default>': { grammar: REF, what: 'default branch detected by the skill' },
|
|
'<diff-base>': { grammar: REF, what: 'diff base ref detected by the skill' },
|
|
'<detected-base-branch>': { grammar: REF, what: 'base branch detected by the skill' },
|
|
'<branch-name>': { grammar: REF, what: 'current branch (git branch --show-current)' },
|
|
'<prior-review-commit>': { grammar: /^[0-9a-f]{7,40}$/, what: 'commit sha' },
|
|
|
|
// Paths printed by earlier gstack blocks or chosen by the skill.
|
|
'<body-file-name>': { grammar: ID, what: 'basename of the mktemp body file printed by the previous block' },
|
|
'<title-file-name>': { grammar: ID, what: 'basename of the mktemp title file printed by the previous block' },
|
|
'<gate-file-name>': { grammar: ID, what: 'basename of the mktemp gate-command file printed by the previous block (/eval-plan price-gate)' },
|
|
'<reply-file-name>': { grammar: ID, what: 'basename of the mktemp reply file printed by the previous block' },
|
|
'<redact-file-name>': { grammar: ID, what: 'basename of the mktemp draft file printed by the free-text block' },
|
|
'<approach-file-name>': { grammar: ID, what: 'basename of the mktemp approach file printed by the free-text block' },
|
|
'<greptile-dir>': { grammar: PATH, what: 'mktemp directory printed by the Greptile fetch block' },
|
|
'<SNAPSHOT_TOOL>': { grammar: QUOTED, what: 'snapshot tool path printed by autoplan', quoted: true },
|
|
'<SOURCE_PLAN>': { grammar: QUOTED, what: 'plan path', quoted: true },
|
|
'<ACTIVE_PLAN>': { grammar: QUOTED, what: 'plan path', quoted: true },
|
|
'<RESTORE_PATH>': { grammar: QUOTED, what: 'restore path printed by autoplan', quoted: true },
|
|
'<methodologyPath>': { grammar: QUOTED, what: 'methodology path printed by autoplan', quoted: true },
|
|
'<CEO_STEP0_CHECKPOINT>': { grammar: QUOTED, what: 'checkpoint path printed by autoplan', quoted: true },
|
|
'<AMENDMENT_CHECKPOINT>': { grammar: QUOTED, what: 'checkpoint path printed by autoplan', quoted: true },
|
|
'<prepared-prompt-file>': { grammar: QUOTED, what: 'prompt file path printed by the prepare step', quoted: true },
|
|
'<run-dir>': { grammar: QUOTED, what: 'run directory printed by an earlier block', quoted: true },
|
|
'<ASIDE_DIR>': { grammar: QUOTED, what: 'Aside output directory printed by an earlier block', quoted: true },
|
|
'<ASIDE_DIR or $_TMP>': { grammar: QUOTED, what: 'Aside output directory printed by an earlier block', quoted: true },
|
|
'<REPORT_DIR from Setup>': { grammar: QUOTED, what: 'report directory printed by Setup', quoted: true },
|
|
'<the FILE path printed above>': { grammar: QUOTED, what: 'file path printed by the previous block', quoted: true },
|
|
'<install dir from Step 2>': { grammar: QUOTED, what: 'install directory printed by Step 2', quoted: true },
|
|
'<APPROVED path>': { grammar: QUOTED, what: 'approved design path', quoted: true },
|
|
'<first path from the printed saved list>': { grammar: QUOTED, what: 'image path printed by the design tool', quoted: true },
|
|
'<design-path>': { grammar: QUOTED, what: 'design doc path', quoted: true },
|
|
'<review-directory>': { grammar: QUOTED, what: 'review directory printed by mktemp', quoted: true },
|
|
'<round-1.json>': { grammar: QUOTED, what: 'round file path', quoted: true },
|
|
'<round-1.json if present>': { grammar: QUOTED, what: 'round file path', quoted: true },
|
|
'<round-2.json if present>': { grammar: QUOTED, what: 'round file path', quoted: true },
|
|
'<round-3.json if present>': { grammar: QUOTED, what: 'round file path', quoted: true },
|
|
'<source-dir>': { grammar: QUOTED, what: 'app source directory', quoted: true },
|
|
'<user-provided-path>': { grammar: QUOTED, what: 'directory the user named', quoted: true },
|
|
'<detected-directory>': { grammar: QUOTED, what: 'directory detected by the skill', quoted: true },
|
|
'<staged>': { grammar: QUOTED, what: 'staged HTML path', quoted: true },
|
|
'<stagedDir>': { grammar: QUOTED, what: 'staged directory printed by an earlier block', quoted: true },
|
|
'<outdir>': { grammar: PATH, what: 'output directory' },
|
|
'<path>': { grammar: PATH, what: 'repository path from discovery' },
|
|
'<plan-path>': { grammar: PATH, what: 'bound plan file path' },
|
|
'<prereg-path>': { grammar: PATH, what: 'preregistration file path' },
|
|
'<results-path>': { grammar: PATH, what: 'results table file path' },
|
|
'<sketch-dir>': { grammar: PATH, what: 'sketch directory printed by an earlier block' },
|
|
'<path-to-finalized.html>': { grammar: PATH, what: 'finalized HTML path' },
|
|
'<path-to-page.html>': { grammar: PATH, what: 'saved page HTML path' },
|
|
'<finalized.html>': { grammar: PATH, what: 'finalized HTML path' },
|
|
'<motion-dir>': { grammar: PATH, what: 'run directory printed by the /design-motion Step 0 MOTION_DIR line' },
|
|
'<approved-variant.png>': { grammar: PATH, what: 'approved variant image path' },
|
|
'<failing-test-file>': { grammar: PATH, what: 'test file path' },
|
|
'<source-file-under-test>': { grammar: PATH, what: 'source file path' },
|
|
'<affected-files>': { grammar: PATHS, what: 'space-separated file paths', list: true },
|
|
'<only-changed-files>': { grammar: PATHS, what: 'space-separated file paths', list: true },
|
|
'<only-verified-source-and-regression-files>': { grammar: PATHS, what: 'space-separated file paths', list: true },
|
|
'<scope>': { grammar: PATHS, what: 'path or glob', list: true },
|
|
|
|
'<project eval command>': { grammar: /^[A-Za-z0-9._/@+:,%=~ -]+$/, what: 'the eval command the project documents (words and flags only)', list: true },
|
|
|
|
// Quoted-only values: URLs, selectors, search words and commands.
|
|
'<url>': { grammar: URL, what: 'http(s) URL', quoted: true },
|
|
'<page-url>': { grammar: URL, what: 'http(s) URL', quoted: true },
|
|
'<base-url>': { grammar: URL, what: 'http(s) URL', quoted: true },
|
|
'<target-url>': { grammar: URL, what: 'http(s) URL', quoted: true },
|
|
'<TARGET_URL>': { grammar: URL, what: 'http(s) URL', quoted: true },
|
|
'<site>': { grammar: URL, what: 'http(s) URL', quoted: true },
|
|
'<selector>': { grammar: QUOTED, what: 'CSS selector', quoted: true },
|
|
'<row-selector>': { grammar: QUOTED, what: 'CSS selector', quoted: true },
|
|
'<title-selector>': { grammar: QUOTED, what: 'CSS selector', quoted: true },
|
|
'<your-keyword>': { grammar: QUOTED, what: 'search words', quoted: true },
|
|
'<keywords>': { grammar: QUOTED, what: 'search words', quoted: true },
|
|
'<keyword1>': { grammar: QUOTED, what: 'search word', quoted: true },
|
|
'<keyword2>': { grammar: QUOTED, what: 'search word', quoted: true },
|
|
'<keyword3>': { grammar: QUOTED, what: 'search word', quoted: true },
|
|
'<tests>': { grammar: QUOTED, what: 'test command', quoted: true },
|
|
'<vitest>': { grammar: QUOTED, what: 'test command', quoted: true },
|
|
|
|
// External-host renders: Claude Code outside voice and the gbrain page save.
|
|
'<gstack-runtime-root>': { grammar: QUOTED, what: 'installed runtime path', quoted: true },
|
|
'<fresh-or-resume>': { grammar: oneOf('fresh', 'resume'), what: 'Claude Code session mode' },
|
|
'<feature-slug>': { grammar: ID, what: 'feature slug for the gbrain page' },
|
|
'<page-file>': { grammar: QUOTED, what: 'mktemp page file path printed by the previous command', quoted: true },
|
|
|
|
// A3 group 1 (Aside prompts, design briefs, design approval feedback): mktemp basenames.
|
|
'<prompt-file-name>': { grammar: ID, what: 'basename of the mktemp prompt file (Aside, Codex, benchmark) printed by the previous block' },
|
|
'<brief-file-name>': { grammar: ID, what: 'basename of the mktemp design brief file printed by the free-text block' },
|
|
'<feedback-file-name>': { grammar: ID, what: 'basename of the mktemp design feedback file printed by the previous block' },
|
|
|
|
// A3 group 2 (Codex prompts and misc): mktemp basenames, mode switches,
|
|
// the benchmark prompt path and the sync-gbrain orchestrator flags.
|
|
'<focus-file-name>': { grammar: ID, what: 'basename of the mktemp Codex review focus file printed by the free-text block' },
|
|
'<receipt line>': { grammar: /^OFFICE_HOURS_VERDICT round=[1-3] sha256=[0-9a-f]{64} path=\/[^'"`$\\\n]+$/, what: 'office-hours reviewer verdict receipt (one line)', quoted: true },
|
|
'<body-top-file-name>': { grammar: ID, what: 'basename of the mktemp PR body draft (through ## Documentation) printed by the free-text block' },
|
|
'<body-rest-file-name>': { grammar: ID, what: 'basename of the mktemp PR body draft (after the documentation section) printed by the free-text block' },
|
|
'<rejected-file-name>': { grammar: ID, what: 'basename of the mktemp rejected-test list printed by the free-text block' },
|
|
'<new|resume>': { grammar: oneOf('new', 'resume'), what: 'Codex consult session mode' },
|
|
'<prompt-path>': { grammar: QUOTED, what: 'benchmark prompt file path', quoted: true },
|
|
'<user-args>': {
|
|
grammar: /^(?:(?:--(?:incremental|full|dry-run|quiet|no-code|no-memory|no-brain-sync|code-only|dream|no-dream|allow-reclone|prune-gone-worktrees)|--sources [a-z][a-z0-9_,-]*)(?: (?=-)|$))*$/,
|
|
what: 'gstack-gbrain-sync flags (space-separated; --sources takes one comma-separated type list)',
|
|
list: true,
|
|
},
|
|
};
|