Review plan gives a preregistration with only its plain data the write-up review, and stops counting data rows as code (GST-3238) (#4544)

A preregistration whose diff is only its write-up, text and data files
now plans as a write-up (one reviewer, one round); scoring, judge, guard
and gate edits keep their full review, and CI is unchanged.

Results rows no longer count as runtime code, and a fix after an
approved review stop that changes only a data file now gets its
follow-up review instead of stopping.

Nothing to do for readers; this follows Garry's rule that
preregistrations without scoring code get one review round, and fixes a
stop reported by Founderstar.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Review-Finding: v1/outside-voice#1 reader=outside-voice
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-11-19 23:17:49 +00:00
1 parent 9ea26ecc42
commit 858679423c
10 files changed
+185 -19

No files matched your search

+12 -6
View File
@@ -117,10 +117,12 @@ TEST_PATH_RE='(^|/)(tests?|spec|__tests__)/|(^|/)tests?\.py$|(^|/)test_[^/]+\.py
# Data files (lib/review-depth.ts DATA_PATH) outside the config locations its
# DATA_CONFIG and agent-prose rules keep code: .github/, config/, dotfiles,
# *.config.*, .gstack/. Case-insensitive, as DATA_PATH is.
# *.config.*, .gstack/. Case-insensitive, as DATA_PATH is. A data file sets no
# category but is never unmatched: its rows are data, and lib/review-depth.ts's
# data rule decides its class.
DATA_FILE_RE='\.(json|jsonl|ndjson|csv|tsv|png|jpe?g|gif|webp|bmp|ico|avif)$'
DATA_CONFIG_RE='(^|/)\.github/|(^|/)config/|(^|/)\.[^/]+$|(^|/)[^/]+\.config\.[^/]+$|^\.gstack/'
_session_data() {
_data_file() {
local r=1
shopt -s nocasematch
if [[ "$1" =~ $DATA_FILE_RE ]] && [[ ! "$1" =~ $DATA_CONFIG_RE ]]; then r=0; fi
@@ -234,6 +236,7 @@ if [ "${#FILES_LIST[@]}" -eq 0 ]; then
fi
UNMATCHED=()
N_UNMATCHED=0
# Categories are INDEPENDENT booleans (#2299): a single first-match-wins case
# made them mutually exclusive, so Button.test.jsx set FRONTEND but not TESTS
@@ -328,7 +331,7 @@ for f in ${FILES_LIST[@]+"${FILES_LIST[@]}"}; do
# session handling, so it is not auth; session config still is.
case "$f" in
*jwt*|*permission*|*role*) m_auth=true ;;
*session*) _session_data "$f" || m_auth=true ;;
*session*) _data_file "$f" || m_auth=true ;;
# Apple permissions (GST-3041): Info.plist holds URL schemes, App Transport
# Security exceptions and usage strings; entitlements grant capabilities
# (keychain groups, push, HealthKit); the privacy manifest declares data use.
@@ -363,8 +366,9 @@ for f in ${FILES_LIST[@]+"${FILES_LIST[@]}"}; do
if [ "$m_frontend" = false ] && [ "$m_prompts" = false ] && [ "$m_tests" = false ] \
&& [ "$m_docs" = false ] && [ "$m_config" = false ] && [ "$m_migrations" = false ] \
&& [ "$m_api" = false ] && [ "$m_auth" = false ] && [ "$m_backend" = false ]; then
&& [ "$m_api" = false ] && [ "$m_auth" = false ] && [ "$m_backend" = false ] && ! _data_file "$f"; then
UNMATCHED+=("$f")
N_UNMATCHED=$((N_UNMATCHED + 1))
fi
done
@@ -465,8 +469,10 @@ _print_flags
# Changed files but ZERO category matches (#2526): a classifier bug, an
# unrecognised layout, or a new top-level directory would otherwise present
# as "no reviewers needed" with the skip invisible. Trip loudly instead.
if [ "$FRONTEND" = false ] && [ "$BACKEND" = false ] && [ "$PROMPTS" = false ] \
# as "no reviewers needed" with the skip invisible. Trip loudly instead. A
# range of data files alone (a one-string fix to a .json after a granted stop,
# FOUN-505) is no classifier miss: it prints every flag false and exits 0.
if [ "$N_UNMATCHED" -gt 0 ] && [ "$FRONTEND" = false ] && [ "$BACKEND" = false ] && [ "$PROMPTS" = false ] \
&& [ "$TESTS" = false ] && [ "$DOCS" = false ] && [ "$CONFIG" = false ] \
&& [ "$MIGRATIONS" = false ] && [ "$API" = false ] && [ "$AUTH" = false ]; then
echo "SCOPE_ERROR=unmatched"
@@ -0,0 +1,4 @@
---
important: blocker
---
- **A preregistration with only its write-up and plain data now gets the write-up review: one reviewer, one round.** A diff of a preregistration's markdown, text and data files (a directory or file named `prereg` or `preregistration`, such as gstack-evals' `wiki/plans/<id>-prereg/`) plans as docs even when code reads its data; a scorer, judge, grader, guard or gate edit beside it keeps its full review, and CI is unchanged. Data rows (JSONL, NDJSON, CSV and TSV, and JSON in a docs, results or preregistration folder) no longer count as runtime lines: harness-evals#432's results planned 17,773 runtime lines, of which 147 were code. A fix after a granted review stop that changes only a data file is now read and gets its fix-check, where before `gstack-diff-scope` called the range unmatched and the plan stopped with "no fix to read".
+33 -1
View File
@@ -424,9 +424,41 @@ function); every path-only caller (the receipt's gate-scan decision, the Step 16
post-review route, /autoplan's docs-only scope) keeps data files code. A numeric claim
in a data file still schedules the outside voice.
<a id="data-rows"></a>
**Data rows are never runtime lines.** A JSONL, NDJSON, CSV or TSV file that code
names keeps its class (code), so its readers still read it, but its lines never count
as runtime lines, and neither do a JSON file's under a docs, wiki, results or reports
directory or a [preregistration](#preregistrations) path: a results PR's trial rows
are not code (harness-evals#432 counted 17,773 runtime lines at its head, 147 of them
code). A JSON file code names anywhere else keeps counting, because it can be config
a guard reads (harness-evals' `scripts/public-safety/allow.json`), and so do config
files (the list above), agent prose, gate paths, executables and binary files.
<a id="preregistrations"></a>
**Preregistrations are write-ups (R114).** A preregistration that adds no scoring
or guard code gets the write-up review: one reviewer, one round. Detection is by
path, because that is how gstack-evals and harness-evals lay preregistrations out
(`wiki/plans/<date>-<code>-<slug>-prereg/README.md` with its `.txt`, `.tsv` and
`.json` beside it, `docs/benchmarks/<id>/prereg-harbor.json`, `lanes/<lane>/prereg.md`)
and their front matter has no preregistration marker (`kind: plan` also covers
non-preregistration plans). `PREREG_PATH` matches a directory or file name with a
`prereg` or `preregistration` segment (`-prereg/`, `prereg2.md`, `-prereg-addendum-2/`);
a data file under it counts as docs even when code names it, so a diff of a
preregistration's markdown, text and data plans as docs. Anything else keeps its
current depth: code inside or beside the preregistration (a scorer, judge, grader or
rubric is the eval risk class and plans full; a `draw.py` or `power.py` stays code at
light), a gate path or guard edit (gate class), and data a preregistration names
outside its own path. The exemption sits on top of the existing catches: the gate
scan still reads the preregistration's data (a tagged hunk or a missing scan is gate
class) and so do the risk classes, as they read any data file code names (a secret
in it is full).
An amendment directory named without `prereg` (gstack-evals' `-pre-run-change-<n>/`)
keeps its current depth; name it `-prereg-amendment-<n>` to get the write-up review.
R81 is unchanged: a preregistration still runs full CI.
| Depth | When | Pass 1 dispatches |
|---|---|---|
| **light**, runtime 0 | docs, tests, eval write-ups, or only registered outputs | docs: the core checklist only (R81). Tests: one **batched** reader on the Testing checklist, which leaves docs and registered outputs out of its diff (R51: tests get one reviewer). Only registered outputs: one batched reader on the code checklists over the whole diff. The outside voice also reads a diff that publishes numbers |
| **light**, runtime 0 | docs, tests, eval write-ups, [preregistrations](#preregistrations) with only their plain data, or only registered outputs | docs: the core checklist only (R81). Tests: one **batched** reader on the Testing checklist, which leaves docs and registered outputs out of its diff (R51: tests get one reviewer). Only registered outputs: one batched reader on the code checklists over the whole diff. The outside voice also reads a diff that publishes numbers |
| **light**, runtime above 0 | code with no risk class, at any size (R51: size never raises the depth) | core, one **batched** specialist (the batched checklists below plus the Red Team prompt in one context), the native adversarial pass (kept: backtest inconclusive, GSTA-147 D5) and the outside voice on numbers |
| **standard** | `--depth-floor standard`, or a light diff raised by a fixed blocker or a CRITICAL | one batched specialist at `medium` and the Red Team at 200 runtime lines (or on a CRITICAL or a raise); the batched reader leaves docs and registered outputs out of its diff; adversarial and outside voice read everything |
| **full** | gate class, a [risk class](#risk-classes), or any classifier error | one batched specialist on the whole diff, the Red Team always, no hit-rate gating |
+17 -7
View File
@@ -13,7 +13,7 @@ import * as fs from 'node:fs';
import * as path from 'node:path';
import type { ResultCodeName } from './result-codes';
import { matchesAny } from './ship-policy';
import { decodeCharRefs, DOC_SECRET_LINE, matchRisk, RISK_CLASSES, RISK_RULES, SECRET_PATH, secretPath, SVG_PATH, svgActive, TOKEN_PATH, type RiskClass } from './review-risk';
import { decodeCharRefs, DOC_SECRET_LINE, matchRisk, RESULTS_TREE, RISK_CLASSES, RISK_RULES, SECRET_PATH, secretPath, SVG_PATH, svgActive, TOKEN_PATH, type RiskClass } from './review-risk';
export type DiffClass = 'docs' | 'tests' | 'code' | 'gate';
export type PathClass = 'docs' | 'tests' | 'code';
@@ -69,6 +69,10 @@ export const isDataConfig = (p: string) => DATA_CONFIG.some(r => r.test(p));
export const BINARY_DATA_PATH = /\.(sqlite3?|db|duckdb|parquet|arrow|feather|avro|orc|npy|npz|h5|hdf5|pdf|mp3|mp4|mov|wav|webm|log|out)$|(^|\/)logs\/[^/.]+$/i;
/** Data files: JSON, CSV, TSV and raster images. YAML and TOML are never data here: deploy configs and API specs stay code. An SVG is data only as a drawing (svgDrawing); executable SVG stays code. */
export const DATA_PATH = /\.(json|jsonl|ndjson|csv|tsv|png|jpe?g|gif|webp|bmp|ico|avif)$/i;
/** Row data, never runtime lines whether code names it or not (harness-evals#432 counted its results rows as code); JSON counts too in a docs, results or preregistration tree. */
export const DATA_ROWS = /\.(jsonl|ndjson|csv|tsv)$/i;
/** A preregistration's own directory or file (R114): a path segment named prereg or preregistration (wiki/plans/<id>-prereg/, prereg-harbor.json, lanes/<x>/prereg2.md). */
export const PREREG_PATH = /(^|\/|[-_.])(prereg|preregistration)s?\d*([-_./]|$)/i;
/** Data-shaped files tools read by convention, with no code naming them: always code. */
const DATA_CONFIG = [
/(^|\/)\.github\//, /(^|\/)config\//, /(^|\/)\.[^/]+$/, /(^|\/)[^/]+\.config\.[^/]+$/, HARNESS_PINS,
@@ -168,7 +172,7 @@ export function svgDrawing(text: string): boolean {
/** A hash of the constants and patterns that decide a depth, not of this file's bytes: a comment edit keeps it, a threshold edit changes it. */
export const DEPTH_RULES = createHash('sha256').update(JSON.stringify({
RISK_RULES, SMALL_DIFF_LINES, RED_TEAM_LINES, LARGE_DIFF_LINES, FULL_PASS_CAP, LIGHT_FIXED, BATCH_SCOPED, SCOPED_RANGE_LINES, ROLE_EFFORT, SVG_ELEMENTS: [...SVG_ELEMENTS],
patterns: Object.entries({ BINARY_DATA_PATH: [BINARY_DATA_PATH], GATE_PATH, AGENT_PROSE, TEST_PATH, RELEASE_METADATA, DOC_PATH: [DOC_PATH], DATA_PATH: [DATA_PATH], DATA_CONFIG, SVG_MARKUP: [SVG_MARKUP], SVG_REFUSED, SVG_HREF: [SVG_HREF], SVG_STYLE_OPEN: [SVG_STYLE_OPEN], SVG_STYLE_CLOSE: [SVG_STYLE_CLOSE], SVG_STYLE_TAG: [SVG_STYLE_TAG], SVG_TAG_NAME: [SVG_TAG_NAME], SVG_ATTR: [SVG_ATTR], SVG_CSS_REF: [SVG_CSS_REF], SVG_CSS_REFUSED: [SVG_CSS_REFUSED] }).map(([k, v]) => [k, v.map(r => `/${r.source}/${r.flags}`)]),
patterns: Object.entries({ BINARY_DATA_PATH: [BINARY_DATA_PATH], DATA_ROWS: [DATA_ROWS], PREREG_PATH: [PREREG_PATH], GATE_PATH, AGENT_PROSE, TEST_PATH, RELEASE_METADATA, DOC_PATH: [DOC_PATH], DATA_PATH: [DATA_PATH], DATA_CONFIG, SVG_MARKUP: [SVG_MARKUP], SVG_REFUSED, SVG_HREF: [SVG_HREF], SVG_STYLE_OPEN: [SVG_STYLE_OPEN], SVG_STYLE_CLOSE: [SVG_STYLE_CLOSE], SVG_STYLE_TAG: [SVG_STYLE_TAG], SVG_TAG_NAME: [SVG_TAG_NAME], SVG_ATTR: [SVG_ATTR], SVG_CSS_REF: [SVG_CSS_REF], SVG_CSS_REFUSED: [SVG_CSS_REFUSED] }).map(([k, v]) => [k, v.map(r => `/${r.source}/${r.flags}`)]),
})).digest('hex').slice(0, 16);
/**
@@ -258,7 +262,11 @@ const sanitize = (s: string) => s.replace(/[\0-\x1f]+/g, ' ').slice(0, 200);
const isOutput = (i: ClassInput, p: string) => !!i.registry?.outputs.length && matchesAny(i.registry.outputs, p);
export const unconsumedData = (i: Pick<ClassInput, 'consumed' | 'drawings' | 'root' | 'executables'>, p: string) => !i.executables?.includes(p) && Array.isArray(i.consumed) && !i.consumed.includes(p) && (dataCandidate(p, i.root) || (!!i.drawings?.includes(p) && svgCandidate(p, i.root)));
const binaryData = (i: ClassInput, p: string) => !i.executables?.includes(p) && BINARY_DATA_PATH.test(p) && classifyPath(p, i.root) === 'code' && ![...AGENT_PROSE, ...GATE_PATH, TOKEN_PATH, ...DATA_CONFIG].some(r => r.test(p));
const pathClass = (i: ClassInput, p: string): PathClass => i.executables?.includes(p) && !i.excluded?.has(p) ? 'code' : (i.excluded?.has(p) || (i.stampOnly ?? []).includes(p) || unconsumedData(i, p) || binaryData(i, p) ? 'docs' : classifyPath(p, i.root));
/** A preregistration's plain data (R114): a data candidate under PREREG_PATH, docs even when code names it; its guard and risk checks still run (classifyDiff). */
const preregData = (i: ClassInput, p: string) => !i.executables?.includes(p) && PREREG_PATH.test(p) && dataCandidate(p, i.root);
/** A data file whose lines are rows, not runtime lines (DATA_ROWS): a JSON elsewhere, such as a guard's allowlist, can be config and keeps counting. */
const dataRows = (i: ClassInput, e: PathLines) => !e.binary && !i.executables?.includes(e.path) && dataCandidate(e.path, i.root) && (DATA_ROWS.test(e.path) || (/\.json$/i.test(e.path) && (RESULTS_TREE.test(e.path) || PREREG_PATH.test(e.path))));
const pathClass = (i: ClassInput, p: string): PathClass => i.executables?.includes(p) && !i.excluded?.has(p) ? 'code' : (i.excluded?.has(p) || (i.stampOnly ?? []).includes(p) || unconsumedData(i, p) || binaryData(i, p) || preregData(i, p) ? 'docs' : classifyPath(p, i.root));
/** The diff's class, raised to gate by any gate signal; `errors` lists every input that fails the depth closed; `runtimeLines` counts code-class lines outside the base registry's outputs. */
export function classifyDiff(i: ClassInput): ClassResult {
@@ -270,12 +278,14 @@ export function classifyDiff(i: ClassInput): ClassResult {
for (const p of i.paths) { const c = pathClass(i, p); if (CLASS_RANK[c] > CLASS_RANK[cls]) raise(c, `${c}: ${p}`); }
const data = i.paths.filter(p => unconsumedData(i, p));
if (data.length) reasons.push(`docs: ${data.length} data path(s) no code names`);
const prereg = i.paths.filter(p => !i.excluded?.has(p) && preregData(i, p) && !unconsumedData(i, p));
if (prereg.length) reasons.push(`docs: ${prereg.length} preregistration data path(s) (R114)`);
const s = i.scope ?? {};
if (s.SCOPE_ERROR) {
raise('code', `scope error: ${s.SCOPE_ERROR}`);
errors.push({ code: 'scope_error', detail: sanitize(s.SCOPE_ERROR), input: 'gstack-diff-scope <base>', repair: 'run gstack-diff-scope <base> by hand and fix what it reports' });
}
if (CLASS_RANK[cls] >= CLASS_RANK.tests) {
if (CLASS_RANK[cls] >= CLASS_RANK.tests || prereg.length) {
if (i.gateTagged === null) {
raise('gate', 'gate scan unavailable');
errors.push({ code: 'gate_scan_unavailable', detail: 'gstack-gate-diff reported no GATE_SUMMARY', input: 'gstack-gate-diff <base>', repair: 'run gstack-gate-diff <base> and read its GATE_ERROR line, or pass the recorded scan as --gate-tagged <n>' });
@@ -289,7 +299,7 @@ export function classifyDiff(i: ClassInput): ClassResult {
for (const p of i.paths) if (GATE_PATH.some(r => r.test(p)) || secretPath(p, [...(i.added?.[p] ?? []), ...(i.removed?.[p] ?? [])])) { raise('gate', `gate path: ${p}`); break; }
if (i.registry?.error) errors.push(i.registry.error);
}
const riskHits = CLASS_RANK[cls] >= CLASS_RANK.code ? matchRisk({ codePaths: i.paths.filter(p => pathClass(i, p) === 'code'), executables: i.executables, added: i.added, removed: i.removed, scopeAuth: s.SCOPE_AUTH === 'true', ...(i.riskPaths?.length ? { isRiskPath: (p: string) => matchesAny(i.riskPaths!, p) && !DATA_PATH.test(p) } : {}) }) : new Map<RiskClass, string>();
const riskHits = CLASS_RANK[cls] >= CLASS_RANK.code || prereg.length ? matchRisk({ codePaths: i.paths.filter(p => pathClass(i, p) === 'code' || prereg.includes(p)), executables: i.executables, added: i.added, removed: i.removed, scopeAuth: s.SCOPE_AUTH === 'true', ...(i.riskPaths?.length ? { isRiskPath: (p: string) => matchesAny(i.riskPaths!, p) && !DATA_PATH.test(p) } : {}) }) : new Map<RiskClass, string>();
const docSecret = i.paths.find(p => DOC_PATH.test(p) && (i.added?.[p] ?? []).some(l => DOC_SECRET_LINE.some(r => r.test(l))));
if (docSecret && !riskHits.has('security')) riskHits.set('security', `docs that can print a credential: ${docSecret}`);
const byRank = (a: RiskClass, b: RiskClass) => RISK_CLASSES.indexOf(a) - RISK_CLASSES.indexOf(b);
@@ -301,7 +311,7 @@ export function classifyDiff(i: ClassInput): ClassResult {
if (!i.perPath) return { class: cls, reasons, errors, runtimeLines: null, risk, detected, declared };
let runtimeLines = 0;
for (const e of i.perPath) {
if (pathClass(i, e.path) !== 'code' || isOutput(i, e.path)) continue;
if (pathClass(i, e.path) !== 'code' || isOutput(i, e.path) || dataRows(i, e)) continue;
if (e.binary) errors.push({ code: 'binary_code_path', detail: 'git reports a binary change, so its lines cannot be counted', input: e.path, repair: "register it as an output of an entry with a command in the base branch's .gstack/generated.json, or accept the full review" });
else runtimeLines += e.lines;
}
@@ -701,7 +711,7 @@ export function planReview(i: PlanInput): ReviewPlan {
/** Risk class read from the paths and lines (gate class, an R51 or review.risk_paths class, a classifier error), or a floor above light (FOUN-64 item 7: never the recorded review_depth). */
const riskClass = (c: ClassResult, depth: DepthResult) => c.class === 'gate' || c.risk.length > 0 || c.errors.length > 0 || depth.depth !== 'light';
/** Whether a light diff may take the platform depth (FOUN-64 item 2): docs or tests only, no numeric claims, no results, prereg, eval or checker path. The receipt and the queue ask the same. */
export const platformFits = (cls: DiffClass, paths: readonly string[], runtimeLines: number, numbers: NumbersResult) => (cls === 'docs' || cls === 'tests') && runtimeLines === 0 && !numbers.publishes && !paths.some(p => CLAIM_PATH.test(p));
export const platformFits = (cls: DiffClass, paths: readonly string[], runtimeLines: number, numbers: NumbersResult) => (cls === 'docs' || cls === 'tests') && runtimeLines === 0 && !numbers.publishes && !paths.some(p => CLAIM_PATH.test(p) || PREREG_PATH.test(p));
/** Paths that carry results or preregistration claims: never the platform depth (FOUN-64 item 2). */
const CLAIM_PATH = /(^|\/|[-_.])(results?|prereg|preregistration|benchmarks?|evals?|scorers?|checkers?)([-_./]|$)/i;
+14
View File
@@ -430,6 +430,20 @@ describe('exit-code contract (#2526)', () => {
expect(vars.SCOPE_API).toBe('false');
});
// Value: protects=FOUN-505, a fix range of one data .json reads as data, not unmatched, so a granted fix-check can plan; an unknown
// file beside data still fails closed; fails_when=a data-only range exits 2 or data hides an unknown file; seam=none; no_claim=none
test('a data-only range prints every flag false and exits 0; an unknown file beside data is still unmatched (FOUN-505)', () => {
const data = runScopeFull(createRepo(['wiki/labels.json', 'results/rows.tsv']));
expect(data.status).toBe(0);
expect(data.vars.SCOPE_ERROR).toBeUndefined();
expect(Object.values(data.vars).every((v) => v === 'false')).toBe(true);
const mixed = runScopeFull(createRepo(['wiki/labels.json', 'weird/layout.xyz']));
expect(mixed.status).toBe(2);
expect(mixed.vars.SCOPE_ERROR).toBe('unmatched');
expect(mixed.stdout).toContain('# unmatched: weird/layout.xyz');
expect(mixed.stdout).not.toContain('# unmatched: wiki/labels.json');
});
// Value: protects=an Apple file type no arm knows still fails closed after Swift and Xcode got arms; fails_when=a broad
// *.swift-style or catch-all arm swallows unknown extensions; why_new=GST-3041 added Apple arms; seam=none; no_claim=none
test('an unknown Apple file type (a Metal shader) still fails closed (GST-3041)', () => {
+1 -1
View File
@@ -1 +1 @@
{"sha":"769ebe909a205e7968d6e5fb8e9d0ce08d896d358d1d3bae984a3c2a68696796","replay":"GST-3187"}
{"sha":"82f87a1db38f5ccd82e5786840176e0142c318bd81ace0367ac2e03cb2d3524d","replay":"GST-3238"}
+1
View File
@@ -0,0 +1 @@
{"replay":"GST-3238","justification":"GST-3238 planner no-loss check (R114, Garry 'do it' 2026-11-19): deterministic replay, no paid run. Over 460 first-parent main commits (gstack-evals 200 to 3df53ef, harness-evals 60 to its main, gstack 200 to ab6a38b), planReview and classifyDiff from main ab6a38b and from this branch, on each commit's collectCommit input (scope {} and gate scan 0 on both sides, panel and lean shapes), give the same depth on all 460 and the same class, risk classes and dispatches on 458; 34 differ only in runtime lines, always down (JSONL, NDJSON, CSV and TSV rows, and JSON under a docs, wiki, results, reports or prereg path), never up. The other 2 drop readers only where R114 or the data-rows rule says: gstack-evals 09d7e2ab50 (/cso advisories preregistration: README.md, power.txt, prereg.json) goes code to docs, panel batched+adversarial+outside voice to outside voice alone; 9a34366049 (a results PR of markdown, reasons.tsv and sessions.jsonl only) stays code at light with runtime 82 to 0, losing the native adversarial pass (panel) and trading the single reader for the batched reader (lean), outside voice kept. No change touches a commit with a risk class, gate class or a guard or scoring path; a first version that also uncounted JSON everywhere dropped the adversarial pass on harness-evals 4c2290601b and 07fa25cd7b (scripts/public-safety allow lists) and was narrowed before this pin. Saved findings were not re-scored, so this shows the change reaches only its intended diffs, not that no past blocker came from the dropped readers. Reviewer briefs, checklists and the transition table are unchanged (GStack Minder drive handoffs/GSTA-663/r114-planner-noloss/)."}
+9
View File
@@ -195,6 +195,15 @@ describe('C6 platform depth (FOUN-32, founderstar#496): a planned skip needs the
expect(planReview(base({ paths: ['docs/benchmarks/run/results/summary.md'], shape: on(ev) })).mode).toBe('lean');
expect(planReview(base({ paths: ['evals/prereg.md'], shape: on(ev) })).mode).toBe('lean');
});
// Value: protects=R114 makes numbered and plural prereg data docs, and the platform skip must still refuse it;
// fails_when=lanes/x/prereg2.json or a preregs/ folder plans the platform depth; seam=platformFits
test('every preregistration spelling PREREG_PATH accepts refuses the platform depth (prereg2.json, preregs/)', () => {
const ev = { id: '1', head: HEAD, reviewer: 'capy-ai[bot]' };
for (const p of ['lanes/x/prereg2.json', 'wiki/plans/a-preregs/arms.json']) {
expect(platformFits('docs', [p], 0, scanNumbers({}))).toBe(false);
expect(planReview(base({ paths: [p], shape: on(ev) })).mode).not.toBe('platform');
}
});
test('risk class and published numbers never take the platform depth', () => {
const ev = { id: '1', head: HEAD, reviewer: 'capy-ai[bot]' };
expect(planReview(base({ paths: ['lib/billing.ts'], shape: on(ev) })).mode).toBe('panel');
+75 -2
View File
@@ -3,7 +3,7 @@ import { spawnSync } from 'node:child_process';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { classifyDiff, classifyPath, dataCandidate, depthFor, gateScanNeeded, depthOfCommitInput, diffCommand, isGatePath, nextStep, planReview, reviewPlanLine, scanNumbers, DEPTH_RULES, ROLE_EFFORT, FULL_PASS_CAP, SCOPED_RANGE_LINES, type DepthError, type PlanInput } from '../lib/review-depth';
import { classifyDiff, classifyPath, dataCandidate, PREREG_PATH, depthFor, gateScanNeeded, depthOfCommitInput, diffCommand, isGatePath, nextStep, planReview, reviewPlanLine, scanNumbers, DEPTH_RULES, ROLE_EFFORT, FULL_PASS_CAP, SCOPED_RANGE_LINES, type DepthError, type PlanInput } from '../lib/review-depth';
import { summarizeReviewLoop } from '../lib/review-scope';
import { sectionTexts } from '../lib/review-brief';
import { createHash } from 'node:crypto';
@@ -772,7 +772,7 @@ describe('GSTA-167 part 4a: data files no code names are docs (classifyDiff only
test('a consumed data file, an unscanned diff and an overflowing scan all keep the file code', () => {
for (const consumed of [['wiki/post/data.json'], undefined, 'all' as const]) {
const c = classifyDiff({ paths: ['wiki/post/data.json'], perPath: [json(80)], consumed, gateTagged: 0 });
expect([c.class, c.runtimeLines]).toEqual(['code', 80]);
expect([c.class, c.runtimeLines]).toEqual(['code', 0]);
}
});
test.each([
@@ -810,6 +810,79 @@ describe('GSTA-167 part 4a: data files no code names are docs (classifyDiff only
});
});
describe('R114: a preregistration and its plain data are a write-up; data rows are never runtime lines', () => {
const dir = 'wiki/plans/2026-11-19-geva-559-ship-panel-judged-sample-prereg';
const prereg = [`${dir}/README.md`, `${dir}/power.txt`, `${dir}/arms.json`, `${dir}/route-table.tsv`];
const consumed = [`${dir}/arms.json`, `${dir}/route-table.tsv`];
const input = (extra: string[] = [], o: Partial<PlanInput> = {}) => {
const paths = [...prereg, ...extra];
return { paths, perPath: paths.map(p => ({ path: p, lines: 40, binary: false })), consumed, gateTagged: 0, scope: {}, diffLines: 40 * paths.length, ...o };
};
const lean: PlanInput['shape'] = { panel: 'off', readerModel: 'm', platform: 'off', head: 'h', platformReview: null };
// Value: protects=R114, a preregistration whose data code names plans as the docs write-up it is (one reviewer, one round), under
// both the panel and the lean table; fails_when=a named prereg data file makes the diff code or counts runtime lines; seam=ClassInput.consumed
test('a prereg-only diff whose data code names is docs, light and runtime 0, and plans exactly as a docs write-up', () => {
const c = classifyDiff(input());
expect([c.class, c.runtimeLines, c.risk, c.errors]).toEqual(['docs', 0, [], []]);
expect(c.reasons).toContain('docs: 2 preregistration data path(s) (R114)');
const writeUp = { paths: [`${dir}/README.md`], perPath: [{ path: `${dir}/README.md`, lines: 40, binary: false }], gateTagged: 0, scope: {}, diffLines: 40 };
const numbers = scanNumbers({ [`${dir}/README.md`]: ['The bar: prove-it loads in at least 39 of 60 bugs (65%).'] });
for (const o of [{}, { numbers }, { shape: lean }, { shape: lean, numbers }] as Partial<PlanInput>[]) {
const p = planReview(base({ ...input([], o) }));
const w = planReview(base({ ...writeUp, ...o }));
expect([p.class, p.depth, p.runtimeLines, roles(p), p.next.step]).toEqual(['docs', 'light', 0, roles(w), w.next.step]);
}
expect(roles(planReview(base({ ...input([], { numbers }) })))).toEqual(['outside-voice:outside-voice:host']);
expect(depthOfCommitInput(input()).depth).toBe('light');
const sessions = depthOfCommitInput(input([`${dir}/sessions.txt`], { scope: { SCOPE_AUTH: 'true', SCOPE_DOCS: 'true' } }));
expect([sessions.class, sessions.depth]).toEqual(['docs', 'light']);
});
// Value: protects=R114's exception, scoring, judge or checker code keeps its depth whether it sits beside the prereg or inside it;
// fails_when=a scorer next to a preregistration plans light; seam=lib/review-risk.ts eval class
test('a prereg plus a scoring module is full depth, inside or outside the prereg directory', () => {
for (const scorer of ['lib/scorers/pass-rate.ts', `${dir}/judge.py`, 'src/grading/rubric.ts']) {
const d = depthOfCommitInput(input([scorer]));
expect([d.class, d.depth, d.risk]).toEqual(['code', 'full', ['eval']]);
}
});
// Value: protects=R114's exception, a guard edit beside a preregistration keeps the full gate review; fails_when=a gate path,
// a tagged gate hunk or an unavailable gate scan in a prereg diff plans light; seam=classifyDiff's gate signals
test('a prereg plus a guard edit is full depth: a gate path, a tagged hunk in its data, or no gate scan', () => {
for (const guard of ['lib/gate-thresholds.json', '.github/workflows/evals.yml', 'bin/gstack-pregate']) {
const d = depthOfCommitInput(input([guard]));
expect([d.class, d.depth]).toEqual(['gate', 'full']);
}
expect(depthOfCommitInput(input([], { gateTagged: 1 }))).toMatchObject({ class: 'gate', depth: 'full' });
const unscanned = depthOfCommitInput(input([], { gateTagged: null }));
expect([unscanned.class, unscanned.depth, unscanned.errors.map(e => e.code)]).toEqual(['gate', 'full', ['gate_scan_unavailable']]);
});
// Value: protects=the R114 exemption sits on top of the existing catches: code in a prereg stays code, a secret in its named
// data still reads as security risk, and executable or out-of-prereg data stays code; fails_when=any of them plans as docs
test('the exemption covers only plain data under a prereg path; every other catch still applies', () => {
const draw = depthOfCommitInput(input([`${dir}/draw.py`]));
expect([draw.class, draw.depth, draw.runtimeLines]).toEqual(['code', 'light', 40]);
const secret = depthOfCommitInput(input([], { added: { [`${dir}/arms.json`]: ['"cmd": "curl -H \\"x-api-key: $OPENAI_API_KEY\\""'] } }));
expect([secret.depth, secret.risk]).toEqual(['full', ['secret']]);
expect(classifyDiff(input([], { executables: [`${dir}/arms.json`] })).class).toBe('code');
expect(classifyDiff({ paths: [`${dir}/README.md`, 'lanes/x/arms.json'], consumed: ['lanes/x/arms.json'], gateTagged: 0 }).class).toBe('code');
});
test.each([
['wiki/plans/2026-11-19-capy-b1d-hook-default-prereg/route-table.tsv', true], ['docs/benchmarks/2026-11-10-gstack-skill-ablation/prereg-harbor-arms.json', true],
['lanes/review-replay/panel/eval2/prereg2.md', true], ['wiki/plans/2026-10-31-geva-65-lanea-shakedown-prereg-addendum-2/arms.json', true], ['evals/preregistration/arms.csv', true],
['wiki/plans/2026-11-18-geva-553-office-hours-rewrite-pre-run-change-1/arms.json', false], ['src/preregistered.json', false], ['wiki/plans/x-results/rows.jsonl', false],
])('PREREG_PATH %s -> %s', (p, hit) => expect(PREREG_PATH.test(p)).toBe(hit));
// Value: protects=harness-evals#432, whose results rows code names by basename planned 17,773 runtime lines; fails_when=a text data
// row counts as a runtime line, or a config file, a guard's JSON allowlist or a binary named image stops counting;
// seam=classifyDiff runtimeLines; no_claim=a JSON guard allowlist outside a docs or results tree (harness-evals' scripts/public-safety/allow.json) keeps counting
test('data rows never count as runtime lines, named or not; a JSON outside a docs or results tree, config files and code still do', () => {
const r = 'docs/benchmarks/2026-11-10-x/results';
const paths = [`${r}/main/opus-full/trials.jsonl`, `${r}/audit.json`, 'docs/benchmarks/2026-11-10-x/locks/diet.lock.json', `${r}/rows.tsv`, 'lanes/x/runs/proxy-usage.ndjson', 'lanes/x/table.csv', 'scripts/analyze.ts', 'package.json', 'scripts/public-safety/allow.json'];
const lines = (p: string) => p.endsWith('.ts') ? 12 : p === 'package.json' ? 2 : p.endsWith('allow.json') ? 19 : 5000;
const c = classifyDiff({ paths, perPath: paths.map(p => ({ path: p, lines: lines(p), binary: false })), consumed: paths.filter(p => !p.endsWith('.ts') && p !== 'package.json'), gateTagged: 0 });
expect([c.class, c.runtimeLines]).toEqual(['code', 12 + 2 + 19]);
});
});
describe('lean ensemble: one batched specialist at standard and full depth (GSTA-167 part 2)', () => {
const fixture = (name: string): PlanInput => {
const x = JSON.parse(fs.readFileSync(path.join(ROOT, 'test/fixtures/review-depth', `${name}.json`), 'utf8'));
+19 -2
View File
@@ -137,8 +137,8 @@ describe('the stop record binds only a review gstack can tie to one commit (stop
});
/** A fixture repository at the stop head with a PR body file, and the bins run against it. */
function stopRepo() {
const f = shipFixture({ 'src/x.ts': 'export const x = 1;\n' }, { 'src/x.ts': 'export const x = 2;\n' });
function stopRepo(base: Record<string, string> = {}, branch: Record<string, string> = {}) {
const f = shipFixture({ 'src/x.ts': 'export const x = 1;\n', ...base }, { 'src/x.ts': 'export const x = 2;\n', ...branch });
fxs.push(f);
const run = (bin: string, args: string[]) => f.run(path.join(BIN, bin), args, { GSTACK_APPROVALS_MIRROR: '0' });
const head = () => f.git('rev-parse', 'HEAD').stdout.trim();
@@ -193,6 +193,23 @@ describe('the CLI seams', () => {
expect(r.run('gstack-ship-receipt', ['history', '--body', r.body, '--pr', '7', '--base', 'main', '--run', 'fc2']).stdout).toContain('PR_REVIEW_GRANT: not applied (grant ' + g + ' was claimed by run fc1');
});
// Value: protects=FOUN-505, a granted stop whose fix is one string in a data .json (named by code or not) reads its fix range and plans
// the fix-check; fails_when=gstack-diff-scope calls the range unmatched, FIX_RANGE prints unread and the plan stops with no fix to
// read; seam=gstack-diff-scope, gstack-review-plan's fix range, classifyDiff's data rows
test('a data-only fix after a granted stop is read (runtime 0) and plans the fix-check (FOUN-505)', () => {
for (const named of [false, true]) {
const labels = '{"label": "pass"}\n';
const r = stopRepo({ 'data/labels.json': labels }, named ? { 'src/x.ts': "export const x = 2; // reads labels.json\n" } : {});
const s = r.head();
const g = grantId(r.run('gstack-approvals', ['round-grant', '--pr', '7', '--head', s, '--kind', 'fix-check', '--stop-run', 'ro', '--findings', 'f#1', '--reason', 'x', '--by', 'Captain GStack Minder']).stdout)!;
r.f.write({ 'data/labels.json': '{"label": "fail"}\n' }); r.f.commit('fix the label');
const plan = r.run('gstack-review-plan', ['--base', 'main', '--gate-tagged', '0', '--prior-rounds', '3', '--prior-full', '3', '--pr', '7', '--round-grant', g, '--stop', s]);
expect(plan.status, plan.stderr).toBe(0);
expect(plan.stdout).not.toContain('unread');
expect(plan.stdout).toContain('next=fix-check');
}
});
test('X15/G7: the plan refuses a fix-check grant for another stop head and an extra-round grant with --stop, and plans no fix-check without a fix', () => {
const r = stopRepo();
const s = r.head();