mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-03 07:18:03 +01:00
v1.93.4.0 fix(review-ledger): a finding's own Severity field sets its label, not severity words in its text (GST-976) (#3867)
* fix(review-ledger): a finding's Severity field sets its label, body severity words never raise it (GST-976) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(review-ledger): Severity counts only in field position, locus-first and emoji-led labels lead, an inline field floors the next (GST-976 review) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(review-ledger): linear lead-label prefix, all-caps leads before a space, fields after a parenthesis or without a colon at the line start (GST-976 review) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(review-ledger): a lead's file:line has no runaway backtracking and may be a range; a mid-line Severity field must close its clause (GST-976 review) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(review-ledger): a field at the line's start needs no closing clause; mid-line fields accept : / and → after the label (GST-976 fix-check) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: changelog fragment and review-policy collect step for GST-976 Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(review-ledger): a Severity field's label chain (High/Critical) takes its strongest label; mid-line fields keep the clause-closing set (GST-976 scoped check) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(review-ledger): never weaker than main invariant over pinned inputs, fixtures and generated field shapes; a multi-label Severity field counts as none, lowercase fields keep their label (GST-976) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: GST-976 re-audit, multi-label and lowercase Severity fields Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed
1 parent
47079fd89f
commit
a2d7a6f0fe
9 files changed
+185
-22
No files matched your search
@@ -1,5 +1,13 @@
|
||||
# Changelog
|
||||
|
||||
## [1.93.4.0] - 2026-11-02
|
||||
|
||||
**/ship's finding collector reads severity from a finding's own Severity field, so a severity word in the finding's text no longer raises it.**
|
||||
|
||||
### Fixed
|
||||
|
||||
- When a reader's finding carries its own `Severity:` field, the collector uses that field. A word like "P0" in the finding's text (a plan phase name, for example) no longer makes it a blocker the reader marked `Blocks: no`. A label leading the line (`CRITICAL:`, `[P0]`, `- High:`, a label heading) still sets the floor, `Blocks: yes` still makes a finding CRITICAL, and the native voice's FIXABLE still collects as CRITICAL. A field that names more than one label ("High/Critical") counts as no field, and a lowercase `severity:` keeps its label. Lines with no Severity field collect as before. (GST-976)
|
||||
|
||||
## [1.93.3.0] - 2026-11-02
|
||||
|
||||
**The queue now refuses a head whose required targeted review is missing or failed (GST-966).** On gstack-evals#140, the receipt showed `REVIEW_TARGETED: required=auth receipt=legacy verdict=missing` and `gstack-ship-queue ready` still exited 0, so a skipped required reader looked the same as a clean one.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# gstack digest v1.93.3.0 — regenerate/re-copy after upgrading gstack
|
||||
# gstack digest v1.93.4.0 — regenerate/re-copy after upgrading gstack
|
||||
|
||||
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
|
||||
for agent hosts without a full skill install. The full skills add workflows,
|
||||
|
||||
@@ -540,10 +540,11 @@ ones, and the arm lost blockers. `bin/gstack-review-dispatch` closes both ends:
|
||||
3. `collect` hashes each reader's raw output and appends its findings to
|
||||
`merged.jsonl` with the label as written (a JSON line's `severity`, or the most
|
||||
severe of a prose line's Critical, High, Medium, Low, P0-P3, FIXABLE or INVESTIGATE
|
||||
wherever they stand, headings and table rows included; a top-level list item under a
|
||||
wherever they stand, headings and table rows included, except that a line with its own `Severity` field takes that
|
||||
field's label (any letter case; a field stating more than one label, "High/Critical", counts as none), so a severity word in the finding's text does not raise it, and a label leading the line stays its floor (GST-976); a top-level list item under a
|
||||
label-only heading takes the stronger of the two). A `Severity:` or `Blocks:` line
|
||||
continues the finding above it until a list item, heading, bold paragraph opener or a
|
||||
second line for the same field starts another: `Severity` raises its label,
|
||||
second line for the same field starts another: `Severity` sets its label (a label leading the finding stays its floor),
|
||||
`Blocks: yes` makes it CRITICAL and `Blocks: no` never lowers it; an inline
|
||||
`Blocks: yes` is CRITICAL too, and with no finding above it is one. The native
|
||||
adversarial voice's FIXABLE collects as CRITICAL (normalizer v2, GSTA-167). An unnumbered, unlabeled bullet with no file:line
|
||||
|
||||
+40
-10
@@ -8,8 +8,8 @@
|
||||
* judgment each lead-judgment change: an added or promoted finding, with its reason
|
||||
* judge the verdict on the parent's judged set (ok, or the dropped and downgraded ids)
|
||||
*
|
||||
* Findings reach merged.jsonl only through collect, with each voice's label as written (a prose line's most severe one,
|
||||
* raised by the Severity line below it), except that a prose voice's `Blocks: yes` and the native voice's FIXABLE collect
|
||||
* Findings reach merged.jsonl only through collect, with each voice's label as written (a prose finding's own Severity field when
|
||||
* it has one, else its line's most severe label), except that a prose voice's `Blocks: yes` and the native voice's FIXABLE collect
|
||||
* as CRITICAL (GSTA-167 normalizer v2).
|
||||
* The parent's judged set may add or promote a finding and may fold a duplicate into one at least
|
||||
* as severe; it never drops or downgrades one. verify replays every check from the ledger alone.
|
||||
@@ -67,6 +67,20 @@ const META = new RegExp(`^(?:[-*+]\\s+)?\\**\\s*(Severity|Blocks)\\**\\s*(?::|\\
|
||||
/** The severity and Blocks a prose finding's own fields state (GST-766): recorded beside the collected label, never in its place. */
|
||||
export interface Stated { severity?: string; blocks?: string }
|
||||
const STATED_SEVERITY = new RegExp(`(?<![A-Za-z0-9])Severity\\**\\s*:?\\s*\\**\\s*(${LABELS})(?![A-Za-z0-9])`, 'gi');
|
||||
/**
|
||||
* A finding's own `Severity` field: at the line's start (its colon optional there, "Severity Low"), or after a separator with its colon
|
||||
* ("1. Drops rows (q.ts:7). Severity: Low", "(t.md:3) Severity: Low", "- Severity: P0"), its label closing the clause. With one present a severity word in the body never raises the label (GST-976); prose that mentions a
|
||||
* severity ("pass 1 had Severity: Low", "filed at severity low", "(Severity: Low in pass 1)") is not a field.
|
||||
*/
|
||||
const SEVERITY_FIELD = new RegExp(`(?:^(?:[-*+>]\\s+|\\d+[.)]\\s+)?\\**\\s*Severity\\**\\s*:?\\s*\\**\\s*((?:${LABELS})(?:\\s*[/→]\\s*\\**\\s*(?:${LABELS}))*)(?![A-Za-z0-9])|(?:[.;,|()]\\s*|\\s[—–-]\\s*)\\**\\s*Severity\\**\\s*:\\s*\\**\\s*((?:${LABELS})(?:\\s*[/→]\\s*\\**\\s*(?:${LABELS}))*)(?![A-Za-z0-9])(?=\\**\\s*(?:$|[.;,|()*—–]|-\\s|Blocks)))`, 'gi');
|
||||
/**
|
||||
* A label leading its line past list, quote, heading, emphasis and emoji marks (one mark per step, so a long divider stays linear)
|
||||
* and an optional file:line, closed by a mark or a separator, or by a space, comma or parenthesis after an all-caps word ("[P0] …",
|
||||
* "1. **High**: …", "### CRITICAL: …", "1. api.ts:10 CRITICAL: …", "1. CRITICAL missing auth"): the reader's own label, which a
|
||||
* Severity field never lowers. "P0 pilot …" is not one.
|
||||
*/
|
||||
const FIELD_LABELS = new RegExp(ANY_LABEL.source, 'gi');
|
||||
const LEAD_LABEL = new RegExp(`^(?:[^\\p{L}\\p{N}\\s]\\s*|\\d+[.)]\\s*)*(?:\\w[\\w./-]*:\\d+(?:[-–:,]\\s*\\d+)*[^\\p{L}\\p{N}\\s]*\\s+(?:[^\\p{L}\\p{N}\\s]\\s*)*)?(${LABELS})(?=[*_\`\\])]|\\s*[:|—–]|\\s+-\\s|\\s*$|(?<=[A-Z]{3})[\\s,(.])`, 'u');
|
||||
const STATED_BLOCKS = /(?<![A-Za-z0-9])Blocks\**\s*:?\s*\**\s*(yes|no)(?![A-Za-z0-9])/gi;
|
||||
function stated(raw: string): Stated | undefined {
|
||||
if (raw.startsWith('{')) return undefined;
|
||||
@@ -80,15 +94,17 @@ export interface MergedLine { id: string; reader: string; label: string; stated?
|
||||
/**
|
||||
* One reader's findings as written: a JSON object per line (severity kept as written); else each prose line carrying a severity or
|
||||
* FIXABLE/INVESTIGATE label, at the most severe label on it, and each top-level list item under a label-only heading (the stronger
|
||||
* of the two labels). A `Severity` or `Blocks` field line continues the open finding, once per field: `Severity` raises its label,
|
||||
* `Blocks: yes` makes it CRITICAL, `Blocks: no` never lowers it; with no finding open it is one, carrying the paragraph's subject.
|
||||
* of the two labels). A line with a `Severity` field takes that field's label instead (a field stating more than one label, "High/Critical", counts as none), whatever its
|
||||
* body names (GST-976: "the P0 pilot" collected as P0 under `Severity: Low`); a label leading the line and an earlier field of the same
|
||||
* finding are its floor. A `Severity` or `Blocks` field line continues the open finding, once per field: `Severity` sets its label, `Blocks: yes` makes it CRITICAL, `Blocks: no` never lowers it; with no finding open it is one, carrying
|
||||
* the paragraph's subject.
|
||||
* A list item, a heading, a bold or post-blank paragraph at the margin, or a line naming a file:line after the open finding's fields
|
||||
* starts the next finding, except an unlabeled, unnumbered bullet with no file:line under a paragraph finding whose fields have not
|
||||
* come yet, which is its detail and joins its text (GSTA-196: 4 findings with detail bullets collected as 8). An inline
|
||||
* `Blocks: yes` is CRITICAL too. The native voice's FIXABLE collects as CRITICAL; other labels stay as written for lead judgment. When in doubt the parser merges rather than drops: a lost blocker costs the gate.
|
||||
*/
|
||||
export function parseFindings(reader: string, output: string): Array<{ label: string; raw: string }> {
|
||||
type Finding = { label: string; raw: string; fields: Set<string>; list: boolean };
|
||||
type Finding = { label: string; raw: string; fields: Set<string>; list: boolean; lead?: string };
|
||||
const out: Finding[] = [];
|
||||
let section: string | null = null;
|
||||
let current: Finding | null = null;
|
||||
@@ -97,8 +113,17 @@ export function parseFindings(reader: string, output: string): Array<{ label: st
|
||||
let treeSeen = false;
|
||||
const role = reader.split('/')[0];
|
||||
const named = (l: string) => (role === 'adversarial-native' && l.toUpperCase() === 'FIXABLE' ? 'CRITICAL' : l);
|
||||
const strongest = (t: string) => [...t.matchAll(ALL_LABELS)].map(m => named(m[1]!)).reduce<string | undefined>((a, b) => (a === undefined ? b : stronger(a, b)), undefined);
|
||||
const push = (label: string, raw: string, fields: string[] = []): Finding => { const f = { label, raw, fields: new Set(fields), list: LIST_ITEM.test(raw) }; out.push(f); return f; };
|
||||
const strongestOf = (labels: Array<string | undefined>) => labels.reduce<string | undefined>((a, b) => (a === undefined ? b : b === undefined ? a : stronger(a, b)), undefined);
|
||||
const strongest = (t: string) => strongestOf([...t.matchAll(ALL_LABELS)].map(m => named(m[1]!)));
|
||||
const fieldOf = (t: string) => { const ls = [...t.matchAll(SEVERITY_FIELD)].flatMap(m => (m[1] ?? m[2]!).match(FIELD_LABELS)!); return ls.length === 1 ? ls[0] : undefined; };
|
||||
const critical = (t: string) => BLOCKS_YES.test(t) || (role === 'adversarial-native' && /(?<![A-Za-z0-9])FIXABLE(?![A-Za-z0-9])/.test(t));
|
||||
const leadOf = (t: string) => { const l = LEAD_LABEL.exec(t)?.[1]; return l === undefined ? undefined : named(l); };
|
||||
const labelOf = (t: string) => {
|
||||
const field = fieldOf(t);
|
||||
if (field === undefined) return strongest(t);
|
||||
return critical(t) ? 'CRITICAL' : strongestOf([leadOf(t), field])!;
|
||||
};
|
||||
const push = (label: string, raw: string, fields: string[] = [], lead?: string): Finding => { const f = { label, raw, fields: new Set(fields), list: LIST_ITEM.test(raw), lead }; out.push(f); return f; };
|
||||
for (const line of output.replace(/\r\n/g, '\n').split('\n')) {
|
||||
const t = line.trim();
|
||||
if (!t) { gap = true; continue; }
|
||||
@@ -107,7 +132,7 @@ export function parseFindings(reader: string, output: string): Array<{ label: st
|
||||
gap = false;
|
||||
if (/^NO FINDINGS\.?$/.test(t) || /^Recommendation:/i.test(t)) { current = null; continue; }
|
||||
const indented = line !== line.trimStart();
|
||||
const label = strongest(t);
|
||||
const label = labelOf(t);
|
||||
const blocks = BLOCKS_YES.test(t);
|
||||
const listItem = !indented && LIST_ITEM.test(t);
|
||||
const locus = LOCUS.test(t);
|
||||
@@ -117,6 +142,7 @@ export function parseFindings(reader: string, output: string): Array<{ label: st
|
||||
if (current && !current.fields.has(field)) {
|
||||
current.fields.add(field);
|
||||
if (blocks) current.label = 'CRITICAL';
|
||||
else if (field === 'severity' && fieldOf(t) !== undefined && !critical(current.raw)) current.label = strongestOf([current.lead, label])!;
|
||||
else if (label) current.label = stronger(current.label, label);
|
||||
current.raw += ` | ${t}`;
|
||||
} else if (blocks || label) current = push(blocks ? 'CRITICAL' : label!, head ? `${head} | ${t}` : t, [field]);
|
||||
@@ -129,7 +155,7 @@ export function parseFindings(reader: string, output: string): Array<{ label: st
|
||||
if (t.startsWith('{')) {
|
||||
try {
|
||||
const o = JSON.parse(t) as Record<string, unknown>;
|
||||
if (typeof o.severity === 'string') { current = push(o.severity, t); continue; }
|
||||
if (typeof o.severity === 'string') { current = push(o.severity, t, [], o.severity); continue; }
|
||||
} catch { /* prose that starts with a brace */ }
|
||||
}
|
||||
const detail = listItem && /^[-*+]\s/.test(t) && !locus && !label && !section && !!current && !current.list && !current.fields.size;
|
||||
@@ -138,7 +164,11 @@ export function parseFindings(reader: string, output: string): Array<{ label: st
|
||||
if (fresh) { current = null; head = t; }
|
||||
const inherited = section && listItem ? section : undefined;
|
||||
const found = label && inherited ? stronger(inherited, label) : label ?? inherited;
|
||||
if (found) { current = push(blocks ? 'CRITICAL' : found, t); if (!indented) head = t; continue; }
|
||||
if (found) {
|
||||
current = push(blocks ? 'CRITICAL' : found, t, [], strongestOf([leadOf(t), inherited, fieldOf(t)]));
|
||||
if (!indented) head = t;
|
||||
continue;
|
||||
}
|
||||
if (blocks && !t.startsWith('{')) {
|
||||
if (current) { current.label = 'CRITICAL'; current.raw += ` | ${t}`; continue; }
|
||||
current = push('CRITICAL', head && head !== t ? `${head} | ${t}` : t);
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gstack",
|
||||
"version": "1.93.3",
|
||||
"version": "1.93.4",
|
||||
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
{"note":"parseFindings labels at main 82e9d7ec (GST-976): every input the review-dispatch and prose-voice-floor tests pass it, every reader fixture here, and generated Severity field shapes. Regenerate only from a frozen main.","main":"82e9d7ec7a2c222f20b04aaf5b965fa50d7b6bdb","entries":[{"source":"pinned","reader":"adversarial-native","output":"FIXABLE: race in a.ts:3\nSeverity: Medium\nBlocks: no","main":["CRITICAL"]},{"source":"pinned","reader":"adversarial-native","output":"INVESTIGATE: odd lock in b.ts:9\nSeverity: Low\nBlocks: yes","main":["CRITICAL"]},{"source":"pinned","reader":"adversarial-native","output":"INVESTIGATE: odd lock in b.ts:9\nBlocks: no","main":["INVESTIGATE"]},{"source":"pinned","reader":"adversarial-native","output":"FIXABLE: race in a.ts:3. Severity: Medium. Blocks: no","main":["CRITICAL"]},{"source":"pinned","reader":"adversarial-native","output":"1. FIXABLE: race in a.ts:3 — Severity: Low, Blocks: no","main":["CRITICAL"]},{"source":"pinned","reader":"adversarial-native","output":"- FIXABLE: race in a.ts:3\n- Severity: Medium\n- Blocks: no","main":["CRITICAL"]},{"source":"pinned","reader":"outside-voice","output":"1. Drops the last row (q.ts:7). Severity: Critical. Blocks: no","main":["Critical"]},{"source":"pinned","reader":"outside-voice","output":"1. Cache evicts sessions (c.ts:4)\n Severity: High","main":["High"]},{"source":"pinned","reader":"outside-voice","output":"1. Retry storm (r.ts:2)\n Severity: Medium\n Blocks: yes","main":["CRITICAL"]},{"source":"pinned","reader":"outside-voice","output":"1. Index drift (i.ts:5). Severity: Low. Blocks: Yes, it loses the row","main":["CRITICAL"]},{"source":"pinned","reader":"outside-voice","output":"1. Naming (n.ts:1)\n Severity: Low\n Blocks: no","main":["Low"]},{"source":"pinned","reader":"specialist/testing","output":"REVIEWED_TREE: d0dd42d6a88fbc2a6abca4f6b2dfde7e35520c5a /home/user/.capy/work/gst976/wt\nNO FINDINGS\n","main":[]},{"source":"pinned","reader":"specialist/testing","output":"REVIEWED_TREE: d0dd42d6a88fbc2a6abca4f6b2dfde7e35520c5a /home/user/.capy/work/gst976/wt\n{\"severity\":\"CRITICAL\",\"path\":\"a.ts\",\"line\":3,\"category\":\"logic\",\"summary\":\"off by one\",\"specialist\":\"testing\"}\n{\"severity\":\"INFORMATIONAL\",\"path\":\"a.ts\",\"line\":9,\"category\":\"style\",\"summary\":\"name\",\"specialist\":\"testing\"}\n","main":["CRITICAL","INFORMATIONAL"]},{"source":"pinned","reader":"outside-voice","output":"REVIEWED_TREE: d0dd42d6a88fbc2a6abca4f6b2dfde7e35520c5a /home/user/.capy/work/gst976/wt\nFindings:\n1. **High**: retry loop never backs off (q.ts:78)\n2. Low - log line is noisy\nRecommendation: Fix the retry because it DoSes the pool\n","main":["High","Low"]},{"source":"pinned","reader":"specialist/testing","output":"REVIEWED_TREE: d0dd42d6a88fbc2a6abca4f6b2dfde7e35520c5a /home/user/.capy/work/gst976/wt\n{\"severity\":\"CRITICAL\",\"path\":\"a.ts\",\"line\":3,\"category\":\"logic\",\"summary\":\"x\",\"specialist\":\"testing\"}\n","main":["CRITICAL"]},{"source":"pinned","reader":"red-team","output":"REVIEWED_TREE: d0dd42d6a88fbc2a6abca4f6b2dfde7e35520c5a /home/user/.capy/work/gst976/wt\n{\"severity\":\"CRITICAL\",\"path\":\"b.ts\",\"line\":1,\"category\":\"x\",\"summary\":\"y\",\"specialist\":\"red-team\"}\n","main":["CRITICAL"]},{"source":"pinned","reader":"specialist/testing","output":"REVIEWED_TREE: d0dd42d6a88fbc2a6abca4f6b2dfde7e35520c5a /home/user/.capy/work/gst976/wt\n{\"severity\":\"CRITICAL\",\"path\":\"a.ts\",\"line\":3,\"category\":\"logic\",\"summary\":\"x\",\"specialist\":\"testing\"}\n{\"severity\":\"INFORMATIONAL\",\"path\":\"b.ts\",\"line\":1,\"category\":\"n\",\"summary\":\"y\",\"specialist\":\"testing\"}\n","main":["CRITICAL","INFORMATIONAL"]},{"source":"pinned","reader":"r","output":"NO FINDINGS\n","main":[]},{"source":"pinned","reader":"r","output":"## High\n1. The auth check skips webhooks (a.ts:4)\n - Impact: nested evidence\n **Fix:**\n- second one\n## Notes\n- not a finding\n**Critical findings**\n- real one\n**Recommendations**\n- consider renaming foo\n","main":["High","High","Critical"]},{"source":"pinned","reader":"r","output":"1. **Unbounded retry in queue.ts:78** (FIXABLE): loops on 429\n2. `auth.ts:42` skips token expiry check — INVESTIGATE\n### CRITICAL: missing auth check in api.ts:10\n#### [P1] token not expired\n### High: authorization bypass (auth.ts:14)\n- **HIGH**: leak\n| High | race in queue.ts:4 |\nClassification: INVESTIGATE\n7. **Informational: reads every file**","main":["FIXABLE","INVESTIGATE","CRITICAL","P1","High","HIGH","High","INVESTIGATE","Informational"]},{"source":"pinned","reader":"r","output":"FIXABLE: race in a.ts:3\nINVESTIGATE: odd lock\nRecommendation: Fix the race because x\n","main":["FIXABLE","INVESTIGATE"]},{"source":"pinned","reader":"outside-voice","output":"No CRITICAL issues except the token leak in log.ts:3","main":["CRITICAL"]},{"source":"pinned","reader":"fix-check","output":"No CRITICAL issues except the token leak in lLine truncated
|
||||
@@ -4,7 +4,7 @@ import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { BriefError, checkReviewedTree, promptHash, renderBrief, sectionTexts, workspaceParagraph, workspaceStep } from '../lib/review-brief';
|
||||
import { collect, judge, parseFindings, readJsonl, type MergedLine } from '../lib/review-ledger';
|
||||
import { collect, judge, parseFindings, rank, readJsonl, type MergedLine } from '../lib/review-ledger';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const BIN = path.join(ROOT, 'bin', 'gstack-review-dispatch');
|
||||
@@ -305,8 +305,9 @@ describe('gstack-review-dispatch sent, collect, judge and verify: nothing change
|
||||
|
||||
// Value: protects=a negated severity word still collects at its label, so a reader's closing prose ("found nothing CRITICAL") is a
|
||||
// known phantom lead judgment folds and never a dropped line (GSTA-167 S1 reverts GST-766's negation drop); fails_when=any negated
|
||||
// form collects nothing or a weaker label; why_new=GST-766's drop lost blockers named in exception clauses; seam=parseFindings;
|
||||
// no_claim=that a negated line is a real finding (judge folds phantoms)
|
||||
// form collects nothing or a weaker label, except under the finding's own Severity field, which decides it (GST-976);
|
||||
// why_new=GST-766's drop lost blockers named in exception clauses; seam=parseFindings; no_claim=that a negated line is a real
|
||||
// finding (judge folds phantoms)
|
||||
test('a negated severity word still collects at its label: GEVA-92 fix-check#5 and its negated forms are known phantoms', () => {
|
||||
const fixCheck = [
|
||||
'REVIEWED_TREE: bedd511efa0fa16c9277821073c68e438dc760d2 /workspace/gstack-evals',
|
||||
@@ -332,7 +333,7 @@ describe('gstack-review-dispatch sent, collect, judge and verify: nothing change
|
||||
expect(parseFindings('r', 'No CRITICAL lock guards the write in Makefile:7\nSeverity: CRITICAL').map(f => f.label)).toEqual(['CRITICAL']);
|
||||
expect(parseFindings('r', '1. auth.ts:3 LOW: log line too verbose\nNo CRITICAL lock guards the write in Makefile:7\nSeverity: CRITICAL').map(f => [f.label, f.raw])).toEqual([['LOW', '1. auth.ts:3 LOW: log line too verbose'], ['CRITICAL', 'No CRITICAL lock guards the write in Makefile:7 | Severity: CRITICAL']]);
|
||||
expect(parseFindings('r', '1. auth.ts:3 LOW: log line too verbose\nNo CRITICAL issues remain.').map(f => f.label)).toEqual(['LOW', 'CRITICAL']);
|
||||
expect(parseFindings('outside-voice', '- a.ts:3 race. Severity: Low. Blocks: no\n Separately, nothing CRITICAL about b.ts:9, but it leaks fds.\n Severity: High').map(f => [f.label, f.raw])).toEqual([['Low', '- a.ts:3 race. Severity: Low. Blocks: no'], ['CRITICAL', 'Separately, nothing CRITICAL about b.ts:9, but it leaks fds. | Severity: High']]);
|
||||
expect(parseFindings('outside-voice', '- a.ts:3 race. Severity: Low. Blocks: no\n Separately, nothing CRITICAL about b.ts:9, but it leaks fds.\n Severity: High').map(f => [f.label, f.raw])).toEqual([['Low', '- a.ts:3 race. Severity: Low. Blocks: no'], ['High', 'Separately, nothing CRITICAL about b.ts:9, but it leaks fds. | Severity: High']]);
|
||||
expect(parseFindings('r', '1. A CRITICAL race in queue.ts:78 that no lock covers').map(f => f.label)).toEqual(['CRITICAL']);
|
||||
expect(parseFindings('r', '- A divide by zero CRITICAL bug in parse()').map(f => f.label)).toEqual(['CRITICAL']);
|
||||
expect(parseFindings('adversarial-native', 'FIXABLE: no timeout on the fetch in a.ts:3').map(f => f.label)).toEqual(['CRITICAL']);
|
||||
@@ -406,8 +407,8 @@ describe('gstack-review-dispatch sent, collect, judge and verify: nothing change
|
||||
expect(parseFindings('outside-voice', '> Blocks: yes').map(f => f.label)).toEqual(['CRITICAL']);
|
||||
});
|
||||
// Value: protects=a later finding's Severity/Blocks line never lands on the finding before it, and an explicit Severity label
|
||||
// decides its line (#3660 pass 2 Red Team); fails_when=the later blocker's text is lost and the earlier finding takes its
|
||||
// CRITICAL, or 'Severity: Low (not High)' collects as High; seam=parseFindings
|
||||
// decides its line unless a stronger label leads that line (#3660 pass 2 Red Team, GST-976); fails_when=the later blocker's text
|
||||
// is lost and the earlier finding takes its CRITICAL, or 'Severity: Low (not High)' collects as High; seam=parseFindings
|
||||
test('prose after a finding\'s field, a blank-line paragraph or an indented block starts the next finding; an explicit Severity label wins', () => {
|
||||
const raws = (o: string) => parseFindings('outside-voice', o).map(f => [f.label, f.raw]);
|
||||
const two = [['Low', 'A in a.ts:1. | Severity: Low'], ['CRITICAL', 'B in b.ts:2. | Blocks: yes']];
|
||||
@@ -603,7 +604,7 @@ describe('gstack-review-dispatch sent, collect, judge and verify: nothing change
|
||||
for (const line of ['There are no CRITICAL findings.', 'No CRITICAL or High findings to report.', 'I found nothing CRITICAL.', "So there's nothing at CRITICAL.",
|
||||
'Neither finding is CRITICAL.', 'Nothing here is CRITICAL.', 'Neither fix is CRITICAL.', 'Nothing in this diff is CRITICAL.', 'This is not CRITICAL.'])
|
||||
expect([line, parseFindings('outside-voice', line).map(f => f.label)]).toEqual([line, ['CRITICAL']]);
|
||||
expect(parseFindings('outside-voice', 'Severity: Low (not High)').map(f => f.label)).toEqual(['High']);
|
||||
expect(parseFindings('outside-voice', 'Severity: Low (not High)').map(f => f.label)).toEqual(['Low']);
|
||||
});
|
||||
|
||||
// Value: protects=item layouts collect as on main: an indented JSON finding keeps its JSON severity, an indented label heading sets
|
||||
@@ -626,6 +627,82 @@ describe('gstack-review-dispatch sent, collect, judge and verify: nothing change
|
||||
.toEqual([['CRITICAL', '{"severity":"low","summary":"race"} | this blocks the merge. Blocks: yes']]);
|
||||
});
|
||||
|
||||
// Value: protects=a finding's own Severity and Blocks fields set its collected label, so a severity word in its body never raises it
|
||||
// (gstack-evals #143: the outside voice marked six findings Blocks: no, and the "P0" in #1, #2 and #4 was the plan's phase name),
|
||||
// while a real P0 or CRITICAL in label position, a Severity: P0 field and Blocks: yes still collect as blocking; fails_when=a body
|
||||
// token outranks the finding's fields, or a label-position P0/CRITICAL or Blocks: yes collects below CRITICAL rank;
|
||||
// why_new=GST-976: #1, #2 and #4 collected as P0, judge refused the downgrade (JUDGMENT_REJECTED), verify stayed dirty and the
|
||||
// receipt needed accepted_by; seam=parseFindings, collect and judge; no_claim=that a body-only label on a line with no Severity field
|
||||
// is a real finding (it still collects at that label, a known phantom judge may fold; GSTA-167 S1 keeps body labels with no fields
|
||||
// because dropping them lost blockers)
|
||||
test('GST-976: a severity word in a finding body never outranks its Severity and Blocks fields; real P0 and CRITICAL still block', () => {
|
||||
const blocking = (label: string) => rank(label) === 4;
|
||||
// The #143 outside-voice reply, cut to each finding's title, its P0 sentence and its fields (gstack-evals is private).
|
||||
const reply = [
|
||||
'The problems below are about accounting and what the post leaves out. None of them breaks a test, production or custody.',
|
||||
'',
|
||||
'1. **The per-task cost leaves out model spend.** The P0 and bulk go or no-go will re-project cost from these figures, so they will come out too low. Severity: Medium. Blocks: no.',
|
||||
'',
|
||||
'2. **One round-2 K1 pair is not accounted for.** The post sets a precedent for P0 yield math, so that drop should be named. Severity: Low. Blocks: no.',
|
||||
'',
|
||||
'3. **The two probe populations are not reconciled.** The stage table probes 189 candidates. Severity: Low. Blocks: no.',
|
||||
'',
|
||||
"4. **The P0 impact of the shortfall is missing.** The plan's P0 (line 323) needs 6 K4 replays and 2 decoys. The post says the shortfall but not that it blocks P0. Severity: Low. Blocks: no.",
|
||||
'',
|
||||
'6. **Some wording is ambiguous.** Two examples:',
|
||||
' - "Repositories 4 (3 for build tasks)" is set against a target of "at least 5 repositories".',
|
||||
'',
|
||||
' Severity: Low. Blocks: no.',
|
||||
'',
|
||||
"Recommendation: ship after a one-line fix because finding 1's per-task cost leaves out measured model spend.",
|
||||
].join('\n');
|
||||
const d = tmp();
|
||||
const merged = path.join(d, 'merged.jsonl');
|
||||
const r = collect(path.join(d, 'l.jsonl'), 'outside-voice', reply, merged);
|
||||
expect(r.lines.map(l => [l.label, l.stated])).toEqual([['Medium', { severity: 'Medium', blocks: 'no' }], ...Array(4).fill(['Low', { severity: 'Low', blocks: 'no' }])]);
|
||||
expect(r.lines.filter(l => blocking(l.label)).map(l => l.id)).toEqual([]);
|
||||
const judged = path.join(d, 'judged.jsonl');
|
||||
fs.writeFileSync(judged, r.lines.map(l => JSON.stringify({ id: l.id, severity: l.stated!.severity })).join('\n') + '\n');
|
||||
expect(judge(readJsonl<MergedLine>(merged), readJsonl(judged))).toMatchObject({ ok: true, downgraded: [] });
|
||||
const real: Array<[string, string]> = [['[P0] Token leak in log.ts:3', 'P0'], ['P0: token leak in log.ts:3', 'P0'], ['1. Token leak in log.ts:3\n Severity: P0', 'P0'],
|
||||
['1. The P0 pilot drops the last row (q.ts:7). Severity: P0. Blocks: no', 'P0'], ['**Critical**\n- Token leak in log.ts:3', 'Critical'], ['## P0\n- Token leak in log.ts:3', 'P0'],
|
||||
['### CRITICAL: missing auth check in api.ts:10', 'CRITICAL'], ['1. The P0 pilot loses rows on retry (r.ts:2). Severity: Low. Blocks: yes', 'CRITICAL'],
|
||||
['1. The P0 pilot loses rows on retry (r.ts:2)\n Severity: Medium\n Blocks: yes', 'CRITICAL']];
|
||||
for (const [output, label] of real) {
|
||||
const got = parseFindings('outside-voice', output).map(f => f.label);
|
||||
expect([output, got, got.map(blocking)]).toEqual([output, [label], [true]]);
|
||||
}
|
||||
expect(parseFindings('outside-voice', '1. The P0 pilot table drops a row (t.md:3)').map(f => f.label)).toEqual(['P0']);
|
||||
// Review of this fix (gst976-p1): a locus-first or emoji-led label, an inline field the next field line follows, a second field on
|
||||
// the line, and prose that only mentions a severity never lower a blocker the reader labeled.
|
||||
const kept: Array<[string, string]> = [['1. api.ts:10 CRITICAL: missing auth. Severity: Low', 'CRITICAL'], ['1. api.ts:10 CRITICAL: missing auth\n Severity: Low', 'CRITICAL'],
|
||||
['1. 🔴 CRITICAL: missing auth. Severity: Low', 'CRITICAL'], ['1. Token leak in auth.ts:3 lets anyone read sessions. Severity: Critical\n\nSeverity: Low', 'Critical'],
|
||||
['- auth.ts:3 token leak. Severity: Low (pre-exploit), Severity: Critical (once exposed)', 'Critical'],
|
||||
['- Missing auth on DELETE /users lets any user wipe accounts (CRITICAL); the old ticket filed it at severity low.', 'CRITICAL'],
|
||||
['- api.ts:10 lets any user wipe accounts. Earlier triage said Severity: Low; it is CRITICAL.', 'CRITICAL']];
|
||||
for (const [output, label] of kept) expect([output, parseFindings('outside-voice', output).map(f => f.label)]).toEqual([output, [label]]);
|
||||
// gst976-p2: an all-caps lead before a space, comma or parenthesis; a field after a locus's parenthesis or with no colon at the
|
||||
// line's start; and a long run of marks before a label stays linear.
|
||||
const p2: Array<[string, string]> = [['1. CRITICAL missing auth. Severity: Low', 'CRITICAL'], ['1. CRITICAL, missing auth on DELETE. Severity: Low', 'CRITICAL'],
|
||||
['1. CRITICAL (api.ts:10) missing auth. Severity: Low', 'CRITICAL'], ['1. The P0 pilot table drops a row (t.md:3) Severity: Low', 'Low'],
|
||||
['1. The P0 pilot table drops a row\nSeverity Low', 'Low'], ['1. The P0 pilot table drops a row\n**Severity** Low', 'Low'],
|
||||
['1. Bypass x.ts:3 is CRITICAL (Severity: Low in pass 1)', 'CRITICAL'], ['Still open: x.ts:3 is CRITICAL, Severity: Low in pass 1 understated it', 'CRITICAL'],
|
||||
['1. lib/x.ts:42-58 CRITICAL: drops rows. Severity: Low', 'CRITICAL'], ['1. lib/x.ts:42:7 CRITICAL: drops rows. Severity: Low', 'CRITICAL'],
|
||||
['1. (lib/x.ts:42, 50) CRITICAL: drops rows. Severity: Low', 'CRITICAL'],
|
||||
['1. The P0 pilot table drops a row\nSeverity: Low: only the pilot', 'Low'], ['1. The P0 pilot table drops a row\nSeverity: Low because it is internal', 'Low'],
|
||||
['1. The P0 pilot table drops a row\nSeverity: Low/Medium', 'P0'], ['Severity: Low: only the P0 pilot table', 'Low'],
|
||||
['1. The P0 pilot drops a row. Severity: Low/P3', 'P0'], ['1. drops rows\nSeverity: High/Critical', 'Critical'],
|
||||
['Severity: Medium/CRITICAL', 'CRITICAL'], ['1. x.ts:3 drops prod rows. Severity: Low/CRITICAL', 'CRITICAL'],
|
||||
['1. x.ts:3 is CRITICAL (Severity: Low/Medium in pass 1)', 'CRITICAL'], ['1. x.ts:3 is CRITICAL (Severity: Low: pass 1)', 'CRITICAL'],
|
||||
['Still open: x.ts:3 is P0, Severity: Low: pass 1 understated it', 'P0']];
|
||||
for (const [output, label] of p2) expect([output, parseFindings('outside-voice', output).map(f => f.label)]).toEqual([output, [label]]);
|
||||
const t0 = performance.now();
|
||||
for (const run of ['='.repeat(60), '-'.repeat(60), '|' + '-'.repeat(60) + '|', '🔴'.repeat(60), '. '.repeat(60), '-.'.repeat(60), '`'.repeat(60)])
|
||||
for (const tail of [' P0 pilot table', ' High-risk', ' CRITICAL findings follow', ' a.ts:3 P0 pilot']) parseFindings('outside-voice', run + tail + '. Severity: Low');
|
||||
parseFindings('outside-voice', '-'.repeat(16000) + ' P0 pilot table. Severity: Low');
|
||||
expect(performance.now() - t0).toBeLessThan(1000);
|
||||
});
|
||||
|
||||
// Value: protects=a non-review the same dispatch later answers with a review is voided (counted, VERIFY clean), and a failed outside
|
||||
// voice recorded with --unavailable reads UNAVAILABLE (missing coverage, never a review) without blocking, while --unavailable is
|
||||
// refused for any other reader; fails_when=one outside-voice timeout or one refusal makes VERIFY dirty for good, or --unavailable
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
import { parseFindings, rank } from '../lib/review-ledger';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const LABELS = 'CRITICAL|INFORMATIONAL|Critical|Informational|High|HIGH|Medium|MEDIUM|Low|LOW|P[0-3]|FIXABLE|INVESTIGATE';
|
||||
const CHAIN = `(?:${LABELS})(?:\\s*[/→]\\s*\\**\\s*(?:${LABELS}))*`;
|
||||
/** A finding's own Severity field as GST-976 defines it: at a line's start (colon optional), or after a separator with its colon and a closed clause. */
|
||||
const FIELD = new RegExp(`(?:(?:^|\\| )(?:[-*+>]\\s+|\\d+[.)]\\s+)?\\**\\s*Severity\\**\\s*:?\\s*\\**\\s*(${CHAIN})(?![A-Za-z0-9])|(?:[.;,|()]\\s*|\\s[—–-]\\s*)\\**\\s*Severity\\**\\s*:\\s*\\**\\s*(${CHAIN})(?![A-Za-z0-9])(?=\\**\\s*(?:$|[.;,|()*—–]|-\\s|Blocks)))`, 'gi');
|
||||
const LEAD = new RegExp(`(?:^|\\| )[^\\p{L}\\p{N}]*(?:\\d+[.)]\\s*[^\\p{L}\\p{N}]*)?(?:\\w[\\w./-]*:\\d+(?:[-–:,]\\s*\\d+)*\\S*\\s+[^\\p{L}\\p{N}]*)?(?:${LABELS})(?![a-z])`, 'u');
|
||||
const HEADING = new RegExp(`^(?:#{1,6}\\s*|\\*\\*)(?:${LABELS})\\b`, 'm');
|
||||
const LABEL = new RegExp(`${LABELS}`, 'gi');
|
||||
|
||||
describe('parseFindings is never weaker than main (GST-976 invariant)', () => {
|
||||
// Value: protects=no input the suite pins, no reader fixture and no Severity field shape collects below the label main (82e9d7ec)
|
||||
// collected, unless the finding's own Severity field states exactly one label, and such a finding collects at that label unless a
|
||||
// leading label, a label heading, Blocks: yes or the native voice's FIXABLE raises it; fails_when=a field chain (High/Critical) keeps
|
||||
// its weaker label (b3ddd6c) or a field whose clause runs on stops counting and a body P0 raises the finding (f3810d4);
|
||||
// why_new=three GST-976 review rounds each found a shape the row tables missed; seam=parseFindings; no_claim=that main's label was
|
||||
// right (it is the floor), or coverage of stored private reader outputs (checked outside the repo)
|
||||
test('every pinned input, reader fixture and generated field shape', () => {
|
||||
const { entries, shapes } = JSON.parse(fs.readFileSync(path.join(ROOT, 'test/fixtures/review-ledger/main-labels.json'), 'utf8'));
|
||||
const { bodies, fields: tails, joins, readers, generated } = shapes;
|
||||
for (const [b, f, j, r, main] of generated) entries.push({ reader: readers[r], output: bodies[b] + joins[j] + tails[f], main });
|
||||
const bad: string[] = [];
|
||||
let findings = 0;
|
||||
for (const e of entries as Array<{ reader: string; output?: string; fixture?: string; main: string[] }>) {
|
||||
const output = e.output ?? fs.readFileSync(path.join(ROOT, 'test/fixtures', e.fixture!), 'utf8');
|
||||
const got = parseFindings(e.reader, output);
|
||||
if (!e.main.length) continue;
|
||||
if (got.length !== e.main.length) { bad.push(`count ${got.length}/${e.main.length} ${JSON.stringify(output.slice(0, 80))}`); continue; }
|
||||
got.forEach((f, i) => {
|
||||
findings++;
|
||||
const fields = [...f.raw.matchAll(FIELD)].map(m => (m[1] ?? m[2])!);
|
||||
const single = fields.length === 1 && fields[0]!.match(LABEL)!.length === 1 ? fields[0]!.match(LABEL)![0] : undefined;
|
||||
const raised = /Blocks:?\**:?\s*\**\s*yes/i.test(f.raw) || (e.reader.startsWith('adversarial-native') && /FIXABLE/.test(f.raw)) || LEAD.test(f.raw) || HEADING.test(output);
|
||||
const ok = single === undefined ? (rank(f.label) ?? 0) >= (rank(e.main[i]) ?? 0)
|
||||
: (rank(f.label) ?? 0) >= (rank(single) ?? 0) && (raised || (rank(f.label) ?? 0) === (rank(single) ?? 0));
|
||||
if (!ok) bad.push(`${e.reader} main=${e.main[i]} got=${f.label} field=${single ?? '-'} ${JSON.stringify(f.raw.slice(0, 100))}`);
|
||||
});
|
||||
}
|
||||
expect(bad).toEqual([]);
|
||||
expect(findings).toBeGreaterThan(1000);
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user