mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-06 00:44:07 +01:00
fix(review): collect stops turning a reader's no-defect closing sentence into a CRITICAL finding (#3990)
* fix(ship): a reader's no-defect closing sentence is not a CRITICAL finding (GST-435)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs(changelog): fragment for GST-435
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(ship): linear summary grammar, keep a summary line with other prose on record, skip wrapped continuations (GST-435 pass-1 red team)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Review-Finding: p1/red-team#1 reader=red-team
Review-Finding: p1/red-team#2 reader=red-team
Review-Finding: p1/red-team#3 reader=red-team
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(ship): a summary line under a run-on or exception line in its paragraph keeps its label (GST-435 outside voice)
Review-Finding: p1/outside-voice#2 reader=outside-voice
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs(ship): closingSummary checks only the line directly above (GST-435 fix-check note)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* Revert "docs(ship): closingSummary checks only the line directly above (GST-435 fix-check note)"
This reverts commit daf526b94a.
---------
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
7ba250ba60
commit
b244e6d11a
8 files changed
+226
-31
No files matched your search
@@ -0,0 +1 @@
|
||||
- **A reviewer's "I found nothing CRITICAL" no longer counts as a CRITICAL finding.** /ship's review collect step used to turn a reader's own closing sentence into a blocking finding when the sentence only denied a severity ("I found nothing CRITICAL.", "I found no defect that is CRITICAL.", "Neither finding is CRITICAL."), so the review loop counted blockers that named no defect and a Captain had to grant an exception. A sentence that is wholly a denial, a count of findings or "nothing blocks", with no file, line, exception clause or field on its line, now collects nothing at the label it denies; when the same line also holds other prose, that line stays on record as an INFORMATIONAL note instead of a blocker. Every real blocker still collects: a file:line, an exception clause ("except the token leak in log.ts:3"), "nothing else" or "no other", a defect named in the same sentence, another labelled sentence, a list item or a Severity or Blocks field keeps the line at its label. (GST-435)
|
||||
+14
-2
@@ -712,8 +712,20 @@ ones, and the arm lost blockers. `bin/gstack-review-dispatch` closes both ends:
|
||||
`Severity` or `Blocks` line comes, so those fields never split off as a second record (GST-1031). An unnumbered, unlabeled bullet with no file:line
|
||||
under a paragraph finding whose Severity or Blocks line has not come yet is that finding's detail, so the field line after it
|
||||
lands on the finding; a numbered item is its own finding.
|
||||
A negated severity word still collects at its label ("found nothing CRITICAL", "No P0/P1 issues"), so a reader's
|
||||
closing summary is a known phantom: it collects at that label. When it is a summary candidate (no lead label, no
|
||||
A reader's own no-defect closing sentence is no finding at the label it denies (GST-435): a prose line with no
|
||||
list, heading, bold or lead label, no `Severity` or `Blocks` field, no FIXABLE or INVESTIGATE, no file:line or path
|
||||
and no exception clause, with no line above it in its paragraph that runs on (no closing period) or names an
|
||||
exception, and whose every labelled sentence is
|
||||
wholly a denial ("I found nothing CRITICAL.", "I found no defect that is CRITICAL.", "No P0/P1 issues.", "Neither
|
||||
finding is CRITICAL."), a count ("I found three INFORMATIONAL issues and nothing CRITICAL" collects at
|
||||
INFORMATIONAL) or "nothing blocks", with labels spelled as collect spells them. Such a line collects nothing when it
|
||||
is only those sentences; when it also holds an unlabelled sentence ("Here is what I checked:", or one that may
|
||||
describe a defect) it stays on record at INFORMATIONAL, never as a blocker. A quantifier that points at a finding
|
||||
elsewhere ("nothing else", "no other", "remaining", "new"), a named defect in the same sentence ("No auth check
|
||||
guards the admin route, so nothing CRITICAL") or any other labelled sentence on the line keeps the whole line at its
|
||||
label, and a reply whose only text is a no-defect sentence, with no clean marker, is still no review. Any other negated severity word
|
||||
still collects at its label, so a closing summary with a clause of its own ("I checked these against the post and
|
||||
found nothing CRITICAL") is a known phantom: it collects at that label. When it is a summary candidate (no lead label, no
|
||||
`Severity` or `Blocks` field, and no file:line or exception clause in any sentence naming a label: "so nothing is
|
||||
CRITICAL", "the test covers the CRITICAL fix"), collect marks it `summary_candidate` and lead judgment may lower it
|
||||
with a reason, which judge records as `dismissed` (GST-1031); otherwise lead judgment can fold it only into a finding
|
||||
|
||||
+41
-4
@@ -90,7 +90,7 @@ export const statesNoBlocks = (raw: string) => { if (raw.startsWith('{') || BLOC
|
||||
/**
|
||||
* A summary candidate (GST-1031): a prose finding whose every severity word is denied ("so nothing is CRITICAL", "No CRITICAL issues")
|
||||
* or names a fix ("the test covers the CRITICAL fix"), with no lead label, no Severity or Blocks field, no FIXABLE or INVESTIGATE tag,
|
||||
* and no file:line or exception clause in any sentence naming a label. A defect called CRITICAL ("this is CRITICAL") is none. It still collects at its label (a blocker named in another sentence stays on record, GST-435); lead
|
||||
* and no file:line or exception clause in any sentence naming a label. A defect called CRITICAL ("this is CRITICAL") is none. It still collects at its label (a blocker named in another sentence stays on record), unless closingSummary reads it as a no-defect sentence (GST-435); lead
|
||||
* judgment may lower it with a reason, which judge records as dismissed. "No CRITICAL issues except the token leak in log.ts:3" is no candidate.
|
||||
*/
|
||||
const SENTENCE = /(?<=[.!?])\s+/;
|
||||
@@ -101,6 +101,38 @@ const DENIED_OR_FIXED = new RegExp(`(?:(?<![A-Za-z])(?:nothing|none|no|neither|z
|
||||
export const summaryCandidate = (t: string) => ANY_LABEL.test(t) && !ANY_LABEL.test(t.replace(DENIED_OR_FIXED, ' ')) && !t.startsWith('{') && !t.startsWith('**') && !LIST_ITEM.test(t) && !HEADING.test(t) && !LEAD_LABEL.test(t)
|
||||
&& !t.match(SEVERITY_FIELD) && !t.match(STATED_BLOCKS) && [...t.matchAll(ALL_LABELS)].every(m => !!rank(m[1]))
|
||||
&& t.split(SENTENCE).filter(s => ANY_LABEL.test(s)).every(s => !PATH_LOCUS.test(s) && !EXCEPTION.test(s));
|
||||
/**
|
||||
* A reader's own no-defect closing sentence (GST-435: gstack-evals #194 "I found nothing CRITICAL.", #202 "I found no defect that is
|
||||
* CRITICAL..."): a prose line whose paragraph has no line above it that runs on or names an exception, with no list, heading, bold or lead label, no
|
||||
* Severity or Blocks field, no FIXABLE or INVESTIGATE, no file:line or path and no exception clause, whose every labelled sentence is
|
||||
* wholly one of these clauses, its labels spelled as ALL_LABELS spells them: a denial (`no CRITICAL issues`, `no defect that is
|
||||
* CRITICAL`, `nothing at CRITICAL`, `there are no High findings`, `neither finding is CRITICAL`), a count (`I found three
|
||||
* INFORMATIONAL issues`) or `nothing blocks`. It yields the labels its counts name and whether the line also holds an unlabelled
|
||||
* sentence, which may describe a defect, so parseFindings keeps that line on record at INFORMATIONAL; the denied label is never a
|
||||
* finding. A quantifier that points at a finding elsewhere (`nothing else`, `no other`, `remaining`), a named defect in the same
|
||||
* sentence (`No auth check guards the admin route, so nothing CRITICAL`) or any other labelled sentence keeps the line as it was.
|
||||
*/
|
||||
const SEV = `[*\`]*(?:CRITICAL|INFORMATIONAL|Critical|Informational|High|HIGH|Medium|MEDIUM|Low|LOW|P[0-3])[*\`]*`;
|
||||
const SEV_LIST = `${SEV}(?:\\s*(?:,|/|\\bor\\b|\\band\\b|\\bnor\\b)\\s*${SEV})*(?![A-Za-z0-9])`;
|
||||
const NOUN = '(?:issues?|findings?|defects?|bugs?|problems?|regressions?|blockers?|concerns?|items?|notes?|ones?)';
|
||||
const DENY = `(?<![A-Za-z])(?:(?:no|zero)\\s+${SEV_LIST}(?:\\s+${NOUN})?|(?:no|zero)\\s+${NOUN}\\s+(?:that\\s+is|that's|that\\s+are|which\\s+is|which\\s+are|at|rated)\\s+(?:at\\s+)?${SEV_LIST}(?:\\s+(?:severity|level))?`
|
||||
+ `|nothing\\s+(?:at\\s+)?${SEV_LIST}(?:\\s+(?:severity|level))?|there(?:'s|\\s+is|\\s+are|\\s+was|\\s+were)\\s+(?:still\\s+)?(?:no|zero|nothing(?:\\s+at)?)\\s+${SEV_LIST}(?:\\s+${NOUN})?`
|
||||
+ `|(?:nothing|none\\s+of\\s+(?:them|these|those|the\\s+(?:findings|fixes|notes|issues|changes))|neither(?:\\s+(?:finding|fix|note|issue|change|one)|\\s+of\\s+(?:them|these|the\\s+(?:findings|fixes|notes))))`
|
||||
+ `(?:\\s+(?:here|in\\s+(?:this|the)\\s+(?:diff|change|fix|patch|range|PR)))?\\s+(?:is|are|was|were)\\s+(?:at\\s+)?${SEV_LIST})`;
|
||||
const SUMMARY_ITEM = `(?:(?:(?:I|we)\\s+(?:also\\s+)?(?:found|find|see|saw|spotted|identified|noticed|observed|have|had|raised|report)\\s+)?`
|
||||
+ `(?:(?:one|two|three|four|five|six|seven|eight|nine|ten|\\d+|a\\s+few|several|some|an?)\\s+${SEV_LIST}\\s+${NOUN}|${DENY})`
|
||||
+ `|(?:nothing|none\\s+of\\s+(?:them|these|the\\s+(?:findings|fixes|notes))|no\\s+(?:finding|issue)s?)\\s+(?:is\\s+|are\\s+)?(?:blocking|blocks?)(?:\\s+(?:landing|the\\s+merge|merging))?)`
|
||||
+ `(?:\\s+(?:here|found|to\\s+report|remain(?:s|ing)?|in\\s+(?:this|the)\\s+(?:diff|change|changes|fix|fixes|patch|range|PR|review|pass|commit)))*`;
|
||||
const SUMMARY_SENTENCE = new RegExp(`^(?:(?:so|overall|in\\s+short|in\\s+summary|in\\s+total|to\\s+sum\\s+up)[,:]?\\s+)?${SUMMARY_ITEM}(?:(?:\\s*[,;]\\s*|\\s+)(?:and\\s+)?${SUMMARY_ITEM})*\\s*[.!…]*$`);
|
||||
const DENIED = new RegExp(DENY, 'g');
|
||||
export const closingSummary = (t: string): { labels: string[]; prose: boolean } | undefined => {
|
||||
if (t.startsWith('{') || t.startsWith('**') || LIST_ITEM.test(t) || HEADING.test(t) || LEAD_LABEL.test(t) || t.match(SEVERITY_FIELD) || t.match(STATED_BLOCKS) || BLOCKS_YES.test(t)
|
||||
|| LOCUS.test(t) || PATH_LOCUS.test(t) || EXCEPTION.test(t) || /(?<![A-Za-z0-9])(?:FIXABLE|INVESTIGATE)(?![A-Za-z0-9])/i.test(t)) return undefined;
|
||||
const sentences = t.split(SENTENCE).map(s => s.trim());
|
||||
const labelled = sentences.filter(s => ANY_LABEL.test(s)).map(s => s.replace(/^[A-Z](?=[a-z])/, c => c.toLowerCase()));
|
||||
if (!labelled.length || !labelled.every(s => SUMMARY_SENTENCE.test(s))) return undefined;
|
||||
return { labels: labelled.flatMap(s => [...s.replace(DENIED, ' ').matchAll(ALL_LABELS)].map(m => m[1]!)), prose: labelled.length < sentences.length };
|
||||
};
|
||||
/**
|
||||
* The native reader's own verdict line (GSTA-258 L4: #3901 "There are three low-severity findings, all FIXABLE, and none blocks."): a prose
|
||||
* line, no list item, heading, lead label, file:line, Severity or Blocks field, whose only labels are FIXABLE or INVESTIGATE and which says
|
||||
@@ -143,6 +175,7 @@ export function parseFindings(reader: string, output: string, fixableRaises = tr
|
||||
let current: Finding | null = null;
|
||||
let head = '';
|
||||
let gap = false;
|
||||
let prev = '';
|
||||
let treeSeen = false;
|
||||
const role = reader.split('/')[0];
|
||||
const named = (l: string) => (role === 'adversarial-native' && l.toUpperCase() === 'FIXABLE' ? 'CRITICAL' : l);
|
||||
@@ -151,8 +184,10 @@ export function parseFindings(reader: string, output: string, fixableRaises = tr
|
||||
const fieldOf = (t: string) => { const ls = [...t.matchAll(SEVERITY_FIELD)].flatMap(m => (m[1] ?? m[2]!).match(FIELD_LABELS)!); return ls.length === 1 && (fixableRaises || role !== 'adversarial-native' || rank(ls[0]!)) ? ls[0] : undefined; };
|
||||
const critical = (t: string) => BLOCKS_YES.test(t) || (fixableRaises && role === 'adversarial-native' && /(?<![A-Za-z0-9])FIXABLE(?![A-Za-z0-9])/.test(t));
|
||||
const leadOf = (t: string) => { const l = LEAD_LABEL.exec(t)?.[1]; return l === undefined || !fixableRaises ? l : named(l); };
|
||||
const labelOf = (t: string) => {
|
||||
const labelOf = (t: string, continued: boolean) => {
|
||||
if (role === 'adversarial-native' && quietSummary(t)) return 'FIXABLE';
|
||||
const summary = continued ? undefined : closingSummary(t);
|
||||
if (summary) return strongestOf([...summary.labels.map(named), summary.prose ? 'INFORMATIONAL' : undefined]);
|
||||
const field = fieldOf(t);
|
||||
if (field === undefined) return strongest(t);
|
||||
return critical(t) ? 'CRITICAL' : strongestOf([leadOf(t), field])!;
|
||||
@@ -160,13 +195,15 @@ export function parseFindings(reader: string, output: string, fixableRaises = tr
|
||||
const push = (label: string, raw: string, fields: string[] = [], lead?: string): Finding => { const f = { label, raw, fields: new Set(fields), list: LIST_ITEM.test(raw), lead }; out.push(f); return f; };
|
||||
for (const line of output.replace(/\r\n/g, '\n').split('\n')) {
|
||||
const t = line.trim();
|
||||
if (!t) { gap = true; continue; }
|
||||
if (!t) { gap = true; prev = ''; continue; }
|
||||
if (!treeSeen && isReviewedTreeLine(t)) { treeSeen = true; continue; }
|
||||
const afterGap = gap;
|
||||
const continued = !!prev && !HEADING.test(prev) && (EXCEPTION.test(prev) || !/[.!?…}]["'”’)\]*_`]*$/.test(prev));
|
||||
prev = t;
|
||||
gap = false;
|
||||
if (/^NO FINDINGS\.?$/.test(t) || /^Recommendation:/i.test(t)) { current = null; continue; }
|
||||
const indented = line !== line.trimStart();
|
||||
const label = labelOf(t);
|
||||
const label = labelOf(t, continued);
|
||||
const blocks = BLOCKS_YES.test(t);
|
||||
const listItem = !indented && LIST_ITEM.test(t);
|
||||
const locus = LOCUS.test(t);
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
{"severity":"INFORMATIONAL","path":"wiki/plans/p/README.md","line":91,"summary":"The run-order step still says the run goes on when every linked log holds only no-contract-skill lines, where the readout stops it."}
|
||||
{"severity":"INFORMATIONAL","path":"wiki/plans/p/hooks.ts.txt","line":46,"summary":"ls() now swallows every readdirSync error, so an unreadable arm folder is skipped without a word."}
|
||||
|
||||
I found no defect that is CRITICAL. Here is what I checked:
|
||||
- **The silent-hook rule.** All five outcomes the hook can trace are in the logged count, so a working hook can't be read as silent.
|
||||
- **The lost rule.** Its result feeds only the tally and the exposure label, and no other caller reads it.
|
||||
- **Tests weakened.** None.
|
||||
@@ -0,0 +1,9 @@
|
||||
{"severity":"INFORMATIONAL","path":"wiki/plans/p/hooks.ts.txt","line":97,"summary":"The watch still checks a path and then opens it, so a folder swapped between the two can race the check."}
|
||||
{"severity":"INFORMATIONAL","path":"wiki/plans/p/hooks.ts.txt","line":186,"summary":"lost only looks at sessions that have a delta.json, so a stopped session whose log was missed reads log_lost=no."}
|
||||
{"severity":"INFORMATIONAL","path":"wiki/plans/p/README.md","line":173,"summary":"The new sentence says a delta.json listing is enough, which does not hold when the agent can write its own state root."}
|
||||
|
||||
I found nothing CRITICAL. I checked these and found them sound:
|
||||
- **Attachment hashes:** they match the README front matter, and the wiki check passes on this tree.
|
||||
- **Missing-list parsing:** the parse at line 187 matches the hook's trace format (stop-hook.ts:134).
|
||||
- **State root:** it's a real directory, so the realpath check doesn't refuse real logs.
|
||||
- **Dry run:** the planted-link case does catch the old behaviour.
|
||||
@@ -0,0 +1,110 @@
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { collect, judge, ledgerBlockers, parseFindings, readJsonl, recordJudgment, summaryCandidate, type MergedLine } from '../lib/review-ledger';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const fixture = (name: string) => fs.readFileSync(path.join(ROOT, 'test/fixtures/review-ledger', name), 'utf8');
|
||||
const labels = (output: string, reader = 'fix-check') => parseFindings(reader, output).map(f => f.label);
|
||||
|
||||
describe("a reader's no-defect closing sentence is not a finding (GST-435)", () => {
|
||||
// Value: protects=a fix-check's own closing sentence that denies every severity it names ("I found nothing CRITICAL",
|
||||
// "I found no defect that is CRITICAL") collects nothing, so the review loop stops counting a blocker that names no defect;
|
||||
// fails_when=any of these sentences collects a finding; why_new=gstack-evals #194 (blocking_found=4 from "I found nothing
|
||||
// CRITICAL") and #202 (fix-check#3 from "I found no defect that is CRITICAL...") each needed a Captain exception because judge
|
||||
// refuses to lower a collected CRITICAL; seam=parseFindings and collect; no_claim=that every summary shape is recognized (one
|
||||
// that opens with a defect clause, "None of them breaks a caller, so there's nothing at CRITICAL", still collects as a candidate)
|
||||
test('the quoted closing sentences and plain negations collect nothing', () => {
|
||||
for (const line of ['I found nothing CRITICAL.', 'I found nothing CRITICAL', 'I found no defect that is CRITICAL.', 'I found no defect that is CRITICAL...', 'Neither finding is CRITICAL.',
|
||||
'No CRITICAL issues.', 'no CRITICAL issues', 'Nothing blocking, nothing CRITICAL.', 'Nothing blocking; no CRITICAL issues.', 'There are no CRITICAL findings.',
|
||||
'No CRITICAL or High findings to report.', "So there's nothing at CRITICAL.", 'Nothing here is CRITICAL.', 'Neither fix is CRITICAL.', 'Nothing in this diff is CRITICAL.',
|
||||
'No CRITICAL findings remain.', 'We found no P0 or P1 issues.', 'None of the fixes blocks landing, and nothing is CRITICAL.'])
|
||||
expect([line, labels(line)]).toEqual([line, []]);
|
||||
for (const line of ['I found no CRITICAL defects. Here is what I checked:', 'I found nothing CRITICAL. I checked these and found them sound:',
|
||||
'No CRITICAL issues. The webhook signature check was removed entirely.', 'The admin route never checks the session token. I found nothing CRITICAL.'])
|
||||
expect([line, labels(line)]).toEqual([line, ['INFORMATIONAL']]);
|
||||
});
|
||||
|
||||
test('the #194 and #202 fix-check replies collect no blocker: the closing line stays on record as a note', () => {
|
||||
for (const [name, json] of [['gst-435-closing-nothing-critical.txt', 4], ['gst-435-closing-no-defect.txt', 3]] as const) {
|
||||
const d = fs.mkdtempSync(path.join(os.tmpdir(), 'gst435-'));
|
||||
const ledger = path.join(d, 'l.jsonl');
|
||||
const merged = path.join(d, 'm.jsonl');
|
||||
fs.writeFileSync(path.join(d, 'out.txt'), fixture(name));
|
||||
const c = collect(ledger, 'fix-check', fixture(name), merged, path.join(d, 'out.txt'));
|
||||
expect([name, c.review, c.lines.map(l => l.label)]).toEqual([name, true, Array(json).fill('INFORMATIONAL')]);
|
||||
fs.writeFileSync(path.join(d, 'j.jsonl'), c.lines.map(l => JSON.stringify({ id: l.id, severity: l.label })).join('\n') + '\n');
|
||||
expect(recordJudgment(ledger, c.lines, readJsonl(path.join(d, 'j.jsonl')), path.join(d, 'j.jsonl'), merged).ok).toBe(true);
|
||||
expect(ledgerBlockers(ledger)).toEqual([]);
|
||||
fs.rmSync(d, { recursive: true, force: true });
|
||||
}
|
||||
const d = fs.mkdtempSync(path.join(os.tmpdir(), 'gst435-'));
|
||||
const merged = path.join(d, 'm.jsonl');
|
||||
const r = collect(path.join(d, 'l.jsonl'), 'fix-check', fixture('geva-77-fix-check.txt'), merged);
|
||||
expect(r.lines.map(l => [l.id, l.label])).toEqual([['fix-check#1', 'INFORMATIONAL'], ['fix-check#2', 'INFORMATIONAL'], ['fix-check#3', 'INFORMATIONAL']]);
|
||||
expect(r.lines[2]!.raw).toStartWith('Neither finding is CRITICAL.');
|
||||
expect(judge(readJsonl<MergedLine>(merged), r.lines.map(l => ({ id: l.id, severity: l.label }))).ok).toBe(true);
|
||||
fs.rmSync(d, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('a count summary keeps the label it counts and drops only the one it denies', () => {
|
||||
expect(labels('I found three INFORMATIONAL issues and nothing CRITICAL.')).toEqual(['INFORMATIONAL']);
|
||||
expect(labels('I found one High issue and nothing CRITICAL.')).toEqual(['High']);
|
||||
expect(labels('None of the fixes blocks landing; I found four INFORMATIONAL defects and no CRITICAL.')).toEqual(['INFORMATIONAL']);
|
||||
});
|
||||
|
||||
// Value: protects=collect stays linear on a crafted reply: a long run of joined labels that ends in a non-matching token parses in
|
||||
// milliseconds; fails_when=the summary grammar backtracks exponentially (each label matching two ways), which made 24 labels take 19 s;
|
||||
// why_new=GST-435 pass-1 red team measured it; seam=parseFindings; no_claim=a timing bound on any other parser path
|
||||
test('a long run of joined labels parses in linear time', () => {
|
||||
for (const join of [' and ', ' or ', ', ', '/']) {
|
||||
const line = 'No ' + Array(40).fill('CRITICAL').join(join) + ' x';
|
||||
const t = performance.now();
|
||||
expect(labels(line)).toEqual(['CRITICAL']);
|
||||
expect(performance.now() - t).toBeLessThan(250);
|
||||
}
|
||||
});
|
||||
|
||||
test('a reply whose only text is a no-defect sentence, with no clean marker, is still no review', () => {
|
||||
const d = fs.mkdtempSync(path.join(os.tmpdir(), 'gst435-'));
|
||||
const r = collect(path.join(d, 'l.jsonl'), 'fix-check', 'I found nothing CRITICAL.\n', path.join(d, 'm.jsonl'));
|
||||
expect([r.review, r.lines]).toEqual([false, []]);
|
||||
fs.rmSync(d, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// Value: protects=a real blocker keeps collecting at its label: a sentence that names a file:line or a defect, an exception clause
|
||||
// (GST-796), an "other"/"else" quantifier that implies a blocker named elsewhere, a list item, heading, bold opener, field or
|
||||
// Blocks: yes, or any second labelled sentence that is not itself a denial; fails_when=any of them collects nothing or a weaker
|
||||
// label; why_new=GST-766's negation drop lost real blockers and was reverted; seam=parseFindings; no_claim=that each is a real finding
|
||||
test('negative controls: a locus, a named defect, an exception, a quantifier or a structured line keeps its label', () => {
|
||||
const kept: Array<[string, string]> = [['The token leak in log.ts:3 is CRITICAL.', 'CRITICAL'], ['I found nothing CRITICAL in auth.ts:9.', 'CRITICAL'],
|
||||
['No CRITICAL issues (a.ts:3).', 'CRITICAL'], ['I found nothing CRITICAL. The token leak in log.ts:3 is CRITICAL.', 'CRITICAL'],
|
||||
['I found nothing CRITICAL. The retry loop never ends, which is High.', 'CRITICAL'], ['No CRITICAL issues except the token leak in log.ts:3', 'CRITICAL'],
|
||||
['There is no CRITICAL issue other than the race in a.ts:3.', 'CRITICAL'], ['Nothing CRITICAL apart from the unescaped query.', 'CRITICAL'],
|
||||
['No new CRITICAL findings, but the race still blocks.', 'CRITICAL'], ['I found no defect that is CRITICAL, but the retry loop never ends.', 'CRITICAL'],
|
||||
['No auth check guards the admin route, so nothing CRITICAL.', 'CRITICAL'], ['There is no CRITICAL lock around the refund write, so two workers double-pay.', 'CRITICAL'],
|
||||
['No input validation is CRITICAL.', 'CRITICAL'], ['Nothing guards the write path the failure of which is CRITICAL', 'CRITICAL'], ['Nothing else is CRITICAL.', 'CRITICAL'],
|
||||
['No other issue is CRITICAL.', 'CRITICAL'], ['No remaining finding is CRITICAL.', 'CRITICAL'], ['None of the blockers are CRITICAL.', 'CRITICAL'], ['This is not CRITICAL.', 'CRITICAL'],
|
||||
['Otherwise nothing CRITICAL.', 'CRITICAL'], ['- No CRITICAL issues.', 'CRITICAL'], ['1. No CRITICAL issues.', 'CRITICAL'], ['**No CRITICAL issues.**', 'CRITICAL'],
|
||||
['CRITICAL: none of the writes are atomic.', 'CRITICAL'], ['No CRITICAL issues. Severity: High', 'High'], ['No CRITICAL issues. Blocks: yes', 'CRITICAL'],
|
||||
['{"severity":"CRITICAL","summary":"I found nothing CRITICAL"}', 'CRITICAL'], ['No CRITICAL — the upload handler drops auth.', 'CRITICAL']];
|
||||
for (const [line, label] of kept) expect([line, labels(line)]).toEqual([line, [label]]);
|
||||
expect(labels('1. Token leak in log.ts:3 must be fixed before landing.\n\nNothing else is CRITICAL.')).toEqual(['CRITICAL']);
|
||||
expect(labels('The token leak blocks the merge.\nNo other issue is CRITICAL.')).toEqual(['CRITICAL']);
|
||||
expect(labels('## CRITICAL\n- No CRITICAL issues in the fix.')).toEqual(['CRITICAL']);
|
||||
expect(labels('I found nothing CRITICAL. Here is what I checked:\n- **Auth:** the token check at auth.ts:9 is CRITICAL and missing.')).toEqual(['INFORMATIONAL', 'CRITICAL']);
|
||||
expect(labels('No FIXABLE findings.', 'adversarial-native')).toEqual(['CRITICAL']);
|
||||
expect(labels('Apart from the SQL injection in db.ts:40, I found\nnothing CRITICAL.')).toEqual(['CRITICAL']);
|
||||
for (const wrap of ['Apart from the SQL injection in db.ts:40 —\nnothing CRITICAL.\nRecommendation: request changes because the SQL injection exposes the database',
|
||||
'Apart from the SQL injection in db.ts:40\nI found nothing CRITICAL.', 'Apart from the SQL injection in `db.ts:40`\nnothing CRITICAL.', 'The guard was removed (see auth)\nno CRITICAL issue.',
|
||||
'Apart from the SQL injection in db.ts:40, the rest is fine.\nI found nothing CRITICAL.', '1. auth.ts:3 LOW: log line too verbose\nNo CRITICAL issues remain.'])
|
||||
expect([wrap, labels(wrap).at(-1)]).toEqual([wrap, 'CRITICAL']);
|
||||
expect(labels('The rest of the diff is fine.\nI found nothing CRITICAL.')).toEqual([]);
|
||||
expect(labels('The webhook check is gone and I found\nno CRITICAL issues.')).toEqual(['CRITICAL']);
|
||||
expect(labels('I found nothing CRITICAL and two high issues.')).toEqual(['CRITICAL']);
|
||||
expect(labels('## Summary\nI found nothing CRITICAL.')).toEqual([]);
|
||||
expect(labels('High risk in the retry path. No CRITICAL issues.')).toEqual(['CRITICAL']);
|
||||
expect(summaryCandidate('No auth check guards the admin route, so nothing CRITICAL.')).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -339,11 +339,11 @@ describe('gstack-review-dispatch sent, collect, judge and verify: nothing change
|
||||
expect([prose, reader, parseFindings(reader, prose).map(f => f.label)]).toEqual([prose, reader, ['CRITICAL']]);
|
||||
});
|
||||
|
||||
// Value: protects=a negated severity word still collects at its label, so a reader's closing prose ("found nothing CRITICAL") is a
|
||||
// known phantom lead judgment folds and never a dropped line (GSTA-167 S1 reverts GST-766's negation drop); fails_when=any negated
|
||||
// form collects nothing or a weaker label, except under the finding's own Severity field, which decides it (GST-976);
|
||||
// why_new=GST-766's drop lost blockers named in exception clauses; seam=parseFindings; no_claim=that a negated line is a real
|
||||
// finding (judge folds phantoms)
|
||||
// Value: protects=a negated severity word still collects at its label, so a reader's closing prose ("found nothing CRITICAL" after a
|
||||
// clause of its own) is a known phantom lead judgment folds and never a dropped line (GSTA-167 S1 reverts GST-766's negation drop),
|
||||
// except a sentence that is wholly a no-defect summary (GST-435), which is no finding; fails_when=any other negated form collects
|
||||
// nothing or a weaker label, except under the finding's own Severity field, which decides it (GST-976); why_new=GST-766's drop lost
|
||||
// blockers named in exception clauses; seam=parseFindings; no_claim=that a negated line is a real finding (judge folds phantoms)
|
||||
test('a negated severity word still collects at its label: GEVA-92 fix-check#5 and its negated forms are known phantoms', () => {
|
||||
const fixCheck = [
|
||||
'REVIEWED_TREE: bedd511efa0fa16c9277821073c68e438dc760d2 /workspace/gstack-evals',
|
||||
@@ -355,14 +355,16 @@ describe('gstack-review-dispatch sent, collect, judge and verify: nothing change
|
||||
'- The minimum p of 2^-15.',
|
||||
].join('\n');
|
||||
expect(parseFindings('fix-check', fixCheck).map(f => f.label)).toEqual(['INFORMATIONAL', 'INFORMATIONAL', 'CRITICAL']);
|
||||
const kept: Array<[string, string]> = [['Found no CRITICAL issues in the fix range.', 'CRITICAL'], ['Nothing CRITICAL or HIGH remains.', 'CRITICAL'], ['This is not CRITICAL.', 'CRITICAL'],
|
||||
const kept: Array<[string, string]> = [['Found no CRITICAL issues in the fix range.', 'CRITICAL'], ['This is not CRITICAL.', 'CRITICAL'],
|
||||
['I found zero Critical findings, and none High.', 'Critical'], ['The retry path is safe; without CRITICAL impact.', 'CRITICAL'], ['No new CRITICAL issues introduced by the fix.', 'CRITICAL'],
|
||||
['No remaining CRITICAL or HIGH findings.', 'CRITICAL'], ['**No** CRITICAL issues found.', 'CRITICAL'], ['No **CRITICAL** issues found.', 'CRITICAL'], ['No `CRITICAL` issue.', 'CRITICAL'],
|
||||
['No [P1] issues remain.', 'P1'], ['Nothing at CRITICAL severity remains.', 'CRITICAL'], ['No CRITICAL regressions introduced by the fix.', 'CRITICAL'], ['Nothing CRITICAL to report.', 'CRITICAL'],
|
||||
['No remaining CRITICAL or HIGH findings.', 'CRITICAL'], ['**No** CRITICAL issues found.', 'CRITICAL'],
|
||||
['No [P1] issues remain.', 'P1'], ['No CRITICAL regressions introduced by the fix.', 'CRITICAL'],
|
||||
['There is nothing CRITICAL in the fix range.', 'CRITICAL'], ['No CRITICAL — the fix is sound.', 'CRITICAL'], ['Found no CRITICAL or HIGH regressions.', 'CRITICAL'], ['No HIGH-risk changes.', 'HIGH'],
|
||||
['No CRITICAL/HIGH findings.', 'CRITICAL'], ['No P0/P1 issues.', 'P0'], ['No CRITICAL, HIGH or MEDIUM issues.', 'CRITICAL'],
|
||||
['Checked collect() at lib/review-ledger.ts:189; no CRITICAL issues remain.', 'CRITICAL'], ['No CRITICAL issues; served on localhost:3000.', 'CRITICAL']];
|
||||
for (const [prose, label] of kept) expect([prose, parseFindings('fix-check', prose).map(f => f.label)]).toEqual([prose, [label]]);
|
||||
for (const prose of ['Nothing CRITICAL or HIGH remains.', 'No **CRITICAL** issues found.', 'No `CRITICAL` issue.', 'Nothing at CRITICAL severity remains.', 'Nothing CRITICAL to report.',
|
||||
'No CRITICAL/HIGH findings.', 'No P0/P1 issues.', 'No CRITICAL, HIGH or MEDIUM issues.'])
|
||||
expect([prose, parseFindings('fix-check', prose).map(f => f.label)]).toEqual([prose, []]);
|
||||
expect(parseFindings('r', 'Not CRITICAL, but High: the cache drops a row (c.ts:4)').map(f => f.label)).toEqual(['CRITICAL']);
|
||||
expect(parseFindings('r', 'No CRITICAL, High: the cache drops a row (c.ts:4)').map(f => f.label)).toEqual(['CRITICAL']);
|
||||
expect(parseFindings('r', 'Nothing CRITICAL / HIGH: race in q.ts:7').map(f => f.label)).toEqual(['CRITICAL']);
|
||||
@@ -618,32 +620,35 @@ describe('gstack-review-dispatch sent, collect, judge and verify: nothing change
|
||||
expect(parseFindings('adversarial-native', 'No rate limit is FIXABLE (login.ts:9)').map(f => f.label)).toEqual(['CRITICAL']);
|
||||
});
|
||||
|
||||
// Value: protects=a reader's own closing summary is a known phantom that keeps its label (HARN-15's Red Team and fix-check replies,
|
||||
// GEVA-77's "Neither finding is CRITICAL.", "found nothing CRITICAL"): it collects, and judge refuses to drop or lower it; the
|
||||
// follow-up tags such lines summary_candidate so judge may dismiss a tagged one on record; fails_when=a summary line collects nothing
|
||||
// or a weaker label, which would mean a drop path is back; why_new=GST-435 removed the drop after it lost blockers in three review
|
||||
// rounds; seam=parseFindings, collect and judge; no_claim=that a phantom is free (it stays CRITICAL in the judged set)
|
||||
test('known phantoms: closing summaries keep their label until the summary_candidate follow-up', () => {
|
||||
// Value: protects=a reader's own closing summary that opens with a clause of its own is a known phantom that keeps its label (HARN-15's
|
||||
// ship2 Red Team "Red-team pass is done: …", harness-evals #152's "…, so there's nothing at CRITICAL"): it collects, and judge refuses
|
||||
// to drop or lower it unless it is a summary_candidate lowered with a reason; a sentence that is wholly a no-defect summary (GEVA-77's
|
||||
// "Neither finding is CRITICAL.", HARN-15 ship2 p2's count summaries) is no finding at the label it denies (GST-435);
|
||||
// fails_when=a phantom collects nothing or a weaker label, or a wholly no-defect sentence still collects its denied label;
|
||||
// why_new=GST-435 removed the negation drop after it lost blockers in three review rounds, then narrowed it to whole summary
|
||||
// sentences; seam=parseFindings, collect and judge; no_claim=that a phantom is free (it stays CRITICAL in the judged set)
|
||||
test('known phantoms: closing summaries with a clause of their own keep their label; wholly no-defect sentences collect none', () => {
|
||||
const fixture = (name: string) => fs.readFileSync(path.join(ROOT, 'test/fixtures/review-ledger', name), 'utf8');
|
||||
const replies: Array<[string, string, number]> = [['red-team', 'harn15-ship2-red-team.txt', 3], ['red-team', 'harn15-ship2-p2-red-team.txt', 3],
|
||||
['fix-check', 'harn15-ship2-p2-fix-check.txt', 4], ['fix-check', 'harn15-152-p2-fix-check.txt', 2]];
|
||||
for (const [reader, name, json] of replies) {
|
||||
const replies: Array<[string, string, string, number]> = [['red-team', 'harn15-ship2-red-team.txt', 'CRITICAL', 3], ['red-team', 'harn15-ship2-p2-red-team.txt', 'INFORMATIONAL', 3],
|
||||
['fix-check', 'harn15-ship2-p2-fix-check.txt', 'INFORMATIONAL', 4], ['fix-check', 'harn15-152-p2-fix-check.txt', 'CRITICAL', 2]];
|
||||
for (const [reader, name, summary, json] of replies) {
|
||||
const found = parseFindings(reader, fixture(name));
|
||||
expect([name, found.map(f => f.label)]).toEqual([name, ['CRITICAL', ...Array(json).fill('INFORMATIONAL')]]);
|
||||
expect([name, found.map(f => f.label)]).toEqual([name, [summary, ...Array(json).fill('INFORMATIONAL')]]);
|
||||
expect(found[0]!.raw).toMatch(/(no|nothing( at)?) CRITICAL/);
|
||||
}
|
||||
const fc = fixture('geva-77-fix-check.txt');
|
||||
const d = tmp();
|
||||
const merged = path.join(d, 'merged.jsonl');
|
||||
const r = collect(path.join(d, 'l.jsonl'), 'fix-check', fc, merged);
|
||||
expect([r.review, r.lines.map(l => [l.id, l.label])]).toEqual([true, [['fix-check#1', 'INFORMATIONAL'], ['fix-check#2', 'INFORMATIONAL'], ['fix-check#3', 'CRITICAL']]]);
|
||||
expect([r.review, r.lines.map(l => [l.id, l.label])]).toEqual([true, [['fix-check#1', 'INFORMATIONAL'], ['fix-check#2', 'INFORMATIONAL'], ['fix-check#3', 'INFORMATIONAL']]]);
|
||||
expect(r.lines[2]!.raw).toStartWith('Neither finding is CRITICAL.');
|
||||
const judged = path.join(d, 'judged.jsonl');
|
||||
fs.writeFileSync(judged, r.lines.map(l => JSON.stringify({ id: l.id, severity: 'INFORMATIONAL' })).join('\n') + '\n');
|
||||
expect(judge(readJsonl<MergedLine>(merged), readJsonl(judged)).ok).toBe(false);
|
||||
const phantom = collect(path.join(d, 'l2.jsonl'), 'fix-check', fixture('harn15-152-p2-fix-check.txt'), path.join(d, 'm2.jsonl'));
|
||||
fs.writeFileSync(path.join(d, 'j2.jsonl'), phantom.lines.map(l => JSON.stringify({ id: l.id, severity: 'INFORMATIONAL' })).join('\n') + '\n');
|
||||
expect(judge(readJsonl<MergedLine>(path.join(d, 'm2.jsonl')), readJsonl(path.join(d, 'j2.jsonl'))).ok).toBe(false);
|
||||
for (const line of ['There are no CRITICAL findings.', 'No CRITICAL or High findings to report.', 'I found nothing CRITICAL.', "So there's nothing at CRITICAL.",
|
||||
'Neither finding is CRITICAL.', 'Nothing here is CRITICAL.', 'Neither fix is CRITICAL.', 'Nothing in this diff is CRITICAL.', 'This is not CRITICAL.'])
|
||||
expect([line, parseFindings('outside-voice', line).map(f => f.label)]).toEqual([line, ['CRITICAL']]);
|
||||
'Neither finding is CRITICAL.', 'Nothing here is CRITICAL.', 'Neither fix is CRITICAL.', 'Nothing in this diff is CRITICAL.'])
|
||||
expect([line, parseFindings('outside-voice', line).map(f => f.label)]).toEqual([line, []]);
|
||||
expect(parseFindings('outside-voice', 'This is not CRITICAL.').map(f => f.label)).toEqual(['CRITICAL']);
|
||||
expect(parseFindings('outside-voice', 'Severity: Low (not High)').map(f => f.label)).toEqual(['Low']);
|
||||
});
|
||||
|
||||
|
||||
@@ -12,6 +12,18 @@ const LEAD = new RegExp(`(?:^|\\| )[^\\p{L}\\p{N}]*(?:\\d+[.)]\\s*[^\\p{L}\\p{N}
|
||||
const HEADING = new RegExp(`^(?:#{1,6}\\s*|\\*\\*)(?:${LABELS})\\b`, 'm');
|
||||
const LABEL = new RegExp(`${LABELS}`, 'gi');
|
||||
const LEAD_NO_FIXABLE = new RegExp(LEAD.source.replace('|FIXABLE|INVESTIGATE', ''), 'u');
|
||||
/**
|
||||
* GST-435: a reader's own no-defect closing sentence is no finding at the label it denies, so these pinned inputs collect fewer or
|
||||
* weaker records than main; each is held to its exact new labels on both classes. Every other input keeps main as its floor.
|
||||
*/
|
||||
const NO_DEFECT_SUMMARIES: Record<string, string[]> = {
|
||||
'Nothing CRITICAL or HIGH remains.': [], 'No **CRITICAL** issues found.': [], 'No `CRITICAL` issue.': [], 'Nothing at CRITICAL severity remains.': [],
|
||||
'Nothing CRITICAL to report.': [], 'No CRITICAL/HIGH findings.': [], 'No P0/P1 issues.': [], 'No CRITICAL, HIGH or MEDIUM issues.': [],
|
||||
'There are no CRITICAL findings.': [], 'No CRITICAL or High findings to report.': [],
|
||||
'I found nothing CRITICAL.': [], "So there's nothing at CRITICAL.": [], 'Neither finding is CRITICAL.': [], 'Nothing here is CRITICAL.': [], 'Neither fix is CRITICAL.': [],
|
||||
'Nothing in this diff is CRITICAL.': [], 'review-ledger/geva-77-fix-check.txt': Array(3).fill('INFORMATIONAL'),
|
||||
'review-ledger/harn15-ship2-p2-red-team.txt': Array(4).fill('INFORMATIONAL'), 'review-ledger/harn15-ship2-p2-fix-check.txt': Array(5).fill('INFORMATIONAL'),
|
||||
};
|
||||
|
||||
describe('parseFindings is never weaker than main (GST-976 invariant)', () => {
|
||||
// Value: protects=no input the suite pins, no reader fixture and no Severity field shape collects below the label main (82e9d7ec)
|
||||
@@ -32,6 +44,8 @@ describe('parseFindings is never weaker than main (GST-976 invariant)', () => {
|
||||
for (const e of entries as Array<{ reader: string; output?: string; fixture?: string; main: string[] }>) {
|
||||
const output = e.output ?? fs.readFileSync(path.join(ROOT, 'test/fixtures', e.fixture!), 'utf8');
|
||||
const got = parseFindings(e.reader, output, raises);
|
||||
const summary = NO_DEFECT_SUMMARIES[e.fixture ?? output];
|
||||
if (summary) { if (JSON.stringify(got.map(f => f.label)) !== JSON.stringify(summary)) bad.push(`GST-435 ${JSON.stringify(got.map(f => f.label))} ${JSON.stringify(output.slice(0, 80))}`); continue; }
|
||||
if (!e.main.length) continue;
|
||||
if (got.length !== e.main.length) { bad.push(`count ${got.length}/${e.main.length} ${JSON.stringify(output.slice(0, 80))}`); continue; }
|
||||
got.forEach((f, i) => {
|
||||
|
||||
Reference in new issue
Block a user