diff --git a/changelog.d/capy-gst620-sealed-identifiers.md b/changelog.d/capy-gst620-sealed-identifiers.md index d5a90c5b8..87190c978 100644 --- a/changelog.d/capy-gst620-sealed-identifiers.md +++ b/changelog.d/capy-gst620-sealed-identifiers.md @@ -2,12 +2,12 @@ ### Added - **`sealed.label_keys` and `sealed.identifier_keys` in `.gstack/pregate.json`.** With `label_keys` set, `sealed.label_join` checks every .json, .jsonl, .ndjson, .csv, .tsv and Markdown pipe table a published commit adds, modifies, renames or copies. A record that holds a label field and an identifier (a field matching an `identifier_keys` glob, by default `head`, `*_sha256`, `*_bytes`, `files` and similar names, or any 7 to 64 character hex string, as a value or as the key of an object) is a join. The guard fingerprints each join in memory and compares it with the parent's version of the file, so unchanged rows and reformatting stay clear while a replaced, added or widened row refuses even when the row count stays the same. Findings name the file, the commit and the field names (`keys=stratum,tags,head,plan_bytes`), never the values, on stderr, in `--json` and in the pre-push hook. -- **`sealed.label_join_unchecked`.** A supported file over 4 MiB, one that does not parse, one nested deeper than 64 levels, or a blob git cannot read refuses instead of passing unread. The `SEALED:` line counts these as `unchecked=`. Like every sealed finding, it can be waived through the base branch's `.gstack/sealed-allow.tsv`; `allow_paths` never exempt either rule. -- **The policy only tightens.** Both lists are the union of the base branch's and the pushed policy's, each resolved to its default first, so a pushed config can turn the rule on or widen it but cannot turn it off or narrow it. +- **`sealed.label_join_unchecked`.** A file whose text never names a label key cannot hold a join and is skipped without parsing, so a commented `tsconfig.json` passes. A supported file over 4 MiB, one that names a label key and does not parse, one nested deeper than 64 levels, or a blob git cannot read refuses instead of passing unread. The `SEALED:` line counts these as `unchecked=`. Like every sealed finding, it can be waived through the base branch's `.gstack/sealed-allow.tsv`; `allow_paths` never exempt either rule. +- **The policy only tightens.** A field can count as both a label and an identifier, so adding identifier names to `label_keys` never hides a join. Both lists are the union of the base branch's and the pushed policy's, each resolved to its default first, so a pushed config can turn the rule on or widen it but cannot turn it off or narrow it. ### Changed - **docs/pregate.md#sealed** documents the rule, its normalization (`Tree SHA-256` matches as `tree_sha_256`) and its known limits: joins across files, joins by reference, columnar JSON, labels in file names and prose. Its version-skew note now names the new keys. A gstack older than this release refuses them as unknown, so raise `min_gstack_version` before adopting them. The unknown-key error now lists `label_keys` and `identifier_keys` among the valid keys. ### Tests - `test/sealed-guard.test.ts`: the GSTA-2 panel refuses through `gstack-pregate sealed`, `--json` and the pre-push hook, and warns on a private destination, with no value in any output. It failed on main. A golden captured from main 2d1b55d76 covers seven pushes (refused, clear, exempt, waived, warned, error, no policy) without `label_keys`; stdout, stderr, `--json`, the hook's stderr and every exit code match it byte for byte. -- `test/sealed-join.test.ts`: each format, human table headers, hex values under unlisted keys, maps keyed by sha, dotted key names refused in the config, decimal strings that stay clear, replacement at the same count, delete plus add, an added identifier field, labels then identifiers a commit later, renames, merges, five unchecked cases, waivers, `allow_paths`, pushed attempts to narrow or disable the rule, and `label_keys: []` matching an absent key. +- `test/sealed-join.test.ts`: each format, human table headers, hex values under unlisted keys, maps keyed by sha, quoted TSV, a commented tsconfig.json that stays clear, dotted key names refused in the config, identifier names added to a pushed `label_keys`, decimal strings that stay clear, replacement at the same count, delete plus add, an added identifier field, labels then identifiers a commit later, renames, merges, five unchecked cases, waivers, `allow_paths`, pushed attempts to narrow or disable the rule, and `label_keys: []` matching an absent key. diff --git a/docs/pregate.md b/docs/pregate.md index 42d50606e..b8643c588 100644 --- a/docs/pregate.md +++ b/docs/pregate.md @@ -999,7 +999,7 @@ gstack-pregate sealed [--range ] [--remote origin] [--base ] [--js `plan_bytes` tells a model under test which unit it is. A record is a JSON object at any depth (its own fields plus one level of nested objects, named like `meta.head`), a JSONL or NDJSON line, a CSV or TSV row, or a row of a - Markdown pipe table. Field names and table headers are compared after + Markdown pipe table (CSV and TSV cells may be quoted). Field names and table headers are compared after trimming, lowercasing and turning spaces and hyphens into underscores (`Tree SHA-256` is `tree_sha_256`); this is the rule's own normalization, separate from `case_insensitive`. An identifier is a field whose name @@ -1010,7 +1010,9 @@ gstack-pregate sealed [--range ] [--remote origin] [--base ] [--js to 64 hex characters with at least one digit and one letter a to f. An object key of that shape counts as an identifier of the object it holds (shown as `(key)`), so a map keyed by sha joins too. Numbers count only - through the field name. Both lists take bare field names: a nested field + through the field name. A field can be both a label and an identifier, and + a record joins when it has a label field and a different identifier field, + so adding identifier names to `label_keys` never hides a join. Both lists take bare field names: a nested field matches by its own name (`stratum`, never `meta.stratum`). For each supported file a commit adds, modifies, renames or copies, the guard fingerprints every joined record (sha256 over its label and identifier fields and their values, kept in @@ -1018,7 +1020,9 @@ gstack-pregate sealed [--range ] [--remote origin] [--base ] [--js rename or copy source; every parent for a merge). A fingerprint no parent has is a finding, so unchanged records and formatting-only edits stay clear, and a replaced, added or widened record refuses even when the count stays - the same. A supported file over 4 MiB, one that does not parse (one bad + the same. A file whose text never names a label key cannot hold a join, so + it is skipped without parsing (a commented `tsconfig.json` passes). A + supported file over 4 MiB, one that names a label key and does not parse (one bad JSONL line is enough), one nested deeper than 64 levels, or a blob git cannot read is `sealed.label_join_unchecked`. It refuses too, and the `SEALED:` line counts it as `unchecked=`. Findings print field names diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index e9968ddeb..93084f5d8 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -3905,8 +3905,8 @@ it). A publishable file kept next to sealed data (a `MANIFEST.json` of hashes and counts) can be exempted from the path rules with a `sealed.allow_paths` glob in the base branch's `.gstack/pregate.json`; the text rules still read it ([docs/pregate.md#sealed](pregate.md#sealed)). For `sealed.label_join`, drop -the identifier fields (or the labels) from the published file, or move it under -a sealed path; `allow_paths` never exempt it. A reviewed false positive gets +the identifier fields (or the labels) from the published file, or keep the file +out of the public repository; `allow_paths` never exempt it. A reviewed false positive gets one line in the base branch's `.gstack/sealed-allow.tsv`: `rulepathshareason`, where `sha` is the full commit sha (a shortened one is ignored and named on stderr). A private diff --git a/lib/sealed-guard.ts b/lib/sealed-guard.ts index d8ebeb9b2..05875c249 100644 --- a/lib/sealed-guard.ts +++ b/lib/sealed-guard.ts @@ -304,13 +304,14 @@ export function renderSealed(r: SealedReport, context: string): string { } const vis = `visibility: ${r.visibility}${r.visibility === 'unknown' ? ', counted public' : ''}; ${r.visibilitySource}`; const body = r.findings.map(f => ` ${f.rule} ${where(f)} commit ${f.commit.slice(0, 12)}${f.keys ? ` keys=${f.keys.join(',')}` : ''}`); - const join = r.findings.some(f => f.rule.startsWith('sealed.label_join')) ? [`sealed.label_join: a record puts a sealed.label_keys field next to an identifier (keys= lists the label fields, then the identifier fields); drop one side from the published file or move it under a sealed path. sealed.label_join_unchecked: a supported file the guard could not read (over 4 MiB, unparseable, or nested deeper than 64); allow_paths never exempt either rule`] : []; + const join = r.findings.some(f => f.rule.startsWith('sealed.label_join')) ? [`sealed.label_join: a record puts a sealed.label_keys field next to an identifier (keys= lists the label fields, then the identifier fields); drop one side from the published file or keep the file out of the public repository. sealed.label_join_unchecked: a supported file the guard could not read (over 4 MiB, unparseable, or nested deeper than 64); allow_paths never exempt either rule`] : []; if (r.status === 'error') { lines.push(`\n⛔ ${context} BLOCKED the publication of ${r.target} — ${r.error}.`); lines.push(`Fix the sealed policy (gstack-pregate validate), then retry; an unreadable policy never passes (SEALED_OUTPUT_REFUSED)`); } else if (r.status === 'refused') { lines.push(`\n⛔ ${context} BLOCKED the publication of ${r.target} (${vis}) — sealed or eval output in ${r.commits} published commit(s):`, ...body, ...join); - lines.push(`Remove it from every published commit before this branch reaches a public repository, exempt a publishable path with a sealed.allow_paths glob in the base branch's ${SEALED_CONFIG_PATH}, or waive one reviewed finding with a \`rulepathfull-commit-shareason\` line in the base branch's ${SEALED_ALLOW_PATH}. A private destination: set ${VISIBILITY_OVERRIDE_KEY} "private" in the base branch's ${SEALED_CONFIG_PATH} (SEALED_OUTPUT_REFUSED)`); + const exemptable = r.findings.some(f => !f.rule.startsWith('sealed.label_join')) ? ` exempt a publishable path with a sealed.allow_paths glob in the base branch's ${SEALED_CONFIG_PATH},` : ''; + lines.push(`Remove it from every published commit before this branch reaches a public repository,${exemptable} or waive one reviewed finding with a \`rulepathfull-commit-shareason\` line in the base branch's ${SEALED_ALLOW_PATH}. A private destination: set ${VISIBILITY_OVERRIDE_KEY} "private" in the base branch's ${SEALED_CONFIG_PATH} (SEALED_OUTPUT_REFUSED)`); } else if (r.status === 'warned') { lines.push(`${context}: warning — sealed or eval output in a push to a private destination (${vis}); it would be refused for a public one:`, ...body, ...join); } diff --git a/lib/sealed-join.ts b/lib/sealed-join.ts index eb78d8918..24e8afeba 100644 --- a/lib/sealed-join.ts +++ b/lib/sealed-join.ts @@ -48,13 +48,10 @@ function classifier(p: JoinPolicy): (fields: Field[]) => { fp: string; labels: s const labels = new Set(p.label_keys.map(normalizeKey)); const ids = p.identifier_keys.map(g => new RegExp(`^${normalizeKey(g).replace(/[.+^${}()|[\]\\?]/g, '\\$&').replace(/\*/g, '.*')}$`)); return fields => { - const l: Field[] = []; const i: Field[] = []; - for (const f of fields) { - if (labels.has(f.key)) l.push(f); - else if ((present(f.value) && ids.some(re => re.test(f.key))) || hexValue(f.value)) i.push(f); - } - if (!l.length || !i.length) return null; - const fp = createHash('sha256').update([...l, ...i].map(f => `${f.name}=${JSON.stringify(f.value)}`).sort().join('\0')).digest('hex'); + const l = fields.filter(f => labels.has(f.key)); + const i = fields.filter(f => (present(f.value) && ids.some(re => re.test(f.key))) || hexValue(f.value)); + if (!l.some(a => i.some(b => b !== a))) return null; + const fp = createHash('sha256').update([...new Set([...l, ...i])].map(f => `${f.name}=${JSON.stringify(f.value)}`).sort().join('\0')).digest('hex'); return { fp, labels: l.map(f => shown(f.name)), ids: i.map(f => shown(f.name)) }; }; } @@ -80,7 +77,7 @@ function jsonRecords(root: unknown, out: Field[][]): boolean { return true; } -function csvRows(text: string): string[][] | null { +function delimitedRows(text: string, sep: string): string[][] | null { const rows: string[][] = []; let row: string[] = []; let cell = ''; let quoted = false; for (let i = 0; i < text.length; i++) { const c = text[i]!; @@ -89,7 +86,7 @@ function csvRows(text: string): string[][] | null { else if (text[i + 1] === '"') { cell += '"'; i++; } else quoted = false; } else if (c === '"' && cell.trim() === '') { quoted = true; cell = ''; } - else if (c === ',') { row.push(cell); cell = ''; } + else if (c === sep) { row.push(cell); cell = ''; } else if (c === '\n' || c === '\r') { if (c === '\r' && text[i + 1] === '\n') i++; row.push(cell); rows.push(row); row = []; cell = ''; } else cell += c; } @@ -131,7 +128,7 @@ export function fileRecords(file: string, text: string): Field[][] | null { } } catch { return null; } if (ext === 'md') { for (const t of markdownTables(body)) tableRecords(t, out); return out; } - const rows = ext === 'tsv' ? body.split(/\r?\n/).map(l => l.split('\t')) : csvRows(body); + const rows = delimitedRows(body, ext === 'tsv' ? '\t' : ','); if (!rows) return null; tableRecords(rows.filter(r => r.some(c => c.trim())), out); return out; @@ -139,6 +136,8 @@ export function fileRecords(file: string, text: string): Field[][] | null { /** Joined-record fingerprints of one file's text; null when it cannot be checked. */ export function fileJoins(file: string, text: string, p: JoinPolicy): Joins | null { + const folded = normalizeKey(text); + if (!text.includes('\\u') && !p.label_keys.some(k => folded.includes(normalizeKey(k)))) return new Map(); const records = fileRecords(file, text); if (!records) return null; const classify = classifier(p); @@ -180,15 +179,13 @@ function changedFiles(repoRoot: string, commits: string[]): Change[] { return out; } -/** Blob text by sha through one `cat-file --batch-check` and chunked `--batch` reads; `big` over the cap, `missing` when unreadable. */ -function readBlobs(repoRoot: string, shas: string[]): Map { - const out = new Map(); - if (!shas.length) return out; +/** Each blob's text by sha, handed to `use` as it is read (one `cat-file --batch-check`, then chunked `--batch` reads); null over the cap or when unreadable. */ +function readBlobs(repoRoot: string, shas: string[], use: (sha: string, text: string | null) => void): void { + if (!shas.length) return; const small: Array<[string, number]> = []; for (const line of gitBuffer(repoRoot, ['cat-file', '--batch-check'], shas.join('\n') + '\n', 'cat-file --batch-check').toString('utf8').split('\n').filter(Boolean)) { const [sha, type, size] = line.split(' '); - if (type !== 'blob') out.set(sha!, { unread: 'missing' }); - else if (Number(size) > JOIN_MAX_BYTES) out.set(sha!, { unread: 'big' }); + if (type !== 'blob' || Number(size) > JOIN_MAX_BYTES) use(sha!, null); else small.push([sha!, Number(size)]); } const decoder = new TextDecoder(); @@ -200,25 +197,23 @@ function readBlobs(repoRoot: string, shas: string[]): Map [c.blob, ...c.parents]))]); + const ext = (file: string) => (/\.[a-z]+$/i.exec(file)?.[0] ?? '').toLowerCase(); + const wanted = new Map>(); + for (const c of changes) for (const sha of [c.blob, ...c.parents]) wanted.set(sha, (wanted.get(sha) ?? new Set()).add(ext(c.file))); const cache = new Map(); - const joinsOf = (sha: string, file: string) => { - const key = `${sha}\0${(/\.[a-z]+$/i.exec(file)?.[0] ?? '').toLowerCase()}`; - if (!cache.has(key)) { const text = blobs.get(sha); cache.set(key, typeof text === 'string' ? fileJoins(file, text, p) : null); } - return cache.get(key)!; - }; + readBlobs(repoRoot, [...wanted.keys()], (sha, text) => { for (const e of wanted.get(sha)!) cache.set(`${sha}\0${e}`, text === null ? null : fileJoins(`f${e}`, text, p)); }); + const joinsOf = (sha: string, file: string) => cache.get(`${sha}\0${ext(file)}`) ?? null; const out: SealedFinding[] = []; for (const c of changes) { const now = joinsOf(c.blob, c.file); diff --git a/test/sealed-join.test.ts b/test/sealed-join.test.ts index 5523d01c4..f53ad620f 100644 --- a/test/sealed-join.test.ts +++ b/test/sealed-join.test.ts @@ -87,6 +87,8 @@ describe('formats and matching', () => { 'j.json': `{"units":{"${HEAD_A}":{"stratum":"decoy","tags":["t"]}}}\n`, 'k.jsonl': `{"${HEAD_B}":{"stratum":"seeded"}}\n`, 'l.json': '{"units":{"unit-one":{"stratum":"decoy"}}}\n', + 'm.tsv': '"Stratum"\t"head"\n"decoy"\t"5b7db3a"\n', + 'tsconfig.json': '{\n // a comment, so not strict JSON\n "compilerOptions": { "strict": true }\n}\n', }); expect(found(s).sort((a, b) => a.file.localeCompare(b.file))).toEqual([ { rule: 'sealed.label_join', file: 'a.jsonl', commit: sha, keys: ['stratum', 'meta.head'] }, @@ -96,6 +98,7 @@ describe('formats and matching', () => { { rule: 'sealed.label_join', file: 'e.json', commit: sha, keys: ['tags', 'id'] }, { rule: 'sealed.label_join', file: 'j.json', commit: sha, keys: ['stratum', 'tags', '(key)'] }, { rule: 'sealed.label_join', file: 'k.jsonl', commit: sha, keys: ['stratum', '(key)'] }, + { rule: 'sealed.label_join', file: 'm.tsv', commit: sha, keys: ['stratum', 'head'] }, ]); }); @@ -192,6 +195,7 @@ describe('fail closed, waivers, allow_paths and the policy union (gstack-pregate const r = preflight(s); expect(r.code).toBe(1); expect(r.out).toMatch(/findings=1 .* exempt=0 policy=/); + expect(r.err).not.toContain('sealed.allow_paths glob'); landOnMain(s, { '.gstack/sealed-allow.tsv': `sealed.label_join\twiki/board/panel.json\t${sha.slice(0, 12)}\tsynthetic fixture\n` }); const short = preflight(s); expect(short.code).toBe(1); @@ -204,7 +208,7 @@ describe('fail closed, waivers, allow_paths and the policy union (gstack-pregate test('a push can enable or widen the rule but never disable or narrow it', () => { const leak = { 'wiki/panel.json': panel([unit('u1', HEAD_A)]) }; - for (const pushed of [{ paths: ['heldout/**'] }, { ...LABELED, identifier_keys: ['zzz'] }, { label_keys: [] as string[], identifier_keys: [] as string[] }]) { + for (const pushed of [{ paths: ['heldout/**'] }, { ...LABELED, identifier_keys: ['zzz'] }, { label_keys: [] as string[], identifier_keys: [] as string[] }, { label_keys: [...LABELED.label_keys, 'head', 'plan_bytes'] }]) { const s = scratch({ '.gstack/pregate.json': policy(LABELED) }); commit(s, { '.gstack/pregate.json': policy(pushed), ...leak }); const r = preflight(s);