diff --git a/changelog.d/gst-1243-public-export-owner-words.md b/changelog.d/gst-1243-public-export-owner-words.md index 8048ce00a..270635e8d 100644 --- a/changelog.d/gst-1243-public-export-owner-words.md +++ b/changelog.d/gst-1243-public-export-owner-words.md @@ -4,10 +4,10 @@ - **Whose words.** Every owner reply the private run ever recorded counts: decision rows, `run.json`'s gate and the journal (`attempts.jsonl` keeps a reply a later answer at the same gate replaced), the words a carried gate reused, and a reason a human gave for `close --deferred --by human`. Replies that parsed as gate grammar are the decision code. An agent's own words (`answer_words`) and an agent's deferral reason stay. - **Dropped fields.** The public copy drops `owner_reply` everywhere and a carry record's `words`, and replaces a human deferral's `reason` with `[owner words redacted]`. The decision itself stays: `chosen`, the gate-grammar `reply`, `owner_mode` and `owner_reply_parsed`. - **gstack's renderings.** A human's deferral reason is replaced where gstack writes it into fixed text (voice `detail`, consensus head, gate title, a blocker's `why`, the run log), and carried words in the `CARRIED:` line. Each rendering is matched with its delimiters on both sides, so a one-word reason never rewrites a longer one that starts with it. -- **Redacted quotes.** Every exported file, sealed `*.bound.md` reviewer results included, is scanned. HTML entities are decoded, zero-width characters dropped, hyphen-newline wraps joined, Markdown link targets skipped and contractions split; a word is a run of letters and digits, and any other character separates words. A reply of 3 or more words is a quote wherever it appears whole; from 8 words, any 8 consecutive words of it are a quote too. Replies of 1 or 2 words ("yes", "do it") are not scanned for. A quote extends over neighboring words of the same reply, and the marker replaces the original characters from its first word to its last. -- **Echoes stay.** Words the owner repeated from text recorded before his reply that he didn't write (the plan at an earlier gate, that gate's items, reviewer results bound before it) stay; only the new words around them are redacted. +- **Redacted quotes.** Every exported file, sealed `*.bound.md` reviewer results included, is scanned. Before matching, the text is normalized: JSON string escapes and HTML entities decoded, format characters dropped by Unicode category (`\p{Cf}`, variation selectors, the combining grapheme joiner), HTML tag names read as spaces and hyphen-newline wraps joined. It is read both as rendered (comments, footnote markers, reference definitions and link targets and titles dropped, link text kept) and as written, so markup can neither split a quote nor hide one. Contractions split ("can't" and "cannot" read "can not", "it's" "it is", "I'd" "I would", and -n't, -'re, -'ve, -'ll, -'d). A word is a run of letters and digits, and any other character separates words. A reply of 3 or more words is a quote wherever it appears whole; from 8 words, any 8 consecutive words of it are a quote too. Replies of 1 or 2 words ("yes", "do it") are not scanned for. A quote extends over neighboring words of the same reply, and the marker replaces the original characters from its first word to its last. +- **Echoes stay.** Words the owner repeated from text recorded before his reply that he didn't write (the plan at an earlier gate, that gate's items, reviewer results bound before it) stay; only the new words around them are redacted. Every run of 3 or more words a reply adds between echoes is matched on its own (and by any 8 of its words from 8), so those words are redacted even where they are quoted without the echo. - **Refusal.** A quote or rendering inside plan.md code, which export never rewrites, stops the export with `ARTIFACT_OWNER_WORDS`, file:line and exit 3, and nothing is written. `gstack-artifact validate --public --private ` runs the same check. The error never prints the words. Each changed file gets an `OWNER_WORDS: dropped= redacted=` line. ### Tests -- `test/autoplan-public-owner-words.test.ts`: a GEVA-shaped bundle (3 `owner_reply` rows, a carried gate, one reviewer quoting a reply on one line and one reflowing a reply across blockquote lines, a control file with no owner words) exports with every owner field dropped and both quotes redacted, while the control and the round-1 reviewer files stay byte-identical. Also: a quote in plan.md code refuses; validate catches a quote or rendering added later; a human's deferral reason leaves gate.json's `why` and every other rendering, an agent's stays; a one-word human reason leaves an agent's longer one alone; a reply replaced at the same gate is still redacted; a rendering never rewrites plan.md code and an exact one there refuses; echoed plan and finding text stays; a 4-word reply is redacted. -- `test/autoplan-owner-words.test.ts`: the matching rule (1-2 word and grammar replies, whole phrases, the 8-word window, eleven kinds of glue, extension, echoes, kept code spans, bounded renderings) and a ratchet on every template that renders a deferral reason. +- `test/autoplan-public-owner-words.test.ts`: a GEVA-shaped bundle (3 `owner_reply` rows, a carried gate, one reviewer quoting a reply on one line and one reflowing a reply across blockquote lines, a control file with no owner words) exports with every owner field dropped and both quotes redacted, while the control and the round-1 reviewer files stay byte-identical. Also: a quote in plan.md code refuses; validate catches a quote or rendering added later; a human's deferral reason leaves gate.json's `why` and every other rendering, an agent's stays; a one-word human reason leaves an agent's longer one alone; a reply replaced at the same gate is still redacted; a rendering never rewrites plan.md code and an exact one there refuses; echoed plan and finding text stays; a 4-word reply is redacted; a short quote split by a tag or a reference link is redacted and validates; the new words a reply added to an echoed plan line are redacted when quoted alone, and plan.md stays byte-identical. +- `test/autoplan-owner-words.test.ts`: the matching rule (1-2 word and grammar replies, whole phrases, the 8-word window, eleven kinds of glue, fifteen kinds of markup and invisible characters (tags, comments, reference links, footnotes, link titles, literal JSON escapes, zero-width and soft-hyphen entities, direction marks and isolates, the combining grapheme joiner, a variation selector), redaction offsets over markup and escapes, contractions, new words added to an echo, extension, echoes, kept code spans, bounded renderings) and a ratchet on every template that renders a deferral reason. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 3ab7a781d..419725aea 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -4701,9 +4701,10 @@ the journal), the words a carried gate reused, or a reason a human gave for deferring a phase, in prose or in one of gstack's renderings of that reason. Nothing a person wrote leaves without their click. A reply of 3 or more words is a quote wherever it appears whole, and from 8 words any 8 consecutive words -are one, compared across punctuation, line breaks, entities, Markdown and case; -replies of 1 or 2 words are not scanned for, and words the owner echoed from -earlier text he didn't write are not his. See +are one, compared across punctuation, line breaks, JSON escapes, entities, HTML and +Markdown markup, invisible format characters, contractions and case; replies +of 1 or 2 words are not scanned for, and words the owner echoed from earlier +text he didn't write are not his (the words a reply adds to an echo still are). See [the public copy](unattended.md#the-public-copy). `gstack-artifact validate --public --private /run.json` reports each quote with file:line, and `gstack-autoplan export --public` runs that check on a staging copy and writes nothing when it fails. diff --git a/docs/unattended.md b/docs/unattended.md index 19161ca4b..e1adc5e87 100644 --- a/docs/unattended.md +++ b/docs/unattended.md @@ -316,11 +316,19 @@ owner's own words: nothing a person wrote leaves without their click (GST-1243). both sides, so a one-word reason never rewrites a longer one that starts with it. - **Redacted quotes.** Every exported file, sealed reviewer results - (`*.bound.md`) included, is scanned for quotes. HTML entities are decoded, - zero-width characters dropped, a hyphen-newline wrap joined, Markdown link - targets skipped and contractions split ("don't" reads "do not"); a word is a - run of letters and digits, and everything else (spaces, line breaks, dashes, - slashes, quote marks, `>` markers, JSON or attribute glue) separates words. + (`*.bound.md`) included, is scanned for quotes. Before matching, the text is + normalized: JSON string escapes (`\n`, `\u2019`) and HTML entities are + decoded, format characters are dropped by Unicode category (`\p{Cf}`, such as + zero-width spaces, soft hyphens and direction marks, plus variation selectors + and the combining grapheme joiner), an HTML tag name reads as a space, and a + hyphen-newline wrap is joined. The text is read twice: as rendered (HTML + comments, footnote markers, reference definitions and link targets and titles + dropped, link text kept) and as written (all of those kept as words), so a + quote is found whether markup splits it or hides it. Contractions split + ("don't" reads "do not", "can't" and "cannot" read "can not", "it's" "it is", + "I'd" "I would"). A word is a run of letters and digits, and everything else + (spaces, line breaks, dashes, slashes, quote marks, `>` markers, JSON or + attribute glue) separates words. Words compare case-insensitively. A reply of 3 or more words is a quote wherever it appears whole; from 8 words, any 8 consecutive words of it are a quote too. Replies of 1 or 2 words ("yes", "do it") are not scanned for: @@ -330,7 +338,10 @@ owner's own words: nothing a person wrote leaves without their click (GST-1243). - **Echoes stay.** Words the owner repeated from text recorded before the reply, which he didn't write (the plan at an earlier gate, that gate's items, the reviewer results bound before it), are not his words: they stay, and only the - new words around them are redacted. Prose with no quote keeps its bytes. One + new words around them are redacted. Every run of 3 or more words a reply adds + between echoes is matched on its own too (and by any 8 of its words from 8), + so the new words are redacted even where they are quoted without the echo. + Prose with no quote keeps its bytes. One `OWNER_WORDS: dropped= redacted=` line names each changed file. - **Refusal.** Export never rewrites code in plan.md, so a quote or rendering inside a code span or fence there stops the export with diff --git a/lib/autoplan-owner-words.ts b/lib/autoplan-owner-words.ts index ad72c660d..29aa0ef8b 100644 --- a/lib/autoplan-owner-words.ts +++ b/lib/autoplan-owner-words.ts @@ -20,24 +20,32 @@ * ratchets that list). Each is matched with its own delimiters on both sides, * so a one-word reason never rewrites a longer one that starts with it. * - * Quotes. Text is normalized with an offset map back to the original: HTML - * entities decoded, zero-width characters dropped, a hyphen-newline wrap - * joined, a Markdown link target dropped, and contractions split ("don't" - * reads "do not"). Words are runs of Unicode letters, marks and digits with - * inner apostrophes; everything else separates them. Words compare NFKC and - * lowercased. A reply of 1 or 2 words is not scanned for (codes and grammar; - * the reply field itself is dropped). A reply of MIN_TOKENS (3) or more is a - * quote wherever it appears whole, at word boundaries; from WINDOW (8) words, - * any WINDOW consecutive words of it are a quote too. A quote then extends - * over the neighboring words that are also words of that reply, so no owner - * word is left on either side. The redaction replaces the original - * characters from the first quoted word to the last. + * Quotes. Text is read in two views, each with an offset map back to the + * original. Both decode JSON string escapes (\n, \u2019) and HTML entities (an + * unknown name reads as a space), drop format characters by category (\p{Cf}, + * variation selectors, the combining grapheme joiner), read an HTML tag name as + * a space and join a hyphen-newline wrap. The rendered view also drops HTML + * comments, footnote markers, reference definitions and link targets and + * titles, keeping link text; the raw view keeps those as words, so owner words + * inside a comment or a title are found too. Contractions split: "don't" reads + * "do not", "can't" and "cannot" read "can not", "it's" "it is", "I'd" "I + * would", and -n't, -'re, -'ve, -'ll, -'d, -'m likewise. Words are runs of + * Unicode letters, marks and digits with inner apostrophes; everything else + * separates them. Words compare NFKC and lowercased. A reply of 1 or 2 words is + * not scanned for (codes and grammar; the reply field itself is dropped). A + * reply of MIN_TOKENS (3) or more is a quote wherever it appears whole, at word + * boundaries; from WINDOW (8) words, any WINDOW consecutive words of it are a + * quote too. A quote then extends over the neighboring words that are also + * words of that reply, so no owner word is left on either side. The redaction + * replaces the original characters from the first quoted word to the last. * * Echoes. A quoted window (or whole short reply) is not the owner's words when * the same words were already recorded before the reply, in text the owner * didn't write: the plan at each earlier gate, those gates' items, and the - * reviewer results bound before it (`prior`). Those words stay; new words - * around them are still redacted. + * reviewer results bound before it (`prior`). Those words stay. Each run of + * MIN_TOKENS or more reply words left between echoed windows is a needle of + * its own (whole, and by WINDOW from WINDOW words), so the words a reply added + * to an echo are redacted wherever they are quoted, with or without the echo. */ import * as fs from 'node:fs'; import * as path from 'node:path'; @@ -52,41 +60,72 @@ export interface OwnerSource { text: string; parsed?: boolean; prior?: string[] interface Indexed { keys: string[]; set: Set; prior: string } export interface OwnerWords { replies: Indexed[]; placements: RegExp[] } -const ENTITIES: Record = { nbsp: ' ', amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", mdash: '\u2014', ndash: '\u2013', hellip: '\u2026', lsquo: '\u2018', rsquo: '\u2019', ldquo: '\u201c', rdquo: '\u201d', shy: '', zwj: '', zwnj: '' }; -const CONTRACTIONS: Record = { "won't": ['will', 'not'], "can't": ['can', 'not'], "shan't": ['shall', 'not'] }; -const SUFFIXES: Array<[string, string]> = [["n't", 'not'], ["'re", 'are'], ["'ve", 'have'], ["'ll", 'will'], ["'m", 'am']]; +/** A normalized string with, per UTF-16 unit, the original [from, to) range it came from. */ +interface Mapped { s: string; from: number[]; to: number[] } +type Piece = string | [number, number]; +/** Rewrite each match of `re` as its pieces: a string (mapped to the whole match) or a kept [start, end) range, whose edges take the match's edges. */ +function pass(m: Mapped, re: RegExp, pieces: (x: RegExpMatchArray) => Piece[]): Mapped { + const out: Mapped = { s: '', from: [], to: [] }; + const copy = (a: number, b: number) => { out.s += m.s.slice(a, b); for (let k = a; k < b; k++) { out.from.push(m.from[k]!); out.to.push(m.to[k]!); } }; + let last = 0; + for (const x of m.s.matchAll(re)) { + const a = x.index!, b = a + x[0].length; + if (b === a) continue; + copy(last, a); + for (const piece of pieces(x)) { + if (typeof piece !== 'string') { + copy(piece[0], piece[1]); + if (piece[1] > piece[0]) { out.from[out.s.length - (piece[1] - piece[0])] = m.from[a]!; out.to[out.s.length - 1] = m.to[b - 1]!; } + continue; + } + for (const c of piece) { out.s += c; out.from.push(m.from[a]!); out.to.push(m.to[b - 1]!); } + } + last = b; + } + copy(last, m.s.length); + return out; +} -const ENTITY = /&(#\d+|#x[0-9a-f]+|[a-z]+);/iy; -const LINK_TARGET = /\]\([^\s)]*\)/y; -const WRAP = /-[ \t]*\r?\n[ \t>]*(?=[\p{L}\p{N}])/uy; -const sticky = (re: RegExp, text: string, at: number) => { re.lastIndex = at; return re.exec(text); }; +/** Common punctuation and spaces, and every HTML named entity for a format character (U+00AD, U+200B-U+200F, U+2060-U+2063). */ +const NAMED: Record = { amp: 0x26, lt: 0x3c, gt: 0x3e, quot: 0x22, apos: 0x27, nbsp: 0xa0, mdash: 0x2014, ndash: 0x2013, hellip: 0x2026, lsquo: 0x2018, rsquo: 0x2019, ldquo: 0x201c, rdquo: 0x201d, shy: 0xad, zerowidthspace: 0x200b, negativeverythinspace: 0x200b, negativethinspace: 0x200b, negativemediumspace: 0x200b, negativethickspace: 0x200b, zwnj: 0x200c, zwj: 0x200d, lrm: 0x200e, rlm: 0x200f, nobreak: 0x2060, applyfunction: 0x2061, af: 0x2061, invisibletimes: 0x2062, it: 0x2062, invisiblecomma: 0x2063, ic: 0x2063 }; +const JSON_ESCAPES: Record = { n: ' ', r: ' ', t: ' ', b: ' ', f: ' ', '"': '"', '\\': '\\', '/': '/' }; +/** What a rendered page doesn't show as words: HTML comments, reference definitions, footnote markers, link targets and titles. */ +const RENDERED: Array<[RegExp, (x: RegExpMatchArray) => Piece[]]> = [ + [//g, () => [' ']], + [/^[ \t]*\[[^\]\n]+\]:[ \t]*\S+(?:[ \t]+(?:"[^"\n]*"|'[^'\n]*'|\([^)\n]*\)))?[ \t]*$/gm, () => [' ']], + [/\[\^[^\]\s]+\]/g, () => [' ']], + [/\[([^\]\n]*)\](?:\[[^\]\n]*\]|\((?:[^()\s]|\([^()\s]*\))*(?:\s+(?:"[^"\n]*"|'[^'\n]*'|\([^)\n]*\)))?\s*\))/gd, x => [' ', x.indices![1]!, ' ']], +]; + +/** The text words are read from, in the raw or the rendered view (the header's Quotes paragraph). */ +function normalize(text: string, rendered: boolean): Mapped { + let m: Mapped = { s: text, from: Array.from({ length: text.length }, (_, i) => i), to: Array.from({ length: text.length }, (_, i) => i + 1) }; + m = pass(m, /\\(?:u([0-9a-fA-F]{4})|([nrtbf"\\/]))/g, x => [x[1] ? String.fromCharCode(parseInt(x[1], 16)) : JSON_ESCAPES[x[2]!]!]); + m = pass(m, /&(?:#(\d{1,7})|#x([0-9a-f]{1,6})|([a-z][a-z0-9]*));/gi, x => { + const code = x[1] ? parseInt(x[1], 10) : x[2] ? parseInt(x[2], 16) : NAMED[x[3]!.toLowerCase()]; + return [code !== undefined && code > 0 && code <= 0x10ffff ? String.fromCodePoint(code) : ' ']; + }); + m = pass(m, /[\p{Cf}\p{Variation_Selector}\u034f]/gu, () => []); + m = pass(m, /<\/?[A-Za-z][\w:-]*/g, () => [' ']); + if (rendered) for (const [re, pieces] of RENDERED) m = pass(m, re, pieces); + return pass(m, /(?<=[\p{L}\p{N}])-[ \t]*\r?\n[ \t>]*(?=[\p{L}\p{N}])/gu, () => []); +} + +const WHOLE: Record = { "can't": ['can', 'not'], cannot: ['can', 'not'], "won't": ['will', 'not'], "shan't": ['shall', 'not'] }; +const SUFFIXES: Array<[string, string]> = [["n't", 'not'], ["'re", 'are'], ["'ve", 'have'], ["'ll", 'will'], ["'d", 'would'], ["'m", 'am']]; +/** Pronouns whose -'s reads "is"; on any other word it is a possessive and stays. */ +const IS_BASES = new Set(['it', 'that', 'there', 'here', 'what', 'who', 'where', 'he', 'she']); interface Token { key: string; start: number; end: number } -/** Words of `text` with their original [start, end) offsets, after the normalization the header describes. */ -export function tokens(text: string): Token[] { - let norm = ''; - const from: number[] = [], to: number[] = []; - const push = (s: string, a: number, b: number) => { for (let k = 0; k < s.length; k++) { norm += s[k]; from.push(a); to.push(b); } }; - for (let i = 0; i < text.length;) { - const c = text[i]!; - const e = c === '&' ? sticky(ENTITY, text, i) : null; - const code = e?.[1]!.startsWith('#') ? (e[1]![1]?.toLowerCase() === 'x' ? parseInt(e[1]!.slice(2), 16) : parseInt(e[1]!.slice(1), 10)) : undefined; - const decoded = e ? (code !== undefined ? (code > 0 && code <= 0x10ffff ? String.fromCodePoint(code) : undefined) : ENTITIES[e[1]!.toLowerCase()]) : undefined; - if (e && decoded !== undefined) { push(decoded, i, i + e[0].length); i += e[0].length; continue; } - if (/[\u200b-\u200d\u2060\ufeff\u00ad]/.test(c)) { i++; continue; } - const l = c === ']' ? sticky(LINK_TARGET, text, i) : null; - if (l) { push(' ', i, i + l[0].length); i += l[0].length; continue; } - const w = c === '-' && /[\p{L}\p{N}]/u.test(norm.slice(-1)) ? sticky(WRAP, text, i) : null; - if (w) { i += w[0].length; continue; } - push(c, i, i + 1); - i++; - } +/** Words of `text` in the rendered (default) or raw view, with their original [start, end) offsets. */ +export function tokens(text: string, rendered = true): Token[] { + const m = normalize(text, rendered); const out: Token[] = []; - for (const m of norm.matchAll(/[\p{L}\p{M}\p{N}]+(?:['\u2019\u02bc][\p{L}\p{M}\p{N}]+)*/gu)) { - const start = from[m.index!]!, end = to[m.index! + m[0].length - 1]!; - const key = m[0].normalize('NFKC').replace(/[\u2019\u02bc]/g, "'").toLowerCase(); + for (const w of m.s.matchAll(/[\p{L}\p{M}\p{N}]+(?:['\u2019\u02bc][\p{L}\p{M}\p{N}]+)*/gu)) { + const start = m.from[w.index!]!, end = m.to[w.index! + w[0].length - 1]!; + const key = w[0].normalize('NFKC').replace(/[\u2019\u02bc]/g, "'").toLowerCase(); const suffix = SUFFIXES.find(([s]) => key.endsWith(s) && key.length > s.length); - const parts = CONTRACTIONS[key] ?? (suffix ? [key.slice(0, -suffix[0].length), suffix[1]] : [key]); + const parts = WHOLE[key] ?? (key.endsWith("'s") && IS_BASES.has(key.slice(0, -2)) ? [key.slice(0, -2), 'is'] : suffix ? [key.slice(0, -suffix[0].length), suffix[1]] : [key]); for (const k of parts) out.push({ key: k, start, end }); } return out; @@ -96,16 +135,34 @@ const escape = (s: string) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); /** Where gstack renders a human's deferral reason (R): the voice detail, the consensus head, the gate title, a blocker's why, the run log. */ const DEFERRAL_RENDERINGS = (r: string) => [`^deferred: (${r})$`, `\\((${r}); by human\\)`, `accepted-with-findings at round \\d+ \\((${r})\\): `, `closed deferred \\((${r})\\) with this finding open`, `, by human\\): (${r}); \\d+ open finding`]; -/** The scan index for owner replies (1-2 word and grammar replies skipped), deferral reasons and carried words. */ +/** + * The scan index: each owner reply and deferral reason of MIN_TOKENS+ words (grammar skipped), each run of MIN_TOKENS+ + * of its words between echoed windows, and the placements of deferral reasons and carried words. + */ export function ownerIndex(replies: OwnerSource[], o: { deferred?: OwnerSource[]; carried?: string[] } = {}): OwnerWords { - const keysOf = (texts: string[]) => ` ${texts.map(t => tokens(t).map(k => k.key).join(' ')).join(' \u0001 ')} `; + const keysOf = (texts: string[]) => ` ${texts.flatMap(t => [tokens(t, false), tokens(t)].map(ts => ts.map(k => k.key).join(' '))).join(' \u0001 ')} `; const seen = new Set(); - const indexed = [...replies, ...(o.deferred ?? [])].flatMap(s => { - const keys = tokens(s.text).map(t => t.key); - if (s.parsed || keys.length < MIN_TOKENS || seen.has(keys.join(' '))) return []; + const indexed: Indexed[] = []; + const add = (keys: string[], prior: string) => { + if (keys.length < MIN_TOKENS || seen.has(keys.join(' '))) return; seen.add(keys.join(' ')); - return [{ keys, set: new Set(keys), prior: keysOf(s.prior ?? []) }]; - }); + indexed.push({ keys, set: new Set(keys), prior }); + }; + for (const s of [...replies, ...(o.deferred ?? [])]) { + if (s.parsed) continue; + const keys = tokens(s.text).map(t => t.key); + const prior = keysOf(s.prior ?? []); + add(keys, prior); + const size = Math.min(keys.length, WINDOW); + const echo = new Set(); + for (let j = 0; j + size <= keys.length; j++) if (prior.includes(` ${keys.slice(j, j + size).join(' ')} `)) for (let k = j; k < j + size; k++) echo.add(k); + for (let j = 0; echo.size && j < keys.length;) { + let k = j; + while (k < keys.length && !echo.has(k)) k++; + if (k > j) add(keys.slice(j, k), prior); + j = k + 1; + } + } const placements = [ ...(o.deferred ?? []).flatMap(d => DEFERRAL_RENDERINGS(escape(d.text))), ...(o.carried ?? []).map(w => `words=(${escape(JSON.stringify(w))})(?= by=|$)`), @@ -113,12 +170,11 @@ export function ownerIndex(replies: OwnerSource[], o: { deferred?: OwnerSource[] return { replies: indexed, placements }; } -/** Every owner-words span in `text` (placements, then quotes less echoes), merged, as original [start, end) offsets. */ +/** Every owner-words span in `text` (placements, then quotes less echoes, in both views), merged, as original [start, end) offsets. */ export function ownerSpans(text: string, owner: OwnerWords): Span[] { const spans: Span[] = []; for (const re of owner.placements) for (const m of text.matchAll(re)) spans.push([m.indices![1]![0], m.indices![1]![1]]); - if (owner.replies.length) { - const ts = tokens(text); + for (const ts of owner.replies.length ? [tokens(text, false), tokens(text)] : []) { const keys = ts.map(t => t.key); for (const r of owner.replies) { const size = Math.min(r.keys.length, WINDOW); diff --git a/test/autoplan-owner-words.test.ts b/test/autoplan-owner-words.test.ts index 01320b845..c27c681ca 100644 --- a/test/autoplan-owner-words.test.ts +++ b/test/autoplan-owner-words.test.ts @@ -51,6 +51,52 @@ describe('the quote rule', () => { for (const word of ['take', 'flag', 'want', 'bang', 'roll']) expect(r.text).not.toContain(word); }); + test.each([ + ['an HTML tag', 'Jane Doe
vetoed it'], + ['bold tags', 'Jane Doe vetoed it'], + ['an HTML comment', 'Jane Doe vetoed it'], + ['a reference link', '[Jane Doe][1] vetoed it\n\n[1]: https://x.test "the source"'], + ['a footnote marker', 'Jane Doe[^1] vetoed it'], + ['a link title', '[Jane Doe](https://x.test "the source") vetoed it'], + ['a literal JSON newline escape', '{"q": "Jane\\nDoe vetoed it"}'], + ['a zero-width space entity', 'Jane Do​e vetoed it'], + ['a soft hyphen entity', 'Jane Do­e vetoed it'], + ['a named zero-width entity', 'Jane Do​e vetoed it'], + ['a left-to-right mark', 'Jane Do\u200ee vetoed it'], + ['a right-to-left mark', 'Jane Do\u200fe vetoed it'], + ['a directional isolate', 'Jane \u2066Doe\u2069 vetoed it'], + ['a combining grapheme joiner', 'Jane Do\u034fe vetoed it'], + ['a variation selector', 'Jane Doe\ufe0f vetoed it'], + ])('%s does not hide a short quote', (_, text) => { + const r = redactOwnerWords(text, ownerIndex([{ text: 'Jane Doe vetoed it' }])); + expect(r.redacted).toBeGreaterThanOrEqual(1); + for (const word of ['Jane', 'vetoed']) expect(r.text).not.toContain(word); + }); + + test('the redaction covers the original characters, markup inside the quote included', () => { + expect(redactOwnerWords('X: Jane Doe vetoed it.', ownerIndex([{ text: 'Jane Doe vetoed it' }])).text).toBe(`X: ${MARK}.`); + expect(redactOwnerWords('{"q": "I don\\u2019t want it now"}', ownerIndex([{ text: "I don't want it now" }])).text).toBe(`{"q": "${MARK}"}`); + }); + + test.each([ + ['I cannot accept it', "I can't accept it"], + ['I cannot accept it', 'I can not accept it'], + ['It is too risky now', "It's too risky now"], + ['I would ship it today', "I'd ship it today"], + ['We will not ship it', "We won't ship it"], + ['They are not ready yet', "They aren't ready yet"], + ['They are not ready yet', "They're not ready yet"], + ])('the reply %p matches %p', (reply, text) => { + expect(redactOwnerWords(text, ownerIndex([{ text: reply }])).redacted).toBe(1); + }); + + test('new words a reply adds to echoed text are a quote on their own, whatever the whole reply\'s length', () => { + const line = 'Persist the attempt count on the job row so a restarted worker resumes the schedule.'; + const o = ownerIndex([{ text: `${line.slice(0, -1)}, and page Jane Doe on call.`, prior: [line] }]); + expect(redactOwnerWords('Note: and page Jane Doe on call.', o).text).toBe(`Note: ${MARK}.`); + expect(redactOwnerWords(line, o).redacted).toBe(0); + }); + test('a quote extends over the neighboring words of the same reply, so none is left on either side', () => { const reply = 'Honestly I do not want a big bang rollout on the retry path'; expect(quoted('Well honestly, I don\u2019t want a big bang rollout on the retry path!', reply)).toEqual(['honestly, I don\u2019t want a big bang rollout on the retry path']); diff --git a/test/autoplan-public-owner-words.test.ts b/test/autoplan-public-owner-words.test.ts index e8d6d68f8..0bc558d97 100644 --- a/test/autoplan-public-owner-words.test.ts +++ b/test/autoplan-public-owner-words.test.ts @@ -277,4 +277,23 @@ describe("export --public drops the owner's own words (GST-1243)", () => { expect(fs.readFileSync(path.join(pub, 'ceo-native.a5.bound.md'), 'utf8')).toContain(`Context: ${MARK}, per the relay.`); expect(allText(pub)).not.toContain('Jane Doe'); }); + + test('markup and reference links around a short quote do not hide it from export or validate', () => { + const { out, dir } = ownerBundle({ wa: 'Jane Doe vetoed it', ceo: { native: 'Context: Jane Doe
vetoed it, and [Jane Doe][1] vetoed it.\n\n[1]: https://x.test' }, eng: {} }); + const pub = path.join(dir, 'public'); + expect(r.ap(['export', '--out', out, '--public', pub]).status).toBe(0); + const bound = fs.readFileSync(path.join(pub, 'ceo-native.a5.bound.md'), 'utf8'); + expect(bound).not.toContain('Jane'); + expect(bound).toContain(`Context: ${MARK}, and ${MARK}.`); + expect(artifact(['validate', '--public', pub, '--private', path.join(out, 'run.json')]).status).toBe(0); + }); + + test('new words a reply adds to an echoed plan line are redacted when quoted alone', () => { + const PLAN_LINE = 'Persist the attempt count on the job row so a restarted worker resumes the schedule.'; + const { out, dir } = ownerBundle({ wb: `${PLAN_LINE.slice(0, -1)}, and page Jane Doe on call.`, ceo: { native: 'Ops note: and page Jane Doe on call.' }, eng: {} }); + const pub = path.join(dir, 'public'); + expect(r.ap(['export', '--out', out, '--public', pub]).status).toBe(0); + expect(fs.readFileSync(path.join(pub, 'ceo-native.a5.bound.md'), 'utf8')).toContain(`Ops note: ${MARK}.`); + expect(fs.readFileSync(path.join(pub, 'plan.md'))).toEqual(fs.readFileSync(path.join(out, 'plan.md'))); + }); });