fix(autoplan): normalize markup, entities, escapes and format characters before matching owner words; each new run in an echoing reply is its own needle (GST-1243 review)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

changelog.d/gst-1243-public-export-owner-words.md

@@ -4,10 +4,10 @@
 - **Whose words.** Every owner reply the private run ever recorded counts: decision rows, `run.json`'s gate and the journal (`attempts.jsonl` keeps a reply a later answer at the same gate replaced), the words a carried gate reused, and a reason a human gave for `close --deferred --by human`. Replies that parsed as gate grammar are the decision code. An agent's own words (`answer_words`) and an agent's deferral reason stay.
 - **Dropped fields.** The public copy drops `owner_reply` everywhere and a carry record's `words`, and replaces a human deferral's `reason` with `[owner words redacted]`. The decision itself stays: `chosen`, the gate-grammar `reply`, `owner_mode` and `owner_reply_parsed`.
 - **gstack's renderings.** A human's deferral reason is replaced where gstack writes it into fixed text (voice `detail`, consensus head, gate title, a blocker's `why`, the run log), and carried words in the `CARRIED:` line. Each rendering is matched with its delimiters on both sides, so a one-word reason never rewrites a longer one that starts with it.
-- **Redacted quotes.** Every exported file, sealed `*.bound.md` reviewer results included, is scanned. HTML entities are decoded, zero-width characters dropped, hyphen-newline wraps joined, Markdown link targets skipped and contractions split; a word is a run of letters and digits, and any other character separates words. A reply of 3 or more words is a quote wherever it appears whole; from 8 words, any 8 consecutive words of it are a quote too. Replies of 1 or 2 words ("yes", "do it") are not scanned for. A quote extends over neighboring words of the same reply, and the marker replaces the original characters from its first word to its last.
+- **Redacted quotes.** Every exported file, sealed `*.bound.md` reviewer results included, is scanned. Before matching, the text is normalized: JSON string escapes and HTML entities decoded, format characters dropped by Unicode category (`\p{Cf}`, variation selectors, the combining grapheme joiner), HTML tag names read as spaces and hyphen-newline wraps joined. It is read both as rendered (comments, footnote markers, reference definitions and link targets and titles dropped, link text kept) and as written, so markup can neither split a quote nor hide one. Contractions split ("can't" and "cannot" read "can not", "it's" "it is", "I'd" "I would", and -n't, -'re, -'ve, -'ll, -'d). A word is a run of letters and digits, and any other character separates words. A reply of 3 or more words is a quote wherever it appears whole; from 8 words, any 8 consecutive words of it are a quote too. Replies of 1 or 2 words ("yes", "do it") are not scanned for. A quote extends over neighboring words of the same reply, and the marker replaces the original characters from its first word to its last.
-- **Echoes stay.** Words the owner repeated from text recorded before his reply that he didn't write (the plan at an earlier gate, that gate's items, reviewer results bound before it) stay; only the new words around them are redacted.
+- **Echoes stay.** Words the owner repeated from text recorded before his reply that he didn't write (the plan at an earlier gate, that gate's items, reviewer results bound before it) stay; only the new words around them are redacted. Every run of 3 or more words a reply adds between echoes is matched on its own (and by any 8 of its words from 8), so those words are redacted even where they are quoted without the echo.
 - **Refusal.** A quote or rendering inside plan.md code, which export never rewrites, stops the export with `ARTIFACT_OWNER_WORDS`, file:line and exit 3, and nothing is written. `gstack-artifact validate --public <dir> --private <run.json>` runs the same check. The error never prints the words. Each changed file gets an `OWNER_WORDS: <file> dropped=<n> redacted=<m>` line.
 
 ### Tests
-- `test/autoplan-public-owner-words.test.ts`: a GEVA-shaped bundle (3 `owner_reply` rows, a carried gate, one reviewer quoting a reply on one line and one reflowing a reply across blockquote lines, a control file with no owner words) exports with every owner field dropped and both quotes redacted, while the control and the round-1 reviewer files stay byte-identical. Also: a quote in plan.md code refuses; validate catches a quote or rendering added later; a human's deferral reason leaves gate.json's `why` and every other rendering, an agent's stays; a one-word human reason leaves an agent's longer one alone; a reply replaced at the same gate is still redacted; a rendering never rewrites plan.md code and an exact one there refuses; echoed plan and finding text stays; a 4-word reply is redacted.
+- `test/autoplan-public-owner-words.test.ts`: a GEVA-shaped bundle (3 `owner_reply` rows, a carried gate, one reviewer quoting a reply on one line and one reflowing a reply across blockquote lines, a control file with no owner words) exports with every owner field dropped and both quotes redacted, while the control and the round-1 reviewer files stay byte-identical. Also: a quote in plan.md code refuses; validate catches a quote or rendering added later; a human's deferral reason leaves gate.json's `why` and every other rendering, an agent's stays; a one-word human reason leaves an agent's longer one alone; a reply replaced at the same gate is still redacted; a rendering never rewrites plan.md code and an exact one there refuses; echoed plan and finding text stays; a 4-word reply is redacted; a short quote split by a tag or a reference link is redacted and validates; the new words a reply added to an echoed plan line are redacted when quoted alone, and plan.md stays byte-identical.
-- `test/autoplan-owner-words.test.ts`: the matching rule (1-2 word and grammar replies, whole phrases, the 8-word window, eleven kinds of glue, extension, echoes, kept code spans, bounded renderings) and a ratchet on every template that renders a deferral reason.
+- `test/autoplan-owner-words.test.ts`: the matching rule (1-2 word and grammar replies, whole phrases, the 8-word window, eleven kinds of glue, fifteen kinds of markup and invisible characters (tags, comments, reference links, footnotes, link titles, literal JSON escapes, zero-width and soft-hyphen entities, direction marks and isolates, the combining grapheme joiner, a variation selector), redaction offsets over markup and escapes, contractions, new words added to an echo, extension, echoes, kept code spans, bounded renderings) and a ratchet on every template that renders a deferral reason.

docs/troubleshooting.md

@@ -4701,9 +4701,10 @@ the journal), the words a carried gate reused, or a reason a human gave for
 deferring a phase, in prose or in one of gstack's renderings of that reason.
 Nothing a person wrote leaves without their click. A reply of 3 or more words
 is a quote wherever it appears whole, and from 8 words any 8 consecutive words
-are one, compared across punctuation, line breaks, entities, Markdown and case;
+are one, compared across punctuation, line breaks, JSON escapes, entities, HTML and
-replies of 1 or 2 words are not scanned for, and words the owner echoed from
+Markdown markup, invisible format characters, contractions and case; replies
-earlier text he didn't write are not his. See
+of 1 or 2 words are not scanned for, and words the owner echoed from earlier
+text he didn't write are not his (the words a reply adds to an echo still are). See
 [the public copy](unattended.md#the-public-copy). `gstack-artifact validate --public <dir> --private
 <run>/run.json` reports each quote with file:line, and `gstack-autoplan export
 --public` runs that check on a staging copy and writes nothing when it fails.

docs/unattended.md

@@ -316,11 +316,19 @@ owner's own words: nothing a person wrote leaves without their click (GST-1243).
   both sides, so a one-word reason never rewrites a longer one that starts
   with it.
 - **Redacted quotes.** Every exported file, sealed reviewer results
-  (`*.bound.md`) included, is scanned for quotes. HTML entities are decoded,
+  (`*.bound.md`) included, is scanned for quotes. Before matching, the text is
-  zero-width characters dropped, a hyphen-newline wrap joined, Markdown link
+  normalized: JSON string escapes (`\n`, `\u2019`) and HTML entities are
-  targets skipped and contractions split ("don't" reads "do not"); a word is a
+  decoded, format characters are dropped by Unicode category (`\p{Cf}`, such as
-  run of letters and digits, and everything else (spaces, line breaks, dashes,
+  zero-width spaces, soft hyphens and direction marks, plus variation selectors
-  slashes, quote marks, `>` markers, JSON or attribute glue) separates words.
+  and the combining grapheme joiner), an HTML tag name reads as a space, and a
+  hyphen-newline wrap is joined. The text is read twice: as rendered (HTML
+  comments, footnote markers, reference definitions and link targets and titles
+  dropped, link text kept) and as written (all of those kept as words), so a
+  quote is found whether markup splits it or hides it. Contractions split
+  ("don't" reads "do not", "can't" and "cannot" read "can not", "it's" "it is",
+  "I'd" "I would"). A word is a run of letters and digits, and everything else
+  (spaces, line breaks, dashes, slashes, quote marks, `>` markers, JSON or
+  attribute glue) separates words.
   Words compare case-insensitively. A reply of 3 or more words is a quote
   wherever it appears whole; from 8 words, any 8 consecutive words of it are a
   quote too. Replies of 1 or 2 words ("yes", "do it") are not scanned for:
@@ -330,7 +338,10 @@ owner's own words: nothing a person wrote leaves without their click (GST-1243).
 - **Echoes stay.** Words the owner repeated from text recorded before the reply,
   which he didn't write (the plan at an earlier gate, that gate's items, the
   reviewer results bound before it), are not his words: they stay, and only the
-  new words around them are redacted. Prose with no quote keeps its bytes. One
+  new words around them are redacted. Every run of 3 or more words a reply adds
+  between echoes is matched on its own too (and by any 8 of its words from 8),
+  so the new words are redacted even where they are quoted without the echo.
+  Prose with no quote keeps its bytes. One
   `OWNER_WORDS: <file> dropped=<n> redacted=<m>` line names each changed file.
 - **Refusal.** Export never rewrites code in plan.md, so a quote or rendering
   inside a code span or fence there stops the export with

lib/autoplan-owner-words.ts

@@ -20,24 +20,32 @@
  * ratchets that list). Each is matched with its own delimiters on both sides,
  * so a one-word reason never rewrites a longer one that starts with it.
  *
- * Quotes. Text is normalized with an offset map back to the original: HTML
+ * Quotes. Text is read in two views, each with an offset map back to the
- * entities decoded, zero-width characters dropped, a hyphen-newline wrap
+ * original. Both decode JSON string escapes (\n, \u2019) and HTML entities (an
- * joined, a Markdown link target dropped, and contractions split ("don't"
+ * unknown name reads as a space), drop format characters by category (\p{Cf},
- * reads "do not"). Words are runs of Unicode letters, marks and digits with
+ * variation selectors, the combining grapheme joiner), read an HTML tag name as
- * inner apostrophes; everything else separates them. Words compare NFKC and
+ * a space and join a hyphen-newline wrap. The rendered view also drops HTML
- * lowercased. A reply of 1 or 2 words is not scanned for (codes and grammar;
+ * comments, footnote markers, reference definitions and link targets and
- * the reply field itself is dropped). A reply of MIN_TOKENS (3) or more is a
+ * titles, keeping link text; the raw view keeps those as words, so owner words
- * quote wherever it appears whole, at word boundaries; from WINDOW (8) words,
+ * inside a comment or a title are found too. Contractions split: "don't" reads
- * any WINDOW consecutive words of it are a quote too. A quote then extends
+ * "do not", "can't" and "cannot" read "can not", "it's" "it is", "I'd" "I
- * over the neighboring words that are also words of that reply, so no owner
+ * would", and -n't, -'re, -'ve, -'ll, -'d, -'m likewise. Words are runs of
- * word is left on either side. The redaction replaces the original
+ * Unicode letters, marks and digits with inner apostrophes; everything else
- * characters from the first quoted word to the last.
+ * separates them. Words compare NFKC and lowercased. A reply of 1 or 2 words is
+ * not scanned for (codes and grammar; the reply field itself is dropped). A
+ * reply of MIN_TOKENS (3) or more is a quote wherever it appears whole, at word
+ * boundaries; from WINDOW (8) words, any WINDOW consecutive words of it are a
+ * quote too. A quote then extends over the neighboring words that are also
+ * words of that reply, so no owner word is left on either side. The redaction
+ * replaces the original characters from the first quoted word to the last.
  *
  * Echoes. A quoted window (or whole short reply) is not the owner's words when
  * the same words were already recorded before the reply, in text the owner
  * didn't write: the plan at each earlier gate, those gates' items, and the
- * reviewer results bound before it (`prior`). Those words stay; new words
+ * reviewer results bound before it (`prior`). Those words stay. Each run of
- * around them are still redacted.
+ * MIN_TOKENS or more reply words left between echoed windows is a needle of
+ * its own (whole, and by WINDOW from WINDOW words), so the words a reply added
+ * to an echo are redacted wherever they are quoted, with or without the echo.
  */
 import * as fs from 'node:fs';
 import * as path from 'node:path';
@@ -52,41 +60,72 @@ export interface OwnerSource { text: string; parsed?: boolean; prior?: string[]
 interface Indexed { keys: string[]; set: Set<string>; prior: string }
 export interface OwnerWords { replies: Indexed[]; placements: RegExp[] }
 
-const ENTITIES: Record<string, string> = { nbsp: ' ', amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", mdash: '\u2014', ndash: '\u2013', hellip: '\u2026', lsquo: '\u2018', rsquo: '\u2019', ldquo: '\u201c', rdquo: '\u201d', shy: '', zwj: '', zwnj: '' };
+/** A normalized string with, per UTF-16 unit, the original [from, to) range it came from. */
-const CONTRACTIONS: Record<string, string[]> = { "won't": ['will', 'not'], "can't": ['can', 'not'], "shan't": ['shall', 'not'] };
+interface Mapped { s: string; from: number[]; to: number[] }
-const SUFFIXES: Array<[string, string]> = [["n't", 'not'], ["'re", 'are'], ["'ve", 'have'], ["'ll", 'will'], ["'m", 'am']];
+type Piece = string | [number, number];
+/** Rewrite each match of `re` as its pieces: a string (mapped to the whole match) or a kept [start, end) range, whose edges take the match's edges. */
+function pass(m: Mapped, re: RegExp, pieces: (x: RegExpMatchArray) => Piece[]): Mapped {
+  const out: Mapped = { s: '', from: [], to: [] };
+  const copy = (a: number, b: number) => { out.s += m.s.slice(a, b); for (let k = a; k < b; k++) { out.from.push(m.from[k]!); out.to.push(m.to[k]!); } };
+  let last = 0;
+  for (const x of m.s.matchAll(re)) {
+    const a = x.index!, b = a + x[0].length;
+    if (b === a) continue;
+    copy(last, a);
+    for (const piece of pieces(x)) {
+      if (typeof piece !== 'string') {
+        copy(piece[0], piece[1]);
+        if (piece[1] > piece[0]) { out.from[out.s.length - (piece[1] - piece[0])] = m.from[a]!; out.to[out.s.length - 1] = m.to[b - 1]!; }
+        continue;
+      }
+      for (const c of piece) { out.s += c; out.from.push(m.from[a]!); out.to.push(m.to[b - 1]!); }
+    }
+    last = b;
+  }
+  copy(last, m.s.length);
+  return out;
+}
 
-const ENTITY = /&(#\d+|#x[0-9a-f]+|[a-z]+);/iy;
+/** Common punctuation and spaces, and every HTML named entity for a format character (U+00AD, U+200B-U+200F, U+2060-U+2063). */
-const LINK_TARGET = /\]\([^\s)]*\)/y;
+const NAMED: Record<string, number> = { amp: 0x26, lt: 0x3c, gt: 0x3e, quot: 0x22, apos: 0x27, nbsp: 0xa0, mdash: 0x2014, ndash: 0x2013, hellip: 0x2026, lsquo: 0x2018, rsquo: 0x2019, ldquo: 0x201c, rdquo: 0x201d, shy: 0xad, zerowidthspace: 0x200b, negativeverythinspace: 0x200b, negativethinspace: 0x200b, negativemediumspace: 0x200b, negativethickspace: 0x200b, zwnj: 0x200c, zwj: 0x200d, lrm: 0x200e, rlm: 0x200f, nobreak: 0x2060, applyfunction: 0x2061, af: 0x2061, invisibletimes: 0x2062, it: 0x2062, invisiblecomma: 0x2063, ic: 0x2063 };
-const WRAP = /-[ \t]*\r?\n[ \t>]*(?=[\p{L}\p{N}])/uy;
+const JSON_ESCAPES: Record<string, string> = { n: ' ', r: ' ', t: ' ', b: ' ', f: ' ', '"': '"', '\\': '\\', '/': '/' };
-const sticky = (re: RegExp, text: string, at: number) => { re.lastIndex = at; return re.exec(text); };
+/** What a rendered page doesn't show as words: HTML comments, reference definitions, footnote markers, link targets and titles. */
+const RENDERED: Array<[RegExp, (x: RegExpMatchArray) => Piece[]]> = [
+  [/<!--[\s\S]*?-->/g, () => [' ']],
+  [/^[ \t]*\[[^\]\n]+\]:[ \t]*\S+(?:[ \t]+(?:"[^"\n]*"|'[^'\n]*'|\([^)\n]*\)))?[ \t]*$/gm, () => [' ']],
+  [/\[\^[^\]\s]+\]/g, () => [' ']],
+  [/\[([^\]\n]*)\](?:\[[^\]\n]*\]|\((?:[^()\s]|\([^()\s]*\))*(?:\s+(?:"[^"\n]*"|'[^'\n]*'|\([^)\n]*\)))?\s*\))/gd, x => [' ', x.indices![1]!, ' ']],
+];
+
+/** The text words are read from, in the raw or the rendered view (the header's Quotes paragraph). */
+function normalize(text: string, rendered: boolean): Mapped {
+  let m: Mapped = { s: text, from: Array.from({ length: text.length }, (_, i) => i), to: Array.from({ length: text.length }, (_, i) => i + 1) };
+  m = pass(m, /\\(?:u([0-9a-fA-F]{4})|([nrtbf"\\/]))/g, x => [x[1] ? String.fromCharCode(parseInt(x[1], 16)) : JSON_ESCAPES[x[2]!]!]);
+  m = pass(m, /&(?:#(\d{1,7})|#x([0-9a-f]{1,6})|([a-z][a-z0-9]*));/gi, x => {
+    const code = x[1] ? parseInt(x[1], 10) : x[2] ? parseInt(x[2], 16) : NAMED[x[3]!.toLowerCase()];
+    return [code !== undefined && code > 0 && code <= 0x10ffff ? String.fromCodePoint(code) : ' '];
+  });
+  m = pass(m, /[\p{Cf}\p{Variation_Selector}\u034f]/gu, () => []);
+  m = pass(m, /<\/?[A-Za-z][\w:-]*/g, () => [' ']);
+  if (rendered) for (const [re, pieces] of RENDERED) m = pass(m, re, pieces);
+  return pass(m, /(?<=[\p{L}\p{N}])-[ \t]*\r?\n[ \t>]*(?=[\p{L}\p{N}])/gu, () => []);
+}
+
+const WHOLE: Record<string, string[]> = { "can't": ['can', 'not'], cannot: ['can', 'not'], "won't": ['will', 'not'], "shan't": ['shall', 'not'] };
+const SUFFIXES: Array<[string, string]> = [["n't", 'not'], ["'re", 'are'], ["'ve", 'have'], ["'ll", 'will'], ["'d", 'would'], ["'m", 'am']];
+/** Pronouns whose -'s reads "is"; on any other word it is a possessive and stays. */
+const IS_BASES = new Set(['it', 'that', 'there', 'here', 'what', 'who', 'where', 'he', 'she']);
 
 interface Token { key: string; start: number; end: number }
-/** Words of `text` with their original [start, end) offsets, after the normalization the header describes. */
+/** Words of `text` in the rendered (default) or raw view, with their original [start, end) offsets. */
-export function tokens(text: string): Token[] {
+export function tokens(text: string, rendered = true): Token[] {
-  let norm = '';
+  const m = normalize(text, rendered);
-  const from: number[] = [], to: number[] = [];
-  const push = (s: string, a: number, b: number) => { for (let k = 0; k < s.length; k++) { norm += s[k]; from.push(a); to.push(b); } };
-  for (let i = 0; i < text.length;) {
-    const c = text[i]!;
-    const e = c === '&' ? sticky(ENTITY, text, i) : null;
-    const code = e?.[1]!.startsWith('#') ? (e[1]![1]?.toLowerCase() === 'x' ? parseInt(e[1]!.slice(2), 16) : parseInt(e[1]!.slice(1), 10)) : undefined;
-    const decoded = e ? (code !== undefined ? (code > 0 && code <= 0x10ffff ? String.fromCodePoint(code) : undefined) : ENTITIES[e[1]!.toLowerCase()]) : undefined;
-    if (e && decoded !== undefined) { push(decoded, i, i + e[0].length); i += e[0].length; continue; }
-    if (/[\u200b-\u200d\u2060\ufeff\u00ad]/.test(c)) { i++; continue; }
-    const l = c === ']' ? sticky(LINK_TARGET, text, i) : null;
-    if (l) { push(' ', i, i + l[0].length); i += l[0].length; continue; }
-    const w = c === '-' && /[\p{L}\p{N}]/u.test(norm.slice(-1)) ? sticky(WRAP, text, i) : null;
-    if (w) { i += w[0].length; continue; }
-    push(c, i, i + 1);
-    i++;
-  }
   const out: Token[] = [];
-  for (const m of norm.matchAll(/[\p{L}\p{M}\p{N}]+(?:['\u2019\u02bc][\p{L}\p{M}\p{N}]+)*/gu)) {
+  for (const w of m.s.matchAll(/[\p{L}\p{M}\p{N}]+(?:['\u2019\u02bc][\p{L}\p{M}\p{N}]+)*/gu)) {
-    const start = from[m.index!]!, end = to[m.index! + m[0].length - 1]!;
+    const start = m.from[w.index!]!, end = m.to[w.index! + w[0].length - 1]!;
-    const key = m[0].normalize('NFKC').replace(/[\u2019\u02bc]/g, "'").toLowerCase();
+    const key = w[0].normalize('NFKC').replace(/[\u2019\u02bc]/g, "'").toLowerCase();
     const suffix = SUFFIXES.find(([s]) => key.endsWith(s) && key.length > s.length);
-    const parts = CONTRACTIONS[key] ?? (suffix ? [key.slice(0, -suffix[0].length), suffix[1]] : [key]);
+    const parts = WHOLE[key] ?? (key.endsWith("'s") && IS_BASES.has(key.slice(0, -2)) ? [key.slice(0, -2), 'is'] : suffix ? [key.slice(0, -suffix[0].length), suffix[1]] : [key]);
     for (const k of parts) out.push({ key: k, start, end });
   }
   return out;
@@ -96,16 +135,34 @@ const escape = (s: string) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
 /** Where gstack renders a human's deferral reason (R): the voice detail, the consensus head, the gate title, a blocker's why, the run log. */
 const DEFERRAL_RENDERINGS = (r: string) => [`^deferred: (${r})$`, `\\((${r}); by human\\)`, `accepted-with-findings at round \\d+ \\((${r})\\): `, `closed deferred \\((${r})\\) with this finding open`, `, by human\\): (${r}); \\d+ open finding`];
 
-/** The scan index for owner replies (1-2 word and grammar replies skipped), deferral reasons and carried words. */
+/**
+ * The scan index: each owner reply and deferral reason of MIN_TOKENS+ words (grammar skipped), each run of MIN_TOKENS+
+ * of its words between echoed windows, and the placements of deferral reasons and carried words.
+ */
 export function ownerIndex(replies: OwnerSource[], o: { deferred?: OwnerSource[]; carried?: string[] } = {}): OwnerWords {
-  const keysOf = (texts: string[]) => ` ${texts.map(t => tokens(t).map(k => k.key).join(' ')).join(' \u0001 ')} `;
+  const keysOf = (texts: string[]) => ` ${texts.flatMap(t => [tokens(t, false), tokens(t)].map(ts => ts.map(k => k.key).join(' '))).join(' \u0001 ')} `;
   const seen = new Set<string>();
-  const indexed = [...replies, ...(o.deferred ?? [])].flatMap(s => {
+  const indexed: Indexed[] = [];
-    const keys = tokens(s.text).map(t => t.key);
+  const add = (keys: string[], prior: string) => {
-    if (s.parsed || keys.length < MIN_TOKENS || seen.has(keys.join(' '))) return [];
+    if (keys.length < MIN_TOKENS || seen.has(keys.join(' '))) return;
     seen.add(keys.join(' '));
-    return [{ keys, set: new Set(keys), prior: keysOf(s.prior ?? []) }];
+    indexed.push({ keys, set: new Set(keys), prior });
-  });
+  };
+  for (const s of [...replies, ...(o.deferred ?? [])]) {
+    if (s.parsed) continue;
+    const keys = tokens(s.text).map(t => t.key);
+    const prior = keysOf(s.prior ?? []);
+    add(keys, prior);
+    const size = Math.min(keys.length, WINDOW);
+    const echo = new Set<number>();
+    for (let j = 0; j + size <= keys.length; j++) if (prior.includes(` ${keys.slice(j, j + size).join(' ')} `)) for (let k = j; k < j + size; k++) echo.add(k);
+    for (let j = 0; echo.size && j < keys.length;) {
+      let k = j;
+      while (k < keys.length && !echo.has(k)) k++;
+      if (k > j) add(keys.slice(j, k), prior);
+      j = k + 1;
+    }
+  }
   const placements = [
     ...(o.deferred ?? []).flatMap(d => DEFERRAL_RENDERINGS(escape(d.text))),
     ...(o.carried ?? []).map(w => `words=(${escape(JSON.stringify(w))})(?= by=|$)`),
@@ -113,12 +170,11 @@ export function ownerIndex(replies: OwnerSource[], o: { deferred?: OwnerSource[]
   return { replies: indexed, placements };
 }
 
-/** Every owner-words span in `text` (placements, then quotes less echoes), merged, as original [start, end) offsets. */
+/** Every owner-words span in `text` (placements, then quotes less echoes, in both views), merged, as original [start, end) offsets. */
 export function ownerSpans(text: string, owner: OwnerWords): Span[] {
   const spans: Span[] = [];
   for (const re of owner.placements) for (const m of text.matchAll(re)) spans.push([m.indices![1]![0], m.indices![1]![1]]);
-  if (owner.replies.length) {
+  for (const ts of owner.replies.length ? [tokens(text, false), tokens(text)] : []) {
-    const ts = tokens(text);
     const keys = ts.map(t => t.key);
     for (const r of owner.replies) {
       const size = Math.min(r.keys.length, WINDOW);

test/autoplan-owner-words.test.ts

@@ -51,6 +51,52 @@ describe('the quote rule', () => {
     for (const word of ['take', 'flag', 'want', 'bang', 'roll']) expect(r.text).not.toContain(word);
   });
 
+  test.each([
+    ['an HTML tag', 'Jane Doe<br>vetoed it'],
+    ['bold tags', '<b>Jane Doe</b> vetoed it'],
+    ['an HTML comment', 'Jane <!-- aside --> Doe vetoed it'],
+    ['a reference link', '[Jane Doe][1] vetoed it\n\n[1]: https://x.test "the source"'],
+    ['a footnote marker', 'Jane Doe[^1] vetoed it'],
+    ['a link title', '[Jane Doe](https://x.test "the source") vetoed it'],
+    ['a literal JSON newline escape', '{"q": "Jane\\nDoe vetoed it"}'],
+    ['a zero-width space entity', 'Jane Do&#8203;e vetoed it'],
+    ['a soft hyphen entity', 'Jane Do&#173;e vetoed it'],
+    ['a named zero-width entity', 'Jane Do&ZeroWidthSpace;e vetoed it'],
+    ['a left-to-right mark', 'Jane Do\u200ee vetoed it'],
+    ['a right-to-left mark', 'Jane Do\u200fe vetoed it'],
+    ['a directional isolate', 'Jane \u2066Doe\u2069 vetoed it'],
+    ['a combining grapheme joiner', 'Jane Do\u034fe vetoed it'],
+    ['a variation selector', 'Jane Doe\ufe0f vetoed it'],
+  ])('%s does not hide a short quote', (_, text) => {
+    const r = redactOwnerWords(text, ownerIndex([{ text: 'Jane Doe vetoed it' }]));
+    expect(r.redacted).toBeGreaterThanOrEqual(1);
+    for (const word of ['Jane', 'vetoed']) expect(r.text).not.toContain(word);
+  });
+
+  test('the redaction covers the original characters, markup inside the quote included', () => {
+    expect(redactOwnerWords('X: <b>Jane Doe</b> vetoed it.', ownerIndex([{ text: 'Jane Doe vetoed it' }])).text).toBe(`X: <b>${MARK}.`);
+    expect(redactOwnerWords('{"q": "I don\\u2019t want it now"}', ownerIndex([{ text: "I don't want it now" }])).text).toBe(`{"q": "${MARK}"}`);
+  });
+
+  test.each([
+    ['I cannot accept it', "I can't accept it"],
+    ['I cannot accept it', 'I can not accept it'],
+    ['It is too risky now', "It's too risky now"],
+    ['I would ship it today', "I'd ship it today"],
+    ['We will not ship it', "We won't ship it"],
+    ['They are not ready yet', "They aren't ready yet"],
+    ['They are not ready yet', "They're not ready yet"],
+  ])('the reply %p matches %p', (reply, text) => {
+    expect(redactOwnerWords(text, ownerIndex([{ text: reply }])).redacted).toBe(1);
+  });
+
+  test('new words a reply adds to echoed text are a quote on their own, whatever the whole reply\'s length', () => {
+    const line = 'Persist the attempt count on the job row so a restarted worker resumes the schedule.';
+    const o = ownerIndex([{ text: `${line.slice(0, -1)}, and page Jane Doe on call.`, prior: [line] }]);
+    expect(redactOwnerWords('Note: and page Jane Doe on call.', o).text).toBe(`Note: ${MARK}.`);
+    expect(redactOwnerWords(line, o).redacted).toBe(0);
+  });
+
   test('a quote extends over the neighboring words of the same reply, so none is left on either side', () => {
     const reply = 'Honestly I do not want a big bang rollout on the retry path';
     expect(quoted('Well honestly, I don\u2019t want a big bang rollout on the retry path!', reply)).toEqual(['honestly, I don\u2019t want a big bang rollout on the retry path']);

test/autoplan-public-owner-words.test.ts

@@ -277,4 +277,23 @@ describe("export --public drops the owner's own words (GST-1243)", () => {
     expect(fs.readFileSync(path.join(pub, 'ceo-native.a5.bound.md'), 'utf8')).toContain(`Context: ${MARK}, per the relay.`);
     expect(allText(pub)).not.toContain('Jane Doe');
   });
+
+  test('markup and reference links around a short quote do not hide it from export or validate', () => {
+    const { out, dir } = ownerBundle({ wa: 'Jane Doe vetoed it', ceo: { native: 'Context: Jane Doe<br>vetoed it, and [Jane Doe][1] vetoed it.\n\n[1]: https://x.test' }, eng: {} });
+    const pub = path.join(dir, 'public');
+    expect(r.ap(['export', '--out', out, '--public', pub]).status).toBe(0);
+    const bound = fs.readFileSync(path.join(pub, 'ceo-native.a5.bound.md'), 'utf8');
+    expect(bound).not.toContain('Jane');
+    expect(bound).toContain(`Context: ${MARK}, and ${MARK}.`);
+    expect(artifact(['validate', '--public', pub, '--private', path.join(out, 'run.json')]).status).toBe(0);
+  });
+
+  test('new words a reply adds to an echoed plan line are redacted when quoted alone', () => {
+    const PLAN_LINE = 'Persist the attempt count on the job row so a restarted worker resumes the schedule.';
+    const { out, dir } = ownerBundle({ wb: `${PLAN_LINE.slice(0, -1)}, and page Jane Doe on call.`, ceo: { native: 'Ops note: and page Jane Doe on call.' }, eng: {} });
+    const pub = path.join(dir, 'public');
+    expect(r.ap(['export', '--out', out, '--public', pub]).status).toBe(0);
+    expect(fs.readFileSync(path.join(pub, 'ceo-native.a5.bound.md'), 'utf8')).toContain(`Ops note: ${MARK}.`);
+    expect(fs.readFileSync(path.join(pub, 'plan.md'))).toEqual(fs.readFileSync(path.join(out, 'plan.md')));
+  });
 });