From 1c28c09b763307fb4c125fef595dc297d7d10446 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sun, 25 Oct 2026 16:54:50 -0700 Subject: [PATCH] fix(plan-tune): map the clicked AskUserQuestion label to the registered option key at capture (#2623) (#3394) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- CHANGELOG.md | 14 ++++ VERSION | 2 +- agents-digest/gstack-AGENTS.md | 2 +- bin/gstack-question-log | 29 +++++++- package.json | 2 +- scripts/psychographic-signals.ts | 83 +++++++++++++++++++++ test/plan-tune.test.ts | 120 +++++++++++++++++++++++++++++++ 7 files changed, 246 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 38c5d4c45..234ee019f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,19 @@ # Changelog +## [1.91.223.0] - 2026-10-25 + +**/plan-tune's inferred profile finally moves: a clicked answer is logged as the question's registered option key, not its label.** #2623 showed every inferred dimension stuck at 0.5 on a 1,286-event log: the AskUserQuestion hook recorded labels like "A) Fix it now (recommended)", while the signal map that turns answers into dimension nudges is keyed by registered option keys like `fix-now`, so 0 of 1,286 answers matched. Now five hook-captured label answers to the registered architecture-finding question move `architecture_care` from 0.5 to 0.632, and `--trace` lists all five. + +### Fixed +- **`bin/gstack-question-log`**: on a hook-captured event (`source: hook`) for a registered `question_id`, the clicked label is mapped to that question's registered option key before it is written, and the label is kept in a new `choice_label` field. A label in `recommended` is mapped the same way, so `followed_recommendation` compares keys. Ad-hoc and `hook-` ids and unmatched or ambiguous labels are written as given. Rows from other writers (the agent's own log call, auto-decide, Codex import, typed free text) are unchanged, because readers such as the plan-ceo-review auto-decide check match the full mode names those rows carry. Existing logs are not rewritten. +- **`scripts/psychographic-signals.ts`**: new `resolveOptionKey(question_id, label)`. It strips the `A)` selector and `(recommended)` suffix, then matches the label's leading words against the question's own options only: each key's own words (`fix-now` → "fix now") plus a short cue list ("Add to TODOs" → `defer`, "Ship as-is" → `skip`, "HOLD SCOPE" → `hold`). The longest cue wins and a tie stays unmapped. + +### Tests +- `test/plan-tune.test.ts`: every registered option resolves from its own words, bare and as an `A) … (recommended)` label; 20 common phrasings, including the four plan-ceo-review mode names, map to their keys; negative controls (unregistered ids, a question with no options, a label naming no option, a mid-sentence cue, another question's key, a number that looks like a selector) stay unmapped. Two capture tests run the real bin: hook labels become keys with `choice_label` kept, while agent rows (including a `HOLD SCOPE` auto-decide row), ad-hoc ids, unmatched labels and free text are untouched, and a picked label follows a key recommendation. Both capture tests fail against the old bin. + +### Gate integrity +- No test expectation, threshold, budget, timeout, trial count or case kind changed. Existing tests that pin label rows from the agent, auto-decide and Codex import paths pass unchanged. + ## [1.91.222.0] - 2026-10-25 **Five flaky tests now fail only for real defects, and a judge reply that isn't valid JSON says so in its failure line (daily flaky-fix pass, GSTA-91).** `context-restore-fragment-match` failed 0/3 on main cb99df4, although every run restored the right checkpoint: to find the checkpoint whose title matches, the agent often prints all three, and the case failed whenever another checkpoint's body appeared anywhere in the transcript. `test/gstack-ff-update.test.ts` counted every `sleep 300` on the machine, so heartbeats from parallel test files changed its count (remote pregate at #3412's head: expected "1", received "3"). The `GSTACK_CAPY=1` install-admission case in `test/gstack-skill-start.test.ts` makes ten `gstack-skill-start` spawns under bun's 5 s default timeout and took 3.6–5.7 s on a 4-core machine. The QA-only fixture's late scenarios mocked a 1000 ms capture budget but ran each attempt's real setup inside it. The setup-browser-cookies workflow judge in train-4 gate 40173759210 returned JSON with an unescaped quote in its reasoning, and the flake record read only "JSON Parse error: Expected '}'". diff --git a/VERSION b/VERSION index f9722c1cb..999c96dfa 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.91.222.0 +1.91.223.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index b0dec46e0..fc6fd4bae 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.91.222.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.91.223.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/bin/gstack-question-log b/bin/gstack-question-log index e1722f95f..7d7fd7b00 100755 --- a/bin/gstack-question-log +++ b/bin/gstack-question-log @@ -18,8 +18,13 @@ # door_type — one-way | two-way # (optional — looked up from registry if omitted) # options_count — number of options presented (positive integer) -# user_choice — key user selected (free string; registry-options preferred) -# recommended — option key the agent recommended (optional) +# user_choice — key user selected (free string; registry-options preferred). +# On hook-captured events (source=hook) for a registered +# question_id, the clicked UI label is mapped to its +# registered option key (#2623) and kept in choice_label. +# Unmatched labels and other sources stay as given. +# recommended — option key the agent recommended (optional; mapped +# the same way on hook-captured events) # followed_recommendation — bool (optional — computed if both present) # session_id — stable session identifier # ts — ISO 8601 timestamp (auto-injected if missing) @@ -46,8 +51,9 @@ INPUT="$1" TMPERR=$(mktemp "${TMPDIR:-/tmp}/gstack-question-log.XXXXXX") || { echo "gstack-question-log: could not create a temp file in ${TMPDIR:-/tmp}, so the question was not logged. Fix: point TMPDIR at a writable directory." >&2; exit 1; } trap 'rm -f "$TMPERR"' EXIT set +e -VALIDATED=$(printf '%s' "$INPUT" | GSTACK_LIB_DIR="$SCRIPT_DIR/../lib" bun --no-env-file --config=/dev/null -e " +VALIDATED=$(printf '%s' "$INPUT" | GSTACK_LIB_DIR="$SCRIPT_DIR/../lib" GSTACK_SCRIPTS_DIR="$SCRIPT_DIR/../scripts" bun --no-env-file --config=/dev/null -e " const { hasInjection } = await import(process.env.GSTACK_LIB_DIR + '/jsonl-store.ts'); +const { resolveOptionKey } = await import(process.env.GSTACK_SCRIPTS_DIR + '/psychographic-signals.ts'); const path = require('path'); const raw = await Bun.stdin.text(); let j; @@ -173,6 +179,23 @@ if (j.recommended !== undefined) { if (j.recommended.length > 64) j.recommended = j.recommended.slice(0, 64); } +// Map the clicked UI label to its registered option key (#2623). SIGNAL_MAP +// and question preferences are keyed by the registry's option keys, but the +// PostToolUse hook only sees the label. Hook events and registered ids only; +// an unmatched or ambiguous label stays as given. Agent-written rows keep +// what the agent logged: skills such as plan-ceo-review log full mode names +// that their own readers match on. +if (j.source === 'hook') { + const key = resolveOptionKey(j.question_id, j.user_choice); + if (key !== undefined && key !== j.user_choice) { + j.choice_label = j.user_choice; + j.user_choice = key; + } + if (j.recommended !== undefined) { + j.recommended = resolveOptionKey(j.question_id, j.recommended) ?? j.recommended; + } +} + // followed_recommendation — compute if both sides present. An __unknown__ // choice means extraction failed, not that the user rejected the // recommendation — leave the field absent so metrics can't be poisoned. diff --git a/package.json b/package.json index 7f40e8da9..28dbec666 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.91.222", + "version": "1.91.223", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", diff --git a/scripts/psychographic-signals.ts b/scripts/psychographic-signals.ts index decc8c6d5..43d4a89ab 100644 --- a/scripts/psychographic-signals.ts +++ b/scripts/psychographic-signals.ts @@ -223,6 +223,89 @@ export const SIGNAL_MAP: Record> = { }, }; +/** + * Label cues: option key → leading words of UI labels that mean that key. + * + * The AskUserQuestion hook only sees the label the user clicked ("A) Fix it + * now (recommended)"), never a stable key, while SIGNAL_MAP is keyed by the + * registry's option keys (#2623). resolveOptionKey() maps a label back to a + * key when bin/gstack-question-log records a hook event. Every key also + * matches its own words ("fix-now" → "fix now"), so only phrasings beyond + * that need an entry here. Cues are + * lowercase words separated by single spaces and match at the start of the + * normalized label; candidates are limited to the question's own registered + * options, and the longest cue wins. + */ +const OPTION_LABEL_CUES: Record = { + accept: ['yes', 'approve', 'agree', 'add', 'include', 'apply', 'adopt', 'enable', 'go ahead', 'do it', 'proceed', 'run', 'create', 'merge', 'roll back', 'migrate', 'open', 'promote', 'reorganize', 'polish'], + reject: ['no', 'disagree', 'decline', 'cancel', 'abort', 'dont', 'do not', 'keep current', 'keep as is', 'keep it as is'], + skip: ['no', 'no thanks', 'not now', 'dont', 'do not', 'pass', 'drop', 'ignore', 'leave it', 'ship as is', 'ship it as is'], + defer: ['later', 'fix later', 'add to todos', 'add to todo', 'add a todo', 'todo', 'postpone', 'backlog', 'park', 'track it'], + 'fix-now': ['fix', 'fix it', 'fix them', 'fix before shipping'], + 'fix-anyway': ['fix', 'fix it'], + investigate: ['dig in', 'look into', 'debug', 'diagnose', 'root cause'], + 'ack-and-ship': ['ack', 'acknowledge', 'ship', 'ship anyway', 'ship it'], + 'accept-risk': ['accept', 'accept the risk', 'live with it'], + 'false-positive': ['not a bug', 'mark false positive', 'mark as false positive', 'mark it false positive'], + 'add-test': ['add', 'add tests', 'add the test', 'write', 'write the test', 'test it'], + expand: ['scope expansion'], + selective: ['cherry pick', 'selective expansion'], + hold: ['keep', 'keep scope', 'hold scope'], + reduce: ['cut', 'trim', 'shrink', 'scope down', 'scope reduction'], + revise: ['edit', 'change', 'rework'], + restart: ['start over', 'redo'], + 'build-now': ['build', 'build it now', 'include in this pr'], + 'keep-local': ['keep'], + 'keep-vendored': ['keep'], + 'yes-upgrade': ['yes', 'upgrade'], + 'always-auto': ['always'], + 'not-now': ['later', 'snooze'], + 'never-ask': ['never', 'disable'], +}; + +function normalizeOptionLabel(label: string): string { + return label + .toLowerCase() + .replace(/\s*\(recommended\)\s*$/, '') + .replace(/['\u2019]/g, '') + .replace(/^\s*(?:option\s+)?[a-z0-9]\s*(?:[).:]|-(?=\s))\s+/, '') + .replace(/[^a-z0-9]+/g, ' ') + .trim(); +} + +/** + * Map a chosen UI label to the registered option key of its question. + * + * Returns the key when exactly one of the question's registered options + * matches with the longest cue; undefined for unregistered ids, questions + * without options, unmatched labels and ties, so an unclear answer stays + * unmapped instead of nudging the wrong dimension. + */ +export function resolveOptionKey(question_id: string, choice: string): string | undefined { + const options = (QUESTIONS as Record)[question_id]?.options as string[] | undefined; + if (!options?.length) return undefined; + if (options.includes(choice)) return choice; + const label = normalizeOptionLabel(choice); + if (!label) return undefined; + let best: string | undefined; + let bestWords = 0; + let tie = false; + for (const key of options) { + for (const cue of [key.replace(/-/g, ' '), ...(OPTION_LABEL_CUES[key] ?? [])]) { + if (label !== cue && !label.startsWith(cue + ' ')) continue; + const words = cue.split(' ').length; + if (words > bestWords) { + best = key; + bestWords = words; + tie = false; + } else if (words === bestWords && best !== key) { + tie = true; + } + } + } + return tie ? undefined : best; +} + /** * Apply a user choice for a question to the running dimension totals. * diff --git a/test/plan-tune.test.ts b/test/plan-tune.test.ts index a615e95fe..413ed07f9 100644 --- a/test/plan-tune.test.ts +++ b/test/plan-tune.test.ts @@ -33,6 +33,7 @@ import { validateRegistrySignalKeys, newDimensionTotals, normalizeToDimensionValue, + resolveOptionKey, ALL_DIMENSIONS, } from '../scripts/psychographic-signals'; import { @@ -290,6 +291,125 @@ describe('psychographic signal map', () => { }); }); +// ----------------------------------------------------------------------- +// #2623 — chosen UI label → registered option key at capture time +// ----------------------------------------------------------------------- + +describe('option label → registered key (#2623)', () => { + const words = (key: string) => key.replace(/-/g, ' '); + const title = (s: string) => s.charAt(0).toUpperCase() + s.slice(1); + + test('every registered option resolves from its own words, as a bare label and as an AUQ label', () => { + const misses: string[] = []; + for (const q of Object.values(QUESTIONS as Record)) { + for (const key of (q.options ?? []) as string[]) { + for (const label of [key, words(key), `A) ${title(words(key))} (recommended)`, `B. ${title(words(key))}`]) { + if (resolveOptionKey(q.id, label) !== key) misses.push(`${q.id}: ${JSON.stringify(label)} → ${resolveOptionKey(q.id, label)}`); + } + } + } + expect(misses).toEqual([]); + }); + + test('common phrasings map to the question\'s own key', () => { + const cases: Array<[string, string, string]> = [ + ['plan-eng-review-arch-finding', 'A) Fix it now (recommended)', 'fix-now'], + ['plan-eng-review-arch-finding', 'Fix later', 'defer'], + ['plan-eng-review-arch-finding', 'B) Defer to TODOS.md', 'defer'], + ['plan-eng-review-arch-finding', 'C) Accept the risk and ship', 'accept-risk'], + ['plan-ceo-review-expansion-proposal', 'A) Add to plan (recommended)', 'accept'], + ['plan-ceo-review-expansion-proposal', 'B) Add to TODOs', 'defer'], + ['plan-ceo-review-mode', 'A) Expand scope (Recommended)', 'expand'], + ['plan-ceo-review-mode', 'A) SCOPE EXPANSION', 'expand'], + ['plan-ceo-review-mode', 'B) SELECTIVE EXPANSION', 'selective'], + ['plan-ceo-review-mode', 'C) HOLD SCOPE (recommended)', 'hold'], + ['plan-ceo-review-mode', 'D) SCOPE REDUCTION', 'reduce'], + ['plan-ceo-review-mode', 'Cherry-pick expansions', 'selective'], + ['plan-ceo-review-mode', 'Cut to the minimum', 'reduce'], + ['ship-pre-landing-review-fix', 'Ship as-is', 'skip'], + ['ship-test-failure-triage', 'Ship anyway, the failures are known', 'ack-and-ship'], + ['review-finding-fix', 'Not a bug, the input is trusted', 'false-positive'], + ['land-and-deploy-merge-confirm', 'Yes, merge it', 'accept'], + ['land-and-deploy-merge-confirm', 'Don’t merge yet', 'reject'], + ['plan-eng-review-test-gap', 'Add the test now', 'add-test'], + ['office-hours-mode-goal', 'Hackathon / demo', 'hackathon'], + ]; + for (const [id, label, key] of cases) { + expect({ id, label, key: resolveOptionKey(id, label) }).toEqual({ id, label, key }); + } + }); + + test('negative controls stay unmapped', () => { + // Unregistered (ad-hoc or hook-hash) ids: no registered options to map to. + expect(resolveOptionKey('plan-eng-review-ci-typecheck', 'Fix now')).toBeUndefined(); + expect(resolveOptionKey('hook-0123456789', 'Accept')).toBeUndefined(); + // Registered question with no options list. + expect(resolveOptionKey('plan-devex-review-persona', 'Accept')).toBeUndefined(); + // A label that names none of the question's options. + expect(resolveOptionKey('plan-eng-review-arch-finding', 'Restore the parallel typecheck job')).toBeUndefined(); + // Cues anchor at the start of the label, never mid-sentence. + expect(resolveOptionKey('review-sql-safety', 'Ship it, then fix')).toBeUndefined(); + expect(resolveOptionKey('review-sql-safety', "Don't fix")).toBeUndefined(); + // Another question's key is not a candidate. + expect(resolveOptionKey('qa-tier', 'Fix now')).toBeUndefined(); + // A selector-looking number is not stripped into a match. + expect(resolveOptionKey('qa-tier', '1.5x deeper')).toBeUndefined(); + expect(resolveOptionKey('plan-eng-review-arch-finding', '')).toBeUndefined(); + }); + + function logAndRead(entries: Array>): Array> { + const tmpHome = fs.mkdtempSync(path.join(require('os').tmpdir(), 'gstack-2623-')); + try { + const { spawnSync } = require('child_process'); + const env = { ...process.env, GSTACK_HOME: tmpHome, GSTACK_QUESTION_LOG_NO_DERIVE: '1' }; + const logBin = path.join(ROOT, 'bin', 'gstack-question-log'); + for (const e of entries) { + const r = spawnSync(logBin, [JSON.stringify({ skill: 'plan-eng-review', question_summary: 'q?', ...e })], { env, cwd: ROOT, encoding: 'utf-8', timeout: 30_000 }); + expect(r.status).toBe(0); + } + const [slug] = fs.readdirSync(path.join(tmpHome, 'projects')); + return fs + .readFileSync(path.join(tmpHome, 'projects', slug, 'question-log.jsonl'), 'utf-8') + .trim() + .split('\n') + .map((l) => JSON.parse(l)); + } finally { + fs.rmSync(tmpHome, { recursive: true, force: true }); + } + } + + test('gstack-question-log records the key and keeps the label for hook events on registered ids only', () => { + const [hook, agentKey, agentMode, adHoc, unmatched, freeText] = logAndRead([ + { question_id: 'plan-eng-review-arch-finding', user_choice: 'A) Fix it now (recommended)', recommended: 'Fix it now', source: 'hook' }, + { question_id: 'plan-eng-review-arch-finding', user_choice: 'defer', recommended: 'fix-now' }, + { skill: 'plan-ceo-review', question_id: 'plan-ceo-review-mode', user_choice: 'HOLD SCOPE', recommended: 'HOLD SCOPE', auto_decided: true }, + { question_id: 'plan-eng-review-ci-typecheck', user_choice: 'Fix now', source: 'hook' }, + { question_id: 'plan-eng-review-arch-finding', user_choice: 'Restore the parallel typecheck job', source: 'hook' }, + { question_id: 'plan-eng-review-arch-finding', user_choice: 'Fix it but rename the module first', free_text: 'Fix it but rename the module first', source: 'auq-other' }, + ]); + expect(hook).toMatchObject({ user_choice: 'fix-now', choice_label: 'A) Fix it now (recommended)', recommended: 'fix-now', followed_recommendation: true }); + expect(agentKey).toMatchObject({ user_choice: 'defer', recommended: 'fix-now', followed_recommendation: false }); + expect(agentKey.choice_label).toBeUndefined(); + // Agent-written rows keep what the agent logged; the auto-decide harness + // matches plan-ceo-review's full mode names. + expect(agentMode).toMatchObject({ source: 'agent', user_choice: 'HOLD SCOPE', recommended: 'HOLD SCOPE', followed_recommendation: true }); + expect(agentMode.choice_label).toBeUndefined(); + expect(adHoc).toMatchObject({ user_choice: 'Fix now' }); + expect(adHoc.choice_label).toBeUndefined(); + expect(unmatched).toMatchObject({ user_choice: 'Restore the parallel typecheck job' }); + expect(unmatched.choice_label).toBeUndefined(); + expect(freeText).toMatchObject({ user_choice: 'Fix it but rename the module first', source: 'auq-other' }); + expect(freeText.choice_label).toBeUndefined(); + }); + + test('a picked label scores as following a key recommendation', () => { + const [row] = logAndRead([ + { question_id: 'plan-ceo-review-mode', user_choice: 'A) Expand scope (Recommended)', recommended: 'expand', source: 'hook' }, + ]); + expect(row).toMatchObject({ user_choice: 'expand', followed_recommendation: true }); + }); +}); + // ----------------------------------------------------------------------- // Archetypes // -----------------------------------------------------------------------