mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-02 17:40:02 +02:00
Merge remote-tracking branch 'origin/capy/rel-c' into capy/audit-fix-wave
This commit is contained in:
garrytan committed
commit
2dae4bf944
25 files changed
+1158
-88
No files matched your search
@@ -93,7 +93,13 @@ RUN curl --retry 5 --retry-delay 5 --retry-connrefused -fsSL https://bun.sh/inst
|
||||
# skillify HOME discovery on 2.1.237, guard/freeze hooks on 2.1.162).
|
||||
# Bump deliberately, via a PR that runs the PTY gate against the new TUI.
|
||||
# test/ci-image-cli-pin.test.ts fails the free suite if this pin is removed.
|
||||
RUN npm i -g @anthropic-ai/claude-code@2.1.251
|
||||
# 2.1.284 (from 2.1.251, 2026-09-29): 2.1.251 logs
|
||||
# [claude-code:unrecognized_model] for the eval model claude-fable-5-1;
|
||||
# 2.1.284 recognizes it. Local canary: the gate PTY smoke subset parsed on
|
||||
# both TUIs (7/8 pass on 2.1.284, 8/8 on 2.1.251; the one red was the model
|
||||
# still in WebSearch at 300 s), and plan-design-review-plan-mode passed at
|
||||
# 293 s on 2.1.284 where 2.1.251 timed out at 300 s.
|
||||
RUN npm i -g @anthropic-ai/claude-code@2.1.284
|
||||
|
||||
# Playwright system deps (Chromium) — needed for browse E2E tests
|
||||
RUN npx playwright install-deps chromium
|
||||
|
||||
@@ -115,6 +115,8 @@ the repo, traverse submodule worktrees, or execute filters. Note excluded symlin
|
||||
submodule, ignored, unavailable or unreadable source. Handle deletions explicitly.
|
||||
Do not call an absent or unreadable overlay clean. Current raw content may differ
|
||||
even when a clean filter would produce the same Git tree.
|
||||
Sessions have a bounded number of turns. Read related files together: parallel
|
||||
host reads or one read-only command per step, not one file per turn.
|
||||
|
||||
## Start with recent work
|
||||
|
||||
|
||||
@@ -109,6 +109,8 @@ the repo, traverse submodule worktrees, or execute filters. Note excluded symlin
|
||||
submodule, ignored, unavailable or unreadable source. Handle deletions explicitly.
|
||||
Do not call an absent or unreadable overlay clean. Current raw content may differ
|
||||
even when a clean filter would produce the same Git tree.
|
||||
Sessions have a bounded number of turns. Read related files together: parallel
|
||||
host reads or one read-only command per step, not one file per turn.
|
||||
|
||||
## Start with recent work
|
||||
|
||||
|
||||
@@ -767,6 +767,8 @@ review design — real visuals, not text descriptions."
|
||||
|
||||
The ONLY time you skip mockups is when:
|
||||
- `DESIGN_NOT_AVAILABLE` was printed (designer binary not found)
|
||||
- The first `$D` generation command fails before producing an image (for
|
||||
example `No OpenAI API key found`): treat it exactly as `DESIGN_NOT_AVAILABLE`
|
||||
- The plan has zero UI scope (pure backend/API/infrastructure)
|
||||
|
||||
If the user explicitly says "skip mockups" or "text only", respect that. Otherwise, generate.
|
||||
@@ -928,7 +930,7 @@ Note which direction was approved. This becomes the visual reference for all sub
|
||||
|
||||
**Multiple variants/screens:** If the user asked for multiple variants (e.g., "5 versions of the homepage"), generate ALL as separate variant sets with their own comparison boards. Each screen/variant set gets its own subdirectory under `designs/`. Complete all mockup generation and user selection before starting review passes.
|
||||
|
||||
**If `DESIGN_NOT_AVAILABLE`:** Tell the user: "The gstack designer isn't set up yet. Run `$D setup` to enable visual mockups. Proceeding with text-only review, but you're missing the best part." Then proceed to review passes with text-based review.
|
||||
**If `DESIGN_NOT_AVAILABLE`:** Tell the user: "The gstack designer isn't set up yet. Run `$D setup` to enable visual mockups. Proceeding with text-only review, but you're missing the best part." Then proceed to review passes with text-based review. Do not substitute hand-built HTML/CSS wireframes, screenshots or a comparison board of your own: they delay the first review question by minutes and are not designer output.
|
||||
|
||||
## Design Outside Voices (independent)
|
||||
|
||||
|
||||
@@ -205,6 +205,8 @@ review design — real visuals, not text descriptions."
|
||||
|
||||
The ONLY time you skip mockups is when:
|
||||
- `DESIGN_NOT_AVAILABLE` was printed (designer binary not found)
|
||||
- The first `$D` generation command fails before producing an image (for
|
||||
example `No OpenAI API key found`): treat it exactly as `DESIGN_NOT_AVAILABLE`
|
||||
- The plan has zero UI scope (pure backend/API/infrastructure)
|
||||
|
||||
If the user explicitly says "skip mockups" or "text only", respect that. Otherwise, generate.
|
||||
@@ -264,7 +266,7 @@ Note which direction was approved. This becomes the visual reference for all sub
|
||||
|
||||
**Multiple variants/screens:** If the user asked for multiple variants (e.g., "5 versions of the homepage"), generate ALL as separate variant sets with their own comparison boards. Each screen/variant set gets its own subdirectory under `designs/`. Complete all mockup generation and user selection before starting review passes.
|
||||
|
||||
**If `DESIGN_NOT_AVAILABLE`:** Tell the user: "The gstack designer isn't set up yet. Run `$D setup` to enable visual mockups. Proceeding with text-only review, but you're missing the best part." Then proceed to review passes with text-based review.
|
||||
**If `DESIGN_NOT_AVAILABLE`:** Tell the user: "The gstack designer isn't set up yet. Run `$D setup` to enable visual mockups. Proceeding with text-only review, but you're missing the best part." Then proceed to review passes with text-based review. Do not substitute hand-built HTML/CSS wireframes, screenshots or a comparison board of your own: they delay the first review question by minutes and are not designer output.
|
||||
|
||||
{{DESIGN_OUTSIDE_VOICES}}
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@ import {
|
||||
} from './helpers/arm-benchmark-harness';
|
||||
import {
|
||||
armJudge, buildArmJudgePrompt, parseArmJudgeResponse,
|
||||
ARM_JUDGE_ATTEMPTS, callJudge,
|
||||
callJudge,
|
||||
} from './helpers/llm-judge';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
@@ -182,28 +182,25 @@ describe('arm benchmark selftest (free, no API)', () => {
|
||||
expect(score.construct).toBe('none');
|
||||
});
|
||||
|
||||
test('armJudge: bounded retry-on-malformed — recovers once, then gives up', async () => {
|
||||
// Malformed first, valid second: recovers within the 2-attempt bound.
|
||||
test('armJudge: a malformed verdict is a failed sample, never re-asked', async () => {
|
||||
let calls = 0;
|
||||
const flaky = (async () => {
|
||||
const malformedFirst = (async () => {
|
||||
calls++;
|
||||
return calls === 1
|
||||
? { over_engineering: 9, construct: 'garbage' }
|
||||
: { over_engineering: 2, construct: 'repository layer in app.js', reasoning: 'ok' };
|
||||
}) as unknown as typeof callJudge;
|
||||
const recovered = await armJudge('ticket', 'diff --git a/x b/x\n+1\n', { call: flaky });
|
||||
expect(recovered.over_engineering).toBe(2);
|
||||
expect(calls).toBe(ARM_JUDGE_ATTEMPTS);
|
||||
await expect(armJudge('ticket', 'diff --git a/x b/x\n+1\n', { call: malformedFirst }))
|
||||
.rejects.toThrow(/malformed verdict \(never resampled\)/);
|
||||
expect(calls).toBe(1);
|
||||
|
||||
// Always malformed: throws after exactly ARM_JUDGE_ATTEMPTS attempts.
|
||||
let badCalls = 0;
|
||||
const alwaysBad = (async () => {
|
||||
badCalls++;
|
||||
return { nonsense: true };
|
||||
let goodCalls = 0;
|
||||
const wellFormed = (async () => {
|
||||
goodCalls++;
|
||||
return { over_engineering: 2, construct: 'repository layer in app.js', reasoning: 'ok' };
|
||||
}) as unknown as typeof callJudge;
|
||||
await expect(armJudge('ticket', 'diff --git a/x b/x\n+1\n', { call: alwaysBad }))
|
||||
.rejects.toThrow(/no well-formed verdict after 2 attempts/);
|
||||
expect(badCalls).toBe(ARM_JUDGE_ATTEMPTS);
|
||||
expect((await armJudge('ticket', 'diff --git a/x b/x\n+1\n', { call: wellFormed })).over_engineering).toBe(2);
|
||||
expect(goodCalls).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -10,6 +10,7 @@ import captured_ceo_hold_commitment_ar from './fixtures/ceo-hold-commitment-ar.j
|
||||
import captured_ceo_hold_posture_ag from './fixtures/ceo-hold-posture-ag.json';
|
||||
import retainedPreservationCaptures_ceo_hold_posture_ag from './fixtures/ceo-hold-preservation-f359.json';
|
||||
import captured_ceo_mode_colon_at from './fixtures/ceo-mode-colon-at.json';
|
||||
import scrolledReview from './fixtures/ceo-mode-scrolled-review-36606688266.json';
|
||||
import fs_ceo_mode_full_ad from 'node:fs';
|
||||
import os_ceo_mode_full_ad from 'node:os';
|
||||
import path_ceo_mode_full_ad from 'node:path';
|
||||
@@ -1840,3 +1841,55 @@ test('AD v2 prerequisite requires the active native packet identity',()=>{
|
||||
for(const delta of [{answered:true},{failed:true},{sessionId:''},{toolUseId:''}]){const call={...pending(),...delta};const x=frame(call,2);expect(planCountPrerequisitePick(x.routing,x.active)).toBeNull();}
|
||||
});
|
||||
});
|
||||
|
||||
describe('mode submission when the review panel scrolls past the viewport', () => {
|
||||
// Run 36606688266 bundled routing, learnings and the mode choice into one
|
||||
// native call. Its review panel was taller than the terminal, so the tab bar
|
||||
// scrolled away and the harness never submitted HOLD SCOPE.
|
||||
const scrolledTranscript = scrolledReview.transcript as unknown as PlanCountTranscript;
|
||||
const scrolledCall = scrolledTranscript.calls[0] as NativePlanQuestionCall;
|
||||
const scrolledSubmit = (screen: string, screenText: string, mode: 'HOLD SCOPE' | 'SCOPE EXPANSION' = 'HOLD SCOPE',
|
||||
selected: NativePlanQuestionCall = scrolledCall, native: PlanCountTranscript = scrolledTranscript) =>
|
||||
ceoModeSubmissionInput(screen, selected, mode, native, new Set(), screenText);
|
||||
|
||||
test('the captured viewport has no tab bar and ends at the focused Submit prompt', () => {
|
||||
expect(scrolledReview.screen).not.toMatch(/←[^\r\n]+✔\s*Submit\s*→/);
|
||||
expect(scrolledReview.screen.trimEnd()).toMatch(/❯ 1\. Submit answers\s+2\. Cancel$/);
|
||||
expect(scrolledCall.questions.map(q => q.header)).toEqual(['Routing', 'Learnings', 'Review mode']);
|
||||
});
|
||||
|
||||
test('the complete scrolled review submits the selected mode once', () => {
|
||||
expect(scrolledSubmit(scrolledReview.screen, scrolledReview.screenText)).toBe('\r');
|
||||
const seen = new Set<string>();
|
||||
expect(ceoModeSubmissionInput(scrolledReview.screen, scrolledCall, 'HOLD SCOPE', scrolledTranscript, seen, scrolledReview.screenText)).toBe('\r');
|
||||
expect(ceoModeSubmissionInput(scrolledReview.screen, scrolledCall, 'HOLD SCOPE', scrolledTranscript, seen, scrolledReview.screenText)).toBeNull();
|
||||
});
|
||||
|
||||
test('without the accumulated screen text a barless viewport cannot submit', () => {
|
||||
expect(scrolledSubmit(scrolledReview.screen, '')).toBeNull();
|
||||
});
|
||||
|
||||
test('a review showing another mode is not an acknowledgement of the target mode', () => {
|
||||
expect(scrolledSubmit(scrolledReview.screen, scrolledReview.screenText, 'SCOPE EXPANSION')).toBeNull();
|
||||
});
|
||||
|
||||
for (const [name, change] of [
|
||||
['an answer no option offers', (text: string) => text.replace(/→ Enable cross-project \(recommended\)(?![\s\S]*→ Enable cross-project)/, '→ Upload learnings')],
|
||||
['an altered question', (text: string) => text.replace(/D2 — Let gstack(?![\s\S]*D2 — Let gstack)/, 'D2 — Never let gstack')],
|
||||
['a quoted review', (text: string) => text.replace(/Review your answers(?![\s\S]*Review your answers)/, 'Quoted example:\nReview your answers')],
|
||||
['output after the prompt', (text: string) => `${text}\nMore text`],
|
||||
] as const) test(`the scrolled route rejects ${name}`, () => {
|
||||
expect(scrolledSubmit(scrolledReview.screen, change(scrolledReview.screenText))).toBeNull();
|
||||
});
|
||||
|
||||
test('the viewport must still end at the focused Submit prompt', () => {
|
||||
expect(scrolledSubmit(scrolledReview.screen.replace('❯ 1. Submit answers', ' 1. Submit answers\n❯ 2. Cancel'), scrolledReview.screenText)).toBeNull();
|
||||
});
|
||||
|
||||
test('an answered or changed native call cannot be submitted again', () => {
|
||||
expect(scrolledSubmit(scrolledReview.screen, scrolledReview.screenText, 'HOLD SCOPE', { ...scrolledCall, answered: true })).toBeNull();
|
||||
const other = structuredClone(scrolledCall);
|
||||
other.questions[1]!.question += ' (changed)';
|
||||
expect(scrolledSubmit(scrolledReview.screen, scrolledReview.screenText, 'HOLD SCOPE', other)).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -1,7 +1,12 @@
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import type { NativePlanQuestionCall } from './helpers/plan-count-transcript';
|
||||
import { isEngBatchingIssueAUQ } from './helpers/eng-seeded-coverage';
|
||||
import { nativePlanCallFingerprint } from './helpers/claude-pty-runner';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { createEngBatchingIssueCounter, isEngBatchingIssueAUQ } from './helpers/eng-seeded-coverage';
|
||||
import { engSetupAUQ, hasCompletePlanReport, nativePlanCallFingerprint } from './helpers/claude-pty-runner';
|
||||
import batchingCapture from './fixtures/eng-batching-unsourced-brief-36606688266.json';
|
||||
import bulletTargetCapture from './fixtures/eng-batching-bullet-target-rerun.json';
|
||||
|
||||
function question(call: NativePlanQuestionCall, text: string) {
|
||||
const answer = call.answers![call.questions[0]!.question]!;
|
||||
@@ -78,3 +83,112 @@ describe('batching caller counts completed issue decisions across setup boundari
|
||||
expect(check(quoted)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('batching replay of run 36606688266 (unsourced native briefs)', () => {
|
||||
// Run 36606688266 asked one native question per finding (D1-D9 bound to
|
||||
// ledger records R1-R9, D10 a TODO follow-up) but cited no PLAN.md line in the
|
||||
// native brief, so the old detector counted zero review decisions.
|
||||
const FLOOR = 3;
|
||||
const calls = batchingCapture.calls as unknown as NativePlanQuestionCall[];
|
||||
|
||||
function count(plan: string, edit: (calls: NativePlanQuestionCall[]) => void = () => {}) {
|
||||
const copy = structuredClone(calls);
|
||||
edit(copy);
|
||||
const counter = createEngBatchingIssueCounter(() => plan, engSetupAUQ);
|
||||
const counted = copy.filter((call, index) => counter.isReviewAUQ(nativePlanCallFingerprint(call, 0, true), copy.slice(0, index)));
|
||||
return { counted: counted.length, issues: counter.trace.map(entry => entry.issue) };
|
||||
}
|
||||
|
||||
test('the recorded failing verdict is the detector, not the review', () => {
|
||||
expect(batchingCapture.recordedOutcome).toEqual({ outcome: 'completion_summary', step0Count: 10, reviewCount: 0 });
|
||||
expect(calls.every(call => call.answered && call.questions.length === 1)).toBe(true);
|
||||
});
|
||||
|
||||
test('each ledger-bound native decision counts once without a native source citation', () => {
|
||||
const { counted, issues } = count(batchingCapture.plan);
|
||||
expect(issues).toEqual(['R1', 'R2', 'R3', 'R4', 'R5', 'R6', 'R7', 'R8', 'R9'].map(id => `record:${id}`));
|
||||
expect(counted).toBeGreaterThanOrEqual(FLOOR);
|
||||
});
|
||||
|
||||
test('a re-asked decision cannot inflate the count', () => {
|
||||
const { counted } = count(batchingCapture.plan, all => {
|
||||
const again = structuredClone(all[0]!);
|
||||
again.toolUseId += '-again';
|
||||
all.splice(1, 0, again);
|
||||
});
|
||||
expect(counted).toBe(9);
|
||||
});
|
||||
|
||||
const target = 'Review target (fixed): `PLAN.md`';
|
||||
for (const [name, plan] of [
|
||||
['a foreign target', batchingCapture.plan.replace(target, 'Review target (fixed): `OTHER.md`')],
|
||||
['a mixed target', batchingCapture.plan.replace(target, 'Review target (fixed): `OTHER.md` and `PLAN.md`')],
|
||||
['two target declarations', batchingCapture.plan.replace(target, `${target}\nReview target (fixed): \`PLAN.md\``)],
|
||||
['no target declaration', batchingCapture.plan.replace(target, 'Report scope: the fixture repo')],
|
||||
['a report title for another plan', batchingCapture.plan.replace('# Engineering review: Add background job retry framework', '# Engineering review: Replace all customer data')],
|
||||
['an archived report title', batchingCapture.plan.replace('# Engineering review:', '# Archived engineering review:')],
|
||||
['a copied H1 naming another plan', batchingCapture.plan.replace('# Plan: Add background job retry framework', '# Plan: Replace all customer data')],
|
||||
] as const) test(`the unsourced route rejects ${name}`, () => {
|
||||
expect(count(plan).counted).toBe(0);
|
||||
});
|
||||
|
||||
test('the unsourced route rejects a native brief naming another plan or file', () => {
|
||||
const rename = (from: string, to: string) => (all: NativePlanQuestionCall[]) => {
|
||||
for (const call of all) call.questions[0]!.question = call.questions[0]!.question.replace(from, to);
|
||||
};
|
||||
expect(count(batchingCapture.plan, rename('plan "Add background job retry framework"', 'plan "Replace all customer data"')).counted).toBe(0);
|
||||
expect(count(batchingCapture.plan, rename('plan "Add background job retry framework"', 'plan "Add background job retry framework", OTHER.md')).counted).toBe(0);
|
||||
expect(count(batchingCapture.plan, rename('plan "Add background job retry framework"', 'the plan')).counted).toBe(0);
|
||||
});
|
||||
|
||||
test('a saved record whose brief title differs from the native question does not bind it', () => {
|
||||
const plan = batchingCapture.plan.replace(/^Question D1:\n.*$/m, 'Question D1:\nD1 — Some other decision?');
|
||||
expect(count(plan).issues).not.toContain('record:R1');
|
||||
});
|
||||
|
||||
test('the completed report is the early outcome point; a partial report is not', () => {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'eng-batching-report-'));
|
||||
try {
|
||||
const report = path.join(dir, 'report.md');
|
||||
fs.writeFileSync(report, batchingCapture.plan);
|
||||
expect(hasCompletePlanReport(report, 0, Date.now() + 1_000)).toBe(true);
|
||||
fs.writeFileSync(report, batchingCapture.plan.slice(0, batchingCapture.plan.indexOf('## Completion summary')));
|
||||
expect(hasCompletePlanReport(report, 0, Date.now() + 1_000)).toBe(false);
|
||||
fs.writeFileSync(report, batchingCapture.plan.replace('## GSTACK REVIEW REPORT', '```\n## GSTACK REVIEW REPORT') + '\n```\n');
|
||||
expect(hasCompletePlanReport(report, 0, Date.now() + 1_000)).toBe(false);
|
||||
} finally {
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('batching replay of a 2.1.284 rerun (bullet target, unnamed plan)', () => {
|
||||
// Eleven separate native questions; the briefs name no plan and the report
|
||||
// declares '- **Review target (fixed):** `/abs/PLAN.md`' under '# Eng Review — PLAN.md: <plan>'.
|
||||
const calls = bulletTargetCapture.calls as unknown as NativePlanQuestionCall[];
|
||||
const count = (plan: string) => {
|
||||
const counter = createEngBatchingIssueCounter(() => plan, engSetupAUQ);
|
||||
calls.forEach((call, index) => counter.isReviewAUQ(nativePlanCallFingerprint(call, 0, true), calls.slice(0, index)));
|
||||
return counter.trace.map(entry => entry.issue);
|
||||
};
|
||||
|
||||
test('the recorded verdict counted none of the separate decisions', () => {
|
||||
expect(bulletTargetCapture.recordedOutcome).toMatchObject({ reviewCount: 0 });
|
||||
expect(calls.length).toBe(11);
|
||||
});
|
||||
|
||||
test('ledger-bound decisions count once each through the report target field', () => {
|
||||
expect(count(bulletTargetCapture.plan).length).toBe(9);
|
||||
});
|
||||
|
||||
for (const [name, change] of [
|
||||
['a foreign target file', (plan: string) => plan.replace(/(Review target \(fixed\):\*\* `[^`]*\/)PLAN\.md`/, '$1OTHER.md`')],
|
||||
['a second target declaration', (plan: string) => plan.replace('- **Review target (fixed):**', '- **Review target (fixed):** `OTHER.md`\n- **Review target (fixed):**')],
|
||||
['no target declaration', (plan: string) => plan.replace('- **Review target (fixed):**', '- **Report scope:**')],
|
||||
['an archived report title', (plan: string) => plan.replace('# Eng Review —', '# Archived Eng Review —')],
|
||||
] as const) test(`the bullet target route rejects ${name}`, () => {
|
||||
const plan = change(bulletTargetCapture.plan);
|
||||
expect(plan).not.toBe(bulletTargetCapture.plan);
|
||||
expect(count(plan)).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,72 @@
|
||||
{
|
||||
"source": "run 36606688266 plan-ceo-mode-routing HOLD SCOPE: final viewport, accumulated screen text from the last tab frame, and the pending native call",
|
||||
"screen": " \u2502 ELI10: gstack works best when your project's CLAUDE.md includes skill routing rules, so requests like \"review this\n \u2502 diff\" route to the right skill automatically. This is a plain text section appended to CLAUDE.md.\n \u2502 Stakes if we pick wrong: without it you invoke skills by name manually; with it, plain requests auto-route. Either\n \u2502 is reversible.\n \u2502 Recommendation: A because auto-routing removes a step from every future session and costs one commit.\n \u2502 Note: options differ in kind, not coverage \u2014 no completeness score.\n \u2502 Net: convenience now vs. one extra committed section in CLAUDE.md. Plan mode blocks file edits, so if you pick A\n \u2502 the append + commit happens after this review exits plan mode.\n \u2192 Add routing rules (recommended)\n \u2502 \u25cf D2 \u2014 Let gstack search learnings from your other projects on this machine?\n \u2502 Project/branch/task: gstack-plan-count-FwyQuk on main; one-time gstack setup prompt.\n \u2502 ELI10: gstack saves small lessons per project (\"this test runner needs flag X\"). Cross-project mode also searches\n \u2502 lessons from your other local projects when reviewing this one. Everything stays on this machine.\n \u2502 Stakes if we pick wrong: too narrow and you miss patterns you already learned elsewhere; too broad and a client\n \u2502 codebase could surface a lesson from another client's repo in a review.\n \u2502 Recommendation: A because this is a solo-style environment and the data never leaves the machine.\n \u2502 Note: options differ in kind, not coverage \u2014 no completeness score.\n \u2502 Net: more recall vs. strict per-project isolation.\n \u2192 Enable cross-project (recommended)\n \u2502 \u25cf D3 \u2014 R2: Which review mode for the saved-views plan?\n \u2502 Project/branch/task: gstack-plan-count-FwyQuk on main; reviewing PLAN.md \"Add saved project views\".\n \u2502 ELI10: The mode sets my posture for the rest of the review. Expansion pushes for the biggest version, Hold Scope\n \u2502 stress-tests exactly what you wrote, Reduction strips to the smallest shippable core, and Selective holds your\n \u2502 scope while offering a few add-ons one at a time for you to accept or decline.\n \u2502 Stakes if we pick wrong: too ambitious and a small feature balloons; too strict and we ship personal-only views\n \u2502 when the goal (\"team members repeatedly recreate filters\") may really be a shared-view problem, forcing a second\n \u2502 migration later.\n \u2502 Recommendation: SELECTIVE EXPANSION because the plan is an added capability of ~8\u201310 files, but its member-only\n \u2502 scoping is the one fact that could be wrong: every incumbent ships shared views too, and the table shape decides\n \u2502 whether adding them later is a column or a rewrite. Selective lets you rule on that once without committing to a\n \u2502 bigger build.\n \u2502 Note: options differ in kind, not coverage \u2014 no completeness score.\n \u2502 Net: how much of the review is spent challenging scope vs. hardening the scope you already chose.\n \u2192 HOLD SCOPE\n\nReady to submit your answers?\n\n\u276f 1. Submit answers\n 2. Cancel\n",
|
||||
"screenText": "omething.\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n 4. Chat about this\n\nEnter to select \u00b7 Tab/Arrow keys to navigate \u00b7 Esc to cancel\n\n\n\n \u2612 Learnings \u2610 Review mode \n3R2:Which review mode for the saved-views plan?\nreviewing PLAN.md \"Add saved project views\".\nThe mode sts y posture for the rest of e review. Expansion pushes forthe biggest version, Hld Scop \nstress-testsexactly what youwt, Reduction strpso the smallst shippable cre, and Selective holds your scope \nwhil ofering a few add-onsone at time for you o accept or dcline.\nStakes if we pick wrong:too ambitious and asmall featureballoons; too strict and we ship personal-only views when \nthe goal (\"teammembers repeatedly recreate filters\") ay really be shared-viw problm, forcing a second migration \nlar.\nRcomendation: SELECTIVE EXPANSION becaue the plan is an added capability of ~8\u201310 files, but its member-only \n\u2502scoping is the one fact that could be wrong: every incumbent ships shared views too, and the table shape decides \n\u2502whether adding them later is a column or a rewrite. Selective lets you rule on that once without committing to a \n\u2502bigger build.\n\u2502Note: options differ in kind, not coverage \u2014 no completeness score.\n\u2502Net: how much of the review is spent challenging scope vs. hardening the scope you already chose.\n\n\u276f1.SELECTIVE EXPANSION (recommended)\n\u2705 Keep your fou approach bullets as the bselineand hardens hemwith fullrigor\ufffd\u2705 Offers each expansion \n (shared views, default view, cleanup) as a separate add/defer/skip call\ufffd\u274c A few more decision questions than Hold \n Scope before the deep review starts\n2.HOLDSCOPE\n\u2705 Maximum rigor on exactly what is written: error paths, edge cases, tests, observability\ufffd\u2705 Fastest path to an \n implementation-ready plan wiho scopquetions\ufffd\u274c Shared views and table-shape futureproofing get flagged, not \noffered; possible second migration later\n\n3.SCOPEEXPANSION\n\n\u2705Designstheplatonicsaved-viewsfeature:personal+shared,defaults,sharelinks,cleanup\ufffd\u2705Bestlong-term\n\narchitectureupfront;nofollow-upmigrations\ufffd\u274cTurnsa~10-filefeatureintoamulti-surfacebuildbeforethe\n\ntwo-weekpilotprovesreuse\n\n4.SCOPEREDUCTION\n\n\u2705Findsthesmallestcorethatteststhepilothypothesis(maybecreate/list/applyonly)\ufffd\u2705Lowestriskand\n\nfastesttothetwo-weekreusemeasurement\ufffd\u274cUpdate/deleteandpickerpolishgetdeferred;pilotmaymeasurea\n\nclunkyversionofthefeature\n\n5.Typesomething.\n\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n\n6.Chataboutthis\n\n\n\nEntertoselect\u00b7Tab/Arrowkeystonavigate\u00b7Esctocancel\n\n\n\nReview your answers\n \u2502 \u25cf D1 \u2014 Add gstack skill routing rules to this project'sCLAUDE.md?\n \u2502 Project/branch/task: gstack-plan-count-FwyQuk on main; one-time gstack setup prompt.\n \u2502 ELI10: gstack works best when your project's CLAUDE.md includes skill routing rules, so requests like \"review this\n \u2502 diff\" route to the right skill automatically. This is a plain text section appended to CLAUDE.md.\n \u2502 Stakes if we pick wrong: without it you invoke skills by name manually;withit,plainrequestsauto-route.Either\n \u2502 is reversible.\n \u2502 Recommendation: A because auto-routing removes a step from every future session and costs one commit.\n \u2502 Note:optionsdifferinkind,notcoverage\u2014nocompletenessscore.\n \u2502 Net: convenience now vs. one extra committed section in CLAUDE.md. Plan mode blocks file edits, so if you pickA\n \u2502 the appenLine truncated
|
||||
"transcript": {
|
||||
"status": "ready",
|
||||
"calls": [
|
||||
{
|
||||
"sessionId": "4ea226bf-5a31-4342-8141-c76094320f94",
|
||||
"toolUseId": "toolu_01Ee4tS4SoMXqLyS14ayssEy",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D1 \u2014 Add gstack skill routing rules to this project's CLAUDE.md?\nProject/branch/task: gstack-plan-count-FwyQuk on main; one-time gstack setup prompt.\nELI10: gstack works best when your project's CLAUDE.md includes skill routing rules, so requests like \"review this diff\" route to the right skill automatically. This is a plain text section appended to CLAUDE.md.\nStakes if we pick wrong: without it you invoke skills by name manually; with it, plain requests auto-route. Either is reversible.\nRecommendation: A because auto-routing removes a step from every future session and costs one commit.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: convenience now vs. one extra committed section in CLAUDE.md. Plan mode blocks file edits, so if you pick A the append + commit happens after this review exits plan mode.",
|
||||
"header": "Routing",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Add routing rules (recommended)",
|
||||
"description": "\u2705 Plain requests (\"ship this\", \"find the bug\") route to the right skill without naming it\n\u2705 One short section, committed once; teammates get the same routing\n\u274c Adds a gstack-specific block to CLAUDE.md; applied after plan mode exits, not now"
|
||||
},
|
||||
{
|
||||
"label": "No thanks, manual invocation",
|
||||
"description": "\u2705 CLAUDE.md stays exactly as it is; no tool-specific text in the repo\n\u2705 You keep full control over when a skill runs\n\u274c Every skill needs its /name typed; re-enable later via gstack-config routing_declined false"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"question": "D2 \u2014 Let gstack search learnings from your other projects on this machine?\nProject/branch/task: gstack-plan-count-FwyQuk on main; one-time gstack setup prompt.\nELI10: gstack saves small lessons per project (\"this test runner needs flag X\"). Cross-project mode also searches lessons from your other local projects when reviewing this one. Everything stays on this machine.\nStakes if we pick wrong: too narrow and you miss patterns you already learned elsewhere; too broad and a client codebase could surface a lesson from another client's repo in a review.\nRecommendation: A because this is a solo-style environment and the data never leaves the machine.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: more recall vs. strict per-project isolation.",
|
||||
"header": "Learnings",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Enable cross-project (recommended)",
|
||||
"description": "\u2705 Lessons from other local projects surface here when they match a finding\n\u2705 Local only; nothing leaves the machine, and it is a one-line config flip to undo\n\u274c Not ideal if you work on multiple client codebases where mixing context is a concern"
|
||||
},
|
||||
{
|
||||
"label": "Keep project-scoped only",
|
||||
"description": "\u2705 Strict isolation: reviews here only see this project's lessons\n\u2705 Safe default for multi-client work or shared machines\n\u274c Patterns you already learned on another repo will not be suggested here"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"question": "D3 \u2014 R2: Which review mode for the saved-views plan?\nProject/branch/task: gstack-plan-count-FwyQuk on main; reviewing PLAN.md \"Add saved project views\".\nELI10: The mode sets my posture for the rest of the review. Expansion pushes for the biggest version, Hold Scope stress-tests exactly what you wrote, Reduction strips to the smallest shippable core, and Selective holds your scope while offering a few add-ons one at a time for you to accept or decline.\nStakes if we pick wrong: too ambitious and a small feature balloons; too strict and we ship personal-only views when the goal (\"team members repeatedly recreate filters\") may really be a shared-view problem, forcing a second migration later.\nRecommendation: SELECTIVE EXPANSION because the plan is an added capability of ~8\u201310 files, but its member-only scoping is the one fact that could be wrong: every incumbent ships shared views too, and the table shape decides whether adding them later is a column or a rewrite. Selective lets you rule on that once without committing to a bigger build.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: how much of the review is spent challenging scope vs. hardening the scope you already chose.",
|
||||
"header": "Review mode",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "SELECTIVE EXPANSION (recommended)",
|
||||
"description": "\u2705 Keeps your four approach bullets as the baseline and hardens them with full rigor\n\u2705 Offers each expansion (shared views, default view, cleanup) as a separate add/defer/skip call\n\u274c A few more decision questions than Hold Scope before the deep review starts"
|
||||
},
|
||||
{
|
||||
"label": "HOLD SCOPE",
|
||||
"description": "\u2705 Maximum rigor on exactly what is written: error paths, edge cases, tests, observability\n\u2705 Fastest path to an implementation-ready plan with no scope questions\n\u274c Shared views and table-shape futureproofing get flagged, not offered; possible second migration later"
|
||||
},
|
||||
{
|
||||
"label": "SCOPE EXPANSION",
|
||||
"description": "\u2705 Designs the platonic saved-views feature: personal + shared, defaults, share links, cleanup\n\u2705 Best long-term architecture up front; no follow-up migrations\n\u274c Turns a ~10-file feature into a multi-surface build before the two-week pilot proves reuse"
|
||||
},
|
||||
{
|
||||
"label": "SCOPE REDUCTION",
|
||||
"description": "\u2705 Finds the smallest core that tests the pilot hypothesis (maybe create/list/apply only)\n\u2705 Lowest risk and fastest to the two-week reuse measurement\n\u274c Update/delete and picker polish get deferred; pilot may measure a clunky version of the feature"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": false,
|
||||
"failed": false
|
||||
}
|
||||
],
|
||||
"assistantMessages": []
|
||||
}
|
||||
}
|
||||
+359
@@ -0,0 +1,359 @@
|
||||
{
|
||||
"source": "local targeted rerun smoke-2.1.284-1790711269 (Claude Code 2.1.284) of plan-eng-multi-finding-batching: observation.json transcript.calls and the saved report replayed from its Write/Edit inputs",
|
||||
"recordedOutcome": {
|
||||
"outcome": "collection_complete",
|
||||
"step0Count": 11,
|
||||
"reviewCount": 0
|
||||
},
|
||||
"calls": [
|
||||
{
|
||||
"sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
|
||||
"toolUseId": "toolu_01XnWRrh4F44QdznytAGmriy",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D1 \u2014 Reuse the job library's retry hooks or roll a custom scheduler?\nProject/branch/task: main branch, retry-framework plan; adding retries to 5 background workers.\nELI10: The job library you already use has retry hooks built in, and your plan says your custom version would be \"the same shape.\" Building your own copy inside each worker means five hand-written schedulers to keep correct, versus configuring one curve the library already knows how to run. The plan's reason for going custom is \"full control over the curve,\" and most retry hook APIs give you that through a backoff callback.\nStakes if we pick wrong: five bespoke schedulers drift apart, each grows its own bugs (no jitter, no cap, retry storms), and nobody at 3am knows which curve a given worker actually runs.\nRecommendation: A because the plan admits the library version has the same shape, and a custom curve is usually a config callback, not a new scheduler.\nCompleteness: A=9/10, B=5/10, C=n/a (investigation, decides nothing)\nPros / cons:\nA) Library hooks + custom curve (recommended)\n \u2705 One scheduler the library already tests; the curve becomes a per-worker config function (human: ~1 day / CC: ~20 min)\n \u2705 Attempt-count persistence, max attempts and dead-letter usually come along for free with the hooks\n \u274c If the hook API cannot accept an arbitrary curve function, that is a new fact and R1 reopens\nB) Custom inline scheduler (as planned)\n \u2705 Total control of delay math and logging, independent of the library's release cycle\n \u2705 No dependency on hook semantics nobody has read yet (human: ~3 days / CC: ~1 hr)\n \u274c Five hand-rolled schedulers to keep in sync, plus reimplementing attempt persistence and terminal handling\nC) Investigate first: bounded read of the hook API\n \u2705 Settles \"can the hooks take a custom curve\" with evidence before committing either way\n \u2705 Cheap: ~30 min human / ~3 min CC reading the hook signature and docs\n \u274c Decides nothing; R1 stays pending and the review pauses on this question\nNet: You are trading a library dependency you already carry for five copies of scheduler code you would own forever.",
|
||||
"header": "Retry mechanism",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Library hooks + custom curve (recommended)",
|
||||
"description": "Use the job library's built-in retry hooks; supply each worker's backoff curve as a callback/config. One scheduler the library already tests. Attempt persistence, max attempts and dead-letter usually included. Human ~1 day / CC ~20 min. Risk: if the hook API cannot take an arbitrary curve, R1 reopens. Completeness 9/10."
|
||||
},
|
||||
{
|
||||
"label": "Custom inline scheduler (as planned)",
|
||||
"description": "Roll the exponential-backoff scheduler inline in each of the 5 workers as PLAN.md:7-9 proposes. Total control of delay math and logging. Human ~3 days / CC ~1 hr. Cost: five schedulers to keep in sync, plus attempt persistence and terminal handling rebuilt by hand. Completeness 5/10."
|
||||
},
|
||||
{
|
||||
"label": "Investigate hook API first",
|
||||
"description": "Bounded ~30 min human / ~3 min CC read of the library's retry hook signature and docs, then return to this question. Approves nothing; R1 stays pending; R2\u2013R6 unchanged."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D1 \u2014 Reuse the job library's retry hooks or roll a custom scheduler?\nProject/branch/task: main branch, retry-framework plan; adding retries to 5 background workers.\nELI10: The job library you already use has retry hooks built in, and your plan says your custom version would be \"the same shape.\" Building your own copy inside each worker means five hand-written schedulers to keep correct, versus configuring one curve the library already knows how to run. The plan's reason for going custom is \"full control over the curve,\" and most retry hook APIs give you that through a backoff callback.\nStakes if we pick wrong: five bespoke schedulers drift apart, each grows its own bugs (no jitter, no cap, retry storms), and nobody at 3am knows which curve a given worker actually runs.\nRecommendation: A because the plan admits the library version has the same shape, and a custom curve is usually a config callback, not a new scheduler.\nCompleteness: A=9/10, B=5/10, C=n/a (investigation, decides nothing)\nPros / cons:\nA) Library hooks + custom curve (recommended)\n \u2705 One scheduler the library already tests; the curve becomes a per-worker config function (human: ~1 day / CC: ~20 min)\n \u2705 Attempt-count persistence, max attempts and dead-letter usually come along for free with the hooks\n \u274c If the hook API cannot accept an arbitrary curve function, that is a new fact and R1 reopens\nB) Custom inline scheduler (as planned)\n \u2705 Total control of delay math and logging, independent of the library's release cycle\n \u2705 No dependency on hook semantics nobody has read yet (human: ~3 days / CC: ~1 hr)\n \u274c Five hand-rolled schedulers to keep in sync, plus reimplementing attempt persistence and terminal handling\nC) Investigate first: bounded read of the hook API\n \u2705 Settles \"can the hooks take a custom curve\" with evidence before committing either way\n \u2705 Cheap: ~30 min human / ~3 min CC reading the hook signature and docs\n \u274c Decides nothing; R1 stays pending and the review pauses on this question\nNet: You are trading a library dependency you already carry for five copies of scheduler code you would own forever.": "Library hooks + custom curve (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T19:51:49.159Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
|
||||
"toolUseId": "toolu_015Ri9YxuhvdexzxG5KqBTTc",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D2 \u2014 What delivery guarantee does processWebhookJob() keep once it can retry?\nProject/branch/task: main branch, retry-framework plan; the webhook worker is one of the 5 workers gaining retries through library hooks (D1).\nELI10: Today the webhook worker sends each event at most once: if the send fails or times out, the event is dropped, never duplicated. A retry cannot tell \"the request never arrived\" apart from \"it arrived but the response got lost,\" so any retry after a timeout can deliver the same event twice. Adding retries silently flips the guarantee from at-most-once to at-least-once. That is a contract change your webhook receivers depend on, and the plan does not name it.\nStakes if we pick wrong: receivers that are not idempotent process duplicate events (double emails, double charges, double state transitions); or, if we keep dropping on ambiguity, the retry framework never fixes the webhook worker's lost events.\nRecommendation: B because losing events is usually worse than duplicates, and a stable idempotency key makes duplicates safe for receivers; this is still a contract call you know better than the review does.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nPros / cons:\nA) Keep at-most-once: retry only provably-unsent failures\n \u2705 No duplicate deliveries ever; existing receivers keep working with no change on their side\n \u2705 Still recovers the clear cases: connection refused, DNS failure, local enqueue error (human: ~1 day / CC: ~30 min)\n \u274c Timeouts and 5xx-after-send still drop events, so the biggest source of loss stays; needs per-attempt failure classification\nB) Move to at-least-once with a stable idempotency key (recommended)\n \u2705 Every event eventually reaches the receiver; retries after timeouts are safe because the event id stays constant across attempts\n \u2705 Matches Stripe, Shopify and Svix practice; receivers dedupe on the key (human: ~1.5 days / CC: ~30 min incl. docs)\n \u274c Contract change: receivers must dedupe; needs a documented header, changelog entry and migration note for existing receivers\nC) Exclude processWebhookJob() from retries\n \u2705 Zero semantic change for receivers; the other 4 workers still get retries\n \u2705 Smallest diff and no receiver communication (human: ~1 hr / CC: ~5 min)\n \u274c The webhook worker keeps losing events on every transient failure, which is likely why the plan touched it\nNet: Never-duplicate-but-lossy, never-lossy-but-receivers-must-dedupe, or leave the webhook worker exactly as it is.",
|
||||
"header": "Webhook delivery",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Keep at-most-once (retry only pre-send failures)",
|
||||
"description": "Webhook worker retries only failures provably raised before the request was written (connection refused, DNS, local error). Timeouts and 5xx-after-send still drop the event. No duplicates; receivers unchanged. Needs per-attempt failure classification. Human ~1 day / CC ~30 min."
|
||||
},
|
||||
{
|
||||
"label": "At-least-once + idempotency key (recommended)",
|
||||
"description": "Webhook worker retries all transient failures; every delivery carries a stable event id / idempotency key header constant across attempts. Documented contract change with changelog and receiver migration note. Receivers dedupe on the key. Human ~1.5 days / CC ~30 min."
|
||||
},
|
||||
{
|
||||
"label": "Exclude webhook worker from retries",
|
||||
"description": "`processWebhookJob()` keeps today's at-most-once, no-retry behavior; the other 4 workers retry via library hooks per D1. Smallest diff, no receiver impact, webhook events still lost on transient failure. Human ~1 hr / CC ~5 min."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D2 \u2014 What delivery guarantee does processWebhookJob() keep once it can retry?\nProject/branch/task: main branch, retry-framework plan; the webhook worker is one of the 5 workers gaining retries through library hooks (D1).\nELI10: Today the webhook worker sends each event at most once: if the send fails or times out, the event is dropped, never duplicated. A retry cannot tell \"the request never arrived\" apart from \"it arrived but the response got lost,\" so any retry after a timeout can deliver the same event twice. Adding retries silently flips the guarantee from at-most-once to at-least-once. That is a contract change your webhook receivers depend on, and the plan does not name it.\nStakes if we pick wrong: receivers that are not idempotent process duplicate events (double emails, double charges, double state transitions); or, if we keep dropping on ambiguity, the retry framework never fixes the webhook worker's lost events.\nRecommendation: B because losing events is usually worse than duplicates, and a stable idempotency key makes duplicates safe for receivers; this is still a contract call you know better than the review does.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nPros / cons:\nA) Keep at-most-once: retry only provably-unsent failures\n \u2705 No duplicate deliveries ever; existing receivers keep working with no change on their side\n \u2705 Still recovers the clear cases: connection refused, DNS failure, local enqueue error (human: ~1 day / CC: ~30 min)\n \u274c Timeouts and 5xx-after-send still drop events, so the biggest source of loss stays; needs per-attempt failure classification\nB) Move to at-least-once with a stable idempotency key (recommended)\n \u2705 Every event eventually reaches the receiver; retries after timeouts are safe because the event id stays constant across attempts\n \u2705 Matches Stripe, Shopify and Svix practice; receivers dedupe on the key (human: ~1.5 days / CC: ~30 min incl. docs)\n \u274c Contract change: receivers must dedupe; needs a documented header, changelog entry and migration note for existing receivers\nC) Exclude processWebhookJob() from retries\n \u2705 Zero semantic change for receivers; the other 4 workers still get retries\n \u2705 Smallest diff and no receiver communication (human: ~1 hr / CC: ~5 min)\n \u274c The webhook worker keeps losing events on every transient failure, which is likely why the plan touched it\nNet: Never-duplicate-but-lossy, never-lossy-but-receivers-must-dedupe, or leave the webhook worker exactly as it is.": "Keep at-most-once (retry only pre-send failures)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T19:54:01.993Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
|
||||
"toolUseId": "toolu_0132i9qkrzXUVyovu4GZxADn",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D3 \u2014 When a job runs out of retries, where does it go?\nProject/branch/task: main branch, retry-framework plan; retry bounds for all 5 workers running through library hooks (D1).\nELI10: Right now the plan describes the curve between retries but never says how many retries there are or what happens to a job that keeps failing. Without a limit, a poisoned job retries forever and eats worker capacity. With a limit but no landing spot, the job disappears with one log line nobody reads. A dead-letter store keeps the failed job, its payload reference and its last error so someone can inspect and replay it.\nStakes if we pick wrong: either an infinite-retry job starves the queue, or real work silently vanishes after the last attempt and the first sign is a customer asking where their data went.\nRecommendation: A because a dead-letter store plus an alert is a few dozen lines with library hooks, and it turns \"job vanished\" into \"job parked, here is why.\"\nCompleteness: A=10/10, B=5/10, C=3/10\nPros / cons:\nA) Bounded attempts + dead-letter store + alert (recommended)\n \u2705 Exhausted or fatal jobs are kept with last error and attempt history; operators can inspect and replay (human: ~1 day / CC: ~20 min)\n \u2705 Metric and alert on dead-letter growth turns a silent failure into a page at the right time\n \u274c One more table or queue to own, plus a small replay path to build and test\nB) Bounded attempts, log and drop\n \u2705 Simplest bound: `maxAttempts` default 5 per worker, one error log on exhaustion (human: ~2 hr / CC: ~5 min)\n \u2705 No new storage; nothing to operate\n \u274c Exhausted jobs are gone; recovery means replaying from upstream sources by hand, if that is even possible\nC) Leave to library defaults\n \u2705 Zero plan text and zero decision now\n \u2705 Whatever the library does is at least consistent across the 5 workers\n \u274c Nobody knows the limit or the terminal behavior until an incident teaches them; 3am failure mode\nNet: You are trading one small dead-letter store for never having to ask \"where did that job go.\"",
|
||||
"header": "Retry exhaustion",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Bounded + dead-letter + alert (recommended)",
|
||||
"description": "`maxAttempts` default 5 with per-worker override. On exhaustion or fatal error the job lands in a dead-letter store (table or queue) with last error, attempt history and payload reference. Metric and alert on dead-letter growth. Manual replay path. Human ~1 day / CC ~20 min. Completeness 10/10."
|
||||
},
|
||||
{
|
||||
"label": "Bounded, log and drop",
|
||||
"description": "`maxAttempts` default 5 with per-worker override. On exhaustion, log at error level with the last error and drop the job. No new storage, no replay. Human ~2 hr / CC ~5 min. Completeness 5/10."
|
||||
},
|
||||
{
|
||||
"label": "Library defaults, unspecified",
|
||||
"description": "Do not write attempt limits or terminal behavior into the plan; accept whatever the library does by default. Completeness 3/10."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D3 \u2014 When a job runs out of retries, where does it go?\nProject/branch/task: main branch, retry-framework plan; retry bounds for all 5 workers running through library hooks (D1).\nELI10: Right now the plan describes the curve between retries but never says how many retries there are or what happens to a job that keeps failing. Without a limit, a poisoned job retries forever and eats worker capacity. With a limit but no landing spot, the job disappears with one log line nobody reads. A dead-letter store keeps the failed job, its payload reference and its last error so someone can inspect and replay it.\nStakes if we pick wrong: either an infinite-retry job starves the queue, or real work silently vanishes after the last attempt and the first sign is a customer asking where their data went.\nRecommendation: A because a dead-letter store plus an alert is a few dozen lines with library hooks, and it turns \"job vanished\" into \"job parked, here is why.\"\nCompleteness: A=10/10, B=5/10, C=3/10\nPros / cons:\nA) Bounded attempts + dead-letter store + alert (recommended)\n \u2705 Exhausted or fatal jobs are kept with last error and attempt history; operators can inspect and replay (human: ~1 day / CC: ~20 min)\n \u2705 Metric and alert on dead-letter growth turns a silent failure into a page at the right time\n \u274c One more table or queue to own, plus a small replay path to build and test\nB) Bounded attempts, log and drop\n \u2705 Simplest bound: `maxAttempts` default 5 per worker, one error log on exhaustion (human: ~2 hr / CC: ~5 min)\n \u2705 No new storage; nothing to operate\n \u274c Exhausted jobs are gone; recovery means replaying from upstream sources by hand, if that is even possible\nC) Leave to library defaults\n \u2705 Zero plan text and zero decision now\n \u2705 Whatever the library does is at least consistent across the 5 workers\n \u274c Nobody knows the limit or the terminal behavior until an incident teaches them; 3am failure mode\nNet: You are trading one small dead-letter store for never having to ask \"where did that job go.\"": "Bounded + dead-letter + alert (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T19:55:16.802Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
|
||||
"toolUseId": "toolu_01GMRg1vxVPnVCefhfuCSKEw",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D4 \u2014 Should retry delays be randomized (jitter)?\nProject/branch/task: main branch, retry-framework plan; the backoff curve each worker supplies to the library hooks (D1).\nELI10: When many jobs fail at the same moment because a shared dependency went down, a pure exponential curve makes them all retry at the same moments too, so the recovering dependency gets hit by a wave on every step. Jitter randomizes each job's delay so the retries spread out. It is one line inside the curve callback each worker already supplies.\nStakes if we pick wrong: synchronized retry waves knock a recovering dependency back over (the classic thundering herd); or, with jitter, per-job retry timing becomes slightly less predictable and tests need a seeded random source.\nRecommendation: A because full jitter gives the least contention in AWS's published analysis and costs one line in a callback you are already writing.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nPros / cons:\nA) Full jitter: random(0, exponentialDelay) (recommended)\n \u2705 Best spread of retries and lowest total contention after a shared outage (AWS Builders' Library)\n \u2705 One line inside the D1 curve callback; RNG injected so tests stay deterministic (human: ~1 hr / CC: ~5 min)\n \u274c An individual retry can fire almost immediately; minimum wait is not guaranteed\nB) Equal jitter: half fixed, half random\n \u2705 Guarantees a minimum wait of half the exponential delay while still spreading retries\n \u2705 Same one-line cost and same injectable RNG as full jitter (human: ~1 hr / CC: ~5 min)\n \u274c Slightly more contention than full jitter in the same analysis, and one more parameter to explain\nC) No jitter: deterministic curve\n \u2705 Fully deterministic; trivial to reason about and to assert exact delays in tests\n \u2705 Zero extra code beyond the exponential curve\n \u274c Every job that failed together retries together; retry storms on recovery are the expected outcome\nNet: One random() call now versus a synchronized retry wave the first time a dependency has a bad hour.",
|
||||
"header": "Jitter",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Full jitter (recommended)",
|
||||
"description": "delay = random(0, exponentialDelay) inside each worker's curve callback. Best spread, lowest contention. RNG injectable so tests are deterministic. Human ~1 hr / CC ~5 min."
|
||||
},
|
||||
{
|
||||
"label": "Equal jitter",
|
||||
"description": "delay = exponentialDelay/2 + random(0, exponentialDelay/2). Guarantees a minimum wait; slightly more contention than full jitter. RNG injectable. Human ~1 hr / CC ~5 min."
|
||||
},
|
||||
{
|
||||
"label": "No jitter",
|
||||
"description": "Deterministic exponential curve, no randomization. Simplest to test; retries synchronize after a shared outage."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D4 \u2014 Should retry delays be randomized (jitter)?\nProject/branch/task: main branch, retry-framework plan; the backoff curve each worker supplies to the library hooks (D1).\nELI10: When many jobs fail at the same moment because a shared dependency went down, a pure exponential curve makes them all retry at the same moments too, so the recovering dependency gets hit by a wave on every step. Jitter randomizes each job's delay so the retries spread out. It is one line inside the curve callback each worker already supplies.\nStakes if we pick wrong: synchronized retry waves knock a recovering dependency back over (the classic thundering herd); or, with jitter, per-job retry timing becomes slightly less predictable and tests need a seeded random source.\nRecommendation: A because full jitter gives the least contention in AWS's published analysis and costs one line in a callback you are already writing.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nPros / cons:\nA) Full jitter: random(0, exponentialDelay) (recommended)\n \u2705 Best spread of retries and lowest total contention after a shared outage (AWS Builders' Library)\n \u2705 One line inside the D1 curve callback; RNG injected so tests stay deterministic (human: ~1 hr / CC: ~5 min)\n \u274c An individual retry can fire almost immediately; minimum wait is not guaranteed\nB) Equal jitter: half fixed, half random\n \u2705 Guarantees a minimum wait of half the exponential delay while still spreading retries\n \u2705 Same one-line cost and same injectable RNG as full jitter (human: ~1 hr / CC: ~5 min)\n \u274c Slightly more contention than full jitter in the same analysis, and one more parameter to explain\nC) No jitter: deterministic curve\n \u2705 Fully deterministic; trivial to reason about and to assert exact delays in tests\n \u2705 Zero extra code beyond the exponential curve\n \u274c Every job that failed together retries together; retry storms on recovery are the expected outcome\nNet: One random() call now versus a synchronized retry wave the first time a dependency has a bad hour.": "Full jitter (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T19:56:30.842Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
|
||||
"toolUseId": "toolu_01XLVRABVWXpSZRBNni2esSC",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D5 \u2014 Should the backoff delay have a ceiling?\nProject/branch/task: main branch, retry-framework plan; the curve parameters each worker passes to the library hooks (D1, jittered per D4).\nELI10: Exponential backoff doubles the wait after each failure. That is fine for 5 attempts, but D3 lets each worker raise its attempt count, and a worker set to 15 attempts from a 1 second base would wait about 4.5 hours before its last try; at 20 attempts it would wait 6 days. A cap says \"never wait longer than X between attempts,\" so the curve grows and then flattens. It is one min() call in the callback.\nStakes if we pick wrong: without a cap, a worker with a higher attempt count silently turns into a multi-day wait that looks like a stuck job; with a cap, one more number to document per worker.\nRecommendation: A because the cap is one min() and it makes \"how long can this job be delayed\" a question with an answer.\nCompleteness: A=9/10, B=4/10\nPros / cons:\nA) Cap each delay: default 10 min, per-worker override (recommended)\n \u2705 Worst-case wait between attempts is bounded and documented for every worker (human: ~1 hr / CC: ~5 min)\n \u2705 Also pins the curve defaults (base 1 s, multiplier 2) so all 5 workers start from the same documented numbers\n \u274c One more config value per worker to document and keep sane alongside maxAttempts\nB) No cap\n \u2705 Zero code; the curve is exactly the exponential the plan describes\n \u2705 Fewer knobs to explain\n \u274c Any worker that raises maxAttempts past ~12 gets hour-to-day waits nobody intended\nNet: One min() now versus a job that looks stuck for six days the first time someone bumps an attempt count.",
|
||||
"header": "Delay cap",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Cap each delay (recommended)",
|
||||
"description": "delay = min(jitteredExponential, maxDelay). `maxDelay` default 10 minutes with per-worker override. Curve defaults documented: base 1 s, multiplier 2, per-worker override. Human ~1 hr / CC ~5 min. Completeness 9/10."
|
||||
},
|
||||
{
|
||||
"label": "No cap",
|
||||
"description": "Raw exponential curve with no ceiling. Zero code, fewer knobs; high attempt counts produce hour-to-day waits. Completeness 4/10."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D5 \u2014 Should the backoff delay have a ceiling?\nProject/branch/task: main branch, retry-framework plan; the curve parameters each worker passes to the library hooks (D1, jittered per D4).\nELI10: Exponential backoff doubles the wait after each failure. That is fine for 5 attempts, but D3 lets each worker raise its attempt count, and a worker set to 15 attempts from a 1 second base would wait about 4.5 hours before its last try; at 20 attempts it would wait 6 days. A cap says \"never wait longer than X between attempts,\" so the curve grows and then flattens. It is one min() call in the callback.\nStakes if we pick wrong: without a cap, a worker with a higher attempt count silently turns into a multi-day wait that looks like a stuck job; with a cap, one more number to document per worker.\nRecommendation: A because the cap is one min() and it makes \"how long can this job be delayed\" a question with an answer.\nCompleteness: A=9/10, B=4/10\nPros / cons:\nA) Cap each delay: default 10 min, per-worker override (recommended)\n \u2705 Worst-case wait between attempts is bounded and documented for every worker (human: ~1 hr / CC: ~5 min)\n \u2705 Also pins the curve defaults (base 1 s, multiplier 2) so all 5 workers start from the same documented numbers\n \u274c One more config value per worker to document and keep sane alongside maxAttempts\nB) No cap\n \u2705 Zero code; the curve is exactly the exponential the plan describes\n \u2705 Fewer knobs to explain\n \u274c Any worker that raises maxAttempts past ~12 gets hour-to-day waits nobody intended\nNet: One min() now versus a job that looks stuck for six days the first time someone bumps an attempt count.": "Cap each delay (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T19:57:33.114Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
|
||||
"toolUseId": "toolu_01SrWycxMxofjLp1hxkciPj9",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D6 \u2014 Which errors should the four non-webhook workers retry, and which go straight to dead-letter?\nProject/branch/task: main branch, retry-framework plan; error handling inside the 4 non-webhook workers' library retry hooks (D1). The webhook worker's rule is already fixed by D2.\nELI10: Not every failure is worth retrying. A timeout or a \"service busy\" reply will likely pass on the next try. A validation error, a missing record or a bug that throws will fail the same way five times in a row, burning worker time and delaying the dead-letter record (D3) by the whole backoff curve. Classifying errors sends the hopeless ones to dead-letter immediately and spends retries only on the ones that can recover. The open question is what to do with an error nobody has classified yet.\nStakes if we pick wrong: either a bug retries five times per job across a whole queue before anyone sees it, or a transient error that nobody thought to list dead-letters real work on its first failure.\nRecommendation: A because classification is a short list per worker, and defaulting unknown errors to retryable never loses work: the worst case is five wasted attempts, not a dropped job.\nCompleteness: A=10/10, B=5/10, C=8/10\nPros / cons:\nA) Classify; unknown errors retry (recommended)\n \u2705 Hopeless errors (validation, 4xx, missing record, TypeError) land in dead-letter on attempt 1 with the real cause visible (human: ~half day / CC: ~15 min)\n \u2705 Unlisted errors still retry, so a forgotten transient class costs attempts, never data\n \u274c Each worker maintains a small error-class list, and a new fatal class retries needlessly until someone adds it\nB) Retry everything until maxAttempts\n \u2705 No lists to maintain; identical behavior in all 4 workers (human: ~0 / CC: ~0)\n \u2705 Impossible to misclassify a transient error as fatal\n \u274c A deploy with a bug retries every affected job 5 times over the full curve before dead-lettering; queue capacity burns and diagnosis is delayed\nC) Classify; unknown errors are fatal\n \u2705 Zero wasted attempts on anything not explicitly known to be transient (human: ~half day / CC: ~15 min)\n \u2705 Dead-letter fills fast, so new error classes surface quickly\n \u274c Any transient error missing from the list dead-letters real work on its first failure, which is the exact loss the retry framework exists to prevent\nNet: A short list per worker plus a safe default, versus either wasted retries on bugs or lost work on unlisted transients.",
|
||||
"header": "Error classes",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Classify; unknown \u2192 retryable (recommended)",
|
||||
"description": "Each of the 4 workers declares retryable classes (timeout, connection error, 429/503, deadlock/serialization failure) and fatal classes (validation error, 4xx other than 429, missing record, programming error). Fatal \u2192 dead-letter immediately without consuming attempts. Unclassified errors retry. Human ~half day / CC ~15 min. Completeness 10/10."
|
||||
},
|
||||
{
|
||||
"label": "Retry everything to maxAttempts",
|
||||
"description": "No classification. Every error consumes an attempt until `maxAttempts`, then dead-letter per D3. Zero code; bugs retry 5 times per job. Completeness 5/10."
|
||||
},
|
||||
{
|
||||
"label": "Classify; unknown \u2192 fatal",
|
||||
"description": "Same declared classes as A, but unclassified errors go to dead-letter immediately. No wasted attempts; unlisted transient errors lose work on first failure. Human ~half day / CC ~15 min. Completeness 8/10."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D6 \u2014 Which errors should the four non-webhook workers retry, and which go straight to dead-letter?\nProject/branch/task: main branch, retry-framework plan; error handling inside the 4 non-webhook workers' library retry hooks (D1). The webhook worker's rule is already fixed by D2.\nELI10: Not every failure is worth retrying. A timeout or a \"service busy\" reply will likely pass on the next try. A validation error, a missing record or a bug that throws will fail the same way five times in a row, burning worker time and delaying the dead-letter record (D3) by the whole backoff curve. Classifying errors sends the hopeless ones to dead-letter immediately and spends retries only on the ones that can recover. The open question is what to do with an error nobody has classified yet.\nStakes if we pick wrong: either a bug retries five times per job across a whole queue before anyone sees it, or a transient error that nobody thought to list dead-letters real work on its first failure.\nRecommendation: A because classification is a short list per worker, and defaulting unknown errors to retryable never loses work: the worst case is five wasted attempts, not a dropped job.\nCompleteness: A=10/10, B=5/10, C=8/10\nPros / cons:\nA) Classify; unknown errors retry (recommended)\n \u2705 Hopeless errors (validation, 4xx, missing record, TypeError) land in dead-letter on attempt 1 with the real cause visible (human: ~half day / CC: ~15 min)\n \u2705 Unlisted errors still retry, so a forgotten transient class costs attempts, never data\n \u274c Each worker maintains a small error-class list, and a new fatal class retries needlessly until someone adds it\nB) Retry everything until maxAttempts\n \u2705 No lists to maintain; identical behavior in all 4 workers (human: ~0 / CC: ~0)\n \u2705 Impossible to misclassify a transient error as fatal\n \u274c A deploy with a bug retries every affected job 5 times over the full curve before dead-lettering; queue capacity burns and diagnosis is delayed\nC) Classify; unknown errors are fatal\n \u2705 Zero wasted attempts on anything not explicitly known to be transient (human: ~half day / CC: ~15 min)\n \u2705 Dead-letter fills fast, so new error classes surface quickly\n \u274c Any transient error missing from the list dead-letters real work on its first failure, which is the exact loss the retry framework exists to prevent\nNet: A short list per worker plus a safe default, versus either wasted retries on bugs or lost work on unlisted transients.": "Classify; unknown \u2192 retryable (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T19:58:40.484Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
|
||||
"toolUseId": "toolu_013bXU7e1WUvhP4r6Nh2cigC",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D7 \u2014 One shared retry-policy module, or five copies and \"refactor later\"?\nProject/branch/task: main branch, retry-framework plan; how the 5 workers carry the behavior approved in D3\u2013D6.\nELI10: After D1 the library does the scheduling, but every worker still has to hand it the same four things: a jittered, capped curve, an error classifier, a dead-letter handoff and an attempt log line. The plan copies that block into five files and promises to clean up later. \"Later\" for copy-pasted retry code usually means the fifth copy drifts (no cap, wrong jitter) and nobody notices until an incident. The alternative is one small module that each worker configures with its own numbers and error lists.\nStakes if we pick wrong: five curves that silently disagree, five places to fix the next retry bug, and five test suites that each cover a slightly different subset; or, with a shared module, one bug that hits all five workers at once (mitigated by the module's own tests).\nRecommendation: A because the behavior is identical by construction (D3\u2013D6 fixed it), the module is under 100 lines, and it removes more lines than it adds while making the retry rules testable once.\nCompleteness: A=10/10, B=4/10, C=7/10\nPros / cons:\nA) One shared retry-policy module (recommended)\n \u2705 Curve, classifier, dead-letter handoff, attempt log and config validation are tested once and behave the same in all 5 workers (human: ~1 day / CC: ~20 min)\n \u2705 Estimated 15\u201390 implementation lines saved; the helper's test suite replaces five near-duplicate suites\n \u274c A bug in the module reaches all 5 workers; the module's own tests are the guard\nB) Five inline copies, refactor later (as planned)\n \u2705 No shared dependency between workers; each can be changed in isolation (human: ~1.5 days / CC: ~30 min)\n \u2705 Matches the plan text exactly; nothing new to name or place\n \u274c Five copies to keep in sync and five test suites to write; \"later\" rarely arrives for retry glue\nC) Extract the curve builder only\n \u2705 The math most likely to drift (jitter + cap) lives in one place (human: ~1 day / CC: ~15 min)\n \u2705 Smaller shared surface than A\n \u274c Classifier, dead-letter handoff and log line are still copied five times, so most of the duplication and its tests remain\nNet: One under-100-line module now, or five copies plus a promise.",
|
||||
"header": "Shared module",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "One shared retry-policy module (recommended)",
|
||||
"description": "Small module: buildBackoff (base/multiplier/maxDelay/rng), classify (per-worker retryable/fatal lists), toDeadLetter (persist + metric), logAttempt, and startup config validation. All 5 workers use it with their own values. Human ~1 day / CC ~20 min. Completeness 10/10."
|
||||
},
|
||||
{
|
||||
"label": "Five inline copies (as planned)",
|
||||
"description": "Each worker carries its own curve, classifier, dead-letter handoff and log line; refactor deferred. Human ~1.5 days / CC ~30 min. Completeness 4/10."
|
||||
},
|
||||
{
|
||||
"label": "Extract curve builder only",
|
||||
"description": "Shared buildBackoff (jitter + cap) only; classifier, dead-letter handoff and log line stay inline in each of the 5 workers. Human ~1 day / CC ~15 min. Completeness 7/10."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D7 \u2014 One shared retry-policy module, or five copies and \"refactor later\"?\nProject/branch/task: main branch, retry-framework plan; how the 5 workers carry the behavior approved in D3\u2013D6.\nELI10: After D1 the library does the scheduling, but every worker still has to hand it the same four things: a jittered, capped curve, an error classifier, a dead-letter handoff and an attempt log line. The plan copies that block into five files and promises to clean up later. \"Later\" for copy-pasted retry code usually means the fifth copy drifts (no cap, wrong jitter) and nobody notices until an incident. The alternative is one small module that each worker configures with its own numbers and error lists.\nStakes if we pick wrong: five curves that silently disagree, five places to fix the next retry bug, and five test suites that each cover a slightly different subset; or, with a shared module, one bug that hits all five workers at once (mitigated by the module's own tests).\nRecommendation: A because the behavior is identical by construction (D3\u2013D6 fixed it), the module is under 100 lines, and it removes more lines than it adds while making the retry rules testable once.\nCompleteness: A=10/10, B=4/10, C=7/10\nPros / cons:\nA) One shared retry-policy module (recommended)\n \u2705 Curve, classifier, dead-letter handoff, attempt log and config validation are tested once and behave the same in all 5 workers (human: ~1 day / CC: ~20 min)\n \u2705 Estimated 15\u201390 implementation lines saved; the helper's test suite replaces five near-duplicate suites\n \u274c A bug in the module reaches all 5 workers; the module's own tests are the guard\nB) Five inline copies, refactor later (as planned)\n \u2705 No shared dependency between workers; each can be changed in isolation (human: ~1.5 days / CC: ~30 min)\n \u2705 Matches the plan text exactly; nothing new to name or place\n \u274c Five copies to keep in sync and five test suites to write; \"later\" rarely arrives for retry glue\nC) Extract the curve builder only\n \u2705 The math most likely to drift (jitter + cap) lives in one place (human: ~1 day / CC: ~15 min)\n \u2705 Smaller shared surface than A\n \u274c Classifier, dead-letter handoff and log line are still copied five times, so most of the duplication and its tests remain\nNet: One under-100-line module now, or five copies plus a promise.": "One shared retry-policy module (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T20:00:17.086Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
|
||||
"toolUseId": "toolu_01Ly1JT6sRQHG4ZVpQ7SUETD",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D8 \u2014 How do we prove processWebhookJob() still sends each event at most once?\nProject/branch/task: main branch, retry-framework plan; regression coverage for the rewritten webhook worker (D2 fixed the behavior to keep).\nELI10: The webhook worker is being rewritten and it carries a promise to receivers: an event is never sent twice. D2 kept that promise while adding retries for failures that happen before anything is sent. A rewrite with no test for the promise means the first duplicate email or double charge is found by a customer. The test is straightforward: a fake receiver counts sends, and we assert the count is exactly one across every failure pattern. The question is how deep to go: assertions against the worker alone, a run through the real library hooks, or both.\nStakes if we pick wrong: a retry path nobody tested sends duplicates to non-idempotent receivers, or a hook wiring mistake means pre-send failures never actually retry and the framework quietly does nothing for webhooks.\nRecommendation: A because the unit layer pins each failure class cheaply and the integration layer is the only thing that catches hook wiring and attempt persistence, which is where retry bugs actually live.\nCompleteness: A=10/10, B=7/10, C=7/10\nPros / cons:\nA) Unit + integration through the library hooks (recommended)\n \u2705 Every failure class (pre-send, timeout, 5xx, reset, success) asserted in isolation with a recording fake transport (human: ~1 day / CC: ~20 min)\n \u2705 One end-to-end run through the real hooks with a fake receiver catches wiring and attempt-persistence bugs the unit layer cannot see\n \u274c Two test layers to maintain; the integration test needs the library's test harness or an in-process queue\nB) Unit tests only\n \u2705 Fast, deterministic, no queue infrastructure in the test run (human: ~half day / CC: ~10 min)\n \u2705 Pins the classifier and the send-count contract per failure class\n \u274c Never exercises the real hook registration, so a miswired hook passes tests and never retries in production\nC) Integration test only\n \u2705 Exercises the real path receivers depend on (human: ~half day / CC: ~10 min)\n \u2705 Fewer tests to write\n \u274c Slower, and a failure tells you \"something duplicated\" without pointing at which failure class; edge classes get skipped for time\nNet: Cheap isolated assertions plus one real-path run, versus trusting either layer alone to protect a promise made to external receivers.",
|
||||
"header": "Webhook regression",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Unit + integration (recommended)",
|
||||
"description": "Unit: fake transport records every send; assert exactly 1 send after pre-send retries, 0 further sends after timeout/5xx/reset with a dead-letter entry, 1 send on success. Integration: real library hooks + fake receiver, same assertions, plus attempt count survives a simulated worker restart. Human ~1 day / CC ~20 min. Completeness 10/10."
|
||||
},
|
||||
{
|
||||
"label": "Unit tests only",
|
||||
"description": "The unit assertions from A against the worker with a fake transport; no run through the real library hooks. Human ~half day / CC ~10 min. Completeness 7/10."
|
||||
},
|
||||
{
|
||||
"label": "Integration test only",
|
||||
"description": "The integration run from A only; no isolated per-failure-class assertions. Human ~half day / CC ~10 min. Completeness 7/10."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D8 \u2014 How do we prove processWebhookJob() still sends each event at most once?\nProject/branch/task: main branch, retry-framework plan; regression coverage for the rewritten webhook worker (D2 fixed the behavior to keep).\nELI10: The webhook worker is being rewritten and it carries a promise to receivers: an event is never sent twice. D2 kept that promise while adding retries for failures that happen before anything is sent. A rewrite with no test for the promise means the first duplicate email or double charge is found by a customer. The test is straightforward: a fake receiver counts sends, and we assert the count is exactly one across every failure pattern. The question is how deep to go: assertions against the worker alone, a run through the real library hooks, or both.\nStakes if we pick wrong: a retry path nobody tested sends duplicates to non-idempotent receivers, or a hook wiring mistake means pre-send failures never actually retry and the framework quietly does nothing for webhooks.\nRecommendation: A because the unit layer pins each failure class cheaply and the integration layer is the only thing that catches hook wiring and attempt persistence, which is where retry bugs actually live.\nCompleteness: A=10/10, B=7/10, C=7/10\nPros / cons:\nA) Unit + integration through the library hooks (recommended)\n \u2705 Every failure class (pre-send, timeout, 5xx, reset, success) asserted in isolation with a recording fake transport (human: ~1 day / CC: ~20 min)\n \u2705 One end-to-end run through the real hooks with a fake receiver catches wiring and attempt-persistence bugs the unit layer cannot see\n \u274c Two test layers to maintain; the integration test needs the library's test harness or an in-process queue\nB) Unit tests only\n \u2705 Fast, deterministic, no queue infrastructure in the test run (human: ~half day / CC: ~10 min)\n \u2705 Pins the classifier and the send-count contract per failure class\n \u274c Never exercises the real hook registration, so a miswired hook passes tests and never retries in production\nC) Integration test only\n \u2705 Exercises the real path receivers depend on (human: ~half day / CC: ~10 min)\n \u2705 Fewer tests to write\n \u274c Slower, and a failure tells you \"something duplicated\" without pointing at which failure class; edge classes get skipped for time\nNet: Cheap isolated assertions plus one real-path run, versus trusting either layer alone to protect a promise made to external receivers.": "Unit + integration (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T20:02:01.530Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
|
||||
"toolUseId": "toolu_01WZ2b5HaUXKgTwZxqfKn3pd",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D9 \u2014 Cache the dependency graph across retries now, or measure first?\nProject/branch/task: main branch, retry-framework plan; per-attempt cost inside the 5 workers running through library hooks (D1).\nELI10: The plan worries that each retry reloads the job and rebuilds its dependency graph from scratch. After D1 the reload is just the library handing the job to the worker, which happens anyway. The rebuild is real extra CPU, but only on retries, and D3 caps those at 5 per failing job. Storing the graph on the first attempt would add a write to every job, including the large majority that succeed first time, to save work on the few that fail. Nobody has measured how long the rebuild takes.\nStakes if we pick wrong: either we add a write and a staleness risk to every job to fix a cost nobody measured, or a genuinely slow rebuild keeps burning worker time on retries and we only find out under load.\nRecommendation: C because an unmeasured optimization that taxes the happy path is the wrong trade; two timing metrics make the real decision cheap and data-driven.\nNote: options differ in kind (persisted cache vs in-process memo vs measure first) \u2014 no completeness score.\nPros / cons:\nA) Persist the graph with the job on attempt 1\n \u2705 Retries never recompute; cost is paid once per job regardless of which worker instance retries (human: ~1 day / CC: ~20 min)\n \u2705 Simple to reason about once the invalidation rule (payload version) is in place\n \u274c Adds a write and stored blob to every job, including the ones that never retry; stale-graph bugs if the payload changes between attempts\nB) In-process memo (bounded LRU)\n \u2705 No persistence, no schema change; a few lines around the graph builder (human: ~2 hr / CC: ~10 min)\n \u2705 Zero cost on the happy path beyond a map insert\n \u274c Retries after a 10-minute delay usually land on a different worker instance, so the hit rate is low and unpredictable\nC) Measure first: timing metrics + p95 budget (recommended)\n \u2705 Two metrics (graph compute ms, payload bytes) per attempt tell you whether this is 2 ms or 2 s before anyone writes cache code (human: ~1 hr / CC: ~5 min)\n \u2705 No happy-path cost, no staleness risk, and the retry-policy module already logs per attempt (D7) so the hook point exists\n \u274c If the rebuild is genuinely slow, retries stay expensive until the follow-up lands\nNet: Add a write to every job to save CPU on the few that retry, or spend an hour on metrics and decide with numbers.",
|
||||
"header": "Graph cache",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Persist the graph with the job",
|
||||
"description": "Compute once on attempt 1, store the graph beside the job row, reuse on retries, invalidate when the payload version changes. Adds a write to every job. Human ~1 day / CC ~20 min."
|
||||
},
|
||||
{
|
||||
"label": "In-process memo (bounded LRU)",
|
||||
"description": "Memoize the graph per worker instance keyed by job id + payload hash, bounded LRU. No persistence; low hit rate when retries land on another instance. Human ~2 hr / CC ~10 min."
|
||||
},
|
||||
{
|
||||
"label": "Measure first (recommended)",
|
||||
"description": "No cache. Add per-attempt timing metrics (job load ms, graph compute ms, payload bytes) via the retry-policy module's attempt log, set a p95 budget, and revisit caching with data. Human ~1 hr / CC ~5 min."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D9 \u2014 Cache the dependency graph across retries now, or measure first?\nProject/branch/task: main branch, retry-framework plan; per-attempt cost inside the 5 workers running through library hooks (D1).\nELI10: The plan worries that each retry reloads the job and rebuilds its dependency graph from scratch. After D1 the reload is just the library handing the job to the worker, which happens anyway. The rebuild is real extra CPU, but only on retries, and D3 caps those at 5 per failing job. Storing the graph on the first attempt would add a write to every job, including the large majority that succeed first time, to save work on the few that fail. Nobody has measured how long the rebuild takes.\nStakes if we pick wrong: either we add a write and a staleness risk to every job to fix a cost nobody measured, or a genuinely slow rebuild keeps burning worker time on retries and we only find out under load.\nRecommendation: C because an unmeasured optimization that taxes the happy path is the wrong trade; two timing metrics make the real decision cheap and data-driven.\nNote: options differ in kind (persisted cache vs in-process memo vs measure first) \u2014 no completeness score.\nPros / cons:\nA) Persist the graph with the job on attempt 1\n \u2705 Retries never recompute; cost is paid once per job regardless of which worker instance retries (human: ~1 day / CC: ~20 min)\n \u2705 Simple to reason about once the invalidation rule (payload version) is in place\n \u274c Adds a write and stored blob to every job, including the ones that never retry; stale-graph bugs if the payload changes between attempts\nB) In-process memo (bounded LRU)\n \u2705 No persistence, no schema change; a few lines around the graph builder (human: ~2 hr / CC: ~10 min)\n \u2705 Zero cost on the happy path beyond a map insert\n \u274c Retries after a 10-minute delay usually land on a different worker instance, so the hit rate is low and unpredictable\nC) Measure first: timing metrics + p95 budget (recommended)\n \u2705 Two metrics (graph compute ms, payload bytes) per attempt tell you whether this is 2 ms or 2 s before anyone writes cache code (human: ~1 hr / CC: ~5 min)\n \u2705 No happy-path cost, no staleness risk, and the retry-policy module already logs per attempt (D7) so the hook point exists\n \u274c If the rebuild is genuinely slow, retries stay expensive until the follow-up lands\nNet: Add a write to every job to save CPU on the few that retry, or spend an hour on metrics and decide with numbers.": "Persist the graph with the job"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T20:04:03.405Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
|
||||
"toolUseId": "toolu_01WjGqEAsM8pPCNSzxpWyPSV",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D10 \u2014 Track dead-letter retention as a TODO, skip it, or build it now?\nProject/branch/task: main branch, retry-framework plan; follow-up to the dead-letter store approved in D3.\nELI10: The dead-letter store keeps every job that ran out of retries or hit a fatal error. Nothing ever removes them. That is fine for months, then the table is large, the growth alert query slows, and nobody remembers why. A purge job with a retention period fixes it, but the retention period is a judgment call about how long failed-job evidence must stay around.\nStakes if we pick wrong: build it now with the wrong retention and you delete evidence of lost work; skip it and the store becomes an unbounded table someone discovers during an incident.\nRecommendation: A because the store is new, growth is slow, and the retention period deserves an owner's answer rather than a default picked inside a retry PR; the TODO carries a concrete trigger.\nCompleteness: A=6/10, B=2/10, C=10/10\nPros / cons:\nA) Add to TODOS.md with a trigger (recommended)\n \u2705 Keeps this PR right-sized: the retry framework ships without a retention debate attached\n \u2705 Trigger (10k rows or 3 months) means the TODO fires before growth matters (human: ~5 min / CC: ~1 min now)\n \u274c Unbounded growth until someone acts on the TODO; the ceiling is a slow query, not data loss\nB) Skip\n \u2705 Nothing to track or build\n \u2705 Zero effort now\n \u274c The store grows forever with no record that anyone considered it\nC) Build now in this PR\n \u2705 Store ships bounded from day one: purge job, 90-day default, keep flag, tests (human: ~2 hr / CC: ~10 min)\n \u2705 No follow-up to forget\n \u274c Expands this PR with a scheduled job and a retention default nobody has agreed to; deletes evidence if the default is wrong\nNet: A tracked follow-up with a trigger, versus a bigger PR that guesses how long failed-job evidence should live.",
|
||||
"header": "DLQ retention",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Add to TODOS.md (recommended)",
|
||||
"description": "Record the TODO (what/why/pros/cons/context/depends-on) with trigger: build when the dead-letter store passes 10k rows or at 3 months, whichever first. Human ~5 min / CC ~1 min. Completeness 6/10."
|
||||
},
|
||||
{
|
||||
"label": "Skip \u2014 not valuable enough",
|
||||
"description": "Do not track retention. Completeness 2/10."
|
||||
},
|
||||
{
|
||||
"label": "Build it now in this PR",
|
||||
"description": "Scheduled purge job, retention config default 90 days, keep flag, tests, shipped with the dead-letter store. Human ~2 hr / CC ~10 min. Completeness 10/10."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D10 \u2014 Track dead-letter retention as a TODO, skip it, or build it now?\nProject/branch/task: main branch, retry-framework plan; follow-up to the dead-letter store approved in D3.\nELI10: The dead-letter store keeps every job that ran out of retries or hit a fatal error. Nothing ever removes them. That is fine for months, then the table is large, the growth alert query slows, and nobody remembers why. A purge job with a retention period fixes it, but the retention period is a judgment call about how long failed-job evidence must stay around.\nStakes if we pick wrong: build it now with the wrong retention and you delete evidence of lost work; skip it and the store becomes an unbounded table someone discovers during an incident.\nRecommendation: A because the store is new, growth is slow, and the retention period deserves an owner's answer rather than a default picked inside a retry PR; the TODO carries a concrete trigger.\nCompleteness: A=6/10, B=2/10, C=10/10\nPros / cons:\nA) Add to TODOS.md with a trigger (recommended)\n \u2705 Keeps this PR right-sized: the retry framework ships without a retention debate attached\n \u2705 Trigger (10k rows or 3 months) means the TODO fires before growth matters (human: ~5 min / CC: ~1 min now)\n \u274c Unbounded growth until someone acts on the TODO; the ceiling is a slow query, not data loss\nB) Skip\n \u2705 Nothing to track or build\n \u2705 Zero effort now\n \u274c The store grows forever with no record that anyone considered it\nC) Build now in this PR\n \u2705 Store ships bounded from day one: purge job, 90-day default, keep flag, tests (human: ~2 hr / CC: ~10 min)\n \u2705 No follow-up to forget\n \u274c Expands this PR with a scheduled job and a retention default nobody has agreed to; deletes evidence if the default is wrong\nNet: A tracked follow-up with a trigger, versus a bigger PR that guesses how long failed-job evidence should live.": "Add to TODOS.md (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T20:05:54.778Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
|
||||
"toolUseId": "toolu_01QrMt8ZhcL4wV3cKEjo3MsA",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D11 \u2014 Track the webhook event-id header and at-least-once opt-in as a TODO, skip it, or ship the header now?\nProject/branch/task: main branch, retry-framework plan; follow-up to D2 (webhook worker stays at-most-once).\nELI10: D2 kept the promise that a webhook is never sent twice, which means a send that times out is still lost. The usual fix is to stamp every event with a stable id so receivers can ignore duplicates, and then retry freely. That is a contract change, so it was declined for this PR. The question is whether to track it, drop it, or at least ship the harmless id header now so receivers can start deduping before the semantics ever change.\nStakes if we pick wrong: lost webhook events keep landing in dead-letter with no plan to stop the loss; or a header change rides along in a retry PR without receiver communication.\nRecommendation: A because this is a receiver-facing contract change that deserves its own PR and docs, and the dead-letter store (D3) will produce the loss numbers that justify it; the trigger is concrete.\nCompleteness: A=6/10, B=2/10, C=8/10\nPros / cons:\nA) Add to TODOS.md with a trigger (recommended)\n \u2705 Keeps the retry PR free of webhook contract changes; the TODO fires on measured loss (human: ~5 min / CC: ~1 min now)\n \u2705 Dead-letter counts of post-send failures give the case for it with real numbers\n \u274c Webhook events lost to timeouts stay lost until the TODO is acted on\nB) Skip\n \u2705 Nothing to track\n \u2705 Zero effort now\n \u274c No record that at-most-once was a deliberate trade with a known cost\nC) Ship the stable event-id header now, at-least-once later\n \u2705 Receivers can start deduping today; the header is harmless under at-most-once (human: ~2 hr / CC: ~10 min)\n \u2705 Makes the eventual semantics change a config flip instead of a payload change\n \u274c Adds a webhook payload change and receiver docs to a retry PR; still needs the TODO for the semantics\nNet: Track it with a loss-based trigger, or ship a small header change now inside a PR about retries.",
|
||||
"header": "Webhook TODO",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Add to TODOS.md (recommended)",
|
||||
"description": "Record the TODO with trigger: post-send dead-letter entries exceed 1% of webhook sends in any week, or a receiver requests redelivery. Human ~5 min / CC ~1 min. Completeness 6/10."
|
||||
},
|
||||
{
|
||||
"label": "Skip \u2014 not valuable enough",
|
||||
"description": "Do not track. Completeness 2/10."
|
||||
},
|
||||
{
|
||||
"label": "Ship the header now",
|
||||
"description": "Add a stable per-event id header to every webhook delivery in this PR; D2 semantics unchanged; at-least-once opt-in remains a TODO. Human ~2 hr / CC ~10 min. Completeness 8/10."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D11 \u2014 Track the webhook event-id header and at-least-once opt-in as a TODO, skip it, or ship the header now?\nProject/branch/task: main branch, retry-framework plan; follow-up to D2 (webhook worker stays at-most-once).\nELI10: D2 kept the promise that a webhook is never sent twice, which means a send that times out is still lost. The usual fix is to stamp every event with a stable id so receivers can ignore duplicates, and then retry freely. That is a contract change, so it was declined for this PR. The question is whether to track it, drop it, or at least ship the harmless id header now so receivers can start deduping before the semantics ever change.\nStakes if we pick wrong: lost webhook events keep landing in dead-letter with no plan to stop the loss; or a header change rides along in a retry PR without receiver communication.\nRecommendation: A because this is a receiver-facing contract change that deserves its own PR and docs, and the dead-letter store (D3) will produce the loss numbers that justify it; the trigger is concrete.\nCompleteness: A=6/10, B=2/10, C=8/10\nPros / cons:\nA) Add to TODOS.md with a trigger (recommended)\n \u2705 Keeps the retry PR free of webhook contract changes; the TODO fires on measured loss (human: ~5 min / CC: ~1 min now)\n \u2705 Dead-letter counts of post-send failures give the case for it with real numbers\n \u274c Webhook events lost to timeouts stay lost until the TODO is acted on\nB) Skip\n \u2705 Nothing to track\n \u2705 Zero effort now\n \u274c No record that at-most-once was a deliberate trade with a known cost\nC) Ship the stable event-id header now, at-least-once later\n \u2705 Receivers can start deduping today; the header is harmless under at-most-once (human: ~2 hr / CC: ~10 min)\n \u2705 Makes the eventual semantics change a config flip instead of a payload change\n \u274c Adds a webhook payload change and receiver docs to a retry PR; still needs the TODO for the semantics\nNet: Track it with a loss-based trigger, or ship a small header change now inside a PR about retries.": "Add to TODOS.md (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T20:07:04.232Z"
|
||||
}
|
||||
],
|
||||
"plan": "# Eng Review \u2014 PLAN.md: Add background job retry framework\n\n- **Review target (fixed):** `/tmp/gstack-plan-count-vrYrwf/PLAN.md` \u2014 \"Plan: Add background job retry framework\"\n- **Report file:** `/tmp/gstack-e2e-plan-eng-batching-PLhMrg/gstack-test-plan-eng-batching.md` (destination explicitly requested by the user)\n- **Skill:** `/plan-eng-review` \u00b7 session `256191-1790711293-1222b505` \u00b7 2026-09-29 \u00b7 branch `main` @ `82eaa12`\n- **Evidence available:** the repository contains only `PLAN.md` and `CLAUDE.md`. No worker files, job library, or `processWebhookJob()` source exist in this checkout. Findings below quote the plan text (file:line) and are calibrated as plan-level, not code-verified.\n\n## Original plan (unchanged copy of PLAN.md lines 4-24)\n\n```markdown\n# Plan: Add background job retry framework\n\n## Architecture\nWe'll roll a custom exponential-backoff scheduler inline in each worker\nrather than use the existing job library's built-in retry hooks. Same\nshape as the library version, but we want full control over the curve.\n\n## Code quality\nThe retry envelope (compute delay, log attempt, dispatch) is duplicated\nacross 5 worker files with copy-pasted bodies. We will leave the\nduplication for now and refactor \"later.\"\n\n## Tests\nThe existing `processWebhookJob()` flow gets rewritten as part of this\nchange. No regression test for the prior at-most-once delivery guarantee\nis planned.\n\n## Performance\nOn every retry we re-fetch the full job payload from the database, then\niterate the payload to recompute the dependency graph. Could cache the\ngraph on the first attempt; not planned.\n```\n\n## Scope Challenge\n\n### A. Assessment\n- **Already solves it:** the job library's built-in retry hooks (PLAN.md:8 admits \"same shape as the library version\"). Library source not in checkout; hook API unverified.\n- **Complexity count (estimate from plan text):** 5 worker files (PLAN.md:13) + `processWebhookJob()` (PLAN.md:17, likely one of the five) = 5\u20136 changed files; 0 new classes/services (scheduler is inline). Under thresholds \u2192 complexity gate B skipped.\n- **Search check:** [Layer 1] library retry hooks + backoff callback; jitter, delay cap, dead-letter, idempotency key are standard practice (AWS Builders' Library; Hookdeck/Svix idempotency guides).\n- **TODOS.md:** none. **Distribution:** no new artifacts.\n\n### C. Findings (plan-level; no code in checkout)\n1. `[P1] (confidence: 8/10) PLAN.md:7-9` \u2014 rebuilding a retry scheduler the job library already provides. \u2192 R1 / D1\n2. `[P1] (confidence: 7/10) PLAN.md:17-19` \u2014 retrying `processWebhookJob()` changes at-most-once to at-least-once delivery; semantics change, not just a missing test. \u2192 Section 1\n3. `[P2] (confidence: 7/10) PLAN.md:7-9` \u2014 retry policy bounds unspecified (max attempts, delay cap, jitter, dead-letter, retryable vs fatal errors). \u2192 Section 1\n4. `[P2] (confidence: 7/10) PLAN.md:12-14` \u2014 five copy-pasted retry envelopes. \u2192 Section 2\n5. `[P1] (confidence: 8/10) PLAN.md:18-19` \u2014 no regression test for a rewritten flow with a stated guarantee (Regression Rule). \u2192 Section 3\n6. `[P2] (confidence: 6/10) PLAN.md:22-24` \u2014 full payload refetch + graph recompute on every retry. \u2192 Section 4\n\nScope Challenge result: **scope accepted as-is** (D1 changed the mechanism to library retry hooks; no feature was cut, so this is not a scope reduction). Dispositions: finding 1 accepted via D1 (R1 approved); findings 2\u20136 pending in their sections.\n\n## Section 1 \u2014 Architecture review\n\nWorking plan after D1: all 5 workers retry through the job library's hooks; each worker supplies its own backoff curve.\n\n```\nRETRY STATE MACHINE (per job, owned by the library after D1)\n\n enqueue \u2500\u2500\u25b6 [attempt n] \u2500\u2500success\u2500\u2500\u25b6 DONE\n \u2502\n \u251c\u2500 fatal error (R3d: non-retryable class) \u2500\u2500\u25b6 FAILED \u2500\u2500\u25b6 dead-letter (R3a)\n \u2502\n \u2514\u2500 transient error / timeout\n \u2502\n \u251c\u2500 n >= maxAttempts (R3a) \u2500\u2500\u25b6 FAILED \u2500\u2500\u25b6 dead-letter (R3a)\n \u2502\n \u2514\u2500 delay = min(base\u00b72^n (+ jitter R3b), cap R3c) \u2500\u2500\u25b6 [attempt n+1]\n\n Webhook worker only: timeout after the request was written is AMBIGUOUS \u2014\n the receiver may already have the event. A retry here = possible duplicate (R2).\n```\n\nFindings:\n- `[P1] (confidence: 7/10) PLAN.md:17-19` \u2014 \"The existing `processWebhookJob()` flow gets rewritten ... prior at-most-once delivery guarantee.\" Adding retries flips the webhook worker from at-most-once to at-least-once: a retry after an ambiguous timeout can deliver the same event twice. The plan treats this as a missing test; it is a deliLine truncated
|
||||
}
|
||||
@@ -0,0 +1,311 @@
|
||||
{
|
||||
"source": "run 36606688266 plan-eng-multi-finding-batching observation.json (transcript.calls) and the saved report replayed from its Write/Edit inputs",
|
||||
"recordedOutcome": {
|
||||
"outcome": "completion_summary",
|
||||
"step0Count": 10,
|
||||
"reviewCount": 0
|
||||
},
|
||||
"calls": [
|
||||
{
|
||||
"sessionId": "cd04b55a-5f1a-4672-b3ed-40ea1b1636bc",
|
||||
"toolUseId": "toolu_011vfjpt7sgeSRMw1zhHATN8",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D1 — Use the job library's retry hooks or roll a custom inline scheduler?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: The job library you already run has a retry system built in. The plan wants to rebuild the same thing by hand inside each of the 5 workers, only so the delay curve can be tuned. Retry systems are easy to get subtly wrong: losing the attempt count when a worker restarts, retrying forever, or retrying twice at once. The library has already paid for those bugs; a hand-rolled copy pays for them again in production.\nStakes if we pick wrong: a hand-rolled scheduler that drops attempt state on restart or double-schedules turns one failed job into duplicate side effects or an infinite retry loop, with no dead-letter to catch it.\nRecommendation: A because the plan admits the shapes are identical, and the curve is pluggable in the library through a strategy function, so A delivers the same control with far less new code. (human: ~1 day / CC: ~20 min for A; human: ~1 week / CC: ~2 h for B plus ongoing ownership)\nCompleteness: A=9/10, B=5/10\nPros / cons:\nA) Library hooks + custom curve (recommended)\n ✅ Attempt counting, persistence across restarts and terminal handling come from tested library code, not new code\n ✅ The custom curve still lives in one strategy function, so \"full control over the curve\" is preserved\n ❌ Requires confirming the library's hook accepts a custom delay function; if it does not, we fall back to B for the curve only\nB) Custom inline scheduler\n ✅ Zero dependency on the library's retry semantics or its upgrade cadence\n ✅ Any curve shape, any bookkeeping, no hook constraints\n ❌ Rebuilds attempt state, restart persistence, concurrency guards and dead-lettering by hand, and those are the parts that fail at 3am\nNet: you are trading a one-line strategy function against owning a second retry engine forever.",
|
||||
"header": "Retry engine",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Library hooks + custom curve (recommended)",
|
||||
"description": "Register the exponential-backoff curve as one custom backoff strategy function with the job library's built-in retry hooks. Attempt counting, persistence across worker restarts, and terminal/dead-letter handling come from the library. Verify the hook accepts a delay function first; if it does not, fall back to a custom curve only for delay computation while keeping library scheduling. R2-R5 stay pending. Completeness 9/10. human: ~1 day / CC: ~20 min."
|
||||
},
|
||||
{
|
||||
"label": "Custom inline scheduler",
|
||||
"description": "Keep the plan as written: a custom exponential-backoff scheduler inline in each worker, bypassing the library's retry hooks. Attempt state, restart persistence, concurrency guards and terminal handling must be designed and tested by hand (tracked under R3). R2-R5 stay pending. Completeness 5/10. human: ~1 week / CC: ~2 h plus ongoing ownership."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D1 — Use the job library's retry hooks or roll a custom inline scheduler?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: The job library you already run has a retry system built in. The plan wants to rebuild the same thing by hand inside each of the 5 workers, only so the delay curve can be tuned. Retry systems are easy to get subtly wrong: losing the attempt count when a worker restarts, retrying forever, or retrying twice at once. The library has already paid for those bugs; a hand-rolled copy pays for them again in production.\nStakes if we pick wrong: a hand-rolled scheduler that drops attempt state on restart or double-schedules turns one failed job into duplicate side effects or an infinite retry loop, with no dead-letter to catch it.\nRecommendation: A because the plan admits the shapes are identical, and the curve is pluggable in the library through a strategy function, so A delivers the same control with far less new code. (human: ~1 day / CC: ~20 min for A; human: ~1 week / CC: ~2 h for B plus ongoing ownership)\nCompleteness: A=9/10, B=5/10\nPros / cons:\nA) Library hooks + custom curve (recommended)\n ✅ Attempt counting, persistence across restarts and terminal handling come from tested library code, not new code\n ✅ The custom curve still lives in one strategy function, so \"full control over the curve\" is preserved\n ❌ Requires confirming the library's hook accepts a custom delay function; if it does not, we fall back to B for the curve only\nB) Custom inline scheduler\n ✅ Zero dependency on the library's retry semantics or its upgrade cadence\n ✅ Any curve shape, any bookkeeping, no hook constraints\n ❌ Rebuilds attempt state, restart persistence, concurrency guards and dead-lettering by hand, and those are the parts that fail at 3am\nNet: you are trading a one-line strategy function against owning a second retry engine forever.": "Library hooks + custom curve (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T17:46:48.252Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "cd04b55a-5f1a-4672-b3ed-40ea1b1636bc",
|
||||
"toolUseId": "toolu_0182pLbYfj4VpsRE57iBzMYQ",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D2 — What delivery guarantee does `processWebhookJob()` keep once it retries?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: Today a webhook is sent once; if it fails, it is gone (at-most-once). The moment you retry, a request that timed out after the customer already received it gets sent again, so the customer sees the same event twice. You have to pick: either only retry when you are sure the request never left, or retry freely but stamp every attempt with the same id so the customer can ignore repeats. The plan does neither and just retries.\nStakes if we pick wrong: customers process duplicate events (double orders, double emails) with no way to detect them, or you ship a retry feature that almost never fires because most webhook failures are timeouts.\nRecommendation: B because it is the standard webhook contract (retry on timeout/5xx, stable event id per attempt) and is the only option where retrying actually improves delivery while giving receivers a way to dedupe. This is a receiver-visible contract change; A is the right pick if you cannot communicate it to receivers.\nCompleteness: A=7/10, B=9/10, C=3/10\nPros / cons:\nA) Keep at-most-once\n ✅ No change to what receivers see; the existing guarantee and its regression test stay valid as-is\n ✅ Smallest blast radius: no new headers, no receiver communication needed\n ❌ Retries only fire on connect/DNS/pre-send errors; timeouts and 5xx go straight to terminal, so most real failures are still not retried\nB) At-least-once + idempotency key (recommended)\n ✅ Timeouts and 5xx are retried, so delivery reliability actually improves for receivers\n ✅ Same delivery id on every attempt lets receivers dedupe; this is the contract Stripe/GitHub-style webhooks use\n ❌ Receiver-visible contract change: duplicates become possible and receivers must be told to dedupe on the id\nC) Plain retry (plan as written)\n ✅ Least code: no header, no classification, just retry on any failure\n ✅ Ships fastest\n ❌ Duplicates reach receivers with no way to tell them apart; silent double side effects\nNet: A keeps the promise but retries little; B changes the promise but makes retries worth having; C breaks the promise silently.",
|
||||
"header": "Webhook semantics",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Keep at-most-once",
|
||||
"description": "Preserve at-most-once for `processWebhookJob()`. Retry only failures where the request provably never left the process (connection refused, DNS failure, errors before send). Timeouts, 5xx and any post-send ambiguity are terminal and go to whatever terminal handling R3 decides. No new headers; receiver contract unchanged. R3, R4, R5, R7 stay pending. Completeness 7/10. human: ~half day / CC: ~10 min."
|
||||
},
|
||||
{
|
||||
"label": "At-least-once + idempotency key (recommended)",
|
||||
"description": "Retry timeouts and 5xx as well as pre-send failures. Every attempt carries the same stable delivery id (e.g. an `Idempotency-Key`/event-id header derived from the job id, generated once, persisted with the job). 4xx other than 408/429 is non-retryable for webhooks. Document to receivers that duplicates are possible and must be deduped on the id. R3, R4, R5, R7 stay pending. Completeness 9/10. human: ~1 day / CC: ~20 min."
|
||||
},
|
||||
{
|
||||
"label": "Plain retry (plan as written)",
|
||||
"description": "Retry `processWebhookJob()` on any failure with no delivery id and no classification. Duplicates reach receivers undetectably. R3, R4, R5, R7 stay pending. Completeness 3/10. human: ~1 h / CC: ~5 min."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D2 — What delivery guarantee does `processWebhookJob()` keep once it retries?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: Today a webhook is sent once; if it fails, it is gone (at-most-once). The moment you retry, a request that timed out after the customer already received it gets sent again, so the customer sees the same event twice. You have to pick: either only retry when you are sure the request never left, or retry freely but stamp every attempt with the same id so the customer can ignore repeats. The plan does neither and just retries.\nStakes if we pick wrong: customers process duplicate events (double orders, double emails) with no way to detect them, or you ship a retry feature that almost never fires because most webhook failures are timeouts.\nRecommendation: B because it is the standard webhook contract (retry on timeout/5xx, stable event id per attempt) and is the only option where retrying actually improves delivery while giving receivers a way to dedupe. This is a receiver-visible contract change; A is the right pick if you cannot communicate it to receivers.\nCompleteness: A=7/10, B=9/10, C=3/10\nPros / cons:\nA) Keep at-most-once\n ✅ No change to what receivers see; the existing guarantee and its regression test stay valid as-is\n ✅ Smallest blast radius: no new headers, no receiver communication needed\n ❌ Retries only fire on connect/DNS/pre-send errors; timeouts and 5xx go straight to terminal, so most real failures are still not retried\nB) At-least-once + idempotency key (recommended)\n ✅ Timeouts and 5xx are retried, so delivery reliability actually improves for receivers\n ✅ Same delivery id on every attempt lets receivers dedupe; this is the contract Stripe/GitHub-style webhooks use\n ❌ Receiver-visible contract change: duplicates become possible and receivers must be told to dedupe on the id\nC) Plain retry (plan as written)\n ✅ Least code: no header, no classification, just retry on any failure\n ✅ Ships fastest\n ❌ Duplicates reach receivers with no way to tell them apart; silent double side effects\nNet: A keeps the promise but retries little; B changes the promise but makes retries worth having; C breaks the promise silently.": "Keep at-most-once"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T17:48:33.604Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "cd04b55a-5f1a-4672-b3ed-40ea1b1636bc",
|
||||
"toolUseId": "toolu_012ueyTcioa6A2y9YE4gYPcR",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D3 — How many times may a job retry, and where does it go when it gives up?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: A retry curve without a stopping point is a job that runs forever when the thing it depends on is down for good. You need a maximum number of tries, and you need a place for jobs that used up their tries (a dead-letter set) so someone can look at them and replay them later. Otherwise failed work quietly disappears or quietly never stops.\nStakes if we pick wrong: either a poisoned job hammers a downstream forever and starves healthy jobs, or failed webhooks and jobs vanish with only a log line nobody reads.\nRecommendation: A because the library already provides the failed set, so the dead-letter and alert cost a config line and one log call, and it is the only option where an operator can find and replay a lost job.\nCompleteness: A=9/10, B=6/10, C=2/10\nPros / cons:\nA) Bounded + dead-letter + alert (recommended)\n ✅ Exhausted jobs are inspectable and replayable from the library's failed set, with the last error attached\n ✅ One structured log line plus a metric on dead-letter entry makes a downstream outage visible within minutes\n ❌ Needs a per-worker ceiling value and a dead-letter retention/cleanup policy to be chosen and documented\nB) Bounded + log-and-drop\n ✅ Bounds the retry loop with the least configuration\n ✅ No dead-letter retention to manage\n ❌ A dropped job is gone; the only trace is a log line, so replay after an outage is impossible\nC) Unbounded (plan as written)\n ✅ No ceiling to tune; a job eventually succeeds if the dependency ever recovers\n ✅ Zero extra code\n ❌ Permanently failing jobs retry forever, consume worker capacity and never surface as a problem\nNet: you are choosing whether a job that cannot succeed becomes a visible artifact, a log line, or a permanent background load.",
|
||||
"header": "Attempt ceiling",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Bounded + dead-letter + alert (recommended)",
|
||||
"description": "Set a maximum attempt count per worker (default 5, overridable per worker, configured in the same place as the backoff strategy). On exhaustion or on a non-retryable error, the job lands in the library's dead-letter/failed set with its last error; emit one structured error log and a metric on entry. Webhook timeouts/5xx (terminal per R2) land here too. Document the retention/replay procedure. R4, R5 stay pending. Completeness 9/10. human: ~half day / CC: ~15 min."
|
||||
},
|
||||
{
|
||||
"label": "Bounded + log-and-drop",
|
||||
"description": "Set the same per-worker maximum attempt count (default 5). On exhaustion, log the error and drop the job; no dead-letter set, no metric, no replay. R4, R5 stay pending. Completeness 6/10. human: ~1 h / CC: ~5 min."
|
||||
},
|
||||
{
|
||||
"label": "Unbounded (plan as written)",
|
||||
"description": "No attempt ceiling; the exponential curve continues until the job succeeds. No terminal handling. Conflicts with R2, which needs a terminal destination for webhook timeouts. R4, R5 stay pending. Completeness 2/10. human: 0 / CC: 0."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D3 — How many times may a job retry, and where does it go when it gives up?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: A retry curve without a stopping point is a job that runs forever when the thing it depends on is down for good. You need a maximum number of tries, and you need a place for jobs that used up their tries (a dead-letter set) so someone can look at them and replay them later. Otherwise failed work quietly disappears or quietly never stops.\nStakes if we pick wrong: either a poisoned job hammers a downstream forever and starves healthy jobs, or failed webhooks and jobs vanish with only a log line nobody reads.\nRecommendation: A because the library already provides the failed set, so the dead-letter and alert cost a config line and one log call, and it is the only option where an operator can find and replay a lost job.\nCompleteness: A=9/10, B=6/10, C=2/10\nPros / cons:\nA) Bounded + dead-letter + alert (recommended)\n ✅ Exhausted jobs are inspectable and replayable from the library's failed set, with the last error attached\n ✅ One structured log line plus a metric on dead-letter entry makes a downstream outage visible within minutes\n ❌ Needs a per-worker ceiling value and a dead-letter retention/cleanup policy to be chosen and documented\nB) Bounded + log-and-drop\n ✅ Bounds the retry loop with the least configuration\n ✅ No dead-letter retention to manage\n ❌ A dropped job is gone; the only trace is a log line, so replay after an outage is impossible\nC) Unbounded (plan as written)\n ✅ No ceiling to tune; a job eventually succeeds if the dependency ever recovers\n ✅ Zero extra code\n ❌ Permanently failing jobs retry forever, consume worker capacity and never surface as a problem\nNet: you are choosing whether a job that cannot succeed becomes a visible artifact, a log line, or a permanent background load.": "Bounded + dead-letter + alert (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T17:50:10.303Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "cd04b55a-5f1a-4672-b3ed-40ea1b1636bc",
|
||||
"toolUseId": "toolu_016fgg5mWHBjFGWxQPRz9Whr",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D4 — Add jitter to the backoff curve, or keep it deterministic?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: When a downstream service goes down, hundreds of jobs across all 5 workers fail at the same moment. With a pure exponential curve they all wake up at exactly the same moment too, and hit the recovering service as one wave, which can knock it over again. Jitter adds a random spread to each delay so the retries trickle back instead of stampeding.\nStakes if we pick wrong: a downstream that recovers from an outage gets re-flattened by your own synchronized retry wave, turning a 2-minute blip into a 20-minute incident.\nRecommendation: A because it is two lines inside the strategy function you already own and it is the standard mitigation for retry storms; deterministic curves are only useful in tests, which can seed or stub the random source.\nCompleteness: A=9/10, B=6/10\nPros / cons:\nA) Equal jitter (recommended)\n ✅ Retries after a shared outage spread across the window instead of returning as one synchronized burst\n ✅ Lives inside the single strategy function from D1, so every worker gets it with no per-worker code\n ❌ Curve tests need an injectable random source to stay deterministic\nB) No jitter (pure curve)\n ✅ Exact, predictable retry times that are easy to reason about and assert in tests\n ✅ Zero extra code beyond the curve itself\n ❌ All jobs that fail together retry together, so the retry framework itself becomes a traffic amplifier during outages\nNet: predictability in tests against stampede protection in production; the test cost is one injected random source.",
|
||||
"header": "Jitter",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Equal jitter (recommended)",
|
||||
"description": "Inside the single backoff strategy function, compute the exponential delay and return half of it plus a random amount up to the other half (equal jitter). The random source is injectable so tests can pin it. Required proof: unit test that returned delays fall within [curve/2, curve] for each attempt, and that a pinned random source gives a deterministic value. R5 stays pending. Completeness 9/10. human: ~1 h / CC: ~5 min."
|
||||
},
|
||||
{
|
||||
"label": "No jitter (pure curve)",
|
||||
"description": "Return the exact exponential curve value with no random spread. Required proof: unit test of the exact value per attempt. R5 stays pending. Completeness 6/10. human: ~30 min / CC: ~3 min."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D4 — Add jitter to the backoff curve, or keep it deterministic?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: When a downstream service goes down, hundreds of jobs across all 5 workers fail at the same moment. With a pure exponential curve they all wake up at exactly the same moment too, and hit the recovering service as one wave, which can knock it over again. Jitter adds a random spread to each delay so the retries trickle back instead of stampeding.\nStakes if we pick wrong: a downstream that recovers from an outage gets re-flattened by your own synchronized retry wave, turning a 2-minute blip into a 20-minute incident.\nRecommendation: A because it is two lines inside the strategy function you already own and it is the standard mitigation for retry storms; deterministic curves are only useful in tests, which can seed or stub the random source.\nCompleteness: A=9/10, B=6/10\nPros / cons:\nA) Equal jitter (recommended)\n ✅ Retries after a shared outage spread across the window instead of returning as one synchronized burst\n ✅ Lives inside the single strategy function from D1, so every worker gets it with no per-worker code\n ❌ Curve tests need an injectable random source to stay deterministic\nB) No jitter (pure curve)\n ✅ Exact, predictable retry times that are easy to reason about and assert in tests\n ✅ Zero extra code beyond the curve itself\n ❌ All jobs that fail together retry together, so the retry framework itself becomes a traffic amplifier during outages\nNet: predictability in tests against stampede protection in production; the test cost is one injected random source.": "Equal jitter (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T17:51:07.968Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "cd04b55a-5f1a-4672-b3ed-40ea1b1636bc",
|
||||
"toolUseId": "toolu_01AGgMxL2cDbt1vqtMqNz8dH",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D5 — Should workers name errors that must not be retried, or retry every failure to the ceiling?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: Some failures fix themselves if you wait (a database hiccup, a slow API). Others never will (a payload that fails validation, a revoked API key). Retrying the second kind five times with growing delays just wastes capacity and delays the moment someone notices. Letting each worker say \"these error types are permanent\" sends them straight to the dead-letter set on the first try.\nStakes if we pick wrong: a bad payload burns 5 attempts and up to the full backoff window before it surfaces, and during a bad deploy every job does this at once.\nRecommendation: A because it is a small per-worker list, the dead-letter path already exists from D3, and it turns a permanent failure into an immediate signal instead of a delayed one. Medium confidence on which types are permanent; verify against the actual error classes when implementing.\nCompleteness: A=9/10, B=6/10\nPros / cons:\nA) Explicit non-retryable list (recommended)\n ✅ Permanent failures reach the dead-letter set on attempt 1, so operators see bad payloads or revoked credentials within seconds\n ✅ Unknown errors still default to retry, so nothing transient is accidentally dropped\n ❌ Each worker needs a short, reviewed list of permanent error types, and a wrong entry makes a transient error permanent\nB) Retry everything to ceiling\n ✅ No classification to get wrong; behavior is identical for every worker\n ✅ Nothing to maintain when new error types appear\n ❌ Permanent failures consume the full attempt budget and backoff window before anyone can see them\nNet: a short reviewed list per worker against a guaranteed delay on every permanent failure.",
|
||||
"header": "Error classes",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Explicit non-retryable list (recommended)",
|
||||
"description": "Each of the 4 non-webhook workers declares its non-retryable error types (validation errors, auth/permission errors, malformed payload). Those bypass retry and land in the dead-letter set (R3) on attempt 1 with the error attached. Any error not on the list retries per R3/R4. Required proof: per worker, one test that a listed error goes to dead-letter without a retry, and one test that an unlisted error retries. Completeness 9/10. human: ~half day / CC: ~15 min."
|
||||
},
|
||||
{
|
||||
"label": "Retry everything to ceiling",
|
||||
"description": "No classification. Every failure in the 4 non-webhook workers retries per R3/R4 until the ceiling, then lands in dead-letter. Required proof: covered by R3 tests. Completeness 6/10. human: 0 / CC: 0."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D5 — Should workers name errors that must not be retried, or retry every failure to the ceiling?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: Some failures fix themselves if you wait (a database hiccup, a slow API). Others never will (a payload that fails validation, a revoked API key). Retrying the second kind five times with growing delays just wastes capacity and delays the moment someone notices. Letting each worker say \"these error types are permanent\" sends them straight to the dead-letter set on the first try.\nStakes if we pick wrong: a bad payload burns 5 attempts and up to the full backoff window before it surfaces, and during a bad deploy every job does this at once.\nRecommendation: A because it is a small per-worker list, the dead-letter path already exists from D3, and it turns a permanent failure into an immediate signal instead of a delayed one. Medium confidence on which types are permanent; verify against the actual error classes when implementing.\nCompleteness: A=9/10, B=6/10\nPros / cons:\nA) Explicit non-retryable list (recommended)\n ✅ Permanent failures reach the dead-letter set on attempt 1, so operators see bad payloads or revoked credentials within seconds\n ✅ Unknown errors still default to retry, so nothing transient is accidentally dropped\n ❌ Each worker needs a short, reviewed list of permanent error types, and a wrong entry makes a transient error permanent\nB) Retry everything to ceiling\n ✅ No classification to get wrong; behavior is identical for every worker\n ✅ Nothing to maintain when new error types appear\n ❌ Permanent failures consume the full attempt budget and backoff window before anyone can see them\nNet: a short reviewed list per worker against a guaranteed delay on every permanent failure.": "Explicit non-retryable list (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T17:52:12.006Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "cd04b55a-5f1a-4672-b3ed-40ea1b1636bc",
|
||||
"toolUseId": "toolu_016w6Sj4WK8TpN3bzensgKcm",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D6 — Extract one shared retry policy module now, or keep 5 copy-pasted envelopes?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: You just decided the curve shape, jitter, the attempt ceiling, the dead-letter alert and how errors are classified. Each of those has to live somewhere. If the retry envelope stays copy-pasted in 5 workers, every one of those decisions is copied 5 times, and the first bug fix or tuning change has to be made in 5 places and tested 5 times. One small shared module applies each decision once and every worker gets it.\nStakes if we pick wrong: a curve or ceiling bug fixed in 4 of 5 workers, or a jitter change that lands in 3, and nobody notices until the fifth worker stampedes a downstream.\nRecommendation: A because the plan already states the 5 bodies are identical, the module is about 50 lines, the migration is mechanical per worker, and doing the refactor commit before the behavior commit keeps each step reviewable. (human: ~1 day / CC: ~30 min)\nCompleteness: A=9/10, B=6/10, C=3/10\nPros / cons:\nA) Extract now, migrate all 5 (recommended)\n ✅ Curve, jitter, ceiling, dead-letter alert and classifier shape are each implemented and tested exactly once\n ✅ Refactor commit lands before the behavior-change commit, so each is small and reviewable on its own\n ❌ A defect in the shared module affects all 5 workers at once; the contract tests are the guard\nB) Extract now, migrate webhook only\n ✅ Smallest first step; proves the module against the worker whose behavior is changing anyway\n ✅ Other 4 workers are untouched in this change, so their risk is zero for now\n ❌ Leaves 4 copies carrying the new policy by hand, so the duplication the plan already called out gets worse, not better\nC) Leave duplication\n ✅ No refactor risk in this change at all\n ✅ Matches the plan as written\n ❌ Every approved policy decision is copy-pasted 5 times and drifts from the first fix onward\nNet: one 50-line module now against 5 hand-maintained copies of every retry decision.",
|
||||
"header": "Shared module",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Extract now, migrate all 5 (recommended)",
|
||||
"description": "Create one `retryPolicy` module exporting backoffStrategy(attempt, rng) with equal jitter and a configurable max-delay clamp, DEFAULT_MAX_ATTEMPTS, onDeadLetter(job, err) emitting the structured log and metric, and isNonRetryable(err, list). Structured attempt log fields: job id, attempt, delay, error class, decision. Inline ASCII state diagram in the module header. All 5 workers register with the library through it, each passing its own non-retryable list. Land the refactor commit before the behavior-change commit. Required proof: shared-contract unit tests for each export plus one integration test per worker that the library invokes the shared policy on failure. Completeness 9/10. human: ~1 day / CC: ~30 min."
|
||||
},
|
||||
{
|
||||
"label": "Extract now, migrate webhook only",
|
||||
"description": "Create the same `retryPolicy` module and migrate only the webhook worker in this change. The other 4 workers keep their copied envelopes and apply R3/R4/R5 by hand until a follow-up (TODO). Required proof: shared-contract unit tests plus one webhook integration test. Completeness 6/10. human: ~half day / CC: ~15 min."
|
||||
},
|
||||
{
|
||||
"label": "Leave duplication",
|
||||
"description": "Keep 5 copy-pasted envelopes as the plan proposes. Apply R3/R4/R5 policy in each copy. No shared module, no shared tests; per-copy tests only. Completeness 3/10. human: ~1 day of copy-paste / CC: ~20 min."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D6 — Extract one shared retry policy module now, or keep 5 copy-pasted envelopes?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: You just decided the curve shape, jitter, the attempt ceiling, the dead-letter alert and how errors are classified. Each of those has to live somewhere. If the retry envelope stays copy-pasted in 5 workers, every one of those decisions is copied 5 times, and the first bug fix or tuning change has to be made in 5 places and tested 5 times. One small shared module applies each decision once and every worker gets it.\nStakes if we pick wrong: a curve or ceiling bug fixed in 4 of 5 workers, or a jitter change that lands in 3, and nobody notices until the fifth worker stampedes a downstream.\nRecommendation: A because the plan already states the 5 bodies are identical, the module is about 50 lines, the migration is mechanical per worker, and doing the refactor commit before the behavior commit keeps each step reviewable. (human: ~1 day / CC: ~30 min)\nCompleteness: A=9/10, B=6/10, C=3/10\nPros / cons:\nA) Extract now, migrate all 5 (recommended)\n ✅ Curve, jitter, ceiling, dead-letter alert and classifier shape are each implemented and tested exactly once\n ✅ Refactor commit lands before the behavior-change commit, so each is small and reviewable on its own\n ❌ A defect in the shared module affects all 5 workers at once; the contract tests are the guard\nB) Extract now, migrate webhook only\n ✅ Smallest first step; proves the module against the worker whose behavior is changing anyway\n ✅ Other 4 workers are untouched in this change, so their risk is zero for now\n ❌ Leaves 4 copies carrying the new policy by hand, so the duplication the plan already called out gets worse, not better\nC) Leave duplication\n ✅ No refactor risk in this change at all\n ✅ Matches the plan as written\n ❌ Every approved policy decision is copy-pasted 5 times and drifts from the first fix onward\nNet: one 50-line module now against 5 hand-maintained copies of every retry decision.": "Extract now, migrate all 5 (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T17:53:59.810Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "cd04b55a-5f1a-4672-b3ed-40ea1b1636bc",
|
||||
"toolUseId": "toolu_01Pv8bKkKS2rSamqKz1rvNX1",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D7 — How do we protect the existing `processWebhookJob()` behavior through the rewrite?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: You are rewriting the code that sends webhooks to customers, and there are no tests around it. The rewrite is supposed to keep everything the same except how failures are handled. Without tests written against the current code first, there is no way to know whether the new version still sends the same request, with the same headers and signature, exactly once. The cheapest insurance is to pin the current behavior in tests before changing a line, then keep them green.\nStakes if we pick wrong: a subtly different request body or signature ships to every webhook receiver at once, or a duplicate send slips through, and the first signal is a customer complaint.\nRecommendation: A because characterization tests are cheap with AI, they are the only way to detect an unintended difference in a rewrite, and they become the permanent contract suite for the webhook path. (human: ~1 day / CC: ~20 min)\nCompleteness: A=9/10, B=6/10\nPros / cons:\nA) Characterize first, then rewrite (recommended)\n ✅ Pins request shape, headers, signature and success bookkeeping against the current code, so any unintended difference fails a test\n ✅ Intentional changes (pre-send retry, dead-letter) are written as explicit tests, so the diff between old and new behavior is documented\n ❌ Requires reading the current implementation carefully and a day of test writing before the rewrite starts\nB) At-most-once assertions only\n ✅ Covers the one guarantee the plan named as at risk\n ✅ Faster to write; no characterization pass\n ❌ Request shape, headers, signature and success bookkeeping can change silently and no test notices\nNet: pin the whole current contract for a day of work, or protect one guarantee and hope the rest survived.",
|
||||
"header": "Webhook regression",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Characterize first, then rewrite (recommended)",
|
||||
"description": "Before modifying `processWebhookJob()`, write characterization tests (e.g. `processWebhookJob.test`) against the current implementation asserting: exact request body, headers and signature for a fixed payload; exactly one send on success, on timeout and on 5xx; success bookkeeping. Keep them green through the rewrite. Then add intentional-difference tests: pre-send failure schedules a retry; timeout/5xx lands in dead-letter with no second send. Flag the suite CRITICAL in the plan. R8 stays pending. Completeness 9/10. human: ~1 day / CC: ~20 min."
|
||||
},
|
||||
{
|
||||
"label": "At-most-once assertions only",
|
||||
"description": "After the rewrite, add tests asserting exactly one send on success, timeout and 5xx, and a retry on pre-send failure. No characterization of request shape, headers, signature or success bookkeeping. R8 stays pending. Completeness 6/10. human: ~2 h / CC: ~5 min."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D7 — How do we protect the existing `processWebhookJob()` behavior through the rewrite?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: You are rewriting the code that sends webhooks to customers, and there are no tests around it. The rewrite is supposed to keep everything the same except how failures are handled. Without tests written against the current code first, there is no way to know whether the new version still sends the same request, with the same headers and signature, exactly once. The cheapest insurance is to pin the current behavior in tests before changing a line, then keep them green.\nStakes if we pick wrong: a subtly different request body or signature ships to every webhook receiver at once, or a duplicate send slips through, and the first signal is a customer complaint.\nRecommendation: A because characterization tests are cheap with AI, they are the only way to detect an unintended difference in a rewrite, and they become the permanent contract suite for the webhook path. (human: ~1 day / CC: ~20 min)\nCompleteness: A=9/10, B=6/10\nPros / cons:\nA) Characterize first, then rewrite (recommended)\n ✅ Pins request shape, headers, signature and success bookkeeping against the current code, so any unintended difference fails a test\n ✅ Intentional changes (pre-send retry, dead-letter) are written as explicit tests, so the diff between old and new behavior is documented\n ❌ Requires reading the current implementation carefully and a day of test writing before the rewrite starts\nB) At-most-once assertions only\n ✅ Covers the one guarantee the plan named as at risk\n ✅ Faster to write; no characterization pass\n ❌ Request shape, headers, signature and success bookkeeping can change silently and no test notices\nNet: pin the whole current contract for a day of work, or protect one guarantee and hope the rest survived.": "Characterize first, then rewrite (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T17:55:40.441Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "cd04b55a-5f1a-4672-b3ed-40ea1b1636bc",
|
||||
"toolUseId": "toolu_01N7xKjvDw5oJhSmyLK3zRAR",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D8 — Do the per-worker integration tests run the real job library, or a mocked hook?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: The whole point of D1 was that the library remembers attempt counts across worker restarts and keeps failed jobs somewhere you can find them. A test that fakes the library cannot check either of those; it only checks that your function was called. Running the real library against a throwaway test queue is slower but proves the parts you are relying on actually behave.\nStakes if we pick wrong: the retry count resets on every deploy or the failed set is empty when you need it, and every test was green because the mock said so.\nRecommendation: A because the library's persistence and failed set are load-bearing assumptions from D1 and D3, and a mock cannot verify either; the cost is a test backend fixture the library almost certainly already ships.\nCompleteness: A=9/10, B=6/10\nPros / cons:\nA) Real library backend in tests (recommended)\n ✅ Proves attempt count survives a worker restart and that exhausted jobs are actually in the failed set with their last error\n ✅ Catches library-version behavior changes and off-by-one attempt numbering that a stub would hide\n ❌ Slower suite and a test backend fixture to maintain (in-process queue or container)\nB) Mocked library hooks\n ✅ Fast, deterministic, no external fixture\n ✅ Enough to prove the worker wiring calls the shared policy\n ❌ Restart persistence and failed-set contents stay unverified, which are exactly the guarantees D1 and D3 depend on\nNet: a slower fixture that verifies the library promises you are betting on, or fast tests that trust them.",
|
||||
"header": "Integration depth",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Real library backend in tests (recommended)",
|
||||
"description": "Per-worker integration tests (5) run the actual job library against a test backend (in-process or containerized queue). Assertions: strategy invoked with real attempt numbers; attempt count survives a simulated worker restart mid-backoff; after the ceiling the job is in the failed set with its last error; a listed non-retryable error is in the failed set after attempt 1. Mark [E2E]. Completeness 9/10. human: ~1 day / CC: ~30 min."
|
||||
},
|
||||
{
|
||||
"label": "Mocked library hooks",
|
||||
"description": "Per-worker tests stub the library retry hook and assert the shared policy is invoked with the expected arguments. No restart or failed-set verification. Completeness 6/10. human: ~2 h / CC: ~10 min."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D8 — Do the per-worker integration tests run the real job library, or a mocked hook?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: The whole point of D1 was that the library remembers attempt counts across worker restarts and keeps failed jobs somewhere you can find them. A test that fakes the library cannot check either of those; it only checks that your function was called. Running the real library against a throwaway test queue is slower but proves the parts you are relying on actually behave.\nStakes if we pick wrong: the retry count resets on every deploy or the failed set is empty when you need it, and every test was green because the mock said so.\nRecommendation: A because the library's persistence and failed set are load-bearing assumptions from D1 and D3, and a mock cannot verify either; the cost is a test backend fixture the library almost certainly already ships.\nCompleteness: A=9/10, B=6/10\nPros / cons:\nA) Real library backend in tests (recommended)\n ✅ Proves attempt count survives a worker restart and that exhausted jobs are actually in the failed set with their last error\n ✅ Catches library-version behavior changes and off-by-one attempt numbering that a stub would hide\n ❌ Slower suite and a test backend fixture to maintain (in-process queue or container)\nB) Mocked library hooks\n ✅ Fast, deterministic, no external fixture\n ✅ Enough to prove the worker wiring calls the shared policy\n ❌ Restart persistence and failed-set contents stay unverified, which are exactly the guarantees D1 and D3 depend on\nNet: a slower fixture that verifies the library promises you are betting on, or fast tests that trust them.": "Real library backend in tests (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T17:56:39.620Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "cd04b55a-5f1a-4672-b3ed-40ea1b1636bc",
|
||||
"toolUseId": "toolu_01XEdmtg1U25aqBrEGFb8dfg",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D9 — Cache the dependency graph across retries, or recompute it on every attempt?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: Every time a job retries, the plan reads the whole payload from the database again and rebuilds the same dependency graph from it. The payload never changes between attempts, so the answer is always the same. With up to 5 attempts, that is up to 5 reads and 5 builds per failing job, and failing jobs pile up exactly when something is already down. Building once and storing the result with the job removes almost all of that.\nStakes if we pick wrong: a downstream outage turns into a database load spike from your own retries, or you spend effort caching something that turns out to be cheap.\nRecommendation: A because the graph is derived from an immutable payload, the library already stores job data per attempt, and the guard (payload hash check) makes the cache safe; it also removes the redundant DB fetch. Confidence is medium: if payloads are tiny and the graph build is microseconds, C is acceptable and this becomes a TODO.\nCompleteness: A=9/10, B=5/10, C=4/10\nPros / cons:\nA) Compute once, store on job (recommended)\n ✅ Retries read no extra payload and build no graph; outage-time DB load drops from 5N to about N\n ✅ Survives worker restarts and works across workers because the cache lives in the job data, not in a process\n ❌ Adds serialized graph size to each job record and needs a payload-hash guard to stay correct\nB) In-process memo\n ✅ Simple to add, no change to job data shape\n ✅ Helps when the same worker process picks up the retry\n ❌ Retries usually land on a different worker or after a restart, so the memo misses most of the time and still re-fetches the payload\nC) Leave as-is\n ✅ Zero new code and no cache correctness to reason about\n ✅ Bounded at 5 attempts by D3, so the waste is finite\n ❌ Every retry storm during an outage multiplies database reads by up to 5\nNet: one persisted derived value with a hash guard, or accept a 5x read multiplier exactly when the system is least healthy.",
|
||||
"header": "Graph caching",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Compute once, store on job (recommended)",
|
||||
"description": "On attempt 1, read the payload (from the library's job data if it carries it, else one DB fetch), build the dependency graph, and persist the serialized graph plus a payload hash in the job data. On later attempts, verify the hash and deserialize; on mismatch, rebuild. Required proof: unit test that attempt 2+ performs no DB fetch and no graph build when the hash matches; test that a hash mismatch triggers a rebuild. Completeness 9/10. human: ~half day / CC: ~15 min."
|
||||
},
|
||||
{
|
||||
"label": "In-process memo",
|
||||
"description": "Memoize the built graph per job id in worker memory. Payload re-fetch unchanged. Required proof: test that a second attempt in the same process reuses the graph. Completeness 5/10. human: ~1 h / CC: ~5 min."
|
||||
},
|
||||
{
|
||||
"label": "Leave as-is",
|
||||
"description": "Re-fetch the payload and rebuild the graph on every attempt, as the plan proposes. No new tests. Completeness 4/10. human: 0 / CC: 0."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D9 — Cache the dependency graph across retries, or recompute it on every attempt?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: Every time a job retries, the plan reads the whole payload from the database again and rebuilds the same dependency graph from it. The payload never changes between attempts, so the answer is always the same. With up to 5 attempts, that is up to 5 reads and 5 builds per failing job, and failing jobs pile up exactly when something is already down. Building once and storing the result with the job removes almost all of that.\nStakes if we pick wrong: a downstream outage turns into a database load spike from your own retries, or you spend effort caching something that turns out to be cheap.\nRecommendation: A because the graph is derived from an immutable payload, the library already stores job data per attempt, and the guard (payload hash check) makes the cache safe; it also removes the redundant DB fetch. Confidence is medium: if payloads are tiny and the graph build is microseconds, C is acceptable and this becomes a TODO.\nCompleteness: A=9/10, B=5/10, C=4/10\nPros / cons:\nA) Compute once, store on job (recommended)\n ✅ Retries read no extra payload and build no graph; outage-time DB load drops from 5N to about N\n ✅ Survives worker restarts and works across workers because the cache lives in the job data, not in a process\n ❌ Adds serialized graph size to each job record and needs a payload-hash guard to stay correct\nB) In-process memo\n ✅ Simple to add, no change to job data shape\n ✅ Helps when the same worker process picks up the retry\n ❌ Retries usually land on a different worker or after a restart, so the memo misses most of the time and still re-fetches the payload\nC) Leave as-is\n ✅ Zero new code and no cache correctness to reason about\n ✅ Bounded at 5 attempts by D3, so the waste is finite\n ❌ Every retry storm during an outage multiplies database reads by up to 5\nNet: one persisted derived value with a hash guard, or accept a 5x read multiplier exactly when the system is least healthy.": "Compute once, store on job (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T17:58:34.047Z"
|
||||
},
|
||||
{
|
||||
"sessionId": "cd04b55a-5f1a-4672-b3ed-40ea1b1636bc",
|
||||
"toolUseId": "toolu_012XHAmgKwXGVbhDkoCK1K1R",
|
||||
"questions": [
|
||||
{
|
||||
"question": "D10 — Record the webhook at-least-once upgrade as a TODO?\nProject/branch/task: gstack-plan-count-yWJb6k on main, retry framework plan; follow-up to D2.\nELI10: In D2 you chose to keep webhooks at \"send at most once\", so a slow or erroring receiver means that delivery is dropped into the failed set instead of retried. The fix (retry with a delivery id the receiver can dedupe) needs receiver work first. This question only decides whether we write that follow-up down in TODOS.md so it does not get lost.\nStakes if we pick wrong: skipped, the only trace is a code comment and a decision-log row; built now, this PR grows and contradicts the D2 call.\nRecommendation: A because the upgrade has an external prerequisite and a clear trigger, which is exactly what a TODO is for.\nNote: options differ in kind, not coverage — no completeness score.\nPros / cons:\nA) Add to TODOS.md (recommended)\n ✅ Creates TODOS.md with the What/Why/Context/Depends-on record, findable by /retro and future reviews\n ✅ Zero implementation cost now; the marker in code and the TODO entry point at each other\n ❌ One more file in the repo that someone has to keep honest as work lands\nB) Skip\n ✅ No new file; the decision log and gstack-shortcut marker already carry the trigger\n ✅ Avoids a TODO nobody may pick up if receivers never add dedupe\n ❌ The trigger lives only in a comment and a JSONL row, easy to miss when receivers do change\nC) Build it now in this PR\n ✅ Ships the stronger delivery guarantee in the same change as the retry framework\n ✅ Reuses the retryPolicy module while it is fresh\n ❌ Reverses D2 and depends on receiver dedupe that does not exist yet, so duplicates would reach receivers\nNet: a TODO entry now versus relying on a code marker alone; building now is off the table until receivers can dedupe.",
|
||||
"header": "Webhook TODO",
|
||||
"multiSelect": false,
|
||||
"options": [
|
||||
{
|
||||
"label": "Add to TODOS.md (recommended)",
|
||||
"description": "Create TODOS.md at implementation time with the TODO record (What/Why/Context/Depends-on) under a `## Workers` section, P3, effort M. No product code change."
|
||||
},
|
||||
{
|
||||
"label": "Skip",
|
||||
"description": "Do not create TODOS.md. The decision log entry and the gstack-shortcut marker remain the only trail."
|
||||
},
|
||||
{
|
||||
"label": "Build it now in this PR",
|
||||
"description": "Extend the accepted scope to at-least-once delivery with delivery id and idempotency key; would reopen D2."
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"answered": true,
|
||||
"failed": false,
|
||||
"answers": {
|
||||
"D10 — Record the webhook at-least-once upgrade as a TODO?\nProject/branch/task: gstack-plan-count-yWJb6k on main, retry framework plan; follow-up to D2.\nELI10: In D2 you chose to keep webhooks at \"send at most once\", so a slow or erroring receiver means that delivery is dropped into the failed set instead of retried. The fix (retry with a delivery id the receiver can dedupe) needs receiver work first. This question only decides whether we write that follow-up down in TODOS.md so it does not get lost.\nStakes if we pick wrong: skipped, the only trace is a code comment and a decision-log row; built now, this PR grows and contradicts the D2 call.\nRecommendation: A because the upgrade has an external prerequisite and a clear trigger, which is exactly what a TODO is for.\nNote: options differ in kind, not coverage — no completeness score.\nPros / cons:\nA) Add to TODOS.md (recommended)\n ✅ Creates TODOS.md with the What/Why/Context/Depends-on record, findable by /retro and future reviews\n ✅ Zero implementation cost now; the marker in code and the TODO entry point at each other\n ❌ One more file in the repo that someone has to keep honest as work lands\nB) Skip\n ✅ No new file; the decision log and gstack-shortcut marker already carry the trigger\n ✅ Avoids a TODO nobody may pick up if receivers never add dedupe\n ❌ The trigger lives only in a comment and a JSONL row, easy to miss when receivers do change\nC) Build it now in this PR\n ✅ Ships the stronger delivery guarantee in the same change as the retry framework\n ✅ Reuses the retryPolicy module while it is fresh\n ❌ Reverses D2 and depends on receiver dedupe that does not exist yet, so duplicates would reach receivers\nNet: a TODO entry now versus relying on a code marker alone; building now is off the table until receivers can dedupe.": "Add to TODOS.md (recommended)"
|
||||
},
|
||||
"unansweredQuestionIndices": [],
|
||||
"answeredAt": "2026-09-29T18:01:55.194Z"
|
||||
}
|
||||
],
|
||||
"plan": "# Engineering review: Add background job retry framework\n\nReview target (fixed): `PLAN.md` in `/home/runner/.cache/gstack-paid-shard-mmiSh1/tmp/gstack-plan-count-yWJb6k` (branch `main`, commit `0ad2479`).\nReport file: this file (path requested by the user).\nReviewer: /plan-eng-review, session `642-1790703811-6005ed0c`, 2026-09-29.\n\n## Original plan (unchanged copy)\n\n# Plan: Add background job retry framework\n\n## Architecture\nWe'll roll a custom exponential-backoff scheduler inline in each worker\nrather than use the existing job library's built-in retry hooks. Same\nshape as the library version, but we want full control over the curve.\n\n## Code quality\nThe retry envelope (compute delay, log attempt, dispatch) is duplicated\nacross 5 worker files with copy-pasted bodies. We will leave the\nduplication for now and refactor \"later.\"\n\n## Tests\nThe existing `processWebhookJob()` flow gets rewritten as part of this\nchange. No regression test for the prior at-most-once delivery guarantee\nis planned.\n\n## Performance\nOn every retry we re-fetch the full job payload from the database, then\niterate the payload to recompute the dependency graph. Could cache the\ngraph on the first attempt; not planned.\n\n## Scope Challenge record\n\nEvidence available: plan text only. The repo contains `PLAN.md` and `CLAUDE.md`; the 5 worker files, `processWebhookJob()`, the job library and its retry hooks are `not available` in this checkout. Findings quote plan lines and are calibrated as plan-text findings.\n\nComplexity count (estimates from plan text): ~5-6 changed files (5 worker files; `processWebhookJob()` may live in one of them), 0 new classes/services (scheduler is inline). Below the 8-file / 2-class gate, so the complexity selectors (B) are skipped.\n\nSearch check: Aside unavailable, host WebSearch used. Industry default [Layer 1]: library built-in retry, exponential backoff + jitter, bounded attempts, dead-letter, idempotent handlers.\n\n## Decision ledger\n\n### R1: Retry scheduler mechanism (library hooks vs custom inline scheduler)\nFinding: SC-1, P1, confidence 8/10, PLAN.md:7-9, reviewer: plan-eng-review (native)\nPlan baseline: original proposal, \"custom exponential-backoff scheduler inline in each worker rather than use the existing job library's built-in retry hooks\" (PLAN.md:7-9). Nothing approved yet.\nRuntime evidence: unknown. Job library and worker files not available in this checkout; plan text states the library has built-in retry hooks and the custom version is the \"same shape\".\nComparison grid:\n\n| Choice | Current | A) Library hooks + custom curve | B) Custom inline scheduler |\n|---|---|---|---|\n| R1 retry mechanism | custom inline scheduler (proposed) | library retry hooks, backoff supplied as one strategy function | custom scheduler inline per worker, as proposed |\n| Backoff curve ownership | \"full control\" wanted | full control via strategy function (verify hook accepts a function; else fall back to B) | full control |\n| Attempt count persistence / terminal handling | unspecified | inherited from library | must be hand-built (pending, R3) |\n| R2 webhook delivery semantics | pending | pending | pending |\n| R3 attempt bound + dead-letter | pending | pending | pending |\n| R4 jitter | pending | pending | pending |\n| R5 shared envelope | pending | pending | pending |\n\nQuestion D1:\nD1 — Use the job library's retry hooks or roll a custom inline scheduler?\nProject/branch/task: `main` of the plan fixture repo, plan \"Add background job retry framework\".\nELI10: The job library you already run has a retry system built in. The plan wants to rebuild the same thing by hand inside each of the 5 workers, only so the delay curve can be tuned. Retry systems are easy to get subtly wrong: losing the attempt count when a worker restarts, retrying forever, or retrying twice at once. The library has already paid for those bugs; a hand-rolled copy pays for them again in production.\nStakes if we pick wrong: a hand-rolled scheduler that drops attempt state on restart or double-schedules turns one failed job into duplicate side effects or an infinite retry loop, with no dead-letter to catch it.\nRecommendation: A because the plan admits the shapes are identical, and the curve is pluggable in the library through a strategy function, so A delivers the same control with far less new code. (human: ~1 day / CC: ~20 min for A; human: ~1 week / CC: ~2 h for B plus ongoing ownership)\nCompleteness: A=9/10, B=5/10\nPros / cons:\nA) Library hooks + custom curve (recommended)\n ✅ Attempt counting, persistence across restarts and terminal handling come from tested library code, not new code\n ✅ The custom curve still lives in one strategy function, so \"full control over the curve\" is preserved\n ❌ Requires confirming the library's hook accepts a custom delay function; if it does not, we fall back to B for the curve only\nB) Custom inline scheduler\n ✅ Zero dependencLine truncated
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"source": "local rerun smoke-2.1.284-1790709409 (Claude Code 2.1.284) of plan-eng-multi-finding-batching: the Create pane stayed unanswered for 1,372 s because its title row was cropped above the file row",
|
||||
"cwd": "/tmp/gstack-plan-count-Z3cntL",
|
||||
"screen": " ../gstack-e2e-plan-eng-batching-DINQ9m/gstack-test-plan-eng-batching.md\n\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\n 1 # Eng Review \u2014 Plan: Add background job retry framework\n 2\n 3 Review target (fixed): `/tmp/gstack-plan-count-Z3cntL/PLAN.md` on branch `main` (commit 844c6ae)\n 4 Reviewer: /plan-eng-review (Claude, session 196868-1790709430-09cebc2c), 2026-09-29\n 5 Report file: this file (user-requested destination)\n 6\n 7 ## Original plan (unchanged copy)\n 8\n 9 # Plan: Add background job retry framework\n 10\n 11 ## Architecture\n 12 We'll roll a custom exponential-backoff scheduler inline in each worker\n 13 rather than use the existing job library's built-in retry hooks. Same\n 14 shape as the library version, but we want full control over the curve.\n 15\n 16 ## Code quality\n 17 The retry envelope (compute delay, log attempt, dispatch) is duplicated\n 18 across 5 worker files with copy-pasted bodies. We will leave the\n 19 duplication for now and refactor \"later.\"\n 20\n 21 ## Tests\n 22 The existing `processWebhookJob()` flow gets rewritten as part of this\n 23 change. No regression test for the prior at-most-once delivery guarantee\n 24 is planned.\n 25\n 26 ## Performance\n 27 On every retry we re-fetch the full job payload from the database, then\n 28 iterate the payload to recompute the dependency graph. Could cache the\n 29 graph on the first attempt; not planned.\n\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\u254c\n Do you want to create gstack-test-plan-eng-batching.md?\n \u276f 1. Yes\n 2. Yes, and switch to accept edits (auto-approve file edits and common file commands) for this session; Yes, and\n always allow access to /tmp/gstack-e2e-plan-eng-batching-DINQ9m for this session (shift+tab)\n 3. No\n\n Esc to cancel \u00b7 Tab to amend\n",
|
||||
"event": {
|
||||
"sessionId": "16914623-f7a4-49ca-bea5-fb4b56274cb1",
|
||||
"toolUseId": "toolu_01BaNDAjTWTr5bf2sdwWC4DA",
|
||||
"input": {
|
||||
"file_path": "/tmp/gstack-e2e-plan-eng-batching-DINQ9m/gstack-test-plan-eng-batching.md",
|
||||
"content": "# Eng Review \u2014 Plan: Add background job retry framework\n\nReview target (fixed): `/tmp/gstack-plan-count-Z3cntL/PLAN.md` on branch `main` (commit 844c6ae)\nReviewer: /plan-eng-review (Claude, session 196868-1790709430-09cebc2c), 2026-09-29\nReport file: this file (user-requested destination)\n\n## Original plan (unchanged copy)\n\n# Plan: Add background job retry framework\n\n## Architecture\nWe'll roll a custom exponential-backoff scheduler inline in each worker\nrather than use the existing job library's built-in retry hooks. Same\nshape as the library version, but we want full control over the curve.\n\n## Code quality\nThe retry envelope (compute delay, log attempt, dispatch) is duplicated\nacross 5 worker files with copy-pasted bodies. We will leave the\nduplication for now and refactor \"later.\"\n\n## Tests\nThe existing `processWebhookJob()` flow gets rewritten as part of this\nchange. No regression test for the prior at-most-once delivery guarantee\nis planned.\n\n## Performance\nOn every retry we re-fetch the full job payload from the database, then\niterate the payload to recompute the dependency graph. Could cache the\ngraph on the first attempt; not planned.\n"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -149,7 +149,7 @@ function hasNativePostureProse(text: string, posture: RegExp): boolean {
|
||||
/** Finish the selected native mode packet before waiting for its answer. */
|
||||
export function ceoModeSubmissionInput(
|
||||
visible: string, selected: NativePlanQuestionCall | undefined, targetMode: CeoMode,
|
||||
transcript: PlanCountTranscript, submitted: Set<string>,
|
||||
transcript: PlanCountTranscript, submitted: Set<string>, screenText = '',
|
||||
): string | null {
|
||||
if (!selected || selected.answered || selected.failed || !selected.sessionId || !selected.toolUseId ||
|
||||
transcript.status !== 'ready' || selected.questions.length < 2 ||
|
||||
@@ -161,16 +161,28 @@ export function ceoModeSubmissionInput(
|
||||
const modeQuestions = selected.questions.filter(q => q.options.filter(o => modeTitle(o.label)).length >= 2);
|
||||
if (modeQuestions.length !== 1 || findCeoModeOption(modeQuestions[0]!.options.map((o, i) =>
|
||||
({index:i + 1, label:o.label})), targetMode) === null) return null;
|
||||
const bar = posturePacketBar(visible);
|
||||
if (!bar || !bar.answered.every(Boolean) || JSON.stringify(bar.headers) !== JSON.stringify(
|
||||
selected.questions.map(q => q.header.trim().replace(/\s+/g, ' '))) ||
|
||||
planCountSubmissionInput(visible) !== '\r') return null;
|
||||
const rawBar = [...visible.matchAll(/←[^\r\n]+✔\s*Submit\s*→/g)].at(-1)!;
|
||||
const preceding = visible.slice(0, rawBar.index);
|
||||
if (/```|~~~|^\s*>|\b(?:example|quoted|source)[^:\n]*:\s*$/im.test(preceding)) return null;
|
||||
const compact = (text: string) => text.replace(/\s+/g, '');
|
||||
const panel = compact(visible.slice(rawBar.index! + rawBar[0].length)
|
||||
.replace(/^[ \t]*[│┃] ?/gm, '').replace(/^[ \t]*[●⏺] ?/gm, ''));
|
||||
const quotedContext = /```|~~~|^\s*>|\b(?:example|quoted|source)[^:\n]*:\s*$/im;
|
||||
const bar = posturePacketBar(visible);
|
||||
let review: string;
|
||||
if (bar) {
|
||||
if (!bar.answered.every(Boolean) || JSON.stringify(bar.headers) !== JSON.stringify(
|
||||
selected.questions.map(q => q.header.trim().replace(/\s+/g, ' '))) ||
|
||||
planCountSubmissionInput(visible) !== '\r') return null;
|
||||
const rawBar = [...visible.matchAll(/←[^\r\n]+✔\s*Submit\s*→/g)].at(-1)!;
|
||||
if (quotedContext.test(visible.slice(0, rawBar.index))) return null;
|
||||
review = visible.slice(rawBar.index! + rawBar[0].length);
|
||||
} else {
|
||||
// A review taller than the terminal scrolls its tab bar and heading off
|
||||
// the viewport (run 36606688266). The viewport must still end at the
|
||||
// focused Submit prompt; the accumulated screen text then supplies the
|
||||
// one complete review panel, authenticated below exactly as with a bar.
|
||||
const heading = screenText.lastIndexOf('Review your answers');
|
||||
if (heading < 0 || !compact(visible).endsWith(BARLESS_SUBMIT_END) ||
|
||||
quotedContext.test(screenText.slice(0, heading).split('\n').slice(-3).join('\n'))) return null;
|
||||
review = screenText.slice(heading);
|
||||
}
|
||||
const panel = compact(review.replace(/^[ \t]*[│┃] ?/gm, '').replace(/^[ \t]*[●⏺] ?/gm, ''));
|
||||
// Authenticate the complete review panel against native questions and
|
||||
// offered answers. An intended keypress or a selected-mode echo is not an ACK.
|
||||
let prefixes = ['Reviewyouranswers'];
|
||||
|
||||
@@ -1987,7 +1987,7 @@ function conflictingDesignClosure(text: string): boolean {
|
||||
new RegExp(`(?:^|[.!?;]\\s+|\\n)(?:If|When|Once|Unless|Assuming|Provided)\\b[^.!?\\n]*\\b${owner}\\b`, 'i').test(text);
|
||||
}
|
||||
|
||||
function hasCompletePlanReport(expectedPlanPath: string, minimumMtime: number, maximumMtime: number,
|
||||
export function hasCompletePlanReport(expectedPlanPath: string, minimumMtime: number, maximumMtime: number,
|
||||
allowRunHeaderForFailure = false, requiredReview?: 'Design'): boolean {
|
||||
if (!path.isAbsolute(expectedPlanPath)) return false;
|
||||
try {
|
||||
|
||||
@@ -229,6 +229,18 @@ export function createEvalCollector(suite: string): EvalCollector | null {
|
||||
}
|
||||
|
||||
/** DRY helper to record an E2E test result into the eval collector. */
|
||||
/** Exit reasons for an API or transport failure (session-runner.ts). */
|
||||
const INFRA_EXIT_REASONS = new Set(['error_api', 'timeout_startup', 'error_output_stream']);
|
||||
|
||||
/** API/transport error or CLI crash before the first model turn: INFRA, never a
|
||||
* verdict on the product. Any assistant event or counted turn means the model
|
||||
* ran, so its refusal, timeout or wrong answer stays an ordinary failure. */
|
||||
export function isPreTurnInfraFailure(result: Pick<SkillTestResult, 'exitReason' | 'transcript' | 'costEstimate'>): boolean {
|
||||
return result.costEstimate.turnsUsed === 0
|
||||
&& (INFRA_EXIT_REASONS.has(result.exitReason) || /^exit_code_\d+$/.test(result.exitReason))
|
||||
&& !result.transcript.some(event => event?.type === 'assistant');
|
||||
}
|
||||
|
||||
export function recordE2E(
|
||||
evalCollector: EvalCollector | null,
|
||||
name: string,
|
||||
@@ -241,9 +253,11 @@ export function recordE2E(
|
||||
? `${result.toolCalls[result.toolCalls.length - 1].tool}(${JSON.stringify(result.toolCalls[result.toolCalls.length - 1].input).slice(0, 60)})`
|
||||
: undefined;
|
||||
|
||||
const passed = extra?.passed ?? (result.exitReason === 'success' && result.browseErrors.length === 0);
|
||||
evalCollector?.addTest({
|
||||
name, suite, tier: 'e2e',
|
||||
passed: result.exitReason === 'success' && result.browseErrors.length === 0,
|
||||
passed,
|
||||
...(!passed && isPreTurnInfraFailure(result) ? { failure_class: 'infra' as const } : {}),
|
||||
duration_ms: result.duration,
|
||||
cost_usd: result.costEstimate.estimatedCost,
|
||||
transcript: result.transcript,
|
||||
|
||||
@@ -117,6 +117,9 @@ export function isEngBatchingIssueAUQ(fp: AskUserQuestionFingerprint, priorCalls
|
||||
return !priorCalls.some(prior => batchingIssueNumber(prior) === issue);
|
||||
}
|
||||
|
||||
// The report's target declaration field (Target / Review target / Reviewed target, optionally qualified).
|
||||
const TARGET_FIELD = /^(?:Reviewed |Review )?target(?: \([^)\n]*\))?:/i;
|
||||
|
||||
/** A native brief can use its D number and topic while its stable R identity
|
||||
* lives in the required saved ledger. Count that owned choice, not a title
|
||||
* spelling. This does not approve the row or validate the implementation. */
|
||||
@@ -160,26 +163,31 @@ function recordedBatchingIssue(call: NativePlanQuestionCall, savedPlan: string):
|
||||
const rawSourceNames = [...(lines[1] ?? '').matchAll(/\b[\w./-]+\.md\b/g)];
|
||||
const directSource = sourceNames.length > 0 && sourceNames.every(name => name === 'PLAN.md') &&
|
||||
new Set([...metadata.matchAll(/\bPLAN\.md:([1-9]\d*(?:[-–][1-9]\d*)?)\b/g)].map(match => match[1])).size <= 1;
|
||||
const targetName = (s: string) => clean(s).replace(/^Eng(?:ineering)? review:\s*/i, '')
|
||||
const targetName = (s: string) => clean(s).replace(/^Eng(?:ineering)? review\s*[:—–-]\s*/i, '')
|
||||
.replace(/^Plan\s*[:—–-]\s*/i, '').toLowerCase();
|
||||
const named = [...(lines[1] ?? '').matchAll(/"(Plan:\s*[^"\n]+)"|“(Plan:\s*[^”\n]+)”/g)]
|
||||
.map(match => targetName(match[1] ?? match[2]!));
|
||||
const named = [...(lines[1] ?? '').matchAll(/"(Plan:\s*[^"\n]+)"|“(Plan:\s*[^”\n]+)”|\b[Pp]lan\s+"([^"\n]+)"|\b[Pp]lan\s+“([^”\n]+)”/g)]
|
||||
.map(match => targetName(match[1] ?? match[2] ?? match[3] ?? match[4]!));
|
||||
const titles = tokens.slice(0, start).filter(token => token.type === 'heading' && token.depth === 1);
|
||||
// Target declarations are fields, whatever their list or emphasis markup.
|
||||
const targetFields = tokens.slice(0, start).flatMap((token, at) => {
|
||||
if (token.type !== 'paragraph' || !currentHeading(at)) return [];
|
||||
if ((token.type !== 'paragraph' && token.type !== 'list') || !currentHeading(at)) return [];
|
||||
const previous = tokens.slice(0, at).filter(t => t.type !== 'space').at(-1);
|
||||
const quotedContext = /\b(?:quoted|copied|historical|example|hypothetical|archived)\b[^\n]*:\s*$/i;
|
||||
if (previous?.type === 'paragraph' && quotedContext.test(previous.raw)) return [];
|
||||
const parts = token.raw.split('\n');
|
||||
return parts.filter((line, i) => /^Reviewed target:/.test(line) &&
|
||||
const parts = token.raw.split('\n').map(line => line.replace(/^\s*(?:[-*+]|\d+[.)])\s+/, '').replace(/[*_]/g, '').trim());
|
||||
return parts.filter((line, i) => TARGET_FIELD.test(line) &&
|
||||
!parts.slice(0, i).some(part => quotedContext.test(part)));
|
||||
});
|
||||
const namedSource = !rawSourceNames.length && named.length === 1 && titles.length === 1 &&
|
||||
const targetFiles = targetFields.length === 1 ? [...targetFields[0]!.matchAll(/[\w./-]*[\w-]+\.md\b/g)].map(match => match[0]) : [];
|
||||
// An unsourced brief inherits the report's one current PLAN.md target; its
|
||||
// ledger record still supplies the cited finding. A brief that names its plan
|
||||
// must name the report title's plan, and an unfenced copy of that plan may
|
||||
// add its own H1 only when it names that same plan.
|
||||
const namedSource = !rawSourceNames.length && named.length <= 1 && titles.length >= 1 &&
|
||||
titles[0]!.type === 'heading' && currentHeading(tokens.indexOf(titles[0]!)) &&
|
||||
/^Eng(?:ineering)? review:\s*Plan\s*[:—–-]/i.test(clean(titles[0]!.text)) &&
|
||||
targetName(titles[0]!.text) === named[0] && targetFields.length === 1 &&
|
||||
/^Reviewed target:\s*`?PLAN\.md`?(?:\s|$)/.test(targetFields[0]!) &&
|
||||
[...targetFields[0]!.matchAll(/\b[\w./-]+\.md\b/g)].length === 1;
|
||||
targetFiles.length === 1 && targetFiles[0]!.split('/').at(-1) === 'PLAN.md' &&
|
||||
(named.length === 0 || /^Eng(?:ineering)? review\s*[:—–-]\s*\S/i.test(clean(titles[0]!.text)) &&
|
||||
titles.every(title => title.type === 'heading' && targetName(title.text) === named[0]));
|
||||
if (!directSource && !namedSource) return;
|
||||
const withdrawn = (value: string, owners: string) => new RegExp(
|
||||
`(?:^|[.!?;]\\s+|\\n)(?:Correction:\\s*)?(?:${owners}) (?:is|was|has been) ["“'‘]?(?:withdrawn|cancelled|canceled|rejected|superseded|resolved|closed|hypothetical|not current|no longer current)\\b`, 'i').test(prose(value, true));
|
||||
@@ -265,7 +273,7 @@ function recordedBatchingIssue(call: NativePlanQuestionCall, savedPlan: string):
|
||||
if (questions.length !== 1) continue;
|
||||
const inlineBrief = fields[questions[0]!]!.slice(marker.length).trim();
|
||||
const inline = Boolean(inlineBrief);
|
||||
if (!inline && (namedSource || clean(fields[questions[0]! + 1] ?? '') !== clean(title))) continue;
|
||||
if (!inline && clean(fields[questions[0]! + 1] ?? '') !== clean(title)) continue;
|
||||
const sources = [...finding[0]!.matchAll(/\b([\w./-]+\.md)(?::([1-9]\d*(?:[-–][1-9]\d*)?))?\b/g)];
|
||||
if (sources.length !== 1 || sources[0]![1] !== 'PLAN.md' ||
|
||||
!inline && !sources[0]![2] || source && sources[0]![2] !== source) continue;
|
||||
|
||||
@@ -512,9 +512,6 @@ export interface ArmJudgeScore {
|
||||
*/
|
||||
export const ARM_JUDGE_MODEL = CLAUDE_FRONTIER_EVAL_MODEL;
|
||||
|
||||
/** Bounded retry-on-malformed loop: total attempts, not extra retries. */
|
||||
export const ARM_JUDGE_ATTEMPTS = 2;
|
||||
|
||||
/**
|
||||
* Build the over-engineering rubric prompt. Exported (pure) so the free
|
||||
* selftest can verify prompt construction without any API call.
|
||||
@@ -587,10 +584,10 @@ export function parseArmJudgeResponse(raw: unknown): ArmJudgeScore {
|
||||
*
|
||||
* - Zero-diff arms are VALID scored cells: the agent built nothing, so the
|
||||
* score is deterministically 0/"none" — no API call.
|
||||
* - Bounded retry-on-malformed: ARM_JUDGE_ATTEMPTS total attempts. callJudge
|
||||
* already retries 429s internally; this loop covers malformed/refused JSON.
|
||||
* - One sample, never re-asked: a malformed or refused verdict is a failed
|
||||
* sample. callJudge's transport-level 429 backoff is not a verdict retry.
|
||||
* - `opts.call` is an injection seam so the free selftest can exercise the
|
||||
* retry bound without spending API money. Defaults to the real callJudge.
|
||||
* malformed path without spending API money. Defaults to the real callJudge.
|
||||
*/
|
||||
export async function armJudge(
|
||||
task: string,
|
||||
@@ -605,18 +602,10 @@ export async function armJudge(
|
||||
};
|
||||
}
|
||||
const call = opts?.call ?? callJudge;
|
||||
const prompt = buildArmJudgePrompt(task, diff);
|
||||
let lastError: unknown;
|
||||
for (let attempt = 1; attempt <= ARM_JUDGE_ATTEMPTS; attempt++) {
|
||||
try {
|
||||
const raw = await call<Record<string, unknown>>(prompt, ARM_JUDGE_MODEL);
|
||||
return parseArmJudgeResponse(raw);
|
||||
} catch (err) {
|
||||
lastError = err;
|
||||
}
|
||||
const raw = await call<Record<string, unknown>>(buildArmJudgePrompt(task, diff), ARM_JUDGE_MODEL);
|
||||
try {
|
||||
return parseArmJudgeResponse(raw);
|
||||
} catch (err) {
|
||||
throw new Error(`armJudge: malformed verdict (never resampled) — ${err instanceof Error ? err.message : String(err)}`);
|
||||
}
|
||||
throw new Error(
|
||||
`armJudge: no well-formed verdict after ${ARM_JUDGE_ATTEMPTS} attempts — `
|
||||
+ (lastError instanceof Error ? lastError.message : String(lastError)),
|
||||
);
|
||||
}
|
||||
@@ -284,7 +284,11 @@ function currentCreatePreview(preview: string, r: any, config: string, cwd: stri
|
||||
if(event.name!=='Write'||`${event.sessionId}:${event.toolUseId}`!==r.pendingId||event.input?.file_path!==r.expected||
|
||||
Date.parse(event.timestamp)<startedAt||typeof event.input.content!=='string'||
|
||||
Buffer.byteLength(event.input.content)>MAX_WRITE_INPUT_BYTES) return false;
|
||||
const source=event.input.content.split(/\r?\n/), rows=preview.split('\n');
|
||||
// A crop can keep the pane's file row and rule above the preview while its
|
||||
// "Create file" title scrolls away. That row must name the owned path.
|
||||
const header=/^ {0,3}(?![1-9]\d*(?:[ \t]|\n))(\S[^\n]*)\n[╌─━]{3,}[ \t]*\n/.exec(preview);
|
||||
if(header && path.resolve(cwd,header[1]!.trim())!==r.expected) return false;
|
||||
const source=event.input.content.split(/\r?\n/), rows=preview.slice(header?.[0].length ?? 0).split('\n');
|
||||
const numbered:Array<{line:number;text:string}>=[];
|
||||
let leading='';
|
||||
for(const row of rows) {
|
||||
|
||||
@@ -6,14 +6,16 @@
|
||||
* negative coverage: hand-graded good/bad recommendation strings, asserted
|
||||
* against the same threshold the production E2E tests use (>= 4).
|
||||
*
|
||||
* Costs ~$0.04 per run (4 Haiku calls + 3 deterministic-only fixtures).
|
||||
* Each fixture is a pre-registered 3-sample judge panel: numeric substance
|
||||
* gates on the panel mean, the boolean checks on a 2-of-3 majority, and an
|
||||
* erroring sample fails the panel (never resampled). Costs ~$0.12 per run.
|
||||
* Touchfile-gated to test/helpers/llm-judge.ts so it fires on rubric
|
||||
* tweaks but not every test run. Runs only under EVALS=1 with an API key.
|
||||
*/
|
||||
|
||||
import { expect } from 'bun:test';
|
||||
import { CAPTURE_MS } from './helpers/eval-budgets';
|
||||
import { judgeRecommendation } from './helpers/llm-judge';
|
||||
import { judgePanel, judgePanelMajority, judgePanelMean, judgePanelReasoning, judgeRecommendation } from './helpers/llm-judge';
|
||||
import { describeIfSelected, testIfSelected } from './helpers/e2e-helpers';
|
||||
|
||||
// Fixtures wrap a realistic AskUserQuestion shape so the judge sees the menu
|
||||
@@ -37,13 +39,24 @@ C) Hybrid — V1 client-side, V1.5 promotes to gbrain
|
||||
Net: optimize for V1 ship velocity vs long-term agent reusability.`;
|
||||
}
|
||||
|
||||
async function judgeRecommendationPanel(text: string) {
|
||||
const samples = await judgePanel(() => judgeRecommendation(text));
|
||||
return {
|
||||
present: judgePanelMajority(samples, 'present'),
|
||||
commits: judgePanelMajority(samples, 'commits'),
|
||||
has_because: judgePanelMajority(samples, 'has_because'),
|
||||
reason_substance: judgePanelMean(samples, ['reason_substance']).reason_substance,
|
||||
reasoning: judgePanelReasoning(samples),
|
||||
};
|
||||
}
|
||||
|
||||
describeIfSelected('judgeRecommendation rubric sanity', ['llm-judge-recommendation'], () => {
|
||||
testIfSelected('llm-judge-recommendation', async () => {
|
||||
// Run all 7 fixtures sequentially in one test entry so the eval-store sees
|
||||
// a single result; individual assertions surface as failed expectations.
|
||||
|
||||
// SUBSTANCE 5: option-specific reason that contrasts an alternative.
|
||||
const good5 = await judgeRecommendation(buildAUQ(
|
||||
const good5 = await judgeRecommendationPanel(buildAUQ(
|
||||
'Recommendation: Choose C because hybrid ships V1 in gstack-only without blocking on cross-repo gbrain coordination, and locks the migration path before other agents take a hard dependency.',
|
||||
));
|
||||
expect(good5.present).toBe(true);
|
||||
@@ -55,7 +68,7 @@ describeIfSelected('judgeRecommendation rubric sanity', ['llm-judge-recommendati
|
||||
).toBeGreaterThanOrEqual(4);
|
||||
|
||||
// SUBSTANCE 4: concrete option-specific reason without alternative comparison.
|
||||
const good4 = await judgeRecommendation(buildAUQ(
|
||||
const good4 = await judgeRecommendationPanel(buildAUQ(
|
||||
'Recommendation: Choose B because client-side composition uses MCP tools that already exist in gstack and avoids any gbrain release dependency for V1.',
|
||||
));
|
||||
expect(good4.present).toBe(true);
|
||||
@@ -65,7 +78,7 @@ describeIfSelected('judgeRecommendation rubric sanity', ['llm-judge-recommendati
|
||||
).toBeGreaterThanOrEqual(4);
|
||||
|
||||
// SUBSTANCE ~1: boilerplate.
|
||||
const bad1 = await judgeRecommendation(buildAUQ(
|
||||
const bad1 = await judgeRecommendationPanel(buildAUQ(
|
||||
'Recommendation: Choose B because it is better.',
|
||||
));
|
||||
expect(bad1.present).toBe(true);
|
||||
@@ -76,7 +89,7 @@ describeIfSelected('judgeRecommendation rubric sanity', ['llm-judge-recommendati
|
||||
).toBeLessThan(4);
|
||||
|
||||
// SUBSTANCE ~3: generic.
|
||||
const bad3 = await judgeRecommendation(buildAUQ(
|
||||
const bad3 = await judgeRecommendationPanel(buildAUQ(
|
||||
'Recommendation: Choose B because it is faster.',
|
||||
));
|
||||
expect(bad3.present).toBe(true);
|
||||
@@ -87,7 +100,7 @@ describeIfSelected('judgeRecommendation rubric sanity', ['llm-judge-recommendati
|
||||
).toBeLessThan(4);
|
||||
|
||||
// NO BECAUSE: missing causal connective.
|
||||
const noBecause = await judgeRecommendation(buildAUQ(
|
||||
const noBecause = await judgeRecommendationPanel(buildAUQ(
|
||||
'Recommendation: Choose B (it has the best tradeoffs).',
|
||||
));
|
||||
expect(noBecause.present).toBe(true);
|
||||
@@ -95,7 +108,7 @@ describeIfSelected('judgeRecommendation rubric sanity', ['llm-judge-recommendati
|
||||
expect(noBecause.reason_substance).toBe(1);
|
||||
|
||||
// NO RECOMMENDATION: line missing entirely.
|
||||
const noRec = await judgeRecommendation(`D1 — Where should the smarts live?
|
||||
const noRec = await judgeRecommendationPanel(`D1 — Where should the smarts live?
|
||||
ELI10: ...
|
||||
Pros / cons:
|
||||
A) Server-side
|
||||
@@ -146,7 +159,7 @@ Net: ...`);
|
||||
],
|
||||
] as Array<[string, string, boolean]>;
|
||||
for (const [label, text, shouldPass] of crossModelCases) {
|
||||
const score = await judgeRecommendation(text);
|
||||
const score = await judgeRecommendationPanel(text);
|
||||
expect(score.present, `[cross-model:${label}] present should be true`).toBe(true);
|
||||
expect(score.has_because, `[cross-model:${label}] has_because should be true`).toBe(true);
|
||||
if (shouldPass) {
|
||||
@@ -175,7 +188,7 @@ Net: ...`);
|
||||
['whichever fits', 'Recommendation: whichever fits the team — A or B both work.'],
|
||||
];
|
||||
for (const [label, text] of hedgeForms) {
|
||||
const score = await judgeRecommendation(buildAUQ(text));
|
||||
const score = await judgeRecommendationPanel(buildAUQ(text));
|
||||
expect(score.present, `[hedge:${label}] present should be true`).toBe(true);
|
||||
expect(
|
||||
score.commits,
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import {expect, test} from 'bun:test';
|
||||
import {describe, expect, test} from 'bun:test';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
@@ -6,6 +6,7 @@ import {createFilePermissionRecorder, recordFilePermission, currentFilePermissio
|
||||
import {readPlanCountTranscript} from './helpers/plan-count-transcript';
|
||||
import {createPlanCountPermissionGuard} from './helpers/claude-pty-runner';
|
||||
import captures from './fixtures/plan-create-combined-permission-70b.json';
|
||||
import croppedTitle from './fixtures/plan-create-cropped-title-batching.json';
|
||||
|
||||
function fixture(captured: typeof captures[number]) {
|
||||
const dir=fs.mkdtempSync(path.join(os.tmpdir(),'create-combined-'));
|
||||
@@ -101,3 +102,50 @@ for(const captured of captures) {
|
||||
}finally{f.close();}
|
||||
});
|
||||
}
|
||||
|
||||
describe('Create pane cropped below its title (plan-eng-multi-finding-batching, CLI 2.1.284)', () => {
|
||||
function cropped() {
|
||||
const dir=fs.mkdtempSync(path.join(os.tmpdir(),'create-cropped-'));
|
||||
const cwd=path.join(dir,path.basename(croppedTitle.cwd)), config=path.join(dir,'config');fs.mkdirSync(cwd);
|
||||
const originalDir=path.dirname(croppedTitle.event.input.file_path);
|
||||
const expected=path.join(dir,path.basename(originalDir),path.basename(croppedTitle.event.input.file_path));
|
||||
fs.mkdirSync(path.dirname(expected));
|
||||
const {sessionId,toolUseId:id}=croppedTitle.event, timestamp=new Date().toISOString();
|
||||
const journal=path.join(config,'projects','owned',sessionId+'.jsonl');fs.mkdirSync(path.dirname(journal),{recursive:true});
|
||||
const recorder=createFilePermissionRecorder(cwd,config,expected)!;
|
||||
const input={...croppedTitle.event.input,file_path:expected};
|
||||
fs.writeFileSync(journal,JSON.stringify({cwd,sessionId,isSidechain:false,timestamp,
|
||||
message:{role:'assistant',content:[{type:'text',text:'Writing the review report.'},{type:'tool_use',id,name:'Write',input}]}})+'\n');
|
||||
recordFilePermission(JSON.stringify({hook_event_name:'PreToolUse',tool_name:'Write',session_id:sessionId,
|
||||
tool_use_id:id,cwd,transcript_path:journal,tool_input:input}),recorder.file,cwd,config,expected);
|
||||
// The relative file row keeps its captured sibling layout; only the footer's
|
||||
// absolute directory is rebound to this fixture.
|
||||
const screen=croppedTitle.screen.replaceAll(originalDir,path.dirname(expected));
|
||||
const read=(s=screen)=>currentFilePermissionEpoch(recorder.file,expected,cwd,config,Date.now()-1000,readPlanCountTranscript(config,cwd),s);
|
||||
return {screen,read,id:`${sessionId}:${id}`,close(){recorder.dispose();fs.rmSync(dir,{recursive:true,force:true});}};
|
||||
}
|
||||
|
||||
test('the captured pane shows its file row and rule but not the Create file title', () => {
|
||||
expect(croppedTitle.screen).not.toMatch(/(?:^|\n) {0,3}Create file[ \t]*\n/);
|
||||
expect(croppedTitle.screen.split('\n')[0]).toBe(' ../gstack-e2e-plan-eng-batching-DINQ9m/gstack-test-plan-eng-batching.md');
|
||||
});
|
||||
|
||||
test('the owned file row binds the pending Write and grants it once', () => {
|
||||
const f=cropped();try {
|
||||
const epoch=f.read();expect(epoch?.pendingId).toBe(f.id);
|
||||
const guard=createPlanCountPermissionGuard();
|
||||
expect(guard(f.screen,'',epoch)).toBe('grant');
|
||||
expect(guard(f.screen,'',epoch)).toBe('handled');
|
||||
}finally{f.close();}
|
||||
});
|
||||
|
||||
for(const [name,change] of [
|
||||
['a file row naming another file',(s:string)=>s.replace('/gstack-test-plan-eng-batching.md\n','/other.md\n')],
|
||||
['a file row in another directory',(s:string)=>s.replace(' ../gstack-e2e-plan-eng-batching-DINQ9m/',' ../elsewhere/')],
|
||||
['an edited preview row',(s:string)=>s.replace('We will leave the','We will fix the')],
|
||||
] as const) test(`the cropped pane rejects ${name}`,()=>{
|
||||
const f=cropped();try {
|
||||
const screen=change(f.screen);expect(screen).not.toBe(f.screen);expect(f.read(screen)).toBeFalsy();
|
||||
}finally{f.close();}
|
||||
});
|
||||
});
|
||||
@@ -4,7 +4,7 @@ import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { CAPTURE_LONG_MS } from './helpers/eval-budgets';
|
||||
import { recordE2E } from './helpers/e2e-helpers';
|
||||
import { isPreTurnInfraFailure, recordE2E } from './helpers/e2e-helpers';
|
||||
import { EvalCollector, isFinalizedEvalResultFile, listEvalJsonFiles, type EvalTestEntry } from './helpers/eval-store';
|
||||
import { OFFICE_HOURS_BUN_GRACE_MS, runRecordedOfficeHoursAttempt } from './helpers/office-hours-attempt';
|
||||
import { isPaidTestFile } from './helpers/paid-test-set';
|
||||
@@ -257,3 +257,37 @@ test('report deadline aborts, records once, cleans up and ignores late completio
|
||||
test('report recording controls stay outside the paid test filename patterns', () => {
|
||||
expect(isPaidTestFile('test/plan-review-report-recording.test.ts')).toBe(false);
|
||||
});
|
||||
|
||||
// A pre-turn API/transport failure is INFRA; once the model has run, a failure
|
||||
// keeps its ordinary class.
|
||||
function runnerResult(exitReason: string, turnsUsed = 0, transcript: any[] = [{ type: 'system', subtype: 'init' }]): any {
|
||||
return { exitReason, transcript, toolCalls: [], browseErrors: [], duration: 1, output: '',
|
||||
costEstimate: { inputChars: 1, outputChars: 0, estimatedTokens: 0, estimatedCost: 0, turnsUsed } };
|
||||
}
|
||||
function recordedClass(result: any, extra?: Partial<EvalTestEntry>) {
|
||||
const collector = new EvalCollector('e2e');
|
||||
recordE2E(collector, 'infra-probe', 'Infra probe', result, extra);
|
||||
return (collector as any).tests.at(-1)?.failure_class;
|
||||
}
|
||||
|
||||
test.each(['error_api', 'timeout_startup', 'error_output_stream', 'exit_code_1'])(
|
||||
'a %s before the first model turn is recorded as infra', exitReason => {
|
||||
expect(isPreTurnInfraFailure(runnerResult(exitReason))).toBe(true);
|
||||
expect(recordedClass(runnerResult(exitReason))).toBe('infra');
|
||||
});
|
||||
|
||||
test.each([
|
||||
['a timeout after model work', runnerResult('timeout')],
|
||||
['max turns', runnerResult('error_max_turns', 24)],
|
||||
['an API error after a turn', runnerResult('error_api', 3)],
|
||||
['an API error after an assistant message', runnerResult('error_api', 0, [{ type: 'assistant', message: { content: [{ type: 'text', text: 'I cannot help with that.' }] } }])],
|
||||
['a successful run', runnerResult('success', 2)],
|
||||
] as const)('%s is not infra', (_name, result) => {
|
||||
expect(isPreTurnInfraFailure(result)).toBe(false);
|
||||
expect(recordedClass(result)).toBeUndefined();
|
||||
});
|
||||
|
||||
test('an explicit pass or class from the caller wins', () => {
|
||||
expect(recordedClass(runnerResult('error_api'), { passed: true })).toBeUndefined();
|
||||
expect(recordedClass(runnerResult('error_api'), { failure_class: 'assertion' })).toBe('assertion');
|
||||
});
|
||||
@@ -260,7 +260,7 @@ describeE2E('/plan-ceo-review mode routing (gate)', () => {
|
||||
}
|
||||
const currentInput = await session.currentScreen();
|
||||
capture('awaiting_posture', currentInput, transcript);
|
||||
const modeSubmit = ceoModeSubmissionInput(currentInput, question.nativeCall, c.mode, transcript, submittedModePackets);
|
||||
const modeSubmit = ceoModeSubmissionInput(currentInput, question.nativeCall, c.mode, transcript, submittedModePackets, session.visibleText());
|
||||
if (modeSubmit !== null) { session.send(modeSubmit); continue; }
|
||||
const pendingQuestion = readPendingQuestion(session.pendingQuestionFile, fixture.cwd,
|
||||
session.hermeticConfigDir, selectionStartedAt, transcript);
|
||||
|
||||
@@ -35,6 +35,7 @@ import {
|
||||
engStep0Boundary,
|
||||
engSetupAUQ,
|
||||
engFirstReviewAUQ,
|
||||
hasCompletePlanReport,
|
||||
} from './helpers/claude-pty-runner';
|
||||
import { FORCING_BATCHING_ENG } from './fixtures/forcing-finding-seeds';
|
||||
import { createEngBatchingIssueCounter } from './helpers/eng-seeded-coverage';
|
||||
@@ -53,6 +54,7 @@ describeE2E('/plan-eng-review multi-finding batching regression (periodic)', ()
|
||||
test(
|
||||
`4-finding plan emits >= ${FLOOR} review-phase AskUserQuestions (no batching)`,
|
||||
async () => {
|
||||
const startedAt = Date.now();
|
||||
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-e2e-plan-eng-batching-'));
|
||||
const planPath = path.join(tmpDir, 'gstack-test-plan-eng-batching.md');
|
||||
const followUpPrompt = FORCING_BATCHING_ENG.replaceAll(FIXTURE_PLAN_PATH, planPath);
|
||||
@@ -85,8 +87,12 @@ describeE2E('/plan-eng-review multi-finding batching regression (periodic)', ()
|
||||
// review decisions exist, a batching regression can no longer occur
|
||||
// in this attempt; stop instead of letting the review run to the
|
||||
// ceiling (run 36385945043: floor at 6m41s, ceiling at 12m13s).
|
||||
// A completed report ends the review, so the count is final there:
|
||||
// grade it now rather than waiting out the session (run 36606688266
|
||||
// wrote its report at 1,248 s and closed at 1,318 s).
|
||||
isCollectionComplete: (_transcript, fingerprints) =>
|
||||
fingerprints.filter(fp => !fp.preReview && !fp.administrative).length >= FLOOR,
|
||||
fingerprints.filter(fp => !fp.preReview && !fp.administrative).length >= FLOOR ||
|
||||
hasCompletePlanReport(planPath, startedAt, Date.now()),
|
||||
reviewCountCeiling: N + 3, // hard cap above floor + tolerance
|
||||
// Supplied prerequisites: routing setup and cross-project learnings are
|
||||
// already declined, so the attempt starts at the review (setup answers
|
||||
|
||||
@@ -8,6 +8,7 @@ import {
|
||||
createEvalCollector, finalizeEvalCollector,
|
||||
} from './helpers/e2e-helpers';
|
||||
import { extractSkillSections, REVIEW_E2E_SECTIONS } from './helpers/skill-fixture';
|
||||
import { expectContract } from './helpers/eval-store';
|
||||
import { spawnSync } from 'child_process';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
@@ -291,7 +292,8 @@ Important: The design checklist should catch issues like blacklisted fonts, smal
|
||||
|
||||
console.log(`Design review detected ${detected}/7 planted checklist signals; detector rows surfaced: ${detectorSeen}`);
|
||||
expect(detected).toBeGreaterThanOrEqual(4); // the LLM-checklist bar, unchanged by the detector
|
||||
expect(detectorSeen).toBe(true); // the fake engine's rows are deterministic; the review must carry them
|
||||
// The fake engine's rows are deterministic; carrying them is the contract.
|
||||
expectContract(detectorSeen, 'review-design-lite: the review omitted the mechanical detector rows', { collector: evalCollector, name: '/review design lite' });
|
||||
}
|
||||
}, CAPTURE_MS + REVIEW_FINALIZE_MS);
|
||||
});
|
||||
|
||||
@@ -4,7 +4,7 @@ import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
import { CAPTURE_LONG_MS } from './helpers/eval-budgets';
|
||||
import { describeE2ETier, e2eTierEnabled } from './helpers/e2e-gate';
|
||||
import { EvalCollector } from './helpers/eval-store';
|
||||
import { EvalCollector, expectContract } from './helpers/eval-store';
|
||||
import { sharedLibsPlanExcerpt } from './helpers/shared-libs-plan-excerpt';
|
||||
import { createSharedPlanReuseSelector } from './helpers/shared-libs-plan-actor';
|
||||
import {
|
||||
@@ -51,17 +51,24 @@ async function assertJudgment(report: string, criteria: Record<string, string>)
|
||||
for (const key of Object.keys(criteria)) expect(judgment.checks[key], `${key}: ${judgment.reasoning}`).toBe(true);
|
||||
}
|
||||
|
||||
function assertReadOnly(f: SharedLibsFixture, before: Record<string, string>, result: any) {
|
||||
function assertReadOnly(f: SharedLibsFixture, before: Record<string, string>, result: any, name: string) {
|
||||
// Read-only is the contract of every audit, even where the recommendation
|
||||
// itself is a tolerated judgment call.
|
||||
const record = { collector, name };
|
||||
result.providerRequests = readRequests(f);
|
||||
expect(sharedReadOnlyViolations(result.toolCalls, result.providerRequests)).toEqual([]);
|
||||
const violations = sharedReadOnlyViolations(result.toolCalls, result.providerRequests);
|
||||
expectContract(violations.length === 0, `read-only: disallowed commands or provider requests ${JSON.stringify(violations)}`, record);
|
||||
const expected = { ...before }, after = snapshotFixture(f.root);
|
||||
// Only the source-provider instrumentation can change. Snapshot the outer
|
||||
// fixture as well as the repository; inspect commands for writes beyond it.
|
||||
delete expected[path.relative(f.root, f.trace)];
|
||||
delete after[path.relative(f.root, f.trace)];
|
||||
expect(after).toEqual(expected);
|
||||
expect(fs.existsSync(f.hookTrace) ? fs.readFileSync(f.hookTrace, 'utf8') : '').toBe('');
|
||||
expect(fs.readdirSync(f.state)).toEqual([]);
|
||||
const changed = [...new Set([...Object.keys(expected), ...Object.keys(after)])].filter(file => expected[file] !== after[file]);
|
||||
expectContract(changed.length === 0, `read-only: fixture files changed: ${changed.join(', ')}`, record);
|
||||
const hookTrace = fs.existsSync(f.hookTrace) ? fs.readFileSync(f.hookTrace, 'utf8') : '';
|
||||
expectContract(hookTrace === '', `read-only: a configured hook ran: ${hookTrace.slice(0, 500)}`, record);
|
||||
const state = fs.readdirSync(f.state);
|
||||
expectContract(state.length === 0, `read-only: gstack state written: ${state.join(', ')}`, record);
|
||||
}
|
||||
|
||||
describeE2E('Shared-code opportunity and coordination judgment (periodic)', () => {
|
||||
@@ -77,7 +84,7 @@ describeE2E('Shared-code opportunity and coordination judgment (periodic)', () =
|
||||
const before = snapshotFixture(f.root);
|
||||
await judgedCapture(attempt, 'empty', 'shared-libs-opportunity-judgment', () => runSharedCapture(f, 'shared-libs-opportunity-judgment',
|
||||
`Run /deslop-shared-libs using ${instructions} and return the report.`, attempt), async result => {
|
||||
assertReadOnly(f, before, result);
|
||||
assertReadOnly(f, before, result, 'shared-libs-opportunity-judgment');
|
||||
await assertJudgment(result.output, {
|
||||
valid_empty: 'There is only README, .gitignore and a unique one-line src/version.ts, and successful empty PR results. It reports no worthwhile sharing opportunities and does not fabricate callers or blame unavailable history/API access.',
|
||||
});
|
||||
@@ -110,7 +117,7 @@ describeE2E('Shared-code opportunity and coordination judgment (periodic)', () =
|
||||
const before = snapshotFixture(f.root);
|
||||
await judgedCapture(attempt, 'opportunity', 'shared-libs-opportunity-judgment', () => runSharedCapture(f, 'shared-libs-opportunity-judgment',
|
||||
`Run /deslop-shared-libs using ${instructions}. Review the active TypeScript and Python areas and return the requested report.`, attempt), async result => {
|
||||
assertReadOnly(f, before, result);
|
||||
assertReadOnly(f, before, result, 'shared-libs-opportunity-judgment');
|
||||
expect(result.output).toContain(f.tip.slice(0, 7));
|
||||
expect(result.output).toContain('lib/retry-after.ts');
|
||||
expect(JSON.stringify(result.transcript ?? result.toolCalls)).toMatch(/inventory\.py|search\.py|negative inventory|src\/\*\.py/);
|
||||
@@ -143,7 +150,7 @@ describeE2E('Shared-code opportunity and coordination judgment (periodic)', () =
|
||||
const before = snapshotFixture(f.root);
|
||||
await judgedCapture(attempt, 'audit', 'shared-libs-pr-coverage', () => runSharedCapture(f, 'shared-libs-pr-coverage',
|
||||
`Run /deslop-shared-libs using ${instructions}. Recent PR 7 mentions https://github.com/fixture/shared-libs/pull/42 as related work. Return the report after checking coordination within the skill's budget.`, attempt), async result => {
|
||||
assertReadOnly(f, before, result);
|
||||
assertReadOnly(f, before, result, 'shared-libs-pr-coverage');
|
||||
const requests = readRequests(f).filter(row => row.tool === 'gh' || row.tool === 'curl');
|
||||
const endpoints = requests.map(row => row.endpoint || '');
|
||||
expect(endpoints.some(endpoint => /\/pulls\/42\/files/.test(endpoint))).toBe(true);
|
||||
|
||||
Reference in new issue
Block a user