gstack/test/fixtures/eng-batching-bullet-target-rerun.json

{
 "source": "local targeted rerun smoke-2.1.284-1790711269 (Claude Code 2.1.284) of plan-eng-multi-finding-batching: observation.json transcript.calls and the saved report replayed from its Write/Edit inputs",
 "recordedOutcome": {
  "outcome": "collection_complete",
  "step0Count": 11,
  "reviewCount": 0
 },
 "calls": [
  {
   "sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
   "toolUseId": "toolu_01XnWRrh4F44QdznytAGmriy",
   "questions": [
    {
     "question": "D1 \u2014 Reuse the job library's retry hooks or roll a custom scheduler?\nProject/branch/task: main branch, retry-framework plan; adding retries to 5 background workers.\nELI10: The job library you already use has retry hooks built in, and your plan says your custom version would be \"the same shape.\" Building your own copy inside each worker means five hand-written schedulers to keep correct, versus configuring one curve the library already knows how to run. The plan's reason for going custom is \"full control over the curve,\" and most retry hook APIs give you that through a backoff callback.\nStakes if we pick wrong: five bespoke schedulers drift apart, each grows its own bugs (no jitter, no cap, retry storms), and nobody at 3am knows which curve a given worker actually runs.\nRecommendation: A because the plan admits the library version has the same shape, and a custom curve is usually a config callback, not a new scheduler.\nCompleteness: A=9/10, B=5/10, C=n/a (investigation, decides nothing)\nPros / cons:\nA) Library hooks + custom curve (recommended)\n  \u2705 One scheduler the library already tests; the curve becomes a per-worker config function (human: ~1 day / CC: ~20 min)\n  \u2705 Attempt-count persistence, max attempts and dead-letter usually come along for free with the hooks\n  \u274c If the hook API cannot accept an arbitrary curve function, that is a new fact and R1 reopens\nB) Custom inline scheduler (as planned)\n  \u2705 Total control of delay math and logging, independent of the library's release cycle\n  \u2705 No dependency on hook semantics nobody has read yet (human: ~3 days / CC: ~1 hr)\n  \u274c Five hand-rolled schedulers to keep in sync, plus reimplementing attempt persistence and terminal handling\nC) Investigate first: bounded read of the hook API\n  \u2705 Settles \"can the hooks take a custom curve\" with evidence before committing either way\n  \u2705 Cheap: ~30 min human / ~3 min CC reading the hook signature and docs\n  \u274c Decides nothing; R1 stays pending and the review pauses on this question\nNet: You are trading a library dependency you already carry for five copies of scheduler code you would own forever.",
     "header": "Retry mechanism",
     "multiSelect": false,
     "options": [
      {
       "label": "Library hooks + custom curve (recommended)",
       "description": "Use the job library's built-in retry hooks; supply each worker's backoff curve as a callback/config. One scheduler the library already tests. Attempt persistence, max attempts and dead-letter usually included. Human ~1 day / CC ~20 min. Risk: if the hook API cannot take an arbitrary curve, R1 reopens. Completeness 9/10."
      },
      {
       "label": "Custom inline scheduler (as planned)",
       "description": "Roll the exponential-backoff scheduler inline in each of the 5 workers as PLAN.md:7-9 proposes. Total control of delay math and logging. Human ~3 days / CC ~1 hr. Cost: five schedulers to keep in sync, plus attempt persistence and terminal handling rebuilt by hand. Completeness 5/10."
      },
      {
       "label": "Investigate hook API first",
       "description": "Bounded ~30 min human / ~3 min CC read of the library's retry hook signature and docs, then return to this question. Approves nothing; R1 stays pending; R2\u2013R6 unchanged."
      }
     ]
    }
   ],
   "answered": true,
   "failed": false,
   "answers": {
    "D1 \u2014 Reuse the job library's retry hooks or roll a custom scheduler?\nProject/branch/task: main branch, retry-framework plan; adding retries to 5 background workers.\nELI10: The job library you already use has retry hooks built in, and your plan says your custom version would be \"the same shape.\" Building your own copy inside each worker means five hand-written schedulers to keep correct, versus configuring one curve the library already knows how to run. The plan's reason for going custom is \"full control over the curve,\" and most retry hook APIs give you that through a backoff callback.\nStakes if we pick wrong: five bespoke schedulers drift apart, each grows its own bugs (no jitter, no cap, retry storms), and nobody at 3am knows which curve a given worker actually runs.\nRecommendation: A because the plan admits the library version has the same shape, and a custom curve is usually a config callback, not a new scheduler.\nCompleteness: A=9/10, B=5/10, C=n/a (investigation, decides nothing)\nPros / cons:\nA) Library hooks + custom curve (recommended)\n  \u2705 One scheduler the library already tests; the curve becomes a per-worker config function (human: ~1 day / CC: ~20 min)\n  \u2705 Attempt-count persistence, max attempts and dead-letter usually come along for free with the hooks\n  \u274c If the hook API cannot accept an arbitrary curve function, that is a new fact and R1 reopens\nB) Custom inline scheduler (as planned)\n  \u2705 Total control of delay math and logging, independent of the library's release cycle\n  \u2705 No dependency on hook semantics nobody has read yet (human: ~3 days / CC: ~1 hr)\n  \u274c Five hand-rolled schedulers to keep in sync, plus reimplementing attempt persistence and terminal handling\nC) Investigate first: bounded read of the hook API\n  \u2705 Settles \"can the hooks take a custom curve\" with evidence before committing either way\n  \u2705 Cheap: ~30 min human / ~3 min CC reading the hook signature and docs\n  \u274c Decides nothing; R1 stays pending and the review pauses on this question\nNet: You are trading a library dependency you already carry for five copies of scheduler code you would own forever.": "Library hooks + custom curve (recommended)"
   },
   "unansweredQuestionIndices": [],
   "answeredAt": "2026-09-29T19:51:49.159Z"
  },
  {
   "sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
   "toolUseId": "toolu_015Ri9YxuhvdexzxG5KqBTTc",
   "questions": [
    {
     "question": "D2 \u2014 What delivery guarantee does processWebhookJob() keep once it can retry?\nProject/branch/task: main branch, retry-framework plan; the webhook worker is one of the 5 workers gaining retries through library hooks (D1).\nELI10: Today the webhook worker sends each event at most once: if the send fails or times out, the event is dropped, never duplicated. A retry cannot tell \"the request never arrived\" apart from \"it arrived but the response got lost,\" so any retry after a timeout can deliver the same event twice. Adding retries silently flips the guarantee from at-most-once to at-least-once. That is a contract change your webhook receivers depend on, and the plan does not name it.\nStakes if we pick wrong: receivers that are not idempotent process duplicate events (double emails, double charges, double state transitions); or, if we keep dropping on ambiguity, the retry framework never fixes the webhook worker's lost events.\nRecommendation: B because losing events is usually worse than duplicates, and a stable idempotency key makes duplicates safe for receivers; this is still a contract call you know better than the review does.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nPros / cons:\nA) Keep at-most-once: retry only provably-unsent failures\n  \u2705 No duplicate deliveries ever; existing receivers keep working with no change on their side\n  \u2705 Still recovers the clear cases: connection refused, DNS failure, local enqueue error (human: ~1 day / CC: ~30 min)\n  \u274c Timeouts and 5xx-after-send still drop events, so the biggest source of loss stays; needs per-attempt failure classification\nB) Move to at-least-once with a stable idempotency key (recommended)\n  \u2705 Every event eventually reaches the receiver; retries after timeouts are safe because the event id stays constant across attempts\n  \u2705 Matches Stripe, Shopify and Svix practice; receivers dedupe on the key (human: ~1.5 days / CC: ~30 min incl. docs)\n  \u274c Contract change: receivers must dedupe; needs a documented header, changelog entry and migration note for existing receivers\nC) Exclude processWebhookJob() from retries\n  \u2705 Zero semantic change for receivers; the other 4 workers still get retries\n  \u2705 Smallest diff and no receiver communication (human: ~1 hr / CC: ~5 min)\n  \u274c The webhook worker keeps losing events on every transient failure, which is likely why the plan touched it\nNet: Never-duplicate-but-lossy, never-lossy-but-receivers-must-dedupe, or leave the webhook worker exactly as it is.",
     "header": "Webhook delivery",
     "multiSelect": false,
     "options": [
      {
       "label": "Keep at-most-once (retry only pre-send failures)",
       "description": "Webhook worker retries only failures provably raised before the request was written (connection refused, DNS, local error). Timeouts and 5xx-after-send still drop the event. No duplicates; receivers unchanged. Needs per-attempt failure classification. Human ~1 day / CC ~30 min."
      },
      {
       "label": "At-least-once + idempotency key (recommended)",
       "description": "Webhook worker retries all transient failures; every delivery carries a stable event id / idempotency key header constant across attempts. Documented contract change with changelog and receiver migration note. Receivers dedupe on the key. Human ~1.5 days / CC ~30 min."
      },
      {
       "label": "Exclude webhook worker from retries",
       "description": "`processWebhookJob()` keeps today's at-most-once, no-retry behavior; the other 4 workers retry via library hooks per D1. Smallest diff, no receiver impact, webhook events still lost on transient failure. Human ~1 hr / CC ~5 min."
      }
     ]
    }
   ],
   "answered": true,
   "failed": false,
   "answers": {
    "D2 \u2014 What delivery guarantee does processWebhookJob() keep once it can retry?\nProject/branch/task: main branch, retry-framework plan; the webhook worker is one of the 5 workers gaining retries through library hooks (D1).\nELI10: Today the webhook worker sends each event at most once: if the send fails or times out, the event is dropped, never duplicated. A retry cannot tell \"the request never arrived\" apart from \"it arrived but the response got lost,\" so any retry after a timeout can deliver the same event twice. Adding retries silently flips the guarantee from at-most-once to at-least-once. That is a contract change your webhook receivers depend on, and the plan does not name it.\nStakes if we pick wrong: receivers that are not idempotent process duplicate events (double emails, double charges, double state transitions); or, if we keep dropping on ambiguity, the retry framework never fixes the webhook worker's lost events.\nRecommendation: B because losing events is usually worse than duplicates, and a stable idempotency key makes duplicates safe for receivers; this is still a contract call you know better than the review does.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nPros / cons:\nA) Keep at-most-once: retry only provably-unsent failures\n  \u2705 No duplicate deliveries ever; existing receivers keep working with no change on their side\n  \u2705 Still recovers the clear cases: connection refused, DNS failure, local enqueue error (human: ~1 day / CC: ~30 min)\n  \u274c Timeouts and 5xx-after-send still drop events, so the biggest source of loss stays; needs per-attempt failure classification\nB) Move to at-least-once with a stable idempotency key (recommended)\n  \u2705 Every event eventually reaches the receiver; retries after timeouts are safe because the event id stays constant across attempts\n  \u2705 Matches Stripe, Shopify and Svix practice; receivers dedupe on the key (human: ~1.5 days / CC: ~30 min incl. docs)\n  \u274c Contract change: receivers must dedupe; needs a documented header, changelog entry and migration note for existing receivers\nC) Exclude processWebhookJob() from retries\n  \u2705 Zero semantic change for receivers; the other 4 workers still get retries\n  \u2705 Smallest diff and no receiver communication (human: ~1 hr / CC: ~5 min)\n  \u274c The webhook worker keeps losing events on every transient failure, which is likely why the plan touched it\nNet: Never-duplicate-but-lossy, never-lossy-but-receivers-must-dedupe, or leave the webhook worker exactly as it is.": "Keep at-most-once (retry only pre-send failures)"
   },
   "unansweredQuestionIndices": [],
   "answeredAt": "2026-09-29T19:54:01.993Z"
  },
  {
   "sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
   "toolUseId": "toolu_0132i9qkrzXUVyovu4GZxADn",
   "questions": [
    {
     "question": "D3 \u2014 When a job runs out of retries, where does it go?\nProject/branch/task: main branch, retry-framework plan; retry bounds for all 5 workers running through library hooks (D1).\nELI10: Right now the plan describes the curve between retries but never says how many retries there are or what happens to a job that keeps failing. Without a limit, a poisoned job retries forever and eats worker capacity. With a limit but no landing spot, the job disappears with one log line nobody reads. A dead-letter store keeps the failed job, its payload reference and its last error so someone can inspect and replay it.\nStakes if we pick wrong: either an infinite-retry job starves the queue, or real work silently vanishes after the last attempt and the first sign is a customer asking where their data went.\nRecommendation: A because a dead-letter store plus an alert is a few dozen lines with library hooks, and it turns \"job vanished\" into \"job parked, here is why.\"\nCompleteness: A=10/10, B=5/10, C=3/10\nPros / cons:\nA) Bounded attempts + dead-letter store + alert (recommended)\n  \u2705 Exhausted or fatal jobs are kept with last error and attempt history; operators can inspect and replay (human: ~1 day / CC: ~20 min)\n  \u2705 Metric and alert on dead-letter growth turns a silent failure into a page at the right time\n  \u274c One more table or queue to own, plus a small replay path to build and test\nB) Bounded attempts, log and drop\n  \u2705 Simplest bound: `maxAttempts` default 5 per worker, one error log on exhaustion (human: ~2 hr / CC: ~5 min)\n  \u2705 No new storage; nothing to operate\n  \u274c Exhausted jobs are gone; recovery means replaying from upstream sources by hand, if that is even possible\nC) Leave to library defaults\n  \u2705 Zero plan text and zero decision now\n  \u2705 Whatever the library does is at least consistent across the 5 workers\n  \u274c Nobody knows the limit or the terminal behavior until an incident teaches them; 3am failure mode\nNet: You are trading one small dead-letter store for never having to ask \"where did that job go.\"",
     "header": "Retry exhaustion",
     "multiSelect": false,
     "options": [
      {
       "label": "Bounded + dead-letter + alert (recommended)",
       "description": "`maxAttempts` default 5 with per-worker override. On exhaustion or fatal error the job lands in a dead-letter store (table or queue) with last error, attempt history and payload reference. Metric and alert on dead-letter growth. Manual replay path. Human ~1 day / CC ~20 min. Completeness 10/10."
      },
      {
       "label": "Bounded, log and drop",
       "description": "`maxAttempts` default 5 with per-worker override. On exhaustion, log at error level with the last error and drop the job. No new storage, no replay. Human ~2 hr / CC ~5 min. Completeness 5/10."
      },
      {
       "label": "Library defaults, unspecified",
       "description": "Do not write attempt limits or terminal behavior into the plan; accept whatever the library does by default. Completeness 3/10."
      }
     ]
    }
   ],
   "answered": true,
   "failed": false,
   "answers": {
    "D3 \u2014 When a job runs out of retries, where does it go?\nProject/branch/task: main branch, retry-framework plan; retry bounds for all 5 workers running through library hooks (D1).\nELI10: Right now the plan describes the curve between retries but never says how many retries there are or what happens to a job that keeps failing. Without a limit, a poisoned job retries forever and eats worker capacity. With a limit but no landing spot, the job disappears with one log line nobody reads. A dead-letter store keeps the failed job, its payload reference and its last error so someone can inspect and replay it.\nStakes if we pick wrong: either an infinite-retry job starves the queue, or real work silently vanishes after the last attempt and the first sign is a customer asking where their data went.\nRecommendation: A because a dead-letter store plus an alert is a few dozen lines with library hooks, and it turns \"job vanished\" into \"job parked, here is why.\"\nCompleteness: A=10/10, B=5/10, C=3/10\nPros / cons:\nA) Bounded attempts + dead-letter store + alert (recommended)\n  \u2705 Exhausted or fatal jobs are kept with last error and attempt history; operators can inspect and replay (human: ~1 day / CC: ~20 min)\n  \u2705 Metric and alert on dead-letter growth turns a silent failure into a page at the right time\n  \u274c One more table or queue to own, plus a small replay path to build and test\nB) Bounded attempts, log and drop\n  \u2705 Simplest bound: `maxAttempts` default 5 per worker, one error log on exhaustion (human: ~2 hr / CC: ~5 min)\n  \u2705 No new storage; nothing to operate\n  \u274c Exhausted jobs are gone; recovery means replaying from upstream sources by hand, if that is even possible\nC) Leave to library defaults\n  \u2705 Zero plan text and zero decision now\n  \u2705 Whatever the library does is at least consistent across the 5 workers\n  \u274c Nobody knows the limit or the terminal behavior until an incident teaches them; 3am failure mode\nNet: You are trading one small dead-letter store for never having to ask \"where did that job go.\"": "Bounded + dead-letter + alert (recommended)"
   },
   "unansweredQuestionIndices": [],
   "answeredAt": "2026-09-29T19:55:16.802Z"
  },
  {
   "sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
   "toolUseId": "toolu_01GMRg1vxVPnVCefhfuCSKEw",
   "questions": [
    {
     "question": "D4 \u2014 Should retry delays be randomized (jitter)?\nProject/branch/task: main branch, retry-framework plan; the backoff curve each worker supplies to the library hooks (D1).\nELI10: When many jobs fail at the same moment because a shared dependency went down, a pure exponential curve makes them all retry at the same moments too, so the recovering dependency gets hit by a wave on every step. Jitter randomizes each job's delay so the retries spread out. It is one line inside the curve callback each worker already supplies.\nStakes if we pick wrong: synchronized retry waves knock a recovering dependency back over (the classic thundering herd); or, with jitter, per-job retry timing becomes slightly less predictable and tests need a seeded random source.\nRecommendation: A because full jitter gives the least contention in AWS's published analysis and costs one line in a callback you are already writing.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nPros / cons:\nA) Full jitter: random(0, exponentialDelay) (recommended)\n  \u2705 Best spread of retries and lowest total contention after a shared outage (AWS Builders' Library)\n  \u2705 One line inside the D1 curve callback; RNG injected so tests stay deterministic (human: ~1 hr / CC: ~5 min)\n  \u274c An individual retry can fire almost immediately; minimum wait is not guaranteed\nB) Equal jitter: half fixed, half random\n  \u2705 Guarantees a minimum wait of half the exponential delay while still spreading retries\n  \u2705 Same one-line cost and same injectable RNG as full jitter (human: ~1 hr / CC: ~5 min)\n  \u274c Slightly more contention than full jitter in the same analysis, and one more parameter to explain\nC) No jitter: deterministic curve\n  \u2705 Fully deterministic; trivial to reason about and to assert exact delays in tests\n  \u2705 Zero extra code beyond the exponential curve\n  \u274c Every job that failed together retries together; retry storms on recovery are the expected outcome\nNet: One random() call now versus a synchronized retry wave the first time a dependency has a bad hour.",
     "header": "Jitter",
     "multiSelect": false,
     "options": [
      {
       "label": "Full jitter (recommended)",
       "description": "delay = random(0, exponentialDelay) inside each worker's curve callback. Best spread, lowest contention. RNG injectable so tests are deterministic. Human ~1 hr / CC ~5 min."
      },
      {
       "label": "Equal jitter",
       "description": "delay = exponentialDelay/2 + random(0, exponentialDelay/2). Guarantees a minimum wait; slightly more contention than full jitter. RNG injectable. Human ~1 hr / CC ~5 min."
      },
      {
       "label": "No jitter",
       "description": "Deterministic exponential curve, no randomization. Simplest to test; retries synchronize after a shared outage."
      }
     ]
    }
   ],
   "answered": true,
   "failed": false,
   "answers": {
    "D4 \u2014 Should retry delays be randomized (jitter)?\nProject/branch/task: main branch, retry-framework plan; the backoff curve each worker supplies to the library hooks (D1).\nELI10: When many jobs fail at the same moment because a shared dependency went down, a pure exponential curve makes them all retry at the same moments too, so the recovering dependency gets hit by a wave on every step. Jitter randomizes each job's delay so the retries spread out. It is one line inside the curve callback each worker already supplies.\nStakes if we pick wrong: synchronized retry waves knock a recovering dependency back over (the classic thundering herd); or, with jitter, per-job retry timing becomes slightly less predictable and tests need a seeded random source.\nRecommendation: A because full jitter gives the least contention in AWS's published analysis and costs one line in a callback you are already writing.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nPros / cons:\nA) Full jitter: random(0, exponentialDelay) (recommended)\n  \u2705 Best spread of retries and lowest total contention after a shared outage (AWS Builders' Library)\n  \u2705 One line inside the D1 curve callback; RNG injected so tests stay deterministic (human: ~1 hr / CC: ~5 min)\n  \u274c An individual retry can fire almost immediately; minimum wait is not guaranteed\nB) Equal jitter: half fixed, half random\n  \u2705 Guarantees a minimum wait of half the exponential delay while still spreading retries\n  \u2705 Same one-line cost and same injectable RNG as full jitter (human: ~1 hr / CC: ~5 min)\n  \u274c Slightly more contention than full jitter in the same analysis, and one more parameter to explain\nC) No jitter: deterministic curve\n  \u2705 Fully deterministic; trivial to reason about and to assert exact delays in tests\n  \u2705 Zero extra code beyond the exponential curve\n  \u274c Every job that failed together retries together; retry storms on recovery are the expected outcome\nNet: One random() call now versus a synchronized retry wave the first time a dependency has a bad hour.": "Full jitter (recommended)"
   },
   "unansweredQuestionIndices": [],
   "answeredAt": "2026-09-29T19:56:30.842Z"
  },
  {
   "sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
   "toolUseId": "toolu_01XLVRABVWXpSZRBNni2esSC",
   "questions": [
    {
     "question": "D5 \u2014 Should the backoff delay have a ceiling?\nProject/branch/task: main branch, retry-framework plan; the curve parameters each worker passes to the library hooks (D1, jittered per D4).\nELI10: Exponential backoff doubles the wait after each failure. That is fine for 5 attempts, but D3 lets each worker raise its attempt count, and a worker set to 15 attempts from a 1 second base would wait about 4.5 hours before its last try; at 20 attempts it would wait 6 days. A cap says \"never wait longer than X between attempts,\" so the curve grows and then flattens. It is one min() call in the callback.\nStakes if we pick wrong: without a cap, a worker with a higher attempt count silently turns into a multi-day wait that looks like a stuck job; with a cap, one more number to document per worker.\nRecommendation: A because the cap is one min() and it makes \"how long can this job be delayed\" a question with an answer.\nCompleteness: A=9/10, B=4/10\nPros / cons:\nA) Cap each delay: default 10 min, per-worker override (recommended)\n  \u2705 Worst-case wait between attempts is bounded and documented for every worker (human: ~1 hr / CC: ~5 min)\n  \u2705 Also pins the curve defaults (base 1 s, multiplier 2) so all 5 workers start from the same documented numbers\n  \u274c One more config value per worker to document and keep sane alongside maxAttempts\nB) No cap\n  \u2705 Zero code; the curve is exactly the exponential the plan describes\n  \u2705 Fewer knobs to explain\n  \u274c Any worker that raises maxAttempts past ~12 gets hour-to-day waits nobody intended\nNet: One min() now versus a job that looks stuck for six days the first time someone bumps an attempt count.",
     "header": "Delay cap",
     "multiSelect": false,
     "options": [
      {
       "label": "Cap each delay (recommended)",
       "description": "delay = min(jitteredExponential, maxDelay). `maxDelay` default 10 minutes with per-worker override. Curve defaults documented: base 1 s, multiplier 2, per-worker override. Human ~1 hr / CC ~5 min. Completeness 9/10."
      },
      {
       "label": "No cap",
       "description": "Raw exponential curve with no ceiling. Zero code, fewer knobs; high attempt counts produce hour-to-day waits. Completeness 4/10."
      }
     ]
    }
   ],
   "answered": true,
   "failed": false,
   "answers": {
    "D5 \u2014 Should the backoff delay have a ceiling?\nProject/branch/task: main branch, retry-framework plan; the curve parameters each worker passes to the library hooks (D1, jittered per D4).\nELI10: Exponential backoff doubles the wait after each failure. That is fine for 5 attempts, but D3 lets each worker raise its attempt count, and a worker set to 15 attempts from a 1 second base would wait about 4.5 hours before its last try; at 20 attempts it would wait 6 days. A cap says \"never wait longer than X between attempts,\" so the curve grows and then flattens. It is one min() call in the callback.\nStakes if we pick wrong: without a cap, a worker with a higher attempt count silently turns into a multi-day wait that looks like a stuck job; with a cap, one more number to document per worker.\nRecommendation: A because the cap is one min() and it makes \"how long can this job be delayed\" a question with an answer.\nCompleteness: A=9/10, B=4/10\nPros / cons:\nA) Cap each delay: default 10 min, per-worker override (recommended)\n  \u2705 Worst-case wait between attempts is bounded and documented for every worker (human: ~1 hr / CC: ~5 min)\n  \u2705 Also pins the curve defaults (base 1 s, multiplier 2) so all 5 workers start from the same documented numbers\n  \u274c One more config value per worker to document and keep sane alongside maxAttempts\nB) No cap\n  \u2705 Zero code; the curve is exactly the exponential the plan describes\n  \u2705 Fewer knobs to explain\n  \u274c Any worker that raises maxAttempts past ~12 gets hour-to-day waits nobody intended\nNet: One min() now versus a job that looks stuck for six days the first time someone bumps an attempt count.": "Cap each delay (recommended)"
   },
   "unansweredQuestionIndices": [],
   "answeredAt": "2026-09-29T19:57:33.114Z"
  },
  {
   "sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
   "toolUseId": "toolu_01SrWycxMxofjLp1hxkciPj9",
   "questions": [
    {
     "question": "D6 \u2014 Which errors should the four non-webhook workers retry, and which go straight to dead-letter?\nProject/branch/task: main branch, retry-framework plan; error handling inside the 4 non-webhook workers' library retry hooks (D1). The webhook worker's rule is already fixed by D2.\nELI10: Not every failure is worth retrying. A timeout or a \"service busy\" reply will likely pass on the next try. A validation error, a missing record or a bug that throws will fail the same way five times in a row, burning worker time and delaying the dead-letter record (D3) by the whole backoff curve. Classifying errors sends the hopeless ones to dead-letter immediately and spends retries only on the ones that can recover. The open question is what to do with an error nobody has classified yet.\nStakes if we pick wrong: either a bug retries five times per job across a whole queue before anyone sees it, or a transient error that nobody thought to list dead-letters real work on its first failure.\nRecommendation: A because classification is a short list per worker, and defaulting unknown errors to retryable never loses work: the worst case is five wasted attempts, not a dropped job.\nCompleteness: A=10/10, B=5/10, C=8/10\nPros / cons:\nA) Classify; unknown errors retry (recommended)\n  \u2705 Hopeless errors (validation, 4xx, missing record, TypeError) land in dead-letter on attempt 1 with the real cause visible (human: ~half day / CC: ~15 min)\n  \u2705 Unlisted errors still retry, so a forgotten transient class costs attempts, never data\n  \u274c Each worker maintains a small error-class list, and a new fatal class retries needlessly until someone adds it\nB) Retry everything until maxAttempts\n  \u2705 No lists to maintain; identical behavior in all 4 workers (human: ~0 / CC: ~0)\n  \u2705 Impossible to misclassify a transient error as fatal\n  \u274c A deploy with a bug retries every affected job 5 times over the full curve before dead-lettering; queue capacity burns and diagnosis is delayed\nC) Classify; unknown errors are fatal\n  \u2705 Zero wasted attempts on anything not explicitly known to be transient (human: ~half day / CC: ~15 min)\n  \u2705 Dead-letter fills fast, so new error classes surface quickly\n  \u274c Any transient error missing from the list dead-letters real work on its first failure, which is the exact loss the retry framework exists to prevent\nNet: A short list per worker plus a safe default, versus either wasted retries on bugs or lost work on unlisted transients.",
     "header": "Error classes",
     "multiSelect": false,
     "options": [
      {
       "label": "Classify; unknown \u2192 retryable (recommended)",
       "description": "Each of the 4 workers declares retryable classes (timeout, connection error, 429/503, deadlock/serialization failure) and fatal classes (validation error, 4xx other than 429, missing record, programming error). Fatal \u2192 dead-letter immediately without consuming attempts. Unclassified errors retry. Human ~half day / CC ~15 min. Completeness 10/10."
      },
      {
       "label": "Retry everything to maxAttempts",
       "description": "No classification. Every error consumes an attempt until `maxAttempts`, then dead-letter per D3. Zero code; bugs retry 5 times per job. Completeness 5/10."
      },
      {
       "label": "Classify; unknown \u2192 fatal",
       "description": "Same declared classes as A, but unclassified errors go to dead-letter immediately. No wasted attempts; unlisted transient errors lose work on first failure. Human ~half day / CC ~15 min. Completeness 8/10."
      }
     ]
    }
   ],
   "answered": true,
   "failed": false,
   "answers": {
    "D6 \u2014 Which errors should the four non-webhook workers retry, and which go straight to dead-letter?\nProject/branch/task: main branch, retry-framework plan; error handling inside the 4 non-webhook workers' library retry hooks (D1). The webhook worker's rule is already fixed by D2.\nELI10: Not every failure is worth retrying. A timeout or a \"service busy\" reply will likely pass on the next try. A validation error, a missing record or a bug that throws will fail the same way five times in a row, burning worker time and delaying the dead-letter record (D3) by the whole backoff curve. Classifying errors sends the hopeless ones to dead-letter immediately and spends retries only on the ones that can recover. The open question is what to do with an error nobody has classified yet.\nStakes if we pick wrong: either a bug retries five times per job across a whole queue before anyone sees it, or a transient error that nobody thought to list dead-letters real work on its first failure.\nRecommendation: A because classification is a short list per worker, and defaulting unknown errors to retryable never loses work: the worst case is five wasted attempts, not a dropped job.\nCompleteness: A=10/10, B=5/10, C=8/10\nPros / cons:\nA) Classify; unknown errors retry (recommended)\n  \u2705 Hopeless errors (validation, 4xx, missing record, TypeError) land in dead-letter on attempt 1 with the real cause visible (human: ~half day / CC: ~15 min)\n  \u2705 Unlisted errors still retry, so a forgotten transient class costs attempts, never data\n  \u274c Each worker maintains a small error-class list, and a new fatal class retries needlessly until someone adds it\nB) Retry everything until maxAttempts\n  \u2705 No lists to maintain; identical behavior in all 4 workers (human: ~0 / CC: ~0)\n  \u2705 Impossible to misclassify a transient error as fatal\n  \u274c A deploy with a bug retries every affected job 5 times over the full curve before dead-lettering; queue capacity burns and diagnosis is delayed\nC) Classify; unknown errors are fatal\n  \u2705 Zero wasted attempts on anything not explicitly known to be transient (human: ~half day / CC: ~15 min)\n  \u2705 Dead-letter fills fast, so new error classes surface quickly\n  \u274c Any transient error missing from the list dead-letters real work on its first failure, which is the exact loss the retry framework exists to prevent\nNet: A short list per worker plus a safe default, versus either wasted retries on bugs or lost work on unlisted transients.": "Classify; unknown \u2192 retryable (recommended)"
   },
   "unansweredQuestionIndices": [],
   "answeredAt": "2026-09-29T19:58:40.484Z"
  },
  {
   "sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
   "toolUseId": "toolu_013bXU7e1WUvhP4r6Nh2cigC",
   "questions": [
    {
     "question": "D7 \u2014 One shared retry-policy module, or five copies and \"refactor later\"?\nProject/branch/task: main branch, retry-framework plan; how the 5 workers carry the behavior approved in D3\u2013D6.\nELI10: After D1 the library does the scheduling, but every worker still has to hand it the same four things: a jittered, capped curve, an error classifier, a dead-letter handoff and an attempt log line. The plan copies that block into five files and promises to clean up later. \"Later\" for copy-pasted retry code usually means the fifth copy drifts (no cap, wrong jitter) and nobody notices until an incident. The alternative is one small module that each worker configures with its own numbers and error lists.\nStakes if we pick wrong: five curves that silently disagree, five places to fix the next retry bug, and five test suites that each cover a slightly different subset; or, with a shared module, one bug that hits all five workers at once (mitigated by the module's own tests).\nRecommendation: A because the behavior is identical by construction (D3\u2013D6 fixed it), the module is under 100 lines, and it removes more lines than it adds while making the retry rules testable once.\nCompleteness: A=10/10, B=4/10, C=7/10\nPros / cons:\nA) One shared retry-policy module (recommended)\n  \u2705 Curve, classifier, dead-letter handoff, attempt log and config validation are tested once and behave the same in all 5 workers (human: ~1 day / CC: ~20 min)\n  \u2705 Estimated 15\u201390 implementation lines saved; the helper's test suite replaces five near-duplicate suites\n  \u274c A bug in the module reaches all 5 workers; the module's own tests are the guard\nB) Five inline copies, refactor later (as planned)\n  \u2705 No shared dependency between workers; each can be changed in isolation (human: ~1.5 days / CC: ~30 min)\n  \u2705 Matches the plan text exactly; nothing new to name or place\n  \u274c Five copies to keep in sync and five test suites to write; \"later\" rarely arrives for retry glue\nC) Extract the curve builder only\n  \u2705 The math most likely to drift (jitter + cap) lives in one place (human: ~1 day / CC: ~15 min)\n  \u2705 Smaller shared surface than A\n  \u274c Classifier, dead-letter handoff and log line are still copied five times, so most of the duplication and its tests remain\nNet: One under-100-line module now, or five copies plus a promise.",
     "header": "Shared module",
     "multiSelect": false,
     "options": [
      {
       "label": "One shared retry-policy module (recommended)",
       "description": "Small module: buildBackoff (base/multiplier/maxDelay/rng), classify (per-worker retryable/fatal lists), toDeadLetter (persist + metric), logAttempt, and startup config validation. All 5 workers use it with their own values. Human ~1 day / CC ~20 min. Completeness 10/10."
      },
      {
       "label": "Five inline copies (as planned)",
       "description": "Each worker carries its own curve, classifier, dead-letter handoff and log line; refactor deferred. Human ~1.5 days / CC ~30 min. Completeness 4/10."
      },
      {
       "label": "Extract curve builder only",
       "description": "Shared buildBackoff (jitter + cap) only; classifier, dead-letter handoff and log line stay inline in each of the 5 workers. Human ~1 day / CC ~15 min. Completeness 7/10."
      }
     ]
    }
   ],
   "answered": true,
   "failed": false,
   "answers": {
    "D7 \u2014 One shared retry-policy module, or five copies and \"refactor later\"?\nProject/branch/task: main branch, retry-framework plan; how the 5 workers carry the behavior approved in D3\u2013D6.\nELI10: After D1 the library does the scheduling, but every worker still has to hand it the same four things: a jittered, capped curve, an error classifier, a dead-letter handoff and an attempt log line. The plan copies that block into five files and promises to clean up later. \"Later\" for copy-pasted retry code usually means the fifth copy drifts (no cap, wrong jitter) and nobody notices until an incident. The alternative is one small module that each worker configures with its own numbers and error lists.\nStakes if we pick wrong: five curves that silently disagree, five places to fix the next retry bug, and five test suites that each cover a slightly different subset; or, with a shared module, one bug that hits all five workers at once (mitigated by the module's own tests).\nRecommendation: A because the behavior is identical by construction (D3\u2013D6 fixed it), the module is under 100 lines, and it removes more lines than it adds while making the retry rules testable once.\nCompleteness: A=10/10, B=4/10, C=7/10\nPros / cons:\nA) One shared retry-policy module (recommended)\n  \u2705 Curve, classifier, dead-letter handoff, attempt log and config validation are tested once and behave the same in all 5 workers (human: ~1 day / CC: ~20 min)\n  \u2705 Estimated 15\u201390 implementation lines saved; the helper's test suite replaces five near-duplicate suites\n  \u274c A bug in the module reaches all 5 workers; the module's own tests are the guard\nB) Five inline copies, refactor later (as planned)\n  \u2705 No shared dependency between workers; each can be changed in isolation (human: ~1.5 days / CC: ~30 min)\n  \u2705 Matches the plan text exactly; nothing new to name or place\n  \u274c Five copies to keep in sync and five test suites to write; \"later\" rarely arrives for retry glue\nC) Extract the curve builder only\n  \u2705 The math most likely to drift (jitter + cap) lives in one place (human: ~1 day / CC: ~15 min)\n  \u2705 Smaller shared surface than A\n  \u274c Classifier, dead-letter handoff and log line are still copied five times, so most of the duplication and its tests remain\nNet: One under-100-line module now, or five copies plus a promise.": "One shared retry-policy module (recommended)"
   },
   "unansweredQuestionIndices": [],
   "answeredAt": "2026-09-29T20:00:17.086Z"
  },
  {
   "sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
   "toolUseId": "toolu_01Ly1JT6sRQHG4ZVpQ7SUETD",
   "questions": [
    {
     "question": "D8 \u2014 How do we prove processWebhookJob() still sends each event at most once?\nProject/branch/task: main branch, retry-framework plan; regression coverage for the rewritten webhook worker (D2 fixed the behavior to keep).\nELI10: The webhook worker is being rewritten and it carries a promise to receivers: an event is never sent twice. D2 kept that promise while adding retries for failures that happen before anything is sent. A rewrite with no test for the promise means the first duplicate email or double charge is found by a customer. The test is straightforward: a fake receiver counts sends, and we assert the count is exactly one across every failure pattern. The question is how deep to go: assertions against the worker alone, a run through the real library hooks, or both.\nStakes if we pick wrong: a retry path nobody tested sends duplicates to non-idempotent receivers, or a hook wiring mistake means pre-send failures never actually retry and the framework quietly does nothing for webhooks.\nRecommendation: A because the unit layer pins each failure class cheaply and the integration layer is the only thing that catches hook wiring and attempt persistence, which is where retry bugs actually live.\nCompleteness: A=10/10, B=7/10, C=7/10\nPros / cons:\nA) Unit + integration through the library hooks (recommended)\n  \u2705 Every failure class (pre-send, timeout, 5xx, reset, success) asserted in isolation with a recording fake transport (human: ~1 day / CC: ~20 min)\n  \u2705 One end-to-end run through the real hooks with a fake receiver catches wiring and attempt-persistence bugs the unit layer cannot see\n  \u274c Two test layers to maintain; the integration test needs the library's test harness or an in-process queue\nB) Unit tests only\n  \u2705 Fast, deterministic, no queue infrastructure in the test run (human: ~half day / CC: ~10 min)\n  \u2705 Pins the classifier and the send-count contract per failure class\n  \u274c Never exercises the real hook registration, so a miswired hook passes tests and never retries in production\nC) Integration test only\n  \u2705 Exercises the real path receivers depend on (human: ~half day / CC: ~10 min)\n  \u2705 Fewer tests to write\n  \u274c Slower, and a failure tells you \"something duplicated\" without pointing at which failure class; edge classes get skipped for time\nNet: Cheap isolated assertions plus one real-path run, versus trusting either layer alone to protect a promise made to external receivers.",
     "header": "Webhook regression",
     "multiSelect": false,
     "options": [
      {
       "label": "Unit + integration (recommended)",
       "description": "Unit: fake transport records every send; assert exactly 1 send after pre-send retries, 0 further sends after timeout/5xx/reset with a dead-letter entry, 1 send on success. Integration: real library hooks + fake receiver, same assertions, plus attempt count survives a simulated worker restart. Human ~1 day / CC ~20 min. Completeness 10/10."
      },
      {
       "label": "Unit tests only",
       "description": "The unit assertions from A against the worker with a fake transport; no run through the real library hooks. Human ~half day / CC ~10 min. Completeness 7/10."
      },
      {
       "label": "Integration test only",
       "description": "The integration run from A only; no isolated per-failure-class assertions. Human ~half day / CC ~10 min. Completeness 7/10."
      }
     ]
    }
   ],
   "answered": true,
   "failed": false,
   "answers": {
    "D8 \u2014 How do we prove processWebhookJob() still sends each event at most once?\nProject/branch/task: main branch, retry-framework plan; regression coverage for the rewritten webhook worker (D2 fixed the behavior to keep).\nELI10: The webhook worker is being rewritten and it carries a promise to receivers: an event is never sent twice. D2 kept that promise while adding retries for failures that happen before anything is sent. A rewrite with no test for the promise means the first duplicate email or double charge is found by a customer. The test is straightforward: a fake receiver counts sends, and we assert the count is exactly one across every failure pattern. The question is how deep to go: assertions against the worker alone, a run through the real library hooks, or both.\nStakes if we pick wrong: a retry path nobody tested sends duplicates to non-idempotent receivers, or a hook wiring mistake means pre-send failures never actually retry and the framework quietly does nothing for webhooks.\nRecommendation: A because the unit layer pins each failure class cheaply and the integration layer is the only thing that catches hook wiring and attempt persistence, which is where retry bugs actually live.\nCompleteness: A=10/10, B=7/10, C=7/10\nPros / cons:\nA) Unit + integration through the library hooks (recommended)\n  \u2705 Every failure class (pre-send, timeout, 5xx, reset, success) asserted in isolation with a recording fake transport (human: ~1 day / CC: ~20 min)\n  \u2705 One end-to-end run through the real hooks with a fake receiver catches wiring and attempt-persistence bugs the unit layer cannot see\n  \u274c Two test layers to maintain; the integration test needs the library's test harness or an in-process queue\nB) Unit tests only\n  \u2705 Fast, deterministic, no queue infrastructure in the test run (human: ~half day / CC: ~10 min)\n  \u2705 Pins the classifier and the send-count contract per failure class\n  \u274c Never exercises the real hook registration, so a miswired hook passes tests and never retries in production\nC) Integration test only\n  \u2705 Exercises the real path receivers depend on (human: ~half day / CC: ~10 min)\n  \u2705 Fewer tests to write\n  \u274c Slower, and a failure tells you \"something duplicated\" without pointing at which failure class; edge classes get skipped for time\nNet: Cheap isolated assertions plus one real-path run, versus trusting either layer alone to protect a promise made to external receivers.": "Unit + integration (recommended)"
   },
   "unansweredQuestionIndices": [],
   "answeredAt": "2026-09-29T20:02:01.530Z"
  },
  {
   "sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
   "toolUseId": "toolu_01WZ2b5HaUXKgTwZxqfKn3pd",
   "questions": [
    {
     "question": "D9 \u2014 Cache the dependency graph across retries now, or measure first?\nProject/branch/task: main branch, retry-framework plan; per-attempt cost inside the 5 workers running through library hooks (D1).\nELI10: The plan worries that each retry reloads the job and rebuilds its dependency graph from scratch. After D1 the reload is just the library handing the job to the worker, which happens anyway. The rebuild is real extra CPU, but only on retries, and D3 caps those at 5 per failing job. Storing the graph on the first attempt would add a write to every job, including the large majority that succeed first time, to save work on the few that fail. Nobody has measured how long the rebuild takes.\nStakes if we pick wrong: either we add a write and a staleness risk to every job to fix a cost nobody measured, or a genuinely slow rebuild keeps burning worker time on retries and we only find out under load.\nRecommendation: C because an unmeasured optimization that taxes the happy path is the wrong trade; two timing metrics make the real decision cheap and data-driven.\nNote: options differ in kind (persisted cache vs in-process memo vs measure first) \u2014 no completeness score.\nPros / cons:\nA) Persist the graph with the job on attempt 1\n  \u2705 Retries never recompute; cost is paid once per job regardless of which worker instance retries (human: ~1 day / CC: ~20 min)\n  \u2705 Simple to reason about once the invalidation rule (payload version) is in place\n  \u274c Adds a write and stored blob to every job, including the ones that never retry; stale-graph bugs if the payload changes between attempts\nB) In-process memo (bounded LRU)\n  \u2705 No persistence, no schema change; a few lines around the graph builder (human: ~2 hr / CC: ~10 min)\n  \u2705 Zero cost on the happy path beyond a map insert\n  \u274c Retries after a 10-minute delay usually land on a different worker instance, so the hit rate is low and unpredictable\nC) Measure first: timing metrics + p95 budget (recommended)\n  \u2705 Two metrics (graph compute ms, payload bytes) per attempt tell you whether this is 2 ms or 2 s before anyone writes cache code (human: ~1 hr / CC: ~5 min)\n  \u2705 No happy-path cost, no staleness risk, and the retry-policy module already logs per attempt (D7) so the hook point exists\n  \u274c If the rebuild is genuinely slow, retries stay expensive until the follow-up lands\nNet: Add a write to every job to save CPU on the few that retry, or spend an hour on metrics and decide with numbers.",
     "header": "Graph cache",
     "multiSelect": false,
     "options": [
      {
       "label": "Persist the graph with the job",
       "description": "Compute once on attempt 1, store the graph beside the job row, reuse on retries, invalidate when the payload version changes. Adds a write to every job. Human ~1 day / CC ~20 min."
      },
      {
       "label": "In-process memo (bounded LRU)",
       "description": "Memoize the graph per worker instance keyed by job id + payload hash, bounded LRU. No persistence; low hit rate when retries land on another instance. Human ~2 hr / CC ~10 min."
      },
      {
       "label": "Measure first (recommended)",
       "description": "No cache. Add per-attempt timing metrics (job load ms, graph compute ms, payload bytes) via the retry-policy module's attempt log, set a p95 budget, and revisit caching with data. Human ~1 hr / CC ~5 min."
      }
     ]
    }
   ],
   "answered": true,
   "failed": false,
   "answers": {
    "D9 \u2014 Cache the dependency graph across retries now, or measure first?\nProject/branch/task: main branch, retry-framework plan; per-attempt cost inside the 5 workers running through library hooks (D1).\nELI10: The plan worries that each retry reloads the job and rebuilds its dependency graph from scratch. After D1 the reload is just the library handing the job to the worker, which happens anyway. The rebuild is real extra CPU, but only on retries, and D3 caps those at 5 per failing job. Storing the graph on the first attempt would add a write to every job, including the large majority that succeed first time, to save work on the few that fail. Nobody has measured how long the rebuild takes.\nStakes if we pick wrong: either we add a write and a staleness risk to every job to fix a cost nobody measured, or a genuinely slow rebuild keeps burning worker time on retries and we only find out under load.\nRecommendation: C because an unmeasured optimization that taxes the happy path is the wrong trade; two timing metrics make the real decision cheap and data-driven.\nNote: options differ in kind (persisted cache vs in-process memo vs measure first) \u2014 no completeness score.\nPros / cons:\nA) Persist the graph with the job on attempt 1\n  \u2705 Retries never recompute; cost is paid once per job regardless of which worker instance retries (human: ~1 day / CC: ~20 min)\n  \u2705 Simple to reason about once the invalidation rule (payload version) is in place\n  \u274c Adds a write and stored blob to every job, including the ones that never retry; stale-graph bugs if the payload changes between attempts\nB) In-process memo (bounded LRU)\n  \u2705 No persistence, no schema change; a few lines around the graph builder (human: ~2 hr / CC: ~10 min)\n  \u2705 Zero cost on the happy path beyond a map insert\n  \u274c Retries after a 10-minute delay usually land on a different worker instance, so the hit rate is low and unpredictable\nC) Measure first: timing metrics + p95 budget (recommended)\n  \u2705 Two metrics (graph compute ms, payload bytes) per attempt tell you whether this is 2 ms or 2 s before anyone writes cache code (human: ~1 hr / CC: ~5 min)\n  \u2705 No happy-path cost, no staleness risk, and the retry-policy module already logs per attempt (D7) so the hook point exists\n  \u274c If the rebuild is genuinely slow, retries stay expensive until the follow-up lands\nNet: Add a write to every job to save CPU on the few that retry, or spend an hour on metrics and decide with numbers.": "Persist the graph with the job"
   },
   "unansweredQuestionIndices": [],
   "answeredAt": "2026-09-29T20:04:03.405Z"
  },
  {
   "sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
   "toolUseId": "toolu_01WjGqEAsM8pPCNSzxpWyPSV",
   "questions": [
    {
     "question": "D10 \u2014 Track dead-letter retention as a TODO, skip it, or build it now?\nProject/branch/task: main branch, retry-framework plan; follow-up to the dead-letter store approved in D3.\nELI10: The dead-letter store keeps every job that ran out of retries or hit a fatal error. Nothing ever removes them. That is fine for months, then the table is large, the growth alert query slows, and nobody remembers why. A purge job with a retention period fixes it, but the retention period is a judgment call about how long failed-job evidence must stay around.\nStakes if we pick wrong: build it now with the wrong retention and you delete evidence of lost work; skip it and the store becomes an unbounded table someone discovers during an incident.\nRecommendation: A because the store is new, growth is slow, and the retention period deserves an owner's answer rather than a default picked inside a retry PR; the TODO carries a concrete trigger.\nCompleteness: A=6/10, B=2/10, C=10/10\nPros / cons:\nA) Add to TODOS.md with a trigger (recommended)\n  \u2705 Keeps this PR right-sized: the retry framework ships without a retention debate attached\n  \u2705 Trigger (10k rows or 3 months) means the TODO fires before growth matters (human: ~5 min / CC: ~1 min now)\n  \u274c Unbounded growth until someone acts on the TODO; the ceiling is a slow query, not data loss\nB) Skip\n  \u2705 Nothing to track or build\n  \u2705 Zero effort now\n  \u274c The store grows forever with no record that anyone considered it\nC) Build now in this PR\n  \u2705 Store ships bounded from day one: purge job, 90-day default, keep flag, tests (human: ~2 hr / CC: ~10 min)\n  \u2705 No follow-up to forget\n  \u274c Expands this PR with a scheduled job and a retention default nobody has agreed to; deletes evidence if the default is wrong\nNet: A tracked follow-up with a trigger, versus a bigger PR that guesses how long failed-job evidence should live.",
     "header": "DLQ retention",
     "multiSelect": false,
     "options": [
      {
       "label": "Add to TODOS.md (recommended)",
       "description": "Record the TODO (what/why/pros/cons/context/depends-on) with trigger: build when the dead-letter store passes 10k rows or at 3 months, whichever first. Human ~5 min / CC ~1 min. Completeness 6/10."
      },
      {
       "label": "Skip \u2014 not valuable enough",
       "description": "Do not track retention. Completeness 2/10."
      },
      {
       "label": "Build it now in this PR",
       "description": "Scheduled purge job, retention config default 90 days, keep flag, tests, shipped with the dead-letter store. Human ~2 hr / CC ~10 min. Completeness 10/10."
      }
     ]
    }
   ],
   "answered": true,
   "failed": false,
   "answers": {
    "D10 \u2014 Track dead-letter retention as a TODO, skip it, or build it now?\nProject/branch/task: main branch, retry-framework plan; follow-up to the dead-letter store approved in D3.\nELI10: The dead-letter store keeps every job that ran out of retries or hit a fatal error. Nothing ever removes them. That is fine for months, then the table is large, the growth alert query slows, and nobody remembers why. A purge job with a retention period fixes it, but the retention period is a judgment call about how long failed-job evidence must stay around.\nStakes if we pick wrong: build it now with the wrong retention and you delete evidence of lost work; skip it and the store becomes an unbounded table someone discovers during an incident.\nRecommendation: A because the store is new, growth is slow, and the retention period deserves an owner's answer rather than a default picked inside a retry PR; the TODO carries a concrete trigger.\nCompleteness: A=6/10, B=2/10, C=10/10\nPros / cons:\nA) Add to TODOS.md with a trigger (recommended)\n  \u2705 Keeps this PR right-sized: the retry framework ships without a retention debate attached\n  \u2705 Trigger (10k rows or 3 months) means the TODO fires before growth matters (human: ~5 min / CC: ~1 min now)\n  \u274c Unbounded growth until someone acts on the TODO; the ceiling is a slow query, not data loss\nB) Skip\n  \u2705 Nothing to track or build\n  \u2705 Zero effort now\n  \u274c The store grows forever with no record that anyone considered it\nC) Build now in this PR\n  \u2705 Store ships bounded from day one: purge job, 90-day default, keep flag, tests (human: ~2 hr / CC: ~10 min)\n  \u2705 No follow-up to forget\n  \u274c Expands this PR with a scheduled job and a retention default nobody has agreed to; deletes evidence if the default is wrong\nNet: A tracked follow-up with a trigger, versus a bigger PR that guesses how long failed-job evidence should live.": "Add to TODOS.md (recommended)"
   },
   "unansweredQuestionIndices": [],
   "answeredAt": "2026-09-29T20:05:54.778Z"
  },
  {
   "sessionId": "0117efe3-a002-42c1-aa59-d45032643e13",
   "toolUseId": "toolu_01QrMt8ZhcL4wV3cKEjo3MsA",
   "questions": [
    {
     "question": "D11 \u2014 Track the webhook event-id header and at-least-once opt-in as a TODO, skip it, or ship the header now?\nProject/branch/task: main branch, retry-framework plan; follow-up to D2 (webhook worker stays at-most-once).\nELI10: D2 kept the promise that a webhook is never sent twice, which means a send that times out is still lost. The usual fix is to stamp every event with a stable id so receivers can ignore duplicates, and then retry freely. That is a contract change, so it was declined for this PR. The question is whether to track it, drop it, or at least ship the harmless id header now so receivers can start deduping before the semantics ever change.\nStakes if we pick wrong: lost webhook events keep landing in dead-letter with no plan to stop the loss; or a header change rides along in a retry PR without receiver communication.\nRecommendation: A because this is a receiver-facing contract change that deserves its own PR and docs, and the dead-letter store (D3) will produce the loss numbers that justify it; the trigger is concrete.\nCompleteness: A=6/10, B=2/10, C=8/10\nPros / cons:\nA) Add to TODOS.md with a trigger (recommended)\n  \u2705 Keeps the retry PR free of webhook contract changes; the TODO fires on measured loss (human: ~5 min / CC: ~1 min now)\n  \u2705 Dead-letter counts of post-send failures give the case for it with real numbers\n  \u274c Webhook events lost to timeouts stay lost until the TODO is acted on\nB) Skip\n  \u2705 Nothing to track\n  \u2705 Zero effort now\n  \u274c No record that at-most-once was a deliberate trade with a known cost\nC) Ship the stable event-id header now, at-least-once later\n  \u2705 Receivers can start deduping today; the header is harmless under at-most-once (human: ~2 hr / CC: ~10 min)\n  \u2705 Makes the eventual semantics change a config flip instead of a payload change\n  \u274c Adds a webhook payload change and receiver docs to a retry PR; still needs the TODO for the semantics\nNet: Track it with a loss-based trigger, or ship a small header change now inside a PR about retries.",
     "header": "Webhook TODO",
     "multiSelect": false,
     "options": [
      {
       "label": "Add to TODOS.md (recommended)",
       "description": "Record the TODO with trigger: post-send dead-letter entries exceed 1% of webhook sends in any week, or a receiver requests redelivery. Human ~5 min / CC ~1 min. Completeness 6/10."
      },
      {
       "label": "Skip \u2014 not valuable enough",
       "description": "Do not track. Completeness 2/10."
      },
      {
       "label": "Ship the header now",
       "description": "Add a stable per-event id header to every webhook delivery in this PR; D2 semantics unchanged; at-least-once opt-in remains a TODO. Human ~2 hr / CC ~10 min. Completeness 8/10."
      }
     ]
    }
   ],
   "answered": true,
   "failed": false,
   "answers": {
    "D11 \u2014 Track the webhook event-id header and at-least-once opt-in as a TODO, skip it, or ship the header now?\nProject/branch/task: main branch, retry-framework plan; follow-up to D2 (webhook worker stays at-most-once).\nELI10: D2 kept the promise that a webhook is never sent twice, which means a send that times out is still lost. The usual fix is to stamp every event with a stable id so receivers can ignore duplicates, and then retry freely. That is a contract change, so it was declined for this PR. The question is whether to track it, drop it, or at least ship the harmless id header now so receivers can start deduping before the semantics ever change.\nStakes if we pick wrong: lost webhook events keep landing in dead-letter with no plan to stop the loss; or a header change rides along in a retry PR without receiver communication.\nRecommendation: A because this is a receiver-facing contract change that deserves its own PR and docs, and the dead-letter store (D3) will produce the loss numbers that justify it; the trigger is concrete.\nCompleteness: A=6/10, B=2/10, C=8/10\nPros / cons:\nA) Add to TODOS.md with a trigger (recommended)\n  \u2705 Keeps the retry PR free of webhook contract changes; the TODO fires on measured loss (human: ~5 min / CC: ~1 min now)\n  \u2705 Dead-letter counts of post-send failures give the case for it with real numbers\n  \u274c Webhook events lost to timeouts stay lost until the TODO is acted on\nB) Skip\n  \u2705 Nothing to track\n  \u2705 Zero effort now\n  \u274c No record that at-most-once was a deliberate trade with a known cost\nC) Ship the stable event-id header now, at-least-once later\n  \u2705 Receivers can start deduping today; the header is harmless under at-most-once (human: ~2 hr / CC: ~10 min)\n  \u2705 Makes the eventual semantics change a config flip instead of a payload change\n  \u274c Adds a webhook payload change and receiver docs to a retry PR; still needs the TODO for the semantics\nNet: Track it with a loss-based trigger, or ship a small header change now inside a PR about retries.": "Add to TODOS.md (recommended)"
   },
   "unansweredQuestionIndices": [],
   "answeredAt": "2026-09-29T20:07:04.232Z"
  }
 ],
 "plan": "# Eng Review \u2014 PLAN.md: Add background job retry framework\n\n- **Review target (fixed):** `/tmp/gstack-plan-count-vrYrwf/PLAN.md` \u2014 \"Plan: Add background job retry framework\"\n- **Report file:** `/tmp/gstack-e2e-plan-eng-batching-PLhMrg/gstack-test-plan-eng-batching.md` (destination explicitly requested by the user)\n- **Skill:** `/plan-eng-review` \u00b7 session `256191-1790711293-1222b505` \u00b7 2026-09-29 \u00b7 branch `main` @ `82eaa12`\n- **Evidence available:** the repository contains only `PLAN.md` and `CLAUDE.md`. No worker files, job library, or `processWebhookJob()` source exist in this checkout. Findings below quote the plan text (file:line) and are calibrated as plan-level, not code-verified.\n\n## Original plan (unchanged copy of PLAN.md lines 4-24)\n\n```markdown\n# Plan: Add background job retry framework\n\n## Architecture\nWe'll roll a custom exponential-backoff scheduler inline in each worker\nrather than use the existing job library's built-in retry hooks. Same\nshape as the library version, but we want full control over the curve.\n\n## Code quality\nThe retry envelope (compute delay, log attempt, dispatch) is duplicated\nacross 5 worker files with copy-pasted bodies. We will leave the\nduplication for now and refactor \"later.\"\n\n## Tests\nThe existing `processWebhookJob()` flow gets rewritten as part of this\nchange. No regression test for the prior at-most-once delivery guarantee\nis planned.\n\n## Performance\nOn every retry we re-fetch the full job payload from the database, then\niterate the payload to recompute the dependency graph. Could cache the\ngraph on the first attempt; not planned.\n```\n\n## Scope Challenge\n\n### A. Assessment\n- **Already solves it:** the job library's built-in retry hooks (PLAN.md:8 admits \"same shape as the library version\"). Library source not in checkout; hook API unverified.\n- **Complexity count (estimate from plan text):** 5 worker files (PLAN.md:13) + `processWebhookJob()` (PLAN.md:17, likely one of the five) = 5\u20136 changed files; 0 new classes/services (scheduler is inline). Under thresholds \u2192 complexity gate B skipped.\n- **Search check:** [Layer 1] library retry hooks + backoff callback; jitter, delay cap, dead-letter, idempotency key are standard practice (AWS Builders' Library; Hookdeck/Svix idempotency guides).\n- **TODOS.md:** none. **Distribution:** no new artifacts.\n\n### C. Findings (plan-level; no code in checkout)\n1. `[P1] (confidence: 8/10) PLAN.md:7-9` \u2014 rebuilding a retry scheduler the job library already provides. \u2192 R1 / D1\n2. `[P1] (confidence: 7/10) PLAN.md:17-19` \u2014 retrying `processWebhookJob()` changes at-most-once to at-least-once delivery; semantics change, not just a missing test. \u2192 Section 1\n3. `[P2] (confidence: 7/10) PLAN.md:7-9` \u2014 retry policy bounds unspecified (max attempts, delay cap, jitter, dead-letter, retryable vs fatal errors). \u2192 Section 1\n4. `[P2] (confidence: 7/10) PLAN.md:12-14` \u2014 five copy-pasted retry envelopes. \u2192 Section 2\n5. `[P1] (confidence: 8/10) PLAN.md:18-19` \u2014 no regression test for a rewritten flow with a stated guarantee (Regression Rule). \u2192 Section 3\n6. `[P2] (confidence: 6/10) PLAN.md:22-24` \u2014 full payload refetch + graph recompute on every retry. \u2192 Section 4\n\nScope Challenge result: **scope accepted as-is** (D1 changed the mechanism to library retry hooks; no feature was cut, so this is not a scope reduction). Dispositions: finding 1 accepted via D1 (R1 approved); findings 2\u20136 pending in their sections.\n\n## Section 1 \u2014 Architecture review\n\nWorking plan after D1: all 5 workers retry through the job library's hooks; each worker supplies its own backoff curve.\n\n```\nRETRY STATE MACHINE (per job, owned by the library after D1)\n\n  enqueue \u2500\u2500\u25b6 [attempt n] \u2500\u2500success\u2500\u2500\u25b6 DONE\n                 \u2502\n                 \u251c\u2500 fatal error (R3d: non-retryable class) \u2500\u2500\u25b6 FAILED \u2500\u2500\u25b6 dead-letter (R3a)\n                 \u2502\n                 \u2514\u2500 transient error / timeout\n                        \u2502\n                        \u251c\u2500 n >= maxAttempts (R3a) \u2500\u2500\u25b6 FAILED \u2500\u2500\u25b6 dead-letter (R3a)\n                        \u2502\n                        \u2514\u2500 delay = min(base\u00b72^n (+ jitter R3b), cap R3c) \u2500\u2500\u25b6 [attempt n+1]\n\n  Webhook worker only: timeout after the request was written is AMBIGUOUS \u2014\n  the receiver may already have the event. A retry here = possible duplicate (R2).\n```\n\nFindings:\n- `[P1] (confidence: 7/10) PLAN.md:17-19` \u2014 \"The existing `processWebhookJob()` flow gets rewritten ... prior at-most-once delivery guarantee.\" Adding retries flips the webhook worker from at-most-once to at-least-once: a retry after an ambiguous timeout can deliver the same event twice. The plan treats this as a missing test; it is a delivery-contract change for every receiver. \u2192 R2 / D2\n- `[P2] (confidence: 7/10) PLAN.md:7-9` \u2014 \"custom exponential-backoff scheduler ... full control over the curve.\" The curve is named but no bound is: no max attempts or terminal handling (dead-letter), no jitter, no delay cap, no retryable-vs-fatal error classification. Each is an independent choice. \u2192 R3a (terminal handling), R3b (jitter), R3c (delay cap), R3d (error classification)\n- `[P2] (confidence: 6/10) PLAN.md:22` \u2014 \"On every retry we re-fetch the full job payload from the database\" implies attempt state lives in the DB between attempts. With D1 (library hooks) attempt counting and persistence are library-owned; no separate decision. Performance side \u2192 Section 4 (R6).\n- Realistic production failure: the downstream webhook receiver is down for 2 hours. All webhook jobs fail together, then retry together when it recovers (thundering herd without jitter, R3b), and jobs past max attempts must land somewhere visible (R3a) rather than vanish.\n- Diagram: the retry state machine above belongs inline in the shared retry config module once written.\n- Distribution: no new artifacts; no CI/CD change.\n\nSection 1 dispositions: finding 1 \u2192 R2 approved A (D2, at-most-once kept); finding 2 \u2192 R3a approved A (D3), R3b approved A (D4), R3c approved A (D5), R3d approved A (D6); finding 3 \u2192 resolved by D1 (library owns attempt state), performance side in Section 4. Section 1 total: 3 findings, 0 open.\n\n## Section 2 \u2014 Code quality review\n\nFindings:\n- `[P2] (confidence: 7/10) PLAN.md:12-14` \u2014 \"duplicated across 5 worker files with copy-pasted bodies. We will leave the duplication for now and refactor 'later.'\" After D1\u2013D6 the per-worker glue is identical by construction; five copies of jitter/cap/classifier/dead-letter code is the drift risk, not a style nit. Shared-code rubric evidence is in the R4 record. \u2192 R4 / D7\n- `[P2] (confidence: 6/10) PLAN.md:7-9` \u2014 error-handling gap: the plan never says what happens when the dead-letter write itself fails (store down). Necessary implementation of the approved D3 contract, not a new choice: if `toDeadLetter` throws, the job stays in the library's failed state, an error log with the original error and the dead-letter failure fires, and a metric increments. Never swallow both errors. Test required (Section 3).\n- `[P2] (confidence: 6/10) PLAN.md:22` \u2014 edge case: the job row is deleted between attempts, so the refetch returns nothing. Under D6 \"missing record\" is fatal \u2192 dead-letter immediately with the payload reference. Add to each worker's fatal list explicitly; test required.\n- `[P3] (confidence: 6/10)` \u2014 config edge cases: `maxAttempts` 0 or negative, `maxDelay` below base, missing per-worker override. Validate at worker startup and fail fast (part of R4 option A/C; in B, five copies of the validation). Test required.\n- Diagrams: the retry state machine (Section 1) belongs inline as a comment in the shared module (R4 A/C) or in each worker (R4 B).\n- Debt check: with D1\u2013D6 approved, no premature abstraction remains in the plan; the only fragility is the duplication itself.\n\nSection 2 dispositions: finding 1 \u2192 R4 approved A (D7, shared module); findings 2\u20134 carried as necessary implementation/proof of D3, D6 and D7 (no new choice). Section 2 total: 4 findings, 0 open.\n\n## Section 3 \u2014 Test review\n\n**Test framework detection:** CLAUDE.md has no Testing section. Auto-detect in the checkout: no runtime markers, no test config, `TESTFILES:0`. Framework unknown; the plan proposes none, so no framework question. Assertions below are framework-neutral.\n\n**Step 1\u20132: traced codepaths and user flows (all proposed; no runnable source in checkout).** Entry points: each worker's job handler wired to the library retry hooks (D1) via the shared module (D7); the webhook worker's send path (D2); the dead-letter store and its replay path (D3).\n\n**Step 3\u20134: coverage diagram**\n\n```\nCODE PATHS (proposed)                                              USER / OPERATOR FLOWS\n[+] retry-policy module (D7)                                       [+] Dead-letter operations (D3)\n  \u251c\u2500\u2500 buildBackoff()                                                 \u251c\u2500\u2500 [GAP] [\u2192E2E] alert fires when dead-letter count grows\n  \u2502   \u251c\u2500\u2500 [GAP] 0 \u2264 delay \u2264 base\u00b7mult^n (seeded RNG, D4)             \u251c\u2500\u2500 [GAP]        operator sees last error + attempt history\n  \u2502   \u251c\u2500\u2500 [GAP] delay capped at maxDelay for large n (D5)            \u251c\u2500\u2500 [GAP] [\u2192E2E] replay re-enqueues a non-webhook job once\n  \u2502   \u2514\u2500\u2500 [GAP] defaults apply when worker sets nothing              \u2514\u2500\u2500 [GAP]        webhook replay shows \"may duplicate\" warning\n  \u251c\u2500\u2500 classify()\n  \u2502   \u251c\u2500\u2500 [GAP] retryable class \u2192 retryable                        [+] Webhook receiver experience (D2)\n  \u2502   \u251c\u2500\u2500 [GAP] fatal class \u2192 fatal                                  \u251c\u2500\u2500 [GAP] [\u2192E2E] receiver gets each event exactly once\n  \u2502   \u2514\u2500\u2500 [GAP] unknown error \u2192 retryable (D6 default)               \u2514\u2500\u2500 [GAP]        receiver down 2h: no duplicate on recovery\n  \u251c\u2500\u2500 toDeadLetter()\n  \u2502   \u251c\u2500\u2500 [GAP] persists record + emits metric                     [+] Error states\n  \u2502   \u2514\u2500\u2500 [GAP] persist fails \u2192 job stays failed, both errors logged  \u251c\u2500\u2500 [GAP]        bug deploy: fatal \u2192 dead-letter on attempt 1\n  \u251c\u2500\u2500 logAttempt()                                                   \u2514\u2500\u2500 [GAP]        poisoned job: stops at maxAttempts, parked\n  \u2502   \u2514\u2500\u2500 [GAP] fields: job id, attempt, delay, error class\n  \u2514\u2500\u2500 validateConfig()\n      \u251c\u2500\u2500 [GAP] maxAttempts < 1 \u2192 startup failure\n      \u2514\u2500\u2500 [GAP] maxDelay < base \u2192 startup failure\n[+] 4 non-webhook workers \u00d7 hook wiring (D1)\n  \u251c\u2500\u2500 [GAP] [\u2192E2E] transient error \u2192 retried with module delay\n  \u251c\u2500\u2500 [GAP] [\u2192E2E] fatal error \u2192 dead-letter, no attempts consumed\n  \u251c\u2500\u2500 [GAP] [\u2192E2E] exhaustion at maxAttempts \u2192 dead-letter\n  \u2514\u2500\u2500 [GAP]        job row deleted between attempts \u2192 fatal\n[+] processWebhookJob() (D2)  \u2014 REGRESSION, CRITICAL \u2192 R5 / D8\n  \u251c\u2500\u2500 [GAP] pre-send failure (refused/DNS/TLS/local) \u2192 retry, then exactly 1 send\n  \u251c\u2500\u2500 [GAP] timeout after send \u2192 0 further sends, dead-letter entry\n  \u251c\u2500\u2500 [GAP] 5xx after send \u2192 0 further sends, dead-letter entry\n  \u251c\u2500\u2500 [GAP] connection reset mid-response \u2192 0 further sends\n  \u251c\u2500\u2500 [GAP] success \u2192 1 send, no dead-letter\n  \u2514\u2500\u2500 [GAP] [\u2192E2E] attempt count survives worker restart mid-curve\n\nLLM integration: none in this plan \u2014 no eval scope.\n\nCOVERAGE: 0/27 paths tested (0%)  |  Code paths: 0/20 (0%)  |  User flows: 0/7 (0%)\nQUALITY: \u2605\u2605\u2605:0 \u2605\u2605:0 \u2605:0  |  GAPS: 27 (8 E2E, 0 eval)\n```\n\nLegend: \u2605\u2605\u2605 behavior + edge + error  |  \u2605\u2605 happy path  |  \u2605 smoke check  |  [\u2192E2E] = needs integration test  |  [\u2192EVAL] = needs LLM eval\n\n**Regression Rule:** the `processWebhookJob()` rewrite puts an existing guarantee at risk with no coverage planned (PLAN.md:18-19). Behavior to preserve (fixed by D2): at most one send per event, ever. Intentional change: pre-send failures now retry. Coverage is required; D8 settles how. \u2192 R5\n\n**Step 5: tests to add.** Required proof of approved behaviors (D3\u2013D7), no new choice: every `[GAP]` under the retry-policy module and the 4 workers above, as unit tests in the module's suite plus one integration test per worker through the real hooks. Pending: the webhook regression contract's assertions and depth (R5 / D8). Test Plan Artifact is written after D8.\n\nSection 3 dispositions: CRITICAL regression gap \u2192 R5 approved A (D8, unit + integration); 26 other gaps carried as required proof of D3\u2013D7 (no new choice). Test Plan Artifact written (path in Completion summary). Section 3 total: 27 gaps identified, 0 open decisions.\n\n## Section 4 \u2014 Performance review\n\nFindings:\n- `[P2] (confidence: 6/10) PLAN.md:22-24` \u2014 \"On every retry we re-fetch the full job payload ... recompute the dependency graph. Could cache the graph on the first attempt; not planned.\" Under D1 the refetch is the library's normal dequeue; only the graph recompute is extra, bounded to maxAttempts (D3) per failing job. Unmeasured. Medium confidence, verify this is actually an issue. \u2192 R6 / D9\n- `[P3] (confidence: 5/10)` \u2014 dead-letter store (D3) grows without bound if entries are never replayed or purged. No retention policy in scope. Medium confidence. \u2192 TODO candidate 1 (retention/purge policy).\n- `[P3] (confidence: 6/10)` \u2014 the dead-letter growth alert (D3) should read a counter metric emitted by `toDeadLetter`, not run `COUNT(*)` on the store per job. Implementation guidance inside the approved D3 work; no new choice.\n- N+1: none introduced; the per-attempt job load is one row by id. Memory: jitter RNG and classifier lists are negligible; payload size unknown (metric proposed in R6 option C).\n\nSection 4 dispositions: finding 1 \u2192 R6 approved A (D9, persist the graph with the job; user's call over the measure-first recommendation, adds a schema migration and 4 tests); finding 2 \u2192 T1 (TODO question D10); finding 3 \u2192 guidance inside approved D3 work. Section 4 total: 3 findings, 0 open.\n\n## Decision ledger\n\n### R1: Retry scheduling mechanism (library hooks vs custom inline scheduler)\nFinding: #1, P1, confidence 8/10, PLAN.md:7-9, reviewer: plan-eng-review (Claude)\nPlan baseline: original proposal \u2014 custom exponential-backoff scheduler inline in each of 5 workers; library retry hooks bypassed (PLAN.md:7-9). Nothing approved yet.\nRuntime evidence: unknown \u2014 job library not in checkout. Plan states the library version has the same shape (PLAN.md:8). Whether its hooks accept a custom curve callback is unverified.\nComparison grid:\n\n| Choice | Current | A) Library hooks + custom curve | B) Custom inline scheduler | C) Investigate hook API first |\n|---|---|---|---|---|\n| R1 retry scheduling mechanism | custom inline scheduler (proposed, unapproved) | library retry hooks; per-worker backoff curve supplied as a callback/config | custom scheduler inline in each worker, as planned | undecided; bounded ~30 min read of the library's hook signature/docs, then return to R1 |\n| R2 webhook delivery semantics | pending | pending | pending | pending |\n| R3 retry policy bounds (attempts/cap/jitter/dead-letter) | pending | pending (library config likely hosts them; not decided here) | pending | pending |\n| R4 envelope duplication across 5 workers | pending | pending (largely dissolves if library owns delay+dispatch; not decided here) | pending | pending |\n| R5 regression test for at-most-once | pending | pending | pending | pending |\n| R6 payload refetch / graph cache | pending | pending | pending | pending |\n\nQuestion D1:\nD1 \u2014 Reuse the job library's retry hooks or roll a custom scheduler?\nProject/branch/task: main branch, retry-framework plan; adding retries to 5 background workers.\nELI10: The job library you already use has retry hooks built in, and your plan says your custom version would be \"the same shape.\" Building your own copy inside each worker means five hand-written schedulers to keep correct, versus configuring one curve the library already knows how to run. The plan's reason for going custom is \"full control over the curve,\" and most retry hook APIs give you that through a backoff callback.\nStakes if we pick wrong: five bespoke schedulers drift apart, each grows its own bugs (no jitter, no cap, retry storms), and nobody at 3am knows which curve a given worker actually runs.\nRecommendation: A because the plan admits the library version has the same shape, and a custom curve is usually a config callback, not a new scheduler.\nCompleteness: A=9/10, B=5/10, C=n/a (investigation, decides nothing)\nPros / cons:\nA) Library hooks + custom curve (recommended)\n  \u2705 One scheduler the library already tests; the curve becomes a per-worker config function (human: ~1 day / CC: ~20 min)\n  \u2705 Attempt-count persistence, max attempts and dead-letter usually come along for free with the hooks\n  \u274c If the hook API cannot accept an arbitrary curve function, that is a new fact and R1 reopens\nB) Custom inline scheduler (as planned)\n  \u2705 Total control of delay math and logging, independent of the library's release cycle\n  \u2705 No dependency on hook semantics nobody has read yet (human: ~3 days / CC: ~1 hr)\n  \u274c Five hand-rolled schedulers to keep in sync, plus reimplementing attempt persistence and terminal handling\nC) Investigate first: bounded read of the hook API\n  \u2705 Settles \"can the hooks take a custom curve\" with evidence before committing either way\n  \u2705 Cheap: ~30 min human / ~3 min CC reading the hook signature and docs\n  \u274c Decides nothing; R1 stays pending and the review pauses on this question\nNet: You are trading a library dependency you already carry for five copies of scheduler code you would own forever.\nHeader: Retry mechanism\nOptions:\nA) Library hooks + custom curve (recommended)\nUse the job library's built-in retry hooks; supply each worker's backoff curve as a callback/config. One scheduler the library already tests. Attempt persistence, max attempts and dead-letter usually included. Human ~1 day / CC ~20 min. Risk: if the hook API cannot take an arbitrary curve, R1 reopens. Completeness 9/10.\nB) Custom inline scheduler (as planned)\nRoll the exponential-backoff scheduler inline in each of the 5 workers as PLAN.md:7-9 proposes. Total control of delay math and logging. Human ~3 days / CC ~1 hr. Cost: five schedulers to keep in sync, plus attempt persistence and terminal handling rebuilt by hand. Completeness 5/10.\nC) Investigate hook API first\nBounded ~30 min human / ~3 min CC read of the library's retry hook signature and docs, then return to this question. Approves nothing; R1 stays pending; R2\u2013R6 unchanged.\n\nState: approved\nActual answer: A) Library hooks + custom curve \u2014 user answer to D1\nAccepted scope: Use the job library's built-in retry hooks in all 5 workers (including the `processWebhookJob()` worker); supply each worker's backoff curve as a callback/config; do not build the custom inline scheduler from PLAN.md:7-9. Condition carried: if the hook API cannot accept an arbitrary curve function, R1 reopens as a new fact. R2\u2013R6 remain pending and are unchanged by this answer.\nHistory: none\n\n### R2: Delivery guarantee for `processWebhookJob()` once it can retry\nFinding: Section 1 finding 1 (Scope Challenge #2), P1, confidence 7/10, PLAN.md:17-19, reviewer: plan-eng-review (Claude)\nPlan baseline: existing behavior is at-most-once delivery (PLAN.md:18). The proposed rewrite adds retries and states no delivery guarantee. Nothing approved for R2.\nRuntime evidence: unknown \u2014 `processWebhookJob()` source not in checkout. Industry practice (Hookdeck, Svix, Stripe, Shopify) is at-least-once delivery with a stable per-event idempotency key.\nComparison grid:\n\n| Choice | Current | A) Keep at-most-once | B) At-least-once + idempotency key | C) Exclude webhook worker from retries |\n|---|---|---|---|---|\n| R2 webhook delivery semantics | at-most-once (existing); rewrite unspecified | at-most-once kept; retry only failures provably raised before the request was written (connection refused, DNS, local error); timeouts/5xx-after-send still drop | at-least-once; a stable event id/idempotency key header constant across attempts; documented contract change + receiver migration note | webhook worker keeps today's at-most-once behavior with no retries; other 4 workers retry per D1 |\n| R1 retry mechanism | approved A (D1) | fixed | fixed | fixed for the 4 non-webhook workers; webhook worker untouched |\n| R3a\u2013R3d retry bounds | pending | pending | pending | pending |\n| R4 envelope duplication | pending | pending | pending | pending |\n| R5 regression test for the webhook flow | pending (behavior to preserve depends on R2) | pending | pending | pending |\n| R6 payload refetch / graph cache | pending | pending | pending | pending |\n\nQuestion D2:\nD2 \u2014 What delivery guarantee does processWebhookJob() keep once it can retry?\nProject/branch/task: main branch, retry-framework plan; the webhook worker is one of the 5 workers gaining retries through library hooks (D1).\nELI10: Today the webhook worker sends each event at most once: if the send fails or times out, the event is dropped, never duplicated. A retry cannot tell \"the request never arrived\" apart from \"it arrived but the response got lost,\" so any retry after a timeout can deliver the same event twice. Adding retries silently flips the guarantee from at-most-once to at-least-once. That is a contract change your webhook receivers depend on, and the plan does not name it.\nStakes if we pick wrong: receivers that are not idempotent process duplicate events (double emails, double charges, double state transitions); or, if we keep dropping on ambiguity, the retry framework never fixes the webhook worker's lost events.\nRecommendation: B because losing events is usually worse than duplicates, and a stable idempotency key makes duplicates safe for receivers; this is still a contract call you know better than the review does.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nPros / cons:\nA) Keep at-most-once: retry only provably-unsent failures\n  \u2705 No duplicate deliveries ever; existing receivers keep working with no change on their side\n  \u2705 Still recovers the clear cases: connection refused, DNS failure, local enqueue error (human: ~1 day / CC: ~30 min)\n  \u274c Timeouts and 5xx-after-send still drop events, so the biggest source of loss stays; needs per-attempt failure classification\nB) Move to at-least-once with a stable idempotency key (recommended)\n  \u2705 Every event eventually reaches the receiver; retries after timeouts are safe because the event id stays constant across attempts\n  \u2705 Matches Stripe, Shopify and Svix practice; receivers dedupe on the key (human: ~1.5 days / CC: ~30 min incl. docs)\n  \u274c Contract change: receivers must dedupe; needs a documented header, changelog entry and migration note for existing receivers\nC) Exclude processWebhookJob() from retries\n  \u2705 Zero semantic change for receivers; the other 4 workers still get retries\n  \u2705 Smallest diff and no receiver communication (human: ~1 hr / CC: ~5 min)\n  \u274c The webhook worker keeps losing events on every transient failure, which is likely why the plan touched it\nNet: Never-duplicate-but-lossy, never-lossy-but-receivers-must-dedupe, or leave the webhook worker exactly as it is.\nHeader: Webhook delivery\nOptions:\nA) Keep at-most-once (retry only pre-send failures)\nWebhook worker retries only failures provably raised before the request was written (connection refused, DNS, local error). Timeouts and 5xx-after-send still drop the event. No duplicates; receivers unchanged. Needs per-attempt failure classification. Human ~1 day / CC ~30 min.\nB) At-least-once + idempotency key (recommended)\nWebhook worker retries all transient failures; every delivery carries a stable event id / idempotency key header constant across attempts. Documented contract change with changelog and receiver migration note. Receivers dedupe on the key. Human ~1.5 days / CC ~30 min.\nC) Exclude webhook worker from retries\n`processWebhookJob()` keeps today's at-most-once, no-retry behavior; the other 4 workers retry via library hooks per D1. Smallest diff, no receiver impact, webhook events still lost on transient failure. Human ~1 hr / CC ~5 min.\n\nState: approved\nActual answer: A) Keep at-most-once (retry only pre-send failures) \u2014 user answer to D2 (not the recommended option; user's contract call)\nAccepted scope: `processWebhookJob()` keeps the at-most-once delivery guarantee. It retries (via library hooks, D1) only failures provably raised before any request bytes were written: connection refused, DNS failure, TLS handshake failure, local serialization/enqueue error. Any failure after the request is written (timeout, 5xx, connection reset mid-response) is terminal for that event: no retry, dropped as today, logged with the failure class. Necessary implementation carried as common work: per-attempt pre-send vs post-send failure classification inside the webhook worker, plus its tests (Section 3, R5 regression contract: at most one send per event, ever). No idempotency header, no receiver contract change. R3a\u2013R3d, R4, R6 unchanged and pending; R3d (retryable vs fatal classification for the other 4 workers) remains its own choice.\nHistory: none\n\n### R3a: Terminal handling \u2014 max attempts and what happens when they run out\nFinding: Section 1 finding 2 (Scope Challenge #3), P2, confidence 7/10, PLAN.md:7-9, reviewer: plan-eng-review (Claude)\nPlan baseline: original proposal names \"full control over the curve\" (PLAN.md:9) and no attempt limit or terminal outcome. Nothing approved for R3a.\nRuntime evidence: unknown \u2014 no worker or library source in checkout. Practice: bounded attempts with a dead-letter store; never silently drop (AWS Builders' Library; retry-pattern references).\nComparison grid:\n\n| Choice | Current | A) Bounded + dead-letter + alert | B) Bounded, log and drop | C) Library defaults, unspecified |\n|---|---|---|---|---|\n| R3a terminal handling | unspecified | `maxAttempts` default 5, per-worker override; on exhaustion or fatal error the job lands in a dead-letter store (table or queue) with last error, attempt history and payload ref; metric + alert on dead-letter growth; manual replay path | `maxAttempts` default 5, per-worker override; on exhaustion log at error level with last error and drop the job | whatever the library does by default; not written into the plan |\n| R1 retry mechanism | approved A (D1) | fixed | fixed | fixed |\n| R2 webhook delivery | approved A (D2) | fixed | fixed | fixed |\n| R3b jitter | pending | pending | pending | pending |\n| R3c delay cap | pending | pending | pending | pending |\n| R3d error classification (4 non-webhook workers) | pending | pending | pending | pending |\n| R4 envelope duplication | pending | pending | pending | pending |\n| R5 regression test | pending | pending | pending | pending |\n| R6 payload refetch / graph cache | pending | pending | pending | pending |\n\nQuestion D3:\nD3 \u2014 When a job runs out of retries, where does it go?\nProject/branch/task: main branch, retry-framework plan; retry bounds for all 5 workers running through library hooks (D1).\nELI10: Right now the plan describes the curve between retries but never says how many retries there are or what happens to a job that keeps failing. Without a limit, a poisoned job retries forever and eats worker capacity. With a limit but no landing spot, the job disappears with one log line nobody reads. A dead-letter store keeps the failed job, its payload reference and its last error so someone can inspect and replay it.\nStakes if we pick wrong: either an infinite-retry job starves the queue, or real work silently vanishes after the last attempt and the first sign is a customer asking where their data went.\nRecommendation: A because a dead-letter store plus an alert is a few dozen lines with library hooks, and it turns \"job vanished\" into \"job parked, here is why.\"\nCompleteness: A=10/10, B=5/10, C=3/10\nPros / cons:\nA) Bounded attempts + dead-letter store + alert (recommended)\n  \u2705 Exhausted or fatal jobs are kept with last error and attempt history; operators can inspect and replay (human: ~1 day / CC: ~20 min)\n  \u2705 Metric and alert on dead-letter growth turns a silent failure into a page at the right time\n  \u274c One more table or queue to own, plus a small replay path to build and test\nB) Bounded attempts, log and drop\n  \u2705 Simplest bound: `maxAttempts` default 5 per worker, one error log on exhaustion (human: ~2 hr / CC: ~5 min)\n  \u2705 No new storage; nothing to operate\n  \u274c Exhausted jobs are gone; recovery means replaying from upstream sources by hand, if that is even possible\nC) Leave to library defaults\n  \u2705 Zero plan text and zero decision now\n  \u2705 Whatever the library does is at least consistent across the 5 workers\n  \u274c Nobody knows the limit or the terminal behavior until an incident teaches them; 3am failure mode\nNet: You are trading one small dead-letter store for never having to ask \"where did that job go.\"\nHeader: Retry exhaustion\nOptions:\nA) Bounded + dead-letter + alert (recommended)\n`maxAttempts` default 5 with per-worker override. On exhaustion or fatal error the job lands in a dead-letter store (table or queue) with last error, attempt history and payload reference. Metric and alert on dead-letter growth. Manual replay path. Human ~1 day / CC ~20 min. Completeness 10/10.\nB) Bounded, log and drop\n`maxAttempts` default 5 with per-worker override. On exhaustion, log at error level with the last error and drop the job. No new storage, no replay. Human ~2 hr / CC ~5 min. Completeness 5/10.\nC) Library defaults, unspecified\nDo not write attempt limits or terminal behavior into the plan; accept whatever the library does by default. Completeness 3/10.\n\nState: approved\nActual answer: A) Bounded + dead-letter + alert \u2014 user answer to D3\nAccepted scope: All 5 workers: `maxAttempts` default 5 with per-worker override (library config, D1). On exhaustion or on a fatal (non-retryable) error the job lands in a dead-letter store (table or queue) recording last error, attempt history and payload reference. Metric and alert on dead-letter growth. Manual replay path. Tests for the exhaustion path, the fatal-error path and replay are common work of this behavior. Reconciliation with D2: a webhook post-send failure is terminal for that event and is recorded in the dead-letter store (not auto-retried, no duplicate send); manual replay of a webhook entry is an explicit operator action and the replay UI/docs must say it may duplicate. R3b, R3c, R3d, R4, R5, R6 unchanged and pending.\nHistory: none\n\n### R3b: Jitter on retry delays\nFinding: Section 1 finding 2 (Scope Challenge #3), P2, confidence 7/10, PLAN.md:7-9, reviewer: plan-eng-review (Claude)\nPlan baseline: \"custom exponential-backoff ... full control over the curve\" (PLAN.md:7-9); no jitter mentioned. Nothing approved for R3b.\nRuntime evidence: unknown \u2014 no worker source in checkout. AWS Builders' Library analysis: full jitter gives the least contention and total work; deterministic exponential curves synchronize retries after a shared outage.\nComparison grid:\n\n| Choice | Current | A) Full jitter | B) Equal jitter | C) No jitter |\n|---|---|---|---|---|\n| R3b jitter | unspecified (deterministic curve implied) | delay = random(0, exponentialDelay) inside each worker's curve callback; RNG injectable for tests | delay = exponentialDelay/2 + random(0, exponentialDelay/2); RNG injectable for tests | deterministic exponentialDelay; no randomization |\n| R1 retry mechanism | approved A (D1) | fixed | fixed | fixed |\n| R2 webhook delivery | approved A (D2) | fixed | fixed | fixed |\n| R3a terminal handling | approved A (D3) | fixed | fixed | fixed |\n| R3c delay cap | pending | pending | pending | pending |\n| R3d error classification (4 non-webhook workers) | pending | pending | pending | pending |\n| R4 envelope duplication | pending | pending | pending | pending |\n| R5 regression test | pending | pending | pending | pending |\n| R6 payload refetch / graph cache | pending | pending | pending | pending |\n\nQuestion D4:\nD4 \u2014 Should retry delays be randomized (jitter)?\nProject/branch/task: main branch, retry-framework plan; the backoff curve each worker supplies to the library hooks (D1).\nELI10: When many jobs fail at the same moment because a shared dependency went down, a pure exponential curve makes them all retry at the same moments too, so the recovering dependency gets hit by a wave on every step. Jitter randomizes each job's delay so the retries spread out. It is one line inside the curve callback each worker already supplies.\nStakes if we pick wrong: synchronized retry waves knock a recovering dependency back over (the classic thundering herd); or, with jitter, per-job retry timing becomes slightly less predictable and tests need a seeded random source.\nRecommendation: A because full jitter gives the least contention in AWS's published analysis and costs one line in a callback you are already writing.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nPros / cons:\nA) Full jitter: random(0, exponentialDelay) (recommended)\n  \u2705 Best spread of retries and lowest total contention after a shared outage (AWS Builders' Library)\n  \u2705 One line inside the D1 curve callback; RNG injected so tests stay deterministic (human: ~1 hr / CC: ~5 min)\n  \u274c An individual retry can fire almost immediately; minimum wait is not guaranteed\nB) Equal jitter: half fixed, half random\n  \u2705 Guarantees a minimum wait of half the exponential delay while still spreading retries\n  \u2705 Same one-line cost and same injectable RNG as full jitter (human: ~1 hr / CC: ~5 min)\n  \u274c Slightly more contention than full jitter in the same analysis, and one more parameter to explain\nC) No jitter: deterministic curve\n  \u2705 Fully deterministic; trivial to reason about and to assert exact delays in tests\n  \u2705 Zero extra code beyond the exponential curve\n  \u274c Every job that failed together retries together; retry storms on recovery are the expected outcome\nNet: One random() call now versus a synchronized retry wave the first time a dependency has a bad hour.\nHeader: Jitter\nOptions:\nA) Full jitter (recommended)\ndelay = random(0, exponentialDelay) inside each worker's curve callback. Best spread, lowest contention. RNG injectable so tests are deterministic. Human ~1 hr / CC ~5 min.\nB) Equal jitter\ndelay = exponentialDelay/2 + random(0, exponentialDelay/2). Guarantees a minimum wait; slightly more contention than full jitter. RNG injectable. Human ~1 hr / CC ~5 min.\nC) No jitter\nDeterministic exponential curve, no randomization. Simplest to test; retries synchronize after a shared outage.\n\nState: approved\nActual answer: A) Full jitter \u2014 user answer to D4\nAccepted scope: Every worker's backoff curve callback (D1) applies full jitter: delay = random(0, exponentialDelay). The random source is injectable so unit tests assert exact delays with a seeded RNG. Tests for the jitter bounds (0 \u2264 delay \u2264 exponentialDelay) are common work of this behavior. R3c, R3d, R4, R5, R6 unchanged and pending.\nHistory: none\n\n### R3c: Ceiling on the backoff delay (delay cap)\nFinding: Section 1 finding 2 (Scope Challenge #3), P2, confidence 7/10, PLAN.md:7-9, reviewer: plan-eng-review (Claude)\nPlan baseline: exponential backoff with \"full control over the curve\" (PLAN.md:7-9); no base delay, multiplier or ceiling stated. Nothing approved for R3c.\nRuntime evidence: unknown \u2014 no worker source in checkout. With D3's per-worker `maxAttempts` override, an uncapped doubling curve from a 1 s base reaches ~4.5 h at attempt 15 and ~6 days at attempt 20.\nComparison grid:\n\n| Choice | Current | A) Cap each delay | B) No cap |\n|---|---|---|---|\n| R3c delay cap | unspecified | delay = min(jitteredExponential, maxDelay); `maxDelay` default 10 min, per-worker override; curve defaults documented as base 1 s, multiplier 2, per-worker override | no ceiling; delay follows the raw exponential curve |\n| R1 retry mechanism | approved A (D1) | fixed | fixed |\n| R2 webhook delivery | approved A (D2) | fixed | fixed |\n| R3a terminal handling | approved A (D3) | fixed | fixed |\n| R3b jitter | approved A (D4) | fixed | fixed |\n| R3d error classification (4 non-webhook workers) | pending | pending | pending |\n| R4 envelope duplication | pending | pending | pending |\n| R5 regression test | pending | pending | pending |\n| R6 payload refetch / graph cache | pending | pending | pending |\n\nQuestion D5:\nD5 \u2014 Should the backoff delay have a ceiling?\nProject/branch/task: main branch, retry-framework plan; the curve parameters each worker passes to the library hooks (D1, jittered per D4).\nELI10: Exponential backoff doubles the wait after each failure. That is fine for 5 attempts, but D3 lets each worker raise its attempt count, and a worker set to 15 attempts from a 1 second base would wait about 4.5 hours before its last try; at 20 attempts it would wait 6 days. A cap says \"never wait longer than X between attempts,\" so the curve grows and then flattens. It is one min() call in the callback.\nStakes if we pick wrong: without a cap, a worker with a higher attempt count silently turns into a multi-day wait that looks like a stuck job; with a cap, one more number to document per worker.\nRecommendation: A because the cap is one min() and it makes \"how long can this job be delayed\" a question with an answer.\nCompleteness: A=9/10, B=4/10\nPros / cons:\nA) Cap each delay: default 10 min, per-worker override (recommended)\n  \u2705 Worst-case wait between attempts is bounded and documented for every worker (human: ~1 hr / CC: ~5 min)\n  \u2705 Also pins the curve defaults (base 1 s, multiplier 2) so all 5 workers start from the same documented numbers\n  \u274c One more config value per worker to document and keep sane alongside maxAttempts\nB) No cap\n  \u2705 Zero code; the curve is exactly the exponential the plan describes\n  \u2705 Fewer knobs to explain\n  \u274c Any worker that raises maxAttempts past ~12 gets hour-to-day waits nobody intended\nNet: One min() now versus a job that looks stuck for six days the first time someone bumps an attempt count.\nHeader: Delay cap\nOptions:\nA) Cap each delay (recommended)\ndelay = min(jitteredExponential, maxDelay). `maxDelay` default 10 minutes with per-worker override. Curve defaults documented: base 1 s, multiplier 2, per-worker override. Human ~1 hr / CC ~5 min. Completeness 9/10.\nB) No cap\nRaw exponential curve with no ceiling. Zero code, fewer knobs; high attempt counts produce hour-to-day waits. Completeness 4/10.\n\nState: approved\nActual answer: A) Cap each delay \u2014 user answer to D5\nAccepted scope: Every worker's curve callback (D1) computes delay = min(jitteredExponential, maxDelay) with `maxDelay` default 10 minutes and per-worker override. Curve defaults documented: base 1 s, multiplier 2, per-worker override. Tests asserting the cap is honored at high attempt numbers and that defaults apply when a worker sets nothing are common work of this behavior. R3d, R4, R5, R6 unchanged and pending.\nHistory: none\n\n### R3d: Retryable vs fatal error classification (4 non-webhook workers)\nFinding: Section 1 finding 2 (Scope Challenge #3), P2, confidence 7/10, PLAN.md:7-9, reviewer: plan-eng-review (Claude)\nPlan baseline: the plan retries on failure with no distinction between transient and permanent errors (PLAN.md:7-9). D2 already fixed the webhook worker's classification (pre-send vs post-send); this row covers the other 4 workers only. Nothing approved for R3d.\nRuntime evidence: unknown \u2014 no worker source in checkout. Practice: retry timeouts, connection errors, 429/503, DB deadlocks/serialization failures; never retry validation errors, 4xx other than 429, missing records, or programming errors (search check, Section A).\nComparison grid:\n\n| Choice | Current | A) Classify; unknown \u2192 retryable | B) Retry everything to maxAttempts | C) Classify; unknown \u2192 fatal |\n|---|---|---|---|---|\n| R3d error classification (4 non-webhook workers) | unspecified (every error retried) | each worker declares retryable error classes (timeout, connection error, 429/503, deadlock/serialization failure) and fatal classes (validation error, 4xx other than 429, missing record, programming error such as TypeError); fatal \u2192 dead-letter immediately (D3) without consuming attempts; unclassified errors default to retryable | no classification; every error consumes an attempt until `maxAttempts`, then dead-letter (D3) | same declared classes as A; unclassified errors default to fatal \u2192 dead-letter immediately |\n| R1 retry mechanism | approved A (D1) | fixed | fixed | fixed |\n| R2 webhook delivery | approved A (D2) | fixed | fixed | fixed |\n| R3a terminal handling | approved A (D3) | fixed | fixed | fixed |\n| R3b jitter | approved A (D4) | fixed | fixed | fixed |\n| R3c delay cap | approved A (D5) | fixed | fixed | fixed |\n| R4 envelope duplication | pending | pending | pending | pending |\n| R5 regression test | pending | pending | pending | pending |\n| R6 payload refetch / graph cache | pending | pending | pending | pending |\n\nQuestion D6:\nD6 \u2014 Which errors should the four non-webhook workers retry, and which go straight to dead-letter?\nProject/branch/task: main branch, retry-framework plan; error handling inside the 4 non-webhook workers' library retry hooks (D1). The webhook worker's rule is already fixed by D2.\nELI10: Not every failure is worth retrying. A timeout or a \"service busy\" reply will likely pass on the next try. A validation error, a missing record or a bug that throws will fail the same way five times in a row, burning worker time and delaying the dead-letter record (D3) by the whole backoff curve. Classifying errors sends the hopeless ones to dead-letter immediately and spends retries only on the ones that can recover. The open question is what to do with an error nobody has classified yet.\nStakes if we pick wrong: either a bug retries five times per job across a whole queue before anyone sees it, or a transient error that nobody thought to list dead-letters real work on its first failure.\nRecommendation: A because classification is a short list per worker, and defaulting unknown errors to retryable never loses work: the worst case is five wasted attempts, not a dropped job.\nCompleteness: A=10/10, B=5/10, C=8/10\nPros / cons:\nA) Classify; unknown errors retry (recommended)\n  \u2705 Hopeless errors (validation, 4xx, missing record, TypeError) land in dead-letter on attempt 1 with the real cause visible (human: ~half day / CC: ~15 min)\n  \u2705 Unlisted errors still retry, so a forgotten transient class costs attempts, never data\n  \u274c Each worker maintains a small error-class list, and a new fatal class retries needlessly until someone adds it\nB) Retry everything until maxAttempts\n  \u2705 No lists to maintain; identical behavior in all 4 workers (human: ~0 / CC: ~0)\n  \u2705 Impossible to misclassify a transient error as fatal\n  \u274c A deploy with a bug retries every affected job 5 times over the full curve before dead-lettering; queue capacity burns and diagnosis is delayed\nC) Classify; unknown errors are fatal\n  \u2705 Zero wasted attempts on anything not explicitly known to be transient (human: ~half day / CC: ~15 min)\n  \u2705 Dead-letter fills fast, so new error classes surface quickly\n  \u274c Any transient error missing from the list dead-letters real work on its first failure, which is the exact loss the retry framework exists to prevent\nNet: A short list per worker plus a safe default, versus either wasted retries on bugs or lost work on unlisted transients.\nHeader: Error classes\nOptions:\nA) Classify; unknown \u2192 retryable (recommended)\nEach of the 4 workers declares retryable classes (timeout, connection error, 429/503, deadlock/serialization failure) and fatal classes (validation error, 4xx other than 429, missing record, programming error). Fatal \u2192 dead-letter immediately without consuming attempts. Unclassified errors retry. Human ~half day / CC ~15 min. Completeness 10/10.\nB) Retry everything to maxAttempts\nNo classification. Every error consumes an attempt until `maxAttempts`, then dead-letter per D3. Zero code; bugs retry 5 times per job. Completeness 5/10.\nC) Classify; unknown \u2192 fatal\nSame declared classes as A, but unclassified errors go to dead-letter immediately. No wasted attempts; unlisted transient errors lose work on first failure. Human ~half day / CC ~15 min. Completeness 8/10.\n\nState: approved\nActual answer: A) Classify; unknown \u2192 retryable \u2014 user answer to D6\nAccepted scope: Each of the 4 non-webhook workers declares retryable error classes (timeout, connection error, 429/503, deadlock/serialization failure) and fatal classes (validation error, 4xx other than 429, missing record, programming error). Fatal errors go to the dead-letter store (D3) immediately without consuming attempts. Unclassified errors are retryable. Tests for each class direction and the unknown-error default are common work of this behavior. Webhook worker classification stays as fixed by D2. R4, R5, R6 unchanged and pending.\nHistory: none\n\n### R4: Shared retry-policy module vs five inline copies\nFinding: Section 2 finding 1 (Scope Challenge #4), P2, confidence 7/10, PLAN.md:12-14, reviewer: plan-eng-review (Claude)\nPlan baseline: \"The retry envelope (compute delay, log attempt, dispatch) is duplicated across 5 worker files with copy-pasted bodies. We will leave the duplication for now and refactor 'later.'\" (PLAN.md:12-14). Nothing approved for R4.\nRuntime evidence: unknown \u2014 worker files not in checkout. After D1 the library owns dispatch and scheduling; what each worker still supplies is identical behavior by construction: the jittered, capped curve callback (D4, D5), the error classifier shape (D6), the dead-letter handoff (D3) and the attempt log line. Callers: the 5 workers named in PLAN.md:13 \u2014 **proposed callers, labelled as plan assumptions, not verified source**.\nShared-code rubric:\n- Callers: 5 proposed workers (PLAN.md:13). Same behavior required by D3\u2013D6 with only per-worker values (attempts, maxDelay, error lists) differing.\n- Reuse before extracting: the library hooks (D1) are the reuse; the helper is the thin glue that configures them identically.\n- Helper contract (small): `buildBackoff({base=1s, multiplier=2, maxDelay=10min, rng})` \u2192 curve callback; `classify(error, {retryable, fatal})` \u2192 `retryable | fatal`; `toDeadLetter(job, error, attempts)` \u2192 persists record + emits metric; `logAttempt(job, n, delay, errorClass)`; config validation at startup (maxAttempts \u2265 1, maxDelay \u2265 base). Blast radius: a helper bug affects all 5 workers, mitigated by the helper's own tests.\n- Line estimate (ranges, plan-level): inline per worker \u2248 25\u201340 lines \u00d7 5 = 125\u2013200 removed; helper \u2248 60\u201380 added; per-worker config \u2248 8 \u00d7 5 = 40 added. Implementation savings \u2248 15\u201390 lines. Tests: one helper suite (~100 lines) replaces five near-identical suites; total change likely still shrinks, but caller-integration tests may make the first PR grow.\nComparison grid:\n\n| Choice | Current | A) One shared retry-policy module | B) Five inline copies (as planned) | C) Extract curve builder only |\n|---|---|---|---|---|\n| R4 envelope duplication | 5 copy-pasted envelopes, refactor \"later\" | one small module (buildBackoff, classify, toDeadLetter, logAttempt, config validation) used by all 5 workers; each worker keeps only its values | each worker carries its own curve, classifier, dead-letter handoff and log line | shared `buildBackoff` only; classifier, dead-letter handoff and log line stay inline in each worker |\n| R1 retry mechanism | approved A (D1) | fixed | fixed | fixed |\n| R2 webhook delivery | approved A (D2) | fixed | fixed | fixed |\n| R3a terminal handling | approved A (D3) | fixed | fixed | fixed |\n| R3b jitter | approved A (D4) | fixed | fixed | fixed |\n| R3c delay cap | approved A (D5) | fixed | fixed | fixed |\n| R3d error classification | approved A (D6) | fixed | fixed | fixed |\n| R5 regression test | pending | pending | pending | pending |\n| R6 payload refetch / graph cache | pending | pending | pending | pending |\n\nQuestion D7:\nD7 \u2014 One shared retry-policy module, or five copies and \"refactor later\"?\nProject/branch/task: main branch, retry-framework plan; how the 5 workers carry the behavior approved in D3\u2013D6.\nELI10: After D1 the library does the scheduling, but every worker still has to hand it the same four things: a jittered, capped curve, an error classifier, a dead-letter handoff and an attempt log line. The plan copies that block into five files and promises to clean up later. \"Later\" for copy-pasted retry code usually means the fifth copy drifts (no cap, wrong jitter) and nobody notices until an incident. The alternative is one small module that each worker configures with its own numbers and error lists.\nStakes if we pick wrong: five curves that silently disagree, five places to fix the next retry bug, and five test suites that each cover a slightly different subset; or, with a shared module, one bug that hits all five workers at once (mitigated by the module's own tests).\nRecommendation: A because the behavior is identical by construction (D3\u2013D6 fixed it), the module is under 100 lines, and it removes more lines than it adds while making the retry rules testable once.\nCompleteness: A=10/10, B=4/10, C=7/10\nPros / cons:\nA) One shared retry-policy module (recommended)\n  \u2705 Curve, classifier, dead-letter handoff, attempt log and config validation are tested once and behave the same in all 5 workers (human: ~1 day / CC: ~20 min)\n  \u2705 Estimated 15\u201390 implementation lines saved; the helper's test suite replaces five near-duplicate suites\n  \u274c A bug in the module reaches all 5 workers; the module's own tests are the guard\nB) Five inline copies, refactor later (as planned)\n  \u2705 No shared dependency between workers; each can be changed in isolation (human: ~1.5 days / CC: ~30 min)\n  \u2705 Matches the plan text exactly; nothing new to name or place\n  \u274c Five copies to keep in sync and five test suites to write; \"later\" rarely arrives for retry glue\nC) Extract the curve builder only\n  \u2705 The math most likely to drift (jitter + cap) lives in one place (human: ~1 day / CC: ~15 min)\n  \u2705 Smaller shared surface than A\n  \u274c Classifier, dead-letter handoff and log line are still copied five times, so most of the duplication and its tests remain\nNet: One under-100-line module now, or five copies plus a promise.\nHeader: Shared module\nOptions:\nA) One shared retry-policy module (recommended)\nSmall module: buildBackoff (base/multiplier/maxDelay/rng), classify (per-worker retryable/fatal lists), toDeadLetter (persist + metric), logAttempt, and startup config validation. All 5 workers use it with their own values. Human ~1 day / CC ~20 min. Completeness 10/10.\nB) Five inline copies (as planned)\nEach worker carries its own curve, classifier, dead-letter handoff and log line; refactor deferred. Human ~1.5 days / CC ~30 min. Completeness 4/10.\nC) Extract curve builder only\nShared buildBackoff (jitter + cap) only; classifier, dead-letter handoff and log line stay inline in each of the 5 workers. Human ~1 day / CC ~15 min. Completeness 7/10.\n\nState: approved\nActual answer: A) One shared retry-policy module \u2014 user answer to D7\nAccepted scope: One small retry-policy module providing `buildBackoff({base, multiplier, maxDelay, rng})`, `classify(error, {retryable, fatal})`, `toDeadLetter(job, error, attempts)` (persist + metric, and on persist failure: job stays in the library's failed state, error log carries both errors, metric increments), `logAttempt(job, n, delay, errorClass)` and startup config validation (maxAttempts \u2265 1, maxDelay \u2265 base). All 5 workers use it with their own values (attempts, maxDelay, error lists); the webhook worker's pre-send/post-send rule (D2) is its classifier input. The retry state machine diagram lives inline in this module. The module's own test suite plus one integration test per worker are common work of this behavior. R5, R6 unchanged and pending.\nHistory: none\n\n### R5: Regression coverage for `processWebhookJob()` at-most-once delivery\nFinding: Section 3 CRITICAL regression gap (Scope Challenge #5), P1, confidence 8/10, PLAN.md:17-19, reviewer: plan-eng-review (Claude)\nPlan baseline: \"No regression test for the prior at-most-once delivery guarantee is planned.\" (PLAN.md:18-19). Behavior to preserve is fixed by D2: at most one HTTP send per event, ever; retries only on pre-send failures; post-send failures terminal \u2192 dead-letter (D3). Intentional differences: pre-send failures now retry (previously dropped). No approved acceptance assertions or test depth yet.\nRuntime evidence: unknown \u2014 `processWebhookJob()` and its tests are not in checkout; TESTFILES:0, no framework detected. Regression Rule: coverage is required; the question is how, not whether.\nComparison grid:\n\n| Choice | Current | A) Unit + integration through the library hooks | B) Unit tests on the classifier and worker only | C) Integration test only |\n|---|---|---|---|---|\n| R5 regression coverage | none planned | unit: fake transport records every send; pre-send failure \u00d7(maxAttempts\u22121) then success \u2192 exactly 1 send; timeout after send \u2192 0 further sends, dead-letter entry; 5xx \u2192 0 further sends, dead-letter; reset mid-response \u2192 0 further sends; success \u2192 1 send, no dead-letter. Integration: real library hooks + fake receiver, same assertions end to end, plus attempt count persisted across a simulated worker restart | the unit assertions from A against the worker with a fake transport; no run through the real library hooks | the integration run from A only; no isolated unit assertions |\n| R1 retry mechanism | approved A (D1) | fixed | fixed | fixed |\n| R2 webhook delivery | approved A (D2) | fixed | fixed | fixed |\n| R3a\u2013R3d | approved A (D3\u2013D6) | fixed | fixed | fixed |\n| R4 shared module | approved A (D7) | fixed | fixed | fixed |\n| R6 payload refetch / graph cache | pending | pending | pending | pending |\n\nQuestion D8:\nD8 \u2014 How do we prove processWebhookJob() still sends each event at most once?\nProject/branch/task: main branch, retry-framework plan; regression coverage for the rewritten webhook worker (D2 fixed the behavior to keep).\nELI10: The webhook worker is being rewritten and it carries a promise to receivers: an event is never sent twice. D2 kept that promise while adding retries for failures that happen before anything is sent. A rewrite with no test for the promise means the first duplicate email or double charge is found by a customer. The test is straightforward: a fake receiver counts sends, and we assert the count is exactly one across every failure pattern. The question is how deep to go: assertions against the worker alone, a run through the real library hooks, or both.\nStakes if we pick wrong: a retry path nobody tested sends duplicates to non-idempotent receivers, or a hook wiring mistake means pre-send failures never actually retry and the framework quietly does nothing for webhooks.\nRecommendation: A because the unit layer pins each failure class cheaply and the integration layer is the only thing that catches hook wiring and attempt persistence, which is where retry bugs actually live.\nCompleteness: A=10/10, B=7/10, C=7/10\nPros / cons:\nA) Unit + integration through the library hooks (recommended)\n  \u2705 Every failure class (pre-send, timeout, 5xx, reset, success) asserted in isolation with a recording fake transport (human: ~1 day / CC: ~20 min)\n  \u2705 One end-to-end run through the real hooks with a fake receiver catches wiring and attempt-persistence bugs the unit layer cannot see\n  \u274c Two test layers to maintain; the integration test needs the library's test harness or an in-process queue\nB) Unit tests only\n  \u2705 Fast, deterministic, no queue infrastructure in the test run (human: ~half day / CC: ~10 min)\n  \u2705 Pins the classifier and the send-count contract per failure class\n  \u274c Never exercises the real hook registration, so a miswired hook passes tests and never retries in production\nC) Integration test only\n  \u2705 Exercises the real path receivers depend on (human: ~half day / CC: ~10 min)\n  \u2705 Fewer tests to write\n  \u274c Slower, and a failure tells you \"something duplicated\" without pointing at which failure class; edge classes get skipped for time\nNet: Cheap isolated assertions plus one real-path run, versus trusting either layer alone to protect a promise made to external receivers.\nHeader: Webhook regression\nOptions:\nA) Unit + integration (recommended)\nUnit: fake transport records every send; assert exactly 1 send after pre-send retries, 0 further sends after timeout/5xx/reset with a dead-letter entry, 1 send on success. Integration: real library hooks + fake receiver, same assertions, plus attempt count survives a simulated worker restart. Human ~1 day / CC ~20 min. Completeness 10/10.\nB) Unit tests only\nThe unit assertions from A against the worker with a fake transport; no run through the real library hooks. Human ~half day / CC ~10 min. Completeness 7/10.\nC) Integration test only\nThe integration run from A only; no isolated per-failure-class assertions. Human ~half day / CC ~10 min. Completeness 7/10.\n\nState: approved\nActual answer: A) Unit + integration \u2014 user answer to D8\nAccepted scope: CRITICAL regression contract for `processWebhookJob()`: behavior preserved = at most one HTTP send per event, ever (D2); intentional change = pre-send failures now retry. Unit tests with a recording fake transport assert: pre-send failure \u00d7(maxAttempts\u22121) then success \u2192 exactly 1 send; timeout after send \u2192 0 further sends and a dead-letter entry; 5xx after send \u2192 0 further sends and a dead-letter entry; connection reset mid-response \u2192 0 further sends; success \u2192 1 send and no dead-letter. Integration test through the real library hooks with a fake receiver repeats those assertions end to end and asserts the attempt count survives a simulated worker restart mid-curve. R6 unchanged and pending.\nHistory: none\n\n### R6: Payload refetch and dependency-graph recompute on every retry\nFinding: Section 4 finding 1 (Scope Challenge #6), P2, confidence 6/10, PLAN.md:22-24, reviewer: plan-eng-review (Claude)\nPlan baseline: \"On every retry we re-fetch the full job payload from the database, then iterate the payload to recompute the dependency graph. Could cache the graph on the first attempt; not planned.\" (PLAN.md:22-24). Nothing approved for R6.\nRuntime evidence: unknown \u2014 no worker source, payload sizes or timings in checkout. Under D1 the \"re-fetch\" is the library dequeuing the job for the attempt, so it is not extra work; the graph recompute is extra CPU, bounded to `maxAttempts` (default 5, D3) per failing job. Failing jobs are the minority; a cache written on the first attempt taxes every successful job to save work on the failing few. Medium confidence, verify with measurements.\nComparison grid:\n\n| Choice | Current | A) Persist the graph with the job | B) In-process memo (LRU by job id + payload hash) | C) Measure first, no cache |\n|---|---|---|---|---|\n| R6 payload refetch / graph cache | recompute on every attempt | compute once on attempt 1, store the graph beside the job row, reuse on retries, invalidate on payload version change | memoize per worker instance, bounded LRU; hits only when the same instance runs the retry | no cache; add per-attempt timing metrics (job load ms, graph compute ms, payload bytes) and a p95 budget; revisit with data |\n| R1\u2013R5 | approved A (D1\u2013D8) | fixed | fixed | fixed |\n\nQuestion D9:\nD9 \u2014 Cache the dependency graph across retries now, or measure first?\nProject/branch/task: main branch, retry-framework plan; per-attempt cost inside the 5 workers running through library hooks (D1).\nELI10: The plan worries that each retry reloads the job and rebuilds its dependency graph from scratch. After D1 the reload is just the library handing the job to the worker, which happens anyway. The rebuild is real extra CPU, but only on retries, and D3 caps those at 5 per failing job. Storing the graph on the first attempt would add a write to every job, including the large majority that succeed first time, to save work on the few that fail. Nobody has measured how long the rebuild takes.\nStakes if we pick wrong: either we add a write and a staleness risk to every job to fix a cost nobody measured, or a genuinely slow rebuild keeps burning worker time on retries and we only find out under load.\nRecommendation: C because an unmeasured optimization that taxes the happy path is the wrong trade; two timing metrics make the real decision cheap and data-driven.\nNote: options differ in kind (persisted cache vs in-process memo vs measure first) \u2014 no completeness score.\nPros / cons:\nA) Persist the graph with the job on attempt 1\n  \u2705 Retries never recompute; cost is paid once per job regardless of which worker instance retries (human: ~1 day / CC: ~20 min)\n  \u2705 Simple to reason about once the invalidation rule (payload version) is in place\n  \u274c Adds a write and stored blob to every job, including the ones that never retry; stale-graph bugs if the payload changes between attempts\nB) In-process memo (bounded LRU)\n  \u2705 No persistence, no schema change; a few lines around the graph builder (human: ~2 hr / CC: ~10 min)\n  \u2705 Zero cost on the happy path beyond a map insert\n  \u274c Retries after a 10-minute delay usually land on a different worker instance, so the hit rate is low and unpredictable\nC) Measure first: timing metrics + p95 budget (recommended)\n  \u2705 Two metrics (graph compute ms, payload bytes) per attempt tell you whether this is 2 ms or 2 s before anyone writes cache code (human: ~1 hr / CC: ~5 min)\n  \u2705 No happy-path cost, no staleness risk, and the retry-policy module already logs per attempt (D7) so the hook point exists\n  \u274c If the rebuild is genuinely slow, retries stay expensive until the follow-up lands\nNet: Add a write to every job to save CPU on the few that retry, or spend an hour on metrics and decide with numbers.\nHeader: Graph cache\nOptions:\nA) Persist the graph with the job\nCompute once on attempt 1, store the graph beside the job row, reuse on retries, invalidate when the payload version changes. Adds a write to every job. Human ~1 day / CC ~20 min.\nB) In-process memo (bounded LRU)\nMemoize the graph per worker instance keyed by job id + payload hash, bounded LRU. No persistence; low hit rate when retries land on another instance. Human ~2 hr / CC ~10 min.\nC) Measure first (recommended)\nNo cache. Add per-attempt timing metrics (job load ms, graph compute ms, payload bytes) via the retry-policy module's attempt log, set a p95 budget, and revisit caching with data. Human ~1 hr / CC ~5 min.\n\nState: approved\nActual answer: A) Persist the graph with the job \u2014 user answer to D9 (not the recommended option; user's call)\nAccepted scope: Compute the dependency graph once on attempt 1 and persist it beside the job row (column or side table keyed by job id) together with the payload version. On retry, load the stored graph when the payload version matches; otherwise recompute and overwrite. A failed graph read falls back to recompute and never blocks a retry. Schema migration is part of the work. Tests (common work): first attempt writes the graph; retry reuses it without recompute; payload version change invalidates and recomputes; graph read failure falls back to recompute. No timing metrics (option C not chosen). No other choice changed.\nHistory: none\n\n### T1: TODO \u2014 dead-letter retention / purge policy\nFinding: Section 4 finding 2, P3, confidence 5/10, no plan line (gap in D3's approved dead-letter store), reviewer: plan-eng-review (Claude)\nPlan baseline: D3 approved a dead-letter store with no retention or purge. Nothing approved for T1.\nRuntime evidence: unknown \u2014 store does not exist yet. Growth rate = failed jobs only; slow, unbounded.\nTODO record:\n- **What:** a scheduled purge of dead-letter entries older than a configurable retention (default 90 days), skipping entries flagged keep.\n- **Why:** the store grows without bound; a year of failed jobs becomes a slow query behind the alert and the replay UI.\n- **Pros:** bounded storage; predictable query cost; a clear answer to \"how long do we keep failed jobs.\"\n- **Cons:** purging deletes the only record of lost work; wrong default destroys evidence; one more scheduled job to run.\n- **Context:** dead-letter store is new in this PR (D3); growth only matters after months. Start in the retry-policy module's dead-letter code; add a scheduled job and a config value.\n- **Depends on / blocked by:** D3 dead-letter store shipped; retention period agreed with whoever owns incident evidence.\nComparison grid:\n\n| Choice | Current | A) Add to TODOS.md | B) Skip | C) Build now in this PR |\n|---|---|---|---|---|\n| T1 dead-letter retention | none | tracked TODO with trigger: build when the store passes 10k rows or at 3 months, whichever first | not tracked | scheduled purge job, retention config default 90 days, keep flag, tests, in this PR |\n| R1\u2013R6 | approved (D1\u2013D9) | fixed | fixed | fixed |\n\nQuestion D10:\nD10 \u2014 Track dead-letter retention as a TODO, skip it, or build it now?\nProject/branch/task: main branch, retry-framework plan; follow-up to the dead-letter store approved in D3.\nELI10: The dead-letter store keeps every job that ran out of retries or hit a fatal error. Nothing ever removes them. That is fine for months, then the table is large, the growth alert query slows, and nobody remembers why. A purge job with a retention period fixes it, but the retention period is a judgment call about how long failed-job evidence must stay around.\nStakes if we pick wrong: build it now with the wrong retention and you delete evidence of lost work; skip it and the store becomes an unbounded table someone discovers during an incident.\nRecommendation: A because the store is new, growth is slow, and the retention period deserves an owner's answer rather than a default picked inside a retry PR; the TODO carries a concrete trigger.\nCompleteness: A=6/10, B=2/10, C=10/10\nPros / cons:\nA) Add to TODOS.md with a trigger (recommended)\n  \u2705 Keeps this PR right-sized: the retry framework ships without a retention debate attached\n  \u2705 Trigger (10k rows or 3 months) means the TODO fires before growth matters (human: ~5 min / CC: ~1 min now)\n  \u274c Unbounded growth until someone acts on the TODO; the ceiling is a slow query, not data loss\nB) Skip\n  \u2705 Nothing to track or build\n  \u2705 Zero effort now\n  \u274c The store grows forever with no record that anyone considered it\nC) Build now in this PR\n  \u2705 Store ships bounded from day one: purge job, 90-day default, keep flag, tests (human: ~2 hr / CC: ~10 min)\n  \u2705 No follow-up to forget\n  \u274c Expands this PR with a scheduled job and a retention default nobody has agreed to; deletes evidence if the default is wrong\nNet: A tracked follow-up with a trigger, versus a bigger PR that guesses how long failed-job evidence should live.\nHeader: DLQ retention\nOptions:\nA) Add to TODOS.md (recommended)\nRecord the TODO (what/why/pros/cons/context/depends-on) with trigger: build when the dead-letter store passes 10k rows or at 3 months, whichever first. Human ~5 min / CC ~1 min. Completeness 6/10.\nB) Skip \u2014 not valuable enough\nDo not track retention. Completeness 2/10.\nC) Build it now in this PR\nScheduled purge job, retention config default 90 days, keep flag, tests, shipped with the dead-letter store. Human ~2 hr / CC ~10 min. Completeness 10/10.\n\nState: approved\nActual answer: A) Add to TODOS.md \u2014 user answer to D10 (accepted shortcut, completeness 6/10; logged with ceiling and trigger)\nAccepted scope: TODO \"dead-letter retention / purge policy\" with the record above, trigger: build when the dead-letter store passes 10k rows or at 3 months after the store ships (2026-12-29 at the latest if it ships now), whichever first. Ceiling: unbounded table growth until then; slow query, no data loss. TODOS.md is not writable in plan mode: content presented as **not persisted** in the report. When implementing D3, mark the dead-letter persist site with `gstack-shortcut(dec-<id>): unbounded growth, upgrade when store > 10k rows or 3 months`. No implementation approved.\nHistory: none\n\n### T2: TODO \u2014 stable webhook event id header and opt-in at-least-once delivery\nFinding: Section 1 finding 1 follow-up (D2 chose at-most-once), P3, confidence 6/10, PLAN.md:17-19, reviewer: plan-eng-review (Claude)\nPlan baseline: D2 approved at-most-once for `processWebhookJob()`; post-send failures drop the event into dead-letter. Nothing approved for T2.\nRuntime evidence: unknown \u2014 webhook payload/headers not in checkout. Industry practice: stable per-event id header (Stripe `id`, Shopify `X-Shopify-Webhook-Id`, Svix `webhook-id`) plus at-least-once delivery.\nTODO record:\n- **What:** add a stable per-event id header to every webhook delivery, then offer receivers an opt-in at-least-once mode (retry post-send failures) once they dedupe on that id.\n- **Why:** under D2, every timeout or 5xx after send still loses the event; the standard cure is at-least-once with an idempotency key, which D2 declined for now.\n- **Pros:** closes the remaining webhook loss path; matches what receivers expect from major providers; the header alone is harmless under at-most-once.\n- **Cons:** contract change requiring receiver communication and docs; per-receiver opt-in adds a config dimension to the webhook worker.\n- **Context:** start from the D2 decision record and the dead-letter entries with post-send failure class; those counts show how much is being lost. Header first, semantics second.\n- **Depends on / blocked by:** D2 decision (would be superseded for opted-in receivers); D3 dead-letter store for measuring loss; receiver docs channel.\nComparison grid:\n\n| Choice | Current | A) Add to TODOS.md | B) Skip | C) Ship the header now, semantics later |\n|---|---|---|---|---|\n| T2 webhook event id + at-least-once opt-in | none | tracked TODO with trigger: post-send dead-letter entries exceed 1% of webhook sends in any week, or a receiver requests redelivery | not tracked | stable event id header added to every webhook in this PR (no semantics change, D2 intact); at-least-once opt-in stays a TODO |\n| R2 webhook delivery | approved A (D2) | fixed | fixed | fixed (header only, no retry-after-send) |\n| R1, R3\u2013R6, T1 | approved (D1, D3\u2013D10) | fixed | fixed | fixed |\n\nQuestion D11:\nD11 \u2014 Track the webhook event-id header and at-least-once opt-in as a TODO, skip it, or ship the header now?\nProject/branch/task: main branch, retry-framework plan; follow-up to D2 (webhook worker stays at-most-once).\nELI10: D2 kept the promise that a webhook is never sent twice, which means a send that times out is still lost. The usual fix is to stamp every event with a stable id so receivers can ignore duplicates, and then retry freely. That is a contract change, so it was declined for this PR. The question is whether to track it, drop it, or at least ship the harmless id header now so receivers can start deduping before the semantics ever change.\nStakes if we pick wrong: lost webhook events keep landing in dead-letter with no plan to stop the loss; or a header change rides along in a retry PR without receiver communication.\nRecommendation: A because this is a receiver-facing contract change that deserves its own PR and docs, and the dead-letter store (D3) will produce the loss numbers that justify it; the trigger is concrete.\nCompleteness: A=6/10, B=2/10, C=8/10\nPros / cons:\nA) Add to TODOS.md with a trigger (recommended)\n  \u2705 Keeps the retry PR free of webhook contract changes; the TODO fires on measured loss (human: ~5 min / CC: ~1 min now)\n  \u2705 Dead-letter counts of post-send failures give the case for it with real numbers\n  \u274c Webhook events lost to timeouts stay lost until the TODO is acted on\nB) Skip\n  \u2705 Nothing to track\n  \u2705 Zero effort now\n  \u274c No record that at-most-once was a deliberate trade with a known cost\nC) Ship the stable event-id header now, at-least-once later\n  \u2705 Receivers can start deduping today; the header is harmless under at-most-once (human: ~2 hr / CC: ~10 min)\n  \u2705 Makes the eventual semantics change a config flip instead of a payload change\n  \u274c Adds a webhook payload change and receiver docs to a retry PR; still needs the TODO for the semantics\nNet: Track it with a loss-based trigger, or ship a small header change now inside a PR about retries.\nHeader: Webhook TODO\nOptions:\nA) Add to TODOS.md (recommended)\nRecord the TODO with trigger: post-send dead-letter entries exceed 1% of webhook sends in any week, or a receiver requests redelivery. Human ~5 min / CC ~1 min. Completeness 6/10.\nB) Skip \u2014 not valuable enough\nDo not track. Completeness 2/10.\nC) Ship the header now\nAdd a stable per-event id header to every webhook delivery in this PR; D2 semantics unchanged; at-least-once opt-in remains a TODO. Human ~2 hr / CC ~10 min. Completeness 8/10.\n\nState: approved\nActual answer: A) Add to TODOS.md \u2014 user answer to D11 (accepted shortcut, completeness 6/10; logged with ceiling and trigger)\nAccepted scope: TODO \"stable webhook event id header + opt-in at-least-once delivery\" with the record above, trigger: post-send dead-letter entries exceed 1% of webhook sends in any week, or a receiver requests redelivery. Ceiling: webhook events lost to post-send failures stay lost (parked in dead-letter, D3). TODOS.md not writable in plan mode: content presented as **not persisted**. No implementation approved; D2 unchanged.\nHistory: none\n\n**Approval readiness: PASS** \u2014 checked R1 (D1=A), R2 (D2=A), R3a (D3=A), R3b (D4=A), R3c (D5=A), R3d (D6=A), R4 (D7=A), R5 (D8=A), R6 (D9=A), T1 (D10=A), T2 (D11=A). Every accepted remedy cites its own user answer; the R5 regression contract is approved with explicit assertions; no deferral is unresolved. Total elapsed retry window: considered, not a choice (bounded by D3 maxAttempts \u00d7 D5 maxDelay \u2248 50 min at defaults).\n\n## Working plan (after review)\n\n1. **Mechanism (D1):** all 5 workers retry through the job library's built-in retry hooks. No custom inline scheduler. Reopen only if the hook API cannot take a custom curve callback.\n2. **Shared retry-policy module (D7):** `buildBackoff`, `classify`, `toDeadLetter`, `logAttempt`, `validateConfig`; retry state-machine diagram inline. Each worker supplies its values only.\n3. **Curve (D4, D5):** full jitter, delay = min(random(0, base\u00b72^n), maxDelay); defaults base 1 s, multiplier 2, maxDelay 10 min; injectable RNG; per-worker override.\n4. **Terminal handling (D3):** maxAttempts default 5, per-worker override; exhaustion or fatal error \u2192 dead-letter store (last error, attempt history, payload ref); metric + alert on growth; manual replay; persist failure keeps the job failed and logs both errors.\n5. **Error classes (D6):** 4 non-webhook workers declare retryable and fatal classes; fatal \u2192 dead-letter immediately; unknown \u2192 retryable.\n6. **Webhook worker (D2):** at-most-once preserved. Retry only pre-send failures (refused, DNS, TLS, local). Post-send timeout/5xx/reset \u2192 terminal, dead-letter, no re-send. Webhook replay from dead-letter warns it may duplicate.\n7. **Regression proof (D8):** unit tests with a recording fake transport per failure class + integration through the real hooks with a fake receiver and a simulated restart. CRITICAL.\n8. **Graph persistence (D9):** compute once on attempt 1, persist beside the job with payload version; reuse on retry; invalidate on version change; read failure \u2192 recompute. Schema migration + 4 tests.\n9. **TODOs (D10, D11):** dead-letter retention; webhook event-id header + at-least-once opt-in. Not persisted (plan mode).\n\n## NOT in scope\n- **Idempotency key / at-least-once webhooks:** deferred to TODO (D11); D2 chose at-most-once for this PR.\n- **Dead-letter retention / purge:** deferred to TODO (D10); store is new and growth is slow.\n- **Total elapsed retry window:** not needed; D3 \u00d7 D5 bounds the window at ~50 min with defaults.\n- **Graph timing metrics (R6 option C):** not chosen; D9 persists the graph instead.\n- **Custom inline scheduler (PLAN.md:7-9):** replaced by library hooks (D1).\n- **Per-receiver circuit breaker for webhooks:** not raised in the plan; separate scope if post-send failures cluster by receiver.\n\n## What already exists\n- **Reused:** the job library's retry hooks and attempt-count persistence (D1). Unverified in this checkout; the plan itself states the library version has the same shape (PLAN.md:8).\n- **Assumed existing, unverified:** structured logging and a metrics/alerting pipeline for the dead-letter growth alert (D3).\n- **New:** retry-policy module (D7, shared-code rubric evidence in R4), dead-letter store + replay path (D3), graph persistence column/table + migration (D9).\n- **Rebuilt:** nothing. The custom scheduler from the original plan is dropped.\n\n## Diagrams\n- Retry state machine: Section 1 above; goes inline in the retry-policy module (D7).\n- Coverage diagram: Section 3 above.\n- Files needing inline diagrams once written: the retry-policy module (state machine); the webhook worker (pre-send vs post-send failure split, D2).\n\n## Failure modes\n| New path | Realistic production failure | Test coverage (approved) | Error handling (approved) | User-visible? |\n|---|---|---|---|---|\n| Hook wiring (D1) in each worker | hook registered wrong \u2192 no retries ever happen | integration test per worker (Section 3), webhook integration (D8) | n/a, caught by tests | silent without tests \u2192 covered |\n| Dead-letter persist (D3) | dead-letter store down while a job exhausts | unit test: persist failure path | job stays failed; both errors logged; metric | operator sees log + metric |\n| Webhook receiver down 2 h (D2) | every send times out after write | unit + integration per failure class | terminal \u2192 dead-letter, no re-send; alert on growth | operator alert; receiver gets no duplicate |\n| Poisoned job (D3, D6) | throws the same error forever | exhaustion test; fatal-class test | fatal \u2192 dead-letter on attempt 1; else at maxAttempts | dead-letter entry with cause |\n| Graph read (D9) | stored graph corrupt or missing | fallback test | recompute, never block the retry | none |\n| Worker crash mid-curve (D1) | process killed between attempts | integration restart test (D8) | library persists attempt count | none |\n| Config error (D7) | maxAttempts 0 or maxDelay < base | validateConfig tests | fail fast at startup | deploy fails loudly |\n\n**Critical gaps flagged: 0** (every new path has both approved test coverage and approved error handling).\n\n## Worktree parallelization strategy\n\n| Step | Modules touched | Depends on |\n|------|----------------|------------|\n| S1 retry-policy module + unit tests | retry-policy (new) | \u2014 |\n| S2 dead-letter store, metric, alert, replay | dead-letter store (new), migrations, metrics | \u2014 (agree `toDeadLetter` signature with S1 first) |\n| S3 graph persistence + migration + tests | job storage / migrations, graph builder | \u2014 |\n| S4 wire 4 non-webhook workers + integration tests | workers/, retry-policy | S1, S2 |\n| S5 webhook worker rewrite + regression tests | workers/ (webhook), retry-policy | S1, S2 |\n| S6 config defaults, docs, inline diagram | retry-policy, docs | S1 |\n\n**Parallel lanes:** Lane A: S1 \u2192 S4 + S5 + S6 (S4/S5 touch disjoint worker files) \u00b7 Lane B: S2 \u00b7 Lane C: S3.\n**Execution order:** Launch A(S1) + B + C. Merge all three. Then S4, S5, S6 in parallel. Merge.\n**Conflict flags:** S1/S2 share the `toDeadLetter` contract \u2014 fix the signature before launching. S3 and S4/S5 both touch each worker's job-load path \u2014 land S3 first or coordinate the graph-load call site.\n\n## Implementation Tasks\nSynthesized from this review's findings. Each task derives from a specific finding above. Run with Claude Code or Codex; checkbox as you ship.\n\n- [ ] **T1 (P1, human: ~1 day / CC: ~20 min)** \u2014 retry-policy module \u2014 Build `buildBackoff` (full jitter, cap, defaults), `classify`, `toDeadLetter` (persist + metric + persist-failure handling), `logAttempt`, `validateConfig`, inline state-machine diagram, unit tests for every branch\n  - Surfaced by: Code quality \u2014 R4/D7 \"duplicated across 5 worker files\"; Section 1 \u2014 R3b/R3c/R3d\n  - Files: retry-policy module (new; path not in checkout)\n  - Verify: module unit suite green: jitter bounds, cap at high n, defaults, class directions, unknown \u2192 retryable, persist-failure path, config validation\n- [ ] **T2 (P1, human: ~1.5 days / CC: ~30 min)** \u2014 dead-letter store \u2014 Schema + write path (last error, attempt history, payload ref), growth counter metric + alert, manual replay path, webhook replay \"may duplicate\" warning, `gstack-shortcut` marker for retention (D10)\n  - Surfaced by: Architecture \u2014 R3a/D3 \"no attempt limit or terminal outcome\"; Performance \u2014 alert reads a counter, not COUNT(*)\n  - Files: dead-letter store + migration (new), metrics config\n  - Verify: exhaustion \u2192 entry; fatal \u2192 entry on attempt 1; alert fires on growth; replay re-enqueues once; webhook replay shows warning\n- [ ] **T3 (P1, human: ~1.5 days / CC: ~30 min)** \u2014 webhook worker \u2014 Rewrite `processWebhookJob()` on library hooks with pre-send vs post-send classification; at-most-once preserved; CRITICAL regression tests (unit fake transport + integration fake receiver + restart)\n  - Surfaced by: Tests \u2014 R5/D8 \"No regression test for the prior at-most-once delivery guarantee\"; Architecture \u2014 R2/D2\n  - Files: workers/ webhook worker (path not in checkout), its tests\n  - Verify: exactly 1 send after pre-send retries; 0 further sends after timeout/5xx/reset with dead-letter entry; attempt count survives restart\n- [ ] **T4 (P1, human: ~1.5 days / CC: ~30 min)** \u2014 4 non-webhook workers \u2014 Wire each to library hooks with module config (attempts, maxDelay, retryable/fatal lists); remove inline envelopes; one integration test per worker\n  - Surfaced by: Scope Challenge \u2014 R1/D1 \"roll a custom exponential-backoff scheduler inline\"; Section 1 \u2014 R3d/D6\n  - Files: workers/ (4 files; paths not in checkout)\n  - Verify: transient \u2192 retried with module delay; fatal \u2192 dead-letter, no attempts consumed; exhaustion \u2192 dead-letter; deleted job row \u2192 fatal\n- [ ] **T5 (P2, human: ~1 day / CC: ~20 min)** \u2014 graph persistence \u2014 Migration for graph + payload version beside the job; store on attempt 1; reuse on retry; invalidate on version change; read failure \u2192 recompute\n  - Surfaced by: Performance \u2014 R6/D9 \"re-fetch the full job payload ... recompute the dependency graph\"\n  - Files: job storage migration (new), graph builder call site in workers/\n  - Verify: 4 tests: first-attempt write, retry reuse, version invalidation, read-failure fallback\n- [ ] **T6 (P2, human: ~2 hr / CC: ~10 min)** \u2014 config + docs \u2014 Document defaults (base 1 s, \u00d72, maxDelay 10 min, maxAttempts 5) and per-worker overrides; document at-most-once webhook semantics and replay caveat\n  - Surfaced by: Architecture \u2014 R3a\u2013R3c/D3\u2013D5; R2/D2\n  - Files: retry-policy module docs, worker config, README/runbook\n  - Verify: docs review; `validateConfig` rejects bad values at startup\n- [ ] **T7 (P3, human: ~10 min / CC: ~2 min)** \u2014 TODOS.md \u2014 Add the two accepted TODOs (dead-letter retention; webhook event-id header + at-least-once opt-in) with triggers\n  - Surfaced by: TODOS.md updates \u2014 D10, D11 (not persisted in plan mode)\n  - Files: TODOS.md (new)\n  - Verify: entries present with What/Why/Pros/Cons/Context/Depends-on\n\nEffort ratios assumed: features ~30x, tests ~50x, bug fix with regression ~20x, architecture ~5x.\n\n## TODOS.md content (accepted, **not persisted** \u2014 plan mode forbids repo writes)\n1. **Dead-letter retention / purge policy** \u2014 record in ledger T1 (D10). Trigger: store > 10k rows or 3 months after ship.\n2. **Stable webhook event-id header + opt-in at-least-once** \u2014 record in ledger T2 (D11). Trigger: post-send dead-letter entries > 1% of webhook sends in any week, or a receiver requests redelivery.\n\n## Unresolved decisions that may bite you later\nNone. All 11 decisions (D1\u2013D11) answered.\n\n## Suppressed findings (appendix, confidence \u2264 4)\n- `[P3] (confidence: 4/10)` \u2014 total elapsed retry window bound; bounded in practice by D3 \u00d7 D5. Not promoted.\n- `[P3] (confidence: 3/10)` \u2014 per-receiver circuit breaker for the webhook worker; no evidence of receiver clustering in the plan. Listed under NOT in scope.\n\n## Completion summary\n- Step 0: Scope Challenge \u2014 scope accepted as-is (mechanism changed to library hooks per D1; no feature cut)\n- Architecture Review: 3 issues found\n- Code Quality Review: 4 issues found\n- Test Review: diagram produced, 27 gaps identified\n- Performance Review: 3 issues found\n- NOT in scope: written\n- What already exists: written\n- TODOS.md updates: 2 items proposed to user (both accepted; not persisted)\n- Failure modes: 0 critical gaps flagged\n- Unresolved decisions: 0 in this review\n- Outside voice: provider codex, disabled (codex_reviews disabled; no native replacement dispatched)\n- Parallelization: 3 lanes, 3 parallel / 2 sequential steps (S1 \u2192 S4/S5/S6)\n- Lake Score: 4/8 = 10/10 choices / answered coverage choices (D3, D6, D7, D8 at 10/10; D1, D5 at 9/10; D10, D11 at 6/10 accepted shortcuts; D2, D4, D9 were kind choices, excluded)\n- Test Plan Artifact: `~/.gstack/projects/gstack-plan-count-vrYrwf/user-main-eng-review-test-plan-20260929-200336.md`\n- Implementation Tasks JSONL: `~/.gstack/projects/gstack-plan-count-vrYrwf/tasks-eng-review-20260929-200911.jsonl` (7 tasks)\n\n## GSTACK REVIEW REPORT\n\n| Review | Trigger | Why | Runs | Status | Findings |\n|--------|---------|-----|------|--------|----------|\n| CEO Review | `/plan-ceo-review` | Scope & strategy | 0 | \u2014 | \u2014 |\n| Outside Review | codex via `/plan-eng-review` Outside Voice (host: claude, phase: plan-review) | Independent 2nd opinion | 1 | DISABLED | skipped \u2014 codex_reviews disabled |\n| Eng Review | `/plan-eng-review` | Architecture & tests (required) | 1 | ISSUES OPEN | 37 issues, 0 critical gaps (this run; logged at finish step 3) |\n| Design Review | `/plan-design-review` | UI/UX gaps | 0 | \u2014 | \u2014 |\n| DX Review | `/plan-devex-review` | Developer experience gaps | 0 | \u2014 | \u2014 |\n\n**OUTSIDE COVERAGE:** provider codex, phase plan-review, outside_status disabled (codex_reviews disabled in gstack config), no findings; no native replacement dispatched. Re-enable: `gstack-config set codex_reviews enabled`.\n\n**VERDICT:** No reviews CLEARED. Eng Review is ISSUES OPEN: 37 mapped issues (3 architecture, 4 code quality, 27 test gaps, 3 performance), all resolved into decisions D1\u2013D11 and tasks T1\u2013T7, 0 critical gaps \u2014 eng review required.\n\nNO UNRESOLVED DECISIONS\n"
}