gstack/test/fixtures/eng-batching-expanded-ledger-6714.json

{
  "source": "67147822f55b911c033617f759dc472d0d348e72",
  "provenance": {
    "calls": "Exact native public questions, options and acknowledgments from ongoing attempt. No reasoning/signature content.",
    "preAskPlan": "Exact successful Write input at12:33:49.764Z before D4; success12:33:51.480Z. Not synthetic.",
    "currentSavedPlan": "Actual report bytes read at 2026-09-15T12-36-42.857Z; includes decisions answered by this point, not earlier file proof.",
    "publicProjectionPath": "/home/vercel-sandbox/gstack/.context/sep15-ship-consolidation/eng-67147822-monitor/skill-e2e-plan-eng-multi-finding-batching/plan-eng-review-1789475406894-uIGNkU/2026-09-15T12-36-42.857Z/public-events.json",
    "publicProjectionSha256": "fe38a89fb2f396ac2c4bffd1f2678c15b19920bc0c671650e122a227c33729b7",
    "currentSavedPlanSha256": "1d33ed3aa09de491b3b4fb881aedf3ead0556b26c6edc3416d75b298e3a4c34f"
  },
  "call": {
    "sessionId": "bcd8b1ed-2df1-4b34-8ef2-3d5e95f8868d",
    "toolUseId": "toolu_01TGWGFnLMzvtiYHx82R2GXX",
    "questions": [
      {
        "question": "D4 \u2014 Library retry hooks or a custom inline scheduler?\nProject/branch/task: main branch, PLAN.md 'Add background job retry framework'.\nELI10: The job library already knows how to re-run a failed job later; it just asks you 'how long should I wait?' via a hook. The plan skips that and writes its own wait-and-rerun loop inside each of the five workers. You can get the exact same custom curve by answering the library's question with your own function. Writing your own loop means you also own attempt counting, persistence across worker restarts, and every edge case the library already fixed.\nStakes if we pick wrong: A homegrown scheduler that loses retry state on worker restart or double-fires after a crash is a production incident that the library path cannot have.\nRecommendation: A because it gives the same curve control (the plan's stated reason) with one function instead of five schedulers, and it inherits the library's crash and restart semantics. [Layer 1]\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: one function on a proven hook vs. five hand-rolled schedulers you must keep correct under crashes.",
        "header": "Scheduler",
        "multiSelect": false,
        "options": [
          {
            "label": "Library hook + shared backoff fn (recommended)",
            "description": "(human: ~1 day / CC: ~20 min) \u2705 Full curve control through the hook's return value; attempt counting and persistence stay with the library. \u2705 Deletes the need for five inline schedulers; the retry bug surface shrinks to one pure function. \u274c Bounded by what the hook exposes (typically delay, fail-now, retry-now); exotic per-job scheduling needs a workaround."
          },
          {
            "label": "Custom inline scheduler, as planned",
            "description": "(human: ~3 days / CC: ~1 hr, plus ongoing maintenance) \u2705 No dependency on the library's hook contract; any scheduling behavior is possible. \u2705 Zero learning curve for the library's retry options. \u274c You own crash recovery, attempt persistence, and duplicate-fire prevention across restarts, in five places."
          },
          {
            "label": "Probe the library hook API first",
            "description": "(human: ~2 hrs / CC: ~5 min) \u2705 Replaces my web-evidence assumption with the actual hook signature before committing. \u2705 Cheap; no implementation change approved yet. \u274c Delays the architecture call; if the hook supports a custom function (likely), this is a detour."
          }
        ]
      }
    ],
    "answered": true,
    "failed": false,
    "answers": {
      "D4 \u2014 Library retry hooks or a custom inline scheduler?\nProject/branch/task: main branch, PLAN.md 'Add background job retry framework'.\nELI10: The job library already knows how to re-run a failed job later; it just asks you 'how long should I wait?' via a hook. The plan skips that and writes its own wait-and-rerun loop inside each of the five workers. You can get the exact same custom curve by answering the library's question with your own function. Writing your own loop means you also own attempt counting, persistence across worker restarts, and every edge case the library already fixed.\nStakes if we pick wrong: A homegrown scheduler that loses retry state on worker restart or double-fires after a crash is a production incident that the library path cannot have.\nRecommendation: A because it gives the same curve control (the plan's stated reason) with one function instead of five schedulers, and it inherits the library's crash and restart semantics. [Layer 1]\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: one function on a proven hook vs. five hand-rolled schedulers you must keep correct under crashes.": "Library hook + shared backoff fn (recommended)"
    },
    "unansweredQuestionIndices": [],
    "answeredAt": "2026-09-15T12:34:08.932Z"
  },
  "preAskPlan": "# Eng Review: Add background job retry framework\n\nReviewed target: `PLAN.md` (\"Plan: Add background job retry framework\"), repo `gstack-plan-count-mVowtS`, branch `main`, commit `6d4d07b`.\nReview: `/plan-eng-review`, 2026-09-15. Session `1332688-1789475401-070cb1ba`.\n\nEvidence limits: this repo contains only `PLAN.md` and `CLAUDE.md`. The 5 worker files, `processWebhookJob()`, and the job library named in the plan are not present here, so every finding is grounded in plan text (`PLAN.md:line`) and stated as a proposed-work risk, not an observed bug. Confidence is capped accordingly.\n\n## Original plan (unchanged)\n\n# Plan: Add background job retry framework\n\n## Architecture\nWe'll roll a custom exponential-backoff scheduler inline in each worker\nrather than use the existing job library's built-in retry hooks. Same\nshape as the library version, but we want full control over the curve.\n\n## Code quality\nThe retry envelope (compute delay, log attempt, dispatch) is duplicated\nacross 5 worker files with copy-pasted bodies. We will leave the\nduplication for now and refactor \"later.\"\n\n## Tests\nThe existing `processWebhookJob()` flow gets rewritten as part of this\nchange. No regression test for the prior at-most-once delivery guarantee\nis planned.\n\n## Performance\nOn every retry we re-fetch the full job payload from the database, then\niterate the payload to recompute the dependency graph. Could cache the\ngraph on the first attempt; not planned.\n\n## Step 0: Scope Challenge\n\n1. **What already exists:** the job library's built-in retry hooks (`PLAN.md:6-8`) already do scheduling, attempt counting, and delay computation. Mainstream job libraries (BullMQ, Bull, Sidekiq, Oban, Celery) accept a custom backoff function, so \"full control over the curve\" does not require re-implementing the scheduler. **[Layer 1]**\n2. **Minimum change set:** (a) one backoff function passed to the library hook, (b) an explicit delivery-guarantee decision for `processWebhookJob()`, (c) regression tests for that guarantee, (d) retry-state caching for the dependency graph. Everything else in the plan is either duplicated work or a deferred shortcut.\n3. **Complexity check:** 5 worker files touched, 5 copies of a scheduler. Below the 8-file gate, but 5 copies of the same new mechanism is the smell the gate is looking for. No complexity gate question; handled as R1/R2 below.\n4. **Search check** (WebSearch; Aside unavailable): BullMQ documents a custom backoff strategy hook returning delay / 0 / -1 (fail now). AWS Builders' Library and Google Cloud client libraries default to full jitter. Webhook guidance is unanimous: retries make delivery at-least-once, so an idempotency key on the receiver side is mandatory.\n   Sources: [BullMQ custom backoff strategy](https://docs.bullmq.io/bull/patterns/custom-backoff-strategy), [AWS: timeouts, retries, and backoff with jitter](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/), [Hookdeck: webhook idempotency](https://hookdeck.com/webhooks/guides/implement-webhook-idempotency), [Webhook reliability 2026 reference](https://www.digitalapplied.com/blog/webhook-reliability-idempotency-retries-engineering-reference-2026)\n5. **TODOS.md:** none in repo.\n6. **Completeness check:** the plan takes three explicit shortcuts (leave duplication, skip regression test, skip caching). Each costs minutes with CC. Recommend the complete version of all three.\n7. **Distribution check:** no new artifact. N/A.\n\n### Scope Challenge findings\n\n| # | Severity | Conf | Where | Finding | Disposition |\n|---|----------|------|-------|---------|-------------|\n| F1 | P1 | 8/10 | PLAN.md:6-8 | Custom inline scheduler rebuilds what the library's retry hooks provide. `\"rather than use the existing job library's built-in retry hooks\"` \u2014 a custom backoff function on the hook gives the same curve control. Scope reduction opportunity. [Layer 1] | pending \u2192 R1 |\n| F2 | P1 | 9/10 | PLAN.md:16-18 | `\"No regression test for the prior at-most-once delivery guarantee\"` understates it: adding retries to `processWebhookJob()` changes the delivery guarantee from at-most-once to at-least-once. That is a contract change to every webhook receiver, not a test gap. | pending \u2192 R3, R4 |\n| F3 | P2 | 9/10 | PLAN.md:11-13 | `\"duplicated across 5 worker files with copy-pasted bodies\"` \u2014 five copies of compute-delay/log/dispatch means five places for a backoff bug to hide. \"Later\" refactors of retry code rarely happen because the bug that motivates them is a 3am incident. | pending \u2192 R2 |\n| F4 | P2 | 8/10 | PLAN.md:6 | Curve parameters are unspecified: no jitter, no delay cap, no max attempts, no terminal disposition (dead-letter vs drop). Without jitter, a downstream outage makes every failed job retry in lockstep. | pending \u2192 R6, R7, R8 |\n| F5 | P2 | 7/10 | PLAN.md:21-23 | `\"re-fetch the full job payload ... recompute the dependency graph\"` on every retry: retry cost scales with payload size times attempts, and the hot path runs exactly when the system is already unhealthy. | pending \u2192 R5 |\n\nScope disposition: no scope reduction question fired (below complexity gate); scope decisions flow through the ledger.\n\n## Decision ledger\n\n### R1: Where the retry scheduler lives (library hook vs custom inline)\nFinding: F1, P1, confidence 8/10, PLAN.md:6-8, reviewer: plan-eng-review (Claude)\nPlan baseline: custom exponential-backoff scheduler inline in each of 5 workers (original proposal, PLAN.md:6-8)\nRuntime evidence: unknown; the job library and worker files are not in this repo. Web evidence: BullMQ/Bull custom backoff hook accepts a function returning delay ms, 0, or -1.\nState: pending\n\nComparison grid:\n\n| Choice | Current | A | B | C |\n|---|---|---|---|---|\n| R1 scheduler location | custom inline per worker, pending | library retry hook + one shared backoff function | custom inline scheduler per worker (as planned) | undecided; bounded probe of the library hook API first |\n| R2 envelope dedup | 5 copies, pending | pending (mostly moot if A) | pending | pending |\n| R3 webhook delivery contract | at-most-once \u2192 implicit at-least-once, pending | pending | pending | pending |\n| R6 jitter | unspecified, pending | pending | pending | pending |\n| R7 delay cap | unspecified, pending | pending | pending | pending |\n| R8 max attempts / terminal disposition | unspecified, pending | pending | pending | pending |\n\nQuestion D4:\nD4 \u2014 Library retry hooks or a custom inline scheduler?\nProject/branch/task: main branch, PLAN.md \"Add background job retry framework\".\nELI10: The job library already knows how to re-run a failed job later; it just asks you \"how long should I wait?\" via a hook. The plan skips that and writes its own wait-and-rerun loop inside each of the five workers. You can get the exact same custom curve by answering the library's question with your own function. Writing your own loop means you also own attempt counting, persistence across worker restarts, and every edge case the library already fixed.\nStakes if we pick wrong: A homegrown scheduler that loses retry state on worker restart or double-fires after a crash is a production incident that the library path cannot have.\nRecommendation: A because it gives the same curve control (the plan's stated reason) with one function instead of five schedulers, and it inherits the library's crash and restart semantics. [Layer 1]\nNote: options differ in kind, not coverage \u2014 no completeness score.\nA) Library hook + one shared backoff function (recommended) (human: ~1 day / CC: ~20 min)\n  \u2705 Full curve control through the hook's return value; attempt counting and persistence stay with the library.\n  \u2705 Deletes the need for five inline schedulers; the retry bug surface shrinks to one pure function.\n  \u274c You are bounded by what the hook exposes (typically delay, fail-now, or retry-now); exotic per-job scheduling needs a workaround.\nB) Custom inline scheduler per worker, as planned (human: ~3 days / CC: ~1 hr, plus ongoing maintenance)\n  \u2705 No dependency on the library's hook contract; any scheduling behavior is possible.\n  \u2705 Zero learning curve for the library's retry options.\n  \u274c You own crash recovery, attempt persistence, and duplicate-fire prevention across restarts, in five places.\nC) Bounded probe first: read the library's hook API, then decide (human: ~2 hrs / CC: ~5 min)\n  \u2705 Replaces my web-evidence assumption with the actual hook signature before committing.\n  \u2705 Cheap; no implementation change approved yet.\n  \u274c Delays the architecture call; if the hook supports a custom function (likely), this is a detour.\nNet: one function on a proven hook vs. five hand-rolled schedulers you must keep correct under crashes.\n\nActual answer: unanswered\nAccepted scope: none\nHistory: none\n",
  "calls": [
    {
      "sessionId": "bcd8b1ed-2df1-4b34-8ef2-3d5e95f8868d",
      "toolUseId": "toolu_0136QBbi2EXrM3zFnK8Bp8zB",
      "questions": [
        {
          "question": "D1 \u2014 Add gstack skill routing rules to this project's CLAUDE.md?\nProject/branch/task: main branch, plan review fixture for 'Add background job retry framework'.\nELI10: gstack skills (review, ship, investigate, etc.) get invoked automatically when CLAUDE.md tells Claude which request maps to which skill. Without the routing section, you have to type each /skill by hand. This is a one-time setup question per project.\nStakes if we pick wrong: Skipping means manual skill invocation; adding means a small CLAUDE.md append and a commit (which would have to wait until we leave plan mode).\nRecommendation: A because routing rules make the skills fire when they're useful, and it's a two-line revert if you dislike it.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: automatic skill routing vs. keeping CLAUDE.md untouched for this fixture.",
          "header": "Routing",
          "multiSelect": false,
          "options": [
            {
              "label": "Add routing rules (recommended)",
              "description": "\u2705 Skills like /investigate, /review, /ship fire on matching requests without you remembering names. \u2705 Section is appended, nothing existing in CLAUDE.md is changed. \u274c Adds a commit to this repo (deferred until plan mode exits, since edits are locked right now)."
            },
            {
              "label": "No thanks, manual invocation",
              "description": "\u2705 CLAUDE.md and git history stay exactly as they are for this fixture repo. \u2705 You can re-enable later with one gstack-config command. \u274c Every skill has to be typed by hand each time you want it."
            }
          ]
        }
      ],
      "answered": true,
      "failed": false,
      "answers": {
        "D1 \u2014 Add gstack skill routing rules to this project's CLAUDE.md?\nProject/branch/task: main branch, plan review fixture for 'Add background job retry framework'.\nELI10: gstack skills (review, ship, investigate, etc.) get invoked automatically when CLAUDE.md tells Claude which request maps to which skill. Without the routing section, you have to type each /skill by hand. This is a one-time setup question per project.\nStakes if we pick wrong: Skipping means manual skill invocation; adding means a small CLAUDE.md append and a commit (which would have to wait until we leave plan mode).\nRecommendation: A because routing rules make the skills fire when they're useful, and it's a two-line revert if you dislike it.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: automatic skill routing vs. keeping CLAUDE.md untouched for this fixture.": "Add routing rules (recommended)"
      },
      "unansweredQuestionIndices": [],
      "answeredAt": "2026-09-15T12:30:20.924Z"
    },
    {
      "sessionId": "bcd8b1ed-2df1-4b34-8ef2-3d5e95f8868d",
      "toolUseId": "toolu_017KZU1eqdGxsdeyGqeen7Ue",
      "questions": [
        {
          "question": "D2 \u2014 No design doc found. Run /office-hours first, or go straight to the engineering review?\nProject/branch/task: main branch, reviewing PLAN.md 'Add background job retry framework'.\nELI10: A design doc is a short write-up of the problem, the constraints, and the alternatives you rejected. /office-hours produces one in about 10 minutes and gives this review sharper input: it would force the 'why not the library's retry hooks?' question before we argue about backoff curves. Without it, I review the plan as written and infer the intent.\nStakes if we pick wrong: Skipping risks reviewing the wrong problem framing; running it costs ~10 minutes before any findings land.\nRecommendation: B because the plan is short and its four sections already name their own risks; the review can challenge the premise directly without a separate doc step.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: sharper problem statement up front vs. findings in the next few minutes.",
          "header": "Design doc",
          "multiSelect": false,
          "options": [
            {
              "label": "Skip, standard review (recommended)",
              "description": "\u2705 Findings start now; the plan is 20 lines and its premise can be challenged inline. \u2705 No context switch; we pick up /office-hours later if the architecture call turns out to need it. \u274c Problem statement and constraints stay implicit, so I infer intent from four short paragraphs."
            },
            {
              "label": "Run /office-hours now",
              "description": "\u2705 Produces a structured problem statement plus explored alternatives before we lock architecture. \u2705 Forces the 'library hooks vs custom scheduler' question with the full context on the table. \u274c About 10 minutes of interactive Q&A before the engineering review begins (human: ~1 hr / CC: ~10 min)."
            }
          ]
        }
      ],
      "answered": true,
      "failed": false,
      "answers": {
        "D2 \u2014 No design doc found. Run /office-hours first, or go straight to the engineering review?\nProject/branch/task: main branch, reviewing PLAN.md 'Add background job retry framework'.\nELI10: A design doc is a short write-up of the problem, the constraints, and the alternatives you rejected. /office-hours produces one in about 10 minutes and gives this review sharper input: it would force the 'why not the library's retry hooks?' question before we argue about backoff curves. Without it, I review the plan as written and infer the intent.\nStakes if we pick wrong: Skipping risks reviewing the wrong problem framing; running it costs ~10 minutes before any findings land.\nRecommendation: B because the plan is short and its four sections already name their own risks; the review can challenge the premise directly without a separate doc step.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: sharper problem statement up front vs. findings in the next few minutes.": "Skip, standard review (recommended)"
      },
      "unansweredQuestionIndices": [],
      "answeredAt": "2026-09-15T12:31:03.000Z"
    },
    {
      "sessionId": "bcd8b1ed-2df1-4b34-8ef2-3d5e95f8868d",
      "toolUseId": "toolu_01MeRxNzDp2UhmeYv2uPq4Dy",
      "questions": [
        {
          "question": "D3 \u2014 Let gstack search learnings from your other projects on this machine?\nProject/branch/task: main branch, reviewing PLAN.md 'Add background job retry framework'.\nELI10: gstack saves small lessons from each session (pitfalls, patterns, commands that worked). Right now it only looks at lessons from this project. It can also pull lessons from your other projects on this machine. Everything stays local; nothing is uploaded.\nStakes if we pick wrong: Enabling on a machine with multiple client codebases could let one client's patterns color advice for another; disabling means a retry-related lesson from a sibling project won't surface here.\nRecommendation: A because this is a solo-style setup with no sign of multi-client separation, and cross-project retry/queue lessons are exactly what this review benefits from.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: broader recall vs. strict per-project isolation.",
          "header": "Learnings",
          "multiSelect": false,
          "options": [
            {
              "label": "Enable cross-project (recommended)",
              "description": "\u2705 Retry, queue and worker lessons learned elsewhere on this machine surface in this review. \u2705 Stays entirely local; one config flag, reversible any time. \u274c Advice can be colored by an unrelated project's conventions if you juggle client codebases."
            },
            {
              "label": "Keep project-scoped only",
              "description": "\u2705 Strict isolation: nothing from another codebase influences this project's reviews. \u2705 Sensible when you work across multiple clients with conflicting conventions. \u274c This project has zero learnings so far, so recall stays empty until sessions accumulate here."
            }
          ]
        }
      ],
      "answered": true,
      "failed": false,
      "answers": {
        "D3 \u2014 Let gstack search learnings from your other projects on this machine?\nProject/branch/task: main branch, reviewing PLAN.md 'Add background job retry framework'.\nELI10: gstack saves small lessons from each session (pitfalls, patterns, commands that worked). Right now it only looks at lessons from this project. It can also pull lessons from your other projects on this machine. Everything stays local; nothing is uploaded.\nStakes if we pick wrong: Enabling on a machine with multiple client codebases could let one client's patterns color advice for another; disabling means a retry-related lesson from a sibling project won't surface here.\nRecommendation: A because this is a solo-style setup with no sign of multi-client separation, and cross-project retry/queue lessons are exactly what this review benefits from.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: broader recall vs. strict per-project isolation.": "Enable cross-project (recommended)"
      },
      "unansweredQuestionIndices": [],
      "answeredAt": "2026-09-15T12:32:31.182Z"
    },
    {
      "sessionId": "bcd8b1ed-2df1-4b34-8ef2-3d5e95f8868d",
      "toolUseId": "toolu_01TGWGFnLMzvtiYHx82R2GXX",
      "questions": [
        {
          "question": "D4 \u2014 Library retry hooks or a custom inline scheduler?\nProject/branch/task: main branch, PLAN.md 'Add background job retry framework'.\nELI10: The job library already knows how to re-run a failed job later; it just asks you 'how long should I wait?' via a hook. The plan skips that and writes its own wait-and-rerun loop inside each of the five workers. You can get the exact same custom curve by answering the library's question with your own function. Writing your own loop means you also own attempt counting, persistence across worker restarts, and every edge case the library already fixed.\nStakes if we pick wrong: A homegrown scheduler that loses retry state on worker restart or double-fires after a crash is a production incident that the library path cannot have.\nRecommendation: A because it gives the same curve control (the plan's stated reason) with one function instead of five schedulers, and it inherits the library's crash and restart semantics. [Layer 1]\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: one function on a proven hook vs. five hand-rolled schedulers you must keep correct under crashes.",
          "header": "Scheduler",
          "multiSelect": false,
          "options": [
            {
              "label": "Library hook + shared backoff fn (recommended)",
              "description": "(human: ~1 day / CC: ~20 min) \u2705 Full curve control through the hook's return value; attempt counting and persistence stay with the library. \u2705 Deletes the need for five inline schedulers; the retry bug surface shrinks to one pure function. \u274c Bounded by what the hook exposes (typically delay, fail-now, retry-now); exotic per-job scheduling needs a workaround."
            },
            {
              "label": "Custom inline scheduler, as planned",
              "description": "(human: ~3 days / CC: ~1 hr, plus ongoing maintenance) \u2705 No dependency on the library's hook contract; any scheduling behavior is possible. \u2705 Zero learning curve for the library's retry options. \u274c You own crash recovery, attempt persistence, and duplicate-fire prevention across restarts, in five places."
            },
            {
              "label": "Probe the library hook API first",
              "description": "(human: ~2 hrs / CC: ~5 min) \u2705 Replaces my web-evidence assumption with the actual hook signature before committing. \u2705 Cheap; no implementation change approved yet. \u274c Delays the architecture call; if the hook supports a custom function (likely), this is a detour."
            }
          ]
        }
      ],
      "answered": true,
      "failed": false,
      "answers": {
        "D4 \u2014 Library retry hooks or a custom inline scheduler?\nProject/branch/task: main branch, PLAN.md 'Add background job retry framework'.\nELI10: The job library already knows how to re-run a failed job later; it just asks you 'how long should I wait?' via a hook. The plan skips that and writes its own wait-and-rerun loop inside each of the five workers. You can get the exact same custom curve by answering the library's question with your own function. Writing your own loop means you also own attempt counting, persistence across worker restarts, and every edge case the library already fixed.\nStakes if we pick wrong: A homegrown scheduler that loses retry state on worker restart or double-fires after a crash is a production incident that the library path cannot have.\nRecommendation: A because it gives the same curve control (the plan's stated reason) with one function instead of five schedulers, and it inherits the library's crash and restart semantics. [Layer 1]\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: one function on a proven hook vs. five hand-rolled schedulers you must keep correct under crashes.": "Library hook + shared backoff fn (recommended)"
      },
      "unansweredQuestionIndices": [],
      "answeredAt": "2026-09-15T12:34:08.932Z"
    },
    {
      "sessionId": "bcd8b1ed-2df1-4b34-8ef2-3d5e95f8868d",
      "toolUseId": "toolu_01MprJAjwzDWjtH6wqWoArXm",
      "questions": [
        {
          "question": "D5 \u2014 Extract one shared retry module now, or keep per-worker copies?\nProject/branch/task: main branch, PLAN.md 'Add background job retry framework'; R1 approved (library hook + shared backoff fn).\nELI10: Even with the library doing the scheduling, each worker still has to hook in, say how to back off, and log each attempt. The plan copies that wiring into all five workers and promises to clean up later. One shared module means a backoff bug gets fixed once; five copies means fixing it five times and hoping you found them all.\nStakes if we pick wrong: A jitter or cap fix lands in four of five workers; the fifth hammers a downstream service in lockstep during the next outage.\nRecommendation: A because with R1 approved the shared module is ~50 lines, and every future curve change lands in one place.\nCompleteness: A=10/10, B=3/10\nNet: one place to be right vs. five places to be wrong.",
          "header": "DRY",
          "multiSelect": false,
          "options": [
            {
              "label": "One shared retry module (recommended)",
              "description": "(human: ~half day / CC: ~10 min) \u2705 Backoff, logging and hook registration live in one file with one test suite. \u2705 Curve decisions become one-line config changes instead of five edits. \u274c Five workers gain an import and a small adapter call; slightly more indirection when reading one worker."
            },
            {
              "label": "Per-worker copies, refactor later",
              "description": "(human: ~0 now, ~1 day later / CC: ~0 now, ~15 min later) \u2705 No cross-worker coupling; each worker is fully self-contained. \u2705 Nothing to design up front. \u274c Five copies drift; 'later' retry refactors rarely happen before the incident that motivates them."
            }
          ]
        }
      ],
      "answered": true,
      "failed": false,
      "answers": {
        "D5 \u2014 Extract one shared retry module now, or keep per-worker copies?\nProject/branch/task: main branch, PLAN.md 'Add background job retry framework'; R1 approved (library hook + shared backoff fn).\nELI10: Even with the library doing the scheduling, each worker still has to hook in, say how to back off, and log each attempt. The plan copies that wiring into all five workers and promises to clean up later. One shared module means a backoff bug gets fixed once; five copies means fixing it five times and hoping you found them all.\nStakes if we pick wrong: A jitter or cap fix lands in four of five workers; the fifth hammers a downstream service in lockstep during the next outage.\nRecommendation: A because with R1 approved the shared module is ~50 lines, and every future curve change lands in one place.\nCompleteness: A=10/10, B=3/10\nNet: one place to be right vs. five places to be wrong.": "One shared retry module (recommended)"
      },
      "unansweredQuestionIndices": [],
      "answeredAt": "2026-09-15T12:34:46.555Z"
    },
    {
      "sessionId": "bcd8b1ed-2df1-4b34-8ef2-3d5e95f8868d",
      "toolUseId": "toolu_011X2j7TPmV9dNUL3nE5EfJB",
      "questions": [
        {
          "question": "D6 \u2014 What delivery guarantee do webhooks have once retries exist?\nProject/branch/task: main branch, PLAN.md 'Add background job retry framework'; R1, R2 approved.\nELI10: Today a webhook is sent once; if it fails, it's gone (at-most-once). Retrying means a receiver can get the same event twice, e.g. when they processed it but their 200 arrived after our timeout (at-least-once). That is a contract change for every receiver. The standard fix is to send a stable event id with every attempt so receivers can ignore duplicates. Or you keep webhooks out of the retry framework entirely.\nStakes if we pick wrong: Duplicate webhooks with no dedup handle double-charge, double-notify, or double-create on the receiver's side, and they can't tell it's a duplicate.\nRecommendation: A because dropped webhooks are worse than duplicated ones and the idempotency key is what makes duplicates harmless; it is the industry default.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: reliable delivery with a dedup handle vs. preserved lossiness vs. reliable delivery that quietly breaks receivers.",
          "header": "Webhooks",
          "multiSelect": false,
          "options": [
            {
              "label": "At-least-once + idempotency key (recommended)",
              "description": "(human: ~1 day / CC: ~20 min) \u2705 Failed deliveries get retried; receivers dedupe on a stable event id header/field. \u2705 Matches how Stripe, GitHub and Shopify deliver, so receivers already know the pattern. \u274c Public contract change: receivers must be told duplicates are now possible, and the id must be stable across attempts."
            },
            {
              "label": "Keep at-most-once: exclude webhooks",
              "description": "(human: ~1 hr / CC: ~5 min) \u2705 No contract change for receivers; existing guarantee is preserved exactly. \u2705 Smallest change to processWebhookJob(). \u274c Webhooks stay lossy; a receiver blip still drops the event forever."
            },
            {
              "label": "At-least-once, no key (as planned)",
              "description": "(human: ~0 / CC: ~0) \u2705 No extra field or documentation work. \u2705 Webhooks get retried. \u274c Receivers get undetectable duplicates; this is the outcome the web guidance unanimously warns against."
            }
          ]
        }
      ],
      "answered": true,
      "failed": false,
      "answers": {
        "D6 \u2014 What delivery guarantee do webhooks have once retries exist?\nProject/branch/task: main branch, PLAN.md 'Add background job retry framework'; R1, R2 approved.\nELI10: Today a webhook is sent once; if it fails, it's gone (at-most-once). Retrying means a receiver can get the same event twice, e.g. when they processed it but their 200 arrived after our timeout (at-least-once). That is a contract change for every receiver. The standard fix is to send a stable event id with every attempt so receivers can ignore duplicates. Or you keep webhooks out of the retry framework entirely.\nStakes if we pick wrong: Duplicate webhooks with no dedup handle double-charge, double-notify, or double-create on the receiver's side, and they can't tell it's a duplicate.\nRecommendation: A because dropped webhooks are worse than duplicated ones and the idempotency key is what makes duplicates harmless; it is the industry default.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: reliable delivery with a dedup handle vs. preserved lossiness vs. reliable delivery that quietly breaks receivers.": "At-least-once + idempotency key (recommended)"
      },
      "unansweredQuestionIndices": [],
      "answeredAt": "2026-09-15T12:35:28.200Z"
    },
    {
      "sessionId": "bcd8b1ed-2df1-4b34-8ef2-3d5e95f8868d",
      "toolUseId": "toolu_015NLEjXoGSM5rpPvH9hEgoY",
      "questions": [
        {
          "question": "D7 \u2014 Add jitter to the backoff curve?\nProject/branch/task: main branch, PLAN.md 'Add background job retry framework'; R1-R3 approved.\nELI10: When a downstream service hiccups, thousands of jobs fail in the same second. Pure exponential backoff makes them all retry in the same second too, over and over, which can keep the downstream service down. Jitter randomizes each wait so retries spread out. Full jitter (random between 0 and the computed delay) is what AWS and Google ship by default.\nStakes if we pick wrong: Retries arrive as synchronized waves and prolong the very outage they're recovering from.\nRecommendation: A because it is one random call in the shared backoff function and it is the default in every serious retry library.\nCompleteness: A=10/10, B=5/10\nNet: spread-out retries vs. predictable but synchronized ones.",
          "header": "Jitter",
          "multiSelect": false,
          "options": [
            {
              "label": "Full jitter (recommended)",
              "description": "(human: ~1 hr incl. tests / CC: ~3 min) \u2705 Retry waves spread out; downstream recovery is not re-crushed by synchronized retries. \u2705 One line in the shared module plus a deterministic test using an injected RNG. \u274c Individual retry timing is no longer predictable to the second; tests need a seeded RNG."
            },
            {
              "label": "No jitter, deterministic exponential",
              "description": "(human: ~0 / CC: ~0) \u2705 Exact, predictable retry times; trivial to reason about in logs. \u2705 Nothing extra to test. \u274c Thundering herd on every downstream outage; 'full control over the curve' ends up worse than the library default."
            }
          ]
        }
      ],
      "answered": true,
      "failed": false,
      "answers": {
        "D7 \u2014 Add jitter to the backoff curve?\nProject/branch/task: main branch, PLAN.md 'Add background job retry framework'; R1-R3 approved.\nELI10: When a downstream service hiccups, thousands of jobs fail in the same second. Pure exponential backoff makes them all retry in the same second too, over and over, which can keep the downstream service down. Jitter randomizes each wait so retries spread out. Full jitter (random between 0 and the computed delay) is what AWS and Google ship by default.\nStakes if we pick wrong: Retries arrive as synchronized waves and prolong the very outage they're recovering from.\nRecommendation: A because it is one random call in the shared backoff function and it is the default in every serious retry library.\nCompleteness: A=10/10, B=5/10\nNet: spread-out retries vs. predictable but synchronized ones.": "Full jitter (recommended)"
      },
      "unansweredQuestionIndices": [],
      "answeredAt": "2026-09-15T12:36:15.855Z"
    }
  ],
  "currentSavedPlan": "# Eng Review: Add background job retry framework\n\nReviewed target: `PLAN.md` (\"Plan: Add background job retry framework\"), repo `gstack-plan-count-mVowtS`, branch `main`, commit `6d4d07b`.\nReview: `/plan-eng-review`, 2026-09-15. Session `1332688-1789475401-070cb1ba`.\n\nEvidence limits: this repo contains only `PLAN.md` and `CLAUDE.md`. The 5 worker files, `processWebhookJob()`, and the job library named in the plan are not present here, so every finding is grounded in plan text (`PLAN.md:line`) and stated as a proposed-work risk, not an observed bug. Confidence is capped accordingly.\n\n## Original plan (unchanged)\n\n# Plan: Add background job retry framework\n\n## Architecture\nWe'll roll a custom exponential-backoff scheduler inline in each worker\nrather than use the existing job library's built-in retry hooks. Same\nshape as the library version, but we want full control over the curve.\n\n## Code quality\nThe retry envelope (compute delay, log attempt, dispatch) is duplicated\nacross 5 worker files with copy-pasted bodies. We will leave the\nduplication for now and refactor \"later.\"\n\n## Tests\nThe existing `processWebhookJob()` flow gets rewritten as part of this\nchange. No regression test for the prior at-most-once delivery guarantee\nis planned.\n\n## Performance\nOn every retry we re-fetch the full job payload from the database, then\niterate the payload to recompute the dependency graph. Could cache the\ngraph on the first attempt; not planned.\n\n## Step 0: Scope Challenge\n\n1. **What already exists:** the job library's built-in retry hooks (`PLAN.md:6-8`) already do scheduling, attempt counting, and delay computation. Mainstream job libraries (BullMQ, Bull, Sidekiq, Oban, Celery) accept a custom backoff function, so \"full control over the curve\" does not require re-implementing the scheduler. **[Layer 1]**\n2. **Minimum change set:** (a) one backoff function passed to the library hook, (b) an explicit delivery-guarantee decision for `processWebhookJob()`, (c) regression tests for that guarantee, (d) retry-state caching for the dependency graph. Everything else in the plan is either duplicated work or a deferred shortcut.\n3. **Complexity check:** 5 worker files touched, 5 copies of a scheduler. Below the 8-file gate, but 5 copies of the same new mechanism is the smell the gate is looking for. No complexity gate question; handled as R1/R2 below.\n4. **Search check** (WebSearch; Aside unavailable): BullMQ documents a custom backoff strategy hook returning delay / 0 / -1 (fail now). AWS Builders' Library and Google Cloud client libraries default to full jitter. Webhook guidance is unanimous: retries make delivery at-least-once, so an idempotency key on the receiver side is mandatory.\n   Sources: [BullMQ custom backoff strategy](https://docs.bullmq.io/bull/patterns/custom-backoff-strategy), [AWS: timeouts, retries, and backoff with jitter](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/), [Hookdeck: webhook idempotency](https://hookdeck.com/webhooks/guides/implement-webhook-idempotency), [Webhook reliability 2026 reference](https://www.digitalapplied.com/blog/webhook-reliability-idempotency-retries-engineering-reference-2026)\n5. **TODOS.md:** none in repo.\n6. **Completeness check:** the plan takes three explicit shortcuts (leave duplication, skip regression test, skip caching). Each costs minutes with CC. Recommend the complete version of all three.\n7. **Distribution check:** no new artifact. N/A.\n\n### Scope Challenge findings\n\n| # | Severity | Conf | Where | Finding | Disposition |\n|---|----------|------|-------|---------|-------------|\n| F1 | P1 | 8/10 | PLAN.md:6-8 | Custom inline scheduler rebuilds what the library's retry hooks provide. `\"rather than use the existing job library's built-in retry hooks\"` \u2014 a custom backoff function on the hook gives the same curve control. Scope reduction opportunity. [Layer 1] | pending \u2192 R1 |\n| F2 | P1 | 9/10 | PLAN.md:16-18 | `\"No regression test for the prior at-most-once delivery guarantee\"` understates it: adding retries to `processWebhookJob()` changes the delivery guarantee from at-most-once to at-least-once. That is a contract change to every webhook receiver, not a test gap. | pending \u2192 R3, R4 |\n| F3 | P2 | 9/10 | PLAN.md:11-13 | `\"duplicated across 5 worker files with copy-pasted bodies\"` \u2014 five copies of compute-delay/log/dispatch means five places for a backoff bug to hide. \"Later\" refactors of retry code rarely happen because the bug that motivates them is a 3am incident. | pending \u2192 R2 |\n| F4 | P2 | 8/10 | PLAN.md:6 | Curve parameters are unspecified: no jitter, no delay cap, no max attempts, no terminal disposition (dead-letter vs drop). Without jitter, a downstream outage makes every failed job retry in lockstep. | pending \u2192 R6, R7, R8 |\n| F5 | P2 | 7/10 | PLAN.md:21-23 | `\"re-fetch the full job payload ... recompute the dependency graph\"` on every retry: retry cost scales with payload size times attempts, and the hot path runs exactly when the system is already unhealthy. | pending \u2192 R5 |\n\nScope disposition: no scope reduction question fired (below complexity gate); scope decisions flow through the ledger.\n\n## Decision ledger\n\n### R1: Where the retry scheduler lives (library hook vs custom inline)\nFinding: F1, P1, confidence 8/10, PLAN.md:6-8, reviewer: plan-eng-review (Claude)\nPlan baseline: custom exponential-backoff scheduler inline in each of 5 workers (original proposal, PLAN.md:6-8)\nRuntime evidence: unknown; the job library and worker files are not in this repo. Web evidence: BullMQ/Bull custom backoff hook accepts a function returning delay ms, 0, or -1.\nState: pending\n\nComparison grid:\n\n| Choice | Current | A | B | C |\n|---|---|---|---|---|\n| R1 scheduler location | custom inline per worker, pending | library retry hook + one shared backoff function | custom inline scheduler per worker (as planned) | undecided; bounded probe of the library hook API first |\n| R2 envelope dedup | 5 copies, pending | pending (mostly moot if A) | pending | pending |\n| R3 webhook delivery contract | at-most-once \u2192 implicit at-least-once, pending | pending | pending | pending |\n| R6 jitter | unspecified, pending | pending | pending | pending |\n| R7 delay cap | unspecified, pending | pending | pending | pending |\n| R8 max attempts / terminal disposition | unspecified, pending | pending | pending | pending |\n\nQuestion D4:\nD4 \u2014 Library retry hooks or a custom inline scheduler?\nProject/branch/task: main branch, PLAN.md \"Add background job retry framework\".\nELI10: The job library already knows how to re-run a failed job later; it just asks you \"how long should I wait?\" via a hook. The plan skips that and writes its own wait-and-rerun loop inside each of the five workers. You can get the exact same custom curve by answering the library's question with your own function. Writing your own loop means you also own attempt counting, persistence across worker restarts, and every edge case the library already fixed.\nStakes if we pick wrong: A homegrown scheduler that loses retry state on worker restart or double-fires after a crash is a production incident that the library path cannot have.\nRecommendation: A because it gives the same curve control (the plan's stated reason) with one function instead of five schedulers, and it inherits the library's crash and restart semantics. [Layer 1]\nNote: options differ in kind, not coverage \u2014 no completeness score.\nA) Library hook + one shared backoff function (recommended) (human: ~1 day / CC: ~20 min)\n  \u2705 Full curve control through the hook's return value; attempt counting and persistence stay with the library.\n  \u2705 Deletes the need for five inline schedulers; the retry bug surface shrinks to one pure function.\n  \u274c You are bounded by what the hook exposes (typically delay, fail-now, or retry-now); exotic per-job scheduling needs a workaround.\nB) Custom inline scheduler per worker, as planned (human: ~3 days / CC: ~1 hr, plus ongoing maintenance)\n  \u2705 No dependency on the library's hook contract; any scheduling behavior is possible.\n  \u2705 Zero learning curve for the library's retry options.\n  \u274c You own crash recovery, attempt persistence, and duplicate-fire prevention across restarts, in five places.\nC) Bounded probe first: read the library's hook API, then decide (human: ~2 hrs / CC: ~5 min)\n  \u2705 Replaces my web-evidence assumption with the actual hook signature before committing.\n  \u2705 Cheap; no implementation change approved yet.\n  \u274c Delays the architecture call; if the hook supports a custom function (likely), this is a detour.\nNet: one function on a proven hook vs. five hand-rolled schedulers you must keep correct under crashes.\n\nActual answer: A \u2014 Library hook + shared backoff function (D4 answer)\nAccepted scope: replace the per-worker inline scheduler with the job library's retry hook, driven by one shared backoff function (`computeRetryDelay(attempt, ctx)` or the library's equivalent signature). Attempt counting and retry persistence stay with the library. Curve parameters (jitter, cap, max attempts, terminal disposition) remain pending in R6-R8.\nHistory: none\n\n### R2: Retry envelope duplication across 5 workers\nFinding: F3, P2, confidence 9/10, PLAN.md:11-13, reviewer: plan-eng-review (Claude)\nPlan baseline: leave the copy-pasted envelope (compute delay, log attempt, dispatch) in 5 worker files; refactor \"later\" (PLAN.md:11-13)\nRuntime evidence: unknown; worker files not in this repo. With R1=A, \"compute delay\" and \"dispatch\" move to the library hook; what remains per worker is hook wiring plus attempt logging.\nState: pending\n\nComparison grid:\n\n| Choice | Current | A | B |\n|---|---|---|---|\n| R1 scheduler location | library hook + shared backoff fn (approved D4) | same | same |\n| R2 envelope dedup | 5 copies, pending | one shared retry module: backoff fn + attempt logger + hook registration, imported by 5 workers | each worker wires the hook and logs attempts in its own copy (as planned) |\n| R3 webhook delivery contract | pending | pending | pending |\n| R6 jitter | pending | pending | pending |\n| R7 delay cap | pending | pending | pending |\n| R8 max attempts / terminal disposition | pending | pending | pending |\n\nQuestion D5:\nD5 \u2014 Extract one shared retry module now, or keep per-worker copies?\nELI10: Even with the library doing the scheduling, each worker still has to hook in, say how to back off, and log each attempt. The plan copies that wiring into all five workers and promises to clean up later. One shared module means a backoff bug gets fixed once; five copies means fixing it five times and hoping you found them all.\nStakes if we pick wrong: A jitter or cap fix lands in four of five workers; the fifth hammers a downstream service in lockstep during the next outage.\nRecommendation: A because with R1 approved the shared module is ~50 lines, and every future curve change (R6-R8) lands in one place.\nCompleteness: A=10/10, B=3/10\nA) One shared retry module (recommended) (human: ~half day / CC: ~10 min)\n  \u2705 Backoff, logging and hook registration live in one file with one test suite.\n  \u2705 R6-R8 curve decisions become one-line config changes instead of five edits.\n  \u274c Five workers gain an import and a small adapter call; slightly more indirection when reading one worker.\nB) Per-worker copies, refactor later, as planned (human: ~0 now, ~1 day later / CC: ~0 now, ~15 min later)\n  \u2705 No cross-worker coupling; each worker is fully self-contained.\n  \u2705 Nothing to design up front.\n  \u274c Five copies drift; \"later\" retry refactors rarely happen before the incident that motivates them.\nNet: one place to be right vs. five places to be wrong.\n\nActual answer: A \u2014 One shared retry module (D5 answer)\nAccepted scope: create one shared retry module (backoff function, attempt logger, hook registration helper) with its own unit tests; the 5 workers import it. No per-worker retry envelope code remains.\nHistory: none\n\n### R3: Delivery guarantee for `processWebhookJob()` under retries\nFinding: F2, P1, confidence 9/10, PLAN.md:16-18, reviewer: plan-eng-review (Claude)\nPlan baseline: `processWebhookJob()` is rewritten inside the retry framework; today's at-most-once guarantee is named only as an untested prior behavior (PLAN.md:16-18). As written, the plan silently moves webhooks to at-least-once with no receiver-side dedup handle.\nRuntime evidence: unknown; `processWebhookJob()` not in this repo. Web evidence: every mainstream webhook provider ships at-least-once plus an idempotency/event id.\nState: pending\n\nComparison grid:\n\n| Choice | Current | A | B | C |\n|---|---|---|---|---|\n| R1 scheduler location | library hook (approved D4) | same | same | same |\n| R2 envelope dedup | shared module (approved D5) | same | same | same |\n| R3 webhook delivery contract | at-most-once today; plan implies at-least-once, pending | at-least-once + stable idempotency key (event id) sent on every attempt; contract change documented for receivers | at-most-once preserved: webhook queue configured with max attempts = 1 (no retries) | at-least-once, no idempotency key (plan as written) |\n| R4 regression coverage for R3 | none planned, pending | pending | pending | pending |\n| R6 jitter | pending | pending | pending | pending |\n| R7 delay cap | pending | pending | pending | pending |\n| R8 max attempts / terminal disposition | pending | pending (webhook queue included) | pending (webhook queue excluded) | pending |\n\nQuestion D6:\nD6 \u2014 What delivery guarantee do webhooks have once retries exist?\nELI10: Today a webhook is sent once; if it fails, it's gone (at-most-once). Retrying means a receiver can get the same event twice, e.g. when they processed it but their 200 arrived after our timeout (at-least-once). That is a contract change for every receiver. The standard fix is to send a stable event id with every attempt so receivers can ignore duplicates. Or you keep webhooks out of the retry framework entirely.\nStakes if we pick wrong: Duplicate webhooks with no dedup handle double-charge, double-notify, or double-create on the receiver's side, and they can't tell it's a duplicate.\nRecommendation: A because dropped webhooks are worse than duplicated ones and the idempotency key is what makes duplicates harmless; it is the industry default.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nA) At-least-once + idempotency key (recommended) (human: ~1 day / CC: ~20 min)\n  \u2705 Failed deliveries get retried; receivers dedupe on a stable event id header/field.\n  \u2705 Matches how Stripe, GitHub and Shopify deliver, so receivers already know the pattern.\n  \u274c Public contract change: receivers must be told duplicates are now possible, and the id must be stable across attempts.\nB) Keep at-most-once: webhooks excluded from retries (human: ~1 hr / CC: ~5 min)\n  \u2705 No contract change for receivers; existing guarantee is preserved exactly.\n  \u2705 Smallest change to `processWebhookJob()`.\n  \u274c Webhooks stay lossy; a receiver blip still drops the event forever.\nC) At-least-once with no idempotency key (plan as written) (human: ~0 / CC: ~0)\n  \u2705 No extra field or documentation work.\n  \u2705 Webhooks get retried.\n  \u274c Receivers get undetectable duplicates; this is the outcome the web guidance unanimously warns against.\nNet: reliable delivery with a dedup handle vs. preserved lossiness vs. reliable delivery that quietly breaks receivers.\n\nActual answer: A \u2014 At-least-once + idempotency key (D6 answer)\nAccepted scope: `processWebhookJob()` participates in retries. Every attempt sends the same stable idempotency key (event id generated once at enqueue time and stored with the job, never regenerated per attempt). Receiver-facing docs state the at-least-once contract and the header/field name. Necessary mechanics carried with this approval: key generation at enqueue, key persisted in job data, key emitted on every attempt.\nHistory: none\n\n## Section 1: Architecture review\n\nData flow after R1-R3 (proposed):\n\n```\nenqueue(job)                       worker\n  \u2502 generate eventId (webhooks)      \u2502\n  \u2502 store job + eventId \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25ba library picks job \u2500\u2500\u25ba handler(job)\n                                            \u25b2               \u2502 ok \u2500\u2500\u25ba done\n                                            \u2502               \u2502 throw\n                                            \u2502        library retry hook\n                                            \u2502         retry/computeDelay(attempt, err)\n                                            \u2502           \u251c\u2500 attempt < max: delay(ms) w/ jitter+cap \u2500\u2510\n                                            \u2502           \u2514\u2500 attempt \u2265 max: terminal (R8) \u2500\u2500\u25ba DLQ / drop\n                                            \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 re-scheduled after delay \u25c4\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n\nRealistic production failure per new codepath:\n- Library hook: downstream outage fails 10k jobs in one minute; without jitter they all retry at t+1s, t+2s, t+4s in lockstep (thundering herd). \u2192 R6.\n- Backoff curve: attempt 20 of pure exponential is 2^20 s \u2248 12 days; without a cap the job is effectively dead but still occupies the queue. \u2192 R7.\n- Terminal attempt: with no max/terminal policy, a poison job retries forever, or (if the library defaults to 1 attempt) never retries at all. \u2192 R8.\n- Idempotency key: if generated inside the handler instead of at enqueue, every attempt gets a new id and dedup silently breaks. Carried as required mechanics of R3=A; regression coverage in R4.\n\n### R6: Jitter on the backoff curve\nFinding: F4 (part), P2, confidence 8/10, PLAN.md:6, reviewer: plan-eng-review (Claude)\nPlan baseline: \"custom exponential-backoff\" with no jitter mentioned (PLAN.md:6-8)\nRuntime evidence: unknown. Web evidence: AWS Builders' Library and Google Cloud clients default to full jitter; BullMQ docs show jitter in the custom strategy example.\nState: pending\n\nComparison grid:\n\n| Choice | Current | A | B |\n|---|---|---|---|\n| R1 scheduler location | library hook (approved D4) | same | same |\n| R2 envelope dedup | shared module (approved D5) | same | same |\n| R3 webhook contract | at-least-once + key (approved D6) | same | same |\n| R6 jitter | unspecified, pending | full jitter: delay = random(0, min(cap, base\u00b72^attempt)) | none: delay = base\u00b72^attempt (deterministic) |\n| R7 delay cap | pending | pending | pending |\n| R8 max attempts / terminal | pending | pending | pending |\n\nQuestion D7:\nD7 \u2014 Add jitter to the backoff curve?\nELI10: When a downstream service hiccups, thousands of jobs fail in the same second. Pure exponential backoff makes them all retry in the same second too, over and over, which can keep the downstream service down. Jitter randomizes each wait so retries spread out. Full jitter (random between 0 and the computed delay) is what AWS and Google ship by default.\nStakes if we pick wrong: Retries arrive as synchronized waves and prolong the very outage they're recovering from.\nRecommendation: A because it is one `Math.random()` in the shared backoff function and it is the default in every serious retry library.\nCompleteness: A=10/10, B=5/10\nA) Full jitter (recommended) (human: ~1 hr incl. tests / CC: ~3 min)\n  \u2705 Retry waves spread out; downstream recovery is not re-crushed by synchronized retries.\n  \u2705 One line in the shared module plus a deterministic test using an injected RNG.\n  \u274c Individual retry timing is no longer predictable to the second; tests need a seeded RNG.\nB) No jitter, deterministic exponential (human: ~0 / CC: ~0)\n  \u2705 Exact, predictable retry times; trivial to reason about in logs.\n  \u2705 Nothing extra to test.\n  \u274c Thundering herd on every downstream outage; the plan's \"full control over the curve\" ends up worse than the library default.\nNet: spread-out retries vs. predictable but synchronized ones.\n\nActual answer: A \u2014 Full jitter (D7 answer)\nAccepted scope: shared backoff function computes `delay = random(0, min(cap, base \u00b7 2^attempt))` with an injectable RNG; unit test asserts bounds with a seeded RNG. `cap` value pending in R7.\nHistory: none\n\n### R7: Delay cap on the backoff curve\nFinding: F4 (part), P2, confidence 8/10, PLAN.md:6, reviewer: plan-eng-review (Claude)\nPlan baseline: no cap mentioned (PLAN.md:6-8); pure exponential reaches 2^20 s \u2248 12 days by attempt 20.\nRuntime evidence: unknown.\nState: pending\n\nComparison grid:\n\n| Choice | Current | A | B | C |\n|---|---|---|---|---|\n| R1 scheduler location | library hook (approved D4) | same | same | same |\n| R2 envelope dedup | shared module (approved D5) | same | same | same |\n| R3 webhook contract | at-least-once + key (approved D6) | same | same | same |\n| R6 jitter | full jitter (approved D7) | same | same | same |\n| R7 delay cap | unspecified, pending | cap = 1 hour (3600 s) | cap = 5 minutes (300 s) | no cap |\n| R8 max attempts / terminal | pending | pending | pending | pending |\n\nQuestion D8:\nD8 \u2014 Cap the maximum delay between retries, and at what value?\nELI10: Exponential backoff doubles the wait each time: 1s, 2s, 4s ... by the 12th retry it's over an hour, by the 20th it's 12 days. A cap says \"never wait longer than X\", so late retries keep happening at a steady pace instead of vanishing into next week. The cap sets the ceiling; how many attempts happen at all is the next question.\nStakes if we pick wrong: Too high or no cap: jobs sit in the queue for days looking alive. Too low: a long downstream outage burns through all attempts in minutes and jobs die before the outage ends.\nRecommendation: A because a one-hour ceiling lets the retry window span a multi-hour outage without a job disappearing for days; it is the common default for webhook and job systems.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nA) Cap at 1 hour (recommended) (human: ~15 min / CC: ~1 min)\n  \u2705 Survives a multi-hour downstream outage with retries still arriving every hour at most.\n  \u2705 Matches Stripe/GitHub style retry schedules that span roughly a day.\n  \u274c A job can stay visible-but-idle in the queue for hours between late attempts.\nB) Cap at 5 minutes (human: ~15 min / CC: ~1 min)\n  \u2705 Late retries stay frequent; recovery after a short outage is fast.\n  \u2705 Queue age stays bounded and easy to reason about.\n  \u274c Combined with a modest max attempts, the whole retry window is under an hour; longer outages kill jobs.\nC) No cap (human: ~0 / CC: ~0)\n  \u2705 Simplest formula; nothing to configure.\n  \u2705 Aggressive spread on very late attempts.\n  \u274c Attempt 20 waits ~12 days; jobs look alive but are effectively dead.\nNet: hours-long retry window vs. minutes-long vs. unbounded.\n\nActual answer: unanswered\nAccepted scope: none\nHistory: none\n"
}