gstack/test/fixtures/dx-manual-handoff-ao.json

{
  "provenance": {
    "source": "be36e2c57cc6bf4762b9c6a81cd4d72a6615baa7",
    "captureSha256": "6f8bb677f9c463977bcb338a17d2c1766cb525e851880df9d96454debfdb5ebb",
    "publicSha256": "58759e3d05a9b94c927b5ce74304557a082f0c8f52fce7738fe681ef1c2dfcc2",
    "reportMtimeMs": 1789040805987.1885,
    "startedAt": 1789040160000.0,
    "scope": "Exact final substantive call and completed manual DX navigation call, with their four public use/result events. Native reasoning is excluded."
  },
  "calls": [
    {
      "sessionId": "ccefcae1-e3c3-482a-a69d-a0815761495a",
      "toolUseId": "toolu_01WStSiGHC7YCNzhRM5h7SSA",
      "questions": [
        {
          "question": "D12 \u2014 TODO candidate: remove the Client.evaluate() compatibility alias at the version named in the 2.0 changelog. Track it?\nProject/branch/task: EvalKit SDK beta polish, branch main, TODOS.md pass after the eight review passes.\nELI10: D7 keeps `Client.evaluate()` as a warning alias so v1 scripts don't break on 2.0. Aliases are only kind if they eventually go away; otherwise the API carries two names forever and the deprecation warning becomes noise. What: delete the alias and its warning at the stated version (2.1 or 3.0). Why: one public name per action. Pros: clean surface, warning stays meaningful. Cons: any straggler still on the old name breaks then, which is the point of the notice period. Context: alias lives in evalkit/client.py; migration guide and codemod from D7 are the remediation path. Depends on: D7 landing in 2.0.0b1 and the changelog naming the removal version.\nStakes if we pick wrong: Without a tracked item, the alias quietly becomes permanent and the deprecation warning lies.\nRecommendation: A because the removal is a promise made in the 2.0 changelog, and a TODO is how the promise survives three months.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: record the follow-through now, or rely on someone remembering at 2.1.",
          "header": "TODO",
          "multiSelect": false,
          "options": [
            {
              "label": "Add to TODOS.md (recommended)",
              "description": "\u2705 Removal is scheduled and visible, with the version, file, and remediation path written down. \u2705 Keeps the DeprecationWarning honest for developers who see it. \u274c TODOS.md is written after plan mode exits, as part of implementation."
            },
            {
              "label": "Skip",
              "description": "\u2705 Nothing extra to track; the changelog already names the removal version. \u2705 Zero work now. \u274c Aliases without a tracked removal tend to become permanent."
            },
            {
              "label": "Build it now",
              "description": "\u2705 Nothing to build now: the alias must exist through 2.0 first, so this option just schedules the removal into the plan. \u2705 Equivalent to A for this item. \u274c Removing the alias in the same release that introduces it defeats D7."
            }
          ]
        }
      ],
      "answered": true,
      "failed": false,
      "answers": {
        "D12 \u2014 TODO candidate: remove the Client.evaluate() compatibility alias at the version named in the 2.0 changelog. Track it?\nProject/branch/task: EvalKit SDK beta polish, branch main, TODOS.md pass after the eight review passes.\nELI10: D7 keeps `Client.evaluate()` as a warning alias so v1 scripts don't break on 2.0. Aliases are only kind if they eventually go away; otherwise the API carries two names forever and the deprecation warning becomes noise. What: delete the alias and its warning at the stated version (2.1 or 3.0). Why: one public name per action. Pros: clean surface, warning stays meaningful. Cons: any straggler still on the old name breaks then, which is the point of the notice period. Context: alias lives in evalkit/client.py; migration guide and codemod from D7 are the remediation path. Depends on: D7 landing in 2.0.0b1 and the changelog naming the removal version.\nStakes if we pick wrong: Without a tracked item, the alias quietly becomes permanent and the deprecation warning lies.\nRecommendation: A because the removal is a promise made in the 2.0 changelog, and a TODO is how the promise survives three months.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: record the follow-through now, or rely on someone remembering at 2.1.": "Add to TODOS.md (recommended)"
      },
      "unansweredQuestionIndices": [],
      "answeredAt": "2026-09-10T11:44:53.729Z"
    },
    {
      "sessionId": "ccefcae1-e3c3-482a-a69d-a0815761495a",
      "toolUseId": "toolu_01HmgyQrnejyK5bQHRNQ6c2C",
      "questions": [
        {
          "question": "D13 \u2014 DX review complete (5/10 -> 8.5/10). What should happen next?\nProject/branch/task: EvalKit SDK beta polish, branch main, review chaining after the Review Readiness Dashboard.\nELI10: The DX review found five broken developer-facing contracts (first-run CI gate, missing quickstart file, reversed argument order, bare auth error, hard v1-to-v2 rename) and two polish additions (demo next-step footer, TTHW re-measurement gate). All are written into the plan as tasks T1 to T9. The readiness dashboard shows only this DX review has run; Eng Review, which normally gates shipping, has zero runs, and the outside Codex voice is disabled. Three of the fixes (T1, T3, T5) change SDK runtime behavior that is maintained outside this repo, which is exactly what an eng review would validate. You said earlier you will handle subsequent reviews manually.\nStakes if we pick wrong: Chaining a review you didn't want wastes your time; skipping one you did want leaves the runtime changes unvalidated until implementation.\nRecommendation: D because you stated up front that you will run subsequent reviews yourself, so the plan should exit with the DX findings and leave the eng gate to you.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: hand off now as you asked, or chain the eng review here.",
          "header": "Next steps",
          "multiSelect": false,
          "options": [
            {
              "label": "Skip, I'll handle next steps manually (recommended)",
              "description": "\u2705 Matches your stated intent to run further reviews yourself. \u2705 Plan exits now with all DX decisions and tasks recorded; nothing else is started. \u274c Eng Review stays at zero runs until you invoke it, so the ship gate is still open."
            },
            {
              "label": "Run /plan-eng-review next (required gate)",
              "description": "\u2705 Validates the three runtime-affecting tasks (T1, T3, T5) and the cross-repo dependency on the SDK runtime. \u2705 Clears the required Eng Review row on the readiness dashboard. \u274c Contradicts your note that subsequent reviews are yours to run."
            },
            {
              "label": "Ready to implement, run /devex-review after shipping",
              "description": "\u2705 Goes straight to building T1 to T9; the boomerang /devex-review later checks measured TTHW against the under-2-minute target. \u2705 Fastest path to a testable beta. \u274c Skips the eng gate on runtime changes made in a separately maintained codebase."
            }
          ]
        }
      ],
      "answered": true,
      "failed": false,
      "answers": {
        "D13 \u2014 DX review complete (5/10 -> 8.5/10). What should happen next?\nProject/branch/task: EvalKit SDK beta polish, branch main, review chaining after the Review Readiness Dashboard.\nELI10: The DX review found five broken developer-facing contracts (first-run CI gate, missing quickstart file, reversed argument order, bare auth error, hard v1-to-v2 rename) and two polish additions (demo next-step footer, TTHW re-measurement gate). All are written into the plan as tasks T1 to T9. The readiness dashboard shows only this DX review has run; Eng Review, which normally gates shipping, has zero runs, and the outside Codex voice is disabled. Three of the fixes (T1, T3, T5) change SDK runtime behavior that is maintained outside this repo, which is exactly what an eng review would validate. You said earlier you will handle subsequent reviews manually.\nStakes if we pick wrong: Chaining a review you didn't want wastes your time; skipping one you did want leaves the runtime changes unvalidated until implementation.\nRecommendation: D because you stated up front that you will run subsequent reviews yourself, so the plan should exit with the DX findings and leave the eng gate to you.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: hand off now as you asked, or chain the eng review here.": "Skip, I'll handle next steps manually (recommended)"
      },
      "unansweredQuestionIndices": [],
      "answeredAt": "2026-09-10T11:47:05.544Z"
    }
  ],
  "events": [
    {
      "sessionId": "ccefcae1-e3c3-482a-a69d-a0815761495a",
      "timestamp": "2026-09-10T11:44:51.951Z",
      "toolUseId": "toolu_01WStSiGHC7YCNzhRM5h7SSA",
      "kind": "use",
      "name": "AskUserQuestion",
      "input": {
        "questions": [
          {
            "question": "D12 \u2014 TODO candidate: remove the Client.evaluate() compatibility alias at the version named in the 2.0 changelog. Track it?\nProject/branch/task: EvalKit SDK beta polish, branch main, TODOS.md pass after the eight review passes.\nELI10: D7 keeps `Client.evaluate()` as a warning alias so v1 scripts don't break on 2.0. Aliases are only kind if they eventually go away; otherwise the API carries two names forever and the deprecation warning becomes noise. What: delete the alias and its warning at the stated version (2.1 or 3.0). Why: one public name per action. Pros: clean surface, warning stays meaningful. Cons: any straggler still on the old name breaks then, which is the point of the notice period. Context: alias lives in evalkit/client.py; migration guide and codemod from D7 are the remediation path. Depends on: D7 landing in 2.0.0b1 and the changelog naming the removal version.\nStakes if we pick wrong: Without a tracked item, the alias quietly becomes permanent and the deprecation warning lies.\nRecommendation: A because the removal is a promise made in the 2.0 changelog, and a TODO is how the promise survives three months.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: record the follow-through now, or rely on someone remembering at 2.1.",
            "header": "TODO",
            "multiSelect": false,
            "options": [
              {
                "label": "Add to TODOS.md (recommended)",
                "description": "\u2705 Removal is scheduled and visible, with the version, file, and remediation path written down. \u2705 Keeps the DeprecationWarning honest for developers who see it. \u274c TODOS.md is written after plan mode exits, as part of implementation."
              },
              {
                "label": "Skip",
                "description": "\u2705 Nothing extra to track; the changelog already names the removal version. \u2705 Zero work now. \u274c Aliases without a tracked removal tend to become permanent."
              },
              {
                "label": "Build it now",
                "description": "\u2705 Nothing to build now: the alias must exist through 2.0 first, so this option just schedules the removal into the plan. \u2705 Equivalent to A for this item. \u274c Removing the alias in the same release that introduces it defeats D7."
              }
            ]
          }
        ]
      },
      "messageId": "msg_011Ceuhc9naVhxPysfBCE4kk",
      "requestId": "req_011Ceuhc8ecEbRpv4KC6HBMF"
    },
    {
      "sessionId": "ccefcae1-e3c3-482a-a69d-a0815761495a",
      "timestamp": "2026-09-10T11:44:53.729Z",
      "toolUseId": "toolu_01WStSiGHC7YCNzhRM5h7SSA",
      "kind": "result",
      "content": "Your questions have been answered: \"D12 \u2014 TODO candidate: remove the Client.evaluate() compatibility alias at the version named in the 2.0 changelog. Track it?\nProject/branch/task: EvalKit SDK beta polish, branch main, TODOS.md pass after the eight review passes.\nELI10: D7 keeps `Client.evaluate()` as a warning alias so v1 scripts don't break on 2.0. Aliases are only kind if they eventually go away; otherwise the API carries two names forever and the deprecation warning becomes noise. What: delete the alias and its warning at the stated version (2.1 or 3.0). Why: one public name per action. Pros: clean surface, warning stays meaningful. Cons: any straggler still on the old name breaks then, which is the point of the notice period. Context: alias lives in evalkit/client.py; migration guide and codemod from D7 are the remediation path. Depends on: D7 landing in 2.0.0b1 and the changelog naming the removal version.\nStakes if we pick wrong: Without a tracked item, the alias quietly becomes permanent and the deprecation warning lies.\nRecommendation: A because the removal is a promise made in the 2.0 changelog, and a TODO is how the promise survives three months.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: record the follow-through now, or rely on someone remembering at 2.1.\"=\"Add to TODOS.md (recommended)\". You can now continue with these answers in mind.",
      "isError": false
    },
    {
      "sessionId": "ccefcae1-e3c3-482a-a69d-a0815761495a",
      "timestamp": "2026-09-10T11:47:03.476Z",
      "toolUseId": "toolu_01HmgyQrnejyK5bQHRNQ6c2C",
      "kind": "use",
      "name": "AskUserQuestion",
      "input": {
        "questions": [
          {
            "question": "D13 \u2014 DX review complete (5/10 -> 8.5/10). What should happen next?\nProject/branch/task: EvalKit SDK beta polish, branch main, review chaining after the Review Readiness Dashboard.\nELI10: The DX review found five broken developer-facing contracts (first-run CI gate, missing quickstart file, reversed argument order, bare auth error, hard v1-to-v2 rename) and two polish additions (demo next-step footer, TTHW re-measurement gate). All are written into the plan as tasks T1 to T9. The readiness dashboard shows only this DX review has run; Eng Review, which normally gates shipping, has zero runs, and the outside Codex voice is disabled. Three of the fixes (T1, T3, T5) change SDK runtime behavior that is maintained outside this repo, which is exactly what an eng review would validate. You said earlier you will handle subsequent reviews manually.\nStakes if we pick wrong: Chaining a review you didn't want wastes your time; skipping one you did want leaves the runtime changes unvalidated until implementation.\nRecommendation: D because you stated up front that you will run subsequent reviews yourself, so the plan should exit with the DX findings and leave the eng gate to you.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: hand off now as you asked, or chain the eng review here.",
            "header": "Next steps",
            "multiSelect": false,
            "options": [
              {
                "label": "Skip, I'll handle next steps manually (recommended)",
                "description": "\u2705 Matches your stated intent to run further reviews yourself. \u2705 Plan exits now with all DX decisions and tasks recorded; nothing else is started. \u274c Eng Review stays at zero runs until you invoke it, so the ship gate is still open."
              },
              {
                "label": "Run /plan-eng-review next (required gate)",
                "description": "\u2705 Validates the three runtime-affecting tasks (T1, T3, T5) and the cross-repo dependency on the SDK runtime. \u2705 Clears the required Eng Review row on the readiness dashboard. \u274c Contradicts your note that subsequent reviews are yours to run."
              },
              {
                "label": "Ready to implement, run /devex-review after shipping",
                "description": "\u2705 Goes straight to building T1 to T9; the boomerang /devex-review later checks measured TTHW against the under-2-minute target. \u2705 Fastest path to a testable beta. \u274c Skips the eng gate on runtime changes made in a separately maintained codebase."
              }
            ]
          }
        ]
      },
      "messageId": "msg_011CeuhmYbX1oEQYPBG3LsB6",
      "requestId": "req_011CeuhmWfvcvvZ55UoJKUKy"
    },
    {
      "sessionId": "ccefcae1-e3c3-482a-a69d-a0815761495a",
      "timestamp": "2026-09-10T11:47:05.544Z",
      "toolUseId": "toolu_01HmgyQrnejyK5bQHRNQ6c2C",
      "kind": "result",
      "content": "Your questions have been answered: \"D13 \u2014 DX review complete (5/10 -> 8.5/10). What should happen next?\nProject/branch/task: EvalKit SDK beta polish, branch main, review chaining after the Review Readiness Dashboard.\nELI10: The DX review found five broken developer-facing contracts (first-run CI gate, missing quickstart file, reversed argument order, bare auth error, hard v1-to-v2 rename) and two polish additions (demo next-step footer, TTHW re-measurement gate). All are written into the plan as tasks T1 to T9. The readiness dashboard shows only this DX review has run; Eng Review, which normally gates shipping, has zero runs, and the outside Codex voice is disabled. Three of the fixes (T1, T3, T5) change SDK runtime behavior that is maintained outside this repo, which is exactly what an eng review would validate. You said earlier you will handle subsequent reviews manually.\nStakes if we pick wrong: Chaining a review you didn't want wastes your time; skipping one you did want leaves the runtime changes unvalidated until implementation.\nRecommendation: D because you stated up front that you will run subsequent reviews yourself, so the plan should exit with the DX findings and leave the eng gate to you.\nNote: options differ in kind, not coverage \u2014 no completeness score.\nNet: hand off now as you asked, or chain the eng review here.\"=\"Skip, I'll handle next steps manually (recommended)\". You can now continue with these answers in mind.",
      "isError": false
    }
  ],
  "planReadyRequests": [
    {
      "sessionId": "ccefcae1-e3c3-482a-a69d-a0815761495a",
      "toolUseId": "toolu_014xATA6mrMbBupvHdm97A33",
      "timestamp": "2026-09-10T11:47:30.675Z",
      "failed": false
    }
  ],
  "reportContent": "# Plan: EvalKit SDK beta release polish (DX review)\n\n## Context\n\nEvalKit is a Python SDK for ML engineers evaluating LLM responses. The beta\nrelease plan (PLAN.md) ships the current SDK contracts unchanged. This review\nruns /plan-devex-review in DX POLISH mode against the documented contracts in\nREADME.md, docs/current-contracts.md, docs/api.md, docs/benchmarks.md, and\ndocs/package-contents.txt, and records which contracts the beta should repair.\n\nSettled inputs (from the repo, not re-asked):\n- Persona: ML engineer, writes Python daily, terminal-first (README.md:3-5)\n- Target: first real result in under 2 minutes (docs/benchmarks.md:8)\n- Magical-moment vehicle: `python -m evalkit.demo` terminal demo (README.md:14-17)\n- Mode: DX POLISH, existing release scope (README.md:5-6, PLAN.md)\n- Product type: Library/SDK (Python package on PyPI)\n\nSetup decisions made in this session:\n- D0: Add gstack skill routing rules to CLAUDE.md. Deferred until plan mode\n  exits (plan mode allows edits to this file only). Post-review step: append\n  the routing section to CLAUDE.md and commit it.\n- D1: Skipped /office-hours; the inputs it would produce are already documented.\n\n## Developer Persona Card\n\n```\nTARGET DEVELOPER PERSONA\n========================\nWho:       ML engineer evaluating LLM responses, writes Python daily\nContext:   Terminal session, wants a local eval result before wiring the SDK\n           into production CI\nTolerance: Under 2 minutes to first real result (agreed target); peers land\n           in 2-4 minutes, so anything past ~5 minutes reads as broken\nExpects:   pip install, one command, real scores on screen; a key step only\n           when they go live; errors that say what to change\n```\n\n## Competitive DX Benchmark\n\nSource: docs/benchmarks.md internal study (same developer, same machine,\nmeasured from before install to first real evaluation result). Web research\nunavailable in this session (Aside not installed); reference tiers are the\nskill's built-in Stripe/Vercel/Firebase/Docker benchmarks.\n\n```\nCOMPETITIVE DX BENCHMARK\n=========================\nTool           | TTHW   | Notable DX choice                          | Source\nPeer SDK A     | 2 min  | No first-run gate; result on first command | docs/benchmarks.md\nPeer SDK C     | 3 min  |                                             | docs/benchmarks.md\nPeer SDK B     | 4 min  |                                             | docs/benchmarks.md\nEvalKit (now)  | 6 min  | 5 of 6 minutes is the mandatory CI wait     | docs/benchmarks.md\nEvalKit target | <2 min | Champion tier (agreed)                      | docs/benchmarks.md:8\n```\n\n## Developer Empathy Narrative (confirmed, D2)\n\nI'm an ML engineer with a notebook full of LLM outputs I need to score. I open\nthe README. Install is one line, `python -m pip install evalkit==2.0.0b1`,\nfine. Next line says run `python examples/first_eval.py`. I run it: No such\nfile or directory. The package inventory confirms that file isn't shipped\nanywhere. Thirty seconds in and the quickstart is already wrong. I read on and\nfind `python -m evalkit.demo`. I run it. It prints \"Verifying the\nsample-project binding with EvalKit CI; inspect\nhttps://ci.evalkit.example/checks/<id>; normally completes within 300s.\" Then a\nstderr line every 30 seconds: \"Waiting for CI check: 90s elapsed of 300s.\"\nFive minutes for a keyless demo of bundled sample data. Peer SDK A gave me a\nreal score in two minutes total. I alt-tab and half forget about it. When it\nfinally prints example scores, I'm mildly pleased but wary. I create a key,\nexport EVALKIT_API_KEY, mistype one character, and get\n`AuthError(\"request failed\")`. No code, no cause, no hint it was the key. I\nopen docs/api.md to write my first real call and see\n`run_eval(dataset, evaluator)` next to `run_batch(evaluator, dataset)`,\nreversed positional order, no keyword requirement. I know I'll swap them at\nleast once. And I notice my v1 scripts calling `Client.evaluate()` will simply\nbreak on v2 with no alias or warning.\n\n## Friction Point Decisions (Step 0F)\n\n### D3. First-run CI gate (Hello World) \u2014 DECIDED: move the check off first-run\n- Evidence: docs/current-contracts.md:3-5 (mandatory 300s remote CI check on\n  every first local evaluation, no skip, no offline path); docs/benchmarks.md\n  (6 min measured, 5 min is the wait; target < 2 min).\n- Change: `python -m evalkit.demo` and the first local `run_eval` return\n  immediately using the shipped mock transport / offline sample data. The\n  sample-project binding check moves to the first live (keyed) evaluation and\n  to noninteractive CI mode. The existing progress line, pre-countdown\n  explanation, EVALKIT_CI_TIMEOUT message, and help link are retained verbatim\n  at the new location.\n- Docs to update: README.md:17-23 (remove \"waits for the mandatory CI check\"\n  from the demo description), docs/current-contracts.md:3-10.\n- Coordination: runtime is maintained separately (README.md:39-40); the plan\n  must name the runtime change as a beta dependency, not assume it.\n- Expected effect: TTHW from 6 min to roughly 1 min (install ~30s, demo ~10s).\n\n### D4. Quickstart points at a missing file (Hello World) \u2014 DECIDED: demo first, ship first_eval.py too\n- Evidence: README.md:11 references examples/first_eval.py; docs/package-contents.txt:8-9\n  says it is absent from the wheel and the examples archive.\n- Change: README quickstart step 2 becomes `python -m evalkit.demo` with the\n  expected output block (README.md:34-36) shown right under it. Step 3 is\n  `python examples/first_eval.py`, a real dataset + evaluator + `run_eval`\n  call that ships in both the package and the examples archive, with its own\n  expected output. Add a release check that the example is present in both\n  artifacts.\n\n### D5. Reversed positional order in run_eval / run_batch (Real Usage) \u2014 DECIDED: same order, keyword-only\n- Evidence: docs/api.md:5-9.\n- Change: both signatures become `run_eval(*, dataset, evaluator)` and\n  `run_batch(*, dataset, evaluator)` (keyword-only; positional use raises a\n  TypeError that names both parameters and shows the corrected call). Update\n  docs/api.md, README example, examples/first_eval.py, and docstrings to the\n  keyword form. List run_batch's order change in the 2.0 changelog and the\n  migration guide (see D7).\n\n### D6. AuthError(\"request failed\") (Debug) \u2014 DECIDED: code + cause + fix + docs link\n- Evidence: docs/api.md:11-13; contrast docs/current-contracts.md:21-24 (every\n  other error already has cause, argument, fix, redaction).\n- Change: `AuthError` carries `code` (EVALKIT_AUTH_MISSING_KEY when\n  EVALKIT_API_KEY is unset/empty; EVALKIT_AUTH_INVALID_KEY when rejected;\n  EVALKIT_AUTH_REVOKED_KEY when the console reports revoked), a message that\n  names the cause and the fix (\"Set EVALKIT_API_KEY\" or \"Create or rotate a key\n  at https://console.evalkit.example/settings/api-keys, then export it\"), a\n  masked key suffix only (never the full key), and a help URL per code. Add\n  the codes to the API reference alongside EVALKIT_CI_TIMEOUT. Update\n  docs/api.md:11-13.\n\n### D7. Client.evaluate() removed without alias (Upgrade) \u2014 DECIDED: alias + warning + guide + codemod\n- Evidence: docs/api.md:15-18.\n- Change: keep `Client.evaluate()` as a thin alias for `Client.run()` that\n  emits `DeprecationWarning(\"Client.evaluate() is deprecated and will be\n  removed in <version>; use Client.run()\")` once per process. Write\n  docs/migrating-v1-to-v2.md covering: the rename, the run_batch argument\n  order change (D5), the AuthError codes (D6), the first-run behavior change\n  (D3). Ship a codemod (`python -m evalkit.codemod v1-to-v2 <path>` or an\n  equivalent documented sed recipe) with a test fixture. Changelog gains a\n  \"Breaking changes\" block naming both changes and the alias removal version.\n\n## Developer Journey Map (after 0F)\n\n```\nSTAGE           | DEVELOPER DOES                         | FRICTION POINTS                         | STATUS\n----------------|----------------------------------------|-----------------------------------------|--------\n1. Discover     | Reads README.md intro                  | none; persona and value stated in 3 lines| ok\n2. Install      | python -m pip install evalkit==2.0.0b1 | none; one command, Py3.10+, no Docker    | ok\n3. Hello World  | python -m evalkit.demo                 | D4 quickstart pointed at missing file    | fixed\n                |                                        | D3 5-minute CI gate before any output    | fixed\n4. Real Usage   | key -> export -> run_eval(...)         | D5 reversed positional order             | fixed\n5. Debug        | Bad key on first live call             | D6 AuthError(\"request failed\")           | fixed\n6. Upgrade      | pip install -U from v1                 | D7 hard removal of Client.evaluate()     | fixed\n```\n\n## First-Time Developer Confusion Report (confirmed, D8: address all)\n\n```\nFIRST-TIME DEVELOPER REPORT\n============================\nPersona: ML engineer, Python daily, terminal-first\nAttempting: EvalKit 2.0.0b1 getting started (current contracts)\n\nCONFUSION LOG:\nT+0:00  README, pip install. Works.                                        (#0 none)\nT+0:30  python examples/first_eval.py -> No such file. \"Is my install     (#1 -> D4, addressed)\n        broken?\" README.md:11 vs docs/package-contents.txt:8\nT+1:00  python -m evalkit.demo -> \"Verifying the sample-project binding    (#2 -> D3, addressed)\n        with EvalKit CI... normally completes within 300s.\" Why does a\n        keyless demo on bundled data need remote CI?\nT+1:30  \"Waiting for CI check: 30s elapsed of 300s\". Alt-tabs away.       (#2 cont.)\nT+6:00  Scores print. Creates key, exports with a typo. First live call:  (#3 -> D6, addressed)\n        AuthError(\"request failed\"). Network? SDK? My code?\nT+8:00  Writes run_batch in run_eval's argument order. Wrong result or    (#4 -> D5, addressed)\n        type error, unsure which.\nT+9:00  Old v1 script: AttributeError on Client.evaluate. No warning,     (#5 -> D7, addressed)\n        no guide.\nT+9:30  Final state: succeeded, 9.5 min, five \"is it me?\" moments,\n        against a 2-minute target.\n```\n\n## Review Pass Findings\n\n### Pass 1: Getting Started \u2014 3/10 -> 9/10\n- Before: TTHW 6 min (target < 2), first quickstart command fails.\n- Fixed by D3 (CI gate off first-run), D4 (quickstart order + shipped example).\n- D10 (new, DECIDED): demo prints a next-step footer after the overall score:\n  `Next: create a key at https://console.evalkit.example/settings/api-keys,\n  then run: export EVALKIT_API_KEY=... && python examples/first_eval.py`.\n  Skip the key line when EVALKIT_API_KEY is already set. Update README.md:34-36\n  and the demo output fixture.\n- Ideal sequence (3 steps, ~1 min): pip install (~30s) -> python -m evalkit.demo\n  (~10s, scores + next step) -> export key + python examples/first_eval.py.\n- Remaining gap to 10: key creation still requires a browser visit; acceptable\n  for a live call, out of POLISH scope to change.\n\n### Pass 2: API/SDK Design \u2014 4/10 -> 8/10\n- Fixed by D5. Retained: bounded timeouts, retries, rate limits, idempotent\n  evaluation IDs (docs/current-contracts.md:23-24). No further issues.\n\n### Pass 3: Error Messages \u2014 5/10 -> 9/10\n- Traced: (1) invalid key: `AuthError(\"request failed\")` -> D6 tiered codes.\n  (2) CI timeout: already Rust-tier (code, URL, help link) and retained at its\n  new location under D3. (3) swapped run_batch args: silent/opaque -> D5\n  keyword-only TypeError naming both parameters. No further issues.\n\n### Pass 4: Documentation \u2014 4/10 -> 8/10\n- Fixed by D4 (quickstart), D7 (migration guide), D3 (README.md:17-23 and\n  docs/current-contracts.md:3-10 rewritten). Docs are version-pinned to\n  2.0.0b1. No further issues.\n\n### Pass 5: Upgrade Path \u2014 2/10 -> 9/10\n- Fixed by D7 (alias, DeprecationWarning, guide, codemod, removal version in\n  changelog). No further issues.\n\n### Pass 6: Developer Environment \u2014 9/10 (no change)\n- Python 3.10+, macOS/Linux/Windows, no Docker, type annotations, mock\n  transport, offline sample data, noninteractive CI mode all exist\n  (docs/current-contracts.md:26-28). No issues.\n\n### Pass 7: Community & Ecosystem \u2014 7/10 (no change)\n- Support contact, changelog, contributor guide exist. No community program\n  proposed; correct for POLISH scope. No issues.\n\n### Pass 8: DX Measurement \u2014 6/10 -> 9/10\n- Existing: timing instrumentation and post-beta survey (docs/benchmarks.md:9-10).\n- D11 (new, DECIDED): repeat the docs/benchmarks.md protocol on the 2.0.0b1\n  release candidate with D3, D4, D10 applied; record the measured TTHW in\n  docs/benchmarks.md beside the 6-minute baseline. Release gate: under 2\n  minutes or the beta waits. Gives /devex-review a real post-ship baseline.\n\n### Outside voice\nCodex review skipped (codex_reviews disabled). Re-enable:\n`gstack-config set codex_reviews enabled`. No substitute reviewer was run;\noutside coverage for this plan is recorded as disabled.\n\n## Magical Moment Specification\n\nVehicle: copy-paste terminal demo, `python -m evalkit.demo`, keyless, uses the\nshipped mock transport and bundled sample_responses.json, prints per-example\nscores and an overall score (README.md:31-36). Already in the package\n(docs/package-contents.txt). Blocker to the moment: the demo waits on the\nmandatory 5-minute remote CI check before printing anything useful.\n\nImplementation requirements after this review:\n- D3: demo returns immediately; no CI check on the keyless path.\n- D10: demo output ends with a next-step footer pointing at key creation and\n  `python examples/first_eval.py`.\n- Documented output (README.md:34-36) updated to the new five-line form and\n  covered by the demo output test fixture.\n\n## Competitive DX Benchmark (post-review)\n\n```\nTool           | TTHW        | Notes\nPeer SDK A     | 2 min       | measured, docs/benchmarks.md\nPeer SDK C     | 3 min       | measured\nPeer SDK B     | 4 min       | measured\nEvalKit before | 6 min       | measured; 5 min is the first-run CI wait\nEvalKit after  | ~1 min est. | D3 + D4 + D10; MUST be measured per D11 before beta\n```\n\n## NOT in scope (considered, deferred)\n\n- Hosted playground / browser sandbox: persona is terminal-first and the\n  terminal demo is the approved vehicle (README.md:14-17).\n- In-terminal key provisioning (`evalkit login`): would remove the one\n  remaining browser visit, but adds an auth flow to a POLISH release.\n- Additional language bindings, community program, new hosted service:\n  excluded by docs/current-contracts.md:29-30.\n- Consolidating `Client.run()` with `run_eval`/`run_batch` naming: no evidence\n  in the docs of what differs; revisit after beta feedback.\n\n## What already exists (reuse, do not rebuild)\n\n- Mock transport and offline sample data (docs/current-contracts.md:27) carry\n  the keyless demo and first local eval under D3.\n- Noninteractive CI mode (docs/current-contracts.md:27) is the natural home\n  for the sample-project binding check moved by D3.\n- CI timeout error format: code + URL + help link (docs/current-contracts.md:15-18)\n  is the template for the D6 AuthError codes.\n- Secret redaction in errors (docs/current-contracts.md:22) applies to the\n  masked key suffix in D6.\n- Timing instrumentation and survey (docs/benchmarks.md:9-10) power D11.\n- Changelog, API reference, contributor guide, support contact exist;\n  D5/D6/D7 add entries rather than new documents, except the migration guide.\n\n## TODOS.md updates\n\n- D12 (accepted): **Remove `Client.evaluate()` alias.** What: delete the\n  alias and DeprecationWarning at the version named in the 2.0 changelog.\n  Why: one public name per action; keeps the warning honest. Pros: clean\n  surface. Cons: stragglers break at that version (the notice period is the\n  mitigation). Context: alias in evalkit/client.py; remediation is\n  docs/migrating-v1-to-v2.md and the codemod. Depends on: D7 shipped in\n  2.0.0b1, changelog names the removal version. Written to TODOS.md when\n  plan mode exits.\n\n## DX Scorecard\n\n```\n+====================================================================+\n|              DX PLAN REVIEW \u2014 SCORECARD                             |\n+====================================================================+\n| Dimension            | Score  | Prior  | Trend  |\n|----------------------|--------|--------|--------|\n| Getting Started      |  9/10  |  3/10  | +6 \u2191   |\n| API/CLI/SDK          |  8/10  |  4/10  | +4 \u2191   |\n| Error Messages       |  9/10  |  5/10  | +4 \u2191   |\n| Documentation        |  8/10  |  4/10  | +4 \u2191   |\n| Upgrade Path         |  9/10  |  2/10  | +7 \u2191   |\n| Dev Environment      |  9/10  |  9/10  |  0 =   |\n| Community            |  7/10  |  7/10  |  0 =   |\n| DX Measurement       |  9/10  |  6/10  | +3 \u2191   |\n+--------------------------------------------------------------------+\n| TTHW                 | ~1 min (est., gate D11) | 6 min | \u2191          |\n| Competitive Rank     | Champion (pending D11 measurement)           |\n| Magical Moment       | designed via terminal demo (python -m evalkit.demo) |\n| Product Type         | Library/SDK (Python)                          |\n| Mode                 | POLISH                                        |\n| Overall DX           |  8.5/10 |  5.0/10 | +3.5 \u2191 |\n+====================================================================+\n| DX PRINCIPLE COVERAGE                                               |\n| Zero Friction      | covered (D3, D4)                               |\n| Learn by Doing     | covered (demo + shipped first_eval.py)         |\n| Fight Uncertainty  | covered (D6, D10, retained CI timeout error)   |\n| Opinionated + Escape Hatches | covered (keyword-only + alias path)  |\n| Code in Context    | covered (first_eval.py: real dataset/evaluator)|\n| Magical Moments    | covered (demo returns in seconds, D3)          |\n+====================================================================+\n```\n\"Prior\" = the plan as drafted before this review. No earlier DX review exists.\nCommunity at 7 is the only dimension below 8; no adoption blocker there, and\ncommunity programs are excluded from this release by contract.\n\n## DX Implementation Checklist\n\n```\nDX IMPLEMENTATION CHECKLIST\n============================\n[ ] Time to hello world < 2 min, measured per D11\n[ ] Installation is one command (already: pip install evalkit==2.0.0b1)\n[ ] First run produces meaningful output within seconds (D3)\n[ ] Magical moment delivered via terminal demo, with next-step footer (D10)\n[ ] Every error message has: problem + cause + fix + docs link (D6 closes the last gap)\n[ ] API naming is guessable without docs (D5: one order, keyword-only)\n[ ] Every parameter has a sensible default (unchanged; no new params)\n[ ] Docs have copy-paste examples that actually work (D4)\n[ ] Examples show real use cases, not just hello world (first_eval.py)\n[ ] Upgrade path documented with migration guide (D7)\n[ ] Breaking changes have deprecation warnings + codemods (D7)\n[x] Type annotations included (existing)\n[x] Works in CI/CD without special configuration (noninteractive mode exists;\n    binding check relocated there by D3)\n[x] Free tier available, no credit card required (keyless demo)\n[x] Changelog exists and is maintained (add Breaking changes block, D5/D7)\n[ ] Search works in documentation (not assessed; no docs site in scope)\n[x] Support channel exists (support contact documented)\n```\n\n## Implementation Tasks\nSynthesized from this review's findings. Each task derives from a specific\nfinding above. Run with Claude Code or Codex; checkbox as you ship.\n\n- [ ] **T1 (P1, human: ~2 days / CC: ~30 min)** \u2014 runtime first-run \u2014 Move the sample-project binding CI check off the keyless demo and first local eval; run it before the first live (keyed) evaluation and in noninteractive CI mode, keeping the existing progress line, pre-countdown explanation, EVALKIT_CI_TIMEOUT message, and help link.\n  - Surfaced by: Friction D3 \u2014 docs/current-contracts.md:3-10, docs/benchmarks.md\n  - Files: evalkit/demo.py, evalkit/client.py, docs/current-contracts.md, README.md:17-23\n  - Verify: fresh venv, `pip install` then `python -m evalkit.demo` prints scores in under 15s with no network; first keyed `run_eval` performs the check once\n- [ ] **T2 (P1, human: ~half day / CC: ~10 min)** \u2014 quickstart + packaging \u2014 Reorder README quickstart to demo first; author and ship examples/first_eval.py in the wheel and examples archive with expected output; add a release check for its presence.\n  - Surfaced by: Friction D4 \u2014 README.md:11, docs/package-contents.txt:8-9\n  - Files: README.md, examples/first_eval.py, packaging manifest, docs/package-contents.txt\n  - Verify: `unzip -l` of the built wheel and the examples archive both list examples/first_eval.py; `python examples/first_eval.py` runs after install\n- [ ] **T3 (P1, human: ~1 day / CC: ~15 min)** \u2014 public API \u2014 Make run_eval and run_batch `(*, dataset, evaluator)` keyword-only with a TypeError that names both parameters and the corrected call; update docs/api.md, README, example, docstrings, changelog.\n  - Surfaced by: Friction D5 \u2014 docs/api.md:5-9\n  - Files: evalkit/__init__.py, evalkit/client.py, docs/api.md, examples/first_eval.py, CHANGELOG\n  - Verify: `run_batch(ds, ev)` raises the named TypeError; `run_batch(dataset=ds, evaluator=ev)` succeeds; api.md shows keyword form\n- [ ] **T4 (P1, human: ~1 day / CC: ~15 min)** \u2014 auth errors \u2014 AuthError carries code (MISSING_KEY / INVALID_KEY / REVOKED_KEY), cause, fix, masked key suffix, help URL; codes added to the API reference beside EVALKIT_CI_TIMEOUT.\n  - Surfaced by: Friction D6 \u2014 docs/api.md:11-13\n  - Files: evalkit/client.py, docs/api.md, API reference error table\n  - Verify: unset key, wrong key, revoked key each produce a distinct code and message containing the console URL; full key never appears in output\n- [ ] **T5 (P1, human: ~1.5 days / CC: ~20 min)** \u2014 upgrade path \u2014 Keep Client.evaluate() as a DeprecationWarning alias for Client.run(); write docs/migrating-v1-to-v2.md (rename, run_batch order, AuthError codes, first-run change); ship codemod with fixture; changelog Breaking changes block names the removal version.\n  - Surfaced by: Friction D7 \u2014 docs/api.md:15-18\n  - Files: evalkit/client.py, docs/migrating-v1-to-v2.md, evalkit/codemod.py (or documented recipe), CHANGELOG\n  - Verify: v1 sample script runs under 2.0.0b1 with one DeprecationWarning; codemod converts the fixture to Client.run() and keyword args\n- [ ] **T6 (P2, human: ~1 hour / CC: ~5 min)** \u2014 demo output \u2014 Append next-step footer to `python -m evalkit.demo` (console URL, export line, first_eval.py command; skip key line if EVALKIT_API_KEY is set); update README.md:34-36 and the output fixture.\n  - Surfaced by: Pass 1 D10\n  - Files: evalkit/demo.py, README.md\n  - Verify: demo output ends with the footer; with EVALKIT_API_KEY set the key line is omitted\n- [ ] **T7 (P2, human: ~half day / CC: n/a)** \u2014 release gate \u2014 Re-run the docs/benchmarks.md onboarding protocol on the release candidate; record the measured TTHW; block the beta if 2 minutes or more.\n  - Surfaced by: Pass 8 D11\n  - Files: docs/benchmarks.md, release checklist\n  - Verify: docs/benchmarks.md has a dated 2.0.0b1 row under 2 minutes\n- [ ] **T8 (P3, human: ~1 hour / CC: ~5 min)** \u2014 follow-up \u2014 TODOS.md entry: remove Client.evaluate() alias at the changelog's stated version.\n  - Surfaced by: TODO D12\n  - Files: TODOS.md\n  - Verify: entry present with version, file, and remediation path\n- [ ] **T9 (P3, human: ~5 min / CC: ~1 min)** \u2014 repo setup \u2014 Append gstack skill routing section to CLAUDE.md and commit (`chore: add gstack skill routing rules to CLAUDE.md`), after plan mode exits.\n  - Surfaced by: D0 onboarding prompt\n  - Files: CLAUDE.md\n  - Verify: `git log -1` shows the commit; CLAUDE.md ends with the Skill routing section\n\n_No new tasks from Pass 2, Pass 3, Pass 4, Pass 5 beyond T3-T5; no new tasks from Pass 6 or Pass 7._\n\n## Verification (end to end)\n\n1. Fresh virtualenv on Linux or macOS; `python -m pip install evalkit==2.0.0b1` from the release candidate index.\n2. `python -m evalkit.demo` with network disabled: three score lines plus the next-step footer, under 15 seconds.\n3. `export EVALKIT_API_KEY=bad` then `python examples/first_eval.py`: AuthError with EVALKIT_AUTH_INVALID_KEY, console URL, masked key.\n4. Valid key: first_eval.py performs the binding check once (progress lines retained), then prints a real result.\n5. v1 script calling `Client.evaluate()`: runs, emits one DeprecationWarning naming `Client.run()`.\n6. `run_batch(ds, ev)` positional: TypeError names dataset and evaluator.\n7. Stopwatch run of steps 1 to 4 by a developer who has not used the beta: under 2 minutes to first real result (D11); record in docs/benchmarks.md.\n\n## Unresolved Decisions\n\nNone. D0 through D12 were each answered.\n\n## GSTACK REVIEW REPORT\n\n| Review | Trigger | Why | Runs | Status | Findings |\n|--------|---------|-----|------|--------|----------|\n| CEO Review | `/plan-ceo-review` | Scope & strategy | 0 | \u2014 | \u2014 |\n| Outside Review | codex via `/plan-devex-review` | Independent 2nd opinion | 1 | disabled | codex_reviews disabled; no substitute reviewer run |\n| Eng Review | `/plan-eng-review` | Architecture & tests (required) | 0 | \u2014 | \u2014 |\n| Design Review | `/plan-design-review` | UI/UX gaps | 0 | \u2014 | \u2014 |\n| DX Review | `/plan-devex-review` | Developer experience gaps | 1 | clean | score: 5/10 \u2192 8.5/10, TTHW: 6 min \u2192 <2 min (est., gated by D11); 5 contract repairs + 2 polish additions |\n\n- **OUTSIDE COVERAGE:** provider codex, phase plan-review, outside_status disabled (2026-09-10T11:44:36Z). No findings; coverage intentionally absent.\n- **VERDICT:** DX CLEARED \u2014 eng review required\n\nNO UNRESOLVED DECISIONS\n",
  "screen": "  |                    REVIEW READINESS DASHBOARD                       |\n  +====================================================================+\n  | Review          | Runs | Last Run            | Status    | Required |\n  |-----------------|------|---------------------|-----------|----------|\n  | Eng Review      |  0   | \u2014                   | \u2014         | YES      |\n  | CEO Review      |  0   | \u2014                   | \u2014         | no       |\n  | Design Review   |  0   | \u2014                   | \u2014         | no       |\n  | Adversarial     |  0   | \u2014                   | \u2014         | no       |\n  | Outside Voice   |  1   | 2026-09-10 11:44    | DISABLED  | no       |\n  | DX Review       |  1   | 2026-09-10 11:46    | CLEAR     | no       |\n  +--------------------------------------------------------------------+\n  | VERDICT: NOT CLEARED \u2014 eng review required (you chose to run it manually) |\n  +====================================================================+\n\n  STATUS: DONE. Plan written to\n  /tmp/gstack-paid-shard-H02ip0/tmp/gstack-e2e-plan-devex-AnQlB9/gstack-test-plan-devex.md.\n\n  What the review decided, all on the recommended option:\n  - Five contract repairs (T1 to T5, P1): CI check moved off the keyless demo and first local eval; quickstart reordered\n    with examples/first_eval.py actually shipped; run_eval and run_batch aligned and keyword-only; AuthError gets\n    codes, cause, fix, and a masked key; Client.evaluate() kept as a deprecation alias with a migration guide and\n    codemod.\n  - Two polish additions (T6, T7, P2): demo prints a next-step footer; the onboarding study is re-run on the release\n    candidate as an under-2-minute ship gate.\n  - Follow-ups (T8, T9, P3): TODO to remove the alias at the stated version; append gstack routing rules to CLAUDE.md\n    and commit once plan mode exits.\n\n  Scores moved from 5.0 to 8.5 overall, with TTHW estimated at about 1 minute pending the T7 measurement. Outside Codex\n  review is disabled in this install and was recorded as such, not substituted. One learning logged: this repo\n  pre-settles the Step 0 inputs, so future DX reviews should skip straight to the contract issues. Eng review is still\n  open and left to you, as requested.\n\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n Exit plan mode?\n\n  Claude wants to exit plan mode\n\n  \u276f 1. Yes, and switch to default (ask each time) for this session\n    2. No\n"
}