mirror of
https://github.com/garrytan/gstack.git
synced 2026-05-07 05:56:41 +02:00
Merge remote-tracking branch 'origin/main' into garrytan/workspace-aware-ship
Rebumped v1.8.0.0 -> v1.11.0.0 (minor-past main's v1.10.1.0) using bin/gstack-next-version — the same queue-aware path this branch introduces. CHANGELOG repositioned so v1.11.0.0 sits above main's new entries (v1.10.1.0 / v1.10.0.0 / v1.9.0.0). Conflicts resolved: - VERSION, package.json: rebumped to v1.11.0.0 (util-picked) - bin/gstack-config: merged both lists (workspace_root + gbrain keys) - CHANGELOG.md: hoisted v1.11.0.0 entry above main's new entries Pre-existing failures in main (4) documented but not fixed in this PR: 1. gstack-brain-sync secret scan > blocks bearer-json (brain-sync tests) 2. no files larger than 2MB (security-bench fixture, already TODO'd) 3. selectTests > skill-specific change (touchfiles scoping) 4. Opus 4.7 overlay pacing directive (expectation stale after v1.10.1.0 removed the Fan out nudge) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -38,22 +38,6 @@
|
||||
**Priority:** P3 (nice-to-have, not blocking anyone yet)
|
||||
**Depends on:** `/context-save` + `/context-restore` rename stable in production (v1.0.1.0+). Research: does Conductor expose a spawn-workspace CLI?
|
||||
|
||||
## P0: Verify Opus 4.7 fanout nudge inside Claude Code harness (next rev)
|
||||
|
||||
**What:** Re-run the fanout A/B from `test/skill-e2e-opus-47.test.ts` against Opus 4.7 **inside Claude Code's interactive harness**, not via `claude -p`. The current eval calls `claude -p` as a subprocess, which does not load SKILL.md content as system context and uses different tool wiring than the live Claude Code session. Build a small harness (Claude Code extension hook, direct API call with the same system prompt Claude Code uses, or a scripted MCP invocation) that reproduces the real tool_use context, then run the same 3-file-read A/B with and without the `model-overlays/opus-4-7.md` overlay. Record parallel-tool-call count in the first assistant turn for each arm.
|
||||
|
||||
**Why:** v1.6.1.0 shipped a rewritten "Fan out explicitly" nudge with a concrete tool_use example (`[Read(a), Read(b), Read(c)]`). Under `claude -p` on `claude-opus-4-7`, both overlay-ON and overlay-OFF arms emitted zero parallel tool calls in the first turn. The routing A/B worked fine in the same harness (3/3 positives routed correctly), so the gap is specific to fanout, and likely specific to how `claude -p` constructs system prompts and tool schemas. Without measurement inside the real harness, we do not know whether the nudge ever lands for a real user. The PR went to production with the fanout claim asserted but unverified; this TODO closes that loop.
|
||||
|
||||
**Pros:** Produces the "actually shipped fanout" measurement the ship-quality review flagged as missing. If the nudge works in Claude Code harness, we can gate it with a `periodic` eval and stop worrying. If it does not, we know to rewrite or drop the nudge rather than carry dead prompt weight. Either answer is better than the current "unverified."
|
||||
|
||||
**Cons:** Requires instrumenting Claude Code's harness (or a faithful replica) rather than the easier `claude -p` path. A faithful replica needs the same system prompt, the same tool definitions, and the same stop-sequence handling. Estimated one afternoon to wire, plus $3-5 per eval run.
|
||||
|
||||
**Context:** See `~/.gstack/projects/garrytan-gstack/evals/1.6.0.0-feat-opus-4.7-migration-e2e-opus-47-*.json` for the raw transcripts showing 0 parallel calls in first turn across both arms. The overlay is at `model-overlays/opus-4-7.md` with an explicit wrong/right tool_use example. The eval file at `test/skill-e2e-opus-47.test.ts` has the full setup including per-skill SKILL.md install, CLAUDE.md routing block, and overlay inlining.
|
||||
|
||||
**Effort:** M (human: ~1 day / CC: ~45 min for the harness wiring, plus the eval run cost)
|
||||
**Priority:** P0 (ship-quality commitment from v1.6.1.0 — do not let it drift)
|
||||
**Depends on / blocked by:** Access to Claude Code's system prompt + tool schema (or a reproducible way to mirror them). May require a small MCP server or a direct Messages API call that mirrors Claude Code's session setup.
|
||||
|
||||
## P0: PACING_UPDATES_V0 — Louise's fatigue root cause (V1.1)
|
||||
|
||||
**What:** Implement the pacing overhaul extracted from PLAN_TUNING_V1. Full design in `docs/designs/PACING_UPDATES_V0.md`. Requires: session-state model, `phase` field in question-log schema, registry extension for dynamic findings, pacing as skill-template control flow (not preamble prose), `bin/gstack-flip-decision` command, migration-prompt budget rule, first-run preamble audit, ranking threshold calibration from real V0 data, one-way-door uncapped rule, concrete verification values.
|
||||
@@ -1288,6 +1272,15 @@ Shipped in v0.6.5. TemplateContext in gen-skill-docs.ts bakes skill name into pr
|
||||
|
||||
## Completed
|
||||
|
||||
### Overlay efficacy harness + Opus 4.7 fanout nudge removal (v1.10.1.0)
|
||||
- Built `test/skill-e2e-overlay-harness.test.ts`, a parametric periodic-tier eval that drives `@anthropic-ai/claude-agent-sdk` and measures first-turn fanout rate (overlay-ON vs overlay-OFF) across registered fixtures
|
||||
- Measured the original "Fan out explicitly" overlay nudge: baseline Opus 4.7 = 70% first-turn fanout on toy prompt, with our nudge = 10%, with Anthropic's own canonical `<use_parallel_tool_calls>` text = 0%
|
||||
- Removed the counterproductive nudge from `model-overlays/opus-4-7.md`
|
||||
- Shipped 36-test free-tier unit suite for the SDK runner + strict fixture validator
|
||||
- Registered `overlay-harness-opus-4-7-fanout-{toy,realistic}` in E2E_TOUCHFILES and E2E_TIERS
|
||||
- Total investigation cost: ~$7 across 3 eval runs
|
||||
**Completed:** v1.10.1.0
|
||||
|
||||
### CI eval pipeline (v0.9.9.0)
|
||||
- GitHub Actions eval upload on Ubicloud runners ($0.006/run)
|
||||
- Within-file test concurrency (test() → testConcurrentIfSelected())
|
||||
|
||||
Reference in New Issue
Block a user