diff --git a/eval-plan/SKILL.md b/eval-plan/SKILL.md index e4a390ff6..402c07414 100644 --- a/eval-plan/SKILL.md +++ b/eval-plan/SKILL.md @@ -428,6 +428,8 @@ the budget the run will use. The checklist printed must be all `ok`: spends output tokens thinking, and a smaller budget truncates the answer the scorer reads. - `CHECK: retry-prompt` — the exact text the run sends on a retry, printed verbatim; a retry with no text is a repeat of the same failure at the same price. +- `CHECK: reservation-fits-cap` (with `--cap `) — one call, the largest row plus the + output budget, priced, fits under the cap; a provider refuses a reservation the cap cannot hold. - `DRY_RUN_ROW:` lines name rows missing a field; fix the rows, not the prompt. `DRY_RUN_RESULT: pass` writes `dry-run.json` to `--out`; the pilot refuses to start on @@ -459,8 +461,19 @@ the preregistration, when `status=budget_exhausted`, or when `verdict=exceeds`. ## Step 4: The approval ask +Price every paid gate the plan names (a benchmark or eval command that spends) from the +repo's call-plan file; nothing is executed (`docs/eval-calls.md` in gstack): + +```bash +~/.claude/skills/gstack/bin/gstack-eval-plan price-gate --gate "" --cap-usd +``` + +Each `GATE_PRICE:` line goes into the ask. `EVAL_GATE_UNPRICED`, `EVAL_PLAN_STALE`, +`EVAL_PRICE_MISSING` or `EVAL_GATE_OVER_CAP` means that gate is not approvable as priced; +`--allow-unpriced ""` only on the owner's word. + Present, in this order and nothing before it: the `PILOT_PROJECTION:` line, the spend -table, the pilot's per-bar results, and the preregistration's stop rules. Then ask one +table, each gate's `GATE_PRICE:` line, the pilot's per-bar results, and the preregistration's stop rules. Then ask one question with the projection as the recommended option only when `verdict=within`: - A) Run `` items at the projected cost (recommended when within the cap) diff --git a/eval-plan/SKILL.md.tmpl b/eval-plan/SKILL.md.tmpl index 584ce2b45..12f8136d3 100644 --- a/eval-plan/SKILL.md.tmpl +++ b/eval-plan/SKILL.md.tmpl @@ -100,6 +100,8 @@ the budget the run will use. The checklist printed must be all `ok`: spends output tokens thinking, and a smaller budget truncates the answer the scorer reads. - `CHECK: retry-prompt` — the exact text the run sends on a retry, printed verbatim; a retry with no text is a repeat of the same failure at the same price. +- `CHECK: reservation-fits-cap` (with `--cap `) — one call, the largest row plus the + output budget, priced, fits under the cap; a provider refuses a reservation the cap cannot hold. - `DRY_RUN_ROW:` lines name rows missing a field; fix the rows, not the prompt. `DRY_RUN_RESULT: pass` writes `dry-run.json` to `--out`; the pilot refuses to start on @@ -131,8 +133,19 @@ the preregistration, when `status=budget_exhausted`, or when `verdict=exceeds`. ## Step 4: The approval ask +Price every paid gate the plan names (a benchmark or eval command that spends) from the +repo's call-plan file; nothing is executed (`docs/eval-calls.md` in gstack): + +```bash +~/.claude/skills/gstack/bin/gstack-eval-plan price-gate --gate "" --cap-usd +``` + +Each `GATE_PRICE:` line goes into the ask. `EVAL_GATE_UNPRICED`, `EVAL_PLAN_STALE`, +`EVAL_PRICE_MISSING` or `EVAL_GATE_OVER_CAP` means that gate is not approvable as priced; +`--allow-unpriced ""` only on the owner's word. + Present, in this order and nothing before it: the `PILOT_PROJECTION:` line, the spend -table, the pilot's per-bar results, and the preregistration's stop rules. Then ask one +table, each gate's `GATE_PRICE:` line, the pilot's per-bar results, and the preregistration's stop rules. Then ask one question with the projection as the recommended option only when `verdict=within`: - A) Run `` items at the projected cost (recommended when within the cap)