mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-05 00:03:15 +01:00
feat(eval-plan): the approval step prices every paid gate with price-gate
Step 4 runs gstack-eval-plan price-gate for each paid gate the plan names and puts its GATE_PRICE line in the ask; the dry-run checklist documents reservation-fits-cap under --cap. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
958c7b34c1
commit
b26b000ee9
2 files changed
+28
-2
No files matched your search
+14
-1
@@ -428,6 +428,8 @@ the budget the run will use. The checklist printed must be all `ok`:
|
||||
spends output tokens thinking, and a smaller budget truncates the answer the scorer reads.
|
||||
- `CHECK: retry-prompt` — the exact text the run sends on a retry, printed verbatim; a retry
|
||||
with no text is a repeat of the same failure at the same price.
|
||||
- `CHECK: reservation-fits-cap` (with `--cap <cap-usd>`) — one call, the largest row plus the
|
||||
output budget, priced, fits under the cap; a provider refuses a reservation the cap cannot hold.
|
||||
- `DRY_RUN_ROW:` lines name rows missing a field; fix the rows, not the prompt.
|
||||
|
||||
`DRY_RUN_RESULT: pass` writes `dry-run.json` to `--out`; the pilot refuses to start on
|
||||
@@ -459,8 +461,19 @@ the preregistration, when `status=budget_exhausted`, or when `verdict=exceeds`.
|
||||
|
||||
## Step 4: The approval ask
|
||||
|
||||
Price every paid gate the plan names (a benchmark or eval command that spends) from the
|
||||
repo's call-plan file; nothing is executed (`docs/eval-calls.md` in gstack):
|
||||
|
||||
```bash
|
||||
~/.claude/skills/gstack/bin/gstack-eval-plan price-gate --gate "<exact gate command>" --cap-usd <cap-usd>
|
||||
```
|
||||
|
||||
Each `GATE_PRICE:` line goes into the ask. `EVAL_GATE_UNPRICED`, `EVAL_PLAN_STALE`,
|
||||
`EVAL_PRICE_MISSING` or `EVAL_GATE_OVER_CAP` means that gate is not approvable as priced;
|
||||
`--allow-unpriced "<reason>"` only on the owner's word.
|
||||
|
||||
Present, in this order and nothing before it: the `PILOT_PROJECTION:` line, the spend
|
||||
table, the pilot's per-bar results, and the preregistration's stop rules. Then ask one
|
||||
table, each gate's `GATE_PRICE:` line, the pilot's per-bar results, and the preregistration's stop rules. Then ask one
|
||||
question with the projection as the recommended option only when `verdict=within`:
|
||||
|
||||
- A) Run `<total-items>` items at the projected cost (recommended when within the cap)
|
||||
|
||||
+14
-1
@@ -100,6 +100,8 @@ the budget the run will use. The checklist printed must be all `ok`:
|
||||
spends output tokens thinking, and a smaller budget truncates the answer the scorer reads.
|
||||
- `CHECK: retry-prompt` — the exact text the run sends on a retry, printed verbatim; a retry
|
||||
with no text is a repeat of the same failure at the same price.
|
||||
- `CHECK: reservation-fits-cap` (with `--cap <cap-usd>`) — one call, the largest row plus the
|
||||
output budget, priced, fits under the cap; a provider refuses a reservation the cap cannot hold.
|
||||
- `DRY_RUN_ROW:` lines name rows missing a field; fix the rows, not the prompt.
|
||||
|
||||
`DRY_RUN_RESULT: pass` writes `dry-run.json` to `--out`; the pilot refuses to start on
|
||||
@@ -131,8 +133,19 @@ the preregistration, when `status=budget_exhausted`, or when `verdict=exceeds`.
|
||||
|
||||
## Step 4: The approval ask
|
||||
|
||||
Price every paid gate the plan names (a benchmark or eval command that spends) from the
|
||||
repo's call-plan file; nothing is executed (`docs/eval-calls.md` in gstack):
|
||||
|
||||
```bash
|
||||
~/.claude/skills/gstack/bin/gstack-eval-plan price-gate --gate "<exact gate command>" --cap-usd <cap-usd>
|
||||
```
|
||||
|
||||
Each `GATE_PRICE:` line goes into the ask. `EVAL_GATE_UNPRICED`, `EVAL_PLAN_STALE`,
|
||||
`EVAL_PRICE_MISSING` or `EVAL_GATE_OVER_CAP` means that gate is not approvable as priced;
|
||||
`--allow-unpriced "<reason>"` only on the owner's word.
|
||||
|
||||
Present, in this order and nothing before it: the `PILOT_PROJECTION:` line, the spend
|
||||
table, the pilot's per-bar results, and the preregistration's stop rules. Then ask one
|
||||
table, each gate's `GATE_PRICE:` line, the pilot's per-bar results, and the preregistration's stop rules. Then ask one
|
||||
question with the projection as the recommended option only when `verdict=within`:
|
||||
|
||||
- A) Run `<total-items>` items at the projected cost (recommended when within the cap)
|
||||
|
||||
Reference in new issue
Block a user