chore: cap audit — remove distill rate cap, loosen size/budget gates

Plan-tune cathedral follow-up. The 3/day distill cap was theatrical: at
~$0.01 per Haiku call, even a runaway loop firing every minute would cost
~$14/day, and free-text events are rare enough that the natural input
rate self-limits to 1-2 fires/day. Count caps don't protect against
runaway bugs (which fire 1000x/second, not 4 times/day) but DO punish
heavy users who'd legitimately distill multiple times during a busy week.

Removed: 3/day rate cap on bin/gstack-distill-free-text. --status output
swapped from "TODAY: N / 3" to "TODAY: N run(s), $X" so users see what
they're spending instead of how close they are to a meaningless count.

Loosened (caps that exist for real-runaway protection, not normal scope):
- EVALS_BUDGET_HARD_CAP_GATE   $25 → $200/run
- EVALS_BUDGET_HARD_CAP_PERIODIC $70 → $500/run
- EVALS_BUDGET_HARD_CAP        $30 → $300/run (umbrella fallback)
- GSTACK_SIZE_BUDGET_RATIO     1.05 → 1.50 per-skill ratio
- plan-review preamble byte budget 40K → 60K

Principle: caps exist to catch obvious bugs (infinite retry, model price
change, prompt blowup), not to gate legitimate scope growth. Set high
enough that real growth never trips them, only bug territory does.
Adjusted defaults are 4-8× historical worst case, leaving ample headroom
for the next 12 months of legitimate expansion.

Tests updated: distill-free-text removes the 3-test rate-cap describe
block in favor of "no rate cap" assertion that 10 runs/day pass. Other
budget tests still pass because they were never near the old ceilings.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-28 17:59:28 -07:00
parent 401a3258a3
commit e9ad7527ea
5 changed files with 41 additions and 69 deletions
+13 -9
View File
@@ -41,20 +41,24 @@ import { logBudgetOverride } from './helpers/budget-override';
* v1.45.0.0 T5 — hard eval cost cap.
*
* Per-tier defaults (override via env):
* EVALS_BUDGET_HARD_CAP_GATE default $25/run
* EVALS_BUDGET_HARD_CAP_PERIODIC default $70/run
* EVALS_BUDGET_HARD_CAP umbrella cap if a tier-specific isn't set; default $30
* EVALS_BUDGET_HARD_CAP_GATE default $200/run
* EVALS_BUDGET_HARD_CAP_PERIODIC default $500/run
* EVALS_BUDGET_HARD_CAP umbrella cap if a tier-specific isn't set; default $300
* EVALS_BUDGET_OVERRIDE_REASON if set, override fires AND audit-logs to
* ~/.gstack/analytics/spend-overrides.jsonl
*
* Caps are dollars-per-run, not dollars-per-test. A test that legitimately
* gets more expensive should bake into the baseline; a runaway eval (infinite
* retry, model price change) gets stopped here.
* Caps are dollars-per-run, not dollars-per-test. The cap exists to catch
* runaway evals (infinite retry, model price change, prompt-blowup bug),
* NOT to gate legitimate scope growth. Set high enough that real growth
* never trips it — only obvious-bug territory does. Adjusted v1.52.0.0
* (cathedral cap audit): $25 → $200 gate, $70 → $500 periodic. Prior
* defaults tripped on normal-scope expansion; new ceilings are 8× the
* historical worst-case eval run.
*/
const DEFAULT_HARD_CAP_USD = Number(process.env.EVALS_BUDGET_HARD_CAP) || 30;
const DEFAULT_HARD_CAP_USD = Number(process.env.EVALS_BUDGET_HARD_CAP) || 300;
const TIER_CAPS: Record<'e2e' | 'llm-judge', number> = {
e2e: Number(process.env.EVALS_BUDGET_HARD_CAP_GATE) || DEFAULT_HARD_CAP_USD,
'llm-judge': Number(process.env.EVALS_BUDGET_HARD_CAP_PERIODIC) || Math.max(70, DEFAULT_HARD_CAP_USD),
e2e: Number(process.env.EVALS_BUDGET_HARD_CAP_GATE) || Math.min(200, DEFAULT_HARD_CAP_USD),
'llm-judge': Number(process.env.EVALS_BUDGET_HARD_CAP_PERIODIC) || Math.max(500, DEFAULT_HARD_CAP_USD),
};
function currentGitBranch(): string {