mirror of
https://github.com/garrytan/gstack.git
synced 2026-07-06 16:17:56 +02:00
chore: cap audit — remove distill rate cap, loosen size/budget gates
Plan-tune cathedral follow-up. The 3/day distill cap was theatrical: at ~$0.01 per Haiku call, even a runaway loop firing every minute would cost ~$14/day, and free-text events are rare enough that the natural input rate self-limits to 1-2 fires/day. Count caps don't protect against runaway bugs (which fire 1000x/second, not 4 times/day) but DO punish heavy users who'd legitimately distill multiple times during a busy week. Removed: 3/day rate cap on bin/gstack-distill-free-text. --status output swapped from "TODAY: N / 3" to "TODAY: N run(s), $X" so users see what they're spending instead of how close they are to a meaningless count. Loosened (caps that exist for real-runaway protection, not normal scope): - EVALS_BUDGET_HARD_CAP_GATE $25 → $200/run - EVALS_BUDGET_HARD_CAP_PERIODIC $70 → $500/run - EVALS_BUDGET_HARD_CAP $30 → $300/run (umbrella fallback) - GSTACK_SIZE_BUDGET_RATIO 1.05 → 1.50 per-skill ratio - plan-review preamble byte budget 40K → 60K Principle: caps exist to catch obvious bugs (infinite retry, model price change, prompt blowup), not to gate legitimate scope growth. Set high enough that real growth never trips them, only bug territory does. Adjusted defaults are 4-8× historical worst case, leaving ample headroom for the next 12 months of legitimate expansion. Tests updated: distill-free-text removes the 3-test rate-cap describe block in favor of "no rate cap" assertion that 10 runs/day pass. Other budget tests still pass because they were never near the old ceilings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -41,20 +41,24 @@ import { logBudgetOverride } from './helpers/budget-override';
|
||||
* v1.45.0.0 T5 — hard eval cost cap.
|
||||
*
|
||||
* Per-tier defaults (override via env):
|
||||
* EVALS_BUDGET_HARD_CAP_GATE default $25/run
|
||||
* EVALS_BUDGET_HARD_CAP_PERIODIC default $70/run
|
||||
* EVALS_BUDGET_HARD_CAP umbrella cap if a tier-specific isn't set; default $30
|
||||
* EVALS_BUDGET_HARD_CAP_GATE default $200/run
|
||||
* EVALS_BUDGET_HARD_CAP_PERIODIC default $500/run
|
||||
* EVALS_BUDGET_HARD_CAP umbrella cap if a tier-specific isn't set; default $300
|
||||
* EVALS_BUDGET_OVERRIDE_REASON if set, override fires AND audit-logs to
|
||||
* ~/.gstack/analytics/spend-overrides.jsonl
|
||||
*
|
||||
* Caps are dollars-per-run, not dollars-per-test. A test that legitimately
|
||||
* gets more expensive should bake into the baseline; a runaway eval (infinite
|
||||
* retry, model price change) gets stopped here.
|
||||
* Caps are dollars-per-run, not dollars-per-test. The cap exists to catch
|
||||
* runaway evals (infinite retry, model price change, prompt-blowup bug),
|
||||
* NOT to gate legitimate scope growth. Set high enough that real growth
|
||||
* never trips it — only obvious-bug territory does. Adjusted v1.52.0.0
|
||||
* (cathedral cap audit): $25 → $200 gate, $70 → $500 periodic. Prior
|
||||
* defaults tripped on normal-scope expansion; new ceilings are 8× the
|
||||
* historical worst-case eval run.
|
||||
*/
|
||||
const DEFAULT_HARD_CAP_USD = Number(process.env.EVALS_BUDGET_HARD_CAP) || 30;
|
||||
const DEFAULT_HARD_CAP_USD = Number(process.env.EVALS_BUDGET_HARD_CAP) || 300;
|
||||
const TIER_CAPS: Record<'e2e' | 'llm-judge', number> = {
|
||||
e2e: Number(process.env.EVALS_BUDGET_HARD_CAP_GATE) || DEFAULT_HARD_CAP_USD,
|
||||
'llm-judge': Number(process.env.EVALS_BUDGET_HARD_CAP_PERIODIC) || Math.max(70, DEFAULT_HARD_CAP_USD),
|
||||
e2e: Number(process.env.EVALS_BUDGET_HARD_CAP_GATE) || Math.min(200, DEFAULT_HARD_CAP_USD),
|
||||
'llm-judge': Number(process.env.EVALS_BUDGET_HARD_CAP_PERIODIC) || Math.max(500, DEFAULT_HARD_CAP_USD),
|
||||
};
|
||||
|
||||
function currentGitBranch(): string {
|
||||
|
||||
Reference in New Issue
Block a user