mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-05 00:03:15 +01:00
fix(review): docs and results PRs stop paying for a money reader; reported costs aren't money (GST-961, GST-962) (#3942)
* test(review-risk): failing tests for incidental money signals (GST-961, GST-962) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(review-risk): unread data files and cost_usd reads no longer raise money (GST-961, GST-962) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(review-risk): keep reservations, estimates, spend guards and pricing tables money (GST-961, GST-962 review) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(review-risk): a <> inequality on a reported cost stays money (GST-962 round-2 grant) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * changelog: GST-961+962 fragment under the branch slug Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: money row names the full reported-cost rule and the results-tree exemption Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(review): receipts/ joins the config price-key carve-out (#3920's review-exclude fixture) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(review-pins): the merged repin's justification says the no-loss replays were not rerun (GST-961+962) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
ad6e1db159
commit
526f604f91
12 files changed
+140
-21
No files matched your search
@@ -0,0 +1,8 @@
|
||||
### Fixed
|
||||
|
||||
- **A docs or data PR no longer gets a money reader because a file name says "cost", and summing or printing a `cost_usd` field no longer counts as money code (GST-961, GST-962).** On gstack-evals#138, a results write-up planned as docs at light depth still dispatched a second, targeted money reader, because `wiki/.../cost-gstack-full.tsv` has "cost" in its name. On gstack-evals#139, an analysis module that only sums and prints per-trial `cost_usd` raised the money class. Now a data file that no code reads, under `docs/`, `wiki/`, `results/` or `reports/`, never counts as money for its name or for a reported cost. A `spend_cap` line or a paid-provider call in that file still counts. A `_usd` or `Usd` name is not money only when it is a reported cost (made of words like cost, total, per, trial or sum) on a line with no cap, limit, budget, price, charge, reservation or estimate word, no `<`, `>` or `<>` comparison and no number set on it. Reservations, estimates, amounts, refunds, spend guards and pricing tables all still count. Over the last 100 main commits, no commit's depth, class, risk classes or targeted classes changed.
|
||||
|
||||
### Tests
|
||||
|
||||
- `test/review-targeted.test.ts`: #138's file list plans no targeted reader, and the same file read by code still plans money. A `cost_usd` that is only summed or printed names no money. Positive controls: caps, budgets, limits, optional-chain and wrapped comparisons, `Math.min`, charges and refunds, reservations and estimates from `lib/eval-plan.ts` and `lib/autoplan-run.ts`, the `trial_reservation_usd` validator, a pricing table outside a results tree, a spend cap in an unread results file, and a secret in an unread data file.
|
||||
- `test/review-risk-redos.test.ts`: `usdMoney` and its comparison patterns stay under 250 ms on 100,000-character lines.
|
||||
@@ -429,7 +429,7 @@ and matching is token-bounded: `costume` never reads as cost.
|
||||
|
||||
| Class | Signal |
|
||||
|---|---|
|
||||
| money | a path segment naming cost, spend, budget, ledger, billing, price, fee or invoice; or a changed line with an identifier ending in `_usd` or `Usd`, any `spend_cap` spelling, or a call to a paid provider `lib/pricing.ts` prices |
|
||||
| money | a path segment naming cost, spend, budget, ledger, billing, price, fee or invoice; or a changed line with an identifier ending in `_usd` or `Usd`, unless every such name on the line is a reported cost (made only of words like cost, total, per, trial, mean or sum) and the line has no cap, limit, budget, spend, price, fee, charge, refund, amount, reservation or estimate word (the full list is `MONEY_USD_WORDS` in `lib/review-risk.ts`), no `<` or `>` comparison and no number set on it: summing or printing `cost_usd` is not money (GST-962); any `spend_cap` spelling; or a call to a paid provider `lib/pricing.ts` prices. A data file no code reads (the same consumer scan the depth uses) under a `doc/`, `docs/`, `wiki/`, `results/` or `reports/` directory counts as money only for a `spend_cap` or paid-provider line, never for its name, a `_usd` or `Usd` identifier (reported cost or not) or a metering word (GST-961); every other class still reads it |
|
||||
| gate | a hunk the gate scan tagged: a relaxed threshold, a skipped test, a loosened check |
|
||||
| auth | `SCOPE_AUTH` from `gstack-diff-scope`, on a code path with an auth, session, jwt, permission or role token in its name; a data file named for sessions (`recovered-sessions.jsonl`) outside a config location does not set `SCOPE_AUTH`, and `SCOPE_AUTH` fails safe: any name token containing auth sets it (`auth.ts`, `/auth/`, `oauth`, `useAuth.ts`, `nextauthmiddleware.ts`) except the authorship words `author`, `authors`, `authored`, `authoring`, `authorship`, `authority`, `authorities`, `coauthor` and `coauthors` |
|
||||
| secret | a secrets, credentials, keychain or token path, or a changed line that reads a key-named environment variable (`OPENAI_API_KEY`, `GH_TOKEN`) or a credential store |
|
||||
|
||||
+1
-1
@@ -254,7 +254,7 @@ export interface ClassResult { class: DiffClass; reasons: string[]; errors: Dept
|
||||
|
||||
const sanitize = (s: string) => s.replace(/[\0-\x1f]+/g, ' ').slice(0, 200);
|
||||
const isOutput = (i: ClassInput, p: string) => !!i.registry?.outputs.length && matchesAny(i.registry.outputs, p);
|
||||
const unconsumedData = (i: ClassInput, p: string) => !i.executables?.includes(p) && Array.isArray(i.consumed) && !i.consumed.includes(p) && (dataCandidate(p, i.root) || (!!i.drawings?.includes(p) && svgCandidate(p, i.root)));
|
||||
export const unconsumedData = (i: Pick<ClassInput, 'consumed' | 'drawings' | 'root' | 'executables'>, p: string) => !i.executables?.includes(p) && Array.isArray(i.consumed) && !i.consumed.includes(p) && (dataCandidate(p, i.root) || (!!i.drawings?.includes(p) && svgCandidate(p, i.root)));
|
||||
const binaryData = (i: ClassInput, p: string) => !i.executables?.includes(p) && BINARY_DATA_PATH.test(p) && classifyPath(p, i.root) === 'code' && ![...AGENT_PROSE, ...GATE_PATH, ...DATA_CONFIG].some(r => r.test(p));
|
||||
const pathClass = (i: ClassInput, p: string): PathClass => i.executables?.includes(p) && !i.excluded?.has(p) ? 'code' : (i.excluded?.has(p) || (i.stampOnly ?? []).includes(p) || unconsumedData(i, p) || binaryData(i, p) ? 'docs' : classifyPath(p, i.root));
|
||||
|
||||
|
||||
+32
-10
@@ -31,14 +31,33 @@ const SPEND_BIND = ['reserve', 'reserves', 'reserved', 'reservation', 'reservati
|
||||
const SPEND_OF = ['ledger', 'ledgers', 'spend', 'budget', 'budgets', 'usd', 'cost', 'costs', 'credits', 'billing', 'cap', 'caps'];
|
||||
export const SPEND_PAIRS: readonly (readonly [string, readonly string[]])[] = [...SPEND_BIND.map(w => [w, SPEND_OF] as const), ['cap', ['spend', 'usd', 'budget', 'ledger']]];
|
||||
export const PROSE_PATH = /\.(md|mdx|markdown|txt|rst|adoc|tmpl)$/i;
|
||||
/** Identifiers ending in _usd, Usd or cap_usd, any spend_cap spelling, and paid-provider call sites (the providers lib/pricing.ts prices). */
|
||||
export const MONEY_LINE = [
|
||||
/\b[A-Za-z0-9_]*_usd\b/i, /\b[a-z][A-Za-z0-9]*Usd\b/, /\bspend[_-]?cap\b/i,
|
||||
/** Identifiers ending in _usd or Usd: money unless the line only reads or reports a cost (usdMoney). */
|
||||
export const MONEY_USD = [/\b[A-Za-z0-9_]*_usd\b/i, /\b[a-z][A-Za-z0-9]*Usd\b/];
|
||||
const MONEY_USD_ALL = MONEY_USD.map(r => new RegExp(r.source, r.flags + 'g'));
|
||||
/** The words a reported cost's name is made of (GST-962: gstack-evals#139's per-trial cost_usd); any other word, a reservation, estimate or amount, keeps the name money. */
|
||||
export const REPORTED_COST_WORDS = ['cost', 'costs', 'total', 'totals', 'usd', 'per', 'trial', 'trials', 'item', 'items', 'mean', 'avg', 'average', 'median', 'sum', 'streamed', 'transcript', 'observed', 'actual', 'reported', 'recorded'];
|
||||
/** A reported cost is still money on a line naming a cap, limit, budget, price, charge, reservation or estimate. */
|
||||
export const MONEY_USD_WORDS = ['cap', 'caps', 'limit', 'limits', 'max', 'maximum', 'min', 'budget', 'budgets', 'ceiling', 'allowance', 'threshold', 'spend', 'spent', 'spending', 'price', 'prices', 'pricing', 'fee', 'fees', 'bill', 'billing', 'paid', 'payment', 'charge', 'refund', 'amount', 'invoice', 'reserve', 'reservation', 'estimate'];
|
||||
/** A reported cost compared with `<` or `>` (never an arrow or a shift), or set to a number. */
|
||||
export const USD_COMPARE = [/<>|(?<![=\-<>])[<>](?![<>])/, /(?:_usd|Usd)\b["']?\s{0,20}[:=]\s{0,20}-?\d/i];
|
||||
/** Docs and results trees: an unread data file there names no money by its path or a reported cost (GST-961). */
|
||||
export const RESULTS_TREE = /(^|\/)(?:docs?|wiki|results|reports)\//i;
|
||||
/** Any spend_cap spelling and paid-provider call sites (the providers lib/pricing.ts prices). */
|
||||
export const PAID_MONEY_LINE = [
|
||||
/\bspend[_-]?cap\b/i,
|
||||
/\bapi\.(?:anthropic|openai|voyageai)\.com\b/, /\bgenerativelanguage\.googleapis\.com\b/, /\bopenrouter\.ai\b/,
|
||||
/\bnew\s+(?:Anthropic|OpenAI|AnthropicBedrock|AnthropicVertex)\s*\(/, /\.(?:messages|chat\.completions|responses|embeddings)\.create\s*\(/,
|
||||
];
|
||||
/** The MONEY_LINE entries a config file or results script still answers to: a spend cap and a paid-provider call site, never a bare _usd value. */
|
||||
export const PAID_MONEY_LINE = MONEY_LINE.slice(2);
|
||||
export const MONEY_LINE = [...MONEY_USD, ...PAID_MONEY_LINE];
|
||||
/** A config file's price or rate key (input_price_usd, cost_per_1k_usd): money outside a results or receipts tree (PRICE_KEY_EXEMPT), where #3920's path-only rule otherwise keeps config values out (GST-961+962). */
|
||||
export const PRICE_KEY_EXEMPT = /(^|\/)(?:docs?|wiki|results|reports|receipts)\//i;
|
||||
export const PRICE_USD_KEY = /(?:^|[^a-z])(?:price|prices|pricing|rate|rates)_[a-z0-9_]*usd\b|cost_?per|per_?token|per_?1[km]\b/i;
|
||||
/** A _usd or Usd name is money unless every one on the line is a reported cost the line only reads: no cap word, comparison or number set. */
|
||||
export const usdMoney = (line: string) => {
|
||||
const names = MONEY_USD_ALL.flatMap(r => [...line.matchAll(r)].map(m => m[0]));
|
||||
if (!names.length) return false;
|
||||
return names.some(n => tokens(n).some(x => !REPORTED_COST_WORDS.includes(x))) || tokens(line).some(x => MONEY_USD_WORDS.includes(x)) || USD_COMPARE.some(r => r.test(line));
|
||||
};
|
||||
export const AUTH_PATH_TOKENS = ['auth', 'authn', 'authz', 'oauth', 'oauth2', 'jwt', 'session', 'sessions', 'permission', 'permissions', 'role', 'roles'];
|
||||
export const AUTH_PATH_PREFIXES = ['authenticat', 'authoriz'];
|
||||
export const SECRET_PATH = /(^|\/)(secrets?|credentials?|keychain|token[s]?)([._/-]|$)/i;
|
||||
@@ -121,8 +140,8 @@ const namesMoney = (t: readonly string[], words: readonly string[]) => t.some(x
|
||||
/** A hash of the patterns that decide the risk classes, not of this file's bytes; lib/review-depth.ts folds it into DEPTH_RULES. */
|
||||
export const RISK_RULES = createHash('sha256').update(JSON.stringify({
|
||||
RISK_CLASSES, MONEY_PATH_TOKENS, AUTH_PATH_TOKENS, AUTH_PATH_PREFIXES, KEY_NAME, SECURITY_PATH_TOKENS, DATA_PATH_TOKENS, EVAL_PATH_TOKENS,
|
||||
MONEY_LINE_TOKENS, MONEY_PAIRS, SPEND_PAIRS,
|
||||
patterns: Object.entries({ PROSE_PATH: [PROSE_PATH], DOC_SECRET_LINE, CONFIG_DATA_PATH: [CONFIG_DATA_PATH], STATS_SCRIPT: [STATS_SCRIPT], MONEY_LINE, PAID_MONEY_LINE, SECRET_LINE, SECRET_PATH: [SECRET_PATH], SECURITY_LINE, SECURITY_JOINED, DATA_LINE, SVG_PATH: [SVG_PATH], SVG_ACTIVE }).map(([k, v]) => [k, v.map(r => '/' + r.source + '/' + r.flags)]),
|
||||
MONEY_LINE_TOKENS, MONEY_PAIRS, SPEND_PAIRS, REPORTED_COST_WORDS, MONEY_USD_WORDS,
|
||||
patterns: Object.entries({ PROSE_PATH: [PROSE_PATH], DOC_SECRET_LINE, CONFIG_DATA_PATH: [CONFIG_DATA_PATH], STATS_SCRIPT: [STATS_SCRIPT], MONEY_LINE, PAID_MONEY_LINE, USD_COMPARE, RESULTS_TREE: [RESULTS_TREE], PRICE_USD_KEY: [PRICE_USD_KEY], PRICE_KEY_EXEMPT: [PRICE_KEY_EXEMPT], SECRET_LINE, SECRET_PATH: [SECRET_PATH], SECURITY_LINE, SECURITY_JOINED, DATA_LINE, SVG_PATH: [SVG_PATH], SVG_ACTIVE }).map(([k, v]) => [k, v.map(r => '/' + r.source + '/' + r.flags)]),
|
||||
})).digest('hex').slice(0, 16);
|
||||
|
||||
export const tokens = (p: string) => p.replace(/([a-z0-9])([A-Z])/g, '$1 $2').toLowerCase().split(/[^a-z0-9]+/).filter(Boolean);
|
||||
@@ -136,6 +155,8 @@ export interface RiskInput {
|
||||
scopeAuth: boolean;
|
||||
/** Executables (lib/review-collect.ts): never a config file or results script, so their lines are always read. */
|
||||
executables?: readonly string[];
|
||||
/** Code paths no code reads (an unconsumed data file or drawing): under RESULTS_TREE their name and reported costs name no money; a spend cap or paid call in them still does (GST-961). */
|
||||
dataPaths?: readonly string[];
|
||||
/** The base policy's review.risk_paths matcher: a code path it matches is risk class `policy`. */
|
||||
isRiskPath?: (p: string) => boolean;
|
||||
}
|
||||
@@ -154,16 +175,17 @@ export function matchRisk(i: RiskInput): Map<RiskClass, string> {
|
||||
const prose = PROSE_PATH.test(p);
|
||||
if (i.scopeAuth && t.some(x => AUTH_PATH_TOKENS.includes(x) || AUTH_PATH_PREFIXES.some(a => x.startsWith(a)))) hit('auth', 'auth path: ' + p);
|
||||
if (SECRET_PATH.test(p)) hit('secret', 'secret path: ' + p);
|
||||
if (namesMoney(t, MONEY_PATH_TOKENS)) hit('money', 'money path: ' + p);
|
||||
const money = !(i.dataPaths?.includes(p) && RESULTS_TREE.test(p));
|
||||
if (money && namesMoney(t, MONEY_PATH_TOKENS)) hit('money', 'money path: ' + p);
|
||||
if (t.some(x => SECURITY_PATH_TOKENS.includes(x))) hit('security', 'security path: ' + p);
|
||||
if (t.some(x => DATA_PATH_TOKENS.includes(x))) hit('data', 'data path: ' + p);
|
||||
if (!config && t.some(x => EVAL_PATH_TOKENS.includes(x))) hit('eval', 'eval scorer path: ' + p);
|
||||
if (SVG_PATH.test(p) && [i.added?.[p], i.removed?.[p]].some(l => l && svgActive(l.join('\n')))) hit('security', 'active SVG lines: ' + p);
|
||||
for (const line of [...(i.added?.[p] ?? []), ...(i.removed?.[p] ?? [])]) {
|
||||
if (!why.has('secret') && SECRET_LINE.some(r => r.test(line))) hit('secret', 'secret line: ' + p);
|
||||
if (!why.has('money') && pathOnly && PAID_MONEY_LINE.some(r => r.test(line))) hit('money', 'spend cap or paid call: ' + p);
|
||||
if (!why.has('money') && pathOnly && (PAID_MONEY_LINE.some(r => r.test(line)) || (!PRICE_KEY_EXEMPT.test(p) && MONEY_USD.some(r => r.test(line)) && PRICE_USD_KEY.test(line)))) hit('money', 'spend cap, paid call or price key: ' + p);
|
||||
if (pathOnly) continue;
|
||||
if (!why.has('money') && (MONEY_LINE.some(r => r.test(line)) || namesMoney(tokens(line), MONEY_LINE_TOKENS) || (!prose && pairs(tokens(line), SPEND_PAIRS)))) hit('money', 'money line: ' + p);
|
||||
if (!why.has('money') && (PAID_MONEY_LINE.some(r => r.test(line)) || (money && (usdMoney(line) || namesMoney(tokens(line), MONEY_LINE_TOKENS) || (!prose && pairs(tokens(line), SPEND_PAIRS)))))) hit('money', 'money line: ' + p);
|
||||
if (!why.has('security') && SECURITY_LINE.some(r => r.test(line))) hit('security', 'security line: ' + p);
|
||||
if (!why.has('data') && DATA_LINE.some(r => r.test(line))) hit('data', 'data line: ' + p);
|
||||
}
|
||||
|
||||
@@ -20,8 +20,8 @@ import { spawnSync } from 'node:child_process';
|
||||
import { createHash } from 'node:crypto';
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
import { classifyPath } from './review-depth';
|
||||
import { AUTH_PATH_PREFIXES, AUTH_PATH_TOKENS, KEY_NAME, MONEY_LINE, MONEY_LINE_TOKENS, MONEY_PAIRS, MONEY_PATH_TOKENS, PROSE_PATH, SECRET_LINE, SECRET_PATH, SPEND_PAIRS, matchRisk } from './review-risk';
|
||||
import { classifyPath, unconsumedData, type ClassInput } from './review-depth';
|
||||
import { AUTH_PATH_PREFIXES, AUTH_PATH_TOKENS, KEY_NAME, MONEY_LINE, MONEY_LINE_TOKENS, MONEY_PAIRS, MONEY_PATH_TOKENS, MONEY_USD_WORDS, PROSE_PATH, REPORTED_COST_WORDS, RESULTS_TREE, SECRET_LINE, SECRET_PATH, SPEND_PAIRS, USD_COMPARE, matchRisk } from './review-risk';
|
||||
|
||||
export const TARGETED_CLASSES = ['money', 'gate', 'auth', 'secret'] as const;
|
||||
export type TargetedClass = (typeof TARGETED_CLASSES)[number];
|
||||
@@ -31,8 +31,8 @@ export const secretLine = (line: string) => SECRET_LINE.some(r => r.test(line));
|
||||
|
||||
/** A hash of the patterns that decide the classes, not of this file's bytes: a comment edit keeps it, a pattern edit changes it. */
|
||||
export const TARGETED_RULES = createHash('sha256').update(JSON.stringify({
|
||||
TARGETED_CLASSES, MONEY_PATH_TOKENS, MONEY_LINE_TOKENS, MONEY_PAIRS, SPEND_PAIRS, AUTH_PATH_TOKENS, AUTH_PATH_PREFIXES, KEY_NAME,
|
||||
patterns: Object.entries({ MONEY_LINE, SECRET_LINE, SECRET_PATH: [SECRET_PATH], PROSE_PATH: [PROSE_PATH] }).map(([k, v]) => [k, v.map(r => `/${r.source}/${r.flags}`)]),
|
||||
TARGETED_CLASSES, MONEY_PATH_TOKENS, MONEY_LINE_TOKENS, MONEY_PAIRS, SPEND_PAIRS, REPORTED_COST_WORDS, MONEY_USD_WORDS, AUTH_PATH_TOKENS, AUTH_PATH_PREFIXES, KEY_NAME,
|
||||
patterns: Object.entries({ MONEY_LINE, USD_COMPARE, RESULTS_TREE: [RESULTS_TREE], SECRET_LINE, SECRET_PATH: [SECRET_PATH], PROSE_PATH: [PROSE_PATH] }).map(([k, v]) => [k, v.map(r => `/${r.source}/${r.flags}`)]),
|
||||
})).digest('hex').slice(0, 16);
|
||||
|
||||
export interface TargetedInput {
|
||||
@@ -47,6 +47,9 @@ export interface TargetedInput {
|
||||
root?: string | readonly string[];
|
||||
/** Collector failures (a diff git could not list). */
|
||||
errors?: readonly { code: string; input: string }[];
|
||||
/** The collector's consumer scan and drawings (ClassInput): a data file no code reads names no money. */
|
||||
consumed?: ClassInput['consumed'];
|
||||
drawings?: ClassInput['drawings'];
|
||||
}
|
||||
export interface TargetedResult { classes: TargetedClass[]; reasons: string[]; errors: string[]; rules: string }
|
||||
|
||||
@@ -61,7 +64,7 @@ export function targetedClasses(i: TargetedInput): TargetedResult {
|
||||
if (errors.length) return { classes: [...TARGETED_CLASSES], reasons: errors.map(e => `fail-closed: ${e}`), errors, rules: TARGETED_RULES };
|
||||
const why = new Map<TargetedClass, string>();
|
||||
if ((i.gateTagged ?? 0) > 0) why.set('gate', `gate scan tagged ${i.gateTagged} hunk(s)`);
|
||||
for (const [c, reason] of matchRisk({ codePaths: code, added: i.added, removed: i.removed, scopeAuth: s!.SCOPE_AUTH === 'true' })) if ((TARGETED_CLASSES as readonly string[]).includes(c)) why.set(c as TargetedClass, reason);
|
||||
for (const [c, reason] of matchRisk({ codePaths: code, dataPaths: code.filter(p => unconsumedData(i, p)), added: i.added, removed: i.removed, scopeAuth: s!.SCOPE_AUTH === 'true' })) if ((TARGETED_CLASSES as readonly string[]).includes(c)) why.set(c as TargetedClass, reason);
|
||||
const classes = TARGETED_CLASSES.filter(c => why.has(c));
|
||||
return { classes, reasons: classes.map(c => why.get(c)!), errors: [], rules: TARGETED_RULES };
|
||||
}
|
||||
|
||||
@@ -1 +1 @@
|
||||
{"sha":"6688963750c14f03cdbb249dcaa27ebd9e9c2c62170f4e142ec8ded28af010a1","replay":"GSTA-258-L7"}
|
||||
{"sha":"d860aa6e2881c395ee2a64426518f42d79c03e1e67752574d9e39a2019200d9c","replay":"GSTA-258-L7+GST-961+962"}
|
||||
+1
-1
@@ -1 +1 @@
|
||||
{"sha":"1d03d5cd8f0cae5f01af85502004849c9fa2545009957f2c4c44a1359d9d0fc0","replay":"GSTA-258-L7"}
|
||||
{"sha":"d4a16e78c76f4346ef3a4ce792c47c6ef5b4c2574f2474f50e7046f89cb2bc69","replay":"GSTA-258-L7+GST-961+962"}
|
||||
@@ -0,0 +1 @@
|
||||
{"replay":"GST-961+962","justification":"GST-961+962 verdict no-loss replay: a data file no code reads (the depth's consumer scan) names no money by its path or rows, under docs/, wiki/, results/ or reports/ (a spend cap or paid call there still does), and a _usd or Usd name is not money only when every such name is a reported cost on a line with no cap, budget, price, charge, reservation or estimate word, no <, > or <> comparison and no number set; over the last 100 first-parent main commits to bf2bcf14b, main's code against this branch, gate scan fixed at 0, no commit changes depth, class, risk classes or targeted classes (~/.capy/work/gst898/replay961.ts on GSTA-249's machine); the 21 changed lines that no longer read as money lines are cost sums and reports (scripts/lib/paid-report.ts, two eval-plan per-item means), tests and docs; lib/review-depth.ts only exports unconsumedData; reviewer text is unchanged"}
|
||||
@@ -0,0 +1 @@
|
||||
{"replay":"GSTA-258-C+GST-961+962","justification":"rule 25 trial merge of #3920 and GST-961+962: lib/review-risk.ts and lib/review-depth.ts carry both changes; the GST-961+962 no-loss replay and the GSTA-258-C checks both hold on the merged files; PAID_MONEY_LINE is GST-961+962's list (the spend cap and provider calls), which GSTA-258-C's slice(2) named; in a config file a _usd price or rate key (PRICE_USD_KEY) is money outside a results tree (GSTA-36 ruling, option 1), so HARN-1's results-tree list_price_usd stays path-only and an unread data/rates.json input_price_usd stays money; rerun the no-loss replay on any change"}
|
||||
@@ -0,0 +1 @@
|
||||
{"replay":"GSTA-258-L7+GST-961+962","justification":"rule 19 main merge of 08cc447 (GSTA-258 L7, #3972) into GST-961+962 (#3942): lib/review-risk.ts keeps both RiskInput fields (L7's executables and GST-961's dataPaths) and matchRisk is the clean textual union; lib/review-depth.ts exports unconsumedData with L7's executables guard, so an executable is never a dataPath and keeps every money trigger (the only interaction, and it only adds risk); this repin relies on the GST-961+962 no-loss replay and the GSTA-258-L7 planner no-loss check, which were not rerun on the merged files; the merge adds no rule of its own, and the focused tests of both (430 in 9 review-* files, GSTA-268) pass on the merged tree; reviewer text is unchanged; rerun the no-loss replay on any change"}
|
||||
@@ -5,7 +5,7 @@
|
||||
*/
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { SVG_REFUSED, svgDrawing } from '../lib/review-depth';
|
||||
import { DATA_LINE, MONEY_LINE, SECRET_LINE, SECRET_PATH, SECURITY_JOINED, SECURITY_LINE, matchRisk } from '../lib/review-risk';
|
||||
import { DATA_LINE, MONEY_LINE, SECRET_LINE, SECRET_PATH, SECURITY_JOINED, SECURITY_LINE, USD_COMPARE, matchRisk, usdMoney } from '../lib/review-risk';
|
||||
|
||||
const L = 100_000;
|
||||
const ms = (f: () => unknown) => { const t = performance.now(); f(); return performance.now() - t; };
|
||||
@@ -27,7 +27,7 @@ describe('linear time on hostile input', () => {
|
||||
run('', 'a_'), run('', 'aU'), run('a', 'A'), run('spend', '_'), run('new Anthropic', ' '), run('.messages.create', ' '),
|
||||
run('$', 'A_'), run('${', 'A_'), run('', '$A_'), run('process.env.', 'A_'), run('process.env[', ' '), run('os.environ[', ' '), run('os.getenv(', ' '), run('ENV[', ' '),
|
||||
];
|
||||
test.each(Object.entries({ SECURITY_LINE, SECURITY_JOINED, DATA_LINE, SECRET_LINE, MONEY_LINE, SECRET_PATH: [SECRET_PATH] }))('every %s pattern finishes under 100 ms on each 100k-character hostile line', (_n, patterns) => {
|
||||
test.each(Object.entries({ SECURITY_LINE, SECURITY_JOINED, DATA_LINE, SECRET_LINE, MONEY_LINE, USD_COMPARE, SECRET_PATH: [SECRET_PATH] }))('every %s pattern finishes under 100 ms on each 100k-character hostile line', (_n, patterns) => {
|
||||
const slow = patterns.flatMap(r => hostile.map(([label, s]) => [r.source.slice(0, 50), label, Math.round(ms(() => r.test(s)))] as const)).filter(([, , t]) => t >= 100);
|
||||
expect(slow).toEqual([]);
|
||||
}, 120_000);
|
||||
@@ -37,6 +37,11 @@ describe('linear time on hostile input', () => {
|
||||
expect(ms(() => { hit = security(line); })).toBeLessThan(250);
|
||||
expect(hit).toBe(false);
|
||||
}, 60_000);
|
||||
test('usdMoney (GST-962) returns in under 250 ms on each 100k-character line of _usd names, comparisons and member chains', () => {
|
||||
const lines = [run('', 'x_usd '), run('', 'aUsd '), run('', '> a.x_usd '), run('', '=> t.cost_usd '), run('>', ' '), run('> ', 'a.', 'x_usd'), run('', 'a_', 'usd'), run('', '<'), run('', 'cost_usd>'), run('', 'cost_usd: '), run('x_usd', ' '), run('x_usd:', ' ')];
|
||||
const slow = lines.map(([label, s]) => [label, Math.round(ms(() => usdMoney(s)))] as const).filter(([, t]) => t >= 250);
|
||||
expect(slow).toEqual([]);
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
describe('the rm rule matches what R51 matched', () => {
|
||||
|
||||
@@ -421,3 +421,81 @@ describe('the receipt record and the queue line', () => {
|
||||
expect(summarizeReviewLoop([{ skill: 'review', via: 'ship', timestamp: '2026-10-30T10:00:00Z', dispatches: [{ role: 'targeted', prompt_bytes: 40 }] }], '2026-10-30T09:00:00Z')).toMatchObject({ dispatches: { targeted: 1 }, est_input_tokens: 10 });
|
||||
});
|
||||
});
|
||||
|
||||
describe('incidental money signals (GST-961, GST-962)', () => {
|
||||
// Value: protects=R51's one reviewer on docs and data PRs and on code that only reports a cost; fails_when=a data file no code
|
||||
// reads gets a money reader because its name says cost (gstack-evals#138), or reading a cost_usd field raises money
|
||||
// (gstack-evals#139), or a narrowed rule stops firing on a cap, a budget, a spend check, a billing path or a provider call;
|
||||
// seam=lib/review-risk.ts matchRisk through targetedClasses and gstack-review-plan
|
||||
test('a data file no code reads never names money, by its path or its rows; every other class still reads it', () => {
|
||||
const tsv = 'wiki/results/cost-gstack-full.tsv';
|
||||
expect(classesOf(input([tsv], { consumed: [] }))).toEqual([]);
|
||||
expect(classesOf(input([tsv], { consumed: [], added: { [tsv]: ['arm\tcost_usd\tcredits', 'A\t1.20\t300'] } }))).toEqual([]);
|
||||
expect(classesOf(input([tsv], { consumed: [tsv] }))).toEqual(['money']);
|
||||
expect(classesOf(input([tsv]))).toEqual(['money']);
|
||||
expect(classesOf(input([tsv], { consumed: 'all' }))).toEqual(['money']);
|
||||
expect(classesOf(input(['wiki/credentials.json'], { consumed: [] }))).toEqual(['secret']);
|
||||
expect(classesOf(input(['wiki/keys.json'], { consumed: [], added: { 'wiki/keys.json': ['{"k": "$OPENAI_API_KEY"}'] } }))).toEqual(['secret']);
|
||||
expect(classesOf(input(['src/billing/invoice.ts'], { consumed: [] }))).toEqual(['money']);
|
||||
expect(classesOf(input(['prices/us-east.json'], { consumed: [] }))).toEqual(['money']);
|
||||
expect(classesOf(input(['data/rates.json'], { consumed: [], added: { 'data/rates.json': ['{"input_price_usd": 3}'] } }))).toEqual(['money']);
|
||||
expect(classesOf(input(['config/models.yaml'], { consumed: 'all', added: { 'config/models.yaml': [' cost_per_1k_usd: 0.3'] } }))).toEqual(['money']);
|
||||
expect(classesOf(input(['evals/config.json'], { consumed: 'all', added: { 'evals/config.json': [' "output_price_usd": 15,'] } }))).toEqual(['money']);
|
||||
expect(classesOf(input(['docs/b/results/run.json'], { consumed: 'all', added: { 'docs/b/results/run.json': [' "list_price_usd": 1.2485,'] } }))).toEqual([]);
|
||||
expect(classesOf(input(['evals/config.json'], { consumed: 'all', added: { 'evals/config.json': [' "lease_usd": 32.9,'] } }))).toEqual([]);
|
||||
expect(classesOf(input([tsv], { consumed: [], added: { [tsv]: ['spend_cap\t500'] } }))).toEqual(['money']);
|
||||
});
|
||||
|
||||
test('a cost_usd field read or reported names no money; a cap, budget, limit, spend or comparison still does', () => {
|
||||
const at = (line: string) => classesOf(input(['analysis/summarize.ts'], { added: { 'analysis/summarize.ts': [line] } }));
|
||||
for (const line of [
|
||||
'const total = trials.reduce((s, t) => s + t.cost_usd, 0);',
|
||||
'console.log(`cost: ${row.cost_usd.toFixed(2)}`);',
|
||||
'rows.push({ arm, cost_usd: r.cost_usd, tokens });',
|
||||
'const perTrial = totalCostUsd / trials.length;',
|
||||
'return { costUsd: sum(rows.map(r => r.costUsd)) };',
|
||||
]) expect(at(line), line).toEqual([]);
|
||||
for (const line of [
|
||||
'const max_cost_usd = 5;',
|
||||
'if (total_cost_usd > budget) throw new Error("over budget");',
|
||||
'if (budget <= run.cost_usd) stop();',
|
||||
'const remainingBudgetUsd = cap - spent;',
|
||||
'const spendLimitUsd = 25;',
|
||||
'while (spent_usd < cap) step();',
|
||||
'const spend_cap_usd = 5;',
|
||||
'const MAX_SPEND_USD = 10;',
|
||||
' if (spent > capUsd) throw new Error("over cap");',
|
||||
'const client = new Anthropic();',
|
||||
'await meter.charge(user, amount);',
|
||||
'if (limit < run?.cost_usd) stop();',
|
||||
'if ((run.cost_usd ?? 0) > limit) stop();',
|
||||
'if (limit <= (row.cost_usd ?? 0)) stop();',
|
||||
'const capped = Math.min(cost_usd, cap);',
|
||||
'if (Number(row.cost_usd) > budget) stop();',
|
||||
'const intent = await stripe.paymentIntents.create({ amount: amount_usd * 100 });',
|
||||
'await refund(user, refund_usd);',
|
||||
'const cost_usd = input_tokens * PRICE_PER_MTOK;',
|
||||
'const check = { attempt, estimate_usd: 0 };',
|
||||
'rows.push({ arm, cost_usd: 0 });',
|
||||
' return { ...r, reservation_usd: estimateCostUsd({ input: Math.ceil(r.in_tokens / r.calls), output: r.max_out_tokens }, r.model) };',
|
||||
' const maxEstimate = Math.max(0, ...record.attempts.map(a => a.estimate_usd));',
|
||||
' const worst = summary.worst_case_usd + workers * maxEstimate;',
|
||||
' reserve(file, attempt, a.estimate_usd, { label });',
|
||||
' a.estimate_usd = usd;',
|
||||
' ship_measure_ask_per_trial_usd) echo "200" ;;',
|
||||
" trial_reservation_usd: v => (typeof v === 'number' && v > 0 && v <= 100000 ? null : 'x'),",
|
||||
'if ($cost_usd <> 0) return;',
|
||||
'SELECT id FROM runs WHERE cost_usd <> 0;',
|
||||
]) expect(at(line), line).toEqual(['money']);
|
||||
});
|
||||
|
||||
test('gstack-review-plan on #138\'s shape: a docs PR with a cost-named data file dispatches no targeted reader', () => {
|
||||
const docs = planRepo({ head: { 'wiki/results/cost-gstack-full.tsv': 'arm\tcost_usd\nA\t1.20\n', 'wiki/results/README.md': '# Results\nArm A cost 1.20.\n' } }).run([]);
|
||||
expect(docs.status, docs.stderr).toBe(0);
|
||||
expect(field(docs.stdout, 'TARGETED:', 'classes')).toBe('none');
|
||||
expect(docs.stdout).not.toContain('DISPATCH: targeted');
|
||||
const read = planRepo({ head: { 'wiki/results/cost-gstack-full.tsv': 'arm\tcost_usd\nA\t1.20\n', 'src/load.ts': 'export const f = "wiki/results/cost-gstack-full.tsv";\n' } }).run([]);
|
||||
expect(read.status, read.stderr).toBe(0);
|
||||
expect(field(read.stdout, 'TARGETED:', 'classes')).toBe('money');
|
||||
}, 60_000);
|
||||
});
|
||||
Reference in new issue
Block a user