mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-01 06:24:41 +01:00
* docs: Oct 7 follow-up wave plan (pre-review) * docs: Oct 7 follow-up wave plan, reviewed through /autoplan * docs: Oct 7 wave plan, record final gate decisions (UC1 accepted, UC2 rejected) * docs: credit #1822 to @Mike-E-Log and #1743 to @habassa5 (v1.64.0.0) The v1.64.0.0 CHANGELOG entry credited the extension token fix (#1822) and the polyfill exited promise (#1743) to the wrong handle, and the extension-sender-auth test header repeated the first slip. Commit history is not rewritten. Fixes #3054. Co-authored-by: Mike Ilog <48659643+Mike-E-Log@users.noreply.github.com> Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(context-restore): offer newer checkpoints from nested repos and task worktrees /context-restore could present an older checkpoint as the latest with no error in two layouts: 1. A session started in a non-git parent saved from a nested repo, so the checkpoint went to the nested repo's bucket. Restore in the parent now lists checkpoints of other buckets whose project_root lies below the cwd (BELOW_CWD; the 200 newest are read, and from $HOME only each project's newest). /context-save also leaves a JSON pointer (checkpoint path, bucket, project_root) in the starting directory's bucket when $CLAUDE_PROJECT_DIR is set and files elsewhere; restore lists those as POINTER, in or out of git. 2. A plan checkpoint on the main tree's branch shadowed a newer checkpoint saved in a ticket worktree. When another branch holds a newer checkpoint for the same task (a worktree below the main tree, read from the new worktree: stamp; equal normalized titles; or a shared #123 / ABC-123 token in titles or branch names), restore prints NEWER_TASK and the skill shows both and proposes the newer one. Generic titles (wip, checkpoint, save, notes, todo) never match on title alone, and the worktree rule applies only when restoring from the main tree, so sibling worktrees still never shadow each other. The branch-first ordering itself is unchanged. Refs #3065. Co-authored-by: Thomas Gernhardt <t.gernhardt@softwareprojekt.net> Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(redact-prepush): list each MEDIUM finding with rule, file line and fix The pre-push hook printed only a MEDIUM count, which nobody could review without re-running the engine by hand. It now prints one line per finding: rule id, file and file line, and a fix, never the matched value. Line numbers come from the hunk headers (combined --cc headers included), so a finding in a later hunk or past a scan-chunk boundary points at the real file line. Findings from the per-commit catch-up path name the commit whose file line it is, so content removed before the pushed tip is still locatable. Plan items: CEO-7 (summary half), DX-9 (fix lines), ENG-8 (line mapping). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(redact-prepush): stop flagging your own email and already-public authors (#3060) The hook never passed the engine's selfEmail / repoPublicEmails, so every push that added the pusher's address, or a commit author's, raised a MEDIUM pii.email warning for an address the destination already shows. The hook now reads `git config user.email` and one bounded `git log` of author and committer addresses, raw and mailmapped (%ae %aE %ce %cE): the history of every pushed tip (those commits become public with the push), each remote tip being replaced, and, for a new branch or an unknown remote tip pushed to a configured remote, that remote's own tracking refs. A URL or unknown destination never borrows tracking refs, and another remote's history never counts. The read stops at 50,000 commits (the newest authors are used) and 5 seconds. An error or the timeout passes no history and, when an email finding is listed, prints one line saying suppression was limited for this push. The two inputs fail independently. HIGH findings and every other rule are untouched. Controls: a stranger's address, an address only another remote knows, a URL push next to a matching remote, and a first push of a repo with no history all still report. Plan items: A3, CEO-7, CEO-23 (A3 half), DX-9 (partial history), ENG-8 (history scope, mailmap). Co-authored-by: akmandhania <143462686+akmandhania@users.noreply.github.com> Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(redact-prepush): per-address allowlist for pii.email (gstack.redact.allowEmail) `git config --add gstack.redact.allowEmail <address>` allows that address for MEDIUM pii.email findings only, so a team alias or support address that has to be in the repo stops warning without reaching for --no-verify. It never affects HIGH findings or any other rule, and the pii.email fix line names it. Controls: a different address still reports, and a HIGH secret next to an allowed address still blocks. Plan item: DX-17. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(troubleshooting): pre-push pii.email findings, exemptions and allowlist Plan item: DX-12 (pre-push touchpoint). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(redact-prepush): assemble fixture addresses at runtime The email fixtures were literals, so this branch's own push listed 47 pii.email findings in its test file. Build them from a helper instead, the way the neighbouring tests build credential and IP fixtures. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(browse): offer the user's signed-in tab, name silent aside repl failures Rules (rendered into every browsing skill, kept short): - Rule 1: before the first openTab, offer the user's tabs on the target origin (title and origin only; the rest of listBrowserTabs() stays private) and attach only after the user confirms one. - Rule 4: a second sign-in wall after a confirmed sign-in means the session is bound to the tab or URL; offer that tab instead of another sign-in. - Rule 6 no longer suggests an early return; rule 7 says a fast [ok without GSTACK_STEP_OK is an abort. Cookbook (browse, devex-review): a "Silent failures" list (evaluate returns only serializable values, no top-level return, locator(sel).press(key), an empty DOM read is about the selector until the screenshot and the request's response say otherwise, read toggle state before clicking) and a signed-in tab recipe that prints only target-origin tabs and never closes the user's tab. Parity caps for design-consultation, land-and-deploy, qa and the browse skeleton are raised to the measured sizes, with notes. Refs #3063, #3064. Co-authored-by: Thomas Gernhardt <t.gernhardt@softwareprojekt.net> Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(codex): run gstack-codex-probe as a command, one subcommand per probe Generated skills now execute bin/gstack-codex-probe (select-model, check-auth, show-sandbox, check-sandbox, probe-model, check-version, show-first-use-notice, run-with-timeout, log-event, log-hang, help) and capture its KEY: value lines, so no helper state crosses into the calling shell and the preflight works from bash, zsh or sh. select-model keeps --model > GSTACK_CODEX_MODEL > config.toml precedence; run-with-timeout supervises with the existing gtimeout/timeout/ watchdog chain and returns 124 on a deadline; an unknown subcommand exits 64. Sourcing keeps working for skills rendered before the upgrade and prints one deprecation line naming /gstack-upgrade; it and the zsh self-locate block are removed in a release on or after 2026-10-21 (DX-18). Parity caps raised with measured notes for the longer capture lines. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(gstack-post): one argv-only, redaction-scanned write path for PR and issue text bin/gstack-post posts a PR/MR or issue comment, a review-comment reply, a PR title or body, a new PR/MR or a new issue from agent-written files. Every value reaches gh/glab as its own argv element in --flag=<value> form or on stdin, never through a shell string, and a target is a number or a URL of this repository on the same host. The helper owns the redaction scan: it reads each file once, scans those exact bytes (visibility from config, then gh/glab; the pusher's own email suppressed) and sends the same bytes. HIGH is refused with exit 1 and no confirmation can post it. MEDIUM exits 2 with RULE:/LINE:/PART: lines and a TOKEN: digest of the bytes, destination, operation and findings; only --confirm <token> for those exact bytes posts, so any edit needs a new confirmation. gh/glab failures exit 3, usage errors 64. It also carries the projectCards REST fallback and the title read-back that /ship did by hand. The logic lives in lib/gstack-post.ts behind an injectable runner, so its tests run in the Windows free lane; the stub-gh executable test is POSIX. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(autoplan): guard verifies dispatches on Claude Code 2.1.29x (schema strip, unflushed call, named causes) Cause A: Claude Code journals the model's raw Agent input but hands PreToolUse the schema-parsed input; with the fork-subagent gate on (the interactive default in 2.1.292) run_in_background is dropped, so the journal record never deep-equaled the payload. nativeToolInput now drops only that documented strip for Agent. Agent input is bound on both paths: keys limited to prompt, description, subagent_type (general-purpose) and run_in_background; a model override or any other key is denied. Cause B: when a ready journal lacks the current call, the guard evaluates the payload placed after the last journaled record instead of polling for a record that may arrive only with the tool result. A phase transition counts a report only when a later journaled record follows it (and, on the journal path, it is not in the current call's message). One batch rule covers both paths; a background reviewer keeps the invocation armed until its completion notice. Every outcome now has a stable reason code, disposition and anchor (autoplan/bin/guard-reasons.ts). UC1: Claude-Code-caused states (unrecognized shape, journal lag, rewritten journal, oversized record, unchecked-version schema strip) allow with a visible warning; integrity failures stay denials. No denial advises a bare retry. Every guarded decision appends one content-free line to analytics/autoplan-guard.jsonl, trimmed to 1,000 lines past 256 KiB. Co-authored-by: Jared <182125291+yolo-jared@users.noreply.github.com> Co-authored-by: crblabs <293123359+crblabs@users.noreply.github.com> Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(autoplan): publish each phase report with a no-op call; foreground requests are conditional Phase-close step 6 sends the report in a message whose only tool call is the Bash no-op ': autoplan-published <phase>', and step 7 makes the next guarded call in a later message, so the guard sees the report flushed before the next phase entry regardless of Claude Code's journal timing. Every site that asked for run_in_background: false now asks for it only when the Agent schema offers it and says how to recover when a dispatch runs in the background anyway (wait for its completion notice). Parity caps rise by the measured bytes. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(ship,spec,review): PR and issue text goes out only through gstack-post /ship: Step 18 writes the title into an agent-written file. An existing PR's title is piped from gh/glab through gstack-pr-title-rewrite.sh --stdin (new), so neither the agent nor a shell string carries it; a new title is written with the file tool and its version prefix checked in the file. Step 19 composes the body as before and publishes with gstack-post pr-create, or pr-body then pr-title, which scan the exact bytes they send. The separate title and body scans, the hand-rolled REST fallback and the title self-check are gone (the helper does all three). Title preparation moved into the pr-body section, so the skeleton shrinks. /spec files its issue with gstack-post issue-create and drops the pre-issue re-scan; Greptile replies use gstack-post reply and pr-comment. scripts/resolvers/redact-doc.ts loses its PR and issue sinks, and a test keeps any template from rendering one again. Tests that pinned the old scan-then-gh wiring now pin the gstack-post wiring; the publication tests run the real compose and publish blocks against a stub gh with the real helper (HIGH in title or body refused, MEDIUM token flow). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * ci: pin Claude Code 2.1.292 in the CI image 2.1.292 is the npm latest users run. Its interactive default turns on the fork-subagent gate, which strips run_in_background from the Agent schema and backgrounds reviewers; CI on 2.1.284 never saw the /autoplan guard break. The image tag is a content hash, so every eval workflow picks up the new image without a workflow edit. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(brain-sync): one drain lock for every writer to the artifacts repo gstack-artifacts-init, gstack-brain-restore, gstack-gbrain-source-wireup and the skill-start merge ran git in $GSTACK_HOME without the drain's lock, so the drain could never tell an abandoned .git/index.lock from a live one. They now take the same mkdir lock (bin/gstack-brain-lock.sh): the drain and the skill-start merge skip when it is held, user-run writers wait for it (GSTACK_BRAIN_LOCK_WAIT, default 120 s), and a child of the holder (the drain's daily worktree advance, restore's wireup) inherits it by token. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(brain-sync): hold back only the flagged file, keep coupled artifacts whole One file with secret-shaped text used to block every push, silently. The drain now attributes scanner hits to files in one pass over the staged diff (split on its diff --git headers), holds back each flagged file with the rest of its coupled group, unstages it, and requires the remaining staged diff to re-scan clean; anything short of that falls back to a full unstage and status blocked. Held files stay queued and are re-scanned every drain. - Status file gains held_count, drainable, last_drain_at, last_push_at and held[] (path, scanner rule, dependents, both fixes); new status held. - Coupled groups come from an explicit map (decisions.jsonl and decisions.active.json): a queued member stages the whole group, a group stages only when its snapshot matches its log in the index, a skipped member stops the whole group. - --unskip-file undoes --skip-file and queues the path again. - An index lock older than 10 minutes is removed under the drain lock; a younger one, a failed git add, a failed commit (hooks included) keep the queue and report status error. A verified empty staged diff is the only "nothing to commit". git add runs once, batched. - The scanner reads bytes, so undecodable content no longer crashes it into a false clean; matched text never reaches the status file. - An empty-queue run with unpushed sync commits stays push_failed. - Privacy-held duplicate records collapse to one per path; --status counts a spool too deep for a shell glob. - --attention prints skill start's fixed attention lines from the sync status table in the usage header. Co-authored-by: v639dragoon <matthewsbrickley@gmail.com> Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(skill-start): ARTIFACTS_SYNC attention lines for a stuck sync Skill start now prints one fixed attention line per sync-status-table row that applies (held, blocked, push_failed, error, unknown, stale-push, stale-drain), each naming gstack-brain-sync by absolute path. Values from the status file are clamped (status code to the table, counts to digits, timestamps to their shape), so the status message, file paths and scanner output never reach the line. The preamble tells the model to surface the line in one sentence and continue. Skill-start tests are generated from the table in bin/gstack-brain-sync, and docs/gbrain-sync-errors.md carries the same table. Co-authored-by: v639dragoon <matthewsbrickley@gmail.com> Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(outside-review): severity words in label position count as findings The Oct 7 /autoplan run's Codex CEO and DX reviews labeled seven and eight findings High/Medium without [Pn] tags, and the classifier printed VERDICT: clean / FINDINGS: none for both. lib/outside-review-result.ts now reads severity words in label position (Severity: High, a leading High: / High — / [High], **High**, a table cell): critical/high block like P0/P1, medium/low are advisory. Words in prose (high-level, low-risk, no critical findings) never count. A review or structured response with neither tags, labels nor an explicit no-findings statement is unverified, never clean (DX-11, ENG-10). Design-direction proposals move to a new proposal gate so they keep needing only their Recommendation line, and review prompts ask for a severity per finding. Fixtures reproduce the CEO/DX label shapes, the eng [P1] output, a table and prose negative controls; the stored Codex panel keeps its verdicts. docs/troubleshooting.md explains how the verdict is read. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat: gstack-doctor readiness report and setup --status Codex row (B5) bin/gstack-doctor prints one row per check without starting a skill: install root (the registry rows behind ./setup --status), state root, Bun against the floor, hook parse check, Codex mode and self-locate via gstack-codex-probe subcommands, the cached model probe with its age (--live runs the paid probe-model), artifacts-sync health from the status file with the status code clamped, the browse bundle, Claude Code version, the largest session journal and the last five schema-1 guard reason codes. Rows are ok, warn, not configured or fail; only fail exits non-zero; fix lines name helpers by absolute install-root path. ./setup --status gains a read-only Codex row (bin/gstack-codex-status.sh, shared with the doctor) and ends with the doctor's absolute path. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: bug-report template asks for doctor output; README troubleshooting entry; doctor test in the Windows lane The template gives the Claude Code doctor path and a lookup for other hosts and names no bare gstack- command (asserted in test/gstack-doctor.test.ts). The doctor test runs every script through explicit bash argv, so it joins the curated Windows free subset. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(troubleshooting): readiness check section points at gstack-doctor Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(question-tuning,plan-tune,gbrain): free text reaches helpers only through agent-written files Question tuning no longer pipes the question summary through printf: a registered id needs no summary, and for an unregistered id the agent writes the summary to .gstack/tmp/qt.txt and passes --summary-file (new). The helper resolves a relative path against the git top level, removes scratch files under .gstack/tmp, and asks normally when the file is missing. /plan-tune tune events carry the user's own words again: they go into a file the agent writes and reach gstack-question-preference --write through --free-text-file (new), still behind the injection filter and the user-origin gate. docs/gbrain-write-surfaces.md drops its heredoc templates for the agent-written page file on stdin, matching the per-skill save block, and the generic save block stops showing a quoted --content argument. The tuning prose was tightened so every preamble stays within its size cap. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(evals): daily autoplan-schema-canary on the latest Claude Code A periodic rule case installs the latest published Claude Code with install scripts off, verifies registry signatures, and runs one headless session with the fork-subagent gate on whose PreToolUse hook records an Agent and a Read payload. Each payload must match its journal record under the guard's own comparison, keep the journal's key set apart from the documented run_in_background strip, and stay inside the Agent allowlist; a background Agent's completion notice must keep the shape the guard reads. evals-periodic.yml runs it daily on its own schedule and concurrency group (read-only token); the weekly census also includes it. A free test proves the comparison fails on an injected strip. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(doctor): read the artifacts sync held count from held_count, the field the drain writes * fix(artifacts sync): the init and restore pre-commit hooks catch Bearer/Basic/Token-prefixed JSON auth headers like the drain does * feat(greptile-early): detect Greptile, ask once on public repos, open the early draft, wait for the review bin/gstack-greptile-early backs /ship's early Greptile PR: - detect: a root .greptile/ folder (which takes precedence), greptile.json, or a Greptile comment on the repo in the last 90 days; triggerOnDrafts from the effective root config; repo visibility; public-repo consent. It prints the signal, that /ship will push and open a draft now, the 10-minute wait cap and the absolute gstack-config command that turns it off. - consent yes|no: the public-repo answer, remembered per repo in the project's state. - open: opens the draft through gstack-post with a body saying /ship opened it early, and posts one @greptileai comment through gstack-post when the config does not set triggerOnDrafts: true. An open PR is left alone. - wait: polls every 30 s for a completed Greptile check run, a Greptile review of the head commit or a summary naming it; each call returns within about 90 s and reports timeout 10 minutes after the PR opened. gstack-config gains ship_greptile_early (default true, invalid values rejected): on automatically whenever the repo uses Greptile, off per user. The logic is in lib/greptile-early.ts behind an injectable runner, so its fixture-repo tests run in the Windows lane; an end-to-end test drives the real helper and gstack-post against a stub gh. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(ship): early Greptile PR at Step 6.5, bounded wait at Step 10 (#3020) When the repo uses Greptile, /ship pushes and opens a draft PR right after its free tests pass (new lazy section greptile-early, Step 6.5), so Greptile reviews while Steps 7-9 run. A public repo asks once, remembered per repo; declining keeps today's order. Step 10 waits for that review with gstack-greptile-early wait and takes the existing unavailable route on a timeout, never a claim of zero comments. Step 18 replaces the provisional title, Step 19 marks the draft ready unless a draft was requested, and a /ship that stops after Step 6.5 leaves the draft open with one comment saying why. ship's union cap rises to 1.5275 with the measured note (B4 +836 bytes, B2 +4,732 bytes); the step-numbering guard allows 6.5; ship goldens are refreshed for B4 and B2. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(autoplan): the publication no-op is 'true autoplan-published <phase>' Measured on Claude Code 2.1.292 in a default-settings interactive session: a bare ':' Bash command asks for approval ('This command requires approval'), and after a background reviewer's completion notice starts a new turn the skill's allowed tools no longer cover it, so the phase-close no-op prompted the user. 'true' with arguments runs without approval and prints nothing. The guard only needs a later journaled record, so the command changes everywhere it is named. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(evals): live autoplan-guard-pty case on the pinned Claude Code A gate-tier rule case (one trial, haiku) drives an interactive PTY session with default settings through a project skill that carries /autoplan's own hook block: init, Phase 1 entry, a CEO reviewer dispatch that sends run_in_background, a prepared close packet, the report published with the no-op, and the Phase 2 entry in a later message. It reads outcomes from the guard decision log (three verified allows) and fails if the no-op raises a permission card; other cards are approved and printed. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(autoplan): bounded, append-tolerant owned-journal read (A2) The publication guard no longer reads the whole parent journal or refuses it at 32 MiB. lib/claude-owned-journal.ts streams the bytes present at open to the last complete line (pass one), keeps a content-free index per record and decides ownership, ancestry and causal order on it with the same checks and reason codes as before. Pass two reloads in full only what the evaluation reads (autoplan/bin/guard-journal.ts): every literal init and its result, the current call, and inside the invocation window (from the latest human turn at or before the bound chain's first init) guarded and close-packet Read results, structured Bash results, error results, assistant text after the first close Read and active-plan mutations, plus the earlier Read a dedup reply cites even before the window (ENG-4). - lib/claude-journal-records.ts: per-record event extraction, record metadata and the causal-order checks, shared by readPlanCountTranscript and the bounded reader. The batch reader keeps its whole-file read and cap. - Appends after open never fail a read; same device and inode and no shrink are required (identity). A second read in one hook invocation must hash bytes [0, first size) the same, or it is `rewritten` (UC1 unverified); the unrecognized-shape double read uses that prefix hash. `changing` is removed (REMOVED_CODES; its anchor stays). - OWNED_RECORD_MAX_BYTES (32 MiB): only one record over it is too_large. OWNED_RETAINED_MAX_BYTES (32 MiB): over it is the new oversized_invocation fallback denial. GSTACK_TRANSCRIPT_TEST_MAX_BYTES and GSTACK_TRANSCRIPT_TEST_RETAINED_BYTES can only lower them. - ENG-5: evaluation groups uses and results by tool id once (no per-tool rescans). CEO-25 / ENG-5 spike (120 MiB synthetic, 31,242 records, 4 vCPU Linux, Bun 1.4.2): a full streaming rescan (parse every line, small index, sha256) took 254-326 ms at 147-161 MiB peak RSS; a pass-one index cache under the state root (load cached index, verify a 64 KiB tail hash, parse the tail) took 26-29 ms at 47 MiB. The rescan meets the 1 s budget, so the cache is not adopted: it would add trusted persistent state and invalidation for no budget need. A backward tail read cannot prove competing_root or cycle, so it was not a candidate. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(autoplan): journal-read benchmark and its dispatch-only workflow (CEO-2, CEO-6) scripts/measure-journal-read.ts builds two synthetic 120 MiB parent journals around a real /autoplan invocation (mixed: ~28k records, most bytes inside the invocation; large-invocation: ~89k records with a current invocation of many small records), keeps appending while the guard reads, and measures one hook call in a fresh process (wall ms, peak RSS). The workflow runs it on ubuntu-latest and macos-latest on workflow_dispatch only. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(evals): autoplan-long-session, a resumed >100 MiB session enters Phase 1 CEO-15 with ENG-6's observer: a real session is compacted, its journal padded past 100 MiB with owned records before the compact boundary, and the resumed interactive session runs the scripted phase boundary; the guard decision log must show a verified Phase 1 allow. A free test proves that observer on a padded journal through the lower-only bound seams (the PTY counting reader refuses the same journal), with a sidechain control that must deny. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * v1.91.34.0: CHANGELOG, TODOS, version; harness decision for test-pr-profile; ship title-step test reads its new section * docs(changelog): use an example.com address in the allowEmail example * test: assemble GitHub-token fixtures at runtime in the new brain-sync and skill-start tests (CI credential scan) * fix: integration repairs across the wave (Step 6.5 fence, printed helper paths on Windows, probe usage without a heredoc) and tests pinned to the moved title and gstack-post contracts * test(doctor): the healthy fixture builds browse.exe on Windows, the name setup uses there * test(shared-libs): recognize Claude Code 2.1.292's internal git probe prefix (format.pretty and per-hook disables), with its CI receipt * fix(ship-measure): report lists v1 measurements as an older format instead of crashing --------- Co-authored-by: Mike Ilog <48659643+Mike-E-Log@users.noreply.github.com> Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> Co-authored-by: Thomas Gernhardt <t.gernhardt@softwareprojekt.net> Co-authored-by: akmandhania <143462686+akmandhania@users.noreply.github.com> Co-authored-by: Jared <182125291+yolo-jared@users.noreply.github.com> Co-authored-by: crblabs <293123359+crblabs@users.noreply.github.com> Co-authored-by: v639dragoon <matthewsbrickley@gmail.com>
705 lines
32 KiB
TypeScript
Executable File
705 lines
32 KiB
TypeScript
Executable File
#!/usr/bin/env bun
|
|
/**
|
|
* gstack-redact-prepush — git pre-push hook that scans the diff being pushed for
|
|
* HIGH-severity credentials and blocks the push on a hit.
|
|
*
|
|
* THIS IS A GUARDRAIL, NOT ENFORCEMENT. `git push --no-verify` bypasses it, as
|
|
* does `GSTACK_REDACT_PREPUSH=skip`. It catches accidental credential pushes,
|
|
* the most common real-world leak. It does NOT scan history, binary/LFS/submodule
|
|
* files, or non-added lines. History scanning is /cso's job.
|
|
*
|
|
* Git pre-push interface: refs are read from STDIN, one per line:
|
|
* <local ref> <local sha> <remote ref> <remote sha>
|
|
* We scan the ADDED lines of <remote sha>..<local sha> per ref (what's being
|
|
* pushed). Special cases:
|
|
* - remote sha all-zeroes → new branch: diff against merge-base with the
|
|
* remote's default branch (fallback: scan all commits unique to local ref).
|
|
* - local sha all-zeroes → branch delete: nothing to scan, skip.
|
|
* - force-push → remote..local still gives the net new content.
|
|
*
|
|
* Behavior:
|
|
* - HIGH finding in added lines → print + exit 1 (block), for public AND private.
|
|
* - MEDIUM → warn (non-blocking), one line per finding naming its rule,
|
|
* file line and fix, never the matched value. LOW/WARN → silent.
|
|
* - `pii.email` skips the pusher's own address, `gstack.redact.allowEmail`
|
|
* entries, and addresses already in the destination's commit metadata
|
|
* (see emailScanOptions).
|
|
* - GSTACK_REDACT_PREPUSH=skip → log + exit 0 (escape valve).
|
|
*
|
|
* Installed/uninstalled via `gstack-redact install-prepush-hook` (see the
|
|
* gstack-redact CLI), which chains any pre-existing hook.
|
|
*/
|
|
import { spawnSync } from "child_process";
|
|
import * as fs from "fs";
|
|
import * as path from "path";
|
|
import { normalizeWithMap, scan, type Finding, type ScanOptions } from "../lib/redact-engine";
|
|
import { mkdirpSync } from "../lib/fs-utils";
|
|
import { resolveStateRoot } from "../lib/state-root";
|
|
|
|
const ZERO = /^0+$/;
|
|
let emptyTree: string | undefined;
|
|
function emptyTreeOid(): string {
|
|
if (emptyTree) return emptyTree;
|
|
const oid = gitStrict(["hash-object", "-w", "-t", "tree", "--stdin"]).trim();
|
|
if (!/^(?:[0-9a-f]{40}|[0-9a-f]{64})$/.test(oid)) throw new Error("git could not resolve the empty tree");
|
|
emptyTree = oid;
|
|
return oid;
|
|
}
|
|
|
|
/**
|
|
* Permissive git for legitimately-fallible PROBES (symbolic-ref, rev-parse,
|
|
* merge-base) where a non-zero exit is normal control flow. The DIFF call
|
|
* must NOT use this — see gitStrict (#1946 fail-closed).
|
|
*/
|
|
function git(args: string[]): string {
|
|
const r = spawnSync("git", args, { encoding: "utf8", maxBuffer: 64 * 1024 * 1024 });
|
|
return r.status === 0 ? (r.stdout ?? "") : "";
|
|
}
|
|
|
|
/**
|
|
* Fail-closed git for the diff that decides whether the push is scanned
|
|
* (#1946). status !== 0 covers repo errors; status === null covers a killed
|
|
* process AND maxBuffer overflow — the oversized-diff case is exactly where
|
|
* a large secret-bearing blob is most likely, so "couldn't read the diff"
|
|
* must block, not silently allow.
|
|
*/
|
|
function gitStrict(args: string[]): string {
|
|
const r = spawnSync("git", args, { encoding: "utf8", maxBuffer: 64 * 1024 * 1024 });
|
|
// status !== 0 covers BOTH a non-zero exit AND null (process killed by a
|
|
// signal or maxBuffer overflow — null !== 0 is true).
|
|
if (r.status !== 0) {
|
|
throw new Error(
|
|
`git ${args[0]} failed (status=${r.status ?? "killed/overflow"}): ${(r.stderr ?? "").slice(0, 300)}`,
|
|
);
|
|
}
|
|
return r.stdout ?? "";
|
|
}
|
|
|
|
/** True when the object exists in the local odb (cat-file -e signals via exit code). */
|
|
function objectExists(sha: string): boolean {
|
|
const r = spawnSync("git", ["cat-file", "-e", sha], { encoding: "utf8" });
|
|
return r.status === 0;
|
|
}
|
|
|
|
/**
|
|
* The remote-tracking exclusion used when narrowing to "commits new to the
|
|
* remote" (#2592 catch-up merges, #2573 rebased force-pushes).
|
|
*
|
|
* Narrowed to the PUSH TARGET's namespace (S1): a bare `--remotes` excludes
|
|
* commits reachable from ANY remote-tracking ref, so a secret that had only
|
|
* ever been fetched from (or pushed to) a private/local-path remote was never
|
|
* scanned when later pushed to a PUBLIC remote — "already left this machine"
|
|
* is not "already reached THIS remote". Git hands pre-push the push remote's
|
|
* name as $1 (and its URL as $2); the installed hook wrapper forwards "$@".
|
|
* A URL push has no tracking namespace, even if origin happens to contain
|
|
* the same commits. Direct invocations without Git's argv keep the historical
|
|
* all-remotes behavior for compatibility.
|
|
*/
|
|
type PushTarget = { kind: "remote"; name: string } | { kind: "url" } | { kind: "unknown" };
|
|
let cachedPushTarget: PushTarget | undefined;
|
|
function pushTarget(): PushTarget {
|
|
if (cachedPushTarget) return cachedPushTarget;
|
|
const name = process.argv[2];
|
|
const url = process.argv[3];
|
|
if (!name && !url) return cachedPushTarget = { kind: "unknown" };
|
|
if (!name || !url) return cachedPushTarget = { kind: "url" };
|
|
const remotes = git(["remote"]).split("\n");
|
|
if (!remotes.includes(name)) return cachedPushTarget = { kind: "url" };
|
|
const pushUrls = git(["remote", "get-url", "--push", "--all", name]).trim().split("\n");
|
|
const fetchUrls = git(["remote", "get-url", "--all", name]).trim().split("\n");
|
|
return cachedPushTarget = pushUrls.includes(url) && fetchUrls[0] === url
|
|
? { kind: "remote", name }
|
|
: { kind: "url" };
|
|
}
|
|
|
|
function remotesExclusionArgs(): string[] {
|
|
const target = pushTarget();
|
|
if (target.kind === "remote") return [`--remotes=${target.name}/*`];
|
|
return target.kind === "unknown" ? ["--remotes"] : [];
|
|
}
|
|
|
|
function defaultRemoteBranch(): string | null {
|
|
const target = pushTarget();
|
|
if (target.kind === "url") return null;
|
|
const remote = target.kind === "remote" ? target.name : "origin";
|
|
const sym = git(["symbolic-ref", `refs/remotes/${remote}/HEAD`]).trim();
|
|
if (sym.startsWith(`refs/remotes/${remote}/`) && git(["rev-parse", "--verify", `${sym}^{commit}`]).trim()) return sym;
|
|
for (const b of [`${remote}/main`, `${remote}/master`]) {
|
|
const qualified = `refs/remotes/${b}`;
|
|
if (git(["rev-parse", "--verify", `${qualified}^{commit}`]).trim()) return qualified;
|
|
}
|
|
return null;
|
|
}
|
|
|
|
/**
|
|
* Base commit for a new remote ref, ordered from most precise to most
|
|
* conservative. Returns null when nothing can anchor the range.
|
|
*/
|
|
function unknownRemoteTipBase(localSha: string): string | null {
|
|
// 1. The common case: a merge-base with the remote's default branch.
|
|
const defaultBranch = defaultRemoteBranch();
|
|
const base = defaultBranch ? git(["merge-base", localSha, defaultBranch]).trim() : "";
|
|
if (base) return base;
|
|
|
|
// 2. No merge-base. The target's default branch may be unavailable
|
|
// (named trunk/develop, remote/HEAD unset), or history may be disjoint.
|
|
// Anything reachable from localSha but from NO remote-tracking
|
|
// branch is what this push actually adds; the parent of its oldest commit
|
|
// is the real base.
|
|
//
|
|
// Without this we drop straight to EMPTY_TREE and re-scan content that is
|
|
// already on the remote. That is not merely wasteful, it is wrong in two
|
|
// ways: a secret pushed long ago gets re-reported as if THIS push
|
|
// introduced it (telling the operator to rotate a key over someone else's
|
|
// old commit), and on any real repository the input overshoots the
|
|
// engine's byte cap, so `engine.input_too_large` blocks having scanned
|
|
// NOTHING — "scans more, never less" inverted into "scans nothing".
|
|
//
|
|
// The exclusion is scoped to the PUSH TARGET's tracking refs (see
|
|
// remotesExclusionArgs): content on some OTHER remote has left this machine,
|
|
// but it has not reached the remote being pushed to — a secret that only
|
|
// ever hit a private remote must still be scanned on its way to a public
|
|
// one (S1).
|
|
const exclusions = remotesExclusionArgs();
|
|
const newCommits = git(["rev-list", "--reverse", localSha, ...(exclusions.length ? ["--not", ...exclusions] : [])]).trim();
|
|
if (newCommits) {
|
|
const oldest = newCommits.split("\n")[0];
|
|
const parent = git(["rev-parse", "--verify", `${oldest}^`]).trim();
|
|
if (parent) return parent;
|
|
// Oldest new commit is a root commit: there is no parent to anchor on.
|
|
}
|
|
|
|
// 3. Nothing to anchor on — a genuinely fresh repository with no remote refs.
|
|
// Every commit IS new content, so scanning it all is the correct answer.
|
|
return null;
|
|
}
|
|
|
|
/**
|
|
* The commits this push actually adds — reachable from localSha and from NO
|
|
* remote-tracking ref.
|
|
*
|
|
* ⚠ WHY THIS EXISTS RATHER THAN A TWO-DOT RANGE.
|
|
*
|
|
* `remoteSha..localSha` is "everything new on this branch", which is NOT the
|
|
* same as "everything new to the remote". Merge origin/main into a feature
|
|
* branch and every commit main gained since the branch's last push becomes an
|
|
* added line — content that is already published, already scanned, and not
|
|
* this push's doing.
|
|
*
|
|
* Two things follow, and both were observed:
|
|
*
|
|
* · FALSE HIGH FINDINGS. A placeholder connection string in a test fixture,
|
|
* already merged to main by someone else, blocked an unrelated push as
|
|
* `db.url_with_password` — telling the operator to rotate a credential
|
|
* over a fixture they had never touched. A
|
|
* guard that cries wolf on catch-up merges is a guard people learn to
|
|
* bypass reflexively — which is exactly how a real secret gets through.
|
|
* · OVERSIZED SCANS. The comment on SCAN_CHUNK_BYTES below records a
|
|
* 1,146,782-byte diff from "a feature branch catching up to a busy main"
|
|
* blowing the engine's 1 MiB cap. Same root cause, treated there as a size
|
|
* problem and solved by slicing. Narrowing the range fixes the size too.
|
|
*
|
|
* A two-dot range cannot express this: after merging main, neither the remote
|
|
* tip nor the merge-base with main is an ancestor of the other, so no single
|
|
* base excludes both. `rev-list --not --remotes=<push-remote>/*` is the
|
|
* operation that does, and this file already reasons that way in
|
|
* `unknownRemoteTipBase` step 2. The exclusion is scoped to the push target's
|
|
* tracking namespace (see remotesExclusionArgs): the upstream commits a catch-up
|
|
* merge brings in came from the SAME remote being pushed to, so scoping keeps
|
|
* the #2592 fix intact while a secret known only to some OTHER (private)
|
|
* remote is still scanned on its way to this one (S1).
|
|
*
|
|
* Each commit is diffed alone. `--cc` on a merge shows only the conflict
|
|
* RESOLUTION — content that exists in no parent — so a secret introduced while
|
|
* resolving a merge is still caught, while an ordinary merge contributes
|
|
* nothing. Returns null when the notion does not apply, so callers fall back.
|
|
*/
|
|
function addedLinesFromNewCommits(localSha: string, remoteSha: string): AddedFile[] | null {
|
|
// remoteSha is what git TELLS us the remote has, and it is authoritative in a
|
|
// way `--remotes` is not: remote-tracking refs can be absent (a fresh clone
|
|
// that never fetched, a push to a remote with no tracking ref) or stale. Drop
|
|
// it and a repo with no tracking refs excludes NOTHING — every commit ever
|
|
// made reads as "new", which re-introduces the false positives from the other
|
|
// direction. So it stays the base; `--remotes` only ADDS exclusions on top.
|
|
if (ZERO.test(remoteSha) || !objectExists(remoteSha)) return null;
|
|
|
|
const narrowed = git(["rev-list", localSha, "--not", remoteSha, ...remotesExclusionArgs()]).trim();
|
|
if (!narrowed) return null;
|
|
|
|
// If excluding remote-tracking refs changes nothing, this push has no
|
|
// catch-up commits and the plain range already describes it exactly. Defer to
|
|
// it. That is not just an optimization: it keeps every push that ISN'T a
|
|
// catch-up merge on the original gitStrict diff path, so the fail-closed
|
|
// guarantee (#1946) and its regression test keep exercising the code they
|
|
// were written for. A narrowing that silently retired that test would be a
|
|
// worse trade than the false positives it set out to fix.
|
|
const plain = git(["rev-list", `${remoteSha}..${localSha}`]).trim();
|
|
const asSet = (s: string) => s.split("\n").filter(Boolean).sort().join("\n");
|
|
if (asSet(narrowed) === asSet(plain)) return null;
|
|
|
|
const shas = narrowed.split("\n").filter(Boolean);
|
|
// A rewrite of long history should fall back rather than shell out per commit.
|
|
if (shas.length > 500) return null;
|
|
const out: AddedFile[] = [];
|
|
for (const sha of shas) {
|
|
// gitStrict: a failed diff must never read as "nothing added" (#1946).
|
|
// Collected per commit so a finding names the commit whose file line it is.
|
|
out.push(...collectAddedLines(gitStrict([
|
|
"show", "--unified=0", "--no-color", "--no-ext-diff", "--no-textconv",
|
|
"--src-prefix=a/", "--dst-prefix=b/", "--cc", "--format=", sha,
|
|
]), sha));
|
|
}
|
|
return out;
|
|
}
|
|
|
|
/**
|
|
* Added lines grouped by the file they were added to (`+++ b/` path; empty
|
|
* when unknown). `lines[i]` is the file line number of the i-th line of
|
|
* `text`, read from the hunk headers; `commit` is set when the lines came from
|
|
* one commit's diff rather than the net diff of the pushed range.
|
|
*/
|
|
interface AddedFile { file: string; text: string; lines: number[]; commit?: string }
|
|
|
|
/** Return the added lines for a ref update being pushed. */
|
|
function addedLinesFor(localSha: string, remoteSha: string): AddedFile[] {
|
|
// Preferred ONLY when this push carries catch-up commits: scanning them again
|
|
// is the bug. Every other shape falls through to the range logic below.
|
|
const fromNew = addedLinesFromNewCommits(localSha, remoteSha);
|
|
if (fromNew !== null) return fromNew;
|
|
|
|
let range: string;
|
|
if (ZERO.test(remoteSha) || !objectExists(remoteSha)) {
|
|
const base = ZERO.test(remoteSha) ? unknownRemoteTipBase(localSha) : null;
|
|
range = base ? `${base}..${localSha}` : `${emptyTreeOid()}..${localSha}`;
|
|
} else {
|
|
// Existing branch (incl. force-push): net new content remote..local.
|
|
range = `${remoteSha}..${localSha}`;
|
|
}
|
|
// -U0: only changed lines; we keep lines starting with '+' (added), drop the
|
|
// +++ file header. Unified diff added lines start with a single '+'.
|
|
// Strict (#1946): a failed diff used to return "" and the push sailed
|
|
// through unscanned — fail open on the exact path the guard exists for.
|
|
//
|
|
// --no-ext-diff: a user's `diff.external` driver replaces the entire diff
|
|
// with its own output — with one set, `git diff` emits zero '+' lines, so an
|
|
// unhardened scanner reads an empty diff and exits 0 on a push full of
|
|
// secrets. Reachable from ordinary user config, not hypothetical. (#2498)
|
|
// --no-textconv: a .gitattributes textconv driver can likewise rewrite
|
|
// content before we ever see it. (#2498)
|
|
const diff = gitStrict([
|
|
"diff", "--unified=0", "--no-color", "--no-ext-diff", "--no-textconv",
|
|
"--src-prefix=a/", "--dst-prefix=b/", range,
|
|
]);
|
|
return collectAddedLines(diff);
|
|
}
|
|
|
|
/**
|
|
* Added-line text from a unified diff. Shared by both range strategies so the
|
|
* hunk-aware header handling below cannot drift between them.
|
|
*/
|
|
function collectAddedLines(diff: string, commit?: string): AddedFile[] {
|
|
const added: AddedFile[] = [];
|
|
// Hunk-aware header skip (#2498): `+++ ` is only a FILE HEADER outside a
|
|
// hunk. Inside a hunk, an added content line whose text begins with "++"
|
|
// renders as "+++<content>" — the old blanket startsWith("+++") skip
|
|
// silently dropped exactly those lines from the scan.
|
|
let inHunk = false;
|
|
let file = "";
|
|
let fileLines: string[] = [];
|
|
let lineNumbers: number[] = [];
|
|
// File line tracking from hunk headers. A combined (--cc) header has one
|
|
// more `@` than it has parents, and each body line one prefix column per
|
|
// parent; a line belongs to the resulting file unless a column says `-`.
|
|
let parents = 1;
|
|
let nextLine = 0;
|
|
const flush = () => {
|
|
const lines = releaseVersionExempt(file, fileLines);
|
|
if (lines.length) added.push({ file, text: lines.join("\n"), lines: lineNumbers, ...(commit ? { commit } : {}) });
|
|
fileLines = [];
|
|
lineNumbers = [];
|
|
};
|
|
for (const line of diff.split("\n")) {
|
|
// `diff --` rather than `diff --git`: a merge scanned with --cc emits
|
|
// `diff --cc <path>`, so a --git-only reset left inHunk true across file
|
|
// boundaries and read the next file's `+++ b/...` header as content. Only
|
|
// noise (it over-scans, never under-scans), but the boundary is real.
|
|
if (line.startsWith("diff --")) { flush(); file = ""; inHunk = false; continue; }
|
|
if (line.startsWith("@@")) {
|
|
inHunk = true;
|
|
parents = (/^@+/.exec(line)?.[0].length ?? 2) - 1;
|
|
nextLine = Number(/ \+(\d+)(?:,\d+)? @+/.exec(line)?.[1] ?? 0);
|
|
continue;
|
|
}
|
|
if (!inHunk && line.startsWith("+++ ")) { file = line.startsWith("+++ b/") ? line.slice(6) : ""; continue; }
|
|
if (!inHunk && (line.startsWith("+++") || line.startsWith("---"))) continue;
|
|
const inResult = inHunk && !line.startsWith("\\") && !line.slice(0, parents).includes("-");
|
|
const lineNumber = inResult ? nextLine++ : 0;
|
|
if (line.startsWith("+")) {
|
|
fileLines.push(line.slice(1));
|
|
lineNumbers.push(lineNumber);
|
|
}
|
|
}
|
|
flush();
|
|
return added;
|
|
}
|
|
|
|
const VERSION_FILE_CONTENT = /^\d+(?:\.\d+)+$/;
|
|
const CHANGELOG_RELEASE_VERSION = /^(#{1,6}[\t ]*\[?v?)\d+(?:\.\d+){3}(?=[\]\t ]|$)/;
|
|
|
|
/**
|
|
* #2856: a 4-part release version (MAJOR.MINOR.PATCH.MICRO) is shaped exactly like a
|
|
* public IPv4 address, so /ship's own VERSION bump and CHANGELOG heading
|
|
* raised pii.ip_public on every release. Exempt only those two release
|
|
* artifacts, decided from the pushed diff alone (works in shallow clones):
|
|
* a root VERSION whose entire added content is one version string, and the
|
|
* version token of a root CHANGELOG.md release heading (the rest of the
|
|
* heading is still scanned). Every other file — and any other position in
|
|
* these two — is scanned unchanged; the engine's global allowlist is untouched.
|
|
*/
|
|
function releaseVersionExempt(file: string, lines: string[]): string[] {
|
|
if (file === "VERSION" && lines.length === 1 && VERSION_FILE_CONTENT.test(lines[0].trim())) return [];
|
|
if (file === "CHANGELOG.md") return lines.map((l) => l.replace(CHANGELOG_RELEASE_VERSION, "$1VERSION"));
|
|
return lines;
|
|
}
|
|
|
|
/**
|
|
* Byte budget per scan() call. Kept comfortably under redact-engine's
|
|
* DEFAULT_MAX_BYTES (1 MiB) so a slice never trips its oversize guard.
|
|
*/
|
|
const SCAN_CHUNK_BYTES = 768 * 1024;
|
|
const SCAN_OVERLAP_CHARS = 16 * 1024;
|
|
const MAX_SCAN_BYTES = 1024 * 1024;
|
|
|
|
/**
|
|
* Scan complete-line core slices with bounded left and right context. Admit
|
|
* only findings whose original captured start lies inside that core.
|
|
*
|
|
* Why: the engine refuses input over its byte cap and fails closed, which is
|
|
* right for one scan() call but wrong as a push policy — a feature branch
|
|
* catching up to a busy main legitimately produces more added lines than the
|
|
* cap (1,146,782 bytes against the 1 MiB default in the push that prompted
|
|
* this, and only ~7% of that was the lockfile). The push then blocked on
|
|
* `engine.input_too_large` — a size error naming no credential — which trains
|
|
* people to reach for --no-verify, defeating the guardrail far more thoroughly
|
|
* than a large diff does.
|
|
*
|
|
* A match may need context on either adjacent line (`nearWindow` reaches 300
|
|
* normalized characters). The full core lines keep opaque spans intact; the
|
|
* engine's normalization map selects context in normalized units without
|
|
* changing the raw input. A partial context may create an artificial boundary,
|
|
* but its findings are never admitted as core findings.
|
|
*
|
|
* Fail-closed is preserved: a SINGLE line over the budget is still passed to
|
|
* the engine intact, so a genuinely unscannable blob (minified bundle,
|
|
* embedded base64) trips input_too_large and blocks exactly as before.
|
|
*
|
|
* The engine reports line/col relative to each slice; an admitted finding's
|
|
* `line` is rewritten to its 1-based line within `added`, which the caller
|
|
* maps to the file line. Do not lift this into the engine, where callers rely
|
|
* on absolute line numbers.
|
|
*/
|
|
function scanAddedLines(added: string, opts: Parameters<typeof scan>[1]): Finding[] {
|
|
const findings: Finding[] = [];
|
|
const cores: Array<{ start: number; end: number }> = [];
|
|
const lines = added.split("\n");
|
|
let start = 0;
|
|
let end = 0;
|
|
let bytes = 0;
|
|
for (let i = 0; i < lines.length; i++) {
|
|
const line = lines[i];
|
|
const newline = i < lines.length - 1 ? 1 : 0;
|
|
const lineBytes = Buffer.byteLength(line, "utf8") + newline;
|
|
if (bytes && bytes + lineBytes > SCAN_CHUNK_BYTES) {
|
|
cores.push({ start, end });
|
|
start = end;
|
|
bytes = 0;
|
|
}
|
|
end += line.length + newline;
|
|
bytes += lineBytes;
|
|
if (lineBytes > SCAN_CHUNK_BYTES) {
|
|
cores.push({ start, end });
|
|
start = end;
|
|
bytes = 0;
|
|
}
|
|
}
|
|
if (bytes) cores.push({ start, end });
|
|
|
|
const contextStart = (boundary: number): number => {
|
|
if (!boundary) return 0;
|
|
let length = Math.min(boundary, SCAN_OVERLAP_CHARS * 2);
|
|
while (true) {
|
|
const from = boundary - length;
|
|
const raw = added.slice(from, boundary);
|
|
if (Buffer.byteLength(raw, "utf8") > MAX_SCAN_BYTES) return from;
|
|
const { normalized, map } = normalizeWithMap(raw);
|
|
if (!from || normalized.length > SCAN_OVERLAP_CHARS) {
|
|
return from + (map[Math.max(0, normalized.length - SCAN_OVERLAP_CHARS - 1)] ?? 0);
|
|
}
|
|
length = Math.min(boundary, length * 2);
|
|
}
|
|
};
|
|
|
|
const contextEnd = (boundary: number): number => {
|
|
if (boundary === added.length) return boundary;
|
|
let length = Math.min(added.length - boundary, SCAN_OVERLAP_CHARS * 2);
|
|
while (true) {
|
|
const to = boundary + length;
|
|
const raw = added.slice(boundary, to);
|
|
if (Buffer.byteLength(raw, "utf8") > MAX_SCAN_BYTES) return to;
|
|
const { normalized, map } = normalizeWithMap(raw);
|
|
if (to === added.length || normalized.length > SCAN_OVERLAP_CHARS) {
|
|
return boundary + (map[Math.min(normalized.length, SCAN_OVERLAP_CHARS + 1)] ?? length);
|
|
}
|
|
length = Math.min(added.length - boundary, length * 2);
|
|
}
|
|
};
|
|
|
|
const addedLineStarts = [0];
|
|
for (let i = 0; i < added.length; i++) if (added[i] === "\n") addedLineStarts.push(i + 1);
|
|
const lineAt = (offset: number): number => {
|
|
let lo = 0;
|
|
let hi = addedLineStarts.length - 1;
|
|
while (lo < hi) {
|
|
const mid = (lo + hi + 1) >> 1;
|
|
if (addedLineStarts[mid] <= offset) lo = mid;
|
|
else hi = mid - 1;
|
|
}
|
|
return lo + 1;
|
|
};
|
|
|
|
for (const core of cores) {
|
|
const left = contextStart(core.start);
|
|
const text = added.slice(left, contextEnd(core.end));
|
|
const result = scan(text, opts);
|
|
if (result.oversize) {
|
|
findings.push(...result.findings);
|
|
return findings;
|
|
}
|
|
const starts = [0];
|
|
for (let i = 0; i < text.length; i++) if (text[i] === "\n") starts.push(i + 1);
|
|
for (const finding of result.findings) {
|
|
const offset = starts[finding.line - 1] + finding.col - 1 + left;
|
|
if (offset >= core.start && offset < core.end) findings.push({ ...finding, line: lineAt(offset) });
|
|
}
|
|
}
|
|
|
|
return findings;
|
|
}
|
|
|
|
function logSkip(reason: string): void {
|
|
try {
|
|
const home = resolveStateRoot();
|
|
const dir = path.join(home, "security");
|
|
// mkdirpSync, not bare mkdirSync: bun-on-Windows EEXIST (#2635). This site
|
|
// is try-wrapped by the caller, so the old failure was a silent skip-log
|
|
// loss rather than a crash — the fix makes the log survive, not un-crash.
|
|
mkdirpSync(dir);
|
|
fs.appendFileSync(
|
|
path.join(dir, "prepush-skip.jsonl"),
|
|
JSON.stringify({ ts: new Date().toISOString(), reason }) + "\n",
|
|
);
|
|
} catch {
|
|
// best-effort; never block a push because logging failed
|
|
}
|
|
}
|
|
|
|
const HISTORY_MAX_COMMITS = 50_000;
|
|
const HISTORY_TIMEOUT_MS = 5_000;
|
|
|
|
function configValues(args: string[]): string[] {
|
|
return git(["config", ...args]).split("\n").map((v) => v.trim()).filter(Boolean);
|
|
}
|
|
|
|
/**
|
|
* Addresses `pii.email` treats as not a new leak (#3060): the pusher's own
|
|
* (`git config user.email`), the ones listed in `gstack.redact.allowEmail`,
|
|
* and the author and committer addresses (raw and mailmapped) the destination
|
|
* will hold after this push. That last set is the history of every pushed tip
|
|
* (pushed commits become public with the push, third-party authors included),
|
|
* of each remote tip being replaced, and, for a ref whose remote tip is new or
|
|
* unknown, the push remote's own tracking refs. A URL or unknown destination
|
|
* never borrows a remote's tracking refs; another remote's history never counts.
|
|
*
|
|
* Each input fails on its own and a failure omits only that input, which is
|
|
* the behavior before these exemptions existed. The history read is bounded:
|
|
* past the commit cap it uses the newest authors it read; an error or the
|
|
* timeout passes no history, and the reason is returned so the report can say
|
|
* suppression was limited.
|
|
*/
|
|
function emailScanOptions(historyRevs: string[], trackingRemote: string | null): {
|
|
options: Pick<ScanOptions, "selfEmail" | "repoPublicEmails">;
|
|
limited?: string;
|
|
} {
|
|
const selfEmail = configValues(["--get", "user.email"])[0];
|
|
const allowed = configValues(["--get-all", "gstack.redact.allowEmail"]);
|
|
const r = spawnSync("git", [
|
|
"log", "--no-show-signature", "--no-color", `--max-count=${HISTORY_MAX_COMMITS}`, "--format=%ae%n%aE%n%ce%n%cE",
|
|
...historyRevs, ...(trackingRemote ? [`--remotes=${trackingRemote}/*`] : []), "--",
|
|
], { encoding: "utf8", maxBuffer: 64 * 1024 * 1024, timeout: HISTORY_TIMEOUT_MS, killSignal: "SIGKILL" });
|
|
const timedOut = (r.error as NodeJS.ErrnoException | undefined)?.code === "ETIMEDOUT";
|
|
const history = r.status === 0 && !r.error
|
|
? (r.stdout ?? "").split("\n").map((e) => e.trim()).filter((e) => e.includes("@"))
|
|
: null;
|
|
return {
|
|
options: {
|
|
...(selfEmail ? { selfEmail } : {}),
|
|
repoPublicEmails: [...new Set([...(history ?? []), ...allowed])],
|
|
},
|
|
...(history ? {} : { limited: timedOut ? `reading commit history took over ${HISTORY_TIMEOUT_MS / 1000} s` : "reading commit history failed" }),
|
|
};
|
|
}
|
|
|
|
/** One MEDIUM report line: rule id, file line (and commit), and the fix. Never the matched value. */
|
|
function mediumLine(f: Finding, part: AddedFile): string {
|
|
const line = part.lines[f.line - 1];
|
|
const where = `${part.file || "(unknown file)"}${line ? `:${line}` : ""}` +
|
|
(part.commit ? ` in commit ${part.commit.slice(0, 12)}` : "");
|
|
const fix = f.id === "pii.email"
|
|
? "remove the address, or allow it: git config --add gstack.redact.allowEmail <address>"
|
|
: "remove the value before it becomes public";
|
|
return ` MEDIUM ${f.id} ${where} fix: ${fix}\n`;
|
|
}
|
|
|
|
function main() {
|
|
if ((process.env.GSTACK_REDACT_PREPUSH || "").toLowerCase() === "skip") {
|
|
logSkip(process.env.GSTACK_REDACT_PREPUSH_REASON || "env-skip");
|
|
process.stderr.write("gstack-redact-prepush: skipped via GSTACK_REDACT_PREPUSH=skip\n");
|
|
process.exit(0);
|
|
}
|
|
|
|
const stdin = fs.readFileSync(0, "utf8");
|
|
const refs = stdin
|
|
.split("\n")
|
|
.map((l) => l.trim())
|
|
.filter(Boolean)
|
|
.map((l) => l.split(/\s+/));
|
|
|
|
const parts: AddedFile[] = [];
|
|
const historyRevs = new Set<string>();
|
|
let trackingRemote: string | null = null;
|
|
|
|
for (const fields of refs) {
|
|
// Fail CLOSED on a ref line we cannot parse (#2498): git hands pre-push
|
|
// exactly "<local ref> <local sha> <remote ref> <remote sha>" — anything
|
|
// else means we cannot tell WHAT is being pushed, and silently skipping
|
|
// it would leave that ref unscanned.
|
|
const [, localSha, , remoteSha] = fields;
|
|
const shaShaped = (s: string | undefined) => !!s && /^[0-9a-f]{40,64}$/i.test(s);
|
|
if (fields.length !== 4 || !shaShaped(localSha) || !shaShaped(remoteSha)) {
|
|
process.stderr.write(
|
|
"\n⛔ gstack-redact-prepush BLOCKED the push — could not parse a pre-push ref line, " +
|
|
"so its content cannot be scanned.\n" +
|
|
` line: ${JSON.stringify(fields.join(" "))}\n` +
|
|
"Bypass if you're sure: GSTACK_REDACT_PREPUSH=skip git push (or git push --no-verify)\n",
|
|
);
|
|
process.exit(1);
|
|
}
|
|
if (ZERO.test(localSha!)) continue; // branch delete → nothing pushed
|
|
let added: AddedFile[];
|
|
try {
|
|
added = addedLinesFor(localSha, remoteSha || "0");
|
|
} catch (err) {
|
|
// Fail CLOSED (#1946): if we can't compute the pushed diff we can't
|
|
// scan it, and unscanned-but-allowed is the failure mode this hook
|
|
// exists to prevent.
|
|
process.stderr.write(
|
|
"\n⛔ gstack-redact-prepush BLOCKED the push — could not compute the pushed diff, " +
|
|
"so it cannot be scanned for credentials.\n" +
|
|
` (${err instanceof Error ? err.message.split("\n")[0] : String(err)})\n` +
|
|
"Bypass if you're sure: GSTACK_REDACT_PREPUSH=skip git push (or git push --no-verify)\n",
|
|
);
|
|
process.exit(1);
|
|
}
|
|
parts.push(...added.filter((part) => part.text.trim()));
|
|
historyRevs.add(localSha);
|
|
if (!ZERO.test(remoteSha) && objectExists(remoteSha)) historyRevs.add(remoteSha);
|
|
else {
|
|
const target = pushTarget();
|
|
if (target.kind === "remote") trackingRemote = target.name;
|
|
}
|
|
}
|
|
|
|
const allHigh: Finding[] = [];
|
|
const mediums: string[] = [];
|
|
let emailMedium = false;
|
|
const emails = parts.length ? emailScanOptions([...historyRevs], trackingRemote) : { options: {} };
|
|
// Visibility doesn't change HIGH behavior; pass private so nothing is treated
|
|
// as public-strict (HIGH blocks regardless either way).
|
|
// Sliced (see scanAddedLines) so a large-but-legitimate diff is actually
|
|
// scanned rather than blocked unscanned on the engine's size cap.
|
|
// Scanned per file so a pattern can use the file's type (#3048).
|
|
for (const part of parts) {
|
|
const opts = { repoVisibility: "private" as const, ...emails.options, ...(part.file ? { sourcePath: part.file } : {}) };
|
|
for (const f of scanAddedLines(part.text, opts)) {
|
|
if (f.severity === "HIGH") allHigh.push(f);
|
|
else if (f.severity === "MEDIUM") {
|
|
mediums.push(mediumLine(f, part));
|
|
if (f.id === "pii.email") emailMedium = true;
|
|
}
|
|
}
|
|
}
|
|
|
|
if (mediums.length > 0) {
|
|
process.stderr.write(
|
|
`gstack-redact-prepush: ${mediums.length} MEDIUM finding(s) in pushed diff (PII/internal). ` +
|
|
"Not blocking. Review before this becomes public.\n",
|
|
);
|
|
for (const line of mediums) process.stderr.write(line);
|
|
if (emailMedium && "limited" in emails && emails.limited) {
|
|
process.stderr.write(
|
|
`gstack-redact-prepush: existing-email suppression was limited for this push (${emails.limited}), ` +
|
|
"so an address already in the history may be listed above.\n",
|
|
);
|
|
}
|
|
}
|
|
|
|
if (allHigh.length > 0) {
|
|
// A scan that could not RUN is not a scan that FOUND something. Reporting
|
|
// "credential(s) in the pushed diff — rotate the credential" for an
|
|
// `engine.*` finding tells the operator to rotate a secret that was never
|
|
// detected, on a diff that was never read. Blocking is still right (fail
|
|
// closed), but the reason must be the true one: a guardrail that cries wolf
|
|
// is a guardrail that gets bypassed by reflex, which is worse than none.
|
|
// The computed range can be much larger than the visible tip diff, and a
|
|
// single over-cap line is deliberately withheld rather than partly scanned.
|
|
const unscanned = allHigh.filter((f) => f.id.startsWith("engine."));
|
|
const secrets = allHigh.filter((f) => !f.id.startsWith("engine."));
|
|
|
|
if (secrets.length > 0) {
|
|
process.stderr.write(
|
|
"\n⛔ gstack-redact-prepush BLOCKED the push — credential(s) in the pushed diff:\n\n",
|
|
);
|
|
for (const f of secrets) {
|
|
process.stderr.write(` HIGH ${f.id} ${f.preview}\n`);
|
|
}
|
|
process.stderr.write(
|
|
"\nRotate the credential (a pushed secret is compromised) and remove it from the diff.\n",
|
|
);
|
|
}
|
|
|
|
if (unscanned.length > 0) {
|
|
process.stderr.write(
|
|
"\n⛔ gstack-redact-prepush BLOCKED the push — the diff could NOT be scanned.\n" +
|
|
" No credential was found; none was looked for. Blocking fail-closed.\n\n",
|
|
);
|
|
for (const f of unscanned) {
|
|
process.stderr.write(` ${f.id}: ${f.description}\n`);
|
|
}
|
|
process.stderr.write(
|
|
"\nA long line, large first push, or unavailable remote tip may exceed the\n" +
|
|
"per-slice safety cap. Check the destination and range; scan the diff yourself\n" +
|
|
"before bypassing: `git diff <base>..HEAD | grep -inE \'password|secret|token|api.?key\'`.\n",
|
|
);
|
|
}
|
|
|
|
process.stderr.write(
|
|
"This is a guardrail: `git push --no-verify` or `GSTACK_REDACT_PREPUSH=skip git push` bypass it.\n",
|
|
);
|
|
process.exit(1);
|
|
}
|
|
|
|
process.exit(0);
|
|
}
|
|
|
|
main();
|