mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 22:14:25 +01:00
v1.91.42.0 fix(cso): lift the snapshot file-count ceiling, bound report lists, actionable capacity errors (#3078)
* fix(cso): give snapshot.json its own 16 MiB cap and actionable capacity errors /cso refused any repository with more than about 3,100 tracked files before an audit started: the snapshot manifest was pretty-printed and both its writer and every reader capped it at the shared 1 MiB private-state limit (#3068, manifest half of #2993). - snapshot.json is written compactly and capped by SNAPSHOT_MANIFEST_LIMIT (16 MiB), a constant local to the snapshot module. - readJson takes a per-artifact cap; readSnapshotManifest is the only reader of snapshot.json and passes the snapshot cap through recovery and the private-file size check. MAX_STATE_FILE and MAX_OUTPUT stay 1 MiB. - The manifest-cap and 64 MiB source-cap errors name the bound cap, the measured value, what counts toward it, the next step and #2993. A single file that would cross 64 MiB now hits the same message before it is read. - Older pretty-printed manifests still load (recheck, inspect). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(cso): bound per-entry report and evidence lists with omitted counts Lifting the manifest cap alone moved the failure into the audit: report.json listed one transformation per transformed entry plus one coverage gap or exclusion per omitted entry (1.97 MB for 5,000 nested paths), and sensitive-evidence.json grew per scanner hit (1.68 MB), both under the 1 MiB private-state cap that saveReport and the start stdout share. - boundedList (state.ts) keeps the leading entries that fit a byte budget, measured as the pretty-printed JSON they add at their nesting depth, and ends the list with one summary carrying the omitted count. - report.json snapshot lists share one 256 KiB budget (gaps 128 KiB, transformations 96 KiB, exclusions 32 KiB); coverage counts and status are computed before trimming, so they stay exact. - sensitive-evidence.json keeps 512 KiB of per-file evidence plus an omitted-count record. - Snapshot coverage and publicSnapshotPath no longer scan the whole manifest per entry. Test: 5,000 long nested paths, mostly excluded or withheld, 999 files with scanner hits, driven through start, inspect, submit, finish and recheck. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(cso): admit Git path listings above 1 MiB during snapshot capture With the manifest cap lifted, the next file-count wall was the shared 1 MiB process-output cap on Git's path listings: ls-files/ls-tree --name-only failed at roughly 15-20k files (23,749-file grafana/grafana fails there) and the comparison-base ls-tree near 8-10k. runProcess lets raw callers, whose output is never printed or persisted, opt in to up to 16 MiB; git() forwards an optional cap, and the four snapshot listings pass SNAPSHOT_MANIFEST_LIMIT. Every other caller keeps the MAX_OUTPUT clamp. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(cso): explain the capacity refusals and file the follow-ups docs/troubleshooting.md#cso-capacity covers the snapshot-list cap, the 64 MiB source cap and the tracked-symlink refusal. TODOS.md records the symlink design call, the #2993 aggregate cap, the full-manifest inspect output and repair-bundle transformations. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore(release): v1.91.42.0 /cso lifts its file-count ceiling Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
c7554824cb
commit
52eec18c63
11 files changed
+538
-67
No files matched your search
@@ -1,5 +1,47 @@
|
||||
# Changelog
|
||||
|
||||
## [1.91.42.0] - 2026-10-08
|
||||
|
||||
**/cso lifts its file-count ceiling: repositories with tens of thousands of files reach the audit.**
|
||||
**Repositories with more than 64 MiB of tracked source, or with a tracked symlink, are still refused, and the message now says which limit applied and what to do.**
|
||||
|
||||
/cso refused any repository with more than about 3,100 files before an audit started, with "Snapshot manifest exceeds the 1 MiB private-state admission limit" (#3068). The snapshot's file list shared a 1 MiB limit with every other private file /cso keeps, and so did the report it writes, the list of files with secret-like findings, and Git's own file listing. The snapshot list now has its own 16 MiB limit (about 50,000 files) and is written compactly. Per-file lists in the report and the evidence file keep their first entries and end with one line saying how many more there are. The counts in the report stay exact.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Shallow clones measured on 2026-10-08 with `start --offline`, before and after this release:
|
||||
|
||||
| Repository | Files | Tracked source | Before | After |
|
||||
|---|---|---|---|---|
|
||||
| sveltejs/svelte | 9,182 | 6.9 MiB | refused (file list over 1 MiB) | audit starts; 2.8 MiB file list, 10 KB report |
|
||||
| facebook/react | 7,252 | 38.7 MiB | refused (file list over 1 MiB) | audit starts; 2.3 MiB file list, 53 KB report |
|
||||
| laravel/framework | 3,439 | 25.7 MiB | refused (file list over 1 MiB) | audit starts; 1.0 MiB file list, 30 KB report |
|
||||
| django/django | 7,085 | 44.0 MiB | refused (4 tracked symlinks) | refused (symlinks); starts once they are removed |
|
||||
| rails/rails | 5,007 | 38.1 MiB | refused (1 tracked symlink) | refused (symlink); starts once it is removed |
|
||||
| hashicorp/terraform | 5,562 | 25.6 MiB | refused (10 tracked symlinks) | refused (symlinks); starts once they are removed |
|
||||
| grafana/grafana | 23,749 | 210.7 MiB | refused (Git file listing over 1 MiB) | refused (symlinks); then the 64 MiB source cap |
|
||||
| microsoft/vscode | 20,238 | 540.0 MiB | refused (Git file listing over 1 MiB) | refused (symlink); then the 64 MiB source cap |
|
||||
|
||||
On a synthetic 5,000-file repository with long nested paths, where most files are excluded or withheld and 1,000 files carry secret-like strings, the report is 267 KB and the evidence file 524 KB. Unbounded, the same lists took 1.97 MB and 1.68 MB, over the 1 MiB limit. A free test drives that shape through start, inspect, findings, finish and recheck.
|
||||
|
||||
### What this means for you
|
||||
|
||||
Run `/gstack-upgrade`, then run `/cso` on a repository with more than 3,100 files: the audit starts instead of stopping at the snapshot. If /cso still refuses, the message names the limit (the 16 MiB snapshot list, the 64 MiB source cap, or a symlink), the measured size, what counts toward it and the next step. No setting raises these limits yet; the 64 MiB source cap is tracked in #2993, and `docs/troubleshooting.md#cso-capacity` explains all three.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Fixed
|
||||
- /cso refused repositories with more than about 3,100 files before an audit. The snapshot file list is written compactly and capped at 16 MiB; every other private file keeps its 1 MiB limit. Snapshot lists written by earlier releases still load in recheck and inspect. (#3068)
|
||||
- Report coverage gaps, exclusions and transformations, and the evidence file of secret-like findings, keep their first entries and end with an omitted count, so a large repository no longer overflows the report partway through an audit.
|
||||
- Git's file listings during capture are allowed up to the snapshot list size, so repositories past roughly 11,000 to 23,000 files, depending on path length (7,000 to 10,000 with `--diff` or `--base`), no longer fail with "Could not read bounded Git metadata: ls-tree exceeded the output limit".
|
||||
- The snapshot-list and 64 MiB source-cap errors state the measured value, the limit, what counts toward it, the next step and the #2993 link. A single file that would cross 64 MiB now gets the same message before it is read. (#2993)
|
||||
|
||||
#### For contributors
|
||||
- `readJson` takes a per-artifact size cap. `readSnapshotManifest(dir)` is the only reader of `snapshot.json`, and a source test fails on a direct read.
|
||||
- `boundedList` in `lib/cso/state.ts` bounds a per-entry list by the pretty-printed bytes it adds and appends one omitted-count item.
|
||||
- `runProcess` lets raw callers opt in to up to 16 MiB of output; everything else keeps the `MAX_OUTPUT` clamp, which stays 1 MiB.
|
||||
- Thanks to @almoatasemm for the report (#3068) and @saanjay for the capacity diagnosis of the manifest, reader and source caps (#2993).
|
||||
|
||||
## [1.91.38.0] - 2026-10-07
|
||||
|
||||
**`/ios-qa` works on iPads, and a dropped USB route no longer restarts the app you are testing.**
|
||||
|
||||
@@ -2,6 +2,31 @@
|
||||
|
||||
## NEXT PRIORITY
|
||||
|
||||
### P1/P2: /cso capacity follow-ups (filed 2026-10-08)
|
||||
|
||||
Left open by v1.91.42.0, which lifted the snapshot file-count ceiling.
|
||||
|
||||
- **Tracked symlinks refuse the whole /cso run** — `capture()` fails closed on
|
||||
any tracked symlink ("Symlink or special source file"). 5 of 8 mid-size OSS
|
||||
repos measured for v1.91.42.0 stop there (django, rails, terraform, grafana,
|
||||
vscode; 1-10 symlinks each, mostly test fixtures). Recording them as unread
|
||||
exclusions in the manifest, like dependency trees, is a security design call
|
||||
for Garry. **Effort:** M. **Priority:** P1.
|
||||
- **64 MiB aggregate source cap (#2993)** — files over 1 MiB are withheld from
|
||||
the audit but still read and counted. Stream-hash them without counting
|
||||
toward the cap, report them as unread coverage, and add an admission estimate
|
||||
to `doctor`. grafana (211 MiB) and vscode (540 MiB) hit it once symlinks are
|
||||
out of the way. A `GSTACK_CSO_SNAPSHOT_MAX_BYTES` knob was deferred with it.
|
||||
**Effort:** L. **Priority:** P2.
|
||||
- **`inspect` prints the whole public manifest** — about 2.8 MB of stdout for
|
||||
5,000 files, more than an agent's tool output keeps. The skill reads paths
|
||||
from it, so trimming it is a skill-contract change with evals.
|
||||
**Effort:** M. **Priority:** P2.
|
||||
- **Repair bundles carry every transformation** — `lib/cso/verification.ts`
|
||||
binds the full transformation list into a 1 MiB immutable bundle. Only
|
||||
runtime verification writes bundles, and no qualified runtime catalog exists
|
||||
yet. **Effort:** M. **Priority:** P3.
|
||||
|
||||
### P2/P3: iOS QA fix-wave follow-ups (filed 2026-10-07, v1.91.38.0)
|
||||
|
||||
Left open by the iPad and route-drop release, each with its reason.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# gstack digest v1.91.38.0 — regenerate/re-copy after upgrading gstack
|
||||
# gstack digest v1.91.42.0 — regenerate/re-copy after upgrading gstack
|
||||
|
||||
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
|
||||
for agent hosts without a full skill install. The full skills add workflows,
|
||||
|
||||
@@ -674,6 +674,24 @@ Linux) with a local Docker socket. On Windows, static assessment is the
|
||||
supported mode; if the refusal named a user directory, install Docker Desktop
|
||||
under Program Files.
|
||||
|
||||
<a id="cso-capacity"></a>
|
||||
### `Snapshot manifest cap exceeded: ...` / `64 MiB source cap exceeded: ...` / `Symlink or special source file: <path>`
|
||||
|
||||
**Meaning.** /cso copies the repository into a private snapshot before any
|
||||
audit starts, and refuses the whole run when the snapshot would be incomplete
|
||||
or too large. Each message names the cap that applied:
|
||||
|
||||
- **Snapshot manifest cap (16 MiB).** One entry per tracked or nonignored
|
||||
untracked file; about 50,000 files fit.
|
||||
- **64 MiB source cap.** The full size of every such file outside dependency
|
||||
and VCS directories counts, including files over 1 MiB whose contents are
|
||||
withheld from the audit.
|
||||
- **Symlinks.** A tracked symlink anywhere in the repository stops the run.
|
||||
|
||||
**Fix.** No setting raises these caps. Run /cso on a smaller checkout that
|
||||
holds the code you want audited and no symlinks. Add your file count or size to
|
||||
[#2993](https://github.com/garrytan/gstack/issues/2993).
|
||||
|
||||
<a id="conductor-auq-hook-removed"></a>
|
||||
### `removed the AskUserQuestion preference hook: it breaks Conductor's native AskUserQuestion (#2207). ...`
|
||||
|
||||
|
||||
+66
-46
@@ -35,9 +35,10 @@ import {
|
||||
validateFinding,
|
||||
validateVerificationRequest,
|
||||
} from './contracts';
|
||||
import { capture, containedFile, assertSnapshot } from './snapshot';
|
||||
import { capture, containedFile, assertSnapshot, readSnapshotManifest } from './snapshot';
|
||||
import {
|
||||
assertStateOutside,
|
||||
boundedList,
|
||||
event,
|
||||
finalizeReplayTemporary,
|
||||
loadReport,
|
||||
@@ -470,32 +471,42 @@ function planned(scope: string): CoverageRecord[] {
|
||||
evidence: [],
|
||||
}));
|
||||
}
|
||||
// One 256 KiB budget for the per-entry snapshot lists in report.json, split so
|
||||
// material gaps keep the largest share; findings and events keep the rest.
|
||||
const SNAPSHOT_DISCLOSURE_BUDGET = { gaps: 128 * 1024, transformations: 96 * 1024, exclusions: 32 * 1024 };
|
||||
const UNLISTED = "to keep report.json within its 1 MiB bound; the run's snapshot manifest keeps every entry";
|
||||
function snapshotCoverage(manifest: Awaited<ReturnType<typeof capture>>, scope: string): CoverageRecord {
|
||||
const omitted = manifest.entries.filter((entry) => !entry.executionHash),
|
||||
excluded = omitted.filter((entry) => entry.transformation?.startsWith('excluded:'));
|
||||
excluded = omitted.filter((entry) => entry.transformation?.startsWith('excluded:')),
|
||||
excludedSet = new Set(excluded);
|
||||
// Classify every unexplained omission as a material coverage gap. Coverage
|
||||
// must not depend on transformation prose retaining a particular prefix.
|
||||
const unread = omitted.filter((entry) => !excluded.includes(entry));
|
||||
const unread = omitted.filter((entry) => !excludedSet.has(entry));
|
||||
const deleted = manifest.deletedPaths ?? [],
|
||||
captured = manifest.entries.filter((entry) => entry.executionHash).length,
|
||||
captured = manifest.entries.length - omitted.length,
|
||||
missing = unread.length + deleted.length;
|
||||
return {
|
||||
domain: 'snapshot-inputs',
|
||||
scope,
|
||||
status: missing ? (captured ? 'partial' : 'not_assessed') : 'assessed',
|
||||
method: 'fail-closed captured source inventory',
|
||||
gaps: [
|
||||
...unread.map(
|
||||
(entry) =>
|
||||
`${publicSnapshotPath(manifest, entry.path).path}: in-scope source payload was unread and withheld from static and runtime assessment`,
|
||||
),
|
||||
...deleted.map(
|
||||
(item) =>
|
||||
`${publicSnapshotPath(manifest, item.path).path}: tracked source is deleted from the worktree; only retained history is available for assessment`,
|
||||
),
|
||||
],
|
||||
exclusions: excluded.map(
|
||||
(entry) => `${publicSnapshotPath(manifest, entry.path).path}: ${entry.transformation}`,
|
||||
gaps: boundedList(
|
||||
missing,
|
||||
(index) =>
|
||||
index < unread.length
|
||||
? `${publicSnapshotPath(manifest, unread[index].path, unread[index].pathId).path}: in-scope source payload was unread and withheld from static and runtime assessment`
|
||||
: `${publicSnapshotPath(manifest, deleted[index - unread.length].path, deleted[index - unread.length].pathId).path}: tracked source is deleted from the worktree; only retained history is available for assessment`,
|
||||
{ bytes: SNAPSHOT_DISCLOSURE_BUDGET.gaps },
|
||||
4,
|
||||
(count) => `${count} more unread or deleted in-scope inputs are not listed ${UNLISTED}`,
|
||||
),
|
||||
exclusions: boundedList(
|
||||
excluded.length,
|
||||
(index) =>
|
||||
`${publicSnapshotPath(manifest, excluded[index].path, excluded[index].pathId).path}: ${excluded[index].transformation}`,
|
||||
{ bytes: SNAPSHOT_DISCLOSURE_BUDGET.exclusions },
|
||||
4,
|
||||
(count) => `${count} more explicit exclusions are not listed ${UNLISTED}`,
|
||||
),
|
||||
evidence: [
|
||||
`${captured} sanitized execution input${captured === 1 ? '' : 's'} captured; ${excluded.length} explicit non-executable exclusion${excluded.length === 1 ? '' : 's'}; ${unread.length} unread in-scope input${unread.length === 1 ? '' : 's'}; ${deleted.length} tracked deletion${deleted.length === 1 ? '' : 's'}`,
|
||||
@@ -607,6 +618,8 @@ async function start(
|
||||
fs.rmSync(run.dir, { recursive: true, force: true });
|
||||
throw error;
|
||||
}
|
||||
const transformed = manifest.entries.filter((entry) => entry.transformation),
|
||||
deletedPaths = manifest.deletedPaths ?? [];
|
||||
const report: RunReportV3 = {
|
||||
schemaVersion: 3,
|
||||
runId: run.runId,
|
||||
@@ -621,18 +634,25 @@ async function start(
|
||||
snapshotHash: manifest.executionHash,
|
||||
originalHash: manifest.originalHash,
|
||||
baseCommit: manifest.baseCommit,
|
||||
transformations: [
|
||||
...manifest.entries
|
||||
.filter((entry) => entry.transformation)
|
||||
.map((entry) => ({
|
||||
path: publicSnapshotPath(manifest, entry.path).path,
|
||||
handling: entry.transformation!,
|
||||
})),
|
||||
...(manifest.deletedPaths ?? []).map((item) => ({
|
||||
path: publicSnapshotPath(manifest, item.path).path,
|
||||
handling: 'tracked source deleted; retained history only',
|
||||
})),
|
||||
],
|
||||
transformations: boundedList(
|
||||
transformed.length + deletedPaths.length,
|
||||
(index) =>
|
||||
index < transformed.length
|
||||
? {
|
||||
path: publicSnapshotPath(manifest, transformed[index].path, transformed[index].pathId).path,
|
||||
handling: transformed[index].transformation!,
|
||||
}
|
||||
: {
|
||||
path: publicSnapshotPath(manifest, deletedPaths[index - transformed.length].path).path,
|
||||
handling: 'tracked source deleted; retained history only',
|
||||
},
|
||||
{ bytes: SNAPSHOT_DISCLOSURE_BUDGET.transformations },
|
||||
3,
|
||||
(count) => ({
|
||||
path: '[not listed]',
|
||||
handling: `${count} more transformations are not listed ${UNLISTED}`,
|
||||
}),
|
||||
),
|
||||
},
|
||||
application: {
|
||||
actors: [],
|
||||
@@ -655,7 +675,7 @@ async function start(
|
||||
event(
|
||||
report,
|
||||
'snapshot',
|
||||
`Captured ${manifest.entries.length} source entries; ${manifest.entries.filter((e) => e.transformation).length + (manifest.deletedPaths?.length ?? 0)} transformations disclosed`,
|
||||
`Captured ${manifest.entries.length} source entries; ${transformed.length + deletedPaths.length} transformations disclosed`,
|
||||
);
|
||||
if (policy.mode === 'comprehensive') {
|
||||
const plan = inspectPreparation(join(run.dir, 'snapshot'));
|
||||
@@ -958,9 +978,9 @@ function recoveryEvents(dir: string): string[] {
|
||||
function publicSnapshotPath(
|
||||
manifest: SnapshotManifest,
|
||||
path: string,
|
||||
pathId = manifest.entries.find((item) => item.path === path)?.pathId,
|
||||
): { path: string; displayPath?: string } {
|
||||
const entry = manifest.entries.find((item) => item.path === path),
|
||||
handle = snapshotPathHandle(entry?.pathId ?? snapshotPathId(manifest.root, path));
|
||||
const handle = snapshotPathHandle(pathId ?? snapshotPathId(manifest.root, path));
|
||||
let displayPath: string;
|
||||
try {
|
||||
displayPath = redact(path);
|
||||
@@ -974,8 +994,8 @@ function publicSnapshotManifest(manifest: SnapshotManifest): Record<string, unkn
|
||||
...manifest,
|
||||
root: PUBLIC_SOURCE_ROOT,
|
||||
entries: manifest.entries.map((entry) => {
|
||||
const { path, pathId: _, ...rest } = entry;
|
||||
return { ...rest, ...publicSnapshotPath(manifest, path) };
|
||||
const { path, pathId, ...rest } = entry;
|
||||
return { ...rest, ...publicSnapshotPath(manifest, path, pathId) };
|
||||
}),
|
||||
...(manifest.deletedPaths?.length
|
||||
? { deletedPaths: manifest.deletedPaths.map((item) => publicSnapshotPath(manifest, item.path)) }
|
||||
@@ -1059,7 +1079,7 @@ function originalBoundary(report: RunReportV3): {
|
||||
throw new CsoError('INVALID_SCHEMA', 'Recheck evidence requires a linked original finding');
|
||||
const dir = runDirectory(report.parent.runId),
|
||||
original = loadReport(dir),
|
||||
manifest = readJson(join(dir, 'snapshot.json')) as SnapshotManifest,
|
||||
manifest = readSnapshotManifest(dir),
|
||||
finding = original.findings.find((item) => item.id === report.parent!.findingId);
|
||||
if (report.repoId !== original.repoId)
|
||||
throw new CsoError('INCOMPATIBLE_INPUT', 'Recheck repository identity differs from the original audit');
|
||||
@@ -1295,7 +1315,7 @@ function submit(args: string[]) {
|
||||
const input = rawInput as SubmissionV3;
|
||||
return withLock(dir, () => {
|
||||
const report = loadReport(dir),
|
||||
manifest = readJson(join(dir, 'snapshot.json'));
|
||||
manifest = readSnapshotManifest(dir);
|
||||
assertSnapshot(dir, manifest);
|
||||
requireReportingTime(report);
|
||||
if (report.status !== 'running')
|
||||
@@ -1401,7 +1421,7 @@ function finish(args: string[]) {
|
||||
if (args.length) throw new CsoError('INVALID_ARGUMENT', 'finish takes only a run ID');
|
||||
return withLock(dir, () => {
|
||||
const report = loadReport(dir),
|
||||
manifest = readJson(join(dir, 'snapshot.json'));
|
||||
manifest = readSnapshotManifest(dir);
|
||||
assertSnapshot(dir, manifest);
|
||||
reclaimDeadAttemptScratch(dir);
|
||||
for (const c of report.coverage)
|
||||
@@ -1478,7 +1498,7 @@ async function inspect(args: string[]) {
|
||||
const { dir } = run(args);
|
||||
if (args.length) throw new CsoError('INVALID_ARGUMENT', 'inspect takes one run ID');
|
||||
const report = loadReport(dir),
|
||||
manifest = readJson(join(dir, 'snapshot.json')) as SnapshotManifest;
|
||||
manifest = readSnapshotManifest(dir);
|
||||
assertSnapshot(dir, manifest);
|
||||
const rawSensitive = readJson(join(dir, 'sensitive-evidence.json')),
|
||||
sensitiveEvidence = Array.isArray(rawSensitive)
|
||||
@@ -1508,7 +1528,7 @@ async function inspect(args: string[]) {
|
||||
async function read(args: string[]) {
|
||||
const { dir } = run(args);
|
||||
if (args.length !== 1) throw new CsoError('INVALID_ARGUMENT', 'read requires one path or opaque handle');
|
||||
const manifest = readJson(join(dir, 'snapshot.json')) as SnapshotManifest;
|
||||
const manifest = readSnapshotManifest(dir);
|
||||
assertSnapshot(dir, manifest);
|
||||
const selected = resolveSnapshotPath(manifest, args[0], true),
|
||||
full = containedFile(join(dir, 'readable'), selected.path),
|
||||
@@ -1517,7 +1537,7 @@ async function read(args: string[]) {
|
||||
}
|
||||
async function history(args: string[]) {
|
||||
const { dir } = run(args),
|
||||
manifest = readJson(join(dir, 'snapshot.json')) as SnapshotManifest;
|
||||
manifest = readSnapshotManifest(dir);
|
||||
assertSnapshot(dir, manifest);
|
||||
let selected: string | undefined;
|
||||
if (args.length) selected = resolveSnapshotPath(manifest, args.shift(), false, 'History path', true).path;
|
||||
@@ -1540,7 +1560,7 @@ function resume(args: string[]) {
|
||||
if (args.length) throw new CsoError('INVALID_ARGUMENT', 'resume takes one run ID');
|
||||
return withLock(dir, () => {
|
||||
const report = loadReport(dir),
|
||||
manifest = readJson(join(dir, 'snapshot.json'));
|
||||
manifest = readSnapshotManifest(dir);
|
||||
if (report.status === 'finished')
|
||||
throw new CsoError('INVALID_SCHEMA', 'A finished audit cannot be resumed');
|
||||
assertSnapshot(dir, manifest);
|
||||
@@ -1677,7 +1697,7 @@ async function scanner(args: string[], sarif = false) {
|
||||
id,
|
||||
runId: report.runId,
|
||||
runDir: dir,
|
||||
manifest: readJson(join(dir, 'snapshot.json')),
|
||||
manifest: readSnapshotManifest(dir),
|
||||
policy: report.policy,
|
||||
executionDeadline: Date.parse(report.deadline) - 60_000,
|
||||
platform: platform(),
|
||||
@@ -1828,7 +1848,7 @@ async function recheck(args: string[], dependencies: CsoCliDependencies) {
|
||||
throw new CsoError('INVALID_SCHEMA', 'Recheck requires a finished original audit');
|
||||
// Keep the original immutable while the fresh snapshot is captured and
|
||||
// until its child lineage report has been durably published.
|
||||
const oldManifest = readJson(join(originalDir, 'snapshot.json'));
|
||||
const oldManifest = readSnapshotManifest(originalDir);
|
||||
const preserveBase = original.policy.diff || Boolean(original.source.baseCommit),
|
||||
report = await start(
|
||||
[
|
||||
@@ -1928,7 +1948,7 @@ function testPlan(args: string[]) {
|
||||
if (args.length !== 1 || !['node', 'bun', 'python', 'rails'].includes(args[0]))
|
||||
throw new CsoError('INVALID_ARGUMENT', 'test-plan requires one supported stack');
|
||||
const stack = args[0] as 'node' | 'bun' | 'python' | 'rails',
|
||||
manifest = readJson(join(dir, 'snapshot.json')) as SnapshotManifest;
|
||||
manifest = readSnapshotManifest(dir);
|
||||
assertSnapshot(dir, manifest);
|
||||
const preparation = inspectPreparation(join(dir, 'snapshot'), stack);
|
||||
if (preparation.status !== 'ready')
|
||||
@@ -1953,7 +1973,7 @@ function runtimePlan(args: string[]) {
|
||||
const port = Number(rawPort);
|
||||
if (!Number.isInteger(port) || port < 1024 || port > 65535)
|
||||
throw new CsoError('INVALID_ARGUMENT', '--port must be an integer from 1024 to 65535');
|
||||
const manifest = readJson(join(dir, 'snapshot.json')) as SnapshotManifest;
|
||||
const manifest = readSnapshotManifest(dir);
|
||||
assertSnapshot(dir, manifest);
|
||||
const preparation = inspectPreparation(join(dir, 'snapshot'), stack);
|
||||
if (preparation.status !== 'ready')
|
||||
@@ -2186,7 +2206,7 @@ async function verify(args: string[], dependencies: CsoCliDependencies) {
|
||||
request = validateVerificationRequest(raw);
|
||||
return await withLock(dir, async () => {
|
||||
const report = loadReport(dir),
|
||||
manifest = readJson(join(dir, 'snapshot.json')) as SnapshotManifest;
|
||||
manifest = readSnapshotManifest(dir);
|
||||
assertSnapshot(dir, manifest);
|
||||
requireTime(report);
|
||||
if (report.policy.mode !== 'comprehensive' || report.status !== 'running')
|
||||
@@ -2510,7 +2530,7 @@ async function replay(args: string[], dependencies: CsoCliDependencies) {
|
||||
throw new CsoError('INCOMPATIBLE_INPUT', 'Supplied source does not match the bundle input hashes');
|
||||
};
|
||||
if (fs.existsSync(retained)) {
|
||||
manifest = readJson(join(stored.dir, 'snapshot.json'));
|
||||
manifest = readSnapshotManifest(stored.dir);
|
||||
const expiresAt =
|
||||
typeof manifest?.expiresAt === 'string' ? Date.parse(manifest.expiresAt) : Number.NaN;
|
||||
if (!Number.isFinite(expiresAt) || new Date(expiresAt).toISOString() !== manifest.expiresAt)
|
||||
|
||||
+5
-2
@@ -521,6 +521,8 @@ export function commandTimeoutMs(
|
||||
Math.min(phase === 'preparation' ? PREPARATION_COMMAND_TIMEOUT_MS : COMMAND_TIMEOUT_MS, deadline - now),
|
||||
);
|
||||
}
|
||||
/** Raw output is never printed or persisted, so a caller may opt in to more than MAX_OUTPUT (snapshot path listings). */
|
||||
const RAW_OUTPUT_LIMIT = 16 * 1024 * 1024;
|
||||
export async function runProcess(
|
||||
file: string,
|
||||
args: string[],
|
||||
@@ -546,7 +548,7 @@ export async function runProcess(
|
||||
throw new CsoError('INVALID_ARGUMENT', 'Invalid child argument');
|
||||
const hardened = hardenGit(file, args);
|
||||
args = hardened.args;
|
||||
const cap = Math.min(opts.maxBytes ?? MAX_OUTPUT, MAX_OUTPUT);
|
||||
const cap = Math.min(opts.maxBytes ?? MAX_OUTPUT, opts.raw ? RAW_OUTPUT_LIMIT : MAX_OUTPUT);
|
||||
return new Promise((resolve, reject) => {
|
||||
const child = spawn(file, args, {
|
||||
cwd: opts.cwd,
|
||||
@@ -640,12 +642,13 @@ export async function runProcess(
|
||||
child.stdin.end(opts.input);
|
||||
});
|
||||
}
|
||||
export async function git(repo: string, args: string[], home: string): Promise<string> {
|
||||
export async function git(repo: string, args: string[], home: string, maxBytes?: number): Promise<string> {
|
||||
const result = await runProcess(executable('git'), ['--no-optional-locks', '-C', repo, ...args], {
|
||||
cwd: home,
|
||||
env: childEnvironment(home),
|
||||
raw: true,
|
||||
timeoutMs: 15_000,
|
||||
maxBytes,
|
||||
});
|
||||
if (result.code || result.timedOut || result.truncated) {
|
||||
// Git stderr and argv can contain repository paths, refs, and configured
|
||||
|
||||
+68
-11
@@ -15,7 +15,7 @@ import {
|
||||
MAX_OUTPUT,
|
||||
} from './contracts';
|
||||
import { childEnvironment, executable, git, redact, runProcess } from './process';
|
||||
import { secureDirectory, writeHelperJson, writeJson } from './state';
|
||||
import { boundedList, readJson, secureDirectory, writeHelperJson, writeJson } from './state';
|
||||
import { scan } from '../redact-engine';
|
||||
import { atomicWriteSync } from '../fs-atomic';
|
||||
|
||||
@@ -52,6 +52,22 @@ function omittedPath(path: string): boolean {
|
||||
const SECRET_FILE =
|
||||
/(?:^|\/)(?:\.env(?:\..*)?|\.npmrc|\.yarnrc(?:\.yml)?|\.pypirc|pip\.conf|credentials(?:\.yml(?:\.enc)?)?|master\.key|id_(?:rsa|ed25519)|.*\.(?:pem|p12|pfx|key)|AGENTS\.md|CLAUDE\.md|GEMINI\.md|bunfig\.toml)$/i;
|
||||
const SOURCE_LIMIT = 64 * 1024 * 1024;
|
||||
/** snapshot.json holds one entry per source file, so it gets its own cap; other private state stays at 1 MiB. */
|
||||
export const SNAPSHOT_MANIFEST_LIMIT = 16 * 1024 * 1024;
|
||||
const SENSITIVE_EVIDENCE_BUDGET = 512 * 1024;
|
||||
const CAPACITY_ISSUE = 'https://github.com/garrytan/gstack/issues/2993';
|
||||
const COUNTED_SOURCE =
|
||||
'every tracked or nonignored untracked file outside dependency and VCS directories (node_modules, .git, .venv, vendor/bundle and similar)';
|
||||
const mib = (bytes: number) => `${(bytes / 1024 / 1024).toFixed(1)} MiB`;
|
||||
export function snapshotManifestCapMessage(entries: number, bytes: number): string {
|
||||
return `Snapshot manifest cap exceeded: ${entries} source entries need a ${mib(bytes)} (${bytes} byte) manifest; the snapshot manifest cap is ${mib(SNAPSHOT_MANIFEST_LIMIT)}. One entry is recorded for ${COUNTED_SOURCE}. Next step: run /cso on a smaller checkout of the code you want audited, and add your entry count to ${CAPACITY_ISSUE}. No supported workaround raises this cap.`;
|
||||
}
|
||||
export function sourceCapMessage(bytes: number, files: number): string {
|
||||
return `64 MiB source cap exceeded: ${files} source files hold about ${mib(bytes)} (${bytes} bytes); the source admission cap is ${mib(SOURCE_LIMIT)}. The full size of ${COUNTED_SOURCE} counts, including files over 1 MiB whose payloads are withheld from the audit. Next step: run /cso on a smaller checkout of the code you want audited, and add your measured size to ${CAPACITY_ISSUE}. No supported workaround raises this cap.`;
|
||||
}
|
||||
export function readSnapshotManifest(runDir: string): SnapshotManifest {
|
||||
return readJson(join(runDir, 'snapshot.json'), SNAPSHOT_MANIFEST_LIMIT) as SnapshotManifest;
|
||||
}
|
||||
const SNAPSHOT_ENTRY_LIMIT = 100_000;
|
||||
const GIT_POINTER_LIMIT = 8192;
|
||||
export interface SnapshotCaptureLimits {
|
||||
@@ -447,9 +463,19 @@ async function paths(
|
||||
// for, while still collecting nonignored untracked source.
|
||||
admission.time();
|
||||
const [working, head] = await Promise.all([
|
||||
git(repo, ['ls-files', '--cached', '--others', '--exclude-standard', '-z'], home),
|
||||
git(
|
||||
repo,
|
||||
['ls-files', '--cached', '--others', '--exclude-standard', '-z'],
|
||||
home,
|
||||
SNAPSHOT_MANIFEST_LIMIT,
|
||||
),
|
||||
headCommit
|
||||
? git(repo, ['ls-tree', '-r', '-z', '--name-only', '--full-tree', headCommit, '--'], home)
|
||||
? git(
|
||||
repo,
|
||||
['ls-tree', '-r', '-z', '--name-only', '--full-tree', headCommit, '--'],
|
||||
home,
|
||||
SNAPSHOT_MANIFEST_LIMIT,
|
||||
)
|
||||
: Promise.resolve(''),
|
||||
]),
|
||||
seen = new Set<string>();
|
||||
@@ -478,6 +504,7 @@ async function rejectSpecialFiles(
|
||||
repo,
|
||||
['ls-files', '--others', '--ignored', '--exclude-standard', '--directory', '-z'],
|
||||
home,
|
||||
SNAPSHOT_MANIFEST_LIMIT,
|
||||
),
|
||||
ignoredDirectories = new Set<string>();
|
||||
admission.time();
|
||||
@@ -611,10 +638,23 @@ export async function capture(
|
||||
manifest.baseCommit = (await git(repo, ['rev-parse', '--verify', `${base}^{commit}`], home)).trim();
|
||||
guard();
|
||||
}
|
||||
const sourceCapError = () => {
|
||||
let measured = 0,
|
||||
files = 0;
|
||||
for (const candidate of list) {
|
||||
if (noReadPath(candidate)) continue;
|
||||
try {
|
||||
measured += fs.lstatSync(join(repo, candidate)).size;
|
||||
files++;
|
||||
} catch {}
|
||||
}
|
||||
return new CsoError('MISSING_INPUT', sourceCapMessage(Math.max(measured, total), files));
|
||||
};
|
||||
for (const path of list) {
|
||||
guard();
|
||||
let size: number;
|
||||
try {
|
||||
fs.lstatSync(join(repo, path));
|
||||
size = fs.lstatSync(join(repo, path)).size;
|
||||
} catch (error: any) {
|
||||
if (error?.code === 'ENOENT') {
|
||||
absentPaths.add(path);
|
||||
@@ -640,11 +680,11 @@ export async function capture(
|
||||
guard();
|
||||
continue;
|
||||
}
|
||||
if (total + size > SOURCE_LIMIT) throw sourceCapError();
|
||||
const { data, mode } = readStable(repo, path, SOURCE_LIMIT);
|
||||
total += data.length;
|
||||
guard();
|
||||
if (total > SOURCE_LIMIT)
|
||||
throw new CsoError('MISSING_INPUT', 'Source exceeds the 64 MiB snapshot admission limit');
|
||||
if (total > SOURCE_LIMIT) throw sourceCapError();
|
||||
const entry: SnapshotEntry = {
|
||||
path,
|
||||
pathId: snapshotPathId(repo, path),
|
||||
@@ -749,7 +789,12 @@ export async function capture(
|
||||
canonical(entries.filter((e) => e.executionHash).map((e) => [e.path, e.executionHash, e.mode])),
|
||||
);
|
||||
if (manifest.baseCommit) {
|
||||
const tree = await git(repo, ['ls-tree', '-r', '-z', '--full-tree', manifest.baseCommit, '--'], home),
|
||||
const tree = await git(
|
||||
repo,
|
||||
['ls-tree', '-r', '-z', '--full-tree', manifest.baseCommit, '--'],
|
||||
home,
|
||||
SNAPSHOT_MANIFEST_LIMIT,
|
||||
),
|
||||
baseFiles = new Map<string, { hash: string; mode: string }>();
|
||||
guard();
|
||||
for (const row of tree.split('\0').filter(Boolean)) {
|
||||
@@ -858,15 +903,27 @@ export async function capture(
|
||||
guard();
|
||||
assertAbsent();
|
||||
admission.time();
|
||||
writeHelperJson(join(runDir, 'sensitive-evidence.json'), sensitiveEvidence);
|
||||
writeHelperJson(
|
||||
join(runDir, 'sensitive-evidence.json'),
|
||||
boundedList<Record<string, unknown>>(
|
||||
sensitiveEvidence.length,
|
||||
(index) => sensitiveEvidence[index],
|
||||
{ bytes: SENSITIVE_EVIDENCE_BUDGET },
|
||||
1,
|
||||
(omitted) => ({
|
||||
omitted,
|
||||
note: `${omitted} more files with sensitive-pattern findings are not listed, to keep this artifact within its 1 MiB bound`,
|
||||
}),
|
||||
),
|
||||
);
|
||||
// The manifest contains helper-computed identities and source pathnames but
|
||||
// never source payloads. Persist it exactly in private state: generic
|
||||
// content redaction would silently break the path/hash identity relation.
|
||||
const serialized = JSON.stringify(manifest, null, 2);
|
||||
if (Buffer.byteLength(serialized) + 1 > MAX_OUTPUT)
|
||||
const serialized = JSON.stringify(manifest);
|
||||
if (Buffer.byteLength(serialized) + 1 > SNAPSHOT_MANIFEST_LIMIT)
|
||||
throw new CsoError(
|
||||
'MISSING_INPUT',
|
||||
'Snapshot manifest exceeds the 1 MiB private-state admission limit',
|
||||
snapshotManifestCapMessage(entries.length, Buffer.byteLength(serialized) + 1),
|
||||
);
|
||||
atomicWriteSync(join(runDir, 'snapshot.json'), serialized + '\n', { mode: 0o600 });
|
||||
return manifest;
|
||||
|
||||
+35
-5
@@ -814,7 +814,7 @@ export function writeJsonExclusive(path: string, value: unknown): void {
|
||||
throw new CsoError('PERSISTENCE_FAILED', 'Private immutable artifact could not be written');
|
||||
}
|
||||
}
|
||||
function readPrivateJson(path: string): unknown {
|
||||
function readPrivateJson(path: string, maxBytes: number): unknown {
|
||||
let fd: number | undefined;
|
||||
try {
|
||||
const before = exactLstat(path);
|
||||
@@ -823,7 +823,7 @@ function readPrivateJson(path: string): unknown {
|
||||
before.isSymbolicLink() ||
|
||||
before.nlink !== 1 ||
|
||||
before.size <= 0 ||
|
||||
before.size > MAX_STATE_FILE ||
|
||||
before.size > maxBytes ||
|
||||
(process.getuid && before.uid !== process.getuid()) ||
|
||||
(process.platform !== 'win32' && (before.mode & 0o077) !== 0)
|
||||
)
|
||||
@@ -857,16 +857,46 @@ function readPrivateJson(path: string): unknown {
|
||||
} catch {}
|
||||
}
|
||||
}
|
||||
export function readJson(path: string): any {
|
||||
/** maxBytes is a per-artifact cap; only the snapshot manifest passes a larger one. */
|
||||
export function readJson(path: string, maxBytes = MAX_STATE_FILE): any {
|
||||
try {
|
||||
secureDirectory(dirname(path));
|
||||
recoverAtomicNoReplaceJson(path, { label: 'Private immutable artifact', maxBytes: MAX_STATE_FILE });
|
||||
return readPrivateJson(path);
|
||||
recoverAtomicNoReplaceJson(path, { label: 'Private immutable artifact', maxBytes });
|
||||
return readPrivateJson(path, maxBytes);
|
||||
} catch (e) {
|
||||
if (e instanceof CsoError) throw e;
|
||||
throw new CsoError('MISSING_INPUT', 'Private state file is missing or invalid');
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Keep the leading items of a per-entry list that fit `budget.bytes`, measured
|
||||
* as pretty-printed JSON nested `depth` levels deep, and replace the rest with
|
||||
* one summary item carrying the omitted count. A shared budget object bounds
|
||||
* several lists in one artifact together.
|
||||
*/
|
||||
export function boundedList<T>(
|
||||
total: number,
|
||||
at: (index: number) => T,
|
||||
budget: { bytes: number },
|
||||
depth: number,
|
||||
summary: (omitted: number) => T,
|
||||
): T[] {
|
||||
const indent = ' '.repeat(2 * depth),
|
||||
size = (item: T) =>
|
||||
Buffer.byteLength(indent + JSON.stringify(item, null, 2).replaceAll('\n', '\n' + indent) + ',\n'),
|
||||
reserve = size(summary(total)),
|
||||
kept: T[] = [];
|
||||
for (let index = 0; index < total; index++) {
|
||||
const item = at(index),
|
||||
bytes = size(item);
|
||||
if (index < total - 1 ? bytes + reserve > budget.bytes : bytes > budget.bytes) break;
|
||||
kept.push(item);
|
||||
budget.bytes -= bytes;
|
||||
}
|
||||
if (kept.length === total) return kept;
|
||||
budget.bytes -= reserve;
|
||||
return [...kept, summary(total - kept.length)];
|
||||
}
|
||||
export const PUBLIC_SOURCE_ROOT = '<REDACTED-internal.user_path>';
|
||||
/** A report is public evidence; the real root remains in the private snapshot. */
|
||||
export function publicReport(report: RunReportV3): RunReportV3 {
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gstack",
|
||||
"version": "1.91.38",
|
||||
"version": "1.91.42",
|
||||
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
|
||||
@@ -0,0 +1,276 @@
|
||||
// /cso file-count ceiling (#3068, manifest half of #2993): snapshot.json has its own
|
||||
// 16 MiB cap and compact form, every per-entry list in a 1 MiB private artifact is
|
||||
// bounded with an omitted count, and both capacity errors say what to do next.
|
||||
import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { MAX_OUTPUT } from '../lib/cso/contracts';
|
||||
import {
|
||||
SNAPSHOT_MANIFEST_LIMIT,
|
||||
capture,
|
||||
readSnapshotManifest,
|
||||
snapshotManifestCapMessage,
|
||||
sourceCapMessage,
|
||||
} from '../lib/cso/snapshot';
|
||||
import { boundedList, newRun, readJson } from '../lib/cso/state';
|
||||
import { runProcess } from '../lib/cso/process';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const launcher = path.join(ROOT, 'bin', process.platform === 'win32' ? 'gstack-cso-launcher.exe' : 'gstack-cso-launcher');
|
||||
const MIB = 1024 * 1024;
|
||||
const ISSUE = 'https://github.com/garrytan/gstack/issues/2993';
|
||||
const KEY = ['pass', 'word'].join('');
|
||||
let root = '';
|
||||
let state = '';
|
||||
const originalHome = process.env.GSTACK_HOME;
|
||||
|
||||
function git(repo: string, ...args: string[]): string {
|
||||
const result = spawnSync('/usr/bin/git', ['-C', repo, ...args], { encoding: 'utf8', env: { HOME: root, PATH: '/usr/bin:/bin' }, timeout: 60_000 });
|
||||
if (result.status) throw new Error(result.stderr);
|
||||
return result.stdout;
|
||||
}
|
||||
function cso(repo: string, args: string[]) {
|
||||
return spawnSync(launcher, args, { cwd: repo, encoding: 'utf8', env: { HOME: root, GSTACK_HOME: state, PATH: '/usr/bin:/bin' }, timeout: 120_000, maxBuffer: 64 * MIB });
|
||||
}
|
||||
function makeRepo(name: string, files: Record<string, string | Buffer>): string {
|
||||
const repo = path.join(root, name);
|
||||
fs.mkdirSync(repo);
|
||||
git(repo, 'init', '-q');
|
||||
git(repo, 'config', 'user.email', 'fixture@example.test');
|
||||
git(repo, 'config', 'user.name', 'Fixture');
|
||||
for (const [file, body] of Object.entries(files)) {
|
||||
fs.mkdirSync(path.dirname(path.join(repo, file)), { recursive: true });
|
||||
fs.writeFileSync(path.join(repo, file), body);
|
||||
}
|
||||
git(repo, 'add', '-A');
|
||||
git(repo, 'commit', '-qm', 'fixture');
|
||||
return repo;
|
||||
}
|
||||
function runDir(run: { repoId: string; runId: string }): string {
|
||||
return path.join(state, 'security', 'cso', run.repoId, run.runId);
|
||||
}
|
||||
function writeInput(name: string, value: unknown): string {
|
||||
const file = path.join(root, name);
|
||||
fs.writeFileSync(file, JSON.stringify(value));
|
||||
return file;
|
||||
}
|
||||
const application = { actors: ['tenant user'], assets: ['tenant records'], entrypoints: ['GET /users/:id'], tenantBoundaries: ['tenant id'], sensitiveOperations: ['record read'], invariants: ['tenant isolation'] };
|
||||
const finding = { title: 'Cross-tenant user read', rootCause: 'Tenant query omits caller tenant predicate', location: { path: 'src/users.ts', line: 1, symbol: 'tenantQuery' }, advisoryIds: [], severity: 'high', confidence: 'high', confidenceRationale: 'The caller-to-query trace and missing tenant predicate directly support the finding', evidence: 'supported', attackerControl: 'Authenticated caller chooses the record ID', impact: 'Another tenant record is returned', scenario: 'A tenant supplies a known record ID owned by another tenant and receives that record', trace: ['GET /users/:id', 'tenantQuery', 'findUnique by id'], references: ['src/users.ts:1', 'OWASP API1:2023'], recommendation: 'Bind the lookup predicate to the authenticated tenant identifier', challenge: { reviewer: 'independent-2', independent: true, mode: 'independent_agent', callers: 'Authenticated route forwards the ID', controls: 'Authentication does not bind tenant', counterevidence: 'Opaque IDs reduce guessing but do not authorize', conclusion: 'The authorization invariant is absent' } };
|
||||
function completeEvidence(report: any, findings: unknown[]) {
|
||||
return { application, findings, coverage: report.coverage.filter((item: any) => !['snapshot-inputs', 'history-inputs'].includes(item.domain)).map((item: any) => ({ ...item, status: 'assessed', method: 'fresh caller and boundary trace', gaps: [], evidence: ['current captured source'] })), gaps: [] };
|
||||
}
|
||||
const USERS = 'export const tenantQuery = (id:string) => db.user.findUnique({where:{id}})\n';
|
||||
// Long nested paths where most files are excluded or withheld and many carry
|
||||
// scanner hits: the shape that overflowed report.json and sensitive-evidence.json.
|
||||
function hostileFiles(count: number): Record<string, string | Buffer> {
|
||||
const files: Record<string, string | Buffer> = { 'src/users.ts': USERS };
|
||||
for (let i = 1; i < count; i++) {
|
||||
const dir = `packages/very-long-workspace-package-name-${i % 50}/src/components/deeply/nested/feature-area-${i % 17}`;
|
||||
switch (i % 5) {
|
||||
case 0:
|
||||
files[`${dir}/module_file_number_${i}.ts`] = Array.from({ length: 12 }, (_, k) => `${KEY} = "Sup3rS3cretValue${i}x${k}"\n`).join('');
|
||||
break;
|
||||
case 1:
|
||||
files[`node_modules/${dir}/dependency_file_number_${i}.js`] = `module.exports=${i}\n`;
|
||||
break;
|
||||
case 2:
|
||||
files[`.claude/${dir}/agent_config_file_number_${i}.md`] = `cfg ${i}\n`;
|
||||
break;
|
||||
case 3:
|
||||
files[`${dir}/binary_asset_number_${i}.bin`] = Buffer.from(`a\0b${i}`);
|
||||
break;
|
||||
default:
|
||||
files[`${dir}/certificate_number_${i}.pem`] = `not a key ${i}\n`;
|
||||
}
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
||||
beforeAll(() => {
|
||||
root = fs.mkdtempSync(path.join(os.tmpdir(), 'cso-ceiling-'));
|
||||
state = path.join(root, 'state');
|
||||
process.env.GSTACK_HOME = state;
|
||||
});
|
||||
afterAll(() => {
|
||||
if (originalHome === undefined) delete process.env.GSTACK_HOME;
|
||||
else process.env.GSTACK_HOME = originalHome;
|
||||
fs.rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('CSO snapshot manifest cap', () => {
|
||||
test('the snapshot cap is local to snapshot.json; shared limits stay 1 MiB', () => {
|
||||
expect(MAX_OUTPUT).toBe(MIB);
|
||||
expect(SNAPSHOT_MANIFEST_LIMIT).toBe(16 * MIB);
|
||||
const dir = fs.mkdtempSync(path.join(root, 'caps-'));
|
||||
fs.chmodSync(dir, 0o700);
|
||||
const big = JSON.stringify({ padding: 'x'.repeat(MIB + 16) });
|
||||
for (const name of ['report.json', 'snapshot.json']) fs.writeFileSync(path.join(dir, name), big, { mode: 0o600 });
|
||||
expect(() => readJson(path.join(dir, 'report.json'))).toThrow('Invalid private state file');
|
||||
expect(() => readJson(path.join(dir, 'snapshot.json'))).toThrow('Invalid private state file');
|
||||
expect(readSnapshotManifest(dir)).toEqual(JSON.parse(big));
|
||||
fs.writeFileSync(path.join(dir, 'snapshot.json'), JSON.stringify({ padding: 'x'.repeat(16 * MIB) }), { mode: 0o600 });
|
||||
expect(() => readSnapshotManifest(dir)).toThrow('Invalid private state file');
|
||||
});
|
||||
|
||||
test('every snapshot.json read in the helper goes through readSnapshotManifest', () => {
|
||||
const files = fs.readdirSync(path.join(ROOT, 'lib', 'cso')).filter((file) => file.endsWith('.ts'));
|
||||
for (const file of files) {
|
||||
const source = fs.readFileSync(path.join(ROOT, 'lib', 'cso', file), 'utf8');
|
||||
expect(source).not.toMatch(/readJson\(\s*join\([^)]*'snapshot\.json'\)\s*\)/);
|
||||
}
|
||||
const cli = fs.readFileSync(path.join(ROOT, 'lib', 'cso', 'cli.ts'), 'utf8');
|
||||
expect(cli.match(/readSnapshotManifest\(/g)?.length).toBeGreaterThanOrEqual(13);
|
||||
const snapshot = fs.readFileSync(path.join(ROOT, 'lib', 'cso', 'snapshot.ts'), 'utf8');
|
||||
expect(snapshot).toContain("readJson(join(runDir, 'snapshot.json'), SNAPSHOT_MANIFEST_LIMIT)");
|
||||
expect(snapshot).toContain('> SNAPSHOT_MANIFEST_LIMIT');
|
||||
expect(snapshot).not.toContain('JSON.stringify(manifest, null, 2)');
|
||||
});
|
||||
|
||||
test('both capacity errors name the bound cap, measured value, what counts, next step and #2993', () => {
|
||||
const manifest = snapshotManifestCapMessage(61_234, 17_000_000);
|
||||
expect(manifest).toStartWith('Snapshot manifest cap exceeded:');
|
||||
for (const part of ['61234 source entries', '17000000 byte', 'cap is 16.0 MiB', 'One entry is recorded for every tracked', 'Next step:', ISSUE, 'No supported workaround'])
|
||||
expect(manifest).toContain(part);
|
||||
const source = sourceCapMessage(346_199_207, 7_561);
|
||||
expect(source).toStartWith('64 MiB source cap exceeded:');
|
||||
for (const part of ['7561 source files', '346199207 bytes', 'cap is 64.0 MiB', 'files over 1 MiB whose payloads are withheld', 'Next step:', ISSUE, 'No supported workaround'])
|
||||
expect(source).toContain(part);
|
||||
});
|
||||
|
||||
test('the 64 MiB source cap still fails closed, with the actionable message on stderr', () => {
|
||||
const repo = makeRepo('over-source-cap', { 'src/users.ts': USERS, 'assets/a.bin': Buffer.alloc(40 * MIB, 1), 'assets/b.bin': Buffer.alloc(25 * MIB, 1) });
|
||||
const result = cso(repo, ['start', '--repo', repo, '--offline']);
|
||||
expect(result.status).not.toBe(0);
|
||||
expect(result.stdout).toBe('');
|
||||
const error = JSON.parse(result.stderr).error;
|
||||
expect(error.code).toBe('MISSING_INPUT');
|
||||
expect(error.message).toStartWith('64 MiB source cap exceeded: 3 source files hold about 65.0 MiB');
|
||||
expect(error.message).toContain(ISSUE);
|
||||
expect(error.message).toContain('No supported workaround');
|
||||
}, 120_000);
|
||||
});
|
||||
|
||||
describe('CSO bounded per-entry lists', () => {
|
||||
test('keeps leading items, appends one omitted-count summary, and leaves short lists untouched', () => {
|
||||
const items = Array.from({ length: 1000 }, (_, i) => `entry-${i}`);
|
||||
expect(boundedList(3, (i) => items[i], { bytes: 4096 }, 2, (n) => `${n} omitted`)).toEqual(items.slice(0, 3));
|
||||
const budget = { bytes: 2048 },
|
||||
kept = boundedList(items.length, (i) => items[i], budget, 2, (n) => `${n} omitted`);
|
||||
expect(kept.length).toBeLessThan(items.length);
|
||||
expect(kept.at(-1)).toBe(`${items.length - kept.length + 1} omitted`);
|
||||
expect(kept.slice(0, -1)).toEqual(items.slice(0, kept.length - 1));
|
||||
expect(budget.bytes).toBeGreaterThanOrEqual(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('CSO above the old file-count ceiling', () => {
|
||||
test('a 2,500-file repository still captures and round-trips', async () => {
|
||||
const files: Record<string, string> = {};
|
||||
for (let i = 1; i <= 2500; i++) files[`src/pkg/sub/file_number_${i}.ts`] = `export const v${i} = ${i};\n`;
|
||||
const repo = makeRepo('plain-2500', files),
|
||||
run = newRun(repo),
|
||||
manifest = await capture(repo, run.dir);
|
||||
expect(manifest.entries).toHaveLength(2500);
|
||||
expect(readSnapshotManifest(run.dir)).toEqual(manifest);
|
||||
}, 120_000);
|
||||
|
||||
test('Git path listings over 1 MiB are admitted, with and without a comparison base', async () => {
|
||||
const files: Record<string, string> = {},
|
||||
dir = Array.from({ length: 6 }, (_, i) => `workspace-segment-with-a-long-descriptive-name-${i}`).join('/');
|
||||
for (let i = 1; i <= 5000; i++) files[`${dir}/file_number_${i}.ts`] = `export const v${i} = ${i};\n`;
|
||||
const repo = makeRepo('long-listing', files);
|
||||
expect(Buffer.byteLength(git(repo, 'ls-files', '-z'))).toBeGreaterThan(MIB);
|
||||
const plain = await capture(repo, newRun(repo).dir);
|
||||
expect(plain.entries).toHaveLength(5000);
|
||||
const based = await capture(repo, newRun(repo).dir, 'HEAD');
|
||||
expect(based.entries).toHaveLength(5000);
|
||||
expect(based.changedPaths).toEqual([]);
|
||||
}, 180_000);
|
||||
|
||||
test('only raw callers can opt in to output above MAX_OUTPUT', async () => {
|
||||
const cwd = fs.mkdtempSync(path.join(root, 'raw-')),
|
||||
script = ['-c', 'head -c 1200000 /dev/zero | tr "\\0" a'],
|
||||
env = { PATH: '/usr/bin:/bin' };
|
||||
expect((await runProcess('/bin/sh', script, { cwd, env, maxBytes: 2 * MIB })).truncated).toBe(true);
|
||||
expect((await runProcess('/bin/sh', script, { cwd, env, raw: true })).truncated).toBe(true);
|
||||
const raw = await runProcess('/bin/sh', script, { cwd, env, raw: true, maxBytes: 2 * MIB });
|
||||
expect(raw.truncated).toBe(false);
|
||||
expect(raw.stdout).toHaveLength(1_200_000);
|
||||
});
|
||||
|
||||
test('5,000 long nested paths, mostly excluded or withheld with many scanner hits, run start to final report and recheck', () => {
|
||||
const repo = makeRepo('hostile-5000', hostileFiles(5000)),
|
||||
started = cso(repo, ['start', '--repo', repo, '--scope', 'auth', '--offline']);
|
||||
expect(started.stderr).toBe('');
|
||||
expect(started.status).toBe(0);
|
||||
expect(Buffer.byteLength(started.stdout)).toBeLessThan(MIB);
|
||||
const run = JSON.parse(started.stdout),
|
||||
dir = runDir(run);
|
||||
|
||||
const rawManifest = fs.readFileSync(path.join(dir, 'snapshot.json'), 'utf8'),
|
||||
manifest = readSnapshotManifest(dir);
|
||||
expect(Buffer.byteLength(rawManifest)).toBeGreaterThan(MIB);
|
||||
expect(rawManifest).toBe(JSON.stringify(manifest) + '\n');
|
||||
expect(manifest.entries).toHaveLength(5000);
|
||||
|
||||
const evidence = readJson(path.join(dir, 'sensitive-evidence.json'));
|
||||
expect(fs.statSync(path.join(dir, 'sensitive-evidence.json')).size).toBeLessThan(MIB);
|
||||
expect(evidence.at(-1).note).toContain(`${evidence.at(-1).omitted} more files with sensitive-pattern findings are not listed`);
|
||||
expect(evidence.length - 1 + evidence.at(-1).omitted).toBe(999);
|
||||
|
||||
const initial = readJson(path.join(dir, 'report.json')),
|
||||
snapshot = initial.coverage.find((item: any) => item.domain === 'snapshot-inputs'),
|
||||
transformations = initial.source.transformations;
|
||||
expect(fs.statSync(path.join(dir, 'report.json')).size).toBeLessThan(400 * 1024);
|
||||
expect(snapshot.evidence).toEqual(['1000 sanitized execution inputs captured; 3000 explicit non-executable exclusions; 1000 unread in-scope inputs; 0 tracked deletions']);
|
||||
expect(snapshot.status).toBe('partial');
|
||||
const omittedCount = (line: string) => Number(line.match(/^(\d+) more/)?.[1]);
|
||||
expect(snapshot.gaps.length - 1 + omittedCount(snapshot.gaps.at(-1))).toBe(1000);
|
||||
expect(snapshot.gaps.at(-1)).toContain('more unread or deleted in-scope inputs are not listed to keep report.json within its 1 MiB bound');
|
||||
expect(snapshot.exclusions.length - 1 + omittedCount(snapshot.exclusions.at(-1))).toBe(3000);
|
||||
expect(transformations.at(-1).path).toBe('[not listed]');
|
||||
expect(transformations.length - 1 + omittedCount(transformations.at(-1).handling)).toBe(4999);
|
||||
expect(transformations.length).toBeGreaterThan(100);
|
||||
expect(initial.events[0].message).toBe('Captured 5000 source entries; 4999 transformations disclosed');
|
||||
|
||||
const inspected = cso(repo, ['inspect', run.runId]);
|
||||
expect(inspected.status).toBe(0);
|
||||
expect(JSON.parse(inspected.stdout).manifest.entries).toHaveLength(5000);
|
||||
|
||||
expect(cso(repo, ['submit', run.runId, writeInput('hostile-evidence.json', completeEvidence(initial, [finding]))]).status).toBe(0);
|
||||
const finished = cso(repo, ['finish', run.runId]);
|
||||
expect(finished.stderr).toBe('');
|
||||
expect(finished.status).toBe(0);
|
||||
const final = readJson(path.join(dir, 'report.json'));
|
||||
expect(final.status).toBe('finished');
|
||||
expect(final.findings).toHaveLength(1);
|
||||
expect(fs.statSync(path.join(dir, 'report.json')).size).toBeLessThan(MIB);
|
||||
|
||||
const rechecked = cso(repo, ['recheck', final.findings[0].id, '--repo', repo]);
|
||||
expect(rechecked.stderr).toBe('');
|
||||
expect(rechecked.status).toBe(0);
|
||||
const child = JSON.parse(rechecked.stdout);
|
||||
expect(child.parent).toEqual({ runId: run.runId, findingId: final.findings[0].id, kind: 'recheck' });
|
||||
expect(readSnapshotManifest(runDir({ ...child, repoId: run.repoId })).entries).toHaveLength(5000);
|
||||
}, 300_000);
|
||||
|
||||
test('a retained pretty-printed snapshot.json from an older helper still loads in recheck', () => {
|
||||
const repo = makeRepo('legacy-pretty', { 'src/users.ts': USERS, 'src/other.ts': 'export const unrelated = true\n' }),
|
||||
run = JSON.parse(cso(repo, ['start', '--repo', repo, '--scope', 'auth', '--offline']).stdout),
|
||||
dir = runDir(run),
|
||||
initial = readJson(path.join(dir, 'report.json'));
|
||||
expect(cso(repo, ['submit', run.runId, writeInput('legacy-evidence.json', completeEvidence(initial, [finding]))]).status).toBe(0);
|
||||
expect(cso(repo, ['finish', run.runId]).status).toBe(0);
|
||||
const file = path.join(dir, 'snapshot.json');
|
||||
fs.writeFileSync(file, JSON.stringify(JSON.parse(fs.readFileSync(file, 'utf8')), null, 2) + '\n');
|
||||
expect(fs.readFileSync(file, 'utf8')).toContain('\n "entries": [');
|
||||
expect(cso(repo, ['inspect', run.runId]).status).toBe(0);
|
||||
const findingId = readJson(path.join(dir, 'report.json')).findings[0].id,
|
||||
rechecked = cso(repo, ['recheck', findingId, '--run', run.runId, '--repo', repo]);
|
||||
expect(rechecked.stderr).toBe('');
|
||||
expect(rechecked.status).toBe(0);
|
||||
expect(JSON.parse(rechecked.stdout).parent.runId).toBe(run.runId);
|
||||
}, 120_000);
|
||||
});
|
||||
Reference in new issue
Block a user