From 8f103baa713c2c7cdf34aeb152f022db569c7b11 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sat, 17 Oct 2026 18:54:14 -0700 Subject: [PATCH] v1.91.131.0 feat(pregate): Python import lanes (#3201) and a missing-dependencies regen diagnostic (#3252) * feat(pregate): lanes follows Python imports and mock.patch targets The import graph only parsed JS/TS imports, so in a Python repo every touched module failed `lanes` with PREGATE_NO_LANE even though SOURCE_EXT, TEST_FILE_RE and the pytest runner/summary already understood Python. Python sources now contribute `import a.b`, `from a.b import c` (c may be a submodule), parenthesised multi-line lists and relative `from ..pkg import x` edges, resolved to `a/b.py` or `a/b/__init__.py` from the repository root. Package `__init__.py` files on the path get edges too, since Python executes them on import. Modules outside the repo resolve to nothing. In test files, dotted string literals such as `'pkg.mod.Class.method'` (mock.patch targets) resolve to the longest dotted prefix that is a repo module. Co-Authored-By: Claude Opus 5.5 * fix(regen): report missing dependencies as REGEN_DEPS_MISSING, not REGEN_STALE When package.json declares dependencies and node_modules is missing, the scratch worktree runs the generator without them. A generator that degrades and still exits 0 then looked stale, and the fix line sent people to rerun the generator. gstack-regen check and the pregate regen check now report "dependencies missing; run " with its own result code and troubleshooting anchor; the install command follows the lockfile. The fixture reproduces the shape GSTA-31 saw in gbrain. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(pregate): keep Python import scanning linear and conservative Review of #3201 on the integrated tree found two quadratic paths on untrusted Python source and three ways a test could be left unselected: - The from-import module spec let two repetitions share a dot, so a failed match on a long run of dots backtracked quadratically. The spec is now dots followed by an optional word-led name. - Dotted literals and imports resolved every prefix with repeated split/join. Names are capped at 32 segments, deeper than any module path. - A literal that also named a repo file skipped its mock.patch reading; both readings now add edges. - A package and a same-named module both map (Python loads the package). - Imports after `;` or a compound statement's `:` are recognised. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(pregate): linear Python string blanking, non-empty from-spec, repo-bounded depth Second review round on the Python import scanner: - An empty from-spec let the whitespace on both sides of it compete, so `from` followed by a long run of spaces backtracked quadratically. The spec now starts with a dot or a word character. - Strings and comments are blanked by a single linear scanner that honours backslash escapes, so an escaped triple quote no longer ends a docstring early and swallows the imports after it. - The fixed 32-segment cap dropped real deep modules. Dotted names are now bounded by the repo's deepest path, which no module can exceed. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(changelog): unreleased entry for Python import lanes and REGEN_DEPS_MISSING Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * v1.91.126.0 stamp Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * release slot 1.91.126.0: back to unstamped (CHANGELOG entry unreleased) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(changelog): resolve the main-merge conflict (Unreleased entry above 1.91.127.0) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * v1.91.131.0 stamp Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: Scott Gigante Co-authored-by: Claude Opus 5.5 Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- CHANGELOG.md | 21 +++++++ VERSION | 2 +- agents-digest/gstack-AGENTS.md | 2 +- bin/gstack-doctor-components.sh | 1 + bin/gstack-regen | 4 +- docs/pregate.md | 4 +- docs/troubleshooting.md | 12 ++++ lib/pregate/checks/regen.ts | 2 + lib/pregate/import-graph.ts | 108 +++++++++++++++++++++++++++++++- lib/regen.ts | 26 +++++++- lib/result-codes-release.ts | 9 ++- package.json | 2 +- test/gstack-regen.test.ts | 50 ++++++++++++++- test/pregate-selection.test.ts | 77 +++++++++++++++++++++++ 14 files changed, 306 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 796a13f7d..ac4418ce4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,26 @@ # Changelog +## [1.91.131.0] - 2026-10-18 + +**`gstack-pregate` finds the tests for Python files, and a missing `node_modules` no longer reads as "stale generated files".** In a Python repo, the `lanes` check failed almost every touched `.py` file with `PREGATE_NO_LANE`, because the import graph only parsed JavaScript and TypeScript. In a FastAPI service with about 1,500 non-test modules, Scott Gigante's change (#3201) cut the unmapped count from 1,473 to 780, and most of the rest are migrations and scripts no import reaches. Separately, a fresh checkout without `node_modules` made the pre-gate report `REGEN_STALE` and suggest rerunning the generator, which sent people the wrong way. + +### Added +- **Python import edges in `lanes`** (thanks @scottgigante-hubflow, #3201). `import a.b`, `from a.b import c`, parenthesised lists and relative `from . import x` resolve from the repository root to `a/b.py` or `a/b/__init__.py`, plus the package `__init__.py` files Python runs on the way in. Stdlib and site-packages imports add nothing. In test files, a dotted `mock.patch` target such as `'pkg.mod.Class.method'` maps to its module. +- **`REGEN_DEPS_MISSING`.** When a regen command exits 0 with different outputs, `package.json` declares dependencies and `node_modules` is missing, `gstack-regen check` and the pre-gate say "dependencies missing (no node_modules); run bun install, then rerun the check" instead of `REGEN_STALE`. The install command follows the lockfile. The pre-gate check is `incomplete`, never `pass`. + +### Fixed +- **The Python scanner stays linear on hostile input and does not drop tests.** Review found several problems before landing, all fixed: + - Long runs of dots or spaces after `from` no longer backtrack quadratically. + - Deep dotted names stay linear, and real deep modules still map. + - An escaped triple quote no longer swallows the imports after it. + - A literal that names a repo file still gets its `mock.patch` reading. + - A package and a same-named module both map. + - Imports after `;` or `:` count. + +### If you… +- **…run `gstack-pregate` in a Python repo:** touched modules now select the pytest files that import them. `conftest.py`, `src/` layouts and `importlib` calls still need `.gstack/pregate.json` `dependencies` declarations. +- **…see `REGEN_DEPS_MISSING`:** run the printed install command, then rerun the check. + ## [1.91.130.0] - 2026-10-18 **Every PR plan now shows what a tighter, exercised-path case map would run, without changing what actually runs.** Today a change to a bin or lib file reaches almost every gate case, because selection follows every mention of the file. The exercised-path map counts only what a case really executes: its touchfiles, its test's imports, the shell its skill runs, and the bins its recorded trials invoked. It runs in shadow mode. Replayed over every PR run that had a gate red between 2026-10-07 and 2026-10-17 (37 runs, 47 failed gate cases), it keeps all 47: `bun run test:exercised-map --replay`. diff --git a/VERSION b/VERSION index 97bde5585..a1846f406 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.91.130.0 +1.91.131.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index 895f1b42e..e0a4e9ee8 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.91.130.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.91.131.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/bin/gstack-doctor-components.sh b/bin/gstack-doctor-components.sh index 463567aae..eea90fd19 100644 --- a/bin/gstack-doctor-components.sh +++ b/bin/gstack-doctor-components.sh @@ -267,6 +267,7 @@ gstack_result_anchor() { APPROVAL_SOURCE_UNAVAILABLE) printf '%s' 'approval-source-unavailable' ;; STATE_MATRIX_INCOMPLETE) printf '%s' 'state-matrix-incomplete' ;; WTREE_UNAVAILABLE) printf '%s' 'wtree-unavailable' ;; + REGEN_DEPS_MISSING) printf '%s' 'regen-deps-missing' ;; *) return 1 ;; esac } diff --git a/bin/gstack-regen b/bin/gstack-regen index 4281915f3..39657a47e 100755 --- a/bin/gstack-regen +++ b/bin/gstack-regen @@ -119,8 +119,8 @@ function main(argv: string[]): number { let outcome; try { outcome = runRegen({ repoRoot: root, registry: loaded.registry, mode: cmd, only: list(flag(rest, '--only')), touched: list(flag(rest, '--touched')) }); } catch (e: any) { process.stderr.write(resultLine(`gstack-regen ${cmd}`, 'PREGATE_INCOMPLETE', e.message) + '\n'); process.stdout.write(formatResult('regen', 'incomplete', loaded.source.label) + '\n'); return EXIT.fail; } - const bad = outcome.results.filter(r => r.status === 'stale' || r.status === 'failed'); - const status = outcome.results.some(r => r.status === 'failed') ? 'incomplete' : 'complete'; + const bad = outcome.results.filter(r => r.status === 'stale' || r.status === 'failed' || r.status === 'deps-missing'); + const status = outcome.results.some(r => r.status === 'failed' || r.status === 'deps-missing') ? 'incomplete' : 'complete'; if (json) { process.stdout.write(JSON.stringify({ mode: cmd, registry: loaded.source.label, wtree: outcome.wtree, results: outcome.results, ok: bad.length === 0 }) + '\n'); return bad.length ? EXIT.fail : EXIT.ok; } process.stdout.write(`registry: ${loaded.source.label}\n`); for (const r of outcome.results) process.stdout.write(renderEntry(r) + '\n'); diff --git a/docs/pregate.md b/docs/pregate.md index 6e9e4f0e0..b988c06ab 100644 --- a/docs/pregate.md +++ b/docs/pregate.md @@ -41,7 +41,9 @@ paths, at most 64 per list); commands are one line under 512 bytes. ` for entries whose inputs or outputs intersect) in a scratch `git worktree` seeded from the working tree, with `node_modules` linked rather than copied, and reports stale outputs (`REGEN_STALE`, exit 1) without touching - the tree. An entry with no `command` is tracked only (goldens regenerated by + the tree. When outputs differ but `package.json` declares dependencies and + `node_modules` is missing, the entry is `REGEN_DEPS_MISSING` (install, then + rerun) instead of stale. An entry with no `command` is tracked only (goldens regenerated by another entry). - `gstack-regen write` regenerates in place: what `gstack-restamp` and the Step 3 conflict rule use. On a conflict in a registered output, take either diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index d802e845b..e2719bca0 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -2789,6 +2789,18 @@ take either side, then regenerate. worktree has no `node_modules` of its own, so a command that needs them must reference the repository's (gstack's entries do through `bun run`). + +### `REGEN_DEPS_MISSING` + +**Meaning.** A registered command exited 0 and its outputs differ from the +committed ones, but `package.json` declares dependencies and the repository has +no `node_modules`, so the scratch worktree ran the generator without them. The +difference may come from the missing dependencies rather than stale files, so +the entry is reported as unverified, not `REGEN_STALE`. + +**Fix.** Run the printed install command (`bun install`, or the one matching the +repository's lockfile), then rerun `gstack-regen check` or the pre-gate. + ### `PREGATE_CONFIG_INVALID` diff --git a/lib/pregate/checks/regen.ts b/lib/pregate/checks/regen.ts index 510512230..dd63dd003 100644 --- a/lib/pregate/checks/regen.ts +++ b/lib/pregate/checks/regen.ts @@ -27,7 +27,9 @@ function run(ctx: PregateContext): CheckResult { const lines = results.map(renderEntry); const stale = results.filter(r => r.status === 'stale'); const failed = results.filter(r => r.status === 'failed'); + const depsMissing = results.filter(r => r.status === 'deps-missing'); ctx.explain.push(...results.map(r => `regen ${r.id}: selected because touched ∩ (${selected.find(e => e.id === r.id)?.inputs.join(',') || 'always'} ∪ outputs) ≠ ∅`)); + if (depsMissing.length) return { id: 'regen', stage: 'preflight', status: 'incomplete', inputs, lines, code: 'REGEN_DEPS_MISSING', detail: `dependencies missing; ${depsMissing[0]!.fix} (${depsMissing.length} entr${depsMissing.length === 1 ? 'y' : 'ies'} unverified: ${depsMissing.map(d => d.id).join(', ')})`, fix: depsMissing[0]!.fix }; if (failed.length) return { id: 'regen', stage: 'preflight', status: 'incomplete', inputs, lines, code: 'REGEN_COMMAND_FAILED', detail: `${failed.length} regen command(s) failed: ${failed.map(f => f.id).join(', ')}`, fix: failed.map(f => f.fix).join('; ') }; if (stale.length) return { id: 'regen', stage: 'preflight', status: 'fail', inputs, lines, code: 'REGEN_STALE', detail: `${stale.length} stale: ${stale.flatMap(s => s.changed.concat(s.outside)).join(' ')}`, fix: stale.map(s => s.fix).join('; ') }; return { id: 'regen', stage: 'preflight', status: 'pass', inputs, lines, detail: `${results.length} entr${results.length === 1 ? 'y' : 'ies'} fresh${results.some(r => r.status === 'listed') ? ' (hand-regenerated outputs listed)' : ''}` }; diff --git a/lib/pregate/import-graph.ts b/lib/pregate/import-graph.ts index 56ad8d989..9db6f4fd6 100644 --- a/lib/pregate/import-graph.ts +++ b/lib/pregate/import-graph.ts @@ -2,10 +2,13 @@ * pregate/import-graph — which test files depend on which repo files (plan * C8 `lanes`). The reverse graph unions four sources, each kept as provenance * for `--explain`: - * 1. static imports and requires (Bun.Transpiler.scanImports); + * 1. static imports and requires (Bun.Transpiler.scanImports), and Python + * `import` / `from … import` statements resolved from the repository root + * or, when relative, from the importing file's package; * 2. repo-path string literals, including `join('a', 'b')` segment runs * (a test that spawns `bin/gstack-doctor` names it in a string, never an - * import) and literal directories (`test/fixtures/x` → `x/**`); + * import) and literal directories (`test/fixtures/x` → `x/**`), and in + * Python, dotted `mock.patch` targets (`'pkg.mod.Class.method'`); * 3. the `bin/` → `test/*.test.*` convention (with and without * the `gstack-` prefix); * 4. explicit `.gstack/pregate.json` `dependencies` declarations. @@ -41,6 +44,10 @@ const scanner = new Bun.Transpiler({ loader: 'tsx' }); const QUOTED = /['"`]([A-Za-z0-9_@][A-Za-z0-9_./@()[\]+-]*)['"`]/g; const JOIN = /\bjoin\(([^()]*)\)/g; const RESOLVE_EXT = ['', '.ts', '.tsx', '.js', '.mjs', '.cjs', '/index.ts', '/index.js']; +// A statement starts a line or follows `;` or a compound statement's `:`. The module spec is +// non-empty and starts with a dot or a word character, so no two repetitions can share a dot or a space. +const PY_IMPORT = /(?:^|[;:])[ \t\f]*(?:from[ \t]+(\.+(?:\w[\w.]*)?|\w[\w.]*)[ \t]+import[ \t]+(\([^)]*\)|[^\n;]+)|import[ \t]+([^\n;]+))/gm; +const PY_DOTTED = /^[A-Za-z_]\w*(?:\.[A-Za-z_]\w*)+$/; export function isTestFile(rel: string): boolean { return TEST_FILE_RE.test(rel); } @@ -185,6 +192,95 @@ function pathLike(config: PathAliases, ref: string): boolean { return chosen !== null || others.length > 0; } +const pathDepths = new WeakMap, number>(); +/** + * Segments in the repo's deepest path. No module name resolves deeper, so + * dotted-name resolution is bounded by the repo rather than by the source. + */ +function maxPathDepth(files: Set): number { + let depth = pathDepths.get(files); + if (depth === undefined) { depth = 0; for (const f of files) depth = Math.max(depth, f.split('/').length); pathDepths.set(files, depth); } + return depth; +} + +/** The leading dotted name of each comma-separated `name [as alias]` clause. */ +function pyNames(list: string): string[] { + return list.replace(/[()]/g, '').split(',').map(p => p.trim().split(/\s+/)[0]!).filter(n => n && n !== '*'); +} + +/** `from ` → the absolute dotted package it names, or null when it climbs above the repository root. */ +function pyFromBase(rel: string, spec: string): string | null { + const dots = /^\.*/.exec(spec)![0].length; + const rest = spec.slice(dots); + if (!dots) return rest; + const pkg = path.posix.dirname(rel).split('/').filter(s => s !== '.'); + if (dots - 1 > pkg.length) return null; + return [...pkg.slice(0, pkg.length - (dots - 1)), ...(rest ? [rest] : [])].join('.'); +} + +/** + * Python source with every string blanked to `''` and every comment dropped, in + * one linear pass, so a `#` inside a string and a quote inside a comment both + * stay inert. Backslash escapes are honoured; an unterminated string runs to + * its line end, or to the end of the file when triple-quoted. + */ +function pythonCode(source: string): string { + const parts: string[] = []; + let from = 0; + for (let i = 0; i < source.length;) { + const c = source[i]!; + if (c === '#') { parts.push(source.slice(from, i)); const end = source.indexOf('\n', i); i = from = end < 0 ? source.length : end; continue; } + if (c !== '"' && c !== "'") { i++; continue; } + parts.push(source.slice(from, i), "''"); + const close = source.startsWith(c + c + c, i) ? c + c + c : c; + let j = i + close.length; + while (j < source.length && !source.startsWith(close, j) && (close.length === 3 || source[j] !== '\n')) j += source[j] === '\\' ? 2 : 1; + i = from = Math.min(source.length, source.startsWith(close, j) ? j + close.length : j); + } + parts.push(source.slice(from)); + return parts.join(''); +} + +/** Dotted modules a Python source imports; `from a import b` also yields `a.b`, which may be a submodule. */ +function pythonImports(rel: string, source: string): string[] { + const code = pythonCode(source).replace(/\\\r?\n/g, ' '); + const out: string[] = []; + for (const m of code.matchAll(PY_IMPORT)) { + if (m[3]) { out.push(...pyNames(m[3])); continue; } + const base = pyFromBase(rel, m[1]!); + if (base === null) continue; + if (base) out.push(base); + for (const name of pyNames(m[2]!)) out.push(base ? `${base}.${name}` : name); + } + return out; +} + +/** + * A dotted module → its repo files (`a/b/__init__.py`, which Python prefers, + * and `a/b.py`; both when both exist) plus the package `__init__.py` files + * Python executes on the way; [] for a module outside the repo (stdlib, + * site-packages), including one deeper than any repo path. + */ +function pythonModuleFiles(files: Set, dotted: string): string[] { + const parts = dotted.split('.'); + if (parts.length > maxPathDepth(files) || parts.some(p => !p)) return []; + const p = parts.join('/'); + const found = [`${p}/__init__.py`, `${p}.py`].filter(f => files.has(f)); + if (!found.length) return []; + const inits = parts.slice(0, -1).map((_, i) => `${parts.slice(0, i + 1).join('/')}/__init__.py`).filter(f => files.has(f)); + return [...found, ...inits]; +} + +/** `'pkg.mod.Class.attr'` (a `mock.patch` target) → the longest dotted prefix that is a repo module. */ +function pythonPatchTarget(files: Set, ref: string): string[] { + const parts = ref.split('.').slice(0, maxPathDepth(files)); + for (let n = parts.length; n >= 1; n--) { + const hit = pythonModuleFiles(files, parts.slice(0, n).join('.')); + if (hit.length) return hit; + } + return []; +} + function addEdge(graph: Graph, dependency: string, dependent: string, via: Edge['via']): void { if (dependency === dependent) return; const list = graph.reverse.get(dependency) ?? []; @@ -210,11 +306,17 @@ export function referencesIn(files: Set, dirs: Set, rel: string, // The transpiler erases type-only imports; a types module still binds its importers' contracts. for (const m of source.matchAll(/^\s*(?:import|export)\s+type\b[^;'"]*?\bfrom\s+['"]([^'"]+)['"]/gm)) importOf(m[1]!); } + const python = rel.endsWith('.py'); + if (python) for (const mod of pythonImports(rel, source)) for (const t of pythonModuleFiles(files, mod)) out.push({ target: t, via: 'import' }); // Literals carry dependencies only from the test tree and from bin/ and shell // sources (a test spawns a bin; a bash bin calls a sibling). Prose and // manifests name everything and would make every file a hub. if (!scansLiterals(rel)) return out; - for (const m of source.matchAll(QUOTED)) for (const target of resolveRefs(files, dirs, rel, m[1]!, true, config)) out.push({ target, via: 'literal' }); + for (const m of source.matchAll(QUOTED)) { + const targets = resolveRefs(files, dirs, rel, m[1]!, true, config); + for (const target of targets) out.push({ target, via: 'literal' }); + if (python && PY_DOTTED.test(m[1]!)) for (const f of pythonPatchTarget(files, m[1]!)) out.push({ target: f, via: 'literal' }); + } for (const m of source.matchAll(JOIN)) { const segments = [...m[1]!.matchAll(/['"`]([^'"`]+)['"`]/g)].map(s => s[1]!); if (segments.length < 2) continue; diff --git a/lib/regen.ts b/lib/regen.ts index c3672a627..2a8538647 100644 --- a/lib/regen.ts +++ b/lib/regen.ts @@ -156,7 +156,7 @@ export function scratchWorktree(repoRoot: string, prefix = 'gstack-regen-'): Scr export interface EntryResult { id: string; command: string | null; - status: 'fresh' | 'stale' | 'failed' | 'listed' | 'written' | 'skipped'; + status: 'fresh' | 'stale' | 'failed' | 'listed' | 'written' | 'skipped' | 'deps-missing'; /** Output files whose bytes changed (stale) or were written. */ changed: string[]; /** Files the command wrote outside its declared outputs (a refusal in `check`). */ @@ -189,6 +189,20 @@ function diffSnapshots(a: Map, b: Map): string[] return [...changed].filter(p => p !== 'node_modules').sort(); } +/** + * The install command for a repository whose package.json declares + * dependencies but has no node_modules (the scratch worktree links none + * then), picked from its lockfile; null when nothing is missing. + */ +export function missingDependencies(repoRoot: string): string | null { + if (fs.existsSync(path.join(repoRoot, 'node_modules'))) return null; + let pkg: { dependencies?: Record; devDependencies?: Record }; + try { pkg = JSON.parse(fs.readFileSync(path.join(repoRoot, 'package.json'), 'utf8')) ?? {}; } catch { return null; } + if (!Object.keys(pkg.dependencies ?? {}).length && !Object.keys(pkg.devDependencies ?? {}).length) return null; + const lockfiles: Array<[string, string]> = [['bun.lock', 'bun install'], ['bun.lockb', 'bun install'], ['pnpm-lock.yaml', 'pnpm install'], ['yarn.lock', 'yarn install'], ['package-lock.json', 'npm install']]; + return lockfiles.find(([file]) => fs.existsSync(path.join(repoRoot, file)))?.[1] ?? 'bun install'; +} + function runCommand(cwd: string, command: string, env: NodeJS.ProcessEnv, timeoutMs: number): { exit: number | null; stderr: string } { const r = runBashSync(`set -o pipefail; ${command}`, { cwd, encoding: 'utf8', timeout: timeoutMs, maxBuffer: 64 * 1024 * 1024, env: { ...env, GSTACK_REGEN_SCRATCH: cwd } }); return { exit: r.status, stderr: ((r.stderr ?? '') + (r.error ? `\n${r.error.message}` : '')).trim().slice(-4000) }; @@ -207,7 +221,9 @@ export interface RunRegenOptions { * `check`: run every selected command in one scratch worktree and compare its * outputs with the working tree; `write`: run them in the repository. An * entry with `command: null` is `listed` (its fix line names the outputs an - * author refreshes by hand). Never runs a command the caller did not opt into: + * author refreshes by hand). A `check` whose outputs differ while the + * repository's dependencies are not installed is `deps-missing`, not `stale`: + * the generator may have run without them. Never runs a command the caller did not opt into: * the bin enforces --allow-repo-commands before calling this. */ export function runRegen(o: RunRegenOptions): { results: EntryResult[]; wtree: string | null } { @@ -215,6 +231,7 @@ export function runRegen(o: RunRegenOptions): { results: EntryResult[]; wtree: s const timeout = o.timeoutMs ?? 600_000; const results: EntryResult[] = []; let scratch: Scratch | null = null; + const install = o.mode === 'check' ? missingDependencies(o.repoRoot) : null; const fixFor = (e: RegenEntry) => e.command ? `${e.command} # then commit ${e.outputs.join(' ')}` : `regenerate by hand: ${e.outputs.join(' ')}${e.note ? ` (${e.note})` : ''}`; try { if (o.mode === 'check' && entries.some(e => e.command)) scratch = scratchWorktree(o.repoRoot); @@ -230,7 +247,9 @@ export function runRegen(o: RunRegenOptions): { results: EntryResult[]; wtree: s const ms = Date.now() - started; if (run.exit !== 0) { results.push({ id: e.id, command: e.command, status: 'failed', changed, outside, exit: run.exit, stderr: run.stderr, ms, fix: fixFor(e) }); continue; } if (o.mode === 'check') { - results.push({ id: e.id, command: e.command, status: changed.length || outside.length ? 'stale' : 'fresh', changed, outside, exit: 0, ms, fix: fixFor(e) }); + const differs = changed.length > 0 || outside.length > 0; + if (differs && install) results.push({ id: e.id, command: e.command, status: 'deps-missing', changed, outside, exit: 0, ms, fix: `run ${install}, then rerun the check` }); + else results.push({ id: e.id, command: e.command, status: differs ? 'stale' : 'fresh', changed, outside, exit: 0, ms, fix: fixFor(e) }); // Reset the scratch so the next entry's diff is its own. git(cwd, ['checkout', '-q', '--', '.']); git(cwd, ['clean', '-fdq']); } else results.push({ id: e.id, command: e.command, status: changed.length ? 'written' : 'fresh', changed, outside, exit: 0, ms, fix: fixFor(e) }); @@ -247,6 +266,7 @@ export function renderEntry(r: EntryResult): string { if (r.status === 'fresh' || r.status === 'skipped') return head; if (r.status === 'listed') return `${head} — ${r.fix}`; if (r.status === 'failed') return `${head} — exit ${r.exit}: ${(r.stderr ?? '').split('\n').pop() ?? ''}; fix: run \`${r.command}\` and read its output (REGEN_COMMAND_FAILED)`; + if (r.status === 'deps-missing') return `${head} — dependencies missing (no node_modules); ${r.fix} — differs without them: ${[...r.changed, ...r.outside].join(' ')} (REGEN_DEPS_MISSING)`; const parts = [head]; if (r.changed.length) parts.push(`stale: ${r.changed.join(' ')}`); if (r.outside.length) parts.push(`outside declared outputs: ${r.outside.join(' ')}`); diff --git a/lib/result-codes-release.ts b/lib/result-codes-release.ts index 19f291a69..42ff9b088 100644 --- a/lib/result-codes-release.ts +++ b/lib/result-codes-release.ts @@ -1,6 +1,7 @@ /** * Result codes for the release path: gstack-ship-queue and the ship-policy - * floor, pregate timing and its working-tree fingerprint, approval classes and + * floor, pregate timing and its working-tree fingerprint, regen with missing + * dependencies, approval classes and * the state-machine matrix. Kept beside lib/result-codes.ts (which spreads * this table into RESULT_CODES) so * the owner module stays within the module-size ratchet; the same rules apply: @@ -83,4 +84,10 @@ export const RELEASE_RESULT_CODES = { summary: 'cannot fingerprint the working tree: bin/gstack-wtree could not start (on Windows, no Git Bash was found) or it ran and failed', fix: 'on Windows point GSTACK_GIT_BASH at Git Bash\'s bash.exe (Git Bash: export GSTACK_GIT_BASH=\'C:/Program Files/Git/bin/bash.exe\'; PowerShell: $env:GSTACK_GIT_BASH = \'C:\\Program Files\\Git\\bin\\bash.exe\'), or install Git for Windows (winget install --id Git.Git -e); elsewhere rerun ./setup so bin/gstack-wtree is present; then rerun gstack-pregate', }, + // Fix wave W5 (GSTA-31): pregate regen without installed dependencies. + REGEN_DEPS_MISSING: { + anchor: 'regen-deps-missing', + summary: 'a regen command exited 0 with different outputs while package.json declares dependencies and node_modules is missing, so the outputs were not verified', + fix: 'install the dependencies (the printed command, such as bun install), then rerun the check', + }, } as const satisfies Record; diff --git a/package.json b/package.json index 1a72f3fc1..d0f2a1ac6 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.91.130", + "version": "1.91.131", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", diff --git a/test/gstack-regen.test.ts b/test/gstack-regen.test.ts index 5c527b24e..f718d6ec9 100644 --- a/test/gstack-regen.test.ts +++ b/test/gstack-regen.test.ts @@ -9,13 +9,24 @@ import { afterEach, describe, expect, test } from 'bun:test'; import * as fs from 'node:fs'; import * as path from 'node:path'; import { EXIT } from '../lib/headless-artifacts'; -import { parseRegistry, selectEntries, validateRegistry } from '../lib/regen'; +import { missingDependencies, parseRegistry, selectEntries, validateRegistry } from '../lib/regen'; import { BASE_REGISTRY, cleanup, commit, makeFixtureRepo, readFile, runBin, write, type FixtureRepo } from './helpers/pregate-fixture'; const repos: FixtureRepo[] = []; const fixture = (opts?: Parameters[0]) => { const r = makeFixtureRepo(opts); repos.push(r); return r; }; afterEach(() => { while (repos.length) cleanup(repos.pop()!); }); +/** gbrain's shape (GSTA-31): the generator degrades without its dependency and still exits 0. */ +const DEPS_FIXTURE = { + registry: { entries: [{ id: 'digest', command: 'node gen.js', outputs: ['generated/**'], inputs: ['src/version.js', 'gen.js'] }] }, + baseFiles: { + 'package.json': JSON.stringify({ name: 'fixture', version: '1.0.0', dependencies: { 'fixture-dep': '1.0.0' } }, null, 2) + '\n', + 'bun.lock': '{}\n', + 'gen.js': "const fs=require('fs');const {VERSION}=require('./src/version.js');let style='plain';try{style=require('fixture-dep').style;}catch{}fs.writeFileSync('generated/digest.md',`# digest v${VERSION} (${style})\\n`);\n", + 'generated/digest.md': '# digest v1.0.0 (fancy)\n', + }, +}; + describe('validateRegistry (schema + containment, pure)', () => { test('the fixture registry validates; command null is a listed entry; inputs default to always', () => { expect(validateRegistry(BASE_REGISTRY)).toEqual([]); @@ -147,6 +158,43 @@ describe('gstack-regen (bin)', () => { expect(all.out).toContain('regen digest: stale'); }); + test('missing dependencies: a generator that exits 0 without node_modules is REGEN_DEPS_MISSING (install, then rerun), not REGEN_STALE; installed deps make it fresh', () => { + const repo = fixture(DEPS_FIXTURE); + commit(repo, { 'src/version.js': "module.exports = { VERSION: '1.0.0' };\n// touched\n" }); + const r = runBin(repo, 'gstack-regen', ['check', '--base', 'main', '--allow-repo-commands']); + expect(r.status).toBe(EXIT.fail); + expect(r.out).toMatch(/^regen digest: deps-missing \(\d+ ms\) — dependencies missing \(no node_modules\); run bun install, then rerun the check — differs without them: generated\/digest\.md \(REGEN_DEPS_MISSING\)$/m); + expect(r.out).not.toContain('REGEN_STALE'); + expect(r.out).toContain('GSTACK_RESULT: skill=regen status=incomplete'); + const json = JSON.parse(runBin(repo, 'gstack-regen', ['check', '--base', 'main', '--allow-repo-commands', '--json']).out); + expect(json.results.map((x: any) => [x.id, x.status])).toEqual([['digest', 'deps-missing']]); + expect(repo.git('status', '--porcelain')).toBe(''); + + write(repo.work, 'node_modules/fixture-dep/index.js', "exports.style = 'fancy';\n"); + const installed = runBin(repo, 'gstack-regen', ['check', '--base', 'main', '--allow-repo-commands']); + expect(installed.status).toBe(EXIT.ok); + expect(installed.out).toContain('regen digest: fresh'); + + // With dependencies installed, a real input change is still stale. + commit(repo, { 'src/version.js': "module.exports = { VERSION: '1.1.0' };\n" }); + const stale = runBin(repo, 'gstack-regen', ['check', '--base', 'main', '--allow-repo-commands']); + expect(stale.status).toBe(EXIT.fail); + expect(stale.out).toContain('regen digest: stale'); + expect(stale.out).toContain('(REGEN_STALE)'); + expect(stale.out).not.toContain('REGEN_DEPS_MISSING'); + }); + + test('missing dependencies: the install command follows the lockfile', () => { + const repo = fixture(DEPS_FIXTURE); + expect(missingDependencies(repo.work)).toBe('bun install'); + fs.rmSync(path.join(repo.work, 'bun.lock')); + write(repo.work, 'package-lock.json', '{}\n'); + expect(missingDependencies(repo.work)).toBe('npm install'); + fs.mkdirSync(path.join(repo.work, 'node_modules')); + expect(missingDependencies(repo.work)).toBeNull(); + expect(missingDependencies(fixture().work)).toBeNull(); + }); + test('validate, list and init: an invalid working-tree registry is REGEN_REGISTRY_INVALID; init refuses to overwrite', () => { const repo = fixture(); expect(runBin(repo, 'gstack-regen', ['validate']).out).toContain('REGISTRY_VALID: .gstack/generated.json'); diff --git a/test/pregate-selection.test.ts b/test/pregate-selection.test.ts index fe839ffe0..59404fd5f 100644 --- a/test/pregate-selection.test.ts +++ b/test/pregate-selection.test.ts @@ -81,6 +81,66 @@ describe('import graph (static imports ∪ repo-path literals ∪ bin convention expect(testsFor(g, 'test/fixtures/sample.md').provenance[0]).toContain('(literal)'); expect(testsFor(g, 'notes/scratch.md')).toMatchObject({ tests: [], declaredNone: false }); }); + + test('python: absolute, from-import submodule, relative and parenthesised imports and mock.patch targets map; stdlib, comments and docstrings do not', () => { + const repo = fixture(); + commit(repo, { + 'pkg/__init__.py': '', + 'pkg/core.py': 'import os\nfrom pkg.util import helper\n', + 'pkg/util.py': 'def helper():\n return 1\n', + 'pkg/multi.py': 'X = 1\n', + 'pkg/patched.py': 'class Thing:\n def run(self):\n return 1\n', + 'pkg/ghost.py': 'G = 1\n', + 'pkg/sub/__init__.py': '', + 'pkg/sub/sibling.py': 'S = 1\n', + 'pkg/sub/leaf.py': 'from .. import core\nfrom . import sibling\n', + 'tests/test_core.py': 'import pkg.core as core\n', + 'tests/test_leaf.py': 'from pkg.sub import leaf\n', + 'tests/test_multi.py': 'from pkg import (\n multi, # the module\n util,\n)\nimport json, os.path\n', + 'tests/test_patch.py': "from unittest import mock\n\n@mock.patch('pkg.patched.Thing.run')\ndef test_run(run):\n pass\n", + 'tests/test_ghost.py': '"""\nimport pkg.ghost\n"""\n# from pkg import ghost\nX = "from pkg import ghost"\n', + }); + const g = buildGraph(repo.work); + expect(testsFor(g, 'pkg/core.py').tests).toEqual(['tests/test_core.py', 'tests/test_leaf.py']); + expect(testsFor(g, 'pkg/util.py').tests).toEqual(['tests/test_core.py', 'tests/test_leaf.py', 'tests/test_multi.py']); + expect(testsFor(g, 'pkg/sub/sibling.py').tests).toEqual(['tests/test_leaf.py']); + expect(testsFor(g, 'pkg/multi.py').tests).toEqual(['tests/test_multi.py']); + expect(testsFor(g, 'pkg/patched.py').provenance).toEqual(['tests/test_patch.py ← pkg/patched.py (literal)']); + expect(testsFor(g, 'pkg/sub/__init__.py').tests).toEqual(['tests/test_leaf.py']); + expect(testsFor(g, 'pkg/__init__.py').tests).toEqual(['tests/test_core.py', 'tests/test_leaf.py', 'tests/test_multi.py', 'tests/test_patch.py']); + expect(testsFor(g, 'pkg/ghost.py').tests).toEqual([]); + expect([...g.reverse.keys()].filter(k => k.endsWith('.py')).sort()).toEqual([ + 'pkg/__init__.py', 'pkg/core.py', 'pkg/multi.py', 'pkg/patched.py', 'pkg/sub/__init__.py', 'pkg/sub/leaf.py', 'pkg/sub/sibling.py', 'pkg/util.py', + ]); + }); + + test('python: a package and a same-named module both map, statements after ; and : map, escaped triple quotes and deep modules map, a path literal does not hide its patch target, and adversarial input stays linear', () => { + const repo = fixture(); + commit(repo, { + 'app.py': 'A = 1\n', + 'app/__init__.py': '', + 'pkg/__init__.py': '', + 'pkg/core.py': 'C = 1\n', + 'pkg/patched.py': 'class Thing:\n pass\n', + 'pkg.patched.Thing.run': 'a file whose name is also a patch target\n', + 'tests/test_app.py': 'import app\n', + 'tests/test_semicolon.py': 'import os; import pkg.core\n', + 'tests/test_inline.py': 'def test_core():\n if True: from pkg.core import C\n', + 'tests/test_patch.py': "from unittest import mock\n\n@mock.patch('pkg.patched.Thing.run')\ndef test_run(run):\n pass\n", + 'tests/test_adversarial.py': `from ${'.'.repeat(100_000)}x\nfrom ${' '.repeat(100_000)}x\nX = '${Array.from({ length: 20_000 }, () => 'pkg').join('.')}'\n${'\\"""'.repeat(20_000)}\n`, + 'tests/test_escaped.py': 'DOC = """text \\"""\nstill doc"""\nimport pkg.core\nTAIL = """x"""\n', + [`${Array.from({ length: 33 }, (_, i) => `d${i}`).join('/')}/leaf.py`]: 'L = 1\n', + 'tests/test_deep.py': `import ${Array.from({ length: 33 }, (_, i) => `d${i}`).join('.')}.leaf\n`, + }); + const g = buildGraph(repo.work); + expect(testsFor(g, 'app/__init__.py').tests).toEqual(['tests/test_app.py']); + expect(testsFor(g, 'app.py').tests).toEqual(['tests/test_app.py']); + expect(testsFor(g, 'pkg/core.py').tests).toEqual(['tests/test_escaped.py', 'tests/test_inline.py', 'tests/test_semicolon.py']); + expect(testsFor(g, `${Array.from({ length: 33 }, (_, i) => `d${i}`).join('/')}/leaf.py`).tests).toEqual(['tests/test_deep.py']); + expect(testsFor(g, 'pkg/patched.py').tests).toEqual(['tests/test_patch.py']); + expect(testsFor(g, 'pkg.patched.Thing.run').tests).toEqual(['tests/test_patch.py']); + expect(testsFor(g, 'pkg/__init__.py').tests).toEqual(['tests/test_adversarial.py', 'tests/test_escaped.py', 'tests/test_inline.py', 'tests/test_patch.py', 'tests/test_semicolon.py']); + }); }); describe('literals (pure)', () => { @@ -189,6 +249,23 @@ describe('gstack-pregate (bin)', () => { expect(readOut(repo).stage2_ms).toBeUndefined(); }); + test('missing dependencies: regen is incomplete with REGEN_DEPS_MISSING and the install command, not REGEN_STALE', () => { + const repo = fixture({ + registry: { entries: [{ id: 'digest', command: 'node gen.js', outputs: ['generated/**'], inputs: ['src/version.js', 'gen.js'] }] }, + baseFiles: { + 'package.json': JSON.stringify({ name: 'fixture', version: '1.0.0', scripts: { test: 'node --test test/*.test.js' }, devDependencies: { 'fixture-dep': '1.0.0' } }, null, 2) + '\n', + 'gen.js': "const fs=require('fs');const {VERSION}=require('./src/version.js');let style='plain';try{style=require('fixture-dep').style;}catch{}fs.writeFileSync('generated/digest.md',`# digest v${VERSION} (${style})\\n`);\n", + 'generated/digest.md': '# digest v1.0.0 (fancy)\n', + }, + }); + commit(repo, { 'src/version.js': "module.exports = { VERSION: '1.0.0' };\n// touched\n" }); + const r = pregate(repo, ['--stage', 'preflight', '--only', 'regen']); + expect(r.status).not.toBe(EXIT.ok); + expect(r.out).toMatch(/^preflight {2}regen {5}incomplete .*dependencies missing; run bun install, then rerun the check \(1 entry unverified: digest\) \(REGEN_DEPS_MISSING\)/m); + expect(r.err).toContain('(REGEN_DEPS_MISSING)'); + expect(r.out + r.err).not.toContain('REGEN_STALE'); + }); + test('p0-4-scratch-script: zz-probe.sh at the root fails strays (downgraded to warn only when the base config says so); a conventional root file passes', () => { const strict = fixture({ pregate: { ...BASE_PREGATE, warn: [] } }); commit(strict, { 'zz-probe.sh': '#!/bin/sh\necho hi\n', 'CONTRIBUTING.md': '# c\n', 'notes.txt': 'x\n', 'newtop/thing.js': 'module.exports = 1;\n' });