mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 14:06:50 +01:00
A1: verify gbrain landing before stamping transcripts, plus --reconcile
gbrain import exits 0 while skipping pages (returned per-file failures, malformed filenames, managed-import Pending receipts), and its failure ledger exists only for git import dirs, so a skipped page was stamped ingested and never retried (field reports: 37 and 69 lost entries). Write, verify, then stamp (lib/memory-ingest-landing.ts, a new ratcheted module): - State schema 2 with a transition table (staged -> imported_unverified -> ingested, plus refused, quarantined, unrecoverable). Schema 1 files migrate in place: entries become ingested in the legacy "default" source and a reconcile pass is marked pending. - Named failures come from --json `failures`, stderr "Skipped <rel>:" lines and the ledger; a failure gbrain does not attribute refuses the whole batch. After 3 refused batches the batch is bisected and the page that poisons it is quarantined (printed even under --quiet). - Landing check: one `gbrain list --source-id` per source per run and at most 50 `gbrain get --json` content checks against a gstack_content_sha256 frontmatter field gstack now writes. Lookup errors and pglite_busy mean "not yet checked", never absent; pages over the cap stay imported_unverified and are not re-imported. - Each entry records its gbrain source_id. Remote-http mode records "staged", never "ingested". - The source fingerprint comes from the exact bytes parsed, so a transcript that grows during import is not stamped as its newer self (the old unscanned contract test is migrated; it pinned that bug). - --reconcile mode (bounded by --limit, default 200; resumable; --dry-run; backs up the state file) prints "reconcile: checked N, present M, re-queued K, not yet checked L". --request-reconcile records a pending pass without calling gbrain; the v1.91.17.0 migration (placeholder name) uses it, and the next ingest run does one bounded pass. - One lock for every state writer; a state-save failure fails the run. The #1802 static pins in regression-1611 move to the new structure (remote-http returns before importBatch; the finally honors preserveStaging). Co-authored-by: afshaker <andrew@phone2.io>
This commit is contained in:
garrytanandafshaker committed
1 parent
96f473abda
commit
97d4e58dcc
11 files changed
+1939
-612
No files matched your search
+723
-598
File diff suppressed because it is too large.
Load diff
@@ -255,6 +255,125 @@ and state recorded under the old name maps to the new one. Run
|
||||
|
||||
---
|
||||
|
||||
## `[memory-ingest] FAILED <path>: <error> (left un-stamped; retried next run)`
|
||||
|
||||
**Problem.** gbrain refused one staged page during `/sync-gbrain`'s memory
|
||||
stage (for example invalid frontmatter or an oversize page).
|
||||
|
||||
**Cause.** `gbrain import` named the page as a per-file failure (in its
|
||||
`--json` `failures` list or a `Skipped <path>:` line). Before this release,
|
||||
such pages were marked ingested and never retried.
|
||||
|
||||
**Fix.** Nothing is lost: the page is not marked ingested and is retried on
|
||||
every run. If the same page keeps failing, read the error, fix the source
|
||||
file it came from, then run `/sync-gbrain`.
|
||||
|
||||
---
|
||||
|
||||
## `[memory-ingest] re-queued <slug>: not found in gbrain source <id> after import (retried next run)`
|
||||
|
||||
**Problem.** gbrain said a page was imported (or unchanged), but the landing
|
||||
check could not find it in the source it was imported into, or found
|
||||
different content.
|
||||
|
||||
**Cause.** Usually a managed import that was accepted but is still
|
||||
publishing (`Pending:` in gbrain's output), or a page gbrain skipped without
|
||||
naming it.
|
||||
|
||||
**Fix.** Nothing to do; the next `/sync-gbrain` imports it again. If it
|
||||
repeats on every run, upgrade gbrain (`gstack-gbrain-install`).
|
||||
|
||||
---
|
||||
|
||||
## `[memory-ingest] ERR: gbrain reported N failure(s) it did not attribute to a staged page ... Refusing to advance state.`
|
||||
|
||||
**Problem.** The memory stage failed and marked nothing ingested.
|
||||
|
||||
**Cause.** gbrain counted failures it did not name (it stops printing them
|
||||
after five of the same kind), so gstack cannot tell which pages landed.
|
||||
|
||||
**Fix.** Re-run `/sync-gbrain`. After three refused runs the batch is split
|
||||
in halves automatically to find the page gbrain cannot import; that page is
|
||||
quarantined (next entry) and the rest import.
|
||||
|
||||
---
|
||||
|
||||
## `[memory-ingest] quarantined <slug> (<source path>): gbrain refuses every batch that contains it (...)`
|
||||
|
||||
**Problem.** One page kept making gbrain refuse the whole batch.
|
||||
|
||||
**Cause.** After three refused batches, gstack imported the batch in halves
|
||||
until it isolated the page. Every other page was imported.
|
||||
|
||||
**Fix.** The page is retried automatically when its source file changes.
|
||||
To retry now, fix or remove the source file named in the message, then run
|
||||
`/sync-gbrain`.
|
||||
|
||||
---
|
||||
|
||||
## `[memory-ingest] reconcile: checked N, present M, re-queued K, not yet checked L (run again to continue)`
|
||||
|
||||
**Problem.** Not an error. This is the one-time catch-up after upgrading:
|
||||
gstack re-checks transcripts it had marked ingested and re-queues the ones
|
||||
missing from the brain, so transcripts lost by older versions import again.
|
||||
|
||||
**Cause.** Older versions could mark a page ingested that gbrain had
|
||||
skipped. The upgrade records that a reconcile is pending; each ingest run
|
||||
then checks a bounded batch (200 entries) until done.
|
||||
|
||||
**Fix.** Nothing to do; each `/sync-gbrain` continues where the last one
|
||||
stopped. To run it now or check progress:
|
||||
```bash
|
||||
gstack-memory-ingest --reconcile --dry-run # report only
|
||||
gstack-memory-ingest --reconcile # re-check the next 200 (--limit N)
|
||||
```
|
||||
"not yet checked" counts pages whose lookup failed (gbrain busy or
|
||||
unreachable); they are never treated as missing. "unrecoverable" counts pages
|
||||
missing from the brain whose transcript file no longer exists. The state file
|
||||
is backed up to `~/.gstack/.transcript-ingest-state.json.pre-reconcile.bak`
|
||||
before each pass.
|
||||
|
||||
---
|
||||
|
||||
## `[memory-ingest] reconcile: not run (...)`
|
||||
|
||||
**Problem.** A reconcile pass was requested but could not run.
|
||||
|
||||
**Cause.** gbrain is not installed or lacks `import`, or the brain is a
|
||||
remote-HTTP brain (its pages are staged for the brain admin's pull, so there
|
||||
is nothing to check locally).
|
||||
|
||||
**Fix.** For a missing gbrain, run `/setup-gbrain`, then
|
||||
`gstack-memory-ingest --reconcile`. For a remote-HTTP brain, nothing to do.
|
||||
|
||||
---
|
||||
|
||||
## `[memory-ingest] ERR: another memory ingest (pid <N>) is writing <state file>; not run.`
|
||||
|
||||
**Problem.** The memory stage did nothing this run.
|
||||
|
||||
**Cause.** Another ingest, reconcile or `--request-reconcile` holds the
|
||||
state lock (`~/.gstack/.transcript-ingest-state.json.lock`). A lock left by a
|
||||
process that no longer exists is taken over automatically.
|
||||
|
||||
**Fix.** Wait for the other run to finish, then run `/sync-gbrain` again.
|
||||
|
||||
---
|
||||
|
||||
## `[memory-ingest] ERR: could not save ingest state <path>: <error>.`
|
||||
|
||||
**Problem.** The memory stage failed after importing.
|
||||
|
||||
**Cause.** The state file could not be written (disk full, permissions, or a
|
||||
directory in its place). The run fails instead of pretending it recorded
|
||||
progress; pages it imported are re-checked next run, which is cheap because
|
||||
gbrain skips unchanged content.
|
||||
|
||||
**Fix.** Free disk space or fix permissions on `~/.gstack/`, then run
|
||||
`/sync-gbrain`.
|
||||
|
||||
---
|
||||
|
||||
## Nothing is syncing but I expect it to
|
||||
|
||||
**Not an error, but a common gotcha.** Check in order:
|
||||
|
||||
Executable
+28
@@ -0,0 +1,28 @@
|
||||
#!/usr/bin/env bash
|
||||
# Migration: v1.91.17.0 — severe fix wave (memory ingest).
|
||||
# Placeholder name: the release queue may rename this file at /ship.
|
||||
#
|
||||
# Each step is independent and guarded. Idempotent and non-fatal: every path
|
||||
# exits 0, and no step calls gbrain or the network unless it has work to do.
|
||||
set -u
|
||||
_gstack_migration_dir="${BASH_SOURCE[0]//\\//}"; _gstack_migration_dir="${_gstack_migration_dir%/*}"
|
||||
. "${_gstack_migration_dir}/../../bin/gstack-state-root.sh" 2>/dev/null || { echo "$0: cannot resolve the gstack state root: ${_gstack_migration_dir}/../../bin/gstack-state-root.sh is missing. fix: reinstall with ./setup or /gstack-upgrade (docs/state-root.md)" >&2; exit 1; }
|
||||
gstack_state_root_select; GH="$_gstack_sr_root"
|
||||
|
||||
# Step 1 (A1, #2778): older versions could mark a transcript ingested that
|
||||
# gbrain had skipped. Record that a reconcile pass is pending; the next memory
|
||||
# ingest re-checks stamped pages in bounded batches and re-queues the missing
|
||||
# ones. This only writes a flag under the ingest's state lock; it never calls
|
||||
# gbrain.
|
||||
if [ -f "$GH/.transcript-ingest-state.json" ]; then
|
||||
INGEST="${_gstack_migration_dir}/../../bin/gstack-memory-ingest.ts"
|
||||
if command -v bun >/dev/null 2>&1 && [ -f "$INGEST" ]; then
|
||||
if bun "$INGEST" --request-reconcile --quiet; then
|
||||
echo "memory ingest: reconcile pending; the next /sync-gbrain re-checks transcripts already marked ingested."
|
||||
else
|
||||
echo "memory ingest: could not record a pending reconcile now; run: gstack-memory-ingest --reconcile" >&2
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
exit 0
|
||||
@@ -0,0 +1,512 @@
|
||||
/**
|
||||
* Memory ingest landing contract (A1, #2778): write, verify, then stamp.
|
||||
*
|
||||
* `gbrain import` exits 0 while skipping pages (returned per-file failures,
|
||||
* malformed filenames, managed-import `Pending:` receipts), and its failure
|
||||
* ledger is written only for git import dirs, so a staged page that never
|
||||
* landed used to be stamped ingested and never retried. This module owns:
|
||||
*
|
||||
* - the versioned ingest state and its transition table,
|
||||
* - parsing `gbrain import --json` plus its stderr into named failures and
|
||||
* a whole-batch refusal when a failure cannot be named,
|
||||
* - the landing check: one `gbrain list` per source per run for presence,
|
||||
* and `gbrain get --json` (bounded per run) for the
|
||||
* `gstack_content_sha256` frontmatter field gstack writes,
|
||||
* - `--reconcile`: a bounded, resumable re-check of stamped entries,
|
||||
* - the single writer lock and fail-loud state saves.
|
||||
*
|
||||
* Capability facts (gbrain is installed at latest HEAD, floor 0.20.0):
|
||||
* `import --json` reports `failures`/`unchanged`/`malformed_skipped` from
|
||||
* 0.48.5.0 and `source_id` from 0.50.1.0; `list`/`get` honor `--source-id`
|
||||
* from 0.46.25. Capability is detected by behavior, never by version string.
|
||||
* A lookup error or a busy PGLite means "not yet checked", never "absent".
|
||||
*/
|
||||
|
||||
import { closeSync, copyFileSync, existsSync, mkdirSync, openSync, readFileSync, renameSync, rmSync, writeFileSync, writeSync } from "fs";
|
||||
import { dirname } from "path";
|
||||
|
||||
// ── State schema ───────────────────────────────────────────────────────────
|
||||
|
||||
export const STATE_SCHEMA_VERSION = 2;
|
||||
/** Source recorded for entries stamped before per-source imports existed. */
|
||||
export const LEGACY_SOURCE_ID = "default";
|
||||
export const DEFAULT_GET_CAP = 50;
|
||||
export const DEFAULT_RECONCILE_LIMIT = 200;
|
||||
export const BISECT_AFTER_REFUSALS = 3;
|
||||
export const LIST_LIMIT = 1_000_000;
|
||||
|
||||
export type EntryStatus =
|
||||
| "staged" // remote-http: written to the publishable staging dir; the brain admin indexes it
|
||||
| "imported_unverified" // import reported it landed; presence/content not yet confirmed
|
||||
| "ingested" // landing check confirmed it in its recorded source
|
||||
| "refused" // stamped earlier, then found absent or different in the brain; re-queued for the next run
|
||||
| "quarantined" // isolated as the page that poisons its batch; retried only when the source changes
|
||||
| "unrecoverable"; // absent from the brain and its source file is gone
|
||||
|
||||
export interface StateEntry {
|
||||
mtime_ns: number;
|
||||
sha256: string;
|
||||
page_slug: string;
|
||||
partial?: boolean;
|
||||
status: EntryStatus;
|
||||
/** gbrain source the page was imported into ("default" for pre-source entries). */
|
||||
source_id: string;
|
||||
/** Value of the gstack_content_sha256 frontmatter field written into the page. */
|
||||
content_sha256?: string;
|
||||
/** When the page was last imported (or staged). */
|
||||
ingested_at?: string;
|
||||
verified_at?: string;
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
export interface IngestState {
|
||||
schema_version: typeof STATE_SCHEMA_VERSION;
|
||||
last_writer: string;
|
||||
last_full_walk?: string;
|
||||
reconcile?: { pending?: boolean; cursor?: string | null; unchecked_in_sweep?: number };
|
||||
/**
|
||||
* Whole-batch refusals per source path (the bisect trigger). A page that did
|
||||
* not land is never stamped, so the count lives outside `sessions`.
|
||||
*/
|
||||
batch_refusals?: Record<string, number>;
|
||||
/** Transcript sources registered by gstack (A4), cached so registration runs once. */
|
||||
sources?: Record<string, { repo: string; registered_at: string }>;
|
||||
sessions: Record<string, StateEntry>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Allowed transitions. "new" is the absence of an entry. Every write goes
|
||||
* through setEntry(), which rejects anything not listed here.
|
||||
*/
|
||||
export const TRANSITIONS: Record<EntryStatus | "new", readonly EntryStatus[]> = {
|
||||
new: ["staged", "imported_unverified", "ingested", "refused", "quarantined"],
|
||||
staged: ["staged", "imported_unverified", "ingested", "refused", "quarantined"],
|
||||
imported_unverified: ["imported_unverified", "ingested", "refused", "quarantined", "unrecoverable", "staged"],
|
||||
ingested: ["ingested", "imported_unverified", "refused", "quarantined", "unrecoverable", "staged"],
|
||||
refused: ["staged", "imported_unverified", "ingested", "refused", "quarantined"],
|
||||
quarantined: ["staged", "imported_unverified", "ingested", "refused", "quarantined"],
|
||||
unrecoverable: ["staged", "imported_unverified", "ingested", "refused", "unrecoverable"],
|
||||
};
|
||||
|
||||
export function setEntry(state: IngestState, path: string, next: StateEntry): void {
|
||||
const from: EntryStatus | "new" = state.sessions[path]?.status ?? "new";
|
||||
if (!TRANSITIONS[from].includes(next.status)) {
|
||||
throw new Error(`ingest state: illegal transition ${from} -> ${next.status} for ${path}`);
|
||||
}
|
||||
state.sessions[path] = next;
|
||||
}
|
||||
|
||||
/** Entries whose page is not considered landed are re-prepared every run. */
|
||||
export function isRetryable(entry: StateEntry | undefined): boolean {
|
||||
return !entry || entry.status === "refused";
|
||||
}
|
||||
|
||||
export function emptyState(): IngestState {
|
||||
return { schema_version: STATE_SCHEMA_VERSION, last_writer: "gstack-memory-ingest", sessions: {} };
|
||||
}
|
||||
|
||||
/**
|
||||
* Migrate a parsed state file. Schema 1 entries were stamped without any
|
||||
* landing check and without a source, so they become `ingested` in the
|
||||
* legacy `default` source and a reconcile pass is marked pending.
|
||||
*/
|
||||
export function migrateState(raw: unknown): { state: IngestState; migrated: boolean } | null {
|
||||
const r = raw as { schema_version?: unknown; sessions?: unknown } | null;
|
||||
if (!r || typeof r !== "object" || typeof r.sessions !== "object" || r.sessions === null) return null;
|
||||
if (r.schema_version === STATE_SCHEMA_VERSION) return { state: r as IngestState, migrated: false };
|
||||
if (r.schema_version !== 1) return null;
|
||||
const v1 = r as { last_writer?: string; last_full_walk?: string; sessions: Record<string, Record<string, unknown>> };
|
||||
const state: IngestState = {
|
||||
...emptyState(),
|
||||
last_full_walk: v1.last_full_walk,
|
||||
reconcile: { pending: true, cursor: null, unchecked_in_sweep: 0 },
|
||||
};
|
||||
for (const [path, e] of Object.entries(v1.sessions)) {
|
||||
if (!e || typeof e !== "object") continue;
|
||||
state.sessions[path] = {
|
||||
mtime_ns: Number(e.mtime_ns) || 0,
|
||||
sha256: String(e.sha256 ?? ""),
|
||||
page_slug: String(e.page_slug ?? ""),
|
||||
...(e.partial ? { partial: true } : {}),
|
||||
status: "ingested",
|
||||
source_id: LEGACY_SOURCE_ID,
|
||||
...(typeof e.ingested_at === "string" ? { ingested_at: e.ingested_at } : {}),
|
||||
};
|
||||
}
|
||||
return { state, migrated: true };
|
||||
}
|
||||
|
||||
export type LoadedState =
|
||||
| { kind: "ok"; state: IngestState; migrated: boolean }
|
||||
| { kind: "absent"; state: IngestState }
|
||||
| { kind: "unreadable"; state: IngestState; backup: string | null; reason: string };
|
||||
|
||||
/** Load the state file. An unreadable or unknown file is backed up, never silently reused. */
|
||||
export function loadStateFile(path: string): LoadedState {
|
||||
if (!existsSync(path)) return { kind: "absent", state: emptyState() };
|
||||
let raw: string;
|
||||
try {
|
||||
raw = readFileSync(path, "utf-8");
|
||||
} catch (err) {
|
||||
return { kind: "unreadable", state: emptyState(), backup: null, reason: (err as Error).message };
|
||||
}
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(raw);
|
||||
} catch {
|
||||
return { kind: "unreadable", state: emptyState(), backup: backupStateFile(path, `${path}.bak`), reason: "corrupt JSON" };
|
||||
}
|
||||
const migrated = migrateState(parsed);
|
||||
if (!migrated) {
|
||||
const v = (parsed as { schema_version?: unknown } | null)?.schema_version;
|
||||
return { kind: "unreadable", state: emptyState(), backup: backupStateFile(path, `${path}.bak`), reason: `unknown schema_version ${String(v)}` };
|
||||
}
|
||||
return { kind: "ok", ...migrated };
|
||||
}
|
||||
|
||||
/** Atomic save. Throws on failure: a run whose state could not be saved has failed. */
|
||||
export function saveStateFile(path: string, state: IngestState): void {
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
const tmp = `${path}.tmp.${process.pid}`;
|
||||
try {
|
||||
writeFileSync(tmp, JSON.stringify(state, null, 2), "utf-8");
|
||||
renameSync(tmp, path);
|
||||
} catch (err) {
|
||||
try { rmSync(tmp, { force: true }); } catch {}
|
||||
throw new Error(`could not save ingest state ${path}: ${(err as Error).message}`);
|
||||
}
|
||||
}
|
||||
|
||||
export function backupStateFile(path: string, dest: string): string | null {
|
||||
if (!existsSync(path)) return null;
|
||||
try {
|
||||
copyFileSync(path, dest);
|
||||
return dest;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// ── Single writer lock ─────────────────────────────────────────────────────
|
||||
|
||||
function pidAlive(pid: number): boolean {
|
||||
if (!Number.isInteger(pid) || pid <= 0) return false;
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch (err) {
|
||||
return (err as NodeJS.ErrnoException).code === "EPERM";
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* One lock for every state writer (ingest, reconcile, the reconcile-pending
|
||||
* flag). A lock left by a dead process is taken over.
|
||||
*/
|
||||
export function acquireStateLock(lockPath: string): { ok: true; release: () => void } | { ok: false; holder: number } {
|
||||
mkdirSync(dirname(lockPath), { recursive: true });
|
||||
for (let attempt = 0; attempt < 2; attempt++) {
|
||||
try {
|
||||
const fd = openSync(lockPath, "wx");
|
||||
writeSync(fd, `${process.pid}\n`);
|
||||
closeSync(fd);
|
||||
return {
|
||||
ok: true,
|
||||
release: () => {
|
||||
try {
|
||||
if (readFileSync(lockPath, "utf-8").trim() === String(process.pid)) rmSync(lockPath, { force: true });
|
||||
} catch {}
|
||||
},
|
||||
};
|
||||
} catch (err) {
|
||||
if ((err as NodeJS.ErrnoException).code !== "EEXIST") throw err;
|
||||
const holder = Number.parseInt(readFileSync(lockPath, "utf-8").trim(), 10);
|
||||
if (pidAlive(holder) && holder !== process.pid) return { ok: false, holder };
|
||||
rmSync(lockPath, { force: true });
|
||||
}
|
||||
}
|
||||
return { ok: false, holder: -1 };
|
||||
}
|
||||
|
||||
// ── gbrain import result ───────────────────────────────────────────────────
|
||||
|
||||
export interface ImportReport {
|
||||
status?: string;
|
||||
imported?: number;
|
||||
skipped?: number;
|
||||
errors?: number;
|
||||
unchanged?: number;
|
||||
malformed_skipped?: number;
|
||||
failures?: Array<{ path?: unknown; error?: unknown }>;
|
||||
source_id?: string;
|
||||
total_files?: number;
|
||||
}
|
||||
|
||||
/** The `--json` summary object on the last JSON line, or null when absent. */
|
||||
export function parseImportReport(stdout: string): ImportReport | null {
|
||||
const lines = stdout.split("\n").map((s) => s.trim()).filter(Boolean);
|
||||
for (let i = lines.length - 1; i >= 0; i--) {
|
||||
const line = lines[i];
|
||||
if (!line.startsWith("{") || !line.endsWith("}")) continue;
|
||||
try {
|
||||
const parsed = JSON.parse(line);
|
||||
if (parsed && typeof parsed === "object" && "imported" in parsed) return parsed as ImportReport;
|
||||
} catch {}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-file failures gbrain prints on stderr: ` Skipped <rel>: <err>` for a
|
||||
* failure importFile returns (printed every time), ` Warning: skipped <rel>:
|
||||
* <err>` for a thrown one (only the first 5 per error class), and
|
||||
* ` Skipped (malformed filename — rename to import): <rel>`.
|
||||
*/
|
||||
export function parseStderrFailures(stderr: string): Array<{ path: string; error: string; thrown: boolean }> {
|
||||
const out: Array<{ path: string; error: string; thrown: boolean }> = [];
|
||||
for (const raw of stderr.split("\n")) {
|
||||
const line = raw.replace(/\r$/, "");
|
||||
let m = line.match(/^\s+Warning: skipped (\S+): (.*)$/);
|
||||
if (m) { out.push({ path: m[1], error: m[2], thrown: true }); continue; }
|
||||
m = line.match(/^\s+Skipped \(malformed filename[^)]*\): (.*)$/);
|
||||
if (m) { out.push({ path: m[1].trim(), error: "malformed filename", thrown: false }); continue; }
|
||||
m = line.match(/^\s+Skipped (\S+): (.*)$/);
|
||||
if (m) out.push({ path: m[1], error: m[2], thrown: false });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export interface BatchVerdict {
|
||||
/** Staged rel path → gbrain's error, for every failure gbrain named. */
|
||||
named: Map<string, string>;
|
||||
/** Set when a failure cannot be attributed to a page: the whole batch is refused. */
|
||||
refuseAll?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide which staged pages failed. Named failures come from `--json`
|
||||
* `failures`, stderr and the sync-failures ledger. Any failure that cannot
|
||||
* be named (gbrain stops printing thrown failures after 5 per class, or a
|
||||
* staged page was never collected) refuses the whole batch.
|
||||
*/
|
||||
export function classifyImport(
|
||||
report: ImportReport,
|
||||
stderr: string,
|
||||
ledgerPaths: Iterable<string>,
|
||||
staged: ReadonlySet<string>,
|
||||
/** Files gbrain should account for when it differs from the mapped pages (a resumed stage). */
|
||||
stagedTotal: number = staged.size,
|
||||
): BatchVerdict {
|
||||
const named = new Map<string, string>();
|
||||
const unmapped: string[] = [];
|
||||
const note = (path: string, error: string) => {
|
||||
if (staged.has(path)) {
|
||||
if (!named.has(path)) named.set(path, error);
|
||||
} else if (!unmapped.includes(path)) unmapped.push(path);
|
||||
};
|
||||
const jsonFailures = Array.isArray(report.failures) ? report.failures : null;
|
||||
for (const f of jsonFailures ?? []) {
|
||||
if (typeof f?.path === "string") note(f.path, String(f.error ?? "unknown error"));
|
||||
}
|
||||
const stderrFailures = parseStderrFailures(stderr);
|
||||
for (const f of stderrFailures) note(f.path, f.error);
|
||||
// The ledger is shared by every gbrain process, so only entries naming a
|
||||
// page of this batch count; it is complete for the run when present.
|
||||
let ledgerCount = 0;
|
||||
for (const p of ledgerPaths) {
|
||||
if (!staged.has(p)) continue;
|
||||
ledgerCount++;
|
||||
if (!named.has(p)) named.set(p, "recorded in ~/.gbrain/sync-failures.jsonl");
|
||||
}
|
||||
|
||||
const errors = report.errors ?? 0;
|
||||
// With `failures` (gbrain >= 0.48.5.0) every error is listed there; before
|
||||
// that, `errors` counted thrown failures, named by "Warning: skipped" lines
|
||||
// (the first 5 per class) or by the ledger for git import dirs.
|
||||
const namedErrors = jsonFailures
|
||||
? jsonFailures.length
|
||||
: Math.max(stderrFailures.filter((f) => f.thrown).length, ledgerCount);
|
||||
const hidden = Math.max(0, errors - namedErrors);
|
||||
const accounted = (report.imported ?? 0) + (report.skipped ?? 0);
|
||||
const expected = stagedTotal - named.size;
|
||||
let refuseAll: string | undefined;
|
||||
if (accounted < expected) {
|
||||
refuseAll =
|
||||
`gbrain import accounted for ${accounted} of ${expected} staged page(s) ` +
|
||||
`(imported=${report.imported ?? 0}, skipped=${report.skipped ?? 0})` +
|
||||
(report.total_files !== undefined ? `; gbrain collected ${report.total_files} file(s)` : "");
|
||||
} else if (hidden > 0 || unmapped.length > 0) {
|
||||
refuseAll =
|
||||
`gbrain reported ${hidden + unmapped.length} failure(s) it did not attribute to a staged page` +
|
||||
(unmapped.length > 0 ? ` (${unmapped.slice(0, 5).join(", ")})` : "");
|
||||
}
|
||||
return { named, ...(refuseAll ? { refuseAll } : {}) };
|
||||
}
|
||||
|
||||
// ── Landing check ──────────────────────────────────────────────────────────
|
||||
|
||||
export type GbrainRun = (args: string[]) => { status: number | null; stdout: string; stderr: string };
|
||||
|
||||
/** The slug gbrain stores for a staged path (gbrain core/sync.ts slugifyPath). */
|
||||
export function gbrainSlug(slug: string): string {
|
||||
return slug
|
||||
.replace(/\.mdx?$/i, "")
|
||||
.split("/")
|
||||
.map((seg) =>
|
||||
seg
|
||||
.normalize("NFD")
|
||||
.replace(/[\u0300-\u036f]/g, "")
|
||||
.normalize("NFC")
|
||||
.toLowerCase()
|
||||
.replace(/[^\p{L}\p{N}\p{M}.\s_-]/gu, "")
|
||||
.replace(/\s+/g, "-")
|
||||
.replace(/-+/g, "-")
|
||||
.replace(/^-|-$/g, ""),
|
||||
)
|
||||
.filter(Boolean)
|
||||
.join("/");
|
||||
}
|
||||
|
||||
export interface LandingLookups {
|
||||
/** Presence per source: slug set, or null when the lookup failed (not yet checked). */
|
||||
list(sourceId: string): Set<string> | null;
|
||||
/** The page's gstack_content_sha256, null when absent from the page, undefined when the lookup failed. */
|
||||
contentSha(sourceId: string, slug: string): string | null | undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* gbrain-backed lookups with a per-run list cache. `list --source-id` is
|
||||
* probed once with an id that cannot exist: a gbrain that ignores the flag
|
||||
* answers it, and its unscoped listing cannot prove absence from a source.
|
||||
*/
|
||||
export function gbrainLookups(run: GbrainRun): LandingLookups {
|
||||
const cache = new Map<string, Set<string> | null>();
|
||||
let scoped: boolean | null = null;
|
||||
return {
|
||||
list(sourceId) {
|
||||
if (cache.has(sourceId)) return cache.get(sourceId)!;
|
||||
if (scoped === null) scoped = run(["list", "--source-id", "gstack-capability-probe-0", "--limit", "1"]).status !== 0;
|
||||
let slugs: Set<string> | null = null;
|
||||
if (scoped) {
|
||||
const r = run(["list", "--source-id", sourceId, "--limit", String(LIST_LIMIT), "--sort", "slug"]);
|
||||
if (r.status === 0 && !r.stdout.trimStart().startsWith("{")) {
|
||||
slugs = new Set();
|
||||
for (const line of r.stdout.split("\n")) {
|
||||
const slug = line.split("\t")[0]?.trim();
|
||||
if (slug && line.includes("\t")) slugs.add(slug);
|
||||
}
|
||||
}
|
||||
}
|
||||
cache.set(sourceId, slugs);
|
||||
return slugs;
|
||||
},
|
||||
contentSha(sourceId, slug) {
|
||||
const r = run(["get", slug, "--source-id", sourceId, "--json"]);
|
||||
if (r.status !== 0) return undefined;
|
||||
try {
|
||||
const page = JSON.parse(r.stdout) as { slug?: string; frontmatter?: Record<string, unknown> };
|
||||
if (!page || typeof page !== "object" || typeof page.slug !== "string") return undefined;
|
||||
const v = page.frontmatter?.gstack_content_sha256;
|
||||
return typeof v === "string" ? v : null;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export type LandingOutcome = "ingested" | "absent" | "mismatch" | "unchecked";
|
||||
|
||||
/**
|
||||
* Check one entry against its recorded source. Presence comes from the
|
||||
* source's list; content is checked with `get` while `budget.gets` lasts.
|
||||
* Entries without a content hash (stamped before this release) are
|
||||
* confirmed by presence alone.
|
||||
*/
|
||||
export function checkLanding(entry: StateEntry, lookups: LandingLookups, budget: { gets: number }): LandingOutcome {
|
||||
const slugs = lookups.list(entry.source_id);
|
||||
if (!slugs) return "unchecked";
|
||||
const slug = gbrainSlug(entry.page_slug);
|
||||
if (!slugs.has(slug)) return "absent";
|
||||
if (!entry.content_sha256) return "ingested";
|
||||
if (budget.gets <= 0) return "unchecked";
|
||||
budget.gets--;
|
||||
const sha = lookups.contentSha(entry.source_id, slug);
|
||||
if (sha === undefined) return "unchecked";
|
||||
return sha === entry.content_sha256 ? "ingested" : "mismatch";
|
||||
}
|
||||
|
||||
// ── Reconcile ──────────────────────────────────────────────────────────────
|
||||
|
||||
export interface ReconcileSummary {
|
||||
checked: number;
|
||||
present: number;
|
||||
requeued: number;
|
||||
notChecked: number;
|
||||
unrecoverable: number;
|
||||
/** True when this pass finished a full sweep with nothing left unchecked. */
|
||||
complete: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Re-check up to `limit` stamped entries (resuming after the stored cursor)
|
||||
* against their recorded source. A page missing from its source is
|
||||
* re-queued when its transcript still exists, otherwise counted
|
||||
* unrecoverable; pages are never moved between sources. A dry run computes
|
||||
* the same summary without changing state.
|
||||
*/
|
||||
export function reconcile(
|
||||
state: IngestState,
|
||||
lookups: LandingLookups,
|
||||
opts: { limit: number; dryRun: boolean; fileExists: (path: string) => boolean; getCap?: number },
|
||||
): ReconcileSummary {
|
||||
const rec = state.reconcile ?? {};
|
||||
const cursor = rec.cursor ?? null;
|
||||
const eligible = Object.keys(state.sessions)
|
||||
.filter((p) => ["ingested", "imported_unverified"].includes(state.sessions[p].status))
|
||||
.sort();
|
||||
const pending = eligible.filter((p) => cursor === null || p > cursor);
|
||||
const batch = pending.slice(0, opts.limit);
|
||||
const budget = { gets: opts.getCap ?? DEFAULT_GET_CAP };
|
||||
const s: ReconcileSummary = { checked: 0, present: 0, requeued: 0, notChecked: 0, unrecoverable: 0, complete: false };
|
||||
const now = new Date().toISOString();
|
||||
for (const path of batch) {
|
||||
const entry = state.sessions[path];
|
||||
const outcome = checkLanding(entry, lookups, budget);
|
||||
if (outcome === "unchecked") {
|
||||
s.notChecked++;
|
||||
continue;
|
||||
}
|
||||
s.checked++;
|
||||
if (outcome === "ingested") {
|
||||
s.present++;
|
||||
if (!opts.dryRun && entry.status !== "ingested") setEntry(state, path, { ...entry, status: "ingested", verified_at: now });
|
||||
continue;
|
||||
}
|
||||
const exists = opts.fileExists(path);
|
||||
if (exists) s.requeued++;
|
||||
else s.unrecoverable++;
|
||||
if (opts.dryRun) continue;
|
||||
setEntry(state, path, exists
|
||||
? { ...entry, status: "refused", reason: outcome === "absent" ? `absent from source ${entry.source_id} (reconcile)` : "content differs from the staged page (reconcile)" }
|
||||
: { ...entry, status: "unrecoverable", reason: `absent from source ${entry.source_id} and the source file is gone` });
|
||||
}
|
||||
const sweepDone = batch.length === pending.length;
|
||||
const unchecked = (rec.unchecked_in_sweep ?? 0) + s.notChecked;
|
||||
s.complete = sweepDone && unchecked === 0;
|
||||
if (!opts.dryRun) {
|
||||
state.reconcile = sweepDone
|
||||
? { pending: !s.complete, cursor: null, unchecked_in_sweep: 0 }
|
||||
: { pending: true, cursor: batch[batch.length - 1] ?? cursor, unchecked_in_sweep: unchecked };
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
export function formatReconcileSummary(s: ReconcileSummary, dryRun: boolean): string {
|
||||
return (
|
||||
`reconcile${dryRun ? " (dry run)" : ""}: checked ${s.checked}, present ${s.present}, ` +
|
||||
`re-queued ${s.requeued}, not yet checked ${s.notChecked}` +
|
||||
(s.unrecoverable > 0 ? `, unrecoverable ${s.unrecoverable} (source file gone)` : "") +
|
||||
(s.complete ? "" : " (run again to continue)")
|
||||
);
|
||||
}
|
||||
+2
-1
@@ -31,7 +31,8 @@
|
||||
{ "file": "test/helpers/pty/runners/floor.ts", "movedFrom": "test/helpers/claude-pty-runner.ts" },
|
||||
{ "file": "test/helpers/pty/runners/observation.ts", "movedFrom": "test/helpers/claude-pty-runner.ts" },
|
||||
{ "file": "test/helpers/pty/screen.ts", "movedFrom": "test/helpers/pty-screen.ts" },
|
||||
{ "file": "test/helpers/pty/session.ts", "movedFrom": "test/helpers/claude-pty-runner.ts" }
|
||||
{ "file": "test/helpers/pty/session.ts", "movedFrom": "test/helpers/claude-pty-runner.ts" },
|
||||
{ "file": "lib/memory-ingest-landing.ts", "movedFrom": "(new: A1 memory ingest landing check, state transitions and reconcile)" }
|
||||
],
|
||||
"residualFiles": [
|
||||
{ "file": "scripts/test-free-shards.ts", "maxLines": 2361, "reason": "free lane policy after the shard engine took spawn, sandbox, logs, seeds and flags" },
|
||||
|
||||
@@ -482,11 +482,18 @@ if (process.env.LIMIT_STAGE_WRITES === '1') {
|
||||
expect(readFileSync(join(home, "imports"), "utf8").trim().split("\n")).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("keeps the no-scan import stamping contract unchanged", () => {
|
||||
// A1 changes the old no-scan contract on purpose: without --scan-secrets
|
||||
// the stamp used to describe the file as it was AFTER the import (its
|
||||
// newer self), so the appended record was never imported.
|
||||
it("does not stamp an append during an unscanned import, and the next run imports it (A1)", () => {
|
||||
const path = source();
|
||||
env.APPEND_DURING_IMPORT = path;
|
||||
env.APPEND_RECORD = appendRecord();
|
||||
expect(run().status).toBe(0);
|
||||
expect(sessions()[path]).toBeUndefined();
|
||||
delete env.APPEND_DURING_IMPORT;
|
||||
expect(run().status).toBe(0);
|
||||
expect(imported()[0].body).toContain("late ordinary update");
|
||||
expect(sessions()[path]).toMatchObject({ sha256: createHash("sha256").update(readFileSync(path)).digest("hex") });
|
||||
});
|
||||
});
|
||||
@@ -958,7 +965,7 @@ describe("gstack-memory-ingest CLI", () => {
|
||||
// ── State file behavior ────────────────────────────────────────────────────
|
||||
|
||||
describe("gstack-memory-ingest state file", () => {
|
||||
it("--incremental on empty home creates state file with schema_version: 1", () => {
|
||||
it("--incremental on empty home creates state file with schema_version: 2", () => {
|
||||
const home = makeTestHome();
|
||||
const gstackHome = join(home, ".gstack");
|
||||
mkdirSync(gstackHome, { recursive: true });
|
||||
@@ -967,7 +974,7 @@ describe("gstack-memory-ingest state file", () => {
|
||||
const statePath = join(gstackHome, ".transcript-ingest-state.json");
|
||||
expect(existsSync(statePath)).toBe(true);
|
||||
const state = JSON.parse(readFileSync(statePath, "utf-8"));
|
||||
expect(state.schema_version).toBe(1);
|
||||
expect(state.schema_version).toBe(2);
|
||||
expect(state.last_writer).toBe("gstack-memory-ingest");
|
||||
rmSync(home, { recursive: true, force: true });
|
||||
});
|
||||
@@ -984,7 +991,7 @@ describe("gstack-memory-ingest state file", () => {
|
||||
expect(existsSync(statePath + ".bak")).toBe(true);
|
||||
|
||||
const fresh = JSON.parse(readFileSync(statePath, "utf-8"));
|
||||
expect(fresh.schema_version).toBe(1);
|
||||
expect(fresh.schema_version).toBe(2);
|
||||
rmSync(home, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
/**
|
||||
* v1.91.17.0 migration (placeholder name; the release queue may rename it):
|
||||
* step 1 records a pending memory-ingest reconcile (A1) without calling gbrain.
|
||||
*/
|
||||
import { describe, test, expect, beforeEach, afterEach } from "bun:test";
|
||||
import * as fs from "fs";
|
||||
import * as os from "os";
|
||||
import * as path from "path";
|
||||
import { spawnSync } from "child_process";
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, "..");
|
||||
const MIGRATION = path.join(ROOT, "gstack-upgrade", "migrations", "v1.91.17.0.sh");
|
||||
|
||||
let home: string;
|
||||
let gstackHome: string;
|
||||
let bin: string;
|
||||
|
||||
beforeEach(() => {
|
||||
home = fs.mkdtempSync(path.join(os.tmpdir(), "mig-v1.91.17-"));
|
||||
gstackHome = path.join(home, ".gstack");
|
||||
bin = path.join(home, "bin");
|
||||
fs.mkdirSync(bin, { recursive: true });
|
||||
fs.writeFileSync(path.join(bin, "gbrain"), `#!/bin/sh\necho "$@" >> "${home}/gbrain-calls.log"\nexit 1\n`, { mode: 0o755 });
|
||||
});
|
||||
afterEach(() => fs.rmSync(home, { recursive: true, force: true }));
|
||||
|
||||
function run() {
|
||||
const r = spawnSync("bash", [MIGRATION], {
|
||||
env: { PATH: `${bin}:${process.env.PATH}`, HOME: home, GSTACK_HOME: gstackHome },
|
||||
encoding: "utf-8",
|
||||
cwd: home,
|
||||
timeout: 30_000,
|
||||
});
|
||||
return { code: r.status, stdout: r.stdout || "", stderr: r.stderr || "" };
|
||||
}
|
||||
|
||||
describe("v1.91.17.0 migration: memory reconcile pending (A1)", () => {
|
||||
test("marks a reconcile pending on an existing ingest state and never calls gbrain", () => {
|
||||
fs.mkdirSync(gstackHome, { recursive: true });
|
||||
const statePath = path.join(gstackHome, ".transcript-ingest-state.json");
|
||||
fs.writeFileSync(statePath, JSON.stringify({
|
||||
schema_version: 1, last_writer: "gstack-memory-ingest",
|
||||
sessions: { "/t/a.jsonl": { mtime_ns: 1, sha256: "h", ingested_at: "2026-09-01T00:00:00Z", page_slug: "transcripts/x/a" } },
|
||||
}));
|
||||
const first = run();
|
||||
expect(first.code).toBe(0);
|
||||
expect(first.stdout).toContain("reconcile pending");
|
||||
const state = JSON.parse(fs.readFileSync(statePath, "utf-8"));
|
||||
expect(state.schema_version).toBe(2);
|
||||
expect(state.reconcile.pending).toBe(true);
|
||||
expect(state.sessions["/t/a.jsonl"]).toMatchObject({ status: "ingested", source_id: "default" });
|
||||
expect(run().code).toBe(0);
|
||||
expect(fs.existsSync(path.join(home, "gbrain-calls.log"))).toBe(false);
|
||||
});
|
||||
|
||||
test("is a no-op without an ingest state", () => {
|
||||
expect(run().code).toBe(0);
|
||||
expect(fs.existsSync(path.join(gstackHome, ".transcript-ingest-state.json"))).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,117 @@
|
||||
/**
|
||||
* A fake `gbrain` CLI backed by a JSON file, for memory-ingest landing tests
|
||||
* (A1/A4). It speaks the gbrain HEAD (0.60) shapes gstack relies on:
|
||||
* `import <dir> --json [--source-id]`, `list --source-id --limit`,
|
||||
* `get <slug> --source-id --json`, `sources list --json`, `sources add`.
|
||||
*
|
||||
* Behavior switches (env, read by the fake at run time):
|
||||
* FAKE_FAIL=<substr> pages whose path contains it fail by name (HEAD `failures`)
|
||||
* FAKE_PENDING=<substr> accepted as a managed-import Pending receipt: counted, never stored
|
||||
* FAKE_POISON=<substr> any batch containing it fails without naming a page
|
||||
* FAKE_OLD_JSON=1 pre-0.48: no failures/unchanged/source_id keys, and a
|
||||
* returned failure counts only as skipped (exit 0)
|
||||
* FAKE_LIST_UNSCOPED=1 `list` ignores --source-id (pre-0.46.25)
|
||||
* FAKE_BUSY=1 list/get answer {"error":"pglite_busy"} with exit 1
|
||||
* FAKE_NO_SOURCES=1 `sources add` is unsupported
|
||||
* Every call is appended to $HOME/gbrain-calls.log.
|
||||
*/
|
||||
import { chmodSync, mkdirSync, writeFileSync } from "fs";
|
||||
import { join } from "path";
|
||||
|
||||
export function installFakeBrain(home: string): { binDir: string; brainFile: string; callLog: string } {
|
||||
const binDir = join(home, "fake-brain-bin");
|
||||
mkdirSync(binDir, { recursive: true });
|
||||
const brainFile = join(home, "fake-brain.json");
|
||||
const callLog = join(home, "gbrain-calls.log");
|
||||
writeFileSync(join(binDir, "gbrain"), `#!${process.execPath}
|
||||
const fs = require("fs");
|
||||
const path = require("path");
|
||||
const args = process.argv.slice(2);
|
||||
const env = process.env;
|
||||
fs.appendFileSync(${JSON.stringify(callLog)}, JSON.stringify(args) + "\\n");
|
||||
const brainFile = ${JSON.stringify(brainFile)};
|
||||
const brain = fs.existsSync(brainFile) ? JSON.parse(fs.readFileSync(brainFile, "utf8")) : { sources: { default: { federated: true } }, pages: {} };
|
||||
const save = () => fs.writeFileSync(brainFile, JSON.stringify(brain, null, 2));
|
||||
const flag = (name) => { const i = args.indexOf(name); return i === -1 ? undefined : args[i + 1]; };
|
||||
const busy = () => { if (env.FAKE_BUSY === "1") { console.log(JSON.stringify({ error: "pglite_busy" })); process.exit(1); } };
|
||||
const cmd = args[0];
|
||||
if (cmd === "--help") { console.log("Commands:\\n import <dir> Import"); process.exit(0); }
|
||||
if (cmd === "import") {
|
||||
const dir = args[1];
|
||||
const source = flag("--source-id") ?? "default";
|
||||
if (!brain.sources[source]) { console.error("Source not found: " + source); process.exit(1); }
|
||||
const files = [];
|
||||
(function walk(d) {
|
||||
for (const e of fs.readdirSync(d, { withFileTypes: true })) {
|
||||
if (e.name.startsWith(".")) continue;
|
||||
const p = path.join(d, e.name);
|
||||
if (e.isDirectory()) walk(p); else if (e.name.endsWith(".md")) files.push(path.relative(dir, p));
|
||||
}
|
||||
})(dir);
|
||||
if (env.FAKE_POISON && files.some((f) => f.includes(env.FAKE_POISON))) {
|
||||
console.error(" (suppressing further errors)");
|
||||
console.log(JSON.stringify({ status: "partial", imported: 0, skipped: files.length, errors: 1, unchanged: files.length - 1, failures: [], source_id: source, total_files: files.length }));
|
||||
process.exit(1);
|
||||
}
|
||||
let imported = 0, skipped = 0, errors = 0;
|
||||
const failures = [];
|
||||
for (const rel of files) {
|
||||
if (env.FAKE_FAIL && rel.includes(env.FAKE_FAIL)) {
|
||||
// Before 0.48.5.0 a returned per-file failure counted only as skipped (exit 0).
|
||||
skipped++; if (env.FAKE_OLD_JSON !== "1") errors++;
|
||||
failures.push({ path: rel, error: "Invalid YAML frontmatter" });
|
||||
console.error(" Skipped " + rel + ": Invalid YAML frontmatter");
|
||||
continue;
|
||||
}
|
||||
if (env.FAKE_PENDING && rel.includes(env.FAKE_PENDING)) {
|
||||
skipped++;
|
||||
console.error(" Pending: " + rel + " was accepted and is still publishing; rerun to confirm it.");
|
||||
continue;
|
||||
}
|
||||
const body = fs.readFileSync(path.join(dir, rel), "utf8");
|
||||
const sha = (body.match(/^gstack_content_sha256: (\\S+)$/m) || [])[1] ?? null;
|
||||
const slug = rel.replace(/\\.md$/, "").toLowerCase();
|
||||
brain.pages[source] ??= {};
|
||||
if (brain.pages[source][slug]?.sha === sha) skipped++; else imported++;
|
||||
brain.pages[source][slug] = { sha };
|
||||
}
|
||||
save();
|
||||
const report = env.FAKE_OLD_JSON === "1"
|
||||
? { status: "success", imported, skipped, errors, total_files: files.length }
|
||||
: { status: errors ? "partial" : "success", imported, skipped, errors, unchanged: skipped - failures.length, malformed_skipped: 0, failures, source_id: source, total_files: files.length };
|
||||
console.log(JSON.stringify(report));
|
||||
process.exit(errors ? 1 : 0);
|
||||
}
|
||||
if (cmd === "list") {
|
||||
busy();
|
||||
const source = env.FAKE_LIST_UNSCOPED === "1" ? undefined : flag("--source-id");
|
||||
if (source !== undefined && !brain.sources[source]) { console.error("Source not found: " + source); process.exit(1); }
|
||||
const slugs = Object.entries(brain.pages).filter(([s]) => source === undefined || s === source).flatMap(([, p]) => Object.keys(p));
|
||||
console.log(slugs.length ? slugs.map((s) => s + "\\ttranscript\\t2026-10-03\\t" + s).join("\\n") : "No pages found.");
|
||||
process.exit(0);
|
||||
}
|
||||
if (cmd === "get") {
|
||||
busy();
|
||||
const source = flag("--source-id") ?? "default";
|
||||
const page = brain.pages[source]?.[args[1]];
|
||||
if (!page) { console.error("Page not found: " + args[1]); process.exit(1); }
|
||||
console.log(JSON.stringify({ slug: args[1], source_id: source, frontmatter: page.sha ? { gstack_content_sha256: page.sha } : {} }));
|
||||
process.exit(0);
|
||||
}
|
||||
if (cmd === "sources" && args[1] === "list") {
|
||||
console.log(JSON.stringify({ sources: Object.entries(brain.sources).map(([id, s]) => ({ id, federated: s.federated, page_count: Object.keys(brain.pages[id] ?? {}).length })) }));
|
||||
process.exit(0);
|
||||
}
|
||||
if (cmd === "sources" && args[1] === "add") {
|
||||
if (env.FAKE_NO_SOURCES === "1") { console.error("Unknown command: sources add"); process.exit(2); }
|
||||
brain.sources[args[2]] = { federated: args.includes("--federated") };
|
||||
save();
|
||||
console.log("Source added: " + args[2]);
|
||||
process.exit(0);
|
||||
}
|
||||
console.error("Unknown command: " + cmd);
|
||||
process.exit(2);
|
||||
`);
|
||||
chmodSync(join(binDir, "gbrain"), 0o755);
|
||||
return { binDir, brainFile, callLog };
|
||||
}
|
||||
@@ -0,0 +1,355 @@
|
||||
/**
|
||||
* A1 (#2778): memory ingest stamps a page only after confirming it landed.
|
||||
*
|
||||
* Unit tests drive lib/memory-ingest-landing.ts directly; CLI tests run
|
||||
* bin/gstack-memory-ingest.ts against a fake gbrain that keeps a JSON brain
|
||||
* and speaks gbrain HEAD's import/list/get shapes (test/helpers/fake-gbrain-brain.ts).
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach } from "bun:test";
|
||||
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync, readdirSync } from "fs";
|
||||
import { tmpdir } from "os";
|
||||
import { join } from "path";
|
||||
import { spawn, spawnSync } from "child_process";
|
||||
import {
|
||||
acquireStateLock,
|
||||
checkLanding,
|
||||
classifyImport,
|
||||
formatReconcileSummary,
|
||||
gbrainLookups,
|
||||
migrateState,
|
||||
reconcile,
|
||||
saveStateFile,
|
||||
setEntry,
|
||||
emptyState,
|
||||
type IngestState,
|
||||
type LandingLookups,
|
||||
type StateEntry,
|
||||
} from "../lib/memory-ingest-landing";
|
||||
import { installFakeBrain } from "./helpers/fake-gbrain-brain";
|
||||
|
||||
const SCRIPT = join(import.meta.dir, "..", "bin", "gstack-memory-ingest.ts");
|
||||
|
||||
const entry = (over: Partial<StateEntry> = {}): StateEntry => ({
|
||||
mtime_ns: 1, sha256: "s", page_slug: "ceo-plans/demo/2026-10-03-a", status: "ingested", source_id: "default", ...over,
|
||||
});
|
||||
|
||||
function lookupsFrom(pages: Record<string, Record<string, string | null>> | null, opts: { busyGet?: boolean } = {}): LandingLookups & { gets: number } {
|
||||
const l = {
|
||||
gets: 0,
|
||||
list: (source: string) => (pages ? new Set(Object.keys(pages[source] ?? {})) : null),
|
||||
contentSha: (source: string, slug: string) => {
|
||||
l.gets++;
|
||||
if (opts.busyGet) return undefined;
|
||||
const v = pages?.[source]?.[slug];
|
||||
return v === undefined ? undefined : v;
|
||||
},
|
||||
};
|
||||
return l;
|
||||
}
|
||||
|
||||
describe("landing state (unit)", () => {
|
||||
it("migrates a schema-1 file: entries become ingested in the legacy default source, reconcile pending", () => {
|
||||
const m = migrateState({ schema_version: 1, last_writer: "x", sessions: { "/a.jsonl": { mtime_ns: 5, sha256: "h", ingested_at: "t", page_slug: "p" } } });
|
||||
expect(m?.migrated).toBe(true);
|
||||
expect(m?.state.schema_version).toBe(2);
|
||||
expect(m?.state.sessions["/a.jsonl"]).toMatchObject({ status: "ingested", source_id: "default", page_slug: "p", ingested_at: "t" });
|
||||
expect(m?.state.reconcile?.pending).toBe(true);
|
||||
expect(migrateState({ schema_version: 999, sessions: {} })).toBeNull();
|
||||
});
|
||||
|
||||
it("rejects transitions outside the table", () => {
|
||||
const state = emptyState();
|
||||
setEntry(state, "/a", entry({ status: "ingested" }));
|
||||
setEntry(state, "/a", entry({ status: "unrecoverable" }));
|
||||
expect(() => setEntry(state, "/a", entry({ status: "quarantined" }))).toThrow(/illegal transition unrecoverable -> quarantined/);
|
||||
});
|
||||
|
||||
it("names HEAD --json failures and treats Pending receipts as counted, not failed", () => {
|
||||
const staged = new Set(["a.md", "b.md", "c.md"]);
|
||||
const v = classifyImport(
|
||||
{ imported: 1, skipped: 2, errors: 1, unchanged: 1, failures: [{ path: "b.md", error: "Invalid YAML" }], source_id: "default" },
|
||||
" Skipped b.md: Invalid YAML\n Pending: c.md was accepted and is still publishing; rerun to confirm it.\n",
|
||||
[],
|
||||
staged,
|
||||
);
|
||||
expect([...v.named]).toEqual([["b.md", "Invalid YAML"]]);
|
||||
expect(v.refuseAll).toBeUndefined();
|
||||
});
|
||||
|
||||
it("reads pre-0.48 stderr failures and refuses the batch when a thrown failure is unnamed", () => {
|
||||
const staged = new Set(["a.md", "b.md"]);
|
||||
const named = classifyImport({ imported: 1, skipped: 1, errors: 0 }, " Skipped b.md: too large\n", [], staged);
|
||||
expect([...named.named.keys()]).toEqual(["b.md"]);
|
||||
expect(named.refuseAll).toBeUndefined();
|
||||
const hidden = classifyImport({ imported: 1, skipped: 1, errors: 1 }, "", [], staged);
|
||||
expect(hidden.refuseAll).toMatch(/1 failure\(s\) it did not attribute/);
|
||||
const short = classifyImport({ imported: 0, skipped: 0, errors: 0, total_files: 0 }, "", [], staged);
|
||||
expect(short.refuseAll).toMatch(/accounted for 0 of 2 staged page\(s\)/);
|
||||
});
|
||||
|
||||
it("checks presence per recorded source and content within the get budget", () => {
|
||||
const l = lookupsFrom({ default: { "ceo-plans/demo/2026-10-03-a": "abc" } });
|
||||
const budget = { gets: 1 };
|
||||
expect(checkLanding(entry({ content_sha256: "abc", status: "imported_unverified" }), l, budget)).toBe("ingested");
|
||||
expect(checkLanding(entry({ content_sha256: "abc", status: "imported_unverified" }), l, budget)).toBe("unchecked");
|
||||
expect(checkLanding(entry({ content_sha256: "zzz" }), l, { gets: 1 })).toBe("mismatch");
|
||||
expect(checkLanding(entry({ source_id: "gstack-tx-x" }), l, { gets: 1 })).toBe("absent");
|
||||
expect(checkLanding(entry(), lookupsFrom(null), { gets: 1 })).toBe("unchecked");
|
||||
expect(checkLanding(entry({ content_sha256: "abc" }), lookupsFrom({ default: { "ceo-plans/demo/2026-10-03-a": "abc" } }, { busyGet: true }), { gets: 1 })).toBe("unchecked");
|
||||
});
|
||||
|
||||
it("treats pglite_busy and a list that ignores --source-id as not yet checked", () => {
|
||||
const busy = gbrainLookups(() => ({ status: 1, stdout: '{"error":"pglite_busy"}', stderr: "" }));
|
||||
expect(busy.list("default")).toBeNull();
|
||||
const unscoped = gbrainLookups(() => ({ status: 0, stdout: "a\ttranscript\t2026-10-03\ta\n", stderr: "" }));
|
||||
expect(unscoped.list("default")).toBeNull();
|
||||
const calls: string[][] = [];
|
||||
const scoped = gbrainLookups((a) => {
|
||||
calls.push(a);
|
||||
return a.includes("gstack-capability-probe-0") ? { status: 1, stdout: "", stderr: "Source not found" } : { status: 0, stdout: "x/y\ttranscript\t2026-10-03\tt\n", stderr: "" };
|
||||
});
|
||||
expect([...scoped.list("default")!]).toEqual(["x/y"]);
|
||||
scoped.list("default");
|
||||
expect(calls).toHaveLength(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe("reconcile (unit)", () => {
|
||||
function state(): IngestState {
|
||||
const s = emptyState();
|
||||
for (const name of ["a", "b", "c", "d"]) s.sessions[`/t/${name}.jsonl`] = entry({ page_slug: `transcripts/x/${name}` });
|
||||
return s;
|
||||
}
|
||||
const brain = { default: { "transcripts/x/a": null, "transcripts/x/b": null } };
|
||||
|
||||
it("finds legacy default-source entries present and re-queues only missing pages whose file still exists", () => {
|
||||
const s = state();
|
||||
const sum = reconcile(s, lookupsFrom(brain), { limit: 10, dryRun: false, fileExists: (p) => p !== "/t/d.jsonl" });
|
||||
expect(sum).toMatchObject({ checked: 4, present: 2, requeued: 1, unrecoverable: 1, notChecked: 0, complete: true });
|
||||
expect(s.sessions["/t/c.jsonl"].status).toBe("refused");
|
||||
expect(s.sessions["/t/d.jsonl"].status).toBe("unrecoverable");
|
||||
expect(s.reconcile?.pending).toBe(false);
|
||||
expect(formatReconcileSummary(sum, false)).toBe("reconcile: checked 4, present 2, re-queued 1, not yet checked 0, unrecoverable 1 (source file gone)");
|
||||
});
|
||||
|
||||
it("--dry-run changes nothing; --limit bounds the pass and the next pass resumes; a third pass is a no-op", () => {
|
||||
const s = state();
|
||||
const before = JSON.stringify(s);
|
||||
reconcile(s, lookupsFrom(brain), { limit: 10, dryRun: true, fileExists: () => true });
|
||||
expect(JSON.stringify(s)).toBe(before);
|
||||
const first = reconcile(s, lookupsFrom(brain), { limit: 2, dryRun: false, fileExists: () => true });
|
||||
expect(first).toMatchObject({ checked: 2, complete: false });
|
||||
expect(formatReconcileSummary(first, false)).toMatch(/\(run again to continue\)$/);
|
||||
const second = reconcile(s, lookupsFrom(brain), { limit: 2, dryRun: false, fileExists: () => true });
|
||||
expect(second).toMatchObject({ checked: 2, present: 0, requeued: 2, complete: true });
|
||||
const third = reconcile(s, lookupsFrom(brain), { limit: 2, dryRun: false, fileExists: () => true });
|
||||
expect(third).toMatchObject({ checked: 2, present: 2, requeued: 0 });
|
||||
});
|
||||
|
||||
it("a failed lookup is not yet checked, never absent, and keeps the reconcile pending", () => {
|
||||
const s = state();
|
||||
const sum = reconcile(s, lookupsFrom(null), { limit: 10, dryRun: false, fileExists: () => true });
|
||||
expect(sum).toMatchObject({ checked: 0, notChecked: 4, requeued: 0, complete: false });
|
||||
expect(Object.values(s.sessions).every((e) => e.status === "ingested")).toBe(true);
|
||||
expect(s.reconcile?.pending).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("state writers (unit)", () => {
|
||||
let dir = "";
|
||||
beforeEach(() => { dir = mkdtempSync(join(tmpdir(), "gstack-landing-lock-")); });
|
||||
afterEach(() => rmSync(dir, { recursive: true, force: true }));
|
||||
|
||||
it("one lock: a live holder blocks, a dead holder's lock is taken over", async () => {
|
||||
const lockPath = join(dir, "state.json.lock");
|
||||
const sleeper = spawn("sleep", ["30"]);
|
||||
try {
|
||||
writeFileSync(lockPath, `${sleeper.pid}\n`);
|
||||
expect(acquireStateLock(lockPath)).toEqual({ ok: false, holder: sleeper.pid! });
|
||||
} finally {
|
||||
sleeper.kill("SIGKILL");
|
||||
}
|
||||
await new Promise((r) => sleeper.on("exit", r));
|
||||
const lock = acquireStateLock(lockPath);
|
||||
expect(lock.ok).toBe(true);
|
||||
if (lock.ok) lock.release();
|
||||
expect(existsSync(lockPath)).toBe(false);
|
||||
});
|
||||
|
||||
it("a state save failure throws instead of being swallowed", () => {
|
||||
const target = join(dir, "state.json");
|
||||
mkdirSync(target);
|
||||
expect(() => saveStateFile(target, emptyState())).toThrow(/could not save ingest state/);
|
||||
});
|
||||
});
|
||||
|
||||
// ── CLI against the fake brain ─────────────────────────────────────────────
|
||||
|
||||
describe("gstack-memory-ingest landing check (CLI, fake gbrain)", () => {
|
||||
let home = "";
|
||||
let gstackHome = "";
|
||||
let env: Record<string, string> = {};
|
||||
let brain: ReturnType<typeof installFakeBrain>;
|
||||
|
||||
beforeEach(() => {
|
||||
home = mkdtempSync(join(tmpdir(), "gstack-landing-"));
|
||||
gstackHome = join(home, ".gstack");
|
||||
mkdirSync(join(gstackHome, "projects", "demo", "ceo-plans"), { recursive: true });
|
||||
brain = installFakeBrain(home);
|
||||
env = { HOME: home, GSTACK_HOME: gstackHome, PATH: `${brain.binDir}:/usr/bin:/bin`, TMPDIR: home };
|
||||
});
|
||||
afterEach(() => rmSync(home, { recursive: true, force: true }));
|
||||
|
||||
function plans(...names: string[]): string[] {
|
||||
return names.map((n) => {
|
||||
const p = join(gstackHome, "projects", "demo", "ceo-plans", `${n}.md`);
|
||||
writeFileSync(p, `# Plan ${n}\n\nbody of ${n}\n`);
|
||||
return p;
|
||||
});
|
||||
}
|
||||
function run(args: string[] = [], extra: Record<string, string> = {}) {
|
||||
const r = spawnSync(process.execPath, [SCRIPT, "--sources", "ceo-plan", ...args], {
|
||||
env: { ...env, ...extra }, cwd: home, encoding: "utf-8", timeout: 60_000,
|
||||
});
|
||||
return { status: r.status, stdout: r.stdout || "", stderr: r.stderr || "" };
|
||||
}
|
||||
const statePath = () => join(gstackHome, ".transcript-ingest-state.json");
|
||||
const sessions = (): Record<string, StateEntry> => existsSync(statePath()) ? JSON.parse(readFileSync(statePath(), "utf8")).sessions : {};
|
||||
const calls = (): string[][] => existsSync(brain.callLog) ? readFileSync(brain.callLog, "utf8").trim().split("\n").filter(Boolean).map((l) => JSON.parse(l)) : [];
|
||||
|
||||
it("stamps verified pages ingested with their recorded source and content hash", () => {
|
||||
const [a] = plans("alpha");
|
||||
const r = run(["--bulk", "--quiet"]);
|
||||
expect(r.status).toBe(0);
|
||||
expect(sessions()[a]).toMatchObject({ status: "ingested", source_id: "default" });
|
||||
expect(sessions()[a].content_sha256).toMatch(/^[0-9a-f]{64}$/);
|
||||
expect(calls().some((c) => c[0] === "get" && c.includes("--json"))).toBe(true);
|
||||
});
|
||||
|
||||
it("a page gbrain names as failed (exit 1, HEAD failures[]) is left un-stamped and printed under --quiet", () => {
|
||||
const [a, b] = plans("alpha", "broken");
|
||||
const r = run(["--bulk", "--quiet"], { FAKE_FAIL: "broken" });
|
||||
expect(r.stderr).toMatch(/\[memory-ingest\] FAILED ceo-plans\/demo\/\S+broken\.md: Invalid YAML frontmatter/);
|
||||
expect(sessions()[a]?.status).toBe("ingested");
|
||||
expect(sessions()[b]).toBeUndefined();
|
||||
expect(r.status).toBe(0);
|
||||
});
|
||||
|
||||
it("pre-0.48 gbrain: a page skipped with exit 0 and only a stderr 'Skipped' line is never stamped (#2778 field case)", () => {
|
||||
const [a, b] = plans("alpha", "broken");
|
||||
const r = run(["--bulk", "--quiet"], { FAKE_FAIL: "broken", FAKE_OLD_JSON: "1" });
|
||||
expect(r.status).toBe(0);
|
||||
expect(r.stderr).toMatch(/FAILED ceo-plans\/demo\/\S+broken\.md/);
|
||||
expect(sessions()[b]).toBeUndefined();
|
||||
expect(sessions()[a].source_id).toBe("default");
|
||||
});
|
||||
|
||||
it("a Pending receipt (counted unchanged, never stored) is re-queued after the landing check, not stamped ingested", () => {
|
||||
const [a, p] = plans("alpha", "pending");
|
||||
const r = run(["--bulk", "--quiet"], { FAKE_PENDING: "pending" });
|
||||
expect(r.stderr).toMatch(/re-queued ceo-plans\/demo\/\S+pending: not found in gbrain source default after import/);
|
||||
expect(sessions()[a].status).toBe("ingested");
|
||||
expect(sessions()[p].status).toBe("refused");
|
||||
const again = run(["--incremental", "--quiet"]);
|
||||
expect(again.status).toBe(0);
|
||||
expect(sessions()[p].status).toBe("ingested");
|
||||
});
|
||||
|
||||
it("without a usable lookup (list ignores --source-id, or PGLite busy) pages stay imported_unverified and are not re-imported", () => {
|
||||
const [a] = plans("alpha");
|
||||
expect(run(["--bulk", "--quiet"], { FAKE_LIST_UNSCOPED: "1" }).status).toBe(0);
|
||||
expect(sessions()[a].status).toBe("imported_unverified");
|
||||
const imports = () => calls().filter((c) => c[0] === "import").length;
|
||||
const before = imports();
|
||||
expect(run(["--incremental", "--quiet"], { FAKE_BUSY: "1" }).status).toBe(0);
|
||||
expect(imports()).toBe(before);
|
||||
expect(sessions()[a].status).toBe("imported_unverified");
|
||||
expect(run(["--incremental", "--quiet"]).status).toBe(0);
|
||||
expect(sessions()[a].status).toBe("ingested");
|
||||
});
|
||||
|
||||
it("an unnamed failure refuses the batch; after 3 refusals the batch is bisected and the poison page quarantined", () => {
|
||||
const pages = plans("p1", "p2", "poison", "p4");
|
||||
for (let i = 1; i <= 2; i++) {
|
||||
const r = run(["--bulk", "--quiet"], { FAKE_POISON: "poison" });
|
||||
expect(r.status).toBe(1);
|
||||
expect(r.stderr).toMatch(/did not attribute to a staged page/);
|
||||
expect(Object.keys(sessions())).toHaveLength(0);
|
||||
}
|
||||
const r = run(["--bulk", "--quiet"], { FAKE_POISON: "poison" });
|
||||
expect(r.status).toBe(0);
|
||||
expect(r.stderr).toMatch(/quarantined ceo-plans\/demo\/\S+poison .*gbrain refuses every batch that contains it/);
|
||||
const s = sessions();
|
||||
expect(s[pages[2]].status).toBe("quarantined");
|
||||
for (const p of [pages[0], pages[1], pages[3]]) expect(s[p].status).toBe("ingested");
|
||||
const next = run(["--bulk", "--quiet"], { FAKE_POISON: "poison" });
|
||||
expect(next.status).toBe(0);
|
||||
expect(sessions()[pages[2]].status).toBe("quarantined");
|
||||
});
|
||||
|
||||
it("remote-http mode records pages as staged, never ingested", () => {
|
||||
const [a] = plans("alpha");
|
||||
writeFileSync(join(home, ".claude.json"), JSON.stringify({ mcpServers: { gbrain: { type: "http", url: "http://fixture.invalid/mcp" } } }));
|
||||
expect(run(["--bulk", "--quiet"]).status).toBe(0);
|
||||
expect(sessions()[a]).toMatchObject({ status: "staged" });
|
||||
expect(calls().filter((c) => c[0] === "import")).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("--reconcile re-queues a stamped page missing from its recorded source, backs up state, and supports --dry-run and --limit", () => {
|
||||
const [a, b] = plans("alpha", "beta");
|
||||
writeFileSync(statePath(), JSON.stringify({
|
||||
schema_version: 1, last_writer: "gstack-memory-ingest",
|
||||
sessions: Object.fromEntries([a, b].map((p) => [p, { mtime_ns: 1, sha256: "old", ingested_at: "2026-09-01T00:00:00Z", page_slug: `ceo-plans/demo/${p.endsWith("alpha.md") ? "x-alpha" : "x-beta"}` }])),
|
||||
}));
|
||||
writeFileSync(brain.brainFile, JSON.stringify({ sources: { default: { federated: true } }, pages: { default: { "ceo-plans/demo/x-alpha": { sha: null } } } }));
|
||||
const dry = run(["--reconcile", "--dry-run"]);
|
||||
expect(dry.stderr).toContain("reconcile (dry run): checked 2, present 1, re-queued 1, not yet checked 0");
|
||||
expect(JSON.parse(readFileSync(statePath(), "utf8")).schema_version).toBe(1);
|
||||
const limited = run(["--reconcile", "--limit", "1"]);
|
||||
expect(limited.stderr).toContain("reconcile: checked 1, present 1, re-queued 0, not yet checked 0 (run again to continue)");
|
||||
expect(existsSync(`${statePath()}.pre-reconcile.bak`)).toBe(true);
|
||||
const rest = run(["--reconcile"]);
|
||||
expect(rest.stderr).toContain("reconcile: checked 1, present 0, re-queued 1, not yet checked 0");
|
||||
expect(sessions()[b].status).toBe("refused");
|
||||
expect(sessions()[a]).toMatchObject({ status: "ingested", source_id: "default" });
|
||||
});
|
||||
|
||||
it("--request-reconcile records a pending pass without calling gbrain; the next ingest runs one bounded pass and prints the summary under --quiet", () => {
|
||||
const [a] = plans("alpha");
|
||||
expect(run(["--bulk", "--quiet"]).status).toBe(0);
|
||||
rmSync(brain.callLog, { force: true });
|
||||
const req = run(["--request-reconcile", "--quiet"]);
|
||||
expect(req.status).toBe(0);
|
||||
expect(calls()).toEqual([]);
|
||||
expect(JSON.parse(readFileSync(statePath(), "utf8")).reconcile.pending).toBe(true);
|
||||
const next = run(["--incremental", "--quiet"]);
|
||||
expect(next.stderr).toContain("[memory-ingest] reconcile: checked 1, present 1, re-queued 0, not yet checked 0");
|
||||
expect(JSON.parse(readFileSync(statePath(), "utf8")).reconcile.pending).toBe(false);
|
||||
expect(sessions()[a].status).toBe("ingested");
|
||||
});
|
||||
|
||||
it("a state file that cannot be saved fails the run", () => {
|
||||
plans("alpha");
|
||||
mkdirSync(statePath());
|
||||
const r = run(["--bulk", "--quiet"]);
|
||||
expect(r.status).toBe(1);
|
||||
expect(r.stderr).toMatch(/could not save ingest state/);
|
||||
});
|
||||
|
||||
it("a live writer's lock stops a second run before it touches state", () => {
|
||||
plans("alpha");
|
||||
writeFileSync(`${statePath()}.lock`, `${process.pid}\n`);
|
||||
const r = run(["--bulk", "--quiet"]);
|
||||
expect(r.status).toBe(1);
|
||||
expect(r.stderr).toContain(`another memory ingest (pid ${process.pid})`);
|
||||
expect(existsSync(statePath())).toBe(false);
|
||||
expect(calls().filter((c) => c[0] === "import")).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("--help lists --reconcile as a mode", () => {
|
||||
const r = spawnSync(process.execPath, [SCRIPT, "--help"], { env, encoding: "utf-8", timeout: 20_000 });
|
||||
expect(r.stderr).toMatch(/\[--probe\|--incremental\|--bulk\|--reconcile\]/);
|
||||
expect(r.stderr).toMatch(/--dry-run/);
|
||||
expect(readdirSync(gstackHome).filter((f) => f.startsWith(".staging-ingest-"))).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -164,7 +164,7 @@ describe("V1 memory ingest pipeline E2E", () => {
|
||||
rmSync(home, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("--incremental writes a state file with schema_version: 1 + last_writer", () => {
|
||||
it("--incremental writes a state file with schema_version: 2 + last_writer", () => {
|
||||
const home = makeFixtureHome();
|
||||
const { gstackHome } = setupFixture(home);
|
||||
const env = fakeGbrainEnv(home, gstackHome);
|
||||
@@ -174,7 +174,7 @@ describe("V1 memory ingest pipeline E2E", () => {
|
||||
const statePath = join(gstackHome, ".transcript-ingest-state.json");
|
||||
expect(existsSync(statePath)).toBe(true);
|
||||
const state = JSON.parse(readFileSync(statePath, "utf-8"));
|
||||
expect(state.schema_version).toBe(1);
|
||||
expect(state.schema_version).toBe(2);
|
||||
expect(state.last_writer).toBe("gstack-memory-ingest");
|
||||
expect(typeof state.last_full_walk).toBe("string");
|
||||
|
||||
|
||||
@@ -373,10 +373,15 @@ describe("#1802 D1 — remote-http finally gate (static invariant)", () => {
|
||||
"utf-8",
|
||||
);
|
||||
|
||||
test("finally gates cleanupStagingDir on !remoteHttpMode", () => {
|
||||
// Tolerates additional guards (e.g. C3's !preserveStaging) in the same
|
||||
// condition — the load-bearing invariant is that remote-http never deletes.
|
||||
expect(ingest).toMatch(/if \(!remoteHttpMode[^)]*\) cleanupStagingDir\(stagingDir\)/);
|
||||
test("the remote-http branch returns before any staging cleanup can run", () => {
|
||||
// A1 moved remote-http staging into stageForRemoteBrain, which ingestPass
|
||||
// returns before importBatch (the only owner of the cleanup finally) runs.
|
||||
// The load-bearing invariant is unchanged: remote-http never deletes.
|
||||
const at = ingest.indexOf("function stageForRemoteBrain(");
|
||||
expect(at).toBeGreaterThan(-1);
|
||||
const body = ingest.slice(at, ingest.indexOf("\n}\n", at));
|
||||
expect(body).not.toMatch(/cleanupStagingDir/);
|
||||
expect(ingest).toMatch(/if \(remoteHttpMode\) return stageForRemoteBrain\(/);
|
||||
});
|
||||
|
||||
test("the only finally-scoped cleanup call is the gated one", () => {
|
||||
@@ -411,9 +416,7 @@ describe("#1802 C3 — import-timeout preserve (static invariant)", () => {
|
||||
});
|
||||
|
||||
test("finally honors preserveStaging", () => {
|
||||
expect(ingest).toMatch(
|
||||
/if \(!remoteHttpMode && !preserveStaging\) cleanupStagingDir\(stagingDir\)/,
|
||||
);
|
||||
expect(ingest).toMatch(/if \(!preserveStaging\) cleanupStagingDir\(stagingDir\)/);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
Reference in new issue
Block a user