mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 14:06:50 +01:00
garrytanandafshaker
97d4e58dcc
gbrain import exits 0 while skipping pages (returned per-file failures, malformed filenames, managed-import Pending receipts), and its failure ledger exists only for git import dirs, so a skipped page was stamped ingested and never retried (field reports: 37 and 69 lost entries). Write, verify, then stamp (lib/memory-ingest-landing.ts, a new ratcheted module): - State schema 2 with a transition table (staged -> imported_unverified -> ingested, plus refused, quarantined, unrecoverable). Schema 1 files migrate in place: entries become ingested in the legacy "default" source and a reconcile pass is marked pending. - Named failures come from --json `failures`, stderr "Skipped <rel>:" lines and the ledger; a failure gbrain does not attribute refuses the whole batch. After 3 refused batches the batch is bisected and the page that poisons it is quarantined (printed even under --quiet). - Landing check: one `gbrain list --source-id` per source per run and at most 50 `gbrain get --json` content checks against a gstack_content_sha256 frontmatter field gstack now writes. Lookup errors and pglite_busy mean "not yet checked", never absent; pages over the cap stay imported_unverified and are not re-imported. - Each entry records its gbrain source_id. Remote-http mode records "staged", never "ingested". - The source fingerprint comes from the exact bytes parsed, so a transcript that grows during import is not stamped as its newer self (the old unscanned contract test is migrated; it pinned that bug). - --reconcile mode (bounded by --limit, default 200; resumable; --dry-run; backs up the state file) prints "reconcile: checked N, present M, re-queued K, not yet checked L". --request-reconcile records a pending pass without calling gbrain; the v1.91.17.0 migration (placeholder name) uses it, and the next ingest run does one bounded pass. - One lock for every state writer; a state-save failure fails the run. The #1802 static pins in regression-1611 move to the new structure (remote-http returns before importBatch; the finally honors preserveStaging). Co-authored-by: afshaker <andrew@phone2.io>