CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-14 17:05:28 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix(brain-sync): per-record spool dir — the enqueue/drain race dies structurally

Browse Source 
Producers appended lines to .brain-queue.jsonl while the drain re-read and
os.replace'd it; the in-code comment admitted a lockless append between the
re-read and the replace was lost. Locks and rename-rotation designs were both
reviewed and rejected (each retained a tail race); the shipped design is a
maildir-style spool: one FILE per record in .brain-queue.d/ (tmp + atomic
rename), the drain snapshots filenames, processes, and deletes exactly what
it snapshotted. Writer and drainer never share an inode — nothing to race.

Semantics: at-least-once (a crash between process and unlink re-drains;
downstream content-hash dedup absorbs duplicates); retained (privacy-held)
records keep their files; unparseable records are kept + warned, never
destroyed. Legacy .brain-queue.jsonl migrates atomically on the next drain
(crash-leftover .migrating files recovered too); status/drop-queue count both
surfaces; discover-new writes spool records and advances its cursor
per-record-written. The preamble's queue-depth line switches to spool count
in this wave's template block.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-17 10:47:26 -07:00
co-authored by Claude Fable 5
parent e57b2798fd
commit 6df30370b3
4 changed files with 470 additions and 152 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bin/gstack-brain-uninstall
+4 -1
View File
@@ -19,7 +19,8 @@
# .gitattributes — merge driver declarations
# .brain-allowlist — sync path list
# .brain-privacy-map.json — sync privacy classifier
# .brain-queue.jsonl — pending queue
# .brain-queue.d/ — pending spool (one file per record)
# .brain-queue.jsonl — legacy pending queue (pre-spool)
# .brain-discover-cursor — discover-new cursor
# .brain-last-push — timestamp marker
# .brain-skip.txt — user-maintained skip list
@@ -118,7 +119,9 @@ rm -f "$GSTACK_HOME/.gitignore" 2>/dev/null || true
rm -f "$GSTACK_HOME/.gitattributes" 2>/dev/null || true
rm -f "$GSTACK_HOME/.brain-allowlist" 2>/dev/null || true
rm -f "$GSTACK_HOME/.brain-privacy-map.json" 2>/dev/null || true
rm -rf "$GSTACK_HOME/.brain-queue.d" 2>/dev/null || true
rm -f "$GSTACK_HOME/.brain-queue.jsonl" 2>/dev/null || true
rm -f "$GSTACK_HOME/.brain-queue.jsonl.migrating" 2>/dev/null || true
rm -f "$GSTACK_HOME/.brain-discover-cursor" 2>/dev/null || true
rm -f "$GSTACK_HOME/.brain-last-push" 2>/dev/null || true
rm -f "$GSTACK_HOME/.brain-last-pull" 2>/dev/null || true