anonymous_github/docs/shared-cache.md

Shared blob cache

The optional shared filesystem cache stores immutable source content across projects. It uses Node's built-in SQLite module and requires Node 22.15 or newer. All API and streamer processes on one host must mount the same local cache directory. SQLite WAL is not a distributed lock service: do not put this directory on NFS or share it across hosts. S3 storage continues to use the existing storage implementation.

Enable the integrated implementation with SHARED_CACHE_ENABLED=true. The default is false so a code deployment does not silently migrate production data. The other settings are independent rollback controls:

• SHARED_CACHE_PATH: defaults to .shared-cache-v1 inside FOLDER.
• SHARED_CACHE_BYTES: source pool budget, default 10 GiB.
• SHARED_CACHE_RESULT_BYTES: transformed output budget, default 512 MiB.
• SHARED_CACHE_MAX_FILE_BYTES: maximum source admission size, default 32 MiB.
• SHARED_CACHE_FREE_BYTES: free-device-space reserve, default 1 GiB.
• SHARED_CACHE_LOW_WATER: eviction target fraction, default 0.9.
• SHARED_CACHE_GZIP: store gzip when beneficial, default true.
• SHARED_CACHE_GZIP_MIN_SAVING: minimum fractional saving, default 0.10.
• SHARED_CACHE_DECISIONS: enable unchanged decisions and changed-output reuse, default true.
• CACHE_MONITORING_ENABLED: collect cache observations, default true. Monitoring is independent of shared storage and also records legacy cache/inspection events.

Logical user quotas remain based on repository file sizes. Shared-pool deduplication, compression and eviction do not change user quota usage.

These are initial limits, not capacity recommendations from the production sample. Restart processes together when changing pool limits. Reserve headroom for SQLite, filesystem directories, existing caches and other application temporary files. The budget accounts for blob allocation with a conservative 4 KiB minimum plus in-flight reservations of three times the maximum object size and a small overhead. Keep each pool budget larger than this reservation. Admission can bypass the cache while other fills/readers occupy the budget. Oversized source objects are streamed without admission, still subject to the application's file-size limit.

Git blobs are verified against the requested Git SHA before publication. Resolved LFS content is admitted only after verifying both the Git pointer and its SHA-256 payload digest and size. Gzip is a representation of the same uncompressed content. Files are written privately, synced and atomically renamed before index publication. Fills and readers have renewable leases. Eviction skips readers, and stale fills are recovered before their reservations are released. LRU access timestamps are coarsened to 30 seconds. Orphan reconciliation runs at most hourly across processes. Alias and reuse-history indexes are bounded; decisions expire after seven days and are capped at 100,000 entries.

Existing generation caches are read as a migration fallback and verified before promotion. They are not deleted during promotion, so rollback remains possible. The new pool budget does not include legacy copies: monitor total filesystem usage and retire legacy data only after validating rollout. Uploaded ZIP originals and other durable source material are untouched. Repository removal invalidates access and project mappings, while shared blobs remain subject to global eviction.

Transient SQLite write contention bypasses caching instead of failing file requests. Publication installs a reader lease before releasing its transaction, including duplicate blobs. Decisions retain sniffed text classification for consistent MIME headers on repeat reads; binary responses omit the range-denial header.

The HTTP fast path applies to individual file responses backed by the shared source pool, including small text files. Decisions include content identity, path, project context, all transformation options, generation/revision, mask, hostname and engine version. Unchanged decisions serve the original representation. Changed output has its own bounded pool. Gzip negotiation honors quality values, uses Vary, and preserves MIME type. File validators are weak when shared delivery is enabled so identity and gzip representations can revalidate the same logical content. HEAD continues to use metadata only. Range requests receive a complete response; this implementation does not add random access into gzip files. Website rendering keeps its renderer cache; ZIP scans do not admit bulk entries into the source blob pool.

The admin overview's Cache section reads /api/admin/cache?minutes=60 behind the existing admin authorization. Windows are 15 minutes, one hour, 24 hours and seven days. SQLite telemetry batches at most 512 groups every five seconds and retains minute aggregates for 48 hours and hourly aggregates for 30 days. No paths, terms, user IDs or blob hashes appear in metric labels. The UI distinguishes unavailable telemetry, dropped events, disabled storage and empty windows. Rows separate file, website, ZIP, background and known-monitor traffic. Monitor classification is an operational heuristic, not a complete bot detector.

Hit counts and byte hit rates measure source lookups; misses classify first observed keys, known misses and eviction refetches within bounded seven-day history. Reuse intervals are consecutive blob accesses, not distinct people or browser visits. Eviction time measures idle age; codec and inspection timings are elapsed work time, not CPU samples. Cached decisions are separate from actual inspections. Minute and hour boundaries make the displayed windows approximate.

Run the reproducible acceptance workload with:

mkdir -p benchmark-results/cache
node scripts/cache-workload.js benchmark-results/cache/workload.json
npx mocha test/shared-blob-cache.test.js test/shared-cache-http.test.js

The JSON report asserts cross-project source reuse, unchanged-decision reuse, changed-output reuse, gzip delivery and budget enforcement. Its compression ratio comes from synthetic text and must not be presented as a production projection. Production rollout should compare request/byte hit rates, eviction refetches, p95 latency, upstream errors and free disk over a representative week, with large files, ZIP work and known monitors visible separately.

Benefits and costs

Choice	Benefit	Cost or limit
Global content identity	Reuses identical files across paths and projects; renames do not duplicate content	Requires verified hashes and authorization on every request; uses digest-prefix directories to avoid one huge directory
Byte budget with approximate LRU	Predictable blob footprint and automatic reclamation	Evicted files incur upstream fetches; active readers can temporarily prevent admission
Selective gzip	Reduces payload storage and can serve compressed bytes directly	First-fill compression costs work; identity consumers and uncached anonymization require decoding
Rule-aware decisions and results	Repeated unchanged reads skip regex inspection; changed output is reusable	More metadata and a separate output budget; settings changes produce new decisions
SQLite coordination and telemetry	Cross-process coordination without another service; admin can measure actual reuse and rewrite work	Local-filesystem scope; short synchronous transactions; metadata/headroom are outside blob budgets

Rollout sequence

1. Deploy with shared storage disabled and collect legacy hits, inspections and codec timings. Keep known-monitor traffic separate from file readers.
2. Enable the shared cache on the single-host filesystem deployment with both byte budgets configured. Keep legacy copies for rollback.
3. Compare a representative week of byte/request hits, rewrite decisions, eviction refetches, request latency, upstream errors and whole-device free space. Adjust admission size and budgets from those observations.
4. Retire legacy cache copies only after the rollback window. Uploaded originals remain durable and are never candidates for shared-pool eviction.

The implementation and acceptance workload are local changes; this work does not enable or migrate production storage.

Production observations used for this design

The read-only audit on 2026-11-02 found 425,312 source files across 31,522 projects, containing 144.32 GB of payload. Deduplication within projects could remove 1.87 GB; global deduplication could remove 2.93 GB total (about 2.03%), including another 1.06 GB across projects. One file could not be read. These are decimal GB and payload savings, not a compression estimate or the expected new working-set size.

The preceding 24-hour access sample contained 29,980 successful file-body responses, including 9,249 repeat responses (30.9%). URL repetitions are not the same as blob cache hits. A separate approximately two-hour metric window recorded 2,302 source hits out of 3,600 lookups (63.9%). This supports retaining a bounded source cache; it does not establish the optimal byte budget.

In the uniform anonymization replay, 937 resident files were measured: 764 text, 173 binary, and 21 with changed output. Thirteen resolved files exceeded the 4 MiB sample limit, and missing files/mappings were excluded. The sample's mean gzip-6 work was 1.46 ms per file and mean decompression was 0.24 ms; these replay timings are not live-request latency or a forecast for large files. Continuous telemetry is needed to replace these samples with representative workload measurements.

Metadata durability and accounting

Reader leases, access history, decisions and telemetry use NORMAL WAL commits. Reservations, publication intent and final blob publication use FULL commits; payloads are synced before rename. Recent disposable metadata may be lost after system failure. Recovery retains reservations until orphan files are removed or accounted for. SIGKILL recovery is tested; physical power failure is not simulated.

Admin statistics use transactional counters and an indexed oldest-access lookup. History and decisions are capped at 100,000 rows each, aliases at 200,000. Expired history/decision cleanup removes at most 256 rows per pass. Access history has 30-second resolution. Stop all cache processes before upgrading an existing index and restart them on the same version; do not mix old and new writers.

Performance and validation

Local synthetic measurements on Node 24.14.1, dated 2026-11-02, compared the initial FULL-commit implementation with optimized bookkeeping. Warm serial changed-text medians fell from 44.9 to 7.3 ms at 16 KiB, 45.9 to 9.0 ms at 256 KiB, and 60.5 to 19.0 ms at 2 MiB. At 100,000 synthetic blob rows, admin stats p95 fell from 78.7 to 0.50 ms and decision-write p95 from 45.5 to 1.11 ms.

The optimized runs verified 2,500 HTTP responses. The 4 MiB eviction probe peaked at 3,997,696 allocated bytes. Its oversized sequential working set had no hits: a bounded LRU cache does not guarantee useful reuse. Measurements used loopback HTTP, warm OS caches and a shared host; they exclude production traffic, TLS, authorization overhead and multi-process throughput. They are not capacity estimates.

On the PR checkout based on main at d0b217a, the production build, scoped lint and 199 cache, HTTP, admin access, HEAD/revalidation and Vue UI tests passed. Coverage includes process crash recovery, concurrent fills, active readers, LFS verification, and public submodule origins without parent credentials. Admin layout was also checked at 1280 px and 390 px during implementation.

Generate fresh reports locally; generated output is ignored by Git:

npx tsc
node scripts/cache-index-performance.js
CACHE_BENCH_COMPILED=1 node scripts/cache-performance.js
CACHE_BENCH_COMPILED=1 node scripts/cache-performance.js benchmark-results/cache/repeat.json --shared

Default reports go to benchmark-results/cache/. Scripts also accept an explicit output path. Keep raw measurements with the evaluation that produced them rather than committing generated reports to the documentation directory.