CalvinBackup/anonymous_github
1
0 Fork 0
mirror of https://github.com/tdurieux/anonymous_github.git synced 2026-11-14 00:07:00 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge origin/main into feat/ui-redesign

Browse Source 
Backend, scripts, docs and their tests take main's newer code where both
sides changed the same lines (the shared blob cache, GitHubStream and
markdown fixes landed on main after the Go migration branch was cut).
The Dockerfile keeps the Go build from the migration, plus main's
owner-alert-protocol label; .gitignore keeps both sides' entries.

Frontend keeps the redesign and folds in main's features: Stripe billing
in the New anonymization submit bar, GitHub access repair, redaction
syntax help, the preview expand toggle, configurable redaction mask,
folder counts, reviewer-route header rules, link expiry (now in the
viewer's snapshot details) and the explorer error card. main's
narrow-only wrap logic is dropped because the redesign wraps on desktop
and scrolls sideways on phones.

Tests: 1534 passing with SESSION_SECRET set (as in CI). Three main tests
were updated for the agreed viewer design (no site header in the viewer,
no repository-wide file count).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Thomas DurieuxandClaude Opus 5.5 committed 2026-11-08 14:07:33 +00:00
2 parents 53cac6ac39 19ce699003
commit c685d833ab
211 files changed
+24523 -3270
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/github-app-setup.md
+12 -1
View File
@@ -19,7 +19,10 @@ Configure:
- Repository permissions: **Contents: read-only**, **Metadata: read-only**,
**Pull requests: read-only**, **Pages: read-only**. Do not grant write permissions.
Pull requests read also permits [reading PR issue comments](https://docs.github.com/en/rest/issues/comments#list-issue-comments); Issues permission is unnecessary.
- Keep user access token expiration enabled. Private email permission is unnecessary.
- User permissions: **Email addresses: read-only** to save a verified address at
sign-in, including addresses hidden from the public GitHub profile. If this
permission is unavailable, owners can enter an address in the email form.
- Keep user access token expiration enabled.
- Leave **Request user authorization (OAuth) during installation** unchecked.
Anonymous GitHub authorizes the user before opening installation; the setup
redirect must remain available afterward.
@@ -105,6 +108,14 @@ Public reads use a [repository-scoped App user token](https://docs.github.com/en
Each API request renews through its own repository binding, rechecking identity
and public visibility while keeping the same user quota.
Existing installation bindings retain their installation checks.
If a previously public source becomes private, its existing public-only binding
continues to reject private reads. Select the repository in the App installation,
then open the anonymization's Edit page and click **Repair GitHub access**. This
verifies installation access to the same GitHub repository ID and published commit
before replacing the binding. It preserves the anonymous URL, snapshot and settings.
Signing in again alone does not change a resource's binding. The Connections page
also provides **Reconnect read-only access** for this purpose.
This uses GitHub's documented [public resource access for App user tokens](https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/choosing-permissions-for-a-github-app).
The explicit **Use existing OAuth access** choice handles repositories not yet
docs/markdown-compatibility.md
+5 -1
View File
@@ -17,6 +17,10 @@ GitHub service features such as user mentions, issue-status badges, commit refer
The sanitizer runs on all Markdown-produced HTML. Math and diagram renderers insert their own generated markup through unpredictable, per-document markers after sanitization. TeX runs with `trust: false`; Mermaid runs with `securityLevel: strict`. Source HTML cannot opt into this bypass. Task inputs are always disabled checkboxes.
Run `npm run build:ui` before `npx mocha test/markdown-compatibility.test.js test/markdown-sanitization.test.js test/mermaid-security.test.js test/asset-build.test.js`. The browser tests exercise the built bundle, not a copy of the renderer.
Run `npm run build:ui` before `npx mocha test/markdown-compatibility.test.js test/markdown-sanitization.test.js test/markdown-security.test.js test/mermaid-security.test.js test/asset-build.test.js`. The browser tests exercise the built bundle, not a copy of the renderer.
References: [GitHub basic syntax](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax), [math](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/writing-mathematical-expressions), [diagrams](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-diagrams), and the [GFM specification](https://github.github.com/gfm/).
Diagram inputs are limited to one million characters and 60,000 coordinates. Maps also enforce nesting and TopoJSON expansion limits before conversion. Rejected diagrams display as code.
Mermaid, DOMPurify and KaTeX are npm dependencies so dependency audits cover their versions. After upgrading them, run `npm run vendor:markdown` followed by `npm run build:ui` and commit the refreshed assets. Security tests require the vendored files to match the installed packages.
docs/paid-hosting.md
+271
View File
@@ -0,0 +1,271 @@
# Paid hosting
Billing is opt-in. `BILLING_ENABLED=false` preserves self-hosted behavior. This implementation uses Stripe Checkout and the customer portal. No card details are stored in the application.
## Offers
Configure four EUR prices in Stripe. The amounts below are suggested launch prices, not hard-coded charges:
- Author monthly: €5/month, recurring monthly.
- Author annual: €49/year, recurring yearly.
- Author review pass: €15 once, one existing repository for 180 days from checkout creation.
- Conference review package: €199 once, up to 100 repositories during a fixed review window of at most 180 days.
Checkout uses the configured Stripe Price ID. The settings page displays its actual amount and cadence. Do not configure trials, metered prices, tiered prices, adjustable quantities, or subscription changes to unrelated products in the customer portal. New subscriptions need a paid invoice before access is granted. A failed renewal does not extend paid time. Cancellation at period end keeps access until that period ends; immediate cancellation ends new paid usage immediately.
Only repository hosting is covered. Gists and pull requests retain their existing behavior. Lab plans are not included.
Default capacities, all measured in source-file bytes:
- Free account: 5 repositories, 2 GiB total.
- Author subscription: 20 repositories, 10 GiB total, and never-expiring repositories while subscribed.
- Repository pass: 1 repository, 2 GiB.
- Conference: 100 repositories, 50 GiB total.
These are admission limits, not a measure or guarantee of physical disk consumption. Original files, anonymized caches, metadata and backups need additional capacity. Review production usage before enabling billing. File-size safeguards and request throttles still apply to paid users. Reviewers do not need accounts or payment.
## Configuration
Set these environment variables in your deployment's secret/configuration manager:
```text
BILLING_ENABLED=true
BILLING_ORIGIN=https://anonymous.example.org
STRIPE_SECRET_KEY=<Stripe secret key>
STRIPE_WEBHOOK_SECRET=<Stripe webhook signing secret>
STRIPE_AUTHOR_MONTHLY_PRICE=<monthly recurring EUR price ID>
STRIPE_AUTHOR_ANNUAL_PRICE=<annual recurring EUR price ID>
STRIPE_AUTHOR_PASS_PRICE=<one-time EUR price ID>
STRIPE_CONFERENCE_PRICE=<one-time EUR price ID>
BILLING_AUTOMATIC_TAX=false
```
Price IDs can be omitted to hide offers. Use test-mode keys and prices for staging. `BILLING_ORIGIN` must be the application's canonical HTTPS origin; HTTP is accepted only for localhost development. Redirects never use request-supplied hosts or return URLs. Enable and configure Stripe Tax before setting `BILLING_AUTOMATIC_TAX=true`; otherwise configure the appropriate tax treatment in Stripe before launch. Receipts, seller details and portal settings also need configuration in Stripe.
Capacity overrides are positive integers:
```text
FREE_AUTHOR_REPOSITORIES=5
FREE_AUTHOR_STORAGE=2147483648
PAID_AUTHOR_REPOSITORIES=20
PAID_AUTHOR_STORAGE=10737418240
PAID_PASS_STORAGE=2147483648
PAID_CONFERENCE_REPOSITORIES=100
PAID_CONFERENCE_STORAGE=53687091200
```
Changing these values changes the allowance of existing plans too. Keep existing Price IDs configured for as long as subscriptions using them remain active. Plan and conference review-window changes are intentionally locked in this first version.
Configure the Stripe customer portal for payment-method updates, invoice downloads and cancellation. Disable arbitrary subscription product switches. Configure a snapshot webhook at `/api/billing/webhook` using API version `2026-10-28.endive`, matching the installed Stripe SDK, for:
- `checkout.session.completed`
- `checkout.session.async_payment_succeeded`
- `customer.subscription.created`
- `customer.subscription.updated`
- `customer.subscription.deleted`
- `customer.subscription.paused`
- `customer.subscription.resumed`
- `invoice.paid`
- `invoice.payment_failed`
- `charge.refunded`
- `charge.dispute.created`
The webhook is mounted before JSON parsing so Stripe verifies the original request bytes. Failed durable receipt returns 500 for Stripe retries. Processing runs asynchronously from the stored event inbox. Duplicate deliveries do not extend passes or allocate duplicate grants. Subscription events retrieve current Stripe state and use a database revision check to prevent stale concurrent writes. Full refunds and disputes suspend the affected paid grant; partial refunds preserve access; reinstatement after a resolved dispute requires operator review.
## Lifecycle and migration
New paid conferences remain `pending_payment` until a verified checkout activates them. Collect billing addresses in Stripe. Historic conference plans remain editable and keep their existing status, but are no longer offered to new conferences. There is no automatic charge for historical estimated usage.
Existing repositories are not deleted or hidden on rollout or subscription expiry. New repositories are marked `billingManaged`; their file trees must fit an allowance before publication. A repository that fails this check remains available in its owner's dashboard for upgrading and retrying. Buy a pass by entering that repository's anonymized ID in Settings. Automatic refresh pauses for billing-managed repositories on the free allowance, including after paid access expires. Owners can still request a refresh within that allowance. Existing review links retain their configured expiration dates. This is not a guarantee of perpetual hosting.
Allowances are selected in order: paid conference, repository pass, author subscription, free account. MongoDB quota ledgers atomically reserve count and source-file bytes. They initially count existing repositories using stored size metadata. Reservations never shrink during refresh, so a concurrent or failed build cannot release capacity still in use. Confirmed removed repositories release reservations on the next admission. Expired repositories and orphan reservations remain reserved until operator reconciliation. This deliberately favors conservative capacity accounting over overselling. The account usage display uses reserved capacity once a ledger exists. A moved repository can retain conservative reservations in its previous scope until removal.
For billing-managed repositories, on-demand recovery and ZIP downloads from truncated trees are rejected with `repository_tree_incomplete`. This prevents serving unaccounted files beyond the indexed tree. A complete source tree is required; refresh after reducing repository size or excluded content upstream.
Billing records are in separate `billingaccounts`, `billinggrants`, `billingquotas` and `billinglocks` collections. They are not serialized on anonymous repository pages. Account deletion cancels recurring subscriptions and expires any known open checkout before deleting content. Minimal provider IDs and financial records remain for reconciliation. Do not disable billing configuration while active subscriptions still need webhook processing or cancellation.
## Verification and operation
Run focused policy and MongoDB tests against a disposable database:
```sh
npx mocha test/billing.test.js
BILLING_TEST_MONGODB_URI=mongodb://127.0.0.1:27017/billing_test_paid npx mocha test/billing-integration.test.js
npm run test:ui
npm run build
```
The integration suite deletes data in its test collections and requires a database name beginning with `billing_test_`. It covers concurrent admission, duplicate fulfillment, customer binding, refunds, subscription cancellation, failed renewal, stale event writes and conference activation. It uses a fake Stripe client and real MongoDB; it does not charge cards.
Before production rollout, run a Stripe test-mode purchase for each configured price, renewal with a test clock, failed renewal, cancellation, refund and webhook retry. Confirm currency, tax behavior, invoice details, capacities and backups. Inspect webhook delivery failures in Stripe. The in-app Refresh payment status action reconciles the latest checkout and known subscriptions; it is not a substitute for webhooks. Failed checkout requests preserve their idempotency key so retrying the same plan recovers the session.
Back up billing collections with the rest of MongoDB. Watch source bytes, physical storage, bandwidth and request rates separately. Stripe fees, taxes and refunds reduce gross revenue.
### Coauthors and the free allowance
Only repositories owned by an account consume its allowance. Joining another author's repository does not consume a slot, and coauthors do not need subscriptions. A coauthor's own subscription does not increase the owner's allowance. Edits and refreshes use the owner's allowance, or the repository's paid conference/pass allowance. If that allowance is full, a collaborator's attempt to refresh or enlarge the repository can also be blocked. Existing review links remain available.
The free defaults are 5 repositories and 2 GiB, revised from the initial 10-repository proposal after the November 2, 2026 production audit. The audit counted ready, unexpired repositories; stored sizes were estimates rather than a fresh recount. Paid author subscriptions retain 20 repositories, 10 GiB and automatic refresh.
### Never-expiring repositories
Monthly and annual author subscribers can choose Never expire. This requires the repository owner's subscription even when a conference pays its storage allowance. A coauthor subscription or a one-time review pass does not unlock it.
Newly selected Never expire repositories store a billing deadline 90 days after the paid-through date. The five-minute maintenance job extends that deadline on renewal and changes overdue repositories to ordinary removal expiration. Cancellation does not shorten the paid-through period. Revocation does not extend a previously recorded deadline. Renewal during the grace period preserves the repository; after expiration, the author must renew and explicitly reactivate it.
Legacy never-expiring repositories have no billing deadline and remain unchanged. Disabling billing disables this reconciliation. Free users can choose a dated expiration within one year and extend it manually. Never expire does not exempt a repository from quotas, abuse controls, or service availability.
Real sandbox validation and repeatable API test instructions are recorded in [Stripe sandbox validation](stripe-sandbox-validation.md).
## Billing recovery and operation
`BILLING_CHECKOUT_ENABLED=false` stops new purchases while webhooks, reconciliation,
the customer portal, and cancellation remain available. Keep `BILLING_ENABLED=true`
and the existing Stripe price IDs configured while subscriptions exist.
Verified webhook routing references are durably inserted into `billingevents` before
HTTP acknowledgement. A ten-second worker claims events with a five-minute lease,
retries failures with exponential backoff, and marks an event failed after 12 failed
processing attempts. Repeated delivery cannot create another inbox record. A worker
that dies after fulfillment can replay the operation without allocating another grant.
Completed envelopes are removed after 90 days; failed events remain for investigation.
The inbox deliberately omits customer names, addresses, card data, and full payloads.
Do not register Connect events on this application's platform webhook.
Every five minutes, a separate job selects up to ten due billing accounts. Each account
is normally reconciled daily; failures retry after 15 minutes. Reconciliation paginates
subscriptions, checkout sessions, and charges, including historical purchases absent
from MongoDB. It also reconstructs confirmed paid periods from invoice lines when the
latest invoice failed. This full customer-history scan favors recovery at the current
scale. Measure Stripe request volume before shortening the daily interval.
Before the 90-day deadline triggers expiration, maintenance performs a fresh owner
reconciliation. Provider or database failures defer that expiration and log an error.
Legacy never-expiring repositories remain excluded. The recorded grace period does not
restart during retries. Ordinary repository requests continue using local entitlements.
Checkout retries retain their key and exact creation parameters. Sessions have an
explicit expiry. After 23 hours, unresolved requests are searched by purchase-attempt
metadata before any retry. After 24 hours, an unmatched attempt can be replaced only if
its immutable request contains an expired session deadline. Older records without that
proof require operator review. Uncertain customer creation also stops after 23 hours;
customer discovery can recover the mapping without creating another customer.
Partial refunds preserve access. Full refunds and disputes suspend the affected grant.
`billingrevocations` records the payment decision even if its grant does not exist yet.
A dispute after a reviewed refund requires review again. Reinstatement is an explicit operator
decision; it does not create paid time or restart a subscription. Do not grant a refund
merely to cancel renewal.
The existing application error log receives billing failures and five-minute health
warnings for failed events, old pending events, unresolved checkout attempts, and failed
reconciliation. These warnings also appear in the existing admin error viewer. External
email or paging delivery still requires deployment monitoring configuration.
### Operator commands
Run with the application's normal server configuration and database access. Inspection
is read-only; mutation commands require `BILLING_OPERATOR` and an audit reason. They
write an intent record to `billingaudits` before performing work. Inspect resulting
state after a command failure; an audit intent alone does not prove completion.
```sh
node -r ts-node/register src/scripts/billing-operations.ts health
node -r ts-node/register src/scripts/billing-operations.ts inspect OWNER_ID
BILLING_OPERATOR=operator-name node -r ts-node/register src/scripts/billing-operations.ts reconcile OWNER_ID 'Investigating missing paid access'
BILLING_OPERATOR=operator-name node -r ts-node/register src/scripts/billing-operations.ts replay test:platform:evt_ID 'Retry after correcting configuration'
BILLING_OPERATOR=operator-name node -r ts-node/register src/scripts/billing-operations.ts drain all 'Process queued recovery work'
BILLING_OPERATOR=operator-name node -r ts-node/register src/scripts/billing-operations.ts discover all 'Recover customer mappings after restore'
BILLING_OPERATOR=operator-name node -r ts-node/register src/scripts/billing-operations.ts reinstate pi_ID 'Dispute resolved; access reinstatement approved'
```
Discovery binds only customers tagged for this application to existing users. Conflicting
customer mappings require investigation. `reinstate` reviews a payment reversal and
keeps a subscription suspended if other unreviewed payment reversals remain. Reconcile
the owner afterward. Restrict shell access to authorized operators; these are not public
HTTP endpoints.
### Restore and launch checks
Back up `billingaccounts`, `billinggrants`, `billingquotas`, `billinglocks`, `billingevents`,
`billingrevocations`, `billingprices`, and `billingaudits` with repository data. Test a restore into an
isolated environment with sandbox credentials. Keep destructive repository maintenance
stopped during restore, discover customer mappings, reconcile restored owners, drain
queued events, and inspect health before re-enabling normal scheduling. Never connect a
restored test copy to live Stripe credentials.
Before launch, configure seller details, live prices, tax treatment, invoice information,
portal restrictions, production email delivery, backup retention, and external alerts.
These are deployment/business settings, not changes made by this implementation. Grace
period email reminders are not currently sent by the app.
The recovery suite uses a disposable MongoDB database and simulated Stripe failures:
```sh
BILLING_TEST_MONGODB_URI=mongodb://127.0.0.1:27029/billing_test_recovery npx mocha test/billing.test.js test/billing-integration.test.js test/billing-reliability.test.js
```
## Payment within the repository workflow
Repository hosting choices now appear in the anonymization form when capacity or a
new Never expire selection requires them. Free dated submissions and coauthor work
keep their existing flow. Conference admission and final capacity are still checked
server-side; an account usage display cannot authorize or permanently block admission.
Existing repository owners can also buy a repository-scoped pass from that form.
Before leaving for hosted Stripe Checkout, the app flushes pending field edits and
saves a private task draft. `/api/billing/draft` requires authentication and the billing
CSRF token. Drafts retain only the form fields needed to resume, are limited to 64 KiB,
and expire after seven days. Retrieval checks the owner and expiry even before MongoDB's
TTL sweep runs. Account deletion also removes these drafts.
Checkout returns to the draft's validated anonymization route, or the conference
being purchased, rather than always returning to Settings. No request-supplied external
return URL is accepted. An open or uncertain checkout attempt cannot be reused for a
different task. Returning restores the draft, refreshes billing and capacity, and leaves
submission to the author. The return token is consumed after restoration so later
navigation cannot overwrite newer edits. Confirmation does not create a repository.
The payment form remains Stripe-hosted. Embedded Checkout is not required for this
workflow and is not enabled by this change. Settings remain available for invoices
and subscription management.
Validation: 40 billing tests with disposable MongoDB and 160 frontend tests passed.
Additional HTTP checks cover draft CSRF, authentication, ownership and no-store headers.
The built UI was checked at desktop and mobile widths using simulated payment responses;
redaction edits survived the checkout return. These workflow changes were not charged
through a fresh real Stripe sandbox payment or deployed to production.
### Current application layout
The anonymization page keeps the main configuration sections visible beside a live
README preview. Source controls retain the existing GitHub connection selector,
manage-access action and branch refresh button. Conference code is a visible optional
field. Hosting choices appear within configuration, with saved drafts preserved through
Stripe Checkout and return.
Settings separates anonymization defaults, hosting and billing, and account actions.
The header Upgrade action sits beside the user menu and opens the plan comparison.
Active subscribers see subscription management instead of another purchase offer.
The donation entry remains available.
Local browser checks used simulated payment responses for the final layout. The real
Stripe sandbox checks and their limitations are documented in
[stripe-sandbox-validation.md](stripe-sandbox-validation.md).
### Resolved source-file storage
Git LFS pointer sizes do not represent cached object sizes. Source-cache fills reserve
bytes beyond the tree size before writing them, in blocks of at most 1 MiB. The ledger
keeps per-path high-water marks so retries and concurrent fills do not charge twice.
Reservations remain conservative across generations and failed fills and require
operator reconciliation to release unused capacity. This also covers public raw
downloads and expanded submodule content. Shared-cache hits and legacy content promoted
to the shared cache also reserve their resolved size before serving. Legacy cache hits
outside the shared cache are not retroactively measured by this write-time guard.
Accepted one-time prices are stored in `billingprices` when checkout starts and when
existing purchases are fulfilled. Hiding an offer by removing its configured price
does not remove this mapping or invalidate paid grants. Include the mapping in backups
so reconciliation can reconstruct grants for retired offers and process their refunds.
docs/performance-monitoring.md
+65 -269
View File
@@ -89,274 +89,70 @@ invalidate them.
Raw Markdown served after a rendering failure has a separate validator, so later
requests retry rendering when the transient failure clears.
## Production failures and restarts
Run `node scripts/check-production-logs.js 15` from a machine with Node and the
Docker CLI to summarize recent application logs and container states. For a
remote Docker host, use
`DOCKER_HOST=ssh://ubuntu@host node scripts/check-production-logs.js 15`.
The optional
window is 1–1440 minutes. The report includes completed 5xx responses, interrupted
requests, token-refresh failures, missing-file warnings, native unhandled stream
errors, and Docker restart counts. It prints counts and container metadata;
request URLs, repository IDs, file paths, credentials, and Docker environment
variables are excluded. Restart counts cover the container's lifetime, while log
counts cover the selected window. Compare restart counts with the last deployment
baseline even when every container is currently healthy.
The API and streamers log `process_fatal_error` through Node's
`uncaughtExceptionMonitor`. This records fatal errors before the default process
exit; it does not resume execution or swallow uncaught exceptions. Native stderr
may describe the same crash as well, so fatal log indicators are not a count of
unique process exits. Use Docker restart counts to confirm exits.
Legacy OAuth reset requests are coalesced by owner, credential, and OAuth app.
A failed 401/404 reset backs off for 30 minutes; transient failures back off for
one minute. A replaced credential or changed OAuth configuration gets a new key.
GitHub App and personal access tokens do not use the OAuth reset endpoint. Refresh
failure leaves the credential unchanged and continues through normal token
validation and fallback.
Confirmed GitHub content 404s back off for 30 seconds in a bounded process-local
cache. The key includes repository, path, commit, blob SHA, cache generation,
revision, and credential. Permission failures and transport errors are not cached.
This reduces repeated upstream misses without changing the file tree or treating
an unavailable file as a successful cache fill. Persistent 404s still require
checking the repository's source snapshot. Application rate limits remain in
force; 429 replies now include `rate_limited`, `resetAt`, and `Retry-After` so
clients can wait for the actual store reset.
## Large repository metadata
GitHub tree preparation consumes pages of plain metadata and persists at most
500 entries per insert. Preparation does not read the finished tree back into
memory. Traversal rejects more than 100,000 queued folders with
`repository_tree_too_large` and HTTP 413. Folder listings and folder-count responses reject results above 10,000
entries with `file_listing_too_large` and HTTP 413. Narrower folder reads and
the bounded filename search remain available. This limit also applies to
recursive internal listing calls.
A preparation reset switches tree generation and retires the old file rows and
active path-index key atomically. It does not wait for metadata deletion before
enqueueing the replacement. Maintenance discovers these retirement markers and
retries interrupted deletion. Activation performs at most ten batches per
retired metadata collection before deferring the remainder to maintenance.
Each metadata delete selects at most 500 IDs, rechecks its filter during the
write, and yields between batches. Expiration and removal also check lifecycle
ownership between batches. `file-tree` logs report deleted rows and elapsed
time as `metadata cleanup complete` or `metadata cleanup deferred`.
Run `npm run migrate:performance` when upgrading to add the sparse
`retiredPathIndexKeys` maintenance index without dropping existing indexes.
The constrained-heap regression fixture can be run without network or MongoDB:
`node --max-old-space-size=256 test/fixtures/large-repository-tree.js`.
## Shared blob cache
The optional shared filesystem cache stores immutable source content across projects.
It uses Node's built-in SQLite module and requires Node 22.15 or newer. All API and
streamer processes on one host must mount the same local cache directory. SQLite
WAL is not a distributed lock service: do not put this directory on NFS or share
it across hosts. S3 storage continues to use the existing storage implementation.
Enable the integrated implementation with `SHARED_CACHE_ENABLED=true`. The default
is false so a code deployment does not silently migrate production data. The other
settings are independent rollback controls:
- `SHARED_CACHE_PATH`: defaults to `.shared-cache-v1` inside `FOLDER`.
- `SHARED_CACHE_BYTES`: source pool budget, default 10 GiB.
- `SHARED_CACHE_RESULT_BYTES`: transformed output budget, default 512 MiB.
- `SHARED_CACHE_MAX_FILE_BYTES`: maximum source admission size, default 32 MiB.
- `SHARED_CACHE_FREE_BYTES`: free-device-space reserve, default 1 GiB.
- `SHARED_CACHE_LOW_WATER`: eviction target fraction, default 0.9.
- `SHARED_CACHE_GZIP`: store gzip when beneficial, default true.
- `SHARED_CACHE_GZIP_MIN_SAVING`: minimum fractional saving, default 0.10.
- `SHARED_CACHE_DECISIONS`: enable unchanged decisions and changed-output reuse,
default true.
- `CACHE_MONITORING_ENABLED`: collect cache observations, default true. Monitoring
is independent of shared storage and also records legacy cache/inspection events.
These are initial limits, not capacity recommendations from the production sample.
Restart processes together when changing pool limits. Reserve headroom for SQLite,
filesystem directories, existing caches and other application temporary files.
The budget accounts for blob allocation with a conservative 4 KiB minimum plus
in-flight reservations of three times the maximum object size and a small overhead.
Keep each pool budget larger than this reservation. Admission can bypass the cache
while other fills/readers occupy the budget. Oversized source objects are streamed
without admission, still subject to the application's file-size limit.
Git blobs are verified against the requested Git SHA before publication. Resolved
LFS content is admitted only after verifying both the Git pointer and its SHA-256
payload digest and size. Gzip is a representation of the same uncompressed content.
Files are written privately, synced and atomically renamed before index publication.
Fills and readers have renewable leases. Eviction skips readers, and stale fills
are recovered before their reservations are released. LRU access timestamps are
coarsened to 30 seconds. Orphan reconciliation runs at most hourly across processes.
Alias and reuse-history indexes are bounded; decisions expire after seven days and
are capped at 100,000 entries.
Existing generation caches are read as a migration fallback and verified before
promotion. They are not deleted during promotion, so rollback remains possible.
The new pool budget does not include legacy copies: monitor total filesystem usage
and retire legacy data only after validating rollout. Uploaded ZIP originals and
other durable source material are untouched. Repository removal invalidates access
and project mappings, while shared blobs remain subject to global eviction.
The HTTP fast path applies to individual file responses backed by the shared source
pool, including small text files. Decisions include content identity, path, project
context, all transformation options, generation/revision, mask, hostname and engine
version. Unchanged decisions serve the original representation. Changed output has
its own bounded pool. Gzip negotiation honors quality values, uses `Vary`, and
preserves MIME type. File validators are weak when shared delivery is enabled so
identity and gzip representations can revalidate the same logical content. HEAD
continues to use metadata only. Range requests receive a complete response; this
implementation does not add random access into gzip files. Website rendering keeps
its renderer cache; ZIP scans do not admit bulk entries into the source blob pool.
The admin overview's Cache section reads `/api/admin/cache?minutes=60` behind the
existing admin authorization. Windows are 15 minutes, one hour, 24 hours and seven
days. SQLite telemetry batches at most 512 groups every five seconds and retains
minute aggregates for 48 hours and hourly aggregates for 30 days. No paths, terms,
user IDs or blob hashes appear in metric labels. The UI distinguishes unavailable
telemetry, dropped events, disabled storage and empty windows. Rows separate file,
website, ZIP, background and known-monitor traffic. Monitor classification is an
operational heuristic, not a complete bot detector.
Hit counts and byte hit rates measure source lookups; misses classify first observed
keys, known misses and eviction refetches within bounded seven-day history. Reuse
intervals are consecutive blob accesses, not distinct people or browser visits.
Eviction time measures idle age; codec and inspection timings are elapsed work time,
not CPU samples. Cached decisions are separate from actual inspections. Minute and
hour boundaries make the displayed windows approximate.
Run the reproducible acceptance workload with:
```sh
node scripts/cache-workload.js docs/cache-workload-results.json
npx mocha test/shared-blob-cache.test.js test/shared-cache-http.test.js
```
The JSON report asserts cross-project source reuse, unchanged-decision reuse,
changed-output reuse, gzip delivery and budget enforcement. Its compression ratio
comes from synthetic text and must not be presented as a production projection.
Production rollout should compare request/byte hit rates, eviction refetches, p95
latency, upstream errors and free disk over a representative week, with large files,
ZIP work and known monitors visible separately.
### Benefits and costs
| Choice | Benefit | Cost or limit |
| --- | --- | --- |
| Global content identity | Reuses identical files across paths and projects; renames do not duplicate content | Requires verified hashes and authorization on every request; uses digest-prefix directories to avoid one huge directory |
| Byte budget with approximate LRU | Predictable blob footprint and automatic reclamation | Evicted files incur upstream fetches; active readers can temporarily prevent admission |
| Selective gzip | Reduces payload storage and can serve compressed bytes directly | First-fill compression costs work; identity consumers and uncached anonymization require decoding |
| Rule-aware decisions and results | Repeated unchanged reads skip regex inspection; changed output is reusable | More metadata and a separate output budget; settings changes produce new decisions |
| SQLite coordination and telemetry | Cross-process coordination without another service; admin can measure actual reuse and rewrite work | Local-filesystem scope; short synchronous transactions; metadata/headroom are outside blob budgets |
### Rollout sequence
1. Deploy with shared storage disabled and collect legacy hits, inspections and codec
timings. Keep known-monitor traffic separate from file readers.
2. Enable the shared cache on the single-host filesystem deployment with both byte
budgets configured. Keep legacy copies for rollback.
3. Compare a representative week of byte/request hits, rewrite decisions, eviction
refetches, request latency, upstream errors and whole-device free space. Adjust
admission size and budgets from those observations.
4. Retire legacy cache copies only after the rollback window. Uploaded originals
remain durable and are never candidates for shared-pool eviction.
The implementation and acceptance workload are local changes; this work does not
enable or migrate production storage.
### Production observations used for this design
The read-only audit on 2026-11-02 found 425,312 source files across 31,522 projects,
containing 144.32 GB of payload. Deduplication within projects could remove 1.87 GB;
global deduplication could remove 2.93 GB total (about 2.03%), including another
1.06 GB across projects. One file could not be read. These are decimal GB and
payload savings, not a compression estimate or the expected new working-set size.
The preceding 24-hour access sample contained 29,980 successful file-body responses,
including 9,249 repeat responses (30.9%). URL repetitions are not the same as blob
cache hits. A separate approximately two-hour metric window recorded 2,302 source
hits out of 3,600 lookups (63.9%). This supports retaining a bounded source cache;
it does not establish the optimal byte budget.
In the uniform anonymization replay, 937 resident files were measured: 764 text,
173 binary, and 21 with changed output. Thirteen resolved files exceeded the 4 MiB
sample limit, and missing files/mappings were excluded. The sample's mean gzip-6
work was 1.46 ms per file and mean decompression was 0.24 ms; these replay timings
are not live-request latency or a forecast for large files. Continuous telemetry is
needed to replace these samples with representative workload measurements.
### Local acceptance evidence
The checked-in `cache-workload-results.json` is generated by assertions, not hand
written. Five file requests across two projects needed one upstream source fill;
unchanged and changed responses reused decisions, and changed bytes occupied one
result blob. The 64 KiB eviction workload ended at 53,248 allocated bytes with no
remaining fill reservations. Compression figures describe this synthetic fixture.
Validation also covers separate operating-system producers, failed/invalid fills,
crash publication recovery, active-reader eviction protection, LFS pointer/payload
verification, gzip negotiation, rule changes, admin authorization and HEAD/304.
The disposable MongoDB/Redis performance integration run passed 130 tests. The
admin Cache section was checked at 1280 px and 390 px, including filter layout,
scrollable tables, empty filters and unavailable telemetry.
The final complete default test run recorded 1,062 passing, 159 pending and one
failure: `test/error-codes.test.js` reports nine missing frontend translations for
the pre-existing maintenance API changes. Cache tests passed in that full run.
TypeScript compilation, the scoped lint checks (existing warnings only), the UI
build and `git diff --check` passed. Production rollout was not performed.
## Initial cache performance evaluation, 2026-11-02
The initial implementation passed functional checks, but its metadata write path regressed small-file performance. The following measurements are preserved as the before-optimization baseline; the implementation and follow-up results below supersede the recommendations in this section. These are local measurements, not production capacity estimates.
### Warm changed-text responses
Each percentile pools two batches. Serial measurements contain 32 requests; concurrency-eight measurements contain 64. The small/medium baseline reads a local raw source and repeats anonymization/gzip. The 2 MiB baseline additionally uses the existing transformed-file cache, with 99 measured cache hits out of 100 requests. Authentication and Redis coordination are outside this comparison.
| File | Baseline median, ms | Current median, ms | Experimental NORMAL median, ms | Baseline p95 at 8 clients, ms | Current p95 at 8 clients, ms | NORMAL p95 at 8 clients, ms |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| 16 KiB | 8.3 | 44.9 | 6.9 | 71.8 | 462.1 | 46.8 |
| 256 KiB | 36.5 | 45.9 | 11.7 | 250.8 | 444.1 | 89.6 |
| 2 MiB | 236.2 | 60.5 | 26.0 | 929.2 | 528.2 | 324.8 |
### Findings
- The cache hit path performs synchronous SQLite writes for reader leases, reuse history and release. The temporary-database A/B changes only pool connections to `PRAGMA synchronous=NORMAL`; the measured reduction supports commit latency as a major bottleneck. This is a diagnostic experiment, not an implemented fix.
- NORMAL in WAL mode can lose recent committed transactions following power loss or a system crash. Validate recovery/accounting before changing the durability of disposable cache indexes. See [SQLite synchronous documentation](https://www.sqlite.org/pragma.html#pragma_synchronous).
- For changed 2 MiB content at eight clients, current shared caching reduced combined client/server CPU from 253.2 to 55.6 ms per request. Peak sampled RSS was 308.0 versus 296.7 MiB. These are whole-process values, including runtime, HTTP client and worker threads.
- Synthetic text transferred about 62.2% fewer bytes with gzip. Incompressible 256 KiB binary data was stored and served as identity, avoiding ineffective gzip storage. These compression ratios are not production projections.
- Cold responses include first-use initialization. For 16 KiB changed text they took 145.9 ms in the baseline, 709.8 ms currently, and 212.0 ms in the NORMAL experiment. Do not use this single cold sample as steady-state miss latency.
- At 100,000 synthetic blob rows, a single pool stats query took 78.7 ms p95 synchronously; saving a decision took 45.5 ms p95. Cache/materialize admin aggregates and reduce retention scans on request paths. Dummy blob rows have no files, so this measures metadata scaling, not full directory reconciliation.
- A 4 MiB pool cycling sequentially over 100 distinct 64 KiB objects stayed below budget, but all 300 reads fetched again. Its working set was larger than the pool. Reuse, request order and admission policy must inform the budget; LRU alone cannot guarantee hits.
### Reproduce and limitations
Node v24.14.1, Linux, Intel Xeon E312xx (Sandy Bridge, IBRS update), 12 logical CPUs. This is a shared host with background load, warm OS page cache, loopback HTTP and a synthetic 20 ms upstream wait. It excludes real upstream bandwidth, TLS, reverse proxies, database authorization, multi-process throughput and production request distributions. Small samples and fixed scenario order mean p95 and speedup ratios are descriptive. An initial ts-node exploration was superseded by the compiled results below. The metadata probe does no worker execution, and excludes TypeScript loading from its timed samples.
```sh
npx tsc --incremental false
CACHE_BENCH_COMPILED=1 node scripts/cache-performance.js
CACHE_BENCH_COMPILED=1 node scripts/cache-performance.js docs/cache-performance-normal-results.json --normal
CACHE_BENCH_COMPILED=1 node scripts/cache-performance.js docs/cache-performance-legacy-results.json --legacy
node scripts/cache-index-performance.js
```
All 2,700 HTTP responses were checked against expected bytes or a validated wire digest. Eviction assertions passed. Raw latency samples, CPU, RSS, event-loop delay, storage and metric counters are in `cache-performance-results.json`, `cache-performance-normal-results.json`, `cache-performance-legacy-results.json` and `cache-index-performance-results.json`. `cache-performance-summary.json` contains pooled comparisons.
Before rollout: reduce per-read durable commits without weakening active-reader eviction protection; consider NORMAL only for rebuildable indexes with crash/recovery tests; replace full admin stats scans with cached aggregates; then replay a representative mixed workload with several server processes. Application defaults and production were not changed by this evaluation.
## Optimized implementation, 2026-11-02
The pool now uses NORMAL WAL commits for reader leases, coarsened access history,
anonymization decisions and telemetry. Admission reservations, publication intent
before rename, and final blob/alias publication still use FULL commits. Payload
fsync remains in place. Reader leases still become visible immediately to other
processes and protect active streams from eviction. This saves hit-path disk
flushes without making payload accounting depend on disposable history.
Recent disposable metadata and telemetry can be lost after system failure.
Reservation and publication recovery conservatively retain accounting until files
are indexed or removed; missing payload files are invalidated and fetched again.
The SIGKILL test stops an actual producer after rename and before index publication,
then verifies reservation retention, orphan cleanup and successful refill. This
tests process failure, not a physical power cut.
Admin pool statistics now read transactional counters and an indexed oldest-access
lookup. Counters are initialized once for an existing index and maintained by
SQLite triggers in the same transactions as row changes. Decision/history/alias
retention deletes only excess rows through indexes. Expired decision/history
cleanup is limited to 256 rows each per admission or maintenance pass. Access
history persists at most once per key per 30 seconds, so reuse-window boundaries
have that resolution. Metadata remains capped at 100,000 decisions/history rows
and 200,000 aliases per pool.
Stop all cache-using processes together before upgrading an existing index, then
restart them on the same version. Do not mix old and new writers during migration.
The admin API response shape is unchanged. Shared caching remains opt-in with
`SHARED_CACHE_ENABLED=true`; production configuration was not changed.
The tradeoff is intentional: cache hits avoid durable flushes, while cold fills
still pay for durable publication and compression. Trigger updates add a small
amount of write work in exchange for constant-time aggregate reads. Approximate
LRU and fixed budgets can still thrash when the working set exceeds the pool.
### Follow-up measurements and validation
All 2,200 responses in the complete optimized HTTP run and all 300 in the focused
repeat matched expected output. The repeat ran after this task's other tests and
benchmarks ended. The early scenarios in the complete run overlapped the metadata
probe, so both results are retained rather than treating the faster run as a
universal estimate. These remain small synthetic samples on a shared host.
- 16 KiB changed text: warm serial median 44.9 → 7.3 ms; eight-client p95 462.1 → 114.9 ms.
- 256 KiB changed text: warm serial median 45.9 → 9.0 ms; eight-client p95 444.1 → 107.6 ms.
- 2048 KiB changed text: warm serial median 60.5 → 19.0 ms; eight-client p95 528.2 → 216.2 ms.
At 100,000 synthetic blob rows, admin stats p95 fell from 78.70 to 0.50 ms,
decision-write p95 from 45.47 to 1.11 ms, and warm-source-hit p95 from 82.99 to
6.37 ms. The 4 MiB eviction probe peaked at 3,997,696 allocated bytes; all budget
assertions passed. Its oversized sequential working set still produced no hits.
The final full suite recorded 1,065 passing, 159 pending and the existing single
error-code translation failure (nine maintenance API codes). New counter,
durability-mode and SIGKILL recovery tests passed, as did HTTP and admin UI
regressions. TypeScript, scoped lint and diff whitespace checks passed.
Production rollout and representative multi-process throughput testing remain
outside this local evaluation.
Reproduce the follow-up without overwriting the original baselines:
```sh
npx tsc
CACHE_BENCH_COMPILED=1 node scripts/cache-performance.js docs/cache-performance-optimized-results.json
node scripts/cache-index-performance.js docs/cache-index-performance-optimized-results.json
CACHE_BENCH_COMPILED=1 node scripts/cache-performance.js docs/cache-performance-optimized-repeat-results.json --shared
```
Raw results and the pooled comparison are in the corresponding
`cache-*-optimized-*.json` files.
See [Shared blob cache](shared-cache.md) for configuration, storage and quota
accounting, admin metrics, production observations, rollout and benchmark commands.
docs/review-callback-handoff.md
+13
View File
@@ -0,0 +1,13 @@
# Returning owner consent to review
An optional callback registry maps the authenticated intent's client and callback IDs to one exact HTTPS URL. Configuration rejects credentials, queries, fragments, noncanonical paths, duplicate IDs and duplicate JSON fields. It permits at most 32 entries and a 4 KiB form-action policy. Callback selection never uses a browser-supplied address.
After recording consent, the browser can request `POST /api/review-consent/handoff` with its signed ticket. The existing transport requires the current authenticated session, same origin and a session-bound CSRF token. The backend resolves the signed intent's callback before issuing a completion code and checks the current repository authority and saved consent. The transport reloads the session before returning the result. A missing registry leaves this endpoint unavailable.
The page submits the completion envelope as the sole `completion` field in a native HTML POST form. Codes never enter URL parameters or browser storage. The page's form-action policy allows only the configured destinations and its own origin. It removes the temporary form when CSP blocks submission and shows a retry/status message. A late result after an account change cannot submit a form. The signed ticket stays in a controller closure for a retry; it is not submitted to the callback.
The initial cross-site POST does not carry a review session cookie configured with SameSite=Strict. The review receiver must serve a non-caching landing page, then use a same-origin authenticated request and saved initiating-browser state before exchanging the code. Receiving the POST alone must not establish a binding. That receiver and startup activation are separate unfinished work. This change does not mount the upstream consent router or enable an integration.
Validation covers callback configuration, current-session HTTP access, concurrent logout, signed callback scope, consent requirements, browser POST contents, account changes and malformed completions. The disposable MongoDB suite contains 23 passing tests. The local 100-replay binding benchmark measured median 2.20 ms and p95 2.69 ms on a one-CPU, 1 GiB container; this excludes provider traffic. Browser fixture results use synthetic HTTP backend responses and test TLS certificates, so they do not establish provider or certificate-verification behavior.
The full local suite passed 762 tests, with 74 pending tests for optional fixtures. The targeted browser/page suite passed 17 tests, registry/HTTP tests passed 16 with one optional performance case pending, and TypeScript, targeted lint and the UI build passed. Chromium at 1280 and 320 pixels completed the native cross-site POST with no referrer or Strict cookie, no overflow, no external requests, no script errors and no WCAG A/AA violations. A third case blocked the destination with CSP and verified zero callback deliveries and removal of the code form. The first browser harness waited for a navigation that CSP intentionally blocked; changing the assertion to inspect the current document completed this case without changing application code.
docs/review-completion-exchange.md
+15
View File
@@ -0,0 +1,15 @@
# Completion codes and binding receipts
The owner consent backend now saves the authenticated review intent with its immutable consent receipt. Existing consent records that lack this scope cannot issue completion codes. No caller may supply replacement submission, entitlement or policy fields.
`completion(actor, ticket)` requires the same current browser account and session, a matching confirmed consent and current repository authority. It issues one code per intent, expiring at the earliest of five minutes, intent expiry or retention expiry. Repeating issuance returns the same code without extending its lifetime. The database stores a random nonce and a code hash. A domain-separated HMAC using the private consent signing key makes the code recoverable after response loss without storing it in plaintext. Retain that key for the code lifetime; changing it fails closed for outstanding issuance retries.
`exchange(authenticatedClientId, command)` requires a separately verified service client. The argument is an internal authority boundary, not authentication by itself. First exchange checks the code, expiry, current account and repository authority, then atomically inserts a pending binding, a client-scoped request key and the original receipt. A later write failure rolls all three back. Competing transactions may report unavailable; callers explicitly retry the original command. The backend never automatically repeats a transaction.
Exact authenticated exchanges recover the original receipt after response loss, code expiry or later owner changes. They cannot create another binding. Changed codes under the committed request ID conflict, a spent code cannot be used with another request ID, and an existing request ID cannot bind another intent. `receipt(authenticatedClientId, intentId)` provides post-expiry reconciliation without accepting an expired code for first use. Both operations remain client scoped and expose only protocol identifiers and policy. They never return source coordinates, account IDs or repository IDs.
Bindings remain pending. No snapshot, artifact access, retained bytes or entitlement fulfillment is claimed. Browser completion POSTs, callback registration, service HTTP authentication, private configuration, startup wiring and activation remain separate tasks. The existing browser confirmation response intentionally does not expose the completion code yet.
Receipt recovery uses majority reads outside transactions. The factory limits active authority transactions and service reads to four each. MongoDB queries have three-second limits, majority writes have a three-second acknowledgment timeout and commits have a five-second limit. The serving connection must also have bounded selection/socket/pool waits. Secrets and encoded completion payloads must never be logged.
Tests use a disposable local MongoDB replica set and cover current ownership, absent consent, cross-session and cross-client attempts, altered codes, expiry, explicit replay, concurrent exchange, request-key reuse, rollback and persistence without raw tokens or codes. Set `TEST_REVIEW_CONSENT_MONGO` to the disposable replica-set URI and run `npx mocha test/review-owner-consent.test.js test/review-consent-http.test.js`. `TEST_REVIEW_COMPLETION_PERF_REPORT` records 100 immutable receipt replays; it excludes browser, TLS and provider latency.
docs/review-completion-http.md
+15
View File
@@ -0,0 +1,15 @@
# Completion service transport
`createReviewCompletionService` serves the exact `POST /service/v1/completions/exchange` and `GET /service/v1/intents/{intentId}/receipt` routes when explicitly mounted before body parsing and browser authentication. Startup does not mount it. An absent scoped registry returns typed 404 responses without touching the backend.
The private registry has exactly `version: 1` and `keys`. Each key contains `clientId`, `keyId`, `tokenSHA256`, `notBefore`, `notAfter` and `scopes`. Identifiers, ASCII-token hashing and whole-second validity windows follow the capabilities registry. Each key must explicitly authorize `completion.exchange`, `receipt.read`, or both. The existing capabilities-only configuration is rejected because it has no operation scopes. Limits are 64 KiB, 64 keys, 32 clients and 16 keys per client. Duplicate fields, key identities, digests and scope entries are rejected.
Requests use `X-4open-Artifact-Client-Id`, `X-4open-Artifact-Service-Key-Id` and `Authorization: Bearer <token>`. TLS is required, directly or through explicitly trusted proxy configuration. Browser cookies, origin/referrer/fetch metadata, duplicate headers, encoded bodies, query strings and route aliases are rejected. Authorization precedes body reads. Constant-time digest comparison and the key window are checked, with validity rechecked before backend execution and before returning a result.
The transport accepts at most 64 KiB of strict JSON. The shared decoder rejects invalid UTF-8, duplicate decoded keys and excessive nesting. Commands have exactly the v1 exchange fields. A successful receipt must contain exactly the expected client/intent scope and v1 receipt fields. Errors contain a fixed protocol code and fresh random correlation ID, never provider or database diagnostics. Responses disable caching, referrers and CORS. Rejections close the connection.
A client is limited to 60 requests per minute across its rotation keys. At most four operations run concurrently. Each request has a 15-second deadline. A disconnected or timed-out caller does not release an in-progress backend slot until the backend settles. A write may commit after response loss; callers must retain and explicitly retry their original request. No automatic retry occurs here.
Tests use real loopback HTTP, synthetic trusted-proxy TLS metadata and a simulated backend. They do not measure incoming TLS, MongoDB or provider latency. Set `TEST_REVIEW_COMPLETION_HTTP_PERF` to capture 50 authenticated receipt reads. The opt-in review-repository interoperability test also passes against this actual TypeScript handler over certificate-verified HTTPS. It covers explicit replay, changed-request conflicts, post-expiry receipt reads, policy mismatch, scope separation and untrusted certificates. Its backend remains synthetic. Credentials, startup wiring, browser callback handoff and activation remain unfinished; capabilities still advertise no available integration features.
The dedicated CI MongoDB fixture uses one CPU, a 1 GiB memory cap and a 512 MiB temporary data mount. The earlier 256 MiB mount failed during replica-set initialization with a no-space error; the increased fixture is still isolated and removed after testing. This does not change production storage or resource limits.
docs/review-consent-http.md
+21
View File
@@ -0,0 +1,21 @@
# Browser transport for review consent
`createReviewConsentRouter` exposes the consent backend through three routes when explicitly mounted at `/api/review-consent`: `GET /csrf`, `POST /preview` and `POST /confirm`. Application startup does not mount this router. Browser UI, private configuration, startup ordering and activation remain unfinished.
The factory requires a canonical HTTPS origin, the consent backend and a dedicated private 32-byte CSRF key shared by serving processes. Mount it after Passport/session authentication and before any JSON parser. The current startup ordering must change before integration; mounting after a parser fails closed. Transport must be HTTPS directly or through the application's explicitly trusted proxy configuration.
Identity comes from Passport's authenticated account and a fresh lookup in the session store. Neither request JSON nor a bearer credential can supply it. POSTs require the exact configured Origin and a constant-time-checked CSRF token bound to the account and session. CSRF tokens are derived with a protocol-specific HMAC; issuing one never saves a session. A late CSRF response therefore cannot recreate a session destroyed by a concurrent logout. Authority is reloaded after backend work before returning a preview or confirmation response.
Requests use exact paths without queries, strict methods, closed JSON fields and literal access/retention acceptance. Cross-origin or cross-site context, bearer authentication, duplicate security headers, content encoding and unexpected content types are rejected. The CSRF GET accepts no body. JSON is limited to 12 KiB; tickets to 8 KiB. A 15-second total deadline closes incomplete requests, and browser disconnects cancel pending preview consumption. Authenticated requests are limited to 60 per account per minute with at most 1,024 live counters.
Responses disable caching and referrers. Confirmations include only the request ID, accepted policy and confirmation time. Internal account/repository identifiers and provider/database errors are omitted. A confirmation already accepted by the backend can commit despite a lost response; the UI must preserve the same ticket and request ID for explicit recovery. This transport does not create artifact bindings or bypass final review-side authority checks.
## Validation
The tests use real Express and express-session middleware with a disposable in-memory session store, a synthetic backend and loopback HTTP representing the trusted TLS proxy boundary. They do not measure real MongoDB or provider latency. TLS verification of the service client and MongoDB authority transactions are tested separately.
Coverage includes session-derived identity, CSRF replay across sessions, cross-origin/bearer rejection, bounded closed payloads, exact routes, session revocation during preview, no session resurrection during a CSRF/logout race, response redaction, fixed errors, rate limits, unauthorized incomplete uploads, browser cancellation and the authenticated 15-second slow-upload deadline. The optional benchmark performs 50 fresh loopback HTTP preview requests. The CI database job runs these HTTP tests alongside the consent transaction tests and retains both performance reports.
The final local targeted run passes 12 cases in 16 seconds. Transport-only timings are mean 4.94 ms, median 4.51 ms and p95 7.26 ms. TypeScript and targeted lint pass.
The final local broader suite passes 723 cases with 63 opt-in/environment-dependent cases pending. The separate CI replica-set job exercises the otherwise skipped consent database tests as well as this transport suite.
docs/review-consent-ui.md
+13
View File
@@ -0,0 +1,13 @@
# Artifact consent page
`/review-link` serves a separate page without the ordinary site's third-party widget or profile header. Its content security policy allows only first-party scripts and connections. The page rejects URL queries and noncanonical aliases, disables caching and referrers, and accepts only filename-shaped manifest entries.
The handoff fragment contains exactly `clientId`, `intentId` and `token`. The application removes it from the current history entry before starting application requests or the router. Intent credentials and signed preview tickets stay in a controller closure. They are not rendered or written to browser storage. Reloading loses the handoff; users must reopen a new link from their submission. Signed-out users sign in before reopening the link.
Owners enter the anonymous repository ID and request a preview. The backend supplies the repository name, access policy, retention deadline and expiry. Confirmation requires two separate unchecked approvals. A failed confirmation keeps the same signed ticket and random request ID for an explicit retry. Repository changes are disabled while that confirmation is uncertain. Account changes and page disposal cancel requests and discard late results. Success says that consent is recorded and that artifact linking remains incomplete.
The page does not mount the consent API or advertise available integration capabilities. Startup configuration, final binding exchange, provider activation and a review-side link launcher remain unfinished. The consent HTTP router still needs to be mounted after authentication and before body parsing.
Validation uses the compiled frontend bundles. Targeted tests cover fragment removal, storage, approval flags, replay identity, departure during CSRF, account changes during preview, invalid handoffs and policies, endpoint unavailability, expiry, sign-in and mismatched receipts. Server tests check canonical paths, headers and unsafe manifests. Chromium checks at 1280 and 320 pixels exercise the full synthetic form flow with CSP enabled and check overflow, external requests, script errors and WCAG A/AA rules. No live database or provider request is involved.
Run `npm run build:ui`, then `npx mocha test/review-consent-page.test.js test/review-consent-ui.test.js`. Set `TEST_REVIEW_PAGE_PERF_REPORT` to a private output path to measure 100 isolated HTML responses over loopback HTTP. This benchmark excludes browser rendering, database operations and provider latency.
docs/review-intent-client.md
+20
View File
@@ -0,0 +1,20 @@
# Review intent consumption client
`src/server/service/review-intent-client.ts` consumes a saved review-side link intent over authenticated HTTPS. The caller supplies a trusted, exact HTTPS origin, a client-bound service credential with a validity window, and an explicit request ID. The returned policy and opaque identifiers come from the authenticated review response. Browser-supplied access, retention and callback fields are rejected.
The factory returns only `consume`; it copies credential values into a closure. A call makes one POST to `/service/v1/artifact-intents/consume`, validates the peer certificate with Node's trust store, and sends no browser cookies or origin. Redirects and automatic retries are disabled. An uncertain result requires the caller to retain the same request ID for an explicit retry.
Limits are four active attempts, a ten-second total deadline, five-second inactivity timeout, 16 KiB response headers and 64 KiB body. An AbortSignal cancels the attempt. Invalid credentials or payloads fail before I/O. Responses must match the configured client and requested intent, use the v1 closed schema and valid future timestamps, and omit cookies and content encoding. Duplicate JSON keys, including escaped spellings, invalid UTF-8 and unknown fields fail. Errors expose only a fixed category; remote bodies and credentials are not included.
This module is not wired into a route or configured in production. It does not authenticate an artifact owner, obtain consent, create a binding, exchange a completion code, or establish browser callback authority. Those are separate pending tasks. Credential distribution and upstream deployment also remain pending.
## Validation
Local tests use a disposable certificate and loopback HTTPS server. A child process loads that certificate through `NODE_EXTRA_CA_CERTS` at startup; an independent child with ordinary trust rejects it before sending credentials. No TLS bypass is used. Tests cover successful policy reads, copied configuration, request-ID replay, rejection before I/O, status classification without retry, malformed and oversized responses, duplicate keys, four-slot saturation and recovery, cancellation, credential expiry, inactivity and slow-stream total deadlines.
- `npx mocha test/review-intent-client.test.js`: 11 behavioral cases; one performance case is opt-in.
- `npm test`: 712 pass, 50 pending opt-in or environment-dependent cases.
- Full TypeScript check and targeted ESLint pass.
- Opt-in 100-request performance run: fresh loopback TLS connections, mean 12.36 ms, median 12 ms, p95 15 ms. These are local synthetic timings, not an external-service latency promise.
Run the measured test with `TEST_REVIEW_INTENT_PERF=1 TEST_REVIEW_INTENT_PERF_REPORT=/tmp/review-intent-perf.json npx mocha test/review-intent-client.test.js`.
docs/review-owner-consent.md
+23
View File
@@ -0,0 +1,23 @@
# Review owner consent backend
`createReviewOwnerConsent` prepares a signed policy preview from the authenticated review intent consumer and records explicit access/retention acceptance in MongoDB. It is an unwired backend. Browser routes, CSRF protection, UI, completion-code exchange and activation remain unfinished. Capabilities continue to advertise no available integration features.
The caller must provide an account ID and session ID derived from a current authenticated browser session. Never populate this principal from request JSON or treat a service credential as an owner's identity. The factory also needs the configured intent consumer, a dedicated random 32-byte signing key held outside Git, and a connected MongoDB replica set. Configure bounded server-selection and socket timeouts on that connection. Tests use MongoDB 7 with a three-second server-selection timeout and ten-second socket timeout.
Preview checks the current account and repository before consuming the review intent, then checks them again afterward. The network request holds no MongoDB transaction or authority lock. The account must be active or have the legacy unset active status; the repository must be ready and unexpired. Ownership uses the current account ObjectId. Coauthors must match the account's current numeric GitHub ID. Legacy username-only coauthors and unrelated administrators cannot provide consent.
The signed preview binds the saved intent and exact policy to the account, hashed session ID and repository ObjectId. Changing the account, session, repository or policy invalidates confirmation. The signing context is specific to this protocol; key rotation invalidates outstanding previews. Confirmation requires both access and retention acceptance and an explicit 32-hex request ID. Preview and replay expire with the intent or retention policy.
Confirmation writes one immutable `review_owner_consents` record per client/intent. It records the account, repository, authority role, exact policy, request ID, confirmation time and a hash of the signed preview. It stores no raw session ID, intent token, service credential or preview ticket. Explicit retries return the same receipt only after current authority checks; another request ID or repository conflicts. No binding is created by accepting a policy.
Transactions touch the current account and repository before writing or reading the receipt. Ownership transfer, coauthor removal and account disablement therefore serialize with confirmation. The private `reviewConsentRevision` fields are excluded from ordinary Mongoose reads. Four transactions may be active per factory. Database commands and commits have limits; transactions are not automatically retried. A conflict or uncertain commit returns a fixed `unavailable` error, and the caller must retain the same ticket and request ID for explicit recovery.
The final binding flow must still revalidate the initiating review session and saved intent scope with the review service. This backend proves current upstream owner/coauthor consent, not continuing review-side author authority, immutable snapshots, restricted file access or preservation. Receipt retention/export and the HTTP adapter remain separate work; this change enables no provider or public route.
## Validation
The opt-in suite runs against a disposable MongoDB replica set. It covers literal acceptance, immutable replay, stable-ID coauthors, administrator/username rejection, ticket tampering, account/session binding, disabled accounts, ownership changes, removed coauthors, archived/expired repositories, permission changes during upstream I/O, concurrent confirmations, expiry, second-repository conflicts and authority-field privacy. A held ownership-change transaction forces a real write conflict; the explicit retry is denied and no receipt is created.
Run `TEST_REVIEW_CONSENT_MONGO='mongodb://127.0.0.1:28743/?replicaSet=review-consent-test' npx mocha test/review-owner-consent.test.js`. Set `TEST_REVIEW_CONSENT_PERF_REPORT=/tmp/review-consent-perf.json` to include 100 durable receipt replays. The CI job starts and removes its own pinned MongoDB image and preserves the performance report. Local timings are synthetic and are not a production latency guarantee.
Local validation passes 12 targeted cases with the benchmark enabled, 723 full-suite cases with 51 opt-in/environment-dependent cases pending, the full TypeScript check and targeted lint. The final 100-replay run measured mean 12.85 ms, median 12.82 ms and p95 14.89 ms against a one-CPU, 512 MiB disposable database. An earlier performance attempt returned `unavailable` while the privacy fixture was automatically building indexes. The log does not establish the underlying database error. The final fixture disables automatic schema/index creation and creates its collections explicitly.
docs/review-service-capabilities.md
+84
View File
@@ -0,0 +1,84 @@
# Review service capabilities
`GET /service/v1/capabilities` is an optional server-to-server endpoint for the
`4open.artifacts/1` contract. It returns the authenticated client ID, the supported
contract version, and an empty `available` list. Linking, snapshots, protected
files, retention, publication, entitlements and usage are not advertised. Clients
must decline those workflows until their required capabilities are implemented.
The endpoint is mounted before JSON parsing, browser sessions and passport. The
entire `/service` namespace returns a typed 404 when configuration is absent.
Only the exact GET path is accepted. Query strings, browser context headers,
cookies, bodies, duplicate authentication headers and unknown paths are rejected.
Rejections close the connection immediately, including requests whose declared
body has not arrived. Responses contain no credentials, owner identity, repository data, cookies or CORS
permission; they have `no-store` and `no-referrer` headers. No database, queue or
provider call is involved.
## Private configuration
Leave `REVIEW_SERVICE_KEYS` unset until activation is approved. If configured,
its value is a JSON object with exactly `version: 1` and `keys`. Each key has
exactly these fields:
- `clientId`: 32 lowercase hexadecimal characters.
- `keyId`: 1–64 ASCII letters, digits, underscores or hyphens.
- `tokenSHA256`: SHA-256 of the **ASCII 64-character lowercase hexadecimal bearer
token**, stored as 64 lowercase hexadecimal characters. Do not hash the decoded
token bytes and do not store a plaintext token in this registration.
- `notBefore` and `notAfter`: UTC timestamps, `YYYY-MM-DDTHH:mm:ssZ`, in 2000–2099.
The start is inclusive and the end exclusive.
Use private runtime configuration outside Git. The caller sends
`Authorization: Bearer <token>`, `X-4open-Artifact-Client-Id` and
`X-4open-Artifact-Service-Key-Id`. Keys grant only capabilities reads. Generate a
separate token for this operation and environment; never reuse GitHub credentials,
webhook secrets or future exchange credentials. The Go review client requires
HTTPS with a trusted server certificate; a deployment must provide that transport.
The registry is bounded to 64 KiB, 64 keys, 32 clients and 16 keys per client.
Duplicate client/key pairs and reused digests are rejected. Unknown fields,
plaintext secret fields, invalid timestamps and empty configuration fail startup
with a generic diagnostic. Configuration uses ordinary JSON object parsing.
Credentials are copied into a private registry and compared by constant-time
SHA-256 digest equality. Overlapping validity windows permit rotation. Remove a
key and restart to revoke it; this is not hot reloading.
Authenticated clients share 60 reads per minute across their keys. Counters are
bounded by registered clients, reset lazily, and have no timer. Limits are local
to each server process; they are not a distributed quota. A 429 response supplies
a typed failure and a bounded retry delay. This endpoint does not accept browser
session or repository-owner authority.
## Verification
`mocha --no-config --no-package test/review-capabilities.test.js` passes eight
HTTP/configuration scenarios, including 60 concurrent reads, cross-client and
wrong-key rejection, raw duplicate headers, exact path checks, browser context,
expiry, overlapping keys and restart revocation. Add
`RUN_REVIEW_SERVICE_BENCHMARK=1` for a 150-request synthetic loopback measurement.
The clean-install run averaged 2.035 ms, median 1.732 ms and p95 2.417 ms with fresh
HTTP connections. These are local tooling measurements, not production latency.
A local HTTPS interoperability rehearsal uses the existing Go
`artifactclient.NewCapabilitiesReader` at review source `ee07d06`. It verifies
certificate trust, accepts the empty capability response, rejects both unsupported
access policies and rejects a wrong credential. No external provider is contacted.
A clean `npm ci` install, the full TypeScript check and emitted build, UI build,
and targeted ESLint pass. The full test suite reports 700 passing in 36 seconds
and 49 pending opt-in cases; those skipped cases are not claimed as tested.
The capability suite separately passes eight cases including its benchmark.
An initial check used dependencies copied from another checkout and produced
TypeScript diagnostics. Those installed versions differed from the lockfile;
reinstalling the locked dependencies resolved every diagnostic without source
changes. Use the lockfile when reproducing these results.
The endpoint is implemented and tested locally. Upstream deployment, private key
distribution, ownership consent and the remaining artifact workflows are pending.
A raw-socket regression first reproduced a rejected request waiting for its unread
10 MB body. Closing failed connections makes the same header-only request return
400 and close immediately; the complete eight-case endpoint suite and full
TypeScript checking pass. Full PR CI is repeated for this correction.
docs/shared-cache.md
+210
View File
@@ -0,0 +1,210 @@
# Shared blob cache
The optional shared filesystem cache stores immutable source content across projects.
It uses Node's built-in SQLite module and requires Node 22.15 or newer. All API and
streamer processes on one host must mount the same local cache directory. SQLite
WAL is not a distributed lock service: do not put this directory on NFS or share
it across hosts. S3 storage continues to use the existing storage implementation.
Enable the integrated implementation with `SHARED_CACHE_ENABLED=true`. The default
is false so a code deployment does not silently migrate production data. The other
settings are independent rollback controls:
- `SHARED_CACHE_PATH`: defaults to `.shared-cache-v1` inside `FOLDER`.
- `SHARED_CACHE_BYTES`: source pool budget, default 10 GiB.
- `SHARED_CACHE_RESULT_BYTES`: transformed output budget, default 512 MiB.
- `SHARED_CACHE_MAX_FILE_BYTES`: maximum source admission size, default 32 MiB.
- `SHARED_CACHE_FREE_BYTES`: free-device-space reserve, default 1 GiB.
- `SHARED_CACHE_LOW_WATER`: eviction target fraction, default 0.9.
- `SHARED_CACHE_GZIP`: store gzip when beneficial, default true.
- `SHARED_CACHE_GZIP_MIN_SAVING`: minimum fractional saving, default 0.10.
- `SHARED_CACHE_DECISIONS`: enable unchanged decisions and changed-output reuse,
default true.
- `CACHE_MONITORING_ENABLED`: collect cache observations, default true. Monitoring
is independent of shared storage and also records legacy cache/inspection events.
Logical user quotas remain based on repository file sizes. Shared-pool deduplication,
compression and eviction do not change user quota usage.
These are initial limits, not capacity recommendations from the production sample.
Restart processes together when changing pool limits. Reserve headroom for SQLite,
filesystem directories, existing caches and other application temporary files.
The budget accounts for blob allocation with a conservative 4 KiB minimum plus
in-flight reservations of three times the maximum object size and a small overhead.
A shared SQLite ledger reserves physical headroom across source and result pools
before producers write bytes. Its check uses a fresh free-space sample under a
cross-process lock and includes all outstanding reservations. Reservations remain
until cleanup or crash recovery completes. This is conservative while fills have
already written some bytes, and cannot reserve space against unrelated applications.
Keep each pool budget larger than this reservation. Admission can bypass the cache
while other fills/readers occupy the budget. Oversized source objects are streamed
without admission, still subject to the application's file-size limit.
Git blobs are verified against the requested Git SHA before publication. Resolved
LFS content is admitted only after verifying both the Git pointer and its SHA-256
payload digest and size. Gzip is a representation of the same uncompressed content.
Files are written privately, synced and atomically renamed before index publication.
Fills and readers have renewable leases. Eviction skips readers, and stale fills
are recovered before their reservations are released. LRU access timestamps are
coarsened to 30 seconds. Orphan reconciliation runs at most hourly across processes.
Alias and reuse-history indexes are bounded; decisions expire after seven days and
are capped at 100,000 entries.
Existing generation caches are read as a migration fallback and verified before
promotion. They are not deleted during promotion, so rollback remains possible.
The new pool budget does not include legacy copies: monitor total filesystem usage
and retire legacy data only after validating rollout. Uploaded ZIP originals and
other durable source material are untouched. Repository removal invalidates access
and project mappings, while shared blobs remain subject to global eviction.
Transient SQLite write contention bypasses caching instead of failing file requests.
Reader heartbeats retry contention within their last confirmed lease window and
terminate only if renewal cannot succeed before expiry or the lease is removed.
Admission-bypass streams retain the minimum expected-size check.
Publication installs a reader lease before releasing its transaction, including
duplicate blobs. Decisions retain sniffed text classification for consistent MIME
headers on repeat reads; binary responses omit the range-denial header.
Shared result production and delivery use the same repository lifecycle checks as
the legacy transformed cache. Client disconnects cancel the active transformer and
its worker signal, releasing spool reservations. Completed evictions are committed
even when pinned readers still prevent a new admission.
The HTTP fast path applies to individual file responses backed by the shared source
pool, including small text files. Decisions include content identity, path, project
context, all transformation options, generation/revision, mask, hostname and engine
version. Unchanged decisions serve the original representation. Changed output has
its own bounded pool. Gzip negotiation honors quality values, uses `Vary`, and
preserves MIME type. File validators are weak when shared delivery is enabled so
identity and gzip representations can revalidate the same logical content. HEAD
continues to use metadata only. Range requests receive a complete response; this
implementation does not add random access into gzip files. Website rendering keeps
its renderer cache; ZIP scans do not admit bulk entries into the source blob pool.
The admin overview's Cache section reads `/api/admin/cache?minutes=60` behind the
existing admin authorization. Windows are 15 minutes, one hour, 24 hours and seven
days. SQLite telemetry batches at most 512 groups every five seconds and retains
minute aggregates for 48 hours and hourly aggregates for 30 days. No paths, terms,
user IDs or blob hashes appear in metric labels. The UI distinguishes unavailable
telemetry, dropped events, disabled storage and empty windows. Rows separate file,
website, ZIP, background and known-monitor traffic. Monitor classification is an
operational heuristic, not a complete bot detector.
Hit counts and byte hit rates measure source lookups; misses classify first observed
keys, known misses and eviction refetches within bounded seven-day history. Reuse
intervals are consecutive blob accesses, not distinct people or browser visits.
Eviction time measures idle age; codec and inspection timings are elapsed work time,
not CPU samples. Cached decisions are separate from actual inspections. Minute and
hour boundaries make the displayed windows approximate.
Run the reproducible acceptance workload with:
```sh
mkdir -p benchmark-results/cache
node scripts/cache-workload.js benchmark-results/cache/workload.json
npx mocha test/shared-blob-cache.test.js test/shared-cache-http.test.js
```
The JSON report asserts cross-project source reuse, unchanged-decision reuse,
changed-output reuse, gzip delivery and budget enforcement. Its compression ratio
comes from synthetic text and must not be presented as a production projection.
Production rollout should compare request/byte hit rates, eviction refetches, p95
latency, upstream errors and free disk over a representative week, with large files,
ZIP work and known monitors visible separately.
### Benefits and costs
| Choice | Benefit | Cost or limit |
| --- | --- | --- |
| Global content identity | Reuses identical files across paths and projects; renames do not duplicate content | Requires verified hashes and authorization on every request; uses digest-prefix directories to avoid one huge directory |
| Byte budget with approximate LRU | Predictable blob footprint and automatic reclamation | Evicted files incur upstream fetches; active readers can temporarily prevent admission |
| Selective gzip | Reduces payload storage and can serve compressed bytes directly | First-fill compression costs work; identity consumers and uncached anonymization require decoding |
| Rule-aware decisions and results | Repeated unchanged reads skip regex inspection; changed output is reusable | More metadata and a separate output budget; settings changes produce new decisions |
| SQLite coordination and telemetry | Cross-process coordination without another service; admin can measure actual reuse and rewrite work | Local-filesystem scope; short synchronous transactions; metadata/headroom are outside blob budgets |
### Rollout sequence
1. Deploy with shared storage disabled and collect legacy hits, inspections and codec
timings. Keep known-monitor traffic separate from file readers.
2. Enable the shared cache on the single-host filesystem deployment with both byte
budgets configured. Keep legacy copies for rollback.
3. Compare a representative week of byte/request hits, rewrite decisions, eviction
refetches, request latency, upstream errors and whole-device free space. Adjust
admission size and budgets from those observations.
4. Retire legacy cache copies only after the rollback window. Uploaded originals
remain durable and are never candidates for shared-pool eviction.
The implementation and acceptance workload are local changes; this work does not
enable or migrate production storage.
### Production observations used for this design
The read-only audit on 2026-11-02 found 425,312 source files across 31,522 projects,
containing 144.32 GB of payload. Deduplication within projects could remove 1.87 GB;
global deduplication could remove 2.93 GB total (about 2.03%), including another
1.06 GB across projects. One file could not be read. These are decimal GB and
payload savings, not a compression estimate or the expected new working-set size.
The preceding 24-hour access sample contained 29,980 successful file-body responses,
including 9,249 repeat responses (30.9%). URL repetitions are not the same as blob
cache hits. A separate approximately two-hour metric window recorded 2,302 source
hits out of 3,600 lookups (63.9%). This supports retaining a bounded source cache;
it does not establish the optimal byte budget.
In the uniform anonymization replay, 937 resident files were measured: 764 text,
173 binary, and 21 with changed output. Thirteen resolved files exceeded the 4 MiB
sample limit, and missing files/mappings were excluded. The sample's mean gzip-6
work was 1.46 ms per file and mean decompression was 0.24 ms; these replay timings
are not live-request latency or a forecast for large files. Continuous telemetry is
needed to replace these samples with representative workload measurements.
### Metadata durability and accounting
Reader leases, access history, decisions and telemetry use NORMAL WAL commits.
Reservations, publication intent and final blob publication use FULL commits;
payloads are synced before rename. Recent disposable metadata may be lost after
system failure. Recovery retains reservations until orphan files are removed or
accounted for. SIGKILL recovery is tested; physical power failure is not simulated.
Admin statistics use transactional counters and an indexed oldest-access lookup.
History and decisions are capped at 100,000 rows each, aliases at 200,000. Expired
history/decision cleanup removes at most 256 rows per pass. Access history has
30-second resolution. Stop all cache processes before upgrading an existing index
and restart them on the same version; do not mix old and new writers.
`scripts/deploy.sh` coordinates restarts when existing containers have shared caching
enabled. For first activation or an explicit budget/configuration change, run
`COORDINATED_RESTART=1 bash scripts/deploy.sh`. It stops and verifies every old
application writer before starting replacements. This briefly interrupts service.
### Performance and validation
Local synthetic measurements on Node 24.14.1, dated 2026-11-02, compared the initial
FULL-commit implementation with optimized bookkeeping. Warm serial changed-text
medians fell from 44.9 to 7.3 ms at 16 KiB, 45.9 to 9.0 ms at 256 KiB, and 60.5 to
19.0 ms at 2 MiB. At 100,000 synthetic blob rows, admin stats p95 fell from 78.7 to
0.50 ms and decision-write p95 from 45.5 to 1.11 ms.
The optimized runs verified 2,500 HTTP responses. The 4 MiB eviction probe peaked
at 3,997,696 allocated bytes. Its oversized sequential working set had no hits:
a bounded LRU cache does not guarantee useful reuse. Measurements used loopback
HTTP, warm OS caches and a shared host; they exclude production traffic, TLS,
authorization overhead and multi-process throughput. They are not capacity estimates.
On the PR checkout based on main at d0b217a, the production build, scoped lint and
199 cache, HTTP, admin access, HEAD/revalidation and Vue UI tests passed. Coverage
includes process crash recovery, concurrent fills, active readers, LFS verification,
and public submodule origins without parent credentials. Admin layout was also
checked at 1280 px and 390 px during implementation.
Generate fresh reports locally; generated output is ignored by Git:
```sh
npx tsc
node scripts/cache-index-performance.js
CACHE_BENCH_COMPILED=1 node scripts/cache-performance.js
CACHE_BENCH_COMPILED=1 node scripts/cache-performance.js benchmark-results/cache/repeat.json --shared
```
Default reports go to `benchmark-results/cache/`. Scripts also accept an explicit
output path. Keep raw measurements with the evaluation that produced them rather
than committing generated reports to the documentation directory.
docs/stripe-sandbox-validation.md
+104
View File
@@ -0,0 +1,104 @@
# Stripe sandbox validation
Validated November 2, 2026 using a real Stripe sandbox, disposable local MongoDB, Stripe-hosted Checkout and the customer portal. The Stripe CLI forwarded real signed events to the application's raw webhook route.
## Results
The repeatable API suite passed 16 scenarios:
- Checkout creation, reuse, unpaid-access denial and session expiry for all four plans.
- Successful monthly and annual subscription payments and author entitlements.
- Duplicate subscription prevention for both recurring plans.
- Scheduled cancellation preserves the paid period.
- Immediate cancellation removes paid capacity; never-expiring repositories retain the recorded grace deadline and enter expiration after it.
- A declined initial payment grants no subscription access.
- Subscription refunds revoke access. Repeated signed events are idempotent; altered payloads fail signature verification; later subscription sync does not undo revocation.
- A Stripe test clock advances a monthly subscription through successful renewal, failed renewal and recovery. Only paid invoices advance paid-through time.
- Account deletion cancels its subscription.
Interactive checks also passed:
- A repository pass purchased through hosted Checkout activated through a real webhook.
- A conference purchase changed pending payment to ready through a real webhook.
- Monthly Checkout displayed an insufficient-funds decline without granting access, then activated after retry with a successful test card.
- Annual Checkout granted no access after failed 3-D Secure authentication and activated after successful authentication.
- Refunding the repository pass revoked its grant through a real webhook.
- The customer portal showed billing details and invoices and scheduled cancellation at the paid-period boundary.
- A disputed sandbox subscription payment triggered a real dispute webhook, revoked access and remained revoked after resync.
The local billing regression suite passed 23 tests, including quota concurrency, coauthor accounting, authorization/CSRF, grace periods and signature checks. TypeScript and targeted lint checks passed.
## Fixes found by real Stripe testing
1. Open subscription Checkout sessions do not yet have a subscription ID. Fulfillment now ignores unfinished sessions instead of reporting an invalid purchase.
2. The customer portal can schedule cancellation using cancel_at while cancel_at_period_end remains false. The app now recognizes either representation when displaying cancelled renewal.
## Repeating the API checks
Create a private JSON file outside the repository with a secret field containing a sandbox secret key. The script rejects live keys and verifies Stripe reports sandbox mode before creating fixtures.
Run against a disposable MongoDB database whose name begins with billing_test_:
```sh
STRIPE_SANDBOX_CREDENTIALS=/absolute/path/to/private-credentials.json \
BILLING_TEST_MONGODB_URI=mongodb://127.0.0.1:27029/billing_test_stripe \
node scripts/test-stripe-sandbox.js
```
The script creates tagged test customers, one product and four test prices. It deletes its customers and test clocks and deactivates its prices/product afterward. Stripe retains transaction history. It does not change existing prices or production settings. Test price amounts are fixtures, not a published price commitment.
## Limits of this validation
- This covers the implemented card-payment flows, not every Stripe payment method or account configuration.
- Automatic tax was disabled. Tax registration, live tax calculation, live settlement, seller details and production email delivery were not tested.
- The test clock changes Stripe time, not the application's wall clock. Failed-renewal tests verify paid-through does not advance; local tests exercise access expiry and the 90-day boundary.
- Hosted Checkout, portal and live CLI-forwarded webhook checks were interactive. The repeatable API suite uses real Stripe responses with locally generated webhook signatures for its HTTP replay test.
- No production deployment or live payment occurred. Sandbox customers and active subscriptions from this run were cleaned up; products and prices were archived.
## Reliability follow-up
The repeatable 16-scenario sandbox suite passed again after introducing durable webhook
receipt, asynchronous processing, checkout expiry, and provider reconciliation. The
HTTP refund replay check now explicitly drains the inbox before asserting entitlements.
No live payments or production deployment were performed.
The local billing suites now include crash recovery, concurrent event receipt, retry
exhaustion, privacy of stored envelopes, reconstruction after missing local grants,
missed renewal recovery, refund-before-fulfillment ordering, partial refunds, uncertain
checkout recovery, purchase shutdown, and expiration deferral during provider outages.
These are deterministic failure simulations against disposable MongoDB, not a claim
that production backup infrastructure or external alert delivery has been tested.
Final follow-up checks passed: 38 local billing tests, 155 frontend regression/UI tests,
TypeScript compilation, the production build, and targeted ESLint checks. The 16-scenario
real Stripe sandbox suite also passed with historical invoice recovery enabled.
## Full application UI check — November 2, 2026
Used the built application in the collaborative browser, production billing routes,
an isolated MongoDB database and real Stripe sandbox Checkout and customer portal.
GitHub login was replaced with a test-only session fixture outside the repository.
| UI flow | Observed result |
| --- | --- |
| Free account | Five repositories and 2 GiB displayed. |
| Monthly subscription | Insufficient-funds decline granted no access; successful retry activated 20 repositories and 10 GiB. |
| Duplicate subscription | App rejected another subscription and displayed the explanation. |
| Customer portal | Billing details and invoices loaded; cancellation stopped renewal while preserving paid access. |
| Annual subscription | Failed 3-D Secure granted no access; successful authentication activated annual access. |
| Delayed webhook processing | With the worker paused, the checkout return recovered annual access from Stripe. Both billing and quota panels reflected the upgrade. Queued events completed after resuming the worker. |
| Abandoned repository pass | Returning without payment granted no access. Reopening reused the same checkout session. |
| Paid repository pass | Successful checkout created a repository-scoped grant. |
| Refunded pass | A sandbox API refund generated real webhooks; refreshing in the UI showed revoked access. Refund initiation was an operator action, not an app UI feature. |
| Conference package | Checkout from the pending conference activated the package; the conference page showed Ready. |
Two UI issues were fixed during this run: checkout reconciliation now refreshes the
quota panel, and revoked purchases no longer display a future paid-access date.
The quota issue has a regression test. Four targeted UI tests and the UI build passed;
both fixes were also checked in the real browser. No failed or pending received
webhook events remained at the final pre-cleanup inspection.
This run does not cover GitHub OAuth, repository ingestion, production deployment,
tax, email delivery, or the passage of the 90-day grace period. Those should not be
inferred from the payment UI results. Temporary customers/subscriptions were deleted,
prices and product archived, and the disposable local services removed after testing.
docs/website-request-budgets.md
+31
View File
@@ -0,0 +1,31 @@
# Website requests and folder counts
Website assets under `/w/` have a separate Redis-backed request budget. The
default `WEBSITE_RATE_LIMIT=2000` allows 2,000 requests per visitor in 15 minutes,
including signed-out visitors. Requests across repositories share this budget.
They do not consume the `/api` or `/github` allowance. Website slowdown, access
checks, file-size limits and streaming protections still apply. Rejections
include `Retry-After` and the existing JSON `rate_limited` response.
Visitor identity uses the existing trusted-proxy address resolver. Do not trust
arbitrary forwarding headers or change `TRUST_PROXY` without verifying the
actual proxy chain. API and maintenance-token rate policies are unchanged.
The explorer requests `/api/repo/:repoId/files/counts?path=…` after loading each
directory, once per directory per viewer initialization. The path is the same
anonymized path used for browsing. A response contains recursive file counts for
that directory and its immediate children, rather than a repository-wide map.
Counts use the captured active tree generation and sum anonymized child-name
collisions. Entering a colliding directory resolves the same original directory
as browsing it. No original paths are returned.
The aggregation has a two-second database deadline, allows disk spill, and
returns at most 10,000 child groups plus the parent total. It still scans files
in the requested subtree; root counts can scan the whole active tree. No
cross-request cache bypasses authorization or retains counts from old snapshots.
The legacy endpoint without `path` retains its existing bounded response.
Counts are optional. If a count request fails, the explorer continues browsing
without unavailable badges and stops issuing further count requests until the
viewer is reinitialized. It does not label the number of loaded tree entries as
the repository's total file count. File listing guards remain in place.