This directory implements the API, authentication, repository processing, file serving, storage adapters and background jobs in Go. The existing browser application remains JavaScript. Node is used to build browser assets, run differential tests and perform offline data migration; it is absent from the runtime image.

The root Docker deployment runs Go. The Node backend remains available through the explicit rollback configuration. Do not switch production based only on route registration or the synthetic benchmarks below. Live OAuth callbacks, installation changes, existing database upgrades, and browser flows need validation against an isolated staging copy.

Build and test

Use Go 1.27.3, Docker, and the repository's installed Node dependencies for differential checks.

cd backend
go build -o ../anonymous-server ./cmd/server
go build -o ../go-contract ./cmd/contract
go build -o ../go-benchmark ./cmd/benchmark
go test ./...
./scripts/test-integration.sh
cd ..
GO_CONTRACT_BINARY="$PWD/go-contract" node backend/contracts/differential.cjs
GO_CONTRACT_BINARY="$PWD/go-contract" node backend/contracts/file-types.cjs
SESSION_SECRET=synthetic-test node backend/contracts/resources.cjs --check
node backend/contracts/routes.cjs --check
node backend/contracts/cache-metrics.cjs --check
node backend/contracts/cache-report-state.cjs --check
node backend/contracts/readme-decoding.cjs --check
node backend/contracts/compression.cjs --check
node backend/contracts/json-http.cjs --check
node backend/contracts/validation.cjs --check
node backend/contracts/options.cjs --check
node backend/contracts/dates.cjs --check
node backend/contracts/reactivation.cjs --check
node backend/contracts/oauth-tokens.cjs --check
node backend/contracts/oauth-profile.cjs --check
node backend/contracts/github-responses.cjs --check
node backend/contracts/github-quota.cjs --check
node backend/contracts/worker-queues.cjs --check
node backend/contracts/blob-index.cjs --check
node backend/contracts/content-verification.cjs --check
node backend/contracts/legacy-source-paths.cjs --check
node backend/contracts/shared-result-key.cjs --check
node backend/contracts/shared-http.cjs --check
SESSION_SECRET=synthetic-test node backend/contracts/file-metadata.cjs --check
node backend/contracts/download-headers.cjs --check
node backend/contracts/github-pagination.cjs --check
node backend/contracts/conferences.cjs --check
node backend/contracts/conference-output.cjs --check
node backend/contracts/conference-options.cjs --check
node backend/contracts/admin-filters.cjs --check
node backend/contracts/admin-pagination.cjs --check
node backend/contracts/admin-csv.cjs --check
node backend/contracts/user-membership.cjs --check
node backend/contracts/admin-user.cjs --check
node backend/contracts/user-defaults.cjs --check
node backend/contracts/quota.cjs --check
node backend/contracts/token-names.cjs --check
node backend/contracts/project-names.cjs --check
node backend/contracts/dashboard-cursors.cjs --check
node backend/contracts/dashboard-limits.cjs --check
node backend/contracts/dashboard-cursor-output.cjs --check
node backend/contracts/dashboard-filters.cjs --check
node backend/contracts/dashboard-pipeline.cjs --check
node backend/contracts/document-cleanup.cjs --check
node backend/contracts/webhook-http.cjs --check
GO_BENCHMARK_BINARY="$PWD/go-benchmark" node backend/contracts/benchmark.cjs

The contract scripts accept paths/environment documented in their source; they compare the current TypeScript implementation with the Go contract executable. Integration tests use disposable Docker services, random database names and synthetic credentials. Plain go test skips database integration unless its test environment variables are provided. scripts/test-integration.sh supplies them and runs the race detector.

Build the application image from the repository root:

docker build -t anonymous-github-go:local .

Exclude local database dumps and unrelated large directories from the build context. The runtime runs as UID/GID 10001 and needs writable repository storage and temporary space. Both Compose configurations add a read-only root filesystem, dropped capabilities, process/memory limits and a health check.

The separate .github/workflows/go-backend.yml workflow runs formatting, static checks, integration/race tests, output comparisons, vulnerability scanning and the image build.

Configuration

The backend reads environment variables, not .env files itself. Compose loads the environment file. Preserve the existing SESSION_SECRET, CREDENTIAL_KEYS, and CREDENTIAL_ACTIVE_KEY_ID to retain sessions and encrypted GitHub grants. It supports the existing MongoDB, Redis, OAuth, GitHub App and S3 variable names in internal/server/config.go.

For local execution, set PUBLIC_DIR to the root public directory and FOLDER to repository storage. Set MONGODB_URI explicitly when testing. MONGODB_DATABASE defaults to production; a database path in the URI takes precedence. Local HTTP testing requires COOKIE_SECURE=false; production behind HTTPS requires true.

TRUST_PROXY supports hop counts, IPs/CIDRs, and the existing named ranges. Configure it for the actual proxy topology; forwarded headers from untrusted peers are ignored. GitHub App private keys can come from GITHUB_APP_PRIVATE_KEY_FILE or the existing environment variable.

Anonymization defaults to one child process, a queue of 16, and a 512 MiB weighted admission budget. ANONYMIZATION_WORKERS permits up to four workers; ANONYMIZATION_MEMORY_MB controls admission. A child receives no server credentials. Its 384 MiB Go heap target is a soft limit; the container memory limit is the hard boundary. Input is capped at 100 MiB, output at 400 MiB, and execution at 15 seconds. Large concurrent downloads also consume temporary storage.

Verified behavior

Implementation verification is complete under the documented policy of preserving supported functionality without reproducing TypeScript bugs. The final full backend race suite, 64 CI contract commands, static checks and isolated deployment smoke test passed. Beta is healthy. Production-data cutover and shipping additional provider adapters remain separate operational work. See reports/implementation-completion-audit.json for requirement-by-requirement evidence and limits.

Reports in reports/ record the comparison and benchmark results. reports/compatibility-audit.json records the 64-command CI contract run, race/integration results, static checks, source fingerprints and rollback-image checks. reports/implementation-completion-audit.json maps the implementation requirements to their verification evidence. Its limits distinguish automated evidence from pending live staging validation:

499 exact differential cases cover text, paths, invalid UTF-8 bytes, regex behavior, self-links and cross-language encrypted credentials.
7,876 file classification cases compare the installed Node MIME database and binary heuristic with Go.
Integration and race tests exercise creation, tree publication, anonymized file output, access control, scoped and expired tokens, banned sessions, App permission restrictions and revocation during token issuance.
A hardened non-root container passed HTTP smoke checks and browser rendering of a Markdown script-injection fixture.
Tests reject truncated upstream content, unsafe raw-content redirects, filesystem traversal and symlink escapes. S3 protocol tests cover atomic publication and deletion boundaries.
All 112 mounted TypeScript API routes are extracted from the router AST into contracts/routes.json. CI regenerates the inventory in check mode and verifies the first matching Go route template, so a generic route cannot conceal a missing specific endpoint. Healthcheck and webhook dispatch are exercised through HTTP. This checks dispatch, not every status code and JSON response. /api/admin/cache reports Go cache events with the TypeScript time windows, latency buckets and row schema. Fifteen contracts execute the TypeScript report to check window coercion and bucket definitions.
govulncheck found zero reachable vulnerabilities and zero vulnerabilities in imported packages. It reported four advisories in required modules outside the packages used by this application; see reports/govulncheck.txt.

GitHub HTTP responses are mocked in automated integration tests. S3 tests use an HTTP protocol fixture rather than a production provider. Those tests do not establish compatibility with every provider configuration.

Request performance monitoring

Go now collects request duration, first-byte latency, encoded response bytes, interruption, 5xx and slow-request counts. Labels are registered route templates or fixed fallbacks; raw repository IDs, filenames and query strings are excluded. A 512-group buffer flushes every 15 seconds into the existing performance:v1 Redis minute-hash format with two-hour retention. Reports aggregate API and retained streamer history over 15 or 60 minutes, including histogram upper bounds and first-byte counts. Uncertain failed writes are dropped rather than replayed, and Redis outages report monitoring unavailable. HTTP response streaming and compression remain unchanged.

The focused race/integration command is GO_BINARY=/path/to/go backend/scripts/test-integration.sh -run 'TestRequestMetric|TestBackendIntegration/request_performance_history'. It checks histogram bounds, encoded bytes, HEAD, cancellation, route-label privacy, bounded buffering, Redis failure handling, cross-instance history and time-window selection. Go process samples are collected at most once every 15 seconds, with process CPU deltas, RSS/heap, cgroup memory limits, active requests, worker activity and process-owned TCP socket states. Redis retains live instances for 90 seconds and a bounded runtime series for two hours; reports exclude stale instances. Go event-loop measurements are explicitly unavailable, and the UI handles nullable values. Every HTTP response has a generated UUID X-Request-ID; incoming API request IDs are ignored. Structured completion logs use that ID, fixed route labels, status, encoded bytes and accumulated work-stage durations. Slow and interrupted requests log at warning level; successful health checks are quiet. Raw paths, query strings, authorization headers and panic details are excluded. Legacy source/transformed cache hits and misses also enter the performance report, independently of detailed cache-monitoring configuration.

Request-scoped stages measure repository lookup, source authorization, provider body reads, shared-cache lookup/fill, worker admission waits and worker execution. Cache hits do not count as worker executions or cache fills. Slot and memory admission waits are separate samples. Labels are fixed and provider-independent; background jobs do not enter HTTP stage metrics. Go has no separate streamer-header stage. These metrics are deployed to beta. The full race-enabled backend suite, 160 UI tests, live OAuth/App import checks and browser metrics checks passed; see reports/compatibility-audit.json and reports/shadow-performance-browser-validation.json.

Admin activity history

The admin overview always reads the last 30 UTC days and computes fresh current totals, independently of the public statistics cache and days query parameter. Its optional history and error-summary reads preserve the overview when those telemetry reads fail. Public and admin history use the same UTC-day merge: replace the first row belonging to today, retain other rows in stable date order, and never mutate cached totals or source rows. Twenty fixtures extracted from TypeScript cover non-midnight snapshots, duplicates, leap days and timezone boundaries. An isolated integration test seeds a stale public cache and verifies fresh admin totals, the fixed window, and malformed optional error telemetry.

Public statistics caches

Current totals use a one-entry local cache and the shared TypeScript Redis key perf:home-stats:v1. A renewable, ownership-checked Redis lease coalesces fills across instances. Redis read or write failures fall back to MongoDB results; cache I/O has bounded timeouts. History has a 16-entry FIFO cache keyed by UTC day and normalized window. Both local caches coalesce concurrent fills, retain successful results for one hour, and retry failures immediately. Returned maps are copied so callers cannot mutate cached results.

Twenty-eight extracted TypeScript contracts cover plain and repeated days parameters, integer prefixes, zero, negatives, hexadecimal values and the 365-day cap. Isolated integration tests cover shared-cache reuse, malformed data, a competing lease, denied writes, unavailable Redis and UTC rollover. Public history also uses bounded Express-compatible query normalization. Its 1,274 differential fixtures cover repeated and indexed arrays, nested objects, mixed values, malformed escapes, prototype-like names and parser limits. Other query consumers still require an audit before claiming application-wide query parity. History returns every snapshot in the selected date range; it does not cap the result at 366 rows. An integration fixture verifies 400 distinct timestamps on one day plus current and future snapshots.

Queue metric charts

Queue history accepts the TypeScript ranges 1h, 6h, 24h and 7d, and returns the selected string in range. Unrecognized strings retain that response value and use the one-hour window. Inherited range-object names such as toString preserve TypeScript's empty-series behavior. Reports contain one point per minute, including zero-valued empty minutes. Successful reads are cached for 30 seconds across at most 12 queue/window combinations; Redis read failures return an empty chart and are not cached. Worker durations round to the nearest millisecond and clamp negatives to zero. Minute hashes retain the TypeScript seven-day-plus-one-hour TTL.

contracts/queue-metrics.cjs --check executes the actual TypeScript recorder and reader against deterministic Redis fixtures. Seventeen contracts and Redis integration tests cover the windows, output shape, rounding, cache expiry, queue isolation, failure recovery and retention. check-shadow-auth.cjs --bootstrap-admin --app --monitoring-only verifies the live admin charts without importing a repository or requiring upstream GitHub requests.

Queue overview cards include 24-hour completed/failed totals from the same minute history. Go process samples advertise the actual running job loops for each queue; the overview counts fresh API instances and reports the first instance's concurrency, matching the TypeScript field meaning. Graceful shutdown removes the instance from live samples when Redis is available; unclean exits age out after 90 seconds. Paused loops still count as workers. Delayed jobs include their actual availableAt as delayUntil, and started/finished jobs expose processedOn/finishedOn timestamps.

Admin error monitoring

The Go error ring retains 5,000 entries and uses the TypeScript severity buckets, noisy-error exclusions, hourly level/module/code counters and 48-hour counter TTL. Routine 401/403/404 errors and the TypeScript allowlisted client errors contribute to counters without filling the recent-entry ring. Slow and interrupted HTTP requests contribute warning records with generated request IDs and fixed route labels. Unknown errors retain generic codes; raw upstream messages, credentials, paths and query strings are excluded.

A 128-entry in-memory buffer shares the 15-second performance flush lifecycle. Admin error reads flush pending entries before querying Redis. Overflow and failed batches increment the process-local dropped count; uncertain writes are not replayed. Clearing error history removes the ring, Redis dropped counter and hourly keys, preserving unrelated keys. As in TypeScript, the process-local dropped count is cumulative. Clearing refuses more than 10,000 hourly keys, and pagination offsets saturate at JavaScript's largest safe integer. LOG_LEVEL accepts debug/info/warn/error, with the same production/development defaults as TypeScript.

contracts/error-log.cjs --check verifies 40 fixtures extracted from the actual TypeScript classifier and error-list pagination expressions. Integration tests cover ring retention, malformed legacy entries, counters, TTL, cancellation warnings and clearing on isolated Redis infrastructure.

Provider adapters

internal/provider defines repository access, metadata, commits, trees, file readers, public URLs and self-link prefixes. Optional FileMetadata supports per-file recovery from truncated trees. Optional Discovery and Documents interfaces cover repository lists/branches/readmes and normalized snippets/pull requests. Discovery.List receives provider.ListRequest{OwnerID, Force} so each adapter can implement its own cache and refresh policy. A fixture adapter verifies that the HTTP route forwards refresh requests independently of GitHub. The shared lifecycle, MongoDB publication, anonymization and file/ZIP handlers use these interfaces. GitHub protocol calls live in the GitHub adapter; stored ZIP repositories use the archive adapter.

Register another adapter with server.New(config, store, adapter). Implement provider.Repositories and any optional capabilities you need. Its Branch method must resolve an actual branch, not a same-named tag, and return provider.ErrBranchNotFound only after establishing that the branch is absent. Authorization and temporary upstream failures must remain distinct. The registry rejects duplicate IDs and typed-nil adapters, and captures each adapter ID once. Lookups, ordering and advertised capabilities use that registered name even if an adapter later reports a different ID; extensions cannot replace built-in authentication by changing their ID during registration. Implementations live within the backend module because the interfaces are under internal/.

Create a repository with the existing POST /api/repo body and add source.provider, for example:

{"repoId":"example","fullName":"team/subgroup/project","source":{"provider":"your-forge","branch":"main","commit":"opaque-revision"},"terms":["Alice"],"options":{"expirationMode":"never","link":true}}

The adapter resolves immutable repository IDs and returns an opaque, server-owned access binding. Request bodies cannot supply credentials, bindings, upstream API URLs or immutable IDs. Third-party bindings are stored in providerAccess; GitHub retains githubAccess. Existing GitHub source types without a provider field keep working. Unknown providers and unsupported capabilities fail explicitly. Updating a resource cannot switch its provider. Discovery and save operations validate the adapter’s declared provider and binding; metadata must contain a name and immutable ID and must agree with the connected repository ID. HTTP tests inject mismatched adapter responses and verify that they create neither cached repository metadata nor anonymous resources.

Reconnect saves require the owner, the unchanged repository name and the preview's external repository ID. Presence of reconnectRepositoryId triggers validation even when its JSON value is null. GitHub previews use positive numeric gh_ IDs; other providers retain their own opaque IDs under their registered prefix. Invalid requests are rejected before provider calls. The returned repository identity must match the preview before branch or commit resolution and before cache writes. Missing branches return branch_not_specified; authorization, rate-limit and upstream failures retain their provider error category and reset time. Tests exercise the save handler's null case, preflight rejection, immutable-ID mismatches, wrapped branch failures and a successful opaque-ID reconnect.

Authenticated discovery endpoints are:

GET /api/providers: registered IDs and capabilities.
GET /api/providers/:provider/repositories: repositories accessible to the current user.
GET /api/providers/:provider/repository/metadata?name=team/subgroup/project.
GET /api/providers/:provider/repository/branches?name=team/subgroup/project.
GET /api/providers/:provider/repository/readme?name=team/subgroup/project&branch=main.
GET /api/providers/:provider/gist/:id and /api/providers/:provider/pr/:id?name=team/subgroup/project: source previews.

Source metadata, branch and README previews accept anonymizedRepoId. Shared routes verify local owner/coauthor/admin access and the requested provider/name before using the saved owner's binding. Provider authorization still runs on every preview. Metadata identity must match the saved immutable ID; branch and README lookups use the resolved name after a rename. reconnect=1 is owner-only, rejects archived resources and selects a new binding using the saved connection kind. It returns preview metadata without changing the anonymous resource. A non-GitHub integration fixture covers coauthor access, revoked grants, replacement identities, unchanged saved bindings and unauthorized requests making no provider calls. The legacy repositoryID query selects a cached metadata record. It supplies an optional ConnectRequest.RepositoryID from that record's provider-scoped external ID, never a credential or client-supplied immutable ID. Adapters must authorize the hinted identity; the shared route rejects a conflicting connection or metadata identity. A missing cache record falls back to name-based connection, and reconnect=1 ignores the hint. Saved-source identities cannot be overridden by a conflicting hint. If a legacy saved identity cannot be recovered, a hint is rejected rather than assigning a new identity to the owner's binding. Tests cover opaque IDs, GitHub OAuth rename resolution, App connection identity, malformed/cross-provider hints, revocation and explicit reconnect. GitHub preview metadata uses an optional RepositoryPreviews capability. Ordinary requests reuse a stored snapshot only after upstream access and identity checks. force=1, reconnect=1 and cache misses fetch current fields, including the Pages branch/path. Cached reads do not rewrite metadata and therefore cannot overwrite a concurrent refresh. Pages 403/404 responses preserve any existing page source; other failures prevent cache publication. Providers without this optional capability continue returning fresh metadata. The optional DiscoveryPreviews capability provides branch-list and README cache behavior. GitHub reuses nonempty branch lists, refreshes them when forced, and retains stored README text by branch name as TypeScript does. README reads select the requested branch, the metadata default branch, or master, and fetch by the selected commit. Missing branches and README failures return readme_not_available; empty README text is fetched again. Preview routes still verify current upstream repository access before serving cached discovery data. Cache revision checks prevent a late branch refresh from replacing newer data, and README writes match both branch name and commit. Integration tests cover normal and forced reads, branch output with cached README text, defaults, error statuses, access denial and concurrent cache changes. Providers without these optional cache capabilities keep their existing discovery behavior.

Existing GitHub routes remain available. Gist/PR creation also accepts source.provider and uses the adapter's ConnectDocument and Document methods. Normalize document payloads to the existing saved schema before returning them. An immutable repository ID returned in ConnectDocument’s Access.Ref.ID is persisted as a server-owned source field and passed back on edits, downloads and refreshes. Adapters must retain an existing ID; replacing or dropping it is rejected. Client-supplied immutable IDs are ignored. Providers without document support return provider_capability_not_supported.

Adapters can return or wrap provider.ErrNotFound, ErrAccessDenied, ErrRateLimited, ErrUpstream, ErrBranchNotFound and ErrCapability without importing the server package. These map to stable public codes and HTTP statuses; wrapper text and upstream response bodies are not exposed. provider.RateLimitError{ResetAt: instant} carries an optional reset time, serialized as Unix milliseconds. The same categories are retained in document and terminal preparation failures. Unknown errors remain generic server errors.

Adapters must authorize access using the local owner and their saved binding, honor context cancellation, bound upstream responses, validate remote URLs/redirects and prevent credentials crossing hosts. Providers without ContentVerifier use Entry.Size as a minimum downloaded size in both shared-cache fills and admission fallback. Providers with a verifier own size interpretation, including pointer-versus-payload differences; their Open method must detect truncated upstream responses even when cache verification is bypassed. SelfLinks contains trusted literal repository URL prefixes, most specific first. An empty slice disables source-link rewriting. File paths still pass shared traversal validation; file processing retains size, output and worker limits. No new provider credentials or endpoints are accepted from public requests.

The test provider uses nested namespaces and opaque revisions. Integration tests exercise repository creation, preparation, transformed files, ZIPs, unchanged-commit renames, refresh, removal, document creation and previews, denied private reads, revocation, unsupported capabilities and attempted provider switching. It makes no GitHub requests. This is an extension contract test, not a shipped GitLab/Bitbucket integration. Login, OAuth, installation webhooks, claims and GitHub coauthor lookup remain provider-specific; a new provider must supply its own credential connection flow.

TypeScript output contracts

contracts/resources.cjs runs the actual TypeScript classes and generates 1,048 fixtures: 1,024 public gist/PR combinations across all eight output flags, plus 24 private repository/gist/PR responses across lifecycle states. resource_contract_test.go compares Go output to those fixtures. CI regenerates them in memory with --check to catch TypeScript drift. To intentionally update the contract, run the script without --check, review the fixture changes, then run Go tests.

This fixed absent-field versus null handling, gist field filtering, empty arrays, coauthor fields, and sizes for non-ready repositories. File search omits SHA fields and rejects slash-containing queries, matching TypeScript. Folder counts roll up original paths before anonymizing and summing collisions. Rate-limited repository responses retain resetAt.

Public gist/PR options now refresh documents older than 24 hours when updates are enabled, and recover non-ready documents older than five minutes. Owners bypass these public refresh checks, matching TypeScript. Recovery preserves the existing gist 425 and PR 404 response for the triggering request; the next request sees the published result. Integration tests cover update opt-out, revoked access and removal during an upstream download. Revision checks prevent refresh from resurrecting removed content.

Repository public options recover stale non-ready states after five minutes, preserving queued jobs and inactive lifecycle states. Automatic source updates run from options requests rather than file reads. A short database claim prevents overlapping refreshes and is released after success or failure, so transient upstream failures do not suppress retries for a day. Tests cover fresh/stale states, duplicate views, failed-grant retries, changed commits and concurrent removal. ZIP HEAD requests return headers without provider access or transformation; disabling downloads blocks administrators as well.

GitHub branch resolution paginates branch lists and fetches commit metadata by the resolved immutable revision. Confirmed missing branches transition the repository to error with branch_not_found, detach its old tree and retire that generation. Authorization failures preserve the saved tree. Tree publication requires an unexpired staging lease; cleanup retires leases before deleting rows and discovers late inserts through staging markers. Legacy rows without generation IDs are retired too. Content cleanup honors TypeScript generation prefixes and retirement markers, removes obsolete legacy cache files, and preserves current generations and ZIP originals.

Manual repository refresh uses a five-minute lease, rejects active/waiting preparation jobs, removes terminal jobs before retry, and leaves ready repositories unchanged when the branch commit has not changed. Refresh writes require the same lease and repository revision. Changed revisions detach and retire old tree/cache generations before enqueueing replacement work. If reset or enqueue fails, only that refresh may mark its preparation as preparation_interrupted; concurrent removal and replacement leases remain protected. Integration tests inject a MongoDB enqueue rejection and verify that retry succeeds.

Expiration extension permits queued/downloading resources, matching TypeScript. Repository extension preserves the saved commit and queues reactivation only for expired repositories; gist/PR extension downloads synchronously. Gist/PR create, update and explicit refresh now return after publication, and create/update return the full private document. Settings edits retain the original source identity. Document deletion completes before returning removed; a repeated delete returns 410. An explicit owner-authorized document refresh can restore it, while concurrent lifecycle changes during a fetch remain revision-protected. Repository deletion remains queued, as in the current TypeScript route. Removal retries are allowed while removal or expiration is in progress; already removed repositories return is_removed with 410. Tests cover ten initial states, ownership, file/path cleanup, and retry after an injected enqueue failure. Both administrative cache deletion and cache jobs change the cache revision before deleting stored bytes. Successful file, ZIP and project-page content reads clear isReseted, refresh statusDate and clear the old status message, matching TypeScript. They leave size recomputation to tree preparation. Concurrent reads tolerate this bookkeeping update, while content publication checks retain lifecycle, tree, settings, access, anonymization and cache-revision guards. Tests cover failed access, concurrent transformed reads and a newer reset or removal during a read. Repository enqueue failures from save/reactivation use the same guarded preparation_interrupted recovery. Unchanged-source settings saves preserve queued/downloading/error states and existing preparation jobs without requiring upstream credentials. Expired or removed snapshots are reactivated only with a future expiration or never; otherwise their inactive state remains unchanged. The TypeScript reactivation helper supplies 60 contracts, exercised through HTTP with a revoked non-GitHub grant.

contracts/validation.cjs extracts the actual TypeScript validator declarations with the TypeScript parser. Its 90 fixtures cover required-field precedence, resource IDs, commit syntax and PR number formats. Go unit tests compare the validator results, and integration tests exercise rejected requests through HTTP. Edit authorization precedes body validation. An omitted provider ID on an existing resource retains that provider’s revision format; explicit provider switches are rejected. PR numeric strings such as 1.0 and 0x10 retain their numeric value. CI detects drift in both output and validation fixtures. contracts/options.cjs adds 126 option contracts using the actual route update functions and Mongoose schemas, checked through both Go unit tests and HTTP/database integration. Each resource keeps its own option fields; repository page sources retain only branch and path. Falsy expiration inputs are omitted. Truthy expiration inputs use JavaScript Date construction, including named months, overflow days, arrays, numeric milliseconds and year 0001. Shared resource access and list expiry treat year 0001 and the Unix epoch as stored dates for repositories, gists and pull requests. HTTP responses serialize native Go and MongoDB dates with JavaScript’s UTC millisecond format, including extended years and null for out-of-range dates. contracts/dates.cjs checks 213 native JavaScript date outputs. Option integration tests compare HTTP output as well as stored values. Date-looking strings remain unchanged. Typed provider maps, slices, arrays and ordered BSON documents also normalize native dates; byte slices and custom JSON encoders retain their wire representations. Boolean options match Mongoose’s explicit casts for null, booleans, 0/1 and the strings true, false, 0, 1, yes and no. Unsupported boolean inputs return invalid_options. Invalid resource options return 400 on both creates and edits, with no partial writes. This deliberately differs from TypeScript/Mongoose, whose creates can return 500 and whose edits can silently omit invalid fields. Valid expiration-mode and page-source strings retain Mongoose scalar/document-ID casts; empty page-source objects are consistently omitted. The isolated database differential tests compare valid persistence and independently enforce atomic rejection of invalid input. Numeric and boolean terms are cast to strings before storage, including JavaScript decimal/exponent formatting and negative-zero normalization. The option contracts also compare transformed text against the TypeScript anonymizer. Null terms, objects and nested arrays are rejected; TypeScript can persist null terms but then fails when compiling them. Go retains bounded request/term sizes; malformed types that trigger TypeScript runtime exceptions receive structured validation errors.

These fixtures cover the listed contracts, not every possible API response or live authentication flow. Production cutover still requires the staging checks below.

Performance results

Three alternating fresh Node and Go processes ran each fixture. The median measures transformation only, excluding startup, compilation, network, database and child-process overhead. Output SHA-256 hashes agree.

2 MB numeric input, 47 absent terms: Node 7.44 ms, Go 2.25 ms; Go 3.31 times faster.
100 MB numeric input, 47 absent terms: Node 234.29 ms, Go 117.23 ms; Go 2.00 times faster.
50 MB with sparse replacements: Node 509.48 ms, Go 593.56 ms; Go 1.17 times slower.
22 MB with 1.2 million replacements: Node 373.11 ms, Go 503.78 ms; Go 1.35 times slower.

These results do not support an overall speedup claim. The archive HTTP benchmark below measures one full-server workload. Shared-cache serving, external provider latency, cold start and GitHub rate-limit usage still need separate workload measurements. Raw repository content is cached through the shared provider layer on filesystem or S3 storage. Cache identities include provider, immutable repository ID, commit, blob revision, tree generation and cache revision. Cached reads still authorize access. Fills have a 120-second deadline, a 64-request process limit, Redis coordination, bounded atomic writes and lifecycle checks before and after publication. Retirement markers and admin cache revisions reject late fills. Tests cover repeated and concurrent reads, revocation, truncated cache entries, short downloads, revision changes and removal during a fill. ZIP originals bypass this cache. Go uses its own cache-key namespace, so the first read after cutover can still fetch upstream. Redis caches anonymous path indexes, and installation tokens are cached within their expiry and local revocation revision.

Full-server archive HTTP benchmark

Run GO_BINARY=/path/to/go backend/scripts/benchmark-http.sh --output=backend/reports/http-archive-benchmark.json from the repository root. Add --quick for a smoke run. The script builds the Go executable, transpiles the current TypeScript to JavaScript, starts disposable MongoDB/Redis containers and launches the actual server entrypoints. Each process receives a new database, storage directory and synthetic credential keys. Local environment files and production credentials are not inherited. The temporary Node layout includes the normal queue processors and required emoji module.

Four archive fixtures cover changed text, absent terms in a 2 MiB file, JavaScript word boundaries and 2 MiB binary content. Every successful response must match its expected SHA-256. Each runtime uses one anonymization worker and a 512 MiB admission budget. Three fresh-process rounds alternate runtime order. Each round measures seven distinct cold repository lookups, warms one repository, then sends 100 requests at concurrency 1 and 100 at concurrency 8. Cold means a new repository/cache identity, not a cold operating-system page cache. HTTP uses loopback keep-alive connections and identity encoding. Distinct forwarded client IPs prevent per-client slowdown from dominating the workload; both servers explicitly trust only loopback proxy traffic.

The report contains individual latency samples, throughput, p95 time to first byte, approximate process-tree CPU and RSS sampled every 10 ms. CPU accounting includes waited-for worker processes and waits for post-response page-view writes to drain. RSS includes live child workers discovered across all OS threads; a short-lived peak between samples can be missed. MongoDB, Redis and load-generator resources are excluded. Runtime versions, source/binary/script hashes, CPU details and cgroup limits identify the run. These are complete archive file-route measurements on a development host. They do not measure shared-cache gzip, external provider authorization/network latency, S3, startup time or production capacity.

The recorded run in reports/http-archive-benchmark.json checked 4,410 successful file responses across 21 successful fixture runs. At concurrency 8, median throughput across three rounds was:

64 KiB changed text: Node 49.65 requests/s, Go 99.83 requests/s, a 2.01× increase. Median per-round p95 fell from 209.84 to 88.19 ms.
128 KiB JavaScript-boundary regex: Node 12.19 requests/s, Go 80.52 requests/s, a 6.61× increase. Median per-round p95 fell from 952.03 to 104.97 ms.
2 MiB binary: Node 37.19 requests/s, Go 49.07 requests/s, a 1.32× increase. Median per-round p95 fell from 305.57 to 282.82 ms.

These ratios apply only to this archive workload and these implementations. The report also retains cold samples, concurrency-1 results, CPU and sampled process-tree RSS. Run node backend/scripts/render-http-benchmark.cjs to regenerate the interactive reports/http-archive-benchmark.html comparison from the JSON. No production capacity estimate follows from this development-host run.

The current TypeScript baseline fails the large-text fixture with HTTP 500: its Mongoose term array cannot be cloned into the anonymization worker. The benchmark records this known DataCloneError as a failed baseline and reports no speedup for it. Other errors or any output mismatch fail the run. A minimal reproduction using the actual repository model confirms that structuredClone({terms: model.options.terms}) throws while cloning a copied plain array succeeds. The benchmark does not patch that baseline.

Regex optimization

A separate before/after run of the Go implementation measured 920 to 600 ms for sparse replacements and 876 to 454 ms for dense replacements. Allocation volume stayed approximately unchanged on these fixtures. The absent-term fixtures improved by 1.41 to 2.11 times. See reports/regex-optimization.json for samples and source hashes. Differences between benchmark runs reflect timing variability; compare versions within the same run.

The implementation uses a byte lookup table for ASCII letter detection, avoids redundant classification during case folding, and translates fallback rune positions incrementally. The 2.2 MB Unicode fallback fixtures allocated about 16.6 MB per operation instead of 59.8 MB, a 72% reduction. This measures allocation volume, not peak memory.

Expanded tests also found and fixed five pre-existing output mismatches caused by the fallback library's Unicode word boundaries. Explicit JavaScript-compatible assertions restore the intended output. This correction has a cost: the boundary-bearing fallback fixture took 442 ms versus 385 ms before; the lookaround-only fixture improved from 338 to 262 ms. Regex timeouts, output limits and rule ordering remain in force.

To compare builds, run backend/contracts/regex-benchmark.cjs with GO_BASELINE_BINARY and GO_BENCHMARK_BINARY pointing to their respective cmd/benchmark executables. The script checks output hashes before reporting results. BenchmarkDenseReplacements, BenchmarkFallbackOffsets and BenchmarkFallbackLookaround provide Go profiling targets.

App token responses are validated consistently at callback, grant persistence and refresh. contracts/oauth-tokens.cjs extracts the actual TypeScript exchange function and supplies 33 cases covering token types, lifetimes and upstream statuses. Fractional lifetimes are preserved; values that would overflow Go durations are rejected. Token responses must be complete JSON within 64 KiB. Grant replacement clears stale refresh locks and absent expiry/refresh fields, and invalidates cached repository discovery. Integration tests verify encrypted replacement and that invalid responses leave the existing credential revision unchanged. HTTP callback tests cover new and existing accounts, signed-session rotation, replay, cross-session state, expiry, cancellation, wrong callback type and changed authenticated/upstream identities. GitHub numeric IDs retain JavaScript’s decimal string format rather than scientific notation for ordinary large IDs. Username reuse alone cannot link a different upstream identity. Refresh tests cover concurrent callers sharing one token rotation, revocation and newer login winning over an in-flight refresh, and cancellation releasing only its own lock. GitHub API requests and token exchange have a 20-second deadline. contracts/github-responses.cjs checks 44 status/header combinations against the actual TypeScript client, including an empty Retry-After header. Malformed, trailing and oversized API JSON responses are rejected with stable errors; response bodies are closed without exposing their contents. Refresh-lock cleanup uses a separate five-second context so request cancellation does not strand the lock. These tests use synthetic tokens and a mocked upstream; live OAuth/App callback validation remains a staging requirement.

Signed GitHub webhooks accept bodies up to 2 MiB. Installation IDs must be JSON numbers within JavaScript's safe-integer range; numeric strings, fractions and unsafe integers cannot change installation state. Authorization-revocation events ignore missing or falsy sender IDs and reconcile every matching account, without the former 100-account cap. Isolated database tests cover signature tampering, foreign App IDs, deletion blocking, the body-size boundary and replay across 101 working grants.

Installation reconciliation shares one implementation between webhook delivery and background retries. Successful reconciliation updates account ID, login and type, applies JavaScript truthiness to suspension state, records the check time and replaces the pending revision. The update requires the original revision, so replayed responses and responses arriving after a newer deletion cannot restore access or overwrite metadata. Database tests cover these races and verify that upstream failures, foreign App responses and missing account objects leave the installation blocked and pending.

App-grant reconciliation replaces the revision when revoking a grant and never clears a concurrent revocation. When reconciliation refreshes a token, successful verification clears the pending flag on the revision carrying that exact token; it does not leave the refreshed grant inaccessible until the next retry. Database tests cover working and rejected tokens, refresh success/failure, upstream outages, a newer login during verification and concurrent revocation with or without a revision change.

contracts/webhook-http.cjs executes the actual TypeScript webhook router through Express and records 78 HTTP contracts. Go tests compare statuses and JSON errors for malformed and primitive JSON, event-specific null handling, content types, disabled Apps, invalid signatures and the exact 2 MiB boundary. Oversized bodies return 413. Parser-generated HTML error pages are not reproduced; Go returns structured JSON errors. Compressed webhook bodies support gzip, deflate, Brotli and Zstandard. Signature verification uses decompressed bytes, with a 2 MiB output limit. Tests include corrupt/truncated streams, compressed-byte signatures, unsupported encodings and exact decoded-size boundaries. Go additionally limits wire input to 8 MiB and Zstandard decoder memory/window to 32 MiB. Brotli uses the pure-Go github.com/andybalholm/brotli decoder; Zstandard uses the existing klauspost/compress dependency.

The GitHub connection summary selects GitHub credentials explicitly, so additional providers cannot hide those grants behind a credential-list limit. Repository and pull-request summaries include all matching records rather than stopping at 10,000, while other providers remain excluded from GitHub migration controls. OAuth connection state reads the stored encrypted token, matching TypeScript; malformed envelopes fail instead of appearing connected. App failures retain safe API error codes and suspension flags use JavaScript truthiness. Database tests cover twelve unrelated provider grants, 10,001 records of each resource type, foreign/removed/other-owner exclusions, upstream failures and malformed credentials.

Connection summaries project only display fields and stream resource rows into the response, avoiding a second slice of complete database documents. Credential-state reads select provider/revoked fields; stored-token reads select only the encrypted access token. A MongoDB command monitor checks the actual replies during the 20,002-resource test and verifies that large synthetic payloads, duplicated source tokens and private binding metadata are absent. This reduces unnecessary data transfer and decoding; it is not an end-to-end throughput benchmark.

GitHub connection migration resolves existing App bindings by immutable repository ID, verifies commits/PRs at the resolved name, and saves renamed sources. Shared repository names/URLs update only when their cached external ID matches. Preview requests leave both resource and cache records unchanged. Stale PR downloads become eligible after five minutes; fresh downloads remain busy. Migration compares the original BSON source and full binding, preserving embedded-field order and rejecting concurrent binding changes even without a revision bump. Integration tests cover public-read renames, replacement identities, previews, stale/fresh PRs, status-date races and strict ID types. Other providers remain excluded from this GitHub-specific operation.

App installation flows can suggest a private repository using an existing OAuth grant, falling back to the App user grant when OAuth is absent. Existing owner installations lead to organization/user settings; new installations receive suggested owner and repository IDs. Invisible repositories fall back to the generic installation page. Setup callbacks consume and persist removal of state before validation, reject repeated state/installation-ID query values, and clear discovery data only after a valid owned flow. Tests cover settings targets, suggestions, fallback credentials, wrong owners, expiry, replay, repeated parameters and navigation-only callbacks without state. Upstream installation responses remain mocked; live browser/App setup is a staging gate.

App OAuth flows persist installation intent and the validated repository suggestion in their session-bound state. Successful callbacks redirect to installation using that saved intent; callback parameters cannot replace it. App login defaults to /connections for authenticated users and /dashboard for anonymous users. Repeated state/code parameters are rejected before token exchange, and disabling new App connections blocks their callbacks. Database/Redis integration tests cover these round trips, default/unsafe return destinations, repeated query values and consumed state on rejected callbacks.

App OAuth profile validation has 28 contracts extracted from the actual TypeScript callback predicate. GitHub profile IDs must be JSON safe integers, and login values must pass the same truthiness check before account lookup. Numeric strings, fractional/unsafe IDs and falsy logins return github_app_reconnect_required (403). Callback integration tests verify that each rejected profile leaves the user collection and grants unchanged, does not authenticate the session, and consumes its OAuth state, including a numeric-string ID matching an existing account.

Repository connection selection uses TypeScript's stored OAuth-token semantics and returns github_oauth_required when that grant is missing. Public App repository IDs must be JSON safe integers before binding or reauthorization; strings and fractional IDs cannot pass through numeric coercion. Persisted bindings accept BSON integer representations but reject malformed IDs, and public-read bindings reject any installation-ID field, including explicit null. Integration tests cover missing/corrupt OAuth grants, stored-token selection, safe-integer boundaries, malformed upstream/stored identities and changed identities during public reauthorization.

Installation-token access retries pending installation reconciliation before checking blocked state, matching TypeScript's recovery path. Minted tokens require a nonempty string token and a parseable expiry; invalid responses are neither returned nor cached. The cache resets at 1,000 entries and reuses tokens only when more than a minute remains. Integration tests verify retry-on-access, durable state after upstream failure, per-read user authorization on cache hits, installation blocks, scoped permissions, invalid issuance responses and cache capacity/expiry. Non-string token/expiry fields are rejected explicitly rather than coerced.

Conference listings and member views no longer silently truncate at 10,000 records, and removal processes members beyond the former 100,000-row limit. Capacity integration tests create 10,001 conferences and 100,001 members, checking complete unique output, expiry of the final conference, and cleanup of the last member despite partial failure from preceding archived members. These limits are separate from the explicit per-repository file-count bound.

Conference listings discover pending membership transitions in bounded batches of up to 500 IDs, projecting only transition endpoints. Only affected conferences run the existing guarded replay and reload path. A command-monitored 10,001-conference test requires 22 find commands when no transitions are pending. Additional checks replay a transition across batch boundaries, preserve original billing timestamps and leave unrelated pending events untouched. This is a database-query-count improvement, not a production latency benchmark.

Failed-job retries capture the complete selected snapshot using projected job metadata, prepare each eligible resource, and transition jobs individually with a failed-state guard. A resource or job write failure does not stop other retries; returned counts include only successful transitions. Jobs becoming active remain active, and failures arriving after the snapshot are left for a later retry. Integration tests place a valid preparation job beyond 100,000 earlier records, inject database validation failures, and verify resource state, active-job isolation, action isolation and snapshot boundaries. Unsupported preparation kinds remain failed rather than being reported as successfully retried.

Compatibility decisions and remaining validation

MongoDB document formats, AES-GCM envelopes, session signatures, Redis session keys, immutable GitHub identity bindings and maintenance-token scopes are retained. New jobs use a separate MongoDB gojobs collection. The offline handoff tool imports existing BullMQ jobs through its public API; Go runtime workers do not read BullMQ Redis internals. Staged tree publication uses revision checks, and maintenance retries cleanup and pending webhook reconciliation.

The following differences are deliberate or require staging review:

Plaintext legacy GitHub tokens are not read. Complete the existing credential migration before cutover.
An unbound legacy account is not linked automatically by matching its GitHub username. Existing authenticated owners need an administrator-verified identity recovery procedure. The old username-only linking behavior is unsafe.
Repository claims require upstream admin permission or existing local ownership; public read permission alone is insufficient.
Markdown project pages use the existing browser renderer in a sandbox without allow-same-origin. Rendering now requires JavaScript; a plaintext fallback is provided.
Go file ETags use a separate namespace. Existing caches will revalidate. Archive contents are generated afresh; compressed ZIP bytes and timestamps are not expected to match Node.
Regex terms are limited to 1,000, each at most 8,192 bytes. General regex matching rejects more than one million matches. API-specific term limits can be stricter. Oversized inputs fail rather than silently returning partial output.
Runtime/performance administration reflects the Go process and worker pool. It does not recreate Node process metrics or every historical per-route/per-stage series. The Linux overview supplies the existing browser fields for normalized one-minute CPU load, host memory, process RSS and root-filesystem usage. These preserve the TypeScript host-level meanings; they are not container quota utilization. Unsupported platform measurements are null. The overview also preserves rolling 24-hour active/new repository and new-user counts using the same timestamp fields as TypeScript, regardless of resource status. Integration fixtures cover recent, old and missing dates, including recently created removed repositories.
Repository membership changes carry a durable conferenceMembership event until both conference histories are updated. Drain these events before rollback to Node; Node does not replay them. Existing histories are preserved rather than inferred from repository status.
Existing repository rename/reconnect cases, legacy cache retirement, account recovery, conference billing, all admin filters, browser rendering and exact error/output parity are not exhaustively validated. These are production cutover gates, not covered by the route dispatch test.

Conference option persistence has 58 contracts from the actual TypeScript form helper and Mongoose schema. Boolean and string casts, null/empty dates, omitted or empty options, unknown fields and rejected types are checked through Go unit tests and HTTP/database tests. Non-owner responses retain the option representation and omit billing. Conference form validation has 121 contracts extracted from the TypeScript validator. Both creates and edits validate plans and required billing fields in the same order. Editing restores ready status and preserves the saved ID, owners, plan, billing and repository history. HTTP integration tests cover rejected edits without mutation, edits of ready/expired/removed conferences, creation and method-specific missing-conference errors. An additional 54 actual TypeScript/Mongoose contracts verify start/end Date construction and persistence. Legacy date strings, overflow dates, arrays, numeric milliseconds and coercion failures use the shared JavaScript date parser. Expiration options follow Mongoose casting, including numeric strings outside the year range and rejected booleans. Invalid stored dates/options return the TypeScript save error, 500 internal_error, without mutating the document. ISO date-only input is parsed as UTC. Listing conferences expires overdue ready conferences before returning, as in TypeScript. Expiry uses open membership records, overrides repository-level never expiration, and preserves already inactive statuses and timestamps. Integration tests cover ten repository states and closed or absent membership records. Conference removal waits for file, path-index and storage cleanup before returning. It attempts all open members, preserves archived repositories and closed memberships, and reports partial failure while keeping the conference removed. Tests cover owner authorization, synchronous cleanup, partial failure and repeated removal. History reconciliation preserves BSON document order in its compare-and-swap guard. contracts/conference-output.cjs adds 288 exact output cases from TypeScript, covering fractional rates, closed and open intervals, future additions/removals, conference-end caps, missing rates and long date ranges. Billing uses JavaScript millisecond arithmetic and operation order, including null for non-finite totals. Year 0001 is a valid conference end or membership removal date, not a missing-date sentinel. HTTP/database expiry and removal checks preserve memberships closed at year 0001 or the Unix epoch. Repository saves update membership history immediately for any provider. Joining opens an entry, switching closes the prior entry, leaving records a removal date, and rejoining resets the first existing entry as TypeScript does. An unchanged conference preserves its billing date and does not prevent settings edits after the conference expires. HTTP/database tests cover these transitions. A pending membership event is saved atomically with the repository revision. Failed history writes are replayed on later saves, conference reads or maintenance using the original timestamp. Strictly increasing save timestamps and event IDs protect newer transitions from stale replay. Fault-injection integration tests reject conference writes, recover both histories without shifting billing dates, and verify that delayed replay cannot undo a later move. Pending events have a sparse index.

Admin list filters have 111 contracts extracted from the TypeScript route handlers. They cover literal search escaping, owner/conference/date filters, status flags and query output shape. Unknown owners return no repositories. Conference status flags apply, and the repository preparing filter includes preparing/downloading entries. Storage totals appear only on repository responses. HTTP integration tests cover status selection, unknown owners, administrator access and credential exclusion. Another 138 TypeScript contracts cover pagination and sorting. Admin lists parse extended query values with JavaScript integer-prefix and Number rounding behavior, including signed decimal/hexadecimal values and arrays. Page numbers no longer have the former one-million cap. Limit retains TypeScript's upper bound of 1,000. Negative limits use MongoDB's single-batch find behavior for repository/conference lists and fail for the user aggregation. Negative skips fail instead of silently using a default. Sort/direction/CSV selection require scalar values as in TypeScript. The sort pattern is compiled once and rejects operator-prefixed fields and trailing newlines. HTTP tests verify ordered pages, bracketed values, large pages, invalid MongoDB pagination, administrator access and CSV bypass of pagination. Extended filters retain distinct find/count and aggregation semantics, covered below. CSV exports have 45 exact TypeScript output contracts covering empty/minimal rows, leading whitespace, quotes, commas, line breaks, Unicode, dates and numeric formatting. Go matches the empty repository header, filename quoting and lack of a trailing newline. Exports retain the 50,000-row limit and credential exclusion.

User repository membership has 12 contracts extracted from TypeScript. Owner, stable GitHub coauthor identity and legacy username entries with absent/null/empty IDs are shared by user lists, dashboard queries and admin user repository lists. Legacy fallback uses one matching coauthor entry and does not override a conflicting stable ID. User repository/gist/PR lists and admin user repository lists mark overdue ready rows as expiring, then reload affected records. Integration tests cover coauthor visibility, inactive states, a concurrent expiration extension and deletion.

Admin account mutations have HTTP/database coverage for missing users, non-admin denial, immediate promote/demote effects, ban/activation and removed-account activation. Banning attempts every eligible owned repository and continues after individual update/enqueue failures, reporting only successful queue submissions. Failed enqueues retain removing for retry; already removing/removed repositories and coauthor-only resources are excluded. Activation changes status without reconstructing erased identity data or tokens. A MongoDB queue validator injects failure to verify partial-success behavior and cleanup retry.

Admin user details populate cached repository references in their saved order, preserve duplicates and omit missing references. Eight contracts use the actual TypeScript serializer and Mongoose population/defaults. Responses include identity metadata, saved defaults, project names and token-use dates while excluding tokens and credential envelopes. Contracts cover null/default metadata; HTTP tests cover authorization, populated output, provider identity and non-mutating reads. Cached repository metadata now records its provider in source, preventing new non-GitHub records from receiving the GitHub display default.

Saved user defaults have 38 Mongoose update-cast contracts and four read contracts. They cover scalar/array terms, string and boolean casts, nulls, omitted defaults and unknown fields. The defaults endpoint retains its schema’s string-valued page, separately from repository option validation. HTTP tests compare stored values and subsequent reads and verify that rejected casts leave prior defaults unchanged. The signed-in profile omits an absent photo and retains an explicit null photo.

Quota accounting has 40 cache-validity contracts and an exact selection-filter contract from TypeScript. It counts owned, ready, non-expired repositories, uses valid positive-file or computed-empty caches, and aggregates uncached file sizes in one MongoDB query across current tree generations. Results are cached only while status, tree generation and file-metadata revision still match. Integration tests cover cached/uncached and empty sizes, expired/other-owner exclusions, legacy trees, repeat reads and stale-cache-write rejection.

GitHub user search uses TypeScript's two-UTF-16-unit query minimum. Regression tests cover ASCII, accented characters, CJK, combining characters and supplementary-plane characters, and verify that short queries return an empty array before credential lookup or upstream access.

Combined GitHub discovery preserves OAuth ordering and lets App metadata replace duplicate IDs without moving their positions. Any App discovery failure discards all partial App results and permits independently connected OAuth discovery; without OAuth, the failure is returned. This fallback applies only to discovery, never resource authorization. Database integration tests cover both connections separately and together, installation and repository failures, partial-result rejection, suspended/foreign installations and connection-specific credentials. Pagination limits return an error instead of silently returning an incomplete list. OAuth-only discovery reuses saved repository references unless the list is empty or force=1 is requested. It reuses existing metadata rows, returns fresh metadata on refresh, omits missing references, and preserves an existing list on upstream failure. A discovery revision prevents in-flight responses from undoing grant replacement, disconnect or account removal. Cache integration tests cover these transitions, including invalidation during a fetch. Existing metadata is loaded in one query; only new rows require inserts.

Coauthor management applies method-specific permissions: list access uses owner/coauthor/admin membership, adding requires owner/admin, and removal also permits the matched coauthor's stable identity or legacy username. Unauthorized mutations return 401; unauthorized reads return 403. HTTP tests cover inactive repository states, case-insensitive repository lookup, stable-ID and legacy self-removal, absent targets and persisted response output. Stable IDs prevent reused usernames from claiming another identity. Mutation guards preserve the original BSON array so embedded field order does not produce false concurrent-update conflicts.

Admin API-token names have 22 contracts extracted from the TypeScript route, including falsy defaults, array/object stringification, JavaScript whitespace, long names and rejected shadowed toString properties. Token lists reload current records so the response includes the request's last-used timestamp. Creation and listing return user_not_found if the account disappears after authentication. Integration tests verify hash-only storage, plaintext disclosure only on creation, admin-only access, immediate revocation, repeated deletion and malformed-ID error parity.

Private project names have 37 contracts extracted from TypeScript's validator. Inputs require strings, lengths use UTF-16 units, and trimming/control-character rules match JavaScript. Naming requires membership in the user's dashboard, including stable-ID repository coauthors; admin access alone does not grant membership. HTTP/database tests cover separate names for owner and coauthor, denial for unrelated admins and document coauthors, exact artifact-ID matching, list expiry, and clearing both the user map and dashboard-name record. Names remain outside shared artifact records and the handler does not depend on the source provider.

Dashboard-name reconciliation streams existing names and artifact-ID projections, with MongoDB writes in batches of 250. It has no collection-size cap and does not load artifact bodies for backfill. Integration tests reconcile 603 private names across batch boundaries, repair stale entries, backfill missing entries, repeat safely, recover an empty-map clear and preserve other users' names. Membership includes stable-ID repository coauthors and excludes unrelated artifacts. A process-local cache matches TypeScript's five-minute TTL after completion and 64-entry FIFO bound. Concurrent requests share in-flight work, failed calculations can be retried, and successful private-name writes invalidate the cache before the dashboard-name write. Tests cover TTL, capacity, concurrency, cancellation, panic cleanup, in-flight invalidation, cached database reads and refresh after invalidation.

Dashboard cursors have 1,968 contracts evaluated through the actual TypeScript pipeline builder. Non-date sort values must be strings or numbers; objects cannot enter MongoDB predicates. Sort and tie-break types are checked without coercion. The decoder accepts padded/standard base64 and whitespace as Node does. Date values normalize supported ISO strings, numeric milliseconds, null and booleans before querying. HTTP tests exercise the contracts and page through provider-independent repository records without duplicates. Dashboard date strings now use the shared ISO/legacy parser verified against the installed TypeScript pipeline; dashboard filters have 28 contracts evaluated through the TypeScript pipeline and BSON serializer. Search uses JavaScript whitespace and a 256-UTF-16-unit bound, including replacement of a split surrogate when encoded for MongoDB. Repeated query values follow TypeScript string/array semantics. HTTP tests cover Unicode search boundaries, literal regex characters, repeated filters and provider-independent repository data. Emitted cursors have 28 exact-byte contracts from the TypeScript encoding expression, including field order, HTML characters, literal backslashes, Unicode line separators, numeric values and millisecond dates with extended years. HTTP pagination compares emitted cursors with TypeScript-generated fixtures. Dashboard page limits have 53 contracts extracted from both TypeScript pagination expressions. Go matches flooring, clamping, defaults, radix and exponent syntax, repeated query values and JavaScript whitespace. HTTP tests verify each case's page size and continuation behavior.

The dashboard aggregation builder has 2,103 complete-pipeline contracts from the actual TypeScript function. They compare projections, owner membership, private-name lookup, expiry, attention, sort fallbacks, filters, cursor predicates, facets and page bounds together. Multi-key sort order is checked separately so JSON object comparison cannot hide an ordering change. Raw query fixtures also cover indexed arrays, nested objects, malformed escaping and string-conversion failures. The handler preserves JavaScript truthiness, strict scalar attention checks, native array type filters and invalid empty-array cursors. The HTTP handler uses this tested builder; database integration tests continue to exercise execution and pagination. These contracts cover their supplied query cases, not every possible legacy date string or malformed stored record.

Legacy user resource lists and the admin user-repository view return the complete membership set, matching TypeScript rather than silently stopping at 10,000 rows. Database integration tests create 10,001 records for each resource type and verify unique results, private-name output and expiry beyond the former boundary. The repository case includes a stable-ID coauthor record and checks the admin view as well. The separate dashboard endpoint retains its bounded cursor pagination.

Account removal scrubs duplicated legacy credentials from every owned artifact, including already-removed tombstones, without rewriting their removal timestamps. An isolated database integration test makes both upstream grant revocations fail and verifies mandatory local deletion of all provider credentials, private names and the session, plus profile anonymization and token removal. Other owners' records and shared organization installations remain unchanged. Owned content is removed synchronously in repository/PR/gist order before grant revocation and account anonymization. Archived repositories stop deletion with the account still active. Storage failures leave a durable removing state and preserve the account, grants and session for retry. A second isolated integration test injects a closed-storage failure, retries cleanup, verifies file/path-index and embedded-payload removal before revocation, and preserves coauthor-only repositories. These fixtures do not establish production storage or live upstream revocation compatibility.

Embedded-document cleanup has eight contracts executing the actual TypeScript resetSate methods. Gist removal/expiry clears files, comments, description and owner login while retaining visibility and dates. Pull-request cleanup clears body, title, diff, comments and base/head repository names, resets state/draft/merged flags and removes the merge date while retaining other metadata. Database tests apply each contract to both removal and expiry, including provider-specific metadata, and check repeated cleanup.

Grant revocation reads the stored encrypted token directly, including expired or revoked App grants, without attempting refresh. Missing grants still clear local connection state. Upstream deletion has a 20-second deadline; 2xx, 404 and 422 permit local cleanup, while other statuses and transport failures preserve the local grant. Integration tests cover both OAuth and App grants across these outcomes, response-body closure, discovery-cache invalidation and isolation of OAuth fields during App disconnection. Legacy plaintext grants must be migrated before cutover as described below.

Queue administration and workers share the same action groups, including expiry in the remove queue. Controls use opaque job IDs and retain each job's resource kind and provider-independent action. Missing jobs return 404 and active retries return 409. Single removal and forced queue emptying also remove active records, matching TypeScript; they do not cancel work already running. Worker lease checks prevent an old completion from updating a replacement queue record. Drain includes waiting and delayed jobs. Listings select up to 200 jobs per state, prioritize active work, and search both IDs and names. Integration tests cover these behaviors, document retries and cross-queue isolation. Queue storage remains separate from BullMQ and requires the cutover procedure below.

GitHub installation, repository, branch and document-comment pagination uses a shared cancellable visitor without the former 1,000-page ceiling. App endpoints stop on a short page or an access-selection match. OAuth repository lists, branches and comments follow the next Link exactly, including opaque cursors and short or empty intermediate pages. A full page without a next link is final. HTTP 409 produces an empty page for these Octokit-backed lists only. The existing per-response byte limit still applies. Failed list requests do not publish partial results. Unit tests exercise page 1,001 for bare arrays and both App response envelopes, exact page boundaries, cancellation, early matches and upstream errors. Database tests select repositories beyond that boundary by name and immutable ID, verify that installation revocation still wins, and reject late upstream failures without falling back to public access. Another 38 contracts execute the installed Octokit paginator and compare returned IDs and requested URLs, including whitespace and relation parsing. Pagination links must use HTTPS on api.github.com, without user information, custom ports, fragments or unsafe paths; tests reject credential redirection attempts before a second request. These pagination details remain inside the GitHub adapter; other providers implement their own discovery protocol.

Isolated shadow rehearsal

compose.shadow.yml runs a separate Go application, MongoDB, Redis and repository volumes on the development/backup server. It does not share production data, sessions, encrypted credentials, jobs or caches. The application and databases use an internal Docker network with no outbound access. A fixed-upstream Nginx gateway exposes only 127.0.0.1:5001. Database ports are not published. This is an isolated rehearsal, not automatic production traffic mirroring.

From the repository root:

node backend/scripts/init-shadow.cjs
docker compose --env-file .env.go-shadow -f backend/compose.shadow.yml up -d --build --wait
node backend/scripts/check-shadow.cjs

The initializer creates owner-readable shadow-only secrets and preserves existing values on subsequent runs. .env.go-shadow is Git-ignored. The check first verifies network, storage, credentials and port isolation, then seeds a synthetic shadow-demo archive and checks nine exact anonymized responses, a ZIP response and public route/authentication behavior. View it at http://127.0.0.1:5001/r/shadow-demo/. From another computer, forward a local port over SSH to this server, for example ssh -N -L 5001:127.0.0.1:5001 USER@SHADOW_SERVER.

Both GitHub login methods are disabled in the base stack. For the public beta deployment, compose.shadow-beta.yml enables legacy OAuth, secure cookies, the https://beta.durieux.me/github/auth callback and an outbound network for the Go application. MongoDB and Redis remain on the private network. Its gateway preserves HTTPS forwarding from Caddy, and Go trusts the two proxy hops. GitHub App login remains disabled.

The beta OAuth client ID/secret are stored as SHADOW_CLIENT_ID and SHADOW_CLIENT_SECRET in .env.go-shadow; they are separate from production credentials. Keep using the overlay when restarting this deployment:

docker compose --env-file .env.go-shadow -f backend/compose.shadow.yml -f backend/compose.shadow-beta.yml up -d --wait
node backend/scripts/check-shadow.cjs --beta

shadow-beta.caddy is installed at /etc/caddy/conf.d/anonymous-go-beta.caddy on this server. It routes beta to the loopback gateway and adds X-Anonymous-Deployment: go-shadow-beta so propagation can be checked without guessing from page appearance. The beta DNS previously reached the production origin; it was moved to this server for isolated testing. Production's application, callbacks and data are unchanged. The public demo is https://beta.durieux.me/r/shadow-demo/. The user completed legacy OAuth authorization. The resulting active account and live GitHub repository access have been checked. Production accounts and repositories are not imported into beta.

node backend/scripts/check-shadow-auth.cjs --bootstrap-admin is a mutating beta check, intended only after explicit authorization to promote the sole active GitHub beta account. It verifies the isolated containers and database before granting admin access and adding a hashed temporary API key. It checks authenticated endpoints, API key issuance and revocation, secure session cookies, cookie-only authentication and logout invalidation, live provider discovery, repository creation and worker completion, exact anonymized README and ZIP output, removal and denial of removed content. ZIP verification uses Python 3 standard-library CRC and archive checks. Keys remain in memory and are revoked during cleanup; the account remains an admin. The script refuses ambiguous account selection and records sanitized results in reports/shadow-authenticated-deployment.json. It does not test private repositories, GitHub App installation flows, production-data migration or production rollback.

The beta GitHub App overlay is compose.shadow-app.yml, applied after both shadow files. It enables App login alongside legacy OAuth, mounts secrets/github-app-beta.pem read-only and reads the SHADOW_GITHUB_APP_* settings from the ignored .env.go-shadow. The key is owned by runtime UID 10001 with mode 0400. Start this configuration with docker compose --env-file .env.go-shadow -f backend/compose.shadow.yml -f backend/compose.shadow-beta.yml -f backend/compose.shadow-app.yml up -d --wait. Use node backend/scripts/check-shadow.cjs --app for isolation, redirects and public output; add --app to the authenticated check for this deployment. These checks permit only the exact read-only beta key bind mount in addition to shadow volumes. Omitting the App overlay on a later up disables App login again.

App 5594414 (anonymous-github-beta) authenticated successfully with its signing key. Its initial installation list was empty. Configure the App callback as https://beta.durieux.me/github/app/callback and webhook URL as https://beta.durieux.me/github/app/webhook. The webhook secret is stored as SHADOW_GITHUB_APP_WEBHOOK_SECRET in .env.go-shadow; configure that same value privately in GitHub. Signed local ping admission does not prove GitHub webhook delivery. The latest App deployment evidence and remaining user authorization/installation steps are in reports/shadow-app-deployment.json.

The App was subsequently installed and authorized. node backend/scripts/check-shadow-auth.cjs --bootstrap-admin --app-flow explicitly selects github-app for the public tdurieux/synctex-js fixture, verifies the persisted App binding, compares its anonymized LICENSE with the immutable upstream source, checks ZIP CRC and the exact LICENSE entry, and removes the test resource. Results are in reports/shadow-app-authenticated-deployment.json. The live installation webhook initially returned 401 because GitHub had no signing secret. Setting the GitHub-side secret to match beta and retrying that delivery returned 204; Go persisted an unblocked installation with reconciliation complete. Private repositories and live revocation/suspension transitions have not been exercised.

For a later replay phase, first prepare sanitized fixtures with private bindings, tokens, account identities and pending jobs removed. Replay an explicit list of public read requests into the isolated stack and compare decoded output and latency. Do not mirror login callbacks, webhooks, uploads, authenticated requests or writes. Even public reads can update view counters or enqueue refresh work, so a shadow must never point to production MongoDB, Redis or writable repository storage.

Stop this stack with docker compose --env-file .env.go-shadow -f backend/compose.shadow.yml stop. Its named volumes remain available for the next rehearsal. No production service needs to stop.

Default deployment

The root Dockerfile and docker-compose.yml run Go. The API handles file streaming and background jobs in the same service; no Node streamer is needed. Browser assets still build with Node. The runtime uses UID 10001, a read-only root filesystem, a writable temporary filesystem, dropped capabilities and a 45-second shutdown grace period. Prepare repository-storage permissions for UID 10001 before startup. Shared-cache paths need writable mounts too when configured outside repository storage.

For a new installation, configure .env, create storage with install -d -m 0750 -o 10001 -g 10001 repositories using an account permitted to set ownership, and run docker compose up -d --build. Use APP_ENV_FILE and REPOSITORIES_DIR to select another environment file or storage directory. The default exposed port is 5000. HTTPS is required with the default secure session cookies; set COOKIE_SECURE=false only for local HTTP development.

For an existing installation, complete the cutover checks below first. The separate backend/compose.yml remains available for a staged proxy switch on port 5001. To use the root stack instead, stop the Node API and all streamers with docker compose -f docker-compose.node.yml stop anonymous_github streamer, then start docker compose up -d --build anonymous_github. Keep the same Compose project name so database volumes and networks remain attached. This does not remove stopped streamer containers.

After building the image, run python3 backend/scripts/test-deployment.py anonymous-github-go:local to rehearse the root Compose configuration. The test uses synthetic credentials, random loopback ports and disposable volumes. It checks startup, runtime permissions, static routes and graceful shutdown without accessing deployment data. CI runs the same check.

The preserved Node configuration is Dockerfile.node with docker-compose.node.yml; its image tag is separate from Go. It uses Debian for both dependency installation and runtime so native MongoDB compression modules load correctly. CI builds the rollback image and checks Zstandard and SQLite at runtime. After completing the rollback data checks below, build/start it with docker compose -f docker-compose.node.yml up -d --build anonymous_github streamer. Stop Go first. GitHub App mounts use docker-compose.github-app.yml for Go and docker-compose.node.github-app.yml for Node.

Cutover

Take restorable MongoDB, Redis and repository-storage snapshots. Rehearse restoring them to an isolated staging environment. Keep staging OAuth callbacks and webhooks separate from production.
On the staging copy, run the existing Node credential migration verifier. If legacy credentials remain, follow that tool's maintenance-mode procedure while all writers are stopped. Do not print key material or tokens.
Validate startup against the copied indexes and data, FS/S3 permissions for UID 10001, OAuth and App callbacks, session continuity, token scopes, revocation, representative large repositories, Git LFS, uploads, PRs, gists, conferences and browser project pages. Compare JSON and file output for representative production fixtures. Benchmark complete HTTP requests and memory under expected concurrency.
Before production cutover, block new writes, then either drain BullMQ or use the offline handoff below. Stop the Node API, all streamers, workers and scheduled writers before switching runtimes. Do not run both job implementations against the same data. Keep MongoDB, Redis and backup services running. Do not use down --volumes or --remove-orphans.
Set GO_BACKEND_NETWORK to the existing database Docker network and GO_ENV_FILE to the environment file. Grant UID 10001 access to repository storage using your existing ownership/ACL policy. Check configuration without printing interpolated secrets using docker compose -f backend/compose.yml config --quiet.
Start with docker compose -f backend/compose.yml up -d --build. The default listener is localhost port 5001. Verify health and authenticated/public smoke tests before switching the reverse proxy. Reopen writes only after the checks succeed.

The Go image can also run behind the existing load balancer without Compose. All replicas must share MongoDB, Redis, credentials and repository storage. /healthcheck verifies the HTTP process; it is not a database readiness probe.

Rollback

Stop Go and its workers before starting Node. For a lossless rollback after Go has accepted writes, drain pending conferenceMembership events, inspect pending gojobs, reconcile resource states, and enqueue corresponding work through Node's supported queue APIs. Never edit BullMQ Redis internals. Preserve the new file generations and encrypted credentials. The browser renderer remains compatible with the existing asset build.

A snapshot rollback discards writes accepted after the snapshot. Choose that only as an explicit recovery decision. Rehearse rollback in staging, including private names, conference histories, newly published trees, removal jobs and sessions. Do not assume changing the container image alone reverses data changes.

Cache telemetry

CACHE_MONITORING_ENABLED defaults to true. Go records provider-independent source cache hits, misses, fills, stored bytes, fill waits, admission bypasses and failures. Per-repository raw-cache reads use TypeScript's legacy_source_hit and legacy_source_miss labels and byte counts. Shared-cache reads use source_hit and source_miss, so the report does not mix the two cache paths. Request traffic is classified as file, website, ZIP, monitor or background; client-supplied telemetry labels are ignored. Events contain no paths, repository names, IDs, tokens or anonymization terms.

Recording uses a mutex-protected buffer capped at 512 minute/event/traffic groups. A worker flushes every five seconds with a three-second database deadline. MongoDB aggregates counts across instances; TTL indexes retain minute data for 48 hours and hourly data for 30 days. The admin report flushes local pending events, supports the same 15-minute, one-hour, one-day and seven-day windows, and sends Cache-Control: no-store. Concurrent flushes return an unavailable report instead of accumulating blocked requests. Failed batches are discarded without replay; the dropped count is a conservative upper bound if a write partially succeeded. Graceful shutdown flushes pending events.

The report returns the configured shared-cache state, filesystem support and live source/result budget summaries. Statistics remain visible when event monitoring is disabled, matching TypeScript. Missing optional caches produce null summaries; an unreadable cache index returns the generic unavailable response without partial statistics or internal error details. Shared decision and compression events enter the Go metrics buffer. The offline telemetry importer preserves TypeScript history in the Go report.

To transfer cache telemetry, stop the Node processes that write the shared cache and flush their pending metrics before shutdown. Use the original telemetry.sqlite with its adjacent WAL files, or a consistent SQLite backup. Copying only the database file while writers are running can omit committed history. Run from the repository root with its Node dependencies installed:

export MONGODB_URI='mongodb://127.0.0.1:27017'
export MONGODB_DATABASE='anonymous_github'
export CACHE_TELEMETRY_FILE='/absolute/shared-cache/telemetry.sqlite'
node backend/scripts/migrate-cache-telemetry.cjs dry-run
NODE_CACHE_WRITERS_STOPPED=1 node backend/scripts/migrate-cache-telemetry.cjs apply

The dry run validates the full read-only SQLite snapshot and prints row counts and its digest without writing to MongoDB. Apply requires the explicit stopped-writer acknowledgement. Import runs in batches of 500 with a two-million-row limit. A single manifest in gocacheimports binds the destination to this snapshot; a different snapshot is rejected. Interrupted runs can be resumed with the same source, and completed replays do not restore expired rows. This imports one deployment's shared telemetry database, not multiple independent histories. Retain the manifest until the migration is retired.

Imported counters use separate IDs in gocachemetrics, so concurrent Go events add to the report without replacing historical totals. Minute/hour buckets, latency buckets, dropped counts and original expiration times are preserved. Only known event and traffic labels are accepted, including TypeScript's inspected_changed, inspected_unchanged, inspected_passthrough and inspected_binary events and legacy_source_hit / legacy_source_miss; health hostnames and process IDs are hashed. The importer leaves SQLite unchanged. The integration rehearsal generates history using the actual TypeScript telemetry module, verifies all four Go report windows, resumes an interrupted import, checks concurrent retries and rejects changed or malformed snapshots.

Transformed-file cache

Text-file requests now reuse successful worker outputs. Keys include a SHA-256 digest of the actual input, matcher version, normalized anonymization options, provider-specific self links, mask, hostname, repository identity and revision. Provider authorization and current repository revision are checked before reading source data and again before returning transformed output, including cache hits. Failed transformations are not cached.

Each Go process maintains a private temporary directory with a one-hour TTL, at most 128 entries, a 512 MiB retained-data budget and a 128 MiB per-entry admission limit. Identical concurrent requests share one worker result. Existing worker admission limits also bound input spooling and cache-copy concurrency. Cache failures fall back to transformation; malformed or truncated entries are discarded and repaired. Cache reset, removal, expiration and generation cleanup invalidate local entries. Maintenance prunes idle entries, and shutdown removes the directory. Replicas have separate caches; revision checks prevent reuse after a change handled by another replica.

The cache reports legacy_result_hit and legacy_result_miss through the admin telemetry endpoint. Integration tests cover byte-identical cached output, changed bytes under an unchanged provider revision, settings changes, reset, revoked access and eight concurrent reads launching one worker. This implements the legacy transformed-output fallback. Shared result-cache decisions are implemented separately below.

Worker-cache measurements and cancellation

Run go run ./cmd/cache-benchmark from backend/ to measure cold and warm worker paths. The command launches the actual isolated worker, runs seven samples per mode, hashes every output, and fails on an output mismatch or a worker launch during the warm phase. Cold samples remove that fixture's cached result before each run. Timings include input spooling/hashing, cache publication or copying, output reading/hashing, and cleanup.

The recorded development-host run in reports/transformed-cache-benchmark.jsonl measured these medians: 60 KB text, 15.13 ms cold / 3.28 ms warm (4.61×); 2 MiB with absent terms, 139.43 / 78.16 ms (1.78×); 2 MiB with replacements, 258.84 / 95.90 ms (2.70×); and 128 KiB JavaScript-boundary input, 58.41 / 6.17 ms (9.46×). Each warm fixture launched zero workers. These are seven sequential samples on a shared development host, not a TypeScript comparison or an HTTP throughput measurement. Provider access, authorization, MongoDB and Redis are excluded.

Input spooling, cache copying and cache publication check cancellation between reads. Canceled operations release admission slots and remove partial output. An arbitrary blocking io.Reader must still supply its own interruption mechanism; network readers retain their request context. Real-worker tests verify that tighter output limits cannot reuse larger cached output, failed transformations are not cached, and mask changes select a different result.

GitHub README decoding

GitHub README previews apply the same Buffer.from(content, encoding).toString("utf-8") behavior as TypeScript. Contracts extracted from the actual TypeScript expression cover permissive standard/URL-safe base64, missing padding, ignored characters, invalid UTF-8 replacement, hexadecimal prefixes, UTF-16 unit narrowing and encoding aliases. This decoder belongs to the GitHub adapter; other providers continue returning normalized text through Discovery.Readme.

Invalid encodings and unsupported content shapes become readme_not_available (404) at the preview endpoint and do not overwrite a valid cached README. Decoded byte buffers, including array-like allocations and UTF-16 encoding expansion, are capped at 32 MiB (UTF-8 replacement text can be up to three times that size) to prevent a tiny upstream JSON object from requesting unbounded memory. This is an intentional bound beyond Node's permissive Buffer constructor. Integration tests verify decoded HTTP text, cached reads, branch JSON output and cache preservation on decode failure.

HTTP response compression

Eligible API and static responses negotiate Brotli (quality 4), gzip or deflate using the TypeScript middleware's preference rules. Known-length responses below 1 KiB stay uncompressed; unknown-length streams can compress immediately. Vary: Accept-Encoding is retained even for eligible responses below the threshold. Cache-Control: no-transform, existing non-identity content encodings, HEAD requests and noncompressible MIME types prevent compression. Repository file routes remain excluded, and the GitHub webhook stays before the compression middleware, matching TypeScript's mount order.

The wrapper streams through bounded compressor buffers, preserves flushing and response-controller access, removes stale content lengths, and propagates cancellation and write failures. It does not buffer an entire streamed response. JSON responses provide their encoded length before header commitment so the threshold applies correctly. MIME compressibility data is generated from the installed mime-db, including explicit false entries.

contracts/compression.cjs runs 117 HTTP cases through Express with the actual compression options extracted from src/server/index.ts. Go tests compare negotiated encoding, Vary, content length, status and decoded body hashes. Compressed bytes may differ between compressor implementations; decoded output must match. Additional tests exercise static responses through Server.ServeHTTP, webhook exclusion, flushing, failed writers, cancellation and HTTP/2.

JSON output and conditional responses

JSON responses now use compact output and strong ETags derived from the transmitted JSON bytes. The shared response path handles GET/HEAD/QUERY freshness, weak validators, validator lists, wildcards, Cache-Control: no-cache, and If-None-Match precedence over If-Modified-Since. It strips entity headers and bodies for 204/304 responses and suppresses HEAD bodies. File revalidation uses the same validator matching rules. Authentication, local permissions and provider authorization still run before returning 304; integration tests cover unauthenticated users, strangers and revoked provider access.

Sixty-six HTTP fixtures run through Express with the application's strong-ETag setting and compare status, body, content type, length and ETag. JSON U+2028/U+2029 handling matches JSON.stringify without changing literal backslash escapes. Go map keys retain deterministic sorting, so tags are computed from Go's actual response order rather than assuming TypeScript insertion order.

Non-finite numeric values serialize as null. Invalid provider values or cyclic/excessively nested data produce a structured 500 before committing a successful status; internal encoder errors are not exposed. Response normalization caps nesting at 100 levels to bound recursive processing, and custom JSON encoders still receive normal serialization error handling. Maintenance-token creation defers its 201 status until encoding succeeds. Failed network writes abort the response instead of appending an error document to partial JSON.

Response recovery tracks header commitment beneath compression, including routes that bypass compression. Panics before final headers return a generic 500 and discard stale file/download headers. Panics or route errors after headers abort the connection, matching Express finalhandler, without appending JSON or completing a truncated compressed stream. Real HTTP tests cover plain and gzip streams, explicit aborts, flush-only responses, and recovery after informational 103 headers. Panic values are never included in the error response or request-panic log.

Provider quota delays in background jobs

Preparation jobs that receive a provider quota error enter delayed until the reset instant. The admin API reports them as delayed, and maintenance enqueue cannot reset their deadline. Quota delays do not consume the three ordinary failure attempts or increment completed/failed queue metrics. Repository status becomes queue with the existing rate_limited:<milliseconds> message, so the public response retains its retry time. Missing or expired reset times use a one-minute fallback. Resource revision checks protect concurrent removal/reconnect, and job lease checks protect replacement workers. Successful retries clear the quota message and stale job error fields.

Adapters can return provider.RateLimitError{ResetAt: ...} or provider.ErrRateLimited; they do not need GitHub-specific job logic. The GitHub adapter supplies reset times from X-RateLimit-Reset and Retry-After, taking the later valid instant. Its existing public error code stays unchanged. Integration tests exercise repeated quota delays on the last ordinary retry, delayed claim eligibility, stale leases, concurrent removal, successful publication and the actual worker loop with a non-GitHub adapter.

Go uses a separate queue. Existing BullMQ work can be imported with the offline handoff tool below; starting Go alone does not migrate it.

Shared provider quota gates

internal/quotagate stores provider/scoped reset times in Redis and a bounded 2,048-entry local fallback. Scope keys are SHA-256 hashes of an unambiguous provider/scope pair. Redis contains only the hash and reset timestamp. Atomic updates retain the latest reset under concurrent writers; expiry releases scopes automatically. Redis failures preserve observed local limits and permit unknown scopes, matching TypeScript's best-effort quota coordination. Cancellation stops the caller before an upstream request.

GitHub API, blob and raw-file requests check the gate before dispatch. Quota responses extend it; a known limit returns the existing 429 error immediately, allowing background jobs to release their slot and enter the delayed queue. This differs from TypeScript's in-request sleep: Go does not hold an HTTP request open for a long provider cooldown. Existing resource authorization still runs and a quota gate never authorizes access. Blob quota errors no longer trigger another request through raw fallback.

App-user scopes survive access-token renewal, installation scopes span repository-scoped token renewal, and App scopes survive JWT renewal. OAuth tokens use their digest as the scope; anonymous calls share the anonymous scope. Credential-to-scope registrations store token digests and are bounded to 2,048 entries. Go uses its own hashed Redis namespace and imports active TypeScript gh_rate_gate: records when a credential is used. It checks both the token-hash key and the applicable installation/App-user key, preserving the latest valid reset. Imported records survive in Go after the legacy keys expire or are removed; later TypeScript extensions are picked up on subsequent checks. Legacy records are never changed or deleted by Go, and a shorter legacy delay cannot shorten a Go gate.

Other adapters can construct quotagate.New(store.Redis), call Current(ctx, providerID, scope) before upstream requests, and call Extend(ctx, providerID, scope, reset) after quota responses. Choose the scope from server-owned credential/account metadata, not an HTTP caller's scope parameter. Return provider.RateLimitError{ResetAt: reset} when gated so the shared job scheduler handles the delay. Distinct providers remain isolated even if their scope strings are identical.

Tests cover concurrent max-reset updates through real Redis, independent instances, provider/credential isolation, expiry, malformed stored timestamps, local fallback after Redis failure, cancellation, memory bounds, token rotation and suppressed upstream requests. These gates coordinate known limits; they do not serialize requests that were already in flight when the first quota response arrived.

Legacy quota import needs no credential export or Redis-wide scan. It uses the server's existing authenticated credential binding. Ten contracts execute the TypeScript token-key function and registration scope expressions, covering ordinary credentials, anonymous requests, installations, App users and Unicode tokens. Integration tests seed those exact Redis keys and verify suppressed HTTP calls, import persistence across Go instances and renewed credentials, unchanged legacy values/expiries, max-reset handling, cancellation and expiry. Malformed, noninteger, expired and out-of-range timestamps are ignored. The bridge is one-way: new Go quota observations are not written back into TypeScript's namespace. BullMQ jobs still require a separate handoff.

Offline BullMQ handoff

node backend/scripts/migrate-bullmq.cjs defaults to a dry run. Run it from a checkout with npm ci dependencies installed. Set MONGODB_URI, MONGODB_DATABASE, REDIS_HOSTNAME and REDIS_PORT to the same services and database used by the application. The database name is required explicitly. The command reads the application's three queue names through the installed BullMQ library and never decodes or edits private Redis job records.

Block new writes and take verified MongoDB/Redis/storage backups. Run node backend/scripts/migrate-bullmq.cjs pause. It pauses the three BullMQ queues and Go queue controls. Already running jobs are not cancelled.
Let active jobs finish, then stop all Node and Go producers, workers, streamers and scheduled writers. A paused queue still accepts new jobs, so pausing alone is insufficient. The tool checks for connected BullMQ workers, active Go jobs and unpaused queues; those checks cannot prove that every external producer has stopped.
Run node backend/scripts/migrate-bullmq.cjs dry-run. Review the job counts, resource-transition count and source digest. No job/resource/checkpoint documents are written by a dry run. Resolve conflicting existing Go work before applying.
Run node backend/scripts/migrate-bullmq.cjs apply. Require a successful exit and complete: true. If interrupted, keep both runtimes stopped and rerun the same command. A process killed outright can retain its migration lock for up to one minute. The snapshot digest must remain unchanged; investigate a changed snapshot instead of deleting the checkpoints.
Start Go with its queues still paused, inspect the imported queue and resource states, and run the staging smoke checks. Resume Go queues through the admin controls once those checks pass. Keep the Node queues paused. Retain their records and backups until the cutover is accepted.

Each source job gets its own stable Go ID and import checkpoint, including duplicate jobs for one repository. Waiting order, priorities, future availability, attempt counts, fixed/exponential backoff and configured retry limits are retained. Failed history is imported as failed; its public error is sanitized to job_failed, and raw payload fields, exception text and stack traces are not copied. Missing repositories produce failed jobs rather than recreated resources. Live preparation jobs transition eligible resources to preparing; live removal requests transition them to removing. Removal wins when both actions are pending. Terminal resources are not revived by preparation jobs. Resource changes use revision guards. The ordinary Go enqueue path recognizes imported live jobs and does not add duplicate maintenance work.

gojobimports records the frozen source digest and each resource/job checkpoint. A retry after an interrupted write completes the checkpoint without resetting an existing Go job. Completed checkpoints also prevent jobs deleted later by an administrator from being recreated. An atomic Redis lease prevents simultaneous import commands. Source queue jobs are never removed or resumed by the tool. Custom backoff functions, jitter, LIFO, dependency flows and recurring schedules are rejected explicitly; the current application does not create these, and they need a separate design if introduced.

Before Go workers run, rollback can use the verified pre-cutover backup and paused source queues. After Go has processed imported work or accepted writes, resuming the retained BullMQ snapshot would replay work. Stop Go and reconcile queue/resource changes using the rollback procedure above; retaining the source snapshot does not make that later rollback automatic.

The integration suite creates real BullMQ jobs in disposable Redis, imports them into disposable MongoDB, injects interruption after resource and job writes, and then runs Go workers against a non-GitHub provider. It covers duplicates, priority order, delayed eligibility, retry success/exhaustion, completed-checkpoint replay prevention, sanitized history and unsafe preconditions. This is an automated rehearsal, not evidence of a production cutover.

Imported failed removal jobs receive an automatic recovery pass at Go worker startup, matching TypeScript's startup recovery intent. A paused startup pass remains pending until maintenance observes that the queue has resumed. It leaves active jobs and unrelated failed jobs alone and records migration.removalRecovered so periodic maintenance cannot repeatedly reset an exhausted imported job. A later process startup can retry it again, as TypeScript does. Normal cleanup maintenance still handles resources already in removing.

Both automatic and admin-requested retries recheck the resource immediately before cleanup. An imported removal can recover an error resource only when its anonymization date is absent or no newer than the source job, or when its current owner is still banned. Already removed resources are not revived. Without a current owner ban, restored ready resources and newer preparation errors are left alone. MongoDB date comparisons retain TypeScript's treatment of null/string values. Revision checks also protect owner transfers, provider-binding replacement and anonymization-date changes during the claim. Tests cover single, bulk and automatic retries, pause/resume behavior, stale requests and a real failed BullMQ removal imported and completed by a Go worker.

Repository removal and expiry snapshot exact storage object names and MongoDB row IDs, then verify the resource revision before deleting. Duplicate workers cannot recursively delete a newer generation published after restoration. Deletions are retryable and batched in groups of 1,000 for S3 and MongoDB; filesystem deletion never descends into a captured path that has become a directory. The snapshot uses the existing 100,000-object storage-list bound and a 1,000,000-row bound per collection. Exceeding either bound leaves the resource pending cleanup and reports an error rather than deleting a partial snapshot. Empty filesystem directories can remain. This relies on Go's immutable content generation paths and read-only archive originals; Node workers and external storage writers must remain stopped after cutover. A regression test pauses a worker logically after snapshot capture, completes removal and restoration through another worker, and verifies that repeated stale cleanup preserves the new content and rows.

Go runs independent worker pools matching TypeScript: 3 download workers, 5 removal/expiry workers and 5 cache workers per process. Each pool observes only its own pause key and claims only its queue's actions. Downloads cannot occupy cleanup slots. Pausing leaves in-flight work running and prevents subsequent claims once workers observe the pause; resuming wakes idle workers within the one-second polling interval. RunJobs returns after cancellation has stopped all its worker loops. The TypeScript worker-capacity contract and an integration test with blocked non-GitHub downloads and blocked S3 deletions verify all three capacities, pause isolation, resumption and shutdown. Cache reset snapshots exact file names before changing its cache revision, so delayed deletion cannot erase newly fetched content under that revision. Its snapshot uses the storage listing limit of 100,000 objects; larger caches require cleanup before retry. The S3 integration test publishes a file under the new revision while deletion is blocked, then verifies that it survives.

The Go executable handles SIGINT and SIGTERM by cancelling queue work and stopping new HTTP connections, then allowing up to 30 seconds for in-flight HTTP requests and worker loops to finish. It closes the anonymization cache, storage and database only after both have stopped. A bind failure starts no workers and exits with status 1. An unexpected HTTP server stop also cancels workers and drains requests. If the grace period expires, it closes HTTP connections and exits with status 1 without closing shared dependencies underneath unfinished work. The Compose service allows 45 seconds before a forced container stop, leaving time for metric flushing and database disconnect after a successful drain. Tests cover requests finishing during drain, blocked workers, grace expiry, bind failure, unexpected listener closure, and clean SIGINT/SIGTERM exits from the built binary against disposable MongoDB/Redis.

Shared-cache index migration is under implementation in internal/blobcache. Its SQLite schema is generated from the actual TypeScript BlobCache constructor, including accounting triggers. Go can open TypeScript-created indexes, backfill older indexes without summary counters, exchange transformation decisions with Node, and report the same statistics. The Go driver is pure Go, preserving the CGO-free build. The source-cache methods are connected to provider content serving when enabled. Shared transformed-content decisions and decoded output serving are connected when enabled. The process-local transformed cache remains the fallback. Stored-gzip HTTP negotiation is implemented for shared-cache file responses.

Shared blob reads support TypeScript's Git and SHA-256 digest paths, aliases, identity files, decoded gzip and stored gzip bytes. Readers acquire two-minute SQLite leases and renew every 30 seconds; EOF, errors, cancellation and close release them. Cache shutdown closes active readers before its filesystem root and database. A Node eviction test verifies that Go's pins protect an open blob and that releasing the reader permits eviction. Reads stay inside an os.Root, reject non-regular files, defer gzip initialization until first use, and bound decoded output to the recorded size. Missing or damaged payloads invalidate only the captured publication filename, preserving a replacement for the same digest. Unlike TypeScript's size-only open check, Go also discards blobs after detected gzip corruption or decoded-size mismatch. Reader renewal fails closed if its lease is missing or expired. These methods remain provider-neutral; authorization is the caller's responsibility and must precede lookup.

Shared-cache maintenance now recovers expired publishers, expires reader leases, prunes old decisions/history, and evicts unpinned blobs in access order. It retains TypeScript's 512-candidate eviction limit and 256-row decision/history expiry batches. Expired fills are read in batches of 256 and recovered within one transaction; published or temporary files are removed before reservations are released. A failed unlink retains the reservation for retry, including temporary-spool failures that TypeScript currently ignores. Indexed publications remain intact. Hourly orphan scans stream directory entries in groups of 128, reject a symlinked scan root, do not traverse nested symlinks, and stop beyond 64 directory levels. Scans preserve both indexed filenames and files belonging to live publishers; failed scans release their own scheduling claim for retry. Tests compare Go and Node eviction results, exercise strict expiry boundaries and batch limits, protect active readers and publishers, and inject cleanup failures. A process-crash rehearsal kills a real Node publisher after rename but before indexing, recovers its payload and reservation in Go, and verifies that Node can refill the same key afterward. When the shared source cache is enabled, the server runs maintenance at startup and once per minute, with a 30-second deadline. Shutdown cancels and joins maintenance before closing the cache.

Shared-cache fills now reserve disk space, coordinate producers through SQLite leases, bound input size, compute Git/SHA-256 proofs, and publish identity or gzip blobs with content deduplication. Provider adapters supply an optional verifier; without one, content uses SHA-256. A verifier can select either computed digest but cannot publish a different content hash. Publication journals commit before rename, and a temporary reader pin protects the result until its caller acquires a reader. Cancellation closes upstream streams; producers and verifiers must honor their context. Cache shutdown cancels fills and joins cleanup. Transactions retry a busy writer lock for up to two seconds before running their callback; callbacks with filesystem side effects are never replayed. Reader release retries within its one-second cleanup deadline, with lease expiry as fallback.

Fill tests cover oversized streams and hints, verifier failures, incorrect digests, disk/budget admission, timeouts, lost leases, shutdown during producer/read/verification, concurrent cache instances, deduplication, and cleanup of reservations and temporary files. A separate Node process reads Go-published Git identity and SHA-256 gzip blobs and waits for an in-progress Go fill without invoking its own producer. Fill admission currently requires Linux disk-allocation accounting; other platforms return disk_accounting_not_supported. Metrics are exposed through a nonblocking observer callback for fills, hits/misses, reuse, eviction, compression and decompression. The observer must not call back into the cache. The source APIs are connected to HTTP content serving when enabled. This does not establish complete backend parity or an HTTP performance improvement.

Providers can optionally implement provider.ContentVerifier to bind downloaded bytes to their immutable entry revisions. It receives cache-computed Git SHA, SHA-256 and size, and returns a verified content digest. Authorization remains separate and mandatory before cache access. The GitHub adapter accepts matching Git content directly; otherwise it fetches the original Git blob as a bounded LFS pointer, checks its Git hash, version, payload SHA-256 and size, and returns the payload's SHA-256 identity. Pointer verification has a 30-second deadline and a 1,024-byte limit, closes streams on cancellation, participates in quota gating, and does not follow redirects or raw-content fallbacks. Twenty-three contracts extracted from TypeScript cover ordinary blobs, LFS fields, duplicate fields, line-ending differences, hash/size failures and pointer limits. Additional tests cover cancellation, bounded reads, upstream failures and request restrictions. The shared-source serving adapter invokes this capability. The legacy raw-cache path used when shared caching is disabled does not invoke it.

Set SHARED_CACHE_ENABLED=true with filesystem storage to use the shared source cache. Go reads the TypeScript SHARED_CACHE_PATH, SHARED_CACHE_BYTES, SHARED_CACHE_MAX_FILE_BYTES, SHARED_CACHE_GZIP, SHARED_CACHE_GZIP_MIN_SAVING, SHARED_CACHE_FREE_BYTES and SHARED_CACHE_LOW_WATER settings with the same defaults. S3 and archive originals retain their existing storage paths. Provider reads authorize before lookup and again before returning a hit or completed fill, with lifecycle checks rejecting removal or revision changes during download. Admission failures fall back to an upstream stream bounded by the request, a 120-second deadline and the normal file-size limit. Other errors propagate. GitHub supplies its verified Git revision as a globally reusable alias. Providers with opaque revisions use a namespace containing provider, repository identity, commit, path and cache lifecycle; they can optionally implement ContentIdentity alongside ContentVerifier when their revisions prove a globally reusable identity. Cache events retain request traffic classification through the metrics buffer. Integration tests exercise non-GitHub HTTP reads, hits, revocation before and during fills, lifecycle changes and budget fallback. Verified legacy-file promotion and decoded shared transformed-output serving are implemented. Stored-gzip HTTP negotiation is implemented for shared-cache file responses.

Shared-source misses can promote existing Go raw-cache files, TypeScript generation-based files, or pre-generation flat files. Promotion requires ContentVerifier; providers without one fetch upstream rather than treating file location or size as proof. Cache paths use the TypeScript JSON serialization and generation hash, with 18 contracts extracted from the actual path expressions. A rejected legacy proof or failed legacy read retries once from upstream, with access/lifecycle checks before that download. Admission bypass always downloads upstream and never serves unverified legacy bytes. Legacy files remain unchanged for rollback. Twenty-four integration cases cover three layouts with valid, corrupt and short files, providers without verification, admission fallback, revoked access, and revocation/removal during verification. The source_legacy_promotion metric records promotion attempts, matching TypeScript.

With shared caching enabled, SHARED_CACHE_DECISIONS defaults to true and SHARED_CACHE_RESULT_BYTES defaults to 512 MiB. Result objects are bounded by the smaller of the source object limit and one eighth of the result budget. The result cache uses TypeScript's default compression saving and low-water thresholds and joins source-cache maintenance/shutdown. Changed output uses a SHA-256 result blob; byte-identical output and binary content save a decision pointing back to the source, without storing a duplicate. Misses run the existing isolated Go worker with its input/output and deadline limits. Admission failures preserve the original source stream for the existing transformation fallback. Decision hits still recheck provider access and repository lifecycle before returning output. Missing result bytes trigger a new transformation.

Decision keys follow TypeScript's anonymizer-v2 hashing algorithm, checked by 12 extracted contracts, and include source digest, path, generation, cache revision, rewrite options, mask, hostname and text extensions. Go adds explicit sourceURLs to represent provider-specific rewrite rules. This distinguishes Go decisions from old Node decisions lacking that field, so the first Go transformation can still miss the decision cache after cutover; blob formats remain shared. Integration tests verify changed/unchanged/binary output, zero worker starts on warm reads through a second cache connection, no redundant unchanged blobs, missing-result recovery, revoked access, cache reset and admission fallback. The file route selects stored-gzip or decoded readers using the shared HTTP response path described below.

Concurrent shared-cache initialization retries SQLITE_BUSY across connection setup, WAL selection and transactional schema/backfill, with a five-second overall deadline and short SQLite waits. Only idempotent initialization SQL is replayed; publication callbacks with filesystem side effects still run at most once. Tests repeat eight simultaneous openers and verify that canceling a blocked initialization leaves an existing decision intact.

Six contracts extracted from the TypeScript admin cache handler cover disabled caching, source-only and source/result caches, S3, and failures of either cache index. Integration tests compare live blob/decision counts, pinned readers and budgets with telemetry history, and exercise the HTTP endpoint with monitoring disabled. Anonymous and non-admin requests receive 401; administrators receive Cache-Control: no-store. Cache-index failure returns HTTP 200 with available: false and the same generic reason as TypeScript.

Shared-cache file responses now follow the installed TypeScript accepts/negotiator behavior for gzip and identity, including quality values, wildcards, quoted parameters and rejected requests. Stored gzip is sent byte-for-byte with its stored length; decoded identity uses the original length. Identity-stored content stays identity when allowed, even with a gzip preference. If identity is forbidden but gzip is accepted, Go compresses that stream while serving it. Responses set Vary: Accept-Encoding and Accept-Ranges: none; requests accepting neither supported representation receive an empty 406. The public file route returns HEAD metadata before shared-cache negotiation. It does not read source bytes, populate caches, launch transformation workers or count a view, even when the client rejects all encodings. Shared caching uses weak ETags and Vary: Accept-Encoding on HEAD and conditional responses. Authorization precedes negotiation, and original/result reader leases remain pinned while a stored representation is opened. Source or result admission fallback retains the existing streaming behavior.

The HTTP contract suite executes TypeScript's actual sendSharedContent and compares 108 GET/HEAD responses, including content length, encoding, Vary, range headers and decoded bytes. It seeds real Node cache files and checks that Go sends the stored gzip bytes without recompression. Database integration tests exercise the public file route with a non-GitHub provider, changed output, warm gzip/identity reads, metadata-only HEAD, GET 406 responses, no extra warm workers, revoked access and lease cleanup.

The file metadata suite runs the actual TypeScript public route to capture 115 HEAD and conditional-response contracts. Go integration tests compare MIME types, weak/strong validator behavior, cache headers, sandbox policies and attachment selection with shared caching on and off. Sandbox classification includes MIME aliases and XML suffix types, and applies before 304 responses. Known text filenames without a MIME mapping use text/plain on HEAD. Empty download= retains the inline policy. Tests assert zero upstream opens, worker starts, cache blobs/decisions and page-view increments, and reject revoked grants before HEAD or conditional responses.

File and ZIP downloads use the installed Express filename-header format, including ASCII fallbacks, UTF-8 extended filenames, quoted characters and literal percent escapes. A 277-case contract suite compares actual Express attachment headers, including every byte-range character and supplementary Unicode characters. Control characters are encoded rather than inserted into headers. Public-route tests compare exact attachment headers on HEAD and 304 responses, and preserve the octet-stream MIME fallback for unknown download extensions. Manual file-route 304 responses retain the attachment MIME type as Express does. Go uses the lowercase header spelling to avoid HTTP/1 suppression of that metadata; protocol tests verify one Content-Type header and no response body on HTTP/1 and HTTP/2. JSON responses retain their existing entity-header stripping rules.

Shared-source admission fallback retains EOF and read failures after closing its download context. MIME sniffing can consume an initial EOF; subsequent reads must still see EOF rather than cancellation. Generic-provider fallback rejects bodies shorter than Entry.Size, while allowing larger bodies within the existing limit. Source read failures retain a typed error through input spooling: truncated text returns storage_write_size_mismatch with HTTP 502, other text input-read failures return file_not_available with HTTP 502, and term/worker failures keep the existing 422 response. Error details are not exposed. Budget and object-limit integration cases cover short/empty text, short binary data, failed reads, complete/expanded bodies, provider-owned size semantics, exactly-once closure, no failed-output worker starts, empty cache reservations and correct view counts.

App login, OAuth callbacks and installation routes now use parsed query types for strict scalar checks. Mixed scalar/bracketed state or code values are rejected before token exchange; array/object error values cancel authorization. Installation suggestions ignore non-scalar IDs and repository names, and setup rejects non-scalar state rather than treating it as absent. The 1,274 Express fixtures check scalar shape and truthiness as well as string conversion. Focused integration tests assert no upstream calls for invalid OAuth callbacks. node backend/scripts/check-shadow-oauth-query.cjs verifies beta rejection and consumed-state replay with fresh anonymous sessions; it keeps state/cookies out of reports and verifies the sessions remain unauthenticated. Pending OAuth attempts are indexed by session and authorization provider. Redis scripts atomically replace earlier attempts and consume the current attempt before checking callback state, including missing or bracket-only state. A foreign browser cannot consume another browser's attempt, and App and legacy OAuth keep separate pending slots. Superseded flow records are deleted; pending indexes and records expire after ten minutes. In-flight logins started before this storage change must restart, while existing authenticated sessions remain valid. Integration coverage includes simultaneous callbacks consuming a flow exactly once.

OAuth lifecycle validation can use node backend/scripts/check-shadow-oauth-query.cjs --local when this server cannot resolve public DNS. This connects to the local HTTPS proxy while preserving the beta hostname and certificate verification, and records that access mode in the report. It does not prove public-DNS reachability. The latest lifecycle change passed all 50 contract commands and the full backend race suite; disabled-provider callback checks now run before pending-flow consumption. Focused integration tests verify that disabled App, disabled new connections and disabled legacy OAuth return 503, retain the pending attempt and allow its use after re-enabling.

Performance and cache telemetry queries share bounded numeric-window parsing after Express-compatible query conversion. Decimal, exponent, hexadecimal, binary and octal forms select the same windows as TypeScript; bracketed arrays and conversion errors preserve its behavior. Thirty-five extracted TypeScript window fixtures cover these paths. Performance reports still select only 15 or 60 minutes, and cache reports retain their existing supported windows and fallback.

Queue metrics, queue overview search/selection and error-log pagination use the shared extended query parser. Empty arrays retain JavaScript truthiness, queue selection preserves repeated-value coercion, and error offsets/limits accept bracketed values with existing numeric bounds. Twenty-nine extracted TypeScript fixtures and database integration cases cover these behaviors; the beta monitoring script also checks their public HTTP output using its temporary admin key.

File download flags now use parsed-value truthiness: empty arrays and objects request an attachment, while an empty scalar retains inline handling. Preview and repository-list force flags follow TypeScript loose equality, so force[]=1 refreshes; reconnect remains a strict scalar check. Fifty-three extracted query fixtures and provider-independent file metadata, discovery and authorized preview tests cover these distinctions. node backend/scripts/check-shadow-file-query.cjs performs read-only HEAD checks against the synthetic beta repository. Other preview binding, cache-hint and branch inputs remain part of the query audit.

Preview credential selection requires a scalar anonymizedRepoId to use a saved resource. Mixed scalar/array inputs follow the caller's connection path. New connection choices must be scalar; GitHub accepts only its two explicit choices, while other providers retain their own opaque names. Saved previews ignore query-supplied connection choices and use the authorized owner binding. Seventeen extracted TypeScript fixtures, provider-probe integration tests and live beta rejection checks cover these rules. Cache-hint, branch and generic repository-name query handling remain in the audit.

File search now parses extended query values, measures the trimmed query in UTF-16 code units and applies Unicode lowercasing compatible with the covered JavaScript cases. Twenty-two fixtures extracted from the TypeScript route and search method cover dotted I, Greek sigma, emoji, whitespace, literal search and malformed query shapes. Integration tests compare HTTP result bodies and verify archived repositories remain inaccessible even for short or malformed queries. Errors retain Go's stricter no-store cache policy. Search metadata retains the existing TypeScript repository-status policy rather than revalidating provider grants per request; file-content authorization has separate coverage. File listings, search and counts now scan a MongoDB cursor without the former 100,000-row cap. The cursor projects name/path/size, reads batches of 256 and closes on completion, cancellation or errors. Search stops at 500 matches. Connected TypeScript search-index ordering and path resolution still require parity review.

The large-tree integration fixture inserts 100,001 files and verifies that the final row appears in folder listings, search and counts; it also checks the 500-result search limit. This proves the former truncation boundary is removed. It does not establish a throughput improvement: sparse searches still scan the tree and anonymize paths per request, unlike TypeScript's persisted search index.

Folder listing requires a scalar path query, accepts original paths and resolves default-mask/custom-replacement paths to one original directory. Exact original paths take precedence over replacement collisions, with transformed collisions resolved in original path/name order. The resolved directory is applied to the MongoDB query before reading rows. Listings now include TypeScript's eight-character SHA-1 of the stored SHA for non-empty files; zero-length files and directories omit it. Four fixtures extracted from the TypeScript projection function verify output, alongside provider-independent path integration tests. Truncated-tree recovery and trailing-separator edge cases still need review. The file-content index is audited separately from folder listing.

File-content lookup now tries original paths before the transformed index. The index reads all rows through a sorted cursor and retains the first transformed collision in original path/name order. Its versioned cache key includes metadata revision, source commit and anonymization date; publication checks lock ownership atomically. Integration coverage includes original-path precedence, metadata/commit changes, stale builders and a transformed lookup beyond 100,000 rows. The Redis index streams batches with a 500-row cap and a 1 MiB byte target instead of retaining the complete index in Go. A row exceeding the byte target is written alone. It has no total encoded-size guard or fixed build deadline: the request context and renewed 60-second ownership lease bound each build. Atomic HSETNX batches retain the first sorted collision, including across batches. A staging sequence rejects a lost temporary hash rather than publishing a partial index. The 10-second competing-builder wait and 10-minute published-cache TTL remain. Lookups read readiness and the requested entry atomically. An evicted index is rebuilt once; repeated eviction returns a retryable 503, while a real miss in a complete current index remains 404. Lock release completes before retry, and repository revision changes return 409. Redis storage/expiry differs from TypeScript’s MongoDB index; matching internal storage is not itself a compatibility requirement. Truncated-file recovery, lifecycle publication fences and complete cache invalidation parity remain to verify.

Path lookup now verifies its repository snapshot before returning either an original or transformed row. Index builders check the same snapshot before and after Redis publication. The check covers metadata, tree generation, terms, commit, anonymization date, settings, content-cache revision, provider-grant revision and terminal status. Changes return repository_changed rather than exposing a row from the retired snapshot. Status-date-only cache bookkeeping does not invalidate paths. MongoDB and Redis are not a single transaction: an old revision's key can briefly remain after a concurrent change, but successful reads recheck the database and reject that snapshot. Unused versioned keys expire normally. This does not replace content-provider authorization.

Truncated-file recovery uses optional provider.FileMetadata, implemented by the GitHub adapter with commit-pinned Contents API requests. Shared code limits recovery to recorded truncated folders and paths unchanged by anonymization, validates returned metadata, and checks authorization before and after the lookup. Persisted rows carry metadataPending until a revision-checked update invalidates path and size caches. Exact-path retries finish pending invalidation. Upstream permission failures remain errors rather than being hidden as missing files. Folder recovery edge cases remain to audit; the new path primarily covers migrated truncated-tree records.

New pending recovery rows also retain the repository snapshot that produced them, excluding the metadata revision that completion itself changes. A retry discards a still-pending row whose commit, settings or grant snapshot is no longer current and returns repository_changed. The next request can fetch current metadata. This guards interrupted recovery across repository changes; imported pending rows without provenance retain the legacy retry behavior.

The minute maintenance cycle now scans metadataPending rows through a cursor backed by a partial index. It finishes invalidation for current and legacy-generation rows, removes pending rows with missing repositories or retired generations/statuses, and continues after individual failures. The row-level pending predicate prevents a stale cursor from deleting already-completed work. Maintenance can invalidate metadata for archived repositories, while public lookups still reject archived content. Recovery provenance checks discard new pending rows from stale source snapshots. Focused tests cover these states and confirm archived lookups stay blocked.

Providers can implement optional PartialTrees.TreeWithTruncation to return entries and incomplete directory paths together. The empty path marks the root. Shared preparation validates these paths and publishes truncatedFolders with the new tree generation; existing adapters can continue implementing only Repositories.Tree. GitHub keeps available entries when a non-recursive directory response remains truncated and records that directory instead of failing the entire import. The former 100,000-entry rejection is removed from GitHub traversal and shared preparation. Isolated tests import 100,001 files and cover incomplete root/child responses. This still uses an in-memory tree snapshot; no constant-memory import or throughput improvement is claimed.

GitHub submodule commit entries are retained as entries without file sizes, rather than being dropped. The partial-tree integration test verifies their presence and verifies that the public options endpoint anonymizes truncated folder names.

Folder resolution now uses Node-compatible POSIX dirname/basename behavior for its exact lookup, including trailing and repeated separators. Eighteen Node-derived contracts and HTTP listing tests cover this behavior. Paths that do not trigger mask/custom-replacement resolution remain exact directory queries. Resolved rows complete pending metadata invalidation when needed, and an index miss can use the existing guarded per-file recovery path, matching the TypeScript lookup used by folder listing. Recovery validates a relative path before contacting any provider. This does not add upstream directory discovery: GitHub's recovery capability returns regular-file metadata only.

Adapter wrappers must account for optional Go interfaces when embedding another adapter. Overriding Tree alone does not override an inherited TreeWithTruncation method, which preparation prefers when available. Implement the optional method explicitly or embed the narrower Repositories interface when the wrapper owns tree behavior. The partial-tree test adapter is separate from the common fixture so retry, rate-limit and worker-capacity wrappers exercise their own Tree overrides.

The full backend race suite passed after the path/tree changes, including worker retry, quota-delay and independent-queue tests. All 55 CI fixture checks and both differential commands passed: 499 behavior cases, three cross-language credential cases and 7,876 file-type cases. Formatting and go vet passed. Govulncheck found no reachable vulnerabilities or affected imported packages, but reported four module-only advisories in unused x/crypto SSH/OpenPGP packages. See fullValidationAfterPathAndTreeParity in reports/compatibility-audit.json for source fingerprints and evidence. Remaining query and index-limit findings are current-state findings; historical findings are retained separately.

Preview cache IDs now use the extended query parser and explicit-ID casting checked against the installed Mongoose model. Repeated/bracketed ID lists, wrapped _id values, uppercase hexadecimal IDs and legacy 12-character ASCII IDs retain their casting behavior. Invalid casts return a generic internal_error, and scalar reconnect=1 bypasses the hint. The lookup still rejects cross-provider records and requires upstream authorization/identity checks. Operator-shaped objects are deliberately rejected instead of forwarding arbitrary query selectors to MongoDB; this is a stricter compatibility boundary. Sixteen Mongoose contracts and provider-hint integration tests cover the supported inputs.

README preview branch queries now use extended-query parsing and JavaScript-compatible string coercion before reaching a provider. Missing or empty scalar branches retain the default; singleton arrays select their value, repeated values join with commas, and a truthy array containing an empty string returns readme_not_available instead of silently selecting the default. Twelve contracts extracted from the TypeScript README method cover selection. Integration tests verify cached output, authorization before malformed-branch handling, and opaque branch names for another provider. For GitHub, selection is deferred until branch discovery succeeds. Empty branch lists return readme_not_available without coercing the query, and discovery errors take precedence.

Legacy GitHub user search now preserves the extended-query value through the route's length gate and credential lookup. Singleton arrays return an empty list; larger arrays and ordinary objects reach credential lookup before failing Octokit's string requirement. Objects with a length property use JavaScript numeric comparison. Scalar searches retain UTF-16 length and Octokit's treatment of literal plus signs as search separators. Twenty-four contracts execute the actual TypeScript callback with the installed Octokit endpoint serializer; integration tests check upstream query values and returned user fields. This route remains GitHub-specific, with no change to provider interfaces.

The TypeScript repository/PR preview routes take repository names from path parameters, not a name query. Go's generic provider name query is an extension, so its scalar behavior is not a TypeScript parity gap.

Admin status flags now use extended-query loose equality, search rejects truthy non-string values before owner lookup, and role=admin requires a scalar value. The 111 actual-route filter contracts include bracketed/repeated values, malformed objects and missing-owner ordering. Repository and conference results use MongoDB Find with the existing credential-excluding projection; user results retain aggregation for repoCount. Three installed Node-driver contracts verify negative-limit single-batch semantics, and integration tests exercise these behaviors. Admin date parsing is covered by the date contracts described below.

The optional SelectedReadmePreviews capability accepts a BranchMatch callback from shared routing. Adapters discover branches first and evaluate the callback against branch names; query objects and Express semantics stay in server code. GitHub implements it alongside the existing DiscoveryPreviews interface, which remains available to other adapters. Nine contracts execute the actual TypeScript README method to check malformed input against populated/empty branch lists and discovery failures. Integration tests compare those errors and call ordering, retain cached-read authorization checks, and exercise the new capability with an opaque-name non-GitHub provider.

Admin owner, user-status and repository-conference predicates now use explicit Mongoose-compatible string-field casting. Find/count/CSV filters support repeated values, wrapped string IDs and the installed schema's field operators; unsupported operators and invalid casts fail. The echoed filter retains Mongoose's nested operator mutations, while aggregation uses the original snapshot. This distinction can produce different totals and result counts for arrays, or reject scalar operator values in JSON mode even when CSV succeeds. The 117 model-casting contracts and 27 database differential cases execute the actual Mongoose models and TypeScript callbacks against isolated records to verify these cases. The latter requires Node and the repository npm dependencies during integration tests; the deployed Go server does not use Node. Operator predicates remain inside administrator-only routes and cannot become top-level MongoDB commands.

Admin date ranges now use extended-query truthiness and deferred string coercion, including malformed-value errors after the missing-owner short circuit. ISO forms support normalized day overflow, 24:00, extended years, millisecond truncation and JavaScript time clipping. Legacy forms support numeric/month-name dates, comments, time and timezone names using V8 parsing rules. The 632 contracts execute TypeScript parseDateRange; 44 database differential cases now include date selection and CSV filtering. Parsing is verified for the current UTC deployment. Alternate local timezones are not covered. The V8 BSD license is retained in licenses/V8-LICENSE and copied into the runtime image. Dashboard cursors also use the shared date-string parser.

The dashboard cursor contracts apply the date corpus to anonymizeDate, descending lastView and options.expirationDate. HTTP tests retain the original cursor checks and exercise representative legacy/array dates. Boundary tests verify that normalized ISO, named and array dates select the same next repository without duplicating the previous record. Numeric, null and boolean date conversions retain their existing behavior. Conference form dates use the shared JavaScript Date constructor behavior; validation, persistence and billing contracts cover the supported inputs.

Transformed-path indexing validates pending recovered metadata before publication. Valid rows complete recovery and rebuild under the new metadata revision; stale recovery snapshots are rejected with repository_changed. The cache namespace is versioned to discard indexes built without these checks. Regression tests cover both cases through the non-GitHub provider fixture. Compatibility means preserving supported behavior, not reproducing TypeScript bugs.

Gist and pull-request source previews now use the same connection-query validation as repository previews. Repeated and object-shaped connection choices return invalid_connection before provider access. Custom provider scalar connection names remain supported. The remaining direct incoming query reads select names on generic provider routes; legacy GitHub routes use path parameters. See reports/query-consumer-audit.json.

Persisted-date checks distinguish valid timestamps from absent or malformed values. Year-0001 dates remain usable for stale-record recovery, refresh, content generation identity, conference transition replay, job timestamps, and source commit dates. Repository lastUpdateDate selects the latest valid stored date without using year 0001 as a lower bound. Missing or invalid refresh timestamps do not trigger automatic recovery.

Replacement-heavy ASCII transforms now avoid full-file lowercase copies and repeated output classification. A monotonic case-insensitive finder retains ordered replacement and uses a bounded fallback for dense false candidates. Three-run paired medians improved the 50 MB sparse case from 592 to 270 ms and the 22 MB dense case from 394 to 190 ms, with roughly 50% and 38% lower allocated bytes. A separate same-machine Node comparison measured Go at 346 versus 488 ms and 226 versus 387 ms. Absent-term timings varied and were slower than the Go baseline in the paired run; these are synthetic transform measurements, not HTTP throughput claims. See reports/replacement-path-validation.json and its linked benchmark reports.

GitHub permission freshness

GITHUB_PERMISSION_CACHE_SECONDS defaults to 0 (check GitHub each time) and accepts 0–3600. The beta App overlay sets 3600. Successful repository access/visibility checks can be reused until that fixed deadline; hits never extend it, and failures are not cached. This permits GitHub permission changes without a processed webhook to take up to an hour to become visible on beta. Local account status, credential revocation/reconciliation and installation blocks are still checked on every access. Installation webhook revisions, changed user tokens and binding revisions select new cache entries. The process-local cache holds at most 10,000 hashed keys and clears on restart or capacity; installation tokens retain their own expiry and scope validation. This setting does not cache the full authorization result or change other providers.

Directory counts and website file lookup

Folder counts now group file rows by original directory in MongoDB, accumulate ancestor totals, then anonymize each distinct folder once. Empty files still count; directory rows do not. Counts for colliding anonymized folders are added together.

Website file requests first use a targeted MongoDB lookup for unchanged, mask-free paths without custom replacements. That path verifies the anonymized name, duplicate originals, descendants and the active snapshot. Successful ordinary-path proofs are cached for up to ten minutes in a bounded 1,000-entry process cache keyed by the repository path revision; cache hits still validate the live snapshot. Other file requests use the revision-bound Redis path index rather than scanning and anonymizing the whole file tree on every read. Index v5 records duplicate anonymized paths and parents containing colliding children, so /w/ continues to reject ambiguous paths even across index-build batches. The legacy file endpoint retains its existing first-match behavior. Website lookups validate the repository snapshot on cache hits, rebuild after eviction and settings changes, and retain provider authorization before content delivery. The first lookup of a masked/custom path can still pay the index-build cost; directory/index-page rendering retains the full listing fallback.

ZIP memory

MongoDB connections default to Snappy wire compression. The driver's zstd encoder pool can retain hundreds of MiB of windows on machines with many CPUs, including during per-entry ZIP authorization queries. An explicit compressors option in the MongoDB URI overrides the default. ZIP payloads still use Deflate.

ZIP downloads read projected file metadata in batches of 128, with no file-count truncation. They reuse one Deflate writer and one 32 KiB copy buffer. File bodies and transformed results remain disk-backed; the ZIP central directory and duplicate-name set still grow with the number of entries. Each entry retains content authorization and revision checks. Cursor errors and cancellation abort the response rather than finalizing an incomplete archive.

Streaming upstream archives into ZIP downloads

GitHub ZIP downloads now use one commit-pinned tarball request, consumed incrementally. provider.Archives is an optional capability; other providers and uploaded archives retain the existing per-file path. The GitHub adapter permits HTTPS redirects only to GitHub API/codeload hosts and strips the API credential on the codeload redirect. Its body deadline is 30 minutes, including client backpressure.

Three stages download and verify an entry, transform it, and compress its output into the client ZIP. Two unbuffered handoffs bound the pipeline to one downloading input, one transforming entry and one output being sent. Slow clients stop further preparation. The existing eight-transfer admission limit and worker memory limits still apply. Source entries use a reused 32 KiB copy buffer and temporary disk files, never a whole-repository spool. Regex transformation still requires an individual complete text file; it does not run on arbitrary byte chunks. ZIP central-directory and duplicate-path metadata grow with the entry count.

Git blob hashes are verified before transformation. Symlinks become regular files containing their Git blob target text; links are never followed. LFS pointers, export-subst differences and export-ignore omissions use the normal provider file path to preserve output. Missing metadata from truncated trees is resolved against the same commit. Output authorization and repository revisions are checked after preparation and before every entry is released. Cancellation, corrupt gzip/truncated input, invalid paths and duplicate anonymized names abort the response and clean up both stages; a corrupt stream is never finalized as a valid ZIP.

The integration test deliberately pauses the upstream before the second file and requires the first ZIP payload to reach the client before releasing it. It also checks cancellation, temporary-file cleanup, corrupted input, repository revocation, archive export exceptions, and zero blob requests for an ordinary streamed archive. This path requests an upstream archive on each download; the earlier local ZIP memory benchmark does not measure GitHub network latency.

ZIP requests have a 30-minute overall deadline. Other API requests retain their two-minute deadline. ZIP socket writes renew a one-minute write deadline on every write/flush, so an active slow transfer can outlive the HTTP server's ordinary two-minute write timeout while a stalled socket still expires. Tests cover an expired server write deadline between ZIP entries and a client that stops reading entirely. Stream failures now log a fixed stage and reason with the request ID, distinguishing client cancellation, request expiry, I/O timeouts and preparation failures without exposing source paths, URLs or credentials.

Anonymization workers now release execution slots and memory reservations as soon as the child has exited and its output is on disk, including cache hits. Output file handles remain valid until the caller's cleanup. A slow reader therefore cannot hold the only worker slot and cause another transform to hit the 30-second admission timeout. Request transfer limits and the ZIP pipeline's unbuffered handoffs still bound outstanding output files. The worker test keeps multiple cold/cached outputs open with one configured worker and verifies that both execution and memory admission are available before cleanup.

Authenticated browser sessions persist for 30 days and renew on activity at most once per day, giving a 29–30 day idle lifetime. Redis expiry and the Secure/HttpOnly/SameSite=Lax cookie are renewed together. Existing valid session cookies upgrade on their next authenticated request. Renewal uses compare-and-set so it cannot resurrect a logged-out session or overwrite concurrent session changes. Account status is still checked on every request. Anonymous OAuth sessions remain browser-session cookies with a 24-hour Redis lifetime; OAuth state still expires after ten minutes. Preserve the session secret and Redis volume across deployments.

README.md Unescape Escape

Go backend