Commit Graph
351 Commits
Author SHA1 Message Date
tdurieux f0fdd9250b fix: improve pdf rendering and html 2026-08-06 14:35:21 -07:00
Thomas DurieuxandGitHub debd83c079 feat: add optional MongoDB replica deployment (#765) 2026-07-30 02:28:19 +02:00
Thomas DurieuxandGitHub f040557ae8 fix: preserve state across backend updates (#756) 2026-07-22 04:29:58 +02:00
Thomas DurieuxandGitHub beef8387ae fix: close account authorization gaps (#755) 2026-07-22 04:29:04 +02:00
Thomas DurieuxandGitHub aab6ccf7fc fix: harden anonymization privacy paths (#754) 2026-07-22 04:28:01 +02:00
5ef10dee9e fix: dashboard menu clickability + expiration date UX (#753)
* fix: dashboard menu clickability + expiration date UX

Dashboard actions menu:
- Inactive (expired/removed) rows dimmed via their cells instead of the
  row, so `opacity` no longer creates a stacking context that trapped the
  actions dropdown beneath later rows and made its items unclickable.
- Add an "Extend 6 months" menu item for expired repos/PRs/gists.

Expiration form (anonymize):
- Fix the "After , the content will be removed." blank date: guard the
  helper text and add min/max validation feedback so an invalid pick no
  longer nulls the model into a broken sentence.
- Add a `min` (today) so past dates can no longer be selected, and
  compute min/max from local date parts (not UTC) to avoid a timezone
  off-by-one in the native picker.
- Default expiration is now 6 months (single source of truth, removing a
  latent double-offset bug); max stays at 1 year.
- Block submitting a missing/out-of-range expiration date.

Backend:
- New POST /:id/extend endpoint for repos, PRs and gists that pushes the
  expiration +6 months and re-anonymizes so expired items come back
  online, mirroring the refresh flow. Shared extendExpirationDate helper.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: add frontend translation for invalid_status error code

The new /extend endpoints throw an "invalid_status" AnonymousError, which
the error-code coverage test requires to have a locale entry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 14:10:38 +02:00
6a820408a1 fix: rate-limit visitors by real IP regardless of Cloudflare proxy hops (#750)
Since the night of 2026-07-16 production keyed the rate limiter on
Cloudflare edge IPs instead of visitor IPs: every visitor routed through
the same edge shared one 175-req/15min bucket, causing widespread 429s
(confirmed by probing: sequential requests alternated 200/429 across
fresh and exhausted buckets).

Root cause: 'trust proxy' used a fixed hop count (TRUST_PROXY=1), which
silently breaks whenever the proxy chain gains or loses an
X-Forwarded-For entry (e.g. a Cloudflare-side change).

- TRUST_PROXY now accepts a comma-separated subnet list; the new default
  'loopback,uniquelocal,cloudflare' expands Cloudflare's published IP
  ranges so Express skips trusted proxies no matter how many entries
  they add. Plain numbers keep the legacy hop-count behavior.
- If resolution still stops at a Cloudflare address (visitor missing
  from X-Forwarded-For entirely), the limiter key falls back to
  cf-connecting-ip — safe because request.ip can only be a Cloudflare
  address when the whole chain to it is trusted.
- CIDR matching uses Node's built-in net.BlockList (no new dependency).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 06:44:21 +02:00
tdurieux 1c1993f972 fix: try to fix gist 2026-07-02 14:53:26 +03:00
Thomas DurieuxandGitHub 839582c657 Fix .bat anonymization, truncated-tree misses, submodule warning, account deletion (#742)
* fix: anonymize Windows batch scripts (#735)

mime-types maps .bat to application/x-msdownload, the same MIME type as
.exe/.dll, so batch scripts were classified as binary and streamed
through without any anonymization. Special-case .bat/.cmd as text before
the MIME lookup, keeping .exe/.dll binary.

* fix: recover files missing from truncated tree listings (#738)

GitHub truncates tree listings of very large repositories. Folders whose
listing was truncated are recorded in truncatedFolders, but files that
fell outside the listing never reached the database, so requesting them
returned 404 file_not_found even though they exist on GitHub — and a
force refresh could not help.

When a file lookup misses and its directory is under a truncated folder,
fetch the file metadata directly from GitHub's contents API (object
media type, so it works past the 1MB inline limit), cache it in the
database, and serve it normally.

* feat: warn when a repository uses git submodules (#737)

GitHub archives and tree listings never include submodule contents, so
submodules end up as empty folders in the anonymized repository, which
surprises users. Detect a root .gitmodules file and show a warning
banner in the explorer explaining that submodule contents are not
included.

* feat: allow users to delete their account (#741)

Add DELETE /api/user: removes all anonymized repositories, gists, and
pull requests owned by the user, best-effort revokes the GitHub OAuth
grant, and scrubs personal data (username, emails, tokens, GitHub id,
photo) from the user record. The record itself is kept with a
placeholder username so removed repoIds stay reserved and owner
references remain resolvable.

The settings page gains an Account section with a confirmed delete
button.

* fix: add missing error translations for token_expired and job_is_active

The error-code coverage test failed because both backend codes had no
frontend translation.
2026-07-02 13:35:48 +02:00
e4ffd74068 Security hardening + gist UI fixes (#731)
* security: harden against XSS, ReDoS, path traversal, and injection

Defensive fixes across the server, storage, and viewer:

- XSS (CWE-79): sanitise rendered notebooks with DOMPurify, escape file
  names interpolated into AngularJS expressions (escapeNgString), set
  Mermaid securityLevel to 'strict', and stop urlRel2abs from returning
  javascript:/vbscript:/data:text/html URLs.
- Path traversal / zip-slip (CWE-22/23/24): validate URL-derived path
  components before they reach the storage layer (file/webview routes +
  StorageBase.assertSafePath) and sanitise zip entry names on extract for
  both the filesystem and S3 backends.
- ReDoS (CWE-1333): escape anonymization terms with catastrophic
  backtracking shapes to literals instead of compiling them as regexes.
- Secret hardening (CWE-798): require SESSION_SECRET / OAuth creds / DB
  password in production, random dev SESSION_SECRET fallback.
- Rate-limit spoofing (CWE-290): derive request.ip via trust-proxy hop
  count instead of the client-settable cf-connecting-ip header.
- NoSQL injection (CWE-943): allow only plain field paths as admin sort keys.
- Reject malformed streamer requests missing required string fields.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ui): make gists reachable/visible and clarify the ZIP button

- Gist & PR routes now accept a trailing slash (/gist/:id/:path*?), so the
  dashboard links (which end in "/") resolve to the gist/PR page instead of
  falling through to the 404 route (#725).
- Gist viewer picks the default tab after content loads, defaulting to
  "files" when files exist; previously the ng-init ran before the async
  load and a files-only gist rendered blank under the hidden comments tab.
- Explorer toolbar: relabel ZIP to "Full repo ZIP" with a tooltip, and add
  tooltips to Raw/Download clarifying they apply to the current file (#721).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: report SAML-enforced orgs clearly instead of "token expired"

When a repo's organization enforces SAML SSO, GitHub returns a 403 whose
message differs from the OAuth-App-restriction case. That 403 fell through
to the generic handler and surfaced as "token_expired", pushing users to
re-login when the real fix is authorizing their token for the org. Detect
the "SAML enforcement" message and raise a dedicated, actionable error
instead (#379, #550).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* security: catch nested quantified groups in ReDoS guard and backslash path traversal

- hasCatastrophicBacktracking now scans across nested parens ([\s\S]*?)
  so shapes like ((a+))+ are detected; comment reframed as a heuristic
  backstop rather than a proof.
- file route path-traversal check now rejects backslash separators and a
  leading backslash, covering Windows-style "..\" payloads (CWE-22/25).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore(dev): track dev-proxy script, ignore .DS_Store and .claude/

scripts/dev-proxy.js is referenced by the "dev:ui" npm script but was
never committed, breaking the command on a fresh clone. Add it and
ignore local-only macOS/Claude Code files.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 13:50:55 +02:00
tdurieux 39fadd6cf0 fix: auth issue & profile save issue 2026-05-18 16:04:09 +03:00
tdurieux e73ad48115 fix: file download large repo 2026-05-12 21:00:47 +03:00
tdurieux 898f18919e fix rate limit 2026-05-12 20:51:08 +03:00
tdurieux 427e26062e fix 2026-05-12 20:34:32 +03:00
tdurieux 03e18fd572 repo change + daily stat improvements 2026-05-11 12:10:17 +03:00
tdurieux 3eeed23609 handle memory issues 2026-05-07 21:01:07 +03:00
tdurieux 369fd8edb2 redis cache 2026-05-07 15:55:28 +03:00
tdurieux b37a814f3a improve queue 2026-05-07 14:58:36 +03:00
tdurieux f817a29a4b loading improvements 2026-05-07 08:30:31 +03:00
tdurieux 2de08c3df3 Add missing error handlers on stream pipelines
- AnonymizedFile.anonymizedContent(): propagate content errors to the
  anonymizer so callers see the failure instead of hanging.
- AnonymizedFile.send() local path: add error handler on the anonymizer
  transform between content and response pipes.
- S3.send(): handle errors on the S3 body stream to avoid unhandled
  emits crashing the process.
- S3.archive() / FileSystem.archive(): propagate read-stream errors
  to the file transformer so archiver sees the failure.
- Add frontend translations for new error codes.
2026-05-07 07:47:29 +03:00
tdurieux 7a163f2d35 Fix streamer crash and misclassified transient GitHub errors
Add missing error handler on the anonymizer transform stream in the
streamer route — without it, an upstream error tears down the pipe and
the anonymizer emits an unhandled error that crashes the process
(surfacing as ECONNRESET to the main server).

Classify transient network errors (ReadError, ECONNRESET, ETIMEDOUT)
as upstream_error/502 instead of file_not_found/404 so they are
distinguishable in logs and don't cache-poison downstream.

Update handleError tests to match the existing sanitization behavior
that returns internal_error for non-AnonymousError instances.
2026-05-07 07:44:15 +03:00
tdurieux 4ab8e0d1cd Handle GitHub 409 "repository is empty" error in getCommitInfo
When a GitHub repo has no commits, the API returns 409 which was
unhandled, causing raw HttpError warnings. Now throws repo_empty
AnonymousError consistent with the existing convention.
2026-05-07 07:42:05 +03:00
tdurieux fbbc694747 improve styling 2026-05-07 07:34:30 +03:00
tdurieux e59527bc78 Remove all user repositories when banning
Use removeQueue instead of cacheQueue so each repo transitions to
REMOVING status and is fully deleted, not just cache-cleared.
2026-05-07 06:04:55 +03:00
tdurieux 9292c19392 Handle GitHub 422 errors as commit_not_found and sanitize error responses 2026-05-07 05:54:51 +03:00
tdurieux 9403f15ac3 Align error logging with admin dashboard field conventions
All warn/error log calls now use field names the dashboard's decorate()
function recognizes: `code` for the error code pill, `httpStatus` for the
status badge and severity bucket, `url` for the sidebar link, and
`repoId` for the repository link.

Key changes:
- Streamer errors surface code, httpStatus, url, and nested err in Raw tab
- Nested `{ err: serializeError(e) }` replaced with spread pattern so
  error fields (name, message, status) appear at the top level
- Raw Error objects in catch blocks now go through serializeError()
- Rate limit, token, and PR 404 warnings include code + httpStatus
- Dashboard stack walker traverses both `cause` and `err` chains
- Dashboard Raw tab renders repoId, filePath, upstream*, err, and cause
- trimRawArg recursively trims stacks in nested err/cause chains
- clampPayload strips heavy nested fields before falling back to
  truncated placeholder, preserving flat diagnostic fields
2026-05-07 05:54:18 +03:00
tdurieux b8cfe293ea Fix BullMQ "Custom Id cannot be integers" error by prefixing jobId 2026-05-07 05:53:26 +03:00
tdurieux 8fc7ac5175 Add user ban/activate feature
Add admin endpoints to ban and activate users, block banned users
from all auth flows (OAuth, token login, bearer auth), and invalidate
existing sessions on next request. Includes frontend translation and
user detail page ban/activate buttons.
2026-05-07 05:41:12 +03:00
tdurieux d9104c2ec2 Update commit on branch refresh and validate commit exists on save
Refresh button now always updates the commit to the latest SHA instead
of preserving the stale one in edit mode. Both create and update routes
verify the commit still exists on GitHub before persisting.
2026-05-06 21:14:53 +03:00
tdurieux bd8656206a fix persistance bugs 2026-05-06 20:00:59 +03:00
tdurieux 67cb2538b1 fix old github download repos 2026-05-06 19:37:16 +03:00
tdurieux da78708b7b Improve error handling 2026-05-06 18:43:36 +03:00
tdurieux aae6eae6eb handle rate limit 2026-05-06 17:50:01 +03:00
tdurieux 6bad6c2f09 fix bugs and report better errors 2026-05-06 17:26:47 +03:00
tdurieux 3b27816702 fix incremental 2026-05-06 17:12:58 +03:00
tdurieux 804bbffb7a Improve error handling 2026-05-06 17:03:19 +03:00
tdurieux 48e782946a Improve error handling 2026-05-06 16:56:07 +03:00
tdurieux cf2f172aca fix(gist): set gist subpaths individually to avoid CastError
Mongoose treats `gist` as a nested path, not a sub-schema, so
set("gist", payload) mis-casts the inner subdoc arrays and fails
validation with 'Cast to [string] failed' at gist.files.0. Set each
subpath individually so the files/comments arrays cast correctly.
2026-05-06 16:52:48 +03:00
tdurieux dcb524c8c1 Improve error handling 2026-05-06 16:45:22 +03:00
tdurieux 3613c895c8 improve logging 2026-05-06 16:31:10 +03:00
tdurieux 873c910dd3 Improve error dashboard 2026-05-06 16:12:37 +03:00
tdurieuxandClaude Opus 4.7 6f418d6332 log AnonymousError context detail in serialized output
Surface the path/repo/url tied to an AnonymousError when it gets
serialized for logging — previously logs only carried name, message,
and httpStatus, which made file_not_found entries impossible to trace
back to a specific file or repo. Extract the existing detail formatting
out of toString() into a public detail() method, harden it against
AnonymizedFile getters that can throw, and have serializeError include
the result as a "detail" field.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-06 13:46:06 +03:00
tdurieux 06a098fba7 perf(deploy): faster builds and zero-downtime streamer rollover
- Multi-stage Dockerfile with BuildKit npm cache mounts and a separate
  prod-deps stage so source edits don't reinstall or prune.
- Tighter .dockerignore to shrink build context.
- Healthchecks: add start_period and tighten interval/retries so
  containers report healthy as soon as the process is actually ready
  instead of after a full polling interval.
- Move recoverStuckPreparing() off the startup critical path; the
  recovery sweep now runs in the background after app.listen.
- depends_on uses condition: service_healthy and the obsolete
  compose 'version' key is gone.
- New scripts/build.sh + scripts/deploy.sh: deploy.sh builds, exits
  early if the image is unchanged, runs a blue/green streamer swap
  (scale to 2N, wait healthy in parallel, drop olds), then recreates
  the API with --no-deps to avoid compose's depends_on re-poll.
2026-05-06 13:38:19 +03:00
tdurieux 371693dc3b fix: end stream for files <= 150 bytes in resolveLfsPointer
The LFS-pointer probe buffered up to 150 bytes before deciding whether
to forward the blob or swap to the raw URL. For blobs that fit entirely
in the probe, decide() ran from the source's end event and attached
data/end listeners to an already-ended stream, so out.end() was never
called. The response hung until upstream timed out and storage.write
left an incomplete cached copy, which then forced a re-fetch on every
subsequent read.

Pass a sourceEnded flag through decide() and end the output directly
when the source has already finished. Also skip the GitHub blob fetch
when the tree size is already over MAX_FILE_SIZE, surfacing
file_too_big instead of a translated 422.
2026-05-06 13:10:25 +03:00
tdurieux 7dd6d872e9 Improve error handling 2026-05-06 12:44:07 +03:00
tdurieux e1cf8e3a00 Improve error handling 2026-05-06 12:26:12 +03:00
tdurieux c4182b5b2a fix: prevent E11000 on concurrent getRepositoryFromGitHub
Replace check-then-insert with atomic findOneAndUpdate upsert keyed on
externalId, plus a single E11000 retry fallback. Eliminates the duplicate
key race when two requests resolve the same gh_<id> concurrently.
2026-05-06 12:22:42 +03:00
tdurieux c2d43164d0 error logging improvement, regex fix 2026-05-06 11:16:12 +03:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>tdurieuxThomas Durieux
e34f45522f chore(deps): bump uuid and bullmq (#691)
* chore(deps): bump uuid and bullmq

Removes [uuid](https://github.com/uuidjs/uuid). It's no longer used after updating ancestor dependency [bullmq](https://github.com/taskforcesh/bullmq). These dependencies need to be updated together.


Removes `uuid`

Updates `bullmq` from 2.4.0 to 5.76.5
- [Release notes](https://github.com/taskforcesh/bullmq/releases)
- [Commits](https://github.com/taskforcesh/bullmq/compare/v2.4.0...v5.76.5)

---
updated-dependencies:
- dependency-name: bullmq
  dependency-version: 5.76.5
  dependency-type: direct:production
- dependency-name: uuid
  dependency-version: 
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix(bullmq): adapt isRunning + getJobs typing for v5 API

Worker.isRunning became a method (was a property in v2), and
Queue.getJobs now requires a mutable JobType[] (was string[]).

* clean up

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: tdurieux <durieuxthomas@hotmail.com>
Co-authored-by: Thomas Durieux <5577568+tdurieux@users.noreply.github.com>
2026-05-06 08:40:12 +02:00
tdurieuxandClaude Opus 4.7 7e0f900c7a fix: resolve eslint unused-var and useless-assignment warnings
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-06 08:22:45 +03:00