Commit Graph
20 Commits
Author SHA1 Message Date
Thomas Durieux 7aa9557905 perf: cut MongoDB scans in repository workflows (#780)
* Add repository name index

* Optimize repository file counts

* Optimize MongoDB query workloads
2026-08-20 12:29:26 +02:00
Thomas Durieux 839582c657 Fix .bat anonymization, truncated-tree misses, submodule warning, account deletion (#742)
* fix: anonymize Windows batch scripts (#735)

mime-types maps .bat to application/x-msdownload, the same MIME type as
.exe/.dll, so batch scripts were classified as binary and streamed
through without any anonymization. Special-case .bat/.cmd as text before
the MIME lookup, keeping .exe/.dll binary.

* fix: recover files missing from truncated tree listings (#738)

GitHub truncates tree listings of very large repositories. Folders whose
listing was truncated are recorded in truncatedFolders, but files that
fell outside the listing never reached the database, so requesting them
returned 404 file_not_found even though they exist on GitHub — and a
force refresh could not help.

When a file lookup misses and its directory is under a truncated folder,
fetch the file metadata directly from GitHub's contents API (object
media type, so it works past the 1MB inline limit), cache it in the
database, and serve it normally.

* feat: warn when a repository uses git submodules (#737)

GitHub archives and tree listings never include submodule contents, so
submodules end up as empty folders in the anonymized repository, which
surprises users. Detect a root .gitmodules file and show a warning
banner in the explorer explaining that submodule contents are not
included.

* feat: allow users to delete their account (#741)

Add DELETE /api/user: removes all anonymized repositories, gists, and
pull requests owned by the user, best-effort revokes the GitHub OAuth
grant, and scrubs personal data (username, emails, tokens, GitHub id,
photo) from the user record. The record itself is kept with a
placeholder username so removed repoIds stay reserved and owner
references remain resolvable.

The settings page gains an Account section with a confirmed delete
button.

* fix: add missing error translations for token_expired and job_is_active

The error-code coverage test failed because both backend codes had no
frontend translation.
2026-07-02 13:35:48 +02:00
tdurieux b37a814f3a improve queue 2026-05-07 14:58:36 +03:00
tdurieux fbbc694747 improve styling 2026-05-07 07:34:30 +03:00
tdurieux b8cfe293ea Fix BullMQ "Custom Id cannot be integers" error by prefixing jobId 2026-05-07 05:53:26 +03:00
tdurieux da78708b7b Improve error handling 2026-05-06 18:43:36 +03:00
tdurieux dcb524c8c1 Improve error handling 2026-05-06 16:45:22 +03:00
tdurieux 3613c895c8 improve logging 2026-05-06 16:31:10 +03:00
tdurieux c2d43164d0 error logging improvement, regex fix 2026-05-06 11:16:12 +03:00
tdurieux f8c91ca0af multiple fixes 2026-05-05 10:32:31 +03:00
tdurieuxandClaude Opus 4.7 5b72b630c4 fix: silent-truncation, token-refresh, and content-type bugs across hot paths
Follow-up review pass after the cache fixes turned up several bugs in
the same family — silent failures that look like success to the client,
plus content-correctness issues in the ZIP and per-file delivery paths.

- zipStream: stop calling archive.finalize() on upstream/parser errors.
  That produced a valid-looking ZIP (200 OK, archive opens) silently
  missing entries — same class as #694, but worse because the user has
  no signal anything went wrong. Destroy the response on failure
  instead so the client sees a connection drop.
- zipStream: apply per-repo image/pdf gates inside the entry handler.
  The single-file /file/... endpoint refuses to serve those types
  via AnonymizedFile.isFileSupported when image=false / pdf=false, but
  the ZIP shipped them anyway — privacy-relevant for maintainers who
  toggle image=false to suppress identifying screenshots. Threaded
  contentOptions through both ZIP entry points (direct and streamer).
- GitHubUtils.getToken: validate the OAuth token-refresh response
  before persisting. On a non-2xx response or a body without a string
  token, we used to overwrite the stored token with `undefined`, which
  then propagated as `Authorization: token undefined` to every API
  call — 401 even on public repos, with the config.GITHUB_TOKEN
  fallback unreachable because the field was no longer falsy.
- AnonymizedFile.send (streamer branch): forward Content-Type from the
  upstream streamer response. got.stream(...).pipe(res) carries body
  bytes only, so the parent response had no Content-Type and browsers
  guessed (text rendered as download, etc.). Also resolve on
  res.on("finish") in addition to "close" — keep-alive sockets stay
  open long after the response is delivered, delaying countView().
- Repository.updateIfNeeded: persist a renamed source.repositoryName
  even when the commit hasn't changed. Previously the new value lived
  in memory only and was overwritten on the next reload, so the
  rename detection ran every request.
- Repository.anonymize: stop materialising a dummy {path:"",name:""}
  FileModel for empty repos. That row collided with the special case
  in AnonymizedFile.getFileInfo and surfaced in unfiltered listings.
- streamer/route POST /: reject filePath segments containing ".." or
  empty parts. Defence in depth — the parent server validates against
  FileModel before calling, but the streamer joins filePath straight
  into the storage path, so any future caller forwarding an
  unvalidated path could traverse out of the repo root.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-05 09:19:05 +03:00
tdurieux a5f66d6844 multiple fixes 2026-05-03 15:30:54 +02:00
Thomas Durieux 188066e91d Fix 9 bugs and add 103 tests for core anonymization, config, and routing (#669) 2026-04-15 09:41:00 +02:00
Thomas Durieux f4209110c7 Fix all 93 ESLint issues (3 errors, 90 warnings) (#666) 2026-04-15 09:04:22 +02:00
tdurieux dcf483ea03 feat: improve download anonymized repository 2024-05-06 11:52:32 +02:00
tdurieux 710f7328e7 feat: flatten file tree for better performance 2024-04-26 10:32:09 +01:00
tdurieux 61c6a79949 feat: check repo status before update 2024-04-11 15:25:45 +01:00
tdurieux 27c54b0182 feat: add link to website if enabled 2024-04-05 12:10:07 +01:00
tdurieux 42c3a58a46 feat: add edit button on repo view 2024-04-03 19:11:10 +01:00
tdurieux 4d12641c7e feat: introduce streamers that handle the stream and anonymization from github 2024-04-03 11:13:01 +01:00