* fix: anonymize Windows batch scripts (#735)
mime-types maps .bat to application/x-msdownload, the same MIME type as
.exe/.dll, so batch scripts were classified as binary and streamed
through without any anonymization. Special-case .bat/.cmd as text before
the MIME lookup, keeping .exe/.dll binary.
* fix: recover files missing from truncated tree listings (#738)
GitHub truncates tree listings of very large repositories. Folders whose
listing was truncated are recorded in truncatedFolders, but files that
fell outside the listing never reached the database, so requesting them
returned 404 file_not_found even though they exist on GitHub — and a
force refresh could not help.
When a file lookup misses and its directory is under a truncated folder,
fetch the file metadata directly from GitHub's contents API (object
media type, so it works past the 1MB inline limit), cache it in the
database, and serve it normally.
* feat: warn when a repository uses git submodules (#737)
GitHub archives and tree listings never include submodule contents, so
submodules end up as empty folders in the anonymized repository, which
surprises users. Detect a root .gitmodules file and show a warning
banner in the explorer explaining that submodule contents are not
included.
* feat: allow users to delete their account (#741)
Add DELETE /api/user: removes all anonymized repositories, gists, and
pull requests owned by the user, best-effort revokes the GitHub OAuth
grant, and scrubs personal data (username, emails, tokens, GitHub id,
photo) from the user record. The record itself is kept with a
placeholder username so removed repoIds stay reserved and owner
references remain resolvable.
The settings page gains an Account section with a confirmed delete
button.
* fix: add missing error translations for token_expired and job_is_active
The error-code coverage test failed because both backend codes had no
frontend translation.
Add missing error handler on the anonymizer transform stream in the
streamer route — without it, an upstream error tears down the pipe and
the anonymizer emits an unhandled error that crashes the process
(surfacing as ECONNRESET to the main server).
Classify transient network errors (ReadError, ECONNRESET, ETIMEDOUT)
as upstream_error/502 instead of file_not_found/404 so they are
distinguishable in logs and don't cache-poison downstream.
Update handleError tests to match the existing sanitization behavior
that returns internal_error for non-AnonymousError instances.
The LFS-pointer probe buffered up to 150 bytes before deciding whether
to forward the blob or swap to the raw URL. For blobs that fit entirely
in the probe, decide() ran from the source's end event and attached
data/end listeners to an already-ended stream, so out.end() was never
called. The response hung until upstream timed out and storage.write
left an incomplete cached copy, which then forced a re-fetch on every
subsequent read.
Pass a sourceEnded flag through decide() and end the output directly
when the source has already finished. Also skip the GitHub blob fetch
when the tree size is already over MAX_FILE_SIZE, surfacing
file_too_big instead of a translated 422.
Follow-up to the GitHubStream cache fixes. The same poisoned-cache
class existed in the GitHubDownload path and a few related spots:
- GitHubDownload.download: wipe pre-existing state before extracting
and write a .anon-complete marker only after a successful extract.
On error, rm the partial cache so a retry starts clean. getFileContent
and getFiles now gate on the marker instead of "any file/folder
exists," so a half-extracted tree can never be served as canonical.
- GitHubDownload.getFileContent: validate cached file size against the
upstream FileModel size (via the new AnonymizedFile.size()), same
guard as GitHubStream. getFiles filters the marker from the listing.
- FileSystem.listFiles: drop the bogus stats.ino.toString() as sha.
An inode isn't a content hash; anything comparing it to a Git blob
sha would silently disagree. Leave undefined.
- S3.write: remove the fire-and-forget data.on("error") -> this.rm(...).
Multipart Upload doesn't commit partial objects, so there was nothing
to clean up, and the handler raced retries and could delete a
previously-good object on a transient source-stream hiccup. The
size-validated read path recovers from any other undersized objects.
- GitHubStream.resolveLfsPointer: drop the post-decision early-return
in blobStream.on("error"). Currently redundant with the inner
listener, but removes the future-refactor footgun.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
A failed/interrupted GitHub fetch could leave a 0-byte or truncated
file in the local cache. Subsequent reads happily streamed the empty
content as the file's body — visible to users as an "Empty file" with
HTTP 200. Reproduced on artifact-70B6/Lethe/configs.py (#694).
- FileSystem.write: stream into a sibling .tmp and rename into place
only on finish. Stream errors discard the tmp and leave any prior
cached file untouched. Drop the utf-8 encoding that was silently
corrupting binary blobs.
- GitHubStream.getFileContentCache: accept an expected size and treat
cached.size < expected as a poisoned cache (truncated fetch) → rm
and re-fetch. cached.size >= expected is accepted, which keeps
Git LFS-resolved files (whose FileModel.size is the pointer size)
working.
- AnonymizedFile: expose size() and pass it through to the streamer
alongside sha so the cache check has the upstream size.
Existing poisoned entries self-heal on next access.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Files tracked by Git LFS used to come out as the pointer text:
version https://git-lfs.github.com/spec/v1
oid sha256:...
size ...
…because GitHub's blob API returns the pointer, not the resolved
content. Detect that prefix on the first ~150 bytes of the blob stream
and switch to a fresh fetch via the web raw URL
(github.com/<owner>/<repo>/raw/<commit>/<path>), which auto-redirects
to media.githubusercontent.com and resolves the LFS object — auth
header carries through. Non-LFS files are forwarded through the
existing pipeline unchanged.
Fixes#95.
Thomas Durieux