Commit Graph
23 Commits
Author SHA1 Message Date
Thomas Durieux 6434337923 perf: bound request work and resource usage (#866)
* perf: bound request work and resource usage

* fix: address performance review regressions

* fix: guard content expiration and dashboard state

* fix: tolerate cache outages and retire derived paths

* fix: revalidate expiration claims and status pagination

* fix: retry interrupted repository metadata cleanup

* fix: retire obsolete content cache storage

* fix: activate tree metadata and recheck expiration claims

* fix: fence cache publication across expiration

* fix: recover staged trees and enforce cache lifecycle cleanup

* fix: prune expired transforms during database outages

* fix: finish conference removals and reload expiration candidates
2026-10-08 03:38:19 +01:00
Thomas Durieux e5dac99598 fix: distinguish upstream stream failures from storage errors 2026-10-07 12:24:23 +00:00
tdurieux 6777537d44 fix: restrict S3 deletion to exact keys and descendants 2026-09-06 09:18:37 +02:00
tdurieux 5e84bfaa5b fix: surface partial S3 object deletion failures 2026-09-06 09:18:32 +02:00
Thomas DurieuxandClaude Opus 4.8 e4ffd74068 Security hardening + gist UI fixes (#731)
* security: harden against XSS, ReDoS, path traversal, and injection

Defensive fixes across the server, storage, and viewer:

- XSS (CWE-79): sanitise rendered notebooks with DOMPurify, escape file
  names interpolated into AngularJS expressions (escapeNgString), set
  Mermaid securityLevel to 'strict', and stop urlRel2abs from returning
  javascript:/vbscript:/data:text/html URLs.
- Path traversal / zip-slip (CWE-22/23/24): validate URL-derived path
  components before they reach the storage layer (file/webview routes +
  StorageBase.assertSafePath) and sanitise zip entry names on extract for
  both the filesystem and S3 backends.
- ReDoS (CWE-1333): escape anonymization terms with catastrophic
  backtracking shapes to literals instead of compiling them as regexes.
- Secret hardening (CWE-798): require SESSION_SECRET / OAuth creds / DB
  password in production, random dev SESSION_SECRET fallback.
- Rate-limit spoofing (CWE-290): derive request.ip via trust-proxy hop
  count instead of the client-settable cf-connecting-ip header.
- NoSQL injection (CWE-943): allow only plain field paths as admin sort keys.
- Reject malformed streamer requests missing required string fields.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ui): make gists reachable/visible and clarify the ZIP button

- Gist & PR routes now accept a trailing slash (/gist/:id/:path*?), so the
  dashboard links (which end in "/") resolve to the gist/PR page instead of
  falling through to the 404 route (#725).
- Gist viewer picks the default tab after content loads, defaulting to
  "files" when files exist; previously the ng-init ran before the async
  load and a files-only gist rendered blank under the hidden comments tab.
- Explorer toolbar: relabel ZIP to "Full repo ZIP" with a tooltip, and add
  tooltips to Raw/Download clarifying they apply to the current file (#721).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: report SAML-enforced orgs clearly instead of "token expired"

When a repo's organization enforces SAML SSO, GitHub returns a 403 whose
message differs from the OAuth-App-restriction case. That 403 fell through
to the generic handler and surfaced as "token_expired", pushing users to
re-login when the real fix is authorizing their token for the org. Detect
the "SAML enforcement" message and raise a dedicated, actionable error
instead (#379, #550).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* security: catch nested quantified groups in ReDoS guard and backslash path traversal

- hasCatastrophicBacktracking now scans across nested parens ([\s\S]*?)
  so shapes like ((a+))+ are detected; comment reframed as a heuristic
  backstop rather than a proof.
- file route path-traversal check now rejects backslash separators and a
  leading backslash, covering Windows-style "..\" payloads (CWE-22/25).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore(dev): track dev-proxy script, ignore .DS_Store and .claude/

scripts/dev-proxy.js is referenced by the "dev:ui" npm script but was
never committed, breaking the command on a fresh clone. Add it and
ignore local-only macOS/Claude Code files.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 13:50:55 +02:00
tdurieux 2de08c3df3 Add missing error handlers on stream pipelines
- AnonymizedFile.anonymizedContent(): propagate content errors to the
  anonymizer so callers see the failure instead of hanging.
- AnonymizedFile.send() local path: add error handler on the anonymizer
  transform between content and response pipes.
- S3.send(): handle errors on the S3 body stream to avoid unhandled
  emits crashing the process.
- S3.archive() / FileSystem.archive(): propagate read-stream errors
  to the file transformer so archiver sees the failure.
- Add frontend translations for new error codes.
2026-05-07 07:47:29 +03:00
tdurieux 9403f15ac3 Align error logging with admin dashboard field conventions
All warn/error log calls now use field names the dashboard's decorate()
function recognizes: `code` for the error code pill, `httpStatus` for the
status badge and severity bucket, `url` for the sidebar link, and
`repoId` for the repository link.

Key changes:
- Streamer errors surface code, httpStatus, url, and nested err in Raw tab
- Nested `{ err: serializeError(e) }` replaced with spread pattern so
  error fields (name, message, status) appear at the top level
- Raw Error objects in catch blocks now go through serializeError()
- Rate limit, token, and PR 404 warnings include code + httpStatus
- Dashboard stack walker traverses both `cause` and `err` chains
- Dashboard Raw tab renders repoId, filePath, upstream*, err, and cause
- trimRawArg recursively trims stacks in nested err/cause chains
- clampPayload strips heavy nested fields before falling back to
  truncated placeholder, preserving flat diagnostic fields
2026-05-07 05:54:18 +03:00
tdurieux da78708b7b Improve error handling 2026-05-06 18:43:36 +03:00
tdurieux dcb524c8c1 Improve error handling 2026-05-06 16:45:22 +03:00
tdurieux c2d43164d0 error logging improvement, regex fix 2026-05-06 11:16:12 +03:00
tdurieuxandClaude Opus 4.7 f413a30313 fix(cache): make Zip-source caches atomic and robust to partial state
Follow-up to the GitHubStream cache fixes. The same poisoned-cache
class existed in the GitHubDownload path and a few related spots:

- GitHubDownload.download: wipe pre-existing state before extracting
  and write a .anon-complete marker only after a successful extract.
  On error, rm the partial cache so a retry starts clean. getFileContent
  and getFiles now gate on the marker instead of "any file/folder
  exists," so a half-extracted tree can never be served as canonical.
- GitHubDownload.getFileContent: validate cached file size against the
  upstream FileModel size (via the new AnonymizedFile.size()), same
  guard as GitHubStream. getFiles filters the marker from the listing.
- FileSystem.listFiles: drop the bogus stats.ino.toString() as sha.
  An inode isn't a content hash; anything comparing it to a Git blob
  sha would silently disagree. Leave undefined.
- S3.write: remove the fire-and-forget data.on("error") -> this.rm(...).
  Multipart Upload doesn't commit partial objects, so there was nothing
  to clean up, and the handler raced retries and could delete a
  previously-good object on a transient source-stream hiccup. The
  size-validated read path recovers from any other undersized objects.
- GitHubStream.resolveLfsPointer: drop the post-decision early-return
  in blobStream.on("error"). Currently redundant with the inner
  listener, but removes the future-refactor footgun.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-05 08:54:42 +03:00
tdurieuxandClaude Opus 4.7 9adff11e74 fix(cache): atomic file writes and size-validated cache reads
A failed/interrupted GitHub fetch could leave a 0-byte or truncated
file in the local cache. Subsequent reads happily streamed the empty
content as the file's body — visible to users as an "Empty file" with
HTTP 200. Reproduced on artifact-70B6/Lethe/configs.py (#694).

- FileSystem.write: stream into a sibling .tmp and rename into place
  only on finish. Stream errors discard the tmp and leave any prior
  cached file untouched. Drop the utf-8 encoding that was silently
  corrupting binary blobs.
- GitHubStream.getFileContentCache: accept an expected size and treat
  cached.size < expected as a poisoned cache (truncated fetch) → rm
  and re-fetch. cached.size >= expected is accepted, which keeps
  Git LFS-resolved files (whose FileModel.size is the pointer size)
  working.
- AnonymizedFile: expose size() and pass it through to the streamer
  alongside sha so the cache check has the upstream size.

Existing poisoned entries self-heal on next access.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-05 08:47:41 +03:00
Thomas Durieux 188066e91d Fix 9 bugs and add 103 tests for core anonymization, config, and routing (#669) 2026-04-15 09:41:00 +02:00
Thomas Durieux 8198a4b44a Standardize error responses with consistent format and human-readable messages (#667) 2026-04-15 09:27:08 +02:00
Thomas Durieux f4209110c7 Fix all 93 ESLint issues (3 errors, 90 warnings) (#666) 2026-04-15 09:04:22 +02:00
Thomas Durieux 655ae92c4c Remove OpenTelemetry tracing infrastructure (#662) 2026-04-15 04:39:08 +02:00
Thomas Durieux f3641c8ce3 Set up CI with ESLint linter and Mocha test runner (#661) 2026-04-15 04:34:03 +02:00
tdurieux b0fa5e6689 fix: hot fix, replace repoID by repoId 2024-04-26 12:40:56 +01:00
tdurieux 710f7328e7 feat: flatten file tree for better performance 2024-04-26 10:32:09 +01:00
tdurieux 1d4bab7866 fix: fix webview & improve download progress 2024-04-03 18:25:33 +01:00
tdurieux 83c55fdfbf fix: typo 2024-04-03 13:27:05 +01:00
tdurieux db67f53b2c fix: fix GitHubDownload 2024-04-03 13:24:34 +01:00
tdurieux 4d12641c7e feat: introduce streamers that handle the stream and anonymization from github 2024-04-03 11:13:01 +01:00