Commit Graph

640 Commits

Author SHA1 Message Date
tdurieux afd9f36cfb improve admin overview 2026-05-11 12:10:17 +03:00
tdurieux 03e18fd572 repo change + daily stat improvements 2026-05-11 12:10:17 +03:00
dependabot[bot] b03c4b437c chore(deps): bump fast-xml-builder from 1.1.5 to 1.2.0 (#707)
Bumps [fast-xml-builder](https://github.com/NaturalIntelligence/fast-xml-builder) from 1.1.5 to 1.2.0.
- [Changelog](https://github.com/NaturalIntelligence/fast-xml-builder/blob/main/CHANGELOG.md)
- [Commits](https://github.com/NaturalIntelligence/fast-xml-builder/compare/v1.1.5...v1.2.0)

---
updated-dependencies:
- dependency-name: fast-xml-builder
  dependency-version: 1.2.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-09 13:31:56 +02:00
tdurieux 3eeed23609 handle memory issues 2026-05-07 21:01:07 +03:00
tdurieux 369fd8edb2 redis cache 2026-05-07 15:55:28 +03:00
tdurieux b37a814f3a improve queue 2026-05-07 14:58:36 +03:00
tdurieux f817a29a4b loading improvements 2026-05-07 08:30:31 +03:00
tdurieux 2de08c3df3 Add missing error handlers on stream pipelines
- AnonymizedFile.anonymizedContent(): propagate content errors to the
  anonymizer so callers see the failure instead of hanging.
- AnonymizedFile.send() local path: add error handler on the anonymizer
  transform between content and response pipes.
- S3.send(): handle errors on the S3 body stream to avoid unhandled
  emits crashing the process.
- S3.archive() / FileSystem.archive(): propagate read-stream errors
  to the file transformer so archiver sees the failure.
- Add frontend translations for new error codes.
2026-05-07 07:47:29 +03:00
tdurieux 7a163f2d35 Fix streamer crash and misclassified transient GitHub errors
Add missing error handler on the anonymizer transform stream in the
streamer route — without it, an upstream error tears down the pipe and
the anonymizer emits an unhandled error that crashes the process
(surfacing as ECONNRESET to the main server).

Classify transient network errors (ReadError, ECONNRESET, ETIMEDOUT)
as upstream_error/502 instead of file_not_found/404 so they are
distinguishable in logs and don't cache-poison downstream.

Update handleError tests to match the existing sanitization behavior
that returns internal_error for non-AnonymousError instances.
2026-05-07 07:44:15 +03:00
tdurieux 4ab8e0d1cd Handle GitHub 409 "repository is empty" error in getCommitInfo
When a GitHub repo has no commits, the API returns 409 which was
unhandled, causing raw HttpError warnings. Now throws repo_empty
AnonymousError consistent with the existing convention.
2026-05-07 07:42:05 +03:00
tdurieux fbbc694747 improve styling 2026-05-07 07:34:30 +03:00
tdurieux e59527bc78 Remove all user repositories when banning
Use removeQueue instead of cacheQueue so each repo transitions to
REMOVING status and is fully deleted, not just cache-cleared.
2026-05-07 06:04:55 +03:00
tdurieux 9292c19392 Handle GitHub 422 errors as commit_not_found and sanitize error responses 2026-05-07 05:54:51 +03:00
tdurieux 9403f15ac3 Align error logging with admin dashboard field conventions
All warn/error log calls now use field names the dashboard's decorate()
function recognizes: `code` for the error code pill, `httpStatus` for the
status badge and severity bucket, `url` for the sidebar link, and
`repoId` for the repository link.

Key changes:
- Streamer errors surface code, httpStatus, url, and nested err in Raw tab
- Nested `{ err: serializeError(e) }` replaced with spread pattern so
  error fields (name, message, status) appear at the top level
- Raw Error objects in catch blocks now go through serializeError()
- Rate limit, token, and PR 404 warnings include code + httpStatus
- Dashboard stack walker traverses both `cause` and `err` chains
- Dashboard Raw tab renders repoId, filePath, upstream*, err, and cause
- trimRawArg recursively trims stacks in nested err/cause chains
- clampPayload strips heavy nested fields before falling back to
  truncated placeholder, preserving flat diagnostic fields
2026-05-07 05:54:18 +03:00
tdurieux b8cfe293ea Fix BullMQ "Custom Id cannot be integers" error by prefixing jobId 2026-05-07 05:53:26 +03:00
tdurieux 8fc7ac5175 Add user ban/activate feature
Add admin endpoints to ban and activate users, block banned users
from all auth flows (OAuth, token login, bearer auth), and invalidate
existing sessions on next request. Includes frontend translation and
user detail page ban/activate buttons.
2026-05-07 05:41:12 +03:00
tdurieux 48256e743c Style Ko-fi widgets to match light/dark theme
Floating button now initializes with theme-aware colors and updates
on toggle. Status page iframe uses a tuned CSS filter in dark mode
to blend with the warm palette.
2026-05-06 21:38:52 +03:00
tdurieux dfa5a2e2fd Fix repo link on admin errors page to point to repo view 2026-05-06 21:38:31 +03:00
tdurieux 1204eaffa9 Fix admin repository links and remove buttons
- Use $location.search() instead of window.location.search for URL
  params so cross-page links (owner, conference, search filters) work
  with AngularJS client-side navigation
- Add missing removeRepository() in both repos and user detail controllers
- Fix removeCache() spurious $scope.$apply() that caused digest errors
- Add confirmation prompts and list refresh after remove/cache operations
2026-05-06 21:27:57 +03:00
tdurieux d9104c2ec2 Update commit on branch refresh and validate commit exists on save
Refresh button now always updates the commit to the latest SHA instead
of preserving the stale one in edit mode. Both create and update routes
verify the commit still exists on GitHub before persisting.
2026-05-06 21:14:53 +03:00
tdurieux d1d6257512 fix audio url 2026-05-06 20:37:50 +03:00
tdurieux 2f6ec41a2c block indexing webpage as well 2026-05-06 20:36:04 +03:00
tdurieux bd8656206a fix persistance bugs 2026-05-06 20:00:59 +03:00
tdurieux 67cb2538b1 fix old github download repos 2026-05-06 19:37:16 +03:00
tdurieux da78708b7b Improve error handling 2026-05-06 18:43:36 +03:00
tdurieux aae6eae6eb handle rate limit 2026-05-06 17:50:01 +03:00
tdurieux c1e18f82a9 Improve error handling 2026-05-06 17:39:43 +03:00
tdurieux 6bad6c2f09 fix bugs and report better errors 2026-05-06 17:26:47 +03:00
tdurieux 3b27816702 fix incremental 2026-05-06 17:12:58 +03:00
tdurieux 804bbffb7a Improve error handling 2026-05-06 17:03:19 +03:00
tdurieux 48e782946a Improve error handling 2026-05-06 16:56:07 +03:00
tdurieux b2461088e8 fix test 2026-05-06 16:55:50 +03:00
tdurieux cf2f172aca fix(gist): set gist subpaths individually to avoid CastError
Mongoose treats `gist` as a nested path, not a sub-schema, so
set("gist", payload) mis-casts the inner subdoc arrays and fails
validation with 'Cast to [string] failed' at gist.files.0. Set each
subpath individually so the files/comments arrays cast correctly.
2026-05-06 16:52:48 +03:00
tdurieux dcb524c8c1 Improve error handling 2026-05-06 16:45:22 +03:00
tdurieux 3613c895c8 improve logging 2026-05-06 16:31:10 +03:00
tdurieux 873c910dd3 Improve error dashboard 2026-05-06 16:12:37 +03:00
tdurieux 6f418d6332 log AnonymousError context detail in serialized output
Surface the path/repo/url tied to an AnonymousError when it gets
serialized for logging — previously logs only carried name, message,
and httpStatus, which made file_not_found entries impossible to trace
back to a specific file or repo. Extract the existing detail formatting
out of toString() into a public detail() method, harden it against
AnonymizedFile getters that can throw, and have serializeError include
the result as a "detail" field.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-06 13:46:06 +03:00
tdurieux 7dc48bccd2 fix(build): isolate parallel npm cache mounts and bump node to 22
- Give the deps and prod-deps stages distinct BuildKit cache IDs with
  sharing=locked so concurrent npm ci runs no longer race on
  /root/.npm and fail with ENOTEMPTY.
- Drop the post-install 'npm cache clean --force' in prod-deps; it
  was wiping the cache mount it had just filled.
- Update base images from node:21 to node:22 (LTS) to silence
  EBADENGINE warnings from packages requiring node >= 22.
2026-05-06 13:40:17 +03:00
tdurieux 06a098fba7 perf(deploy): faster builds and zero-downtime streamer rollover
- Multi-stage Dockerfile with BuildKit npm cache mounts and a separate
  prod-deps stage so source edits don't reinstall or prune.
- Tighter .dockerignore to shrink build context.
- Healthchecks: add start_period and tighten interval/retries so
  containers report healthy as soon as the process is actually ready
  instead of after a full polling interval.
- Move recoverStuckPreparing() off the startup critical path; the
  recovery sweep now runs in the background after app.listen.
- depends_on uses condition: service_healthy and the obsolete
  compose 'version' key is gone.
- New scripts/build.sh + scripts/deploy.sh: deploy.sh builds, exits
  early if the image is unchanged, runs a blue/green streamer swap
  (scale to 2N, wait healthy in parallel, drop olds), then recreates
  the API with --no-deps to avoid compose's depends_on re-poll.
2026-05-06 13:38:19 +03:00
tdurieux 371693dc3b fix: end stream for files <= 150 bytes in resolveLfsPointer
The LFS-pointer probe buffered up to 150 bytes before deciding whether
to forward the blob or swap to the raw URL. For blobs that fit entirely
in the probe, decide() ran from the source's end event and attached
data/end listeners to an already-ended stream, so out.end() was never
called. The response hung until upstream timed out and storage.write
left an incomplete cached copy, which then forced a re-fetch on every
subsequent read.

Pass a sourceEnded flag through decide() and end the output directly
when the source has already finished. Also skip the GitHub blob fetch
when the tree size is already over MAX_FILE_SIZE, surfacing
file_too_big instead of a translated 422.
2026-05-06 13:10:25 +03:00
tdurieux 7dd6d872e9 Improve error handling 2026-05-06 12:44:07 +03:00
tdurieux e1cf8e3a00 Improve error handling 2026-05-06 12:26:12 +03:00
tdurieux c4182b5b2a fix: prevent E11000 on concurrent getRepositoryFromGitHub
Replace check-then-insert with atomic findOneAndUpdate upsert keyed on
externalId, plus a single E11000 retry fallback. Eliminates the duplicate
key race when two requests resolve the same gh_<id> concurrently.
2026-05-06 12:22:42 +03:00
tdurieux 75b67448ef fix 2026-05-06 11:24:48 +03:00
tdurieux 1254c56223 fix: dedupe folder contents on re-fetch (#701)
getFiles blindly appended fetched entries to $scope.files, so
re-opening a folder duplicated its children in the tree. Drop any
existing entries at the requested path before appending.
2026-05-06 11:16:12 +03:00
tdurieux c2d43164d0 error logging improvement, regex fix 2026-05-06 11:16:12 +03:00
dependabot[bot] e34f45522f chore(deps): bump uuid and bullmq (#691)
* chore(deps): bump uuid and bullmq

Removes [uuid](https://github.com/uuidjs/uuid). It's no longer used after updating ancestor dependency [bullmq](https://github.com/taskforcesh/bullmq). These dependencies need to be updated together.


Removes `uuid`

Updates `bullmq` from 2.4.0 to 5.76.5
- [Release notes](https://github.com/taskforcesh/bullmq/releases)
- [Commits](https://github.com/taskforcesh/bullmq/compare/v2.4.0...v5.76.5)

---
updated-dependencies:
- dependency-name: bullmq
  dependency-version: 5.76.5
  dependency-type: direct:production
- dependency-name: uuid
  dependency-version: 
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix(bullmq): adapt isRunning + getJobs typing for v5 API

Worker.isRunning became a method (was a property in v2), and
Queue.getJobs now requires a mutable JobType[] (was string[]).

* clean up

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: tdurieux <durieuxthomas@hotmail.com>
Co-authored-by: Thomas Durieux <5577568+tdurieux@users.noreply.github.com>
2026-05-06 08:40:12 +02:00
tdurieux 2528f42b59 add tests 2026-05-06 08:23:22 +03:00
tdurieux 7e0f900c7a fix: resolve eslint unused-var and useless-assignment warnings
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-06 08:22:45 +03:00
dependabot[bot] e306adcc97 chore(deps): bump ip-address and socks (#697)
Bumps [ip-address](https://github.com/beaugunderson/ip-address) and [socks](https://github.com/JoshGlazebrook/socks). These dependencies needed to be updated together.

Updates `ip-address` from 9.0.5 to 10.2.0
- [Commits](https://github.com/beaugunderson/ip-address/commits)

Updates `socks` from 2.8.4 to 2.8.8
- [Release notes](https://github.com/JoshGlazebrook/socks/releases)
- [Commits](https://github.com/JoshGlazebrook/socks/compare/2.8.4...2.8.8)

---
updated-dependencies:
- dependency-name: ip-address
  dependency-version: 10.2.0
  dependency-type: indirect
- dependency-name: socks
  dependency-version: 2.8.8
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-06 07:21:33 +02:00