mirror of
https://github.com/tdurieux/anonymous_github.git
synced 2026-10-18 13:50:50 +02:00
fix: avoid timeouts for safe ZIP anonymization (#873)
This commit is contained in:
1 parent
40c369641b
commit
4fd31a59c8
7 files changed
+159
-12
No files matched your search
@@ -67,7 +67,17 @@ Run `npm run benchmark:performance` for CPU comparisons. Add `-- --memory-only`
|
||||
|
||||
Before deploying the readers, run `npm run migrate:performance` with the application's database environment. This adds the membership, file-generation, mapping, private-name, maintenance and cleanup-recovery indexes; it never drops indexes. It also queues legacy stream-cache cleanup once. Maintenance preserves managed content caches and ZIP source roots. Rerun it if you applied the migration before the cleanup-recovery indexes were added. The command passed twice against the disposable MongoDB fixture and retained existing indexes. Deploy the compiled worker file with the application build. Source-mode workers use ts-node and have additional startup cost.
|
||||
|
||||
Production rollout still needs latency percentiles under the real mixed workload, cleanup-backlog observation, and real S3 multipart/network-failure checks. The local S3 fixture exercises SDK requests and short-upload publication but does not emulate all service failure modes. No deployment or production data changes were performed.
|
||||
Production rollout still needs latency percentiles under the real mixed workload, cleanup-backlog observation, and real S3 multipart/network-failure checks. The local S3 fixture exercises SDK requests and short-upload publication but does not emulate all service failure modes. PRs #866 and #872 were deployed on 8 October 2026 after clean Codex reviews. API and streamer health, static assets, public repository browsing, and a complete 699-entry ZIP download passed production checks; all 12 required indexes were verified without replacing MongoDB, Redis, or the backup container.
|
||||
|
||||
### Production ZIP anonymization timeouts
|
||||
|
||||
The later log check found four failed ZIP requests for one repository. Its nine terms compiled to 18 RE2 variants with no JavaScript fallback; its largest files were about 8 MB. The one-second synchronous deadline also applied inside workers, so safe whole-file matching could abort a ZIP. A synthetic 7.84 MB dataset with nine safe rules reproduced that error. Merely increasing the worker timeout was insufficient because absent regex terms still triggered repeated RE2 scans over the whole input.
|
||||
|
||||
Regex terms now use a conservative mandatory-prefix absence check where possible. Alternation and unfamiliar syntax keep the full matcher; the optimization never decides replacements. Worker transformation gets ten seconds within the pool's existing fifteen-second deadline. JavaScript-only backtracking replacements keep their own one-second deadline, and synchronous callers retain the one-second whole-operation limit. Output is still published only after complete anonymization succeeds, with the existing memory, spool, cancellation, and queue limits.
|
||||
|
||||
Regression coverage includes safe regex jobs, dense Unicode replacements, a large ZIP entry followed by another entry, prefix parity with the original RE2 matcher, and worker recovery after a backtracking timeout. Build and lint passed, with seven existing lint warnings; the complete suite passed 955 tests with 51 pending and zero failures using disposable MongoDB/Redis and `PERFORMANCE_STREAMER_COMPILED=1`. These include 125 performance tests. A separate 100-case prefix parity probe also passed.
|
||||
|
||||
The candidate Docker image passed the 7.84 MB safe-regex ZIP fixture through real Express compression on Node 22, with plain and gzip responses and zero remaining drain listeners. Its native-regex guard rejected a backtracking job after one second without publishing partial output, and the next worker job succeeded. The affected repository's nine rules were also replayed against synthetic numeric input in isolated containers with one CPU: the deployed image hit its one-second timeout, while the candidate completed in 2.4 seconds. This replay used no original repository file contents and changed no live application containers. Token-refresh warnings, missing-branch HTTP status handling, and backup-retention cleanup remain separate follow-up issues.
|
||||
|
||||
## Baseline and implementation rules
|
||||
|
||||
|
||||
@@ -108,14 +108,14 @@
|
||||
],
|
||||
"tests": {
|
||||
"fullSuite": {
|
||||
"passing": 950,
|
||||
"passing": 955,
|
||||
"pending": 51,
|
||||
"failing": 0,
|
||||
"note": "Clean committed checkout after the ZIP HTTP rollout follow-up, with disposable MongoDB/Redis and memory-backed temporary storage. Compiled streamer startup/lifecycle/idle-cleanup integration remains enabled. Two real HTTP tests cover Express compression, plain and gzip output, forced backpressure, listener cleanup and chunked upstream archives.",
|
||||
"note": "Isolated worktree with disposable MongoDB/Redis, memory-backed temporary storage, and PERFORMANCE_STREAMER_COMPILED=1. New coverage verifies safe-regex prefix parity, large worker and ZIP jobs, dense Unicode replacements, one-second native-regex rejection without partial output, and worker recovery.",
|
||||
"validationDate": "2026-10-08"
|
||||
},
|
||||
"performanceSuite": {
|
||||
"passing": 122,
|
||||
"passing": 125,
|
||||
"note": "Included in the latest clean full-suite run with disposable MongoDB and Redis."
|
||||
},
|
||||
"uiSuite": {
|
||||
@@ -153,6 +153,31 @@
|
||||
"gzip": "passed",
|
||||
"forcedBackpressure": "passed",
|
||||
"drainListenerCleanup": "passed"
|
||||
},
|
||||
"zipAnonymization": {
|
||||
"safeRegexFixtureBytes": 7840011,
|
||||
"denseUnicodeFixtureBytes": 20800000,
|
||||
"workerProcessingDeadlineMs": 10000,
|
||||
"workerJobDeadlineMs": 15000,
|
||||
"nativeRegexDeadlineMs": 1000,
|
||||
"extraPrefixParityCases": 100,
|
||||
"compiledNode22Http": {
|
||||
"plain": "passed",
|
||||
"gzip": "passed",
|
||||
"drainListenerCleanup": "passed"
|
||||
},
|
||||
"compiledNode22Guard": {
|
||||
"partialOutput": false,
|
||||
"workerRecovery": "passed"
|
||||
},
|
||||
"productionRulesReplay": {
|
||||
"scope": "Synthetic numeric input using the affected repository rules, in isolated Node 22 containers limited to one CPU. No original file contents or identifying terms were printed; live application containers were unchanged.",
|
||||
"terms": 9,
|
||||
"inputBytes": 7840032,
|
||||
"baseline": "Script execution timed out after 1000ms",
|
||||
"candidateComplete": true,
|
||||
"candidateElapsedMs": 2401.216523
|
||||
}
|
||||
}
|
||||
},
|
||||
"sockets": {
|
||||
|
||||
Reference in new issue
Block a user