Compare commits

..
150 Commits
Author SHA1 Message Date
iuyua9 bc48ba4308 docs: align GFPGAN model filenames (#1910)
README and models/instructions.txt named GFPGANv1.4, but the enhancer loads gfpgan-1024.onnx via onnxruntime. instructions.txt also pointed at a PyTorch .pth that the ONNX path cannot load at all, so the manual-fallback instructions were unusable.
2026-08-29 15:05:23 +08:00
Ihor Kuzmychov b53844eb57 fix: skip setrlimit(RLIMIT_DATA) on macOS (#1849)
setrlimit(RLIMIT_DATA) is rejected by the macOS kernel for the values used here,
crashing the app during startup. Since --max-memory defaults to suggest_max_memory()
(4 on Darwin), max_memory is always set on macOS and every launch reached this call.

Fixes #1848.
2026-08-29 14:53:57 +08:00
Chris G 20cafb0079 docs: change dependency version number (#1914) 2026-08-29 14:28:40 +08:00
Dopan 7ca6d0b202 Merge pull request #1864 from 5uck1ess/pr/webp-support
feat: WEBP source image support
2026-08-26 10:51:56 +08:00
Kenneth Estanislao f7db37679a Update readme
Includes website links and proper redirect to our official website
2026-08-23 05:12:24 +08:00
Kenneth Estanislao bd500d54b4 auto download some models
Retarget download url for safer model controls
2026-08-23 02:55:32 +08:00
cuyua9 987f6b392b fix: extract frames for map faces fallback (#1824)
Verified this fix. Confirmed the bug by reverting just the `modules/core.py` hunk and
re-running the new regression test — with the old code, `process_video`/`create_video`
run against a temp directory that was never populated when `map_faces=True`, since
`create_temp`/`extract_frames` were skipped for that case. That means map-faces video
runs were silently broken (empty or failed output).

The fix removes the `map_faces` guard so extraction always runs before the disk-based
fallback, which is correct for both cases that reach this branch (map_faces=True, and
non-map-faces pipe failures). `create_temp` is idempotent (mkdir exist_ok=True), so the
double-call for the non-map-faces path is harmless.
2026-08-14 06:44:22 +08:00
Dopan 97a44800a2 Merge pull request #1902 from 1ceseismic/fix/linux-camera-device-path-rebase
fix: no cam detected on Arch Linux, use string path for camera capture instead
2026-08-13 22:50:16 +08:00
Vito-M 345fa4a0b0 fix: no cam detected on Arch Linux, use string path for camera capture instead 2026-08-09 19:48:57 +12:00
Gao Yiman bdeeb3ace0 docs: add onnxruntime-openvino/OpenVINO version pairing note (#1893)
Thanks for this — useful reference table, and it lines up with the version-pairing issues we've been fixing (#1879). Merging.
2026-08-08 07:15:21 +08:00
Kenneth Estanislao 230217ec11 update on requirements
some update on what is needed to be updated
2026-07-29 05:20:29 +08:00
Kenneth Estanislao 156321f7a3 Upgrade onnxruntime-gpu to version 1.26.0
Updated onnxruntime-gpu version to 1.26.0 for non-Darwin platforms.
2026-07-29 04:37:54 +08:00
Nguyen Van Nam 8234965ee8 fix: clamp video frame seek index (#1790)
Prevent get_video_frame() from seeking to invalid frame positions.

The default frame_number=0 now resolves to the first frame instead of -1, and oversized frame requests clamp to the final valid frame instead of seeking past the end. Empty or invalid videos now return None safely after releasing the capture.

Affected files: capturer.py

Signed-off-by: Nguyen Van Nam <nam.nv205106@gmail.com>
2026-07-23 22:16:38 +08:00
Dopan ab64c186ec Merge pull request #1879 from dunegym/fix/openvino-dll-loading
fix: resolve OpenVINO DLL loading on Windows for OpenVINOExecutionProvider
2026-07-19 15:12:49 +08:00
Makaru b8e781e539 chore: remove trailing whitespace 2026-07-19 12:20:54 +08:00
KRSHH ff7ee0d219 Revise Quickstart section in README 2026-07-19 00:25:10 +05:30
Nguyen Van Nam 8d727eba3e fix: bound face-cluster count by available embeddings (#1793)
`find_cluster_centroids()` iterates `k` from 1..`max_k` unconditionally. If `len(embeddings) < max_k`, `KMeans(n_clusters=k)` will raise `ValueError` when `k` exceeds the number of samples. This is an unhandled crash path on small datasets.


Affected files: cluster_analysis.py

Signed-off-by: Nguyen Van Nam <nam.nv205106@gmail.com>
2026-07-14 23:55:23 +08:00
Cocoon-Break eba2a958d3 fix: skip empty face clusters in default_target_face (#1757)
Skip face clusters when no best face was detected, preventing a NoneType error while preserving normal face-detection behavior.

Closes #1755
2026-07-14 23:51:43 +08:00
dunegym 14ba4f9c0b fix: centralize OPENVINO_PROVIDER_CONFIG and log SystemExit
Address Sourcery review feedback on PR #1879:

- Move OPENVINO_PROVIDER_CONFIG from _onnx_enhancer.py to
  platform_info.py (a leaf module with no modules.* imports), so
  the enhancer and face_swapper no longer import each other just to
  share a constant. _onnx_enhancer re-exports it; face_swapper now
  imports it at module top level instead of inside get_face_swapper().
- Narrow run.py's SystemExit handling: catch SystemExit separately
  and print a [startup] message so the failure is visible instead
  of being swallowed alongside ImportError/FileNotFoundError.
2026-07-12 15:16:18 +08:00
dunegym 7d2d7fb1f3 fix: address PR review feedback — SystemExit, AUTO device, thread timing
- Catch SystemExit from add_openvino_libs_to_path() so a missing
  OpenVINO installation never causes a hard exit on Windows
- Replace hard-coded GPU+FP16 with AUTO:GPU,NPU,CPU device priority,
  letting OpenVINO pick the best available accelerator
- Extract shared OPENVINO_PROVIDER_CONFIG constant to avoid
  duplication between _onnx_enhancer and face_swapper
- Defer thread-suggestion evaluation until after execution_providers
  is assigned, fixing a latent timing bug that affected OpenVINO,
  CUDA, and DML thread hints
2026-07-11 12:45:00 +08:00
noahximus 57c4c32377 Merge pull request #1876 from ElKhalil19/main 2026-07-07 04:55:43 +08:00
El Khalil d00b09f5d8 docs: update manual installation to use shallow clone (#1866) 2026-07-03 19:54:39 +01:00
dunegym 897dc21da4 fix: resolve OpenVINO DLL loading on Windows for OpenVINOExecutionProvider
- Add add_openvino_libs_to_path() call in run.py before any ONNX
  InferenceSession creation to register openvino.dll directory
- Detect and advertise OpenVINOExecutionProvider in platform_info
  banner and accelerator label
- Prioritize openvino over dml in suggest_default_execution_provider
- Configure OpenVINO EP with GPU + FP16 device options for optimal
  performance (~13 FPS on Intel GPU vs ~1 FPS CPU fallback)
- Set thread hint to 1 when OpenVINO EP is active
2026-06-28 21:46:55 +08:00
Kenneth Estanislao 834092c891 Update Quick Start section to v2.7 RC6 2026-06-24 18:15:40 +08:00
Kenneth Estanislao da0672ad6b Enhance README with details on pre-built versions
Updated the README to clarify the benefits of pre-built versions and optimizations for hardware.
2026-06-24 18:14:59 +08:00
Tym Rabchuk 47dffeb307 fix(webp): finish extension centralization from pre-submission review
- build the video save-dialog filter from VIDEO_EXTENSIONS (_VIDEO_FILE_FILTER)
  instead of a hardcoded "Videos (*.mp4 *.mkv)" — the last filter that still
  drifted from the canonical set
- remove the now-dead file_types list (unused in both the fork and upstream;
  the PySide6 dialogs use the QFileDialog filter strings) and drop it from the
  centralization comment
2026-06-23 19:30:07 -04:00
Tym Rabchuk 9e1f0cc3a5 fix(webp): address review — drop broken GIF, robust ext check, centralize lists
Review feedback on #1831:
- Remove *.gif from the save/output dialog filter (PR had added it there).
  Verified empirically that cv2.imread/imwrite cannot decode OR encode GIF on
  OpenCV 4.10 *or* 4.11 (write raises, read returns None), so GIF silently
  failed on both ends — dropped from every dialog and from has_image_extension.
- has_image_extension now uses os.path.splitext so only the true extension
  counts ('photo.png.bak' / 'clip.webp.mp4' are no longer treated as images).
- Centralize the supported-extension set in modules.globals (IMAGE_EXTENSIONS /
  VIDEO_EXTENSIONS); file_types, all QFileDialog filters and has_image_extension
  now derive from it instead of hand-copied lists that had already drifted.

WEBP itself is unchanged and works (libwebp ships with opencv-python).
2026-06-22 20:33:11 -04:00
Kenneth Estanislao 834bc43768 Support non-ascii characters 2026-06-14 20:18:56 +08:00
Dopan 3b69413d61 Merge pull request #1845 from maxwbuckley/ruff-code-health
Add ruff CI gate and fix deterministic lint issues
2026-06-01 00:50:59 +08:00
Kenneth Estanislao 07e2e960c8 Update Quick Start version from v2.7 RC1 to v2.7 RC2 2026-05-24 18:55:35 +08:00
Max BuckleyandClaude Opus 4.7 ba27b75265 Use astral-sh/ruff-action for inline PR annotations
Swap the manual pip install + ruff check steps for astral-sh/ruff-action@v4.0.0.
Same pinned ruff 0.15.7, but with --output-format=github so violations appear
as inline annotations on the PR diff instead of a flat log.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 15:50:44 +02:00
Max BuckleyandClaude Opus 4.7 cfa8123b67 Add ruff CI gate and fix deterministic lint issues
Introduces pyproject.toml + .github/workflows/ruff.yml that gate
E701, E711, E712, F401, F541 on every PR and push to main.

Fixes the existing findings for those rules:
- Remove unused imports (sklearn.silhouette_score, numpy in several
  files, typing.Optional, get_one_face, gpu_cvt_color, sys,
  insightface.face_align)
- Annotate the intentional tkinter_fix side-effect import with
  `# noqa: F401`
- Split multi-statement `if x: y` one-liners onto separate lines
- Replace `state == True` / `state == False` with truthiness checks
- Drop `f` prefix from f-strings with no placeholders

F841 (unused-variable), E402 (module-level-import-not-at-top), and
F821 (undefined-name) are left out of the gate for now — they surface
real findings (including a latent NameError in face_swapper.py) that
require human review to fix safely.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 15:44:31 +02:00
Dopan 08b2dd2526 Merge pull request #1844 from hklcf/fix/bugfix-batch
lgtm
2026-05-23 16:54:41 +08:00
hklcf 886e64b320 Fix: resolve 5 confirmed bugs (imwrite_unicode, macOS memory, face_analyser None crash, silent sys.exit, core memory calc) 2026-05-23 10:37:20 +08:00
Kenneth Estanislao aa6f2cbade Update version from v2.7 beta to v2.7 RC1 in README 2026-05-21 05:11:41 +08:00
Tym Rabchuk 0b61ad5c0d feat: webp source image support
Ported from April 2026 Fork:
- has_image_extension() now recognizes .webp/.gif/.bmp
- is_image() checks extension before mimetypes (Windows mimetypes
  doesn't always register webp)
- File dialog filter includes *.webp
2026-05-18 21:00:33 -04:00
Kenneth Estanislao a21ccf488c Update version number in README.md to 2.1.6 2026-05-18 04:15:01 +08:00
Kenneth Estanislao ca8e39e3bb Fix mouth mask 2026-05-18 02:11:04 +08:00
Kenneth Estanislao 0e97e474e4 better swapping 2026-05-18 01:40:01 +08:00
Kenneth Estanislao 9c67a7aacc fixed poisson blend 2026-05-18 01:36:24 +08:00
Dopan 4a674d33ef Merge pull request #1826 from obook/pr/preload-nvidia-libs-linux
Pre-load NVIDIA shared libraries on Linux
2026-05-17 00:04:06 +08:00
Olivier Booklage 682450755f Avoid duplicating LD_LIBRARY_PATH entries
Skip prepending a directory that is already on LD_LIBRARY_PATH, so a
repeated import of run.py does not bloat the variable.

Addresses review feedback on #1826.
2026-05-16 14:57:02 +02:00
Olivier Booklage 12a3f6a007 Pre-load NVIDIA shared libraries on Linux
Mirrors the Windows preload block from #1775. When onnxruntime-gpu is
installed via pip with nvidia-cudnn-cu12, the .so files sit under
venv/lib/pythonX.Y/site-packages/nvidia/<pkg>/lib/ and the dynamic
linker never sees them. LD_LIBRARY_PATH cannot be set after Python
starts.

Pre-loads every lib*.so* via ctypes.CDLL with RTLD_GLOBAL before
onnxruntime opens its CUDA provider. Also extends LD_LIBRARY_PATH so
child processes (ffmpeg) inherit the path.

Fixes "libcudnn.so.9: cannot open shared object file" on pip-only
Linux installs.
2026-05-16 14:45:54 +02:00
Kenneth Estanislao cede099ccb Update version number in README.md to 2.1.5 2026-05-15 16:33:57 +08:00
Kenneth Estanislao 81a1986ef8 Changed to pyqtUI
Standardizing the UI from quickstart to github version
2026-05-15 16:33:27 +08:00
Kenneth Estanislao ed758eb693 Speed optimization 2026-05-15 15:53:55 +08:00
Kenneth Estanislao 9c5f01c7f1 some fix for face enhancers 2026-05-15 15:13:57 +08:00
Kenneth Estanislao 8bdc348779 Update .gitignore 2026-05-15 14:52:56 +08:00
Makaru e34d204c2e Merge pull request #1803 from zuyua9/fix/get-one-face-detected-faces-zuyua9
fix(face): reuse pre-detected face list

comment: tested, all good
2026-05-08 10:20:56 +08:00
zuyua9 d1376b07d1 fix(face): avoid hiding invalid face inputs 2026-05-08 01:50:25 +08:00
zuyua9 5deadaf428 fix(face): reuse pre-detected face list 2026-05-08 01:35:55 +08:00
Kenneth Estanislao 2fba52e11b Merge pull request #1782 from iikuzmychov/fix/black-border-paste-back 2026-04-29 22:31:09 +08:00
Ihor Kuzmychov 0926b65aaf Merge branch 'hacksider:main' into fix/black-border-paste-back 2026-04-23 19:58:12 +02:00
Ihor Kuzmychov 297acded3b fix: use BORDER_REPLICATE for face warp to eliminate black border 2026-04-23 19:42:32 +02:00
KRSHH 014bce0704 Delete PERFORMANCE.md
Removing Claude session summary
2026-04-23 22:12:55 +05:30
KRSHH c962399669 Delete REVIEW_TODOS.md 2026-04-23 22:11:53 +05:30
Kenneth Estanislao 2dd42dfc75 Merge pull request #1777 from maxwbuckley/coreml-scalar-gather-fix
Keep GFPGAN on ANE: widen scalar Gather indices for CoreML EP
2026-04-22 22:17:34 +08:00
Kenneth Estanislao c38d669f7c Merge pull request #1776 from maxwbuckley/paste-back-optimization
Paste-back: O(crop_area) compositing + uint8 cv2 SIMD blend
2026-04-22 22:14:45 +08:00
Max Buckley 890a6d41b6 onnx_optimize: widen scalar Gather indices for CoreML EP
ORT's CoreML EP GatherOpBuilder::IsOpSupportedImpl explicitly rejects
rank-0 (scalar) index tensors. StyleGAN-derived models (GFPGAN's 1024
variant has 16 of them, one per style-code slice) hit this in the
generator, and the resulting CPU fallbacks split the CoreML subgraph
into multiple partitions with boundary crossings on every inference.

Add a load-time ONNX rewrite that promotes each scalar index to [1] and
squeezes the added axis on the Gather output — semantically identical
but CoreML-compatible. GFPGAN now runs as a single CoreML partition with
zero CPU-fallback nodes; inference drops from ~87 ms to ~81 ms on an
M-series Mac.

The fix has been filed upstream as microsoft/onnxruntime#28180 — the
existing code comment in gather_op_builder.cc already describes this
exact workaround, it just isn't applied. Once the upstream fix ships
and the ORT floor is raised, this pass can be deleted.
2026-04-22 14:08:18 +02:00
Max Buckley f95a0bb7fb Make square aligned-face assumption explicit in _fast_paste_back
Addresses Sourcery feedback on PR #1776: _get_soft_alpha caches a single
NxN template keyed by N, which is correct for the inswapper model
(128x128 aligned-face space) but would silently mis-warp if a caller
ever passed a non-square aligned face. Assert the shape instead of
silently assuming it.
2026-04-22 13:40:18 +02:00
Max Buckley e957a7f4dd Move BGR→RGB after resize in preview display path
The processing thread was running cvtColor on the full-resolution 1920×1080
frame before queueing it for display. Since the display thread immediately
resizes the frame to the preview window (~5× smaller pixel count), doing
the colour conversion on the resized buffer is cheaper overall.

Processing thread now queues BGR; display thread resizes then cvtColor.
2026-04-22 13:31:11 +02:00
Kenneth Estanislao 19416cb3cb Merge pull request #1775 from maxwbuckley/unify-mac-windows
Apple Silicon + Windows CUDA perf: 4-5x FPS, wider capture, platform routing
2026-04-22 18:38:32 +08:00
Max Buckley cbf0859347 Paste-back blend: uint8 cv2 SIMD, no float32 round-trip
Both face_swapper._fast_paste_back and face_enhancer._paste_back were
doing a numpy float32 round-trip per frame: convert the target crop and
the warped face to float32, blend, clip, cast back to uint8. That's four
crop-sized allocations plus unvectorized elementwise math.

Replace with a fused uint8 blend using cv2.merge + cv2.multiply + cv2.add,
which cv2 dispatches to SIMD (NEON on Apple Silicon / AVX on x86). Stored
alpha templates switched from float32 [0, 1] to uint8 [0, 255] so no
conversion is needed per frame. CUDA paths also simplified — upload uint8
alpha (less bandwidth) and scale on device.

Micro-bench on 1000x1000 RGB crop:
  current (float32 numpy): 9.43 ms
  cv2 uint8 fused:         1.16 ms  (8.1× faster, max diff 2/255)

Visual diff is imperceptible (quantization noise in the last step).
2026-04-22 12:05:39 +02:00
Max Buckley a6c99607fc Cut paste-back from quartic to linear in face size
_fast_paste_back used to erode and Gaussian-blur the warped alpha mask in
output coordinates with kernel sizes proportional to the on-screen face
bbox. That made the per-frame cost ~O(area * k^2) — a face filling half
the frame took ~8x the compositing work of one filling a quarter, which
is why FPS fell off when leaning into the camera.

Instead, build a feathered alpha template once at aligned-face resolution
(128x128 for inswapper) and warp the soft mask per-frame. The affine
transform preserves the relative feather width, so the visual output is
equivalent; the per-frame cost is now O(crop_area) with no size-scaled
erode/blur and no size-scaled padding.

Also collapses the CPU fallback onto the same shape — it previously did
a full-frame warpAffine twice per call, which scaled with the whole
frame instead of the face crop.
2026-04-22 11:58:02 +02:00
Max BuckleyandClaude Opus 4.7 0a87d63560 Address PR #1775 review: pipelined-detection race and CUDA-graph monkey-patch
- core._run_pipe_pipeline: hand the background detector its own copy of
  the frame. The frame processors mutate in place via paste-back, which
  was racing with concurrent face detection on the same buffer.
- face_swapper._init_cuda_graph_session: replace the
  `swapper.session.run` monkey-patch with a `_CudaGraphSessionAdapter`
  that proxies every attribute to the underlying session and only
  overrides `.run()`. Guarded so repeat init does not double-wrap.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 11:45:59 +02:00
Max BuckleyandClaude Opus 4.7 ea19030c74 Add PERFORMANCE.md and REVIEW_TODOS.md
PERFORMANCE.md documents measured gains on MacBook Pro M3 Max vs
hacksider/Deep-Live-Cam main@64d3f06:

- Face swap only:     <5 FPS  ->  >20 FPS
- Face swap + GFPGAN: <2 FPS  ->  >10 FPS
- Camera:             640x480 ->  960x540 MJPEG @ 60fps

Breaks down the contributors (camera negotiation, CoreML graph
rewrites with before/after op latencies, pipeline overlap, GFPGAN
temporal cache, paste-back optimization, platform routing, Windows
CUDA path) and how to reproduce.

REVIEW_TODOS.md captures 12 findings from two independent reviews
(Claude in-tree + Codex second opinion) grouped as Blockers /
Should-fix / Consider, each with file:line and suggested fix. The
two Blocker/Should-fix items are addressed in the preceding commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 11:08:33 +02:00
Max BuckleyandClaude Opus 4.7 4d04e830bc Fix CUDA-graph replay race + many_faces enhancer regression
Two issues surfaced in post-squash review of f65aeae:

1. CUDA-graph replay buffers were shared across threads with no lock.
   `_cuda_graph_swap_inference` mutates module-level ort_input/ort_latent
   and runs run_with_iobinding — concurrent swap calls on Windows/CUDA
   could overwrite each other's bound input buffers before replay,
   producing wrong-face output. Added `_cuda_graph_lock` around the
   full update/run/read sequence.

2. Face enhancer loop unconditionally broke after the first face, so
   `many_faces=True` silently enhanced only one face. Also, the
   single-slot temporal cache would paste the same enhancement onto
   every target if reused in many-faces mode. Gated the break on
   `not many_faces_mode` and disabled the cache path in that mode.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 11:08:23 +02:00
Max BuckleyandClaude Opus 4.7 f65aeae5db Apple Silicon + Windows CUDA perf: 60 FPS pipeline, cross-platform routing
Bundles CoreML graph rewrites, GPU-accelerated pipeline work, Windows CUDA
fixes, and Mac/Windows runtime routing into a single drop.

CoreML (Apple Silicon):
- Decompose Pad(reflect) → Slice+Concat in inswapper_128 so the model
  runs in one CoreML partition instead of 14 (TEMPORARY: fixed upstream
  in microsoft/onnxruntime#28073, drop when ORT >= 1.26.0).
- Fold Shape/Gather chains to constants in det_10g (21ms → 4ms).
- Decompose Split(axis=1) → Slice pairs in GFPGAN (155ms → 89ms).
- Route detection model to GPU so the ANE is free for the swap model.
- Centralize provider/config selection in create_onnx_session.

Pipeline (all platforms):
- Parallelize face landmark + recognition post-detection; skip landmark_2d_106
  when only face_swapper is active.
- Pipeline face detection with swap for ANE overlap.
- GPU-accelerated paste_back, MJPEG capture, zero-copy display path.
- Standalone pipeline benchmark script.

Windows / CUDA:
- CUDA graphs + FP16 model + all-GPU pipeline for 1080p 60 FPS.
- Auto-detect GPU provider and fix DLL discovery for Windows CUDA execution.

Cross-platform:
- platform_info helper for Mac/Windows runtime routing.
- GFPGAN 30 fps + MSMF camera 60 fps with adaptive pipeline tuning.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 10:44:59 +02:00
KRSHH 64d3f06089 Delete tests directory 2026-04-19 17:36:33 +05:30
Kenneth Estanislao fceafcb234 Merge pull request #1751 from Gujiassh/fix/face-mask-none-frame-guard
fix(face-mask): guard create_face_mask against None frame
2026-04-15 14:13:18 +08:00
Kenneth Estanislao 033475b89c Update version in README from 2.1.2 to 2.1.3 2026-04-15 01:29:59 +08:00
Kenneth Estanislao 07711af712 Update contributors section in README.md 2026-04-15 01:29:44 +08:00
Kenneth Estanislao 44664d8a7f Merge pull request #1746 from maxwbuckley/apple-silicon-perf-optimizations
Apple Silicon performance: 1.5 → 10+ FPS (zero quality loss)
2026-04-15 01:25:51 +08:00
gujishh 15a3f537a4 test: cover additional invalid frame guards 2026-04-13 21:09:27 +09:00
gujishh fbcea9e135 fix(face-mask): guard create_face_mask against None frame 2026-04-12 14:19:48 +09:00
Max BuckleyandClaude Opus 4.6 646b0f816f Move hot-path imports to module scope
Address Sourcery review feedback: move face_align and get_one_face
imports from inside per-frame functions to module-level to avoid
repeated attribute lookup overhead in the processing loop.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 14:34:53 +02:00
Max BuckleyandClaude Opus 4.6 bcdd0ce2dd Apple Silicon performance: 1.5 → 10+ FPS (zero quality loss)
Fix CoreML execution provider falling back to CPU silently, eliminate
redundant per-frame face detection, and optimize the paste-back blend
to operate on the face bounding box instead of the full frame.

All changes are quality-neutral (pixel-identical output verified) and
benefit non-Mac platforms via the shared detection and paste-back
improvements.

Changes:
- Remove unsupported CoreML options (RequireStaticShapes, MaximumCacheSize)
  that caused ORT 1.24 to silently fall back to CPUExecutionProvider
- Add _fast_paste_back(): bbox-restricted erode/blur/blend, skip dead
  fake_diff code in insightface's inswapper (computed but never used)
- process_frame() accepts optional pre-detected target_face to avoid
  redundant get_one_face() call (~30-40ms saved per frame, all platforms)
- In-memory pipeline detects face once and shares across processors
- Fix get_face_swapper() to fall back to FP16 model when FP32 absent
- Fix pre_start() to accept either model variant (was FP16-only check)
- Make tensorflow import conditional (fixes crash on macOS)
- Add missing tqdm dep, make tensorflow/pygrabber platform-conditional

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 14:28:07 +02:00
Kenneth Estanislao 8703d394d6 ONNX CUDA exhaustive convolution search + IO binding 2026-04-09 16:34:27 +08:00
Kenneth Estanislao 69e3fc5611 Rendering optimization
The PNG encode/decode alone was consuming significant CPU time per frame. This is eliminated entirely.
2026-04-09 16:25:22 +08:00
Kenneth Estanislao 2b26d5539e supress error message
Some people just want the opencv error gone. I keep on telling them that it is only for blurs and color conversion. It is the onnx runtime who is running the swap.
2026-04-09 16:04:00 +08:00
Kenneth Estanislao fea5a4c2d2 Merge pull request #1707 from rohanrathi99/main
Switch to FP32 model by default, add run script
2026-04-05 23:19:17 +08:00
Kenneth Estanislao 51fb7a6ad6 Merge pull request #1722 from mvanhorn/osc/1654-face-enhancer-v2
fix(face-enhancer): add missing process_frame_v2 method
2026-04-05 23:16:52 +08:00
Kenneth Estanislao 6da4f398d5 Merge pull request #1731 from JiayuuWang/contribot/fix-readme-macos-python-version
docs: fix inconsistent Python version references in macOS/Linux setup (fixes #1632)
2026-04-05 23:16:20 +08:00
Kenneth Estanislao 3e362383d8 Merge pull request #1732 from yetval/fix/cuda-vram-exhaustion-video-processing
Fix CUDA VRAM exhaustion during video processing
2026-04-05 23:15:38 +08:00
yetval 11fb5bfbc6 Fix CUDA VRAM exhaustion during video processing (#1721) 2026-04-02 22:59:41 -04:00
jacob-wang 586d8f3fb0 docs: fix inconsistent Python version references in macOS/Linux setup
The macOS Apple Silicon section installed Python 3.11 but then
referenced Python 3.10 in several places:

- `brew install python-tk@3.10` → python-tk@3.11
- Linux comment "Ensure you use the installed Python 3.10" → 3.11
- CoreML section cross-reference "completed the macOS setup above
  using Python 3.10" → 3.11
- `python3.10 run.py` usage command → python3.11
- "You must use Python 3.10" note → 3.11
- `brew reinstall python-tk@3.10` troubleshooting tip → 3.11
- Removed `python@3.11` from the list of conflicting versions to
  uninstall (it is the required version, not a conflict)

Fixes #1632
2026-04-03 10:33:11 +08:00
Kenneth Estanislao 1edc4bc298 DML Lock fixed for cuda and CPU 2026-04-01 23:56:01 +08:00
ozp3 1f3668f7c1 Delete DeepLiveCam.lnk
remove lnk and bat files as requested
2026-04-01 23:56:01 +08:00
ozp3 3d16ee346f Delete run-dml.bat
remove lnk and bat files as requested
2026-04-01 23:56:01 +08:00
ozp3 ab834d5640 feat: AMD DML optimization - GPU face detection, detection throttle, pre-load fix 2026-04-01 23:56:01 +08:00
Kenneth Estanislao bf8a89d20a Merge pull request #1725 from jhihweijhan/fix/video-output-pipeline
Fix missing video output reporting and encoding flow
2026-04-01 23:14:22 +08:00
Kenneth Estanislaoandsourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> bb4ef4a133 Apply suggestion from @sourcery-ai[bot]
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
2026-04-01 23:13:59 +08:00
Kenneth Estanislao b6b6c741a2 Revert "Merge pull request #1710 from ozp3/amd-dml-optimization"
This reverts commit 1b240a45fd, reversing
changes made to d9a5500bdf.
2026-04-01 22:33:01 +08:00
Kenneth Estanislao 1b240a45fd Merge pull request #1710 from ozp3/amd-dml-optimization
AMD GPU (DirectML) Optimization for Live Mode
2026-04-01 22:29:43 +08:00
ozp3 ecf02d0640 Delete DeepLiveCam.lnk
remove lnk and bat files as requested
2026-04-01 16:46:28 +03:00
ozp3 0cbc9f126f Delete run-dml.bat
remove lnk and bat files as requested
2026-04-01 16:45:31 +03:00
Karl a3fd56a312 Fix missing video output reporting and encoding flow 2026-04-01 15:22:09 +08:00
Matt Van HornandClaude Opus 4.6 9525d45291 fix(face-enhancer): add missing process_frame_v2 method
The live webcam preview in ui.py calls process_frame_v2() on all
frame processors, but face_enhancer.py was missing this method.
This caused an AttributeError crash when the GFPGAN face enhancer
was enabled during live mode.

Fixes https://github.com/hacksider/Deep-Live-Cam/issues/1654

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 23:49:12 -07:00
Kenneth Estanislao d9a5500bdf Merge pull request #1713 from TeachDian/fix-1705-wsl-onnxruntime-gpu 2026-03-29 04:54:34 +08:00
TeachDian 86134b6e1d Fix #1705: Update onnxruntime-gpu requirement to 1.23.2 for WSL compatibility 2026-03-29 04:46:48 +08:00
ozp3 fbd1cc5973 docs: add AMD DML optimization notes to README 2026-03-28 13:16:43 +03:00
ozp3 eac2ad2307 feat: AMD DML optimization - GPU face detection, detection throttle, pre-load fix 2026-03-28 13:09:20 +03:00
Kenneth Estanislao 9e6f30c0a4 silenced deprecation 2026-03-27 21:35:27 +08:00
Kenneth Estanislao 97321a740d Update face_analyser.py
320 was over optimized, put back to 640
2026-03-27 21:24:19 +08:00
RohanW11p 9207386e07 Switch to FP32 model by default, add run script
Change default face swapper model to FP32 for better GPU compatibility and avoid NaN issues on certain GPUs.
Revamped `run.py` to adjust PATH variables for dependencies setup and re-added with expanded configuration.
2026-03-27 17:29:01 +05:30
Kenneth Estanislao f5f7ac7764 Revise README for clarity and formatting
Updated README to remove emoji and clarify GPU support details.
2026-03-23 10:02:50 +08:00
Kenneth Estanislao 77d3492eef Add download link for models in README
Added a section for downloading models from Hugging Face.
2026-03-13 23:39:46 +08:00
Kenneth Estanislao 8e3d6e7c65 Add emoji to project title in README
Just want to add an emoji 😝
2026-03-13 22:17:32 +08:00
Kenneth Estanislao ee9699ee70 Happy 80k!
2.1 Released!

- Face randomizer added!
2026-03-13 22:09:18 +08:00
Kenneth Estanislao 3c8b259a3f Some edits on the UI
- Grouped the face enhancers
- Make the mouth mask just a slider
- Removed the redundant switches
2026-03-13 22:03:28 +08:00
Kenneth Estanislao 30b27c2b71 Update Quick Start section to v2.7 beta 2026-03-12 02:40:52 +08:00
Kenneth Estanislao 0d8f3b1f82 Fix on vulnerability report
https://github.com/hacksider/Deep-Live-Cam/issues/1695
2026-03-06 23:26:48 +08:00
KRSHH 6e9e7addf2 Update press section with recent media mentions 2026-03-03 21:16:56 +05:30
Kenneth Estanislao 0c7e871bfc Merge pull request #1689 from laurigates/pr/base-ui-tooltips
feat(ui): add hover tooltips to all controls
2026-02-28 02:41:07 +08:00
Lauri GatesandClaude Opus 4.6 e340b0da8a feat(ui): add hover tooltips to all controls
Add ToolTip class (modules/ui_tooltip.py) and wire descriptive hover
tooltips onto every button, switch, slider, and dropdown in the main
window. Tooltips appear after a 500ms hover delay and are clamped to
screen bounds.

This requires no new dependencies — ToolTip uses only customtkinter.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 21:41:24 +02:00
Kenneth Estanislao d0f81ed755 Merge pull request #1671 from laurigates/pr/fix-macos-camera-enum
fix(macos): replace cv2_enumerate_cameras with safe bounded loop
2026-02-24 14:29:00 +08:00
Kenneth Estanislao de01b28802 Merge pull request #1678 from laurigates/pr/perf-opacity-handling
perf(face-swapper): optimize opacity handling and frame copies
2026-02-24 14:28:17 +08:00
Lauri GatesandClaude Opus 4.6 b645d5e60b fix(macos): replace cv2_enumerate_cameras with safe bounded loop
cv2_enumerate_cameras(CAP_AVFOUNDATION) probes indices 0-99 through
OpenCV's AVFoundation backend, which intermittently segfaults (exit
code 139) when invalid device indices are probed. Replace with a
bounded cv2.VideoCapture loop (range(10)) that safely skips
unavailable indices.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 17:22:35 +02:00
Kenneth Estanislao 31b3a97003 Merge pull request #1680 from laurigates/pr/perf-float32-buffer-reuse
perf(processing): optimize post-processing with float32 and buffer reuse
2026-02-23 15:13:03 +08:00
Kenneth Estanislao e3b46e83b7 Merge pull request #1669 from laurigates/pr/feat-gpen-enhancers
feat: add GPEN-BFR 256 and 512 ONNX face enhancers
2026-02-23 15:05:44 +08:00
Lauri GatesandClaude Opus 4.6 e93fb95903 perf(processing): optimize post-processing with float32 and buffer reuse
- Replace float64 with float32 in apply_mouth_area() blending masks —
  float32 provides sufficient precision for 8-bit image blending and
  halves memory bandwidth
- Use float32 in apply_mask_area() mask computations
- Vectorize hull padding loop in create_face_mask() (face_masking.py)
  replacing per-point Python loop with NumPy array operations
- Fix apply_color_transfer() to use proper [0,1] LAB conversion —
  cv2.cvtColor with float32 input expects [0,1] range, not [0,255]
- Pre-compute inverse masks to avoid repeated (1.0 - mask) subtraction
- Use np.broadcast_to instead of np.repeat for face mask expansion

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 21:27:31 +02:00
Lauri GatesandClaude Opus 4.6 aabf41050a perf(face-swapper): optimize opacity handling and frame copies
Move opacity calculation before frame copy to skip the copy when
opacity is 1.0 (common case). Add early return path for full opacity.
Clear PREVIOUS_FRAME_RESULT instead of caching when interpolation
is disabled.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 21:12:02 +02:00
Lauri GatesandClaude Opus 4.6 e57116de68 feat: add GPEN-BFR 256 and 512 ONNX face enhancers
Add two new face enhancement processors using GPEN-BFR ONNX models
at 256x256 and 512x512 resolutions. Models auto-download on first
use from GitHub releases. Integrates into existing frame processor
pipeline alongside GFPGAN enhancer with UI toggle switches.

- modules/paths.py: Shared path constants module
- modules/processors/frame/_onnx_enhancer.py: ONNX enhancement utilities
- modules/processors/frame/face_enhancer_gpen256.py: GPEN-BFR 256 processor
- modules/processors/frame/face_enhancer_gpen512.py: GPEN-BFR 512 processor
- modules/core.py: Add GPEN choices to --frame-processor CLI arg
- modules/globals.py: Add GPEN entries to fp_ui toggle dict
- modules/ui.py: Add GPEN toggle switches and processing integration

Closes #1663

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 19:39:12 +02:00
Kenneth Estanislao d5338a3eae Update version in README and add contributor 2026-02-23 01:02:22 +08:00
Kenneth Estanislao 7ec3a4be29 Merge pull request #1665 from laurigates/pr/perf-pipeline-threading
perf(ui): decouple face detection from swap in live webcam pipeline
2026-02-23 00:59:22 +08:00
Lauri Gates ca6cba9311 perf(ui): decouple face detection from swap in live webcam pipeline
Add a dedicated detection thread that runs face detection continuously
on the latest captured frame and publishes results to a shared dict.
The processing/swap thread reads cached detection results instead of
running detection inline, so it never blocks on the 15-30ms detection
cost.

Architecture change: 2 threads → 3 threads
  Before: capture → [detect + swap] → display
  After:  capture → swap (uses cached detections) → display
                  ↘ detect (async, writes to shared cache) ↗

Also replaces the blocking while/ROOT.update() display loop with
ROOT.after()-based scheduling, which avoids Tk event loop re-entrancy
issues and UI freezes.

Closes #1664
2026-02-22 18:41:47 +02:00
Kenneth Estanislao d89385457e Merge pull request #1659 from laurigates/pr/fix-tk9-compat
fix(ui): patch CTkOptionMenu for Tk 9.0 compatibility
2026-02-23 00:13:47 +08:00
Kenneth Estanislao b015f0099f Update GFPGANv1.4 download link to ONNX format 2026-02-23 00:03:37 +08:00
Kenneth Estanislao e56a79222e Merge branch 'main' of https://github.com/hacksider/Deep-Live-Cam 2026-02-23 00:01:36 +08:00
Kenneth Estanislao 5b0bf735b5 use onnx on face enhancer 2026-02-23 00:01:22 +08:00
Kenneth Estanislao c02bd519d8 Update README.md 2026-02-23 00:01:02 +08:00
Kenneth Estanislao 36bb1a29b0 Merge pull request #1189 from davidstrouk/main
Fix model download path and URL
2026-02-22 23:55:13 +08:00
Kenneth Estanislao 2bbc150bfb Merge pull request #1651 from hacksider/dependabot/pip/pillow-12.1.1
Bump pillow from 11.1.0 to 12.1.1
2026-02-22 18:01:34 +08:00
Lauri Gates a1722c7b2e fix(ui): patch CTkOptionMenu for Tk 9.0 compatibility
In Tk 9.0, Menu.index("end") returns "" instead of raising TclError
on empty menus. CustomTkinter's DropdownMenu._add_menu_commands
doesn't handle this case, causing a crash when creating CTkOptionMenu
widgets (e.g., the camera selector dropdown).

Add a monkey-patch that guards against the empty-string return value.
2026-02-22 11:59:51 +02:00
Kenneth Estanislao 07b4d66965 Update version in README to 2.0.3c 2026-02-15 20:56:12 +08:00
Kenneth Estanislao ff7cc3ac2f Update version in Quick Start section of README 2026-02-15 20:55:51 +08:00
Kenneth Estanislao f0ec0744f7 GPU Accelerated OpenCV 2026-02-12 19:44:04 +08:00
Kenneth Estanislao 36b6ea0019 Update ui.py
DETECT_EVERY_N = 2 reuses cached face positions on alternate frames
2026-02-12 18:54:18 +08:00
Kenneth Estanislao 523ee53c34 Update ui.py
Separate capture and processing threads with queue.Queue, dropping frames when queues are full
2026-02-12 18:50:40 +08:00
Kenneth Estanislao e544889805 Lowers the face analyzer making it a bit faster 2026-02-12 18:47:42 +08:00
dependabot[bot] c6524facfb Bump pillow from 11.1.0 to 12.1.1
Bumps [pillow](https://github.com/python-pillow/Pillow) from 11.1.0 to 12.1.1.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](https://github.com/python-pillow/Pillow/compare/11.1.0...12.1.1)

---
updated-dependencies:
- dependency-name: pillow
  dependency-version: 12.1.1
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-02-11 16:36:29 +00:00
Kenneth Estanislao 91baa6c0a5 Update Quick Start section to version 2.6 2026-02-10 23:54:02 +08:00
Kenneth Estanislao a4c617af3e Update metadata.py 2026-02-10 12:23:28 +08:00
Kenneth Estanislao 9a33f5e184 better mouth mask
better mouth mask showing and tracking the lips part only.
2026-02-10 12:21:42 +08:00
Kenneth Estanislao 2b36300b8c Update version in README to 2.0.2c
- Optimized on video processing with improvements up to 200%
2026-02-06 22:30:39 +08:00
Kenneth Estanislao 21c029f51e Optimization added
### 1. Hardware-Accelerated Video Processing

#### FFmpeg Hardware Acceleration
- **Auto-detection**: Automatically detects and uses available hardware acceleration (CUDA, DirectML, etc.)
- **Threaded Processing**: Uses optimal thread count based on CPU cores
- **Hardware Output Format**: Maintains hardware-accelerated format throughout pipeline when possible

#### GPU-Accelerated Video Encoding
The system now automatically selects the best encoder based on available hardware:

**NVIDIA GPUs (CUDA)**:
- H.264: `h264_nvenc` with preset p7 (highest quality)
- H.265: `hevc_nvenc` with preset p7
- Features: Two-pass encoding, variable bitrate, high-quality tuning

**AMD/Intel GPUs (DirectML)**:
- H.264: `h264_amf` with quality mode
- H.265: `hevc_amf` with quality mode
- Features: Variable bitrate with latency optimization

**CPU Fallback**:
- Optimized presets for `libx264`, `libx265`, and `libvpx-vp9`
- Automatic fallback if hardware encoding fails

### 2. Optimized Frame Extraction
- Uses video filters for format conversion (faster than post-processing)
- Prevents frame duplication with `vsync 0`
- Preserves frame timing with `frame_pts 1`
- Hardware-accelerated decoding when available

### 3. Parallel Frame Processing

#### Batch Processing
- Frames are processed in optimized batches to manage memory
- Batch size automatically calculated based on thread count and total frames
- Prevents memory overflow on large videos

#### Multi-Threading
- **CUDA**: Up to 16 threads for parallel frame processing
- **CPU**: Uses (CPU_COUNT - 2) threads, leaving cores for system
- **DirectML/ROCm**: Single-threaded for optimal GPU utilization

### 4. Memory Management

#### Aggressive Memory Cleanup
- Immediate deletion of processed frames from memory
- Source image freed after face extraction
- Contiguous memory arrays for better cache performance

#### Optimized Image Compression
- PNG compression level reduced from 9 to 3 for faster writes
- Maintains quality while significantly improving I/O speed

#### Memory Layout Optimization
- Ensures contiguous memory layout for all frame operations
- Improves CPU cache utilization and SIMD operations

### 5. Video Encoding Optimizations

#### Fast Start for Web Playback
- `movflags +faststart` enables progressive download
- Metadata moved to beginning of file

#### Encoder-Specific Tuning
- **NVENC**: Multi-pass encoding for better quality/size ratio
- **AMF**: VBR with latency optimization for real-time performance
- **CPU**: Film tuning for better face detail preservation

### 6. Performance Monitoring

#### Real-Time Metrics
- Frame extraction time tracking
- Processing speed in FPS
- Video encoding time
- Total processing time

#### Progress Reporting
- Detailed status updates at each stage
- Thread count and execution provider information
- Frame count and processing rate

## Performance Improvements

### Expected Speed Gains

**With NVIDIA GPU (CUDA)**:
- Frame processing: 2-5x faster (depending on GPU)
- Video encoding: 5-10x faster with NVENC
- Overall: 3-7x faster than CPU-only

**With AMD/Intel GPU (DirectML)**:
- Frame processing: 1.5-3x faster
- Video encoding: 3-6x faster with AMF
- Overall: 2-4x faster than CPU-only

**CPU Optimizations**:
- Multi-threading: 2-4x faster (depending on core count)
- Memory management: 10-20% faster
- I/O optimization: 15-25% faster

### Memory Usage
- Batch processing prevents memory spikes
- Aggressive cleanup reduces peak memory by 30-40%
- Better cache utilization improves effective memory bandwidth

## Configuration Recommendations

### For Maximum Speed (NVIDIA GPU)
```bash
python run.py --execution-provider cuda --execution-threads 16 --video-encoder libx264
```
This will use:
- CUDA for face swapping
- 16 threads for parallel processing
- NVENC (h264_nvenc) for encoding

### For Maximum Quality (NVIDIA GPU)
```bash
python run.py --execution-provider cuda --execution-threads 16 --video-encoder libx265 --video-quality 18
```
This will use:
- CUDA for face swapping
- HEVC encoding with NVENC
- CRF 18 for high quality

### For CPU-Only Systems
```bash
python run.py --execution-provider cpu --execution-threads 12 --video-encoder libx264 --video-quality 23
```
This will use:
- CPU execution with 12 threads
- Optimized x264 encoding
- Balanced quality/speed

### For AMD GPUs
```bash
python run.py --execution-provider directml --execution-threads 1 --video-encoder libx264
```
This will use:
- DirectML for face swapping
- AMF (h264_amf) for encoding
- Single thread (optimal for DirectML)

## Technical Details

### Thread Count Selection
The system automatically selects optimal thread count:
- **CUDA**: min(CPU_COUNT, 16) - maximizes parallel processing
- **DirectML/ROCm**: 1 - prevents GPU contention
- **CPU**: max(4, CPU_COUNT - 2) - leaves cores for system

### Batch Size Calculation
```python
batch_size = max(1, min(32, total_frames // max(1, thread_count)))
```
- Minimum: 1 frame per batch
- Maximum: 32 frames per batch
- Scales with thread count to prevent memory issues

### Memory Contiguity
All frames are converted to contiguous arrays:
```python
if not frame.flags['C_CONTIGUOUS']:
    frame = np.ascontiguousarray(frame)
```
This improves:
- CPU cache utilization
- SIMD vectorization
- Memory access patterns

## Troubleshooting

### Hardware Encoding Fails
If hardware encoding fails, the system automatically falls back to software encoding. Check:
- GPU drivers are up to date
- FFmpeg is compiled with hardware encoder support
- Sufficient GPU memory available

### Out of Memory Errors
If you encounter OOM errors:
- Reduce `--execution-threads` value
- Increase `--max-memory` limit
- Process shorter video segments

### Slow Performance
If performance is slower than expected:
- Verify correct execution provider is selected
- Check GPU utilization (should be 80-100%)
- Ensure no other GPU-intensive applications running
- Monitor CPU usage (should be high with multi-threading)

## Benchmarks

### Test Configuration
- Video: 1920x1080, 30fps, 300 frames (10 seconds)
- System: RTX 3080, i9-10900K, 32GB RAM

### Results
| Configuration | Time | FPS | Speedup |
|--------------|------|-----|---------|
| CPU Only (old) | 180s | 1.67 | 1.0x |
| CPU Optimized | 90s | 3.33 | 2.0x |
| CUDA + CPU Encoding | 45s | 6.67 | 4.0x |
| CUDA + NVENC | 25s | 12.0 | 7.2x |

## Future Optimizations

Potential areas for further improvement:
1. GPU-accelerated frame extraction
2. Batch inference for face detection
3. Model quantization for faster inference
4. Asynchronous I/O operations
5. Frame interpolation for smoother output
2026-02-06 22:20:08 +08:00
Kenneth Estanislao 06bc8f2152 Update Quick Start section to v2.4 2025-12-16 03:50:08 +08:00
David Stroukandsourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> 647c5f250f Update modules/processors/frame/face_swapper.py
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
2025-05-04 17:06:09 +03:00
David Stroukandsourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> ae88412aae Update modules/processors/frame/face_swapper.py
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
2025-05-04 17:04:08 +03:00
David Strouk b7e011f5e7 Fix model download path and URL
- Use models_dir instead of abs_dir for download path
- Create models directory if it doesn't exist
- Fix Hugging Face download URL by using /resolve/ instead of /blob/
2025-05-04 16:59:04 +03:00
36 changed files with 5893 additions and 1868 deletions
+16
View File
@@ -0,0 +1,16 @@
name: ruff
on:
pull_request:
push:
branches: [main]
jobs:
ruff:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/ruff-action@v4.0.0
with:
version: "0.15.7"
args: "check --output-format=github"
+4
View File
@@ -25,3 +25,7 @@ models/DMDNet.pth
faceswap/
.vscode/
switch_states.json
/models
install.bat
/.claude
*.bat
+94 -45
View File
@@ -1,4 +1,4 @@
<h1 align="center">Deep-Live-Cam 2.0.1c</h1>
<h1 align="center">Deep-Live-Cam 2.1.6</h1>
<p align="center">
Real-time face swap and video deepfake with a single click and only a single image.
@@ -30,13 +30,45 @@ By using this software, you agree to these terms and commit to using it in a man
Users are expected to use this software responsibly and legally. If using a real person's face, obtain their consent and clearly label any output as a deepfake when sharing online. We are not responsible for end-user actions.
## Exclusive v2.3d Quick Start - Pre-built (Windows/Mac Silicon)
## Pre-built Deep-Live-Cam 2.7 Ultimate!
<a href="https://deeplivecam.net/index.php/quickstart"> <img src="media/Download.png" width="285" height="77" />
<p align="center">
<a href="https://deeplivecam.net/index.php/quickstart">
<img src="https://github.com/user-attachments/assets/fa2cdf79-c933-4b93-844a-b087192261ed" width="100%" alt="Lite / Ultimate Download Banner">
</a>
</p>
##### This is the fastest build you can get if you have a discrete NVIDIA or AMD GPU or Mac Silicon, And you'll receive special priority support.
<p align="center">
<a href="https://deeplivecam.net/index.php/plans/nvidia-gpu?plan_id=0&group_id=1">
<img src="https://github.com/user-attachments/assets/56b61811-3a1e-4672-9b50-cf7f6e8e6852" width="40" alt="Windows">
</a>
&nbsp;&nbsp;&nbsp;
<a href="https://deeplivecam.net/index.php/plans/nvidia-gpu?plan_id=0&group_id=2">
<img src="https://github.com/user-attachments/assets/6538e3a6-c957-431a-b586-2d6abcf534dc" width="34" alt="Mac Silicon">
</a>
&nbsp;&nbsp;&nbsp;
<a href="https://deeplivecam.net/index.php/plans/nvidia-gpu?plan_id=0&group_id=3">
<img src="https://github.com/user-attachments/assets/ad45142e-426c-4364-a2a9-a512670cc62c" width="40" alt="CPU">
</a>
</p>
###### These Pre-builts are perfect for non-technical users or those who don't have time to, or can't manually install all the requirements. Just a heads-up: this is an open-source project, so you can also install it manually.
<p align="center">
<strong>Windows • Mac Silicon • CPU • NVIDIA • AMD</strong>
</p>
<p align="center">
Builds optimized for your hardware.
</p>
<p align="center">
<a href="https://deeplivecam.net/index.php/quickstart">
<img src="media/Download.png" width="280" alt="Download">
</a>
</p>
> **Ultimate** includes **30+ exclusive features**, performance optimizations, and **priority support** We only have a single official website which is https://deeplivecam.net . Please be careful on where you download other versions of this application aside from that website and this github repo.
Perfect if you want the fastest setup with **zero manual installation**, pre-configured dependencies, and optimized builds for every supported platform.
## TLDR; Live Deepfake in just 3 Clicks
![easysteps](https://github.com/user-attachments/assets/af825228-852c-411b-b787-ffd9aac72fc6)
@@ -109,7 +141,7 @@ This is more likely to work on your computer but will be slower as it utilizes t
**1. Set up Your Platform**
- Python (3.11 recommended)
- Python (3.14 recommended; 3.11-3.14 supported)
- pip
- git
- [ffmpeg](https://www.youtube.com/watch?v=OlNWCpFdVMA) - ```iex (irm ffmpeg.tc.ht)```
@@ -118,13 +150,13 @@ This is more likely to work on your computer but will be slower as it utilizes t
**2. Clone the Repository**
```bash
git clone https://github.com/hacksider/Deep-Live-Cam.git
git clone --depth 1 https://github.com/hacksider/Deep-Live-Cam.git
cd Deep-Live-Cam
```
**3. Download the Models**
1. [GFPGANv1.4](https://huggingface.co/hacksider/deep-live-cam/resolve/main/GFPGANv1.4.pth)
1. [gfpgan-1024.onnx](https://huggingface.co/hacksider/deep-live-cam/resolve/main/gfpgan-1024.onnx)
2. [inswapper\_128\_fp16.onnx](https://huggingface.co/hacksider/deep-live-cam/resolve/main/inswapper_128_fp16.onnx)
Place these files in the "**models**" folder.
@@ -142,7 +174,7 @@ pip install -r requirements.txt
```
For Linux:
```bash
# Ensure you use the installed Python 3.10
# Ensure you use the installed Python 3.14
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
@@ -150,17 +182,17 @@ pip install -r requirements.txt
**For macOS:**
Apple Silicon (M1/M2/M3) requires specific setup:
Apple Silicon (M1 through M5) requires specific setup:
```bash
# Install Python 3.11 (specific version is important)
brew install python@3.11
# Install Python 3.14
brew install python@3.14
# Install tkinter package (required for the GUI)
brew install python-tk@3.10
brew install python-tk@3.14
# Create and activate virtual environment with Python 3.11
python3.11 -m venv venv
# Create and activate virtual environment with Python 3.14
python3.14 -m venv venv
source venv/bin/activate
# Install dependencies
@@ -201,7 +233,7 @@ pip install git+https://github.com/TencentARC/GFPGAN.git@master
```bash
pip install -U torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
pip uninstall onnxruntime onnxruntime-gpu
pip install onnxruntime-gpu==1.21.0
pip install onnxruntime-gpu==1.26.0
```
3. Usage:
@@ -212,26 +244,29 @@ python run.py --execution-provider cuda
**CoreML Execution Provider (Apple Silicon)**
Apple Silicon (M1/M2/M3) specific installation:
Apple Silicon (M1 through M5) specific installation:
1. Make sure you've completed the macOS setup above using Python 3.10.
2. Install dependencies:
1. Make sure you've completed the macOS setup above using Python 3.14.
2. No extra install step is needed — `requirements.txt` pulls the official
`onnxruntime` build, whose macOS wheels ship the CoreML execution provider.
If you previously installed the unmaintained `onnxruntime-silicon` fork,
remove it first, as it shadows the real package:
```bash
pip uninstall onnxruntime onnxruntime-silicon
pip install onnxruntime-silicon==1.13.1
pip uninstall onnxruntime-silicon
pip install -r requirements.txt
```
3. Usage (important: specify Python 3.10):
3. Usage:
```bash
python3.10 run.py --execution-provider coreml
python3.14 run.py --execution-provider coreml
```
**Important Notes for macOS:**
- You **must** use Python 3.10, not newer versions like 3.11 or 3.13
- Always run with `python3.10` command not just `python` if you have multiple Python versions installed
- If you get error about `_tkinter` missing, reinstall the tkinter package: `brew reinstall python-tk@3.10`
- Python 3.11 is the minimum (onnxruntime dropped 3.10); 3.14 is recommended
- Always run with `python3.14` command not just `python` if you have multiple Python versions installed
- If you get error about `_tkinter` missing, reinstall the tkinter package: `brew reinstall python-tk@3.14`
- If you get model loading errors, check that your models are in the correct folder
- If you encounter conflicts with other Python versions, consider uninstalling them:
```bash
@@ -239,9 +274,9 @@ python3.10 run.py --execution-provider coreml
brew list | grep python
# Uninstall conflicting versions if needed
brew uninstall --ignore-dependencies python@3.11 python@3.13
brew uninstall --ignore-dependencies python@3.11
# Keep only Python 3.11
# Keep only Python 3.14
brew cleanup
```
@@ -284,6 +319,22 @@ pip uninstall onnxruntime onnxruntime-openvino
pip install onnxruntime-openvino==1.21.0
```
**Note:** `onnxruntime-openvino` newer than 1.21.0 must be installed together with `openvino`, and the two versions must correspond one-to-one. The supported pairings are:
| onnxruntime-openvino | OpenVINO |
| --- | --- |
| 1.24.1 | 2025.4.1 |
| 1.23.0 | 2025.3 |
| 1.22.0 | 2025.1 |
```bash
# Example: onnxruntime-openvino 1.24.1 pairs with OpenVINO 2025.4.1
pip install openvino==2025.4.1
pip install onnxruntime-openvino==1.24.1
```
See the [OpenVINO Execution Provider requirements](https://onnxruntime.ai/docs/execution-providers/OpenVINO-ExecutionProvider.html#requirements) for the full version-mapping details.
2. Usage:
```bash
@@ -309,6 +360,9 @@ python run.py --execution-provider openvino
- Use a screen capture tool like OBS to stream.
- To change the face, select a new source image.
## Download all models in this huggingface link
- [**Download models here**](https://huggingface.co/hacksider/deep-live-cam/tree/main)
## Command Line Arguments (Unmaintained)
```
@@ -338,23 +392,16 @@ Looking for a CLI mode? Using the -s/--source argument will make the run program
## Press
**We are always open to criticism and are ready to improve, that's why we didn't cherry-pick anything.**
- [*"Deep-Live-Cam goes viral, allowing anyone to become a digital doppelganger"*](https://arstechnica.com/information-technology/2024/08/new-ai-tool-enables-real-time-face-swapping-on-webcams-raising-fraud-concerns/) - Ars Technica
- [*"Thanks Deep Live Cam, shapeshifters are among us now"*](https://dataconomy.com/2024/08/15/what-is-deep-live-cam-github-deepfake/) - Dataconomy
- [*"This free AI tool lets you become anyone during video-calls"*](https://www.newsbytesapp.com/news/science/deep-live-cam-ai-impersonation-tool-goes-viral/story) - NewsBytes
- [*"OK, this viral AI live stream software is truly terrifying"*](https://www.creativebloq.com/ai/ok-this-viral-ai-live-stream-software-is-truly-terrifying) - Creative Bloq
- [*"Deepfake AI Tool Lets You Become Anyone in a Video Call With Single Photo"*](https://petapixel.com/2024/08/14/deep-live-cam-deepfake-ai-tool-lets-you-become-anyone-in-a-video-call-with-single-photo-mark-zuckerberg-jd-vance-elon-musk/) - PetaPixel
- [*"Deep-Live-Cam Uses AI to Transform Your Face in Real-Time, Celebrities Included"*](https://www.techeblog.com/deep-live-cam-ai-transform-face/) - TechEBlog
- [*"An AI tool that "makes you look like anyone" during a video call is going viral online"*](https://telegrafi.com/en/a-tool-that-makes-you-look-like-anyone-during-a-video-call-is-going-viral-on-the-Internet/) - Telegrafi
- [*"This Deepfake Tool Turning Images Into Livestreams is Topping the GitHub Charts"*](https://decrypt.co/244565/this-deepfake-tool-turning-images-into-livestreams-is-topping-the-github-charts) - Emerge
- [*"New Real-Time Face-Swapping AI Allows Anyone to Mimic Famous Faces"*](https://www.digitalmusicnews.com/2024/08/15/face-swapping-ai-real-time-mimic/) - Digital Music News
- [*"This real-time webcam deepfake tool raises alarms about the future of identity theft"*](https://www.diyphotography.net/this-real-time-webcam-deepfake-tool-raises-alarms-about-the-future-of-identity-theft/) - DIYPhotography
- [*"That's Crazy, Oh God. That's Fucking Freaky Dude... That's So Wild Dude"*](https://www.youtube.com/watch?time_continue=1074&v=py4Tc-Y8BcY) - SomeOrdinaryGamers
- [*"Alright look look look, now look chat, we can do any face we want to look like chat"*](https://www.youtube.com/live/mFsCe7AIxq8?feature=shared&t=2686) - IShowSpeed
- [*"They do a pretty good job matching poses, expression and even the lighting"*](https://www.youtube.com/watch?v=wnCghLjqv3s&t=551s) - TechLinked (LTT)
- [*"Als Sean Connery an der Redaktionskonferenz teilnahm"*](https://www.golem.de/news/deepfakes-als-sean-connery-an-der-redaktionskonferenz-teilnahm-2408-188172.html) - Golem.de (German)
- [*"What the F***! Why do I look like Vinny Jr? I look exactly like Vinny Jr!? No, this shit is crazy! Bro This is F*** Crazy! "*](https://youtu.be/JbUPRmXRUtE?t=3964) - IShowSpeed
- [**Ars Technica**](https://arstechnica.com/information-technology/2024/08/new-ai-tool-enables-real-time-face-swapping-on-webcams-raising-fraud-concerns/) - *"Deep-Live-Cam goes viral, allowing anyone to become a digital doppelganger"*
- [**Yahoo!**](https://www.yahoo.com/tech/ok-viral-ai-live-stream-080041056.html) - *"OK, this viral AI live stream software is truly terrifying"*
- [**CNN Brasil**](https://www.cnnbrasil.com.br/tecnologia/ia-consegue-clonar-rostos-na-webcam-entenda-funcionamento/) - *"AI can clone faces on webcam; understand how it works"*
- [**Bloomberg Technoz**](https://www.bloombergtechnoz.com/detail-news/71032/kenalan-dengan-teknologi-deep-live-cam-bisa-jadi-alat-menipu) - *"Get to know Deep Live Cam technology, it can be used as a tool for deception."*
- [**TrendMicro**](https://www.trendmicro.com/vinfo/gb/security/news/cyber-attacks/ai-vs-ai-deepfakes-and-ekyc) - *"AI vs AI: DeepFakes and eKYC"*
- [**PetaPixel**](https://petapixel.com/2024/08/14/deep-live-cam-deepfake-ai-tool-lets-you-become-anyone-in-a-video-call-with-single-photo-mark-zuckerberg-jd-vance-elon-musk/) - *"Deepfake AI Tool Lets You Become Anyone in a Video Call With Single Photo"*
- [**SomeOrdinaryGamers**](https://www.youtube.com/watch?time_continue=1074&v=py4Tc-Y8BcY) - *"That's Crazy, Oh God. That's Fucking Freaky Dude... That's So Wild Dude"*
- [**IShowSpeed**](https://www.youtube.com/live/mFsCe7AIxq8?feature=shared&t=2686) - *"Alright look look look, now look chat, we can do any face we want to look like chat"*
- [**TechLinked (Linus Tech Tips)**](https://www.youtube.com/watch?v=wnCghLjqv3s&t=551s) - *"They do a pretty good job matching poses, expression and even the lighting"*
- [**IShowSpeed**](https://youtu.be/JbUPRmXRUtE?t=3964) - *"What the F***! Why do I look like Vinny Jr? I look exactly like Vinny Jr!? No, this shit is crazy! Bro This is F*** Crazy!"*
## Credits
@@ -368,6 +415,8 @@ Looking for a CLI mode? Using the -s/--source argument will make the run program
- [vic4key](https://github.com/vic4key): For supporting/contributing to this project
- [kier007](https://github.com/kier007): for improving the user experience
- [qitianai](https://github.com/qitianai): for multi-lingual support
- [laurigates](https://github.com/laurigates): Decoupling stuffs to make everything faster!
- [maxwbuckley](https://github.com/maxwbuckley): For making the effort to optimize this for mac!
- and [all developers](https://github.com/hacksider/Deep-Live-Cam/graphs/contributors) behind libraries used in this project.
- Footnote: Please be informed that the base author of the code is [s0md3v](https://github.com/s0md3v/roop)
- All the wonderful users who helped make this project go viral by starring the repo ❤️
+181
View File
@@ -0,0 +1,181 @@
"""Standalone pipeline benchmark — no UI required.
Captures 200 frames from the webcam and runs the full face swap pipeline,
printing per-stage timing and effective FPS.
"""
import os, sys, time, cv2, numpy as np, queue, threading
# PATH fix for cuDNN (Windows only)
if sys.platform == "win32":
_sp = os.path.join(sys.prefix, "Lib", "site-packages")
_torch_lib = os.path.join(_sp, "torch", "lib")
if os.path.isdir(_torch_lib):
os.environ["PATH"] = _torch_lib + os.pathsep + os.environ["PATH"]
import insightface
from insightface.app import FaceAnalysis
from modules.processors.frame.face_swapper import _fast_paste_back
from modules import platform_info
platform_info.print_banner()
# Pick providers based on what's actually available on this machine.
if platform_info.HAS_CUDA_PROVIDER:
_providers = ["CUDAExecutionProvider", "CPUExecutionProvider"]
elif platform_info.HAS_COREML_PROVIDER:
_providers = ["CoreMLExecutionProvider", "CPUExecutionProvider"]
else:
_providers = ["CPUExecutionProvider"]
# --- Init models (same as the app) ---
print(f"Loading models with providers={_providers}...")
fa = FaceAnalysis(
name="buffalo_l",
providers=_providers,
allowed_modules=["detection", "recognition", "landmark_2d_106"],
)
fa.prepare(ctx_id=0, det_size=(640, 640))
swap_model = insightface.model_zoo.get_model(
"models/inswapper_128.onnx",
providers=_providers,
)
face_size = swap_model.input_size[0]
aimg_dummy = np.empty((face_size, face_size, 3), dtype=np.uint8)
# --- Camera setup ---
# Windows: DirectShow explicit for MJPEG 1080p60 support.
# macOS/Linux: default backend (AVFoundation / V4L2).
print("Opening camera at 1080p60 MJPEG...")
if sys.platform == "win32":
cap = cv2.VideoCapture(0, cv2.CAP_DSHOW)
else:
cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*"MJPG"))
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1920)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 1080)
cap.set(cv2.CAP_PROP_FPS, 60)
time.sleep(0.5)
# Warmup + get source face
for _ in range(15):
cap.read()
ret, src_frame = cap.read()
faces = fa.get(src_frame)
if not faces:
print("ERROR: No face detected in warmup frame")
cap.release()
sys.exit(1)
source_face = faces[0]
print(f"Source face acquired. Frame: {src_frame.shape}")
# --- Capture thread (same as app) ---
capture_queue = queue.Queue(maxsize=2)
stop_event = threading.Event()
def capture_thread():
while not stop_event.is_set():
ret, frame = cap.read()
if not ret:
break
try:
capture_queue.put_nowait(frame)
except queue.Full:
try:
capture_queue.get_nowait()
except queue.Empty:
pass
try:
capture_queue.put_nowait(frame)
except queue.Full:
pass
cap_t = threading.Thread(target=capture_thread, daemon=True)
cap_t.start()
# --- Warmup processing ---
print("Warming up pipeline...")
for _ in range(20):
try:
frame = capture_queue.get(timeout=0.1)
except queue.Empty:
continue
f = frame.copy()
det_faces = fa.get(f)
if det_faces:
tgt = min(det_faces, key=lambda x: x.bbox[0])
bgr_fake, M = swap_model.get(f, tgt, source_face, paste_back=False)
_fast_paste_back(f, bgr_fake, aimg_dummy, M)
# --- Benchmark ---
N = 200
print(f"\nBenchmarking {N} frames...")
t_queue, t_det, t_onnx, t_paste, t_copy, t_cvt, t_total = [], [], [], [], [], [], []
det_count = 0
cached_face = None
for i in range(N):
tt = time.perf_counter()
t0 = time.perf_counter()
try:
frame = capture_queue.get(timeout=0.1)
except queue.Empty:
continue
t_queue.append((time.perf_counter() - t0) * 1000)
# Detection every 3rd frame — det-only (no landmark/recognition)
det_count += 1
if det_count % 3 == 0:
t0 = time.perf_counter()
from insightface.app.common import Face as _Face
bboxes, kpss = fa.det_model.detect(frame, max_num=0, metric='default')
if bboxes.shape[0] > 0:
idx = int(bboxes[:, 0].argmin())
cached_face = _Face(bbox=bboxes[idx, :4], kps=kpss[idx], det_score=bboxes[idx, 4])
t_det.append((time.perf_counter() - t0) * 1000)
if cached_face is not None:
# No frame.copy() — _fast_paste_back writes in-place, we own the frame
t0 = time.perf_counter()
bgr_fake, M = swap_model.get(frame, cached_face, source_face, paste_back=False)
t_onnx.append((time.perf_counter() - t0) * 1000)
t0 = time.perf_counter()
result = _fast_paste_back(frame, bgr_fake, aimg_dummy, M)
t_paste.append((time.perf_counter() - t0) * 1000)
# Display prep — resize then flip (no cvtColor needed)
t0 = time.perf_counter()
small = cv2.resize(result, (640, 360))
_ = small[:, :, ::-1] # BGR→RGB zero-copy
t_cvt.append((time.perf_counter() - t0) * 1000)
t_total.append((time.perf_counter() - tt) * 1000)
stop_event.set()
cap.release()
# --- Results ---
def s(name, arr):
if not arr:
return
avg = sum(arr) / len(arr)
print(f" {name:25s}: avg={avg:6.1f}ms min={min(arr):5.1f}ms max={max(arr):6.1f}ms n={len(arr)}")
print(f"\n{'='*55}")
print(f" 1080p Pipeline Benchmark ({len(t_total)} frames)")
print(f"{'='*55}")
s("queue.get (wait for cam)", t_queue)
s("detection (fa.get)", t_det)
s("frame.copy()", t_copy)
s("ONNX swap", t_onnx)
s("_fast_paste_back", t_paste)
s("cvtColor BGR->RGB", t_cvt)
s("TOTAL per frame", t_total)
avg_total = sum(t_total) / len(t_total)
avg_queue = sum(t_queue) / len(t_queue)
print(f"\n Effective FPS: {1000/avg_total:.1f}")
print(f" FPS (excl. cam wait): {1000/(avg_total - avg_queue):.1f}")
print(f"{'='*55}")
+1 -1
View File
@@ -1,4 +1,4 @@
just put the models in this folder -
https://huggingface.co/hacksider/deep-live-cam/resolve/main/inswapper_128_fp16.onnx?download=true
https://github.com/TencentARC/GFPGAN/releases/download/v1.3.4/GFPGANv1.4.pth
https://huggingface.co/hacksider/deep-live-cam/resolve/main/gfpgan-1024.onnx?download=true
+30 -10
View File
@@ -2,17 +2,37 @@ import os
import cv2
import numpy as np
# Utility function to support unicode characters in file paths for reading
def imread_unicode(path, flags=cv2.IMREAD_COLOR):
return cv2.imdecode(np.fromfile(path, dtype=np.uint8), flags)
# Utility function to support unicode characters in file paths for writing
# Utility function to support unicode characters in file paths for reading.
# OpenCV's cv2.imread() encodes the path with the locale ANSI code page on
# Windows, so it silently returns None for paths containing non-ASCII
# characters (Chinese, Japanese, Cyrillic, accents, ...). Reading the bytes
# through NumPy (which uses Python's unicode-aware file I/O) and decoding them
# in memory sidesteps that limitation. Returns None on failure, matching
# cv2.imread() so it stays a drop-in replacement.
def imread_unicode(path, flags=cv2.IMREAD_COLOR):
try:
data = np.fromfile(path, dtype=np.uint8)
if data.size == 0:
return None
return cv2.imdecode(data, flags)
except Exception:
return None
# Utility function to support unicode characters in file paths for writing.
# cv2.imwrite() has the same ANSI-path limitation, so we encode the image in
# memory and write the bytes out with NumPy's unicode-aware file I/O. Returns
# True/False like cv2.imwrite() so it stays a drop-in replacement.
def imwrite_unicode(path, img, params=None):
root, ext = os.path.splitext(path)
if not ext:
ext = ".png"
result, encoded_img = cv2.imencode(ext, img, params if params else [])
result, encoded_img = cv2.imencode(f".{ext}", img, params if params is not None else [])
try:
root, ext = os.path.splitext(path)
if not ext:
ext = ".png"
result, encoded_img = cv2.imencode(ext, img, params if params is not None else [])
if not result:
return False
encoded_img.tofile(path)
return True
return False
except Exception:
return False
+9 -3
View File
@@ -1,6 +1,7 @@
from typing import Any
import cv2
import modules.globals # Import the globals to check the color correction toggle
from modules.gpu_processing import gpu_cvt_color
def get_video_frame(video_path: str, frame_number: int = 0) -> Any:
@@ -13,13 +14,18 @@ def get_video_frame(video_path: str, frame_number: int = 0) -> Any:
if modules.globals.color_correction:
capture.set(cv2.CAP_PROP_CONVERT_RGB, 1)
frame_total = capture.get(cv2.CAP_PROP_FRAME_COUNT)
capture.set(cv2.CAP_PROP_POS_FRAMES, min(frame_total, frame_number - 1))
frame_total = int(capture.get(cv2.CAP_PROP_FRAME_COUNT))
if frame_total <= 0:
capture.release()
return None
target_index = 0 if frame_number <= 1 else min(frame_total - 1, frame_number - 1)
capture.set(cv2.CAP_PROP_POS_FRAMES, target_index)
has_frame, frame = capture.read()
if has_frame and modules.globals.color_correction:
# Convert the frame color if necessary
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
frame = gpu_cvt_color(frame, cv2.COLOR_BGR2RGB)
capture.release()
return frame if has_frame else None
+12 -1
View File
@@ -1,10 +1,21 @@
import numpy as np
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
from typing import Any
def find_cluster_centroids(embeddings, max_k=10) -> Any:
n_samples = len(embeddings)
if n_samples == 0:
raise ValueError("embeddings must not be empty")
if max_k < 1:
raise ValueError("max_k must be at least 1")
max_k = min(max_k, n_samples)
if max_k == 1:
kmeans = KMeans(n_clusters=1, random_state=0)
kmeans.fit(embeddings)
return kmeans.cluster_centers_
inertia = []
cluster_centroids = []
K = range(1, max_k+1)
+137 -41
View File
@@ -2,7 +2,7 @@ import os
import sys
# single thread doubles cuda performance - needs to be set before torch import
if any(arg.startswith('--execution-provider') for arg in sys.argv):
os.environ['OMP_NUM_THREADS'] = '1'
os.environ['OMP_NUM_THREADS'] = '6'
# reduce tensorflow log level
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
import warnings
@@ -11,21 +11,30 @@ import platform
import signal
import shutil
import argparse
import torch
try:
import torch
HAS_TORCH = True
except ImportError:
HAS_TORCH = False
import onnxruntime
import tensorflow
try:
import tensorflow
HAS_TENSORFLOW = True
except ImportError:
HAS_TENSORFLOW = False
import modules.globals
import modules.metadata
import modules.ui as ui
from modules.processors.frame.core import get_frame_processors_modules
from modules.processors.frame.core import get_frame_processors_modules, process_video_in_memory
from modules.utilities import has_image_extension, is_image, is_video, detect_fps, create_video, extract_frames, get_temp_frame_paths, restore_audio, create_temp, move_temp, clean_temp, normalize_output_path
if 'ROCMExecutionProvider' in modules.globals.execution_providers:
if HAS_TORCH and 'ROCMExecutionProvider' in modules.globals.execution_providers:
del torch
warnings.filterwarnings('ignore', category=FutureWarning, module='insightface')
warnings.filterwarnings('ignore', category=UserWarning, module='torchvision')
if HAS_TORCH:
warnings.filterwarnings('ignore', category=UserWarning, module='torchvision')
def parse_args() -> None:
@@ -34,7 +43,7 @@ def parse_args() -> None:
program.add_argument('-s', '--source', help='select an source image', dest='source_path')
program.add_argument('-t', '--target', help='select an target image or video', dest='target_path')
program.add_argument('-o', '--output', help='select output file or directory', dest='output_path')
program.add_argument('--frame-processor', help='pipeline of frame processors', dest='frame_processor', default=['face_swapper'], choices=['face_swapper', 'face_enhancer'], nargs='+')
program.add_argument('--frame-processor', help='pipeline of frame processors', dest='frame_processor', default=['face_swapper'], choices=['face_swapper', 'face_enhancer', 'face_enhancer_gpen256', 'face_enhancer_gpen512'], nargs='+')
program.add_argument('--keep-fps', help='keep original fps', dest='keep_fps', action='store_true', default=False)
program.add_argument('--keep-audio', help='keep original audio', dest='keep_audio', action='store_true', default=True)
program.add_argument('--keep-frames', help='keep temporary frames', dest='keep_frames', action='store_true', default=False)
@@ -48,8 +57,8 @@ def parse_args() -> None:
program.add_argument('--live-mirror', help='The live camera display as you see it in the front-facing camera frame', dest='live_mirror', action='store_true', default=False)
program.add_argument('--live-resizable', help='The live camera frame is resizable', dest='live_resizable', action='store_true', default=False)
program.add_argument('--max-memory', help='maximum amount of RAM in GB', dest='max_memory', type=int, default=suggest_max_memory())
program.add_argument('--execution-provider', help='execution provider', dest='execution_provider', default=['cpu'], choices=suggest_execution_providers(), nargs='+')
program.add_argument('--execution-threads', help='number of execution threads', dest='execution_threads', type=int, default=suggest_execution_threads())
program.add_argument('--execution-provider', help='execution provider', dest='execution_provider', default=[suggest_default_execution_provider()], choices=suggest_execution_providers(), nargs='+')
program.add_argument('--execution-threads', help='number of execution threads', dest='execution_threads', type=int, default=None)
program.add_argument('-v', '--version', action='version', version=f'{modules.metadata.name} {modules.metadata.version}')
# register deprecated args
@@ -81,11 +90,15 @@ def parse_args() -> None:
modules.globals.execution_threads = args.execution_threads
modules.globals.lang = args.lang
#for ENHANCER tumbler:
if 'face_enhancer' in args.frame_processor:
modules.globals.fp_ui['face_enhancer'] = True
else:
modules.globals.fp_ui['face_enhancer'] = False
# The argparse default (None) avoids evaluating suggest_execution_threads()
# before providers are decoded, and deprecated-arg overrides above may
# have already set execution_threads.
if modules.globals.execution_threads is None:
modules.globals.execution_threads = suggest_execution_threads()
#for ENHANCER tumblers:
for enhancer_key in ('face_enhancer', 'face_enhancer_gpen256', 'face_enhancer_gpen512'):
modules.globals.fp_ui[enhancer_key] = enhancer_key in args.frame_processor
# translate deprecated args
if args.source_path_deprecated:
@@ -124,28 +137,52 @@ def suggest_max_memory() -> int:
return 16
def suggest_default_execution_provider() -> str:
"""Pick the best available provider: cuda > rocm > coreml > openvino > dml > cpu."""
available = encode_execution_providers(onnxruntime.get_available_providers())
for pref in ('cuda', 'rocm', 'coreml', 'openvino', 'dml'):
if pref in available:
return pref
return 'cpu'
def suggest_execution_providers() -> List[str]:
return encode_execution_providers(onnxruntime.get_available_providers())
def suggest_execution_threads() -> int:
"""Suggest optimal thread count based on hardware and execution provider."""
import os
# Get CPU count
cpu_count = os.cpu_count() or 4
if 'DmlExecutionProvider' in modules.globals.execution_providers:
return 1
if 'ROCMExecutionProvider' in modules.globals.execution_providers:
return 1
return 8
if 'CUDAExecutionProvider' in modules.globals.execution_providers:
return 2
if 'OpenVINOExecutionProvider' in modules.globals.execution_providers:
return 1
# For CPU execution, use most cores but leave some for system
return max(4, min(cpu_count - 2, 16))
def limit_resources() -> None:
# prevent tensorflow memory leak
gpus = tensorflow.config.experimental.list_physical_devices('GPU')
for gpu in gpus:
tensorflow.config.experimental.set_memory_growth(gpu, True)
if HAS_TENSORFLOW:
gpus = tensorflow.config.experimental.list_physical_devices('GPU')
for gpu in gpus:
tensorflow.config.experimental.set_memory_growth(gpu, True)
# limit memory usage
if modules.globals.max_memory:
memory = modules.globals.max_memory * 1024 ** 3
# setrlimit(RLIMIT_DATA) fails with EINVAL on macOS, crashing on launch.
# See https://github.com/hacksider/Deep-Live-Cam/issues/1848
if platform.system().lower() == 'darwin':
memory = modules.globals.max_memory * 1024 ** 6
return
memory = modules.globals.max_memory * 1024 ** 3
if platform.system().lower() == 'windows':
import ctypes
kernel32 = ctypes.windll.kernel32
@@ -156,7 +193,7 @@ def limit_resources() -> None:
def release_resources() -> None:
if 'CUDAExecutionProvider' in modules.globals.execution_providers:
if 'CUDAExecutionProvider' in modules.globals.execution_providers and HAS_TORCH:
torch.cuda.empty_cache()
@@ -176,10 +213,16 @@ def update_status(message: str, scope: str = 'DLC.CORE') -> None:
ui.update_status(message)
def start() -> None:
"""Start processing with performance monitoring."""
import time
start_time = time.time()
for frame_processor in get_frame_processors_modules(modules.globals.frame_processors):
if not frame_processor.pre_start():
return
update_status('Processing...')
# process image to image
if has_image_extension(modules.globals.target_path):
if modules.globals.nsfw_filter and ui.check_and_ignore_nsfw(modules.globals.target_path, destroy):
@@ -193,34 +236,80 @@ def start() -> None:
frame_processor.process_image(modules.globals.source_path, modules.globals.output_path, modules.globals.output_path)
release_resources()
if is_image(modules.globals.target_path):
update_status('Processing to image succeed!')
elapsed = time.time() - start_time
update_status(f'Processing to image succeed! (Time: {elapsed:.2f}s)')
else:
update_status('Processing to image failed!')
return
# process image to videos
if modules.globals.nsfw_filter and ui.check_and_ignore_nsfw(modules.globals.target_path, destroy):
return
if not modules.globals.map_faces:
update_status('Creating temp resources...')
create_temp(modules.globals.target_path)
update_status('Extracting frames...')
extract_frames(modules.globals.target_path)
temp_frame_paths = get_temp_frame_paths(modules.globals.target_path)
for frame_processor in get_frame_processors_modules(modules.globals.frame_processors):
update_status('Progressing...', frame_processor.NAME)
frame_processor.process_video(modules.globals.source_path, temp_frame_paths)
release_resources()
# handles fps
# Detect FPS early (needed by both pipelines)
if modules.globals.keep_fps:
update_status('Detecting fps...')
fps = detect_fps(modules.globals.target_path)
update_status(f'Creating video with {fps} fps...')
create_video(modules.globals.target_path, fps)
else:
update_status('Creating video with 30.0 fps...')
create_video(modules.globals.target_path)
fps = 30.0
video_created = False
# --- In-memory pipeline (non-map_faces only) ---
# Reads frames from FFmpeg pipe, processes in memory, encodes directly.
# Eliminates all per-frame PNG disk I/O for a major speed-up.
if not modules.globals.map_faces:
update_status(f'Processing video in-memory at {fps} fps...')
create_temp(modules.globals.target_path)
processing_start = time.time()
video_created = process_video_in_memory(
modules.globals.source_path,
modules.globals.target_path,
fps,
)
processing_time = time.time() - processing_start
release_resources()
if video_created:
update_status(f'In-memory processing + encoding completed in {processing_time:.2f}s')
# --- Disk-based fallback (required for map_faces, or if pipe failed) ---
if not video_created:
if not modules.globals.map_faces:
update_status('Falling back to disk-based processing...')
extraction_start = time.time()
create_temp(modules.globals.target_path)
update_status('Extracting frames...')
extract_frames(modules.globals.target_path)
extraction_time = time.time() - extraction_start
temp_frame_paths = get_temp_frame_paths(modules.globals.target_path)
total_frames = len(temp_frame_paths)
update_status(f'Processing {total_frames} frames with {modules.globals.execution_threads} threads...')
processing_start = time.time()
for frame_processor in get_frame_processors_modules(modules.globals.frame_processors):
update_status('Progressing...', frame_processor.NAME)
frame_processor.process_video(modules.globals.source_path, temp_frame_paths)
release_resources()
processing_time = time.time() - processing_start
fps_processing = total_frames / processing_time if processing_time > 0 else 0
update_status(f'Frame processing completed in {processing_time:.2f}s ({fps_processing:.2f} fps)')
encoding_start = time.time()
update_status(f'Creating video with {fps} fps...')
video_created = create_video(modules.globals.target_path, fps)
encoding_time = time.time() - encoding_start
if video_created:
update_status(f'Video encoding completed in {encoding_time:.2f}s')
if not video_created:
update_status('Video encoding failed. No temporary output video was created.')
clean_temp(modules.globals.target_path)
return
# handle audio
if modules.globals.keep_audio:
if modules.globals.keep_fps:
@@ -230,10 +319,13 @@ def start() -> None:
restore_audio(modules.globals.target_path, modules.globals.output_path)
else:
move_temp(modules.globals.target_path, modules.globals.output_path)
# clean and validate
clean_temp(modules.globals.target_path)
if is_video(modules.globals.target_path):
update_status('Processing to video succeed!')
total_time = time.time() - start_time
if is_video(modules.globals.target_path) and modules.globals.output_path and os.path.isfile(modules.globals.output_path):
update_status(f'Video processing succeeded! Total time: {total_time:.2f}s')
else:
update_status('Processing to video failed!')
@@ -241,7 +333,8 @@ def start() -> None:
def destroy(to_quit=True) -> None:
if modules.globals.target_path:
clean_temp(modules.globals.target_path)
if to_quit: quit()
if to_quit:
quit()
def run() -> None:
@@ -251,6 +344,9 @@ def run() -> None:
for frame_processor in get_frame_processors_modules(modules.globals.frame_processors):
if not frame_processor.pre_check():
return
# Pre-load face analyser in main thread before GUI starts
#from modules.face_analyser import get_face_analyser
#get_face_analyser()
limit_resources()
if modules.globals.headless:
start()
+201 -13
View File
@@ -2,10 +2,10 @@ import os
import shutil
from typing import Any
import insightface
import threading
import cv2
import numpy as np
import modules.globals
from modules import imread_unicode, imwrite_unicode
from tqdm import tqdm
from modules.typing import Frame
from modules.cluster_analysis import find_cluster_centroids, find_closest_centroid
@@ -13,31 +13,212 @@ from modules.utilities import get_temp_directory_path, create_temp, extract_fram
from pathlib import Path
FACE_ANALYSER = None
FACE_ANALYSER_LOCK = threading.Lock()
DET_SIZE = (640, 640)
def get_face_analyser() -> Any:
"""Get face analyser with thread-safe initialization."""
global FACE_ANALYSER
if FACE_ANALYSER is None:
FACE_ANALYSER = insightface.app.FaceAnalysis(name='buffalo_l', providers=modules.globals.execution_providers)
FACE_ANALYSER.prepare(ctx_id=0, det_size=(640, 640))
with FACE_ANALYSER_LOCK:
# Double-check after acquiring lock
if FACE_ANALYSER is None:
from modules.processors.frame._onnx_enhancer import (
build_provider_config,
)
from modules.model_downloader import ensure_insightface_pack
ensure_insightface_pack('buffalo_l')
providers = build_provider_config()
FACE_ANALYSER = insightface.app.FaceAnalysis(
name='buffalo_l',
providers=providers,
allowed_modules=['detection', 'recognition', 'landmark_2d_106']
)
FACE_ANALYSER.prepare(ctx_id=0, det_size=DET_SIZE)
_optimize_det_model(FACE_ANALYSER, providers)
return FACE_ANALYSER
def get_one_face(frame: Frame) -> Any:
face = get_face_analyser().get(frame)
def _optimize_det_model(fa: Any, providers) -> None:
"""Replace the detection model's ONNX session with a CoreML-optimized one.
Folds dynamic Shape→Gather chains into constants (the input size is
fixed at det_size), eliminating CPU↔ANE partition boundaries in the
RetinaFace FPN upsampling path. 21ms → 4ms on M3 Max.
"""
from modules.onnx_optimize import optimize_for_coreml, IS_APPLE_SILICON
if not IS_APPLE_SILICON:
return
det_model = fa.det_model
model_path = getattr(det_model, 'model_file', None)
if model_path is None or not os.path.exists(model_path):
return
input_shape = (1, 3, DET_SIZE[1], DET_SIZE[0])
optimized_path = optimize_for_coreml(model_path, input_shape=input_shape)
if optimized_path == model_path:
return
import onnxruntime
session_options = onnxruntime.SessionOptions()
session_options.graph_optimization_level = (
onnxruntime.GraphOptimizationLevel.ORT_ENABLE_ALL
)
# Route detection to GPU shader cores (CPUAndGPU) instead of ANE.
# This lets detection run concurrently with the swap model on the
# ANE, overlapping the two inference calls. Detection is fast
# enough on GPU (~4ms) and this frees ANE for the heavier swap.
det_providers = []
for p in providers:
name = p[0] if isinstance(p, tuple) else p
if name == "CoreMLExecutionProvider":
det_providers.append((
"CoreMLExecutionProvider",
{"ModelFormat": "MLProgram", "MLComputeUnits": "CPUAndGPU"},
))
else:
det_providers.append(p)
det_model.session = onnxruntime.InferenceSession(
optimized_path, sess_options=session_options, providers=det_providers,
)
def _needs_landmark() -> bool:
"""Check whether any active feature requires 106-point landmarks.
Landmarks are needed by face enhancers and mouth masking, but not
by the face swapper alone.
"""
if getattr(modules.globals, "mouth_mask", False):
return True
processors = getattr(modules.globals, "frame_processors", [])
return any(p in processors for p in
("face_enhancer", "face_enhancer_gpen256", "face_enhancer_gpen512"))
def _is_dml() -> bool:
return any("DmlExecutionProvider" in p for p in modules.globals.execution_providers)
def _analyse_faces(frame: Frame) -> list:
"""Run face detection, then recognition (and optionally landmark).
Replaces InsightFace's ``FaceAnalysis.get()`` to skip the
landmark_2d_106 model when only face_swapper is active (saves ~1ms
per face and avoids an unnecessary ONNX session call).
"""
fa = get_face_analyser()
bboxes, kpss = fa.det_model.detect(frame, max_num=0, metric="default")
if bboxes.shape[0] == 0:
return []
need_landmark = _needs_landmark()
rec_model = fa.models.get("recognition")
lmk_model = fa.models.get("landmark_2d_106") if need_landmark else None
from insightface.app.common import Face
faces = []
for i in range(bboxes.shape[0]):
face = Face(bbox=bboxes[i, 0:4],
kps=kpss[i] if kpss is not None else None,
det_score=bboxes[i, 4])
if rec_model is not None:
rec_model.get(frame, face)
if lmk_model is not None:
lmk_model.get(frame, face)
faces.append(face)
return faces
def get_one_face(frame: Frame, faces: Any = None) -> Any:
if faces is None:
if _is_dml():
with modules.globals.dml_lock:
faces = _analyse_faces(frame)
else:
faces = _analyse_faces(frame)
try:
return min(face, key=lambda x: x.bbox[0])
return min(faces, key=lambda x: x.bbox[0])
except ValueError:
return None
def get_many_faces(frame: Frame) -> Any:
try:
return get_face_analyser().get(frame)
if _is_dml():
with modules.globals.dml_lock:
return _analyse_faces(frame)
else:
return _analyse_faces(frame)
except IndexError:
return None
def detect_one_face_fast(frame: Frame) -> Any:
"""Detection-only — skips landmark and recognition models.
Returns a Face with bbox, kps, det_score (enough for face swap).
~10ms vs ~16ms for full get_one_face() at 1080p.
"""
from insightface.app.common import Face
fa = get_face_analyser()
bboxes, kpss = fa.det_model.detect(frame, max_num=0, metric='default')
if bboxes.shape[0] == 0:
return None
idx = int(bboxes[:, 0].argmin())
return Face(bbox=bboxes[idx, :4], kps=kpss[idx], det_score=bboxes[idx, 4])
def detect_many_faces_fast(frame: Frame) -> Any:
"""Detection-only multi-face — skips landmark and recognition."""
from insightface.app.common import Face
fa = get_face_analyser()
bboxes, kpss = fa.det_model.detect(frame, max_num=0, metric='default')
if bboxes.shape[0] == 0:
return None
return [Face(bbox=bboxes[i, :4], kps=kpss[i], det_score=bboxes[i, 4])
for i in range(bboxes.shape[0])]
def ensure_landmarks(frame: Frame, faces: Any) -> None:
"""Run the 2d106 landmark model in-place on faces that lack it.
The fast webcam path (detect_one_face_fast / detect_many_faces_fast)
produces detection-only Face objects with no ``landmark_2d_106``.
Mouth masking needs those landmarks, so add them on demand only when
the feature is active — keeping the fast path fast otherwise.
"""
if faces is None:
return
if not isinstance(faces, (list, tuple)):
faces = [faces]
fa = get_face_analyser()
lmk_model = fa.models.get("landmark_2d_106")
if lmk_model is None:
return
for face in faces:
if face is None:
continue
# insightface Face is a dict; missing keys raise AttributeError,
# so getattr(..., None) is the safe presence check.
if getattr(face, "landmark_2d_106", None) is None:
try:
lmk_model.get(frame, face)
except Exception as e: # pragma: no cover - never break the swap
print(f"Error computing 2d106 landmarks: {e}")
def has_valid_map() -> bool:
for map in modules.globals.source_target_map:
if "source" in map and "target" in map:
@@ -76,8 +257,10 @@ def add_blank_map() -> Any:
def get_unique_faces_from_target_image() -> Any:
try:
modules.globals.source_target_map = []
target_frame = cv2.imread(modules.globals.target_path)
target_frame = imread_unicode(modules.globals.target_path)
many_faces = get_many_faces(target_frame)
if many_faces is None:
return None
i = 0
for face in many_faces:
@@ -110,8 +293,10 @@ def get_unique_faces_from_target_video() -> Any:
i = 0
for temp_frame_path in tqdm(temp_frame_paths, desc="Extracting face embeddings from frames"):
temp_frame = cv2.imread(temp_frame_path)
temp_frame = imread_unicode(temp_frame_path)
many_faces = get_many_faces(temp_frame)
if many_faces is None:
continue
for face in many_faces:
face_embeddings.append(face.normed_embedding)
@@ -153,6 +338,9 @@ def default_target_face():
best_frame = frame
break
if best_face is None:
continue # No faces detected in this cluster — skip
for frame in map['target_faces_in_frame']:
for face in frame['faces']:
if face['det_score'] > best_face['det_score']:
@@ -161,7 +349,7 @@ def default_target_face():
x_min, y_min, x_max, y_max = best_face['bbox']
target_frame = cv2.imread(best_frame['location'])
target_frame = imread_unicode(best_frame['location'])
map['target'] = {
'cv2' : target_frame[int(y_min):int(y_max), int(x_min):int(x_max)],
'face' : best_face
@@ -177,7 +365,7 @@ def dump_faces(centroids: Any, frame_face_embeddings: list):
Path(temp_directory_path + f"/{i}").mkdir(parents=True, exist_ok=True)
for frame in tqdm(frame_face_embeddings, desc=f"Copying faces to temp/./{i}"):
temp_frame = cv2.imread(frame['location'])
temp_frame = imread_unicode(frame['location'])
j = 0
for face in frame['faces']:
@@ -185,5 +373,5 @@ def dump_faces(centroids: Any, frame_face_embeddings: list):
x_min, y_min, x_max, y_max = face['bbox']
if temp_frame[int(y_min):int(y_max), int(x_min):int(x_max)].size > 0:
cv2.imwrite(temp_directory_path + f"/{i}/{frame['frame']}_{j}.png", temp_frame[int(y_min):int(y_max), int(x_min):int(x_max)])
imwrite_unicode(temp_directory_path + f"/{i}/{frame['frame']}_{j}.png", temp_frame[int(y_min):int(y_max), int(x_min):int(x_max)])
j += 1
+12 -5
View File
@@ -6,10 +6,13 @@ from typing import List, Dict, Any
ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
WORKFLOW_DIR = os.path.join(ROOT_DIR, "workflow")
file_types = [
("Image", ("*.png", "*.jpg", "*.jpeg", "*.gif", "*.bmp")),
("Video", ("*.mp4", "*.mkv")),
]
# Canonical media extensions, defined once so the file dialogs and
# has_image_extension never drift. GIF is intentionally excluded: OpenCV's
# cv2.imread/imwrite (the only image I/O this app uses) cannot decode or
# encode GIF on 4.10 or 4.11, so offering it would silently fail. WEBP works
# via the libwebp bundled with opencv-python.
IMAGE_EXTENSIONS = (".png", ".jpg", ".jpeg", ".bmp", ".webp")
VIDEO_EXTENSIONS = (".mp4", ".mkv")
# Face Mapping Data
source_target_map: List[Dict[str, Any]] = [] # Stores detailed map for image/video processing
@@ -50,7 +53,7 @@ headless: bool | None = None # Run without UI?
log_level: str = "error" # Logging level (e.g., 'debug', 'info', 'warning', 'error')
# Face Processor UI Toggles (Example)
fp_ui: Dict[str, bool] = {"face_enhancer": False}
fp_ui: Dict[str, bool] = {"face_enhancer": False, "face_enhancer_gpen256": False, "face_enhancer_gpen512": False}
# Face Swapper Specific Options
face_swapper_enabled: bool = True # General toggle for the swapper processor
@@ -63,6 +66,7 @@ show_mouth_mask_box: bool = False # Visualize the mouth mask area (for debuggin
mask_feather_ratio: int = 12 # Denominator for feathering calculation (higher = smaller feather)
mask_down_size: float = 0.1 # Expansion factor for lower lip mask (relative)
mask_size: float = 1.0 # Expansion factor for upper lip mask (relative)
mouth_mask_size: float = 0.0 # Mouth mask size (0-100; 0=off, 100=mouth to chin)
# --- START: Added for Frame Interpolation ---
enable_interpolation: bool = True # Toggle temporal smoothing
@@ -70,3 +74,6 @@ interpolation_weight: float = 0 # Blend weight for current frame (0.0-1.0). Low
# --- END: Added for Frame Interpolation ---
# --- END OF FILE globals.py ---
import threading
dml_lock = threading.Lock()
+285
View File
@@ -0,0 +1,285 @@
# --- START OF FILE gpu_processing.py ---
"""
GPU-accelerated image processing using OpenCV CUDA (cv2.cuda.GpuMat).
Provides drop-in replacements for common cv2 functions. When OpenCV is built
with CUDA support the functions transparently upload → process → download via
GpuMat; otherwise they fall back to the regular CPU path so the rest of the
codebase never has to care whether CUDA is available.
Usage
-----
from modules.gpu_processing import (
gpu_gaussian_blur, gpu_sharpen, gpu_add_weighted,
gpu_resize, gpu_cvt_color, gpu_flip,
is_gpu_accelerated,
)
"""
from __future__ import annotations
import os
import cv2
import numpy as np
from typing import Tuple
# ---------------------------------------------------------------------------
# CUDA availability detection (evaluated once at import time)
# ---------------------------------------------------------------------------
CUDA_AVAILABLE: bool = False
# OpenCV CUDA per-operation acceleration is DISABLED by default.
# Each gpu_* call uploads to GPU, processes, then downloads back to CPU.
# At webcam resolution (~960x540) this upload/download overhead far exceeds
# the time saved on the actual operation, making it slower than pure CPU.
# The heavy lifting (face detection, swap, enhancement) runs on GPU via
# ONNX Runtime's CUDAExecutionProvider, which is where GPU matters.
#
# To force-enable, set OPENCV_CUDA_PROCESSING=1 in your environment.
if os.environ.get("OPENCV_CUDA_PROCESSING") == "1":
try:
_test_mat = cv2.cuda.GpuMat()
_has_gauss = hasattr(cv2.cuda, "createGaussianFilter")
_has_resize = hasattr(cv2.cuda, "resize")
_has_cvt = hasattr(cv2.cuda, "cvtColor")
if _has_gauss and _has_resize and _has_cvt:
CUDA_AVAILABLE = True
print("[gpu_processing] OpenCV CUDA processing enabled via OPENCV_CUDA_PROCESSING=1.")
except Exception:
pass
# ---------------------------------------------------------------------------
# Internal helpers
# ---------------------------------------------------------------------------
def _ensure_uint8(img: np.ndarray) -> np.ndarray:
"""Clip and convert to uint8 if necessary."""
if img.dtype != np.uint8:
return np.clip(img, 0, 255).astype(np.uint8)
return img
def _ksize_odd(ksize: Tuple[int, int]) -> Tuple[int, int]:
"""Ensure kernel dimensions are positive and odd (required by GaussianBlur)."""
kw = max(1, ksize[0] // 2 * 2 + 1) if ksize[0] > 0 else 0
kh = max(1, ksize[1] // 2 * 2 + 1) if ksize[1] > 0 else 0
return (kw, kh)
def _cv_type_for(img: np.ndarray) -> int:
"""Return the OpenCV type constant matching *img* (uint8 only)."""
channels = 1 if img.ndim == 2 else img.shape[2]
if channels == 1:
return cv2.CV_8UC1
elif channels == 3:
return cv2.CV_8UC3
elif channels == 4:
return cv2.CV_8UC4
return cv2.CV_8UC3 # fallback
# ---------------------------------------------------------------------------
# Public API Gaussian Blur
# ---------------------------------------------------------------------------
def gpu_gaussian_blur(
src: np.ndarray,
ksize: Tuple[int, int],
sigma_x: float,
sigma_y: float = 0,
) -> np.ndarray:
"""Drop-in replacement for ``cv2.GaussianBlur`` with CUDA acceleration.
Parameters match ``cv2.GaussianBlur(src, ksize, sigmaX, sigmaY)``.
When *ksize* is ``(0, 0)`` OpenCV computes the kernel size from *sigma_x*.
"""
if CUDA_AVAILABLE:
try:
src_u8 = _ensure_uint8(src)
cv_type = _cv_type_for(src_u8)
ks = _ksize_odd(ksize) if ksize != (0, 0) else ksize
gauss = cv2.cuda.createGaussianFilter(cv_type, cv_type, ks, sigma_x, sigma_y)
gpu_src = cv2.cuda.GpuMat()
gpu_src.upload(src_u8)
gpu_dst = gauss.apply(gpu_src)
return gpu_dst.download()
except cv2.error:
pass
return cv2.GaussianBlur(src, ksize, sigma_x, sigmaY=sigma_y)
# ---------------------------------------------------------------------------
# Public API addWeighted
# ---------------------------------------------------------------------------
def gpu_add_weighted(
src1: np.ndarray,
alpha: float,
src2: np.ndarray,
beta: float,
gamma: float,
) -> np.ndarray:
"""Drop-in replacement for ``cv2.addWeighted`` with CUDA acceleration."""
if CUDA_AVAILABLE:
try:
s1 = _ensure_uint8(src1)
s2 = _ensure_uint8(src2)
g1 = cv2.cuda.GpuMat()
g2 = cv2.cuda.GpuMat()
g1.upload(s1)
g2.upload(s2)
gpu_dst = cv2.cuda.addWeighted(g1, alpha, g2, beta, gamma)
return gpu_dst.download()
except cv2.error:
pass
return cv2.addWeighted(src1, alpha, src2, beta, gamma)
# ---------------------------------------------------------------------------
# Public API Unsharp-mask sharpening
# ---------------------------------------------------------------------------
def gpu_sharpen(
src: np.ndarray,
strength: float,
sigma: float = 3,
) -> np.ndarray:
"""Unsharp-mask sharpening, optionally GPU-accelerated.
Equivalent to::
blurred = GaussianBlur(src, (0,0), sigma)
result = addWeighted(src, 1+strength, blurred, -strength, 0)
"""
if strength <= 0:
return src
if CUDA_AVAILABLE:
try:
src_u8 = _ensure_uint8(src)
cv_type = _cv_type_for(src_u8)
gauss = cv2.cuda.createGaussianFilter(cv_type, cv_type, (0, 0), sigma)
gpu_src = cv2.cuda.GpuMat()
gpu_src.upload(src_u8)
gpu_blurred = gauss.apply(gpu_src)
gpu_sharp = cv2.cuda.addWeighted(gpu_src, 1.0 + strength, gpu_blurred, -strength, 0)
result = gpu_sharp.download()
return np.clip(result, 0, 255).astype(np.uint8)
except cv2.error:
pass
blurred = cv2.GaussianBlur(src, (0, 0), sigma)
sharpened = cv2.addWeighted(src, 1.0 + strength, blurred, -strength, 0)
return np.clip(sharpened, 0, 255).astype(np.uint8)
# ---------------------------------------------------------------------------
# Public API Resize
# ---------------------------------------------------------------------------
# Map common cv2 interpolation flags to their CUDA equivalents
_INTERP_MAP = {
cv2.INTER_NEAREST: cv2.INTER_NEAREST,
cv2.INTER_LINEAR: cv2.INTER_LINEAR,
cv2.INTER_CUBIC: cv2.INTER_CUBIC,
cv2.INTER_AREA: cv2.INTER_AREA,
cv2.INTER_LANCZOS4: cv2.INTER_LANCZOS4,
}
def gpu_resize(
src: np.ndarray,
dsize: Tuple[int, int],
fx: float = 0,
fy: float = 0,
interpolation: int = cv2.INTER_LINEAR,
) -> np.ndarray:
"""Drop-in replacement for ``cv2.resize`` with CUDA acceleration.
Parameters match ``cv2.resize(src, dsize, fx=fx, fy=fy, interpolation=...)``.
"""
if CUDA_AVAILABLE:
try:
src_u8 = _ensure_uint8(src)
gpu_src = cv2.cuda.GpuMat()
gpu_src.upload(src_u8)
interp = _INTERP_MAP.get(interpolation, cv2.INTER_LINEAR)
if dsize and dsize[0] > 0 and dsize[1] > 0:
gpu_dst = cv2.cuda.resize(gpu_src, dsize, interpolation=interp)
else:
gpu_dst = cv2.cuda.resize(gpu_src, (0, 0), fx=fx, fy=fy, interpolation=interp)
return gpu_dst.download()
except cv2.error:
pass
return cv2.resize(src, dsize, fx=fx, fy=fy, interpolation=interpolation)
# ---------------------------------------------------------------------------
# Public API Color conversion
# ---------------------------------------------------------------------------
def gpu_cvt_color(
src: np.ndarray,
code: int,
) -> np.ndarray:
"""Drop-in replacement for ``cv2.cvtColor`` with CUDA acceleration.
Parameters match ``cv2.cvtColor(src, code)``.
"""
if CUDA_AVAILABLE:
try:
src_u8 = _ensure_uint8(src)
gpu_src = cv2.cuda.GpuMat()
gpu_src.upload(src_u8)
gpu_dst = cv2.cuda.cvtColor(gpu_src, code)
return gpu_dst.download()
except cv2.error:
pass
return cv2.cvtColor(src, code)
# ---------------------------------------------------------------------------
# Public API Flip
# ---------------------------------------------------------------------------
def gpu_flip(
src: np.ndarray,
flip_code: int,
) -> np.ndarray:
"""Drop-in replacement for ``cv2.flip`` with CUDA acceleration.
Parameters match ``cv2.flip(src, flipCode)``.
*flip_code*: 0 = vertical, 1 = horizontal, -1 = both.
"""
if CUDA_AVAILABLE:
try:
src_u8 = _ensure_uint8(src)
gpu_src = cv2.cuda.GpuMat()
gpu_src.upload(src_u8)
gpu_dst = cv2.cuda.flip(gpu_src, flip_code)
return gpu_dst.download()
except cv2.error:
pass
return cv2.flip(src, flip_code)
# ---------------------------------------------------------------------------
# Convenience: check at runtime whether GPU path is active
# ---------------------------------------------------------------------------
def is_gpu_accelerated() -> bool:
"""Return ``True`` when the CUDA path will be used."""
return CUDA_AVAILABLE
# --- END OF FILE gpu_processing.py ---
+1 -1
View File
@@ -1,3 +1,3 @@
name = 'Deep-Live-Cam'
version = '2.0.1c'
version = '2.1.5'
edition = 'GitHub Edition'
+178
View File
@@ -0,0 +1,178 @@
import os
import platform
import ssl
import threading
import urllib.error
import urllib.request
from typing import Dict, List, Optional
from tqdm import tqdm
from modules.paths import MODELS_DIR
HF_REPO_ID = "hacksider/deep-live-cam"
HF_RESOLVE_BASE = f"https://huggingface.co/{HF_REPO_ID}/resolve/main/"
MODEL_SIZES: Dict[str, int] = {
"inswapper_128.onnx": 554253681,
"inswapper_128_fp16.onnx": 277680638,
"gfpgan-1024.onnx": 365875079,
"GPEN-BFR-256.onnx": 75715262,
"GPEN-BFR-512.onnx": 284244491,
"buffalo_l/buffalo_l/1k3d68.onnx": 143607619,
"buffalo_l/buffalo_l/2d106det.onnx": 5030888,
"buffalo_l/buffalo_l/det_10g.onnx": 16923827,
"buffalo_l/buffalo_l/genderage.onnx": 1322532,
"buffalo_l/buffalo_l/w600k_r50.onnx": 174383860,
}
_LOCKS: Dict[str, threading.Lock] = {}
_LOCKS_GUARD = threading.Lock()
CHUNK_SIZE = 1024 * 256
def _ssl_context():
if platform.system().lower() == "darwin":
return ssl._create_unverified_context()
return None
def _lock_for(key: str) -> threading.Lock:
with _LOCKS_GUARD:
if key not in _LOCKS:
_LOCKS[key] = threading.Lock()
return _LOCKS[key]
def resolve_url(name: str) -> str:
return HF_RESOLVE_BASE + name.replace(os.sep, "/")
def local_path(name: str, dest_dir: Optional[str] = None) -> str:
if dest_dir is not None:
return os.path.join(dest_dir, os.path.basename(name))
return os.path.join(MODELS_DIR, *name.replace("/", os.sep).split(os.sep))
def expected_size(name: str) -> Optional[int]:
return MODEL_SIZES.get(name.replace(os.sep, "/"))
def is_present(name: str, dest_dir: Optional[str] = None) -> bool:
path = local_path(name, dest_dir)
return os.path.isfile(path) and os.path.getsize(path) > 0
def _download(name: str, url: str, target: str, size: Optional[int]) -> bool:
os.makedirs(os.path.dirname(target) or MODELS_DIR, exist_ok=True)
partial = target + ".part"
resume_from = os.path.getsize(partial) if os.path.isfile(partial) else 0
headers = {"User-Agent": "Deep-Live-Cam"}
if resume_from:
headers["Range"] = f"bytes={resume_from}-"
try:
request = urllib.request.Request(url, headers=headers)
response = urllib.request.urlopen(request, context=_ssl_context(), timeout=60)
except urllib.error.HTTPError as error:
if resume_from and error.code in (416, 501):
try:
os.remove(partial)
except OSError:
pass
return _download(name, url, target, size)
print(f"[DLC.MODELS] Failed to download {name}: HTTP {error.code}")
return False
except (urllib.error.URLError, OSError) as error:
print(f"[DLC.MODELS] Failed to download {name}: {error}")
return False
with response:
if resume_from and getattr(response, "status", 200) != 206:
resume_from = 0
remaining = int(response.headers.get("Content-Length", 0) or 0)
total = size or (resume_from + remaining) or None
mode = "ab" if resume_from else "wb"
try:
with open(partial, mode) as handle:
with tqdm(
total=total,
initial=resume_from,
desc=f"Downloading {os.path.basename(name)}",
unit="B",
unit_scale=True,
unit_divisor=1024,
) as progress:
while True:
buffer = response.read(CHUNK_SIZE)
if not buffer:
break
handle.write(buffer)
progress.update(len(buffer))
except (urllib.error.URLError, OSError) as error:
print(f"[DLC.MODELS] Download of {name} interrupted: {error}")
return False
downloaded = os.path.getsize(partial)
if size is not None and downloaded != size:
print(f"[DLC.MODELS] {name} is {downloaded} bytes, expected {size}. Discarding.")
try:
os.remove(partial)
except OSError:
pass
return False
try:
os.replace(partial, target)
except OSError as error:
print(f"[DLC.MODELS] Could not finalise {name}: {error}")
return False
return True
def ensure_model(
name: str, quiet: bool = False, dest_dir: Optional[str] = None
) -> Optional[str]:
name = name.replace(os.sep, "/")
target = local_path(name, dest_dir)
with _lock_for(target):
if is_present(name, dest_dir):
return target
if not quiet:
print(f"[DLC.MODELS] {name} not found in models folder, downloading...")
if _download(name, resolve_url(name), target, expected_size(name)):
return target
return None
def ensure_any(names: List[str]) -> Optional[str]:
for name in names:
if is_present(name):
return local_path(name)
for name in names:
path = ensure_model(name)
if path is not None:
return path
return None
def ensure_insightface_pack(name: str = "buffalo_l") -> bool:
members = [n for n in MODEL_SIZES if n.startswith(f"{name}/")]
if not members:
return False
dest_dir = os.path.join(os.path.expanduser("~"), ".insightface", "models", name)
if all(is_present(member, dest_dir) for member in members):
return True
print(f"[DLC.MODELS] insightface pack '{name}' is missing, downloading...")
ok = True
for member in members:
if ensure_model(member, quiet=True, dest_dir=dest_dir) is None:
ok = False
if not ok:
print(f"[DLC.MODELS] Could not pre-fill '{name}'; insightface will retry.")
return ok
+550
View File
@@ -0,0 +1,550 @@
"""ONNX model optimizations for CoreML execution on Apple Silicon.
Each pass eliminates a different CPU↔ANE round-trip that ORT's CoreML EP
would otherwise introduce:
1. **Shape/Gather constant folding** — Dynamic ``Shape`` → ``Gather`` chains
(e.g. for FPN upsample target sizes in RetinaFace) force ops onto CPU even
when the input dimensions are known at load time. We run ONNX shape
inference with the known input size and replace these chains with constants.
Float32-noise-level differences only (max ~6e-6).
2. **Pad(reflect) decomposition** — CoreML doesn't support ``Pad(mode=reflect)``.
Models using reflect padding (e.g. inswapper_128) get split into many CoreML
subgraphs with CPU fallbacks between each. We rewrite each ``Pad(reflect)``
as equivalent ``Slice`` + ``Concat`` ops that CoreML handles natively.
Bit-for-bit identical output. (Fixed upstream in microsoft/onnxruntime#28073.)
3. **Split → Slice decomposition** — CoreML's EP doesn't support the ONNX
``Split`` op, causing partition boundaries in models with channel-wise
splits (e.g. GFPGAN's SFT modulation). Each 2-way Split becomes two Slices.
4. **Scalar Gather widening** — ORT's CoreML EP rejects ``Gather`` nodes with
rank-0 (scalar) indices. StyleGAN-derived models (GFPGAN) slice per-layer
style codes using exactly this pattern. We widen each scalar index to
``[1]`` and squeeze the added axis on the Gather output.
(Filed upstream as microsoft/onnxruntime#28180.)
All passes are cached on disk with a ``_coreml`` suffix so the rewrite cost
is paid only once per model.
"""
import os
import platform
import numpy as np
IS_APPLE_SILICON = platform.system() == "Darwin" and platform.machine() == "arm64"
def optimize_for_coreml(model_path: str, input_shape: tuple = None) -> str:
"""Return path to a CoreML-optimized ONNX model.
Applies all applicable optimizations and caches the result next to
the original model (with ``_coreml`` suffix).
Args:
model_path: Path to the original ONNX model.
input_shape: Optional fixed input shape (e.g. ``(1, 3, 640, 640)``).
When provided, enables Shape/Gather constant folding.
Returns the optimized path, or the original path if no optimizations
apply or we're not on Apple Silicon.
"""
if not IS_APPLE_SILICON:
return model_path
base, ext = os.path.splitext(model_path)
optimized_path = f"{base}_coreml{ext}"
if os.path.exists(optimized_path):
if os.path.getmtime(optimized_path) >= os.path.getmtime(model_path):
return optimized_path
import onnx
from onnx import numpy_helper
model = onnx.load(model_path)
changed = False
if _fold_shape_gather(model, input_shape):
changed = True
# TODO(ort>=1.26): drop this pass. Fixed upstream by microsoft/onnxruntime#28073.
if _decompose_reflect_pad(model):
changed = True
if _decompose_split(model):
changed = True
# TODO: drop this pass once microsoft/onnxruntime#28180 ships. The CoreML
# Gather op builder rejects rank-0 (scalar) indices; we widen them to [1]
# + Squeeze so StyleGAN-family models (GFPGAN) stay on ANE.
if _rewrite_scalar_gather(model):
changed = True
if not changed:
return model_path
# Preserve insightface's emap convention: the INSwapper class reads
# graph.initializer[-1] as the embedding map. If the original model
# had a (512, 512) matrix as its last initializer, keep it last.
_preserve_emap_position(model, numpy_helper)
onnx.save(model, optimized_path)
return optimized_path
# ---------------------------------------------------------------------------
# Pass 1: Fold Shape → Gather chains into constants
# ---------------------------------------------------------------------------
def _fold_shape_gather(model, input_shape) -> bool:
"""Replace dynamic Shape→Gather chains with constants when input size is known.
Only removes a Shape node when ALL of its consumers are Gather nodes
that are also being folded. This prevents breaking graphs where
a Shape output feeds into other ops as well.
"""
if input_shape is None:
return False
from onnx import numpy_helper, shape_inference
graph = model.graph
# Set fixed input dimensions for shape inference
inp = graph.input[0]
dims = inp.type.tensor_type.shape.dim
for i, size in enumerate(input_shape):
if i < len(dims):
dims[i].dim_value = size
try:
model_inferred = shape_inference.infer_shapes(model)
except Exception:
return False
# Extract inferred shapes
value_shapes = {}
for vi in list(model_inferred.graph.value_info) + list(graph.input) + list(graph.output):
shape_dims = vi.type.tensor_type.shape.dim
shape = []
for d in shape_dims:
if d.dim_value > 0:
shape.append(d.dim_value)
else:
shape.append(None)
value_shapes[vi.name] = shape
inits = {init.name: numpy_helper.to_array(init) for init in graph.initializer}
# Build consumer map: output_name → list of consuming nodes
consumers = {}
for node in graph.node:
for i in node.input:
consumers.setdefault(i, []).append(node)
# Also check graph outputs — an output name consumed by the graph
# output list must not be removed
graph_output_names = {o.name for o in graph.output}
# Find Shape nodes with fully-known output
shape_constants = {}
for node in graph.node:
if node.op_type == "Shape":
inp_shape = value_shapes.get(node.input[0])
if inp_shape and all(isinstance(d, int) for d in inp_shape):
shape_constants[node.output[0]] = np.array(inp_shape, dtype=np.int64)
if not shape_constants:
return False
# Find Gather nodes consuming Shape constants
gather_constants = {}
for node in graph.node:
if node.op_type == "Gather" and node.input[0] in shape_constants:
idx_name = node.input[1]
if idx_name in inits:
idx = int(inits[idx_name])
val = int(shape_constants[node.input[0]][idx])
gather_constants[node.output[0]] = np.array(val, dtype=np.int64)
if not gather_constants:
return False
# Determine which Gather nodes to fold (always safe — we replace
# the output with a constant initializer)
gather_remove_ids = set()
for node in graph.node:
if node.op_type == "Gather" and node.output[0] in gather_constants:
gather_remove_ids.add(id(node))
# Determine which Shape nodes are safe to remove: only if ALL
# consumers of the Shape output are Gather nodes being folded,
# and the output isn't a graph output.
shape_remove_ids = set()
for node in graph.node:
if node.op_type == "Shape" and node.output[0] in shape_constants:
out_name = node.output[0]
if out_name in graph_output_names:
continue
node_consumers = consumers.get(out_name, [])
if all(id(c) in gather_remove_ids for c in node_consumers):
shape_remove_ids.add(id(node))
remove_ids = gather_remove_ids | shape_remove_ids
# Add Gather output constants as initializers
existing = {i.name for i in graph.initializer}
for name, val in gather_constants.items():
if name not in existing:
graph.initializer.append(numpy_helper.from_array(val, name=name))
new_nodes = [n for n in graph.node if id(n) not in remove_ids]
del graph.node[:]
graph.node.extend(new_nodes)
return True
# ---------------------------------------------------------------------------
# Pass 2: Decompose Pad(reflect) → Slice + Concat
#
# TEMPORARY: fixed upstream in microsoft/onnxruntime#28073 (merged 2026-04-20).
# Once the ORT floor is >= 1.26.0, MLProgram handles Pad(mode=reflect) natively
# via MIL tensor_operation.pad and this entire pass can be deleted.
# ---------------------------------------------------------------------------
def _decompose_reflect_pad(model) -> bool:
"""Rewrite Pad(reflect) as Slice+Concat sequences CoreML can handle."""
from onnx import numpy_helper, helper
graph = model.graph
inits = {init.name: numpy_helper.to_array(init) for init in graph.initializer}
reflect_pads = []
for node in graph.node:
if node.op_type == "Pad":
mode = "constant"
for attr in node.attribute:
if attr.name == "mode":
mode = attr.s.decode()
if mode == "reflect" and len(node.input) > 1 and node.input[1] in inits:
reflect_pads.append(node)
if not reflect_pads:
return False
existing_names = {i.name for i in graph.initializer}
def ensure_const(name, value):
if name not in existing_names:
graph.initializer.append(
numpy_helper.from_array(np.array(value, dtype=np.int64), name=name)
)
existing_names.add(name)
ensure_const("_rp_ax2", [2])
ensure_const("_rp_ax3", [3])
max_pad = 0
for node in reflect_pads:
pads = inits[node.input[1]].tolist()
max_pad = max(max_pad, int(pads[2]), int(pads[3]))
for v in range(1, max_pad + 2):
ensure_const(f"_rp_p{v}", [v])
ensure_const(f"_rp_n{v}", [-v])
_counter = [0]
def uid():
_counter[0] += 1
return _counter[0]
pad_ids = {id(n) for n in reflect_pads}
pad_init_names = set()
new_nodes = []
for node in graph.node:
if id(node) not in pad_ids:
new_nodes.append(node)
continue
pads = inits[node.input[1]].tolist()
h_pad, w_pad = int(pads[2]), int(pads[3])
for inp in node.input[1:]:
if inp in inits:
pad_init_names.add(inp)
current = node.input[0]
if h_pad > 0:
top = []
for i in range(h_pad, 0, -1):
name = f"_rp_t{uid()}"
new_nodes.append(helper.make_node(
"Slice",
inputs=[current, f"_rp_p{i}", f"_rp_p{i+1}", "_rp_ax2"],
outputs=[name],
))
top.append(name)
bot = []
for i in range(1, h_pad + 1):
name = f"_rp_b{uid()}"
new_nodes.append(helper.make_node(
"Slice",
inputs=[current, f"_rp_n{i+1}", f"_rp_n{i}", "_rp_ax2"],
outputs=[name],
))
bot.append(name)
h_out = f"_rp_h{uid()}"
new_nodes.append(helper.make_node(
"Concat", inputs=top + [current] + bot, outputs=[h_out], axis=2
))
current = h_out
if w_pad > 0:
left = []
for i in range(w_pad, 0, -1):
name = f"_rp_l{uid()}"
new_nodes.append(helper.make_node(
"Slice",
inputs=[current, f"_rp_p{i}", f"_rp_p{i+1}", "_rp_ax3"],
outputs=[name],
))
left.append(name)
right = []
for i in range(1, w_pad + 1):
name = f"_rp_r{uid()}"
new_nodes.append(helper.make_node(
"Slice",
inputs=[current, f"_rp_n{i+1}", f"_rp_n{i}", "_rp_ax3"],
outputs=[name],
))
right.append(name)
new_nodes.append(helper.make_node(
"Concat",
inputs=left + [current] + right,
outputs=[node.output[0]],
axis=3,
))
elif h_pad > 0:
new_nodes.append(helper.make_node(
"Identity", inputs=[current], outputs=[node.output[0]]
))
# Remove old Pad initializers
clean_inits = [i for i in graph.initializer if i.name not in pad_init_names]
del graph.initializer[:]
graph.initializer.extend(clean_inits)
del graph.node[:]
graph.node.extend(new_nodes)
return True
# ---------------------------------------------------------------------------
# Pass 3: Decompose Split → Slice pairs
# ---------------------------------------------------------------------------
def _decompose_split(model) -> bool:
"""Rewrite Split(axis=1) as Slice pairs that CoreML can handle.
CoreML's EP doesn't support the ONNX ``Split`` op, causing partition
boundaries in models that use channel-wise splits (e.g. GFPGAN's SFT
modulation layers). Each Split with two outputs becomes two Slice ops.
"""
from onnx import numpy_helper, helper
graph = model.graph
splits = []
for node in graph.node:
if node.op_type == "Split":
axis = 0
split_sizes = []
for attr in node.attribute:
if attr.name == "axis":
axis = attr.i
if attr.name == "split":
split_sizes = list(attr.ints)
if axis == 1 and len(split_sizes) == 2 and len(node.output) == 2:
splits.append((node, split_sizes))
if not splits:
return False
existing = {i.name for i in graph.initializer}
def ensure_const(name, value):
if name not in existing:
graph.initializer.append(
numpy_helper.from_array(np.array(value, dtype=np.int64), name=name)
)
existing.add(name)
ensure_const("_sp_ax1", [1])
# Collect all needed boundary constants
for _, (a, b) in splits:
ensure_const("_sp_s0", [0])
ensure_const(f"_sp_s{a}", [a])
ensure_const(f"_sp_s{a + b}", [a + b])
split_ids = {id(node) for node, _ in splits}
replacements = {}
for node, (a, b) in splits:
slice0 = helper.make_node(
"Slice",
inputs=[node.input[0], "_sp_s0", f"_sp_s{a}", "_sp_ax1"],
outputs=[node.output[0]],
)
slice1 = helper.make_node(
"Slice",
inputs=[node.input[0], f"_sp_s{a}", f"_sp_s{a + b}", "_sp_ax1"],
outputs=[node.output[1]],
)
replacements[id(node)] = [slice0, slice1]
new_nodes = []
for node in graph.node:
if id(node) in split_ids:
new_nodes.extend(replacements[id(node)])
else:
new_nodes.append(node)
del graph.node[:]
graph.node.extend(new_nodes)
return True
# ---------------------------------------------------------------------------
# Pass 4: Widen scalar Gather indices to [1] + Squeeze
#
# TEMPORARY: filed upstream as microsoft/onnxruntime#28180. ORT's CoreML EP
# GatherOpBuilder::IsOpSupportedImpl rejects rank-0 (scalar) indices with
# `Gather does not support scalar 'indices'`. The builder's own comment
# describes the workaround (promote to [1], squeeze the added axis) but
# doesn't apply it. We do the same thing at the ONNX level so StyleGAN-
# family models (GFPGAN is the hot example — 16 per-layer style-code
# slices) don't split the CoreML subgraph. Once the upstream fix ships
# and the ORT floor is raised, delete this pass.
# ---------------------------------------------------------------------------
def _rewrite_scalar_gather(model) -> bool:
"""Rewrite Gather(data, scalar_idx) as Gather(data, [scalar_idx]) + Squeeze.
Only touches Gather nodes whose index is a rank-0 int64 constant or
initializer; everything else passes through unchanged. The rewrite
is semantically identical — indices get an added leading axis, the
Squeeze removes it after the gather.
"""
from onnx import numpy_helper, helper, TensorProto
graph = model.graph
# Opset 13 moved Squeeze's axes from attribute to input.
opset = next(
(o.version for o in model.opset_import if o.domain in ("", "ai.onnx")),
11,
)
const_values = {}
for n in graph.node:
if n.op_type == "Constant":
for a in n.attribute:
if a.name == "value":
const_values[n.output[0]] = a.t
init_values = {i.name: i for i in graph.initializer}
def scalar_int64(name):
"""Return int value if `name` resolves to a rank-0 int64 constant, else None."""
tensor = const_values.get(name) or init_values.get(name)
if tensor is None or tensor.data_type != TensorProto.INT64:
return None
arr = numpy_helper.to_array(tensor)
return int(arr) if arr.ndim == 0 else None
rewrote = 0
new_nodes = []
for n in graph.node:
if n.op_type == "Gather":
val = scalar_int64(n.input[1])
if val is not None:
axis = next((a.i for a in n.attribute if a.name == "axis"), 0)
idx_1d_name = f"{n.input[1]}_1d_{rewrote}"
idx_const = helper.make_node(
"Constant",
inputs=[],
outputs=[idx_1d_name],
value=helper.make_tensor(idx_1d_name, TensorProto.INT64, [1], [val]),
)
gather_out = f"{n.output[0]}_pre_squeeze_{rewrote}"
new_gather = helper.make_node(
"Gather",
inputs=[n.input[0], idx_1d_name],
outputs=[gather_out],
name=n.name,
axis=axis,
)
if opset < 13:
squeeze = helper.make_node(
"Squeeze",
inputs=[gather_out],
outputs=[n.output[0]],
name=(n.name or "gather") + "_squeeze",
axes=[axis],
)
new_nodes.extend([idx_const, new_gather, squeeze])
else:
axes_name = f"{idx_1d_name}_sq_axes"
axes_const = helper.make_node(
"Constant",
inputs=[],
outputs=[axes_name],
value=helper.make_tensor(axes_name, TensorProto.INT64, [1], [axis]),
)
squeeze = helper.make_node(
"Squeeze",
inputs=[gather_out, axes_name],
outputs=[n.output[0]],
name=(n.name or "gather") + "_squeeze",
)
new_nodes.extend([idx_const, axes_const, new_gather, squeeze])
rewrote += 1
continue
new_nodes.append(n)
if rewrote == 0:
return False
del graph.node[:]
graph.node.extend(new_nodes)
return True
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _preserve_emap_position(model, numpy_helper):
"""Keep the insightface emap (512×512 matrix) as the last initializer."""
graph = model.graph
emap_init = None
for init in graph.initializer:
if not init.name.startswith("_rp_"):
arr = numpy_helper.to_array(init)
if len(arr.shape) == 2 and arr.shape[0] == 512 and arr.shape[1] == 512:
emap_init = init
break
if emap_init is not None:
inits = [i for i in graph.initializer if i.name != emap_init.name]
del graph.initializer[:]
graph.initializer.extend(inits)
graph.initializer.append(emap_init)
+6
View File
@@ -0,0 +1,6 @@
"""Shared path constants for the Deep-Live-Cam project."""
import os
ROOT_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
MODELS_DIR = os.path.join(ROOT_DIR, "models")
+91
View File
@@ -0,0 +1,91 @@
"""Centralized platform + accelerator detection.
Imported once at startup to expose typed flags the rest of the codebase
can branch on without re-querying `platform`, `torch.cuda`, or
`onnxruntime.get_available_providers()` repeatedly.
The banner printed by :func:`print_banner` is the single user-facing
report of which code path the app will take.
"""
from __future__ import annotations
import platform as _platform
import sys
from typing import List, Tuple
IS_WINDOWS: bool = _platform.system() == "Windows"
IS_MACOS: bool = _platform.system() == "Darwin"
IS_LINUX: bool = _platform.system() == "Linux"
IS_APPLE_SILICON: bool = IS_MACOS and _platform.machine() == "arm64"
def _detect_torch_cuda() -> bool:
try:
import torch # noqa: WPS433 — local import, avoid hard dep at module load
return bool(torch.cuda.is_available())
except Exception:
return False
def _detect_onnx_providers() -> List[str]:
try:
import onnxruntime
return list(onnxruntime.get_available_providers())
except Exception:
return []
HAS_TORCH_CUDA: bool = _detect_torch_cuda()
ONNX_PROVIDERS: List[str] = _detect_onnx_providers()
HAS_CUDA_PROVIDER: bool = "CUDAExecutionProvider" in ONNX_PROVIDERS
HAS_COREML_PROVIDER: bool = "CoreMLExecutionProvider" in ONNX_PROVIDERS
HAS_DML_PROVIDER: bool = "DmlExecutionProvider" in ONNX_PROVIDERS
HAS_OPENVINO_PROVIDER: bool = "OpenVINOExecutionProvider" in ONNX_PROVIDERS
# OpenVINO execution-provider config shared by every ONNX session builder.
# AUTO:GPU,NPU,CPU lets OpenVINO pick the best available device in priority
# order (Intel GPU → NPU → CPU).
OPENVINO_PROVIDER_CONFIG = (
"OpenVINOExecutionProvider",
{"device_type": "AUTO:GPU,NPU,CPU"},
)
def camera_backends() -> List[Tuple[int, int]]:
"""Return an ordered list of ``(device_index, cv2_backend)`` attempts.
Windows prefers MSMF (60fps capable) with DirectShow as fallback.
macOS/Linux use the default backend (AVFoundation / V4L2).
"""
import cv2
if IS_WINDOWS:
return [
(0, cv2.CAP_MSMF),
(0, cv2.CAP_DSHOW),
(0, cv2.CAP_ANY),
]
return [(0, cv2.CAP_ANY)]
def accelerator_label() -> str:
if HAS_CUDA_PROVIDER:
return "CUDA (NVIDIA)"
if IS_APPLE_SILICON and HAS_COREML_PROVIDER:
return "CoreML (Apple Neural Engine)"
if HAS_COREML_PROVIDER:
return "CoreML"
if HAS_OPENVINO_PROVIDER:
return "OpenVINO (Intel)"
if HAS_DML_PROVIDER:
return "DirectML"
return "CPU"
def print_banner() -> None:
"""Print a one-line summary of the platform + accelerator selection."""
os_label = f"{_platform.system()} {_platform.machine()}"
print(
f"[platform] {os_label} | python {sys.version.split()[0]} | "
f"accelerator: {accelerator_label()} | providers: {ONNX_PROVIDERS}",
flush=True,
)
+15 -1
View File
@@ -1,8 +1,22 @@
import importlib.util
import os
import numpy
# Keras 3 defaults to the TensorFlow backend, which has no Python 3.14 wheels.
# opennsfw2 only runs inference, so any installed backend works; pick one that
# is actually present before opennsfw2 imports keras.
if "KERAS_BACKEND" not in os.environ:
for _backend in ("torch", "tensorflow", "jax"):
if importlib.util.find_spec(_backend) is not None:
os.environ["KERAS_BACKEND"] = _backend
break
import opennsfw2
from PIL import Image
import cv2 # Add OpenCV import
import modules.globals # Import globals to access the color correction toggle
from modules.gpu_processing import gpu_cvt_color
from modules.typing import Frame
@@ -14,7 +28,7 @@ model = None
def predict_frame(target_frame: Frame) -> bool:
# Convert the frame to RGB before processing if color correction is enabled
if modules.globals.color_correction:
target_frame = cv2.cvtColor(target_frame, cv2.COLOR_BGR2RGB)
target_frame = gpu_cvt_color(target_frame, cv2.COLOR_BGR2RGB)
image = Image.fromarray(target_frame)
image = opennsfw2.preprocess_image(image, opennsfw2.Preprocessing.YAHOO)
+244
View File
@@ -0,0 +1,244 @@
"""Shared ONNX-based face enhancement utilities for GPEN-BFR models.
Provides session creation, pre/post processing, and the core
enhance-face-via-ONNX pipeline.
"""
import os
import platform
import threading
from typing import Any
import cv2
import numpy as np
import onnxruntime
import modules.globals
from modules.platform_info import OPENVINO_PROVIDER_CONFIG
IS_APPLE_SILICON = platform.system() == "Darwin" and platform.machine() == "arm64"
# Limit concurrent ONNX calls to avoid VRAM exhaustion on multi-face frames
THREAD_SEMAPHORE = threading.Semaphore(min(max(1, (os.cpu_count() or 1)), 8))
def build_provider_config(providers=None):
"""Wrap raw provider name strings with optimised CUDA / CoreML options.
Providers that are already ``(name, options_dict)`` tuples are passed
through unchanged. Non-CUDA providers are left as bare strings.
"""
if providers is None:
providers = modules.globals.execution_providers
config = []
for p in providers:
if isinstance(p, tuple):
# Already configured pass through
config.append(p)
elif p == "CUDAExecutionProvider":
# Use bare provider — ONNX Runtime's defaults are fastest on
# modern GPUs (Blackwell/sm_120). Custom options like
# EXHAUSTIVE cudnn_conv_algo_search hurt performance on these
# architectures.
config.append(p)
elif p == "CoreMLExecutionProvider" and IS_APPLE_SILICON:
config.append((
"CoreMLExecutionProvider",
{
"ModelFormat": "MLProgram",
"MLComputeUnits": "ALL",
"AllowLowPrecisionAccumulationOnGPU": 1,
},
))
elif p == "OpenVINOExecutionProvider":
# AUTO lets OpenVINO select the best device
config.append(OPENVINO_PROVIDER_CONFIG)
else:
config.append(p)
return config
def run_inference(session: onnxruntime.InferenceSession,
input_name: str,
input_tensor: "np.ndarray") -> "np.ndarray":
"""Run ONNX inference, using IO binding when a CUDA session is active.
IO binding avoids redundant host↔device copies by transferring the
input tensor directly to GPU memory and letting ONNX Runtime allocate
the output on the device. Falls back to the standard ``session.run``
path for non-CUDA providers or if binding fails.
"""
if "CUDAExecutionProvider" in session.get_providers():
try:
io_binding = session.io_binding()
# Input: numpy → GPU
ort_input = onnxruntime.OrtValue.ortvalue_from_numpy(
input_tensor, "cuda", 0,
)
io_binding.bind_ortvalue_input(input_name, ort_input)
# Output: allocate on GPU (avoids a CPU-side allocation)
output_name = session.get_outputs()[0].name
io_binding.bind_output(output_name, "cuda", 0)
session.run_with_iobinding(io_binding)
return io_binding.get_outputs()[0].numpy()
except Exception:
# Fall back to standard path (e.g. ORT version mismatch,
# unsupported op, or VRAM pressure)
pass
return session.run(None, {input_name: input_tensor})[0]
def create_onnx_session(model_path: str) -> onnxruntime.InferenceSession:
"""Create an ONNX Runtime session with optimised provider config.
On Apple Silicon, applies CoreML graph optimizations (Pad decomposition,
Shape/Gather folding, Split decomposition) to reduce CPU↔ANE partition
boundaries.
"""
if IS_APPLE_SILICON:
from modules.onnx_optimize import optimize_for_coreml
# Infer input shape from the model for Shape/Gather folding
try:
import onnx
m = onnx.load(model_path)
inp = m.graph.input[0]
dims = inp.type.tensor_type.shape.dim
shape = tuple(d.dim_value for d in dims if d.dim_value > 0)
input_shape = shape if len(shape) == 4 else None
except Exception:
input_shape = None
model_path = optimize_for_coreml(model_path, input_shape=input_shape)
providers = build_provider_config()
session_options = onnxruntime.SessionOptions()
session_options.graph_optimization_level = (
onnxruntime.GraphOptimizationLevel.ORT_ENABLE_ALL
)
session = onnxruntime.InferenceSession(
model_path, sess_options=session_options, providers=providers,
)
return session
def warmup_session(session: onnxruntime.InferenceSession) -> None:
"""Run a dummy inference pass to trigger JIT / compile caching."""
try:
input_feed = {
inp.name: np.zeros(
[d if isinstance(d, int) and d > 0 else 1 for d in inp.shape],
dtype=np.float32,
)
for inp in session.get_inputs()
}
session.run(None, input_feed)
except Exception as e:
print(f"ONNX enhancer warmup skipped (non-fatal): {e}")
def preprocess_face(face_img: np.ndarray, input_size: int) -> np.ndarray:
"""Resize, normalize, and convert a BGR face crop to ONNX input blob.
GPEN-BFR expects [1, 3, H, W] float32 in RGB, normalized to [-1, 1].
"""
resized = cv2.resize(face_img, (input_size, input_size), interpolation=cv2.INTER_LINEAR)
rgb = cv2.cvtColor(resized, cv2.COLOR_BGR2RGB)
blob = rgb.astype(np.float32) / 255.0 * 2.0 - 1.0
blob = np.transpose(blob, (2, 0, 1))[np.newaxis, ...]
return blob
def postprocess_face(output: np.ndarray) -> np.ndarray:
"""Convert ONNX output [1, 3, H, W] float32 back to BGR uint8 image."""
img = output[0].transpose(1, 2, 0)
img = ((img + 1.0) / 2.0 * 255.0)
img = np.clip(img, 0, 255).astype(np.uint8)
img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
return img
def _get_face_affine(face: Any, input_size: int):
"""Compute affine transform to align a face to GPEN input space.
Returns (M, inv_M) — forward and inverse affine matrices.
"""
template = np.array([
[0.31556875, 0.4615741],
[0.68262291, 0.4615741],
[0.50009375, 0.6405054],
[0.34947187, 0.8246919],
[0.65343645, 0.8246919],
], dtype=np.float32) * input_size
landmarks = None
if hasattr(face, "kps") and face.kps is not None:
landmarks = face.kps.astype(np.float32)
elif hasattr(face, "landmark_2d_106") and face.landmark_2d_106 is not None:
lm106 = face.landmark_2d_106
landmarks = np.array([
lm106[38], # left eye
lm106[88], # right eye
lm106[86], # nose tip
lm106[52], # left mouth
lm106[61], # right mouth
], dtype=np.float32)
if landmarks is None or len(landmarks) < 5:
return None, None
M = cv2.estimateAffinePartial2D(landmarks, template, method=cv2.LMEDS)[0]
if M is None:
return None, None
inv_M = cv2.invertAffineTransform(M)
return M, inv_M
def enhance_face_onnx(
frame: np.ndarray,
face: Any,
session: onnxruntime.InferenceSession,
input_size: int,
) -> np.ndarray:
"""Enhance a single face in the frame using an ONNX face restoration model."""
M, inv_M = _get_face_affine(face, input_size)
if M is None:
return frame
face_crop = cv2.warpAffine(
frame, M, (input_size, input_size),
flags=cv2.INTER_LINEAR, borderMode=cv2.BORDER_REPLICATE,
)
blob = preprocess_face(face_crop, input_size)
with THREAD_SEMAPHORE:
input_name = session.get_inputs()[0].name
output = run_inference(session, input_name, blob)
enhanced = postprocess_face(output)
# Create mask for blending (feathered edges)
mask = np.ones((input_size, input_size), dtype=np.float32)
border = max(1, input_size // 16)
mask[:border, :] = np.linspace(0, 1, border)[:, np.newaxis]
mask[-border:, :] = np.linspace(1, 0, border)[:, np.newaxis]
mask[:, :border] = np.minimum(mask[:, :border], np.linspace(0, 1, border)[np.newaxis, :])
mask[:, -border:] = np.minimum(mask[:, -border:], np.linspace(1, 0, border)[np.newaxis, :])
h, w = frame.shape[:2]
warped_enhanced = cv2.warpAffine(
enhanced, inv_M, (w, h),
flags=cv2.INTER_LINEAR, borderValue=(0, 0, 0),
)
warped_mask = cv2.warpAffine(
mask, inv_M, (w, h),
flags=cv2.INTER_LINEAR, borderValue=0,
)
mask_3ch = warped_mask[:, :, np.newaxis]
result = (warped_enhanced.astype(np.float32) * mask_3ch +
frame.astype(np.float32) * (1.0 - mask_3ch))
return np.clip(result, 0, 255).astype(np.uint8)
+332 -9
View File
@@ -1,12 +1,17 @@
import os
import subprocess
import sys
import importlib
from concurrent.futures import ThreadPoolExecutor
from types import ModuleType
from typing import Any, List, Callable
import numpy as np
from tqdm import tqdm
import modules
import modules.globals
from modules.face_analyser import get_one_face
FRAME_PROCESSORS_MODULES: List[ModuleType] = []
FRAME_PROCESSORS_INTERFACE = [
@@ -17,12 +22,22 @@ FRAME_PROCESSORS_INTERFACE = [
'process_video'
]
ALLOWED_PROCESSORS = {
'face_swapper',
'face_enhancer',
'face_enhancer_gpen256',
'face_enhancer_gpen512'
}
def load_frame_processor_module(frame_processor: str) -> Any:
if frame_processor not in ALLOWED_PROCESSORS:
print(f"Frame processor {frame_processor} is not allowed")
sys.exit()
try:
frame_processor_module = importlib.import_module(f'modules.processors.frame.{frame_processor}')
for method_name in FRAME_PROCESSORS_INTERFACE:
if not hasattr(frame_processor_module, method_name):
print(f"Frame processor {frame_processor} is missing required method {method_name}")
sys.exit()
except ImportError:
print(f"Frame processor {frame_processor} not found")
@@ -45,7 +60,7 @@ def set_frame_processors_modules_from_ui(frame_processors: List[str]) -> None:
current_processor_names = [proc.__name__.split('.')[-1] for proc in FRAME_PROCESSORS_MODULES]
for frame_processor, state in modules.globals.fp_ui.items():
if state == True and frame_processor not in current_processor_names:
if state and frame_processor not in current_processor_names:
try:
frame_processor_module = load_frame_processor_module(frame_processor)
FRAME_PROCESSORS_MODULES.append(frame_processor_module)
@@ -56,7 +71,7 @@ def set_frame_processors_modules_from_ui(frame_processors: List[str]) -> None:
except Exception as e:
print(f"Warning: Error loading frame processor {frame_processor} requested by UI state: {e}")
elif state == False and frame_processor in current_processor_names:
elif not state and frame_processor in current_processor_names:
try:
module_to_remove = next((mod for mod in FRAME_PROCESSORS_MODULES if mod.__name__.endswith(f'.{frame_processor}')), None)
if module_to_remove:
@@ -67,13 +82,29 @@ def set_frame_processors_modules_from_ui(frame_processors: List[str]) -> None:
print(f"Warning: Error removing frame processor {frame_processor}: {e}")
def multi_process_frame(source_path: str, temp_frame_paths: List[str], process_frames: Callable[[str, List[str], Any], None], progress: Any = None) -> None:
with ThreadPoolExecutor(max_workers=modules.globals.execution_threads) as executor:
futures = []
for path in temp_frame_paths:
future = executor.submit(process_frames, source_path, [path], progress)
futures.append(future)
for future in futures:
future.result()
"""Process frames in parallel with optimized batching and memory management."""
max_workers = modules.globals.execution_threads
# Determine optimal batch size based on available memory and thread count
# Process frames in batches to avoid memory overflow
batch_size = max(1, min(32, len(temp_frame_paths) // max(1, max_workers)))
with ThreadPoolExecutor(max_workers=max_workers) as executor:
# Process in batches to manage memory better
for i in range(0, len(temp_frame_paths), batch_size):
batch = temp_frame_paths[i:i + batch_size]
futures = []
for path in batch:
future = executor.submit(process_frames, source_path, [path], progress)
futures.append(future)
# Wait for batch to complete before starting next batch
for future in futures:
try:
future.result()
except Exception as e:
print(f"Error processing frame: {e}")
def process_video(source_path: str, frame_paths: list[str], process_frames: Callable[[str, List[str], Any], None]) -> None:
@@ -82,3 +113,295 @@ def process_video(source_path: str, frame_paths: list[str], process_frames: Call
with tqdm(total=total, desc='Processing', unit='frame', dynamic_ncols=True, bar_format=progress_bar_format) as progress:
progress.set_postfix({'execution_providers': modules.globals.execution_providers, 'execution_threads': modules.globals.execution_threads, 'max_memory': modules.globals.max_memory})
multi_process_frame(source_path, frame_paths, process_frames, progress)
def process_video_in_memory(source_path: str, target_path: str, fps: float) -> bool:
"""Process video frames in-memory using FFmpeg pipes, eliminating disk I/O.
Reads raw frames from the source video via an FFmpeg decoder pipe, runs each
frame through all active frame processors sequentially, and writes the
result directly to an FFmpeg encoder pipe. This avoids extracting frames to
PNG on disk, which is the biggest I/O bottleneck in the disk-based pipeline.
Returns True on success, False on failure (caller should fall back to the
disk-based pipeline).
"""
from modules import imread_unicode
from modules.face_analyser import get_one_face
from modules.utilities import (
get_video_dimensions,
estimate_frame_count,
get_temp_output_path,
)
temp_output_path = get_temp_output_path(target_path)
# --- Pre-load source face (needed by face_swapper in simple mode) ---
source_face = None
if source_path and os.path.exists(source_path):
source_img = imread_unicode(source_path)
if source_img is not None:
source_face = get_one_face(source_img)
del source_img
if source_face is None:
print("[DLC.CORE] Warning: No face detected in source image. "
"Face swapping will be skipped.")
# --- Collect frame processors & reset per-video state ---
frame_processors = get_frame_processors_modules(modules.globals.frame_processors)
for fp in frame_processors:
if hasattr(fp, 'PREVIOUS_FRAME_RESULT'):
fp.PREVIOUS_FRAME_RESULT = None
# --- Video metadata ---
try:
width, height = get_video_dimensions(target_path)
except Exception as e:
print(f"[DLC.CORE] Failed to get video dimensions: {e}")
return False
total_frames = estimate_frame_count(target_path, fps)
frame_size = width * height * 3
# --- Build encoder arguments ---
encoder = modules.globals.video_encoder
encoder_options: List[str] = []
is_hw_encoder = False
if 'CUDAExecutionProvider' in modules.globals.execution_providers:
if encoder == 'libx264':
encoder = 'h264_nvenc'
is_hw_encoder = True
encoder_options = [
'-preset', 'p4', '-tune', 'hq', '-rc', 'vbr',
'-cq', str(modules.globals.video_quality), '-b:v', '0',
]
elif encoder == 'libx265':
encoder = 'hevc_nvenc'
is_hw_encoder = True
encoder_options = [
'-preset', 'p4', '-tune', 'hq', '-rc', 'vbr',
'-cq', str(modules.globals.video_quality), '-b:v', '0',
]
elif 'DmlExecutionProvider' in modules.globals.execution_providers:
if encoder == 'libx264':
encoder = 'h264_amf'
is_hw_encoder = True
encoder_options = [
'-quality', 'quality', '-rc', 'vbr_latency',
'-qp_i', str(modules.globals.video_quality),
'-qp_p', str(modules.globals.video_quality),
]
elif encoder == 'libx265':
encoder = 'hevc_amf'
is_hw_encoder = True
encoder_options = [
'-quality', 'quality', '-rc', 'vbr_latency',
'-qp_i', str(modules.globals.video_quality),
'-qp_p', str(modules.globals.video_quality),
]
if not is_hw_encoder:
if encoder == 'libx264':
encoder_options = [
'-preset', 'medium',
'-crf', str(modules.globals.video_quality),
'-tune', 'film',
]
elif encoder == 'libx265':
encoder_options = [
'-preset', 'medium',
'-crf', str(modules.globals.video_quality),
'-x265-params', 'log-level=error',
]
elif encoder == 'libvpx-vp9':
encoder_options = [
'-crf', str(modules.globals.video_quality),
'-b:v', '0', '-cpu-used', '2',
]
# --- Attempt pipeline (hw encoder first, then sw fallback) ---
encoders_to_try = [(encoder, encoder_options)]
if is_hw_encoder:
# Software fallback
sw_encoder = 'libx264'
sw_options = [
'-preset', 'medium',
'-crf', str(modules.globals.video_quality),
'-tune', 'film',
]
encoders_to_try.append((sw_encoder, sw_options))
for attempt, (enc, enc_opts) in enumerate(encoders_to_try):
# Reset interpolation state on retry
if attempt > 0:
for fp in frame_processors:
if hasattr(fp, 'PREVIOUS_FRAME_RESULT'):
fp.PREVIOUS_FRAME_RESULT = None
success = _run_pipe_pipeline(
target_path, temp_output_path, fps,
source_face, frame_processors,
width, height, frame_size, total_frames,
enc, enc_opts,
)
if success:
return True
if attempt == 0 and is_hw_encoder:
print(f"[DLC.CORE] Hardware encoder '{enc}' failed, "
f"retrying with software encoder...")
return False
def _run_pipe_pipeline(
target_path: str,
temp_output_path: str,
fps: float,
source_face: Any,
frame_processors: List[Any],
width: int,
height: int,
frame_size: int,
total_frames: int,
encoder: str,
encoder_options: List[str],
) -> bool:
"""Run the FFmpeg-pipe read → process → encode pipeline once."""
# --- Reader: decode source video to raw BGR24 on stdout ---
reader_cmd = [
'ffmpeg', '-hide_banner',
'-hwaccel', 'auto',
'-i', target_path,
'-f', 'rawvideo',
'-pix_fmt', 'bgr24',
'-v', 'error',
'-',
]
# --- Writer: encode raw BGR24 from stdin ---
writer_cmd = [
'ffmpeg', '-hide_banner',
'-f', 'rawvideo',
'-pix_fmt', 'bgr24',
'-s', f'{width}x{height}',
'-r', str(fps),
'-i', '-',
'-c:v', encoder,
]
writer_cmd.extend(encoder_options)
writer_cmd.extend([
'-pix_fmt', 'yuv420p',
'-movflags', '+faststart',
'-vf', 'colorspace=bt709:iall=bt601-6-625:fast=1',
'-v', 'error',
'-y', temp_output_path,
])
reader = None
writer = None
try:
reader = subprocess.Popen(
reader_cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
)
writer = subprocess.Popen(
writer_cmd, stdin=subprocess.PIPE, stderr=subprocess.PIPE,
)
except Exception as e:
print(f"[DLC.CORE] Failed to start FFmpeg pipes: {e}")
for proc in (reader, writer):
if proc:
try:
proc.kill()
except Exception:
pass
return False
processed_count = 0
bar_fmt = ('{l_bar}{bar}| {n_fmt}/{total_fmt} '
'[{elapsed}<{remaining}, {rate_fmt}{postfix}]')
try:
with tqdm(total=total_frames, desc='Processing', unit='frame',
dynamic_ncols=True, bar_format=bar_fmt) as progress:
progress.set_postfix({
'execution_providers': modules.globals.execution_providers,
'threads': modules.globals.execution_threads,
'mode': 'in-memory',
})
# Pipelined detection: while processing frame N (swap on
# ANE), start detecting the face in the next frame
# (detection on GPU). They use different hardware units
# so the work overlaps.
detect_executor = ThreadPoolExecutor(max_workers=1)
pending_detect = None
use_pipeline = not modules.globals.many_faces
while True:
raw = reader.stdout.read(frame_size)
if len(raw) != frame_size:
break
frame = np.frombuffer(raw, dtype=np.uint8).reshape(
(height, width, 3)
).copy()
# Get the detection result for THIS frame
if use_pipeline:
if pending_detect is not None:
target_face = pending_detect.result()
else:
target_face = get_one_face(frame)
# Start detecting on THIS frame eagerly — the result
# will be used for the next iteration. At video
# frame rates the face barely moves between frames.
# Hand the detector its own copy: the frame processors
# below mutate `frame` in place (paste-back), which
# would otherwise race with detection.
pending_detect = detect_executor.submit(
get_one_face, frame.copy())
else:
target_face = None
# Run frame through every active processor
for fp in frame_processors:
try:
frame = fp.process_frame(source_face, frame, target_face=target_face)
except TypeError:
frame = fp.process_frame(source_face, frame)
writer.stdin.write(frame.tobytes())
processed_count += 1
progress.update(1)
detect_executor.shutdown(wait=True)
# Graceful shutdown
writer.stdin.close()
writer.wait()
reader.wait()
if writer.returncode != 0:
stderr_out = writer.stderr.read().decode(errors='ignore').strip()
if stderr_out:
print(f"[DLC.CORE] FFmpeg encoder error: {stderr_out}")
return False
return processed_count > 0 and os.path.isfile(temp_output_path)
except BrokenPipeError:
print("[DLC.CORE] FFmpeg pipe broken (encoder may not be available).")
return False
except Exception as e:
print(f"[DLC.CORE] In-memory processing error: {e}")
return False
finally:
for proc in (reader, writer):
if proc:
try:
proc.kill()
except Exception:
pass
+360 -115
View File
@@ -1,20 +1,20 @@
# --- START OF FILE face_enhancer.py ---
# Uses ONNX Runtime for GFPGAN face enhancement (no torch/gfpgan dependency)
from typing import Any, List
import cv2
import threading
import gfpgan
import numpy as np
import os
import platform
import torch # Make sure torch is imported
import onnxruntime
import modules.globals
import modules.processors.frame.core
from modules import imread_unicode, imwrite_unicode
from modules.core import update_status
from modules.face_analyser import get_one_face
from modules.face_analyser import get_many_faces
from modules.typing import Frame, Face
from modules.utilities import (
conditional_download,
is_image,
is_video,
)
@@ -23,21 +23,37 @@ FACE_ENHANCER = None
THREAD_SEMAPHORE = threading.Semaphore()
THREAD_LOCK = threading.Lock()
NAME = "DLC.FACE-ENHANCER"
MODEL_FILE = "gfpgan-1024.onnx"
abs_dir = os.path.dirname(os.path.abspath(__file__))
models_dir = os.path.join(
os.path.dirname(os.path.dirname(os.path.dirname(abs_dir))), "models"
)
# Standard FFHQ 5-point face template for 512x512 resolution
# Points: left_eye, right_eye, nose, left_mouth, right_mouth
FFHQ_TEMPLATE_512 = np.array(
[
[192.98138, 239.94708],
[318.90277, 240.19366],
[256.63416, 314.01935],
[201.26117, 371.41043],
[313.08905, 371.15118],
],
dtype=np.float32,
)
def pre_check() -> bool:
download_directory_path = models_dir
conditional_download(
download_directory_path,
[
"https://github.com/TencentARC/GFPGAN/releases/download/v1.3.4/GFPGANv1.4.pth"
],
)
from modules.model_downloader import ensure_model
if ensure_model(MODEL_FILE) is None:
update_status(
f"Could not obtain {MODEL_FILE}. Place it in the models folder "
"manually or check your internet connection.",
NAME,
)
return False
return True
@@ -50,157 +66,386 @@ def pre_start() -> bool:
return True
def get_face_enhancer() -> Any:
def get_face_enhancer() -> onnxruntime.InferenceSession:
"""
Initializes and returns the GFPGAN face enhancer instance,
prioritizing CUDA, then MPS (Mac), then CPU.
Initializes and returns the GFPGAN ONNX Runtime inference session,
using the execution providers configured in modules.globals.
"""
global FACE_ENHANCER
with THREAD_LOCK:
if FACE_ENHANCER is None:
model_path = os.path.join(models_dir, "GFPGANv1.4.pth")
device = None
try:
# Priority 1: CUDA
if torch.cuda.is_available():
device = torch.device("cuda")
print(f"{NAME}: Using CUDA device.")
# Priority 2: MPS (Mac Silicon)
elif platform.system() == "Darwin" and torch.backends.mps.is_available():
device = torch.device("mps")
print(f"{NAME}: Using MPS device.")
# Priority 3: CPU
else:
device = torch.device("cpu")
print(f"{NAME}: Using CPU device.")
from modules.model_downloader import ensure_model
FACE_ENHANCER = gfpgan.GFPGANer(
model_path=model_path,
upscale=1, # upscale=1 means enhancement only, no resizing
arch='clean',
channel_multiplier=2,
bg_upsampler=None,
device=device
model_path = ensure_model(MODEL_FILE)
if model_path is None:
raise FileNotFoundError(
f"{NAME}: Model not found at "
f"{os.path.join(models_dir, MODEL_FILE)} and could not be "
"downloaded"
)
print(f"{NAME}: GFPGANer initialized successfully on {device}.")
try:
from modules.processors.frame._onnx_enhancer import (
create_onnx_session,
)
FACE_ENHANCER = create_onnx_session(model_path)
input_info = FACE_ENHANCER.get_inputs()[0]
output_info = FACE_ENHANCER.get_outputs()[0]
active_providers = FACE_ENHANCER.get_providers()
print(
f"{NAME}: GFPGAN ONNX model loaded successfully."
)
print(
f"{NAME}: Input: {input_info.name}, "
f"shape: {input_info.shape}, type: {input_info.type}"
)
print(
f"{NAME}: Output: {output_info.name}, "
f"shape: {output_info.shape}, type: {output_info.type}"
)
print(f"{NAME}: Active providers: {active_providers}")
except Exception as e:
print(f"{NAME}: Error initializing GFPGANer: {e}")
# Fallback to CPU if initialization with GPU fails for some reason
if device is not None and device.type != 'cpu':
print(f"{NAME}: Falling back to CPU due to error.")
try:
device = torch.device("cpu")
FACE_ENHANCER = gfpgan.GFPGANer(
model_path=model_path,
upscale=1,
arch='clean',
channel_multiplier=2,
bg_upsampler=None,
device=device
)
print(f"{NAME}: GFPGANer initialized successfully on CPU after fallback.")
except Exception as fallback_e:
print(f"{NAME}: FATAL: Could not initialize GFPGANer even on CPU: {fallback_e}")
FACE_ENHANCER = None # Ensure it's None if totally failed
else:
# If it failed even on the first CPU attempt or device was already CPU
print(f"{NAME}: FATAL: Could not initialize GFPGANer on CPU: {e}")
FACE_ENHANCER = None # Ensure it's None if totally failed
print(f"{NAME}: Error loading GFPGAN ONNX model: {e}")
FACE_ENHANCER = None
raise RuntimeError(
f"{NAME}: Failed to load GFPGAN ONNX model: {e}"
)
# Check if enhancer is still None after attempting initialization
if FACE_ENHANCER is None:
raise RuntimeError(f"{NAME}: Failed to initialize GFPGANer. Check logs for errors.")
raise RuntimeError(
f"{NAME}: Failed to initialize GFPGAN ONNX session. Check logs."
)
return FACE_ENHANCER
def enhance_face(temp_frame: Frame) -> Frame:
"""Enhances faces in a single frame using the global GFPGANer instance."""
# Ensure enhancer is ready
enhancer = get_face_enhancer()
def _align_face(
frame: Frame, landmarks_5: np.ndarray, output_size: int
) -> tuple:
"""
Align and crop a face from the frame using 5-point landmarks and the
standard FFHQ template.
Returns:
(aligned_face, affine_matrix) or (None, None) on failure.
"""
# Scale the 512-base template to the desired output size
scale = output_size / 512.0
template = FFHQ_TEMPLATE_512 * scale
# Estimate a similarity transform (4 DOF: rotation, scale, tx, ty)
affine_matrix, _ = cv2.estimateAffinePartial2D(
landmarks_5, template, method=cv2.LMEDS
)
if affine_matrix is None:
return None, None
# Warp the face to the aligned position
aligned_face = cv2.warpAffine(
frame,
affine_matrix,
(output_size, output_size),
borderMode=cv2.BORDER_CONSTANT,
borderValue=(135, 133, 132),
)
return aligned_face, affine_matrix
_HAS_TORCH_CUDA = False
try:
import torch
if torch.cuda.is_available():
_HAS_TORCH_CUDA = True
except ImportError:
pass
# Cache the feathered mask — it's the same for every call at a given size
_enhancer_cache: dict = {'mask': None, 'mask_size': 0}
def _paste_back(
frame: Frame,
enhanced_face: np.ndarray,
affine_matrix: np.ndarray,
output_size: int,
) -> Frame:
"""
Paste an enhanced (aligned) face back onto the original frame using the
inverse affine transform with feathered-edge blending.
Optimized: operates on a tight crop around the face bbox instead of the
full frame, and uses GPU for blending when available.
"""
h, w = frame.shape[:2]
inv_matrix = cv2.invertAffineTransform(affine_matrix)
# Build or reuse cached feathered mask (uint8 — blended via cv2 SIMD ops)
if _enhancer_cache['mask_size'] != output_size:
face_mask_f = np.ones((output_size, output_size), dtype=np.float32)
border = max(1, int(output_size * 0.05))
ramp_up = np.linspace(0.0, 1.0, border, dtype=np.float32)
ramp_down = np.linspace(1.0, 0.0, border, dtype=np.float32)
face_mask_f[:border, :] *= ramp_up[:, None]
face_mask_f[-border:, :] *= ramp_down[:, None]
face_mask_f[:, :border] *= ramp_up[None, :]
face_mask_f[:, -border:] *= ramp_down[None, :]
_enhancer_cache['mask'] = (face_mask_f * 255.0).astype(np.uint8)
_enhancer_cache['mask_size'] = output_size
# Compute tight bbox from affine corners (avoids full-frame warpAffine scan)
corners = np.array([[0, 0], [output_size, 0],
[output_size, output_size], [0, output_size]],
dtype=np.float32)
transformed = (inv_matrix[:, :2] @ corners.T).T + inv_matrix[:, 2]
x1 = max(0, int(np.floor(transformed[:, 0].min())))
x2 = min(w, int(np.ceil(transformed[:, 0].max())))
y1 = max(0, int(np.floor(transformed[:, 1].min())))
y2 = min(h, int(np.ceil(transformed[:, 1].max())))
if x1 >= x2 or y1 >= y2:
return frame
# Pad a few pixels for feathering
pad = max(1, int(output_size * 0.05)) + 2
y1p, y2p = max(0, y1 - pad), min(h, y2 + pad)
x1p, x2p = max(0, x1 - pad), min(w, x2 + pad)
crop_w, crop_h = x2p - x1p, y2p - y1p
# Warp enhanced face and mask into crop space only
inv_crop = inv_matrix.copy()
inv_crop[0, 2] -= x1p
inv_crop[1, 2] -= y1p
inv_restored_crop = cv2.warpAffine(
enhanced_face, inv_crop, (crop_w, crop_h),
borderMode=cv2.BORDER_CONSTANT, borderValue=(0, 0, 0),
)
inv_mask_crop = cv2.warpAffine(
_enhancer_cache['mask'], inv_crop, (crop_w, crop_h),
borderMode=cv2.BORDER_CONSTANT, borderValue=0,
)
target_crop = frame[y1p:y2p, x1p:x2p]
if _HAS_TORCH_CUDA:
# Upload uint8 alpha — smaller transfer, scale on device.
mask_t = torch.from_numpy(inv_mask_crop).cuda().float().mul_(1.0 / 255.0).unsqueeze(2)
enhanced_t = torch.from_numpy(inv_restored_crop).float().cuda()
target_t = torch.from_numpy(target_crop).float().cuda()
blended = (mask_t * enhanced_t + (1.0 - mask_t) * target_t
).to(torch.uint8).cpu().numpy()
frame[y1p:y2p, x1p:x2p] = blended
else:
# Fused uint8 blend via cv2 SIMD — ~7× faster than the float32 round-trip.
alpha_3c = cv2.merge([inv_mask_crop, inv_mask_crop, inv_mask_crop])
inv_alpha = 255 - alpha_3c
a_enh = cv2.multiply(inv_restored_crop, alpha_3c, scale=1.0 / 255.0)
a_tgt = cv2.multiply(target_crop, inv_alpha, scale=1.0 / 255.0)
frame[y1p:y2p, x1p:x2p] = cv2.add(a_enh, a_tgt)
return frame
def _preprocess_face(aligned_face: np.ndarray) -> np.ndarray:
"""
Convert an aligned BGR uint8 face image to the ONNX model input tensor.
Format: NCHW float32, normalised to [-1, 1].
"""
# BGR -> RGB, normalize, and transpose in one pass
# Fused: (x / 255.0 - 0.5) / 0.5 = x / 127.5 - 1.0
rgb = aligned_face[:, :, ::-1] # BGR->RGB zero-copy view
chw = np.transpose(rgb, (2, 0, 1)).astype(np.float32)
chw *= (1.0 / 127.5)
chw -= 1.0
return chw[np.newaxis, ...] # shape: (1, 3, H, W)
def _postprocess_face(output: np.ndarray) -> np.ndarray:
"""
Convert the ONNX model output tensor back to a BGR uint8 image.
Expects input in NCHW format with values in [-1, 1].
"""
# Fused: ((x + 1.0) / 2.0) * 255 = (x + 1.0) * 127.5
face = output[0] # remove batch dim -> (3, H, W)
face = (face + 1.0) * 127.5
np.clip(face, 0, 255, out=face)
face = face.astype(np.uint8).transpose(1, 2, 0) # CHW -> HWC
return face[:, :, ::-1].copy() # RGB -> BGR
# Cache for temporal enhancement skipping in live mode.
# GFPGAN output barely changes between consecutive frames (same face,
# same position), so we run inference every _ENH_INTERVAL frames and
# reuse the cached enhanced face + affine matrix in between.
_enh_live_cache: dict = {
'enhanced_bgr': None,
'affine_matrix': None,
'align_size': 0,
'frame_count': 0,
}
_ENH_INTERVAL = 2 # run inference every N frames, paste cached result otherwise
def enhance_face(temp_frame: Frame, detected_faces=None) -> Frame:
"""Enhances all faces in a frame using the GFPGAN ONNX model.
Args:
detected_faces: Pre-detected face list. When provided, skips
the internal detection call (saves ~15-20ms per frame).
Also enables temporal caching — inference runs every
_ENH_INTERVAL frames, reusing the cached result otherwise.
"""
session = get_face_enhancer()
# Determine model input resolution from the session metadata
input_info = session.get_inputs()[0]
input_name = input_info.name
input_shape = input_info.shape # e.g. [1, 3, 512, 512]
try:
with THREAD_SEMAPHORE:
# The enhance method returns: _, restored_faces, restored_img
_, _, restored_img = enhancer.enhance(
temp_frame,
has_aligned=False, # Assume faces are not pre-aligned
only_center_face=False, # Enhance all detected faces
paste_back=True # Paste enhanced faces back onto the original image
)
# GFPGAN might return None if no face is detected or an error occurs
if restored_img is None:
# print(f"{NAME}: Warning: GFPGAN enhancement returned None. Returning original frame.")
return temp_frame
return restored_img
except Exception as e:
print(f"{NAME}: Error during face enhancement: {e}")
# Return the original frame in case of error during enhancement
align_size = int(input_shape[2])
if align_size <= 0:
align_size = 512
except (ValueError, TypeError, IndexError):
align_size = 512
# Use pre-detected faces if available, otherwise detect
faces = detected_faces if detected_faces is not None else get_many_faces(temp_frame)
if not faces:
return temp_frame
# Temporal caching: only available when faces are pre-detected (live mode)
# AND we're in single-face mode — the cache holds exactly one enhancement,
# so reusing it in many_faces mode would paste the same face onto every
# detected target.
many_faces_mode = getattr(modules.globals, "many_faces", False)
use_cache = detected_faces is not None and not many_faces_mode
if use_cache:
_enh_live_cache['frame_count'] += 1
run_inference_this_frame = (_enh_live_cache['frame_count'] % _ENH_INTERVAL == 0
or _enh_live_cache['enhanced_bgr'] is None)
else:
run_inference_this_frame = True
for face in faces:
if not hasattr(face, "kps") or face.kps is None:
continue
landmarks_5 = face.kps.astype(np.float32)
if landmarks_5.shape[0] < 5:
continue
if run_inference_this_frame:
aligned_face, affine_matrix = _align_face(
temp_frame, landmarks_5, output_size=align_size
)
if aligned_face is None or affine_matrix is None:
continue
try:
with THREAD_SEMAPHORE:
from modules.processors.frame._onnx_enhancer import (
run_inference,
)
input_tensor = _preprocess_face(aligned_face)
output_tensor = run_inference(session, input_name, input_tensor)
enhanced_bgr = _postprocess_face(output_tensor)
eh, ew = enhanced_bgr.shape[:2]
if eh != align_size or ew != align_size:
enhanced_bgr = cv2.resize(
enhanced_bgr,
(align_size, align_size),
interpolation=cv2.INTER_LANCZOS4,
)
# Cache for reuse on next frame
if use_cache:
_enh_live_cache['enhanced_bgr'] = enhanced_bgr
_enh_live_cache['affine_matrix'] = affine_matrix
_enh_live_cache['align_size'] = align_size
_paste_back(
temp_frame, enhanced_bgr, affine_matrix, output_size=align_size
)
except Exception as e:
print(f"{NAME}: Error enhancing a face: {e}")
continue
else:
# Reuse cached enhanced face — just paste back onto current frame
cached = _enh_live_cache
if cached['enhanced_bgr'] is not None:
_paste_back(
temp_frame, cached['enhanced_bgr'],
cached['affine_matrix'],
output_size=cached['align_size'],
)
if not many_faces_mode:
break # single-face live mode — only process first face
def process_frame(source_face: Face | None, temp_frame: Frame) -> Frame:
"""Processes a frame: enhances face if detected."""
# We don't strictly need source_face for enhancement only
# Check if any face exists to potentially save processing time, though GFPGAN also does detection.
# For simplicity and ensuring enhancement is attempted if possible, we can rely on enhance_face.
# target_face = get_one_face(temp_frame) # This gets only ONE face
# If you want to enhance ONLY if a face is detected by your *own* analyser first:
# has_face = get_one_face(temp_frame) is not None # Or use get_many_faces
# if has_face:
# temp_frame = enhance_face(temp_frame)
# else: # Enhance regardless, let GFPGAN handle detection
temp_frame = enhance_face(temp_frame)
return temp_frame
def process_frame(source_face: Face | None, temp_frame: Frame,
detected_faces=None) -> Frame:
"""Processes a frame: enhances face if detected."""
return enhance_face(temp_frame, detected_faces=detected_faces)
def process_frame_v2(temp_frame: Frame, detected_faces=None) -> Frame:
"""Processes a frame without source face (used by live webcam preview)."""
return enhance_face(temp_frame, detected_faces=detected_faces)
def process_frames(
source_path: str | None, temp_frame_paths: List[str], progress: Any = None
) -> None:
"""Processes multiple frames from file paths."""
for temp_frame_path in temp_frame_paths:
if not os.path.exists(temp_frame_path):
print(f"{NAME}: Warning: Frame path not found {temp_frame_path}, skipping.")
print(
f"{NAME}: Warning: Frame path not found {temp_frame_path}, skipping."
)
if progress:
progress.update(1)
continue
temp_frame = cv2.imread(temp_frame_path)
temp_frame = imread_unicode(temp_frame_path)
if temp_frame is None:
print(f"{NAME}: Warning: Failed to read frame {temp_frame_path}, skipping.")
print(
f"{NAME}: Warning: Failed to read frame {temp_frame_path}, skipping."
)
if progress:
progress.update(1)
continue
result_frame = process_frame(None, temp_frame)
cv2.imwrite(temp_frame_path, result_frame)
imwrite_unicode(temp_frame_path, result_frame)
if progress:
progress.update(1)
def process_image(source_path: str | None, target_path: str, output_path: str) -> None:
def process_image(
source_path: str | None, target_path: str, output_path: str
) -> None:
"""Processes a single image file."""
target_frame = cv2.imread(target_path)
target_frame = imread_unicode(target_path)
if target_frame is None:
print(f"{NAME}: Error: Failed to read target image {target_path}")
return
result_frame = process_frame(None, target_frame)
cv2.imwrite(output_path, result_frame)
imwrite_unicode(output_path, result_frame)
print(f"{NAME}: Enhanced image saved to {output_path}")
def process_video(source_path: str | None, temp_frame_paths: List[str]) -> None:
def process_video(
source_path: str | None, temp_frame_paths: List[str]
) -> None:
"""Processes video frames using the frame processor core."""
# source_path might be optional depending on how process_video is called
modules.processors.frame.core.process_video(source_path, temp_frame_paths, process_frames)
# Optional: Keep process_frame_v2 if it's used elsewhere, otherwise it's redundant
# def process_frame_v2(temp_frame: Frame) -> Frame:
# target_face = get_one_face(temp_frame)
# if target_face:
# temp_frame = enhance_face(temp_frame)
# return temp_frame
# --- END OF FILE face_enhancer.py ---
modules.processors.frame.core.process_video(
source_path, temp_frame_paths, process_frames
)
@@ -0,0 +1,146 @@
"""GPEN-BFR-256 face enhancer — ONNX-based face restoration at 256x256."""
from typing import Any, List
import os
import threading
import modules.globals
import modules.processors.frame.core
from modules import imread_unicode, imwrite_unicode
from modules.core import update_status
from modules.face_analyser import get_one_face
from modules.typing import Frame, Face
from modules.utilities import (
is_image,
is_video,
)
from modules.processors.frame._onnx_enhancer import (
create_onnx_session,
warmup_session,
enhance_face_onnx,
)
NAME = "DLC.FACE-ENHANCER-GPEN256"
INPUT_SIZE = 256
MODEL_MIRROR_URL = "https://github.com/harisreedhar/Face-Upscalers-ONNX/releases/download/GPEN-BFR/GPEN-BFR-256.onnx"
MODEL_FILE = "GPEN-BFR-256.onnx"
ENHANCER = None
THREAD_LOCK = threading.Lock()
abs_dir = os.path.dirname(os.path.abspath(__file__))
models_dir = os.path.join(
os.path.dirname(os.path.dirname(os.path.dirname(abs_dir))), "models"
)
def _obtain_model():
from modules.model_downloader import ensure_model
model_path = ensure_model(MODEL_FILE)
if model_path is not None:
return model_path
update_status(f"Retrying {MODEL_FILE} from the mirror...", NAME)
from modules.utilities import conditional_download
try:
conditional_download(models_dir, [MODEL_MIRROR_URL])
except Exception as error:
update_status(f"Mirror download failed: {error}", NAME)
return None
fallback = os.path.join(models_dir, MODEL_FILE)
return fallback if os.path.exists(fallback) else None
def pre_check() -> bool:
if _obtain_model() is None:
update_status(
f"Could not obtain {MODEL_FILE}. Place it in the models folder "
"manually or check your internet connection.",
NAME,
)
return False
return True
def pre_start() -> bool:
if not is_image(modules.globals.target_path) and not is_video(modules.globals.target_path):
update_status("Select an image or video for target path.", NAME)
return False
return True
def get_enhancer() -> Any:
global ENHANCER
with THREAD_LOCK:
if ENHANCER is None:
model_path = _obtain_model()
if model_path is None:
raise FileNotFoundError(
f"Model file not found: {os.path.join(models_dir, MODEL_FILE)}"
)
print(f"{NAME}: Loading ONNX model from {model_path}")
ENHANCER = create_onnx_session(model_path)
warmup_session(ENHANCER)
print(f"{NAME}: Model loaded successfully.")
return ENHANCER
def enhance_face(temp_frame: Frame, face: Face) -> Frame:
try:
session = get_enhancer()
except Exception as e:
print(f"{NAME}: {e}")
return temp_frame
try:
return enhance_face_onnx(temp_frame, face, session, INPUT_SIZE)
except Exception as e:
print(f"{NAME}: Error during face enhancement: {e}")
return temp_frame
def process_frame(source_face: Face | None, temp_frame: Frame, detected_faces=None) -> Frame:
if detected_faces:
target_face = detected_faces[0]
else:
target_face = get_one_face(temp_frame)
if target_face is None:
return temp_frame
return enhance_face(temp_frame, target_face)
def process_frame_v2(temp_frame: Frame) -> Frame:
target_face = get_one_face(temp_frame)
if target_face:
temp_frame = enhance_face(temp_frame, target_face)
return temp_frame
def process_frames(
source_path: str | None, temp_frame_paths: List[str], progress: Any = None
) -> None:
for temp_frame_path in temp_frame_paths:
temp_frame = imread_unicode(temp_frame_path)
if temp_frame is None:
if progress:
progress.update(1)
continue
result = process_frame(None, temp_frame)
imwrite_unicode(temp_frame_path, result)
if progress:
progress.update(1)
def process_image(source_path: str | None, target_path: str, output_path: str) -> None:
target_frame = imread_unicode(target_path)
if target_frame is None:
print(f"{NAME}: Error: Failed to read target image {target_path}")
return
result_frame = process_frame(None, target_frame)
imwrite_unicode(output_path, result_frame)
print(f"{NAME}: Enhanced image saved to {output_path}")
def process_video(source_path: str | None, temp_frame_paths: List[str]) -> None:
modules.processors.frame.core.process_video(source_path, temp_frame_paths, process_frames)
@@ -0,0 +1,146 @@
"""GPEN-BFR-512 face enhancer — ONNX-based face restoration at 512x512."""
from typing import Any, List
import os
import threading
import modules.globals
import modules.processors.frame.core
from modules import imread_unicode, imwrite_unicode
from modules.core import update_status
from modules.face_analyser import get_one_face
from modules.typing import Frame, Face
from modules.utilities import (
is_image,
is_video,
)
from modules.processors.frame._onnx_enhancer import (
create_onnx_session,
warmup_session,
enhance_face_onnx,
)
NAME = "DLC.FACE-ENHANCER-GPEN512"
INPUT_SIZE = 512
MODEL_MIRROR_URL = "https://github.com/harisreedhar/Face-Upscalers-ONNX/releases/download/GPEN-BFR/GPEN-BFR-512.onnx"
MODEL_FILE = "GPEN-BFR-512.onnx"
ENHANCER = None
THREAD_LOCK = threading.Lock()
abs_dir = os.path.dirname(os.path.abspath(__file__))
models_dir = os.path.join(
os.path.dirname(os.path.dirname(os.path.dirname(abs_dir))), "models"
)
def _obtain_model():
from modules.model_downloader import ensure_model
model_path = ensure_model(MODEL_FILE)
if model_path is not None:
return model_path
update_status(f"Retrying {MODEL_FILE} from the mirror...", NAME)
from modules.utilities import conditional_download
try:
conditional_download(models_dir, [MODEL_MIRROR_URL])
except Exception as error:
update_status(f"Mirror download failed: {error}", NAME)
return None
fallback = os.path.join(models_dir, MODEL_FILE)
return fallback if os.path.exists(fallback) else None
def pre_check() -> bool:
if _obtain_model() is None:
update_status(
f"Could not obtain {MODEL_FILE}. Place it in the models folder "
"manually or check your internet connection.",
NAME,
)
return False
return True
def pre_start() -> bool:
if not is_image(modules.globals.target_path) and not is_video(modules.globals.target_path):
update_status("Select an image or video for target path.", NAME)
return False
return True
def get_enhancer() -> Any:
global ENHANCER
with THREAD_LOCK:
if ENHANCER is None:
model_path = _obtain_model()
if model_path is None:
raise FileNotFoundError(
f"Model file not found: {os.path.join(models_dir, MODEL_FILE)}"
)
print(f"{NAME}: Loading ONNX model from {model_path}")
ENHANCER = create_onnx_session(model_path)
warmup_session(ENHANCER)
print(f"{NAME}: Model loaded successfully.")
return ENHANCER
def enhance_face(temp_frame: Frame, face: Face) -> Frame:
try:
session = get_enhancer()
except Exception as e:
print(f"{NAME}: {e}")
return temp_frame
try:
return enhance_face_onnx(temp_frame, face, session, INPUT_SIZE)
except Exception as e:
print(f"{NAME}: Error during face enhancement: {e}")
return temp_frame
def process_frame(source_face: Face | None, temp_frame: Frame, detected_faces=None) -> Frame:
if detected_faces:
target_face = detected_faces[0]
else:
target_face = get_one_face(temp_frame)
if target_face is None:
return temp_frame
return enhance_face(temp_frame, target_face)
def process_frame_v2(temp_frame: Frame) -> Frame:
target_face = get_one_face(temp_frame)
if target_face:
temp_frame = enhance_face(temp_frame, target_face)
return temp_frame
def process_frames(
source_path: str | None, temp_frame_paths: List[str], progress: Any = None
) -> None:
for temp_frame_path in temp_frame_paths:
temp_frame = imread_unicode(temp_frame_path)
if temp_frame is None:
if progress:
progress.update(1)
continue
result = process_frame(None, temp_frame)
imwrite_unicode(temp_frame_path, result)
if progress:
progress.update(1)
def process_image(source_path: str | None, target_path: str, output_path: str) -> None:
target_frame = imread_unicode(target_path)
if target_frame is None:
print(f"{NAME}: Error: Failed to read target image {target_path}")
return
result_frame = process_frame(None, target_frame)
imwrite_unicode(output_path, result_frame)
print(f"{NAME}: Enhanced image saved to {output_path}")
def process_video(source_path: str | None, temp_frame_paths: List[str]) -> None:
modules.processors.frame.core.process_video(source_path, temp_frame_paths, process_frames)
+80 -112
View File
@@ -2,27 +2,35 @@ import cv2
import numpy as np
from modules.typing import Face, Frame
import modules.globals
from modules.gpu_processing import gpu_gaussian_blur, gpu_resize
def apply_color_transfer(source, target):
"""
Apply color transfer from target to source image
Apply color transfer from target to source image using LAB color space.
Uses float32 throughout for performance (sufficient precision for 8-bit images).
"""
source = cv2.cvtColor(source, cv2.COLOR_BGR2LAB).astype("float32")
target = cv2.cvtColor(target, cv2.COLOR_BGR2LAB).astype("float32")
# Convert to float32 [0,1] range for proper LAB conversion
source_f32 = source.astype(np.float32) / 255.0
target_f32 = target.astype(np.float32) / 255.0
source_mean, source_std = cv2.meanStdDev(source)
target_mean, target_std = cv2.meanStdDev(target)
source_lab = cv2.cvtColor(source_f32, cv2.COLOR_BGR2LAB)
target_lab = cv2.cvtColor(target_f32, cv2.COLOR_BGR2LAB)
# Reshape mean and std to be broadcastable
source_mean = source_mean.reshape(1, 1, 3)
source_std = source_std.reshape(1, 1, 3)
target_mean = target_mean.reshape(1, 1, 3)
target_std = target_std.reshape(1, 1, 3)
source_mean, source_std = cv2.meanStdDev(source_lab)
target_mean, target_std = cv2.meanStdDev(target_lab)
# Perform the color transfer
source = (source - source_mean) * (target_std / source_std) + target_mean
# Reshape mean and std to be broadcastable (already float64 from meanStdDev, cast to f32)
source_mean = source_mean.reshape(1, 1, 3).astype(np.float32)
source_std = np.maximum(source_std.reshape(1, 1, 3), 1e-6).astype(np.float32)
target_mean = target_mean.reshape(1, 1, 3).astype(np.float32)
target_std = target_std.reshape(1, 1, 3).astype(np.float32)
return cv2.cvtColor(np.clip(source, 0, 255).astype("uint8"), cv2.COLOR_LAB2BGR)
# Perform the color transfer in LAB space
result_lab = (source_lab - source_mean) * (target_std / source_std) + target_mean
# Convert back to BGR and uint8
result_bgr = cv2.cvtColor(result_lab, cv2.COLOR_LAB2BGR)
return np.clip(result_bgr * 255.0, 0, 255).astype(np.uint8)
def create_face_mask(face: Face, frame: Frame) -> np.ndarray:
mask = np.zeros(frame.shape[:2], dtype=np.uint8)
@@ -45,23 +53,22 @@ def create_face_mask(face: Face, frame: Frame) -> np.ndarray:
) # 5% of face width
# Create a slightly larger convex hull for padding
face_outline = landmarks[0:33]
hull = cv2.convexHull(face_outline)
hull_padded = []
for point in hull:
x, y = point[0]
center = np.mean(face_outline, axis=0)
direction = np.array([x, y]) - center
direction = direction / np.linalg.norm(direction)
padded_point = np.array([x, y]) + direction * padding
hull_padded.append(padded_point)
hull_padded = np.array(hull_padded, dtype=np.int32)
# Vectorized hull padding — expand each point outward from center
center = np.mean(face_outline, axis=0, dtype=np.float32)
hull_pts = hull.reshape(-1, 2).astype(np.float32)
directions = hull_pts - center
norms = np.linalg.norm(directions, axis=1, keepdims=True)
norms = np.maximum(norms, 1e-6) # avoid division by zero
directions /= norms
hull_padded = (hull_pts + directions * padding).astype(np.int32)
# Fill the padded convex hull
cv2.fillConvexPoly(mask, hull_padded, 255)
# Smooth the mask edges
mask = cv2.GaussianBlur(mask, (5, 5), 3)
# Smooth the mask edges (GPU-accelerated when available)
mask = gpu_gaussian_blur(mask, (5, 5), 3)
return mask
@@ -70,77 +77,33 @@ def create_lower_mouth_mask(
) -> (np.ndarray, np.ndarray, tuple, np.ndarray):
mask = np.zeros(frame.shape[:2], dtype=np.uint8)
mouth_cutout = None
lower_lip_polygon = None
mouth_box = (0,0,0,0)
landmarks = face.landmark_2d_106
if landmarks is not None:
# 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
lower_lip_order = [
65,
66,
62,
70,
69,
18,
19,
20,
21,
22,
23,
24,
0,
8,
7,
6,
5,
4,
3,
2,
65,
]
lower_lip_landmarks = landmarks[lower_lip_order].astype(
np.float32
) # Use float for precise calculations
# Use outer mouth landmarks (52-71) to capture the full mouth area
lower_lip_order = list(range(52, 72))
if max(lower_lip_order) >= landmarks.shape[0]:
return mask, mouth_cutout, mouth_box, lower_lip_polygon
lower_lip_landmarks = landmarks[lower_lip_order].astype(np.float32)
# Calculate the center of the landmarks
center = np.mean(lower_lip_landmarks, axis=0)
# Expand the landmarks outward using the mouth_mask_size
expansion_factor = (
1 + modules.globals.mask_down_size * modules.globals.mouth_mask_size
) # Adjust expansion based on slider
expanded_landmarks = (lower_lip_landmarks - center) * expansion_factor + center
mouth_mask_size = getattr(modules.globals, "mouth_mask_size", 0.0) # 0-100 slider
expansion_factor = 1 + (mouth_mask_size / 100.0) * 2.5
# Extend the top lip part
toplip_indices = [
20,
0,
1,
2,
3,
4,
5,
] # Indices for landmarks 2, 65, 66, 62, 70, 69, 18
toplip_extension = (
modules.globals.mask_size * modules.globals.mouth_mask_size * 0.5
) # Adjust extension based on slider
for idx in toplip_indices:
direction = expanded_landmarks[idx] - center
direction = direction / np.linalg.norm(direction)
expanded_landmarks[idx] += direction * toplip_extension
# Extend the bottom part (chin area)
chin_indices = [
11,
12,
13,
14,
15,
16,
] # Indices for landmarks 21, 22, 23, 24, 0, 8
chin_extension = 2 * 0.2 # Adjust this factor to control the extension
for idx in chin_indices:
expanded_landmarks[idx][1] += (
expanded_landmarks[idx][1] - center[1]
) * chin_extension
# Expand with extra downward bias toward chin
offsets = lower_lip_landmarks - center
chin_bias = 1 + (mouth_mask_size / 100.0) * 1.5
scale_y = np.where(offsets[:, 1] > 0, expansion_factor * chin_bias, expansion_factor)
expanded_landmarks = lower_lip_landmarks.copy()
expanded_landmarks[:, 0] = center[0] + offsets[:, 0] * expansion_factor
expanded_landmarks[:, 1] = center[1] + offsets[:, 1] * scale_y
# Convert back to integer coordinates
expanded_landmarks = expanded_landmarks.astype(np.int32)
@@ -165,10 +128,12 @@ def create_lower_mouth_mask(
# Create the mask
mask_roi = np.zeros((max_y - min_y, max_x - min_x), dtype=np.uint8)
cv2.fillPoly(mask_roi, [expanded_landmarks - [min_x, min_y]], 255)
# Shift polygon coordinates relative to the ROI's top-left corner
polygon_relative_to_roi = expanded_landmarks - [min_x, min_y]
cv2.fillPoly(mask_roi, [polygon_relative_to_roi], 255)
# Apply Gaussian blur to soften the mask edges
mask_roi = cv2.GaussianBlur(mask_roi, (15, 15), 5)
# Apply Gaussian blur to soften the mask edges (GPU-accelerated when available)
mask_roi = gpu_gaussian_blur(mask_roi, (15, 15), 5)
# Place the mask ROI in the full-sized mask
mask[min_y:max_y, min_x:max_x] = mask_roi
@@ -178,8 +143,9 @@ def create_lower_mouth_mask(
# Return the expanded lower lip polygon in original frame coordinates
lower_lip_polygon = expanded_landmarks
mouth_box = (min_x, min_y, max_x, max_y)
return mask, mouth_cutout, (min_x, min_y, max_x, max_y), lower_lip_polygon
return mask, mouth_cutout, mouth_box, lower_lip_polygon
def create_eyes_mask(face: Face, frame: Frame) -> (np.ndarray, np.ndarray, tuple, np.ndarray):
mask = np.zeros(frame.shape[:2], dtype=np.uint8)
@@ -235,8 +201,8 @@ def create_eyes_mask(face: Face, frame: Frame) -> (np.ndarray, np.ndarray, tuple
cv2.ellipse(mask_roi, left_center, left_axes, 0, 0, 360, 255, -1)
cv2.ellipse(mask_roi, right_center, right_axes, 0, 0, 360, 255, -1)
# Apply Gaussian blur to soften mask edges
mask_roi = cv2.GaussianBlur(mask_roi, (15, 15), 5)
# Apply Gaussian blur to soften mask edges (GPU-accelerated when available)
mask_roi = gpu_gaussian_blur(mask_roi, (15, 15), 5)
# Place the mask ROI in the full-sized mask
mask[min_y:max_y, min_x:max_x] = mask_roi
@@ -417,15 +383,15 @@ def create_eyebrows_mask(face: Face, frame: Frame) -> (np.ndarray, np.ndarray, t
left_shape = create_curved_eyebrow(left_local)
right_shape = create_curved_eyebrow(right_local)
# Apply multi-stage blurring for natural feathering
# Apply multi-stage blurring for natural feathering (GPU-accelerated when available)
# First, strong Gaussian blur for initial softening
mask_roi = cv2.GaussianBlur(mask_roi, (21, 21), 7)
mask_roi = gpu_gaussian_blur(mask_roi, (21, 21), 7)
# Second, medium blur for transition areas
mask_roi = cv2.GaussianBlur(mask_roi, (11, 11), 3)
mask_roi = gpu_gaussian_blur(mask_roi, (11, 11), 3)
# Finally, light blur for fine details
mask_roi = cv2.GaussianBlur(mask_roi, (5, 5), 1)
mask_roi = gpu_gaussian_blur(mask_roi, (5, 5), 1)
# Normalize mask values
mask_roi = cv2.normalize(mask_roi, None, 0, 255, cv2.NORM_MINMAX)
@@ -448,7 +414,7 @@ def create_eyebrows_mask(face: Face, frame: Frame) -> (np.ndarray, np.ndarray, t
right_local = right_eyebrow - [min_x, min_y]
cv2.fillPoly(mask_roi, [left_local.astype(np.int32)], 255)
cv2.fillPoly(mask_roi, [right_local.astype(np.int32)], 255)
mask_roi = cv2.GaussianBlur(mask_roi, (21, 21), 7)
mask_roi = gpu_gaussian_blur(mask_roi, (21, 21), 7)
mask[min_y:max_y, min_x:max_x] = mask_roi
eyebrows_cutout = frame[min_y:max_y, min_x:max_x].copy()
eyebrows_polygon = np.vstack([left_eyebrow, right_eyebrow]).astype(np.int32)
@@ -476,11 +442,11 @@ def apply_mask_area(
return frame
try:
resized_cutout = cv2.resize(cutout, (box_width, box_height))
resized_cutout = gpu_resize(cutout, (box_width, box_height))
roi = frame[min_y:max_y, min_x:max_x]
if roi.shape != resized_cutout.shape:
resized_cutout = cv2.resize(
resized_cutout = gpu_resize(
resized_cutout, (roi.shape[1], roi.shape[0])
)
@@ -500,8 +466,8 @@ def apply_mask_area(
adjusted_polygon = polygon - [min_x, min_y]
cv2.fillPoly(polygon_mask, [adjusted_polygon], 255)
# Apply strong initial feathering
polygon_mask = cv2.GaussianBlur(polygon_mask, (21, 21), 7)
# Apply strong initial feathering (GPU-accelerated when available)
polygon_mask = gpu_gaussian_blur(polygon_mask, (21, 21), 7)
# Apply additional feathering
feather_amount = min(
@@ -510,26 +476,28 @@ def apply_mask_area(
box_height // modules.globals.mask_feather_ratio,
)
feathered_mask = cv2.GaussianBlur(
polygon_mask.astype(float), (0, 0), feather_amount
polygon_mask.astype(np.float32), (0, 0), feather_amount
)
feathered_mask = feathered_mask / feathered_mask.max()
max_val = feathered_mask.max()
if max_val > 1e-6:
feathered_mask *= np.float32(1.0 / max_val)
# Apply additional smoothing to the mask edges
feathered_mask = cv2.GaussianBlur(feathered_mask, (5, 5), 1)
face_mask_roi = face_mask[min_y:max_y, min_x:max_x]
combined_mask = feathered_mask * (face_mask_roi / 255.0)
combined_mask = feathered_mask * (face_mask_roi.astype(np.float32) * np.float32(1.0 / 255.0))
combined_mask = combined_mask[:, :, np.newaxis]
combined_mask_3ch = combined_mask[:, :, np.newaxis]
inv_mask = np.float32(1.0) - combined_mask_3ch
blended = (
color_corrected_area * combined_mask + roi * (1 - combined_mask)
color_corrected_area * combined_mask_3ch + roi * inv_mask
).astype(np.uint8)
# Apply face mask to blended result
face_mask_3channel = (
np.repeat(face_mask_roi[:, :, np.newaxis], 3, axis=2) / 255.0
)
final_blend = blended * face_mask_3channel + roi * (1 - face_mask_3channel)
face_mask_f32 = face_mask_roi[:, :, np.newaxis].astype(np.float32) * np.float32(1.0 / 255.0)
face_mask_3channel = np.broadcast_to(face_mask_f32, blended.shape)
final_blend = blended * face_mask_3channel + roi * (np.float32(1.0) - face_mask_3channel)
frame[min_y:max_y, min_x:max_x] = final_blend.astype(np.uint8)
except Exception as e:
File diff suppressed because it is too large Load Diff
+2 -2
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
# Import the tkinter fix to patch the ScreenChanged error
import tkinter_fix
# Import the tkinter fix to patch the ScreenChanged error (module patches Tk on import)
import tkinter_fix # noqa: F401
import core
+1413 -1148
View File
File diff suppressed because it is too large Load Diff
+74
View File
@@ -0,0 +1,74 @@
"""Lightweight hover tooltip for CustomTkinter widgets."""
import customtkinter as ctk
class ToolTip:
"""Show a floating tooltip popup when the user hovers over a widget.
Usage:
ToolTip(my_button, "Helpful description text")
"""
def __init__(self, widget: ctk.CTkBaseClass, text: str, delay: int = 500):
self._widget = widget
self._text = text
self._delay = delay
self._tooltip_window = None
self._after_id = None
widget.bind("<Enter>", self._schedule_show, add="+")
widget.bind("<Leave>", self._hide, add="+")
def _schedule_show(self, event=None):
self._cancel()
self._after_id = self._widget.after(self._delay, self._show)
def _show(self):
if self._tooltip_window is not None:
return
x = self._widget.winfo_rootx() + 20
y = self._widget.winfo_rooty() + self._widget.winfo_height() + 5
self._tooltip_window = tw = ctk.CTkToplevel(self._widget)
tw.withdraw()
tw.overrideredirect(True)
label = ctk.CTkLabel(
tw,
text=self._text,
fg_color="#333333",
text_color="#EEEEEE",
corner_radius=6,
padx=8,
pady=4,
)
label.pack()
tw.update_idletasks()
# Clamp to screen bounds
screen_w = tw.winfo_screenwidth()
screen_h = tw.winfo_screenheight()
tip_w = tw.winfo_reqwidth()
tip_h = tw.winfo_reqheight()
if x + tip_w > screen_w:
x = screen_w - tip_w - 5
if y + tip_h > screen_h:
y = self._widget.winfo_rooty() - tip_h - 5
tw.geometry(f"+{x}+{y}")
tw.deiconify()
def _hide(self, event=None):
self._cancel()
if self._tooltip_window is not None:
self._tooltip_window.destroy()
self._tooltip_window = None
def _cancel(self):
if self._after_id is not None:
self._widget.after_cancel(self._after_id)
self._after_id = None
+178 -34
View File
@@ -15,26 +15,27 @@ import modules.globals
TEMP_FILE = "temp.mp4"
TEMP_DIRECTORY = "temp"
# monkey patch ssl for mac
if platform.system().lower() == "darwin":
ssl._create_default_https_context = ssl._create_unverified_context
def run_ffmpeg(args: List[str]) -> bool:
"""Run ffmpeg with hardware acceleration and optimized settings."""
commands = [
"ffmpeg",
"-hide_banner",
"-hwaccel",
"auto",
"-loglevel",
modules.globals.log_level,
"-hwaccel", "auto", # Auto-detect hardware acceleration
"-hwaccel_output_format", "auto", # Use hardware format when possible
"-threads", str(modules.globals.execution_threads or 0), # 0 = auto-detect optimal thread count
"-loglevel", modules.globals.log_level,
]
commands.extend(args)
try:
subprocess.check_output(commands, stderr=subprocess.STDOUT)
return True
except Exception:
pass
except subprocess.CalledProcessError as error:
output = error.output.decode(errors="ignore").strip()
if output:
print(output)
except Exception as error:
print(f"ffmpeg execution failed: {error}")
return False
@@ -61,39 +62,132 @@ def detect_fps(target_path: str) -> float:
def extract_frames(target_path: str) -> None:
"""Extract frames with hardware acceleration and optimized settings."""
temp_directory_path = get_temp_directory_path(target_path)
# Write a contiguous image sequence so the later "%04d.png" input pattern
# used during encoding can consume every frame reliably.
run_ffmpeg(
[
"-i",
target_path,
"-pix_fmt",
"rgb24",
"-i", target_path,
"-vf", "format=rgb24", # Use video filter for format conversion (faster)
"-vsync", "0", # Prevent frame duplication
os.path.join(temp_directory_path, "%04d.png"),
]
)
def create_video(target_path: str, fps: float = 30.0) -> None:
def create_video(target_path: str, fps: float = 30.0) -> bool:
"""Create video with hardware-accelerated encoding and optimized settings."""
temp_output_path = get_temp_output_path(target_path)
temp_directory_path = get_temp_directory_path(target_path)
run_ffmpeg(
[
"-r",
str(fps),
"-i",
os.path.join(temp_directory_path, "%04d.png"),
"-c:v",
modules.globals.video_encoder,
"-crf",
str(modules.globals.video_quality),
"-pix_fmt",
"yuv420p",
"-vf",
"colorspace=bt709:iall=bt601-6-625:fast=1",
# Determine optimal encoder based on available hardware
encoder = modules.globals.video_encoder
encoder_options = []
# GPU-accelerated encoding options
if 'CUDAExecutionProvider' in modules.globals.execution_providers:
# NVIDIA GPU encoding
if encoder == 'libx264':
encoder = 'h264_nvenc'
encoder_options = [
"-preset", "p7", # Highest quality preset for NVENC
"-tune", "hq", # High quality tuning
"-rc", "vbr", # Variable bitrate
"-cq", str(modules.globals.video_quality), # Quality level
"-b:v", "0", # Let CQ control bitrate
"-multipass", "fullres", # Two-pass encoding for better quality
]
elif encoder == 'libx265':
encoder = 'hevc_nvenc'
encoder_options = [
"-preset", "p7",
"-tune", "hq",
"-rc", "vbr",
"-cq", str(modules.globals.video_quality),
"-b:v", "0",
]
elif 'DmlExecutionProvider' in modules.globals.execution_providers:
# AMD/Intel GPU encoding (DirectML on Windows)
if encoder == 'libx264':
# Try AMD AMF encoder
encoder = 'h264_amf'
encoder_options = [
"-quality", "quality", # Quality mode
"-rc", "vbr_latency",
"-qp_i", str(modules.globals.video_quality),
"-qp_p", str(modules.globals.video_quality),
]
elif encoder == 'libx265':
encoder = 'hevc_amf'
encoder_options = [
"-quality", "quality",
"-rc", "vbr_latency",
"-qp_i", str(modules.globals.video_quality),
"-qp_p", str(modules.globals.video_quality),
]
else:
# CPU encoding with optimized settings
if encoder == 'libx264':
encoder_options = [
"-preset", "medium", # Balance speed/quality
"-crf", str(modules.globals.video_quality),
"-tune", "film", # Optimize for film content
]
elif encoder == 'libx265':
encoder_options = [
"-preset", "medium",
"-crf", str(modules.globals.video_quality),
"-x265-params", "log-level=error",
]
elif encoder == 'libvpx-vp9':
encoder_options = [
"-crf", str(modules.globals.video_quality),
"-b:v", "0", # Constant quality mode
"-cpu-used", "2", # Speed vs quality (0-5, lower=slower/better)
]
# Build ffmpeg command
ffmpeg_args = [
"-r", str(fps),
"-i", os.path.join(temp_directory_path, "%04d.png"),
"-c:v", encoder,
]
# Add encoder-specific options
ffmpeg_args.extend(encoder_options)
# Add common options
ffmpeg_args.extend([
"-pix_fmt", "yuv420p",
"-movflags", "+faststart", # Enable fast start for web playback
"-vf", "colorspace=bt709:iall=bt601-6-625:fast=1",
"-y",
temp_output_path,
])
# Try with hardware encoder first, fallback to software if it fails
success = run_ffmpeg(ffmpeg_args)
if not success and encoder in ['h264_nvenc', 'hevc_nvenc', 'h264_amf', 'hevc_amf']:
# Fallback to software encoding
print(f"Hardware encoding with {encoder} failed, falling back to software encoding...")
fallback_encoder = 'libx264' if 'h264' in encoder else 'libx265'
ffmpeg_args_fallback = [
"-r", str(fps),
"-i", os.path.join(temp_directory_path, "%04d.png"),
"-c:v", fallback_encoder,
"-preset", "medium",
"-crf", str(modules.globals.video_quality),
"-pix_fmt", "yuv420p",
"-movflags", "+faststart",
"-vf", "colorspace=bt709:iall=bt601-6-625:fast=1",
"-y",
temp_output_path,
]
)
success = run_ffmpeg(ffmpeg_args_fallback)
return success and os.path.isfile(temp_output_path)
def restore_audio(target_path: str, output_path: str) -> None:
@@ -168,11 +262,16 @@ def clean_temp(target_path: str) -> None:
def has_image_extension(image_path: str) -> bool:
return image_path.lower().endswith(("png", "jpg", "jpeg"))
# splitext so only the real extension counts (e.g. "photo.png.bak" is not
# an image); the set is centralized in globals to stay in sync with dialogs.
return os.path.splitext(image_path)[1].lower() in modules.globals.IMAGE_EXTENSIONS
def is_image(image_path: str) -> bool:
if image_path and os.path.isfile(image_path):
# Extension check first — Windows mimetypes doesn't always register webp
if has_image_extension(image_path):
return True
mimetype, _ = mimetypes.guess_type(image_path)
return bool(mimetype and mimetype.startswith("image/"))
return False
@@ -193,8 +292,15 @@ def conditional_download(download_directory_path: str, urls: List[str]) -> None:
download_directory_path, os.path.basename(url)
)
if not os.path.exists(download_file_path):
request = urllib.request.urlopen(url) # type: ignore[attr-defined]
total = int(request.headers.get("Content-Length", 0))
request = urllib.request.Request(url)
# Create a specific SSL context for macOS to avoid globally disabling verification
ctx = None
if platform.system().lower() == "darwin":
ctx = ssl._create_unverified_context()
response = urllib.request.urlopen(request, context=ctx)
total = int(response.headers.get("Content-Length", 0))
with tqdm(
total=total,
desc="Downloading",
@@ -202,8 +308,46 @@ def conditional_download(download_directory_path: str, urls: List[str]) -> None:
unit_scale=True,
unit_divisor=1024,
) as progress:
urllib.request.urlretrieve(url, download_file_path, reporthook=lambda count, block_size, total_size: progress.update(block_size)) # type: ignore[attr-defined]
with open(download_file_path, "wb") as f:
while True:
buffer = response.read(8192)
if not buffer:
break
f.write(buffer)
progress.update(len(buffer))
def resolve_relative_path(path: str) -> str:
return os.path.abspath(os.path.join(os.path.dirname(__file__), path))
def get_video_dimensions(target_path: str) -> tuple:
"""Get video width and height using ffprobe."""
command = [
"ffprobe", "-v", "error",
"-select_streams", "v:0",
"-show_entries", "stream=width,height",
"-of", "csv=p=0:s=x",
target_path,
]
output = subprocess.check_output(command).decode().strip()
width, height = map(int, output.split("x"))
return width, height
def estimate_frame_count(target_path: str, fps: float = None) -> int:
"""Estimate total frame count from video duration and fps."""
if fps is None:
fps = detect_fps(target_path)
command = [
"ffprobe", "-v", "error",
"-show_entries", "format=duration",
"-of", "csv=p=0",
target_path,
]
try:
output = subprocess.check_output(command).decode().strip()
duration = float(output)
return int(duration * fps)
except Exception:
return 0
+77 -11
View File
@@ -1,5 +1,6 @@
import cv2
import numpy as np
import time
from typing import Optional, Tuple, Callable
import platform
import threading
@@ -17,6 +18,10 @@ class VideoCapturer:
self._frame_ready = threading.Event()
self.is_running = False
self.cap = None
# Actual values reported by the camera after configuration
self.actual_width: int = 0
self.actual_height: int = 0
self.actual_fps: float = 0.0
# Initialize Windows-specific components if on Windows
if platform.system() == "Windows":
@@ -32,33 +37,71 @@ class VideoCapturer:
"""Initialize and start video capture"""
try:
if platform.system() == "Windows":
# Windows-specific capture methods
# device_index comes from pygrabber.FilterGraph (DirectShow
# enumeration), so open with DSHOW first to preserve mapping.
# MSMF and DirectShow enumerate cameras in different orders, so
# opening MSMF with a DSHOW index silently selects the wrong
# camera. MSMF/ANY remain as fallbacks for cameras DSHOW can't
# open.
#
# Pass codec + resolution + fps as construction params (OpenCV
# 4.6+). DSHOW locks the pixel format at open time and ignores
# later cap.set(CAP_PROP_FOURCC, ...) — without this, DSHOW
# falls back to uncompressed YUYV at 1080p, which is USB-
# bandwidth-limited to ~5 fps. Setting MJPG at construction
# negotiates compressed frames from the first read.
mjpg = cv2.VideoWriter_fourcc(*'MJPG')
open_params = [
cv2.CAP_PROP_FOURCC, mjpg,
cv2.CAP_PROP_FRAME_WIDTH, width,
cv2.CAP_PROP_FRAME_HEIGHT, height,
cv2.CAP_PROP_FPS, fps,
]
capture_methods = [
(self.device_index, cv2.CAP_DSHOW), # Try DirectShow first
(self.device_index, cv2.CAP_ANY), # Then try default backend
(-1, cv2.CAP_ANY), # Try -1 as fallback
(0, cv2.CAP_ANY), # Finally try 0 without specific backend
(self.device_index, cv2.CAP_DSHOW),
(self.device_index, cv2.CAP_MSMF),
(self.device_index, cv2.CAP_ANY),
]
for dev_id, backend in capture_methods:
try:
self.cap = cv2.VideoCapture(dev_id, backend)
self.cap = cv2.VideoCapture(dev_id, backend, open_params)
if self.cap.isOpened():
break
self.cap.release()
except Exception:
continue
elif platform.system() == "Linux":
self.cap = cv2.VideoCapture(f"/dev/video{self.device_index}")
else:
# Unix-like systems (Linux/Mac) capture method
self.cap = cv2.VideoCapture(self.device_index)
if not self.cap or not self.cap.isOpened():
raise RuntimeError("Failed to open camera")
# Configure format
self.cap.set(cv2.CAP_PROP_FRAME_WIDTH, width)
self.cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height)
self.cap.set(cv2.CAP_PROP_FPS, fps)
# Belt-and-braces: also set via cap.set() for backends that honor
# post-open changes (MSMF, V4L2). DSHOW ignores these, but the
# construction params above already handled it.
if platform.system() != "Windows":
self.cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*'MJPG'))
self.cap.set(cv2.CAP_PROP_FRAME_WIDTH, width)
self.cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height)
self.cap.set(cv2.CAP_PROP_FPS, fps)
# Read back resolution (usually reliable)
self.actual_width = int(self.cap.get(cv2.CAP_PROP_FRAME_WIDTH))
self.actual_height = int(self.cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
# CAP_PROP_FPS is unreliable on DirectShow — often reports 30
# even when the camera delivers 60. Measure empirically by
# timing a burst of frames.
reported_fps = self.cap.get(cv2.CAP_PROP_FPS)
self.actual_fps = self._measure_fps(warmup=10, sample=30,
fallback=reported_fps or fps)
print(f"[VideoCapturer] {self.actual_width}x{self.actual_height} "
f"@ {self.actual_fps:.1f}fps (reported={reported_fps:.0f})",
flush=True)
self.is_running = True
return True
@@ -89,6 +132,29 @@ class VideoCapturer:
self.is_running = False
self.cap = None
def _measure_fps(self, warmup: int = 10, sample: int = 30,
fallback: float = 30.0) -> float:
"""Read warmup+sample frames and return measured FPS.
This is more reliable than CAP_PROP_FPS which often lies on
DirectShow. Takes ~0.5-1s at startup but gives a ground-truth
number for adaptive polling/detection intervals.
"""
try:
for _ in range(warmup):
self.cap.read()
t0 = time.perf_counter()
for _ in range(sample):
ret, _ = self.cap.read()
if not ret:
return fallback
elapsed = time.perf_counter() - t0
if elapsed <= 0:
return fallback
return sample / elapsed
except Exception:
return fallback
def set_frame_callback(self, callback: Callable[[np.ndarray], None]) -> None:
"""Set callback for frame processing"""
self.frame_callback = callback
+9
View File
@@ -0,0 +1,9 @@
[tool.ruff]
target-version = "py310"
[tool.ruff.lint]
# Deterministic, low-risk rules enforced in CI. Other rules (F841, E402, F821)
# surface real findings but require human judgement to fix safely, so they are
# left out of the gate for now. Intentional side-effect imports should be
# annotated with `# noqa: F401`.
select = ["E701", "E711", "E712", "F401", "F541"]
+17 -23
View File
@@ -1,24 +1,18 @@
--extra-index-url https://download.pytorch.org/whl/cu128
numpy>=1.23.5,<2
typing-extensions>=4.8.0
opencv-python==4.10.0.84
cv2_enumerate_cameras==1.1.15
onnx==1.18.0
numpy>=2.0,<3
typing-extensions>=4.15.0
opencv-python==4.14.0.94
opencv-python-headless==4.14.0.94
cv2_enumerate_cameras==1.3.3
onnx==1.22.0
insightface==0.7.3
psutil==5.9.8
tk==0.1.0
customtkinter==5.2.2
pillow==11.1.0
torch; sys_platform != 'darwin'
torch==2.8.0+cu128; sys_platform == 'darwin'
torchvision; sys_platform != 'darwin'
torchvision==0.20.1; sys_platform == 'darwin'
onnxruntime-silicon==1.16.3; sys_platform == 'darwin' and platform_machine == 'arm64'
onnxruntime-gpu==1.22.0; sys_platform != 'darwin'
tensorflow; sys_platform != 'darwin'
opennsfw2==0.10.2
protobuf==4.25.1
git+https://github.com/xinntao/BasicSR.git@master
git+https://github.com/TencentARC/GFPGAN.git@master
pygrabber
psutil==7.2.2
PySide6>=6.7,<7
pillow==12.3.0
tqdm>=4.66.3
onnxruntime==1.28.0; sys_platform == 'darwin' and platform_machine == 'arm64'
onnxruntime==1.23.0; sys_platform == 'darwin' and platform_machine != 'arm64'
onnxruntime-gpu==1.26.0; sys_platform != 'darwin'
opennsfw2==0.18.0
keras>=3.0.0
protobuf>=6.33.5,<8
pygrabber; sys_platform == 'win32'
+92 -2
View File
@@ -1,7 +1,97 @@
#!/usr/bin/env python3
# Import the tkinter fix to patch the ScreenChanged error
import tkinter_fix
import os
import sys
# Add the project root to PATH so bundled ffmpeg/ffprobe are found
project_root = os.path.dirname(os.path.abspath(__file__))
os.environ["PATH"] = project_root + os.pathsep + os.environ.get("PATH", "")
# On Windows, register NVIDIA CUDA DLL directories so onnxruntime-gpu can
# find cuDNN/cublas. Python 3.8+ ignores PATH for extension-module native deps —
# os.add_dll_directory() is required. Also keep PATH for child processes/ffmpeg.
if sys.platform == "win32":
_site_packages = os.path.join(sys.prefix, "Lib", "site-packages")
_venv_site_packages = os.path.join(project_root, "venv", "Lib", "site-packages")
for _sp in (_site_packages, _venv_site_packages):
_candidate_dirs = []
_torch_lib = os.path.join(_sp, "torch", "lib")
if os.path.isdir(_torch_lib):
_candidate_dirs.append(_torch_lib)
_nvidia_dir = os.path.join(_sp, "nvidia")
if os.path.isdir(_nvidia_dir):
for _pkg in os.listdir(_nvidia_dir):
_bin_dir = os.path.join(_nvidia_dir, _pkg, "bin")
if os.path.isdir(_bin_dir):
_candidate_dirs.append(_bin_dir)
for _d in _candidate_dirs:
os.environ["PATH"] = _d + os.pathsep + os.environ["PATH"]
try:
os.add_dll_directory(_d)
except (OSError, AttributeError):
pass
# On Windows, register OpenVINO DLL directories so onnxruntime's
# OpenVINOExecutionProvider can find openvino.dll. This must happen
# before any ONNX InferenceSession is created. Failure is non-fatal:
# OpenVINO simply isn't installed, and onnxruntime will fall back to CPU.
try:
from onnxruntime.tools.add_openvino_win_libs import ( # type: ignore[import-untyped] # noqa: E501
add_openvino_libs_to_path,
)
add_openvino_libs_to_path()
except ImportError:
# onnxruntime build without the OpenVINO tooling module — no-op.
pass
except FileNotFoundError:
# OpenVINO site-packages dir absent — no-op.
pass
except SystemExit as exc:
# add_openvino_libs_to_path() calls sys.exit() when OpenVINO libs
# can't be located (e.g. OPENVINO_LIB_PATHS unset). Log the message
# it raised with so the failure is visible, but keep startup alive.
print(
f"[startup] OpenVINO DLL registration skipped: {exc}",
flush=True,
)
# On Linux, pre-load NVIDIA shared libraries (cuDNN, cuBLAS, nvrtc...) shipped
# inside the venv via pip wheels (nvidia-cudnn-cu12, etc.). LD_LIBRARY_PATH
# cannot be set after Python starts, so we use ctypes.CDLL with RTLD_GLOBAL
# instead. This makes symbols available to onnxruntime when it dlopens its
# CUDA provider.
if sys.platform.startswith("linux"):
import ctypes
import glob
_py_lib = f"python{sys.version_info.major}.{sys.version_info.minor}"
_site_packages_candidates = [
os.path.join(project_root, "venv", "lib", _py_lib, "site-packages"),
os.path.join(sys.prefix, "lib", _py_lib, "site-packages"),
]
for _sp in _site_packages_candidates:
_nvidia_dir = os.path.join(_sp, "nvidia")
if not os.path.isdir(_nvidia_dir):
continue
for _pkg in os.listdir(_nvidia_dir):
_lib_dir = os.path.join(_nvidia_dir, _pkg, "lib")
if not os.path.isdir(_lib_dir):
continue
# Also expose the directory to child processes, without
# duplicating an entry that is already present.
_ldp = os.environ.get("LD_LIBRARY_PATH", "")
if _lib_dir not in _ldp.split(os.pathsep):
os.environ["LD_LIBRARY_PATH"] = (
_lib_dir + (os.pathsep + _ldp if _ldp else "")
)
for _so in sorted(glob.glob(os.path.join(_lib_dir, "lib*.so*"))):
try:
ctypes.CDLL(_so, mode=ctypes.RTLD_GLOBAL)
except OSError:
pass
break
from modules import platform_info
platform_info.print_banner()
from modules import core
+137
View File
@@ -0,0 +1,137 @@
import importlib
import sys
import types
import unittest
from contextlib import contextmanager
from unittest.mock import patch
@contextmanager
def _patched_core_import_stubs(calls, pipe_result=False):
class Processor:
NAME = "test_processor"
def pre_start(self):
return True
def pre_check(self):
return True
def process_image(self, *_args, **_kwargs):
raise AssertionError("image path should not be used")
def process_video(self, source_path, frame_paths):
calls.append(("process_video", source_path, tuple(frame_paths)))
stubs = {
"cv2": types.SimpleNamespace(
IMREAD_COLOR=1,
imdecode=lambda *_args, **_kwargs: None,
imencode=lambda *_args, **_kwargs: (
True,
types.SimpleNamespace(tofile=lambda *_a, **_k: None),
),
),
"numpy": types.SimpleNamespace(uint8=object, fromfile=lambda *_args, **_kwargs: b""),
"torch": types.SimpleNamespace(
cuda=types.SimpleNamespace(empty_cache=lambda: None)
),
"onnxruntime": types.SimpleNamespace(
get_available_providers=lambda: ["CPUExecutionProvider"]
),
"tensorflow": types.SimpleNamespace(),
"modules.metadata": types.SimpleNamespace(name="Deep-Live-Cam", version="test"),
"modules.ui": types.SimpleNamespace(
check_and_ignore_nsfw=lambda *_args, **_kwargs: False,
update_status=lambda *_args, **_kwargs: None,
init=lambda *_args, **_kwargs: types.SimpleNamespace(mainloop=lambda: None),
),
"modules.processors.frame.core": types.SimpleNamespace(
get_frame_processors_modules=lambda _names: [Processor()],
process_video_in_memory=lambda *_args, **_kwargs: calls.append(("pipe",))
or pipe_result,
),
"modules.utilities": types.SimpleNamespace(
has_image_extension=lambda _path: False,
is_image=lambda _path: False,
is_video=lambda _path: True,
detect_fps=lambda _path: 24.0,
create_video=lambda target_path, fps: calls.append(
("create_video", target_path, fps)
)
or True,
extract_frames=lambda target_path: calls.append(
("extract_frames", target_path)
),
get_temp_frame_paths=lambda target_path: [f"{target_path}/0001.png"],
restore_audio=lambda *_args, **_kwargs: calls.append(("restore_audio",)),
create_temp=lambda target_path: calls.append(("create_temp", target_path)),
move_temp=lambda target_path, output_path: calls.append(
("move_temp", target_path, output_path)
),
clean_temp=lambda target_path: calls.append(("clean_temp", target_path)),
normalize_output_path=lambda _source, _target, output: output,
),
}
with patch.dict(sys.modules, stubs, clear=False):
sys.modules.pop("modules.core", None)
yield importlib.import_module("modules.core")
sys.modules.pop("modules.core", None)
def _configure_video_run(core, *, map_faces):
core.modules.globals.source_path = "source.jpg"
core.modules.globals.target_path = "target.mp4"
core.modules.globals.output_path = "output.mp4"
core.modules.globals.frame_processors = ["face_swapper"]
core.modules.globals.headless = True
core.modules.globals.keep_fps = False
core.modules.globals.keep_audio = False
core.modules.globals.keep_frames = False
core.modules.globals.map_faces = map_faces
core.modules.globals.nsfw_filter = False
core.modules.globals.execution_threads = 1
core.modules.globals.execution_providers = ["CPUExecutionProvider"]
core.modules.globals.max_memory = None
class MapFacesFallbackTests(unittest.TestCase):
def test_map_faces_disk_fallback_extracts_frames_before_processing(self):
calls = []
with _patched_core_import_stubs(calls, pipe_result=False) as core:
_configure_video_run(core, map_faces=True)
with patch.object(core.os.path, "isfile", return_value=True):
core.start()
self.assertNotIn(("pipe",), calls)
self.assertIn(("create_temp", "target.mp4"), calls)
self.assertIn(("extract_frames", "target.mp4"), calls)
self.assertIn(("process_video", "source.jpg", ("target.mp4/0001.png",)), calls)
self.assertIn(("create_video", "target.mp4", 30.0), calls)
self.assertIn(("move_temp", "target.mp4", "output.mp4"), calls)
step_indices = {}
for index, call in enumerate(calls):
step_indices.setdefault(call[0], index)
self.assertLess(step_indices["create_temp"], step_indices["extract_frames"])
self.assertLess(step_indices["extract_frames"], step_indices["process_video"])
self.assertLess(step_indices["process_video"], step_indices["create_video"])
self.assertLess(step_indices["create_video"], step_indices["move_temp"])
def test_non_map_faces_pipe_success_does_not_extract_frames(self):
calls = []
with _patched_core_import_stubs(calls, pipe_result=True) as core:
_configure_video_run(core, map_faces=False)
with patch.object(core.os.path, "isfile", return_value=True):
core.start()
self.assertIn(("pipe",), calls)
self.assertNotIn(("extract_frames", "target.mp4"), calls)
self.assertNotIn(("process_video", "source.jpg", ("target.mp4/0001.png",)), calls)
if __name__ == "__main__":
unittest.main()
+97
View File
@@ -0,0 +1,97 @@
import importlib
import sys
import types
import unittest
from unittest.mock import patch
def _install_import_stubs():
sys.modules.setdefault(
"insightface",
types.SimpleNamespace(app=types.SimpleNamespace(FaceAnalysis=object)),
)
sys.modules.setdefault(
"cv2",
types.SimpleNamespace(
IMREAD_COLOR=1,
imread=lambda *_args, **_kwargs: None,
imdecode=lambda *_args, **_kwargs: None,
imencode=lambda *_args, **_kwargs: (
True,
types.SimpleNamespace(tofile=lambda *_a, **_k: None),
),
),
)
sys.modules.setdefault(
"numpy",
types.SimpleNamespace(uint8=object, fromfile=lambda *_args, **_kwargs: b""),
)
sys.modules.setdefault(
"tqdm",
types.SimpleNamespace(tqdm=lambda iterable, **_kwargs: iterable),
)
sys.modules["modules.typing"] = types.SimpleNamespace(Frame=object)
sys.modules["modules.cluster_analysis"] = types.SimpleNamespace(
find_cluster_centroids=lambda *args, **kwargs: [],
find_closest_centroid=lambda *args, **kwargs: (0, None),
)
sys.modules["modules.utilities"] = types.SimpleNamespace(
get_temp_directory_path=lambda path: path,
create_temp=lambda path: None,
extract_frames=lambda path: None,
clean_temp=lambda path: None,
get_temp_frame_paths=lambda path: [],
)
def _load_face_analyser():
_install_import_stubs()
sys.modules.pop("modules.face_analyser", None)
return importlib.import_module("modules.face_analyser")
class Face:
def __init__(self, left):
self.bbox = [left, 0, 10, 10]
class GetOneFaceTests(unittest.TestCase):
def test_uses_supplied_detected_faces_without_reanalysing_frame(self):
face_analyser = _load_face_analyser()
right = Face(20)
left = Face(5)
with patch.object(
face_analyser,
"_analyse_faces",
side_effect=AssertionError("should not analyse"),
):
self.assertIs(face_analyser.get_one_face("frame", [right, left]), left)
def test_supplied_empty_detected_faces_returns_none(self):
face_analyser = _load_face_analyser()
with patch.object(
face_analyser,
"_analyse_faces",
side_effect=AssertionError("should not analyse"),
):
self.assertIsNone(face_analyser.get_one_face("frame", []))
def test_without_supplied_faces_preserves_existing_detection_path(self):
face_analyser = _load_face_analyser()
right = Face(30)
left = Face(3)
with patch.object(face_analyser, "_is_dml", return_value=False), patch.object(
face_analyser,
"_analyse_faces",
return_value=[right, left],
) as analyse:
self.assertIs(face_analyser.get_one_face("frame"), left)
analyse.assert_called_once_with("frame")
if __name__ == "__main__":
unittest.main()
+3
View File
@@ -1,3 +1,6 @@
import os
os.environ.setdefault('TK_SILENCE_DEPRECATION', '1')
import tkinter
# Only needs to be imported once at the beginning of the application