Compare commits

..
Author SHA1 Message Date
henryruhs b6a1adb7a5 add groups to capabilities 2026-07-02 20:39:09 +02:00
henryruhs 4895615334 keep v4 style 2026-07-01 17:21:33 +02:00
henryruhs df2bda6145 merge master into v4 - post adjustments 2026-07-01 17:10:31 +02:00
henryruhs 942d435245 merge master into v4 - post adjustments 2026-07-01 16:02:19 +02:00
henryruhs 5d0b2afbe9 merge master into v4 - post adjustments 2026-07-01 14:44:53 +02:00
henryruhs e0921ba4b5 merge master into v4 - post adjustments 2026-07-01 14:29:55 +02:00
henryruhs 0b976f80a8 merge master into v4 - post adjustments 2026-07-01 14:28:46 +02:00
henryruhs bd1448805c merge master into v4 - post adjustments 2026-07-01 13:36:59 +02:00
henryruhs 8a9c596fde merge master into v4 - post adjustments 2026-07-01 13:34:54 +02:00
300470e7c7 port/master-into-v4 (#1176)
* 3.7.0 (#1175)

* mark as next, introduce dynamic scale for face debugger

* use latest onnxruntime

* update within Gradio 5

* Remove system memory limit (#986)

* remove system memory limit from ui

* remove system memory limit from args.py

* flatten the face store

* prevent countless importlib.import_module calls

* remove --onnxruntime from install.py

* remove --onnxruntime from install.py

* resolve static inference providers to fix macos (#1127)

* resolve static inference providers to fix macos

* fix lint

* restore old behaviour

* restore old behaviour

* handle ghost and uniface as well

* adjust condition for ghost and uniface

* fix Gradio gallery styles

* remove face store (#1132)

* fix dataflow in streamer

* Face selector auto mode (#1137)

* introduce face selector auto mode

* introduce face selector auto mode

* introduce face selector auto mode

* correct way is to pass source_vision_frames

* make the world a better place

* fix dataflow in faceswapper, no read of files withing inner methods (#1148)

* fix dataflow in faceswapper, no read of files withing inner methods

* fix lint

* adjust code more

* adjust code more

* bring back the face store but for source and reference only (#1149)

* bring back the face store but for source and reference only

* fix ci

* minor improvement

* guard for tobytes()

* drop condition in select_faces()

* Replace CONFIG_PARSER global with @lru_cache (#1147)

* remove global config_parser

* fix import order

* remove lambda

* remove unused block

* optimize app context detection

* decouple common modules from core (#1152)

* decouple common modules from core

* remove that nonsense

* remove that nonsense

* minor adjustment to workflows

* Tag HEVC output as hvc1 and move moov atom to the front (#1153)

* Tag HEVC output as hvc1 and move moov atom to the front

ffmpeg defaults HEVC in MP4 to the 'hev1' sample entry and leaves the moov
atom at the tail. Apple players (QuickTime, Finder QuickLook) refuse to decode
'hev1' and stall reading a tail-placed moov on large files, so hevc_nvenc /
libx265 renders cannot be previewed on macOS.

- add ffmpeg_builder.set_video_tag(): emit `-tag:v hvc1` for every HEVC
  encoder (libx265, hevc_nvenc, hevc_amf, hevc_qsv, hevc_videotoolbox).
  Applied in merge_video where the encoder is known; `-c:v copy` in the audio
  mux / concat steps preserves the tag.
- add ffmpeg_builder.set_faststart(): emit `-movflags +faststart`, applied in
  restore_audio / replace_audio / concat_video which write the final output.

H.264 and other codecs are left untouched. Verified on a real hevc_nvenc
render: hev1 hung QuickLook (no thumbnail); after the patch the file is hvc1
with a front-placed moov and QuickLook generates a thumbnail.

* Restrict hvc1 tag and faststart to quicktime containers

Gate set_video_tag / set_faststart on the output container format
(m4v, mov, mp4) via get_file_format(), so non-quicktime muxers no longer
receive -tag:v hvc1 / -movflags +faststart. Trim test_set_video_tag to a
single positive and negative assertion.

Addresses review on #1153.

* Move hvc1 tag and faststart gates into ffmpeg_builder

Rename set_video_tag / set_faststart to conditional_* and push the
container-format gate (m4v, mov, mp4) inside the builders, keeping
ffmpeg.py free of inline conditionals. Matches the set_image_quality
pattern. Addresses review on #1153.

* post cleanup after merge

* Pack target frames (#1158)

* pack target frames

* add todos

* add todos, resolve todos

* resolve todos

* change names

* revert to single target frame for select faces

* fix lint

* return empty frame

* get() have no default

* Fix trim (#1162)

* fix trim

* fix trim

* rename ffmpeg builder method

* rename to temp_frame_set and temp_frame_pattern

---------

Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>
Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>

* Implement face tracker (#1163)

* add face tracker

* change get_nearest_track_face -> get_nearest_track_index

* create face_creator.py and move methods around

* add type FaceTrack

* naming

* remove iou test, don't belong there

* fix spaces

* rename to interpolate_points

* rename to find_best_face_track

* just track_faces

* cleanp

* previous next naming

* remove >= and >=

* rename

* remove helper from test and use face from source.jpg

* make get_anchor_indices more readable

* track_faces() call before and is forwarded to select_faces

* change to interpolate_faces

* rename methods

* rename methods

* rename variables

* remove dtype

* move face_anlyser -> face_creator

* claenup face_creator.py

* move tests to dedicated test face detector

* move tracking inside select_faces

* simplify face_tracker (#1165)

* minor renaming

* improve face_tracker test (#1166)

* improve face_tracker test

* cleanup

* Add target frame amount (#1167)

* introduce --target-frame-amount

* add ui

* make track_faces conditional

* update choices.py

* fix []

* rename component file to frame_process.py

* fix track preview (#1168)

* introduce face origin (#1169)

* add guard to prevent failure

* show and hide voice extractor according to lip syncer

* rename average_face_coordinates to average_face_geometry

* use static faces for select_faces()

* face store with lock (#1171)

* face store with lock

* face store with lock

* remove refill color from bbox

* adjust tests and handle frame_position proper way

* enforce similar naming

* introduce face tracker score

* introduce face tracker score

* fix/audio-trim-alignment (#1173)

* fix audio offset

* fix audio offset

* remove reference_frame_number check

---------

Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>

* reduce face tracker score from 0 to 0.5

* mark as 3.7.0

* make face tracker stateless

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>
Co-authored-by: kazuki nakai <kazuki.nakai@agiletec.net>
Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>

* update preview

* fix wording

* fix wording

* last minute change to frame distribution

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>
Co-authored-by: kazuki nakai <kazuki.nakai@agiletec.net>
Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>
2026-07-01 11:51:54 +02:00
Henry RuhsandGitHub 5f403ada65 remove opus frame size (#1160)
* remove opus frame size

* improve naming, follow audio.py
2026-06-15 19:19:51 +02:00
henryruhs c6aed91698 use dynamic audio frame size 2026-06-15 17:14:24 +02:00
Henry RuhsandGitHub 2854bf1db0 Resolve final stream TODOs (#1155)
* refactor a lot

* refactor a lot

* fix test

* fix test
2026-06-15 11:37:25 +02:00
Henry RuhsandGitHub d5271d21a1 minor adjustment for workflow setup and clear (#1157)
* minor adjustment for workflow setup and clear

* minor adjustment for workflow setup and clear
2026-06-14 22:57:01 +02:00
Henry RuhsandGitHub e7d22e84bf use Buffer and BufferPack everywhere (#1154) 2026-06-11 13:03:29 +02:00
henryruhs 57189c638e optimize app context detection 2026-06-09 22:30:37 +02:00
Henry RuhsandGitHub 00fb89d4f1 Best performance to code ratio for stream (#1150)
* queue with futures, kill deque, add couple of todos

* resolve couple of todos

* add more todos

* add more todos and resolve others

* add more todos and resolve others

* fix test

* fix collapse

* adjust naming a bit
2026-06-09 22:16:51 +02:00
henryruhs 67cc3de934 just ignore type 2026-06-06 15:31:14 +02:00
henryruhs accad0a2cb fix ci 2026-06-06 13:53:30 +02:00
henryruhs 46d1575c9b fix ci 2026-06-06 13:41:28 +02:00
henryruhs f3bc5ffb4b changes for vp9 support 2026-06-06 12:39:14 +02:00
henryruhs 5287ce6dcb add todos 2026-06-06 11:45:38 +02:00
henryruhs 5d6258e17e add vp9 support 2026-06-06 11:39:53 +02:00
henryruhs 036c5c0225 add vp9 support 2026-06-06 11:15:00 +02:00
henryruhs 2e884941f8 resolve todos 2026-06-06 08:36:20 +02:00
henryruhs 0ef5de1d02 guard for tobytes()
(cherry picked from commit 0cbfa5a415)
2026-06-05 21:47:57 +02:00
henryruhs dcb9a7a59b minor improvement
(cherry picked from commit fd3e8b82e2)
2026-06-05 21:47:57 +02:00
henryruhs 3f6738dd1d fix ci
(cherry picked from commit fbfb21df3b)
2026-06-05 21:47:57 +02:00
henryruhs 19f8f5f206 bring back the face store but for source and reference only
(cherry picked from commit 9d8a5be714)
2026-06-05 21:47:57 +02:00
775985645e Push based receive with queue (#1146)
* move to push based receive

* move to push based receive, fix mocks

* fix tests

* add todos

* remove asyncio

* remove asyncio

* resolve todos

* move to queue without events

* prevent debug spam

* concurrent stream inference

stream_video.py: pipeline face-swap inference across execution_thread_count workers (ThreadPoolExecutor + bounded in-flight deque, ordered encode) to keep the GPU busy during encode

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* add todos

* add todos

* add missing state

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-05 00:18:57 +02:00
henryruhs 1f494f54db fix hashes 2026-06-03 10:43:16 +02:00
henryruhs 476a21cc7a fix hashes 2026-06-03 10:43:09 +02:00
henryruhs 121c3a5af6 update hashes 2026-06-03 10:36:00 +02:00
Henry RuhsandGitHub 2ac9b70550 Feat/remb both direction (#1145)
* revamp remb to take both directions

* extract more methods
2026-06-03 10:31:33 +02:00
henryruhs d9553b12e8 remove copy of buffer 2026-06-02 17:10:25 +02:00
henryruhs 3bda73699c fix fast seeking 2026-06-02 16:29:58 +02:00
Henry RuhsandGitHub 7181b41f2d Feat/finalize stream (#1144)
* break stream helper into pieces

* remove todos
2026-06-02 16:05:21 +02:00
harisreedhar 314ee61826 add create_event method 2026-06-02 03:06:16 +05:30
HarisreedharandGitHub beeb1d99e9 Refactor(stream-helper): split encode/receive loops and unify audio/video structure (#1142)
* split encode loop

* unify audio and video encode loop methods

* improve variable names

* split receive methods

* test improve

* try to avoid != with even more weird approach

* remove source_path check

* remove empty variables

* fix lint

* avoid not in condition

* rename

* fix lint
2026-06-02 01:50:40 +05:30
Henry RuhsandGitHub e2e4e6a95b Refactor/stream helper testing (#1141)
* fix tests part1

* improve testing

* use real audio

* more renaming

* switch to hash assertions

* switch to hash assertions

* switch to hash assertions

* switch to hash assertions

* skip on macos
2026-06-01 18:42:37 +02:00
3eeb505c86 Refactor(stream-helper): pass peer context objects to receivers, fix codec guard, move YUV conversion into decoders (#1140)
* extract numpy.empty(0) into an empty_vision_frame variable so the sentinel intent is clear

* bring back todos

* bring back todos

* rename opus_buffer to audio

* move reshape and cvtColor into decoder modules

* add audio_codec check

* add audio_codec check

* rename audio video buffer

* simplify receive methods and cleanup test

* todo

* revert decode methods to return pointer

* remove duplicate test_stream_helper — tests live in test_api_stream_helper

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-01 17:21:06 +05:30
harisreedhar 02f684e220 bring back todos 2026-06-01 15:48:10 +05:30
harisreedhar a88d8ead49 bring back todos 2026-06-01 15:45:44 +05:30
harisreedhar 7735c3740f bring back todos 2026-06-01 15:19:28 +05:30
henryruhs f2de9f1843 rename to video pack and audio pack 2026-06-01 11:41:25 +02:00
HarisreedharandGitHub aea36a9e55 Replace queues with timestamped deques and fix Audio Video sync (#1139)
* change to deque

* remove single line condition

* change threading to asyncio

* bringback todos
2026-06-01 15:03:33 +05:30
Henry RuhsandGitHub 0f5f75ba51 Cleanup/testing suite (#1136)
* clean testing suite

* clean testing suite part2

* clean testing suite part3

* add todos

* extend testing suite and kill some mutants

* fix hashes

* fix lint

* fix test

* fix test
2026-06-01 08:54:37 +02:00
Henry RuhsandGitHub 460c65004b Add available event (#1134)
* remove sleep with available event

* reorder methods, caller has to follow variable names of consumer, reorder tests methods

* more todos for naming
2026-05-30 16:39:48 +05:30
Henry RuhsandGitHub 1ac0e3e9a4 remove face store (#1132) (#1133) 2026-05-30 11:55:58 +02:00
HarisreedharandGitHub 9f6a19c1d2 feat(rtc): 2-way REMB with in-place encoder bitrate update (#1131)
* implement remb other direction

* update test
2026-05-30 04:41:10 +05:30
henryruhs 3ff327e670 add todos 2026-05-29 18:17:51 +02:00
henryruhs 2553ad7cad add todos 2026-05-29 18:10:15 +02:00
HarisreedharandGitHub 6b9ddd9a4f feat(rtc): REMB bitrate adaptation with in-place encoder update (#1130)
* improve test

* fix lint

* cleanup

* cleanup
2026-05-29 21:15:03 +05:30
henryruhs 871559cb6a clean and simplify tests 2026-05-29 15:44:07 +02:00
HarisreedharandGitHub 2a8672b54d Implement tier 1 REMB (#1129)
* implement tier 1

* fix lint

* cleanup

* cleanup

* fix lint

* use clear_remb method

* use single callback

* improve test

* improve test

* improve test

* improve test

* improve test
2026-05-29 19:00:27 +05:30
henryruhs cc0af9175a ban byte string 2026-05-26 19:15:11 +02:00
harisreedhar d77be89177 improve encoder collect 2026-05-26 22:25:55 +05:30
henryruhs e813d7df95 improve performance for decoder collect 2026-05-26 18:29:12 +02:00
henryruhs 6eaabe123e remove --onnxruntime from install.py 2026-05-26 08:16:49 +02:00
henryruhs e2bac200d6 remove --onnxruntime from install.py 2026-05-26 08:14:48 +02:00
HarisreedharandGitHub 4fe79483ea Fix stream lifecycle bugs: threading, RTP sync, and resource cleanup (#1125)
* fix executor thread not terminating after stream deletion

* fix stream shutdown and thread lifecycle

* add todo

* cleanup

* cleanup

* cleanup

* cleanup

* audio_queue.put() → get_nowait() + put_nowait()

* rename test

* fix test

* merge tests

* cleanup tests

* cleanup tests

* cleanup tests

* simplify test logic with mock

* cleanup

* cleanup hard to read stream_helper.py

* introduce rtc_peer.has_peers

* fix lint

* add todos

* fix test hash
2026-05-22 16:29:45 +05:30
HarisreedharandGitHub 520dcbfd6b Refine stream helper: queue-based loops, AV1/WHIP support, endpoint separation (#1124)
* rearrange methods

* add test_stream_helper.py

* improve tests

* use deque

* move decoder to recieve methods

* remove cleanup_peer

* add destroy_stream

* make run_peer_loop more readable

* make video and audio method simlar

* change deque to queue to avoid extra thread event

* remove negative condition

* cleanup

* remove wait_for_frame

* cleanup

* cleanup

* fix process_image

* fix lint

* cleanup

* remove last_time

* add todos
2026-05-20 23:43:14 +05:30
Henry RuhsandGitHub 48869bedf0 Follow WHIP specs (#1123)
* follow more specs of whip

* pass the location header value via API router

* fix CI

* remove more queries
2026-05-19 17:05:53 +02:00
Henry RuhsandGitHub 927857d70d cleanup decoders tests (#1122)
* cleanup mostly decoders tests, also a bit encoders

* cleanup mostly decoders tests, also a bit encoders
2026-05-19 12:57:57 +02:00
Henry RuhsandGitHub fbacb24fcc Tiny refactor of codecs (#1121)
* improve performance using pointers

* simplify decoder's collect

* simplify decoder's collect

* add threading to decoders

* fix test

* switch back to return bytes

* fix macos
2026-05-19 10:31:53 +02:00
c00ea92f35 Migrate to WHIP (#1120)
* migrate to whip part1

* migrate to whip part2

* migrate to whip part3

* migrate to whip part4

* migrate to whip/whep with bidirectional

* migrate to whip/whep with bidirectional

* use next library

* add _next to lid datachannel files

* cleanup and add todos

* use internal helper rtcGetPayloadTypesForCodec

* fix lint

* refactor decode()

* move logic to codecs

* move logic to codecs

* break encoders and decoders into multiple files

* break encoders and decoders into multiple files

* cleanup more

* drop action for stream endpoints, keep type for self documentation

* restore the v4 store

* fix: align frame_width and frame_height to even in both collect() and read_resolution() in both decoders.

---------

Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>
2026-05-18 16:16:06 +02:00
HarisreedharandGitHub c48c238f88 Combine encode loop methods (#1119)
* combine run_aom_encode_loop and run_vpx_encode_loop to encode_video_loop

* run_opus_encode_loop -> encode_audio_loop

* use else instead of continue

* rename to video_codec
2026-05-16 23:29:36 +05:30
Henry RuhsandGitHub dd1ded1408 Refactor/rtc cleanup 3 (#1118)
* tweak rtc store and make the decision to ban trivial testing

* clear todos for rtc_test, remove redundant tests

* clear todos for rtc_test, remove redundant tests

* break negotiation out of rtc flow, introduce create_sdp_answer and set_remote_description

* add todo

* move timeline control to the stream helper, clean send_audio|video_to_peers

* rename some methods

* fix test

* introduce detect_sdp_media

* introduce detect_sdp_media
2026-05-16 09:06:04 +02:00
Henry RuhsandGitHub 95435f842c use datachannel to create proper rtc_track_init (#1117)
* use datachannel to create proper rtc_track_init

* fix lint

* fix lint
2026-05-15 19:14:36 +02:00
HarisreedharandGitHub 0019d3ad0f Refactor stream_helper: queue-based audio/video loops with unified threading (#1116)
* rearrange methods following the flow

* add test_stream_helper.py

* fix lint

* fix lint

* refactor audio flow to match video by replacing dequeue with queue

* remove unused keyframe interval

* remove try block

* remove while True

* simplify run_aom_encode_loop and run_vp8_encode_loop

* cleanup names

* simplify run_opus_encode_loop

* move opus_encoder creation to run_opus_encode_loop

* add todos

* fix lint

* update todos and tests
2026-05-15 21:47:30 +05:30
Henry RuhsandGitHub 532464032b More RTC cleanup (#1115)
* reduce create_peer_connection like crazy

* turns out that we dont event and callback/while+sleep magic for create_sdp_offer and negotiate_sdp_answer

* test for audio/video to peers

* flag rtc methods to be revised

* fix lint
2026-05-15 15:10:51 +02:00
Henry RuhsandGitHub ad3b582c49 resolve todos in stream endpoint (#1114) 2026-05-15 12:40:50 +02:00
Henry RuhsandGitHub 98adce8a2b Refactor RTC structure (#1113)
* refactor rtc part1

* skip for macos

* merge create spd and create sdp offer

* fix lint

* add test for create_sdp_offer

* better naming for negotiate method as we get an answer

* extend tests based on mutations

* remove dead code

* rename rtc store and related methods

* clean store, move sender logic to stream helper under apis

* generate tests for rtc store
2026-05-15 11:46:51 +02:00
henryruhs 3def6c8fcd skip for macos 2026-05-15 08:47:28 +02:00
henryruhs 504f2240f7 make test more robust 2026-05-14 23:19:49 +02:00
henryruhs 061522e5b7 use bytes() over empty byte literals 2026-05-14 22:45:02 +02:00
henryruhs 37420eac7c use bytes() over empty byte literals 2026-05-14 22:44:56 +02:00
henryruhs 1562fe2fee kill the stream helper in tests 2026-05-14 22:41:54 +02:00
henryruhs dc74e1c783 event driven test_stream_video 2026-05-14 22:21:36 +02:00
henryruhs a097034889 skip test for macos 2026-05-14 19:32:14 +02:00
henryruhs d8d9d5a280 fix test for macos 2026-05-14 18:44:02 +02:00
henryruhs eeb342ce36 fix macos for aom encode 2026-05-14 17:20:18 +02:00
henryruhs b8d00d6389 get rid of aom obus stuff again 2026-05-14 16:44:09 +02:00
Henry RuhsandGitHub 18a487347a av1 support integrated (#1112) 2026-05-14 16:11:23 +02:00
Henry RuhsandGitHub b607e4a99e AV1 codec support (#1111)
* restructure xxx_encoders, introduce av1 codec

* get rid of strip_temporal_delimiters

* improve testing

* fix test for macos

* improve testing
2026-05-14 13:30:18 +02:00
Henry RuhsandGitHub b1bc0ea43c aom library for av1 support (#1110) 2026-05-14 11:41:28 +02:00
henryruhs 912d7eaa52 adjust create_string_buffer that could be measured 2026-05-13 15:31:54 +02:00
Henry RuhsandGitHub 832d954df6 use bytes over pointer for opus encoder (#1109)
* use bytes over pointer for opus encoder

* use bytes over pointer for opus encoder
2026-05-13 14:41:26 +02:00
henryruhs 78a068107a clean audio encoder too 2026-05-13 13:41:09 +02:00
Henry RuhsandGitHub 9e1c068938 improve naming, remove flags as not needed (#1108)
* improve naming, remove flags as not needed

* fix lint
2026-05-13 13:34:37 +02:00
HarisreedharandGitHub bff222a12f try to unify structure of encode_opus_buffer and encode_vpx_buffer (#1107) 2026-05-13 16:26:30 +05:30
Henry RuhsandGitHub 9453a042a1 hash based test for test_stream_image (#1104)
* hash based test for test_stream_image

* new todo for test_video_stream

* new todo for test_video_stream
2026-05-12 19:51:19 +05:30
5e39c60b5c Improve encoder tests with hash assertion (#1103)
* improve test_encode_opus_buffer

* try different hash per os

* fix lint

* add windows check

* update windows hash

* fix test and lint

* update windows hash

* update CI for test_video_encoder.py

* update hash for macos

* update method to use single cpu

* update mac hash

* update windows hash

* cleanup

* restore ci.yml

* remove argument defaults

* selected CI tests

* selected CI tests

* restore ci.yml

---------

Co-authored-by: henryruhs <info@henryruhs.com>
2026-05-12 16:36:06 +05:30
HarisreedharandGitHub 8690ccf49e rename methods (#1102) 2026-05-12 13:22:33 +05:30
Henry RuhsandGitHub e53cb63577 QA - Encoder Testing (#1101)
* testing for audio and video encoders, minor cleanups

* fix lint

* finish create_vpx_encoder, adjust unrelated order of width vs height args
2026-05-12 08:23:27 +02:00
henryruhs 717ff0aa33 restore todos 2026-05-11 21:23:50 +02:00
harisreedhar 4922be4ad3 fix audio 2026-05-11 22:30:00 +05:30
henryruhs fdf1b841b2 setup_platform is no longer needed, bring back conda.py 2026-05-11 17:42:51 +02:00
HarisreedharandGitHub 92296fc5a5 move run_video_encode_loop (#1100) 2026-05-11 21:04:56 +05:30
henryruhs 6968d8fe47 switch to final library repo 2026-05-11 16:36:23 +02:00
Henry Ruhs b6549e873b Refactor/windows libraries (#1099)
* ship libssl and libcrypto in windows

* ship libssl and libcrypto in windows
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs ab7110eb92 Add audio encoder (#1096)
* add audio encoder

* add todos

* add todos

* cleannup and add todos

* fix lint

---------

Co-authored-by: henryruhs <info@henryruhs.com>
2026-05-11 16:36:23 +02:00
Henry Ruhs 9321b41e8e ship libssl and libcrypto in macos (#1097) 2026-05-11 16:36:23 +02:00
henryruhs 450075e20e fix windows 2026-05-11 16:36:23 +02:00
Henry Ruhs 20b392f760 ship libssl and libcrypto in linux (#1095) 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs bb1b8ccf13 Add video_encoder.py (#1094)
* add video_encoder.py

* rename pts to presentation_timestamp

* improve test

* improve test

* cleanup

* cleanup

* fix lint
2026-05-11 16:36:23 +02:00
henryruhs 6eace4ce29 cleanup 2026-05-11 16:36:23 +02:00
henryruhs 6fe735618e cleanup 2026-05-11 16:36:23 +02:00
henryruhs 527bb0ff45 cleanup 2026-05-11 16:36:23 +02:00
henryruhs e05d13c47e cleanup 2026-05-11 16:36:23 +02:00
Henry Ruhs 8bbb6e7062 Cleanup/stream part1 (#1093)
* use find_library from cytypes, enable rtc tests again

* install libvpx and libopus for unix on CI

* bug found when sending audio first

* no need for source path guard

* add more tests for libraries

* add more tests for libraries

* fix testing

* disable tests to see what happens

* disable tests to see what happens

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* hope to solve everything via ENV

* hope to solve everything via ENV

* hope to solve everything via ENV

* hope to solve everything via ENV

* hope to solve everything via ENV

* hope to solve everything via ENV

* hope to solve everything via ENV

* hope to solve everything via ENV

* fix testing

* fix testing

* fix testing

* fix testing

* fix testing

* fix testing

* switch to self hosted libraries

* fixes for macos

* switch to self hosted libraries

* switch to self hosted libraries

* switch to self hosted libraries

* switch to self hosted libraries

* switch to self hosted libraries

* switch to self hosted libraries

* switch to self hosted libraries

* switch to self hosted libraries
2026-05-11 16:36:23 +02:00
Henry Ruhs 76c413a2c1 Refactor/ffmpeg less stream (#1092)
* remove ffmpeg from stream to use opus and vpx, add bunch of todos

* fix testing

* improve download checkout

* fix datachannel download, fix super dirty test clients - setup logic does not belong there

* fix testing
2026-05-11 16:36:23 +02:00
Henry Ruhs 430b16ce56 introduce opus and vpx to libraries (#1091) 2026-05-11 16:36:23 +02:00
henryruhs 9a390bd5bc more uniform codebase for libraries 2026-05-11 16:36:23 +02:00
Henry Ruhs 97e0df01b1 move datachannel to libraries and follow datachannel conventions (#1090)
* move datachannel to libraries and follow new datachannel_module convention

* move datachannel to libraries and follow new datachannel_module convention
2026-05-11 16:36:23 +02:00
Henry Ruhs a8db033033 move stream mode to query parameter (#1089) 2026-05-11 16:36:23 +02:00
henryruhs fda6b7f69d fix test 2026-05-11 16:36:23 +02:00
henryruhs 4198cca18f disable broken tests 2026-05-11 16:36:23 +02:00
henryruhs 9a2d57ae54 refactor detect_websocket_stream_mode and related tests a bit, disable broken tests 2026-05-11 16:36:23 +02:00
henryruhs fe002dc821 skip potential broken tests 2026-05-11 16:36:23 +02:00
henryruhs f8f5d6197d skip potential broken tests 2026-05-11 16:36:23 +02:00
henryruhs 2290e4ea57 giving up - set driver to 0.0.0 for amd 2026-05-11 16:36:23 +02:00
henryruhs 6ec9257e92 add driver version back for amd 2026-05-11 16:36:23 +02:00
henryruhs d727558900 load rocm version for dedicated library 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 07c1c936af Refine RTC bindings: callback-based SDP negotiation, peer state tracking, and type cleanup (#1088)
* Refine RTC bindings: callback-based SDP negotiation, peer state tracking, and type cleanup

* fix lint

* restore peer_connection and rename methods

* remove flags, unused_methods and improve tests

* fix indent
2026-05-11 16:36:23 +02:00
henryruhs 7322bd5d52 fix memory for amd 2026-05-11 16:36:23 +02:00
henryruhs 7f40516f71 use define_xxx for the type() factories 2026-05-11 16:36:23 +02:00
henryruhs ef8567bd3a adjust naming and fix version lookups 2026-05-11 16:36:23 +02:00
henryruhs f8c90b4b25 simplify memory call for amd 2026-05-11 16:36:23 +02:00
henryruhs 343b7f6aad simplify memory call for amd 2026-05-11 16:36:23 +02:00
henryruhs 9b878431d3 rename video memory to memory, add amdsmi library 2026-05-11 16:36:23 +02:00
henryruhs 74a9df35a6 scope the nvidia_ml stuff into module alias 2026-05-11 16:36:23 +02:00
henryruhs 0816bea6d3 cleanup 2026-05-11 16:36:23 +02:00
harisreedharandhenryruhs 6c1475f720 fix lint 2026-05-11 16:36:23 +02:00
henryruhs b549a92a35 cleanup 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 949d9cd276 Move RTC init helpers to bindings layer and clean up structs (#1086)
* converted from type() factory to proper ctypes.Structure subclasses

* change class to method

* bring back rtcSetLocalDescription

* rename rtc_bindings.py to datachannel.py

* rename rtc_library to datachannel_library and rearrange methods

* cleanup
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 99fa2875a9 Fix SDP line endings and media description builder (#1085)
* SDP \r\n fix + build_media_description

* improve tests and remove default arguments

* improve tests

* Refine RTC types and tests with loopback SDP validation

* cleanup
2026-05-11 16:36:23 +02:00
henryruhs b9445fd3a4 add TODOS 2026-05-11 16:36:23 +02:00
henryruhs 66ee9144fb add TODOS 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs dfaa1f9cd4 Replace aiortc with libdatachannel direct pipeline (#1083)
* fix stdin close error

* Refactor stream endpoint, fix encoder thread safety and improve tests

* fix and improve test

* remove not None

* use Enum

* use Enum and add todo

* remove poll
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs a2aedc8814 Add stream helper utilities and IVF frame iterator (#1082)
* Add stream helper utilities and IVF frame iterator

* fix lint

* some cosmetics

* fix lint

* changes

* improve test

* improve types and test

* add todo for better bitrate calculation
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs cc8bfc1af4 Implement RTC store (#1081)
* implement RTC store

* fix ffmpeg_builder

* add RtcSdpOffer type
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 31a16982f3 Refactor RTC library initialization and peer connection API (#1079)
* Replace the mutable RTC_LIBRARY global with @lru_cache on create_static_rtc_library. Expose all RTC_CONFIGURATION fields as parameters with defaults on create_peer_connection. Split add_media_tracks into add_video_track and add_audio_track, each with configurable
   defaults and descriptive parameter names (sync_source_id, canonical_name). Use os.linesep for SDP media descriptions. Simplify init_ctypes to return None and remove the redundant null guard.

* cleanup
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs db545b8ae4 libdatachannel implementation Part 1 (#1077)
* add libdatachannel

* move some methods to rtc_helper.py

* move some methods to rtc_helper.py

* open_vp8_encoder -> spawn_stream_encoder

* update url (linux only)

* fix lint

* remove RTC_STATE and add unit tests

* remove convert_to_raw_rgb

* remove 'utf-8' from decode

* cleanup

* cleanup

* cleanup

* move rtc types to rtc_bindings.py

* remove load_library

* cleanup

* add todos and some cleanup

* fix lint
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 701a1b5f9e Remove upload queue & media chunk reader (#1076)
* remove upload queue & media chunk reader

* remove macos large

---------

Co-authored-by: henryruhs <info@henryruhs.com>
2026-05-11 16:36:23 +02:00
henryruhs 1f80aa735a remove macos large 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs ccfc5f61b6 Fix symatically wrong usage in tests (#1075)
* fix symatically wrong usage in tests

* add target-240p.jpg everywhere

* combine both target
2026-05-11 16:36:23 +02:00
henryruhs 6e7bcb599a once invalid value causes other to fail too 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 47b703f4f5 Ignore set state for non api scope (#1073)
* ignore set state for non api scope

* replace 404 with 422

* remove two loops

* use HTTP_422_UNPROCESSABLE_CONTENT
2026-05-11 16:36:23 +02:00
henryruhs 7111af232c add todos 2026-05-11 16:36:23 +02:00
henryruhs 2709515c39 fix stuck ffmpeg due multi thread lock 2026-05-11 16:36:23 +02:00
henryruhs a9a4adb083 make ci great again 2026-05-11 16:36:23 +02:00
Henry Ruhs eac03796f7 Fix/ffmpeg sanitize (#1074)
* fix sanitize for videos

* fix macos

* fix macos
2026-05-11 16:36:23 +02:00
Henry Ruhs 5790e03009 Feat/ffprobe v2 (#1072)
* follow the todos

* extend to support bit_rate and more

* simplify like crazy

* simplify like crazy

* minor changes

* clean testing

* clean testing

* kill pipe resolver helpers

* kill pipe resolver helpers

* bit rate seems to be different on CI

* use .splitlines() over .split(os.linesep)

* skip test for windows

* hack testing
2026-05-11 16:36:23 +02:00
henryruhs fe7f402474 fix more todos 2026-05-11 16:36:23 +02:00
henryruhs 0fd6a403b8 stop passing format 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs df38778558 fix refresh_session does not validate expiry before refreshing (#1071) 2026-05-11 16:36:23 +02:00
henryruhs e15a2dec76 add tons of todos 2026-05-11 16:36:23 +02:00
henryruhs 399e07261d add tons of todos 2026-05-11 16:36:23 +02:00
henryruhs 61c67c8637 fix macos 2026-05-11 16:36:23 +02:00
henryruhs a16bd54493 add meaningful tests 2026-05-11 16:36:23 +02:00
henryruhs edf8914da3 add meaningful tests 2026-05-11 16:36:23 +02:00
henryruhs 2aced392e1 cleanup code part1 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 2b76f3381f Assets ffmpeg stream upload (#1069)
* ffmpeg sanitize

* fix type

* fix type

* add config

* ChunkQueue -> UploadQueue

* revert assets.py

* move resolve methods to ffmpeg_builder.py

* Refactor ffmpeg.py

* Remove partial import

* improve test

* remove put(None)

* cleanup

* without poll() not working

* ChunkReader -> MediaChunkReader

* improve assert by replacing generic is_file

* naming and cleanup
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs f40ee8335a Fix hardcoded fps (#1068)
* fix hardcoded fps

* fallback to temp_fps when output_video_fps is None

* fix test_ffmpeg

* change temp-fps -> output-audio-fps

* cleanup

* cleanup
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs b9c1ff8185 add test when get_metrics_set() returns None (#1067) 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 48bae74001 fix false vram numbers (#1065) 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 6654abbdca Fix bad caching of graphic devices (#1064)
* fix bad caching of graphic devices

* restore without cache

* restore without cache

* restore without cache

* remove detect_static_graphic_devices and add resolve_static_cudnn_conv_algo_search
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs facd908196 Dynamic bitrate for webrtc stream (#1063)
* use custom aiortc

* update naming to bitrate_
2026-05-11 16:36:23 +02:00
henryruhs 5c6b247449 merge stuff 2026-05-11 16:36:23 +02:00
Henry Ruhs efab505adb asset validation and image encoder lookup (#1058)
* asset validation and image encoder lookup

* asset validation and image encoder lookup

* asset validation and image encoder lookup
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 246d48e079 add download action (#1057) 2026-05-11 16:36:23 +02:00
henryruhs 980c2d3939 fix stream of videos 2026-05-11 16:36:23 +02:00
harisreedharandhenryruhs f0cacba52b fix execution.py import order 2026-05-11 16:36:23 +02:00
henryruhs 61aa016659 burn ui with fire 2026-05-11 16:36:23 +02:00
henryruhsandClaude Opus 4.6 dedf2bf829 updates for v4
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 16:36:22 +02:00
Harisreedharandhenryruhs ad1a6c9ea3 Implement basic webrtc stream (#1054)
* implement basic webrtc_stream

* add aiortc to requirements.txt

* update aiortc version

* rename variables with rtc_ prefix

* changes

* changes

* change helper to assert_helper and stream_helper

* rename variables with rtc_ prefix

* add error handling

* return whole connection

* remove monkey patch and some cleaning

* cleanup

* tiny adjustments

* tiny adjustments

* proper typing and naming for rtc offer set

* - remove async from on_video_track method
- rename source -> target
- add audio

* audio always before video

---------

Co-authored-by: henryruhs <info@henryruhs.com>
2026-05-11 16:36:03 +02:00
henryruhs 9c0859ade0 fix benchmarker, prevent path traveling via job-id 2026-05-11 16:36:01 +02:00
Harisreedharandhenryruhs 1ba2adc10b Rename process to stream (#1053)
* rename process to stream

* remove /image and add webrtc_stream_video scaffolding

* remove _image
2026-05-11 16:35:16 +02:00
Henry Ruhs 078041461d remove more type ignore (#1052) 2026-05-11 16:35:16 +02:00
Henry Ruhs 5ca5e1c2af Better args types part2 (#1051)
* remove helper methods finally

* state becomes the total truth now

* state becomes the total truth now

* state becomes the total truth now

* state becomes the total truth now

* add ini file
2026-05-11 16:35:16 +02:00
Harisreedharandhenryruhs 401118106a Remove unused code to improve test coverage (#1050)
* remove unused code

* move create_session_guard to middlewares
2026-05-11 16:35:16 +02:00
Henry Ruhs f69fa62fa2 better args types (#1049)
* fix lint and remove unused types, restructure helpers

* fix lint

* fix lint

* fix lint
2026-05-11 16:35:16 +02:00
Henry Ruhs ef47b17a4a rename args store to capability store (#1048)
* rename args store to capability store

* fix lint and remove unused types
2026-05-11 16:35:16 +02:00
Harisreedharandhenryruhs 7c078644ad Improve tests (#1047)
* improve tests

* does it hurt someone to have crazy numbers?

---------

Co-authored-by: henryruhs <info@henryruhs.com>
2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 47bdcd67f1 fix pre-check issue 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs f83ff29116 remove pre_check 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 91067c6394 remove program 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 1e8c042473 remove while True and websocket code 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 1f0af5906d hopefully fix test 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 887a6ac1c7 pre_check() on test instead 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 4338d16d17 pre_check() when command == api 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs a21c23173e fix naming and remove unwanted state initialization 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs cd3f6f2224 init 'face_selector_mode' to many 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs f2d9eb7f0b remove temp storing to disk 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs c59596e096 update exception 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 4a0f4bcb9a update test 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs b5a89971c4 process image 2026-05-11 16:35:16 +02:00
Harisreedharandhenryruhs 53845585ef Feat/refactor args (#1045)
* follow the new argument naming convention

* revert

* use get_xxx_arguments

* Simple fix Args

* Fix more stuff

---------

Co-authored-by: henryruhs <info@henryruhs.com>
2026-05-11 16:35:16 +02:00
henryruhs 16d2fbf6f2 fix args store 2026-05-11 16:35:16 +02:00
henryruhs 7a6ef94b26 bump dependencies 2026-05-11 16:35:16 +02:00
henryruhs 9486750f38 allow register_arguments to be nested, fix types, adjust processors 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 1181307a0e reformat test_api_state.py 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs e7a28cf072 introduce type ArgumentSet 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs ecc280a93f remove unwanted assertions 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs aa83142ab5 remove value None as default 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 2a60280182 remove value None as default 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs b9c11815f8 add test_api_capabilities.py 2026-05-11 16:35:16 +02:00
henryruhs accb615181 add get_xxx_set to args store 2026-05-11 16:35:16 +02:00
henryruhs 5f70cbd0a3 fix spacing quite a bit 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 485041c5cc conditional choices, speaking Alias ArgumentValue, different formatting for program.py 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs e16f5bca11 fix test 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 02ba86308e add capabilities 2026-05-11 16:35:16 +02:00
henryruhs fddb5c927b remove unused method 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs f0dc10a2f5 add capabilities endpoint 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 53d0ed0e1c hopefully fix CI 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs f71bbab71e remove has methods 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 300b4f73a1 add has_amd_execution_provider and has_nvidia_execution_provider 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 7e00035fdd fix cache 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs f475ec4bbc remove try block, make detect_graphic_devices stateless, cosmetics 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 9e63832740 fix test 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs ef0bfe81e2 add amd metric 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs f2f09fc215 move detect_execution_devices to system.py 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 73a298cd97 add processor metric 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs b1947ffaa2 add network metrics 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs b7e80bdafa fix test 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs a94015c5ee add memory metrics 2026-05-11 16:35:15 +02:00
harisreedharandhenryruhs 14b1ed6b76 add memory metrics 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs 2624c281eb move metrics.py to endpoints 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs bae6169274 some cosmetics 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs 9a27c8346c add disk to metric test 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs e4b8adb218 change unit to GB 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs 4cd588ebc8 add drive_path 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs aaebb56804 make detect_disk_metrics stateless 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs fa16752a6c add disk metric 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs 09dc2ec50b cleanup 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs 3a25476ca6 add pytest-mock 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs 01baf8d2ea improve test 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs a818e7f610 remove instance check 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs 5f6e0cc567 fix test 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs e2738f1f14 add metrics 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs db5ffdc449 add metrics 2026-05-11 16:35:12 +02:00
henryruhs 2db7ad426a simplify code - no need for helper methods at this point 2026-05-11 16:35:12 +02:00
Harisreedharandhenryruhs abeef470cc Remove xml parsing for gpu metrics (#1030)
* remove xml parsing

* remove aitop
2026-05-11 16:35:10 +02:00
Henry Ruhs 0c18508f70 Revisit and cleanup ffprobe integration (#1027)
* Revisit and cleanup ffprobe integration

* Revisit and cleanup ffprobe integration
2026-05-11 16:34:28 +02:00
henryruhs 60def0f30d Asset related fixes 2026-05-11 16:34:28 +02:00
henryruhs c2676fa471 Asset related fixes 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs dcfbbc041f add missing endpoints 2026-05-11 16:34:28 +02:00
henryruhs dc500eb398 Add process manager to the right place 2026-05-11 16:34:28 +02:00
henryruhs 3575aeea86 Fix for Windows 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 06a3854fbb Revert "hopefully fixes windows CI error"
This reverts commit 3295726347.
2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs d1a068b9d5 hopefully fixes windows CI error 2026-05-11 16:34:28 +02:00
henryruhs 38ece8f7d0 Fix stuff 2026-05-11 16:34:28 +02:00
henryruhs 5fa088457e FFmpeg powered sanitization, Chunk based upload write 2026-05-11 16:34:28 +02:00
henryruhs 77c1682078 Hopefully fix Windows 2026-05-11 16:34:28 +02:00
henryruhs 40f5b403a4 Simplify testing 2026-05-11 16:34:28 +02:00
henryruhs 908f3d2873 Revamp the upload 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 02e8a13adb upload asset endpoint 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs ec3e6d777d upload asset endpoint 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 71c79f4e96 upload asset endpoint 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 746b3249ef upload asset endpoint 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs e3aa3d15f5 upload asset endpoint 2026-05-11 16:34:28 +02:00
henryruhs 2c07d8ca45 Polish asset store and helpers 2026-05-11 16:34:28 +02:00
Harisreedharandhenryruhs 6e84970e0d Feat/audio metadata (#1019)
* audio metadata

* audio metadata

* audio metadata

* audio metadata

* audio metadata

* audio metadata

* audio metadata
2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs f84b2cfc00 asset store update 2026-05-11 16:34:28 +02:00
henryruhs 56842a1c58 Minor cleanup 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 98384a55df state api updates 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs a01bed0bda state api updates 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 6edf6c7266 state api updates 2026-05-11 16:34:28 +02:00
Henry Ruhs 516b5039d0 Move to endpoints directory (#1012)
* Move to endpoints directory

* Move to endpoints directory
2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs dae96f90a1 create to_image.py 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 9bf9ee40a7 audio to image as frames 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 621bddd174 audio to image as frames 2026-05-11 16:34:28 +02:00
henryruhs 5cbc50a5aa Rename local(s) to locale(s) 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 03bbc10251 workflows rename 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 1e2a66e7b0 workflows rename 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 1b2fffd82a image to video as sequence 2026-05-11 16:34:28 +02:00
Henry Ruhs 97801ceab3 feat/ping-endpoint (#1001)
* api: add WebSocket /ping endpoint and update session guard to support WebSocket subprotocol auth; add tests (test_api_ping.py)

* Initial websocket support using ping

* Initial websocket support using ping

* Initial websocket support using ping

* Combine imports
2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 69463b34bc use common analyse_image method 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs e42939df9c to video unification 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs c7ab933a8e to video unification 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 92ec60933e detect workflow 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs f754828d08 changes
restructure conditional methods to a fall-through pattern

common process_temp_frame for all workflow
2026-05-11 16:34:28 +02:00
Henry Ruhs 403e89eeac Feat/session context (#993)
* Add simple session context

* Add simple session context
2026-05-11 16:34:28 +02:00
Henry Ruhs 672d65d61a Add simple path isolation (#992) 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 678f4edd4c update ffmpeg.set_loop
add test

introduce spawn_frames
2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 38e8610523 changes 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 3444194aed remove --keep-temp 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs c4f1d14d71 Part 2 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 0aa56d54bb Part 2 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs cec736d03a fix 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 2485a409b8 part 1 2026-05-11 16:34:28 +02:00
henryruhs 35881a4f8d Switch workflow args order in tests, Remove old choices in processors 2026-05-11 16:34:28 +02:00
henryruhs 86cd7cbc8f Switch workflow args order in tests, Remove old choices in processors 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 7666411a3d add todo
add test

cleanup

remove -w

move --workflow position

fix test

add --worflow, audio-to-image, image-to-image, image-to-video
2026-05-11 16:34:27 +02:00
Henry Ruhs b333280b8d Scope for Args (#988)
* Add API scopes

* Add API scopes

* Add API scopes

* Add API scopes

* Add API scopes

* Add API scopes

* Add API scopes

* Add API scopes

* Remove system memory limit (#986)

* Add session_id, make token size more reasonable (#983)

* Add session_id, make token size more reasonable

* Use more direct approach

* Fix more stuff

* Fix ignore comments

* Fix naming

* Fix lint
2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs 9664bb98db changes 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs 139e73589a changes 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs f11e8eaaea changes 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs b0e849a251 rename to target_path 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs d224e47d38 fix exit-helper 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs 7b1570dc3f refactor temp handling from target-path to output-path 2026-05-11 16:31:45 +02:00
henryruhs 85eeaf39ca Burn it with fire 2026-05-11 16:31:45 +02:00
henryruhs a10b740a80 Burn it with fire 2026-05-11 16:31:45 +02:00
Henry Ruhs 85b85b31a2 Remove system memory limit (#986) 2026-05-11 16:31:45 +02:00
Henry Ruhs ba884a19b1 Add session_id, make token size more reasonable (#983)
* Add session_id, make token size more reasonable

* Use more direct approach
2026-05-11 16:31:45 +02:00
Henry Ruhs 87678da498 Local API (#982)
* Introduce API scelleton

* Raw impl for session

* Simple state endpoint

* Apply _body naming

* Finalize session testing and comment out tons of useless code

* Clean and refactor part1

* Clean and refactor part2

* Clean and refactor part2

* Clean and refactor part2

* Clean and refactor part2

* Refactor middleware

* Refactor middleware

* Clean and refactor part3

* TDD and 2 beers

* TDD and 2 beers

* Complete state endpoints

* You can only set what is already present

* Use only JSON as response

* Use default logger

* Improve auth extraction

* Extend api command with more args

* Adjust API messages
2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs f1149ebc84 remove output_path argument 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs 7870e67fdc remove output_path argument from merge_video() 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs 05264ac760 remove output_video_fps argument from merge_video() 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs 71916e4e66 rename method argument 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs c903438c3f fix 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs 51d4ec63b0 remove same file extension constraint 2026-05-11 16:31:45 +02:00
Henry Ruhs 92b1e1b3e3 Refactor reusable workflow tasks (#980)
* Refactor reusable workflow tasks

* Refactor reusable workflow tasks

* Make it borderline again
2026-05-11 16:31:45 +02:00
Henry Ruhs 1173493e84 Feels so good to get rid of Gradio (#978) 2026-05-11 16:31:45 +02:00
henryruhs daf906097e Mark es temporary v4 2026-05-11 16:31:43 +02:00
Henry RuhsandGitHub 5b7d145aa7 Patch 3.6.1 (#1078)
* Avast intercepts ssl cert and breaks curl

* modernize dependencies

* fix sanitizer for int range

* comment out tests

* disable testing for CI

* disable testing for CI
2026-04-19 21:17:39 +02:00
Henry RuhsandGitHub 519360bcd6 Update LICENSE.md 2026-04-01 09:10:08 +02:00
57fcb86b82 3.6.0 (#1062)
* mark as next

* add fran model

* add support for corridor key (#1060)

* introduce despill color

* simplify the apply dispill color

* finalize naming for both fill and despill

* follow vision_frame convension

* patch fran model

* adjust fran urls

* Feat/dynamic env setup (#1061)

* dynamic environment setup

* dynamic environment setup

* fix fran model

* prevent directml using incompatible corridor_key model

* fix environment setup for windows

* switch to corridor_key_1024 and corridor_key_2048

* switch to corridor_key_1024 and corridor_key_2048

* mark it as 3.6.0

* rename environment to conda

* rename environment to conda

* fix testing for face analyser

* some background remove cosmetics

* some background remove cosmetics

* some background remove cosmetics

* update preview

---------

Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>
2026-03-16 15:08:43 +01:00
henryruhs 2cc05d4fba update licenses 2026-03-09 21:24:53 +01:00
Henry RuhsandGitHub a498f3d618 Patch 3.5.4 (#1055)
* remove insecure flag from curl

* eleminate repating definitons

* limit processors and ui layouts by choices

* follow couple of v4 standards

* use more secure mkstemp

* dynamic cache path for execution providers

* fix benchmarker, prevent path traveling via job-id

* fix order in execution provider choices

* resort by prioroty

* introduce support for QNN

* close file description for Windows to stop crying

* prevent ConnectionResetError under windows

* needed for nested .caches directory as onnxruntime does not create it

* different approach to silent asyncio

* update dependencies

* simplify the name to just inference providers

* switch to trt_builder_optimization_level 4
2026-03-08 11:00:45 +01:00
c7976ec9d4 Fix list literal spacing to use [ x, y ] style (#1044)
Enforce consistent space inside square brackets for all list literals
across source and test files.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-02-18 09:18:03 +01:00
Henry RuhsandGitHub 8801668562 3.5.3
* honor webcam resolution to avoid stripe mismatch, update dependencies

* avoid version conflicts

* enforce prores video extraction to 8 bit

* make the installer more robust on execution switch

* make the installer more robust on execution switch

* improve the installer env handling

* different approach to handle env
2026-02-11 09:35:08 +01:00
Henry RuhsandGitHub a7f3de3dbc Rename local(s) to locale(s) (#1008) 2025-12-17 18:37:51 +01:00
Henry RuhsandGitHub 666c15f9da Patch 3.5.2 (#1002)
* Remove old files

* Fix some spacing

* Introduce retry to download

* More testing

* Better installer scripting (#994)

* Better installer scripting

* Add migraphx installer support

* Add migraphx installer support

* Ignore issue

* Ignore issue

* Make --force-install optional
2025-12-13 08:37:15 +01:00
420d738a6b 3.5.1 (#985)
* Fix type for device id

* Fix type for device id

* Fix order

* Remove comma

* Replace Generator typing

* Replace Generator typing

* Conditional kill myself (#984)

* Introduce conditional remove mask

* fix

---------

Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>

---------

Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>
2025-11-19 17:39:42 +01:00
henryruhs 81c5e85dea Remove invalid command 2025-11-07 12:37:21 +01:00
8bf9170577 3.5.0 (#977)
* Mark as NEXT

* Reduce caching to avoid RAM explosion

* Reduce caching to avoid RAM explosion

* Update dependencies

* add face-detector-pad-factor

* update facefusion.ini

* fix test

* change pad to margin

* fix order

* add prepare margin

* use 50% max margin

* Minor fixes part2

* Minor fixes part3

* Minor fixes part4

* Minor fixes part1

* Downgrade onnxruntime as of BiRefNet broken on CPU

add test

update

update facefusion.ini

add birefnet

* rename models

add more models

* Fix versions

* Add .claude to gitignore

* add normalize color

add 4 channel

add colors

* worflows

* cleanup

* cleanup

* cleanup

* cleanup

* add more models (#961)

* Fix naming

* changes

* Fix style and mock Gradio

* Fix style and mock Gradio

* Fix style and mock Gradio

* apply clamp

* remove clamp

* Add normalizer test

* Introduce sanitizer for the rescue (#963)

* Introduce sanitizer for the rescue

* Introduce sanitizer for the rescue

* Introduce sanitizer for the rescue

* prepare ffmpeg for alpha support

* Some cleanup

* Some cleanup

* Fix CI

* List as TypeAlias is not allowed (#967)

* List as TypeAlias is not allowed

* List as TypeAlias is not allowed

* List as TypeAlias is not allowed

* List as TypeAlias is not allowed

* Add mpeg and mxf support (#968)

* Add mpeg support

* Add mxf support

* Adjust fix_xxx_encoder for the new formats

* Extend output pattern for batch-run (#969)

* Extend output pattern for batch-run

* Add {target_extension} to allowed mixed files

* Catch invalid output pattern keys

* alpha support

* cleanup

* cleanup

* add ProcessorOutputs type

* fix preview and streamer, support alpha for background_remover

* Refactor/open close processors (#972)

* Introduce open/close processors

* Add locales for translator

* Introduce __autoload__ for translator

* More cleanup

* Fix import issues

* Resolve the scope situation for locals

* Fix installer by not using translator

* Fixes after merge

* Fixes after merge

* Fix translator keys in ui

* Use LOCALS in installer

* Update and partial fix DirectML

* Use latest onnxruntime

* Fix performance

* Fix lint issues

* fix mask

* fix lint

* fix lint

* Remove default from translator.get()

* remove 'framerate='

* fix test

* Rename and reorder models

* Align naming

* add alpha preview

* fix frame-by-frame

* Add alpha effect via css

* preview support alpha channel

* fix preview modes

* Use official assets repositories

* Add support for u2net_cloth

* fix naming

* Add more models

* Add vendor, license and year direct to the models

* Add vendor, license and year direct to the models

* Update dependencies, Minor CSS adjustment

* Ready for 3.5.0

* Fix naming

* Update about messages

* Fix return

* Use groups to show/hide

* Update preview

* Conditional merge mask

* Conditional merge mask

* Fix import order

---------

Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>
Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>
2025-11-03 14:05:15 +01:00
189d750621 3.4.2 (#975)
* Reduce caching to avoid RAM explosion

* Reduce caching to avoid RAM explosion

* Reduce caching to avoid RAM explosion

* fix bounding_box scale

* Bump to 3.4.2

* More conservative on audio caching

* Fix coverage issue

* Fix coverage issue

---------

Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>
2025-10-29 11:51:42 +01:00
Henry RuhsandGitHub f3be23d19b Patch (#947)
* Fix preview when using frame enhancer

* Fix version conflict numpy vs. cv2

* Use latest numpy

* Introduce scale_face() to match size of temp frames and target frames

* Remove hardcoded backend for camera under Windows

* Up and downgrade some dependencies

* Up and downgrade some dependencies

* Up and downgrade some dependencies
2025-09-11 16:58:39 +02:00
henryruhs 16e84b43ce Update preview 2025-09-08 10:54:22 +02:00
da0da3a4b4 Next (#945)
* Rename calcXXX to calculateXXX

* Add migraphx support

* Add migraphx support

* Add migraphx support

* Add migraphx support

* Add migraphx support

* Add migraphx support

* Use True for the flags

* Add migraphx support

* add face-swapper-weight

* add face-swapper-weight to facefusion.ini

* changes

* change choice

* Fix typing for xxxWeight

* Feat/log inference session (#906)

* Log inference session, Introduce time helper

* Log inference session, Introduce time helper

* Log inference session, Introduce time helper

* Log inference session, Introduce time helper

* Mark as NEXT

* Follow industry standard x1, x2, y1 and y2

* Follow industry standard x1, x2, y1 and y2

* Follow industry standard in terms of naming (#908)

* Follow industry standard in terms of naming

* Improve xxx_embedding naming

* Fix norm vs. norms

* Reduce timeout to 5

* Sort out voice_extractor once again

* changes

* Introduce many to the occlusion mask (#910)

* Introduce many to the occlusion mask

* Then we use minimum

* Add support for wmv

* Run platform tests before has_execution_provider (#911)

* Add support for wmv

* Introduce benchmark mode (#912)

* Honestly makes no difference to me

* Honestly makes no difference to me

* Fix wording

* Bring back YuNet (#922)

* Reintroduce YuNet without cv2 dependency

* Fix variable naming

* Avoid RGB to YUV colorshift using libx264rgb

* Avoid RGB to YUV colorshift using libx264rgb

* Make libx264 the default again

* Make libx264 the default again

* Fix types in ffmpeg builder

* Fix quality stuff in ffmpeg builder

* Fix quality stuff in ffmpeg builder

* Add libx264rgb to test

* Revamp Processors (#923)

* Introduce new concept of pure target frames

* Radical refactoring of process flow

* Introduce new concept of pure target frames

* Fix webcam

* Minor improvements

* Minor improvements

* Use deque for video processing

* Use deque for video processing

* Extend the video manager

* Polish deque

* Polish deque

* Deque is not even used

* Improve speed with multiple futures

* Fix temp frame mutation and

* Fix RAM usage

* Remove old types and manage method

* Remove execution_queue_count

* Use init_state for benchmarker to avoid issues

* add voice extractor option

* Change the order of voice extractor in code

* Use official download urls

* Use official download urls

* add gui

* fix preview

* Add remote updates for voice extractor

* fix crash on headless-run

* update test_job_helper.py

* Fix it for good

* Remove pointless method

* Fix types and unused imports

* Revamp reference (#925)

* Initial revamp of face references

* Initial revamp of face references

* Initial revamp of face references

* Terminate find_similar_faces

* Improve find mutant faces

* Improve find mutant faces

* Move sort where it belongs

* Forward reference vision frame

* Forward reference vision frame also in preview

* Fix reference selection

* Use static video frame

* Fix CI

* Remove reference type from frame processors

* Improve some naming

* Fix types and unused imports

* Fix find mutant faces

* Fix find mutant faces

* Fix imports

* Correct naming

* Correct naming

* simplify pad

* Improve webcam performance on highres

* Camera manager (#932)

* Introduce webcam manager

* Fix order

* Rename to camera manager, improve video manager

* Fix CI

* Remove optional

* Fix naming in webcam options

* Avoid using temp faces (#933)

* output video scale

* Fix imports

* output image scale

* upscale fix (not limiter)

* add unit test scale_resolution & remove unused methods

* fix and add test

* fix

* change pack_resolution

* fix tests

* Simplify output scale testing

* Fix benchmark UI

* Fix benchmark UI

* Update dependencies

* Introduce REAL multi gpu support using multi dimensional inference pool (#935)

* Introduce REAL multi gpu support using multi dimensional inference pool

* Remove the MULTI:GPU flag

* Restore "processing stop"

* Restore "processing stop"

* Remove old templates

* Go fill in with caching

* add expression restorer areas

* re-arrange

* rename method

* Fix stop for extract frames and merge video

* Replace arcface_converter models with latest crossface models

* Replace arcface_converter models with latest crossface models

* Move module logs to debug mode

* Refactor/streamer (#938)

* Introduce webcam manager

* Fix order

* Rename to camera manager, improve video manager

* Fix CI

* Fix naming in webcam options

* Move logic over to streamer

* Fix streamer, improve webcam experience

* Improve webcam experience

* Revert method

* Revert method

* Improve webcam again

* Use release on capture instead

* Only forward valid frames

* Fix resolution logging

* Add AVIF support

* Add AVIF support

* Limit avif to unix systems

* Drop avif

* Drop avif

* Drop avif

* Default to Documents in the UI if output path is not set

* Update wording.py (#939)

"succeed" is grammatically incorrect in the given context. To succeed is the infinitive form of the verb. Correct would be either "succeeded" or alternatively a form involving the noun "success".

* Fix more grammar issue

* Fix more grammar issue

* Sort out caching

* Move webcam choices back to UI

* Move preview options to own file (#940)

* Fix Migraphx execution provider

* Fix benchmark

* Reuse blend frame method

* Fix CI

* Fix CI

* Fix CI

* Hotfix missing check in face debugger, Enable logger for preview

* Fix reference selection (#942)

* Fix reference selection

* Fix reference selection

* Fix reference selection

* Fix reference selection

* Side by side preview (#941)

* Initial side by side preview

* More work on preview, remove UI only stuff from vision.py

* Improve more

* Use fit frame

* Add different fit methods for vision

* Improve preview part2

* Improve preview part3

* Improve preview part4

* Remove none as choice

* Remove useless methods

* Fix CI

* Fix naming

* use 1024 as preview resolution default

* Fix fit_cover_frame

* Uniform fit_xxx_frame methods

* Add back disabled logger

* Use ui choices alias

* Extract select face logic from processors (#943)

* Extract select face logic from processors to use it for face by face in preview

* Fix order

* Remove old code

* Merge methods

* Refactor face debugger (#944)

* Refactor huge method of face debugger

* Remove text metrics from face debugger

* Remove useless copy of temp frame

* Resort methods

* Fix spacing

* Remove old method

* Fix hard exit to work without signals

* Prevent upscaling for face-by-face

* Switch to version

* Improve exiting

---------

Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>
Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>
Co-authored-by: Rafael Tappe Maestro <rafael@tappemaestro.com>
2025-09-08 10:43:58 +02:00
henryruhs 7b8bea4e0a Make HyperSwap the default 2025-07-10 15:22:41 +02:00
henryruhs b3678c4e4c Fix Gradio 2025-07-10 15:10:44 +02:00
Henry RuhsandGitHub ac3504d03b Hotfix release (#907)
* Revert to old-fashion face store

* Bump release

* Don't be evil

* Don't be evil
2025-07-07 20:21:39 +02:00
Henry RuhsandGitHub 26c701cd62 Merge pull request #902 from facefusion/next
3.3.0
2025-06-22 17:19:10 +02:00
henryruhs 5c1b6ad5ad Remove weight multiplier from edtalk 2025-06-22 14:38:56 +02:00
henryruhs 1e0f6890d5 Update preview 2025-06-22 11:52:22 +02:00
henryruhs 9d169164f6 Remove statistics 2025-06-22 10:52:03 +02:00
henryruhs 2193794501 Mark as 3.3.0 2025-06-22 10:45:53 +02:00
henryruhs 8f2687801b Final preparations 2025-06-21 12:55:10 +02:00
henryruhs 43e1e4bf44 Prepare HyperSwap models 2025-06-20 15:34:21 +02:00
henryruhs 7e6c97d275 CPU is faster then CoreML 2025-06-20 10:53:07 +02:00
henryruhs 6bc948b94f Cosmetic spaces 2025-06-19 08:54:30 +02:00
henryruhs 9e9b272660 Fix CI 2025-06-19 08:42:04 +02:00
henryruhs 6f9997bd31 Fix lip syncer 2025-06-19 08:41:06 +02:00
henryruhs fc501754cd Introduce vision area based hashing 2025-06-18 00:43:53 +02:00
henryruhs 93756d1295 Update face store to faster hashing 2025-06-18 00:24:30 +02:00
henryruhs b7275b6b06 Always render the CLI table 2025-06-17 23:54:39 +02:00
henryruhs 6773ab4d97 Fix paste back 2025-06-17 23:18:04 +02:00
henryruhs c390c6d882 Add HyperSwap (no models yet) 2025-06-17 19:24:16 +02:00
henryruhs b76ca08dea cosmetics changes on lip syncer 2025-06-17 19:18:46 +02:00
henryruhs d1385fed0d Basic state manager test 2025-06-17 10:47:02 +02:00
henryruhs 5112a15564 Basic state manager test 2025-06-17 10:08:30 +02:00
henryruhs a8655f58b6 Polish paste back 2025-06-16 22:46:17 +02:00
henryruhs 9235802b03 Polish paste back 2025-06-16 22:25:35 +02:00
henryruhs c65574d6ac Polish paste back 2025-06-16 22:01:22 +02:00
Henry RuhsandGitHub 129d1fd76c Merge pull request #901 from facefusion/feat/optimized-pasteback
Optimized pasteback
2025-06-16 19:01:43 +02:00
harisreedhar d1e8fbab08 optimized pasteback 2025-06-16 20:02:59 +05:30
henryruhs 4200a646f6 Polish benchmark and orders 2025-06-16 16:02:12 +02:00
henryruhs be26611693 Polish benchmark and orders 2025-06-16 15:59:40 +02:00
Henry RuhsandGitHub 5218d852ec Merge pull request #900 from facefusion/vibe/benchmark-command
Benchmark command
2025-06-16 13:12:49 +02:00
henryruhs 6340d67c96 Polish types 2025-06-16 12:27:15 +02:00
henryruhs 505432cfde This types are needed 2025-06-16 12:08:18 +02:00
henryruhs 13772f8400 This types are never needed 2025-06-16 12:04:31 +02:00
henryruhs bf651f5420 Fix import 2025-06-16 12:03:20 +02:00
henryruhs 4f20419484 Minor wording 2025-06-16 12:01:11 +02:00
henryruhs 9e1d60fd07 Fix UI 2025-06-16 11:52:52 +02:00
henryruhs 56d714368a Fix UI 2025-06-16 11:44:22 +02:00
henryruhs 0feecfd9d7 Finalize choices and types 2025-06-16 11:21:48 +02:00
henryruhs 26d0a5b9bb Vibe coded benchmark command part5 2025-06-16 10:32:31 +02:00
henryruhs 293dff56ed Vibe coded benchmark command part4 2025-06-16 09:54:37 +02:00
henryruhs 0f92cdfab1 Vibe coded benchmark command part3 2025-06-15 20:26:20 +02:00
henryruhs 45fbac0220 Vibe coded benchmark command part3 2025-06-15 20:23:52 +02:00
henryruhs 703c64bed5 Vibe coded benchmark command part3 2025-06-15 19:53:12 +02:00
Henry RuhsandGitHub ca77ff62ee Merge pull request #898 from facefusion/feat/area_mask
area mask
2025-06-15 17:29:55 +02:00
henryruhs 19400a92bd Vibe coded benchmark command part2 2025-06-14 11:00:52 +02:00
henryruhs 589c317f19 Vibe coded benchmark command 2025-06-13 23:15:34 +02:00
henryruhs 543013fcf3 Implement area mask, refactor box mask 2025-06-13 18:42:14 +02:00
henryruhs 4d34a92f6f Implement area mask, refactor box mask 2025-06-13 18:39:37 +02:00
henryruhs 55e9df9c39 Fix mypy 2025-06-13 10:28:48 +02:00
henryruhs dec8797d58 Fix lint 2025-06-13 10:26:44 +02:00
harisreedhar bdc186f67b lipsyncer: revert to old bbox setting 2025-06-13 12:29:50 +05:30
henryruhs 17a94c0604 Introduce signal_exit() 2025-06-12 23:40:40 +02:00
henryruhs cee560e89a Ban lambda usage 2025-06-12 23:02:39 +02:00
henryruhs 3298ed144b Ban lambda usage 2025-06-12 23:01:15 +02:00
henryruhs 74a40cec2e Fix wav2lip 2025-06-12 18:13:19 +02:00
henryruhs 3df70b14ca Kill some elif 2025-06-12 17:54:30 +02:00
henryruhs 885a9b044f Kill some elif and else 2025-06-12 17:43:54 +02:00
henryruhs 4150cda64e Fix spacing 2025-06-12 17:36:48 +02:00
Henry RuhsandGitHub 75344fa0dd Merge pull request #897 from facefusion/organize-lipsyncer
Add Edtalk and re organize lipsyncer
2025-06-12 17:13:30 +02:00
harisreedhar 7905cfe6a3 stage 1 OCD 2025-06-12 20:07:41 +05:30
harisreedhar 36cad4d1b7 add edtalk 2025-06-12 19:08:26 +05:30
harisreedhar fb05361dc3 space 2025-06-12 17:18:37 +05:30
harisreedhar 8aec81d63b re-arrange to methods 2025-06-12 17:07:51 +05:30
henryruhs f712570d1e Fix spacing 2025-06-12 13:01:29 +02:00
HarisreedharandGitHub fdb28e4fcc Merge pull request #896 from facefusion/feat/face-areas
Landmark powered Face Areas
2025-06-12 16:20:39 +05:30
harisreedhar ff1320fbcb update face_mask_area_set 2025-06-12 16:19:10 +05:30
henryruhs ed67b83eb3 Fix wrong area 2025-06-12 11:14:53 +02:00
henryruhs 53641e1779 Rename head to face 2025-06-11 15:11:19 +02:00
henryruhs e51c4e4782 Cosmetic changes 2025-06-11 15:01:55 +02:00
henryruhs 41bcea7ee3 Cosmetic changes 2025-06-11 15:01:24 +02:00
henryruhs c44bfca389 Add landmark powered face areas 2025-06-11 14:52:41 +02:00
henryruhs 2cad038ca3 Add landmark powered face areas 2025-06-11 14:40:34 +02:00
henryruhs a460b0d390 Polish content analyser 2025-06-10 23:01:26 +02:00
henryruhs f79770aa13 Polish content analyser 2025-06-10 20:03:27 +02:00
henryruhs 8e80ab0d21 Polish content analyser 2025-06-10 19:30:58 +02:00
Henry RuhsandGitHub f65aabfd72 Merge pull request #895 from facefusion/multimodel-content-analyser
Reduce content analyser false positives
2025-06-10 18:21:10 +02:00
harisreedhar 6b03388f76 fix test 2025-06-10 18:05:20 +05:30
harisreedhar 2309b4d79a multi model content analyser 2025-06-10 17:38:51 +05:30
henryruhs aab2db1478 Fix typo in test job runner 2025-06-10 08:38:55 +02:00
henryruhs a26713353f Fix typo in test job runner 2025-06-10 08:25:02 +02:00
henryruhs d766bfaef3 Finish job testing 2025-06-10 08:15:20 +02:00
henryruhs 9fc32f2d94 Introduce lip syncer weight 2025-06-09 16:32:40 +02:00
henryruhs 918dd2c0d6 Introduce lip syncer weight 2025-06-09 15:33:11 +02:00
henryruhs 28081e32f1 Introduce lip syncer weight 2025-06-09 15:05:56 +02:00
henryruhs f05f74f5a1 Introduce lip syncer weight 2025-06-09 15:02:28 +02:00
henryruhs 076c313842 Change of plans 2025-06-04 22:29:46 +02:00
henryruhs 911f8d280c Fix lint 2025-05-29 10:54:11 +02:00
henryruhs a02df0be7e Try to fix UDP stream 2025-05-28 22:09:01 +02:00
henryruhs 00047858a8 Try to fix UDP stream 2025-05-28 21:47:11 +02:00
henryruhs 52c3391aa4 Try to fix UDP stream 2025-05-28 21:32:11 +02:00
henryruhs 1695e95ba5 Undo set_hardware_accelerator 2025-05-27 21:46:19 +02:00
henryruhs 7ba3db5fe3 Add set_hardware_accelerator for ffmpeg 2025-05-27 20:47:35 +02:00
henryruhs e5046a444e Turns out ultra_sharp_2_x4 needs huge tiles 2025-05-27 13:02:59 +02:00
henryruhs bbad9a4732 Add ultra_sharp_2_x4 to frame enhancers 2025-05-27 12:48:37 +02:00
henryruhs aab2e0aec5 Introduce fix_audio_encoder and fix_video_encoder 2025-05-25 18:13:58 +02:00
henryruhs 854f895403 Introduce fix_audio_encoder and fix_video_encoder 2025-05-25 18:02:37 +02:00
henryruhs 1b5621cac5 Fix libopus test 2025-05-25 15:52:52 +02:00
henryruhs 84e86ad91e Improve testing 2025-05-25 15:45:38 +02:00
henryruhs 441305afe5 Improve testing 2025-05-25 15:37:41 +02:00
henryruhs 4021ddf3b7 Fix audio bitrate for libopus 2025-05-25 13:00:06 +02:00
henryruhs 3c36440262 More testing for audio restore/replace and video merge 2025-05-24 21:00:19 +02:00
henryruhs f9b906850b Stub get_available_encoder_set() 2025-05-23 23:05:56 +02:00
henryruhs 5b37c9b45b CI is very sensitive 2025-05-23 22:25:52 +02:00
henryruhs b1786ddd0b CI is very sensitive 2025-05-23 22:25:01 +02:00
henryruhs 7c65323558 CI is very sensitive 2025-05-23 22:05:48 +02:00
henryruhs df8d4317a7 Fallbacks for audio 2025-05-23 21:43:58 +02:00
henryruhs c273a834f6 Renaming temp_xxx_path 2025-05-23 20:56:55 +02:00
henryruhs 14bb3e32c2 Testing for merge_video() 2025-05-23 20:18:11 +02:00
henryruhs 068f6c86d1 More edge case testing 2025-05-23 19:20:56 +02:00
henryruhs fc0de4eec2 More edge case testing 2025-05-23 19:06:38 +02:00
henryruhs 834c17340d More edge case testing 2025-05-23 18:53:03 +02:00
henryruhs 04319fb3c6 More edge case testing 2025-05-23 13:09:54 +02:00
henryruhs 68da1cf266 More edge case testing 2025-05-23 09:17:51 +02:00
henryruhs 52000114ba Fix merge edge cases 2025-05-22 22:44:24 +02:00
henryruhs 23343eaa76 Testing for merge_video() 2025-05-22 20:58:46 +02:00
henryruhs 377c728a91 Fix spacing 2025-05-22 16:46:51 +02:00
Henry RuhsandGitHub 5005cb6098 Merge pull request #886 from facefusion/feat/video_manager
feat/video manager
2025-05-22 14:38:22 +02:00
henryruhs 42e72f8d7e Fix CI 2025-05-22 11:00:50 +02:00
henryruhs 4343bf5629 Fix moving temp files for Windows 2025-05-22 08:49:12 +02:00
henryruhs f50cb74099 Introduce video manager to handle broken videos 2025-05-22 09:29:15 +02:00
henryruhs 62483fdfad Introduce video manager to handle broken videos 2025-05-22 09:26:36 +02:00
henryruhs 8470ebee4f Minor spacing in audio.py 2025-05-21 14:52:47 +02:00
henryruhs 6f3d8ffcb0 Remove unnecessary code block 2025-05-21 14:42:34 +02:00
henryruhs 5bbe0ec481 Minor spacing in audio.py 2025-05-20 16:31:00 +02:00
henryruhs 801108faca Minor rename in vision.py 2025-05-20 15:48:55 +02:00
henryruhs 94535ab9dc Update onnxruntime 2025-05-18 18:12:56 +02:00
henryruhs 7c2308f19e Replace Patrick Stewart model 2025-05-16 15:42:44 +02:00
henryruhs 1687ed7b58 Add more DeepFaceLive models 2025-05-05 14:16:57 +02:00
henryruhs 7a85d8b2cd Fix typo 2025-05-05 14:00:43 +02:00
henryruhs 0e01ca131d Introduce NEXT 2025-05-05 01:47:14 +02:00
Henry RuhsandGitHub a6fb48780c 3.2.0
3.2.0
2025-04-29 11:33:12 +02:00
henryruhs 0b7349bde0 Remove old config 2025-04-28 10:56:26 +02:00
henryruhs 3e563a3eb9 Prepare 3.2.0 2025-04-28 10:20:17 +02:00
henryruhs 1baa134126 Prepare 3.2.0 2025-04-28 10:03:29 +02:00
henryruhs 4e5675aa2c Use correct class for temp file formats 2025-04-28 10:03:29 +02:00
henryruhs 162025161a Use correct class for temp file formats 2025-04-28 10:03:29 +02:00
henryruhs 5118c7229f Rename arcface_128_v2 to arcface_128 2025-04-28 10:03:29 +02:00
henryruhs 72a0edb6ba Use module_name everywhere 2025-04-28 10:03:29 +02:00
henryruhs 84b9b60e6e Update onnxruntime 2025-04-28 10:03:29 +02:00
henryruhs 7f4b232a4f Allow deep nested wording 2025-04-28 10:03:29 +02:00
henryruhs 5ebefb6831 Allow deep nested wording 2025-04-28 10:03:29 +02:00
henryruhs b6156f6073 Allow deep nested wording 2025-04-28 10:03:29 +02:00
henryruhs ff645b9069 Allow deep nested wording 2025-04-28 10:03:29 +02:00
henryruhs 106c1f20b3 Remove useless global 2025-04-28 10:03:29 +02:00
henryruhs aaebc018ce Update dependencies 2025-04-28 10:03:29 +02:00
henryruhs b9356d50db Add support for rawvideo 2025-04-28 10:03:29 +02:00
henryruhs b9021a8250 Fix reading wrong config section 2025-04-28 10:03:29 +02:00
henryruhs 5db1e3421c Enable fast prediction for CoreML 2025-04-28 10:03:29 +02:00
henryruhs bb5504c7a4 Use latest onnxruntime and enable model cache for CoreML 2025-04-28 10:03:29 +02:00
henryruhs 096d31cad6 Fix CI 2025-04-28 10:03:29 +02:00
henryruhs 57ef628138 Make get_first and get_last safe 2025-04-28 10:03:29 +02:00
henryruhs 4c09f88b42 Make get_first and get_last safe 2025-04-28 10:03:29 +02:00
henryruhs 1c5a2337e8 Make get_first and get_last safe 2025-04-28 10:03:29 +02:00
henryruhs 89c0493acd Follow the config parser way part4 2025-04-28 10:03:29 +02:00
henryruhs b480c76e35 Follow the config parser way part3 2025-04-28 10:03:29 +02:00
henryruhs 3250f39708 Follow the config parser way part2 2025-04-28 10:03:29 +02:00
henryruhs b4e54e6502 Fix CI 2025-04-28 10:03:29 +02:00
henryruhs 671acae887 Follow the config parser way 2025-04-28 10:03:29 +02:00
henryruhs 6f62d41595 Introduce resolve_cudnn_conv_algo_search 2025-04-28 10:03:29 +02:00
henryruhs e56eabe981 Enable macos for CI again 2025-04-28 10:03:29 +02:00
henryruhs 65939b0c79 Add license 2025-04-28 10:03:29 +02:00
henryruhs dca3c31e51 Remove macos again 2025-04-28 10:03:29 +02:00
henryruhs 22b9e89eff Let's try macos 13 2025-04-28 10:03:29 +02:00
henryruhs 9d7b69b73d Let's try macos 14 2025-04-28 10:03:29 +02:00
henryruhs bdf068521a Disable macos as ffmpeg fails 2025-04-28 10:03:29 +02:00
henryruhs 02d500cdb2 Change naming of config parser 2025-04-28 10:03:29 +02:00
henryruhs a07b77b31a Change naming of config parser 2025-04-28 10:03:29 +02:00
henryruhs d43147ffe9 Add space 2025-04-28 10:03:29 +02:00
henryruhs 679731c02b Use different hashing for the face store 2025-04-28 10:03:29 +02:00
henryruhs 27f77f92fb Use different hashing for the face store 2025-04-28 10:03:29 +02:00
henryruhs a91cb9a3ed Add multi GPU support for openvino 2025-04-28 10:03:29 +02:00
henryruhs 7bdd084c20 Add multi GPU support for openvino 2025-04-28 10:03:29 +02:00
henryruhs 6c49a9779b Update dependencies and fix layout 2025-04-28 10:03:28 +02:00
henryruhs d459ae981e Rename to detect_face_landmark() 2025-04-28 10:03:25 +02:00
henryruhs f1a4011254 Fix CI 2025-04-28 10:03:25 +02:00
henryruhs 288c66f5c0 Get rid of suggest_execution_provider() via the insert trick 2025-04-28 10:03:25 +02:00
henryruhs d56d651568 Get rid of suggest_execution_provider() via the insert trick 2025-04-28 10:03:25 +02:00
henryruhs 0fdbd167f0 Use flac over aac if present 2025-04-28 10:03:25 +02:00
henryruhs 1d045028ab Change license to OpenRAIL 2025-04-28 10:03:25 +02:00
henryruhs d03e3b3ed7 Add audio encoders pcm_s16le and pcm_s32le 2025-04-28 10:03:25 +02:00
henryruhs 48aaba2786 Undo non-fixes 2025-04-28 10:03:25 +02:00
henryruhs aa1007ff6a Fix blank screen in replace_audio() 2025-04-28 10:03:25 +02:00
henryruhs 2bb91a8098 Fix blank screen in restore_audio() 2025-04-28 10:03:25 +02:00
Henry Ruhs 944306a19c Introduce TypeAlias everywhere (#869)
* Introduce TypeAlias everywhere

* Undo changes
2025-04-28 10:03:25 +02:00
henryruhs 7dbe17596f Create preview when video is not playable 2025-04-28 10:03:25 +02:00
Henry Ruhs d43cf5cdde Fix/extrating merging progress bar total (#868)
* Introduce predict_video_frame_total() to fix progress bar while extracting and merging

* Introduce predict_video_frame_total() to fix progress bar while extracting and merging

* Use floor over ceil
2025-04-28 10:03:25 +02:00
henryruhs c0e856ef72 Fix audio quality mapping for aac 2025-04-28 10:03:25 +02:00
henryruhs 81f34bec2b Fix audio quality mapping for aac 2025-04-28 10:03:25 +02:00
henryruhs 75eba39f95 Fix the media resolution order 2025-04-28 10:03:25 +02:00
henryruhs b0f37ba46c Update Gradio 2025-04-28 10:03:25 +02:00
henryruhs 41514c2d67 Fix model size detection 2025-04-28 10:03:25 +02:00
henryruhs 977d8d91b5 Move restricted temp_video_fps to core 2025-04-28 10:03:25 +02:00
henryruhs c7abeb4785 Minor fix 2025-04-28 10:03:25 +02:00
henryruhs 96878917b5 Prevent false calculation for rate 2025-04-28 10:03:25 +02:00
Henry Ruhs f3bbd3e16f Qa/follow set naming (#867)
* Follow set naming

* Follow set naming

* Disable type hints

* Uniform order
2025-04-28 10:03:25 +02:00
henryruhs 1bdc02014c Fix broken inference pools part2 2025-04-28 10:03:25 +02:00
henryruhs b438b899e8 Fix broken inference pools 2025-04-28 10:03:25 +02:00
henryruhs 858acd2fe3 Remove conditional_exit() and introduce early exit 2025-04-28 10:03:25 +02:00
Henry Ruhs 5e3ab88f63 Fix broken inference pools (#866)
* Fix broken inference pools

* Fix broken inference pools
2025-04-28 10:03:25 +02:00
henryruhs 7ceb3389dc Adjust naming 2025-04-28 10:03:25 +02:00
henryruhs 44165eb0b7 Rename typing to types 2025-04-28 10:03:25 +02:00
henryruhs 997649ed4a Update ico 2025-04-28 10:03:25 +02:00
henryruhs c25f046c2b Add support for flac and opus audio format 2025-04-28 10:03:25 +02:00
henryruhs 359dddc121 Add support for flac and opus audio format 2025-04-28 10:03:25 +02:00
henryruhs c9296de559 Add m4a and m4v support 2025-04-28 10:03:25 +02:00
henryruhs e47140ea27 Fix spacing 2025-04-28 10:03:25 +02:00
henryruhs 616914d38f Improve the face selector experience 2025-04-28 10:03:25 +02:00
henryruhs 6cc21d7a91 Remove default value 2025-04-28 10:03:25 +02:00
henryruhs 6732a82af4 Fix audio range in restore audio 2025-04-28 10:03:25 +02:00
henryruhs 51a809310e Fix audio range in restore audio 2025-04-28 10:03:25 +02:00
henryruhs ddd83d0606 Fix audio range in restore audio 2025-04-28 10:03:25 +02:00
Henry Ruhs 296eea8577 Feat/halt on error (#862)
* Introduce halt-on-error

* Introduce halt-on-error

* Fix wording
2025-04-28 10:03:25 +02:00
henryruhs efc9652df4 Extend flake8 rules 2025-04-28 10:03:25 +02:00
henryruhs 2fc4d2b02e Fix typo 2025-04-28 10:03:25 +02:00
Henry Ruhs c70b45bd39 Feat/improve content analyser (#861)
* Introduce fit_frame to improve content analyser, rename resize_frame_resolution to restrict_frame

* Fix CI, Add some spaces

* Normalize according to face detector
2025-04-28 10:03:25 +02:00
henryruhs e79a99fac4 Skip the new test 2025-04-28 10:03:25 +02:00
henryruhs 5dc986d6b2 Skip the new test 2025-04-28 10:03:25 +02:00
henryruhs 2892dd484d Use latest Gradio 2025-04-28 10:03:25 +02:00
henryruhs 1eed88b0f5 Override for Gradio's nasty convert_video_to_playable_mp4() 2025-04-28 10:03:25 +02:00
henryruhs c363faa757 Remove Gradio preview interception for now 2025-04-28 10:03:25 +02:00
henryruhs cd6d2012f6 Remove upload event handler 2025-04-28 10:03:25 +02:00
henryruhs 1c06023160 Remove upload event handler 2025-04-28 10:03:25 +02:00
henryruhs 2e16b10200 Remove upload event handler 2025-04-28 10:03:25 +02:00
henryruhs 5f82c8dc7d Add some spacing 2025-04-28 10:03:25 +02:00
Henry Ruhs d260c28cf3 Feat/available encoders (#860)
* Introduce available audio encoders and video encoders

* Introduce available audio encoders and video encoders

* Introduce available audio encoders and video encoders

* Introduce available audio encoders and video encoders

* Add flac to audio encoders
2025-04-28 10:03:25 +02:00
henryruhs 30d9b038e4 Fix CI 2025-04-28 10:03:25 +02:00
henryruhs 02d11435a6 Simplify set_audio_sample_size() 2025-04-28 10:03:25 +02:00
henryruhs b9eb6a7cc0 Fix typo in get_file_format 2025-04-28 10:03:25 +02:00
henryruhs 4f32ed7868 Rename to read_video_frame 2025-04-28 10:03:25 +02:00
Henry Ruhs 0e6ee69c53 Feat/content analyser pro (#859)
* Update to Yolo powered content analyser

* Update to Yolo powered content analyser

* Fix typing

* Drop bounding boxes and NMS check

* Drop bounding boxes and NMS check

* Fix CI
2025-04-28 10:03:25 +02:00
henryruhs 87e3a80491 We name it InferenceSessionProvider 2025-04-28 10:03:25 +02:00
henryruhs dbbf3445b6 We name it InferenceSessionProvider 2025-04-28 10:03:25 +02:00
Henry Ruhs 5270bd679c Follow yolo convention, renaming in face detector (#858)
* Follow yolo convention, renaming in face detector

* Follow yolo convention, renaming in face detector
2025-04-28 10:03:25 +02:00
henryruhs 0ecebc8c82 Fix import 2025-04-28 10:03:25 +02:00
henryruhs f4255e66fa Add new model by Druuzil 2025-04-28 10:03:25 +02:00
henryruhs a45754892d Add logical blocks 2025-04-28 10:03:25 +02:00
henryruhs 9dd397579c Remove useless strip 2025-04-28 10:03:25 +02:00
henryruhs bedd75920d Update dependencies 2025-04-28 10:03:25 +02:00
henryruhs 3a437fdc92 Update dependencies 2025-04-28 10:03:25 +02:00
Henry Ruhs 3b80d66bf4 Feat/better resolve execution (#856)
* A better way to resolve execution providers

* Fix issues

* Fix issues
2025-04-28 10:03:25 +02:00
henryruhs 330f86a4e4 Use 255.0 everywhere 2025-04-28 10:03:25 +02:00
henryruhs 2f66bde116 get_file_extension needs lower() 2025-04-28 10:03:25 +02:00
Harisreedharandhenryruhs 01fff3c419 change face distance range to 0-1 (#855) 2025-04-28 10:03:25 +02:00
henryruhs e5278996d1 Add tiff image format 2025-04-28 10:03:25 +02:00
Henry Ruhs faf5020051 Suggest best execution provider, Simplify ONNXRUNTIME_SET (#854) 2025-04-28 10:03:25 +02:00
henryruhs 732f096da0 Reintroduce the hack for coreml 2025-04-28 10:03:25 +02:00
henryruhs 2d95e409cb Remove default values 2025-04-28 10:03:25 +02:00
henryruhs 0553ef4766 Drop has_inference_model and solve issue on Gradio side 2025-04-28 10:03:25 +02:00
henryruhs ed8e25dbb2 Drop has_inference_model and solve issue on Gradio side 2025-04-28 10:03:25 +02:00
henryruhs b11cb07aea Move render table to cli helper 2025-04-28 10:03:25 +02:00
henryruhs 964fab8724 no, jpeg IS a format 2025-04-28 10:03:24 +02:00
henryruhs 9d9805a03b jpeg is not a format 2025-04-28 10:03:24 +02:00
harisreedharandhenryruhs 169d578a14 jpeg support 2025-04-28 10:03:24 +02:00
henryruhs 63de6a8d8a Use numpy to transform ranges 2025-04-28 10:03:24 +02:00
henryruhs 56ba551630 Fix set video quality 2025-04-28 10:03:24 +02:00
henryruhs b8ebcf9fe3 Fix set audio quality 2025-04-28 10:03:24 +02:00
Henry Ruhs 8a9e08f3a2 Feat/commands builder (#852)
* Protype for ffmpeg builder

* Protype for ffmpeg builder

* Add curl builder

* Fix typing import

* Adjust commands indent

* Protype for ffmpeg builder part2

* Protype for ffmpeg builder part3

* Protype for ffmpeg builder part3

* Add chain() helper to the builders

* Protype for ffmpeg builder part4

* Protype for ffmpeg builder part5

* Protoype for ffmpeg builder part5

* Protoype for ffmpeg builder part6

* Allow dynamic audio size

* Fix testing

* Protoype for ffmpeg builder part7

* Fix and polish ffmpeg builder

* Hardcode the log level for ffmpeg

* More ffmpeg rework

* Prototype for ffmpeg builder part8

* Prototype for ffmpeg builder part9

* Fix CI

* Fix Styles

* Add lazy testing, User Agent for CURL

* More testing

* More testing
2025-04-28 10:03:24 +02:00
Henry Ruhs 7f90ca72bb Add another xseg model, Simplify download mapping (#851) 2025-04-28 10:03:24 +02:00
Henry Ruhs 71092cb951 Improve and simplify installer (#850) 2025-04-28 10:03:24 +02:00
henryruhs fbe2e57da6 Fix typing import 2025-04-28 10:03:24 +02:00
henryruhs 99c514c58f Introduce Commands type 2025-04-28 10:03:24 +02:00
henryruhs 36ddef4ba1 Adjust wording 2025-04-28 10:03:24 +02:00
Henry Ruhs 5b76f54332 Feat/more audio settings (#849)
* Add more audio settings, revamp some ffmpeg commands

* Add more audio settings, revamp some ffmpeg commands

* Add more audio settings, revamp some ffmpeg commands

* Add more audio settings, revamp some ffmpeg commands
2025-04-28 10:03:24 +02:00
Henry Ruhs c5bc7c50a5 Fix/remote inference pool lookups (#848)
* Fix edge case when offline and inference session has no model, Prevent inference session creation

* Fix edge case when offline and inference session has no model, Prevent inference session creation

* Fix edge case when offline and inference session has no model, Prevent inference session creation

* Fix edge case when offline and inference session has no model, Prevent inference session creation
2025-04-28 10:03:24 +02:00
Henry Ruhs 87350eb45f Support for download provider mirrors (#847) 2025-04-28 10:03:24 +02:00
Henry Ruhs 6f0675030e Feat/custom file format handling (#845)
* Purge filetype dependency, Rename file_extension to file_format, Introduce custom format detections

* Changed a lot

* Purge filetype dependency, Rename file_extension to file_format, Introduce custom format detections

* Fix stuff

* Fix stuff

* Simplify all the is_ and has_ methods

* Simplify all the is_ and has_ methods

* Use the new helper on more places

* Introduce are_ next to is_ and has_

* Get rid of the type-ignores

* Add more video types
2025-04-28 10:03:24 +02:00
henryruhs bb32135af2 Add 10 seconds timeout for curl 2025-04-28 10:03:24 +02:00
henryruhs 749abc3406 Fix CI 2025-04-28 10:03:24 +02:00
henryruhs 612fd70e54 Remove Windows-only path sanitization 2025-04-28 10:03:24 +02:00
henryruhs eb2f794ece Extend testing 2025-04-28 10:03:24 +02:00
henryruhs efc49c367a Introduce NEXT 2025-04-28 10:03:23 +02:00
henryruhs d4fba8421f Fix gradio using pinned pydantic 2025-03-28 12:07:01 +01:00
henryruhs 1e3bab9644 Hotfix Geforce 16 series 2025-01-02 22:13:31 +01:00
henryruhs 197773c346 Hotfix Geforce 16 series 2025-01-02 22:03:48 +01:00
henryruhs 8656411336 Bump version 2024-12-31 18:34:51 +01:00
Henry RuhsandGitHub 8efd7c1fa0 Revert CoreML fallbacks (#841) 2024-12-31 18:34:39 +01:00
Henry RuhsandGitHub b48eac30d0 Temporary use the http links 2024-12-29 10:11:59 +01:00
Henry RuhsandGitHub befd75421f Update FUNDING.yml 2024-12-29 01:07:23 +01:00
311 changed files with 17852 additions and 9804 deletions
+2
View File
@@ -0,0 +1,2 @@
[run]
patch = subprocess
+1 -1
View File
@@ -1,5 +1,5 @@
[flake8] [flake8]
select = E3, E4, F, I1, I2 select = E22, E23, E24, E27, E3, E4, E7, F, I1, I2
per-file-ignores = facefusion.py:E402, install.py:E402 per-file-ignores = facefusion.py:E402, install.py:E402
plugins = flake8-import-order plugins = flake8-import-order
application_import_names = facefusion application_import_names = facefusion
+1 -1
View File
@@ -1 +1 @@
custom: [ buymeacoffee.com/henryruhs, ko-fi.com/henryruhs, paypal.me/henryruhs ] custom: [ buymeacoffee.com/facefusion, ko-fi.com/facefusion ]
Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.3 MiB

+8 -2
View File
@@ -33,8 +33,11 @@ jobs:
uses: actions/setup-python@v5 uses: actions/setup-python@v5
with: with:
python-version: '3.12' python-version: '3.12'
- run: python install.py --onnxruntime default --skip-conda - run: python install.py default --skip-conda
- run: pip install pytest - run: pip install pytest
- run: pip install pytest-mock
- run: pip install httpx
- run: pip install python-multipart
- run: pytest - run: pytest
report: report:
needs: test needs: test
@@ -48,10 +51,13 @@ jobs:
uses: actions/setup-python@v5 uses: actions/setup-python@v5
with: with:
python-version: '3.12' python-version: '3.12'
- run: python install.py --onnxruntime default --skip-conda - run: python install.py default --skip-conda
- run: pip install coveralls - run: pip install coveralls
- run: pip install pytest - run: pip install pytest
- run: pip install pytest-cov - run: pip install pytest-cov
- run: pip install pytest-mock
- run: pip install httpx
- run: pip install python-multipart
- run: pytest tests --cov facefusion - run: pytest tests --cov facefusion
- run: coveralls --service github - run: coveralls --service github
env: env:
+3 -1
View File
@@ -1,6 +1,8 @@
__pycache__ __pycache__
.assets .assets
.claude
.caches .caches
.jobs
.idea .idea
.jobs
.libraries
.vscode .vscode
+2 -2
View File
@@ -1,3 +1,3 @@
MIT license OpenRAIL-AS license
Copyright (c) 2024 Henry Ruhs Copyright (c) 2026 Henry Ruhs
+4 -9
View File
@@ -5,19 +5,13 @@ FaceFusion
[![Build Status](https://img.shields.io/github/actions/workflow/status/facefusion/facefusion/ci.yml.svg?branch=master)](https://github.com/facefusion/facefusion/actions?query=workflow:ci) [![Build Status](https://img.shields.io/github/actions/workflow/status/facefusion/facefusion/ci.yml.svg?branch=master)](https://github.com/facefusion/facefusion/actions?query=workflow:ci)
[![Coverage Status](https://img.shields.io/coveralls/facefusion/facefusion.svg)](https://coveralls.io/r/facefusion/facefusion) [![Coverage Status](https://img.shields.io/coveralls/facefusion/facefusion.svg)](https://coveralls.io/r/facefusion/facefusion)
![License](https://img.shields.io/badge/license-MIT-green) ![License](https://img.shields.io/badge/license-OpenRAIL--AS-green)
Preview
-------
![Preview](https://raw.githubusercontent.com/facefusion/facefusion/master/.github/preview.png?sanitize=true)
Installation Installation
------------ ------------
Be aware, the [installation](https://docs.facefusion.io/installation) needs technical skills and is not recommended for beginners. In case you are not comfortable using a terminal, our [Windows Installer](https://windows-installer.facefusion.io) and [macOS Installer](https://macos-installer.facefusion.io) get you started. Be aware, the [installation](https://docs.facefusion.io/installation) needs technical skills and is not recommended for beginners. In case you are not comfortable using a terminal, our [Windows Installer](http://windows-installer.facefusion.io) and [macOS Installer](http://macos-installer.facefusion.io) get you started.
Usage Usage
@@ -34,9 +28,10 @@ options:
commands: commands:
run run the program run run the program
headless-run run the program in headless mode
batch-run run the program in batch mode batch-run run the program in batch mode
force-download force automate downloads and exit force-download force automate downloads and exit
benchmark benchmark the program
api start the API server
job-list list jobs by status job-list list jobs by status
job-create create a drafted job job-create create a drafted job
job-submit submit a drafted job to become a queued job job-submit submit a drafted job to become a queued job
BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 94 KiB

After

Width:  |  Height:  |  Size: 20 KiB

+42 -17
View File
@@ -1,3 +1,6 @@
[workflow]
workflow =
[paths] [paths]
temp_path = temp_path =
jobs_path = jobs_path =
@@ -13,6 +16,7 @@ output_pattern =
[face_detector] [face_detector]
face_detector_model = face_detector_model =
face_detector_size = face_detector_size =
face_detector_margin =
face_detector_angles = face_detector_angles =
face_detector_score = face_detector_score =
@@ -31,39 +35,54 @@ reference_face_position =
reference_face_distance = reference_face_distance =
reference_frame_number = reference_frame_number =
[face_tracker]
face_tracker_score =
[face_masker] [face_masker]
face_occluder_model = face_occluder_model =
face_parser_model = face_parser_model =
face_mask_types = face_mask_types =
face_mask_areas =
face_mask_regions =
face_mask_blur = face_mask_blur =
face_mask_padding = face_mask_padding =
face_mask_regions =
[voice_extractor]
voice_extractor_model =
[frame_extraction] [frame_extraction]
trim_frame_start = trim_frame_start =
trim_frame_end = trim_frame_end =
temp_frame_format = temp_frame_format =
keep_temp =
[frame_distribution]
target_frame_amount =
[output_creation] [output_creation]
output_image_quality = output_image_quality =
output_image_resolution = output_image_scale =
output_audio_encoder = output_audio_encoder =
output_audio_quality =
output_audio_volume =
output_audio_fps =
output_video_encoder = output_video_encoder =
output_video_preset = output_video_preset =
output_video_quality = output_video_quality =
output_video_resolution = output_video_scale =
output_video_fps = output_video_fps =
skip_audio =
[processors] [processors]
processors = processors =
age_modifier_model = age_modifier_model =
age_modifier_direction = age_modifier_direction =
background_remover_model =
background_remover_fill_color =
background_remover_despill_color =
deep_swapper_model = deep_swapper_model =
deep_swapper_morph = deep_swapper_morph =
expression_restorer_model = expression_restorer_model =
expression_restorer_factor = expression_restorer_factor =
expression_restorer_areas =
face_debugger_items = face_debugger_items =
face_editor_model = face_editor_model =
face_editor_eyebrow_direction = face_editor_eyebrow_direction =
@@ -85,31 +104,37 @@ face_enhancer_blend =
face_enhancer_weight = face_enhancer_weight =
face_swapper_model = face_swapper_model =
face_swapper_pixel_boost = face_swapper_pixel_boost =
face_swapper_weight =
frame_colorizer_model = frame_colorizer_model =
frame_colorizer_size = frame_colorizer_size =
frame_colorizer_blend = frame_colorizer_blend =
frame_enhancer_model = frame_enhancer_model =
frame_enhancer_blend = frame_enhancer_blend =
lip_syncer_model = lip_syncer_model =
lip_syncer_weight =
[uis]
open_browser =
ui_layouts =
ui_workflow =
[execution]
execution_device_id =
execution_providers =
execution_thread_count =
execution_queue_count =
[download] [download]
download_providers = download_providers =
download_scope = download_scope =
[benchmark]
benchmark_mode =
benchmark_resolutions =
benchmark_cycle_count =
[api]
api_host =
api_port =
api_security_strategy =
[execution]
execution_device_ids =
execution_providers =
execution_thread_count =
[memory] [memory]
video_memory_strategy = video_memory_strategy =
system_memory_limit =
[misc] [misc]
log_level = log_level =
halt_on_error =
+2 -1
View File
@@ -4,7 +4,8 @@ import os
os.environ['OMP_NUM_THREADS'] = '1' os.environ['OMP_NUM_THREADS'] = '1'
from facefusion import core from facefusion import conda, core
if __name__ == '__main__': if __name__ == '__main__':
conda.setup()
core.cli() core.cli()
+14
View File
@@ -0,0 +1,14 @@
from typing import Optional
from starlette.datastructures import Headers
from starlette.types import Scope
def get_sec_websocket_protocol(scope : Scope) -> Optional[str]:
protocol_header = Headers(scope = scope).get('Sec-WebSocket-Protocol')
if protocol_header:
protocol, _, _ = protocol_header.partition(',')
return protocol.strip()
return None
+91
View File
@@ -0,0 +1,91 @@
import asyncio
import os
import uuid
from typing import List, Optional
from starlette.datastructures import UploadFile
import facefusion.choices
from facefusion import ffmpeg, process_manager, state_manager
from facefusion.filesystem import create_directory, get_file_extension, get_file_format, is_audio, is_image, is_video
from facefusion.types import ImageMetadata, MediaType
from facefusion.vision import detect_image_resolution
def extract_image_metadata(file_path : str) -> ImageMetadata:
metadata : ImageMetadata =\
{
'resolution': detect_image_resolution(file_path)
}
return metadata
def detect_media_type_by_path(file_path : str) -> Optional[MediaType]:
if is_audio(file_path):
return 'audio'
if is_image(file_path):
return 'image'
if is_video(file_path):
return 'video'
return None
def detect_media_type_by_format(file_format : str) -> Optional[MediaType]:
if file_format in facefusion.choices.audio_set:
return 'audio'
if file_format in facefusion.choices.image_set:
return 'image'
if file_format in facefusion.choices.video_set:
return 'video'
return None
def validate_asset_files(upload_files : List[UploadFile]) -> bool:
available_encoder_set = ffmpeg.get_static_available_encoder_set()
for upload_file in upload_files:
file_format = get_file_format(upload_file.filename)
media_type = detect_media_type_by_format(file_format)
if media_type == 'audio' and facefusion.choices.audio_set.get(file_format) not in available_encoder_set.get('audio'): #type:ignore[call-overload]
return False
if media_type == 'image' and facefusion.choices.image_set.get(file_format) not in available_encoder_set.get('image'): #type:ignore[call-overload]
return False
if media_type == 'video' and facefusion.choices.video_set.get(file_format) not in available_encoder_set.get('video'): #type:ignore[call-overload]
return False
return True
async def save_asset_files(upload_files : List[UploadFile]) -> List[str]:
asset_paths : List[str] = []
api_security_strategy = state_manager.get_item('api_security_strategy')
for upload_file in upload_files:
file_format = get_file_format(upload_file.filename)
file_extension = get_file_extension(upload_file.filename)
media_type = detect_media_type_by_format(file_format)
temp_path = state_manager.get_temp_path()
create_directory(temp_path)
asset_file_name = uuid.uuid4().hex
asset_path = os.path.join(temp_path, asset_file_name + file_extension)
file_content = await upload_file.read()
process_manager.start()
if media_type == 'audio' and await asyncio.to_thread(ffmpeg.sanitize_audio, file_content, asset_path, api_security_strategy):
asset_paths.append(asset_path)
if media_type == 'image' and await asyncio.to_thread(ffmpeg.sanitize_image, file_content, asset_path):
asset_paths.append(asset_path)
if media_type == 'video' and await asyncio.to_thread(ffmpeg.sanitize_video, file_content, asset_path, api_security_strategy):
asset_paths.append(asset_path)
process_manager.end()
return asset_paths
+92
View File
@@ -0,0 +1,92 @@
import uuid
from datetime import datetime, timedelta
from typing import List, Optional, cast
from facefusion.apis.asset_helper import detect_media_type_by_path, extract_image_metadata
from facefusion.ffprobe import extract_audio_metadata, extract_video_metadata
from facefusion.filesystem import get_file_format, get_file_name, get_file_size
from facefusion.types import AssetId, AssetSet, AssetStore, AssetType, AudioAsset, AudioFormat, ImageAsset, ImageFormat, SessionId, VideoAsset, VideoFormat
ASSET_STORE : AssetStore = {}
def create_asset(session_id : SessionId, asset_type : AssetType, asset_path : str) -> Optional[AudioAsset | ImageAsset | VideoAsset]:
asset_id = str(uuid.uuid4())
asset_name = get_file_name(asset_path)
asset_format = get_file_format(asset_path)
asset_size = get_file_size(asset_path)
media_type = detect_media_type_by_path(asset_path)
created_at = datetime.now()
expires_at = created_at + timedelta(hours = 2)
if session_id not in ASSET_STORE:
ASSET_STORE[session_id] = {}
if media_type == 'audio':
ASSET_STORE[session_id][asset_id] = cast(AudioAsset,
{
'id': asset_id,
'created_at': created_at,
'expires_at': expires_at,
'type': asset_type,
'media': media_type,
'name': asset_name,
'format': cast(AudioFormat, asset_format),
'size': asset_size,
'path': asset_path,
'metadata': extract_audio_metadata(asset_path)
})
if media_type == 'image':
ASSET_STORE[session_id][asset_id] = cast(ImageAsset,
{
'id': asset_id,
'created_at': created_at,
'expires_at': expires_at,
'type': asset_type,
'media': media_type,
'name': asset_name,
'format': cast(ImageFormat, asset_format),
'size': asset_size,
'path': asset_path,
'metadata': extract_image_metadata(asset_path)
})
if media_type == 'video':
ASSET_STORE[session_id][asset_id] = cast(VideoAsset,
{
'id': asset_id,
'created_at': created_at,
'expires_at': expires_at,
'type': asset_type,
'media': media_type,
'name': asset_name,
'format': cast(VideoFormat, asset_format),
'size': asset_size,
'path': asset_path,
'metadata': extract_video_metadata(asset_path)
})
return ASSET_STORE[session_id].get(asset_id)
def get_assets(session_id : SessionId) -> Optional[AssetSet]:
return ASSET_STORE.get(session_id)
def get_asset(session_id : SessionId, asset_id : AssetId) -> Optional[AudioAsset | ImageAsset | VideoAsset]:
if session_id in ASSET_STORE:
return ASSET_STORE.get(session_id).get(asset_id)
return None
def delete_assets(session_id : SessionId, asset_ids : List[AssetId]) -> None:
if session_id in ASSET_STORE:
for asset_id in asset_ids:
if asset_id in ASSET_STORE.get(session_id):
del ASSET_STORE[session_id][asset_id]
return None
def clear() -> None:
ASSET_STORE.clear()
+58
View File
@@ -0,0 +1,58 @@
from types import ModuleType
from typing import List
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.cors import CORSMiddleware
from starlette.routing import Route, WebSocketRoute
from facefusion.apis.endpoints.assets import delete_assets, get_asset, get_assets, upload_asset
from facefusion.apis.endpoints.capabilities import get_capabilities
from facefusion.apis.endpoints.metrics import get_metrics, websocket_metrics
from facefusion.apis.endpoints.ping import websocket_ping
from facefusion.apis.endpoints.session import create_session, destroy_session, get_session, refresh_session
from facefusion.apis.endpoints.state import get_state, set_state
from facefusion.apis.endpoints.stream import delete_stream, post_stream, websocket_stream
from facefusion.apis.middlewares.session import create_session_guard
from facefusion.libraries import aom as aom_module, datachannel as datachannel_module, opus as opus_module, vpx as vpx_module
def get_common_modules() -> List[ModuleType]:
return [ aom_module, datachannel_module, opus_module, vpx_module ]
def pre_check() -> bool:
for common_module in get_common_modules():
if not common_module.pre_check():
return False
return True
def create_api() -> Starlette:
session_guard = Middleware(create_session_guard)
routes =\
[
Route('/session', create_session, methods = [ 'POST' ]),
Route('/session', get_session, methods = [ 'GET' ], middleware = [ session_guard ]),
Route('/session', refresh_session, methods = [ 'PUT' ]),
Route('/session', destroy_session, methods = [ 'DELETE' ], middleware = [ session_guard ]),
Route('/state', get_state, methods = [ 'GET' ], middleware = [ session_guard ]),
Route('/state', set_state, methods = [ 'PUT' ], middleware = [ session_guard ]),
Route('/assets', get_assets, methods = [ 'GET' ], middleware = [ session_guard ]),
Route('/assets', upload_asset, methods = [ 'POST' ], middleware = [ session_guard ]),
Route('/assets/{asset_id}', get_asset, methods = [ 'GET' ], middleware = [ session_guard ]),
Route('/assets', delete_assets, methods = [ 'DELETE' ], middleware = [ session_guard ]),
Route('/capabilities', get_capabilities, methods = [ 'GET' ]),
Route('/metrics', get_metrics, methods = [ 'GET' ], middleware = [ session_guard ]),
Route('/stream', post_stream, methods = [ 'POST' ], middleware = [ session_guard ]),
Route('/stream', delete_stream, methods = [ 'DELETE' ], name = 'delete_stream', middleware = [ session_guard ]),
WebSocketRoute('/metrics', websocket_metrics, middleware = [ session_guard ]),
WebSocketRoute('/ping', websocket_ping, middleware = [ session_guard ]),
WebSocketRoute('/stream', websocket_stream, middleware = [ session_guard ])
]
api = Starlette(routes = routes)
api.add_middleware(CORSMiddleware, allow_origins = [ '*' ], allow_methods = [ '*' ], allow_headers = [ '*' ])
return api
+135
View File
@@ -0,0 +1,135 @@
import os
from typing import List
from starlette.requests import Request
from starlette.responses import FileResponse, JSONResponse, Response
from starlette.status import HTTP_200_OK, HTTP_201_CREATED, HTTP_400_BAD_REQUEST, HTTP_404_NOT_FOUND, HTTP_415_UNSUPPORTED_MEDIA_TYPE
from facefusion import session_context, session_manager
from facefusion.apis import asset_store
from facefusion.apis.asset_helper import save_asset_files, validate_asset_files
from facefusion.apis.endpoints.session import extract_access_token
from facefusion.filesystem import remove_file
async def upload_asset(request : Request) -> Response:
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
asset_type = request.query_params.get('type')
if session_id and asset_type in [ 'source', 'target' ]:
session_context.set_session_id(session_id)
form = await request.form()
upload_files = form.getlist('file')
if upload_files and validate_asset_files(upload_files):
asset_paths = await save_asset_files(upload_files)
if asset_paths:
asset_ids : List[str] = []
for asset_path in asset_paths:
asset = asset_store.create_asset(session_id, asset_type, asset_path)
if asset:
asset_id = asset.get('id')
if asset_id:
asset_ids.append(asset_id)
if asset_ids:
return JSONResponse(
{
'asset_ids': asset_ids
}, status_code = HTTP_201_CREATED)
return Response(status_code = HTTP_415_UNSUPPORTED_MEDIA_TYPE)
return Response(status_code = HTTP_400_BAD_REQUEST)
async def get_assets(request : Request) -> Response:
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
if session_id:
asset_set = asset_store.get_assets(session_id)
assets = []
if asset_set:
for asset in asset_set.values():
assets.append(
{
'id': asset.get('id'),
'created_at': asset.get('created_at').isoformat(),
'expires_at': asset.get('expires_at').isoformat(),
'type': asset.get('type'),
'media': asset.get('media'),
'name': asset.get('name'),
'format': asset.get('format'),
'size': asset.get('size'),
'metadata': asset.get('metadata')
})
return JSONResponse(
{
'assets': assets
}, status_code = HTTP_200_OK)
return Response(status_code = HTTP_400_BAD_REQUEST)
async def get_asset(request : Request) -> Response:
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
asset_id = request.path_params.get('asset_id')
if session_id and asset_id:
asset = asset_store.get_asset(session_id, asset_id)
if asset:
if request.query_params.get('action') == 'download':
asset_path = asset.get('path')
if os.path.exists(asset_path):
return FileResponse(asset_path, filename = asset.get('name'))
return JSONResponse(
{
'id': asset.get('id'),
'created_at': asset.get('created_at').isoformat(),
'expires_at': asset.get('expires_at').isoformat(),
'type': asset.get('type'),
'media': asset.get('media'),
'name': asset.get('name'),
'format': asset.get('format'),
'size': asset.get('size'),
'metadata': asset.get('metadata')
}, status_code = HTTP_200_OK)
return Response(status_code = HTTP_404_NOT_FOUND)
async def delete_assets(request : Request) -> Response:
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
body = await request.json()
asset_ids = body.get('asset_ids')
if session_id and asset_ids:
asset_set = asset_store.get_assets(session_id)
if asset_set:
for asset_id in asset_ids:
if asset_id in asset_set:
asset = asset_set.get(asset_id)
if asset:
remove_file(asset.get('path'))
asset_store.delete_assets(session_id, asset_ids)
return Response(status_code = HTTP_200_OK)
return Response(status_code = HTTP_404_NOT_FOUND)
+20
View File
@@ -0,0 +1,20 @@
from starlette.requests import Request
from starlette.responses import JSONResponse
from starlette.status import HTTP_200_OK
import facefusion.choices
from facefusion import capability_store
async def get_capabilities(request : Request) -> JSONResponse:
capabilities =\
{
'formats':
{
'audio': facefusion.choices.audio_formats,
'image': facefusion.choices.image_formats,
'video': facefusion.choices.video_formats
},
'arguments': capability_store.get_api_capability_group()
}
return JSONResponse(capabilities, status_code = HTTP_200_OK)
+32
View File
@@ -0,0 +1,32 @@
import asyncio
from starlette.requests import Request
from starlette.responses import JSONResponse, Response
from starlette.status import HTTP_404_NOT_FOUND
from starlette.websockets import WebSocket
from facefusion.apis.api_helper import get_sec_websocket_protocol
from facefusion.system import get_metrics_set
async def get_metrics(request : Request) -> Response:
metrics_set = get_metrics_set()
if metrics_set:
return JSONResponse(metrics_set)
return Response(status_code = HTTP_404_NOT_FOUND)
async def websocket_metrics(websocket : WebSocket) -> None:
subprotocol = get_sec_websocket_protocol(websocket.scope)
await websocket.accept(subprotocol = subprotocol)
try:
while True:
metrics_set = get_metrics_set()
await websocket.send_json(metrics_set)
await asyncio.sleep(2)
except Exception:
pass
+16
View File
@@ -0,0 +1,16 @@
from starlette.websockets import WebSocket
from facefusion.apis.api_helper import get_sec_websocket_protocol
async def websocket_ping(websocket : WebSocket) -> None:
subprotocol = get_sec_websocket_protocol(websocket.scope)
await websocket.accept(subprotocol = subprotocol)
try:
while True:
await websocket.receive()
except Exception:
pass
+75
View File
@@ -0,0 +1,75 @@
import os
import secrets
from starlette.requests import Request
from starlette.responses import JSONResponse
from starlette.status import HTTP_200_OK, HTTP_201_CREATED, HTTP_401_UNAUTHORIZED
from facefusion import session_context, session_manager, translator
from facefusion.apis.session_helper import extract_access_token
async def create_session(request : Request) -> JSONResponse:
body = await request.json()
if not body.get('api_key') or body.get('api_key') == os.getenv('FACEFUSION_API_KEY'):
session_id = secrets.token_urlsafe(16)
session = session_manager.create_session()
session_context.set_session_id(session_id)
session_manager.set_session(session_id, session)
return JSONResponse(
{
'access_token': session.get('access_token'),
'refresh_token': session.get('refresh_token')
}, status_code = HTTP_201_CREATED)
return JSONResponse(
{
'message': translator.get('something_went_wrong', 'facefusion.apis')
}, status_code = HTTP_401_UNAUTHORIZED)
async def get_session(request : Request) -> JSONResponse:
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
session = session_manager.get_session(session_id)
return JSONResponse(
{
'access_token': session.get('access_token'),
'refresh_token': session.get('refresh_token'),
'created_at': session.get('created_at').isoformat(),
'expires_at': session.get('expires_at').isoformat()
}, status_code = HTTP_200_OK)
async def refresh_session(request : Request) -> JSONResponse:
body = await request.json()
for session_id, session in session_manager.SESSIONS.items():
if session.get('refresh_token') == body.get('refresh_token') and session_manager.validate_session(session_id):
__session__ = session_manager.create_session()
session_manager.set_session(session_id, __session__)
return JSONResponse(
{
'access_token': __session__.get('access_token'),
'refresh_token': __session__.get('refresh_token')
}, status_code = HTTP_200_OK)
return JSONResponse(
{
'message': translator.get('something_went_wrong', 'facefusion.apis')
}, status_code = HTTP_401_UNAUTHORIZED)
async def destroy_session(request : Request) -> JSONResponse:
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
session_manager.clear_session(session_id)
return JSONResponse(
{
'message': translator.get('ok', 'facefusion.apis')
}, status_code = HTTP_200_OK)
+93
View File
@@ -0,0 +1,93 @@
from starlette.requests import Request
from starlette.responses import JSONResponse
from starlette.status import HTTP_200_OK, HTTP_400_BAD_REQUEST, HTTP_404_NOT_FOUND, HTTP_422_UNPROCESSABLE_CONTENT
from facefusion import args_helper, capability_store, session_manager, state_manager, translator
from facefusion.apis import asset_store
from facefusion.apis.endpoints.session import extract_access_token
async def get_state(request : Request) -> JSONResponse:
api_args = args_helper.extract_api_args(state_manager.get_state())
return JSONResponse(state_manager.collect_state(api_args), status_code = HTTP_200_OK)
async def set_state(request : Request) -> JSONResponse:
__api_args__ = {}
action = request.query_params.get('action')
asset_type = request.query_params.get('type')
if action == 'select' and asset_type == 'source':
return await select_source(request)
if action == 'select' and asset_type == 'target':
return await select_target(request)
body = await request.json()
api_args = capability_store.get_api_arguments()
for key, value in body.items():
if key not in api_args:
return JSONResponse(
{
'message': translator.get('invalid_state_key', 'facefusion.apis')
}, status_code = HTTP_400_BAD_REQUEST)
__api_args__[key] = value
if __api_args__:
for key, value in __api_args__.items():
state_manager.set_item(key, value)
__api_args__ = args_helper.extract_api_args(state_manager.get_state())
return JSONResponse(state_manager.collect_state(__api_args__), status_code = HTTP_200_OK)
return JSONResponse({}, status_code = HTTP_422_UNPROCESSABLE_CONTENT)
async def select_source(request : Request) -> JSONResponse:
body = await request.json()
asset_ids = body.get('asset_ids')
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
if isinstance(asset_ids, list) and session_id:
source_paths = []
for asset_id in asset_ids:
asset = asset_store.get_asset(session_id, asset_id)
if asset:
source_paths.append(asset.get('path'))
state_manager.set_item('source_paths', source_paths)
__api_args__ = args_helper.extract_api_args(state_manager.get_state())
return JSONResponse(state_manager.collect_state(__api_args__), status_code = HTTP_200_OK)
return JSONResponse(
{
'message': translator.get('source_asset_not_found', 'facefusion.apis')
}, status_code = HTTP_404_NOT_FOUND)
async def select_target(request : Request) -> JSONResponse:
body = await request.json()
asset_id = body.get('asset_id')
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
if isinstance(asset_id, str) and session_id:
asset = asset_store.get_asset(session_id, asset_id)
if asset:
state_manager.set_item('target_path', asset.get('path'))
__api_args__ = args_helper.extract_api_args(state_manager.get_state())
return JSONResponse(state_manager.collect_state(__api_args__), status_code = HTTP_200_OK)
return JSONResponse(
{
'message': translator.get('target_asset_not_found', 'facefusion.apis')
}, status_code = HTTP_404_NOT_FOUND)
+52
View File
@@ -0,0 +1,52 @@
from starlette.requests import Request
from starlette.responses import Response
from starlette.status import HTTP_200_OK, HTTP_201_CREATED, HTTP_404_NOT_FOUND
from starlette.websockets import WebSocket, WebSocketState
from facefusion import session_context, session_manager
from facefusion.apis.api_helper import get_sec_websocket_protocol
from facefusion.apis.session_helper import extract_access_token
from facefusion.apis.stream_manager import destroy_stream, process_image, process_video
async def websocket_stream(websocket : WebSocket) -> None:
subprotocol = get_sec_websocket_protocol(websocket.scope)
access_token = extract_access_token(websocket.scope)
session_id = session_manager.find_session_id(access_token)
session_context.set_session_id(session_id)
await websocket.accept(subprotocol = subprotocol)
await process_image(websocket)
if websocket.client_state == WebSocketState.CONNECTED:
await websocket.close()
async def post_stream(request : Request) -> Response:
headers =\
{
'Location': request.url_for('delete_stream').path
}
content_type = request.headers.get('content-type')
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
session_context.set_session_id(session_id)
if session_id and content_type == 'application/sdp':
sdp_offer = await request.body()
sdp_answer = process_video(session_id, sdp_offer.decode())
if sdp_answer:
return Response(sdp_answer, status_code = HTTP_201_CREATED, media_type = 'application/sdp', headers = headers)
return Response(status_code = HTTP_404_NOT_FOUND)
async def delete_stream(request : Request) -> Response:
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
if session_id and destroy_stream(session_id):
return Response(status_code = HTTP_200_OK)
return Response(status_code = HTTP_404_NOT_FOUND)
+15
View File
@@ -0,0 +1,15 @@
from facefusion.types import Locales
LOCALES : Locales =\
{
'en':
{
'ok': 'ok',
'something_went_wrong': 'something went wrong',
'invalid_access_token': 'invalid access token',
'invalid_refresh_token': 'invalid refresh token',
'source_asset_not_found': 'source asset not found',
'target_asset_not_found': 'target asset not found',
'invalid_state_key': 'invalid state key'
}
}
+34
View File
@@ -0,0 +1,34 @@
from starlette.responses import JSONResponse
from starlette.status import HTTP_401_UNAUTHORIZED, HTTP_426_UPGRADE_REQUIRED
from starlette.types import ASGIApp, Receive, Scope, Send
from facefusion import session_manager, translator
from facefusion.apis.session_helper import extract_access_token
def create_session_guard(app : ASGIApp) -> ASGIApp:
async def middleware(scope : Scope, receive : Receive, send : Send) -> None:
access_token = extract_access_token(scope)
if access_token:
session_id = session_manager.find_session_id(access_token)
if session_id:
if session_manager.validate_session(session_id):
return await app(scope, receive, send)
response = JSONResponse(
{
'message': translator.get('invalid_access_token', 'facefusion.apis')
}, status_code = HTTP_426_UPGRADE_REQUIRED)
return await response(scope, receive, send)
response = JSONResponse(
{
'message': translator.get('invalid_access_token', 'facefusion.apis')
}, status_code = HTTP_401_UNAUTHORIZED)
return await response(scope, receive, send)
return middleware
+29
View File
@@ -0,0 +1,29 @@
from typing import Optional
from starlette.datastructures import Headers
from starlette.types import Scope
from facefusion.apis.api_helper import get_sec_websocket_protocol
from facefusion.types import Token
def extract_access_token(scope : Scope) -> Optional[Token]:
if scope.get('type') == 'http':
auth_header = Headers(scope = scope).get('Authorization')
if auth_header:
auth_prefix, _, access_token = auth_header.partition(' ')
if auth_prefix.lower() == 'bearer' and access_token:
return access_token
if scope.get('type') == 'websocket':
subprotocol = get_sec_websocket_protocol(scope)
if subprotocol:
protocol_prefix, _, access_token = subprotocol.partition('.')
if protocol_prefix == 'access_token' and access_token:
return access_token
return None
+67
View File
@@ -0,0 +1,67 @@
from functools import partial
from queue import Queue
from typing import Optional, Tuple
import numpy
from facefusion import rtc
from facefusion.apis.stream_event import create_receive_event
from facefusion.codecs import opus_decoder, opus_encoder
from facefusion.types import AudioCodec, AudioFrame, Buffer, OpusDecoder, RtcPeer, RtcPeerAudio, Time
def run_audio_encode_loop(rtc_peer : RtcPeer, audio_queue : Queue[Tuple[Time, AudioFrame]]) -> None:
audio_codec = rtc_peer.get('audio').get('codec')
temp_audio_time, temp_audio_frame = audio_queue.get()
audio_encoder = opus_encoder.create(48000, 2)
while numpy.any(temp_audio_frame):
audio_buffer = opus_encoder.encode(audio_encoder, temp_audio_frame.tobytes(), 2)
if audio_buffer:
audio_timestamp = rtc.convert_time_to_timestamp(audio_codec, temp_audio_time)
rtc.send_audio(rtc_peer, audio_buffer, audio_timestamp)
temp_audio_time, temp_audio_frame = audio_queue.get()
opus_encoder.destroy(audio_encoder)
def receive_audio_frames(rtc_peer_audio : RtcPeerAudio, audio_queue : Queue[Tuple[Time, AudioFrame]]) -> None:
audio_track = rtc_peer_audio.get('receiver_track')
audio_codec = rtc_peer_audio.get('codec')
audio_decoder = create_audio_decoder(audio_codec)
audio_frame_handler = partial(handle_audio_frame, audio_codec, audio_decoder, audio_queue)
receive_event = create_receive_event(audio_track, audio_frame_handler)
receive_event.wait()
empty_audio_frame = numpy.empty(0)
audio_queue.put((0.0, empty_audio_frame))
destroy_audio_decoder(audio_codec, audio_decoder)
def decode_audio_frame(audio_codec : AudioCodec, audio_decoder : OpusDecoder, input_buffer : Buffer) -> Optional[Buffer]:
if audio_codec == 'opus':
return opus_decoder.decode(audio_decoder, input_buffer, 2)
return None
def create_audio_decoder(audio_codec : AudioCodec) -> Optional[OpusDecoder]:
if audio_codec == 'opus':
return opus_decoder.create(48000, 2)
return None
def destroy_audio_decoder(audio_codec : AudioCodec, audio_decoder : OpusDecoder) -> None:
if audio_codec == 'opus':
opus_decoder.destroy(audio_decoder)
def handle_audio_frame(audio_codec : AudioCodec, audio_decoder : OpusDecoder, audio_queue : Queue[Tuple[Time, AudioFrame]], audio_buffer : Buffer, audio_timestamp : int) -> None:
audio_frame = decode_audio_frame(audio_codec, audio_decoder, audio_buffer)
if audio_frame:
audio_frame = numpy.frombuffer(audio_frame, dtype = numpy.float32)
audio_time = rtc.convert_timestamp_to_time(audio_codec, audio_timestamp)
audio_queue.put((audio_time, audio_frame))
+30
View File
@@ -0,0 +1,30 @@
import ctypes
import threading
from functools import partial
from facefusion.libraries import datachannel as datachannel_module
from facefusion.types import FrameHandler
def create_receive_event(track : int, frame_handler : FrameHandler) -> threading.Event:
datachannel_library = datachannel_module.create_static_library()
receive_event = threading.Event()
frame_callback = ctypes.CFUNCTYPE(None, ctypes.c_int, ctypes.c_void_p, ctypes.c_int, ctypes.c_void_p, ctypes.c_void_p)(partial(dispatch_frame, frame_handler))
close_callback = ctypes.CFUNCTYPE(None, ctypes.c_int, ctypes.c_void_p)(partial(dispatch_event, receive_event))
datachannel_library.rtcSetFrameCallback(track, frame_callback)
datachannel_library.rtcSetClosedCallback(track, close_callback)
receive_event.frame_callback = frame_callback # type: ignore[attr-defined]
receive_event.close_callback = close_callback # type: ignore[attr-defined]
return receive_event
def dispatch_frame(frame_handler : FrameHandler, track : int, data : ctypes.c_void_p, size : int, info : ctypes.c_void_p, pointer : ctypes.c_void_p) -> None:
frame_buffer = ctypes.string_at(data, size)
frame_timestamp = ctypes.cast(info, ctypes.POINTER(ctypes.c_uint32)).contents.value
frame_handler(frame_buffer, frame_timestamp)
def dispatch_event(event : threading.Event, track : int, pointer : ctypes.c_void_p) -> None:
event.set()
+136
View File
@@ -0,0 +1,136 @@
import ctypes
import threading
from collections.abc import AsyncIterator
from concurrent.futures import Future, ThreadPoolExecutor
from queue import Queue
from typing import Optional, Tuple
import cv2
import numpy
from starlette.websockets import WebSocket
from facefusion import rtc, rtc_store, state_manager, streamer
from facefusion.apis.stream_audio import receive_audio_frames, run_audio_encode_loop
from facefusion.apis.stream_video import receive_video_frames, run_video_encode_loop
from facefusion.libraries import datachannel as datachannel_module
from facefusion.types import AudioCodec, AudioFrame, BufferPack, PeerConnection, RtcPeer, RtcPeerAudio, SdpAnswer, SdpOffer, SessionId, Time, VideoCodec, VisionFrame
async def process_image(websocket : WebSocket) -> None:
capture_vision_frame = await anext(receive_vision_frames(websocket), None)
if numpy.any(capture_vision_frame):
output_vision_frame = streamer.process_stream_frame(capture_vision_frame)
is_success, output_frame_buffer = cv2.imencode('.jpg', output_vision_frame)
if is_success:
await websocket.send_bytes(output_frame_buffer.tobytes())
async def receive_vision_frames(websocket : WebSocket) -> AsyncIterator[VisionFrame]:
websocket_event = await websocket.receive()
while websocket_event.get('type') == 'websocket.receive':
frame_buffer = websocket_event.get('bytes') or bytes()
vision_frame = cv2.imdecode(numpy.frombuffer(frame_buffer, numpy.uint8), cv2.IMREAD_COLOR)
if numpy.any(vision_frame):
yield vision_frame
websocket_event = await websocket.receive()
def process_video(session_id : SessionId, sdp_offer : SdpOffer) -> Optional[SdpAnswer]:
video_codec : VideoCodec = 'vp8'
if rtc.get_payload_type(sdp_offer, 'vp9'):
video_codec = 'vp9'
if rtc.get_payload_type(sdp_offer, 'av1'):
video_codec = 'av1'
video_payload_type = rtc.get_payload_type(sdp_offer, video_codec)
if video_payload_type:
peer_connection : PeerConnection = rtc.create_peer_connection()
video_receiver_track = rtc.add_video_track(peer_connection, 'recvonly', video_codec, video_payload_type)
video_sender_track = rtc.add_video_track(peer_connection, 'sendonly', video_codec, video_payload_type)
sender_bitrate = ctypes.c_uint(0)
receiver_bitrate = ctypes.c_uint(8000)
rtc.wire_sender_bitrate(video_sender_track, sender_bitrate)
audio_codec : AudioCodec = 'opus'
audio_payload_type = rtc.get_payload_type(sdp_offer, audio_codec)
if audio_payload_type:
audio_receiver_track = rtc.add_audio_track(peer_connection, 'recvonly', audio_codec, audio_payload_type)
audio_sender_track = rtc.add_audio_track(peer_connection, 'sendonly', audio_codec, audio_payload_type)
rtc.set_remote_description(peer_connection, sdp_offer)
sdp_answer = rtc.create_sdp_answer(peer_connection)
if sdp_answer:
rtc_peer : RtcPeer =\
{
'peer_connection': peer_connection,
'video':
{
'sender_track': video_sender_track,
'receiver_track': video_receiver_track,
'codec': video_codec
},
'sender_bitrate': sender_bitrate,
'receiver_bitrate': receiver_bitrate
}
if audio_payload_type:
rtc_peer['audio'] = RtcPeerAudio(
sender_track = audio_sender_track,
receiver_track = audio_receiver_track,
codec = audio_codec
)
rtc_store.init_peers(session_id)
rtc_store.get_peers(session_id).append(rtc_peer)
threading.Thread(target = run_peer_loop, args = (session_id, rtc_peer), daemon = True).start()
return sdp_answer
datachannel_module.create_static_library().rtcDeletePeerConnection(peer_connection)
return None
def run_peer_loop(session_id : SessionId, rtc_peer : RtcPeer) -> None:
execution_thread_count = state_manager.get_item('execution_thread_count')
video_queue : Queue[Tuple[Time, Future[BufferPack]]] = Queue(maxsize = execution_thread_count)
audio_queue : Queue[Tuple[Time, AudioFrame]] = Queue(maxsize = execution_thread_count * 10)
video_executor = ThreadPoolExecutor(max_workers = execution_thread_count)
video_receiver_thread = threading.Thread(target = receive_video_frames, args = (rtc_peer.get('video'), video_queue, video_executor), daemon = True)
video_encoder_thread = threading.Thread(target = run_video_encode_loop, args = (rtc_peer, video_queue), daemon = True)
video_receiver_thread.start()
video_encoder_thread.start()
if rtc_peer.get('audio'):
audio_receiver_thread = threading.Thread(target = receive_audio_frames, args = (rtc_peer.get('audio'), audio_queue), daemon = True)
audio_encoder_thread = threading.Thread(target = run_audio_encode_loop, args = (rtc_peer, audio_queue), daemon = True)
audio_receiver_thread.start()
audio_encoder_thread.start()
audio_receiver_thread.join()
audio_encoder_thread.join()
video_receiver_thread.join()
video_encoder_thread.join()
video_executor.shutdown(wait = True)
rtc_store.delete_peers(session_id)
def destroy_stream(session_id : SessionId) -> bool:
if rtc_store.has_peers(session_id):
rtc_store.delete_peers(session_id)
return not rtc_store.has_peers(session_id)
return False
+174
View File
@@ -0,0 +1,174 @@
from concurrent.futures import Future, ThreadPoolExecutor
from functools import partial
from queue import Queue
from typing import Optional, Tuple
import cv2
import numpy
from facefusion import rtc, streamer
from facefusion.apis.stream_event import create_receive_event
from facefusion.codecs import aom_decoder, aom_encoder, vpx_decoder, vpx_encoder
from facefusion.types import AomDecoder, AomEncoder, BitRate, Buffer, BufferPack, Resolution, RtcPeer, RtcPeerVideo, Time, VideoCodec, VisionFrame, VpxDecoder, VpxEncoder
def run_video_encode_loop(rtc_peer : RtcPeer, video_queue : Queue[Tuple[Time, Future[BufferPack]]]) -> None:
video_codec = rtc_peer.get('video').get('codec')
video_time, video_future = video_queue.get()
video_pack = video_future.result()
video_buffer = video_pack.get('buffer')
video_resolution = video_pack.get('resolution')
if video_buffer:
temp_resolution : Resolution = video_resolution
temp_bitrate : BitRate = 8000
video_encoder = create_video_encoder(video_codec, temp_resolution, temp_bitrate)
frame_index = 0
while video_buffer:
sender_bitrate = rtc_peer.get('sender_bitrate').value
if video_resolution[0] - temp_resolution[0] or video_resolution[1] - temp_resolution[1]:
temp_resolution = video_resolution
update_video_encoder_resolution(video_codec, video_encoder, temp_resolution)
if sender_bitrate > 0 and sender_bitrate - temp_bitrate:
temp_bitrate = sender_bitrate
update_video_encoder_bitrate(video_codec, video_encoder, temp_bitrate)
__video_buffer__ = encode_video_frame(video_codec, video_encoder, video_buffer, temp_resolution, frame_index)
if __video_buffer__:
video_timestamp = rtc.convert_time_to_timestamp(video_codec, video_time)
rtc.send_video(rtc_peer, __video_buffer__, video_timestamp)
receiver_bitrate = rtc_peer.get('receiver_bitrate').value
rtc.adapt_receiver_bitrate(rtc_peer, receiver_bitrate)
frame_index += 1
video_time, video_future = video_queue.get()
video_pack = video_future.result()
video_buffer = video_pack.get('buffer')
video_resolution = video_pack.get('resolution')
destroy_video_encoder(video_codec, video_encoder)
rtc.clear_bitrate(rtc_peer)
def receive_video_frames(rtc_peer_video : RtcPeerVideo, video_queue : Queue[Tuple[Time, Future[BufferPack]]], video_executor : ThreadPoolExecutor) -> None:
video_track = rtc_peer_video.get('receiver_track')
video_codec = rtc_peer_video.get('codec')
video_decoder = create_video_decoder(video_codec)
video_frame_handler = partial(handle_video_frame, video_codec, video_decoder, video_queue, video_executor)
receive_event = create_receive_event(video_track, video_frame_handler)
receive_event.wait()
empty_future : Future[BufferPack] = Future()
empty_future.set_result(BufferPack(buffer = bytes(), resolution = (0, 0)))
video_queue.put((0.0, empty_future))
destroy_video_decoder(video_codec, video_decoder)
def process_video_frame(input_vision_frame : VisionFrame) -> BufferPack:
output_vision_frame = streamer.process_stream_frame(input_vision_frame)
output_resolution : Resolution = (output_vision_frame.shape[1], output_vision_frame.shape[0])
output_buffer = cv2.cvtColor(output_vision_frame, cv2.COLOR_BGR2YUV_I420).tobytes()
return BufferPack(buffer = output_buffer, resolution = output_resolution)
def decode_video_frame(video_codec : VideoCodec, video_decoder : VpxDecoder | AomDecoder, input_buffer : Buffer) -> Optional[VisionFrame]:
if video_codec == 'av1':
aom_pointer = aom_decoder.decode(video_decoder, input_buffer)
if aom_pointer:
return normalize_vision_frame(aom_pointer)
if video_codec in [ 'vp8', 'vp9' ]:
vpx_pointer = vpx_decoder.decode(video_decoder, input_buffer)
if vpx_pointer:
return normalize_vision_frame(vpx_pointer)
return None
def encode_video_frame(video_codec : VideoCodec, video_encoder : VpxEncoder | AomEncoder, input_buffer : Buffer, frame_resolution : Resolution, frame_index : int) -> Buffer:
if video_codec == 'av1':
return aom_encoder.encode(video_encoder, input_buffer, frame_resolution, frame_index)
if video_codec in [ 'vp8', 'vp9' ]:
return vpx_encoder.encode(video_encoder, input_buffer, frame_resolution, frame_index)
return bytes()
def normalize_vision_frame(frame_pointer : BufferPack) -> VisionFrame:
frame_width, frame_height = frame_pointer.get('resolution')
vision_frame = numpy.frombuffer(frame_pointer.get('buffer'), dtype = numpy.uint8).reshape((frame_height * 3 // 2, frame_width))
return cv2.cvtColor(vision_frame, cv2.COLOR_YUV2BGR_I420)
def create_video_decoder(video_codec : VideoCodec) -> Optional[VpxDecoder | AomDecoder]:
if video_codec == 'av1':
return aom_decoder.create(8)
if video_codec in [ 'vp8', 'vp9' ]:
return vpx_decoder.create(video_codec, 8)
return None
def create_video_encoder(video_codec : VideoCodec, frame_resolution : Resolution, bitrate : BitRate) -> Optional[VpxEncoder | AomEncoder]:
if video_codec == 'av1':
return aom_encoder.create(frame_resolution, bitrate, 8, 10)
if video_codec in [ 'vp8', 'vp9' ]:
return vpx_encoder.create(video_codec, frame_resolution, bitrate, 8, 10)
return None
def destroy_video_decoder(video_codec : VideoCodec, video_decoder : VpxDecoder | AomDecoder) -> None:
if video_codec == 'av1':
aom_decoder.destroy(video_decoder)
if video_codec in [ 'vp8', 'vp9' ]:
vpx_decoder.destroy(video_decoder)
def destroy_video_encoder(video_codec : VideoCodec, video_encoder : VpxEncoder | AomEncoder) -> None:
if video_codec == 'av1':
aom_encoder.destroy(video_encoder)
if video_codec in [ 'vp8', 'vp9' ]:
vpx_encoder.destroy(video_encoder)
def update_video_encoder_resolution(video_codec : VideoCodec, video_encoder : VpxEncoder | AomEncoder, frame_resolution : Resolution) -> bool:
if video_codec == 'av1':
return aom_encoder.update_resolution(video_encoder, frame_resolution)
if video_codec in [ 'vp8', 'vp9' ]:
return vpx_encoder.update_resolution(video_encoder, frame_resolution)
return False
def update_video_encoder_bitrate(video_codec : VideoCodec, video_encoder : VpxEncoder | AomEncoder, bitrate : BitRate) -> bool:
if video_codec == 'av1':
return aom_encoder.update_bitrate(video_encoder, bitrate)
if video_codec in [ 'vp8', 'vp9' ]:
return vpx_encoder.update_bitrate(video_encoder, bitrate)
return False
def handle_video_frame(video_codec : VideoCodec, video_decoder : VpxDecoder | AomDecoder, video_queue : Queue[Tuple[Time, Future[BufferPack]]], video_executor : ThreadPoolExecutor, video_buffer : Buffer, video_timestamp : int) -> None:
vision_frame = decode_video_frame(video_codec, video_decoder, video_buffer)
if numpy.any(vision_frame) and video_queue.qsize() < video_queue.maxsize:
video_future = video_executor.submit(process_video_frame, vision_frame)
video_time = rtc.convert_timestamp_to_time(video_codec, video_timestamp)
video_queue.put((video_time, video_future))
+6 -4
View File
@@ -1,16 +1,18 @@
import os import os
import sys import sys
from facefusion.typing import AppContext from facefusion.types import AppContext
def detect_app_context() -> AppContext: def detect_app_context() -> AppContext:
jobs_path = os.path.join('facefusion', 'jobs')
apis_path = os.path.join('facefusion', 'apis')
frame = sys._getframe(1) frame = sys._getframe(1)
while frame: while frame:
if os.path.join('facefusion', 'jobs') in frame.f_code.co_filename: if jobs_path in frame.f_code.co_filename:
return 'cli' return 'cli'
if os.path.join('facefusion', 'uis') in frame.f_code.co_filename: if apis_path in frame.f_code.co_filename:
return 'ui' return 'api'
frame = frame.f_back frame = frame.f_back
return 'cli' return 'cli'
@@ -1,66 +1,32 @@
from facefusion import state_manager from typing import Union
from facefusion.filesystem import is_image, is_video, list_directory
from facefusion.jobs import job_store from facefusion.capability_store import get_api_arguments, get_cli_arguments, get_sys_arguments
from facefusion.normalizer import normalize_fps, normalize_padding from facefusion.filesystem import get_file_name, is_video, resolve_file_paths
from facefusion.normalizer import normalize_fps, normalize_space
from facefusion.processors.core import get_processors_modules from facefusion.processors.core import get_processors_modules
from facefusion.typing import ApplyStateItem, Args from facefusion.processors.types import ProcessorState
from facefusion.vision import create_image_resolutions, create_video_resolutions, detect_image_resolution, detect_video_fps, detect_video_resolution, pack_resolution from facefusion.types import ApplyStateItem, Args, State
from facefusion.vision import detect_video_fps
def reduce_step_args(args : Args) -> Args:
step_args =\
{
key: args[key] for key in args if key in job_store.get_step_keys()
}
return step_args
def reduce_job_args(args : Args) -> Args:
job_args =\
{
key: args[key] for key in args if key in job_store.get_job_keys()
}
return job_args
def collect_step_args() -> Args:
step_args =\
{
key: state_manager.get_item(key) for key in job_store.get_step_keys() #type:ignore[arg-type]
}
return step_args
def collect_job_args() -> Args:
job_args =\
{
key: state_manager.get_item(key) for key in job_store.get_job_keys() #type:ignore[arg-type]
}
return job_args
def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None: def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
# general
apply_state_item('command', args.get('command')) apply_state_item('command', args.get('command'))
# paths apply_state_item('workflow', args.get('workflow'))
apply_state_item('temp_path', args.get('temp_path')) apply_state_item('temp_path', args.get('temp_path'))
apply_state_item('jobs_path', args.get('jobs_path')) apply_state_item('jobs_path', args.get('jobs_path'))
apply_state_item('source_paths', args.get('source_paths')) apply_state_item('source_paths', args.get('source_paths'))
apply_state_item('target_path', args.get('target_path')) apply_state_item('target_path', args.get('target_path'))
apply_state_item('output_path', args.get('output_path')) apply_state_item('output_path', args.get('output_path'))
# patterns
apply_state_item('source_pattern', args.get('source_pattern')) apply_state_item('source_pattern', args.get('source_pattern'))
apply_state_item('target_pattern', args.get('target_pattern')) apply_state_item('target_pattern', args.get('target_pattern'))
apply_state_item('output_pattern', args.get('output_pattern')) apply_state_item('output_pattern', args.get('output_pattern'))
# face detector
apply_state_item('face_detector_model', args.get('face_detector_model')) apply_state_item('face_detector_model', args.get('face_detector_model'))
apply_state_item('face_detector_size', args.get('face_detector_size')) apply_state_item('face_detector_size', args.get('face_detector_size'))
apply_state_item('face_detector_margin', normalize_space(args.get('face_detector_margin')))
apply_state_item('face_detector_angles', args.get('face_detector_angles')) apply_state_item('face_detector_angles', args.get('face_detector_angles'))
apply_state_item('face_detector_score', args.get('face_detector_score')) apply_state_item('face_detector_score', args.get('face_detector_score'))
# face landmarker
apply_state_item('face_landmarker_model', args.get('face_landmarker_model')) apply_state_item('face_landmarker_model', args.get('face_landmarker_model'))
apply_state_item('face_landmarker_score', args.get('face_landmarker_score')) apply_state_item('face_landmarker_score', args.get('face_landmarker_score'))
# face selector
apply_state_item('face_selector_mode', args.get('face_selector_mode')) apply_state_item('face_selector_mode', args.get('face_selector_mode'))
apply_state_item('face_selector_order', args.get('face_selector_order')) apply_state_item('face_selector_order', args.get('face_selector_order'))
apply_state_item('face_selector_age_start', args.get('face_selector_age_start')) apply_state_item('face_selector_age_start', args.get('face_selector_age_start'))
@@ -70,65 +36,94 @@ def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('reference_face_position', args.get('reference_face_position')) apply_state_item('reference_face_position', args.get('reference_face_position'))
apply_state_item('reference_face_distance', args.get('reference_face_distance')) apply_state_item('reference_face_distance', args.get('reference_face_distance'))
apply_state_item('reference_frame_number', args.get('reference_frame_number')) apply_state_item('reference_frame_number', args.get('reference_frame_number'))
# face masker apply_state_item('face_tracker_score', args.get('face_tracker_score'))
apply_state_item('face_occluder_model', args.get('face_occluder_model')) apply_state_item('face_occluder_model', args.get('face_occluder_model'))
apply_state_item('face_parser_model', args.get('face_parser_model')) apply_state_item('face_parser_model', args.get('face_parser_model'))
apply_state_item('face_mask_types', args.get('face_mask_types')) apply_state_item('face_mask_types', args.get('face_mask_types'))
apply_state_item('face_mask_blur', args.get('face_mask_blur')) apply_state_item('face_mask_areas', args.get('face_mask_areas'))
apply_state_item('face_mask_padding', normalize_padding(args.get('face_mask_padding')))
apply_state_item('face_mask_regions', args.get('face_mask_regions')) apply_state_item('face_mask_regions', args.get('face_mask_regions'))
# frame extraction apply_state_item('face_mask_blur', args.get('face_mask_blur'))
apply_state_item('face_mask_padding', normalize_space(args.get('face_mask_padding')))
apply_state_item('voice_extractor_model', args.get('voice_extractor_model'))
apply_state_item('trim_frame_start', args.get('trim_frame_start')) apply_state_item('trim_frame_start', args.get('trim_frame_start'))
apply_state_item('trim_frame_end', args.get('trim_frame_end')) apply_state_item('trim_frame_end', args.get('trim_frame_end'))
apply_state_item('temp_frame_format', args.get('temp_frame_format')) apply_state_item('temp_frame_format', args.get('temp_frame_format'))
apply_state_item('keep_temp', args.get('keep_temp')) apply_state_item('target_frame_amount', args.get('target_frame_amount'))
# output creation
apply_state_item('output_image_quality', args.get('output_image_quality')) apply_state_item('output_image_quality', args.get('output_image_quality'))
if is_image(args.get('target_path')): apply_state_item('output_image_scale', args.get('output_image_scale'))
output_image_resolution = detect_image_resolution(args.get('target_path'))
output_image_resolutions = create_image_resolutions(output_image_resolution)
if args.get('output_image_resolution') in output_image_resolutions:
apply_state_item('output_image_resolution', args.get('output_image_resolution'))
else:
apply_state_item('output_image_resolution', pack_resolution(output_image_resolution))
apply_state_item('output_audio_encoder', args.get('output_audio_encoder')) apply_state_item('output_audio_encoder', args.get('output_audio_encoder'))
apply_state_item('output_audio_quality', args.get('output_audio_quality'))
apply_state_item('output_audio_volume', args.get('output_audio_volume'))
apply_state_item('output_audio_fps', normalize_fps(args.get('output_audio_fps')))
apply_state_item('output_video_encoder', args.get('output_video_encoder')) apply_state_item('output_video_encoder', args.get('output_video_encoder'))
apply_state_item('output_video_preset', args.get('output_video_preset')) apply_state_item('output_video_preset', args.get('output_video_preset'))
apply_state_item('output_video_quality', args.get('output_video_quality')) apply_state_item('output_video_quality', args.get('output_video_quality'))
if is_video(args.get('target_path')): apply_state_item('output_video_scale', args.get('output_video_scale'))
output_video_resolution = detect_video_resolution(args.get('target_path'))
output_video_resolutions = create_video_resolutions(output_video_resolution)
if args.get('output_video_resolution') in output_video_resolutions:
apply_state_item('output_video_resolution', args.get('output_video_resolution'))
else:
apply_state_item('output_video_resolution', pack_resolution(output_video_resolution))
if args.get('output_video_fps') or is_video(args.get('target_path')): if args.get('output_video_fps') or is_video(args.get('target_path')):
output_video_fps = normalize_fps(args.get('output_video_fps')) or detect_video_fps(args.get('target_path')) output_video_fps = normalize_fps(args.get('output_video_fps')) or detect_video_fps(args.get('target_path'))
apply_state_item('output_video_fps', output_video_fps) apply_state_item('output_video_fps', output_video_fps)
apply_state_item('skip_audio', args.get('skip_audio'))
# processors available_processors = [ get_file_name(file_path) for file_path in resolve_file_paths('facefusion/processors/modules') ]
available_processors = [ file.get('name') for file in list_directory('facefusion/processors/modules') ]
apply_state_item('processors', args.get('processors')) apply_state_item('processors', args.get('processors'))
for processor_module in get_processors_modules(available_processors): for processor_module in get_processors_modules(available_processors):
processor_module.apply_args(args, apply_state_item) processor_module.apply_args(args, apply_state_item)
# uis
apply_state_item('open_browser', args.get('open_browser')) apply_state_item('execution_device_ids', args.get('execution_device_ids'))
apply_state_item('ui_layouts', args.get('ui_layouts'))
apply_state_item('ui_workflow', args.get('ui_workflow'))
# execution
apply_state_item('execution_device_id', args.get('execution_device_id'))
apply_state_item('execution_providers', args.get('execution_providers')) apply_state_item('execution_providers', args.get('execution_providers'))
apply_state_item('execution_thread_count', args.get('execution_thread_count')) apply_state_item('execution_thread_count', args.get('execution_thread_count'))
apply_state_item('execution_queue_count', args.get('execution_queue_count'))
# download
apply_state_item('download_providers', args.get('download_providers')) apply_state_item('download_providers', args.get('download_providers'))
apply_state_item('download_scope', args.get('download_scope')) apply_state_item('download_scope', args.get('download_scope'))
# memory apply_state_item('benchmark_mode', args.get('benchmark_mode'))
apply_state_item('benchmark_resolutions', args.get('benchmark_resolutions'))
apply_state_item('benchmark_cycle_count', args.get('benchmark_cycle_count'))
apply_state_item('api_host', args.get('api_host'))
apply_state_item('api_port', args.get('api_port'))
apply_state_item('api_security_strategy', args.get('api_security_strategy'))
apply_state_item('video_memory_strategy', args.get('video_memory_strategy')) apply_state_item('video_memory_strategy', args.get('video_memory_strategy'))
apply_state_item('system_memory_limit', args.get('system_memory_limit'))
# misc
apply_state_item('log_level', args.get('log_level')) apply_state_item('log_level', args.get('log_level'))
# jobs apply_state_item('halt_on_error', args.get('halt_on_error'))
apply_state_item('job_id', args.get('job_id')) apply_state_item('job_id', args.get('job_id'))
apply_state_item('job_status', args.get('job_status')) apply_state_item('job_status', args.get('job_status'))
apply_state_item('step_index', args.get('step_index')) apply_state_item('step_index', args.get('step_index'))
def extract_api_args(state : Union[State, ProcessorState]) -> Args:
api_args =\
{
key: state.get(key) for key in state if key in get_api_arguments()
}
return api_args
def extract_cli_args(state : Union[State, ProcessorState]) -> Args:
cli_args =\
{
key: state.get(key) for key in state if key in get_cli_arguments()
}
return cli_args
def extract_sys_args(state : Union[State, ProcessorState]) -> Args:
sys_args =\
{
key: state.get(key) for key in state if key in get_sys_arguments()
}
return sys_args
def extract_step_args(state : Union[State, ProcessorState]) -> Args:
step_args =\
{
key: state.get(key) for key in state if key in get_cli_arguments() and key not in get_sys_arguments()
}
return step_args
def filter_step_args(args : Args) -> Args:
step_args =\
{
key: args.get(key) for key in args if key in get_cli_arguments() and key not in get_sys_arguments()
}
return step_args
+67 -35
View File
@@ -1,27 +1,30 @@
import math
from functools import lru_cache from functools import lru_cache
from typing import Any, List, Optional from typing import Any, List, Optional, Tuple
import numpy import numpy
import scipy import scipy
from numpy._typing import NDArray from numpy.typing import NDArray
from facefusion.ffmpeg import read_audio_buffer from facefusion.ffmpeg import read_audio_buffer
from facefusion.filesystem import is_audio from facefusion.filesystem import is_audio
from facefusion.typing import Audio, AudioFrame, Fps, Mel, MelFilterBank, Spectrogram from facefusion.media_helper import restrict_trim_frame
from facefusion.types import Audio, AudioFrame, Duration, Fps, Mel, MelFilterBank, Spectrogram
from facefusion.voice_extractor import batch_extract_voice from facefusion.voice_extractor import batch_extract_voice
@lru_cache(maxsize = 128) @lru_cache(maxsize = 64)
def read_static_audio(audio_path : str, fps : Fps) -> Optional[List[AudioFrame]]: def read_static_audio(audio_path : str, fps : Fps) -> Optional[List[AudioFrame]]:
return read_audio(audio_path, fps) return read_audio(audio_path, fps)
def read_audio(audio_path : str, fps : Fps) -> Optional[List[AudioFrame]]: def read_audio(audio_path : str, fps : Fps) -> Optional[List[AudioFrame]]:
sample_rate = 48000 audio_sample_rate = 48000
channel_total = 2 audio_sample_size = 16
audio_channel_total = 2
if is_audio(audio_path): if is_audio(audio_path):
audio_buffer = read_audio_buffer(audio_path, sample_rate, channel_total) audio_buffer = read_audio_buffer(audio_path, audio_sample_rate, audio_sample_size, audio_channel_total)
audio = numpy.frombuffer(audio_buffer, dtype = numpy.int16).reshape(-1, 2) audio = numpy.frombuffer(audio_buffer, dtype = numpy.int16).reshape(-1, 2)
audio = prepare_audio(audio) audio = prepare_audio(audio)
spectrogram = create_spectrogram(audio) spectrogram = create_spectrogram(audio)
@@ -30,21 +33,22 @@ def read_audio(audio_path : str, fps : Fps) -> Optional[List[AudioFrame]]:
return None return None
@lru_cache(maxsize = 128) @lru_cache(maxsize = 64)
def read_static_voice(audio_path : str, fps : Fps) -> Optional[List[AudioFrame]]: def read_static_voice(audio_path : str, fps : Fps) -> Optional[List[AudioFrame]]:
return read_voice(audio_path, fps) return read_voice(audio_path, fps)
def read_voice(audio_path : str, fps : Fps) -> Optional[List[AudioFrame]]: def read_voice(audio_path : str, fps : Fps) -> Optional[List[AudioFrame]]:
sample_rate = 48000 voice_sample_rate = 48000
channel_total = 2 voice_sample_size = 16
chunk_size = 240 * 1024 voice_channel_total = 2
step_size = 180 * 1024 voice_chunk_size = 240 * 1024
voice_step_size = 180 * 1024
if is_audio(audio_path): if is_audio(audio_path):
audio_buffer = read_audio_buffer(audio_path, sample_rate, channel_total) audio_buffer = read_audio_buffer(audio_path, voice_sample_rate, voice_sample_size, voice_channel_total)
audio = numpy.frombuffer(audio_buffer, dtype = numpy.int16).reshape(-1, 2) audio = numpy.frombuffer(audio_buffer, dtype = numpy.int16).reshape(-1, 2)
audio = batch_extract_voice(audio, chunk_size, step_size) audio = batch_extract_voice(audio, voice_chunk_size, voice_step_size)
audio = prepare_voice(audio) audio = prepare_voice(audio)
spectrogram = create_spectrogram(audio) spectrogram = create_spectrogram(audio)
audio_frames = extract_audio_frames(spectrogram, fps) audio_frames = extract_audio_frames(spectrogram, fps)
@@ -60,6 +64,20 @@ def get_audio_frame(audio_path : str, fps : Fps, frame_number : int = 0) -> Opti
return None return None
def extract_audio_frames(spectrogram : Spectrogram, fps : Fps) -> List[AudioFrame]:
audio_frames = []
mel_filter_total = 80
audio_step_size = 16
indices = numpy.arange(0, spectrogram.shape[1], mel_filter_total / fps).astype(numpy.int16)
indices = indices[indices >= audio_step_size]
for index in indices:
start = max(0, index - audio_step_size)
audio_frames.append(spectrogram[:, start:index])
return audio_frames
def get_voice_frame(audio_path : str, fps : Fps, frame_number : int = 0) -> Optional[AudioFrame]: def get_voice_frame(audio_path : str, fps : Fps, frame_number : int = 0) -> Optional[AudioFrame]:
if is_audio(audio_path): if is_audio(audio_path):
voice_frames = read_static_voice(audio_path, fps) voice_frames = read_static_voice(audio_path, fps)
@@ -70,8 +88,8 @@ def get_voice_frame(audio_path : str, fps : Fps, frame_number : int = 0) -> Opti
def create_empty_audio_frame() -> AudioFrame: def create_empty_audio_frame() -> AudioFrame:
mel_filter_total = 80 mel_filter_total = 80
step_size = 16 audio_step_size = 16
audio_frame = numpy.zeros((mel_filter_total, step_size)).astype(numpy.int16) audio_frame = numpy.zeros((mel_filter_total, audio_step_size)).astype(numpy.int16)
return audio_frame return audio_frame
@@ -84,10 +102,10 @@ def prepare_audio(audio : Audio) -> Audio:
def prepare_voice(audio : Audio) -> Audio: def prepare_voice(audio : Audio) -> Audio:
sample_rate = 48000 audio_sample_rate = 48000
resample_rate = 16000 audio_resample_rate = 16000
audio_resample_factor = round(len(audio) * audio_resample_rate / audio_sample_rate)
audio = scipy.signal.resample(audio, int(len(audio) * resample_rate / sample_rate)) audio = scipy.signal.resample(audio, audio_resample_factor)
audio = prepare_audio(audio) audio = prepare_audio(audio)
return audio return audio
@@ -101,19 +119,20 @@ def convert_mel_to_hertz(mel : Mel) -> NDArray[Any]:
def create_mel_filter_bank() -> MelFilterBank: def create_mel_filter_bank() -> MelFilterBank:
audio_sample_rate = 16000
audio_frequency_min = 55.0
audio_frequency_max = 7600.0
mel_filter_total = 80 mel_filter_total = 80
mel_bin_total = 800 mel_bin_total = 800
sample_rate = 16000
min_frequency = 55.0
max_frequency = 7600.0
mel_filter_bank = numpy.zeros((mel_filter_total, mel_bin_total // 2 + 1)) mel_filter_bank = numpy.zeros((mel_filter_total, mel_bin_total // 2 + 1))
mel_frequency_range = numpy.linspace(convert_hertz_to_mel(min_frequency), convert_hertz_to_mel(max_frequency), mel_filter_total + 2) mel_frequency_range = numpy.linspace(convert_hertz_to_mel(audio_frequency_min), convert_hertz_to_mel(audio_frequency_max), mel_filter_total + 2)
indices = numpy.floor((mel_bin_total + 1) * convert_mel_to_hertz(mel_frequency_range) / sample_rate).astype(numpy.int16) indices = numpy.floor((mel_bin_total + 1) * convert_mel_to_hertz(mel_frequency_range) / audio_sample_rate).astype(numpy.int16)
for index in range(mel_filter_total): for index in range(mel_filter_total):
start = indices[index] start = indices[index]
end = indices[index + 1] end = indices[index + 1]
mel_filter_bank[index, start:end] = scipy.signal.windows.triang(end - start) mel_filter_bank[index, start:end] = scipy.signal.windows.triang(end - start)
return mel_filter_bank return mel_filter_bank
@@ -126,14 +145,27 @@ def create_spectrogram(audio : Audio) -> Spectrogram:
return spectrogram return spectrogram
def extract_audio_frames(spectrogram : Spectrogram, fps : Fps) -> List[AudioFrame]: def count_audio_frame_total(audio_path : str, fps : Fps) -> int:
mel_filter_total = 80 audio_duration = detect_audio_duration(audio_path)
step_size = 16 if audio_duration > 0:
audio_frames = [] return math.ceil(audio_duration * fps)
indices = numpy.arange(0, spectrogram.shape[1], mel_filter_total / fps).astype(numpy.int16) return 0
indices = indices[indices >= step_size]
for index in indices:
start = max(0, index - step_size) def detect_audio_duration(audio_path : str) -> Duration:
audio_frames.append(spectrogram[:, start:index]) audio_sample_rate = 48000
return audio_frames audio_sample_size = 16
audio_channel_total = 2
if is_audio(audio_path):
audio_buffer = read_audio_buffer(audio_path, audio_sample_rate, audio_sample_size, audio_channel_total)
if audio_buffer:
audio = numpy.frombuffer(audio_buffer, dtype = numpy.int16).reshape(-1, audio_channel_total)
audio_duration = len(audio) / audio_sample_rate
return audio_duration
return 0
def restrict_trim_audio_frame(audio_path : str, fps : Fps, trim_frame_start : Optional[int], trim_frame_end : Optional[int]) -> Tuple[int, int]:
audio_frame_total = count_audio_frame_total(audio_path, fps)
return restrict_trim_frame(audio_frame_total, trim_frame_start, trim_frame_end)
+111
View File
@@ -0,0 +1,111 @@
import hashlib
import os
import statistics
import tempfile
from time import perf_counter
from typing import Iterator, List
import facefusion.choices
from facefusion import content_analyser, core, state_manager
from facefusion.cli_helper import render_table
from facefusion.download import conditional_download, resolve_download_url
from facefusion.face_store import clear_faces
from facefusion.filesystem import get_file_extension
from facefusion.types import BenchmarkCycleSet
from facefusion.vision import count_video_frame_total, detect_video_fps
def pre_check() -> bool:
conditional_download('.assets/examples',
[
resolve_download_url('examples-3.0.0', 'source.jpg'),
resolve_download_url('examples-3.0.0', 'source.mp3'),
resolve_download_url('examples-3.0.0', 'target-240p.mp4'),
resolve_download_url('examples-3.0.0', 'target-360p.mp4'),
resolve_download_url('examples-3.0.0', 'target-540p.mp4'),
resolve_download_url('examples-3.0.0', 'target-720p.mp4'),
resolve_download_url('examples-3.0.0', 'target-1080p.mp4'),
resolve_download_url('examples-3.0.0', 'target-1440p.mp4'),
resolve_download_url('examples-3.0.0', 'target-2160p.mp4')
])
return True
def run() -> Iterator[List[BenchmarkCycleSet]]:
benchmark_resolutions = state_manager.get_item('benchmark_resolutions')
benchmark_cycle_count = state_manager.get_item('benchmark_cycle_count')
state_manager.init_item('source_paths', [ '.assets/examples/source.jpg', '.assets/examples/source.mp3' ])
state_manager.init_item('face_landmarker_score', 0)
state_manager.init_item('temp_frame_format', 'bmp')
state_manager.init_item('output_audio_volume', 0)
state_manager.init_item('output_video_preset', 'ultrafast')
state_manager.init_item('video_memory_strategy', 'tolerant')
benchmarks = []
target_paths = [ facefusion.choices.benchmark_set.get(benchmark_resolution) for benchmark_resolution in benchmark_resolutions if benchmark_resolution in facefusion.choices.benchmark_set ]
for target_path in target_paths:
state_manager.init_item('target_path', target_path)
state_manager.init_item('output_path', suggest_output_path(state_manager.get_item('target_path')))
benchmarks.append(cycle(benchmark_cycle_count))
yield benchmarks
def cycle(cycle_count : int) -> BenchmarkCycleSet:
process_times = []
video_frame_total = count_video_frame_total(state_manager.get_item('target_path'))
state_manager.init_item('output_video_fps', detect_video_fps(state_manager.get_item('target_path')))
if state_manager.get_item('benchmark_mode') == 'warm':
core.conditional_process()
for index in range(cycle_count):
if state_manager.get_item('benchmark_mode') == 'cold':
content_analyser.analyse_image.cache_clear()
content_analyser.analyse_video.cache_clear()
clear_faces()
start_time = perf_counter()
core.conditional_process()
end_time = perf_counter()
process_times.append(end_time - start_time)
average_run = round(statistics.mean(process_times), 2)
fastest_run = round(min(process_times), 2)
slowest_run = round(max(process_times), 2)
relative_fps = round(video_frame_total * cycle_count / sum(process_times), 2)
return\
{
'target_path': state_manager.get_item('target_path'),
'cycle_count': cycle_count,
'average_run': average_run,
'fastest_run': fastest_run,
'slowest_run': slowest_run,
'relative_fps': relative_fps
}
def suggest_output_path(target_path : str) -> str:
target_file_extension = get_file_extension(target_path)
return os.path.join(tempfile.gettempdir(), hashlib.sha1(target_path.encode()).hexdigest() + target_file_extension)
def render() -> None:
benchmarks = []
headers =\
[
'target_path',
'cycle_count',
'average_run',
'fastest_run',
'slowest_run',
'relative_fps'
]
for benchmark in run():
benchmarks = benchmark
contents = [ list(benchmark_set.values()) for benchmark_set in benchmarks ]
render_table(headers, contents)
+53
View File
@@ -0,0 +1,53 @@
from typing import List
import cv2
from facefusion.types import CameraPoolSet
CAMERA_POOL_SET : CameraPoolSet =\
{
'capture': {}
}
def get_local_camera_capture(camera_id : int) -> cv2.VideoCapture:
camera_key = str(camera_id)
if camera_key not in CAMERA_POOL_SET.get('capture'):
camera_capture = cv2.VideoCapture(camera_id)
if camera_capture.isOpened():
CAMERA_POOL_SET['capture'][camera_key] = camera_capture
return CAMERA_POOL_SET.get('capture').get(camera_key)
def get_remote_camera_capture(camera_url : str) -> cv2.VideoCapture:
if camera_url not in CAMERA_POOL_SET.get('capture'):
camera_capture = cv2.VideoCapture(camera_url)
if camera_capture.isOpened():
CAMERA_POOL_SET['capture'][camera_url] = camera_capture
return CAMERA_POOL_SET.get('capture').get(camera_url)
def clear_camera_pool() -> None:
for camera_capture in CAMERA_POOL_SET.get('capture').values():
camera_capture.release()
CAMERA_POOL_SET['capture'].clear()
def detect_local_camera_ids(id_start : int, id_end : int) -> List[int]:
local_camera_ids = []
for camera_id in range(id_start, id_end):
cv2.utils.logging.setLogLevel(0)
camera_capture = get_local_camera_capture(camera_id)
cv2.utils.logging.setLogLevel(3)
if camera_capture and camera_capture.isOpened():
local_camera_ids.append(camera_id)
return local_camera_ids
+65
View File
@@ -0,0 +1,65 @@
from argparse import Action
from typing import Dict, List
from facefusion.types import CapabilityGroup, CapabilitySet, CapabilityStore, Group, Scope
CAPABILITY_STORE : CapabilityStore =\
{
'api': {},
'cli': {},
'sys': {}
}
def get_api_capability_group() -> CapabilityGroup:
capability_group : CapabilityGroup = {}
for name, value in CAPABILITY_STORE.get('api').items():
for group in value.get('groups'):
capability_group.setdefault(group, {})[name] = value
return capability_group
def get_api_capability_set() -> Dict[str, CapabilitySet]:
return CAPABILITY_STORE.get('api')
def get_cli_capability_set() -> Dict[str, CapabilitySet]:
return CAPABILITY_STORE.get('cli')
def get_sys_capability_set() -> Dict[str, CapabilitySet]:
return CAPABILITY_STORE.get('sys')
def get_api_arguments() -> List[str]:
return list(get_api_capability_set().keys())
def get_cli_arguments() -> List[str]:
return list(get_cli_capability_set().keys())
def get_sys_arguments() -> List[str]:
return list(get_sys_capability_set().keys())
def register_capability_set(actions : List[Action], scopes : List[Scope], groups : List[Group]) -> None:
for action in actions:
value : CapabilitySet =\
{
'default': action.default,
'groups': groups
}
if action.choices:
value['choices'] = list(action.choices)
for scope in scopes:
if scope == 'api':
CAPABILITY_STORE['api'][action.dest] = value
if scope == 'cli':
CAPABILITY_STORE['cli'][action.dest] = value
if scope == 'sys':
CAPABILITY_STORE['sys'][action.dest] = value
+110 -36
View File
@@ -1,25 +1,34 @@
import logging import logging
from typing import List, Sequence from typing import List, Sequence, get_args
from facefusion.common_helper import create_float_range, create_int_range from facefusion.common_helper import create_float_range, create_int_range
from facefusion.typing import Angle, DownloadProvider, DownloadProviderSet, DownloadScope, ExecutionProvider, ExecutionProviderSet, FaceDetectorModel, FaceDetectorSet, FaceLandmarkerModel, FaceMaskRegion, FaceMaskRegionSet, FaceMaskType, FaceOccluderModel, FaceParserModel, FaceSelectorMode, FaceSelectorOrder, Gender, JobStatus, LogLevel, LogLevelSet, OutputAudioEncoder, OutputVideoEncoder, OutputVideoPreset, Race, Score, TempFrameFormat, UiWorkflow, VideoMemoryStrategy from facefusion.types import Angle, ApiSecurityStrategy, AudioEncoder, AudioFormat, AudioSet, BenchmarkMode, BenchmarkResolution, BenchmarkSet, DownloadProvider, DownloadProviderSet, DownloadScope, ExecutionProvider, ExecutionProviderSet, FaceDetectorModel, FaceDetectorSet, FaceLandmarkerModel, FaceMaskArea, FaceMaskAreaSet, FaceMaskRegion, FaceMaskRegionSet, FaceMaskType, FaceOccluderModel, FaceParserModel, FaceSelectorGender, FaceSelectorMode, FaceSelectorOrder, FaceSelectorRace, Gender, ImageEncoder, ImageFormat, ImageSet, JobStatus, LogLevel, LogLevelSet, Race, Score, TempFrameFormat, VideoEncoder, VideoFormat, VideoMemoryStrategy, VideoPreset, VideoSet, VoiceExtractorModel, WorkFlow
face_detector_set : FaceDetectorSet =\ face_detector_set : FaceDetectorSet =\
{ {
'many': [ '640x640' ], 'many': [ '640x640' ],
'retinaface': [ '160x160', '320x320', '480x480', '512x512', '640x640' ], 'retinaface': [ '160x160', '320x320', '480x480', '512x512', '640x640' ],
'scrfd': [ '160x160', '320x320', '480x480', '512x512', '640x640' ], 'scrfd': [ '160x160', '320x320', '480x480', '512x512', '640x640' ],
'yoloface': [ '640x640' ] 'yolo_face': [ '640x640' ],
'yunet': [ '640x640' ]
}
face_detector_models : List[FaceDetectorModel] = list(get_args(FaceDetectorModel))
face_landmarker_models : List[FaceLandmarkerModel] = list(get_args(FaceLandmarkerModel))
face_selector_modes : List[FaceSelectorMode] = list(get_args(FaceSelectorMode))
face_selector_orders : List[FaceSelectorOrder] = list(get_args(FaceSelectorOrder))
genders : List[Gender] = list(get_args(Gender))
races : List[Race] = list(get_args(Race))
face_selector_genders : List[FaceSelectorGender] = list(get_args(FaceSelectorGender))
face_selector_races : List[FaceSelectorRace] = list(get_args(FaceSelectorRace))
face_occluder_models : List[FaceOccluderModel] = list(get_args(FaceOccluderModel))
face_parser_models : List[FaceParserModel] = list(get_args(FaceParserModel))
face_mask_types : List[FaceMaskType] = list(get_args(FaceMaskType))
face_mask_area_set : FaceMaskAreaSet =\
{
'upper-face': [ 0, 1, 2, 31, 32, 33, 34, 35, 14, 15, 16, 26, 25, 24, 23, 22, 21, 20, 19, 18, 17 ],
'lower-face': [ 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 35, 34, 33, 32, 31 ],
'mouth': [ 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67 ]
} }
face_detector_models : List[FaceDetectorModel] = list(face_detector_set.keys())
face_landmarker_models : List[FaceLandmarkerModel] = [ 'many', '2dfan4', 'peppa_wutz' ]
face_selector_modes : List[FaceSelectorMode] = [ 'many', 'one', 'reference' ]
face_selector_orders : List[FaceSelectorOrder] = [ 'left-right', 'right-left', 'top-bottom', 'bottom-top', 'small-large', 'large-small', 'best-worst', 'worst-best' ]
face_selector_genders : List[Gender] = [ 'female', 'male' ]
face_selector_races : List[Race] = [ 'white', 'black', 'latino', 'asian', 'indian', 'arabic' ]
face_occluder_models : List[FaceOccluderModel] = [ 'xseg_1', 'xseg_2' ]
face_parser_models : List[FaceParserModel] = [ 'bisenet_resnet_18', 'bisenet_resnet_34' ]
face_mask_types : List[FaceMaskType] = [ 'box', 'occlusion', 'region' ]
face_mask_region_set : FaceMaskRegionSet =\ face_mask_region_set : FaceMaskRegionSet =\
{ {
'skin': 1, 'skin': 1,
@@ -33,43 +42,103 @@ face_mask_region_set : FaceMaskRegionSet =\
'upper-lip': 12, 'upper-lip': 12,
'lower-lip': 13 'lower-lip': 13
} }
face_mask_regions : List[FaceMaskRegion] = list(face_mask_region_set.keys()) face_mask_areas : List[FaceMaskArea] = list(get_args(FaceMaskArea))
temp_frame_formats : List[TempFrameFormat] = [ 'bmp', 'jpg', 'png' ] face_mask_regions : List[FaceMaskRegion] = list(get_args(FaceMaskRegion))
output_audio_encoders : List[OutputAudioEncoder] = [ 'aac', 'libmp3lame', 'libopus', 'libvorbis' ]
output_video_encoders : List[OutputVideoEncoder] = [ 'libx264', 'libx265', 'libvpx-vp9', 'h264_nvenc', 'hevc_nvenc', 'h264_amf', 'hevc_amf', 'h264_qsv', 'hevc_qsv', 'h264_videotoolbox', 'hevc_videotoolbox' ]
output_video_presets : List[OutputVideoPreset] = [ 'ultrafast', 'superfast', 'veryfast', 'faster', 'fast', 'medium', 'slow', 'slower', 'veryslow' ]
image_template_sizes : List[float] = [ 0.25, 0.5, 0.75, 1, 1.5, 2, 2.5, 3, 3.5, 4 ] voice_extractor_models : List[VoiceExtractorModel] = list(get_args(VoiceExtractorModel))
video_template_sizes : List[int] = [ 240, 360, 480, 540, 720, 1080, 1440, 2160, 4320 ]
workflows : List[WorkFlow] = [ 'auto', 'audio-to-image:frames', 'audio-to-image:video', 'image-to-image', 'image-to-video', 'image-to-video:frames' ]
audio_set : AudioSet =\
{
'flac': 'flac',
'm4a': 'aac',
'mp3': 'libmp3lame',
'ogg': 'flac',
'opus': 'libopus',
'wav': 'pcm_s16le'
}
image_set : ImageSet =\
{
'bmp': 'bmp',
'jpeg': 'mjpeg',
'png': 'png',
'tiff': 'tiff',
'webp': 'libwebp'
}
video_set : VideoSet =\
{
'avi': 'mpeg4',
'm4v': 'libx264',
'mkv': 'libx264',
'mov': 'libx264',
'mp4': 'libx264',
'mpeg': 'mpeg1video',
'mxf': 'mpeg2video',
'webm': 'libvpx-vp9',
'wmv': 'msmpeg4'
}
audio_formats : List[AudioFormat] = list(get_args(AudioFormat))
image_formats : List[ImageFormat] = list(get_args(ImageFormat))
video_formats : List[VideoFormat] = list(get_args(VideoFormat))
temp_frame_formats : List[TempFrameFormat] = list(get_args(TempFrameFormat))
audio_encoders : List[AudioEncoder] = list(get_args(AudioEncoder))
image_encoders : List[ImageEncoder] = list(get_args(ImageEncoder))
video_encoders : List[VideoEncoder] = list(get_args(VideoEncoder))
video_presets : List[VideoPreset] = list(get_args(VideoPreset))
benchmark_modes : List[BenchmarkMode] = list(get_args(BenchmarkMode))
benchmark_set : BenchmarkSet =\
{
'240p': '.assets/examples/target-240p.mp4',
'360p': '.assets/examples/target-360p.mp4',
'540p': '.assets/examples/target-540p.mp4',
'720p': '.assets/examples/target-720p.mp4',
'1080p': '.assets/examples/target-1080p.mp4',
'1440p': '.assets/examples/target-1440p.mp4',
'2160p': '.assets/examples/target-2160p.mp4'
}
benchmark_resolutions : List[BenchmarkResolution] = list(get_args(BenchmarkResolution))
execution_provider_set : ExecutionProviderSet =\ execution_provider_set : ExecutionProviderSet =\
{ {
'cpu': 'CPUExecutionProvider',
'coreml': 'CoreMLExecutionProvider',
'cuda': 'CUDAExecutionProvider', 'cuda': 'CUDAExecutionProvider',
'directml': 'DmlExecutionProvider', 'tensorrt': 'TensorrtExecutionProvider',
'openvino': 'OpenVINOExecutionProvider',
'rocm': 'ROCMExecutionProvider', 'rocm': 'ROCMExecutionProvider',
'tensorrt': 'TensorrtExecutionProvider' 'migraphx': 'MIGraphXExecutionProvider',
'coreml': 'CoreMLExecutionProvider',
'openvino': 'OpenVINOExecutionProvider',
'qnn': 'QNNExecutionProvider',
'directml': 'DmlExecutionProvider',
'cpu': 'CPUExecutionProvider'
} }
execution_providers : List[ExecutionProvider] = list(execution_provider_set.keys()) execution_providers : List[ExecutionProvider] = list(get_args(ExecutionProvider))
download_provider_set : DownloadProviderSet =\ download_provider_set : DownloadProviderSet =\
{ {
'github': 'github':
{ {
'url': 'https://github.com', 'urls':
[
'https://github.com'
],
'path': '/facefusion/facefusion-assets/releases/download/{base_name}/{file_name}' 'path': '/facefusion/facefusion-assets/releases/download/{base_name}/{file_name}'
}, },
'huggingface': 'huggingface':
{ {
'url': 'https://huggingface.co', 'urls':
[
'https://huggingface.co',
'https://hf-mirror.com'
],
'path': '/facefusion/{base_name}/resolve/main/{file_name}' 'path': '/facefusion/{base_name}/resolve/main/{file_name}'
} }
} }
download_providers : List[DownloadProvider] = list(download_provider_set.keys()) download_providers : List[DownloadProvider] = list(get_args(DownloadProvider))
download_scopes : List[DownloadScope] = [ 'lite', 'full' ] download_scopes : List[DownloadScope] = list(get_args(DownloadScope))
video_memory_strategies : List[VideoMemoryStrategy] = [ 'strict', 'moderate', 'tolerant' ] video_memory_strategies : List[VideoMemoryStrategy] = list(get_args(VideoMemoryStrategy))
api_security_strategies : List[ApiSecurityStrategy] = list(get_args(ApiSecurityStrategy))
log_level_set : LogLevelSet =\ log_level_set : LogLevelSet =\
{ {
@@ -78,20 +147,25 @@ log_level_set : LogLevelSet =\
'info': logging.INFO, 'info': logging.INFO,
'debug': logging.DEBUG 'debug': logging.DEBUG
} }
log_levels : List[LogLevel] = list(log_level_set.keys()) log_levels : List[LogLevel] = list(get_args(LogLevel))
ui_workflows : List[UiWorkflow] = [ 'instant_runner', 'job_runner', 'job_manager' ] job_statuses : List[JobStatus] = list(get_args(JobStatus))
job_statuses : List[JobStatus] = [ 'drafted', 'queued', 'completed', 'failed' ]
benchmark_cycle_count_range : Sequence[int] = create_int_range(1, 10, 1)
execution_thread_count_range : Sequence[int] = create_int_range(1, 32, 1) execution_thread_count_range : Sequence[int] = create_int_range(1, 32, 1)
execution_queue_count_range : Sequence[int] = create_int_range(1, 4, 1) face_detector_margin_range : Sequence[int] = create_int_range(0, 100, 1)
system_memory_limit_range : Sequence[int] = create_int_range(0, 128, 4)
face_detector_angles : Sequence[Angle] = create_int_range(0, 270, 90) face_detector_angles : Sequence[Angle] = create_int_range(0, 270, 90)
face_detector_score_range : Sequence[Score] = create_float_range(0.0, 1.0, 0.05) face_detector_score_range : Sequence[Score] = create_float_range(0.0, 1.0, 0.05)
face_landmarker_score_range : Sequence[Score] = create_float_range(0.0, 1.0, 0.05) face_landmarker_score_range : Sequence[Score] = create_float_range(0.0, 1.0, 0.05)
face_mask_blur_range : Sequence[float] = create_float_range(0.0, 1.0, 0.05) face_mask_blur_range : Sequence[float] = create_float_range(0.0, 1.0, 0.05)
face_mask_padding_range : Sequence[int] = create_int_range(0, 100, 1) face_mask_padding_range : Sequence[int] = create_int_range(0, 100, 1)
face_selector_age_range : Sequence[int] = create_int_range(0, 100, 1) face_selector_age_range : Sequence[int] = create_int_range(0, 100, 1)
reference_face_distance_range : Sequence[float] = create_float_range(0.0, 1.5, 0.05) reference_face_distance_range : Sequence[float] = create_float_range(0.0, 1.0, 0.05)
face_tracker_score_range : Sequence[Score] = create_float_range(0.0, 0.5, 0.05)
target_frame_amount_range : Sequence[int] = create_int_range(0, 10, 1)
output_image_quality_range : Sequence[int] = create_int_range(0, 100, 1) output_image_quality_range : Sequence[int] = create_int_range(0, 100, 1)
output_image_scale_range : Sequence[float] = create_float_range(0.25, 8.0, 0.25)
output_audio_quality_range : Sequence[int] = create_int_range(0, 100, 1)
output_audio_volume_range : Sequence[int] = create_int_range(0, 100, 1)
output_video_quality_range : Sequence[int] = create_int_range(0, 100, 1) output_video_quality_range : Sequence[int] = create_int_range(0, 100, 1)
output_video_scale_range : Sequence[float] = create_float_range(0.25, 8.0, 0.25)
+35
View File
@@ -0,0 +1,35 @@
from typing import List, Tuple
from facefusion.logger import get_package_logger
from facefusion.types import TableContent, TableHeader
def render_table(headers : List[TableHeader], contents : List[List[TableContent]]) -> None:
package_logger = get_package_logger()
table_column, table_separator = create_table_parts(headers, contents)
package_logger.critical(table_separator)
package_logger.critical(table_column.format(*headers))
package_logger.critical(table_separator)
for content in contents:
content = [ str(value) for value in content ]
package_logger.critical(table_column.format(*content))
package_logger.critical(table_separator)
def create_table_parts(headers : List[TableHeader], contents : List[List[TableContent]]) -> Tuple[str, str]:
column_parts = []
separator_parts = []
widths = [ len(header) for header in headers ]
for content in contents:
for index, value in enumerate(content):
widths[index] = max(widths[index], len(str(value)))
for width in widths:
column_parts.append('{:<' + str(width) + '}')
separator_parts.append('-' * width)
return '| ' + ' | '.join(column_parts) + ' |', '+-' + '-+-'.join(separator_parts) + '-+'
View File
+70
View File
@@ -0,0 +1,70 @@
import ctypes
import struct
from typing import Optional
from facefusion.libraries import aom as aom_module
from facefusion.types import AomDecoder, Buffer, BufferPack
def create(thread_count : int) -> Optional[AomDecoder]:
aom_library = aom_module.create_static_library()
if aom_library:
aom_decoder = ctypes.create_string_buffer(128)
aom_codec = ctypes.c_void_p.in_dll(aom_library, 'aom_codec_av1_dx_algo')
config_buffer = ctypes.create_string_buffer(128)
struct.pack_into('I', config_buffer, 0, thread_count)
struct.pack_into('I', config_buffer, 12, 1)
if aom_library.aom_codec_dec_init_ver(aom_decoder, ctypes.byref(aom_codec), config_buffer, 0, 22) == 0:
return aom_decoder
return None
def decode(aom_decoder : AomDecoder, input_buffer : Buffer) -> Optional[BufferPack]:
aom_library = aom_module.create_static_library()
if aom_library and input_buffer:
input_total = len(input_buffer)
temp_buffer = ctypes.create_string_buffer(input_buffer)
if aom_library.aom_codec_decode(aom_decoder, temp_buffer, input_total, None) == 0:
address = aom_library.aom_codec_get_frame(aom_decoder, ctypes.byref(ctypes.c_void_p(0)))
if address:
frame_width = ctypes.c_uint.from_address(address + 28).value & ~1
frame_height = ctypes.c_uint.from_address(address + 32).value & ~1
return BufferPack(
buffer = collect(address, frame_width, frame_height),
resolution = (frame_width, frame_height)
)
return None
def collect(address : int, frame_width : int, frame_height : int) -> Buffer:
output_parts = []
for index in range(3):
plane_pointer = ctypes.c_void_p.from_address(address + 64 + index * 8).value
stride = ctypes.c_int.from_address(address + 88 + index * 4).value
plane_width = frame_width >> (index > 0)
plane_height = frame_height >> (index > 0)
if stride == plane_width:
output_parts.append(ctypes.string_at(plane_pointer, plane_width * plane_height))
else:
for row in range(plane_height):
output_parts.append(ctypes.string_at(plane_pointer + row * stride, plane_width))
return bytes().join(output_parts)
def destroy(aom_decoder : AomDecoder) -> None:
aom_library = aom_module.create_static_library()
if aom_library:
aom_library.aom_codec_destroy(aom_decoder)
+94
View File
@@ -0,0 +1,94 @@
import ctypes
import struct
from typing import Optional
from facefusion.libraries import aom as aom_module
from facefusion.types import AomEncoder, BitRate, Buffer, Resolution
def create(frame_resolution : Resolution, bitrate : BitRate, thread_count : int, cpu_count : int) -> Optional[AomEncoder]:
aom_library = aom_module.create_static_library()
if aom_library:
aom_encoder = ctypes.create_string_buffer(1152)
aom_codec = ctypes.c_void_p.in_dll(aom_library, 'aom_codec_av1_cx_algo')
config_buffer = ctypes.create_string_buffer(1024)
if aom_library.aom_codec_enc_config_default(ctypes.byref(aom_codec), config_buffer, 1) == 0:
struct.pack_into('I', config_buffer, 4, thread_count)
struct.pack_into('I', config_buffer, 12, frame_resolution[0])
struct.pack_into('I', config_buffer, 16, frame_resolution[1])
struct.pack_into('I', config_buffer, 136, bitrate)
struct.pack_into('I', config_buffer, 192, 30)
if aom_library.aom_codec_enc_init_ver(aom_encoder, ctypes.byref(aom_codec), config_buffer, 0, 25) == 0:
aom_library.aom_codec_control(aom_encoder, 13, ctypes.c_int(cpu_count))
aom_library.aom_codec_control(aom_encoder, 75, ctypes.c_int(1))
aom_library.aom_codec_control(aom_encoder, 106, ctypes.c_int(0))
aom_library.aom_codec_control(aom_encoder, 122, ctypes.c_int(0))
aom_library.aom_codec_control(aom_encoder, 123, ctypes.c_int(0))
ctypes.memmove(ctypes.addressof(aom_encoder) + 128, config_buffer, 1024)
return aom_encoder
return None
def encode(aom_encoder : AomEncoder, input_buffer : Buffer, frame_resolution : Resolution, frame_index : int) -> Buffer:
aom_library = aom_module.create_static_library()
output_buffer = bytes()
if aom_library:
temp_buffer = ctypes.create_string_buffer(256)
encode_buffer = ctypes.create_string_buffer(input_buffer)
if aom_library.aom_img_wrap(temp_buffer, 0x102, frame_resolution[0], frame_resolution[1], 1, encode_buffer) and aom_library.aom_codec_encode(aom_encoder, temp_buffer, frame_index, 1, 0, 1) == 0:
output_buffer = collect(aom_encoder)
return output_buffer
def collect(aom_encoder : AomEncoder) -> Buffer:
aom_library = aom_module.create_static_library()
output_parts = []
packet_cursor = ctypes.c_void_p(0)
packet = aom_library.aom_codec_get_cx_data(aom_encoder, ctypes.byref(packet_cursor))
while packet:
if ctypes.c_int.from_address(packet).value == 0:
buffer_pointer = ctypes.c_void_p.from_address(packet + 8).value
buffer_size = ctypes.c_size_t.from_address(packet + 16).value
output_parts.append(ctypes.string_at(buffer_pointer, buffer_size))
packet = aom_library.aom_codec_get_cx_data(aom_encoder, ctypes.byref(packet_cursor))
return bytes().join(output_parts)
def update_resolution(aom_encoder : AomEncoder, frame_resolution : Resolution) -> bool:
aom_library = aom_module.create_static_library()
if aom_library:
struct.pack_into('I', aom_encoder, 128 + 12, frame_resolution[0])
struct.pack_into('I', aom_encoder, 128 + 16, frame_resolution[1])
return aom_library.aom_codec_enc_config_set(aom_encoder, ctypes.cast(ctypes.addressof(aom_encoder) + 128, ctypes.c_void_p)) == 0
return False
def update_bitrate(aom_encoder : AomEncoder, bitrate : BitRate) -> bool:
aom_library = aom_module.create_static_library()
if aom_library:
struct.pack_into('I', aom_encoder, 128 + 136, bitrate)
return aom_library.aom_codec_enc_config_set(aom_encoder, ctypes.cast(ctypes.addressof(aom_encoder) + 128, ctypes.c_void_p)) == 0
return False
def destroy(aom_encoder : AomEncoder) -> None:
aom_library = aom_module.create_static_library()
if aom_library:
aom_library.aom_codec_destroy(aom_encoder)
+38
View File
@@ -0,0 +1,38 @@
import ctypes
from typing import Optional
from facefusion.libraries import opus as opus_module
from facefusion.types import Buffer, OpusDecoder
def create(sample_rate : int, channel_total : int) -> Optional[OpusDecoder]:
opus_library = opus_module.create_static_library()
if opus_library:
return opus_library.opus_decoder_create(sample_rate, channel_total, ctypes.byref(ctypes.c_int(0)))
return None
def decode(opus_decoder : OpusDecoder, input_buffer : Buffer, channel_total : int) -> Buffer:
opus_library = opus_module.create_static_library()
output_buffer = bytes()
if opus_library:
input_total = len(input_buffer)
sample_size = ctypes.sizeof(ctypes.c_float)
sample_total = opus_library.opus_decoder_get_nb_samples(opus_decoder, input_buffer, input_total)
sample_buffer = (ctypes.c_float * (sample_total * channel_total))()
output_total = opus_library.opus_decode_float(opus_decoder, input_buffer, input_total, sample_buffer, sample_total, 0)
if output_total:
output_buffer = ctypes.string_at(ctypes.addressof(sample_buffer), output_total * channel_total * sample_size)
return output_buffer
def destroy(opus_decoder : OpusDecoder) -> None:
opus_library = opus_module.create_static_library()
if opus_library:
opus_library.opus_decoder_destroy(opus_decoder)
+38
View File
@@ -0,0 +1,38 @@
import ctypes
from typing import Optional
from facefusion.libraries import opus as opus_module
from facefusion.types import Buffer, OpusEncoder
def create(sample_rate : int, channel_total : int) -> Optional[OpusEncoder]:
opus_library = opus_module.create_static_library()
if opus_library:
return opus_library.opus_encoder_create(sample_rate, channel_total, 2049, ctypes.byref(ctypes.c_int(0)))
return None
def encode(opus_encoder : OpusEncoder, input_buffer : Buffer, channel_total : int) -> Buffer:
opus_library = opus_module.create_static_library()
output_buffer = bytes()
if opus_library:
sample_size = ctypes.sizeof(ctypes.c_float)
sample_total = len(input_buffer) // (sample_size * channel_total)
sample_buffer = (ctypes.c_float * (sample_total * channel_total)).from_buffer_copy(input_buffer)
temp_buffer = ctypes.create_string_buffer(2048)
output_total = opus_library.opus_encode_float(opus_encoder, sample_buffer, sample_total, temp_buffer, 2048)
if output_total:
output_buffer = temp_buffer.raw[:output_total]
return output_buffer
def destroy(opus_encoder : OpusEncoder) -> None:
opus_library = opus_module.create_static_library()
if opus_library:
opus_library.opus_encoder_destroy(opus_encoder)
+74
View File
@@ -0,0 +1,74 @@
import ctypes
import struct
from typing import Optional
from facefusion.libraries import vpx as vpx_module
from facefusion.types import Buffer, BufferPack, VpxDecoder, VxpVideoCodec
def create(video_codec : VxpVideoCodec, thread_count : int) -> Optional[VpxDecoder]:
vpx_library = vpx_module.create_static_library()
if vpx_library:
vpx_decoder = ctypes.create_string_buffer(64)
vpx_algo = 'vpx_codec_vp8_dx_algo'
if video_codec == 'vp9':
vpx_algo = 'vpx_codec_vp9_dx_algo'
vpx_codec = ctypes.c_void_p.in_dll(vpx_library, vpx_algo)
config_buffer = ctypes.create_string_buffer(128)
struct.pack_into('I', config_buffer, 0, thread_count)
if vpx_library.vpx_codec_dec_init_ver(vpx_decoder, ctypes.byref(vpx_codec), config_buffer, 0, 12) == 0:
return vpx_decoder
return None
def decode(vpx_decoder : VpxDecoder, input_buffer : Buffer) -> Optional[BufferPack]:
vpx_library = vpx_module.create_static_library()
if vpx_library and input_buffer:
input_total = len(input_buffer)
temp_buffer = ctypes.create_string_buffer(input_buffer)
if vpx_library.vpx_codec_decode(vpx_decoder, temp_buffer, input_total, None, 0) == 0:
address = vpx_library.vpx_codec_get_frame(vpx_decoder, ctypes.byref(ctypes.c_void_p(0)))
if address:
frame_width = ctypes.c_uint.from_address(address + 24).value & ~1
frame_height = ctypes.c_uint.from_address(address + 28).value & ~1
return BufferPack(
buffer = collect(address, frame_width, frame_height),
resolution = (frame_width, frame_height)
)
return None
def collect(address : int, frame_width : int, frame_height : int) -> Buffer:
output_parts = []
for index in range(3):
plane_pointer = ctypes.c_void_p.from_address(address + 48 + index * 8).value
stride = ctypes.c_int.from_address(address + 80 + index * 4).value
plane_width = frame_width >> (index > 0)
plane_height = frame_height >> (index > 0)
if stride == plane_width:
output_parts.append(ctypes.string_at(plane_pointer, plane_width * plane_height))
else:
for row in range(plane_height):
output_parts.append(ctypes.string_at(plane_pointer + row * stride, plane_width))
return bytes().join(output_parts)
def destroy(vpx_decoder : VpxDecoder) -> None:
vpx_library = vpx_module.create_static_library()
if vpx_library:
vpx_library.vpx_codec_destroy(vpx_decoder)
+104
View File
@@ -0,0 +1,104 @@
import ctypes
import struct
from typing import Optional
from facefusion.libraries import vpx as vpx_module
from facefusion.types import BitRate, Buffer, Resolution, VpxEncoder, VxpVideoCodec
def create(video_codec : VxpVideoCodec, frame_resolution : Resolution, bitrate : BitRate, thread_count : int, cpu_count : int) -> Optional[VpxEncoder]:
vpx_library = vpx_module.create_static_library()
if vpx_library:
vpx_encoder = ctypes.create_string_buffer(640)
vpx_algo = 'vpx_codec_vp8_cx_algo'
if video_codec == 'vp9':
vpx_algo = 'vpx_codec_vp9_cx_algo'
vpx_codec = ctypes.c_void_p.in_dll(vpx_library, vpx_algo)
config_buffer = ctypes.create_string_buffer(512)
if vpx_library.vpx_codec_enc_config_default(ctypes.byref(vpx_codec), config_buffer, 0) == 0:
struct.pack_into('I', config_buffer, 4, thread_count)
struct.pack_into('I', config_buffer, 12, frame_resolution[0])
struct.pack_into('I', config_buffer, 16, frame_resolution[1])
struct.pack_into('I', config_buffer, 28, 1)
struct.pack_into('I', config_buffer, 36, 0)
struct.pack_into('I', config_buffer, 44, 0)
struct.pack_into('I', config_buffer, 72, 0)
struct.pack_into('I', config_buffer, 112, bitrate)
struct.pack_into('I', config_buffer, 116, 2)
struct.pack_into('I', config_buffer, 120, 50)
struct.pack_into('I', config_buffer, 124, 50)
struct.pack_into('I', config_buffer, 128, 50)
if vpx_library.vpx_codec_enc_init_ver(vpx_encoder, ctypes.byref(vpx_codec), config_buffer, 0, 39) == 0:
vpx_library.vpx_codec_control_(vpx_encoder, 13, ctypes.c_int(cpu_count))
vpx_library.vpx_codec_control_(vpx_encoder, 12, ctypes.c_int(0))
vpx_library.vpx_codec_control_(vpx_encoder, 27, ctypes.c_int(10))
ctypes.memmove(ctypes.addressof(vpx_encoder) + 64, config_buffer, 512)
return vpx_encoder
return None
def encode(vpx_encoder : VpxEncoder, input_buffer : Buffer, frame_resolution : Resolution, frame_index : int) -> Buffer:
vpx_library = vpx_module.create_static_library()
output_buffer = bytes()
if vpx_library:
temp_buffer = ctypes.create_string_buffer(256)
encode_buffer = ctypes.create_string_buffer(input_buffer)
if vpx_library.vpx_img_wrap(temp_buffer, 0x102, frame_resolution[0], frame_resolution[1], 1, encode_buffer) and vpx_library.vpx_codec_encode(vpx_encoder, temp_buffer, frame_index, 1, 0, 1) == 0:
output_buffer = collect(vpx_encoder)
return output_buffer
def collect(vpx_encoder : VpxEncoder) -> Buffer:
vpx_library = vpx_module.create_static_library()
output_parts = []
packet_cursor = ctypes.c_void_p(0)
packet = vpx_library.vpx_codec_get_cx_data(vpx_encoder, ctypes.byref(packet_cursor))
while packet:
if ctypes.c_int.from_address(packet).value == 0:
buffer_pointer = ctypes.c_void_p.from_address(packet + 8).value
buffer_size = ctypes.c_size_t.from_address(packet + 16).value
output_parts.append(ctypes.string_at(buffer_pointer, buffer_size))
packet = vpx_library.vpx_codec_get_cx_data(vpx_encoder, ctypes.byref(packet_cursor))
return bytes().join(output_parts)
def update_resolution(vpx_encoder : VpxEncoder, frame_resolution : Resolution) -> bool:
vpx_library = vpx_module.create_static_library()
if vpx_library:
struct.pack_into('I', vpx_encoder, 64 + 12, frame_resolution[0])
struct.pack_into('I', vpx_encoder, 64 + 16, frame_resolution[1])
return vpx_library.vpx_codec_enc_config_set(vpx_encoder, ctypes.cast(ctypes.addressof(vpx_encoder) + 64, ctypes.c_void_p)) == 0
return False
def update_bitrate(vpx_encoder : VpxEncoder, bitrate : BitRate) -> bool:
vpx_library = vpx_module.create_static_library()
if vpx_library:
struct.pack_into('I', vpx_encoder, 64 + 112, bitrate)
return vpx_library.vpx_codec_enc_config_set(vpx_encoder, ctypes.cast(ctypes.addressof(vpx_encoder) + 64, ctypes.c_void_p)) == 0
return False
def destroy(vpx_encoder : VpxEncoder) -> None:
vpx_library = vpx_module.create_static_library()
if vpx_library:
vpx_library.vpx_codec_destroy(vpx_encoder)
+27 -9
View File
@@ -1,5 +1,5 @@
import platform import platform
from typing import Any, Optional, Sequence from typing import Any, Iterable, Optional, Reversible, Sequence
def is_linux() -> bool: def is_linux() -> bool:
@@ -15,11 +15,11 @@ def is_windows() -> bool:
def create_int_metavar(int_range : Sequence[int]) -> str: def create_int_metavar(int_range : Sequence[int]) -> str:
return '[' + str(int_range[0]) + '..' + str(int_range[-1]) + ':' + str(calc_int_step(int_range)) + ']' return '[' + str(int_range[0]) + '..' + str(int_range[-1]) + ':' + str(calculate_int_step(int_range)) + ']'
def create_float_metavar(float_range : Sequence[float]) -> str: def create_float_metavar(float_range : Sequence[float]) -> str:
return '[' + str(float_range[0]) + '..' + str(float_range[-1]) + ':' + str(calc_float_step(float_range)) + ']' return '[' + str(float_range[0]) + '..' + str(float_range[-1]) + ':' + str(calculate_float_step(float_range)) + ']'
def create_int_range(start : int, end : int, step : int) -> Sequence[int]: def create_int_range(start : int, end : int, step : int) -> Sequence[int]:
@@ -42,31 +42,49 @@ def create_float_range(start : float, end : float, step : float) -> Sequence[flo
return float_range return float_range
def calc_int_step(int_range : Sequence[int]) -> int: def calculate_int_step(int_range : Sequence[int]) -> int:
return int_range[1] - int_range[0] return int_range[1] - int_range[0]
def calc_float_step(float_range : Sequence[float]) -> float: def calculate_float_step(float_range : Sequence[float]) -> float:
return round(float_range[1] - float_range[0], 2) return round(float_range[1] - float_range[0], 2)
def cast_int(value : Any) -> Optional[Any]: def cast_int(value : Any) -> Optional[int]:
try: try:
return int(value) return int(value)
except (ValueError, TypeError): except (ValueError, TypeError):
return None return None
def cast_float(value : Any) -> Optional[Any]: def cast_float(value : Any) -> Optional[float]:
try: try:
return float(value) return float(value)
except (ValueError, TypeError): except (ValueError, TypeError):
return None return None
def cast_bool(value : Any) -> Optional[bool]:
if value == 'True':
return True
if value == 'False':
return False
return None
def get_first(__list__ : Any) -> Any: def get_first(__list__ : Any) -> Any:
return next(iter(__list__), None) if isinstance(__list__, Iterable):
return next(iter(__list__), None)
return None
def get_middle(__list__ : Any) -> Any:
if isinstance(__list__, Sequence) and __list__:
return __list__[len(__list__) // 2]
return None
def get_last(__list__ : Any) -> Any: def get_last(__list__ : Any) -> Any:
return next(reversed(__list__), None) if isinstance(__list__, Reversible):
return next(reversed(__list__), None)
return None
+41
View File
@@ -0,0 +1,41 @@
import os
import sys
from typing import List
from facefusion.common_helper import is_linux, is_windows
def setup() -> None:
conda_prefix = os.getenv('CONDA_PREFIX')
conda_ready = os.getenv('CONDA_READY')
if conda_prefix and not conda_ready:
if is_linux():
python_id = 'python' + str(sys.version_info.major) + '.' + str(sys.version_info.minor)
library_paths : List[str] =\
[
os.path.join(conda_prefix, 'lib'),
os.path.join(conda_prefix, 'lib', python_id, 'site-packages', 'tensorrt_libs')
]
library_paths = list(filter(os.path.exists, library_paths))
if library_paths:
if os.getenv('LD_LIBRARY_PATH'):
library_paths.append(os.getenv('LD_LIBRARY_PATH'))
os.environ['LD_LIBRARY_PATH'] = os.pathsep.join(library_paths)
os.environ['CONDA_READY'] = '1'
os.execv(sys.executable, [ sys.executable ] + sys.argv)
if is_windows():
library_paths =\
[
os.path.join(conda_prefix, 'Lib'),
os.path.join(conda_prefix, 'Lib', 'site-packages', 'tensorrt_libs')
]
library_paths = list(filter(os.path.exists, library_paths))
if library_paths:
if os.getenv('PATH'):
library_paths.append(os.getenv('PATH'))
os.environ['PATH'] = os.pathsep.join(library_paths)
os.environ['CONDA_READY'] = '1'
+49 -76
View File
@@ -1,92 +1,65 @@
from configparser import ConfigParser from configparser import ConfigParser
from typing import Any, List, Optional from functools import lru_cache
from typing import List, Optional
from facefusion import state_manager from facefusion import state_manager
from facefusion.common_helper import cast_float, cast_int from facefusion.common_helper import cast_bool, cast_float, cast_int
CONFIG = None
def get_config() -> ConfigParser: @lru_cache
global CONFIG def get_static_config_parser() -> ConfigParser:
config_parser = ConfigParser()
if CONFIG is None: config_parser.read(state_manager.get_item('config_path'), encoding = 'utf-8')
CONFIG = ConfigParser() return config_parser
CONFIG.read(state_manager.get_item('config_path'), encoding = 'utf-8')
return CONFIG
def clear_config() -> None: def get_str_value(section : str, option : str, fallback : Optional[str] = None) -> Optional[str]:
global CONFIG config_parser = get_static_config_parser()
CONFIG = None if config_parser.has_option(section, option) and config_parser.get(section, option).strip():
return config_parser.get(section, option)
return fallback
def get_str_value(key : str, fallback : Optional[str] = None) -> Optional[str]: def get_int_value(section : str, option : str, fallback : Optional[str] = None) -> Optional[int]:
value = get_value_by_notation(key) config_parser = get_static_config_parser()
if value or fallback: if config_parser.has_option(section, option) and config_parser.get(section, option).strip():
return str(value or fallback) return config_parser.getint(section, option)
return cast_int(fallback)
def get_float_value(section : str, option : str, fallback : Optional[str] = None) -> Optional[float]:
config_parser = get_static_config_parser()
if config_parser.has_option(section, option) and config_parser.get(section, option).strip():
return config_parser.getfloat(section, option)
return cast_float(fallback)
def get_bool_value(section : str, option : str, fallback : Optional[str] = None) -> Optional[bool]:
config_parser = get_static_config_parser()
if config_parser.has_option(section, option) and config_parser.get(section, option).strip():
return config_parser.getboolean(section, option)
return cast_bool(fallback)
def get_str_list(section : str, option : str, fallback : Optional[str] = None) -> Optional[List[str]]:
config_parser = get_static_config_parser()
if config_parser.has_option(section, option) and config_parser.get(section, option).strip():
return config_parser.get(section, option).split()
if fallback:
return fallback.split()
return None return None
def get_int_value(key : str, fallback : Optional[str] = None) -> Optional[int]: def get_int_list(section : str, option : str, fallback : Optional[str] = None) -> Optional[List[int]]:
value = get_value_by_notation(key) config_parser = get_static_config_parser()
if value or fallback: if config_parser.has_option(section, option) and config_parser.get(section, option).strip():
return cast_int(value or fallback) return list(map(int, config_parser.get(section, option).split()))
return None if fallback:
return list(map(int, fallback.split()))
def get_float_value(key : str, fallback : Optional[str] = None) -> Optional[float]:
value = get_value_by_notation(key)
if value or fallback:
return cast_float(value or fallback)
return None
def get_bool_value(key : str, fallback : Optional[str] = None) -> Optional[bool]:
value = get_value_by_notation(key)
if value == 'True' or fallback == 'True':
return True
if value == 'False' or fallback == 'False':
return False
return None
def get_str_list(key : str, fallback : Optional[str] = None) -> Optional[List[str]]:
value = get_value_by_notation(key)
if value or fallback:
return [ str(value) for value in (value or fallback).split(' ') ]
return None
def get_int_list(key : str, fallback : Optional[str] = None) -> Optional[List[int]]:
value = get_value_by_notation(key)
if value or fallback:
return [ cast_int(value) for value in (value or fallback).split(' ') ]
return None
def get_float_list(key : str, fallback : Optional[str] = None) -> Optional[List[float]]:
value = get_value_by_notation(key)
if value or fallback:
return [ cast_float(value) for value in (value or fallback).split(' ') ]
return None
def get_value_by_notation(key : str) -> Optional[Any]:
config = get_config()
if '.' in key:
section, name = key.split('.')
if section in config and name in config[section]:
return config[section][name]
if key in config:
return config[key]
return None return None
+171 -56
View File
@@ -1,67 +1,138 @@
from functools import lru_cache from functools import lru_cache
from typing import Tuple
import cv2
import numpy import numpy
from tqdm import tqdm from tqdm import tqdm
from facefusion import inference_manager, state_manager, wording from facefusion import inference_manager, state_manager, translator
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url
from facefusion.filesystem import resolve_relative_path from facefusion.filesystem import resolve_relative_path
from facefusion.thread_helper import conditional_thread_semaphore from facefusion.thread_helper import conditional_thread_semaphore
from facefusion.typing import DownloadScope, Fps, InferencePool, ModelOptions, ModelSet, VisionFrame from facefusion.types import Detection, DownloadScope, DownloadSet, Fps, InferencePool, ModelSet, VisionFrame
from facefusion.vision import detect_video_fps, get_video_frame, read_image from facefusion.vision import detect_video_fps, fit_contain_frame, read_image, read_video_frame
PROBABILITY_LIMIT = 0.80
RATE_LIMIT = 10
STREAM_COUNTER = 0 STREAM_COUNTER = 0
@lru_cache(maxsize = None) @lru_cache()
def create_static_model_set(download_scope : DownloadScope) -> ModelSet: def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
return\ return\
{ {
'open_nsfw': 'nsfw_1':
{ {
'__metadata__':
{
'vendor': 'EraX',
'license': 'Apache-2.0',
'year': 2024
},
'hashes': 'hashes':
{ {
'content_analyser': 'content_analyser':
{ {
'url': resolve_download_url('models-3.0.0', 'open_nsfw.hash'), 'url': resolve_download_url('models-3.3.0', 'nsfw_1.hash'),
'path': resolve_relative_path('../.assets/models/open_nsfw.hash') 'path': resolve_relative_path('../.assets/models/nsfw_1.hash')
} }
}, },
'sources': 'sources':
{ {
'content_analyser': 'content_analyser':
{ {
'url': resolve_download_url('models-3.0.0', 'open_nsfw.onnx'), 'url': resolve_download_url('models-3.3.0', 'nsfw_1.onnx'),
'path': resolve_relative_path('../.assets/models/open_nsfw.onnx') 'path': resolve_relative_path('../.assets/models/nsfw_1.onnx')
} }
}, },
'size': (224, 224), 'size': (640, 640),
'mean': [ 104, 117, 123 ] 'mean': (0.0, 0.0, 0.0),
'standard_deviation': (1.0, 1.0, 1.0)
},
'nsfw_2':
{
'__metadata__':
{
'vendor': 'Marqo',
'license': 'Apache-2.0',
'year': 2024
},
'hashes':
{
'content_analyser':
{
'url': resolve_download_url('models-3.3.0', 'nsfw_2.hash'),
'path': resolve_relative_path('../.assets/models/nsfw_2.hash')
}
},
'sources':
{
'content_analyser':
{
'url': resolve_download_url('models-3.3.0', 'nsfw_2.onnx'),
'path': resolve_relative_path('../.assets/models/nsfw_2.onnx')
}
},
'size': (384, 384),
'mean': (0.5, 0.5, 0.5),
'standard_deviation': (0.5, 0.5, 0.5)
},
'nsfw_3':
{
'__metadata__':
{
'vendor': 'Freepik',
'license': 'MIT',
'year': 2025
},
'hashes':
{
'content_analyser':
{
'url': resolve_download_url('models-3.3.0', 'nsfw_3.hash'),
'path': resolve_relative_path('../.assets/models/nsfw_3.hash')
}
},
'sources':
{
'content_analyser':
{
'url': resolve_download_url('models-3.3.0', 'nsfw_3.onnx'),
'path': resolve_relative_path('../.assets/models/nsfw_3.onnx')
}
},
'size': (448, 448),
'mean': (0.48145466, 0.4578275, 0.40821073),
'standard_deviation': (0.26862954, 0.26130258, 0.27577711)
} }
} }
def get_inference_pool() -> InferencePool: def get_inference_pool() -> InferencePool:
model_sources = get_model_options().get('sources') model_names = [ 'nsfw_1', 'nsfw_2', 'nsfw_3' ]
return inference_manager.get_inference_pool(__name__, model_sources) _, model_source_set = collect_model_downloads()
return inference_manager.get_inference_pool(__name__, model_names, model_source_set)
def clear_inference_pool() -> None: def clear_inference_pool() -> None:
inference_manager.clear_inference_pool(__name__) model_names = [ 'nsfw_1', 'nsfw_2', 'nsfw_3' ]
inference_manager.clear_inference_pool(__name__, model_names)
def get_model_options() -> ModelOptions: def collect_model_downloads() -> Tuple[DownloadSet, DownloadSet]:
return create_static_model_set('full').get('open_nsfw') model_set = create_static_model_set('full')
model_hash_set = {}
model_source_set = {}
for content_analyser_model in [ 'nsfw_1', 'nsfw_2', 'nsfw_3' ]:
model_hash_set[content_analyser_model] = model_set.get(content_analyser_model).get('hashes').get('content_analyser')
model_source_set[content_analyser_model] = model_set.get(content_analyser_model).get('sources').get('content_analyser')
return model_hash_set, model_source_set
def pre_check() -> bool: def pre_check() -> bool:
model_hashes = get_model_options().get('hashes') model_hash_set, model_source_set = collect_model_downloads()
model_sources = get_model_options().get('sources')
return conditional_download_hashes(model_hashes) and conditional_download_sources(model_sources) return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set)
def analyse_stream(vision_frame : VisionFrame, video_fps : Fps) -> bool: def analyse_stream(vision_frame : VisionFrame, video_fps : Fps) -> bool:
@@ -74,53 +145,97 @@ def analyse_stream(vision_frame : VisionFrame, video_fps : Fps) -> bool:
def analyse_frame(vision_frame : VisionFrame) -> bool: def analyse_frame(vision_frame : VisionFrame) -> bool:
vision_frame = prepare_frame(vision_frame) return detect_nsfw(vision_frame)
probability = forward(vision_frame)
return probability > PROBABILITY_LIMIT
def forward(vision_frame : VisionFrame) -> float: @lru_cache()
content_analyser = get_inference_pool().get('content_analyser')
with conditional_thread_semaphore():
probability = content_analyser.run(None,
{
'input': vision_frame
})[0][0][1]
return probability
def prepare_frame(vision_frame : VisionFrame) -> VisionFrame:
model_size = get_model_options().get('size')
model_mean = get_model_options().get('mean')
vision_frame = cv2.resize(vision_frame, model_size).astype(numpy.float32)
vision_frame -= numpy.array(model_mean).astype(numpy.float32)
vision_frame = numpy.expand_dims(vision_frame, axis = 0)
return vision_frame
@lru_cache(maxsize = None)
def analyse_image(image_path : str) -> bool: def analyse_image(image_path : str) -> bool:
vision_frame = read_image(image_path) vision_frame = read_image(image_path)
return analyse_frame(vision_frame) return analyse_frame(vision_frame)
@lru_cache(maxsize = None) @lru_cache()
def analyse_video(video_path : str, trim_frame_start : int, trim_frame_end : int) -> bool: def analyse_video(video_path : str, trim_frame_start : int, trim_frame_end : int) -> bool:
video_fps = detect_video_fps(video_path) video_fps = detect_video_fps(video_path)
frame_range = range(trim_frame_start, trim_frame_end) frame_range = range(trim_frame_start, trim_frame_end)
rate = 0.0 rate = 0.0
total = 0
counter = 0 counter = 0
with tqdm(total = len(frame_range), desc = wording.get('analysing'), unit = 'frame', ascii = ' =', disable = state_manager.get_item('log_level') in [ 'warn', 'error' ]) as progress: with tqdm(total = len(frame_range), desc = translator.get('analysing'), unit = 'frame', ascii = ' =', disable = state_manager.get_item('log_level') in [ 'warn', 'error' ]) as progress:
for frame_number in frame_range: for frame_number in frame_range:
if frame_number % int(video_fps) == 0: if frame_number % int(video_fps) == 0:
vision_frame = get_video_frame(video_path, frame_number) vision_frame = read_video_frame(video_path, frame_number)
if analyse_frame(vision_frame):
counter += 1 if numpy.any(vision_frame):
rate = counter * int(video_fps) / len(frame_range) * 100 total += 1
progress.update()
if analyse_frame(vision_frame):
counter += 1
if counter > 0 and total > 0:
rate = counter / total * 100
progress.set_postfix(rate = rate) progress.set_postfix(rate = rate)
return rate > RATE_LIMIT progress.update()
return bool(rate > 10.0)
def detect_nsfw(vision_frame : VisionFrame) -> bool:
is_nsfw_1 = detect_with_nsfw_1(vision_frame)
is_nsfw_2 = detect_with_nsfw_2(vision_frame)
is_nsfw_3 = detect_with_nsfw_3(vision_frame)
return is_nsfw_1 and is_nsfw_2 or is_nsfw_1 and is_nsfw_3 or is_nsfw_2 and is_nsfw_3
def detect_with_nsfw_1(vision_frame : VisionFrame) -> bool:
detect_vision_frame = prepare_detect_frame(vision_frame, 'nsfw_1')
detection = forward_nsfw(detect_vision_frame, 'nsfw_1')
detection_score = numpy.max(numpy.amax(detection[:, 4:], axis = 1))
return bool(detection_score > 0.2)
def detect_with_nsfw_2(vision_frame : VisionFrame) -> bool:
detect_vision_frame = prepare_detect_frame(vision_frame, 'nsfw_2')
detection = forward_nsfw(detect_vision_frame, 'nsfw_2')
detection_score = detection[0] - detection[1]
return bool(detection_score > 0.25)
def detect_with_nsfw_3(vision_frame : VisionFrame) -> bool:
detect_vision_frame = prepare_detect_frame(vision_frame, 'nsfw_3')
detection = forward_nsfw(detect_vision_frame, 'nsfw_3')
detection_score = (detection[2] + detection[3]) - (detection[0] + detection[1])
return bool(detection_score > 10.5)
def forward_nsfw(vision_frame : VisionFrame, model_name : str) -> Detection:
content_analyser = get_inference_pool().get(model_name)
with conditional_thread_semaphore():
detection = content_analyser.run(None,
{
'input': vision_frame
})[0]
if model_name in [ 'nsfw_2', 'nsfw_3' ]:
return detection[0]
return detection
def prepare_detect_frame(temp_vision_frame : VisionFrame, model_name : str) -> VisionFrame:
model_set = create_static_model_set('full').get(model_name)
model_size = model_set.get('size')
model_mean = model_set.get('mean')
model_standard_deviation = model_set.get('standard_deviation')
detect_vision_frame = fit_contain_frame(temp_vision_frame, model_size)
detect_vision_frame = detect_vision_frame[:, :, ::-1] / 255.0
detect_vision_frame -= model_mean
detect_vision_frame /= model_standard_deviation
detect_vision_frame = numpy.expand_dims(detect_vision_frame.transpose(2, 0, 1), axis = 0).astype(numpy.float32)
return detect_vision_frame
+165 -296
View File
@@ -1,87 +1,86 @@
import inspect
import itertools import itertools
import shutil import shutil
import signal import signal
import sys import sys
from time import time from time import time
import numpy import uvicorn
from facefusion import content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, logger, process_manager, state_manager, voice_extractor, wording import facefusion.apis.core
from facefusion.args import apply_args, collect_job_args, reduce_job_args, reduce_step_args from facefusion import args_helper, benchmarker, cli_helper, content_analyser, hash_helper, logger, state_manager, translator
from facefusion.common_helper import get_first from facefusion.args_helper import apply_args
from facefusion.content_analyser import analyse_image, analyse_video
from facefusion.download import conditional_download_hashes, conditional_download_sources from facefusion.download import conditional_download_hashes, conditional_download_sources
from facefusion.exit_helper import conditional_exit, graceful_exit, hard_exit from facefusion.exit_helper import hard_exit, signal_exit
from facefusion.face_analyser import get_average_face, get_many_faces, get_one_face from facefusion.filesystem import get_file_extension, has_audio, has_image, has_video
from facefusion.face_selector import sort_and_filter_faces from facefusion.filesystem import get_file_name, resolve_file_paths, resolve_file_pattern
from facefusion.face_store import append_reference_face, clear_reference_faces, get_reference_faces
from facefusion.ffmpeg import copy_image, extract_frames, finalize_image, merge_video, replace_audio, restore_audio
from facefusion.filesystem import filter_audio_paths, is_image, is_video, list_directory, resolve_file_pattern
from facefusion.jobs import job_helper, job_manager, job_runner from facefusion.jobs import job_helper, job_manager, job_runner
from facefusion.jobs.job_list import compose_job_list from facefusion.jobs.job_list import compose_job_list
from facefusion.memory import limit_system_memory
from facefusion.processors.core import get_processors_modules from facefusion.processors.core import get_processors_modules
from facefusion.program import create_program from facefusion.program import create_program
from facefusion.program_helper import validate_args from facefusion.program_helper import validate_args
from facefusion.statistics import conditional_log_statistics from facefusion.types import Args, ErrorCode, WorkFlow
from facefusion.temp_helper import clear_temp_directory, create_temp_directory, get_temp_file_path, get_temp_frame_paths, move_temp_file from facefusion.workflows import audio_to_image, audio_to_image_as_frames, image_to_image, image_to_video, image_to_video_as_frames
from facefusion.typing import Args, ErrorCode
from facefusion.vision import get_video_frame, pack_resolution, read_image, read_static_images, restrict_image_resolution, restrict_trim_frame, restrict_video_fps, restrict_video_resolution, unpack_resolution
def cli() -> None: def cli() -> None:
signal.signal(signal.SIGINT, lambda signal_number, frame: graceful_exit(0)) if pre_check():
program = create_program() signal.signal(signal.SIGINT, signal_exit)
program = create_program()
if validate_args(program): if validate_args(program):
args = vars(program.parse_args()) args = vars(program.parse_args())
apply_args(args, state_manager.init_item) apply_args(args, state_manager.init_item)
if state_manager.get_item('command'): if state_manager.get_item('command'):
logger.init(state_manager.get_item('log_level')) logger.init(state_manager.get_item('log_level'))
route(args) route(args)
else:
program.print_help()
else: else:
program.print_help() hard_exit(2)
else: else:
hard_exit(2) hard_exit(2)
def route(args : Args) -> None: def route(args : Args) -> None:
system_memory_limit = state_manager.get_item('system_memory_limit')
if system_memory_limit and system_memory_limit > 0:
limit_system_memory(system_memory_limit)
if state_manager.get_item('command') == 'force-download': if state_manager.get_item('command') == 'force-download':
error_code = force_download() error_code = force_download()
return conditional_exit(error_code) hard_exit(error_code)
if state_manager.get_item('command') == 'benchmark':
if not common_pre_check() or not processors_pre_check() or not benchmarker.pre_check():
hard_exit(2)
benchmarker.render()
if state_manager.get_item('command') == 'api':
if not common_pre_check() or not processors_pre_check() or not facefusion.apis.core.pre_check():
hard_exit(2)
logger.info(translator.get('api_started').format(host = state_manager.get_item('api_host'), port = state_manager.get_item('api_port')), __name__)
uvicorn.run(facefusion.apis.core.create_api(), host = state_manager.get_item('api_host'), port = state_manager.get_item('api_port'))
hard_exit(1)
if state_manager.get_item('command') in [ 'job-list', 'job-create', 'job-submit', 'job-submit-all', 'job-delete', 'job-delete-all', 'job-add-step', 'job-remix-step', 'job-insert-step', 'job-remove-step' ]: if state_manager.get_item('command') in [ 'job-list', 'job-create', 'job-submit', 'job-submit-all', 'job-delete', 'job-delete-all', 'job-add-step', 'job-remix-step', 'job-insert-step', 'job-remove-step' ]:
if not job_manager.init_jobs(state_manager.get_item('jobs_path')): if not job_manager.init_jobs(state_manager.get_jobs_path()):
hard_exit(1) hard_exit(1)
error_code = route_job_manager(args) error_code = route_job_manager(args)
hard_exit(error_code) hard_exit(error_code)
if not pre_check():
return conditional_exit(2)
if state_manager.get_item('command') == 'run':
import facefusion.uis.core as ui
if not common_pre_check() or not processors_pre_check(): if state_manager.get_item('command') == 'run':
return conditional_exit(2) if not job_manager.init_jobs(state_manager.get_jobs_path()):
for ui_layout in ui.get_ui_layouts_modules(state_manager.get_item('ui_layouts')):
if not ui_layout.pre_check():
return conditional_exit(2)
ui.init()
ui.launch()
if state_manager.get_item('command') == 'headless-run':
if not job_manager.init_jobs(state_manager.get_item('jobs_path')):
hard_exit(1) hard_exit(1)
error_core = process_headless(args) error_code = process_headless(args)
hard_exit(error_core) hard_exit(error_code)
if state_manager.get_item('command') == 'batch-run': if state_manager.get_item('command') == 'batch-run':
if not job_manager.init_jobs(state_manager.get_item('jobs_path')): if not job_manager.init_jobs(state_manager.get_jobs_path()):
hard_exit(1) hard_exit(1)
error_core = process_batch(args) error_code = process_batch(args)
hard_exit(error_core) hard_exit(error_code)
if state_manager.get_item('command') in [ 'job-run', 'job-run-all', 'job-retry', 'job-retry-all' ]: if state_manager.get_item('command') in [ 'job-run', 'job-run-all', 'job-retry', 'job-retry-all' ]:
if not job_manager.init_jobs(state_manager.get_item('jobs_path')): if not job_manager.init_jobs(state_manager.get_jobs_path()):
hard_exit(1) hard_exit(1)
error_code = route_job_runner() error_code = route_job_runner()
hard_exit(error_code) hard_exit(error_code)
@@ -89,30 +88,23 @@ def route(args : Args) -> None:
def pre_check() -> bool: def pre_check() -> bool:
if sys.version_info < (3, 10): if sys.version_info < (3, 10):
logger.error(wording.get('python_not_supported').format(version = '3.10'), __name__) logger.error(translator.get('python_not_supported').format(version = '3.10'), __name__)
return False return False
if not shutil.which('curl'): if not shutil.which('curl'):
logger.error(wording.get('curl_not_installed'), __name__) logger.error(translator.get('curl_not_installed'), __name__)
return False return False
if not shutil.which('ffmpeg'): if not shutil.which('ffmpeg'):
logger.error(wording.get('ffmpeg_not_installed'), __name__) logger.error(translator.get('ffmpeg_not_installed'), __name__)
return False return False
return True return True
def common_pre_check() -> bool: def common_pre_check() -> bool:
common_modules =\ content_analyser_content = inspect.getsource(content_analyser).encode()
[
content_analyser,
face_classifier,
face_detector,
face_landmarker,
face_masker,
face_recognizer,
voice_extractor
]
return all(module.pre_check() for module in common_modules) return hash_helper.create_hash(content_analyser_content) == '975d67d6'
def processors_pre_check() -> bool: def processors_pre_check() -> bool:
@@ -123,27 +115,24 @@ def processors_pre_check() -> bool:
def force_download() -> ErrorCode: def force_download() -> ErrorCode:
common_modules =\ download_scope = state_manager.get_item('download_scope')
[ available_processors = [ get_file_name(file_path) for file_path in resolve_file_paths('facefusion/processors/modules') ]
content_analyser,
face_classifier,
face_detector,
face_landmarker,
face_masker,
face_recognizer,
voice_extractor
]
available_processors = [ file.get('name') for file in list_directory('facefusion/processors/modules') ]
processor_modules = get_processors_modules(available_processors) processor_modules = get_processors_modules(available_processors)
common_modules = []
for processor_module in processor_modules:
for common_module in processor_module.get_common_modules():
if common_module not in common_modules:
common_modules.append(common_module)
for module in common_modules + processor_modules: for module in common_modules + processor_modules:
if hasattr(module, 'create_static_model_set'): if hasattr(module, 'create_static_model_set'):
for model in module.create_static_model_set(state_manager.get_item('download_scope')).values(): for model in module.create_static_model_set(download_scope).values():
model_hashes = model.get('hashes') model_hash_set = model.get('hashes')
model_sources = model.get('sources') model_source_set = model.get('sources')
if model_hashes and model_sources: if model_hash_set and model_source_set:
if not conditional_download_hashes(model_hashes) or not conditional_download_sources(model_sources): if not conditional_download_hashes(model_hash_set) or not conditional_download_sources(model_source_set):
return 1 return 1
return 0 return 0
@@ -154,107 +143,119 @@ def route_job_manager(args : Args) -> ErrorCode:
job_headers, job_contents = compose_job_list(state_manager.get_item('job_status')) job_headers, job_contents = compose_job_list(state_manager.get_item('job_status'))
if job_contents: if job_contents:
logger.table(job_headers, job_contents) cli_helper.render_table(job_headers, job_contents)
return 0 return 0
return 1 return 1
if state_manager.get_item('command') == 'job-create': if state_manager.get_item('command') == 'job-create':
if job_manager.create_job(state_manager.get_item('job_id')): if job_manager.create_job(state_manager.get_item('job_id')):
logger.info(wording.get('job_created').format(job_id = state_manager.get_item('job_id')), __name__) logger.info(translator.get('job_created').format(job_id = state_manager.get_item('job_id')), __name__)
return 0 return 0
logger.error(wording.get('job_not_created').format(job_id = state_manager.get_item('job_id')), __name__) logger.error(translator.get('job_not_created').format(job_id = state_manager.get_item('job_id')), __name__)
return 1 return 1
if state_manager.get_item('command') == 'job-submit': if state_manager.get_item('command') == 'job-submit':
if job_manager.submit_job(state_manager.get_item('job_id')): if job_manager.submit_job(state_manager.get_item('job_id')):
logger.info(wording.get('job_submitted').format(job_id = state_manager.get_item('job_id')), __name__) logger.info(translator.get('job_submitted').format(job_id = state_manager.get_item('job_id')), __name__)
return 0 return 0
logger.error(wording.get('job_not_submitted').format(job_id = state_manager.get_item('job_id')), __name__) logger.error(translator.get('job_not_submitted').format(job_id = state_manager.get_item('job_id')), __name__)
return 1 return 1
if state_manager.get_item('command') == 'job-submit-all': if state_manager.get_item('command') == 'job-submit-all':
if job_manager.submit_jobs(): if job_manager.submit_jobs(state_manager.get_item('halt_on_error')):
logger.info(wording.get('job_all_submitted'), __name__) logger.info(translator.get('job_all_submitted'), __name__)
return 0 return 0
logger.error(wording.get('job_all_not_submitted'), __name__) logger.error(translator.get('job_all_not_submitted'), __name__)
return 1 return 1
if state_manager.get_item('command') == 'job-delete': if state_manager.get_item('command') == 'job-delete':
if job_manager.delete_job(state_manager.get_item('job_id')): if job_manager.delete_job(state_manager.get_item('job_id')):
logger.info(wording.get('job_deleted').format(job_id = state_manager.get_item('job_id')), __name__) logger.info(translator.get('job_deleted').format(job_id = state_manager.get_item('job_id')), __name__)
return 0 return 0
logger.error(wording.get('job_not_deleted').format(job_id = state_manager.get_item('job_id')), __name__) logger.error(translator.get('job_not_deleted').format(job_id = state_manager.get_item('job_id')), __name__)
return 1 return 1
if state_manager.get_item('command') == 'job-delete-all': if state_manager.get_item('command') == 'job-delete-all':
if job_manager.delete_jobs(): if job_manager.delete_jobs(state_manager.get_item('halt_on_error')):
logger.info(wording.get('job_all_deleted'), __name__) logger.info(translator.get('job_all_deleted'), __name__)
return 0 return 0
logger.error(wording.get('job_all_not_deleted'), __name__) logger.error(translator.get('job_all_not_deleted'), __name__)
return 1 return 1
if state_manager.get_item('command') == 'job-add-step': if state_manager.get_item('command') == 'job-add-step':
step_args = reduce_step_args(args) step_args = args_helper.filter_step_args(args)
if job_manager.add_step(state_manager.get_item('job_id'), step_args): if job_manager.add_step(state_manager.get_item('job_id'), step_args):
logger.info(wording.get('job_step_added').format(job_id = state_manager.get_item('job_id')), __name__) logger.info(translator.get('job_step_added').format(job_id = state_manager.get_item('job_id')), __name__)
return 0 return 0
logger.error(wording.get('job_step_not_added').format(job_id = state_manager.get_item('job_id')), __name__) logger.error(translator.get('job_step_not_added').format(job_id = state_manager.get_item('job_id')), __name__)
return 1 return 1
if state_manager.get_item('command') == 'job-remix-step': if state_manager.get_item('command') == 'job-remix-step':
step_args = reduce_step_args(args) step_args = args_helper.filter_step_args(args)
if job_manager.remix_step(state_manager.get_item('job_id'), state_manager.get_item('step_index'), step_args): if job_manager.remix_step(state_manager.get_item('job_id'), state_manager.get_item('step_index'), step_args):
logger.info(wording.get('job_remix_step_added').format(job_id = state_manager.get_item('job_id'), step_index = state_manager.get_item('step_index')), __name__) logger.info(translator.get('job_remix_step_added').format(job_id = state_manager.get_item('job_id'), step_index = state_manager.get_item('step_index')), __name__)
return 0 return 0
logger.error(wording.get('job_remix_step_not_added').format(job_id = state_manager.get_item('job_id'), step_index = state_manager.get_item('step_index')), __name__) logger.error(translator.get('job_remix_step_not_added').format(job_id = state_manager.get_item('job_id'), step_index = state_manager.get_item('step_index')), __name__)
return 1 return 1
if state_manager.get_item('command') == 'job-insert-step': if state_manager.get_item('command') == 'job-insert-step':
step_args = reduce_step_args(args) step_args = args_helper.filter_step_args(args)
if job_manager.insert_step(state_manager.get_item('job_id'), state_manager.get_item('step_index'), step_args): if job_manager.insert_step(state_manager.get_item('job_id'), state_manager.get_item('step_index'), step_args):
logger.info(wording.get('job_step_inserted').format(job_id = state_manager.get_item('job_id'), step_index = state_manager.get_item('step_index')), __name__) logger.info(translator.get('job_step_inserted').format(job_id = state_manager.get_item('job_id'), step_index = state_manager.get_item('step_index')), __name__)
return 0 return 0
logger.error(wording.get('job_step_not_inserted').format(job_id = state_manager.get_item('job_id'), step_index = state_manager.get_item('step_index')), __name__) logger.error(translator.get('job_step_not_inserted').format(job_id = state_manager.get_item('job_id'), step_index = state_manager.get_item('step_index')), __name__)
return 1 return 1
if state_manager.get_item('command') == 'job-remove-step': if state_manager.get_item('command') == 'job-remove-step':
if job_manager.remove_step(state_manager.get_item('job_id'), state_manager.get_item('step_index')): if job_manager.remove_step(state_manager.get_item('job_id'), state_manager.get_item('step_index')):
logger.info(wording.get('job_step_removed').format(job_id = state_manager.get_item('job_id'), step_index = state_manager.get_item('step_index')), __name__) logger.info(translator.get('job_step_removed').format(job_id = state_manager.get_item('job_id'), step_index = state_manager.get_item('step_index')), __name__)
return 0 return 0
logger.error(wording.get('job_step_not_removed').format(job_id = state_manager.get_item('job_id'), step_index = state_manager.get_item('step_index')), __name__) logger.error(translator.get('job_step_not_removed').format(job_id = state_manager.get_item('job_id'), step_index = state_manager.get_item('step_index')), __name__)
return 1 return 1
return 1 return 1
def route_job_runner() -> ErrorCode: def route_job_runner() -> ErrorCode:
if state_manager.get_item('command') == 'job-run': if state_manager.get_item('command') == 'job-run':
logger.info(wording.get('running_job').format(job_id = state_manager.get_item('job_id')), __name__) logger.info(translator.get('running_job').format(job_id = state_manager.get_item('job_id')), __name__)
if job_runner.run_job(state_manager.get_item('job_id'), process_step): if job_runner.run_job(state_manager.get_item('job_id'), process_step):
logger.info(wording.get('processing_job_succeed').format(job_id = state_manager.get_item('job_id')), __name__) logger.info(translator.get('processing_job_succeeded').format(job_id = state_manager.get_item('job_id')), __name__)
return 0 return 0
logger.info(wording.get('processing_job_failed').format(job_id = state_manager.get_item('job_id')), __name__) logger.info(translator.get('processing_job_failed').format(job_id = state_manager.get_item('job_id')), __name__)
return 1 return 1
if state_manager.get_item('command') == 'job-run-all': if state_manager.get_item('command') == 'job-run-all':
logger.info(wording.get('running_jobs'), __name__) logger.info(translator.get('running_jobs'), __name__)
if job_runner.run_jobs(process_step): if job_runner.run_jobs(process_step, state_manager.get_item('halt_on_error')):
logger.info(wording.get('processing_jobs_succeed'), __name__) logger.info(translator.get('processing_jobs_succeeded'), __name__)
return 0 return 0
logger.info(wording.get('processing_jobs_failed'), __name__) logger.info(translator.get('processing_jobs_failed'), __name__)
return 1 return 1
if state_manager.get_item('command') == 'job-retry': if state_manager.get_item('command') == 'job-retry':
logger.info(wording.get('retrying_job').format(job_id = state_manager.get_item('job_id')), __name__) logger.info(translator.get('retrying_job').format(job_id = state_manager.get_item('job_id')), __name__)
if job_runner.retry_job(state_manager.get_item('job_id'), process_step): if job_runner.retry_job(state_manager.get_item('job_id'), process_step):
logger.info(wording.get('processing_job_succeed').format(job_id = state_manager.get_item('job_id')), __name__) logger.info(translator.get('processing_job_succeeded').format(job_id = state_manager.get_item('job_id')), __name__)
return 0 return 0
logger.info(wording.get('processing_job_failed').format(job_id = state_manager.get_item('job_id')), __name__) logger.info(translator.get('processing_job_failed').format(job_id = state_manager.get_item('job_id')), __name__)
return 1 return 1
if state_manager.get_item('command') == 'job-retry-all': if state_manager.get_item('command') == 'job-retry-all':
logger.info(wording.get('retrying_jobs'), __name__) logger.info(translator.get('retrying_jobs'), __name__)
if job_runner.retry_jobs(process_step): if job_runner.retry_jobs(process_step, state_manager.get_item('halt_on_error')):
logger.info(wording.get('processing_jobs_succeed'), __name__) logger.info(translator.get('processing_jobs_succeeded'), __name__)
return 0 return 0
logger.info(wording.get('processing_jobs_failed'), __name__) logger.info(translator.get('processing_jobs_failed'), __name__)
return 1 return 1
return 2 return 2
def process_headless(args : Args) -> ErrorCode: def process_headless(args : Args) -> ErrorCode:
job_id = job_helper.suggest_job_id('headless') job_id = job_helper.suggest_job_id('headless')
step_args = reduce_step_args(args) step_args = args_helper.filter_step_args(args)
if job_manager.create_job(job_id) and job_manager.add_step(job_id, step_args) and job_manager.submit_job(job_id) and job_runner.run_job(job_id, process_step): if job_manager.create_job(job_id) and job_manager.add_step(job_id, step_args) and job_manager.submit_job(job_id) and job_runner.run_job(job_id, process_step):
return 0 return 0
@@ -263,17 +264,21 @@ def process_headless(args : Args) -> ErrorCode:
def process_batch(args : Args) -> ErrorCode: def process_batch(args : Args) -> ErrorCode:
job_id = job_helper.suggest_job_id('batch') job_id = job_helper.suggest_job_id('batch')
step_args = reduce_step_args(args) step_args = args_helper.filter_step_args(args)
job_args = reduce_job_args(args) source_paths = resolve_file_pattern(step_args.get('source_pattern'))
source_paths = resolve_file_pattern(job_args.get('source_pattern')) target_paths = resolve_file_pattern(step_args.get('target_pattern'))
target_paths = resolve_file_pattern(job_args.get('target_pattern'))
if job_manager.create_job(job_id): if job_manager.create_job(job_id):
if source_paths and target_paths: if source_paths and target_paths:
for index, (source_path, target_path) in enumerate(itertools.product(source_paths, target_paths)): for index, (source_path, target_path) in enumerate(itertools.product(source_paths, target_paths)):
step_args['source_paths'] = [ source_path ] step_args['source_paths'] = [ source_path ]
step_args['target_path'] = target_path step_args['target_path'] = target_path
step_args['output_path'] = job_args.get('output_pattern').format(index = index)
try:
step_args['output_path'] = step_args.get('output_pattern').format(index = index, source_name = get_file_name(source_path), target_name = get_file_name(target_path), target_extension = get_file_extension(target_path))
except KeyError:
return 1
if not job_manager.add_step(job_id, step_args): if not job_manager.add_step(job_id, step_args):
return 1 return 1
if job_manager.submit_job(job_id) and job_runner.run_job(job_id, process_step): if job_manager.submit_job(job_id) and job_runner.run_job(job_id, process_step):
@@ -282,7 +287,12 @@ def process_batch(args : Args) -> ErrorCode:
if not source_paths and target_paths: if not source_paths and target_paths:
for index, target_path in enumerate(target_paths): for index, target_path in enumerate(target_paths):
step_args['target_path'] = target_path step_args['target_path'] = target_path
step_args['output_path'] = job_args.get('output_pattern').format(index = index)
try:
step_args['output_path'] = step_args.get('output_pattern').format(index = index, target_name = get_file_name(target_path), target_extension = get_file_extension(target_path))
except KeyError:
return 1
if not job_manager.add_step(job_id, step_args): if not job_manager.add_step(job_id, step_args):
return 1 return 1
if job_manager.submit_job(job_id) and job_runner.run_job(job_id, process_step): if job_manager.submit_job(job_id) and job_runner.run_job(job_id, process_step):
@@ -291,12 +301,13 @@ def process_batch(args : Args) -> ErrorCode:
def process_step(job_id : str, step_index : int, step_args : Args) -> bool: def process_step(job_id : str, step_index : int, step_args : Args) -> bool:
clear_reference_faces()
step_total = job_manager.count_step_total(job_id) step_total = job_manager.count_step_total(job_id)
step_args.update(collect_job_args()) cli_args = args_helper.extract_cli_args(state_manager.get_state())
apply_args(step_args, state_manager.set_item) args = cli_args.copy()
args.update(step_args)
apply_args(args, state_manager.set_item)
logger.info(wording.get('processing_step').format(step_current = step_index + 1, step_total = step_total), __name__) logger.info(translator.get('processing_step').format(step_current = step_index + 1, step_total = step_total), __name__)
if common_pre_check() and processors_pre_check(): if common_pre_check() and processors_pre_check():
error_code = conditional_process() error_code = conditional_process()
return error_code == 0 return error_code == 0
@@ -305,179 +316,37 @@ def process_step(job_id : str, step_index : int, step_args : Args) -> bool:
def conditional_process() -> ErrorCode: def conditional_process() -> ErrorCode:
start_time = time() start_time = time()
if state_manager.get_item('workflow') == 'auto':
state_manager.set_item('workflow', detect_workflow())
for processor_module in get_processors_modules(state_manager.get_item('processors')): for processor_module in get_processors_modules(state_manager.get_item('processors')):
if not processor_module.pre_process('output'): if not processor_module.pre_process('output'):
return 2 return 2
conditional_append_reference_faces()
if is_image(state_manager.get_item('target_path')): if state_manager.get_item('workflow') == 'audio-to-image:video':
return process_image(start_time) return audio_to_image.process(start_time)
if is_video(state_manager.get_item('target_path')): if state_manager.get_item('workflow') == 'audio-to-image:frames':
return process_video(start_time) return audio_to_image_as_frames.process(start_time)
if state_manager.get_item('workflow') == 'image-to-image':
return image_to_image.process(start_time)
if state_manager.get_item('workflow') == 'image-to-video':
return image_to_video.process(start_time)
if state_manager.get_item('workflow') == 'image-to-video:frames':
return image_to_video_as_frames.process(start_time)
return 0 return 0
def conditional_append_reference_faces() -> None: def detect_workflow() -> WorkFlow:
if 'reference' in state_manager.get_item('face_selector_mode') and not get_reference_faces(): if has_video([ state_manager.get_item('target_path') ]):
source_frames = read_static_images(state_manager.get_item('source_paths')) if get_file_extension(state_manager.get_item('output_path')):
source_faces = get_many_faces(source_frames) return 'image-to-video'
source_face = get_average_face(source_faces) return 'image-to-video:frames'
if is_video(state_manager.get_item('target_path')):
reference_frame = get_video_frame(state_manager.get_item('target_path'), state_manager.get_item('reference_frame_number'))
else:
reference_frame = read_image(state_manager.get_item('target_path'))
reference_faces = sort_and_filter_faces(get_many_faces([ reference_frame ]))
reference_face = get_one_face(reference_faces, state_manager.get_item('reference_face_position'))
append_reference_face('origin', reference_face)
if source_face and reference_face: if has_audio(state_manager.get_item('source_paths')) and has_image([ state_manager.get_item('target_path') ]):
for processor_module in get_processors_modules(state_manager.get_item('processors')): if get_file_extension(state_manager.get_item('output_path')):
abstract_reference_frame = processor_module.get_reference_frame(source_face, reference_face, reference_frame) return 'audio-to-image:video'
if numpy.any(abstract_reference_frame): return 'audio-to-image:frames'
abstract_reference_faces = sort_and_filter_faces(get_many_faces([ abstract_reference_frame ]))
abstract_reference_face = get_one_face(abstract_reference_faces, state_manager.get_item('reference_face_position'))
append_reference_face(processor_module.__name__, abstract_reference_face)
return 'image-to-image'
def process_image(start_time : float) -> ErrorCode:
if analyse_image(state_manager.get_item('target_path')):
return 3
# clear temp
logger.debug(wording.get('clearing_temp'), __name__)
clear_temp_directory(state_manager.get_item('target_path'))
# create temp
logger.debug(wording.get('creating_temp'), __name__)
create_temp_directory(state_manager.get_item('target_path'))
# copy image
process_manager.start()
temp_image_resolution = pack_resolution(restrict_image_resolution(state_manager.get_item('target_path'), unpack_resolution(state_manager.get_item('output_image_resolution'))))
logger.info(wording.get('copying_image').format(resolution = temp_image_resolution), __name__)
if copy_image(state_manager.get_item('target_path'), temp_image_resolution):
logger.debug(wording.get('copying_image_succeed'), __name__)
else:
logger.error(wording.get('copying_image_failed'), __name__)
process_manager.end()
return 1
# process image
temp_file_path = get_temp_file_path(state_manager.get_item('target_path'))
for processor_module in get_processors_modules(state_manager.get_item('processors')):
logger.info(wording.get('processing'), processor_module.__name__)
processor_module.process_image(state_manager.get_item('source_paths'), temp_file_path, temp_file_path)
processor_module.post_process()
if is_process_stopping():
process_manager.end()
return 4
# finalize image
logger.info(wording.get('finalizing_image').format(resolution = state_manager.get_item('output_image_resolution')), __name__)
if finalize_image(state_manager.get_item('target_path'), state_manager.get_item('output_path'), state_manager.get_item('output_image_resolution')):
logger.debug(wording.get('finalizing_image_succeed'), __name__)
else:
logger.warn(wording.get('finalizing_image_skipped'), __name__)
# clear temp
logger.debug(wording.get('clearing_temp'), __name__)
clear_temp_directory(state_manager.get_item('target_path'))
# validate image
if is_image(state_manager.get_item('output_path')):
seconds = '{:.2f}'.format((time() - start_time) % 60)
logger.info(wording.get('processing_image_succeed').format(seconds = seconds), __name__)
conditional_log_statistics()
else:
logger.error(wording.get('processing_image_failed'), __name__)
process_manager.end()
return 1
process_manager.end()
return 0
def process_video(start_time : float) -> ErrorCode:
trim_frame_start, trim_frame_end = restrict_trim_frame(state_manager.get_item('target_path'), state_manager.get_item('trim_frame_start'), state_manager.get_item('trim_frame_end'))
if analyse_video(state_manager.get_item('target_path'), trim_frame_start, trim_frame_end):
return 3
# clear temp
logger.debug(wording.get('clearing_temp'), __name__)
clear_temp_directory(state_manager.get_item('target_path'))
# create temp
logger.debug(wording.get('creating_temp'), __name__)
create_temp_directory(state_manager.get_item('target_path'))
# extract frames
process_manager.start()
temp_video_resolution = pack_resolution(restrict_video_resolution(state_manager.get_item('target_path'), unpack_resolution(state_manager.get_item('output_video_resolution'))))
temp_video_fps = restrict_video_fps(state_manager.get_item('target_path'), state_manager.get_item('output_video_fps'))
logger.info(wording.get('extracting_frames').format(resolution = temp_video_resolution, fps = temp_video_fps), __name__)
if extract_frames(state_manager.get_item('target_path'), temp_video_resolution, temp_video_fps, trim_frame_start, trim_frame_end):
logger.debug(wording.get('extracting_frames_succeed'), __name__)
else:
if is_process_stopping():
process_manager.end()
return 4
logger.error(wording.get('extracting_frames_failed'), __name__)
process_manager.end()
return 1
# process frames
temp_frame_paths = get_temp_frame_paths(state_manager.get_item('target_path'))
if temp_frame_paths:
for processor_module in get_processors_modules(state_manager.get_item('processors')):
logger.info(wording.get('processing'), processor_module.__name__)
processor_module.process_video(state_manager.get_item('source_paths'), temp_frame_paths)
processor_module.post_process()
if is_process_stopping():
return 4
else:
logger.error(wording.get('temp_frames_not_found'), __name__)
process_manager.end()
return 1
# merge video
logger.info(wording.get('merging_video').format(resolution = state_manager.get_item('output_video_resolution'), fps = state_manager.get_item('output_video_fps')), __name__)
if merge_video(state_manager.get_item('target_path'), state_manager.get_item('output_video_resolution'), state_manager.get_item('output_video_fps')):
logger.debug(wording.get('merging_video_succeed'), __name__)
else:
if is_process_stopping():
process_manager.end()
return 4
logger.error(wording.get('merging_video_failed'), __name__)
process_manager.end()
return 1
# handle audio
if state_manager.get_item('skip_audio'):
logger.info(wording.get('skipping_audio'), __name__)
move_temp_file(state_manager.get_item('target_path'), state_manager.get_item('output_path'))
else:
source_audio_path = get_first(filter_audio_paths(state_manager.get_item('source_paths')))
if source_audio_path:
if replace_audio(state_manager.get_item('target_path'), source_audio_path, state_manager.get_item('output_path')):
logger.debug(wording.get('replacing_audio_succeed'), __name__)
else:
if is_process_stopping():
process_manager.end()
return 4
logger.warn(wording.get('replacing_audio_skipped'), __name__)
move_temp_file(state_manager.get_item('target_path'), state_manager.get_item('output_path'))
else:
if restore_audio(state_manager.get_item('target_path'), state_manager.get_item('output_path'), state_manager.get_item('output_video_fps'), trim_frame_start, trim_frame_end):
logger.debug(wording.get('restoring_audio_succeed'), __name__)
else:
if is_process_stopping():
process_manager.end()
return 4
logger.warn(wording.get('restoring_audio_skipped'), __name__)
move_temp_file(state_manager.get_item('target_path'), state_manager.get_item('output_path'))
# clear temp
logger.debug(wording.get('clearing_temp'), __name__)
clear_temp_directory(state_manager.get_item('target_path'))
# validate video
if is_video(state_manager.get_item('output_path')):
seconds = '{:.2f}'.format((time() - start_time))
logger.info(wording.get('processing_video_succeed').format(seconds = seconds), __name__)
conditional_log_statistics()
else:
logger.error(wording.get('processing_video_failed'), __name__)
process_manager.end()
return 1
process_manager.end()
return 0
def is_process_stopping() -> bool:
if process_manager.is_stopping():
process_manager.end()
logger.info(wording.get('processing_stopped'), __name__)
return process_manager.is_pending()
+32
View File
@@ -0,0 +1,32 @@
import itertools
import shutil
from typing import List
from facefusion import metadata
from facefusion.types import Command
def run(commands : List[Command]) -> List[Command]:
user_agent = metadata.get('name') + '/' + metadata.get('version')
return [ shutil.which('curl'), '--user-agent', user_agent, '--location', '--silent', '--ssl-no-revoke' ] + commands
def chain(*commands : List[Command]) -> List[Command]:
return list(itertools.chain(*commands))
def ping(url : str) -> List[Command]:
return [ '-I', url ]
def download(url : str, download_file_path : str) -> List[Command]:
return [ '--create-dirs', '--continue-at', '-', '--output', download_file_path, url ]
def set_timeout(timeout : int) -> List[Command]:
return [ '--connect-timeout', str(timeout) ]
def set_retry(retry : int) -> List[Command]:
return [ '--retry', str(retry) ]
+54 -42
View File
@@ -1,5 +1,4 @@
import os import os
import shutil
import subprocess import subprocess
from functools import lru_cache from functools import lru_cache
from typing import List, Optional, Tuple from typing import List, Optional, Tuple
@@ -8,15 +7,14 @@ from urllib.parse import urlparse
from tqdm import tqdm from tqdm import tqdm
import facefusion.choices import facefusion.choices
from facefusion import logger, process_manager, state_manager, wording from facefusion import curl_builder, logger, process_manager, state_manager, translator
from facefusion.filesystem import get_file_size, is_file, remove_file from facefusion.filesystem import get_file_name, get_file_size, is_file, remove_file
from facefusion.hash_helper import validate_hash from facefusion.hash_helper import validate_hash
from facefusion.typing import DownloadProvider, DownloadSet from facefusion.types import Buffer, Command, DownloadProvider, DownloadSet
def open_curl(args : List[str]) -> subprocess.Popen[bytes]: def open_curl(commands : List[Command]) -> subprocess.Popen[Buffer]:
commands = [ shutil.which('curl'), '--silent', '--insecure', '--location' ] commands = curl_builder.run(commands)
commands.extend(args)
return subprocess.Popen(commands, stdin = subprocess.PIPE, stdout = subprocess.PIPE) return subprocess.Popen(commands, stdin = subprocess.PIPE, stdout = subprocess.PIPE)
@@ -28,8 +26,12 @@ def conditional_download(download_directory_path : str, urls : List[str]) -> Non
download_size = get_static_download_size(url) download_size = get_static_download_size(url)
if initial_size < download_size: if initial_size < download_size:
with tqdm(total = download_size, initial = initial_size, desc = wording.get('downloading'), unit = 'B', unit_scale = True, unit_divisor = 1024, ascii = ' =', disable = state_manager.get_item('log_level') in [ 'warn', 'error' ]) as progress: with tqdm(total = download_size, initial = initial_size, desc = translator.get('downloading'), unit = 'B', unit_scale = True, unit_divisor = 1024, ascii = ' =', disable = state_manager.get_item('log_level') in [ 'warn', 'error' ]) as progress:
commands = [ '--create-dirs', '--continue-at', '-', '--output', download_file_path, url ] commands = curl_builder.chain(
curl_builder.download(url, download_file_path),
curl_builder.set_timeout(5),
curl_builder.set_retry(5)
)
open_curl(commands) open_curl(commands)
current_size = initial_size current_size = initial_size
progress.set_postfix(download_providers = state_manager.get_item('download_providers'), file_name = download_file_name) progress.set_postfix(download_providers = state_manager.get_item('download_providers'), file_name = download_file_name)
@@ -40,9 +42,12 @@ def conditional_download(download_directory_path : str, urls : List[str]) -> Non
progress.update(current_size - progress.n) progress.update(current_size - progress.n)
@lru_cache(maxsize = None) @lru_cache(maxsize = 64)
def get_static_download_size(url : str) -> int: def get_static_download_size(url : str) -> int:
commands = [ '-I', url ] commands = curl_builder.chain(
curl_builder.ping(url),
curl_builder.set_timeout(5)
)
process = open_curl(commands) process = open_curl(commands)
lines = reversed(process.stdout.readlines()) lines = reversed(process.stdout.readlines())
@@ -55,65 +60,68 @@ def get_static_download_size(url : str) -> int:
return 0 return 0
@lru_cache(maxsize = None) @lru_cache(maxsize = 64)
def ping_static_url(url : str) -> bool: def ping_static_url(url : str) -> bool:
commands = [ '-I', url ] commands = curl_builder.chain(
curl_builder.ping(url),
curl_builder.set_timeout(5)
)
process = open_curl(commands) process = open_curl(commands)
process.communicate() process.communicate()
return process.returncode == 0 return process.returncode == 0
def conditional_download_hashes(hashes : DownloadSet) -> bool: def conditional_download_hashes(hash_set : DownloadSet) -> bool:
hash_paths = [ hashes.get(hash_key).get('path') for hash_key in hashes.keys() ] hash_paths = [ hash_set.get(hash_key).get('path') for hash_key in hash_set.keys() ]
process_manager.check() process_manager.check()
_, invalid_hash_paths = validate_hash_paths(hash_paths) _, invalid_hash_paths = validate_hash_paths(hash_paths)
if invalid_hash_paths: if invalid_hash_paths:
for index in hashes: for index in hash_set:
if hashes.get(index).get('path') in invalid_hash_paths: if hash_set.get(index).get('path') in invalid_hash_paths:
invalid_hash_url = hashes.get(index).get('url') invalid_hash_url = hash_set.get(index).get('url')
if invalid_hash_url: if invalid_hash_url:
download_directory_path = os.path.dirname(hashes.get(index).get('path')) download_directory_path = os.path.dirname(hash_set.get(index).get('path'))
conditional_download(download_directory_path, [ invalid_hash_url ]) conditional_download(download_directory_path, [ invalid_hash_url ])
valid_hash_paths, invalid_hash_paths = validate_hash_paths(hash_paths) valid_hash_paths, invalid_hash_paths = validate_hash_paths(hash_paths)
for valid_hash_path in valid_hash_paths: for valid_hash_path in valid_hash_paths:
valid_hash_file_name, _ = os.path.splitext(os.path.basename(valid_hash_path)) valid_hash_file_name = get_file_name(valid_hash_path)
logger.debug(wording.get('validating_hash_succeed').format(hash_file_name = valid_hash_file_name), __name__) logger.debug(translator.get('validating_hash_succeeded').format(hash_file_name = valid_hash_file_name), __name__)
for invalid_hash_path in invalid_hash_paths: for invalid_hash_path in invalid_hash_paths:
invalid_hash_file_name, _ = os.path.splitext(os.path.basename(invalid_hash_path)) invalid_hash_file_name = get_file_name(invalid_hash_path)
logger.error(wording.get('validating_hash_failed').format(hash_file_name = invalid_hash_file_name), __name__) logger.error(translator.get('validating_hash_failed').format(hash_file_name = invalid_hash_file_name), __name__)
if not invalid_hash_paths: if not invalid_hash_paths:
process_manager.end() process_manager.end()
return not invalid_hash_paths return not invalid_hash_paths
def conditional_download_sources(sources : DownloadSet) -> bool: def conditional_download_sources(source_set : DownloadSet) -> bool:
source_paths = [ sources.get(source_key).get('path') for source_key in sources.keys() ] source_paths = [ source_set.get(source_key).get('path') for source_key in source_set.keys() ]
process_manager.check() process_manager.check()
_, invalid_source_paths = validate_source_paths(source_paths) _, invalid_source_paths = validate_source_paths(source_paths)
if invalid_source_paths: if invalid_source_paths:
for index in sources: for index in source_set:
if sources.get(index).get('path') in invalid_source_paths: if source_set.get(index).get('path') in invalid_source_paths:
invalid_source_url = sources.get(index).get('url') invalid_source_url = source_set.get(index).get('url')
if invalid_source_url: if invalid_source_url:
download_directory_path = os.path.dirname(sources.get(index).get('path')) download_directory_path = os.path.dirname(source_set.get(index).get('path'))
conditional_download(download_directory_path, [ invalid_source_url ]) conditional_download(download_directory_path, [ invalid_source_url ])
valid_source_paths, invalid_source_paths = validate_source_paths(source_paths) valid_source_paths, invalid_source_paths = validate_source_paths(source_paths)
for valid_source_path in valid_source_paths: for valid_source_path in valid_source_paths:
valid_source_file_name, _ = os.path.splitext(os.path.basename(valid_source_path)) valid_source_file_name = get_file_name(valid_source_path)
logger.debug(wording.get('validating_source_succeed').format(source_file_name = valid_source_file_name), __name__) logger.debug(translator.get('validating_source_succeeded').format(source_file_name = valid_source_file_name), __name__)
for invalid_source_path in invalid_source_paths: for invalid_source_path in invalid_source_paths:
invalid_source_file_name, _ = os.path.splitext(os.path.basename(invalid_source_path)) invalid_source_file_name = get_file_name(invalid_source_path)
logger.error(wording.get('validating_source_failed').format(source_file_name = invalid_source_file_name), __name__) logger.error(translator.get('validating_source_failed').format(source_file_name = invalid_source_file_name), __name__)
if remove_file(invalid_source_path): if remove_file(invalid_source_path):
logger.error(wording.get('deleting_corrupt_source').format(source_file_name = invalid_source_file_name), __name__) logger.error(translator.get('deleting_corrupt_source').format(source_file_name = invalid_source_file_name), __name__)
if not invalid_source_paths: if not invalid_source_paths:
process_manager.end() process_manager.end()
@@ -129,6 +137,7 @@ def validate_hash_paths(hash_paths : List[str]) -> Tuple[List[str], List[str]]:
valid_hash_paths.append(hash_path) valid_hash_paths.append(hash_path)
else: else:
invalid_hash_paths.append(hash_path) invalid_hash_paths.append(hash_path)
return valid_hash_paths, invalid_hash_paths return valid_hash_paths, invalid_hash_paths
@@ -141,6 +150,7 @@ def validate_source_paths(source_paths : List[str]) -> Tuple[List[str], List[str
valid_source_paths.append(source_path) valid_source_paths.append(source_path)
else: else:
invalid_source_paths.append(source_path) invalid_source_paths.append(source_path)
return valid_source_paths, invalid_source_paths return valid_source_paths, invalid_source_paths
@@ -148,16 +158,18 @@ def resolve_download_url(base_name : str, file_name : str) -> Optional[str]:
download_providers = state_manager.get_item('download_providers') download_providers = state_manager.get_item('download_providers')
for download_provider in download_providers: for download_provider in download_providers:
if ping_download_provider(download_provider): download_url = resolve_download_url_by_provider(download_provider, base_name, file_name)
return resolve_download_url_by_provider(download_provider, base_name, file_name) if download_url:
return download_url
return None return None
def ping_download_provider(download_provider : DownloadProvider) -> bool:
download_provider_value = facefusion.choices.download_provider_set.get(download_provider)
return ping_static_url(download_provider_value.get('url'))
def resolve_download_url_by_provider(download_provider : DownloadProvider, base_name : str, file_name : str) -> Optional[str]: def resolve_download_url_by_provider(download_provider : DownloadProvider, base_name : str, file_name : str) -> Optional[str]:
download_provider_value = facefusion.choices.download_provider_set.get(download_provider) download_provider_value = facefusion.choices.download_provider_set.get(download_provider)
return download_provider_value.get('url') + download_provider_value.get('path').format(base_name = base_name, file_name = file_name)
for download_provider_url in download_provider_value.get('urls'):
if ping_static_url(download_provider_url):
return download_provider_url + download_provider_value.get('path').format(base_name = base_name, file_name = file_name)
return None
+90 -88
View File
@@ -1,15 +1,15 @@
import shutil import os
import subprocess
import xml.etree.ElementTree as ElementTree
from functools import lru_cache from functools import lru_cache
from typing import Any, List, Optional from typing import List, Tuple
from onnxruntime import get_available_providers, set_default_logger_severity import onnxruntime
import facefusion.choices import facefusion.choices
from facefusion.typing import ExecutionDevice, ExecutionProvider, ValueAndUnit from facefusion.filesystem import create_directory, is_directory
from facefusion.system import detect_graphic_devices
from facefusion.types import ExecutionProvider, InferenceOptionSet, InferenceProvider
set_default_logger_severity(3) onnxruntime.set_default_logger_severity(3)
def has_execution_provider(execution_provider : ExecutionProvider) -> bool: def has_execution_provider(execution_provider : ExecutionProvider) -> bool:
@@ -17,115 +17,117 @@ def has_execution_provider(execution_provider : ExecutionProvider) -> bool:
def get_available_execution_providers() -> List[ExecutionProvider]: def get_available_execution_providers() -> List[ExecutionProvider]:
inference_execution_providers = get_available_providers() inference_session_providers = onnxruntime.get_available_providers()
available_execution_providers = [] available_execution_providers : List[ExecutionProvider] = []
for execution_provider, execution_provider_value in facefusion.choices.execution_provider_set.items(): for execution_provider, execution_provider_value in facefusion.choices.execution_provider_set.items():
if execution_provider_value in inference_execution_providers: if execution_provider_value in inference_session_providers:
available_execution_providers.append(execution_provider) index = facefusion.choices.execution_providers.index(execution_provider)
available_execution_providers.insert(index, execution_provider)
return available_execution_providers return available_execution_providers
def create_inference_execution_providers(execution_device_id : str, execution_providers : List[ExecutionProvider]) -> List[Any]: def create_inference_providers(execution_device_id : int, execution_providers : List[ExecutionProvider]) -> List[InferenceProvider]:
inference_execution_providers : List[Any] = [] inference_providers : List[InferenceProvider] = []
cache_path = resolve_cache_path()
for execution_provider in execution_providers: for execution_provider in execution_providers:
if execution_provider == 'cuda': if execution_provider == 'cuda':
inference_execution_providers.append((facefusion.choices.execution_provider_set.get(execution_provider), inference_providers.append((facefusion.choices.execution_provider_set.get(execution_provider),
{
'device_id': execution_device_id
}))
if execution_provider == 'tensorrt':
inference_execution_providers.append((facefusion.choices.execution_provider_set.get(execution_provider),
{ {
'device_id': execution_device_id, 'device_id': execution_device_id,
'trt_engine_cache_enable': True, 'cudnn_conv_algo_search': resolve_static_cudnn_conv_algo_search(tuple(execution_providers))
'trt_engine_cache_path': '.caches',
'trt_timing_cache_enable': True,
'trt_timing_cache_path': '.caches',
'trt_builder_optimization_level': 5
})) }))
if execution_provider == 'openvino':
inference_execution_providers.append((facefusion.choices.execution_provider_set.get(execution_provider), if execution_provider == 'tensorrt':
inference_option_set : InferenceOptionSet =\
{ {
'device_type': 'GPU' if execution_device_id == '0' else 'GPU.' + execution_device_id, 'device_id': execution_device_id
'precision': 'FP32' }
})) if is_directory(cache_path) or create_directory(cache_path):
inference_option_set.update(
{
'trt_engine_cache_enable': True,
'trt_engine_cache_path': cache_path,
'trt_timing_cache_enable': True,
'trt_timing_cache_path': cache_path,
'trt_builder_optimization_level': 4
})
inference_providers.append((facefusion.choices.execution_provider_set.get(execution_provider), inference_option_set))
if execution_provider in [ 'directml', 'rocm' ]: if execution_provider in [ 'directml', 'rocm' ]:
inference_execution_providers.append((facefusion.choices.execution_provider_set.get(execution_provider), inference_providers.append((facefusion.choices.execution_provider_set.get(execution_provider),
{ {
'device_id': execution_device_id 'device_id': execution_device_id
})) }))
if execution_provider == 'migraphx':
inference_option_set =\
{
'device_id': execution_device_id
}
if is_directory(cache_path) or create_directory(cache_path):
inference_option_set.update(
{
'migraphx_model_cache_dir': cache_path
})
inference_providers.append((facefusion.choices.execution_provider_set.get(execution_provider), inference_option_set))
if execution_provider == 'coreml': if execution_provider == 'coreml':
inference_execution_providers.append(facefusion.choices.execution_provider_set.get(execution_provider)) inference_option_set =\
{
'SpecializationStrategy': 'FastPrediction'
}
if is_directory(cache_path) or create_directory(cache_path):
inference_option_set.update(
{
'ModelCacheDirectory': cache_path
})
inference_providers.append((facefusion.choices.execution_provider_set.get(execution_provider), inference_option_set))
if execution_provider == 'openvino':
inference_providers.append((facefusion.choices.execution_provider_set.get(execution_provider),
{
'device_type': resolve_openvino_device_type(execution_device_id),
'precision': 'FP32'
}))
if execution_provider == 'qnn':
inference_providers.append((facefusion.choices.execution_provider_set.get(execution_provider),
{
'device_id': execution_device_id,
'backend_type': 'htp'
}))
if 'cpu' in execution_providers: if 'cpu' in execution_providers:
inference_execution_providers.append(facefusion.choices.execution_provider_set.get('cpu')) inference_providers.append(facefusion.choices.execution_provider_set.get('cpu'))
return inference_execution_providers return inference_providers
def run_nvidia_smi() -> subprocess.Popen[bytes]: def resolve_cache_path() -> str:
commands = [ shutil.which('nvidia-smi'), '--query', '--xml-format' ] return os.path.join('.caches', onnxruntime.get_version_string())
return subprocess.Popen(commands, stdout = subprocess.PIPE)
@lru_cache(maxsize = None) @lru_cache()
def detect_static_execution_devices() -> List[ExecutionDevice]: def resolve_static_cudnn_conv_algo_search(execution_providers : Tuple[ExecutionProvider, ...]) -> str:
return detect_execution_devices() return resolve_cudnn_conv_algo_search(list(execution_providers))
def detect_execution_devices() -> List[ExecutionDevice]: def resolve_cudnn_conv_algo_search(execution_providers : List[ExecutionProvider]) -> str:
execution_devices : List[ExecutionDevice] = [] if has_execution_provider('cuda') or has_execution_provider('tensorrt'):
graphic_devices = detect_graphic_devices(execution_providers)
product_names = ('GeForce GTX 1630', 'GeForce GTX 1650', 'GeForce GTX 1660')
try: for graphic_device in graphic_devices:
output, _ = run_nvidia_smi().communicate() if graphic_device.get('product').get('name').startswith(product_names):
root_element = ElementTree.fromstring(output) return 'DEFAULT'
except Exception:
root_element = ElementTree.Element('xml')
for gpu_element in root_element.findall('gpu'): return 'EXHAUSTIVE'
execution_devices.append(
{
'driver_version': root_element.findtext('driver_version'),
'framework':
{
'name': 'CUDA',
'version': root_element.findtext('cuda_version')
},
'product':
{
'vendor': 'NVIDIA',
'name': gpu_element.findtext('product_name').replace('NVIDIA', '').strip()
},
'video_memory':
{
'total': create_value_and_unit(gpu_element.findtext('fb_memory_usage/total')),
'free': create_value_and_unit(gpu_element.findtext('fb_memory_usage/free'))
},
'temperature':
{
'gpu': create_value_and_unit(gpu_element.findtext('temperature/gpu_temp')),
'memory': create_value_and_unit(gpu_element.findtext('temperature/memory_temp'))
},
'utilization':
{
'gpu': create_value_and_unit(gpu_element.findtext('utilization/gpu_util')),
'memory': create_value_and_unit(gpu_element.findtext('utilization/memory_util'))
}
})
return execution_devices
def create_value_and_unit(text : str) -> Optional[ValueAndUnit]: def resolve_openvino_device_type(execution_device_id : int) -> str:
if ' ' in text: if execution_device_id == 0:
value, unit = text.split(' ') return 'GPU'
return 'GPU.' + str(execution_device_id)
return\
{
'value': int(value),
'unit': str(unit)
}
return None
+15 -7
View File
@@ -1,26 +1,34 @@
import os
import signal import signal
import sys import sys
from time import sleep from time import sleep
from types import FrameType
from facefusion import process_manager, state_manager from facefusion import process_manager, state_manager
from facefusion.temp_helper import clear_temp_directory from facefusion.temp_helper import clear_temp_directory
from facefusion.typing import ErrorCode from facefusion.types import ErrorCode
def fatal_exit(error_code : ErrorCode) -> None:
os._exit(error_code)
def hard_exit(error_code : ErrorCode) -> None: def hard_exit(error_code : ErrorCode) -> None:
signal.signal(signal.SIGINT, signal.SIG_IGN)
sys.exit(error_code) sys.exit(error_code)
def conditional_exit(error_code : ErrorCode) -> None: def signal_exit(signum : int, frame : FrameType) -> None:
if state_manager.get_item('command') == 'headless-run': graceful_exit(0)
hard_exit(error_code)
def graceful_exit(error_code : ErrorCode) -> None: def graceful_exit(error_code : ErrorCode) -> None:
signal.signal(signal.SIGINT, signal.SIG_IGN)
process_manager.stop() process_manager.stop()
while process_manager.is_processing(): while process_manager.is_processing():
sleep(0.5) sleep(0.5)
if state_manager.get_item('target_path'):
clear_temp_directory(state_manager.get_item('target_path')) if state_manager.get_item('output_path'):
clear_temp_directory(state_manager.get_temp_path(), state_manager.get_item('output_path'))
hard_exit(error_code) hard_exit(error_code)
-124
View File
@@ -1,124 +0,0 @@
from typing import List, Optional
import numpy
from facefusion import state_manager
from facefusion.common_helper import get_first
from facefusion.face_classifier import classify_face
from facefusion.face_detector import detect_faces, detect_rotated_faces
from facefusion.face_helper import apply_nms, convert_to_face_landmark_5, estimate_face_angle, get_nms_threshold
from facefusion.face_landmarker import detect_face_landmarks, estimate_face_landmark_68_5
from facefusion.face_recognizer import calc_embedding
from facefusion.face_store import get_static_faces, set_static_faces
from facefusion.typing import BoundingBox, Face, FaceLandmark5, FaceLandmarkSet, FaceScoreSet, Score, VisionFrame
def create_faces(vision_frame : VisionFrame, bounding_boxes : List[BoundingBox], face_scores : List[Score], face_landmarks_5 : List[FaceLandmark5]) -> List[Face]:
faces = []
nms_threshold = get_nms_threshold(state_manager.get_item('face_detector_model'), state_manager.get_item('face_detector_angles'))
keep_indices = apply_nms(bounding_boxes, face_scores, state_manager.get_item('face_detector_score'), nms_threshold)
for index in keep_indices:
bounding_box = bounding_boxes[index]
face_score = face_scores[index]
face_landmark_5 = face_landmarks_5[index]
face_landmark_5_68 = face_landmark_5
face_landmark_68_5 = estimate_face_landmark_68_5(face_landmark_5_68)
face_landmark_68 = face_landmark_68_5
face_landmark_score_68 = 0.0
face_angle = estimate_face_angle(face_landmark_68_5)
if state_manager.get_item('face_landmarker_score') > 0:
face_landmark_68, face_landmark_score_68 = detect_face_landmarks(vision_frame, bounding_box, face_angle)
if face_landmark_score_68 > state_manager.get_item('face_landmarker_score'):
face_landmark_5_68 = convert_to_face_landmark_5(face_landmark_68)
face_landmark_set : FaceLandmarkSet =\
{
'5': face_landmark_5,
'5/68': face_landmark_5_68,
'68': face_landmark_68,
'68/5': face_landmark_68_5
}
face_score_set : FaceScoreSet =\
{
'detector': face_score,
'landmarker': face_landmark_score_68
}
embedding, normed_embedding = calc_embedding(vision_frame, face_landmark_set.get('5/68'))
gender, age, race = classify_face(vision_frame, face_landmark_set.get('5/68'))
faces.append(Face(
bounding_box = bounding_box,
score_set = face_score_set,
landmark_set = face_landmark_set,
angle = face_angle,
embedding = embedding,
normed_embedding = normed_embedding,
gender = gender,
age = age,
race = race
))
return faces
def get_one_face(faces : List[Face], position : int = 0) -> Optional[Face]:
if faces:
position = min(position, len(faces) - 1)
return faces[position]
return None
def get_average_face(faces : List[Face]) -> Optional[Face]:
embeddings = []
normed_embeddings = []
if faces:
first_face = get_first(faces)
for face in faces:
embeddings.append(face.embedding)
normed_embeddings.append(face.normed_embedding)
return Face(
bounding_box = first_face.bounding_box,
score_set = first_face.score_set,
landmark_set = first_face.landmark_set,
angle = first_face.angle,
embedding = numpy.mean(embeddings, axis = 0),
normed_embedding = numpy.mean(normed_embeddings, axis = 0),
gender = first_face.gender,
age = first_face.age,
race = first_face.race
)
return None
def get_many_faces(vision_frames : List[VisionFrame]) -> List[Face]:
many_faces : List[Face] = []
for vision_frame in vision_frames:
if numpy.any(vision_frame):
static_faces = get_static_faces(vision_frame)
if static_faces:
many_faces.extend(static_faces)
else:
all_bounding_boxes = []
all_face_scores = []
all_face_landmarks_5 = []
for face_detector_angle in state_manager.get_item('face_detector_angles'):
if face_detector_angle == 0:
bounding_boxes, face_scores, face_landmarks_5 = detect_faces(vision_frame)
else:
bounding_boxes, face_scores, face_landmarks_5 = detect_rotated_faces(vision_frame, face_detector_angle)
all_bounding_boxes.extend(bounding_boxes)
all_face_scores.extend(face_scores)
all_face_landmarks_5.extend(face_landmarks_5)
if all_bounding_boxes and all_face_scores and all_face_landmarks_5 and state_manager.get_item('face_detector_score') > 0:
faces = create_faces(vision_frame, all_bounding_boxes, all_face_scores, all_face_landmarks_5)
if faces:
many_faces.extend(faces)
set_static_faces(vision_frame, faces)
return many_faces
+18 -9
View File
@@ -8,15 +8,21 @@ from facefusion.download import conditional_download_hashes, conditional_downloa
from facefusion.face_helper import warp_face_by_face_landmark_5 from facefusion.face_helper import warp_face_by_face_landmark_5
from facefusion.filesystem import resolve_relative_path from facefusion.filesystem import resolve_relative_path
from facefusion.thread_helper import conditional_thread_semaphore from facefusion.thread_helper import conditional_thread_semaphore
from facefusion.typing import Age, DownloadScope, FaceLandmark5, Gender, InferencePool, ModelOptions, ModelSet, Race, VisionFrame from facefusion.types import Age, DownloadScope, FaceLandmark5, Gender, InferencePool, ModelOptions, ModelSet, Race, VisionFrame
@lru_cache(maxsize = None) @lru_cache()
def create_static_model_set(download_scope : DownloadScope) -> ModelSet: def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
return\ return\
{ {
'fairface': 'fairface':
{ {
'__metadata__':
{
'vendor': 'dchen236',
'license': 'CC-BY-4.0',
'year': 2021
},
'hashes': 'hashes':
{ {
'face_classifier': 'face_classifier':
@@ -42,12 +48,15 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
def get_inference_pool() -> InferencePool: def get_inference_pool() -> InferencePool:
model_sources = get_model_options().get('sources') model_names = [ 'fairface' ]
return inference_manager.get_inference_pool(__name__, model_sources) model_source_set = get_model_options().get('sources')
return inference_manager.get_inference_pool(__name__, model_names, model_source_set)
def clear_inference_pool() -> None: def clear_inference_pool() -> None:
inference_manager.clear_inference_pool(__name__) model_names = [ 'fairface' ]
inference_manager.clear_inference_pool(__name__, model_names)
def get_model_options() -> ModelOptions: def get_model_options() -> ModelOptions:
@@ -55,10 +64,10 @@ def get_model_options() -> ModelOptions:
def pre_check() -> bool: def pre_check() -> bool:
model_hashes = get_model_options().get('hashes') model_hash_set = get_model_options().get('hashes')
model_sources = get_model_options().get('sources') model_source_set = get_model_options().get('sources')
return conditional_download_hashes(model_hashes) and conditional_download_sources(model_sources) return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set)
def classify_face(temp_vision_frame : VisionFrame, face_landmark_5 : FaceLandmark5) -> Tuple[Gender, Age, Race]: def classify_face(temp_vision_frame : VisionFrame, face_landmark_5 : FaceLandmark5) -> Tuple[Gender, Age, Race]:
@@ -67,7 +76,7 @@ def classify_face(temp_vision_frame : VisionFrame, face_landmark_5 : FaceLandmar
model_mean = get_model_options().get('mean') model_mean = get_model_options().get('mean')
model_standard_deviation = get_model_options().get('standard_deviation') model_standard_deviation = get_model_options().get('standard_deviation')
crop_vision_frame, _ = warp_face_by_face_landmark_5(temp_vision_frame, face_landmark_5, model_template, model_size) crop_vision_frame, _ = warp_face_by_face_landmark_5(temp_vision_frame, face_landmark_5, model_template, model_size)
crop_vision_frame = crop_vision_frame.astype(numpy.float32)[:, :, ::-1] / 255 crop_vision_frame = crop_vision_frame.astype(numpy.float32)[:, :, ::-1] / 255.0
crop_vision_frame -= model_mean crop_vision_frame -= model_mean
crop_vision_frame /= model_standard_deviation crop_vision_frame /= model_standard_deviation
crop_vision_frame = crop_vision_frame.transpose(2, 0, 1) crop_vision_frame = crop_vision_frame.transpose(2, 0, 1)
+209
View File
@@ -0,0 +1,209 @@
from typing import List, Optional
import numpy
from facefusion import face_store, state_manager
from facefusion.common_helper import get_first, get_middle
from facefusion.face_classifier import classify_face
from facefusion.face_detector import detect_faces, detect_faces_by_angle
from facefusion.face_helper import apply_nms, average_points, convert_to_face_landmark_5, estimate_face_angle, get_nms_threshold
from facefusion.face_landmarker import detect_face_landmark, estimate_face_landmark_68_5
from facefusion.face_recognizer import calculate_face_embedding
from facefusion.types import BoundingBox, Face, FaceLandmark5, FaceLandmarkSet, FaceScoreSet, Score, VisionFrame
def create_faces(vision_frame : VisionFrame, bounding_boxes : List[BoundingBox], face_scores : List[Score], face_landmarks_5 : List[FaceLandmark5]) -> List[Face]:
faces = []
nms_threshold = get_nms_threshold(state_manager.get_item('face_detector_model'), state_manager.get_item('face_detector_angles'))
keep_indices = apply_nms(bounding_boxes, face_scores, state_manager.get_item('face_detector_score'), nms_threshold)
for index in keep_indices:
bounding_box = bounding_boxes[index]
face_score = face_scores[index]
face_landmark_5 = face_landmarks_5[index]
face_landmark_5_68 = face_landmark_5
face_landmark_68_5 = estimate_face_landmark_68_5(face_landmark_5_68)
face_landmark_68 = face_landmark_68_5
face_landmark_score_68 = 0.0
face_angle = estimate_face_angle(face_landmark_68_5)
if state_manager.get_item('face_landmarker_score') > 0:
face_landmark_68, face_landmark_score_68 = detect_face_landmark(vision_frame, bounding_box, face_angle)
if face_landmark_score_68 > state_manager.get_item('face_landmarker_score'):
face_landmark_5_68 = convert_to_face_landmark_5(face_landmark_68)
face_landmark_set : FaceLandmarkSet =\
{
'5': face_landmark_5,
'5/68': face_landmark_5_68,
'68': face_landmark_68,
'68/5': face_landmark_68_5
}
face_score_set : FaceScoreSet =\
{
'detector': face_score,
'landmarker': face_landmark_score_68
}
face_embedding, face_embedding_norm = calculate_face_embedding(vision_frame, face_landmark_set.get('5/68'))
gender, age, race = classify_face(vision_frame, face_landmark_set.get('5/68'))
faces.append(Face(
origin = 'detect',
bounding_box = bounding_box,
score_set = face_score_set,
landmark_set = face_landmark_set,
angle = face_angle,
embedding = face_embedding,
embedding_norm = face_embedding_norm,
gender = gender,
age = age,
race = race
))
return faces
def get_one_face(faces : List[Face], position : int = 0) -> Optional[Face]:
if faces:
position = min(position, len(faces) - 1)
return faces[position]
return None
def get_many_faces(vision_frames : List[VisionFrame]) -> List[Face]:
many_faces : List[Face] = []
for vision_frame in vision_frames:
if numpy.any(vision_frame):
all_bounding_boxes = []
all_face_scores = []
all_face_landmarks_5 = []
for face_detector_angle in state_manager.get_item('face_detector_angles'):
if face_detector_angle == 0:
bounding_boxes, face_scores, face_landmarks_5 = detect_faces(vision_frame)
else:
bounding_boxes, face_scores, face_landmarks_5 = detect_faces_by_angle(vision_frame, face_detector_angle)
all_bounding_boxes.extend(bounding_boxes)
all_face_scores.extend(face_scores)
all_face_landmarks_5.extend(face_landmarks_5)
if all_bounding_boxes and all_face_scores and all_face_landmarks_5 and state_manager.get_item('face_detector_score') > 0:
faces = create_faces(vision_frame, all_bounding_boxes, all_face_scores, all_face_landmarks_5)
if faces:
many_faces.extend(faces)
return many_faces
def get_static_faces(vision_frames : List[VisionFrame]) -> List[Face]:
many_faces : List[Face] = []
for vision_frame in vision_frames:
faces = face_store.get_faces(vision_frame)
if not faces:
with face_store.resolve_lock(vision_frame):
faces = face_store.get_faces(vision_frame)
if not faces:
faces = get_many_faces([ vision_frame ])
if faces:
face_store.set_faces(vision_frame, faces)
many_faces.extend(faces)
return many_faces
def refill_faces(faces : List[Optional[Face]]) -> List[Face]:
fill_faces = []
anchor_index_previous = -1
for index, face in enumerate(faces):
if face:
for gap_index in range(anchor_index_previous + 1, index):
average_factor = (gap_index - anchor_index_previous) / (index - anchor_index_previous)
average_face = average_face_geometry([faces[anchor_index_previous], face], average_factor)
fill_faces.append(average_face)
fill_faces.append(face)
anchor_index_previous = index
return fill_faces
def average_face_geometry(faces : List[Face], average_factor : float) -> Face:
face_first = get_first(faces)
face_middle = get_middle(faces)
face_anchor = face_middle
if average_factor < 0.5:
face_anchor = face_first
landmark_set : FaceLandmarkSet =\
{
'5': average_points(face_first.landmark_set.get('5'), face_middle.landmark_set.get('5'), average_factor),
'5/68': average_points(face_first.landmark_set.get('5/68'), face_middle.landmark_set.get('5/68'), average_factor),
'68': average_points(face_first.landmark_set.get('68'), face_middle.landmark_set.get('68'), average_factor),
'68/5': average_points(face_first.landmark_set.get('68/5'), face_middle.landmark_set.get('68/5'), average_factor)
}
return Face(
origin = 'refill',
bounding_box = average_points(face_first.bounding_box, face_middle.bounding_box, average_factor),
score_set = face_anchor.score_set,
landmark_set = landmark_set,
angle = estimate_face_angle(landmark_set.get('68/5')),
embedding = face_anchor.embedding,
embedding_norm = face_anchor.embedding_norm,
gender = face_anchor.gender,
age = face_anchor.age,
race = face_anchor.race
)
def average_face_identity(faces : List[Face]) -> Optional[Face]:
face_embeddings = []
face_embeddings_norm = []
if faces:
first_face = get_first(faces)
for face in faces:
face_embeddings.append(face.embedding)
face_embeddings_norm.append(face.embedding_norm)
return Face(
origin = first_face.origin,
bounding_box = first_face.bounding_box,
score_set = first_face.score_set,
landmark_set = first_face.landmark_set,
angle = first_face.angle,
embedding = numpy.mean(face_embeddings, axis = 0),
embedding_norm = numpy.mean(face_embeddings_norm, axis = 0),
gender = first_face.gender,
age = first_face.age,
race = first_face.race
)
return None
def scale_face(target_face : Face, target_vision_frame : VisionFrame, temp_vision_frame : VisionFrame) -> Face:
scale_x = temp_vision_frame.shape[1] / target_vision_frame.shape[1]
scale_y = temp_vision_frame.shape[0] / target_vision_frame.shape[0]
bounding_box = target_face.bounding_box * [ scale_x, scale_y, scale_x, scale_y ]
landmark_set =\
{
'5': target_face.landmark_set.get('5') * numpy.array([ scale_x, scale_y ]),
'5/68': target_face.landmark_set.get('5/68') * numpy.array([ scale_x, scale_y ]),
'68': target_face.landmark_set.get('68') * numpy.array([ scale_x, scale_y ]),
'68/5': target_face.landmark_set.get('68/5') * numpy.array([ scale_x, scale_y ])
}
return target_face._replace(
bounding_box = bounding_box,
landmark_set = landmark_set
)
+231 -87
View File
@@ -1,24 +1,30 @@
from typing import List, Tuple from functools import lru_cache
from typing import List, Sequence, Tuple
import cv2 import cv2
import numpy import numpy
from charset_normalizer.md import lru_cache
from facefusion import inference_manager, state_manager from facefusion import inference_manager, state_manager
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url
from facefusion.face_helper import create_rotated_matrix_and_size, create_static_anchors, distance_to_bounding_box, distance_to_face_landmark_5, normalize_bounding_box, transform_bounding_box, transform_points from facefusion.face_helper import create_rotation_matrix_and_size, create_static_anchors, distance_to_bounding_box, distance_to_face_landmark_5, normalize_bounding_box, transform_bounding_box, transform_points
from facefusion.filesystem import resolve_relative_path from facefusion.filesystem import resolve_relative_path
from facefusion.thread_helper import thread_semaphore from facefusion.thread_helper import thread_semaphore
from facefusion.typing import Angle, BoundingBox, Detection, DownloadScope, DownloadSet, FaceLandmark5, InferencePool, ModelSet, Score, VisionFrame from facefusion.types import Angle, BoundingBox, Detection, DownloadScope, DownloadSet, FaceLandmark5, InferencePool, Margin, ModelSet, Score, VisionFrame
from facefusion.vision import resize_frame_resolution, unpack_resolution from facefusion.vision import restrict_frame, unpack_resolution
@lru_cache(maxsize = None) @lru_cache()
def create_static_model_set(download_scope : DownloadScope) -> ModelSet: def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
return\ return\
{ {
'retinaface': 'retinaface':
{ {
'__metadata__':
{
'vendor': 'InsightFace',
'license': 'Non-Commercial',
'year': 2020
},
'hashes': 'hashes':
{ {
'retinaface': 'retinaface':
@@ -38,6 +44,12 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
}, },
'scrfd': 'scrfd':
{ {
'__metadata__':
{
'vendor': 'InsightFace',
'license': 'Non-Commercial',
'year': 2021
},
'hashes': 'hashes':
{ {
'scrfd': 'scrfd':
@@ -55,11 +67,17 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
} }
} }
}, },
'yoloface': 'yolo_face':
{ {
'__metadata__':
{
'vendor': 'derronqi',
'license': 'GPL-3.0',
'year': 2022
},
'hashes': 'hashes':
{ {
'yoloface': 'yolo_face':
{ {
'url': resolve_download_url('models-3.0.0', 'yoloface_8n.hash'), 'url': resolve_download_url('models-3.0.0', 'yoloface_8n.hash'),
'path': resolve_relative_path('../.assets/models/yoloface_8n.hash') 'path': resolve_relative_path('../.assets/models/yoloface_8n.hash')
@@ -67,85 +85,123 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
}, },
'sources': 'sources':
{ {
'yoloface': 'yolo_face':
{ {
'url': resolve_download_url('models-3.0.0', 'yoloface_8n.onnx'), 'url': resolve_download_url('models-3.0.0', 'yoloface_8n.onnx'),
'path': resolve_relative_path('../.assets/models/yoloface_8n.onnx') 'path': resolve_relative_path('../.assets/models/yoloface_8n.onnx')
} }
} }
},
'yunet':
{
'__metadata__':
{
'vendor': 'OpenCV',
'license': 'MIT',
'year': 2023
},
'hashes':
{
'yunet':
{
'url': resolve_download_url('models-3.4.0', 'yunet_2023_mar.hash'),
'path': resolve_relative_path('../.assets/models/yunet_2023_mar.hash')
}
},
'sources':
{
'yunet':
{
'url': resolve_download_url('models-3.4.0', 'yunet_2023_mar.onnx'),
'path': resolve_relative_path('../.assets/models/yunet_2023_mar.onnx')
}
}
} }
} }
def get_inference_pool() -> InferencePool: def get_inference_pool() -> InferencePool:
_, model_sources = collect_model_downloads() model_names = [ state_manager.get_item('face_detector_model') ]
return inference_manager.get_inference_pool(__name__, model_sources) _, model_source_set = collect_model_downloads()
return inference_manager.get_inference_pool(__name__, model_names, model_source_set)
def clear_inference_pool() -> None: def clear_inference_pool() -> None:
inference_manager.clear_inference_pool(__name__) model_names = [ state_manager.get_item('face_detector_model') ]
inference_manager.clear_inference_pool(__name__, model_names)
def collect_model_downloads() -> Tuple[DownloadSet, DownloadSet]: def collect_model_downloads() -> Tuple[DownloadSet, DownloadSet]:
model_hashes = {}
model_sources = {}
model_set = create_static_model_set('full') model_set = create_static_model_set('full')
model_hash_set = {}
model_source_set = {}
if state_manager.get_item('face_detector_model') in [ 'many', 'retinaface' ]: for face_detector_model in [ 'retinaface', 'scrfd', 'yolo_face', 'yunet' ]:
model_hashes['retinaface'] = model_set.get('retinaface').get('hashes').get('retinaface') if state_manager.get_item('face_detector_model') in [ 'many', face_detector_model ]:
model_sources['retinaface'] = model_set.get('retinaface').get('sources').get('retinaface') model_hash_set[face_detector_model] = model_set.get(face_detector_model).get('hashes').get(face_detector_model)
model_source_set[face_detector_model] = model_set.get(face_detector_model).get('sources').get(face_detector_model)
if state_manager.get_item('face_detector_model') in [ 'many', 'scrfd' ]: return model_hash_set, model_source_set
model_hashes['scrfd'] = model_set.get('scrfd').get('hashes').get('scrfd')
model_sources['scrfd'] = model_set.get('scrfd').get('sources').get('scrfd')
if state_manager.get_item('face_detector_model') in [ 'many', 'yoloface' ]:
model_hashes['yoloface'] = model_set.get('yoloface').get('hashes').get('yoloface')
model_sources['yoloface'] = model_set.get('yoloface').get('sources').get('yoloface')
return model_hashes, model_sources
def pre_check() -> bool: def pre_check() -> bool:
model_hashes, model_sources = collect_model_downloads() model_hash_set, model_source_set = collect_model_downloads()
return conditional_download_hashes(model_hashes) and conditional_download_sources(model_sources) return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set)
def detect_faces(vision_frame : VisionFrame) -> Tuple[List[BoundingBox], List[Score], List[FaceLandmark5]]: def detect_faces(vision_frame : VisionFrame) -> Tuple[List[BoundingBox], List[Score], List[FaceLandmark5]]:
margin_top, margin_right, margin_bottom, margin_left = prepare_margin(vision_frame)
margin_vision_frame = numpy.pad(vision_frame, ((margin_top, margin_bottom), (margin_left, margin_right), (0, 0)))
all_bounding_boxes : List[BoundingBox] = [] all_bounding_boxes : List[BoundingBox] = []
all_face_scores : List[Score] = [] all_face_scores : List[Score] = []
all_face_landmarks_5 : List[FaceLandmark5] = [] all_face_landmarks_5 : List[FaceLandmark5] = []
if state_manager.get_item('face_detector_model') in [ 'many', 'retinaface' ]: if state_manager.get_item('face_detector_model') in [ 'many', 'retinaface' ]:
bounding_boxes, face_scores, face_landmarks_5 = detect_with_retinaface(vision_frame, state_manager.get_item('face_detector_size')) bounding_boxes, face_scores, face_landmarks_5 = detect_with_retinaface(margin_vision_frame, state_manager.get_item('face_detector_size'))
all_bounding_boxes.extend(bounding_boxes) all_bounding_boxes.extend(bounding_boxes)
all_face_scores.extend(face_scores) all_face_scores.extend(face_scores)
all_face_landmarks_5.extend(face_landmarks_5) all_face_landmarks_5.extend(face_landmarks_5)
if state_manager.get_item('face_detector_model') in [ 'many', 'scrfd' ]: if state_manager.get_item('face_detector_model') in [ 'many', 'scrfd' ]:
bounding_boxes, face_scores, face_landmarks_5 = detect_with_scrfd(vision_frame, state_manager.get_item('face_detector_size')) bounding_boxes, face_scores, face_landmarks_5 = detect_with_scrfd(margin_vision_frame, state_manager.get_item('face_detector_size'))
all_bounding_boxes.extend(bounding_boxes) all_bounding_boxes.extend(bounding_boxes)
all_face_scores.extend(face_scores) all_face_scores.extend(face_scores)
all_face_landmarks_5.extend(face_landmarks_5) all_face_landmarks_5.extend(face_landmarks_5)
if state_manager.get_item('face_detector_model') in [ 'many', 'yoloface' ]: if state_manager.get_item('face_detector_model') in [ 'many', 'yolo_face' ]:
bounding_boxes, face_scores, face_landmarks_5 = detect_with_yoloface(vision_frame, state_manager.get_item('face_detector_size')) bounding_boxes, face_scores, face_landmarks_5 = detect_with_yolo_face(margin_vision_frame, state_manager.get_item('face_detector_size'))
all_bounding_boxes.extend(bounding_boxes) all_bounding_boxes.extend(bounding_boxes)
all_face_scores.extend(face_scores) all_face_scores.extend(face_scores)
all_face_landmarks_5.extend(face_landmarks_5) all_face_landmarks_5.extend(face_landmarks_5)
all_bounding_boxes = [ normalize_bounding_box(all_bounding_box) for all_bounding_box in all_bounding_boxes ] if state_manager.get_item('face_detector_model') == 'yunet':
bounding_boxes, face_scores, face_landmarks_5 = detect_with_yunet(margin_vision_frame, state_manager.get_item('face_detector_size'))
all_bounding_boxes.extend(bounding_boxes)
all_face_scores.extend(face_scores)
all_face_landmarks_5.extend(face_landmarks_5)
all_bounding_boxes = [ normalize_bounding_box(all_bounding_box) - numpy.array([ margin_left, margin_top, margin_left, margin_top ]) for all_bounding_box in all_bounding_boxes ]
all_face_landmarks_5 = [ all_face_landmark_5 - numpy.array([ margin_left, margin_top ]) for all_face_landmark_5 in all_face_landmarks_5 ]
return all_bounding_boxes, all_face_scores, all_face_landmarks_5 return all_bounding_boxes, all_face_scores, all_face_landmarks_5
def detect_rotated_faces(vision_frame : VisionFrame, angle : Angle) -> Tuple[List[BoundingBox], List[Score], List[FaceLandmark5]]: def prepare_margin(vision_frame : VisionFrame) -> Margin:
rotated_matrix, rotated_size = create_rotated_matrix_and_size(angle, vision_frame.shape[:2][::-1]) margin_top = int(vision_frame.shape[0] * numpy.interp(state_manager.get_item('face_detector_margin')[0], [ 0, 100 ], [ 0, 0.5 ]))
rotated_vision_frame = cv2.warpAffine(vision_frame, rotated_matrix, rotated_size) margin_right = int(vision_frame.shape[1] * numpy.interp(state_manager.get_item('face_detector_margin')[1], [ 0, 100 ], [ 0, 0.5 ]))
rotated_inverse_matrix = cv2.invertAffineTransform(rotated_matrix) margin_bottom = int(vision_frame.shape[0] * numpy.interp(state_manager.get_item('face_detector_margin')[2], [ 0, 100 ], [ 0, 0.5 ]))
bounding_boxes, face_scores, face_landmarks_5 = detect_faces(rotated_vision_frame) margin_left = int(vision_frame.shape[1] * numpy.interp(state_manager.get_item('face_detector_margin')[3], [ 0, 100 ], [ 0, 0.5 ]))
bounding_boxes = [ transform_bounding_box(bounding_box, rotated_inverse_matrix) for bounding_box in bounding_boxes ] return margin_top, margin_right, margin_bottom, margin_left
face_landmarks_5 = [ transform_points(face_landmark_5, rotated_inverse_matrix) for face_landmark_5 in face_landmarks_5 ]
def detect_faces_by_angle(vision_frame : VisionFrame, face_angle : Angle) -> Tuple[List[BoundingBox], List[Score], List[FaceLandmark5]]:
rotation_matrix, rotation_size = create_rotation_matrix_and_size(face_angle, vision_frame.shape[:2][::-1])
rotation_vision_frame = cv2.warpAffine(vision_frame, rotation_matrix, rotation_size)
rotation_inverse_matrix = cv2.invertAffineTransform(rotation_matrix)
bounding_boxes, face_scores, face_landmarks_5 = detect_faces(rotation_vision_frame)
bounding_boxes = [ transform_bounding_box(bounding_box, rotation_inverse_matrix) for bounding_box in bounding_boxes ]
face_landmarks_5 = [ transform_points(face_landmark_5, rotation_inverse_matrix) for face_landmark_5 in face_landmarks_5 ]
return bounding_boxes, face_scores, face_landmarks_5 return bounding_boxes, face_scores, face_landmarks_5
@@ -156,37 +212,40 @@ def detect_with_retinaface(vision_frame : VisionFrame, face_detector_size : str)
feature_strides = [ 8, 16, 32 ] feature_strides = [ 8, 16, 32 ]
feature_map_channel = 3 feature_map_channel = 3
anchor_total = 2 anchor_total = 2
face_detector_score = state_manager.get_item('face_detector_score')
face_detector_width, face_detector_height = unpack_resolution(face_detector_size) face_detector_width, face_detector_height = unpack_resolution(face_detector_size)
temp_vision_frame = resize_frame_resolution(vision_frame, (face_detector_width, face_detector_height)) temp_vision_frame = restrict_frame(vision_frame, (face_detector_width, face_detector_height))
ratio_height = vision_frame.shape[0] / temp_vision_frame.shape[0] ratio_height = vision_frame.shape[0] / temp_vision_frame.shape[0]
ratio_width = vision_frame.shape[1] / temp_vision_frame.shape[1] ratio_width = vision_frame.shape[1] / temp_vision_frame.shape[1]
detect_vision_frame = prepare_detect_frame(temp_vision_frame, face_detector_size) detect_vision_frame = prepare_detect_frame(temp_vision_frame, face_detector_size)
detect_vision_frame = normalize_detect_frame(detect_vision_frame, [ -1, 1 ])
detection = forward_with_retinaface(detect_vision_frame) detection = forward_with_retinaface(detect_vision_frame)
for index, feature_stride in enumerate(feature_strides): for index, feature_stride in enumerate(feature_strides):
keep_indices = numpy.where(detection[index] >= state_manager.get_item('face_detector_score'))[0] face_scores_raw = detection[index]
keep_indices = numpy.where(face_scores_raw >= face_detector_score)[0]
if numpy.any(keep_indices): if numpy.any(keep_indices):
stride_height = face_detector_height // feature_stride stride_height = face_detector_height // feature_stride
stride_width = face_detector_width // feature_stride stride_width = face_detector_width // feature_stride
anchors = create_static_anchors(feature_stride, anchor_total, stride_height, stride_width) anchors = create_static_anchors(feature_stride, anchor_total, stride_width, stride_height)
bounding_box_raw = detection[index + feature_map_channel] * feature_stride bounding_boxes_raw = detection[index + feature_map_channel] * feature_stride
face_landmark_5_raw = detection[index + feature_map_channel * 2] * feature_stride face_landmarks_5_raw = detection[index + feature_map_channel * 2] * feature_stride
for bounding_box in distance_to_bounding_box(anchors, bounding_box_raw)[keep_indices]: for bounding_box_raw in distance_to_bounding_box(anchors, bounding_boxes_raw)[keep_indices]:
bounding_boxes.append(numpy.array( bounding_boxes.append(numpy.array(
[ [
bounding_box[0] * ratio_width, bounding_box_raw[0] * ratio_width,
bounding_box[1] * ratio_height, bounding_box_raw[1] * ratio_height,
bounding_box[2] * ratio_width, bounding_box_raw[2] * ratio_width,
bounding_box[3] * ratio_height, bounding_box_raw[3] * ratio_height
])) ]))
for score in detection[index][keep_indices]: for face_score_raw in face_scores_raw[keep_indices]:
face_scores.append(score[0]) face_scores.append(face_score_raw[0])
for face_landmark_5 in distance_to_face_landmark_5(anchors, face_landmark_5_raw)[keep_indices]: for face_landmark_raw_5 in distance_to_face_landmark_5(anchors, face_landmarks_5_raw)[keep_indices]:
face_landmarks_5.append(face_landmark_5 * [ ratio_width, ratio_height ]) face_landmarks_5.append(face_landmark_raw_5 * [ ratio_width, ratio_height ])
return bounding_boxes, face_scores, face_landmarks_5 return bounding_boxes, face_scores, face_landmarks_5
@@ -198,73 +257,139 @@ def detect_with_scrfd(vision_frame : VisionFrame, face_detector_size : str) -> T
feature_strides = [ 8, 16, 32 ] feature_strides = [ 8, 16, 32 ]
feature_map_channel = 3 feature_map_channel = 3
anchor_total = 2 anchor_total = 2
face_detector_score = state_manager.get_item('face_detector_score')
face_detector_width, face_detector_height = unpack_resolution(face_detector_size) face_detector_width, face_detector_height = unpack_resolution(face_detector_size)
temp_vision_frame = resize_frame_resolution(vision_frame, (face_detector_width, face_detector_height)) temp_vision_frame = restrict_frame(vision_frame, (face_detector_width, face_detector_height))
ratio_height = vision_frame.shape[0] / temp_vision_frame.shape[0] ratio_height = vision_frame.shape[0] / temp_vision_frame.shape[0]
ratio_width = vision_frame.shape[1] / temp_vision_frame.shape[1] ratio_width = vision_frame.shape[1] / temp_vision_frame.shape[1]
detect_vision_frame = prepare_detect_frame(temp_vision_frame, face_detector_size) detect_vision_frame = prepare_detect_frame(temp_vision_frame, face_detector_size)
detect_vision_frame = normalize_detect_frame(detect_vision_frame, [ -1, 1 ])
detection = forward_with_scrfd(detect_vision_frame) detection = forward_with_scrfd(detect_vision_frame)
for index, feature_stride in enumerate(feature_strides): for index, feature_stride in enumerate(feature_strides):
keep_indices = numpy.where(detection[index] >= state_manager.get_item('face_detector_score'))[0] face_scores_raw = detection[index]
keep_indices = numpy.where(face_scores_raw >= face_detector_score)[0]
if numpy.any(keep_indices): if numpy.any(keep_indices):
stride_height = face_detector_height // feature_stride stride_height = face_detector_height // feature_stride
stride_width = face_detector_width // feature_stride stride_width = face_detector_width // feature_stride
anchors = create_static_anchors(feature_stride, anchor_total, stride_height, stride_width) anchors = create_static_anchors(feature_stride, anchor_total, stride_width, stride_height)
bounding_box_raw = detection[index + feature_map_channel] * feature_stride bounding_boxes_raw = detection[index + feature_map_channel] * feature_stride
face_landmark_5_raw = detection[index + feature_map_channel * 2] * feature_stride face_landmarks_5_raw = detection[index + feature_map_channel * 2] * feature_stride
for bounding_box in distance_to_bounding_box(anchors, bounding_box_raw)[keep_indices]: for bounding_box_raw in distance_to_bounding_box(anchors, bounding_boxes_raw)[keep_indices]:
bounding_boxes.append(numpy.array( bounding_boxes.append(numpy.array(
[ [
bounding_box[0] * ratio_width, bounding_box_raw[0] * ratio_width,
bounding_box[1] * ratio_height, bounding_box_raw[1] * ratio_height,
bounding_box[2] * ratio_width, bounding_box_raw[2] * ratio_width,
bounding_box[3] * ratio_height, bounding_box_raw[3] * ratio_height
])) ]))
for score in detection[index][keep_indices]: for face_score_raw in face_scores_raw[keep_indices]:
face_scores.append(score[0]) face_scores.append(face_score_raw[0])
for face_landmark_5 in distance_to_face_landmark_5(anchors, face_landmark_5_raw)[keep_indices]: for face_landmark_raw_5 in distance_to_face_landmark_5(anchors, face_landmarks_5_raw)[keep_indices]:
face_landmarks_5.append(face_landmark_5 * [ ratio_width, ratio_height ]) face_landmarks_5.append(face_landmark_raw_5 * [ ratio_width, ratio_height ])
return bounding_boxes, face_scores, face_landmarks_5 return bounding_boxes, face_scores, face_landmarks_5
def detect_with_yoloface(vision_frame : VisionFrame, face_detector_size : str) -> Tuple[List[BoundingBox], List[Score], List[FaceLandmark5]]: def detect_with_yolo_face(vision_frame : VisionFrame, face_detector_size : str) -> Tuple[List[BoundingBox], List[Score], List[FaceLandmark5]]:
bounding_boxes = [] bounding_boxes = []
face_scores = [] face_scores = []
face_landmarks_5 = [] face_landmarks_5 = []
face_detector_score = state_manager.get_item('face_detector_score')
face_detector_width, face_detector_height = unpack_resolution(face_detector_size) face_detector_width, face_detector_height = unpack_resolution(face_detector_size)
temp_vision_frame = resize_frame_resolution(vision_frame, (face_detector_width, face_detector_height)) temp_vision_frame = restrict_frame(vision_frame, (face_detector_width, face_detector_height))
ratio_height = vision_frame.shape[0] / temp_vision_frame.shape[0] ratio_height = vision_frame.shape[0] / temp_vision_frame.shape[0]
ratio_width = vision_frame.shape[1] / temp_vision_frame.shape[1] ratio_width = vision_frame.shape[1] / temp_vision_frame.shape[1]
detect_vision_frame = prepare_detect_frame(temp_vision_frame, face_detector_size) detect_vision_frame = prepare_detect_frame(temp_vision_frame, face_detector_size)
detection = forward_with_yoloface(detect_vision_frame) detect_vision_frame = normalize_detect_frame(detect_vision_frame, [ 0, 1 ])
detection = forward_with_yolo_face(detect_vision_frame)
detection = numpy.squeeze(detection).T detection = numpy.squeeze(detection).T
bounding_box_raw, score_raw, face_landmark_5_raw = numpy.split(detection, [ 4, 5 ], axis = 1) bounding_boxes_raw, face_scores_raw, face_landmarks_5_raw = numpy.split(detection, [ 4, 5 ], axis = 1)
keep_indices = numpy.where(score_raw > state_manager.get_item('face_detector_score'))[0] keep_indices = numpy.where(face_scores_raw > face_detector_score)[0]
if numpy.any(keep_indices): if numpy.any(keep_indices):
bounding_box_raw, face_landmark_5_raw, score_raw = bounding_box_raw[keep_indices], face_landmark_5_raw[keep_indices], score_raw[keep_indices] bounding_boxes_raw, face_scores_raw, face_landmarks_5_raw = bounding_boxes_raw[keep_indices], face_scores_raw[keep_indices], face_landmarks_5_raw[keep_indices]
for bounding_box in bounding_box_raw: for bounding_box_raw in bounding_boxes_raw:
bounding_boxes.append(numpy.array( bounding_boxes.append(numpy.array(
[ [
(bounding_box[0] - bounding_box[2] / 2) * ratio_width, (bounding_box_raw[0] - bounding_box_raw[2] / 2) * ratio_width,
(bounding_box[1] - bounding_box[3] / 2) * ratio_height, (bounding_box_raw[1] - bounding_box_raw[3] / 2) * ratio_height,
(bounding_box[0] + bounding_box[2] / 2) * ratio_width, (bounding_box_raw[0] + bounding_box_raw[2] / 2) * ratio_width,
(bounding_box[1] + bounding_box[3] / 2) * ratio_height, (bounding_box_raw[1] + bounding_box_raw[3] / 2) * ratio_height
])) ]))
face_scores = score_raw.ravel().tolist() face_scores = face_scores_raw.ravel().tolist()
face_landmark_5_raw[:, 0::3] = (face_landmark_5_raw[:, 0::3]) * ratio_width face_landmarks_5_raw[:, 0::3] = (face_landmarks_5_raw[:, 0::3]) * ratio_width
face_landmark_5_raw[:, 1::3] = (face_landmark_5_raw[:, 1::3]) * ratio_height face_landmarks_5_raw[:, 1::3] = (face_landmarks_5_raw[:, 1::3]) * ratio_height
for face_landmark_5 in face_landmark_5_raw: for face_landmark_raw_5 in face_landmarks_5_raw:
face_landmarks_5.append(numpy.array(face_landmark_5.reshape(-1, 3)[:, :2])) face_landmarks_5.append(numpy.array(face_landmark_raw_5.reshape(-1, 3)[:, :2]))
return bounding_boxes, face_scores, face_landmarks_5
def detect_with_yunet(vision_frame : VisionFrame, face_detector_size : str) -> Tuple[List[BoundingBox], List[Score], List[FaceLandmark5]]:
bounding_boxes = []
face_scores = []
face_landmarks_5 = []
feature_strides = [ 8, 16, 32 ]
feature_map_channel = 3
anchor_total = 1
face_detector_score = state_manager.get_item('face_detector_score')
face_detector_width, face_detector_height = unpack_resolution(face_detector_size)
temp_vision_frame = restrict_frame(vision_frame, (face_detector_width, face_detector_height))
ratio_height = vision_frame.shape[0] / temp_vision_frame.shape[0]
ratio_width = vision_frame.shape[1] / temp_vision_frame.shape[1]
detect_vision_frame = prepare_detect_frame(temp_vision_frame, face_detector_size)
detect_vision_frame = normalize_detect_frame(detect_vision_frame, [ 0, 255 ])
detection = forward_with_yunet(detect_vision_frame)
for index, feature_stride in enumerate(feature_strides):
face_scores_raw = (detection[index] * detection[index + feature_map_channel]).reshape(-1)
keep_indices = numpy.where(face_scores_raw >= face_detector_score)[0]
if numpy.any(keep_indices):
stride_height = face_detector_height // feature_stride
stride_width = face_detector_width // feature_stride
anchors = create_static_anchors(feature_stride, anchor_total, stride_width, stride_height)
bounding_boxes_center = detection[index + feature_map_channel * 2].squeeze(0)[:, :2] * feature_stride + anchors
bounding_boxes_size = numpy.exp(detection[index + feature_map_channel * 2].squeeze(0)[:, 2:4]) * feature_stride
face_landmarks_5_raw = detection[index + feature_map_channel * 3].squeeze(0)
bounding_boxes_raw = numpy.stack(
[
bounding_boxes_center[:, 0] - bounding_boxes_size[:, 0] / 2,
bounding_boxes_center[:, 1] - bounding_boxes_size[:, 1] / 2,
bounding_boxes_center[:, 0] + bounding_boxes_size[:, 0] / 2,
bounding_boxes_center[:, 1] + bounding_boxes_size[:, 1] / 2
], axis = -1)
for bounding_box_raw in bounding_boxes_raw[keep_indices]:
bounding_boxes.append(numpy.array(
[
bounding_box_raw[0] * ratio_width,
bounding_box_raw[1] * ratio_height,
bounding_box_raw[2] * ratio_width,
bounding_box_raw[3] * ratio_height
]))
face_scores.extend(face_scores_raw[keep_indices])
face_landmarks_5_raw = numpy.concatenate(
[
face_landmarks_5_raw[:, [ 0, 1 ]] * feature_stride + anchors,
face_landmarks_5_raw[:, [ 2, 3 ]] * feature_stride + anchors,
face_landmarks_5_raw[:, [ 4, 5 ]] * feature_stride + anchors,
face_landmarks_5_raw[:, [ 6, 7 ]] * feature_stride + anchors,
face_landmarks_5_raw[:, [ 8, 9 ]] * feature_stride + anchors
], axis = -1).reshape(-1, 5, 2)
for face_landmark_raw_5 in face_landmarks_5_raw[keep_indices]:
face_landmarks_5.append(face_landmark_raw_5 * [ ratio_width, ratio_height ])
return bounding_boxes, face_scores, face_landmarks_5 return bounding_boxes, face_scores, face_landmarks_5
@@ -293,8 +418,20 @@ def forward_with_scrfd(detect_vision_frame : VisionFrame) -> Detection:
return detection return detection
def forward_with_yoloface(detect_vision_frame : VisionFrame) -> Detection: def forward_with_yolo_face(detect_vision_frame : VisionFrame) -> Detection:
face_detector = get_inference_pool().get('yoloface') face_detector = get_inference_pool().get('yolo_face')
with thread_semaphore():
detection = face_detector.run(None,
{
'input': detect_vision_frame
})
return detection
def forward_with_yunet(detect_vision_frame : VisionFrame) -> Detection:
face_detector = get_inference_pool().get('yunet')
with thread_semaphore(): with thread_semaphore():
detection = face_detector.run(None, detection = face_detector.run(None,
@@ -309,6 +446,13 @@ def prepare_detect_frame(temp_vision_frame : VisionFrame, face_detector_size : s
face_detector_width, face_detector_height = unpack_resolution(face_detector_size) face_detector_width, face_detector_height = unpack_resolution(face_detector_size)
detect_vision_frame = numpy.zeros((face_detector_height, face_detector_width, 3)) detect_vision_frame = numpy.zeros((face_detector_height, face_detector_width, 3))
detect_vision_frame[:temp_vision_frame.shape[0], :temp_vision_frame.shape[1], :] = temp_vision_frame detect_vision_frame[:temp_vision_frame.shape[0], :temp_vision_frame.shape[1], :] = temp_vision_frame
detect_vision_frame = (detect_vision_frame - 127.5) / 128.0
detect_vision_frame = numpy.expand_dims(detect_vision_frame.transpose(2, 0, 1), axis = 0).astype(numpy.float32) detect_vision_frame = numpy.expand_dims(detect_vision_frame.transpose(2, 0, 1), axis = 0).astype(numpy.float32)
return detect_vision_frame return detect_vision_frame
def normalize_detect_frame(detect_vision_frame : VisionFrame, normalize_range : Sequence[int]) -> VisionFrame:
if normalize_range == [ -1, 1 ]:
return (detect_vision_frame - 127.5) / 128.0
if normalize_range == [ 0, 1 ]:
return detect_vision_frame / 255.0
return detect_vision_frame
+78 -36
View File
@@ -5,9 +5,9 @@ import cv2
import numpy import numpy
from cv2.typing import Size from cv2.typing import Size
from facefusion.typing import Anchors, Angle, BoundingBox, Distance, FaceDetectorModel, FaceLandmark5, FaceLandmark68, Mask, Matrix, Points, Scale, Score, Translation, VisionFrame, WarpTemplate, WarpTemplateSet from facefusion.types import Anchors, Angle, BoundingBox, Distance, FaceDetectorModel, FaceLandmark5, FaceLandmark68, Mask, Matrix, Points, Scale, Score, Translation, VisionFrame, WarpTemplate, WarpTemplateSet
WARP_TEMPLATES : WarpTemplateSet =\ WARP_TEMPLATE_SET : WarpTemplateSet =\
{ {
'arcface_112_v1': numpy.array( 'arcface_112_v1': numpy.array(
[ [
@@ -25,7 +25,7 @@ WARP_TEMPLATES : WarpTemplateSet =\
[ 0.37097589, 0.82469196 ], [ 0.37097589, 0.82469196 ],
[ 0.63151696, 0.82325089 ] [ 0.63151696, 0.82325089 ]
]), ]),
'arcface_128_v2': numpy.array( 'arcface_128': numpy.array(
[ [
[ 0.36167656, 0.40387734 ], [ 0.36167656, 0.40387734 ],
[ 0.63696719, 0.40235469 ], [ 0.63696719, 0.40235469 ],
@@ -69,8 +69,8 @@ WARP_TEMPLATES : WarpTemplateSet =\
def estimate_matrix_by_face_landmark_5(face_landmark_5 : FaceLandmark5, warp_template : WarpTemplate, crop_size : Size) -> Matrix: def estimate_matrix_by_face_landmark_5(face_landmark_5 : FaceLandmark5, warp_template : WarpTemplate, crop_size : Size) -> Matrix:
normed_warp_template = WARP_TEMPLATES.get(warp_template) * crop_size warp_template_norm = WARP_TEMPLATE_SET.get(warp_template) * crop_size
affine_matrix = cv2.estimateAffinePartial2D(face_landmark_5, normed_warp_template, method = cv2.RANSAC, ransacReprojThreshold = 100)[0] affine_matrix = cv2.estimateAffinePartial2D(face_landmark_5, warp_template_norm, method = cv2.RANSAC, ransacReprojThreshold = 100)[0]
return affine_matrix return affine_matrix
@@ -81,7 +81,7 @@ def warp_face_by_face_landmark_5(temp_vision_frame : VisionFrame, face_landmark_
def warp_face_by_bounding_box(temp_vision_frame : VisionFrame, bounding_box : BoundingBox, crop_size : Size) -> Tuple[VisionFrame, Matrix]: def warp_face_by_bounding_box(temp_vision_frame : VisionFrame, bounding_box : BoundingBox, crop_size : Size) -> Tuple[VisionFrame, Matrix]:
source_points = numpy.array([ [ bounding_box[0], bounding_box[1] ], [bounding_box[2], bounding_box[1] ], [ bounding_box[0], bounding_box[3] ] ]).astype(numpy.float32) source_points = numpy.array([ [ bounding_box[0], bounding_box[1] ], [ bounding_box[2], bounding_box[1] ], [ bounding_box[0], bounding_box[3] ] ]).astype(numpy.float32)
target_points = numpy.array([ [ 0, 0 ], [ crop_size[0], 0 ], [ 0, crop_size[1] ] ]).astype(numpy.float32) target_points = numpy.array([ [ 0, 0 ], [ crop_size[0], 0 ], [ 0, crop_size[1] ] ]).astype(numpy.float32)
affine_matrix = cv2.getAffineTransform(source_points, target_points) affine_matrix = cv2.getAffineTransform(source_points, target_points)
if bounding_box[2] - bounding_box[0] > crop_size[0] or bounding_box[3] - bounding_box[1] > crop_size[1]: if bounding_box[2] - bounding_box[0] > crop_size[0] or bounding_box[3] - bounding_box[1] > crop_size[1]:
@@ -98,39 +98,59 @@ def warp_face_by_translation(temp_vision_frame : VisionFrame, translation : Tran
return crop_vision_frame, affine_matrix return crop_vision_frame, affine_matrix
def paste_back(temp_vision_frame : VisionFrame, crop_vision_frame : VisionFrame, crop_mask : Mask, affine_matrix : Matrix) -> VisionFrame: def paste_back(temp_vision_frame : VisionFrame, crop_vision_frame : VisionFrame, crop_vision_mask : Mask, affine_matrix : Matrix) -> VisionFrame:
paste_bounding_box, paste_matrix = calculate_paste_area(temp_vision_frame, crop_vision_frame, affine_matrix)
x1, y1, x2, y2 = paste_bounding_box
paste_width = x2 - x1
paste_height = y2 - y1
inverse_vision_mask = cv2.warpAffine(crop_vision_mask, paste_matrix, (paste_width, paste_height)).clip(0, 1)
inverse_vision_mask = numpy.expand_dims(inverse_vision_mask, axis = -1)
inverse_vision_frame = cv2.warpAffine(crop_vision_frame, paste_matrix, (paste_width, paste_height), borderMode = cv2.BORDER_REPLICATE)
temp_vision_frame = temp_vision_frame.copy()
paste_vision_frame = temp_vision_frame[y1:y2, x1:x2]
paste_vision_frame = paste_vision_frame * (1 - inverse_vision_mask) + inverse_vision_frame * inverse_vision_mask
temp_vision_frame[y1:y2, x1:x2] = paste_vision_frame.astype(temp_vision_frame.dtype)
return temp_vision_frame
def calculate_paste_area(temp_vision_frame : VisionFrame, crop_vision_frame : VisionFrame, affine_matrix : Matrix) -> Tuple[BoundingBox, Matrix]:
temp_height, temp_width = temp_vision_frame.shape[:2]
crop_height, crop_width = crop_vision_frame.shape[:2]
inverse_matrix = cv2.invertAffineTransform(affine_matrix) inverse_matrix = cv2.invertAffineTransform(affine_matrix)
temp_size = temp_vision_frame.shape[:2][::-1] crop_points = numpy.array([ [ 0, 0 ], [ crop_width, 0 ], [ crop_width, crop_height ], [ 0, crop_height ] ])
inverse_mask = cv2.warpAffine(crop_mask, inverse_matrix, temp_size).clip(0, 1) paste_region_points = transform_points(crop_points, inverse_matrix)
inverse_vision_frame = cv2.warpAffine(crop_vision_frame, inverse_matrix, temp_size, borderMode = cv2.BORDER_REPLICATE) paste_region_point_min = numpy.floor(paste_region_points.min(axis = 0)).astype(int)
paste_vision_frame = temp_vision_frame.copy() paste_region_point_max = numpy.ceil(paste_region_points.max(axis = 0)).astype(int)
paste_vision_frame[:, :, 0] = inverse_mask * inverse_vision_frame[:, :, 0] + (1 - inverse_mask) * temp_vision_frame[:, :, 0] x1, y1 = numpy.clip(paste_region_point_min, 0, [ temp_width, temp_height ])
paste_vision_frame[:, :, 1] = inverse_mask * inverse_vision_frame[:, :, 1] + (1 - inverse_mask) * temp_vision_frame[:, :, 1] x2, y2 = numpy.clip(paste_region_point_max, 0, [ temp_width, temp_height ])
paste_vision_frame[:, :, 2] = inverse_mask * inverse_vision_frame[:, :, 2] + (1 - inverse_mask) * temp_vision_frame[:, :, 2] paste_bounding_box = numpy.array([ x1, y1, x2, y2 ])
return paste_vision_frame paste_matrix = inverse_matrix.copy()
paste_matrix[0, 2] -= x1
paste_matrix[1, 2] -= y1
return paste_bounding_box, paste_matrix
@lru_cache(maxsize = None) @lru_cache()
def create_static_anchors(feature_stride : int, anchor_total : int, stride_height : int, stride_width : int) -> Anchors: def create_static_anchors(feature_stride : int, anchor_total : int, stride_width : int, stride_height : int) -> Anchors:
y, x = numpy.mgrid[:stride_height, :stride_width][::-1] x, y = numpy.mgrid[:stride_width, :stride_height]
anchors = numpy.stack((y, x), axis = -1) anchors = numpy.stack((y, x), axis = -1)
anchors = (anchors * feature_stride).reshape((-1, 2)) anchors = (anchors * feature_stride).reshape((-1, 2))
anchors = numpy.stack([ anchors ] * anchor_total, axis = 1).reshape((-1, 2)) anchors = numpy.stack([ anchors ] * anchor_total, axis = 1).reshape((-1, 2))
return anchors return anchors
def create_rotated_matrix_and_size(angle : Angle, size : Size) -> Tuple[Matrix, Size]: def create_rotation_matrix_and_size(angle : Angle, size : Size) -> Tuple[Matrix, Size]:
rotated_matrix = cv2.getRotationMatrix2D((size[0] / 2, size[1] / 2), angle, 1) rotation_matrix = cv2.getRotationMatrix2D((size[0] / 2, size[1] / 2), angle, 1)
rotated_size = numpy.dot(numpy.abs(rotated_matrix[:, :2]), size) rotation_size = numpy.dot(numpy.abs(rotation_matrix[:, :2]), size)
rotated_matrix[:, -1] += (rotated_size - size) * 0.5 #type:ignore[misc] rotation_matrix[:, -1] += (rotation_size - size) * 0.5 #type:ignore[misc]
rotated_size = int(rotated_size[0]), int(rotated_size[1]) rotation_size = int(rotation_size[0]), int(rotation_size[1])
return rotated_matrix, rotated_size return rotation_matrix, rotation_size
def create_bounding_box(face_landmark_68 : FaceLandmark68) -> BoundingBox: def create_bounding_box(face_landmark_68 : FaceLandmark68) -> BoundingBox:
min_x, min_y = numpy.min(face_landmark_68, axis = 0) x1, y1 = numpy.min(face_landmark_68, axis = 0)
max_x, max_y = numpy.max(face_landmark_68, axis = 0) x2, y2 = numpy.max(face_landmark_68, axis = 0)
bounding_box = normalize_bounding_box(numpy.array([ min_x, min_y, max_x, max_y ])) bounding_box = normalize_bounding_box(numpy.array([ x1, y1, x2, y2 ]))
return bounding_box return bounding_box
@@ -208,9 +228,9 @@ def estimate_face_angle(face_landmark_68 : FaceLandmark68) -> Angle:
return face_angle return face_angle
def apply_nms(bounding_boxes : List[BoundingBox], face_scores : List[Score], score_threshold : float, nms_threshold : float) -> Sequence[int]: def apply_nms(bounding_boxes : List[BoundingBox], scores : List[Score], score_threshold : float, nms_threshold : float) -> Sequence[int]:
normed_bounding_boxes = [ (x1, y1, x2 - x1, y2 - y1) for (x1, y1, x2, y2) in bounding_boxes ] bounding_boxes_norm = [ (x1, y1, x2 - x1, y2 - y1) for (x1, y1, x2, y2) in bounding_boxes ]
keep_indices = cv2.dnn.NMSBoxes(normed_bounding_boxes, face_scores, score_threshold = score_threshold, nms_threshold = nms_threshold) keep_indices = cv2.dnn.NMSBoxes(bounding_boxes_norm, scores, score_threshold = score_threshold, nms_threshold = nms_threshold)
return keep_indices return keep_indices
@@ -226,9 +246,31 @@ def get_nms_threshold(face_detector_model : FaceDetectorModel, face_detector_ang
return 0.4 return 0.4
def merge_matrix(matrices : List[Matrix]) -> Matrix: def merge_matrix(temp_matrices : List[Matrix]) -> Matrix:
merged_matrix = numpy.vstack([ matrices[0], [ 0, 0, 1 ] ]) matrix = numpy.vstack([ temp_matrices[0], [ 0, 0, 1 ] ])
for matrix in matrices[1:]:
matrix = numpy.vstack([ matrix, [ 0, 0, 1 ] ]) for temp_matrix in temp_matrices[1:]:
merged_matrix = numpy.dot(merged_matrix, matrix) temp_matrix = numpy.vstack([ temp_matrix, [ 0, 0, 1 ] ])
return merged_matrix[:2, :] matrix = numpy.dot(temp_matrix, matrix)
return matrix[:2, :]
def calculate_bounding_box_overlap(bounding_box_a : BoundingBox, bounding_box_b : BoundingBox) -> float:
intersection_x1 = max(bounding_box_a[0], bounding_box_b[0])
intersection_y1 = max(bounding_box_a[1], bounding_box_b[1])
intersection_x2 = min(bounding_box_a[2], bounding_box_b[2])
intersection_y2 = min(bounding_box_a[3], bounding_box_b[3])
intersection = max(0, intersection_x2 - intersection_x1) * max(0, intersection_y2 - intersection_y1)
bounding_box_area = (bounding_box_a[2] - bounding_box_a[0]) * (bounding_box_a[3] - bounding_box_a[1])
reference_bounding_box_area = (bounding_box_b[2] - bounding_box_b[0]) * (bounding_box_b[3] - bounding_box_b[1])
union = bounding_box_area + reference_bounding_box_area - intersection
if union > 0:
return intersection / union
return 0.0
def average_points(points_previous : Points, points_next : Points, average_factor : float) -> Points:
return points_previous * (1 - average_factor) + points_next * average_factor
+43 -25
View File
@@ -6,18 +6,24 @@ import numpy
from facefusion import inference_manager, state_manager from facefusion import inference_manager, state_manager
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url
from facefusion.face_helper import create_rotated_matrix_and_size, estimate_matrix_by_face_landmark_5, transform_points, warp_face_by_translation from facefusion.face_helper import create_rotation_matrix_and_size, estimate_matrix_by_face_landmark_5, transform_points, warp_face_by_translation
from facefusion.filesystem import resolve_relative_path from facefusion.filesystem import resolve_relative_path
from facefusion.thread_helper import conditional_thread_semaphore from facefusion.thread_helper import conditional_thread_semaphore
from facefusion.typing import Angle, BoundingBox, DownloadScope, DownloadSet, FaceLandmark5, FaceLandmark68, InferencePool, ModelSet, Prediction, Score, VisionFrame from facefusion.types import Angle, BoundingBox, DownloadScope, DownloadSet, FaceLandmark5, FaceLandmark68, InferencePool, ModelSet, Prediction, Score, VisionFrame
@lru_cache(maxsize = None) @lru_cache()
def create_static_model_set(download_scope : DownloadScope) -> ModelSet: def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
return\ return\
{ {
'2dfan4': '2dfan4':
{ {
'__metadata__':
{
'vendor': 'breadbread1984',
'license': 'MIT',
'year': 2018
},
'hashes': 'hashes':
{ {
'2dfan4': '2dfan4':
@@ -38,6 +44,12 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
}, },
'peppa_wutz': 'peppa_wutz':
{ {
'__metadata__':
{
'vendor': 'Unknown',
'license': 'Apache-2.0',
'year': 2023
},
'hashes': 'hashes':
{ {
'peppa_wutz': 'peppa_wutz':
@@ -58,6 +70,12 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
}, },
'fan_68_5': 'fan_68_5':
{ {
'__metadata__':
{
'vendor': 'FaceFusion',
'license': 'OpenRAIL-M',
'year': 2024
},
'hashes': 'hashes':
{ {
'fan_68_5': 'fan_68_5':
@@ -79,43 +97,43 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
def get_inference_pool() -> InferencePool: def get_inference_pool() -> InferencePool:
_, model_sources = collect_model_downloads() model_names = [ state_manager.get_item('face_landmarker_model'), 'fan_68_5' ]
return inference_manager.get_inference_pool(__name__, model_sources) _, model_source_set = collect_model_downloads()
return inference_manager.get_inference_pool(__name__, model_names, model_source_set)
def clear_inference_pool() -> None: def clear_inference_pool() -> None:
inference_manager.clear_inference_pool(__name__) model_names = [ state_manager.get_item('face_landmarker_model'), 'fan_68_5' ]
inference_manager.clear_inference_pool(__name__, model_names)
def collect_model_downloads() -> Tuple[DownloadSet, DownloadSet]: def collect_model_downloads() -> Tuple[DownloadSet, DownloadSet]:
model_set = create_static_model_set('full') model_set = create_static_model_set('full')
model_hashes =\ model_hash_set =\
{ {
'fan_68_5': model_set.get('fan_68_5').get('hashes').get('fan_68_5') 'fan_68_5': model_set.get('fan_68_5').get('hashes').get('fan_68_5')
} }
model_sources =\ model_source_set =\
{ {
'fan_68_5': model_set.get('fan_68_5').get('sources').get('fan_68_5') 'fan_68_5': model_set.get('fan_68_5').get('sources').get('fan_68_5')
} }
if state_manager.get_item('face_landmarker_model') in [ 'many', '2dfan4' ]: for face_landmarker_model in [ '2dfan4', 'peppa_wutz' ]:
model_hashes['2dfan4'] = model_set.get('2dfan4').get('hashes').get('2dfan4') if state_manager.get_item('face_landmarker_model') in [ 'many', face_landmarker_model ]:
model_sources['2dfan4'] = model_set.get('2dfan4').get('sources').get('2dfan4') model_hash_set[face_landmarker_model] = model_set.get(face_landmarker_model).get('hashes').get(face_landmarker_model)
model_source_set[face_landmarker_model] = model_set.get(face_landmarker_model).get('sources').get(face_landmarker_model)
if state_manager.get_item('face_landmarker_model') in [ 'many', 'peppa_wutz' ]: return model_hash_set, model_source_set
model_hashes['peppa_wutz'] = model_set.get('peppa_wutz').get('hashes').get('peppa_wutz')
model_sources['peppa_wutz'] = model_set.get('peppa_wutz').get('sources').get('peppa_wutz')
return model_hashes, model_sources
def pre_check() -> bool: def pre_check() -> bool:
model_hashes, model_sources = collect_model_downloads() model_hash_set, model_source_set = collect_model_downloads()
return conditional_download_hashes(model_hashes) and conditional_download_sources(model_sources) return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set)
def detect_face_landmarks(vision_frame : VisionFrame, bounding_box : BoundingBox, face_angle : Angle) -> Tuple[FaceLandmark68, Score]: def detect_face_landmark(vision_frame : VisionFrame, bounding_box : BoundingBox, face_angle : Angle) -> Tuple[FaceLandmark68, Score]:
face_landmark_2dfan4 = None face_landmark_2dfan4 = None
face_landmark_peppa_wutz = None face_landmark_peppa_wutz = None
face_landmark_score_2dfan4 = 0.0 face_landmark_score_2dfan4 = 0.0
@@ -136,14 +154,14 @@ def detect_with_2dfan4(temp_vision_frame: VisionFrame, bounding_box: BoundingBox
model_size = create_static_model_set('full').get('2dfan4').get('size') model_size = create_static_model_set('full').get('2dfan4').get('size')
scale = 195 / numpy.subtract(bounding_box[2:], bounding_box[:2]).max().clip(1, None) scale = 195 / numpy.subtract(bounding_box[2:], bounding_box[:2]).max().clip(1, None)
translation = (model_size[0] - numpy.add(bounding_box[2:], bounding_box[:2]) * scale) * 0.5 translation = (model_size[0] - numpy.add(bounding_box[2:], bounding_box[:2]) * scale) * 0.5
rotated_matrix, rotated_size = create_rotated_matrix_and_size(face_angle, model_size) rotation_matrix, rotation_size = create_rotation_matrix_and_size(face_angle, model_size)
crop_vision_frame, affine_matrix = warp_face_by_translation(temp_vision_frame, translation, scale, model_size) crop_vision_frame, affine_matrix = warp_face_by_translation(temp_vision_frame, translation, scale, model_size)
crop_vision_frame = cv2.warpAffine(crop_vision_frame, rotated_matrix, rotated_size) crop_vision_frame = cv2.warpAffine(crop_vision_frame, rotation_matrix, rotation_size)
crop_vision_frame = conditional_optimize_contrast(crop_vision_frame) crop_vision_frame = conditional_optimize_contrast(crop_vision_frame)
crop_vision_frame = crop_vision_frame.transpose(2, 0, 1).astype(numpy.float32) / 255.0 crop_vision_frame = crop_vision_frame.transpose(2, 0, 1).astype(numpy.float32) / 255.0
face_landmark_68, face_heatmap = forward_with_2dfan4(crop_vision_frame) face_landmark_68, face_heatmap = forward_with_2dfan4(crop_vision_frame)
face_landmark_68 = face_landmark_68[:, :, :2][0] / 64 * 256 face_landmark_68 = face_landmark_68[:, :, :2][0] / 64 * 256
face_landmark_68 = transform_points(face_landmark_68, cv2.invertAffineTransform(rotated_matrix)) face_landmark_68 = transform_points(face_landmark_68, cv2.invertAffineTransform(rotation_matrix))
face_landmark_68 = transform_points(face_landmark_68, cv2.invertAffineTransform(affine_matrix)) face_landmark_68 = transform_points(face_landmark_68, cv2.invertAffineTransform(affine_matrix))
face_landmark_score_68 = numpy.amax(face_heatmap, axis = (2, 3)) face_landmark_score_68 = numpy.amax(face_heatmap, axis = (2, 3))
face_landmark_score_68 = numpy.mean(face_landmark_score_68) face_landmark_score_68 = numpy.mean(face_landmark_score_68)
@@ -155,15 +173,15 @@ def detect_with_peppa_wutz(temp_vision_frame : VisionFrame, bounding_box : Bound
model_size = create_static_model_set('full').get('peppa_wutz').get('size') model_size = create_static_model_set('full').get('peppa_wutz').get('size')
scale = 195 / numpy.subtract(bounding_box[2:], bounding_box[:2]).max().clip(1, None) scale = 195 / numpy.subtract(bounding_box[2:], bounding_box[:2]).max().clip(1, None)
translation = (model_size[0] - numpy.add(bounding_box[2:], bounding_box[:2]) * scale) * 0.5 translation = (model_size[0] - numpy.add(bounding_box[2:], bounding_box[:2]) * scale) * 0.5
rotated_matrix, rotated_size = create_rotated_matrix_and_size(face_angle, model_size) rotation_matrix, rotation_size = create_rotation_matrix_and_size(face_angle, model_size)
crop_vision_frame, affine_matrix = warp_face_by_translation(temp_vision_frame, translation, scale, model_size) crop_vision_frame, affine_matrix = warp_face_by_translation(temp_vision_frame, translation, scale, model_size)
crop_vision_frame = cv2.warpAffine(crop_vision_frame, rotated_matrix, rotated_size) crop_vision_frame = cv2.warpAffine(crop_vision_frame, rotation_matrix, rotation_size)
crop_vision_frame = conditional_optimize_contrast(crop_vision_frame) crop_vision_frame = conditional_optimize_contrast(crop_vision_frame)
crop_vision_frame = crop_vision_frame.transpose(2, 0, 1).astype(numpy.float32) / 255.0 crop_vision_frame = crop_vision_frame.transpose(2, 0, 1).astype(numpy.float32) / 255.0
crop_vision_frame = numpy.expand_dims(crop_vision_frame, axis = 0) crop_vision_frame = numpy.expand_dims(crop_vision_frame, axis = 0)
prediction = forward_with_peppa_wutz(crop_vision_frame) prediction = forward_with_peppa_wutz(crop_vision_frame)
face_landmark_68 = prediction.reshape(-1, 3)[:, :2] / 64 * model_size[0] face_landmark_68 = prediction.reshape(-1, 3)[:, :2] / 64 * model_size[0]
face_landmark_68 = transform_points(face_landmark_68, cv2.invertAffineTransform(rotated_matrix)) face_landmark_68 = transform_points(face_landmark_68, cv2.invertAffineTransform(rotation_matrix))
face_landmark_68 = transform_points(face_landmark_68, cv2.invertAffineTransform(affine_matrix)) face_landmark_68 = transform_points(face_landmark_68, cv2.invertAffineTransform(affine_matrix))
face_landmark_score_68 = prediction.reshape(-1, 3)[:, 2].mean() face_landmark_score_68 = prediction.reshape(-1, 3)[:, 2].mean()
face_landmark_score_68 = numpy.interp(face_landmark_score_68, [ 0, 0.95 ], [ 0, 1 ]) face_landmark_score_68 = numpy.interp(face_landmark_score_68, [ 0, 0.95 ], [ 0, 1 ])
+114 -52
View File
@@ -3,22 +3,27 @@ from typing import List, Tuple
import cv2 import cv2
import numpy import numpy
from cv2.typing import Size
import facefusion.choices import facefusion.choices
from facefusion import inference_manager, state_manager from facefusion import inference_manager, state_manager
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url
from facefusion.filesystem import resolve_relative_path from facefusion.filesystem import resolve_relative_path
from facefusion.thread_helper import conditional_thread_semaphore from facefusion.thread_helper import conditional_thread_semaphore
from facefusion.typing import DownloadScope, DownloadSet, FaceLandmark68, FaceMaskRegion, InferencePool, Mask, ModelSet, Padding, VisionFrame from facefusion.types import DownloadScope, DownloadSet, FaceLandmark68, FaceMaskArea, FaceMaskRegion, InferencePool, Mask, ModelSet, Padding, VisionFrame
@lru_cache(maxsize = None) @lru_cache()
def create_static_model_set(download_scope : DownloadScope) -> ModelSet: def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
return\ return\
{ {
'xseg_1': 'xseg_1':
{ {
'__metadata__':
{
'vendor': 'DeepFaceLab',
'license': 'GPL-3.0',
'year': 2021
},
'hashes': 'hashes':
{ {
'face_occluder': 'face_occluder':
@@ -39,6 +44,12 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
}, },
'xseg_2': 'xseg_2':
{ {
'__metadata__':
{
'vendor': 'DeepFaceLab',
'license': 'GPL-3.0',
'year': 2021
},
'hashes': 'hashes':
{ {
'face_occluder': 'face_occluder':
@@ -57,8 +68,40 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
}, },
'size': (256, 256) 'size': (256, 256)
}, },
'xseg_3':
{
'__metadata__':
{
'vendor': 'DeepFaceLab',
'license': 'GPL-3.0',
'year': 2021
},
'hashes':
{
'face_occluder':
{
'url': resolve_download_url('models-3.2.0', 'xseg_3.hash'),
'path': resolve_relative_path('../.assets/models/xseg_3.hash')
}
},
'sources':
{
'face_occluder':
{
'url': resolve_download_url('models-3.2.0', 'xseg_3.onnx'),
'path': resolve_relative_path('../.assets/models/xseg_3.onnx')
}
},
'size': (256, 256)
},
'bisenet_resnet_18': 'bisenet_resnet_18':
{ {
'__metadata__':
{
'vendor': 'yakhyo',
'license': 'MIT',
'year': 2024
},
'hashes': 'hashes':
{ {
'face_parser': 'face_parser':
@@ -79,6 +122,12 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
}, },
'bisenet_resnet_34': 'bisenet_resnet_34':
{ {
'__metadata__':
{
'vendor': 'yakhyo',
'license': 'MIT',
'year': 2024
},
'hashes': 'hashes':
{ {
'face_parser': 'face_parser':
@@ -101,46 +150,43 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
def get_inference_pool() -> InferencePool: def get_inference_pool() -> InferencePool:
_, model_sources = collect_model_downloads() model_names = [ state_manager.get_item('face_occluder_model'), state_manager.get_item('face_parser_model') ]
return inference_manager.get_inference_pool(__name__, model_sources) _, model_source_set = collect_model_downloads()
return inference_manager.get_inference_pool(__name__, model_names, model_source_set)
def clear_inference_pool() -> None: def clear_inference_pool() -> None:
inference_manager.clear_inference_pool(__name__) model_names = [ state_manager.get_item('face_occluder_model'), state_manager.get_item('face_parser_model') ]
inference_manager.clear_inference_pool(__name__, model_names)
def collect_model_downloads() -> Tuple[DownloadSet, DownloadSet]: def collect_model_downloads() -> Tuple[DownloadSet, DownloadSet]:
model_hashes = {}
model_sources = {}
model_set = create_static_model_set('full') model_set = create_static_model_set('full')
model_hash_set = {}
model_source_set = {}
if state_manager.get_item('face_occluder_model') == 'xseg_1': for face_occluder_model in [ 'xseg_1', 'xseg_2', 'xseg_3' ]:
model_hashes['xseg_1'] = model_set.get('xseg_1').get('hashes').get('face_occluder') if state_manager.get_item('face_occluder_model') in [ 'many', face_occluder_model ]:
model_sources['xseg_1'] = model_set.get('xseg_1').get('sources').get('face_occluder') model_hash_set[face_occluder_model] = model_set.get(face_occluder_model).get('hashes').get('face_occluder')
model_source_set[face_occluder_model] = model_set.get(face_occluder_model).get('sources').get('face_occluder')
if state_manager.get_item('face_occluder_model') == 'xseg_2': for face_parser_model in [ 'bisenet_resnet_18', 'bisenet_resnet_34' ]:
model_hashes['xseg_2'] = model_set.get('xseg_2').get('hashes').get('face_occluder') if state_manager.get_item('face_parser_model') == face_parser_model:
model_sources['xseg_2'] = model_set.get('xseg_2').get('sources').get('face_occluder') model_hash_set[face_parser_model] = model_set.get(face_parser_model).get('hashes').get('face_parser')
model_source_set[face_parser_model] = model_set.get(face_parser_model).get('sources').get('face_parser')
if state_manager.get_item('face_parser_model') == 'bisenet_resnet_18': return model_hash_set, model_source_set
model_hashes['bisenet_resnet_18'] = model_set.get('bisenet_resnet_18').get('hashes').get('face_parser')
model_sources['bisenet_resnet_18'] = model_set.get('bisenet_resnet_18').get('sources').get('face_parser')
if state_manager.get_item('face_parser_model') == 'bisenet_resnet_34':
model_hashes['bisenet_resnet_34'] = model_set.get('bisenet_resnet_34').get('hashes').get('face_parser')
model_sources['bisenet_resnet_34'] = model_set.get('bisenet_resnet_34').get('sources').get('face_parser')
return model_hashes, model_sources
def pre_check() -> bool: def pre_check() -> bool:
model_hashes, model_sources = collect_model_downloads() model_hash_set, model_source_set = collect_model_downloads()
return conditional_download_hashes(model_hashes) and conditional_download_sources(model_sources) return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set)
@lru_cache(maxsize = None) def create_box_mask(crop_vision_frame : VisionFrame, face_mask_blur : float, face_mask_padding : Padding) -> Mask:
def create_static_box_mask(crop_size : Size, face_mask_blur : float, face_mask_padding : Padding) -> Mask: crop_size = crop_vision_frame.shape[:2][::-1]
blur_amount = int(crop_size[0] * 0.5 * face_mask_blur) blur_amount = int(crop_size[0] * 0.5 * face_mask_blur)
blur_area = max(blur_amount // 2, 1) blur_area = max(blur_amount // 2, 1)
box_mask : Mask = numpy.ones(crop_size).astype(numpy.float32) box_mask : Mask = numpy.ones(crop_size).astype(numpy.float32)
@@ -148,29 +194,55 @@ def create_static_box_mask(crop_size : Size, face_mask_blur : float, face_mask_p
box_mask[-max(blur_area, int(crop_size[1] * face_mask_padding[2] / 100)):, :] = 0 box_mask[-max(blur_area, int(crop_size[1] * face_mask_padding[2] / 100)):, :] = 0
box_mask[:, :max(blur_area, int(crop_size[0] * face_mask_padding[3] / 100))] = 0 box_mask[:, :max(blur_area, int(crop_size[0] * face_mask_padding[3] / 100))] = 0
box_mask[:, -max(blur_area, int(crop_size[0] * face_mask_padding[1] / 100)):] = 0 box_mask[:, -max(blur_area, int(crop_size[0] * face_mask_padding[1] / 100)):] = 0
if blur_amount > 0: if blur_amount > 0:
box_mask = cv2.GaussianBlur(box_mask, (0, 0), blur_amount * 0.25) box_mask = cv2.GaussianBlur(box_mask, (0, 0), blur_amount * 0.25)
return box_mask return box_mask
def create_occlusion_mask(crop_vision_frame : VisionFrame) -> Mask: def create_occlusion_mask(crop_vision_frame : VisionFrame) -> Mask:
face_occluder_model = state_manager.get_item('face_occluder_model') temp_masks = []
model_size = create_static_model_set('full').get(face_occluder_model).get('size')
prepare_vision_frame = cv2.resize(crop_vision_frame, model_size) if state_manager.get_item('face_occluder_model') == 'many':
prepare_vision_frame = numpy.expand_dims(prepare_vision_frame, axis = 0).astype(numpy.float32) / 255 model_names = [ 'xseg_1', 'xseg_2', 'xseg_3' ]
prepare_vision_frame = prepare_vision_frame.transpose(0, 1, 2, 3) else:
occlusion_mask = forward_occlude_face(prepare_vision_frame) model_names = [ state_manager.get_item('face_occluder_model') ]
occlusion_mask = occlusion_mask.transpose(0, 1, 2).clip(0, 1).astype(numpy.float32)
occlusion_mask = cv2.resize(occlusion_mask, crop_vision_frame.shape[:2][::-1]) for model_name in model_names:
model_size = create_static_model_set('full').get(model_name).get('size')
prepare_vision_frame = cv2.resize(crop_vision_frame, model_size)
prepare_vision_frame = numpy.expand_dims(prepare_vision_frame, axis = 0).astype(numpy.float32) / 255.0
prepare_vision_frame = prepare_vision_frame.transpose(0, 1, 2, 3)
temp_mask = forward_occlude_face(prepare_vision_frame, model_name)
temp_mask = temp_mask.transpose(0, 1, 2).clip(0, 1).astype(numpy.float32)
temp_mask = cv2.resize(temp_mask, crop_vision_frame.shape[:2][::-1])
temp_masks.append(temp_mask)
occlusion_mask = numpy.minimum.reduce(temp_masks)
occlusion_mask = (cv2.GaussianBlur(occlusion_mask.clip(0, 1), (0, 0), 5).clip(0.5, 1) - 0.5) * 2 occlusion_mask = (cv2.GaussianBlur(occlusion_mask.clip(0, 1), (0, 0), 5).clip(0.5, 1) - 0.5) * 2
return occlusion_mask return occlusion_mask
def create_area_mask(crop_vision_frame : VisionFrame, face_landmark_68 : FaceLandmark68, face_mask_areas : List[FaceMaskArea]) -> Mask:
crop_size = crop_vision_frame.shape[:2][::-1]
landmark_points = []
for face_mask_area in face_mask_areas:
if face_mask_area in facefusion.choices.face_mask_area_set:
landmark_points.extend(facefusion.choices.face_mask_area_set.get(face_mask_area))
convex_hull = cv2.convexHull(face_landmark_68[landmark_points].astype(numpy.int32))
area_mask = numpy.zeros(crop_size).astype(numpy.float32)
cv2.fillConvexPoly(area_mask, convex_hull, 1.0) #type:ignore[call-overload]
area_mask = (cv2.GaussianBlur(area_mask.clip(0, 1), (0, 0), 5).clip(0.5, 1) - 0.5) * 2
return area_mask
def create_region_mask(crop_vision_frame : VisionFrame, face_mask_regions : List[FaceMaskRegion]) -> Mask: def create_region_mask(crop_vision_frame : VisionFrame, face_mask_regions : List[FaceMaskRegion]) -> Mask:
face_parser_model = state_manager.get_item('face_parser_model') model_name = state_manager.get_item('face_parser_model')
model_size = create_static_model_set('full').get(face_parser_model).get('size') model_size = create_static_model_set('full').get(model_name).get('size')
prepare_vision_frame = cv2.resize(crop_vision_frame, model_size) prepare_vision_frame = cv2.resize(crop_vision_frame, model_size)
prepare_vision_frame = prepare_vision_frame[:, :, ::-1].astype(numpy.float32) / 255 prepare_vision_frame = prepare_vision_frame[:, :, ::-1].astype(numpy.float32) / 255.0
prepare_vision_frame = numpy.subtract(prepare_vision_frame, numpy.array([ 0.485, 0.456, 0.406 ]).astype(numpy.float32)) prepare_vision_frame = numpy.subtract(prepare_vision_frame, numpy.array([ 0.485, 0.456, 0.406 ]).astype(numpy.float32))
prepare_vision_frame = numpy.divide(prepare_vision_frame, numpy.array([ 0.229, 0.224, 0.225 ]).astype(numpy.float32)) prepare_vision_frame = numpy.divide(prepare_vision_frame, numpy.array([ 0.229, 0.224, 0.225 ]).astype(numpy.float32))
prepare_vision_frame = numpy.expand_dims(prepare_vision_frame, axis = 0) prepare_vision_frame = numpy.expand_dims(prepare_vision_frame, axis = 0)
@@ -182,18 +254,8 @@ def create_region_mask(crop_vision_frame : VisionFrame, face_mask_regions : List
return region_mask return region_mask
def create_mouth_mask(face_landmark_68 : FaceLandmark68) -> Mask: def forward_occlude_face(prepare_vision_frame : VisionFrame, model_name : str) -> Mask:
convex_hull = cv2.convexHull(face_landmark_68[numpy.r_[3:14, 31:36]].astype(numpy.int32)) face_occluder = get_inference_pool().get(model_name)
mouth_mask : Mask = numpy.zeros((512, 512)).astype(numpy.float32)
mouth_mask = cv2.fillConvexPoly(mouth_mask, convex_hull, 1.0) #type:ignore[call-overload]
mouth_mask = cv2.erode(mouth_mask.clip(0, 1), numpy.ones((21, 3)))
mouth_mask = cv2.GaussianBlur(mouth_mask, (0, 0), sigmaX = 1, sigmaY = 15)
return mouth_mask
def forward_occlude_face(prepare_vision_frame : VisionFrame) -> Mask:
face_occluder_model = state_manager.get_item('face_occluder_model')
face_occluder = get_inference_pool().get(face_occluder_model)
with conditional_thread_semaphore(): with conditional_thread_semaphore():
occlusion_mask : Mask = face_occluder.run(None, occlusion_mask : Mask = face_occluder.run(None,
@@ -205,8 +267,8 @@ def forward_occlude_face(prepare_vision_frame : VisionFrame) -> Mask:
def forward_parse_face(prepare_vision_frame : VisionFrame) -> Mask: def forward_parse_face(prepare_vision_frame : VisionFrame) -> Mask:
face_parser_model = state_manager.get_item('face_parser_model') model_name = state_manager.get_item('face_parser_model')
face_parser = get_inference_pool().get(face_parser_model) face_parser = get_inference_pool().get(model_name)
with conditional_thread_semaphore(): with conditional_thread_semaphore():
region_mask : Mask = face_parser.run(None, region_mask : Mask = face_parser.run(None,
+24 -15
View File
@@ -8,15 +8,21 @@ from facefusion.download import conditional_download_hashes, conditional_downloa
from facefusion.face_helper import warp_face_by_face_landmark_5 from facefusion.face_helper import warp_face_by_face_landmark_5
from facefusion.filesystem import resolve_relative_path from facefusion.filesystem import resolve_relative_path
from facefusion.thread_helper import conditional_thread_semaphore from facefusion.thread_helper import conditional_thread_semaphore
from facefusion.typing import DownloadScope, Embedding, FaceLandmark5, InferencePool, ModelOptions, ModelSet, VisionFrame from facefusion.types import DownloadScope, Embedding, FaceLandmark5, InferencePool, ModelOptions, ModelSet, VisionFrame
@lru_cache(maxsize = None) @lru_cache()
def create_static_model_set(download_scope : DownloadScope) -> ModelSet: def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
return\ return\
{ {
'arcface': 'arcface':
{ {
'__metadata__':
{
'vendor': 'InsightFace',
'license': 'Non-Commercial',
'year': 2018
},
'hashes': 'hashes':
{ {
'face_recognizer': 'face_recognizer':
@@ -40,12 +46,15 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
def get_inference_pool() -> InferencePool: def get_inference_pool() -> InferencePool:
model_sources = get_model_options().get('sources') model_names = [ 'arcface' ]
return inference_manager.get_inference_pool(__name__, model_sources) model_source_set = get_model_options().get('sources')
return inference_manager.get_inference_pool(__name__, model_names, model_source_set)
def clear_inference_pool() -> None: def clear_inference_pool() -> None:
inference_manager.clear_inference_pool(__name__) model_names = [ 'arcface' ]
inference_manager.clear_inference_pool(__name__, model_names)
def get_model_options() -> ModelOptions: def get_model_options() -> ModelOptions:
@@ -53,32 +62,32 @@ def get_model_options() -> ModelOptions:
def pre_check() -> bool: def pre_check() -> bool:
model_hashes = get_model_options().get('hashes') model_hash_set = get_model_options().get('hashes')
model_sources = get_model_options().get('sources') model_source_set = get_model_options().get('sources')
return conditional_download_hashes(model_hashes) and conditional_download_sources(model_sources) return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set)
def calc_embedding(temp_vision_frame : VisionFrame, face_landmark_5 : FaceLandmark5) -> Tuple[Embedding, Embedding]: def calculate_face_embedding(temp_vision_frame : VisionFrame, face_landmark_5 : FaceLandmark5) -> Tuple[Embedding, Embedding]:
model_template = get_model_options().get('template') model_template = get_model_options().get('template')
model_size = get_model_options().get('size') model_size = get_model_options().get('size')
crop_vision_frame, matrix = warp_face_by_face_landmark_5(temp_vision_frame, face_landmark_5, model_template, model_size) crop_vision_frame, matrix = warp_face_by_face_landmark_5(temp_vision_frame, face_landmark_5, model_template, model_size)
crop_vision_frame = crop_vision_frame / 127.5 - 1 crop_vision_frame = crop_vision_frame / 127.5 - 1
crop_vision_frame = crop_vision_frame[:, :, ::-1].transpose(2, 0, 1).astype(numpy.float32) crop_vision_frame = crop_vision_frame[:, :, ::-1].transpose(2, 0, 1).astype(numpy.float32)
crop_vision_frame = numpy.expand_dims(crop_vision_frame, axis = 0) crop_vision_frame = numpy.expand_dims(crop_vision_frame, axis = 0)
embedding = forward(crop_vision_frame) face_embedding = forward(crop_vision_frame)
embedding = embedding.ravel() face_embedding = face_embedding.ravel()
normed_embedding = embedding / numpy.linalg.norm(embedding) face_embedding_norm = face_embedding / numpy.linalg.norm(face_embedding)
return embedding, normed_embedding return face_embedding, face_embedding_norm
def forward(crop_vision_frame : VisionFrame) -> Embedding: def forward(crop_vision_frame : VisionFrame) -> Embedding:
face_recognizer = get_inference_pool().get('face_recognizer') face_recognizer = get_inference_pool().get('face_recognizer')
with conditional_thread_semaphore(): with conditional_thread_semaphore():
embedding = face_recognizer.run(None, face_embedding = face_recognizer.run(None,
{ {
'input': crop_vision_frame 'input': crop_vision_frame
})[0] })[0]
return embedding return face_embedding
+96 -32
View File
@@ -2,67 +2,131 @@ from typing import List
import numpy import numpy
import facefusion.choices
from facefusion import state_manager from facefusion import state_manager
from facefusion.typing import Face, FaceSelectorOrder, FaceSet, Gender, Race from facefusion.common_helper import get_first, get_middle
from facefusion.face_creator import get_one_face, get_static_faces
from facefusion.face_tracker import track_faces
from facefusion.types import Face, FaceSelectorOrder, Gender, Race, Score, VisionFrame
def find_similar_faces(faces : List[Face], reference_faces : FaceSet, face_distance : float) -> List[Face]: def select_faces(reference_vision_frame : VisionFrame, source_vision_frames : List[VisionFrame], target_vision_frames : List[VisionFrame]) -> List[Face]:
similar_faces : List[Face] = [] source_faces = get_static_faces(source_vision_frames)
if faces and reference_faces: if state_manager.get_item('face_tracker_score') > 0:
for reference_set in reference_faces: target_faces = track_faces(target_vision_frames, state_manager.get_item('face_tracker_score'))
if not similar_faces: else:
for reference_face in reference_faces[reference_set]: target_faces = get_static_faces([ get_middle(target_vision_frames) ])
for face in faces:
if compare_faces(face, reference_face, face_distance): if state_manager.get_item('face_selector_mode') == 'many':
similar_faces.append(face) return sort_and_filter_faces(source_faces, target_faces)
return similar_faces
if state_manager.get_item('face_selector_mode') == 'one':
target_face = get_one_face(sort_and_filter_faces(source_faces, target_faces))
if target_face:
return [ target_face ]
if state_manager.get_item('face_selector_mode') == 'reference':
reference_faces = get_static_faces([ reference_vision_frame ])
reference_faces = sort_and_filter_faces(source_faces, reference_faces)
reference_face = get_one_face(reference_faces, state_manager.get_item('reference_face_position'))
if reference_face:
match_faces = find_match_faces([ reference_face ], target_faces, state_manager.get_item('reference_face_distance'))
return match_faces
return []
def find_match_faces(reference_faces : List[Face], target_faces : List[Face], face_distance : float) -> List[Face]:
match_faces : List[Face] = []
for reference_face in reference_faces:
if reference_face:
for index, target_face in enumerate(target_faces):
if compare_faces(target_face, reference_face, face_distance):
match_faces.append(target_faces[index])
return match_faces
def compare_faces(face : Face, reference_face : Face, face_distance : float) -> bool: def compare_faces(face : Face, reference_face : Face, face_distance : float) -> bool:
current_face_distance = calc_face_distance(face, reference_face) current_face_distance = calculate_face_distance(face, reference_face)
current_face_distance = float(numpy.interp(current_face_distance, [ 0, 2 ], [ 0, 1 ]))
return current_face_distance < face_distance return current_face_distance < face_distance
def calc_face_distance(face : Face, reference_face : Face) -> float: def calculate_face_distance(face : Face, reference_face : Face) -> float:
if hasattr(face, 'normed_embedding') and hasattr(reference_face, 'normed_embedding'): if hasattr(face, 'embedding_norm') and hasattr(reference_face, 'embedding_norm'):
return 1 - numpy.dot(face.normed_embedding, reference_face.normed_embedding) return 1 - numpy.dot(face.embedding_norm, reference_face.embedding_norm)
return 0 return 0
def sort_and_filter_faces(faces : List[Face]) -> List[Face]: def sort_and_filter_faces(source_faces : List[Face], target_faces : List[Face]) -> List[Face]:
if faces: if target_faces:
if state_manager.get_item('face_selector_order'): if state_manager.get_item('face_selector_order'):
faces = sort_faces_by_order(faces, state_manager.get_item('face_selector_order')) target_faces = sort_faces_by_order(target_faces, state_manager.get_item('face_selector_order'))
if state_manager.get_item('face_selector_gender'):
faces = filter_faces_by_gender(faces, state_manager.get_item('face_selector_gender')) face_selector_gender = state_manager.get_item('face_selector_gender')
if state_manager.get_item('face_selector_race'): face_selector_race = state_manager.get_item('face_selector_race')
faces = filter_faces_by_race(faces, state_manager.get_item('face_selector_race'))
if source_faces and face_selector_gender == 'auto' or face_selector_race == 'auto':
source_face = get_first(sort_faces_by_order(source_faces, 'large-small'))
if source_face:
if face_selector_gender == 'auto':
face_selector_gender = source_face.gender
if face_selector_race == 'auto':
face_selector_race = source_face.race
if face_selector_gender in facefusion.choices.genders:
target_faces = filter_faces_by_gender(target_faces, face_selector_gender)
if face_selector_race in facefusion.choices.races:
target_faces = filter_faces_by_race(target_faces, face_selector_race)
if state_manager.get_item('face_selector_age_start') or state_manager.get_item('face_selector_age_end'): if state_manager.get_item('face_selector_age_start') or state_manager.get_item('face_selector_age_end'):
faces = filter_faces_by_age(faces, state_manager.get_item('face_selector_age_start'), state_manager.get_item('face_selector_age_end')) target_faces = filter_faces_by_age(target_faces, state_manager.get_item('face_selector_age_start'), state_manager.get_item('face_selector_age_end'))
return faces
return target_faces
def sort_faces_by_order(faces : List[Face], order : FaceSelectorOrder) -> List[Face]: def sort_faces_by_order(faces : List[Face], order : FaceSelectorOrder) -> List[Face]:
if order == 'left-right': if order == 'left-right':
return sorted(faces, key = lambda face: face.bounding_box[0]) return sorted(faces, key = get_bounding_box_left)
if order == 'right-left': if order == 'right-left':
return sorted(faces, key = lambda face: face.bounding_box[0], reverse = True) return sorted(faces, key = get_bounding_box_left, reverse = True)
if order == 'top-bottom': if order == 'top-bottom':
return sorted(faces, key = lambda face: face.bounding_box[1]) return sorted(faces, key = get_bounding_box_top)
if order == 'bottom-top': if order == 'bottom-top':
return sorted(faces, key = lambda face: face.bounding_box[1], reverse = True) return sorted(faces, key = get_bounding_box_top, reverse = True)
if order == 'small-large': if order == 'small-large':
return sorted(faces, key = lambda face: (face.bounding_box[2] - face.bounding_box[0]) * (face.bounding_box[3] - face.bounding_box[1])) return sorted(faces, key = get_bounding_box_area)
if order == 'large-small': if order == 'large-small':
return sorted(faces, key = lambda face: (face.bounding_box[2] - face.bounding_box[0]) * (face.bounding_box[3] - face.bounding_box[1]), reverse = True) return sorted(faces, key = get_bounding_box_area, reverse = True)
if order == 'best-worst': if order == 'best-worst':
return sorted(faces, key = lambda face: face.score_set.get('detector'), reverse = True) return sorted(faces, key = get_face_detector_score, reverse = True)
if order == 'worst-best': if order == 'worst-best':
return sorted(faces, key = lambda face: face.score_set.get('detector')) return sorted(faces, key = get_face_detector_score)
return faces return faces
def get_bounding_box_left(face : Face) -> float:
return face.bounding_box[0]
def get_bounding_box_top(face : Face) -> float:
return face.bounding_box[1]
def get_bounding_box_area(face : Face) -> float:
return (face.bounding_box[2] - face.bounding_box[0]) * (face.bounding_box[3] - face.bounding_box[1])
def get_face_detector_score(face : Face) -> Score:
return face.score_set.get('detector')
def filter_faces_by_gender(faces : List[Face], gender : Gender) -> List[Face]: def filter_faces_by_gender(faces : List[Face], gender : Gender) -> List[Face]:
filter_faces = [] filter_faces = []
+26 -37
View File
@@ -1,53 +1,42 @@
import hashlib import threading
from typing import List, Optional from typing import List, Optional
import numpy import numpy
from facefusion.typing import Face, FaceSet, FaceStore, VisionFrame from facefusion.hash_helper import create_hash
from facefusion.types import Face, FaceStore, VisionFrame
FACE_STORE : FaceStore =\ FACE_STORE : FaceStore = {}
{
'static_faces': {},
'reference_faces': {}
}
def get_face_store() -> FaceStore: def get_faces(vision_frame : VisionFrame) -> Optional[List[Face]]:
return FACE_STORE if numpy.any(vision_frame):
vision_hash = create_hash(vision_frame.tobytes())
if FACE_STORE.get(vision_hash):
return FACE_STORE.get(vision_hash).get('faces')
def get_static_faces(vision_frame : VisionFrame) -> Optional[List[Face]]:
frame_hash = create_frame_hash(vision_frame)
if frame_hash in FACE_STORE['static_faces']:
return FACE_STORE['static_faces'][frame_hash]
return None return None
def set_static_faces(vision_frame : VisionFrame, faces : List[Face]) -> None: def set_faces(vision_frame : VisionFrame, faces : List[Face]) -> None:
frame_hash = create_frame_hash(vision_frame) if numpy.any(vision_frame):
if frame_hash: vision_hash = create_hash(vision_frame.tobytes())
FACE_STORE['static_faces'][frame_hash] = faces FACE_STORE.setdefault(vision_hash,
{
'lock': threading.Lock()
})['faces'] = faces
def clear_static_faces() -> None: def resolve_lock(vision_frame : VisionFrame) -> threading.Lock:
FACE_STORE['static_faces'] = {} if numpy.any(vision_frame):
vision_hash = create_hash(vision_frame.tobytes())
return FACE_STORE.setdefault(vision_hash,
{
'lock': threading.Lock()
}).get('lock')
return threading.Lock()
def create_frame_hash(vision_frame : VisionFrame) -> Optional[str]: def clear_faces() -> None:
return hashlib.sha1(vision_frame.tobytes()).hexdigest() if numpy.any(vision_frame) else None FACE_STORE.clear()
def get_reference_faces() -> Optional[FaceSet]:
if FACE_STORE['reference_faces']:
return FACE_STORE['reference_faces']
return None
def append_reference_face(name : str, face : Face) -> None:
if name not in FACE_STORE['reference_faces']:
FACE_STORE['reference_faces'][name] = []
FACE_STORE['reference_faces'][name].append(face)
def clear_reference_faces() -> None:
FACE_STORE['reference_faces'] = {}
+61
View File
@@ -0,0 +1,61 @@
from typing import List
from facefusion.common_helper import get_first, get_last
from facefusion.face_creator import get_static_faces, refill_faces
from facefusion.face_helper import calculate_bounding_box_overlap
from facefusion.types import Face, FaceTrack, Score, VisionFrame
def track_faces(vision_frames : List[VisionFrame], score : Score) -> List[Face]:
target_index = len(vision_frames) // 2
face_tracks = create_face_tracks(vision_frames, score)
temp_faces = []
for face_track in face_tracks:
track_indices = sorted(face_track)
track_index_first = get_first(track_indices)
track_index_last = get_last(track_indices)
track_range = range(track_index_first, track_index_last + 1)
if target_index in track_range:
fill_faces = []
for index in track_range:
fill_faces.append(face_track.get(index))
temp_faces.append(refill_faces(fill_faces)[target_index - track_index_first])
return temp_faces
def create_face_tracks(vision_frames : List[VisionFrame], score : Score) -> List[FaceTrack]:
face_tracks : List[FaceTrack] = []
for frame_index, vision_frame in enumerate(vision_frames):
for face in get_static_faces([ vision_frame ]):
face_track = select_face_track(face_tracks, face, score)
if face_track:
face_track[frame_index] = face
else:
face_tracks.append(
{
frame_index : face
})
return face_tracks
def select_face_track(face_tracks : List[FaceTrack], face : Face, score : Score) -> FaceTrack:
select_track : FaceTrack = {}
select_score = score
for face_track in face_tracks:
track_face = face_track.get(get_last(face_track))
track_score = calculate_bounding_box_overlap(face.bounding_box, track_face.bounding_box)
if track_score > select_score:
select_score = track_score
select_track = face_track
return select_track
+301 -141
View File
@@ -1,29 +1,32 @@
import os import os
import shutil
import subprocess import subprocess
import tempfile import tempfile
from typing import List, Optional from functools import lru_cache, partial
from typing import List, Optional, cast
import filetype
from tqdm import tqdm from tqdm import tqdm
from facefusion import logger, process_manager, state_manager, wording import facefusion.choices
from facefusion.filesystem import remove_file from facefusion import ffmpeg_builder, logger, process_manager, state_manager, translator
from facefusion.temp_helper import get_temp_file_path, get_temp_frame_paths, get_temp_frames_pattern from facefusion.filesystem import get_file_format, remove_file
from facefusion.typing import AudioBuffer, Fps, OutputVideoPreset, UpdateProgress from facefusion.temp_helper import get_temp_file_path, get_temp_frames_pattern
from facefusion.vision import count_trim_frame_total, detect_video_duration, restrict_video_fps from facefusion.types import ApiSecurityStrategy, AudioEncoder, Buffer, Command, EncoderSet, Fps, Resolution, SampleRate, UpdateProgress, VideoEncoder, VideoFormat
from facefusion.vision import detect_video_duration, detect_video_fps, pack_resolution, predict_video_frame_total
def run_ffmpeg_with_progress(args: List[str], update_progress : UpdateProgress) -> subprocess.Popen[bytes]: def run_ffmpeg_with_progress(commands : List[Command], update_progress : UpdateProgress) -> subprocess.Popen[Buffer]:
log_level = state_manager.get_item('log_level') log_level = state_manager.get_item('log_level')
commands = [ shutil.which('ffmpeg'), '-hide_banner', '-nostats', '-loglevel', 'error', '-progress', '-' ] commands.extend(ffmpeg_builder.set_progress())
commands.extend(args) commands.extend(ffmpeg_builder.cast_stream())
commands = ffmpeg_builder.run(commands)
process = subprocess.Popen(commands, stderr = subprocess.PIPE, stdout = subprocess.PIPE) process = subprocess.Popen(commands, stderr = subprocess.PIPE, stdout = subprocess.PIPE)
while process_manager.is_processing(): while process_manager.is_processing():
try: try:
while __line__ := process.stdout.readline().decode().lower(): while __line__ := process.stdout.readline().decode().lower():
if process_manager.is_stopping():
process.terminate()
if 'frame=' in __line__: if 'frame=' in __line__:
_, frame_number = __line__.split('frame=') _, frame_number = __line__.split('frame=')
update_progress(int(frame_number)) update_progress(int(frame_number))
@@ -35,15 +38,23 @@ def run_ffmpeg_with_progress(args: List[str], update_progress : UpdateProgress)
continue continue
return process return process
if process_manager.is_stopping():
process.terminate()
return process return process
def run_ffmpeg(args : List[str]) -> subprocess.Popen[bytes]: def update_progress(progress : tqdm, frame_number : int) -> None:
progress.update(frame_number - progress.n)
def run_ffmpeg_with_pipe(commands : List[Command], file_content : Buffer) -> subprocess.Popen[Buffer]:
commands = ffmpeg_builder.run(commands)
process = subprocess.Popen(commands, stdin = subprocess.PIPE, stderr = subprocess.PIPE, stdout = subprocess.PIPE)
process.communicate(input = file_content)
return process
def run_ffmpeg(commands : List[Command]) -> subprocess.Popen[Buffer]:
log_level = state_manager.get_item('log_level') log_level = state_manager.get_item('log_level')
commands = [ shutil.which('ffmpeg'), '-hide_banner', '-nostats', '-loglevel', 'error' ] commands = ffmpeg_builder.run(commands)
commands.extend(args)
process = subprocess.Popen(commands, stderr = subprocess.PIPE, stdout = subprocess.PIPE) process = subprocess.Popen(commands, stderr = subprocess.PIPE, stdout = subprocess.PIPE)
while process_manager.is_processing(): while process_manager.is_processing():
@@ -57,174 +68,323 @@ def run_ffmpeg(args : List[str]) -> subprocess.Popen[bytes]:
if process_manager.is_stopping(): if process_manager.is_stopping():
process.terminate() process.terminate()
return process return process
def open_ffmpeg(args : List[str]) -> subprocess.Popen[bytes]: def open_ffmpeg(commands : List[Command]) -> subprocess.Popen[Buffer]:
commands = [ shutil.which('ffmpeg'), '-loglevel', 'quiet' ] commands = ffmpeg_builder.run(commands)
commands.extend(args)
return subprocess.Popen(commands, stdin = subprocess.PIPE, stdout = subprocess.PIPE) return subprocess.Popen(commands, stdin = subprocess.PIPE, stdout = subprocess.PIPE)
def log_debug(process : subprocess.Popen[bytes]) -> None: def log_debug(process : subprocess.Popen[Buffer]) -> None:
_, stderr = process.communicate() _, stderr = process.communicate()
errors = stderr.decode().split(os.linesep) errors = stderr.decode().splitlines()
for error in errors: for error in errors:
if error.strip(): if error.strip():
logger.debug(error.strip(), __name__) logger.debug(error.strip(), __name__)
def extract_frames(target_path : str, temp_video_resolution : str, temp_video_fps : Fps, trim_frame_start : int, trim_frame_end : int) -> bool: def get_available_encoder_set() -> EncoderSet:
extract_frame_total = count_trim_frame_total(target_path, trim_frame_start, trim_frame_end) available_encoder_set : EncoderSet =\
temp_frames_pattern = get_temp_frames_pattern(target_path, '%08d') {
commands = [ '-i', target_path, '-s', str(temp_video_resolution), '-q:v', '0' ] 'audio': [],
'image': [],
'video': []
}
commands = ffmpeg_builder.chain(
ffmpeg_builder.get_encoders()
)
process = run_ffmpeg(commands)
if isinstance(trim_frame_start, int) and isinstance(trim_frame_end, int): while line := process.stdout.readline().decode().lower():
commands.extend([ '-vf', 'trim=start_frame=' + str(trim_frame_start) + ':end_frame=' + str(trim_frame_end) + ',fps=' + str(temp_video_fps) ]) if line.startswith(' a'):
elif isinstance(trim_frame_start, int): audio_encoder = line.split()[1]
commands.extend([ '-vf', 'trim=start_frame=' + str(trim_frame_start) + ',fps=' + str(temp_video_fps) ])
elif isinstance(trim_frame_end, int):
commands.extend([ '-vf', 'trim=end_frame=' + str(trim_frame_end) + ',fps=' + str(temp_video_fps) ])
else:
commands.extend([ '-vf', 'fps=' + str(temp_video_fps) ])
commands.extend([ '-vsync', '0', temp_frames_pattern ])
with tqdm(total = extract_frame_total, desc = wording.get('extracting'), unit = 'frame', ascii = ' =', disable = state_manager.get_item('log_level') in [ 'warn', 'error' ]) as progress: if audio_encoder in facefusion.choices.audio_encoders and audio_encoder not in available_encoder_set.get('audio'):
process = run_ffmpeg_with_progress(commands, lambda frame_number: progress.update(frame_number - progress.n)) available_encoder_set['audio'].append(audio_encoder) #type:ignore[arg-type]
if line.startswith(' v'):
vision_encoder = line.split()[1]
if vision_encoder in facefusion.choices.image_encoders and vision_encoder not in available_encoder_set.get('image'):
available_encoder_set['image'].append(vision_encoder) #type:ignore[arg-type]
if vision_encoder in facefusion.choices.video_encoders and vision_encoder not in available_encoder_set.get('video'):
available_encoder_set['video'].append(vision_encoder) #type:ignore[arg-type]
return available_encoder_set
@lru_cache(maxsize = None)
def get_static_available_encoder_set() -> EncoderSet:
return get_available_encoder_set()
def extract_frames(target_path : str, output_path : str, temp_video_resolution : Resolution, temp_video_fps : Fps, trim_frame_start : int, trim_frame_end : int) -> bool:
extract_frame_total = predict_video_frame_total(target_path, temp_video_fps, trim_frame_start, trim_frame_end)
temp_frames_pattern = get_temp_frames_pattern(state_manager.get_temp_path(), output_path, state_manager.get_item('temp_frame_format'), '%08d')
commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input(target_path),
ffmpeg_builder.set_media_resolution(pack_resolution(temp_video_resolution)),
ffmpeg_builder.set_frame_quality(0),
ffmpeg_builder.enforce_pixel_format('rgb24'),
ffmpeg_builder.select_frame_range(trim_frame_start, trim_frame_end, temp_video_fps),
ffmpeg_builder.prevent_frame_drop(),
ffmpeg_builder.set_start_number(trim_frame_start),
ffmpeg_builder.set_output(temp_frames_pattern)
)
with tqdm(total = extract_frame_total, desc = translator.get('extracting'), unit = 'frame', ascii = ' =', disable = state_manager.get_item('log_level') in [ 'warn', 'error' ]) as progress:
process = run_ffmpeg_with_progress(commands, partial(update_progress, progress))
return process.returncode == 0 return process.returncode == 0
def merge_video(target_path : str, output_video_resolution : str, output_video_fps: Fps) -> bool: def spawn_frames(target_path : str, output_path : str, temp_video_resolution : Resolution, temp_video_fps : Fps, trim_frame_start : int, trim_frame_end : int) -> bool:
spawn_frame_total = trim_frame_end - trim_frame_start
duration = spawn_frame_total / temp_video_fps
temp_frames_pattern = get_temp_frames_pattern(state_manager.get_temp_path(), output_path, state_manager.get_item('temp_frame_format'), '%08d')
commands = ffmpeg_builder.chain(
ffmpeg_builder.set_loop(),
ffmpeg_builder.set_input(target_path),
ffmpeg_builder.set_video_duration(duration),
ffmpeg_builder.set_video_fps(temp_video_fps),
ffmpeg_builder.set_media_resolution(pack_resolution(temp_video_resolution)),
ffmpeg_builder.set_output(temp_frames_pattern)
)
with tqdm(total = spawn_frame_total, desc = translator.get('spawning'), unit = 'frame', ascii = ' =', disable = state_manager.get_item('log_level') in [ 'warn', 'error' ]) as progress:
process = run_ffmpeg_with_progress(commands, partial(update_progress, progress))
return process.returncode == 0
def copy_image(target_path : str, output_path : str, temp_image_resolution : Resolution) -> bool:
temp_image_path = get_temp_file_path(state_manager.get_temp_path(), output_path)
commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input(target_path),
ffmpeg_builder.set_media_resolution(pack_resolution(temp_image_resolution)),
ffmpeg_builder.set_image_quality(target_path, 100),
ffmpeg_builder.force_output(temp_image_path)
)
return run_ffmpeg(commands).returncode == 0
def finalize_image(output_path : str, output_image_resolution : Resolution) -> bool:
output_image_quality = state_manager.get_item('output_image_quality')
temp_image_path = get_temp_file_path(state_manager.get_temp_path(), output_path)
commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input(temp_image_path),
ffmpeg_builder.set_media_resolution(pack_resolution(output_image_resolution)),
ffmpeg_builder.set_image_quality(output_path, output_image_quality),
ffmpeg_builder.force_output(output_path)
)
return run_ffmpeg(commands).returncode == 0
def read_audio_buffer(target_path : str, audio_sample_rate : SampleRate, audio_sample_size : int, audio_channel_total : int) -> Optional[Buffer]:
commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input(target_path),
ffmpeg_builder.ignore_video_stream(),
ffmpeg_builder.set_audio_sample_rate(audio_sample_rate),
ffmpeg_builder.set_audio_sample_size(audio_sample_size),
ffmpeg_builder.set_audio_channel_total(audio_channel_total),
ffmpeg_builder.cast_stream()
)
process = open_ffmpeg(commands)
audio_buffer, _ = process.communicate()
if process.returncode == 0:
return audio_buffer
return None
def restore_audio(target_path : str, output_path : str, trim_frame_start : int, trim_frame_end : int) -> bool:
output_audio_encoder = state_manager.get_item('output_audio_encoder')
output_audio_quality = state_manager.get_item('output_audio_quality')
output_audio_volume = state_manager.get_item('output_audio_volume')
target_video_fps = detect_video_fps(target_path)
temp_video_path = get_temp_file_path(state_manager.get_temp_path(), output_path)
temp_video_format = cast(VideoFormat, get_file_format(output_path))
temp_video_duration = detect_video_duration(temp_video_path)
output_video_format = cast(VideoFormat, get_file_format(output_path))
output_audio_encoder = fix_audio_encoder(temp_video_format, output_audio_encoder)
commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input(temp_video_path),
ffmpeg_builder.select_media_range(trim_frame_start, trim_frame_end, target_video_fps),
ffmpeg_builder.set_input(target_path),
ffmpeg_builder.copy_video_encoder(),
ffmpeg_builder.set_audio_encoder(output_audio_encoder),
ffmpeg_builder.set_audio_quality(output_audio_encoder, output_audio_quality),
ffmpeg_builder.set_audio_volume(output_audio_volume),
ffmpeg_builder.select_media_stream('0:v:0'),
ffmpeg_builder.select_media_stream('1:a:0'),
ffmpeg_builder.set_video_duration(temp_video_duration),
ffmpeg_builder.set_faststart(output_video_format),
ffmpeg_builder.force_output(output_path)
)
return run_ffmpeg(commands).returncode == 0
def replace_audio(audio_path : str, output_path : str) -> bool:
output_audio_encoder = state_manager.get_item('output_audio_encoder')
output_audio_quality = state_manager.get_item('output_audio_quality')
output_audio_volume = state_manager.get_item('output_audio_volume')
temp_video_path = get_temp_file_path(state_manager.get_temp_path(), output_path)
temp_video_format = cast(VideoFormat, get_file_format(output_path))
temp_video_duration = detect_video_duration(temp_video_path)
output_video_format = cast(VideoFormat, get_file_format(output_path))
output_audio_encoder = fix_audio_encoder(temp_video_format, output_audio_encoder)
commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input(temp_video_path),
ffmpeg_builder.set_input(audio_path),
ffmpeg_builder.copy_video_encoder(),
ffmpeg_builder.set_audio_encoder(output_audio_encoder),
ffmpeg_builder.set_audio_quality(output_audio_encoder, output_audio_quality),
ffmpeg_builder.set_audio_volume(output_audio_volume),
ffmpeg_builder.set_video_duration(temp_video_duration),
ffmpeg_builder.set_faststart(output_video_format),
ffmpeg_builder.force_output(output_path)
)
return run_ffmpeg(commands).returncode == 0
def merge_video(target_path : str, output_path : str, temp_video_fps : Fps, output_video_fps : Fps, output_video_resolution : Resolution, trim_frame_start : int, trim_frame_end : int) -> bool:
output_video_encoder = state_manager.get_item('output_video_encoder') output_video_encoder = state_manager.get_item('output_video_encoder')
output_video_quality = state_manager.get_item('output_video_quality') output_video_quality = state_manager.get_item('output_video_quality')
output_video_preset = state_manager.get_item('output_video_preset') output_video_preset = state_manager.get_item('output_video_preset')
merge_frame_total = len(get_temp_frame_paths(target_path)) merge_frame_total = predict_video_frame_total(target_path, output_video_fps, trim_frame_start, trim_frame_end)
temp_video_fps = restrict_video_fps(target_path, output_video_fps) temp_video_path = get_temp_file_path(state_manager.get_temp_path(), output_path)
temp_file_path = get_temp_file_path(target_path) temp_video_format = cast(VideoFormat, get_file_format(output_path))
temp_frames_pattern = get_temp_frames_pattern(target_path, '%08d') temp_frames_pattern = get_temp_frames_pattern(state_manager.get_temp_path(), output_path, state_manager.get_item('temp_frame_format'), '%08d')
is_webm = filetype.guess_mime(target_path) == 'video/webm'
if is_webm: output_video_encoder = fix_video_encoder(temp_video_format, output_video_encoder)
output_video_encoder = 'libvpx-vp9' commands = ffmpeg_builder.chain(
commands = [ '-r', str(temp_video_fps), '-i', temp_frames_pattern, '-s', str(output_video_resolution), '-c:v', output_video_encoder ] ffmpeg_builder.set_input_fps(temp_video_fps),
if output_video_encoder in [ 'libx264', 'libx265' ]: ffmpeg_builder.set_start_number(trim_frame_start),
output_video_compression = round(51 - (output_video_quality * 0.51)) ffmpeg_builder.set_input(temp_frames_pattern),
commands.extend([ '-crf', str(output_video_compression), '-preset', output_video_preset ]) ffmpeg_builder.set_media_resolution(pack_resolution(output_video_resolution)),
if output_video_encoder in [ 'libvpx-vp9' ]: ffmpeg_builder.set_video_encoder(output_video_encoder),
output_video_compression = round(63 - (output_video_quality * 0.63)) ffmpeg_builder.set_video_tag(output_video_encoder, temp_video_format),
commands.extend([ '-crf', str(output_video_compression) ]) ffmpeg_builder.set_video_quality(output_video_encoder, output_video_quality),
if output_video_encoder in [ 'h264_nvenc', 'hevc_nvenc' ]: ffmpeg_builder.set_video_preset(output_video_encoder, output_video_preset),
output_video_compression = round(51 - (output_video_quality * 0.51)) ffmpeg_builder.concat(
commands.extend([ '-cq', str(output_video_compression), '-preset', map_nvenc_preset(output_video_preset) ]) ffmpeg_builder.set_video_fps(output_video_fps),
if output_video_encoder in [ 'h264_amf', 'hevc_amf' ]: ffmpeg_builder.keep_video_alpha(output_video_encoder)
output_video_compression = round(51 - (output_video_quality * 0.51)) ),
commands.extend([ '-qp_i', str(output_video_compression), '-qp_p', str(output_video_compression), '-quality', map_amf_preset(output_video_preset) ]) ffmpeg_builder.set_pixel_format(output_video_encoder),
if output_video_encoder in [ 'h264_videotoolbox', 'hevc_videotoolbox' ]: ffmpeg_builder.force_output(temp_video_path)
commands.extend([ '-q:v', str(output_video_quality) ]) )
commands.extend([ '-vf', 'framerate=fps=' + str(output_video_fps), '-pix_fmt', 'yuv420p', '-colorspace', 'bt709', '-y', temp_file_path ])
with tqdm(total = merge_frame_total, desc = wording.get('merging'), unit = 'frame', ascii = ' =', disable = state_manager.get_item('log_level') in [ 'warn', 'error' ]) as progress: with tqdm(total = merge_frame_total, desc = translator.get('merging'), unit = 'frame', ascii = ' =', disable = state_manager.get_item('log_level') in [ 'warn', 'error' ]) as progress:
process = run_ffmpeg_with_progress(commands, lambda frame_number: progress.update(frame_number - progress.n)) process = run_ffmpeg_with_progress(commands, partial(update_progress, progress))
return process.returncode == 0 return process.returncode == 0
def concat_video(output_path : str, temp_output_paths : List[str]) -> bool: def concat_video(output_path : str, temp_output_paths : List[str]) -> bool:
output_audio_encoder = state_manager.get_item('output_audio_encoder') file_descriptor, concat_video_path = tempfile.mkstemp()
concat_video_path = tempfile.mktemp() os.close(file_descriptor)
with open(concat_video_path, 'w') as concat_video_file: with open(concat_video_path, 'w') as concat_video_file:
for temp_output_path in temp_output_paths: for temp_output_path in temp_output_paths:
concat_video_file.write('file \'' + os.path.abspath(temp_output_path) + '\'' + os.linesep) concat_video_file.write('file \'' + os.path.abspath(temp_output_path) + '\'' + os.linesep)
concat_video_file.flush() concat_video_file.flush()
concat_video_file.close() concat_video_file.close()
commands = [ '-f', 'concat', '-safe', '0', '-i', concat_video_file.name, '-c:v', 'copy', '-c:a', output_audio_encoder, '-y', os.path.abspath(output_path) ]
output_path = os.path.abspath(output_path)
output_video_format = cast(VideoFormat, get_file_format(output_path))
commands = ffmpeg_builder.chain(
ffmpeg_builder.unsafe_concat(),
ffmpeg_builder.set_input(concat_video_file.name),
ffmpeg_builder.copy_video_encoder(),
ffmpeg_builder.copy_audio_encoder(),
ffmpeg_builder.set_faststart(output_video_format),
ffmpeg_builder.force_output(output_path)
)
process = run_ffmpeg(commands) process = run_ffmpeg(commands)
process.communicate() process.communicate()
remove_file(concat_video_path) remove_file(concat_video_path)
return process.returncode == 0 return process.returncode == 0
def copy_image(target_path : str, temp_image_resolution : str) -> bool: def sanitize_audio(file_content : Buffer, asset_path : str, security_strategy : ApiSecurityStrategy) -> bool:
temp_file_path = get_temp_file_path(target_path) if security_strategy == 'strict':
temp_image_compression = calc_image_compression(target_path, 100) commands = ffmpeg_builder.chain(
commands = [ '-i', target_path, '-s', str(temp_image_resolution), '-q:v', str(temp_image_compression), '-y', temp_file_path ] ffmpeg_builder.set_input('pipe:0'),
return run_ffmpeg(commands).returncode == 0 ffmpeg_builder.deep_copy_audio(),
ffmpeg_builder.strip_metadata(),
ffmpeg_builder.force_output(asset_path)
)
return run_ffmpeg_with_pipe(commands, file_content).returncode == 0
commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input('pipe:0'),
ffmpeg_builder.copy_audio_encoder(),
ffmpeg_builder.strip_metadata(),
ffmpeg_builder.force_output(asset_path)
)
return run_ffmpeg_with_pipe(commands, file_content).returncode == 0
def finalize_image(target_path : str, output_path : str, output_image_resolution : str) -> bool: def sanitize_image(file_content : Buffer, asset_path : str) -> bool:
output_image_quality = state_manager.get_item('output_image_quality') commands = ffmpeg_builder.chain(
temp_file_path = get_temp_file_path(target_path) ffmpeg_builder.set_input('pipe:0'),
output_image_compression = calc_image_compression(target_path, output_image_quality) ffmpeg_builder.deep_copy_image(),
commands = [ '-i', temp_file_path, '-s', str(output_image_resolution), '-q:v', str(output_image_compression), '-y', output_path ] ffmpeg_builder.strip_metadata(),
return run_ffmpeg(commands).returncode == 0 ffmpeg_builder.force_output(asset_path)
)
return run_ffmpeg_with_pipe(commands, file_content).returncode == 0
def calc_image_compression(image_path : str, image_quality : int) -> int: def sanitize_video(file_content : Buffer, asset_path : str, security_strategy : ApiSecurityStrategy) -> bool:
is_webp = filetype.guess_mime(image_path) == 'image/webp' if security_strategy == 'strict':
if is_webp: available_video_encoders = get_static_available_encoder_set().get('video')
image_quality = 100 - image_quality commands = ffmpeg_builder.chain(
return round(31 - (image_quality * 0.31)) ffmpeg_builder.set_input('pipe:0'),
ffmpeg_builder.set_video_encoder(available_video_encoders[0]),
ffmpeg_builder.set_video_preset(available_video_encoders[0], 'ultrafast'),
ffmpeg_builder.set_pixel_format(available_video_encoders[0]),
ffmpeg_builder.deep_copy_video(),
ffmpeg_builder.deep_copy_audio(),
ffmpeg_builder.strip_metadata(),
ffmpeg_builder.force_output(asset_path)
)
return run_ffmpeg_with_pipe(commands, file_content).returncode == 0
commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input('pipe:0'),
ffmpeg_builder.copy_video_encoder(),
ffmpeg_builder.copy_audio_encoder(),
ffmpeg_builder.strip_metadata(),
ffmpeg_builder.force_output(asset_path)
)
return run_ffmpeg_with_pipe(commands, file_content).returncode == 0
def read_audio_buffer(target_path : str, sample_rate : int, channel_total : int) -> Optional[AudioBuffer]: def fix_audio_encoder(video_format : VideoFormat, audio_encoder : AudioEncoder) -> AudioEncoder:
commands = [ '-i', target_path, '-vn', '-f', 's16le', '-acodec', 'pcm_s16le', '-ar', str(sample_rate), '-ac', str(channel_total), '-' ] if video_format == 'avi' and audio_encoder == 'libopus':
process = open_ffmpeg(commands) return 'aac'
audio_buffer, _ = process.communicate() if video_format in [ 'm4v', 'mpeg', 'wmv' ]:
if process.returncode == 0: return 'aac'
return audio_buffer if video_format == 'mov' and audio_encoder in [ 'flac', 'libopus' ]:
return None return 'aac'
if video_format == 'mxf':
return 'pcm_s16le'
if video_format == 'webm':
return 'libopus'
return audio_encoder
def restore_audio(target_path : str, output_path : str, output_video_fps : Fps, trim_frame_start : int, trim_frame_end : int) -> bool: def fix_video_encoder(video_format : VideoFormat, video_encoder : VideoEncoder) -> VideoEncoder:
output_audio_encoder = state_manager.get_item('output_audio_encoder') if video_format in [ 'm4v', 'mpeg', 'mxf', 'wmv' ]:
temp_file_path = get_temp_file_path(target_path) return 'libx264'
temp_video_duration = detect_video_duration(temp_file_path) if video_format in [ 'mkv', 'mp4' ] and video_encoder == 'rawvideo':
commands = [ '-i', temp_file_path ] return 'libx264'
if video_format == 'mov' and video_encoder == 'libvpx-vp9':
if isinstance(trim_frame_start, int): return 'libx264'
start_time = trim_frame_start / output_video_fps if video_format == 'webm':
commands.extend([ '-ss', str(start_time) ]) return 'libvpx-vp9'
if isinstance(trim_frame_end, int): return video_encoder
end_time = trim_frame_end / output_video_fps
commands.extend([ '-to', str(end_time) ])
commands.extend([ '-i', target_path, '-c:v', 'copy', '-c:a', output_audio_encoder, '-map', '0:v:0', '-map', '1:a:0', '-t', str(temp_video_duration), '-y', output_path ])
return run_ffmpeg(commands).returncode == 0
def replace_audio(target_path : str, audio_path : str, output_path : str) -> bool:
output_audio_encoder = state_manager.get_item('output_audio_encoder')
temp_file_path = get_temp_file_path(target_path)
temp_video_duration = detect_video_duration(temp_file_path)
commands = [ '-i', temp_file_path, '-i', audio_path, '-c:v', 'copy', '-c:a', output_audio_encoder, '-t', str(temp_video_duration), '-y', output_path ]
return run_ffmpeg(commands).returncode == 0
def map_nvenc_preset(output_video_preset : OutputVideoPreset) -> Optional[str]:
if output_video_preset in [ 'ultrafast', 'superfast', 'veryfast', 'faster', 'fast' ]:
return 'fast'
if output_video_preset == 'medium':
return 'medium'
if output_video_preset in [ 'slow', 'slower', 'veryslow' ]:
return 'slow'
return None
def map_amf_preset(output_video_preset : OutputVideoPreset) -> Optional[str]:
if output_video_preset in [ 'ultrafast', 'superfast', 'veryfast' ]:
return 'speed'
if output_video_preset in [ 'faster', 'fast', 'medium' ]:
return 'balanced'
if output_video_preset in [ 'slow', 'slower', 'veryslow' ]:
return 'quality'
return None
def map_qsv_preset(output_video_preset : OutputVideoPreset) -> Optional[str]:
if output_video_preset in [ 'ultrafast', 'superfast', 'veryfast', 'faster', 'fast' ]:
return 'fast'
if output_video_preset == 'medium':
return 'medium'
if output_video_preset in [ 'slow', 'slower', 'veryslow' ]:
return 'slow'
return None
+309
View File
@@ -0,0 +1,309 @@
import itertools
import shutil
from typing import List, Optional
import numpy
from facefusion.filesystem import get_file_format
from facefusion.types import AudioEncoder, Command, CommandSet, Duration, Fps, SampleRate, StreamMode, VideoEncoder, VideoFormat, VideoPreset
def run(commands : List[Command]) -> List[Command]:
return [ shutil.which('ffmpeg'), '-loglevel', 'error' ] + commands
def chain(*commands : List[Command]) -> List[Command]:
return list(itertools.chain(*commands))
def concat(*__commands__ : List[Command]) -> List[Command]:
commands = []
command_set : CommandSet = {}
for command in __commands__:
for argument, value in zip(command[::2], command[1::2]):
command_set.setdefault(argument, []).append(value)
for argument, values in command_set.items():
commands.append(argument)
commands.append(','.join(values))
return commands
def get_encoders() -> List[Command]:
return [ '-encoders' ]
def set_hardware_accelerator(value : str) -> List[Command]:
return [ '-hwaccel', value ]
def set_progress() -> List[Command]:
return [ '-progress' ]
def set_input(input_path : str) -> List[Command]:
return [ '-i', input_path ]
def set_input_fps(input_fps : Fps) -> List[Command]:
return [ '-r', str(input_fps) ]
def set_start_number(frame_number : int) -> List[Command]:
return [ '-start_number', str(frame_number) ]
def set_output(output_path : str) -> List[Command]:
return [ output_path ]
def force_output(output_path : str) -> List[Command]:
return [ '-y', output_path ]
def set_loop() -> List[Command]:
return [ '-loop', '1' ]
def cast_stream() -> List[Command]:
return [ '-' ]
def set_stream_mode(stream_mode : StreamMode) -> List[Command]:
if stream_mode == 'udp':
return [ '-f', 'mpegts' ]
if stream_mode == 'v4l2':
return [ '-f', 'v4l2' ]
return []
def set_stream_quality(stream_quality : int) -> List[Command]:
return [ '-b:v', str(stream_quality) + 'k' ]
def unsafe_concat() -> List[Command]:
return [ '-f', 'concat', '-safe', '0' ]
def enforce_pixel_format(pixel_format : str) -> List[Command]:
return [ '-pix_fmt', pixel_format ]
def strip_metadata() -> List[Command]:
return [ '-map_metadata', '-1' ]
def set_pixel_format(video_encoder : VideoEncoder) -> List[Command]:
if video_encoder == 'rawvideo':
return [ '-pix_fmt', 'rgb24' ]
if video_encoder == 'libvpx-vp9':
return [ '-pix_fmt', 'yuva420p' ]
return [ '-pix_fmt', 'yuv420p' ]
def set_frame_quality(frame_quality : int) -> List[Command]:
return [ '-q:v', str(frame_quality) ]
def select_frame_range(frame_start : int, frame_end : int, video_fps : Fps) -> List[Command]:
if isinstance(frame_start, int) and isinstance(frame_end, int):
return [ '-vf', 'trim=start_frame=' + str(frame_start) + ':end_frame=' + str(frame_end) + ',fps=' + str(video_fps) ]
if isinstance(frame_start, int):
return [ '-vf', 'trim=start_frame=' + str(frame_start) + ',fps=' + str(video_fps) ]
if isinstance(frame_end, int):
return [ '-vf', 'trim=end_frame=' + str(frame_end) + ',fps=' + str(video_fps) ]
return [ '-vf', 'fps=' + str(video_fps) ]
def prevent_frame_drop() -> List[Command]:
return [ '-vsync', '0' ]
def select_media_range(frame_start : int, frame_end : int, media_fps : Fps) -> List[Command]:
commands = []
if isinstance(frame_start, int):
commands.extend([ '-ss', str(frame_start / media_fps) ])
if isinstance(frame_end, int):
commands.extend([ '-to', str(frame_end / media_fps) ])
return commands
def select_media_stream(media_stream : str) -> List[Command]:
return [ '-map', media_stream ]
def set_media_resolution(video_resolution : str) -> List[Command]:
return [ '-s', video_resolution ]
def deep_copy_audio() -> List[Command]:
return [ '-q:a', '0' ]
def set_audio_encoder(audio_codec : str) -> List[Command]:
return [ '-c:a', audio_codec ]
def copy_audio_encoder() -> List[Command]:
return set_audio_encoder('copy')
def set_audio_sample_rate(audio_sample_rate : SampleRate) -> List[Command]:
return [ '-ar', str(audio_sample_rate) ]
def set_audio_sample_size(audio_sample_size : int) -> List[Command]:
if audio_sample_size == 16:
return [ '-f', 's16le' ]
if audio_sample_size == 32:
return [ '-f', 's32le' ]
return []
def set_audio_channel_total(audio_channel_total : int) -> List[Command]:
return [ '-ac', str(audio_channel_total) ]
def set_audio_quality(audio_encoder : AudioEncoder, audio_quality : int) -> List[Command]:
if audio_encoder == 'aac':
audio_compression = numpy.round(numpy.interp(audio_quality, [ 0, 100 ], [ 0.1, 2.0 ]), 1).astype(float).item()
return [ '-q:a', str(audio_compression) ]
if audio_encoder == 'libmp3lame':
audio_compression = numpy.round(numpy.interp(audio_quality, [ 0, 100 ], [ 9, 0 ])).astype(int).item()
return [ '-q:a', str(audio_compression) ]
if audio_encoder == 'libopus':
audio_bit_rate = numpy.round(numpy.interp(audio_quality, [ 0, 100 ], [ 64, 256 ])).astype(int).item()
return [ '-b:a', str(audio_bit_rate) + 'k' ]
if audio_encoder == 'libvorbis':
audio_compression = numpy.round(numpy.interp(audio_quality, [ 0, 100 ], [ -1, 10 ]), 1).astype(float).item()
return [ '-q:a', str(audio_compression) ]
return []
def set_audio_volume(audio_volume : int) -> List[Command]:
return [ '-filter:a', 'volume=' + str(audio_volume / 100) ]
def deep_copy_image() -> List[Command]:
return [ '-q:v', '0' ]
def set_image_quality(image_path : str, image_quality : int) -> List[Command]:
if get_file_format(image_path) == 'webp':
return [ '-q:v', str(image_quality) ]
image_compression = round(31 - (image_quality * 0.31))
return [ '-q:v', str(image_compression) ]
def deep_copy_video() -> List[Command]:
return [ '-q:v', '0' ]
def set_video_encoder(video_encoder : str) -> List[Command]:
return [ '-c:v', video_encoder ]
def copy_video_encoder() -> List[Command]:
return set_video_encoder('copy')
def set_faststart(video_format : VideoFormat) -> List[Command]:
if video_format in [ 'm4v', 'mov', 'mp4' ]:
return [ '-movflags', '+faststart' ]
return []
def set_video_tag(video_encoder : VideoEncoder, video_format : VideoFormat) -> List[Command]:
if video_format in [ 'm4v', 'mov', 'mp4' ] and video_encoder in [ 'libx265', 'hevc_nvenc', 'hevc_amf', 'hevc_qsv', 'hevc_videotoolbox' ]:
return [ '-tag:v', 'hvc1' ]
return []
def set_video_quality(video_encoder : VideoEncoder, video_quality : int) -> List[Command]:
if video_encoder in [ 'libx264', 'libx264rgb', 'libx265' ]:
video_compression = numpy.round(numpy.interp(video_quality, [ 0, 100 ], [ 51, 0 ])).astype(int).item()
return [ '-crf', str(video_compression) ]
if video_encoder == 'libvpx-vp9':
video_compression = numpy.round(numpy.interp(video_quality, [ 0, 100 ], [ 63, 0 ])).astype(int).item()
return [ '-crf', str(video_compression) ]
if video_encoder in [ 'h264_nvenc', 'hevc_nvenc' ]:
video_compression = numpy.round(numpy.interp(video_quality, [ 0, 100 ], [ 51, 0 ])).astype(int).item()
return [ '-cq', str(video_compression) ]
if video_encoder in [ 'h264_amf', 'hevc_amf' ]:
video_compression = numpy.round(numpy.interp(video_quality, [ 0, 100 ], [ 51, 0 ])).astype(int).item()
return [ '-qp_i', str(video_compression), '-qp_p', str(video_compression), '-qp_b', str(video_compression) ]
if video_encoder in [ 'h264_qsv', 'hevc_qsv' ]:
video_compression = numpy.round(numpy.interp(video_quality, [ 0, 100 ], [ 51, 0 ])).astype(int).item()
return [ '-qp', str(video_compression) ]
if video_encoder in [ 'h264_videotoolbox', 'hevc_videotoolbox' ]:
video_bit_rate = numpy.round(numpy.interp(video_quality, [ 0, 100 ], [ 1024, 50512 ])).astype(int).item()
return [ '-b:v', str(video_bit_rate) + 'k' ]
return []
def set_video_preset(video_encoder : VideoEncoder, video_preset : VideoPreset) -> List[Command]:
if video_encoder in [ 'libx264', 'libx264rgb', 'libx265' ]:
return [ '-preset', video_preset ]
if video_encoder in [ 'h264_nvenc', 'hevc_nvenc' ]:
return [ '-preset', map_nvenc_preset(video_preset) ]
if video_encoder in [ 'h264_amf', 'hevc_amf' ]:
return [ '-quality', map_amf_preset(video_preset) ]
if video_encoder in [ 'h264_qsv', 'hevc_qsv' ]:
return [ '-preset', map_qsv_preset(video_preset) ]
return []
def set_video_fps(video_fps : Fps) -> List[Command]:
return [ '-vf', 'fps=' + str(video_fps) ]
def set_video_duration(video_duration : Duration) -> List[Command]:
return [ '-t', str(video_duration) ]
def keep_video_alpha(video_encoder : VideoEncoder) -> List[Command]:
if video_encoder == 'libvpx-vp9':
return [ '-vf', 'format=yuva420p' ]
return []
def capture_video() -> List[Command]:
return [ '-f', 'rawvideo', '-pix_fmt', 'rgb24' ]
def ignore_video_stream() -> List[Command]:
return [ '-vn' ]
def map_nvenc_preset(video_preset : VideoPreset) -> Optional[str]:
if video_preset in [ 'ultrafast', 'superfast', 'veryfast', 'faster', 'fast' ]:
return 'fast'
if video_preset == 'medium':
return 'medium'
if video_preset in [ 'slow', 'slower', 'veryslow' ]:
return 'slow'
return None
def map_amf_preset(video_preset : VideoPreset) -> Optional[str]:
if video_preset in [ 'ultrafast', 'superfast', 'veryfast' ]:
return 'speed'
if video_preset in [ 'faster', 'fast', 'medium' ]:
return 'balanced'
if video_preset in [ 'slow', 'slower', 'veryslow' ]:
return 'quality'
return None
def map_qsv_preset(video_preset : VideoPreset) -> Optional[str]:
if video_preset in [ 'ultrafast', 'superfast', 'veryfast' ]:
return 'veryfast'
if video_preset in [ 'faster', 'fast', 'medium', 'slow', 'slower', 'veryslow' ]:
return video_preset
return None
+84
View File
@@ -0,0 +1,84 @@
import subprocess
from typing import Dict, List
from facefusion import ffprobe_builder
from facefusion.types import AudioMetadata, Buffer, Command, Fps, VideoMetadata
def run_ffprobe(commands : List[Command]) -> subprocess.Popen[Buffer]:
commands = ffprobe_builder.run(commands)
return subprocess.Popen(commands, stderr = subprocess.PIPE, stdout = subprocess.PIPE)
def probe_entries(media_path : str, entries : List[str]) -> Dict[str, str]:
media_entries = {}
commands = ffprobe_builder.chain(
ffprobe_builder.show_entries(entries),
ffprobe_builder.format_to_key_value(),
ffprobe_builder.set_input(media_path)
)
output, _ = run_ffprobe(commands).communicate()
if output:
lines = output.decode().strip().splitlines()
for line in lines:
if '=' in line:
key, value = line.split('=', 1)
media_entries[key] = value
return media_entries
def extract_audio_metadata(audio_path : str) -> AudioMetadata:
audio_entries = probe_entries(audio_path, [ 'duration', 'sample_rate', 'channels', 'bit_rate' ])
duration = float(audio_entries.get('duration'))
sample_rate = int(audio_entries.get('sample_rate'))
frame_total = int(duration * sample_rate)
channel_total = int(audio_entries.get('channels'))
bit_rate = int(audio_entries.get('bit_rate'))
audio_metadata : AudioMetadata =\
{
'duration' : duration,
'frame_total' : frame_total,
'channel_total' : channel_total,
'sample_rate' : sample_rate,
'bit_rate' : bit_rate
}
return audio_metadata
def extract_video_metadata(video_path : str) -> VideoMetadata:
video_entries = probe_entries(video_path, [ 'duration', 'width', 'height', 'r_frame_rate', 'bit_rate' ])
duration = float(video_entries.get('duration'))
fps = extract_video_fps(video_entries.get('r_frame_rate'))
frame_total = int(duration * fps)
width = int(video_entries.get('width'))
height = int(video_entries.get('height'))
bit_rate = int(video_entries.get('bit_rate'))
video_metadata : VideoMetadata =\
{
'duration' : duration,
'frame_total' : frame_total,
'fps' : fps,
'resolution' : (width, height),
'bit_rate' : bit_rate
}
return video_metadata
def extract_video_fps(frame_rate : str) -> Fps:
if frame_rate and '/' in frame_rate:
numerator, denominator = frame_rate.split('/')
if int(numerator) and int(denominator):
return int(numerator) / int(denominator)
return 0.0
+29
View File
@@ -0,0 +1,29 @@
import itertools
import shutil
from typing import List
from facefusion.types import Command
def run(commands : List[Command]) -> List[Command]:
return [ shutil.which('ffprobe'), '-loglevel', 'error' ] + commands
def chain(*commands : List[Command]) -> List[Command]:
return list(itertools.chain(*commands))
def show_entries(entries : List[str]) -> List[Command]:
return [ '-show_entries', 'stream=' + ','.join(entries) ]
def format_to_value() -> List[Command]:
return [ '-of', 'default=noprint_wrappers=1:nokey=1' ]
def format_to_key_value() -> List[Command]:
return [ '-of', 'default=noprint_wrappers=1' ]
def set_input(input_path : str) -> List[Command]:
return [ '-i', input_path ]
+98 -70
View File
@@ -1,16 +1,9 @@
import glob import glob
import os import os
import shutil import shutil
from pathlib import Path
from typing import List, Optional from typing import List, Optional
import filetype import facefusion.choices
from facefusion.common_helper import is_windows
from facefusion.typing import File
if is_windows():
import ctypes
def get_file_size(file_path : str) -> int: def get_file_size(file_path : str) -> int:
@@ -19,54 +12,88 @@ def get_file_size(file_path : str) -> int:
return 0 return 0
def same_file_extension(file_paths : List[str]) -> bool: def get_file_name(file_path : str) -> Optional[str]:
file_extensions : List[str] = [] file_name, _ = os.path.splitext(os.path.basename(file_path))
for file_path in file_paths: if file_name:
_, file_extension = os.path.splitext(file_path.lower()) return file_name
return None
if file_extensions and file_extension not in file_extensions:
return False def get_file_extension(file_path : str) -> Optional[str]:
file_extensions.append(file_extension) _, file_extension = os.path.splitext(file_path)
return True
if file_extension:
return file_extension.lower()
return None
def get_file_format(file_path : str) -> Optional[str]:
file_extension = get_file_extension(file_path)
if file_extension:
if file_extension == '.jpg':
return 'jpeg'
if file_extension == '.tif':
return 'tiff'
if file_extension == '.mpg':
return 'mpeg'
return file_extension.lstrip('.')
return None
def is_file(file_path : str) -> bool: def is_file(file_path : str) -> bool:
return bool(file_path and os.path.isfile(file_path)) if file_path:
return os.path.isfile(file_path)
def is_directory(directory_path : str) -> bool:
return bool(directory_path and os.path.isdir(directory_path))
def in_directory(file_path : str) -> bool:
if file_path and not is_directory(file_path):
return is_directory(os.path.dirname(file_path))
return False return False
def is_audio(audio_path : str) -> bool: def is_audio(audio_path : str) -> bool:
return is_file(audio_path) and filetype.helpers.is_audio(audio_path) return is_file(audio_path) and get_file_format(audio_path) in facefusion.choices.audio_formats
def has_audio(audio_paths : List[str]) -> bool: def has_audio(audio_paths : List[str]) -> bool:
if audio_paths: if audio_paths:
return any(is_audio(audio_path) for audio_path in audio_paths) return any(map(is_audio, audio_paths))
return False
def are_audios(audio_paths : List[str]) -> bool:
if audio_paths:
return all(map(is_audio, audio_paths))
return False return False
def is_image(image_path : str) -> bool: def is_image(image_path : str) -> bool:
return is_file(image_path) and filetype.helpers.is_image(image_path) return is_file(image_path) and get_file_format(image_path) in facefusion.choices.image_formats
def has_image(image_paths: List[str]) -> bool: def has_image(image_paths : List[str]) -> bool:
if image_paths: if image_paths:
return any(is_image(image_path) for image_path in image_paths) return any(is_image(image_path) for image_path in image_paths)
return False return False
def are_images(image_paths : List[str]) -> bool:
if image_paths:
return all(map(is_image, image_paths))
return False
def is_video(video_path : str) -> bool: def is_video(video_path : str) -> bool:
return is_file(video_path) and filetype.helpers.is_video(video_path) return is_file(video_path) and get_file_format(video_path) in facefusion.choices.video_formats
def has_video(video_paths : List[str]) -> bool:
if video_paths:
return any(map(is_video, video_paths))
return False
def are_videos(video_paths : List[str]) -> bool:
if video_paths:
return all(map(is_video, video_paths))
return False
def filter_audio_paths(paths : List[str]) -> List[str]: def filter_audio_paths(paths : List[str]) -> List[str]:
@@ -81,24 +108,6 @@ def filter_image_paths(paths : List[str]) -> List[str]:
return [] return []
def resolve_relative_path(path : str) -> str:
return os.path.abspath(os.path.join(os.path.dirname(__file__), path))
def sanitize_path_for_windows(full_path : str) -> Optional[str]:
buffer_size = 0
while True:
unicode_buffer = ctypes.create_unicode_buffer(buffer_size)
buffer_limit = ctypes.windll.kernel32.GetShortPathNameW(full_path, unicode_buffer, buffer_size) #type:ignore[attr-defined]
if buffer_size > buffer_limit:
return unicode_buffer.value
if buffer_limit == 0:
return None
buffer_size = buffer_limit
def copy_file(file_path : str, move_path : str) -> bool: def copy_file(file_path : str, move_path : str) -> bool:
if is_file(file_path): if is_file(file_path):
shutil.copy(file_path, move_path) shutil.copy(file_path, move_path)
@@ -120,31 +129,18 @@ def remove_file(file_path : str) -> bool:
return False return False
def create_directory(directory_path : str) -> bool: def resolve_file_paths(directory_path : str) -> List[str]:
if directory_path and not is_file(directory_path): file_paths : List[str] = []
Path(directory_path).mkdir(parents = True, exist_ok = True)
return is_directory(directory_path)
return False
def list_directory(directory_path : str) -> Optional[List[File]]:
if is_directory(directory_path): if is_directory(directory_path):
file_paths = sorted(os.listdir(directory_path)) file_names_and_extensions = sorted(os.listdir(directory_path))
files: List[File] = []
for file_path in file_paths: for file_name_and_extension in file_names_and_extensions:
file_name, file_extension = os.path.splitext(file_path) if not file_name_and_extension.startswith(('.', '__')):
file_path = os.path.join(directory_path, file_name_and_extension)
file_paths.append(file_path)
if not file_name.startswith(('.', '__')): return file_paths
files.append(
{
'name': file_name,
'extension': file_extension,
'path': os.path.join(directory_path, file_path)
})
return files
return None
def resolve_file_pattern(file_pattern : str) -> List[str]: def resolve_file_pattern(file_pattern : str) -> List[str]:
@@ -153,8 +149,40 @@ def resolve_file_pattern(file_pattern : str) -> List[str]:
return [] return []
def is_directory(directory_path : str) -> bool:
if directory_path:
return os.path.isdir(directory_path)
return False
def in_directory(file_path : str) -> bool:
if file_path:
directory_path = os.path.dirname(file_path)
if directory_path:
return not is_directory(file_path) and is_directory(directory_path)
return False
def create_directory(directory_path : str) -> bool:
if directory_path and not is_file(directory_path):
os.makedirs(directory_path, exist_ok = True)
return is_directory(directory_path)
return False
def move_directory(directory_path : str, move_path : str) -> bool:
if is_directory(directory_path):
shutil.move(directory_path, move_path)
return is_directory(move_path)
return False
def remove_directory(directory_path : str) -> bool: def remove_directory(directory_path : str) -> bool:
if is_directory(directory_path): if is_directory(directory_path):
shutil.rmtree(directory_path, ignore_errors = True) shutil.rmtree(directory_path, ignore_errors = True)
return not is_directory(directory_path) return not is_directory(directory_path)
return False return False
def resolve_relative_path(path : str) -> str:
return os.path.abspath(os.path.join(os.path.dirname(__file__), path))
+8 -7
View File
@@ -2,19 +2,20 @@ import os
import zlib import zlib
from typing import Optional from typing import Optional
from facefusion.filesystem import is_file from facefusion.filesystem import get_file_name, is_file
from facefusion.types import Buffer
def create_hash(content : bytes) -> str: def create_hash(buffer : Buffer) -> str:
return format(zlib.crc32(content), '08x') return format(zlib.crc32(buffer), '08x')
def validate_hash(validate_path : str) -> bool: def validate_hash(validate_path : str) -> bool:
hash_path = get_hash_path(validate_path) hash_path = get_hash_path(validate_path)
if is_file(hash_path): if is_file(hash_path):
with open(hash_path, 'r') as hash_file: with open(hash_path) as hash_file:
hash_content = hash_file.read().strip() hash_content = hash_file.read()
with open(validate_path, 'rb') as validate_file: with open(validate_path, 'rb') as validate_file:
validate_content = validate_file.read() validate_content = validate_file.read()
@@ -25,8 +26,8 @@ def validate_hash(validate_path : str) -> bool:
def get_hash_path(validate_path : str) -> Optional[str]: def get_hash_path(validate_path : str) -> Optional[str]:
if is_file(validate_path): if is_file(validate_path):
validate_directory_path, _ = os.path.split(validate_path) validate_directory_path, file_name_and_extension = os.path.split(validate_path)
validate_file_name, _ = os.path.splitext(_) validate_file_name = get_file_name(file_name_and_extension)
return os.path.join(validate_directory_path, validate_file_name + '.hash') return os.path.join(validate_directory_path, validate_file_name + '.hash')
return None return None
+75 -36
View File
@@ -1,63 +1,102 @@
from time import sleep import importlib
import random
from functools import lru_cache
from time import sleep, time
from typing import List from typing import List
from onnxruntime import InferenceSession from onnxruntime import InferenceSession
from facefusion import process_manager, state_manager from facefusion import logger, process_manager, state_manager, translator
from facefusion.app_context import detect_app_context from facefusion.app_context import detect_app_context
from facefusion.execution import create_inference_execution_providers from facefusion.common_helper import is_windows
from facefusion.thread_helper import thread_lock from facefusion.execution import create_inference_providers, has_execution_provider
from facefusion.typing import DownloadSet, ExecutionProvider, InferencePool, InferencePoolSet from facefusion.exit_helper import fatal_exit
from facefusion.filesystem import get_file_name, is_file
from facefusion.time_helper import calculate_end_time
from facefusion.types import DownloadSet, ExecutionProvider, InferencePool, InferencePoolSet, InferenceProvider
INFERENCE_POOLS : InferencePoolSet =\ INFERENCE_POOL_SET : InferencePoolSet =\
{ {
'cli': {}, #type:ignore[typeddict-item] 'cli': {},
'ui': {} #type:ignore[typeddict-item] 'api': {}
} }
def get_inference_pool(model_context : str, model_sources : DownloadSet) -> InferencePool: def get_inference_pool(module_name : str, model_names : List[str], model_source_set : DownloadSet) -> InferencePool:
global INFERENCE_POOLS while process_manager.is_checking():
sleep(0.5)
execution_device_ids = state_manager.get_item('execution_device_ids')
execution_providers = state_manager.get_item('execution_providers')
app_context = detect_app_context()
with thread_lock(): for execution_device_id in execution_device_ids:
while process_manager.is_checking(): inference_context = get_inference_context(module_name, model_names, execution_device_id, execution_providers)
sleep(0.5)
app_context = detect_app_context()
inference_context = get_inference_context(model_context)
if app_context == 'cli' and INFERENCE_POOLS.get('ui').get(inference_context): if app_context == 'cli' and INFERENCE_POOL_SET.get('api').get(inference_context):
INFERENCE_POOLS['cli'][inference_context] = INFERENCE_POOLS.get('ui').get(inference_context) INFERENCE_POOL_SET['cli'][inference_context] = INFERENCE_POOL_SET.get('api').get(inference_context)
if app_context == 'ui' and INFERENCE_POOLS.get('cli').get(inference_context): if app_context == 'api' and INFERENCE_POOL_SET.get('cli').get(inference_context):
INFERENCE_POOLS['ui'][inference_context] = INFERENCE_POOLS.get('cli').get(inference_context) INFERENCE_POOL_SET['api'][inference_context] = INFERENCE_POOL_SET.get('cli').get(inference_context)
if not INFERENCE_POOLS.get(app_context).get(inference_context): if not INFERENCE_POOL_SET.get(app_context).get(inference_context):
INFERENCE_POOLS[app_context][inference_context] = create_inference_pool(model_sources, state_manager.get_item('execution_device_id'), state_manager.get_item('execution_providers')) inference_providers = resolve_static_inference_providers(module_name, execution_device_id)
INFERENCE_POOL_SET[app_context][inference_context] = create_inference_pool(model_source_set, inference_providers)
return INFERENCE_POOLS.get(app_context).get(inference_context) current_inference_context = get_inference_context(module_name, model_names, random.choice(execution_device_ids), execution_providers)
return INFERENCE_POOL_SET.get(app_context).get(current_inference_context)
def create_inference_pool(model_sources : DownloadSet, execution_device_id : str, execution_providers : List[ExecutionProvider]) -> InferencePool: def create_inference_pool(model_source_set : DownloadSet, inference_providers : List[InferenceProvider]) -> InferencePool:
inference_pool : InferencePool = {} inference_pool : InferencePool = {}
for model_name in model_sources.keys(): for model_name in model_source_set.keys():
inference_pool[model_name] = create_inference_session(model_sources.get(model_name).get('path'), execution_device_id, execution_providers) model_path = model_source_set.get(model_name).get('path')
if is_file(model_path):
inference_pool[model_name] = create_inference_session(model_path, inference_providers)
return inference_pool return inference_pool
def clear_inference_pool(model_context : str) -> None: def clear_inference_pool(module_name : str, model_names : List[str]) -> None:
global INFERENCE_POOLS execution_device_ids = state_manager.get_item('execution_device_ids')
execution_providers = state_manager.get_item('execution_providers')
app_context = detect_app_context() app_context = detect_app_context()
inference_context = get_inference_context(model_context)
if INFERENCE_POOLS.get(app_context).get(inference_context): if is_windows() and has_execution_provider('directml'):
del INFERENCE_POOLS[app_context][inference_context] INFERENCE_POOL_SET[app_context].clear()
for execution_device_id in execution_device_ids:
inference_context = get_inference_context(module_name, model_names, execution_device_id, execution_providers)
if INFERENCE_POOL_SET.get(app_context).get(inference_context):
del INFERENCE_POOL_SET[app_context][inference_context]
def create_inference_session(model_path : str, execution_device_id : str, execution_providers : List[ExecutionProvider]) -> InferenceSession: def create_inference_session(model_path : str, inference_providers : List[InferenceProvider]) -> InferenceSession:
inference_execution_providers = create_inference_execution_providers(execution_device_id, execution_providers) model_file_name = get_file_name(model_path)
return InferenceSession(model_path, providers = inference_execution_providers) start_time = time()
try:
inference_session = InferenceSession(model_path, providers = inference_providers)
logger.debug(translator.get('loading_model_succeeded').format(model_name = model_file_name, seconds = calculate_end_time(start_time)), __name__)
return inference_session
except Exception:
logger.error(translator.get('loading_model_failed').format(model_name = model_file_name), __name__)
fatal_exit(1)
def get_inference_context(model_context : str) -> str: def get_inference_context(module_name : str, model_names : List[str], execution_device_id : int, execution_providers : List[ExecutionProvider]) -> str:
inference_context = model_context + '.' + '_'.join(state_manager.get_item('execution_providers')) inference_context = '.'.join([ module_name ] + model_names + [ str(execution_device_id) ] + list(execution_providers))
return inference_context return inference_context
@lru_cache()
def resolve_static_inference_providers(module_name : str, execution_device_id : int) -> List[InferenceProvider]:
module = importlib.import_module(module_name)
execution_providers = state_manager.get_item('execution_providers')
if hasattr(module, 'resolve_inference_providers'):
inference_providers = getattr(module, 'resolve_inference_providers')()
if inference_providers:
return inference_providers
return create_inference_providers(execution_device_id, execution_providers)
+45 -64
View File
@@ -3,91 +3,72 @@ import shutil
import signal import signal
import subprocess import subprocess
import sys import sys
import tempfile
from argparse import ArgumentParser, HelpFormatter from argparse import ArgumentParser, HelpFormatter
from typing import Dict, Tuple from functools import partial
from types import FrameType
from facefusion import metadata, wording from facefusion import metadata
from facefusion.common_helper import is_linux, is_macos, is_windows from facefusion.common_helper import is_linux, is_windows
ONNXRUNTIMES : Dict[str, Tuple[str, str]] = {} LOCALES =\
{
if is_macos(): 'install_dependency': 'install the {dependency} package',
ONNXRUNTIMES['default'] = ('onnxruntime', '1.20.1') 'force_reinstall': 'force reinstall of packages',
else: 'skip_conda': 'skip the conda environment check',
ONNXRUNTIMES['default'] = ('onnxruntime', '1.20.1') 'conda_not_activated': 'conda is not activated'
ONNXRUNTIMES['cuda'] = ('onnxruntime-gpu', '1.20.1') }
ONNXRUNTIMES['openvino'] = ('onnxruntime-openvino', '1.20.0') ONNXRUNTIME_SET =\
if is_linux(): {
ONNXRUNTIMES['rocm'] = ('onnxruntime-rocm', '1.19.0') 'default': ('onnxruntime', '1.26.0')
}
if is_windows() or is_linux():
ONNXRUNTIME_SET['cuda'] = ('onnxruntime-gpu', '1.26.0')
ONNXRUNTIME_SET['openvino'] = ('onnxruntime-openvino', '1.24.1')
if is_windows(): if is_windows():
ONNXRUNTIMES['directml'] = ('onnxruntime-directml', '1.17.3') ONNXRUNTIME_SET['directml'] = ('onnxruntime-directml', '1.24.4')
ONNXRUNTIME_SET['qnn'] = ('onnxruntime-qnn', '1.24.4')
if is_linux():
ONNXRUNTIME_SET['migraphx'] = ('onnxruntime-migraphx', '1.25.0')
ONNXRUNTIME_SET['rocm'] = ('onnxruntime-rocm', '1.22.2.post1')
def cli() -> None: def cli() -> None:
signal.signal(signal.SIGINT, lambda signal_number, frame: sys.exit(0)) signal.signal(signal.SIGINT, signal_exit)
program = ArgumentParser(formatter_class = lambda prog: HelpFormatter(prog, max_help_position = 50)) program = ArgumentParser(formatter_class = partial(HelpFormatter, max_help_position = 50))
program.add_argument('--onnxruntime', help = wording.get('help.install_dependency').format(dependency = 'onnxruntime'), choices = ONNXRUNTIMES.keys(), required = True) program.add_argument('onnxruntime', help = LOCALES.get('install_dependency').format(dependency = 'onnxruntime'), choices = ONNXRUNTIME_SET.keys())
program.add_argument('--skip-conda', help = wording.get('help.skip_conda'), action = 'store_true') program.add_argument('--force-reinstall', help = LOCALES.get('force_reinstall'), action = 'store_true')
program.add_argument('--skip-conda', help = LOCALES.get('skip_conda'), action = 'store_true')
program.add_argument('-v', '--version', version = metadata.get('name') + ' ' + metadata.get('version'), action = 'version') program.add_argument('-v', '--version', version = metadata.get('name') + ' ' + metadata.get('version'), action = 'version')
run(program) run(program)
def signal_exit(signum : int, frame : FrameType) -> None:
sys.exit(0)
def run(program : ArgumentParser) -> None: def run(program : ArgumentParser) -> None:
args = program.parse_args() args = program.parse_args()
has_conda = 'CONDA_PREFIX' in os.environ has_conda = 'CONDA_PREFIX' in os.environ
onnxruntime_name, onnxruntime_version = ONNXRUNTIMES.get(args.onnxruntime)
if not args.skip_conda and not has_conda: if not args.skip_conda and not has_conda:
sys.stdout.write(wording.get('conda_not_activated') + os.linesep) sys.stdout.write(LOCALES.get('conda_not_activated') + os.linesep)
sys.exit(1) sys.exit(1)
subprocess.call([ shutil.which('pip'), 'install', '-r', 'requirements.txt', '--force-reinstall' ]) commands = [ shutil.which('pip'), 'install' ]
if args.onnxruntime == 'rocm': if args.force_reinstall:
python_id = 'cp' + str(sys.version_info.major) + str(sys.version_info.minor) commands.append('--force-reinstall')
if python_id in [ 'cp310', 'cp312' ]: with open('requirements.txt') as file:
wheel_name = 'onnxruntime_rocm-' + onnxruntime_version + '-' + python_id + '-' + python_id + '-linux_x86_64.whl'
wheel_path = os.path.join(tempfile.gettempdir(), wheel_name)
wheel_url = 'https://repo.radeon.com/rocm/manylinux/rocm-rel-6.3.1/' + wheel_name
subprocess.call([ shutil.which('curl'), '--silent', '--location', '--continue-at', '-', '--output', wheel_path, wheel_url ])
subprocess.call([ shutil.which('pip'), 'uninstall', 'onnxruntime', wheel_path, '-y', '-q' ])
subprocess.call([ shutil.which('pip'), 'install', wheel_path, '--force-reinstall' ])
os.remove(wheel_path)
else:
subprocess.call([ shutil.which('pip'), 'uninstall', 'onnxruntime', onnxruntime_name, '-y', '-q' ])
subprocess.call([ shutil.which('pip'), 'install', onnxruntime_name + '==' + onnxruntime_version, '--force-reinstall' ])
if args.onnxruntime == 'cuda' and has_conda: for line in file.readlines():
library_paths = [] __line__ = line.strip()
if not __line__.startswith('onnxruntime'):
commands.append(__line__)
if is_linux(): onnxruntime_name, onnxruntime_version = ONNXRUNTIME_SET.get(args.onnxruntime)
if os.getenv('LD_LIBRARY_PATH'): commands.append(onnxruntime_name + '==' + onnxruntime_version)
library_paths = os.getenv('LD_LIBRARY_PATH').split(os.pathsep)
python_id = 'python' + str(sys.version_info.major) + '.' + str(sys.version_info.minor) subprocess.call([ shutil.which('pip'), 'uninstall', 'onnxruntime', onnxruntime_name, '-y', '-q' ])
library_paths.extend(
[
os.path.join(os.getenv('CONDA_PREFIX'), 'lib'),
os.path.join(os.getenv('CONDA_PREFIX'), 'lib', python_id, 'site-packages', 'tensorrt_libs')
])
library_paths = list(dict.fromkeys([ library_path for library_path in library_paths if os.path.exists(library_path) ]))
subprocess.call([ shutil.which('conda'), 'env', 'config', 'vars', 'set', 'LD_LIBRARY_PATH=' + os.pathsep.join(library_paths) ]) subprocess.call(commands)
if is_windows():
if os.getenv('PATH'):
library_paths = os.getenv('PATH').split(os.pathsep)
library_paths.extend(
[
os.path.join(os.getenv('CONDA_PREFIX'), 'Lib'),
os.path.join(os.getenv('CONDA_PREFIX'), 'Lib', 'site-packages', 'tensorrt_libs')
])
library_paths = list(dict.fromkeys([ library_path for library_path in library_paths if os.path.exists(library_path) ]))
subprocess.call([ shutil.which('conda'), 'env', 'config', 'vars', 'set', 'PATH=' + os.pathsep.join(library_paths) ])
if args.onnxruntime in [ 'rocm', 'directml' ]:
subprocess.call([ shutil.which('pip'), 'install', 'numpy==1.26.4', '--force-reinstall' ])
+10 -3
View File
@@ -2,12 +2,19 @@ import os
from datetime import datetime from datetime import datetime
from typing import Optional from typing import Optional
from facefusion.filesystem import get_file_extension, get_file_name
def get_step_output_path(job_id : str, step_index : int, output_path : str) -> Optional[str]: def get_step_output_path(job_id : str, step_index : int, output_path : str) -> Optional[str]:
if output_path: if output_path:
output_directory_path, _ = os.path.split(output_path) output_directory_path, output_file_path = os.path.split(output_path)
output_file_name, output_file_extension = os.path.splitext(_) output_file_name = get_file_name(output_file_path)
return os.path.join(output_directory_path, output_file_name + '-' + job_id + '-' + str(step_index) + output_file_extension) output_file_extension = get_file_extension(output_file_path)
if output_file_name and output_file_extension:
return os.path.join(output_directory_path, output_file_name + '-' + job_id + '-' + str(step_index) + output_file_extension)
if output_file_path and output_directory_path:
return os.path.join(output_directory_path, output_file_path + '-' + job_id + '-' + str(step_index))
return None return None
+6 -6
View File
@@ -1,15 +1,15 @@
from datetime import datetime from datetime import datetime
from typing import Optional, Tuple from typing import List, Optional, Tuple
from facefusion.date_helper import describe_time_ago
from facefusion.jobs import job_manager from facefusion.jobs import job_manager
from facefusion.typing import JobStatus, TableContents, TableHeaders from facefusion.time_helper import describe_time_ago
from facefusion.types import JobStatus, TableContent, TableHeader
def compose_job_list(job_status : JobStatus) -> Tuple[TableHeaders, TableContents]: def compose_job_list(job_status : JobStatus) -> Tuple[List[TableHeader], List[List[TableContent]]]:
jobs = job_manager.find_jobs(job_status) jobs = job_manager.find_jobs(job_status)
job_headers : TableHeaders = [ 'job id', 'steps', 'date created', 'date updated', 'job status' ] job_headers : List[TableHeader] = [ 'job id', 'steps', 'date created', 'date updated', 'job status' ]
job_contents : TableContents = [] job_contents : List[List[TableContent]] = []
for index, job_id in enumerate(jobs): for index, job_id in enumerate(jobs):
if job_manager.validate_job(job_id): if job_manager.validate_job(job_id):
+21 -14
View File
@@ -3,11 +3,12 @@ from copy import copy
from typing import List, Optional from typing import List, Optional
import facefusion.choices import facefusion.choices
from facefusion.date_helper import get_current_date_time from facefusion.filesystem import create_directory, get_file_name, is_directory, is_file, move_file, remove_directory, remove_file, resolve_file_pattern
from facefusion.filesystem import create_directory, is_directory, is_file, move_file, remove_directory, remove_file, resolve_file_pattern
from facefusion.jobs.job_helper import get_step_output_path from facefusion.jobs.job_helper import get_step_output_path
from facefusion.json import read_json, write_json from facefusion.json import read_json, write_json
from facefusion.typing import Args, Job, JobSet, JobStatus, JobStep, JobStepStatus from facefusion.sanitizer import sanitize_job_id
from facefusion.time_helper import get_current_date_time
from facefusion.types import Args, Job, JobSet, JobStatus, JobStep, JobStepStatus
JOBS_PATH : Optional[str] = None JOBS_PATH : Optional[str] = None
@@ -48,14 +49,17 @@ def submit_job(job_id : str) -> bool:
return False return False
def submit_jobs() -> bool: def submit_jobs(halt_on_error : bool) -> bool:
drafted_job_ids = find_job_ids('drafted') drafted_job_ids = find_job_ids('drafted')
has_error = False
if drafted_job_ids: if drafted_job_ids:
for job_id in drafted_job_ids: for job_id in drafted_job_ids:
if not submit_job(job_id): if not submit_job(job_id):
return False has_error = True
return True if halt_on_error:
return False
return not has_error
return False return False
@@ -63,24 +67,27 @@ def delete_job(job_id : str) -> bool:
return delete_job_file(job_id) return delete_job_file(job_id)
def delete_jobs() -> bool: def delete_jobs(halt_on_error : bool) -> bool:
job_ids = find_job_ids('drafted') + find_job_ids('queued') + find_job_ids('failed') + find_job_ids('completed') job_ids = find_job_ids('drafted') + find_job_ids('queued') + find_job_ids('failed') + find_job_ids('completed')
has_error = False
if job_ids: if job_ids:
for job_id in job_ids: for job_id in job_ids:
if not delete_job(job_id): if not delete_job(job_id):
return False has_error = True
return True if halt_on_error:
return False
return not has_error
return False return False
def find_jobs(job_status : JobStatus) -> JobSet: def find_jobs(job_status : JobStatus) -> JobSet:
job_ids = find_job_ids(job_status) job_ids = find_job_ids(job_status)
jobs : JobSet = {} job_set : JobSet = {}
for job_id in job_ids: for job_id in job_ids:
jobs[job_id] = read_job_file(job_id) job_set[job_id] = read_job_file(job_id)
return jobs return job_set
def find_job_ids(job_status : JobStatus) -> List[str]: def find_job_ids(job_status : JobStatus) -> List[str]:
@@ -90,7 +97,7 @@ def find_job_ids(job_status : JobStatus) -> List[str]:
job_ids = [] job_ids = []
for job_path in job_paths: for job_path in job_paths:
job_id, _ = os.path.splitext(os.path.basename(job_path)) job_id = get_file_name(job_path)
job_ids.append(job_id) job_ids.append(job_id)
return job_ids return job_ids
@@ -182,7 +189,6 @@ def set_step_status(job_id : str, step_index : int, step_status : JobStepStatus)
if job: if job:
steps = job.get('steps') steps = job.get('steps')
if has_step(job_id, step_index): if has_step(job_id, step_index):
steps[step_index]['status'] = step_status steps[step_index]['status'] = step_status
return update_job_file(job_id, job) return update_job_file(job_id, job)
@@ -256,5 +262,6 @@ def find_job_path(job_id : str) -> Optional[str]:
def get_job_file_name(job_id : str) -> Optional[str]: def get_job_file_name(job_id : str) -> Optional[str]:
if job_id: if job_id:
job_id = sanitize_job_id(job_id)
return job_id + '.json' return job_id + '.json'
return None return None
+42 -15
View File
@@ -1,7 +1,9 @@
import os
from facefusion.ffmpeg import concat_video from facefusion.ffmpeg import concat_video
from facefusion.filesystem import is_image, is_video, move_file, remove_file from facefusion.filesystem import are_images, are_videos, copy_file, create_directory, is_directory, is_file, move_directory, move_file, remove_directory, remove_file, resolve_file_paths
from facefusion.jobs import job_helper, job_manager from facefusion.jobs import job_helper, job_manager
from facefusion.typing import JobOutputSet, JobStep, ProcessStep from facefusion.types import JobOutputSet, JobStep, ProcessStep
def run_job(job_id : str, process_step : ProcessStep) -> bool: def run_job(job_id : str, process_step : ProcessStep) -> bool:
@@ -16,14 +18,17 @@ def run_job(job_id : str, process_step : ProcessStep) -> bool:
return False return False
def run_jobs(process_step : ProcessStep) -> bool: def run_jobs(process_step : ProcessStep, halt_on_error : bool) -> bool:
queued_job_ids = job_manager.find_job_ids('queued') queued_job_ids = job_manager.find_job_ids('queued')
has_error = False
if queued_job_ids: if queued_job_ids:
for job_id in queued_job_ids: for job_id in queued_job_ids:
if not run_job(job_id, process_step): if not run_job(job_id, process_step):
return False has_error = True
return True if halt_on_error:
return False
return not has_error
return False return False
@@ -35,14 +40,17 @@ def retry_job(job_id : str, process_step : ProcessStep) -> bool:
return False return False
def retry_jobs(process_step : ProcessStep) -> bool: def retry_jobs(process_step : ProcessStep, halt_on_error : bool) -> bool:
failed_job_ids = job_manager.find_job_ids('failed') failed_job_ids = job_manager.find_job_ids('failed')
has_error = False
if failed_job_ids: if failed_job_ids:
for job_id in failed_job_ids: for job_id in failed_job_ids:
if not retry_job(job_id, process_step): if not retry_job(job_id, process_step):
return False has_error = True
return True if halt_on_error:
return False
return not has_error
return False return False
@@ -53,6 +61,8 @@ def run_step(job_id : str, step_index : int, step : JobStep, process_step : Proc
output_path = step_args.get('output_path') output_path = step_args.get('output_path')
step_output_path = job_helper.get_step_output_path(job_id, step_index, output_path) step_output_path = job_helper.get_step_output_path(job_id, step_index, output_path)
if is_directory(output_path):
return move_directory(output_path, step_output_path) and job_manager.set_step_status(job_id, step_index, 'completed')
return move_file(output_path, step_output_path) and job_manager.set_step_status(job_id, step_index, 'completed') return move_file(output_path, step_output_path) and job_manager.set_step_status(job_id, step_index, 'completed')
job_manager.set_step_status(job_id, step_index, 'failed') job_manager.set_step_status(job_id, step_index, 'failed')
return False return False
@@ -73,13 +83,26 @@ def finalize_steps(job_id : str) -> bool:
output_set = collect_output_set(job_id) output_set = collect_output_set(job_id)
for output_path, temp_output_paths in output_set.items(): for output_path, temp_output_paths in output_set.items():
if all(map(is_video, temp_output_paths)): has_videos = are_videos(temp_output_paths)
has_images = are_images(temp_output_paths)
if has_videos:
if not concat_video(output_path, temp_output_paths): if not concat_video(output_path, temp_output_paths):
return False return False
if any(map(is_image, temp_output_paths)): if not has_videos and has_images:
for temp_output_path in temp_output_paths: for temp_output_path in temp_output_paths:
if not move_file(temp_output_path, output_path): if not move_file(temp_output_path, output_path):
return False return False
if not has_videos and not has_images:
if not create_directory(output_path):
return False
for temp_output_path in temp_output_paths:
if is_directory(temp_output_path):
temp_frame_paths = resolve_file_paths(temp_output_path)
for temp_frame_path in temp_frame_paths:
if not copy_file(temp_frame_path, os.path.join(output_path, os.path.basename(temp_frame_path))):
return False
return True return True
@@ -88,19 +111,23 @@ def clean_steps(job_id: str) -> bool:
for temp_output_paths in output_set.values(): for temp_output_paths in output_set.values():
for temp_output_path in temp_output_paths: for temp_output_path in temp_output_paths:
if not remove_file(temp_output_path): if is_file(temp_output_path):
return False if not remove_file(temp_output_path):
return False
if is_directory(temp_output_path):
if not remove_directory(temp_output_path):
return False
return True return True
def collect_output_set(job_id : str) -> JobOutputSet: def collect_output_set(job_id : str) -> JobOutputSet:
steps = job_manager.get_steps(job_id) steps = job_manager.get_steps(job_id)
output_set : JobOutputSet = {} job_output_set : JobOutputSet = {}
for index, step in enumerate(steps): for index, step in enumerate(steps):
output_path = step.get('args').get('output_path') output_path = step.get('args').get('output_path')
if output_path: if output_path:
step_output_path = job_manager.get_step_output_path(job_id, index, output_path) step_output_path = job_manager.get_step_output_path(job_id, index, output_path)
output_set.setdefault(output_path, []).append(step_output_path) job_output_set.setdefault(output_path, []).append(step_output_path)
return output_set return job_output_set
-27
View File
@@ -1,27 +0,0 @@
from typing import List
from facefusion.typing import JobStore
JOB_STORE : JobStore =\
{
'job_keys': [],
'step_keys': []
}
def get_job_keys() -> List[str]:
return JOB_STORE.get('job_keys')
def get_step_keys() -> List[str]:
return JOB_STORE.get('step_keys')
def register_job_keys(step_keys : List[str]) -> None:
for step_key in step_keys:
JOB_STORE['job_keys'].append(step_key)
def register_step_keys(job_keys : List[str]) -> None:
for job_key in job_keys:
JOB_STORE['step_keys'].append(job_key)
+2 -2
View File
@@ -3,13 +3,13 @@ from json import JSONDecodeError
from typing import Optional from typing import Optional
from facefusion.filesystem import is_file from facefusion.filesystem import is_file
from facefusion.typing import Content from facefusion.types import Content
def read_json(json_path : str) -> Optional[Content]: def read_json(json_path : str) -> Optional[Content]:
if is_file(json_path): if is_file(json_path):
try: try:
with open(json_path, 'r') as json_file: with open(json_path) as json_file:
return json.load(json_file) return json.load(json_file)
except JSONDecodeError: except JSONDecodeError:
pass pass
+115
View File
@@ -0,0 +1,115 @@
import ctypes
import ctypes.util
from functools import lru_cache
from typing import List, Optional
@lru_cache
def create_static_library() -> Optional[ctypes.CDLL]:
library_path = ctypes.util.find_library('amd_smi')
if library_path:
library = ctypes.CDLL(library_path)
if library:
return init_ctypes(library)
return None
def init_ctypes(library : ctypes.CDLL) -> ctypes.CDLL:
library.amdsmi_init.argtypes = [ ctypes.c_uint64 ]
library.amdsmi_init.restype = ctypes.c_uint32
library.amdsmi_shut_down.argtypes = []
library.amdsmi_shut_down.restype = ctypes.c_uint32
library.amdsmi_get_socket_handles.argtypes = [ ctypes.POINTER(ctypes.c_uint32), ctypes.POINTER(ctypes.c_void_p) ]
library.amdsmi_get_socket_handles.restype = ctypes.c_uint32
library.amdsmi_get_processor_handles.argtypes = [ ctypes.c_void_p, ctypes.POINTER(ctypes.c_uint32), ctypes.POINTER(ctypes.c_void_p) ]
library.amdsmi_get_processor_handles.restype = ctypes.c_uint32
library.amdsmi_get_gpu_vram_usage.argtypes = [ ctypes.c_void_p, ctypes.c_void_p ]
library.amdsmi_get_gpu_vram_usage.restype = ctypes.c_uint32
library.amdsmi_get_gpu_activity.argtypes = [ ctypes.c_void_p, ctypes.c_void_p ]
library.amdsmi_get_gpu_activity.restype = ctypes.c_uint32
library.amdsmi_get_gpu_asic_info.argtypes = [ ctypes.c_void_p, ctypes.c_void_p ]
library.amdsmi_get_gpu_asic_info.restype = ctypes.c_uint32
library.amdsmi_get_temp_metric.argtypes = [ ctypes.c_void_p, ctypes.c_uint32, ctypes.c_uint32, ctypes.POINTER(ctypes.c_int64) ]
library.amdsmi_get_temp_metric.restype = ctypes.c_uint32
return library
def find_device_handles(amd_smi_library : ctypes.CDLL) -> List[ctypes.c_void_p]:
device_handles : List[ctypes.c_void_p] = []
socket_count = ctypes.c_uint32()
amd_smi_library.amdsmi_get_socket_handles(ctypes.byref(socket_count), ctypes.POINTER(ctypes.c_void_p)())
socket_handles = (ctypes.c_void_p * socket_count.value)()
amd_smi_library.amdsmi_get_socket_handles(ctypes.byref(socket_count), socket_handles)
for socket_index in range(socket_count.value):
device_count = ctypes.c_uint32()
amd_smi_library.amdsmi_get_processor_handles(socket_handles[socket_index], ctypes.byref(device_count), ctypes.POINTER(ctypes.c_void_p)())
processor_handles = (ctypes.c_void_p * device_count.value)()
amd_smi_library.amdsmi_get_processor_handles(socket_handles[socket_index], ctypes.byref(device_count), processor_handles)
for device_index in range(device_count.value):
device_handles.append(ctypes.c_void_p(processor_handles[device_index]))
return device_handles
def define_product_info() -> ctypes.Structure:
return type('AMDSMI_ASIC_INFO', (ctypes.Structure,),
{
'_pack_': 1,
'_fields_':
[
('market_name', ctypes.c_char * 256),
('vendor_id', ctypes.c_uint32),
('vendor_name', ctypes.c_char * 256),
('subvendor_id', ctypes.c_uint32),
('device_id', ctypes.c_uint64),
('rev_id', ctypes.c_uint32),
('asic_serial', ctypes.c_char * 256),
('oam_id', ctypes.c_uint32),
('num_of_compute_units', ctypes.c_uint32),
('padding', ctypes.c_ubyte * 4),
('target_graphics_version', ctypes.c_uint64),
('subsystem_id', ctypes.c_uint32),
('reserved', ctypes.c_uint32 * 21)
]
})()
def define_device_memory() -> ctypes.Structure:
return type('AMDSMI_VRAM_USAGE', (ctypes.Structure,),
{
'_pack_': 1,
'_fields_':
[
('vram_total', ctypes.c_uint32),
('vram_used', ctypes.c_uint32),
('reserved', ctypes.c_uint32 * 2)
]
})()
def define_device_utilization() -> ctypes.Structure:
return type('AMDSMI_ENGINE_USAGE', (ctypes.Structure,),
{
'_pack_': 1,
'_fields_':
[
('gfx_activity', ctypes.c_uint32),
('umc_activity', ctypes.c_uint32),
('mm_activity', ctypes.c_uint32),
('reserved', ctypes.c_uint32 * 13)
]
})()
+134
View File
@@ -0,0 +1,134 @@
import ctypes
from functools import lru_cache
from typing import Optional
from facefusion.common_helper import is_linux, is_macos, is_windows
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url_by_provider
from facefusion.filesystem import resolve_relative_path
from facefusion.types import LibrarySet
@lru_cache
def create_static_library_set() -> Optional[LibrarySet]:
if is_linux():
return\
{
'hashes':
{
'aom':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libaom.hash'),
'path': resolve_relative_path('../.libraries/libaom.hash')
}
},
'sources':
{
'aom':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libaom.so'),
'path': resolve_relative_path('../.libraries/libaom.so')
}
}
}
if is_macos():
return\
{
'hashes':
{
'aom':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libaom.hash'),
'path': resolve_relative_path('../.libraries/libaom.hash')
}
},
'sources':
{
'aom':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libaom.dylib'),
'path': resolve_relative_path('../.libraries/libaom.dylib')
}
}
}
if is_windows():
return\
{
'hashes':
{
'aom':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/aom.hash'),
'path': resolve_relative_path('../.libraries/aom.hash')
}
},
'sources':
{
'aom':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/aom.dll'),
'path': resolve_relative_path('../.libraries/aom.dll')
}
}
}
return None
def pre_check() -> bool:
library_hash_set = create_static_library_set().get('hashes')
library_source_set = create_static_library_set().get('sources')
return conditional_download_hashes(library_hash_set) and conditional_download_sources(library_source_set)
@lru_cache
def create_static_library() -> Optional[ctypes.CDLL]:
library_path = create_static_library_set().get('sources').get('aom').get('path')
if library_path:
if is_windows():
library = ctypes.CDLL(library_path, winmode = 0)
else:
library = ctypes.CDLL(library_path)
if library:
return init_ctypes(library)
return None
def init_ctypes(library : ctypes.CDLL) -> ctypes.CDLL:
library.aom_codec_enc_config_default.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_uint ]
library.aom_codec_enc_config_default.restype = ctypes.c_int
library.aom_codec_enc_init_ver.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_long, ctypes.c_int ]
library.aom_codec_enc_init_ver.restype = ctypes.c_int
library.aom_codec_encode.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_int64, ctypes.c_ulong, ctypes.c_long, ctypes.c_ulong ]
library.aom_codec_encode.restype = ctypes.c_int
library.aom_codec_get_cx_data.argtypes = [ ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p) ]
library.aom_codec_get_cx_data.restype = ctypes.c_void_p
library.aom_codec_enc_config_set.argtypes = [ ctypes.c_void_p, ctypes.c_void_p ]
library.aom_codec_enc_config_set.restype = ctypes.c_int
library.aom_codec_destroy.argtypes = [ ctypes.c_void_p ]
library.aom_codec_destroy.restype = ctypes.c_int
library.aom_img_wrap.argtypes = [ ctypes.c_void_p, ctypes.c_int, ctypes.c_uint, ctypes.c_uint, ctypes.c_uint, ctypes.c_void_p ]
library.aom_img_wrap.restype = ctypes.c_void_p
library.aom_codec_control.argtypes = [ ctypes.c_void_p, ctypes.c_int, ctypes.c_int ]
library.aom_codec_control.restype = ctypes.c_int
library.aom_codec_dec_init_ver.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_long, ctypes.c_int ]
library.aom_codec_dec_init_ver.restype = ctypes.c_int
library.aom_codec_decode.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_uint, ctypes.c_void_p ]
library.aom_codec_decode.restype = ctypes.c_int
library.aom_codec_get_frame.argtypes = [ ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p) ]
library.aom_codec_get_frame.restype = ctypes.c_void_p
return library
+294
View File
@@ -0,0 +1,294 @@
import ctypes
from functools import lru_cache
from typing import Optional
from facefusion.common_helper import is_linux, is_macos, is_windows
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url_by_provider
from facefusion.filesystem import resolve_relative_path
from facefusion.types import LibrarySet
@lru_cache
def create_static_library_set() -> Optional[LibrarySet]:
if is_linux():
return\
{
'hashes':
{
'crypto':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libcrypto.hash'),
'path': resolve_relative_path('../.libraries/libcrypto.hash')
},
'datachannel':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libdatachannel_next.hash'),
'path': resolve_relative_path('../.libraries/libdatachannel_next.hash')
},
'ssl':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libssl.hash'),
'path': resolve_relative_path('../.libraries/libssl.hash')
}
},
'sources':
{
'crypto':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libcrypto.so'),
'path': resolve_relative_path('../.libraries/libcrypto.so')
},
'datachannel':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libdatachannel_next.so'),
'path': resolve_relative_path('../.libraries/libdatachannel_next.so')
},
'ssl':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libssl.so'),
'path': resolve_relative_path('../.libraries/libssl.so')
}
}
}
if is_macos():
return\
{
'hashes':
{
'crypto':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libcrypto.hash'),
'path': resolve_relative_path('../.libraries/libcrypto.hash')
},
'datachannel':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libdatachannel_next.hash'),
'path': resolve_relative_path('../.libraries/libdatachannel_next.hash')
},
'ssl':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libssl.hash'),
'path': resolve_relative_path('../.libraries/libssl.hash')
}
},
'sources':
{
'crypto':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libcrypto.dylib'),
'path': resolve_relative_path('../.libraries/libcrypto.dylib')
},
'datachannel':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libdatachannel_next.dylib'),
'path': resolve_relative_path('../.libraries/libdatachannel_next.dylib')
},
'ssl':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libssl.dylib'),
'path': resolve_relative_path('../.libraries/libssl.dylib')
}
}
}
if is_windows():
return\
{
'hashes':
{
'crypto':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/crypto.hash'),
'path': resolve_relative_path('../.libraries/crypto.hash')
},
'datachannel':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/datachannel_next.hash'),
'path': resolve_relative_path('../.libraries/datachannel_next.hash')
},
'ssl':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/ssl.hash'),
'path': resolve_relative_path('../.libraries/ssl.hash')
}
},
'sources':
{
'crypto':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/crypto.dll'),
'path': resolve_relative_path('../.libraries/crypto.dll')
},
'datachannel':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/datachannel_next.dll'),
'path': resolve_relative_path('../.libraries/datachannel_next.dll')
},
'ssl':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/ssl.dll'),
'path': resolve_relative_path('../.libraries/ssl.dll')
}
}
}
return None
def pre_check() -> bool:
library_hash_set = create_static_library_set().get('hashes')
library_source_set = create_static_library_set().get('sources')
return conditional_download_hashes(library_hash_set) and conditional_download_sources(library_source_set)
@lru_cache
def create_static_library() -> Optional[ctypes.CDLL]:
crypto_source_path = create_static_library_set().get('sources').get('crypto').get('path')
datachannel_source_path = create_static_library_set().get('sources').get('datachannel').get('path')
ssl_source_path = create_static_library_set().get('sources').get('ssl').get('path')
if crypto_source_path and datachannel_source_path and ssl_source_path:
if is_windows():
ctypes.CDLL(crypto_source_path, winmode = 0)
ctypes.CDLL(ssl_source_path, winmode = 0)
library = ctypes.CDLL(datachannel_source_path, winmode = 0)
else:
ctypes.CDLL(crypto_source_path)
ctypes.CDLL(ssl_source_path)
library = ctypes.CDLL(datachannel_source_path)
if library:
return init_ctypes(library)
return None
def init_ctypes(library : ctypes.CDLL) -> ctypes.CDLL:
library.rtcInitLogger.argtypes = [ ctypes.c_int, ctypes.CFUNCTYPE(None, ctypes.c_int, ctypes.c_char_p) ]
library.rtcInitLogger.restype = None
library.rtcInitLogger(2, ctypes.CFUNCTYPE(None, ctypes.c_int, ctypes.c_char_p)(0))
library.rtcCreatePeerConnection.restype = ctypes.c_int
library.rtcDeletePeerConnection.argtypes = [ ctypes.c_int ]
library.rtcDeletePeerConnection.restype = ctypes.c_int
library.rtcSetLocalDescription.argtypes = [ ctypes.c_int, ctypes.c_char_p ]
library.rtcSetLocalDescription.restype = ctypes.c_int
library.rtcSetRemoteDescription.argtypes = [ ctypes.c_int, ctypes.c_char_p, ctypes.c_char_p ]
library.rtcSetRemoteDescription.restype = ctypes.c_int
library.rtcAddTrackEx.restype = ctypes.c_int
library.rtcSendMessage.argtypes = [ ctypes.c_int, ctypes.c_void_p, ctypes.c_int ]
library.rtcSendMessage.restype = ctypes.c_int
library.rtcSetAV1Packetizer.restype = ctypes.c_int
library.rtcSetVP8Packetizer.restype = ctypes.c_int
library.rtcSetVP9Packetizer.restype = ctypes.c_int
library.rtcChainRtcpSrReporter.argtypes = [ ctypes.c_int ]
library.rtcChainRtcpSrReporter.restype = ctypes.c_int
library.rtcSetTrackRtpTimestamp.argtypes = [ ctypes.c_int, ctypes.c_uint32 ]
library.rtcSetTrackRtpTimestamp.restype = ctypes.c_int
library.rtcIsOpen.argtypes = [ ctypes.c_int ]
library.rtcIsOpen.restype = ctypes.c_bool
library.rtcChainRtcpNackResponder.argtypes = [ ctypes.c_int, ctypes.c_uint ]
library.rtcChainRtcpNackResponder.restype = ctypes.c_int
library.rtcGetLocalDescription.argtypes = [ ctypes.c_int, ctypes.c_char_p, ctypes.c_int ]
library.rtcGetLocalDescription.restype = ctypes.c_int
library.rtcSetOpusPacketizer.restype = ctypes.c_int
library.rtcGetPayloadTypesForCodec.argtypes = [ ctypes.c_char_p, ctypes.c_char_p, ctypes.POINTER(ctypes.c_int), ctypes.c_int ]
library.rtcGetPayloadTypesForCodec.restype = ctypes.c_int
library.rtcSetAV1Depacketizer.argtypes = [ ctypes.c_int, ctypes.c_int ]
library.rtcSetAV1Depacketizer.restype = ctypes.c_int
library.rtcSetVP8Depacketizer.restype = ctypes.c_int
library.rtcSetVP9Depacketizer.restype = ctypes.c_int
library.rtcSetOpusDepacketizer.restype = ctypes.c_int
library.rtcChainRtcpReceivingSession.argtypes = [ ctypes.c_int ]
library.rtcChainRtcpReceivingSession.restype = ctypes.c_int
library.rtcSetFrameCallback.argtypes = [ ctypes.c_int, ctypes.CFUNCTYPE(None, ctypes.c_int, ctypes.c_void_p, ctypes.c_int, ctypes.c_void_p, ctypes.c_void_p) ]
library.rtcSetFrameCallback.restype = ctypes.c_int
library.rtcSetUserPointer.argtypes = [ ctypes.c_int, ctypes.c_void_p ]
library.rtcSetUserPointer.restype = None
library.rtcChainRembHandler.argtypes = [ ctypes.c_int, ctypes.CFUNCTYPE(None, ctypes.c_int, ctypes.c_uint, ctypes.c_void_p) ]
library.rtcChainRembHandler.restype = ctypes.c_int
library.rtcRequestBitrate.argtypes = [ ctypes.c_int, ctypes.c_uint ]
library.rtcRequestBitrate.restype = ctypes.c_int
library.rtcSetClosedCallback.argtypes = [ ctypes.c_int, ctypes.CFUNCTYPE(None, ctypes.c_int, ctypes.c_void_p) ]
library.rtcSetClosedCallback.restype = ctypes.c_int
return library
def define_rtc_configuration() -> ctypes.Structure:
return type('RTC_CONFIGURATION', (ctypes.Structure,),
{
'_fields_':
[
('iceServers', ctypes.POINTER(ctypes.c_char_p)),
('iceServersCount', ctypes.c_int),
('proxyServer', ctypes.c_char_p),
('bindAddress', ctypes.c_char_p),
('certificateType', ctypes.c_int),
('iceTransportPolicy', ctypes.c_int),
('enableIceTcp', ctypes.c_bool),
('enableIceUdpMux', ctypes.c_bool),
('disableAutoNegotiation', ctypes.c_bool),
('forceMediaTransport', ctypes.c_bool),
('portRangeBegin', ctypes.c_ushort),
('portRangeEnd', ctypes.c_ushort),
('mtu', ctypes.c_int),
('maxMessageSize', ctypes.c_int)
]
})()
def define_rtc_track_init() -> ctypes.Structure:
return type('RTC_TRACK_INIT', (ctypes.Structure,),
{
'_fields_':
[
('direction', ctypes.c_int),
('codec', ctypes.c_int),
('payloadType', ctypes.c_int),
('ssrc', ctypes.c_uint32),
('mid', ctypes.c_char_p),
('name', ctypes.c_char_p),
('msid', ctypes.c_char_p),
('trackId', ctypes.c_char_p),
('profile', ctypes.c_char_p)
]
})()
def define_rtc_packetizer_init() -> ctypes.Structure:
return type('RTC_PACKETIZER_INIT', (ctypes.Structure,),
{
'_fields_':
[
('ssrc', ctypes.c_uint32),
('cname', ctypes.c_char_p),
('payloadType', ctypes.c_uint8),
('clockRate', ctypes.c_uint32),
('sequenceNumber', ctypes.c_uint16),
('timestamp', ctypes.c_uint32),
('maxFragmentSize', ctypes.c_uint16),
('nalSeparator', ctypes.c_int),
('obuPacketization', ctypes.c_int)
]
})()
+88
View File
@@ -0,0 +1,88 @@
import ctypes
import ctypes.util
from functools import lru_cache
from typing import List, Optional
@lru_cache
def create_static_library() -> Optional[ctypes.CDLL]:
library_path = ctypes.util.find_library('nvidia-ml') or ctypes.util.find_library('nvml')
if library_path:
library = ctypes.CDLL(library_path)
if library:
return init_ctypes(library)
return None
def init_ctypes(library : ctypes.CDLL) -> ctypes.CDLL:
library.nvmlInit_v2.argtypes = []
library.nvmlInit_v2.restype = ctypes.c_int
library.nvmlShutdown.argtypes = []
library.nvmlShutdown.restype = ctypes.c_int
library.nvmlDeviceGetCount_v2.argtypes = [ ctypes.POINTER(ctypes.c_uint) ]
library.nvmlDeviceGetCount_v2.restype = ctypes.c_int
library.nvmlSystemGetDriverVersion.argtypes = [ ctypes.c_char_p, ctypes.c_uint ]
library.nvmlSystemGetDriverVersion.restype = ctypes.c_int
library.nvmlSystemGetCudaDriverVersion.argtypes = [ ctypes.POINTER(ctypes.c_int) ]
library.nvmlSystemGetCudaDriverVersion.restype = ctypes.c_int
library.nvmlDeviceGetHandleByIndex_v2.argtypes = [ ctypes.c_uint, ctypes.POINTER(ctypes.c_void_p) ]
library.nvmlDeviceGetHandleByIndex_v2.restype = ctypes.c_int
library.nvmlDeviceGetName.argtypes = [ ctypes.c_void_p, ctypes.c_char_p, ctypes.c_uint ]
library.nvmlDeviceGetName.restype = ctypes.c_int
library.nvmlDeviceGetMemoryInfo.argtypes = [ ctypes.c_void_p, ctypes.c_void_p ]
library.nvmlDeviceGetMemoryInfo.restype = ctypes.c_int
library.nvmlDeviceGetTemperature.argtypes = [ ctypes.c_void_p, ctypes.c_int, ctypes.POINTER(ctypes.c_uint) ]
library.nvmlDeviceGetTemperature.restype = ctypes.c_int
library.nvmlDeviceGetUtilizationRates.argtypes = [ ctypes.c_void_p, ctypes.c_void_p ]
library.nvmlDeviceGetUtilizationRates.restype = ctypes.c_int
return library
def find_device_handles(nvidia_ml_library : ctypes.CDLL) -> List[ctypes.c_void_p]:
device_handles : List[ctypes.c_void_p] = []
device_count = ctypes.c_uint()
nvidia_ml_library.nvmlDeviceGetCount_v2(ctypes.byref(device_count))
for device_id in range(device_count.value):
device_handle = ctypes.c_void_p()
nvidia_ml_library.nvmlDeviceGetHandleByIndex_v2(device_id, ctypes.byref(device_handle))
device_handles.append(device_handle)
return device_handles
def define_device_memory() -> ctypes.Structure:
return type('NVML_MEMORY', (ctypes.Structure,),
{
'_fields_':
[
('total', ctypes.c_ulonglong),
('free', ctypes.c_ulonglong),
('used', ctypes.c_ulonglong)
]
})()
def define_device_utilization() -> ctypes.Structure:
return type('NVML_UTILIZATION', (ctypes.Structure,),
{
'_fields_':
[
('gpu', ctypes.c_uint),
('memory', ctypes.c_uint)
]
})()
+124
View File
@@ -0,0 +1,124 @@
import ctypes
from functools import lru_cache
from typing import Optional
from facefusion.common_helper import is_linux, is_macos, is_windows
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url_by_provider
from facefusion.filesystem import resolve_relative_path
from facefusion.types import LibrarySet
@lru_cache
def create_static_library_set() -> Optional[LibrarySet]:
if is_linux():
return\
{
'hashes':
{
'opus':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libopus.hash'),
'path': resolve_relative_path('../.libraries/libopus.hash')
}
},
'sources':
{
'opus':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libopus.so'),
'path': resolve_relative_path('../.libraries/libopus.so')
}
}
}
if is_macos():
return\
{
'hashes':
{
'opus':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libopus.hash'),
'path': resolve_relative_path('../.libraries/libopus.hash')
}
},
'sources':
{
'opus':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libopus.dylib'),
'path': resolve_relative_path('../.libraries/libopus.dylib')
}
}
}
if is_windows():
return\
{
'hashes':
{
'opus':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/opus.hash'),
'path': resolve_relative_path('../.libraries/opus.hash')
}
},
'sources':
{
'opus':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/opus.dll'),
'path': resolve_relative_path('../.libraries/opus.dll')
}
}
}
return None
def pre_check() -> bool:
library_hash_set = create_static_library_set().get('hashes')
library_source_set = create_static_library_set().get('sources')
return conditional_download_hashes(library_hash_set) and conditional_download_sources(library_source_set)
@lru_cache
def create_static_library() -> Optional[ctypes.CDLL]:
library_path = create_static_library_set().get('sources').get('opus').get('path')
if library_path:
if is_windows():
library = ctypes.CDLL(library_path, winmode = 0)
else:
library = ctypes.CDLL(library_path)
if library:
return init_ctypes(library)
return None
def init_ctypes(library : ctypes.CDLL) -> ctypes.CDLL:
library.opus_encoder_create.argtypes = [ ctypes.c_int, ctypes.c_int, ctypes.c_int, ctypes.POINTER(ctypes.c_int) ]
library.opus_encoder_create.restype = ctypes.c_void_p
library.opus_encode_float.argtypes = [ ctypes.c_void_p, ctypes.POINTER(ctypes.c_float), ctypes.c_int, ctypes.c_char_p, ctypes.c_int ]
library.opus_encode_float.restype = ctypes.c_int
library.opus_encoder_destroy.argtypes = [ ctypes.c_void_p ]
library.opus_encoder_destroy.restype = None
library.opus_decoder_create.argtypes = [ ctypes.c_int, ctypes.c_int, ctypes.POINTER(ctypes.c_int) ]
library.opus_decoder_create.restype = ctypes.c_void_p
library.opus_decode_float.argtypes = [ ctypes.c_void_p, ctypes.c_char_p, ctypes.c_int, ctypes.POINTER(ctypes.c_float), ctypes.c_int, ctypes.c_int ]
library.opus_decode_float.restype = ctypes.c_int
library.opus_decoder_get_nb_samples.argtypes = [ ctypes.c_void_p, ctypes.c_char_p, ctypes.c_int ]
library.opus_decoder_get_nb_samples.restype = ctypes.c_int
library.opus_decoder_destroy.argtypes = [ ctypes.c_void_p ]
library.opus_decoder_destroy.restype = None
return library
+24
View File
@@ -0,0 +1,24 @@
import ctypes
import ctypes.util
from functools import lru_cache
from typing import Optional
@lru_cache
def create_static_library() -> Optional[ctypes.CDLL]:
library_path = ctypes.util.find_library('rocm-core')
if library_path:
library = ctypes.CDLL(library_path)
if library:
return init_ctypes(library)
return None
def init_ctypes(library : ctypes.CDLL) -> ctypes.CDLL:
library.getROCmVersion.argtypes = [ ctypes.POINTER(ctypes.c_uint), ctypes.POINTER(ctypes.c_uint), ctypes.POINTER(ctypes.c_uint) ]
library.getROCmVersion.restype = ctypes.c_int
return library
+134
View File
@@ -0,0 +1,134 @@
import ctypes
from functools import lru_cache
from typing import Optional
from facefusion.common_helper import is_linux, is_macos, is_windows
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url_by_provider
from facefusion.filesystem import resolve_relative_path
from facefusion.types import LibrarySet
@lru_cache
def create_static_library_set() -> Optional[LibrarySet]:
if is_linux():
return\
{
'hashes':
{
'vpx':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libvpx.hash'),
'path': resolve_relative_path('../.libraries/libvpx.hash')
}
},
'sources':
{
'vpx':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libvpx.so'),
'path': resolve_relative_path('../.libraries/libvpx.so')
}
}
}
if is_macos():
return\
{
'hashes':
{
'vpx':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libvpx.hash'),
'path': resolve_relative_path('../.libraries/libvpx.hash')
}
},
'sources':
{
'vpx':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libvpx.dylib'),
'path': resolve_relative_path('../.libraries/libvpx.dylib')
}
}
}
if is_windows():
return\
{
'hashes':
{
'vpx':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/vpx.hash'),
'path': resolve_relative_path('../.libraries/vpx.hash')
}
},
'sources':
{
'vpx':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/vpx.dll'),
'path': resolve_relative_path('../.libraries/vpx.dll')
}
}
}
return None
def pre_check() -> bool:
library_hash_set = create_static_library_set().get('hashes')
library_source_set = create_static_library_set().get('sources')
return conditional_download_hashes(library_hash_set) and conditional_download_sources(library_source_set)
@lru_cache
def create_static_library() -> Optional[ctypes.CDLL]:
library_path = create_static_library_set().get('sources').get('vpx').get('path')
if library_path:
if is_windows():
library = ctypes.CDLL(library_path, winmode = 0)
else:
library = ctypes.CDLL(library_path)
if library:
return init_ctypes(library)
return None
def init_ctypes(library : ctypes.CDLL) -> ctypes.CDLL:
library.vpx_codec_enc_config_default.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_uint ]
library.vpx_codec_enc_config_default.restype = ctypes.c_int
library.vpx_codec_enc_init_ver.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_long, ctypes.c_int ]
library.vpx_codec_enc_init_ver.restype = ctypes.c_int
library.vpx_codec_encode.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_int64, ctypes.c_ulong, ctypes.c_long, ctypes.c_ulong ]
library.vpx_codec_encode.restype = ctypes.c_int
library.vpx_codec_get_cx_data.argtypes = [ ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p) ]
library.vpx_codec_get_cx_data.restype = ctypes.c_void_p
library.vpx_codec_enc_config_set.argtypes = [ ctypes.c_void_p, ctypes.c_void_p ]
library.vpx_codec_enc_config_set.restype = ctypes.c_int
library.vpx_codec_destroy.argtypes = [ ctypes.c_void_p ]
library.vpx_codec_destroy.restype = ctypes.c_int
library.vpx_img_wrap.argtypes = [ ctypes.c_void_p, ctypes.c_int, ctypes.c_uint, ctypes.c_uint, ctypes.c_uint, ctypes.c_void_p ]
library.vpx_img_wrap.restype = ctypes.c_void_p
library.vpx_codec_control_.argtypes = [ ctypes.c_void_p, ctypes.c_int, ctypes.c_int ]
library.vpx_codec_control_.restype = ctypes.c_int
library.vpx_codec_dec_init_ver.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_long, ctypes.c_int ]
library.vpx_codec_dec_init_ver.restype = ctypes.c_int
library.vpx_codec_decode.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_uint, ctypes.c_void_p, ctypes.c_long ]
library.vpx_codec_decode.restype = ctypes.c_int
library.vpx_codec_get_frame.argtypes = [ ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p) ]
library.vpx_codec_get_frame.restype = ctypes.c_void_p
return library
+279
View File
@@ -0,0 +1,279 @@
from facefusion.types import Locales
LOCALES : Locales =\
{
'en':
{
'conda_not_activated': 'conda is not activated',
'python_not_supported': 'python version is not supported, upgrade to {version} or higher',
'curl_not_installed': 'curl is not installed',
'ffmpeg_not_installed': 'ffmpeg is not installed',
'creating_temp': 'creating temporary resources',
'extracting_frames': 'extracting frames with a resolution of {resolution} and {fps} frames per second',
'extracting_frames_succeeded': 'extracting frames succeeded',
'extracting_frames_failed': 'extracting frames failed',
'spawning_frames_succeeded': 'spawning frames succeeded',
'spawning_frames_failed': 'spawning frames failed',
'analysing': 'analysing',
'extracting': 'extracting',
'spawning': 'spawning',
'streaming': 'streaming',
'processing': 'processing',
'merging': 'merging',
'downloading': 'downloading',
'temp_frames_not_found': 'temporary frames not found',
'copying_image': 'copying image with a resolution of {resolution}',
'copying_image_succeeded': 'copying image succeeded',
'copying_image_failed': 'copying image failed',
'finalizing_image': 'finalizing image with a resolution of {resolution}',
'finalizing_image_succeeded': 'finalizing image succeeded',
'finalizing_image_skipped': 'finalizing image skipped',
'merging_video': 'merging video with a resolution of {resolution} and {fps} frames per second',
'merging_video_succeeded': 'merging video succeeded',
'merging_video_failed': 'merging video failed',
'skipping_audio': 'skipping audio',
'replacing_audio_succeeded': 'replacing audio succeeded',
'replacing_audio_skipped': 'replacing audio skipped',
'restoring_audio_succeeded': 'restoring audio succeeded',
'restoring_audio_skipped': 'restoring audio skipped',
'clearing_temp': 'clearing temporary resources',
'processing_stopped': 'processing stopped',
'processing_image_succeeded': 'processing to image succeeded in {seconds} seconds',
'processing_image_failed': 'processing to image failed',
'processing_frames_succeeded': 'processing to frames succeeded in {seconds} seconds',
'processing_frames_failed': 'processing to frames failed',
'processing_video_succeeded': 'processing to video succeeded in {seconds} seconds',
'processing_video_failed': 'processing to video failed',
'choose_image_source': 'choose an image for the source',
'choose_audio_source': 'choose an audio for the source',
'choose_video_target': 'choose a video for the target',
'choose_image_or_video_target': 'choose an image or video for the target',
'specify_image_or_video_output': 'specify the output image or video within a directory',
'match_target_and_output_extension': 'match the target and output extension',
'no_source_face_detected': 'no source face detected',
'processor_not_loaded': 'processor {processor} could not be loaded',
'processor_not_implemented': 'processor {processor} not implemented correctly',
'stream_not_loaded': 'stream {stream_mode} could not be loaded',
'stream_not_supported': 'stream not supported',
'api_started': 'started API on {host}:{port}',
'job_created': 'job {job_id} created',
'job_not_created': 'job {job_id} not created',
'job_submitted': 'job {job_id} submitted',
'job_not_submitted': 'job {job_id} not submitted',
'job_all_submitted': 'jobs submitted',
'job_all_not_submitted': 'jobs not submitted',
'job_deleted': 'job {job_id} deleted',
'job_not_deleted': 'job {job_id} not deleted',
'job_all_deleted': 'jobs deleted',
'job_all_not_deleted': 'jobs not deleted',
'job_step_added': 'step added to job {job_id}',
'job_step_not_added': 'step not added to job {job_id}',
'job_remix_step_added': 'step {step_index} remixed from job {job_id}',
'job_remix_step_not_added': 'step {step_index} not remixed from job {job_id}',
'job_step_inserted': 'step {step_index} inserted to job {job_id}',
'job_step_not_inserted': 'step {step_index} not inserted to job {job_id}',
'job_step_removed': 'step {step_index} removed from job {job_id}',
'job_step_not_removed': 'step {step_index} not removed from job {job_id}',
'running_job': 'running queued job {job_id}',
'running_jobs': 'running all queued jobs',
'retrying_job': 'retrying failed job {job_id}',
'retrying_jobs': 'retrying all failed jobs',
'processing_job_succeeded': 'processing of job {job_id} succeeded',
'processing_jobs_succeeded': 'processing of all jobs succeeded',
'processing_job_failed': 'processing of job {job_id} failed',
'processing_jobs_failed': 'processing of all jobs failed',
'processing_step': 'processing step {step_current} of {step_total}',
'validating_hash_succeeded': 'validating hash for {hash_file_name} succeeded',
'validating_hash_failed': 'validating hash for {hash_file_name} failed',
'validating_source_succeeded': 'validating source for {source_file_name} succeeded',
'validating_source_failed': 'validating source for {source_file_name} failed',
'deleting_corrupt_source': 'deleting corrupt source for {source_file_name}',
'loading_model_succeeded': 'loading model {model_name} succeeded in {seconds} seconds',
'loading_model_failed': 'loading model {model_name} failed',
'time_ago_now': 'just now',
'time_ago_minutes': '{minutes} minutes ago',
'time_ago_hours': '{hours} hours and {minutes} minutes ago',
'time_ago_days': '{days} days, {hours} hours and {minutes} minutes ago',
'point': '.',
'comma': ',',
'colon': ':',
'question_mark': '?',
'exclamation_mark': '!',
'help':
{
'install_dependency': 'choose the variant of {dependency} to install',
'skip_conda': 'skip the conda environment check',
'workflow': 'choose the workflow',
'config_path': 'choose the config file to override defaults',
'temp_path': 'specify the directory for the temporary resources',
'jobs_path': 'specify the directory to store jobs',
'source_paths': 'choose the image or audio paths',
'target_path': 'choose the image or video path',
'output_path': 'specify the image or video within a directory',
'source_pattern': 'choose the image or audio pattern',
'target_pattern': 'choose the image or video pattern',
'output_pattern': 'specify the image or video pattern',
'face_detector_model': 'choose the model responsible for detecting the faces',
'face_detector_size': 'specify the frame size provided to the face detector',
'face_detector_margin': 'apply top, right, bottom and left margin to the frame',
'face_detector_angles': 'specify the angles to rotate the frame before detecting faces',
'face_detector_score': 'filter the detected faces based on the confidence score',
'face_landmarker_model': 'choose the model responsible for detecting the face landmarks',
'face_landmarker_score': 'filter the detected face landmarks based on the confidence score',
'face_selector_mode': 'use reference based tracking or simple matching',
'face_selector_order': 'specify the order of the detected faces',
'face_selector_age_start': 'filter the detected faces based on the starting age',
'face_selector_age_end': 'filter the detected faces based on the ending age',
'face_selector_gender': 'filter the detected faces based on their gender',
'face_selector_race': 'filter the detected faces based on their race',
'reference_face_position': 'specify the position used to create the reference face',
'reference_face_distance': 'specify the similarity between the reference face and target face',
'reference_frame_number': 'specify the frame used to create the reference face',
'face_tracker_score': 'specify the overlap score used to match the tracked faces',
'face_occluder_model': 'choose the model responsible for the occlusion mask',
'face_parser_model': 'choose the model responsible for the region mask',
'face_mask_types': 'mix and match different face mask types (choices: {choices})',
'face_mask_areas': 'choose the items used for the area mask (choices: {choices})',
'face_mask_regions': 'choose the items used for the region mask (choices: {choices})',
'face_mask_blur': 'specify the degree of blur applied to the box mask',
'face_mask_padding': 'apply top, right, bottom and left padding to the box mask',
'voice_extractor_model': 'choose the model responsible for extracting the voices',
'trim_frame_start': 'specify the starting frame of the target video',
'trim_frame_end': 'specify the ending frame of the target video',
'temp_frame_format': 'specify the temporary resources format',
'target_frame_amount': 'specify the amount of target frames forwarded to the processor',
'output_image_quality': 'specify the image quality which translates to the image compression',
'output_image_scale': 'specify the image scale based on the target image',
'output_audio_encoder': 'specify the encoder used for the audio',
'output_audio_quality': 'specify the audio quality which translates to the audio compression',
'output_audio_volume': 'specify the audio volume based on the target video',
'output_audio_fps': 'specify the fps used when converting audio to video frames',
'output_video_encoder': 'specify the encoder used for the video',
'output_video_preset': 'balance fast video processing and video file size',
'output_video_quality': 'specify the video quality which translates to the video compression',
'output_video_scale': 'specify the video scale based on the target video',
'output_video_fps': 'specify the video fps based on the target video',
'processors': 'load a single or multiple processors (choices: {choices}, ...)',
'background-remover-model': 'choose the model responsible for removing the background',
'background-remover-color': 'apply red, green blue and alpha values of the background',
'download_providers': 'download using different providers (choices: {choices}, ...)',
'download_scope': 'specify the download scope',
'benchmark_mode': 'choose the benchmark mode',
'benchmark_resolutions': 'choose the resolutions for the benchmarks (choices: {choices}, ...)',
'benchmark_cycle_count': 'specify the amount of cycles per benchmark',
'api_host': 'specify the API host',
'api_port': 'specify the API port',
'api_security_strategy': 'specify the API security strategy used for sanitizing uploaded assets',
'execution_device_ids': 'specify the devices used for processing',
'execution_providers': 'inference using different providers (choices: {choices}, ...)',
'execution_thread_count': 'specify the amount of parallel threads while processing',
'video_memory_strategy': 'balance fast processing and low VRAM usage',
'log_level': 'adjust the message severity displayed in the terminal',
'halt_on_error': 'halt the program once an error occurred',
'run': 'run the program',
'batch_run': 'run the program in batch mode',
'force_download': 'force automate downloads and exit',
'benchmark': 'benchmark the program',
'api': 'start the API server',
'job_id': 'specify the job id',
'job_status': 'specify the job status',
'step_index': 'specify the step index',
'job_list': 'list jobs by status',
'job_create': 'create a drafted job',
'job_submit': 'submit a drafted job to become a queued job',
'job_submit_all': 'submit all drafted jobs to become a queued jobs',
'job_delete': 'delete a drafted, queued, failed or completed job',
'job_delete_all': 'delete all drafted, queued, failed and completed jobs',
'job_add_step': 'add a step to a drafted job',
'job_remix_step': 'remix a previous step from a drafted job',
'job_insert_step': 'insert a step to a drafted job',
'job_remove_step': 'remove a step from a drafted job',
'job_run': 'run a queued job',
'job_run_all': 'run all queued jobs',
'job_retry': 'retry a failed job',
'job_retry_all': 'retry all failed jobs'
},
'about':
{
'fund': 'fund ai workstation',
'subscribe': 'become a member',
'join': 'join our community'
},
'uis':
{
'apply_button': 'APPLY',
'benchmark_mode_dropdown': 'BENCHMARK MODE',
'benchmark_cycle_count_slider': 'BENCHMARK CYCLE COUNT',
'benchmark_resolutions_checkbox_group': 'BENCHMARK RESOLUTIONS',
'clear_button': 'CLEAR',
'common_options_checkbox_group': 'OPTIONS',
'download_providers_checkbox_group': 'DOWNLOAD PROVIDERS',
'execution_providers_checkbox_group': 'EXECUTION PROVIDERS',
'execution_thread_count_slider': 'EXECUTION THREAD COUNT',
'face_detector_angles_checkbox_group': 'FACE DETECTOR ANGLES',
'face_detector_model_dropdown': 'FACE DETECTOR MODEL',
'face_detector_margin_slider': 'FACE DETECTOR MARGIN',
'face_detector_score_slider': 'FACE DETECTOR SCORE',
'face_detector_size_dropdown': 'FACE DETECTOR SIZE',
'face_landmarker_model_dropdown': 'FACE LANDMARKER MODEL',
'face_landmarker_score_slider': 'FACE LANDMARKER SCORE',
'face_mask_blur_slider': 'FACE MASK BLUR',
'face_mask_padding_bottom_slider': 'FACE MASK PADDING BOTTOM',
'face_mask_padding_left_slider': 'FACE MASK PADDING LEFT',
'face_mask_padding_right_slider': 'FACE MASK PADDING RIGHT',
'face_mask_padding_top_slider': 'FACE MASK PADDING TOP',
'face_mask_areas_checkbox_group': 'FACE MASK AREAS',
'face_mask_regions_checkbox_group': 'FACE MASK REGIONS',
'face_mask_types_checkbox_group': 'FACE MASK TYPES',
'face_selector_age_range_slider': 'FACE SELECTOR AGE',
'face_selector_gender_dropdown': 'FACE SELECTOR GENDER',
'face_selector_mode_dropdown': 'FACE SELECTOR MODE',
'face_selector_order_dropdown': 'FACE SELECTOR ORDER',
'face_selector_race_dropdown': 'FACE SELECTOR RACE',
'face_tracker_score_slider': 'FACE TRACKER SCORE',
'face_occluder_model_dropdown': 'FACE OCCLUDER MODEL',
'face_parser_model_dropdown': 'FACE PARSER MODEL',
'voice_extractor_model_dropdown': 'VOICE EXTRACTOR MODEL',
'job_list_status_checkbox_group': 'JOB STATUS',
'job_manager_job_action_dropdown': 'JOB_ACTION',
'job_manager_job_id_dropdown': 'JOB ID',
'job_manager_step_index_dropdown': 'STEP INDEX',
'job_runner_job_action_dropdown': 'JOB ACTION',
'job_runner_job_id_dropdown': 'JOB ID',
'log_level_dropdown': 'LOG LEVEL',
'output_audio_encoder_dropdown': 'OUTPUT AUDIO ENCODER',
'output_audio_quality_slider': 'OUTPUT AUDIO QUALITY',
'output_audio_volume_slider': 'OUTPUT AUDIO VOLUME',
'output_image_or_video': 'OUTPUT',
'output_image_quality_slider': 'OUTPUT IMAGE QUALITY',
'output_image_scale_slider': 'OUTPUT IMAGE SCALE',
'output_path_textbox': 'OUTPUT PATH',
'output_video_encoder_dropdown': 'OUTPUT VIDEO ENCODER',
'output_video_fps_slider': 'OUTPUT VIDEO FPS',
'output_video_preset_dropdown': 'OUTPUT VIDEO PRESET',
'output_video_quality_slider': 'OUTPUT VIDEO QUALITY',
'output_video_scale_slider': 'OUTPUT VIDEO SCALE',
'preview_frame_slider': 'PREVIEW FRAME',
'preview_image': 'PREVIEW',
'preview_mode_dropdown': 'PREVIEW MODE',
'preview_resolution_dropdown': 'PREVIEW RESOLUTION',
'processors_checkbox_group': 'PROCESSORS',
'reference_face_distance_slider': 'REFERENCE FACE DISTANCE',
'reference_face_gallery': 'REFERENCE FACE',
'refresh_button': 'REFRESH',
'source_file': 'SOURCE',
'start_button': 'START',
'stop_button': 'STOP',
'target_file': 'TARGET',
'temp_frame_format_dropdown': 'TEMP FRAME FORMAT',
'terminal_textbox': 'TERMINAL',
'trim_frame_slider': 'TRIM FRAME',
'video_memory_strategy_dropdown': 'VIDEO MEMORY STRATEGY',
'webcam_fps_slider': 'WEBCAM FPS',
'webcam_image': 'WEBCAM',
'webcam_device_id_dropdown': 'WEBCAM DEVICE ID',
'webcam_mode_radio': 'WEBCAM MODE',
'webcam_resolution_dropdown': 'WEBCAM RESOLUTION'
}
}
}
+6 -38
View File
@@ -1,9 +1,8 @@
from logging import Logger, basicConfig, getLogger from logging import Logger, basicConfig, getLogger
from typing import Tuple
import facefusion.choices import facefusion.choices
from facefusion.common_helper import get_first, get_last from facefusion.common_helper import get_first, get_last
from facefusion.typing import LogLevel, TableContents, TableHeaders from facefusion.types import LogLevel
def init(log_level : LogLevel) -> None: def init(log_level : LogLevel) -> None:
@@ -32,46 +31,15 @@ def error(message : str, module_name : str) -> None:
def create_message(message : str, module_name : str) -> str: def create_message(message : str, module_name : str) -> str:
scopes = module_name.split('.') module_names = module_name.split('.')
first_scope = get_first(scopes) first_module_name = get_first(module_names)
last_scope = get_last(scopes) last_module_name = get_last(module_names)
if first_scope and last_scope: if first_module_name and last_module_name:
return '[' + first_scope.upper() + '.' + last_scope.upper() + '] ' + message return '[' + first_module_name.upper() + '.' + last_module_name.upper() + '] ' + message
return message return message
def table(headers : TableHeaders, contents : TableContents) -> None:
package_logger = get_package_logger()
table_column, table_separator = create_table_parts(headers, contents)
package_logger.info(table_separator)
package_logger.info(table_column.format(*headers))
package_logger.info(table_separator)
for content in contents:
content = [ value if value else '' for value in content ]
package_logger.info(table_column.format(*content))
package_logger.info(table_separator)
def create_table_parts(headers : TableHeaders, contents : TableContents) -> Tuple[str, str]:
column_parts = []
separator_parts = []
widths = [ len(header) for header in headers ]
for content in contents:
for index, value in enumerate(content):
widths[index] = max(widths[index], len(str(value)))
for width in widths:
column_parts.append('{:<' + str(width) + '}')
separator_parts.append('-' * width)
return '| ' + ' | '.join(column_parts) + ' |', '+-' + '-+-'.join(separator_parts) + '-+'
def enable() -> None: def enable() -> None:
get_package_logger().disabled = False get_package_logger().disabled = False
+17
View File
@@ -0,0 +1,17 @@
from typing import Optional, Tuple
def restrict_trim_frame(frame_total : int, trim_frame_start : Optional[int], trim_frame_end : Optional[int]) -> Tuple[int, int]:
if isinstance(trim_frame_start, int):
trim_frame_start = max(0, min(trim_frame_start, frame_total))
if isinstance(trim_frame_end, int):
trim_frame_end = max(0, min(trim_frame_end, frame_total))
if isinstance(trim_frame_start, int) and isinstance(trim_frame_end, int):
return trim_frame_start, trim_frame_end
if isinstance(trim_frame_start, int):
return trim_frame_start, frame_total
if isinstance(trim_frame_end, int):
return 0, trim_frame_end
return 0, frame_total
-21
View File
@@ -1,21 +0,0 @@
from facefusion.common_helper import is_macos, is_windows
if is_windows():
import ctypes
else:
import resource
def limit_system_memory(system_memory_limit : int = 1) -> bool:
if is_macos():
system_memory_limit = system_memory_limit * (1024 ** 6)
else:
system_memory_limit = system_memory_limit * (1024 ** 3)
try:
if is_windows():
ctypes.windll.kernel32.SetProcessWorkingSetSize(-1, ctypes.c_size_t(system_memory_limit), ctypes.c_size_t(system_memory_limit)) #type:ignore[attr-defined]
else:
resource.setrlimit(resource.RLIMIT_DATA, (system_memory_limit, system_memory_limit))
return True
except Exception:
return False
+3 -5
View File
@@ -4,14 +4,12 @@ METADATA =\
{ {
'name': 'FaceFusion', 'name': 'FaceFusion',
'description': 'Industry leading face manipulation platform', 'description': 'Industry leading face manipulation platform',
'version': '3.1.0', 'version': 'v4',
'license': 'MIT', 'license': 'OpenRAIL-AS',
'author': 'Henry Ruhs', 'author': 'Henry Ruhs',
'url': 'https://facefusion.io' 'url': 'https://facefusion.io'
} }
def get(key : str) -> Optional[str]: def get(key : str) -> Optional[str]:
if key in METADATA: return METADATA.get(key)
return METADATA.get(key)
return None
+2 -2
View File
@@ -2,10 +2,10 @@ from functools import lru_cache
import onnx import onnx
from facefusion.typing import ModelInitializer from facefusion.types import ModelInitializer
@lru_cache(maxsize = None) @lru_cache()
def get_static_model_initializer(model_path : str) -> ModelInitializer: def get_static_model_initializer(model_path : str) -> ModelInitializer:
model = onnx.load(model_path) model = onnx.load(model_path)
return onnx.numpy_helper.to_array(model.graph.initializer[-1]) return onnx.numpy_helper.to_array(model.graph.initializer[-1])
+22 -10
View File
@@ -1,17 +1,29 @@
from typing import List, Optional from typing import List, Optional
from facefusion.typing import Fps, Padding from facefusion.types import Color, Fps, Padding
def normalize_padding(padding : Optional[List[int]]) -> Optional[Padding]: def normalize_color(channels : Optional[List[int]]) -> Optional[Color]:
if padding and len(padding) == 1: if channels and len(channels) == 1:
return tuple([ padding[0] ] * 4) #type:ignore[return-value] return tuple([ channels[0], channels[0], channels[0], 255 ]) #type:ignore[return-value]
if padding and len(padding) == 2: if channels and len(channels) == 2:
return tuple([ padding[0], padding[1], padding[0], padding[1] ]) #type:ignore[return-value] return tuple([ channels[0], channels[1], channels[0], 255 ]) #type:ignore[return-value]
if padding and len(padding) == 3: if channels and len(channels) == 3:
return tuple([ padding[0], padding[1], padding[2], padding[1] ]) #type:ignore[return-value] return tuple([ channels[0], channels[1], channels[2], 255 ]) #type:ignore[return-value]
if padding and len(padding) == 4: if channels and len(channels) == 4:
return tuple(padding) #type:ignore[return-value] return tuple(channels) #type:ignore[return-value]
return None
def normalize_space(spaces : Optional[List[int]]) -> Optional[Padding]:
if spaces and len(spaces) == 1:
return tuple([spaces[0]] * 4) #type:ignore[return-value]
if spaces and len(spaces) == 2:
return tuple([ spaces[0], spaces[1], spaces[0], spaces[1] ]) #type:ignore[return-value]
if spaces and len(spaces) == 3:
return tuple([ spaces[0], spaces[1], spaces[2], spaces[1] ]) #type:ignore[return-value]
if spaces and len(spaces) == 4:
return tuple(spaces) #type:ignore[return-value]
return None return None
+1 -9
View File
@@ -1,6 +1,4 @@
from typing import Generator, List from facefusion.types import ProcessState
from facefusion.typing import ProcessState, QueuePayload
PROCESS_STATE : ProcessState = 'pending' PROCESS_STATE : ProcessState = 'pending'
@@ -45,9 +43,3 @@ def stop() -> None:
def end() -> None: def end() -> None:
set_process_state('pending') set_process_state('pending')
def manage(queue_payloads : List[QueuePayload]) -> Generator[QueuePayload, None, None]:
for query_payload in queue_payloads:
if is_processing():
yield query_payload
+7 -61
View File
@@ -1,15 +1,10 @@
import importlib import importlib
import os
from concurrent.futures import ThreadPoolExecutor, as_completed
from queue import Queue
from types import ModuleType from types import ModuleType
from typing import Any, List from typing import Any, List
from tqdm import tqdm from facefusion import logger, translator
from facefusion import logger, state_manager, wording
from facefusion.exit_helper import hard_exit from facefusion.exit_helper import hard_exit
from facefusion.typing import ProcessFrames, QueuePayload
PROCESSORS_METHODS =\ PROCESSORS_METHODS =\
[ [
@@ -17,29 +12,26 @@ PROCESSORS_METHODS =\
'clear_inference_pool', 'clear_inference_pool',
'register_args', 'register_args',
'apply_args', 'apply_args',
'get_common_modules',
'pre_check', 'pre_check',
'pre_process', 'pre_process',
'post_process', 'post_process',
'get_reference_frame', 'process_frame'
'process_frame',
'process_frames',
'process_image',
'process_video'
] ]
def load_processor_module(processor : str) -> Any: def load_processor_module(processor : str) -> Any:
try: try:
processor_module = importlib.import_module('facefusion.processors.modules.' + processor) processor_module = importlib.import_module('facefusion.processors.modules.' + processor + '.core')
for method_name in PROCESSORS_METHODS: for method_name in PROCESSORS_METHODS:
if not hasattr(processor_module, method_name): if not hasattr(processor_module, method_name):
raise NotImplementedError raise NotImplementedError
except ModuleNotFoundError as exception: except ModuleNotFoundError as exception:
logger.error(wording.get('processor_not_loaded').format(processor = processor), __name__) logger.error(translator.get('processor_not_loaded').format(processor = processor), __name__)
logger.debug(exception.msg, __name__) logger.debug(exception.msg, __name__)
hard_exit(1) hard_exit(1)
except NotImplementedError: except NotImplementedError:
logger.error(wording.get('processor_not_implemented').format(processor = processor), __name__) logger.error(translator.get('processor_not_implemented').format(processor = processor), __name__)
hard_exit(1) hard_exit(1)
return processor_module return processor_module
@@ -51,49 +43,3 @@ def get_processors_modules(processors : List[str]) -> List[ModuleType]:
processor_module = load_processor_module(processor) processor_module = load_processor_module(processor)
processor_modules.append(processor_module) processor_modules.append(processor_module)
return processor_modules return processor_modules
def multi_process_frames(source_paths : List[str], temp_frame_paths : List[str], process_frames : ProcessFrames) -> None:
queue_payloads = create_queue_payloads(temp_frame_paths)
with tqdm(total = len(queue_payloads), desc = wording.get('processing'), unit = 'frame', ascii = ' =', disable = state_manager.get_item('log_level') in [ 'warn', 'error' ]) as progress:
progress.set_postfix(execution_providers = state_manager.get_item('execution_providers'))
with ThreadPoolExecutor(max_workers = state_manager.get_item('execution_thread_count')) as executor:
futures = []
queue : Queue[QueuePayload] = create_queue(queue_payloads)
queue_per_future = max(len(queue_payloads) // state_manager.get_item('execution_thread_count') * state_manager.get_item('execution_queue_count'), 1)
while not queue.empty():
future = executor.submit(process_frames, source_paths, pick_queue(queue, queue_per_future), progress.update)
futures.append(future)
for future_done in as_completed(futures):
future_done.result()
def create_queue(queue_payloads : List[QueuePayload]) -> Queue[QueuePayload]:
queue : Queue[QueuePayload] = Queue()
for queue_payload in queue_payloads:
queue.put(queue_payload)
return queue
def pick_queue(queue : Queue[QueuePayload], queue_per_future : int) -> List[QueuePayload]:
queues = []
for _ in range(queue_per_future):
if not queue.empty():
queues.append(queue.get())
return queues
def create_queue_payloads(temp_frame_paths : List[str]) -> List[QueuePayload]:
queue_payloads = []
temp_frame_paths = sorted(temp_frame_paths, key = os.path.basename)
for frame_number, frame_path in enumerate(temp_frame_paths):
frame_payload : QueuePayload =\
{
'frame_number': frame_number,
'frame_path': frame_path
}
queue_payloads.append(frame_payload)
return queue_payloads
+4 -4
View File
@@ -3,7 +3,7 @@ from typing import Tuple
import numpy import numpy
import scipy import scipy
from facefusion.processors.typing import LivePortraitExpression, LivePortraitPitch, LivePortraitRoll, LivePortraitRotation, LivePortraitYaw from facefusion.processors.types import LivePortraitExpression, LivePortraitPitch, LivePortraitRoll, LivePortraitRotation, LivePortraitYaw
EXPRESSION_MIN = numpy.array( EXPRESSION_MIN = numpy.array(
[ [
@@ -63,15 +63,15 @@ def limit_expression(expression : LivePortraitExpression) -> LivePortraitExpress
return numpy.clip(expression, EXPRESSION_MIN, EXPRESSION_MAX) return numpy.clip(expression, EXPRESSION_MIN, EXPRESSION_MAX)
def limit_euler_angles(target_pitch : LivePortraitPitch, target_yaw : LivePortraitYaw, target_roll : LivePortraitRoll, output_pitch : LivePortraitPitch, output_yaw : LivePortraitYaw, output_roll : LivePortraitRoll) -> Tuple[LivePortraitPitch, LivePortraitYaw, LivePortraitRoll]: def limit_angle(target_pitch : LivePortraitPitch, target_yaw : LivePortraitYaw, target_roll : LivePortraitRoll, output_pitch : LivePortraitPitch, output_yaw : LivePortraitYaw, output_roll : LivePortraitRoll) -> Tuple[LivePortraitPitch, LivePortraitYaw, LivePortraitRoll]:
pitch_min, pitch_max, yaw_min, yaw_max, roll_min, roll_max = calc_euler_limits(target_pitch, target_yaw, target_roll) pitch_min, pitch_max, yaw_min, yaw_max, roll_min, roll_max = calculate_euler_limits(target_pitch, target_yaw, target_roll)
output_pitch = numpy.clip(output_pitch, pitch_min, pitch_max) output_pitch = numpy.clip(output_pitch, pitch_min, pitch_max)
output_yaw = numpy.clip(output_yaw, yaw_min, yaw_max) output_yaw = numpy.clip(output_yaw, yaw_min, yaw_max)
output_roll = numpy.clip(output_roll, roll_min, roll_max) output_roll = numpy.clip(output_roll, roll_min, roll_max)
return output_pitch, output_yaw, output_roll return output_pitch, output_yaw, output_roll
def calc_euler_limits(pitch : LivePortraitPitch, yaw : LivePortraitYaw, roll : LivePortraitRoll) -> Tuple[float, float, float, float, float, float]: def calculate_euler_limits(pitch : LivePortraitPitch, yaw : LivePortraitYaw, roll : LivePortraitRoll) -> Tuple[float, float, float, float, float, float]:
pitch_min = -30.0 pitch_min = -30.0
pitch_max = 30.0 pitch_max = 30.0
yaw_min = -60.0 yaw_min = -60.0
@@ -1,250 +0,0 @@
from argparse import ArgumentParser
from functools import lru_cache
from typing import List
import cv2
import numpy
import facefusion.choices
import facefusion.jobs.job_manager
import facefusion.jobs.job_store
import facefusion.processors.core as processors
from facefusion import config, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, inference_manager, logger, process_manager, state_manager, wording
from facefusion.common_helper import create_int_metavar
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url
from facefusion.execution import has_execution_provider
from facefusion.face_analyser import get_many_faces, get_one_face
from facefusion.face_helper import merge_matrix, paste_back, scale_face_landmark_5, warp_face_by_face_landmark_5
from facefusion.face_masker import create_occlusion_mask, create_static_box_mask
from facefusion.face_selector import find_similar_faces, sort_and_filter_faces
from facefusion.face_store import get_reference_faces
from facefusion.filesystem import in_directory, is_image, is_video, resolve_relative_path, same_file_extension
from facefusion.processors import choices as processors_choices
from facefusion.processors.typing import AgeModifierDirection, AgeModifierInputs
from facefusion.program_helper import find_argument_group
from facefusion.thread_helper import thread_semaphore
from facefusion.typing import ApplyStateItem, Args, DownloadScope, Face, InferencePool, ModelOptions, ModelSet, ProcessMode, QueuePayload, UpdateProgress, VisionFrame
from facefusion.vision import match_frame_color, read_image, read_static_image, write_image
@lru_cache(maxsize = None)
def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
return\
{
'styleganex_age':
{
'hashes':
{
'age_modifier':
{
'url': resolve_download_url('models-3.1.0', 'styleganex_age.hash'),
'path': resolve_relative_path('../.assets/models/styleganex_age.hash')
}
},
'sources':
{
'age_modifier':
{
'url': resolve_download_url('models-3.1.0', 'styleganex_age.onnx'),
'path': resolve_relative_path('../.assets/models/styleganex_age.onnx')
}
},
'templates':
{
'target': 'ffhq_512',
'target_with_background': 'styleganex_384'
},
'sizes':
{
'target': (256, 256),
'target_with_background': (384, 384)
}
}
}
def get_inference_pool() -> InferencePool:
model_sources = get_model_options().get('sources')
return inference_manager.get_inference_pool(__name__, model_sources)
def clear_inference_pool() -> None:
inference_manager.clear_inference_pool(__name__)
def get_model_options() -> ModelOptions:
age_modifier_model = state_manager.get_item('age_modifier_model')
return create_static_model_set('full').get(age_modifier_model)
def register_args(program : ArgumentParser) -> None:
group_processors = find_argument_group(program, 'processors')
if group_processors:
group_processors.add_argument('--age-modifier-model', help = wording.get('help.age_modifier_model'), default = config.get_str_value('processors.age_modifier_model', 'styleganex_age'), choices = processors_choices.age_modifier_models)
group_processors.add_argument('--age-modifier-direction', help = wording.get('help.age_modifier_direction'), type = int, default = config.get_int_value('processors.age_modifier_direction', '0'), choices = processors_choices.age_modifier_direction_range, metavar = create_int_metavar(processors_choices.age_modifier_direction_range))
facefusion.jobs.job_store.register_step_keys([ 'age_modifier_model', 'age_modifier_direction' ])
def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('age_modifier_model', args.get('age_modifier_model'))
apply_state_item('age_modifier_direction', args.get('age_modifier_direction'))
def pre_check() -> bool:
model_hashes = get_model_options().get('hashes')
model_sources = get_model_options().get('sources')
return conditional_download_hashes(model_hashes) and conditional_download_sources(model_sources)
def pre_process(mode : ProcessMode) -> bool:
if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')):
logger.error(wording.get('choose_image_or_video_target') + wording.get('exclamation_mark'), __name__)
return False
if mode == 'output' and not in_directory(state_manager.get_item('output_path')):
logger.error(wording.get('specify_image_or_video_output') + wording.get('exclamation_mark'), __name__)
return False
if mode == 'output' and not same_file_extension([ state_manager.get_item('target_path'), state_manager.get_item('output_path') ]):
logger.error(wording.get('match_target_and_output_extension') + wording.get('exclamation_mark'), __name__)
return False
return True
def post_process() -> None:
read_static_image.cache_clear()
if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]:
clear_inference_pool()
if state_manager.get_item('video_memory_strategy') == 'strict':
content_analyser.clear_inference_pool()
face_classifier.clear_inference_pool()
face_detector.clear_inference_pool()
face_landmarker.clear_inference_pool()
face_masker.clear_inference_pool()
face_recognizer.clear_inference_pool()
def modify_age(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame:
model_templates = get_model_options().get('templates')
model_sizes = get_model_options().get('sizes')
face_landmark_5 = target_face.landmark_set.get('5/68').copy()
crop_vision_frame, affine_matrix = warp_face_by_face_landmark_5(temp_vision_frame, face_landmark_5, model_templates.get('target'), model_sizes.get('target'))
extend_face_landmark_5 = scale_face_landmark_5(face_landmark_5, 0.875)
extend_vision_frame, extend_affine_matrix = warp_face_by_face_landmark_5(temp_vision_frame, extend_face_landmark_5, model_templates.get('target_with_background'), model_sizes.get('target_with_background'))
extend_vision_frame_raw = extend_vision_frame.copy()
box_mask = create_static_box_mask(model_sizes.get('target_with_background'), state_manager.get_item('face_mask_blur'), (0, 0, 0, 0))
crop_masks =\
[
box_mask
]
if 'occlusion' in state_manager.get_item('face_mask_types'):
occlusion_mask = create_occlusion_mask(crop_vision_frame)
combined_matrix = merge_matrix([ extend_affine_matrix, cv2.invertAffineTransform(affine_matrix) ])
occlusion_mask = cv2.warpAffine(occlusion_mask, combined_matrix, model_sizes.get('target_with_background'))
crop_masks.append(occlusion_mask)
crop_vision_frame = prepare_vision_frame(crop_vision_frame)
extend_vision_frame = prepare_vision_frame(extend_vision_frame)
age_modifier_direction = numpy.array(numpy.interp(state_manager.get_item('age_modifier_direction'), [-100, 100], [2.5, -2.5])).astype(numpy.float32)
extend_vision_frame = forward(crop_vision_frame, extend_vision_frame, age_modifier_direction)
extend_vision_frame = normalize_extend_frame(extend_vision_frame)
extend_vision_frame = match_frame_color(extend_vision_frame_raw, extend_vision_frame)
extend_affine_matrix *= (model_sizes.get('target')[0] * 4) / model_sizes.get('target_with_background')[0]
crop_mask = numpy.minimum.reduce(crop_masks).clip(0, 1)
crop_mask = cv2.resize(crop_mask, (model_sizes.get('target')[0] * 4, model_sizes.get('target')[1] * 4))
paste_vision_frame = paste_back(temp_vision_frame, extend_vision_frame, crop_mask, extend_affine_matrix)
return paste_vision_frame
def forward(crop_vision_frame : VisionFrame, extend_vision_frame : VisionFrame, age_modifier_direction : AgeModifierDirection) -> VisionFrame:
age_modifier = get_inference_pool().get('age_modifier')
age_modifier_inputs = {}
if has_execution_provider('coreml'):
age_modifier.set_providers([ facefusion.choices.execution_provider_set.get('cpu') ])
for age_modifier_input in age_modifier.get_inputs():
if age_modifier_input.name == 'target':
age_modifier_inputs[age_modifier_input.name] = crop_vision_frame
if age_modifier_input.name == 'target_with_background':
age_modifier_inputs[age_modifier_input.name] = extend_vision_frame
if age_modifier_input.name == 'direction':
age_modifier_inputs[age_modifier_input.name] = age_modifier_direction
with thread_semaphore():
crop_vision_frame = age_modifier.run(None, age_modifier_inputs)[0][0]
return crop_vision_frame
def prepare_vision_frame(vision_frame : VisionFrame) -> VisionFrame:
vision_frame = vision_frame[:, :, ::-1] / 255.0
vision_frame = (vision_frame - 0.5) / 0.5
vision_frame = numpy.expand_dims(vision_frame.transpose(2, 0, 1), axis = 0).astype(numpy.float32)
return vision_frame
def normalize_extend_frame(extend_vision_frame : VisionFrame) -> VisionFrame:
model_sizes = get_model_options().get('sizes')
extend_vision_frame = numpy.clip(extend_vision_frame, -1, 1)
extend_vision_frame = (extend_vision_frame + 1) / 2
extend_vision_frame = extend_vision_frame.transpose(1, 2, 0).clip(0, 255)
extend_vision_frame = (extend_vision_frame * 255.0)
extend_vision_frame = extend_vision_frame.astype(numpy.uint8)[:, :, ::-1]
extend_vision_frame = cv2.resize(extend_vision_frame, (model_sizes.get('target')[0] * 4, model_sizes.get('target')[1] * 4), interpolation = cv2.INTER_AREA)
return extend_vision_frame
def get_reference_frame(source_face : Face, target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame:
return modify_age(target_face, temp_vision_frame)
def process_frame(inputs : AgeModifierInputs) -> VisionFrame:
reference_faces = inputs.get('reference_faces')
target_vision_frame = inputs.get('target_vision_frame')
many_faces = sort_and_filter_faces(get_many_faces([ target_vision_frame ]))
if state_manager.get_item('face_selector_mode') == 'many':
if many_faces:
for target_face in many_faces:
target_vision_frame = modify_age(target_face, target_vision_frame)
if state_manager.get_item('face_selector_mode') == 'one':
target_face = get_one_face(many_faces)
if target_face:
target_vision_frame = modify_age(target_face, target_vision_frame)
if state_manager.get_item('face_selector_mode') == 'reference':
similar_faces = find_similar_faces(many_faces, reference_faces, state_manager.get_item('reference_face_distance'))
if similar_faces:
for similar_face in similar_faces:
target_vision_frame = modify_age(similar_face, target_vision_frame)
return target_vision_frame
def process_frames(source_path : List[str], queue_payloads : List[QueuePayload], update_progress : UpdateProgress) -> None:
reference_faces = get_reference_faces() if 'reference' in state_manager.get_item('face_selector_mode') else None
for queue_payload in process_manager.manage(queue_payloads):
target_vision_path = queue_payload['frame_path']
target_vision_frame = read_image(target_vision_path)
output_vision_frame = process_frame(
{
'reference_faces': reference_faces,
'target_vision_frame': target_vision_frame
})
write_image(target_vision_path, output_vision_frame)
update_progress(1)
def process_image(source_path : str, target_path : str, output_path : str) -> None:
reference_faces = get_reference_faces() if 'reference' in state_manager.get_item('face_selector_mode') else None
target_vision_frame = read_static_image(target_path)
output_vision_frame = process_frame(
{
'reference_faces': reference_faces,
'target_vision_frame': target_vision_frame
})
write_image(output_path, output_vision_frame)
def process_video(source_paths : List[str], temp_frame_paths : List[str]) -> None:
processors.multi_process_frames(None, temp_frame_paths, process_frames)

Some files were not shown because too many files have changed in this diff Show More