Compare commits

...
Author SHA1 Message Date
henryruhs b6a1adb7a5 add groups to capabilities 2026-07-02 20:39:09 +02:00
henryruhs 4895615334 keep v4 style 2026-07-01 17:21:33 +02:00
henryruhs df2bda6145 merge master into v4 - post adjustments 2026-07-01 17:10:31 +02:00
henryruhs 942d435245 merge master into v4 - post adjustments 2026-07-01 16:02:19 +02:00
henryruhs 5d0b2afbe9 merge master into v4 - post adjustments 2026-07-01 14:44:53 +02:00
henryruhs e0921ba4b5 merge master into v4 - post adjustments 2026-07-01 14:29:55 +02:00
henryruhs 0b976f80a8 merge master into v4 - post adjustments 2026-07-01 14:28:46 +02:00
henryruhs bd1448805c merge master into v4 - post adjustments 2026-07-01 13:36:59 +02:00
henryruhs 8a9c596fde merge master into v4 - post adjustments 2026-07-01 13:34:54 +02:00
300470e7c7 port/master-into-v4 (#1176)
* 3.7.0 (#1175)

* mark as next, introduce dynamic scale for face debugger

* use latest onnxruntime

* update within Gradio 5

* Remove system memory limit (#986)

* remove system memory limit from ui

* remove system memory limit from args.py

* flatten the face store

* prevent countless importlib.import_module calls

* remove --onnxruntime from install.py

* remove --onnxruntime from install.py

* resolve static inference providers to fix macos (#1127)

* resolve static inference providers to fix macos

* fix lint

* restore old behaviour

* restore old behaviour

* handle ghost and uniface as well

* adjust condition for ghost and uniface

* fix Gradio gallery styles

* remove face store (#1132)

* fix dataflow in streamer

* Face selector auto mode (#1137)

* introduce face selector auto mode

* introduce face selector auto mode

* introduce face selector auto mode

* correct way is to pass source_vision_frames

* make the world a better place

* fix dataflow in faceswapper, no read of files withing inner methods (#1148)

* fix dataflow in faceswapper, no read of files withing inner methods

* fix lint

* adjust code more

* adjust code more

* bring back the face store but for source and reference only (#1149)

* bring back the face store but for source and reference only

* fix ci

* minor improvement

* guard for tobytes()

* drop condition in select_faces()

* Replace CONFIG_PARSER global with @lru_cache (#1147)

* remove global config_parser

* fix import order

* remove lambda

* remove unused block

* optimize app context detection

* decouple common modules from core (#1152)

* decouple common modules from core

* remove that nonsense

* remove that nonsense

* minor adjustment to workflows

* Tag HEVC output as hvc1 and move moov atom to the front (#1153)

* Tag HEVC output as hvc1 and move moov atom to the front

ffmpeg defaults HEVC in MP4 to the 'hev1' sample entry and leaves the moov
atom at the tail. Apple players (QuickTime, Finder QuickLook) refuse to decode
'hev1' and stall reading a tail-placed moov on large files, so hevc_nvenc /
libx265 renders cannot be previewed on macOS.

- add ffmpeg_builder.set_video_tag(): emit `-tag:v hvc1` for every HEVC
  encoder (libx265, hevc_nvenc, hevc_amf, hevc_qsv, hevc_videotoolbox).
  Applied in merge_video where the encoder is known; `-c:v copy` in the audio
  mux / concat steps preserves the tag.
- add ffmpeg_builder.set_faststart(): emit `-movflags +faststart`, applied in
  restore_audio / replace_audio / concat_video which write the final output.

H.264 and other codecs are left untouched. Verified on a real hevc_nvenc
render: hev1 hung QuickLook (no thumbnail); after the patch the file is hvc1
with a front-placed moov and QuickLook generates a thumbnail.

* Restrict hvc1 tag and faststart to quicktime containers

Gate set_video_tag / set_faststart on the output container format
(m4v, mov, mp4) via get_file_format(), so non-quicktime muxers no longer
receive -tag:v hvc1 / -movflags +faststart. Trim test_set_video_tag to a
single positive and negative assertion.

Addresses review on #1153.

* Move hvc1 tag and faststart gates into ffmpeg_builder

Rename set_video_tag / set_faststart to conditional_* and push the
container-format gate (m4v, mov, mp4) inside the builders, keeping
ffmpeg.py free of inline conditionals. Matches the set_image_quality
pattern. Addresses review on #1153.

* post cleanup after merge

* Pack target frames (#1158)

* pack target frames

* add todos

* add todos, resolve todos

* resolve todos

* change names

* revert to single target frame for select faces

* fix lint

* return empty frame

* get() have no default

* Fix trim (#1162)

* fix trim

* fix trim

* rename ffmpeg builder method

* rename to temp_frame_set and temp_frame_pattern

---------

Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>
Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>

* Implement face tracker (#1163)

* add face tracker

* change get_nearest_track_face -> get_nearest_track_index

* create face_creator.py and move methods around

* add type FaceTrack

* naming

* remove iou test, don't belong there

* fix spaces

* rename to interpolate_points

* rename to find_best_face_track

* just track_faces

* cleanp

* previous next naming

* remove >= and >=

* rename

* remove helper from test and use face from source.jpg

* make get_anchor_indices more readable

* track_faces() call before and is forwarded to select_faces

* change to interpolate_faces

* rename methods

* rename methods

* rename variables

* remove dtype

* move face_anlyser -> face_creator

* claenup face_creator.py

* move tests to dedicated test face detector

* move tracking inside select_faces

* simplify face_tracker (#1165)

* minor renaming

* improve face_tracker test (#1166)

* improve face_tracker test

* cleanup

* Add target frame amount (#1167)

* introduce --target-frame-amount

* add ui

* make track_faces conditional

* update choices.py

* fix []

* rename component file to frame_process.py

* fix track preview (#1168)

* introduce face origin (#1169)

* add guard to prevent failure

* show and hide voice extractor according to lip syncer

* rename average_face_coordinates to average_face_geometry

* use static faces for select_faces()

* face store with lock (#1171)

* face store with lock

* face store with lock

* remove refill color from bbox

* adjust tests and handle frame_position proper way

* enforce similar naming

* introduce face tracker score

* introduce face tracker score

* fix/audio-trim-alignment (#1173)

* fix audio offset

* fix audio offset

* remove reference_frame_number check

---------

Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>

* reduce face tracker score from 0 to 0.5

* mark as 3.7.0

* make face tracker stateless

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>
Co-authored-by: kazuki nakai <kazuki.nakai@agiletec.net>
Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>

* update preview

* fix wording

* fix wording

* last minute change to frame distribution

---------

Co-authored-by: Harisreedhar <46858047+harisreedhar@users.noreply.github.com>
Co-authored-by: kazuki nakai <kazuki.nakai@agiletec.net>
Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>
2026-07-01 11:51:54 +02:00
Henry RuhsandGitHub 5f403ada65 remove opus frame size (#1160)
* remove opus frame size

* improve naming, follow audio.py
2026-06-15 19:19:51 +02:00
henryruhs c6aed91698 use dynamic audio frame size 2026-06-15 17:14:24 +02:00
Henry RuhsandGitHub 2854bf1db0 Resolve final stream TODOs (#1155)
* refactor a lot

* refactor a lot

* fix test

* fix test
2026-06-15 11:37:25 +02:00
Henry RuhsandGitHub d5271d21a1 minor adjustment for workflow setup and clear (#1157)
* minor adjustment for workflow setup and clear

* minor adjustment for workflow setup and clear
2026-06-14 22:57:01 +02:00
Henry RuhsandGitHub e7d22e84bf use Buffer and BufferPack everywhere (#1154) 2026-06-11 13:03:29 +02:00
henryruhs 57189c638e optimize app context detection 2026-06-09 22:30:37 +02:00
Henry RuhsandGitHub 00fb89d4f1 Best performance to code ratio for stream (#1150)
* queue with futures, kill deque, add couple of todos

* resolve couple of todos

* add more todos

* add more todos and resolve others

* add more todos and resolve others

* fix test

* fix collapse

* adjust naming a bit
2026-06-09 22:16:51 +02:00
henryruhs 67cc3de934 just ignore type 2026-06-06 15:31:14 +02:00
henryruhs accad0a2cb fix ci 2026-06-06 13:53:30 +02:00
henryruhs 46d1575c9b fix ci 2026-06-06 13:41:28 +02:00
henryruhs f3bc5ffb4b changes for vp9 support 2026-06-06 12:39:14 +02:00
henryruhs 5287ce6dcb add todos 2026-06-06 11:45:38 +02:00
henryruhs 5d6258e17e add vp9 support 2026-06-06 11:39:53 +02:00
henryruhs 036c5c0225 add vp9 support 2026-06-06 11:15:00 +02:00
henryruhs 2e884941f8 resolve todos 2026-06-06 08:36:20 +02:00
henryruhs 0ef5de1d02 guard for tobytes()
(cherry picked from commit 0cbfa5a415)
2026-06-05 21:47:57 +02:00
henryruhs dcb9a7a59b minor improvement
(cherry picked from commit fd3e8b82e2)
2026-06-05 21:47:57 +02:00
henryruhs 3f6738dd1d fix ci
(cherry picked from commit fbfb21df3b)
2026-06-05 21:47:57 +02:00
henryruhs 19f8f5f206 bring back the face store but for source and reference only
(cherry picked from commit 9d8a5be714)
2026-06-05 21:47:57 +02:00
775985645e Push based receive with queue (#1146)
* move to push based receive

* move to push based receive, fix mocks

* fix tests

* add todos

* remove asyncio

* remove asyncio

* resolve todos

* move to queue without events

* prevent debug spam

* concurrent stream inference

stream_video.py: pipeline face-swap inference across execution_thread_count workers (ThreadPoolExecutor + bounded in-flight deque, ordered encode) to keep the GPU busy during encode

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* add todos

* add todos

* add missing state

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-05 00:18:57 +02:00
henryruhs 1f494f54db fix hashes 2026-06-03 10:43:16 +02:00
henryruhs 476a21cc7a fix hashes 2026-06-03 10:43:09 +02:00
henryruhs 121c3a5af6 update hashes 2026-06-03 10:36:00 +02:00
Henry RuhsandGitHub 2ac9b70550 Feat/remb both direction (#1145)
* revamp remb to take both directions

* extract more methods
2026-06-03 10:31:33 +02:00
henryruhs d9553b12e8 remove copy of buffer 2026-06-02 17:10:25 +02:00
henryruhs 3bda73699c fix fast seeking 2026-06-02 16:29:58 +02:00
Henry RuhsandGitHub 7181b41f2d Feat/finalize stream (#1144)
* break stream helper into pieces

* remove todos
2026-06-02 16:05:21 +02:00
harisreedhar 314ee61826 add create_event method 2026-06-02 03:06:16 +05:30
HarisreedharandGitHub beeb1d99e9 Refactor(stream-helper): split encode/receive loops and unify audio/video structure (#1142)
* split encode loop

* unify audio and video encode loop methods

* improve variable names

* split receive methods

* test improve

* try to avoid != with even more weird approach

* remove source_path check

* remove empty variables

* fix lint

* avoid not in condition

* rename

* fix lint
2026-06-02 01:50:40 +05:30
Henry RuhsandGitHub e2e4e6a95b Refactor/stream helper testing (#1141)
* fix tests part1

* improve testing

* use real audio

* more renaming

* switch to hash assertions

* switch to hash assertions

* switch to hash assertions

* switch to hash assertions

* skip on macos
2026-06-01 18:42:37 +02:00
3eeb505c86 Refactor(stream-helper): pass peer context objects to receivers, fix codec guard, move YUV conversion into decoders (#1140)
* extract numpy.empty(0) into an empty_vision_frame variable so the sentinel intent is clear

* bring back todos

* bring back todos

* rename opus_buffer to audio

* move reshape and cvtColor into decoder modules

* add audio_codec check

* add audio_codec check

* rename audio video buffer

* simplify receive methods and cleanup test

* todo

* revert decode methods to return pointer

* remove duplicate test_stream_helper — tests live in test_api_stream_helper

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-01 17:21:06 +05:30
harisreedhar 02f684e220 bring back todos 2026-06-01 15:48:10 +05:30
harisreedhar a88d8ead49 bring back todos 2026-06-01 15:45:44 +05:30
harisreedhar 7735c3740f bring back todos 2026-06-01 15:19:28 +05:30
henryruhs f2de9f1843 rename to video pack and audio pack 2026-06-01 11:41:25 +02:00
HarisreedharandGitHub aea36a9e55 Replace queues with timestamped deques and fix Audio Video sync (#1139)
* change to deque

* remove single line condition

* change threading to asyncio

* bringback todos
2026-06-01 15:03:33 +05:30
Henry RuhsandGitHub 0f5f75ba51 Cleanup/testing suite (#1136)
* clean testing suite

* clean testing suite part2

* clean testing suite part3

* add todos

* extend testing suite and kill some mutants

* fix hashes

* fix lint

* fix test

* fix test
2026-06-01 08:54:37 +02:00
Henry RuhsandGitHub 460c65004b Add available event (#1134)
* remove sleep with available event

* reorder methods, caller has to follow variable names of consumer, reorder tests methods

* more todos for naming
2026-05-30 16:39:48 +05:30
Henry RuhsandGitHub 1ac0e3e9a4 remove face store (#1132) (#1133) 2026-05-30 11:55:58 +02:00
HarisreedharandGitHub 9f6a19c1d2 feat(rtc): 2-way REMB with in-place encoder bitrate update (#1131)
* implement remb other direction

* update test
2026-05-30 04:41:10 +05:30
henryruhs 3ff327e670 add todos 2026-05-29 18:17:51 +02:00
henryruhs 2553ad7cad add todos 2026-05-29 18:10:15 +02:00
HarisreedharandGitHub 6b9ddd9a4f feat(rtc): REMB bitrate adaptation with in-place encoder update (#1130)
* improve test

* fix lint

* cleanup

* cleanup
2026-05-29 21:15:03 +05:30
henryruhs 871559cb6a clean and simplify tests 2026-05-29 15:44:07 +02:00
HarisreedharandGitHub 2a8672b54d Implement tier 1 REMB (#1129)
* implement tier 1

* fix lint

* cleanup

* cleanup

* fix lint

* use clear_remb method

* use single callback

* improve test

* improve test

* improve test

* improve test

* improve test
2026-05-29 19:00:27 +05:30
henryruhs cc0af9175a ban byte string 2026-05-26 19:15:11 +02:00
harisreedhar d77be89177 improve encoder collect 2026-05-26 22:25:55 +05:30
henryruhs e813d7df95 improve performance for decoder collect 2026-05-26 18:29:12 +02:00
henryruhs 6eaabe123e remove --onnxruntime from install.py 2026-05-26 08:16:49 +02:00
henryruhs e2bac200d6 remove --onnxruntime from install.py 2026-05-26 08:14:48 +02:00
HarisreedharandGitHub 4fe79483ea Fix stream lifecycle bugs: threading, RTP sync, and resource cleanup (#1125)
* fix executor thread not terminating after stream deletion

* fix stream shutdown and thread lifecycle

* add todo

* cleanup

* cleanup

* cleanup

* cleanup

* audio_queue.put() → get_nowait() + put_nowait()

* rename test

* fix test

* merge tests

* cleanup tests

* cleanup tests

* cleanup tests

* simplify test logic with mock

* cleanup

* cleanup hard to read stream_helper.py

* introduce rtc_peer.has_peers

* fix lint

* add todos

* fix test hash
2026-05-22 16:29:45 +05:30
HarisreedharandGitHub 520dcbfd6b Refine stream helper: queue-based loops, AV1/WHIP support, endpoint separation (#1124)
* rearrange methods

* add test_stream_helper.py

* improve tests

* use deque

* move decoder to recieve methods

* remove cleanup_peer

* add destroy_stream

* make run_peer_loop more readable

* make video and audio method simlar

* change deque to queue to avoid extra thread event

* remove negative condition

* cleanup

* remove wait_for_frame

* cleanup

* cleanup

* fix process_image

* fix lint

* cleanup

* remove last_time

* add todos
2026-05-20 23:43:14 +05:30
Henry RuhsandGitHub 48869bedf0 Follow WHIP specs (#1123)
* follow more specs of whip

* pass the location header value via API router

* fix CI

* remove more queries
2026-05-19 17:05:53 +02:00
Henry RuhsandGitHub 927857d70d cleanup decoders tests (#1122)
* cleanup mostly decoders tests, also a bit encoders

* cleanup mostly decoders tests, also a bit encoders
2026-05-19 12:57:57 +02:00
Henry RuhsandGitHub fbacb24fcc Tiny refactor of codecs (#1121)
* improve performance using pointers

* simplify decoder's collect

* simplify decoder's collect

* add threading to decoders

* fix test

* switch back to return bytes

* fix macos
2026-05-19 10:31:53 +02:00
c00ea92f35 Migrate to WHIP (#1120)
* migrate to whip part1

* migrate to whip part2

* migrate to whip part3

* migrate to whip part4

* migrate to whip/whep with bidirectional

* migrate to whip/whep with bidirectional

* use next library

* add _next to lid datachannel files

* cleanup and add todos

* use internal helper rtcGetPayloadTypesForCodec

* fix lint

* refactor decode()

* move logic to codecs

* move logic to codecs

* break encoders and decoders into multiple files

* break encoders and decoders into multiple files

* cleanup more

* drop action for stream endpoints, keep type for self documentation

* restore the v4 store

* fix: align frame_width and frame_height to even in both collect() and read_resolution() in both decoders.

---------

Co-authored-by: harisreedhar <h4harisreedhar.s.s@gmail.com>
2026-05-18 16:16:06 +02:00
HarisreedharandGitHub c48c238f88 Combine encode loop methods (#1119)
* combine run_aom_encode_loop and run_vpx_encode_loop to encode_video_loop

* run_opus_encode_loop -> encode_audio_loop

* use else instead of continue

* rename to video_codec
2026-05-16 23:29:36 +05:30
Henry RuhsandGitHub dd1ded1408 Refactor/rtc cleanup 3 (#1118)
* tweak rtc store and make the decision to ban trivial testing

* clear todos for rtc_test, remove redundant tests

* clear todos for rtc_test, remove redundant tests

* break negotiation out of rtc flow, introduce create_sdp_answer and set_remote_description

* add todo

* move timeline control to the stream helper, clean send_audio|video_to_peers

* rename some methods

* fix test

* introduce detect_sdp_media

* introduce detect_sdp_media
2026-05-16 09:06:04 +02:00
Henry RuhsandGitHub 95435f842c use datachannel to create proper rtc_track_init (#1117)
* use datachannel to create proper rtc_track_init

* fix lint

* fix lint
2026-05-15 19:14:36 +02:00
HarisreedharandGitHub 0019d3ad0f Refactor stream_helper: queue-based audio/video loops with unified threading (#1116)
* rearrange methods following the flow

* add test_stream_helper.py

* fix lint

* fix lint

* refactor audio flow to match video by replacing dequeue with queue

* remove unused keyframe interval

* remove try block

* remove while True

* simplify run_aom_encode_loop and run_vp8_encode_loop

* cleanup names

* simplify run_opus_encode_loop

* move opus_encoder creation to run_opus_encode_loop

* add todos

* fix lint

* update todos and tests
2026-05-15 21:47:30 +05:30
Henry RuhsandGitHub 532464032b More RTC cleanup (#1115)
* reduce create_peer_connection like crazy

* turns out that we dont event and callback/while+sleep magic for create_sdp_offer and negotiate_sdp_answer

* test for audio/video to peers

* flag rtc methods to be revised

* fix lint
2026-05-15 15:10:51 +02:00
Henry RuhsandGitHub ad3b582c49 resolve todos in stream endpoint (#1114) 2026-05-15 12:40:50 +02:00
Henry RuhsandGitHub 98adce8a2b Refactor RTC structure (#1113)
* refactor rtc part1

* skip for macos

* merge create spd and create sdp offer

* fix lint

* add test for create_sdp_offer

* better naming for negotiate method as we get an answer

* extend tests based on mutations

* remove dead code

* rename rtc store and related methods

* clean store, move sender logic to stream helper under apis

* generate tests for rtc store
2026-05-15 11:46:51 +02:00
henryruhs 3def6c8fcd skip for macos 2026-05-15 08:47:28 +02:00
henryruhs 504f2240f7 make test more robust 2026-05-14 23:19:49 +02:00
henryruhs 061522e5b7 use bytes() over empty byte literals 2026-05-14 22:45:02 +02:00
henryruhs 37420eac7c use bytes() over empty byte literals 2026-05-14 22:44:56 +02:00
henryruhs 1562fe2fee kill the stream helper in tests 2026-05-14 22:41:54 +02:00
henryruhs dc74e1c783 event driven test_stream_video 2026-05-14 22:21:36 +02:00
henryruhs a097034889 skip test for macos 2026-05-14 19:32:14 +02:00
henryruhs d8d9d5a280 fix test for macos 2026-05-14 18:44:02 +02:00
henryruhs eeb342ce36 fix macos for aom encode 2026-05-14 17:20:18 +02:00
henryruhs b8d00d6389 get rid of aom obus stuff again 2026-05-14 16:44:09 +02:00
Henry RuhsandGitHub 18a487347a av1 support integrated (#1112) 2026-05-14 16:11:23 +02:00
Henry RuhsandGitHub b607e4a99e AV1 codec support (#1111)
* restructure xxx_encoders, introduce av1 codec

* get rid of strip_temporal_delimiters

* improve testing

* fix test for macos

* improve testing
2026-05-14 13:30:18 +02:00
Henry RuhsandGitHub b1bc0ea43c aom library for av1 support (#1110) 2026-05-14 11:41:28 +02:00
henryruhs 912d7eaa52 adjust create_string_buffer that could be measured 2026-05-13 15:31:54 +02:00
Henry RuhsandGitHub 832d954df6 use bytes over pointer for opus encoder (#1109)
* use bytes over pointer for opus encoder

* use bytes over pointer for opus encoder
2026-05-13 14:41:26 +02:00
henryruhs 78a068107a clean audio encoder too 2026-05-13 13:41:09 +02:00
Henry RuhsandGitHub 9e1c068938 improve naming, remove flags as not needed (#1108)
* improve naming, remove flags as not needed

* fix lint
2026-05-13 13:34:37 +02:00
HarisreedharandGitHub bff222a12f try to unify structure of encode_opus_buffer and encode_vpx_buffer (#1107) 2026-05-13 16:26:30 +05:30
Henry RuhsandGitHub 9453a042a1 hash based test for test_stream_image (#1104)
* hash based test for test_stream_image

* new todo for test_video_stream

* new todo for test_video_stream
2026-05-12 19:51:19 +05:30
5e39c60b5c Improve encoder tests with hash assertion (#1103)
* improve test_encode_opus_buffer

* try different hash per os

* fix lint

* add windows check

* update windows hash

* fix test and lint

* update windows hash

* update CI for test_video_encoder.py

* update hash for macos

* update method to use single cpu

* update mac hash

* update windows hash

* cleanup

* restore ci.yml

* remove argument defaults

* selected CI tests

* selected CI tests

* restore ci.yml

---------

Co-authored-by: henryruhs <info@henryruhs.com>
2026-05-12 16:36:06 +05:30
HarisreedharandGitHub 8690ccf49e rename methods (#1102) 2026-05-12 13:22:33 +05:30
Henry RuhsandGitHub e53cb63577 QA - Encoder Testing (#1101)
* testing for audio and video encoders, minor cleanups

* fix lint

* finish create_vpx_encoder, adjust unrelated order of width vs height args
2026-05-12 08:23:27 +02:00
henryruhs 717ff0aa33 restore todos 2026-05-11 21:23:50 +02:00
harisreedhar 4922be4ad3 fix audio 2026-05-11 22:30:00 +05:30
henryruhs fdf1b841b2 setup_platform is no longer needed, bring back conda.py 2026-05-11 17:42:51 +02:00
HarisreedharandGitHub 92296fc5a5 move run_video_encode_loop (#1100) 2026-05-11 21:04:56 +05:30
henryruhs 6968d8fe47 switch to final library repo 2026-05-11 16:36:23 +02:00
Henry Ruhs b6549e873b Refactor/windows libraries (#1099)
* ship libssl and libcrypto in windows

* ship libssl and libcrypto in windows
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs ab7110eb92 Add audio encoder (#1096)
* add audio encoder

* add todos

* add todos

* cleannup and add todos

* fix lint

---------

Co-authored-by: henryruhs <info@henryruhs.com>
2026-05-11 16:36:23 +02:00
Henry Ruhs 9321b41e8e ship libssl and libcrypto in macos (#1097) 2026-05-11 16:36:23 +02:00
henryruhs 450075e20e fix windows 2026-05-11 16:36:23 +02:00
Henry Ruhs 20b392f760 ship libssl and libcrypto in linux (#1095) 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs bb1b8ccf13 Add video_encoder.py (#1094)
* add video_encoder.py

* rename pts to presentation_timestamp

* improve test

* improve test

* cleanup

* cleanup

* fix lint
2026-05-11 16:36:23 +02:00
henryruhs 6eace4ce29 cleanup 2026-05-11 16:36:23 +02:00
henryruhs 6fe735618e cleanup 2026-05-11 16:36:23 +02:00
henryruhs 527bb0ff45 cleanup 2026-05-11 16:36:23 +02:00
henryruhs e05d13c47e cleanup 2026-05-11 16:36:23 +02:00
Henry Ruhs 8bbb6e7062 Cleanup/stream part1 (#1093)
* use find_library from cytypes, enable rtc tests again

* install libvpx and libopus for unix on CI

* bug found when sending audio first

* no need for source path guard

* add more tests for libraries

* add more tests for libraries

* fix testing

* disable tests to see what happens

* disable tests to see what happens

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* debug ci

* hope to solve everything via ENV

* hope to solve everything via ENV

* hope to solve everything via ENV

* hope to solve everything via ENV

* hope to solve everything via ENV

* hope to solve everything via ENV

* hope to solve everything via ENV

* hope to solve everything via ENV

* fix testing

* fix testing

* fix testing

* fix testing

* fix testing

* fix testing

* switch to self hosted libraries

* fixes for macos

* switch to self hosted libraries

* switch to self hosted libraries

* switch to self hosted libraries

* switch to self hosted libraries

* switch to self hosted libraries

* switch to self hosted libraries

* switch to self hosted libraries

* switch to self hosted libraries
2026-05-11 16:36:23 +02:00
Henry Ruhs 76c413a2c1 Refactor/ffmpeg less stream (#1092)
* remove ffmpeg from stream to use opus and vpx, add bunch of todos

* fix testing

* improve download checkout

* fix datachannel download, fix super dirty test clients - setup logic does not belong there

* fix testing
2026-05-11 16:36:23 +02:00
Henry Ruhs 430b16ce56 introduce opus and vpx to libraries (#1091) 2026-05-11 16:36:23 +02:00
henryruhs 9a390bd5bc more uniform codebase for libraries 2026-05-11 16:36:23 +02:00
Henry Ruhs 97e0df01b1 move datachannel to libraries and follow datachannel conventions (#1090)
* move datachannel to libraries and follow new datachannel_module convention

* move datachannel to libraries and follow new datachannel_module convention
2026-05-11 16:36:23 +02:00
Henry Ruhs a8db033033 move stream mode to query parameter (#1089) 2026-05-11 16:36:23 +02:00
henryruhs fda6b7f69d fix test 2026-05-11 16:36:23 +02:00
henryruhs 4198cca18f disable broken tests 2026-05-11 16:36:23 +02:00
henryruhs 9a2d57ae54 refactor detect_websocket_stream_mode and related tests a bit, disable broken tests 2026-05-11 16:36:23 +02:00
henryruhs fe002dc821 skip potential broken tests 2026-05-11 16:36:23 +02:00
henryruhs f8f5d6197d skip potential broken tests 2026-05-11 16:36:23 +02:00
henryruhs 2290e4ea57 giving up - set driver to 0.0.0 for amd 2026-05-11 16:36:23 +02:00
henryruhs 6ec9257e92 add driver version back for amd 2026-05-11 16:36:23 +02:00
henryruhs d727558900 load rocm version for dedicated library 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 07c1c936af Refine RTC bindings: callback-based SDP negotiation, peer state tracking, and type cleanup (#1088)
* Refine RTC bindings: callback-based SDP negotiation, peer state tracking, and type cleanup

* fix lint

* restore peer_connection and rename methods

* remove flags, unused_methods and improve tests

* fix indent
2026-05-11 16:36:23 +02:00
henryruhs 7322bd5d52 fix memory for amd 2026-05-11 16:36:23 +02:00
henryruhs 7f40516f71 use define_xxx for the type() factories 2026-05-11 16:36:23 +02:00
henryruhs ef8567bd3a adjust naming and fix version lookups 2026-05-11 16:36:23 +02:00
henryruhs f8c90b4b25 simplify memory call for amd 2026-05-11 16:36:23 +02:00
henryruhs 343b7f6aad simplify memory call for amd 2026-05-11 16:36:23 +02:00
henryruhs 9b878431d3 rename video memory to memory, add amdsmi library 2026-05-11 16:36:23 +02:00
henryruhs 74a9df35a6 scope the nvidia_ml stuff into module alias 2026-05-11 16:36:23 +02:00
henryruhs 0816bea6d3 cleanup 2026-05-11 16:36:23 +02:00
harisreedharandhenryruhs 6c1475f720 fix lint 2026-05-11 16:36:23 +02:00
henryruhs b549a92a35 cleanup 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 949d9cd276 Move RTC init helpers to bindings layer and clean up structs (#1086)
* converted from type() factory to proper ctypes.Structure subclasses

* change class to method

* bring back rtcSetLocalDescription

* rename rtc_bindings.py to datachannel.py

* rename rtc_library to datachannel_library and rearrange methods

* cleanup
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 99fa2875a9 Fix SDP line endings and media description builder (#1085)
* SDP \r\n fix + build_media_description

* improve tests and remove default arguments

* improve tests

* Refine RTC types and tests with loopback SDP validation

* cleanup
2026-05-11 16:36:23 +02:00
henryruhs b9445fd3a4 add TODOS 2026-05-11 16:36:23 +02:00
henryruhs 66ee9144fb add TODOS 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs dfaa1f9cd4 Replace aiortc with libdatachannel direct pipeline (#1083)
* fix stdin close error

* Refactor stream endpoint, fix encoder thread safety and improve tests

* fix and improve test

* remove not None

* use Enum

* use Enum and add todo

* remove poll
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs a2aedc8814 Add stream helper utilities and IVF frame iterator (#1082)
* Add stream helper utilities and IVF frame iterator

* fix lint

* some cosmetics

* fix lint

* changes

* improve test

* improve types and test

* add todo for better bitrate calculation
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs cc8bfc1af4 Implement RTC store (#1081)
* implement RTC store

* fix ffmpeg_builder

* add RtcSdpOffer type
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 31a16982f3 Refactor RTC library initialization and peer connection API (#1079)
* Replace the mutable RTC_LIBRARY global with @lru_cache on create_static_rtc_library. Expose all RTC_CONFIGURATION fields as parameters with defaults on create_peer_connection. Split add_media_tracks into add_video_track and add_audio_track, each with configurable
   defaults and descriptive parameter names (sync_source_id, canonical_name). Use os.linesep for SDP media descriptions. Simplify init_ctypes to return None and remove the redundant null guard.

* cleanup
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs db545b8ae4 libdatachannel implementation Part 1 (#1077)
* add libdatachannel

* move some methods to rtc_helper.py

* move some methods to rtc_helper.py

* open_vp8_encoder -> spawn_stream_encoder

* update url (linux only)

* fix lint

* remove RTC_STATE and add unit tests

* remove convert_to_raw_rgb

* remove 'utf-8' from decode

* cleanup

* cleanup

* cleanup

* move rtc types to rtc_bindings.py

* remove load_library

* cleanup

* add todos and some cleanup

* fix lint
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 701a1b5f9e Remove upload queue & media chunk reader (#1076)
* remove upload queue & media chunk reader

* remove macos large

---------

Co-authored-by: henryruhs <info@henryruhs.com>
2026-05-11 16:36:23 +02:00
henryruhs 1f80aa735a remove macos large 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs ccfc5f61b6 Fix symatically wrong usage in tests (#1075)
* fix symatically wrong usage in tests

* add target-240p.jpg everywhere

* combine both target
2026-05-11 16:36:23 +02:00
henryruhs 6e7bcb599a once invalid value causes other to fail too 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 47b703f4f5 Ignore set state for non api scope (#1073)
* ignore set state for non api scope

* replace 404 with 422

* remove two loops

* use HTTP_422_UNPROCESSABLE_CONTENT
2026-05-11 16:36:23 +02:00
henryruhs 7111af232c add todos 2026-05-11 16:36:23 +02:00
henryruhs 2709515c39 fix stuck ffmpeg due multi thread lock 2026-05-11 16:36:23 +02:00
henryruhs a9a4adb083 make ci great again 2026-05-11 16:36:23 +02:00
Henry Ruhs eac03796f7 Fix/ffmpeg sanitize (#1074)
* fix sanitize for videos

* fix macos

* fix macos
2026-05-11 16:36:23 +02:00
Henry Ruhs 5790e03009 Feat/ffprobe v2 (#1072)
* follow the todos

* extend to support bit_rate and more

* simplify like crazy

* simplify like crazy

* minor changes

* clean testing

* clean testing

* kill pipe resolver helpers

* kill pipe resolver helpers

* bit rate seems to be different on CI

* use .splitlines() over .split(os.linesep)

* skip test for windows

* hack testing
2026-05-11 16:36:23 +02:00
henryruhs fe7f402474 fix more todos 2026-05-11 16:36:23 +02:00
henryruhs 0fd6a403b8 stop passing format 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs df38778558 fix refresh_session does not validate expiry before refreshing (#1071) 2026-05-11 16:36:23 +02:00
henryruhs e15a2dec76 add tons of todos 2026-05-11 16:36:23 +02:00
henryruhs 399e07261d add tons of todos 2026-05-11 16:36:23 +02:00
henryruhs 61c67c8637 fix macos 2026-05-11 16:36:23 +02:00
henryruhs a16bd54493 add meaningful tests 2026-05-11 16:36:23 +02:00
henryruhs edf8914da3 add meaningful tests 2026-05-11 16:36:23 +02:00
henryruhs 2aced392e1 cleanup code part1 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 2b76f3381f Assets ffmpeg stream upload (#1069)
* ffmpeg sanitize

* fix type

* fix type

* add config

* ChunkQueue -> UploadQueue

* revert assets.py

* move resolve methods to ffmpeg_builder.py

* Refactor ffmpeg.py

* Remove partial import

* improve test

* remove put(None)

* cleanup

* without poll() not working

* ChunkReader -> MediaChunkReader

* improve assert by replacing generic is_file

* naming and cleanup
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs f40ee8335a Fix hardcoded fps (#1068)
* fix hardcoded fps

* fallback to temp_fps when output_video_fps is None

* fix test_ffmpeg

* change temp-fps -> output-audio-fps

* cleanup

* cleanup
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs b9c1ff8185 add test when get_metrics_set() returns None (#1067) 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 48bae74001 fix false vram numbers (#1065) 2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 6654abbdca Fix bad caching of graphic devices (#1064)
* fix bad caching of graphic devices

* restore without cache

* restore without cache

* restore without cache

* remove detect_static_graphic_devices and add resolve_static_cudnn_conv_algo_search
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs facd908196 Dynamic bitrate for webrtc stream (#1063)
* use custom aiortc

* update naming to bitrate_
2026-05-11 16:36:23 +02:00
henryruhs 5c6b247449 merge stuff 2026-05-11 16:36:23 +02:00
Henry Ruhs efab505adb asset validation and image encoder lookup (#1058)
* asset validation and image encoder lookup

* asset validation and image encoder lookup

* asset validation and image encoder lookup
2026-05-11 16:36:23 +02:00
Harisreedharandhenryruhs 246d48e079 add download action (#1057) 2026-05-11 16:36:23 +02:00
henryruhs 980c2d3939 fix stream of videos 2026-05-11 16:36:23 +02:00
harisreedharandhenryruhs f0cacba52b fix execution.py import order 2026-05-11 16:36:23 +02:00
henryruhs 61aa016659 burn ui with fire 2026-05-11 16:36:23 +02:00
henryruhsandClaude Opus 4.6 dedf2bf829 updates for v4
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-11 16:36:22 +02:00
Harisreedharandhenryruhs ad1a6c9ea3 Implement basic webrtc stream (#1054)
* implement basic webrtc_stream

* add aiortc to requirements.txt

* update aiortc version

* rename variables with rtc_ prefix

* changes

* changes

* change helper to assert_helper and stream_helper

* rename variables with rtc_ prefix

* add error handling

* return whole connection

* remove monkey patch and some cleaning

* cleanup

* tiny adjustments

* tiny adjustments

* proper typing and naming for rtc offer set

* - remove async from on_video_track method
- rename source -> target
- add audio

* audio always before video

---------

Co-authored-by: henryruhs <info@henryruhs.com>
2026-05-11 16:36:03 +02:00
henryruhs 9c0859ade0 fix benchmarker, prevent path traveling via job-id 2026-05-11 16:36:01 +02:00
Harisreedharandhenryruhs 1ba2adc10b Rename process to stream (#1053)
* rename process to stream

* remove /image and add webrtc_stream_video scaffolding

* remove _image
2026-05-11 16:35:16 +02:00
Henry Ruhs 078041461d remove more type ignore (#1052) 2026-05-11 16:35:16 +02:00
Henry Ruhs 5ca5e1c2af Better args types part2 (#1051)
* remove helper methods finally

* state becomes the total truth now

* state becomes the total truth now

* state becomes the total truth now

* state becomes the total truth now

* add ini file
2026-05-11 16:35:16 +02:00
Harisreedharandhenryruhs 401118106a Remove unused code to improve test coverage (#1050)
* remove unused code

* move create_session_guard to middlewares
2026-05-11 16:35:16 +02:00
Henry Ruhs f69fa62fa2 better args types (#1049)
* fix lint and remove unused types, restructure helpers

* fix lint

* fix lint

* fix lint
2026-05-11 16:35:16 +02:00
Henry Ruhs ef47b17a4a rename args store to capability store (#1048)
* rename args store to capability store

* fix lint and remove unused types
2026-05-11 16:35:16 +02:00
Harisreedharandhenryruhs 7c078644ad Improve tests (#1047)
* improve tests

* does it hurt someone to have crazy numbers?

---------

Co-authored-by: henryruhs <info@henryruhs.com>
2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 47bdcd67f1 fix pre-check issue 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs f83ff29116 remove pre_check 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 91067c6394 remove program 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 1e8c042473 remove while True and websocket code 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 1f0af5906d hopefully fix test 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 887a6ac1c7 pre_check() on test instead 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 4338d16d17 pre_check() when command == api 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs a21c23173e fix naming and remove unwanted state initialization 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs cd3f6f2224 init 'face_selector_mode' to many 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs f2d9eb7f0b remove temp storing to disk 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs c59596e096 update exception 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 4a0f4bcb9a update test 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs b5a89971c4 process image 2026-05-11 16:35:16 +02:00
Harisreedharandhenryruhs 53845585ef Feat/refactor args (#1045)
* follow the new argument naming convention

* revert

* use get_xxx_arguments

* Simple fix Args

* Fix more stuff

---------

Co-authored-by: henryruhs <info@henryruhs.com>
2026-05-11 16:35:16 +02:00
henryruhs 16d2fbf6f2 fix args store 2026-05-11 16:35:16 +02:00
henryruhs 7a6ef94b26 bump dependencies 2026-05-11 16:35:16 +02:00
henryruhs 9486750f38 allow register_arguments to be nested, fix types, adjust processors 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 1181307a0e reformat test_api_state.py 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs e7a28cf072 introduce type ArgumentSet 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs ecc280a93f remove unwanted assertions 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs aa83142ab5 remove value None as default 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 2a60280182 remove value None as default 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs b9c11815f8 add test_api_capabilities.py 2026-05-11 16:35:16 +02:00
henryruhs accb615181 add get_xxx_set to args store 2026-05-11 16:35:16 +02:00
henryruhs 5f70cbd0a3 fix spacing quite a bit 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 485041c5cc conditional choices, speaking Alias ArgumentValue, different formatting for program.py 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs e16f5bca11 fix test 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 02ba86308e add capabilities 2026-05-11 16:35:16 +02:00
henryruhs fddb5c927b remove unused method 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs f0dc10a2f5 add capabilities endpoint 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 53d0ed0e1c hopefully fix CI 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs f71bbab71e remove has methods 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 300b4f73a1 add has_amd_execution_provider and has_nvidia_execution_provider 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 7e00035fdd fix cache 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs f475ec4bbc remove try block, make detect_graphic_devices stateless, cosmetics 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 9e63832740 fix test 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs ef0bfe81e2 add amd metric 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs f2f09fc215 move detect_execution_devices to system.py 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs 73a298cd97 add processor metric 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs b1947ffaa2 add network metrics 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs b7e80bdafa fix test 2026-05-11 16:35:16 +02:00
harisreedharandhenryruhs a94015c5ee add memory metrics 2026-05-11 16:35:15 +02:00
harisreedharandhenryruhs 14b1ed6b76 add memory metrics 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs 2624c281eb move metrics.py to endpoints 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs bae6169274 some cosmetics 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs 9a27c8346c add disk to metric test 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs e4b8adb218 change unit to GB 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs 4cd588ebc8 add drive_path 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs aaebb56804 make detect_disk_metrics stateless 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs fa16752a6c add disk metric 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs 09dc2ec50b cleanup 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs 3a25476ca6 add pytest-mock 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs 01baf8d2ea improve test 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs a818e7f610 remove instance check 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs 5f6e0cc567 fix test 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs e2738f1f14 add metrics 2026-05-11 16:35:12 +02:00
harisreedharandhenryruhs db5ffdc449 add metrics 2026-05-11 16:35:12 +02:00
henryruhs 2db7ad426a simplify code - no need for helper methods at this point 2026-05-11 16:35:12 +02:00
Harisreedharandhenryruhs abeef470cc Remove xml parsing for gpu metrics (#1030)
* remove xml parsing

* remove aitop
2026-05-11 16:35:10 +02:00
Henry Ruhs 0c18508f70 Revisit and cleanup ffprobe integration (#1027)
* Revisit and cleanup ffprobe integration

* Revisit and cleanup ffprobe integration
2026-05-11 16:34:28 +02:00
henryruhs 60def0f30d Asset related fixes 2026-05-11 16:34:28 +02:00
henryruhs c2676fa471 Asset related fixes 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs dcfbbc041f add missing endpoints 2026-05-11 16:34:28 +02:00
henryruhs dc500eb398 Add process manager to the right place 2026-05-11 16:34:28 +02:00
henryruhs 3575aeea86 Fix for Windows 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 06a3854fbb Revert "hopefully fixes windows CI error"
This reverts commit 3295726347.
2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs d1a068b9d5 hopefully fixes windows CI error 2026-05-11 16:34:28 +02:00
henryruhs 38ece8f7d0 Fix stuff 2026-05-11 16:34:28 +02:00
henryruhs 5fa088457e FFmpeg powered sanitization, Chunk based upload write 2026-05-11 16:34:28 +02:00
henryruhs 77c1682078 Hopefully fix Windows 2026-05-11 16:34:28 +02:00
henryruhs 40f5b403a4 Simplify testing 2026-05-11 16:34:28 +02:00
henryruhs 908f3d2873 Revamp the upload 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 02e8a13adb upload asset endpoint 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs ec3e6d777d upload asset endpoint 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 71c79f4e96 upload asset endpoint 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 746b3249ef upload asset endpoint 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs e3aa3d15f5 upload asset endpoint 2026-05-11 16:34:28 +02:00
henryruhs 2c07d8ca45 Polish asset store and helpers 2026-05-11 16:34:28 +02:00
Harisreedharandhenryruhs 6e84970e0d Feat/audio metadata (#1019)
* audio metadata

* audio metadata

* audio metadata

* audio metadata

* audio metadata

* audio metadata

* audio metadata
2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs f84b2cfc00 asset store update 2026-05-11 16:34:28 +02:00
henryruhs 56842a1c58 Minor cleanup 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 98384a55df state api updates 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs a01bed0bda state api updates 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 6edf6c7266 state api updates 2026-05-11 16:34:28 +02:00
Henry Ruhs 516b5039d0 Move to endpoints directory (#1012)
* Move to endpoints directory

* Move to endpoints directory
2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs dae96f90a1 create to_image.py 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 9bf9ee40a7 audio to image as frames 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 621bddd174 audio to image as frames 2026-05-11 16:34:28 +02:00
henryruhs 5cbc50a5aa Rename local(s) to locale(s) 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 03bbc10251 workflows rename 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 1e2a66e7b0 workflows rename 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 1b2fffd82a image to video as sequence 2026-05-11 16:34:28 +02:00
Henry Ruhs 97801ceab3 feat/ping-endpoint (#1001)
* api: add WebSocket /ping endpoint and update session guard to support WebSocket subprotocol auth; add tests (test_api_ping.py)

* Initial websocket support using ping

* Initial websocket support using ping

* Initial websocket support using ping

* Combine imports
2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 69463b34bc use common analyse_image method 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs e42939df9c to video unification 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs c7ab933a8e to video unification 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 92ec60933e detect workflow 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs f754828d08 changes
restructure conditional methods to a fall-through pattern

common process_temp_frame for all workflow
2026-05-11 16:34:28 +02:00
Henry Ruhs 403e89eeac Feat/session context (#993)
* Add simple session context

* Add simple session context
2026-05-11 16:34:28 +02:00
Henry Ruhs 672d65d61a Add simple path isolation (#992) 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 678f4edd4c update ffmpeg.set_loop
add test

introduce spawn_frames
2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 38e8610523 changes 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 3444194aed remove --keep-temp 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs c4f1d14d71 Part 2 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 0aa56d54bb Part 2 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs cec736d03a fix 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 2485a409b8 part 1 2026-05-11 16:34:28 +02:00
henryruhs 35881a4f8d Switch workflow args order in tests, Remove old choices in processors 2026-05-11 16:34:28 +02:00
henryruhs 86cd7cbc8f Switch workflow args order in tests, Remove old choices in processors 2026-05-11 16:34:28 +02:00
harisreedharandhenryruhs 7666411a3d add todo
add test

cleanup

remove -w

move --workflow position

fix test

add --worflow, audio-to-image, image-to-image, image-to-video
2026-05-11 16:34:27 +02:00
Henry Ruhs b333280b8d Scope for Args (#988)
* Add API scopes

* Add API scopes

* Add API scopes

* Add API scopes

* Add API scopes

* Add API scopes

* Add API scopes

* Add API scopes

* Remove system memory limit (#986)

* Add session_id, make token size more reasonable (#983)

* Add session_id, make token size more reasonable

* Use more direct approach

* Fix more stuff

* Fix ignore comments

* Fix naming

* Fix lint
2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs 9664bb98db changes 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs 139e73589a changes 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs f11e8eaaea changes 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs b0e849a251 rename to target_path 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs d224e47d38 fix exit-helper 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs 7b1570dc3f refactor temp handling from target-path to output-path 2026-05-11 16:31:45 +02:00
henryruhs 85eeaf39ca Burn it with fire 2026-05-11 16:31:45 +02:00
henryruhs a10b740a80 Burn it with fire 2026-05-11 16:31:45 +02:00
Henry Ruhs 85b85b31a2 Remove system memory limit (#986) 2026-05-11 16:31:45 +02:00
Henry Ruhs ba884a19b1 Add session_id, make token size more reasonable (#983)
* Add session_id, make token size more reasonable

* Use more direct approach
2026-05-11 16:31:45 +02:00
Henry Ruhs 87678da498 Local API (#982)
* Introduce API scelleton

* Raw impl for session

* Simple state endpoint

* Apply _body naming

* Finalize session testing and comment out tons of useless code

* Clean and refactor part1

* Clean and refactor part2

* Clean and refactor part2

* Clean and refactor part2

* Clean and refactor part2

* Refactor middleware

* Refactor middleware

* Clean and refactor part3

* TDD and 2 beers

* TDD and 2 beers

* Complete state endpoints

* You can only set what is already present

* Use only JSON as response

* Use default logger

* Improve auth extraction

* Extend api command with more args

* Adjust API messages
2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs f1149ebc84 remove output_path argument 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs 7870e67fdc remove output_path argument from merge_video() 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs 05264ac760 remove output_video_fps argument from merge_video() 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs 71916e4e66 rename method argument 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs c903438c3f fix 2026-05-11 16:31:45 +02:00
harisreedharandhenryruhs 51d4ec63b0 remove same file extension constraint 2026-05-11 16:31:45 +02:00
Henry Ruhs 92b1e1b3e3 Refactor reusable workflow tasks (#980)
* Refactor reusable workflow tasks

* Refactor reusable workflow tasks

* Make it borderline again
2026-05-11 16:31:45 +02:00
Henry Ruhs 1173493e84 Feels so good to get rid of Gradio (#978) 2026-05-11 16:31:45 +02:00
henryruhs daf906097e Mark es temporary v4 2026-05-11 16:31:43 +02:00
233 changed files with 10386 additions and 6156 deletions
Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.3 MiB

+8 -2
View File
@@ -33,8 +33,11 @@ jobs:
uses: actions/setup-python@v5 uses: actions/setup-python@v5
with: with:
python-version: '3.12' python-version: '3.12'
- run: python install.py --onnxruntime default --skip-conda - run: python install.py default --skip-conda
- run: pip install pytest - run: pip install pytest
- run: pip install pytest-mock
- run: pip install httpx
- run: pip install python-multipart
- run: pytest - run: pytest
report: report:
needs: test needs: test
@@ -48,10 +51,13 @@ jobs:
uses: actions/setup-python@v5 uses: actions/setup-python@v5
with: with:
python-version: '3.12' python-version: '3.12'
- run: python install.py --onnxruntime default --skip-conda - run: python install.py default --skip-conda
- run: pip install coveralls - run: pip install coveralls
- run: pip install pytest - run: pip install pytest
- run: pip install pytest-cov - run: pip install pytest-cov
- run: pip install pytest-mock
- run: pip install httpx
- run: pip install python-multipart
- run: pytest tests --cov facefusion - run: pytest tests --cov facefusion
- run: coveralls --service github - run: coveralls --service github
env: env:
+1
View File
@@ -4,4 +4,5 @@ __pycache__
.caches .caches
.idea .idea
.jobs .jobs
.libraries
.vscode .vscode
+1 -7
View File
@@ -8,12 +8,6 @@ FaceFusion
![License](https://img.shields.io/badge/license-OpenRAIL--AS-green) ![License](https://img.shields.io/badge/license-OpenRAIL--AS-green)
Preview
-------
![Preview](https://raw.githubusercontent.com/facefusion/facefusion/master/.github/preview.png?sanitize=true)
Installation Installation
------------ ------------
@@ -34,10 +28,10 @@ options:
commands: commands:
run run the program run run the program
headless-run run the program in headless mode
batch-run run the program in batch mode batch-run run the program in batch mode
force-download force automate downloads and exit force-download force automate downloads and exit
benchmark benchmark the program benchmark benchmark the program
api start the API server
job-list list jobs by status job-list list jobs by status
job-create create a drafted job job-create create a drafted job
job-submit submit a drafted job to become a queued job job-submit submit a drafted job to become a queued job
+15 -7
View File
@@ -1,3 +1,6 @@
[workflow]
workflow =
[paths] [paths]
temp_path = temp_path =
jobs_path = jobs_path =
@@ -32,6 +35,9 @@ reference_face_position =
reference_face_distance = reference_face_distance =
reference_frame_number = reference_frame_number =
[face_tracker]
face_tracker_score =
[face_masker] [face_masker]
face_occluder_model = face_occluder_model =
face_parser_model = face_parser_model =
@@ -48,7 +54,9 @@ voice_extractor_model =
trim_frame_start = trim_frame_start =
trim_frame_end = trim_frame_end =
temp_frame_format = temp_frame_format =
keep_temp =
[frame_distribution]
target_frame_amount =
[output_creation] [output_creation]
output_image_quality = output_image_quality =
@@ -56,6 +64,7 @@ output_image_scale =
output_audio_encoder = output_audio_encoder =
output_audio_quality = output_audio_quality =
output_audio_volume = output_audio_volume =
output_audio_fps =
output_video_encoder = output_video_encoder =
output_video_preset = output_video_preset =
output_video_quality = output_video_quality =
@@ -104,11 +113,6 @@ frame_enhancer_blend =
lip_syncer_model = lip_syncer_model =
lip_syncer_weight = lip_syncer_weight =
[uis]
open_browser =
ui_layouts =
ui_workflow =
[download] [download]
download_providers = download_providers =
download_scope = download_scope =
@@ -118,6 +122,11 @@ benchmark_mode =
benchmark_resolutions = benchmark_resolutions =
benchmark_cycle_count = benchmark_cycle_count =
[api]
api_host =
api_port =
api_security_strategy =
[execution] [execution]
execution_device_ids = execution_device_ids =
execution_providers = execution_providers =
@@ -125,7 +134,6 @@ execution_thread_count =
[memory] [memory]
video_memory_strategy = video_memory_strategy =
system_memory_limit =
[misc] [misc]
log_level = log_level =
+14
View File
@@ -0,0 +1,14 @@
from typing import Optional
from starlette.datastructures import Headers
from starlette.types import Scope
def get_sec_websocket_protocol(scope : Scope) -> Optional[str]:
protocol_header = Headers(scope = scope).get('Sec-WebSocket-Protocol')
if protocol_header:
protocol, _, _ = protocol_header.partition(',')
return protocol.strip()
return None
+91
View File
@@ -0,0 +1,91 @@
import asyncio
import os
import uuid
from typing import List, Optional
from starlette.datastructures import UploadFile
import facefusion.choices
from facefusion import ffmpeg, process_manager, state_manager
from facefusion.filesystem import create_directory, get_file_extension, get_file_format, is_audio, is_image, is_video
from facefusion.types import ImageMetadata, MediaType
from facefusion.vision import detect_image_resolution
def extract_image_metadata(file_path : str) -> ImageMetadata:
metadata : ImageMetadata =\
{
'resolution': detect_image_resolution(file_path)
}
return metadata
def detect_media_type_by_path(file_path : str) -> Optional[MediaType]:
if is_audio(file_path):
return 'audio'
if is_image(file_path):
return 'image'
if is_video(file_path):
return 'video'
return None
def detect_media_type_by_format(file_format : str) -> Optional[MediaType]:
if file_format in facefusion.choices.audio_set:
return 'audio'
if file_format in facefusion.choices.image_set:
return 'image'
if file_format in facefusion.choices.video_set:
return 'video'
return None
def validate_asset_files(upload_files : List[UploadFile]) -> bool:
available_encoder_set = ffmpeg.get_static_available_encoder_set()
for upload_file in upload_files:
file_format = get_file_format(upload_file.filename)
media_type = detect_media_type_by_format(file_format)
if media_type == 'audio' and facefusion.choices.audio_set.get(file_format) not in available_encoder_set.get('audio'): #type:ignore[call-overload]
return False
if media_type == 'image' and facefusion.choices.image_set.get(file_format) not in available_encoder_set.get('image'): #type:ignore[call-overload]
return False
if media_type == 'video' and facefusion.choices.video_set.get(file_format) not in available_encoder_set.get('video'): #type:ignore[call-overload]
return False
return True
async def save_asset_files(upload_files : List[UploadFile]) -> List[str]:
asset_paths : List[str] = []
api_security_strategy = state_manager.get_item('api_security_strategy')
for upload_file in upload_files:
file_format = get_file_format(upload_file.filename)
file_extension = get_file_extension(upload_file.filename)
media_type = detect_media_type_by_format(file_format)
temp_path = state_manager.get_temp_path()
create_directory(temp_path)
asset_file_name = uuid.uuid4().hex
asset_path = os.path.join(temp_path, asset_file_name + file_extension)
file_content = await upload_file.read()
process_manager.start()
if media_type == 'audio' and await asyncio.to_thread(ffmpeg.sanitize_audio, file_content, asset_path, api_security_strategy):
asset_paths.append(asset_path)
if media_type == 'image' and await asyncio.to_thread(ffmpeg.sanitize_image, file_content, asset_path):
asset_paths.append(asset_path)
if media_type == 'video' and await asyncio.to_thread(ffmpeg.sanitize_video, file_content, asset_path, api_security_strategy):
asset_paths.append(asset_path)
process_manager.end()
return asset_paths
+92
View File
@@ -0,0 +1,92 @@
import uuid
from datetime import datetime, timedelta
from typing import List, Optional, cast
from facefusion.apis.asset_helper import detect_media_type_by_path, extract_image_metadata
from facefusion.ffprobe import extract_audio_metadata, extract_video_metadata
from facefusion.filesystem import get_file_format, get_file_name, get_file_size
from facefusion.types import AssetId, AssetSet, AssetStore, AssetType, AudioAsset, AudioFormat, ImageAsset, ImageFormat, SessionId, VideoAsset, VideoFormat
ASSET_STORE : AssetStore = {}
def create_asset(session_id : SessionId, asset_type : AssetType, asset_path : str) -> Optional[AudioAsset | ImageAsset | VideoAsset]:
asset_id = str(uuid.uuid4())
asset_name = get_file_name(asset_path)
asset_format = get_file_format(asset_path)
asset_size = get_file_size(asset_path)
media_type = detect_media_type_by_path(asset_path)
created_at = datetime.now()
expires_at = created_at + timedelta(hours = 2)
if session_id not in ASSET_STORE:
ASSET_STORE[session_id] = {}
if media_type == 'audio':
ASSET_STORE[session_id][asset_id] = cast(AudioAsset,
{
'id': asset_id,
'created_at': created_at,
'expires_at': expires_at,
'type': asset_type,
'media': media_type,
'name': asset_name,
'format': cast(AudioFormat, asset_format),
'size': asset_size,
'path': asset_path,
'metadata': extract_audio_metadata(asset_path)
})
if media_type == 'image':
ASSET_STORE[session_id][asset_id] = cast(ImageAsset,
{
'id': asset_id,
'created_at': created_at,
'expires_at': expires_at,
'type': asset_type,
'media': media_type,
'name': asset_name,
'format': cast(ImageFormat, asset_format),
'size': asset_size,
'path': asset_path,
'metadata': extract_image_metadata(asset_path)
})
if media_type == 'video':
ASSET_STORE[session_id][asset_id] = cast(VideoAsset,
{
'id': asset_id,
'created_at': created_at,
'expires_at': expires_at,
'type': asset_type,
'media': media_type,
'name': asset_name,
'format': cast(VideoFormat, asset_format),
'size': asset_size,
'path': asset_path,
'metadata': extract_video_metadata(asset_path)
})
return ASSET_STORE[session_id].get(asset_id)
def get_assets(session_id : SessionId) -> Optional[AssetSet]:
return ASSET_STORE.get(session_id)
def get_asset(session_id : SessionId, asset_id : AssetId) -> Optional[AudioAsset | ImageAsset | VideoAsset]:
if session_id in ASSET_STORE:
return ASSET_STORE.get(session_id).get(asset_id)
return None
def delete_assets(session_id : SessionId, asset_ids : List[AssetId]) -> None:
if session_id in ASSET_STORE:
for asset_id in asset_ids:
if asset_id in ASSET_STORE.get(session_id):
del ASSET_STORE[session_id][asset_id]
return None
def clear() -> None:
ASSET_STORE.clear()
+58
View File
@@ -0,0 +1,58 @@
from types import ModuleType
from typing import List
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.middleware.cors import CORSMiddleware
from starlette.routing import Route, WebSocketRoute
from facefusion.apis.endpoints.assets import delete_assets, get_asset, get_assets, upload_asset
from facefusion.apis.endpoints.capabilities import get_capabilities
from facefusion.apis.endpoints.metrics import get_metrics, websocket_metrics
from facefusion.apis.endpoints.ping import websocket_ping
from facefusion.apis.endpoints.session import create_session, destroy_session, get_session, refresh_session
from facefusion.apis.endpoints.state import get_state, set_state
from facefusion.apis.endpoints.stream import delete_stream, post_stream, websocket_stream
from facefusion.apis.middlewares.session import create_session_guard
from facefusion.libraries import aom as aom_module, datachannel as datachannel_module, opus as opus_module, vpx as vpx_module
def get_common_modules() -> List[ModuleType]:
return [ aom_module, datachannel_module, opus_module, vpx_module ]
def pre_check() -> bool:
for common_module in get_common_modules():
if not common_module.pre_check():
return False
return True
def create_api() -> Starlette:
session_guard = Middleware(create_session_guard)
routes =\
[
Route('/session', create_session, methods = [ 'POST' ]),
Route('/session', get_session, methods = [ 'GET' ], middleware = [ session_guard ]),
Route('/session', refresh_session, methods = [ 'PUT' ]),
Route('/session', destroy_session, methods = [ 'DELETE' ], middleware = [ session_guard ]),
Route('/state', get_state, methods = [ 'GET' ], middleware = [ session_guard ]),
Route('/state', set_state, methods = [ 'PUT' ], middleware = [ session_guard ]),
Route('/assets', get_assets, methods = [ 'GET' ], middleware = [ session_guard ]),
Route('/assets', upload_asset, methods = [ 'POST' ], middleware = [ session_guard ]),
Route('/assets/{asset_id}', get_asset, methods = [ 'GET' ], middleware = [ session_guard ]),
Route('/assets', delete_assets, methods = [ 'DELETE' ], middleware = [ session_guard ]),
Route('/capabilities', get_capabilities, methods = [ 'GET' ]),
Route('/metrics', get_metrics, methods = [ 'GET' ], middleware = [ session_guard ]),
Route('/stream', post_stream, methods = [ 'POST' ], middleware = [ session_guard ]),
Route('/stream', delete_stream, methods = [ 'DELETE' ], name = 'delete_stream', middleware = [ session_guard ]),
WebSocketRoute('/metrics', websocket_metrics, middleware = [ session_guard ]),
WebSocketRoute('/ping', websocket_ping, middleware = [ session_guard ]),
WebSocketRoute('/stream', websocket_stream, middleware = [ session_guard ])
]
api = Starlette(routes = routes)
api.add_middleware(CORSMiddleware, allow_origins = [ '*' ], allow_methods = [ '*' ], allow_headers = [ '*' ])
return api
+135
View File
@@ -0,0 +1,135 @@
import os
from typing import List
from starlette.requests import Request
from starlette.responses import FileResponse, JSONResponse, Response
from starlette.status import HTTP_200_OK, HTTP_201_CREATED, HTTP_400_BAD_REQUEST, HTTP_404_NOT_FOUND, HTTP_415_UNSUPPORTED_MEDIA_TYPE
from facefusion import session_context, session_manager
from facefusion.apis import asset_store
from facefusion.apis.asset_helper import save_asset_files, validate_asset_files
from facefusion.apis.endpoints.session import extract_access_token
from facefusion.filesystem import remove_file
async def upload_asset(request : Request) -> Response:
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
asset_type = request.query_params.get('type')
if session_id and asset_type in [ 'source', 'target' ]:
session_context.set_session_id(session_id)
form = await request.form()
upload_files = form.getlist('file')
if upload_files and validate_asset_files(upload_files):
asset_paths = await save_asset_files(upload_files)
if asset_paths:
asset_ids : List[str] = []
for asset_path in asset_paths:
asset = asset_store.create_asset(session_id, asset_type, asset_path)
if asset:
asset_id = asset.get('id')
if asset_id:
asset_ids.append(asset_id)
if asset_ids:
return JSONResponse(
{
'asset_ids': asset_ids
}, status_code = HTTP_201_CREATED)
return Response(status_code = HTTP_415_UNSUPPORTED_MEDIA_TYPE)
return Response(status_code = HTTP_400_BAD_REQUEST)
async def get_assets(request : Request) -> Response:
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
if session_id:
asset_set = asset_store.get_assets(session_id)
assets = []
if asset_set:
for asset in asset_set.values():
assets.append(
{
'id': asset.get('id'),
'created_at': asset.get('created_at').isoformat(),
'expires_at': asset.get('expires_at').isoformat(),
'type': asset.get('type'),
'media': asset.get('media'),
'name': asset.get('name'),
'format': asset.get('format'),
'size': asset.get('size'),
'metadata': asset.get('metadata')
})
return JSONResponse(
{
'assets': assets
}, status_code = HTTP_200_OK)
return Response(status_code = HTTP_400_BAD_REQUEST)
async def get_asset(request : Request) -> Response:
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
asset_id = request.path_params.get('asset_id')
if session_id and asset_id:
asset = asset_store.get_asset(session_id, asset_id)
if asset:
if request.query_params.get('action') == 'download':
asset_path = asset.get('path')
if os.path.exists(asset_path):
return FileResponse(asset_path, filename = asset.get('name'))
return JSONResponse(
{
'id': asset.get('id'),
'created_at': asset.get('created_at').isoformat(),
'expires_at': asset.get('expires_at').isoformat(),
'type': asset.get('type'),
'media': asset.get('media'),
'name': asset.get('name'),
'format': asset.get('format'),
'size': asset.get('size'),
'metadata': asset.get('metadata')
}, status_code = HTTP_200_OK)
return Response(status_code = HTTP_404_NOT_FOUND)
async def delete_assets(request : Request) -> Response:
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
body = await request.json()
asset_ids = body.get('asset_ids')
if session_id and asset_ids:
asset_set = asset_store.get_assets(session_id)
if asset_set:
for asset_id in asset_ids:
if asset_id in asset_set:
asset = asset_set.get(asset_id)
if asset:
remove_file(asset.get('path'))
asset_store.delete_assets(session_id, asset_ids)
return Response(status_code = HTTP_200_OK)
return Response(status_code = HTTP_404_NOT_FOUND)
+20
View File
@@ -0,0 +1,20 @@
from starlette.requests import Request
from starlette.responses import JSONResponse
from starlette.status import HTTP_200_OK
import facefusion.choices
from facefusion import capability_store
async def get_capabilities(request : Request) -> JSONResponse:
capabilities =\
{
'formats':
{
'audio': facefusion.choices.audio_formats,
'image': facefusion.choices.image_formats,
'video': facefusion.choices.video_formats
},
'arguments': capability_store.get_api_capability_group()
}
return JSONResponse(capabilities, status_code = HTTP_200_OK)
+32
View File
@@ -0,0 +1,32 @@
import asyncio
from starlette.requests import Request
from starlette.responses import JSONResponse, Response
from starlette.status import HTTP_404_NOT_FOUND
from starlette.websockets import WebSocket
from facefusion.apis.api_helper import get_sec_websocket_protocol
from facefusion.system import get_metrics_set
async def get_metrics(request : Request) -> Response:
metrics_set = get_metrics_set()
if metrics_set:
return JSONResponse(metrics_set)
return Response(status_code = HTTP_404_NOT_FOUND)
async def websocket_metrics(websocket : WebSocket) -> None:
subprotocol = get_sec_websocket_protocol(websocket.scope)
await websocket.accept(subprotocol = subprotocol)
try:
while True:
metrics_set = get_metrics_set()
await websocket.send_json(metrics_set)
await asyncio.sleep(2)
except Exception:
pass
+16
View File
@@ -0,0 +1,16 @@
from starlette.websockets import WebSocket
from facefusion.apis.api_helper import get_sec_websocket_protocol
async def websocket_ping(websocket : WebSocket) -> None:
subprotocol = get_sec_websocket_protocol(websocket.scope)
await websocket.accept(subprotocol = subprotocol)
try:
while True:
await websocket.receive()
except Exception:
pass
+75
View File
@@ -0,0 +1,75 @@
import os
import secrets
from starlette.requests import Request
from starlette.responses import JSONResponse
from starlette.status import HTTP_200_OK, HTTP_201_CREATED, HTTP_401_UNAUTHORIZED
from facefusion import session_context, session_manager, translator
from facefusion.apis.session_helper import extract_access_token
async def create_session(request : Request) -> JSONResponse:
body = await request.json()
if not body.get('api_key') or body.get('api_key') == os.getenv('FACEFUSION_API_KEY'):
session_id = secrets.token_urlsafe(16)
session = session_manager.create_session()
session_context.set_session_id(session_id)
session_manager.set_session(session_id, session)
return JSONResponse(
{
'access_token': session.get('access_token'),
'refresh_token': session.get('refresh_token')
}, status_code = HTTP_201_CREATED)
return JSONResponse(
{
'message': translator.get('something_went_wrong', 'facefusion.apis')
}, status_code = HTTP_401_UNAUTHORIZED)
async def get_session(request : Request) -> JSONResponse:
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
session = session_manager.get_session(session_id)
return JSONResponse(
{
'access_token': session.get('access_token'),
'refresh_token': session.get('refresh_token'),
'created_at': session.get('created_at').isoformat(),
'expires_at': session.get('expires_at').isoformat()
}, status_code = HTTP_200_OK)
async def refresh_session(request : Request) -> JSONResponse:
body = await request.json()
for session_id, session in session_manager.SESSIONS.items():
if session.get('refresh_token') == body.get('refresh_token') and session_manager.validate_session(session_id):
__session__ = session_manager.create_session()
session_manager.set_session(session_id, __session__)
return JSONResponse(
{
'access_token': __session__.get('access_token'),
'refresh_token': __session__.get('refresh_token')
}, status_code = HTTP_200_OK)
return JSONResponse(
{
'message': translator.get('something_went_wrong', 'facefusion.apis')
}, status_code = HTTP_401_UNAUTHORIZED)
async def destroy_session(request : Request) -> JSONResponse:
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
session_manager.clear_session(session_id)
return JSONResponse(
{
'message': translator.get('ok', 'facefusion.apis')
}, status_code = HTTP_200_OK)
+93
View File
@@ -0,0 +1,93 @@
from starlette.requests import Request
from starlette.responses import JSONResponse
from starlette.status import HTTP_200_OK, HTTP_400_BAD_REQUEST, HTTP_404_NOT_FOUND, HTTP_422_UNPROCESSABLE_CONTENT
from facefusion import args_helper, capability_store, session_manager, state_manager, translator
from facefusion.apis import asset_store
from facefusion.apis.endpoints.session import extract_access_token
async def get_state(request : Request) -> JSONResponse:
api_args = args_helper.extract_api_args(state_manager.get_state())
return JSONResponse(state_manager.collect_state(api_args), status_code = HTTP_200_OK)
async def set_state(request : Request) -> JSONResponse:
__api_args__ = {}
action = request.query_params.get('action')
asset_type = request.query_params.get('type')
if action == 'select' and asset_type == 'source':
return await select_source(request)
if action == 'select' and asset_type == 'target':
return await select_target(request)
body = await request.json()
api_args = capability_store.get_api_arguments()
for key, value in body.items():
if key not in api_args:
return JSONResponse(
{
'message': translator.get('invalid_state_key', 'facefusion.apis')
}, status_code = HTTP_400_BAD_REQUEST)
__api_args__[key] = value
if __api_args__:
for key, value in __api_args__.items():
state_manager.set_item(key, value)
__api_args__ = args_helper.extract_api_args(state_manager.get_state())
return JSONResponse(state_manager.collect_state(__api_args__), status_code = HTTP_200_OK)
return JSONResponse({}, status_code = HTTP_422_UNPROCESSABLE_CONTENT)
async def select_source(request : Request) -> JSONResponse:
body = await request.json()
asset_ids = body.get('asset_ids')
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
if isinstance(asset_ids, list) and session_id:
source_paths = []
for asset_id in asset_ids:
asset = asset_store.get_asset(session_id, asset_id)
if asset:
source_paths.append(asset.get('path'))
state_manager.set_item('source_paths', source_paths)
__api_args__ = args_helper.extract_api_args(state_manager.get_state())
return JSONResponse(state_manager.collect_state(__api_args__), status_code = HTTP_200_OK)
return JSONResponse(
{
'message': translator.get('source_asset_not_found', 'facefusion.apis')
}, status_code = HTTP_404_NOT_FOUND)
async def select_target(request : Request) -> JSONResponse:
body = await request.json()
asset_id = body.get('asset_id')
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
if isinstance(asset_id, str) and session_id:
asset = asset_store.get_asset(session_id, asset_id)
if asset:
state_manager.set_item('target_path', asset.get('path'))
__api_args__ = args_helper.extract_api_args(state_manager.get_state())
return JSONResponse(state_manager.collect_state(__api_args__), status_code = HTTP_200_OK)
return JSONResponse(
{
'message': translator.get('target_asset_not_found', 'facefusion.apis')
}, status_code = HTTP_404_NOT_FOUND)
+52
View File
@@ -0,0 +1,52 @@
from starlette.requests import Request
from starlette.responses import Response
from starlette.status import HTTP_200_OK, HTTP_201_CREATED, HTTP_404_NOT_FOUND
from starlette.websockets import WebSocket, WebSocketState
from facefusion import session_context, session_manager
from facefusion.apis.api_helper import get_sec_websocket_protocol
from facefusion.apis.session_helper import extract_access_token
from facefusion.apis.stream_manager import destroy_stream, process_image, process_video
async def websocket_stream(websocket : WebSocket) -> None:
subprotocol = get_sec_websocket_protocol(websocket.scope)
access_token = extract_access_token(websocket.scope)
session_id = session_manager.find_session_id(access_token)
session_context.set_session_id(session_id)
await websocket.accept(subprotocol = subprotocol)
await process_image(websocket)
if websocket.client_state == WebSocketState.CONNECTED:
await websocket.close()
async def post_stream(request : Request) -> Response:
headers =\
{
'Location': request.url_for('delete_stream').path
}
content_type = request.headers.get('content-type')
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
session_context.set_session_id(session_id)
if session_id and content_type == 'application/sdp':
sdp_offer = await request.body()
sdp_answer = process_video(session_id, sdp_offer.decode())
if sdp_answer:
return Response(sdp_answer, status_code = HTTP_201_CREATED, media_type = 'application/sdp', headers = headers)
return Response(status_code = HTTP_404_NOT_FOUND)
async def delete_stream(request : Request) -> Response:
access_token = extract_access_token(request.scope)
session_id = session_manager.find_session_id(access_token)
if session_id and destroy_stream(session_id):
return Response(status_code = HTTP_200_OK)
return Response(status_code = HTTP_404_NOT_FOUND)
+15
View File
@@ -0,0 +1,15 @@
from facefusion.types import Locales
LOCALES : Locales =\
{
'en':
{
'ok': 'ok',
'something_went_wrong': 'something went wrong',
'invalid_access_token': 'invalid access token',
'invalid_refresh_token': 'invalid refresh token',
'source_asset_not_found': 'source asset not found',
'target_asset_not_found': 'target asset not found',
'invalid_state_key': 'invalid state key'
}
}
+34
View File
@@ -0,0 +1,34 @@
from starlette.responses import JSONResponse
from starlette.status import HTTP_401_UNAUTHORIZED, HTTP_426_UPGRADE_REQUIRED
from starlette.types import ASGIApp, Receive, Scope, Send
from facefusion import session_manager, translator
from facefusion.apis.session_helper import extract_access_token
def create_session_guard(app : ASGIApp) -> ASGIApp:
async def middleware(scope : Scope, receive : Receive, send : Send) -> None:
access_token = extract_access_token(scope)
if access_token:
session_id = session_manager.find_session_id(access_token)
if session_id:
if session_manager.validate_session(session_id):
return await app(scope, receive, send)
response = JSONResponse(
{
'message': translator.get('invalid_access_token', 'facefusion.apis')
}, status_code = HTTP_426_UPGRADE_REQUIRED)
return await response(scope, receive, send)
response = JSONResponse(
{
'message': translator.get('invalid_access_token', 'facefusion.apis')
}, status_code = HTTP_401_UNAUTHORIZED)
return await response(scope, receive, send)
return middleware
+29
View File
@@ -0,0 +1,29 @@
from typing import Optional
from starlette.datastructures import Headers
from starlette.types import Scope
from facefusion.apis.api_helper import get_sec_websocket_protocol
from facefusion.types import Token
def extract_access_token(scope : Scope) -> Optional[Token]:
if scope.get('type') == 'http':
auth_header = Headers(scope = scope).get('Authorization')
if auth_header:
auth_prefix, _, access_token = auth_header.partition(' ')
if auth_prefix.lower() == 'bearer' and access_token:
return access_token
if scope.get('type') == 'websocket':
subprotocol = get_sec_websocket_protocol(scope)
if subprotocol:
protocol_prefix, _, access_token = subprotocol.partition('.')
if protocol_prefix == 'access_token' and access_token:
return access_token
return None
+67
View File
@@ -0,0 +1,67 @@
from functools import partial
from queue import Queue
from typing import Optional, Tuple
import numpy
from facefusion import rtc
from facefusion.apis.stream_event import create_receive_event
from facefusion.codecs import opus_decoder, opus_encoder
from facefusion.types import AudioCodec, AudioFrame, Buffer, OpusDecoder, RtcPeer, RtcPeerAudio, Time
def run_audio_encode_loop(rtc_peer : RtcPeer, audio_queue : Queue[Tuple[Time, AudioFrame]]) -> None:
audio_codec = rtc_peer.get('audio').get('codec')
temp_audio_time, temp_audio_frame = audio_queue.get()
audio_encoder = opus_encoder.create(48000, 2)
while numpy.any(temp_audio_frame):
audio_buffer = opus_encoder.encode(audio_encoder, temp_audio_frame.tobytes(), 2)
if audio_buffer:
audio_timestamp = rtc.convert_time_to_timestamp(audio_codec, temp_audio_time)
rtc.send_audio(rtc_peer, audio_buffer, audio_timestamp)
temp_audio_time, temp_audio_frame = audio_queue.get()
opus_encoder.destroy(audio_encoder)
def receive_audio_frames(rtc_peer_audio : RtcPeerAudio, audio_queue : Queue[Tuple[Time, AudioFrame]]) -> None:
audio_track = rtc_peer_audio.get('receiver_track')
audio_codec = rtc_peer_audio.get('codec')
audio_decoder = create_audio_decoder(audio_codec)
audio_frame_handler = partial(handle_audio_frame, audio_codec, audio_decoder, audio_queue)
receive_event = create_receive_event(audio_track, audio_frame_handler)
receive_event.wait()
empty_audio_frame = numpy.empty(0)
audio_queue.put((0.0, empty_audio_frame))
destroy_audio_decoder(audio_codec, audio_decoder)
def decode_audio_frame(audio_codec : AudioCodec, audio_decoder : OpusDecoder, input_buffer : Buffer) -> Optional[Buffer]:
if audio_codec == 'opus':
return opus_decoder.decode(audio_decoder, input_buffer, 2)
return None
def create_audio_decoder(audio_codec : AudioCodec) -> Optional[OpusDecoder]:
if audio_codec == 'opus':
return opus_decoder.create(48000, 2)
return None
def destroy_audio_decoder(audio_codec : AudioCodec, audio_decoder : OpusDecoder) -> None:
if audio_codec == 'opus':
opus_decoder.destroy(audio_decoder)
def handle_audio_frame(audio_codec : AudioCodec, audio_decoder : OpusDecoder, audio_queue : Queue[Tuple[Time, AudioFrame]], audio_buffer : Buffer, audio_timestamp : int) -> None:
audio_frame = decode_audio_frame(audio_codec, audio_decoder, audio_buffer)
if audio_frame:
audio_frame = numpy.frombuffer(audio_frame, dtype = numpy.float32)
audio_time = rtc.convert_timestamp_to_time(audio_codec, audio_timestamp)
audio_queue.put((audio_time, audio_frame))
+30
View File
@@ -0,0 +1,30 @@
import ctypes
import threading
from functools import partial
from facefusion.libraries import datachannel as datachannel_module
from facefusion.types import FrameHandler
def create_receive_event(track : int, frame_handler : FrameHandler) -> threading.Event:
datachannel_library = datachannel_module.create_static_library()
receive_event = threading.Event()
frame_callback = ctypes.CFUNCTYPE(None, ctypes.c_int, ctypes.c_void_p, ctypes.c_int, ctypes.c_void_p, ctypes.c_void_p)(partial(dispatch_frame, frame_handler))
close_callback = ctypes.CFUNCTYPE(None, ctypes.c_int, ctypes.c_void_p)(partial(dispatch_event, receive_event))
datachannel_library.rtcSetFrameCallback(track, frame_callback)
datachannel_library.rtcSetClosedCallback(track, close_callback)
receive_event.frame_callback = frame_callback # type: ignore[attr-defined]
receive_event.close_callback = close_callback # type: ignore[attr-defined]
return receive_event
def dispatch_frame(frame_handler : FrameHandler, track : int, data : ctypes.c_void_p, size : int, info : ctypes.c_void_p, pointer : ctypes.c_void_p) -> None:
frame_buffer = ctypes.string_at(data, size)
frame_timestamp = ctypes.cast(info, ctypes.POINTER(ctypes.c_uint32)).contents.value
frame_handler(frame_buffer, frame_timestamp)
def dispatch_event(event : threading.Event, track : int, pointer : ctypes.c_void_p) -> None:
event.set()
+136
View File
@@ -0,0 +1,136 @@
import ctypes
import threading
from collections.abc import AsyncIterator
from concurrent.futures import Future, ThreadPoolExecutor
from queue import Queue
from typing import Optional, Tuple
import cv2
import numpy
from starlette.websockets import WebSocket
from facefusion import rtc, rtc_store, state_manager, streamer
from facefusion.apis.stream_audio import receive_audio_frames, run_audio_encode_loop
from facefusion.apis.stream_video import receive_video_frames, run_video_encode_loop
from facefusion.libraries import datachannel as datachannel_module
from facefusion.types import AudioCodec, AudioFrame, BufferPack, PeerConnection, RtcPeer, RtcPeerAudio, SdpAnswer, SdpOffer, SessionId, Time, VideoCodec, VisionFrame
async def process_image(websocket : WebSocket) -> None:
capture_vision_frame = await anext(receive_vision_frames(websocket), None)
if numpy.any(capture_vision_frame):
output_vision_frame = streamer.process_stream_frame(capture_vision_frame)
is_success, output_frame_buffer = cv2.imencode('.jpg', output_vision_frame)
if is_success:
await websocket.send_bytes(output_frame_buffer.tobytes())
async def receive_vision_frames(websocket : WebSocket) -> AsyncIterator[VisionFrame]:
websocket_event = await websocket.receive()
while websocket_event.get('type') == 'websocket.receive':
frame_buffer = websocket_event.get('bytes') or bytes()
vision_frame = cv2.imdecode(numpy.frombuffer(frame_buffer, numpy.uint8), cv2.IMREAD_COLOR)
if numpy.any(vision_frame):
yield vision_frame
websocket_event = await websocket.receive()
def process_video(session_id : SessionId, sdp_offer : SdpOffer) -> Optional[SdpAnswer]:
video_codec : VideoCodec = 'vp8'
if rtc.get_payload_type(sdp_offer, 'vp9'):
video_codec = 'vp9'
if rtc.get_payload_type(sdp_offer, 'av1'):
video_codec = 'av1'
video_payload_type = rtc.get_payload_type(sdp_offer, video_codec)
if video_payload_type:
peer_connection : PeerConnection = rtc.create_peer_connection()
video_receiver_track = rtc.add_video_track(peer_connection, 'recvonly', video_codec, video_payload_type)
video_sender_track = rtc.add_video_track(peer_connection, 'sendonly', video_codec, video_payload_type)
sender_bitrate = ctypes.c_uint(0)
receiver_bitrate = ctypes.c_uint(8000)
rtc.wire_sender_bitrate(video_sender_track, sender_bitrate)
audio_codec : AudioCodec = 'opus'
audio_payload_type = rtc.get_payload_type(sdp_offer, audio_codec)
if audio_payload_type:
audio_receiver_track = rtc.add_audio_track(peer_connection, 'recvonly', audio_codec, audio_payload_type)
audio_sender_track = rtc.add_audio_track(peer_connection, 'sendonly', audio_codec, audio_payload_type)
rtc.set_remote_description(peer_connection, sdp_offer)
sdp_answer = rtc.create_sdp_answer(peer_connection)
if sdp_answer:
rtc_peer : RtcPeer =\
{
'peer_connection': peer_connection,
'video':
{
'sender_track': video_sender_track,
'receiver_track': video_receiver_track,
'codec': video_codec
},
'sender_bitrate': sender_bitrate,
'receiver_bitrate': receiver_bitrate
}
if audio_payload_type:
rtc_peer['audio'] = RtcPeerAudio(
sender_track = audio_sender_track,
receiver_track = audio_receiver_track,
codec = audio_codec
)
rtc_store.init_peers(session_id)
rtc_store.get_peers(session_id).append(rtc_peer)
threading.Thread(target = run_peer_loop, args = (session_id, rtc_peer), daemon = True).start()
return sdp_answer
datachannel_module.create_static_library().rtcDeletePeerConnection(peer_connection)
return None
def run_peer_loop(session_id : SessionId, rtc_peer : RtcPeer) -> None:
execution_thread_count = state_manager.get_item('execution_thread_count')
video_queue : Queue[Tuple[Time, Future[BufferPack]]] = Queue(maxsize = execution_thread_count)
audio_queue : Queue[Tuple[Time, AudioFrame]] = Queue(maxsize = execution_thread_count * 10)
video_executor = ThreadPoolExecutor(max_workers = execution_thread_count)
video_receiver_thread = threading.Thread(target = receive_video_frames, args = (rtc_peer.get('video'), video_queue, video_executor), daemon = True)
video_encoder_thread = threading.Thread(target = run_video_encode_loop, args = (rtc_peer, video_queue), daemon = True)
video_receiver_thread.start()
video_encoder_thread.start()
if rtc_peer.get('audio'):
audio_receiver_thread = threading.Thread(target = receive_audio_frames, args = (rtc_peer.get('audio'), audio_queue), daemon = True)
audio_encoder_thread = threading.Thread(target = run_audio_encode_loop, args = (rtc_peer, audio_queue), daemon = True)
audio_receiver_thread.start()
audio_encoder_thread.start()
audio_receiver_thread.join()
audio_encoder_thread.join()
video_receiver_thread.join()
video_encoder_thread.join()
video_executor.shutdown(wait = True)
rtc_store.delete_peers(session_id)
def destroy_stream(session_id : SessionId) -> bool:
if rtc_store.has_peers(session_id):
rtc_store.delete_peers(session_id)
return not rtc_store.has_peers(session_id)
return False
+174
View File
@@ -0,0 +1,174 @@
from concurrent.futures import Future, ThreadPoolExecutor
from functools import partial
from queue import Queue
from typing import Optional, Tuple
import cv2
import numpy
from facefusion import rtc, streamer
from facefusion.apis.stream_event import create_receive_event
from facefusion.codecs import aom_decoder, aom_encoder, vpx_decoder, vpx_encoder
from facefusion.types import AomDecoder, AomEncoder, BitRate, Buffer, BufferPack, Resolution, RtcPeer, RtcPeerVideo, Time, VideoCodec, VisionFrame, VpxDecoder, VpxEncoder
def run_video_encode_loop(rtc_peer : RtcPeer, video_queue : Queue[Tuple[Time, Future[BufferPack]]]) -> None:
video_codec = rtc_peer.get('video').get('codec')
video_time, video_future = video_queue.get()
video_pack = video_future.result()
video_buffer = video_pack.get('buffer')
video_resolution = video_pack.get('resolution')
if video_buffer:
temp_resolution : Resolution = video_resolution
temp_bitrate : BitRate = 8000
video_encoder = create_video_encoder(video_codec, temp_resolution, temp_bitrate)
frame_index = 0
while video_buffer:
sender_bitrate = rtc_peer.get('sender_bitrate').value
if video_resolution[0] - temp_resolution[0] or video_resolution[1] - temp_resolution[1]:
temp_resolution = video_resolution
update_video_encoder_resolution(video_codec, video_encoder, temp_resolution)
if sender_bitrate > 0 and sender_bitrate - temp_bitrate:
temp_bitrate = sender_bitrate
update_video_encoder_bitrate(video_codec, video_encoder, temp_bitrate)
__video_buffer__ = encode_video_frame(video_codec, video_encoder, video_buffer, temp_resolution, frame_index)
if __video_buffer__:
video_timestamp = rtc.convert_time_to_timestamp(video_codec, video_time)
rtc.send_video(rtc_peer, __video_buffer__, video_timestamp)
receiver_bitrate = rtc_peer.get('receiver_bitrate').value
rtc.adapt_receiver_bitrate(rtc_peer, receiver_bitrate)
frame_index += 1
video_time, video_future = video_queue.get()
video_pack = video_future.result()
video_buffer = video_pack.get('buffer')
video_resolution = video_pack.get('resolution')
destroy_video_encoder(video_codec, video_encoder)
rtc.clear_bitrate(rtc_peer)
def receive_video_frames(rtc_peer_video : RtcPeerVideo, video_queue : Queue[Tuple[Time, Future[BufferPack]]], video_executor : ThreadPoolExecutor) -> None:
video_track = rtc_peer_video.get('receiver_track')
video_codec = rtc_peer_video.get('codec')
video_decoder = create_video_decoder(video_codec)
video_frame_handler = partial(handle_video_frame, video_codec, video_decoder, video_queue, video_executor)
receive_event = create_receive_event(video_track, video_frame_handler)
receive_event.wait()
empty_future : Future[BufferPack] = Future()
empty_future.set_result(BufferPack(buffer = bytes(), resolution = (0, 0)))
video_queue.put((0.0, empty_future))
destroy_video_decoder(video_codec, video_decoder)
def process_video_frame(input_vision_frame : VisionFrame) -> BufferPack:
output_vision_frame = streamer.process_stream_frame(input_vision_frame)
output_resolution : Resolution = (output_vision_frame.shape[1], output_vision_frame.shape[0])
output_buffer = cv2.cvtColor(output_vision_frame, cv2.COLOR_BGR2YUV_I420).tobytes()
return BufferPack(buffer = output_buffer, resolution = output_resolution)
def decode_video_frame(video_codec : VideoCodec, video_decoder : VpxDecoder | AomDecoder, input_buffer : Buffer) -> Optional[VisionFrame]:
if video_codec == 'av1':
aom_pointer = aom_decoder.decode(video_decoder, input_buffer)
if aom_pointer:
return normalize_vision_frame(aom_pointer)
if video_codec in [ 'vp8', 'vp9' ]:
vpx_pointer = vpx_decoder.decode(video_decoder, input_buffer)
if vpx_pointer:
return normalize_vision_frame(vpx_pointer)
return None
def encode_video_frame(video_codec : VideoCodec, video_encoder : VpxEncoder | AomEncoder, input_buffer : Buffer, frame_resolution : Resolution, frame_index : int) -> Buffer:
if video_codec == 'av1':
return aom_encoder.encode(video_encoder, input_buffer, frame_resolution, frame_index)
if video_codec in [ 'vp8', 'vp9' ]:
return vpx_encoder.encode(video_encoder, input_buffer, frame_resolution, frame_index)
return bytes()
def normalize_vision_frame(frame_pointer : BufferPack) -> VisionFrame:
frame_width, frame_height = frame_pointer.get('resolution')
vision_frame = numpy.frombuffer(frame_pointer.get('buffer'), dtype = numpy.uint8).reshape((frame_height * 3 // 2, frame_width))
return cv2.cvtColor(vision_frame, cv2.COLOR_YUV2BGR_I420)
def create_video_decoder(video_codec : VideoCodec) -> Optional[VpxDecoder | AomDecoder]:
if video_codec == 'av1':
return aom_decoder.create(8)
if video_codec in [ 'vp8', 'vp9' ]:
return vpx_decoder.create(video_codec, 8)
return None
def create_video_encoder(video_codec : VideoCodec, frame_resolution : Resolution, bitrate : BitRate) -> Optional[VpxEncoder | AomEncoder]:
if video_codec == 'av1':
return aom_encoder.create(frame_resolution, bitrate, 8, 10)
if video_codec in [ 'vp8', 'vp9' ]:
return vpx_encoder.create(video_codec, frame_resolution, bitrate, 8, 10)
return None
def destroy_video_decoder(video_codec : VideoCodec, video_decoder : VpxDecoder | AomDecoder) -> None:
if video_codec == 'av1':
aom_decoder.destroy(video_decoder)
if video_codec in [ 'vp8', 'vp9' ]:
vpx_decoder.destroy(video_decoder)
def destroy_video_encoder(video_codec : VideoCodec, video_encoder : VpxEncoder | AomEncoder) -> None:
if video_codec == 'av1':
aom_encoder.destroy(video_encoder)
if video_codec in [ 'vp8', 'vp9' ]:
vpx_encoder.destroy(video_encoder)
def update_video_encoder_resolution(video_codec : VideoCodec, video_encoder : VpxEncoder | AomEncoder, frame_resolution : Resolution) -> bool:
if video_codec == 'av1':
return aom_encoder.update_resolution(video_encoder, frame_resolution)
if video_codec in [ 'vp8', 'vp9' ]:
return vpx_encoder.update_resolution(video_encoder, frame_resolution)
return False
def update_video_encoder_bitrate(video_codec : VideoCodec, video_encoder : VpxEncoder | AomEncoder, bitrate : BitRate) -> bool:
if video_codec == 'av1':
return aom_encoder.update_bitrate(video_encoder, bitrate)
if video_codec in [ 'vp8', 'vp9' ]:
return vpx_encoder.update_bitrate(video_encoder, bitrate)
return False
def handle_video_frame(video_codec : VideoCodec, video_decoder : VpxDecoder | AomDecoder, video_queue : Queue[Tuple[Time, Future[BufferPack]]], video_executor : ThreadPoolExecutor, video_buffer : Buffer, video_timestamp : int) -> None:
vision_frame = decode_video_frame(video_codec, video_decoder, video_buffer)
if numpy.any(vision_frame) and video_queue.qsize() < video_queue.maxsize:
video_future = video_executor.submit(process_video_frame, vision_frame)
video_time = rtc.convert_timestamp_to_time(video_codec, video_timestamp)
video_queue.put((video_time, video_future))
+5 -3
View File
@@ -5,12 +5,14 @@ from facefusion.types import AppContext
def detect_app_context() -> AppContext: def detect_app_context() -> AppContext:
jobs_path = os.path.join('facefusion', 'jobs')
apis_path = os.path.join('facefusion', 'apis')
frame = sys._getframe(1) frame = sys._getframe(1)
while frame: while frame:
if os.path.join('facefusion', 'jobs') in frame.f_code.co_filename: if jobs_path in frame.f_code.co_filename:
return 'cli' return 'cli'
if os.path.join('facefusion', 'uis') in frame.f_code.co_filename: if apis_path in frame.f_code.co_filename:
return 'ui' return 'api'
frame = frame.f_back frame = frame.f_back
return 'cli' return 'cli'
@@ -1,14 +1,17 @@
from facefusion import state_manager from typing import Union
from facefusion.capability_store import get_api_arguments, get_cli_arguments, get_sys_arguments
from facefusion.filesystem import get_file_name, is_video, resolve_file_paths from facefusion.filesystem import get_file_name, is_video, resolve_file_paths
from facefusion.jobs import job_store
from facefusion.normalizer import normalize_fps, normalize_space from facefusion.normalizer import normalize_fps, normalize_space
from facefusion.processors.core import get_processors_modules from facefusion.processors.core import get_processors_modules
from facefusion.types import ApplyStateItem, Args from facefusion.processors.types import ProcessorState
from facefusion.types import ApplyStateItem, Args, State
from facefusion.vision import detect_video_fps from facefusion.vision import detect_video_fps
def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None: def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('command', args.get('command')) apply_state_item('command', args.get('command'))
apply_state_item('workflow', args.get('workflow'))
apply_state_item('temp_path', args.get('temp_path')) apply_state_item('temp_path', args.get('temp_path'))
apply_state_item('jobs_path', args.get('jobs_path')) apply_state_item('jobs_path', args.get('jobs_path'))
apply_state_item('source_paths', args.get('source_paths')) apply_state_item('source_paths', args.get('source_paths'))
@@ -33,6 +36,7 @@ def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('reference_face_position', args.get('reference_face_position')) apply_state_item('reference_face_position', args.get('reference_face_position'))
apply_state_item('reference_face_distance', args.get('reference_face_distance')) apply_state_item('reference_face_distance', args.get('reference_face_distance'))
apply_state_item('reference_frame_number', args.get('reference_frame_number')) apply_state_item('reference_frame_number', args.get('reference_frame_number'))
apply_state_item('face_tracker_score', args.get('face_tracker_score'))
apply_state_item('face_occluder_model', args.get('face_occluder_model')) apply_state_item('face_occluder_model', args.get('face_occluder_model'))
apply_state_item('face_parser_model', args.get('face_parser_model')) apply_state_item('face_parser_model', args.get('face_parser_model'))
apply_state_item('face_mask_types', args.get('face_mask_types')) apply_state_item('face_mask_types', args.get('face_mask_types'))
@@ -44,12 +48,13 @@ def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('trim_frame_start', args.get('trim_frame_start')) apply_state_item('trim_frame_start', args.get('trim_frame_start'))
apply_state_item('trim_frame_end', args.get('trim_frame_end')) apply_state_item('trim_frame_end', args.get('trim_frame_end'))
apply_state_item('temp_frame_format', args.get('temp_frame_format')) apply_state_item('temp_frame_format', args.get('temp_frame_format'))
apply_state_item('keep_temp', args.get('keep_temp')) apply_state_item('target_frame_amount', args.get('target_frame_amount'))
apply_state_item('output_image_quality', args.get('output_image_quality')) apply_state_item('output_image_quality', args.get('output_image_quality'))
apply_state_item('output_image_scale', args.get('output_image_scale')) apply_state_item('output_image_scale', args.get('output_image_scale'))
apply_state_item('output_audio_encoder', args.get('output_audio_encoder')) apply_state_item('output_audio_encoder', args.get('output_audio_encoder'))
apply_state_item('output_audio_quality', args.get('output_audio_quality')) apply_state_item('output_audio_quality', args.get('output_audio_quality'))
apply_state_item('output_audio_volume', args.get('output_audio_volume')) apply_state_item('output_audio_volume', args.get('output_audio_volume'))
apply_state_item('output_audio_fps', normalize_fps(args.get('output_audio_fps')))
apply_state_item('output_video_encoder', args.get('output_video_encoder')) apply_state_item('output_video_encoder', args.get('output_video_encoder'))
apply_state_item('output_video_preset', args.get('output_video_preset')) apply_state_item('output_video_preset', args.get('output_video_preset'))
apply_state_item('output_video_quality', args.get('output_video_quality')) apply_state_item('output_video_quality', args.get('output_video_quality'))
@@ -65,9 +70,6 @@ def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
for processor_module in get_processors_modules(available_processors): for processor_module in get_processors_modules(available_processors):
processor_module.apply_args(args, apply_state_item) processor_module.apply_args(args, apply_state_item)
apply_state_item('open_browser', args.get('open_browser'))
apply_state_item('ui_layouts', args.get('ui_layouts'))
apply_state_item('ui_workflow', args.get('ui_workflow'))
apply_state_item('execution_device_ids', args.get('execution_device_ids')) apply_state_item('execution_device_ids', args.get('execution_device_ids'))
apply_state_item('execution_providers', args.get('execution_providers')) apply_state_item('execution_providers', args.get('execution_providers'))
apply_state_item('execution_thread_count', args.get('execution_thread_count')) apply_state_item('execution_thread_count', args.get('execution_thread_count'))
@@ -76,8 +78,10 @@ def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('benchmark_mode', args.get('benchmark_mode')) apply_state_item('benchmark_mode', args.get('benchmark_mode'))
apply_state_item('benchmark_resolutions', args.get('benchmark_resolutions')) apply_state_item('benchmark_resolutions', args.get('benchmark_resolutions'))
apply_state_item('benchmark_cycle_count', args.get('benchmark_cycle_count')) apply_state_item('benchmark_cycle_count', args.get('benchmark_cycle_count'))
apply_state_item('api_host', args.get('api_host'))
apply_state_item('api_port', args.get('api_port'))
apply_state_item('api_security_strategy', args.get('api_security_strategy'))
apply_state_item('video_memory_strategy', args.get('video_memory_strategy')) apply_state_item('video_memory_strategy', args.get('video_memory_strategy'))
apply_state_item('system_memory_limit', args.get('system_memory_limit'))
apply_state_item('log_level', args.get('log_level')) apply_state_item('log_level', args.get('log_level'))
apply_state_item('halt_on_error', args.get('halt_on_error')) apply_state_item('halt_on_error', args.get('halt_on_error'))
apply_state_item('job_id', args.get('job_id')) apply_state_item('job_id', args.get('job_id'))
@@ -85,33 +89,41 @@ def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('step_index', args.get('step_index')) apply_state_item('step_index', args.get('step_index'))
def reduce_step_args(args : Args) -> Args: def extract_api_args(state : Union[State, ProcessorState]) -> Args:
api_args =\
{
key: state.get(key) for key in state if key in get_api_arguments()
}
return api_args
def extract_cli_args(state : Union[State, ProcessorState]) -> Args:
cli_args =\
{
key: state.get(key) for key in state if key in get_cli_arguments()
}
return cli_args
def extract_sys_args(state : Union[State, ProcessorState]) -> Args:
sys_args =\
{
key: state.get(key) for key in state if key in get_sys_arguments()
}
return sys_args
def extract_step_args(state : Union[State, ProcessorState]) -> Args:
step_args =\ step_args =\
{ {
key: args[key] for key in args if key in job_store.get_step_keys() key: state.get(key) for key in state if key in get_cli_arguments() and key not in get_sys_arguments()
} }
return step_args return step_args
def reduce_job_args(args : Args) -> Args: def filter_step_args(args : Args) -> Args:
job_args =\
{
key: args[key] for key in args if key in job_store.get_job_keys()
}
return job_args
def collect_step_args() -> Args:
step_args =\ step_args =\
{ {
key: state_manager.get_item(key) for key in job_store.get_step_keys() #type:ignore[arg-type] key: args.get(key) for key in args if key in get_cli_arguments() and key not in get_sys_arguments()
} }
return step_args return step_args
def collect_job_args() -> Args:
job_args =\
{
key: state_manager.get_item(key) for key in job_store.get_job_keys() #type:ignore[arg-type]
}
return job_args
+30 -2
View File
@@ -1,5 +1,6 @@
import math
from functools import lru_cache from functools import lru_cache
from typing import Any, List, Optional from typing import Any, List, Optional, Tuple
import numpy import numpy
import scipy import scipy
@@ -7,7 +8,8 @@ from numpy.typing import NDArray
from facefusion.ffmpeg import read_audio_buffer from facefusion.ffmpeg import read_audio_buffer
from facefusion.filesystem import is_audio from facefusion.filesystem import is_audio
from facefusion.types import Audio, AudioFrame, Fps, Mel, MelFilterBank, Spectrogram from facefusion.media_helper import restrict_trim_frame
from facefusion.types import Audio, AudioFrame, Duration, Fps, Mel, MelFilterBank, Spectrogram
from facefusion.voice_extractor import batch_extract_voice from facefusion.voice_extractor import batch_extract_voice
@@ -141,3 +143,29 @@ def create_spectrogram(audio : Audio) -> Spectrogram:
spectrogram = scipy.signal.stft(audio, nperseg = mel_bin_total, nfft = mel_bin_total, noverlap = mel_bin_overlap)[2] spectrogram = scipy.signal.stft(audio, nperseg = mel_bin_total, nfft = mel_bin_total, noverlap = mel_bin_overlap)[2]
spectrogram = numpy.dot(mel_filter_bank, numpy.abs(spectrogram)) spectrogram = numpy.dot(mel_filter_bank, numpy.abs(spectrogram))
return spectrogram return spectrogram
def count_audio_frame_total(audio_path : str, fps : Fps) -> int:
audio_duration = detect_audio_duration(audio_path)
if audio_duration > 0:
return math.ceil(audio_duration * fps)
return 0
def detect_audio_duration(audio_path : str) -> Duration:
audio_sample_rate = 48000
audio_sample_size = 16
audio_channel_total = 2
if is_audio(audio_path):
audio_buffer = read_audio_buffer(audio_path, audio_sample_rate, audio_sample_size, audio_channel_total)
if audio_buffer:
audio = numpy.frombuffer(audio_buffer, dtype = numpy.int16).reshape(-1, audio_channel_total)
audio_duration = len(audio) / audio_sample_rate
return audio_duration
return 0
def restrict_trim_audio_frame(audio_path : str, fps : Fps, trim_frame_start : Optional[int], trim_frame_end : Optional[int]) -> Tuple[int, int]:
audio_frame_total = count_audio_frame_total(audio_path, fps)
return restrict_trim_frame(audio_frame_total, trim_frame_start, trim_frame_end)
+2 -2
View File
@@ -9,7 +9,7 @@ import facefusion.choices
from facefusion import content_analyser, core, state_manager from facefusion import content_analyser, core, state_manager
from facefusion.cli_helper import render_table from facefusion.cli_helper import render_table
from facefusion.download import conditional_download, resolve_download_url from facefusion.download import conditional_download, resolve_download_url
from facefusion.face_store import clear_static_faces from facefusion.face_store import clear_faces
from facefusion.filesystem import get_file_extension from facefusion.filesystem import get_file_extension
from facefusion.types import BenchmarkCycleSet from facefusion.types import BenchmarkCycleSet
from facefusion.vision import count_video_frame_total, detect_video_fps from facefusion.vision import count_video_frame_total, detect_video_fps
@@ -64,7 +64,7 @@ def cycle(cycle_count : int) -> BenchmarkCycleSet:
if state_manager.get_item('benchmark_mode') == 'cold': if state_manager.get_item('benchmark_mode') == 'cold':
content_analyser.analyse_image.cache_clear() content_analyser.analyse_image.cache_clear()
content_analyser.analyse_video.cache_clear() content_analyser.analyse_video.cache_clear()
clear_static_faces() clear_faces()
start_time = perf_counter() start_time = perf_counter()
core.conditional_process() core.conditional_process()
+65
View File
@@ -0,0 +1,65 @@
from argparse import Action
from typing import Dict, List
from facefusion.types import CapabilityGroup, CapabilitySet, CapabilityStore, Group, Scope
CAPABILITY_STORE : CapabilityStore =\
{
'api': {},
'cli': {},
'sys': {}
}
def get_api_capability_group() -> CapabilityGroup:
capability_group : CapabilityGroup = {}
for name, value in CAPABILITY_STORE.get('api').items():
for group in value.get('groups'):
capability_group.setdefault(group, {})[name] = value
return capability_group
def get_api_capability_set() -> Dict[str, CapabilitySet]:
return CAPABILITY_STORE.get('api')
def get_cli_capability_set() -> Dict[str, CapabilitySet]:
return CAPABILITY_STORE.get('cli')
def get_sys_capability_set() -> Dict[str, CapabilitySet]:
return CAPABILITY_STORE.get('sys')
def get_api_arguments() -> List[str]:
return list(get_api_capability_set().keys())
def get_cli_arguments() -> List[str]:
return list(get_cli_capability_set().keys())
def get_sys_arguments() -> List[str]:
return list(get_sys_capability_set().keys())
def register_capability_set(actions : List[Action], scopes : List[Scope], groups : List[Group]) -> None:
for action in actions:
value : CapabilitySet =\
{
'default': action.default,
'groups': groups
}
if action.choices:
value['choices'] = list(action.choices)
for scope in scopes:
if scope == 'api':
CAPABILITY_STORE['api'][action.dest] = value
if scope == 'cli':
CAPABILITY_STORE['cli'][action.dest] = value
if scope == 'sys':
CAPABILITY_STORE['sys'][action.dest] = value
+37 -36
View File
@@ -2,7 +2,7 @@ import logging
from typing import List, Sequence, get_args from typing import List, Sequence, get_args
from facefusion.common_helper import create_float_range, create_int_range from facefusion.common_helper import create_float_range, create_int_range
from facefusion.types import Angle, AudioEncoder, AudioFormat, AudioTypeSet, BenchmarkMode, BenchmarkResolution, BenchmarkSet, DownloadProvider, DownloadProviderSet, DownloadScope, EncoderSet, ExecutionProvider, ExecutionProviderSet, FaceDetectorModel, FaceDetectorSet, FaceLandmarkerModel, FaceMaskArea, FaceMaskAreaSet, FaceMaskRegion, FaceMaskRegionSet, FaceMaskType, FaceOccluderModel, FaceParserModel, FaceSelectorMode, FaceSelectorOrder, Gender, ImageFormat, ImageTypeSet, JobStatus, LogLevel, LogLevelSet, Race, Score, TempFrameFormat, UiWorkflow, VideoEncoder, VideoFormat, VideoMemoryStrategy, VideoPreset, VideoTypeSet, VoiceExtractorModel from facefusion.types import Angle, ApiSecurityStrategy, AudioEncoder, AudioFormat, AudioSet, BenchmarkMode, BenchmarkResolution, BenchmarkSet, DownloadProvider, DownloadProviderSet, DownloadScope, ExecutionProvider, ExecutionProviderSet, FaceDetectorModel, FaceDetectorSet, FaceLandmarkerModel, FaceMaskArea, FaceMaskAreaSet, FaceMaskRegion, FaceMaskRegionSet, FaceMaskType, FaceOccluderModel, FaceParserModel, FaceSelectorGender, FaceSelectorMode, FaceSelectorOrder, FaceSelectorRace, Gender, ImageEncoder, ImageFormat, ImageSet, JobStatus, LogLevel, LogLevelSet, Race, Score, TempFrameFormat, VideoEncoder, VideoFormat, VideoMemoryStrategy, VideoPreset, VideoSet, VoiceExtractorModel, WorkFlow
face_detector_set : FaceDetectorSet =\ face_detector_set : FaceDetectorSet =\
{ {
@@ -16,8 +16,10 @@ face_detector_models : List[FaceDetectorModel] = list(get_args(FaceDetectorModel
face_landmarker_models : List[FaceLandmarkerModel] = list(get_args(FaceLandmarkerModel)) face_landmarker_models : List[FaceLandmarkerModel] = list(get_args(FaceLandmarkerModel))
face_selector_modes : List[FaceSelectorMode] = list(get_args(FaceSelectorMode)) face_selector_modes : List[FaceSelectorMode] = list(get_args(FaceSelectorMode))
face_selector_orders : List[FaceSelectorOrder] = list(get_args(FaceSelectorOrder)) face_selector_orders : List[FaceSelectorOrder] = list(get_args(FaceSelectorOrder))
face_selector_genders : List[Gender] = list(get_args(Gender)) genders : List[Gender] = list(get_args(Gender))
face_selector_races : List[Race] = list(get_args(Race)) races : List[Race] = list(get_args(Race))
face_selector_genders : List[FaceSelectorGender] = list(get_args(FaceSelectorGender))
face_selector_races : List[FaceSelectorRace] = list(get_args(FaceSelectorRace))
face_occluder_models : List[FaceOccluderModel] = list(get_args(FaceOccluderModel)) face_occluder_models : List[FaceOccluderModel] = list(get_args(FaceOccluderModel))
face_parser_models : List[FaceParserModel] = list(get_args(FaceParserModel)) face_parser_models : List[FaceParserModel] = list(get_args(FaceParserModel))
face_mask_types : List[FaceMaskType] = list(get_args(FaceMaskType)) face_mask_types : List[FaceMaskType] = list(get_args(FaceMaskType))
@@ -45,48 +47,46 @@ face_mask_regions : List[FaceMaskRegion] = list(get_args(FaceMaskRegion))
voice_extractor_models : List[VoiceExtractorModel] = list(get_args(VoiceExtractorModel)) voice_extractor_models : List[VoiceExtractorModel] = list(get_args(VoiceExtractorModel))
audio_type_set : AudioTypeSet =\ workflows : List[WorkFlow] = [ 'auto', 'audio-to-image:frames', 'audio-to-image:video', 'image-to-image', 'image-to-video', 'image-to-video:frames' ]
audio_set : AudioSet =\
{ {
'flac': 'audio/flac', 'flac': 'flac',
'm4a': 'audio/mp4', 'm4a': 'aac',
'mp3': 'audio/mpeg', 'mp3': 'libmp3lame',
'ogg': 'audio/ogg', 'ogg': 'flac',
'opus': 'audio/opus', 'opus': 'libopus',
'wav': 'audio/x-wav' 'wav': 'pcm_s16le'
} }
image_type_set : ImageTypeSet =\ image_set : ImageSet =\
{ {
'bmp': 'image/bmp', 'bmp': 'bmp',
'jpeg': 'image/jpeg', 'jpeg': 'mjpeg',
'png': 'image/png', 'png': 'png',
'tiff': 'image/tiff', 'tiff': 'tiff',
'webp': 'image/webp' 'webp': 'libwebp'
} }
video_type_set : VideoTypeSet =\ video_set : VideoSet =\
{ {
'avi': 'video/x-msvideo', 'avi': 'mpeg4',
'm4v': 'video/mp4', 'm4v': 'libx264',
'mkv': 'video/x-matroska', 'mkv': 'libx264',
'mp4': 'video/mp4', 'mov': 'libx264',
'mpeg': 'video/mpeg', 'mp4': 'libx264',
'mov': 'video/quicktime', 'mpeg': 'mpeg1video',
'mxf': 'application/mxf', 'mxf': 'mpeg2video',
'webm': 'video/webm', 'webm': 'libvpx-vp9',
'wmv': 'video/x-ms-wmv' 'wmv': 'msmpeg4'
} }
audio_formats : List[AudioFormat] = list(get_args(AudioFormat)) audio_formats : List[AudioFormat] = list(get_args(AudioFormat))
image_formats : List[ImageFormat] = list(get_args(ImageFormat)) image_formats : List[ImageFormat] = list(get_args(ImageFormat))
video_formats : List[VideoFormat] = list(get_args(VideoFormat)) video_formats : List[VideoFormat] = list(get_args(VideoFormat))
temp_frame_formats : List[TempFrameFormat] = list(get_args(TempFrameFormat)) temp_frame_formats : List[TempFrameFormat] = list(get_args(TempFrameFormat))
output_audio_encoders : List[AudioEncoder] = list(get_args(AudioEncoder)) audio_encoders : List[AudioEncoder] = list(get_args(AudioEncoder))
output_video_encoders : List[VideoEncoder] = list(get_args(VideoEncoder)) image_encoders : List[ImageEncoder] = list(get_args(ImageEncoder))
output_encoder_set : EncoderSet =\ video_encoders : List[VideoEncoder] = list(get_args(VideoEncoder))
{ video_presets : List[VideoPreset] = list(get_args(VideoPreset))
'audio': output_audio_encoders,
'video': output_video_encoders
}
output_video_presets : List[VideoPreset] = list(get_args(VideoPreset))
benchmark_modes : List[BenchmarkMode] = list(get_args(BenchmarkMode)) benchmark_modes : List[BenchmarkMode] = list(get_args(BenchmarkMode))
benchmark_set : BenchmarkSet =\ benchmark_set : BenchmarkSet =\
@@ -138,6 +138,7 @@ download_providers : List[DownloadProvider] = list(get_args(DownloadProvider))
download_scopes : List[DownloadScope] = list(get_args(DownloadScope)) download_scopes : List[DownloadScope] = list(get_args(DownloadScope))
video_memory_strategies : List[VideoMemoryStrategy] = list(get_args(VideoMemoryStrategy)) video_memory_strategies : List[VideoMemoryStrategy] = list(get_args(VideoMemoryStrategy))
api_security_strategies : List[ApiSecurityStrategy] = list(get_args(ApiSecurityStrategy))
log_level_set : LogLevelSet =\ log_level_set : LogLevelSet =\
{ {
@@ -148,12 +149,10 @@ log_level_set : LogLevelSet =\
} }
log_levels : List[LogLevel] = list(get_args(LogLevel)) log_levels : List[LogLevel] = list(get_args(LogLevel))
ui_workflows : List[UiWorkflow] = list(get_args(UiWorkflow))
job_statuses : List[JobStatus] = list(get_args(JobStatus)) job_statuses : List[JobStatus] = list(get_args(JobStatus))
benchmark_cycle_count_range : Sequence[int] = create_int_range(1, 10, 1) benchmark_cycle_count_range : Sequence[int] = create_int_range(1, 10, 1)
execution_thread_count_range : Sequence[int] = create_int_range(1, 32, 1) execution_thread_count_range : Sequence[int] = create_int_range(1, 32, 1)
system_memory_limit_range : Sequence[int] = create_int_range(0, 128, 4)
face_detector_margin_range : Sequence[int] = create_int_range(0, 100, 1) face_detector_margin_range : Sequence[int] = create_int_range(0, 100, 1)
face_detector_angles : Sequence[Angle] = create_int_range(0, 270, 90) face_detector_angles : Sequence[Angle] = create_int_range(0, 270, 90)
face_detector_score_range : Sequence[Score] = create_float_range(0.0, 1.0, 0.05) face_detector_score_range : Sequence[Score] = create_float_range(0.0, 1.0, 0.05)
@@ -162,6 +161,8 @@ face_mask_blur_range : Sequence[float] = create_float_range(0.0, 1.0, 0.05)
face_mask_padding_range : Sequence[int] = create_int_range(0, 100, 1) face_mask_padding_range : Sequence[int] = create_int_range(0, 100, 1)
face_selector_age_range : Sequence[int] = create_int_range(0, 100, 1) face_selector_age_range : Sequence[int] = create_int_range(0, 100, 1)
reference_face_distance_range : Sequence[float] = create_float_range(0.0, 1.0, 0.05) reference_face_distance_range : Sequence[float] = create_float_range(0.0, 1.0, 0.05)
face_tracker_score_range : Sequence[Score] = create_float_range(0.0, 0.5, 0.05)
target_frame_amount_range : Sequence[int] = create_int_range(0, 10, 1)
output_image_quality_range : Sequence[int] = create_int_range(0, 100, 1) output_image_quality_range : Sequence[int] = create_int_range(0, 100, 1)
output_image_scale_range : Sequence[float] = create_float_range(0.25, 8.0, 0.25) output_image_scale_range : Sequence[float] = create_float_range(0.25, 8.0, 0.25)
output_audio_quality_range : Sequence[int] = create_int_range(0, 100, 1) output_audio_quality_range : Sequence[int] = create_int_range(0, 100, 1)
View File
+70
View File
@@ -0,0 +1,70 @@
import ctypes
import struct
from typing import Optional
from facefusion.libraries import aom as aom_module
from facefusion.types import AomDecoder, Buffer, BufferPack
def create(thread_count : int) -> Optional[AomDecoder]:
aom_library = aom_module.create_static_library()
if aom_library:
aom_decoder = ctypes.create_string_buffer(128)
aom_codec = ctypes.c_void_p.in_dll(aom_library, 'aom_codec_av1_dx_algo')
config_buffer = ctypes.create_string_buffer(128)
struct.pack_into('I', config_buffer, 0, thread_count)
struct.pack_into('I', config_buffer, 12, 1)
if aom_library.aom_codec_dec_init_ver(aom_decoder, ctypes.byref(aom_codec), config_buffer, 0, 22) == 0:
return aom_decoder
return None
def decode(aom_decoder : AomDecoder, input_buffer : Buffer) -> Optional[BufferPack]:
aom_library = aom_module.create_static_library()
if aom_library and input_buffer:
input_total = len(input_buffer)
temp_buffer = ctypes.create_string_buffer(input_buffer)
if aom_library.aom_codec_decode(aom_decoder, temp_buffer, input_total, None) == 0:
address = aom_library.aom_codec_get_frame(aom_decoder, ctypes.byref(ctypes.c_void_p(0)))
if address:
frame_width = ctypes.c_uint.from_address(address + 28).value & ~1
frame_height = ctypes.c_uint.from_address(address + 32).value & ~1
return BufferPack(
buffer = collect(address, frame_width, frame_height),
resolution = (frame_width, frame_height)
)
return None
def collect(address : int, frame_width : int, frame_height : int) -> Buffer:
output_parts = []
for index in range(3):
plane_pointer = ctypes.c_void_p.from_address(address + 64 + index * 8).value
stride = ctypes.c_int.from_address(address + 88 + index * 4).value
plane_width = frame_width >> (index > 0)
plane_height = frame_height >> (index > 0)
if stride == plane_width:
output_parts.append(ctypes.string_at(plane_pointer, plane_width * plane_height))
else:
for row in range(plane_height):
output_parts.append(ctypes.string_at(plane_pointer + row * stride, plane_width))
return bytes().join(output_parts)
def destroy(aom_decoder : AomDecoder) -> None:
aom_library = aom_module.create_static_library()
if aom_library:
aom_library.aom_codec_destroy(aom_decoder)
+94
View File
@@ -0,0 +1,94 @@
import ctypes
import struct
from typing import Optional
from facefusion.libraries import aom as aom_module
from facefusion.types import AomEncoder, BitRate, Buffer, Resolution
def create(frame_resolution : Resolution, bitrate : BitRate, thread_count : int, cpu_count : int) -> Optional[AomEncoder]:
aom_library = aom_module.create_static_library()
if aom_library:
aom_encoder = ctypes.create_string_buffer(1152)
aom_codec = ctypes.c_void_p.in_dll(aom_library, 'aom_codec_av1_cx_algo')
config_buffer = ctypes.create_string_buffer(1024)
if aom_library.aom_codec_enc_config_default(ctypes.byref(aom_codec), config_buffer, 1) == 0:
struct.pack_into('I', config_buffer, 4, thread_count)
struct.pack_into('I', config_buffer, 12, frame_resolution[0])
struct.pack_into('I', config_buffer, 16, frame_resolution[1])
struct.pack_into('I', config_buffer, 136, bitrate)
struct.pack_into('I', config_buffer, 192, 30)
if aom_library.aom_codec_enc_init_ver(aom_encoder, ctypes.byref(aom_codec), config_buffer, 0, 25) == 0:
aom_library.aom_codec_control(aom_encoder, 13, ctypes.c_int(cpu_count))
aom_library.aom_codec_control(aom_encoder, 75, ctypes.c_int(1))
aom_library.aom_codec_control(aom_encoder, 106, ctypes.c_int(0))
aom_library.aom_codec_control(aom_encoder, 122, ctypes.c_int(0))
aom_library.aom_codec_control(aom_encoder, 123, ctypes.c_int(0))
ctypes.memmove(ctypes.addressof(aom_encoder) + 128, config_buffer, 1024)
return aom_encoder
return None
def encode(aom_encoder : AomEncoder, input_buffer : Buffer, frame_resolution : Resolution, frame_index : int) -> Buffer:
aom_library = aom_module.create_static_library()
output_buffer = bytes()
if aom_library:
temp_buffer = ctypes.create_string_buffer(256)
encode_buffer = ctypes.create_string_buffer(input_buffer)
if aom_library.aom_img_wrap(temp_buffer, 0x102, frame_resolution[0], frame_resolution[1], 1, encode_buffer) and aom_library.aom_codec_encode(aom_encoder, temp_buffer, frame_index, 1, 0, 1) == 0:
output_buffer = collect(aom_encoder)
return output_buffer
def collect(aom_encoder : AomEncoder) -> Buffer:
aom_library = aom_module.create_static_library()
output_parts = []
packet_cursor = ctypes.c_void_p(0)
packet = aom_library.aom_codec_get_cx_data(aom_encoder, ctypes.byref(packet_cursor))
while packet:
if ctypes.c_int.from_address(packet).value == 0:
buffer_pointer = ctypes.c_void_p.from_address(packet + 8).value
buffer_size = ctypes.c_size_t.from_address(packet + 16).value
output_parts.append(ctypes.string_at(buffer_pointer, buffer_size))
packet = aom_library.aom_codec_get_cx_data(aom_encoder, ctypes.byref(packet_cursor))
return bytes().join(output_parts)
def update_resolution(aom_encoder : AomEncoder, frame_resolution : Resolution) -> bool:
aom_library = aom_module.create_static_library()
if aom_library:
struct.pack_into('I', aom_encoder, 128 + 12, frame_resolution[0])
struct.pack_into('I', aom_encoder, 128 + 16, frame_resolution[1])
return aom_library.aom_codec_enc_config_set(aom_encoder, ctypes.cast(ctypes.addressof(aom_encoder) + 128, ctypes.c_void_p)) == 0
return False
def update_bitrate(aom_encoder : AomEncoder, bitrate : BitRate) -> bool:
aom_library = aom_module.create_static_library()
if aom_library:
struct.pack_into('I', aom_encoder, 128 + 136, bitrate)
return aom_library.aom_codec_enc_config_set(aom_encoder, ctypes.cast(ctypes.addressof(aom_encoder) + 128, ctypes.c_void_p)) == 0
return False
def destroy(aom_encoder : AomEncoder) -> None:
aom_library = aom_module.create_static_library()
if aom_library:
aom_library.aom_codec_destroy(aom_encoder)
+38
View File
@@ -0,0 +1,38 @@
import ctypes
from typing import Optional
from facefusion.libraries import opus as opus_module
from facefusion.types import Buffer, OpusDecoder
def create(sample_rate : int, channel_total : int) -> Optional[OpusDecoder]:
opus_library = opus_module.create_static_library()
if opus_library:
return opus_library.opus_decoder_create(sample_rate, channel_total, ctypes.byref(ctypes.c_int(0)))
return None
def decode(opus_decoder : OpusDecoder, input_buffer : Buffer, channel_total : int) -> Buffer:
opus_library = opus_module.create_static_library()
output_buffer = bytes()
if opus_library:
input_total = len(input_buffer)
sample_size = ctypes.sizeof(ctypes.c_float)
sample_total = opus_library.opus_decoder_get_nb_samples(opus_decoder, input_buffer, input_total)
sample_buffer = (ctypes.c_float * (sample_total * channel_total))()
output_total = opus_library.opus_decode_float(opus_decoder, input_buffer, input_total, sample_buffer, sample_total, 0)
if output_total:
output_buffer = ctypes.string_at(ctypes.addressof(sample_buffer), output_total * channel_total * sample_size)
return output_buffer
def destroy(opus_decoder : OpusDecoder) -> None:
opus_library = opus_module.create_static_library()
if opus_library:
opus_library.opus_decoder_destroy(opus_decoder)
+38
View File
@@ -0,0 +1,38 @@
import ctypes
from typing import Optional
from facefusion.libraries import opus as opus_module
from facefusion.types import Buffer, OpusEncoder
def create(sample_rate : int, channel_total : int) -> Optional[OpusEncoder]:
opus_library = opus_module.create_static_library()
if opus_library:
return opus_library.opus_encoder_create(sample_rate, channel_total, 2049, ctypes.byref(ctypes.c_int(0)))
return None
def encode(opus_encoder : OpusEncoder, input_buffer : Buffer, channel_total : int) -> Buffer:
opus_library = opus_module.create_static_library()
output_buffer = bytes()
if opus_library:
sample_size = ctypes.sizeof(ctypes.c_float)
sample_total = len(input_buffer) // (sample_size * channel_total)
sample_buffer = (ctypes.c_float * (sample_total * channel_total)).from_buffer_copy(input_buffer)
temp_buffer = ctypes.create_string_buffer(2048)
output_total = opus_library.opus_encode_float(opus_encoder, sample_buffer, sample_total, temp_buffer, 2048)
if output_total:
output_buffer = temp_buffer.raw[:output_total]
return output_buffer
def destroy(opus_encoder : OpusEncoder) -> None:
opus_library = opus_module.create_static_library()
if opus_library:
opus_library.opus_encoder_destroy(opus_encoder)
+74
View File
@@ -0,0 +1,74 @@
import ctypes
import struct
from typing import Optional
from facefusion.libraries import vpx as vpx_module
from facefusion.types import Buffer, BufferPack, VpxDecoder, VxpVideoCodec
def create(video_codec : VxpVideoCodec, thread_count : int) -> Optional[VpxDecoder]:
vpx_library = vpx_module.create_static_library()
if vpx_library:
vpx_decoder = ctypes.create_string_buffer(64)
vpx_algo = 'vpx_codec_vp8_dx_algo'
if video_codec == 'vp9':
vpx_algo = 'vpx_codec_vp9_dx_algo'
vpx_codec = ctypes.c_void_p.in_dll(vpx_library, vpx_algo)
config_buffer = ctypes.create_string_buffer(128)
struct.pack_into('I', config_buffer, 0, thread_count)
if vpx_library.vpx_codec_dec_init_ver(vpx_decoder, ctypes.byref(vpx_codec), config_buffer, 0, 12) == 0:
return vpx_decoder
return None
def decode(vpx_decoder : VpxDecoder, input_buffer : Buffer) -> Optional[BufferPack]:
vpx_library = vpx_module.create_static_library()
if vpx_library and input_buffer:
input_total = len(input_buffer)
temp_buffer = ctypes.create_string_buffer(input_buffer)
if vpx_library.vpx_codec_decode(vpx_decoder, temp_buffer, input_total, None, 0) == 0:
address = vpx_library.vpx_codec_get_frame(vpx_decoder, ctypes.byref(ctypes.c_void_p(0)))
if address:
frame_width = ctypes.c_uint.from_address(address + 24).value & ~1
frame_height = ctypes.c_uint.from_address(address + 28).value & ~1
return BufferPack(
buffer = collect(address, frame_width, frame_height),
resolution = (frame_width, frame_height)
)
return None
def collect(address : int, frame_width : int, frame_height : int) -> Buffer:
output_parts = []
for index in range(3):
plane_pointer = ctypes.c_void_p.from_address(address + 48 + index * 8).value
stride = ctypes.c_int.from_address(address + 80 + index * 4).value
plane_width = frame_width >> (index > 0)
plane_height = frame_height >> (index > 0)
if stride == plane_width:
output_parts.append(ctypes.string_at(plane_pointer, plane_width * plane_height))
else:
for row in range(plane_height):
output_parts.append(ctypes.string_at(plane_pointer + row * stride, plane_width))
return bytes().join(output_parts)
def destroy(vpx_decoder : VpxDecoder) -> None:
vpx_library = vpx_module.create_static_library()
if vpx_library:
vpx_library.vpx_codec_destroy(vpx_decoder)
+104
View File
@@ -0,0 +1,104 @@
import ctypes
import struct
from typing import Optional
from facefusion.libraries import vpx as vpx_module
from facefusion.types import BitRate, Buffer, Resolution, VpxEncoder, VxpVideoCodec
def create(video_codec : VxpVideoCodec, frame_resolution : Resolution, bitrate : BitRate, thread_count : int, cpu_count : int) -> Optional[VpxEncoder]:
vpx_library = vpx_module.create_static_library()
if vpx_library:
vpx_encoder = ctypes.create_string_buffer(640)
vpx_algo = 'vpx_codec_vp8_cx_algo'
if video_codec == 'vp9':
vpx_algo = 'vpx_codec_vp9_cx_algo'
vpx_codec = ctypes.c_void_p.in_dll(vpx_library, vpx_algo)
config_buffer = ctypes.create_string_buffer(512)
if vpx_library.vpx_codec_enc_config_default(ctypes.byref(vpx_codec), config_buffer, 0) == 0:
struct.pack_into('I', config_buffer, 4, thread_count)
struct.pack_into('I', config_buffer, 12, frame_resolution[0])
struct.pack_into('I', config_buffer, 16, frame_resolution[1])
struct.pack_into('I', config_buffer, 28, 1)
struct.pack_into('I', config_buffer, 36, 0)
struct.pack_into('I', config_buffer, 44, 0)
struct.pack_into('I', config_buffer, 72, 0)
struct.pack_into('I', config_buffer, 112, bitrate)
struct.pack_into('I', config_buffer, 116, 2)
struct.pack_into('I', config_buffer, 120, 50)
struct.pack_into('I', config_buffer, 124, 50)
struct.pack_into('I', config_buffer, 128, 50)
if vpx_library.vpx_codec_enc_init_ver(vpx_encoder, ctypes.byref(vpx_codec), config_buffer, 0, 39) == 0:
vpx_library.vpx_codec_control_(vpx_encoder, 13, ctypes.c_int(cpu_count))
vpx_library.vpx_codec_control_(vpx_encoder, 12, ctypes.c_int(0))
vpx_library.vpx_codec_control_(vpx_encoder, 27, ctypes.c_int(10))
ctypes.memmove(ctypes.addressof(vpx_encoder) + 64, config_buffer, 512)
return vpx_encoder
return None
def encode(vpx_encoder : VpxEncoder, input_buffer : Buffer, frame_resolution : Resolution, frame_index : int) -> Buffer:
vpx_library = vpx_module.create_static_library()
output_buffer = bytes()
if vpx_library:
temp_buffer = ctypes.create_string_buffer(256)
encode_buffer = ctypes.create_string_buffer(input_buffer)
if vpx_library.vpx_img_wrap(temp_buffer, 0x102, frame_resolution[0], frame_resolution[1], 1, encode_buffer) and vpx_library.vpx_codec_encode(vpx_encoder, temp_buffer, frame_index, 1, 0, 1) == 0:
output_buffer = collect(vpx_encoder)
return output_buffer
def collect(vpx_encoder : VpxEncoder) -> Buffer:
vpx_library = vpx_module.create_static_library()
output_parts = []
packet_cursor = ctypes.c_void_p(0)
packet = vpx_library.vpx_codec_get_cx_data(vpx_encoder, ctypes.byref(packet_cursor))
while packet:
if ctypes.c_int.from_address(packet).value == 0:
buffer_pointer = ctypes.c_void_p.from_address(packet + 8).value
buffer_size = ctypes.c_size_t.from_address(packet + 16).value
output_parts.append(ctypes.string_at(buffer_pointer, buffer_size))
packet = vpx_library.vpx_codec_get_cx_data(vpx_encoder, ctypes.byref(packet_cursor))
return bytes().join(output_parts)
def update_resolution(vpx_encoder : VpxEncoder, frame_resolution : Resolution) -> bool:
vpx_library = vpx_module.create_static_library()
if vpx_library:
struct.pack_into('I', vpx_encoder, 64 + 12, frame_resolution[0])
struct.pack_into('I', vpx_encoder, 64 + 16, frame_resolution[1])
return vpx_library.vpx_codec_enc_config_set(vpx_encoder, ctypes.cast(ctypes.addressof(vpx_encoder) + 64, ctypes.c_void_p)) == 0
return False
def update_bitrate(vpx_encoder : VpxEncoder, bitrate : BitRate) -> bool:
vpx_library = vpx_module.create_static_library()
if vpx_library:
struct.pack_into('I', vpx_encoder, 64 + 112, bitrate)
return vpx_library.vpx_codec_enc_config_set(vpx_encoder, ctypes.cast(ctypes.addressof(vpx_encoder) + 64, ctypes.c_void_p)) == 0
return False
def destroy(vpx_encoder : VpxEncoder) -> None:
vpx_library = vpx_module.create_static_library()
if vpx_library:
vpx_library.vpx_codec_destroy(vpx_encoder)
+6
View File
@@ -78,6 +78,12 @@ def get_first(__list__ : Any) -> Any:
return None return None
def get_middle(__list__ : Any) -> Any:
if isinstance(__list__, Sequence) and __list__:
return __list__[len(__list__) // 2]
return None
def get_last(__list__ : Any) -> Any: def get_last(__list__ : Any) -> Any:
if isinstance(__list__, Reversible): if isinstance(__list__, Reversible):
return next(reversed(__list__), None) return next(reversed(__list__), None)
+12 -21
View File
@@ -1,29 +1,20 @@
from configparser import ConfigParser from configparser import ConfigParser
from functools import lru_cache
from typing import List, Optional from typing import List, Optional
from facefusion import state_manager from facefusion import state_manager
from facefusion.common_helper import cast_bool, cast_float, cast_int from facefusion.common_helper import cast_bool, cast_float, cast_int
CONFIG_PARSER = None
@lru_cache
def get_config_parser() -> ConfigParser: def get_static_config_parser() -> ConfigParser:
global CONFIG_PARSER config_parser = ConfigParser()
config_parser.read(state_manager.get_item('config_path'), encoding = 'utf-8')
if CONFIG_PARSER is None: return config_parser
CONFIG_PARSER = ConfigParser()
CONFIG_PARSER.read(state_manager.get_item('config_path'), encoding = 'utf-8')
return CONFIG_PARSER
def clear_config_parser() -> None:
global CONFIG_PARSER
CONFIG_PARSER = None
def get_str_value(section : str, option : str, fallback : Optional[str] = None) -> Optional[str]: def get_str_value(section : str, option : str, fallback : Optional[str] = None) -> Optional[str]:
config_parser = get_config_parser() config_parser = get_static_config_parser()
if config_parser.has_option(section, option) and config_parser.get(section, option).strip(): if config_parser.has_option(section, option) and config_parser.get(section, option).strip():
return config_parser.get(section, option) return config_parser.get(section, option)
@@ -31,7 +22,7 @@ def get_str_value(section : str, option : str, fallback : Optional[str] = None)
def get_int_value(section : str, option : str, fallback : Optional[str] = None) -> Optional[int]: def get_int_value(section : str, option : str, fallback : Optional[str] = None) -> Optional[int]:
config_parser = get_config_parser() config_parser = get_static_config_parser()
if config_parser.has_option(section, option) and config_parser.get(section, option).strip(): if config_parser.has_option(section, option) and config_parser.get(section, option).strip():
return config_parser.getint(section, option) return config_parser.getint(section, option)
@@ -39,7 +30,7 @@ def get_int_value(section : str, option : str, fallback : Optional[str] = None)
def get_float_value(section : str, option : str, fallback : Optional[str] = None) -> Optional[float]: def get_float_value(section : str, option : str, fallback : Optional[str] = None) -> Optional[float]:
config_parser = get_config_parser() config_parser = get_static_config_parser()
if config_parser.has_option(section, option) and config_parser.get(section, option).strip(): if config_parser.has_option(section, option) and config_parser.get(section, option).strip():
return config_parser.getfloat(section, option) return config_parser.getfloat(section, option)
@@ -47,7 +38,7 @@ def get_float_value(section : str, option : str, fallback : Optional[str] = None
def get_bool_value(section : str, option : str, fallback : Optional[str] = None) -> Optional[bool]: def get_bool_value(section : str, option : str, fallback : Optional[str] = None) -> Optional[bool]:
config_parser = get_config_parser() config_parser = get_static_config_parser()
if config_parser.has_option(section, option) and config_parser.get(section, option).strip(): if config_parser.has_option(section, option) and config_parser.get(section, option).strip():
return config_parser.getboolean(section, option) return config_parser.getboolean(section, option)
@@ -55,7 +46,7 @@ def get_bool_value(section : str, option : str, fallback : Optional[str] = None)
def get_str_list(section : str, option : str, fallback : Optional[str] = None) -> Optional[List[str]]: def get_str_list(section : str, option : str, fallback : Optional[str] = None) -> Optional[List[str]]:
config_parser = get_config_parser() config_parser = get_static_config_parser()
if config_parser.has_option(section, option) and config_parser.get(section, option).strip(): if config_parser.has_option(section, option) and config_parser.get(section, option).strip():
return config_parser.get(section, option).split() return config_parser.get(section, option).split()
@@ -65,7 +56,7 @@ def get_str_list(section : str, option : str, fallback : Optional[str] = None) -
def get_int_list(section : str, option : str, fallback : Optional[str] = None) -> Optional[List[int]]: def get_int_list(section : str, option : str, fallback : Optional[str] = None) -> Optional[List[int]]:
config_parser = get_config_parser() config_parser = get_static_config_parser()
if config_parser.has_option(section, option) and config_parser.get(section, option).strip(): if config_parser.has_option(section, option) and config_parser.get(section, option).strip():
return list(map(int, config_parser.get(section, option).split())) return list(map(int, config_parser.get(section, option).split()))
+4 -10
View File
@@ -1,16 +1,14 @@
from functools import lru_cache from functools import lru_cache
from typing import List, Tuple from typing import Tuple
import numpy import numpy
from tqdm import tqdm from tqdm import tqdm
from facefusion import inference_manager, state_manager, translator from facefusion import inference_manager, state_manager, translator
from facefusion.common_helper import is_macos
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url
from facefusion.execution import has_execution_provider
from facefusion.filesystem import resolve_relative_path from facefusion.filesystem import resolve_relative_path
from facefusion.thread_helper import conditional_thread_semaphore from facefusion.thread_helper import conditional_thread_semaphore
from facefusion.types import Detection, DownloadScope, DownloadSet, ExecutionProvider, Fps, InferencePool, ModelSet, VisionFrame from facefusion.types import Detection, DownloadScope, DownloadSet, Fps, InferencePool, ModelSet, VisionFrame
from facefusion.vision import detect_video_fps, fit_contain_frame, read_image, read_video_frame from facefusion.vision import detect_video_fps, fit_contain_frame, read_image, read_video_frame
STREAM_COUNTER = 0 STREAM_COUNTER = 0
@@ -119,12 +117,6 @@ def clear_inference_pool() -> None:
inference_manager.clear_inference_pool(__name__, model_names) inference_manager.clear_inference_pool(__name__, model_names)
def resolve_execution_providers() -> List[ExecutionProvider]:
if is_macos() and has_execution_provider('coreml'):
return [ 'cpu' ]
return state_manager.get_item('execution_providers')
def collect_model_downloads() -> Tuple[DownloadSet, DownloadSet]: def collect_model_downloads() -> Tuple[DownloadSet, DownloadSet]:
model_set = create_static_model_set('full') model_set = create_static_model_set('full')
model_hash_set = {} model_hash_set = {}
@@ -175,6 +167,8 @@ def analyse_video(video_path : str, trim_frame_start : int, trim_frame_end : int
for frame_number in frame_range: for frame_number in frame_range:
if frame_number % int(video_fps) == 0: if frame_number % int(video_fps) == 0:
vision_frame = read_video_frame(video_path, frame_number) vision_frame = read_video_frame(video_path, frame_number)
if numpy.any(vision_frame):
total += 1 total += 1
if analyse_frame(vision_frame): if analyse_frame(vision_frame):
+66 -64
View File
@@ -5,19 +5,22 @@ import signal
import sys import sys
from time import time from time import time
from facefusion import benchmarker, cli_helper, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, hash_helper, logger, state_manager, translator, voice_extractor import uvicorn
from facefusion.args import apply_args, collect_job_args, reduce_job_args, reduce_step_args
import facefusion.apis.core
from facefusion import args_helper, benchmarker, cli_helper, content_analyser, hash_helper, logger, state_manager, translator
from facefusion.args_helper import apply_args
from facefusion.download import conditional_download_hashes, conditional_download_sources from facefusion.download import conditional_download_hashes, conditional_download_sources
from facefusion.exit_helper import hard_exit, signal_exit from facefusion.exit_helper import hard_exit, signal_exit
from facefusion.filesystem import get_file_extension, get_file_name, is_image, is_video, resolve_file_paths, resolve_file_pattern from facefusion.filesystem import get_file_extension, has_audio, has_image, has_video
from facefusion.filesystem import get_file_name, resolve_file_paths, resolve_file_pattern
from facefusion.jobs import job_helper, job_manager, job_runner from facefusion.jobs import job_helper, job_manager, job_runner
from facefusion.jobs.job_list import compose_job_list from facefusion.jobs.job_list import compose_job_list
from facefusion.memory import limit_system_memory
from facefusion.processors.core import get_processors_modules from facefusion.processors.core import get_processors_modules
from facefusion.program import create_program from facefusion.program import create_program
from facefusion.program_helper import validate_args from facefusion.program_helper import validate_args
from facefusion.types import Args, ErrorCode from facefusion.types import Args, ErrorCode, WorkFlow
from facefusion.workflows import image_to_image, image_to_video from facefusion.workflows import audio_to_image, audio_to_image_as_frames, image_to_image, image_to_video, image_to_video_as_frames
def cli() -> None: def cli() -> None:
@@ -41,11 +44,6 @@ def cli() -> None:
def route(args : Args) -> None: def route(args : Args) -> None:
system_memory_limit = state_manager.get_item('system_memory_limit')
if system_memory_limit and system_memory_limit > 0:
limit_system_memory(system_memory_limit)
if state_manager.get_item('command') == 'force-download': if state_manager.get_item('command') == 'force-download':
error_code = force_download() error_code = force_download()
hard_exit(error_code) hard_exit(error_code)
@@ -55,37 +53,34 @@ def route(args : Args) -> None:
hard_exit(2) hard_exit(2)
benchmarker.render() benchmarker.render()
if state_manager.get_item('command') == 'api':
if not common_pre_check() or not processors_pre_check() or not facefusion.apis.core.pre_check():
hard_exit(2)
logger.info(translator.get('api_started').format(host = state_manager.get_item('api_host'), port = state_manager.get_item('api_port')), __name__)
uvicorn.run(facefusion.apis.core.create_api(), host = state_manager.get_item('api_host'), port = state_manager.get_item('api_port'))
hard_exit(1)
if state_manager.get_item('command') in [ 'job-list', 'job-create', 'job-submit', 'job-submit-all', 'job-delete', 'job-delete-all', 'job-add-step', 'job-remix-step', 'job-insert-step', 'job-remove-step' ]: if state_manager.get_item('command') in [ 'job-list', 'job-create', 'job-submit', 'job-submit-all', 'job-delete', 'job-delete-all', 'job-add-step', 'job-remix-step', 'job-insert-step', 'job-remove-step' ]:
if not job_manager.init_jobs(state_manager.get_item('jobs_path')): if not job_manager.init_jobs(state_manager.get_jobs_path()):
hard_exit(1) hard_exit(1)
error_code = route_job_manager(args) error_code = route_job_manager(args)
hard_exit(error_code) hard_exit(error_code)
if state_manager.get_item('command') == 'run': if state_manager.get_item('command') == 'run':
import facefusion.uis.core as ui if not job_manager.init_jobs(state_manager.get_jobs_path()):
if not common_pre_check() or not processors_pre_check():
hard_exit(2)
for ui_layout in ui.get_ui_layouts_modules(state_manager.get_item('ui_layouts')):
if not ui_layout.pre_check():
hard_exit(2)
ui.init()
ui.launch()
if state_manager.get_item('command') == 'headless-run':
if not job_manager.init_jobs(state_manager.get_item('jobs_path')):
hard_exit(1) hard_exit(1)
error_code = process_headless(args) error_code = process_headless(args)
hard_exit(error_code) hard_exit(error_code)
if state_manager.get_item('command') == 'batch-run': if state_manager.get_item('command') == 'batch-run':
if not job_manager.init_jobs(state_manager.get_item('jobs_path')): if not job_manager.init_jobs(state_manager.get_jobs_path()):
hard_exit(1) hard_exit(1)
error_code = process_batch(args) error_code = process_batch(args)
hard_exit(error_code) hard_exit(error_code)
if state_manager.get_item('command') in [ 'job-run', 'job-run-all', 'job-retry', 'job-retry-all' ]: if state_manager.get_item('command') in [ 'job-run', 'job-run-all', 'job-retry', 'job-retry-all' ]:
if not job_manager.init_jobs(state_manager.get_item('jobs_path')): if not job_manager.init_jobs(state_manager.get_jobs_path()):
hard_exit(1) hard_exit(1)
error_code = route_job_runner() error_code = route_job_runner()
hard_exit(error_code) hard_exit(error_code)
@@ -107,21 +102,9 @@ def pre_check() -> bool:
def common_pre_check() -> bool: def common_pre_check() -> bool:
common_modules =\
[
content_analyser,
face_classifier,
face_detector,
face_landmarker,
face_masker,
face_recognizer,
voice_extractor
]
content_analyser_content = inspect.getsource(content_analyser).encode() content_analyser_content = inspect.getsource(content_analyser).encode()
content_analyser_hash = hash_helper.create_hash(content_analyser_content)
return all(module.pre_check() for module in common_modules) and content_analyser_hash == 'b14e7b92' return hash_helper.create_hash(content_analyser_content) == '975d67d6'
def processors_pre_check() -> bool: def processors_pre_check() -> bool:
@@ -132,22 +115,19 @@ def processors_pre_check() -> bool:
def force_download() -> ErrorCode: def force_download() -> ErrorCode:
common_modules =\ download_scope = state_manager.get_item('download_scope')
[
content_analyser,
face_classifier,
face_detector,
face_landmarker,
face_masker,
face_recognizer,
voice_extractor
]
available_processors = [ get_file_name(file_path) for file_path in resolve_file_paths('facefusion/processors/modules') ] available_processors = [ get_file_name(file_path) for file_path in resolve_file_paths('facefusion/processors/modules') ]
processor_modules = get_processors_modules(available_processors) processor_modules = get_processors_modules(available_processors)
common_modules = []
for processor_module in processor_modules:
for common_module in processor_module.get_common_modules():
if common_module not in common_modules:
common_modules.append(common_module)
for module in common_modules + processor_modules: for module in common_modules + processor_modules:
if hasattr(module, 'create_static_model_set'): if hasattr(module, 'create_static_model_set'):
for model in module.create_static_model_set(state_manager.get_item('download_scope')).values(): for model in module.create_static_model_set(download_scope).values():
model_hash_set = model.get('hashes') model_hash_set = model.get('hashes')
model_source_set = model.get('sources') model_source_set = model.get('sources')
@@ -203,7 +183,7 @@ def route_job_manager(args : Args) -> ErrorCode:
return 1 return 1
if state_manager.get_item('command') == 'job-add-step': if state_manager.get_item('command') == 'job-add-step':
step_args = reduce_step_args(args) step_args = args_helper.filter_step_args(args)
if job_manager.add_step(state_manager.get_item('job_id'), step_args): if job_manager.add_step(state_manager.get_item('job_id'), step_args):
logger.info(translator.get('job_step_added').format(job_id = state_manager.get_item('job_id')), __name__) logger.info(translator.get('job_step_added').format(job_id = state_manager.get_item('job_id')), __name__)
@@ -212,7 +192,7 @@ def route_job_manager(args : Args) -> ErrorCode:
return 1 return 1
if state_manager.get_item('command') == 'job-remix-step': if state_manager.get_item('command') == 'job-remix-step':
step_args = reduce_step_args(args) step_args = args_helper.filter_step_args(args)
if job_manager.remix_step(state_manager.get_item('job_id'), state_manager.get_item('step_index'), step_args): if job_manager.remix_step(state_manager.get_item('job_id'), state_manager.get_item('step_index'), step_args):
logger.info(translator.get('job_remix_step_added').format(job_id = state_manager.get_item('job_id'), step_index = state_manager.get_item('step_index')), __name__) logger.info(translator.get('job_remix_step_added').format(job_id = state_manager.get_item('job_id'), step_index = state_manager.get_item('step_index')), __name__)
@@ -221,7 +201,7 @@ def route_job_manager(args : Args) -> ErrorCode:
return 1 return 1
if state_manager.get_item('command') == 'job-insert-step': if state_manager.get_item('command') == 'job-insert-step':
step_args = reduce_step_args(args) step_args = args_helper.filter_step_args(args)
if job_manager.insert_step(state_manager.get_item('job_id'), state_manager.get_item('step_index'), step_args): if job_manager.insert_step(state_manager.get_item('job_id'), state_manager.get_item('step_index'), step_args):
logger.info(translator.get('job_step_inserted').format(job_id = state_manager.get_item('job_id'), step_index = state_manager.get_item('step_index')), __name__) logger.info(translator.get('job_step_inserted').format(job_id = state_manager.get_item('job_id'), step_index = state_manager.get_item('step_index')), __name__)
@@ -275,7 +255,7 @@ def route_job_runner() -> ErrorCode:
def process_headless(args : Args) -> ErrorCode: def process_headless(args : Args) -> ErrorCode:
job_id = job_helper.suggest_job_id('headless') job_id = job_helper.suggest_job_id('headless')
step_args = reduce_step_args(args) step_args = args_helper.filter_step_args(args)
if job_manager.create_job(job_id) and job_manager.add_step(job_id, step_args) and job_manager.submit_job(job_id) and job_runner.run_job(job_id, process_step): if job_manager.create_job(job_id) and job_manager.add_step(job_id, step_args) and job_manager.submit_job(job_id) and job_runner.run_job(job_id, process_step):
return 0 return 0
@@ -284,10 +264,9 @@ def process_headless(args : Args) -> ErrorCode:
def process_batch(args : Args) -> ErrorCode: def process_batch(args : Args) -> ErrorCode:
job_id = job_helper.suggest_job_id('batch') job_id = job_helper.suggest_job_id('batch')
step_args = reduce_step_args(args) step_args = args_helper.filter_step_args(args)
job_args = reduce_job_args(args) source_paths = resolve_file_pattern(step_args.get('source_pattern'))
source_paths = resolve_file_pattern(job_args.get('source_pattern')) target_paths = resolve_file_pattern(step_args.get('target_pattern'))
target_paths = resolve_file_pattern(job_args.get('target_pattern'))
if job_manager.create_job(job_id): if job_manager.create_job(job_id):
if source_paths and target_paths: if source_paths and target_paths:
@@ -296,7 +275,7 @@ def process_batch(args : Args) -> ErrorCode:
step_args['target_path'] = target_path step_args['target_path'] = target_path
try: try:
step_args['output_path'] = job_args.get('output_pattern').format(index = index, source_name = get_file_name(source_path), target_name = get_file_name(target_path), target_extension = get_file_extension(target_path)) step_args['output_path'] = step_args.get('output_pattern').format(index = index, source_name = get_file_name(source_path), target_name = get_file_name(target_path), target_extension = get_file_extension(target_path))
except KeyError: except KeyError:
return 1 return 1
@@ -310,7 +289,7 @@ def process_batch(args : Args) -> ErrorCode:
step_args['target_path'] = target_path step_args['target_path'] = target_path
try: try:
step_args['output_path'] = job_args.get('output_pattern').format(index = index, target_name = get_file_name(target_path), target_extension = get_file_extension(target_path)) step_args['output_path'] = step_args.get('output_pattern').format(index = index, target_name = get_file_name(target_path), target_extension = get_file_extension(target_path))
except KeyError: except KeyError:
return 1 return 1
@@ -323,8 +302,10 @@ def process_batch(args : Args) -> ErrorCode:
def process_step(job_id : str, step_index : int, step_args : Args) -> bool: def process_step(job_id : str, step_index : int, step_args : Args) -> bool:
step_total = job_manager.count_step_total(job_id) step_total = job_manager.count_step_total(job_id)
step_args.update(collect_job_args()) cli_args = args_helper.extract_cli_args(state_manager.get_state())
apply_args(step_args, state_manager.set_item) args = cli_args.copy()
args.update(step_args)
apply_args(args, state_manager.set_item)
logger.info(translator.get('processing_step').format(step_current = step_index + 1, step_total = step_total), __name__) logger.info(translator.get('processing_step').format(step_current = step_index + 1, step_total = step_total), __name__)
if common_pre_check() and processors_pre_check(): if common_pre_check() and processors_pre_check():
@@ -336,15 +317,36 @@ def process_step(job_id : str, step_index : int, step_args : Args) -> bool:
def conditional_process() -> ErrorCode: def conditional_process() -> ErrorCode:
start_time = time() start_time = time()
if state_manager.get_item('workflow') == 'auto':
state_manager.set_item('workflow', detect_workflow())
for processor_module in get_processors_modules(state_manager.get_item('processors')): for processor_module in get_processors_modules(state_manager.get_item('processors')):
if not processor_module.pre_process('output'): if not processor_module.pre_process('output'):
return 2 return 2
if is_image(state_manager.get_item('target_path')): if state_manager.get_item('workflow') == 'audio-to-image:video':
return audio_to_image.process(start_time)
if state_manager.get_item('workflow') == 'audio-to-image:frames':
return audio_to_image_as_frames.process(start_time)
if state_manager.get_item('workflow') == 'image-to-image':
return image_to_image.process(start_time) return image_to_image.process(start_time)
if is_video(state_manager.get_item('target_path')): if state_manager.get_item('workflow') == 'image-to-video':
return image_to_video.process(start_time) return image_to_video.process(start_time)
if state_manager.get_item('workflow') == 'image-to-video:frames':
return image_to_video_as_frames.process(start_time)
return 0 return 0
def detect_workflow() -> WorkFlow:
if has_video([ state_manager.get_item('target_path') ]):
if get_file_extension(state_manager.get_item('output_path')):
return 'image-to-video'
return 'image-to-video:frames'
if has_audio(state_manager.get_item('source_paths')) and has_image([ state_manager.get_item('target_path') ]):
if get_file_extension(state_manager.get_item('output_path')):
return 'audio-to-image:video'
return 'audio-to-image:frames'
return 'image-to-image'
+2 -2
View File
@@ -10,10 +10,10 @@ import facefusion.choices
from facefusion import curl_builder, logger, process_manager, state_manager, translator from facefusion import curl_builder, logger, process_manager, state_manager, translator
from facefusion.filesystem import get_file_name, get_file_size, is_file, remove_file from facefusion.filesystem import get_file_name, get_file_size, is_file, remove_file
from facefusion.hash_helper import validate_hash from facefusion.hash_helper import validate_hash
from facefusion.types import Command, DownloadProvider, DownloadSet from facefusion.types import Buffer, Command, DownloadProvider, DownloadSet
def open_curl(commands : List[Command]) -> subprocess.Popen[bytes]: def open_curl(commands : List[Command]) -> subprocess.Popen[Buffer]:
commands = curl_builder.run(commands) commands = curl_builder.run(commands)
return subprocess.Popen(commands, stdin = subprocess.PIPE, stdout = subprocess.PIPE) return subprocess.Popen(commands, stdin = subprocess.PIPE, stdout = subprocess.PIPE)
+14 -75
View File
@@ -1,15 +1,13 @@
import os import os
import shutil
import subprocess
import xml.etree.ElementTree as ElementTree
from functools import lru_cache from functools import lru_cache
from typing import List, Optional from typing import List, Tuple
import onnxruntime import onnxruntime
import facefusion.choices import facefusion.choices
from facefusion.filesystem import create_directory, is_directory from facefusion.filesystem import create_directory, is_directory
from facefusion.types import ExecutionDevice, ExecutionProvider, InferenceOptionSet, InferenceProvider, ValueAndUnit from facefusion.system import detect_graphic_devices
from facefusion.types import ExecutionProvider, InferenceOptionSet, InferenceProvider
onnxruntime.set_default_logger_severity(3) onnxruntime.set_default_logger_severity(3)
@@ -39,7 +37,7 @@ def create_inference_providers(execution_device_id : int, execution_providers :
inference_providers.append((facefusion.choices.execution_provider_set.get(execution_provider), inference_providers.append((facefusion.choices.execution_provider_set.get(execution_provider),
{ {
'device_id': execution_device_id, 'device_id': execution_device_id,
'cudnn_conv_algo_search': resolve_cudnn_conv_algo_search() 'cudnn_conv_algo_search': resolve_static_cudnn_conv_algo_search(tuple(execution_providers))
})) }))
if execution_provider == 'tensorrt': if execution_provider == 'tensorrt':
@@ -112,12 +110,18 @@ def resolve_cache_path() -> str:
return os.path.join('.caches', onnxruntime.get_version_string()) return os.path.join('.caches', onnxruntime.get_version_string())
def resolve_cudnn_conv_algo_search() -> str: @lru_cache()
execution_devices = detect_static_execution_devices() def resolve_static_cudnn_conv_algo_search(execution_providers : Tuple[ExecutionProvider, ...]) -> str:
return resolve_cudnn_conv_algo_search(list(execution_providers))
def resolve_cudnn_conv_algo_search(execution_providers : List[ExecutionProvider]) -> str:
if has_execution_provider('cuda') or has_execution_provider('tensorrt'):
graphic_devices = detect_graphic_devices(execution_providers)
product_names = ('GeForce GTX 1630', 'GeForce GTX 1650', 'GeForce GTX 1660') product_names = ('GeForce GTX 1630', 'GeForce GTX 1650', 'GeForce GTX 1660')
for execution_device in execution_devices: for graphic_device in graphic_devices:
if execution_device.get('product').get('name').startswith(product_names): if graphic_device.get('product').get('name').startswith(product_names):
return 'DEFAULT' return 'DEFAULT'
return 'EXHAUSTIVE' return 'EXHAUSTIVE'
@@ -127,68 +131,3 @@ def resolve_openvino_device_type(execution_device_id : int) -> str:
if execution_device_id == 0: if execution_device_id == 0:
return 'GPU' return 'GPU'
return 'GPU.' + str(execution_device_id) return 'GPU.' + str(execution_device_id)
def run_nvidia_smi() -> subprocess.Popen[bytes]:
commands = [ shutil.which('nvidia-smi'), '--query', '--xml-format' ]
return subprocess.Popen(commands, stdout = subprocess.PIPE)
@lru_cache()
def detect_static_execution_devices() -> List[ExecutionDevice]:
return detect_execution_devices()
def detect_execution_devices() -> List[ExecutionDevice]:
execution_devices : List[ExecutionDevice] = []
try:
output, _ = run_nvidia_smi().communicate()
root_element = ElementTree.fromstring(output)
except Exception:
root_element = ElementTree.Element('xml')
for gpu_element in root_element.findall('gpu'):
execution_devices.append(
{
'driver_version': root_element.findtext('driver_version'),
'framework':
{
'name': 'CUDA',
'version': root_element.findtext('cuda_version')
},
'product':
{
'vendor': 'NVIDIA',
'name': gpu_element.findtext('product_name').replace('NVIDIA', '').strip()
},
'video_memory':
{
'total': create_value_and_unit(gpu_element.findtext('fb_memory_usage/total')),
'free': create_value_and_unit(gpu_element.findtext('fb_memory_usage/free'))
},
'temperature':
{
'gpu': create_value_and_unit(gpu_element.findtext('temperature/gpu_temp')),
'memory': create_value_and_unit(gpu_element.findtext('temperature/memory_temp'))
},
'utilization':
{
'gpu': create_value_and_unit(gpu_element.findtext('utilization/gpu_util')),
'memory': create_value_and_unit(gpu_element.findtext('utilization/memory_util'))
}
})
return execution_devices
def create_value_and_unit(text : str) -> Optional[ValueAndUnit]:
if ' ' in text:
value, unit = text.split()
return\
{
'value': int(value),
'unit': str(unit)
}
return None
+2 -2
View File
@@ -28,7 +28,7 @@ def graceful_exit(error_code : ErrorCode) -> None:
while process_manager.is_processing(): while process_manager.is_processing():
sleep(0.5) sleep(0.5)
if state_manager.get_item('target_path'): if state_manager.get_item('output_path'):
clear_temp_directory(state_manager.get_item('target_path')) clear_temp_directory(state_manager.get_temp_path(), state_manager.get_item('output_path'))
hard_exit(error_code) hard_exit(error_code)
@@ -2,14 +2,13 @@ from typing import List, Optional
import numpy import numpy
from facefusion import state_manager from facefusion import face_store, state_manager
from facefusion.common_helper import get_first from facefusion.common_helper import get_first, get_middle
from facefusion.face_classifier import classify_face from facefusion.face_classifier import classify_face
from facefusion.face_detector import detect_faces, detect_faces_by_angle from facefusion.face_detector import detect_faces, detect_faces_by_angle
from facefusion.face_helper import apply_nms, convert_to_face_landmark_5, estimate_face_angle, get_nms_threshold from facefusion.face_helper import apply_nms, average_points, convert_to_face_landmark_5, estimate_face_angle, get_nms_threshold
from facefusion.face_landmarker import detect_face_landmark, estimate_face_landmark_68_5 from facefusion.face_landmarker import detect_face_landmark, estimate_face_landmark_68_5
from facefusion.face_recognizer import calculate_face_embedding from facefusion.face_recognizer import calculate_face_embedding
from facefusion.face_store import get_static_faces, set_static_faces
from facefusion.types import BoundingBox, Face, FaceLandmark5, FaceLandmarkSet, FaceScoreSet, Score, VisionFrame from facefusion.types import BoundingBox, Face, FaceLandmark5, FaceLandmarkSet, FaceScoreSet, Score, VisionFrame
@@ -47,7 +46,9 @@ def create_faces(vision_frame : VisionFrame, bounding_boxes : List[BoundingBox],
} }
face_embedding, face_embedding_norm = calculate_face_embedding(vision_frame, face_landmark_set.get('5/68')) face_embedding, face_embedding_norm = calculate_face_embedding(vision_frame, face_landmark_set.get('5/68'))
gender, age, race = classify_face(vision_frame, face_landmark_set.get('5/68')) gender, age, race = classify_face(vision_frame, face_landmark_set.get('5/68'))
faces.append(Face( faces.append(Face(
origin = 'detect',
bounding_box = bounding_box, bounding_box = bounding_box,
score_set = face_score_set, score_set = face_score_set,
landmark_set = face_landmark_set, landmark_set = face_landmark_set,
@@ -68,40 +69,11 @@ def get_one_face(faces : List[Face], position : int = 0) -> Optional[Face]:
return None return None
def get_average_face(faces : List[Face]) -> Optional[Face]:
face_embeddings = []
face_embeddings_norm = []
if faces:
first_face = get_first(faces)
for face in faces:
face_embeddings.append(face.embedding)
face_embeddings_norm.append(face.embedding_norm)
return Face(
bounding_box = first_face.bounding_box,
score_set = first_face.score_set,
landmark_set = first_face.landmark_set,
angle = first_face.angle,
embedding = numpy.mean(face_embeddings, axis = 0),
embedding_norm = numpy.mean(face_embeddings_norm, axis = 0),
gender = first_face.gender,
age = first_face.age,
race = first_face.race
)
return None
def get_many_faces(vision_frames : List[VisionFrame]) -> List[Face]: def get_many_faces(vision_frames : List[VisionFrame]) -> List[Face]:
many_faces : List[Face] = [] many_faces : List[Face] = []
for vision_frame in vision_frames: for vision_frame in vision_frames:
if numpy.any(vision_frame): if numpy.any(vision_frame):
static_faces = get_static_faces(vision_frame)
if static_faces:
many_faces.extend(static_faces)
else:
all_bounding_boxes = [] all_bounding_boxes = []
all_face_scores = [] all_face_scores = []
all_face_landmarks_5 = [] all_face_landmarks_5 = []
@@ -120,10 +92,104 @@ def get_many_faces(vision_frames : List[VisionFrame]) -> List[Face]:
if faces: if faces:
many_faces.extend(faces) many_faces.extend(faces)
set_static_faces(vision_frame, faces)
return many_faces return many_faces
def get_static_faces(vision_frames : List[VisionFrame]) -> List[Face]:
many_faces : List[Face] = []
for vision_frame in vision_frames:
faces = face_store.get_faces(vision_frame)
if not faces:
with face_store.resolve_lock(vision_frame):
faces = face_store.get_faces(vision_frame)
if not faces:
faces = get_many_faces([ vision_frame ])
if faces:
face_store.set_faces(vision_frame, faces)
many_faces.extend(faces)
return many_faces
def refill_faces(faces : List[Optional[Face]]) -> List[Face]:
fill_faces = []
anchor_index_previous = -1
for index, face in enumerate(faces):
if face:
for gap_index in range(anchor_index_previous + 1, index):
average_factor = (gap_index - anchor_index_previous) / (index - anchor_index_previous)
average_face = average_face_geometry([faces[anchor_index_previous], face], average_factor)
fill_faces.append(average_face)
fill_faces.append(face)
anchor_index_previous = index
return fill_faces
def average_face_geometry(faces : List[Face], average_factor : float) -> Face:
face_first = get_first(faces)
face_middle = get_middle(faces)
face_anchor = face_middle
if average_factor < 0.5:
face_anchor = face_first
landmark_set : FaceLandmarkSet =\
{
'5': average_points(face_first.landmark_set.get('5'), face_middle.landmark_set.get('5'), average_factor),
'5/68': average_points(face_first.landmark_set.get('5/68'), face_middle.landmark_set.get('5/68'), average_factor),
'68': average_points(face_first.landmark_set.get('68'), face_middle.landmark_set.get('68'), average_factor),
'68/5': average_points(face_first.landmark_set.get('68/5'), face_middle.landmark_set.get('68/5'), average_factor)
}
return Face(
origin = 'refill',
bounding_box = average_points(face_first.bounding_box, face_middle.bounding_box, average_factor),
score_set = face_anchor.score_set,
landmark_set = landmark_set,
angle = estimate_face_angle(landmark_set.get('68/5')),
embedding = face_anchor.embedding,
embedding_norm = face_anchor.embedding_norm,
gender = face_anchor.gender,
age = face_anchor.age,
race = face_anchor.race
)
def average_face_identity(faces : List[Face]) -> Optional[Face]:
face_embeddings = []
face_embeddings_norm = []
if faces:
first_face = get_first(faces)
for face in faces:
face_embeddings.append(face.embedding)
face_embeddings_norm.append(face.embedding_norm)
return Face(
origin = first_face.origin,
bounding_box = first_face.bounding_box,
score_set = first_face.score_set,
landmark_set = first_face.landmark_set,
angle = first_face.angle,
embedding = numpy.mean(face_embeddings, axis = 0),
embedding_norm = numpy.mean(face_embeddings_norm, axis = 0),
gender = first_face.gender,
age = first_face.age,
race = first_face.race
)
return None
def scale_face(target_face : Face, target_vision_frame : VisionFrame, temp_vision_frame : VisionFrame) -> Face: def scale_face(target_face : Face, target_vision_frame : VisionFrame, temp_vision_frame : VisionFrame) -> Face:
scale_x = temp_vision_frame.shape[1] / target_vision_frame.shape[1] scale_x = temp_vision_frame.shape[1] / target_vision_frame.shape[1]
scale_y = temp_vision_frame.shape[0] / target_vision_frame.shape[0] scale_y = temp_vision_frame.shape[0] / target_vision_frame.shape[0]
+3 -3
View File
@@ -228,7 +228,7 @@ def detect_with_retinaface(vision_frame : VisionFrame, face_detector_size : str)
if numpy.any(keep_indices): if numpy.any(keep_indices):
stride_height = face_detector_height // feature_stride stride_height = face_detector_height // feature_stride
stride_width = face_detector_width // feature_stride stride_width = face_detector_width // feature_stride
anchors = create_static_anchors(feature_stride, anchor_total, stride_height, stride_width) anchors = create_static_anchors(feature_stride, anchor_total, stride_width, stride_height)
bounding_boxes_raw = detection[index + feature_map_channel] * feature_stride bounding_boxes_raw = detection[index + feature_map_channel] * feature_stride
face_landmarks_5_raw = detection[index + feature_map_channel * 2] * feature_stride face_landmarks_5_raw = detection[index + feature_map_channel * 2] * feature_stride
@@ -273,7 +273,7 @@ def detect_with_scrfd(vision_frame : VisionFrame, face_detector_size : str) -> T
if numpy.any(keep_indices): if numpy.any(keep_indices):
stride_height = face_detector_height // feature_stride stride_height = face_detector_height // feature_stride
stride_width = face_detector_width // feature_stride stride_width = face_detector_width // feature_stride
anchors = create_static_anchors(feature_stride, anchor_total, stride_height, stride_width) anchors = create_static_anchors(feature_stride, anchor_total, stride_width, stride_height)
bounding_boxes_raw = detection[index + feature_map_channel] * feature_stride bounding_boxes_raw = detection[index + feature_map_channel] * feature_stride
face_landmarks_5_raw = detection[index + feature_map_channel * 2] * feature_stride face_landmarks_5_raw = detection[index + feature_map_channel * 2] * feature_stride
@@ -356,7 +356,7 @@ def detect_with_yunet(vision_frame : VisionFrame, face_detector_size : str) -> T
if numpy.any(keep_indices): if numpy.any(keep_indices):
stride_height = face_detector_height // feature_stride stride_height = face_detector_height // feature_stride
stride_width = face_detector_width // feature_stride stride_width = face_detector_width // feature_stride
anchors = create_static_anchors(feature_stride, anchor_total, stride_height, stride_width) anchors = create_static_anchors(feature_stride, anchor_total, stride_width, stride_height)
bounding_boxes_center = detection[index + feature_map_channel * 2].squeeze(0)[:, :2] * feature_stride + anchors bounding_boxes_center = detection[index + feature_map_channel * 2].squeeze(0)[:, :2] * feature_stride + anchors
bounding_boxes_size = numpy.exp(detection[index + feature_map_channel * 2].squeeze(0)[:, 2:4]) * feature_stride bounding_boxes_size = numpy.exp(detection[index + feature_map_channel * 2].squeeze(0)[:, 2:4]) * feature_stride
face_landmarks_5_raw = detection[index + feature_map_channel * 3].squeeze(0) face_landmarks_5_raw = detection[index + feature_map_channel * 3].squeeze(0)
+21 -1
View File
@@ -131,7 +131,7 @@ def calculate_paste_area(temp_vision_frame : VisionFrame, crop_vision_frame : Vi
@lru_cache() @lru_cache()
def create_static_anchors(feature_stride : int, anchor_total : int, stride_height : int, stride_width : int) -> Anchors: def create_static_anchors(feature_stride : int, anchor_total : int, stride_width : int, stride_height : int) -> Anchors:
x, y = numpy.mgrid[:stride_width, :stride_height] x, y = numpy.mgrid[:stride_width, :stride_height]
anchors = numpy.stack((y, x), axis = -1) anchors = numpy.stack((y, x), axis = -1)
anchors = (anchors * feature_stride).reshape((-1, 2)) anchors = (anchors * feature_stride).reshape((-1, 2))
@@ -254,3 +254,23 @@ def merge_matrix(temp_matrices : List[Matrix]) -> Matrix:
matrix = numpy.dot(temp_matrix, matrix) matrix = numpy.dot(temp_matrix, matrix)
return matrix[:2, :] return matrix[:2, :]
def calculate_bounding_box_overlap(bounding_box_a : BoundingBox, bounding_box_b : BoundingBox) -> float:
intersection_x1 = max(bounding_box_a[0], bounding_box_b[0])
intersection_y1 = max(bounding_box_a[1], bounding_box_b[1])
intersection_x2 = min(bounding_box_a[2], bounding_box_b[2])
intersection_y2 = min(bounding_box_a[3], bounding_box_b[3])
intersection = max(0, intersection_x2 - intersection_x1) * max(0, intersection_y2 - intersection_y1)
bounding_box_area = (bounding_box_a[2] - bounding_box_a[0]) * (bounding_box_a[3] - bounding_box_a[1])
reference_bounding_box_area = (bounding_box_b[2] - bounding_box_b[0]) * (bounding_box_b[3] - bounding_box_b[1])
union = bounding_box_area + reference_bounding_box_area - intersection
if union > 0:
return intersection / union
return 0.0
def average_points(points_previous : Points, points_next : Points, average_factor : float) -> Points:
return points_previous * (1 - average_factor) + points_next * average_factor
+41 -16
View File
@@ -2,26 +2,35 @@ from typing import List
import numpy import numpy
import facefusion.choices
from facefusion import state_manager from facefusion import state_manager
from facefusion.face_analyser import get_many_faces, get_one_face from facefusion.common_helper import get_first, get_middle
from facefusion.face_creator import get_one_face, get_static_faces
from facefusion.face_tracker import track_faces
from facefusion.types import Face, FaceSelectorOrder, Gender, Race, Score, VisionFrame from facefusion.types import Face, FaceSelectorOrder, Gender, Race, Score, VisionFrame
def select_faces(reference_vision_frame : VisionFrame, target_vision_frame : VisionFrame) -> List[Face]: def select_faces(reference_vision_frame : VisionFrame, source_vision_frames : List[VisionFrame], target_vision_frames : List[VisionFrame]) -> List[Face]:
target_faces = get_many_faces([ target_vision_frame ]) source_faces = get_static_faces(source_vision_frames)
if state_manager.get_item('face_tracker_score') > 0:
target_faces = track_faces(target_vision_frames, state_manager.get_item('face_tracker_score'))
else:
target_faces = get_static_faces([ get_middle(target_vision_frames) ])
if state_manager.get_item('face_selector_mode') == 'many': if state_manager.get_item('face_selector_mode') == 'many':
return sort_and_filter_faces(target_faces) return sort_and_filter_faces(source_faces, target_faces)
if state_manager.get_item('face_selector_mode') == 'one': if state_manager.get_item('face_selector_mode') == 'one':
target_face = get_one_face(sort_and_filter_faces(target_faces)) target_face = get_one_face(sort_and_filter_faces(source_faces, target_faces))
if target_face: if target_face:
return [ target_face ] return [ target_face ]
if state_manager.get_item('face_selector_mode') == 'reference': if state_manager.get_item('face_selector_mode') == 'reference':
reference_faces = get_many_faces([ reference_vision_frame ]) reference_faces = get_static_faces([ reference_vision_frame ])
reference_faces = sort_and_filter_faces(reference_faces) reference_faces = sort_and_filter_faces(source_faces, reference_faces)
reference_face = get_one_face(reference_faces, state_manager.get_item('reference_face_position')) reference_face = get_one_face(reference_faces, state_manager.get_item('reference_face_position'))
if reference_face: if reference_face:
match_faces = find_match_faces([ reference_face ], target_faces, state_manager.get_item('reference_face_distance')) match_faces = find_match_faces([ reference_face ], target_faces, state_manager.get_item('reference_face_distance'))
return match_faces return match_faces
@@ -53,17 +62,33 @@ def calculate_face_distance(face : Face, reference_face : Face) -> float:
return 0 return 0
def sort_and_filter_faces(faces : List[Face]) -> List[Face]: def sort_and_filter_faces(source_faces : List[Face], target_faces : List[Face]) -> List[Face]:
if faces: if target_faces:
if state_manager.get_item('face_selector_order'): if state_manager.get_item('face_selector_order'):
faces = sort_faces_by_order(faces, state_manager.get_item('face_selector_order')) target_faces = sort_faces_by_order(target_faces, state_manager.get_item('face_selector_order'))
if state_manager.get_item('face_selector_gender'):
faces = filter_faces_by_gender(faces, state_manager.get_item('face_selector_gender')) face_selector_gender = state_manager.get_item('face_selector_gender')
if state_manager.get_item('face_selector_race'): face_selector_race = state_manager.get_item('face_selector_race')
faces = filter_faces_by_race(faces, state_manager.get_item('face_selector_race'))
if source_faces and face_selector_gender == 'auto' or face_selector_race == 'auto':
source_face = get_first(sort_faces_by_order(source_faces, 'large-small'))
if source_face:
if face_selector_gender == 'auto':
face_selector_gender = source_face.gender
if face_selector_race == 'auto':
face_selector_race = source_face.race
if face_selector_gender in facefusion.choices.genders:
target_faces = filter_faces_by_gender(target_faces, face_selector_gender)
if face_selector_race in facefusion.choices.races:
target_faces = filter_faces_by_race(target_faces, face_selector_race)
if state_manager.get_item('face_selector_age_start') or state_manager.get_item('face_selector_age_end'): if state_manager.get_item('face_selector_age_start') or state_manager.get_item('face_selector_age_end'):
faces = filter_faces_by_age(faces, state_manager.get_item('face_selector_age_start'), state_manager.get_item('face_selector_age_end')) target_faces = filter_faces_by_age(target_faces, state_manager.get_item('face_selector_age_start'), state_manager.get_item('face_selector_age_end'))
return faces
return target_faces
def sort_faces_by_order(faces : List[Face], order : FaceSelectorOrder) -> List[Face]: def sort_faces_by_order(faces : List[Face], order : FaceSelectorOrder) -> List[Face]:
+29 -15
View File
@@ -1,28 +1,42 @@
import threading
from typing import List, Optional from typing import List, Optional
import numpy
from facefusion.hash_helper import create_hash from facefusion.hash_helper import create_hash
from facefusion.types import Face, FaceStore, VisionFrame from facefusion.types import Face, FaceStore, VisionFrame
FACE_STORE : FaceStore =\ FACE_STORE : FaceStore = {}
{
'static_faces': {}
}
def get_face_store() -> FaceStore: def get_faces(vision_frame : VisionFrame) -> Optional[List[Face]]:
return FACE_STORE if numpy.any(vision_frame):
def get_static_faces(vision_frame : VisionFrame) -> Optional[List[Face]]:
vision_hash = create_hash(vision_frame.tobytes()) vision_hash = create_hash(vision_frame.tobytes())
return FACE_STORE.get('static_faces').get(vision_hash)
if FACE_STORE.get(vision_hash):
return FACE_STORE.get(vision_hash).get('faces')
return None
def set_static_faces(vision_frame : VisionFrame, faces : List[Face]) -> None: def set_faces(vision_frame : VisionFrame, faces : List[Face]) -> None:
if numpy.any(vision_frame):
vision_hash = create_hash(vision_frame.tobytes()) vision_hash = create_hash(vision_frame.tobytes())
if vision_hash: FACE_STORE.setdefault(vision_hash,
FACE_STORE['static_faces'][vision_hash] = faces {
'lock': threading.Lock()
})['faces'] = faces
def clear_static_faces() -> None: def resolve_lock(vision_frame : VisionFrame) -> threading.Lock:
FACE_STORE['static_faces'].clear() if numpy.any(vision_frame):
vision_hash = create_hash(vision_frame.tobytes())
return FACE_STORE.setdefault(vision_hash,
{
'lock': threading.Lock()
}).get('lock')
return threading.Lock()
def clear_faces() -> None:
FACE_STORE.clear()
+61
View File
@@ -0,0 +1,61 @@
from typing import List
from facefusion.common_helper import get_first, get_last
from facefusion.face_creator import get_static_faces, refill_faces
from facefusion.face_helper import calculate_bounding_box_overlap
from facefusion.types import Face, FaceTrack, Score, VisionFrame
def track_faces(vision_frames : List[VisionFrame], score : Score) -> List[Face]:
target_index = len(vision_frames) // 2
face_tracks = create_face_tracks(vision_frames, score)
temp_faces = []
for face_track in face_tracks:
track_indices = sorted(face_track)
track_index_first = get_first(track_indices)
track_index_last = get_last(track_indices)
track_range = range(track_index_first, track_index_last + 1)
if target_index in track_range:
fill_faces = []
for index in track_range:
fill_faces.append(face_track.get(index))
temp_faces.append(refill_faces(fill_faces)[target_index - track_index_first])
return temp_faces
def create_face_tracks(vision_frames : List[VisionFrame], score : Score) -> List[FaceTrack]:
face_tracks : List[FaceTrack] = []
for frame_index, vision_frame in enumerate(vision_frames):
for face in get_static_faces([ vision_frame ]):
face_track = select_face_track(face_tracks, face, score)
if face_track:
face_track[frame_index] = face
else:
face_tracks.append(
{
frame_index : face
})
return face_tracks
def select_face_track(face_tracks : List[FaceTrack], face : Face, score : Score) -> FaceTrack:
select_track : FaceTrack = {}
select_score = score
for face_track in face_tracks:
track_face = face_track.get(get_last(face_track))
track_score = calculate_bounding_box_overlap(face.bounding_box, track_face.bounding_box)
if track_score > select_score:
select_score = track_score
select_track = face_track
return select_track
+129 -32
View File
@@ -1,7 +1,7 @@
import os import os
import subprocess import subprocess
import tempfile import tempfile
from functools import partial from functools import lru_cache, partial
from typing import List, Optional, cast from typing import List, Optional, cast
from tqdm import tqdm from tqdm import tqdm
@@ -10,11 +10,11 @@ import facefusion.choices
from facefusion import ffmpeg_builder, logger, process_manager, state_manager, translator from facefusion import ffmpeg_builder, logger, process_manager, state_manager, translator
from facefusion.filesystem import get_file_format, remove_file from facefusion.filesystem import get_file_format, remove_file
from facefusion.temp_helper import get_temp_file_path, get_temp_frames_pattern from facefusion.temp_helper import get_temp_file_path, get_temp_frames_pattern
from facefusion.types import AudioBuffer, AudioEncoder, Command, EncoderSet, Fps, Resolution, UpdateProgress, VideoEncoder, VideoFormat from facefusion.types import ApiSecurityStrategy, AudioEncoder, Buffer, Command, EncoderSet, Fps, Resolution, SampleRate, UpdateProgress, VideoEncoder, VideoFormat
from facefusion.vision import detect_video_duration, detect_video_fps, pack_resolution, predict_video_frame_total from facefusion.vision import detect_video_duration, detect_video_fps, pack_resolution, predict_video_frame_total
def run_ffmpeg_with_progress(commands : List[Command], update_progress : UpdateProgress) -> subprocess.Popen[bytes]: def run_ffmpeg_with_progress(commands : List[Command], update_progress : UpdateProgress) -> subprocess.Popen[Buffer]:
log_level = state_manager.get_item('log_level') log_level = state_manager.get_item('log_level')
commands.extend(ffmpeg_builder.set_progress()) commands.extend(ffmpeg_builder.set_progress())
commands.extend(ffmpeg_builder.cast_stream()) commands.extend(ffmpeg_builder.cast_stream())
@@ -45,7 +45,14 @@ def update_progress(progress : tqdm, frame_number : int) -> None:
progress.update(frame_number - progress.n) progress.update(frame_number - progress.n)
def run_ffmpeg(commands : List[Command]) -> subprocess.Popen[bytes]: def run_ffmpeg_with_pipe(commands : List[Command], file_content : Buffer) -> subprocess.Popen[Buffer]:
commands = ffmpeg_builder.run(commands)
process = subprocess.Popen(commands, stdin = subprocess.PIPE, stderr = subprocess.PIPE, stdout = subprocess.PIPE)
process.communicate(input = file_content)
return process
def run_ffmpeg(commands : List[Command]) -> subprocess.Popen[Buffer]:
log_level = state_manager.get_item('log_level') log_level = state_manager.get_item('log_level')
commands = ffmpeg_builder.run(commands) commands = ffmpeg_builder.run(commands)
process = subprocess.Popen(commands, stderr = subprocess.PIPE, stdout = subprocess.PIPE) process = subprocess.Popen(commands, stderr = subprocess.PIPE, stdout = subprocess.PIPE)
@@ -65,14 +72,14 @@ def run_ffmpeg(commands : List[Command]) -> subprocess.Popen[bytes]:
return process return process
def open_ffmpeg(commands : List[Command]) -> subprocess.Popen[bytes]: def open_ffmpeg(commands : List[Command]) -> subprocess.Popen[Buffer]:
commands = ffmpeg_builder.run(commands) commands = ffmpeg_builder.run(commands)
return subprocess.Popen(commands, stdin = subprocess.PIPE, stdout = subprocess.PIPE) return subprocess.Popen(commands, stdin = subprocess.PIPE, stdout = subprocess.PIPE)
def log_debug(process : subprocess.Popen[bytes]) -> None: def log_debug(process : subprocess.Popen[Buffer]) -> None:
_, stderr = process.communicate() _, stderr = process.communicate()
errors = stderr.decode().split(os.linesep) errors = stderr.decode().splitlines()
for error in errors: for error in errors:
if error.strip(): if error.strip():
@@ -83,6 +90,7 @@ def get_available_encoder_set() -> EncoderSet:
available_encoder_set : EncoderSet =\ available_encoder_set : EncoderSet =\
{ {
'audio': [], 'audio': [],
'image': [],
'video': [] 'video': []
} }
commands = ffmpeg_builder.chain( commands = ffmpeg_builder.chain(
@@ -94,22 +102,29 @@ def get_available_encoder_set() -> EncoderSet:
if line.startswith(' a'): if line.startswith(' a'):
audio_encoder = line.split()[1] audio_encoder = line.split()[1]
if audio_encoder in facefusion.choices.output_audio_encoders: if audio_encoder in facefusion.choices.audio_encoders and audio_encoder not in available_encoder_set.get('audio'):
index = facefusion.choices.output_audio_encoders.index(audio_encoder) #type:ignore[arg-type] available_encoder_set['audio'].append(audio_encoder) #type:ignore[arg-type]
available_encoder_set['audio'].insert(index, audio_encoder) #type:ignore[arg-type]
if line.startswith(' v'):
video_encoder = line.split()[1]
if video_encoder in facefusion.choices.output_video_encoders: if line.startswith(' v'):
index = facefusion.choices.output_video_encoders.index(video_encoder) #type:ignore[arg-type] vision_encoder = line.split()[1]
available_encoder_set['video'].insert(index, video_encoder) #type:ignore[arg-type]
if vision_encoder in facefusion.choices.image_encoders and vision_encoder not in available_encoder_set.get('image'):
available_encoder_set['image'].append(vision_encoder) #type:ignore[arg-type]
if vision_encoder in facefusion.choices.video_encoders and vision_encoder not in available_encoder_set.get('video'):
available_encoder_set['video'].append(vision_encoder) #type:ignore[arg-type]
return available_encoder_set return available_encoder_set
def extract_frames(target_path : str, temp_video_resolution : Resolution, temp_video_fps : Fps, trim_frame_start : int, trim_frame_end : int) -> bool: @lru_cache(maxsize = None)
def get_static_available_encoder_set() -> EncoderSet:
return get_available_encoder_set()
def extract_frames(target_path : str, output_path : str, temp_video_resolution : Resolution, temp_video_fps : Fps, trim_frame_start : int, trim_frame_end : int) -> bool:
extract_frame_total = predict_video_frame_total(target_path, temp_video_fps, trim_frame_start, trim_frame_end) extract_frame_total = predict_video_frame_total(target_path, temp_video_fps, trim_frame_start, trim_frame_end)
temp_frames_pattern = get_temp_frames_pattern(target_path, '%08d') temp_frames_pattern = get_temp_frames_pattern(state_manager.get_temp_path(), output_path, state_manager.get_item('temp_frame_format'), '%08d')
commands = ffmpeg_builder.chain( commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input(target_path), ffmpeg_builder.set_input(target_path),
ffmpeg_builder.set_media_resolution(pack_resolution(temp_video_resolution)), ffmpeg_builder.set_media_resolution(pack_resolution(temp_video_resolution)),
@@ -117,6 +132,7 @@ def extract_frames(target_path : str, temp_video_resolution : Resolution, temp_v
ffmpeg_builder.enforce_pixel_format('rgb24'), ffmpeg_builder.enforce_pixel_format('rgb24'),
ffmpeg_builder.select_frame_range(trim_frame_start, trim_frame_end, temp_video_fps), ffmpeg_builder.select_frame_range(trim_frame_start, trim_frame_end, temp_video_fps),
ffmpeg_builder.prevent_frame_drop(), ffmpeg_builder.prevent_frame_drop(),
ffmpeg_builder.set_start_number(trim_frame_start),
ffmpeg_builder.set_output(temp_frames_pattern) ffmpeg_builder.set_output(temp_frames_pattern)
) )
@@ -125,8 +141,26 @@ def extract_frames(target_path : str, temp_video_resolution : Resolution, temp_v
return process.returncode == 0 return process.returncode == 0
def copy_image(target_path : str, temp_image_resolution : Resolution) -> bool: def spawn_frames(target_path : str, output_path : str, temp_video_resolution : Resolution, temp_video_fps : Fps, trim_frame_start : int, trim_frame_end : int) -> bool:
temp_image_path = get_temp_file_path(target_path) spawn_frame_total = trim_frame_end - trim_frame_start
duration = spawn_frame_total / temp_video_fps
temp_frames_pattern = get_temp_frames_pattern(state_manager.get_temp_path(), output_path, state_manager.get_item('temp_frame_format'), '%08d')
commands = ffmpeg_builder.chain(
ffmpeg_builder.set_loop(),
ffmpeg_builder.set_input(target_path),
ffmpeg_builder.set_video_duration(duration),
ffmpeg_builder.set_video_fps(temp_video_fps),
ffmpeg_builder.set_media_resolution(pack_resolution(temp_video_resolution)),
ffmpeg_builder.set_output(temp_frames_pattern)
)
with tqdm(total = spawn_frame_total, desc = translator.get('spawning'), unit = 'frame', ascii = ' =', disable = state_manager.get_item('log_level') in [ 'warn', 'error' ]) as progress:
process = run_ffmpeg_with_progress(commands, partial(update_progress, progress))
return process.returncode == 0
def copy_image(target_path : str, output_path : str, temp_image_resolution : Resolution) -> bool:
temp_image_path = get_temp_file_path(state_manager.get_temp_path(), output_path)
commands = ffmpeg_builder.chain( commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input(target_path), ffmpeg_builder.set_input(target_path),
ffmpeg_builder.set_media_resolution(pack_resolution(temp_image_resolution)), ffmpeg_builder.set_media_resolution(pack_resolution(temp_image_resolution)),
@@ -136,19 +170,19 @@ def copy_image(target_path : str, temp_image_resolution : Resolution) -> bool:
return run_ffmpeg(commands).returncode == 0 return run_ffmpeg(commands).returncode == 0
def finalize_image(target_path : str, output_path : str, output_image_resolution : Resolution) -> bool: def finalize_image(output_path : str, output_image_resolution : Resolution) -> bool:
output_image_quality = state_manager.get_item('output_image_quality') output_image_quality = state_manager.get_item('output_image_quality')
temp_image_path = get_temp_file_path(target_path) temp_image_path = get_temp_file_path(state_manager.get_temp_path(), output_path)
commands = ffmpeg_builder.chain( commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input(temp_image_path), ffmpeg_builder.set_input(temp_image_path),
ffmpeg_builder.set_media_resolution(pack_resolution(output_image_resolution)), ffmpeg_builder.set_media_resolution(pack_resolution(output_image_resolution)),
ffmpeg_builder.set_image_quality(target_path, output_image_quality), ffmpeg_builder.set_image_quality(output_path, output_image_quality),
ffmpeg_builder.force_output(output_path) ffmpeg_builder.force_output(output_path)
) )
return run_ffmpeg(commands).returncode == 0 return run_ffmpeg(commands).returncode == 0
def read_audio_buffer(target_path : str, audio_sample_rate : int, audio_sample_size : int, audio_channel_total : int) -> Optional[AudioBuffer]: def read_audio_buffer(target_path : str, audio_sample_rate : SampleRate, audio_sample_size : int, audio_channel_total : int) -> Optional[Buffer]:
commands = ffmpeg_builder.chain( commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input(target_path), ffmpeg_builder.set_input(target_path),
ffmpeg_builder.ignore_video_stream(), ffmpeg_builder.ignore_video_stream(),
@@ -160,6 +194,7 @@ def read_audio_buffer(target_path : str, audio_sample_rate : int, audio_sample_s
process = open_ffmpeg(commands) process = open_ffmpeg(commands)
audio_buffer, _ = process.communicate() audio_buffer, _ = process.communicate()
if process.returncode == 0: if process.returncode == 0:
return audio_buffer return audio_buffer
return None return None
@@ -170,9 +205,10 @@ def restore_audio(target_path : str, output_path : str, trim_frame_start : int,
output_audio_quality = state_manager.get_item('output_audio_quality') output_audio_quality = state_manager.get_item('output_audio_quality')
output_audio_volume = state_manager.get_item('output_audio_volume') output_audio_volume = state_manager.get_item('output_audio_volume')
target_video_fps = detect_video_fps(target_path) target_video_fps = detect_video_fps(target_path)
temp_video_path = get_temp_file_path(target_path) temp_video_path = get_temp_file_path(state_manager.get_temp_path(), output_path)
temp_video_format = cast(VideoFormat, get_file_format(temp_video_path)) temp_video_format = cast(VideoFormat, get_file_format(output_path))
temp_video_duration = detect_video_duration(temp_video_path) temp_video_duration = detect_video_duration(temp_video_path)
output_video_format = cast(VideoFormat, get_file_format(output_path))
output_audio_encoder = fix_audio_encoder(temp_video_format, output_audio_encoder) output_audio_encoder = fix_audio_encoder(temp_video_format, output_audio_encoder)
commands = ffmpeg_builder.chain( commands = ffmpeg_builder.chain(
@@ -186,18 +222,20 @@ def restore_audio(target_path : str, output_path : str, trim_frame_start : int,
ffmpeg_builder.select_media_stream('0:v:0'), ffmpeg_builder.select_media_stream('0:v:0'),
ffmpeg_builder.select_media_stream('1:a:0'), ffmpeg_builder.select_media_stream('1:a:0'),
ffmpeg_builder.set_video_duration(temp_video_duration), ffmpeg_builder.set_video_duration(temp_video_duration),
ffmpeg_builder.set_faststart(output_video_format),
ffmpeg_builder.force_output(output_path) ffmpeg_builder.force_output(output_path)
) )
return run_ffmpeg(commands).returncode == 0 return run_ffmpeg(commands).returncode == 0
def replace_audio(target_path : str, audio_path : str, output_path : str) -> bool: def replace_audio(audio_path : str, output_path : str) -> bool:
output_audio_encoder = state_manager.get_item('output_audio_encoder') output_audio_encoder = state_manager.get_item('output_audio_encoder')
output_audio_quality = state_manager.get_item('output_audio_quality') output_audio_quality = state_manager.get_item('output_audio_quality')
output_audio_volume = state_manager.get_item('output_audio_volume') output_audio_volume = state_manager.get_item('output_audio_volume')
temp_video_path = get_temp_file_path(target_path) temp_video_path = get_temp_file_path(state_manager.get_temp_path(), output_path)
temp_video_format = cast(VideoFormat, get_file_format(temp_video_path)) temp_video_format = cast(VideoFormat, get_file_format(output_path))
temp_video_duration = detect_video_duration(temp_video_path) temp_video_duration = detect_video_duration(temp_video_path)
output_video_format = cast(VideoFormat, get_file_format(output_path))
output_audio_encoder = fix_audio_encoder(temp_video_format, output_audio_encoder) output_audio_encoder = fix_audio_encoder(temp_video_format, output_audio_encoder)
commands = ffmpeg_builder.chain( commands = ffmpeg_builder.chain(
@@ -208,26 +246,29 @@ def replace_audio(target_path : str, audio_path : str, output_path : str) -> boo
ffmpeg_builder.set_audio_quality(output_audio_encoder, output_audio_quality), ffmpeg_builder.set_audio_quality(output_audio_encoder, output_audio_quality),
ffmpeg_builder.set_audio_volume(output_audio_volume), ffmpeg_builder.set_audio_volume(output_audio_volume),
ffmpeg_builder.set_video_duration(temp_video_duration), ffmpeg_builder.set_video_duration(temp_video_duration),
ffmpeg_builder.set_faststart(output_video_format),
ffmpeg_builder.force_output(output_path) ffmpeg_builder.force_output(output_path)
) )
return run_ffmpeg(commands).returncode == 0 return run_ffmpeg(commands).returncode == 0
def merge_video(target_path : str, temp_video_fps : Fps, output_video_resolution : Resolution, output_video_fps : Fps, trim_frame_start : int, trim_frame_end : int) -> bool: def merge_video(target_path : str, output_path : str, temp_video_fps : Fps, output_video_fps : Fps, output_video_resolution : Resolution, trim_frame_start : int, trim_frame_end : int) -> bool:
output_video_encoder = state_manager.get_item('output_video_encoder') output_video_encoder = state_manager.get_item('output_video_encoder')
output_video_quality = state_manager.get_item('output_video_quality') output_video_quality = state_manager.get_item('output_video_quality')
output_video_preset = state_manager.get_item('output_video_preset') output_video_preset = state_manager.get_item('output_video_preset')
merge_frame_total = predict_video_frame_total(target_path, output_video_fps, trim_frame_start, trim_frame_end) merge_frame_total = predict_video_frame_total(target_path, output_video_fps, trim_frame_start, trim_frame_end)
temp_video_path = get_temp_file_path(target_path) temp_video_path = get_temp_file_path(state_manager.get_temp_path(), output_path)
temp_video_format = cast(VideoFormat, get_file_format(temp_video_path)) temp_video_format = cast(VideoFormat, get_file_format(output_path))
temp_frames_pattern = get_temp_frames_pattern(target_path, '%08d') temp_frames_pattern = get_temp_frames_pattern(state_manager.get_temp_path(), output_path, state_manager.get_item('temp_frame_format'), '%08d')
output_video_encoder = fix_video_encoder(temp_video_format, output_video_encoder) output_video_encoder = fix_video_encoder(temp_video_format, output_video_encoder)
commands = ffmpeg_builder.chain( commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input_fps(temp_video_fps), ffmpeg_builder.set_input_fps(temp_video_fps),
ffmpeg_builder.set_start_number(trim_frame_start),
ffmpeg_builder.set_input(temp_frames_pattern), ffmpeg_builder.set_input(temp_frames_pattern),
ffmpeg_builder.set_media_resolution(pack_resolution(output_video_resolution)), ffmpeg_builder.set_media_resolution(pack_resolution(output_video_resolution)),
ffmpeg_builder.set_video_encoder(output_video_encoder), ffmpeg_builder.set_video_encoder(output_video_encoder),
ffmpeg_builder.set_video_tag(output_video_encoder, temp_video_format),
ffmpeg_builder.set_video_quality(output_video_encoder, output_video_quality), ffmpeg_builder.set_video_quality(output_video_encoder, output_video_quality),
ffmpeg_builder.set_video_preset(output_video_encoder, output_video_preset), ffmpeg_builder.set_video_preset(output_video_encoder, output_video_preset),
ffmpeg_builder.concat( ffmpeg_builder.concat(
@@ -254,11 +295,13 @@ def concat_video(output_path : str, temp_output_paths : List[str]) -> bool:
concat_video_file.close() concat_video_file.close()
output_path = os.path.abspath(output_path) output_path = os.path.abspath(output_path)
output_video_format = cast(VideoFormat, get_file_format(output_path))
commands = ffmpeg_builder.chain( commands = ffmpeg_builder.chain(
ffmpeg_builder.unsafe_concat(), ffmpeg_builder.unsafe_concat(),
ffmpeg_builder.set_input(concat_video_file.name), ffmpeg_builder.set_input(concat_video_file.name),
ffmpeg_builder.copy_video_encoder(), ffmpeg_builder.copy_video_encoder(),
ffmpeg_builder.copy_audio_encoder(), ffmpeg_builder.copy_audio_encoder(),
ffmpeg_builder.set_faststart(output_video_format),
ffmpeg_builder.force_output(output_path) ffmpeg_builder.force_output(output_path)
) )
process = run_ffmpeg(commands) process = run_ffmpeg(commands)
@@ -267,6 +310,60 @@ def concat_video(output_path : str, temp_output_paths : List[str]) -> bool:
return process.returncode == 0 return process.returncode == 0
def sanitize_audio(file_content : Buffer, asset_path : str, security_strategy : ApiSecurityStrategy) -> bool:
if security_strategy == 'strict':
commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input('pipe:0'),
ffmpeg_builder.deep_copy_audio(),
ffmpeg_builder.strip_metadata(),
ffmpeg_builder.force_output(asset_path)
)
return run_ffmpeg_with_pipe(commands, file_content).returncode == 0
commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input('pipe:0'),
ffmpeg_builder.copy_audio_encoder(),
ffmpeg_builder.strip_metadata(),
ffmpeg_builder.force_output(asset_path)
)
return run_ffmpeg_with_pipe(commands, file_content).returncode == 0
def sanitize_image(file_content : Buffer, asset_path : str) -> bool:
commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input('pipe:0'),
ffmpeg_builder.deep_copy_image(),
ffmpeg_builder.strip_metadata(),
ffmpeg_builder.force_output(asset_path)
)
return run_ffmpeg_with_pipe(commands, file_content).returncode == 0
def sanitize_video(file_content : Buffer, asset_path : str, security_strategy : ApiSecurityStrategy) -> bool:
if security_strategy == 'strict':
available_video_encoders = get_static_available_encoder_set().get('video')
commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input('pipe:0'),
ffmpeg_builder.set_video_encoder(available_video_encoders[0]),
ffmpeg_builder.set_video_preset(available_video_encoders[0], 'ultrafast'),
ffmpeg_builder.set_pixel_format(available_video_encoders[0]),
ffmpeg_builder.deep_copy_video(),
ffmpeg_builder.deep_copy_audio(),
ffmpeg_builder.strip_metadata(),
ffmpeg_builder.force_output(asset_path)
)
return run_ffmpeg_with_pipe(commands, file_content).returncode == 0
commands = ffmpeg_builder.chain(
ffmpeg_builder.set_input('pipe:0'),
ffmpeg_builder.copy_video_encoder(),
ffmpeg_builder.copy_audio_encoder(),
ffmpeg_builder.strip_metadata(),
ffmpeg_builder.force_output(asset_path)
)
return run_ffmpeg_with_pipe(commands, file_content).returncode == 0
def fix_audio_encoder(video_format : VideoFormat, audio_encoder : AudioEncoder) -> AudioEncoder: def fix_audio_encoder(video_format : VideoFormat, audio_encoder : AudioEncoder) -> AudioEncoder:
if video_format == 'avi' and audio_encoder == 'libopus': if video_format == 'avi' and audio_encoder == 'libopus':
return 'aac' return 'aac'
+46 -8
View File
@@ -5,7 +5,7 @@ from typing import List, Optional
import numpy import numpy
from facefusion.filesystem import get_file_format from facefusion.filesystem import get_file_format
from facefusion.types import AudioEncoder, Command, CommandSet, Duration, Fps, StreamMode, VideoEncoder, VideoPreset from facefusion.types import AudioEncoder, Command, CommandSet, Duration, Fps, SampleRate, StreamMode, VideoEncoder, VideoFormat, VideoPreset
def run(commands : List[Command]) -> List[Command]: def run(commands : List[Command]) -> List[Command]:
@@ -51,6 +51,10 @@ def set_input_fps(input_fps : Fps) -> List[Command]:
return [ '-r', str(input_fps) ] return [ '-r', str(input_fps) ]
def set_start_number(frame_number : int) -> List[Command]:
return [ '-start_number', str(frame_number) ]
def set_output(output_path : str) -> List[Command]: def set_output(output_path : str) -> List[Command]:
return [ output_path ] return [ output_path ]
@@ -59,6 +63,10 @@ def force_output(output_path : str) -> List[Command]:
return [ '-y', output_path ] return [ '-y', output_path ]
def set_loop() -> List[Command]:
return [ '-loop', '1' ]
def cast_stream() -> List[Command]: def cast_stream() -> List[Command]:
return [ '-' ] return [ '-' ]
@@ -83,6 +91,10 @@ def enforce_pixel_format(pixel_format : str) -> List[Command]:
return [ '-pix_fmt', pixel_format ] return [ '-pix_fmt', pixel_format ]
def strip_metadata() -> List[Command]:
return [ '-map_metadata', '-1' ]
def set_pixel_format(video_encoder : VideoEncoder) -> List[Command]: def set_pixel_format(video_encoder : VideoEncoder) -> List[Command]:
if video_encoder == 'rawvideo': if video_encoder == 'rawvideo':
return [ '-pix_fmt', 'rgb24' ] return [ '-pix_fmt', 'rgb24' ]
@@ -127,12 +139,8 @@ def set_media_resolution(video_resolution : str) -> List[Command]:
return [ '-s', video_resolution ] return [ '-s', video_resolution ]
def set_image_quality(image_path : str, image_quality : int) -> List[Command]: def deep_copy_audio() -> List[Command]:
if get_file_format(image_path) == 'webp': return [ '-q:a', '0' ]
return [ '-q:v', str(image_quality) ]
image_compression = round(31 - (image_quality * 0.31))
return [ '-q:v', str(image_compression) ]
def set_audio_encoder(audio_codec : str) -> List[Command]: def set_audio_encoder(audio_codec : str) -> List[Command]:
@@ -143,7 +151,7 @@ def copy_audio_encoder() -> List[Command]:
return set_audio_encoder('copy') return set_audio_encoder('copy')
def set_audio_sample_rate(audio_sample_rate : int) -> List[Command]: def set_audio_sample_rate(audio_sample_rate : SampleRate) -> List[Command]:
return [ '-ar', str(audio_sample_rate) ] return [ '-ar', str(audio_sample_rate) ]
@@ -179,6 +187,22 @@ def set_audio_volume(audio_volume : int) -> List[Command]:
return [ '-filter:a', 'volume=' + str(audio_volume / 100) ] return [ '-filter:a', 'volume=' + str(audio_volume / 100) ]
def deep_copy_image() -> List[Command]:
return [ '-q:v', '0' ]
def set_image_quality(image_path : str, image_quality : int) -> List[Command]:
if get_file_format(image_path) == 'webp':
return [ '-q:v', str(image_quality) ]
image_compression = round(31 - (image_quality * 0.31))
return [ '-q:v', str(image_compression) ]
def deep_copy_video() -> List[Command]:
return [ '-q:v', '0' ]
def set_video_encoder(video_encoder : str) -> List[Command]: def set_video_encoder(video_encoder : str) -> List[Command]:
return [ '-c:v', video_encoder ] return [ '-c:v', video_encoder ]
@@ -187,6 +211,18 @@ def copy_video_encoder() -> List[Command]:
return set_video_encoder('copy') return set_video_encoder('copy')
def set_faststart(video_format : VideoFormat) -> List[Command]:
if video_format in [ 'm4v', 'mov', 'mp4' ]:
return [ '-movflags', '+faststart' ]
return []
def set_video_tag(video_encoder : VideoEncoder, video_format : VideoFormat) -> List[Command]:
if video_format in [ 'm4v', 'mov', 'mp4' ] and video_encoder in [ 'libx265', 'hevc_nvenc', 'hevc_amf', 'hevc_qsv', 'hevc_videotoolbox' ]:
return [ '-tag:v', 'hvc1' ]
return []
def set_video_quality(video_encoder : VideoEncoder, video_quality : int) -> List[Command]: def set_video_quality(video_encoder : VideoEncoder, video_quality : int) -> List[Command]:
if video_encoder in [ 'libx264', 'libx264rgb', 'libx265' ]: if video_encoder in [ 'libx264', 'libx264rgb', 'libx265' ]:
video_compression = numpy.round(numpy.interp(video_quality, [ 0, 100 ], [ 51, 0 ])).astype(int).item() video_compression = numpy.round(numpy.interp(video_quality, [ 0, 100 ], [ 51, 0 ])).astype(int).item()
@@ -269,3 +305,5 @@ def map_qsv_preset(video_preset : VideoPreset) -> Optional[str]:
if video_preset in [ 'faster', 'fast', 'medium', 'slow', 'slower', 'veryslow' ]: if video_preset in [ 'faster', 'fast', 'medium', 'slow', 'slower', 'veryslow' ]:
return video_preset return video_preset
return None return None
+84
View File
@@ -0,0 +1,84 @@
import subprocess
from typing import Dict, List
from facefusion import ffprobe_builder
from facefusion.types import AudioMetadata, Buffer, Command, Fps, VideoMetadata
def run_ffprobe(commands : List[Command]) -> subprocess.Popen[Buffer]:
commands = ffprobe_builder.run(commands)
return subprocess.Popen(commands, stderr = subprocess.PIPE, stdout = subprocess.PIPE)
def probe_entries(media_path : str, entries : List[str]) -> Dict[str, str]:
media_entries = {}
commands = ffprobe_builder.chain(
ffprobe_builder.show_entries(entries),
ffprobe_builder.format_to_key_value(),
ffprobe_builder.set_input(media_path)
)
output, _ = run_ffprobe(commands).communicate()
if output:
lines = output.decode().strip().splitlines()
for line in lines:
if '=' in line:
key, value = line.split('=', 1)
media_entries[key] = value
return media_entries
def extract_audio_metadata(audio_path : str) -> AudioMetadata:
audio_entries = probe_entries(audio_path, [ 'duration', 'sample_rate', 'channels', 'bit_rate' ])
duration = float(audio_entries.get('duration'))
sample_rate = int(audio_entries.get('sample_rate'))
frame_total = int(duration * sample_rate)
channel_total = int(audio_entries.get('channels'))
bit_rate = int(audio_entries.get('bit_rate'))
audio_metadata : AudioMetadata =\
{
'duration' : duration,
'frame_total' : frame_total,
'channel_total' : channel_total,
'sample_rate' : sample_rate,
'bit_rate' : bit_rate
}
return audio_metadata
def extract_video_metadata(video_path : str) -> VideoMetadata:
video_entries = probe_entries(video_path, [ 'duration', 'width', 'height', 'r_frame_rate', 'bit_rate' ])
duration = float(video_entries.get('duration'))
fps = extract_video_fps(video_entries.get('r_frame_rate'))
frame_total = int(duration * fps)
width = int(video_entries.get('width'))
height = int(video_entries.get('height'))
bit_rate = int(video_entries.get('bit_rate'))
video_metadata : VideoMetadata =\
{
'duration' : duration,
'frame_total' : frame_total,
'fps' : fps,
'resolution' : (width, height),
'bit_rate' : bit_rate
}
return video_metadata
def extract_video_fps(frame_rate : str) -> Fps:
if frame_rate and '/' in frame_rate:
numerator, denominator = frame_rate.split('/')
if int(numerator) and int(denominator):
return int(numerator) / int(denominator)
return 0.0
+29
View File
@@ -0,0 +1,29 @@
import itertools
import shutil
from typing import List
from facefusion.types import Command
def run(commands : List[Command]) -> List[Command]:
return [ shutil.which('ffprobe'), '-loglevel', 'error' ] + commands
def chain(*commands : List[Command]) -> List[Command]:
return list(itertools.chain(*commands))
def show_entries(entries : List[str]) -> List[Command]:
return [ '-show_entries', 'stream=' + ','.join(entries) ]
def format_to_value() -> List[Command]:
return [ '-of', 'default=noprint_wrappers=1:nokey=1' ]
def format_to_key_value() -> List[Command]:
return [ '-of', 'default=noprint_wrappers=1' ]
def set_input(input_path : str) -> List[Command]:
return [ '-i', input_path ]
+7 -9
View File
@@ -42,15 +42,6 @@ def get_file_format(file_path : str) -> Optional[str]:
return None return None
def same_file_extension(first_file_path : str, second_file_path : str) -> bool:
first_file_extension = get_file_extension(first_file_path)
second_file_extension = get_file_extension(second_file_path)
if first_file_extension and second_file_extension:
return get_file_extension(first_file_path) == get_file_extension(second_file_path)
return False
def is_file(file_path : str) -> bool: def is_file(file_path : str) -> bool:
if file_path: if file_path:
return os.path.isfile(file_path) return os.path.isfile(file_path)
@@ -179,6 +170,13 @@ def create_directory(directory_path : str) -> bool:
return False return False
def move_directory(directory_path : str, move_path : str) -> bool:
if is_directory(directory_path):
shutil.move(directory_path, move_path)
return is_directory(move_path)
return False
def remove_directory(directory_path : str) -> bool: def remove_directory(directory_path : str) -> bool:
if is_directory(directory_path): if is_directory(directory_path):
shutil.rmtree(directory_path, ignore_errors = True) shutil.rmtree(directory_path, ignore_errors = True)
+3 -2
View File
@@ -3,10 +3,11 @@ import zlib
from typing import Optional from typing import Optional
from facefusion.filesystem import get_file_name, is_file from facefusion.filesystem import get_file_name, is_file
from facefusion.types import Buffer
def create_hash(content : bytes) -> str: def create_hash(buffer : Buffer) -> str:
return format(zlib.crc32(content), '08x') return format(zlib.crc32(buffer), '08x')
def validate_hash(validate_path : str) -> bool: def validate_hash(validate_path : str) -> bool:
+24 -17
View File
@@ -1,5 +1,6 @@
import importlib import importlib
import random import random
from functools import lru_cache
from time import sleep, time from time import sleep, time
from typing import List from typing import List
@@ -12,12 +13,12 @@ from facefusion.execution import create_inference_providers, has_execution_provi
from facefusion.exit_helper import fatal_exit from facefusion.exit_helper import fatal_exit
from facefusion.filesystem import get_file_name, is_file from facefusion.filesystem import get_file_name, is_file
from facefusion.time_helper import calculate_end_time from facefusion.time_helper import calculate_end_time
from facefusion.types import DownloadSet, ExecutionProvider, InferencePool, InferencePoolSet from facefusion.types import DownloadSet, ExecutionProvider, InferencePool, InferencePoolSet, InferenceProvider
INFERENCE_POOL_SET : InferencePoolSet =\ INFERENCE_POOL_SET : InferencePoolSet =\
{ {
'cli': {}, 'cli': {},
'ui': {} 'api': {}
} }
@@ -25,37 +26,38 @@ def get_inference_pool(module_name : str, model_names : List[str], model_source_
while process_manager.is_checking(): while process_manager.is_checking():
sleep(0.5) sleep(0.5)
execution_device_ids = state_manager.get_item('execution_device_ids') execution_device_ids = state_manager.get_item('execution_device_ids')
execution_providers = resolve_execution_providers(module_name) execution_providers = state_manager.get_item('execution_providers')
app_context = detect_app_context() app_context = detect_app_context()
for execution_device_id in execution_device_ids: for execution_device_id in execution_device_ids:
inference_context = get_inference_context(module_name, model_names, execution_device_id, execution_providers) inference_context = get_inference_context(module_name, model_names, execution_device_id, execution_providers)
if app_context == 'cli' and INFERENCE_POOL_SET.get('ui').get(inference_context): if app_context == 'cli' and INFERENCE_POOL_SET.get('api').get(inference_context):
INFERENCE_POOL_SET['cli'][inference_context] = INFERENCE_POOL_SET.get('ui').get(inference_context) INFERENCE_POOL_SET['cli'][inference_context] = INFERENCE_POOL_SET.get('api').get(inference_context)
if app_context == 'ui' and INFERENCE_POOL_SET.get('cli').get(inference_context): if app_context == 'api' and INFERENCE_POOL_SET.get('cli').get(inference_context):
INFERENCE_POOL_SET['ui'][inference_context] = INFERENCE_POOL_SET.get('cli').get(inference_context) INFERENCE_POOL_SET['api'][inference_context] = INFERENCE_POOL_SET.get('cli').get(inference_context)
if not INFERENCE_POOL_SET.get(app_context).get(inference_context): if not INFERENCE_POOL_SET.get(app_context).get(inference_context):
INFERENCE_POOL_SET[app_context][inference_context] = create_inference_pool(model_source_set, execution_device_id, execution_providers) inference_providers = resolve_static_inference_providers(module_name, execution_device_id)
INFERENCE_POOL_SET[app_context][inference_context] = create_inference_pool(model_source_set, inference_providers)
current_inference_context = get_inference_context(module_name, model_names, random.choice(execution_device_ids), execution_providers) current_inference_context = get_inference_context(module_name, model_names, random.choice(execution_device_ids), execution_providers)
return INFERENCE_POOL_SET.get(app_context).get(current_inference_context) return INFERENCE_POOL_SET.get(app_context).get(current_inference_context)
def create_inference_pool(model_source_set : DownloadSet, execution_device_id : int, execution_providers : List[ExecutionProvider]) -> InferencePool: def create_inference_pool(model_source_set : DownloadSet, inference_providers : List[InferenceProvider]) -> InferencePool:
inference_pool : InferencePool = {} inference_pool : InferencePool = {}
for model_name in model_source_set.keys(): for model_name in model_source_set.keys():
model_path = model_source_set.get(model_name).get('path') model_path = model_source_set.get(model_name).get('path')
if is_file(model_path): if is_file(model_path):
inference_pool[model_name] = create_inference_session(model_path, execution_device_id, execution_providers) inference_pool[model_name] = create_inference_session(model_path, inference_providers)
return inference_pool return inference_pool
def clear_inference_pool(module_name : str, model_names : List[str]) -> None: def clear_inference_pool(module_name : str, model_names : List[str]) -> None:
execution_device_ids = state_manager.get_item('execution_device_ids') execution_device_ids = state_manager.get_item('execution_device_ids')
execution_providers = resolve_execution_providers(module_name) execution_providers = state_manager.get_item('execution_providers')
app_context = detect_app_context() app_context = detect_app_context()
if is_windows() and has_execution_provider('directml'): if is_windows() and has_execution_provider('directml'):
@@ -67,12 +69,11 @@ def clear_inference_pool(module_name : str, model_names : List[str]) -> None:
del INFERENCE_POOL_SET[app_context][inference_context] del INFERENCE_POOL_SET[app_context][inference_context]
def create_inference_session(model_path : str, execution_device_id : int, execution_providers : List[ExecutionProvider]) -> InferenceSession: def create_inference_session(model_path : str, inference_providers : List[InferenceProvider]) -> InferenceSession:
model_file_name = get_file_name(model_path) model_file_name = get_file_name(model_path)
start_time = time() start_time = time()
try: try:
inference_providers = create_inference_providers(execution_device_id, execution_providers)
inference_session = InferenceSession(model_path, providers = inference_providers) inference_session = InferenceSession(model_path, providers = inference_providers)
logger.debug(translator.get('loading_model_succeeded').format(model_name = model_file_name, seconds = calculate_end_time(start_time)), __name__) logger.debug(translator.get('loading_model_succeeded').format(model_name = model_file_name, seconds = calculate_end_time(start_time)), __name__)
return inference_session return inference_session
@@ -87,9 +88,15 @@ def get_inference_context(module_name : str, model_names : List[str], execution_
return inference_context return inference_context
def resolve_execution_providers(module_name : str) -> List[ExecutionProvider]: @lru_cache()
def resolve_static_inference_providers(module_name : str, execution_device_id : int) -> List[InferenceProvider]:
module = importlib.import_module(module_name) module = importlib.import_module(module_name)
execution_providers = state_manager.get_item('execution_providers')
if hasattr(module, 'resolve_execution_providers'): if hasattr(module, 'resolve_inference_providers'):
return getattr(module, 'resolve_execution_providers')() inference_providers = getattr(module, 'resolve_inference_providers')()
return state_manager.get_item('execution_providers')
if inference_providers:
return inference_providers
return create_inference_providers(execution_device_id, execution_providers)
+4 -4
View File
@@ -19,23 +19,23 @@ LOCALES =\
} }
ONNXRUNTIME_SET =\ ONNXRUNTIME_SET =\
{ {
'default': ('onnxruntime', '1.24.4') 'default': ('onnxruntime', '1.26.0')
} }
if is_windows() or is_linux(): if is_windows() or is_linux():
ONNXRUNTIME_SET['cuda'] = ('onnxruntime-gpu', '1.24.4') ONNXRUNTIME_SET['cuda'] = ('onnxruntime-gpu', '1.26.0')
ONNXRUNTIME_SET['openvino'] = ('onnxruntime-openvino', '1.24.1') ONNXRUNTIME_SET['openvino'] = ('onnxruntime-openvino', '1.24.1')
if is_windows(): if is_windows():
ONNXRUNTIME_SET['directml'] = ('onnxruntime-directml', '1.24.4') ONNXRUNTIME_SET['directml'] = ('onnxruntime-directml', '1.24.4')
ONNXRUNTIME_SET['qnn'] = ('onnxruntime-qnn', '1.24.4') ONNXRUNTIME_SET['qnn'] = ('onnxruntime-qnn', '1.24.4')
if is_linux(): if is_linux():
ONNXRUNTIME_SET['migraphx'] = ('onnxruntime-migraphx', '1.24.2') ONNXRUNTIME_SET['migraphx'] = ('onnxruntime-migraphx', '1.25.0')
ONNXRUNTIME_SET['rocm'] = ('onnxruntime-rocm', '1.22.2.post1') ONNXRUNTIME_SET['rocm'] = ('onnxruntime-rocm', '1.22.2.post1')
def cli() -> None: def cli() -> None:
signal.signal(signal.SIGINT, signal_exit) signal.signal(signal.SIGINT, signal_exit)
program = ArgumentParser(formatter_class = partial(HelpFormatter, max_help_position = 50)) program = ArgumentParser(formatter_class = partial(HelpFormatter, max_help_position = 50))
program.add_argument('--onnxruntime', help = LOCALES.get('install_dependency').format(dependency = 'onnxruntime'), choices = ONNXRUNTIME_SET.keys(), required = True) program.add_argument('onnxruntime', help = LOCALES.get('install_dependency').format(dependency = 'onnxruntime'), choices = ONNXRUNTIME_SET.keys())
program.add_argument('--force-reinstall', help = LOCALES.get('force_reinstall'), action = 'store_true') program.add_argument('--force-reinstall', help = LOCALES.get('force_reinstall'), action = 'store_true')
program.add_argument('--skip-conda', help = LOCALES.get('skip_conda'), action = 'store_true') program.add_argument('--skip-conda', help = LOCALES.get('skip_conda'), action = 'store_true')
program.add_argument('-v', '--version', version = metadata.get('name') + ' ' + metadata.get('version'), action = 'version') program.add_argument('-v', '--version', version = metadata.get('name') + ' ' + metadata.get('version'), action = 'version')
+2
View File
@@ -13,6 +13,8 @@ def get_step_output_path(job_id : str, step_index : int, output_path : str) -> O
if output_file_name and output_file_extension: if output_file_name and output_file_extension:
return os.path.join(output_directory_path, output_file_name + '-' + job_id + '-' + str(step_index) + output_file_extension) return os.path.join(output_directory_path, output_file_name + '-' + job_id + '-' + str(step_index) + output_file_extension)
if output_file_path and output_directory_path:
return os.path.join(output_directory_path, output_file_path + '-' + job_id + '-' + str(step_index))
return None return None
+2
View File
@@ -6,6 +6,7 @@ import facefusion.choices
from facefusion.filesystem import create_directory, get_file_name, is_directory, is_file, move_file, remove_directory, remove_file, resolve_file_pattern from facefusion.filesystem import create_directory, get_file_name, is_directory, is_file, move_file, remove_directory, remove_file, resolve_file_pattern
from facefusion.jobs.job_helper import get_step_output_path from facefusion.jobs.job_helper import get_step_output_path
from facefusion.json import read_json, write_json from facefusion.json import read_json, write_json
from facefusion.sanitizer import sanitize_job_id
from facefusion.time_helper import get_current_date_time from facefusion.time_helper import get_current_date_time
from facefusion.types import Args, Job, JobSet, JobStatus, JobStep, JobStepStatus from facefusion.types import Args, Job, JobSet, JobStatus, JobStep, JobStepStatus
@@ -261,5 +262,6 @@ def find_job_path(job_id : str) -> Optional[str]:
def get_job_file_name(job_id : str) -> Optional[str]: def get_job_file_name(job_id : str) -> Optional[str]:
if job_id: if job_id:
job_id = sanitize_job_id(job_id)
return job_id + '.json' return job_id + '.json'
return None return None
+24 -3
View File
@@ -1,5 +1,7 @@
import os
from facefusion.ffmpeg import concat_video from facefusion.ffmpeg import concat_video
from facefusion.filesystem import are_images, are_videos, move_file, remove_file from facefusion.filesystem import are_images, are_videos, copy_file, create_directory, is_directory, is_file, move_directory, move_file, remove_directory, remove_file, resolve_file_paths
from facefusion.jobs import job_helper, job_manager from facefusion.jobs import job_helper, job_manager
from facefusion.types import JobOutputSet, JobStep, ProcessStep from facefusion.types import JobOutputSet, JobStep, ProcessStep
@@ -59,6 +61,8 @@ def run_step(job_id : str, step_index : int, step : JobStep, process_step : Proc
output_path = step_args.get('output_path') output_path = step_args.get('output_path')
step_output_path = job_helper.get_step_output_path(job_id, step_index, output_path) step_output_path = job_helper.get_step_output_path(job_id, step_index, output_path)
if is_directory(output_path):
return move_directory(output_path, step_output_path) and job_manager.set_step_status(job_id, step_index, 'completed')
return move_file(output_path, step_output_path) and job_manager.set_step_status(job_id, step_index, 'completed') return move_file(output_path, step_output_path) and job_manager.set_step_status(job_id, step_index, 'completed')
job_manager.set_step_status(job_id, step_index, 'failed') job_manager.set_step_status(job_id, step_index, 'failed')
return False return False
@@ -79,13 +83,26 @@ def finalize_steps(job_id : str) -> bool:
output_set = collect_output_set(job_id) output_set = collect_output_set(job_id)
for output_path, temp_output_paths in output_set.items(): for output_path, temp_output_paths in output_set.items():
if are_videos(temp_output_paths): has_videos = are_videos(temp_output_paths)
has_images = are_images(temp_output_paths)
if has_videos:
if not concat_video(output_path, temp_output_paths): if not concat_video(output_path, temp_output_paths):
return False return False
if are_images(temp_output_paths): if not has_videos and has_images:
for temp_output_path in temp_output_paths: for temp_output_path in temp_output_paths:
if not move_file(temp_output_path, output_path): if not move_file(temp_output_path, output_path):
return False return False
if not has_videos and not has_images:
if not create_directory(output_path):
return False
for temp_output_path in temp_output_paths:
if is_directory(temp_output_path):
temp_frame_paths = resolve_file_paths(temp_output_path)
for temp_frame_path in temp_frame_paths:
if not copy_file(temp_frame_path, os.path.join(output_path, os.path.basename(temp_frame_path))):
return False
return True return True
@@ -94,8 +111,12 @@ def clean_steps(job_id: str) -> bool:
for temp_output_paths in output_set.values(): for temp_output_paths in output_set.values():
for temp_output_path in temp_output_paths: for temp_output_path in temp_output_paths:
if is_file(temp_output_path):
if not remove_file(temp_output_path): if not remove_file(temp_output_path):
return False return False
if is_directory(temp_output_path):
if not remove_directory(temp_output_path):
return False
return True return True
-27
View File
@@ -1,27 +0,0 @@
from typing import List
from facefusion.types import JobStore
JOB_STORE : JobStore =\
{
'job_keys': [],
'step_keys': []
}
def get_job_keys() -> List[str]:
return JOB_STORE.get('job_keys')
def get_step_keys() -> List[str]:
return JOB_STORE.get('step_keys')
def register_job_keys(job_keys : List[str]) -> None:
for job_key in job_keys:
JOB_STORE['job_keys'].append(job_key)
def register_step_keys(step_keys : List[str]) -> None:
for step_key in step_keys:
JOB_STORE['step_keys'].append(step_key)
+115
View File
@@ -0,0 +1,115 @@
import ctypes
import ctypes.util
from functools import lru_cache
from typing import List, Optional
@lru_cache
def create_static_library() -> Optional[ctypes.CDLL]:
library_path = ctypes.util.find_library('amd_smi')
if library_path:
library = ctypes.CDLL(library_path)
if library:
return init_ctypes(library)
return None
def init_ctypes(library : ctypes.CDLL) -> ctypes.CDLL:
library.amdsmi_init.argtypes = [ ctypes.c_uint64 ]
library.amdsmi_init.restype = ctypes.c_uint32
library.amdsmi_shut_down.argtypes = []
library.amdsmi_shut_down.restype = ctypes.c_uint32
library.amdsmi_get_socket_handles.argtypes = [ ctypes.POINTER(ctypes.c_uint32), ctypes.POINTER(ctypes.c_void_p) ]
library.amdsmi_get_socket_handles.restype = ctypes.c_uint32
library.amdsmi_get_processor_handles.argtypes = [ ctypes.c_void_p, ctypes.POINTER(ctypes.c_uint32), ctypes.POINTER(ctypes.c_void_p) ]
library.amdsmi_get_processor_handles.restype = ctypes.c_uint32
library.amdsmi_get_gpu_vram_usage.argtypes = [ ctypes.c_void_p, ctypes.c_void_p ]
library.amdsmi_get_gpu_vram_usage.restype = ctypes.c_uint32
library.amdsmi_get_gpu_activity.argtypes = [ ctypes.c_void_p, ctypes.c_void_p ]
library.amdsmi_get_gpu_activity.restype = ctypes.c_uint32
library.amdsmi_get_gpu_asic_info.argtypes = [ ctypes.c_void_p, ctypes.c_void_p ]
library.amdsmi_get_gpu_asic_info.restype = ctypes.c_uint32
library.amdsmi_get_temp_metric.argtypes = [ ctypes.c_void_p, ctypes.c_uint32, ctypes.c_uint32, ctypes.POINTER(ctypes.c_int64) ]
library.amdsmi_get_temp_metric.restype = ctypes.c_uint32
return library
def find_device_handles(amd_smi_library : ctypes.CDLL) -> List[ctypes.c_void_p]:
device_handles : List[ctypes.c_void_p] = []
socket_count = ctypes.c_uint32()
amd_smi_library.amdsmi_get_socket_handles(ctypes.byref(socket_count), ctypes.POINTER(ctypes.c_void_p)())
socket_handles = (ctypes.c_void_p * socket_count.value)()
amd_smi_library.amdsmi_get_socket_handles(ctypes.byref(socket_count), socket_handles)
for socket_index in range(socket_count.value):
device_count = ctypes.c_uint32()
amd_smi_library.amdsmi_get_processor_handles(socket_handles[socket_index], ctypes.byref(device_count), ctypes.POINTER(ctypes.c_void_p)())
processor_handles = (ctypes.c_void_p * device_count.value)()
amd_smi_library.amdsmi_get_processor_handles(socket_handles[socket_index], ctypes.byref(device_count), processor_handles)
for device_index in range(device_count.value):
device_handles.append(ctypes.c_void_p(processor_handles[device_index]))
return device_handles
def define_product_info() -> ctypes.Structure:
return type('AMDSMI_ASIC_INFO', (ctypes.Structure,),
{
'_pack_': 1,
'_fields_':
[
('market_name', ctypes.c_char * 256),
('vendor_id', ctypes.c_uint32),
('vendor_name', ctypes.c_char * 256),
('subvendor_id', ctypes.c_uint32),
('device_id', ctypes.c_uint64),
('rev_id', ctypes.c_uint32),
('asic_serial', ctypes.c_char * 256),
('oam_id', ctypes.c_uint32),
('num_of_compute_units', ctypes.c_uint32),
('padding', ctypes.c_ubyte * 4),
('target_graphics_version', ctypes.c_uint64),
('subsystem_id', ctypes.c_uint32),
('reserved', ctypes.c_uint32 * 21)
]
})()
def define_device_memory() -> ctypes.Structure:
return type('AMDSMI_VRAM_USAGE', (ctypes.Structure,),
{
'_pack_': 1,
'_fields_':
[
('vram_total', ctypes.c_uint32),
('vram_used', ctypes.c_uint32),
('reserved', ctypes.c_uint32 * 2)
]
})()
def define_device_utilization() -> ctypes.Structure:
return type('AMDSMI_ENGINE_USAGE', (ctypes.Structure,),
{
'_pack_': 1,
'_fields_':
[
('gfx_activity', ctypes.c_uint32),
('umc_activity', ctypes.c_uint32),
('mm_activity', ctypes.c_uint32),
('reserved', ctypes.c_uint32 * 13)
]
})()
+134
View File
@@ -0,0 +1,134 @@
import ctypes
from functools import lru_cache
from typing import Optional
from facefusion.common_helper import is_linux, is_macos, is_windows
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url_by_provider
from facefusion.filesystem import resolve_relative_path
from facefusion.types import LibrarySet
@lru_cache
def create_static_library_set() -> Optional[LibrarySet]:
if is_linux():
return\
{
'hashes':
{
'aom':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libaom.hash'),
'path': resolve_relative_path('../.libraries/libaom.hash')
}
},
'sources':
{
'aom':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libaom.so'),
'path': resolve_relative_path('../.libraries/libaom.so')
}
}
}
if is_macos():
return\
{
'hashes':
{
'aom':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libaom.hash'),
'path': resolve_relative_path('../.libraries/libaom.hash')
}
},
'sources':
{
'aom':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libaom.dylib'),
'path': resolve_relative_path('../.libraries/libaom.dylib')
}
}
}
if is_windows():
return\
{
'hashes':
{
'aom':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/aom.hash'),
'path': resolve_relative_path('../.libraries/aom.hash')
}
},
'sources':
{
'aom':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/aom.dll'),
'path': resolve_relative_path('../.libraries/aom.dll')
}
}
}
return None
def pre_check() -> bool:
library_hash_set = create_static_library_set().get('hashes')
library_source_set = create_static_library_set().get('sources')
return conditional_download_hashes(library_hash_set) and conditional_download_sources(library_source_set)
@lru_cache
def create_static_library() -> Optional[ctypes.CDLL]:
library_path = create_static_library_set().get('sources').get('aom').get('path')
if library_path:
if is_windows():
library = ctypes.CDLL(library_path, winmode = 0)
else:
library = ctypes.CDLL(library_path)
if library:
return init_ctypes(library)
return None
def init_ctypes(library : ctypes.CDLL) -> ctypes.CDLL:
library.aom_codec_enc_config_default.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_uint ]
library.aom_codec_enc_config_default.restype = ctypes.c_int
library.aom_codec_enc_init_ver.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_long, ctypes.c_int ]
library.aom_codec_enc_init_ver.restype = ctypes.c_int
library.aom_codec_encode.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_int64, ctypes.c_ulong, ctypes.c_long, ctypes.c_ulong ]
library.aom_codec_encode.restype = ctypes.c_int
library.aom_codec_get_cx_data.argtypes = [ ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p) ]
library.aom_codec_get_cx_data.restype = ctypes.c_void_p
library.aom_codec_enc_config_set.argtypes = [ ctypes.c_void_p, ctypes.c_void_p ]
library.aom_codec_enc_config_set.restype = ctypes.c_int
library.aom_codec_destroy.argtypes = [ ctypes.c_void_p ]
library.aom_codec_destroy.restype = ctypes.c_int
library.aom_img_wrap.argtypes = [ ctypes.c_void_p, ctypes.c_int, ctypes.c_uint, ctypes.c_uint, ctypes.c_uint, ctypes.c_void_p ]
library.aom_img_wrap.restype = ctypes.c_void_p
library.aom_codec_control.argtypes = [ ctypes.c_void_p, ctypes.c_int, ctypes.c_int ]
library.aom_codec_control.restype = ctypes.c_int
library.aom_codec_dec_init_ver.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_long, ctypes.c_int ]
library.aom_codec_dec_init_ver.restype = ctypes.c_int
library.aom_codec_decode.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_uint, ctypes.c_void_p ]
library.aom_codec_decode.restype = ctypes.c_int
library.aom_codec_get_frame.argtypes = [ ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p) ]
library.aom_codec_get_frame.restype = ctypes.c_void_p
return library
+294
View File
@@ -0,0 +1,294 @@
import ctypes
from functools import lru_cache
from typing import Optional
from facefusion.common_helper import is_linux, is_macos, is_windows
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url_by_provider
from facefusion.filesystem import resolve_relative_path
from facefusion.types import LibrarySet
@lru_cache
def create_static_library_set() -> Optional[LibrarySet]:
if is_linux():
return\
{
'hashes':
{
'crypto':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libcrypto.hash'),
'path': resolve_relative_path('../.libraries/libcrypto.hash')
},
'datachannel':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libdatachannel_next.hash'),
'path': resolve_relative_path('../.libraries/libdatachannel_next.hash')
},
'ssl':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libssl.hash'),
'path': resolve_relative_path('../.libraries/libssl.hash')
}
},
'sources':
{
'crypto':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libcrypto.so'),
'path': resolve_relative_path('../.libraries/libcrypto.so')
},
'datachannel':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libdatachannel_next.so'),
'path': resolve_relative_path('../.libraries/libdatachannel_next.so')
},
'ssl':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libssl.so'),
'path': resolve_relative_path('../.libraries/libssl.so')
}
}
}
if is_macos():
return\
{
'hashes':
{
'crypto':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libcrypto.hash'),
'path': resolve_relative_path('../.libraries/libcrypto.hash')
},
'datachannel':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libdatachannel_next.hash'),
'path': resolve_relative_path('../.libraries/libdatachannel_next.hash')
},
'ssl':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libssl.hash'),
'path': resolve_relative_path('../.libraries/libssl.hash')
}
},
'sources':
{
'crypto':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libcrypto.dylib'),
'path': resolve_relative_path('../.libraries/libcrypto.dylib')
},
'datachannel':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libdatachannel_next.dylib'),
'path': resolve_relative_path('../.libraries/libdatachannel_next.dylib')
},
'ssl':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libssl.dylib'),
'path': resolve_relative_path('../.libraries/libssl.dylib')
}
}
}
if is_windows():
return\
{
'hashes':
{
'crypto':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/crypto.hash'),
'path': resolve_relative_path('../.libraries/crypto.hash')
},
'datachannel':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/datachannel_next.hash'),
'path': resolve_relative_path('../.libraries/datachannel_next.hash')
},
'ssl':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/ssl.hash'),
'path': resolve_relative_path('../.libraries/ssl.hash')
}
},
'sources':
{
'crypto':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/crypto.dll'),
'path': resolve_relative_path('../.libraries/crypto.dll')
},
'datachannel':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/datachannel_next.dll'),
'path': resolve_relative_path('../.libraries/datachannel_next.dll')
},
'ssl':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/ssl.dll'),
'path': resolve_relative_path('../.libraries/ssl.dll')
}
}
}
return None
def pre_check() -> bool:
library_hash_set = create_static_library_set().get('hashes')
library_source_set = create_static_library_set().get('sources')
return conditional_download_hashes(library_hash_set) and conditional_download_sources(library_source_set)
@lru_cache
def create_static_library() -> Optional[ctypes.CDLL]:
crypto_source_path = create_static_library_set().get('sources').get('crypto').get('path')
datachannel_source_path = create_static_library_set().get('sources').get('datachannel').get('path')
ssl_source_path = create_static_library_set().get('sources').get('ssl').get('path')
if crypto_source_path and datachannel_source_path and ssl_source_path:
if is_windows():
ctypes.CDLL(crypto_source_path, winmode = 0)
ctypes.CDLL(ssl_source_path, winmode = 0)
library = ctypes.CDLL(datachannel_source_path, winmode = 0)
else:
ctypes.CDLL(crypto_source_path)
ctypes.CDLL(ssl_source_path)
library = ctypes.CDLL(datachannel_source_path)
if library:
return init_ctypes(library)
return None
def init_ctypes(library : ctypes.CDLL) -> ctypes.CDLL:
library.rtcInitLogger.argtypes = [ ctypes.c_int, ctypes.CFUNCTYPE(None, ctypes.c_int, ctypes.c_char_p) ]
library.rtcInitLogger.restype = None
library.rtcInitLogger(2, ctypes.CFUNCTYPE(None, ctypes.c_int, ctypes.c_char_p)(0))
library.rtcCreatePeerConnection.restype = ctypes.c_int
library.rtcDeletePeerConnection.argtypes = [ ctypes.c_int ]
library.rtcDeletePeerConnection.restype = ctypes.c_int
library.rtcSetLocalDescription.argtypes = [ ctypes.c_int, ctypes.c_char_p ]
library.rtcSetLocalDescription.restype = ctypes.c_int
library.rtcSetRemoteDescription.argtypes = [ ctypes.c_int, ctypes.c_char_p, ctypes.c_char_p ]
library.rtcSetRemoteDescription.restype = ctypes.c_int
library.rtcAddTrackEx.restype = ctypes.c_int
library.rtcSendMessage.argtypes = [ ctypes.c_int, ctypes.c_void_p, ctypes.c_int ]
library.rtcSendMessage.restype = ctypes.c_int
library.rtcSetAV1Packetizer.restype = ctypes.c_int
library.rtcSetVP8Packetizer.restype = ctypes.c_int
library.rtcSetVP9Packetizer.restype = ctypes.c_int
library.rtcChainRtcpSrReporter.argtypes = [ ctypes.c_int ]
library.rtcChainRtcpSrReporter.restype = ctypes.c_int
library.rtcSetTrackRtpTimestamp.argtypes = [ ctypes.c_int, ctypes.c_uint32 ]
library.rtcSetTrackRtpTimestamp.restype = ctypes.c_int
library.rtcIsOpen.argtypes = [ ctypes.c_int ]
library.rtcIsOpen.restype = ctypes.c_bool
library.rtcChainRtcpNackResponder.argtypes = [ ctypes.c_int, ctypes.c_uint ]
library.rtcChainRtcpNackResponder.restype = ctypes.c_int
library.rtcGetLocalDescription.argtypes = [ ctypes.c_int, ctypes.c_char_p, ctypes.c_int ]
library.rtcGetLocalDescription.restype = ctypes.c_int
library.rtcSetOpusPacketizer.restype = ctypes.c_int
library.rtcGetPayloadTypesForCodec.argtypes = [ ctypes.c_char_p, ctypes.c_char_p, ctypes.POINTER(ctypes.c_int), ctypes.c_int ]
library.rtcGetPayloadTypesForCodec.restype = ctypes.c_int
library.rtcSetAV1Depacketizer.argtypes = [ ctypes.c_int, ctypes.c_int ]
library.rtcSetAV1Depacketizer.restype = ctypes.c_int
library.rtcSetVP8Depacketizer.restype = ctypes.c_int
library.rtcSetVP9Depacketizer.restype = ctypes.c_int
library.rtcSetOpusDepacketizer.restype = ctypes.c_int
library.rtcChainRtcpReceivingSession.argtypes = [ ctypes.c_int ]
library.rtcChainRtcpReceivingSession.restype = ctypes.c_int
library.rtcSetFrameCallback.argtypes = [ ctypes.c_int, ctypes.CFUNCTYPE(None, ctypes.c_int, ctypes.c_void_p, ctypes.c_int, ctypes.c_void_p, ctypes.c_void_p) ]
library.rtcSetFrameCallback.restype = ctypes.c_int
library.rtcSetUserPointer.argtypes = [ ctypes.c_int, ctypes.c_void_p ]
library.rtcSetUserPointer.restype = None
library.rtcChainRembHandler.argtypes = [ ctypes.c_int, ctypes.CFUNCTYPE(None, ctypes.c_int, ctypes.c_uint, ctypes.c_void_p) ]
library.rtcChainRembHandler.restype = ctypes.c_int
library.rtcRequestBitrate.argtypes = [ ctypes.c_int, ctypes.c_uint ]
library.rtcRequestBitrate.restype = ctypes.c_int
library.rtcSetClosedCallback.argtypes = [ ctypes.c_int, ctypes.CFUNCTYPE(None, ctypes.c_int, ctypes.c_void_p) ]
library.rtcSetClosedCallback.restype = ctypes.c_int
return library
def define_rtc_configuration() -> ctypes.Structure:
return type('RTC_CONFIGURATION', (ctypes.Structure,),
{
'_fields_':
[
('iceServers', ctypes.POINTER(ctypes.c_char_p)),
('iceServersCount', ctypes.c_int),
('proxyServer', ctypes.c_char_p),
('bindAddress', ctypes.c_char_p),
('certificateType', ctypes.c_int),
('iceTransportPolicy', ctypes.c_int),
('enableIceTcp', ctypes.c_bool),
('enableIceUdpMux', ctypes.c_bool),
('disableAutoNegotiation', ctypes.c_bool),
('forceMediaTransport', ctypes.c_bool),
('portRangeBegin', ctypes.c_ushort),
('portRangeEnd', ctypes.c_ushort),
('mtu', ctypes.c_int),
('maxMessageSize', ctypes.c_int)
]
})()
def define_rtc_track_init() -> ctypes.Structure:
return type('RTC_TRACK_INIT', (ctypes.Structure,),
{
'_fields_':
[
('direction', ctypes.c_int),
('codec', ctypes.c_int),
('payloadType', ctypes.c_int),
('ssrc', ctypes.c_uint32),
('mid', ctypes.c_char_p),
('name', ctypes.c_char_p),
('msid', ctypes.c_char_p),
('trackId', ctypes.c_char_p),
('profile', ctypes.c_char_p)
]
})()
def define_rtc_packetizer_init() -> ctypes.Structure:
return type('RTC_PACKETIZER_INIT', (ctypes.Structure,),
{
'_fields_':
[
('ssrc', ctypes.c_uint32),
('cname', ctypes.c_char_p),
('payloadType', ctypes.c_uint8),
('clockRate', ctypes.c_uint32),
('sequenceNumber', ctypes.c_uint16),
('timestamp', ctypes.c_uint32),
('maxFragmentSize', ctypes.c_uint16),
('nalSeparator', ctypes.c_int),
('obuPacketization', ctypes.c_int)
]
})()
+88
View File
@@ -0,0 +1,88 @@
import ctypes
import ctypes.util
from functools import lru_cache
from typing import List, Optional
@lru_cache
def create_static_library() -> Optional[ctypes.CDLL]:
library_path = ctypes.util.find_library('nvidia-ml') or ctypes.util.find_library('nvml')
if library_path:
library = ctypes.CDLL(library_path)
if library:
return init_ctypes(library)
return None
def init_ctypes(library : ctypes.CDLL) -> ctypes.CDLL:
library.nvmlInit_v2.argtypes = []
library.nvmlInit_v2.restype = ctypes.c_int
library.nvmlShutdown.argtypes = []
library.nvmlShutdown.restype = ctypes.c_int
library.nvmlDeviceGetCount_v2.argtypes = [ ctypes.POINTER(ctypes.c_uint) ]
library.nvmlDeviceGetCount_v2.restype = ctypes.c_int
library.nvmlSystemGetDriverVersion.argtypes = [ ctypes.c_char_p, ctypes.c_uint ]
library.nvmlSystemGetDriverVersion.restype = ctypes.c_int
library.nvmlSystemGetCudaDriverVersion.argtypes = [ ctypes.POINTER(ctypes.c_int) ]
library.nvmlSystemGetCudaDriverVersion.restype = ctypes.c_int
library.nvmlDeviceGetHandleByIndex_v2.argtypes = [ ctypes.c_uint, ctypes.POINTER(ctypes.c_void_p) ]
library.nvmlDeviceGetHandleByIndex_v2.restype = ctypes.c_int
library.nvmlDeviceGetName.argtypes = [ ctypes.c_void_p, ctypes.c_char_p, ctypes.c_uint ]
library.nvmlDeviceGetName.restype = ctypes.c_int
library.nvmlDeviceGetMemoryInfo.argtypes = [ ctypes.c_void_p, ctypes.c_void_p ]
library.nvmlDeviceGetMemoryInfo.restype = ctypes.c_int
library.nvmlDeviceGetTemperature.argtypes = [ ctypes.c_void_p, ctypes.c_int, ctypes.POINTER(ctypes.c_uint) ]
library.nvmlDeviceGetTemperature.restype = ctypes.c_int
library.nvmlDeviceGetUtilizationRates.argtypes = [ ctypes.c_void_p, ctypes.c_void_p ]
library.nvmlDeviceGetUtilizationRates.restype = ctypes.c_int
return library
def find_device_handles(nvidia_ml_library : ctypes.CDLL) -> List[ctypes.c_void_p]:
device_handles : List[ctypes.c_void_p] = []
device_count = ctypes.c_uint()
nvidia_ml_library.nvmlDeviceGetCount_v2(ctypes.byref(device_count))
for device_id in range(device_count.value):
device_handle = ctypes.c_void_p()
nvidia_ml_library.nvmlDeviceGetHandleByIndex_v2(device_id, ctypes.byref(device_handle))
device_handles.append(device_handle)
return device_handles
def define_device_memory() -> ctypes.Structure:
return type('NVML_MEMORY', (ctypes.Structure,),
{
'_fields_':
[
('total', ctypes.c_ulonglong),
('free', ctypes.c_ulonglong),
('used', ctypes.c_ulonglong)
]
})()
def define_device_utilization() -> ctypes.Structure:
return type('NVML_UTILIZATION', (ctypes.Structure,),
{
'_fields_':
[
('gpu', ctypes.c_uint),
('memory', ctypes.c_uint)
]
})()
+124
View File
@@ -0,0 +1,124 @@
import ctypes
from functools import lru_cache
from typing import Optional
from facefusion.common_helper import is_linux, is_macos, is_windows
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url_by_provider
from facefusion.filesystem import resolve_relative_path
from facefusion.types import LibrarySet
@lru_cache
def create_static_library_set() -> Optional[LibrarySet]:
if is_linux():
return\
{
'hashes':
{
'opus':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libopus.hash'),
'path': resolve_relative_path('../.libraries/libopus.hash')
}
},
'sources':
{
'opus':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libopus.so'),
'path': resolve_relative_path('../.libraries/libopus.so')
}
}
}
if is_macos():
return\
{
'hashes':
{
'opus':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libopus.hash'),
'path': resolve_relative_path('../.libraries/libopus.hash')
}
},
'sources':
{
'opus':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libopus.dylib'),
'path': resolve_relative_path('../.libraries/libopus.dylib')
}
}
}
if is_windows():
return\
{
'hashes':
{
'opus':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/opus.hash'),
'path': resolve_relative_path('../.libraries/opus.hash')
}
},
'sources':
{
'opus':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/opus.dll'),
'path': resolve_relative_path('../.libraries/opus.dll')
}
}
}
return None
def pre_check() -> bool:
library_hash_set = create_static_library_set().get('hashes')
library_source_set = create_static_library_set().get('sources')
return conditional_download_hashes(library_hash_set) and conditional_download_sources(library_source_set)
@lru_cache
def create_static_library() -> Optional[ctypes.CDLL]:
library_path = create_static_library_set().get('sources').get('opus').get('path')
if library_path:
if is_windows():
library = ctypes.CDLL(library_path, winmode = 0)
else:
library = ctypes.CDLL(library_path)
if library:
return init_ctypes(library)
return None
def init_ctypes(library : ctypes.CDLL) -> ctypes.CDLL:
library.opus_encoder_create.argtypes = [ ctypes.c_int, ctypes.c_int, ctypes.c_int, ctypes.POINTER(ctypes.c_int) ]
library.opus_encoder_create.restype = ctypes.c_void_p
library.opus_encode_float.argtypes = [ ctypes.c_void_p, ctypes.POINTER(ctypes.c_float), ctypes.c_int, ctypes.c_char_p, ctypes.c_int ]
library.opus_encode_float.restype = ctypes.c_int
library.opus_encoder_destroy.argtypes = [ ctypes.c_void_p ]
library.opus_encoder_destroy.restype = None
library.opus_decoder_create.argtypes = [ ctypes.c_int, ctypes.c_int, ctypes.POINTER(ctypes.c_int) ]
library.opus_decoder_create.restype = ctypes.c_void_p
library.opus_decode_float.argtypes = [ ctypes.c_void_p, ctypes.c_char_p, ctypes.c_int, ctypes.POINTER(ctypes.c_float), ctypes.c_int, ctypes.c_int ]
library.opus_decode_float.restype = ctypes.c_int
library.opus_decoder_get_nb_samples.argtypes = [ ctypes.c_void_p, ctypes.c_char_p, ctypes.c_int ]
library.opus_decoder_get_nb_samples.restype = ctypes.c_int
library.opus_decoder_destroy.argtypes = [ ctypes.c_void_p ]
library.opus_decoder_destroy.restype = None
return library
+24
View File
@@ -0,0 +1,24 @@
import ctypes
import ctypes.util
from functools import lru_cache
from typing import Optional
@lru_cache
def create_static_library() -> Optional[ctypes.CDLL]:
library_path = ctypes.util.find_library('rocm-core')
if library_path:
library = ctypes.CDLL(library_path)
if library:
return init_ctypes(library)
return None
def init_ctypes(library : ctypes.CDLL) -> ctypes.CDLL:
library.getROCmVersion.argtypes = [ ctypes.POINTER(ctypes.c_uint), ctypes.POINTER(ctypes.c_uint), ctypes.POINTER(ctypes.c_uint) ]
library.getROCmVersion.restype = ctypes.c_int
return library
+134
View File
@@ -0,0 +1,134 @@
import ctypes
from functools import lru_cache
from typing import Optional
from facefusion.common_helper import is_linux, is_macos, is_windows
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url_by_provider
from facefusion.filesystem import resolve_relative_path
from facefusion.types import LibrarySet
@lru_cache
def create_static_library_set() -> Optional[LibrarySet]:
if is_linux():
return\
{
'hashes':
{
'vpx':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libvpx.hash'),
'path': resolve_relative_path('../.libraries/libvpx.hash')
}
},
'sources':
{
'vpx':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'linux/libvpx.so'),
'path': resolve_relative_path('../.libraries/libvpx.so')
}
}
}
if is_macos():
return\
{
'hashes':
{
'vpx':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libvpx.hash'),
'path': resolve_relative_path('../.libraries/libvpx.hash')
}
},
'sources':
{
'vpx':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'macos/libvpx.dylib'),
'path': resolve_relative_path('../.libraries/libvpx.dylib')
}
}
}
if is_windows():
return\
{
'hashes':
{
'vpx':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/vpx.hash'),
'path': resolve_relative_path('../.libraries/vpx.hash')
}
},
'sources':
{
'vpx':
{
'url': resolve_download_url_by_provider('huggingface', 'libraries-4.0.0', 'windows/vpx.dll'),
'path': resolve_relative_path('../.libraries/vpx.dll')
}
}
}
return None
def pre_check() -> bool:
library_hash_set = create_static_library_set().get('hashes')
library_source_set = create_static_library_set().get('sources')
return conditional_download_hashes(library_hash_set) and conditional_download_sources(library_source_set)
@lru_cache
def create_static_library() -> Optional[ctypes.CDLL]:
library_path = create_static_library_set().get('sources').get('vpx').get('path')
if library_path:
if is_windows():
library = ctypes.CDLL(library_path, winmode = 0)
else:
library = ctypes.CDLL(library_path)
if library:
return init_ctypes(library)
return None
def init_ctypes(library : ctypes.CDLL) -> ctypes.CDLL:
library.vpx_codec_enc_config_default.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_uint ]
library.vpx_codec_enc_config_default.restype = ctypes.c_int
library.vpx_codec_enc_init_ver.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_long, ctypes.c_int ]
library.vpx_codec_enc_init_ver.restype = ctypes.c_int
library.vpx_codec_encode.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_int64, ctypes.c_ulong, ctypes.c_long, ctypes.c_ulong ]
library.vpx_codec_encode.restype = ctypes.c_int
library.vpx_codec_get_cx_data.argtypes = [ ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p) ]
library.vpx_codec_get_cx_data.restype = ctypes.c_void_p
library.vpx_codec_enc_config_set.argtypes = [ ctypes.c_void_p, ctypes.c_void_p ]
library.vpx_codec_enc_config_set.restype = ctypes.c_int
library.vpx_codec_destroy.argtypes = [ ctypes.c_void_p ]
library.vpx_codec_destroy.restype = ctypes.c_int
library.vpx_img_wrap.argtypes = [ ctypes.c_void_p, ctypes.c_int, ctypes.c_uint, ctypes.c_uint, ctypes.c_uint, ctypes.c_void_p ]
library.vpx_img_wrap.restype = ctypes.c_void_p
library.vpx_codec_control_.argtypes = [ ctypes.c_void_p, ctypes.c_int, ctypes.c_int ]
library.vpx_codec_control_.restype = ctypes.c_int
library.vpx_codec_dec_init_ver.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_long, ctypes.c_int ]
library.vpx_codec_dec_init_ver.restype = ctypes.c_int
library.vpx_codec_decode.argtypes = [ ctypes.c_void_p, ctypes.c_void_p, ctypes.c_uint, ctypes.c_void_p, ctypes.c_long ]
library.vpx_codec_decode.restype = ctypes.c_int
library.vpx_codec_get_frame.argtypes = [ ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p) ]
library.vpx_codec_get_frame.restype = ctypes.c_void_p
return library
+15 -10
View File
@@ -12,8 +12,11 @@ LOCALES : Locales =\
'extracting_frames': 'extracting frames with a resolution of {resolution} and {fps} frames per second', 'extracting_frames': 'extracting frames with a resolution of {resolution} and {fps} frames per second',
'extracting_frames_succeeded': 'extracting frames succeeded', 'extracting_frames_succeeded': 'extracting frames succeeded',
'extracting_frames_failed': 'extracting frames failed', 'extracting_frames_failed': 'extracting frames failed',
'spawning_frames_succeeded': 'spawning frames succeeded',
'spawning_frames_failed': 'spawning frames failed',
'analysing': 'analysing', 'analysing': 'analysing',
'extracting': 'extracting', 'extracting': 'extracting',
'spawning': 'spawning',
'streaming': 'streaming', 'streaming': 'streaming',
'processing': 'processing', 'processing': 'processing',
'merging': 'merging', 'merging': 'merging',
@@ -37,6 +40,8 @@ LOCALES : Locales =\
'processing_stopped': 'processing stopped', 'processing_stopped': 'processing stopped',
'processing_image_succeeded': 'processing to image succeeded in {seconds} seconds', 'processing_image_succeeded': 'processing to image succeeded in {seconds} seconds',
'processing_image_failed': 'processing to image failed', 'processing_image_failed': 'processing to image failed',
'processing_frames_succeeded': 'processing to frames succeeded in {seconds} seconds',
'processing_frames_failed': 'processing to frames failed',
'processing_video_succeeded': 'processing to video succeeded in {seconds} seconds', 'processing_video_succeeded': 'processing to video succeeded in {seconds} seconds',
'processing_video_failed': 'processing to video failed', 'processing_video_failed': 'processing to video failed',
'choose_image_source': 'choose an image for the source', 'choose_image_source': 'choose an image for the source',
@@ -48,10 +53,9 @@ LOCALES : Locales =\
'no_source_face_detected': 'no source face detected', 'no_source_face_detected': 'no source face detected',
'processor_not_loaded': 'processor {processor} could not be loaded', 'processor_not_loaded': 'processor {processor} could not be loaded',
'processor_not_implemented': 'processor {processor} not implemented correctly', 'processor_not_implemented': 'processor {processor} not implemented correctly',
'ui_layout_not_loaded': 'ui layout {ui_layout} could not be loaded',
'ui_layout_not_implemented': 'ui layout {ui_layout} not implemented correctly',
'stream_not_loaded': 'stream {stream_mode} could not be loaded', 'stream_not_loaded': 'stream {stream_mode} could not be loaded',
'stream_not_supported': 'stream not supported', 'stream_not_supported': 'stream not supported',
'api_started': 'started API on {host}:{port}',
'job_created': 'job {job_id} created', 'job_created': 'job {job_id} created',
'job_not_created': 'job {job_id} not created', 'job_not_created': 'job {job_id} not created',
'job_submitted': 'job {job_id} submitted', 'job_submitted': 'job {job_id} submitted',
@@ -99,6 +103,7 @@ LOCALES : Locales =\
{ {
'install_dependency': 'choose the variant of {dependency} to install', 'install_dependency': 'choose the variant of {dependency} to install',
'skip_conda': 'skip the conda environment check', 'skip_conda': 'skip the conda environment check',
'workflow': 'choose the workflow',
'config_path': 'choose the config file to override defaults', 'config_path': 'choose the config file to override defaults',
'temp_path': 'specify the directory for the temporary resources', 'temp_path': 'specify the directory for the temporary resources',
'jobs_path': 'specify the directory to store jobs', 'jobs_path': 'specify the directory to store jobs',
@@ -124,6 +129,7 @@ LOCALES : Locales =\
'reference_face_position': 'specify the position used to create the reference face', 'reference_face_position': 'specify the position used to create the reference face',
'reference_face_distance': 'specify the similarity between the reference face and target face', 'reference_face_distance': 'specify the similarity between the reference face and target face',
'reference_frame_number': 'specify the frame used to create the reference face', 'reference_frame_number': 'specify the frame used to create the reference face',
'face_tracker_score': 'specify the overlap score used to match the tracked faces',
'face_occluder_model': 'choose the model responsible for the occlusion mask', 'face_occluder_model': 'choose the model responsible for the occlusion mask',
'face_parser_model': 'choose the model responsible for the region mask', 'face_parser_model': 'choose the model responsible for the region mask',
'face_mask_types': 'mix and match different face mask types (choices: {choices})', 'face_mask_types': 'mix and match different face mask types (choices: {choices})',
@@ -135,12 +141,13 @@ LOCALES : Locales =\
'trim_frame_start': 'specify the starting frame of the target video', 'trim_frame_start': 'specify the starting frame of the target video',
'trim_frame_end': 'specify the ending frame of the target video', 'trim_frame_end': 'specify the ending frame of the target video',
'temp_frame_format': 'specify the temporary resources format', 'temp_frame_format': 'specify the temporary resources format',
'keep_temp': 'keep the temporary resources after processing', 'target_frame_amount': 'specify the amount of target frames forwarded to the processor',
'output_image_quality': 'specify the image quality which translates to the image compression', 'output_image_quality': 'specify the image quality which translates to the image compression',
'output_image_scale': 'specify the image scale based on the target image', 'output_image_scale': 'specify the image scale based on the target image',
'output_audio_encoder': 'specify the encoder used for the audio', 'output_audio_encoder': 'specify the encoder used for the audio',
'output_audio_quality': 'specify the audio quality which translates to the audio compression', 'output_audio_quality': 'specify the audio quality which translates to the audio compression',
'output_audio_volume': 'specify the audio volume based on the target video', 'output_audio_volume': 'specify the audio volume based on the target video',
'output_audio_fps': 'specify the fps used when converting audio to video frames',
'output_video_encoder': 'specify the encoder used for the video', 'output_video_encoder': 'specify the encoder used for the video',
'output_video_preset': 'balance fast video processing and video file size', 'output_video_preset': 'balance fast video processing and video file size',
'output_video_quality': 'specify the video quality which translates to the video compression', 'output_video_quality': 'specify the video quality which translates to the video compression',
@@ -149,26 +156,25 @@ LOCALES : Locales =\
'processors': 'load a single or multiple processors (choices: {choices}, ...)', 'processors': 'load a single or multiple processors (choices: {choices}, ...)',
'background-remover-model': 'choose the model responsible for removing the background', 'background-remover-model': 'choose the model responsible for removing the background',
'background-remover-color': 'apply red, green blue and alpha values of the background', 'background-remover-color': 'apply red, green blue and alpha values of the background',
'open_browser': 'open the browser once the program is ready',
'ui_layouts': 'launch a single or multiple UI layouts (choices: {choices}, ...)',
'ui_workflow': 'choose the ui workflow',
'download_providers': 'download using different providers (choices: {choices}, ...)', 'download_providers': 'download using different providers (choices: {choices}, ...)',
'download_scope': 'specify the download scope', 'download_scope': 'specify the download scope',
'benchmark_mode': 'choose the benchmark mode', 'benchmark_mode': 'choose the benchmark mode',
'benchmark_resolutions': 'choose the resolutions for the benchmarks (choices: {choices}, ...)', 'benchmark_resolutions': 'choose the resolutions for the benchmarks (choices: {choices}, ...)',
'benchmark_cycle_count': 'specify the amount of cycles per benchmark', 'benchmark_cycle_count': 'specify the amount of cycles per benchmark',
'api_host': 'specify the API host',
'api_port': 'specify the API port',
'api_security_strategy': 'specify the API security strategy used for sanitizing uploaded assets',
'execution_device_ids': 'specify the devices used for processing', 'execution_device_ids': 'specify the devices used for processing',
'execution_providers': 'inference using different providers (choices: {choices}, ...)', 'execution_providers': 'inference using different providers (choices: {choices}, ...)',
'execution_thread_count': 'specify the amount of parallel threads while processing', 'execution_thread_count': 'specify the amount of parallel threads while processing',
'video_memory_strategy': 'balance fast processing and low VRAM usage', 'video_memory_strategy': 'balance fast processing and low VRAM usage',
'system_memory_limit': 'limit the available RAM that can be used while processing',
'log_level': 'adjust the message severity displayed in the terminal', 'log_level': 'adjust the message severity displayed in the terminal',
'halt_on_error': 'halt the program once an error occurred', 'halt_on_error': 'halt the program once an error occurred',
'run': 'run the program', 'run': 'run the program',
'headless_run': 'run the program in headless mode',
'batch_run': 'run the program in batch mode', 'batch_run': 'run the program in batch mode',
'force_download': 'force automate downloads and exit', 'force_download': 'force automate downloads and exit',
'benchmark': 'benchmark the program', 'benchmark': 'benchmark the program',
'api': 'start the API server',
'job_id': 'specify the job id', 'job_id': 'specify the job id',
'job_status': 'specify the job status', 'job_status': 'specify the job status',
'step_index': 'specify the step index', 'step_index': 'specify the step index',
@@ -224,6 +230,7 @@ LOCALES : Locales =\
'face_selector_mode_dropdown': 'FACE SELECTOR MODE', 'face_selector_mode_dropdown': 'FACE SELECTOR MODE',
'face_selector_order_dropdown': 'FACE SELECTOR ORDER', 'face_selector_order_dropdown': 'FACE SELECTOR ORDER',
'face_selector_race_dropdown': 'FACE SELECTOR RACE', 'face_selector_race_dropdown': 'FACE SELECTOR RACE',
'face_tracker_score_slider': 'FACE TRACKER SCORE',
'face_occluder_model_dropdown': 'FACE OCCLUDER MODEL', 'face_occluder_model_dropdown': 'FACE OCCLUDER MODEL',
'face_parser_model_dropdown': 'FACE PARSER MODEL', 'face_parser_model_dropdown': 'FACE PARSER MODEL',
'voice_extractor_model_dropdown': 'VOICE EXTRACTOR MODEL', 'voice_extractor_model_dropdown': 'VOICE EXTRACTOR MODEL',
@@ -257,12 +264,10 @@ LOCALES : Locales =\
'source_file': 'SOURCE', 'source_file': 'SOURCE',
'start_button': 'START', 'start_button': 'START',
'stop_button': 'STOP', 'stop_button': 'STOP',
'system_memory_limit_slider': 'SYSTEM MEMORY LIMIT',
'target_file': 'TARGET', 'target_file': 'TARGET',
'temp_frame_format_dropdown': 'TEMP FRAME FORMAT', 'temp_frame_format_dropdown': 'TEMP FRAME FORMAT',
'terminal_textbox': 'TERMINAL', 'terminal_textbox': 'TERMINAL',
'trim_frame_slider': 'TRIM FRAME', 'trim_frame_slider': 'TRIM FRAME',
'ui_workflow': 'UI WORKFLOW',
'video_memory_strategy_dropdown': 'VIDEO MEMORY STRATEGY', 'video_memory_strategy_dropdown': 'VIDEO MEMORY STRATEGY',
'webcam_fps_slider': 'WEBCAM FPS', 'webcam_fps_slider': 'WEBCAM FPS',
'webcam_image': 'WEBCAM', 'webcam_image': 'WEBCAM',
+17
View File
@@ -0,0 +1,17 @@
from typing import Optional, Tuple
def restrict_trim_frame(frame_total : int, trim_frame_start : Optional[int], trim_frame_end : Optional[int]) -> Tuple[int, int]:
if isinstance(trim_frame_start, int):
trim_frame_start = max(0, min(trim_frame_start, frame_total))
if isinstance(trim_frame_end, int):
trim_frame_end = max(0, min(trim_frame_end, frame_total))
if isinstance(trim_frame_start, int) and isinstance(trim_frame_end, int):
return trim_frame_start, trim_frame_end
if isinstance(trim_frame_start, int):
return trim_frame_start, frame_total
if isinstance(trim_frame_end, int):
return 0, trim_frame_end
return 0, frame_total
-21
View File
@@ -1,21 +0,0 @@
from facefusion.common_helper import is_macos, is_windows
if is_windows():
import ctypes
else:
import resource
def limit_system_memory(system_memory_limit : int = 1) -> bool:
if is_macos():
system_memory_limit = system_memory_limit * (1024 ** 6)
else:
system_memory_limit = system_memory_limit * (1024 ** 3)
try:
if is_windows():
ctypes.windll.kernel32.SetProcessWorkingSetSize(-1, ctypes.c_size_t(system_memory_limit), ctypes.c_size_t(system_memory_limit)) #type:ignore[attr-defined]
else:
resource.setrlimit(resource.RLIMIT_DATA, (system_memory_limit, system_memory_limit))
return True
except Exception:
return False
+1 -1
View File
@@ -4,7 +4,7 @@ METADATA =\
{ {
'name': 'FaceFusion', 'name': 'FaceFusion',
'description': 'Industry leading face manipulation platform', 'description': 'Industry leading face manipulation platform',
'version': '3.6.1', 'version': 'v4',
'license': 'OpenRAIL-AS', 'license': 'OpenRAIL-AS',
'author': 'Henry Ruhs', 'author': 'Henry Ruhs',
'url': 'https://facefusion.io' 'url': 'https://facefusion.io'
+1
View File
@@ -12,6 +12,7 @@ PROCESSORS_METHODS =\
'clear_inference_pool', 'clear_inference_pool',
'register_args', 'register_args',
'apply_args', 'apply_args',
'get_common_modules',
'pre_check', 'pre_check',
'pre_process', 'pre_process',
'post_process', 'post_process',
@@ -1,28 +1,29 @@
from argparse import ArgumentParser from argparse import ArgumentParser
from functools import lru_cache from functools import lru_cache
from types import ModuleType
from typing import List
import cv2 import cv2
import numpy import numpy
import facefusion.capability_store
import facefusion.choices import facefusion.choices
import facefusion.jobs.job_manager import facefusion.jobs.job_manager
import facefusion.jobs.job_store
from facefusion import config, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, inference_manager, logger, state_manager, translator, video_manager from facefusion import config, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, inference_manager, logger, state_manager, translator, video_manager
from facefusion.common_helper import create_int_metavar, is_macos from facefusion.common_helper import create_int_metavar, get_middle
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url
from facefusion.execution import has_execution_provider from facefusion.face_creator import scale_face
from facefusion.face_analyser import scale_face
from facefusion.face_helper import merge_matrix, paste_back, scale_face_landmark_5, warp_face_by_face_landmark_5 from facefusion.face_helper import merge_matrix, paste_back, scale_face_landmark_5, warp_face_by_face_landmark_5
from facefusion.face_masker import create_box_mask, create_occlusion_mask from facefusion.face_masker import create_box_mask, create_occlusion_mask
from facefusion.face_selector import select_faces from facefusion.face_selector import select_faces
from facefusion.filesystem import in_directory, is_image, is_video, resolve_relative_path, same_file_extension from facefusion.filesystem import in_directory, is_image, is_video, resolve_relative_path
from facefusion.processors.modules.age_modifier import choices as age_modifier_choices from facefusion.processors.modules.age_modifier import choices as age_modifier_choices
from facefusion.processors.modules.age_modifier.types import AgeModifierDirection, AgeModifierInputs from facefusion.processors.modules.age_modifier.types import AgeModifierDirection, AgeModifierInputs
from facefusion.processors.types import ProcessorOutputs from facefusion.processors.types import ApplyStateItem, ProcessorOutputs
from facefusion.program_helper import find_argument_group from facefusion.program_helper import find_argument_group
from facefusion.thread_helper import thread_semaphore from facefusion.thread_helper import thread_semaphore
from facefusion.types import ApplyStateItem, Args, DownloadScope, Face, InferencePool, ModelOptions, ModelSet, ProcessMode, VisionFrame from facefusion.types import Args, DownloadScope, Face, InferencePool, ModelOptions, ModelSet, ProcessMode, VisionFrame
from facefusion.vision import match_frame_color, read_static_image, read_static_video_frame from facefusion.vision import match_frame_color, read_static_image, read_static_video_chunk, read_static_video_frame
@lru_cache() @lru_cache()
@@ -124,9 +125,26 @@ def get_model_options() -> ModelOptions:
def register_args(program : ArgumentParser) -> None: def register_args(program : ArgumentParser) -> None:
group_processors = find_argument_group(program, 'processors') group_processors = find_argument_group(program, 'processors')
if group_processors: if group_processors:
group_processors.add_argument('--age-modifier-model', help = translator.get('help.model', __package__), default = config.get_str_value('processors', 'age_modifier_model', 'fran'), choices = age_modifier_choices.age_modifier_models) facefusion.capability_store.register_capability_set(
group_processors.add_argument('--age-modifier-direction', help = translator.get('help.direction', __package__), type = int, default = config.get_int_value('processors', 'age_modifier_direction', '0'), choices = age_modifier_choices.age_modifier_direction_range, metavar = create_int_metavar(age_modifier_choices.age_modifier_direction_range)) [
facefusion.jobs.job_store.register_step_keys([ 'age_modifier_model', 'age_modifier_direction' ]) group_processors.add_argument(
'--age-modifier-model',
help = translator.get('help.model', __package__),
default = config.get_str_value('processors', 'age_modifier_model', 'fran'),
choices = age_modifier_choices.age_modifier_models
),
group_processors.add_argument(
'--age-modifier-direction',
help = translator.get('help.direction', __package__),
type = int,
default = config.get_int_value('processors', 'age_modifier_direction', '0'),
choices = age_modifier_choices.age_modifier_direction_range,
metavar = create_int_metavar(age_modifier_choices.age_modifier_direction_range)
)
],
scopes = [ 'api', 'cli' ],
groups = [ 'age_modifier' ]
)
def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None: def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
@@ -134,10 +152,18 @@ def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('age_modifier_direction', args.get('age_modifier_direction')) apply_state_item('age_modifier_direction', args.get('age_modifier_direction'))
def get_common_modules() -> List[ModuleType]:
return [ content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer ]
def pre_check() -> bool: def pre_check() -> bool:
model_hash_set = get_model_options().get('hashes') model_hash_set = get_model_options().get('hashes')
model_source_set = get_model_options().get('sources') model_source_set = get_model_options().get('sources')
for common_module in get_common_modules():
if not common_module.pre_check():
return False
return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set) return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set)
@@ -145,28 +171,25 @@ def pre_process(mode : ProcessMode) -> bool:
if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')): if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')):
logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__)
return False return False
if state_manager.get_item('workflow') in [ 'audio-to-image:video', 'image-to-image', 'image-to-video' ]:
if mode == 'output' and not in_directory(state_manager.get_item('output_path')): if mode == 'output' and not in_directory(state_manager.get_item('output_path')):
logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__)
return False return False
if mode == 'output' and not same_file_extension(state_manager.get_item('target_path'), state_manager.get_item('output_path')):
logger.error(translator.get('match_target_and_output_extension') + translator.get('exclamation_mark'), __name__)
return False
return True return True
def post_process() -> None: def post_process() -> None:
read_static_image.cache_clear() read_static_image.cache_clear()
read_static_video_frame.cache_clear() read_static_video_frame.cache_clear()
read_static_video_chunk.cache_clear()
video_manager.clear_video_pool() video_manager.clear_video_pool()
if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]: if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]:
clear_inference_pool() clear_inference_pool()
if state_manager.get_item('video_memory_strategy') == 'strict': if state_manager.get_item('video_memory_strategy') == 'strict':
content_analyser.clear_inference_pool() for common_module in get_common_modules():
face_classifier.clear_inference_pool() common_module.clear_inference_pool()
face_detector.clear_inference_pool()
face_landmarker.clear_inference_pool()
face_masker.clear_inference_pool()
face_recognizer.clear_inference_pool()
def modify_age(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame: def modify_age(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame:
@@ -231,9 +254,6 @@ def forward(crop_vision_frame : VisionFrame, extend_vision_frame : VisionFrame,
age_modifier = get_inference_pool().get('age_modifier') age_modifier = get_inference_pool().get('age_modifier')
age_modifier_inputs = {} age_modifier_inputs = {}
if is_macos() and has_execution_provider('coreml'):
age_modifier.set_providers([ facefusion.choices.execution_provider_set.get('cpu') ])
for age_modifier_input in age_modifier.get_inputs(): for age_modifier_input in age_modifier.get_inputs():
if age_modifier_input.name == 'target': if age_modifier_input.name == 'target':
age_modifier_inputs[age_modifier_input.name] = crop_vision_frame age_modifier_inputs[age_modifier_input.name] = crop_vision_frame
@@ -280,10 +300,13 @@ def normalize_extend_frame(extend_vision_frame : VisionFrame) -> VisionFrame:
def process_frame(inputs : AgeModifierInputs) -> ProcessorOutputs: def process_frame(inputs : AgeModifierInputs) -> ProcessorOutputs:
reference_vision_frame = inputs.get('reference_vision_frame') reference_vision_frame = inputs.get('reference_vision_frame')
target_vision_frame = inputs.get('target_vision_frame') source_vision_frames = inputs.get('source_vision_frames')
target_vision_frames = inputs.get('target_vision_frames')
temp_vision_frame = inputs.get('temp_vision_frame') temp_vision_frame = inputs.get('temp_vision_frame')
temp_vision_mask = inputs.get('temp_vision_mask') temp_vision_mask = inputs.get('temp_vision_mask')
target_faces = select_faces(reference_vision_frame, target_vision_frame)
target_vision_frame = get_middle(target_vision_frames)
target_faces = select_faces(reference_vision_frame, source_vision_frames, target_vision_frames)
if target_faces: if target_faces:
for target_face in target_faces: for target_face in target_faces:
@@ -1,4 +1,4 @@
from typing import Any, Literal, TypeAlias, TypedDict from typing import Any, List, Literal, TypeAlias, TypedDict
from numpy.typing import NDArray from numpy.typing import NDArray
@@ -7,7 +7,8 @@ from facefusion.types import Mask, VisionFrame
AgeModifierInputs = TypedDict('AgeModifierInputs', AgeModifierInputs = TypedDict('AgeModifierInputs',
{ {
'reference_vision_frame' : VisionFrame, 'reference_vision_frame' : VisionFrame,
'target_vision_frame' : VisionFrame, 'source_vision_frames' : List[VisionFrame],
'target_vision_frames' : List[VisionFrame],
'temp_vision_frame' : VisionFrame, 'temp_vision_frame' : VisionFrame,
'temp_vision_mask' : Mask 'temp_vision_mask' : Mask
}) })
@@ -1,26 +1,28 @@
from argparse import ArgumentParser from argparse import ArgumentParser
from functools import lru_cache, partial from functools import lru_cache, partial
from types import ModuleType
from typing import List, Tuple from typing import List, Tuple
import cv2 import cv2
import numpy import numpy
import facefusion.capability_store
import facefusion.choices
import facefusion.jobs.job_manager import facefusion.jobs.job_manager
import facefusion.jobs.job_store
from facefusion import config, content_analyser, inference_manager, logger, state_manager, translator, video_manager from facefusion import config, content_analyser, inference_manager, logger, state_manager, translator, video_manager
from facefusion.common_helper import is_macos, is_windows from facefusion.common_helper import is_macos, is_windows
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url
from facefusion.execution import has_execution_provider from facefusion.execution import has_execution_provider
from facefusion.filesystem import in_directory, is_image, is_video, resolve_relative_path, same_file_extension from facefusion.filesystem import in_directory, is_image, is_video, resolve_relative_path
from facefusion.normalizer import normalize_color from facefusion.normalizer import normalize_color
from facefusion.processors.modules.background_remover import choices as background_remover_choices from facefusion.processors.modules.background_remover import choices as background_remover_choices
from facefusion.processors.modules.background_remover.types import BackgroundRemoverInputs from facefusion.processors.modules.background_remover.types import BackgroundRemoverInputs
from facefusion.processors.types import ProcessorOutputs from facefusion.processors.types import ApplyStateItem, ProcessorOutputs
from facefusion.program_helper import find_argument_group from facefusion.program_helper import find_argument_group
from facefusion.sanitizer import sanitize_int_range from facefusion.sanitizer import sanitize_int_range
from facefusion.thread_helper import thread_semaphore from facefusion.thread_helper import thread_semaphore
from facefusion.types import ApplyStateItem, Args, DownloadScope, ExecutionProvider, InferencePool, Mask, ModelOptions, ModelSet, ProcessMode, VisionFrame from facefusion.types import Args, DownloadScope, InferencePool, InferenceProvider, Mask, ModelOptions, ModelSet, ProcessMode, VisionFrame
from facefusion.vision import read_static_image, read_static_video_frame from facefusion.vision import read_static_image, read_static_video_chunk, read_static_video_frame
@lru_cache() @lru_cache()
@@ -477,12 +479,13 @@ def clear_inference_pool() -> None:
inference_manager.clear_inference_pool(__name__, model_names) inference_manager.clear_inference_pool(__name__, model_names)
def resolve_execution_providers() -> List[ExecutionProvider]: def resolve_inference_providers() -> List[InferenceProvider]:
model_type = get_model_options().get('type') model_type = get_model_options().get('type')
if is_macos() and has_execution_provider('coreml') or is_windows() and has_execution_provider('directml') and model_type == 'corridor_key': if is_macos() and has_execution_provider('coreml') or is_windows() and has_execution_provider('directml') and model_type == 'corridor_key':
return [ 'cpu' ] return [ facefusion.choices.execution_provider_set.get('cpu') ]
return state_manager.get_item('execution_providers')
return []
def get_model_options() -> ModelOptions: def get_model_options() -> ModelOptions:
@@ -493,10 +496,32 @@ def get_model_options() -> ModelOptions:
def register_args(program : ArgumentParser) -> None: def register_args(program : ArgumentParser) -> None:
group_processors = find_argument_group(program, 'processors') group_processors = find_argument_group(program, 'processors')
if group_processors: if group_processors:
group_processors.add_argument('--background-remover-model', help = translator.get('help.model', __package__), default = config.get_str_value('processors', 'background_remover_model', 'modnet'), choices = background_remover_choices.background_remover_models) facefusion.capability_store.register_capability_set(
group_processors.add_argument('--background-remover-fill-color', help = translator.get('help.fill_color', __package__), type = partial(sanitize_int_range, int_range = background_remover_choices.background_remover_color_range), default = config.get_int_list('processors', 'background_remover_fill_color', '0 0 0 0'), nargs = '+') [
group_processors.add_argument('--background-remover-despill-color', help = translator.get('help.despill_color', __package__), type = partial(sanitize_int_range, int_range = background_remover_choices.background_remover_color_range), default = config.get_int_list('processors', 'background_remover_despill_color', '0 0 0 0'), nargs = '+') group_processors.add_argument(
facefusion.jobs.job_store.register_step_keys([ 'background_remover_model', 'background_remover_fill_color', 'background_remover_despill_color' ]) '--background-remover-model',
help = translator.get('help.model', __package__),
default = config.get_str_value('processors', 'background_remover_model', 'modnet'),
choices = background_remover_choices.background_remover_models
),
group_processors.add_argument(
'--background-remover-fill-color',
help = translator.get('help.fill_color', __package__),
type = partial(sanitize_int_range, int_range = background_remover_choices.background_remover_color_range),
default = config.get_int_list('processors', 'background_remover_fill_color', '0 0 0 0'),
nargs = '+'
),
group_processors.add_argument(
'--background-remover-despill-color',
help = translator.get('help.despill_color', __package__),
type = partial(sanitize_int_range, int_range = background_remover_choices.background_remover_color_range),
default = config.get_int_list('processors', 'background_remover_despill_color', '0 0 0 0'),
nargs = '+'
)
],
scopes = [ 'api', 'cli' ],
groups = [ 'background_remover' ]
)
def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None: def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
@@ -505,10 +530,18 @@ def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('background_remover_despill_color', normalize_color(args.get('background_remover_despill_color'))) apply_state_item('background_remover_despill_color', normalize_color(args.get('background_remover_despill_color')))
def get_common_modules() -> List[ModuleType]:
return [ content_analyser ]
def pre_check() -> bool: def pre_check() -> bool:
model_hash_set = get_model_options().get('hashes') model_hash_set = get_model_options().get('hashes')
model_source_set = get_model_options().get('sources') model_source_set = get_model_options().get('sources')
for common_module in get_common_modules():
if not common_module.pre_check():
return False
return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set) return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set)
@@ -516,23 +549,25 @@ def pre_process(mode : ProcessMode) -> bool:
if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')): if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')):
logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__)
return False return False
if state_manager.get_item('workflow') in [ 'audio-to-image:video', 'image-to-image', 'image-to-video' ]:
if mode == 'output' and not in_directory(state_manager.get_item('output_path')): if mode == 'output' and not in_directory(state_manager.get_item('output_path')):
logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__)
return False return False
if mode == 'output' and not same_file_extension(state_manager.get_item('target_path'), state_manager.get_item('output_path')):
logger.error(translator.get('match_target_and_output_extension') + translator.get('exclamation_mark'), __name__)
return False
return True return True
def post_process() -> None: def post_process() -> None:
read_static_image.cache_clear() read_static_image.cache_clear()
read_static_video_frame.cache_clear() read_static_video_frame.cache_clear()
read_static_video_chunk.cache_clear()
video_manager.clear_video_pool() video_manager.clear_video_pool()
if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]: if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]:
clear_inference_pool() clear_inference_pool()
if state_manager.get_item('video_memory_strategy') == 'strict': if state_manager.get_item('video_memory_strategy') == 'strict':
content_analyser.clear_inference_pool() for common_module in get_common_modules():
common_module.clear_inference_pool()
def remove_background(temp_vision_frame : VisionFrame) -> Tuple[VisionFrame, Mask]: def remove_background(temp_vision_frame : VisionFrame) -> Tuple[VisionFrame, Mask]:
@@ -1,10 +1,10 @@
from typing import Literal, TypedDict from typing import List, Literal, TypedDict
from facefusion.types import Mask, VisionFrame from facefusion.types import Mask, VisionFrame
BackgroundRemoverInputs = TypedDict('BackgroundRemoverInputs', BackgroundRemoverInputs = TypedDict('BackgroundRemoverInputs',
{ {
'target_vision_frame' : VisionFrame, 'target_vision_frames' : List[VisionFrame],
'temp_vision_frame' : VisionFrame, 'temp_vision_frame' : VisionFrame,
'temp_vision_mask' : Mask 'temp_vision_mask' : Mask
}) })
@@ -1,28 +1,29 @@
from argparse import ArgumentParser from argparse import ArgumentParser
from functools import lru_cache from functools import lru_cache
from typing import Tuple from types import ModuleType
from typing import List, Tuple
import cv2 import cv2
import numpy import numpy
from cv2.typing import Size from cv2.typing import Size
import facefusion.capability_store
import facefusion.jobs.job_manager import facefusion.jobs.job_manager
import facefusion.jobs.job_store
from facefusion import config, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, inference_manager, logger, state_manager, translator, video_manager from facefusion import config, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, inference_manager, logger, state_manager, translator, video_manager
from facefusion.common_helper import create_int_metavar from facefusion.common_helper import create_int_metavar, get_middle
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url_by_provider from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url_by_provider
from facefusion.face_analyser import scale_face from facefusion.face_creator import scale_face
from facefusion.face_helper import paste_back, warp_face_by_face_landmark_5 from facefusion.face_helper import paste_back, warp_face_by_face_landmark_5
from facefusion.face_masker import create_area_mask, create_box_mask, create_occlusion_mask, create_region_mask from facefusion.face_masker import create_area_mask, create_box_mask, create_occlusion_mask, create_region_mask
from facefusion.face_selector import select_faces from facefusion.face_selector import select_faces
from facefusion.filesystem import get_file_name, in_directory, is_image, is_video, resolve_file_paths, resolve_relative_path, same_file_extension from facefusion.filesystem import get_file_name, in_directory, is_image, is_video, resolve_file_paths, resolve_relative_path
from facefusion.processors.modules.deep_swapper import choices as deep_swapper_choices from facefusion.processors.modules.deep_swapper import choices as deep_swapper_choices
from facefusion.processors.modules.deep_swapper.types import DeepSwapperInputs, DeepSwapperMorph from facefusion.processors.modules.deep_swapper.types import DeepSwapperInputs, DeepSwapperMorph
from facefusion.processors.types import ProcessorOutputs from facefusion.processors.types import ApplyStateItem, ProcessorOutputs
from facefusion.program_helper import find_argument_group from facefusion.program_helper import find_argument_group
from facefusion.thread_helper import thread_semaphore from facefusion.thread_helper import thread_semaphore
from facefusion.types import ApplyStateItem, Args, DownloadScope, Face, InferencePool, Mask, ModelOptions, ModelSet, ProcessMode, VisionFrame from facefusion.types import Args, DownloadScope, Face, InferencePool, Mask, ModelOptions, ModelSet, ProcessMode, VisionFrame
from facefusion.vision import conditional_match_frame_color, read_static_image, read_static_video_frame from facefusion.vision import conditional_match_frame_color, read_static_image, read_static_video_chunk, read_static_video_frame
@lru_cache() @lru_cache()
@@ -276,9 +277,26 @@ def get_model_size() -> Size:
def register_args(program : ArgumentParser) -> None: def register_args(program : ArgumentParser) -> None:
group_processors = find_argument_group(program, 'processors') group_processors = find_argument_group(program, 'processors')
if group_processors: if group_processors:
group_processors.add_argument('--deep-swapper-model', help = translator.get('help.model', __package__), default = config.get_str_value('processors', 'deep_swapper_model', 'iperov/elon_musk_224'), choices = deep_swapper_choices.deep_swapper_models) facefusion.capability_store.register_capability_set(
group_processors.add_argument('--deep-swapper-morph', help = translator.get('help.morph', __package__), type = int, default = config.get_int_value('processors', 'deep_swapper_morph', '100'), choices = deep_swapper_choices.deep_swapper_morph_range, metavar = create_int_metavar(deep_swapper_choices.deep_swapper_morph_range)) [
facefusion.jobs.job_store.register_step_keys([ 'deep_swapper_model', 'deep_swapper_morph' ]) group_processors.add_argument(
'--deep-swapper-model',
help = translator.get('help.model', __package__),
default = config.get_str_value('processors', 'deep_swapper_model', 'iperov/elon_musk_224'),
choices = deep_swapper_choices.deep_swapper_models
),
group_processors.add_argument(
'--deep-swapper-morph',
help = translator.get('help.morph', __package__),
type = int,
default = config.get_int_value('processors', 'deep_swapper_morph', '100'),
choices = deep_swapper_choices.deep_swapper_morph_range,
metavar = create_int_metavar(deep_swapper_choices.deep_swapper_morph_range)
)
],
scopes = [ 'api', 'cli' ],
groups = [ 'deep_swapper' ]
)
def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None: def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
@@ -286,10 +304,18 @@ def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('deep_swapper_morph', args.get('deep_swapper_morph')) apply_state_item('deep_swapper_morph', args.get('deep_swapper_morph'))
def get_common_modules() -> List[ModuleType]:
return [ content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer ]
def pre_check() -> bool: def pre_check() -> bool:
model_hash_set = get_model_options().get('hashes') model_hash_set = get_model_options().get('hashes')
model_source_set = get_model_options().get('sources') model_source_set = get_model_options().get('sources')
for common_module in get_common_modules():
if not common_module.pre_check():
return False
if model_hash_set and model_source_set: if model_hash_set and model_source_set:
return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set) return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set)
return True return True
@@ -299,28 +325,25 @@ def pre_process(mode : ProcessMode) -> bool:
if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')): if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')):
logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__)
return False return False
if state_manager.get_item('workflow') in [ 'audio-to-image:video', 'image-to-image', 'image-to-video' ]:
if mode == 'output' and not in_directory(state_manager.get_item('output_path')): if mode == 'output' and not in_directory(state_manager.get_item('output_path')):
logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__)
return False return False
if mode == 'output' and not same_file_extension(state_manager.get_item('target_path'), state_manager.get_item('output_path')):
logger.error(translator.get('match_target_and_output_extension') + translator.get('exclamation_mark'), __name__)
return False
return True return True
def post_process() -> None: def post_process() -> None:
read_static_image.cache_clear() read_static_image.cache_clear()
read_static_video_frame.cache_clear() read_static_video_frame.cache_clear()
read_static_video_chunk.cache_clear()
video_manager.clear_video_pool() video_manager.clear_video_pool()
if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]: if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]:
clear_inference_pool() clear_inference_pool()
if state_manager.get_item('video_memory_strategy') == 'strict': if state_manager.get_item('video_memory_strategy') == 'strict':
content_analyser.clear_inference_pool() for common_module in get_common_modules():
face_classifier.clear_inference_pool() common_module.clear_inference_pool()
face_detector.clear_inference_pool()
face_landmarker.clear_inference_pool()
face_masker.clear_inference_pool()
face_recognizer.clear_inference_pool()
def swap_face(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame: def swap_face(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame:
@@ -411,10 +434,13 @@ def prepare_crop_mask(crop_source_mask : Mask, crop_target_mask : Mask) -> Mask:
def process_frame(inputs : DeepSwapperInputs) -> ProcessorOutputs: def process_frame(inputs : DeepSwapperInputs) -> ProcessorOutputs:
reference_vision_frame = inputs.get('reference_vision_frame') reference_vision_frame = inputs.get('reference_vision_frame')
target_vision_frame = inputs.get('target_vision_frame') source_vision_frames = inputs.get('source_vision_frames')
target_vision_frames = inputs.get('target_vision_frames')
temp_vision_frame = inputs.get('temp_vision_frame') temp_vision_frame = inputs.get('temp_vision_frame')
temp_vision_mask = inputs.get('temp_vision_mask') temp_vision_mask = inputs.get('temp_vision_mask')
target_faces = select_faces(reference_vision_frame, target_vision_frame)
target_vision_frame = get_middle(target_vision_frames)
target_faces = select_faces(reference_vision_frame, source_vision_frames, target_vision_frames)
if target_faces: if target_faces:
for target_face in target_faces: for target_face in target_faces:
@@ -1,4 +1,4 @@
from typing import Any, TypeAlias, TypedDict from typing import Any, List, TypeAlias, TypedDict
from numpy.typing import NDArray from numpy.typing import NDArray
@@ -7,7 +7,8 @@ from facefusion.types import Mask, VisionFrame
DeepSwapperInputs = TypedDict('DeepSwapperInputs', DeepSwapperInputs = TypedDict('DeepSwapperInputs',
{ {
'reference_vision_frame' : VisionFrame, 'reference_vision_frame' : VisionFrame,
'target_vision_frame' : VisionFrame, 'source_vision_frames' : List[VisionFrame],
'target_vision_frames' : List[VisionFrame],
'temp_vision_frame' : VisionFrame, 'temp_vision_frame' : VisionFrame,
'temp_vision_mask' : Mask 'temp_vision_mask' : Mask
}) })
@@ -1,28 +1,29 @@
from argparse import ArgumentParser from argparse import ArgumentParser
from functools import lru_cache from functools import lru_cache
from typing import Tuple from types import ModuleType
from typing import List, Tuple
import cv2 import cv2
import numpy import numpy
import facefusion.capability_store
import facefusion.jobs.job_manager import facefusion.jobs.job_manager
import facefusion.jobs.job_store
from facefusion import config, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, inference_manager, logger, state_manager, translator, video_manager from facefusion import config, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, inference_manager, logger, state_manager, translator, video_manager
from facefusion.common_helper import create_int_metavar from facefusion.common_helper import create_int_metavar, get_middle
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url
from facefusion.face_analyser import scale_face from facefusion.face_creator import scale_face
from facefusion.face_helper import paste_back, warp_face_by_face_landmark_5 from facefusion.face_helper import paste_back, warp_face_by_face_landmark_5
from facefusion.face_masker import create_box_mask, create_occlusion_mask from facefusion.face_masker import create_box_mask, create_occlusion_mask
from facefusion.face_selector import select_faces from facefusion.face_selector import select_faces
from facefusion.filesystem import in_directory, is_image, is_video, resolve_relative_path, same_file_extension from facefusion.filesystem import in_directory, is_image, is_video, resolve_relative_path
from facefusion.processors.live_portrait import create_rotation, limit_expression from facefusion.processors.live_portrait import create_rotation, limit_expression
from facefusion.processors.modules.expression_restorer import choices as expression_restorer_choices from facefusion.processors.modules.expression_restorer import choices as expression_restorer_choices
from facefusion.processors.modules.expression_restorer.types import ExpressionRestorerInputs from facefusion.processors.modules.expression_restorer.types import ExpressionRestorerInputs
from facefusion.processors.types import LivePortraitExpression, LivePortraitFeatureVolume, LivePortraitMotionPoints, LivePortraitPitch, LivePortraitRoll, LivePortraitScale, LivePortraitTranslation, LivePortraitYaw, ProcessorOutputs from facefusion.processors.types import ApplyStateItem, LivePortraitExpression, LivePortraitFeatureVolume, LivePortraitMotionPoints, LivePortraitPitch, LivePortraitRoll, LivePortraitScale, LivePortraitTranslation, LivePortraitYaw, ProcessorOutputs
from facefusion.program_helper import find_argument_group from facefusion.program_helper import find_argument_group
from facefusion.thread_helper import conditional_thread_semaphore, thread_semaphore from facefusion.thread_helper import conditional_thread_semaphore, thread_semaphore
from facefusion.types import ApplyStateItem, Args, DownloadScope, Face, InferencePool, ModelOptions, ModelSet, ProcessMode, VisionFrame from facefusion.types import Args, DownloadScope, Face, InferencePool, ModelOptions, ModelSet, ProcessMode, VisionFrame
from facefusion.vision import read_static_image, read_static_video_frame from facefusion.vision import read_static_image, read_static_video_chunk, read_static_video_frame
@lru_cache() @lru_cache()
@@ -99,10 +100,34 @@ def get_model_options() -> ModelOptions:
def register_args(program : ArgumentParser) -> None: def register_args(program : ArgumentParser) -> None:
group_processors = find_argument_group(program, 'processors') group_processors = find_argument_group(program, 'processors')
if group_processors: if group_processors:
group_processors.add_argument('--expression-restorer-model', help = translator.get('help.model', __package__), default = config.get_str_value('processors', 'expression_restorer_model', 'live_portrait'), choices = expression_restorer_choices.expression_restorer_models) facefusion.capability_store.register_capability_set(
group_processors.add_argument('--expression-restorer-factor', help = translator.get('help.factor', __package__), type = int, default = config.get_int_value('processors', 'expression_restorer_factor', '80'), choices = expression_restorer_choices.expression_restorer_factor_range, metavar = create_int_metavar(expression_restorer_choices.expression_restorer_factor_range)) [
group_processors.add_argument('--expression-restorer-areas', help = translator.get('help.areas', __package__).format(choices = ', '.join(expression_restorer_choices.expression_restorer_areas)), default = config.get_str_list('processors', 'expression_restorer_areas', ' '.join(expression_restorer_choices.expression_restorer_areas)), choices = expression_restorer_choices.expression_restorer_areas, nargs = '+', metavar = 'EXPRESSION_RESTORER_AREAS') group_processors.add_argument(
facefusion.jobs.job_store.register_step_keys([ 'expression_restorer_model', 'expression_restorer_factor', 'expression_restorer_areas' ]) '--expression-restorer-model',
help = translator.get('help.model', __package__),
default = config.get_str_value('processors', 'expression_restorer_model', 'live_portrait'),
choices = expression_restorer_choices.expression_restorer_models
),
group_processors.add_argument(
'--expression-restorer-factor',
help = translator.get('help.factor', __package__),
type = int,
default = config.get_int_value('processors', 'expression_restorer_factor', '80'),
choices = expression_restorer_choices.expression_restorer_factor_range,
metavar = create_int_metavar(expression_restorer_choices.expression_restorer_factor_range)
),
group_processors.add_argument(
'--expression-restorer-areas',
help = translator.get('help.areas', __package__).format(choices = ', '.join(expression_restorer_choices.expression_restorer_areas)),
default = config.get_str_list('processors', 'expression_restorer_areas', ' '.join(expression_restorer_choices.expression_restorer_areas)),
choices = expression_restorer_choices.expression_restorer_areas,
nargs = '+',
metavar = 'EXPRESSION_RESTORER_AREAS'
)
],
scopes = [ 'api', 'cli' ],
groups = [ 'expression_restorer' ]
)
def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None: def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
@@ -111,10 +136,18 @@ def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('expression_restorer_areas', args.get('expression_restorer_areas')) apply_state_item('expression_restorer_areas', args.get('expression_restorer_areas'))
def get_common_modules() -> List[ModuleType]:
return [ content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer ]
def pre_check() -> bool: def pre_check() -> bool:
model_hash_set = get_model_options().get('hashes') model_hash_set = get_model_options().get('hashes')
model_source_set = get_model_options().get('sources') model_source_set = get_model_options().get('sources')
for common_module in get_common_modules():
if not common_module.pre_check():
return False
return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set) return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set)
@@ -125,28 +158,25 @@ def pre_process(mode : ProcessMode) -> bool:
if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')): if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')):
logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__)
return False return False
if state_manager.get_item('workflow') in [ 'audio-to-image:video', 'image-to-image', 'image-to-video' ]:
if mode == 'output' and not in_directory(state_manager.get_item('output_path')): if mode == 'output' and not in_directory(state_manager.get_item('output_path')):
logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__)
return False return False
if mode == 'output' and not same_file_extension(state_manager.get_item('target_path'), state_manager.get_item('output_path')):
logger.error(translator.get('match_target_and_output_extension') + translator.get('exclamation_mark'), __name__)
return False
return True return True
def post_process() -> None: def post_process() -> None:
read_static_image.cache_clear() read_static_image.cache_clear()
read_static_video_frame.cache_clear() read_static_video_frame.cache_clear()
read_static_video_chunk.cache_clear()
video_manager.clear_video_pool() video_manager.clear_video_pool()
if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]: if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]:
clear_inference_pool() clear_inference_pool()
if state_manager.get_item('video_memory_strategy') == 'strict': if state_manager.get_item('video_memory_strategy') == 'strict':
content_analyser.clear_inference_pool() for common_module in get_common_modules():
face_classifier.clear_inference_pool() common_module.clear_inference_pool()
face_detector.clear_inference_pool()
face_landmarker.clear_inference_pool()
face_masker.clear_inference_pool()
face_recognizer.clear_inference_pool()
def restore_expression(target_face : Face, target_vision_frame : VisionFrame, temp_vision_frame : VisionFrame) -> VisionFrame: def restore_expression(target_face : Face, target_vision_frame : VisionFrame, temp_vision_frame : VisionFrame) -> VisionFrame:
@@ -257,10 +287,13 @@ def normalize_crop_frame(crop_vision_frame : VisionFrame) -> VisionFrame:
def process_frame(inputs : ExpressionRestorerInputs) -> ProcessorOutputs: def process_frame(inputs : ExpressionRestorerInputs) -> ProcessorOutputs:
reference_vision_frame = inputs.get('reference_vision_frame') reference_vision_frame = inputs.get('reference_vision_frame')
target_vision_frame = inputs.get('target_vision_frame') source_vision_frames = inputs.get('source_vision_frames')
target_vision_frames = inputs.get('target_vision_frames')
temp_vision_frame = inputs.get('temp_vision_frame') temp_vision_frame = inputs.get('temp_vision_frame')
temp_vision_mask = inputs.get('temp_vision_mask') temp_vision_mask = inputs.get('temp_vision_mask')
target_faces = select_faces(reference_vision_frame, target_vision_frame)
target_vision_frame = get_middle(target_vision_frames)
target_faces = select_faces(reference_vision_frame, source_vision_frames, target_vision_frames)
if target_faces: if target_faces:
for target_face in target_faces: for target_face in target_faces:
@@ -6,7 +6,7 @@ ExpressionRestorerInputs = TypedDict('ExpressionRestorerInputs',
{ {
'reference_vision_frame' : VisionFrame, 'reference_vision_frame' : VisionFrame,
'source_vision_frames' : List[VisionFrame], 'source_vision_frames' : List[VisionFrame],
'target_vision_frame' : VisionFrame, 'target_vision_frames' : List[VisionFrame],
'temp_vision_frame' : VisionFrame, 'temp_vision_frame' : VisionFrame,
'temp_vision_mask' : Mask 'temp_vision_mask' : Mask
}) })
@@ -1,22 +1,25 @@
from argparse import ArgumentParser from argparse import ArgumentParser
from types import ModuleType
from typing import List
import cv2 import cv2
import numpy import numpy
import facefusion.capability_store
import facefusion.jobs.job_manager import facefusion.jobs.job_manager
import facefusion.jobs.job_store
from facefusion import config, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, logger, state_manager, translator, video_manager from facefusion import config, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, logger, state_manager, translator, video_manager
from facefusion.face_analyser import scale_face from facefusion.common_helper import get_middle
from facefusion.face_creator import scale_face
from facefusion.face_helper import warp_face_by_face_landmark_5 from facefusion.face_helper import warp_face_by_face_landmark_5
from facefusion.face_masker import create_area_mask, create_box_mask, create_occlusion_mask, create_region_mask from facefusion.face_masker import create_area_mask, create_box_mask, create_occlusion_mask, create_region_mask
from facefusion.face_selector import select_faces from facefusion.face_selector import select_faces
from facefusion.filesystem import in_directory, is_image, is_video, same_file_extension from facefusion.filesystem import in_directory, is_image, is_video
from facefusion.processors.modules.face_debugger import choices as face_debugger_choices from facefusion.processors.modules.face_debugger import choices as face_debugger_choices
from facefusion.processors.modules.face_debugger.types import FaceDebuggerInputs from facefusion.processors.modules.face_debugger.types import FaceDebuggerInputs
from facefusion.processors.types import ProcessorOutputs from facefusion.processors.types import ApplyStateItem, ProcessorOutputs
from facefusion.program_helper import find_argument_group from facefusion.program_helper import find_argument_group
from facefusion.types import ApplyStateItem, Args, Face, InferencePool, ProcessMode, VisionFrame from facefusion.types import Args, Face, InferencePool, ProcessMode, VisionFrame
from facefusion.vision import read_static_image, read_static_video_frame from facefusion.vision import read_static_image, read_static_video_chunk, read_static_video_frame
def get_inference_pool() -> InferencePool: def get_inference_pool() -> InferencePool:
@@ -30,15 +33,34 @@ def clear_inference_pool() -> None:
def register_args(program : ArgumentParser) -> None: def register_args(program : ArgumentParser) -> None:
group_processors = find_argument_group(program, 'processors') group_processors = find_argument_group(program, 'processors')
if group_processors: if group_processors:
group_processors.add_argument('--face-debugger-items', help = translator.get('help.items', __package__).format(choices = ', '.join(face_debugger_choices.face_debugger_items)), default = config.get_str_list('processors', 'face_debugger_items', 'face-landmark-5/68 face-mask'), choices = face_debugger_choices.face_debugger_items, nargs = '+', metavar = 'FACE_DEBUGGER_ITEMS') facefusion.capability_store.register_capability_set(
facefusion.jobs.job_store.register_step_keys([ 'face_debugger_items' ]) [
group_processors.add_argument(
'--face-debugger-items',
help = translator.get('help.items', __package__).format(choices = ', '.join(face_debugger_choices.face_debugger_items)),
default = config.get_str_list('processors', 'face_debugger_items', 'face-landmark-5/68 face-mask'),
choices = face_debugger_choices.face_debugger_items,
nargs = '+',
metavar = 'FACE_DEBUGGER_ITEMS'
)
],
scopes = [ 'api', 'cli' ],
groups = [ 'face_debugger' ]
)
def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None: def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('face_debugger_items', args.get('face_debugger_items')) apply_state_item('face_debugger_items', args.get('face_debugger_items'))
def get_common_modules() -> List[ModuleType]:
return [ content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer ]
def pre_check() -> bool: def pre_check() -> bool:
for common_module in get_common_modules():
if not common_module.pre_check():
return False
return True return True
@@ -46,26 +68,22 @@ def pre_process(mode : ProcessMode) -> bool:
if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')): if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')):
logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__)
return False return False
if state_manager.get_item('workflow') in [ 'audio-to-image:video', 'image-to-image', 'image-to-video' ]:
if mode == 'output' and not in_directory(state_manager.get_item('output_path')): if mode == 'output' and not in_directory(state_manager.get_item('output_path')):
logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__)
return False return False
if mode == 'output' and not same_file_extension(state_manager.get_item('target_path'), state_manager.get_item('output_path')):
logger.error(translator.get('match_target_and_output_extension') + translator.get('exclamation_mark'), __name__)
return False
return True return True
def post_process() -> None: def post_process() -> None:
read_static_image.cache_clear() read_static_image.cache_clear()
read_static_video_frame.cache_clear() read_static_video_frame.cache_clear()
read_static_video_chunk.cache_clear()
video_manager.clear_video_pool() video_manager.clear_video_pool()
if state_manager.get_item('video_memory_strategy') == 'strict': if state_manager.get_item('video_memory_strategy') == 'strict':
content_analyser.clear_inference_pool() for common_module in get_common_modules():
face_classifier.clear_inference_pool() common_module.clear_inference_pool()
face_detector.clear_inference_pool()
face_landmarker.clear_inference_pool()
face_masker.clear_inference_pool()
face_recognizer.clear_inference_pool()
def debug_face(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame: def debug_face(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame:
@@ -94,21 +112,22 @@ def debug_face(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFra
def draw_bounding_box(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame: def draw_bounding_box(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame:
temp_vision_frame = numpy.ascontiguousarray(temp_vision_frame) temp_vision_frame = numpy.ascontiguousarray(temp_vision_frame)
box_color = 0, 0, 255
border_color = 100, 100, 255
bounding_box = target_face.bounding_box.astype(numpy.int32) bounding_box = target_face.bounding_box.astype(numpy.int32)
x1, y1, x2, y2 = bounding_box x1, y1, x2, y2 = bounding_box
box_color = 0, 0, 255
border_scale = calculate_scale(temp_vision_frame)
border_color = 100, 100, 255
cv2.rectangle(temp_vision_frame, (x1, y1), (x2, y2), box_color, 2) cv2.rectangle(temp_vision_frame, (x1, y1), (x2, y2), box_color, border_scale)
if target_face.angle == 0: if target_face.angle == 0:
cv2.line(temp_vision_frame, (x1, y1), (x2, y1), border_color, 3) cv2.line(temp_vision_frame, (x1, y1), (x2, y1), border_color, border_scale + 1)
if target_face.angle == 180: if target_face.angle == 180:
cv2.line(temp_vision_frame, (x1, y2), (x2, y2), border_color, 3) cv2.line(temp_vision_frame, (x1, y2), (x2, y2), border_color, border_scale + 1)
if target_face.angle == 90: if target_face.angle == 90:
cv2.line(temp_vision_frame, (x2, y1), (x2, y2), border_color, 3) cv2.line(temp_vision_frame, (x2, y1), (x2, y2), border_color, border_scale + 1)
if target_face.angle == 270: if target_face.angle == 270:
cv2.line(temp_vision_frame, (x1, y1), (x1, y2), border_color, 3) cv2.line(temp_vision_frame, (x1, y1), (x1, y2), border_color, border_scale + 1)
return temp_vision_frame return temp_vision_frame
@@ -122,11 +141,15 @@ def draw_face_mask(target_face : Face, temp_vision_frame : VisionFrame) -> Visio
crop_vision_frame, affine_matrix = warp_face_by_face_landmark_5(temp_vision_frame, face_landmark_5_68, 'arcface_128', (512, 512)) crop_vision_frame, affine_matrix = warp_face_by_face_landmark_5(temp_vision_frame, face_landmark_5_68, 'arcface_128', (512, 512))
inverse_matrix = cv2.invertAffineTransform(affine_matrix) inverse_matrix = cv2.invertAffineTransform(affine_matrix)
temp_size = temp_vision_frame.shape[:2][::-1] temp_size = temp_vision_frame.shape[:2][::-1]
mask_scale = calculate_scale(temp_vision_frame)
mask_color = 0, 255, 0 mask_color = 0, 255, 0
if numpy.array_equal(face_landmark_5, face_landmark_5_68): if numpy.array_equal(face_landmark_5, face_landmark_5_68):
mask_color = 255, 255, 0 mask_color = 255, 255, 0
if target_face.origin == 'refill':
mask_color = 0, 165, 255
if 'box' in state_manager.get_item('face_mask_types'): if 'box' in state_manager.get_item('face_mask_types'):
box_mask = create_box_mask(crop_vision_frame, 0, state_manager.get_item('face_mask_padding')) box_mask = create_box_mask(crop_vision_frame, 0, state_manager.get_item('face_mask_padding'))
crop_masks.append(box_mask) crop_masks.append(box_mask)
@@ -149,7 +172,7 @@ def draw_face_mask(target_face : Face, temp_vision_frame : VisionFrame) -> Visio
inverse_vision_frame = cv2.warpAffine(crop_mask, inverse_matrix, temp_size) inverse_vision_frame = cv2.warpAffine(crop_mask, inverse_matrix, temp_size)
inverse_vision_frame = cv2.threshold(inverse_vision_frame, 100, 255, cv2.THRESH_BINARY)[1] inverse_vision_frame = cv2.threshold(inverse_vision_frame, 100, 255, cv2.THRESH_BINARY)[1]
inverse_contours, _ = cv2.findContours(inverse_vision_frame, cv2.RETR_LIST, cv2.CHAIN_APPROX_NONE) inverse_contours, _ = cv2.findContours(inverse_vision_frame, cv2.RETR_LIST, cv2.CHAIN_APPROX_NONE)
cv2.drawContours(temp_vision_frame, inverse_contours, -1, mask_color, 2) cv2.drawContours(temp_vision_frame, inverse_contours, -1, mask_color, mask_scale)
return temp_vision_frame return temp_vision_frame
@@ -157,13 +180,17 @@ def draw_face_mask(target_face : Face, temp_vision_frame : VisionFrame) -> Visio
def draw_face_landmark_5(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame: def draw_face_landmark_5(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame:
temp_vision_frame = numpy.ascontiguousarray(temp_vision_frame) temp_vision_frame = numpy.ascontiguousarray(temp_vision_frame)
face_landmark_5 = target_face.landmark_set.get('5') face_landmark_5 = target_face.landmark_set.get('5')
point_scale = calculate_scale(temp_vision_frame)
point_color = 0, 0, 255 point_color = 0, 0, 255
if target_face.origin == 'refill':
point_color = 0, 165, 255
if numpy.any(face_landmark_5): if numpy.any(face_landmark_5):
face_landmark_5 = face_landmark_5.astype(numpy.int32) face_landmark_5 = face_landmark_5.astype(numpy.int32)
for point in face_landmark_5: for point in face_landmark_5:
cv2.circle(temp_vision_frame, tuple(point), 3, point_color, -1) cv2.circle(temp_vision_frame, tuple(point), point_scale, point_color, -1)
return temp_vision_frame return temp_vision_frame
@@ -172,16 +199,20 @@ def draw_face_landmark_5_68(target_face : Face, temp_vision_frame : VisionFrame)
temp_vision_frame = numpy.ascontiguousarray(temp_vision_frame) temp_vision_frame = numpy.ascontiguousarray(temp_vision_frame)
face_landmark_5 = target_face.landmark_set.get('5') face_landmark_5 = target_face.landmark_set.get('5')
face_landmark_5_68 = target_face.landmark_set.get('5/68') face_landmark_5_68 = target_face.landmark_set.get('5/68')
point_scale = calculate_scale(temp_vision_frame)
point_color = 0, 255, 0 point_color = 0, 255, 0
if numpy.array_equal(face_landmark_5, face_landmark_5_68): if numpy.array_equal(face_landmark_5, face_landmark_5_68):
point_color = 255, 255, 0 point_color = 255, 255, 0
if target_face.origin == 'refill':
point_color = 0, 165, 255
if numpy.any(face_landmark_5_68): if numpy.any(face_landmark_5_68):
face_landmark_5_68 = face_landmark_5_68.astype(numpy.int32) face_landmark_5_68 = face_landmark_5_68.astype(numpy.int32)
for point in face_landmark_5_68: for point in face_landmark_5_68:
cv2.circle(temp_vision_frame, tuple(point), 3, point_color, -1) cv2.circle(temp_vision_frame, tuple(point), point_scale, point_color, -1)
return temp_vision_frame return temp_vision_frame
@@ -190,16 +221,20 @@ def draw_face_landmark_68(target_face : Face, temp_vision_frame : VisionFrame) -
temp_vision_frame = numpy.ascontiguousarray(temp_vision_frame) temp_vision_frame = numpy.ascontiguousarray(temp_vision_frame)
face_landmark_68 = target_face.landmark_set.get('68') face_landmark_68 = target_face.landmark_set.get('68')
face_landmark_68_5 = target_face.landmark_set.get('68/5') face_landmark_68_5 = target_face.landmark_set.get('68/5')
point_scale = calculate_scale(temp_vision_frame)
point_color = 0, 255, 0 point_color = 0, 255, 0
if numpy.array_equal(face_landmark_68, face_landmark_68_5): if numpy.array_equal(face_landmark_68, face_landmark_68_5):
point_color = 255, 255, 0 point_color = 255, 255, 0
if target_face.origin == 'refill':
point_color = 0, 165, 255
if numpy.any(face_landmark_68): if numpy.any(face_landmark_68):
face_landmark_68 = face_landmark_68.astype(numpy.int32) face_landmark_68 = face_landmark_68.astype(numpy.int32)
for point in face_landmark_68: for point in face_landmark_68:
cv2.circle(temp_vision_frame, tuple(point), 3, point_color, -1) cv2.circle(temp_vision_frame, tuple(point), point_scale, point_color, -1)
return temp_vision_frame return temp_vision_frame
@@ -207,23 +242,36 @@ def draw_face_landmark_68(target_face : Face, temp_vision_frame : VisionFrame) -
def draw_face_landmark_68_5(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame: def draw_face_landmark_68_5(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame:
temp_vision_frame = numpy.ascontiguousarray(temp_vision_frame) temp_vision_frame = numpy.ascontiguousarray(temp_vision_frame)
face_landmark_68_5 = target_face.landmark_set.get('68/5') face_landmark_68_5 = target_face.landmark_set.get('68/5')
point_scale = calculate_scale(temp_vision_frame)
point_color = 255, 255, 0 point_color = 255, 255, 0
if target_face.origin == 'refill':
point_color = 0, 165, 255
if numpy.any(face_landmark_68_5): if numpy.any(face_landmark_68_5):
face_landmark_68_5 = face_landmark_68_5.astype(numpy.int32) face_landmark_68_5 = face_landmark_68_5.astype(numpy.int32)
for point in face_landmark_68_5: for point in face_landmark_68_5:
cv2.circle(temp_vision_frame, tuple(point), 3, point_color, -1) cv2.circle(temp_vision_frame, tuple(point), point_scale, point_color, -1)
return temp_vision_frame return temp_vision_frame
def calculate_scale(temp_vision_frame : VisionFrame) -> int:
frame_height, _ = temp_vision_frame.shape[:2]
frame_scale = round(frame_height / 270)
return max(1, min(10, frame_scale))
def process_frame(inputs : FaceDebuggerInputs) -> ProcessorOutputs: def process_frame(inputs : FaceDebuggerInputs) -> ProcessorOutputs:
reference_vision_frame = inputs.get('reference_vision_frame') reference_vision_frame = inputs.get('reference_vision_frame')
target_vision_frame = inputs.get('target_vision_frame') source_vision_frames = inputs.get('source_vision_frames')
target_vision_frames = inputs.get('target_vision_frames')
temp_vision_frame = inputs.get('temp_vision_frame') temp_vision_frame = inputs.get('temp_vision_frame')
temp_vision_mask = inputs.get('temp_vision_mask') temp_vision_mask = inputs.get('temp_vision_mask')
target_faces = select_faces(reference_vision_frame, target_vision_frame)
target_vision_frame = get_middle(target_vision_frames)
target_faces = select_faces(reference_vision_frame, source_vision_frames, target_vision_frames)
if target_faces: if target_faces:
for target_face in target_faces: for target_face in target_faces:
@@ -231,5 +279,3 @@ def process_frame(inputs : FaceDebuggerInputs) -> ProcessorOutputs:
temp_vision_frame = debug_face(target_face, temp_vision_frame) temp_vision_frame = debug_face(target_face, temp_vision_frame)
return temp_vision_frame, temp_vision_mask return temp_vision_frame, temp_vision_mask
@@ -1,11 +1,12 @@
from typing import Literal, TypedDict from typing import List, Literal, TypedDict
from facefusion.types import Mask, VisionFrame from facefusion.types import Mask, VisionFrame
FaceDebuggerInputs = TypedDict('FaceDebuggerInputs', FaceDebuggerInputs = TypedDict('FaceDebuggerInputs',
{ {
'reference_vision_frame' : VisionFrame, 'reference_vision_frame' : VisionFrame,
'target_vision_frame' : VisionFrame, 'source_vision_frames' : List[VisionFrame],
'target_vision_frames' : List[VisionFrame],
'temp_vision_frame' : VisionFrame, 'temp_vision_frame' : VisionFrame,
'temp_vision_mask' : Mask 'temp_vision_mask' : Mask
}) })
+152 -35
View File
@@ -1,28 +1,29 @@
from argparse import ArgumentParser from argparse import ArgumentParser
from functools import lru_cache from functools import lru_cache
from typing import Tuple from types import ModuleType
from typing import List, Tuple
import cv2 import cv2
import numpy import numpy
import facefusion.capability_store
import facefusion.jobs.job_manager import facefusion.jobs.job_manager
import facefusion.jobs.job_store
from facefusion import config, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, inference_manager, logger, state_manager, translator, video_manager from facefusion import config, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, inference_manager, logger, state_manager, translator, video_manager
from facefusion.common_helper import create_float_metavar from facefusion.common_helper import create_float_metavar, get_middle
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url
from facefusion.face_analyser import scale_face from facefusion.face_creator import scale_face
from facefusion.face_helper import paste_back, scale_face_landmark_5, warp_face_by_face_landmark_5 from facefusion.face_helper import paste_back, scale_face_landmark_5, warp_face_by_face_landmark_5
from facefusion.face_masker import create_box_mask from facefusion.face_masker import create_box_mask
from facefusion.face_selector import select_faces from facefusion.face_selector import select_faces
from facefusion.filesystem import in_directory, is_image, is_video, resolve_relative_path, same_file_extension from facefusion.filesystem import in_directory, is_image, is_video, resolve_relative_path
from facefusion.processors.live_portrait import create_rotation, limit_angle, limit_expression from facefusion.processors.live_portrait import create_rotation, limit_angle, limit_expression
from facefusion.processors.modules.face_editor import choices as face_editor_choices from facefusion.processors.modules.face_editor import choices as face_editor_choices
from facefusion.processors.modules.face_editor.types import FaceEditorInputs from facefusion.processors.modules.face_editor.types import FaceEditorInputs
from facefusion.processors.types import LivePortraitExpression, LivePortraitFeatureVolume, LivePortraitMotionPoints, LivePortraitPitch, LivePortraitRoll, LivePortraitRotation, LivePortraitScale, LivePortraitTranslation, LivePortraitYaw, ProcessorOutputs from facefusion.processors.types import ApplyStateItem, LivePortraitExpression, LivePortraitFeatureVolume, LivePortraitMotionPoints, LivePortraitPitch, LivePortraitRoll, LivePortraitRotation, LivePortraitScale, LivePortraitTranslation, LivePortraitYaw, ProcessorOutputs
from facefusion.program_helper import find_argument_group from facefusion.program_helper import find_argument_group
from facefusion.thread_helper import conditional_thread_semaphore, thread_semaphore from facefusion.thread_helper import conditional_thread_semaphore, thread_semaphore
from facefusion.types import ApplyStateItem, Args, DownloadScope, Face, FaceLandmark68, InferencePool, ModelOptions, ModelSet, ProcessMode, VisionFrame from facefusion.types import Args, DownloadScope, Face, FaceLandmark68, InferencePool, ModelOptions, ModelSet, ProcessMode, VisionFrame
from facefusion.vision import read_static_image, read_static_video_frame from facefusion.vision import read_static_image, read_static_video_chunk, read_static_video_frame
@lru_cache() @lru_cache()
@@ -129,22 +130,130 @@ def get_model_options() -> ModelOptions:
def register_args(program : ArgumentParser) -> None: def register_args(program : ArgumentParser) -> None:
group_processors = find_argument_group(program, 'processors') group_processors = find_argument_group(program, 'processors')
if group_processors: if group_processors:
group_processors.add_argument('--face-editor-model', help = translator.get('help.model', __package__), default = config.get_str_value('processors', 'face_editor_model', 'live_portrait'), choices = face_editor_choices.face_editor_models) facefusion.capability_store.register_capability_set(
group_processors.add_argument('--face-editor-eyebrow-direction', help = translator.get('help.eyebrow_direction', __package__), type = float, default = config.get_float_value('processors', 'face_editor_eyebrow_direction', '0'), choices = face_editor_choices.face_editor_eyebrow_direction_range, metavar = create_float_metavar(face_editor_choices.face_editor_eyebrow_direction_range)) [
group_processors.add_argument('--face-editor-eye-gaze-horizontal', help = translator.get('help.eye_gaze_horizontal', __package__), type = float, default = config.get_float_value('processors', 'face_editor_eye_gaze_horizontal', '0'), choices = face_editor_choices.face_editor_eye_gaze_horizontal_range, metavar = create_float_metavar(face_editor_choices.face_editor_eye_gaze_horizontal_range)) group_processors.add_argument(
group_processors.add_argument('--face-editor-eye-gaze-vertical', help = translator.get('help.eye_gaze_vertical', __package__), type = float, default = config.get_float_value('processors', 'face_editor_eye_gaze_vertical', '0'), choices = face_editor_choices.face_editor_eye_gaze_vertical_range, metavar = create_float_metavar(face_editor_choices.face_editor_eye_gaze_vertical_range)) '--face-editor-model',
group_processors.add_argument('--face-editor-eye-open-ratio', help = translator.get('help.eye_open_ratio', __package__), type = float, default = config.get_float_value('processors', 'face_editor_eye_open_ratio', '0'), choices = face_editor_choices.face_editor_eye_open_ratio_range, metavar = create_float_metavar(face_editor_choices.face_editor_eye_open_ratio_range)) help = translator.get('help.model', __package__),
group_processors.add_argument('--face-editor-lip-open-ratio', help = translator.get('help.lip_open_ratio', __package__), type = float, default = config.get_float_value('processors', 'face_editor_lip_open_ratio', '0'), choices = face_editor_choices.face_editor_lip_open_ratio_range, metavar = create_float_metavar(face_editor_choices.face_editor_lip_open_ratio_range)) default = config.get_str_value('processors', 'face_editor_model', 'live_portrait'),
group_processors.add_argument('--face-editor-mouth-grim', help = translator.get('help.mouth_grim', __package__), type = float, default = config.get_float_value('processors', 'face_editor_mouth_grim', '0'), choices = face_editor_choices.face_editor_mouth_grim_range, metavar = create_float_metavar(face_editor_choices.face_editor_mouth_grim_range)) choices = face_editor_choices.face_editor_models
group_processors.add_argument('--face-editor-mouth-pout', help = translator.get('help.mouth_pout', __package__), type = float, default = config.get_float_value('processors', 'face_editor_mouth_pout', '0'), choices = face_editor_choices.face_editor_mouth_pout_range, metavar = create_float_metavar(face_editor_choices.face_editor_mouth_pout_range)) ),
group_processors.add_argument('--face-editor-mouth-purse', help = translator.get('help.mouth_purse', __package__), type = float, default = config.get_float_value('processors', 'face_editor_mouth_purse', '0'), choices = face_editor_choices.face_editor_mouth_purse_range, metavar = create_float_metavar(face_editor_choices.face_editor_mouth_purse_range)) group_processors.add_argument(
group_processors.add_argument('--face-editor-mouth-smile', help = translator.get('help.mouth_smile', __package__), type = float, default = config.get_float_value('processors', 'face_editor_mouth_smile', '0'), choices = face_editor_choices.face_editor_mouth_smile_range, metavar = create_float_metavar(face_editor_choices.face_editor_mouth_smile_range)) '--face-editor-eyebrow-direction',
group_processors.add_argument('--face-editor-mouth-position-horizontal', help = translator.get('help.mouth_position_horizontal', __package__), type = float, default = config.get_float_value('processors', 'face_editor_mouth_position_horizontal', '0'), choices = face_editor_choices.face_editor_mouth_position_horizontal_range, metavar = create_float_metavar(face_editor_choices.face_editor_mouth_position_horizontal_range)) help = translator.get('help.eyebrow_direction', __package__),
group_processors.add_argument('--face-editor-mouth-position-vertical', help = translator.get('help.mouth_position_vertical', __package__), type = float, default = config.get_float_value('processors', 'face_editor_mouth_position_vertical', '0'), choices = face_editor_choices.face_editor_mouth_position_vertical_range, metavar = create_float_metavar(face_editor_choices.face_editor_mouth_position_vertical_range)) type = float,
group_processors.add_argument('--face-editor-head-pitch', help = translator.get('help.head_pitch', __package__), type = float, default = config.get_float_value('processors', 'face_editor_head_pitch', '0'), choices = face_editor_choices.face_editor_head_pitch_range, metavar = create_float_metavar(face_editor_choices.face_editor_head_pitch_range)) default = config.get_float_value('processors', 'face_editor_eyebrow_direction', '0'),
group_processors.add_argument('--face-editor-head-yaw', help = translator.get('help.head_yaw', __package__), type = float, default = config.get_float_value('processors', 'face_editor_head_yaw', '0'), choices = face_editor_choices.face_editor_head_yaw_range, metavar = create_float_metavar(face_editor_choices.face_editor_head_yaw_range)) choices = face_editor_choices.face_editor_eyebrow_direction_range,
group_processors.add_argument('--face-editor-head-roll', help = translator.get('help.head_roll', __package__), type = float, default = config.get_float_value('processors', 'face_editor_head_roll', '0'), choices = face_editor_choices.face_editor_head_roll_range, metavar = create_float_metavar(face_editor_choices.face_editor_head_roll_range)) metavar = create_float_metavar(face_editor_choices.face_editor_eyebrow_direction_range)
facefusion.jobs.job_store.register_step_keys([ 'face_editor_model', 'face_editor_eyebrow_direction', 'face_editor_eye_gaze_horizontal', 'face_editor_eye_gaze_vertical', 'face_editor_eye_open_ratio', 'face_editor_lip_open_ratio', 'face_editor_mouth_grim', 'face_editor_mouth_pout', 'face_editor_mouth_purse', 'face_editor_mouth_smile', 'face_editor_mouth_position_horizontal', 'face_editor_mouth_position_vertical', 'face_editor_head_pitch', 'face_editor_head_yaw', 'face_editor_head_roll' ]) ),
group_processors.add_argument(
'--face-editor-eye-gaze-horizontal',
help = translator.get('help.eye_gaze_horizontal', __package__),
type = float,
default = config.get_float_value('processors', 'face_editor_eye_gaze_horizontal', '0'),
choices = face_editor_choices.face_editor_eye_gaze_horizontal_range,
metavar = create_float_metavar(face_editor_choices.face_editor_eye_gaze_horizontal_range)
),
group_processors.add_argument(
'--face-editor-eye-gaze-vertical',
help = translator.get('help.eye_gaze_vertical', __package__),
type = float,
default = config.get_float_value('processors', 'face_editor_eye_gaze_vertical', '0'),
choices = face_editor_choices.face_editor_eye_gaze_vertical_range,
metavar = create_float_metavar(face_editor_choices.face_editor_eye_gaze_vertical_range)
),
group_processors.add_argument(
'--face-editor-eye-open-ratio',
help = translator.get('help.eye_open_ratio', __package__),
type = float,
default = config.get_float_value('processors', 'face_editor_eye_open_ratio', '0'),
choices = face_editor_choices.face_editor_eye_open_ratio_range,
metavar = create_float_metavar(face_editor_choices.face_editor_eye_open_ratio_range)
),
group_processors.add_argument(
'--face-editor-lip-open-ratio',
help = translator.get('help.lip_open_ratio', __package__),
type = float,
default = config.get_float_value('processors', 'face_editor_lip_open_ratio', '0'),
choices = face_editor_choices.face_editor_lip_open_ratio_range,
metavar = create_float_metavar(face_editor_choices.face_editor_lip_open_ratio_range)
),
group_processors.add_argument(
'--face-editor-mouth-grim',
help = translator.get('help.mouth_grim', __package__),
type = float,
default = config.get_float_value('processors', 'face_editor_mouth_grim', '0'),
choices = face_editor_choices.face_editor_mouth_grim_range,
metavar = create_float_metavar(face_editor_choices.face_editor_mouth_grim_range)
),
group_processors.add_argument(
'--face-editor-mouth-pout',
help = translator.get('help.mouth_pout', __package__),
type = float,
default = config.get_float_value('processors', 'face_editor_mouth_pout', '0'),
choices = face_editor_choices.face_editor_mouth_pout_range,
metavar = create_float_metavar(face_editor_choices.face_editor_mouth_pout_range)
),
group_processors.add_argument(
'--face-editor-mouth-purse',
help = translator.get('help.mouth_purse', __package__),
type = float,
default = config.get_float_value('processors', 'face_editor_mouth_purse', '0'),
choices = face_editor_choices.face_editor_mouth_purse_range,
metavar = create_float_metavar(face_editor_choices.face_editor_mouth_purse_range)
),
group_processors.add_argument(
'--face-editor-mouth-smile',
help = translator.get('help.mouth_smile', __package__),
type = float,
default = config.get_float_value('processors', 'face_editor_mouth_smile', '0'),
choices = face_editor_choices.face_editor_mouth_smile_range,
metavar = create_float_metavar(face_editor_choices.face_editor_mouth_smile_range)
),
group_processors.add_argument(
'--face-editor-mouth-position-horizontal',
help = translator.get('help.mouth_position_horizontal', __package__),
type = float,
default = config.get_float_value('processors', 'face_editor_mouth_position_horizontal', '0'),
choices = face_editor_choices.face_editor_mouth_position_horizontal_range,
metavar = create_float_metavar(face_editor_choices.face_editor_mouth_position_horizontal_range)
),
group_processors.add_argument(
'--face-editor-mouth-position-vertical',
help = translator.get('help.mouth_position_vertical', __package__),
type = float,
default = config.get_float_value('processors', 'face_editor_mouth_position_vertical', '0'),
choices = face_editor_choices.face_editor_mouth_position_vertical_range,
metavar = create_float_metavar(face_editor_choices.face_editor_mouth_position_vertical_range)
),
group_processors.add_argument(
'--face-editor-head-pitch',
help = translator.get('help.head_pitch', __package__),
type = float,
default = config.get_float_value('processors', 'face_editor_head_pitch', '0'),
choices = face_editor_choices.face_editor_head_pitch_range,
metavar = create_float_metavar(face_editor_choices.face_editor_head_pitch_range)
),
group_processors.add_argument(
'--face-editor-head-yaw',
help = translator.get('help.head_yaw', __package__),
type = float,
default = config.get_float_value('processors', 'face_editor_head_yaw', '0'),
choices = face_editor_choices.face_editor_head_yaw_range,
metavar = create_float_metavar(face_editor_choices.face_editor_head_yaw_range)
),
group_processors.add_argument(
'--face-editor-head-roll',
help = translator.get('help.head_roll', __package__),
type = float,
default = config.get_float_value('processors', 'face_editor_head_roll', '0'),
choices = face_editor_choices.face_editor_head_roll_range,
metavar = create_float_metavar(face_editor_choices.face_editor_head_roll_range)
)
],
scopes = [ 'api', 'cli' ],
groups = [ 'face_editor' ]
)
def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None: def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
@@ -165,10 +274,18 @@ def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('face_editor_head_roll', args.get('face_editor_head_roll')) apply_state_item('face_editor_head_roll', args.get('face_editor_head_roll'))
def get_common_modules() -> List[ModuleType]:
return [ content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer ]
def pre_check() -> bool: def pre_check() -> bool:
model_hash_set = get_model_options().get('hashes') model_hash_set = get_model_options().get('hashes')
model_source_set = get_model_options().get('sources') model_source_set = get_model_options().get('sources')
for common_module in get_common_modules():
if not common_module.pre_check():
return False
return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set) return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set)
@@ -176,28 +293,25 @@ def pre_process(mode : ProcessMode) -> bool:
if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')): if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')):
logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__)
return False return False
if state_manager.get_item('workflow') in [ 'audio-to-image:video', 'image-to-image', 'image-to-video' ]:
if mode == 'output' and not in_directory(state_manager.get_item('output_path')): if mode == 'output' and not in_directory(state_manager.get_item('output_path')):
logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__)
return False return False
if mode == 'output' and not same_file_extension(state_manager.get_item('target_path'), state_manager.get_item('output_path')):
logger.error(translator.get('match_target_and_output_extension') + translator.get('exclamation_mark'), __name__)
return False
return True return True
def post_process() -> None: def post_process() -> None:
read_static_image.cache_clear() read_static_image.cache_clear()
read_static_video_frame.cache_clear() read_static_video_frame.cache_clear()
read_static_video_chunk.cache_clear()
video_manager.clear_video_pool() video_manager.clear_video_pool()
if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]: if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]:
clear_inference_pool() clear_inference_pool()
if state_manager.get_item('video_memory_strategy') == 'strict': if state_manager.get_item('video_memory_strategy') == 'strict':
content_analyser.clear_inference_pool() for common_module in get_common_modules():
face_classifier.clear_inference_pool() common_module.clear_inference_pool()
face_detector.clear_inference_pool()
face_landmarker.clear_inference_pool()
face_masker.clear_inference_pool()
face_recognizer.clear_inference_pool()
def edit_face(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame: def edit_face(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame:
@@ -486,10 +600,13 @@ def normalize_crop_frame(crop_vision_frame : VisionFrame) -> VisionFrame:
def process_frame(inputs : FaceEditorInputs) -> ProcessorOutputs: def process_frame(inputs : FaceEditorInputs) -> ProcessorOutputs:
reference_vision_frame = inputs.get('reference_vision_frame') reference_vision_frame = inputs.get('reference_vision_frame')
target_vision_frame = inputs.get('target_vision_frame') source_vision_frames = inputs.get('source_vision_frames')
target_vision_frames = inputs.get('target_vision_frames')
temp_vision_frame = inputs.get('temp_vision_frame') temp_vision_frame = inputs.get('temp_vision_frame')
temp_vision_mask = inputs.get('temp_vision_mask') temp_vision_mask = inputs.get('temp_vision_mask')
target_faces = select_faces(reference_vision_frame, target_vision_frame)
target_vision_frame = get_middle(target_vision_frames)
target_faces = select_faces(reference_vision_frame, source_vision_frames, target_vision_frames)
if target_faces: if target_faces:
for target_face in target_faces: for target_face in target_faces:
@@ -1,11 +1,12 @@
from typing import Literal, TypedDict from typing import List, Literal, TypedDict
from facefusion.types import Mask, VisionFrame from facefusion.types import Mask, VisionFrame
FaceEditorInputs = TypedDict('FaceEditorInputs', FaceEditorInputs = TypedDict('FaceEditorInputs',
{ {
'reference_vision_frame' : VisionFrame, 'reference_vision_frame' : VisionFrame,
'target_vision_frame' : VisionFrame, 'source_vision_frames' : List[VisionFrame],
'target_vision_frames' : List[VisionFrame],
'temp_vision_frame' : VisionFrame, 'temp_vision_frame' : VisionFrame,
'temp_vision_mask' : Mask 'temp_vision_mask' : Mask
}) })
@@ -1,25 +1,27 @@
from argparse import ArgumentParser from argparse import ArgumentParser
from functools import lru_cache from functools import lru_cache
from types import ModuleType
from typing import List
import numpy import numpy
import facefusion.capability_store
import facefusion.jobs.job_manager import facefusion.jobs.job_manager
import facefusion.jobs.job_store
from facefusion import config, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, inference_manager, logger, state_manager, translator, video_manager from facefusion import config, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, inference_manager, logger, state_manager, translator, video_manager
from facefusion.common_helper import create_float_metavar, create_int_metavar from facefusion.common_helper import create_float_metavar, create_int_metavar, get_middle
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url
from facefusion.face_analyser import scale_face from facefusion.face_creator import scale_face
from facefusion.face_helper import paste_back, warp_face_by_face_landmark_5 from facefusion.face_helper import paste_back, warp_face_by_face_landmark_5
from facefusion.face_masker import create_box_mask, create_occlusion_mask from facefusion.face_masker import create_box_mask, create_occlusion_mask
from facefusion.face_selector import select_faces from facefusion.face_selector import select_faces
from facefusion.filesystem import in_directory, is_image, is_video, resolve_relative_path, same_file_extension from facefusion.filesystem import in_directory, is_image, is_video, resolve_relative_path
from facefusion.processors.modules.face_enhancer import choices as face_enhancer_choices from facefusion.processors.modules.face_enhancer import choices as face_enhancer_choices
from facefusion.processors.modules.face_enhancer.types import FaceEnhancerInputs, FaceEnhancerWeight from facefusion.processors.modules.face_enhancer.types import FaceEnhancerInputs, FaceEnhancerWeight
from facefusion.processors.types import ProcessorOutputs from facefusion.processors.types import ApplyStateItem, ProcessorOutputs
from facefusion.program_helper import find_argument_group from facefusion.program_helper import find_argument_group
from facefusion.thread_helper import thread_semaphore from facefusion.thread_helper import thread_semaphore
from facefusion.types import ApplyStateItem, Args, DownloadScope, Face, InferencePool, ModelOptions, ModelSet, ProcessMode, VisionFrame from facefusion.types import Args, DownloadScope, Face, InferencePool, ModelOptions, ModelSet, ProcessMode, VisionFrame
from facefusion.vision import blend_frame, read_static_image, read_static_video_frame from facefusion.vision import blend_frame, read_static_image, read_static_video_chunk, read_static_video_frame
@lru_cache() @lru_cache()
@@ -292,10 +294,34 @@ def get_model_options() -> ModelOptions:
def register_args(program : ArgumentParser) -> None: def register_args(program : ArgumentParser) -> None:
group_processors = find_argument_group(program, 'processors') group_processors = find_argument_group(program, 'processors')
if group_processors: if group_processors:
group_processors.add_argument('--face-enhancer-model', help = translator.get('help.model', __package__), default = config.get_str_value('processors', 'face_enhancer_model', 'gfpgan_1.4'), choices = face_enhancer_choices.face_enhancer_models) facefusion.capability_store.register_capability_set(
group_processors.add_argument('--face-enhancer-blend', help = translator.get('help.blend', __package__), type = int, default = config.get_int_value('processors', 'face_enhancer_blend', '80'), choices = face_enhancer_choices.face_enhancer_blend_range, metavar = create_int_metavar(face_enhancer_choices.face_enhancer_blend_range)) [
group_processors.add_argument('--face-enhancer-weight', help = translator.get('help.weight', __package__), type = float, default = config.get_float_value('processors', 'face_enhancer_weight', '0.5'), choices = face_enhancer_choices.face_enhancer_weight_range, metavar = create_float_metavar(face_enhancer_choices.face_enhancer_weight_range)) group_processors.add_argument(
facefusion.jobs.job_store.register_step_keys([ 'face_enhancer_model', 'face_enhancer_blend', 'face_enhancer_weight' ]) '--face-enhancer-model',
help = translator.get('help.model', __package__),
default = config.get_str_value('processors', 'face_enhancer_model', 'gfpgan_1.4'),
choices = face_enhancer_choices.face_enhancer_models
),
group_processors.add_argument(
'--face-enhancer-blend',
help = translator.get('help.blend', __package__),
type = int,
default = config.get_int_value('processors', 'face_enhancer_blend', '80'),
choices = face_enhancer_choices.face_enhancer_blend_range,
metavar = create_int_metavar(face_enhancer_choices.face_enhancer_blend_range)
),
group_processors.add_argument(
'--face-enhancer-weight',
help = translator.get('help.weight', __package__),
type = float,
default = config.get_float_value('processors', 'face_enhancer_weight', '0.5'),
choices = face_enhancer_choices.face_enhancer_weight_range,
metavar = create_float_metavar(face_enhancer_choices.face_enhancer_weight_range)
)
],
scopes = [ 'api', 'cli' ],
groups = [ 'face_enhancer' ]
)
def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None: def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
@@ -304,10 +330,18 @@ def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('face_enhancer_weight', args.get('face_enhancer_weight')) apply_state_item('face_enhancer_weight', args.get('face_enhancer_weight'))
def get_common_modules() -> List[ModuleType]:
return [ content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer ]
def pre_check() -> bool: def pre_check() -> bool:
model_hash_set = get_model_options().get('hashes') model_hash_set = get_model_options().get('hashes')
model_source_set = get_model_options().get('sources') model_source_set = get_model_options().get('sources')
for common_module in get_common_modules():
if not common_module.pre_check():
return False
return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set) return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set)
@@ -315,28 +349,25 @@ def pre_process(mode : ProcessMode) -> bool:
if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')): if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')):
logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__)
return False return False
if state_manager.get_item('workflow') in [ 'audio-to-image:video', 'image-to-image', 'image-to-video' ]:
if mode == 'output' and not in_directory(state_manager.get_item('output_path')): if mode == 'output' and not in_directory(state_manager.get_item('output_path')):
logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__)
return False return False
if mode == 'output' and not same_file_extension(state_manager.get_item('target_path'), state_manager.get_item('output_path')):
logger.error(translator.get('match_target_and_output_extension') + translator.get('exclamation_mark'), __name__)
return False
return True return True
def post_process() -> None: def post_process() -> None:
read_static_image.cache_clear() read_static_image.cache_clear()
read_static_video_frame.cache_clear() read_static_video_frame.cache_clear()
read_static_video_chunk.cache_clear()
video_manager.clear_video_pool() video_manager.clear_video_pool()
if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]: if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]:
clear_inference_pool() clear_inference_pool()
if state_manager.get_item('video_memory_strategy') == 'strict': if state_manager.get_item('video_memory_strategy') == 'strict':
content_analyser.clear_inference_pool() for common_module in get_common_modules():
face_classifier.clear_inference_pool() common_module.clear_inference_pool()
face_detector.clear_inference_pool()
face_landmarker.clear_inference_pool()
face_masker.clear_inference_pool()
face_recognizer.clear_inference_pool()
def enhance_face(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame: def enhance_face(target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame:
@@ -413,10 +444,13 @@ def blend_paste_frame(temp_vision_frame : VisionFrame, paste_vision_frame : Visi
def process_frame(inputs : FaceEnhancerInputs) -> ProcessorOutputs: def process_frame(inputs : FaceEnhancerInputs) -> ProcessorOutputs:
reference_vision_frame = inputs.get('reference_vision_frame') reference_vision_frame = inputs.get('reference_vision_frame')
target_vision_frame = inputs.get('target_vision_frame') source_vision_frames = inputs.get('source_vision_frames')
target_vision_frames = inputs.get('target_vision_frames')
temp_vision_frame = inputs.get('temp_vision_frame') temp_vision_frame = inputs.get('temp_vision_frame')
temp_vision_mask = inputs.get('temp_vision_mask') temp_vision_mask = inputs.get('temp_vision_mask')
target_faces = select_faces(reference_vision_frame, target_vision_frame)
target_vision_frame = get_middle(target_vision_frames)
target_faces = select_faces(reference_vision_frame, source_vision_frames, target_vision_frames)
if target_faces: if target_faces:
for target_face in target_faces: for target_face in target_faces:
@@ -1,4 +1,4 @@
from typing import Any, Literal, TypeAlias, TypedDict from typing import Any, List, Literal, TypeAlias, TypedDict
from numpy.typing import NDArray from numpy.typing import NDArray
@@ -7,7 +7,8 @@ from facefusion.types import Mask, VisionFrame
FaceEnhancerInputs = TypedDict('FaceEnhancerInputs', FaceEnhancerInputs = TypedDict('FaceEnhancerInputs',
{ {
'reference_vision_frame' : VisionFrame, 'reference_vision_frame' : VisionFrame,
'target_vision_frame' : VisionFrame, 'source_vision_frames' : List[VisionFrame],
'target_vision_frames' : List[VisionFrame],
'temp_vision_frame' : VisionFrame, 'temp_vision_frame' : VisionFrame,
'temp_vision_mask' : Mask 'temp_vision_mask' : Mask
}) })
@@ -1,31 +1,32 @@
from argparse import ArgumentParser from argparse import ArgumentParser
from functools import lru_cache from functools import lru_cache
from types import ModuleType
from typing import List, Optional, Tuple from typing import List, Optional, Tuple
import cv2 import cv2
import numpy import numpy
import facefusion.capability_store
import facefusion.choices import facefusion.choices
import facefusion.jobs.job_manager import facefusion.jobs.job_manager
import facefusion.jobs.job_store
from facefusion import config, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, inference_manager, logger, state_manager, translator, video_manager from facefusion import config, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, inference_manager, logger, state_manager, translator, video_manager
from facefusion.common_helper import get_first, is_macos from facefusion.common_helper import get_first, get_middle, is_macos
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url
from facefusion.execution import has_execution_provider from facefusion.execution import has_execution_provider
from facefusion.face_analyser import get_average_face, get_many_faces, get_one_face, scale_face from facefusion.face_creator import average_face_identity, get_one_face, get_static_faces, scale_face
from facefusion.face_helper import paste_back, warp_face_by_face_landmark_5 from facefusion.face_helper import paste_back, warp_face_by_face_landmark_5
from facefusion.face_masker import create_area_mask, create_box_mask, create_occlusion_mask, create_region_mask from facefusion.face_masker import create_area_mask, create_box_mask, create_occlusion_mask, create_region_mask
from facefusion.face_selector import select_faces, sort_faces_by_order from facefusion.face_selector import select_faces, sort_faces_by_order
from facefusion.filesystem import filter_image_paths, has_image, in_directory, is_image, is_video, resolve_relative_path, same_file_extension from facefusion.filesystem import filter_image_paths, has_image, in_directory, is_image, is_video, resolve_relative_path
from facefusion.model_helper import get_static_model_initializer from facefusion.model_helper import get_static_model_initializer
from facefusion.processors.modules.face_swapper import choices as face_swapper_choices from facefusion.processors.modules.face_swapper import choices as face_swapper_choices
from facefusion.processors.modules.face_swapper.types import FaceSwapperInputs from facefusion.processors.modules.face_swapper.types import FaceSwapperInputs
from facefusion.processors.pixel_boost import explode_pixel_boost, implode_pixel_boost from facefusion.processors.pixel_boost import explode_pixel_boost, implode_pixel_boost
from facefusion.processors.types import ProcessorOutputs from facefusion.processors.types import ApplyStateItem, ProcessorOutputs
from facefusion.program_helper import find_argument_group from facefusion.program_helper import find_argument_group
from facefusion.thread_helper import conditional_thread_semaphore from facefusion.thread_helper import conditional_thread_semaphore
from facefusion.types import ApplyStateItem, Args, DownloadScope, Embedding, Face, InferencePool, ModelOptions, ModelSet, ProcessMode, VisionFrame from facefusion.types import Args, DownloadScope, Embedding, Face, InferencePool, InferenceProvider, ModelOptions, ModelSet, ProcessMode, VisionFrame
from facefusion.vision import read_static_image, read_static_images, read_static_video_frame, unpack_resolution from facefusion.vision import read_static_image, read_static_images, read_static_video_chunk, read_static_video_frame, unpack_resolution
@lru_cache() @lru_cache()
@@ -246,6 +247,7 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
'path': resolve_relative_path('../.assets/models/hyperswap_1a_256.onnx') 'path': resolve_relative_path('../.assets/models/hyperswap_1a_256.onnx')
} }
}, },
'precision': 'fp16',
'type': 'hyperswap', 'type': 'hyperswap',
'template': 'arcface_128', 'template': 'arcface_128',
'size': (256, 256), 'size': (256, 256),
@@ -276,6 +278,7 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
'path': resolve_relative_path('../.assets/models/hyperswap_1b_256.onnx') 'path': resolve_relative_path('../.assets/models/hyperswap_1b_256.onnx')
} }
}, },
'precision': 'fp16',
'type': 'hyperswap', 'type': 'hyperswap',
'template': 'arcface_128', 'template': 'arcface_128',
'size': (256, 256), 'size': (256, 256),
@@ -306,6 +309,7 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
'path': resolve_relative_path('../.assets/models/hyperswap_1c_256.onnx') 'path': resolve_relative_path('../.assets/models/hyperswap_1c_256.onnx')
} }
}, },
'precision': 'fp16',
'type': 'hyperswap', 'type': 'hyperswap',
'template': 'arcface_128', 'template': 'arcface_128',
'size': (256, 256), 'size': (256, 256),
@@ -366,6 +370,7 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
'path': resolve_relative_path('../.assets/models/inswapper_128_fp16.onnx') 'path': resolve_relative_path('../.assets/models/inswapper_128_fp16.onnx')
} }
}, },
'precision': 'fp16',
'type': 'inswapper', 'type': 'inswapper',
'template': 'arcface_128', 'template': 'arcface_128',
'size': (128, 128), 'size': (128, 128),
@@ -486,39 +491,76 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
def get_inference_pool() -> InferencePool: def get_inference_pool() -> InferencePool:
model_names = [ get_model_name() ] model_names = [ state_manager.get_item('face_swapper_model') ]
model_source_set = get_model_options().get('sources') model_source_set = get_model_options().get('sources')
return inference_manager.get_inference_pool(__name__, model_names, model_source_set) return inference_manager.get_inference_pool(__name__, model_names, model_source_set)
def clear_inference_pool() -> None: def clear_inference_pool() -> None:
model_names = [ get_model_name() ] model_names = [ state_manager.get_item('face_swapper_model') ]
inference_manager.clear_inference_pool(__name__, model_names) inference_manager.clear_inference_pool(__name__, model_names)
def resolve_inference_providers() -> List[InferenceProvider]:
model_precision = get_model_options().get('precision')
model_type = get_model_options().get('type')
if is_macos() and has_execution_provider('coreml'):
if model_type in [ 'ghost', 'uniface' ] or model_precision == 'fp16':
return\
[
(facefusion.choices.execution_provider_set.get('coreml'),
{
'ModelFormat': 'MLProgram',
'SpecializationStrategy': 'FastPrediction'
})
]
return []
def get_model_options() -> ModelOptions: def get_model_options() -> ModelOptions:
model_name = get_model_name()
return create_static_model_set('full').get(model_name)
def get_model_name() -> str:
model_name = state_manager.get_item('face_swapper_model') model_name = state_manager.get_item('face_swapper_model')
return create_static_model_set('full').get(model_name)
if is_macos() and has_execution_provider('coreml') and model_name == 'inswapper_128_fp16':
return 'inswapper_128'
return model_name
def register_args(program : ArgumentParser) -> None: def register_args(program : ArgumentParser) -> None:
group_processors = find_argument_group(program, 'processors') group_processors = find_argument_group(program, 'processors')
if group_processors: if group_processors:
group_processors.add_argument('--face-swapper-model', help = translator.get('help.model', __package__), default = config.get_str_value('processors', 'face_swapper_model', 'hyperswap_1a_256'), choices = face_swapper_choices.face_swapper_models) facefusion.capability_store.register_capability_set(
[
group_processors.add_argument(
'--face-swapper-model',
help = translator.get('help.model', __package__),
default = config.get_str_value('processors', 'face_swapper_model', 'hyperswap_1a_256'),
choices = face_swapper_choices.face_swapper_models
)
],
scopes = [ 'api', 'cli' ],
groups = [ 'face_swapper' ]
)
known_args, _ = program.parse_known_args() known_args, _ = program.parse_known_args()
face_swapper_pixel_boost_choices = face_swapper_choices.face_swapper_set.get(known_args.face_swapper_model) face_swapper_pixel_boost_choices = face_swapper_choices.face_swapper_set.get(known_args.face_swapper_model)
group_processors.add_argument('--face-swapper-pixel-boost', help = translator.get('help.pixel_boost', __package__), default = config.get_str_value('processors', 'face_swapper_pixel_boost', get_first(face_swapper_pixel_boost_choices)), choices = face_swapper_pixel_boost_choices) facefusion.capability_store.register_capability_set(
group_processors.add_argument('--face-swapper-weight', help = translator.get('help.weight', __package__), type = float, default = config.get_float_value('processors', 'face_swapper_weight', '0.5'), choices = face_swapper_choices.face_swapper_weight_range) [
facefusion.jobs.job_store.register_step_keys([ 'face_swapper_model', 'face_swapper_pixel_boost', 'face_swapper_weight' ]) group_processors.add_argument(
'--face-swapper-pixel-boost',
help = translator.get('help.pixel_boost', __package__),
default = config.get_str_value('processors', 'face_swapper_pixel_boost', get_first(face_swapper_pixel_boost_choices)),
choices = face_swapper_pixel_boost_choices
),
group_processors.add_argument(
'--face-swapper-weight',
help = translator.get('help.weight', __package__),
type = float,
default = config.get_float_value('processors', 'face_swapper_weight', '0.5'),
choices = face_swapper_choices.face_swapper_weight_range
)
],
scopes = [ 'api', 'cli' ],
groups = [ 'face_swapper' ]
)
def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None: def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
@@ -527,10 +569,18 @@ def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('face_swapper_weight', args.get('face_swapper_weight')) apply_state_item('face_swapper_weight', args.get('face_swapper_weight'))
def get_common_modules() -> List[ModuleType]:
return [ content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer ]
def pre_check() -> bool: def pre_check() -> bool:
model_hash_set = get_model_options().get('hashes') model_hash_set = get_model_options().get('hashes')
model_source_set = get_model_options().get('sources') model_source_set = get_model_options().get('sources')
for common_module in get_common_modules():
if not common_module.pre_check():
return False
return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set) return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set)
@@ -541,7 +591,7 @@ def pre_process(mode : ProcessMode) -> bool:
source_image_paths = filter_image_paths(state_manager.get_item('source_paths')) source_image_paths = filter_image_paths(state_manager.get_item('source_paths'))
source_vision_frames = read_static_images(source_image_paths) source_vision_frames = read_static_images(source_image_paths)
source_faces = get_many_faces(source_vision_frames) source_faces = get_static_faces(source_vision_frames)
if not get_one_face(source_faces): if not get_one_face(source_faces):
logger.error(translator.get('no_source_face_detected') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('no_source_face_detected') + translator.get('exclamation_mark'), __name__)
@@ -551,34 +601,30 @@ def pre_process(mode : ProcessMode) -> bool:
logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__)
return False return False
if state_manager.get_item('workflow') in [ 'audio-to-image:video', 'image-to-image', 'image-to-video' ]:
if mode == 'output' and not in_directory(state_manager.get_item('output_path')): if mode == 'output' and not in_directory(state_manager.get_item('output_path')):
logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__)
return False return False
if mode == 'output' and not same_file_extension(state_manager.get_item('target_path'), state_manager.get_item('output_path')):
logger.error(translator.get('match_target_and_output_extension') + translator.get('exclamation_mark'), __name__)
return False
return True return True
def post_process() -> None: def post_process() -> None:
read_static_image.cache_clear() read_static_image.cache_clear()
read_static_video_frame.cache_clear() read_static_video_frame.cache_clear()
read_static_video_chunk.cache_clear()
video_manager.clear_video_pool() video_manager.clear_video_pool()
if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]: if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]:
get_static_model_initializer.cache_clear() get_static_model_initializer.cache_clear()
clear_inference_pool() clear_inference_pool()
if state_manager.get_item('video_memory_strategy') == 'strict': if state_manager.get_item('video_memory_strategy') == 'strict':
content_analyser.clear_inference_pool() for common_module in get_common_modules():
face_classifier.clear_inference_pool() common_module.clear_inference_pool()
face_detector.clear_inference_pool()
face_landmarker.clear_inference_pool()
face_masker.clear_inference_pool()
face_recognizer.clear_inference_pool()
def swap_face(source_face : Face, target_face : Face, temp_vision_frame : VisionFrame) -> VisionFrame: def swap_face(source_face : Face, target_face : Face, source_vision_frame : VisionFrame, temp_vision_frame : VisionFrame) -> VisionFrame:
model_template = get_model_options().get('template') model_template = get_model_options().get('template')
model_size = get_model_options().get('size') model_size = get_model_options().get('size')
pixel_boost_size = unpack_resolution(state_manager.get_item('face_swapper_pixel_boost')) pixel_boost_size = unpack_resolution(state_manager.get_item('face_swapper_pixel_boost'))
@@ -598,7 +644,7 @@ def swap_face(source_face : Face, target_face : Face, temp_vision_frame : Vision
pixel_boost_vision_frames = implode_pixel_boost(crop_vision_frame, pixel_boost_total, model_size) pixel_boost_vision_frames = implode_pixel_boost(crop_vision_frame, pixel_boost_total, model_size)
for pixel_boost_vision_frame in pixel_boost_vision_frames: for pixel_boost_vision_frame in pixel_boost_vision_frames:
pixel_boost_vision_frame = prepare_crop_frame(pixel_boost_vision_frame) pixel_boost_vision_frame = prepare_crop_frame(pixel_boost_vision_frame)
pixel_boost_vision_frame = forward_swap_face(source_face, target_face, pixel_boost_vision_frame) pixel_boost_vision_frame = forward_swap_face(source_face, target_face, source_vision_frame, pixel_boost_vision_frame)
pixel_boost_vision_frame = normalize_crop_frame(pixel_boost_vision_frame) pixel_boost_vision_frame = normalize_crop_frame(pixel_boost_vision_frame)
temp_vision_frames.append(pixel_boost_vision_frame) temp_vision_frames.append(pixel_boost_vision_frame)
crop_vision_frame = explode_pixel_boost(temp_vision_frames, pixel_boost_total, model_size, pixel_boost_size) crop_vision_frame = explode_pixel_boost(temp_vision_frames, pixel_boost_total, model_size, pixel_boost_size)
@@ -617,18 +663,15 @@ def swap_face(source_face : Face, target_face : Face, temp_vision_frame : Vision
return paste_vision_frame return paste_vision_frame
def forward_swap_face(source_face : Face, target_face : Face, crop_vision_frame : VisionFrame) -> VisionFrame: def forward_swap_face(source_face : Face, target_face : Face, source_vision_frame : VisionFrame, crop_vision_frame : VisionFrame) -> VisionFrame:
face_swapper = get_inference_pool().get('face_swapper') face_swapper = get_inference_pool().get('face_swapper')
model_type = get_model_options().get('type') model_type = get_model_options().get('type')
face_swapper_inputs = {} face_swapper_inputs = {}
if is_macos() and has_execution_provider('coreml') and model_type in [ 'ghost', 'uniface' ]:
face_swapper.set_providers([ facefusion.choices.execution_provider_set.get('cpu') ])
for face_swapper_input in face_swapper.get_inputs(): for face_swapper_input in face_swapper.get_inputs():
if face_swapper_input.name == 'source': if face_swapper_input.name == 'source':
if model_type in [ 'blendswap', 'uniface' ]: if model_type in [ 'blendswap', 'uniface' ]:
face_swapper_inputs[face_swapper_input.name] = prepare_source_frame(source_face) face_swapper_inputs[face_swapper_input.name] = prepare_source_frame(source_face, source_vision_frame)
else: else:
source_embedding = prepare_source_embedding(source_face) source_embedding = prepare_source_embedding(source_face)
source_embedding = balance_source_embedding(source_embedding, target_face.embedding) source_embedding = balance_source_embedding(source_embedding, target_face.embedding)
@@ -654,9 +697,8 @@ def forward_convert_embedding(face_embedding : Embedding) -> Embedding:
return face_embedding return face_embedding
def prepare_source_frame(source_face : Face) -> VisionFrame: def prepare_source_frame(source_face : Face, source_vision_frame : VisionFrame) -> VisionFrame:
model_type = get_model_options().get('type') model_type = get_model_options().get('type')
source_vision_frame = read_static_image(get_first(state_manager.get_item('source_paths')))
if model_type == 'blendswap': if model_type == 'blendswap':
source_vision_frame, _ = warp_face_by_face_landmark_5(source_vision_frame, source_face.landmark_set.get('5/68'), 'arcface_112_v2', (112, 112)) source_vision_frame, _ = warp_face_by_face_landmark_5(source_vision_frame, source_face.landmark_set.get('5/68'), 'arcface_112_v2', (112, 112))
@@ -748,27 +790,31 @@ def extract_source_face(source_vision_frames : List[VisionFrame]) -> Optional[Fa
if source_vision_frames: if source_vision_frames:
for source_vision_frame in source_vision_frames: for source_vision_frame in source_vision_frames:
temp_faces = get_many_faces([source_vision_frame]) temp_faces = get_static_faces([ source_vision_frame ])
temp_faces = sort_faces_by_order(temp_faces, 'large-small') temp_faces = sort_faces_by_order(temp_faces, 'large-small')
if temp_faces: if temp_faces:
source_faces.append(get_first(temp_faces)) source_faces.append(get_first(temp_faces))
return get_average_face(source_faces) return average_face_identity(source_faces)
def process_frame(inputs : FaceSwapperInputs) -> ProcessorOutputs: def process_frame(inputs : FaceSwapperInputs) -> ProcessorOutputs:
reference_vision_frame = inputs.get('reference_vision_frame') reference_vision_frame = inputs.get('reference_vision_frame')
source_vision_frames = inputs.get('source_vision_frames') source_vision_frames = inputs.get('source_vision_frames')
target_vision_frame = inputs.get('target_vision_frame') target_vision_frames = inputs.get('target_vision_frames')
temp_vision_frame = inputs.get('temp_vision_frame') temp_vision_frame = inputs.get('temp_vision_frame')
temp_vision_mask = inputs.get('temp_vision_mask') temp_vision_mask = inputs.get('temp_vision_mask')
target_vision_frame = get_middle(target_vision_frames)
source_face = extract_source_face(source_vision_frames) source_face = extract_source_face(source_vision_frames)
target_faces = select_faces(reference_vision_frame, target_vision_frame) target_faces = select_faces(reference_vision_frame, source_vision_frames, target_vision_frames)
if source_face and target_faces: if source_face and target_faces:
source_vision_frame = get_first(source_vision_frames)
for target_face in target_faces: for target_face in target_faces:
target_face = scale_face(target_face, target_vision_frame, temp_vision_frame) target_face = scale_face(target_face, target_vision_frame, temp_vision_frame)
temp_vision_frame = swap_face(source_face, target_face, temp_vision_frame) temp_vision_frame = swap_face(source_face, target_face, source_vision_frame, temp_vision_frame)
return temp_vision_frame, temp_vision_mask return temp_vision_frame, temp_vision_mask
@@ -6,7 +6,7 @@ FaceSwapperInputs = TypedDict('FaceSwapperInputs',
{ {
'reference_vision_frame' : VisionFrame, 'reference_vision_frame' : VisionFrame,
'source_vision_frames' : List[VisionFrame], 'source_vision_frames' : List[VisionFrame],
'target_vision_frame' : VisionFrame, 'target_vision_frames' : List[VisionFrame],
'temp_vision_frame' : VisionFrame, 'temp_vision_frame' : VisionFrame,
'temp_vision_mask' : Mask 'temp_vision_mask' : Mask
}) })
@@ -1,24 +1,26 @@
from argparse import ArgumentParser from argparse import ArgumentParser
from functools import lru_cache from functools import lru_cache
from types import ModuleType
from typing import List from typing import List
import cv2 import cv2
import numpy import numpy
import facefusion.capability_store
import facefusion.choices
import facefusion.jobs.job_manager import facefusion.jobs.job_manager
import facefusion.jobs.job_store
from facefusion import config, content_analyser, inference_manager, logger, state_manager, translator, video_manager from facefusion import config, content_analyser, inference_manager, logger, state_manager, translator, video_manager
from facefusion.common_helper import create_int_metavar, is_macos from facefusion.common_helper import create_int_metavar, is_macos
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url
from facefusion.execution import has_execution_provider from facefusion.execution import has_execution_provider
from facefusion.filesystem import in_directory, is_image, is_video, resolve_relative_path, same_file_extension from facefusion.filesystem import in_directory, is_image, is_video, resolve_relative_path
from facefusion.processors.modules.frame_colorizer import choices as frame_colorizer_choices from facefusion.processors.modules.frame_colorizer import choices as frame_colorizer_choices
from facefusion.processors.modules.frame_colorizer.types import FrameColorizerInputs from facefusion.processors.modules.frame_colorizer.types import FrameColorizerInputs
from facefusion.processors.types import ProcessorOutputs from facefusion.processors.types import ApplyStateItem, ProcessorOutputs
from facefusion.program_helper import find_argument_group from facefusion.program_helper import find_argument_group
from facefusion.thread_helper import thread_semaphore from facefusion.thread_helper import thread_semaphore
from facefusion.types import ApplyStateItem, Args, DownloadScope, ExecutionProvider, InferencePool, ModelOptions, ModelSet, ProcessMode, VisionFrame from facefusion.types import Args, DownloadScope, InferencePool, InferenceProvider, ModelOptions, ModelSet, ProcessMode, VisionFrame
from facefusion.vision import blend_frame, read_static_image, read_static_video_frame, unpack_resolution from facefusion.vision import blend_frame, read_static_image, read_static_video_chunk, read_static_video_frame, unpack_resolution
@lru_cache() @lru_cache()
@@ -170,10 +172,11 @@ def clear_inference_pool() -> None:
inference_manager.clear_inference_pool(__name__, model_names) inference_manager.clear_inference_pool(__name__, model_names)
def resolve_execution_providers() -> List[ExecutionProvider]: def resolve_inference_providers() -> List[InferenceProvider]:
if is_macos() and has_execution_provider('coreml'): if is_macos() and has_execution_provider('coreml'):
return [ 'cpu' ] return [ facefusion.choices.execution_provider_set.get('cpu') ]
return state_manager.get_item('execution_providers')
return []
def get_model_options() -> ModelOptions: def get_model_options() -> ModelOptions:
@@ -184,10 +187,33 @@ def get_model_options() -> ModelOptions:
def register_args(program : ArgumentParser) -> None: def register_args(program : ArgumentParser) -> None:
group_processors = find_argument_group(program, 'processors') group_processors = find_argument_group(program, 'processors')
if group_processors: if group_processors:
group_processors.add_argument('--frame-colorizer-model', help = translator.get('help.model', __package__), default = config.get_str_value('processors', 'frame_colorizer_model', 'ddcolor'), choices = frame_colorizer_choices.frame_colorizer_models) facefusion.capability_store.register_capability_set(
group_processors.add_argument('--frame-colorizer-size', help = translator.get('help.size', __package__), type = str, default = config.get_str_value('processors', 'frame_colorizer_size', '256x256'), choices = frame_colorizer_choices.frame_colorizer_sizes) [
group_processors.add_argument('--frame-colorizer-blend', help = translator.get('help.blend', __package__), type = int, default = config.get_int_value('processors', 'frame_colorizer_blend', '100'), choices = frame_colorizer_choices.frame_colorizer_blend_range, metavar = create_int_metavar(frame_colorizer_choices.frame_colorizer_blend_range)) group_processors.add_argument(
facefusion.jobs.job_store.register_step_keys([ 'frame_colorizer_model', 'frame_colorizer_blend', 'frame_colorizer_size' ]) '--frame-colorizer-model',
help = translator.get('help.model', __package__),
default = config.get_str_value('processors', 'frame_colorizer_model', 'ddcolor'),
choices = frame_colorizer_choices.frame_colorizer_models
),
group_processors.add_argument(
'--frame-colorizer-size',
help = translator.get('help.size', __package__),
type = str,
default = config.get_str_value('processors', 'frame_colorizer_size', '256x256'),
choices = frame_colorizer_choices.frame_colorizer_sizes
),
group_processors.add_argument(
'--frame-colorizer-blend',
help = translator.get('help.blend', __package__),
type = int,
default = config.get_int_value('processors', 'frame_colorizer_blend', '100'),
choices = frame_colorizer_choices.frame_colorizer_blend_range,
metavar = create_int_metavar(frame_colorizer_choices.frame_colorizer_blend_range)
)
],
scopes = [ 'api', 'cli' ],
groups = [ 'frame_colorizer' ]
)
def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None: def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
@@ -196,10 +222,18 @@ def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('frame_colorizer_size', args.get('frame_colorizer_size')) apply_state_item('frame_colorizer_size', args.get('frame_colorizer_size'))
def get_common_modules() -> List[ModuleType]:
return [ content_analyser ]
def pre_check() -> bool: def pre_check() -> bool:
model_hash_set = get_model_options().get('hashes') model_hash_set = get_model_options().get('hashes')
model_source_set = get_model_options().get('sources') model_source_set = get_model_options().get('sources')
for common_module in get_common_modules():
if not common_module.pre_check():
return False
return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set) return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set)
@@ -207,23 +241,25 @@ def pre_process(mode : ProcessMode) -> bool:
if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')): if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')):
logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__)
return False return False
if state_manager.get_item('workflow') in [ 'audio-to-image:video', 'image-to-image', 'image-to-video' ]:
if mode == 'output' and not in_directory(state_manager.get_item('output_path')): if mode == 'output' and not in_directory(state_manager.get_item('output_path')):
logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__)
return False return False
if mode == 'output' and not same_file_extension(state_manager.get_item('target_path'), state_manager.get_item('output_path')):
logger.error(translator.get('match_target_and_output_extension') + translator.get('exclamation_mark'), __name__)
return False
return True return True
def post_process() -> None: def post_process() -> None:
read_static_image.cache_clear() read_static_image.cache_clear()
read_static_video_frame.cache_clear() read_static_video_frame.cache_clear()
read_static_video_chunk.cache_clear()
video_manager.clear_video_pool() video_manager.clear_video_pool()
if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]: if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]:
clear_inference_pool() clear_inference_pool()
if state_manager.get_item('video_memory_strategy') == 'strict': if state_manager.get_item('video_memory_strategy') == 'strict':
content_analyser.clear_inference_pool() for common_module in get_common_modules():
common_module.clear_inference_pool()
def colorize_frame(temp_vision_frame : VisionFrame) -> VisionFrame: def colorize_frame(temp_vision_frame : VisionFrame) -> VisionFrame:
@@ -1,10 +1,10 @@
from typing import Literal, TypedDict from typing import List, Literal, TypedDict
from facefusion.types import Mask, VisionFrame from facefusion.types import Mask, VisionFrame
FrameColorizerInputs = TypedDict('FrameColorizerInputs', FrameColorizerInputs = TypedDict('FrameColorizerInputs',
{ {
'target_vision_frame' : VisionFrame, 'target_vision_frames' : List[VisionFrame],
'temp_vision_frame' : VisionFrame, 'temp_vision_frame' : VisionFrame,
'temp_vision_mask' : Mask 'temp_vision_mask' : Mask
}) })
@@ -1,23 +1,26 @@
from argparse import ArgumentParser from argparse import ArgumentParser
from functools import lru_cache from functools import lru_cache
from types import ModuleType
from typing import List
import cv2 import cv2
import numpy import numpy
import facefusion.capability_store
import facefusion.choices
import facefusion.jobs.job_manager import facefusion.jobs.job_manager
import facefusion.jobs.job_store
from facefusion import config, content_analyser, inference_manager, logger, state_manager, translator, video_manager from facefusion import config, content_analyser, inference_manager, logger, state_manager, translator, video_manager
from facefusion.common_helper import create_int_metavar, is_macos from facefusion.common_helper import create_int_metavar, is_macos
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url
from facefusion.execution import has_execution_provider from facefusion.execution import has_execution_provider
from facefusion.filesystem import in_directory, is_image, is_video, resolve_relative_path, same_file_extension from facefusion.filesystem import in_directory, is_image, is_video, resolve_relative_path
from facefusion.processors.modules.frame_enhancer import choices as frame_enhancer_choices from facefusion.processors.modules.frame_enhancer import choices as frame_enhancer_choices
from facefusion.processors.modules.frame_enhancer.types import FrameEnhancerInputs from facefusion.processors.modules.frame_enhancer.types import FrameEnhancerInputs
from facefusion.processors.types import ProcessorOutputs from facefusion.processors.types import ApplyStateItem, ProcessorOutputs
from facefusion.program_helper import find_argument_group from facefusion.program_helper import find_argument_group
from facefusion.thread_helper import conditional_thread_semaphore from facefusion.thread_helper import conditional_thread_semaphore
from facefusion.types import ApplyStateItem, Args, DownloadScope, InferencePool, ModelOptions, ModelSet, ProcessMode, VisionFrame from facefusion.types import Args, DownloadScope, InferencePool, InferenceProvider, ModelOptions, ModelSet, ProcessMode, VisionFrame
from facefusion.vision import blend_frame, create_tile_frames, merge_tile_frames, read_static_image, read_static_video_frame from facefusion.vision import blend_frame, create_tile_frames, merge_tile_frames, read_static_image, read_static_video_chunk, read_static_video_frame
@lru_cache() @lru_cache()
@@ -156,6 +159,7 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
'path': resolve_relative_path('../.assets/models/real_esrgan_x2_fp16.onnx') 'path': resolve_relative_path('../.assets/models/real_esrgan_x2_fp16.onnx')
} }
}, },
'precision': 'fp16',
'size': (256, 16, 8), 'size': (256, 16, 8),
'scale': 2 'scale': 2
}, },
@@ -210,6 +214,7 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
'path': resolve_relative_path('../.assets/models/real_esrgan_x4_fp16.onnx') 'path': resolve_relative_path('../.assets/models/real_esrgan_x4_fp16.onnx')
} }
}, },
'precision': 'fp16',
'size': (256, 16, 8), 'size': (256, 16, 8),
'scale': 4 'scale': 4
}, },
@@ -264,6 +269,7 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
'path': resolve_relative_path('../.assets/models/real_esrgan_x8_fp16.onnx') 'path': resolve_relative_path('../.assets/models/real_esrgan_x8_fp16.onnx')
} }
}, },
'precision': 'fp16',
'size': (256, 16, 8), 'size': (256, 16, 8),
'scale': 8 'scale': 8
}, },
@@ -541,41 +547,61 @@ def create_static_model_set(download_scope : DownloadScope) -> ModelSet:
def get_inference_pool() -> InferencePool: def get_inference_pool() -> InferencePool:
model_names = [ get_frame_enhancer_model() ] model_names = [ state_manager.get_item('frame_enhancer_model') ]
model_source_set = get_model_options().get('sources') model_source_set = get_model_options().get('sources')
return inference_manager.get_inference_pool(__name__, model_names, model_source_set) return inference_manager.get_inference_pool(__name__, model_names, model_source_set)
def clear_inference_pool() -> None: def clear_inference_pool() -> None:
model_names = [ get_frame_enhancer_model() ] model_names = [ state_manager.get_item('frame_enhancer_model') ]
inference_manager.clear_inference_pool(__name__, model_names) inference_manager.clear_inference_pool(__name__, model_names)
def resolve_inference_providers() -> List[InferenceProvider]:
model_precision = get_model_options().get('precision')
if is_macos() and has_execution_provider('coreml') and model_precision == 'fp16':
return\
[
(facefusion.choices.execution_provider_set.get('coreml'),
{
'ModelFormat': 'MLProgram',
'SpecializationStrategy': 'FastPrediction'
})
]
return []
def get_model_options() -> ModelOptions: def get_model_options() -> ModelOptions:
model_name = get_frame_enhancer_model() model_name = state_manager.get_item('frame_enhancer_model')
return create_static_model_set('full').get(model_name) return create_static_model_set('full').get(model_name)
def get_frame_enhancer_model() -> str:
frame_enhancer_model = state_manager.get_item('frame_enhancer_model')
if is_macos() and has_execution_provider('coreml'):
if frame_enhancer_model == 'real_esrgan_x2_fp16':
return 'real_esrgan_x2'
if frame_enhancer_model == 'real_esrgan_x4_fp16':
return 'real_esrgan_x4'
if frame_enhancer_model == 'real_esrgan_x8_fp16':
return 'real_esrgan_x8'
return frame_enhancer_model
def register_args(program : ArgumentParser) -> None: def register_args(program : ArgumentParser) -> None:
group_processors = find_argument_group(program, 'processors') group_processors = find_argument_group(program, 'processors')
if group_processors: if group_processors:
group_processors.add_argument('--frame-enhancer-model', help = translator.get('help.model', __package__), default = config.get_str_value('processors', 'frame_enhancer_model', 'span_kendata_x4'), choices = frame_enhancer_choices.frame_enhancer_models) facefusion.capability_store.register_capability_set(
group_processors.add_argument('--frame-enhancer-blend', help = translator.get('help.blend', __package__), type = int, default = config.get_int_value('processors', 'frame_enhancer_blend', '80'), choices = frame_enhancer_choices.frame_enhancer_blend_range, metavar = create_int_metavar(frame_enhancer_choices.frame_enhancer_blend_range)) [
facefusion.jobs.job_store.register_step_keys([ 'frame_enhancer_model', 'frame_enhancer_blend' ]) group_processors.add_argument(
'--frame-enhancer-model',
help = translator.get('help.model', __package__),
default = config.get_str_value('processors', 'frame_enhancer_model', 'span_kendata_x4'),
choices = frame_enhancer_choices.frame_enhancer_models
),
group_processors.add_argument(
'--frame-enhancer-blend',
help = translator.get('help.blend', __package__),
type = int,
default = config.get_int_value('processors', 'frame_enhancer_blend', '80'),
choices = frame_enhancer_choices.frame_enhancer_blend_range,
metavar = create_int_metavar(frame_enhancer_choices.frame_enhancer_blend_range)
)
],
scopes = [ 'api', 'cli' ],
groups = [ 'frame_enhancer' ]
)
def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None: def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
@@ -583,10 +609,18 @@ def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('frame_enhancer_blend', args.get('frame_enhancer_blend')) apply_state_item('frame_enhancer_blend', args.get('frame_enhancer_blend'))
def get_common_modules() -> List[ModuleType]:
return [ content_analyser ]
def pre_check() -> bool: def pre_check() -> bool:
model_hash_set = get_model_options().get('hashes') model_hash_set = get_model_options().get('hashes')
model_source_set = get_model_options().get('sources') model_source_set = get_model_options().get('sources')
for common_module in get_common_modules():
if not common_module.pre_check():
return False
return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set) return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set)
@@ -594,23 +628,25 @@ def pre_process(mode : ProcessMode) -> bool:
if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')): if mode in [ 'output', 'preview' ] and not is_image(state_manager.get_item('target_path')) and not is_video(state_manager.get_item('target_path')):
logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('choose_image_or_video_target') + translator.get('exclamation_mark'), __name__)
return False return False
if state_manager.get_item('workflow') in [ 'audio-to-image:video', 'image-to-image', 'image-to-video' ]:
if mode == 'output' and not in_directory(state_manager.get_item('output_path')): if mode == 'output' and not in_directory(state_manager.get_item('output_path')):
logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__) logger.error(translator.get('specify_image_or_video_output') + translator.get('exclamation_mark'), __name__)
return False return False
if mode == 'output' and not same_file_extension(state_manager.get_item('target_path'), state_manager.get_item('output_path')):
logger.error(translator.get('match_target_and_output_extension') + translator.get('exclamation_mark'), __name__)
return False
return True return True
def post_process() -> None: def post_process() -> None:
read_static_image.cache_clear() read_static_image.cache_clear()
read_static_video_frame.cache_clear() read_static_video_frame.cache_clear()
read_static_video_chunk.cache_clear()
video_manager.clear_video_pool() video_manager.clear_video_pool()
if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]: if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]:
clear_inference_pool() clear_inference_pool()
if state_manager.get_item('video_memory_strategy') == 'strict': if state_manager.get_item('video_memory_strategy') == 'strict':
content_analyser.clear_inference_pool() for common_module in get_common_modules():
common_module.clear_inference_pool()
def enhance_frame(temp_vision_frame : VisionFrame) -> VisionFrame: def enhance_frame(temp_vision_frame : VisionFrame) -> VisionFrame:
@@ -1,10 +1,10 @@
from typing import Literal, TypedDict from typing import List, Literal, TypedDict
from facefusion.types import Mask, VisionFrame from facefusion.types import Mask, VisionFrame
FrameEnhancerInputs = TypedDict('FrameEnhancerInputs', FrameEnhancerInputs = TypedDict('FrameEnhancerInputs',
{ {
'target_vision_frame' : VisionFrame, 'target_vision_frames' : List[VisionFrame],
'temp_vision_frame' : VisionFrame, 'temp_vision_frame' : VisionFrame,
'temp_vision_mask' : Mask 'temp_vision_mask' : Mask
}) })
@@ -1,27 +1,29 @@
from argparse import ArgumentParser from argparse import ArgumentParser
from functools import lru_cache from functools import lru_cache
from types import ModuleType
from typing import List
import cv2 import cv2
import numpy import numpy
import facefusion.capability_store
import facefusion.jobs.job_manager import facefusion.jobs.job_manager
import facefusion.jobs.job_store
from facefusion import config, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, inference_manager, logger, state_manager, translator, video_manager, voice_extractor from facefusion import config, content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, inference_manager, logger, state_manager, translator, video_manager, voice_extractor
from facefusion.audio import read_static_voice from facefusion.audio import read_static_voice
from facefusion.common_helper import create_float_metavar from facefusion.common_helper import create_float_metavar, get_middle
from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url from facefusion.download import conditional_download_hashes, conditional_download_sources, resolve_download_url
from facefusion.face_analyser import scale_face from facefusion.face_creator import scale_face
from facefusion.face_helper import create_bounding_box, paste_back, warp_face_by_bounding_box, warp_face_by_face_landmark_5 from facefusion.face_helper import create_bounding_box, paste_back, warp_face_by_bounding_box, warp_face_by_face_landmark_5
from facefusion.face_masker import create_area_mask, create_box_mask, create_occlusion_mask from facefusion.face_masker import create_area_mask, create_box_mask, create_occlusion_mask
from facefusion.face_selector import select_faces from facefusion.face_selector import select_faces
from facefusion.filesystem import has_audio, resolve_relative_path from facefusion.filesystem import has_audio, resolve_relative_path
from facefusion.processors.modules.lip_syncer import choices as lip_syncer_choices from facefusion.processors.modules.lip_syncer import choices as lip_syncer_choices
from facefusion.processors.modules.lip_syncer.types import LipSyncerInputs, LipSyncerWeight from facefusion.processors.modules.lip_syncer.types import LipSyncerInputs, LipSyncerWeight
from facefusion.processors.types import ProcessorOutputs from facefusion.processors.types import ApplyStateItem, ProcessorOutputs
from facefusion.program_helper import find_argument_group from facefusion.program_helper import find_argument_group
from facefusion.thread_helper import conditional_thread_semaphore from facefusion.thread_helper import conditional_thread_semaphore
from facefusion.types import ApplyStateItem, Args, AudioFrame, DownloadScope, Face, InferencePool, ModelOptions, ModelSet, ProcessMode, VisionFrame from facefusion.types import Args, AudioFrame, DownloadScope, Face, InferencePool, ModelOptions, ModelSet, ProcessMode, VisionFrame
from facefusion.vision import read_static_image, read_static_video_frame from facefusion.vision import read_static_image, read_static_video_chunk, read_static_video_frame
@lru_cache() @lru_cache()
@@ -132,9 +134,26 @@ def get_model_options() -> ModelOptions:
def register_args(program : ArgumentParser) -> None: def register_args(program : ArgumentParser) -> None:
group_processors = find_argument_group(program, 'processors') group_processors = find_argument_group(program, 'processors')
if group_processors: if group_processors:
group_processors.add_argument('--lip-syncer-model', help = translator.get('help.model', __package__), default = config.get_str_value('processors', 'lip_syncer_model', 'wav2lip_gan_96'), choices = lip_syncer_choices.lip_syncer_models) facefusion.capability_store.register_capability_set(
group_processors.add_argument('--lip-syncer-weight', help = translator.get('help.weight', __package__), type = float, default = config.get_float_value('processors', 'lip_syncer_weight', '0.5'), choices = lip_syncer_choices.lip_syncer_weight_range, metavar = create_float_metavar(lip_syncer_choices.lip_syncer_weight_range)) [
facefusion.jobs.job_store.register_step_keys([ 'lip_syncer_model', 'lip_syncer_weight' ]) group_processors.add_argument(
'--lip-syncer-model',
help = translator.get('help.model', __package__),
default = config.get_str_value('processors', 'lip_syncer_model', 'wav2lip_gan_96'),
choices = lip_syncer_choices.lip_syncer_models
),
group_processors.add_argument(
'--lip-syncer-weight',
help = translator.get('help.weight', __package__),
type = float,
default = config.get_float_value('processors', 'lip_syncer_weight', '0.5'),
choices = lip_syncer_choices.lip_syncer_weight_range,
metavar = create_float_metavar(lip_syncer_choices.lip_syncer_weight_range)
)
],
scopes = [ 'api', 'cli' ],
groups = [ 'lip_syncer' ]
)
def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None: def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
@@ -142,10 +161,18 @@ def apply_args(args : Args, apply_state_item : ApplyStateItem) -> None:
apply_state_item('lip_syncer_weight', args.get('lip_syncer_weight')) apply_state_item('lip_syncer_weight', args.get('lip_syncer_weight'))
def get_common_modules() -> List[ModuleType]:
return [ content_analyser, face_classifier, face_detector, face_landmarker, face_masker, face_recognizer, voice_extractor ]
def pre_check() -> bool: def pre_check() -> bool:
model_hash_set = get_model_options().get('hashes') model_hash_set = get_model_options().get('hashes')
model_source_set = get_model_options().get('sources') model_source_set = get_model_options().get('sources')
for common_module in get_common_modules():
if not common_module.pre_check():
return False
return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set) return conditional_download_hashes(model_hash_set) and conditional_download_sources(model_source_set)
@@ -159,18 +186,16 @@ def pre_process(mode : ProcessMode) -> bool:
def post_process() -> None: def post_process() -> None:
read_static_image.cache_clear() read_static_image.cache_clear()
read_static_video_frame.cache_clear() read_static_video_frame.cache_clear()
read_static_video_chunk.cache_clear()
read_static_voice.cache_clear() read_static_voice.cache_clear()
video_manager.clear_video_pool() video_manager.clear_video_pool()
if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]: if state_manager.get_item('video_memory_strategy') in [ 'strict', 'moderate' ]:
clear_inference_pool() clear_inference_pool()
if state_manager.get_item('video_memory_strategy') == 'strict': if state_manager.get_item('video_memory_strategy') == 'strict':
content_analyser.clear_inference_pool() for common_module in get_common_modules():
face_classifier.clear_inference_pool() common_module.clear_inference_pool()
face_detector.clear_inference_pool()
face_landmarker.clear_inference_pool()
face_masker.clear_inference_pool()
face_recognizer.clear_inference_pool()
voice_extractor.clear_inference_pool()
def sync_lip(target_face : Face, source_voice_frame : AudioFrame, temp_vision_frame : VisionFrame) -> VisionFrame: def sync_lip(target_face : Face, source_voice_frame : AudioFrame, temp_vision_frame : VisionFrame) -> VisionFrame:
@@ -282,11 +307,14 @@ def normalize_crop_frame(crop_vision_frame : VisionFrame) -> VisionFrame:
def process_frame(inputs : LipSyncerInputs) -> ProcessorOutputs: def process_frame(inputs : LipSyncerInputs) -> ProcessorOutputs:
reference_vision_frame = inputs.get('reference_vision_frame') reference_vision_frame = inputs.get('reference_vision_frame')
source_vision_frames = inputs.get('source_vision_frames')
source_voice_frame = inputs.get('source_voice_frame') source_voice_frame = inputs.get('source_voice_frame')
target_vision_frame = inputs.get('target_vision_frame') target_vision_frames = inputs.get('target_vision_frames')
temp_vision_frame = inputs.get('temp_vision_frame') temp_vision_frame = inputs.get('temp_vision_frame')
temp_vision_mask = inputs.get('temp_vision_mask') temp_vision_mask = inputs.get('temp_vision_mask')
target_faces = select_faces(reference_vision_frame, target_vision_frame)
target_vision_frame = get_middle(target_vision_frames)
target_faces = select_faces(reference_vision_frame, source_vision_frames, target_vision_frames)
if target_faces: if target_faces:
for target_face in target_faces: for target_face in target_faces:
@@ -1,4 +1,4 @@
from typing import Any, Literal, TypeAlias, TypedDict from typing import Any, List, Literal, TypeAlias, TypedDict
from numpy.typing import NDArray from numpy.typing import NDArray
@@ -7,8 +7,9 @@ from facefusion.types import AudioFrame, Mask, VisionFrame
LipSyncerInputs = TypedDict('LipSyncerInputs', LipSyncerInputs = TypedDict('LipSyncerInputs',
{ {
'reference_vision_frame' : VisionFrame, 'reference_vision_frame' : VisionFrame,
'source_vision_frames' : List[VisionFrame],
'source_voice_frame' : AudioFrame, 'source_voice_frame' : AudioFrame,
'target_vision_frame' : VisionFrame, 'target_vision_frames' : List[VisionFrame],
'temp_vision_frame' : VisionFrame, 'temp_vision_frame' : VisionFrame,
'temp_vision_mask' : Mask 'temp_vision_mask' : Mask
}) })
+7 -3
View File
@@ -1,4 +1,4 @@
from typing import Any, Dict, Tuple, TypeAlias from typing import Any, Callable, Dict, Tuple, TypeAlias
from numpy.typing import NDArray from numpy.typing import NDArray
@@ -14,7 +14,11 @@ LivePortraitRotation : TypeAlias = NDArray[Any]
LivePortraitScale : TypeAlias = NDArray[Any] LivePortraitScale : TypeAlias = NDArray[Any]
LivePortraitTranslation : TypeAlias = NDArray[Any] LivePortraitTranslation : TypeAlias = NDArray[Any]
ProcessorStateKey = str ProcessorStateValue : TypeAlias = Any
ProcessorState : TypeAlias = Dict[ProcessorStateKey, Any] ProcessorStateKey : TypeAlias = str
ProcessorState : TypeAlias = Dict[ProcessorStateKey, ProcessorStateValue]
ProcessorStateSet : TypeAlias = Dict[AppContext, ProcessorState] ProcessorStateSet : TypeAlias = Dict[AppContext, ProcessorState]
ApplyStateItem : TypeAlias = Callable[[ProcessorStateKey, ProcessorStateValue], None]
ProcessorOutputs : TypeAlias = Tuple[VisionFrame, Mask] ProcessorOutputs : TypeAlias = Tuple[VisionFrame, Mask]
+1179 -119
View File
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More