CalvinBackup/neuralchen-SimSwap
1
0
Fork 0
Code Issues 288 Pull Requests 16 Actions Packages Projects Releases 2 Wiki Activity

A few requests for image and video face swapping inference #73

New Issue
Open
opened 2021-07-27 11:41:19 +02:00 by dasasf · 5 comments
dasasf commented 2021-07-27 11:41:19 +02:00 (Migrated from github.com)
Copy Link

Hello. First off, let me congratulate you on your great repo, it is amazing what it can already do. However, I ran into a couple of issues, some small, and hopefully easy to fix, and some larger ones.

  1. Right now, swapping a single image creates the output file "result_whole_swapmulti.jpg", which requires renaming it manually and gets annoying quickly.

Would it be possible to have the output file be named as "title_dst_Picture - title_src_picture.jpg" ? In the demo_file case, the output file would be named "iron_man-specific1.jpg" for example.

  1. Additionally, an option to have a whole folder of pictures either as src, dst or both would be nice.

  2. Face detection has trouble with profile shots, quick camera movement and other angles different from portrait shots. Would it be possible to update insightface to a newer version ? Assuming it would help of course.

For multiface video swaps:

  1. When trying to swap multiple faces, the same person person is often not recognized, especially when the distance to the camera changes or in profile shots. You can mitigate it by running the face swap again using new src/dst pictures with the previous output video but it is time consuming and doesn't always work.

Now, if I understand it correctly, Insightface can output what it believes are different faces, as seen here:

https://camo.githubusercontent.com/94afea3fd9149d1859c601e5f396b2d57ed803d8cc1401fcbdc76e669e162943/68747470733a2f2f696e7369676874666163652e61692f6173736574732f696d672f6769746875622f666163657265636f676e6974696f6e66726f6d766964656f2e504e47

Would it be possible to have an option to run a preliminary face recognition over a video/folder and get a src picture for every different face in the video ? Ideally one could then assign one dst face to all src faces the program thinks are different but are actually the same person.

Thank you for your time and, once again, thank you for the amazing work you have already done.

Hello. First off, let me congratulate you on your great repo, it is amazing what it can already do. However, I ran into a couple of issues, some small, and hopefully easy to fix, and some larger ones. 1) Right now, swapping a single image creates the output file "result_whole_swapmulti.jpg", which requires renaming it manually and gets annoying quickly. Would it be possible to have the output file be named as "title_dst_Picture - title_src_picture.jpg" ? In the demo_file case, the output file would be named "iron_man-specific1.jpg" for example. 2) Additionally, an option to have a whole folder of pictures either as src, dst or both would be nice. 3) Face detection has trouble with profile shots, quick camera movement and other angles different from portrait shots. Would it be possible to update insightface to a newer version ? Assuming it would help of course. For multiface video swaps: 4) When trying to swap multiple faces, the same person person is often not recognized, especially when the distance to the camera changes or in profile shots. You can mitigate it by running the face swap again using new src/dst pictures with the previous output video but it is time consuming and doesn't always work. Now, if I understand it correctly, Insightface can output what it believes are different faces, as seen here: https://camo.githubusercontent.com/94afea3fd9149d1859c601e5f396b2d57ed803d8cc1401fcbdc76e669e162943/68747470733a2f2f696e7369676874666163652e61692f6173736574732f696d672f6769746875622f666163657265636f676e6974696f6e66726f6d766964656f2e504e47 Would it be possible to have an option to run a preliminary face recognition over a video/folder and get a src picture for every **different** face in the video ? Ideally one could then assign one dst face to all src faces the program thinks are different but are actually the same person. Thank you for your time and, once again, thank you for the amazing work you have already done.
👀 1
mata3o commented 2021-08-01 14:16:54 +02:00 (Migrated from github.com)
Copy Link

hey @dasasf could you tell me what commands you use for swapping single images? i cannot for the life of me get image swap to work no matter what while video works just fine

hey @dasasf could you tell me what commands you use for swapping single images? i cannot for the life of me get image swap to work no matter what while video works just fine
dasasf commented 2021-08-04 14:14:21 +02:00 (Migrated from github.com)
Copy Link

What does the error message say when you try ? I noticed image swaps don't work on faces that are too zoomed in, even if they work for video swaps and you can't specifiy an output filename, only the folder. If you do, no output file will be generated.

What does the error message say when you try ? I noticed image swaps don't work on faces that are too zoomed in, even if they work for video swaps and you can't specifiy an output filename, only the folder. If you do, no output file will be generated.
mata3o commented 2021-08-04 14:53:15 +02:00 (Migrated from github.com)
Copy Link

i apologize for the long spam but its the only way i could think to show you what all is happening. I did all install instructions to the letter (twice, clean install the 2nd time) to make sure i didnt miss anything. Would you have a recommendation for picture sizes for swap and also could you tell me what command lines you use for single face single picture swap? Ive tried everything i can think of and all variations of picture face swap(single multi etc). video works like a charm though with no errors. I greatly appreciate your reply and any help or insight you might be able to give, cheers!

(simswap) C:\Users\user1\Desktop\Simswap Overlord\SimSwap-main>python test_wholeimage_swapsingle.py --isTrain false --use_mask --name people --Arc_path arcface_model/arcface_checkpoint.tar --pic_a_path ./demo_file/testpicA.jpg --pic_b_path ./demo_file/testpicB.jpg --output_path ./output/result_whole_swapsingle.jpg
------------ Options -------------
Arc_path: arcface_model/arcface_checkpoint.tar
aspect_ratio: 1.0
batchSize: 8
checkpoints_dir: ./checkpoints
cluster_path: features_clustered_010.npy
data_type: 32
dataroot: ./datasets/cityscapes/
display_winsize: 512
engine: None
export_onnx: None
feat_num: 3
fineSize: 512
fp16: False
gpu_ids: [0]
how_many: 50
id_thres: 0.03
image_size: 224
input_nc: 3
instance_feat: False
isTrain: False
label_feat: False
label_nc: 0
latent_size: 512
loadSize: 1024
load_features: False
local_rank: 0
max_dataset_size: inf
model: pix2pixHD
multisepcific_dir: ./demo_file/multispecific
nThreads: 2
n_blocks_global: 6
n_blocks_local: 3
n_clusters: 10
n_downsample_E: 4
n_downsample_global: 3
n_local_enhancers: 1
name: people
nef: 16
netG: global
ngf: 64
niter_fix_global: 0
no_flip: False
no_instance: False
no_simswaplogo: False
norm: batch
norm_G: spectralspadesyncbatch3x3
ntest: inf
onnx: None
output_nc: 3
output_path: ./output/result_whole_swapsingle.jpg
phase: test
pic_a_path: ./demo_file/testpicA.jpg
pic_b_path: ./demo_file/testpicB.jpg
pic_specific_path: ./crop_224/zrf.jpg
resize_or_crop: scale_width
results_dir: ./results/
semantic_nc: 3
serial_batches: False
temp_path: ./temp_results
tf_log: False
use_dropout: False
use_encoded_image: False
use_mask: True
verbose: False
video_path: ./demo_file/multi_people_1080p.mp4
which_epoch: latest
-------------- End ----------------
(142, 366, 4)
C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.parallel.data_parallel.DataParallel' has changed. Tried to save a patch, but couldn't create a writable file DataParallel.patch. Make sure it doesn't exist and your working directory is writable.
warnings.warn(msg, SourceChangeWarning)
C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.conv.Conv2d' has changed. Tried to save a patch, but couldn't create a writable file Conv2d.patch. Make sure it doesn't exist and your working directory is writable.
warnings.warn(msg, SourceChangeWarning)
C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.batchnorm.BatchNorm2d' has changed. Tried to save a patch, but couldn't create a writable file BatchNorm2d.patch. Make sure it doesn't exist and your working directory is writable.
warnings.warn(msg, SourceChangeWarning)
C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.activation.PReLU' has changed. Tried to save a patch, but couldn't create a writable file PReLU.patch. Make sure it doesn't exist and your working directory is writable.
warnings.warn(msg, SourceChangeWarning)
C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.pooling.MaxPool2d' has changed. Tried to save a patch, but couldn't create a writable file MaxPool2d.patch. Make sure it doesn't exist and your working directory is writable.
warnings.warn(msg, SourceChangeWarning)
C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.container.Sequential' has changed. Tried to save a patch, but couldn't create a writable file Sequential.patch. Make sure it doesn't exist and your working directory is writable.
warnings.warn(msg, SourceChangeWarning)
C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.pooling.AdaptiveAvgPool2d' has changed. Tried to save a patch, but couldn't create a writable file AdaptiveAvgPool2d.patch. Make sure it doesn't exist and your working directory is writable.
warnings.warn(msg, SourceChangeWarning)
C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.linear.Linear' has changed. Tried to save a patch, but couldn't create a writable file Linear.patch. Make sure it doesn't exist and your working directory is writable.
warnings.warn(msg, SourceChangeWarning)
C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.activation.Sigmoid' has changed. Tried to save a patch, but couldn't create a writable file Sigmoid.patch. Make sure it doesn't exist and your working directory is writable.
warnings.warn(msg, SourceChangeWarning)
C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.dropout.Dropout' has changed. Tried to save a patch, but couldn't create a writable file Dropout.patch. Make sure it doesn't exist and your working directory is writable.
warnings.warn(msg, SourceChangeWarning)
C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.batchnorm.BatchNorm1d' has changed. Tried to save a patch, but couldn't create a writable file BatchNorm1d.patch. Make sure it doesn't exist and your working directory is writable.
warnings.warn(msg, SourceChangeWarning)
input mean and std: 127.5 127.5
find model: ./insightface_func/models\antelope\glintr100.onnx recognition
find model: ./insightface_func/models\antelope\scrfd_10g_bnkps.onnx detection
set det-size: (640, 640)
C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\nn\functional.py:3500: UserWarning: The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
"The default behavior for interpolate/upsample with float scale_factor changed "
C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\nn\functional.py:3455: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
"See the documentation of nn.Upsample for details.".format(mode)

************ Done ! ************

i apologize for the long spam but its the only way i could think to show you what all is happening. I did all install instructions to the letter (twice, clean install the 2nd time) to make sure i didnt miss anything. Would you have a recommendation for picture sizes for swap and also could you tell me what command lines you use for single face single picture swap? Ive tried everything i can think of and all variations of picture face swap(single multi etc). video works like a charm though with no errors. I greatly appreciate your reply and any help or insight you might be able to give, cheers! (simswap) C:\Users\user1\Desktop\Simswap Overlord\SimSwap-main>python test_wholeimage_swapsingle.py --isTrain false --use_mask --name people --Arc_path arcface_model/arcface_checkpoint.tar --pic_a_path ./demo_file/testpicA.jpg --pic_b_path ./demo_file/testpicB.jpg --output_path ./output/result_whole_swapsingle.jpg ------------ Options ------------- Arc_path: arcface_model/arcface_checkpoint.tar aspect_ratio: 1.0 batchSize: 8 checkpoints_dir: ./checkpoints cluster_path: features_clustered_010.npy data_type: 32 dataroot: ./datasets/cityscapes/ display_winsize: 512 engine: None export_onnx: None feat_num: 3 fineSize: 512 fp16: False gpu_ids: [0] how_many: 50 id_thres: 0.03 image_size: 224 input_nc: 3 instance_feat: False isTrain: False label_feat: False label_nc: 0 latent_size: 512 loadSize: 1024 load_features: False local_rank: 0 max_dataset_size: inf model: pix2pixHD multisepcific_dir: ./demo_file/multispecific nThreads: 2 n_blocks_global: 6 n_blocks_local: 3 n_clusters: 10 n_downsample_E: 4 n_downsample_global: 3 n_local_enhancers: 1 name: people nef: 16 netG: global ngf: 64 niter_fix_global: 0 no_flip: False no_instance: False no_simswaplogo: False norm: batch norm_G: spectralspadesyncbatch3x3 ntest: inf onnx: None output_nc: 3 output_path: ./output/result_whole_swapsingle.jpg phase: test pic_a_path: ./demo_file/testpicA.jpg pic_b_path: ./demo_file/testpicB.jpg pic_specific_path: ./crop_224/zrf.jpg resize_or_crop: scale_width results_dir: ./results/ semantic_nc: 3 serial_batches: False temp_path: ./temp_results tf_log: False use_dropout: False use_encoded_image: False use_mask: True verbose: False video_path: ./demo_file/multi_people_1080p.mp4 which_epoch: latest -------------- End ---------------- (142, 366, 4) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.parallel.data_parallel.DataParallel' has changed. Tried to save a patch, but couldn't create a writable file DataParallel.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.conv.Conv2d' has changed. Tried to save a patch, but couldn't create a writable file Conv2d.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.batchnorm.BatchNorm2d' has changed. Tried to save a patch, but couldn't create a writable file BatchNorm2d.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.activation.PReLU' has changed. Tried to save a patch, but couldn't create a writable file PReLU.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.pooling.MaxPool2d' has changed. Tried to save a patch, but couldn't create a writable file MaxPool2d.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.container.Sequential' has changed. Tried to save a patch, but couldn't create a writable file Sequential.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.pooling.AdaptiveAvgPool2d' has changed. Tried to save a patch, but couldn't create a writable file AdaptiveAvgPool2d.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.linear.Linear' has changed. Tried to save a patch, but couldn't create a writable file Linear.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.activation.Sigmoid' has changed. Tried to save a patch, but couldn't create a writable file Sigmoid.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.dropout.Dropout' has changed. Tried to save a patch, but couldn't create a writable file Dropout.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\serialization.py:656: SourceChangeWarning: source code of class 'torch.nn.modules.batchnorm.BatchNorm1d' has changed. Tried to save a patch, but couldn't create a writable file BatchNorm1d.patch. Make sure it doesn't exist and your working directory is writable. warnings.warn(msg, SourceChangeWarning) input mean and std: 127.5 127.5 find model: ./insightface_func/models\antelope\glintr100.onnx recognition find model: ./insightface_func/models\antelope\scrfd_10g_bnkps.onnx detection set det-size: (640, 640) C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\nn\functional.py:3500: UserWarning: The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details. "The default behavior for interpolate/upsample with float scale_factor changed " C:\Users\user1\MiniConda3\envs\simswap\lib\site-packages\torch\nn\functional.py:3455: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details. "See the documentation of nn.Upsample for details.".format(mode) ************ Done ! ************
dasasf commented 2021-08-04 15:19:44 +02:00 (Migrated from github.com)
Copy Link

I use the commands described here: https://github.com/neuralchen/SimSwap/blob/main/docs/guidance/usage.md . It looks to me your problem is that you specify an output file name.

Try running this command and see if it works:

python test_wholeimage_swapsingle.py --isTrain false --use_mask --name people --Arc_path arcface_model/arcface_checkpoint.tar --pic_a_path ./demo_file/testpicA.jpg --pic_b_path ./demo_file/testpicB.jpg --output_path ./output/

I use the commands described here: https://github.com/neuralchen/SimSwap/blob/main/docs/guidance/usage.md . It looks to me your problem is that you specify an output file name. Try running this command and see if it works: python test_wholeimage_swapsingle.py --isTrain false --use_mask --name people --Arc_path arcface_model/arcface_checkpoint.tar --pic_a_path ./demo_file/testpicA.jpg --pic_b_path ./demo_file/testpicB.jpg --output_path ./output/
🎉 1 🚀 1
mata3o commented 2021-08-04 19:45:19 +02:00 (Migrated from github.com)
Copy Link

THANK YOU SO MUCH, it does seem the problem was me putting in a file name (i was putting in result_swapsingle for a name) and the problem compiled by it being rather picky on image size for swapping pictures! Still throws all those error messages but it works so im totally fine with that. Again i very much appreciate you taking the time to lend a fresh perspective on it all! Cheers m8

THANK YOU SO MUCH, it does seem the problem was me putting in a file name (i was putting in result_swapsingle for a name) and the problem compiled by it being rather picky on image size for swapping pictures! Still throws all those error messages but it works so im totally fine with that. Again i very much appreciate you taking the time to lend a fresh perspective on it all! Cheers m8
🎉 1
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
GIF
GUI
Increase inference time
A solution to increase inference time
 bug
Something isn't working
 documentation
Improvements or additions to documentation
 duplicate
This issue or pull request already exists
 enhancement
New feature or request
 good first issue
Good for newcomers
 help wanted
Extra attention is needed
 invalid
This doesn't seem right
 question
Further information is requested
 wontfix
This will not be worked on
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
CalvinBackup
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: CalvinBackup/neuralchen-SimSwap#73
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?