neuralchen-SimSwap/SimSwap colab.ipynb

This is a simple example of SimSwap on processing video with multiple faces. You can change the codes for inference based on our other scripts for image or single face swapping.

Code path: https://github.com/neuralchen/SimSwap

Paper path: https://arxiv.org/pdf/2106.06340v1.pdf or https://dl.acm.org/doi/10.1145/3394171.3413630

In [1]:

## make sure you are using a runtime with GPU
## you can check at Runtime/Change runtime type in the top bar.
!nvidia-smi

Mon Jun 21 02:13:20 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.27       Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   45C    P8    10W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Installation

All file changes made by this notebook are temporary. You can try to mount your own google drive to store files if you want.

In [2]:

!git clone https://github.com/neuralchen/SimSwap
!cd SimSwap && git pull

Cloning into 'SimSwap'...
remote: Enumerating objects: 362, done.
remote: Counting objects: 100% (362/362), done.
remote: Compressing objects: 100% (281/281), done.
remote: Total 362 (delta 149), reused 272 (delta 67), pack-reused 0
Receiving objects: 100% (362/362), 101.31 MiB | 32.47 MiB/s, done.
Resolving deltas: 100% (149/149), done.
Already up to date.

In [3]:

!pip install insightface==0.2.1 onnxruntime moviepy
!pip install googledrivedownloader
!pip install imageio==2.4.1

Collecting insightface==0.2.1
  Downloading https://files.pythonhosted.org/packages/ee/1e/6395bbe0db665f187c8e49266cda54fcf661f182192370d409423e4943e4/insightface-0.2.1-py2.py3-none-any.whl
Collecting onnxruntime
[?25l  Downloading https://files.pythonhosted.org/packages/f9/76/3d0f8bb2776961c7335693df06eccf8d099e48fa6fb552c7546867192603/onnxruntime-1.8.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.5MB)
     |████████████████████████████████| 4.5MB 10.2MB/s 
[?25hRequirement already satisfied: moviepy in /usr/local/lib/python3.7/dist-packages (0.2.3.5)
Collecting onnx
[?25l  Downloading https://files.pythonhosted.org/packages/3f/9b/54c950d3256e27f970a83cd0504efb183a24312702deed0179453316dbd0/onnx-1.9.0-cp37-cp37m-manylinux2010_x86_64.whl (12.2MB)
     |████████████████████████████████| 12.2MB 51.4MB/s 
[?25hRequirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from insightface==0.2.1) (3.2.2)
Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from insightface==0.2.1) (4.41.1)
Requirement already satisfied: Pillow in /usr/local/lib/python3.7/dist-packages (from insightface==0.2.1) (7.1.2)
Requirement already satisfied: scikit-image in /usr/local/lib/python3.7/dist-packages (from insightface==0.2.1) (0.16.2)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from insightface==0.2.1) (2.23.0)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from insightface==0.2.1) (0.22.2.post1)
Requirement already satisfied: opencv-python in /usr/local/lib/python3.7/dist-packages (from insightface==0.2.1) (4.1.2.30)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from insightface==0.2.1) (1.19.5)
Requirement already satisfied: easydict in /usr/local/lib/python3.7/dist-packages (from insightface==0.2.1) (1.9)
Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from insightface==0.2.1) (1.4.1)
Requirement already satisfied: flatbuffers in /usr/local/lib/python3.7/dist-packages (from onnxruntime) (1.12)
Requirement already satisfied: protobuf in /usr/local/lib/python3.7/dist-packages (from onnxruntime) (3.12.4)
Requirement already satisfied: decorator<5.0,>=4.0.2 in /usr/local/lib/python3.7/dist-packages (from moviepy) (4.4.2)
Requirement already satisfied: imageio<3.0,>=2.1.2 in /usr/local/lib/python3.7/dist-packages (from moviepy) (2.4.1)
Requirement already satisfied: typing-extensions>=3.6.2.1 in /usr/local/lib/python3.7/dist-packages (from onnx->insightface==0.2.1) (3.7.4.3)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from onnx->insightface==0.2.1) (1.15.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->insightface==0.2.1) (1.3.1)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->insightface==0.2.1) (2.8.1)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->insightface==0.2.1) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->insightface==0.2.1) (2.4.7)
Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image->insightface==0.2.1) (2.5.1)
Requirement already satisfied: PyWavelets>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image->insightface==0.2.1) (1.1.1)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->insightface==0.2.1) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->insightface==0.2.1) (2021.5.30)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->insightface==0.2.1) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->insightface==0.2.1) (1.24.3)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->insightface==0.2.1) (1.0.1)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from protobuf->onnxruntime) (57.0.0)
Installing collected packages: onnx, insightface, onnxruntime
Successfully installed insightface-0.2.1 onnx-1.9.0 onnxruntime-1.8.0
Requirement already satisfied: googledrivedownloader in /usr/local/lib/python3.7/dist-packages (0.4)
Requirement already satisfied: imageio==2.4.1 in /usr/local/lib/python3.7/dist-packages (2.4.1)
Requirement already satisfied: pillow in /usr/local/lib/python3.7/dist-packages (from imageio==2.4.1) (7.1.2)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from imageio==2.4.1) (1.19.5)

In [4]:

import os
os.chdir("SimSwap")
!ls

 crop_224	    models		   test_one_image.py
 data		    options		   test_video_swapmulti.py
 demo_file	    output		   test_video_swapsingle.py
 doc		    README.md		   test_wholeimage_swapmulti.py
 insightface_func  'SimSwap colab.ipynb'   test_wholeimage_swapsingle.py
 LICENSE	    simswaplogo		   util

In [5]:

from google_drive_downloader import GoogleDriveDownloader

### it seems that google drive link may not be permenant, you can find this ID from our open url.
GoogleDriveDownloader.download_file_from_google_drive(file_id='1TLNdIufzwesDbyr_nVTR7Zrx9oRHLM_N',
                                    dest_path='./arcface_model/arcface_checkpoint.tar')
GoogleDriveDownloader.download_file_from_google_drive(file_id='1PXkRiBUYbu1xWpQyDEJvGKeqqUFthJcI',
                                    dest_path='./checkpoints.zip')
!unzip ./checkpoints.zip  -d ./checkpoints

Downloading 1TLNdIufzwesDbyr_nVTR7Zrx9oRHLM_N into ./arcface_model/arcface_checkpoint.tar... Done.
Downloading 1PXkRiBUYbu1xWpQyDEJvGKeqqUFthJcI into ./checkpoints.zip... Done.
Archive:  ./checkpoints.zip
   creating: ./checkpoints/people/
  inflating: ./checkpoints/people/iter.txt  
  inflating: ./checkpoints/people/latest_net_D1.pth  
  inflating: ./checkpoints/people/latest_net_D2.pth  
  inflating: ./checkpoints/people/latest_net_G.pth  
  inflating: ./checkpoints/people/loss_log.txt  
  inflating: ./checkpoints/people/opt.txt  
   creating: ./checkpoints/people/web/
   creating: ./checkpoints/people/web/images/

In [6]:

## You can upload filed manually
# from google.colab import drive
# drive.mount('/content/gdrive')

### Now onedrive file can be downloaded in Colab directly!
### If the link blow is not permanent, you can just download it from the 
### open url(can be found at [our repo]/doc/guidance/preparation.md) and copy the assigned download link here.
### many thanks to woctezuma for this very useful help
!wget --no-check-certificate "https://sh23tw.dm.files.1drv.com/y4mmGiIkNVigkSwOKDcV3nwMJulRGhbtHdkheehR5TArc52UjudUYNXAEvKCii2O5LAmzGCGK6IfleocxuDeoKxDZkNzDRSt4ZUlEt8GlSOpCXAFEkBwaZimtWGDRbpIGpb_pz9Nq5jATBQpezBS6G_UtspWTkgrXHHxhviV2nWy8APPx134zOZrUIbkSF6xnsqzs3uZ_SEX_m9Rey0ykpx9w" -O antelope.zip
!unzip ./antelope.zip -d ./insightface_func/models/

--2021-06-21 02:14:17--  https://sh23tw.dm.files.1drv.com/y4mmGiIkNVigkSwOKDcV3nwMJulRGhbtHdkheehR5TArc52UjudUYNXAEvKCii2O5LAmzGCGK6IfleocxuDeoKxDZkNzDRSt4ZUlEt8GlSOpCXAFEkBwaZimtWGDRbpIGpb_pz9Nq5jATBQpezBS6G_UtspWTkgrXHHxhviV2nWy8APPx134zOZrUIbkSF6xnsqzs3uZ_SEX_m9Rey0ykpx9w
Resolving sh23tw.dm.files.1drv.com (sh23tw.dm.files.1drv.com)... 13.107.42.12
Connecting to sh23tw.dm.files.1drv.com (sh23tw.dm.files.1drv.com)|13.107.42.12|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 248024513 (237M) [application/zip]
Saving to: ‘antelope.zip’

antelope.zip        100%[===================>] 236.53M  6.16MB/s    in 31s     

2021-06-21 02:14:48 (7.66 MB/s) - ‘antelope.zip’ saved [248024513/248024513]

Archive:  ./antelope.zip
   creating: ./insightface_func/models/antelope/
  inflating: ./insightface_func/models/antelope/glintr100.onnx  
  inflating: ./insightface_func/models/antelope/scrfd_10g_bnkps.onnx  

Inference

In [7]:

import cv2
import torch
import fractions
import numpy as np
from PIL import Image
import torch.nn.functional as F
from torchvision import transforms
from models.models import create_model
from options.test_options import TestOptions
from insightface_func.face_detect_crop_multi import Face_detect_crop
from util.videoswap import video_swap
from util.add_watermark import watermark_image

Imageio: 'ffmpeg-linux64-v3.3.1' was not found on your computer; downloading it now.
Try 1. Download from https://github.com/imageio/imageio-binaries/raw/master/ffmpeg/ffmpeg-linux64-v3.3.1 (43.8 MB)
Downloading: 8192/45929032 bytes (0.0%)1286144/45929032 bytes (2.8%)3653632/45929032 bytes (8.0%)7479296/45929032 bytes (16.3%)11526144/45929032 bytes (25.1%)15171584/45929032 bytes (33.0%)18997248/45929032 bytes (41.4%)22724608/45929032 bytes (49.5%)26673152/45929032 bytes (58.1%)30728192/45929032 bytes (66.9%)34725888/45929032 bytes (75.6%)38879232/45929032 bytes (84.7%)42680320/45929032 bytes (92.9%)45929032/45929032 bytes (100.0%)
  Done
File saved as /root/.imageio/ffmpeg/ffmpeg-linux64-v3.3.1.

In [8]:

transformer = transforms.Compose([
        transforms.ToTensor(),
        #transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])

transformer_Arcface = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])

detransformer = transforms.Compose([
        transforms.Normalize([0, 0, 0], [1/0.229, 1/0.224, 1/0.225]),
        transforms.Normalize([-0.485, -0.456, -0.406], [1, 1, 1])
    ])

In [9]:

opt = TestOptions()
opt.initialize()
opt.parser.add_argument('-f') ## dummy arg to avoid bug
opt = opt.parse()
opt.pic_a_path = './demo_file/Iron_man.jpg' ## or replace it with image from your own google drive
opt.video_path = './demo_file/multi_people_1080p.mp4' ## or replace it with video from your own google drive
opt.output_path = './output/demo.mp4'
opt.temp_path = './tmp'
opt.Arc_path = './arcface_model/arcface_checkpoint.tar'
opt.isTrain = False

crop_size = 224

torch.nn.Module.dump_patches = True
model = create_model(opt)
model.eval()


app = Face_detect_crop(name='antelope', root='./insightface_func/models')
app.prepare(ctx_id= 0, det_thresh=0.6, det_size=(640,640))

pic_a = opt.pic_a_path
# img_a = Image.open(pic_a).convert('RGB')
img_a_whole = cv2.imread(pic_a)
img_a_align_crop, _ = app.get(img_a_whole,crop_size)
img_a_align_crop_pil = Image.fromarray(cv2.cvtColor(img_a_align_crop[0],cv2.COLOR_BGR2RGB)) 
img_a = transformer_Arcface(img_a_align_crop_pil)
img_id = img_a.view(-1, img_a.shape[0], img_a.shape[1], img_a.shape[2])

# convert numpy to tensor
img_id = img_id.cuda()

#create latent id
img_id_downsample = F.interpolate(img_id, scale_factor=0.5)
latend_id = model.netArc(img_id_downsample)
latend_id = latend_id.detach().to('cpu')
latend_id = latend_id/np.linalg.norm(latend_id,axis=1,keepdims=True)
latend_id = latend_id.to('cuda')

video_swap(opt.video_path, latend_id, model, app, opt.output_path,temp_results_dir=opt.temp_path)

------------ Options -------------
Arc_path: models/BEST_checkpoint.tar
aspect_ratio: 1.0
batchSize: 8
checkpoints_dir: ./checkpoints
cluster_path: features_clustered_010.npy
data_type: 32
dataroot: ./datasets/cityscapes/
display_winsize: 512
engine: None
export_onnx: None
f: /root/.local/share/jupyter/runtime/kernel-6d955151-4911-464a-824d-f0806d8071f6.json
feat_num: 3
fineSize: 512
fp16: False
gpu_ids: [0]
how_many: 50
image_size: 224
input_nc: 3
instance_feat: False
isTrain: False
label_feat: False
label_nc: 0
latent_size: 512
loadSize: 1024
load_features: False
local_rank: 0
max_dataset_size: inf
model: pix2pixHD
nThreads: 2
n_blocks_global: 6
n_blocks_local: 3
n_clusters: 10
n_downsample_E: 4
n_downsample_global: 3
n_local_enhancers: 1
name: people
nef: 16
netG: global
ngf: 64
niter_fix_global: 0
no_flip: False
no_instance: False
norm: batch
norm_G: spectralspadesyncbatch3x3
ntest: inf
onnx: None
output_nc: 3
output_path: ./output/
phase: test
pic_a_path: ./crop_224/gdg.jpg
pic_b_path: ./crop_224/zrf.jpg
resize_or_crop: scale_width
results_dir: ./results/
semantic_nc: 3
serial_batches: False
temp_path: ./temp_results
tf_log: False
use_dropout: False
use_encoded_image: False
verbose: False
video_path: ./demo_file/multi_people_1080p.mp4
which_epoch: latest
-------------- End ----------------
input mean and std: 127.5 127.5
find model: ./insightface_func/models/antelope/glintr100.onnx recognition
find model: ./insightface_func/models/antelope/scrfd_10g_bnkps.onnx detection
set det-size: (640, 640)

  0%|          | 0/594 [00:00<?, ?it/s]

(142, 366, 4)

100%|██████████| 594/594 [08:45<00:00,  1.13it/s]

[MoviePy] >>>> Building video ./output/demo.mp4
[MoviePy] Writing audio in demoTEMP_MPY_wvf_snd.mp3

100%|██████████| 438/438 [00:00<00:00, 877.18it/s]

[MoviePy] Done.
[MoviePy] Writing video ./output/demo.mp4

100%|██████████| 595/595 [00:53<00:00, 11.15it/s]

[MoviePy] Done.
[MoviePy] >>>> Video ready: ./output/demo.mp4 

In [ ]: