Replace Face based on size binning #2

New Issue

2023-07-29T23:13:39+02:00

Oceanswave commented

2023-07-29 23:13:39 +02:00

(Migrated from github.com)

Is your feature request related to a problem? Please describe.
Often a face that is detected isn't the face that I'd like to replace. Face Similarity/Filtering helps, but often I want to replace a face given the actual face size.

Often times a subject face I'd like to swap is the subject, and thus in the foreground, and large. Secondary faces appear in the background, and are smaller.

Describe the solution you'd like
Given a small/medium/large 'Size' setting on a Face Swap Unit, Loop through the detected faces in the generated image. Create a histogram of these faces. Bin the faces into small/medium/large bins. Swap the face in the specified size bin at the specified index.

An optional Face Swap Unit setting would sort each bin by either the detected generated face age or the face's euclidean distance from the generated image center.

Psudocode:

# Calculate the width of the image and determine the image center
image_height, image_width = img_data.shape[:2]
image_center = (image_width // 2, image_height // 2)

# Calculate areas and categorize bounding boxes
areas = [(face.bbox[2] - face.bbox[0]) * (face.bbox[3] - face.bbox[1]) for face in gender_faces]
 
# Use numpy histogram to categorize into 3 bins
_, bin_edges = np.histogram(areas, bins=3)

# Place the faces in bins
bins = {'small': [], 'medium': [], 'large': []}
for face in gender_faces:
    area = (face.bbox[2] - face.bbox[0]) * (face.bbox[3] - face.bbox[1])
    if area < bin_edges[1]:
        bins['small'].append(face)
    elif area < bin_edges[2]:
        bins['medium'].append(face)
    else:
        bins['large'].append(face)

# Sort bins by setting
sorted_bins = {}
for size in ['large', 'medium', 'small']:
    if (sort = "center")
    # sort by distance from center
      sorted_bins[size] = sorted(
        bins[size], key=lambda face: ((face.bbox[2] + face.bbox[0]) / 2 - image_center[0]) ** 2 + ((face.bbox[3] + face.bbox[1]) / 2 - image_center[1]) ** 2
    # )
    else (sort = "age")
    # sort by age ascending
      sorted_bins[size] = sorted(bins[size], key=lambda face: face.age)

Describe alternatives you've considered
Face similarity, however this doesn't work on novel faces. Adding a named person, e.g. Albert Einstein, sometimes skews the generated image to contain particulars of that person/training that are undesirable in some instances or make it difficult to combine with other aspects contained in the prompt.

Additional context
Upon request.

**Is your feature request related to a problem? Please describe.** Often a face that is detected isn't the face that I'd like to replace. Face Similarity/Filtering helps, but often I want to replace a face given the actual face size. Often times a subject face I'd like to swap is the subject, and thus in the foreground, and large. Secondary faces appear in the background, and are smaller. **Describe the solution you'd like** Given a small/medium/large 'Size' setting on a Face Swap Unit, Loop through the detected faces in the generated image. Create a histogram of these faces. Bin the faces into small/medium/large bins. Swap the face in the specified size bin at the specified index. An optional Face Swap Unit setting would sort each bin by either the detected generated face age or the face's euclidean distance from the generated image center. Psudocode: ``` py # Calculate the width of the image and determine the image center image_height, image_width = img_data.shape[:2] image_center = (image_width // 2, image_height // 2) # Calculate areas and categorize bounding boxes areas = [(face.bbox[2] - face.bbox[0]) * (face.bbox[3] - face.bbox[1]) for face in gender_faces] # Use numpy histogram to categorize into 3 bins _, bin_edges = np.histogram(areas, bins=3) # Place the faces in bins bins = {'small': [], 'medium': [], 'large': []} for face in gender_faces: area = (face.bbox[2] - face.bbox[0]) * (face.bbox[3] - face.bbox[1]) if area < bin_edges[1]: bins['small'].append(face) elif area < bin_edges[2]: bins['medium'].append(face) else: bins['large'].append(face) # Sort bins by setting sorted_bins = {} for size in ['large', 'medium', 'small']: if (sort = "center") # sort by distance from center sorted_bins[size] = sorted( bins[size], key=lambda face: ((face.bbox[2] + face.bbox[0]) / 2 - image_center[0]) ** 2 + ((face.bbox[3] + face.bbox[1]) / 2 - image_center[1]) ** 2 # ) else (sort = "age") # sort by age ascending sorted_bins[size] = sorted(bins[size], key=lambda face: face.age) ``` **Describe alternatives you've considered** Face similarity, however this doesn't work on novel faces. Adding a named person, e.g. Albert Einstein, sometimes skews the generated image to contain particulars of that person/training that are undesirable in some instances or make it difficult to combine with other aspects contained in the prompt. **Additional context** Upon request.

👍 2 ❤️ 1

glucauze commented

2023-07-30 00:49:54 +02:00

(Migrated from github.com)

It's an interesting approach indeed. I think I've pretty much got the idea. From a coding standpoint, it's certainly doable even though it does require quite a lot of adaptations. At first glance, I would say it requires:

modifying get_faces in swapper (in swapper.py) to take the bins into account.
There also needs to be an option in FaceSwapUnitSettings (in faceswaplab_unit_settings.py) to specify the bin (probably an enum).
Changing the parts of the API that correspond to FaceSwapUnitSettings. I use DTOs for the API that must match. It also requires changing the UI part so that it is consistent. So, modify api_utils.py and faceswaplab_unit_ui.py.
passing these choices in the swap_face part of swapper.py. That means modifying this method and process_image_unit in swapper.py

A potentially annoying point is the ergonomics of the interface to allow this choice in a comprehensible way. I think we need to keep the interface relatively simple (it's already pretty complicated ^^). Perhaps a checkbox.

I'm not saying that it's not doable or that I won't do it. But it's just too many modifications for now. Let's just say it's a little too ambitious for the moment. However, if you come up with a working prototype, why not :)

It's an interesting approach indeed. I think I've pretty much got the idea. From a coding standpoint, it's certainly doable even though it does require quite a lot of adaptations. At first glance, I would say it requires: + modifying get_faces in swapper (in swapper.py) to take the bins into account. + There also needs to be an option in FaceSwapUnitSettings (in faceswaplab_unit_settings.py) to specify the bin (probably an enum). + Changing the parts of the API that correspond to FaceSwapUnitSettings. I use DTOs for the API that must match. It also requires changing the UI part so that it is consistent. So, modify api_utils.py and faceswaplab_unit_ui.py. + passing these choices in the swap_face part of swapper.py. That means modifying this method and process_image_unit in swapper.py A potentially annoying point is the ergonomics of the interface to allow this choice in a comprehensible way. I think we need to keep the interface relatively simple (it's already pretty complicated ^^). Perhaps a checkbox. I'm not saying that it's not doable or that I won't do it. But it's just too many modifications for now. Let's just say it's a little too ambitious for the moment. However, if you come up with a working prototype, why not :)

👍 1 ❤️ 1

drdancm commented

2023-11-06 07:09:05 +01:00

(Migrated from github.com)

This problem comes up with photos of an audience with the subject of interest in the foreground, face is large, but the FaceSwap finds really small faces and it is very hard to even count how many of the background faces are going to be targeted.

It's not easy because partial faces are sometimes targeted, and other times they are not.

I suppose it is not possible to solve this an other problems by (User Option) simply having the user draw a virtual x across the target face(s) or some sort of mark, maybe even M for Male and F for Female.

This problem comes up with photos of an audience with the subject of interest in the foreground, face is large, but the FaceSwap finds really small faces and it is very hard to even count how many of the background faces are going to be targeted. It's not easy because partial faces are sometimes targeted, and other times they are not. I suppose it is not possible to solve this an other problems by (User Option) simply having the user draw a virtual x across the target face(s) or some sort of mark, maybe even M for Male and F for Female.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: CalvinBackup/sd-webui-faceswaplab#2