They are all technical experts, so why are the photos taken by my husband so ugly, while the photos taken by it are so natural and beautiful?

Audit expert: Xu Runsheng

PhD in Artificial Intelligence, University of California

With the rise of various live broadcasts, short videos and social media, we seem to be living in a society where there are more beautiful women and handsome men per capita. Is this really the case?

Thanks to the development of various technologies and software, when we appear in front of the camera, the computer algorithm will automatically beautify the captured portraits, so the people in the camera are always beautiful.

So, what algorithms are used behind these software? How do computers recognize faces and bodies and beautify them?

The photo was roughly processed by the beauty camera. Source: pexels.com

Today, let’s take a look at how beauty software works.

There are many similarities between beauty enhancement and painting. Painting starts with drawing a line draft to determine the basic structure of the human body; then coloring is done on the line draft, gradually enriching the details, and finally completing a work of art.

The original human body images we take are also "line drawings". The mobile phone collects the original data and transmits the image to the background for positioning and identification. The purpose of this step is to determine the characteristic position of the human body. Then we start "coloring" and use algorithms and techniques to analyze, process and beautify the facial features or body parts to finally get the beautified image.

It’s just that it takes a long time to complete a painting, while beautification is instantaneous and immediate, which requires powerful computer technology as support.

Next, let’s take facial beautification (such as whitening, skin smoothing, and skin beautification) as an example and talk specifically about the main professional technologies used in beautification.

Several key steps in face beautification: facial key point positioning technology

We can use beauty cameras to apply eye shadow to the eyes and lipstick to the lips, but how can these functions accurately identify specific parts of the face in the image?

Source: Shangguan News

What is used here is the facial key point detection technology, that is, determining the precise position of the facial features. This can be achieved by finding facial feature points such as eyes, eyebrows, nose, mouth, and facial contours.

There are currently two mainstream methods for detecting facial key points: feature-based face detection and image-based face detection. The most commonly used method is feature-based face detection, and the eyes of a person will be recognized first. Generally speaking, the more key points are located, the better the final beauty effect. Currently, the popular numbers of key points are 5, 21, 29, 68, 96, etc., with 68 points being the most classic. As shown in the figure below, the outermost points of the left and right faces are 0 and 16 respectively, the outer and inner corners of the left eye are 36 and 39 respectively, the outer and inner corners of the right eye are 45 and 42 respectively, the chin is 8, the root of the nose and the tip of the nose are 27 and 30 respectively, and the center of the mouth is 66.

Facial key point location

Source: CSDN

The original face data is uploaded to the backend, and the face detection technology recognizes the original image frame. The eyes, eyebrows, T-zone (the area consisting of eyes + nose), mouth, and chin are recognized in turn. The recognized data is put into the face information structure, and then applied to the specific beauty algorithm to beautify specific parts.

Source: Sohu

Microdermabrasion technology

Girls will apply a layer of foundation before putting on makeup. Its main function is to whiten and reduce skin blemishes, which is equivalent to the "skin smoothing" and "whitening" functions in the beauty function.

Skin resurfacing is the most basic step in beauty

Source: Internet

What should a good skin resurfacing effect look like? First and foremost, it is important to make the skin smooth while also preserving as many details as possible (such as skin texture) so that the contours of the facial features are clearly visible. This way, the picture looks more realistic and has a better texture.

Source: TV series stills

Skin resurfacing uses image smoothing technology, because when shooting an image, it will be disturbed by noise, which will reduce the image quality, so the noise must be filtered. The filtered image will become smooth and unnecessary information will be removed.

The tool for filtering image noise is the filter. Commonly used filters include median filtering, edge-preserving filtering, bilateral filtering (also known as Gaussian bilateral filtering), etc. The disadvantage of median filtering and edge-preserving filtering is that they cannot preserve the edge information of the image well; while bilateral filtering can preserve a lot of facial details and ensure that the boundaries will not be blurred, so it is the most commonly used.

Original image (left) and image after bilateral filtering (right)

Source: See watermark

After the portrait is processed with bilateral filtering, we can obviously see that the skin becomes smoother and the contour edges are clear. We can adjust the parameters to obtain different skin refining effects.

Make your skin fairer

Our human eyes have very high pixels and can see a colorful world. So how do computers without "eyes" recognize colors?

The colors in an image are generally represented in RGB mode, where R, G, and B represent red, green, and blue respectively, and the numbers vary from 0 to 255. When the values ​​of R, G, and B are closer to 0, the image becomes darker; and when the values ​​are closer to 255, the color becomes whiter. In addition, there is also the HSV color space (H, S, and V represent color, saturation, and brightness respectively), and these two color spaces can be converted through algorithms.

The corresponding relationship between other color modes and RGB. The angle (i.e. color), saturation, and brightness under the "hue" classification correspond to HSV respectively. Source: Wikipedia. Many whitening technologies convert RGB to HSV before adjusting. First, use an algorithm to perform a large amount of statistics on the color data of skin pixels to obtain the value range of skin color pixels, and then adjust the values ​​of each pixel. When the values ​​of H, S, and V tend to 0°, 0, and 100% respectively, the skin is the whitest and brightest.

HSV color space model

Source: Baidu Encyclopedia

Original image (left) and image after whitening and smoothing (right)

Source: pexels

Make your face smaller

What to do if you don’t want to lose weight but want to have a smaller face? Turn on the face slimming function of the beauty software!

In fact, adjustments such as face thinning, enlarging eyes, and thinning nose all use liquefaction deformation technology. The liquefaction algorithm takes a point as the center. Within its range of influence, we push the center point to change the shape of the image. In the circle, the closer to the center point O, the more obvious the deformation.

The figure below is a schematic diagram of the principle of the liquefaction algorithm. O is the center point of the circle, R is the liquefaction radius, and when we drag O toward OA, the pixel C in the image will be transformed to the position of D. Through a series of formulas, we can calculate the pixel value of each transformed point in the circular area, so that we can get the transformed image.

Schematic diagram of the liquefaction algorithm

We treat the facial area as a circle, select a point as the center, set the liquefaction radius, and the points within the radius can move in a certain direction and deform, and finally render to achieve a face-slimming effect.

Source: pexels

Cameras can help people put on makeup

You can also DIY makeup in the beauty camera, including lipstick, blush, eyebrows, eye makeup, etc., which is very popular among people who don’t know how to put on makeup.

Source: pexels

The camera is implanted with pre-designed makeup templates of different styles. When a face appears, the specific parts will be identified based on facial key point detection. Then the makeup template will be projected onto the specific area of ​​the face. Finally, the makeup and the original image will be efficiently synthesized.

Taking eyebrows as an example, the camera provides multiple sets of eyebrow image templates. First, the key point positions of the eyebrows in the original image are detected, and the first key point of the eyebrows will be used as the starting position for fitting; the algorithm will also adjust the size and range of the eyebrow template to make the template and the original eyebrows fit better; finally, it will be fused on the original image to obtain a natural and realistic eyebrow effect.

Multiple eyebrow options in Beauty Camera

Source: a beauty camera

Beauty is a technical job. Today we only shared the algorithms and technologies related to the basic beauty functions. Beauty also has many advanced functions, such as special effects decoration, cutout, etc. The image algorithms used in these functions are more complex. You can explore more.

Source: pexels