One-click transformation into a cartoon! The technology behind the popular Douyin special effects

Yesterday, Li Ronghao was on the hot search again because of his small eyes.

Under the topic #Special effects can’t make Li Ronghao’s eyes bigger#, there is a video of Li Ronghao and Rainie Yang’s wedding photo being swapped. After the face swap, Li Ronghao’s eyes have completely become a line. Sun Honglei also reposted the Weibo post “The pot calling the kettle black”:

This special effect is too much, Mr. Li, someone is trolling you!

In response to this, netizens’ comments were all “Hahahahahahahaha”.

Careful people may have noticed the TikTok APP logo in the video. That’s right! The function above that is not very friendly to people with small eyes is the new “Transform into Comics” special effect launched by TikTok.

Behind the effect of real people turning into comics with just one click is the Generative Adversarial Network (GAN).

Break the dimensional wall with one click

Currently, the Douyin "Transform into Comics" special effect ranks No. 1 on the "This Week's Real-Time List" (June 15-21, 2002) of the popular challenge list, and has been used by about 8.672 million users.

Among them, many celebrities shot the same video, and the ways of playing were also varied.

For example, Chen He took out a "mysterious key" that looked like a fly swatter to lift the seal of the two-dimensional world.

G.E.M. wipes the glass and changes her face.

During the time Zhang Yixing was rapping, he had no idea how many times he had used his hand gestures to travel between dimensions.

After entering the two-dimensional world, the stars’ hairstyles, facial features and other details are “perfectly restored”, while also adding a touch of playfulness and agility.

However, turning real people into cartoon styles is not uncommon. In recent years, special effects photo apps have emerged one after another. Users want to turn pictures and videos into cartoon styles, ancient styles, hand-painted styles, oil painting styles, or even change gender or age.

Generative Adversarial Networks (GANs)

When it comes to face-changing, one technology cannot be avoided - the generative adversarial network GAN.

Leifeng.com (official account: Leifeng.com) has previously reported that GAN, as a deep learning model, is a relatively new concept in machine learning. It first appeared in the paper "Generative Adversarial Networks" published by Ian Goodfellow, a famous scientist at Google Brain, and his team in October 2014.

Machine learning models are divided into two types: generative models and discriminative models. As the name suggests, their respective functions are to generate and judge based on data sets.

The reason why Ian Goodfellow became the "Father of GANs" is that he creatively proposed a concept of estimating the generative model through adversarial estimation. He trained the generative model and the discriminative model at the same time to let the two compete with each other. The generative model should generate realistic pictures as much as possible to deceive the discriminative model. At the same time, the discriminative model should also try its best to distinguish the pictures generated by the generative model from the original data set.

This process is like two players in a game battling each other. At first, they are both confused, but in the end, the truth becomes clearer and clearer. This kind of spontaneous mutual learning between machines can be said to be very intelligent.

There are many types of GANs, the most common of which is Cycle GAN.

In fact, CycleGAN is an important early attempt at face conversion. It can convert from source to target and also perform reverse conversion, thereby conveniently learning the conversion relationship between the two categories and better ensuring the quality of the converted image.

As shown in the picture above, if a zebra can be turned into a horse, then even if you change the action, the horse can be turned into a zebra again; similarly, summer can become winter, and winter can be turned back into summer after changing the scene.

Although the above-mentioned Tik Tok anime-style face-changing special effects have not disclosed specific technical information, previous attempts to transform human faces into cartoon images did use CycleGAN, and related papers were also published on the preprint platform arXiv in July 2019.

It is worth mentioning that this research utilized the so-called landmark assisted CycleGAN.

According to the paper, the "landmark" here is not the "landmark" we often say, but refers to facial features, so we can understand this technology as a special version of CycleGAN - "CycleGAN with facial feature assistance."

To see how good the technology is, let’s take a look at the results (Leifeng.com Note: the second column from the right is generated by CycleGAN; the first column from the right is generated by CycleGAN with the assistance of facial features).

The beauty in real life turned into a two-dimensional loli.

The cartoon image generated by the handsome guy also looks good.

In addition, there are also research teams that have used GAN to convert low-resolution images into high-resolution images using facial features. I feel like my myopia that I have had for many years has been cured!

After seeing this, do you want to exclaim that it’s amazing?

In fact, there are even more miraculous things, such as transforming living people into living people.

This is the employee personal information page on the website of REGIUM, a Spanish Internet chess board manufacturer. It seems that there is nothing unusual. But in fact, the four engineers in the frame are "dummy" and do not exist in real life.

Leifeng.com learned that these "artificial people" come from a foreign face-making website this person does not exist.com, and these photos are also based on GAN.

However, along with the emergence of face-changing technology, there are also many problems - for example, the ZAO APP that became popular overnight and then faded away in three days, and the Deepfake that is notorious worldwide, have caused concerns from all walks of life.

How to reasonably utilize this type of face-changing technology is a question that needs to be considered.