What did Meng Dehai say in "Storm"? Have you guessed it?

The recent hit TV series "Kuang Bi" has suddenly set off a wave of "solving cases", and many viewers have become famous detectives, trying to solve a secret. However, this secret is not about the case in the play, but about the play itself.

It turns out that the plot of the second half of "Kuangchuan" has many big plot twists, which are inconsistent with the plot logic laid out before. The audience suspected that these plots were changed after the TV series was filmed and then edited and dubbed again.

The picture is taken from the TV series "Kuangchuan"

The audience also provided evidence for this question: many characters' lip movements did not match the lines. The audience stared at the actors' mouths repeatedly and even deciphered the "original lines".

Nowadays, many film and television works are produced by post-production dubbing, and it is common that the lines and lip movements do not match each other. So, is it reliable to guess the lines by looking at the lip movements?

01

Information contained in moving mouth

Let's play a small game first. Please see the picture below:

Lip reading puzzle, Image source: Wikipedia

This is a fun picture puzzle. These twelve people in the picture took a "photo" while saying their names. We know that their names are Oom, Alden, Eastman, Alfred, Arthur, Luke, Fletcher, Matthew, Theodore, Richard, Shirmer and Hisswald. Can you tell who is who?

Please enlarge the picture, read their names, and compare their mouth shapes with your own. You will find that even without the help of English pronunciation, Chinese people can guess roughly. This is because when we speak, the shape of our mouth reflects the sound being made. Just by looking at the mouth shape, we can indeed guess some pronunciations.

Copyright image, no permission to reprint

Scientists have found that although the information of speech is mainly based on voice, the shape of the mouth is also very important. When babies are learning to speak, they pay special attention to the shape of their parents' mouths. If you play a video to babies and deliberately mismatch the voice and the shape of the mouth, the babies will immediately find out and appear very confused. In the early stages of language learning, imitating the shape of the mouth can help learners imitate pronunciation.

People discovered the correspondence between lip shape and speech very early, and used lip shape to assist in understanding language. In ancient times, the deaf relied on lip shape to understand what people were saying, a method called "lip reading". To this day, many schools for the deaf still teach lip reading. People with acquired hearing loss generally do not learn sign language, and some training institutions will also help these people learn lip reading. Using lip reading to assist in listening to speech can help them communicate better. Many deaf people have been using lip reading for a long time, and their lip reading skills are even more advanced. They can look at a person's facial movements and guess what he said.

So, the audience can guess the original lines by looking at the lip movements? You may be disappointed. Even trained lip-reading experts cannot accurately restore the lines. Why is this?

02

Limitations of lip reading

In the 2006 World Cup final, there was a famous foul: French star Zidane suddenly hit Italian player Materazzi's chest with his head. The opponent fell down, and Zidane was sent off with a red card for this foul.

That game was originally the World Cup final that Zidane represented the French national team in. It was also his last game before retirement. His exit in this way surprised the fans.

Image taken from the 2006 World Cup

Combining the video of Zidane's angry exit and his performance at the time, people realized that there was a reason for the foul. Materazzi whispered something to Zidane, which made him lose control. After the game, the two parties had different opinions. Zidane said that Materazzi insulted his sister and mother; Materazzi admitted to the rude words, but denied insulting the other party's family.

The media naturally would not let go of this gossip. They found a lip-reading expert to analyze what Materazzi said from the video at that time. As a result, the results of everyone's interpretation were inconsistent. The following year, Materazzi revealed what he said in an interview, which was also different from the interpretation of the lip-reading expert. Of course, athletes may whitewash their words afterwards, but this example at least shows that people cannot rely on lip reading and cannot guess the original sentence perfectly.

Copyright image, no permission to reprint

Linguists have found that there are about 40 distinguishable pronunciations in English, but many different pronunciations share similar mouth shapes. The same is true for Chinese. You can also try it in front of a mirror. The mouth shapes of Chinese pinyin b and p, k and g, and z, c and s are very similar. This is because when people speak, their throats, tongues and lips must make corresponding movements. However, people can only observe the lips and a little bit of the tongue, and the movement of the throat cannot be seen with the eyes.

When a deaf person is lip-reading, if he knows the topic of the conversation and is familiar with the speaker, he can not only obtain information through lip-reading, but also improve the accuracy based on the context and the other person's speaking habits. However, if he only looks at the lip movements of others from a distance without the support of context, the accuracy of lip-reading will be much lower.

03

After Mute

The lip movements seem easier to recognize?

It seems that it is easy to find that the lip movements do not match when watching the video, but it is not so easy to guess the original lines based on the lip movements alone. However, there is another interesting phenomenon worth mentioning. Some netizens discovered a phenomenon when deciphering the plot: if you watch the video directly, the voice will interfere with the judgment of the lip movements, but if you turn off the video, it will be easier to guess the original lines.

Yes, the information of voice and lip shape will affect each other. Sometimes, as long as you hear the voice, even if the lip shape doesn't match, it won't be too distracting. In recent years, some actors didn't even read out their lines completely during filming of TV series and movies, but relied entirely on post-dubbing, but they were still able to produce works that were not too discordant. Moreover, people also feel when watching dubbed films that the most exquisite dubbing seems to be spoken naturally by the actors, without the feeling of "matching" at all. This is exactly how voice affects the perception of lip shape.

Conversely, lip shape can also affect the perception of speech. Scientists conducted an experiment in which they recorded a video of a person pronouncing the word "ba". Everyone thought everything was normal when they saw the lip shape and heard the sound. Then, the actor's lip shape changed to "fa", but the sound was still "ba". At this time, the sound people heard did not change at all, but when watching the video, they could actually hear a little "fa" sound. This phenomenon is called the "McGurk effect". It shows that when humans talk to others, they automatically integrate the information of speech and lip reading. Changing lip shape can actually affect speech.

04

AI

Why you again?

It seems that the information contained in lip reading is limited, and humans have a limit to understanding speech and lip shape. It is no longer feasible to guess the plot based on lip shape. However, perhaps in the future people will be able to further improve the ability of lip reading, and this will require the help of artificial intelligence. Scientists from Google have found a large number of videos to train computers to improve the accuracy of lip reading. The progress of this research is far less than that of speech recognition, but computers also have a certain degree of accuracy.

Image source: Reference [9]

In addition, an artificial intelligence company called Jali has developed a new technology that allows 3D characters to automatically match their mouths with voice and text input. Some 3D games can provide dubbing in more than a dozen languages, and the expressions and mouth shapes of the characters can correspond to specific languages. If animators were to adjust manually, this would be an impossible task, but with artificial intelligence, exquisite performances can be produced.

Perhaps, in the future, TV dramas will consider using virtual actors to replace real people. After all, the virtual actors' lip movements and lines will always match, and there will be no chance for lip-reading detectives to come out.

Finally, did you solve the lip-reading puzzle at the beginning? Leave a message and tell us your answer!

References:

[1] Sam Loyd's Cyclopedia of Puzzles, 1914

[2] Dodd B(1976). Lip reading in infants: attention to speech presented in- and out-of-synchrony. Cognitive Psychology Oct;11(4):478-84

[3] Xu Cheng. (2013). A review of lip reading research: from the deaf to the normal. Journal of East China Normal University (Educational Science Edition), 31(1): 56-61.

[4] Lei Jianghua, Zhang Fengqin, Fang Junming. (2004). Experimental study on speech recognition of lip-reading Chinese characters by deaf students under word-based conditions. Chinese Journal of Special Education, 53(11), 37-39.

[5] Red card puts an end to the Zinedine Zidane headbutt incident. China News Network, 2007.3.6

[6] Auer, ET (2010). "Investigating speechreading and deafness". Journal of the American Academy of Audiology. 21 (3): 163–8.

[7] McGurk H., MacDonald J. (1976). "Hearing lips and seeing voices". Nature. 264 (5588): 746–748.

[8] Luo Xiaoxiao, Kang Guanlan, Zhou Xiaolin. (2018). Influencing factors and neural basis of the McGurk effect. Advances in Psychological Science, 2018, 26(11): 1935-1951

[9] Chung, JS, Senior, A., Vinyals, O., & Zisserman, A. (2017). Lip Reading Sentences in the Wild. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3444–3453.

[10] Cyberpunk 2077's dialogue was lip-synced by AI. Engadget, 2020.10.20

Author: Chen Chao, Master of Cognitive Neuroscience, Beijing Normal University

Reviewer: Tao Ning, Associate Researcher, Institute of Biophysics, Chinese Academy of Sciences

The cover image and the images in this article are from the copyright library

Reproduction of image content is not authorized