How to identify AI-forged portraits? Look at the eyes!

The painting below is titled "False Mirror" and was created by Belgian surrealist painter Magritte (1898-1967) in 1928. It is currently in the collection of the Museum of Modern Art in New York, USA.

Figure 1: Belgian painter Magritte’s work “False Mirror” [1]

From an artistic perspective, the painting provokes reflections on the nature of visual perception, with each viewer having their own interpretation.

However, from a scientific point of view, the human eye in the painting is actually very consistent with the real human eye in structure. The white area corresponds to the milky white sclera in the human eye, and the blue and white area corresponds to the iris of the human eye, which is usually called the eyeball (or eyeball). The color of the iris is darker than the sclera, and it varies depending on the race, and may be brown or blue. But the darkest part of the eye is the pupil in the center of the iris, which absorbs a lot of light and looks pitch black. The eyes in the painting also have several basic components of the human eye.

Figure 2: The external structure of the human eye: sclera, iris, pupil, cornea [2]

In the painting, the iris is partially filled with blue sky and white clouds, while the black pupil is still clearly visible. This representation is inconsistent with reality because we usually cannot judge what someone sees just by looking at their eyes.

Figure 3: Internal structure of the human eye[3]

The working principle of the human eye is similar to that of a camera: the pupil acts as an aperture to control the amount of light entering, while the retina acts as a photosensitive element to receive light signals. Since the pupil is very small and most of the light is absorbed by the tissues inside the eye, it is difficult for us to directly observe the image formed on the retina from the outside.

Figure 4: Analogy between the human eye and a camera

Of course, sometimes we can also get clues from the outside of the eye. One example is the "red eye" phenomenon that often occurs when taking photos with a flash at night. This is because the pupil is enlarged in a dark environment, and the strong light of the flash is reflected by the congested choroid in the eye, causing the eyes to appear red in the photo.

The optical system of the human eye is mainly composed of the cornea and the lens, both of which are transparent, similar to the lens group of a camera. The cornea is located at the outermost layer of the eye and has a fixed shape; while the lens can change its shape through the action of the ciliary muscle, thereby adjusting the focal length so that objects at different distances can be clearly imaged on the retina. If the adjustment function of the lens group fails, the image of the distant object cannot be focused on the retina, but can only be focused in front of the retina, the image becomes blurred, and myopia occurs in the eye.

Back to what I just said, it is usually impossible to judge what someone is looking at through a photo of his eyes, but it is not absolute. It is still possible in some cases, and this depends on the cornea. The cornea is an ideal optical device. Its smooth surface can refract and reflect light, and it can act as a miniature mirror under appropriate conditions. The "mirror image" formed by this reflection is usually small and unclear, and will also be interfered by the iris texture (the unique texture of the iris is similar to a fingerprint and can be used for identity recognition).

Figure 5: Recovering a clear photo from the “mirror image” reflected by the cornea of ​​the eye [2]

Researchers have designed complex computer algorithms to try to restore a clear image of the surrounding environment from the reflection of the eye[4]. They established a mathematical model, corrected the position and angle, and removed the interference of the iris texture, and finally successfully reconstructed the image of the surrounding environment. Researchers also found that the "mirror image" of the cornea has unique advantages. The curved surface of the cornea can provide a wide-angle panoramic view, while the movement of the eyeball and the movement of the body can capture multiple perspectives. These characteristics make it possible to reconstruct a three-dimensional scene from the reflection of the cornea, achieving an effect that is difficult to achieve with ordinary cameras, and realizing the effect of "seeing the world through someone else's eyes."

Figure 6: Reconstructing the 3D scene in front of us from a video clip containing the reflection of the human eye [4]

It is important to note that these studies were conducted under ideal laboratory conditions, and there are still many difficulties in applying this technology in everyday settings.

The reflection of the cornea can also help us distinguish the authenticity of photos or videos . With the development of image processing technology and artificial intelligence, fake images and videos have become more and more realistic, causing social concerns and a crisis of trust. To meet this challenge, researchers have developed various fake image and video detection technologies. Among them, analyzing the eye details in the image is an effective method.

Of the two photos below, one is a fake. Can you tell which one it is? The clue to solving a cold case comes from a pair of eyes.

Figure 7: Two photos, one real and one fake: Eye corneal reflection images to help you distinguish [5]

Research has found that due to the close proximity of a person's eyes, the corneal reflection images of the left and right eyes in a real photo are usually very similar. Counterfeiters often ignore this detail, resulting in inconsistencies between the two in fake photos. Figure 7 (right) is a typical example. Using this feature, researchers achieved a 94% success rate in distinguishing real and fake photos [5].

In addition, the shape of the pupil can also be used to distinguish the authenticity of a photo [6]. The two photos in Figure 8 are also one real and one fake. The trick to distinguishing the two is that the shape of the pupil of the human eye in the normal photo on the left is usually a regular circle or ellipse, while the shape of the pupil in the AI-forged photo on the right is irregular.

Figure 8: Two photos, one real and one fake: How to tell the difference between pupil shapes [6]

In-depth research on the structure and function of the eyes has not only deepened our understanding of the visual system, but also promoted the development of image processing and information security technology. A lot of inspiration in scientific research and technological applications comes from a pair of eyes.

References

[1] https://www.wikiart.org/en/rene-magritte/the-false-mirror-1928

[2] K. Nishino and SK Nayar, The World in an Eye, 2004 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2004), 444-451 (2004)

[3] https://commons.m.wikimedia.org/wiki/File:Schematic_diagram_of_the_human_eye_zh-hans.svg

[4] H. Alzayer, K. Zhang, B. Feng, C. Metzler, J.-B. Huang, Seeing the World through Your Eyes, arXiv:2306.09348 (2023)

[5] S. Hu, Y. Li and S. Lyu, "Exposing GAN-Generated Faces Using Inconsistent Corneal Specular Highlights," IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2021), 2500-2504 (2021)

[6] H. Guo, S. Hu, X. Wang, M. -C. Chang and S. Lyu, "Eyes Tell All: Irregular Pupil Shapes Reveal GAN-Generated Faces," 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2022), 2904-2908 (2022)

Planning and production

Source: Chinese Optics

Author: Jiao Shuming Guangdong-Hong Kong-Macao Greater Bay Area Quantum Science Center

Reviewer: Gao Hui, Huazhong University of Science and Technology

Editor: Zhong Yanping

Proofread by Xu Lailinlin

The cover image and the images in this article are from the copyright library

Reprinting may lead to copyright disputes