Magic Leap's core technology revealed

Figure 12. Stanford’s light field camera: 16x8 multi-camera array.

As digital camera technology matures, pinhole cameras are becoming smaller and smaller and can be densely integrated, thus reducing the size of light field cameras. However, the size of the lens cannot be reduced, as shown in Figure 13.

Fig. 13. Stanford Light Field Camera: Camera Array.

The real breakthrough comes from bionics. Many insects have compound eyes, which acquire light field information.

Fig. 14. Insect compound eyes: light field cameras.

Humans imitated insects and created lenses similar to compound eyes, as shown in Figure 15, with dozens of small lenses integrated on a large lens. With the improvement of optical technology, people have created thousands of tiny lenses integrated on a plastic film. Based on this idea, Stanford doctoral student Wu Ren founded the light field camera company Lytro.

Figure 15. Prototype of artificial compound eye made by Adobe.

Traditional cameras need to focus first and then take pictures. The slogan of Lytro camera is "take pictures first, then focus". Because of the light field information obtained by Lytro camera, users can synthesize 4D photos of different angles and depths from the 4D light field.

Figure 16. Lytro camera.

Wedding photography as shown in Figure 17: In the same light field photo, we can focus on the groom who is close to the camera; we can also focus on the bride who is far away from the camera.

Figure 17. Lytro wedding photo: The same light field photo can be focused on different areas. The left frame focuses on the groom; the right frame focuses on the bride.

Light field display The traditional display method, screen, LCD/LED, only retains the geometric information and color information of the intersection point where the ray passes through the screen, but does not retain the directional information of the ray. The screen is diffusely reflective, and all rays emitted from a certain point on the screen are of the same color, while light field display requires different rays starting from the same point to have different colors, as shown in Figure 18. Light field display is the core technology of Magic Leap.

Figure 18. Display mode comparison: the left picture is a traditional screen, where all rays passing through a point are the same color; the right picture is a light field display, where different rays passing through a point are different colors.

USC's light field display The University of Southern California proposed and produced a light field display device, as shown in Figures 19 and 20. There is a glass cabinet with four transparent sides. In the middle of the cabinet is a mirror with an angle of 45 degrees to the horizontal plane. A high-speed projector is installed on the top of the cabinet. The projector projects vertically downward, and the light is reflected by the mirror and then emitted horizontally. At the same time, the mirror rotates at high speed. A ghostly transparent human head is suspended in the air. When we walk around the cabinet, we can see all sides of the head, and the head winks at you.

Figure 19. USC Light field display, a floating human head.

Figure 20. USC Light field display for teleconferencing system.

Figure 21. USC Light Field display patent drawing.

FIG21 shows the principle of this light field display device. The 45-degree tilted mirror (114) is driven to rotate by the motor (115), and the graphics processor (130) generates an image and transmits it to the high-speed projector (120). The projector projects the image onto the mirror, which is then reflected and projected horizontally to all directions. In this way, after strict synchronization control, we can display a three-dimensional light field. This device is bulky and expensive, and the high-speed rotating mirror reduces the stability of the system. Any mechanical vibration will affect the light field display effect.

Magic Leap Light Field Display - Flashlight Explanation Magic Leap's core technology is a special light field display device: Fiber Optic Projector. The laser propagates in the optical fiber and is emitted at the end of the fiber, and the output direction is tangent to the fiber. By changing the shape of the fiber in three-dimensional space, especially changing the tangent direction at the fiber end, we can control the direction of the laser emission. This is like holding a flashlight and changing the position and direction of the flashlight to change the direction of the output light column. If we shake our wrist quickly, the light column emitted by the flashlight draws a cone in the air, and this cone hits a wall to form a circumference. By quickly changing the amplitude of the wrist shake, we can control the radius of the circumference, thereby obtaining a series of concentric circles, which cover a disk. If the color of the flashlight's light column changes, we draw a colored disk on the wall. In this way, by quickly shaking a flashlight, we get an image, or cover a cluster of rays. Suppose there are many people standing in different spatial positions, and each of them shakes a flashlight, then we get a light field. This is the principle of Magic Leap's light field display device: the fiber optic projector.

Figure 22. Magic Leap's flashlight.

Figure 22 shows Magic Leap's flashlight. The actuator (206) is equivalent to a person's wrist, and the optical fiber (208) is equivalent to the flashlight. The actuator causes the fiber tip to vibrate periodically, and the fiber tip spirally draws a series of concentric circles. The laser is output through the lens system, drawing a cluster of rays in the air. It is projected onto a plane to illuminate a disk. By synchronously changing the color and intensity, a fiber uses time-sharing technology to obtain an image, as shown in Figure 23.

Figure 23. One fiber produces one image using time-sharing technology.

In Magic Leap's fiber light projector, there are many optical fibers assembled into a two-dimensional array. Each fiber is equivalent to a pinhole camera, and the two-dimensional camera array generates a light field.

Advantages of light field display Compared with binocular stereo vision, light field display has many advantages. There are two ways for humans to obtain three-dimensional depth information, "shape from stereo" and "shape from focus". We use two eyes to look at the same object, and the same point in three-dimensional space is projected onto different pixels on the left and right retinas. Our human brain can reversely calculate the corresponding spatial rays through the pixels on the retina, thereby obtaining the intersection of the two rays and obtaining depth information. This process is "shape from stereo". When each of our eyes looks at an object, the brain automatically adjusts the curvature of the eye's lens so that the object is clearly imaged on the retina. Adjusting the muscle tension of the lens allows the brain to calculate the depth information of the object, which is the so-called "shape from focus". When watching the 3D version of "Avatar", we only used "shape from stereo", and the focal length of the eyes has been fixed because the distance from the eyes to the screen does not change, so there is no "shape from focus" process. However, after a long evolution of humans, these two processes are naturally closely linked. Artificially separating them will cause dizziness. On the contrary, if we use light field display technology, we need both “shape from stereo” and “shape from focus” at the same time, so you won’t feel dizzy when watching. Light field display technology is more natural and healthy.

Challenges of light field display As the beginning of a revolution, Magic Leap's technology faces many challenges. The most direct one is that traditional display technology only needs to calculate a two-dimensional slice in a four-dimensional light field, while light field display needs to calculate the entire four-dimensional light field, and its computational complexity increases by several orders of magnitude, which is one of the technical bottlenecks. At the same time, precise control of mechanical components so that each fiber vibrates stably and naturally, and the vibration pattern must be synchronized with data transmission, and this vibration is not affected by external noise, also requires incredible technology.

It has been 20 years since the concept of digital holographic light field was proposed to the investment frenzy of Magic Leap, and the development history of digital holographic technology is even longer. Light field is essentially geometric optics, while digital holography is wave optics. At present, digital holographic technology is becoming more and more mature. With the invention of blue laser, color digital holographic technology has become possible. The current bottleneck of development is that the amount of calculation is huge, far exceeding the calculation of light field, and the second is that a special kind of crystal is required in digital holographic display, and the refractive index of each pixel can be controlled by voltage. At present, this kind of optical device is still expensive and small in size. We believe that with the widespread acceptance of light field technology, digital holographic technology will also develop rapidly.

The inspiration of light field technology The historical development of light field technology shows us that disruptive technological revolutions often originate from basic science and non-commercial academic circles. It often takes decades for a technology to mature in academia and become a force to be reckoned with in the business world. Magic Leap's technological breakthrough came from the use of endoscopy technology, which shows the importance of cross-border scientific research.

I look forward to the day when all TV and movies are shot with light field cameras, and the audience can dynamically choose the viewing angle at will. Maybe this day will take another twenty years, or maybe only three to five years. I believe that in the near future, all photos on Taobao will be replaced by light field photos, and the Magic Leap helmet will become the standard for every online shopper. ( This article was first published in "Lao Gu Talks Geometry" and is reprinted with the author's permission.)

As a winner of Toutiao's Qingyun Plan and Baijiahao's Bai+ Plan, the 2019 Baidu Digital Author of the Year, the Baijiahao's Most Popular Author in the Technology Field, the 2019 Sogou Technology and Culture Author, and the 2021 Baijiahao Quarterly Influential Creator, he has won many awards, including the 2013 Sohu Best Industry Media Person, the 2015 China New Media Entrepreneurship Competition Beijing Third Place, the 2015 Guangmang Experience Award, the 2015 China New Media Entrepreneurship Competition Finals Third Place, and the 2018 Baidu Dynamic Annual Powerful Celebrity.