Summary: All the AR/VR/3D technologies and developments you should know

We are all so tired of hearing the words 3D and "hologram". From Panasonic's release of the first 3D TV system in 2010 to the current virtual reality and augmented reality technologies, these words have been integrated into our popular culture and have become more and more the focus of our attention. After all, the real world is three-dimensional, so why should we limit our experience to a flat screen?

[[170781]]

The transition from 2D to 3D is a natural process, just as black-and-white movies and television switched to color in the 1950s, but in many ways, the transition from 2D to 3D may be even more significant.

The significance of 3D is not just to present a more credible real world, but to make the digital world a part of real life and make every detail of it as real as the real world.

This coming watershed will have a profound impact on the way we work, learn, socialize, and play. We will be able to transcend the physical limitations of the real world and live in a world that is only as big as our imaginations.

Of course, this shift will also affect the devices we use and how we interact with machines. That’s why companies like Google, Facebook, and Apple are rushing to seize the 3D market as quickly as possible: whoever wins the 3D war will control the next generation of user interaction.

But now it is just a vision. Although some people have tried to open up the 3D market before, we still have to rely on narrow screens to enter the electronic world. Why? Because there are many shortcomings in the current 3D, and the current level of technology is not enough to realize a real and credible world, so consumers are still waiting.

Next, YiViAn would like to give you a comprehensive introduction to 3D technology from a user's perspective, the challenges facing 3D products such as VR/AR headsets and naked-eye 3D displays, and how these products will become an intuitive interactive interface between us and the digital world.

Why do we need 3D images?

Before we delve into 3D technology, we need to understand the principles of this technology.

Try squinting through one eye to thread a needle, and you'll find it's harder than you think. Evolution has altered our 3D perception, allowing us to grasp information about the real world more quickly and accurately.

How do we get depth perception at a physiological level? This is a complex subject. The human eye and brain can sense many depth cues, and when these cues appear in the real world, they reinforce each other to form a clear 3D environment image in our minds.

Stereoscopic vision (binocular parallax)

If we want to use a 3D illusion to trick the visual system and brain, we may not be able to reproduce all the depth cues faithfully, so the 3D experience will be biased. Therefore, we need to understand some of the main depth cues.

Believe it or not, your two eyes see things differently. They see objects from slightly different angles, so the images they receive on your retinas are different.

When we look at distant objects, the parallax between the left and right eyes is relatively small. But when the object is closer, the parallax increases. This difference allows the brain to measure and "feel" the distance of the object, triggering the perception of depth information.

Most current 3D technologies rely on stereoscopic vision to trick the brain into believing it is sensing depth information. These technologies present different images to each eye, and if you've seen 3D movies (such as Avatar and Star Wars), you've probably experienced this effect with 3D glasses.

More advanced "autostereoscopic" display technology can project different images into space in different directions so that the eyes can receive different images without the need for glasses.

Stereoscopic vision is the most obvious depth cue, but it's not the only one. Humans can actually perceive depth using just one eye.

Close one eye and place your index finger in front of your other eye and hold it there. Now move your head slightly up and down and side to side. You will see that the background appears to be moving relative to your finger. More precisely, it will appear that your finger is moving faster than the background.

This phenomenon is called motion parallax, and it's the reason you can perceive depth with one eye. Motion parallax is an important effect if you want to provide a realistic 3D experience, because the viewer is very slightly displaced relative to the screen.

Stereoscopic vision without motion parallax still gives you a sense of depth, but the 3D image will be distorted. For example, buildings in a 3D map will begin to appear distorted, and objects in the background will appear to be deliberately obscured by objects in the foreground, which can be quite uncomfortable if you look closely.

Motion parallax is closely related to our visual perception - in fact, the effect that we see near things appear larger than far things is a depth cue.

Even if you were sitting perfectly still (no motion parallax) and closed one eye (no stereopsis), you would still be able to distinguish between distant and near objects.

Try the finger experiment again. Place your index finger in front of your eyes and keep it still. Then focus on it. As the glasses focus, you will notice that the background becomes blurry. Now focus on the background. You will notice that your finger becomes blurry and the background becomes clear. This is the same as how modern cameras work. Our eyes have the ability to change focus.

The eye achieves its zoom function by changing the shape of the lens through the contraction of the ciliary muscle.

So how do the ciliary muscles of the eyes know how much force they should use to contract? The answer is that our brain has a feedback loop, and the ciliary muscles will continue to contract and relax until the brain gets the clearest image. This is actually an instantaneous action, but if the ciliary muscles are adjusted too frequently, our eyes will feel very tired.

Only objects within two meters can trigger the movement of the ciliary muscles. Beyond this distance, the glasses will begin to relax and focus on infinity.

Conflict between visual convergence and visual accommodation

When your eyes focus on a nearby point, they actually rotate in their eye sockets. The extraocular muscles automatically stretch when you focus, and the brain can sense this movement and interpret it as a depth cue. If you focus on an object within 10 meters, you will feel your eyes converge.

So when we see the world with two eyes, we use two different sets of muscles. One set of muscles is responsible for making the two eyes converge (converge) to the same focus, while the other set of muscles is responsible for adjusting the clarity of the retinal image. If the eyes do not converge properly, we will see double images. If the vision is not adjusted properly, we will see blurry images.

In the real world, vergence and accommodation go hand in hand. In fact, the nerves that trigger these two responses are connected.

However, when watching 3D, virtual reality or augmented reality content, we usually focus on a specific location (such as the screen in a 3D cinema), but the information received by both eyes will make the eyes converge to another distance (such as a dragon rushing out of the screen).

At this time, our brain will find it difficult to coordinate these two conflicting signals, which is why some viewers feel eye fatigue or even nausea when watching 3D movies.

Do you remember those red-cyan 3D glasses made of cardboard? These "filter glasses" allow the left eye to see only red light and the right eye to see only blue light. Normal 3D images superimpose a red image on the left eye and a blue image on the right eye.

When we look through filter glasses, the visual cortex of the brain will fuse the images we see into a three-dimensional scene, and the colors of the picture will also be corrected.

All current glasses-based 3D effects, including virtual reality and augmented head displays, use this working principle - creating stereoscopic vision by physically separating the images seen by the left and right eyes.

When you go to the movies today, you get a pair of 3D glasses that look nothing like polarized glasses. Instead of filtering the image based on color, they filter the image based on the polarization of light.

We can imagine photons as vibrating entities that vibrate either horizontally or vertically.

On a movie screen, a special projector creates two overlapping images. One image sends only horizontally vibrating photons toward the viewer, while the other sends vertically vibrating photons. The polarized lenses of the glasses ensure that the two images reach the appropriate eye.

If you have a pair of polarized glasses (don't steal them from the theater), you can use them to see the polarization of the real world. Hold the glasses about 2 feet away from your line of sight and look through them at a car windshield or the surface of water. As you rotate the glasses 90 degrees, you should see the glare coming through the lenses appear and disappear. This is how polarized glasses work.

Polarized glasses allow the full color spectrum to enter the eye, which will enhance the quality of the 3D image.

In terms of 3D image quality, these glasses can only provide stereoscopic vision as a depth cue. So although you can see the depth of the image, if you leave your seat and walk around the theater, you will not be able to see the objects around you clearly, and the background will move in the opposite direction of the motion parallax, that is, it will follow you.

A more serious problem is the lack of accommodation support, which leads to a conflict between vergence and accommodation. If you stare at a dragon approaching you on the screen, you will quickly experience extreme visual discomfort.

This is also why dragons in movies only fly by very quickly - just to create a startling effect and avoid irritating your eyes.

If you have a 3D TV at home, the glasses that come with it may not be polarized, but use "active shutter" technology. This TV will alternately display the image for the right eye and the left eye, and the glasses will block the corresponding eye in sync with the frequency of the TV. If the switching frequency is fast enough, the brain can mix the two signals into a coherent 3D image.

Why not use polarized glasses? Although some TVs use this technology, polarized images in different directions must come from different pixels, so the resolution of the image seen by the audience will be halved. (But the two images in the cinema are projected on the screen separately, and they can overlap each other, so there is no loss of resolution.)

Active shutter 3D glasses generally only support one depth cue, stereoscopic images. Some more advanced systems use tracking technology to adjust the content, supporting motion parallax by tracking the viewer's head, but only for one viewer.

Virtual reality is a whole new type of 3D rendering. The technology has been getting a lot of attention lately. Many people have already experienced virtual reality headsets, and more people will be able to experience this technology in the coming years. So, how does virtual reality work?

Instead of filtering content from an external screen, VR headsets generate their own binocular images and present them directly to the corresponding eye. VR headsets typically contain two microdisplays (one for the left eye and one for the right eye), whose realistic avatars are magnified and adjusted by optical components and displayed at a specific location in front of the user's eyes.

If the display resolution is high enough, the image can be magnified at a higher ratio, so the user will see a wider field of view and the immersive experience will be better.

Current virtual reality systems like the Oculus Rift track the user's position to add motion parallax effects to stereoscopic vision.

Current versions of VR systems do not support visual accommodation and are prone to conflicts between visual convergence and accommodation. This technical challenge must be solved to ensure a good user experience. These systems cannot fully solve the problem of eye movement (limited range and distortion), but this problem may be solved in the future through eye tracking technology.

Likewise, augmented reality headsets, which blend the digital and real worlds, are becoming more common.

To ensure realism, the augmented reality system not only needs to track the user's head movement in the real world, but also take into account the actual 3D environment in which the user is located.

If recent reports about Hololens and Magic Leap are true, the field of augmented reality has now taken a huge leap forward.

Just as our two eyes see the world differently, real-world light enters the pupil from different directions. In order to trigger visual accommodation, near-eye displays must be able to simulate light independently emitted from all directions. This light signal is called a light field, and it is key to the future of virtual and augmented reality products.

Until this technology is implemented, we will just have to continue to suffer headaches.

These are the challenges that head-mounted displays need to overcome, and another problem they face is integrating into society and culture (such as what happened to Google Glass). If we want to display the world from multiple perspectives at the same time, we need to use a naked-eye 3D display system.

How can we experience 3D images without wearing any device?

Seeing this, I think you should understand that if you want to provide stereoscopic vision, the 3D screen must project image content from different perspectives in different spatial directions. In this way, the audience's left and right eyes will naturally see different images, thereby triggering depth perception, so this system is called "autostereoscopic display".

Because 3D images are projected onto a screen, the automatic stereoscopic display itself can support visual convergence and accommodation as long as the 3D effect is not too exaggerated.

But this does not mean that this system will not cause eye discomfort. In fact, this system also has another problem in the conversion of different visual areas.

The images of this system will have frequent jumps, brightness changes, dark bands, stereoscopic vision interruptions and other problems. The worst thing is that the content seen by the left and right eyes is reversed, resulting in completely opposite 3D feelings.

So how should we solve these problems?

3M began commercializing the technology in 2009.

The prismatic film will be inserted into the film stack of the ordinary LCD screen backlight, and can be illuminated by light from the left or right direction respectively. When the light source comes from the left, the LCD image will be projected on the right area, and when the light source comes from the right, the image will be projected on the left area.

Rapidly switching the direction of the light source and changing the content seen by the left and right eyes on the LCD at the same frequency can produce stereoscopic vision, but the premise is that the screen needs to be placed directly in front of the viewer. If viewed from other angles, the stereoscopic effect will disappear and the image will become flat.

Because the viewing range is very limited, this display technology is commonly referred to as "2.5D".

A cover layer with many small holes is added to a 2D screen. When we observe the screen through these small holes, we cannot see all the pixels below. The pixels we can see actually depend on the viewing angle. The left and right eyes of the observer may see different groups of pixels.

The concept of "parallax barrier" was discovered a century ago, and Sharp first put it into commercial application ten years ago.

The technology has been refined to now feature a switchable barrier, which is essentially another active screen layer that can create a barrier effect or become transparent, reverting the screen to 2D mode to display the full resolution.

Back in 2011, the HTC EVO 3D and LG Optimus 3D made headlines as the world's first 3D-enabled smartphones, but they were just another example of 2.5D technology that only provided 3D effects within a very narrow viewing angle.

Technically, the parallax barrier can be extended to form a wider viewing angle. But the problem is that the wider the viewing angle, the more light you need to block, which will cause excessive power consumption, especially for mobile devices.

A layer of miniature convex lens is covered on a 2D screen. We all know that convex lenses can focus parallel light from distant light sources. Children also use this principle to point at things with a magnifying glass.

These lenses collect the light from the screen pixels and transform it into a beam with a direction, a phenomenon we call collimation.

The direction of the light beam changes with the position of the pixel under the lens. In this way, different pixels will follow the light beam and project in different directions. The lenticular lens technology can ultimately achieve the same effect as the parallax barrier (both use the principle of different groups of pixels seen at different spatial positions), but the lenticular lens will not block any light.

So why don't we see lenticular lens 3D screens on the market?

It's not that no one has tried; Toshiba released its first-generation system in Japan in 2011. But the screens exhibited some unacceptable visual artifacts when viewed closely, largely due to problems with the lenses.

First, screen pixels are usually composed of a small emissive area and a large "black matrix", the latter of which is non-luminous. After being processed by the lens, the emissive area and black matrix of a single pixel will be deflected to different directions in space. This will cause very obvious black areas to appear in the 3D picture. The only way to solve this problem is to "defocus" the lens, but doing so will cause interference between different viewing angles and the image will become blurred.

Second, it is difficult to achieve proper collimation at wide viewing angles with just a single lens. This is why camera lenses and microscopes use compound lenses rather than single lenses. Therefore, a cylindrical lens system can only observe true motion parallax at a narrow viewing angle (about 20 degrees). Beyond this range, the 3D image will repeat itself, it will feel like you are looking at it from the wrong angle, and the image will become increasingly blurry.

The narrow field of view and poor visual transitions are the biggest drawbacks of lenticular screens. For TV systems, current lenticular technology is acceptable if the viewer automatically adjusts their head and doesn't move around.

However, in usage scenarios such as mobile phones and cars, the head will definitely move, so it is difficult for the cylindrical lens system to achieve the ideal effect.

So, how should we design a naked-eye 3D vision system with a wide field of view and smooth transitions?

If I know the relative position of your eyes to the screen, I can calculate the corresponding viewing angle and try to adjust the image to the direction of your eyes. As long as I can detect the position of your eyes quickly and have an equally fast image adjustment mechanism, I can ensure that you can see stereoscopic vision at any viewing angle, and smooth motion parallax rendering.

This is how eye-tracking autostereoscopic screens work.

The benefit of this approach is that the screen only needs to render two perspectives at any given moment, which keeps most of the screen pixels. From a practical point of view, the eye tracking system can be used in conjunction with the current parallax barrier technology to avoid the optical illusions produced by the cylindrical lens system.

But eye tracking isn’t a panacea. For one thing, it can only support one viewer at a time, and it requires additional cameras on the device and complex software running in the background to predict eye positions.

Size and power consumption are not a big issue for TV systems, but they can have a huge impact on mobile devices.

Furthermore, even the best eye-tracking systems can experience delays or errors, often due to lighting changes, eyes being blocked by hair or glasses, the camera detecting another pair of eyes, or the viewer moving their head too quickly. When this system fails, the viewer can see very uncomfortable visual effects.

The latest development in glasses-free 3D technology is diffraction "multi-view" backlit LCD screens. Diffraction is a property of light where light is deflected when it encounters sub-micron objects, so this means we are getting ready to enter the realm of nanotechnology.

You read that right, it’s nanotechnology.

Ordinary LCD screen backlights emit randomly distributed light, that is, each LCD pixel emits light into a wide space. However, diffraction backlights can emit light in a uniform direction (light field), and a certain LCD pixel can be set to emit unidirectional light. In this way, different pixels can send their own signals in different directions.

Like lenticular lenses, diffractive backlighting can make the most of incoming light. But unlike lenticular lenses, diffractive backlighting can handle both small and large light emission angles, as well as fully control the angular spread of each viewing angle. If designed properly, there will be no dark areas between viewing angles, and the generated 3D image will be as clear as a parallax barrier.

Another feature of the diffraction method is that the light adjustment function does not affect the light that passes directly through the screen.

In this way, the transparency of the screen can be completely preserved, so the screen can add a normal backlight underneath and return to full-pixel 2D mode. This paves the way for the development of see-through naked-eye 3D display technology.

The main problem with diffraction methods is color consistency. Diffractive structures usually send different colored light in different directions, and this dispersion phenomenon needs to be offset at the system level.

However, the development of 3D technology will not stop at seeing 3D images, it will open up a whole new paradigm of user interaction.

3D experiences will soon be of a quality suitable for consumption, and we won't have to worry about our 3D experience being interrupted.

In virtual reality and augmented display systems, this means increasing the field of view, supporting visual accommodation, and making the system more responsive to rapid head movements.

In naked-eye 3D display technology, this means providing users with enough freedom of movement and avoiding various visual artifacts such as 3D loss, dark areas, visual jumps, or motion parallax delays.

Once the illusion becomes believable, we tend to forget about the technology behind it and treat the VR world as if it were real, at least until we run into a real wall. If we want to realize this fantasy, we need to consider that the real world has physical responses.

When we convert electronic data into light signals that we can perceive in the real world, we need to send the body's real response data back to the digital world for interaction.

In virtual reality and augmented reality headsets, this can be achieved through sensors and cameras worn by the user or placed in the environment. We can expect to see smart clothing with sensors in the future, but they will likely be bulky.

On a 3D screen, this task can be done directly by the screen. In fact, the Hover Touch technology developed by Synaptics has been able to achieve non-touch finger sensing. Soon, we will only need to move our fingers to interact with holographic images in mid-air.

Once the electronic world understands the user's response mechanism, the two can merge more naturally. In other words, digital signals can open a door on a wall before we hit it.

But wouldn’t it be even better if we could actually see and touch this wall in the virtual world? But how do we bring touch into this world?

This question falls into the realm of tactile feedback, which you've experienced if you've ever turned on vibrate mode on your phone.

For example, gloves and other clothing equipped with vibration units, or using weak electrical signals to simulate skin, if adjusted correctly, different parts of your body will be able to feel touch that matches the visual effects.

Of course, not everyone is suited to wearing a costume full of wires or feeling the electrical current.

For screen-based devices, ultrasonic haptics allow you to touch the screen directly in mid-air, without the need for any smart clothing. This technology emits ultrasonic waves from all sides of the screen, and the intensity of these ultrasonic waves can be adjusted according to the user's finger movements.

Believe it or not, these enhanced sound waves are strong enough to be sensed by your skin, and companies like Ultrahaptics are already preparing to bring this technology to market.

Although virtual reality and augmented display headsets are becoming more and more common today, they still have many limitations in terms of mobility and social interaction, making it difficult to achieve a fully interactive 3D experience. 3D screens using finger haptic technology will be able to overcome this obstacle and allow us to interact with the digital world in a more direct way in the future.

In a recent blog post, I called this platform Holographic Reality and described how holographic displays can be used in all aspects of our daily lives. In the future, all windows, tables, walls and doors will have holographic features, and the communication devices we use in offices, homes, cars and even public places will have holographic elements. We will be able to access virtual worlds anytime and anywhere without wearing a headset or connecting to cables.

Over the past five years, there has been tremendous development in the fields of 3D technology, display technology, and head-mounted displays.

With such rapid technological development, it is not difficult to imagine that in the next five years we will communicate, learn, work, shop or entertain in a panoramic real world. This will be achieved through an advanced 3D ecosystem consisting of products such as head displays and 3D screens.