Virtual reality touches the virtual world in reality: the technological revolution behind the visual feast

As early as 50 years ago, virtual reality has appeared, and now it has the potential to change the human visual experience. In the world reconstructed by virtual reality, we not only have senses such as sight, hearing, touch, and smell, but they will become more and more real, no different from the real world. Corresponding to virtual reality, there are also augmented reality and mixed reality, which are changing our lives while changing the human visual experience. We can use them for shopping, simulated driving, and even use them to treat diseases. Although virtual reality technology still has many defects, it is destined to affect each of us in the near future. This article comes from the sharing of Yang Xubo, a professor at the School of Software, Shanghai Jiaotong University, at the "Xinmin Science Cafe" event recently.

More than 50 years ago, American cinematographer Morton Heilig invented a machine called Sensorama, which was like a large game console today. When we put our heads into this machine, we not only had 3D vision, but also could smell and hear sounds. Due to Heilig and the machine's own shortcomings, such as no interactivity and only relying on pre-made images for playback, the Sensorama machine did not receive a good response at the time.

[[170452]]

The development of head-mounted displays can be traced back to the 1960s. In 1968, Ivan Sutherland, a scientist with a doctorate from the Massachusetts Institute of Technology, made a head-mounted display called the Sword of Damocles in the laboratory. This display uses CRT display technology, which can track the movement of the head. It can sense which direction you look and calculate the picture you see in real time. However, computer capabilities were very poor at that time, and the display pixels were very low. Sutherland also wrote a very famous paper "Ultimate Display" at the time. He envisioned that future displays should be able to distinguish between reality and virtuality. When we put this on, we will see another world. In this virtual world, our feelings are real, a bit like what is described in "The Matrix". But in fact, even now, we are still far from his vision.

As for the idea of ​​head-mounted display, it can be traced back to 1613. In the era of Galileo, there was a design of a helmet telescope. In this way, you can see things far away right in front of you, and you can also have a good sense of immersion. Today's virtual reality technology helmet Oculus looks similar to these early head-mounted display devices, but it is very cheap now. In the past, such devices were very expensive and out of reach for ordinary people. In 2006, Toshiba launched such a device, but the design was a failure and it looked like a monster when worn. In 2014, an innovative company called Oculus made virtual reality popular again, not only because of the revolutionary breakthrough in technology, but also because of its low price, allowing ordinary people to buy the development version of the head-mounted display for only $300 to $350. Facebook spent $2 billion to acquire this company, which also made people look at it with admiration.

Samsung has also designed a cheap head-mounted display Gear VR, which can be used when a Samsung phone is placed in this box. Google's cardboard glasses Cardboard are even cheaper. We can even use Google's open design drawings, buy two lenses, and make them ourselves with cardboard. Google mainly promotes Android phones. When the phone is installed with applications, the phone will be divided into two screens, and the left and right eyes can see different pictures, creating a three-dimensional effect. When the information we rotate is detected by the phone's sensor, we can see the interactive effect, but this is less comfortable than Oculus because the distance is difficult to adjust, and the computing power of mobile phones is much worse than that of PCs, but it can still achieve a good sense of immersion.

Virtual reality technology: touching the virtual world in reality

Virtual Reality (VR) is abbreviated as VR. These two words seem contradictory when combined. One is virtual and the other is real, but they are both embodied in a virtual reality environment. A virtual world is generated by a computer, and people enter this fictional world through a virtual reality system. In this world, we can perceive through vision, hearing, and touch, and some people are also studying smell and taste.

Virtual reality has the following characteristics: computer generation, multi-channel perception, immersion and imagination. When we put the virtual reality head-mounted display in front of our eyes, we can enter another world. Although a movie may also bring immersion, it is not interactive. In the virtual reality system, this immersion is interactive. When we move our heads, we can see scenes from different angles. As for imagination, it should be said that this virtual world is conceived by us. It can be a replica of the real world. For example, when we are in a meeting, after putting on the helmet, we may be in a conference room in San Francisco; it can also be an imaginary cartoon world, a fairy tale world you read in books when you were a child, which does not exist in reality.

The virtual reality system mainly includes the following aspects: first, the content, its core is a real-time computer graphics and simulator, and then the user interface: one input, one output, this is a very simple relationship, the input is mainly corresponding to the tracking of the user's head, eyes, hands and the entire body movements. When this body information is substituted into the virtual reality world, how the body moves, once the system senses it, it will give a corresponding feedback.

In fact, virtual reality is a multidisciplinary field. From a disciplinary perspective, its core is computer graphics and simulation, in addition to computer software and hardware, image and vision, artificial intelligence and other technologies. Because it uses multiple sensory channels such as vision, touch, and hearing, it also involves light, sound, electricity, mechanics, etc. Although Oculus has been designed quite well, it may still be a bit uncomfortable and dizzy to wear it for more than ten minutes. At present, the world's top scholars are studying ergonomics issues and how to design it to be more comfortable. When we put on glasses, some people will feel that their brains are controlled by the three-dimensional virtual environment, which may bring psychological shocks. Therefore, the application of virtual reality will also involve psychology, sociology and other issues.

Virtual content is usually initially constructed on the screen by modelers using 3D computer software. For example, starting from a simple model, a complex character model is gradually constructed. After designing it on the computer, it is given actions and finally the skin texture is attached to create a virtual character image. There is also a kind of virtual content that does not rely on manual work, but on a special camera - a panoramic camera. This camera has many lenses and can capture all 360-degree images. The data can also be displayed in virtual reality. This is an important application direction in the future. During the two sessions, some reporters also used such cameras. With virtual reality helmets, we can watch from all angles and save the data for others to see.

Of course, helmets are not the only form of virtual reality. There are many other forms. For example, CAVE is built with a projector, which can simulate four or six walls. Each wall is projected with a stereo projector. Wearing stereo glasses will give you a sense of immersion, but the cost is very high. In 1999, I participated in the installation of such a system at Zhejiang University. At that time, the equipment cost about 2.5 million RMB. There is also a circular screen projection, which is relatively cheaper and can be seen in museums and exhibition halls. It should be considered semi-immersive because the entire field of view is not completely covered. When people turn around, they feel like they are back in the real world.

Next is the input device. The input device needs to capture human signals. How to bring human motion signals into the virtual environment? The most commonly used device is the motion capture device. For example, the actors here make some movements, which are captured by many cameras and then put on the virtual character. There are two types of input devices: one is the optical motion capture device, which uses a camera plus infrared rays. There are many luminous points on the human body, and the computer's computing power is used to calculate the movement in real time, but it has a disadvantage that it is easily blocked; the other is the inertial sensor device. The user has many of them installed on his body, and there is an inertial data when he moves manually, which can calculate the angle. There is also facial motion capture, which is often used in movies. In the past, many points were added to the face, but now the technology has developed a lot. Instead of laying points on the face, the computer vision algorithm is used to calculate directly, and it can be done in real time.

In addition to face simulation, there is also a data glove. When wearing this special glove, each joint angle of the hand can be sensed by the glove. For example, when simulating holding a cup, a force feedback device gives the hand a feedback force. Although there is no cup, through the external force applied by the device, we can feel that we are holding something like a cup. At present, there are still many technical difficulties in this field that have not been broken through.

To sum up, the essence of virtual reality is a new computing platform, a new communication platform and interactive interface. From the perspective of computer development, the earliest was the mainframe, where a computer occupied the entire house. At that time, people were small and needed to adapt to computers. Later, when personal computers appeared, humans made computers smaller and smaller. When mobile phones appeared, we could calculate at any time. Virtual reality should be the next new computing platform. It is no longer that you feel that there is a screen between you and the computer world, but in the future you will feel that the world constructed by the computer is integrated with the real world. Although it is still displayed through the screen, you may not be able to tell which world you are in in the future. This is a development trend, and computers will gradually adapt to people's habits.

Augmented reality: an equally amazing technology

Many people may not be able to accurately distinguish between virtual reality and augmented reality (AR). There is also another one called mixed reality. Mixed reality is a transition from the real world to the virtual world, which includes virtual reality and augmented reality. There are many representative industries developing augmented reality, such as Google Glass, but it is not very successful. Microsoft has also just launched an augmented reality glasses product Hololens, which is relatively expensive, about 3,000 US dollars. They have a common feature, that is, they are translucent, because we still need to see the real reality scene, while the front of virtual reality glasses represented by Oculus is opaque, others can't see your eyes, and you can't see the things around you. This is the main difference between them, but augmented reality will have a wider application in the future.

For example, when we dress in front of the mirror, we can use augmented reality to change into another dress or skirt, or a suitable style. This is called virtual fitting. It can also be used in remote meetings. After wearing augmented reality glasses, we can see a virtual table and images of participants from a distance. Although they are in different spaces, they are discussing issues in the same environment at this moment. Recently, Microsoft launched a video. Through augmented reality glasses, even if your child is not around, he (she) can be placed in the same room in the augmented reality scenario, making you feel that the child is right in front of you. However, this aspect is not so good now. There are some conceptual elements and many defects. Augmented reality can also be used for car navigation. If we look at the navigation information, we need to look down at the car panel, which may be distracting. Using augmented reality technology, navigation information is displayed directly on the windshield, so that you can see the road information.

Application of virtual reality: unlimited future prospects

In addition to being used in entertainment applications such as movies, TV shows and games, virtual reality also has broad prospects in other non-entertainment application areas. Virtual reality can also be used for virtual fitting, which is Alibaba's "Buy+" plan. The concept was just launched in March this year. After wearing virtual reality glasses, people can choose and buy clothes through virtual models. Many car companies have also begun to use virtual reality glasses for simulated driving, allowing users to experience the feeling of driving, but they cannot really drive on the road. In addition, you can also do virtual tourism. After wearing glasses, you can experience places you have never been to before and feel like you are there. In addition, you can also virtually view or design a house. We can ask the designer to design a virtual picture in advance, and through experience, we can also correct the places that need to be modified.

Daily life experiences can have an impact on our brains, and virtual experiences can also affect the brain. Therefore, virtual reality can also be used for psychotherapy. Many people with depression can get some improvement through virtual reality technology-assisted treatment. For patients with social phobia, they usually dare not talk to many people. In the virtual world, you know it is fake, and your defense will be relaxed, making it easier to communicate with others, thus gradually improving your social skills. For another example, many people are afraid of spiders. Through virtual reality simulation, spiders are "close" to you. Although it is difficult for you to resist the fear in your subconscious mind, you can gradually "desensitize" yourself in this way. Acrophobia can also be alleviated through virtual reality.

Psychologists also use virtual reality for other treatments. For example, burn patients are in great pain, especially when changing their dressings. If a doctor uses virtual reality technology on them and lets them wear a helmet and see a cold world, their brains will respond accordingly and the pain will be reduced. After the 9/11 terrorist attacks in the United States, many people suffered great psychological trauma. Through virtual reality technology, patients are brought back to the environment at that time. This is also a treatment similar to "desensitization", which can relieve patients psychologically. It has been proven that it can have a relatively obvious effect. In addition, there is also treatment for autistic patients. They are unwilling to communicate with others and are difficult to get close to. Through virtual reality technology, they can play games in the virtual world, gradually accept their roles in it, and then exercise their skills in interacting with others. Studies have found that the activity of patients' brains has improved significantly before and after treatment, which shows that virtual reality has played a good role.

In the medical field, virtual reality can also be used to train doctors. Currently, the cost of training doctors is relatively high, with a lack of equipment and a large number of people. We can use virtual reality equipment to help doctors practice surgery. For example, in the virtual world, a device connected to force feedback can simulate a scalpel. When the scalpel touches a virtual organ, there will be a force feedback and the deformation of the organ will be simulated. In this way, doctors can practice surgical skills.

Virtual reality is also very helpful for learning and education. Generally speaking, children are more receptive to visual things. Through virtual reality technology, some relatively abstract things can be presented in an intuitive way, which can improve their memory and learning motivation. In this way, they are willing to learn and find it fun. In addition, it can also enhance cooperation among children.

In general, virtual reality has been around for more than 50 years, but it has only recently become accessible to the public due to its high cost. Virtual reality and augmented reality are related and have their own characteristics, but their core is real-time computer graphics, and their applications are constantly expanding. Through further integration with the Internet, robots, artificial intelligence, and big data, virtual reality will have a wider range of applications in the future. (Speech | Yang Xubo (Professor of the School of Software, Shanghai Jiao Tong University) Compiled by Ye Shuisong)