I fell in the metaverse, it hurts so much!

The baby left the mother's body and came into the world from the moist warm room. The discomfort of the temperature change made him cry for the first time. At the age of 10, he was playful and had poor grades. His mother slapped him on the arm. The pain made him remember that if he didn't work hard, he would be beaten. At the age of 17, he was rejected by the girl he liked. The scorching sun in the summer and the sticky sweat on his body were full of disappointment, which was a sour youth. At the age of 35, he failed in his business. He sat on the beach. The cold sea water hit the last will of this middle-aged man again and again. At the age of 89, lying on the hospital bed, he held his lover's hand. His skin was dry but warm and strong, which made people feel at ease. Then he fell into a deep sleep...

When people look back on their lives, it is not difficult to find that those unforgettable memories are often accompanied by the sense of touch. Because touch is the earliest and most basic sense developed by the human body, it is also the most widely distributed and most complex sensory system in the human body. It is an important way for humans to understand external things and explore the mysteries of the world.

Although people's sense of touch far outweighs vision and hearing, in real life, if we close our eyes we lose our vision and if we cover our ears we lose our hearing. It is difficult to understand the feeling of losing our sense of touch, so we tend to ignore the importance of touch.

The same is true in the metaverse. In the metaverse where vision, hearing and other perceptions have been realized, tactile technology is still at a relatively elementary stage. Zhu Hongwei, a professor at the School of Materials Science and Engineering at Tsinghua University, said that the metaverse integrates various sensory technologies, and the most core part of it is the sensor technology that embodies tactile sensation.

It can be said that tactile sensing technology has reached the conditions that require specific practice, but how to use tactile sensors to shield the human body's original perception, obtain new perception and enhance it to achieve a realistic effect is the key to realizing the metaverse.

From tactile sensing to tactile intelligence

After completing the first Easter egg mission and getting the bonus, the protagonist of "Ready Player One" first did is to buy the coveted X1 touch suit to enter the game world to experience it. The heroine patted him on the shoulder gently, and in an instant the two of them were holding hands and spinning on the dance floor. The X1 suit made him feel the touch and the temperature, just like he was really dancing with someone.

"Immersion" is a necessary condition for traveling in the Metaverse, and the X1 suit is the best solution to solve the problem of immersion. In the Metaverse, if you want to make a wearable device like the X1 tactile suit, the tactile sensor needs to be very intelligent.

Zhu Hongwei believes that tactile sensors include three major parts: sensing materials, software, and hardware, among which sensing materials are the core part. The sensing materials collect the physical signals of external stimuli and convert them into digital signals, which are then transmitted to the data analysis and processing module, and then to the feedback and drive module. In this system, the proportion of materials should be tiny both in terms of space and mass, and at the same time meet the characteristics of lightness, thinness, flexibility, strength, conductivity, transparency, and capacity. Therefore, from the perspective of power consumption, the sensor will adopt new energy power supply modes such as photoelectricity and human motion power supply.

The software level includes data processing and analysis. With the development of digital twin technology and the introduction of the concept of neuromorphic computing, the analysis and processing of data by combining artificial intelligence, machine learning and life morphological computing have greatly expanded the tools and means of tactile signal processing.

The hardware level includes simulation of bionic human body, limbs, and some driving devices. The earliest tactile sensor experience was only simple pressure sensing. With further analysis of touch, researchers found that touch includes comprehensive perceptions such as temperature, humidity, and surface flow. In terms of hardware, it has also evolved from simple pressure sensing to the integration of touch, from handheld to wearable to implantable, to better integrate tactile sensor signals with human perception signals.

Zhu Hongwei believes that the way to realize touch in the metaverse is implantation and wearability. Implantable devices are touch-sensitive chips implanted in the subcutaneous tissue of the skin. There are many implantable biosensors such as blood sugar monitoring and brain-computer interfaces to achieve the effect of observing various vital signs. In the current conception, implantable tactile sensor chips are used to repair the lack of touch in certain special scenarios.

Wearable devices need to be weavable and combined with clothing, or the sensor materials should be attached to the skin for structural design and wiring. "How to integrate and couple the sensor materials, software, and hardware without crosstalk to achieve highly unified reliability, consistency, and durability is the biggest challenge facing the current layout of metaverse sensors," said Zhu Hongwei.

The reality of multiple perceptions

Although theory and practice have proven that tactile sensing is feasible, it will still take time to apply it in the metaverse. Because touch is a complex perception, traditional single pressure sensors and motion sensors cannot meet the scene requirements of the metaverse. The digital content processed by the tactile sensor needs to be closer to the real perception of reality in order to give users a better immersive experience.

Due to the complexity of tactile perception, tactile sensing is destined to be a multidisciplinary technology. From materials to algorithms, and then to drive, it is a multifunctional integrated system. Sensing materials involve material physics and chemistry; algorithms involve calculations and information; feedback details involve electronics and microelectronics. The whole is combined with artificial intelligence, neuromorphic computing and even brain-like devices. Because it involves a wide range of disciplines, the requirements for algorithms are also more stringent.

High-demand algorithms are often considered to require extremely high accuracy and sensitivity. However, Zhu Hongwei does not think that the more sensitive the tactile sensor is, the better. In his opinion, for machines, all sensors have a drawback: they cannot simultaneously meet the contradictory characteristics of sensitivity and reliable stability.

Therefore, once the pursuit of extreme sensitivity, stability and reliability will inevitably decline. How to balance the relationship between the two is a problem that needs to be solved in current sensor technology. Just like our eyes are not as high-definition as cameras, and our sense of smell is not as sensitive as that of search and rescue dogs, but we can still fully experience the beauty of the world.

Another challenge for tactile sensing technology is how to use tactile sensing to block the body's own sense of touch in the metaverse and gain the ability to surpass human perception. Zhu Hongwei said that when a tactile sensor is applied to the human body, two appearances will appear: one is the perception of the tactile sensor acting on the human body, and the other is the tactile perception of the human body itself. In order to have a more immersive experience, the original perception of the body needs to be blocked in the metaverse, and new perceptions need to be obtained, which may require two sets of sensors to be performed at the same time.

The Metaverse is Gaining Tactile Sensitivity

Researchers have never stopped exploring tactile sensing technology. Currently, many laboratories around the world, including Stanford University and Northwestern University in the United States, the University of Tokyo in Japan, and the National University of Singapore, are conducting research in the field of tactile sensors.

The 2021 Nobel Prize in Physiology or Medicine was awarded to American scientists David Julius and Arden Pataptian for their discoveries in the sense of temperature and touch. David Julius used capsaicin (a pungent compound from chili peppers that causes a burning sensation) to identify sensors in the nerve endings of the skin that respond to heat. Arden Pataptian used pressure-sensitive cells to discover a new class of sensors that respond to mechanical stimuli in the skin and internal organs.

Meta (formerly Facebook) Reality Labs (RL), which is entering the Metaverse, launched tactile sensing gloves in March 2021. These gloves can reproduce a series of tactile sensations in real life in the virtual world, including simulating the feeling of human hands touching the texture of materials, pressure feedback, and vibration feedback.

Image source: Meta Reality Labs

This technology draws on recent advances in the field of soft robotics, replacing bulky motors with micro valves. At the same time, Zhu Hongwei believes that finding second-class materials such as graphene that fit the skin is the future development trend of sensor materials. From the current research, traditional sensor materials can also be improved, such as miniaturizing traditional tactile sensor materials and designing distributed or special structures to meet current usage needs.

However, miniaturized tactile sensors also have more stringent requirements in terms of energy consumption. They can use new energy sources such as solar energy and mechanical energy from human movement to continuously supply energy, or use non-conductive resistive sensors that do not consume electricity during non-working hours. In addition, the energy consumption of tactile sensors is also related to the layout and number of device modules, but in general, the energy consumption should be controlled at the microwatt or even nanowatt level.

Zhu Hongwei said that the future development of the Metaverse industry depends on the needs of future human survival and living habits. For example, flexible electronic technology has been developed for many years and the technology is relatively mature, but people's current lifestyles do not have a high enough demand for flexible screens, but are more adapted to traditional rigid screens. But in the world of the Metaverse, flexible electronics and wearable devices will definitely become core components.

Tactile sensors have maintained a very rapid development trend in recent years. With the rise of 5G, the Internet of Things and emerging industries, tactile sensors will usher in greater development opportunities. At the same time, he predicts that in the next 3-5 years, tactile sensors can be applied in certain fields and scenarios of the metaverse.

The poet sang the praises of life coming to Earth: "I was born and finally came here. My eyes are not yet open and my ears can't hear, but I know what a beautiful place this is." When a person enters the metaverse for the first time, he has not seen or heard anything, but his body's perception will tell him - welcome to this wonderful world.