New breakthrough! Chinese scientists develop the world's first brain-like complementary vision chip →

In the modern technological arena, "visual perception technology" is playing a vital role. From self-driving cars to dexterous robots to ubiquitous intelligent monitoring systems, the performance of image sensors directly determines the success of these technologies.

However, when faced with dynamic, changing and unpredictable environments, traditional image sensors often fail to cope with the challenges they face, including limited dynamic range, data redundancy, perception delay and other major aspects.

Dynamic range refers to the range of all pixels in the visible range from the darkest to the brightest that can be captured in an image or video. The larger the dynamic range, the more pixel changes the device can capture, and details from deep black to bright can be displayed more clearly. However, the dynamic range of traditional sensors is very limited, making it difficult to capture clear images in both strong and low light environments.

Data redundancy refers to the fact that high-resolution and high-speed sensors generate a large amount of data, which increases the burden of processing and transmission. Perception delay refers to the fact that due to the limitation of processing speed, sensors are prone to perception delay in a rapidly changing environment, which affects the timeliness of decision-making.

Due to the limited dynamic range, the camera cannot capture a clear portrait in the low-light environment (third image) (Image source: Reference 1)

These problems are particularly evident in areas such as autonomous driving, robotics, and artificial intelligence. For example, in autonomous driving, sensors must be able to quickly and accurately identify road conditions and potential hazards, but traditional sensors often perform poorly when dealing with complex scenarios, such as the sudden appearance of pedestrians or vehicles.

These technical barriers limit the application of image sensors in complex environments and give rise to an urgent need for more advanced visual perception technologies. Scientists are trying to find solutions by continuously studying the excellent human visual system.

Inspiration from the human visual system

The human visual system (HVS) performs well in processing complex visual information. That is, the primary visual cortex, as the initial processing area of ​​visual information, decomposes visual information into primitive components such as color, direction, and motion, and transmits this information to the dorsal and ventral streams, which are then processed through two main pathways:

1

Cognitive Path

The ventral stream connects to the temporal lobe and is primarily responsible for high-precision cognition and detail recognition, such as color and shape. This pathway enables us to see the details and colors of objects clearly and to perceive the environment accurately.

2

Motion Path

The dorsal stream connects to the parietal lobe and is primarily responsible for rapid response and motion detection, such as direction and speed. Through this pathway, we can quickly identify moving objects and respond accordingly, such as avoiding obstacles or chasing targets.

Dorsal stream, ventral stream and primary visual cortex (Image source: VISUAL SYSTEM: CENTRAL PROCESSING)

This dual-path processing method enables humans to perceive and respond efficiently and accurately in various complex environments. Based on the imitation of the human visual system, the research team of Tsinghua University has developed the world's first brain-like complementary visual chip - the Tianmo chip, which breaks through the shortcomings of traditional visual perception chips and provides an unprecedented efficient and accurate visual perception solution.

The birth of Tianmo chip

The design concept of the Tianmou chip is based on in-depth research on the human visual system and adopts a hybrid pixel array and parallel heterogeneous readout architecture.

The hybrid pixel array mimics the cones and rods in the human visual system, which are used for color and motion detection, respectively. Epithelial cells support and protect the cell layer of photoreceptor cells. Cones are mainly responsible for capturing color information, allowing us to see rich color details in bright light. Rods are extremely sensitive to changes in light intensity, and are particularly suitable for low-light environments, helping us see the outlines and movements of objects in dim conditions.

Cones, rods and epithelial cells (Image source: Seeing color)

The parallel heterogeneous readout architecture is the core part of the Tianmou chip . Its role is to convert electrical signals from different pixels (such as cones and rods) into digital data at high speed and high precision. The advantage of this architecture is that it can handle high dynamic range and high speed perception requirements at the same time, effectively reduce data redundancy, and maintain high performance under complex lighting conditions.

The architecture of the Tianmou chip, including the hybrid pixel array and its interaction with multiple paths (Image source: Reference 2)

Through the application of these new technologies, the Tianmou chip has the three characteristics of high-speed perception, wide dynamic range and bandwidth optimization, which solves the shortcomings of traditional sensors.

1

High-speed perception capability

The Tianmo chip can achieve a speed of up to 10,000 frames per second, ensuring that clear images can be captured in rapidly changing environments. This high frame rate perception capability is critical for application scenarios such as autonomous driving and robotics that require real-time perception and response.

2

Wide dynamic range

The dynamic range is calculated in dB (decibel). The dynamic range of traditional sensors is usually between 60 and 80 dB, while the dynamic range of the human eye is about 120 dB. The Tianmou chip has a dynamic range of up to 130 dB, which can provide clear images in both strong and weak light environments. This means that even in complex lighting environments with direct sunlight and shadows, the Tianmou chip can provide delicate picture details.

The signal-to-noise ratio of the Tianmo chip at different optical power densities. By combining the high-gain and low-gain modes of the action path and the cognitive path, the Tianmo chip achieves a wide dynamic range of 130dB. This shows that the chip can provide high-quality images under extremely strong and weak lighting conditions. (Image source: Reference 2)

3

Bandwidth Optimization

Through adaptive technology, the Tianmou chip can reduce bandwidth requirements by 90%, effectively reducing the burden of data transmission and processing. This bandwidth optimization technology not only improves data transmission efficiency, but also reduces energy consumption, making the Tianmou chip more suitable for mobile devices and IoT applications.

The high performance of the Tianmo chip under fast motion and light flicker interference. Through the high-speed response of the action path, the chip can quickly handle unpredictable light flicker events while maintaining low bandwidth consumption. The Tianmo chip shows superior performance in terms of power consumption and bandwidth compared to traditional and neuromorphic vision sensors. (Image source: Reference 2)

Application cases of Tianmo chip

The application of Tianmou chip in autonomous driving system is an important demonstration of its powerful performance. It can provide accurate, fast and robust perception in complex road environments and respond quickly even in corner situations. This is of great significance to improving the safety and reliability of autonomous driving systems.

For example, the Tianmou chip demonstrated its superior performance in dealing with sudden appearances of pedestrians and vehicles during autonomous driving tests, significantly reducing the probability of accidents.

In addition to autonomous driving, the Tianmo chip can also be widely used in drones, security monitoring and other fields. For example, in security monitoring, the Tianmo chip can provide high-quality video images in an environment with drastic light changes, which helps to detect potential security threats in a timely manner. In drone applications, the high dynamic range and high-speed perception capabilities of the Tianmo chip enable drones to navigate and monitor efficiently in complex terrain and lighting conditions.

The performance of the Tianmou chip in a long-distance driving test. During the test, the vehicle encountered extreme situations such as day and night, tunnels, high dynamic range, abnormal objects and complex scenes. The Tianmou chip ensured high-precision perception through seamless synchronization of cognitive path and action path detection results. (Image source: Reference 2)

The results of various experiments show that the Tianmou chip not only has a high dynamic range and high resolution, but also maintains excellent perception under high-speed motion and extreme lighting conditions, and performs far better than traditional sensors in extreme environments.

The Tianmo chip has unlimited possibilities in the future development of science and technology. As technology continues to advance, it will play an indispensable role in more fields. Imagine how the ultra-high-quality visual experience brought by the Tianmo chip in augmented reality (AR) and virtual reality (VR) will completely change the way we perceive and interact? This is just the beginning.

In the future, when the Tianmo chip is deeply integrated with artificial intelligence technology, what disruptive changes will it bring to smart city construction, medical image analysis, industrial automation and other fields? How will it lead us into a more intelligent and connected world? The suspense remains, so we will wait and see.

References

[1]Han, Yuqi & Yu, Xiaohang & Luan, Heng & Suo, Jinli. (2023). Event-Assisted Object Tracking on High-Speed ​​Drones under Harsh Illumination Environment. 10.20944/preprints202312.1056.v1.

[2]Yang, Z., Wang, T., Lin, Y. et al. A vision chip with complementary pathways for open-world sensing. Nature 629, 1027–1033 (2024).

Planning and production

Produced by Science Popularization China

Author: Zheng Shengjie, PhD student in Computational and Neural Systems

Audit丨China Science Expo

Editor: Yang Yaping

Proofread by Xu Lailinlin