When optics “meets” AI, will it become the “killer application” of 5G and 6G?

Computational optics can be understood as an optical imaging method for information encoding. "Its essence is the acquisition and interpretation of light field information. It introduces physical optical information on the basis of geometric optical imaging, takes information transmission as the criterion, and obtains higher-dimensional information through information." Computational optical imaging is the next generation of optoelectronic imaging technology and is the inevitable product of optoelectronic imaging technology entering the information age.

Background - Principles of Photoelectric Imaging

The essence of optoelectronic imaging is the acquisition and interpretation of light field information. The so-called light field interpretation refers to a more in-depth analysis and interpretation of the image information captured in the traditional optoelectronic imaging system. The traditional optoelectronic imaging system can only record the light intensity distribution in two-dimensional space, similar to human vision. However, in reality, the information contained in the imaging system is more than the image we see. Light field interpretation obtains more useful information by analyzing and interpreting this information. Through light field interpretation, we can extract and interpret some information hidden in the image, which leads to computational optical imaging.

Figure 1 Information contained in the light field

What is computational optical imaging?

Based on traditional geometric optics, computational optical imaging organically integrates information from physical optics, such as polarization, phase, orbital angular momentum and other physical quantities. It uses information transmission as a criterion to obtain light field information in multiple dimensions, and combines mathematics and signal processing knowledge to deeply mine light field information. It obtains higher-dimensional information through physical process interpretation, and is the next generation of optoelectronic imaging technology.

Comparison with traditional imaging

Traditional optical imaging systems use an imaging method of "imaging first, then processing", that is, when the imaging effect is not good, it is necessary to use tools such as Photoshop and "Meitu Xiuxiu" to further process the image. At this time, the image processing algorithm is considered a post-processing process and is not considered in the design of the imaging system.

Computational optical imaging uses the imaging method of "modulation first, then shooting, and finally demodulation". The optical system (illumination, optical devices, light detectors) and digital image processing algorithms are considered as a whole, and the front-end imaging elements and back-end data processing complement each other and become an organic whole.

Figure 2 Imaging process of computational optical imaging system

Computational optical imaging modulates more light field information into the original image captured by the sensor by introducing controllable coding into the illumination and imaging system, such as illumination coding, wavefront coding, aperture coding, etc. This image is also called an intermediate image. Because this image modulates a lot of additional light field information, it is often not directly usable and observable. Then in the demodulation stage, the mathematical model based on geometric optics and wave optics established in the above modulation and shooting stage is "reverse solved" to obtain more light field information by computational reconstruction. In other words, the image in "computational imaging" is not directly captured, but calculated by "encoding and decoding". Optical coding includes aperture coding, detector coding and other methods, which is the meaning of calculation in computational imaging. ‍‍‍‍‍‍‍‍‍‍

Development Overview

The idea of ​​"computational imaging" was first applied in the synthetic aperture radar of the photoelectric detection system. In 1984, the University of Colorado proposed the real use of the idea of ​​"computational imaging" to jointly design optics. In 1995, Dowski and Cathey proposed the wavefront coding technology, which was a turning point for computational optical imaging technology to move from theory to practical application. In 2004, Levoy's research group built a multi-camera array system to apply computational optical imaging to synthetic aperture imaging, high-speed photography, and HDR imaging. In 2006, Lytro Company successively launched two commercial-grade handheld microlens light field cameras, Lytro I and Lytro II, which can realize full-depth shooting of full-view shooting after focusing, and light field imaging has since entered the public's field of vision.

Domestic computational optical imaging research is also being carried out in sync with the international community. The Computational Optical Imaging Technology Laboratory of the Institute of Optoelectronics, Chinese Academy of Sciences, has conducted extensive research in computational spectroscopy, light field, and active three-dimensional imaging; the National Information Laboratory and the Institute of Optoelectronic Engineering of Tsinghua University have made important contributions in computational light field imaging and microscopic imaging; the Key Laboratory of Computational Imaging of Xidian University has carried out research based on scattering imaging, polarization imaging, and wide-area high-resolution computational imaging technologies; the Optical Imaging and Computing Laboratory and the Measurement and Imaging Laboratory of Beijing Institute of Technology have also proposed optimized solutions for computational display and computational spectral imaging; the Intelligent Computational Imaging Laboratory of Nanjing University of Science and Technology has achieved excellent results in quantitative phase imaging, digital holographic imaging, and computational three-dimensional imaging.

Technical

The emergence of computational optical imaging technology extends and expands the visual characteristics of the human eye, innovatively extending the information processing and calculation at the end of the traditional optoelectronic imaging process to the imaging process, converting some traditional optical problems that are physically difficult to break through into mathematical and information processing problems , and combining hardware design with software calculations. It fully taps the potential of information processing of optical signals, solves the problems of "invisible", "unclear", and "incomplete" in traditional optoelectronic imaging technology, and develops towards higher resolution, longer range, wider field of view, smaller optical imaging system, and stronger environmental adaptability.

Figure 3 Five development goals of optical imaging technology

Unlike traditional photoelectric imaging, which uses single link calculation and independent optimization, computational imaging introduces the idea of ​​full-link integrated global optimization and incorporates transmission media (such as atmosphere, water bodies, etc.) into the imaging model, which can improve the range and environmental adaptability, breaking through the limits of traditional imaging. Its full link includes light source, imaging target, transmission media, optical system, photosensitive electronic components (CCD), external circuitry, computer, etc.

Figure 4 Full-link optical path imaging model

This article focuses on introducing light field technology from two aspects: light field acquisition and three-dimensional display .

Light Field Acquisition Existing light field acquisition methods can be divided into three types: time-series acquisition, multi-sensor acquisition, and multiplexed acquisition . The multi-viewpoint information of the acquired light field can be used for light field display.

Timing acquisition

Time-series acquisition generally installs a traditional single camera on a mechanical mobile device, and adjusts the mechanical device to acquire images of the target scene from different perspectives. However, this method takes a certain amount of time when moving the mechanical device, so it can only capture static objects. In order to solve this problem, multi-sensor acquisition methods came into being.

Multi-sensor acquisition

Multi-sensor acquisition generally refers to a light field camera based on a camera array. The multi-camera combination method uses multiple traditional cameras to form a camera array, forming a virtual projection reference plane composed of multiple lens projection centers, and a virtual imaging plane composed of multiple CCDs (or CMOS). Multiple cameras are used to simultaneously obtain the light radiation intensity at different viewing angles at the same point in the target scene. The image captured by each camera can be regarded as a sampled image of the light field at different angles.

By adjusting the distance between each sub-camera in the camera array, the entire camera array can be used for different purposes. When the distance between all cameras is relatively small, the entire camera array can be regarded as a monocular camera; when the distance between all cameras is medium-sized, the entire camera array can be regarded as a camera with a synthetic aperture; when the distance between all cameras is large, the entire camera array can be regarded as a multi-camera camera, at which time multi-view information of the object can be obtained, and a panoramic photo can be constructed using the multi-view information.

A representative example is the world's first real-time camera array consisting of 8x8 cameras built by Jason Yang at MIT in 2002, which can simultaneously acquire images of 64 viewing angles of the target scene. In 2005, Bennett Wilburn further increased the number of cameras to about 100, which has stronger capabilities, but is costly and bulky. In 2013, Pelican manufactured a compact 4x4 camera array by optimizing optical design and applying light field super-resolution algorithms. It is an independent light field camera module. The entire device is smaller than a coin and can be integrated into a mobile phone.

Figure 5. Large-scale light field camera array designed by Bennett Wilburn in 2005

Multiplexed imaging

The disadvantage of multi-sensor light field acquisition is that it can only be used to acquire static or slowly moving scenes in a specific area. The multiplexed imaging method can solve this problem. Multiplexed imaging is divided into spatial multiplexing and frequency multiplexing .

Spatial multiplexing : This is achieved by installing a microlens array on the image sensor. For example, a plenoptic camera, also known as a light field camera, is similar in shape, size, and operation to a normal camera. It encodes a four-dimensional light field onto a two-dimensional sensor plane by multiplexing the angular domain (angle information) into the spatial (or frequency) domain.

Let's talk about the principle of light field camera. The surface of objects in real life is generally diffuse reflection. For example, if point A in the figure below is an ideal diffuse reflection surface, it will emit light within a 180-degree range of the hemisphere. Using the lens pinhole imaging, all the light emitted by point A on the surface of the object within a certain angle range is focused on the imaging sensor point A' through the lens, and all the light within the angle range is integrated, and the integration result is used as the pixel value of point A. This greatly increases the signal-to-noise ratio of the imaging, but it also couples the light from point A in all directions within the angle range together.

Figure 6 Principle of lens pinhole imaging

The plenoptic function is a function that describes the change of the light field in space and time. In order to obtain the angle information in the plenoptic function, Adelson proposed the plenoptic camera (also called light field camera) model in 1992. A microlens array was added to the traditional camera. The light on the surface of the object first passes through the main lens, then through the microlens, and finally reaches the imaging sensor. Each microlens captures all the angle distributions of the light at the main lens. As shown in Figure 7, the light emitted by point A on the surface of the object within the FOP (Field Of Parallax, parallax range) angle range (light at different angles at the same point is represented by different colors) enters the camera main lens and focuses on the microlens. The microlens divides the light into 4x4 beams and is recorded by the corresponding 16 pixels on the imaging sensor. Similarly, the light of other luminous points in space, such as points B and C, within their FOP angle range is divided into 4x4 beams and recorded separately.

Figure 7 Image resolution and angular resolution of the lens array 4D light field

In 2011, Lytro launched the world's first consumer light field camera, Lytro, based on the principle of plenoptic camera.

Figure 8 Raytrix light field camera based on microlens array

Spatial multiplexing is the most widely used method for collecting light fields, but it also has problems, that is, image resolution and angular resolution are mutually restricted, and one is traded off. In addition, since the direct distance between microlenses is very small, the parallax angle of the collected light field is small, and the viewpoint can only be changed within a small angle range.

Frequency reuse: In recent years, with the development of information theory technologies such as compressed sensing, in order to solve the problem that spatial multiplexing methods must compromise image resolution and angular resolution, some frequency reuse methods such as mask-based ones have also emerged. The mask-based solution can obtain a light field dictionary by learning the light field in advance, thereby removing the redundancy of the light field and realizing the reconstruction of a complete light field by collecting less data. As shown in Figure 9, a semi-transparent coding mask is added to the imaging optical path of a traditional camera. The light transmittance of each pixel on the mask is different. The light entering the aperture will be modulated by the mask before reaching the imaging sensor, and the light modulated by the mask reaches the imaging sensor. Through the pre-learned complete light field dictionary, the complete light field can be reconstructed from a single captured coded 2D modulated image.

Figure 9 Principle of mask light field camera

Although the light field acquisition scheme based on coded masks can solve the contradiction between image resolution and angular resolution, the transmittance cannot reach 100%, which will cause loss of light signal intensity and lead to low imaging signal-to-noise ratio; at the same time, the final reconstructed light field image is obtained by demodulating the modulated image rather than directly acquiring it through the imaging sensor, so it will rely on the correctness of the learned light field dictionary.

In summary, the existing light field acquisition methods mainly record light fields by sampling light at multiple angles through a multi-sensor camera array or a microlens-based plenoptic camera with spatial multiplexing imaging or a mask-based plenoptic camera with frequency domain multiplexing. Different light field acquisition devices solve certain problems through different technical solutions. According to different scene requirements, breakthroughs are constantly being made in software and hardware. With the continuous advancement of technology, I believe there will be better and lighter solutions for light field acquisition.

3D Display

The real world is three-dimensional. Traditional two-dimensional display can only obtain two-dimensional image information of a certain cross section of a space object, lacking depth information, and has problems such as lack of information and lack of authenticity. Therefore, three-dimensional display technology with better immersion is an important development direction of new display technology.

3D Depth Cues

Three-dimensional depth cues refer to all characteristic information that can provide users with depth perception. They are the key to three-dimensional display effects and are divided into two categories: psychology and physiology.

Psychological depth cues refer to the characteristic information of 3D perception induced by 2D images, mainly including linear perspective, occlusion, shadow, texture and prior knowledge, etc. Physiological depth cues refer to the characteristic information of 3D perception induced by the spatial position relationship of 3D scenes, which can be further divided into binocular depth cues and monocular depth cues.

For 3D display, psychological depth cues are easier to achieve. Traditional ultra-high definition can also achieve a certain sense of 3D through effects such as fine textures and shadows, such as outdoor large-screen naked-eye 3D effects and mobile phone naked-eye 3D ringtones. Physiological cues are relatively difficult to achieve. With the development of technology, binocular stereo vision has become relatively mature and has gradually entered the public's field of vision through VR equipment, naked-eye 3D equipment, etc., while monocular depth cues are relatively more difficult to achieve.

3D Display Technology Classification

According to the different mechanisms for achieving the stereoscopic effect, three-dimensional display can be divided into: assisted 3D display , such as red and blue glasses, where the viewer wears glasses or other visual aids; in addition, there is binocular parallax 3D display , which projects different images to the left and right eyes to produce a stereoscopic effect; and true 3D display , which provides nearly real 3D image information and real physical depth of field, including light field display, volume display, holographic display, etc. Assisted 3D technology has poor effects and is relatively old, so it will not be described in detail here. The following briefly introduces binocular parallax 3D display, light field display, volume display and holographic display.

Figure 10 3D display classification

Binocular parallax 3D

Binocular parallax 3D has traditional methods based on slit gratings and cylindrical lenses, as well as parallax-type three-dimensional displays based on directional backlights, which maintain the panel resolution perceived by the viewer's left or right eye in the viewing area. It uses time-space multiplexing 3D display technology, based on a directional backlight structure design that controls the direction of light emission, combined with high-speed refresh LCD display technology, and uses time multiplexing to achieve full-resolution 3D display effects.

Figure 11 Pointing backlight display principle

Light Field Display

Parallax 3D display presents several discrete viewpoint images at a certain moment, while light field display presents continuous or quasi-continuous viewpoint images within a certain angle range. Light field display technology can be divided into compressed light field display, directional light field display and integrated imaging display according to the implementation principle.

Compressed light field display

It is also called stacked light field display because it uses a multi-layer structure and uses stacked display layers to modulate the light intensity within a certain angle range. Because of its multi-layer screen structure, it can achieve high dynamic range or super-resolution 2D image display. Compressed light field display uses the strong correlation between viewpoint images of a three-dimensional scene to "compress" the target light field at a certain viewing angle into multiple two-dimensional patterns. However, because of this, it will have a lot of crosstalk when displaying unrelated multi-viewpoint images. The compressed light field display has a limited field of view. The viewing angle can be expanded by increasing the number of display layers or increasing the refresh rate, but increasing the number of display layers will increase the hardware and computational complexity, and the optical efficiency will also decrease exponentially.

Pointing light field display

The above problems can be solved to a certain extent. In 2012, G. Wetzstein et al. proposed a new "tensor" display technology by adding a directional backlight panel to the back of a multi-layer LCD. This structure achieves the purpose of expanding the field of view through multi-layer LCD screens and multi-frame rate light field decomposition, and the field of view is increased from the original 10°×10° to 50°×20°.

There is also integrated imaging display . The 3D display technology based on the cylindrical lens array can only provide horizontal parallax, while the integrated display based on the spherical microlens array can provide full parallax images. The principle is shown in Figure 12. When recording the image, the microlens array or pinhole array is used to "integrate" the images of different perspectives on a piece of film. When the image is reproduced, the outgoing light of the multi-perspective image element is reproduced in front of the microlens array, and the element image on the display panel is imaged to different viewpoints to form a complete 3D image. The advantages of integrated imaging display are compact and simple structure, ability to reproduce full parallax 3D images, and can provide motion parallax information. However, its resolution is severely reduced and the field of view is relatively limited.

Figure 12 Integrated photography and integrated imaging

Volume Display

Light field display can provide all kinds of psychological cues and binocular depth cues, and can provide monocular depth cues such as motion parallax and occlusion changes. However, traditional light field display is difficult to provide complete focus cues. Volumetric 3D display can complete the display process within a certain volume of space by lighting up the luminous material or "voxel" in the space, and can provide focus cues through the persistence effect of the human eye, but it requires complex mechanical scanning devices and has poor mobility.

Figure 13 Rotating scanning stereoscopic display principle

Holographic Display

Theoretically, holographic display can provide all kinds of depth clues and is considered to be the ultimate way to achieve three-dimensional display. Holography, also known as holographic photography, refers to the technology of recording the amplitude and phase distribution of light waves on photographic film or dry plate and reproducing the three-dimensional image of an object. According to its physical meaning, it can be divided into two parts: wavefront recording and wavefront reconstruction. In the process of wavefront recording, the object light wave and the reference light wave interfere with each other in the hologram plane, and the intensity of the interference fringes is recorded. Since the amplitude and phase of the reference light wave are known, holography is to convert the complex amplitude signal of the object light wave into an intensity signal, thereby recording all the information of the object light wave. In the process of wavefront reconstruction, a beam of reconstruction light wave is used to illuminate the hologram, and the reconstructed light wave can reconstruct the amplitude and phase information of the object light wave at a specific position through the diffracted light wave of the hologram.

Figure 14 Holographic recording and holographic reconstruction

In summary, with the development of 3D display technology, the effect of 3D display has been significantly improved. Traditional naked-eye 3D based on binocular parallax in psychological and physiological cues has been widely used due to its simple principle and low cost, but because it only provides a limited viewing angle, there is still a lot of room for improvement in experience. In order to provide more physiological and psychological depth cues, light field display technology, volume display technology, and holographic display technology have all developed rapidly, but these technologies often have high implementation costs and are mostly still in the research stage, and are still a long way from real consumer-level applications.

Application

Computational optical imaging technology is widely used in various fields, mainly in the fields of medicine, astronomy, military, industrial detection, etc. In recent years, with the development of virtual/augmented reality, naked-eye 3D and other businesses, computational optical imaging technology has gradually been applied to life and entered the public eye. Through optical imaging technology, people can better understand and grasp the information of the surrounding environment, improve work efficiency, improve the quality of life, and promote the development of science and technology. In today's society, the application of optical imaging technology has become one of the indispensable key technologies in various fields.

Virtual/Augmented Reality

The traditional structured light technology can be used to quickly produce 3D models of objects, and combined with deep learning technology, 3D models can be quickly generated through multi-angle images/videos. For example, the research team of Tsinghua University has developed a new multi-camera large-space dense light field capture system - Den-SOFT, which is the leading data set in the public domain in terms of quality and viewpoint density. It is expected to inspire space-centric light field reconstruction research and provide higher quality 3D scenes for immersive virtual/augmented reality experiences.

Figure 15 Tsinghua Den-SOFT

With only consumer-grade RGB-D cameras, real-time 3D human reconstruction of multiple people can be achieved, and the human body model can be used for real-time AR display, online education, and games, etc. Please refer to the Tsinghua University paper "Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors" and other papers.

Figure 16 Tsinghua University RGB-D real-time 3D reconstruction

In terms of professional-level motion capture drivers, optical motion capture completes motion capture by monitoring and tracking specific light points of the target. High-resolution optical infrared cameras can be used for capture. For example, Lingyun Optics' FZMotion AI multi-modal motion capture system can achieve multi-modal motion capture with high-precision and high-speed AI human motion recognition.

Figure 17 FZMotion AI multimodal motion capture system

In terms of consumer-level drivers, combined with deep learning, real-time multi-person motion capture can be achieved using a small number of consumer-grade RGB-D cameras.

Glasses-free 3D

Consumer-grade naked-eye 3D is developing rapidly. In March 2023, ZTE released the LCD Nubia Pad 3D (in cooperation with Leia, an American company) at the Mobile World Congress. It adopts a switchable directional light source module, a display resolution of 2K, and supports real-time switching between 2D and 3D displays. The price is about 1,300 euros. In terms of content, Nubia and China Mobile Migu have cooperated deeply to jointly bring users a multi-dimensional interactive and immersive new game, video, live broadcast, and music experience, bringing a broader development space for the digital entertainment industry.

Figure 18 ZTE nubia Pad 3D

In the field of light field display, looking glass and the like can realize multi-viewpoint light field display.

Figure 19 Tsinghua University light field display research

Film and TV Production

The Light Stage light field acquisition system is widely used in Hollywood film production. Light Stage is a high-fidelity 3D acquisition and reconstruction platform system developed by Paul Debevec of the ICT Graphic Lab at the University of Southern California. The system focuses on high-fidelity 3D face reconstruction and has been used in Hollywood movie rendering. The first generation system, Light Stage 1, was born in 2000 and has been upgraded to Light Stage 6. The latest generation system is named Light Stage X.

Figure 20 Light Stage 6 acquisition system prototype

In China, companies such as Lingyunguang and Yingmo Technology have also built self-developed acquisition systems similar to Light Stage. For example, Lingyunguang's LuStage digital human light field modeling system can achieve 0.1mm pore-level reconstruction. The system consists of 756 6-color LEDs (RGBWCA) independently designed, developed and independently manufactured, a 375-sided bracket with a diameter of 6.6m, an embedded control platform, and 100 4K cameras. Each light source contains 97 LED beads, and the brightness, light-emitting time, and frequency of each bead are controllable. The entire system can simulate the ambient lighting in the input image or video, and can also light up the light source in a certain direction one by one at a time. It can also realize the gradient lighting of the three-axis XYZ image, and use the camera on the spherical surface to synchronize the light source to collect images, and realize film and television lighting, AI training data collection and other purposes.

Figure 21 LuStage digital human light field modeling system

Medical

In the medical field, the team led by Dai Qionghai, an academician of the Chinese Academy of Engineering and director of the Imaging and Intelligent Technology Laboratory at Tsinghua University, has realized that a high-resolution integrated microscope can be equipped in a mobile phone to stimulate new portable diagnosis of skin health without the need for additional electronic devices. Other skin diseases, such as acne, pemphigus, and psoriasis, can also be easily diagnosed at one time through an integrated microscope and corresponding intelligent algorithms.

Figure 22 Integrated microscope in mobile phone

astronomical

In the field of astronomy, the digital adaptive optics Earth-Moon observation system developed by the team of Academician Dai Qionghai of Tsinghua University has achieved a 400,000-kilometer Earth-Moon imaging test at the Xinglong Observatory of the National Astronomical Observatory. It can be directly connected to the existing optical system; it has reduced the volume of the existing large-aperture optical system and reduced the cost by more than three orders of magnitude; it has achieved the correction of spatially non-uniform environmental turbulence aberrations in multiple areas with a large field of view, and reconstructed high-resolution images.

Figure 23 Digital adaptive earth-moon observation

Computational optical imaging has been applied in various fields, and the market for computational optical imaging is developing rapidly. With the development of the metaverse, computational optical imaging is becoming more and more important in content production and display. With the development of technology, the market prospects of software and hardware involved in computational optical imaging are becoming more and more broad.

Outlook

Traditional photoelectric imaging technology is limited by industrial design ideas and its performance has reached its limit. Computational imaging technology is an inevitable development in the information age.

Light field acquisition is expected to expand new input methods for the metaverse

Traditional photoelectric imaging can only obtain two-dimensional data, while the human eye can perceive information with greater data density and dimensions than traditional cameras. The human eye has a sophisticated structure that can capture multi-dimensional light field information in real scenes in real time. Combined with the brain's calculations, the human eye + brain is a natural and precise computational optical imaging system. The traditional shooting method currently used will lose a large amount of high-dimensional data information, resulting in a significant decrease in acquisition efficiency. If a computational optical system like the human eye + brain is designed, efficient real-time acquisition of real scenes can be achieved. Through 4D light field data, many problems that cannot be solved in current two-dimensional images can be solved, and ultimately true intelligent machine vision can be achieved.

Holographic display is the ultimate form of future 3D display

Recreating a real three-dimensional world through light field technology has been a dream of mankind for many years. The current three-dimensional display technology cannot provide all the depth clues, which will cause various problems such as dizziness and unreal images. Holographic display will display three-dimensional objects in the real world, making the virtual and the real world perfectly integrated, and the boundary between the real and virtual worlds is no longer obvious.

Figure 24 The holographic 3D military sand table depicted in the movie Avatar

Light field + AI is the only way to compute optical imaging

Deep learning has gradually penetrated into multiple optical technology fields and promoted the development of many optical technologies. Deep learning has powerful computing, data evolution and nonlinear inverse problem solving capabilities, providing new ideas and methods for the optimization and solution of more complex optical system designs. At the same time, how to simplify optical systems based on deep learning, reduce optical system costs, and improve light field acquisition effects is also a major problem. At present, the three-dimensional reconstruction technology based on neural radiation fields is changing with each passing day, and behind this is people's infinite expectations for lightweight light field three-dimensional reconstruction.

Computational optical imaging technology is expected to achieve 5G and 6G killer applications

With the development of image encoding and decoding and display hardware technology, the resolution of traditional two-dimensional video has grown from 2K to 4K, 8K, and 16K, which has exceeded the resolution limit of ordinary human eyes. The benefits of further increasing image resolution are already very small. However, with the development of metaverse technology, content has moved from two-dimensional to three-dimensional, and with it comes an exponential increase in the amount of information.

Traditional high-resolution information, which is close to the limit of human vision in two-dimensional planes, is inadequate in three-dimensional displays.

In the post-5G and even 6G era, mobile phone naked-eye 3D display applications represented by micro-nano light field control and metasurface optical technology are likely to drive the leapfrog development of mobile communication networks and become the "killer" application of 5G and even 6G networks. Technologies such as millimeter wave, terahertz and flood-optical communication (which belongs to the category of 6G photonics) can greatly improve network carrying capacity indicators such as communication peak bandwidth, continuous spectrum bandwidth, network latency and traffic density.

The post-5G and even 6G life cycle may be an era of data "explosion". For example, in order to present realistic display effects to users, 3D high-definition video may require end-to-end traffic of more than Gbit per second, and holographic 3D stereo images that do not require glasses require 300 to 300 million pixels. The huge amount of Tbit-level data generated brings unprecedented pressure to mobile networks. First of all, the compression of 3D object data becomes the first challenge faced. Huge data must require a high compression ratio. Secondly, the storage of massive 3D object data is also a difficult problem. The amount of information carried by 3D objects is very huge, and it may also break the current limits in terms of storage and reading and writing. Finally, the transmission and calculation of 3D data are also key issues. How to ensure real-time transmission and calculation of such huge data? The upgrade and transformation of edge computing power seems to be inevitable.

In addition, there is still a blank in the standards for computational optical imaging from production to experience, which needs to be supplemented with technological progress and industrial implementation. With the continuous development of computational imaging technology and the continuous improvement of theory, the computational imaging system will become richer, more three-dimensional and more effective, enabling computational imaging to truly achieve the goals of higher (imaging resolution), farther (detection distance), larger (imaging field of view) and smaller (power consumption and volume).

[References] [1] SHAO Xiaopeng, SU Yun, LIU Jinpeng, et al. Connotation and System of Computational Imaging (Invited) [J]. Acta Photonica Sinica, 2021, 50 (5): 0511001

[2]CHENLiang,YUShao-hua. Research on the Impact of Holographic 3D Display on the Capacity of the Post-5G/6G Network

[3] Qiao Wen, Zhou Fengbin, Chen Linsen. Towards application of mobile devices: the status and future of glasses-free 3D display

[4] WANG Cheng, ZHU Xiang-bing.Research Development of Stereoscopic Display Technology

[5] Gao Chen, Li Ziyin, Wu Rengmao, Li Haifeng, Liu Xu. Development and Prospect of Portable Three-Dimensional Displays

[6] Cao Liangcai, He Zehao, Liu Kexuan, Sui Xiaomeng. Progress and challenges in dynamic holographic 3D display for the metaverse (Invited)

[7] KANG Jiaxin, WANG Wenwen, PENG Yuyan, ZHANG Jiazhen, ZHOU Xiongtu, YAN Qun, GUO Tailiang, ZHANG Yongai. Research Status and Trends of Light Field Display Technology

[8] Fang Lu, Dai Qionghai. Computational light field imaging[J]. Acta Optica Sinica, 2020, 40(1): 1-11.

[9] Zuo Chao, Chen Qian. Computational optical imaging: An overview

[10] Gao Jinming, Guo Jinying, Dai Anli, Situ Guohai. Optical system design: from iterative optimization to artificial intelligence.

[11] Shao Xiaopeng. Computational optical imaging [OL]. Broadcasting and Television Imaging Engineering Center of Xidian University (official account)

Author: Liu Jianlong and Bi Lei

Unit: China Mobile Communications Group Migu Company