Can a small screen also be used to create a "blockbuster"? Let your photos achieve the "depth of field effect" on your phone

1. Summary

On iOS, when users set system wallpapers, they will find that sometimes a small part of the wallpaper appears in front of the lock screen clock, creating a layered visual style. This lock screen effect has been available since iOS 16. Users can choose an ordinary picture as the background, and part of the content in the picture can cover some desktop components in appropriate places. iOS calls this lock screen effect Depth Effect.

2. What is the Depth of Field Effect?

Depth of field is an important concept in photography. It refers to the effect that when the camera lens is focused on a certain point, the scenery within a certain range before and after the focus can form a clear image. The depth of field effect determines which parts of the photo are clear and which parts are blurred, and plays an important role in enhancing the visual impact and artistic expression of the photo.

Depth of field effects are widely used in photography. For example, in portrait photography, photographers often use large apertures and telephoto lenses to create a shallow depth of field effect, blurring the background and highlighting the subject's facial expressions and details. In landscape photography, small apertures and wide-angle lenses are more likely to be used to create a deep depth of field effect, making the entire scene clearly visible.

Figure 1 Depth of field effect diagram

3. How to achieve depth of field effect

There are many ways to achieve depth of field effects, which can be mainly divided into the following three categories: camera simulation, rendering technology and post-processing, as shown in the figure below.

Figure 2 How to achieve depth of field effect

3.1 Camera Simulation

In photography, camera simulation is the most direct way to achieve depth of field effect. By adjusting the various parameters of the camera, you can simulate photos with different depth of field effects.

① Aperture control: Aperture is a key factor affecting depth of field. The larger the aperture (the smaller the aperture value, such as f/2.8), the shallower the depth of field and the more obvious the background blur effect. The smaller the aperture (the larger the aperture value, such as f/16), the deeper the depth of field and the elements in the picture are relatively clear.

②Focal length adjustment: When shooting with a telephoto lens, the depth of field is relatively shallow and the background blur effect is significant. A short-focus lens will make the depth of field deeper and the picture clearer.

③ Shooting distance: The closer the shooting distance, the shallower the depth of field and the more obvious the background blur. On the contrary, the farther the shooting distance, the deeper the depth of field.

3.2 Rendering Technology

In computer graphics and game engines, rendering technology is an important means to achieve depth of field effects. Through algorithm simulation and calculation, images with depth of field effects can be generated in real time or offline.

① Depth information calculation: During the rendering process, the distance of each pixel in the scene relative to the focus (i.e. depth information) is calculated. This is usually done by calculating the Z coordinate of the pixel in the viewing cone.

② Application of blur algorithm: Use blur algorithm (such as Gaussian blur, radial blur, etc.) to blur pixels far from the focus. The degree of blur can be determined based on the depth information of the pixel. The farther from the focus, the higher the degree of blur.

③Depth Map: Depth Map is a commonly used technique to simulate the depth of field effect of a camera. It generates a depth map during the rendering process, which records the distance information from each pixel in the scene to the camera. Then, the image is blurred based on this depth map.

3.3 Post-processing

Post-processing is another important way to achieve depth of field effect. Editing and adjusting photos or images through professional image processing software can further optimize the depth of field effect.

① Blur filter: In post-processing software, you can use the blur filter to blur the image. By adjusting the filter parameters (such as blur radius, blur intensity, etc.), you can control the degree and range of blur.

② Color and contrast adjustment: By adjusting the color and contrast of the image, the depth of field effect can be further enhanced. For example, increasing the saturation and contrast of the foreground can make the foreground more prominent, while reducing the saturation and contrast of the background can make the background more blurred and soft.

③ Layers and masks: In some complex post-processing, you may need to use layers and masks to precisely control the depth of field effect. By creating different layers and applying different blur filters and adjustment effects, and then using masks to composite these layers, you can achieve a more sophisticated depth of field effect.

4. How is the depth of field effect achieved on mobile phones?

We have introduced many ways to achieve depth of field effects. So how do you achieve depth of field effects on photos on mobile phones? Because it is processing existing photos, it is obvious that post-processing is used. To achieve depth of field effects on photos on mobile phones, there are mainly two steps:

① Use AI to process images and distinguish the subject foreground from the background.

② Get the foreground part of the picture, and lay it out in layers with the lock screen component and the original picture to achieve a 3D visual layered effect.

Figure 3 Flow chart of realizing depth of field effect on mobile phone

4.1 Using AI to process images

To achieve the layered effect, you need to first segment the original image and extract the foreground and background areas (in fact, it is simpler to just extract the foreground, which will be introduced later when reorganizing the implementation). To extract the foreground and background of the image, you need to use image segmentation technology. The purpose of image segmentation is to assign each pixel or set of pixels in the image to a specific category or object, so that pixels in the same category or object are similar in some attribute, while pixels between different categories or objects have significant differences in this attribute. A simple schematic diagram of image segmentation is shown below.

Figure 4 Schematic diagram of image segmentation

4.2 Recombination to achieve depth of field effect

Figure 5 Layered settings

After we obtain the foreground and background images of the image, we can rearrange them and the lock screen display controls. The specific layout rules are as follows:

① The first layer (bottom layer) is used to place the image background or original image

② The second layer (middle layer) is used to place controls related to information display

③The third layer (top layer) is used to place the main body of the image, that is, the foreground part of the image obtained by image segmentation

Based on the above-mentioned layered layout method, a 3D-like depth of field effect can be achieved.

5. Practice depth of field effect on Android

In order to verify the above solution, this article practices and implements a method to achieve depth of field effect for ordinary pictures on Android. The specific solution is as follows:

① Based on Huawei's end-side intelligent deep learning framework, achieve image segmentation effect and obtain the image foreground image.

② Use relative layout to layer the original image, content controls, and foreground image to achieve a depth of field effect.

The figure below shows the process of implementing Depth Effect on Android.

Figure 6 Android implementation of Depth Effect flow chart

The first step is to read the original image and perform image segmentation. You can choose one from the existing image segmentation libraries on the market. This article uses Huawei's machine learning kit ML Kit. The specific access method is not described in detail in this article. If you are interested, you can download the demo from the official website. Run the model locally, analyze and process the input image, and obtain the bitmap data of the original image and foreground image.

The second step is to obtain the foreground part of the original image and perform layered layout. In the XML layout file, set the positions of the foreground image, content controls, and original image through relative layout. When the image is segmented successfully, set the obtained foreground image bitmap data to the corresponding ImageView to achieve the depth of field effect. The following is the layout style for achieving the depth of field effect:

After rendering, the final effect is as shown in the figure below. The upper part is the original image, and the lower part is the 3D layered effect achieved after processing.

Figure 7 Depth of field effect achieved on Android

6. Summary

The use of depth of field effects in mobile systems and applications makes the mobile phone interface more beautiful and layered, improving the user experience. With the continuous maturity of AI technology and the continuous improvement of end-side computing power, the application of end-side AI will become more intelligent, personalized and diversified. At the same time, we also look forward to seeing more innovative application scenarios and solutions emerge, bringing users a richer and more exciting experience.

Author: Zheng Rui

Unit: China Mobile Smart Home Operation Center