In-depth understanding of the Android graphics system

1. Introduction to Graphics System

The graphics system is one of the most important subsystems in a computer. The computers and mobile phones we usually use all have graphical interfaces. For ordinary people, a computer without a graphical interface is almost unusable. Today we will talk about the principles behind the graphics system.

1.1 The birth of the graphics system

Early computers did not have graphical interfaces, but command line interfaces. People sat in front of the terminal to input commands, execute commands, and wait for the commands to be completed, and so on. Such computers are more suitable for scientific researchers and science and engineering men, but it is impossible to popularize them to thousands of households. Later, Xerox Palo Alto Research Center (Xerox PARC) took the lead in developing computers with graphical interfaces and proposed the concept of WIMP. WIMP is Window, Icon, Menu, and Pointer. Our current computers are still in WIMP mode. Unfortunately, Xerox did not make computers with graphical interfaces, but it was carried forward by Steve Jobs and Bill Gates. When Steve Jobs visited the PARC, he was stunned by the graphical interface they showed. After returning, he immediately started to develop a graphical interface operating system in his own company. Bill Gates found that Apple's graphical interface was indeed good, and he also started to make his own graphical interface, so there was the Windows system. Later, Apple and Microsoft even sued over the issue of the graphical interface.

1.2 Overall Structure of the Graphics System

Compared with command line mode, the programming mode and software structure of the graphics mode have changed greatly. In command line mode, programmers only need to consider the flow of the program itself, and then deal with the terminal through standard input and output. But when it comes to graphics mode, everything changes. Programmers must first consider how to draw the interface of the program, and then process various events such as program clicks through message loops. Not only has the programming mode of programmers changed, but the way the operating system is implemented has also changed greatly. In command line mode, the operating system only needs to provide a shell, which continuously reads and executes commands. However, in graphics mode, the operating system must first provide a desktop as the starting point for users to use the computer, and also provide a file manager to facilitate users to view and manage files. For programmers, the operating system must also provide a graphics programming interface, a rendering library, and is responsible for synthesizing and displaying all windows. So an important and huge subsystem, the graphics system, was born in the operating system. According to the description in the previous few sentences, let's first look at the simple structure of the graphics system.

It can be seen that the overall structure of the graphics system is quite simple. The GUI process needs a window system to create and manage windows, a rendering system to help draw the interface, and finally a display system to display the image on the monitor.

1.3 Functions of each layer of the graphics system

Knowing the overall structure of the graphics system, let's describe the functions of each layer in detail.

The window system generally runs in the user space as a process, and we call its process DisplayServer. The window system has two responsibilities: one is the window manager, which is responsible for creating, scaling, and destroying windows; the other is the composition manager, which is responsible for compositing the windows drawn by each GUI process into a bitmap and then sending it to the display system for display.

The rendering system exists in the form of a so library and is loaded into the memory space of each GUI process. The rendering system is responsible for executing the drawing commands of the GUI process and generating the corresponding bitmap on the display buffer of the window. Rendering is divided into 2D rendering and 3D rendering. 2D rendering is generally performed by the CPU, and 3D rendering is generally performed by the GPU. However, 2D rendering is often done by the GPU now. However, many ordinary programs do not use the rendering library directly, but use the control library, because it is too troublesome to use the rendering library directly. For example, if we want to draw a button, it is very troublesome to use the rendering library API to draw it, but if we use the control library API, we only need to specify the position, size, style and other attributes to easily draw a button.

The display system exists in the kernel in the form of a driver. The driver is the screen controller driver or the DPU driver. The function of the display system is to display a bitmap formed by all windows on the display. The early display driver model was FBDEV, which was aimed at the screen controller. The screen controller had no computing power and could only receive the bitmap that the window system had synthesized for display. At this time, the synthesis manager of the window system would use the rendering system to synthesize the bitmaps of each window. Synthesis can also be seen as a special rendering. Later, the screen controller gradually developed into a DPU, which had computing power and could perform synthesis operations. At this time, a new display driver model DRM was also born. DRM allowed the window system not to perform synthesis operations, but to send the video memory of each window to itself, perform synthesis operations through the DPU, and then send it to the display for display.

2. Android Graphics System

Android is one of the most popular mobile operating systems. Today we will analyze Android's graphics system in detail.

2.1 Framework Overview

Before talking about Android, let's take a look at the graphics system of Linux distributions. Since the kernel of Android is also Linux, their display systems are the same. Linux's rendering system uses OpenGL and the latest Vulkan, and the control library uses GTK (GNOME) or Qt (KDE). Linux's window system has a long and complex history, dating back to the UNIX era. We won't go into details here, let's just talk about the current situation. On Linux, the protocol and implementation of the window system are clearly separated. The window protocol used by Linux for a long time is called X Window, and the implementation is X.org. However, because X Window is too old, many designs do not meet the current situation, and there is a heavy historical burden. Therefore, someone designed a new window protocol Wayland, and the most popular implementation of Wayland is called Weston. Now most Linux distributions have begun to switch to Wayland/Weston.

Now that we know the graphics system of Linux distributions, let's take a look at the graphics system of Android.

Android's graphics system does not have a clear protocol, and the implementation is the protocol. This is because the Linux system is a standard open source system, and many things like to set a protocol first, and then anyone can implement this protocol. Although Android is also open source, it is directly implemented by Google and used by other manufacturers, so there is no need to set a protocol. The Android graphics system is still very different from the Linux graphics system in specific details. This is because the Linux graphics system is aimed at desktop systems, and the Android graphics system is aimed at mobile systems. The different usage environments and development environments of the two lead to different specific implementation details. One of the biggest differences is that there is no typical window concept in the Android graphics system. In other window systems, there is generally a CreateWindow interface to create a window, and the return value is the window handle, and then we can use this window handle to do other things. But in Android, this logic is not the case. The concept of the window is hidden and scattered in the specific implementation. Programmers are faced with Activity, View, and ViewGroup. The following sections will introduce the various parts of the Android graphics system.

2.2 Rendering System Overview

Android initially used OpenGL ES for 3D rendering and skia for 2D software rendering. Later, in order to optimize 2D rendering, hwui was developed for hardware rendering. hwui is a wrapper of OpenGL ES. Later, hwui called skia, which wrapped OpenGL ES for hardware rendering. Of course, skia also retained the software rendering part. Let's take a look at the figure below.

OpenGL ES system warpper is a standard interface library provided by the system. Its so location is fixed, which is convenient for program loading. Its interface is a standard interface, which is convenient for program use. However, it does not have any implementation logic. All implementation logic is in the closed library provided by GPU manufacturers.

Ordinary APKs do not use these rendering libraries directly, but use the control library provided by the system. Most of the controls provided by Android are in the packages android.view and android.widget.

2.3 Window System Overview

The window system has two responsibilities, the window manager and the composite manager. In Android, these two are not together. The window manager is implemented in the system_server process and is called WindowManagerService (WMS). It is implemented in Java because system_server is a Java process. Why does Android implement WindowManagerService in system_server? This is because there is ActivityManagerService (AMS) in system_server. The two are closely related and it is more appropriate to put them together. The composite manager is implemented in an independent process called SurfaceFlinger. At the beginning, SurfaceFlinger performed the composite directly. Later, due to the rise of hardware composite, SurfaceFlinger no longer performed the composite operation directly, but forwarded the composite operation to the underlying layer. WindowManagerService and SurfaceFlinger interact using Binder inter-process communication. Let's take a look at the figure below:

Google introduced a module called HWC (Hardware Compositor) to handle hardware synthesis. At first, HWC was just a so library running in the SurfaceFlinger process, and later HWC became an independent process. In HWC, there are many closed and semi-open source libraries provided by manufacturers.

This diagram does not show the interaction between the drawing and the APK. The interaction between the window system and the APK consists of two parts. First, when the program creates an Activity, it interacts with the WMS to create a window. There is no typical window concept in Android. PhoneWindow, DecorView, ViewRootImpl, and Surface can be combined together as the concept of a window. Another part that is not shown is the producer-consumer relationship between the rendering of the APK and the synthesis of SurfaceFlinger. This logic will be discussed in the next chapter.

If you want to learn AMS in depth, I recommend reading Luo's WMS article in his Android journey: https://blog.csdn.net/Luoshengyang/article/details/8462738

, and WMS analysis written by Yuan Huihui: http://gityuan.com/2017/01/08/windowmanger/

2.4 Display system overview

The display system is directly related to the screen and belongs to the driver in the kernel. The kernel generally has a driver model for any type of hardware, and all hardware manufacturers develop drivers based on this hardware model. The earliest driver model abstracted from the display was added to FBDEV. Later, with the development of hardware and software, a new driver model DRM was born. Now most systems have turned to DRM, so let's talk about DRM here. Let's draw a picture first:

This structure is actually the structure of many drivers. The kernel defines and implements the DRM Core. Hardware manufacturers extend the structure according to the requirements of the DRM Core, implement function pointers, and then call the registration function to register themselves. The hardware used in the user space will create a device file. The user space can open the device file and use ioctl to call various commands. The ioctl command is defined by the Core, and the specific driver must implement these commands. It is still troublesome for the user space to use the ioctl command directly, so there will also be a libdrm library to encapsulate various ioctl commands and convert them into function interfaces, so that the process is more convenient to use.

3. Producer Consumer Model

Before talking about rendering and compositing, let's first talk about the relationship between them and the process of their interaction.

3.1 Overview

Rendering and synthesis are producer-consumer relationships, so how do they interact with each other? Android implements a producer-consumer model BufferQueue, where producers and consumers interact through BufferQueue. BufferQueue manages GraphicBuffer. The content rendered by the producer is placed on GraphicBuffer, and the source of the consumer's synthesized content comes from GraphicBuffer. GraphicBuffer allocates memory through the Hidl interface Gralloc defined by Google, and Gralloc allocates memory through ION. ION is a cross-space and cross-device memory allocation method based on DMA-BUF. In order to speed up the generation and consumption process, BufferQueue can adopt an asynchronous mode, and when it is asynchronous, it needs to be synchronized. The method adopted for this is Fence. Fence is a cross-space and cross-device synchronization mechanism. Cross-space means between processes and between kernel and user space, and cross-device means between the drivers of two devices or between the driver and the process. Let's draw a picture to see their overall relationship.

3.2 BufferQueue

BufferQueue is the implementation of the production-consumption relationship model of rendering and compositing in Android. Let's first look at how to use BufferQueue.

 void BufferQueue::createBufferQueue(sp<IGraphicBufferProducer>* outProducer,
 sp<IGraphicBufferConsumer>* outConsumer)
 {
 sp<BufferQueueCore> core(new BufferQueueCore());
 sp<IGraphicBufferProducer> producer(new BufferQueueProducer(core));
 sp<IGraphicBufferConsumer> consumer(new BufferQueueConsumer(core));
 *outProducer = producer;
 *outConsumer = consumer;
 }

You can see that creating a BufferQueue is to create a BufferQueueCore, and then use this core as a parameter to create the producer basic interface and the consumer basic interface respectively. Generally, the BufferQueue is created in the consumer process, and then the producer interface is passed to the producer process across processes using Binder. Of course, it can also be the other way around, or both can be cross-process, or neither can be cross-process. The reason why BufferQueue is created in the consumer process in most cases is to make the consumer ready, so that the producer can consume it immediately after it is generated.

Generally speaking, we do not use the original producer or consumer interface directly, but encapsulate them layer by layer, so that the encapsulated interface is more convenient to use. Let's take a look at its encapsulation logic diagram.

This picture shows the use of BufferQueue by APK and SurfaceFlinger. It can be seen that the encapsulation of the original producer interface is generally Surface, but we often see SurfaceControl in the code. What's going on? This is to separate the control right from the drawing right. When the APK starts, it will request WMS to create a window, that is, Surface. WMS then requests SurfaceFlinger to create BufferQueue and obtain its original producer interface. WMS itself encapsulates the original producer as SurfaceControl to control the Surface. Then the original producer is encapsulated as Surface and passed to APK, so that APK only has the right to draw. If APK wants to set the properties of Surface, it has to ask WMS for help. Let's draw a picture below:

Next, let's take a look at the internal management logic of BufferQueue. BufferQueue manages GraphicBuffer, but it does not manage GraphicBuffer directly. Instead, it defines the BufferSlot structure. BufferSlot contains a smart pointer application to GraphicBuffer and a smart pointer reference to Fence, as well as BufferState. BufferQueueCore contains an array of BufferSlots with 64 elements. Since BufferSlots are all referenced by smart pointers, they are empty at first and are only allocated when used. When managing BufferSlots, BufferQueue does not directly operate on them, but manages their subscripts. Let's draw a picture to see.

BufferQueue uses 4 integer containers to manage BufferSlots. The subscripts of BufferSlots have different meanings in different containers. First, the 64 hard-coded definition of BufferQueue is all slots. After creating BufferQueue, we can use the interface function to set how many Buffers we want to use. The unused subscripts will be placed in the container mUnusedSlots, and the used subscripts will be placed in the container mFreeSlots. Then when we use a certain Buffer, whether it is used by the producer or the consumer, its subscript will be placed in the container mActiveBuffers. When the consumer finishes using a Buffer, it will be placed in the container mFreeBuffers. The difference between mFreeBuffers and mFreeSlots is that the former's BufferSlot is already associated with GraphicBuffer, while the latter is just an empty slot. We will talk about the state transition of Buffer in Section 3.4.

3.3 Video Memory Allocation and Synchronization

When we use BufferSlot for the first time, we will allocate GraphicBuffer. So how does GraphicBuffer allocate memory? GraphicBuffer allocates memory through the Gralloc interface defined by Google. The Gralloc interface is implemented through two Hidl interfaces IAllocator and IMapper. Let's draw a picture to see it.

You can see that the final method of allocating memory is ION. ION is a cross-space and cross-device memory allocation method. ION is based on DMA-BUF. Let's talk about DMA-BUF first.

DMA-BUF is a memory sharing mechanism across space and devices. It is just a framework and cannot allocate memory. DMA-BUF is neither DMA nor BUF, but Sharing. DMA-BUF defines two roles: Exporter, responsible for allocating memory. There can only be one importer in a system; Importer, also called User, responsible for using memory. There can be N of them, but generally there are two, one writer, i.e. producer, and one reader, i.e. consumer. Let's draw a picture to see:

Now that we understand DMA-BUF, let's take a look at ION. ION is built on the basis of DMA-BUF. ION can share memory between processes, between processes and the kernel, and between devices thanks to DMA-BUF. ION itself has many heaps, and different heaps are used to allocate different types of memory. ION uses the system heap by default. The code in the kernel can directly use ION's interface. In order to allow user space to use ION, ION creates a device file /dev/ion. User space can use various ioctl commands to use ION, which is obviously not very convenient, so libion ​​was created to help everyone use ION conveniently. To summarize, as shown in the following figure:

After the memory allocation of GraphicBuffer is completed, it can be used for rendering and synthesis. However, we can only perform synchronous operations now, while GPU rendering is asynchronous. In order to improve performance, we need a waiting notification mechanism under asynchronous use. For this reason, Fence is implemented in the kernel. It is mainly used for DMA-BUF, so it is also a cross-space and cross-device mechanism. Therefore, Fence is a wait/notify mechanism across devices and spaces. It has the same semantics as wait/notify in Java and wait/signal of conditional variables in C++. The difference is that the mechanisms in Java and C++ can only be used within the process. Another great feature of Fence is that its notify signal will not be lost. This is because Fence is disposable and thrown away after use. Each time it is used, it needs to be re-applied for one, and it cannot be reused. Therefore, Fences are numbered. Fences not only have numbers, but also contexts. Contexts can be created in different scenarios. Fence numbers under the same context are comparable, and the time with a smaller number is in the front. Fence numbers under different contexts are not comparable. Let's draw a picture to see.

3.4 Production and consumption process

After understanding the previous knowledge, we will look at the specific process of production and consumption. Let's first look at the state change of BufferSlot. The state change of BufferSlot is related to the process of production and consumption. Let's look at the picture first and then explain.

A BufferSlot is initially in the Free state. When the producer is ready to produce, it will first dequeueBuffer, at which point it will get a BufferSlot, and the BufferSlot state will also change to Dequeued. If the BufferSlot obtained was an empty slot before, it will allocate memory. The process was described in the previous section. If it is a slot that has already allocated memory, it will be used directly. Then the producer starts production and puts all the produced memory into the GraphicBuffer. When production is completed, queueBuffer will be called to tell the consumer that I have finished production and you can start consuming. After queueBuffer, the BufferSlot state changes from Dequeue to Queued. At this time, the GraphicBuffer will be encapsulated as a BufferItem structure and put into the mQueue queue. After the consumer receives the message, he will be ready to consume. The consumer first acquireBuffer, obtains a BufferItem from the mQueue queue, and the corresponding BufferSlot state is converted to Acquired. Then the consumer can start consuming. When the consumption is completed, it will call releaseBuffer to indicate that the consumption is completed and return the BufferSlot to the BufferQueue. At this time, the state of the BufferSlot returns to Free.

After understanding the state changes of BufferSlot and the basic process of generation and consumption, let's take a look at the process of generation and consumption under VSync and with Fence. Let's look at the picture first:

First of all, rendering and compositing are two independent threads, which are carried out at the same time. Both sides start to execute when the VSync signal is received. Secondly, rendering and compositing each have an additional thread to perform asynchronous rendering and compositing, which will not block the mainstream thread. The mainstream thread has no blocking operations and will not get stuck. Both asynchronous threads are waiting for the Fence signal and may get stuck. When an asynchronous thread is stuck all the time, such as the compositing thread is stuck, the rendering thread will also get stuck while waiting for the Fence signal, but the main thread can continue to run.

IV. Summary and Review

Through this article, we have a basic understanding of Android's graphics system, and a general understanding of the producer-consumer model of graphics rendering and synthesis. Let's review it again by looking at the picture below:

The graphics system consists of three parts: the rendering system, the window system, and the display system. The rendering system is responsible for helping the GUI process to draw the interface. The window system is responsible for allocating windows to the GUI process, managing windows, and synthesizing all surfaces. The display system is responsible for sending the synthesized images to the display for display. Hardware synthesis is more popular now, and the window system delegates the task of graphics synthesis to the display system through hardware.

References:

https://blog.csdn.net/hexiaolong2009/category_9281458.html

https://blog.csdn.net/hexiaolong2009/category_9705063.html

https://blog.csdn.net/hexiaolong2009/category_10331964.html

http://www.wowotech.net/sort/graphic_subsystem

About the Author:

Cheng Lei is a system development engineer at a major mobile phone manufacturer and honorary editor-in-chief of Yuemachang. His greatest hobby is studying the basic principles of the Linux kernel.