Research on audio and video playback technology in Android system

Author: Hu Rui, Unit: China Mobile Smart Home Operation Center

Labs Guide

With the popularity of smartphones and 5G technology, the demand for video playback has increased rapidly. Whether you watch short videos in your spare time or stay up late to watch TV series, it is no exaggeration to say that video playback is always with you and me. We often hear people talk about some professional terms such as MP4, MKV, H264, H265, AAC, 1080p, 60fps, etc. What do they mean? What is the process of video playback? How is the player framework of the Android platform, which has the largest market share today, designed? This article will take you through these key points of knowledge.

Part 01 Common professional terms

MP4, MKV, and WMV, which we usually talk about, are video file formats. In the industry, we call them containers. The data of audio and video files are stored in containers, and each container has a fixed format. Among them, MP4 is the most common international format. It can be used and played in common playback software and is widely used. Videos recorded with cameras on Android phones are generally in MP4 format. The biggest feature of MKV is that it can accommodate a variety of different types of encoded video, audio, and subtitle streams, commonly known as universal media containers. WMV is a streaming media format launched by Microsoft, which is an extension of the ASF format upgrade. Under the same video quality, the WMV format is very small in size, so it is very suitable for online playback and transmission. After converting WMV files to MP4 files of the same resolution through tools or commands, the file size will become larger. The following figure can roughly represent the relationship between file formats and data.

The original video data is in YUV format, and the original audio data is in PCM format. These two formats of data will take up a lot of storage space. In order to decompress the storage space occupied by audio and video data, the audio and video data needs to be compressed, which is what the industry calls encoding. When playing audio and video files, the compressed audio and video files need to be decompressed, which is decoding. Encoding and decoding are both carried out according to certain specifications. Different specifications have different names, such as H264, H265, and AAC. In other words, H264, H265, and AAC are codec formats. Among them, H265 and H264 are both video codec formats, and both are encoded based on macroblocks. Among them, H264 supports up to 16*16 macroblocks, while H265 supports up to 64*64 macroblocks. Therefore, H265 will be clearer than H264 in detail expression. In the case of consistent pixels, the compression ratio of H265 will be higher than that of H264. Since H265 has a higher compression ratio, it occupies less storage space, thus achieving the purpose of saving bandwidth and achieving smooth playback. This is also the reason why the current popular HD TV series use H265 encoding.

The aspect ratio we often hear refers to the ratio of the width and height of the video screen. Common ratios in life are 16:9 and 4:3. We also often hear terms such as standard definition, high definition, and ultra-high definition. The corresponding professional term is resolution. 720p and below are standard definition, 720p and 1080p are high definition, and 4k and 8k are called ultra-high definition. What exactly do 720p and 1080p mean? At this point, we have to talk about the concept of video resolution. Resolution refers to the pixel value of the width and height of the video, and the unit is Px. Usually the numerical aspect ratio of the video resolution should be equal to the aspect ratio, otherwise the video file will have black edges. The resolution of standard 1080p is 1920×1080, which means that there are 1920 pixels in width and 1080 pixels in height. The higher the resolution, the clearer the video. 30fps and 60fps refer to the frame rate. Video is composed of pictures one by one. Video playback is to show pictures one by one (one picture can also be simply understood as one frame) to the audience at a fixed rate, such as 30 frames per second (30fps) or 60 frames per second (fps), so that the audience feels that the action is continuous. Therefore, the frame rate is the number of frames displayed per second. The higher the frame rate, the smoother the video. The current mainstream video frame rate is 60 frames.

Currently, mainstream Android systems support H264 and H265 hardware encoding and decoding, which can well meet the needs of users. In addition, the Android system reserves convenient and simple interfaces for developers to implement encoding and decoding, such as using MediaRecorder to implement recording (encoding scenario) and using MediaPlayer to implement playback (decoding scenario). The MediaRecorder and MediaPlayer interfaces are very simple and easy to use, and the underlying encoding and decoding logic is shielded, which is very friendly to developers. Next, we will briefly introduce the process of video playback.

Part 02 Video playback process

The main process of a player playing a video is as follows:

1. Read the video file and parse the file format, that is, obtain the container format. There is a sniff function in the DataSource of the Android platform Media framework. This function calls the sniff function of each Extractor. The sniff function of each Extractor returns a score. DataSource will eventually select the Extractor with the largest score.

2. Use the corresponding container format parser (called Extractor on the Android platform) to parse the video file and obtain metadata, audio track, video track index, and audio and video encoder formats. If it is an MP4 file, Mp4Extractor will be used on the Android platform to parse the file;

3. Call the corresponding decoder to decode the audio and video data. On the Android platform, MediaCodec calls the OMX or C2 decoder plug-in for decoding, and returns the decoded data through MediaCodec;

4. Perform audio and video synchronization on the decoded data, and perform frame drop processing when necessary. When performing audio and video synchronization, the video is usually synchronized with the audio, and this is also done on the Android platform. The specific method is to compare the timestamp of the decoded video with the audio timestamp. If the difference between the video timestamp and the audio timestamp is greater than a certain set value, such as 200ms, then the current video frame is discarded;

5. Render the audio and video. On the Android platform, the audio data decoded by the decoder will be handed over to SurfaceFlinger for rendering. The decoded audio data will be processed by AudioTrack and finally output to the speaker.

Now we have a general understanding of the main process of video playback, but as a hard-working bricklayer, I always want to write a video player by myself and experience my own achievements. The following is a brief introduction on how to write a simple video player on the Android platform.

Part 03 Implementing the simplest player on Android

In fact, it is very simple to write a video player in Android. Using the MediaPlayer Java class provided by Android, you can easily implement a player.

 Sample Code:
 String url = https://www.xxx.com/video/test.mp4 ​
 MediaPlayer player = new MediaPlayer ();
 player .setDataSource ( url );
 player . prepare ();
 player . setDisplay ( surfaceView . getHolder ());
 player . start ();

You read that right, it’s that simple.

As a technology enthusiast, you must be curious about how Android MediaPlayer is implemented internally.

Part 04 Android MeiaPlayer player architecture

On the Android platform, the Java layer MediaPlayer only provides an externally exposed interface. In fact, the real logic implementation is in the Native MediaSever. The setDataSource of MediaPlayer will trigger the Native IPC request server to create a player instance, and prepare will trigger the server to find a suitable container parser (MediaExtractor) and parse it to obtain the required decoder information, so prepare is time-consuming. Start will trigger the server to create the corresponding decoder and start decoding, then synchronize the audio and video, and finally display it.

MediaCodec can be considered as a decoder component that interacts with the OMX or C2 decoder plugin.

Feed the decoder with data in the onInputBufferAvailable callback and provide data to the decoder through queueInputBuffer.

Process the decoded data in the onOutputBufferAvailable callback and plan the output buffer to the decoder through releaseOutputBuffer.

On the Android platform, in addition to MediaPlayer, there are many open source player frameworks available for us to use. Below we briefly introduce them in a table.

Part 05 Common open source players

Part 06 The future of Android audio and video

Audio and video technology will never be outdated in the digital age, and the demand will become more and more vigorous. We participate in video conferences at work, watch short videos in life, watch TV series with mobile phones, make video calls on WeChat, and use facial recognition to enter and exit buildings. These are all closely related to audio and video technology. With the development and popularization of 5G technology, there are more and more video scenes. At the same time, with the popularity of the concept of the metaverse, VR and AR have been pushed to the forefront, and VR and AR are closely related to audio and video. Currently, the VR all-in-one machines on the market are basically Android systems. The author believes that there will be more audio and video application scenarios, and audio and video technology will surely add more color to human digital life. ​