A comprehensive guide to image formats that mobile developers should not miss

First, let's talk about several old-fashioned picture formats that everyone is familiar with:

JPEG is the most common image format. It was created in 1992 and is a very old format. It only supports lossy compression. Its compression algorithm can accurately control the compression ratio, exchanging image quality for storage space. Because it is so common, the CPUs of many mobile devices support hard encoding and hard decoding for it.

PNG was born in 1995, a few years later than JPEG. It was designed to replace the GIF format, so it has more similarities with GIF. PNG only supports lossless compression, so its compression ratio has an upper limit. Compared with JPEG and GIF, its biggest advantage is that it supports complete transparent channels.

GIF was born in 1987 and became popular with the first generation of the Internet. It has many disadvantages, such as usually only supporting 256 colors, only 1 bit of transparency channel, and low file compression ratio. Its only advantage is that it supports multi-frame animation. With this feature, it has been popular from the Windows 1.0 era to the present day, and it is still very popular.

After the birth of the above image formats, many companies or groups have tried to improve them or create other better image formats, such as JPEG 2000 by the JPEG group, JPEG-XR by Microsoft, WebP by Google, BPG and FLIF released by individual developers, etc. They have made great progress compared to the old image formats, but for various reasons, only a few formats can become popular. The following three are the emerging formats with relatively strong strength:

APNG is an image format released by Mozilla in 2008, which aims to replace the low-quality GIF animation. It is actually just an extension of the PNG format, so Mozilla has always wanted to merge it into the PNG standard. However, the PNG development team did not accept the APNG extension, but has been promoting its own MNG animation format. The MNG format is too complicated to be supported by any system or browser, and the APNG format has gradually become popular because it is simple and easy to implement. Mozilla's own Firefox first supported APNG, and then Apple's Safari also began to support it. Chrome has also tried to start supporting it. It can be said that the future prospects are very good.

WebP is an image format released by Google in 2010, hoping to replace JPEG with a higher compression ratio. It uses VP8 video intraframe coding as the basis of its algorithm and achieves good compression results. It supports lossy and lossless compression, supports complete transparent channels, supports multi-frame animation, and has no copyright issues. It is an ideal image format. With Google's influence in the online world, WebP has been widely used in a few years. Look at the apps in your phone: Weibo, WeChat, QQ, Taobao, NetEase News, etc., WebP is present in every app. Facebook has gone a step further and used WebP to display sticker animations in the chat interface.

BPG is an ultra-high compression ratio image format released by the famous programmer Fabrice Bellard last year (2014). Some people may not be familiar with this programmer, but everyone must know his works FFmpeg and QEMU. BPG uses HEVC (i.e. H.265) intra-frame coding as the basis of its algorithm. In this regard, it is undoubtedly the most advanced image compression format at present. Compared with JP2, JPEG-XR, and WebP, BPG can provide higher image quality at the same volume. In addition, thanks to its own characteristics based on the video encoding algorithm, it can save multi-frame animations in a very small file size. Fabrice Bellard is smart because he knows that he cannot get the support of major browser manufacturers alone, so he also specially developed a Javascript version of the decoder. Any browser can directly display BPG format images as long as it loads this 76KB JS file. The current reasons that hinder its popularity are the copyright issues of HEVC and its long encoding and decoding time. Although this image format has only been released for a year, many companies have already started trying it, such as Alibaba and Tencent.

Support for mobile image types

How do the mainstream mobile terminals support image formats? Let's take a look at the current image encoding and decoding architectures of Android and iOS respectively:

Android's image encoding and decoding is handled by the Skia graphics library. Skia implements encoding and decoding support for common image formats by hooking up third-party open source libraries. Currently, Android natively supports only JPEG, PNG, GIF, BMP, and WebP (added in Android 4.0), and the only encoding methods that can be directly called in the upper layer are JPEG, PNG, and WebP. Currently, Android does not support direct encoding and decoding of animated images.

The underlying layer of iOS uses ImageIO.framework to implement image encoding and decoding. Currently, the formats natively supported by iOS are: JPEG, JPEG2000, PNG, GIF, BMP, ICO, TIFF, PICT. Since iOS 8.0, ImageIO has added support for APNG, SVG, and RAW formats. At the upper level, developers can directly call ImageIO to encode and decode the above image formats. For animated images, developers can decode animated GIF and APNG, and encode animated GIF.

When importing third-party codec libraries, both platforms made some modifications to them. For example, Android made adjustments to libjpeg to better control memory, and iOS modified libpng to support APNG and added multi-threaded codec features. In addition, iOS developed AppleJPEG.framework specifically for JPEG codecs, achieving higher-performance hard encoding and hard decoding. Only when hard-coded decoding fails, libjpeg will be used.

Encoding and decoding of static images

Since I am currently mainly doing iOS development, the following performance evaluation is based on iPhone. The main test code can be seen here. There are very few test materials, only two:

[[154857]]

[[154858]]

The first one is the Dribbble logo, which includes an alpha channel and is used to test simple, graphic-like images.
The second classic Lena image is used to test photo-like images with rich details.
Each image has four resolutions: 64x64, 128x128, 256x256, and 512x512.
Too little test material may cause some tests to be inaccurate, but it is generally fine for reference.

JPEG

There are currently three well-known JPEG libraries:

libjpeg : The earliest developed and most widely used JPEG library. Since the JPEG standard is too complex and vague, no one else has implemented it, so this library is the de facto standard for JPEG.

libjpeg-turbo : A JPEG library dedicated to improving encoding and decoding speed. It is based on libjpeg and rewrites some codes with SIMD instruction sets (MMX, SSE2, NEON). The official website claims that it has a 2 to 4 times performance improvement over libjpeg.

MozJPEG : A JPEG library dedicated to improving compression ratio. It was released by Mozilla in 2014 and is based on libjpeg-turbo. It has a 5% to 15% compression ratio improvement over libjpeg, but its encoding speed is also much slower.

In addition to the above three libraries, Apple has also developed an AppleJPEG, but it is not open source. It calls the DSP hard encoding and decoding functions provided by the chip. Although it is not as functional as the above three libraries, its performance is very high. In my tests, its encoding and decoding speed is usually 1~2 times that of libjpeg-turbo. Unfortunately, developers cannot directly access this library at present.

Here are the ImageIO (AppleJPEG/libpng) encoding and decoding performance on iPhone 6:

It can be seen that the smaller the quality in JPEG encoding, the smaller the image size, the worse the quality, and the shorter the encoding time. The decoding time is not very different, probably because most of the time is spent on function calls and hardware calls. The default value of quality provided by Apple in its own photo album demo is 0.9. Around this value, a good balance can be achieved between image quality and size, as well as encoding and decoding time.

#p#

PNG

Compared with JPEG, the PNG standard is clearer and simpler, so many companies or individuals have their own PNG encoding and decoding implementations. However, the most widely used one is the libpng library officially released by PNG. The underlying layers of iOS and Android both call this library to implement PNG encoding and decoding.

Here is the PNG encoding and decoding performance on iPhone 6:

It can be seen that when encoding and decoding graphic-type images (few colors and details), the difference between PNG and JPEG is not that big; but for photo-type images (rich in colors and details), PNG is much worse than JPEG in file size and encoding and decoding speed.

Unlike JPEG, PNG is lossless compression and does not provide compression ratio options. Its compression ratio has an upper limit. There are many tools and services optimized for PNG on the Internet, aiming to improve the compression ratio of PNG. The following is a performance comparison of several common PNG compression tools:

pngcrush is a PNG compression tool that comes with Xcode. It can achieve good compression results compared to the images generated by designers using Photoshop. ImageOptim goes a step further and compares each image with multiple compression algorithms, selecting the one with the higher compression ratio, further reducing the file size. TinyPNG.com has an incredibly high compression ratio compared to other tools. It uses a color index table similar to GIF to compress PNG, so it will cause some details of rich color images to be lost. If using TinyPNG, it is best to let the designer see if the color effect is acceptable after the compression is completed.

WebP

The WebP standard is customized by Google, and so far only libwebp released by Google has implemented the codec. Therefore, this library is also the de facto standard for this format.

WebP encoding has several main parameters:

lossless: YES: lossy encoding NO: lossless encoding. The main advantage of WebP lies in lossy encoding, and its lossless encoding performance and compression ratio are average.

quality: [0~100] Image quality, 0 means the worst quality, the smallest file size, and severe loss of details, 100 means the highest image quality, and a larger file size. This parameter is only effective for lossy compression. Google's official recommendation is 75, and Tencent's recommendation for WebP evaluation is also 75. Around this value, WebP can achieve a good balance between compression ratio and image quality.

method: [0~6] Compression ratio, 0 means fast compression, short compression time, average compression quality, 6 means extreme compression, long compression time, good compression quality. This parameter is also effective only for lossy compression. Adjusting this parameter can bring a higher compression ratio of 20% to 40%, but the corresponding encoding time will increase by 5 to 20 times. The value recommended by Google is 4.

For encoding lossless images, quality=0, method=0~3 are relatively suitable parameters, which can save encoding time and have a good compression ratio. For lossless encoding images, quality=75, method=2~4 are relatively suitable parameters, which can achieve a good balance between encoding time, image quality and file size.

WebP decoding has three parameters:

use_threads: Whether to enable pthread multi-threaded decoding. This parameter only works for lossy images with a width greater than 512. When enabled, multi-threaded decoding will be used internally, the CPU usage will be higher, and the decoding time can be shortened by 10% to 20% on average.

bypass_filtering: Whether to disable filtering. This parameter only works for lossy images. After turning it on, it can shorten the decoding time by about 5% to 10%, but it will cause banding in some areas with smooth color transitions.

no_fancy_upsampling: Whether to disable upsampling. This parameter only works for lossy images. In my test, after turning on this parameter, the decoding time will increase by 5-25%, and some image details will be lost. The edges of the lines will increase noise and appear unnatural.

Normally, all three parameters can be set to NO. If you want to achieve higher decoding speed, you can try to enable the two parameters of use_threads and bypass_filtering. However, no_fancy_upsampling does not need to be enabled in any case.

Since there is a lot of WebP test data, only part of the test results of 512x512 size are posted here. If you are interested, you can see the Excel attachment at the end of the article.

For simple graphic images (such as various UI materials within an App), the file size and decoding speed of WebP lossless compression are in some cases even better than PNG. If you want to optimize the size of your App installation package, you can try WebP.

For complex images (such as photos), WebP lossless encoding performance is not good, but lossy encoding performance is very good. The decoding speed of WebP for images of similar quality is not much different from JPEG, but the file compression ratio can be greatly improved.

BPG

BPG is the best known lossy compression format, which can reduce the size of JPEG by 50% at the same quality. Below is a comparison of the classic Lena image, and you can also see a lot of other pictures here. The online comparison of BPG, JPEG, JPEG2000, JPEG-XR, WebP compression effects, the effect is very obvious.

Currently, only libbpg released by the author is available for BPG. However, the author has compiled a Javascript decoder based on libbpg, which greatly expands the scope of use. bpg can be encoded in two ways: lossless and lossy compression. In lossy compression, the compression ratio can be controlled by the quality parameter, which ranges from 0 to 51. The larger the value, the higher the compression ratio. Generally speaking, 25 is a good choice. The default value of the BPG official tool is 28.

libbpg is not currently optimized for ARM NEON, so its performance on mobile devices is average. The following is a performance test on an iPhone 6:

Since bpg encoding takes too long, I did not put the data in a table. You can see that at the same quality, BPG decoding speed is still much slower than JPEG, about 3 to 5 times slower. At present, BPG is suitable for those places that are very sensitive to traffic but not to decoding time. From the news on the Internet, Taobao Mobile and QQ Mobile have tried it, but it is not clear whether they have optimized BPG decoding.

Encoding and decoding of dynamic images

Animated images are very popular on the Internet. They are similar to videos, but are usually simple to implement, small in file size, and widely used. The ancestor of animated images is GIF, which has been popular on the Internet since the Windows 1.0 era and is still difficult to be replaced by other formats today. Although it is very old, the principles used are almost the same as several emerging formats today.

Here is a QQ emoticon in GIF format:

[[154859]]

This expression is made up of 6 static images. Each image has a certain survival time, and playing them continuously forms an animation:

In these pictures, most of the content is similar. In order to compress the file size, the animation format usually supports some special methods to crop similar pictures, retaining only the different parts of the previous and next frames:

When decoding animated images, decoders usually use the so-called "canvas mode" for rendering. Imagine that the playback area is a canvas. Before the first frame is played, the canvas is cleared and then the first frame is completely drawn. When the second frame is played, the canvas is no longer cleared, but only the area different from the first frame is overlaid on the canvas, just like the creation of an oil painting.

The first frame like this is called a key frame (i.e., an I frame, an intra-frame coded frame), and the subsequent frames obtained through compensation calculations are called predictive coded frames (P frames). A well-compressed animated image usually has only a few key frames, and the rest are predictive coded frames; an animated image produced by a poor compression tool is basically full of key frames. Different animated image compression tools usually produce different results.

In addition, animated image formats usually have more detailed parameters to control the drawing process of each frame. The following are several common parameters for GIF/APNG/WebP:

#p#

Disposal Method
Do Not Dispose: Draw the current frame increment onto the canvas without clearing the canvas.
Restore to Background: Before drawing the current frame, clear the canvas to the default background color.
Restore to Previous: Before drawing the next frame, restore the canvas to the previous frame of the current frame

Blend Mode
Blend None: When drawing, all channels (including the Alpha channel) will be overlaid on the canvas, which is equivalent to clearing the specified area of ​​the canvas before drawing.
Blend over: When drawing, the alpha channel will be composited into the canvas, which is the effect of overlapping two images in normal situations.

The above technologies are the basis of common animated image formats. The following introduces the characteristics of different animated image formats.

GIF

GIF has obvious flaws: it usually only supports 256-color indexed colors, which means it can only simulate richer colors through dithering, interpolation, etc.; its alpha channel has only 1 bit, which means that a pixel can only be completely transparent or completely opaque.

[[154862]]

This is a demo picture from Tencent Blog. You can see that GIF has serious "rough edges" due to the problem of Alpha channel. The current common solution is to add a circle of white edges on the edge of the image to reduce this visual effect:

[[154863]]

If you take a close look at the animated emoticons in apps like QQ and WeChat, you will find that almost every emoticon is surrounded by a circle of white border. This is a very helpless solution.

There are many tools for making GIF, but few tools have good effects and high compression ratio. For GIFs that have already been made, using imagemagick to process them can reduce the file size a lot. If you need to convert a video to GIF, Cinemagraph Pro is a good fool-proof tool. Here is an article that introduces how to compress GIF with ffmpeg. Although the parameter adjustment is a bit troublesome, the effect is very ideal.

Below is a non-optimized GIF and a GIF optimized and encoded by ffmpeg. You can see that the difference is very large.

[[154864]]

[[154865]]

APNG

APNG is not officially accepted by PNG yet, so libpng cannot decode APNG directly. But since APNG is just a simple extension based on PNG, it is easy to implement APNG encoding and decoding with a small amount of code on platforms that already support PNG. Chromium only added less than 600 lines of code to support APNG playback, and I also implemented a simple APNG encoding and decoding tool with about 500 lines of C code. In addition, on browsers that support canvas, you can use apng-canvas to directly display APNG animations. The best tool for APNG compression is currently apngasm, and most graphical tools such as Tencent's iSparta are developed based on this tool.

For now, APNG is the best alternative to GIF: it is simple to implement, has a wide range of applications, has a good compression ratio, and has good display effects.

WebP

WebP did not support animated images when it was released in 2010. In 2012, when libwebp v0.2 was released, Google began to try to support animations, but its implementation had many problems and the performance was so poor that the Chrome team never accepted it. It was not until 2013, when libwebp v0.4 was released, that the animation format was stabilized and accepted by Chrome.

WebP animated images are actually multiple single-frame WebP data simply packaged into one file, rather than being extended from a single-frame WebP, so the animated image format is not upward compatible with static images. If you want to support animated images, you first need to add the demux module when compiling libwebp. When decoding WebP, you need to first try to unpack it with WebPDemuxer, and then decode the unpacked single frame with WebPDecode. To facilitate compilation, I wrote a script for packaging iOS static libraries, adding mux and demux modules.

Google provides two simple command line tools for making animated images: gif2webp can convert GIF to WebP, and webpmux can package multiple WebP images into animated images, and there are many adjustable parameters. These two tools are not very good at compressing similar frames, so that in some cases the compression ratio is not as good as APNG, but there is no other better tool to use.

BPG

BPG itself is based on HEVC (H.265) video encoding, and was designed with animated images in mind. Because it fully utilizes the high compression ratio of HEVC and the characteristics of video encoding, its animated image compression ratio far exceeds that of other formats. Here and here are a few examples of BPG animated images, where you can see that at the same quality, BPG animated images are only a few tenths the size of APNG/WebP/GIF.

I wrote a simple method to decode animated images using libbpg here, you can refer to it if you need it.

Animation performance comparison

I converted the following GIF into WebP, APNG, and BPG, and decoded all its frames on an iPhone 6.

[[154866]]

The evaluation results are as follows:

APNG has a slight advantage over GIF in file size, and the decoding time is similar. WebP has a big advantage in both file size and decoding time. BPG has the biggest advantage in file size, but also takes the longest decoding time. So APNG and WebP are both good choices, while BPG still needs performance optimization.

Original link: http://blog.ibireme.com/2015/11/02/mobile_image_benchmark/