Facebook's Superpack data compression technology can reduce Android app size by 20%

Facebook software engineer Sapan Bhatia said in a blog post that the team developed a technology called Superpack that combines compiler analysis with data compression to find size optimizations beyond the capabilities of traditional compression tools. "Superpack pushes the limits of compression and achieves better compression rates than existing compression tools."

Bhatia said that managing app size at Facebook is a unique challenge: Every day, developers check in large amounts of code, and each line of code translates into additional bits in the app that are ultimately downloaded to users' phones. If left unchecked, this added code can make the app bigger and bigger until eventually it takes an unacceptable amount of time to download.

So the Facebook team has been working on Superpack technology. Bhatia said that over the past two years, Facebook has adopted Superpack to control the growth of developer-induced applications and successfully reduced the size of its Android applications. Compared with regular Android APK compression, the size of these applications has been greatly reduced. Compared with Android's default Zip compression, the average savings are more than 20%.

For example, the main Facebook app on Android has actually reduced usage by more than 40% using Superpack, while Messenger has achieved a reduction of more than 30%. See the following figure:

According to the introduction, the advantage of Superpack is that it can compress specific types of code (such as machine code and bytecode), as well as certain types of structured data. Superpack's underlying method is based on insights into the complexity metric of the Kolmogorov algorithm, which defines the information content of a piece of data as the length of the shortest program that can generate that data. This means that data can be compressed by representing it as a program that generates that data, rather than compressing the information itself. When the data is code at the beginning, it can be converted to data with a smaller compressed representation. The program that generates a list of Fibonacci numbers and their indices is a highly compressed representation of a file containing such numbers.

Bhatia said the idea of ​​reducing Kolmogorov complexity itself is not new in the field of compression. What is new about Superpack is that it combines a compiler approach with modern compression techniques to achieve this goal.

Superpack was originally targeted as a tool for reducing the size of mobile apps, but Facebook sees potential in many other areas where data compression is needed. The company is developing a new on-demand executable format that saves disk space by keeping shared libraries compressed and decompressed as they load; and is considering using Superpack for code delta compression to reduce the size of software updates.

Additionally, they are investigating using Superpack as a cold storage compressor to compress rarely used log data and files. "Currently, Superpack is only available to our engineers, but we are eager to bring the benefits of Superpack to everyone. To that end, we are exploring ways to improve the compatibility of our compression work with the Android ecosystem... We may one day consider open sourcing Superpack."

For more details, please check the official blog.

This article is reproduced from OSCHINA

Title of this article: Facebook's Superpack data compression technology can reduce Android App size by 20%

Article URL: https://www.oschina.net/news/160286/facebook-core-data-superpack