Alipay Android package size is extremely compressed

Preface

In this chapter, we will start a new series around "Analysis of Alipay App Construction Optimization", and discuss the specific implementation solutions of the client in dimensions such as "code management", "certificate management", "version management", and "construction packaging", so as to lead everyone to further understand Alipay's continuous optimization under the App construction module.

This section will mainly record how to improve the operation efficiency and quality by compressing the size of the Alipay Android package.

background

The importance of package size is self-evident. It directly affects user downloads and retention, and some manufacturers have mandatory pre-installation requirements that the package size must be less than a certain value. However, with the iterative development of the business, the application will become larger and larger, and the installation package will continue to expand, so reducing the package size is a long-term governance process.

plan

Alipay has also been working hard to optimize package size, and we have introduced many solutions.

• For example: proguard code obfuscation, image conversion from png to tinypng to webp, introduction of 7zip compression solution, etc.

This solution is different from the conventional solutions mentioned above. It directly deletes useless information in dex to instantly reduce the size of the Alipay package by 2.1M without affecting the entire operating logic and performance, and can even reduce the running memory a little.

Solution Introduction

introduction

Before discussing the detailed solution, let me briefly talk about the debugging logic of the entire Java system.

JVM loads .class files when running. In order to make the package size more compact and run more efficiently, Android invented the dalvik and art virtual machines. Both virtual machines run .dex files (of course, the art virtual machine can also run oat files at the same time, which is not discussed in this article).

Therefore, the information in the dex file is completely consistent with the information in the class file. The difference is that the dex file deduplicates the information in the class. A dex contains many class files, and there are relatively large differences in structure. The class is a streaming structure, and the dex is a partitioned structure. Each block is indexed by an offset. In the following, only the dex structure is mentioned, and the class structure is not mentioned. The structure of dex can be represented by the following figure:

The structure of the dex file is actually very clear, divided into several large blocks, header area, index area, data area, and map area. This optimization solution optimizes and deletes the debugItems area in the data area.

What is debugItem used for?

First, we need to know what is stored in debugItem?

It mainly contains two kinds of information:

• Function parameter variables and all local variables
• The correspondence between all instruction set line numbers and source file line numbers

What is the use:

• The first point is actually very obvious. Since it is called debugItem, it must be used for debugging. We usually use this area when using IDE for breakpoints and single-step debugging.
• The second function is to report a crash or actively obtain the call stack, because the virtual machine actually executes the instruction set when it is actually executed. The report stack will report the corresponding source file line number of the crash. At this time, the debugItem is used to obtain the corresponding line number. You can use the following screenshot to understand it more intuitively:

The above picture is a common crash information. The line number in the red box is obtained by searching for this debugItem.

How big is debugItem?

In the case of Alipay, the debug package is 4-5M, and the release package is about 3.5M, accounting for about 5.5% of the dex file size, which is consistent with Google's official data. If this part can be removed directly, wouldn't it be very tempting!

Can debugItem be removed directly?

Obviously not. If it is removed, all the reported crash information will have no line numbers, and all line numbers will become -1, which will make you very confused.

In fact, when running proguard, there is a configuration that can remove or retain this line number information. -keep SourceFile, LineNumberTable has this function. In order to facilitate problem location, almost all development retains this configuration.

Therefore, the core idea of ​​the solution is to remove debugItem, while at the same time getting the correct line number when reporting a crash. As for IDE debugging, this is easier to solve. We only need to process the release package, not the debug package.

Solution 1

The core idea is relatively simple, which is to make the line number search offline, so that the line number correspondence originally stored in the App can be extracted in advance and stored on the server. When the crash is reported, the line number is reversed through the line number table extracted in advance, solving the problem that the crash information is reported without line numbers and cannot be located.

Although the idea is simple, it is still a bit complicated to implement, and it is also tortuous to promote the launch. After several adjustments, the plan can be summarized by the following picture:

As shown above, there are four core points:

• Modify proguard, use proguard to delete debugItem (remove -keep lineNumberTable), and dump a temporary dex before deleting the line number table.
• Modify dexdump, dump the line number table relationship in the temporary dex into a dexcmapping file (the mapping relationship between the instruction set line number and the source file line number), and save it to the server.
• Hook the crash handler of the app runtime and report the instruction line number at the time of the crash to the reverse engineering platform.
• The decryption platform decrypts the correct line number by reporting the instruction set line number and preparing the dexpcmapping file in advance.

The above solution took more than two weeks to develop the entire demo. The other modification points were not very difficult, but the difficulty lies in reporting the instruction set line number.

We know that all crashes will eventually have a throwable object, which stores the entire stack information. After repeated reading of the source code and attempts, I found that the instruction set line number I wanted was actually in this object. This can be illustrated by the following simple diagram:

Before printing the crash stack information, each throwable will call a jni method provided by the art virtual machine and return an internal object called stackTrace which is saved in the Throwable object. The stackTrace object stores the call stack of the entire method, including the instruction set line number. When obtaining the actual stack information later, an art jni method will be called again and the stackTrace method will be passed. The underlying layer will reverse the instruction set line number in the stackTrace object to get the official source file line number.

Well, it’s actually very simple. Reflect and get the stackTrace object in this Throwable, get the instruction set line number, and then report it.

One point to note here is quite disgusting. The implementation of each virtual machine is different. First of all, the name of the internal object, some are called stackTrace, some are called backstrace, and then the type of this internal object is also very different, some are int arrays, some are long arrays, and some are object arrays, but they all have the instruction set line number. Different methods need to be used to parse this object for different virtual machine versions. It is necessary to be compatible with about 4 types of virtual machines, 4.x, 5.x, 6.x, 7.x, and the types after 7.x are unified.

Solution 2

The above solution is actually perfect, there is no compatibility issue. Deletion is done directly by using proguard, and the instruction set line number is obtained directly at the Java layer without various hooks. If you only need to handle crash reports, solution one is enough, but it is far from enough in many scenarios in Alipay.

for example:

• Performance, CPU, and memory exception call stack.
• Java call stack when native crashes.

All of the above cases involve stack information. Solution 1, which uses reflection to call the stackTrace internal object in throwable, cannot handle it at all, so a different method is needed.

My initial idea was to try to hook the art virtual machine. I would look through the source code every day to see what I could hook. But I gave up in the end. One reason was that I was worried about compatibility issues, and the other was that there were too many hook points and I was quite panicked.

Finally, I changed my mind and tried to modify the dex file directly, leaving a small debugItem, so that the instruction set line number and the source file line number are consistent when the system searches for line numbers. In this way, there is nothing to do, and any line number reported by the monitoring is directly converted to the instruction set line number. You only need to modify the dex file. This can be represented by the following diagram:

As shown above: Originally, each method would have a debugInfoItem, and each debugInfoItem would have a mapping relationship between the instruction set line number and the source file line number. The modification I made was actually very simple. I just deleted all the redundant debugInfoItems and left only one debugInfoItem. All methods point to the same debugInfoItem, and the instruction set line number in this debugInfoItem is consistent with the source file line number. In this way, no matter what method is used to check the line number, the instruction set line number is obtained.

There were many pitfalls. In fact, it is not enough to keep only one debugInfoItem. To be compatible with the search methods of all virtual machines, debugInfoItem needs to be partitioned, and the debugInfoItem table cannot be too large. One pitfall was that when performing dex2oat optimization on AndroidO, the debugInfoItem would be traversed frequently, resulting in slower AOT compilation. In the end, it was solved by partitioning debugInfoItem.

This solution is more thorough, and does not require modifying proguard or hooking native. However, if you only need to deal with the line number problem of the crash, then solution 1 is still the first choice. This solution requires a lot of changes. In the early stage, I also studied the file structure of dex every day, and worked on every detail. I only dared to change it when I was relatively confident.

summary

Currently, the solution has been officially launched on Alipay. After several rounds of external verification, it is relatively stable. The overall package size of Alipay has been reduced by about 2.1M, and the actual dex size has been reduced by about 3.5M.

Through this section, we have a preliminary understanding of how Alipay improves the efficiency and quality of App operation through package size compression on the Android client. Regarding the design ideas and specific practices of package size compression on the Android side, we also look forward to your feedback and welcome discussions and exchanges.