Things to know about Android compilation

As an Android engineer, we go through countless compilations every day. For small projects, compilation can be completed in half a minute or 1 or 2 minutes, while for large projects, each compilation may take the time of a cup of coffee. Maybe you will understand more if I tell you the specific numbers. When I was in the WeChat team, it took 5 minutes to fully compile the Debug package, and it took more than 15 minutes to compile the Release package.

If each compilation can be reduced by 1 minute, the WeChat Android team can save 1,200 minutes (40 people in the team × 30 compilations per day × 1 minute). Therefore, optimizing the compilation speed is very important for improving the development efficiency of the entire team.

So how should we optimize the compilation speed? What efforts have WeChat, Google, Facebook and other domestic and foreign manufacturers made? In addition to the compilation speed, what else do you need to know about compilation?

[[282337]]

Compile

Although we compile every day, what exactly is compilation?

You can simply understand compilation as the process of converting a high-level language into a low-level language that can be recognized by a machine or virtual machine. For Android, this process is the process of converting Java or Kotlin into Dalvik bytecode that can be run by the Android virtual machine.

The entire compilation process involves steps such as lexical analysis, syntax analysis, semantic checking, and code optimization. For students who are interested in the underlying compilation principles, you can challenge the three classic masterpieces of compilation principles: Dragon Book, Tiger Book, and Whale Book.

But today our focus is not on the underlying compilation principles, but we hope to discuss what problems Android compilation needs to solve, what challenges we are currently facing, and what solutions domestic and foreign manufacturers have given.

Basic knowledge of Android compilation

Whether it is WeChat compilation optimization or Tinker project, both involve a lot of compilation-related knowledge, so I have done a lot of research on Android compilation and have rich experience. The Android compilation and build process mainly includes three parts: code, resources, and Native Library. The entire process can refer to the build flow chart in the official document.

Gradle is the official compilation tool of Android and it is also an open source project on GitHub. From the update log of Gradle, we can see that the current project is updated very frequently, and basically there will be a new version every one or two months. For Gradle, I feel the most painful is the writing of Gradle Plugin, mainly because Gradle does not have a complete documentation in this regard, so generally you can only rely on reading the source code or breakpoint debugging. Recently, my company is going to use Gradle to develop a channel packaging tool, and I have a deep understanding of the project packaging and building process.

However, compilation is so important, and each company has its own unique situation, so they have to build their own "wheels". Open source projects include Facebook's Buck and Google's Bazel.

Why do we need to "reinvent the wheel" ourselves? There are several reasons:

• Unified compilation tools. Facebook and Google both have dedicated teams responsible for compilation work. They hope that all internal projects use the same set of build tools, including Android, Java, iOS, Go, C++, etc. All projects will benefit from the unified optimization of compilation tools;
• Code organization and management architecture. Facebook and Google have a very special feature in code management, that is, all projects of the entire company are placed in the same warehouse. Therefore, the entire warehouse is very large, so they will not use Git. Currently, Google uses Piper, and Facebook is based on HG, which is also a distributed file system;
• The pursuit of extreme performance. Buck and Bazel do have better performance than Gradle, and they have various compilation optimizations. But they all have some customization flavors, for example, they do not support external dependencies such as Maven and JCenter very well.

"Programmers hate writing documents, and some people don't write documents." Therefore, their documents are relatively few, and it will be painful if you want to do secondary customized development. If you want to switch the compilation tool to Buck and Bazel, you need to make a big decision, and you also need to consider the collaboration with other upstream and downstream projects. Of course, even if we don't use them directly, their internal optimization ideas are also worth learning and referring to.

Gradle, Buck, and Bazel all aim to achieve faster compilation speeds and more powerful code optimization. Let's take a look at what they have done.

Compilation speed

Think back to our Android development career, how much time and life we ​​have wasted on compiling. As I said before, compiling speed is very important to team efficiency.

Regarding compilation speed, we are most concerned about the speed of compiling the Debug package, especially the speed of incremental compilation. We hope to achieve faster debugging. As shown in the figure below, each code verification must go through two steps: compilation and installation.

Here, we look at the Android compilation speed from two dimensions: compilation time and installation time.

• Compile time. Compile Java or Kotlin code into ".class" files, and then compile them into Dex files through dx. For incremental compilation, we hope to compile as little code and resources as possible, and ideally only compile the changed parts. However, due to the dependencies between codes, this is not feasible in most cases. At this time, we can only settle for the second best and hope to compile fewer modules. Android Plugin 3.0 and later versions use Implementation instead of Compile in order to optimize dependencies;
• Installation time. We have to go through signature verification first, and after the verification is successful, there will be a lot of file copying work, such as APK files, Library files, Dex files, etc. After that, we also need to compile Odex files, which is very time-consuming especially on Android 5.0 and 6.0. For incremental compilation, the best optimization is to directly apply the new code without reinstalling the new APK.

For incremental compilation, let me first talk about Instant Run, the official solution of Gradle. Before Android Plugin 2.3, it used Multidex implementation. After Android Plugin 2.3, it uses the Split APK mechanism newly added in Android 5.0.

As shown in the figure below, resources and Manifest are placed in the Base APK. The code in the Base APK only contains the Instant Run framework, and the application's own code is in the Split APK.

Instant Run has three modes. For hot swap and warm swap, we do not need to reinstall the new Split APK. The difference lies in whether to restart the Activity. For cold swap, we need to reinstall the changed Split APK through adb install-multiple -r -t, and the application also needs to be restarted.

Although we don't need to reinstall the Base APK in either mode, which makes Instant Run look very good, but in large projects, its performance is still very poor, mainly due to:

• Multi-process problem. "The app was restarted since it uses multiple processes". If the app has multiple processes, hot swap and warm swap will not work. Since most apps have multiple processes, the speed of Instant Run will be greatly reduced.
• Split APK installation problem. Although the installation of Split APK does not generate Odex files, there will still be signature verification and file copying (the ping-pong mechanism of APK installation). This time takes several seconds to tens of seconds, which is unacceptable.
• Javac issue. Before Gradle 4.6, if the Annotation Processor was used in the project, then sorry, this modification and its dependent modules need full javac, and this process is very slow, which may take dozens of seconds. This problem was not solved until Gradle 4.7. You can refer to this issue for a discussion on the cause of this problem.

You can also look at this issue: "full rebuild if a class contains a constant". If the modified class contains a "public static final" variable, then this modification and its dependent modules also need full javac. Why is this? Because the constant pool will directly compile the value into other classes, Gradle does not know which classes may use this constant.

When asked by Gradle staff, the solution they gave was the following:

 // Original constant definition:
 public   static final int MAGIC = 23 
 
 // Replace constant definitions with methods:
 public   static   int magic() {
 return 23;
 }

For large projects, this is definitely not feasible. As I wrote in the issue, regardless of whether we actually changed the constant, Gradle will mindlessly run the full javac, which is definitely wrong. In fact, we can compare the code modification to see if the value of a constant has actually changed.

However, students who have used Alibaba's Freeline or Mogujie's fast compilation may have questions, why don't their solutions encounter problems with annotations and constants?

In fact, their solutions are faster than Instant Run in most cases, but that’s because they sacrifice correctness. In other words, in pursuit of faster speed, they ignore the compilation errors that may be caused by changes in annotations and constants. As the official solution, Instant Run gives priority to ensuring 100% correctness.

Of course, Google has also discovered various problems with Instant Run. After Android Studio 3.5, a new solution "Apply Changes" will be used to replace Instant Run for devices running Android 8.0 and later. I haven't found more information about this solution yet, but I think the Split APK mechanism should be abandoned.

I have always had an ideal compilation solution in my mind. The Base APK installed by this solution is still just a shell APK, and the real business code is placed in ClassesN.dex of Assets. Its architecture diagram is as follows.

• No installation is required. Still use a method similar to Tinker hot fix, just insert the modified and dependent classes to the front of pathclassloader each time. Students who are not familiar with it can refer to the Qzone solution in "The Evolution of WeChat Android Hot Patch Practice";
• Oatmeal. In order to solve the time-consuming Odex problem of ClassesN.dex in Assets during the first run, we can use the black technology in ReDex mentioned in "Installation Package Optimization": Oatmeal. It can generate a fully interpreted Odex file in less than 100 milliseconds;
• Turn off JIT. We turn off the JIT optimization of the virtual machine by adding android:vmSafeMode="true" in AndroidManifest, so that the problem encountered by Tinker in Android N mixed compilation will not occur.

I have a few more suggestions for optimizing compilation speed:

Replace the compilation machine. For companies with strong financial strength, it is easiest to directly replace the compilation machine with a Mac or other more powerful device;

Build Cache. You can separate most of the projects that do not change often and use the remote cache mode to keep the compiled cache;

Upgrade Gradle and SDK Build Tools. We should upgrade the latest compilation tool chain in time to enjoy the latest optimization results of Google;

Use Buck. Buck is more efficient, whether it is exopackage or incremental compilation of code. But as I said before, if a large project wants to switch to Buck, there are actually many concerns. WeChat connected to Buck in early 2014, but due to problems with collaboration with other projects, it switched back to the Gradle solution in 2015.

In comparison, the Hot Reload second-level compilation feature in Flutter, which is currently the hottest, may be more attractive.

Of course, in the recent versions of Android Studio, Google has also made a lot of other optimizations, such as using AAPT2 to replace AAPT to compile Android resources. AAPT2 implements incremental compilation of resources, which splits resource compilation into two steps: Compile and Link. The former compiles resource files in binary form into Flat format, and the latter merges all files and then packages them.

In addition to AAPT2, Google also introduced d8 and R8. Here are some test data provided by Google, as shown below.

So what are d8 and R8? In addition to optimizing compilation speed, what other functions do they have? You can refer to the following introduction: Android D8 and R8

Code Optimization

For Debug package compilation, we are more concerned about speed. But for Release package, code optimization is more important because we care more about application performance.

Next, I will talk about ProGuard, d8, R8 and ReDex, four code optimization tools that we may use.

ProGuard

In the 12-minute compilation process of the WeChat Release package, ProGuard alone takes 8 minutes. Although ProGuard is really slow, it is used in almost every project. After adding ProGuard, the application build process is as follows:

ProGuard mainly has three functions: obfuscation, tailoring, and optimization. Its entire processing flow is as follows:

The optimizations include more than 30 types such as inlining, modifiers, merging classes and methods, etc. You can refer to the official documentation for detailed introduction and usage.

D8

Android Studio 3.0 introduced d8, which officially became the default tool in 3.1. Its function is to compile ".class" files into Dex files, replacing the previous dx tool.

In addition to faster compilation speed, d8 also has an optimization to reduce the size of generated Dex. According to Google's test results, there will be an optimization of about 3% to 5%.

R8

R8 was introduced in Android Studio 3.1, and its ambition is even higher. Its goal is to replace ProGuard and d8. We can directly use R8 to convert ".class" files into Dex.

At the same time, R8 also supports the three major functions of ProGuard: obfuscation, trimming, and optimization. Since R8 is still in the experimental stage, there is not much information about it online. You can refer to the following information:

ProGuard vs R8:

ProGuard and R8: a comparison of optimizers.

Jake Wharton’s blog has a lot of R8-related articles recently: https://jakewharton.com/blog/.

The ultimate goal of R8 is the same as that of d8, one is to speed up compilation, and the other is to provide more powerful code optimization.

ReDex

If R8 is the tool that will replace ProGuard in the future, then ReDex, which is used internally by Facebook, has already done it. Many projects within Facebook have switched to ReDex and no longer use ProGuard. Unlike ProGuard, its direct input object is Dex, not ".class" files, which means it directly optimizes the final product, and what you see is what you get.

In previous articles, I have mentioned the ReDex project more than once because its functions are so powerful. For details, please refer to the previous article in the column "Package size optimization (Part 1): How to reduce the size of the installation package?"

• Interdex: class reordering and file reordering, Dex sub-packaging optimization;
• Oatmeal: directly generated Odex file;
• StripDebugInfo: Remove Debug information in Dex.

In addition, ReDex features such as Type Erasure and removing Aceess methods in code are also very good features. They are helpful for both package size and application running speed, so I also encourage you to study and practice their usage and effects.

However, the ReDex documentation has not been updated for years, and it contains some Facebook internal customized logic, so it is indeed very inconvenient to use. At present, I mainly study its source code directly, refer to its principles, and then implement it directly.

In fact, there are many useful things in Buck, but the documentation still doesn’t mention anything, so you still need to “read the source code”.

Library Merge and Relinker

• Multi-language split
• Subcontracting support
• ReDex support

Continuous Delivery

Gradle, Buck, and Bazel all represent compilation in a narrow sense. I think compilation in a broad sense should include processes such as packaging and building, code review, code engineering management, and code scanning, which is the continuous integration that the industry has often mentioned recently.

The most commonly used continuous integration tools are Jenkins, GitLab CI, Travis CI, etc. GitHub also provides its own continuous integration service. Every large company has its own continuous integration solution, such as Tencent's RDM, Alibaba's Ferris Wheel, Dianping's MCI, etc.

Let me briefly talk about my experience and views on continuous integration:

• Custom code checking. Every company has its own coding standards. The purpose of code checking is to prevent non-compliant code from being submitted to the remote repository. For example, WeChat has defined a set of code standards and has written special plug-ins to detect them. For example, log standards, new Thread and new Handler cannot be used directly, and violators will be punished. Custom code detection can be completely implemented by yourself or by extending the Findbugs plug-in. For example, Meituan uses Findbugs to implement the Android vulnerability scanning tool Code Arbiter;
• Third-party code inspection. The commonly used code scanning tools in the industry include the paid Coverity and Facebook's open source Infer, which can scan out many problems such as null pointers, multi-threading problems, resource leaks, etc. In addition to increasing the detection process, my biggest experience is that it is necessary to increase staff training at the same time. I have met many developers who directly judge null pointers as null and directly lock multi-threads in order to solve the problems found by scanning, which may cause more serious problems in the end;
• Code Review. For Code Review, integrating GitLab, Phabricator or Gerrit is a good choice. We must attach importance to Code Review, which is also an opportunity to show others our "great" code. And we should be the first Code Reviewer. Before reviewing for others, we should review the code from a third-party perspective. In this way, we can pass the test of this level first, which not only respects other people's time, but also establishes a good technical brand for ourselves.

There are many processes involved in continuous integration, and you need to combine them with the current situation of your team. If you just blindly add processes, it may sometimes be counterproductive.

Summarize

In Android 8.0, Google introduced the Dexlayout library to rearrange classes and methods, and Facebook's Buck also introduced AAPT2 as soon as possible. ReDex, d8, and R8 actually complement each other. It can be seen that Google is also absorbing the knowledge of the community, but at the same time we will also seek ideas from Google's new technology development.

I had another experience when writing today's content. Google spent a lot of effort to solve the problem of Android compilation speed, but the result was not satisfactory. I want to say that if we dare to jump out of the constraints of the system, we may be able to completely solve this problem, just as we can perfectly achieve second-level compilation on Flutter. In fact, the same is true for being a person and doing things. We often fall into the dilemma of local optimal solutions or enter a "thinking circle". At this time, if we can jump out of path dependence and rethink and examine the overall situation from a higher dimension, the experience we get may be completely different.