Re-understanding the R8 compiler from an online question

background

In the past period of time, JD Android APP has optimized the size of the installation package through a series of means such as image compression, image transfer download, resource obfuscation compilation, plug-inization, plug-in post-installation, and hybrid development, and achieved good weight loss benefits. After completing these conventional weight loss optimizations, in order to further optimize the size of the installation package, we investigated the newly launched R8 compiler by Google and learned that the R8 compiler can improve the build efficiency while optimizing the package size. So we started to try to upgrade the AGP version to enable R8 compilation. The upgrade process was not very smooth and encountered the following problems.

Introduction to obfuscation tools

For Android applications, code obfuscation is often used as one of the means to improve the security of the application. Code obfuscation is to convert the source code into a form that is functionally equivalent but difficult to read and understand, which reduces the readability of the code. Even if it is successfully decompiled, it is difficult to derive the true meaning of the code. Code obfuscation can increase the difficulty of decompiling and cracking the application. On the other hand, after code obfuscation, the names of classes, methods or fields are mapped to short and meaningless names, which can also reduce the size of the application package.

01ProGuard

Before AGP3.4.0, ProGuard was used as the default optimization tool in the Android packaging process. ProGuard optimizes the source code in the following four steps: shrink, optimize, obfuscate, and preverify.

• Shrink: remove unused classes, methods, fields, etc.
• Optimize: bytecode optimization, method inlining and other operations;
• Obfuscate: rename class names, method names, field names, etc. with short and meaningless names to increase the difficulty of decompilation;
• Preverify: Preverify the class.

The above four stages can be run independently and are enabled by default. You can turn off the corresponding stages by setting the -dontshrink, -dontoptimize, -dontobfuscate, and -dontpreverify rules in the obfuscation configuration file. After ProGuard performs code compression optimization and obfuscation on the .class file, it will be handed over to the D8 compiler for desugaring and converting the .class file into a .dex file. The execution process is as follows:

Figure 1 ProGuard and D8 optimization process

02R8

After AGP 3.3.0, Google officially introduced R8, which is a replacement for ProGuard, but is compatible with ProGuard's keep rules. R8 integrates optimization processes such as code desugaring, compression, obfuscation, optimization, and dex processing (D8) into one step. After enabling R8 compilation, the project construction efficiency is better than ProGuard in the actual development process. The compilation process is as follows.

Figure 2 R8 optimization process

• Tree shaking: detect and safely remove unused classes, fields, methods, and properties from your app and its library dependencies;
• Resource shrinking: Remove unused resources from your app, including unused resources in your app's library dependencies.
• Obfuscation: shorten class and member names;
• Optimization: Optimize bytecode, simplify code, and more to further reduce the size of your app’s DEX file. For example, if R8 detects that the else {} branch of a given if/else statement is never taken, R8 removes the code for the else {} branch.

When Android officially releases the Release package, we usually enable the optimization function by setting:

 release {
 // Enable code convergence
 minifyEnabled true
 // Enable resource compression
 shrinkResources true
 // Define ProGuard obfuscation rules
 proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
 }

Pitfall experience

01R8 Confusion Rules

1.1 Problem Symptom

First, let's take a look at the mapping.txt file generated after the AGP 3.3.3 build tool packages the obfuscation and observe the mapping relationship of the data classes after obfuscation. The OtherBean data class has an exclusion rule added to the obfuscation configuration file - keep class com.jd.obfuscate.bean.OtherBean{ *; }, while the WatermelonBean data class has no keep rule added. In addition, these two data classes have three fields with the same name: name, color, and shape:

 com.jd.obfuscate.bean.OtherBean -> com.jd.obfuscate.bean.OtherBean:
 java.lang.String name -> name
 java.lang.String color -> color
 java.lang.String shape -> shape
 java.lang.String otherPrice -> otherPrice
 java.lang.String otherComment -> otherComment
 java.lang.String otherProducer -> otherProducer
 java.lang.String otherCategory -> otherCategory
 int otherWeight -> otherWeight
 int otherQuality -> otherQuality
 int otherScore -> otherScore
 com.jd.obfuscate.bean.WatermelonBean -> com.jd.obfuscate.bean.a:
 java.lang.String name -> name
 java.lang.String color -> color
 java.lang.String shape -> shape
 java.lang.String price -> aMR
 java.lang.String comment -> aMS
 java.lang.String producer -> aMT
 java.lang.String cate -> aMU
 int weightAttr -> aMV
 int qualityAttr -> aMW
 int scoreAttr -> aMX

Carefully observe the obfuscated class names and field names of the two data classes. Since the WatermelonBean data class does not have a keep rule added, it is found that the class name and other field names of the WatermelonBean data class are obfuscated into meaningless names, while the fields name, color, and shape are not obfuscated.

Next, let's take a look at how the WatermelonBean data class will be obfuscated after upgrading AGP 3.6.4 and enabling R8 compilation:

 com.jd.obfuscate.bean.WatermelonBean -> bbaaa:
 java.lang.String name -> a
 java.lang.String color -> b
 java.lang.String shape -> c
 java.lang.String price -> d
 java.lang.String comment -> e
 java.lang.String producer -> f
 java.lang.String cate -> g
 int scoreAttr -> j
 int weightAttr -> h
 int qualityAttr -> i

After enabling R8 compilation, it was found that the name, color, and shape fields in the WatermelonBean data class without the keep rule were also obfuscated.

When interacting with the server during development, when using frameworks such as Gson and FastJson to parse server data, deserialization or reflection is involved. The corresponding object data class cannot be confused, and corresponding keep rules should be added, otherwise the json data cannot be parsed into the correct object. So if your project fails to check that all network parsing data classes have been added with corresponding keep rules before the application goes online after upgrading AGP, then there will be problems with business functions.

1.2 Cause Analysis

What causes the above phenomenon?

① When you upgrade AGP and enable R8 compilation, if you look carefully at the packaging log, you will find a warning about the "-useuniqueclassmembernames" obfuscation rule:

 AGPBI: {"kind":"warning","text":"Ignoring option: -useuniqueclassmembernames",
 "sources":[{"file":"xxx/proguard-project.txt","tool":"R8"}

②For those who are familiar with ProGuard obfuscation rules, you should be able to tell at a glance that the above obfuscation phenomenon is caused by the "-useuniqueclassmembernames" obfuscation rule. The explanation of this rule is: generate globally unique obfuscated names for member variables with the same name in different classes after obfuscation. If this rule is not set, methods or fields of different classes may be mapped to meaningless names such as a, b, and c.

③Remove the "-useuniqueclassmembernames" obfuscation rule and verify. Let's look at the obfuscated mapping file of the WatermelonBean data class:

 com.jd.obfuscate.bean.WatermelonBean -> com.jd.obfuscate.bean.a:
 java.lang.String name -> a
 java.lang.String color -> b
 java.lang.String shape -> c
 java.lang.String price -> d
 java.lang.String comment -> e
 java.lang.String producer -> f
 java.lang.String cate -> g
 int weightAttr -> h
 int qualityAttr -> i
 int scoreAttr -> j

It was found that the fields name, color, and shape in the WatermelonBean data class were all obfuscated into letters, confirming that the obfuscation rule caused the class name to be obfuscated while some of its field names were not obfuscated.

How does Proguard obfuscate field names?

Developers who have developed custom Android Gradle plugins should be familiar with the Gradle Transform abstract class. Gradle Transform is a set of standard APIs officially provided by Android to developers to modify .class files during the conversion of class files to dex files during the project build phase. Through these APIs, bytecodes can be manipulated to achieve common functions.

AGP source code viewing method, add dependencies in the project:

 implementation 'com.android.tools.build:gradle:3.3.3'

Locate the source code to com.android.build.gradle.internal.transforms.ProGuardTransform:

 public void transform(final TransformInvocation invocation) {
 final SettableFuture<TransformOutputProvider> resultFuture = SettableFuture.create();
 Job job = new Job(this.getName(), new Task<Void>() {
 public void run(Job<Void> job, JobContext<Void> context) throws IOException {
 ProGuardTransform.this.doMinification(invocation.getInputs(),
 invocation.getReferencedInputs(), invocation.getOutputProvider());
 }
 }, resultFuture);
 SimpleWorkQueue.push(job);
 job.awaitRethrowExceptions();
 }

Create an asynchronous task processed by ProGuard and add it to the WorkQueue queue, execute doMinification() -> runProguard() -> (new ProGuard(this.configuration)).execute();

 //Core method, perform corresponding operations according to the obfuscation rule configuration
 public void execute() throws IOException {
 System.out.println(VERSION);
 // Check the GPL license agreement
 GPL.check();
 if (configuration.printConfiguration != null) {
 // Print configuration file
 printConfiguration();
 }
 // Check whether the obfuscation rules are configured correctly
 new ConfigurationChecker(configuration).check();
 if (configuration.programJars != null &&
 configuration.programJars.hasOutput() &&
 new UpToDateChecker(configuration).check()) {
 return;
 }
 if (configuration.targetClassVersion != 0) {
 configuration.backport = true;
 }
 // Read all class files into class pools, programClassPool and libraryClassPool
 readInput();
 if (configuration.shrink || configuration.optimize ||
 configuration.obfuscate || configuration.preverify) {
 // Clear all JSE pre-verification information in the class
 clearPreverification();
 }
 if (configuration.printSeeds != null || configuration.shrink ||
 configuration.optimize || configuration.obfuscate ||
 configuration.preverify || configuration.backport) {
 // Retrieve class dependencies
 initialize();
 }
 if (configuration.printSeeds != null)
 // Output the kept class to the seeds.txt file
 printSeeds();
 }
 if (configuration.shrink) {
 // Perform compression optimization
 shrink();
 }
 // Optimize code instructions according to the set optimization level: -optimizationpasses 5
 if (configuration.optimize) {
 for (int optimizationPass = 0;
 optimizationPass < configuration.optimizationPasses;
 optimizationPass++) {
 if (!optimize(optimizationPass+1, configuration.optimizationPasses)) {
 break;
 }
 // Shrink again, if we may.
 if (configuration.shrink) {
 // Don't print any usage this time around.
 configuration.printUsage = null;
 configuration.whyAreYouKeeping = null;
 //Compress and optimize again
 shrink();
 }
 }
 // After optimizations such as method inlining and class merging, eliminate the line numbers of all program classes
 linearizeLineNumbers();
 }
 if (configuration.obfuscate) {
 // Execute the obfuscation steps
 obfuscate();
 }
 if (configuration.preverify) {
 // Pre-check
 preverify();
 }
 ......
 }

The above is the basic process of ProGuard execution. Let's focus on the obfuscate() obfuscation method:

 The execute() method performs the actual obfuscation operation:
 new Obfuscator(configuration).execute(programClassPool, libraryClassPool);

The Obfuscator class is the class that actually does the obfuscation, including class obfuscation ClassObfuscator and member obfuscation MemberObfuscator.

In the executed method, two processing logics about the "-useuniqueclassmembernames" rule were found:

 // If the class member names have to correspond globally,
 // link all class members in all classes, otherwise
 // link all non-private methods in all class hierarchies.
 ClassVisitor memberInfoLinker =
 configuration.useUniqueClassMemberNames?
 (ClassVisitor)new AllMemberVisitor(new MethodLinker()) :
 (ClassVisitor)new BottomClassFilter(new MethodLinker());
 programClassPool.classesAccept(memberInfoLinker);


 // Create a visitor for marking the seeds.
 NameMarker nameMarker = new NameMarker();
 ClassPoolVisitor classPoolvisitor =
 new KeepClassSpecificationVisitorFactory(false, false, true)
 .createClassPoolVisitor(configuration.keep,
 nameMarker,
 nameMarker,
 nameMarker,
 null);
 // Mark the seeds.
 programClassPool.accept(classPoolvisitor);


 // Come up with new names for all class members.
 NameFactory nameFactory = new SimpleNameFactory();
 // Maintain a map of names to avoid [descriptor - new name - old name].
 Map descriptorMap = new HashMap();


 // Do the class member names have to be globally unique?
 if (configuration.useUniqueClassMemberNames) {
 // Collect all member names in all classes.
 programClassPool.classesAccept(
 new AllMemberVisitor(
 new MemberNameCollector(configuration.overloadAggressively, descriptorMap)));


 // Assign new names to all members in all classes.
 programClassPool.classesAccept(new AllMemberVisitor(
 new MemberObfuscator(configuration.overloadAggressively, nameFactory, descriptorMap)));
 } else { ...... }

Obfuscation is divided into six steps:

Step 1: If the "-useuniqueclassmembernames" obfuscation rule is set, first create a MethodLinker visitor object through ClassVisitor)new AllMemberVisitor(new MethodLinker()), convert the field information in all classes into a linked list and connect them together, use the field name and field type as the key, query whether the visitorInfo information of the field already exists in the memberMap, if not, call the lastMember() method to try to get the visitorInfo of the field in the linked list and store it in the memberMap; if it can be queried, add the field information as visitorInfo to the linked list of field information:

 public void visitAnyMember(Clazz clazz, Member member) {
 // Get the name and descriptor of the field
 String name = member.getName(clazz);
 String descriptor = member.getDescriptor(clazz);
 String key = name + ' ' + descriptor;
 Member otherMember = (Member)memberMap.get(key);
 if (otherMember == null) {
 // Get the last method in the chain.
 Member thisLastMember = lastMember(member);
 // Store the new class method in the map.
 memberMap.put(key, thisLastMember);
 } else {
 // Link both members.
 link(member, otherMember);
 }
 }
 public static Member lastMember(Member member) {
 Member lastMember = member;
 while (lastMember.getVisitorInfo() != null &&
 lastMember.getVisitorInfo() instanceof Member) {
 lastMember = (Member)lastMember.getVisitorInfo();
 }
 return lastMember;
 }
 private static void link(Member member1, Member member2) {
 // Get the last methods in the both chains.
 Member lastMember1 = lastMember(member1);
 Member lastMember2 = lastMember(member2);
 // Check if both link chains aren't already ending in the same element.
 if (!lastMember1.equals(lastMember2)) {
 // Merge the two chains, with the library members last.
 if (lastMember2 instanceof LibraryMember) {
 lastMember1.setVisitorInfo(lastMember2);
 } else {
 lastMember2.setVisitorInfo(lastMember1);
 }
 }
 }

Step 2: Mark the class name, method or field name to which the keep rule is added (NameMarker) so that it will not be confused. The ProGuard source code makes extensive use of the visitor pattern, accessing the object pool by creating a nameMarker object of the ClassVisitor implementation class, and ultimately executing the method in the NameMarker class:

 public void visitProgramClass(ProgramClass programClass) {
 //Mark the class name in the keep rule
 keepClassName(programClass);
 // Make sure any outer class names are kept as well.
 programClass.attributesAccept(this);
 }


 public void keepClassName(Clazz clazz) {
 ClassObfuscator.setNewClassName(clazz, clazz.getName());
 }

 public void visitProgramField(ProgramClass programClass,
 ProgramField programField) {
 //Mark the field name in the keep rule
 keepFieldName(programClass, programField);
 }

 private void keepFieldName(Clazz clazz, Field field) {
 MemberObfuscator.setFixedNewMemberName(field, field.getName(clazz));
 }

 // Give the field a fixed name
 static void setFixedNewMemberName(Member member, String name) {
 VisitorAccepter lastVisitorAccepter = MethodLinker.lastVisitorAccepter(member);
 if (!(lastVisitorAccepter instanceof LibraryMember) &&
 !(lastVisitorAccepter instanceof MyFixedName)) {
 lastVisitorAccepter.setVisitorInfo(new MyFixedName(name));
 } else {
 lastVisitorAccepter.setVisitorInfo(name);
 }
 }


 public void visitProgramMethod(ProgramClass programClass,
 ProgramMethod programMethod) {
 //Mark the method name in the keep rule
 keepMethodName(programClass, programMethod);
 }

 private void keepMethodName(Clazz clazz, Method method) {
 String name = method.getName(clazz);
 if (!ClassUtil.isInitializer(name)) {
 MemberObfuscator.setFixedNewMemberName(method, name);
 }

It is relatively simple to mark the field keep name, just set it through lastVisitorAccepter.setVisitorInfo(name).

Step 3: Collect the mapping relationships of all members in all classes (MemberNameCollector), first obtain the keep name (visitorInfo) marked in the previous step from the field list, and put the methods or fields of the same type into the same Map<obfuscated name, original name>:

 public void visitAnyMember(Clazz clazz, Member member) {
 String name = member.getName(clazz);
 // Get the member's new name.
 // Get the visitorInfo of the member in the linked list
 String newName = MemberObfuscator.newMemberName(member);
 // keep the name
 if (newName != null) {
 // Get the member's descriptor.
 String descriptor = member.getDescriptor(clazz);
 if (!allowAggressiveOverloading) {
 descriptor = descriptor.substring(0, descriptor.indexOf(')')+1);
 }
 // Put the [descriptor - new name] in the map,
 // creating a new [new name - old name] map if necessary.
 Map nameMap = MemberObfuscator.retrieveNameMap(descriptorMap, descriptor);
 String otherName = (String)nameMap.get(newName);
 if (otherName == null ||
 MemberObfuscator.hasFixedNewMemberName(member) ||
 name.compareTo(otherName) < 0) {
 // Put methods or fields with the same descriptor into the same
 // Map<obfuscated name, original name>
 nameMap.put(newName, name);
 }
 }

Step 4: Create an obfuscator name. If no mapping is found in the keep name list, create a new obfuscator name (MemberObfuscator):

 public void visitAnyMember(Clazz clazz, Member member) {
 String name = member.getName(clazz);
 // Get the member's descriptor.
 String descriptor = member.getDescriptor(clazz);
 // Get the name map, creating a new one if necessary.
 Map nameMap = retrieveNameMap(descriptorMap, descriptor);
 // Get the member's new name.
 // 1. If there is a keep name in front, no obfuscation will be performed.
 // If not, assign a new obfuscated name
 String newName = newMemberName(member);
 // Assign a new one, if necessary.
 if (newName == null) {
 // Find an acceptable new name.
 nameFactory.reset();
 do {
 // 2. Generate a new name
 newName = nameFactory.nextName();
 }
 while (nameMap.containsKey(newName));
 // Remember not to use the new name again
 // in this name space.
 nameMap.put(newName, name);
 // Assign the new name.
 // 3. Set a new name for this member
 setNewMemberName(member, newName);
 }
 }
 static void setNewMemberName(Member member, String name) {
 MethodLinker.lastVisitorAccepter(member).setVisitorInfo(name);
 }

The NameFactory interface class is mainly responsible for generating new obfuscated names. If no custom obfuscationDictionary dictionary is set, the implementation class of the NameFactory interface, the SimpleNameFactory class, mainly generates new obfuscated names through the newName method:

 private static final int CHARACTER_COUNT = 26;

 private String newName(int index) {
 // If we're allowed to generate mixed-case names,
 // we can use twice as
 // many characters.
 int totalCharacterCount = generateMixedCaseNames?
 2 * CHARACTER_COUNT : CHARACTER_COUNT;
 int baseIndex = index / totalCharacterCount;
 int offset = index % totalCharacterCount;
 char newChar = charAt(offset);
 String newName = baseIndex == 0 ?
 new String(new char[] { newChar }):
 (name(baseIndex-1) + newChar);
 return newName;
 }


 private char charAt(int index) {
 return (char)((index < CHARACTER_COUNT ? 'a' -0 :
 'A' - CHARACTER_COUNT) + index);

CHARACTER_COUNT is defined as 26, which means exactly 26 letters. The nextName() method uses the index counter to increment each time a new name is generated, so ProGuard's obfuscated names start from a to z.

Step 5: Apply the obfuscated name, create a ClassRenamer visitor object, add the new obfuscated name to the constant pool through the ConstantPoolEditor object, and update the field name index u2nameIndex to point to the new obfuscated name.

 // Actually apply the new names.
 programClassPool.classesAccept(new ClassRenamer());


 public void visitProgramMember(ProgramClass programClass,
 ProgramMember programMember) {
 // Has the class member name changed?
 String name = programMember.getName(programClass);
 String newName = MemberObfuscator.newMemberName(programMember);
 if (newName != null && !newName.equals(name)) {
 programMember.u2nameIndex = new ConstantPoolEditor(programClass).addUtf8Constant(newName);
 }

Step 6: Constant pool compression. The new obfuscated name is created by adding new data to the constant pool. The original data is not deleted and needs to be repaired. Due to the limited space, we will not analyze it in detail.

Why doesn't the R8 compiler support this obfuscation rule?

From the above phenomenon, it can be seen that the R8 compiler ignored the "-useuniqueclassmembernames" obfuscation rule, but there is no relevant information about this rule in the Google Android development document "Reduce Application Size" user guide. Next, try to find the reason why this rule is ignored at the source code level. By searching for information and searching for the keyword "useuniqueclassmembernames" in the R8 source code repository, I found the code submission record that R8 does not support this rule. The "-useuniqueclassmembernames" obfuscation rule is mainly used for incremental obfuscation in ProGuard, but the main goal of introducing the R8 compiler is to further reduce the size of the application. If this rule is supported in R8, it will only increase the complexity of code obfuscation and bring no real benefits.

What other obfuscation rules does R8 not support?

Looking at the ProguardConfigurationParser.java source code (https://r8.googlesource.com/r8/+/3a100449ba5b490cd13d466b8c7e17dcd500722a/src/main/java/com/android/tools/r8/shaking/ProguardConfigurationParser.java) , the following obfuscation rules will be ignored by the R8 compiler:

 -forceprocessing
 -dontusemixedcaseclassnames
 -dontpreverify
 -experimentalshrinkunusedprotofields
 -filterlibraryjarswithorginalprogramjars
 -dontskipnonpubliclibraryclasses
 -dontskipnonpubliclibraryclassmembers
 -invokebasemethod
 -mergeinterfacesaggressively
 -android
 -shrinkunusedprotofields
 -allowruntypeandignoreoptimizationpasse
 -addconfigurationdebugging
 -assumenoescapingparameters
 -assumenoexternalreturnvalues
 -dump
 -keepparameternames
 -outjars
 -target
 -useuniqueclassmembernames

1.3 Summary of Confusion

If your project meets the following conditions at the same time, it is possible that during the AGP upgrade, field parsing may fail to obtain the corresponding value, resulting in abnormal business functions:

• Added "-useuniqueclassmembernames" obfuscation rule;
• There are multiple data classes with different business lines that contain some of the same fields;
• It cannot be guaranteed that all data classes have keep rules added when parsing network JSON data;
• Planning to upgrade AGP to enable R8 compilation.

Obfuscation suggestions for Android development:

• Develop a custom Gradle plug-in that checks whether the data class has added keep rules, and reminds developers in advance whether there is a risk of confusion through error logs during the project packaging and debugging phase;
• Best practices are formed for obfuscation to guide developers to use obfuscation rules correctly. For example, all data classes are placed in a unified package, and keep rules for the package name are added. If it involves deserialization or reflection, the @Keep annotation can also be added.

02v1 Signature Logic

2.1 v1 signature loss issue

The JD APP debugging package will read the v1 signature information. After upgrading to AGP 3.6.4, the debugging package crashed. After eliminating the cause, it was found that the v1 signature in the debugging package generated by the Android Studio run button was lost.

We usually make signature-related settings in the build.gradle file in the project app directory:

 signingConfigs {
 release {
 storeFile file('xxx')
 storePassword 'xxx'
 keyAlias ​​'xxx'
 keyPassword 'xxx'
 v1SigningEnabled true
 v2SigningEnabled true
 }
 }
 buildTypes {
 debug {
 minifyEnabled false
 shrinkResources false
 zipAlignEnabled true
 signingConfig signingConfigs.release
 }
 release {
 minifyEnabled true
 shrinkResources true
 zipAlignEnabled true
 signingConfig signingConfigs.release
 }
 }

2.2 Cause Analysis

By consulting the AGP 3.6.4 source code, it is found that the IncrementalPackagerBuilder will be created in the PackageAndroidArtifact class when executing the doTask() static method, which involves signature-related code. The code is as follows:

 public IncrementalPackagerBuilder withSigning(
 @Nullable SigningConfigData signingConfig, int minSdk,
 @Nullable Integer targetApi) {
 boolean enableV1Signing =
 enableV1Signing(
 signingConfig.getV1SigningEnabled(),
 signingConfig.getV2SigningEnabled(),
 minSdk,
 targetApi);
 boolean enableV2Signing = (targetApi == null || targetApi >= NO_V1_SDK)
 && signingConfig.getV2SigningEnabled();
 creationDataBuilder.setSigningOptions(
 SigningOptions.builder()
 .setKey(certificateInfo.getKey())
 .setCertificates(certificateInfo.getCertificate())
 .setV1SigningEnabled(enableV1Signing)
 .setV2SigningEnabled(enableV2Signing)
 .build());
 } catch (KeytoolException|FileNotFoundException e) {
 throw new RuntimeException(e);
 }
 return this;
 }

 private static int NO_V1_SDK = 24;
 static boolean enableV1Signing(boolean v1Enabled, boolean v2Enabled,
 int minSdk, @Nullable Integer targetApi) {
 if (!v1Enabled) {
 return false;
 }
 // If there is no v2 signature specified we have to sign with v1 even if the versions are
 // high enough otherwise we would not have signed at all
 if (!v2Enabled) {
 return true;
 }
 // Case where both v1Enabled==true and v2Enabled==true
 return (targetApi == null || targetApi < NO_V1_SDK) && minSdk < NO_V1_SDK;

As can be seen from the enableV1Signing() method above, targetApi refers to the system version of the mobile phone connected to the computer. It will determine whether to enable v1 signature based on whether the system version of the currently connected test machine is less than Android 7.0. If the mobile phone system is greater than Android 7.0, the v1 signature will be invalid.

Solution: Set targetApi to null through reflection and add the following settings to the build.gralde file:

 project.afterEvaluate {
 project.android.getApplicationVariants().all { appVariant ->
 String variantName = appVariant.name.capitalize()
 Task packageTask = project.tasks.findByName("package${variantName}")
 try {
 if (packageTask.getTargetApi() != null) {
 Field field = packageTask.getClass().getSuperclass().getSuperclass().getDeclaredField("targetApi")
 field.setAccessible(true)
 field.set(packageTask, null)
 }
 } catch (Exception e) {
 e.printStackTrace()
 }
 }
 }

Judging from the change records submitted in the source code IncrementalPackagerBuilder.java, the logic of enabling v1/v2 signatures has changed many times.

03Packing process

3.1 Phenomenon

During the packaging process, JD APP uses APT technology to identify annotations in the code, and then generates a json file with the annotation information and stores it in the project assets directory. The json file will then be packaged into the apk. However, after upgrading AGP, the json file is lost during packaging, resulting in functional abnormalities. The cause was found to be that the execution order of some tasks in the packaging process changed:

 Before upgrading AGP:
 > Task :AndroidPhone:compileXXXJavaWithJavac
 > Task :AndroidPhone:mergeXXXAssets
 After upgrading AGP:
 > Task :AndroidPhone:mergeXXXAssets
 > Task :AndroidPhone:compileXXXJavaWithJavac

Because the annotation processing logic is in the compileXXXJavaWithJavac task, after the execution order of the above two tasks changes, the task of merging assets resource files takes precedence over copying json files to the project assets directory, which eventually causes the json file to be lost in the apk package.

3.2 Solution

Set the two tasks to depend on each other, and add the following script to build.gradle:

 project.afterEvaluate {
 project.android.applicationVariants.all {
 def variantName = it.name.capitalize()
 Task compileJavaWithJavacTask = project.tasks.findByName("compile${variantName}JavaWithJavac")
 Task mergeAssetsTask = project.tasks.findByName("merge${variantName}Assets")
 mergeAssetsTask.dependsOn(compileJavaWithJavacTask)
 }
 }

AGP Upgrade Recommendations

• First, compare the apk products before and after the AGP upgrade to check whether there are any missing resource files;
• Create a script tool to compare the mapping.txt files generated before and after the upgrade to check whether the classes or fields to which the keep rules should be added are missing;
• Perform potential risk assessment for ProGuard obfuscation rules that are not supported by the R8 compiler and affect the global environment, such as the obfuscation rule "-useuniqueclassmembernames";
• Comprehensively assess the risks of upgrading AGP and prepare a downgrade plan.

Summarize

This article mainly introduces the pitfalls of JD APP during the upgrade of Android build tool AGP 3.6.4. After the upgrade, the package size was reduced by about 1.5MB. I hope the above pitfalls can help readers who plan to upgrade AGP.