Please enjoy a comprehensive Android obfuscation feast

In the daily development process of Android, obfuscation is an essential skill for us to develop apps. As long as we have personally experienced the process of app packaging and online, we need to more or less understand some basic operations of code obfuscation. So, what exactly is obfuscation? What are its benefits? What are the specific effects? Don't worry, let's explore its "unique" charm one by one.

Introduction to Obfuscation

Obfuscated code is the act of converting the code in a program into a code that is difficult to read and understand according to certain rules.

The benefits of obfuscation

The benefit of obfuscation is its purpose: to make the APK difficult to reverse engineer, which greatly increases the cost of decompilation. In addition, the "obfuscation" in Android can also remove useless resources during packaging, significantly reducing the size of the APK. Finally, it can also avoid the common 64k method reference limit in Android in a workaround way.

Let's first look at the APK structure comparison before and after obfuscation:

From the two pictures above, we can see that after obfuscation, the package name, class name, member name, etc. in our APK are replaced with random, meaningless names, which increases the difficulty of reading and understanding the code and increases the cost of decompilation. Careful friends may also notice that the size of the APK before and after obfuscation has been reduced from 2.7M to 1.4M, which is nearly doubled! Is it really so magical? Haha, it is indeed so magical. Let us slowly unveil its mystery.

Confusion in Android

In Android, the "obfuscation" we usually talk about actually has two meanings, one is the obfuscation of Java code, and the other is the compression of resources. In fact, there is no connection between the two, but they are used together out of habit. So, after saying so much, how do we turn on obfuscation on the Android platform?

Enable obfuscation

 ......
 android {
 buildTypes {
 release {
 minifyEnabled true  
 shrinkResources true  
 proguardFiles getDefaultProguardFile( 'proguard-android.txt' ), 'proguard-rules.pro'  
 }
 }
 }

The above are the basic operations to enable obfuscation. You can enable obfuscation by setting minifyEnabled to true. At the same time, you can set shrinkResources to true to enable resource compression. It is not difficult to see that we usually enable obfuscation when we are building a release package, because obfuscation will increase the compilation time, so it is not recommended to enable it in debug mode. In addition, it should be noted that: enabling resource compression will only be effective if obfuscation is enabled! The proguard-android.txt in the above code represents the default obfuscation rule file provided by the Android system, and proguard-rules.pro is the obfuscation rule we want to customize. As for how to customize the obfuscation rules, we will talk about it next.

Code Obfuscation

In fact, the Java platform provides us with the Proguard obfuscation tool to help us quickly obfuscate the code. According to the official introduction of Java, the specific Chinese definition of Proguard is as follows:

• It is a tool that includes code file compression, optimization, obfuscation and verification functions
• It can detect and remove unused classes, variables, methods and properties
• It is able to optimize bytecode and remove unused instructions
• It can rename the names of classes, variables and methods to meaningless names to achieve obfuscation effect
• Finally, it will validate the processed code, mainly for Java 6 and above and Java ME

Resource compression

In Android, the compiler provides us with another powerful feature: resource compression. Resource compression can help us remove unused resources in the project and dependent repositories, effectively reducing the size of the APK package. Since resource compression and code obfuscation work together, if you need to enable resource compression, remember to enable code obfuscation first, otherwise the following problems will occur:

 ERROR: Removing unused resources requires unused code shrinking to be turned on . See http://d.android.com/r/tools/shrink-resources.html for more information.
 Affected Modules: app

Customize the resources to be retained

When we turn on resource compression, the system will remove all unused resources for us by default. If we need to keep certain specific resources, we can create a marked XML file in our project (such as res/raw/keep.xml), and specify each resource to be kept in the tools:keep attribute, and specify each resource to be discarded in the tools:discard attribute. Both attributes accept a comma-separated list of resource names. Similarly, we can use the character * as a wildcard. For example:

 <?xml version= "1.0" encoding= "utf-8" ?>
 <resources xmlns:tools= "http://schemas.android.com/tools"  
 tools:keep= "@layout/activity_video*,@layout/dialog_update_v2"  
 tools:discard= "@layout/unused_layout,@drawable/unused_selector" />

Enable strict checking mode

Normally, the resource shrinker can accurately determine whether a resource is used by the system. However, if your code (including libraries) calls Resources.getIdentifier(), this means that your code will query the resource name based on a dynamically generated string. In this case, the resource shrinker will take defensive action and mark all resources with matching name formats as potentially used and cannot be removed. For example, the following code will mark all resources with the img_ prefix as used:

 String name = String.format( "img_%1d" , angle + 1);
 res = getResources().getIdentifier( name , "drawable" , getPackageName());

At this point, I can turn on strict resource review mode and only keep resources that are certain to have been used.

Remove backup resources

The Gradle resource compressor only removes resources that are not referenced by the application, which means it will not remove alternative resources for different device configurations. If necessary, we can use the resConfigs property of the Android Gradle plugin to remove alternative resource files that your application does not need (common ones include strings.xml for internationalization support, layout.xml for adaptation, etc.):

 android {
 defaultConfig {
 ...
 //Retain Chinese and English internationalization support
 resConfigs "en" , "zh"  
 }
 }

Customizing obfuscation rules

After tasting the above "side dishes", let's taste the "main course" of this article: custom obfuscation rules. First, let's learn about common obfuscation commands.

keep Command

The keep command mentioned here refers to a series of commands starting with -keep, which are mainly used to retain elements in Java that do not need to be obfuscated. The following are common -keep commands:

-keep

Purpose: To keep the specified classes and members to prevent them from being obfuscated. For example:

 # Keep the classes and class members under the package: com.moos.media.entity
 -keep public class com.moos.media.entity.**
 # Reserved class: NumberProgressBar
 -keep public class com.moos.media.widget.NumberProgressBar {*;}

-keepclassmembers

Function: Keep the members (variables/methods) of the specified class, and they will not be obfuscated. For example:

 # Reserved class members: specific member methods in the MediaUtils class
 -keepclassmembers class com.moos.media.MediaUtils {
 public   static *** getLocalVideos(android.content.Context);
 public   static *** getLocalPictures(android.content.Context);
 }

-keepclasseswithmembers

Effect: Keep the specified class and its members (variables/methods), provided they are not deleted during the compression phase. Similar to -keep:

 # Reserved class: Subclass of BaseMediaEntity
 -keepclasseswithmembers public class * extends com.moos.media.entity.BaseMediaEntity{*;}
 # Reserved class: Implementation class of OnProgressBarListener interface
 -keep public class * implements com.moos.media.widget.OnProgressBarListener {*;}

@Keep

In addition to the above methods, you can also choose to use the @Keep annotation to keep the expected code and prevent it from being obfuscated. For example, we modify a class with @Keep to keep it from being obfuscated:

 @Keep
 data class CloudMusicBean(var createDate: String,
 var id: Long,
 var name : String,
 var url: String,
 val imgUrl: String)

Similarly, we can also let @Keep decorate methods or fields to retain them.

Other commands

dontwarn

The -dontwarn command is usually used when we introduce a new library, and is often used to deal with warnings that cannot be resolved in the library. For example:

 -keep class twitter4j.** { *; }
 -dontwarn twitter4j.**

For other command usage, please refer to the default obfuscation rules provided by the Android system:

 #Do not generate mixed case class names when obfuscating
 -dontusemixedcaseclassnames
 #Do not skip non-public library classes
 -dontskipnonpubliclibraryclasses
 #Log the process of obfuscation
 -verbose
 # Turn off pre-check
 -dontpreverify
 #Turn off optimization
 -dontoptimize
 #Keep annotations
 -keepattributes *Annotation*
 #Retain all class names and local method names that have local methods
 -keepclasseswithmembernames class * {
 native <methods>;
 }
 #Retain the get and set methods of the custom View
 -keepclassmembers public class * extends android. view . View {
 void set *(***);
 *** get*();
 }
 #Retain the method of entering the View and its subclasses in the Activity, such as: onClick(android. view . View )
 -keepclassmembers class * extends android.app.Activity {
 public void *(android. view . View );
 }
 #Keep enumeration
 -keepclassmembers enum * {
 **[]$ VALUES ;
 public *;
 }
 #Keep the serialized class
 -keepclassmembers class * implements android.os.Parcelable {
 public   static final android.os.Parcelable$Creator CREATOR;
 }
 #Keep static members of R file
 -keepclassmembers class **.R$* {
 public   static <fields>;
 }
 -dontwarn android.support.**
 -keep class android.support.annotation.Keep
 -keep @android.support.annotation.Keep class * {*;}
 -keepclasseswithmembers class * {
 @android.support.annotation.Keep <methods>;
 }
 -keepclasseswithmembers class * {
 @android.support.annotation.Keep <fields>;
 }
 -keepclasseswithmembers class * {
 @android.support.annotation.Keep <init>(...);
 }

Confusion "Blacklist"

After we understand the basic commands of obfuscation, many people may still be confused: what content should be obfuscated? In fact, when we use code obfuscation, ProGuard obfuscates most of the code in our project. In order to prevent errors during compilation, we should use the keep command to keep some elements from being obfuscated. Therefore, we only need to know which elements should not be obfuscated:

enumerate

It is inevitable that enumeration types may be used in a project, but they cannot be used in obfuscation. The reason is that there is a values ​​method inside the enumeration class, which will be renamed after obfuscation and throw a NoSuchMethodException. Fortunately, the default obfuscation rules of the Android system have added processing for enumeration classes, so we don't need to do extra work. If you want to know more about the internal details of the enumeration, you can check the source code. I won't go into details due to limited space.

Reflected Elements

Classes, variables, methods, package names, etc. used by reflection should not be obfuscated. The reason is that during code obfuscation, the elements used by reflection will be renamed, but reflection still searches for elements according to the previous names, so NoSuchMethodException and NoSuchFiledException problems often occur.

Entity Class

Entity classes are what we often call "data classes", and of course they are often accompanied by serialization and deserialization operations. Many people should have thought that obfuscation is to transform "elements" that originally have specific meanings into meaningless names. Therefore, after the "baptism" of obfuscation, the key corresponding to the serialized value has become a meaningless field, which is definitely not what we want. At the same time, the deserialization process creates objects fundamentally with the help of reflection. After obfuscation, the key will be changed, so it will also violate our expected effect.

Four major components

The four major components in Android should not be confused either. The reasons are:

1. The four major components need to be registered and declared in the AndroidManifest.xml file before use. However, after obfuscation, the class names of the four major components will be tampered with, and the classes actually used do not match the classes registered in the manifest, so an error occurs.
2. When other applications access components, they may use the package name plus the class name. If obfuscated, they may not be able to find the corresponding component or may generate an exception.

Java method called by JNI

When the Java method called by JNI is obfuscated, the method name becomes a meaningless name, which does not match the original Java method name in C++, so the called method cannot be found.

Others that should not be confused

• Custom controls do not need to be obfuscated
• JavaScript calling Java methods should not be confused
• Java native methods should not be confused
• It is not recommended to confuse third-party libraries referenced in the project

Obfuscated stack trace

After the code is obfuscated by ProGuard, it becomes difficult to read the StackTrace information. Since the method names and class names are obfuscated, it is difficult to locate the problem even if the program crashes. Fortunately, ProGuard provides us with a remedy. Before we proceed, let's take a look at what ProGuard generates after each build.

Obfuscated output

After the obfuscation build is completed, the following files will be generated in the /build/outputs/mapping/release/ directory:

• dump.txt

Describes the internal structure of all class files within the APK.

• mapping.txt

Provides a comparison table of contents before and after obfuscation, which mainly includes classes, methods and class member variables.

• seeds.txt

Lists the classes and members that are not obfuscated.

• usage.txt

Lists the code that was removed from the APK.

Restore the stack trace

After understanding the output content after the obfuscation is built, let's now look at the previous problem: after obfuscation, StackTrace is difficult to locate. How to restore the positioning ability of StackTrace? The system provides us with a retrace tool. Combined with the mapping.txt file mentioned above, the obfuscated crash stack trace information can be restored to the normal StackTrace information. There are two main ways to restore StackTrace. For ease of understanding, we take the following crash information as an example and use two methods to restore it respectively:

 java.lang.RuntimeException: Unable to start activity
 Caused by : kotlin.KotlinNullPointerException
 at com.moos.media.ui.ImageSelectActivity.k(ImageSelectActivity.kt:71)
 at com.moos.media.ui.ImageSelectActivity.onCreate(ImageSelectActivity.kt:58)
 at android.app.Activity.performCreate(Activity.java:6237)
 at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1107)

Through the retrace script tool

First, we need to enter the /tools/proguard/bin directory of the Android SDK path. Here, taking the Mac system as an example, you can see the following content:

You can see the three files above, and proguardgui.sh is the retrace script we need (proguardgui.bat in Windows). In Windows, just double-click the script proguardgui.bat to run it. As for Mac, if you have not made any configuration, just drag the proguardgui.sh script to the terminal that comes with Mac and press Enter to run it. Then, we will see the following interface:

Select the ReTrace column and add the location of the obfuscated mapping.txt file in our project. Then copy the obfuscated crash information to the Obfuscated stack trace column and click the ReTrace! button to restore the crash log information. The result is shown in the figure above. Our previous obfuscated log: at com.moos.media.ui.ImageSelectActivity.k(ImageSelectActivity.kt:71) is restored to at com.moos.media.ui.ImageSelectActivity.initView(ImageSelectActivity.kt:71). ImageSelectActivity.k is the name of the method after obfuscation, and ImageSelectActivity.initView is the name of the method before obfuscation. With the help of the ReTrace tool, we can quickly locate the crash code area as before.

Via the retrace command line

We first need to copy the crash information to a txt file (such as proguard_stacktrace.txt) and save it, then execute the following command (MAC system):

 retrace.sh -verbose mapping.txt proguard_stacktrace.txt

If you are using Windows, you can execute the following command:

 retrace.bat -verbose mapping.txt proguard_stacktrace.txt

The final restored result is the same as before:

Perhaps you find the Unknown Source problem when restoring stackTrace through the above two methods:

It is worth noting that remember to add the following configuration to the obfuscation rules to improve our StackSource search efficiency:

 # Keep the source file name and specific code line number
 -keepattributes SourceFile,LineNumberTable

Additionally, every time we create a release build with ProGuard, the mapping.txt file from the previous version is overwritten, so we have to be careful to save a copy every time we release a new version. By keeping a copy of the mapping.txt file for each release build, we can debug and fix problems with old versions of the app based on the obfuscated StackTraces submitted by users.

Posture-raising operation

As we have seen above, after the APK is obfuscated, the package name, class name, and member name are converted into meaningless and incomprehensible names, which increases the cost of decompilation. Android ProGuard provides us with a default "obfuscation dictionary", which converts element names into lowercase English letters. So, can we define our own obfuscation dictionary? Let's take a look at a picture of the effect first:

Is this wave operation a bit "outstanding"? Haha, I won't keep you in suspense. It's actually very simple. Just generate your own obfuscation dictionary in txt format and then apply it in the obfuscation rules Proguard-rules.pro:

Of course, you can also customize your own "obfuscation dictionary" to increase the difficulty of decompilation.

Along the way, we can find that from the necessity and advantages of obfuscation technology, it is still worth our in-depth study and research. This article only takes you to appreciate the "tip of the iceberg". Due to my limited technical level, if you find any problems or inappropriate explanations, please point them out and correct them.