Solution to resource confusion in Android plug-in

summary

This article introduces the resource confusion problem when plugins use host resources in the Android plugin framework, as well as the causes of the confusion, the industry's general solutions, and the optimization solutions we proposed.

This article will explain step by step in the following order:

• Briefly introduce the dynamic resource part in Android plug-in.
• Briefly introduce some basic knowledge, usage and compilation principles of resources in Android.
• This section describes the resource confusion problem that occurs in plug-in scenarios and the common solutions in the industry.
• This paper introduces a new solution - the resource-free fixation solution, which is used to solve the resource confusion problem.
• Let me introduce a technical point in the free resource fixation solution separately: modify the resource files in the apk.

1. Dynamic resources in Android plug-in

Android has been developed for so many years, and many plug-in/hot fix frameworks have emerged on the market. Both plug-in and hot fix are to achieve dynamic content outside the main apk, including dex (class), res (resources), so (dynamic library), etc. For each type of content, the industry has many implementation solutions. Although the solutions are different, the underlying principles are similar. There are also many articles and open source projects on the Internet for reference.

Glossary

Host: An App that is directly installed on the user's phone. The code in the host is fixed the moment the host is installed on the user's phone and cannot be changed (hot fixes only prevent incorrect logic from running, but do not change the original code).

Plugin: A file independent of the host. A collection of classes, res, so, etc. that need to be dynamically loaded by the host. (This part is usually called patch in hot fixes, but for convenience, it is called plugin here)

Java code: For the convenience of description, the dex in apk is called Java code before compilation and dex after compilation (this statement is not accurate, don’t be misled by me, it is generally Java / Kotlin->class->dex)

When it comes to dynamic Android resources, the ideas are similar:

• Create a Resources for each plugin or add the plugin's resource path to the host AssetManager so that the plugin resources can be loaded smoothly.
• When the plug-in is compiled, the packageId part of the resource ID in the plug-in is modified by configuring the aapt2 parameter to ensure that the plug-in does not conflict with the host resource ID.
• For the host resources used in the plug-in, use the aapt2 parameter to fix the resources to ensure that the host resource ID used by the plug-in remains unchanged after the host is upgraded.

The emergence of aapt2 makes it much easier to fix resources and modify packageId!

Although the dynamic technology of Android resources is very mature, there are still many shortcomings in practice. For example, "fixed resources" are often complained by business colleagues.

2. Introduction to resources in Android

Before introducing resource pinning, let's first briefly introduce the basic knowledge related to resources in Android.

2.1 Resource IDs in Android

After the Android code is compiled into an apk, each resource corresponds to a unique resource id, which is an 8-bit hexadecimal int value 0xPPTTEEEE:

• PP: The first two digits are the PackageId field. System resources are 01, host resource id is 7f, and others such as manufacturer-defined skin packages and webview plug-in resource packages will occupy 02, 03, etc. Therefore, App resources and system resources will never conflict. In order to ensure that plug-ins and host resources do not conflict, plug-in frameworks on the market usually change the PP of plug-in resources to other values, such as 7e and 7d.
• TT: The middle two digits are the TypeId field, which indicates the type of resource, such as anim, drawable, string, etc. There is no strict correspondence between them, and type values ​​are usually assigned in alphabetical order.
• EEEE: The last four digits are the EntryId field, which is used to distinguish resources with different names under the same PackageId and the same TypeId, and are usually allocated in alphabetical order.

Notice:

• By default, resource IDs are assigned in alphabetical order. That is, when a resource named a is added, after recompiling, the resource ID values ​​of the same type after a will be changed.
• aapt2 provides parameters to intervene in the resource ID allocation method. aapt2 will prioritize the ID allocation according to the corresponding relationship configured in the parameters. We call this technology resource fixation, which is also the most commonly used technology in plug-in frameworks to solve resource confusion problems.

2.2 Resource usage in Android

There are usually two ways to use resources in Android:

1. In Java code, access it through the internal class of R. The specific syntax is:

 [<package_name>].R.<resource_type>.<resource_name> 

2. In XML, it is used by symbols, and the specific syntax is:

 @[<package_name>:]<resource_type>/<resource_name> 

In xml, you can also reference style attributes by replacing @ with ?. You can also introduce custom attributes, such as android:layout_width. These two usages do not affect the following introduction.

So what is the difference between these two methods?

From the perspective of code writing, resources are accessed through a resource name (resource_name). Let's decompile the apk and see what it looks like after compilation.

Write the following code in the project app module, library module, and xml respectively

Let's decompile the apk and see how these three codes are represented in the apk.

It can be found that the resources in the appTest method and xml have become numbers (0x7f0e0069), and the resources in the libTest method are still accessed through Lcom/bytedance/lib/R$string;->test

in conclusion:

• The resources referenced in the main module are compiled into values;
• In submodules and aar, the values ​​are indirectly referenced through R's internal classes;
• All resource IDs in the XML are compiled into numerical values. (See the XML attributes in the above picture - layout_width and others are still strings. In fact, they are also resource ID values. These strings are actually useless and can even be directly removed in some package size optimizations).

So why is it referenced by field in libTest method but becomes a number in appTest?

2.3 Simple process of resource compilation in Android

Assume that there is a project with only one app module, which depends on several third-party AARs through the Maven repository. The simplified process of compiling the project is as follows:

1. Download the third-party aar;
2. Compile and link the resources in the app module and the third-party aar through aapt2, and finally generate R.jar and ap_

• R.jar contains all R.class that are finally put into apk, one for each dependency. aapt2 will also assign a unique id value to each resource in alphabetical order by default. Note: Adding or deleting a resource will cause the resource ids behind it to change. aapt2 allows configuration to intervene in the allocation of ids.
• The ap_ file contains all compiled resource files.

3. The java files of the App module are compiled by javac together with R.jar. Since all fields in R.jar are final, all resources referenced by R in the App module are inlined as values. Since the third-party aar is already a class, it does not need to be compiled, so resources are still used through R references;
4. Finally, convert the .class compiled by the app module and the .class in the third-party aar into dex and compress them into the apk together with ap_.

Therefore, it is easy to understand why libTest still uses resources through R, while appTest directly references them by value (inlined).

Although the libTest module is depended on by the app module through source code, it is actually similar in terms of resource compilation, which is not introduced here.

2.4 Summary

Whether the resources in Android are used through Java code or XML, they are ultimately searched through the resource ID value.

Drag the apk into the as file and check the resources.arsc file. You can see that it contains the id index of all resources in the apk, as well as the real resource or value corresponding to the resource name. It is easy to imagine that when the App is running, it also searches for the real resource content through the resource id value through this resource table.

3. Plugins use host resources

3.1 How plugins use host resources

Imagine that we want to make the live broadcast function of the App into a plug-in and distribute it dynamically. Most of the resources required for the live broadcast function are in the live broadcast plug-in, but there are always some resources from the host, such as the resources contained in some common UI components (support/androidx library), etc.

So, suppose there is a picture named icon in the host, and the xml in the live plug-in references this picture through @drawable/icon, and also references it in the code through R.drawable.icon. In fact, there is no icon picture in the live plug-in, but it exists in the host. After the host is compiled, according to the previous knowledge points, the value corresponding to icon in the host is compiled into 0x7f010001.

The plugin itself is also an apk. According to the knowledge points introduced above, after the plugin is compiled, @drawable/icon in the xml will be compiled into a value (0x7f010001), and R.drawable.icon in the java code will also be compiled into a value (0x7f010001) directly or indirectly. When the plugin runs on the host, according to the previous introduction, the plugin will search for 0x7f010001 and find that it can be found, so the host resources are used correctly.

When the plugin is compiled, we will do some processing so that the host id can be referenced in the plugin.

3.2 What are the problems with plugins using host resources?

As mentioned earlier, adding or deleting a resource may cause the IDs of many other resources to be changed.

After our host is compiled, the icon is 0x7f010001. After a plug-in is compiled based on the existing host, the icon referenced in the plug-in is also 0x7f010001, so there is no problem at this time.

After the host iteration, a new resource aicon is added. According to the resource ID allocation rules introduced above, the ID value of aicon in the new version of the host is 0x7f010001, and the ID value of icon is assigned to 0x7f010002. When the old version of the plug-in is sent to the new version of the host, it will still use 0x7f010001 to find the icon in the host, so it will naturally find the wrong one. If you are lucky, it may only cause an abnormal display of the image, but if you are unlucky, it may crash directly.

3.3 How to solve this problem

To solve this problem, the industry currently has a universal and stable solution - resource fixation. When the host is compiled, the resources used by the plug-in are fixed through the parameters provided by aapt2, so that the values ​​of these resources will never change each time the host is packaged.

Disadvantages of resource fixation solution:

1. One plugin corresponds to one host:

• All resources of the host must be fixed. If only the resources used by the plug-in are fixed, when a host has two plug-ins, the two plug-ins will each fix the resources they need to the host. When the code is merged, it is easy to cause conflicts because the fixed values ​​of the resources are not allowed to be repeated;
• When the host is connected to multiple frameworks involving resource fixation, such as plug-in, resource hotfix, game repackaging framework, etc., the resource fixation between these frameworks also needs to be considered unified fixation, which is very costly;
• Resource pinning increases the cost of hosting access to the framework.

2. When a plugin runs on multiple hosts:

• When a plug-in wants to run on multiple hosts, each host needs to perform resource pinning according to the resource usage of the plug-in. Once a host has already pinned a resource, causing a conflict with the resource pinning required by the plug-in, the plug-in needs to compromise with the host and regenerate the pinning rules based on the host's existing resource pinning. This makes it impossible for a plug-in to run on multiple hosts. We currently have a requirement: the same plug-in needs to run on thousands of hosts. If this problem cannot be solved, hundreds or thousands of plug-ins may need to be developed, which is obviously unreasonable.

• Resource pinning increases the cost of hosting access to the framework.

In order to solve the above problems, we have developed a new solution to solve the resource confusion problem.

4. Free resource fixation solution

When the same version of a plug-in runs on different versions or even different apps, the plug-in code is fixed, but the resource ID in the host will change. In order to solve the resource confusion problem, the current idea is to ensure that the resource ID remains unchanged every time the host releases a new version. So is there a way to keep the plug-in consistent with the host's resource ID without constraining the host?

When the plugin is packaged, the host is unknown, and when a plugin runs on multiple hosts, the hosts are also diverse. So there is no way to specify that the plugin should type the id to satisfy the host. As mentioned above, all places in the plugin that reference the host id are constants. So what can we do?

Is it possible to dynamically modify the content of the plug-in when it runs on the host to achieve the effect of matching the plug-in and host ID values?

For example, the plugin uses the host's resource icon, and the corresponding id value is 0x7f010001. When the plugin runs on a host with an icon of 0x7f010002, since the runtime resource search is performed by the id value, we can only know that the plugin is looking for a resource with an id of 0x7f010001. If we can map 0x7f010001 to the string icon by some means, and then use the Resources#getIdentifier method provided by the Android system to dynamically obtain the resource id corresponding to the icon in the current host, we can ensure that the plugin loads the correct resource.

This work requires some work to be done when the plug-in is compiled and run.

4.1 Plugin compile-time work

This section is based on the introduction of agp4.1. There are some differences between different versions, but the overall ideas are similar.

As mentioned above, there are two main situations in which a plug-in uses host resources: 1. Through Java code 2. Through XML.

4.1.1 Handling host resources referenced in Java code

After the Java code is compiled into a class, some of the code that references the host resource ID will be compiled into a numerical value, while others will still be referenced through R. We can easily find the latter, but it is a bit difficult to find the former, because simply scanning the number starting with 0x7f in the class can easily lead to misjudgment and treat a meaningless number as a resource ID.

We have discussed why the resource ID in the class is inlined as a value. So why not just not inline it? We only need to process R.jar during compilation and remove all final fields in the class to ensure that all resource IDs in the plugin that reference the host are referenced through R.

This part requires a certain understanding of the workflow of agp and the development of gradle plugin. It uses the asm bytecode modification technology and the transform api provided by agp. Students who don’t understand it can check it separately. I won’t introduce it in detail here.

Simply put, through these two technologies, class files can be modified when compiling apk.

Start practicing

1. Since R.jar is generated in processResourcesTask, you can write a gradle plugin, get R.jar in doLast of processResourcesTask, modify the bytecode in R.jar, and remove all final modifiers of fields with id starting with 0x7f. This ensures that all references to host resources in the plugin class will not be inlined into values;
2. After the first step, all host resources referenced in the plugin are referenced through R.xx.xx, but the value in the plugin R still cannot correspond to the host. Therefore, we continue to write a transform to scan the places in the plugin that reference resources through R, and use asm to change it from the original R reference to a method call. When the plugin is running, the original code similar to R.drawable.test no longer obtains a constant value, but calls a method to dynamically calculate the corresponding value in the current host. ​

Summarize:

The above mentioned problem of free resource fixation when referencing host resources in the plug-in Java code can be solved through some processing during compilation.

• Advantages: No resource pinning is required.
• shortcoming:

1. Some resources in the plugin are not inlined, which will increase the package size very slightly, but it is not a big problem;
2. The plugin references host resources from the original constant to method call, which reduces the execution efficiency, but this can be solved by caching. At the same time, plug-in itself is a black technology, and sometimes it is worth sacrificing some performance to solve a problem.

4.1.2 Processing resources referenced from the host in XML code

The problem of referencing host resources in XML cannot be solved only at compile time, because XML cannot execute logic like Java code. As mentioned earlier, after XML is compiled, all resources are compiled into numerical values, and we cannot know which host it will run on in the future and what the value will be at compile time. Therefore, the work of modifying resource IDs in XML can only be moved to runtime. Of course, we also need to do something at compile time to assist the modification operation at runtime.

At runtime, we need to modify the resources starting with 0x7f in the apk's xml, and change their values ​​to the correct values ​​corresponding to the current host. However, through xml, we can only get one value, so we can collect the xml files where the host resources used in the plug-in xml are located and their corresponding resource names when the plug-in is compiled. At runtime, we can use the mapRes method mentioned above to get the values ​​that need to be modified.

Start practicing

As mentioned above, aapt2 generates an ap_ file after compilation/linking. This file contains all the compiled resources (including various XML, resources.arsc, AndroidManifest.xml) that will eventually enter the plug-in. We only need to analyze the resources starting with 0x7f referenced in these files, find the corresponding resource name according to R.txt (a file generated by aapt2), record the resource name, id value, and file into a file, and package them into the plug-in apk.

As for how to scan the resources at 0x7f in these files, we used different methods at different stages. You can choose by yourself:

1. Use the aapt2 command to dump file information and analyze the text content after the dump (this is how we compile, which is simple and crude, has poor performance, and is not elegant enough);
2. According to the file format analysis, the file content is parsed and the resource starting with 0x7f is found (more elegant and efficient, this is how we run it).

Summarize:

The above generates a file that stores the information of the host resources used in the plugin xml. It looks like this:

In the previous article, we have been talking about the host resources used in XML. Looking at the configuration file above, why is there resoureces.arsc in fileNames? It is obviously not an XML file.

In fact, after Android resources are compiled, some resource files related to values ​​no longer exist and will be directly placed in resources.arsc. Files such as layout still exist. The layout in resoureces.arsc points to various layout.xml files, while value type resources such as string point to real content. Interested students can open the apk through Android Studio and observe the structure in resources.arsc.

4.2 Plugin Installation Work

As mentioned above, when the plug-in is compiled, some logic is inserted into the Java code to achieve the effect that the plug-in dynamically obtains the resource ID according to the host environment. However, after the XML is compiled, the resource ID is directly compiled into a number, and logic cannot be inserted into the XML. Therefore, we can only modify it according to the host environment before running the plug-in.

There is a plugin installation process before the plugin runs in the host, which is similar to the installation of apk in the Android system. Therefore, you only need to modify the xml and resources.arsc files in the plugin according to the configuration file generated during compilation and the mapRes method before each plugin installation or after the host upgrade.

After determining the timing and content of the modification, the next step is to introduce in detail how to modify these files.

5. Modify the resource files in apk

5.1 How to modify xml and arsc files

After being compiled into apk, layout, drawable, AndroidManifest and other files in Android are no longer regular XML files, but a new file format Android Binary XML, which we call axml here. So how to modify the axml file?

All files have their own file formats. When reading files, the program reads byte arrays and then parses the meaning of each element in the byte array according to the file format. Therefore, we only need to understand the axml file format, parse the file according to the specification, find the location of the resource id in the byte array, map the original resource id to a new value according to the resMap method, and then modify the corresponding part in the byte array. (Fortunately, we only modify an 8-bit hexadecimal number in the axml file. This modification will not change the length, offset and other information of the content in the file, so we can directly replace the corresponding part of the byte array.)

resources.arsc is the resource index table of apk, which records all the resources in apk. For resources of value type, the corresponding content of the resource will all enter resources.arsc, so we also need to modify this file (for example, if the parent of a style is the host resource, we need to modify it). The modification method is similar to XML. You only need to parse the byte array according to the specification, find the offset of the content to be modified, and replace it.

There are many articles on the Internet introducing the file formats of axml and arsc, so I will not describe them in detail here.

Apktool is a powerful, open source reverse tool that can decompile apk into source code. It must also have the code to read axml and arsc in apk, otherwise how can it output an editable xml source code file? So we can directly go to the code to read axml and arsc in apktool. When reading the id belonging to the host in axml, record the offset of the byte array and directly replace the byte subarray at the corresponding position.

aapt2 provides us with the ability to dump resource content, which can help us directly use the "naked eye" to see the content of axml and arsc. With this tool, we can easily confirm the modified content and verify whether the modification is effective. Taking aapt2 in the 30.0 version of build-tools as an example, its command is aapt2 dump apk path --file resource path. If it is not followed by --file resource path, arsc will be dumped directly.

The following is the dump of arsc. You can see that the parent of the last style is a host resource starting with 0x7f.

The following is the dump of activity_plugin1.xml. You can see that the TextView references a resource in the host as the background.

5.2 Modify the xml/arsc file in apk

Above we know how to modify an axml and arsc file. When installing the plugin, we get an apk file, so how to modify the axml and arsc files in the apk?

5.2.1 Modification of recompression method

Apk is actually a zip file. To modify the file content in apk, the simplest way that comes to mind is to read the file in the zipFile, modify it and then re-compress it.

Java provides us with a set of APIs for operating zipFile. We can easily read the contents of the zip file into memory, modify it in memory, and then rewrite it into a new zipFile using ZipOutputStream.

The code is very simple to implement. After the modification is successful, the test finds that it is feasible, so our first step is successful, which means that the way to dynamically modify the plug-in at runtime is feasible.

I was secretly happy to find that the modification process was very time-consuming. Taking the company's live broadcast plug-in as an example (the live broadcast plug-in is about 30 MB, which is a relatively large plug-in), it takes about 8 seconds on devices with version 9.0 and above, about 20 to 40 seconds on devices with version 7 to 8, and about 10 to 20 seconds on devices below 7.x. Although the plug-in installation is done in the background, it is acceptable to increase the time appropriately, but a few tens of seconds of time is obviously unacceptable. Then we can only think of other ways.

Regarding the time differences between the various versions:

Starting from Android 7.0, the official uses ZLIB to provide the implementation of Deflater and Inflater to optimize the decompression and compression algorithm speed (see the comments of Deflater.java and Inflater.java). However, the ZipFileInputStream of 7.x/8.x has a BUFSIZE limit of 8192 when reading data (this limit was removed after 8.x), which increases the number of loops when reading data and reduces efficiency.

Starting from 7.0, ZipFileInpugStream uses the native method ZipFile_read to read data. The following is part of the code of ZipFile_read in android8.0 and android9.0.

5.2.2 Directly modify the byte array of apk

Apk is actually a zip file. For an introduction to zip files, please refer to the official documentation of Zip.

To summarize briefly, a zip file consists of a data area, a central directory record area, and a central directory trailer area (higher versions of zip files add new content).

• Central directory tail area: Through the tail area, we can know the number of files in the zip package, the location of the central directory record area and other information;
• Central Directory Record Area: Through the tail area, we can quickly find each file record in the central directory record area. These records mainly describe the basic properties of the files in the zip package, such as file name, file size, whether it is compressed, compressed size, file offset in the data area, etc.
• Data area: The data area is used to store the real content of the file. According to the content recorded in the central directory record area, the corresponding file metadata and the real data of the file (if compressed, it is the compressed data) can be quickly found in the data area.

Start working

After understanding the format of the zip file, we only need to find the offset of the file data we need to modify in the apk according to the file format protocol, and then directly modify the corresponding byte array in combination with the previous method of modifying the axml/arsc file. With the help of the RandomAccessFile tool provided by java, we can quickly read/write any position of the file.

During the modification process, we found that most of the XML files in the apk are compressed (those in the res/xml directory are generally not compressed). This results in the byte array we get from the apk being compressed axml data. If we want to modify this data, we need to first decompress it using the Deflate algorithm (the Deflate algorithm is generally used in zip files), then modify and recompress it. However, after our modification, the recompressed data may not match the length of the data before the modification. If it is shortened, it is okay to modify the file metadata. If the file length becomes longer, it may cause the offset of subsequent files to change, which will affect the entire system.

Fortunately, we can hack into the plugin packaging process. When we introduced "plugin compile-time work" earlier, we got the files that need to be modified during compilation, so we only need to control the apk packaging not to compress these files (in fact, Android Target30 also requires arsc files not to be compressed). This solves the problem very easily, but of course it will increase the size of the plugin package.

Finally, in the live streaming plug-in, turning on this function will increase the package size by 20kb. For a live streaming plug-in with a size of nearly 30mb, this increase is acceptable and will not affect the host package size. (This increase depends on how much XML the plug-in uses host resources. Generally, the increase of the plug-in should be smaller than that of the live streaming plug-in.)

After the transformation was completed, it was tested that the modification time of the live plug-in on various versions of mobile phones was about 300 to 700ms, and the modification speed increased by 10 to 90 times. Most plug-ins are also smaller than the live plug-in, and the time consumption can be guaranteed to be within 100ms. At the same time, this modification process is only done when the plug-in is installed for the first time or the host is upgraded, and it is completed in the background, so it is completely acceptable.