Some summary thoughts on Android plug-in

In recent years, the mobile development industry has been swept by the "plug-in technology" whirlwind. Major companies have launched their own plug-in frameworks, and various open source frameworks have evaluated their own functional superiority, which is dizzying. With the rapid development of the company's business, the number of projects has increased, but the development resources are limited. How to meet the demand and project growth within limited resources, while being able to quickly respond to problems and iterate new requirements, this is a contradiction. At this time, plug-in technology is booming, and it is necessary to understand the implementation ideas of various mainstream frameworks to see if it can help current work.

It is mainly divided into the following parts

• Plug-in Introduction
• Getting Started
• Implementation principle
• Mainstream framework
• Actual Combat
• summary

Plug-in Introduction

Baidu Encyclopedia defines a plug-in as follows: "It is a program written in accordance with a certain standard application interface. It can only run on the system platform specified by the program and cannot run independently from the specified platform." In other words, a plug-in can provide a dynamic expansion capability, allowing the application to load functions that do not originally belong to the application at runtime, and to achieve dynamic updates and replacements.

So what is "plug-in" in Android? As the name suggests, it means encapsulating some core complex and highly dependent business modules into independent plug-ins, and then combining them differently according to different business needs, dynamically replacing them, and managing and updating the plug-ins. There are two concepts in plug-in that need to be explained:

Host

The so-called host needs to be able to provide a running environment and a context environment for resource calls. It is generally our main APK, the application to be run. As the main project of the application, it implements a framework for loading and managing plug-ins. Plug-ins all rely on the host APK.

Plugins

A plug-in can be imagined as each independent functional module encapsulated as a small APK. The plug-in APK can be put online and offline in the host APK, as well as dynamically updated, through online configuration and update.

So why should we use plug-in technology, what are its advantages, and what benefits can it bring to us? Here are a few simple ones:

• Allow users to upgrade application functions without reinstalling APK, reduce the frequency of version releases, and improve user experience.
• Provides a capability to quickly fix online bugs and updates.
• Load different modules on demand to achieve flexible function configuration and reduce the server's compatibility pressure with old version interfaces.
• Modularity, decoupling, parallel development, 65535 problems.

Getting Started

First of all, we need to know that plug-in technology is a relatively complex field. The complexity lies in the fact that it involves a wide range of knowledge points. It is not only the upper-level application architecture ability, but also requires us to have a certain understanding of the underlying knowledge of the Android system. Here is a brief list of the knowledge points involved:

First of all, we need to introduce Binder. We all know that the core of Android multi-process communication is Binder. Without it, it is really difficult to move forward. Binder involves two layers of technology. You can think of it as a mediator mode. Binder plays the role of an intermediary between the client and the server. If you want to implement the plug-in of the four major components, you need to make changes on Binder. The content of the Binder server cannot be modified. You can only change the client code. Moreover, the client of each of the four major components is different, which requires in-depth research. The best way to learn Binder is AIDL. There are a lot of information on this aspect on the Internet. The easiest way is to write an aidl file to automatically generate a Java class, and then check each method and variable of this Java class, and then look at the four major components. In fact, they are all implemented in a similar way to AIDL.

Secondly, it is the process of App packaging. After the code is written, a packaging operation is performed, which goes through resource packaging, Dex generation, signing and other processes. The most important one is resource packaging, that is, the AAPT step. If the resource ID of the host and the plug-in conflict, one solution is to modify it here.

Third, the installation process of the App on the phone is also very important. Being familiar with the installation process is not only helpful for plug-inization, but also very important when encountering installation bugs. When installing an App on a mobile phone, there will often be download anomalies, prompting that the resource package cannot be parsed. At this time, you need to know where the code for installing the App is. This is only the first step. The second step is to know what specific things to do after the App is downloaded locally. Some directories on the mobile phone cannot be accessed. After the App is downloaded locally, which directory should it be placed in, and what files will be generated. Plug-inization has the concept of incremental updates, how to download an incremental package, from which specific location to take a package locally, what is the specific naming rule of this package, etc. These details must be clearly understood.

Fourth, it is the startup process of the App. How many ways are there to start an Activity? One is to write a startActivity, and the second is to click on the mobile App and start the default Activity in the App through the Launcher mechanism in the mobile phone system. Usually, the second method is the most popular among App developers. So what is the startup principle of the first method? In addition, when starting, where is the Main function? The location of this Main function is very important. We can modify the class where it is located to realize plug-in.

The fifth point is more important . To make Android plug-in, you need to control two places. The first is the loading of the plug-in Dex. How to load the classes in the plug-in Dex into memory? The other is the problem of resource loading. The plug-in may be in Apk or so format. No matter which one, R.id will not be generated, so it cannot be used. There are several solutions to this problem. One is to rewrite methods such as getAsset and getResource of Context, replace the concept, and let the plug-in read the resources in the plug-in, but the disadvantage is that the resource ids of the host and the plug-in will conflict, and AAPT needs to be rewritten. Another way is to rewrite the plug-in list saved in AMS, so that the host and the plug-in can load their own resources separately without conflict. The third method is to execute a script after packaging to modify the resource id in the generated package.

The sixth point is how to solve the workspace problem of developers of different plug-ins after implementing plug-ins. For example, what codes do plug-in 1 and plug-in 2 need to download respectively, and how to run them independently? Just like air tickets and train tickets, how to run only your own plug-in and not run other people's plug-ins? This is a problem of collaborative work. Train tickets and air tickets, the workspaces of these two Android teams are different. At this time, Gradle scripts are needed. Each project has its own warehouse and different packaging scripts. You only need to package your own plug-in with the host project and run it without introducing other plug-ins. What's more powerful is that you can also package and run your own plug-in as an App.

The above introduces the entry-level knowledge of plug-in, a total of six points, each of which requires a lot of time to understand. Otherwise, when facing a plug-in project, you will be confused in many places. As long as you understand these six core points, everything can be solved easily.

Implementation principle

Applying plug-in technology in Android is actually the process of dynamic loading, which is divided into the following steps:

• Copy the executable file (.so/dex/jar/apk, etc.) to the application APP.
• Load the executable file and replace the static resources
• Call specific methods to execute business logic

In Android projects, dynamic loading technology can be roughly divided into two types according to the different executable files loaded:

• Dynamically loading .so libraries
• Dynamically load dex/jar/apk files (this is what dynamic loading generally refers to now)

First, the NDK in Android actually uses dynamic loading, which dynamically loads the .so library and calls its encapsulated methods through JNI. The latter is generally compiled from C/C++ and runs at the Native layer, which is much more efficient than Java code executed at the virtual machine layer. Therefore, Android often uses dynamic loading of .so libraries to complete some tasks that require high performance (such as Bitmap decoding, image Gaussian blur processing, etc.). In addition, since the .so library is compiled from C/C++, it can only be decompiled into assembly code, which is more difficult to crack than the Smali code decompiled from the dex file, so the .so library can also be used in the security field.

Secondly, "dynamic loading of dex/jar/apk files based on ClassLoader" means dynamically loading the dex package compiled from Java code in Android and executing the code logic therein. This is a technology that is rarely used in conventional Android development. This is what we are referring to as dynamic loading.

In Android projects, all Java codes will be compiled into dex files. When Android applications are running, they work by executing the business code logic in the dex files. Using dynamic loading technology, external dex files can be loaded when Android applications are running. Downloading new dex files from the network and replacing the original dex files can achieve the purpose of upgrading applications (changing code logic) without installing new APK files.

Therefore, the ClassLoader mechanism in Android is mainly used to load dex files. The system provides two APIs to choose from:

• PathClassLoader: It can only load APK files that have been installed in the Android system. Therefore, it does not meet the requirements of plug-in and is not considered.
• DexClassLoader: supports loading external APK, Jar or dex files, which meets the requirements of documentation. All plug-in solutions use DexClassloader to load .class files in the plug-in APK.

Mainstream framework

To implement a plug-in framework in Android, the following issues need to be addressed:

• Loading resources and code
• Android life cycle management and component registration
• Conflict resolution between host APK and plugin APK resource references

The following is an analysis of several current mainstream open source frameworks to see the specific implementation ideas and advantages and disadvantages of each framework.

DL dynamic loading framework (end of 2014)

It implements the plug-in framework based on the proxy method, processes the surface of the App, and registers the proxy component in the Manifest. When the plug-in component is started, a proxy component is started first, and then the plug-in component is built and started through this proxy component. The plug-in APK needs to be developed according to certain rules, and the components in the plug-in need to implement the modified subclasses of Activity, FragmentActivity, Service, etc.

The advantages are as follows:

• Plugins need to follow certain rules, so the security aspect can be controlled.
• The solution is simple and suitable for plug-in transformation with a small amount of code.

The disadvantages are as follows:

• Calling component methods through This is not supported, you need to call through that.
• Since the Activity in the APK is not registered, implicitly calling the Activity inside the APK is not supported.
• During the process of plug-in writing and modification, many compatibility issues need to be considered, and joint debugging will be time-consuming and labor-intensive.

DroidPlugin (August 2015)

DroidPlugin is a plug-in framework implemented by 360 Mobile Assistant. It can directly run third-party independent APK files without modifying or installing the APK. It is a new plug-in mechanism, a free-installation operation mechanism, a sandbox (but not a complete sandbox. For users, they don't know what they will do with the APK), and the basis of modularization.

Implementation principle:

• Shared process: Provides a mechanism for Android to run multiple APKs in one process, deceiving the system through the API spoofing mechanism.
• Occupy a pit: By pre-occupying a pit, there is no need to register in the manifest, and service management is achieved through a one-to-many approach.
• Hook mechanism: Dynamic proxy implements function hook, Binder proxy bypasses some system service restrictions, IO redirection (first get the original Object -> Read, then the dynamic proxy hooks the Object -> Write back, to achieve the purpose of concealing the truth).

The program architecture of the plugin host:

The advantages are as follows:

• It supports the four major components of Android, and the components in the plug-in do not need to be registered in the host APK.
• Supports Android 2.3 and above, and all system APIs.
• The codes and resources between plugins and between plugins and the host are completely isolated.
• Process management is implemented, and the empty processes of the plug-in will be recycled in time, occupying low memory.

The disadvantages are as follows:

• The plugin APK does not support custom resource Notification and notification bar restrictions.
• The four major components with special IntentFilter cannot be registered in the plugin APK.
• There is a lack of Hook operations for the Native layer, and some plug-in APKs with Native code are not friendly and may not run properly.
• Since the codes between plugins and plugins, and plugins and hosts are completely isolated, the communication between plugins and plugins, and plugins and hosts can only be done through the communication method at the Android system level.
• Security concerns (can be modified to hook some important information).
• Model adaptation (not all machines can work, because a lot of reflection is used. If the ROM manufacturer deeply customizes the framework layer, the reflection method or class is not there, and the plug-in application may fail easily)

Small (end of 2015)

Small is a lightweight cross-platform plug-in framework based on the concept of "lightweight, transparent, minimal, and cross-platform". Its implementation principles are as follows.

• Dynamically loading classes: We know that many plug-ins have a DexPathList list from the DexClassLoader class, which supports dex/jar/zip/apk file formats, but does not support the .so file format. Therefore, the Small framework packages the .so file into a zip file format, inserts it into the DexPathList collection, and rewrites the dynamically loaded code.
• Resource segmentation: Since the format of Android resources is 0xPPTTNNNN, PP is the package ID, 00-02 belongs to the system, 7f belongs to the application, 03-7e is reserved, and you can do something with this range, TT is the Type, such as attr, layout, string, etc., and NNNN is the global resource ID. Then this framework repackages the resource package and reassigns the resource ID to each plug-in, thus ensuring that the resources of the host and the plug-in do not conflict.
• Dynamic proxy registration: In Android, four major components need to be registered in the manifest before they can be used. So how can they be used without registration? Here, the dynamic proxy mechanism is used for Hook. Before sending AMS, the system is deceived by the component that occupies the pit. After authentication, the component that is actually to be called is restored to achieve the purpose of concealing the truth.

Architecture diagram:

The advantages are as follows:

• All plugin support is built into the hosting package.
• The coding and resource file usage of plug-ins are no different from those of ordinary application development.
• Communication can be facilitated by setting URI, host, Native application plug-in, Web plug-in, online web page, etc.
• Supports Android, iOS, and Html5, and the three can communicate through the same set of Java interfaces.

The disadvantages are as follows:

• Dynamic registration of Services is not currently supported, but this can be circumvented by pre-registering the Service in the host's AndroidManifest.xml file, because the update frequency of Services is usually very low.

Differences from other mainstream frameworks:

 DyLA : Dynamic - load -apk @singwhatiwanna
 DiLA: Direct- Load -apk @FinalLody
 APF : Android-Plugin-Framework @limpoxe
 ACDD : ACDD @bunnyblue
 DyAPK : DynamicAPK @TediWang
 DPG: DroidPlugin @cmzy, 360

Function

transparency

VirtualAPK (June 2017)

VirtualAPK is an open source plug-in framework developed by Didi that supports almost all Android features and four major components.

Architecture diagram:

Implementation ideas:

VirtualAPK has no additional restrictions on plugins, and native apks can be used as plugins. After the plugin project is compiled and generated into apk, it can be loaded through the host App. After each plugin apk is loaded, a separate LoadedPlugin object will be created in the host. As shown in the figure below, through these LoadedPlugin objects, VirtualAPK can manage plugins and give them new meanings, so that they can run like apps installed on the phone.

• Merge the host and plugin ClassLoader. Note that the classes in the plugin cannot be repeated in the host.
• Merge plugin and host resources Reset the packageId of plugin resources and merge plugin resources with host resources
• Remove the plugin package's reference to the host. Use the Gradle plugin to remove the plugin's reference to the host's code and resources during the build.

Features are as follows:

None of the four major components need to be pre-registered in the host manifest, and each component has a complete life cycle.

• Activity: supports display and implicit calling, supports Activity theme and LaunchMode, and supports transparent themes;
• Service: supports explicit and implicit calls, supports start, stop, bind and unbind of Service, and supports Service in cross-process bind plug-in;
• Receiver: supports static and dynamic registration of receivers;
• ContentProvider: supports all provider operations, including CRUD and call methods, and supports cross-process access to the Provider in the plug-in.
• Custom View: support custom View, support custom attributes and style, support animation;
• PendingIntent: supports PendingIntent and its related Alarm, Notification and AppWidget;
• Support plugin Application and meta-data in plugin manifest;
• Support so in plugins.

Excellent compatibility

• It is compatible with almost all Android phones on the market, which has been verified in the Didi Chuxing client.
• In terms of resources, it is adapted to Xiaomi, Vivo, Nubia, etc., and an adaptive adaptation solution is adopted for unknown models.
• There are very few Binder Hooks. Currently, only two Binders are hooked: AMS and IContentProvider. The hook process has been fully adapted for compatibility.
• The plug-in operation logic is isolated from the host, ensuring that any problems with the framework will not affect the normal operation of the host.

Very low invasiveness

• Plugin development is equivalent to native development, and the four major components do not need to inherit specific base classes;
• A streamlined plug-in package, where the plug-in can rely on the code and resources in the host or not;
• The plugin building process is simple and is completed through the Gradle plugin. The whole process is transparent to developers.

The following is a comparison between VirtualAPK and mainstream plug-in frameworks.

RePlugin (July 2017)

RePlugin is a complete, stable, and fully usable plug-in solution developed by the RePlugin Team of 360 Mobile Security. It is also the first solution in the industry to propose "full plug-in" (comprehensive features, full compatibility, and full use).

Framework diagram:

The main advantages are:

• Extremely flexible: The main program does not need to be upgraded (no need to pre-embed components in the Manifest) to support the four new components and even new plug-ins
• Very stable: There is only one Hook point (ClassLoader), without any Binder Hook! This can make its crash rate only "one in ten thousand", and it is perfectly compatible with almost all Android ROMs on the market.
• Rich features: Supports almost all features in "single product" development, including static receivers, Task-Affinity slots, custom themes, process slots, AppCompat, DataBinding, etc.
• Easy to integrate: Whether it is a plug-in or a main program, it can be connected with just "a few lines".
• Mature management: It has a mature and stable "plugin management solution" that supports plug-in installation, upgrade, uninstallation, version management, and even process communication, protocol version, security verification, etc.
• Hundreds of millions of support: 360 Mobile Security has hundreds of millions of users to support it, and more than three years of rigorous verification ensures that the solution used by the App is the most stable and most suitable.

Actual Combat

The main purpose is to test how easy it is to get started with each framework and make different comparisons. Two Demo examples are written here, one is based on the Small framework and the other is based on the VirtualAPK framework, from which you can see the difference.

Small Practice

You need to reference the latest official version, otherwise a BUG will appear when the host and the plug-in merge build.gradle. This is a pitfall, so be careful. Secondly, you must follow certain rules in module naming, such as using app.* for business modules and lib.* for public library modules, which is equivalent to the package name .app., .lib.. Every time you add an activity component to the plug-in, you need to configure the route in the host and then recompile the plug-in. Otherwise, if you run it directly, the newly added activity component will be found in the host, and it will be reported that the component is not in the system manifest, so it is recommended to recompile the plug-in every time it is added or modified. The official said that the service support is not very friendly, so I didn't put it into practice.

VirtualAPK in Practice

One pitfall you need to pay attention to is the build environment. The official instructions require the following version environment, Gradle 2.14.1 and com.android.tools.build 2.1.3. The previous compilation was done with the latest Gradle version, which caused problems all the time. As for whether there are other problems, you can refer to the official documentation.

Specific code

• Small Demo: https://github.com/cr330326/MySmall
• VirtualAPK Demo: https://github.com/cr330326/MyVirtualAPKDemo

summary

As mentioned at the beginning, to implement a plug-in framework, it is nothing more than solving the three typical problems: how to load the plug-in code, how to manage the component life cycle in the plug-in, and how to deal with the conflict between the plug-in resources and the host resources. Each framework has different solutions to these three problems. At the same time, according to the chronological order, the later frameworks tend to absorb the essence of the existing frameworks and then fix the shortcomings of those more milestone frameworks. However, the core idea of ​​these frameworks is to use the proxy mode. Some of them are proxies at the surface layer, and some are proxies at the system application layer. Through the proxy, replacement and concealment are achieved, and finally the Android system is mistakenly believed that calling the plug-in function is the same as calling the native development function, thereby achieving the purpose of plug-in and native compatible programming.