vivo official website App modular development solution-ModularDevTool

Note: This tool is developed based on the open source compilation management tool developed by the vivo Internet client team.

1. Background

Now that client businesses are increasing, most client projects adopt a modular development model, that is, they are divided into multiple modules according to the business to improve team efficiency. For example, the overall architecture of our vivo official website is as shown below, which is divided into 13 modules, each of which is an independent code warehouse.

(Note: For the reason of this division, you can refer to a previous article " Android Modular Development Practice ")

2. Pain Points

Completely isolated code repositories make each module more independent and easier to manage, but it also brings some problems .

1. During the development phase, sub-warehouse development and integrated development and debugging are cumbersome, error-prone, and difficult to track and trace back.

1.1. When there are many modules involved in development, you need to manually pull the code one by one. The code operation of multiple sub-warehouses is very troublesome, and you need to open multiple AndroidStudios for development;

1.2. There are two ways to integrate the sub-warehouse into the main warehouse for development and debugging, but both have significant disadvantages:

(1) Method 1: The sub-warehouse relies on Maven. This method requires continuous release of sub-warehouse snapshots, and the main warehouse then updates the snapshots, which is inefficient.

(2) Method 2: The sub-warehouse relies on code dependencies. That is, you need to manually include the sub-warehouse code pulled to the local in the main warehouse's settings.gradle, and then configure dependencies in build.gradle. The configuration is cumbersome and prone to errors.

1.3. If the main warehouse relies on the sub-warehouse, code conflicts are likely to occur if it is partly Maven dependency and partly code dependency;

1.4. There is no corresponding relationship between the submodule aar and code integrated in apk, which makes it difficult to trace back when troubleshooting.

2. During the version release phase, the process is cumbersome and involves too much repetitive work. The process is as follows:

2.1. Modify the versions of the sub-warehouses one by one, specifying snapshot or release;

2.2. Each sub-warehouse needs to submit the code with the modified version number to git;

2.3. Each sub-warehouse must manually trigger the release of the Maven warehouse;

2.4. Update the version that the main warehouse depends on the sub-warehouse;

2.5、Build Apk;

2.6. If you use the continuous integration system CI, each sub-warehouse needs to be configured with a project, and then the compilation of the sub-warehouses must be started one by one. After all the sub-warehouses are compiled, the compilation of the main warehouse can be started.

3. Solution

In view of the above problems, our optimization ideas are also very clear, that is, to solve the tedious and repetitive operations in an automated way. Finally, ModularDevTool was developed to achieve the following functions:

1. Development stage

1.1. In the main repository, manage all sub-repository codes (pull code, cut branches and other git operations), and manage sub-repository related information (code repository path, branch, version, etc.);

1.2. You only need to open one AS project to develop code for all warehouses;

1.3. One-click switching between the two dependency modes of sub-warehouses (code dependency and Maven dependency), supporting mixed dependency (i.e., some warehouses are code dependent, and some warehouses are Maven dependent);

1.4. Output the submodule version and corresponding commitid during compilation to facilitate code tracing.

2. Version release stage

2.1. You only need to modify the sub-warehouse version number in the main warehouse, and the sub-warehouse does not need to be modified, which saves the process of sub-warehouse code modification and code submission;

2.2. On CI, you only need to configure a main warehouse project to achieve one-click compilation, including sub-warehouse compilation of aar (compilation in the order of dependencies), uploading maven, and compiling apk;

2.3. Three compilation modes are supported on CI:

• OnlyApp: only compile the main repository code to generate apk (provided that the submodule has been published in Maven);
• publishSnapshot: The sub-warehouse compiles and uploads the snapshot version, and then compiles the main warehouse to generate an apk;
• publishRelease: The sub-warehouse compiles and uploads the release version, and then compiles the main warehouse to generate an apk.

4. ModularDevTool Overview

The tool is implemented using shell script + gradle plug-in.

First, take a look at the project directory overview

1. The submodules directory is used to store sub-warehouse codes. The sub-warehouse code is the normal project structure. The submodules directory is as follows:

2. The repositories.xml file is used to configure sub-warehouse information, including module name, code warehouse, branch, version, etc. The specific contents are as follows:

 <?xml versinotallow="1.0" encoding="utf-8" ?>
 < repositories >
 <!-- A repository represents a warehouse, and a warehouse may have multiple modules -->
 < repository >
 <!-- Warehouse name, can be defined at will, mainly used for local quick identification-->
 < name > lib-module </ name >
 <!-- groupid when uploading to Maven -->
 < group > com.vivo.space.lib </ group >
 <!-- Configure all submodules in the repository. If there are multiple modules, add multiple module tags-->
 < modules >
 < module >
 <!-- artifactid when uploading to Maven -->
 <artifactid> vivospace_lib </artifactid>​​​
 <!-- Version number when uploading to Maven-->
 < version > 5.9.8.0-SNAPSHOT </ version >
 <!-- Compilation order priority, the smaller the priority, the higher the priority -->
 < priority > 0 </ priority >
 </ module >
 </ modules >
 <!-- Note that the personal ssh name in the warehouse address should be replaced by the $user placeholder-->
 < repo > ssh://$user@smartgit:xxxx/VivoCode/xxxx_lib </ repo >
 <!-- Development branch, the script is used to automatically switch to this branch-->
 <devbranch> feature_5.9.0.0_xxx_dev </devbranch>​​​
 <!-- When making a release package, you must specify commitId to ensure that the specified code is obtained-->
 <commitid> cb4xxxxxx69d1 </commitid>​​​
 </ repository >
 <!-- Add multiple repositories if multiple repositories -->
 ...
 </ repositories >

3. The vsub.sh script is the entry point for various functions of the tool, such as:

• ./vsub.sh sync: pull all submodule codes, the codes are stored in the submodules directory under the main project
• ./vsub.sh publish: compile all sub-warehouses with one click and publish aar to Maven

4. The subbuild directory is used to output the git commit records of the sub-warehouse, and the subError directory is used to output the log when the sub-warehouse is compiled abnormally.

5. Key Function Implementation

The main functions of ModularDevTool are divided into two categories. One is code management, which is used to batch process git operations. The second is project construction, which realizes functions such as dynamic configuration of sub-module dependencies and sub-module release.

5.1 Code Management

The vsub.sh script encapsulates commonly used git commands and is used to batch process git operations of sub-warehouses. The implementation logic is relatively simple, and the git commands are encapsulated using shell scripts.

For example, the implementation logic of ./vsub.sh -pull is to first cd into the submodules directory (the submodules directory stores all sub-warehouse codes), then traverse into the sub-warehouse directory and execute the git pull --rebase command, so that one command can complete the same git operation on all sub-warehouses. The implementation logic is as follows:

 <!-- ./vsub.sh -pull code logic -- >
 cd submodules
 path = $currPath
 files = $(ls $path )
 for fileName in $files
 do
 if [ ! -d $fileName ]
 then
 continue
 fi
 cd $fileName
 echo -e "\033[33mEntering $fileName \033[0m"
 git pull --rebase
 cd ..
 done

5.2 Project Construction

(1) Sync function

Pull the code of all submodules into the submodules directory of the main project by executing the ./vsub.sh sync command.

The Sync command has three functions:

1) If the sub-warehouse code is not pulled, pull the code and switch to the devbranch configured in repositories.xml;

2) If the sub-warehouse code has been pulled, switch to the devbranch configured in repositories.xml;

3) Considering that in some scenarios (such as Jenkins builds), there may be exceptions when using branches to check out code, add the -c parameter after the sync command, and the commitid configured in repositories.xml will be used to check out the specified branch code.

The Sync process is as follows:

(2) Submodule dependency processing

Previously, when we relied on the code of different sub-warehouses, we needed to manually modify settings.gradle to import the sub-module, and then modify the dependencies in build.gradle, as shown below.

 <!-- settings.gradle -- >
 include ':app' , ':module_name_1' , ':module_name_2' , ':module_name_3' ...

 project( ':module_name_1' ).projectDir = new File( 'E:/AndroidCode/module_name_1/code/' )
 project( ':module_name_2' ).projectDir = new File( 'E:/AndroidCode/module_name_2/code/' )
 project( ':module_name_3' ).projectDir = new File( 'E:/AndroidCode/module_name_3/code/' )
 ... 

 <!-- build.gradle -- >
 dependencies {
 api fileTree(dir: 'libs' , include: [ '*.jar' ])
 //Business submodule begin
 api project ( ':module_name_1' )
 api project ( ':module_name_2' )
 api project ( ':module_name_3' )
 //Business submodule end
 }
 ...

Everyone in the team stores their code in a different location, so after pulling the code in the new version, they need to manually configure it, which is cumbersome.

Based on the sync function, all sub-warehouse codes have been pulled into the submodules directory. Now when building our project, we only need to simply configure local.properties (the local.properties configuration is shown in the figure below) to determine which sub-modules are code dependencies and which sub-modules are Maven dependencies.

 <!-- The key module_name_x indicates the submodule name, the value 0 indicates Maven dependency, and 1 indicates code dependency. The default is Maven dependency, that is, if some submodules are not configured, the default is Maven dependency-- >
 module_name_1 = 0
 module_name_2 = 0
 module_name_3 = 1
 module_name_4 = 1
 module_name_5 = 1
 module_name_6 = 1

The process of submodule dependency processing is as follows:

(3) Publish function

By executing the ./vsub.sh publish command, all submodule aars can be compiled and uploaded to Maven with one click.

The publish command has four main functions:

1) If the sub-warehouse code is not pulled, the sub-warehouse code will be pulled automatically;

2) If you are releasing a snapshot version, switch to the latest code of the devbranch branch, compile the submodule containing the snapshot string in the version to generate an aar and upload it to Maven; otherwise, skip it directly and do not compile;

3) If the release version is released (that is, the -a parameter is specified), switch to the code corresponding to commitid, compile and generate the aar of the release version, and upload it to Maven;

4) The compilation and upload order of the sub-warehouses is executed according to the configured priority.

Note: The above devbranch, version, commitid, priority, etc. are all configuration items in repositories.xml.

The process of publishing a submodule is as follows:

6. ModularDevTool Access

The premise of accessing this solution is that the project adopts a multi-code warehouse approach for modular development. The specific access steps are also relatively simple.

The first step is that the main warehouse depends on the gradle plug-in modular_dev_plugin;

(This plug-in contains four sub-plugins: settings, tools, base, and publish. The settings, tools, and base plug-ins work together to implement sub-warehouse code management and dynamic dependency processing, and the publish plug-in implements the aar publishing of the sub-warehouse)

The second step is to apply the settings plug-in in the main warehouse's settings.gradle, and apply the tools and base plug-ins in the main warehouse's app build.gradle;

Step 3: Add the repositories.xml configuration file and vsub script to the root directory of the main warehouse;

Step 4: The sub-warehouse depends on modular_dev_plugin and applies the publish plug-in;

Step 5: The middle-level sub-warehouse (e.g. App→Shop→Lib, where Shop is the middle-level sub-warehouse) changes the version number of the dependency on the next-level sub-warehouse into a placeholder, which will be automatically replaced with the version number in repositories.xml when the project is built. See the figure below:

 dependencies {
 // The dependency on the lib repository used to depend on a specific version number, but now it has been changed to a "unified" placeholder, which will be automatically replaced with the version number in repositories.xml when the project is built
 api "com.vivo.space.lib:vivospace_lib:unified"
 }

At this point, ModularDevTool has been connected.

7. Current development process

Based on this tool, the development process of our official website is now as follows:

The first step is to clone the main App repository code, checkout the corresponding development branch, and open the project in AndroidStudio;

The second step is to modify the repositories.xml configuration, change devbranch to the corresponding development branch, and change version to the corresponding version number for the sub-warehouse that needs to be developed;

The third step is to check out all submodule codes through the ./vsub.sh sync command;

The fourth step is to modify the sub-warehouse dependency mode in local.properties (Maven dependency or code dependency). After the modification, click Sync , and then you can develop the code normally. The development experience is exactly the same as the single-project multi-module mode.

8. Conclusion

This tool is already very mature and has been used for many years in projects such as vivo wallet and vivo official website. Through this tool, in the development phase, integrated development of multiple business modules is achieved, solving tedious operations such as decentralized management of code repositories and manual configuration of dependencies. In the release phase, multiple compilation modes and one-click packaging capabilities are implemented, which greatly improves the development efficiency of the team, supports parallel iteration of 3+ business lines of the official website app project, and reduces code conflicts by more than 50%.