In-depth understanding of Android Studio Sync process

1. Getting to know Sync

We generally understand Sync as the preparation stage of Android Studio, including parsing project configuration information, downloading remote dependencies to local, updating code index and other preparatory work. After modifying the gradle build file, you need to re-Sync to synchronize the Gradle build configuration information to the IDE, so that the IDE functions can apply the new build configuration in time. These functions include the display of the project's Gradle Task list, dependency information display, etc. Sync is a unique concept in Android Studio. When building an Android application through the Gradle command line program, it only goes through the Initialization, Configuration and Execution life cycle defined by Gradle, and there is no concept of Sync at all. The Sync stage of Android Studio involves multiple roles such as IDE, Gradle, Plugin, etc. Only by sorting out the functions and connections of these roles can we clearly understand the overall architecture of Sync. The following introduces these roles separately:

IDE level: Android Studio is based on IntelliJ IDEA and reuses IntelliJ IDEA's powerful IDE capabilities such as code editor and developer tools. On top of this, Android Studio provides more features to improve Android build efficiency, such as Android simulator, code templates, etc. In addition, Android Studio also feeds back its professional Android application development capabilities to IntelliJ IDEA in the form of Android IDEA Plugin, enabling IntelliJ IDEA to support Android application development. The two empower each other and complement each other.

Gradle level: Gradle is a flexible and powerful open source build system. In addition to providing cross-platform executable programs to support command line execution of Gradle builds, it also provides a special Gradle Tooling API programming SDK for external parties to more conveniently and tightly embed Gradle build capabilities into IDEs. IDEs such as IntelliJ IDEA, Eclipse, and VSCode all use this approach. There are also code modules in the Gradle source code that specifically serve IDEs such as IntelliJ IDEA and Eclipse. The two roles of build tools and IDEs also empower each other and form a powerful combination.

Plugin level: The Plugin level includes Android IDEA Plugin and Android Gradle Plugin. Android IDEA Plugin expands the Android application development capabilities for IntelliJ IDEA/Android Studio; Android Gradle Plugin expands the Android application building capabilities for Gradle. Through these two plugins, Google combines modern, mature and excellent IDE development capabilities and building tools for Android. Compared with the early Eclipse plus ANT building development method, it greatly improves the Android application development efficiency and experience.

2. Sync process analysis

After understanding the roles involved in the Sync phase and the relationships between them, we will delve into the Android Studio source code to sort out the key processes of Sync from the code level.

2.1 Android Studio source code analysis

2.1.1 Function entry and preparation

After the Sync operation is triggered in Android Studio, the top-level entry class GradleSyncInvoker will be called to the GradleProjectResolver that is actually responsible for parsing the Gradle build information. The call chain is shown in the following figure:

Key classes involved in the calling process:

GradleSyncInvoker: The entry class for triggering Sync. In many places in Android Studio where Sync needs to be executed, it is triggered by calling the requestProjectSync method of this class.

ExternalSystemUtil: GradleSyncInvoker, GradleSyncExecutor and other classes are encapsulations specifically for the Sync function. Sync is a unique operation in Android Studio. There is no concept of Sync in IntelliJ IDEA. IntelliJ IDEA triggers the Gradle build information of the parsed project through the [Reload All Gradle Projects] operation and starts execution directly from the ExternalSystemUtil class.

GradleProjectResolver: Responsible for the specific execution of Sync, where the resolveProjectInfo method is where the Sync logic is specifically executed. The definition of this method is as follows:

 public DataNode<ProjectData> resolveProjectInfo(
 @NotNull ExternalSystemTaskId id,
 @NotNull String projectPath,
 boolean isPreviewMode,
 @Nullable S settings,
 @NotNull ExternalSystemTaskNotificationListener listener)
 throws ExternalSystemException, IllegalArgumentException, IllegalStateException

• id: The unique identifier of this Sync operation. You can cancel this Sync task by calling the cancelTask ​​method in GradleProjectResolver.
• projectPath: absolute path of the project
• settings: Configuration parameters of the Sync project, which can set the Java version, etc.
• listener: used to monitor the process and results of this Sync
• isPreviewMode: Whether to enable preview mode. When Android Studio opens a project from an unknown source for the first time, it will ask the developer to choose how to open the project. As shown in the figure below, if you select [Stay in Safe Mode], the project will be run in "preview mode", which means that the IDE can only browse the source code of the project and will not execute or parse any build tasks and scripts.

After entering the resolveProjectInfo method of GradleProjectResolver, the preview mode will be processed first, as shown in the following code. If it is preview mode, the corresponding project data structure will be simply constructed and returned immediately without any parsing behavior:

 if (isPreviewMode) {
 String projectName = new File(projectPath).getName();
 ProjectData projectData = new ProjectData(GradleConstants.SYSTEM_ID, projectName, projectPath, projectPath);
 DataNode<ProjectData> projectDataNode = new DataNode<>(ProjectKeys.PROJECT, projectData, null);
 ......
 return projectDataNode;
 }

When opening a project with Android Studio/IntelliJ IDEA, in addition to specifying the project's root directory, you can also specify a Gradle configuration file. This logic is also reflected in the source code:

 if (projectPathFile.isFile() && projectPath.endsWith(GradleConstants.EXTENSION) && projectPathFile.getParent() != null) {
 projectDir = projectPathFile.getParent();
 if (settings != null) {
 List<String> arguments = settings.getArguments();
 if (!arguments.contains("-b") && !arguments.contains("--build-file")) {
 settings.withArguments("-b", projectPath);
 }
 }
 } else {
 projectDir = projectPath;
 }

You can see that if you open a configuration file instead of the project root directory, the directory where the configuration file is located will be used as the project path, and the configuration file will be specified through the --build-file parameter, which will be passed to Gradle later. After preliminary processing of the project, the BuildEnvironment of the project is obtained through the Gradle Tooling API:

 ModelBuilder<BuildEnvironment> modelBuilder = connection.model(BuildEnvironment.class);

When the Gradle Tooling API is called, the Gradle version of the project configuration will be automatically downloaded. That is to say, the call to obtain the BuildEnvironment Model in the above code itself ensures the download of Gradle, that is, the Gradle specified by the distributionUrl in /grade/wrapper/gradle-wrapper.properties is downloaded to GRADLE_HOME/wrapper/dists.

BuildEnvironment is the Gradle Model provided by Gradle. Gradle Model is a very important concept. When the IDE interacts with Gradle through the Gradle Tooling API, various models are transmitted, such as Gradle's own GradleProject and BuildEnvironment. In addition, you can also register a custom Model through the ToolingModelBuilderRegistry in the Gradle Plugin. For example, if the AndroidProject Model is registered in the Android Gradle Plugin, you can get the AndroidProject Model unique to the Android project through the Gradle Tooling API, thereby obtaining the Android application-related project information provided by the Android Gradle Plugin.

2.1.2 Configure BuildAction

Continuing to analyze the Android Studio Sync source code, a ProjectImportAction is constructed, which implements the BuildAction interface in the Gradle Tooling API. The BuildAction definition is as follows:

 public interface BuildAction<T> extends Serializable {

 T execute(BuildController controller);
 }

BuildAction is the behavior that will be passed to the Gradle build process for execution, and the result data can be serialized and returned to the caller. This BuildAction is crucial. It is the place where Gradle actually communicates. It implements functions such as organizing and generating project information and downloading dependencies. It is the core logic of the Sync process. BuildAction is combined with BuildActionExecuter in the Gradle Tooling API to let Gradle trigger the execution of BuildAction. Before execution, you need to configure JVM parameters, Gradle command line parameters, environment variables and other build information through BuildActionExecuter:

 private static void configureExecutionArgumentsAndVmOptions(@NotNull GradleExecutionSettings executionSettings,
 @NotNull DefaultProjectResolverContext resolverCtx,
 boolean isBuildSrcProject) {
 executionSettings.withArgument("-Didea.sync.active=true");
 if (resolverCtx.isResolveModulePerSourceSet()) {
 executionSettings.withArgument("-Didea.resolveSourceSetDependencies=true");
 }
 if (!isBuildSrcProject) {
 for (GradleBuildParticipant buildParticipant : executionSettings.getExecutionWorkspace().getBuildParticipants()) {
 executionSettings.withArguments(GradleConstants.INCLUDE_BUILD_CMD_OPTION, buildParticipant.getProjectPath());
 }
 }
 GradleImportCustomizer importCustomizer = GradleImportCustomizer.get();
 GradleProjectResolverUtil.createProjectResolvers(resolverCtx).forEachOrdered(extension -> {
 if (importCustomizer == null || importCustomizer.useExtraJvmArgs()) {

 ParametersList parametersList = new ParametersList();
 for (Pair<String, String> jvmArg : extension.getExtraJvmArgs()) {
 parametersList.addProperty(jvmArg.first, jvmArg.second);
 }
 executionSettings.withVmOptions(parametersList.getParameters());
 }

 executionSettings.withArguments(extension.getExtraCommandLineArgs());
 });
 }

There are many codes above. We will focus on the withArgument and withVmOptions methods of GradleExecutionSettings, which are responsible for collecting Gradle command line parameters and JVM parameters respectively. As can be seen from the code, most of these Gradle command line parameters and JVM parameters are collected from extensions. Here, extensions refer to extensions in IntelliJ IDEA Plugin. When used in conjunction with extension points, they can extend the features of IntelliJ IDEA platform or other functional features of IntelliJ IDEA Plugin. For example, Gradle IDEA Plugin provides extension points for project parsing:

 <extensionPoint
 qualifiedName="org.jetbrains.plugins.gradle.projectResolve"
 interface="org.jetbrains.plugins.gradle.service.project.GradleProjectResolverExtension"/>

This extension point is implemented in Android IDEA Plugin, which provides the extension of Android project parsing:

 <extensions defaultExtensionNs="org.jetbrains.plugins.gradle">
 <projectResolve implementation=
 "com.android.tools.idea.gradle.project.sync.idea.AndroidGradleProjectResolver"
 order="first"/>
 ......
 </extensions>

Next, let's look at the parameter configuration logic of BuildAction. The final place to set the JVM parameters is in the prepare method of GradleExecutionHelper:

 List<String> jvmArgs = settings.getJvmArguments();
 BuildEnvironment buildEnvironment = getBuildEnvironment(connection, id, listener, (CancellationToken)null, settings);
 if (!jvmArgs.isEmpty()) {

 Collection<String> merged;
 if (buildEnvironment != null) {


 BuildIdentifier buildIdentifier = getBuildIdentifier(buildEnvironment);
 List<String> buildJvmArguments = buildIdentifier == null || "buildSrc".equals(buildIdentifier.getRootDir().getName())
 ContainerUtil.emptyList()
 : buildEnvironment.getJava().getJvmArguments();
 merged = mergeBuildJvmArguments(buildJvmArguments, jvmArgs);
 } else {
 merged = jvmArgs;
 }
 List<String> filteredArgs = ContainerUtil.mapNotNull(merged, s -> StringUtil.isEmpty(s) ? null : s);
 operation.setJvmArguments(ArrayUtilRt.toStringArray(filteredArgs));
 }

As shown in the code above, the configuration logic of JVM parameters is very simple: merge the JVM parameters previously collected from a series of GradleProjectResolve extensions and stored in GradleExecutionSettings with the JVM parameters in BuildEnvironment, and then call the setJvmArguments method of BuildActionExecuter to set the JVM parameters to BuildAction. Gradle command line parameters are also configured in the prepare method of GradleExecutionHelper:

 ...
 List<String> filteredArgs = new ArrayList<>();
 if (!settings.getArguments().isEmpty()) {
 String loggableArgs = StringUtil.join(obfuscatePasswordParameters(settings.getArguments()), " ");
 LOG.info("Passing command-line args to Gradle Tooling API: " + loggableArgs);

 filteredArgs.addAll(ContainerUtil.mapNotNull(settings.getArguments(), s -> StringUtil.isEmpty(s) ? null : s));
 ...
 }
 filteredArgs.add("-Didea.active=true");
 filteredArgs.add("-Didea.version=" + getIdeaVersion());
 operation.withArguments(ArrayUtilRt.toStringArray(filteredArgs));

For a simplest Kotlin App Demo project, the Gradle command line parameters are as follows:

Focus on the --init-script command line parameter, which can specify an initialization script that will be executed before the project build script. The initialization script is a very flexible mechanism provided by Gradle. In addition to command line configuration, you can also name the initialization script init.gradle and place it in USER_HOME/.gradle/ for configuration. The initialization script allows you to customize the build logic of all projects, such as defining the JDK path and other environmental information for all projects on a specific machine. In the above Kotlin App Demo project, the Android IDEA Plugin extension configures an initialization script with the following content:

 initscript {
 dependencies {
 classpath files([mapPath('/Users/bytedance/IDE/intellij-community/out/production/intellij.android.gradle-tooling'), mapPath('/Users/bytedance/IDE/intellij-community/out/production/intellij.android.gradle-tooling.impl'), mapPath('/Users/bytedance/.m2/repository/org/jetbrains/kotlin/kotlin-stdlib/1.5.10-release-945/kotlin-stdlib-1.5.10-release-945.jar')])
 }
 }
 allprojects {
 apply plugin: com.android.ide.gradle.model.builder.AndroidStudioToolingPlugin
 }

The AndroidStudioToolingPlugin plugin is applied to all projects in the initialization script. This plugin registers the AdditionalClassifierArtifactsModel through the ToolingModelBuilderRegistry. This Gradle Model implements the functions of downloading dependent sources and javadoc:

 class AdditionalClassifierArtifactsModelBuilder : ParameterizedToolingModelBuilder<AdditionalClassifierArtifactsModelParameter> {
 ...

In other words, Android IDEA Plugin provides the download function of dependent sources and javadoc. When the AdditionalClassifierArtifactsModel Gradle Model is obtained through the Gradle Tooling API, the dependent sources and javadoc downloads will be triggered.

After analyzing the JVM parameters of BuildAction and the Gradle command line parameter configuration process, let's take a look at the configuration of BuildAction environment variables. The final configuration is in the setupEnvironment method of GradleExecutionHelper:

 GeneralCommandLine commandLine = new GeneralCommandLine();
 commandLine.withEnvironment(settings.getEnv());
 commandLine.withParentEnvironmentType(
 settings.isPassParentEnvs() ? GeneralCommandLine.ParentEnvironmentType.CONSOLE : GeneralCommandLine.ParentEnvironmentType.NONE);
 Map<String, String> effectiveEnvironment = commandLine.getEffectiveEnvironment();
 operation.setEnvironmentVariables(effectiveEnvironment);

GeneralCommandLine includes all environment variables of the current Java process. Other environment variables and JVM parameters are similarly collected in GradleExecutionSettings. Android Studio will first merge other environment variables with those in GeneralCommandLine and then configure them to BuildAction. For the default sync behavior without any configuration, the environment variables in GradleExecutionSettings are empty and are all provided by GeneralCommandLine.

2.1.3 Execute BuildAction

After analyzing the configuration logic of BuildAction, let's look at what is done in BuildAction. The behavior in BuildAction is no longer in the Android Studio IDE process, but is executed in the Gradle build process. When the IDE interacts with Gradle through the Gradle Tooling API, the main medium is the Gradle Model, and BuildAction is no exception. The specific execution logic of BuildAction can be found in the execute method in its implementation class ProjectImportAction. We only focus on the code related to the Gradle Model in this method:

 public AllModels execute(final BuildController controller) {
 ...
 fetchProjectBuildModels(wrappedController, isProjectsLoadedAction, myGradleBuild);
 addBuildModels(wrappedController, myAllModels, myGradleBuild, isProjectsLoadedAction);
 ...
 }

In BuildAction, the fetchProjectBuildModels and addBuildModels methods are called to obtain the Gradle Model. Let's first analyze the fetchProjectBuildModels method, which further calls the getProjectModels method:

 private List<Runnable> getProjectModels(@NotNull BuildController controller,
 @NotNull final AllModels allModels,
 @NotNull final BasicGradleProject project,
 boolean isProjectsLoadedAction) {
 ...
 Set<ProjectImportModelProvider> modelProviders = getModelProviders(isProjectsLoadedAction);
 for (ProjectImportModelProvider extension : modelProviders) {
 extension.populateProjectModels(controller, project, modelConsumer);
 }
 ...
 }

As shown in the code above, the populateProjectModels method of ProjectImportModelProvider is used to further obtain the Gradle Model. The addBuildModels method of BuildAction is very similar to this:

 private void addBuildModels(@NotNull final ToolingSerializerAdapter serializerAdapter,
 @NotNull BuildController controller,
 @NotNull final AllModels allModels,
 @NotNull final GradleBuild buildModel,
 boolean isProjectsLoadedAction) {
 Set<ProjectImportModelProvider> modelProviders = getModelProviders(isProjectsLoadedAction);
 for (ProjectImportModelProvider extension : modelProviders) {
 extension.populateBuildModels(controller, buildModel, modelConsumer);
 }
 ...
 }

You can see that the same method is handed over to ProjectImportModelProvider to obtain the Gradle Model. The difference is that the former calls populateProjectModels, while the populateBuildModels method is called here. The role of ProjectImportModelProvider is to generate the Gradle Model. ProjectImportModelProvider is provided by a series of Gradle IDEA Plugin extensions, just like JVM parameters and Gradle command line parameters, as shown in the following code:

 for (GradleProjectResolverExtension resolverExtension = tracedResolverChain;
 resolverExtension != null;
 resolverExtension = resolverExtension.getNext()) {
 ...
 ProjectImportModelProvider modelProvider = resolverExtension.getModelProvider();
 if (modelProvider != null) {
 projectImportAction.addProjectImportModelProvider(modelProvider);
 }
 ProjectImportModelProvider projectsLoadedModelProvider = resolverExtension.getProjectsLoadedModelProvider();
 if (projectsLoadedModelProvider != null) {
 projectImportAction.addProjectImportModelProvider(projectsLoadedModelProvider, true);
 }
 }

For Android projects, focus on Android IDEA Plugin, which provides the ProjectImportModelProvider implementation class AndroidExtraModelProvider, and the populateProjectModels method of the implementation class is as follows:

 override fun populateProjectModels(controller: BuildController,
 projectModel: Model,
 modelConsumer: ProjectImportModelProvider.ProjectModelConsumer) {
 controller.findModel(projectModel, GradlePluginModel::class.java)
 ?.also { pluginModel -> modelConsumer.consume(pluginModel, GradlePluginModel::class.java) }
 controller.findModel(projectModel, KaptGradleModel::class.java)
 ?.also { model -> modelConsumer.consume(model, KaptGradleModel::class.java) }
 }

This method uses the Gradle Tooling API to obtain two Gradle models, GradlePluginModel and KaptGradleModel. GradlePluginModel provides information about the Gradle Plugins that have been applied to the project; KaptGradleModel provides information about Kotlin annotation processing. Next, let's look at the populateBuildModels method:

 override fun populateBuildModels(
 controller: BuildController,
 buildModel: GradleBuild,
 consumer: ProjectImportModelProvider.BuildModelConsumer) {
 populateAndroidModels(controller, buildModel, consumer)
 populateProjectSyncIssues(controller, buildModel, consumer)
 }

The populateAndroidModels and populateProjectSyncIssues methods are called respectively. Let's look at the populateProjectSyncIssues method first. It will further obtain the ProjectSyncIssues Gradle Model through the Gradle Tooling API to collect problems that occur during the Sync process. Then analyze the populateAndroidModels method, which is crucial in the entire Sync process. It obtains the Android project-related Gradle Model through the Gradle Tooling API:

 val androidModules: MutableList<AndroidModule> = mutableListOf()
 buildModel.projects.forEach { gradleProject ->
 findParameterizedAndroidModel(controller, gradleProject, AndroidProject::class.java)?.also { androidProject ->
 consumer.consumeProjectModel(gradleProject, androidProject, AndroidProject::class.java)
 val nativeAndroidProject = findParameterizedAndroidModel(controller, gradleProject, NativeAndroidProject::class.java)?.also {
 consumer.consumeProjectModel(gradleProject, it, NativeAndroidProject::class.java)
 }
 androidModules.add(AndroidModule(gradleProject, androidProject, nativeAndroidProject))
 }
 }

As shown in the above code, two Gradle Models AndroidProject and NativeAndroidProject registered by Android Gradle Plugin are obtained respectively. AndroidProject includes key information such as BuildType, Flavors, Variant, Dependency of Android application; NativeAndroidProject includes relevant information of Android C/C++ project such as NDK and NativeToolchain. After obtaining the most critical AndroidProject and NativeAndroidProject, the next step is to process single Variant Sync:

 if (syncActionOptions.isSingleVariantSyncEnabled) {
 chooseSelectedVariants(controller, androidModules, syncActionOptions)
 }

How to understand Single Variant Sync? Variant refers to the product variation of an Android application, such as the simplest Debug and Release version application packages. Variant corresponds to a set of build configurations, which are associated with the project source code structure and dependency list. Android applications may have multiple Variants. If all Variants are constructed during Sync, the overall time consumption may be extremely long, so Android Studio Sync will only construct one Variant by default and support Variant switching. If Single Variant Sync is enabled, the filter parameters will be passed in when obtaining AndroidProject above to inform Android Gradle Plugin that it does not need to construct Variant information when constructing AndroidProject.

2.2 Sync Process Overview

2.2.1 Android Studio perspective

The source code of the Android Studio Sync process is large and complex. This article focuses on the stage from triggering the Sync entry to the end of the Gradle build. There is also the subsequent processing of Gradle data, and the process of the language capability module to index code based on Gradle data. The language capability is a large and complex module, so this article will not expand on it.

The source code analysis section above divides the process from triggering the Sync entry to the end of the Gradle build into the Sync function entry and preparation, configuration of BuildAction, and execution of BuildAction. The overall process is shown in the following figure:

2.2.2 Gradle perspective

The above analysis of Android Studio Sync is from the perspective of IDE. The overall process is relatively complicated. Next, we will sort out the Sync process from the perspective of Gradle, focusing on the behavior on the Gradle side:

As shown in the figure above, for Gradle, in the Sync process, Android Studio will obtain a series of required Gradle Models from Gradle through the Gradle Tooling API. In addition to Gradle itself, Gradle Plugin can also provide a custom Gradle Model. In addition, in the Sync process, Gradle will go through its own defined life cycle. Focusing on this perspective, the process is sorted out as follows:

When the Gradle Model is obtained through BuildAction, the Gradle build behavior will be triggered. The Gradle build will go through the Initialization, Configuration, and Execution phases defined in its own life cycle.

3. Conclusion

This article first introduces the functions and connections of various roles in the Android Studio Sync process, so that you can have a clearer overall understanding of Sync. Then, it deeply analyzes the stages from triggering the Sync entry to the end of the Gradle build from the source code perspective, and explains key concepts such as Gradle Model and BuildAction in detail. Finally, it sorts out the Sync process from the Android Studio perspective and the Gradle perspective respectively.

Through in-depth analysis of the Android Studio Sync process, in addition to a deep understanding of the implementation principle of the Sync function, we also have a deeper understanding of its significance. Sync is an IDE preparation stage defined by Android Studio. In this preparation stage, key IDE functions need to be prepared in advance, and for these functions to be available, they need to obtain the necessary data. Based on this perspective, the optimization direction of the Sync process is also inspired to a certain extent: first, consider the definition of the Sync stage from the product level, and consider omitting or postponing the preparation of functions that are not really necessary for developers; then confirm the minimum data set required for the necessary functions, and unnecessary data can be omitted; finally, for the necessary data, find the fastest way to obtain it through more efficient implementation or caching.

4. Reference Links

https://mp.weixin.qq.com/s/cftj6WueoHlLh-So9REEXQ

https://developer.android.com/studio/intro

https://android.googlesource.com/platform/tools/base/+/studio-master-dev

https://plugins.jetbrains.com/developers