Fastbot Open Source Technical Principles and Architecture

Background

In recent years, the number of mobile applications has exploded, and with it comes users' high demands for application quality. Ensuring the quality of applications is critical to maintaining user loyalty and business success. However, traditional manual testing methods have some challenges, including a lot of time and resources, scalability and maintainability limitations, etc., so automated testing tools for mobile applications have emerged. With the advent of the artificial intelligence era, the field of software testing continues to develop towards intelligence, among which the ability to automatically generate tests has always been a research hotspot of common concern in academia and industry. Relying on automatically generated tests can greatly reduce the workload of writing and maintaining test scripts.

In actual applications, industrial applications often need to be updated to adapt to changing user needs. For example, industrial manufacturers generally release a new version of the core application every week. However, the existing automated testing tools are generally ineffective in this case. They simply rerun the test on the mobile application and lack the use of human experience and knowledge. Fastbot is an automated Android testing tool that combines reinforcement learning and model-based decision algorithms. It was proposed by the ByteDance Software Engineering Laboratory (https://se-research.bytedance.com/) team. It aims to use reinforcement learning technology to gain knowledge from previous test runs through learning and reasoning, so as to achieve faster and more efficient testing results. In addition, Fastbot also provides a personalized expert system, and users can customize various configurations to meet different test scenarios and needs.

The following will introduce the tool's workflow, core concepts, algorithm principles, design implementation, and common configurations one by one, and gain an in-depth understanding of its ability to improve the efficiency of mobile application testing.

Workflow

Fastbot is a reusable model-based automated Android testing tool that leverages reinforcement learning. It accepts a given app version in the form of an APK file as input and outputs coverage reports and found crashes. Fastbot's workflow consists of two main stages, as shown in the figure:

(a)   Pre-test setup. a1 Decompile the APK file and collect static text information of the controls. a2 Install the APK on a set of devices, while a3 load historical test data from previous test runs to populate the model.

(b)   Guided UI exploration. b1 gets the current GUI page from the application under test, b2 identifies and abstracts the available hyper-events (a set of events with the same properties, which will be explained in detail below) on the current page. b3 selects a specific UI event that may increase the Activity coverage, and b4 executes the event. After execution, b5 updates the historical test data and probability model, and b6 updates the reinforcement learning agent.

These steps will be repeated until the time budget is exhausted.

Core Concepts

Technology and algorithm principles

The key idea of ​​Fastbot is to use the prior knowledge stored in the probabilistic model to effectively guide GUI testing. To achieve this, the key step is to decide which UI event to select on the current GUI page, which can quickly improve the Activity coverage. Specifically, given a GUI page, Fastbot extracts all currently available super events and uses the following two strategies to synergistically select the event to execute: (1) model-based event selection, (2) learning-based event selection.

1. Event selection based on probability model

In order to balance the model's exploration of unknown actions and the utilization of known high-reward actions and avoid falling into the deadlock of local optimality, model-based event selection includes two modes: model extension and model utilization.

Model extension: If some super events in the current GUI page have not been included in the probability model 𝑀, Fastbot will start the extension strategy and randomly select a super event that has not been executed. There are two possible situations in practice: 1) the previous test run may not cover all super events, 2) some new features have been added in the current test application version. This mode can help extend the model and prioritize the exploration of potential new features.

The algorithm principle of Fastbot is to use previous knowledge and probability models to select events through two strategic modes: model expansion and model utilization, so as to quickly improve the Activity coverage of GUI testing. The design of this algorithm enables GUI testing to be more efficient and adaptable to different application scenarios.

2. Event selection based on reinforcement learning

The previous probability model can only express one-step guidance information, while reinforcement learning technology can expand one step into multiple steps of guidance information. Fastbot uses the Sarsa N-Step algorithm as the reward function to calculate and update the Q value.

3. Case Explanation

To deepen our understanding, let’s use the Toutiao app to explain several decisions made by the algorithm:

Probabilistic Model Exploration and Exploitation Mode

The probability model M is initialized by recording historical data, as shown in Figure b. After starting the Toutiao app, it enters home Activity1. The current page can be abstracted into 3 hyper-events. These three events are all included in the model, so Fastbot starts the model again to select events using strategies. Looking at the left part of Figure b, we can see that e3 has a 90% probability of staying in Activity1, so it is more inclined to choose e1 and e2. Assuming that after selecting e1, it reaches Activity2, in which e4 is already in the model and 100% returns to Activity1, while e5 and e6 are no longer in the model. At this time, Fastbot will start the exploration mode and randomly select e5 or e6. If e5 is selected and reaches Activity3, the model adds a probability value of 100% for e5-Activity3. At the same time, since the number of executions of e1 is increased by 1, the probability of e1 will also change. The probability of e1-Activity2 becomes 0.7/1.1=63.6%, and the probability of e1-Activity5 The probability becomes 36.4% and is updated to the right half of Figure b.

Update and use of Q table

In Figure a, Activity2, e4, and e6 have not been executed, and e6 is not in the model, so the reward of e1 is higher. Similarly, e7 and e8 are also new events, so the reward of e5 will be higher. Assuming that e1, e2, and e3 have been executed many times, and Activities2, 3, and 4 are all covered, the values ​​in the Q table will be used to calculate the probability of event selection. In the current Activity1, since e1 has the highest reward, it means that it can reach a deeper Activity, so e1 is selected as the next event to be executed.

Design and implementation

The open source version of Fastbot consists of two parts: the client and the local server. The client is responsible for monitoring UI events, receiving and injecting corresponding actions, and the server is responsible for calculation and decision-making. Specifically, the Fastbot client is run on each device, and the GUI page information is sent to the server by monitoring. The server receives the information and returns the selected event based on the algorithm decision. The client receives the event and performs the operation.

The client is written in Java and is used to obtain GUI information, read server decisions, and convert decisions into code that can be executed by the device to operate the device; the local server, that is, the underlying C/C++ code layer corresponding to the Java layer, interacts with the Java layer through the JNI interface in the form of a dynamic link library (.so file). It is used to realize the reading of user preferences, model learning and task decision-making, and convert the decision results into JSON format information and pass it to the Java layer.

The implementation of Fastbot inherits the Monkey native framework. The figure below shows a brief structure of the Fastbot extension code. In the Java layer, the MonkeySourceApeNative class, the event source returned by the Fastbot algorithm, the acquisition of the GUI tree, and the Adapter adapter interface compatible with different Android systems are added. In addition, in order to support the parsing and execution of custom operations, the original events of Monkey are encapsulated.

Common configuration

1. Configure custom events

Fastbot supports custom event sequences, which are suitable for scenarios with incomplete coverage and can be reached through manual configuration in scenarios that Fastbot cannot traverse.

Configuration steps:

1. Create a new max.xpath.actions file (the file name is fixed and cannot be changed)
2. Specify controls and corresponding actions in the case format.

1. prob: probability of occurrence, "prob": 1, represents a 100% probability of occurrence
2. activity: the scene
3. times: number of repetitions, the default value is 1
4. actions: the specific sequence of events to be executed, which specifies the xpath of each operation object, the corresponding action, and the time interval (ms) between the next event. Pay attention to the xpath writing method.

3. Action type (must be capitalized):

4. CLICK: Click. If you want to enter content, add text under action. If there is text, execute text input.

5. LONG_CLICK: long press

6. BACK: Return

7. SCROLL_TOP_DOWN: scroll from top to bottom

8. SCROLL_BOTTOM_UP: Slide from bottom to top

9. SCROLL_LEFT_RIGHT: swipe from left to right

10. SCROLL_RIGHT_LEFT: swipe from right to left

11. After the configuration is complete, push the configuration file to the mobile phone: adb push 路径+max.xpath.actions /sdcard

    
    

Let's take AmazeFileManager as an example:

1. The first case: When the event execution does not involve the jump of Activity, just write all the event sequences in one object.

As shown in the figure, the four actions in the actions field correspond to the actions marked in red in the figure below: (1) Click the menu button to open the menu bar, (2) click the recent file button to go to the corresponding file directory, (3) click the plus button to open the floating options, and (4) click the File button to open the New File dialog box.

 [ { "prob":1, "activity":"com.amaze.filemanager.ui.activities.MainActivity", "times":1, "actions":[ { "xpath":"//*[@content-desc='Navigate up']", "action":"CLICK", "throttle": 2000 }, { "xpath":"//*[@resource-id='com.amaze.filemanager:id/design_menu_item_text' and @text='Recent files']", "action":"CLICK", "throttle": 2000 }, { "xpath":"//*[@resource-id='com.amaze.filemanager:id/sd_main_fab']", "action":"CLICK", "throttle": 2000 }, { "xpath":"//*[@resource-id='com.amaze.filemanager:id/menu_new_file']", "action":"CLICK", "throttle": 2000 } ] } ]

2. The second situation: When the event execution involves the jump of Activity, the event sequences corresponding to different Activities should be stored separately.

The following figure shows an example of jumping from MainActivity to PreferencesActivity.

You can see that there are two objects in the configuration file, corresponding to the event sequence that needs to be executed on the MainActivity and PreferencesActivity pages.

 [ { "prob":1, "activity":"com.amaze.filemanager.ui.activities.MainActivity", "times":1, "actions":[ { "xpath":"//*[@content-desc='Navigate up']", "action":"CLICK", "throttle": 2000 }, { "xpath":"//*[@resource-id='com.amaze.filemanager:id/design_navigation_view']", "action":"SCROLL_BOTTOM_UP", "throttle": 2000 }, { "xpath":"//*[@text='Settings' and @resource-id=['com.amaze.filemanager:id/design_menu_item_text']", "action":"CLICK", "throttle": 2000 } ] }, { "prob":1, "activity":"com.amaze.filemanager.ui.activities.PreferencesActivity", "times":1, "actions":[ { "xpath":"//*[@resource-id='android:id/title' and @text='Appearance']", "action":"CLICK", "throttle": 2000 } ] } ]

3. Case with text input

Configuration format: Select "CLICK" in the action field, and add the "text" field to store the text to be entered.

Taking Tik Tok as an example, the following configuration information can be used to specify the operation of entering account information. The two actions correspond to clicking the use phone button and entering text.

 [ { "prob":1, "activity":"com.ss.android.ugc.aweme.account.login.auth.I18nSignUpActivityWithNoAnimation", "times":1, "actions":[ { "xpath":"//*[@resource-id='com.zhiliaoapp.musically:id/ayo' and @text='Use phone / email / username']", "action":"CLICK", "throttle": 2000 } ] }, { "prob":1, "activity":"com.ss.android.ugc.aweme.account.login.v2.ui.SignUpOrLoginActivity", "times":1, "actions":[ { "xpath":"//*[@text='Phone number' and @resource-id=['com.zhiliaoapp.musically:id/e61']", "action":"CLICK", "text":"12341828506", "throttle": 2000 } ] } ]

2. Block controls

Fastbot supports manual configuration of controls or areas that need to be blocked, such as logging out "halfway" during the test and blocking the logout button.

Configuration steps:

1. Create a new max.widget.black file (the file name is fixed and cannot be changed)
2. Refer to the case format to specify the controls that need to be shielded. The format is as follows:

• bounds: Shields an area so that the controls or coordinates within the area will not be clicked.
• xpath: Find the matching control and block the click of the control.
• xpath+bounds: Finds matching controls and masks the specified area if the controls exist.

3. activity: When activity matches currentactivity, the following matches are performed
4. There are three ways to mask a control or area:
5. After the configuration is complete, push the configuration file to the mobile phone: adb push 路径+max.widget.black /sdcard

The following uses AmazeFileManager as an example to show how to perform control, region, and tree pruning shielding:

1. Block controls

As shown in Figure (a), the black area controls marked by the red frame are manually configured shielding controls. Here, xpath is used to specify the controls that need to be shielded.

 [ { "activity":"com.ss.android.ugc.aweme.main.MainActivity", "xpath":"//*[@content-desc='Navigate up']" }, { "activity":"com.ss.android.ugc.aweme.main.MainActivity", "xpath":"//*[@content-desc='More options']" } ]

2. Shielded Area

As shown in Figure (b), the black area control marked by the red frame is the manually configured shielding area. Here, bounds is used to configure the shielding area.

 [ { "activity":"com.ss.android.ugc.aweme.main.MainActivity", "bounds":"[0,18],[240，60]" } ]

3. Tree pruning shield

Tree pruning and masking means finding the corresponding control in the GUI tree and masking the control by setting the enable property of the control to False.

Configuration steps:

1. Configure the max.tree.pruning file (the file name is fixed and cannot be changed)
2. Refer to the case format to specify the controls that need to be shielded. The format is as follows:

1. activity: When activity matches currentactivity, the following matches are performed
2. Pruning method: configure xpath, find matching controls, and change control properties to shield the controls.

3. After the configuration is complete, push the configuration file to the phone: adb push 路径+max.tree.pruning /sdcard

Example: Still taking AmazeFileManager as an example, as shown in the above figure (c), the black area marked by the red frame is the manually configured tree pruning shielding control, which can be shielded by changing the enabled property of the control to false, or changing other properties to empty.

 [ { "activity":"com.ss.android.ugc.aweme.main.MainActivity", "xpath":"//*[@content-desc='Navigate up']", "enabled":"false" }, { "activity":"com.ss.android.ugc.aweme.main.MainActivity", "xpath":"//*[@content-desc='More options']", "enabled":"false" }, { "activity":"com.ss.android.ugc.aweme.main.MainActivity", "xpath":"//*[@resource-id='com.amaze.filemanager:id/search'", "resourceid":"", "contentdesc":"", "text":"", "classname":"" } ]

More information

Fastbot open source address: https://github.com/bytedance/Fastbot_Android

For more technical details of Fastbot, please refer to the paper: "Fastbot2: Reusable Automated Model-based GUI Testing for Android Enhanced by Reinforcement Learning". Zhengwei Lv, Chao Peng, Zhao Zhang, Ting Su, Kai Liu, Ping Yang. 37th IEEE/ACM International Conference on Automated Software Engineering (ASE 2022).