iOS UI Automation Testing Principles and Application Practices at Trip.com

[[429539]]

Preface

It has been a year since I joined Trip.com. Looking back on my work experience over the past year, I have spent about half of my time working on UI automation testing. As a result, I have conducted a deeper study of automation testing technologies on the iOS platform, and am currently responsible for the construction and maintenance of the department's App automation testing platform. Therefore, I would like to use this article to systematically and comprehensively organize the pitfalls I encountered and the technologies I learned to share with you.

The content of this article is as follows:

• Detailed explanation of the principles of XCUITest, an iOS/macOS UI automation testing framework
• Design of automated testing platform architecture based on Web Service
• Introduction and comparison of Appium and Macaca
• Current state of Trip.com App UI automation testing

Automated testing can be divided into white-box testing, black-box testing, and gray-box testing. This article mainly focuses on the XCUITest testing framework officially provided by Apple, and gradually explains the principles, architectural design ideas, and application scenarios of UI automated testing under the iOS operating system.

Detailed explanation of XCUITest principles

iOS UI Automation Testing Core Technology

In 2015, Apple released the UI automation testing framework XCUITest and integrated it into Xcode7. iOS/macOS UI automation testing relies on two core technologies: XCUITest and Accessibility.

XCUITest is a testing framework integrated into Xcode. If you want to use the UI testing function, you can check the Include Tests option when creating an iOS project, so that the project has the ability to automate testing. Accessibility technology is a complete solution provided by Apple for visually impaired users to use iOS/macOS Apps.

The Xcode project creates a UITests Target and runs tests. Its compiled product, Test App, is essentially a Deamon process that has an independent application lifecycle and is managed by the XCUIApplication type. The UITests Test App process drives the Host App (the main Target product of the project) at runtime and uses the relevant APIs of element review to drive the Host App to simulate user behavior interactions, thereby performing UI automated testing.

For Accessibility technology, developers need to note that the XCUITest framework cannot review all view elements by default, but only review elements whose text can be read by the VoiceOver function. For example, UIButton and UILabel, these views can be understood by voice for visually impaired users, while UIKit view elements such as UIImageView and UIView, which are not friendly to visually impaired people, are not reviewed by default, so Accessibility-related properties must be configured separately during coding to ensure that they support Accessibility and are visible in the element hierarchy of UI automation queries.

Automated testing based on the XCUITest framework and Accessibility technology is beneficial for App data consistency verification, but the UI consistency verification capability is relatively weak. For example, App can verify the results of certain data requests or the existence of a certain element, but the visual display effect still requires manual intervention.

XCUITest framework structure

The XCUITest test framework API mainly includes: UI Element Queries related types, such as XCUIElementQuery, UI Elements related types, such as XCUIElement, and Application Lifecycle test types, such as XCUIApplication.

Next, we create a simple Demo project to learn how to use the XCUITest framework and perform iOS UI automation testing.

Automated testing with Xcode UITests Target

Create a Demo project, check the Include Tests option, and write the following code in ViewController. The Demo project in this article can be accessed at https://github.com/niyaoyao/UITestDemo.

 import UIKit 
 
 class ViewController: UIViewController {
    lazy var testImageView: UIImageView = {
        let testImageView = UIImageView(frame: CGRect(x: 0, y: 0, width: 100, height: 100))
        testImageView.backgroundColor = .red
        testImageView.accessibilityIdentifier = "test imageview"  
 return testImageView
 }()
    lazy var testLabel: UILabel = {
        let testLabel = UILabel(frame: CGRect(x: 0, y: 130, width: 100, height: 20))
        testLabel.backgroundColor = .green
        testLabel.text = "test label"  
 return testLabel
 }()
    lazy var testView: UIView = {
        let testView = UIView(frame: CGRect(x: 0, y: 170, width: 100, height: 50))
        testView.backgroundColor = .blue
        testView.accessibilityIdentifier = "test view"  
 return testView
 }()
    lazy var testButton: UIButton = {
        let testButton = UIButton(frame: CGRect(x: 0, y: 230, width: 100, height: 50))
        testButton.backgroundColor = .yellow
 testButton.setTitle( "Test Button" , for : .normal)
 return testButton
 }() 
     
    override func viewDidLoad() {
 super.viewDidLoad()
 view .addSubview(testImageView)
 view .addSubview(testLabel)
 view .addSubview(testView)
 view .addSubview(testButton)
 }
 }

Source code explanation: The above code creates four view instances, namely UIImageView, UILabel, UIView and UIButton, and adds them to the current page. Among them, UILable and UIButton only set properties such as frame, string, background color, etc., but for UIImageView and UIView, in addition to general view properties, accessibilityIdentifier properties are also set to make UIImageView and UIView support Accessibility function, but setting this property alone does not make these two views visible in the element hierarchy of Accessibility. Next, we will briefly introduce the Accessibility function.

Make your app accessible

Using the Accessibility Inspector

As mentioned in the previous article, Apple will review view elements that can play text through VoiceOver by default. For UIImageView and UIView that do not support Accessibility functions by default, relevant features need to be configured. During the development process, developers can view the Accessibility element hierarchy of different processes through Accessibility Inspector. This application can review elements of iOS and macOS.

Select the Xcode icon menu and choose the Open Developer Tool option, then click Accessibility Inspector to get started.

When we do not set the isAccessibilityElement property, we cannot see the UIImageView and UIView elements in the Accessibility element hierarchy, and can only see the "test label" and "test button". When we set the isAccessibilityElement property of UIImageView and UIView to true, the UIImageView and UIView elements are visible in the element hierarchy.

Accessibility-related properties

 UIAccessibility: var accessibilityLabel: String? { get set }

The accessibilityLabel attribute can solve most Accessibility problems. When the cursor focuses on the element that sets this attribute, its content is a human-readable string that can be read by VoiceOver. However, if it is not a view element that needs to be known by visually impaired users and is only used for automated testing, you do not need to set this attribute.

 UIAccessibility: var accessibilityIdentifier: String? { get set }

The accessibilityIdentifier property is not spoken by VoiceOver but is a developer-targeted string that can be used in situations where you don't want users to manipulate the accessibilityLabel.

 UIAccessibility: var isAccessibilityElement: Bool { get set }

If isAccessibilityElement is not set to true, then the view will not be visible in the Accessibility view hierarchy.

• The default value for this property is false unless the element is a standard UIKit control, in which case, the value is true. —— Apple Documentation

In addition, according to Apple's official introduction, the isAccessibilityElement property of UIControl subclasses is set to true by default.

Manually write test cases

 import XCTest 
 
 class UITestDemoUITests: XCTestCase { 
 
    override func setUpWithError() throws {
 // ...
 } 
 
    override func tearDownWithError() throws {
        // Put teardown code here. This method is called after the invocation of each test method in the class.
 } 
 
    func testExample() throws {
        // UI tests must launch the application that they test.
        let app = XCUIApplication()
 app.launch()
        let label = app.staticTexts[ "test label" ]
        XCTAssertTrue(label.exists)
 let button = app.buttons[ "test button" ]
        XCTAssertTrue(button.exists)
        let imgview = app.images[ "test imageview" ]
        XCTAssertTrue(imgview.exists)
        let view = app.otherElements[ "test view" ]
        XCTAssertTrue( view .exists)
        // Use recording to get started writing UI tests.
        // Use XCTAssert and related functions to verify your tests produce the correct results.
 } 
 
    func testLaunchPerformance() throws {
        if #available(macOS 10.15, iOS 13.0, tvOS 13.0, watchOS 7.0, *) {
            // This measures how long it takes to launch your application.
            measure(metrics: [XCTApplicationLaunchMetric()]) {
                XCUIApplication().launch()
 }
 }
 }
 }

Source code explanation: An instance of the XCUIApplication type is an instance object that manages the life cycle of the Test App. The Accessibility view hierarchy can be obtained through this object, and the existence of an element can be asserted through XCTAssertTrue.

Record interactive behaviors to automatically generate test cases

For relatively complex test cases, you can use the test behavior recording function provided by Xcode to automatically generate code.

UITest Execution Process

Click the play button in front of the function defined by Test or the play button of the corresponding function in Test Navigator to start the UI test. After starting the UI test, the source code will be compiled first, the source code in Target will be compiled into a product, the Test App process will be started, and the App will be started by entering the Test program and executing app.launch(), and then the assertion source code will be executed.

iOS automated testing toolchain

After writing the UITest Target method for basic UI testing, we can use the relevant command line tool chain to script iOS UI automated testing, so that it can be easily integrated into the CI process.

xcodebuild

 xcodebuild test -project UITestDemo.xcodeproj -scheme UITestDemoUITests -destination 'platform=iOS,id=<iPhoneUDID>'  

You can use the above commands to perform automated testing, or you can split the commands into test compilation commands and test execution commands to refine the automated testing process. Test compilation commands:

 xcodebuild build- for -testing -project ****.xcodeproj -scheme **** -configuration Debug -sdk iphonesimulator -destination 'platform=iOS Simulator,id=XXXXX' -derivedDataPath ~/derived_path -quiet COMPILER_INDEX_STORE_ENABLE= NO GCC_WARN_INHIBIT_ALL_WARNINGS=YES | tee build.log

Test execution command:

 xcodebuild test-without-building -xctestrun ****.xctestrun -configuration Debug -sdk iphonesimulator -destination 'platform=iOS Simulator,id=XXXXXX' -derivedDataPath ~/derived_path -resultBundlePath ****.xcresult - only -testing:****-UITests/TargetTests

xcrun simctl

The simctl command is a set of commands from xcrun that provides a series of commands for controlling the iOS simulator.

List the currently started simulators xcrun simctl list devices | grep booted

Start the simulator xcrun simctl boot XXXXX

Shut down the simulator xcrun simctl shutdown XXXXX

Set simulator permissions xcrun simctl privacy XXX grant location-always xx.xx.xxx

Install App xcrun simctl install {} {}'.format(uuid, app_path)

Run the specified App xcrun simctl launch {} {}'.format(uuid, bundle_id)

End the specified App xcrun simctl terminate {} {}'.format(uuid, bundle_id)

Uninstall the specified App xcrun simctl uninstall {} {}'.format(uuid, bundle_id)

ideviceinstaller

Similar to controlling simulators, real iOS devices also have corresponding control command line tool chains, such as ideviceinstaller.

Install the app under apppath ideviceinstaller -i apppath

Install xxx.ipa as the local path of the application ideviceinstaller -u [udid] -i [xxx.ipa]

Uninstall the application ideviceinstaller -u [udid] -U [bundleId]

View third-party applications installed on the device ideviceinstaller -u [udid] -l

Same as above, view the third-party applications installed on the device ideviceinstaller -u [udid] -l -o list_user

View system applications installed on the device ideviceinstaller -u [udid] -l -o list_system

View all applications installed on the device ideviceinstaller -u [udid] -l -o list_all

List all user-installed apps on the phone ideviceinstaller -l

ios-deploy

View the currently linked device ios-deploy -c

Install APP ios-deploy --[xxx.app]

Uninstall application ios-deploy --id [udid] --uninstall_only --bundle_id [bundleId]

View all apps ios-deploy --id [udid] --list_bundle_id

Check whether the app is installed ios-deploy --id [udid] --exists --bundle_id

Using the above command line tool chain, you can connect UI automation test scripts customized according to different projects to the CI process, such as connecting to GitLab Pipelines to intervene in processes such as code review and merge request.

Architecture design based on Web Service

Architecture design of App automated testing platform

From the previous article, we learned that we can use Xcode to create UITest Target, write UITest Case test scripts, and use xcodebuild and other related command tool chains to write automation scripts. Then, we can access the CI/CD process and implement UI automation testing for iOS App, thereby freeing up human resources and reducing the cost of manual testing.

But at the same time, a new problem arises. That is, with frequent business iterations, the test case scripts we write are easily abandoned due to business changes. The test script reuse rate is low, which increases development costs. If apps on different system platforms, such as Android, iOS, and even Web App, can share a set of test scripts, the script reuse rate will be increased, which will reduce development costs and be more conducive to business regression.

In addition, for regression testing of complex businesses, if you want to improve the efficiency of business regression for a large number of test cases, you must improve concurrency and reduce testing time. Therefore, under such a demand, the Facebook team designed an automated testing tool based on Web Service, such as Appium. Another testing tool similar to Appium is Macaca designed by the Alibaba team. The design architecture of this type of testing tool is shown in the figure below.

The architecture of automated testing based on Web Service can be mainly divided into command distribution service Web Service module and UI test driver module.

For the command distribution service module, its task is to build a communication bridge between the general test case script and the underlying driver, and the HTTP RESTful API can meet such cross-platform requirements. Therefore, the Web Service module needs to build an HTTP Web Service for command forwarding, and use the Test Case script in the automated test as the client of the Web Service to send a request to the server of the Web Service. After receiving the request, the server of the Web Service forwards the request to the driver process of the underlying UI test to drive the UI test later.

For the UI test driver module, its main task is to receive the request forwarded by the Web Service Server, trigger the driver process to perform UI automated testing, finally collect the test results, and generate a test report. The underlying driver of the Android operating system is generally the UIAutomator program; while for the iOS system, Appium uses WebDriverAgent and Macaca uses XCTestWD. Both WebDriverAgent and XCTestWD are Xcode projects based on XCUITest, and their technical core is the iOS UI automated testing technology based on XCUITest and Accessibility that we introduced earlier.

The App automated testing platform needs to run the Web Service Server first. As the issuer of test instructions, the Server sends requests to the test driver, thereby driving the Test App process to operate the App. Therefore, you need to start and run the Web Service Server on the Jenkins Slave machine first. For example, create a Web Service on the local port 4722, listen to the requests sent by the Client to the port, and then forward them to the driver layer.

After the driver project (WebDriverAgent or XCTestWD) is compiled successfully, a Runner program will be created and run on the running device. This program is compiled into a Test App using XCUITest. However, unlike the previous Demo, this program will also create a Web Service on the device to receive requests from the Server, process the requests according to the program in the Test App, and finally return the response result to the Server.

For example, when creating a test session, after the WebDriverAgent is compiled successfully, a Web Service will be created on port 8080 of the test device, so that the Web Service Server running on the Jenkins Slave can forward the client's request to the Web Service created by the WebDriverAgent, and then map it through the internal route /wd/hub/session of the WebDriverAgent, find the specific code corresponding to creating the session, save the Session ID value, and return the Session ID as a response result to the Jenkins Web Server. Other test operations such as finding an element, finding an element value, and scrolling an element are similar to the communication process between the Jenkins Web Service C/S and the underlying driver, which is similar to the process of establishing a session.

Therefore, with the UI automation testing tool based on Web Service, we can perform automated testing more efficiently, with higher reusability, support multi-platform and cross-platform testing, and even use its Web Service to build a distributed testing platform. The architecture design based on Jenkins service is shown in the figure below.

According to the architecture design in the figure above, we can use multiple machines to build a Jenkins cluster. According to the requirements of our CI/CD process, we can send requests to the Jenkins Server, and then the Jenkins Server will assign different Jenkins Slaves to execute the Job. Each Jenkins Slave is configured with the UI automated testing platform to drive multiple devices for automated testing. This will realize a distributed automated testing platform, improve concurrency, increase testing efficiency, and reduce the time of regression testing.

Next, we will introduce the simple usage of Appium and Macaca respectively.

Appium Toolchain Matrix

WebDriverAgent

WDA is an iOS UI automatic test driver developed by Facebook based on the XCUITest test framework, similar to Macaca Runner.

Install WDA from source

 git clone https://github.com/appium/WebDriverAgent.git

Modify Team ID for real machine testing

Select the Team of the Apple development account.

Appium Web Service Server

Appium Server is used for HTTP command forwarding and drives the underlying Driver WDA.

Start the Server using Appium Desktop

Download link: https://github.com/appium/appium-desktop/releases/download/v1.21.0/Appium-mac-1.21.0.dmg

Install the Appium App and start the Server using the GUI App.

Start the Server using Appium Command

Install Nodejs dependencies

Execute command line

 npm install -g appium

Start the Server

Execute command line

 appium -a 127.0.0.1 -p 4722

Parameter list: http://appium.io/docs/en/writing-running-appium/server-args/index.html

Port Mapping

Execute command line

 iproxy [LOCAL_TCP_PORT] [DEVICE_TCP_PORT]

http://manpages.ubuntu.com/manpages//trusty/man1/iproxy.1.html

https://github.com/libimobiledevice/libusbmuxd/blob/master/tools/iproxy.c

Port mapping relationship

Execute command line

 appium -a 127.0.0.1 -p 4722 --webdriveragent-port 8123  

If the WDA port is set to 8123 when starting the appium server, the first parameter of the iproxy command must be the local listening port, which can be randomly selected, and the second parameter must correspond to the WDA port specified by the appium command. It can be executed as follows:

 iproxy 8100 8123

Driver Runner storage location

After installing the appium server globally locally, WebDriverAgent.xcodeproj is stored in the following path.

 /usr/ local /lib/node_modules/appium/node_modules/appium-webdriveragent/WebDriverAgent.xcodeproj

Web Service Client——Test Case

Test items

https://github.com/appium/appium/tree/master/sample-code

javascript - webdriverio

Install Dependencies

Execute command line

 cd appium-master/sample-code/javascript-webdriverio
 npm install

Modify the configuration

Modify the capabilities configuration in the test script.

 const iosCaps = {
 platformName: 'iOS' ,
  automationName: 'XCUITest' ,
  deviceName: process.env.IOS_DEVICE_NAME || 'iPhone' ,
 udid: 'iphone udid' ,
  platformVersion: process.env.IOS_PLATFORM_VERSION || '13.6.1' ,
 noReset: true ,
 bundleId: 'your app id' ,
  app: undefined // Will be added in tests
 };

Capabilities documentation https://appium.io/docs/en/writing-running-appium/caps/

https://appium.io/docs/en/writing-running-appium/default-capabilities-arg/

Run Case

Execute command line

 npm test

Macaca Toolchain Matrix

Similar to Appium, Macaca's toolchain matrix also includes Driver, Web Service Server, and Web Service Client.

Install the Macaca toolchain

 # Local Installation
 $ npm i macaca-ios --save-dev  
 # Global installation
 $ npm i macaca-ios -g
 # Install macaca-ios with TEAM_ID
 $ DEVELOPMENT_TEAM_ID=TEAM_ID npm i macaca-ios -g

For more detailed installation process, please refer to the official document https://macacajs.github.io/zh/guide/environment-setup.html#%E5%AE%89%E8%A3%85-node-js

Comparison between Appium and Macaca

Framework names are similar but different

Our UI automation testing platform was initially connected to the Macaca framework only, and an independent repository was maintained for internal platform use. Various problems were encountered during the maintenance process and solved by ourselves. After verification, feedback was given to the official and corresponding solutions were provided. At present, we have also begun to gradually connect to the Appium framework to carry out technical transformation of the existing platform to adapt to more scenarios and ensure the long-term stable and sustainable maintenance of the framework.

Current state of automated testing on Trip.com App

In the daily development and iteration process of Trip.com App, there are many application scenarios for UI automation testing, such as smoke testing, exploratory testing, and UI automation testing platform based on Web Service. Next, we will introduce the roles and functions of different tests in CI/CD.

Application Scenario

Smoke Test

Basic Information

• In programming and software testing, smoke testing (also confidence testing, sanity testing, [1] build verification testing (BVT) [2] [3], and build acceptance testing) refers to preliminary testing that reveals simple but high-level errors that are sufficient to affect the release of a software version. - Wikipedia

In the actual application scenario of Trip.com, the role of smoke testing is mainly to detect Merge Request card points, and its main function is to pre-check the integrated compilation and runtime crash of Trip.com App. For example, in the case of parallel development of multiple modules, some changes made by different teams will cause the problem that the symbol name cannot be found, and smoke testing can pre-check this point, avoid integration and packaging failure, and reduce trial and error costs and time costs.

The specific practice of smoke testing for Trip.com iOS is to create a UITest Target in the main project, write a simple UI view verification program, connect to the GitLab Runner Pipeline, and use the xcodebuild tool chain to perform preliminary verification of the compilation process and runtime robustness to ensure that the code merged into the main branch will not cause obvious major crashes in the App.

Data

During the rapid iterative development process of Trip.com, smoke testing was used as a checkpoint task for Merge Request. Multiple GitLab Runners were used to concurrently execute six smoke test tasks, which greatly reduced the checkpoint verification time. The time for a single smoke test was controlled within 6 minutes. This not only achieved the purpose of verifying the compilation, construction and robustness of the integration package, but also greatly saved the time cost of test verification.

Exploratory Testing

Basic Information

Exploratory Testing is a software testing method. Its characteristic is to explore and develop more different types of testing methods while testing in order to improve the testing process. —— Wikipedia

In the actual application scenario of Trip.com App, the main role of exploratory testing is random testing of App pages, which is mainly used to verify the quality of integrated packaged Apps, randomly click on pages, and collect and count Page View and Crash data, and finally compile them into reports and send them to relevant developers.

Trip.com iOS exploratory testing is a white-box/grey-box UI testing framework developed based on Google eDistantObject and EarlGrey open source projects. Different from XCUITest, which writes test cases and must be combined with Accessibility testing, the white-box/grey-box exploratory testing framework uses the communication between the test app and the host app process to enable the test app to drive the host app to perform UI automation testing, while the app's element review, user interaction, and data collection are all completed in the host app process. Without the restrictions of Accessibility, the white-box/grey-box exploratory testing element review is more comprehensive, the stability is higher, and the test data is correspondingly more comprehensive.

Data

Trip.com exploratory testing is a daily Jenkins task used to verify the stability of the App integration package. It collects all the pages that are touched, can effectively detect crash problems in advance, and send a report of the test results to the R&D team by email. When the number of concurrent tests for iOS is 5, the number of non-repetitive pages that can be effectively touched in 2 hours of testing can reach 180, involving scenarios such as homepage feed flow, travel photography, and order pages. The crash problems collected by the exploratory test will be collected and organized into tables for the crash call stacks, which will be assigned to relevant R&D colleagues to promote the production line to modify the relevant problem code.

UI Automation Testing Platform

Basic Information

The Trip.com App UI automated testing platform is a visual, unified data management quality assurance platform designed and built by the IBU public testing team and IBU public wireless. In order to improve the testing efficiency during the rapid iteration of the App development process, multiple machines are used to build a Jenkins cluster to implement concurrent execution of cases, perform App regression testing, reduce the cost of manpower testing, and report test problems to relevant developers to encourage them to improve functions, thereby ensuring the quality of the App before it goes online. The test framework used is mainly Macaca, and it will be gradually migrated to Appium.

Data

The UI automation platform is currently in the first stage of development. In daily regression testing, for testing of complex business scenarios, when the machine performance is stable and the number of concurrent tests is 6, the total test time can be controlled within 40 minutes, the total number of test cases can reach 209, the total number of steps is 3077, the feature pass rate is 97.14%, and the case pass rate is 98.56%.

The above different automated testing application practices, connected to different CI/CD processes, provide quality assurance for the rapid development and iteration process of Trip.com App.

Summarize

For UI automated testing technology on the iOS platform, the two core technologies officially provided by Apple are XCUITest and Accessibility. In order to enhance reusability and facilitate distributed automated testing, different manufacturers have designed and implemented automated testing platforms based on Web Services. The advantages are easy deployment and cross-platform features. They can make greater use of distributed systems to enhance concurrency, improve testing efficiency, and reduce labor testing costs.

Of course, there are many UI automation frameworks on the market, such as STF and Airtest. The underlying drivers of these frameworks use graphic image recognition to locate App elements. As for the current practice of automated testing applications of Trip.com iOS, it is more based on the XCUITest framework, so this article will not discuss such testing frameworks for the time being. However, no matter what kind of driver is used for automated testing of the App, the overall architecture design will be designed based on the Web Service introduced in this article to achieve the goals of cross-platform, easy integration, and high reuse.

This article is reproduced from the WeChat public account "Swift Community"