Bluepill: LinkedIn's open source iOS parallel UI testing tool

[[182653]]

Not long ago, LinkedIn introduced the company's first open source project Bluepill in its official blog. Bluepill allows developers to run iOS UI tests with multiple simulators on a single machine at the same time, with the goal of providing stable testing tools and scalable testing.

Its awesomeness lies in:

Start multiple iOS simulators on one machine at the same time, automatically run multiple UI tests and generate test reports for each test.

For a more intuitive look, you can look at the official Demo screenshots:

As you can see in the picture, there are 12 simulators running at the same time, which can cover almost all the mainstream iOS versions.

Why develop Bluepill?

LinkedIn's mobile team has proposed a concept called "3x3 philosophy" in R&D, which means setting a goal of releasing three times a day, with no more than three hours between code submission and availability to team members. This poses a great challenge to continuous delivery.

UI testing, as part of continuous delivery, is very time-consuming. You usually need to test all mainstream system versions before releasing them. Bluepill was born to solve this problem.

In order to achieve the 3x3 goal, Bluepill provides two features:

The stability of the test platform is mainly aimed at the "weird problems" of the Xcode iOS simulator. Sometimes the test fails not because of problems with the code, but because the simulator is not powerful enough, can't run or is stuck. According to LinkedIn statistics, 2% of the tests they run fall into this situation.

Scalability of testing. Scalability fundamentally provides the ability to run tests in parallel to cover multiple simulation environments simultaneously.

Bluepill's Black Magic

Previously, the LinkedIn team tried to develop a parallel testing tool based on Facebook's open source xctool. As a result, xctool stopped development and maintenance. After some research, LinkedIn engineers reluctantly gave up and started their own research.

Bluepill uses CoreSimulator to solve stability and scalability issues. CoreSimulator is an undisclosed framework of Apple that has functions such as creating different types of device simulators, selecting runtime versions, and handling notifications provided by simulators. In the article, the author mentioned that LinkedIn used CoreSimulator to isolate Bluepill from the Xcode simulator and enable Bluepill to run tests using multiple simulators in parallel. There is no need to worry about the Xcode simulator version here, because the Xcode simulator is a black box that evolves with each Xcode update.

How to use Bluepill

Using Bluepill is very simple. In the simplest use case, after you build the project in Xcode, you can run Bluepill using the following command:

./bluepill -a ./Sample.app -s ./SampleAppTestScheme.xcscheme -o ./output/

This command will start four simulators. There are also some options that can be used to customize the test environment, including the number of simulators to use, the runtime environment, the number of retries allowed on failed tests, and the wait time when the test is stuck.

Bluepill is available as open source on GitHub under the BSD-2 license. For developers who do not plan to build from source code themselves, Bluepill also provides binary versions.

Further reading

LinkedIn has put a lot of effort into continuous delivery and UI testing to develop Bluepill. You can read related articles on their official blog:

• https://engineering.linkedin.com/blog/2017/01/open-sourcing-bluepill--run-ios-tests-in-multiple-simulators
• https://engineering.linkedin.com/blog/2015/12/managing-ios-continuous-integration-at-enterprise-scale
• https://engineering.linkedin.com/blog/2016/04/3x3--ios-build-speed-and-stability
• https://engineering.linkedin.com/blog/2015/12/test-stability---how-we-make-ui-tests-stable