Testing strategy and testing architecture for mobile applications

Today we will talk about the testing strategy and testing architecture of mobile testing.

First, we limit the scope of mobile applications to smart mobile operating systems (such as Android, iOS, WinPhone, etc.), including mobile phone applications, smart device applications, etc.

The popularity of smart phones and smart devices requires a large number of applications to support them. With the increase in the number of applications and the increase in business complexity, mobile applications are increasingly in need of various tests to ensure the correct and stable operation of applications and devices themselves. Therefore, the demand for mobile application testing is also increasing, and a large number of books on mobile application testing have emerged, such as "Android Mobile Performance Practice", "Tencent iOS Testing Practice", "Mobile APP Performance Evaluation and Optimization", "In-depth Understanding of Android Automated Testing", "Mastering Mobile App Testing Practice: Technology, Tools and Cases", etc.

These books all introduce a lot of mobile application testing practices, but no matter how many books you read, how many testing methods, testing techniques, or testing tools and frameworks you learn, you still need to learn and use testing strategies and testing architectures first. If you don’t have a good testing strategy and testing architecture in the beginning, and blindly conduct various tests, it is likely that you will get half the result with twice the effort.

[[216573]]

For mobile applications, first of all, they are essentially software systems, so general software testing methods and technologies can be used. Secondly, they have embedded characteristics, such as the need for cross-compilation and remote debugging during development, and relatively insufficient hardware resources. Therefore, mobile application testing also has its own special features, such as the need for cross-compilation, remote testing, and various hardware-related tests. The corresponding mobile application testing strategy and testing architecture also have their own special features.

Develop a testing strategy

I divide mobile testing into three types: basic testing, advanced testing, and product testing. Basic testing is the basic guarantee for the correct and rapid delivery of products. Extended testing is mainly to enhance the robustness of the software system. Product testing is mainly conducted from the perspective of products and users. The following lists the three common types of testing.

Basic Test

• Function Test[1].
• Integration Test
• Unit Test
• Contract Test[2]

Advanced Test

• Compatibility Test
• UI Visual Test
• Profiling
• Security Test
• Exception Test[3]
• Monkey Test
• Installation, Upgrade and Uninstall Test
• Endurance Test
• Power Consumption Test
• Network Traffic Test
• Other hardware function special tests[4]

Product Testing

• Usability Test
• A/B Test
• Product Online Test (Product Verification Test or Product Online Test)
• Customer Test[5]

For a small or medium-sized project, resources are often very limited, and it is difficult to carry out comprehensive testing. This is even more true for large projects, where it is even more difficult to have enough resources to carry out all types of testing. In addition, some types of testing may not be possible due to insufficient technical capabilities of team members, or limitations of the test-related technology stack, as well as limitations of the development and testing environment and software system architecture.

Therefore, the key point in developing a testing strategy is to specify it based on the priority of quality requirements and with reference to the various constraints of the team.

First, discuss with PO, PM, etc. to get the priority of product quality requirements, and then specify the corresponding type of test according to the priority. Then formulate the corresponding test methods and test techniques according to the team's resources, project cycle, technical capabilities and various limitations, including whether to use automated testing or manual testing, what testing tools and testing frameworks to use, the scope and extent of testing, etc.

The following table is an example of a typical mobile application test strategy table (this is just a sample table for a simulation project. There should be more information in the real project, and new columns can be added according to the specific situation. And please note that these tests are not necessarily done by testers or QA, but should be completed by the whole team working together):

Obtaining the quality requirement priorities in the table is a relatively cumbersome process, which requires discussion and negotiation with various stakeholders.

According to this test priority table, you will know that you should prioritize resources in high-priority tests. After the high-priority tests are completed to a level acceptable to the team, do the next type of test according to the priority. The priorities in this table are not absolutely unchanged during the development process. If the priorities of stakeholders such as PO and PM for product quality requirements change, the test priorities in this table need to be changed after obtaining the team's consent. Therefore, it is necessary to frequently update the test progress with the team and obtain timely feedback and updates from various roles in the team on testing and product quality requirements.

Secondly, you can think about the relationship and workload between different types of tests based on models such as the test pyramid. However, in many cases, you don’t need to refer to these test models because the complexity of mobile applications is generally not very high. In most cases, the complex business logic in mobile applications will be processed on the server side as much as possible. Therefore, mobile applications are often just a user interaction system. Therefore, you should complete the E2E process test that affects user usage as much as possible before continuing with other types of tests.

However, for projects that implement complex business in mobile applications, the testing strategy should still try to consider the problem of duplication of test cases between test types, try to avoid duplicate use cases, and reduce testing costs.

Developing the test architecture

Through the test priority table, we get a simplified test strategy, and then we should develop a test architecture. Due to the particularity of embedded software, its test architecture is also different from conventional desktop systems and server systems. The following figure shows the functional test architecture corresponding to the above sample test strategy:

The figure only provides further detailed architecture design for functional testing, and does not provide detailed architecture design for other tests such as integration testing, compatibility testing, and stability testing. Interested readers can try it themselves according to the actual situation of their own projects.

Through this architecture diagram, you can compare systems and intuitively understand the distribution, relationship and architecture of various types of tests.

Then, with the test priority table, the team can be better guided to conduct effective testing, such as developing better test plans, developing more suitable automated test systems, etc. It can also more effectively evaluate product quality, such as what types of tests are not done, and those specific aspects have higher risks.

However, any software system has defects and risks. The key is to see how much impact these defects have on developers and users, and whether the risks are within the controllable range. Never try to find and eliminate all defects, but consider the size of the risk, the degree of impact, and other aspects comprehensively, increase the team's confidence in product quality, and avoid causing serious and large-scale impacts on customers.

Note:

• [1]. Backend resident application testing also belongs to functional testing.
• [2] For stand-alone applications, there is no need to consider contract testing.
• [3]. Abnormal tests include weak network tests, such as low-speed network signals, intermittent network connections, network switching, no network, sudden power outages, etc.
• [4]. Other hardware function special tests include hardware function shutdown, hardware function abnormality, etc.
• [5]. User testing involves collecting user usage information and generating test cases that users actually use to test the system.