5 minutes to show you how to play with App automation testing

Preface

App automated testing has always been a field that small and micro teams rarely get involved in. In the context of the rapid iteration of the Internet, as business develops, regression pressure gradually increases. I believe that every time an online accident occurs due to insufficient regression coverage, I am definitely not the only one who is so frustrated that I want to smash the table.

Generally speaking, the testing of small and micro teams, including regression testing, is done manually. Some critical businesses that deviate from the main process are often easily missed by manual regression testing. Manpower is limited, and at this time, the idea of ​​automated testing pops up in your head, and then you just go and study it. But it may end up stopping at research. Not to mention the construction of the automation framework, the various subdivided boundary cases, a thing that is bound to be cumbersome, just think about it and it will be even more cumbersome. If you are an App developer, you don’t have enough manpower for business development, and you are powerless to develop automated scripts! If you are a tester, the business requirements test of each iteration will fill your schedule, not to mention that you are not just responsible for App testing tasks. As a small and micro team, we also want automation, but we don’t have the resources...

First of all, we must understand that App automated testing is not as powerful as you expected, but if you are facing regression testing pain points like me, it can definitely meet your needs. It is not that powerful, so it is not as complicated as you expected. At the same time, its participants should not be limited to testers or developers, so you can have more flexible allocation of resources. Of course, automated testing is a long-term process, and its future is not just regression...

Finally, I will recommend you a less popular automated testing tool: Calabash. I will also provide you with a Calabash introductory tutorial blog and some of my experience using it. I introduce Calabash because I personally think that the features of Calabash are more suitable for small and micro teams with limited resources.

Dahua App Automated Testing

This is just my personal opinion. My knowledge is limited, so please feel free to criticize me.

status quo

Let me start with something that everyone knows. Take Android as an example. Since 2010, the Android development environment has developed rapidly and is now almost mature. Developers are no longer limited to this single development platform and are beginning to seek the unification of Android and iOS development: ReactNactive, WEEX, H5...

The development environment is mature, but the testing environment for mobile clients is lagging behind. Many teams are facing the following challenges:

1. Unlike server-side testing, App testing usually only requires verification of the data itself. App involves interface display and interaction, and automated identification is difficult. 2. Internet companies have been pursuing rapid iterations, and Apps are directly connected to users. Changes to the interface and logic of Apps are commonplace. The stability of writing automated test scripts is very poor. A change to the interface may invalidate all scripts previously linked to this page. 3. Android and iOS dual platforms, web pages. It is almost impossible to rely on a set of code for automated testing in the case of multiple platforms. Considering the need for frequent changes, you know. 4. Compared with manual testing, automated testing may require writing a lot of scripts for various boundary cases. Compared with manual testing, all that may be needed is to design a few scenarios in advance, and other abnormal boundary cases can be observed manually during testing. The cost of automated testing has doubled!

Therefore, a complete mobile automated testing environment often requires the support of a large testing team, but only large companies can afford such a large testing team. Mobile automated testing has always been ignored by small and medium-sized startups. Small companies develop and test by themselves, while medium-sized companies rely on manual verification by testers.

Is automated testing really a closed door for small and micro teams?

Just do it

The current situations mentioned above can only be described as difficult problems. Perhaps it is too early to discuss and solve these problems now. I think you have not yet determined some things clearly. Why don't you follow my ideas first? When you have figured out some of the following things, perhaps these problems can be avoided. If you need to force them, you will be mentally prepared enough.

1. Set up a trial and error plan

In fact, the biggest problem that makes small and micro teams hesitate in the face of automated testing is: input-output ratio! It is difficult to predict whether the testers or developers will get stuck in the quagmire between frequent page changes and script development and maintenance after the implementation of automated testing.

But talking on paper will never produce results, and other large companies are not very useful for reference, because it involves the team, resource allocation, business change frequency, testing tools, the degree of decoupling of script development, etc. However, automated testing is an inevitable trend, so action is the most important thing!

Whether your team is currently suitable or not can only be known by trying. You may want to set up several stages first, and then use the first stage for trial and error:

Phase 1: Assign one person per iteration to complete the automated test case of the main process business, and run two to three iterations, and add exception cases of the main process during this period. Use these three iterations to evaluate the feasibility of subsequent development! Phase 2-(Phase 1 successfully passed): If you think the status of Phase 1 is not bad, then on the basis of maintaining the results of Phase 1, select a new business from the remaining business scenarios according to the business criticality and change frequency, and write an automated test case at the pace of one person per iteration for one business!

Phase 2-(Phase 1 Excessive Failure): Of course, after three iterations of evaluation, as the number of abnormal cases increases and the difficulty of synchronous maintenance increases, you may think that the cost of implementing automated testing is too high, but don't give up the results of these three iterations easily. Please use programming thinking to check all test scripts to see if there is a possibility of extracting similar codes, encapsulating specific View operations, and extracting instructions that are not related to the business. At the same time, consider using all group resources to maintain this set of main process automation cases. Until there are sufficient resources in the future or a better alternative is found, proceed to Phase 3 or restart Phase 1.

Phase 3: After about 5 to 8 iterations, you have successfully survived Phase 2, which means that your automated testing environment is on the right track. At this time, you can appropriately speed up the coverage of automated cases according to the team's resources!

Using one person per iteration for trial and error is a loss you can accept. Of course, don't rush into it. You may have just decided to do automated testing, but you still need to make some detailed plans and some preparations in thought and action!

2. Set clear and simple goals

With a big plan, you also need to clarify the specific needs. The optional needs are probably as follows:

1. Black box test or white box test Although it is UI automation test, it can also be divided into black box or white box, which depends on the precision of the test you want. That is, at this time, you can preliminarily determine what automated testing tools to use. For example, we usually choose Appium, a cross-platform testing tool, in order to be able to use dual platforms, but if you have high testing requirements, taking Android as an example, it is recommended that you use the test framework officially recommended by Android: Espresso, and write the test directly in the Android project. It seems that everyone understands this, but if you can't clarify your needs in the early stage, it may cause great trouble in the later stage! Choose Appium, because it is a black box, encounter some special scenarios or requirements, resulting in individual cases cannot be tested. Or choose Espresso, but later find that there is not such a precise testing requirement, which leads to the inability to cross platforms in the later stage or think that the cost of writing Espresso is too high, and it is difficult to hand over to the testing team.

2. Front-end UI or overall business Take, for example, the scenario of grabbing an order and delivering it. Find an order and click to grab the order, then pick up the food, and finally complete the delivery. This set of UI layer operations that can only be observed with the naked eye is tentatively defined as the front-end UI logic. However, behind this series of operations, the flow of order status causes other data changes, such as: wallet data changes, activity reward data changes, fraudulent order determination, etc. This range belongs to the overall business scope. What we need to determine now is whether the test expectation you want to achieve is UI testing or overall business testing. This directly determines the complexity of your test script. My suggestion is to only test the UI logic, and I also want you to lower your expectations.

First, set a clear and simple goal, and then dive in. If you think too much, it will become more difficult, and in the end, it may just be a thought. This is the next point: lower your expectations!

3. Lower your expectations

In a manual (clicking on the phone) testing environment, we usually verify various cases by operating the App. As long as the operation of the App passes the verification, it can be basically determined that there are no problems with the front-end and back-end. But this premise is that manual verification is the human brain plus the naked eye, which will have more accurate subjective judgments. Involving the front-end and back-end interaction will greatly increase the complexity. Your test cases will not be infinitely numerous and infinitely detailed, but as a human brain, you can find more unreasonable bugs in the limited case execution: unreasonable text wrapping, inaccurate numerical display, wrong button color, incorrect number of Toast displays, etc. Automated testing will rigidly follow the script you wrote. Can you guarantee that your case covers all?

This is also a question that many people will raise when talking about automated testing: how to write so many exception cases? It’s tiring to think about it. Considering the input-output ratio, it’s better to do manual testing.

First of all, we must make it clear that the front-end and back-end automated tests must be separated. One set cannot solve the problem, so many cases can be ignored in the front-end test. In addition, automated testing is a long-term process like project growth. It will take some time for automation to completely replace manual work. Don't think about it all at once. There will still be a lot of cases to write, right? If there are not enough resources, we can run the main process first. Running the main process means that the script dependency (environment construction, view positioning) is relatively mature. Other abnormal case scripts can be modified according to the main process script. We can take our time, and at this time you can let other developers or testers do it. This is what we said above, stage 1.

If you can't figure out these problems, you will find that you can't achieve the results you want under limited conditions of App automated testing, and you will give up under pressure. Convince yourself to lower your expectations!

• Lower your expectations and aim for regression first
• Lower your expectations. Both front-end and back-end automated testing should be done. One set of tests cannot solve the problem.
• Lower your expectations. Client-side automated testing still has many limitations. Manual testing and verification cannot be abandoned completely.
• Lower your expectations. You can’t achieve everything in one go. Automated testing also needs to be improved through iteration. Run it through first and then focus on verifying abnormal cases. Iterate quickly and improve gradually.

4. Specialization

Maybe one of the developers or testers in your team can't bear the pressure, so they start to explore App automated testing. The advantage of developers doing automated testing is that they have more development experience and are more proficient in setting up the test environment. Because they write their own code, they will understand the risk points better and can write targeted cases. The advantage of testers doing automated testing is that they can discover more boundary scenarios and write more comprehensive test cases.

In automated testing, development and testing each have different advantages, but at the same time these are also each other's disadvantages.

The test case must be as comprehensive as possible, and automated testing itself is an engineering project. When writing scripts, if you consider more programming ideas, reasonable coupling and decoupling, it will make the subsequent script writing more convenient. Therefore, it is not recommended to leave automated testing entirely to testers or developers. A close cooperation relationship is best. It is also a solution to the problem of insufficient resources. Although it is a test, the participation of developers will make the test project develop in a better direction.

5. It takes two to tango

First of all, I would like to clarify my point again: in the absence of sufficient resources, the mobile client automation testing should mainly focus on regression testing.

Compared with manual testing, automated testing is very rigid, so it requires very "rigid" data support. If it is an App developer (like me) who does automated testing, do I have to mock all the data? Although the front-end and back-end automation should be separated, as mentioned earlier, we should only care about the front-end logic. But it must be tested in a real service environment (test environment, non-production environment). Mock data is too rigid, and many cases may not be verified. For example, in the registration scenario, it is difficult to verify the abnormal prompt when the registered account cannot be registered again. Mock data is obviously difficult to take into account both normal registration and abnormal registration. And testing in a real environment, wouldn't it be great if a back-end bug is detected? Moreover, it seems that there is no big problem for a completely 2C application like Taobao Mobile to be tested directly in the production environment, but it is more embarrassing for a semi-2C application like Fengniao Delivery. It is obviously impossible to grab an online order directly, so who will send this order in the test environment and how to send it?

So at this time you can't work alone, you have to find a tester to see if you can build a test environment that cooperates with automated testing. Take grabbing orders as an example. I hope there will always be many orders for me to grab in this environment. I am an app for grabbing orders. What else can I do without orders?

If the testers can't handle it, then go and ask the backend students for help. I believe that for the sake of better project, they will not reject you.

Automated testing requires not only automated testing tools but also the support of the testing environment.

summary

In fact, I have said so much just to encourage you to lower your expectations and take action. There will naturally be a large number of automated tools to help you realize the automatic identification and multi-platform of apps, and learning to lower your expectations will also reduce the difficulty of script development and maintenance.