Awesome! A Practical Guide to Mobile App Testing

[[148566]]

Testers are often seen as bug hunters, but have you ever wondered how they actually perform testing? Are you curious about what they actually do and how they add value in a typical technology project?

The author will take you through the thought process of testers and explore the various considerations they make when testing mobile apps. The purpose of this article is to reveal the thought process of testers and show the breadth and depth of what they usually consider.

Questions testers need to ask

The core competency of a tester is to ask challenging and relevant questions. If you can combine investigation and questioning skills with knowledge of technology and products, you will gradually become a good tester.

For example, a tester might ask:

• What platform should this app be used on?
• What exactly does this app do?
• What would happen if I did this?

And so on.

Testers can discover issues in a variety of contexts, whether it’s from conversations, design, documentation, user feedback, or the product itself. The possibilities are endless… so let’s dive in!

Where to start testing

Ideally, testers would have all the latest details on the product they are testing. But in reality, this is rarely the case, so testers, like everyone else, have to make do with the limited information they have. But this is no excuse not to test! Testers can gather information from many different sources, both internal and external.

At this stage, testers can ask these questions:

• What information is there: specifications? Project meetings? User documentation? Knowledgeable team members? Are there support forums or company online forums to help? Are there records of existing bugs?
• On what systems, platforms, and devices does the app run and is tested?
• What type of data does the app handle (e.g. personal information, credit card information, etc.)?
• Does the app integrate with external applications (such as APIs and data sources)?
• Does the app require a specific mobile web page?
• How do existing consumers rate this product?
• How much time is available for testing?
• What are the testing priorities and risks?
• Which users are unhappy with it and why?
• How to publish and update?

Based on the information collected above, the tester can make a test plan. Usually the budget determines the test method. The method of testing in one day, one week or one month will definitely be different. As you gradually become familiar with the team, the workflow and how to solve such problems, it will be easier for you to predict the results.

Case Study: Social Commentary on Facebook App

When I'm collecting information as a tester, I like to use Facebook App as an example because user complaints are everywhere. Below are just some of the reviews posted by users who encountered problems in the iTunes App Store, but there are many more on the Internet.

Facebook App for iPhone has a lot of negative reviews

If I were challenged to test the Facebook app, I would be a fool to take this feedback into account.

Creativity of testers

You may know what the app is supposed to do, but what does it actually do? How do users actually use it? Testers are good at thinking from the outside, trying different things, and constantly asking "what if" and "why" questions.

For example, mobile testers often test with different user roles - this is a bit exaggerated, of course, but the ability to think, analyze and imagine yourself as different users is very inspiring for testing.

Testers might imagine themselves as the following users:

• No experience;
• Very experienced;
• Enthusiasts;
• hacker;
• Competitors;

Of course, there are more roles to choose from, depending on the product you are developing. In addition to the characteristics of the role, its operation behavior and workflow are also important. People often use products in strange ways, such as:

• Returned when he should not have;
• Being impatient and hammering keys multiple times;
• Entering incorrect data;
• Not understanding what to do;
• It may not be set up as required;
• May be complacent and think they know what to do (e.g., often not reading instructions).

When testers encounter these problems, they often discover unexpected bugs. Sometimes, these bugs are trivial, but deeper investigation will reveal more serious problems.

Many questions can be predetermined and tested. When testing mobile apps, the following questions are not necessarily relevant, but you can try asking them:

• Did you do as you were told?
• Did it accomplish the task as designed?
• Isn't it designed to complete the task?
• What happens if it is under constant use or load? Is it unresponsive? Does it crash? Does it update? Is there feedback?
• Will crash reports be fed back to the app?
• What creative, logical, or passive ways might users navigate your site? Do users trust your brand?
• How secure is the user's data?
• Is it possible to be interrupted or cracked?
• What happens when you push it to its limits?
• Will it ask you to turn on related services (such as GPS, Wi-Fi)? What happens if the user turns them on? What happens if they don’t?
• Where do you redirect the user? To the web? Or from the web to the app? Will this cause problems?
• Are the communication process and market feedback consistent with the App’s functions, design, and content?
• What is the login process? Can I log in directly on the App or do I need to go to the web page?
• Does the login integrate with other services, such as logging in with Facebook and Twitter accounts?

Example: RunKeeper'sgyUpdate

RunKeeper is an app that tracks your fitness activities. The latest version has a "goal setting" feature. I was very interested in trying it out, partly from a tester's perspective, but more as a user who really likes the product. But I found some problems:

1. The default unit is pounds, but I want kilograms as the weight unit;

2. Switching between pounds and kilograms doesn’t work at all;

3. When setting goals, it leads to wrong data and charts, which confuses me;

4. Because of item 3, I want to delete the target, but I can't find where to delete it;

5. To solve this problem, I had to change my personal weight value until it was within the "goal setting" range, so that when the goal was reached, I could set a new goal;

6. I will try to add the target again;

Because of the above doubts, I spent more time playing with it to see if I could find other problems;

Here are some screenshots of where the problem was discovered:

The latest version of the app includes a new "Goals" section. When setting the date, I noticed that both the start and end dates can start from 1 AD, and why are there two 1-year options?

Another bug is a typo in the “Current Weight” section. When clearing the data, a typo appears on the “Enter” (the app uses Etner). This is just a minor bug, but it looks very unprofessional.

There is no shortcut to finding problems, you can only try it over and over again. Every app and its team will face many different challenges. However, the typical characteristics of testers are: going beyond the limit, doing something unconventional, changing things around, and keeping testing for a long time (testing for days, weeks or even months, not just a few minutes), even if they know that these things are impossible. These are exactly the scenarios that can be found and elicited.

#p#

Where is all the data?

Testers like to find problems in data, which sometimes makes developers very depressed. In fact, users or software developers are too easily confused in the information flow because many errors may occur, so data and cloud-based services are more important.

You may try to check the problem in the following scenarios:

• The mobile device data is full;
• The tester removed all the data;
• What happens to the data if the tester deletes the App?
• What happens to the data if the tester deletes and reinstalls the App?
• Too much or too little content leads to changes in design and layout;
• Use in different time periods and time zones;
• Data is not synchronized;
• Synchronization is interrupted;
• Data updates affect other services (such as web pages and cloud services);
• Process data quickly or process large amounts of data;
• Using invalid data;

Example: Soup.me’s Error

I tried Soup.me, a web service that can sort personal Instagram photos by map and color, but I didn't use it for long. When I signed up, it prompted me that I didn't have enough photos on Instagram, but I clearly had more than 500 photos in my account. I don't know where the problem is, maybe it's a data problem, maybe it's a presentation problem, or maybe it's an error prompt of the app.

Another example: Quicklytics

Quickytics is a web analytics app for iPad. While using it, even though I have deleted the website configuration from Google Analytics, it still exists. Here are some questions:

I have deleted the website configuration, why is this information still there?

The module on the left does not explain why “the operation cannot be completed”, so can it be improved to avoid confusing users?

Testers also like to test under extreme data. They often use typical users to understand the app, so extreme testing does not take a long time. Data is messy, so testers need to consider the user types of the software and how to test under different data scenarios.

For example, they might try the following scenario:

• Test the limits of user input;
• Test with duplicate data;
• Tested on a brand new phone with no data;
• Tested on an old phone;
• Pre-install different types of data;
• Consider pooling everyone’s resources to conduct testing;
• Automate some tests;
• Test it with some unexpected data to see how it handles;
• Analyze how information and data affect user experience;
• Regardless of whether what the user sees is correct, keep asking questions.

Creating error alerts and messages

Here, I am not going to talk about the design of good error messages from the perspective of a designer, but from the perspective of a user or tester. Error prompts and messages are where testers can easily find problems.

Questions to ask about error messages:

• Consider the following questions:
• Is the UI design of error reminder acceptable?
• Is the error message understandable?
• Are error messages consistent?
• Are these error messages helpful?
• Is the error message appropriate?
• Do these errors conform to conventions and standards?
• Are these error messages inherently safe?
• Are runtime logs and crash reports available to users and developers?
• Have all errors been tested?
• What state will the user be in after handling the error message?
• Are there no error messages when the user should accept them?

Error messages affect the user experience. However, bad or useless error messages are everywhere. Although it is ideal to avoid users encountering error messages, this is almost impossible. The design, implementation and confirmation of error conditions may be contrary to expectations, but testers are often good at discovering unexpected bugs and can carefully consider whether to improve them.

Example of error message:

I really like the example of the Facebook app on the iPhone. Not only does this long and cryptic text try to cover many different scenarios, but it can also get lost in the middle of nowhere.

Maybe the following message box should be included in the counterexample "Hall of Fame"?

Looking at this The Guardian app on iPad, what should I do if I don’t want to “retry”?

#p#

Platform-specific notes

It is critical for any project team member to understand the business, technical, and design constraints of the relevant platforms.

So, what platform-related issues should mobile app testers find out?

• Does it follow the design specifications for this specific platform?
• How does it compare to competitors and designs within the industry?
• Is it suitable for peripheral devices?
• Does the touch screen support gestures such as: tap, double-click, long press, drag, shake, pinch, flick, swipe?
• Is this app understandable?
• What changes when you rotate the device?
• Can I use maps and GPS?
• Is there a user guide?
• Is the email workflow friendly?
• Does it work smoothly when shared over the web? Does it integrate with other social apps or sites?
• Does it work well when users are multitasking and switching between different apps?
• Does it show the progress of time as the user updates it?
• What are the default settings? Have they been adjusted?
• Does using sound effects make a difference?

Example: ChimpStats

ChimpStats is an app for viewing email campaign details on iPad. The first time I used the app, I was in landscape mode. When I needed to enter the API password, I was stuck. I couldn't enter the API password in landscape mode until I switched to portrait mode.

Connection and Disconnection Issues When a connection is intermittent or unexpectedly disconnected, a lot of interesting things can happen.

Have you tried using the App in the following scenarios:

• In a moving environment?
• Wi-Fi connected?
• No Wi-Fi?
• In 3G mode?
• Connecting intermittently?
• Set to airplane mode?
• When a phone call comes in?
• When you receive a message?
• When you receive a reminder notification?
• When the battery is very low or even shuts down automatically?
• When forced to update?
• When you receive a voice message?

This type of testing is the easiest way to find errors and bugs. I strongly recommend that you test in these situations (not just powering it on and making sure it works, but trying the entire user flow and forcing connections and disconnections at specific intervals).

• Does the app provide enough feedback?
• Is the data transfer known to the user?
• Will it grind to a halt and then collapse?
• What happens when it is turned on?
• What happens during mission completion?
• Is it possible to lose unsaved operations?
• Can you ignore a notification? What happens if you ignore it?
• Can you respond to notification alerts? What happens if you respond?
• Is it appropriate to use error messages for certain problems?
• What happens when a login expires or times out?

App Maintenance

It would be easy to speed up the entire testing process and just test it once and forget it, right? Think again.

One problem I have encountered at the moment is that some apps on iPad can no longer be downloaded after being updated. For a user, this is very frustrating.

Maybe, this is something the developer can't control. Who knows? I just know that it doesn't work for the user. I also tried uninstalling the app and reinstalling it, but that didn't solve the problem. I searched a lot online, but found no other solution except some suggestions about updating the operating system. Maybe, I'll try it again next time when I have time.

The key point is this: if an app is tested once and only once (or only for a short period of time), many problems will go undetected. An app itself may not notice the changes, but external conditions can make these problems happen.

When the external environment continues to change, how will the App be affected? Let’s ask ourselves:

• Can I download this app?
• Can I download and install updates?
• Can it still be used after the update?
• Can I update it when many apps are waiting for update?
• What happens to it after the system is updated?
• What happens if the system is not updated?
• Will it automatically sync to other devices via iTunes?
• Does it make sense to automate tasks or tests?
• Will it connect to a web service? What difference will that make?

It is best to test each version of the mobile app after it is released. Each time a new version is released, define the highest priority test first to ensure that it can be performed under various conditions (mainly on mainstream platforms). Over time, testing can become automated. But remember that automation is not a panacea, and problems can only be discovered through human eyes.

Case: Analytics app on iPhone

I've been using this app for 2 years with no issues, but now it's showing zero data for some of my sites (but more than one person visits my site in a month!). Judging by the reviews on the App Store, I'm not the only one with this problem.

Another example is Twitter on iPhone. After updating and launching the app, I instantly saw the following message: "Your timeline data is empty and you haven't followed anyone yet" (but I am an active user with 5 years of experience). I was worried for a while, but fortunately, the message disappeared quickly and then loaded the historical data.

Testing is not a right or wrong judgment

We discussed some aspects of mobile testing, but the premise is: you have questions in order to discover them.

Often, testing is considered to be completely logical, plannable, and predictable, with test scripts and test plans, pass and fail, correct and incorrect feedback. Going through these testing processes is not far from the truth.

Of course, if necessary, we can use the above method to test, but this is not the purpose of testing. We are not only to create test cases and find bugs, but more importantly to find key issues and provide valuable information for the project team to decide when to release the App. And the best way to find those key issues is to ask questions!