Taking K12 products as an example, let’s talk about how to organize user internal testing activities

Before a product function is launched, the final step is usually for testers to write product test cases, which are then deployed online after passing the test.

However, due to the limited manpower of testers, it is difficult to fully simulate the user's usage scenarios, device types and network conditions, and omissions are inevitable. Therefore, adding a small-scale user internal testing phase before the product goes online will help improve the product quality after launch and reduce the risk of bugs.

So, how to organize a user internal testing activity, what points should be paid attention to? Through this review document, we will sort out these issues from scratch.

01 Review your goals

Looking back at all the user internal testing activities, the goals can be divided into the following three categories:

1) Expose untested bugs

If the product involves a relatively large update, such as launching a new function or changing a large range of program codes, you need to focus on the number and type of bugs that appear during the testing process, and pay attention to whether there are any problems that affect user use, and how often they occur; then determine the priority based on the type and number of problems, and connect with production and research colleagues to fix the problems to ensure the quality of the final launch.

2) Anticipate possible risks after product launch

Through internal testing activities, we need to simulate more real-life usage scenarios to discover hidden problems. We can also obtain user usage data for part of the function by tracking data points in the internal testing installation package. From the perspective of product development, we can predict possible risks after the product is launched, avoid them, prepare response plans, and improve product stability.

3) Collect user feedback

Internal testing activities are a good opportunity for product and user communication. We can take this opportunity to collect users' suggestions or comments on products and functions. Of course, it is not necessarily limited to the key test functions each time. Users' opinions on the overall product can all be collected as a reference for subsequent product iterations.

02 Review process

After determining the goals of an internal testing activity, what points should we pay attention to during the activity process? I will sort them out from the following five aspects.

1) Recruitment

Take for example several activities I have organized before. Because the product type is K12 student users, it is mainly concentrated in elementary school grades.

If we directly recruit primary school students as test users, it will affect the students' learning progress. In addition, if serious product quality problems arise, it will be detrimental to the shaping of the product image. Therefore, we will target the group of mothers who are of primary school age.

So how do you find this group of people?

We have set our sights on some regular part-time job groups and forums, and recruited users into the groups by posting task information to facilitate subsequent task guidance. In this link, we also found that people who frequently participate in part-time activities will have their own part-time job circles. By leveraging the power of these users, we can make our recruitment progress produce a fission effect.

When recruiting and managing personnel, we chose the QQ group method. Because QQ groups provide relatively complete group management and group file support, we can reduce our manpower in the process.

If the product is not K12, you can also recruit users who are interested in participating in the internal test through pop-up reminders in the official version of the App, guide these users to actively download the test version of the App, and cooperate with feedback activities to guide everyone to fully test and expose problems. However, this method will have certain restrictions on product types and user attributes, and requires the support of product development colleagues. You can choose according to the actual situation.

2) Activity costs

In order to encourage internal test users to participate more fully in activities, we can set a certain number of prizes, bonuses and other incentives. If it is a product with strong social attributes, we can also distribute virtual items such as in-site medals and titles.

The premise for choosing an incentive method is to ensure that most users will like it, so as to stimulate the motivation for user feedback; if the final activity reward is not significant to the user, the effect of the activity may be greatly reduced.

Since most of the users we recruited for the event came from external part-time forums, we adopted the most direct method - bonuses, as an incentive; the criteria for bonus distribution are based on the length of time participating in the test and the quantity and quality of problem feedback. Since all users participating in the event have been added to the QQ group, we can save manpower in distributing bonuses through group red envelopes.

Through the number of people participating in each internal testing activity, we can estimate the cost expenditure of the activity and report it to the Leader or the cooperating business department. In addition, because invoices cannot be obtained when distributing group red envelopes, it is also necessary to confirm the compliance of the process with the finance and taxation colleagues in advance.

3) Organize the test

Next, we will start preparing for the test. This step is also a very important step, which will directly determine whether our activity goals are achieved.

First, we need to confirm the delivery time of the test version installation package with product development colleagues. In the previous practical operation process, because an activity group had been established during registration, the installation package could be sent to the group to guide users participating in the activity to download and install it.

Please note that during the installation process, iOS users need to download the official Testflight App to participate in the test. Android users need to pay attention to whether their phone prompts "Jump to the official market to install, this version has not passed verification" during the installation process. Users need to be guided to skip these prompts to ensure that users install the version that participates in the event.

So how do you guide users to test newly launched features? I have adopted two approaches here.

Because it is a K12 live teaching product, for new features that can only be tested through live broadcasts, we can coordinate with internal colleagues to simulate the teacher’s actual teaching situation; after setting the live broadcast time, we promptly notify internal test users to reserve time to participate, and then guide the internal test users’ operations during the live broadcast. It should be noted here that if users are guided to participate in the test through live broadcast, then the live broadcast should describe how to operate, the results of the operation, and how to record the problems to facilitate subsequent problem tracing.

In addition to the non-live class functions, you can make a functional operation instruction document in the form of screenshots, screen recordings, GIF images, or flow charts, and use pictures, texts, and videos to guide users' operations and expose problems.

4) Question collection

Because of the difference between the internal beta version and the official version, in order to avoid the internal beta activity affecting the feedback data of the official version, an independent entry for collecting user feedback needs to be selected during the problem collection phase.

In previous activities, I used questionnaires to collect information. I asked users participating in the activities to fill in their account, the time when they encountered the problem, and the corresponding screenshots or reproduced screen recordings. Combined with the product's tracking data, this has basically met the requirements for locating the problem.

After receiving feedback from internal beta test users, we need to sort and organize the problems described by the users and their emergency procedures. In order to calculate the frequency of the same type of feedback, when sorting user feedback, in addition to matching the user's basic information (account, device type, etc.) with the feedback information (problem description + screenshot), you can also replace the user's described problems with different types of words to facilitate the final statistics of the magnitude of the same problem; for feedback content that you do not understand or cannot be sure of, you need to confirm it with the user again and do not make any assumptions.

5) Follow-up

After sorting out the user feedback issues, you can send the document to the product development staff responsible for the project for follow-up.

In the document, for issues that affect usage reported by users, it is necessary to evaluate whether they are bugs and the subsequent repair plan; for user demand suggestions, it is necessary to evaluate whether subsequent iterations are required and the iteration plan.

03 Evaluation Results

Analysis of the results of internal testing activities helps us evaluate the effectiveness and quality of each activity in the following two dimensions:

• Activity number: target number of recruits, actual number of recruits, number of people who installed the beta version, and number of people who reported issues;
• Feedback quality: total number of feedbacks for the activity, number of valid feedbacks, number of feedbacks confirmed to be bugs, and number of feedbacks confirmed to be demand suggestions.

By comparing the conversion rate of each link in each activity, from the actual number of recruits → the number of people who installed the internal beta version → the number of people who reported problems, we can help optimize the form of the activity; and by comparing the changes in the number of effective feedback and the total number of activity feedback, we can help us optimize the feedback education link for internal beta users and help us obtain higher quality feedback.

04 Analyze the causes

In this section, I will talk about two pitfalls I encountered during activities, hoping to help everyone avoid them.

Initially, in order to stimulate the enthusiasm of registered users to find problems, additional feedback rewards were set, and the reward standard was set to the top three in the number of feedbacks; after the first few activities ended, we observed the data and found that the proportion of valid feedback was only about 40%, and the number of repeated feedback and invalid feedback was relatively large.

Therefore, we later changed the feedback reward rules, using "quality + quantity" as the standard, and publicly feedback with better quality, giving a positive boost to users participating in the event; in subsequent internal testing activities, the proportion of effective feedback can be stabilized at around 70%.

There was also an internal testing activity with more than 500 participants. We still used QQ groups to do 1-to-N user management and internal testing education. However, the increase in the number of participants led to a more difficult internal testing education, and the 1-to-N model was obviously no longer applicable.

When distributing rewards to participating users at the end of the event, Tencent officials will also sensitively remind them of the risk of account suspension due to frequent red envelope transactions. In response to this situation, when we set up the activity group, we must also add the backup management account to the group, and at the same time increase the manpower for internal testing education.

From a long-term perspective, discovering a number of high-quality internal beta test users during previous activities and giving these users the ability to promote and manage internal beta test activities will be of great benefit in freeing up our own manpower.

05 Summary of experience

Due to the particularity of the K12 industry, we chose to distribute internal beta packages through QQ groups and recruit external users to test the new features that are about to be launched.

The key to the internal testing activity is to recruit enough users and get more feedback to support product iteration. Compared with the official version, the internal testing version will experience new features in advance. When recruiting, the words "limited recruitment" and "invitation to experience" can help our recruitment progress.

During the internal testing process, it is necessary to do a good job of function promotion and feedback education for internal testing users, and set up appropriate incentive mechanisms to obtain more high-quality feedback.

At the same time, we need to promptly discover users who actively participate in internal testing activities and provide high-quality feedback, and shape these users into the "image characters" in each of our internal testing activities, giving other users participating in the activities a positive influence.

Author: Bai Shuo Operation

Source: Baishuo Operation