Data analysis: 6 A/B tests to successfully cultivate user habits

If you have experienced Duolingo, this language education platform may give you the feeling of playing a game . Design patterns from games are being effectively applied in Duolingo's platform. Interesting game level designs allow more users to stay engaged longer.

Of course this is not a coincidence. Duolingo has long been keenly aware of the power of gamifying education. Gina Gotthilf, vice president of growth, said that any information sent to users - even a small red dot - will provide valuable data. What the team needs to do is test the data, internalize each number, and then guide their next actions based on the test results. Next, we will introduce how they cultivate user behavior habits through A/B testing.

Before that, let me give you some background: Duolingo’s mission is to “provide free education to the world” and its app can be downloaded for free. Currently, it includes 25 languages, and Mandarin will also be included. With more than 200 million users, it is the most downloaded educational application. It currently has billions of data points (about six billion courses are completed each month) and is constantly being iterated.

Four ways to motivate users

The biggest challenge for Duolingo is keeping users motivated. Like other apps, if you want to encourage users and cultivate a fixed behavior pattern, in their case, that means completing the language course every day. In fact, if users cannot access it every day, then their courses are ineffective. Because it would be difficult for one person to study a course, and online learning always brings distractions to users. Product Manager Zan Gilani said: MOOCs (massive open online courses) have a very low completion rate, about only 1-2%. To overcome these challenges, Duolingo uses gaming to change users’ daily habits in four ways.

• Detailed goals
• Clear progress
• External triggers to users
• Let users invest money

In Duolingo, “detailed goals” are integrated into the APP, and users unlock the next course unit by completing a course unit. For example, when you set "fitness" as a goal, this goal is very vague, but it is easy for you to run a mile every day, and it becomes possible to achieve the goal.

Let users "win streak"

Duolingo focuses on persistent usage by users. On this basis, Duolingo has set up a powerful game mechanism to make it consistent with the formation of user usage habits. Duolingo reminds users on apps like Snapchat (a well-known social app with strong relationships), Facebook, and Headspace. Rewards are given when users use the game continuously, that is, when they maintain a winning streak in the game.

At the same time, Duolingo encourages users to set daily goals, which to a certain extent can constrain users to log in to the APP every day. For new users, continuous logins mean having a goal, so the team added a goal-setting function in the account settings. To this day, the company team still uses this indicator to measure work results. "It's like a video game where users have to do something every day to stay on the leaderboard ."

To ensure continuity for users, Gilani publicly displayed an email sent from Duolingo headquarters to users, which explained to users the Internet outage caused by Hurricane Irma and requested users to log in to Duolingo again to stay active.

A/B Test 1: Visualize the Process

Users generally want to see their progress, but the Duolingo app does not allow you to see the number of days a user has progressed. Therefore, the team added this time display at the top of the page. The results are significant: daily activity increased by 3% and 14-day retention increased by 1%.

A/B Test 2: Emphasize the importance of “continuity” to users

Next, to emphasize the user’s “streak value” after each course, Duolingo displays the course progress using a circle as shown below. The result was a 1% increase in daily active users and a 3% increase in 14-day retention.

A/B Test 3: External Trigger

Notifications such as emails and applications can easily make users feel annoyed, but "goal achievement" will make users less disgusted by external triggers. “Come back to Duolingo,” it sounded so bland and annoying. Instead of “Open our app,” Duolingo’s notification says, “Remember, you’ve made it 24 days.” This encouragement can help users maintain their goals. ”

It is worth mentioning here that when pushing, you should pay attention to the time and frequency of push, etc. Whether the timing is grasped correctly is directly related to the success or failure of the push. There are some rules for when to send push notifications during the day, depending on which app you use. The results showed that sending the email 23.5 hours after a user completed a course was most effective in stimulating user engagement. This isn’t surprising, as Zilani says that doing the same thing at the same time every day is the best way to build habits.

Push content also needs to be tested continuously. Some of the language used in Duolingo’s external triggers is also particularly interesting. If a user is “notified” but still does not log in, they will receive a “passive-aggressive notification” that begins with “These reminders don’t seem to be working…” This tone fits the playful nature of the brand very well.

A/B Test 4: Testing the “Winning Streak” Bet

Duolingo has an internal virtual currency, lingots, which users earn for completing lessons and can be used to purchase virtual items that increase certain features of the site, such as access to special language courses—do you want to learn to speak Italian? —or answer a question to find out how you’re doing.

The team tested a project about streak betting, where users could bet on how many consecutive wins they would have and use lingots as bets. This feature surfaced in the latest version of the application.

The result was a 5% increase in 14-day user retention and a 600% increase in IAP (in-app purchase revenue).

It is worth mentioning that Duolingo has always been free to use, and users pay mainly for offline use of the APP (to ensure winning streak) and ad-free.

In this test, Gilani said the team once set a bet on a two-day winning streak, but found that the bet had the smallest increase in 14-day retention and daily active users, so they eventually chose to abandon it.

A/B Test 5: Improving the “Weak Link”

Duolingo found that user activity drops by 9% on weekends compared to weekdays. If users who insist on studying every day accidentally miss the check-in on a certain day, many users will lose the opportunity to win in a row, which makes users very discouraged.

What if a user could skip a day and still keep their streak? Therefore, the team tried to allow users to purchase the "weekend magic weapon" on Friday, which ensures that even if users do not check in on Saturdays and Sundays, it will not hinder their learning progress. Of course, users need to pay 20 lingots for this "amulet". It turns out that telling users “you don’t have to play on weekends” has a huge impact on product adoption: 7-day retention increased by 2.1% and 14-day retention increased by 4%.

A/B Test 6: Enrich the channels for users to win

It’s not just winning streaks that keep users engaged. Duolingo provides users with "badges" that represent their achievements. This is a right that is difficult for users to obtain . Since the launch of the first version, the biggest achievement is that some new users have joined Duolingo. However, this is nothing to celebrate because DAU increased by 0% and D14 retention increased by 0%. After reflection, the team decided that users must see what they have achieved, and that signing up was not an achievement and would not make users feel proud. What users want is the process of collecting medals.

When conducting A/B testing, it is important to understand that if you want an idea to be implemented and bring good results, what you think is the simplest way may not be the simplest. In the term “minimum viable product”, although the word “minimize” comes first, the word “feasibility” is the core.

In the second test, the team designed several different badges and set up a page to store the badges. Badges are distributed in levels. For example, to obtain a perfect badge, users need to go through 5-10 or even more in-depth tests. After the team's efforts, DAU increased by 2%, 14-day retention increased by 2%, and in-app purchases increased by 5%.

Finally, the team summarized and decided to pair product managers or team managers with designers when developing promising features or making secondary attempts (such as Duolingo’s badges) to form a countervailing force and maintain the team’s passion and execution. If a test fails, don’t forget to reflect on it.

An excellent educational product designer can enable more people to go further and learn more. One success story is a Colombian security guard named Edilson, who learned the language through Duolingo and now has a job teaching English at the Security Academy.

So what kind of A/B testing can you use in your apps and services to retain customers?

The author of this article @乔一鸭 is compiled and published by (Qinggua Media). Please indicate the author information and source when reprinting!

Product promotion services: APP promotion services, advertising platform, Longyou Games