How to A/B test your app icon

[[145648]]

In addition to the app name and preview image, the app icon is also one of the key factors for users to decide whether to download the app. Although the app icon is only a few hundred pixels, its role cannot be underestimated.

Last time we discussed the elements of app icon design. No surprise, you already have 3-5 eye-catching app icons.

Now it’s time to make a tough choice: you can only choose one. The icon you choose will be used for app promotion, brand identity, and more. This choice will have far-reaching consequences, so it’s worth doing your homework.

Don’t just pick an icon that your friends or family members like. Make your choice based on data. That’s why A/B testing is so important.

Select the best through A/B testing

Android: Using Google Play's new A/B testing feature

Thanks to Google Experiments, it is relatively easy for Android developers to conduct A/B testing. Google Experiments was announced at the 2015 Google I/O Developer Conference. Developers can see the new "Experiment" option on the development machine and use this feature to test multiple attributes, including icons.

Google Experiments App AB Testing 1 If you want to succeed in all regions, you must conduct global testing. Later you can conduct more complex regional tests. But at this stage, it is most important to have the best icon when the app is launched.

Google Experiments App AB Testing 2 Name the test, select the test user ratio, and check "Hi-res icon" as the test attribute. You will later determine which icon resonates most with users.

Make sure all icons are the same size as Google requires, then upload them. Make sure the images are clear when uploading (if the icons are too small and blurry, the test results will be affected). Now, the test can officially begin.

Google Experiments App AB Testing 3 adds different resolution versions for each icon. Users in different test groups see different icons, and this phenomenon only exists while the experiment is running. There is no need to worry about users being confused when they revisit the actual app page.

With enough data, it should be clear which icon to use. Congratulations, you have tested and found the best app icon.

Don't have the Android app? You can test it on iOS (and other app stores) as well. Read on.

iOS and other app stores: Facebook paid A/B testing

Facebook is a great platform for testing app icons because it has a huge user base, extremely niche targeting options, and is a popular app on both web and mobile platforms.

a. Must be a CPM campaign

Make sure to set your test campaign to cost-per-impression (CPM). This means you will be charged for a certain number of impressions (users will see your banner ad in their feeds). Choosing cost-per-click (CPC) may skew your results, as ads with a high click-through rate will appear more often in players' feeds.

b. Simplify mobile banner ads

During the process, you can test multiple app icon designs through banner ads. The important point is that the banner ad needs to have enough background to let users know what kind of app it is, and the rest of the design depends on the icon.

This is the crossroads where you can take your design (symbol or mascot) in different directions. For example, you can test it with the background color of the icon, the shape of the icon, the mascot, or even the shadow tweaks.

[[145649]]

Will adding this cat have an impact on the target user installation rate? We need test data to prove this.

Once the data is available, the best option will emerge. Simply use the icon that has the most clicks from your target users.

That said, if you’re considering replacing an existing icon, the same approach applies. Use the existing icon as a basis for comparison with the new design. If the data for the original icon is better, then don’t replace it. Otherwise, go with the new icon if the data supports it.

c. Target different consumer groups

Facebook's targeting capabilities are very powerful and can be used to target different groups of mobile apps. Let's see how these icons perform among the target user groups.

The user group targeting options are:

• age

• gender

• Geographic location (Note: Country or region based usually works best)

• Mental state

• Industry Solutions

• income

By refining user targeting through the above options, it is easier to find icons that target users like. For example, if the icon of a beauty product recommendation app is brightly colored or has a makeup-related image, it can attract more young female users aged 18 to 25.

Regarding sample size: When refining your targeting, confirm that the impressions correctly reflect the target audience. You can use an online sample size calculator like this one to determine if there is a noticeable difference in the impressions you get.

d. End in time

When the sample size is reached and the number of impressions for each variant is similar, the test campaign should be ended.

The test activity will continue unless the funds are exhausted or it is terminated manually. If you forget to end it, the data obtained from the test will lose its reference value.

Other Testing Tips

Direct testing

If you want clear results, you need to do a direct test. The test is designed in a completely different way.

We reached out to mobile game publisher Kongregate for some tips on icon testing. "We've found that the best app icon testing is to keep trying different versions from the beginning," said Jeff Gurian, senior marketing director at Kongregate. "Once we find a good version, we start to make various fine-tuning changes until everyone agrees that this is the best version. We often compare existing icons with new designs to determine the validity of the data. This testing is especially important when the source composition and traffic geography are constantly changing."

The above picture shows the results of Kongregate's comparative test of existing icons. It can be seen that the advantages of the winners are mostly very obvious (the third one from the left); after most of the tests, existing icons were optimized.

Jeff also mentioned that a new icon can even attract users who have previously ignored the app. "Our philosophy is that a different or new icon will make users think it's a new game." That's the huge power of a small icon.

Precision testing

Don’t test everything that can be adjusted. Test only those things that are adjusted based on assumptions. Test only those things that the team needs answers to urgently.

For example, if one person on the samurai RPG team is convinced that two katanas as an icon will get more downloads instead of just one, then spot test that. Consistency testing of the game version, name, and icon is not an option.

Identify key indicators

Keep in mind your primary optimization goal. If your app's success is based on retained paying users and you only see an increase in downloads, that means your app icon isn't attracting the right user base. Optimize based on key metrics, not just installs.

Google Experiments cannot test activation and payment levels, these are future features. However, Facebook Tests can provide tracking at this level. Please keep this in mind when testing your icons.

Optimizing key indicators is the key to increasing business volume.

Real-world test results: A/B testing that actually works

When we tested our own app icons, the best performing version had a 15 percentage point higher click-through rate. This means that the app has 15% more users (assuming all of them can be converted into real users), which is not a small number.

This finding points to a trend for all digital products: fine-tuning your design and constant testing can increase the number of users. It’s best to spend more time and effort choosing the icon for your next app. Not only will your users be happy, but your business will also grow organically.