Produced by Tencent experts! AR basic knowledge and design practice case review and summary

I hope that through this article, you can quickly understand the implementation principles of AR, the design process of AR, and the problems encountered in the design process. At the end, I will share some of my experiences and skills in AR design over the years.

I joined Tencent in 2013 and have been responsible for QQ-related design since then. I started to participate in the design of short video APPs in 2015, which is to make QQ's social content entertaining and combine it with some new technologies.

About AR

The word AR is not a strange word to the public, but what is the implementation principle of AR?

The Chinese name of AR is augmented reality technology, which is a technology that calculates the position and angle of camera images in real time and adds corresponding images, videos, and 3D models.

Simply put, it is to superimpose some virtual information on the real world, and then calculate its position and angle changes on the device in real time.

The goal of AR is to place virtual content on the screen in the real world to interact with it, helping the public to obtain information more quickly and intuitively, and AR has strong commercial value. Friends around me often confuse AR with VR. In fact, AR is augmented reality technology, and VR is virtual reality technology.

AR places virtual objects in the real environment around us to enhance the sensory experience of the real world; VR places us in a virtual environment to create an immersive experience that is so realistic that it is hard to tell the difference between the real and the real.

They are divided on the basis of devices. AR has a low threshold for participation and an even lower threshold for operation for the general public, because AR only requires a smartphone with a camera to become a carrier, which can enhance the real world and reshape the user's interactive experience. VR devices are difficult for the general public to access on a daily basis and require more professional equipment and venues.

The core of AR technology is recognition and tracking, which can be implemented in two ways:

• Image detection method;
• Global Positioning Satellite System Law.

Image detection method

Image detection is actually the most common positioning method for AR, and is often used in image recognition and face recognition. Its technical implementation process is mainly divided into three steps:

• Step 1: It is necessary to capture the feature image. The algorithm will plan and process the image and calculate the valid feature points for matching. As shown in Figure 1 below, these small green dots are the valid feature points.

• Step 2: Calculate a three-dimensional coordinate system from the deviated rectangle and establish a space.
• Step 3: Track the changes in the position and angle of the data content in real time.

I just mentioned the green dots. How should designers detect them? Here I recommend a website to everyone - Vuforia. After registering an account and uploading the design drawing, you can directly mark the feature points of the drawing and score it. The more feature points there are, the higher the recognition accuracy.

By observing the two works above and below the first picture, we can find that they are just different in background and depth, and the effective feature points formed are also different, because the algorithm will form more effective feature points when the grayscale contrast is high.

If we observe the second picture again, we can find that there is no valid feature point in the circular part of the sun in the second picture, because the algorithm will exclude some relatively smooth curves or elements as invalid feature points. Such points generally appear at the angle. Therefore, in order to make the feature point score higher, we need to design it as follows:

• Avoid drawing smooth or curved elements;
• Feature points need to be evenly distributed in every corner.

Finally, look at the third picture. Parts of the geometric cloud are repeated in different positions, and the number and position of the valid feature points are the same. Because the repeated regular graphics will be judged as invalid feature points by the algorithm and excluded, so when designing, you need to avoid using some regular graphics and some repeated elements.

The advantages of image detection are simplicity, efficiency and diversity. The disadvantages are that it is easily affected by factors such as environment, light, distance, occlusion and focus, which can cause virtual content to jitter, freeze, misjudge, slow to recognize or even unrecognizable.

There is no need to explain the distance of ambient light. Occlusion depends on the angle and range of the viewfinder. Focus refers to the device, that is, our camera. Our mobile phones will focus slowly on dark colors and gradient colors.

Therefore, when making a visual draft, many conditions need to be met, and multi-environment testing and discussions must be conducted with the developer to come up with a relatively qualified recognition map. An excellent recognition map must ensure its stability and recognition speed.

Let's see how it is applied in actual projects.

In 2016, the team planned the "QQ Olympic Torchbearer" AR event, basically with no experience at all, and constantly tried and made corrections. It took nearly 3,000 drafts from style to final draft within two months to complete the design of 28 identification maps of six Olympic cities. It can be seen that the effect presented by the DEMO is relatively stable, and the recognition speed is very fast, basically completed within one second.

I emphasize the word "excellent" here because we don't want users to feel frustrated when they first come into contact with AR, which would lead to resistance to the new technology. Therefore, we did a lot of debugging on the speed and stability of the recognition rate, and also obtained the Guinness World Record certificate at that time.

This is a small demo from the past, which shows the process of scanning and identifying the content layer by layer through the mobile phone. We just talked about the image scene of the picture detection method. Now let's take a look at the texture recognition scene. Face recognition is actually the edge detection of the image detection method. It can detect the parts of the face and track the movement of these parts to seamlessly fit the virtual content with the body.

For example, the often used special effects widgets, what kind of capabilities do some of the beauty apps on the market have?

• Real-time beauty
• Face fitting
• Action Trigger
• Facial deformation
• Face swap, face removal
• Atmosphere filter
• 3D Model

I won’t elaborate on the rest, I believe everyone understands it.

For some social scenarios of QQ, our team has also developed some special effects widgets unique to QQ, such as the shooting function in the QQ chat interface, and the chat function, which can be seen and played in the video chat. We also used the combination of AI and AR to make a small game during a call between two people, and win the other party's red envelope by making expressions.

[[270343]]

Global Positioning System Act

The Global Positioning System method uses GPS to locate our location and find a plane in the real space so that we can interact and view virtual objects in the real world.

The advantage is that it is suitable for outdoor tracking and can overcome uncertain factors such as lighting and focus in outdoor scenes. The disadvantage is that it is very demanding on the performance of the device, including the current network environment.

You can see there are two unfamiliar words here, one is LBS (Location Based Service) and the other is POI (Point of Interest). LBS is actually a way to obtain location based on mobile location service and configure service GPS. Some AR games are implemented based on LBS, and the implementation method they use is the Global Positioning System. POI is a point of interest, which can represent a building, a store or a scenic spot on the map, etc. It contains four aspects of information, namely name, category, coordinates and classification.

This is an AR game made in Japan in 2016. I believe everyone has played it.

[[270344]]

This game is implemented using the Global Positioning System. The reason why I showed you the short film just now is because Tencent's 2017 New Year QQ operation activity is very similar to its gameplay. Both are based on looking for a red envelope or a treasure on a map, then finding the location, and then completing some tasks to get these red envelopes or gifts.

When doing this operation, we need to consider how to embody the spatial concept of AR, such as making rational use of a real space, because the design draft produced by the designer is no longer in the size of 750×1334. It is a real space that needs to be expanded to guide users to move the mobile phone lens and track the movement and interaction of these virtual objects.

We can reflect a sense of space through the direction of objects, for example, letting users move from far to near, from left to right, from top to bottom, etc.

In terms of map guidance, we can create a three-dimensional space from the level of distance. For example, when it comes to AR points that are relatively close to us, we need to strengthen our information and make some differentiated animation guidance. For places that are farther away, we only need to give a simple sticker or simple element to inform the user that there is also a red envelope or gift in the distance.

[[270345]]

The power of IP is very strong. At that time, baby Q from QQfamily was used as the image of the God of Wealth to send red envelopes to users. Its function is to have an equal dialogue with users, accompany users to complete some tasks, and establish an equal and intimate relationship.

In 2016, through the "QQ Olympic Torchbearer" event, QQ was known as the first social application in China with more than 100 million AR users. The entire event attracted users from 143 countries around the world, and the interactions with AR exceeded nearly 200 million times.

In 2017, the number of users of the "QQ Red Packet" operation activity exceeded 300 million, and it gained a good reputation in the industry and among some users within half a year. QQ is a very young product, and most of its users are students. By combining such activities with new technologies, we have successfully upgraded the QQ brand to be younger, more entertaining, and more technological.

In the past, the design method was to make some innovations in graphics, animations and colors. Now we have added some new technologies, such as AI, AR, sound effects and physical vibrations, which can upgrade the user's sensory experience and make the design more immersive and entertaining.

The reason why AI and AR are well combined is that better AI requires better AR to reflect it.

So everyone says AI is AI and AR is AR, but this is actually wrong because they are very closely related. If I were to make an analogy, “AI is the human brain and AR is some of the human sensory experiences.”

Combining with these new technologies will give designs and activities four advantages, namely topicality, connectivity, immersion and entertainment.

By using new technologies to bring freshness and topicality to users, and effectively connecting elements such as time sound effects and vibration images together, users can be immersed in the experience and focus more on the operation, which further enhances the overall design and the fun of the entire product, and also improves the entertainment value of the entire product.

During last year's World Cup, we launched "QQAR Across the Stadium".

[[270346]]

Compared to the Olympics, the World Cup is not an event that everyone pays attention to, so we spent a lot of time doing user research and trying every way to increase user participation. We analyzed user behavior and psychology, hoping to get users to participate better through new forms of interaction and new technologies.

When doing operational design, you need to plan from three parts:

• How to attract users;
• How to create some surprises during the user experience process;
• We hope that users will have an outlet to express their desires and share the activity in the end.

So how do you attract users to participate in activities? You can use some visual clues, highlight some key information, and satisfy users' curiosity through differentiated design. For example, use the movement trajectory of a football to attract users' attention, arouse their curiosity, and click on the entrance. When they reach the second page, they need to be told the specific content of the activity. This is a relatively light design technique. A more direct way is to give users an activity preview directly after logging into QQ.

This needs to be combined with the QQ scan interface and the scenes in the event to create a new UI style. We can tell users through some color textures and execution animations that this is a high-tech operation activity, a new UI activity style, and a new high-tech style.

Immersive experience often also includes people's sensory and cognitive experience.

We need to extract some design elements from the public's memories and combine them based on current scenarios and applications. So how do we extract some memory elements of users? We can extract its color, texture, sound, graphics, movement, optics and mechanics from some fragments of our memories and integrate them with the design.

When it comes to the topic of time travel, everyone will think of Doraemon's time portal or Doraemon's time machine. Some users will also think of a clip from Doctor Strange, which has been very popular in recent years. We extracted some design elements from the clip of Doctor Strange opening the time portal, such as its movement, mechanics and optical characteristics.

[[270347]]

First of all, we need to educate users about "drawing circles in the air". The term "drawing circles in the air" should not be difficult for everyone to understand. At that time, we invited many Shenzhen university students to the company to do user research. When trying out this activity, we found that many students did not know how to play it. It's not that everyone has not seen Doctor Strange, nor that everyone has no contact with AR, but because this new form of interaction will create an uncertainty in the user's psychology and doubt their own judgment. Everyone is more accustomed to drawing circles directly on the screen, rather than raising their hands, drawing a circle in the air, and opening the portal. Some students even turned their hands over and kept pressing the camera. This kind of situation will occur.

This makes our design more objective. Things that we think we understand well are actually not easy to achieve for some general users.

Based on this research, we combined pictures and texts, and finally optimized more than a dozen solutions. We kept looking for students to do user research, and finally came up with a solution - simulating a real circle drawing scene. We made an animation, and added concise text and operation feedback to guide users' operations. At the same time, it also supports users to draw circles on the screen, but it cannot be done by holding the camera. This is to avoid some interruptions during the user trial.

Regarding AR circle drawing, for designers, designing the animation of this point seems to be very simple, but it is actually very difficult. Developers need to consider issues such as mobile phone performance and QQ engine. In order to ensure a smooth user experience, the design may require some sacrifices.

At that time, we had a restriction that the animation must be within 5 to 8 frames. However, it would cause freezes, color difference, blur, and stiffness. In addition, when we used AE's particle effects to make an animation, it was actually not consistent with the development implementation principle, so we had to constantly debug and restore the effect with the developers. In addition, for example, the particle effects in AE would have a large color difference after exporting to PNG, which all needed to be constantly corrected and solved.

We debugged in many aspects, revised the plan, looked for solutions, and finally achieved a relatively satisfactory result.

After drawing a circle in the air, it will immediately form a 3D stereoscopic portal, which allows us to view 360 degrees and travel back and forth between real and virtual scenes. This requires a lot of faces to do some mapping processing, but there will be technical problems, so we can only use the most concise way to achieve our ideal effect.

Due to some engine and technical limitations, we need to think of many ways to integrate resources. For example, from AR circle drawing to AR gate crossing, and then to a virtual AR arena, there are actually four different resource synthesis, and we need to strictly control the entry and exit of the entire time.

The first step: You need to find ways to use some clever techniques to make it transition naturally.

The second step is to make users feel as if they are in the game. This can be done by enhancing the user's sensory experience from the three aspects of vision, hearing, and touch. The entire AR stadium can be viewed 360 degrees from a first-person perspective, and then combined with some real sound effects on the scene, such as the commentator's narration, to restore a sense of reality.

Part 3: Touch. We know that after the iOS upgrade, the phone has three vibration levels: strong, medium, and weak. We designed the frequency and length of these vibrations to restore the touch of kicking the ball, the ball hitting the goal, and the ball rolling back and forth on the grass. The main purpose is to enhance the user's sense of reality.

Step 4: How to enhance users' desire to share. In addition to creating a real newspaper texture, we layout the content according to users' reading habits and distribute it in every corner. Next, we integrate these elements together based on the color matching of the uniforms of the entire country, different praise copywriting, and the process of users drawing circles. There will be video recordings to create a magic newspaper. These personalized designs can enhance users' sense of participation and make users want to share.

Design Experience

Suggestions for AR design process:

• When defining gameplay, interactions and operations should be as simple as possible to lower the user's participation and operation thresholds, and avoid making users feel frustrated.
• During the interactive stage, we use user research to determine the best onboarding plan to ensure that users understand how to play this activity;
• In the visual stage, time needs to be reserved for development and testing, and repeated debugging to restore the visual effect. The design principles of AR are actually the same as most design principles. It is a pyramid triangle relationship. First of all, applicability must be ensured. Applicability means that there must be some simple and clear operations, and some guidance to suggest that users will use AP or play this activity, so as to avoid causing some psychological frustration to users;
• Stability. It is necessary to ensure that the entire experience process is smooth and provide multi-dimensional operational feedback to avoid accidents or interruptions during the user experience;
• The last thing is pleasure. Through design techniques, users can get some entertainment and satisfaction, immerse themselves in it, and forget the real world.

Suggestions for AR immersive experience:

• Scenario: You need to design a background or plot to attract users and make them want to explore;
• Space: Real space needs to be used rationally;
• Plot: It is necessary to define some detailed goals, so that the user's behavior has some special feedback, and the user has an absolute sense of control over the entire experience;
• Roles: You can create some role relationships and situational designs to enhance user participation;
• Atmosphere: It can enhance the pleasure of the whole experience from three aspects: vision, hearing and touch;
• Rhythm: From simple to challenging, it needs to be gradual and gradual to keep users focused.

Conclusion

Finally, here are some reflections on my AR project.

I used to think that the job of a designer is just to consider how to package the app, how to package the brand, and how to make it look good. In fact, it is not. We need to consider some user experience issues, and we need to have some product thinking. All designs that integrate with new technologies have high risks. A good design is actually the best solution under many conditions.

We will encounter many difficulties and limitations during the design process, and we need to use design to solve some technical problems. We will use these limitations as the starting point for thinking about the problem and provide a more complete solution. At the same time, I also believe that science and technology are constantly advancing, so one day we can achieve our ideal design.