Testin Cloud Testing CTO Xu Kun: Service-driven cloud testing

[[127491]]

Xu Kun, CTO of Testin Cloud Testing. He is a founding employee of PICA, the earliest mobile Internet company in China, and was PICA's vice president. He is also the founder of Shanshuidi Information, a leading HTML5 game development company. As an architect of instant messaging systems with tens of millions of people online, he led the development of a mobile social platform with tens of millions of users.

Testin Cloud Testing was established in June 2011 and has been engaged in mobile Internet APP testing services. Testin wants to turn testing into a cloud-based standardized service. The ultimate vision is to allow R&D teams or APP teams to focus on their own products and marketing, and ultimately solve testing problems through cloud services.

Generally speaking, the APP development model of mobile Internet is an APP team, which probably has several departments. The first is development, which we commonly call technology. The second is product, and generally there are operations and marketing. If it is a team selling to enterprises, there will also be sales, and then there may be after-sales. In addition to these functional departments, there is a testing functional department and an operation and maintenance functional department. Let's take a look at what testing-related services Testin can provide to developers in the process of developing and testing APPs:

Functional testing

For general APP, the first step is function development, then release, and then online promotion. If the APP is not dead, it will repeat this process. In the function development stage, Testin will provide a service called function testing. Testin is about to launch a crowd testing service, which will bring together excellent domestic test PMs, and then provide a solution to all APP developers through PM. Testin is a cloud-based crowdsourcing platform with many testers. Why is there such a service? The development stage generally takes 14 days, including product design and technical development time, and two days are reserved to enter the testing stage. The standard APP development team has 10 people, two products, two technologies, and finally one or two people to do operations, and there is no one in the real testing position. The heavy task and small number of personnel have led to many mobile APPs on the market with many bugs, but it doesn’t matter. For developers, the first step is to have users, but as the number of users increases, it is found that quality problems are becoming more and more prominent, so in order to solve this problem, Testin proposes such a service at this stage to specifically solve this problem. Through a crowd-testing platform with tens of thousands of people, the PM (test project manager) helps design the plan, including writing the tests, and then distributes them to many testers. The PM then summarizes the results, including bugs, and then there will be a test report.

Adaptation test

After the functional test is done without any problems, a problem will arise when the release is near. China now has many mobile phone models and manufacturers, including Xiaomi, Samsung, and Huawei. Each brand has many models. Xiaomi used to have only one model, but now Xiaomi has almost ten models, including Xiaomi 1, 2, 3, and so on. At this time, the version split of each model will become more and more serious. For example, although an APP can run on Xiaomi 4, it cannot run on Meizu phones. The screen resolution of Meizu 4 is different from that of other phones. If you pay attention to mobile phones, you will know that Meizu Pro is a little different. They use MIU, which they made themselves. Huawei also made one, including Vivo and OPPO. Although there is no problem with the function, adaptation has become a very serious problem. The adaptation problem is because the version split, different screen sizes, different mobile phone models, and different Roms will bring various problems. If you want to solve it yourself, how does the general development team do it? He will buy four or five mobile phones, he uses Xiaomi himself, and there are Samsung in the team, and then try whether his APP can run. However, there are about a thousand mobile phones on the market, and China has the most models in the world. Even if it works on Xiaomi 1, it may not work on Xiaomi 2 or Xiaomi 3. Many people use iPhones, and it may work on iPhone 4, but not necessarily on iPhone 5. So Testin has an adaptation test service.

How does Testin do adaptation? It is very similar to crowd testing. Testin is connected to something similar to a cloud testing platform, which has many mobile phones hanging under it, which are real machines one by one. Developers submit the APP to Testin through Testin's Web website, and Testin sends these APPs to more than 4,000 real machines in the warehouse to run. Testin has its own unique technology and an automated script. The APP comes to Testin's adaptation platform, and through this script, it is automatically installed, run, and uninstalled on all mobile phones. The first is whether it can be installed, the second is whether it can be uninstalled, and the third is whether it can perform basic functions normally. At present, Testin's adaptation platform has run a total of about 78 million automated tests. Testin regularly counts some mainstream models in the country, and then purchases them, and then puts the new machines into the computer room for all developers to use.

Application and mobile game performance monitoring

So after the functions and compatibility have been tested, can the application sit back and relax? Of course not. Testin found that no matter how it is tested, no matter what automated means are used, there are actually many problems that have not been truly discovered. Various problems may be seen on the APP user side because they do not know the user's network situation, whether it is a copycat phone, etc. Therefore, Testin launched Crash Master. Before the APP is launched and promoted, the Crash Master SDK is embedded in the APP. So what is the role of this SDK? After the developer adds our SDK to his APP, Testin will provide each APP with a crash background. This background will capture all crashes, flashbacks, and abnormal situations generated when the user uses the APP, and make a backup after capturing.

1. Code-level location crash

After an APP crashes abnormally, Crash Master can monitor the code location that caused the problem, and developers can make modifications based on the information provided by Crash Master.

2. NDK error capture

NDK error, for example, Android phones use a lot of third-party libraries, such as Tencent or Baidu, which have third-party SDKs, most of which are written in C and are dynamic libraries. This dynamic library itself will produce an exception, which is an NDK error, this is called a native error, mainly a C++ error. Crash Master can capture all problems that occur in third parties. Many third-party platforms themselves can cause program crashes and flashbacks.

3. One-to-one user positioning

When the client crashes or crashes, the developer cannot know the identity of the user. When users report a problem after a crash, they may just vent their emotions in the comments, and generally will not explain what their model is, what the network situation is, etc. Crash Master can display the ID of the crashed client, especially for O2O companies, or companies that use mobile phone numbers as IDs, so that they can call the user to check what the problem is.

4. Mobile Game Engine Support

Crash Master supports two mobile game engines, namely cocos2D-X and Unity3D, which are also third-party tools similar to SDK. Developers will use several scripting languages ​​based on this engine, one of which is Lua and the other is JavaScript. Crash Master can capture the exceptions generated by these scripting languages ​​through the SDK service.

APP Application Cases

Mobile APP developers give their requirements to Testin. During the entire development cycle, from function development to release to online launch, Testin will conduct a complete tracking of the developer's APP. Testin's public testing service has currently served companies such as E袋洗, AutoNavi and some others. Adaptation testing service is Testin's earliest and ongoing business. 65% of domestic mobile developers basically use Testin's adaptation testing service. Testin's Crash Master was completed in August 2014, and the Bate version was launched in September. In the three and a half months since then, most of the larger games in China are users of Crash Master. For example, Gunfight, Fishing Master, Papa Three Kingdoms, Defend Carrot, Dena One Piece, most of them are games. It should be in the top ten in terms of user base among third-party SDKs in China.

Crash Master service is about to launch version 1.7.2. From September 2014 to now, it has iterated more than 30 versions, about one version per week, and now there should be more than 30 versions. The SDK package of Crash Master is very small, only 30K, and its stability is beyond doubt, and it can capture more things. Currently, thousands of apps have used its services.

Crash Master has merged the exception captures. When reporting crash exceptions, they are not uploaded one by one, but merged together and reported in categories. In this way, developers can distinguish the causes of exceptions (the program does not crash but the function has problems) and crashes (causing the program to crash). At the same time, this reporting method can reduce traffic for users and prevent the user's APP from causing a surge in traffic due to the SDK.

Crash Master version 1.7.2, a new upgrade of crash + adaptation, Testin deeply integrates the original adaptation and current Crash Master functions, so that the adaptation function can also accurately locate many underlying errors, making the adaptation function more accurate.

Crash Master currently provides local services, and also has an overseas English version, which has been released. Testin has a strategic partnership with Cocos Touch. Based on Crash Master, the Touch North American team and Testin jointly operate and promote it overseas, providing services to more developers in North America and other overseas countries.

Conclusion

Testin is a service-driven team. ------Xu Kun

Testin is based on technology and service-oriented, providing developers with a service to improve the quality of apps. The founding team of Testin all come from R&D backgrounds. In the past, Testin believed that technology could change everything, and even believed that all problems could be solved through automation. After about three years, such as functional testing, it was finally discovered that some problems could not be completely solved through automation, because there are some things that only people can know whether this is right or wrong, so a crowdsourcing testing platform was launched. Problems that can be solved technically, such as automatic capture of crashes, can make the technology more precise and thorough, and then help developers to do crash statistics and make the recognition rate higher. So technology is the foundation of Testin, but what is ultimately delivered to users is a service. The value that Testin brings to customers should be that Testin helps developers discover problems that developers cannot find themselves. This is what Testin is doing in Xu Kun's eyes.