Google’s Li Cong: How was the “6 9s” operation and maintenance achievement achieved?

On April 14-15, 2016, the WOT2016 Internet Operation and Developer Conference hosted by 51CTO Media was held at the JW Marriott Hotel in the Pearl River Delta, Beijing. Adhering to the concept of focusing on technology and serving technical personnel, the WOT brand conference has been successfully held for eight sessions since 2012, accumulating a large number of technical expert resources, and has been unanimously recognized by the majority of IT practitioners and technology enthusiasts, becoming an important technology sharing and exchange platform and network expansion platform in the industry.

At the meeting, 51TO reporters interviewed Li Cong, the leader of Google's engineering team. He has worked at Google for more than seven years and has led the development and maintenance of multiple projects, including front-end, back-end, and offline operations.

How are “6 9s” achieved?

In his speech on the theme of "Operation and Maintenance Concepts and Practices", Li Cong said that the goal of his operation and maintenance concepts and practices is only one, which is 99.9999%. The industry's operation and maintenance experts all know that in the high reliability of software systems (also known as availability, described in English as HA, High Available), there is a standard for measuring its reliability - X 9s, X 9s means the ratio of the normal use time of the system to the total time (1 year) during the one-year use of the software system. This X usually represents the numbers 3 to 5. To achieve the mark of 6 9s, that is, (1-99.9999%)*365*24*60*60=31 seconds, it means that the maximum possible business interruption time of the software system in one year of continuous operation is 31 seconds, which is a very high standard indicator.

When asked by reporters how this standard was achieved, Li Cong said that six nines are difficult for many companies, and are not particularly necessary for many companies. However, for some important projects and services, to achieve six nines, we must first rely on engineering methods to solve software development, release, and operation problems; coordinate different teams through DevOps thinking, reach common goals in terms of organizational structure and management concepts, and achieve high availability. In addition to management, there are also technical methods, such as infrastructure, or at a lower level to the machine level, or at a higher level, such as from testing to release, the entire process, all the various links must be coordinated together, just right, to achieve six nines.

What is my operation and maintenance philosophy?

Li Cong told the reporter that his operation and maintenance concept is still the concept of DevOps, that is, how to promote the two teams to achieve a common goal through cooperation. These can be achieved through organizational structure, composition methods and technical means. For example, he did a very good job in the operation and maintenance of the Google+ project some time ago, which can achieve rapid release while ensuring high availability. He gave an example, for example, if a developer wants to launch a new project, he can submit it today and it can be launched tomorrow. This can be achieved without reducing stability.

Will automated operation and maintenance take away the jobs of operation and maintenance engineers?

IT operation and maintenance has become an important part of IT service. Faced with increasingly complex businesses and increasingly diverse user needs, Li Cong believes that although automated operation and maintenance will not completely replace traditional operation and maintenance, its proportion will increase in the future. So, with the continuous evolution of automated operation and maintenance, are people worried that automated operation and maintenance will take away the jobs of operation and maintenance engineers?

Li Cong said that the jobs of operation and maintenance engineers will not be taken away, but they need to evolve and improve.

"When automated O&M gets better, your O&M engineers will have more advanced work to do. This is actually a better thing for O&M engineers, because they can hand over many of the things that they have done miserably in the past to automation. I think this is a better thing for everyone."

Interview ***

He also mentioned that a mature automated operation and maintenance system should have automatic monitoring, automatic error correction and similar alarm functions, and provide a series of supporting tools, such as rollback and release.

The reporter asked Li Cong what advice he had for young people who are interested in working in the field of operation and maintenance automation. He said that they should find a position that deals more with operation and maintenance, learn more, observe more, and think more, and they will gain something.