Why would Google open source its most important secrets?

Google's technology is one of the main driving forces for the open source of cloud computing. Open source software such as Hadoop and NoSQL databases all originate from Google's technology. However, the search giant itself is a bit conflicted about the open source attitude of its own technology. This entanglement mainly stems from not wanting to give up its own competitive advantage. However, as Microsoft, VMware, EMC and others have embraced the general trend of open source, Google has finally realized the importance of promoting its cloud strategy in an open source environment, even at the cost of open source its biggest secrets. Now this strategy has begun to bear fruit.

Craig McLuckie took the idea to Urs Hölzle, who oversaw Google's global network of data centers, and he didn't like the idea.

McLuckie and two engineers at Google's Seattle office wanted to make Brog an open source project. Brog is a software tool that manages and schedules the resources needed for everything from search to maps, allowing Google to bundle computing tasks and distribute them across its vast global network. For years, Brog has been one of Google's best-kept secrets. And McLuckie wanted to share its blueprints, at least some of them, with the Internet.

"When we first approached Urs, he was actually not happy about the idea," McLuckie said, "because what we were doing was giving away the secret sauce that runs Google as open source technology."

But Hölzle and other Google officials finally approved the project - the name Kubernetes originated from ancient Greece and means captain or driver. Today is the first anniversary of its launch. In the past 12 months, more than 370 programmers have submitted 12,500 "commits" to the project, most of whom are not Googlers, including three top contributors.

Kubernetes represents a strategic shift for Google as it competes with Amazon, Microsoft and others in the cloud computing space. Traditionally, Google has been very protective of the underlying technologies that power its online empire — Borg is a good example, and to some extent it still is. But now that Google is a company that is serious about cloud computing, it is inviting other companies and developers to build websites and applications on its infrastructure (note: similar to the way Amazon did AWS back then), so it is closed and not public. Inviting other companies to develop and run applications on its infrastructure, so open source is more of an equal exchange for the entire software community.

Kubernetes was originally conceived by McLuckie and two other engineers, Joe Beda and Brendan Burns, who are all members of the development team of Google Cloud Services. In their view, the open source project Kubernetes has pushed developers to use Google Compute Engine, a cloud service that provides "virtual machines" for external ideas and developers, on which almost any code can run, while Kubernetes provides effective management methods for running code across environments (cloud services, local data centers). Hölzle eventually agreed that this was worth sacrificing the advantage of keeping his own code. "The cloud is absolutely imperative for Google," McLuckie said. "We have to operate in a new way. We have to bring our expertise to the community."

The Evolution of Google

Google's move is also a response to a larger shift driven by the rise of cloud computing. It's now easy for developers to develop software on a bunch of machines, often using open source. Running specialized software on so many machines is expensive and difficult to meet specific needs. Whether providing services like Google Compute Engine or developing software for such services, cloud companies now have to embrace open source. Microsoft has also recognized this indisputable trend and embraced open source to promote the development of its Azure cloud computing service. VMware and EMC have also done so.

But Google is struggling a bit. Its technology is one of the main drivers of the rise of open source in cloud computing. It's just that the company has always kept a proper distance from the open source movement. Now it's getting closer.

Over the past decade, Google has developed software to manage the running of software on its thousands of machines. It is very suspicious and sees the ability to run large-scale machines as its competitive advantage, and it does not want people to see the face of this software. Then, after a few years, Google will publish 1 or 2 papers on related technologies from time to time, and the open source community will start to copy Google's technology. This is the case with many technologies, such as MapReduce gave birth to Hadoop, and after BigTable was released, a large number of "NoSQL" databases popped up like mushrooms after rain.

The same is true of Borg. A few years ago, engineers at Twitter and the University of California, Berkeley, developed a tool called Mesos that is now the basis for many popular Internet services, including Twitter and Airbnb. There are also a number of projects developing similar tools around the Docker container technology (which is how Borg works, packaging software into software "containers" and delivering them around the Google network).

But now, driven by the needs of its cloud business and other forces, Google is changing its role. It is not just sharing a research paper or making some small online infrastructure open source, but turning the entire system into an open source project (at least to some extent).

As Burns explains, he, Beda, and McLuckie saw that there were a lot of other projects taking advantage of Borg and the idea of ​​containers, so they thought Google could help move the needle. "We felt like people were having trouble putting the pieces together," he says. "We had the pattern. We had 10 years of experience figuring out how to put the pieces together."

To be sure, Kubernetes is not open source Borg. It is not as complex as Borg or Borg's successor, Omega. As it stands, it cannot manage as many machines. "We are still working towards getting close to the scale of Borg," Beda said. "The early focus of Kubernetes is to get the core concepts right."

However, some of the developers of Borg and Omega are also working on Kubernetes, including Uber engineers Eric Brewer, John Wilkes and Brian Grant. It is of great significance for this open source project that people who have done it are working on Kubernetes, and the future goal of this project is to correct the mistakes of Borg and Omega and eventually surpass these two predecessors.

In a world that values ​​open source so much, this is the best way for Google to compete. For many people, containers are the future of software development. And Google is now taking a leading role in that future.

Just run

However, Mark Kropf, who helped develop a similar system at Pivotal, questioned how serious Google is about the project, noting that Beda has left the company. And Ben Uretsky, CEO of cloud service provider Digital Ocean, believes that the use of systems such as Kubernetes is not that widespread. "Container technology is not mature yet," he said. "We are still at the peak of the hype cycle."

But whatever Kubernetes' fate, in many ways it points to the future of software development.

Google uses its own software containers in its own data centers. But when it launched Kubernetes a year ago, it found an open source partner with Docker. While Docker's parent company sought to bundle its container technology with some of its own software tools, Google and the Kubernetes project began to embrace Rocket, a Docker alternative, a completely independent container format.

McLuckie stressed that Kubernetes now supports both Docker and Rocket. He hopes the industry will eventually return to an independent container format, but he explained that it is not about choosing sides. Kubernetes, he said, "should become Switzerland." It sounds a bit idealistic, but this is what cloud companies must do: provide open source software and let others do whatever they want with it. McLuckie said:

The disruption taking place is bigger than any one company.