LinkedIn's three-member team resigned to found Confluent: a real-time information queuing system based on Kafka

A three-member team from LinkedIn started a business - they were the team members who developed the Apache Kafka real-time information queue technology at the time. Based on this technology, Jay Kreps took the lead in founding a new company Confluent, which is committed to providing real-time data processing service solutions for companies in various industries. The other two members are Neha Narkhede and Jun Rao. The company has received $6.9 million in financing from Benchmark, LinkedIn, and Data Collective.

Kreps described Kafka as LinkedIn's "central nervous system," managing the flow of information coming in from various applications, processing it, and then distributing it everywhere.

Unlike traditional enterprise information queuing systems, Kafka processes all data flowing through a company in near real-time. It has built real-time information processing platforms for LinkedIn, Netflix, Uber and Verizon. Confluent's vision is to allow other companies to use this platform. Confluent has learned about their usage models from Kafka users. There are no products yet, but these practices are enough to inspire what kind of products Confluent should build.

Kreps admits that he was initially skeptical that non-network companies would be interested in technologies like Kafka, but changed his mind when he saw the widespread adoption of the technology in the financial services and telecommunications industries. In March of this year, he visited a company called Synapse Wireless, which uses Kafka to support a sensor network system for tracking the hygiene habits of hospital staff.

Therefore, he believes that the demand is absolutely there, and Confluent's advantage lies in near real-time performance. Kreps believes that building a company based on Kafka information processing technology is always more reliable than building a company based on open source stream processing technology such as Apache Storm, because information queues are a more basic component of advanced data processing architectures.

He still remembers that when he first joined LinkedIn, the website only had a batch processing system, and everyone was very happy when the stream processing system was later introduced. Later, they realized that LinkedIn did not have an architecture that supported this system.

“The reason most companies today are lagging behind the best is that they have very little access to data in real time.” Once companies find the right technology stack, they can build IoT or other sensor-based applications, or other businesses that pull large amounts of data from many sources into back-end systems.

“This actually opens up entirely new use cases,” Kreps said, “and they’re use cases that otherwise wouldn’t really be opened up.”