Apple and Siri's seven-year itch: Siri's lonely road

Once upon a time, we thought Siri would be Apple's next big innovation. But seven years after its launch, Siri has become a burden to Apple. This article tells the story of Siri's decline.

[[222968]]

In 2011, Apple held a "Let's talk iPhone" new product launch event at its Cupertino headquarters. It was a historic moment and the first major launch event of new CEO Tim Cook. He tried to continue Steve Jobs' product story through this event. But disappointingly, what was launched was not a brand new iPhone 5, but an updated version of iPhone 4, iPhone 4s.

Siri saved the day at an otherwise disappointing event. The digital assistant wowed reviewers with its ability to respond to voice commands. It could automatically create reminders, schedule events, or provide information about restaurants or the weather. The consensus was that it would be revolutionary. It also indirectly helped boost iPhone 4s sales, with Apple selling four million units in just three days, the most ever sold iPhone at the time.

Now, seven years after its launch, Siri has been criticized. According to early reports from analysts, Siri was the main reason for the performance of Apple's latest product, a $349 HomePod smart speaker. Although the speaker won praise for its stylish design and perfect sound quality, Apple eventually abandoned the speaker's Siri voice function after testing it repeatedly uttered words such as "clunky," "annoying" and "awkward."

To find out how Apple beat competitors such as Amazon and Google in the field of voice assistants, The Information interviewed more than a dozen former Apple employees who were responsible for developing Siri or integrating it into Apple's systems on different teams. Most of them expressed their opinions anonymously.

It is not common for former employees to express their opinions. Apple is trying to reshape its image, and in addition to hardware design, it is also trying to focus on innovation and providing high-quality online services. It has stumbled in the online service business, especially the low-quality iCloud service. With the decline in iPhone sales year by year, online services may hurt Apple's future development.

Before Siri technology was fully mature, Apple decided to embed Siri into the iPhone 4s system. Many of the above-mentioned former employees said that this immediately triggered a heated debate internally: should we continue to fix the flawed architecture? Or should we tear it down and start over? This debate has become more and more lively since Siri was released. This debate is just the beginning. The various Siri teams are more keen to participate in small group struggles and have heated debates on the ideal version of Siri - is it a fast and accurate information acquirer? Or an information assistant that is good at handling complex tasks?

These teams were infighting among leaders and middle managers who lacked Jobs’ vision and influence. Meanwhile, Jobs died of pancreatic cancer the day after Apple launched Siri. These former employees said that lack of leadership and constant turnover put Siri back in trouble. But the main reason for its failure was Apple’s closed culture that did not open up to the outside world, which failed to provide more opportunities for outside developers to create more useful Siri applications.

Bill Stasior, a quiet and humble engineer who was a prominent search expert at Amazon before joining Apple in late 2012, has been in charge of Siri's development department. Stasior is responsible for Siri's core technologies - speech recognition system, natural language processing system and Apple's various search functions. Former Siri employees said that these departments competed with each other for more attention and resources, and conflicts often occurred between departments.

An Apple spokesperson declined to make Stasior available for an interview. In a statement, the company called Siri "the world's most popular voice assistant" and touted its capabilities across Apple devices.

"Siri has made significant progress in performance, scalability, and reliability, using the latest machine learning algorithms to make it more natural-sounding and proactive," Apple said in a statement. "We will continue to invest in machine learning and artificial intelligence to improve the quality of Siri's answers and expand the range of questions it can answer."

To be sure, no company has the perfect digital voice assistant that looks like something out of a sci-fi movie, but Apple’s products are ahead of Amazon and Google in some areas: Siri supports 21 languages, while Amazon’s Alexa only supports 3 languages; Google Assistant supports 8 languages ​​(Google said last month that Google Assistant will soon support more than 30 languages). Siri is used on more than 500 million devices each month and handles 2 billion question and answer requests each week.

Siri now responds to commands much faster than when it was first introduced, and in terms of the number of users, since Siri is embedded in the iPhone, which is said to be one of Apple's best-selling products, they confidently claim that the number of users using Siri far exceeds that of other voice assistants.

Still, Siri's capabilities remain limited compared to its competitors, and in 2016, Apple introduced SiriKit, a tool that gave outside developers the opportunity to build new features for Siri, but it failed to gain much developer interest.

By comparison, rival Amazon Alexa has 25,000 “skills” built by third-party developers. Google Assistant has more than 1 million “actions,” which include commands written by Google (such as navigation through Google Maps) as well as features developed by third parties (Google doesn’t disclose how many third parties have developed). These competing platforms are also quickly catching up to Apple in usage: Google announced in January that Assistant is used on 400 million devices, and Amazon said it sold “tens of millions” of Alexa-based devices during the holiday season alone.

“When Alexa shipped, sales were very solid from day one,” said a former Siri team member. “For Siri, they had to start from scratch to get to that point.”

Footprints in the Universe

Siri was founded in 2007 by Stanford Research Institute, a nonprofit research organization in Menlo Park. Siri was originally conceived as an intelligent digital voice assistant: one that responds to user requests and interacts with users to get the answers they need. For example, ideally, Siri would not just give a restaurant phone number, but would communicate with the user to confirm the details, such as the date, time and number of party members, and then directly book the restaurant.

Siri's designers believe that the only way to solve this problem is to create an open platform that allows external developers to participate in the development of Siri, so that Siri can have far more functions than if it were developed behind closed doors.

“This is about building applications for artificial intelligence,” said one Siri executive. “It should be a way to orchestrate the internet through conversation.”

Norman Winarsky, Siri founder and board member, said Jobs was a big believer in Siri's potential. One day in 2010, Jobs invited Siri CEO Dag Kittlaus and the company's three co-founders to meet at his home. After a few hours, Jobs finally convinced the team to abandon other partners and devote themselves to Apple, and convinced them that they would achieve greater goals with Siri.

“Steve Jobs was a huge inspiration to the team,” Winarsky said of the Siri co-founders. “He made all of them believe that they could make their own mark in the universe.”

In April 2010, Apple acquired Siri for more than $200 million. At the time, Siri had only 24 employees, but Apple quickly expanded Siri's core team talent. 18 months later, when Siri was launched with the iPhone 4s, Siri's team had grown to nearly 100 people.

After the acquisition, Jobs still kept a close eye on Siri. Former employees of the team said that Jobs would attend a meeting every week and give his opinions on Siri's upcoming features. Scott Forstall, who was Apple's senior vice president at the time, was also very supportive of the Siri team. Forstall had a strong insight into what the product could do and liked to get his hands dirty. One former employee said that he would study technology every day and often told everyone when he attended a team meeting: "I discovered 10 more things last night."

To ensure Siri was released on time, the team made some compromises on Siri's original vision, and the "AI App Store" plan had to be shelved. Jobs wanted Siri to focus on some technologies that only Apple could fully control, but early Siri members said that Jobs promised to start building a third-party ecosystem later.

On October 4, 2011, the day after Siri was released, Jobs died of long-term cancer. Siri members said that Jobs' death basically marked the end of Siri's initial plan.

"The original plan for Siri fell through when Steve Jobs died the day after Siri was released," the former employee recalled. "They lost their big picture direction."

The beginning of chaos

After this, the Siri team still had Forstall's support, but Forstall's attention was also distracted by other major projects, including the upcoming Apple Maps. Forstall appointed Richard Williamson, a deputy from the Apple Maps team, to lead the Siri team in an attempt to get everything back on track.

Several former employees said that many decisions made by Williamson were opposed by other members, including only upgrading Siri once a year. This is Apple's common strategy for iOS system upgrades, and Williamson's previous job was to make the software run on the phone from the back-end server update. Siri members said that this model is wrong, Siri should be an online service that is constantly improving, not just updated once a year. Although the server software has made many updates in terms of stability and performance, there were no changes to the Siri architecture in the first year.

Williamson wrote in an email to a request for comment that he thought it was "completely unrealistic" that Siri wasn't being updated consistently, that decisions about the "technical aspects of the software and server infrastructure" were made by his staff, and that he was simply responsible for getting the team back on track.

“The Siri launch was a disaster,” Williamson wrote. “It was slow and had a lot of serious bugs, which was entirely the fault of the old Siri team, not me.”

After the announcement, Kittlaus responded: "This announcement is complete nonsense and the biggest disaster in Apple's history! And all of this is caused by Williamson, the head of Apple Maps. In fact, Siri performed very well at launch, but like any new platform, in the case of unexpected large-scale load, load scalability and 24-hour uninterrupted maintenance are required."

When Siri was first released, the number of users surged beyond expectations and the software began to crash. Engineers were busy keeping the backend running, but the infrastructure simply couldn't handle the massive influx of data. An employee assigned to the infrastructure recalled that 500 servers were initially needed to handle Siri tasks, but after improvements, it was reduced to 5 servers.

“At the time, it was like a war,” recalled another former employee.

Luc Julia, who had worked with Siri co-founder Adam Cheyer at SRI in the 1990s, was named head of the Siri team. Several former Siri employees enjoyed working under Julia, but Williamson wrote that Julia clashed with some people at Apple, including Forstall. When Forstall pressured Julia to leave Siri, Williamson did so. After less than a year of running the Siri team, Julia left.

Williamson wrote in the email that he tried to get the team to implement the SiriKit project to allow outside developers to improve Siri's functionality, but was resisted by the Siri team because Siri's "initial software was very fragile and rigid."

Disagreements between Williamson and the Siri team led to the departure of co-founder Cheyer, one of the Siri team's most loyal supporters, in June 2012. Cheyer left soon after co-founder Kittlaus left a year later, which in turn led to several other founding members of the Siri team leaving the project.

At the end of the year, Apple fired Forstall and Williamson when a catastrophic error occurred in the Apple Maps iOS 6 system. Former Siri employees said they felt sorry for losing Forstall, who always supported their work.

Turn to search

Shortly after Apple acquired Siri, Jobs expressed his unique insights.

In June 2010, Jobs said at the Wall Street Journal's All Things Digital technology conference: "Siri has no intention of getting involved in the search field. They belong to the field of artificial intelligence... We have no plans to enter the search field. This is not an area that we are interested in. Others do it better in this area."

However, a high-quality search engine is still key to digital voice assistants. When users ask questions, the AI ​​needs to access the knowledge base and respond quickly and correctly.

Jobs's view expressed the thoughts of many founding members of the Siri team: Siri will be more powerful than search. One Siri member said that their ideal setting for Siri is similar to "her" in the 2013 Spike Jonze movie, in which Joaquin Phoenix played a lonely man who fell in love with the operating system he was familiar with, "Samantha."

Employees who share this view say they were immediately skeptical when a search expert like Stasior was tapped to lead Siri. Stasior, who has a Ph.D. and a degree in computer science from MIT, served as Amazon's president and CEO from 2006 to 2012 and helped found Amazon's search division, A9.com.

Siri team members believed that Stasior had no background in speech or natural language processing and could not successfully lead the team.

“I realized that even though he was particularly interested in Siri, his ultimate goal was to build a search engine,” said one of Stasior’s employees. “Siri was just a stepping stone.”

When Stasior first joined Apple, its search engine functionality was scattered across the company's various applications, with a lot of duplication. People who worked at the time said Stasior's main job was to integrate Siri with Apple's existing search functions to improve Siri's core performance. He also worked to strengthen Apple's influence in the field of search technology, including web search, media search and Wikipedia data classification.

"Stasior's job is to take search and make it a core asset and a top product at Apple," said a search team employee.

Stasior worked to facilitate a series of acquisitions at Apple to bolster the growth of search talent and technology within Apple. In October 2013, the team acquired Cue, a company that built a personal assistant app that curates a user's calendar through their email, for more than $40 million, according to a TechCrunch report.

Stasior acquired Topsy for more than $200 million in 2013. Topsy technology is used in Spotlight, Apple's built-in search feature for MacOS and iOS, which was first introduced in 2004.

"When Bill sees something good, he'll take control," said an Apple employee who oversees acquisitions. "His goal is to acquire and retain as much great search talent as possible."

The Topsy team eventually grew into a large organization under Stasior, with the number of employees now almost matching that of the Siri team, a former employee said. Topsy CEO Vipul Ved Prakash will continue to lead the search team and report directly to Stasior.

Combining the existing Siri team with Stasior's search team turned out to be a mistake. The Topsy team was reluctant to work with the Siri team, believing that the Siri team was moving slowly and that, while the initial infrastructure had been patched up, it had not been fully replaced since launch.

"I had this idea: Why don't we just start over and build what we need to build and then worry about coordinating the two?" said the former search team member. "But they're still looking at coordinating the two."

The core technology of Siri and Spotlight is a combination of Topsy technology and Siri Data Services, which was based on iTunes' old search technology, improved for Siri and launched in 2013, the former employee said. Siri Data Services handles things like Wikipedia, stocks and movie showtimes, while Topsy sorts Twitter, news and web search results. Prakash planned to integrate all the technologies into one stack, eventually folding the Siri Data Services team into the Topsy team. But because the two are based on different programming languages, it was difficult to coordinate.

The difficulty in integrating the search teams led to some embarrassing consequences: users using Siri or Spotlight would get two completely different answers, each built on the basis of two different search technologies.

Turf Wars

Siri's initial technology used natural language processing to understand the user's intent. Speech processing is software that can detect the user's language. Initially, Apple outsourced the technology to Boston-based Nuance. Later, under Stasior's leadership, Apple began to focus on creating its own speech recognition software.

In 2013, Apple also hired Alex Acero, an experienced Microsoft speech recognition expert, to lead a new speech recognition team. In the same year, Apple also acquired a speech startup, Novaris Technologies, to accumulate expertise in speech technology. Apple plans to combine the new speech recognition technology with Acero's Siri core technology in the hope of improving Siri's conversational performance. The Siri team believes that this technical combination is important and is likely to greatly promote the development of Siri.

After working together for six months, the team was disbanded in late 2014. Acero continued to lead the speech recognition team, while the natural language team was led by Anoop Sinha, a management consultant who previously worked at global consulting firm McKinsey & Company.

Former employees interviewed by The Information said that before this, senior management had never given any explanation for the dissolution, and if this really happened, Sinha would have his own management style.

Several members of the Siri team resented Sinha, who had no background in natural language processing and whose decisions seemed to stem from office politics rather than scientific research.

One former employee recalled one meeting in which Sinha and Acero openly confronted each other, arguing over project allocations and pressuring Stasior to make the final decision. Tensions between the two factions prompted another wave of departures, including experienced speech scientists Gokhan Tur, Chuck Wooters, Tom Kollar and Larry Gillick.

Sinha's LinkedIn profile shows that he left Apple in 2016 and is currently an engineering manager at Facebook. He did not respond to requests for comment.

Many long-time Siri employees joined Kittlaus and Cheyer to form a new company, Viv Labs, and continue to work on building a better digital voice assistant. Stasior was furious. "Can they be trusted?" one former employee recalled.

After learning that Kittlaus and Cheyer continued to attend basketball games between Apple and their former Siri team, Stasior began to worry that they were poaching them. Soon after, the two were banned from Apple's campus, according to Siri employees who were present at the time.

The rise of a powerful rival

In order to make Siri more powerful, Stasior continued to make large-scale acquisitions. In October 2015, he led the acquisition of VocalIQ, a British artificial intelligence startup dedicated to the research of dynamic learning systems.

A former employee of the VocalIQ team said that they believe Siri is actually a "manual operating system" and believe that VocalIQ technology is very helpful for the development of Siri. VocalIQ technology can extract important information from voice interaction and analyze it, constantly adjusting the accuracy of the system. According to relevant sources, Apple has successfully integrated VocalIQ technology into the Siri calendar function.

In addition, under the leadership of former employee Rushin Shah, more automated machine learning technology has been integrated into Siri's domain selection, such as determining whether a request needs to be sent or shared. Rushin Shah's profile on LinkedIn shows that Shah left Apple in 2017 and is currently a senior manager in Facebook's natural language understanding team.

Meanwhile, Apple has ceded its three-year lead in voice assistants to Amazon and Google. In June 2017, The Wall Street Journal wrote that HomePod “is at best a third-place finisher in the home speaker market.”

Two Siri team employees told The Information that in order to cope with this competition, the Siri team did not know about Apple's HomePod project until 2015, or when Amazon's Echo came out at the end of 2014. According to people familiar with the matter, Apple originally planned to launch an ordinary speaker without Siri.

Apple's sudden decision to include Siri in the audio project forced the Siri speech recognition team to deal with weekly meetings with Beats, which Apple acquired for $3 billion two years ago. Chuck Wooters, a former Siri employee, mentioned that in the early meetings, about 25 people participated, including several Beats employees who attended via video.

The Beats team worked on microphone noise reduction and beamforming algorithms to better pick up users' voice commands. In apartments around the Bay Area, Apple installed microphones to collect data and simulate home environments, which they call "room impulse responses."

However, Siri still failed to build a third-party development ecosystem, which was a key element of Siri's original conception. Former employees who have worked on multiple developer kit projects said that after years of shelving the project, Apple finally launched SiriKit in 2016. However, according to multiple people involved, Apple has been working on building a developer kit since 2012.

Apple eventually appointed Vineet Khosla, a member of the original Siri team, to manage the SiriKit project, which connects apps developed by outside developers to Siri. Apple's intention seems to be to enhance Siri's functionality so that it no longer relies on a limited number of local apps, such as making calls or sending messages.

But SiriKit has yet to deliver on its promise. So far, only 10 features are available, such as payments, booking rides, setting to-dos and viewing photos. Some of SiriKit's top engineers have either left the company or left the project. Although Khosla is still at Apple, he is no longer leading the SiriKit team, which is now led by Robby Walker, the current head of Siri, according to people familiar with the matter.

Some former Siri team members pointed out that although Apple has tried to transform itself into a service-oriented company, its core is still product design. Inside the company, the idea of ​​creating a perfect online service is far less strong than the desire to design a revolutionary product, which can also cause many thorny problems like the project managed by Stasior.

"The structure of Apple was so antithetical to what people were trying to do, and one of the things Bill had to do as much as he could was fight that mentality," said one former employee.