iOS10 SiriKit QQ Adaptation Detailed Explanation

1. Overview

Apple opened the SiriKit interface to third-party applications in iOS 10. Currently, QQ has taken the lead in adapting Siri's messaging and calling functions. This means that in iOS 10 you can directly tell Siri to help you send QQ messages and make QQ calls. Doesn't it sound cool?

So what is the experience of using Siri with third-party applications? Which applications can be connected to SiriKit? What work is needed to connect to SiriKit? This article will answer these questions for you one by one.

Figure 1 The effect of sending QQ messages using Siri

2. Introduction to SiriKit

We all know that Siri is the smart voice assistant in iPhone, so what is SiriKit? SiriKit is a development framework provided by Apple for third-party applications to support Siri. In the official documentation, SiriKit divides voice support for different scenarios into different domains. Currently, the domains supported by SiriKit include: VoIP calls, messaging, money transfers, image searches, online car-hailing, CarPlay, and restaurant reservations. In other words, if your application contains one of these functions, you can consider integrating these functions into SiriKit.

When implementing SiriKit related functions, we do not need to actually recognize the voice, as Siri will do the voice recognition work. After Siri recognizes the voice, it will abstract the function to be completed by the voice into an Intent object and pass it to us. Our access work is mainly to deal with these Intent objects, and does not involve natural language processing (NLP) technology.

There are already some articles about SiriKit development on the Internet. You can also refer to Apple's official document SiriKit Programming Guide. This article focuses on the adaptation experience of QQ.

Figure 2 SiriKit Principle

3. SiriKit access

To implement SiriKit functions, you need to add the Intents Extension target in the Xcode project. Like other extensions, Intents Extension is a plug-in that runs independently of the Containing App process and is mainly used to process and confirm intent requests from Siri. If you want Siri to provide some customized interfaces when processing App-related intents, you need to add the Intents UI Extension target. Intents UI Extension is also an independently running plug-in (so to fully support SiriKit, you actually need to add two targets, which is a bit painful). For the development of App Extension, please refer to Apple's App Extension Programming Guide.

Let's take the message sending function in QQ as an example to explain how to access SiriKit:

First, we need to configure the Siri Intents we need to support in the info.plist file of Intents Extentsion, add INSendMessageIntent to IntentsSupported, and if you need to disable a certain function when the screen is locked, add the corresponding Intent to IntentsRestrictedWhileLocked, as shown in Figure 3.

Figure 3 Intent Extentsion info.plist configuration

SiriKit access is mainly divided into two parts: Intents Extension and Intents UI Extension, which are introduced below.

Intents Extension

When we say to Siri, "Use QQ to send a message to Wang Yiran to say hello," Siri will automatically recognize the voice and display the recognized content on the Siri interface. As shown in Figure 4, we can see that a complete message sentence consists of four parts:

App name: Tell Siri which app to use. Siri will automatically identify the app name based on the app's bundle displayname without the need for additional registration.

Message sending intent: Tell Siri to use the message sending function. We found that the message sending function can also be recognized through actual testing. However, Apple did not specify in the document which other words will be recognized as the message sending intent.

Message recipient: Tell Siri who the recipient of the message is. "Wang Yiran" is the nickname of my QQ friend.

Message content: Tell Siri what the message you want to send is. Here the message content is "I am angry."

Figure 4 Confirm message sending interface

The application name and Intent are required, otherwise Siri cannot abstract your "Intent". If the latter two items are missing, we can ask the user to provide further data or ignore them in the implementation. After the recognition is completed, Siri will abstract the message content and the recipient into an INSendMessageIntent and pass it to QQ's Intent Extension.

We can also see from Figure 4 that Siri accurately identified my QQ friend nicknamed "Wang Yiran" from my voice. However, "Wang Yiran" is not a common phrase. So how is this done? The secret lies in that when QQ is running, we synchronize the nicknames of all QQ friends to the Siri cloud, so that Siri can recognize the specific phrases that specific users want to use. For detailed synchronization methods, please refer to the setVocabularyStrings:ofType: method of INVocabulary.

Each domain function has a corresponding Intents in Siri, and each Intents corresponds to a specific handler protocol. For sending messages, the corresponding Intent and handler protocols are INSendMessageIntent and INSendMessageIntentHandling respectively. Just implement the relevant methods in the INSendMessageIntentHandling protocol, and use our INSendMessageIntentHandling object to handle the relevant message sending request when Siri parses the INSendMessageIntent request. The specific process is shown in Figure 5:

Figure 5 Siri sending QQ message process

1)ResolveRecipientsForSendMessage

Process and confirm the recipient name passed by Siri from the Intent, for example, you can confirm whether the name is currently in the QQ friend list, and feedback the resolution result to Siri. The resolution result represents the result of the application's processing of the intent. For sending messages, Table 1 lists several possible resolution results.

Table 1 send resolution result

2)ResolveContent

Similar to the receiver's processing, this method can "modify" the message content recognized by Siri and feed back the resolution result to Siri. For example, QQ has adapted emojis for special words in some messages, such as "angry".

3)ConfirmSendMessage

This method is used to confirm whether to send the message. You can perform some authentication work at this step. After the authentication is passed, confirm the sending, otherwise cancel it. After confirming that it can be sent, the confirmation sending interface will be called up, as shown in Figure 4. If you need to share data from the containing app, refer to the shared container of the app group for the specific implementation solution.

4)HandleSendMessage

As shown in Figure 4, when the user clicks the "Send" button or gives a sending instruction by voice, he will eventually enter this method. In this method, we need to implement the logic of sending messages. After the message is sent successfully, we can call up the interface of successful message sending, as shown in Figure 6.

Figure 6 Message sent successfully interface

Intents UI Extension

For Intent types that support custom interfaces, you can provide a more beautiful custom interface in Intents UI Extension. The implementation of Custom UI is relatively simple, and is implemented through the subclass of UIViewController, just like the development of iOS apps. We need to set the initial viewcontroller or main storyboard in the info.plist file of Intents UI Extension, and use Child Viewcontrollers to achieve differentiated interface display for different types of Intent interfaces.

As shown in Figure 7, when receiving a response from the Intents Extension, the system will call up the Intents UI Extension and load the initial viewcontroller. The configureWithInteraction:context:completion: method of the INUIHostedViewSiriProviding protocol can be used to obtain the intent. For example, in the message sending function, this method will be called back once after the message is confirmed to be sent and sent successfully. According to the type and state of the Intent object, when receiving the callback of the relevant Intent, the corresponding Child Viewcontroller can be presented to realize the customized interface display.

It should be noted here that the process of Intents UI Extension does not exit after the interface is destroyed. It is likely to be in a dormant state in the background and will be awakened when the next response arrives.

Figure 7 Life cycle of an Intents UI extension

4. Conclusion

In general, although Apple has opened limited scenarios for SiriKit this time, from our adaptation experience, we can see that Apple still attaches great importance to Siri. In addition, this is the first time that SiriKit has opened its interface to third-party applications, so there are inevitably some problems. We did encounter some bugs in SiriKit itself during the development process. Most of the bugs were solved after feedback to Apple, but Siri still has some defects in language recognition, such as the recognition of mixed Chinese and English scenes is still not very good. I look forward to Siri's support for Chinese getting better and better in the future, and I also hope that Siri can open more scenarios for third-party application adaptation.

For more exciting content, please follow bugly’s WeChat public account:

Tencent Bugly is a quality monitoring tool designed for mobile developers, helping developers quickly and conveniently locate online application crashes and solutions. The intelligent merging function helps developers to merge and classify the thousands of crashes reported every day according to the root cause. The daily report will list the crashes that affect the most users. The precise positioning function helps developers locate the problematic code line. Real-time reporting can quickly understand the quality of the application after release. It is compatible with the latest iOS and Android official operating systems. Tencent engineers are using it. Come and join us!