iOS 9 multitasking split screen tips

iOS 9 Multitasking Overview

The most eye-catching new feature in iOS 9 is multitasking. Similar plug-ins have appeared in jailbreak development a long time ago, and the Windows Surface series has also had split-screen multitasking features, allowing users to use two or more apps at the same time. Similar features have also been added to iOS 9. Multitasking in iOS 9 has three forms: Slide Over, which temporarily appears and interacts, Split View, which is a true split-screen mode that allows two apps to be operated simultaneously, and Picture in Picture mode, which allows video playback in other apps.

In their documentation on multitasking, Apple clearly states:

Most apps should be compatible with Slide Over and Split View

Because this is one of the core features of iOS 9 and what your users expect to see. On the other hand, supporting multitasking also increases the likelihood that your users will open and use your app. However, multitasking has a limitation. Among iPad devices that can install iOS 9, only the most powerful iPad Air 2 and later models support split view mode, while others such as iPad mini 2, iPad mini 3 and iPad Air only support slide overlay and picture-in-picture modes. This should be a decision made based on the resource and performance limitations of mobile devices to a certain extent. After all, multitasking will only make sense if a good user experience is guaranteed.

For developers, although multiple layouts seem complicated, in fact, if you keep up with Apple's technological pace, it is not very difficult to adapt your iPad app for multitasking. Because the layout used in the sliding overlay mode and split view mode is actually the Compact Width layout, and this concept is the UI layout method based on screen features introduced at WWDC14. If you are already using this layout method, then it can be said that the support for multitasking view is also automatically completed. However, if you have never used or even heard of this layout method, a note I wrote last year may help you have a preliminary understanding of it. In the next section, I will also review the relevant concepts and basic usage.

Adaptive UI Review

Adaptive UI is a concept proposed by Apple in iOS 8. Before that, if we wanted to develop apps for both iPhone and iPad, we would probably write a lot of device judgment code, such as this:

 if UI_USER_INTERFACE_IDIOM() == .Pad {
 // The device is an iPad  
 } 
 
 In addition, if we want to adapt to both horizontal and vertical orientations, we will need code similar to this: 
 
 if UIInterfaceOrientationIsPortrait(orientation) {
 // The screen is vertical  
 }

These judgments and branches are not only difficult to write and read, but also make adaptation development difficult. After iOS 8, developers should no longer rely on such devices to adapt the UI, but should switch to the new Size Class system. Apple distinguishes its mobile devices by size, and designs a combination of Regular and Compact in both vertical and horizontal directions. For example, the width of the iPhone is Compact and the height is Regular in portrait mode, the width of the iPhone 6 Plus is Regular and the height is Compact in landscape mode, and the height and width of other iPhones are Compact in landscape mode; the width and height of the iPad are Regular regardless of the model and orientation. The Size Class of existing devices is shown in the figure below:

With the idea of ​​developing for size classes, we no longer care about the model or size of the specific device, but display content according to the characteristics of a specific size class. With a regular width, we can display more content horizontally, such as displaying the Master and Detail View Controllers at the same time. Similarly, we should no longer care about the problem of device rotation, but instead care about changes in the size class. When developing, if you use Interface Builder, when making the UI, pay attention to configuring appropriate constraints and layouts for different size classes, which is enough in most cases. If you use code, the UITraitCollection class will be the key to using and operating size classes. We can set the appropriate layout based on the traitCollection property of the current working UIViewController, and respond correctly to the UI layout when -willTransitionToTraitCollection:withTransitionCoordinator: and -viewWillTransitionToSize:withTransitionCoordinator: are called.

Although it is not theoretically impossible, it would be extremely painful to operate Size Class purely by hand. We should still use IB as much as possible to reduce this part of the workload and speed up development efficiency.

Multitasking on iPad

For multitasking, slide overlays, and the initial position of split views in iOS 9, the size of the newly opened app will be 1/3 of the device size. However, this ratio is not important. What we need to remember is that the newly opened app will run under the Size Class of Compact Width and Regular Height. In other words, if your iPad app uses Size Class for layout and supports iPhone portrait, then congratulations, you just need to switch to iOS 9 SDK and recompile your app, and you're done.

Since the focus of this article is not to teach you how to develop an Adaptive UI app, I don't intend to go into depth in this area. If you missed the course last year and don't know much about this, this tutorial may help you quickly understand and master these contents. If you want to get started directly and see how multitasking works in iOS 9, you can create a new Master-Detail Application and install it on the iPad simulator. The Master-Detail template project provides us with a good framework for adapting Size Class, so that the project can perform well on any device. You can also observe its performance on iPad with iOS 9.

But not all apps should be adapted to multitasking, such as a game that needs to be played in full screen. If you don't want your app to be used as a secondary app in multitasking, you can add UIRequiresFullScreen to Info.plist and set it to true.

Easy enough? Yes, to adapt to iPad's multitasking, all you need to do is develop a universal app for all platforms according to the standard process, that's all.

Build your app using the iOS 9 SDK.

Supports all directions and corresponding Size Classes;

Use launch storyboard as the app launch page.

Although there is not much content worth mentioning, there are still some small details that need attention.

Some small details worth noting

In the past, there was no such thing as an app sharing the screen with other apps in the foreground, so UIScreen.bounds and the main window's UIWindow.bounds were basically synonymous. But in the era of multitasking, UIWindow may only be 1/3 or 1/2 the size of the screen. If you have used it to define your view in previous apps, it is necessary to make special processing for multitasking. However, although the sliding overlay and split view are both displayed on the right, their Window's origin is still (0, 0), which also makes it convenient for us to define the view.

The second detail is that the Size Class of iPad UI will change. In the past, no matter vertically or horizontally, the iPad screen size was always Regular in both length and width. But in iOS 9, the situation is different. Your app may be opened as an additional app in multitasking mode, and may be dragged by the user to become a full-screen app during multitasking (the Size Class will change from Compact width to Regular). It is even possible that your app will be used as the main app and become a Compact width app because of user dragging:

In other words, you don't know whether or not your app's Size Class will change, and when it will change, because it's all a result of user operations. Therefore, you must take this into full consideration during development, and strive to present a good effect to users when the size changes. It goes without saying that you should design and adjust the UI appropriately according to the screen size, and you should also pay attention to using transitionCoordinator 's -animateAlongsideTransition: to perform layout animation at the right time to make the switch more natural.

Since multitasking makes it possible for multiple apps to run on the same platform, your app will inevitably face the situation of running with other apps. When developing mobile applications, you must never forget the limitations of the device platform. Compared with desktop devices, mobile devices have limited memory, and two or even three apps are running in the foreground at the same time, which requires us to carefully design the use of memory. For general developers, allocating memory reasonably, listening to Memory Warning to release cache and unnecessary view controllers, avoiding circular references, etc., should become the basic skills of daily development that they are proficient in.

The last detail is a demand for perfection. In iOS 9, multitasking also uses the App Switcher to switch between apps. So when your app is switched to the background, the system will save a screenshot of your app's current state for later switching. Your app may now be used as a Regular full-screen app or a Compact layout, so the system will save two screenshots in sequence when taking screenshots. Users may close your app in full-screen mode and then open your app as an additional app through multitasking. At this time, it is best to ensure that the screenshots in the App Switcher are consistent with the screenshots that users see after the app is opened to get the best experience. Maybe this is not a big problem, but if you pursue the ultimate user experience, it is necessary. For those apps that contain user sensitive data and need to blur the screenshots, you now also need to pay attention to processing both layouts of the screenshots at the same time.

Picture-in-picture mode

Another form of multitasking in iOS 9 is the picture-in-picture mode for videos: even if you exit your video app, it can still play while the user is using another app, such as watching an American TV series while writing a diary or sending an email. This is probably a feature that all video apps must support, and it is also very easy to implement:

Build your app using the iOS 9 SDK.

In the app's Capabilities, check "Audio, AirPlay, and Picture in Picture" under Background Modes (it is currently "Audio and AirPlay" in Xcode 7 beta);

Set the AudioSession Category to the appropriate option, such as AVAudioSessionCategoryPlayback

Use AVKit, AVFoundation or WebKit framework to play videos.

In iOS 9, the video playback part of the MediaPlayer framework that has been with us has officially been declared dead. In other words, if you are using MPMoviePlayerViewController or MPMoviePlayerController to play videos, you will not be able to use the picture-in-picture feature, so it is an urgent adaptation task to transition to the new video playback framework as soon as possible. Because the picture-in-picture mode is based on AVPlayerLayer . When switching to picture-in-picture, the layer that is playing the video will be taken out, then shrunk and added to the layer of the new interface. This is also the main reason why the old MediaPlayer framework cannot support picture-in-picture.

If you use AVPlayerViewController , once these simple conditions are met, you should be able to see the picture-in-picture button in the lower right corner when playing a video in full screen using the corresponding framework. Whether you click this button to enter picture-in-picture mode or directly use the Home button to switch to the background, the video that is already playing will shrink to the lower right corner of the screen to become a picture-in-picture and keep playing. In picture-in-picture mode, the system will add a set of default controls to the AVPlayerLayer of the video to control pause/resume, close, and return to the app. There is nothing much to say about the first two controls. If you return to the app, you need to handle the operations after returning. Generally speaking, we hope to restore to full-screen mode and continue playing the video. Because we generally do not operate AVPlayerLayer when AVPlayerViewController is playing, we need to implement - playerViewController:restoreUserInterfaceForPictureInPictureStopWithCompletionHandler: in AVPlayerViewControllerDelegate to rebuild the UI according to the passed ViewController, and return true to the system through CompletionHandler to inform the system that the restoration is successful (of course, if it cannot be restored, false needs to be passed).

We can also use AVPlayerLayer directly to build a custom player. In this case, we need to create an AVPictureInPictureController by passing in the AVPlayerLayer used. AVPictureInPictureController provides an API to check whether the picture-in-picture mode is supported, as well as some other methods to control the picture-in-picture behavior. Unlike using AVPlayerViewController directly, when restoring, the system will return the AVPlayerLayer that was reduced during the picture-in-picture to the previous view. We can use the corresponding methods in AVPictureInPictureControllerDelegate to learn about the execution status of the picture-in-picture and restore the UI based on the situation of our own app.

Summarize

After years of planning, based on AutoLayout and Size Class, Apple released the killer feature of multitasking in iOS 9. It can be said that the demand for running multiple apps on the same screen has existed since the first iPad, and now it is finally here. In OS X 10.11, Apple also introduced similar features to the full-screen mode of OSX apps, which can be said to be a further attempt to unify the two platforms of OSX and iOS.

However, there are still some shortcomings in multitasking on iPad. The biggest problem is that apps are still running in the sandbox, which means that we still cannot communicate between two apps on iOS: for example, if we open a photo app and a note app at the same time, we cannot drag a picture directly into the note. Although there are XPC services in iOS, third-party developers cannot use them now, which still limits the possibility of multitasking to a certain extent.

But in general, the multitasking feature greatly increases the practicality of the iPad, and it will definitely be one of the most commonly used and most desired features in apps in the future. It would be a very cost-effective thing to spend some time learning how to make Adaptive UI and make apps support multitasking.