objc.io#21#Photo framework

introduce

Every day, more photos are taken with an iPhone than with any other camera. Displays on iOS devices get better every year, and back when the iPad first came out without a Retina display, one of the killer features of the big screen was the ability to display your photos and browse your photo library. Ever since the camera became the most important and powerful feature of the iPhone, there has been a huge demand for apps and tools that can manage and process the precious photos in your photo library.

Until the summer of 2014, the only way for developers to access a user's growing photo library was with the AssetsLibrary framework. Over the years, the Camera and Photos apps have evolved significantly, adding many new features, including ways to organize photos by moment. But the AssetsLibrary framework has not kept pace.

With the arrival of iOS 8, Apple provided us with a modern framework - PhotoKit, which performs better than AssetsLibrary and has features that make apps and device photo libraries work seamlessly.

summary

We'll start with a bird's-eye view of the framework's object model: entities and the relationships between them, getting instances of entities, and working with the results.

In addition, our explanation will also involve some resource metadata that is not yet open to developers when using AssetsLibrary.

We'll then discuss loading image data from a resource: the process itself, the myriad of options available, and some gotchas and edge cases.

Finally, we'll talk about observing changes to the photo library through external actors, and learn how to create and submit our own changes.

PhotoKit Object Model

PhotoKit defines an entity graph that is consistent with the model objects presented to users in the system's Photos app. These photo entities are lightweight, immutable objects. All PhotoKit objects inherit from the PHObject abstract base class, whose public interface provides only a localIdentifier property.

PHAsset represents a single resource in the user's photo library and is used to provide metadata about the resource.

Grouped resources are called resource collections, represented by the PHAssetCollection class. A single resource collection can be an album or a moment in the photo library, or a special "smart album". This smart album includes all video collections, recently added items, user favorites, all burst photos, and so on. PHAssetCollection is a subclass of PHCollection.

PHCollectionList represents a set of PHCollections. Because it is itself a PHCollection, a collection list can contain other collection lists, which allows complex collection inheritance. In fact, we can see it in the Moments section of the Photos app: Photos --- Moments --- Featured --- Year, is an example.

Fetch the photo entity

Get vs. Enumerate

Developers who are familiar with the AssetsLibrary framework may remember that AssetsLibrary can use some specific properties to find the required resources, one of which must enumerate the user's resource library to obtain matching resources. It must be admitted that although this API provides some methods to narrow the search domain, it is still very inefficient.

In stark contrast, instances of PhotoKit entities are obtained through retrieval. Those familiar with Core Data will find it closer to PhotoKit in terms of concept and description.

Get Request

Fetching operations are implemented by class methods of the entities described above. Which class/method to use depends on the scope of the problem and the way you want to display and traverse the photo library. All fetch methods are named similarly: class func fetchXXX(..., options: PHFetchOptions) -> PHFetchResult . The options parameter gives us a way to filter and sort the results, similar to the predicate and sortDescriptors parameters of NSFetchRequest .

Get the results

You may have noticed that these fetch operations are not asynchronous. They return a PHFetchResult object, which can be accessed with an NSArray-like interface. It dynamically loads content on demand and caches the most recently requested content. This behavior is similar to the result array returned by NSFetchRequest with the batchSize property set. For PHFetchResult, there is no way to specify this behavior with parameters, but the official documentation guarantees that "even when dealing with a large number of returned results, it can still have excellent performance."

Even if the content of the photo library that satisfies the request changes, the PHFetchResult object returned by the acquisition method will not be automatically updated. In the following sections, we will introduce how to observe changes in the returned PHFetchResult object and handle the updated content.

Transient Collections

You may find that you’ve designed a component that operates on a collection of assets, and you want it to be able to work with any set of assets. PhotoKit makes this easy with temporary asset collections.

You can create a transient asset collection from an array of PHAsset objects or a PHFetchResult object containing assets. The creation operation is completed in the transientAssetCollectionWithAssets(...) and transientAssetCollectionWithFetchResult(...) factory methods of PHAssetCollection. The objects created by these methods can be used like other PHAssetCollection objects. However, these collections will not be stored in the user's photo library and will naturally not be displayed in the Photos application.

Similar to resource collections, you can use the transientCollectionListWithXXX(...) factory methods in PHCollectionList to create transient collection lists.

This is useful when you want to merge two fetch requests.

Photo metadata

As mentioned at the beginning of the article, PhotoKit provides additional metadata about user assets that was previously inaccessible or difficult to access using the ALAssetsLibrary framework.

HDR and Panoramic Photos

You can use the mediaSubtypes property of a photo asset to verify that images in your library were captured with HDR enabled and using the camera app's panorama mode.

Favorite and hide resources

To verify whether an asset has been marked as a favorite or hidden by the user, simply check the favorite and hidden properties of the PHAsset instance.

Burst mode photos

For an asset, if the representsBurst property of its PHAsset is true, it means that this asset is a representative photo in a series of burst photos (multiple photos are taken when the user holds down the shutter). It also has a burstIdentifier property. If you want to get the remaining photos in the burst photos, you can get them by passing this value to the fetchAssetsWithBurstIdentifier(...) method.

Users can mark photos in a burst; in addition, the system automatically uses various heuristics to mark potential representative photos that the user might select. This metadata is accessible through the burstSelectionTypes property of PHAsset. This property is a bitmask composed of three constants: .UserPick for resources manually marked by the user, .AutoPick for potential resources that the user might mark, and .None for unmarked resources.

This screenshot shows how the Photos app automatically tags potential assets that the user might tag in a burst of photos.

Photo loading

Over the past few years of working with user photo libraries, developers have created hundreds (if not thousands) of tricks to make loading and displaying photos more efficient. These tricks deal with dispatching and canceling requests, resizing and cropping images, caching, and more. PhotoKit provides a class that does all of this with a more convenient and modern API: PHImageManager .

Requesting images

Image requests are dispatched through the requestImageForAsset(...) method. This method accepts a PHAsset, the size of the returned image and other optional image options (set through the PHImageRequestOptions parameter object), and a result handler. The return value of this method can be used to cancel the request when the requested data is no longer needed.

Image Sizing and Cropping

Oddly, the parameters for defining the size and cropping of the returned image are distributed in two places. The two parameters targetSize and contentMode are passed directly to the requestImageForAsset(...) method. This content Mode is similar to the contentMode parameter of UIView, and determines whether the photo should be scaled or filled proportionally to the target size. Note: If the photo is not resized or cropped, the method parameters are PHImageManagerMaximumSize and PHImageContentMode.Default.

In addition, PHImageRequestOptions also provides some ways to determine how the image manager should resize the image. The resizeMode property can be set to .Exact (the returned image must match the target size), .Fast (more efficient than .Exact, but the returned image may not be the same size as the target) or .None. It is also worth mentioning that the normalizedCroppingMode property allows us to determine how the image manager should crop the image. Note: If the normalizedcroppingMode value is set, then resizeMode needs to be set to .Exact.

Request delivery and progress

By default, if the image manager decides to use an opportunistic policy, it will deliver a lower-quality version of the image before delivering the higher-quality version to you. You can control this behavior with the deliveryMode property; the default behavior described above is .Opportunistic. If you only want high-quality images and are okay with a longer load time, set the property to .HighQualityFormat. If you want faster load times and can sacrifice a little image quality, set the property to .FastFormat.

You can use the synchronous property of PHImageRequestOptions to make the requestImage... series of methods synchronous operations. Note: When synchronous is set to true, the deliveryMode property will be ignored and treated as .HighQualityFormat.

When setting these parameters, it's important to consider that some of your users may have iCloud Photo Library turned on. The PhotoKit API doesn't necessarily distinguish between device photos and iCloud photos - they're both loaded using the same requestImage method. This means that any image request will potentially be a very slow network request over a cellular network. Keep this in mind when using .HighQualityFormat or making a synchronous request. Note: If you want to ensure that the request doesn't go over the network, set networkAccessAllowed to false.

Another iCloud-related property is progressHandler. You can set it to a PHAssetImageProgressHandler block, which will be automatically called by the image manager when downloading photos from iCloud.

Resource Version

PhotoKit allows apps to make non-destructive modifications to photos. For edited photos, the system keeps a separate copy of the original photo and the adjustments made to the app. When retrieving resources using the image manager, you can specify which version of the image resource should be delivered through the result handler. This can be done by setting the version property: .Current delivers the image with all adjustments and modifications; .Unadjusted delivers the image without any modifications applied; and .Original delivers the image in the original, highest quality format (for example, RAW format data. When the property is set to .Unadjusted, a JPEG is delivered).

You can read more about this aspect of the framework in Sam Davies' article Photo Extensions.

Result handler

The result callback is a block that contains a UIImage variable and an info dictionary as parameters. It can be called multiple times by the image manager throughout the life cycle of the request, depending on the parameters and options of the request.

The info dictionary provides information about the current request status, such as:

Whether the image must be requested from iCloud (if you set networkAccessAllowed to false during initialization, the image must be re-requested) - PHImageResultIsInCloudKey.

Whether the currently delivered UIImage is a degraded format of the final result. This allows you to show a preview image to the user while the high-quality image is being downloaded - PHImageResultIsDegradedKey.

The request ID (which can be used to easily cancel a request), and whether the request has been canceled - PHImageResultRequestIDKey and PHImageCancelledKey.

If no image was provided to the result handler, there will also be an error message in the dictionary - PHImageErrorKey.

These values ​​allow you to update your UI to inform the user, and work with the progressHandler discussed above to indicate their loading status.

cache

When images are about to be displayed on the screen, such as when displaying a large number of resource image thumbnails in a scrolling collection view, it is sometimes very useful to pre-load some images into memory. PhotoKit provides a subclass of PHImageManager to handle this specific usage scenario - PHImageCachingManager.

PHImageCachingManager provides a key method - startCachingImagesForAssets(...). You pass in an array of PHAssets, some request parameters, and some options that will be used when requesting a single image. In addition, there are methods that allow you to notify the cache manager to stop caching a specific list of assets, as well as to stop caching all images.

The allowsCachingHighQualityImages property lets you specify whether the image manager should prepare high-quality images. The default true property works well when caching a short and unchanging list of resources. But it is best to set it to false when caching while swiping quickly on a collection view.

Note: In my experience, using a cache manager can hurt scrolling performance when the user is scrolling extremely quickly on a resource-heavy collection view. It is extremely important to tailor a caching behavior for this specific use case. The size of the cache window, when and how often to move the cache window, the value of the allowsCachingHighQualityImages property - these parameters should all be carefully tuned on a real photo library on the target hardware and tested for performance. Furthermore, you can consider setting these parameters dynamically based on user behavior.

Requesting image data

***, in addition to requesting a normal UIImage, PHImageManager provides another method that can return resource data of NSData object type, including its universal type identifier and the display direction of the image. This method returns the most information about this resource.

Everything changes

We've discussed requesting metadata for resources in the user's photo library, but so far we haven't mentioned how to update the data we retrieve. The photo library is essentially a bunch of mutable state, while the photo entities mentioned in Section 1 are immutable objects. PhotoKit allows you to receive all the information you need about changes to the photo library in order to correctly update your cached state.

Observe changes

First, you need to register a change observer (this observer must conform to the PHPhotoLibraryChangeObserver protocol) with the shared PHPhotoLibrary object using the registerChangeObserver(...) method. Whenever another application or the user makes changes to the photo library that affect any resource or collection of resources you retrieved before the change, the change observer's photoLibraryDidChange(...) method will be called. This method has a single parameter of type PHChange, which you can use to verify whether these changes are related to the retrieved object you are interested in.

Update the obtained results

PHChange provides several methods that allow you to track changes by passing in any PHObject or PHFetchResult object you are interested in. These methods are changeDetailsForObject(...) and changeDetailsForFetchResult(...) . If there are no changes, these methods will return nil, otherwise you can use PHObjectChangeDetails or PHFetchResultChangeDetails objects to observe these changes.

PHObjectChangeDetails provides a reference to the original photo entity object, as well as Boolean values ​​that tell you whether the object's image data has changed or whether the object has been deleted.

PHFetchResultChangeDetails encapsulates information about the changes applied to the PHFetchResult you previously obtained by fetching. PHFetchResultChangeDetails is designed to simplify the update operation of a CollectionView or TableView as much as possible. Its properties map exactly to the information you need when using a typical CollectionView update handler. Note that to correctly update the UITableView/UICollectionView, you must handle the changes in the correct order, which is: RICE - removedIndexes, insertedIndexes, changedIndexes, enumerateMovesWithBlock (if hasMoves is true). In addition, the hasIncrementalChanges property of PHFetchResultChangeDetails can be set to false, which means that the old fetch results should be completely replaced by the new values. In this case, you should call reloadData of UITableView/UICollectionView.

Note: It is not necessary to handle changes in a centralized manner. If multiple components in your app need to handle photo entities, then each of them should have its own PHPhotoLibraryChangeObserver. Components can then query the PHChange object on their own to detect whether (and how) they need to update their own state.

Change with the wind

Now that we know how to observe changes made by users and other applications, let's try making changes ourselves.

Changing existing objects

Making changes to the photo library with PhotoKit is ultimately about creating a change request object that is linked to a resource or collection of resources, then setting the relevant properties of the request object or calling the appropriate method to describe the changes you want to submit. This must be done in the block submitted to the shared PHPhotoLibrary through the performChanges(...) method. Note: You need to be prepared to handle failures in the completion block of the performChanges method. Although it deals with states that can be changed by multiple participants (such as your app, users, other apps, photo extensions, etc.), this approach provides security and is relatively easy to use.

To modify an asset, you need to create a PHAssetChangeRequest. You can then modify the creation date, asset location, whether to hide the asset, whether to treat the asset as a user favorite, etc. In addition, you can also delete the asset from the user's library.

Similarly, to modify an asset collection or collection list, you need to create a PHAssetCollectionChangeRequest or PHCollectionListChangeRequest object. You can then modify the collection title, add or remove collection members, or delete the collection entirely.

Before your changes are committed to the user's photo library, the system will display an explicit permission warning box to the user.

Creating New Objects

Creating a new asset is similar to modifying an existing one. Just use the creationRequestForAssetFromXXX(...) factory method to create a change request and pass in the asset image data (or a URL). If you need to make additional changes to the newly created asset, you can use the placeholderForCreatedAsset property of the creation change request. It will return a usable placeholder in place of the "real" PHAsset reference.

in conclusion

I’ve discussed the basics of PhotoKit, but there’s still so much more to discover. You can learn more by looking at the code throughout the examples, watching the WWDC session videos, digging deeper, and then writing some code of your own! PhotoKit opens up a world of new possibilities for iOS developers, and we’re sure to see many more creative and exciting products built on this foundation in the months and years ahead.