Three major updates to Apple's Core ML framework in 2020: more layer types, model encryption, and CloudKit-based model deployment

This year's Apple WWDC Global Developer Conference brought us software and hardware updates. Among them, Apple's Core ML framework also brought three updates: more layer types, more support for model encryption, and the ability to host model updates on CloudKit.

Mobile machine learning is no longer the hot new thing in 2020. Making applications intelligent has long been an industry standard.

Last year, Apple made a major update to Core ML. This year’s update is much simpler: more layer types, more support for model encryption, and the ability to host model updates on CloudKit.

CoreML is an Apple framework for integrating machine learning models into applications.

The advantage is that Core ML optimizes on-device performance by leveraging the CPU, GPU, and Neural Engine while minimizing its memory footprint and power consumption. Running the model strictly on the user's device eliminates any need for a network connection, which helps keep user data private and apps responsive.

Core ML is also the foundation for several other Apple frameworks and features. Core ML is the core model for vision processing, natural language, speech-to-text, and audio recognition, and Core ML itself is built on low-level primitives such as Accelerate and BNNS and Metal Performance Shaders.

More layer types, bringing more functionality to Core ML

The new layer types are:

The new layer type can only be used in high versions, which means the system must be iOS 14 and macOS 11.0, or higher.

Another useful improvement is the 8-bit quantization operation for InnerProductLayer and BatchedMatMulLayer.

In previous versions of Core ML, users could quantize weights, but the weights would be dequantized to floating point numbers when the model was loaded. With the new int8DynamicQuantize function, weights are kept as 8-bit integer values ​​when loaded, and the actual calculations are also done using integers.

There is a lot of potential for using INT8 for calculations, which could be much faster than floating point operations. But the researchers point out that this is only beneficial for CPUs and not GPUs, as GPUs do prefer floating points. Perhaps an upcoming update to the Neural Engine will support INT8 operations.

Core ML on the CPU can now also use 16-bit floating point operations instead of 32-bit floating point operations (on A11 Bionic and later). 16-bit floats are now a first-class Swift data type. With 16-bit floats natively supported by the CPU, Core ML can be more than twice as fast!

On CloudKit, managed model updates can be implemented

This new Core ML feature lets users update models independently of their apps.

Existing applications do not need to release an application update, but only need to download a new version of the mlmodel file. This is not a new idea, several third-party vendors provide SDKs for this, and it is not difficult to build it yourself.

The advantage of using Apple's solution is that the models are hosted on the Apple Cloud.

You may have multiple models in your app, and a model collection can bundle multiple models together so that when an update is made, the app will update all models at once. You can create these collections on the CloudKit dashboard. In your app, you can also use MLModelCollection to download and manage model updates.

To prepare a Core ML model for deployment, there is a Create Model Archive button in Xcode. This will write a .mlarchive file. The user can upload this model version to the CloudKit dashboard, which will then put it into a model collection. (mlarchive is actually just a zip file containing the contents of the mlmodelc folder.)

Another nice feature is that you can deploy different sets of models to different users. For example, the camera on an iPhone is different than the camera on an iPad, so you might want to create two versions of your model and send one to the iPhone side of the app and another to the iPad side.

Users can create targeting rules for device (iPhone/iPad/TV/Watch), OS and version, region code, language code, and app version.

Deployment of new model versions is not immediate. The app needs to detect at some point that a new model is available and automatically download the model, placing it in the app’s sandbox. But the user has no control over when or how this happens, and Core ML may perform downloads in the background while the phone is not in use.

Therefore, it is better to ship your app using a built-in model as a fallback option, such as a universal model that can handle both iPhone and iPad.

Model encryption prevents proprietary technology from being stolen

In the past, it was very easy to steal a user's Core ML model and paste it into other applications.

Starting with iOS 14 / macOS 11.0, Core ML can automatically encrypt and decrypt models, so strangers can’t peek into the user’s mlmodelc folder. Encryption is available regardless of whether the user has a new CloudKit deployment.

The model mlmodelc compiled by Xcode is not the original mlmodel file. This model will be permanently encrypted on the user's device.

When an application instantiates a model, Core ML automatically decrypts it. This decrypted version is not stored as a file anywhere, only in memory.

First, users need an encryption key, and users don't need to manage this key. There is now a "Create Encryption Key" button in Xcode's Core ML model viewer. When the user clicks this button, Xcode generates a new encryption key associated with the Apple developer team account.

This process creates a new .mlmodelkey ​​file. This key is stored on Apple's servers, but users can also get a local copy needed to encrypt models in Xcode without embedding this encryption key in their applications.

To encrypt a Core ML model, users can add --encrypt YourModel.mlmodelkey ​​to the model's compiler flags. Or if you want to deploy using CloudKit, you need to provide an encryption key when creating a model archive.

To decrypt the model when the app is instantiated, Core ML will need to fetch the encryption key from Apple’s servers over the network, so an internet connection is obviously required. Core ML only requires an internet connection the first time a user uses the model.

If the network is down and the encryption key has not yet been downloaded, the app will not be able to instantiate the Core ML model. Therefore, users should use the new YourModel.load() function. It can complete the handler and prompt the user to respond to the loading error.

This function is easy to implement in the application and effectively solves the problem of misappropriation of proprietary technology.

In addition, Create ML has also been greatly improved this year, providing new templates and training capabilities.

With the new version of Xcode 12, users can pause training and continue, save model checkpoints, and preview the running status of the model, giving users better control over the training process. In CreateML.framework, more APIs for vision and natural language are also provided.