How to build your first machine learning model on your iPhone

introduction

As a data scientist, I have always had a dream that top technology companies continue to launch new products in fields related to me.

If you watched Apple's latest iPhone X announcement, you'll see that the iPhone X has some really cool features like FaceID, Animoji, and Augmented Reality, all of which use machine learning. Being a hacker myself, I decided to try my hand at exploring how to build such a system.

After further investigation, I found an interesting tool, which is CoreML, Apple's official machine learning framework tool for developers. It can be used on any Apple device, including iPhone, Macbook, Apple TV, Apple watch, etc.

[[206275]]

Another interesting revelation is that Apple designed a custom GPU in its latest iPhones, as well as an A11 advanced Bionic processing chip with a Neural Engine for optimized machine learning.

As the core component computing engine becomes increasingly powerful, the iPhone will open up new avenues for machine learning, and CoreML will become increasingly important in the future.

After reading this article, you will understand what Apple CoreML is and why it is gaining momentum. We will also explore the implementation details of CoreML by developing a spam SMS classification app on iPhone.

At the same time, we will also end this article by objectively evaluating the pros and cons of CoreML.

Article Directory:

1. What is CoreML?
2. Build the system
3. Case Study: Implementing a spam SMS classification app on iPhone
4. Pros and Cons of Using CoreML

01. What is CoreML?

This year, Apple has been hyping CoreML at its annual Worldwide Developers Conference (WWDC), similar to Google's I/O conference. To better understand the role of CoreML, we need to understand some background.

Background on CoreML

Interestingly, this is not the first time Apple has released a mobile machine learning framework. Last year it released some of the same framework libraries:

1. Accelerate Framework and Basic Neural Network Subroutines (BNNS) — Efficiently use CPU and use convolutional neural networks for prediction.
2. Metal Performance Shaders CNN (MPSCNN) - Efficiently utilizes the GPU and uses convolutional neural networks for prediction.

The difference between these two framework libraries is that one is optimized for CPU and the other for GPU. This is because sometimes during inference, the CPU is faster than the GPU, while during training, the GPU is almost always faster.

But to improve performance, the frameworks get very close to the underlying hardware, making these hybrid frameworks confusing for developers and difficult to program.

Enter CoreML

CoreML will provide an abstraction layer on top of the two libraries mentioned above and will also provide a simple interface to achieve the same efficiency. Another benefit is that when our app is running, CoreML fully takes care of the context switching between the CPU and GPU.

In other words, if we have a memory-intensive task that involves text processing (natural language processing), CoreML will automatically run on the CPU; and if we have a computationally heavy task, such as image recognition, it will use the GPU; when the app contains both functions, it will automatically switch so that both are utilized to the maximum extent.

What else will CoreML offer?

CoreML also comes with three libraries on top:

1. Vision: This library provides high-performance image analysis and computer vision techniques for face recognition, feature detection, and scene recognition in images and videos.
2. Foundation(NLP): As the name suggests, it provides some functions for natural language processing
3. Gameplay Kit: A library for game development that also provides AI and uses decision trees.

All the libraries mentioned above can be easily used with some simple interfaces and can be used to complete a range of tasks. With the above libraries, the final framework diagram of CoreML is as follows:

Note that the above design provides a nice modular structure for iOS apps. You can use different layers for different tasks and use them in multiple ways (for example, using NLP for image classification in your app). Learn more: Vision, Foundation, and GameplayKit. Ok, now that we have enough theory, it's time to put it into practice!

"Due to WeChat layout limitations, students who need the code can look up the original link at the end of the article to find it themselves."

02. Establish a system

To fully use CoreML, you need to follow these requirements:

1.OS: MacOS (Sierra 10.12 or above)

2. Python 2.7 and pip: Click to download python on mac. Open the terminal and enter the following code to install pip:

 sudo easy_install pip

3.coremltools: This package helps convert your model from Python to a format that CoreML can understand. Enter the following code in the terminal to install it:

 sudo pip install -U coremltools

4. Xcode 9: This is the default software for building applications on Apple devices. Click here to download. Before downloading Xcode, you need to log in with your Apple ID.

After logging in, you will need to verify your Apple ID. You will receive the same notification as the device where your Apple ID is registered.

Click "Allow" and enter the 6-digit password displayed by the website.

When you complete this step, you will see a download option. You can download Xcode from there. Now that we have the system set up, let's move on to the implementation part if we are ready!

03. Case Study: Implementing a spam SMS classification app on iPhone

In this development, we will focus on using CoreML's capabilities in two important ways. Let's get started!

Convert your machine learning model to CoreML format

One of the strengths of CoreML, or should I say a wise decision made by its creators, is the support for converting machine learning models trained in other popular frameworks like sklearn, caffe, xgboost, etc.

The data science community will not be reluctant to try CoreML as they can experiment in their favorite environment, train their models, and then easily import and use them in apps on iOS/macOS.

Here are the frameworks that CoreML supports out of the box:

What is Mlmodel?

To make the conversion process simple, Apple designed its own open format to represent cross-framework machine learning models, namely mlmodel. This model file contains a description of each layer of the model, inputs, outputs, class labels, and any preprocessing required for the data. It also contains the learned parameters (weights and biases).

The conversion process is as follows:

1. Train the model in your favorite framework
2. Use the Python module coremltools to convert the model to .mlmodel format
3. Using the model in your app

In this example, we will train a spam classifier in sklearn and then transfer the model to CoreML.

About the spam text message dataset

The SMS Spam Dataset v.1 is a public SMS labeled text message dataset for mobile spam text message research. It contains 5574 real, uncoded English text messages, which are marked as legitimate (contrived) or spam text messages.

You can download the dataset here.

Building a basic model

We use LinearSVC in sklearn to build the basic model. At the same time, we extract the TF-IDF value of the text message as the model feature. TF-IDF is a method in natural language processing that classifies documents based on the words that uniquely identify the document. If you want to learn more about NLP and tf-idf, you can read this article. The code is as follows:

 import numpy as npimport pandas as pd#Reading in   and parsing dataraw_data = open ( 'SMSSpamCollection.txt' , 'r' )sms_data = [] for line in raw_data: split_line = line.split( "\t" ) sms_data.append(split_line)
 #Splitting data into messages and labels and training and testsms_data = np.array(sms_data)X = sms_data[:, 1]y = sms_data[:, 0]  
 #Build a LinearSVC modelfrom sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.svm import LinearSVC  
 #Build tf-idf vector representation of datavectorizer = TfidfVectorizer()vectorized_text = vectorizer.fit_transform(X)text_clf = LinearSVC()text_clf = text_clf.fit(vectorized_text, y)

Now that our model is built, let's test it with a spam message:

 #Test the modelprint text_clf.predict(vectorizer.transform([ "" "XXXMobileMovieClub: To use your credit, click the WAP link in the next txt message or click here>> http://wap.xxxmobilemovieclub.com?n=QJKGIGHJJGCBL" " " ]))

Interesting, our model works pretty well! Let’s add cross validation:

 # Cross - Validationfrom sklearn.model_selection import cross_val_scorecross_score = cross_val_score(text_clf, vectorized_text, y, cv=10)print cross_scoreprint "mean:" ,np.mean(cross_score)
 Now that the model is built, we need to convert it to .mlmodel format to make it suitable for CoreML. We can do this using the coremltools package that we installed earlier. The following code will convert our model to .mlmodel format.  
 import coremltools  
 # convert   to coreml model  
 coreml_model = coremltools.converters.sklearn. convert (text_clf, "message" , "spam_or_not" )  
 # set parameters of the model  
 coreml_model.short_description = "Classify whether message is spam or not"    
 coreml_model.input_description[ "message" ] = "TFIDF of message to be classified"    
 coreml_model.output_description[ "spam_or_not" ] = "Whether message is spam or not"    
 #save the model  
 coreml_model.save( "SpamMessageClassifier.mlmodel" )

How does this work?

First, we use the coremltools Python toolkit. Then we select a converter to convert the model. In this case, we use converters.sklearn because the model to be converted is built using the sklearn tool. Then, we declare the model object, input variable names, and output variable names within the .convert() brackets. Next, we set the model parameters to add more information about the input and output, and finally use .save() to save the model file that has been converted to CoreML format.

Double-click the model file and it will open in Xcode.

As you can see, the model file shows a lot of information about the type of model, its inputs, outputs, the types of inputs and outputs, etc. I have marked them in red in the image above. You can compare these descriptions with what is provided when converting to .mlmodel.

It’s that simple to import your own model into CoreML. Now that your model is in the Apple ecosystem, the real fun begins!

Note: The full code for this step can be found here. Learn more about coremltools here and the different types of converters provided here.

Using this model in our app

Now that we have trained our model and imported it into CoreML, let’s use it to develop an iPhone spam classification app!

We will run the app on the simulator. The simulator is a piece of software that displays the app's interface and operation as if it were actually running on an iPhone. This saves a lot of time because we can test the code and debug before running the app on an iPhone. Here's what the final product looks like:

Download Project

I have made a simple basic UI for our app and put it on GitHub. Load and run it with the following command:

 git clone https://github.com/mohdsanadzakirizvi/CoreML-on-iPhone.git  
 cd CoreML- on -iPhone/Practice\ App/  
 open coreml\ test.xcodeproj/

This will open our project in Xcode.

I have highlighted three important areas in red in the Xcode window:

1. The play button in the upper left corner is used to start the app running in the simulator.
2. Just below the play button are the files and folders related to our project. This is the project navigation bar, which makes it easy for you to find files and folders in the project.
3. The iPhone 8 is written next to the play button, indicating the device you want to emulate with the simulator. You can click it and select iPhone 7 from the drop-down list.

Let's run the app and see what happens. Click the play button in the upper left corner to run the app in the simulator. Type some text in the box and click the predict button. What happened?

So far, our app does nothing but output the text typed into the box as is.

Add a trained model to your app

Pretty simple:

• Drag your .mlmodel model file into the project navigator in the Xcode window.
• When you are done, a window will pop up with several options. Select the default options and click "Finish".
• When you drag a file into Xcode like this, a reference path to the file is automatically generated in the project, so you can easily get the file in your code.

Compile the model

Before we can use our model for inference, we need to let Xcode compile the model during the build phase. Here are the specific steps:

Select the file with a blue icon in the project navigation bar.

The project settings will open on the right hand side. Click on Compile Sources and select the + icon.

In the new window that appears, select the SpamMessageClassifier.mlmodel file and click Add.

Now every time you run the app, Xcode will compile our machine learning model so that it can be used to make predictions.

Creating the model in code

Any app developed for Apple devices is programmed in Swift. You don’t need to learn Swift but if you are interested in learning more later, you can follow this tutorial.

Select ViewController.swift in the Project Navigator. This file contains most of the code that controls the app’s functionality.

The predictSpam() function on line 24 does most of the work. Delete line 25 and add the following code to the function:

 let enteredMessage = messageTextField.text!  
 if (enteredMessage != "" ){  
 spamLabel.text = ""    
 }  
 // Fetch tfidf representation of text  
 let vec = tfidf(sms: enteredMessage)  
 do {  
 //Get prediction on the text  
 let prediction = try SpamMessageClassifier().prediction(message: vec).spam_or_not  
 print (prediction)  
 if (prediction == "spam" ){  
 spamLabel.text = "SPAM!"   
  }  
 else if(prediction == "ham" ){  
 spamLabel.text = "NOT SPAM"    
 }  
 }  
 catch{  
 spamLabel.text = "No Prediction"    
 }

The above code checks if the user has entered any information into the box. If so, the tfidf() function is called to calculate the tfidf value of the text. Then a SpamMessageClassifier object instance is generated and the .prediction() function is called. This is the same as the .predict() function in sklearn. Then appropriate information is displayed based on the prediction.

But why is tfidf() needed?

Remember that we are training the model based on the tf-idf representation of the text, so our model needs the same form of input. Once we have the information typed into the text box, call the tfidf() function to do the same thing. Let's write the code for this step. Copy the following code and put it after the predictSpam() function:

 //MARK: Functionality code  
 func tfidf(sms: String) -> MLMultiArray{  
 //get path for files  
 let wordsFile = Bundle.main.path(forResource: "wordlist" , ofType: "txt" )  
 let smsFile = Bundle.main.path(forResource: "SMSSpamCollection" , ofType: "txt" )  
 do {  
 // read words file  
 let wordsFileText = try String(contentsOfFile: wordsFile!, encoding: String.Encoding.utf8)  
 var wordsData = wordsFileText.components(separatedBy: .newlines)  
 wordsData.removeLast() // Trailing newline.  
 // read spam collection file  
 let smsFileText = try String(contentsOfFile: smsFile!, encoding: String.Encoding.utf8)  
 var smsData = smsFileText.components(separatedBy: .newlines)  
 smsData.removeLast() // Trailing newline.  
 let wordsInMessage = sms.split(separator: " " ) 
  // create a multi-dimensional array  
 let vectorized = try MLMultiArray(shape: [NSNumber(integerLiteral: wordsData. count )], dataType: MLMultiArrayDataType. double )  
 for i in 0..  
 let word = wordsData[i]  
 if sms.contains (word) {  
 var wordCount = 0  
 for substr in wordsInMessage{  
 if substr.elementsEqual(word{  
 wordCount += 1  
 }  
 } 
  let tf = Double (wordCount) / Double (wordsInMessage. count )  
 var docCount = 0  
 for sms in smsData{  
 if sms.contains (word) {  
 docCount += 1  
 }
 }  
 let idf = log( Double (smsData. count ) / Double (docCount))  
 vectorized[i] = NSNumber(value: tf * idf) 
  } else {  
 vectorized[i] = 0.0  
 }  
 }  
 return vectorized  
 } catch {  
 return MLMultiArray()  
 }  
 }

The above code gets the tfidf representation of the information entered in the text box, to do this read the SMSSpamCollection.txt original database and return the same information. Once you save the project and run the simulator again, your app will run fine.

4. Advantages and disadvantages of CoreML

Like every evolving library, CoreML has pros and cons. Let’s be clear.

advantage:

• Optimize performance on mobile devices, minimizing memory and energy consumption.
• Running on mobile devices ensures user privacy, and there is no need to send data to the server for prediction.
• Running on a mobile device means that predictions can be made even when there is no internet connection, and the response time for the user is much faster.
• Ability to decide whether to run on the CPU or GPU (or both).

Because it can use the CPU, you can run it on the iOS Simulator (iOS Simulator does not support GPU).

It provides many models because it can import models from other mainstream machine learning frameworks:

• Support Vector Machine (SVM)
• Tree ensembles, such as random forests and boosted trees
• Linear Regression and Logistic Regression
• Neural Networks: Feedforward, Convolution, Recurrence

shortcoming:

• Only supervised models are supported, not unsupervised models and reinforcement learning.
• Model retraining on the device is not supported, only prediction can be done.
• If CoreML doesn’t support a certain layer, you can’t use it. It’s not currently possible to extend CoreML with your own layers.
• CoreML tools only support specific versions of a small number of training tools (it doesn't even support tensorflow).
• You cannot see the output of the middle layer, you can only get the prediction result.
• Only regression and classification are supported (clustering, sorting, dimensionality reduction, etc. are not supported).

Conclusion

In this article, we learned about CoreML and applied it to develop iPhone machine learning apps. CoreML is a newer library, so it has its own advantages and disadvantages. One very useful advantage is that it runs on the local device, so it is faster and guarantees data privacy. But at the same time, it is not comprehensive and does not consider the needs of data scientists well enough. I hope that subsequent versions will improve.

If you get stuck at any point, all the code for this article is available on GitHub.