Implementing image recognition in Web development based on Google Vision API and Ionic

1. Introduction

Image recognition allows computers to recognize images in a way similar to humans. In the past, developers had to rely on complex image recognition techniques and algorithms such as pattern recognition to achieve this goal. Now, with the help of Google Vision API, programmers can directly use the image recognition functions provided in these ready-made tools.

This article will show you how to use the Google Vision API for image recognition through a JavaScript web application built on the Ionic framework.

two, Start working

To use the Google Vision API, you need to upload a JSON file that contains the image type you want to detect and the base64 encoding of the image. You need to upload these two pieces of information to the Google Vision API.

Here is an example of a JSON file:

 {
 "requests" :[
 {
 "image" :{
 "content" : "base64-encoded-image"  
 },
 "features" :[
 {
 "type" : "LABEL_DETECTION" ,
 "maxResults" : 1
 }
 ]
 }
 ]
 }

In this example, you must replace the base64-encoded-image part above with the actual base64-encoded string of the image. In addition, you need to provide an object array at the features attribute, which contains information about the type of image you want to detect. Among them, the LABEL_DETECTION attribute value is used to classify the image by specifying a label or description information.

Once the information is returned from the server, it should be in a format similar to the following:

 {  
 "responses" : [  
 {
 "labelAnnotations" : [ 
 
 {
 "mid" : "/m/0bt9lr" ,
 "description" : "dog" ,  
 "score" : 0.89208293
 }
 ]
 } 
 
 ] 
 
 }

Because you specified LABEL_DETECTION and assigned a maxResults value of 1 in the previous request format, a single object is returned in the response array. In the example above, the array name is labelAnnotations.

In addition to using LABEL_DETECTION in the previous example, you can also use the following enumeration data:

l FACE_DETECTION: Detects the face part in the photo and returns the corresponding coordinate value, which is used to draw the face part according to the detected range.

l LANDMARK_DETECTION: Detect landmarks such as Sydney's Opera House or Stonehenge in Wiltshire.

l LOGO_DETECTION: Detect different company logos.

l TEXT_DETECTION: Extract text from images using optical character recognition (OCR) technology.

l SAFE_SEARCH_DETECTION: Classify images based on safe search parameters. The image classifications provided here are divided into: adult, spoof, medical, and violence.

three, Register Cloud Vision API

At the time of writing, Google Cloud Vision API is still in beta, which means developers can try it for free. To do this, you can navigate to the Google Cloud Platform website (https://cloud.google.com/vision/) and click the button "try to free". The above action will take you to a page asking for your business and credit information; but don't worry, Google will not charge you anything more than $300.

Once the above is done, you can create a new project in the Google console, enable billing in the project, and enable the Cloud Vision API. You are advised to skip the usual process but use the option "API Key". Please refer to the image below:

Four, Build the application

Now you are ready to build the application. But first, I want to give a brief overview of the application you are going to build. The application provides a page with all the elements needed to interact with the Cloud Vision API. There is a drop-down list to select which type of image detection the user wants to use, a button to take a photo, a section to display the photo to be taken, and a title section to describe the image displayed.

Here's what the final application will look like:

You can find the source code of this example project on GitHub (https://github.com/sitepoint-editors/ionic-vision).

(one) Install Dependencies

In your working directory, open a new terminal window and install Cordova and Ionic as follows:

 npm install -g cordova ionic

Then, create a new Ionic project using the blank template with the following command:

 ionic start ionic-vision blank

Next, add the platforms you want to use. I only want to install Android, but the code should work on iOS as well. The command is as follows:

 ionic platform add android

Next, you need to install several plugins to interact with the device API so that you can use features such as camera, files, and upload files. The relevant commands are as follows:

 cordova plugin add cordova-plugin-camera
 cordova plugin add cordova-plugin-file  
 cordova plugin add cordova-plugin-file-transfer

Next, use bower to install ngCordova:

 bower install ngCordova

Note: The NgCordova library provides AngularJS wrappers for plugins to install. These wrappers make it easier to use these plugins in an Ionic application.

(two) Adding a Controller

Now, open the www directory and create a file controllers/HomeController.js in the js directory. Finally, add the following code:

 ( function () {
    angular.module( 'starter' )  
    .controller( 'HomeController' , [ '$scope' , '$ionicModal' , '$cordovaFile' , '$cordovaFileTransfer' , '$cordovaCamera' , HomeController]);
 function HomeController($scope, $ionicModal, $cordovaFile, $cordovaFileTransfer, $cordovaCamera){
 var me = this ;
        me.current_image = 'img/koro-sensei.png' ;
    me.image_description = '' ;
        me.detection_type = 'LABEL_DETECTION' ;
        me.detection_types = { 
 
          LABEL_DETECTION: 'label' , 
 
          TEXT_DETECTION: 'text' , 
 
          LOGO_DETECTION: 'logo' , 
 
         LANDMARK_DETECTION: 'landmark'  
 };
 var api_key = 'your-google-api-key' ;  
        $scope.takePicture = function (){
 var options = {
                destinationType: Camera.DestinationType.DATA_URL,  
                sourceType: Camera.PictureSourceType.CAMERA, 
                 targetWidth: 500,  
              targetHeight: 500,
              correctOrientation: true ,
                cameraDirection: 0,
                encodingType: Camera.EncodingType.JPEG  
 };
            $cordovaCamera.getPicture(options).then( function (imagedata){
                me.current_image = "data:image/jpeg;base64," + imagedata;
                me.image_description = '' ;
                me.locale = '' ;
 var vision_api_json = { 
 
 "requests" :[ 
 
 { 
  "image" :{ 
 
 "content" : imagedata 
 
 },
 "features" :[ 
 
 { 
  "type" : me.detection_type, 
 
 "maxResults" : 1 
 
 }
 ]
 }
 ]
 };
 var file_contents = JSON.stringify(vision_api_json);
              $cordovaFile.writeFile( 
 
                    cordova.file.applicationStorageDirectory,
 'file.json' , 
 
                    file_contents, 
 
 true   
                ).then( function (result){
 var headers = { 
  'Content-Type' : 'application/json'   
 
 }; 
                    options.headers = headers;
 var server = 'https://vision.googleapis.com/v1/images:annotate?key=' + api_key;  
 var filePath = cordova.file.applicationStorageDirectory + 'file.json' ;
                    $cordovaFileTransfer.upload(server, filePath, options, true )  
                     .then( function (result){
 var res = JSON.parse(result.response); 
 
 var key = me.detection_types[me.detection_type] + 'Annotations' ;
                            me.image_description = res.responses[0][key][0].description;
                      }, function (err){
                        alert( 'An error occurred while uploading the file' );
 });
                }, function (err){
                    alert( 'An error occurred while trying to write the file' );
 });
            }, function (err){ 
 
              alert( 'An error occurred getting the picture from the camera' ); 
 
 });
 }
 }
 })();

Now, let's break down the code above into pieces. First, create a controller and import the required libraries:

 ( function () {
    angular.module( 'starter' )
    .controller( 'HomeController' , [ '$scope' , '$cordovaFile' , '$cordovaFileTransfer' , '$cordovaCamera' , HomeController]);
 function HomeController($scope, $cordovaFile, $cordovaFileTransfer, $cordovaCamera){
 ...
 }

Set the default data needed by the view in the controller. This includes: a placeholder for the image to be displayed, an empty description position, a default detection type, etc. Usually we use the LABEL_DETECTION option because it is more versatile than other options. Please refer to the following code:

 var me = this ; 
 
 me.current_image = 'img/koro-sensei.png' ; 
 
 me.image_description = '' ; 
 
 me.detection_type = 'LABEL_DETECTION' ;

Next, define an object that contains all the types to be detected and the API key provided by Google:

 me.detection_types = { 
 
 LABEL_DETECTION: 'label' , 
 
 TEXT_DETECTION: 'text' , 
 
 LOGO_DETECTION: 'logo' , 
 
  LANDMARK_DETECTION: 'landmark'    
 };
 var api_key = 'your-google-api-key' ;

Next, create a method to be executed when the camera button is pressed. The code is as follows:

 $scope.takePicture = function (){  
 ...
 };

In this method, first declare the camera plugin-related options and set the destinationType to Camera.DestinationType.DATA_URL. This means that once a picture is selected, the callback function will have the URI data of the image. Because this URI is already base64-encoded data, no conversion is required.

The sourceType is specified as Camera.PictureSourceType.CAMERA so that the image captured by the camera can be used as the data source. Then, the targetWidth and targetHeight values ​​set the appropriate image size. The correctOrientation value is set to true so that it automatically changes the orientation of the image to portrait. The cameraDirection is specified as 0 so that the rear camera can be used. Finally, the encodingType is specified as Camera.EncodingType.JPEG, which allows you to prepend the data URI to data:image/jpeg;base64 to display the image.

 var options = {
    destinationType: Camera.DestinationType.DATA_URL,
    sourceType: Camera.PictureSourceType.CAMERA,
 targetWidth: 500,
 targetHeight: 500,
    correctOrientation: true ,
 cameraDirection: 0,
    encodingType: Camera.EncodingType.JPEG
 };

Calling $cordovaCamera.getPicture will open the default camera application on the device. It takes options as a parameter and then calls then with callbacks for success and error actions. The same design applies to all plugins you will use later.

 $cordovaCamera.getPicture(options).then( function (imagedata){ 
  ...
 }, function (err){ 
 
    alert( 'An error occurred getting the picture from the camera' );
 });

Next, in the success callback function, update the image source (current_image) and reset the description to an empty string:

 me.current_image = "data:image/jpeg;base64," + imagedata; 
 
 me.image_description = '' ;

Then, use the data URI you got from the camera plugin and the detection type (me.detection_type) selected by the user to construct a data object. Then, convert the object to a string; this way, you can use it as content data in the JSON file sent to the API.

 var vision_api_json = { 
 
 "requests" :[ 
 
 {
 "image" :{
 "content" : imagedata
 },
 "features" :[
 {
 "type" : me.detection_type, 
 
 "maxResults" : 1
 }
 ]
 }
 ]  
 };
 var file_contents = JSON.stringify(vision_api_json);

Next, use the Cordova file plugin to write file_contents to the file file.json stored in the root directory of the application sandbox. Also, note that the third parameter of the writeFile method is a Boolean type that specifies whether to create a new file if the file does not exist:

 $cordovaFile.writeFile(
    cordova.file.applicationStorageDirectory, 
 
 'file.json' , 
 
 file_contents, 
 
 true  
 ).then( function (result){
 ...
 }, function (err){
    alert( 'An error occurred while writing to the file' );
 });

When writing content to a file, you need to declare the variables that the file transfer plugin needs to use. The headers variable in the code snippet below corresponds to the http headers of the request. Because you are sending a JSON file, you must set the Content-Type to application/json. In addition, a full path URL is used in server to send the request to the API; a full path is also used in filePath to specify the JSON file to be sent.

 var headers = { 
 
 'Content-Type' : 'application/json'   
 
 };   
 options.headers = headers;
 var server = 'https://vision.googleapis.com/v1/images:annotate?key=' + api_key; 
 
 var filePath = cordova.file.applicationStorageDirectory + 'file.json' ;

Next, send the file to the server using the upload method of the file transfer plugin. Here, the fourth parameter provided to the upload method is a boolean value that sets whether to accept security certificates from all hosts. Once you get a response from the server, you need to convert it into a JavaScript object using the JSON.parse method. Construct a key by concatenating the current detection type and the word Annotations. This allows you to form the string labelAnnotations if the user selected LABEL_DETECTION as the detection type. You can then use this string to extract the actual description of the image.

 $cordovaFileTransfer.upload(server, filePath, options, true ) 
 
 .then( function (result){
 var res = JSON.parse(result.response); 
 
 var key = me.detection_types[me.detection_type] + 'Annotations' ; 
 
        me.image_description = res.responses[0][key][0].description;
 }, function (err){ 
    alert( 'An error occurred while uploading the file' ); 
 
 });

(three) Add a view

Now, you need to create a file template/home.html and add the following code:

 <ion-view title= "IonicVision" ng-controller= "HomeController as home_ctrl" > 
 
    <header class = "bar bar-header bar-stable" > 
 
        <h1 class = "title" >Ionic Vision</h1> 
 
 </header>
    <ion-content class = "has-header padding" >
        <img src= "{{ home_ctrl.current_image }}"   class = "picture" > 
 
        <h3 class = "text-center" ng-show= "home_ctrl.image_description" >{{ home_ctrl.image_description }}</h3>
        <label class = "item item-input item-select" >
          <div class = "input-label" >
 Detection Type
 </div>
          <select ng-model= "home_ctrl.detection_type" > 
 
            <option value= "{{detection_type}}" ng-repeat= "(detection_type, detection_type_value) in home_ctrl.detection_types" >{{detection_type_value}}</option>
 </select>
 </label>
        <button class = "button button-positive button-block" ng-click= "takePicture()" >
 Take Picture 
 
 </button>
 </ion-content> 
 
 </ion-view>

Next, let's analyze the above code piece by piece.

First, create a new ion-view and specify the controller to use:

 <ion-view title= "IonicVision" ng-controller= "HomeController as home_ctrl" > 
 
 </ion-view>

Inside this io-view is the header and ion-content. The ion-content is the UI element you see below the header. Further inside is the image, image description, detection type list, and camera button.

 <header class = "bar bar-header bar-stable" > 
 
    <h1 class = "title" >Ionic Vision</h1>
 </header>
 <ion-content class = "has-header padding" > 
 
    <img src= "{{ home_ctrl.current_image }}"   class = "picture" > 
 
    <h3 class = "text-center" ng-show= "home_ctrl.image_description" >{{ home_ctrl.image_description }}</h3> 
 
    <label class = "item item-input item-select" > 
 
      <div class = "input-label" > 
 
 Detection Type
 </div> 
 
      <select ng-model= "home_ctrl.detection_type" >
        <option value= "{{detection_type}}" ng-repeat= "(detection_type, detection_type_value) in home_ctrl.detection_types" >{{detection_type_value}}</option>
 </select>
 </label>
    <button class = "button button-positive button-block" ng-click= "takePicture()" >
 Take Picture
 </button>
 </ion-content>

five, format

Most of the styling is done automatically by Ionic, so you only need to declare a set of styles. To do this, add the following to your css/style.css file:

 .text- center {
 text-align : center ;  
 }   
 
 .picture {
 max-width : 100% ; 
  max-height : 100% ;  
 }

(one) Putting it together

Open the js/app.js file, which contains the code for initializing Ionic and ngCordova. If you chose to use the blank startup template provided by Ionic, you will notice that most of the code is already filled in. You only need to specify the purpose of ngCordova and edit the content of the config method to point to the file home.html.

 angular.module( 'starter' , [ 'ionic' , 'ngCordova' ])
 .run(function($ionicPlatform) { 
 
  $ionicPlatform.ready(function() { 
 
    if(window.cordova && window.cordova.plugins.Keyboard) { 
 
    cordova.plugins.Keyboard.hideKeyboardAccessoryBar(true); 
 
     cordova.plugins.Keyboard.disableScroll(true); 
 
 }
 if(window.StatusBar) { 
 
      StatusBar.styleDefault();
 } 
 
 }); 
 
 })
 .config(function($stateProvider, $urlRouterProvider) { 
 
 $stateProvider   
 .state( 'home' , { 
 
 url: '/home' , 
 
    templateUrl: 'templates/home.html'   
 
 });  
  $urlRouterProvider.otherwise( '/home' ); 
 
 });

Next, open the index.html file and link to the ng-cordova.js file (just after the ionic.bundle.js file). Then, after the app.js file, link to the HomeController.js file.

Don’t forget to specify starter as the value of ng-app and add ion-nav-view inside the body tag to display the home.html view.

 <!DOCTYPE html >   
 
 < html >  
 < head >  
 < meta   charset = "utf-8" >  
 < meta   name = "viewport"   content = "initial-scale=1, maximum-scale=1, user-scalable=no, width=device-width" >  
 < title > </ title >  
 < link   href = "lib/ionic/css/ionic.css"   rel = "stylesheet" >  
 < link   href = "css/style.css"   rel = "stylesheet" >  
   <!-- IF using Sass (run gulp sass first), then uncomment below and remove the CSS includes above
 < link   href = "css/ionic.app.css"   rel = "stylesheet" >  
 -- >  
 <!-- ionic/angularjs js -->  
 < script   src = "lib/ionic/js/ionic.bundle.js" > </ script >  
 < script   src = "lib/ngCordova/dist/ng-cordova.js" > </ script >  
 <!-- cordova script (this will be a 404 during development) -->  
 < script   src = "cordova.js" > </ script >  
 <!-- your app's js -->  
 < script   src = "js/app.js" > </ script >  
 < script   src = "js/controllers/HomeController.js" > </ script >  
 </ head >   
 
 < body   ng-app = "starter" >   
 
 <ion-nav-view> </ion-nav-view>​​​​   
 
 </ body >   
 
 </ html >  

six, Run the application

Now you can run the above application on your device or simulator by executing the following command:

 ionic run android

seven, summary

In this tutorial, you built an image recognition software using Ionic and the Cloud Vision API. In it, I discussed using different types of image detection such as label, sign, logo, and text detection. However, I did not cover face detection or safe search detection in the article. However, for face detection technology, you can use a framework like Fabric.js. This tool converts the image into a Canvas object and draws circles on the detected faces.

For more information about Cloud Vision API, please read the official documentation (https://cloud.google.com/vision/docs/). Of course, I also hope to learn about your experience and thoughts.

Original title : Image Recognition with the Google Vision API and Ionic

[Translated by 51CTO. Please indicate the original translator and source as 51CTO.com when reprinting on partner sites]