Developing intelligent assistant based on Android TTS service

Introduction

In this article, I will show you a basic Android application that can listen to the user's voice and convert it into text data. In addition, this application can also perform text analysis and then execute corresponding commands to implement data storage and user response functions.

Note: The source code for this article can be downloaded from https://github.com/sitepoint-editors/SpeechApplication.

The program snapshot is as follows:

[[167745]]

Create an application

Open Android Studio and create a new project. Select the minimum version of Android API 18 and add an empty Activity. This is also the only Activity in this project.

To achieve full-screen display of the view, open the configuration file AndroidManifest.xml and set it as follows:

 android:theme= "@style/Theme.AppCompat.NoActionBar"  

This configuration will hide the ActionBar in our current activity.

At this point, you already have a full-screen white background layout view with only one TextView control. To make some improvements, you can add a gradient shape to the RelativeLayout.

Next, right-click the drawable folder and select New->Drawable resource file. Name this resource file background and replace the original content with the following code:

 <?xml version= "1.0" encoding= "UTF-8" ?> 
 
 <shape xmlns:android= "http://schemas.android.com/apk/res/android"   
 
 android:shape= "rectangle" > 
 
 < gradient 
 
 android:type= "linear"   
 
 android:startColor= "#FF85FBFF"   
 
 android:endColor= "#FF008080"   
 
 android:angle= "45" /> 
 
 </shape>

In fact, you can modify the color and angle as you like.

Note: The ImageButton control in the layout uses an image from https://design.google.com/icons/#ic_mic_none. You can download it and add it as a resource.

Next, update the code in the activity_main.xml file:

 <?xml version= "1.0" encoding= "utf-8" ?> 
 
 <RelativeLayoutxmlns:android= "http://schemas.android.com/apk/res/android"   
 
 xmlns:tools= "http://schemas.android.com/tools"   
 
 android:layout_width= "match_parent"   
 
 android:layout_height= "match_parent"   
 
 android:background= "@drawable/background"   
 
 android:id= "@+id/rel"   
 
 tools:context= "com.example.theodhor.speechapplication.MainActivity" > 
 
 <ImageButton 
 
 android:layout_width= "wrap_content"   
 
 android:layout_height= "wrap_content"   
 
 android:id= "@+id/microphoneButton"   
 
 android:layout_centerVertical= "true"   
 
 android:layout_centerHorizontal= "true"   
 
 android:src= "@drawable/ic_mic_none_white_48dp"   
 
 android:background= "@null" /> 
 
 </RelativeLayout>

Add speaking function

Now that the user interface is complete, it is time to write the Java code inside MainActivity.

First, declare a variable TextToSpeech above the onCreate method:

 private TextToSpeechtts;

Then, add the following code to the onCreate method:

 tts = new TextToSpeech(this, new TextToSpeech.OnInitListener() { 
 
 @Override 
 
 public void onInit( int status) { 
 
 if (status == TextToSpeech.SUCCESS) { 
 
 int result = tts.setLanguage(Locale.US); 
 
 if (result == TextToSpeech.LANG_MISSING_DATA || result == TextToSpeech.LANG_NOT_SUPPORTED) { 
 
 Log.e( "TTS" , "This Language is not supported" ); 
 
 } 
 
 speak( "Hello" ); 
 
 } else { 
 
 Log.e( "TTS" , "Initilization Failed!" ); 
 
 } 
 
 } 
 
 });

The above code will start the TextToSpeech service in the system. The speak() method uses a String type parameter, which is the text you want your Android device to read out.

Next, create this method and add the following code:

 private void speak(String text){ 
 
 if (Build.VERSION.SDK_INT>= Build.VERSION_CODES.LOLLIPOP) { 
 
 tts.speak(text, TextToSpeech.QUEUE_FLUSH, null , null ); 
 
 } else { 
 
 tts.speak(text, TextToSpeech.QUEUE_FLUSH, null ); 
 
 } 
 
 }

The Build.VERSION check is used in the above code because calls of the form tts.speak(param,param,param) are deprecated for Android API 5.1.

Create another method after the speak() method to stop the TextToSpeech service when the user closes the program:

 @Override 
 
 public void onDestroy() { 
 
 if (tts != null ) { 
 
 tts.stop(); 
 
 tts.shutdown(); 
 
 } 
 
 super.onDestroy(); 
 
 }

At this point, once you start the program, it can say "Hello". The next step is to make the program listen.

Add listening function

To enable the program to listen, you need to use the microphone button. To do this, add the following code to the onCreate method:

 findViewById(R.id.microphoneButton).setOnClickListener(new View .OnClickListener() { 
 
 @Override 
 
 public void onClick( View v) { 
 
 listen(); 
 
 } 
 
 });

When the ImageButton control is clicked, the following function will be called:

 private void listen(){ 
 
 Intent i = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH); 
 
 i.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_FREE_FORM); 
 
 i.putExtra(RecognizerIntent.EXTRA_LANGUAGE, Locale.getDefault()); 
 
 i.putExtra(RecognizerIntent.EXTRA_PROMPT, "Say something" ); 
 
 try { 
 
 startActivityForResult(i, 100); 
 
 } catch (ActivityNotFoundException a) { 
 
 Toast.makeText(MainActivity.this, "Your device doesn't support Speech Recognition" , Toast.LENGTH_SHORT).show(); 
 
 } 
 
 }

This method will start the listening Activity, which will display a dialog box with a text prompt. The language used for the speech is provided by the device through the Locale.getDefault() method.

The StartActivityForResult(i, 100) method waits for the current activity to return a result. 100 is just a random code attached to the started activity, but it can be any number that suits your needs. When the result is returned from the started activity, it contains this code and uses this code to distinguish multiple results from each other.

To capture the results from the launched activity, you need to add the following override method:

 @Override 
 
 protected void onActivityResult(intrequestCode, intresultCode, Intent data) { 
 
 super.onActivityResult(requestCode, resultCode, data); 
 
 if(requestCode == 100){ 
 
 if (resultCode == RESULT_OK && null != data) { 
 
 ArrayList<String> res = data.getStringArrayListExtra(RecognizerIntent.EXTRA_RESULTS); 
 
 String inSpeech = res.get(0); 
 
 recognition(inSpeech); 
 
 } 
 
 } 
 
 }

This method captures each result from the activity and uses the requestCode to process the language identifier result data. If requestCode equals 100, requestCode equals OK and the data from this result is not null. You will get the result string from res.get(0).

Next, create a new method, recognition, which will take a parameter of type String:

 private void recognition(String text){ 
 
 Log.e( "Speech" , "" +text); 
 
 }

So far, when the user clicks the microphone button, the program can listen to the sound and convert the user's language into text data. The final result will be printed out through the Error log.

Add learning function

To make the program more interesting, in this section you will enable the application to learn some simple things, such as your name. To achieve this, you need to use the local storage feature.

First, add the following code to the onCreate method:

 private SharedPreferences preferences; 
 
 private SharedPreferences.Editor editor; 
 
 private static final String PREFS = "prefs" ; 
 
 private static final String NAME = "name" ; 
 
 private static final String AGE = "age" ; 
 
 private static final String AS_NAME = "as_name" ;

Then, add the following code to the onCreate method:

 preferences = getSharedPreferences(PREFS,0); 
 
 editor = preferences.edit();

First, you need to make the application ask questions, so you need to change speak("Hello") to speak("What is your name?").

Here you can use a simple logic; so when someone asks "What is your name?" and the answer is "My name is Dori", then take the name from the answer. A simple way to do this is to split the answer into strings separated by spaces and get the value of the last index.

So, we need to update the code in the recognition method as follows:

 private void recognition(String text){ 
 
 Log.e( "Speech" , "" +text); 
 
 //creating an array which contains the words of the answer 
 
 String[] speech = text.split( " " ); 
 
 //the last word is our name   
 
 String name = speech[speech.length-1]; 
 
 //we got the name , we can put it in   local storage and save changes 
 
 editor.putString( NAME , name ).apply(); 
 
 //make the app tell our name   
 
 speak( "Your name is " +preferences.getString( NAME , null )); 
 
 }

The recognize method uses all the results from the user's speech. Since the utterances may be different, you can use certain words they may contain to distinguish them.

For example, the code in this method could be:

 private void recognition(String text){ 
 
 Log.e( "Speech" , "" +text); 
 
 String[] speech = text.split( " " ); 
 
 //if the speech contains these words, the user   is saying their name   
 
 if(text. contains ( "my name is" )){ 
 
 String name = speech[speech.length-1]; 
 
 Log.e( "Your name" , "" + name ); 
 
 editor.putString( NAME , name ).apply(); 
 
 speak( "Your name is " +preferences.getString( NAME , null )); 
 
 } 
 
 }

But it's still a simple interaction with the app. You can have it learn your age, or give it a name.

In the same method, you can try these simple conditions:

 //This must be the age 
 
 //Just speak: I am x years old. 
 
 if(text. contains ( "years" ) &&text. contains ( "old" )){ 
 
 String age = speech[speech.length-3]; 
 
 Log.e( "THIS" , "" + age); 
 
 editor.putString(AGE, age).apply(); 
 
 } 
 
 // Then ask it for your age 
 
 if(text. contains ( "how old am I" )){ 
 
 speak( "You are " +preferences.getString(AGE, null )+ " years old." ); 
 
 }

The app can tell you the time:

 //Ask: What time   is it? 
 
 if(text. contains ( "what time is it" )){ 
 
 SimpleDateFormatsdfDate = new SimpleDateFormat( "HH:mm" );//dd/MM/yyyy 
 
 Date now = new Date (); 
 
 String[] strDate = sdfDate.format(now).split( ":" ); 
 
 if(strDate[1]. contains ( "00" ))strDate[1] = "o'clock" ; 
 
 speak( "The time is " + sdfDate.format(now)); 
 
 }

summary

There are more examples in the GitHub project I created (https://github.com/sitepoint-editors/SpeechApplication), so you can fully experiment and develop your own real Android assistant application.

In the end, I hope you enjoyed this tutorial and that you were able to have a truly useful conversation with your phone.