Guest Post: Integrating your Application to work with Cortana in Windows Phone 8.1
Our second guest blog on Community Day 2014 is from our ASP.net/IIS MVP Abhishek Sur. He writes about one of the most favorite features of Windows Phone 8.1, Cortana which makes your phone your personal assistant.
The Windows Phone 8.1 comes up with a lot of interesting features, some of which are really interesting while some are just for eye candy. Among all the features that has been released, one of the most interesting and eye catching feature that people are awaited of is Cortana.
Cortana is a special app that is installed by default with Windows Phone 8.1 and automatically configures itself to run on the hardware search button. This special application invokes a Personal assistant which not just capable of helping you through various settings and invoking various tasks, it can also go beyond the boundaries of the Windows phone. Cortana is your guide, which stores all the settings that you do with your phone, creates a list of your personal choices, create alarms, and remind you something which interests you automatically. The best thing with Cortana is it understands natural language commands.
Here are some of the interesting things that Cortana can do:
1. Phone : Cortana is capable of understanding relationships and can allow you to say commands to call your friends and family. For instance, if you want you can say either “Call my wife” or “Call Riya” to do the same thing, where Riya is the contact saved in my contact list. It will prompt you if it does not know, or otherwise it will go ahead and call that person.
2. Messaging : Cortana can literally send text messages to the contacts. For instance, you can say “Text Abhijit I am busy will be calling you in short while” . This will automatically find “Abhijit” from my contact list and send this message.
3. Calendar: One of the most common issues with people is to deal with Calendar. For a very busy person, adding a calendar entry and reminding on time is quite a big deal. Cortana handles it very good. You can say “Add an appointment to Kunal tomorrow at 3 pm” , and Cortana will do just that. Also you can say, “What do I need to do next?” , and it will tell you the next calendar entry.
4. Reminders: Well, reminders are another interesting problem that people face daily. You can simply say something to remind later, and your Cortana will do just that. For instance, you can say “Remind me tomorrow to call Abhijit on his birthday”, it will add the reminder and work accordingly.
5. Notes : Cortana as it deals with natural language can also add notes. Say “Take notes: Me and Cortana is best friend” , and it takes it on its note book.
6. Alarms: Just like reminders, you can also set Alarms with Cortana. For instance, you can say “Wake me up at 6 O’clock in the morning” or “Set alarm for 6 AM” , and Cortana understands it and add an alarm.
7. Music: Oho, You can start, pause, and change music simply by saying it to Cortana. You can say “Play my music” to start music you recently played, or you can also play a specific song by saying just that.
8. Places: You can also use Places, distance between your locations with the place you say very easily with Cortana. Just say “How much is the distance to Northern Avenue” and it will get you the info.
9. Search: Cortana integrates Bing search engine. Anything and everything, it doesn’t pick up from the list will automatically go to Search. You can say anything to Cortana like “How is today’s weather” or “show me today’s headlines” and Cortana shows it from Bing.
10. App Integration: Not only dealing with Phone features, Cortana can also invoke the Apps that have been installed in your phone and do something it supports. For instance, you can say “Twitter new Tweet Today it could hashtag rain here in hashtag Kolkata need to get an umbrella quick” . Cortana will understand this, open Twitter and add “Today it could #rain here in #Kolkata, need to get an umbrella quick”.
The above image shows the Cortana APP. When you open, you will see something very similar to above. The very right side bottom of the screen is the mike which you can press once and interact with Cortana. The initial screen also allows you to change different settings which can be accessed by the button on the top right section and in the centre, Cortana greeted you with a nice message and also you can click on “see more” link to know more about it.
This is all that exists with Cortana.
Integrating your .NET App with Cortana
Now that you know that Cortana is getting popularity within the Windows Phone ecosystem, it would not be a bad idea to integrate your app with Cortana. Integrating means dealing with voice commands that is passed to your application and do something that your app is capable of. As I gave the example already, Twitter can recognise the command “new tweet” and create the new tweet after opening the Twitter app, it also replaces the hashtag with actual #.
To integrate your application with Cortana, you need to understand three things:
1. Create Voice Command Definitions.
2. Register the VCD file on Start-up.
3. Handle Voice commands when the page is navigated with it as argument.
How to do it…
To do these steps, let us first open Visual Studio 2013 and create a Blank Windows Phone App. Let’s name it as Integrating Cortana Sample. It will prompt the Target OS version, choose it and click Ok.
1. Open WMAppManifest.xml from Properties node of Solution Explorer and double click to open. Add ID_CAP_SPEECH_RECOGNITION, ID_CAP_MICROPHONE, and ID_CAP_NETWORKING as Capabilities, save the file and Close.
2. Once this is done, Right click on the Project node in Solution Explorer, and Add è New Item. Select Voice Command Definition file, name it as “CortanaCommands.vcd”, and add it to the project.
The figure shows the exact item that you need to select to add a VCD template to the solution. The Voice Command Definition is just an XML file which has a schema that Cortana can understand.
3. After the file is added to the Solution, Right click on the file (CortanaCommands.xml) è Select Properties. Choose “Build Action” to Content and choose Copy if newer for” Copy to Output Directory” property.
4. Once this is done, open the file. We will now try to understand each nodes defined inside the VCD file. Let’s see how we change the VCD file.
Let us know see individual elements of the VCD file.
a. Command Prefix: It defines a nick name through which we can call the app. For instance my APP can be named as “Great MVP Dictation Lists” while I can put a nickname as “MVP” such that I can say “MVP” then some series of content and Cortana will pass the content to the app after opening it.
b. Example: This is the example which is shown when you open the List of all apps that is supported.
c. Command: From each sentence you speak, Cortana identifies Command. For instance, in case of Twitter “Twitter new tweet …” the new tweet is the command. In our case we can just say “Text” after “MVP”.
d. Example: This is the example which is shown while opening the App.
e. Listen For: You can say some keywords in forms of square braces which would be ignored and in curly braces which is the command label which is identified when passed to the Application as text. That means whatever I say after “MVP text” goes to the variable {dictation} ignoring the create word.
f. Feedback: This is what Cortana is going to reply.
g. Navigate: This is the target page where Cortana will navigate to when app is opened.
h. Phase Topic: You can improve the experience by defining phase topic for each variable you pass to. The Phase topic could be a Dictation, Commands, Short Message, Natural Language, Search etc. It should be worth noting that “PhaseTopic” are recognised in cloud, not in the device itself. This is an optional key.
While defining each of the elements, don’t forget to hover over the element directly inside Visual Studio. The editor also provides great info as tooltip.
5. Now after the VCD file is created correctly, let us see the step to register the VCD file. The VCD file needs to be registered only once when the Application is first run. We Open App.xaml and go to Application_Launcing event to write this step.
The above code registers the VCD file hence created such that it could be identified by Cortana.
6. Finally let us add some code on MainPage.xaml to do things in real time. We add three TextBlocks on the Content area.
In the above code, the UI creates three TextBlock to show text stacked vertically.
7. As we specified Navigate to “MainPage.xaml”, the page gets automatically opened when Cortana opens the app and a command is identified. Let’s override OnNavigatedTo command with the following code:
8. In the above code, three commands need to be noticed.
a. The reco is the entire recorded Text that you say after your command.
b. The voiceCommandName is the name of the command identified by the app.
c. The {dictation} is the identified key defined on the VCD file at element ListenFor.
9. You can also notice, the text would be displayed on the nodes created on the UI.
10. Run the code, open Cortana and Say “MVP Text I love being an MVP” and see that gets displayed on the screen.
In the above figure it demonstrates how Cortana identifies the command correctly and shows in the App. The three lines define each command identified from the QueryString.
Download Source code
Conclusion
As it is open framework to integrate your app with Cortana, it is very easy to expose each and every feature of your app through Natural language, thus giving great user experience of the app to the users.
More References …
http://msdn.microsoft.com/en-us/library/windows/apps/jj206959(v=vs.105).aspx
http://msdn.microsoft.com/en-us/magazine/jj721592.aspx
I hope you like my post, do comment, and keep in touch.
Comments
- Anonymous
August 23, 2014
Great Information.. - Anonymous
August 23, 2014
Awesome post, I really liked it. I will start integrating all my existing apps with Cortana soon. - Anonymous
August 23, 2014
Thanks for the lovely post. :) - Anonymous
August 24, 2014
Thank you for sharing. - Anonymous
October 13, 2014
Thanks for keeping it concise and clear. And of course it worked like a charm!