Cortana Integration in Universal Windows Platform Apps

Article
1/17/2024

Introduction

This article is all about integrating Cortana(Virtual Voice-activated Personal Assistant) with our app.This means, when user give any instruction/ask question to Cortana related to our app or the thing which your app can fulfil. In this case Cortana communicate with our app and respond back to user accordingly.To use Cortana, your device must have proper region set with the language pack installed which Cortana understands.

Create Sample

In this article i am going to explain step by step procedure to integrate Cortana with UWP app.

Step 1 : Create Windows Universal Project

Open Visual Studio 2015 -> New Project -> Visual C# ->Windows -> Universal -> Blank App

Step 2 : Add Windows Runtime Component

Now lets add Windows Runtime Component which act as a background AppService for this app and gets activate whenever user ask anything to Cortana with the name of our app.We'll see it later.
Right Click on solution -> Add new project -> Add Windows Runtime Component

Step 3 : Add VCD file to project

Now add a Voice Command Definition (VCD) file into our app, which we need to install on app launch.
A voice command is a single utterance with a specific intent, defined in a Voice Command Definition (VCD) file, directed at an installed app through Cortana.
A VCD file defines one or more voice commands, each with a unique intent.

CortanaUWPDemo -> Add ->New Item ->XML File

Copy paste below code in your VCD file.

<?xml version="1.0" encoding="utf-8"?>
<VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.2">
 
  <!--For English US language-->
  <CommandSet xml:lang="en-us" Name="examplevcd">
    <AppName>Assistant</AppName>
    <Example>Ask your query to assistant regarding travels</Example>
    <Command Name="YourKeyword">
      <Example>I want to travel near areas</Example>
      <ListenFor>{UserCommand}</ListenFor>
      <Feedback>In progress...</Feedback>
      <VoiceCommandService Target="CortanaBackgroundTask"/>
    </Command>
    <PhraseTopic Label="UserCommand" Scenario="Natural Language">
    </PhraseTopic>
  </CommandSet>
 
  <!--For English-India Language-->
  <CommandSet xml:lang="en-in" Name="examplevcd">
    <AppName>Assistant</AppName>
    <Example>Ask your query to assistant regarding travels</Example>
    <Command Name="YourKeyword">
      <Example>I want to travel near areas</Example>
      <ListenFor>{UserCommand}</ListenFor>
      <Feedback>In progress...</Feedback>
      <VoiceCommandService Target="CortanaBackgroundTask"/>
    </Command>
    <PhraseTopic Label="UserCommand" Scenario="Natural Language">
    </PhraseTopic>
  </CommandSet>
 
</VoiceCommands>

Here, i added multiple command sets for en-us and en-in.Where,
AppName : Cortana will listen user only with this app name as a prefix of the sentence(Example : Assistant I want to travel near areas.).
Feedback : Text Shows to user while working on creating user response after user queries/ask something.
VoiceCommandService : The name of a App service which activates on calling a AppName or CommandPrefix by user.
PhraseTopic : You can add multiple phrase inside this PhraseTopic for better response and understanding the user.

Step 4 : Install VCD file

Lets write a code in our app to install this VCD file.

public async Task InstallVCD()
        {
            try
            {               
                StorageFile vcdStorageFile = await Package.Current.InstalledLocation.GetFileAsync(@"VCD.xml");
                await Windows.ApplicationModel.VoiceCommands.VoiceCommandDefinitionManager.InstallCommandDefinitionsFromStorageFileAsync(vcdStorageFile);
            }
            catch (Exception ex)
            {
 
            }
        }

Call above method from OnLaunched() of app.xaml.cs file.

Step 5 : Add reference of Windows Runtime Project

Now, we are done with making VCD file,installing VCD file from App, added a Runtime Component into a solution.It's time to give reference of background service to our app.

CortanaUWPDemo -> References -> Right Click ->Add Reference -> Project -> Solution -> Check CortanaUWPDemoBgTask -> Click Ok

Step 6 : Add Background task to Windows Runtime Project

Now the reference of Background service has been added into your UWP app.Now,we need to add BackgroundTask into Windows Runtime Component project.

CortanaUWPDemoBgTask -> Add New item ->Class -> CortanaBackgroundTask
Implements a IBackgrountTask Interface

public sealed  class CortanaBackgroundTask : IBackgroundTask
   {
       public void  Run(IBackgroundTaskInstance taskInstance)
       {
            
       }
   }

This background task(Run()) will called on every voice command by user.

Now add some code for Voice Command Connection and get the text out of it, what user said.
here is the updated code for the Background task.

public sealed  class CortanaBackgroundTask : IBackgroundTask
    {
        BackgroundTaskDeferral serviceDeferral;
        VoiceCommandServiceConnection voiceServiceConnection;
        public async void Run(IBackgroundTaskInstance taskInstance)
        {
            serviceDeferral = taskInstance.GetDeferral();
 
            taskInstance.Canceled += OnTaskCanceled;
            var triggerDetails = taskInstance.TriggerDetails as  AppServiceTriggerDetails;
 
            // This should match the uap:AppService and VoiceCommandService references from the 
            // package manifest and VCD files, respectively. Make sure we've been launched by
            // a Cortana Voice Command.
            if (triggerDetails != null && triggerDetails.Name == "CortanaBackgroundTask")
            {
                try
                {
                    voiceServiceConnection =
                        VoiceCommandServiceConnection.FromAppServiceTriggerDetails(
                            triggerDetails);
 
                    voiceServiceConnection.VoiceCommandCompleted += OnVoiceCommandCompleted;
 
                    // GetVoiceCommandAsync establishes initial connection to Cortana, and must be called prior to any 
                    // messages sent to Cortana. Attempting to use ReportSuccessAsync, ReportProgressAsync, etc
                    // prior to calling this will produce undefined behavior.
                    VoiceCommand voiceCommand = await voiceServiceConnection.GetVoiceCommandAsync();
                    var text = voiceCommand.SpeechRecognitionResult.Text;
 
                    if (text.Equals("I"))
                    {
 
                        RespondTouser("Hi, How are your doing ?");
                    }
                    else
                    {
                        //Here you call up any api's to create a meaning full response to user/call services like Bot/Luis etc... to create a meaningful response.
                        RespondTouser("You Said ," + text);
                    }
                     
                }
                catch (Exception ex)
                {
 
                }
            }
        }
        private async void RespondTouser(string text)
        {
            var userMessage = new  VoiceCommandUserMessage();
            string keepingTripToDestination = text; //How can i Help you?
            userMessage.DisplayMessage = userMessage.SpokenMessage = keepingTripToDestination;
 
            VoiceCommandResponse response = VoiceCommandResponse.CreateResponse(userMessage);
            await voiceServiceConnection.ReportSuccessAsync(response);
        }
 
        private void  OnVoiceCommandCompleted(VoiceCommandServiceConnection sender, VoiceCommandCompletedEventArgs args)
        {
            if (this.serviceDeferral != null)
            {
                this.serviceDeferral.Complete();
            }
        }
 
        private void  OnTaskCanceled(IBackgroundTaskInstance sender, BackgroundTaskCancellationReason reason)
        {
            if (this.serviceDeferral != null)
            {
                this.serviceDeferral.Complete();
            }
        }
    }

Step 7 : Define AppService in Package Manifest file

Now, we've done with our Background App Service.Let our UWP app knows about it.

Add, this below code in package.appmanifest file.

<Extensions>
        <uap:Extension Category="windows.appService" EntryPoint="CortanaUWPDemoBgTask.CortanaBackgroundTask">
          <uap:AppService Name="CortanaBackgroundTask" />
        </uap:Extension>
        <uap:Extension Category="windows.personalAssistantLaunch" />
</Extensions>

Now, we've done all the stuffs, Its time to run our app :)

Output Video

Check out here : View

Source Code

Download full source code from here.

Share via