WP8 Voice Commands: Dynamic Phrase Lists … From Azure!
Over the last couple weeks, I’ve been demonstrating various features of the new speech platform included in Windows Phone 8.0. So far, we’ve covered the following topic areas:
- Start Speech Enabling your apps! … where I consolidated a bunch of resources, and uploaded a video of a “real world” Windows Phone application that I’ve “speech enabled” myself, called Tivo Command.
- How many lines of code does it take to build a complete WP8 speech app? … where we found the answer to be a very small number. Especially considering the fact that our first version of our “Search On” sample app used Voice Commands, Speech Recognition, and Speech Synthesis.
- WP8: Voice Command Phrase Lists … building upon our “Search On” sample app, we saw how we could extend the voice commands our users can use, simply by using <PhraseList> elements inside our Voice Command Definition (VCD) xml file.
- WP8: Voice Command Wildcards … we extended our app to deal with the cases when a user said something unexpected, by using a handy Voice Command Service feature, we call wildcards (we also sometimes call it a garbage rule, or garbage model)
Today, I’m going to pick up where we left off with our “Search On” application, and demonstrate how you as “the developer” behind the app, can update the phrases that users can say to your app, from the server side of the system.
The “speech part” of this is actually quite simple. In fact, if we had a list of strings (List<string>) just laying around inside our app, it would only takes a few lines of code to update our Phrase List contained inside our VCD that’s loaded into the Voice Command Service. The harder part, truly, is getting that list from a server, hosting the list on said server, and dealing with other changes that result from the list being server side.
Since this is a predominantly “speech” blog posting, I thought I’d lead with the “speech” part of the code. If you’re one of those types that likes spoilers, feel free to scan ahead to take a look at the full solution, but I’ll try to make it worth your while to hold off until we get there. :-)
The “Speech” Part
OK. The “speech” part of updating phrase lists is pretty straight forward, like I said. Let’s look at the code…
1: private async Task UpdatePhraseListsAsync()
2: {
3: foreach (VoiceCommandSet cs in VoiceCommandService.InstalledCommandSets.Values)
4: {
5: List<string> updatedListOfPhrases = GetPhrasesForUpdatedSiteToSearchPhraseList(cs.Language.ToLower());
6: await cs.UpdatePhraseListAsync("siteToSearch", updatedListOfPhrases);
7: }
8: }
If you recall, a VCD file can have one or more <CommandSet> elements, representing different “languages” or “dialects”. Our VCD file only has one, for “en-US” at the moment. I’ll probably post an update to our “Search On” sample app in the future, enabling voice commands in more than one language, and thus containing more than one <CommandSet>. But for now, we just have one. Having said that, I’m going to go ahead and write the code for updating the phrase lists in a language neutral way, such that it should just work once that future post actually gets written.
To update the phrase list, as you can see, I’ve created a new method, called UpdatePhraseListsAsync, which simply iterates through all the installed Command Sets, gets a list of strings (the phrases users can say), and then calls UpdatePhraseListAsync on the VoiceCommandSet object we got from the VoiceCommandService.
One caveat I want to warn you about, though. Even if you’ve successfully installed your Command Sets in the VoiceCommandService, the InstalledCommandSets collection may be empty. That happens if you didn’t include a “Name” on the <CommandSet> in your VCD file. We did this because the collection is actually a Dictionary, and thus, you have to have a “key” to look up the “value”. No key, No value. Thus, we have to update our VCD as follows:
6: <CommandSet xml:lang="en-US" Name="commands-en-us">
Great. The speech part is done. Let’s run it! … Wait. We have a lot of other changes. Like, getting the phrases for the siteToSearch phrase list. Let’s take a look at that method by method.
First up, GetPhrasesForUpdatedSiteToSearchPhraseList:
1: private List<string> GetPhrasesForUpdatedSiteToSearchPhraseList(string language)
2: {
3: List<string> phrasesBuiltIn = GetPhrasesForUpdatedSiteToSearchPhraseListFromBuiltInList();
4: List<string> phrasesFromAzure = GetPhrasesForUpdatedSiteToSearchPhraseListFromAzure(language);
5:
6: return phrasesFromAzure.Union(phrasesBuiltIn).ToList();
7: }
First, we get a list of phrases rom the built in list, then a list of phrases from Azure, and then we returned the combined list. Neat, eh?
Wait… Did I just say from Azure? Yeah. I did.
The “Hosting in Azure” Part
While I was considering where to store my list, I remembered that some of my friends over in the Azure team just released a really cool new feature called Azure Mobile Services. You can read about it here from the original announcement in August on ScottGu’s blog, or here from the Build announcement / blog post a couple weeks ago.
There are many things I love about Azure Mobile Services. But I’ll only list a couple here: They have a free 90 day trial; the To Do sample app on their quick start page makes it brain dead easy to build your first app with data hosted in azure; there are APIs for Windows 8, Windows Phone 8, and REST based APIs as well.
OK. You can either follow along here and use my Azure Mobile Service, (hosted here), or you can sign up for your own free trial, and have your own server side solution for your version of our “Search On” sample application.
Setting up your Azure Mobile Service
To set up my Azure Mobile Service. Here’s what I did, step by step (you can do the same, once you have an account):
Log in to the Azure portal here.
Click on the Mobile Services tab:
Click “New” to create a new Mobile Service.
Click “Compute”, “Mobile Service”, and then “Create”. That should land you here:
Pick a URL (I picked robchblogsamples), select “Create a new database”, and pick your subscription. Then click the “Next” arrow.
Confirm the name of your new database, and then fill in the credentials you want to use for your new sql database. When you’re done, click the “Complete” check mark in the lower right hand corner.
That’s it! You now have your very own Azure Mobile Service.
I’d suggest you play with the WP8 To Do application for a few minutes, by clicking on the “Create a new Windows Phone 8” application link on the quick start tab you’re already sitting on … Once you’ve played around with that, come back here…
OK… Was that fun? Seems like you were only gone for a few seconds. :-)
Now, without going into a lot of details today, here’s the quick version of what I did to populate my Azure Mobile Service with a table filled with “rows” of data, that represent different “sites” that the user can search using our “Search On” application:
- Create a new table, called SiteToSearchItem
- Set the permissions on that table such that “Everyone” has read permission
- Modify the WP8 To Do app you just played with, to add “sites” instead of To Do items
- And finally … use that app to add some sites.
Make sense? OK … Let’s do that step by step:
Creating the table, setting the permissions
Create the new table, called SiteToSearchItem, by first, clicking on the “Data” tab… Then click the “Create” button at the bottom, and fill out the fields appropriately:
After clicking the “Complete” check mark in the lower right hand corner, your table will be created. It’ll just take a few seconds.
Now click your new “SiteToSearchItem” table. No data! Well … that makes sense. We haven’t added any yet. Let’s do that now.
Modify the WP8 app to add “sites”
Now, we’re going to modify the WP8 To Do app from the quick start tab in your new Azure Mobile Service, to add “sites” instead of “to do” items. At the highest level, here’s what I did to change that app in that way:
In MainPage.xaml.cs:
- Renamed the ToDoItem class into a new SiteToSearchItem class.
- Removed the two existing “To Do” data members from that class that were “To Do” specific.
- Added public get/set properties for SiteName, Domain, SearchUrlTemplate, and Language.
- Examples for those values would be:
- SiteName = “Bing”, Domain = “www.bing.com”, SearchUrlTemplate = “https://www.bing.com/search?q={0}”, and Language = “en-US”
In MainPage.xaml:
- Removed Checkbox, and To Do Textbox
- Added Textboxes for SiteName, Domain, SearchUrlTemplate, and Language
- Added a ScrollViewer, and Reformatted to make more usable
Here’s what that ended up looking like, in MainPage.xaml.cs:
1: using System;
2: using System.Collections.Generic;
3: using System.Linq;
4: using System.Net;
5: using System.Runtime.Serialization;
6: using System.Windows;
7: using System.Windows.Controls;
8: using System.Windows.Navigation;
9: using Microsoft.Phone.Controls;
10: using Microsoft.Phone.Shell;
11: using robchblogsamples.Resources;
12:
13: using Microsoft.WindowsAzure.MobileServices;
14:
15: namespace robchblogsamples
16: {
17: public class SiteToSearchItem
18: {
19: public int Id { get; set; }
20:
21: [DataMember(Name = "SiteName")]
22: public string SiteName { get; set; }
23:
24: [DataMember(Name = "Language")]
25: public string Language{ get; set; }
26:
27: [DataMember(Name = "Domain")]
28: public string Domain { get; set; }
29:
30: [DataMember(Name = "SearchUrlTemplate")]
31: public string SearchUrlTemplate { get; set; }
32: }
33:
34: public partial class MainPage : PhoneApplicationPage
35: {
36: private MobileServiceCollectionView<SiteToSearchItem> items;
37:
38: private IMobileServiceTable<SiteToSearchItem> _table = App.MobileService.GetTable<SiteToSearchItem>();
39:
40: public MainPage()
41: {
42: InitializeComponent();
43: }
44:
45: private async void InsertItem(SiteToSearchItem item)
46: {
47: await _table.InsertAsync(item);
48: items.Add(item);
49: }
50:
51: private void RefreshItems()
52: {
53: items = _table
54: .Where(item => item.Language == "en-us")
55: .ToCollectionView();
56: ListItems.ItemsSource = items;
57: }
58:
59: private async void UpdateItem(SiteToSearchItem item)
60: {
61: await _table.UpdateAsync(item);
62: items.Remove(item);
63: }
64:
65: private void ButtonRefresh_Click(object sender, RoutedEventArgs e)
66: {
67: RefreshItems();
68: }
69:
70: private void ButtonSave_Click(object sender, RoutedEventArgs e)
71: {
72: var item = new SiteToSearchItem
73: {
74: SiteName = _textboxSiteName.Text,
75: Language = _textboxLanguage.Text,
76: Domain = _textblockDomain.Text,
77: SearchUrlTemplate = _textboxUrlTemplate.Text
78: };
79:
80: InsertItem(item);
81: }
82:
83: protected override void OnNavigatedTo(NavigationEventArgs e)
84: {
85: RefreshItems();
86: }
87: }
88: }
…. and this in MainPage.xaml:
1: <phone:PhoneApplicationPage
2: x:Class="robchblogsamples.MainPage"
3: xmlns="https://schemas.microsoft.com/winfx/2006/xaml/presentation"
4: xmlns:x="https://schemas.microsoft.com/winfx/2006/xaml"
5: xmlns:phone="clr-namespace:Microsoft.Phone.Controls;assembly=Microsoft.Phone"
6: xmlns:shell="clr-namespace:Microsoft.Phone.Shell;assembly=Microsoft.Phone"
7: xmlns:d="https://schemas.microsoft.com/expression/blend/2008"
8: xmlns:mc="https://schemas.openxmlformats.org/markup-compatibility/2006"
9: mc:Ignorable="d"
10: FontFamily="{StaticResource PhoneFontFamilyNormal}"
11: FontSize="{StaticResource PhoneFontSizeNormal}"
12: Foreground="{StaticResource PhoneForegroundBrush}"
13: SupportedOrientations="Portrait" Orientation="Portrait"
14: shell:SystemTray.IsVisible="True">
15:
16: <!--LayoutRoot is the root grid where all page content is placed-->
17: <Grid x:Name="LayoutRoot" Background="Transparent">
18: <Grid.RowDefinitions>
19: <RowDefinition Height="Auto"/>
20: <RowDefinition Height="*"/>
21: </Grid.RowDefinitions>
22:
23: <!--TitlePanel contains the name of the application and page title-->
24: <StackPanel x:Name="TitlePanel" Grid.Row="0" Margin="12,17,0,28">
25: <TextBlock Text="robchblogsamples" Style="{StaticResource PhoneTextNormalStyle}" Margin="12,0"/>
26: </StackPanel>
27: <ScrollViewer Grid.Row="1" Margin="12,0,12,0">
28: <StackPanel x:Name="ContentPanel" Width="456">
29: <TextBlock Text="Enter some text below and click Save to insert a new SiteToSearchItem into your database" TextWrapping="Wrap" Margin="12,0" Height="54"/>
30: <StackPanel>
31: <StackPanel Orientation="Horizontal" HorizontalAlignment="Right">
32: <TextBlock TextWrapping="Wrap" Text="Site Name:" HorizontalAlignment="Center" VerticalAlignment="Center" FontSize="21.333"/>
33: <TextBox x:Name="_textboxSiteName" Text="" Width="300" />
34: </StackPanel>
35: <StackPanel Orientation="Horizontal" HorizontalAlignment="Right">
36: <TextBlock TextWrapping="Wrap" Text="Language:" HorizontalAlignment="Center" VerticalAlignment="Center" FontSize="21.333"/>
37: <TextBox x:Name="_textboxLanguage" Text="" Width="300" />
38: </StackPanel>
39: <StackPanel Orientation="Horizontal" HorizontalAlignment="Right">
40: <TextBlock TextWrapping="Wrap" HorizontalAlignment="Center" VerticalAlignment="Center" FontSize="21.333" Text="Domain:"/>
41: <TextBox x:Name="_textblockDomain" Text="" Width="300" />
42: </StackPanel>
43: <StackPanel Orientation="Horizontal" HorizontalAlignment="Right">
44: <TextBlock TextWrapping="Wrap" Text="Url Template:" HorizontalAlignment="Center" VerticalAlignment="Center" FontSize="21.333"/>
45: <TextBox x:Name="_textboxUrlTemplate" Text="" Width="300" />
46: </StackPanel>
47: </StackPanel>
48: <Button x:Name="ButtonSave" Click="ButtonSave_Click" Content="Save" Margin="0"/>
49: <TextBlock Text="Click refresh below to load the unfinished TodoItems from your database. Use the checkbox to complete and update your TodoItems" TextWrapping="Wrap" Margin="12,36,12,0" Height="107" />
50: <Button x:Name="ButtonRefresh" Click="ButtonRefresh_Click" Content="Refresh" Height="72"/>
51: <phone:LongListSelector x:Name="ListItems" Height="400">
52: <phone:LongListSelector.ItemTemplate>
53: <DataTemplate>
54: <StackPanel>
55: <TextBlock TextWrapping="Wrap" Text="{Binding SiteName, FallbackValue=[SiteName]}" FontSize="32"/>
56: <TextBlock TextWrapping="Wrap" Text="{Binding Language, FallbackValue=[Language]}" FontSize="16"/>
57: <TextBlock TextWrapping="Wrap" Text="{Binding Domain, FallbackValue=[Domain]}" FontSize="16"/>
58: <TextBlock TextWrapping="Wrap" Text="{Binding SearchUrlTemplate, FallbackValue=[SearchUrlTemplate]}" FontSize="16"/>
59: </StackPanel>
60: </DataTemplate>
61: </phone:LongListSelector.ItemTemplate>
62: </phone:LongListSelector>
63: </StackPanel>
64:
65: </ScrollViewer>
66:
67: <!--ContentPanel - place additional content here-->
68: </Grid>
69: </phone:PhoneApplicationPage>
Add some sites
Press F5, and add some sites. :-)
Remember … It’s a very simplistic “Admin UI”… For example, there is no delete. If you need to delete something, you can either:
- Add that feature yourself, or
- Use the Azure Portal to delete your SiteToSearchItem table, re-add the table, and then re-add the sites from our “Admin UI”
The “Other” part…
OK. Got that Azure part all done …
Back to our “Search On” application...
Now it’s time to pull the data down from Azure, in our new GetPhrasesForUpdatedSiteToSearchPhraseListFromAzure method. Basically, this method delegates to another method, called GetItemsFromAzureMobileService, to download the list of “SiteToSearchItem” entries. Once we have our list of sites in a structured class, we’ll use LINQ to populate a List of strings for our phrase list, by adding all the Site Name’s (e.g. “Bing”) via the “SiteName” property from all the “SiteToSearchItem” items in the list of items from Azure.
1: private List<string> GetPhrasesForUpdatedSiteToSearchPhraseListFromAzure(string language)
2: {
3: List<SiteToSearchItem> itemsFromAzure =
4: GetItemsFromAzureMobileService<SiteToSearchItem>(
5: string.Format(_siteToSearchAzureMobileServicesTableUrlTemplate, language),
6: _downloadTimeout);
7:
8: List<string> phrasesFromAzure = new List<string>(
9: from item in itemsFromAzure
10: select item.SiteName);
11:
12: return phrasesFromAzure;
13: }
Here’s what our SiteToSearchItem looks like:
1: public class SiteToSearchItem
2: {
3: public int Id { get; set; }
4:
5: public string Domain { get; set; }
6: public string SiteName { get; set; }
7:
8: public string SearchUrlTemplate { get; set; }
9:
10: public string Language { get; set; }
11: }
Now, since we have a data structure to keep our sites in, instead of a dictionary of strings, let’s use that for our built-in list of sites to search:
1: private List<SiteToSearchItem> _sitesToSearchBuiltInList = new List<SiteToSearchItem>()
2: {
3: { new SiteToSearchItem()
4: { Language = "en-us",
5: SiteName = "Amazon",
6: Domain = "www.amazon.com",
7: SearchUrlTemplate = "https://www.amazon.com/gp/aw/s/ref=is_box_?k={0}" } },
8:
9: { new SiteToSearchItem()
10: { Language = "en-us",
11: SiteName = "Bing",
12: Domain = "www.bing.com",
13: SearchUrlTemplate = "https://www.bing.com/?q={0}" } },
14:
15: { new SiteToSearchItem()
16: { Language = "en-us",
17: SiteName = "CNN",
18: Domain = "www.CNN.com",
19: SearchUrlTemplate = "https://www.cnn.com/search/?query={0}" } },
20:
21: { new SiteToSearchItem()
22: { Language = "en-us",
23: SiteName = "zzzz",
24: Domain = "www.zzzz.com",
25: SearchUrlTemplate = "zzzz" } },
26:
27: { new SiteToSearchItem()
28: { Language = "en-us",
29: SiteName = "Dictionary.com",
30: Domain = "www.dictionary.com",
31: SearchUrlTemplate = "https://dictionary.reference.com/browse/{0}" } },
32:
33: { new SiteToSearchItem()
34: { Language = "en-us",
35: SiteName = "Ebay",
36: Domain = "www.ebay.com",
37: SearchUrlTemplate = "https://www.ebay.com/sch/i.html?_nkw={0}" } },
38:
39: { new SiteToSearchItem()
40: { Language = "en-us",
41: SiteName = "Facebook",
42: Domain = "www.facebook.com",
43: SearchUrlTemplate = "https://www.facebook.com/search/results.php?q={0}" } },
44:
45: { new SiteToSearchItem()
46: { Language = "en-us",
47: SiteName = "Google",
48: Domain = "www.google.com",
49: SearchUrlTemplate = "https://www.google.com/search?hl=en&q={0}" } },
50:
51: { new SiteToSearchItem()
52: { Language = "en-us",
53: SiteName = "Hulu",
54: Domain = "www.hulu.com",
55: SearchUrlTemplate = "https://www.hulu.com/#!search?q={0}" } },
56:
57: { new SiteToSearchItem()
58: { Language = "en-us",
59: SiteName = "IMDB",
60: Domain = "www.imdb.com",
61: SearchUrlTemplate = "https://www.imdb.com/find?s=all&q={0}" } },
62:
63: { new SiteToSearchItem()
64: { Language = "en-us",
65: SiteName = "Linked In",
66: Domain = "www.linkedin.com",
67: SearchUrlTemplate = "https://www.linkedin.com/search/fpsearch?keywords={0}" } },
68:
69: { new SiteToSearchItem()
70: { Language = "en-us",
71: SiteName = "MSN",
72: Domain = "www.msn.com",
73: SearchUrlTemplate = "https://www.bing.com/search?scope=msn&q={0}" } },
74:
75: { new SiteToSearchItem()
76: { Language = "en-us",
77: SiteName = "Twitter",
78: Domain = "www.twitter.com",
79: SearchUrlTemplate = "https://twitter.com/search/{0}" } },
80:
81: { new SiteToSearchItem()
82: { Language = "en-us",
83: SiteName = "Weather.com",
84: Domain = "www.weather.com",
85: SearchUrlTemplate = "https://www.weather.com/info/sitesearch?q={0}" } },
86:
87: { new SiteToSearchItem()
88: { Language = "en-us",
89: SiteName = "You Tube",
90: Domain = "www.youtube.com",
91: SearchUrlTemplate = "https://www.youtube.com/results?search_query={0}" } },
92:
93: { new SiteToSearchItem()
94: { Language = "en-us",
95: SiteName = "Zillow",
96: Domain = "www.zillow.com",
97: SearchUrlTemplate = "https://www.zillow.com/homes/{0}/" } }
98: };
Similar to the GetPhrasesForUpdatedSiteToSearchPhraseListFromAzure method, we’ll use the same LINQ technique to populate our list of phrases for the built-in sites, that we’ve just defined.
1: private List<string> GetPhrasesForUpdatedSiteToSearchPhraseListFromBuiltInList()
2: {
3: return new List<string>(
4: from item in _sitesToSearchBuiltInList
5: select item.SiteName);
6: }
On to the GetItemsFromAzureMobileService method…
There are several ways we can do this. For example, we could use the Azure Mobile Services client library, just like the To Do List (which we turned into our fancy Admin UI app). Or … we could use the REST end points that expose all the data in our tables as an OData end point.
Let’s use the REST-ful approach. I picked this method, because you might not end up using Azure Mobile Services, and you can get more out of copying this code if you go another route. You could download your data from anywhere you can host a JSON list.
Just download the data from the endpoint and convert it to a List of SiteToSearchItem instances.
1: private List<T> GetItemsFromAzureMobileService<T>(string url, int downloadTimeout)
2: {
3: string json = DownloadString(url, downloadTimeout);
4: List<T> itemsFromJson = FromJSON<List<T>>(json != null ? json : "[]");
5: return itemsFromJson;
6: }
Now, let’s add the two new functions that’ll make that work, DownloadString, and FromJSON:
1: private string DownloadString(string uri, int timeoutInMilliseconds)
2: {
3: string result = null;
4:
5: ManualResetEvent webrequestCompleted = new ManualResetEvent(false);
6: AsyncCallback ac = new AsyncCallback((IAsyncResult iar) =>
7: {
8: try
9: {
10: HttpWebRequest request = (HttpWebRequest)iar.AsyncState;
11: HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(iar);
12: using (StreamReader sr = new StreamReader(response.GetResponseStream()))
13: {
14: result = sr.ReadToEnd();
15: }
16: response.Close();
17: }
18: catch (Exception)
19: {
20: }
21: finally
22: {
23: webrequestCompleted.Set();
24: }
25: });
26:
27: HttpWebRequest webrequest = (HttpWebRequest)HttpWebRequest.Create(new Uri(uri));
28: IAsyncResult ar = webrequest.BeginGetResponse(ac, webrequest);
29: if (ar.CompletedSynchronously)
30: {
31: ac.Invoke(ar);
32: }
33:
34: if (webrequestCompleted.WaitOne(timeoutInMilliseconds))
35: {
36: }
37:
38: return result;
39: }
40:
41: private static T FromJSON<T>(string json)
42: {
43: using (MemoryStream ms = new MemoryStream(Encoding.UTF8.GetBytes(json)))
44: {
45: DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(T));
46: return (T)serializer.ReadObject(ms);
47: }
48: }
Since we’re getting our data from the server, we also have to change how we find the URL template given a site name. Here’s how we’re going to do that now:
1: private string GetUrlTemplateForSiteName(string siteName, string language)
2: {
3: string url;
4:
5: if (null == (url = GetUrlTemplateForSiteNameFromBuiltInList(siteName, language)) &&
6: null == (url = GetUrlTemplateForSiteNameFromAzure(siteName, language)))
7: {
8: url = GetUrlTemplateForUnknownSite(siteName);
9: }
10:
11: return url;
12: }
13:
14: private string GetUrlTemplateForSiteNameFromBuiltInList(string siteName, string language)
15: {
16: return GetUrlTemplateForSiteNameFromList(siteName, language, _sitesToSearchBuiltInList);
17: }
18:
19: private string GetUrlTemplateForSiteNameFromAzure(string siteName, string language)
20: {
21: List<SiteToSearchItem> itemsFromAzure =
22: GetItemsFromAzureMobileService<SiteToSearchItem>(
23: string.Format(_siteToSearchAzureMobileServicesTableUrlTemplate, language),
24: _downloadTimeout);
25:
26: return GetUrlTemplateForSiteNameFromList(siteName, language, itemsFromAzure);
27: }
28:
29: private string GetUrlTemplateForSiteNameFromList(string siteName, string language, List<SiteToSearchItem> items)
30: {
31: List<SiteToSearchItem> matchingItems = new List<SiteToSearchItem>(
32: from item in items
33: where item.SiteName.ToLower() == siteName.ToLower() && item.Language == language
34: select item);
35:
36: return matchingItems.Count > 0
37: ? GetUrlTemplateForSiteToSearch(matchingItems.First())
38: : null;
39: }
40:
41: private string GetUrlTemplateForSiteToSearch(SiteToSearchItem item)
42: {
43: return string.IsNullOrWhiteSpace(item.SearchUrlTemplate)
44: ? GetUrlTemplateForUnknownSite(item.Domain)
45: : item.SearchUrlTemplate;
46: }
47:
48: private string GetUrlTemplateForUnknownSite(string siteName)
49: {
50: return siteName.Length > 4 && siteName[siteName.Length - 4] != '.'
51: ? string.Format(_defaultUrlTemplateForUnknownSites, siteName + "%20{0}")
52: : string.Format(_defaultUrlTemplateForUnknownSites, "site:" + siteName + "%20{0}");
53: }
As a performance optimization, since getting the URL template actually blocks on a network operation now (to fetch the data from Azure), let’s start that operation prior to letting the user know that we’re about to do the search. Like this (check out lines 8, 10, and 12):
1: private async void SearchSiteVoiceCommand(IDictionary<string, string> queryString)
2: {
3: string siteName, findText;
4:
5: if (null != (siteName = await GetSiteName(queryString)) &&
6: null != (findText = await GetTextToFindOnSite(queryString, siteName)))
7: {
8: Task<string> asyncGetUrlTemplate = Task.Run<string>(() =>
{ return GetUrlTemplateForSiteName(siteName, GetSpeechRecognitionLanguage()); });
9:
10: await Speak(string.Format("Searching {0} for {1}", siteName, findText));
11:
12: string siteUrlTemplate = await asyncGetUrlTemplate;
13: NavigateToUrl(string.Format(siteUrlTemplate, findText, siteName));
14: }
15: }
We’re almost ready to try it out … Almost. Just a few more changes…
Now, you might be wondering, who actually calls UpdatePhraseListsAsync? Great question. Copy this code into place, and you’ll know the answer:
1: private void EnsureInitVoiceCommandsOnBackgroundThread()
2: {
3: Task.Run(() => EnsureInitVoiceCommands());
4: }
5:
6: private async void EnsureInitVoiceCommands()
7: {
8: await VoiceCommandService.InstallCommandSetsFromFileAsync(new Uri("ms-appx:///vcd.xml"));
9: await UpdatePhraseListsAsync();
10: }
11:
But, didn’t we already install the command sets from the vcd file in the MainPage constructor? Yep, we used to. But now, we’ll be doing that from the HandleNonVoiceCommandInvocation method for now.
1: private async void HandleNonVoiceCommandInvocation()
2: {
3: EnsureInitVoiceCommandsOnBackgroundThread();
4:
5: await Speak("What site would you like to search?");
6: SearchSiteVoiceCommand(new Dictionary<string, string>());
7: }
That’ll work, because, the first run of the app will go thru this method. Ideally, we’d really do something like this from a Background agent, PeriodTask like implementation, but, we’ll save that for another day.
Now we’re ready to try it out. Go for it. F5!!
Did it work? Put a breakpoint on the UpdatePhraseListsAsync method, and step thru…
Here’s the full listing for Search On…
MainPage.xaml.cs:
using System;
using System.Collections.Generic;
using System.Net;
using System.Windows;
using System.Windows.Navigation;
using Microsoft.Phone.Controls;
using Windows.Phone.Speech.VoiceCommands;
using Windows.Phone.Speech.Recognition;
using Windows.Phone.Speech.Synthesis;
using System.Threading.Tasks;
using Microsoft.Phone.Tasks;
using System.Text.RegularExpressions;
using System.Threading;
using System.Text;
using System.Linq;
using System.Runtime.Serialization.Json;
using System.IO;
namespace SearchOn
{
public partial class MainPage : PhoneApplicationPage
{
public MainPage()
{
InitializeComponent();
}
protected override void OnNavigatedTo(NavigationEventArgs e)
{
base.OnNavigatedTo(e);
if (e.NavigationMode == NavigationMode.New && NavigationContext.QueryString.ContainsKey("voiceCommandName"))
{
HandleVoiceCommand(NavigationContext.QueryString);
}
else if (e.NavigationMode == NavigationMode.New)
{
HandleNonVoiceCommandInvocation();
}
else if (e.NavigationMode == NavigationMode.Back && !System.Diagnostics.Debugger.IsAttached)
{
NavigationService.GoBack();
}
}
private async void HandleNonVoiceCommandInvocation()
{
EnsureInitVoiceCommandsOnBackgroundThread();
await Speak("What site would you like to search?");
SearchSiteVoiceCommand(new Dictionary<string, string>());
}
private void HandleVoiceCommand(IDictionary<string, string> queryString)
{
switch (queryString["voiceCommandName"])
{
case "searchSite":
case "searchSiteCatchAll":
SearchSiteVoiceCommand(queryString);
break;
}
}
private async void SearchSiteVoiceCommand(IDictionary<string, string> queryString)
{
string siteName, findText;
if (null != (siteName = await GetSiteName(queryString)) &&
null != (findText = await GetTextToFindOnSite(queryString, siteName)))
{
Task<string> asyncGetUrlTemplate = Task.Run<string>(() => { return GetUrlTemplateForSiteName(siteName, GetSpeechRecognitionLanguage()); });
await Speak(string.Format("Searching {0} for {1}", siteName, findText));
string siteUrlTemplate = await asyncGetUrlTemplate;
NavigateToUrl(string.Format(siteUrlTemplate, findText, siteName));
}
}
private async Task<string> GetSiteName(IDictionary<string, string> queryString)
{
return queryString.ContainsKey("siteToSearch")
? queryString["siteToSearch"]
: NormalizeDotComSuffixes(
await RecognizeTextFromWebSearchGrammar("Ex. \"msdn blogs\""));
}
private async Task<string> GetTextToFindOnSite(IDictionary<string, string> queryString, string siteName)
{
if (!queryString.ContainsKey("siteToSearch"))
{
await Speak(string.Format("Search {0} for what?", siteName));
}
return await RecognizeTextFromWebSearchGrammar("Ex. \"electronics\"");
}
private async Task<string> RecognizeTextFromWebSearchGrammar(string exampleText)
{
string text = null;
try
{
SpeechRecognizerUI sr = new SpeechRecognizerUI();
sr.Recognizer.Grammars.AddGrammarFromPredefinedType("web", SpeechPredefinedGrammar.WebSearch);
sr.Settings.ListenText = "Listening...";
sr.Settings.ExampleText = exampleText;
sr.Settings.ReadoutEnabled = false;
sr.Settings.ShowConfirmation = false;
SpeechRecognitionUIResult result = await sr.RecognizeWithUIAsync();
if (result != null &&
result.ResultStatus == SpeechRecognitionUIStatus.Succeeded &&
result.RecognitionResult != null &&
result.RecognitionResult.TextConfidence != SpeechRecognitionConfidence.Rejected)
{
text = result.RecognitionResult.Text;
}
}
catch
{
}
return text;
}
private string GetSpeechRecognitionLanguage()
{
return InstalledSpeechRecognizers.Default.Language.ToLower();
}
private async Task Speak(string text)
{
SpeechSynthesizer tts = new SpeechSynthesizer();
await tts.SpeakTextAsync(text);
}
private void NavigateToUrl(string url)
{
WebBrowserTask task = new WebBrowserTask();
task.Uri = new Uri(url, UriKind.Absolute);
task.Show();
}
private string GetUrlTemplateForSiteName(string siteName, string language)
{
string url;
if (null == (url = GetUrlTemplateForSiteNameFromBuiltInList(siteName, language)) &&
null == (url = GetUrlTemplateForSiteNameFromAzure(siteName, language)))
{
url = GetUrlTemplateForUnknownSite(siteName);
}
return url;
}
private string GetUrlTemplateForSiteNameFromBuiltInList(string siteName, string language)
{
return GetUrlTemplateForSiteNameFromList(siteName, language, _sitesToSearchBuiltInList);
}
private string GetUrlTemplateForSiteNameFromAzure(string siteName, string language)
{
List<SiteToSearchItem> itemsFromAzure =
GetItemsFromAzureMobileService<SiteToSearchItem>(
string.Format(_siteToSearchAzureMobileServicesTableUrlTemplate, language),
_downloadTimeout);
return GetUrlTemplateForSiteNameFromList(siteName, language, itemsFromAzure);
}
private string GetUrlTemplateForSiteNameFromList(string siteName, string language, List<SiteToSearchItem> items)
{
List<SiteToSearchItem> matchingItems = new List<SiteToSearchItem>(
from item in items
where item.SiteName.ToLower() == siteName.ToLower() && item.Language == language
select item);
return matchingItems.Count > 0
? GetUrlTemplateForSiteToSearch(matchingItems.First())
: null;
}
private string GetUrlTemplateForSiteToSearch(SiteToSearchItem item)
{
return string.IsNullOrWhiteSpace(item.SearchUrlTemplate)
? GetUrlTemplateForUnknownSite(item.Domain)
: item.SearchUrlTemplate;
}
private string GetUrlTemplateForUnknownSite(string siteName)
{
return siteName.Length > 4 && siteName[siteName.Length - 4] != '.'
? string.Format(_defaultUrlTemplateForUnknownSites, siteName + "%20{0}")
: string.Format(_defaultUrlTemplateForUnknownSites, "site:" + siteName + "%20{0}");
}
private string NormalizeDotComSuffixes(string input)
{
Regex re1 = new Regex(@"\s+dot\s+(?<suffix>(com|net|org|gov))\z");
if (input != null && re1.IsMatch(input))
{
input = re1.Replace(input, @".${suffix}").Replace(" ", "");
}
Regex re2 = new Regex(@"\s+dot\s+(?<l1>[A-Z])[.]\s+(?<l2>[A-Z])[.]\s+(?<l3>[A-Z])[.]\z");
if (input != null && re2.IsMatch(input))
{
input = re2.Replace(input, @".${l1}${l2}${l3}").Replace(" ", "");
input = input.Substring(0, input.Length - 4) + input.Substring(input.Length - 4).ToLower();
}
return input;
}
private void EnsureInitVoiceCommandsOnBackgroundThread()
{
Task.Run(() => EnsureInitVoiceCommands());
}
private async void EnsureInitVoiceCommands()
{
await VoiceCommandService.InstallCommandSetsFromFileAsync(new Uri("ms-appx:///vcd.xml"));
await UpdatePhraseListsAsync();
}
private async Task UpdatePhraseListsAsync()
{
foreach (VoiceCommandSet cs in VoiceCommandService.InstalledCommandSets.Values)
{
List<string> updatedListOfPhrases = GetPhrasesForUpdatedSiteToSearchPhraseList(cs.Language.ToLower());
await cs.UpdatePhraseListAsync("siteToSearch", updatedListOfPhrases);
}
}
private List<string> GetPhrasesForUpdatedSiteToSearchPhraseList(string language)
{
List<string> phrasesBuiltIn = GetPhrasesForUpdatedSiteToSearchPhraseListFromBuiltInList();
List<string> phrasesFromAzure = GetPhrasesForUpdatedSiteToSearchPhraseListFromAzure(language);
return phrasesFromAzure.Union(phrasesBuiltIn).ToList();
}
private List<string> GetPhrasesForUpdatedSiteToSearchPhraseListFromBuiltInList()
{
return new List<string>(
from item in _sitesToSearchBuiltInList
select item.SiteName);
}
private List<string> GetPhrasesForUpdatedSiteToSearchPhraseListFromAzure(string language)
{
List<SiteToSearchItem> itemsFromAzure =
GetItemsFromAzureMobileService<SiteToSearchItem>(
string.Format(_siteToSearchAzureMobileServicesTableUrlTemplate, language),
_downloadTimeout);
List<string> phrasesFromAzure = new List<string>(
from item in itemsFromAzure
select item.SiteName);
return phrasesFromAzure;
}
private List<T> GetItemsFromAzureMobileService<T>(string url, int downloadTimeout)
{
string json = DownloadString(url, downloadTimeout);
List<T> itemsFromJson = FromJSON<List<T>>(json != null ? json : "[]");
return itemsFromJson;
}
private string DownloadString(string uri, int timeoutInMilliseconds)
{
string result = null;
ManualResetEvent webrequestCompleted = new ManualResetEvent(false);
AsyncCallback ac = new AsyncCallback((IAsyncResult iar) =>
{
try
{
HttpWebRequest request = (HttpWebRequest)iar.AsyncState;
HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(iar);
using (StreamReader sr = new StreamReader(response.GetResponseStream()))
{
result = sr.ReadToEnd();
}
response.Close();
}
catch (Exception)
{
}
finally
{
webrequestCompleted.Set();
}
});
HttpWebRequest webrequest = (HttpWebRequest)HttpWebRequest.Create(new Uri(uri));
IAsyncResult ar = webrequest.BeginGetResponse(ac, webrequest);
if (ar.CompletedSynchronously)
{
ac.Invoke(ar);
}
if (webrequestCompleted.WaitOne(timeoutInMilliseconds))
{
}
return result;
}
private static T FromJSON<T>(string json)
{
using (MemoryStream ms = new MemoryStream(Encoding.UTF8.GetBytes(json)))
{
DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(T));
return (T)serializer.ReadObject(ms);
}
}
#region private data
private string _defaultUrlTemplateForUnknownSites = "https://m.bing.com/search?q={0}";
private List<SiteToSearchItem> _sitesToSearchBuiltInList = new List<SiteToSearchItem>()
{
{ new SiteToSearchItem()
{ Language = "en-us",
SiteName = "Amazon",
Domain = "www.amazon.com",
SearchUrlTemplate = "https://www.amazon.com/gp/aw/s/ref=is_box_?k={0}" } },
{ new SiteToSearchItem()
{ Language = "en-us",
SiteName = "Bing",
Domain = "www.bing.com",
SearchUrlTemplate = "https://www.bing.com/?q={0}" } },
{ new SiteToSearchItem()
{ Language = "en-us",
SiteName = "CNN",
Domain = "www.CNN.com",
SearchUrlTemplate = "https://www.cnn.com/search/?query={0}" } },
{ new SiteToSearchItem()
{ Language = "en-us",
SiteName = "zzzz",
Domain = "www.zzzz.com",
SearchUrlTemplate = "zzzz" } },
{ new SiteToSearchItem()
{ Language = "en-us",
SiteName = "Dictionary.com",
Domain = "www.dictionary.com",
SearchUrlTemplate = "https://dictionary.reference.com/browse/{0}" } },
{ new SiteToSearchItem()
{ Language = "en-us",
SiteName = "Ebay",
Domain = "www.ebay.com",
SearchUrlTemplate = "https://www.ebay.com/sch/i.html?_nkw={0}" } },
{ new SiteToSearchItem()
{ Language = "en-us",
SiteName = "Facebook",
Domain = "www.facebook.com",
SearchUrlTemplate = "https://www.facebook.com/search/results.php?q={0}" } },
{ new SiteToSearchItem()
{ Language = "en-us",
SiteName = "Google",
Domain = "www.google.com",
SearchUrlTemplate = "https://www.google.com/search?hl=en&q={0}" } },
{ new SiteToSearchItem()
{ Language = "en-us",
SiteName = "Hulu",
Domain = "www.hulu.com",
SearchUrlTemplate = "https://www.hulu.com/#!search?q={0}" } },
{ new SiteToSearchItem()
{ Language = "en-us",
SiteName = "IMDB",
Domain = "www.imdb.com",
SearchUrlTemplate = "https://www.imdb.com/find?s=all&q={0}" } },
{ new SiteToSearchItem()
{ Language = "en-us",
SiteName = "Linked In",
Domain = "www.linkedin.com",
SearchUrlTemplate = "https://www.linkedin.com/search/fpsearch?keywords={0}" } },
{ new SiteToSearchItem()
{ Language = "en-us",
SiteName = "MSN",
Domain = "www.msn.com",
SearchUrlTemplate = "https://www.bing.com/search?scope=msn&q={0}" } },
{ new SiteToSearchItem()
{ Language = "en-us",
SiteName = "Twitter",
Domain = "www.twitter.com",
SearchUrlTemplate = "https://twitter.com/search/{0}" } },
{ new SiteToSearchItem()
{ Language = "en-us",
SiteName = "Weather.com",
Domain = "www.weather.com",
SearchUrlTemplate = "https://www.weather.com/info/sitesearch?q={0}" } },
{ new SiteToSearchItem()
{ Language = "en-us",
SiteName = "You Tube",
Domain = "www.youtube.com",
SearchUrlTemplate = "https://www.youtube.com/results?search_query={0}" } },
{ new SiteToSearchItem()
{ Language = "en-us",
SiteName = "Zillow",
Domain = "www.zillow.com",
SearchUrlTemplate = "https://www.zillow.com/homes/{0}/" } }
};
private string _siteToSearchAzureMobileServicesTableUrlTemplate = "https://robchblogsamples.azure-mobile.net/tables/SiteToSearchItem?$filter=Language%20eq%20'{0}'";
private int _downloadTimeout = 5000;
#endregion
}
public class SiteToSearchItem
{
public int Id { get; set; }
public string Domain { get; set; }
public string SiteName { get; set; }
public string SearchUrlTemplate { get; set; }
public string Language { get; set; }
}
}
Comments
- Anonymous
November 28, 2012
Wow, very perfect article.