Office 365 Search Monitoring at Office365Mon

Today we’re announcing a new feature at Office365Mon that our customers have been asking about for quite some time.  We’ve added the ability to monitor the Search service in your SharePoint Online tenant using our well-established health probe architecture.  This has been frequently requested because Search in SharePoint plays such a pivotal role in content delivery.  There are many out of the box web parts that depend on successfully executing queries to generate the content for the page.  On top of that there are many, many custom applications that are dependent upon the Search service working correctly.  From a developer perspective this has been an approach advocated by Microsoft for several years (including myself when I was in that role) because of the reduction in load it puts on the SharePoint farm as well the ability to pull data from a variety of sources.

When we were first designing this feature it primarily focused on the query aspect of the Search service.  However, after giving a sneak peek at what we were planning with several of my former colleagues at Microsoft, they felt just as passionate (if not more so) that we should also see what we could do about monitoring crawl performance.  Apparently quite a few of them have had customers frustrated about not seeing the content they expected when using Search and in many cases found out that their content had not actually been crawled yet.  There were some challenges in managing this request, but we found a solution that we’re happy with and causes a minimal amount of friction for our customers.  Best of all, like all our other features, I think you’ll find this extremely easy to set up and use.

What Does It Do?

Here’s a screenshot of the configuration page for the Office 365 Search Monitoring feature so you can see just how easy it is to configure what it does for you:

The Office 365 Search Monitoring feature allows you to provide a custom Keyword Query Language (KQL) query that we will execute against the site we’re monitoring for you with Office365Mon.  We have a link to the KQL guide when you go in and configure Search monitoring, and it can be something as simple asquerytext=’sharepoint’ .  As you’ll see if you look at the KQL reference though, the beauty of this is that you can actually get quite sophisticated in your KQL.  You can do things like control how many search results are returned, select a set of properties to return, use different ranking models, enable stemming, enable phonetics, etc.  One of the big reasons why we chose to use KQL directly is because so many of you have written us about custom applications you have built on Search and you want some means to monitor them.  By letting you use KQL, you can use any query that’s relevant to what your app does, and we’ll use that as the basis for monitoring.

Once you have your KQL, we take over from there.  We really monitor three things around the KQL query you’ve provided, based on the feedback and requests we’ve received the last several months:

1. Query latency – you define a maximum query time, and if it takes longer than that for us to get results back from running your KQL, we send out a notification to all of your configured notification options. That includes emails, text messages, and of course – now – webhooks.  If you have apps that are based on the Search service it can be critically important to know when queries are running slowly – based on a latency you decide is needed – so you know whether there are issues with your app, or issues with the Search service that your app is using.
2. Search results change – you can receive notifications when search results change. What this means is that if the set of results changes from the last time we ran your KQL, we’ll send out notifications.  If the only thing that changes is the raw rank value of the items in the search results, we will NOT send out notifications.  However, if all of the search results are the same, but the order of them changes, we DO send out notifications.  It’s also important to underscore that this reflects only on the search results themselves – it has nothing directly to do with the underlying items in the search results.  What that means is that if you change a document that’s included in the search results, we don’t detect and notify for that.  However, if that change caused a change in the search results – either the document no longer shows up, it shows up higher or lower in the search results, etc. – then we would send out notifications.

There are a couple of other things worth noting here as well.  First, we don’t store your actual search results.  This is consistent with how we do things here at Office365Mon – we don’t store usernames and passwords, and we don’t store your data.  What we DO store is a one-way hash of the search results.  When we do another search, we create the hash on the latest results and compare it to the hash we had before, and if they’re different we know the search results have changed.  It’s a one-way hash which means that it cannot be “reverse engineered” or otherwise tampered with to rehydrate the actual search results.

The second thing worth noting here is that in our experience, you may see the search results change quite frequently, even if there have not been any changes to the site content.  This is not a bug or issue with the Office365Mon Search Monitor feature.   We find that there may be hours and sometimes days when the search results come back exactly the same.  Then there are times when you may get different search results on five or six queries in a row.  Again – using the same KQL and the site content has not changed – but the search results ordering changes.  This is something you should be aware of when developing your applications based on the Search service.  If you need to know when it happens, now you can use our Search monitor feature to be alerted to it.  To help you parse through the changes, you can also elect to check the box to Include results in notifications when results change.  When that option is selected, you’ll get the actual search results we received in both emails and webhooks – just not text messages.  Again – these results are not saved anywhere, we are simply forwarding to you what we received.

3. No search results – the other option you have is to get notified when no search results are returned from your KQL. Again, whether you’re using out of the box web parts or a custom application, if you have content in your site that’s dependent upon getting search results, this can be a critical notification to have.

Finally, as described above, the other thing we monitor is content indexing.  With a simple checkbox, you can have us monitor your search index to see how long it is taking to crawl new content.

With all of the features described above, there are several reports that you can also use to stay on top of how things are doing.  More details regarding the reports are described below.  Also, if you want detailed step-by-step instructions for configuring Search monitoring you can get them from our site athttps://www.office365mon.com/configuresearchmon.pdf.  One final point worth making – you can also do all of the configuration programmatically using our Subscription Management API.  See our latest API documentation for details and code samples.

How Does It Do It

The Office 365 Search Monitor uses the same super-scale health probe infrastructure that Office365Mon has been using since Day One.  That enables us to issue and track query responses to your tenant.  To support crawl monitoring though, we had to come up with something a little different, and here’s why.  The monitoring applications we use are all defined in Azure Active Directory, and as part of that definition we describe what rights our applications need.  We always use the least invasive permissions possible to get the job done, so all of our apps are configured with the smallest amount of Read Only rights that we can get away with.  To do crawl monitoring though, we needed a way to determine how long it takes to get new content indexed – so how do we do that?  Well, we need to write a small amount of data to the site we’re monitoring, and then start issuing queries for that content.  We look at when we wrote the content into the site, and how long it takes until it starts showing up in search results, and that’s how we calculate the time it takes to index the content.

As described above though, all of our applications are configure to only have Read rights, so how can we write content to a site?  That’s where we had to add a new item to our toolbox to make it happen, and what we decided to do is to write a SharePoint App.  Yes, the same SharePoint Apps many of you develop to bring your site to life, we built one as well.  We wanted to limit our scope as much as possible, so the app only has rights in the current site it’s installed in – not any other site (i.e. SPWeb) in the site collection, nor any other site collection in the tenant.  When you first configure the Office 365 Search Monitoring feature, if you elect to Monitor search index freshness, the first check we do is to ensure the application is installed in the site being monitored.  If it’s not, we let you know so you can go install the app and try saving your Search monitoring configuration changes again.

The SharePoint App is in (or soon will be in) the Office Store under the name Office365Mon Search Monitor.  You can install it from there into the site being monitored.  In addition to that, since some folks turn off access to the Office Store, when you go to our Configure Office 365 Search Monitor page, we have a link you can use to download a zip file containing the SharePoint App.  If you go that route you’ll need extract the .app file out the zip file, upload it to the App Catalog in your tenant, and then you can install it in the monitored site.

When the app is installed and you save your search monitoring configuration, we’ll look for a custom list we use to store the data used for monitoring the search index freshness.  If it doesn’t exist, we’ll create it.  After that we use the app to create new content in the list so we can monitor how quickly it’s being indexed.

Reporting on Search Monitoring

We’ve built a half-dozen new reports based on the Office 365 Search Monitoring feature.  We have a very full list of Advanced Reports now with your Office365Mon subscription, as you can see in the Report Gallery here:

You can get recent data (i.e. from the last few hours) on query and crawl latency.  Here’s an example of the recent crawl latency report:

You can get daily stats on query and crawl latency – here’s an example of the daily query latency report:

Don’t be fooled by the numbers either – they show exactly why you want to monitor your query latency.  You can look at the graph and see overall the queries are returning data in sub second time.  However, it’s not uncommon to see this, yet still get notifications about queries that have taken 15 or 20 seconds or longer.  That’s what you want to know – your queries normally perform one way, but when they are significantly different it may be impacting the content and performance in your site.

We also have reports that show your monthly query and crawl latency averages, and they are overlaid on top of our service-wide averages so you can see how your performance is compared to your peers that are being monitored by Office365Mon.  In addition to that, in the Basic Reports you can see data just on the average crawl and query latency across our entire service:

In addition to our out of the box reports, you can also use our Report Data API to programmatically retrieve this data via our REST endpoints.  For those of you using our Power BI integration, you will also automatically see monthly crawl and query freshness data show up in Power BI after you refresh your data in there.

How Do I Get It

All existing Office365Mon customers always get all new features free to try for 90 days; everyone has had the feature turned on.  In addition, all new customers also always get ALL our features free to try for 90 days.  The feature is currently in beta, but rolled out now so everyone can begin using it.

That’s it – a quick run-down on a feature that we believe many of you will find extremely useful.  It has great value whether you are using out of the box web parts, or you’ve developed your own custom applications built on Office 365.  As always, if you have suggestions or ideas on how to improve this or any other feature at Office365Mon, please just drop me a note at support@office365mon.com.

From Sunny Phoenix,

Steve