Share via


Notifications overview for VSTS

Core Objects

There are three fundamental objects in the notification system:

Object Storage
Event stored in tbl_ClientEvent
Subscription stored in tbl_EventSubscription
Notification stored in tbl_EventNotification

Workflow

The actual flow can be expressed simply as well:

  • Some part of the product fires an event (WIT, Git, Gallery, etc.)
  • The notification system writes a record to tbl_ClientEvent with the event payload and starts the appropriate event processing job for the event type.
  • The event processing job reads the pending events from tbl_ClientEvent and loads subscriptions for the matching event types from tbl_EventSubscription as well as builtin and contributed subscriptions.
  • Each event is matched against each subscription. If there is a match, a notification is created. After each batch of events (up to 1000 at a time by default), the notifications are flushed to tbl_EventNotification and the appropriate notification delivery job(s) are started.
  • The delivery job reads pending notifications in batches (up to 100 by default) and delivers them.
    • Email messages are delivered in parallel (up to 8 at a time by default)
    • Service Hooks notifications are batched up and sent to the Service Hooks service for actual delivery unless...
    • ...the subscription is for Azure Service Bus for specific hosts/subscriptions - these are delivered directly from the host service without calling to the Service Hooks service at all

Additional Tables

Table Name Usage
tbl_DefaultSubscriptionAdminConfig Admin control over group subscriptions
tbl_DefaultSubscriptionUserOptOut User opt out of group subscriptions
tbl_NotificationLog Notification Diagnostic Logs
tbl_NotificationStatistics Continuously aggregated statistics

Notification Diagnostic Logs

Beginning with M131 and TFS 2018 Update 2

Extended diagnostic logs are now stored in tbl_NotificationLog. These logs can get large so they are cleaned up every few days. However, they are easily accessible via a REST navigate in a web browser. Logs are identified by LogSource and LogId. LogSource is typically a well known ID and LogId is a particular instance of that LogSource. If you happen to know the LogSource and LogId, you can refer to the log directly. Otherwise, you can specify a date range. The format of the REST API URL is:

<your host>/_apis/notification/DiagnosticLogs/LogSource/entries/LogId

or

<your host>/_apis/notification/DiagnosticLogs/LogSource/entries?startTime=time1&endTime=time2

Job Logging

Looking at the result message in Job History can be a big help when troubleshooting what's happening with event processing and notification delivery. However, it is limited to 8k and really isn't structured in any way - it is essentially a sequence of trace messages. Extended diagnostics are recorded for every single job run. The JobId is used as the LogSource and each run of the job gets a unique LogId - it's just a GUID with no further meaning. The result message in JobHistory contains the URL for that particular run.

To get the diagnostic log for the Work Item Processing job (915F48F2-1B64-40D9-A43F-FE2528B4F296) with a LogID of 68DE62E0-AA65-47BB-9369-FCD534B9A69A:

https://mseng.visualstudio.com/_apis/notification/DiagnosticLogs/915F48F2-1B64-40D9-A43F-FE2528B4F296/entries/68DE62E0-AA65-47BB-9369-FCD534B9A69A

If you don't know the LogId, you can ask for entries using a date range like this:

https://mseng.visualstudio.com/_apis/notification/DiagnosticLogs/915F48F2-1B64-40D9-A43F-FE2528B4F296/entries?startTime=2018-03-01T01:05:21.2253819Z&endTime=2018-03-01T12:05:21.2253819Z

This will return an array of logs. There can be a lot of logs and they can be on the larger size of things so the smaller you can make the date range the better.

Subscription Diagnostics Logging

We can enable even more diagnostic logging for a particular subscription. It can be enabled for even processing and/or notification delivery. The format for the LogSource is the subscription ID encoded into a well known GUID.

For event processing the LogSource is 08675309-EEEE-0000-0000-nnnnnnnnnnnn and for notification delivery the format is 08675309-DDDD-0000-0000-nnnnnnnnnnnn where nnnnnnnnnnnn is a leading zero padded subscription Id in decimal. The Job History result message and the diagnostic log will have a list of URLs for all of the subscription diagnostic logs that it generated. Once enabled, subscriptions diagnostics for 25 events for up to an hour and for 25 deliveries for an hour.

You can enable access to subscription diagnostics in the browser by appending diagnostics=true to the notifications page you are viewing. For example, on your personal subscriptions:

<your host>/_notifications?diagnostics=true

This will add an option to the "..." menu to allow you to enable/disable diagnostics for a particular subscription.

These settings are exposed via a REST API as well.

 

To enable processing and delivery diagnostics for a subscription, make an HTTP PUT call with the following body:

 {
  "evaluationTracing": { "enabled": true },
  "deliveryTracing": { "enabled": true }
}

The URL format is:< yourhost>/_apis/notification/subscriptions/SubscriptionId/diagnostics?api-version=4.1-preview.1

If you issue an HTTP GET, you will receive the current status. The results will look like this:

 {
  "evaluationTracing": {
    "enabled": true,
    "endDate": "2018-03-01T20:03:59.929866Z",
    "startDate": "2018-03-01T19:03:59.929866Z",
    "maxTracedEntries": 25
  },
  "deliveryTracing": {
    "enabled": true,
    "endDate": "2018-03-01T20:03:59.929866Z",
    "startDate": "2018-03-01T19:03:59.929866Z",
    "maxTracedEntries": 25
  }
}

Notifications Jobs

Job IDs (as of TFS 2018)

Job Id Job Type Scope(s) Schedule
915f48f2-1b64-40d9-a43f-fe2528b4f296 Event Processing WIT on demand
9a688110-9e33-4cdc-affd-75d16303e7f1 Event Processing Git on demand
a4804dcf-4bb6-4109-b61c-e59c2e8a9ff7 Event Processing Any other publisher on demand
631f49b3-46e1-42ec-8fff-081bd176c18a Email Delivery WIT on demand
8833fc71-42ca-441b-ab12-25314877772d Email Delivery Git on demand
68063fb6-64c6-4b56-924f-501406d6e7e7 Email Delivery System generated on demand
a96d6177-beef-477a-a2ee-2c31433214d0 Email Delivery Any other publisher on demand
b7494ac7-dee6-4342-ad43-5d6944576645 Service Hooks Delivery Non-Service Bus WIT on demand
d745068c-7dab-4135-8665-6a7e1c20f29c Service Hooks Delivery Non-Service Bus Git on demand
a0c22f34-652e-4b9c-a06b-3f4afe4be458 Service Hooks Delivery Non-Service Bus Any other publisher on demand
ce76d8b3-3e04-4758-8430-4e9266d38938 Service Hooks Delivery Service Bus (limited use) WIT on demand
1214f948-151c-4f98-9ae2-1c9816db1eea Service Hooks Delivery Service Bus (limited use) Git on demand
c1c9013c-c4a9-4bb1-acca-7cd6b5ba5126 Service Hooks Delivery Service Bus (limited use) Any other publisher on demand
d6be0cce-5320-45de-9297-596a9be68a37 Group Service Hooks Delivery Any publisher on demand
007b33d4-c189-4576-96dd-36178e2f6803 SOAP Delivery All publishers on demand
0d30ca11-162c-4134-9ba8-42582f3890A5 Notification and Event Cleanup Job Any events, notifications, statistics, diagnostic logs daily
5ec103d2-7505-4705-b78e-2faf19127a12 Subscription Cleanup Job Subscriptions marked for deletion daily
eab3ce0c-a4f4-4460-82a0-6cf6e8bf0a8b Backlog status Notifications and Events on demand

SQL JobHistory Filters

Processing Job

JobId in ('915f48f2-1b64-40d9-a43f-fe2528b4f296', '9a688110-9e33-4cdc-affd-75d16303e7f1', 'a4804dcf-4bb6-4109-b61c-e59c2e8a9ff7')

Email Delivery Jobs

JobId in ('631f49b3-46e1-42ec-8fff-081bd176c18a', '8833fc71-42ca-441b-ab12-25314877772d', 'a96d6177-beef-477a-a2ee-2c31433214d0')

Service Hooks Delivery Jobs

Non Service Bus

JobId in ('b7494ac7-dee6-4342-ad43-5d6944576645', 'd745068c-7dab-4135-8665-6a7e1c20f29c', 'a0c22f34-652e-4b9c-a06b-3f4afe4be458')

Service Bus

JobId in ('ce76d8b3-3e04-4758-8430-4e9266d38938', '1214f948-151c-4f98-9ae2-1c9816db1eea', 'c1c9013c-c4a9-4bb1-acca-7cd6b5ba5126')

Groups

JobId in ('d6be0cce-5320-45de-9297-596a9be68a37')

Notifications Legacy Delivery

JobId in ('007b33d4-c189-4576-96dd-36178e2f6803')

Event Type groupings

WIT Email and Service hooks

JobId in ('915f48f2-1b64-40d9-a43f-fe2528b4f296', '631f49b3-46e1-42ec-8fff-081bd176c18a', 'b7494ac7-dee6-4342-ad43-5d6944576645', 'ce76d8b3-3e04-4758-8430-4e9266d38938')

Git Email and Service hooks

JobId in ('9a688110-9e33-4cdc-affd-75d16303e7f1', '8833fc71-42ca-441b-ab12-25314877772d', 'd745068c-7dab-4135-8665-6a7e1c20f29c', '1214f948-151c-4f98-9ae2-1c9816db1eea')

Any other events Email and Service hooks

JobId in ('a4804dcf-4bb6-4109-b61c-e59c2e8a9ff7', 'a96d6177-beef-477a-a2ee-2c31433214d0', 'a0c22f34-652e-4b9c-a06b-3f4afe4be458', 'c1c9013c-c4a9-4bb1-acca-7cd6b5ba5126')