Notifications overview for VSTS
Core Objects
There are three fundamental objects in the notification system:
| Object | Storage |
|---|---|
| Event | stored in tbl_ClientEvent |
| Subscription | stored in tbl_EventSubscription |
| Notification | stored in tbl_EventNotification |
Workflow
The actual flow can be expressed simply as well:
- Some part of the product fires an event (WIT, Git, Gallery, etc.)
- The notification system writes a record to tbl_ClientEvent with the event payload and starts the appropriate event processing job for the event type.
- The event processing job reads the pending events from tbl_ClientEvent and loads subscriptions for the matching event types from tbl_EventSubscription as well as builtin and contributed subscriptions.
- Each event is matched against each subscription. If there is a match, a notification is created. After each batch of events (up to 1000 at a time by default), the notifications are flushed to tbl_EventNotification and the appropriate notification delivery job(s) are started.
- The delivery job reads pending notifications in batches (up to 100 by default) and delivers them.
- Email messages are delivered in parallel (up to 8 at a time by default)
- Service Hooks notifications are batched up and sent to the Service Hooks service for actual delivery unless...
- ...the subscription is for Azure Service Bus for specific hosts/subscriptions - these are delivered directly from the host service without calling to the Service Hooks service at all
Additional Tables
| Table Name | Usage |
|---|---|
| tbl_DefaultSubscriptionAdminConfig | Admin control over group subscriptions |
| tbl_DefaultSubscriptionUserOptOut | User opt out of group subscriptions |
| tbl_NotificationLog | Notification Diagnostic Logs |
| tbl_NotificationStatistics | Continuously aggregated statistics |
Notification Diagnostic Logs
Beginning with M131 and TFS 2018 Update 2
Extended diagnostic logs are now stored in tbl_NotificationLog. These logs can get large so they are cleaned up every few days. However, they are easily accessible via a REST navigate in a web browser. Logs are identified by LogSource and LogId. LogSource is typically a well known ID and LogId is a particular instance of that LogSource. If you happen to know the LogSource and LogId, you can refer to the log directly. Otherwise, you can specify a date range. The format of the REST API URL is:
<your host>/_apis/notification/DiagnosticLogs/LogSource/entries/LogId
or
<your host>/_apis/notification/DiagnosticLogs/LogSource/entries?startTime=time1&endTime=time2
Job Logging
Looking at the result message in Job History can be a big help when troubleshooting what's happening with event processing and notification delivery. However, it is limited to 8k and really isn't structured in any way - it is essentially a sequence of trace messages. Extended diagnostics are recorded for every single job run. The JobId is used as the LogSource and each run of the job gets a unique LogId - it's just a GUID with no further meaning. The result message in JobHistory contains the URL for that particular run.
To get the diagnostic log for the Work Item Processing job (915F48F2-1B64-40D9-A43F-FE2528B4F296) with a LogID of 68DE62E0-AA65-47BB-9369-FCD534B9A69A:
https://mseng.visualstudio.com/_apis/notification/DiagnosticLogs/915F48F2-1B64-40D9-A43F-FE2528B4F296/entries/68DE62E0-AA65-47BB-9369-FCD534B9A69A
If you don't know the LogId, you can ask for entries using a date range like this:
https://mseng.visualstudio.com/_apis/notification/DiagnosticLogs/915F48F2-1B64-40D9-A43F-FE2528B4F296/entries?startTime=2018-03-01T01:05:21.2253819Z&endTime=2018-03-01T12:05:21.2253819Z
This will return an array of logs. There can be a lot of logs and they can be on the larger size of things so the smaller you can make the date range the better.
Subscription Diagnostics Logging
We can enable even more diagnostic logging for a particular subscription. It can be enabled for even processing and/or notification delivery. The format for the LogSource is the subscription ID encoded into a well known GUID.
For event processing the LogSource is 08675309-EEEE-0000-0000-nnnnnnnnnnnn and for notification delivery the format is 08675309-DDDD-0000-0000-nnnnnnnnnnnn where nnnnnnnnnnnn is a leading zero padded subscription Id in decimal. The Job History result message and the diagnostic log will have a list of URLs for all of the subscription diagnostic logs that it generated. Once enabled, subscriptions diagnostics for 25 events for up to an hour and for 25 deliveries for an hour.
You can enable access to subscription diagnostics in the browser by appending diagnostics=true to the notifications page you are viewing. For example, on your personal subscriptions:
<your host>/_notifications?diagnostics=true
This will add an option to the "..." menu to allow you to enable/disable diagnostics for a particular subscription.
These settings are exposed via a REST API as well.
To enable processing and delivery diagnostics for a subscription, make an HTTP PUT call with the following body:
{
"evaluationTracing": { "enabled": true },
"deliveryTracing": { "enabled": true }
}
The URL format is:< yourhost>/_apis/notification/subscriptions/SubscriptionId/diagnostics?api-version=4.1-preview.1
If you issue an HTTP GET, you will receive the current status. The results will look like this:
{
"evaluationTracing": {
"enabled": true,
"endDate": "2018-03-01T20:03:59.929866Z",
"startDate": "2018-03-01T19:03:59.929866Z",
"maxTracedEntries": 25
},
"deliveryTracing": {
"enabled": true,
"endDate": "2018-03-01T20:03:59.929866Z",
"startDate": "2018-03-01T19:03:59.929866Z",
"maxTracedEntries": 25
}
}
Notifications Jobs
Job IDs (as of TFS 2018)
| Job Id | Job Type | Scope(s) | Schedule |
|---|---|---|---|
| 915f48f2-1b64-40d9-a43f-fe2528b4f296 | Event Processing | WIT | on demand |
| 9a688110-9e33-4cdc-affd-75d16303e7f1 | Event Processing | Git | on demand |
| a4804dcf-4bb6-4109-b61c-e59c2e8a9ff7 | Event Processing | Any other publisher | on demand |
| 631f49b3-46e1-42ec-8fff-081bd176c18a | Email Delivery | WIT | on demand |
| 8833fc71-42ca-441b-ab12-25314877772d | Email Delivery | Git | on demand |
| 68063fb6-64c6-4b56-924f-501406d6e7e7 | Email Delivery | System generated | on demand |
| a96d6177-beef-477a-a2ee-2c31433214d0 | Email Delivery | Any other publisher | on demand |
| b7494ac7-dee6-4342-ad43-5d6944576645 | Service Hooks Delivery Non-Service Bus | WIT | on demand |
| d745068c-7dab-4135-8665-6a7e1c20f29c | Service Hooks Delivery Non-Service Bus | Git | on demand |
| a0c22f34-652e-4b9c-a06b-3f4afe4be458 | Service Hooks Delivery Non-Service Bus | Any other publisher | on demand |
| ce76d8b3-3e04-4758-8430-4e9266d38938 | Service Hooks Delivery Service Bus (limited use) | WIT | on demand |
| 1214f948-151c-4f98-9ae2-1c9816db1eea | Service Hooks Delivery Service Bus (limited use) | Git | on demand |
| c1c9013c-c4a9-4bb1-acca-7cd6b5ba5126 | Service Hooks Delivery Service Bus (limited use) | Any other publisher | on demand |
| d6be0cce-5320-45de-9297-596a9be68a37 | Group Service Hooks Delivery | Any publisher | on demand |
| 007b33d4-c189-4576-96dd-36178e2f6803 | SOAP Delivery | All publishers | on demand |
| 0d30ca11-162c-4134-9ba8-42582f3890A5 | Notification and Event Cleanup Job | Any events, notifications, statistics, diagnostic logs | daily |
| 5ec103d2-7505-4705-b78e-2faf19127a12 | Subscription Cleanup Job | Subscriptions marked for deletion | daily |
| eab3ce0c-a4f4-4460-82a0-6cf6e8bf0a8b | Backlog status | Notifications and Events | on demand |
SQL JobHistory Filters
Processing Job
JobId in ('915f48f2-1b64-40d9-a43f-fe2528b4f296', '9a688110-9e33-4cdc-affd-75d16303e7f1', 'a4804dcf-4bb6-4109-b61c-e59c2e8a9ff7')
Email Delivery Jobs
JobId in ('631f49b3-46e1-42ec-8fff-081bd176c18a', '8833fc71-42ca-441b-ab12-25314877772d', 'a96d6177-beef-477a-a2ee-2c31433214d0')
Service Hooks Delivery Jobs
Non Service Bus
JobId in ('b7494ac7-dee6-4342-ad43-5d6944576645', 'd745068c-7dab-4135-8665-6a7e1c20f29c', 'a0c22f34-652e-4b9c-a06b-3f4afe4be458')
Service Bus
JobId in ('ce76d8b3-3e04-4758-8430-4e9266d38938', '1214f948-151c-4f98-9ae2-1c9816db1eea', 'c1c9013c-c4a9-4bb1-acca-7cd6b5ba5126')
Groups
JobId in ('d6be0cce-5320-45de-9297-596a9be68a37')
Notifications Legacy Delivery
JobId in ('007b33d4-c189-4576-96dd-36178e2f6803')
Event Type groupings
WIT Email and Service hooks
JobId in ('915f48f2-1b64-40d9-a43f-fe2528b4f296', '631f49b3-46e1-42ec-8fff-081bd176c18a', 'b7494ac7-dee6-4342-ad43-5d6944576645', 'ce76d8b3-3e04-4758-8430-4e9266d38938')
Git Email and Service hooks
JobId in ('9a688110-9e33-4cdc-affd-75d16303e7f1', '8833fc71-42ca-441b-ab12-25314877772d', 'd745068c-7dab-4135-8665-6a7e1c20f29c', '1214f948-151c-4f98-9ae2-1c9816db1eea')
Any other events Email and Service hooks
JobId in ('a4804dcf-4bb6-4109-b61c-e59c2e8a9ff7', 'a96d6177-beef-477a-a2ee-2c31433214d0', 'a0c22f34-652e-4b9c-a06b-3f4afe4be458', 'c1c9013c-c4a9-4bb1-acca-7cd6b5ba5126')