Windows Server AppFabric: Extensibility to support remote troubleshooting

A common problem ISV support teams (and multi location enterprises, e.g., Microsoft) have to grapple with is providing debugging assistance to IT Admin and Junior Developers at remote locations. Compounding the problem is restricted access to the machine due to logistical or security considerations. Packaging the meta data, instance data and log files relating to the erroneous instance is not a unique idea; however, this blog is an original attempt to provide a design for the Windows Server AppFabric.

While the blog provides designs to build and consume the package, it does not elaborate on the Workflow Visualizer which is widely covered in technology blogs and MSDN (https://msdn.microsoft.com/en-us/library/cc835242.aspx).

Scenario

Consider a situation where an ISV has created a re-hosted Workflow Designer and Visualizer that their customers use to create, deploy and also debug their solutions. When the customer encounters a problem that they are unable to debug/address; they turn to the ISV support team to help with troubleshooting and finding a solution to the issue. An obvious challenge is that the ISV support team is not on site or does not have direct access to the AppFabric machines to debug the problem. So it’s imperative that the Support Team should be able to reconstruct the erroneous scenario from the customer and log data.

The grahic below (executed within the Visualizer Application) demonstrates an exception being thrown. The customer offline tests and deploys a Workflow; however this Workflow throws an exception in a specific uses case and aborts. The customer is unable to debug and provide a fix. This finally escalates into a call to the ISV Support team.

Per instructions from the ISV Support team, the customer reopens the erroneous project; loads the specific instance of its execution and creates a snapshot using the appropriate interfaces in the Visualizer. This Snapshot is a package of the project, its supporting files (XAMLX and assemblies) and the event data AppFabric collected from the instance in question via the instrumentation elaborated in previous sections.

This ‘package’ will be loaded by the ISV Support team in the Visualizer to ‘reproduce’ the error on the ‘specific’ Workflow along with the related instance data.

Solution

Leveraging the Windows Server AppFabric ability to track Workflow and WCF event data; the ISV's Workflow Designer created a great support feature that allows the customer to quickly and reliably build a ‘package’ and share their meta and instance data with the ‘offline’ ISV Support team.

Instrumentation

In order to keep this ‘execution’ data separate from other event data that AppFabric monitors on the customer's machine, a separate monitoring database is installed that is only for the use of the ISV's Workflow Designer. Setting up this database is quite simple as elaborated in the PowerShell Scripts below.

    1: import-module ApplicationServer
    2: Initialize-ASMonitoringSqlDatabase -Database WorkflowDesignerMonitoringDB -Server .\SqlExpress

 

 

Design and Implementation

Let's take a closer look at the design to build this solution.

Querying for the instance

The customer can pull up a list of the Workflow instances and select the erroneous one with a menu we populate using the AppFabric tables ASWfInstances and ASEventSources.

ASWfInstancesTable

Field	Description	Example data
Id	-	-
WorkflowInstanceId	GUID uniquely identifyies each instance	0f43ee0a-d661-4d14-8657-67003
LatestRecordNumber	-	-
LastEventSourceId	Last file to emit an event in this instance	-
LastEventStatus	Last state of this instance	Aborted/Completed/Idle/Suspended
StartTime	-	-
LastModifiedTime	The last time this instance did anything	-
CurrentDuration	Total execution time (in milliseconds)	3
ExceptionCount	# of exceptions that occurred	1
LastAbortedTime	Time the instance failed	-

Table ASWfInstances above holds information about each instance that has been executed. The fields ExceptionCount, LastAbortedTime and LastEventStatus should all help the user in finding the erroneous instance. However, this table does not contain information about the application or .XAMLX files involved. For this we use LastEventSourceId to combine this data with the information contained in table ASEventSources. With the extra detail that ASEventSources provides it should be possible for the user to identify the instance being sought.

ASEventSourcesTable

Field	Description	Example Data
Id	-	-
Name	Name of the file	Service
Computer	-	Example-PC
Site	IIS Server website	Default Web Site
VirtualPath	IIS application and file	/ExampleApp/Service.xamlx
ApplicationVirtualPath	Just the IIS application	/ExampleApp
ServiceVirtualPath	Just the file	/Service.xamlx

Here is a query used for combining the info from these two tables:

    1: SELECT        wfSvcs.VirtualPath, wfInst.LastModifiedTime, wfInst.LastEventStatus, wfInst.WorkflowInstanceId, wfSvcs.Site, wfSvcs.Name, wfInst.StartTime, wfInst.CurrentDuration, 
    2:                          wfInst.ExceptionCount, wfInst.LastAbortedTime
    3: FROM            ASWfInstances AS wfInst INNER JOIN
    4:                          ASEventSources AS wfSvcs ON wfSvcs.Id = wfInst.LastEventSourceId

Typical results demonstrating this query:

Creating the package

With the instance now selected, the ISV Application gathers the associated event data to be stored in its own monitoring database in preparation for transport. Using this instance’s WorkflowInstanceId the event data is gathered in the following manner:

The ASWfEventsTable is queried using the WorkflowInstanceId as shown below. This table gives us all of the events that occurred during the execution of this instance.

ASWfEventsTable

    1: SELECT        Id, EventTypeId, EventSourceId, ProcessId, TraceLevelId, WorkflowInstanceId, RecordNumber, TrackingProfileId, AnnotationSetId, TimeCreated, Data1Str, Data2Str, 
    2:                          Data3Str, Data4Str, Data5Str, Data6Str, Data7Str, Data8Str, Data1MaxStr, Data1Int
    3: FROM            ASWfEventsTable
    4: WHERE        (WorkflowInstanceId = @InstanceGuid)

Significant fields listed below.

Field	Description	Example Data
EventTypeId	Integer representing type of event (workflow events, wcf events, etc.)	103
EventSourceId	Maps to service this event came from	63
TraceLevelId	Indicates the level of this trace event (e.g., verbose, info, warn, error).	4
WorkflowInstanceId	All events in this instance have this same id.	0f43ee0a-d661-4d14-8657-67003
Data1Str	Context specific event descriptions	Faulted
Data2Str	Context specific event descriptions	Assign
Data5Str	Context specific event descriptions	System.Activities.Statements.Assign

From the result of this query we use the EventSourceId to get information we need from the ASEventSourcesTable as shown in the query that follows. As was stated earlier, this gives us descriptive information to use such as the website, application and source service. The ASEventSourcesTable’s data is described in more detail in the previous section, ‘Querying for the Instance’.

    1: int eventSourceId = asWfEventsTable.First().EventSourceId;
    2: 
    3: SELECT [Id]
    4:       ,[Name]
    5:       ,[Computer]
    6:       ,[Site]
    7:       ,[VirtualPath]
    8:       ,[ApplicationVirtualPath]
    9:       ,[ServiceVirtualPath]
   10:   FROM [AppFabricMonitoring].[dbo].[ASEventSourcesTable]
   11:   WHERE [Id] = @eventSourceId

ASWfEventPropertiesTable

This next table queried for information is the ASWfEventPropertiesTable. This table provides detailed information on the status of each event

    1: ASWfEventPropertiesTable
    2: SELECT                [EventId]
    3:            ,[WfDataSourceId]
    4:            ,[Name]
    5:            ,[Type]
    6:            ,[Value]
    7:            ,[ValueBlob]
    8:            ,[TimeCreated]
    9: FROM ASWfEventPropertiesTable
   10: WHERE [EventId] in (SELECT EventId FROM ASWfEventsTable WHERE [WorkflowInstanceId] = @InstanceGuid)

Significant fields listed below.

Field	Description	Example Data
EventId	The numeric id of the event type.	103
WfDataSourceId	The source of the event property which can be argument, variable or custom data.	0
Name	The name of the event property.	E2EActivityId
Type	The type of the property.	System.Guid
Value	The value of the property.	b82dbde2-d3b0-4ef5-be00-2713820a4128
ValueBlob	The value property when the type is a complex type.	NULL
TimeCreated	Timestamp when the property was emitted.	-

Finally, the ASWfPropertyNamesTable and ASWfTrackingProfilesTable tables are queried. ASWfTrackingProfilesTable is used to capture the tracking profile in use.

    1: ASWfPropertyNamesTable
    2: SELECT [EventSourceId]
    3:       ,[Name]
    4:       ,[Type]
    5: FROM [ASWfPropertyNamesTable]
    6: WHERE [EventSourceId] = @eventSourceId
    7:  
    8: ASWfTrackingProfilesTable
    9: SELECT        Id, Name
   10: FROM            ASWfTrackingProfilesTable

Once the event monitoring information is queried and stored in the ISV’s monitoring database, this data and the project’s files can be written to the package file that is to be sent to ISV Support. This data is visualized on the user’s workflow for visualization of the problem.

Selected instance is visualized on the workflow

With the desired instance visualized, the user chooses to create the package. In this application the package is referred to as a Snapshot. This file allows anyone to see exactly what this workflow was doing at a specific point in time, hence the term snapshot.

The user chooses where to save this Snapshot file, it is created and can then be sent to the ISV Support Team

Loading the package

The ISV Support team will load the package to begin the troubleshooting process. Loading the package requires opening the project and injecting the customer event data into the Monitoring Database used by the ISV Workflow Designer and Visualizer Application. This monitoring database is a separate copy of the monitoring database used specifically for storing the records of a single instance.

AppFabric serializes all of the data it stores; meaning that the customer’s event data in the package (which is serialized) needs to be un-serialized before it can be written to the database.

    1: BinaryFormatter bin = new BinaryFormatter();
    2: FileStream fs = new FileStream(selectedSnapshot, FileMode.Open);
    3: DataSet ds = bin.Deserialize(fs) as DataSet;
    4: fs.Close();

 With the data unpacked into a dataset, it is ready to be placed into the ISV Application’s monitoring database.

    1: AppFabricMonitoringDataSet.ASWfEventsTableDataTable asWfEventsTable =
    2: ds.Tables["ASWfEventsTable"] as AppFabricMonitoringDataSet.ASWfEventsTableDataTable;
    3: AppFabricMonitoringDataSet.ASEventSourcesTableDataTable asEventSourcesTable =
    4: ds.Tables["ASEventSourcesTable"]as AppFabricMonitoringDataSet.ASEventSourcesTableDataTable;
    5: AppFabricMonitoringDataSet.ASWfEventPropertiesTableDataTable asWfEventPropertiesTable =
    6: ds.Tables["ASWfEventPropertiesTable"] as AppFabricMonitoringDataSet.ASWfEventPropertiesTableDataTable;
    7: AppFabricMonitoringDataSet.ASWfPropertyNamesTableDataTable asWfPropertyNamesTable =
    8: ds.Tables["ASWfPropertyNamesTable"] as AppFabricMonitoringDataSet.ASWfPropertyNamesTableDataTable;
    9: AppFabricMonitoringDataSet.ASWfTrackingProfilesTableDataTable asWFTrackingProfilesTable =
   10: ds.Tables["ASWFTrackingProfilesTable"] as AppFabricMonitoringDataSet.ASWfTrackingProfilesTableDataTable;

Before loading the data, we clear the temporary monitoring database.

    1: string conString = System.Configuration.ConfigurationManager.ConnectionStrings["WorkflowDesignerMonitoringDB ConnectionString"].ConnectionString;
    2:                 System.Data.SqlClient.SqlConnection sqlCon = new System.Data.SqlClient.SqlConnection(conString);
    3:  
    4: try
    5: {
    6: sqlCon.Open();
    7: System.Data.SqlClient.SqlCommand cmd = new System.Data.SqlClient.SqlCommand();
    8: cmd.CommandText = "TRUNCATE TABLE [ASWfTrackingProfilesTable]; " +
    9: "TRUNCATE TABLE [ASWfEventPropertiesTable];" +
   10: "TRUNCATE TABLE [ASWfPropertyNamesTable];" +
   11: "TRUNCATE TABLE [ASWfInstancesTable];" +
   12: "TRUNCATE TABLE [ASWfEventsTable];" +
   13: "TRUNCATE TABLE [ASEventSourcesTable];";
   14: cmd.Connection = sqlCon;
   15: cmd.ExecuteNonQuery();
   16: }
   17: finally
   18: {
   19: sqlCon.Close();
   20: }

We then set up a bulk insertion.

    1: System.Data.SqlClient.SqlBulkCopy sbc = new System.Data.SqlClient.SqlBulkCopy(conString,
    2: System.Data.SqlClient.SqlBulkCopyOptions.KeepIdentity | System.Data.SqlClient.SqlBulkCopyOptions.FireTriggers);

Then we insert the records into the appropriate tables.

    1: BulkInsertTable(asWfEventsTable, sbc);
    2: BulkInsertTable(asEventSourcesTable, sbc);
    3: BulkInsertTable(asWfEventPropertiesTable, sbc);
    4: BulkInsertTable(asWfPropertyNamesTable, sbc);
    5: BulkInsertTable(asWFTrackingProfilesTable, sbc);

Once the project and event data is loaded, the ISV Support Team can instantly reproduce the error scenario and troubleshoot the customer’s erroneous Workflow. The Support team has an exact copy of the customer's project with the state reproduced.

Result

Typically, .NET4 Workflows are difficult to debug since the event information relating to these Workflows is ‘hidden’ amongst OS and Server related events. With the advent of Windows Server AppFabric, this event data is now easily accessible from a simple database. Even the ISV's rudimentary Workflow Designer and Visualizer Applications can be designed to package this data for remote debugging.

The error example used in this blog post relates to a variable with the wrong format. While in the real world there will be many more complex use cases, this solution provides a reasonable platform to troubleshoot Workflows, especially in an offsite scenario.

Once the error is ‘fixed’ the Workflow instance is flawlessly executed and can be verified on the Visualizer.

Summary

Windows Server AppFabric provides adequate extensibility for the ISV to build applications that can efficiently manage the lifecycle of .NET4 Workflow Services. This blog post effectively demonstrates the extensibility around remote troubleshooting.

Caveat

The Workflow instances could contain sensitive and confidential data, event info etc. The ISV Application should include designs to obfuscate such information.