Troubleshooting MDT 2012 Monitoring

I mentioned a while back that I wanted to do a blog post talking about how to troubleshoot the new MDT 2012 monitoring feature for Lite Touch deployments, but first I had to actually describe it.  If you haven’t reviewed that post, you might want to check it out first at https://blogs.technet.com/b/mniehaus/archive/2012/03/09/mdt-2012-new-feature-monitoring.aspx.

So now let’s talk about troubleshooting.  First, let’s look at the server side.  You have to enable monitoring on a computer that has MDT 2012 installed.  When you use Deployment Workbench on that computer and check the box to enable monitoring, Workbench will first check to see if the specified monitoring host name is local:

It doesn’t really matter if you specify an IP address, a short host name, or a fully-qualified host name, as long as the clients can resolve whatever you specify.  If you specify a name that Workbench doesn’t think is local (because Workbench itself can’t resolve the name back to an IP address assigned to the current machine), it won’t try to install the monitoring component; instead, it will try to contact that server to see if monitoring is running on that computer.  If it is, great; if it isn’t, you’ll see an error message:

If you look closely at the error “tip” at the end of the “Monitoring host” line, you’ll see a message like “Unable to connect to the specified server and port”.  If you think you specified the local computer name and got that error, then Workbench couldn’t figure out that it was the local computer name (something that’s harder to do than you might think).  If you are specifying a different server and see this error, then it’s having problems communicating with that other server.

Tip #1:   Make sure the name you specify in Workbench can be resolved to the IP address of the current machine.

What does the checkbox do?

You’ve checked the checkbox and can’t see that anything happened.  So what was actually done?  Two things:

1. A new “Microsoft Deployment Toolkit Monitor Service” service was installed on the computer and started.

2. An additional entry was added to the [Default] section of CustomSettings.ini telling the clients how to contact the server, with an entry such as:

   EventService=https://mdt-server.mdt.local:9800

Tip #2:   Make sure the “Microsoft Deployment Toolkit Monitor Service” is installed and running.  If it’s not installed and it should be, you can uncheck the box, click apply, then check the box again and click apply to reinstall it.  If it’s installed but not running, try to start it.

Tip #3:   Make sure the entry was added to CustomSettings.ini by looking at the Rules tab.  Because of a peculiarity with the way Workbench works, if you make any changes to the Rules tab after you’ve clicked the “Enable monitoring” checkbox but before you’ve clicked OK, it’s possible that the changes made on the Rules tab overlay the EventService entry in CustomSettings.ini, but it’s easy enough to put it back manually.

What if the service doesn’t start?

The service has two dependencies:

1. .NET 3.5 SP1 needs to be installed.  That shouldn’t be an issue, because you can’t install MDT 2012 without .NET 3.5 SP1.
2. The ports you specified need to be available for use.  (Generally that’s not an issue either, as 9800 and 9801 aren’t commonly-used TCP ports.  But it is possible to have another application use them.  Fortunately, MDT will happily use other ports.)

So there’s no dependency on IIS or SQL Server.  The service uses .NET to host a web server as part of the service process, and it uses a SQL Compact database (basically a set of DLLs, which ship with MDT, that run in the service process) to store the monitoring information.  It’s designed to be easy to install and run.

Tip #4:   If you try to start the service and it won’t start, that most likely means the ports you chose were already in use.  (If you want to know what’s using the ports, use a tool like TCPView, available from https://technet.microsoft.com/en-us/sysinternals/bb897437.)  Pick different ports.

While it’s always possible that there could be some other reason the service fails, I haven’t seen any other causes.  But if you know the ports are not in use and the service still won’t start, capture a trace using DebugView (https://technet.microsoft.com/en-us/sysinternals/bb896647) to see if it provides any further clues.  If not, contact Microsoft Support for assistance.

Verifying the Monitoring Service

The monitoring service listens on the two ports that you specified.  The first of these ports (9800) is used by computers being deployed to send progress events.  The second (9801) is used by Workbench itself to query information about deployments being monitored.  To make sure these ports are accessible, we can manually connect to each one using Internet Explorer.

To verify the “event port” from the monitor server itself, you can use a URL such as:

https://localhost:9800/MDTMonitorEvent/

If that works, you should see a response like:

That’s a proper response in this case – the web service doesn’t expect to be called in this way (an HTTP GET request instead of an HTTP POST request), so it’s telling you the proper way to call the service.

To verify the “data port” from the monitor server itself, you can use a URL such as:

https://localhost:9801/MDTMonitorData/

This response (which is an ODATA feed in case you are curious) confirms that the data feed is working as expected.

But those are the easy queries – they are using “localhost”, which is almost never subject to firewall restrictions.  Next, you need to try these queries remotely, using the appropriate “remote” URLs:

https://mdt-server:9800/MDTMonitorEvent/

https://mdt-server:9801/MDTMonitorData/

If those work, great.  If they don’t, then you need to make sure that whatever firewall is running on the monitoring server allows the ports you specified (e.g. 9800 and 9801) to be accessed from remote hosts.

Tip #5:   Make sure you can access the monitor service ports both locally and remotely.  Adjust the firewall rules as necessary.

Note that there are other networking “challenges” that can get in the way, e.g. IPSec domain isolation.  In this configuration, computers that aren’t domain-joined, e.g. running from Windows PE, can’t talk to domain-joined computers because they aren’t using encrypted IPsec communication.  This type of configuration will never work – you would need to set up the monitoring service on a “boundary server” that has been configured to allow non-IPsec traffic on the configured ports.  So don’t assume that if a “remote” URL works from a domain-joined machine, it will also work from a workgroup machine (or Windows PE) – know how your network is configured.

From the Client Side

When the EventService task sequence variable is set (via the processing of CustomSettings.ini), each MDT script executed in the task sequence will send an event to the monitor service on the “event port” URL.  When this succeeds, you’ll see a message like this:

If a script is unable to send an event, you’ll see something different:

That’s a clear sign that something isn’t right.  Make sure the service is running, that the firewall ports are open, etc. – the same challenges we already reviewed.

Tip #6:   Check the client logs to make sure the clients are able to talk to the monitoring service.

Another way you might notice an issue:  If the monitor service isn’t running, the clients will still try to connect to it, eventually timing out.  This timeout process will cause a delay at the end of each step in the task sequence, so if you are watching the task sequence progress dialog, you’ll see steps that you never noticed before (because they usually run so fast) now taking a long time.

From Workbench

When you try to look at the monitoring data from Workbench, it calls a PowerShell cmdlet (Get-MDTMonitorData), then that PowerShell cmdlet makes the “data port” query to retrieve the details from the monitoring service.  If the service is working as expected, you can see the list of monitored machines in Workbench.  If the service isn’t working, you’ll see something like this instead:

Good advice, make sure the service is running

Finally

Still having issues?  Post them as comments here, or send me an e-mail at mniehaus@microsoft.com and we’ll try to figure out what’s going on.

Comments

• Anonymous
  January 01, 2003
  It sounds like your servers may have an incorrect system-level proxy configuration or default route - an internet connection isn't required. In fact, my test environment typically is completely isolated and monitoring works fine.

• Anonymous
  January 01, 2003
  Michael, One quick question.  I have several deployment shares (Production, TestLab, Captures), on one server.  Do I need to change the ports so that it is monitored in the proper deployment share?

• Anonymous
  January 01, 2003
  If you want to keep the monitoring separate, you would need to set up three different machines running monitoring.  Otherwise, you can point them all to the same server and see them all together. There's no way to run multiple monitoring services (on different ports) on the same machine.

• Anonymous
  January 01, 2003
  When I first setup my MDT share and integrated DaRT, I believe the systems would automatically show up in the Monitoring window of the MDT Workbench once they fully booted the LiteTouch PE. (I have SkipWelcome set.) I was able to use the Monitoring feature to remotely connect to DaRT's remote control app by just clicking on the button within the monitoring properties of a machine. Then I could enter my credentials for the DeploymentShare and start my LiteTouch deployment remotely. Now when LiteTouch fully boots, the systems do not show in the Monitoring window until after I enter my DeploymentShare credentials. (Which basically means I cannot start a deployment fully remotely.) Is there a problem and if so how do I fix it, or am I just remembering incorrectly? Thanks for any help.

• Anonymous
  July 04, 2012
  I'm having trouble enabling monitoring. How is it done thanks

• Anonymous
  November 26, 2012
  mdt 2012 can not be intelligent judgments dual hard drives Scene: My pc has two hard drives, a data disk, and a system disk. Failure: mdt can not distinguish between the data disk and system disk, the system will often installed to the data disk. We look forward to a solution. email：minianson@hotmail.com

• Anonymous
  April 15, 2013
  With a proxy enabled in IE the service will fail to start

• Anonymous
  February 06, 2014
  Have you ever come across all of the tests passing, and the client logs indicating they are posting events to the server but the events do not show up in the Application Log? MDT Monitoring picks the deployment and we can watch it progress but events are not posted to the App Log.

• Anonymous
  June 12, 2014
  Tip #7 : make sure you have an Internet connection on you MDT server otherwise you always get : "unable to access the monitoring data service at hhtp://xyz:9801/MDTMonitorData." Is there a workaround because our servers normaly don't have an internet connection.

• Anonymous
  June 16, 2014
  It seems that Monitoring-Remote Desktop can't recognize the computer name if the target device did not join the domain. I need to use the traditional method - Use mstsc, type the target device IP address, in order to make a remote desktop connection.

• Anonymous
  June 23, 2014
  The comment has been removed

• Anonymous
  September 23, 2014
  believe it or not, none of these related to issue was causing the service not to start.
  I accidentally noticed windows update service was not running properly, and when I stopped and started it I then notice Deployment server started on it's own.
  

• Anonymous
  September 25, 2014
  this is what fixed the issue for me:
  Navigate to the HKEY_LOCAL_MACHINESYSTEMCurrentControlSetControl
  1- In the Right Pane create the following Key (If not already created) 1.Value: DWORD (32-bit)
  2- Name: ServicesPipeTimeout
  3 -Value Data, Click Decimal and type 60000 (Not less than 60000)

• Anonymous
  August 06, 2015
  I'm having an issue when I run a deployment, nothing is showing under the 'Monitoring' tab in the Workbench.
  I can view http://myserver:9800/mdtmonitorevent/ fine from the server and the client that I just deployed.
  Any suggestions?

• Anonymous
  March 11, 2016
  Michael,We recently upgraded our MDT server to Update 2. After the update, monitoring is not longer working. The service is running and the settings in cs.ini are correct. The bdd.log on the client shows an error from the web service - "Unexpected response from web service: 400 Bad Request"The monitoring service was working perfectly before upgrading to Update 2. Have you seen this issue?I also posted this on technet. I got a few responses from people also having this issue with no solution. https://social.technet.microsoft.com/Forums/en-US/b960ffa7-c9c0-465b-b67d-1787ae2f0270/mdt-2013-update-2-monitoring-errors?forum=mdtAny help would be great. Thanks.-Matt

  • Anonymous
    July 20, 2016
    A reply from MS would be appreciated!

• Anonymous
  March 14, 2016
  Hi, I got an issue with MDT 2012 MDT U1 and SCCM 2007 R3,I've created dummy deployment share just for monitoring, I enabled monitoring with default ports,ports are free, I checked with TCP View, there is no additional entry in the 'Rules' tab, and service is not starting up. I tried different ports, NETBIOS name, FQDN, nothing works. Any ideas?

• Anonymous
  May 26, 2016
  The comment has been removed

  • Anonymous
    June 01, 2016
    It looks like my solution was simply rebooting the remote Distribution Point for the changes to take affect. Monitoring was configured on the Primary Site server but WDS was running on a remote server and I guess the changes didn't make a difference as WDS was not restarted on the remote server.

• Anonymous
  June 07, 2016
  I'm having the same issue as these guys - https://social.technet.microsoft.com/Forums/en-US/b960ffa7-c9c0-465b-b67d-1787ae2f0270/mdt-2013-update-2-monitoring-errors?forum=mdtAnybody has any idea what's going on?

• Anonymous
  August 08, 2016
  Mike,I've had a problem with our logs since I upgraded our deployment share to MDT2013. Every now and then I get an unexpected response from the web service (400 bad request) and I can't seem to get rid of it. I've even tried a clean install of MDT/WDS on a new server (and moved the existing deployment shares to it) but still can't seem to get them to go away. They are always the same error and show up seemingly randomly throughout deployments and captures. I think I need to rebuild my deployment shares from scratch, but I really don't want to because we've spent a lot of time getting a lot of little tweaks in place over the years. If you have any ideas, please let me know. Here is the exact error:Unexpected response from web service: 400 Bad Request Request Error BODY { color: #000000; background-color: white; font-family: Verdana; margin-left: 0px; margin-top: 0px; } #content { margin-left: 30px; font-size: .70em; padding-bottom: 2em; } A:link { color: #336699; font-weight: bold; text-decoration: underline; } A:visited { color: #6699cc; font-weight: bold; text-decoration: underline; } A:active { color: #336699; font-weight: bold; text-decoration: underline; } .heading1 { background-color: #003366; border-bottom: #336699 6px solid; color: #ffffff; font-family: Tahoma; font-size: 26px; font-weight: normal;margin: 0em 0em 10px -20px; padding-bottom: 8px; padding-left: 30px;padding-top: 16px;} pre { font-size:small; background-color: #e5e5cc; padding: 5px; font-family: Courier New; margin-top: 0px; border: 1px #f0f0e0 solid; white-space: pre-wrap; white-space: -pre-wrap; word-wrap: break-word; } table { border-collapse: collapse; border-spacing: 0px; font-family: Verdana;} table th { border-right: 2px white solid; border-bottom: 2px white solid; font-weight: bold; background-color: #cecf9c;} table td { border-right: 2px white solid; border-bottom: 2px white solid; background-color: #e5e5cc;} Request Error The server encountered an error processing the request. The exception message is 'The type initializer for 'ExtentPlaceholderCreator' threw an exception.'. See server logs for more details. The exception stack trace is: at System.Data.Mapping.Update.Internal.Propagator.ExtentPlaceholderCreator.GetPropagatorResultForPrimitiveType(PrimitiveType primitiveType, PropagatorResult& result) at System.Data.Mapping.Update.Internal.Propagator.ExtentPlaceholderCreator.Visit(EdmMember node) at System.Data.Mapping.Update.Internal.Propagator.ExtentPlaceholderCreator.CreateEntitySetPlaceholder(EntitySet entitySet) at System.Data.Mapping.Update.Internal.Propagator.Visit(DbScanExpression node) at System.Data.Mapping.Update.Internal.Propagator.Visit(DbProjectExpression node) at System.Data.Mapping.Update.Internal.UpdateTranslator.<ProduceDynamicCommands>d__0.MoveNext() at System.Linq.Enumerable.<ConcatIterator>d__711.MoveNext() at System.Data.Mapping.Update.Internal.UpdateCommandOrderer..ctor(IEnumerable1 commands, UpdateTranslator translator) at System.Data.Mapping.Update.Internal.UpdateTranslator.Update(IEntityStateManager stateManager, IEntityAdapter adapter) at System.Data.Objects.ObjectContext.SaveChanges(SaveOptions options) at Microsoft.BDD.MonitorService.MonitorEventService.NextID(MDT_MonitorEntities monitorEntities) at Microsoft.BDD.MonitorService.MonitorEventService.PostEvent(String uniqueId, String computerName, String messageId, String severity, String stepName, Int16 currentStep, Int16 totalSteps, String id, String message, String dartIP, String dartPort, String dartTicket, String vmHost, String vmName) at SyncInvokePostEvent(Object , Object[] , Object[] ) at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs) at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc) at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)