Analyzing ddr backlogs

 

The purpose of this post is to assist in troubleshooting ddr backlogs. The main focus will be the analysis of the backlogged data, but we'll also discuss some basics to look at first.

About DDRs (Discovery Data Records) - DDRs are files that contain discovery data in Configuration Manager. They are created when the various discovery methods are run and are then processed on the site server by ddm (discovery data manager). For more information on discovery, see Planning for Discovery in Configuration Manager.

Most of this information applies to both ConfigMgr 2007 and 2012, but there are some differences.

1. 2007 Processes DDRs at every primary site server while discovery data moves up the hierarchy, whereas 2012 processes DDRs at the site that generates them and then replicates the data to other sites using database replication.
2. You'll notice user ddrs have their own subdirectory on 2012 <InstallDirectory>\inboxes\auth\ddm.box\userddrsonly. On 2007, all DDRs are processed from <InstallDirectory>\inboxes\auth\ddm.box.

When experiencing ddr backlogs, here are a few items to get out of the way first:

1. Review ddm.log which is located on the primary site server's <Installation Directory>\logs folder and ensure that the discovery data manager component is processing ddrs without errors.
2. Ensure that you have the proper real-time AV exclusions in place on your site server. Slow ddr processing due to real-time AV scan and/or poor disk performance is the most common cause of ddr backlogs
   • If using SCEP on 2012, you can import the antimalware policy templates that ship with the product from the <InstallDirectory>\AdminConsole\XmlStorage\EPTemplates. Note that if using a non-default template, you'll have to modify it. See my related post here.
   • On 2007 or 2012 with other AV products, ensure that the <InstallDirectory>\Inboxes is excluded from real-time antivirus scans. Scheduling a daily scan is fine. If you'd like more details, there is a detailed post located here.
   • You might be able to identify other processes slowing ddr processing using Process Monitor.
3. If you want to know your processing rate, you can see DDRs processed/minute by viewing the SMS Discovery Data Manager counters on the site server.
   • It is important to have a baseline of what is normal in your environment on your hardware as "normal" can be very different based on many variables
   • In situations where the processing rate is low and all other items are good, check event logs and/or hardware diagnostics for disk problems.
4. Discovery frequency - Consider your discovery frequencies. Often, when modified from default settings, you can easily end up discovering redundant data. With delta discovery enabled, there is rarely a need to increase your full discovery schedule.
   • More information on ConfigMgr 2007 Discovery configuration
   • More information on ConfigMgr 2012 Discovery configuration

Once you have ensured that data is processing at a reasonable rate, the next thing you'll want to look at is how much and what type of data is coming in. Here are some methods that I use to analyze this data.

If you open a ddr file using notepad, you'll find that much of the data is clearly stated.

 

I've attached an example of a ddr. See below.

 

When generating a lot of discovery data, the first question is typically about what discovery method is generating the bulk of the data. We can identify this using the AGENTINFO line in the ddr:

AGENTINFO<Heartbeat Discovery><RP0><08/13/2013 10:28:16>

 

We can see the above example is a heartbeat ddr

You can use findstr to get a list of all discovery types that you currently have queued up and put the results in a txt file.

 

Bring up a cmd prompt and cd to the appropriate directory:

<Installation Directory>\inboxes\auth\ddm.box>

 

Then use the following syntax:

findstr /I "AGENTINFO" *.ddr >D:\Temp\DDR_AgentInfo.txt

 

The output looks like this:

6R0BXZ2O.DDR:AGENTINFO<Heartbeat Discovery><RP0><08/09/2013 16:08:46>

7HL0W9MY.DDR:AGENTINFO<Heartbeat Discovery><RP0><08/09/2013 16:02:25>

adh13wp6.DDR:AGENTINFO<SMS_AD_SECURITY_GROUP_DISCOVERY_AGENT><RP0><08/11/2013 00:00:02>

adhz9c4e.DDR:AGENTINFO<SMS_AD_SECURITY_GROUP_DISCOVERY_AGENT><RP0><08/09/2013 16:24:09>

adsxwcj3.DDR:AGENTINFO<SMS_AD_SYSTEM_DISCOVERY_AGENT><RP0><08/09/2013 16:24:00>

adsy4b3j.DDR:AGENTINFO<SMS_AD_SYSTEM_DISCOVERY_AGENT><RP0><08/11/2013 00:00:02>

AKY0SZ7N.DDR:AGENTINFO<Heartbeat Discovery><RP0><08/09/2013 16:08:18>

svrlnecz.DDR:AGENTINFO<SMS_WINNT_SERVER_DISCOVERY_AGENT><RP0><08/12/2013 12:12:01>

UMK1TZIM.DDR:AGENTINFO<Heartbeat Discovery><RP0><08/09/2013 16:08:05>

 

If you are troubleshooting a backlog, this is likely a large list of ddrs. To make it easier to review the data, I import it into excel.

The best way to import it is dependent on what you plan on doing with the results.

You can import with a delimiter of < for starters. The benefit here is that you'll have different columns for the ddr name, ddr type, site code, and date.

In Excel, simply select the data tab and "From Text" to launch the text import wizard. Select delimited and then choose other for the delimiter and type <:

 

You can then select to filter the data to a particular type to find out where your large counts are:

 

Disclaimer: Manually modifying files in the inboxes is unsupported. Microsoft Support should be engaged prior to doing so in a production environment.

 

Occasionally the ddr backlog is identified after it has grown to a point where the data processing will never catch up. In other words, your processing rate and the data incoming are such that the site server will never be able to clear the queued data. In large hierarchies where a problem has occurred, I've seen ddr backlogs of several million files.

In these cases, you might want to get the ddr name in a text file by itself in order to take action. For example, let's say that you find the vast majority of the data is from Active Directory System Discovery. You investigate further and find that you are discovering a trusted domain that you weren't supposed to. Or someone increased the full discovery schedule to run way too frequently. Once you reach a point where you won't be able to catch up, you might say that you need to purge some of the ddrs. You decide that you need to delete all Active Directory System Discovery ddrs from the system.

If you want to get the name of the ddr file in a column by itself, you can use the LEFT function in Excel:

 

In Excel, filter your list of ddrs to SMS_AD_SYSTEM_DISCOVERY_AGENT. Then copy the list of ddrs (see column E above) into a text file and use PowerShell to delete those ddrs.
First you'd open PowerShell and navigate to the appropriate directory

<Installation Directory>\inboxes\auth\ddm.box

Then you'd run the following where DeleteDDRList.txt is the list of ddrs that you pulled from the spreadsheet:

get-content D:\Temp\DeleteDDRList.txt | %{del $_}

The DeleteDDRList.txt contents look like this:

10U4T03Q.DDR

1KBRW3HX.DDR

21PVQKKM.DDR

etc….

 

Hope this helps!

Ryan Anderson

Z8Y2V6DR.DDR

Comments

• Anonymous
  February 08, 2015
  Great post Ryan!
  
  I got a problem with a client this week regarding the exchange connector initially, he had several errors synchronizing the data from the active-sync to SCCM, I end up resolving the problem with the exchange connector, however the problem was resolved, however I was unable to bring the discovered data from the exchange connector to SCCM to the All Mobiles Collection, and it was exactly as your post mentions, DDR backlog, the environment was configured to use all the discovery methods, and the AD System Discovery was running every 5 minutes! It created a monster backlog, the environment was drowning in DDR files, once we resolved this backlog problem, the mobiles were discovered and the environment was happy again.
  
  Thanks again for this great post!
  
  
  Renato Pacheco
• Anonymous
  October 18, 2015
  Superb Post Ryan, it helps a lot....