Share via

Forefront Threat Management Gateway (TMG) 2010 Troubleshooting Survival Guide

  • Article

We encourage you to enhance this guide by identifying missing areas (scenarios, features, lifecycle...), provide links to and write descriptions of existing content, and providing new content where there are gaps. Join the community!
**
**

[Introduction

](https://social.technet.microsoft.com/wiki/contents/articles/2702.forefront-threat-management-gateway-tmg-2010-troubleshooting-survival-guide/edit.aspx#Intro)Troubleshooting Tools

Troubleshooting Setup

Troubleshooting Outbound Access

Troubleshooting Performance

Troubleshooting E-Mail Protection

Troubleshooting VPN

Troubleshooting Report

Troubleshooting Web Publishing

Introduction

When approaching to troubleshooting any product there are some general guidelines that should be followed. The WSUS Troubleshooting Survival Guide has the seven core steps that should be used. When the subject is TMG, there are many areas that can be explored from the troubleshooting standpoint. This article will cover the main areas and you can expand by adding new areas with core troubleshooting techniques.

Troubleshooting Tools

Forefront TMG 2010 comes with a built in set of tools that can help you to troubleshoot a great variety of scenarios. Those options are located in the Troubleshooting pane as shown in the figure below:

The Troubleshooting pane has the following options:

  • Change Tracking - allows you to enable the capability to track all the changes that are done on TMG. Check a demo of this feature on ISA Server 2006 SP1 here.
  • Traffic Simulator - allows you to troubleshoot access rule, web publishing and server publishing rule by simulating a network traffic that will test one of those scenarios. Check a demo of this feature on ISA Server 2006 SP1 here.
  • Diagnostic Logging - this feature was first introduced in ISA Server 2004 SP3 and it helps to deeply understand each step of the packet evaluation on TMG. Check a demo of this feature on ISA Server 2006 SP1 here.
  • Connectivity Test - performs a simple test against a specific URL.

Note: You can also find an overview of each one of those options on the article Overview of the TMG Firewall’s Troubleshooting Node.

Besides those options there is also an important tool that can be used for proactive and reactive scenarios, this tool is called TMG Best Practices Analyzer. In addition to those direct related TMG tools, there are also other tools that can be very useful while troubleshooting TMG issues, here are some examples:

  • Network Monitor: this tool can assist you analyzing packages and have a good understanding of what’s going on in the TCP/IP level. Here are some sample scenarios where this tool was used:

o    Error 64 - From the Field to the Classroom

o    Error 64 “ The specified network name is no longer available” while browsing Internet through ISA Server 2006

  • TCPView: this tool can assist you by viewing the TCP connections between localhost and different systems. Is similar to netstat –nao command but in a graphic interface. Here a sample scenario where this tool can be used: Unable to Access HTTPS Sites behind TMG 2010
  • Process Monitor: when dealing with Performance issues on TMG, Process Monitor can be a very useful tool for an initial assessment and identification of potential culprits. Here are some sample scenarios where this tool was used:

o    Another Case of High CPU Utilization by wspsrv.exe on Forefront TMG 2010

o    Unable to Install ISA BPA

  • HTTPWatch: this is a great tool that allows you to see the content of pages that are using https and troubleshoot issues involving web access or web publishing rules. Here a sample scenario where this tool can be used: Error “Object doesn’t support this property or method” while browsing a site published by ISA Server 2006.
  • WinDBG: this tool can be used in scenarios where you need to analyze a dump that was gathered during a crash or hang. It can also be used to attach to a process and get more information about that particular process. Here are some sample scenarios where this tool was used:

o    ISA Server 2006 Firewall Service not starting

o    The Curious Case of TMG Stopping Responding in Random days but always during the Morning

Troubleshooting Setup

Now that you know the most common tools to troubleshoot issues on Forefront TMG, it’s time to have a look on how you should approach in order to troubleshoot TMG Setup. Forefront TMG setup introduces a tool called Preparation Tool that assists to install TMG’s pre-requisites components. After the OS is fully prepared the next phase is to install TMG’s components. During the whole setup process TMG stores log’s information at %windir%\temp, the logs that are added to this folder can be found it here.

At this point in time that we already have Forefront TMG 2010 SP1 and many other updates, it is recommended to always run on the latest and greatest version. One way to start Forefront TMG installation by having at least Service Pack 1 is by slipstreaming TMG with SP1, the procedure to do that can be found it here. If you decide to install Forefront TMG 2010 RTM, test the functionality and just after that install SP1 you also can. If you have problems to install Forefront TMG SP1 follow this article.

Here are some important articles that outline major installation issues and how to solve those:

Setup issues are not always related to the TMG installation itself, sometimes a setup also means the lack of capability to join a new TMG to an existing array. Here are some important articles in this area:

Troubleshooting Outbound Access

When troubleshooting Outbound Access on TMG you must understand which area of TMG you should focus first. But even before that you will need to understand the problem and that’s why those seven steps mentioned in the beginning of this article are so important. To determine which area of TMG you should focus while troubleshooting Outbound Access, ask questions, for example: what’s the error message that the client receives when tries to browse to the web site that doesn’t work? Is this the only user experiencing this problem? Does the problem happens all the time or it’s random? Is this the only web site that this user cannot access? If this user logs on in another workstation, does the problem happen? Does the issue happen using any Browser? Does the issue happen when bypass TMG?  The answer for those questions can lead to a more narrowed scenario where you should understand which feature or which setting could be causing this problem. Even better, you could just determine that the issue is not caused by TMG at all.

The core features used in the Secure Web Gateway scenario (Outbound Access) are specified in the table below:

Feature

 

 

Troubleshooting Approach

 

 

Common Problems (Samples)

 

 

URL Filtering

 

 

Troubleshooting URL filtering

 

 

HTTPS Inspection

 

 

Troubleshooting HTTPS inspection

 

 

NIS

 

 

Troubleshooting NIS

 

 

Besides those three core features there are other areas of Outbound Access that also needs attention:

o    Troubleshooting Authentication Issues in ISA Server Using Net Logon Logging

o    Random authentication prompts while accessing internet through ISA Server followed by ISA Server becoming unresponsive

o    Understanding Why ISA Server re-prompts for Authentication when Passwords Expire

o    Another Case where Users are randomly prompted for Authentication while Browsing Internet through ISA Server 2006

o    Troubleshooting Intermittent Pop-up Credentials in ISA Server 2004

  • Caching: sometimes the object that you see while browsing Internet through TMG might not be what you are expecting. Caching issues can also be treated as an important part of Outbound Access troubleshooting. Here are some important articles in this area:

o    Files larger than 512MB are not served from cache after ISA Server firewall service is restarted

o    Unable to download files larger than 4GB through ISA 200x – works fine in TMG

Troubleshooting Performance

When troubleshooting performance issues on TMG it is important to look outside of TMG itself and make sure to have a broader view of the system where TMG is installed as well as the environment. Performance issues on TMG can be located at the OS level itself or an outside element, such as network environment.  The core elements to address are:

By analyzing the OS components in first hand you can eliminate potential issues that are affecting TMG. Performance Monitor is one of the best ways to address issues of this nature, the main counters to be used in this area are:

  • All TMG Counters/*
  • Memory/*
  • Processor/*
  • Network Interface/*
  • Process/*
  • Physical Disk/*
  • Threads/*

Use TMG PAL template in order to analyze the data captured using Performance Monitor, this tool can facilitate the analysis process by giving you a comprehensive report highlighting the main findings.  In some scenarios it is not possible to have a conclusive result based only on Perfmon data, in those cases you might need to capture user or kernel memory dump in order to find out the root cause of the problem. To capture dump you can use the same approach from the article “We are all waiting for you Mr. Disk….are you there?” and once you have the dump you can use the Troubleshooting Forefront TMG 2010 Performance issues Cheat Sheet to analyze it.

The most common causes of performance issue on Forefront TMG are:

  • Disk

o    How Disk Bottleneck can affect TMG Performance?

  • DNS

o    Side Effects of Incorrect DNS configuration on ISA Server: 10060 Connection Timeout Scenario

  • Inappropriate Rule Set

o    ISA Server 2006 stops answering requests

  • Third Party Applications

o    Isolating problems that seems to be related to the ISA Server – Part III

o    ISA Server Stop Answering Requests and Firewall Service Hangs

o    TMG Hangs and requires a manual restart

  • Sizing

o    Port Exhaustion on ISA Server 2006 while Publishing Outlook Anywhere

o    Unable to send messages from Outlook behind Forefront TMG after migrating to Cloud Services

  • Network

o    What can happen when you think that only Windows system needs to be patched

o    Understanding a scenario where TMG drops the packet as spoofed even when the source IP doesn’t belong to the internal network

  • Windows Settings

o    Another performance caveat when troubleshooting TMG or ISA slow browsing behavior

  • TMG Logging

o    Intermittent Performance Problem while Accessing Internet through ISA Server 2006

  • Authentication

o    Hey DC, are you still there?

Troubleshooting E-Mail Protection

E-Mail Protection feature on TMG is a combination of Forefront Protection for Exchange and Exchange Edge on the same server as TMG is installed. If you don’t have those products you shouldn’t enable E-Mail Protection feature in the first place, this will cause issues. It is strongly recommended to review the E-Mail Protection requirements before enabling this feature. Currently (TMG 2010 SP1 + Updates) requires that ALL configurations MUST be done via TMG 2010 Console. When the user changes something that TMG has no control of, TMG doesn’t care and it is up to the user to make sure the settings are duplicated across the array. However, if the user tries to change something that TMG controls it may lead to an invalid configuration and cause TMG to function incorrectly. Therefore TMG will not permit such a change. It will remove the user’s changes by resetting Exchange configuration back to the one in TMG storage. To check for changes we will use ADAM’s built-in support for “checkHighestUSn”, an LDAP query that queries the entire ADAM structure for the highest USN. Changes done directly on Exchange Edge Console/Powershell or FPE Console/Powershell will be overwritten by TMG. When this happens the following alert will appear on TMG:

Here it is a list of the top five more common problems while configuring/administering TMG E-Mail Protection:

Scenario 1: making change directly on Exchange Edge or FPE

  • Result: Forefront TMG Managed Control Service might fail to start with error 0x80070057.
  • Solution: remove the changes that were manually added to Exchange or FPE.

Scenario 2: IPs getting populated on the IP Block List directly on Exchange

  • Result: Forefront TMG Managed Control Service stops and fail to start with error 0x80070057
  • Solution: disable Sender Reputation feature via TMG Console (under Spam Filtering).

Scenario 3: Installing Exchange 2010 SP1 Slipstream during the installation of E-Mail Protection Pre-Reqs

  • Result: Forefront TMG Managed Control Service might fail to start with error 0x80131500 .
  • Solution: don’t use Exchange 2010 SP1 Slipstream while installing the pre-reqs for E-Mail Protection. Use RTM and apply SP1 after having TMG 2010 SP1 Update 1.

Scenario 4: Action: Trying to make changes on settings that are not exposed via TMG Console directly via FPE or ExchangeEdge

  • Result: Forefront TMG Managed Control Service will overwrite the option and undo the change.
  • Solution: don’t use the options that are not exposed via TMG Console

Scenario 5: Action: Install Exchange 2010 SP1 on a Server using E-Mail Protection feature and having TMG 2010 SP1 on it

  • Result: Forefront TMG Managed Control Service might fail to start with error 0x80070057
  • Solution: Install TMG 2010 SP1 Update 1

Keep in mind the following points while troubleshooting E-Mail Protection issues:

  • TMG Live Logging just shows the SMTP connections coming in and out, nothing more than this.
  • TMG Live Logging will be useful is to validate if SMTP connection is established or not and which rule is hitting.
  • TMG Trace (using TMG Data Packager) will go a little further, but not much since it only logs the changes that are applied to the system.
  • Is okay to use Powershell commands or other tools for Data Gathering purpose, as long as is for read only purpose is okay. Direct changes via Powershell on Exchange Edge or FPE will be overwritten by TMG.

TMG Trace (gathered via TMG Data Packager) most likely will not be helpful in the following scenarios:

  • Mail flow issues or NDRs

o    An Exchange Edge expert should be involved.

  • Messages are getting incorrectly stamped as SPAM or getting dropped due to virus detection

o    Identify which setting is controlling that and engage the correct engineer (FPE or Exchange)

Besides that you also have the following articles that can be used while troubleshooting E-Mail Protection on TMG:

Troubleshooting VPN

The VPN feature on Forefront TMG is totally based on Windows Server 2008 functionality, in other words, it depends on RRAS functionality. This means that using the traditional Windows Server Routing and Remote access troubleshooting approach is valid.

VPN Client Access

Here are some resources to assist you during the VPN Client access troubleshooting:

Site to Site VPN

Here are some resources to assist you during the VPN Site to Site troubleshooting:

Troubleshooting Report

When troubleshooting reporting issues there are three core areas that need attention:

  • Reporting configuration
  • Report generation
  • Summary reports.

The general troubleshooting report framework can be found in this article. Some issues arise when TMG 2010 SP1 was launched, such as the one explained in the TMG Reports stop working after installing TMG 2010 SP1 blog post. Here are some other related articles on reporting issues:

Troubleshooting Web Publishing

Forefront TMG 2010 has a set of features that can assist you while deploying a scenario that requires high availability. Here are some core TMG functionalities in this area:

Outlook Web Access (OWA)

Use the core troubleshooting methodology exposed in the article Troubleshooting OWA 2007 Publishing Rules on ISA Server 2006, although the article is for ISA the steps there does apply to TMG. In addition to that, keep in mind the following common problems:

SharePoint

Most of the issues publishing SharePoint through TMG are similar to what we used to have in the past with ISA, which is how to proper configure AAM. Here an example of this scenario: Unable to “Check Out” a Document in MOSS 2007 Published Through ISA Server 2006. In addition to that, it is always recommended to use TMG Data Packager to troubleshoot issues of this nature.

Authentication

Authentication issues in a publishing scenario are usually caused by one of the following components:

 

A – Client to TMG Authentication

  • When client is trying to authenticate against TMG in a publishing scenario you must choose an authentication method that it supported by the client, for example: in the scenario above you have the laptop that is trying to access OWA and a mobile device that is trying to synchronize via Active Sync. If you publish Exchange using Forms Based Authentication for the OWA and use the same rule for Active Sync it will work because FBA falls back to Basic and Active Sync supports basic. Of course the assumption is that you are using HTTPS to avoid passing clear text using Basic Authentication. The most relevant authentication settings in this scenario (A) are located in the Web Listener.

 B – TMG to published Server

  • The delegation authentication that TMG uses against the published server, for example when you are publishing OWA and choose Basic as delegation you must make sure that the Exchange CAS Server is also using basic for OWA folder. The delegation must match with the published server. The most relevant authentication settings in this scenario (B) are located in the Publishing Rule / Delegation Tab.

C – TMG to Authentication Repository

Other sources of investigation in the authentication scenario are described below:

Password Management

When using the built-in functionality of providing users with a warning message and a password change prompt, in case their passwords have expired, or are about to, you may encounter difficulties if your domain uses a fine-grained password policy.

Microsoft Forefront TMG and ISA do not support the use of fine-grained password policies.

Call of Action

This is a living document that we are starting now and giving it to you as a base to expand it. Do you want to get engaged on this? Make sure to read the guidelines from Wiki: How to Contribute and have a great time helping the community to grow.

Note: do not add troubleshooting articles in this Survival Guide, we are working to build a Troubleshooting Survival Guide for Forefront TMG 2010. Once we have it we will post it here.

This article was originally written by:

Yuri Diogenes, Senior Technical Writer
**
Windows Server iX | IT Pro Security

Microsoft Corporation

**--------

Yuri’s Blog: http://blogs.technet.com/yuridiogenes
Team’s Blog: http://blogs.technet.com/b/securitycontent
Twitter: http://twitter.com/yuridiogenes

See Also