通过

RPC Filter and "Enable strict RPC compliance"

  • 项目

Configuring the RPC filter to support DCOM traffic is a particular pain point in ISA Server configuration. This entry provides a quick overview of the filter, the implications of the "Enable strict RPC compliance" setting, and some information on common issues with DCOM traffic.

RPC Filter

ISA Server's RPC filter monitors RPC traffic between hosts, and sets up secondary connections as required for RPC traffic. For outbound RPC requests, ISA Server inspects the traffic flowing between the source and destination. For incoming requests to published RPC servers, ISA Server inspects the traffic flowing between the source and destination, and dynamically opens and closes ports on the external published listener based on the protocols used by the RPC client and server. The RPC filter cannot be applied to traffic tunneled over another protocol, such as RPC over HTTP. When a rule references a protocol that is bound to the RPC filter, then the filter is applied to traffic matching the rule. By default, ISA Server provides three predefined RPC protocols for use by inbound and outbound RPC traffic:

  • The Exchange RPC Server protocol has a list of UUIDs used for publishing Exchange.
  • The RPC Server (all interfaces) protocol is used for publishing other RPC servers.
  • The RPC (all interfaces) protocol is used in access rules for outbound RPC access.

By default the predefined RPC protocols are bound to the RPC filter. You can also create custom protocol RPC definitions using the New RPC Protocol Definition Wizard. When you create a custom RPC protocol using the wizard, the following defaults are applied:

  • Port TCP 135 is enabled for the custom protocol
  • The custom protocol is bound to the RPC filter

Note that traffic defined as "outbound" is not handled by ISA Server based on specific UUIDs, so it isn't possible to set up a custom protocol definition for specific UUIDs. For traffic defined as "incoming", you can create a custom protocol with specific UUIDs, either by selecting them from the endpoint mapper list, or by manually creating them.

Enable strict RPC compliance

For publishing rules ISA Server blocks DCOM traffic, and this setting cannot be modified. For access rules, a default "Enable strict RPC compliance" setting is configured on each RPC rule. With this setting in place, DCOM traffic is blocked. More specifically, any traffic (such as DCOM) that does not start an RPC exchange by communicating with the endpoint mapper is blocked. Turning off the "Enable strict RPC compliance" setting does not specifically allow DCOM traffic. It simply disables filtering for this traffic after the endpoint mapper requirements have been met. To allow DCOM traffic through an RPC access rule, either of the following is required:

  •  An access rule that allows all protocols between the specified source and destination.
  • Alternatively, you can do the following:
    • Create a custom outbound protocol using a port that is not associated with any other application.
    • Configure the RPC application or DCOM endpoint to use the custom protocol port as a static port.
    • Create an access rule to allow the protocol between the required source and destination.

Hints for Troubleshooting RPC Server Publishing

  • Ensure that you are using a recognized protocol definition in the rule. Either use a predefined protocol, or ensure that the custom protocol is defined correctly. Custom protocol definitions should be inbound, TCP port 135, with the correct UUID interfaces.
  • Check that the publishing rule is enabled.
  • Verify that the RPC filter is enabled for the protocol - to handle the secondary connection, and inspect the traffic.
  • For Exchange server publishing use the New Mail Server Publishing Wizard. To publish other RPC servers, create a server publishing rule for the internal server, using the IP address of the adapter associated with the ISA Server External network.
  • Check that the external IP address of the ISA Server and the IP address of the internal server are specified correctly in the rule.
  • Ensure that a network rule between the source and destination networks exists, and that the rule relationship is appropriate to the traffic flow.
  • Check that the RPC client/server application is working without ISA Server in the middle, by using a client in the Internal network to communicate with the published RPC server.

 Common Issues

  • Problem: You cannot use DCOM between a computer in the Remote Management Computers sets and the ISA Server computer
  • Workaround: In the system policy rule, there is no option to configure remote management to allow non-strict RPC traffic, so all DCOM traffic between the Remote Management Computers set and the Local Host network (the ISA Server) is dropped. As a workaround, remove the computer from the set, and create an additional access rule for the same traffic. Then clear the "Enforce strict RPC compliance setting" on the rule.

 

  • Problem: When you request a certificate using the Certificate MMC snap-in, the request fails. This occurs even if the CA is started and you have sufficient permissions to request a certificate.
  • Workaround: This issue occurs because DCOM is required to acquire a certificate (this issue also occurs if you are using CA Web enrollment).
    • If ISA Server is requesting the certificate, disable the "Enforce strict RPC compliance setting" on the system policy rule. To do this, on the Firewall Policy tab of ISA Server Management, click Edit System Policy on the Tasks tab. Select the Active Directory group in the Configuration Groups list. On the General tab, clear the Enforce strict RPC compliance checkbox.
    • If an internal host is requesting the certificate from another network through ISA Server, do the following: in the Firewall Policy tab of ISA Server Management, right-click the access rule allowing the traffic, and then click Configure RPC protocol. On the Protocol tab, clear Enforce strict RPC compliance.
  • NOTE: In either case, after clearing the setting you need a rule to allow all traffic, or a rule for a custom protocol as described in the blog section "Enable strict RPC compliance". For more information on CA configuration in a firewall environment, the following document is a useful reference: http://www.microsoft.com/technet/prodtechnol/windowsserver2003/technologies/security/webenroll.mspx.

 

 

 

  • Problem: WMI scripts fail to run from remote systems
  • Workaround: Create an access rule that allows RPC (All Interfaces) from the Internal network to the Local Host network. After creating the rule, right-click it and select Configure RPC Protocol. On the Protocol tab, clear Enable strict RPC compliance. After clearing the setting, you need arule to allow or traffic, or a custom protocol as described in the blog section "Enable strict RPC compliance".

Rayne Wiselman

ISA Server Product Team

Comments

  • Anonymous
    January 01, 2003
    Applications that want to talk to other servers will often use the Remote Procedure Call (RPC) infrastructure

  • Anonymous
    January 01, 2003
    I wonder why don't you distribute this article inside your team.. We spent a whole day on a Severity A case collecting and recollecting mutiple diagnostic data from ISA until someone suddenly said: "Hey, we don't support WMI, and we don't care if all other MS products rely upon it". Thank's a lot...

  • Anonymous
    January 01, 2003
    So, is it not possible to configure Windows EBS 2008 security server to connect through WMI? If it is possible, how do we connect? I created a filter and unchecked the "Strict RPC compliance" but that didnt help.

  • Anonymous
    January 01, 2003
    Hi Rayne, Excellenet article! Lots of interesting and useful information Thanks! Tom

  • Anonymous
    June 01, 2007
    Thank you for the article, very helpful.   Is this going to be fixed in the next release of ISA?  It's a rather annoying bug...

  • Anonymous
    October 02, 2007
    Are there any issues with RPC publishing on ISA 2006 with Exchange 2007 and Outlook 2007? I can't  'stay' connected, it keeps dropping off and reconnecting. Outlook 2k/Ex07 works though. Thanks Brad

  • Anonymous
    October 17, 2007
    Recently I had to re-install WinXP on a workstation. This caused non-critical login errors in the server security log, event ID 537. I eventually tracked these errors back to automatic certificate enrollment errors, "RPC Server is unavailable", at the workstation. The workstation was requesting a computer certificate. In my first effort at debugging this problem I disabled the "strict RPC compliance", restarted the ISA 2004 firewall, and manually requested the certificate. No luck! Same error! Next I  disabled the RPC filter, restarted the firewall, and requested the certificate. The certificate request worked. I enabled the RPC filter and restarted the firewall. When I checked the logs today everything is normal. I have noticed in several forums that several people were successful at resolving their RPC issues only by turning off the RPC filter. Where should I be looking to fix this problem?

  1. I believe I have a fully patched ISA 2004 server.
  2. The workstation is in the "protected networks" so all protocols should be available. 3. The ISA log does not show any error messages for this workstation. ISA does not "appear" to be preventing the connection.