Site to site IPSEC tunnel Between TMG 2010 on VMware and Cisco

Article
1/17/2024

Issue and Scenario

I recently worked on a case where we were trying to establish a tunnel between TMG 2010 on VMware and a Cisco device. We had configured both sides to match the IPSEC security parameters as per http://technet.microsoft.com/en-us/library/dd441072.aspx . After configuring, we tested the tunnel and connectivity through it but it failed. After ensuring that the configuration on both sides matches as mentioned before, we collected more data to troubleshoot.

Data Collection

We used the TMG data packager with VPN template in repro mode while trying to connect to one of the machine on the remote end from the local end. TMG data packager can be downloaded from here http://www.microsoft.com/download/en/details.aspx?id=17730. Instructions can be found here http://blogs.technet.com/b/sooraj-sec/archive/2010/04/10/instructions-for-isa-data-packager-to-collect-data-in-repro-mode.aspx. This post was written for ISA data packager using web proxy template but in our case we will use VPN template.

Data Analysis

In the IKEEXT logs we found the following:

|x.x.x.x|Looking up MM policy for IKE
[ikeext] 0|x.x.x.x|Policy
GUID: x
LUID: x
Name: x tunnel to network xx
Description: (null)
Flags: 0x00000000
Provider: <unspecified>
Provider data:
Type: IKE Main Mode
Soft expiry: 86400
InitiatorImpersonationType: None
Auth methods: 1
-- 0 --
Type: Preshared Key
Key:
x
Proposals: 1
-- 0 --
Cipher algorithm:
Type: 3DES
Key length: 0
Rounds: 0
Integrity algorithm:
Type: SHA1
Max lifetime (sec): 7200
DH group: 2
QM limit: 0
Flags: 0x00000000
MaxDynamicFilters: 5
[user] |NULL|IkeProcessAcquireDispatch failed with HRESULT 0x80073601(ERROR_IPSEC_IKE_NO_POLICY)
[ikeext] 0|x.x.x.x|Construct IKEHeader
[ikeext] 0|x.x.x.x|Initializing Kerberos SSPI
[user] |x.x.x.x|IkeFindAuthConfig failed with Windows error 87(ERROR_INVALID_PARAMETER)
[user] |x.x.x.x|IkeFindAuthConfig failed with HRESULT 0x80070057(ERROR_INVALID_PARAMETER)
[user] |x.x.x.x|IkeDetermineSspiInfo failed with HRESULT 0x80070057(ERROR_INVALID_PARAMETER)
[user] |x.x.x.x|IkeCreateSspiIke failed with HRESULT 0x80070057(ERROR_INVALID_PARAMETER)
[ikeext] 0|x.x.x.x|WFP free sspi 0000000002F4BF20
[user] |x.x.x.x|IkeGetSspiContext failed with HRESULT 0x80070057(ERROR_INVALID_PARAMETER)
[ikeext] 0|x.x.x.x|Construct SA
[ikeext] 0|x.x.x.x|AUTHIP keying module is not enabled for traffic
[ikeext] 0|x.x.x.x|IKE not sending co-existence Vendor ID
[ikeext] 0|x.x.x.x|Construct VENDOR type MS NT5 ISAKMPOAKLEY
[ikeext] 0|x.x.x.x|Construct VENDOR type RFC 3947
[ikeext] 0|x.x.x.x|Construct VENDOR type draft-ietf-ipsec-nat-t-ike-02

After this Main mode gets established and Quick mode also gets established in one direction and then we get following

Timestamp: x
Flags: 0x00000100
IP version field set
IP version: IPv4
IP protocol: 0
Local address: 0.0.0.0
Remote address: 0.0.0.0
Local Port: 0
Remote Port: 0
Application ID:
User SID: <invalid>
Failure type: IKE/Authip Quick Mode Failure
Type specific info:
Failure error code:0x00003601
No policy configured

For No policy configured, we already made sure that Address ranges on both sides are configured properly as per http://blogs.technet.com/b/sooraj-sec/archive/2011/06/20/ipsec-site-to-site-tunnel-between-tmg-and-cisco-fails-another-scenario.aspx.

Solution/WorkAround

After researching, I found some old cases where this issue was reportedly due to the virtual NICs of VMware. We then started focusing on that part and changed the virtual NICs and configured the virtual switch to permit Promiscuous Mode on it. After doing that we tested again and the issue was resolved.

Share via

Site to site IPSEC tunnel Between TMG 2010 on VMware and Cisco

Additional resources