Azure Virtual Network: Cross-Tenant Vnet-to-Vnet Connection Fails with 'InternalRetriableError

geoffrey 5

We have been using New-AzVirtualNetworkGatewayConnection to establish cross tenant Vnet-to-Vnet connections for years, and even one a few months ago. Now we are getting an error message:


New-AzVirtualNetworkGatewayConnection: 
Line |
  39 |  New-AzVirtualNetworkGatewayConnection `
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The operation cannot be completed due to an internal retriable error. Please retry in sometime.
StatusCode: 500
ReasonPhrase: Internal Server Error
ErrorCode: InternalRetriableError
ErrorMessage: The operation cannot be completed due to an internal retriable error. Please retry in sometime.

for about a week. This error occurs when attempting from either side. We've noticed that we can still make Vnet-to-Vnet connections within the same subscription, which tells me that the admins trying this have proper rights, and also that the Vnets have not hit their ceiling of max connections.

Several attempts adhering to the latest documentation have erred similarly. ( https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-vnet-vnet-rm-ps) following the format of example.

New-AzVirtualNetworkGatewayConnection -Name $Connection51 -ResourceGroupName $RG5 -VirtualNetworkGateway1 $vnet5gw -VirtualNetworkGateway2 $vnet1gw -Location $Location5 -ConnectionType Vnet2Vnet -SharedKey 'AzureA1b2C3'

Sai Prasanna Sinde 3,845 Reputation points Microsoft Vendor

2025-02-12T08:04:24.9733333+00:00
Hi @geoffrey

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

Please go through the below troubleshooting points:

As per the above information and based on the error: "InternalRetriableError", the most likely cause of this issue is a temporary disruption within the Azure Virtual Network Gateway service. While retrying the operation and checking the Azure Service Health dashboard, specifically for any reported issues in the regions where your Virtual Network Gateways reside.

The error message you provided is generic. Look for more detailed logs. Azure Activity Logs should contain more information about the specific error. Filter by the time of your connection attempts and the resource type "Virtual Network Gateway Connection".

Use the Azure Resource Explorer) to examine the properties of your Virtual Network Gateways and the connection you're trying to create. Sometimes, inconsistencies in the configurations can cause these errors.

Make sure that the SKUs of your Virtual Network Gateways are compatible for cross-tenant connections. Basic SKUs might not support it. Standard, HighPerformance, or UltraPerformance are generally required.

If you're on a lower SKU, consider temporarily upgrading one of the gateways to a higher SKU to see if that resolves the issue. (Remember to downgrade if it doesn't help to avoid unnecessary costs)

As we can see you're using -ConnectionType Vnet2Vnet, which is correct for cross-tenant connections. Just double-check you haven't accidentally used a different type.

Use Network Watcher's IP flow verify to check if traffic is being blocked between the VNet gateways. This can help identify if a NSG or firewall rule is interfering.

Network Watcher also has VPN diagnostics capabilities. These can provide deeper insights into the health and status of your VPN gateways and connections.

Make sure that you're using the latest version of the Az PowerShell module. Run Update-Module Az to get the latest version. Sometimes, bugs in older modules can cause these issues.

Kindly let us know if the above helps or you need further assistance on this issue.
Sai Prasanna Sinde 3,845 Reputation points Microsoft Vendor

2025-02-13T08:21:38.4766667+00:00

Hi @geoffrey,

Just checking in to see if you had a chance to see my response to your question.

Please tell us if it was helpful and let us know if you have any questions in the comments below so that we can address them.
geoffrey 5 Reputation points

2025-02-13T15:54:40.45+00:00

Yes, we have gone over the suggestions. The code and azure resources we are using are consistent with what is currently working with other VPN connections in the very same tenants where we are now getting the error. I haven't seen any alert or outage that would affect this. Furthermore, vnet peering is working, but our preferred solution is the vnet-to-vnet, which has failed every attempt for about a week.

Share via

Azure Virtual Network: Cross-Tenant Vnet-to-Vnet Connection Fails with 'InternalRetriableError

Your answer