Managed identity doesn't allow permissions to join subnet, but it's owner on that vnet/subnet

Diko Parvanov 0

I have an Azure managed application, deployed from the Azure Marketplace with an ARM template and with a managed identity assigned to a VM, that lives in the managed resource group. The managed identity has Owner permissions over the resource group so it should be able to do anything - I can create/delete vnets/subets with az cli freely.

This dfa2e8dd-df89-40e2-bbb2-9b77cdcb4d7c is the VM object managed identity.

Now I am trying to add backend servers to an application gateway backend pool using:

az network application-gateway address-pool create --gateway-name $APP_GATEWAY_NAME -n mybackendpooltest -g $RG --servers $IPS

What I am getting back is:

(ApplicationGatewayInsufficientPermissionOnSubnet) Client with object id dfa2e8dd-df89-40e2-bbb2-9b77cdcb4d7c does not have permission on the Virtual Network resource /subscriptions/XXXXXXXXXXXXX/resourceGroups/YYYYYYYYYYYYYY/providers/Microsoft.Network/virtualNetworks/AppGatewayVnet/subnets/AppGatewaySubnet to perform action Microsoft.Network/virtualNetworks/subnets/join/action. For details on the required permissions, please visit https://aka.ms/agsubnetjoin.

When I run the same command with the external tenant that has full access over the managed application - the command succeeds. I have the same permissions as the external object (Owner) of the resource group and all resources.

I tried manually assigning this system managed identity explicit access over the Vnet and Subnet with:

"Network Contributor" role - no help
"Domain Services Contributor" - no help
"Owner" role - no help

The reason I am adding the backend servers later on are that during the ARM template deployment the machines are not known yet and are created afterwards.

What can I do to unblock this?

Silvia Wibowo 5,281 Reputation points Microsoft Employee

2025-01-16T03:01:27.21+00:00

Hi @Diko Parvanov , as you are using managed applications from Marketplace, please refer to their documentation on what you can or cannot do.
Diko Parvanov 0 Reputation points

2025-01-16T05:34:19.86+00:00

Hi @Silvia Wibowo there is no documentation regarding this scenario. For the record the managed application is developed by me, so I have full control on how to configure it and I have granted full Owner permissions on both the system identity of the VM, added also and tried with a user managed identity with Owner permissions - I have the same problem. The object has Owner access to the resource group and all resources, but fails to update anything on an application gateway with the error I've pasted.
Rohith Vinnakota 2,330 Reputation points Microsoft Vendor

2025-01-16T06:39:00.0233333+00:00

Hi @Diko Parvanov

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.

To resolve the issue, you would need to assign the managed identity used by VM, Network Contributor role on the virtual network/subnet, so it has permissions for Application Gateway to join the subnet.

In the Azure portal, this can be done by navigating to the virtual network, select the subnet Application Gateway is deployed in, click Managed users, click Add on Access Control, select Add role assignment, search for Network Contributor, select Network Contributor and click Next, click Select Members, search for the managed identity used by VM and click Select, click Review + assign, click Review + assign again.

Hope this helps!

If above is unclear and/or you are unsure about something add a comment below.

Please don’t forget to close the thread by clicking "Accept the answer" wherever the information provided helps you, as this can be beneficial to other community members.

Regards,

Rohith

Diko Parvanov 0

Ftr, this is the access my object has over the resource group, the vnet and subnet:

az role assignment list --assignee dfa2e8dd-df89-40e2-bbb2-9b77cdcb4d7c --all --output json --query '[].{principalName:principalName, roleDefinitionName:roleDefinitionName, scope:scope}'

[
  {
    "principalName": "2bb366f8-ac24-46fe-bc49-42fdbc9fa939",
    "roleDefinitionName": "Owner",
    "scope": "/subscriptions/XXXXXXXXXXXXX/resourcegroups/YYYYYYYYYYYYYY"
  },
  {
    "principalName": "2bb366f8-ac24-46fe-bc49-42fdbc9fa939",
    "roleDefinitionName": "Domain Services Contributor",
    "scope": "/subscriptions/XXXXXXXXXXXXX/resourceGroups/YYYYYYYYYYYYYY/providers/Microsoft.Network/virtualNetworks/AppGatewayVnet"
  },
  {
    "principalName": "2bb366f8-ac24-46fe-bc49-42fdbc9fa939",
    "roleDefinitionName": "Network Contributor",
    "scope": "/subscriptions/XXXXXXXXXXXXX/resourceGroups/YYYYYYYYYYYYYY/providers/Microsoft.Network/virtualNetworks/AppGatewayVnet"
  },
  {
    "principalName": "2bb366f8-ac24-46fe-bc49-42fdbc9fa939",
    "roleDefinitionName": "Network Contributor",
    "scope": "/subscriptions/XXXXXXXXXXXXX/resourceGroups/YYYYYYYYYYYYYY/providers/Microsoft.Network/virtualNetworks/AppGatewayVnet/subnets/AppGatewaySubnet"
  },
  {
    "principalName": "2bb366f8-ac24-46fe-bc49-42fdbc9fa939",
    "roleDefinitionName": "Contributor",
    "scope": "/subscriptions/XXXXXXXXXXXXX/resourceGroups/YYYYYYYYYYYYYY/providers/Microsoft.Network/virtualNetworks/AppGatewayVnet/subnets/AppGatewaySubnet"
  }
]

and I am still getting this error ApplicationGatewayInsufficientPermissionOnSubnet, also on the UI I get User's image

User's image

But there is no option to fix that.

Diko Parvanov 0 Reputation points

2025-01-16T06:45:32.71+00:00

@Rohith Vinnakota I've done that multiple times, with all possible combinations on both UI and CLI, on both Vnet and Subnet, with all possible roles - Owner, Network Contributor, Domain Services Contributor - none if it works. And I am not using AGIC/AKS here, it's just an application gateway with backend pool of normal VMs.
Diko Parvanov 0 Reputation points

2025-01-16T06:48:00.1533333+00:00

Hi Rohith Vinnakota I just pasted screenshots and output from the CLI with the object permissions and clearly it should have them, is there any other way to fix this?
Rohith Vinnakota 2,330 Reputation points Microsoft Vendor

2025-01-16T07:30:45.8033333+00:00

Hi @Diko Parvanov ,

Thank for reply.

Can you create the custom role with the permissions below and assign this role to the ApplicationSubnet , as mentioned in the comments above

I hope it helps.

Thanks,

Rohith
Diko Parvanov 0 Reputation points

2025-01-16T07:39:53.2633333+00:00

I did that too - didn't help.
Diko Parvanov 0 Reputation points

2025-01-16T07:50:04.65+00:00

This is the customer role I've tried with:
Diko Parvanov 0 Reputation points

2025-01-16T08:15:01.1966667+00:00

Here is the effective role list of the system managed identity on the VM over the subnet and vnet:

and I've created a new role just to test:
Rohith Vinnakota 2,330 Reputation points Microsoft Vendor

2025-01-16T09:58:50.3666667+00:00

Hi @Diko Parvanov

Thank for reply.

Are you adding permissions to user level or client (service principal) level?

If you are using user-level access, can you use the client ID: dfa2e8dd-df89-40e2-bbb2-9b77cdcb4d7c

I hope it helps.

Thanks,

Rohith
Diko Parvanov 0 Reputation points

2025-01-16T10:07:34.0633333+00:00

Hi Rohith Vinnakota,

I am adding the permissions to service principal client dfa2e8dd-df89-40e2-bbb2-9b77cdcb4d7c - this is the object of the managed identity assigned to the VM, where I am running the command.
Diko Parvanov 0 Reputation points

2025-01-16T11:03:32.58+00:00
And the strange thing is - this error appears even when I try to run a command like this:

az network application-gateway ssl-cert create --gateway-name AppGateway -g myRg --name cert --cert-password 123456

So there definitely is something wrong with Azure, as this doesn't touch the networks at all. I presume this is coming from the fact that the PUT http request towards management.azure.com contains the whole application gateway configuration in the request body, thus making the API thing it's trying to update the network config, where in fact it is not.
Rohith Vinnakota 2,330 Reputation points Microsoft Vendor

2025-01-20T15:58:39.8666667+00:00

Hi @Diko Parvanov,

sorry for delay,
We are looking into it and reviewing the documents and will get back to you soon.
Thanks,
Rohith
Vallepu Venkateswarlu 0 Reputation points Microsoft Vendor

2025-01-28T12:11:59.1533333+00:00

Hi @Diko Parvanov
I've done that multiple times, with all possible combinations on both UI and CLI? if you enabled System Managed Identity on VM, The Identity access will only work inside the VM and not from outside Azure VM, like both Portal UI and CLI.

You can use commands to log in with a System Managed Identity in PowerShell using Connect-AzAccount -Identity, and in the CLI using az login --identity inside the Azure VM.

Reference: https://learn.microsoft.com/en-us/cli/azure/authenticate-azure-cli-managed-identity

https://learn.microsoft.com/en-us/powershell/module/az.accounts/connect-azaccount?view=azps-13.1.0
Diko Parvanov 0 Reputation points

2025-01-28T12:33:09.9333333+00:00

What I've done multiple times via the UI and CLI is to assign the access of the custom roles and ownership of the managed identity on the resource group and over the vnet and subnet. I am only using the CLI inside the VM, there is no other option to use it.
Vallepu Venkateswarlu 0 Reputation points Microsoft Vendor

2025-01-29T05:23:31.88+00:00
Can you please share the details below for further clarification?

Please share the output of az account show from the VM, as shown below.

Screenshot of the VM Identity being enabled, as shown below.

Role assignment screenshot to identity

4. Share Vnet and application-gateway resource group screenshot.
Diko Parvanov 0 Reputation points

2025-01-29T07:58:52.23+00:00

I've masked some sensitive information, below the screenshots as requested:
Vallepu Venkateswarlu 0 Reputation points Microsoft Vendor

2025-01-30T07:20:43.6133333+00:00

Hi @Diko Parvanov

Thanks for the replay.

Could you please provide access at the subscription scope for testing, check whether it's working, and also try using PowerShell instead of Azure CLI?
Diko Parvanov 0 Reputation points

2025-01-30T09:31:37.18+00:00

Running with PowerShell the command succeeds, no error, but then the ProvisioningState of the pool is empty and doesn't get created at all. Sames is valid for the certificated, doesn't show in the UI and doesn't get provisioned, but no errors.

How/to whom do I give access over this to troubleshoot?
Vallepu Venkateswarlu 0 Reputation points Microsoft Vendor

2025-01-30T11:37:34.7433333+00:00

Thanks for reply,

Are you facing issues with the VM identity or any other authentication method, like a user or service principal, when trying to perform operations other than pool creation using that identity? Additionally, you can try creating a user-assigned managed identity and performing the same operation from both the VM and the CLI.
Diko Parvanov 0 Reputation points

2025-01-30T13:27:31.8733333+00:00

I am only seeing this with the managed identity. With my managed application tenant and security object that allows the same Owner access I can successfully accomplish the action, however it is required that a script does it via the azure cli inside the VM to automate the process and that fails.

I have a user assigned identity as well, having the same permissions and I've faced the same issues with it.
Diko Parvanov 0 Reputation points

2025-01-30T14:08:00.9533333+00:00

No, all other operations are working as expected, I can create/update/delete any kind of resources in that resource group. No issues with authentication. I tried with user-assigned identity with the same permissions and I still get the same error, by doing az login --identity --username <user assigned identity object id>.
Vallepu Venkateswarlu 0 Reputation points Microsoft Vendor

2025-01-31T05:09:37.8466667+00:00

Is the application gateway in the same resource group or in a different one? Also, could you please assign owner access to the subscription? Additionally, can you confirm if this issue is only affecting a specific application gateway or if it applies to other gateways as well? Lastly, could you test it from another VM?
KapilAnanth-MSFT 48,576 Reputation points Microsoft Employee

2025-01-31T10:05:06.8866667+00:00
@Diko Parvanov ,

Greetings.

From the discussion you had with Rohith Vinnakota and Vallepu Venkateswarlu,

I tried to set up a Lab and I was able to get this working.

The error message is often associated with the resource actually not existing.

I.e., I strongly urge you to double check the spelling of the resource (App Gateway, Resource Group)

See what happens when I use a wrong RG Name

CLI assumes the resource exists, and hence it says it has no permission over it - which in reality is actually a spelling mistake

Alternatively,

Can you try using az network application-gateway address-pool update instead of az network application-gateway address-pool create

i.e., create the Backend Pool along with the App Gateway

Then update the backend Pool

Can you confirm if the issue persists?

I understand the error message you are seeing is a different one, (Subnet Join related) - so if the issue persists even after you have verified the spelling , we will need a specialized 1:1 session, where a support engineer can check the backend logs to pinpoint the issue.

If you have a support plan you may file a support ticket, else please do let us know, we will try and help you get a one-time free technical support.

Thanks,

Kapil
Diko Parvanov 0 Reputation points

2025-01-31T17:33:55.1066667+00:00

Hello @KapilAnanth-MSFT

from what I can see, first you are running within a Windows VM - I am running in a Linux one, so if it is working in the Windows VM with a managed identity, then this surely is a bug in Azure with either az cli or with how it's handling the requests from a Linux VM.

Second - are you running this with a managed identity or did you login with your user account? Because when I run it with a user account it works just fine, it only fails with either managed or user assigned identities to a VM.

I am sure the resource exists, because here you go a screenshot where I can clearly list the pools on the same application gateway and also az accout show shows the type of user, which in my case is systemAssignedIdentity - can you show me what's showing on your end?

Update fails as well:
Diko Parvanov 0 Reputation points

2025-01-31T17:35:47.35+00:00

I have a support ticket opened with Microsoft and I got this response just now:

which makes no sense to me, this really smells like an Azure bug.
KapilAnanth-MSFT 48,576 Reputation points Microsoft Employee

2025-02-03T05:37:05.3966667+00:00
@Diko Parvanov ,

Thanks for sharing the info.

Since you already have a Support incident open with Azure Support, please pass on the feedback to the Engineer.

Meanwhile, I have initiated a private message to gather more details related the SR.

P.S :

I was using Managed Identity enabled VM only - not user login.

Is it possible for you to try the same with Windows VM?

Cheers,

Kapil
Diko Parvanov 0 Reputation points

2025-02-03T07:39:48.18+00:00

Just tested with a Windows 10 VM - same problem.
Diko Parvanov 0 Reputation points

2025-02-03T11:16:46.9133333+00:00

Just verified and this works on a normal resource group, and only fails with a managed application resource group.
KapilAnanth-MSFT 48,576 Reputation points Microsoft Employee

2025-02-04T11:42:10.5233333+00:00

@Diko Parvanov ,

Thanks for sharing the info. I see Azure Managed Application with managed identity is still in preview

Since you have an active support incident open, I suggest you follow up with them as they have access to platform logs and your tenant details.

Cheers,

Kapil