Azure Confidential Ledger ARM template re-deployment infinite loop calls to Ledger_ListBySubscription

Question

Azure Confidential Ledger ARM template re-deployment infinite loop calls to Ledger_ListBySubscription

Dev S 71

When using an ARM template to deploy an Azure Confidential ledger, resource of type Microsoft.ConfidentialLedger/ledgers, we see that the initial deployment of the Ledger runs fine, however, upon rerunning the ARM template, the deployment gets stuck with multiple calls to Ledger_ListBySubscription recorded in the Activity Log. There seems to be no end to the deployment and calls to Ledger_ListBySubscription keep getting logged every minute. The deployment gets stuck even when re-running the ARM template with 0 changes to the initial deployment.

Vinod Pittala 735 Reputation points Microsoft External Staff

2025-01-29T23:15:30.6533333+00:00
Hi Dev S

Welcome to Microsoft Q&A Forum, thank you for posting your query here!

The issue you're encountering, where the deployment of Azure Confidential Ledger gets stuck and repeatedly logs calls to Ledger_ListBySubscription in the Activity Log, could be due to several factors.

Below are possible causes in detailed.

1st cause: When the first deployment of the Azure Confidential Ledger is successful, it creates the required resources. If there is any inconsistency in the state or configuration of the Ledger after the initial deployment (such as a partially completed resource), the ARM template may be repeatedly trying to validate or query the Ledger's state (which is what the Ledger_ListBySubscription API call represents).

Solution: Check the current state of the Ledger resource in the portal. Verify if it is healthy and fully deployed. If you find any inconsistencies or partial deployments, you may need to manually resolve or clean up the resources. It might also help to manually delete the existing resource and try redeploying the ARM template from scratch.

You can also check the activity logs or deployment history that may give more context about the ongoing job.

2nd cause: ARM templates are designed to be idempotent, meaning they should only change the resources if there are changes in the template. However, if the template has issues like incorrect conditions, parameters, or missing dependencies, it might be repeatedly trying to apply the same changes without detecting the existing resource.
Solution: Review your ARM template for potential issues like conditions or parameters that might cause it to continually attempt to apply changes. Pay attention to resource definitions that may be incorrectly marked for recreation or updating.

Below is the reference template that can provide you with further understanding.

https://learn.microsoft.com/en-us/azure/confidential-ledger/quickstart-template?tabs=azure-cli%2CCLI#review-the-template

3rd cause: The Ledger_ListBySubscription API calls are part of the Azure Confidential Ledger's resource provider. If the resource provider is not correctly registered or if permissions have changed, the template might repeatedly attempt to validate or list existing Ledger resources without success.
Solution: Ensure that the resource provider Microsoft.ConfidentialLedger is properly registered in your subscription.

Refer to the below document for your reference.

Create an Microsoft Azure confidential ledger by using Azure Resource Manager template | Microsoft Learn

You can check the status by running the following Azure CLI command:

az provider show --namespace "microsoft.ConfidentialLedger"

If it's not registered, register it with the below Azure CLI command:

az provider register --namespace "microsoft.ConfidentialLedger"

4th cause: Continuous calls to the Ledger_ListBySubscription API could indicate API throttling or rate limiting issues. If the deployment is hitting a rate limit or there is a high volume of requests being sent to the API, the deployment might get stuck in a loop.

Solution: Check for throttling errors or limits in the activity log. If necessary, you could add delay mechanisms or reduce the frequency of retries in the deployment script to prevent hitting the rate limits.

5th cause: If a previous deployment left behind a lock or an active resource that is preventing subsequent changes, the deployment could hang while trying to access the Ledger.

Solution: Check if there are any locks applied to the resource group or resource that could be preventing updates. Remove any unnecessary locks and try the deployment again.

By examining these alternatives, you should be able to identify the root cause of the issue and resolve the recurring deployment problem.

If the provided answer is helpful for you, please “upvote it”, this can be beneficial to other community members.it would be greatly appreciated and helpful to others.

Thanks
Dev S 71 Reputation points

2025-01-30T17:46:58.4966667+00:00

Thanks @Vinod Pittala for the response.

What I've noticed is, when executing the template the first time, as you mentioned, the Resource gets created since it doesn't exist. During this run, the DevOps Principal which executes the pipeline with the Ledger ARM template gets added automatically as an Administrator. Note that we also pass an additional array list of OIDs which should be Administrators on the Ledger.

During the second run, reviewing the Activity Log Change History, I see that the continuous loop of Ledger_ListBySubscription is stuck trying to change the list of Admins. In the Change History I see that during the second run, the SP ID is not present but the array of OIDs we pass exists. So, for some reason, it isn't able to remove the creator of the Ledger (DevOps SP) as an Admin and it gets stuck.

Not sure if this is a problem with the Azure Backend or if we are expected to pass the DevOps SP all the time as an Administrator in the ARM template.
Dev S 71 Reputation points

2025-01-30T21:00:33.89+00:00

I have done further tests and I am able to reproduce the behavior every time we try to add/remove Admins/Contributors on the Ledger using the ARM template. This was working a few weeks back and it seems something has changed in the Azure backend.

Could you please test with the sample template at https://learn.microsoft.com/en-us/azure/confidential-ledger/quickstart-template?tabs=azure-cli%2CCLI#register-the-resource-provider%22https://learn.microsoft.com/en-us/azure/confidential-ledger/quickstart-template?tabs=azure-cli%2CCLI#register-the-resource-provider%22 and confirm that you are able to add/delete admins by subsequent template runs on the same resource?
Madugula Jahnavi 0 Reputation points Microsoft External Staff

2025-01-31T05:36:16.0566667+00:00

Hello Dev S, I understood that you have tried all the approaches given and I see you mentioned a template in the comment, I can try deploying that and let you know.
Madugula Jahnavi 0 Reputation points Microsoft External Staff

2025-02-03T13:13:29.7533333+00:00

@Dev S, are you still looking for an approach!
Dev S 71 Reputation points

2025-02-03T17:34:49.76+00:00

Yes. This still does not work correctly. Changing the AAD principals list through the ARM template fails.
Madugula Jahnavi 0 Reputation points Microsoft External Staff

2025-02-05T07:13:15.5166667+00:00

Hello Dev S, I am able to deploy the ledger ARM template successfully without any conflict. I can share my code here for better understanding.

1 answer

Your answer

Dev S 71 Reputation points

2025-01-30T17:46:58.4966667+00:00

Thanks @Vinod Pittala for the response.

What I've noticed is, when executing the template the first time, as you mentioned, the Resource gets created since it doesn't exist. During this run, the DevOps Principal which executes the pipeline with the Ledger ARM template gets added automatically as an Administrator. Note that we also pass an additional array list of OIDs which should be Administrators on the Ledger.

During the second run, reviewing the Activity Log Change History, I see that the continuous loop of Ledger_ListBySubscription is stuck trying to change the list of Admins. In the Change History I see that during the second run, the SP ID is not present but the array of OIDs we pass exists. So, for some reason, it isn't able to remove the creator of the Ledger (DevOps SP) as an Admin and it gets stuck.

Not sure if this is a problem with the Azure Backend or if we are expected to pass the DevOps SP all the time as an Administrator in the ARM template.
Dev S 71 Reputation points

2025-01-30T21:00:33.89+00:00

I have done further tests and I am able to reproduce the behavior every time we try to add/remove Admins/Contributors on the Ledger using the ARM template. This was working a few weeks back and it seems something has changed in the Azure backend.

Could you please test with the sample template at https://learn.microsoft.com/en-us/azure/confidential-ledger/quickstart-template?tabs=azure-cli%2CCLI#register-the-resource-provider%22https://learn.microsoft.com/en-us/azure/confidential-ledger/quickstart-template?tabs=azure-cli%2CCLI#register-the-resource-provider%22 and confirm that you are able to add/delete admins by subsequent template runs on the same resource?
Madugula Jahnavi 0 Reputation points Microsoft External Staff

2025-01-31T05:36:16.0566667+00:00

Hello Dev S, I understood that you have tried all the approaches given and I see you mentioned a template in the comment, I can try deploying that and let you know.
Madugula Jahnavi 0 Reputation points Microsoft External Staff

2025-02-03T13:13:29.7533333+00:00

@Dev S, are you still looking for an approach!
Dev S 71 Reputation points

2025-02-03T17:34:49.76+00:00

Yes. This still does not work correctly. Changing the AAD principals list through the ARM template fails.
Madugula Jahnavi 0 Reputation points Microsoft External Staff

2025-02-05T07:13:15.5166667+00:00

Hello Dev S, I am able to deploy the ledger ARM template successfully without any conflict. I can share my code here for better understanding.

Answer 1

Madugula Jahnavi 0 Microsoft External Staff

Hello Dev S,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

Here is the complete ARM template I've tried to deploy confidential ledgers with "aad security principal's" ledger role as administrator and was able to perform the operation successfully.

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "ledger": {
            "type": "string"
        },
        "principalId": {
            "type": "string"
        },
        "location": {
            "type": "string",
            "defaultValue": "[resourceGroup().location]"
        }
    },
    "functions": [],
    "variables": {},
    "resources": [{
        "name": "[parameters('ledger')]",
        "type": "Microsoft.ConfidentialLedger/ledgers",
        "apiVersion": "2020-12-01-preview",
        "location": "[parameters('location')]",
        "properties": {
            "ledgerType": "Public",
            "aadBasedSecurityPrincipals": [{
                "principalId": "[parameters('principalId')]",
                "ledgerRoleName": "Administrator"
            }]
        }
    }]
}

Portal view ledger >> properties:

aad

And mainly if you are looking to update the AAD security principals exclusively through an ARM template, I found it is not exactly possible with ARM after exploring on it. To update them, you can use "az confidentialledger update --aad-based-security-principals" Az CLI command along with the ledger name and relevant arguments.

Reference MS Doc: https://learn.microsoft.com/en-us/cli/azure/confidentialledger?view=azure-cli-latest#az-confidentialledger-update

Hope this helps.

If the answer is helpful, please click Accept Answer and kindly upvote it. If you have any further questions about this answer, please click Comment.

Madugula Jahnavi 0 Reputation points Microsoft External Staff

2025-02-06T03:59:58.1366667+00:00

Hello @Dev S, have you looked into above approach if it works in your environment?
Dev S 71 Reputation points

2025-02-06T16:00:30.63+00:00
Thanks for testing this out.

And mainly if you are looking to update the AAD security principals exclusively through an ARM template, I found it is not exactly possible with ARM after exploring on it.

This above is exactly when the ARM template goes into an infinite loop. I can confirm that it was possible to update AAD security principals with just the ARM template a few weeks back, even mid January. Running the same ARM template you tested a second time with an additional "AAD based security principals" item should trigger the infinite loop behavior.

This has an additional consequence too for the following procedure:

When using a Azure DevOps pipeline to deploy a Ledger using an ARM template, the Service Principal used to execute the pipeline gets added automatically as an Administrator when the Ledger is created.

Re-executing the same pipeline triggers the infinite loop behavior. The second run does not create the Ledger but instead updates it since the Ledger already exists. During this operation by the ARM template, the Service Principal used to create the Ledger is not in the "AAD Security Principals" Array and hence there is a difference in what exists and what needs to get added, leaving the ARM template in a loop state.

Furthermore, imagine a Ledger is created using the DevOps pipeline and then an Admin uses Azure Portal to add an additional Admin/Contributor to the Ledger. Rerunning the ARM template will get stuck because now the ARM template tries to update the Ledger AAD Security Principals.

I have a feeling that something has changed in the Ledger Management API a few weeks back and this behavior makes it impossible to use ARM templates for Ledger deployment through pipelines.
Madugula Jahnavi 0 Reputation points Microsoft External Staff

2025-02-06T16:03:52.09+00:00

Dev S, Yes, it is not possible with ARM. I have provided you an alternative workaround with CLI. Isn't it fits in your environment?
Dev S 71 Reputation points

2025-02-06T16:09:03.32+00:00

The workaround doesn't fit with our environment because we deploy a lot of resources along with the Ledger using a DevOps pipeline and these pipelines are executed multiple times. At the moment, the pipelines get stuck because Confidential Ledger stage gets stuck on every run after the first run.

If you could point me to official documentation stating we can't use ARM templates anymore to update AAD Security Principals on Azure Confidential Ledger or that ARM templates in DevOps pipelines cannot be used for a Confidential Ledger, I can bring it up with our team to investigate alternative methods for us.

Thanks.
Madugula Jahnavi 0 Reputation points Microsoft External Staff

2025-02-06T16:18:22.7566667+00:00

Dev S, Let me try to find out a reference regarding it. And do you prefer bicep to work in this scenario.
Dev S 71 Reputation points

2025-02-06T16:18:47.2933333+00:00

Not sure if my previous comment got through. Rewriting it:

The provided workaround is not a good fit for us because, in our current environment we use DevOps pipelines to deploy and update a lot of resources along with the Confidential Ledger. Those pipelines now get stuck at the Confidential Ledger stage on every run except the first run because the ARM template triggers an AAD Security Principal update on the Ledger due to what I mentioned in my previous comment.

If you could point me to official documentation which states that ARM templates cannot be used to update Confidential Ledgers or are not supported inside an Azure DevOps pipeline, I can bring that to the team and we could internally investigate how to change our current process to overcome this limitation.

Thanks.
Dev S 71 Reputation points

2025-02-06T16:20:01.3766667+00:00

Let me try to find out a reference regarding it. And do you prefer bicep to work in this scenario.

We currently just use ARM templates, but I suppose Bicep is also eventually converted to ARM templates, so if it works in Bicep, it should work in ARM templates too.

Thanks again.
Madugula Jahnavi 0 Reputation points Microsoft External Staff

2025-02-06T16:21:16.51+00:00

Dev S, Ok will try to find out a document and provide it here.
SadiqhAhmed-MSFT 48,456 Reputation points Microsoft Employee

2025-02-24T16:29:19.3133333+00:00

@Dev S Sorry for the delayed response. Regarding your ask about documentation for your requirements; not all the scenarios are covered in the documentation. Usually, the documents are developed as source of truth for reference. For specific requirements, you need to work through the same to accomplish your scenario.

References:
https://blog.azinsider.net/deploy-azure-confidential-ledger-using-bicep-language-d88b854efea1

https://learn.microsoft.com/en-us/cli/azure/bicep?view=azure-cli-latest

https://learn.microsoft.com/en-us/graph/templates/quickstart-create-bicep-zero-touch-mode?tabs=CLI

Hope this helps!

Share via

Azure Confidential Ledger ARM template re-deployment infinite loop calls to Ledger_ListBySubscription

1 answer

Your answer