How to deploy an Azure Data Explorer Cluster resource with the ALZ policies activated

REZAI Arash 0

Hi,
I've detected some limitations with a built-in policy definition that is part of the initiative "Deny-PublicPaaSEndpoints" in ALZ and that is preventing people from creating something, which is supposed to be accepted, through the GUI.

This is the policy:

Public network access on Azure Data Explorer should be disabled - 43bc7be6-5e69-4b0d-a2bb-e815557ca673

While it works fine to set "/clusters" to "Disabled" through code, it seems the Portal doesn't explicitly do that by default when choosing the Connectivity method "Private endpoint".

I would like to sort this behavior in the Portal such that colleagues would be able to deploy an Azure Data Explorer Cluster without having to do that through code... but how?

REZAI Arash 0 Reputation points

2024-10-31T14:16:25.1766667+00:00

Hi @Pavan Minukuri ,

Thanks for your reply. However, your reply is wrong on more than one level and I will clarify why.

First of all, built-in policy definitions cannot be edited (refer to https://learn.microsoft.com/en-us/answers/questions/245260/how-can-i-edit-the-built-in-azure-policy-definitio , despite old post, still accurate).

You might want to duplicate it and therefore have a copy of the built-in policy as a custom policy. Yes, then you can edit it... which leads to the second faulty statement. There is no such thing as adding an exception in the edit view of a custom policy. I also followed both the links you provided and didn't find anything backing that up.

BUT the moral of this issue isn't something that should be solved through exemptions (yes, I said exemptions, not exceptions). I managed to temporarily solve the issue with an exemption while I am hopefully waiting for Microsoft to correct their built-in policy, or even better, to have the GUI set that particular value explicitly to Disabled whenever you choose to go for the Private endpoint option.

Although selecting the private endpoint connectivity method, once the Azure Data Explorer Cluster resource has been deployed, navigating to Networking provides the following:

Hence, solution is pretty much known but up to Microsoft to solve. I wonder where you should submit such requests if not through this forum/community. Any ideas?
Pavan Minukuri 520 Reputation points Microsoft Vendor

2024-11-01T04:38:30.8833333+00:00

Hi REZAI Arash,.
Thank you for the detailed feedback and providing the correct information. I apologize for any confusion caused by my previous response.

Regarding your question about submitting requests to Microsoft for policy changes, you can submit feedback and feature requests through the Azure Feedback Forum. This forum allows you to submit feedback, vote on existing feedback, and engage with other users and Microsoft product teams. Here is the link to the Azure Feedback Forum:

https://feedback.azure.com/forums/34192--general-feedback

You can also submit support requests to Microsoft through the Azure portal or by contacting Azure support directly. They can help you troubleshoot issues and provide guidance on how to resolve them.

If you have any further queries, do let us know. If the comment is helpful, please click "Accept comment".
REZAI Arash 0 Reputation points

2024-11-01T14:26:16.82+00:00

Hi again @Pavan Minukuri ,
Okay, so in essence, I have posted this issue in the wrong forum you mean? I doubt it is a "General feedback" but rather "Reporting an issue", but I'll take it as I created the same post following your suggestion and posting it here. Hopefully, someone will pick it up and resolve the issue on MS side soon. In case you know anybody that can take a quick look at it at MS side, please bring them in and I am happy to share any more information with whomever is authorized to solve the issue. Thanks!
Pranay Reddy Madireddy 700 Reputation points Microsoft Vendor

2024-11-01T18:33:09.77+00:00

Hi REZAI Arash

Thank You for Replying back.

I hope you got the things from my side we will support online Q&A only. Regarding the submit forum support request u can reach over there only we don't have contact I will check with my internal team. If in case, I get any update from my internal team will update you
Pavan Minukuri 520 Reputation points Microsoft Vendor

2024-11-04T02:30:14.85+00:00

Hi REZAI Arash
Just want to check is that issue resolved or else if you need any help, please let us know we will always help as you needed.! if you had chance please review once my previous comment.
Thank you.!
REZAI Arash 0 Reputation points

2024-11-08T13:11:24.8033333+00:00

Hi @Pranay Reddy Madireddy and @Pavan Minukuri ,
Unfortunately, the issue still very much exists and is not solved. Nobody is replying in the other forum so I guess it might be valuable to have this question posted here as well in order for someone from MS to maybe pick it up at some point. For now, thanks for your suggestions and unless you can get ahold of someone working with these templates behind the Portal, I have created a workaround such that I exempt whatever subscriptions I want to be able to create a Data Explorer cluster resource from that policy. I believe the policy is correct, however, someone needs to fix the template that is being deployed through the Portal whenever this resource is created.
Pavan Minukuri 520 Reputation points Microsoft Vendor

2024-11-11T18:31:44.8866667+00:00
Hi REZAI Arash
You have successfully created a workaround to exempt certain subscriptions from a policy that restricts the creation of Azure Data Explorer clusters. This suggests that your policy is correctly set up to allow exceptions, which is a crucial step in managing resource access and compliance within Azure.

Template Deployment Issues

The problem you are encountering seems to stem from the ARM template used for deploying the Azure Data Explorer cluster. Here are some key points to consider regarding the ARM template:

Template Structure: Ensure that your ARM template is correctly structured. A typical ARM template for an Azure Data Explorer cluster includes defining resources, specifying properties like SKU, location, and other necessary configurations. For example:

json

{

"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",

"contentVersion": "1.0.0.0",

"resources": [

{ "type": "Microsoft.Kusto/clusters", "apiVersion": "2021-04-01-preview", "name": "[variables('adxClusterName')]", "location": "[variables('location')]", "sku": { "name": "Standard_L16s", "tier": "Standard" }, "properties": { "enableDiskEncryption": false, "dataRetentionTime": "365.00:00:00", "trustedExternalTenants": [] } }

],

"variables": {

"adxClusterName": "adx-cluster", "location": "[resourceGroup().location]"

}

}

Deployment Process: After creating or modifying the ARM template, deploy it through the Azure Portal by navigating to your resource group and selecting “Deploy a custom template.” Make sure to review all parameters before finalizing the deployment.

Verifying Deployment: Once deployed, verify that the cluster is created successfully by checking its status in the Azure Portal or using Azure CLI commands like:

bash

az kusto cluster show --name adx-cluster --resource-group MyResourceGroup

Error Handling: If errors occur during deployment, check for issues related to permissions or misconfigured settings in your ARM template. The Azure portal provides logs that can help identify what went wrong during deployment.

If you have any further queries, do let us know.
Pavan Minukuri 520 Reputation points Microsoft Vendor

2024-11-12T17:57:54.7666667+00:00

Hi REZAI Arash
We haven't heard back from you. Please reply if you have any questions in this matter and we will gladly continue the discussion. Thank you
Pavan Minukuri 520 Reputation points Microsoft Vendor

2024-11-13T16:29:46.53+00:00

Hi REZAI Arash
Just want to check is that issue resolved or else if you need any help, please let us know we will always help as you needed.! if you had chance please review once my previous comment. Thank you.!
REZAI Arash 0 Reputation points

2024-11-14T08:45:31.6466667+00:00

Hello @Pavan Minukuri ,
My issue won't be resolved unless someone changes the way a Data Explorer Cluster is deployed via portal API to align with how it is supposed to be created to not end up with policy issues (i.e. to align with how you would create the same resource through code).
If you are unable to help me with that, I guess you could withdraw my request, but no, the issue hasn't been resolved and won't be either with the proposed solutions. Or well, I can create exemptions successfully but that is just a poor workaround to the original problem, which has to do with the ARM template that the portal uses for creating the resource in the first place.
Pavan Minukuri 520 Reputation points Microsoft Vendor

2024-11-15T19:17:41.5233333+00:00
Hi REZAI Arash
To address the issue you're facing with deploying an Azure Data Explorer (ADX) cluster via the Azure portal API and the associated policy problems, here are some key points and steps to consider:

Understanding the Issue

Policy Restrictions: It appears that your deployment is being hindered by Azure policies, such as the one that requires "Public network access on Azure Data Explorer should be disabled." This policy can restrict how resources are created, especially if they do not align with the expected configurations when deployed through code versus the portal.

ARM Template Discrepancies: The way the Azure portal generates ARM templates for resource creation may differ from how you would define them in your own templates. This discrepancy can lead to compliance issues with existing policies.

Steps to Resolve

Review and Update Policy Definitions

Navigate to the Azure Policy section in the Azure portal.

Locate the relevant policy definition, such as "Public network access on Azure Data Explorer should be disabled."

Consider adding exceptions for specific connectivity methods (e.g., allowing "Private endpoint") to enable successful deployments without violating policies1.

Use ARM Templates for Deployment

Instead of relying on the portal, utilize ARM templates that align with your organization’s policies. This ensures that all necessary configurations are included.

You can reference existing templates or create a new one that explicitly defines properties like restrictOutboundNetworkAccess and any required networking configurations34.

Register Resource Providers

Ensure that the Kusto resource provider is registered in your subscription. If it isn't, you may encounter errors during deployment2.

Create Exemptions as a Temporary Measure

While creating exemptions may not be ideal, it can serve as a temporary workaround while you work on aligning your deployment methods with policy requirements.

Validate Network Configurations

If deploying into a virtual network, ensure that Network Security Groups (NSGs) and route tables are correctly configured to allow traffic as needed. Misconfigurations here can lead to further issues during cluster operations5.

Monitor and Troubleshoot

After making adjustments, monitor the deployment process closely using Azure Monitor or Application Insights to capture any latency or operational issues that arise during cluster creation and usage7.

Conclusion

Addressing these deployment issues will require a combination of policy management, proper use of ARM templates, and ensuring all necessary configurations are in place. If these steps do not resolve your issue, consider reaching out to Azure support for more tailored assistance or further investigation into specific error messages encountered during deployment.

Please Let me know if anything requried.
Pavan Minukuri 520 Reputation points Microsoft Vendor

2024-11-18T16:53:58.19+00:00

Hi REZAI Arash
We haven't heard back from you. Please reply if you have any questions in this matter and we will gladly continue the discussion. Thank You...!
Pavan Minukuri 520 Reputation points Microsoft Vendor

2024-11-20T17:08:32.1466667+00:00

Hi REZAI Arash Just want to check is that issue resolved or else if you need any help, please let us know we will always help as you needed.! if you had chance please review once my previous comment. Thank you...!
REZAI Arash 0 Reputation points

2024-11-21T20:16:14.88+00:00

Hi @Pavan Minukuri ,
Indeed, while none of the above steps would permanently fix the problem, adding an exemption would indeed create a workaround to the problem that is within "The way the Azure portal generates ARM templates for resource creation", which is in fact wrong. The Data Explorer team at MS has finally got alerted and are looking to fix it on their end, through a different community post that I opened for this very same thing.
Like I mentioned earlier, the only way to fix this is not by me doing anything, but by MS fixing how the resource is being created when you click and proceed in the Portal.

1 answer

Pavan Minukuri 520 Reputation points Microsoft Vendor

2024-10-30T17:43:13.6466667+00:00
Hi REZAI Arash,
Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

It seems like you are having trouble with a built-in policy definition that is preventing you from creating something through the GUI. Specifically, the policy "Public network access on Azure Data Explorer should be disabled" is not allowing you to create an Azure Data Explorer Cluster through the Portal when using the Connectivity method "Private endpoint".

To resolve this issue, you can try updating the policy definition to include the "Private endpoint" connectivity method as an exception. Here are the steps to do that:

Go to the Azure Policy page in the Azure Portal.

Select the policy definition "Public network access on Azure Data Explorer should be disabled".

Click on the "Edit" button at the top of the page.

Scroll down to the "Exceptions" section and click on the "Add exception" button.

In the "Add exception" dialog, select "Connectivity method" as the exception type and "Private endpoint" as the connectivity method.

Click on the "Add" button to save the exception.

Click on the "Review + create" button at the top of the page to save the updated policy definition.

With this exception added to the policy definition, your colleagues should be able to deploy an Azure Data Explorer Cluster through the Portal using the "Private endpoint" connectivity method without having to do that through code.

For better understand please refer this link: https://learn.microsoft.com/en-us/azure/governance/policy/tutorials/create-and-manage https://learn.microsoft.com/en-us/azure/governance/policy/assign-policy-portal

If you have any further queries, do let us know, If the answer is helpful, please click "Accept Answer".
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to deploy an Azure Data Explorer Cluster resource with the ALZ policies activated

1 answer

Your answer