Automating Daily Metric Collection and Storage in Private Azure Environment

Yu-Jeong Seo 130

I want to extract the metric values of VMs configured in the Azure environment and automatically save them as CSV files to a Storage Account on a daily basis. While my PowerShell script is working, I am facing difficulties in saving the data to the Storage Account due to the specific constraints of the Azure environment.

To access the Azure Portal, I must go through the proxy of the VM already configured in Azure.
Both the Storage Account and VMs have very restrictive inbound allow rules (Storage Account + Private Link, VM + NSG).
Additionally, there are restrictions in place due to VWAN and Azure Firewall.

Q1: Given these circumstances, is it possible to use Azure Automation to perform daily checks and save the CSV files to the Storage Account? While using a Hybrid Worker to save to Storage is an option, will it work in this constrained environment? I am also unsure about how Azure Automation communicates with the VMs (e.g., via internet traffic, Microsoft network backbone, etc.).

Q2: If Azure Automation is not an option, are there alternative methods to check the resource usage daily? Ultimately, the CPU, memory, and disk usage need to be saved as a CSV file in the Storage Account. (In this case, the storage has restricted access through Private Link.)

I'm sorry for the lengthy questions as my English is not very strong. I really appreciate your help. Thank you!

Sumarigo-MSFT 47,106 Reputation points Microsoft Employee

2024-11-19T06:56:49.81+00:00
@Yu-Jeong Seo Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

Adding more information to the below answer

Q1: Using Azure Automation to Perform Daily Checks and Save CSV Files : Yes

Azure Automation can indeed be used to perform daily checks and save CSV files to a Storage Account. However, considering the restrictive inbound rules and the need to go through a proxy, there are a few things to keep in mind:

Hybrid Worker: Using a Hybrid Worker is a viable option. Hybrid Workers run on-premises or in your Azure VMs and can communicate with your Azure resources without needing to traverse the internet. This setup should work within your constrained environment, as the Hybrid Worker can operate within the same network boundaries as your VMs and Storage Account.

Communication: Azure Automation communicates with VMs and other resources primarily through the Microsoft network backbone. This means that as long as your Hybrid Worker can reach the necessary endpoints within your network, it should be able to perform the required tasks.

Network Configuration: Ensure that your Network Security Groups (NSGs), Virtual WAN (VWAN), and Azure Firewall rules allow the necessary traffic between your Hybrid Worker and the Storage Account. You may need to configure specific rules to permit this communication.

Azure Automation communicates with the VMs using the Azure Hybrid Runbook Worker. The Hybrid Runbook Worker is a lightweight agent that runs on a Windows or Linux computer in your environment. The Hybrid Runbook Worker communicates with Azure Automation over HTTPS, so you will need to ensure that HTTPS traffic is allowed through your firewall.

Q2: Alternative Methods to Check Resource Usage Daily

If Azure Automation is not a suitable option, there are alternative methods to achieve your goal:

Azure Monitor and Log Analytics: You can use Azure Monitor to collect metrics from your VMs and store them in a Log Analytics workspace. From there, you can create scheduled queries to extract the required metrics and save them as CSV files. This approach leverages Azure's built-in monitoring capabilities and can be configured to work within your network constraints.

Azure Functions: Another option is to use Azure Functions to run a script that collects the metrics and saves them to the Storage Account. Azure Functions can be configured to run on a schedule and can operate within the same network boundaries as your VMs and Storage Account.

Custom Script: You can continue using your PowerShell script but modify it to run on a VM within the same network as your Storage Account. This VM can act as a dedicated monitoring server, collecting metrics and saving them to the Storage Account without needing to traverse restrictive network boundaries.

If Azure Automation is not an option, you can use Azure Monitor to check the resource usage daily. Azure Monitor provides a set of pre-built metrics for Azure VMs, including CPU usage, memory usage, and disk usage. You can configure Azure Monitor to collect these metrics on a daily basis and save them to a Log Analytics workspace. From there, you can export the data to a CSV file and save it to your Storage Account.

In summary, using a Hybrid Worker with Azure Automation is a feasible solution, provided you configure your network rules appropriately. Alternatively, leveraging Azure Monitor, Azure Functions, or a custom script running on a dedicated VM can also achieve the desired outcome

Please let us know if you have any further queries. I’m happy to assist you further.

Accepted answer

Ashok Gandhi Kotnana 860 Reputation points Microsoft Vendor

2024-11-19T07:09:21.4166667+00:00

HI @Yu-Jeong Seo

Welcome to Microsoft Q&A Forum, thank you for posting your query here! I have understood your required Please find the below two solution that might help you.

A1) Yes it will work using automation account you need to follow the below steps

1.Create an automation account in the Networking tab of the automation account disable public network access and enable private endpoint
Note: The private endpoint should be created on the same network where your storage account resides so that your automation account will speak to the storage account within your network Please check the configuration below.
Disable the Public IP and enable the private endpoint While creating the private endpoint please make sure you select same network where the storage account is created so that traffic will flow within your internal network
A2) Create a new virtual machine within the same network as your storage account. Enable the System Assigned Managed Identity on the VM and assign it the appropriate permissions to access the storage account. Specifically, grant the identity the "Storage Blob Data Contributor" role. Follow these steps:

Select the identity which you have enabled for the virtual machine to communicate with the storage account. Please select the resource instance below

Please let us know if any help, we are always here to help whenever you need us.

Please do not forget to "Accept the answer” wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.
Ashok Gandhi Kotnana 860 Reputation points Microsoft Vendor

2024-11-20T10:33:14.3566667+00:00

Hi @Yu-Jeong Seo ,

Just want to check if the above answer or comment worked for you or else let us know if any help, we will always help as you needed.!

Please do not forget to "Accept the answer” wherever the information provided helps you, this can be beneficial to other community members.

Ashok Gandhi Kotnana 860 Reputation points Microsoft Vendor

2024-11-21T10:36:08.33+00:00

Hi @Yu-Jeong Seo ,

We haven't heard back from you. Please reply if you have any questions in this matter and we will gladly continue the discussion.

Please do not forget to "Accept the answer” wherever the information provided helps you, this can be beneficial to other community members.

Yu-Jeong Seo 130 Reputation points

2024-11-21T11:46:47.84+00:00

@Ashok Gandhi Kotnana
Thank you! I apologize for the delay in my response as I was testing the suggested method. The results from my environment were successful. I hope it will also work successfully in environments that are slightly different from mine.

Ashok Gandhi Kotnana 860 Reputation points Microsoft Vendor

2024-11-21T11:54:10.2466667+00:00

HI @Yu-Jeong Seo ,

Thank you so much for your response, thank you for testing it, great to hear it worked well for you
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.