Send Prometheus metrics from virtual machines, scale sets, or Kubernetes clusters to an Azure Monitor workspace

Prometheus isn't limited to monitoring Kubernetes clusters. Use Prometheus to monitor applications and services running on your servers, wherever they're running. For example, you can monitor applications running on Virtual Machines, Virtual Machine Scale Sets, or even on-premises servers. You can also send Prometheus metrics to an Azure Monitor workspace from your self-managed cluster and Prometheus server. Install prometheus on your servers and configure remote-write to send metrics to an Azure Monitor workspace.

This article explains how to configure remote-write to send data from a self-managed Prometheus instance to an Azure Monitor workspace.

Remote write options

Self-managed Prometheus can run on Azure and non-Azure environments. The following are authentication options for remote-write to Azure Monitor workspace based on the environment where Prometheus is running.

Azure-managed Virtual Machines, Virtual Machine Scale Sets, and Kubernetes clusters

Use user-assigned managed identity authentication for services running self managed Prometheus in an Azure environment. Azure-managed services include:

  • Azure Virtual Machines
  • Azure Virtual Machine Scale Sets
  • Azure Kubernetes Service (AKS)

To set up remote write for Azure-managed resources, see Remote-write using user-assigned managed identity.

Virtual machines and Kubernetes clusters running on non-Azure environments

If you have virtual machines or a Kubernetes cluster in non-Azure environments, or you have onboarded to Azure Arc, install self-managed Prometheus, and configure remote-write using Microsoft Entra ID application authentication. For more information, see Remote-write using Microsoft Entra ID application authentication.

Onboarding to Azure Arc-enabled services allows you to manage and configure non-Azure virtual machines in Azure. For more Information on onboarding Virtual Machines to Azure Arc-enabled servers, see Azure Arc-enabled servers and Azure Arc-enabled Kubernetes. Arc-enabled services only support Microsoft Entra ID authentication.

Note

System-assigned managed identity is not supported for remote-write to Azure Monitor workspaces. Use user-assigned managed identity or Microsoft Entra ID application authentication.

Prerequisites

Supported versions

  • Prometheus versions greater than v2.45 are required for managed identity authentication.
  • Prometheus versions greater than v2.48 are required for Microsoft Entra ID application authentication.

Azure Monitor workspace

This article covers sending Prometheus metrics to an Azure Monitor workspace. To create an Azure monitor workspace, see Manage an Azure Monitor workspace.

Permissions

Administrator permissions for the cluster or resource are required to complete the steps in this article.

Set up authentication for remote-write

Depending on the environment where Prometheus is running, you can configure remote-write to use user-assigned managed identity or Microsoft Entra ID application authentication to send data to Azure Monitor workspace.

Use the Azure portal or CLI to create a user-assigned managed identity or Microsoft Entra ID application.

Remote-write using user-assigned managed identity authentication

User-assigned managed identity authentication can be used in any Azure-managed environment. If your Prometheus service is running in a non-Azure environment, you can use Microsoft Entra ID application authentication.

To configure a user-assigned managed identity for remote-write to Azure Monitor workspace, complete the following steps.

Create a user-assigned managed identity

To create a user-managed identity to use in your remote-write configuration, see Manage user-assigned managed identities.

Note the value of the clientId of the managed identity that you created. This ID is used in the Prometheus remote write configuration.

Assign the Monitoring Metrics Publisher role to the application

On the workspace's data collection rule, assign the Monitoring Metrics Publisher role to the managed identity.

  1. On the Azure Monitor workspace Overview page, select the Data collection rule link.

    A screenshot showing the data collection rule link on an Azure Monitor workspace page.

  2. On the data collection rule page, select Access control (IAM).

  3. Select Add, and Add role assignment. A screenshot showing the data collection rule.

  4. Search for and select for Monitoring Metrics Publisher, and then select Next. A screenshot showing the role assignment menu for a data collection rule.

  5. Select Managed Identity.

  6. Select Select members.

  7. In the Managed Entity dropdown, User-assigned managed identity.

  8. Select the user-assigned identity that you want to use, then click Select.

  9. Select Review + assign to complete the role assignment.

    A screenshot showing the select members menu for a data collection rule.

Assign the managed identity to a Virtual Machine or Virtual Machine Scale Set

Important

To complete the steps in this section, you must have owner or user access administrator permissions for the Virtual Machine or Virtual Machine Scale Set.

  1. In the Azure portal, go to the cluster, Virtual Machine, or Virtual Machine Scale Set's page.

  2. Select Identity.

  3. Select User assigned.

  4. Select Add.

  5. Select the user assigned managed identity that you created, then select Add.

    A screenshot showing the add user assigned managed identity page.

Assign the managed identity for an Azure Kubernetes Service

For Azure Kubernetes services (AKS), the managed identity must be assigned to the virtual machine scale sets.

AKS creates a resource group containing the virtual machine scale sets. The resource group name is in the format MC_<resource group name>_<AKS cluster name>_<region>. For each Virtual Machine Scale Set in the resource group, assign the managed identity according to the steps in the previous section, Assign the managed identity to a Virtual Machine or Virtual Machine Scale Set.

Configure remote-write

Remote-write is configured in the Prometheus configuration file prometheus.yml, or in the Prometheus Operator.

For more information on configuring remote-write, see the Prometheus.io article: Configuration. For more on tuning the remote write configuration, see Remote write tuning.

To send data to your Azure Monitor Workspace, add the following section to the configuration file (prometheus.yml) of your self-managed Prometheus instance.

remote_write:   
  - url: "<metrics ingestion endpoint for your Azure Monitor workspace>"
# AzureAD configuration.
# The Azure Cloud. Options are 'AzurePublic', 'AzureChina', or 'AzureGovernment'.
    azuread:
      cloud: 'AzurePublic'
      managed_identity:
        client_id: "<client-id of the managed identity>"
      oauth:
        client_id: "<client-id from the Entra app>"
        client_secret: "<client secret from the Entra app>"
        tenant_id: "<Azure subscription tenant Id>"

The url parameter specifies the metrics ingestion endpoint of the Azure Monitor workspace. It can be found on the Overview page of your Azure Monitor workspace in the Azure portal.

A screenshot showing the metrics ingestion endpoint for an Azure Monitor workspace.

Use either the managed_identity, or oauth for Microsoft Entra ID application authentication, depending on your implementation. Remove the object that you're not using.

Find your client ID for the managed identity using the following Azure CLI command:

az identity list --resource-group <resource group name>

For more information, see az identity list.

To find your client for managed identity authentication in the portal, go to the Managed Identities page in the Azure portal and select the relevant identity name. Copy the value of the Client ID from the Identity overview page.

A screenshot showing the client ID on the Identity overview page.

To find the client ID for the Microsoft Entra ID application, use the following CLI or see the first step in the Create a Microsoft Entra ID application using the Azure portal section.

$ az ad app list --display-name < application name>

For more information, see az ad app list.

Note

After editing the configuration file, restart Prometheus for the changes to apply.

Verify that remote-write data is flowing

Use the following methods to verify that Prometheus data is being sent into your Azure Monitor workspace.

Azure Monitor metrics explorer with PromQL

To check if the metrics are flowing to the Azure Monitor workspace, from your Azure Monitor workspace in the Azure portal, select Metrics. Use the metrics explorer to query the metrics that you're expecting from the self-managed Prometheus environment. For more information, see Metrics explorer.

Prometheus explorer in Azure Monitor Workspace

Prometheus Explorer provides a convenient way to interact with Prometheus metrics within your Azure environment, making monitoring and troubleshooting more efficient. To use the Prometheus explorer, go to your Azure Monitor workspace in the Azure portal and select Prometheus Explorer to query the metrics that you're expecting from the self-managed Prometheus environment. For more information, see Prometheus explorer.

Grafana

Use PromQL queries in Grafana to verify that the results return the expected data. To configure Grafana, see getting Grafana setup with Managed Prometheus

Troubleshoot remote write

If remote data isn't appearing in your Azure Monitor workspace, see Troubleshoot remote write for common issues and solutions.

Next steps