Connect Grafana to Azure Monitor Prometheus metrics
Artikel
The most common way to analyze and present Prometheus data is with a Grafana dashboard. You can collect Prometheus metrics in Azure in the following ways:
This article explains how to configure Azure-hosted Prometheus metrics as a data source for Azure Managed Grafana, self-hosted Grafana running on an Azure virtual machine, or a Grafana instance running outside of Azure.
Azure Monitor workspace query endpoint
In Azure, Prometheus data is stored in an Azure Monitor workspace. When configuring the Prometheus data source in Grafana, you use the Query endpoint for your Azure Monitor workspace. To find the query endpoint, open the Overview page for your Azure Monitor workspace in the Azure portal.
When you create an Azure Managed Grafana instance, it's automatically configured with a managed system identity. The identity has the Monitoring Data Reader role assigned to it at the subscription level. This role allows the identity to read data any monitoring data for the subscription. This identity is used to authenticate Grafana to Azure Monitor. You don't need to do anything to configure the identity.
Create the Prometheus data source in Grafana.
To configure Prometheus as a data source, follow these steps:
Open your Azure Managed Grafana workspace in the Azure portal.
Select on the Endpoint to view the Grafana workspace.
Select Connections and then Data sources.
Select Add data source
Search for and select Prometheus.
Paste the query endpoint from your Azure Monitor workspace into the Prometheus server URL field.
Under Authentication, select Azure Auth.
Under Azure Authentication, select Managed Identity from the Authentication dropdown.
Scroll to the bottom of the page and select Save & test.
Self-managed Grafana
The following section describes how to configure self-managed Grafana on an Azure virtual machine to use Azure-hosted Prometheus data.
Configure system identity
Use the following steps to allow access all Azure Monitor workspaces in a resource group or subscription:
Open the Identity page for your virtual machine in the Azure portal.
Set the Status to On.
Select Save.
Select Azure role assignments to review the existing access in your subscription.
If the Monitoring Data Reader role isn't listed for your subscription or resource group, select + Add role assignment
In the Scope dropdown, select either Subscription or Resource group. Selecting Subscription allows access to all Azure Monitor workspaces in the subscription. Selecting Resource group allows access only to Azure Monitor workspaces in the selected resource group.
Select the specific subscription or resource group where your Azure Monitor workspace is located.
From the Role dropdown, select Monitoring Data Reader.
Select Save.
Configure Grafana for Azure Authentication
Versions 9.x and greater of Grafana support Azure Authentication, but it's not enabled by default. To enable Azure Authentication, update your Grafana configuration and restart the Grafana instance. To find your grafana.ini file, review the Configure Grafana document from Grafana Labs.
Enable Azure Authentication using the following steps:
Locate and open the grafana.ini file on your virtual machine.
Under the [auth] section of the configuration file, change the azure_auth_enabled setting to true.
Under the [azure] section of the configuration file, change the managed_identity_enabled setting to true
Restart the Grafana instance.
Create the Prometheus data source in Grafana
Configure Prometheus as a data source using the following steps:
Open Grafana in your browser.
Select Connections and then Data sources.
Select Add data source
Search for and select Prometheus.
Paste the query endpoint from your Azure Monitor workspace into the Prometheus server URL field.
Under Authentication, select Azure Auth.
Under Azure Authentication, select Managed Identity from the Authentication dropdown.
Scroll to the bottom of the page and select Save & test.
Grafana hosted outside of Azure
If your Grafana instance isn't hosted in Azure, you can connect to your Prometheus data in your Azure Monitor workspace using Microsoft Entra ID.
Set up Microsoft Entra ID authentication using the following steps:
Register an app with Microsoft Entra ID.
Grant access for the app to your Azure Monitor workspace.
Configure your self-hosted Grafana with the app's credentials.
Register an app with Microsoft Entra ID
To register an app, open the Active Directory Overview page in the Azure portal.
Select App registration.
On the Register an application page, enter a Name for the application.
Select Register.
Note the Application (client) ID and Directory(Tenant) ID. They're used in the Grafana authentication settings.
On the app's overview page, select Certificates and Secrets.
In the client secrets tab, select New client secret.
Enter a Description.
Select an expiry period from the dropdown and select Add.
Note
Create a process to renew the secret and update your Grafana data source settings before the secret expires.
Once the secret expires Grafana loses the ability to query data from your Azure Monitor workspace.
Copy and save the client secret Value.
Note
Client secret values can only be viewed immediately after creation. Be sure to save the secret before leaving the page.
Allow your app access to your workspace
Allow your app to query data from your Azure Monitor workspace.
Open your Azure Monitor workspace in the Azure portal.
On the Overview page, take note of your Query endpoint. The query endpoint is used when setting up your Grafana data source.
Select Access control (IAM).
Select Add, then Add role assignment from the Access Control (IAM) page.
On the Add role Assignment page, search for Monitoring.
Select Monitoring data reader, then select the Members tab.
You've created your App registration and have assigned it access to query data from your Azure Monitor workspace. The next step is setting up your Prometheus data source in Grafana.
Configure Grafana for Azure Authentication
Grafana now supports connecting to Azure Monitor managed Prometheus using the Prometheus data source. For self-hosted Grafana instances, a configuration change is needed to use the Azure Authentication option in Grafana. For Grafana instances that aren't managed by Azure, make the following changes:
Versions 9.x and greater of Grafana support Azure Authentication, but it's not enabled by default. To enable Azure Authentication, update your Grafana configuration and restart the Grafana instance. To find your grafana.ini file, review the Configure Grafana document from Grafana Labs.
Locate and open the grafana.ini file on your virtual machine.
Identify your Grafana version.
For Grafana 9.0, in the [feature_toggles] section, set prometheus_azure_auth to true.
For Grafana 9.1 and later versions, in the [auth] section, set the azure_auth_enabled setting to true.
Restart the Grafana instance.
Create the Prometheus data source in Grafana
Configure Prometheus as a data source using the following steps:
Open Grafana in your browser.
Select Connections and then Data sources.
Select Add data source
Search for and select Prometheus.
Paste the query endpoint from your Azure Monitor workspace into the URL field.
Under Authentication, select Azure Auth. For earlier Grafana versions, under Auth, turn on Azure Authentication
Under Azure Authentication, select App Registration from the Authentication dropdown.
Enter the Direct(tenant) ID, Application (client) ID, and the Client secret generated when you created your App registration.
Scroll to the bottom of the page and select Save & test.
Frequently asked questions
This section provides answers to common questions.
I am missing all or some of my metrics. How can I troubleshoot?
You can use the troubleshooting guide for ingesting Prometheus metrics from the managed agent here.
Why am I missing metrics that have two labels with the same name but different casing?
Azure managed Prometheus is a case insensitive system. It treats strings, such as metric names, label names, or label values, as the same time series if they differ from another time series only by the case of the string. For more information, see Prometheus metrics overview.
I see some gaps in metric data, why is this occurring?
During node updates, you might see a 1-minute to 2-minute gap in metric data for metrics collected from our cluster level collectors. This gap occurs because the node that the data runs on is being updated as part of a normal update process. This update process affects cluster-wide targets such as kube-state-metrics and custom application targets that are specified. This occurs when your cluster is updated manually or via autoupdate. This behavior is expected and occurs due to the node it runs on being updated. This behavior doesn't affect any of our recommended alert rules.
