Edit

Share via

Azure Monitor and Prometheus

  • Article

Prometheus is a popular open-source monitoring and alerting solution that's widely used in the cloud-native ecosystem. Organizations use Prometheus to monitor and alert on the performance of infrastructure and workloads. It's often used in Kubernetes environments.

You can use Prometheus as an Azure-managed service or as a self-managed service to collect metrics. Prometheus metrics can be collected from your Azure Kubernetes Service (AKS) clusters, Azure Arc-enabled Kubernetes clusters, virtual machines, and virtual machine scale sets.

Prometheus metrics are stored in an Azure Monitor workspace. You can analyze and visualize the data in a workspace by using metrics explorer with Prometheus Query Language (PromQL) and Azure Managed Grafana.

Important

The use of Azure Monitor to manage and host Prometheus is intended for storing information about the service health of customer machines and applications. It's not intended for storing any personal data. We strongly recommend that you don't send any sensitive information (for example, usernames and credit card numbers) into Azure Monitor-hosted Prometheus fields like metric names, label names, or label values.

Azure Monitor managed service for Prometheus

Azure Monitor managed service for Prometheus is a component of Azure Monitor Metrics that provides a fully managed and scalable environment for running Prometheus. It simplifies the deployment, management, and scaling of Prometheus in AKS, so you can focus on monitoring your applications and infrastructure.

As a fully managed service, Azure Monitor managed service for Prometheus automatically deploys Prometheus in AKS or Azure Arc-enabled Kubernetes. The service provides high availability, service-level agreement (SLA) guarantees, and automatic software updates. It provides a highly scalable metrics store that retains data for up to 18 months.

Azure Monitor managed service for Prometheus provides preconfigured alerts, rules, and dashboards. With recommended dashboards from the Prometheus community and native Grafana integration, you can achieve comprehensive monitoring immediately. Azure Monitor managed service for Prometheus integrates with Azure Managed Grafana, and it also works with self-managed Grafana.

Pricing is based on ingestion and query with no additional storage cost. For more information, see the Metrics tab in Azure Monitor pricing.

Enable Azure Monitor managed service for Prometheus

Azure Monitor managed service for Prometheus collects data from AKS and Azure Arc-enabled Kubernetes.

To enable Azure Monitor managed service for Prometheus, you must create an Azure Monitor workspace to store the metrics. You can then onboard services that collect Prometheus metrics:

To enable managed Prometheus for Microsoft Azure air-gapped clouds, contact support.

Azure-hosted self-managed Prometheus

In addition to managed service for Prometheus, you can install and manage your own Prometheus instance and use remote write to store metrics in an Azure Monitor workspace.

By using remote write, you can collect data from self-managed Prometheus servers running in the following environments:

  • Azure virtual machines
  • Azure virtual machine scale sets
  • Azure Arc-enabled servers
  • Self-manged Azure-hosted or Azure Arc-enabled Kubernetes clusters

Self-managed Kubernetes services

Send metrics from self-managed Prometheus on Kubernetes clusters. For more information on remote write to Azure Monitor workspaces for Kubernetes services, see the following articles:

Virtual machines and virtual machine scale sets

Send data from self-managed Prometheus on virtual machines and virtual machine scale sets. The virtual machines can be in an Azure-managed environment or on-premises. For more information, see Send Prometheus metrics from virtual machines, scale sets, or Kubernetes clusters to an Azure Monitor workspace.

Data storage

Prometheus metrics are stored in an Azure Monitor workspace. The data is stored in a time-series database that can be queried via PromQL. You can store data from several Prometheus data sources in a single Azure Monitor workspace. For more information, see Azure Monitor workspace architecture.

Azure Monitor workspaces retain data for 18 months.

Querying and analyzing Prometheus metrics

Prometheus data is retrieved via PromQL. You can write your own queries, use queries from the open-source community, and use Grafana dashboards that include PromQL queries. For more information, see the Querying Prometheus on the Prometheus website.

The following Azure services support querying Prometheus metrics from an Azure Monitor workspace:

Azure Monitor metrics explorer with PromQL

Use metrics explorer with PromQL (preview) to analyze and visualize platform and Prometheus metrics. Metrics explorer with PromQL is available from the Metrics pane in the Azure Monitor workspace where your Prometheus metrics are stored. For more information, see Azure Monitor metrics explorer with PromQL.

Screenshot of a PromQL query in Azure Monitor metrics explorer.

Azure workbooks

Create charts and dashboards powered by Azure Monitor managed service for Prometheus by using Azure workbooks and PromQL queries. For more information, see Query Prometheus metrics using Azure workbooks.

Grafana integration

Visualize Prometheus metrics by using Azure Managed Grafana. Connect your Azure Monitor workspace to a Grafana workspace so that you can use it as a data source in a Grafana dashboard. You then have access to multiple prebuilt dashboards that use Prometheus metrics. You also have the ability to create any number of custom dashboards. For more information, see Link a Grafana workspace.

Prometheus query API

Use PromQL via the REST API to query Prometheus metrics stored in an Azure Monitor workspace. For more information, see Query Prometheus metrics using the API and PromQL.

Rules and alerts

Prometheus supports recording rules and alert rules by using PromQL queries. Azure Monitor managed service for Prometheus automatically deploys rules and alerts. Metrics that recording rules record are stored in the Azure Monitor workspace. Dashboards or other rules can then query the metrics.

You can create and manage alert rules and recording rules by using Azure Monitor managed service for Prometheus rule groups. For your AKS cluster, a set of predefined Prometheus alert rules and recording rules helps you get started quickly.

Alerts that alert rules fire can trigger actions or notifications, as defined in the action groups configured for the alert rule. You can also view fired and resolved Prometheus alerts in the Azure portal, along with other alert types.

Service limits and quotas

Azure Monitor managed service for Prometheus has default limits and quotas for ingestion. When you reach the ingestion limits, throttling can occur. You can request an increase in these limits. For more information, see Azure Monitor service limits.

To monitor and alert on your ingestion metrics, see Monitor Azure Monitor workspace metrics ingestion.

Limitations

The following limitations apply to Azure Monitor managed service for Prometheus:

  • The minimum frequency for scraping and storing metrics is 1 second.
  • During node updates, you might experience gaps that last 1 to 2 minutes in some metric collections from the cluster-level collector. This gap is due to a regular action from Azure Kubernetes Service to update the nodes in your cluster. This behavior doesn't affect recommended alert rules.
  • Managed Prometheus for Windows nodes isn't automatically enabled. To enable monitoring for Windows nodes and pods in your clusters, see Enable Windows metrics collection (preview).

Case sensitivity

Azure Monitor managed service for Prometheus is a case-insensitive system. It treats strings (such as metric names, label names, or label values) as the same time series if they differ from another time series only by the case of the string.

Note

This behavior is different from native open-source Prometheus, which is a case-sensitive system. Self-managed Prometheus instances running in Azure virtual machines, virtual machine scale sets, or Azure Kubernetes Service clusters are case-sensitive systems.

In managed service for Prometheus, the following time series are considered the same:

diskSize(cluster="eastus", node="node1", filesystem="usr_mnt")
diskSize(cluster="eastus", node="node1", filesystem="usr_MNT")

The preceding examples are a single time series in a time series database. The following considerations apply:

  • Any samples ingested against them are stored as if they're scraped or ingested against a single time series.
  • If the preceding examples are ingested with the same time stamp, one of them is randomly dropped.
  • The casing that's stored in the time series database and returned by a query is unpredictable. The same time series might return different casing at different times.
  • Any metric name or label name/value matcher present in the query is retrieved from the time series database through a case-insensitive comparison. If there's a case-sensitive matcher in a query, it's automatically treated as a case-insensitive matcher in string comparisons.

It's a best practice to use a single consistent case to produce or scrape a time series.

Open-source Prometheus treats the preceding examples as two different time series. Any samples scraped or ingested against them are stored separately.

Prometheus references

Following are links to Prometheus documentation:

Related content