How to create Native Prometheus Alert on Managed AKS/Prometheus

KaushikKumar Patel 21 Microsoft Employee

Hey folks, I have Managed Grafana and Prometheus on AKS. Native Opensource Prometheus has been in use for few years, hence has many custom (application/service) related alerting and recording rule setup.

Is there anyway in Azure, where we can deploy these prometheus alert rules without using azure prometheus terraform/bicep logic.

Goal is each app/service team manages their own set of alerts which are deployed as part of their app deployment with other K8s Manifest such as hpa, deployment, keda scaleset, configmap, and such.

This is an important requirement for us since migrating this to tf/bicep will add significant operation overhead.

I already have helm templates for Prometheus alerts, but not sure how to get them applied/deployment to managed AKS/Prometheus.

Akshay kumar Mandha 1,265 Reputation points Microsoft Vendor

2024-11-11T09:57:07.3933333+00:00

Hi KaushikKumar Patel,
Welcome to the Microsoft Q&A Platform. Thank you for posting your query here. We are looking into it and will get back to you soon.

Akshay kumar Mandha 1,265 Microsoft Vendor

Hi KaushikKumar Patel,
Thanks for patience while we are reviewing your thread

Based on your query what I understand do you want to deploy the existing custom rules in AKS without terraform and biceps if that is the case could you please try below things and let us know

1.Create a ConfigMap for Prometheus Rules:

Store your custom Prometheus rules in a ConfigMap. This ConfigMap will be mounted into the Prometheus container.

ConfigMap (prometheus-rules-configmap.yaml):

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-rules
  namespace: monitoring
data:
  prometheus.rules: |
    groups:
    - name: custom_rules
      rules:
      - alert: HighMemoryUsage
        expr: node_memory_Active_bytes / node_memory_MemTotal_bytes * 100 > 80
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High Memory usage detected"
          description: "Memory usage is above 80% for more than 5 minutes."

Modify Prometheus Deployment to Mount ConfigMap:

Update your Prometheus deployment to mount the ConfigMap containing the custom rules.

Prometheus Deployment (prometheus-deployment.yaml):

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus
        image: prom/prometheus
        args:
        - "--config.file=/etc/prometheus/prometheus.yml"
        - "--web.enable-lifecycle"
        volumeMounts:
        - name: prometheus-config
          mountPath: /etc/prometheus
        - name: prometheus-rules
          mountPath: /etc/prometheus/rules
      volumes:
      - name: prometheus-config
        configMap:
          name: prometheus-config
      - name: prometheus-rules
        configMap:
          name: prometheus-rules
          items:
          - key: prometheus.rules
            path: prometheus.rules.yml

Use kubectl apply to deploy the ConfigMap and update the Prometheus deployment in your AKS cluster.

Please let us know if you face any difficulties we will help as you needed and do you have any other question, please tag me in comment

KaushikKumar Patel 21 Reputation points Microsoft Employee

2024-11-13T04:18:45.4633333+00:00

thanks, but this is not viable option since we have 50+ team, about 300+ Microservice with their sets of rules. This will be massive overhead and we are tinkering with azure managed Prometheus container instance, so not sure this is good suggestion or recommendation.
Akshay kumar Mandha 1,265 Reputation points Microsoft Vendor

2024-11-13T13:49:04.7866667+00:00

Hi KaushikKumar Patel,
Thanks for sharing your concerns. I got it your point and understand that managing over, no of teams and more no of microservices manually would be a huge challenge. We are reviewing your query & looking into the Azure Managed Prometheus container instances to help automate and simplify the process, considering the scale and needs of your applications. Your feedback is very helpful as we work on a solution to ensure automation without adding extra overhead. We will provide you with a better solution and suggestions shortly
Akshay kumar Mandha 1,265 Reputation points Microsoft Vendor

2024-11-21T05:10:21.7966667+00:00

Hi KaushikKumar Patel,
I hope doing well,
Have tried any process to integrate in the meantime if you made any progress, please share with us here, we can also support if you are following some process, we can assist you and also we are continuing to search for a solution to this query

Share via

How to create Native Prometheus Alert on Managed AKS/Prometheus

Your answer