Getting Notification for AKS Events

Question

I am trying to get notifications for Kubernetes events as seen in the screenshot attached.

Is there a simple way to get notified if a certain event occurs? Specifically I'm looking to be notified on when a SPOT VM has a preemption and is scheduled for deletion.

Surely I don't have to send these event logs which already exist to azure monitor just to get a notification?

Appreciate any assistance.

Thanks

Answer 1

Hi Jacob Solomon,

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

To get notifications for AKS events like SPOT VM preemption, Azure Monitor is required. You need to enable Diagnostic Settings to send the kube-events logs to a Log Analytics workspace. Once the logs are available, you can set up Log Alerts in Azure Monitor to trigger notifications based on specific conditions, such as SPOT VM preemption.

Go to your AKS cluster in the Azure portal, enable Diagnostic Settings, and send the kube-events logs to a Log Analytics workspace. Use the following query to filter preemption events:

KubeEvents
| where Reason == "Preempted" and Message contains "spot"

Create an alert rule in Azure Monitor based on this query to receive notifications.

For more details, please refer to the below documentation:
Monitor Azure Kubernetes Service (AKS)
Create or edit a log search alert rule

If an answer has been helpful, please consider accept the "Answer" and "Upvote" to help increase visibility of this question for other members of the Microsoft Q&A community. 

Answer 2

Hi @Jacob Solomon,

We have noticed that you rated an answer as not helpful. We appreciate your feedback and are committed to improving your experience with the Q&A. I have worked on finding an alternative solution to address your issue.

Get notifications for AKS events like SPOT VM preemption and deletion- Users can opt in to receive in-VM notifications through Azure Scheduled Events. This notifies you if your VMs are being evicted and you have 30 seconds to finish any jobs and perform shutdown tasks prior to the eviction- https://learn.microsoft.com/en-us/azure/virtual-machine-scale-sets/use-spot#eviction-policy

Scheduled events provide proactive notifications about upcoming events, to Create availability alert rule for Azure virtual machine.

Scheduled Events provides events in the following use cases:

• Platform initiated maintenance (for example, VM reboot, live migration or memory preserving updates for host).
• Virtual machine is running on degraded host hardware that is predicted to fail soon.
• Virtual machine was running on a host that suffered a hardware failure.
• User-initiated maintenance (for example, a user restarts or redeploys a VM).
• Spot VM and Spot scale set instance evictions.

Depends on your configuration of spot vmss eviction type, you need to create new alert rule with values- https://learn.microsoft.com/en-us/azure/azure-monitor/vm/tutorial-monitor-vm-alert-availability#view-vm-availability-metric-in-metrics-explorer

If you have any further queries, do let us know.

I have provided more details, hope it helps. If it does, I would request you to kindly please consider accepting it as an answer and do a thumbs up at “Was it helpful”. This in turn will benefit other community members with similar scenario navigate better to right solution.

Thank you!