Is it possible to send email alerts on Azure container health probes failing?

RSa 0

We have liveness, startup and readiness health probes setup on some Azure Container Apps.

We also can see the log analytics tables however do not seen any health probe data in them.

Is it possible in bicep to create alerts that go to a specific email address should any of these health probes for the container app fail?

1 answer

Vinodh247 23,346 Reputation points MVP

2024-09-09T16:45:54.5966667+00:00
Hi ,

Thanks for reaching out to Microsoft Q&A.

Even if you're not seeing health probe data in Log Analytics tables, make sure that diagnostic logging is enabled for your Azure Container Apps and sending logs to Log Analytics. You can do this by configuring diagnostic settings.

You’ll want to monitor any log data related to container health probes. Once diagnostic settings are configured, you can query these logs in Log Analytics for events like failed health probes.

You might need to check AppLogs or ContainerAppConsoleLogs to get health probe failures.

You can create a log alert rule in Azure Monitor based on your Log Analytics query for health probe failures.

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.
Please sign in to rate this answer.
Mahesh Goud Juvvadi 1,070 Reputation points Microsoft Vendor

2024-09-09T18:28:05.93+00:00

Hi RSa,

Just checking in to see if you have got a chance to see the answer posted by Vinodh247 in resolving the issue.

If it was helpful, please click "Upvote and Accept Answer" on his post to let us know.

We're here to help, so if you have any further questions, don't hesitate to ask.

Thank You.

RSa 0 Reputation points

2024-09-10T08:17:07.1933333+00:00

Hi @Vinodh247 our container app is outputting to the log tables, however I am specifically looking for an example how to query for a health probe entries in the log tables using Bicep (see original question). I can see container app logs there but nothing to do with health probes even though they are setup.

RSa 0 Reputation points

2024-09-10T08:18:39.6033333+00:00

Hi @Vinodh247 our container app is outputting to the log tables, however I am specifically looking for an example how to query for a health probe entries in the log tables using Bicep (see original question). I can see container app logs there but nothing to do with health probes even though they are setup.

RSa 0 Reputation points

2024-09-10T08:18:55.4333333+00:00

Hi @Vinodh247 our container app is outputting to the log tables, however I am specifically looking for an example how to query for a health probe entries in the log tables using Bicep (see original question). I can see container app logs there but nothing to do with health probes even though they are setup.

Vinodh247 23,346 Reputation points MVP

2024-09-10T10:17:55.76+00:00

Make sure you have the necessary diagnostic settings enabled for your container app to capture all relevant logs. This ensures that container logs are fully available in Log Analytics.

// Query to search for potential health probe failure indicators in logs AppLogs | where LogEntry contains "HealthCheck" or LogEntry contains "Health Probe" or LogEntry contains "probe failure" | project TimeGenerated, LogEntry

You can test variations of above query in azure monitor logs to catch entries related to health probes. Probe failures might sometimes show up as generic failures in the application logs, or they may use terms such as "HealthCheck" or "Health Probe" depending on how they're surfaced in your container app.

Post this you can create an alert based on this query using Bicep using the following template.

resource logAlert 'Microsoft.Insights/scheduledQueryRules@2022-04-01' = { name: 'ContainerHealthProbeAlert' location: resourceGroup().location properties: { description: 'Alert on container health probe failures' enabled: true source: { dataSourceId: logAnalytics.id query: ''' AppLogs | where LogEntry contains "HealthCheck" or LogEntry contains "Health Probe" or LogEntry contains "probe failure" | summarize count() by bin(TimeGenerated, 5m) ''' } schedule: { frequencyInMinutes: 5 timeWindowInMinutes: 15 } criteria: { allOf: [ { query: ''' AppLogs | where LogEntry contains "HealthCheck" or LogEntry contains "Health Probe" or LogEntry contains "probe failure" | summarize count() by bin(TimeGenerated, 5m) ''' threshold: 1 timeAggregation: 'Count' operator: 'GreaterThan' } ] } actionGroups: [ { actionGroupId: actionGroup.id } ] } } resource actionGroup 'Microsoft.Insights/actionGroups@2022-04-01' = { name: 'EmailAlertActionGroup' location: resourceGroup().location properties: { enabled: true emailReceivers: [ { name: 'adminEmail' emailAddress: 'your-email@example.com' status: 'Enabled' } ] } }

RSa 0 Reputation points

2024-09-10T12:24:05.4666667+00:00

Thanks, that LogEntry table does not exist, however the below returns some errors, is this the same health probe logs?

ContainerAppSystemLogs_CL | where Reason_s == "ProbeFailed"

Vinodh247 23,346 Reputation points MVP

2024-09-10T13:02:51.7933333+00:00

'ContainerAppSystemLogs_CL' was an example and might be present in different name in your setup. Normally, in log analytics, you'll find logs under tables like ContainerInstanceLog_CL, ContainerAppConsoleLogs or similarly named tables depending on the service and how logging is configured.

See the example query below to look for errors that could include health probe failures:

ContainerInstanceLog_CL | where LogEntry contains "error" or LogEntry contains "fail" | project TimeGenerated, LogEntry, ContainerID

you will have to replace ContainerInstanceLog_CL with the actual name of the log table in your environment.

RSa 0 Reputation points

2024-09-10T13:19:01.9466667+00:00

Thanks one question, if I have found the right table would the log reason be "ProbeFailed"? What do health probes log out as?

Vinodh247 23,346 Reputation points MVP

2024-09-10T13:42:01.2866667+00:00

If i understand correctly about your question:

health probes may not explicitly log an entry as "ProbeFailed" unless your application is designed to log these events deliberately. when health probes fail, they might log entries that include terms like:

"liveness probe failed"

"readiness probe failed"

"unhealthy"

Specific error messages depending on why the probe failed (ex., connection timeouts, HTTP status codes other than 200-399, etc.)

RSa 0 Reputation points

2024-09-10T15:35:08.5066667+00:00

Thanks @Vinodh247 I cannot find similar logs other than "ProbeFailed" Is there a defined list somewhere? Its so wooly to be able to alert on a health probe, seems odd there is nothing first class in Azure Monitor for this.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Is it possible to send email alerts on Azure container health probes failing?

1 answer

Your answer