How can we integrate hadoop logs in insights

Question

Hi ,

Need to integrage spark application logs in azure insights or api insgihts.

And also need to build monitoring dashboard for azure spark hid ingishts

Any inputs please

Answer 1

Hello Panneeru, Vikram,

You can use Azure Monitor Logs to collect data from resources in your cloud and on-premises environments. This service allows you to analyze data across multiple sources, which can include Hadoop logs.

Azure HDInsight clusters generate logs that are written to Azure tables and Azure Blob storage. You can view these logs using Ambari Quick Links or access them directly from the storage account

reference document: https://learn.microsoft.com/en-us/azure/hdinsight/selective-logging-analysis

For programmatic management of log alerts, you can use the Scheduled Query Rules REST API. This API allows you to create, update, or delete log alerts for Application Insights

for monitoring dashboard:

Please see the below document which provides detailed information on how to monitor cluster health and availability using Azure Monitor logs

https://azure.microsoft.com/en-us/blog/monitoring-on-azure-hdinsight-part-2-cluster-health-and-availability/

I hope this answers your question. Please let me know if you have any further questions.