Azure EvenHub integration with Elasticsearch

Question

Hello,

I want to send the events on the Event Hub or Azure Service Bus to Elastic (installed on an Azure VM). There are two integrations that I found - Azure Event Hub Integration and Azure Logstash Integration. Can you help me understand how to make a decision on choosing the integration type?

I will have 100K+ events sent by an Azure function to the Event Hub (or Service Bus) and I want to load them in Elastic cloud via a plugin.

Thanks

Poonam

Answer 1

Hello Poonam , Welcome to MS Q&A

To choose between Azure Event Hub Integration and Azure Logstash Integration for sending events to Elastic, consider the following factors:

Purpose: Azure Event Hubs is designed for high-throughput data streaming scenarios, making it suitable for applications that require the ingestion of large volumes of events with low latency. In contrast, Logstash is a server-side data processing pipeline that can ingest data from various sources, transform it, and send it to your preferred storage solution, including Elastic.

Integration Needs: If your use case involves integrating with various Azure services and you need a serverless event broker, Azure Event Hubs may be more appropriate. If you require more complex data processing and transformation capabilities before sending data to Elastic, Logstash would be a better fit.

Performance: For scenarios where performance is critical and you need to handle billions of events per day, Azure Event Hubs is optimized for such high-throughput scenarios. Logstash can also be configured for performance, but it may require more tuning based on the data volume and processing requirements.

Data Processing: If you need to perform extensive data transformations or enrichments before sending data to Elastic, Logstash provides powerful capabilities to achieve this. Azure Event Hubs focuses more on the ingestion and streaming aspect.

By evaluating your specific requirements against these factors, you can make a more informed decision on which integration to use.

References:

• Select a data ingestion tool
• Connecting IoT Devices to Azure: IoT Hub and Event Hubs

Please let us know if any questions

Kindly accept answer if it helps

Thanks

Deepanshu