KEDA Unable to Connect to Kafka Broker in Same Container Apps Environment.

Question

I have two container apps. One is running an instance of kafka and the other is consuming kafka messages. They both live inside the same container app environment and traffic from outside the environment is not allowed. The consumer app can connect to the kafka without any issues. I can also SSH into the consumer app and connect to the kafka from the console no issues there. I have also created a KEDA scale rule of type 'kafka' on the consumer app which is configured to connect to kafka using the same URL, but it is not able to reach the server (See error below).

Reason: KEDAScalerFailed

Msg: "error creating kafka client: kafka: client has run out of available brokers to talk to: dial tcp xx.x.xxx.xxx:9092: connect: connection refused"

I have configured all network security group rules according to this documentation.

It seems strange that keda can't access the container app environments internal network. I have tried every possible configuration I can think of without any luck (different URL's, directly specifying IP, enable/disable TLS, etc...) . Help is deeply appreciated!

Answer 1

Hi ,

Thanks for reaching out to Microsoft Q&A.

The issue appears to be related to how KEDA interacts with Kafka in a secure Container Apps environment. Here are some steps to troubleshoot and resolve the issue:

• Ensure that the Kafka broker's address in the KEDA scale rule matches what is used by the consumer app. If the consumer app can connect using a specific hostname or IP address, use the same in the KEDA configuration.
• If Kafka is listening on localhost within its container, it might not be accessible from other containers. Update Kafka's advertised.listeners configuration to expose it properly, e.g., PLAINTEXT://<container-app-internal-IP>:9092.
• Both apps are within the same Container Apps environment, so ensure the Internal network type is configured correctly for communication.
• Validate that the kafka container app is exposing port 9092 correctly in its configuration.
• Confirm the NSG and VNET allow traffic on the Kafka broker's port (9092) within the environment. Use tools like telnet or nc from the consumer app's container to ensure the port is reachable.
• Ensure the KEDA scale rule has the correct Kafka broker address, topic, and other configurations.
• If Kafka is configured with TLS or SASL, ensure the correct certificates and authentication configurations are provided to KEDA. Misconfiguration can lead to connection refusals. Disable TLS temporarily for testing (if possible) to isolate the issue.
• Deploy a standalone Kafka client (e.g., Kafka CLI tools) in the same Container Apps environment to validate connectivity using the same configurations as KEDA.
• Increase the verbosity of KEDA logs to gather more details about the connection failure. You can enable debug logging by setting the appropriate logging level in KEDA's deployment.
• Use an external IP or DNS for the Kafka broker to bypass internal networking issues temporarily. Deploy Kafka in a separate environment or VM with clearer networking boundaries and integrate it with KEDA.

Please feel free to click the 'Upvote' (Thumbs-up) button and 'Accept as Answer'. This helps the community by allowing others with similar queries to easily find the solution.