Hi, I have a VNet in a resource group with below elements: A VM with a Kafka broker listening on its public and private IPs and ports. A Standard Load-Balancer in front of the VM with LB rules to Kafka broker listening ports. A Private Link Service attached to above LB. I have a SAP BTP subaccount with Private Link Service. I created a Private Endpoint from the subaccount to above Private Link Service and it is approved. I have an application running on my BTP subaccount space and already bound with the Private Link Service successfully. I tried to consume the message from the Kafka broker through the Private Endpoint but I got connection refused. I tried to use both LB's IP and VM's IP but I had no luck. Any idea why I got the error in open connection to the VM via LB's IP and through Private Endpoint? Regards, Steven

Connect to a service on an VM in backend-pool behind a load balancer through Private Link Service

Peter Ha 0

Hi,

I have a VNet in a resource group with below elements:

A VM with a Kafka broker listening on its public and private IPs and ports.
A Standard Load-Balancer in front of the VM with LB rules to Kafka broker listening ports.
A Private Link Service attached to above LB.
I have a SAP BTP subaccount with Private Link Service. I created a Private Endpoint from the subaccount to above Private Link Service and it is approved.
I have an application running on my BTP subaccount space and already bound with the Private Link Service successfully.

I tried to consume the message from the Kafka broker through the Private Endpoint but I got connection refused. I tried to use both LB's IP and VM's IP but I had no luck.

Any idea why I got the error in open connection to the VM via LB's IP and through Private Endpoint?

Regards,

Steven

Rohith Vinnakota 1,515 Reputation points Microsoft Vendor

2024-12-19T13:03:46.27+00:00
Hi Peter Ha,

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

Could you share the topology diagram?

Can you perform an nslookup from the source machine? For example: nslookup <FQDN>.

Also, please run PSping on the source machine with port 443. For example: psping <privateIP>:<port>.https://learn.microsoft.com/en-us/sysinternals/downloads/psping

what is your requirement here it helps to resolve the issue?

Refer this links:

https://learn.microsoft.com/en-us/azure/private-link/create-private-link-service-portal?tabs=dynamic-ip

https://learn.microsoft.com/en-us/azure/load-balancer/backend-pool-management#limitations

Thanks,
Rohith
Peter Ha 0 Reputation points

2024-12-20T04:22:29.6833333+00:00

Hi @Rohith Vinnakota ,Thanks for your response.

Please find below the topology diagram.

The purpose is to connect to the VM at a specific port through Private Endpoint in Private Link Service.

Unfortunately, I could not run the nslookup command from source system because it's a cloud tenant (SAP BTP Cloud Foundry).

I went through the MS documentation and I could see my setup is the same. It's strange that I could not open the connection to the VM/load balancer using its private IP. I also attached the Private Endpoint with Approved status.

Regards,

Steven

AinhoaGiles 150

The "connection refused" error you're encountering could be due to several potential issues in your configuration. Here are a few things to check:

Kafka Broker Binding Configuration:

Ensure that Kafka is properly configured to listen on both its public and private IPs. Kafka typically binds to either specific IP addresses or all interfaces (0.0.0.0). If it is only listening on the public IP, requests from your private network (via the Private Endpoint) would be refused. Verify the listeners and advertised.listeners settings in your server.properties file.

Network Security Group (NSG) Rules:

  - Check the Network Security Groups (NSGs) associated with both the VM and the Private Endpoint. Ensure that there are inbound and outbound rules allowing traffic on the Kafka broker's ports (usually 9092). For Private Endpoint traffic, make sure the required ports are open.
  
  **Load Balancer Health Probe**:
  
     - Verify that the Load Balancer health probes are correctly configured for the Kafka broker’s listening ports. If the health probe is failing, the Load Balancer may not route traffic properly to your VM.
     
     **Private Link Configuration**:
     
        - Make sure that the Private Link Service and Private Endpoint are configured correctly. Double-check the IP range assigned to the Private Endpoint and ensure that it matches the private IP of the Load Balancer. Also, ensure that the Private Link Service is properly associated with the Load Balancer and that the required ports are exposed.
        
        **Kafka’s `advertised.listeners` Setting**:
        
           - For Kafka, the `advertised.listeners` must be set to the correct IP (the private IP through the Load Balancer, in your case). This helps clients (from the BTP subaccount in this case) resolve the broker's IP correctly when attempting to connect. Make sure this setting is properly configured.
           
           **DNS Resolution**:
           
              - Verify that the Private Endpoint is resolving the correct IP address and DNS name. The DNS name should resolve to the Load Balancer's private IP, not the public IP.
              
              **Private Link Service Binding**:
              
                 - Ensure that the Private Link Service is properly bound to your application in the SAP BTP subaccount. Sometimes, after creating or approving the Private Endpoint, a restart or re-binding may be needed to refresh the connection.
                 
                 **Firewall or ACLs**:
                 
                    - If there are any firewalls or access control lists (ACLs) in your environment, ensure they are not blocking traffic between the Private Endpoint and the Load Balancer or Kafka broker.

Next Steps:

Start by confirming Kafka’s listeners and advertised.listeners settings, making sure both the VM’s private IP and the Load Balancer IP are included.
Ensure the Load Balancer’s health probes and traffic routing are correctly set up.
Review the NSG and security rules to allow traffic through the necessary ports.
Confirm DNS resolution for the Private Endpoint to ensure it resolves correctly to the Load Balancer’s IP.

If these configurations are correct, your Kafka client should be able to connect via the Private Endpoint to the broker through the Load Balancer. The "connection refused" error you're encountering could be due to several potential issues in your configuration. Here are a few things to check:

Kafka Broker Binding Configuration:

Ensure that Kafka is properly configured to listen on both its public and private IPs. Kafka typically binds to either specific IP addresses or all interfaces (0.0.0.0). If it is only listening on the public IP, requests from your private network (via the Private Endpoint) would be refused. Verify the listeners and advertised.listeners settings in your server.properties file.

Network Security Group (NSG) Rules:

  - Check the Network Security Groups (NSGs) associated with both the VM and the Private Endpoint. Ensure that there are inbound and outbound rules allowing traffic on the Kafka broker's ports (usually 9092). For Private Endpoint traffic, make sure the required ports are open.
  
  **Load Balancer Health Probe**:
  
     - Verify that the Load Balancer health probes are correctly configured for the Kafka broker’s listening ports. If the health probe is failing, the Load Balancer may not route traffic properly to your VM.
     
     **Private Link Configuration**:
     
        - Make sure that the Private Link Service and Private Endpoint are configured correctly. Double-check the IP range assigned to the Private Endpoint and ensure that it matches the private IP of the Load Balancer. Also, ensure that the Private Link Service is properly associated with the Load Balancer and that the required ports are exposed.
        
        **Kafka’s `advertised.listeners` Setting**:
        
           - For Kafka, the `advertised.listeners` must be set to the correct IP (the private IP through the Load Balancer, in your case). This helps clients (from the BTP subaccount in this case) resolve the broker's IP correctly when attempting to connect. Make sure this setting is properly configured.
           
           **DNS Resolution**:
           
              - Verify that the Private Endpoint is resolving the correct IP address and DNS name. The DNS name should resolve to the Load Balancer's private IP, not the public IP.
              
              **Private Link Service Binding**:
              
                 - Ensure that the Private Link Service is properly bound to your application in the SAP BTP subaccount. Sometimes, after creating or approving the Private Endpoint, a restart or re-binding may be needed to refresh the connection.
                 
                 **Firewall or ACLs**:
                 
                    - If there are any firewalls or access control lists (ACLs) in your environment, ensure they are not blocking traffic between the Private Endpoint and the Load Balancer or Kafka broker.

Next Steps:

Start by confirming Kafka’s listeners and advertised.listeners settings, making sure both the VM’s private IP and the Load Balancer IP are included.
Ensure the Load Balancer’s health probes and traffic routing are correctly set up.
Review the NSG and security rules to allow traffic through the necessary ports.
Confirm DNS resolution for the Private Endpoint to ensure it resolves correctly to the Load Balancer’s IP.

If these configurations are correct, your Kafka client should be able to connect via the Private Endpoint to the broker through the Load Balancer.

Rohith Vinnakota 1,515 Reputation points Microsoft Vendor

2024-12-20T12:28:07.89+00:00
Hi @Peter Ha ,

Good day!

Could you share the MS documentation to help reproduce the issue in my lab?

You can also look into AinhoaGiles suggestion.

Thanks,
Rohith
Peter Ha 0 Reputation points

2024-12-24T13:44:52.72+00:00

Hi @Rohith Vinnakota ,

I followed the documentation at https://github.com/SAP-samples/btp-build-resilient-apps/blob/main/tutorials/05_setupconnectivity/pl-azure.md to create everything on Azure and BTP.

I also had a look at @AinhoaGiles but I didn't find anything wrong. I also have a simple nginx index page running on the VM and I also cannot access the page using the LB's IP.

Regards,

Steven
Peter Ha 0 Reputation points

2024-12-24T13:49:17.9+00:00

Hi @AinhoaGiles ,

Thanks for your answer. Unfortunately, I didn't find anything strange in my setup. I also tried with a simple HTML page in NGINX and I could not access it via LB's IP.

Regards,

Steven

Share via

Connect to a service on an VM in backend-pool behind a load balancer through Private Link Service

Your answer