Azure Container App Load Balancer TCP RST messages

Question

I'm hosting a TCP server, with 2-way authentication using SSLStream, on Azure Container App instance, running Linux. My solution is written with .net 8. The load balancer that is automatically created with the Azure Container App, does not send the TCP RST packets, once the 4 minutes idle time has passed (this is the default timeout on both VIP and Load Balancer rule). This behavior is causing both the Client and Server to not be aware of the dropped connection.

To tackle with this issue, I set the client to send KeepAlive messages to maintain the TCP connection established.

I need the server to also send KeepAlive messages to the client, in cases where the idleness of the connection is more than 240 seconds. So I'm setting the below to each newly established connection for each client:

Socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true); Socket.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveTime, 240); Socket.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveInterval, 5); Socket.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.TcpKeepAliveRetryCount, 3);

Unfortunately, the KeepAlive messages are not send by the server at all. Note than even if I remove the ethernet plug from the connected client, still no KeepAlive messages are coming from the server (I'm using Wireshark to capture the network traffic).

Trying to set the KeepAlive messages using the IOControl results to an exception, probably because the IOControlCode.KeepAliveValues flag is not supported for Linux environments:

int size = Marshal.SizeOf((uint)0); byte[] keepAlive = new byte[size * 3]; System.Buffer.BlockCopy(BitConverter.GetBytes((uint)1), 0, keepAlive, 0, size); System.Buffer.BlockCopy(BitConverter.GetBytes((uint)240_000), 0, keepAlive, size, size); System.Buffer.BlockCopy(BitConverter.GetBytes((uint)5_000), 0, keepAlive, size * 2, size); Socket.IOControl(IOControlCode.KeepAliveValues, keepAlive, null);

I also tried to set the keepalive option on Kernel level by modifying the sysctl.conf file, by applying the values:

• net.ipv4.tcp_keepalive_time=240
• net.ipv4.tcp_keepalive_intvl=5
• net.ipv4.tcp_keepalive_probes=3
• sysctl -p

Again, no success on managing the server to send the KeepAlive messages to the client (Although I would like to avoid the above because it affects all the active connections to the server)

What I can tell from this behavior, is that I do not have the ability to change the default KeepAlive settings to my Azure Container App, which seams the IdleTime to be at 2 hours. Is there a way to change the KeepAlive settings on Azure Container App and how can I achieve that?

Please note that I'm building and publishing my container images using .NET SDK, not Docker.

Any help would be appreciated.

Answer 1

Hi @Panayiotis Zinieris ,

Thank you for your detailed explanation of the issue. It appears that you're encountering issues with the Azure Load Balancer not consistently sending TCP Reset (RST) packets to both the client and server after idle connections. The behavior you've described, where RST packets are only sent during certain events (such as server restarts or new deployments), may be linked to the idle timeout settings and how the Load Balancer handles idle connections.

As per Azure documentation, the default behavior of the Load Balancer is to silently drop connections when the idle timeout is reached. However, if TCP Reset is enabled, it should ideally send RST packets in both directions to inform the client and server that the connection is no longer valid. If this behavior is not occurring consistently, it could be attributed to several factors, including:

• Make sure the idle timeout settings for your Load Balancer are set correctly. The default timeout is 4 minutes, but you can change it to as long as 100 minutes for certain rules. When the timeout happens, the Load Balancer should send RST packets if it's set up to do so.
• Network problems like lost packets or delays can affect how RST packets are delivered. Using tools like Wireshark to check network traffic can help you see if packets are being sent but not received. Also, if your application doesn’t handle RST packets well or has issues with managing connections, it might not react properly when connections are dropped.
• Even though you said keep-alives are turned off, turning on TCP keep-alives can help keep the connection alive and make it easier to spot broken connections.

If you have explored all the options in the Azure Load Balancer settings and your application configuration, you might want to consider implementing application-level keep-alives or monitoring mechanisms to more effectively detect broken connections.

For your reference, please review the following documentation for further clarification:

• Load Balancer TCP Reset and Idle Timeout

I hope this is helpful! Feel free to reach out if you have any more questions or need further assistance.