Server-sent events and Azure Application Gateway for Containers

Timo Notheisen 20

I have a server that supports server-sent events. When running this server locally, everything works as expected.

But as soon as I deploy it in an AKS Kubernetes cluster in front of an Azure Application Gateway for Containers, the SSE connection times out after 60 seconds. After some research, I found the following documentation: https://learn.microsoft.com/en-us/azure/application-gateway/for-containers/server-sent-events?tabs=server-sent-events-gateway-api. There it's stated that you should create a RoutePolicy and configure a timeout of 0s. But doing so doesn't change anything. The observed timeout remains 60s.

I then played around with the RoutePolicy a bit. And in case the timeout is higher than 0s and lower than 300s, the RoutePolicy works. But (as mentioned above) 0s doesn't seem to do anything at all. And any value greater than 300s leads to a 300s timeout. So the 300s appears to be the maximum.

Kubernetes version: 1.29.11.

alb-controller Helm Chart version: 1.3.7

I created a minimal example, which you can find on GitHub: https://github.com/tnotheis/SseServerTest

You can run the SSE server locally with the following command:

dotnet run dotnet run -p .\SseServerTest\SseServerTest.csproj

Afterwards, open http://localhost:5293/sse in your browser. You will receive 1 event per second, which makes it easy to see after which time a timeout occurs.

You will notice that, when running it locally, there will never be a timeout.

I further added a kubernetes.yaml file, which contains all Kubernetes resources necessary to deploy the application into a cluster with an already existing Azure Application Gateway for Containers. Just make sure to replace the name of the gateway resource as well as the hostname (both places are marked with TODO comments).

Akshay kumar Mandha 2,665 Reputation points Microsoft External Staff

2025-02-06T09:39:23.35+00:00

Hi Timo Notheisen,
Thanks for update providing the information
I'm looking into it and will update on this soon
Akshay kumar Mandha 2,665 Reputation points Microsoft External Staff

2025-02-06T23:04:24.6333333+00:00

Hi Timo Notheisen,
Thank you so much for your patience while I am reviewing your query. I came across a similar issue case found posted another community member. They have shared some troubleshooting steps that might help resolve your issue. Would you mind taking a look into below link
https://learn.microsoft.com/en-us/answers/questions/2087406/increase-application-gateway-load-balancer-gateway
Could you please try and let me know if you have any issue tag me in comment will help you as needed.!
Timo Notheisen 20 Reputation points

2025-02-07T08:44:40.0133333+00:00

Hi @Akshay kumar Mandha ,

this unfortunately doesn't help. It's about the Azure Application Gateway, while we have problems with the Azure Application Gateway for Containers. As far as I understand, these are two completely different products.
Akshay kumar Mandha 2,665 Reputation points Microsoft External Staff

2025-02-10T20:41:39.1233333+00:00

Hi Timo Notheisen,
I hope you are doing well,
I understand your concern, and I suggest that community question and answer. It might be helpful to collect logs as input. I know it involving two different products, I got your point this one not helpful for you on my previous comment I'm working into it continuously to address the issue, and I appreciate your patience while I reviewed your query & thanks, being actively giving response on my comments

I have been looking into the timeout issue, and it appears that the default timeout is 60 seconds, as mentioned in the documentation. as per below screenshot

It seems the issue might be related to the configuration being overridden, but you have said there is no older configuration, could you please recheck the setup, including the naming conventions.
Also, regarding the logs, I noticed as you mentioned there were no error logs. Could you kindly share which logs you are already verified. Could you please let me know this will helps more quickly to resolve your issue
Timo Notheisen 20 Reputation points

2025-02-11T06:53:41.4666667+00:00
It seems the issue might be related to the configuration being overridden, but you have said there is no older configuration, could you please recheck the setup, including the naming conventions.

See the following output. As I said, there is not a single RoutePolicy in the whole cluster, except for the one that's causing the problems.

❯ kubectl get --all-namespaces routepolicies.alb.networking.azure.io NAMESPACE NAME ACCEPTED AGE default sseserver-timeout-policy True 7d

Also, regarding the logs, I noticed as you mentioned there were no error logs. Could you kindly share which logs you are already verified. Could you please let me know this will helps more quickly to resolve your issue

I already sent you the output of the kubectl describe of the RoutePolicy. I'm not aware of any other logs I could check. If you let me know here else to look, I will happily do so.

I could send you the output of kubectl describe of the gateway and the HTTPRoute as well, but they all are looking fine as well, so this would just bloat this conversation. And I don't see why they would fail anyway.

Also: why would there be error logs with a timeout below 1s and higher than 300s, but not for any value in between? That just doesn't make sense.

Did you try to deploy the sample application I sent you? (https://github.com/tnotheis/SseServerTest) Because I'm pretty sure it will fail on your end as well.
Akshay kumar Mandha 2,665 Reputation points Microsoft External Staff

2025-02-12T12:06:19.16+00:00

Hi Timo Notheisen,
I am currently investigating this and trying to address the issue. please allow me some time? I'm finding inputs and will provide update as soon as possible
Akshay kumar Mandha 2,665 Reputation points Microsoft External Staff

2025-02-13T21:57:50.56+00:00

Hi Timo Notheisen,
I'm just trying to understand how things are set up in AKS. Could you put any screenshot or details how you are accessing the application in the browser? I have tested your application locally, and it works, but I want to understand the AKS setup. Could you please provide more details on this? I'm trying to address the issue quickly.
Timo Notheisen 20 Reputation points

2025-02-14T06:29:44.4+00:00
Hi,

In your AKS, you have to

install the ALB controller

Create an Application Gateway for Containers resource

After that, you can create the Kubernetes services I added to the sample repo and access the sample app via the Application Gateway for Containers.
Akshay kumar Mandha 2,665 Reputation points Microsoft External Staff

2025-02-14T21:24:14.5033333+00:00

Hi Timo Notheisen,
I followed the documentation and set up a similar configuration. However, I’m unclear about how the custom DNS hostname is configured in the Application Gateway. and the GitHub repository only includes one YAML file u have used right? and could you please clarify how the custom DNS hostname is set up in your environment? I’m trying to understand it better in order to solve the issue.
Timo Notheisen 20 Reputation points

2025-02-17T06:44:06.6133333+00:00

Hello @Akshay kumar Mandha ,

I just added the Gateway resource to the kubernetes.yaml file. I also added a self signed TLS certificate. All you have to do now is create an Azure Application Gateway for Containers resource using the "bring your own deployment" strategy (see the docs), take its resource id and replace the <gateway-resource-id> placeholder in the kubernetes.yaml file with it.

I further commented out the hostnames property in the HTTPRoute. You should be able to access the gateway via its IP address instead.
Akshay kumar Mandha 2,665 Reputation points Microsoft External Staff

2025-02-17T23:09:08.62+00:00

Hi Timo Notheisen,
Thank you for the information. I am currently reviewing it and will update you shortly.Your patience and understanding are greatly appreciated
Akshay kumar Mandha 2,665 Reputation points Microsoft External Staff

2025-02-20T22:25:40.7566667+00:00

Hi Timo Notheisen,
I hope you're doing well. Just a few things to confirm:

Was the configuration working for you without the routeTimeout: 0s issue before, or is this the first time you're encountering it? Have you tried any other approaches to resolve it?

I'm investigating this further and need more information to find a solution.
Timo Notheisen 20 Reputation points

2025-02-21T11:26:14.4933333+00:00

Hi.

Thank you for your efforts.

Was the configuration working for you without the routeTimeout: 0s issue before, or is this the first time you're encountering it?

I didn't use RoutePolicys until recently. And it has never worked since I've been using it.

Have you tried any other approaches to resolve it?

All I did so far I already documented in this thread.

Have you managed to install everything from my GitHub repo on your site? If so, can you reproduce my problem or is everything working as expected?
Akshay kumar Mandha 2,665 Reputation points Microsoft External Staff

2025-02-25T09:09:45.8666667+00:00

Hi Timo Notheisen,
Due to some restrictions and limited access, I'm unable to set up in my lab. Could you please remove the default name and only mention the timeouts below? remove Default In yaml file try once
Older one
Default
timeouts:
routeTimeout: 0s

Newer One
timeouts:
routeTimeout: 0s

Timo Notheisen 20

You mean like this?

apiVersion: alb.networking.azure.io/v1
kind: RoutePolicy
metadata:
  name: sseservertest
spec:
  targetRef:
    kind: HTTPRoute
    name: sseservertest
    group: gateway.networking.k8s.io
  timeouts:
    routeTimeout: 0s

Because this doesn't work. It gives me the following error when trying to apply:

Error from server (BadRequest): error when creating "routepolicy.yaml": RoutePolicy in version "v1" cannot be handled as a RoutePolicy: strict decoding error: unknown field "spec.timeouts"

Akshay kumar Mandha 2,665 Reputation points Microsoft External Staff

2025-02-26T15:51:19.8666667+00:00

Hi Timo Notheisen,
Yes, this is asked for what I requested. Thank you for looking into it and keeping me informed. I will check with my team.

Akshay kumar Mandha 2,665 Microsoft External Staff

Hi Timo Notheisen,
Could please give it try once if it is work it could be good if not will go for another alternative method Can you mention the timeout in Kind: HTTProute not in Kind: Routepolicy

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: sseservertest
spec:
  hostnames:
    - <hostname>  # Replace with your hostname
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: <gateway-name>  # Replace with your gateway name
  rules:
    - backendRefs:
        - kind: Service
          name: sseservertest
          port: 8080
      matches:
        - path:
            type: PathPrefix
            value: /sse
  timeouts:
    routeTimeout: 0s

Timo Notheisen 20 Reputation points

2025-02-28T11:27:44.3066667+00:00
There is no such field on an HTTPRoute . When trying to apply, it Kubernetes gives me the following error message:

strict decoding error: unknown field "spec.rules[0].timeouts.routeTimeout"

There is in fact a timeouts property (see https://gateway-api.sigs.k8s.io/api-types/httproute/#timeouts-optional), but it doesn't have the routeTimeout property you mentioned. Also, it is only available in the experimental channel. I tried it out anyway (with the request and backendRequest properties the docs mention), but it doesn't have any effect at all, no matter which value we configure.

I don't mean to be rude, but I don't have time for a try and error approach with random guesses. So I would appreciate it if you would only make suggestions that are at least based on correct syntax.

Thanks.

1 answer

Akshay kumar Mandha 2,665 Reputation points Microsoft External Staff

2025-02-04T22:03:00.6666667+00:00

Hi Timo Notheisen,
Thank you for your reply.
I understand your latest comment. Regardless of the timeout settings, it always times out after 60 seconds. If that is the case, This suggests that the timeout might be overridden, possibly due to an older configuration with a default timeout of 60 seconds in the Ingress. Could you please check the configuration for any default settings that might be causing this issue? Let me know if you find any configuration conflicts

Additionally, as you mentioned that it works locally but encounters issues when deployed in AKS, could you please provide some information on how you set up the SSE locally? It would be helpful to get an overview of the overall setup and the steps you followed for both the local environment and the AKS deployment. This will allow us to identify any potential configuration conflicts or differences
Please sign in to rate this answer.
Akshay kumar Mandha 2,665 Reputation points Microsoft External Staff

2025-02-03T20:00:29.93+00:00

Hi Timo Notheisen,
Welcome to the Microsoft Q&A Platform! Thank you for asking your question here
Based on your query,
It is working within the limit of 300 seconds. Beyond that, it does not support. Also, when you set the timeout to 0s, it still times out after 60 seconds. Is that your issue what I have understand if yes means If there are any errors or related logs identified, if possible, could you please share the logs related to this issue? better we can assist more quickly on this thing to address the issue

Timo Notheisen 20 Reputation points

2025-02-04T06:10:56.7533333+00:00

Hi @Akshay kumar Mandha ,

Thank you for your answer.

Your summary is correct, yes. But unfortunately, there are no error logs - except for the ones on the client side, that say that there was a timeout.

The RoutePolicy is Accepted, no matter if the configured timeout is 0s, 300s or 3600s. There aren't even any Kubernetes events (not even after the first apply). Here's the output of kubectl describe:

Name: sseserver-timeout-policy Namespace: default Labels: <none> Annotations: <none> API Version: alb.networking.azure.io/v1 Kind: RoutePolicy Metadata: Creation Timestamp: 2025-02-04T06:08:02Z Generation: 1 Resource Version: 67404617 UID: f56b2bd2-3e0e-42a1-a9f3-d47b0b20d8e1 Spec: Default: Timeouts: Route Timeout: 0s Target Ref: Group: gateway.networking.k8s.io Kind: HTTPRoute Name: sseserver Namespace: default Status: Conditions: Last Transition Time: 2025-02-04T06:08:02Z Message: Valid RoutePolicy Observed Generation: 1 Reason: Accepted Status: True Type: Accepted Events: <none

Timo Notheisen 20 Reputation points

2025-02-05T08:54:57.0333333+00:00

This suggests that the timeout might be overridden, possibly due to an older configuration with a default timeout of 60 seconds

There is no other RoutePolicy anywhere in the whole cluster, if that's what you mean. And I'm not aware of any other way to configure a timeout.

could you please provide some information on how you set up the SSE locally?

I just updated my question with a link to a code example, which should answer your question.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Server-sent events and Azure Application Gateway for Containers

1 answer

Your answer