Azure Service Bus: cannot allocate more handles, are there any metrics and how to solve this?

Question

Past Saturday (July 27) in the morning, our software stopped working because of an azure service bus error.
Cannot allocate more handles. The maximum number of handles is 4999

The sender is a dotnet C# api that uses ServiceBusClient / ServiceBusSender to send the message. ServiceBusClient is retrieved using Dependency Injection.
The servicebus sender is created in the constructor and then the message is send in the [HttpPost] function.
I have now moved the creation of the sender to the [HttpPost] function and call DisposeAsync() after sending the message. This made the service healthy again, although deploying an update causes the appservice to restart which would likely have fixed it too. So I am unsure if this really fixes anything.

The servicebusclient is added to dependency injection using the following code.

var serviceBusNamespace = builder.Configuration["Settings:ServiceBusNamespace"];

builder.Services.AddAzureClients(builder =>
{
    builder.AddServiceBusClientWithNamespace(serviceBusNamespace);
    builder.UseCredential(new DefaultAzureCredential(true));
});

Metrics

• When looking at the 'ActiveConnections' metric, aggregation 'Max'. I always see only 1 connection active. Max Connections Closed/Opened metrics always show 0 or 1.
• There was a peak of 124 incoming requests at july 26 5:15PM, but after that it was at most 5 until the service went down the following morning.

I can not find a metric that shows us coming near the 5000 handles.

Architecture

Sender: Dotnet api running on Premium v3 plan (P1v3), single instance
Using Azure.Messaging.ServiceBus 7.17.5. (I will update this to 7.18.0.)

Receiver: Azure functions app running on the same P1v3 plan
Using Microsoft.Azure.Functions.Worker.Extensions.ServiceBus 5.20.0

ServiceBus: Basic, containing 2 queues (both use the same sender/receiver apps)

Recent changes

In the week before this issue happened, we added a healthcheck. All this does is creating a receiver calling PeekMessageAsync().
After the error occured I also added DisposeAsync to the healthcheck.

Questions

• Am I supposed to call DisposeAsync after sending/peeking a message? (The example code doesn't show this).
• How can I see how many handles I am currently using?
• What can I do to avoid this in the future?

Answer 1

I have now tested this locally with code that does call DisposeAsync and code that doesn't afterwards.

Without DisposeAsync, you will eventually get this error. So I assume using DisposeAsync is the correct usage. (I thought I've read somewhere that ServiceBusClient will handle this and return the same receiver/sender if there was one already created with the same queueName, but that doesn't seem to be the case so I probably misunderstood that part).

Therefore my code change already ensures this won't happen in the future, but it would be nice to have a metric that shows amount of handles in use so this is easier to debug from azure portal and allows me to add an azure-alert to it if we get close to that amount.

Answer 2

We're seeing this problem as well in .NET 8 Isolated Functions. We're injecting a ServiceBusClient and using it to create a class variable ServiceBusReceiver.
I had assumed that this was reused or disposed by the framework but it seems that might not be the case.

I will see if your solution works for us as well.