I got an transient error System.Threading.Tasks.TaskCanceledException: A task was canceled. while my service bus consumer listens for subscription

Question

Hi everybody. Using C# / .Net (net8.0). I've implemented my consumer over Azure.Messaging.ServiceBus. (Version=7.18.2)

The consumer is a part of hosted service. Initialization looks like :

_client = new ServiceBusClient(_connectionString, new ServiceBusClientOptions
{
    RetryOptions = new ServiceBusRetryOptions
    {
        TryTimeout = TimeSpan.FromSeconds(60),
        Mode = ServiceBusRetryMode.Exponential,
        MaxRetries = 6,
        MaxDelay = TimeSpan.FromSeconds(10)
    }
});
    

_processor = _client.CreateProcessor(
    _topicName,
    _cmSubscriptionName,
    new ServiceBusProcessorOptions
    {
        AutoCompleteMessages = false,
        ReceiveMode = ServiceBusReceiveMode.PeekLock,
        MaxConcurrentCalls = 1
    });

_processor.ProcessMessageAsync += OnMessageReceived;
_processor.ProcessErrorAsync += OnError;

await _processor.StartProcessingAsync(cancellationToken);

In OnMessageReceived I implemented a message processing with raising event for subscribers.

The issue that I'm constantly seeing in Azure Dashboard>Investigate>Failures is an error as below:

System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
   at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.ReceiveMessagesAsyncInternal(Int32 maxMessages, Nullable`1 maxWaitTime, TimeSpan timeout, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.<>c.<<ReceiveMessagesAsync>b__44_0>d.MoveNext()
--- End of stack trace from previous location ---
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken, Boolean logTimeoutRetriesAsVerbose)
   at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken, Boolean logTimeoutRetriesAsVerbose)
   at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.ReceiveMessagesAsync(Int32 maxMessages, Nullable`1 maxWaitTime, CancellationToken cancellationToken)
   at Azure.Messaging.ServiceBus.ServiceBusReceiver.ReceiveMessagesAsync(Int32 maxMessages, Nullable`1 maxWaitTime, Boolean isProcessor, CancellationToken cancellationToken)

I've added detailed logging for each step in OnMessage received and OnError - there were no incoming messages during time period mentioned in Azure Dashboard>Investigate>Failures.

I've carefully look through the similar issues and found this one https://learn.microsoft.com/en-us/answers/questions/990564/azure-service-bus-receive-message-fails-with-follo. This answer suggest to wrap all risky places with try catch and log the issues. This doesn't work for me. Could someone advice what might be wrong? if this is a kind if internal AzureSdk logger, can I somehow turn this error logging off to not spam me with issues I can't resolve?