Always On Instance Service Broker/Database Mirroring transport connection endpoint Error

Question

I have a production instance that has 2 AOG. There is no mirroring configured for both groups. Both groups are also in 'Healthy' state.

Upon checking the error log on SQL Server, I noticed that starting from 12/19/20 there has been this error that started at 9:03PM. I went through an dug in further to see that there is a pattern. Every night around 9:03-9:04pm the same error is happening. This does not occur during normal business hours.

Error msg:

An error occurred in a Service Broker/Database Mirroring transport connection endpoint, Error: 8474, State: 11. (Near endpoint role: Target, far endpoint address: '')`

My thought was to stop and restart the endpoints, but I don't think that would be necessary in this case.

Anyone have any thoughts as to why it might be happening on a daily basis around the same time, and how I may be able to solve this error?

Thanks in advance.

Answer 1

Error 8474, specifically state 11 means that the service broker message was corrupt in some way, shape, or form. The initial reaction would be to think there was an issue on the network and to be honest it's possible, but to pass TCP checksums and be delivered to the port without having a different error is suspect.

Instead, this is most likely a network scanner or exploit checker, I wrote about this here. Normally there are other errors along with it on the TDS endpoint, though that may have been whitelisted in your environment. The fact it happens every day around the same time is suspect. You can check the ringbuffer connectivity logs or setup some network packet capturing since you know the timeframe and check the source system sending the packets. I'm sure it'll go back to a "security" server.

Edit: Please note that the use of "Service Broker" here is the ENDPOINT (Database Mirroring, etc.) than the service broker feature inside of SQL Server that can handle async processing with reliable messaging and routing.