PRB: SQL Server Browser service does not start due to Winsock Monitoring Software
Issue
During an upgrade from SQL 2005 to SQL 2008 Express, the SQL Browser service failed to start, thus causing the upgrade to fail. We saw the following errors being reported in detail.txt
2010-07-23 18:49:17 SQLBrowser: SQL Server Browser Install for feature 'SQL_Browser_Redist_SqlBrowser_Cpu32' generated exception, and will invoke retry option. The exception: Microsoft.SqlServer.Configuration.Sco.ScoException: Service 'SQLBrowser' start request failed.
at Microsoft.SqlServer.Configuration.Sco.Service.StartServiceWait(String startParameters, Int32 dwTimeOut)
at Microsoft.SqlServer.Configuration.Sco.Service.StartServiceWait(Int32 dwTimeOut)
During the upgrade the new files are copied over and then the services are started. So sqlbrowser.exe was already on SQL 2008 build 2007.100.1600. 22.
At the same time we observed the following events being logged in the application event logs.
Warning,7/26/2010 6:20:23 PM,SQLBrowser,3,None,The configuration of the AdminConnection\TCP protocol in the SQL instance SQLEXPRESS is not valid.
"The SQLBrowser service port is unavailable for listening, or invalid"
"The SQLBrowser service was unable to establish SQL instance and connectivity discovery"
Troubleshooting
Looking at these messages I suspected two things:
1) Some other application/service is blocking the udp port 1434
2) Filter drivers/AV is preventing browser from using the port.
Here are some of the things I checked:-
1. Checked the service account for SQLBrowser and it was set to LocalSystem. Changing to a Domain Account (same one being used by the SQL service) did not help.
2. It could be an issue with UDP Port 1434 since that is what sqlbrowser listens on. To give you a background, ever since SQL 2000 introduced support for multiple instances on the same machine, the SSRP (SQL Server Resolution Protocol) was developed to listen on UDP Port 1434. Now, SQL Browser replaced SSRP and became a service, but it still uses the same port.
Netstat –abn showed that no-one was using port 1434 and it was free
3. Checked the value of the following registry key
HKLM\Software\Microsoft\Microsoft SQL Server\IsListenerActive
If the value is YES, it means that a SQL 2000 instance is present and it will use the port. But that was no true in this case.
After all this turned out un-fruitful, I collected a dump of sqlbrowser.exe using Debug Diag and loaded the dump using WinDbg. Once I had it loaded, it looked at the list of modules loaded (lm) with the address space of sqlbrowser.exe and found these.
ModLoad: 02b10000 02b1a000 c:\program files\microsoft sql server\100\shared\instapi10.dll
ModLoad: 10000000 10024000 c:\windows\system32\kaseyasp.dll <----------3rd party Module loaded
ModLoad: 75630000 75635000 c:\windows\system32\wshtcpip.dll
ModLoad: 75a10000 75a4b000 c:\windows\system32\mswsock.dll
ModLoad: 75a80000 75a85000 c:\windows\system32\wship6.dll
ModLoad: 76410000 76417000 c:\windows\system32\psapi.dll
ModLoad: 76510000 765d8000 c:\windows\system32\msctf.dll
ModLoad: 77330000 773ad000 c:\windows\system32\usp10.dll
ModLoad: 77610000 77619000 c:\windows\system32\lpk.dll
ModLoad: 777b0000 777ce000 c:\windows\system32\imm32.dll
ModLoad: 77ca0000 77d3d000 c:\windows\system32\user32.dll
ModLoad: 77eb0000 77efb000 c:\windows\system32\gdi32.dll
What first caught my eye was to see something that I did not expect to see in a regular sqlbrowser image, i.e. kaseyasp.dll. This is not a Microsoft DLL and is not something that I would expect to see loaded inside SQL Browser.
I got more information on this module.
0:000> lmvm kaseyasp
start end module name
10000000 10024000 kaseyasp (deferred)
Image path: c:\windows\system32\kaseyasp.dll
Image name: kaseyasp.dll
Timestamp: Sat Mar 08 00:47:26 2008 (47D194C6)
File version: 5.0.0.0
Product version: 5.0.0.0
CompanyName: Kaseya
ProductName: Kaseya Agent Internet Access Protection Extension
InternalName: KaseyaSP
OriginalFilename: KaseyaSP.dll
ProductVersion: 5,0,0,0
FileVersion: 5,0,0,0
FileDescription: Kaseya Agent Extension
LegalCopyright: Copyright © 2001-2008 by Kaseya, All Rights Reserved
LegalTrademarks: Kaseya Virtual System Administrator(tm)
So it looks like my suspicion #2 was turning out to be right. I then checked the machine under services.msc and bingo I found Kaseya Agent service running. Now, since this was loaded inside SQLBrowser it looks to be a filter driver of some kind.
A quick Bing search told me that this piece of software is a Winsock Layered Service Provider that allows monitoring of network traffic on a machine. It has the ability to intercept and monitor network communications (including TDS packets) happening at the WinSock level between the client applications and SQL Server. Now, any 3rd party filter drives loading into the SQL Server process and intercepting our calls is not supported.
The use of third-party detours or similar techniques is not supported in SQL Server
https://support.microsoft.com/kb/920925
Solution
So, here are the options that I had:-
1. Disable Kaseya Agent using MSCONFIG and reboot the server.
2. Uninstall the Kaseya Agent Software on the server and reboot.
Why do we need a reboot?
Sometimes a filter-driver never fully gets un-loaded until a system reboot is done. To be sure, I got this right, a reboot was required.
After uninstalling the Kaseya Agent software, we re-ran the upgrade and SQLBrowser started successfully and the upgrade went smooth.
Just sharing this issue so that anyone out there who encounters something similar with SQL Browser will know a few things to check and this issue might just be the one in your case. Happy Troubleshooting!
Regards,
Sudarshan Narasimhan
Technical Lead, Microsoft SQL Server CSS