SSL Security error [DBNETLIB]ConnectionOpen (SECDoClientHandshake())
This particular resolution is one of its own kind their could have been other possibilites for his error message to raise, generally it comes if you have Expired Certificates or invalid Certificates on the Server for the user account or the computer account, delete the unwanted Certificates on the server and then try to start SQL or Connect should work
Other times this message comes if you had encryption enabled for the Server sometime but no more now, but then internally we are still finding the certificate somehow, though you anyways dont need the SSL certificate, so go to the following Registry and check for the Certificate value it should be EMPTY, but if it has ThumbPrint value of the Certifcate it is a problem, go ahead and make the change for Certificate value set to 0
HKLM\SOFTWARE\Microsoft\MSSQLServer\MSSQLServer\SuperSocketNetLib REG_SZ Certificate=0
Coming back to this unique problem:
Problem Description:
=================
Trying to enable SSL encryption for sql server connections on server
Went through the steps detailed in article 276553 (https://support.microsoft.com/?id=276553).
From one client we were unable to connect to the SQL Server 2000 using encrypted connections {forced encryption from the Client configuration, this client box where its working is a Windows 2003 SP1 box}. From our own desktop workstation {Windows XP sp2}, along with that of one of the other Clients box {Windows 2000} they are unable to connect to the SQL Server through encrypted connections.
Consistently receive the following error:
[DBNETLIB]SSL Security error[DBNETLIB]ConnectionOpen (SECDoClientHandshake()).
We have checked and re-checked and re-re-checked that his local workstation trusts the same root certificate authority (plus all intermediary issuing authorities) as does the SQL Server
The certificate chains are identical on both machines.
We Enabled odbc tracing and attempted to login again from his local box. The log of that attempt is enclosed. It contains the following information:
osql -Sact-dd01 2764-1a88 EXIT SQLDriverConnectW with return code -1
(SQL_ERROR) HDBC 009A1830 HWND 00000000 WCHAR * 0x74329A38 [ -3] "******\ 0" SWORD -3 WCHAR * 0x74329A38 SWORD 2 SWORD * 0x00000000 UWORD 0
DIAG [08001] [Microsoft][ODBC SQL Server Driver][DBNETLIB]SSL Security error (18)
DIAG [01000] [Microsoft][ODBC SQL Server Driver][DBNETLIB]ConnectionOpen (SECDoClientHandshake()). (271)
Operating System:Windows 2003 : where the Encrypted Client connections are working
Operating System: Windows XP Sp2 and Windows 2000 SP4 where the Client connections are not working using Encryption, though if the encryption is not enabled the connections will flow fine
Cause
=====
The cause is a DBNETLIB.DLL {Remember this is not a COM DLL}, the version compatibility of this DLL on the XP and Windows 2000 is the possible culprit, i dont think it is a BUG
Resolution
========
1) Check the for the following DLL's version on the Windows 2003 SP1 or SP2 Box
C:\WINDOWS\system32\DBNETLIB.DLL
Version: 2000.86.1830.0 {Windows 2003 SP1}
2) Now Check the Version for the same DLL on the Windows XP Sp2 and Windows 2000 Sp4 box . In My case the Windows XP SP2 had the following version of the DBNETLIB.DLL Version: 2000.85.1117.0
3) The Windows 2000 SP4 box had the follwoing DLL version in my case Version: 2000.85.1064.0
4) So what i did was, i copied the DLL Version: 2000.86.1830.0 from the Windows 2003 SP1 box to the Windows Xp SP2 box and also to the Windows 2000 SP4 Box
Went to the following location on the Windows XP Sp2 box:
C:\windows\system32 and renamed the existing DBNETLIB.DLL to DBNETLIB.DLL.OLD
After that pasted the copied DLL version 2000.86.1830.0 on the XP Sp2 box in the following location c:\windows\system32\DBNETLIB.DLL {2000.86.1830.0} after that did a refresh and saw that the Version got back to the older version that is 2000.85.1117.0. Made a check in the DLLCACHE {c:\windows\system32\DLLCACHE} and saw that the DBNETLIB.DLL was present there and was having the Version 2000.85.1117.0 {so whenever the connection was made, this DLL from the cache was picked}, hence First paste the DBNETLIB.DLL {with version 2000.86.1830.0} in the DLLCACHE and then replace the actual DLL inside the c:\windows\system32.
Do a refresh and see if the DLL version is Showing as 2000.86.1830.0 on the XP Sp2 Box.
After this again try to connect to the SQL Server from this Client box {Make sure that Force Encryption is Checked in the Client Configuration}
The above resolved the issue with the SSL encryption errors, and the Secured channel is established
5) Repeat the above Steps for a Windows 2000 SP4 box, and this should help in getting past through the problem
NOTE: Never try to Register this DLL, also it wont let you as well, as it is not a COM DLL, also replacing this DLL will never break anything as it is a System DLL and the version is upgraded through MDAC