CryptSignHash fails with NTE_BAD_KEYSET

Hi all,

The other day I worked on the following issue: a customer of mine had developed an ActiveX which they used to sign some data in Internet Explorer. Their signing code was based on CryptoAPI and worked just fine with some certs, but failed with others with a NTE_BAD_KEYSET error.

So we took some traces with my CryptoAPI Tracer script to see which API was returning the error exactly.

This is the API that failed with NTE_BAD_KEYSET error:

 CryptSignHashA (0x1f2c)

IN
hHash
0x3346208

dwKeySpec
AT_KEYEXCHANGE

sDescription
NULL

dwFlags
0

pbSignature
NULL

dwSigLen
46500472

OUT
dwSigLen
46500472

RESULT
CryptSignHashA (0x1f2c) FAILED
LastErrorValue: (HRESULT) 0x80090016 (2148073494) -  Keyset does not exist
LastStatusValue: (NTSTATUS) 0xc0000034 - Object Name not found

My post CryptAcquireContext fails with NTE_BAD_KEYSET refers to CryptAcquireContext API, but the causes of the error should be the same: keys don't exist or we have no permissions to access them.

We can see above that we are trying to sign with AT_KEYEXCHANGE keys.

CryptSignHash Function
"
dwKeySpec [in]
Identifies the private key to use from the provider's container. It can be AT_KEYEXCHANGE or AT_SIGNATURE.
The signature algorithm used is specified when the key pair is originally created.
"

Summing up, we may have two public/private key pairs: AT_KEYEXCHANGE and AT_SIGNATURE. When we generate the keys, we specify which ones we are creating.

CryptGenKey Function
"
In addition to generating session keys for symmetric algorithms, this function can also generate public/private key pairs. Each CryptoAPI client generally possesses two public/private key pairs. To generate one of these key pairs, set the Algid parameter to one of the following values.
Value Meaning
AT_KEYEXCHANGE Key exchange
AT_SIGNATURE Digital signature
"

When we enroll a cert, keys of a given type get created and associated to it. We can use certutil utility on a cert (i.e. "certutil -dump -v cert.pfx") to check if its KeySpec is AT_KEYEXCHANGE or AT_SIGNATURE. In this particular case, working certs have AT_KEYEXCHANGE KeySpec, and failing certs have AT_SIGNATURE.

So we are trying to sign with AT_KEYEXCHANGE keys but the certificates only have AT_SIGNATURE keys.

 

When we get the CSP of our certificate via API, we can find out which kind of keys are associated to it:

CryptAcquireCertificatePrivateKey
"
pdwKeySpec [out]
The address of a DWORD variable that receives additional information about the key. This can be one of the following values.
Value Meaning
AT_KEYEXCHANGE The key pair is a key exchange pair.
AT_SIGNATURE The key pair is a signature pair.
"

The following sample shows how to use CryptAcquireCertificatePrivateKey to get the CSP of a certificate and its Keyspec, and sign with it:

How to sign and verify with CryptoAPI and a user certificate

Note: in this sample I only sign with AT_SIGNATURE. We should modify the sample to get the right KeySpec with CryptAcquireCertificatePrivateKey and pass it to CryptSignHash so it uses the right keys to sign.

I hope this helps.

Regards,

 

Alex (Alejandro Campos Magencio)