NCryptSignHash returns NTE_INVALID_PARAMETER when signing a SHA512 hash

Question

Hello,

I've been updating a program to use CNG but have run into a problem where NCryptSignHash returns NTE_INVALID_PARAMETER when I ask it to sign a SHA512 hash. If I change the hashing algorithm to MD5, SHA1, or SHA256 then the function succeeds fine. I've created a minimal program showing this behavour which contains a sample private key and will attempt to sign the hash of the "Hello, World!" text.

#include <windows.h>
#include <vector>

auto pem =
"-----BEGIN PRIVATE KEY-----"
"MIIBVAIBADANBgkqhkiG9w0BAQEFAASCAT4wggE6AgEAAkEAqPfgaTEWEP3S9w0t"
"gsicURfo+nLW09/0KfOPinhYZ4ouzU+3xC4pSlEp8Ut9FgL0AgqNslNaK34Kq+NZ"
"jO9DAQIDAQABAkAgkuLEHLaqkWhLgNKagSajeobLS3rPT0Agm0f7k55FXVt743hw"
"Ngkp98bMNrzy9AQ1mJGbQZGrpr4c8ZAx3aRNAiEAoxK/MgGeeLui385KJ7ZOYktj"
"hLBNAB69fKwTZFsUNh0CIQEJQRpFCcydunv2bENcN/oBTRw39E8GNv2pIcNxZkcb"
"NQIgbYSzn3Py6AasNj6nEtCfB+i1p3F35TK/87DlPSrmAgkCIQDJLhFoj1gbwRbH"
"/bDRPrtlRUDDx44wHoEhSDRdy77eiQIgE6z/k6I+ChN1LLttwX0galITxmAYrOBh"
"BVl433tgTTQ="
"-----END PRIVATE KEY-----";

auto data = "Hello, World!";

auto algId = NCRYPT_SHA512_ALGORITHM;

int main(int argc, char* argv[])
{
    // Load Private Key
    auto derKeyLength = DWORD{ 0 };
    CryptStringToBinaryA(pem, 0, CRYPT_STRING_BASE64HEADER, nullptr, &derKeyLength, nullptr, nullptr);
    auto derKey = std::vector<uint8_t>(derKeyLength);
    CryptStringToBinaryA(pem, 0, CRYPT_STRING_BASE64HEADER, derKey.data(), &derKeyLength, nullptr, nullptr);

    auto provider = NCRYPT_PROV_HANDLE();
    NCryptOpenStorageProvider(&provider, MS_KEY_STORAGE_PROVIDER, 0);
    auto key = NCRYPT_KEY_HANDLE();
    NCryptImportKey(provider, 0, NCRYPT_PKCS8_PRIVATE_KEY_BLOB, nullptr, &key, derKey.data(), derKey.size(), 0);

    // Hash Data
    auto dummy = DWORD{ 0 };

    auto alg = BCRYPT_ALG_HANDLE();
    BCryptOpenAlgorithmProvider(&alg, algId, nullptr, 0);

    auto objLength = DWORD{ 0 };
    BCryptGetProperty(alg, BCRYPT_OBJECT_LENGTH, reinterpret_cast<PUCHAR>(&objLength), sizeof(DWORD), &dummy, 0);

    auto obj  = std::vector<uint8_t>(objLength);
    auto hash = BCRYPT_HASH_HANDLE();
    BCryptCreateHash(alg, &hash, obj.data(), obj.size(), nullptr, 0, 0);

    auto hashLength = DWORD{ 0 };
    BCryptGetProperty(alg, BCRYPT_HASH_LENGTH, reinterpret_cast<PUCHAR>(&hashLength), sizeof(DWORD), &dummy, 0);

    BCryptHashData(hash, reinterpret_cast<PUCHAR>(const_cast<char*>(data)), strlen(data), 0);

    auto hashBuffer = std::vector<UCHAR>(hashLength);
    BCryptFinishHash(hash, hashBuffer.data(), hashBuffer.size(), 0);

    // Sign Hash
    auto padding = BCRYPT_PKCS1_PADDING_INFO{ algId };
    auto status  = 0;

    auto signatureLength = DWORD{ 0 };
    status = NCryptSignHash(key, &padding, hashBuffer.data(), hashBuffer.size(), nullptr, 0, &signatureLength, BCRYPT_PAD_PKCS1);

    auto signature = std::vector<UCHAR>(signatureLength);
    status = NCryptSignHash(key, &padding, hashBuffer.data(), hashBuffer.size(), signature.data(), signature.size(), &signatureLength, BCRYPT_PAD_PKCS1);

    return 0;
}

In the above sample the second NCryptSignHash call fails with NTE_INVALID_PARAMETER when algId is set to NCRYPT_SHA512_ALGORITHM. My only guess is its something related to padding or alignment? I've done some googling but haven't come up with anything so far. Any advice very welcome!

Thanks, Aidan.

Answer 1

Hello Aidan,

Your sample private key is a 512 bit RSA key. A single encryption operation (without chaining of some sort) can only encrypt 512 bits of plaintext data. By the time ASN.1 encoding and PKCS #1 padding have been applied to the hash data to be signed, the length of data that needs to be encrypted has grown to 94 bytes (752 bits) - too big for a single RSA encryption operation.

Increase your RSA key length to 1024 or larger and you should find that your code works.

BTW, SHA384 produces a "to be signed" block of 78 bytes (624 bits), which is also too long. An SHA256 hash is the longest SHA family hash that can be signed with a 512 bit RSA key.

Gary