CryptDecryptMessage fails, returning an CRYPT_E_ASN1_MEMORY error value
 |
Other Resources Security Developer Center Cryptography Topics on MSDN Follow us on Twitter |
Code that called the CryptDecryptMessage function successfully under previous versions of Windows may fail with a CRYPT_E_ASN1_MEMORY error under Windows 7.
Cause
There is a counter in the ASN structure that is increased every time an octet block is processed when the ASN sequence that wraps the octets is BER encoded. It adds the number of bytes that have already been processed plus the new block size; as the number of processed bytes grows that is added to the counter and the rate at which the counter grows increases. Since the counter is represented by an integer it can overflow. On Windows 7, the ASN code checks for overflow so the function fails. On pre-Windows 7 platforms, the overflow still occurs but it wasn't checked. That counter is not critical for operation and did not cause issues in the code for the previous versions of the OS.
Resolution
Download the hotfix described in Microsoft Knowledge Base article 2480994, "ASN1 out of memory" error when the CryptMsgUpdate function decodes a message that is larger than 5 MB in Windows 7 or in Windows Server 2008 R2 (http://support.microsoft.com/kb/2480994).
You can also call the lower level message functions (CryptMsgOpenToDecode, CryptMsgUpdate, CryptMsgControl) to process BER encoded content.