EnvelopedCMS throws an "ASN1 out of memory" exception with files larger than 5 MB
Hi all,
When using .NET's EnvelopedCMS to decode a file that is larger than 5 MB, you may get the following exception on Windows 7/Server 2008 R2:
Unhandled Exception: System.Security.Cryptography.CryptographicException: ASN1 out of memory.
at System.Security.Cryptography.Pkcs.EnvelopedCms.OpenToDecode(Byte[] encodedMessage)
at System.Security.Cryptography.Pkcs.EnvelopedCms.Decode(Byte[] encodedMessage)
This simple code reproduces the issue:
byte[] encryptedContent = File.ReadAllBytes(@"c:\EncryptedFile.test");
EnvelopedCms cms = new EnvelopedCms();
cms.Decode(encryptedContent);
cms.Decrypt();
Issue won't happen with all files. Additionally, issue won't happen on e.g. Windows XP with the same code and file.
I’ve seen similar issues in the past when dealing with big CMS files, with both EnvelopedCMS and SignedCMS, and I solved them with this sample:
How to call CryptMsg API in streaming mode - LargeCMS full sample
More info:
"ASN1 value too large" error when calling SignedCms.ComputeSignature
Now, in the posts above, the error was similar but not the same: "ASN1 value too large". In this case the issue is a bit different, as the problematic files are not too big (only 5MB or so). It seems to be related to the contents of the file itsef. In any case, the above sample worked on this scenario, too.
With the workaround at hand, I kept investigating and found that this issue is actually a bug on Windows! Fortunatelly we just released a fix for it:
Note that this fix goes on top of Windows 7 SP1.
I hope this helps.
Cheers,
Alex (Alejandro Campos Magencio)