Share via

CryptEncrypt and RSACryptoServiceProvider::Encrypt

  • Article

The RSACryptoServiceProvider class provides two methods, Encrypt and Decrypt which seem to be the managed counterparts to CAPI's CryptEncrypt and CryptDecrypt functions.  However, if you try to encrypt using CAPI and decrypt using managed code, you'll end up with a CryptographicException saying "bad data".  What's going on here?

Since v1.0 of the runtime, the output from RSACryptoServiceProvider::Encrypt has been reversed in memory.  Input to RSACryptoServiceProvider::Decrypt is also swapped before it's processed.  Since the two reversals undo each other, using the RSA class to round trip data works perfectly.

However, if you use CAPI for one of the steps, the memory reverse won't undo itself, and the output won't contain valid padding -- in this case leading to a bad data exception.  The solution is to introduce the reversal yourself.  To interop, you'd need to either:

  1. CryptEncrypt
  2. Reverse
  3. RSACryptoServiceProvider::Decrypt

Which gets the output from CAPI setup so that Decrypt's own memory reverse puts the bytes in a state that they can be decrypted.  Going the other direction would of course be:

  1. RSACryptoServiceProvider::Encrypt
  2. Reverse
  3. CryptDecrypt

The Reverse in this direction undoes the reversal that Encrypt does.  Reversing the memory can be done with a call to Array.Reverse();

Why not change the behavior of Encrypt and Decrypt now?  Doing that would prevent people using the same code on different versions of the runtime from being able to interop with each other.  Even worse, if you wrote some data out to a file while your app was running on a version of the runtime that had the old behavior, and before your app started up again the computer admin uninstalled the old version of the runtime and installed a new version which didn't have this behavior, when your app started back up it wouldn't be able to read its file anymore.

Instead, when writing code that needs to interact with CryptEncrypt and CryptDecrypt, you need to remember to apply this transform to your data in order for your code to work properly.

Updated 11:19: Added a reference to Array.Reverse

Comments

  • Anonymous
    December 05, 2005
    Know your libraries.
    Use System.Array.Reverse http://msdn.microsoft.com/library/en-us/cpref/html/frlrfsystemarrayclassreversetopic.asp

    It's two times faster and bug-free.

  • Anonymous
    December 05, 2005
    Thanks TAG :-) Speed in this case is probably not a huge factor, considering that the maximum size of this array is on the order of 100 or so bytes, but I agree using a built in method is better.

    -Shawn

  • Anonymous
    December 07, 2005
    Was the byte reversal a design decision or just a bug that slipped through in v1?

  • Anonymous
    December 12, 2005
    I'm not sure -- I wasn't involved in crypto in the v1.0 timeframe, and everyone who was has moved on at this point.

    -Shawn

  • Anonymous
    December 12, 2005
    The comment has been removed

  • Anonymous
    December 12, 2005
    The comment has been removed

  • Anonymous
    January 11, 2006
    I think we might be having a problem with this.

    We are finding an XML document signed with an RSA key in .NET 1.1 no longer verifies in .NET 2.0. Further investigation revealed that the .NET 1.1 signing was wrong.

    Is it possible this is being caused by what you are describing is due to RSACryptoProvider reversing some data somewhere? If so, interopobility so we can check signatures on documents signed in .NET 1.1 would be helpful!

  • Anonymous
    January 12, 2006
    Nick,

    I don't think this is the root of your problem. This issue aaffects the Encrypt method only. In XML DSig we use the RSAPKCS1SignatureFormatter to create the signature, and that relys on the SignHash method.

    -Shawn

  • Anonymous
    March 31, 2010
    I spent 3 days trying to figure out whey decryption of CryptEncrypt output is not working I founded by chance when compared the data (byte by byte) Then I stopped in your blog. I felt upset. Such an important issue should be clearly mentioned in the API/Framework documentaion what a mess!!!!

  • Anonymous
    August 16, 2010
    Can somebody send to me an compatibility example with C# and win32 source code? We have an issue and are trying to resolve. Thanks! Ruslan.Tomashevsky@aricent.com