Share via

Don't Roundtrip Ciphertext Via a String Encoding

  • Article

One common mistake that people make when using managed encryption classes is that they attempt to store the result of an encryption operation in a string by using one of the Encoding classes.  That seems to make sense right?  After all, Encoding.ToString() takes a byte[] and converts it to a string which is exactly what they were looking for.  The code might look something like this:

    public static string Encrypt(string data, string password)
    {
        if(String.IsNullOrEmpty(data))
            throw new ArgumentException("No data given");
        if(String.IsNullOrEmpty(password))
            throw new ArgumentException("No password given");

        // setup the encryption algorithm
        Rfc2898DeriveBytes keyGenerator = new Rfc2898DeriveBytes(password, 8);
        Rijndael aes = Rijndael.Create();
        aes.IV = keyGenerator.GetBytes(aes.BlockSize / 8);
        aes.Key = keyGenerator.GetBytes(aes.KeySize / 8);

        // encrypt the data
        byte[] rawData = Encoding.Unicode.GetBytes(data);
        using(MemoryStream memoryStream = new MemoryStream())
        using(CryptoStream cryptoStream = new CryptoStream(memoryStream, aes.CreateEncryptor(), CryptoStreamMode.Write))
        {
            memoryStream.Write(keyGenerator.Salt, 0, keyGenerator.Salt.Length);
            cryptoStream.Write(rawData, 0, rawData.Length);
            cryptoStream.Close();

            byte[] encrypted = memoryStream.ToArray();
            return Encoding.Unicode.GetString(encrypted);
        }
    }

    public static string Decrypt(string data, string password)
    {
        if(String.IsNullOrEmpty(data))
            throw new ArgumentException("No data given");
        if(String.IsNullOrEmpty(password))
            throw new ArgumentException("No password given");

        byte[] rawData = Encoding.Unicode.GetBytes(data);
        if(rawData.Length < 8)
            throw new ArgumentException("Invalid input data");
        
        // setup the decryption algorithm
        byte[] salt = new byte[8];
        for(int i = 0; i < salt.Length; i++)
            salt[i] = rawData[i];

        Rfc2898DeriveBytes keyGenerator = new Rfc2898DeriveBytes(password, salt);
        Rijndael aes = Rijndael.Create();
        aes.IV = keyGenerator.GetBytes(aes.BlockSize / 8);
        aes.Key = keyGenerator.GetBytes(aes.KeySize / 8);

        // decrypt the data
        using(MemoryStream memoryStream = new MemoryStream())
        using(CryptoStream cryptoStream = new CryptoStream(memoryStream, aes.CreateDecryptor(), CryptoStreamMode.Write))
        {
            cryptoStream.Write(rawData, 8, rawData.Length - 8);
            cryptoStream.Close();

            byte[] decrypted = memoryStream.ToArray();
            return Encoding.Unicode.GetString(decrypted);
        }
    }
}

The first mistake some people make is to use ASCII encoding.  This will nearly always fail to work since ASCII is a seven bit encoding, meaning any data that is stored in the most significant bit will be lost.  If your cipherdata can be guaranteed to contain only bytes with values less than 128, then its time to find a new encryption algorithm :-)

So if we don't use ASCII we could use UTF8 or Unicode right?  Those both use all eight bits of a byte.  In fact this approach tended to work with v1.x of the CLR.  However a problem still remains ... just because these encodings use all eight bits of a byte doesn't mean that every arbitrary sequence of bytes represents a valid character in them.  For v2.0 of the framework, the Encoding classes had some work done so that they explicitly reject illegal input sequences (As the other Shawn-with-a-w discusses here).  This leads to bad code that used to work (due to v1.1 not being very strict) to start failing on v2.0 with exceptions along the line of:

Unhandled Exception: System.Security.Cryptography.CryptographicException: Length of the data to decrypt is invalid.
   at System.Security.Cryptography.RijndaelManagedTransform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount)
   at System.Security.Cryptography.CryptoStream.FlushFinalBlock()
   at System.Security.Cryptography.CryptoStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()

Which at first glance looks like the CryptoStream is broken.  However, take a closer look at what's going on.  If we check the encrypted data before converting it into a string in Encrypt and compare that to the raw data after converting back from a string in Decrypt we'll see something along the lines of:

encrypted=array [72] { 111, 49, 30, 0, 29 .... }
rawData=array [68] { 111, 49, 30, 0, 8, ... }

So round tripping through the Unicode encoding caused our data to become corrupted.  That the decryption didn't work due to having an incomplete final block is actually a blessing -- the worst case scenario here is that you end up with some corrupted ciphertext that can still be decrypted -- just to the wrong plaintext.  That results in your code silently working with corrupt data.

You might not see this error all the time either, sometimes you might get lucky and have some ciphertext that is actually valid in the target encoding.  However, eventually you'll run into an error here so you should never be using the Encoding classes for this purpose.  Instead if you want to convert the ciphertext into a string, use Base64 encoding.  Replacing the two conversion lines with:

    byte[] encrypted = memoryStream.ToArray();
    return Convert.ToBase64String(encrypted);

    byte[] rawData = Convert.FromBase64String(data);

Results in code that works every time, since base 64 encoding is guaranteed to be able to accurately represent any input byte sequence.

Comments

  • Anonymous
    November 20, 2005
    The comment has been removed

  • Anonymous
    November 22, 2005
    Hi Robin,

    I haven't seen that problem before. Do you have repro code available?

    -Shawn

  • Anonymous
    December 01, 2005
    @Robin --> I tried the code above and it works fine. Did you add the:

    byte[] encrypted = memoryStream.ToArray();
    return Convert.ToBase64String(encrypted);


    byte[] rawData = Convert.FromBase64String(data);


    changes ?

  • Anonymous
    December 12, 2005
    THANKYOU!!

  • Anonymous
    February 01, 2006
    Flawless!

  • Anonymous
    February 18, 2006
    The comment has been removed

  • Anonymous
    February 22, 2006
    The encryption classes don't care about "special" characters, they only see a stream of bytes.  What problem are you seeing exactly?

    -Shawn

  • Anonymous
    February 23, 2006
    So I think I'm in bad shape.  My encryption function doesn't return the same value in ASP 2.0 as it does in 1.1.  I'm sure its because of the different behavior in Encoding classes.  What can I do?  I have WAY too much data encrypted under my "bad scheme".  I need a way to get the "old encoding methods" from 1.1, and include them in my project.  Any ideas?

    Thanks a bunch.  I'm in a pickle!

    Sam


    Public Function EncryptString(ByVal Source As String) As String

    Dim larrSourceData As Byte()
    Dim larrDestinationData As Byte()

    larrSourceData = Encoding.Unicode.GetBytes(Source)

    Call SetAESValues()
    larrDestinationData = _AESManaged.CreateEncryptor.TransformFinalBlock(larrSourceData, 0, larrSourceData.Length)



    Return Encoding.Unicode.GetString(larrDestinationData)


    End Function

  • Anonymous
    February 26, 2006
    Hi all,

    I have a problem with this code.
    When I encode two times the exactely same string I get a different encrypted string.

    Any help ?

  • Anonymous
    February 27, 2006
    Hi Sam,

    Your best bet is probably to bind old versions of your application to the v1.1 framework via an app.config file, and create a new version of your application which does not use the Encoding classes to store ciphertext.

    When you install the new version, you could have some sort of upgrade utility that is also bound to the v1.1 runtime and reads in the old data, writing it out in base 64.  Or you could have the new version of the application detect old data files and run the upgrade tool automatically.

    -Shawn

  • Anonymous
    February 27, 2006
    Hi Aleks,

    The fact that ciphertext differs does not mean that it's incorrect.  If you're using symmetric encryption, you should chekc that your key, IV, and padding mode are the same.  Asymmetric encryption will always have different output due to reandom padding.

    As long as you can round trip your data, you should be fine.

    -Shawn

  • Anonymous
    February 27, 2006
    OK Now I have it functionning (I hope). It was because of the "salt" that I didn't need.

    The problem is that, for some string, the decryption fail with the old method (without Base64) and the new one too (with Base64).
    Theses strings are passwords and I absolutely need to have it functionning as quick as possible.

    I can send some code by email I you'd like ...

    Thank you shawnfa

  • Anonymous
    November 15, 2006
    I have heard rumours that a certain type of implementation of AES (128bit) has been cracked  (in milliseconds rather than years). If this is true, how can we be sure that the Rijndael implementation within this Crypto namespace is not at risk. Just curious ;)

  • Anonymous
    December 17, 2006
    I hadn't seen anything about AES being cracked, so I'm not sure I can comment on RijndaelManaged :-) -Shawn

  • Anonymous
    December 29, 2006
    How can you store encrypted values in a database if they aren't converted into strings?

  • Anonymous
    January 02, 2007
    Hi John, You could store the encrypted byte array as a blob field in the database, or you can continue to store as a string.  However, when converting to a string do not use the Encoding classes, but instead use Convert.ToBase64String / Convert.FromBase64String.  This will create a string that can always be round-tripped back to the original byte array. -Shawn

  • Anonymous
    January 13, 2007
    Your encryption algorithm may fail moving to .NET 2.0

  • Anonymous
    February 15, 2007
    A different, more secure, Shawn , blogged " Don't Roundtrip Ciphertext Via a String Encoding ". I've

  • Anonymous
    March 31, 2007
    PingBack from http://fernandof.wordpress.com/2007/04/01/encryption-license-issues-when-upgrading-from-net-11-to-net-2/

  • Anonymous
    April 12, 2007
    When I decrypt a binary file I can not be opened! The PDF Document wheb decrypted is blank. Why? amirhussein@gmail.com

  • Anonymous
    April 13, 2007
    Okaaayy... So what if we don't want to use Base64? I'm trying to encrypt and store text that has commas, colons, etc -- more than just the letters and numbers that Base64 includes.

  • Anonymous
    April 13, 2007
    Hi Michelle, Base64 is just used to encode the ciphertext, you certainly do not need to limit your input to characters that appear in the base64 set.  In fact your input to the encryption algorithm doesn't even need to be a string at all. For instance (all hypothetical and not the real encodings): Plaintext: "Here-Is=Some:Plain, Text" Ciphertext: 0x12, 0x34, 0x56, 0x78, ... Cipertext to base64: abcdefg1234== The in the reverse Base64: abcdefg1234== Ciphertext from base64: 0x12, 0x34, 0x56, 0x78 ... Plaintext decrypted: "Here-Is=Some:Plain, Text" -Shawn

  • Anonymous
    April 27, 2007
    The comment has been removed

  • Anonymous
    April 30, 2007
    The comment has been removed

  • Anonymous
    May 07, 2007
    Hi Sergeda, You'll want to use Convert.ToBase64String() here, since you're trying to create a base64 string. -Shawn

  • Anonymous
    May 07, 2007
    Hi Jason, These lines of code jump out at me: byte[] initVectorBytes = Encoding.ASCII.GetBytes(InitVector); byte[] saltValueBytes  = Encoding.ASCII.GetBytes(Salt); are InitVector and Salt both real ASCII strings? -Shawn

  • Anonymous
    January 30, 2009
    The comment has been removed

  • Anonymous
    January 30, 2009
    FromBase64String takes a base64 string as input, not a plaintext string.  You're looking for ToBase64String to convert your "hello" string into base64.  (You'll also need to convert it to a byte array -- so something to the effect of Convert.ToBase64String(Encoding.UTF8.GetBytes("hello"))

  • Anonymous
    February 23, 2009
    Could someone please tell me if my code is suffering from the problem discussed here? I'm in a hurry and need to fix this ASAP. Here is my code:    Public Shared Function Encrypt(ByVal text As String, Optional ByVal additionalKey As String = "") As String        If text Is Nothing Then text = String.Empty        tripleDes.Key = TruncateHash(additionalKey & m_key, tripleDes.KeySize 8)        tripleDes.IV = TruncateHash("", tripleDes.BlockSize 8)        Dim plaintextBytes() As Byte = System.Text.Encoding.Unicode.GetBytes(text)        Dim ms As New System.IO.MemoryStream        Dim encStream As New CryptoStream(ms, tripleDes.CreateEncryptor(), System.Security.Cryptography.CryptoStreamMode.Write)        encStream.Write(plaintextBytes, 0, plaintextBytes.Length)        encStream.FlushFinalBlock()        encStream.Dispose()        Return Convert.ToBase64String(ms.ToArray)    End Function    Public Shared Function Decrypt(ByVal encryptedText As String, Optional ByVal additionalKey As String = "") As String        tripleDes.Key = TruncateHash(additionalKey & m_key, tripleDes.KeySize 8)        tripleDes.IV = TruncateHash("", tripleDes.BlockSize 8)        Dim encryptedBytes() As Byte = Convert.FromBase64String(encryptedtext)        Dim ms As New System.IO.MemoryStream        Dim decStream As New CryptoStream(ms, tripleDes.CreateDecryptor(), System.Security.Cryptography.CryptoStreamMode.Write)        decStream.Write(encryptedBytes, 0, encryptedBytes.Length)        Try            decStream.FlushFinalBlock()        Catch ex As Exception        Finally            decStream.Dispose()        End Try        Return System.Text.Encoding.Unicode.GetString(ms.ToArray) 'Convert.ToBase64String(ms.ToArray)    End Function Thank you really. I don't have the time to read the post carefully.

  • Anonymous
    March 30, 2009
    That's solve my problem Thanks!!

  • Anonymous
    May 21, 2009
    Hi, This is what I am using in my decrypt method..but i m getting the error of bad data Can anyone help me out: public static string DecryptString(string strEncData, string strKey, string strIV) { ICryptoTransform ct; MemoryStream ms; CryptoStream cs; byte[] byt; SymmetricAlgorithm mCSP=SymmetricAlgorithm.Create(); mCSP = new TripleDESCryptoServiceProvider(); mCSP.Key = Convert.FromBase64String(strKey); mCSP.IV = Convert.FromBase64String(strIV); ct = mCSP.CreateDecryptor(mCSP.Key,mCSP.IV); byt = Convert.FromBase64String(strEncData); ms = new MemoryStream(); cs = new CryptoStream(ms,ct, CryptoStreamMode.Write); cs.Write(byt,0,byt.Length); cs.FlushFinalBlock(); cs.Close(); return Encoding.UTF8.GetString(ms.ToArray()); }

  • Anonymous
    July 16, 2009
    The comment has been removed

  • Anonymous
    December 02, 2009
    Maybe there is a better way, maybe I can be enlightend but this is what I came up with. It's all that conversion stuff that has my head spinning. code: #Region "Security"    Public Sub Encrypt(ByVal password As String)        Dim s_aditionalEntropy As Byte() = CreateRandomEntropy()        Dim secret1 As Byte() = Encoding.UTF8.GetBytes(password)        Dim secret2 As String = Convert.ToBase64String(secret1)        Dim secret3 As Byte() = Convert.FromBase64String(secret2)        Dim encryptedSecret As Byte()        'Encrypt the data.        encryptedSecret = ProtectedData.Protect(secret3, s_aditionalEntropy, DataProtectionScope.CurrentUser)        SaveSetting(TITLE, "Settings", "UserP", Convert.ToBase64String(encryptedSecret))        SaveSetting(TITLE, "Settings", "UserE", Convert.ToBase64String(s_aditionalEntropy))    End Sub    Public Function Decrypt() As String        Dim s_aditionalEntropy As Byte()        Dim encryptedSecret As Byte()        encryptedSecret = Convert.FromBase64String(GetSetting(TITLE, "Settings", "UserP", ""))        s_aditionalEntropy = Convert.FromBase64String(GetSetting(TITLE, "Settings", "UserE", ""))        If encryptedSecret.Count <> 0 Then            Dim secret1 As Byte() = ProtectedData.Unprotect(encryptedSecret, s_aditionalEntropy, DataProtectionScope.CurrentUser)            Dim secret2 As String = Convert.ToBase64String(secret1)            Dim secret3 As Byte() = Convert.FromBase64String(secret2)            Dim secret4 As String = Encoding.UTF8.GetString(secret3)            Return secret4        Else            Return ""        End If    End Function    Function CreateRandomEntropy() As Byte()        ' Create a byte array to hold the random value.        Dim entropy(15) As Byte        ' Create a new instance of the RNGCryptoServiceProvider.        ' Fill the array with a random value.        Dim RNG As New RNGCryptoServiceProvider()        RNG.GetBytes(entropy)        ' Return the array.        Return entropy    End Function 'CreateRandomEntropy #End Region

  • Anonymous
    January 29, 2010
    Looks interesting... Seems to have one more problem, cause the returned string, doesnt want to combine with my other variables. Would like to decode the Adress from an server like (MyServer.org)... Seems to work correctly, but if i take this returned string and try to add subfolders variables, like (ServerAdress & Folder1 & Folder2), he ignored the two variables Folder1 and Folder2... Why that ? I search for an solution about 2 days till now.. Anyone any thoughts ??? Greets

  • Anonymous
    February 01, 2010
    I am actually using ToBase64String and FromBase64String and still get dad data error any idea ???????????????????? Here is the code: Ecrypting:   DESCryptoServiceProvider desCrypto = new DESCryptoServiceProvider(); MemoryStream ms = new MemoryStream(); CryptoStream cs = new CryptoStream(ms,desCrypto.CreateDecryptor(EncryptKey,EncryptVactor),CryptoStreamMode.Write); StreamWriter sw = new StreamWriter(cs); sw.Write(valueToEncrypt); ms.Flush(); cs.Flush();                                     sw.Flush(); string result = Convert.ToBase64String(ms.GetBuffer(), 0, (int)ms.Length);   log.Info("value To Encrypt is " + valueToEncrypt.ToString()); log.Info("Encrypted value is " + result); log.Info("Encrypting successfull."); Decripting: DESCryptoServiceProvider desCrypto = new DESCryptoServiceProvider(); byte[] buffer = Convert.FromBase64String(valueToDecrypt);   MemoryStream ms = new MemoryStream(buffer); CryptoStream cs = new CryptoStream(ms, desCrypto.CreateDecryptor(decryptKey,decryptVactor), CryptoStreamMode.Read); StreamReader  sw = new StreamReader(cs);                                     ms.Flush(); cs.Flush(); cs.FlushFinalBlock(); string result = sw.ReadToEnd(); log.Info("value To decrypt is " + valueToDecrypt.ToString()); log.Info("Decrypted value is " + result); log.Info("Decrypting successfull.");

  • Anonymous
    February 24, 2010
    Make sure you call FlushFinalBlock when doing the encryption as well - otherwise the padding won't get added correctly. -Shawn