How to trace CryptoAPI calls

Hi, welcome back,

An application may use CryptoAPI without us developers realizing it. Security classes in .NET Framework use CryptoAPI behind the scenes. CAPICOM.dll uses it, too.

The issue appears when the API returns an error which can help us to find the real cause of the issue, but the code calling it captures the error and returns a completely different one which won't help us at all. Remember for instance my previous post, RSACryptoServiceProvider fails when used with mandatory profiles, where .NET was returning "Cryptographic Service Provider (CSP) for this implementation could not be acquired" but the real error was being returned by CryptAcquireContext: NTE_TEMPORARY_PROFILE. With .NET's error I don't know what's going on. With CryptAcquireContext's I do know.

Fortunately debuggers like Windbg.exe\Cdb.exe (part of our Debugging Tools for Windows) are very handy here. If we want to trace the calls that an application is making to CryptoAPI behind the scenes, we can follow these steps:

1) Attach Windbg or Cdb (command line version of Windbg) to the desired process.

2) Set the following breakpoint:

bm Advapi32!Crypt* ".printf \" \\n>>>>>>>>>>>>>>>>>>>>>>\\n\\n(%#x)\\n\\n\ ", @$tid; .echo CALL;k 1; bp /t @$thread poi(@esp) \" .echo;.echo RESULT; .if(@eax=1) {.echo SUCCEEDED} .else {.echo FAILED;!gle}; .echo;.echo <<<<<<<<<<<<<<<<<<<<<<; G;\"; G;";

Note:  There are other Crypt* APIs on Crypt32.dll, so an additional breakpoint may be set for them. 

3) Continue running the application under the control of the debugger.

 

What is that "bm" command doing exactly? It will be easier to understand it if I write it this way:

 00  bm Advapi32!Crypt* 
01  "
02      .printf \"\\n>>>>>>>>>>>>>>>>>>>>>>\\n\\n(%#x)\\n\\n\", @$tid;
03
04      .echo CALL;
05      k 1;
06
07      bp /t @$thread poi(@esp)
08      \"
09          .echo;
10          .echo RESULT;
11
12          .if(@eax=1)
13          {
14              .echo SUCCEEDED
15          }
16          .else
17          {
18              .echo FAILED;
19              !gle
20          };
21
22          .echo;
23          .echo <<<<<<<<<<<<<<<<<<<<<<;    
24
25          G;
26      \";    
27  
28      G;
29  ";

00) We set a breakpoint on all APIs found in Advapi32.dll which start by "Crypt". The following commands will be executed when the application calls any of those APIs thus reaching any of those breakpoints:

02) We print the id of the thread calling the API (" $tid" pseudo-register). " .printf" works pretty much the same way as our old friend "printf" in C. Instead of ' " ' and ' \n ' we will have to use ' \" ' and ' \\n ', because these commands are already between ' " '.

04) We print the string "CALL" with " .echo".

05) We print the first line of the call stack gotten when the breakpoint is reached ("k 1"). The API we are calling will correspond to that first line. Here we could show the parameters being passed to the API by using "kp" command (if we have symbols -.pdb files- associated to Advapi32.dll), or any other way.

07) Now we want to know what happened after the API finished executing. When we stop on the breakpoint set at 00, the "esp" register (stack pointer) points to the return address where we will go after the call to the API ends. So we set a breakpoint at that address with "bp". And if we don't want weird things to happen on multi-threaded applications, we have to specify on which thread (" $thread" pseudo-register) we are setting that breakpoint.

28) Once we've set the breakpoint at 07, we can continue the execution of the application with "g" command.

Now, let see what happens when we stop on breakpoint set at 07:

09 & 10) We print the string "RESULT".

12) When the API has finished executing, its return value will be stored in the "eax" register. The Crypt* functions in Advapi32 will return 1 if they succeed or 0 if they don't. We use " .if {} .else {} " command to determine which one we got.

14) The API succeeded, so we print the string "SUCCEEDED".

18 & 19) The API failed, so we print the string "FAILED" and show the last error value with " !gle" (Get Last Error) command.

25) Now that we know the result of calling the API, we can continue the execution of the application and trace more APIs.

 

Comments on this:

- Some people may think that setting a breakpoint at 07 is unnecessary, as we could use "gu" command to get to the end of the call instead. But if we use "gu" and a CryptoAPI calls another one, we will miss the return value of the former.

- I tried setting the breakpoint at 07 using " ~~[$tid] bp ... ", but the behavior wasn't correct. I don't know why the debugger was stopping several times on the same breakpoint instead of just one. Using "bp /t @$thread ..." instead worked just fine.

 

I hope this helps.

Cheers,

 

Alex (Alejandro Campos Magencio)

 

PS: I'm currently developing a Windbg\Cdb script based on the principles explained in this post which traces all CryptoAPI calls, but this time showing all their in & out parameters in a very user friendly way. You can see its current version here.