Share via

DllMain : a horror story

  • Article

Last
time I was talking about DllMain and what nasty things can occur if you misuse
it. I have also mentioned that it may not be one of those "I'm always careful it can
never happen to me" situations - things can get out of hand very quickly.

Keep in mind that OS loader has
been evolving over time. OS creators know that not all DLLs are well-behaved and they
have been trying to do their best to minimize the impact of poorly-written DllMains,
however it is still more than possible to shot oneself in the foot.

Let me give you a very simple example
as to how easy this can be (behavior may vary on different OS's, I'm running this
on Windows XP SP1).

Consider the following:

/////////////////////////////////////////////////////////////////////

Dll2.cpp

/////////////////////////////////////////////////////////////////////

HMODULE g_Module;

TCHAR g_tclpszFileName[256];

BOOL APIENTRY DllMain( HINSTANCE
hModule,

                       DWORD ul_reason_for_call,

                       LPVOID
lpReserved

                                                                                )

{

                if (
DLL_PROCESS_ATTACH == ul_reason_for_call )

                {

                                printf("Dll2:DllMain\r\n");

                                g_Module
= hModule;

                                ::GetModuleFileName(
g_Module, g_tclpszFileName, 255 );

                }

   return TRUE;

}

extern "C" __declspec(dllexport) void WINAPIV
OutputModuleInfo2(void)

{

                printf("Enetering
Dll2::OutputModuleInfo2\r\n");

                printf("Name:
%s\r\nHandle 0x%x\r\n", g_tclpszFileName, g_Module );

}

/////////////////////////////////////////////////////////////////////

// Dll1.cpp -
NEVER do this

/////////////////////////////////////////////////////////////////////

typedef void (WINAPIV
*LPFOUTPUTMODULEINFOFUNC) (void);

HMODULE g_Module;

TCHAR g_tclpszFileName[256];

BOOL APIENTRY DllMain( HINSTANCE
hModule,

                       DWORD ul_reason_for_call,

                       LPVOID
lpReserved

                                                                                )

{

                if (
DLL_PROCESS_ATTACH == ul_reason_for_call )

                {

                                printf("Dll1:DllMain\r\n");

g_Module = hModule;

                                ::GetModuleFileName(
g_Module, g_tclpszFileName, 255 );

                               

                                //
Load Dll2 - never EVER do this

                                HMODULE
hModule1 = ::LoadLibrary("Dll2.dll");

                                LPFOUTPUTMODULEINFOFUNC
lpOutputModuleInfo1Func = (LPFOUTPUTMODULEINFOFUNC)::GetProcAddress( hModule1,"OutputModuleInfo2");

                                lpOutputModuleInfo1Func();

                }

    return TRUE;

}

extern "C" __declspec(dllexport) void WINAPIV
OutputModuleInfo1(void)

{

                printf("Enetering
Dll1::OutputModuleInfo1\r\n");

                printf("Name:
%s\r\nHandle 0x%x\r\n", g_tclpszFileName, g_Module );

}

/////////////////////////////////////////////////////////////////////

// Main.cpp

/////////////////////////////////////////////////////////////////////

extern "C" __declspec(dllimport) void WINAPIV
OutputModuleInfo1(void);

extern "C" __declspec(dllimport) void WINAPIV
OutputModuleInfo2(void);

int _tmain(int argc,
_TCHAR* argv[])

{

                OutputModuleInfo1();

                OutputModuleInfo2();

                return 0;

}

What's wrong with this? Let me
count the ways. On top of non-existent error-handling and the fact that we have an
un-paired LoadLibrary() call, this code has a very fundamental problem. Let's
just say that depending on how this code is compiled, it may

-
Run and produce results you expect

-
Run and produce results you don't
expect

-
Blow up with AV

That's right.

Let's dig into it.

First let's see what this code
is supposed to do in the first place. You'll have to bear with me here - it's after
midnight and I haven't been able to come up with something brilliantly meaningful,
but this will just have to do for now.

As you can see, we are dealing
with two DLLs and one EXE that uses those DLLs. First DLL (inventively called Dll2)
- gets its own HMODULE in DLLMain(DLL_PROCESS_ATTACH), gets its name based on that
and stores them away in global variables. Exported function OutputModuleInfo2 simply
prints that out using printf.

Dll1 is almost identical, except
it dynamically calls into Dll2 right after collecting its own information. It's a
little weird, but this is just a primitive example after all.

Main

is a console appliccation that
is statically bruit against export libraries produced by the build of the first two
DLLs and calls both OutputModuleInfo1 and OutputModuleInfo2.

Simple enough? Let's roll.

1.
It
works! It works! Let's
compile everything, but make sure that all three binaries use CRT(C/C++ runtime) dynamically
(/MD compiler option) and that Dll2.lib appears before Dll1.lib in linker options
pertaining to additional input libraries for our console app (something like link.exe
main.obj /out:main.exe dll2.lib dll1.lib). Turns out that is important - we'll
see why in a little bit. When you run the application, it outputs:

Dll2:DllMain

Dll1:DllMain

Enetering
Dll2::OutputModuleInfo2

Name:
c:\Temp\KillDllMain\Dll2.dll

Handle
0x10000000

Enetering
Dll1::OutputModuleInfo1

Name:
c:\Temp\KillDllMain\Dll1.dll

Handle
0x320000

Enetering
Dll2::OutputModuleInfo2

Name:
c:\Temp\KillDllMain\Dll2.dll

Handle
0x10000000

As you see, things seem to be working
fine. One thing that is worth pointing out is that - as you can see from the output
- DllMain for Dll2.dll was called before DllMain of Dll1.dll. Why? Well, technically,
there's no explicit guarantee as to the order of these things - it is all in the loader's
hands. As
I mentioned before, the loader looks at static dependencies and builds a list
of DllMains to be called based on that. But what happens if the order really doesn't
matter? From loader's perspective, Main.exe depends on Dll1 and Dll2 and there's no
reason to choose one over the other (remember, the fact that Dll1 does in fact load
Dll2 is our little dirty secret).

Well,
turns out that the loader seems to be preserving the order in which the imported DLLs
are listed in the Imports Section of the loading executable. You can read all about
the low-level details in Matt
Pietrek's article, but for the purpose of this discussion let's just say that
each PE file (EXE or DLL) knows what binaries it "references" - that is what external
functions it imports - and that a list of those binaries, together with referenced
functions is linked into its PE header. Microsoft Linker seems to build that header
based on the order in which export libraries are supplied, which is why we built our
app the way we did.

                        What
happens if change that? Let's see.

2.
Huh?
But... So now let's
build the same code, only this time supply export libraries in the opposite order
(something like link.exe main.obj /out:main.exe
dll1.lib dll2.lib). Let's run it:

Dll1:DllMain

Enetering
Dll2::OutputModuleInfo2

Name:

Handle
0x0

Dll2:DllMain

Enetering
Dll1::OutputModuleInfo1

Name:
c:\Temp\KillDllMain\Dll1.dll

Handle
0x10000000

Enetering
Dll2::OutputModuleInfo2

Name:
c:\Temp\KillDllMain\Dll2.dll

Handle
0x320000

Interesting... As you see, this
time DllMain from Dll1 got called first. That loaded Dll2 and its OutputModuleInfo2
got called ... before its DllMain! No wonder it printed what it did. Note that the
second call into OutputModuleInfo2 went through just fine because Dll2's DllMain was
called already.

So why in the world is OS loader
acting so dumb? Doesn't it know we are loading Dll2? We have explicitly called LoadLibrary
after all, which loaded it from disk, laid it out in memory, resolved its exports
etc. Why wasn't DllMain called? If you experiment a little, you will find out that
in most cases DllMain of dynamically loaded libraries will be
called, even if the "illegal" LoadLibrary is used to load it. The only case that will
not take place is when OS loader already "knows" about that DLL but hasn't yet called
DllMain on it, which is exactly what happened here.

Main.exe statically depends
on Dll2.dll, so it's already in the loader's plan. It turns out, the loader is not
so willing to change its original plan created based on static dependencies. If new
binaries get thrown in, the loader will stop and dutifully load them; but if the binary
is in fact the "old" one - that is already in the plan - the loader will just skip
it.

Why? My guess is that this works
pretty well for most scenarios. The loader is still trying to be nice and compensate
for our bad behavior. Once we attempt to load something it already knows about, it
simply preserves its current plan - I suspect doing otherwise would cause all kinds
of nasty consequences. Mind you, we are
on no position to complain - we are not supposed to call LoadLibrary from DllMain
in the first place. Keep in mind, these are my speculations - I'm not trying to give
a precise recipe as to how the OS loader can be mistreated, I'm just saying that it
can be done.

So...there you go. In this particular
situation you "just" got the wrong value printed out, but you can imagine that this
can easily cause a wide range of nastiness - AVs for instance. Speaking of which...

3.
What???
How did that happen?... Let's
build the whole thing again, only this time let's use static CRT (/MT or /ML compiler
options). Why should it matter, right?

Now let's run it:  
  
  
  
  
  
*Dll1:DllMain*

. ..
and then... whoa...

First-chance exception at 0x77f57bd2
(ntdll.dll) in MainApp.exe: 0xC0000005: Access violation reading location 0x00000010.

      But
why? If you look at the stack, you will see the following:

      ntdll.dll!_RtlAllocateHeap@12() +
0x24

  Dll2.dll!_heap_alloc(unsigned
int size=0x00000018) Line 212 C

  Dll2.dll!_nh_malloc(unsigned
int size=0x00000018, int nhFlag=0x00000000) Line
113 C

  Dll2.dll!malloc(unsigned
int size=0x00000018) Line 54 + 0xf C

  Dll2.dll!_mtinitlocknum(int
locknum=0x00000011) Line 251 + 0x7 C

  Dll2.dll!_lock(int
locknum=0x00000011) Line 311 + 0x6 C

  Dll2.dll!_lock_file2(int
i=0x00000001, void * s=0x00346b68) Line
267 + 0x9 C

  Dll2.dll!printf(const
char * format=0x0034204c, ...) Line 57
+ 0xd C

  Dll2.dll!OutputModuleInfo2() Line
30 + 0xa C++

> Dll1.dll!DllMain(HINSTANCE__
* hModule=0x10000000, unsigned long ul_reason_for_call=0x00000001, void * lpReserved=0x0012fd30) Line
30 + 0x5 C++

  Dll1.dll!_DllMainCRTStartup(void
* hDllHandle=0x10000000, unsigned long dwReason=0x00000001, void * lpreserved=0x0012fd30) Line
297 + 0xd C

       

So this is caused by calling "printf"
from Dll2's OuputModuleInfo2,
which is sort of strange. If you look some more, you will find that the CRT internal
global _crtheap is NULL, which means that CRT has no heap. Why? You guessed it
- static CRT allocates its heap in DllMain of the owning DLL! If our case DllMain
wasn't called yet, so naturally - no heap.

Ouch. (Incidentally, this means
that just about any CRT call will AV - it's awfully difficult to do anything without
allocating any memory...)

Moral

OK, this is much longer than I
intended... but here's the moral: be
careful. OS loader is
not dumb, and it is as forgiving as it gets, but sometimes it won't be there to help
- simply because it has no idea what your intentions are.

OK, I think I'm officially done
with the topic - I'm feeling much better now :)

Comments