Share via

The NT DLL Loader: DLL_PROCESS_ATTACH reentrancy - step 4 - ramifications of questionable quality

  • Article

Last time I alluded to the world of hurt you're in when the loader is reentered during DLL_PROCESS_ATTACH and the initialization of another DLL failed.

The state of the affairs is pretty derivable from the clues that I've left behind in the series.

First, we know that the loader tracks entry into the initialization function.  Otherwise the recursive/reentrant usage of the loader would have just called the same function.  Once it sees that an init function has been called, it will not call it again for initialization.

So, in the context of the example from last time, foo.dll's initializer failed.  The loader had marked it as entered.  The failure was not propagated.   Even if the example code was better and called FreeLibrary() to release the library in the case that GetProcAddress() failed, the refcount of foo is still non-zero so it stays loaded.

The next behavior is entirely predictable.  When the loader returns back to the outer LoadLibrary() (which was in the middle of initializing all the uninitialized DLLs in the initialization list), it continues on and is very nice and careful not to call any initializers which have not already been called.

Now, foo.dll is loaded, it is not initialized and its initializers will never be called.  Nonetheless since the failure was lost it tries to make progress.

Aside:

The obvious suggestion is to track the fact that the init failed and fail the entire load sequence.  But that's not really possible; you have an arbitrary number of levels of reentrancy here; making this a failure case now when it didn't used to be will break applications.  Once again the black helicopters swoop in and accusations that Windows only tries to be compatible fly around.  Most of all, you've caused an app compat hit by disallowing something that's always been documented not to work.

Now, let's see.  I can spend let's roughly cost it to 320 person-hours dealing with the arguably better but incompatible change I made or I can just let it be and continue to advise against the whole pattern.

Like I said, while it was a fun ride maintaining the loader for a period of time, it was also a little slice of hades too.

Now clearly this behavior needs to be called out better in the documentation.  These blog entries are an attempt at shedding light on the topic focussing mostly on directly observable behavior rather than "contractual obligations".  (app compat turns observable behavior into contractual obligations but app compat is also dialable given the intestinal fortitude and free time to do it.)

Note that this problem is about the closure of the call graph during the duration of the DLL_PROCESS_ATTACH.

Next time, wrapping up DLL_PROCESS_ATTACH reentrancy and then setting the stage for DLL_PROCESS_DETACH issues.

Comments