Share via

New NX APIs added to Windows Vista SP1, Windows XP SP3 and Windows Server 2008

  • Article

In the interests of helping secure the platform, we want more people to opt-in to using Data Execution Prevention (aka DEP aka NX), and we have lowered the barrier to entry for application developers in Windows Vista SP1, Windows XP SP3 and Windows Server 2008.

We've added some new APIs that allow a developer to set DEP on their process at runtime rather than using linker options. The new APIs also give developers some more flexibility if your application uses an older version of the Active Template Library (ATL.) Before I explain the new APIs, let me give you a little history behind ATL and NX.

Some ATL History

ATL has been around for a long time; it's reasonably light-weight and allows developers to build COM components rapidly. It also includes classes for manipulating security descriptors and such; to be honest, it makes working with Windows security objects open to mere mortals.

Older versions of ATL, and by older I mean pre-Visual C++ 2005, used dynamically generated code in small isolated cases. Obviously, without the appropriate APIs this is going to cause problems on a DEP-enabled computer, because you can't execute data. This code is referred to as a "thunk" and versions of ATL in VC++ 2005 and later work correctly with DEP.

The APIs

The most important API added is SetProcessDEPPolicy,   which sets the DEP policy for the running process. You would normally use this function pretty early in main.

The function takes only one flag argument: the policy setting. 

If your program loads 3rd party plug-ins or makes use of older ATL libraries you should use the flag below, as this enhances compatibility:

  • PROCESS_DEP_ENABLE Enable DEP for the process and allow ATL thunk emulation.

If your program does not use legacy or 3rd party plug-ins, nor make use of older ATL libraries, you should use:

  • PROCESS_DEP_ENABLE | PROCESS_DEP_DISABLE_ATL_THUNK_EMULATION Enable DEP for the process, and disallow ATL thunks.

There are two other functions: GetSystemDEPPolicy and GetProcessDEPPolicy; I'm not going to insult your intelligence and explain what they do.

The only negative to these APIs is they must be dynamically loaded because they don't exist on all supported versions of Windows. The following code shows how you can use the functions regardless of Windows version:

 

If you OR the two flags together, it's virtually the same as linking with /NXCOMPAT.

When to use the NX APIs

There are three main reasons to use these new APIs:

  • If your application has some form of in-process extensibility mechanism, and some applications might use older ATL, then you can enable DEP for your process, and the extensibility mechanisms using ATL will function correctly.
  • If you support DEP but want to allow customers to disable DEP if there are serious compatibility issues, then this is the API to use because the argument can be a configuration option.
  • If your application uses an old version of ATL, and you still want to do the right thing by DEP, then use this function. Of course, you really ought to use an updated version of ATL!

One Caveat

I'm only telling you this because it bit me.

There is one caveat that you should know; SetPRocessDEPPolicy often returns error 5 (Access Denied) but this error does not mean the operating system is denying access, it means you are attempt to change DEP policy in a way that is not appropriate. For example, if you link with /NXCOMPAT, and then use this API, you'll get the error. Or, if the operating system is configured to use DEP for all processes all the time no matter what, then you'll see the same error. Finally, you'll get an access denied error if you attempt to call SetPRocessDEPPolicy twice in one application; once the policy is set, it's set for the process lifetime.

In short, don't be overly alarmed if you see this error.

#define PROCESS_DEP_ENABLE                          0x00000001
#define PROCESS_DEP_DISABLE_ATL_THUNK_EMULATION     0x00000002

BOOL SetDEP(__in DWORD dwFlags = PROCESS_DEP_ENABLE) {

       HMODULE hMod = GetModuleHandleW(L"Kernel32.dll");

       if (!hMod) return FALSE;

       typedef BOOL (WINAPI *PSETDEP)(DWORD);

       PSETDEP procSet = (PSETDEP)GetProcAddress(hMod,"SetProcessDEPPolicy");

       if (!procSet) return FALSE;

       return procSet(dwFlags);

}

Comments

  • Anonymous
    January 29, 2008
    PingBack from http://www.windows-vista.luiscorreia.com/new-nx-apis-added-to-windows-vista-sp1-windows-xp-sp3-and-windows/

  • Anonymous
    January 29, 2008
    The comment has been removed

  • Anonymous
    January 29, 2008
    The comment has been removed

  • Anonymous
    January 30, 2008
    The comment has been removed

  • Anonymous
    January 30, 2008
    What about Windows 2k3?  Are there plans for a service pack to add this functionality there as well?

  • Anonymous
    January 30, 2008
    Why provide the option to disable it? it seems that makes easier the job of shellcode exploits.

  • Anonymous
    January 30, 2008
    Curt, you are 100% correct - I will correct the name.

  • Anonymous
    January 30, 2008
    IronGuts If you're running shellcode, then you must have already defeated NX!!

  • Anonymous
    January 30, 2008
    The comment has been removed

  • Anonymous
    January 31, 2008
    A while a go when I posted about the .NET Framework 3.5 and 2.0 SP1 being available for download, Kima

  • Anonymous
    February 01, 2008
    The comment has been removed

  • Anonymous
    February 01, 2008
    Nice, but the APIs missing in Windows are the ones to patch the system i.e. to force downloads of Emergency (critical/wormable ones with an exploit in the wild) or Critical patches Patching APIs would make a HUGE difference in how systems are protected as they could be called by installers (even third party ones) or system management software ... or even tempt people to write inoculating viruses (vaccines);-)

  • Anonymous
    February 02, 2008
    stefan, updating is built into the OS, it's not an app thing. that's why we default new OSs to check for updates every 24hrs

  • Anonymous
    February 02, 2008
    o.s. - THINK the issue relates to the granularity of the underlying APIs, it has a fixed set of errors, and Err5 is one.

  • Anonymous
    February 04, 2008
    Per faciliare la pianificazione di un corretto processo di update di Vista oggi Renato, sul blog di Technet

  • Anonymous
    February 04, 2008
    Per faciliare la pianificazione di un corretto processo di update di Vista oggi Renato, sul blog di Technet

  • Anonymous
    February 15, 2008
    Thank You For Sharin very inforamtive materials with us

  • Anonymous
    February 20, 2008
    There are other libraries besides old ATL that use thunking.  For instance, our app is build with OWLNext which uses thunking for windows in a way similiar to ATL.  Is there a way to turn on DEP but allow these specific thunks to work?

  • Anonymous
    February 29, 2008
    I have DEP problems with Server 2003 Enterprise. Is there a possiblility to get the current state of DEP-settings (via an alternative way for GetProcessDEPPolicy / GetSystemDEPPolicy)? We use a translation tool that modifies the address/code of LoadResString and at that point  our program is being kicked without exception. When the program is in the list it works fine. But that is not acceptable for clients. They shall know what went wrong... Any hint would be welcomed.

  • Anonymous
    March 02, 2008
    The comment has been removed

  • Anonymous
    March 07, 2008
    "'If you're running shellcode, then you must have already defeated NX!!" Not really, in return to libc attacks you are not there yet. If you can change the return address to point to this function you can disable NX in one more convenient way than before. It would be a two stage attack of course. maybe we can go from other side, what cases does it support flipping this flag over and over?

  • Anonymous
    April 07, 2008
    I'm getting a bit confused about the flag PROCESS_DEP_DISABLE_ATL_THUNK_EMULATION. If I have an ATL application using old ATL, shall I set flag to PROCESS_DEP_ENABLE | PROCESS_DEP_DISABLE_ATL_THUNK_EMULATION or  PROCESS_DEP_ENABLE only?

  • Anonymous
    April 08, 2008
    Hi, I’m Eric Lawrence from the Internet Explorer Security Team. With the RSA security conference kicking

  • Anonymous
    September 11, 2008
    First, let me remind you that in my new ongoing quest to read source code to be a better developer ,

  • Anonymous
    September 15, 2008
    Scott Hanselman has a look under Chrome's hood and how it uses the new NX/DEP APIs we added to Windows

  • Anonymous
    May 21, 2009
    Hello, my name is Xiang Fan and I am a developer on the C++ Shanghai team. Today I’d like to talk about

  • Anonymous
    June 01, 2009
    こんにちは、五寳です。 IE7 から実装されているメモリ保護 ( DEP/NX Memory Protection ) の機能ですが、IE8 からは (条件がそろえば) デフォルトで有効になっています。