Understanding DEP/NX

Despite being one of the crucial security features of modern browsers, Data Execution Prevention / No Execute (DEP/NX) is not well understood by most users, even technical experts without a security background.

In this post, I’ll try to provide some insight into how DEP/NX works, explain why you might encounter a DEP/NX crash, and convince you that turning off DEP/NX is almost never the right decision.

More than anything else, I hope you take away two important facts from reading this post:

• In many cases where you encounter a DEP/NX crash, the browser would have crashed anyway.
• The vast majority of DEP/NX crashes are caused by browser add-ons. If you run IE in No Add-ons Mode, it’s very unlikely that you will encounter a DEP/NX crash.

Background

I’ll begin by providing some background information on DEP/NX and how the browser makes use of it.

What is DEP/NX?

DEP/NX is a feature of modern CPUs that allows marking of memory pages as Executable or non-Executable. This allows the CPU to help prevent execution of malicious data placed into memory by an attacker. If the CPU detects that it is about to jump to (begin execution of) data which is in a memory page which is not marked as Executable, the CPU will raise an exception which results in termination of the process.

Stated another way, if DEP/NX determines that if a potentially dangerous jump is about to be made, the process is intentionally “safely crashed” to prevent a potential security exploit. You can get more DEP/NX details here.

Checking Your Protection

You can see which processes are protected by DEP/NX using Task Manager’s Process tab. On Windows XP, you need to use Process Explorer instead. In either case, ensure that the “Data Execution Prevention box” is checked in the View > Select Columns menu, and a column in the process list will show the DEP/NX protection status.

As mentioned last year, Internet Explorer 8 enables DEP/NX protection by default. In IE7 and earlier, DEP/NX was disabled by default due to compatibility concerns that were resolved in IE8.

Opting-in to DEP/NX

Internet Explorer 8 uses the SetProcessDEPPolicy() API to enable DEP/NX. This provides the following benefits versus using the /NXCOMPAT linker flag:

• It allows us to offer an Internet Control Panel checkbox and Group Policy option to disable DEP/NX if desired.
• It enables DEP/NX on Windows XP SP3. The Windows XP loader does not check the NX Compatible bit.
• It ensures that ATL_THUNK_EMULATION, an important compatibility feature, works properly.

Note: New applications without 3rd-party code compatibility concerns, targeted for use on Vista and later, should simply use the /NXCOMPAT linker flag.

Recognizing a DEP/NX Crash in Internet Explorer

When Internet Explorer 8 recovers from a DEP/NX-induced crash, it will not automatically recover the current tabs. This is a security measure designed to help prevent a malicious site from having multiple attempts to exploit a vulnerability. Instead of reloading the tabs, the browser will show the following error page:

Unfortunately, the nature of DEP/NX crashes makes it infeasible for the browser to “pin the blame” on the specific add-on that is responsible for the problem.

Why do DEP/NX Crashes Occur in the Real World?

Now, let’s take a look at why users encounter DEP/NX crashes in the real-world.

When the CPU is about to jump to a non-Executable memory page, there are three possible types of data in that page: malicious code, non-malicious code, and garbage data. I’ll discuss each of these in the following sections.

Jump Target: Malicious code

This is the scenario where DEP/NX shines. In this scenario, an attacker has put malicious data in memory that will be executed as x86 instructions if he can get the CPU to jump to it. The attacker then exploits some vulnerability to induce the CPU to jump to his data, typically using a memory-related vulnerability in an add-on or the browser itself.

In this scenario, the CPU notes that the attacker’s code is not in an executable memory page and prevents the interpretation of the attacker-supplied data as instructions. The attack is foiled and the user’s machine is protected. If not for DEP/NX, the attacker would have been able to execute his instructions and potentially infect the user’s machine with malware, steal their data, or achieve some other nefarious goal.

Now, the obvious next question is: What if the attacker can somehow get his data marked as executable?

The answer is that doing so is intentionally difficult. IE8 blocks the best known trick used to get the attacker’s data in an executable page. That means that the attacker must find some other way to get the memory page containing his instructions marked as executable.

The obvious choice would be for the attacker to call VirtualProtect() directly, passing PAGE_EXECUTE_READ as the flNewProtect flag. However, thanks to Address Space Layout Randomization (ASLR) it is difficult for the attacker to guess where the VirtualProtect function is in memory. If he guesses wrong (and he almost always will), the process will crash and not execute his attack instructions.

Jump Target: Non-malicious code

In this scenario, a browser add-on is designed in such a way that it expects to be able to execute data from memory pages which are not marked as executable, or otherwise makes a bad assumption.

There are a number of possible cases where this may happen.

Case #1: Code Generation

In the first case, the add-on (or the technology it is built upon) depends on the ability to execute dynamically generated instructions at runtime. Examples of this are the Java Virtual Machine (JVM) and the Active Template Library (ATL). These frameworks generate (“JIT compile”) executable code at runtime and jump to it. Older versions of these frameworks did not mark the memory pages containing the generated code as executable and would hence crash when DEP/NX was enabled. The Java team fixed this problem in the JVM years ago, and the ATL team also fixed this problem several versions ago.

Because ATL is so commonly used to build Internet Explorer add-ons, additional work was done to allow Windows to “emulate” the ATL Thunk code which violated DEP/NX, so that even if an add-on was compiled against an ancient version of ATL, ATL Thunk Emulation will ensure that the code runs properly inside Internet Explorer with DEP/NX enabled.

Case #2: Code Rewriting

In another common case, the add-on depends on “thunking” or modifying an existing Internet Explorer API or Windows function at runtime by rewriting the instructions in the existing function’s memory page. In order to accomplish this, the add-on uses VirtualProtect() to change the memory protection of the target page to allow Write, then updates that page with new instructions that point to some code that the add-on would like to have run inside the target function.

If the add-on fails to subsequently call VirtualProtect() to revert the memory protection back to allow Execute, the process will crash with a DEP/NX violation the next time that function is called.

More commonly, the add-on will later change the memory protection back to allow Execute, but the developer ignores the fact that it’s entirely unsafe to perform modification of shared code while any other threads are executing. While an add-on thread is modifying the code in a memory page, if any thread attempts to call any function in the same memory page, the process will crash. Internet Explorer makes extensive use of threads, so such crashes are likely if an add-on uses thunking.

Because timing is a critical factor here, the add-on may seem to “work fine” on one machine (e.g. a slower single-core machine) and always crash on another (e.g. a fast multi-core machine). This problem is just one of the major reasons why function thunking by Add-ons is not supported and is strongly discouraged.

Jump Target: Garbage data

In this scenario, inadvertent memory corruption has occurred such that the CPU is about to jump to arbitrary data somewhere in memory. This scenario is probably the most common source of DEP/NX crashes, particularly when the crash occurs at a seemingly random time, or when a browser tab is closed.

This arbitrary data isn't usually chosen by an attacker, and usually doesn’t even represent sensible x86 instructions. For instance, the jump may be to an address near 0x000000 where no code is loaded (near-null jump) , if a virtual function was called off an object pointer which has been nulled. Or, the jump may be to some other address where code used to exist (stale pointer) but that memory was later freed and potentially reused for another purpose.

In this “garbage data” scenario, the process will almost always crash, even if DEP/NX were not enabled. That’s because the CPU is very unlikely to reliably execute arbitrary data as sensible x86 instructions. Most likely, the process will crash within a microsecond with an exception like “Access Violation”, “Invalid Instruction”,  “Divide by 0” or similar.

Attackers look for this type of memory corruption to use as an entry point in their attacks; they may, for instance, “spray the heap” with many copies of their malicious data, then trigger the memory corruption vulnerability with the hope that the CPU will jump into a copy of their malicious code.

Resolving DEP/NX Problems

Your best bet to resolve DEP/NX problems in Internet Explorer is to first confirm that the problem is caused by a buggy browser add-on. You can do this by running IE in No Add-ons Mode. After confirming that the problem is related to an add-on, you should use the browser’s Manage Add-Ons feature to disable unwanted add-ons and find updated versions of any add-ons that you wish to keep.

If you find that you’re encountering DEP/NX crashes in multiple software applications, it’s possible that you have malicious or buggy system software installed (e.g. malware or a buggy anti-virus product). You should check your system for malware and ensure that you install the latest updates for your system software.

Frequent DEP/NX crashes also suggest that your computer might have a hardware problem (e.g. bad system memory). To help rule out hardware failure, you can use the Windows Memory Diagnostic.

Conclusion

DEP/NX provides an important defense against malicious websites that may try to exploit vulnerabilities in your add-ons or web browser. By ensuring that you are running the latest version of add-ons and system software, you can improve your security and minimize the incidence of DEP/NX crashes. If you're currently using an older version (6 or 7) of Internet Explorer that does not have DEP/NX protections enabled by default, you should upgrade to IE8 as soon as possible.

Thanks for reading!

-Eric

Comments

• Anonymous
  October 11, 2009
  Interesting post. I always enable DEP for "all programs and services except those I select" via the System Properties. There's an excellent program at GRC.com called SecurAble which tells you whether your CPU supports hardware DEP.

• Anonymous
  December 10, 2009
  Over at //bernardodamele.blogspot.com/2009/12/dep-bypass-with-setprocessdeppolicy.html, Bernardo writes a nice post explaining how exploit code could attempt to circumvent DEP on Windows XP by calling SetProcessDEPPolicy. That post fails to note two critical points, however: 1> SetProcessDEPPolicy can only be called once per process, so if the process itself calls the function (as IE8 does on startup) this attack won't work. 2> It's broadly understood that DEP/NX isn't very useful without ASLR. It's ASLR that breaks the shellcode's ability to reliably jump to the SetProcessDEPPolicy function to turn off DEP/NX protection.   The attack demo shown in the post is performed on Windows XP, where that pesky ASLR stuff isn't available.

• Anonymous
  December 27, 2009
  The comment has been removed

• Anonymous
  January 26, 2010
  The comment has been removed

• Anonymous
  July 11, 2010
  In our company we have trouble with every client because of ActiveX-Plugins (since the update IE6 to IE8). Many ActiveX-Plugins (even microsoft support activeX control!) are crashing the IE8. We need the correct function of ActiveX-Plugins, for example for WebEx ActiveX-Plugins. But we don't want to disable DEP, because it's an important security function. I disabled all IE add-ins, but this doesn't help. What can I try or check else? I found many articles in the internet with users .

• Anonymous
  July 11, 2010
  The comment has been removed

• Anonymous
  July 12, 2010
  The comment has been removed

• Anonymous
  July 28, 2010
  Wouldn't the setting "Enable memory protection to help mitigate online attacks" have been better under IE's Security?  Thus, under Local Intranet or Trusted zone it'd be off, but under Internet or Restricted it'd be on.  And we can add sites to the zones depending on our circumstances/observations. As is, I have to uncheck the setting; do what I need to resolve a user's issue; and re-check that setting.  And most sites I've come across are not telling/advising/recommending people to recheck that setting.  One or two users that's fine, but I could see where it could become a headache to support.   I had a user try to download SQL Server 2008 iso from Microsoft MSDN website, and the dep crash page came up.  After we unchecked that settings, we saw that it was trying to get the user to install a File Dowload Manager.  So it DOES happen with a Microsoft site--which we kind of thought was funny/ironic.

• Anonymous
  July 28, 2010
  The comment has been removed

• Anonymous
  August 11, 2010
  The comment has been removed

• Anonymous
  August 12, 2010
  The Windows Update ActiveX control is fully DEP-compatible. If you encounter DEP crashes with WindowsUpdate, either you have hardware failure (bad memory) or conflicting system software (e.g. antivirus) which is improperly thunking system APIs and introducing crashes. I have yet to find ANY ActiveX sample code which is not DEP-compatible. The only incompatible code I know of (e.g. ancient versions of the Sun Java Virtual Machine) are not open-source.

• Anonymous
  August 23, 2010
  Hello Eric, do ActiveX controls that were developed using VB6 can cause DEP/NX crashes as well?

• Anonymous
  July 05, 2011
  Another advantage is that LCIE allows IE to run non-DEP capable add-ons and DEP capable add-ons in separate processes.

• Anonymous
  July 05, 2011
  @Yuhong: No such feature exists. DEP/NX is enabled by default for all processes (and should be left in that configuration). If you disable the feature, then it's off for all processes. Generally speaking, very few things aren't DEP-compatible these days.