Compute Emulator: Invalid access to memory location
Symptom
When running a Windows Azure project in Microsoft Visual Studio, you may encounter the following Windows Azure Compute Emulator: Unexpected Failure dialog:
Windows Azure Tools for Microsoft Visuals Studio prior to SDK 1.7 would display this error dialog:
The seemingly random nature of this issue made reproducing thus troubleshooting exceptionally difficult. A retry loop was added to SDK 1.7 which alleviated the issue for some users.
Root Cause
A bug in legacy 64-bit operating systems caused the kernel address-conversion routines to incorrectly sign-extended the 32-bit address when converting to a 64-bit address. When the sign-extended memory address was accessed, it caused an Invalid Access to memory location with NativeErrorCode 998 error.
Solution
The best solution is to install Windows 8.1; although it was first fixed in Windows 8. If you are using one of the following 64-bit operating systems, there is a hot fix documented in KB 2588507 to correct the address-conversion routines.
- Windows Vista
- Windows Server 2008
- Windows 7
- Windows Server 2008 R2
How did we find the solution?
May thanks to Ryan Johnson and the developers at Wellero for suffering through this nasty bug and for their patience working with us to get this issue resolved. In hindsight, it is easy to see that the issue only reproduces in high memory conditions when running a legacy OS and that the KB applied because Visual Studio was linked to use addresses greater than 2 GB. As Scott Adams recently wrote “failure is where success likes to hide in plain sight.”
For those curious about how we found the solution, start by examining the six errors displayed in the Unexpected Error dialog: System.Net.NetworkInformation.NetworkInformationException Invalid access to memory location with NativeErrorCode 998.
The Windows Azure Authoring Tools running inside of the Visual Studio process calls the .NET method GetActiveTcpListeners to obtain the list of ports currently in use so it doesn’t try to deploy the project to a port already in use. By debugging the kernel, we can see that a kernel function tries to allocate heap memory to process the list. The allocation fails on an address like this: A3BA2E38 with the return value STATUS_ACCESS_VIOLATION (0xc0000005). This status gets translated to ERROR_NOACCESS 998 (0x3e6) “Invalid access to memory location” in a NetworkInformationException by GetActiveTcpListeners.
Now that we understand the error message, we need to understand why we get the error. Visual Studio is a 32bit application so the Windows architecture would normally allocate a total virtual address space of 4GB of which the process can use the lower half x00000000 to x7FFFFFFFF. Luckily, Visual Studio uses the linker option /LARGEADDRESSAWARE in order to handle large addresses. You can verify this by using the dumpbin utility like this:
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE>..\..\vc\bin\dumpbin /headers devenv.exe Microsoft (R) COFF/PE Dumper Version 12.00.21005.1 Copyright (C) Microsoft Corporation. All rights reserved. Dump of file devenv.exe PE signature found File Type: EXECUTABLE IMAGE FILE HEADER VALUES 14C machine (x86) 6 number of sections 524FCB34 time date stamp Sat Oct 05 04:17:56 2013 0 file pointer to symbol table 0 number of symbols E0 size of optional header 122 characteristics Executable Application can handle large (>2GB) addresses 32 bit word machine
As you can see in the dumpbin output this x86 application can handle addresses larger than 2 gigabytes when running on a 64-bit version of the operating system. Since most of us are now running 64-bit OS versions, when Visual Studio runs out of help heap addresses in the lower 2 GB it will return a 32-bit addresses that normally would belong to the kernel like A3BA2E38. You know this by looking at the top bit of the address A which is greater than 7.
A bug in the kernel address-conversion routines incorrectly sign-extended the 32-bit address to a 64-bit address. The incorrectly sign-extended address looked like this: 0xffffffff`a3ba2e38 instead of 0x00000000`a3ba2e38. The sign-extended address is reserved for the kernel so it is not part of the user-mode address pool. Correcting the address-conversion routines eliminates the bad address and everything works as expected.
Comments
Anonymous
November 26, 2013
Awesome! This will save us sooo much time.Anonymous
January 07, 2014
This bug has been the bane of my life for longer than I care to remember. Thanks for this - it will transform my development experience!Anonymous
February 05, 2014
Yes! I see this bug at least once a day. Hopefully the hotfix will fix it for me.Anonymous
February 17, 2014
It's been two weeks and I've been using the emulator every day and haven't seen that bug since, so yay, it's fixed. Thanks!Anonymous
March 24, 2014
I've one question regarding this. I also have the same problem in windws 7 64 bit. But i'am not able to apply your solution because this hotfix is for windoes 7 sp1 only, and i do not have any service pack 1. Is there a alternate way out for me. Please,can you help me because this error is very bad.Anonymous
March 24, 2014
Hi Sudhir,Unfortunately the hotfix is only applicable to Windows 7 SP1. We would recommend you to be on the latest and greatest service pack. Please download the Windows 7 SP1 from www.microsoft.com/.../details.aspxRegards,Anghsuman NayakAnonymous
March 25, 2014
Since this article was originally posted in October 2013, we have yet to reproduce the error. Thank you, William, for your dedication to understanding and resolving this issue.Anonymous
September 22, 2014
Still reproducible on Win 7 SP1. Thank you for deep dive into root causes of this issue and suggested solution!Anonymous
July 08, 2015
support.microsoft.com/.../2588507 links to a hotfix... but nothing is at the end of that link... thanks MS.Anonymous
November 07, 2016
The comment has been removed