Scenario 2 : AssemblyBinder role instances is throwing System.IO.IOException -There is not enough space on the disk and stuck between Busy and Restarting state

Referring to my blog on Azure Cloud Service Troubleshooting Series, this is the 2nd scenario of the lab. Please make sure you have followed the lab setup instructions for Compressor application as per this, to recreate the problem.

 

Symptom

AssemblyBinder role instance of Compressor application is throwing the below unhandled exception in the azure portal blade and stuck between Busy and Restarting state.

 Unhandled Exception: There is not enough space on the disk. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.WriteCore(Byte[] buffer, Int32 offset, Int32 count) at Ionic.Zip.ZipEntry.ExtractAndCrc(Stream archiveStream, Stream targetOutput, Int16 compressionMethod, Int64 compressedFileDataSize, Int64 uncompressedSize) at Ionic.Zip.ZipEntry.ExtractToStream(Stream archiveStream, Stream output, EncryptionAlgorithm encryptionAlgorithm, Int32 expectedCrc32) at Ionic.Zip.ZipEntry.InternalExtractToBaseDir(String baseDir, String password, ZipContainer zipContainer, ZipEntrySource zipEntrySource, String fileName) at Ionic.Zip.ZipFile._InternalExtractAll(String path, Boolean overrideExtractExistingProperty) at AssemblyBinder.WorkerRole.OnStart() in D:\compressor\AssemblyBinder\WorkerRole.cs:line 56 at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.InitializeRoleInternal(RoleType roleTypeEnum) at Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.
<InitializeRole>
b__0() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart() ' [2018-08-12T14:47:25Z] Last exit time: [2018/08/12, 14:47:25.965].

 

Troubleshooting

From the error callstack it looks like this WorkerRole is performing some unzip or extract operation at OnStart() method but it is failing due to insufficient storage. The next obvious question that will come to your mind is where exactly WaWorkerHost.exe process is extracting the file ?

In order to find out the answer, the only tool that comes to my mind which always helps me in troubleshooting these kind of issues is none other than - Process Monitor !  ..... So let's take a ProcMon trace and see what we can find...

 

 

And there you go ..... Gotcha !

WaWorkerHost.exe process is writing some file in it's default temporary directory which has a maximum size of 100 MB, which may become full at some point. Upon navigating to the RoleTemp directory I found that disk space quota is indeed getting exhausted for that directory.

 

We figured out the cause of the problem, but what should I do when I am running out of space in RoleTemp location ? Here is the answer :-) ....Hence I have configured a local storage resource, and pointed the TEMP and TMP directories to point to the path of the local storage resource like below:

 

The local resource declaration must have been added to the service definition file for AssemblyBinder role.

 <?xml version="1.0" encoding="UTF-8"?>
<LocalResources>
   <LocalStorage name="FileStorage" cleanOnRoleRecycle="true" sizeInMB="200" />
</LocalResources>

This modification should be performed within the RoleEntryPoint.OnStart method.

 localResource = RoleEnvironment.GetLocalResource("FileStorage");
Environment.SetEnvironmentVariable("TMP", localResource.RootPath);
Environment.SetEnvironmentVariable("TEMP", localResource.RootPath);

 

I hope you have got an idea how to troubleshoot default TEMP folder size issues in Azure Cloud Service.

Happy Learning !