共用方式為

W3SVC 1310 and who changed my worker process affinity

  • 發行項

 

Randomly, IE will show follow error when customer accessing their web application. Meantime, ASP.NET event 1310 will recorded as well. According to KB 821268 and MSDN, this is the recommended setting for a two CPU system (Dual core is considered as 2 CPU).

Here is what we see in the browser:

Parser Error Message: The value for 'minFreeThreads' must be less than the thread pool limit of 100 threads.

Source Error:

Line 136: </system.data>

Line 137: <system.web>

Line 138: <httpRuntime minFreeThreads="176" minLocalRequestFreeThreads="152" />

Line 139: <processModel maxWorkerThreads="100" maxIoThreads="100" minWorkerThreads="50" />

Line 140:

Source File: C:\WINNT\Microsoft.NET\Framework\v2.0.50727\Config\machine.config Line: 138

Also, we could see 1310 event in the application event log.

Event Type: Warning

Event Source: ASP.NET 2.0.50727.0

Event Category: Web Event

Event ID: 1310

Date: 2009/3/27

Time: 下午 02:54:23

User: N/A

Computer: ZW

Description:

Event code: 3008

Event message: A configuration error has occurred.

…..

…..

Exception information:

    Exception type: ConfigurationErrorsException

    Exception message: The value for 'minFreeThreads' must be less than the thread pool limit of 100 threads. (C:\WINNT\Microsoft.NET\Framework\v2.0.50727\Config\machine.config line 138)

 

Thread information:

    Thread ID: 7

    Thread account name: NT AUTHORITY\NETWORK SERVICE

    Is impersonating: False

    Stack trace: at System.Web.HttpRuntime.FirstRequestInit(HttpContext context)

   at System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context)

   at System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr)

Here is the MSDN suggested settings and its details description. As you can see, customer’s setting is exactly what we suggested.

Configuration Setting

Recommended Value

maxconnection

12 * #CPUs

maxIoThreads

100

maxWorkerThreads

100

minFreeThreads

88 * #CPUs

minLocalRequestFreeThreads

76 * #CPUs

Then why this error happens? Based on MSDN, the configuration item in processModel will be automatically multiplied by the number of available CPUs, and MaxWorkerThreads must be equal or greater than minFreeThreads in httpRuntime. For customer’s setting, threads available to the worker process to service request is 100*2-176=24, which are 12 threads per CPU.

maxWorkerThreads

5 to 100

Configures the maximum amount of worker threads to be used for the process on a per-CPU basis. For example, if this value is 25 on a single-processor server, ASP.NET uses the runtime APIs to set the process limit to 25. On a two-processor server, the limit is set to 50. The default is 20. The value of maxWorkerThreads must be equal to or greater than the minFreeThread attribute setting in the <httpRuntime> configuration section.

Now, come back to the original question, why this error happens? The only factor can affect this is process affinity. Now, we are able to reproduce this problem easily.

1. Create a web application contains hello.aspx

2. Open IE and visit hello.aspx, the page loaded correctly.

3. Open task manager, right click the w3wp.exe and select “set affinity”. Change the worker process’s affinity by un-check one CPU.

4. Open the machine.confog or web.config, make any change and then un-do this change. Save the config file.

5. Open IE and visit hello.aspx again, IE shows the errors and able to find 1310 in event log as well.

Here comes to the new question, who changed my worker process’s affinity? There will be 3 scenarios which may change worker process’s affinity.

1. Other application changed IIS worker process’s affinity by call SetProcessAffinityMask, example Task Manager.

2. User changed the worker process’s affinity by adsutil.vbs. This is done through W3SVC and finally, called SetProcessAffinityMask.

3. The code running in the worker process itself changed the affinity by call SetProcessAffinityMask.

For scenario 1, I don’t know how to find out the culprit application. My Windows colleague send me a checked build of Kernel32.dll, with this build, any process call SetProcessAffinityMask will be crashed at once. I don’t have a chance to test this yet.

I finally find out the problem by attach debugger to worker process and W3SVC service and set breakpoint at SetProcessAffinityMask. Here is the call stack in the worker process’s debugger’s log. A 3rd party DLL used by customer’s web application is calling SetProcessAffinityMask!

# ChildEBP RetAddr 

kernel32!SetProcessAffinityMask(void * hProcess = 0xffffffff, unsigned long dwProcessAffinityMask = 1)

WARNING: Stack unwind information not available. Following frames may be wrong.

AStrangeComponent!DllCanUnloadNow+0x50a76

AStrangeComponent!DllCanUnloadNow+0x10f9a5

…………..

Regards,

 

Zhao Wei

Comments

  • Anonymous
    June 15, 2009
    Three months ago, I have a blog talking about worker process affinity . Now, we have a similar issue