Udostępnij za pośrednictwem

Detecting ephemeral port exhaustion

  • Artykuł

Symptoms

When Windows or Windows Server is out of ephemeral/outbound/dynamic network ports, it will not be able to establish any outbound network connections. This results in a lot of connection failures such as database and/or domain controller connections. If the system is not responding, then try increasing the port range (discussed below) - this change is effective immediately. If the system immediately recovers, then you know you are dealing with this issue. You can either increase the port range which may just delay the problem or you can try to identify the cause.

Detecting user mode port leaks

Ephemeral ports are range of ports that Windows and Windows Server use for outbound communications over the TCP/IP network protocol. When an outbound connection is finished, the port associated to the connection is put into a TIMED_WAIT state for two minutes by default. This allows any lingering packets on the network to be ignored. Windows Server 2008 and later use the IANA range which uses the ports between 49152 and 65535 providing 16,383 ports.

Some applications and services such as Microsoft Exchange Server CAS servers can be very “chatty” and might actually use all 16,383 ports within a two minute time period. The result is connection failures similar to “Couldn’t connect to X, due to no ports available from the end point mapper”.

If you suspect ephemeral port exhaustion, then consider running the following Powershell script called “Log-EphemeralPortStats.ps1” at
https://1drv.ms/f/s!AhuJirRUDDbmkotkPocbTrN0wgKB7Q

Warning: This script is provided as sample code only. Please review it and use at your own risk.

Be aware that this only detects user mode port leaks. If a driver (kernel mode) is leaking ports, then a complete memory dump must be sent to Microsoft Support for analysis to know which driver is responsible. The memory dump must be taken when the system has leaked a significant number of ports.

This script is designed to run in an infinite loop of 1 minute sleep intervals and write to a log file called “EphemeralPortStats.log”. Here is an example of the output it produces:

Computer       DateTime             LocalAddress  #OfEPortsInUse Max#OfEPorts %EPortUsage #OfTcpListeningPorts #OfPids
--------       --------             ------------  -------------- ------------ ----------- -------------------- -------
ETCHEDCHAMPION 8/9/2013 12:37:42 PM 127.0.0.1                  6        16384           0                   15      11
ETCHEDCHAMPION 8/9/2013 12:37:42 PM 172.18.96.192              3        16384           0                   15      10
ETCHEDCHAMPION 8/9/2013 12:37:42 PM 192.168.1.2               69        16384         0.4                   15      17

This script is intended to be ran from the console of the computer suspected to be running low on ephemeral ports and to leave it running. Periodically review the log to see if there was any ephemeral port exhaustion detected.

This script gets the port range from:

netsh int ipv4 show dynamicportrange tcp

Then, correlates this information with the output of:

netstat –ano –p tcp

PsExec can be potentially used to get this information from remote computers, but keep in mind that passwords used in PsExec are sent in the clear over the network.

My PFE colleagues and customers have used this script quit a bit and I hope it will help you as well.

Also, as a temporary work-around, the ephemeral/dynamic/outbound port range can be increased using the following command:

netsh int ipv4 set dynamicport tcp start=10000 num=55535

This change takes affect immediately. No need to restart. No need to reboot. This is about three times larger than the default. If this change has a positive affect when the problem occurs, then you know that this is the issue your system is dealing with.

To set the port range back to default, use the following command:

netsh int ipv4 set dynamicport tcp start=49152 num=16384

To check the port range:

netsh int ipv4 show dynamicportrange tcp

Once you have identify the process consuming the ports, contact the developer of the process for a fix.

Also, consider decreasing the TIMED WAIT delay using the TcpTimedWaitDelay registry key.

Detecting kernel mode port leaks

All of the above covers user mode port leaks - meaning leaks originating from processes. If a driver (kernel mode) leaks a port, then netstat will not be able to report it. Kernel mode port leaks can only be detected by inducing a complete memory dump and then analyzing the dump with the MEX debug extension. Load MEX into WinDBG and then run the following commands to help identify ports associated with drivers.

!mex.afd -conn -report -verbose

!afd -endp -report

!tcpip -p

Once you identify the driver consumer the ports, contact the developer of the driver.

If you suspect a Windows Server 2008 R2 system might be running out of ephemeral ports, then apply the following hotfix:

Kernel sockets leak on a multiprocessor computer that is running Windows Server 2008 R2 or Windows 7
https://support.microsoft.com/en-us/kb/2577795

Windows Server 2012 and later are already patched with this fix.

UPDATE

Windows 10 and Windows Server 2012 R2 introduced the "q" parameter in netstat. This parameter displays all connections, listening ports, and bound non-listening TCP ports. Bound non-listening ports may or may not be associated with an active connection.

Comments

  • Anonymous
    January 01, 2003
    Ah, it must be 4 minutes. My memory must be failing. I was thinking of TcpTimedWaitDelay at technet.microsoft.com/.../cc938217.aspx. It says 4 minutes.

  • Anonymous
    January 01, 2003
    Thank you for the useful information.

  • Anonymous
    August 12, 2013
    Two minutes or four minutes?  I thought four minutes as that is 2xMSL, with each MSL being 120 seconds (2 minutes). TechNet says 4 minutes as well.

  • Anonymous
    August 11, 2014
    Thanks Client awesome script. Very useful. Should be somewhere in Windows monitored or an Event log if running out of ports.
    Can you put it to http://gallery.technet.microsoft.com/scriptcenter/site/upload as well please. Thanks

  • Anonymous
    February 13, 2015
    So what next? How to troubleshoot this? Owner of process is System

  • Anonymous
    March 02, 2015
    To Clint Huffman - Can you please provide the powershell script source code as cannot download from the link provided -https://skydrive.live.com/redir?resid=E6360C54B48A891B!5328

  • Anonymous
    March 04, 2015
    Hi Clint: Good article. Can you please provide the powershell script source sode as cannot download from the link provided.

  • Anonymous
    March 30, 2015
    Thanks for the script. Should this list the actual detail of ports in use or just the number of ports? i.e. if ports start to get exhausted will we need to then run 'netstat -ano -p tcp' to get the detail?

    This is generally happening out of hours so we would not be on the console to check. By the time we do the ports are fine again.

  • Anonymous
    May 15, 2015
    Hi Clint,
    i've taken the liberty of adapting your script to list the number of ephemeral ports in ESTBLISHED, TIME_WAIT and SYN_RECEIVED states, and also listing the top 5 PIDs, with process name and number of ports assigned. any suggestions as to what i can do with it? do you have a listing on the script repository i could append it to?

    • Anonymous
      September 06, 2016
      You can send it directly to me at clinth@microsoft.com and I'll update the script. Thanks!
    • Anonymous
      May 16, 2017
      I don't have a repository for this, but you are welcome to publish the script/code anyway you want.

  • Anonymous
    July 14, 2015
    Hi, How do you fix the issue if you have this happening to an Exchange server?

    • Anonymous
      September 06, 2016
      By increasing the port range with the following command:netsh int ipv4 set dynamicport tcp start=10000 num=55535

  • Anonymous
    July 29, 2015
    I see a lot of talk on how to detect this, but not how to troubleshoot the underlying cause. In my case it is also local SYSTEM and connecting to SQL. But we do not know which of the many services tools and tasks is behind it, as they are hidden behind system.exe

    • Anonymous
      September 06, 2016
      Correct. This blog post only covers user mode port leaks. I'll update it to include kernel mode.

  • Anonymous
    October 02, 2015
    For a quick check of total and used ephemeral ports of a single service (so in my case, a SQL server) do this:

    PS C:UsersRack> netsh int ipv4 show dynamicportrange tcp

    Protocol tcp Dynamic Port Range
    ---------------------------------
    Start Port : 49152
    Number of Ports : 16384

    PS C:UsersRack> netstat -ano | findstr 1433 | measure


    Count : 910
    Average :
    Sum :
    Maximum :
    Minimum :
    Property :

    It will only count the MSSQL connections, but in my case those were his only ephemeral. You can also do this with a couple different ports to get numbers you can sum up.

  • Anonymous
    March 08, 2016
    Do you have the same script working for monitoring UDP port? I tried modifying the script for UDP but not luck.M S Ali

    • Anonymous
      September 06, 2016
      The comment has been removed

  • Anonymous
    March 25, 2016
    Once again, thanks Clint! This is exactly what i´m looking for!

  • Anonymous
    September 01, 2016
    Sorry for the delay. I updated this blog post with a working link to the script.