Freigeben über


Windows Azure: Roles Repetitively Starved of Ports

You just upgraded the OS family or in case you have auto upgrade on for Windows Azure there is a chance that you might see the following errors in case you are using Storage Client and SQL Azure .Net APIs. The other symptom could be sluggishness/slowness/delay in any storage calls like blobs/queues/tables or even SQL Azure calls in case the application uses both Windows storage and SQL Azure. It could also afflict you in case Windows Azure diagnostics are being run as Windows Azure Diagnostics write to Storage.   

Rebooting the VM resolves the issue only for it to come back after a certain time.

Storage 

Server Error in ‘/’ Application.

An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified

Stack Trace:

[SocketException (0x2747): An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full]

Systen. Net. Sockets. Socket. DOBi nd(EndPoint endPoi ntSnapshot, SocketAddress socketAddress) t7985914

System. Net. Sockets. Socket. Internal Bi nd(EndPoint localEP) +243

Systen.Net.Sockets.Socket.BeginConnectEx(EndPoint remoteEP, Boolean flowcontext, AsyncCallback callback, Object state) +307

System. Net. Sockets. Socket. Unsafeßeqi nConnect(EndPoint remoteEP, AsyncCal lback callback, Object state) t71

System.Net.ServicePoint.ConnectSocketlnternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, lAsyncResult asyncResult, In

[WebException: Unable to connect to the remote server]

Microsoft.WindowsAzure.StorageClient.Tasks.Task’1.get_Result() 

Microsoft.WindowsAzure.StorageClient.Tasks.Task’l.ExecuteAndwait() 

Microsoft.WindowsAzure. StorageClient. <LazytnumerateSegmented>d_01.MoveNext()

System.Linq.Enumerable.Count(IEnumerable’1 source) 

 

SQL Server

 Server Error in ‘I’ Application. 
 A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where d originated in the code.
 Exception Details: System.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection SQL Server)
 Source Error:
 An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
 Stack Trace:
 [SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server 
 System.Data.ProviderBase.DbConnectionPool .GetConnection(DbConnection owningObject) +706
 System.Data.ProviderBase.DbConnectionFactory.CetConnection(DbConnection owningConnection) +89
 System.Data.Providerßase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory) +6425518
 System.Data.SqlClient.SqlConnection.Open() +300
 System.Data.EntityClient.EntityConnection.OpenStoreConnectionlf(Boolean openCondition, DbConnection storeConnectionToOpen, DbConnection originalConnection, String exceptionCode, String attemptedOperation, Boolean& closeStoreConnectic
 [EntityException: The underlyin9 provider failed on Open.]
 System.Data.EntityClient.EntityConnection.OpenStoreConnectionlf(Boolean opencondition, DbConnection storeConnectionToOpen, DbConnection originalconnection, String exceptioncode, String attemptedoperation, Boolean& closeStoreConnectic
System.Data.EntityClient.EntityConnection.Open() +142
 System.Data.Objects.ObjectContext.EnsureConnection() +97
 System.Data.Ob1ects.Ob1ectQuery’1.GetResults(Nullabl&1 forMergeoption) +66
 System.Data.Objects.ObjectQuery 1.System.Collections .Generic.IEnumerable<T>.GetEnumeratorO ÷47
 System.Linq.Enumerable.SingleOrDefault(IEnumerable 1 source) +271 
 System.Linq.Queryable.SingleOrDefault(IQueryabl&1 source) +383 
 System.Data.Entity.Internal .Linq.InternalSet’1.FindlnStore(WrappedEntityKey key, String keyValuesParamName) +779 
 System.Data.Entity.Internal .Linq.InternalSet’1.Find(Object[] keyvalues) +182 
 System.Web.Providers.DefaultSessionStateProvider .SetAndReleaseltemExclusive(HttpContext context, String id, SessionStateStoreData item, Object lockld, Boolean newltem) +259 
 System.Web.SessionState.SessionStateModule.OnReleaseState(Object source, EventArgs eventArgs) +929 
 System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +80
 System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +270

 

Open a command prompt in elevated mode and type netstat - anob

If you see a pattern as follows where a lot of ports are held by a single PID as below 

Active Connections

Proto Local Address Foreign Address State PID

[MonAgentHost.exe]

 TCP [::1]:80 [::1]:52108 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:80 [::1]:52109 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:80 [::1]:52110 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:80 [::1]:52111 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:80 [::1]:52112 ESTABLISHED 4 Can not obtain ownership information  

 TCP [::1]:80 [::1]:52108 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:80 [::1]:52109 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:80 [::1]:52110 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:80 [::1]:52111 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:80 [::1]:52112 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:80 [::1]:52108 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:80 [::1]:52109 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:80 [::1]:52110 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:80 [::1]:52111 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:80 [::1]:52112 ESTABLISHED 4 Can not obtain ownership information 

 TCP [::1]:80 [::1]:52108 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:80 [::1]:52109 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:80 [::1]:52110 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:80 [::1]:52111 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:80 [::1]:52112 ESTABLISHED 4 Can not obtain ownership information

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 TCP [::1]:52108 [::1]:80 ESTABLISHED 3188 [WaIISHost.exe]

 

As we can see above that a single process with ID 4 or ID 3188 is holding numerous connections open to the server.

Do an RDP to the instance and please check if KB2750149 or KB2805227 is installed on the machine. The actual cause and resolution steps can be found at https://blogs.msdn.com/b/windowsazurestorage/archive/2013/08/08/net-clients-encountering-port-exhaustion-after-installing-kb2750149-or-kb2805227.aspx