Exceptions Received After Running Successful Modification Operations with Exchange 2010 SP1
In this case, what we were running into, is that when commands are run that make modifications to objects, either in the Exchange Management Console (EMC) or Exchange Management Shell (EMS), an exception is returned:
Error : The cmdlet extension agent with the index 0 has thrown an exception in OnComplete(). The exception is: System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
This was occurring on servers in 2 of the 3 sites the customer's environment was divided into. In one site, the same modifications could be made without error.
Discussing the behavior with one of my colleagues brought up the fact that this might be related to Administrator Audit Logging, which is enabled by default in Exchange 2010 SP1. We ran Set-AdminAuditLogConfig -AdminAuditLogEnabled $False to disable the logging and the exceptions ceased, so it was indeed the logging causing them.
After a bit of searching, I found the information regarding the mailbox we put the audit logging in and how we save it:
The logs will be saved to a dedicated folder under “SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}”. To reduce latency, Exchange Web Services (EWS) is used to send the asynchronous call to CreateItem in the mailbox.
What we had in this case, and I determined it by investigating the IIS logs from the Mailbox Server, is that our calls to EWS were going to the Default Web Site, and the EWS virtual directory was not under the Default Web Site (DWS). There were 2 web sites set up in this customer’s case, and EWS was under their alternate site.
The IIS logs showed:
2010-11-19 13:35:42 172.26.4.21 POST /EWS/Exchange.asmx - 443 - 172.24.152.80 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.4952) 404 0 0 109
So, the question was, why were our EWS calls going to the DWS, instead of the alternate site? After all, standard EWS calls from client access were working OK.
The reason was because we were going between sites, and thus CAS proxying was occurring. When this happens, we use the InternalNLBBypassUrl path specified for the EWS virtual directory. To see the configuration for the virtual directory we run:
Get-WebServicesVirtualDirectory -Server 'servername'.
And what we saw was:
InternalNLBBypassUrl : https://servername.domain.int/EWS/Exchange.asmx
(note, names have been changed)
This resolved to the IP used for the Default Web Site, and so we would try to go there and find no EWS folder.
There are a couple of possible solutions. The simplest is just to set InternalNLBBypassUrl to the same value being used for InternalUrl, which was https://mail.company.com/ews/exchange.asmx. Although this address goes through NLB, it still works fine.
The InternalNLBBypassURL was changed, and after that, no more exceptions.
--------------------
I had another occurrence of the same exceptions, and in that case, the solution was as follows:
We had a Host Name set in the https binding in IIS, and it was configured with the load-balanced/external name being used by the customer.
This meant that IIS wasn't responding to the internal name, and this is what is used by the auditing extension for the EWS calls to the System Mailbox used for it auditing.
Once we removed the host name from the bindings (leaving the bindings as *), everything worked without exceptions.