UM 2010 voicemail delivery failure to Hub Transport
With voicemail delivery issues, the first place to verify is to go to the voicemail folder under \Program Files\Microsoft\Exchange Server\V14\UnifiedMessaging\
If the folder is empty then the voicemail files have left the UM and on its way to the Hub Transport.
If you see voicemail audio files there it means the connection between UM and HT is broken somewhere. This can arise because of one of the following scenarios.
- The HT receive connector is mis-configured. Check the AUTH on the receive connectors
- Verify if you can telnet on hub transport's IP and port number 25
- Verify if HT FQDN is resolving to the correct IP
- Check if any anti-virus software is installed
As an example, see the following events that may come when voicemail delivery is failing.
Event IDs 1436, 1423 and 1446
- <Event xmlns="https://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="MSExchange Unified Messaging" />
<EventID Qualifiers="49156">1446</EventID>
<Level>2</Level>
<Task>2</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-12-20T16:50:40.000000000Z" />
<EventRecordID>252718</EventRecordID>
<Channel>Application</Channel>
<Computer>optimus.com</Computer>
<Security />
</System>
- <EventData>
<Data>C:\Program Files\Microsoft\Exchange Server\V14\UnifiedMessaging\voicemail\1c3ebe7b-c1a4-4b31-8ef4-ebfd978a6239.txt</Data>
<Data>23</Data>
</EventData>
</Event>
- <Event xmlns="https://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="MSExchange Unified Messaging" />
<EventID Qualifiers="32772">1436</EventID>
<Level>3</Level>
<Task>2</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-12-20T16:52:17.000000000Z" />
<EventRecordID>252730</EventRecordID>
<Channel>Application</Channel>
<Computer>optimus.com</Computer>
<Security />
</System>
- <EventData>
<Data>SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}@optimus.com</Data>
<Data>Microsoft.Exchange.Data.Storage.FolderSaveException: An error occurred while saving the changes on the folder "vAAAAAAbVfogqmYRzZvIAKoAL8RaCQBIAFNSVjYwLUVYMS1XMks4LnctZHVidXF1ZS5rMTIuaWEudXMAL289V2VzdGVybiBEdWJ1cXVlIFNjaG9vbCBEaXN0cmljdC9vdT1FeGNoYW5nZSBBZG1pbmlzdHJhdGl2ZSBHcm91cCAoRllESUJPSEYyM1NQRExUKS9jbj1SZWNpcGllbnRzL2NuPVN5c3RlbU1haWxib3h7ZTBkYzFjMjktODkA/w==". Error details: Failed, Property: [0x3428000b] HasUMReportData, PropertyErrorCode: AccessDenied, PropertyErrorDescription: . ---> Microsoft.Exchange.Data.Storage.PropertyErrorException: Property: [0x3428000b] HasUMReportData, PropertyErrorCode: AccessDenied, PropertyErrorDescription: . at Microsoft.Exchange.Data.Storage.StoreObjectPropertyBag.FlushChanges() at Microsoft.Exchange.Data.Storage.CoreMailboxObject.Save() --- End of inner exception stack trace --- at Microsoft.Exchange.Data.Storage.MailboxStoreObject.Save() at Microsoft.Exchange.UM.UMCommon.UMReportUtil.SetMailboxExtendedProperty(MailboxSession session, Boolean hasUMReportData) at Microsoft.Exchange.UM.UMCore.CDRPipelineContext.TrySetExtendedProperty(MailboxSession mbxSession)</Data>
</EventData>
</Event>
- <Event xmlns="https://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="MSExchange Unified Messaging" />
<EventID Qualifiers="49156">1423</EventID>
<Level>2</Level>
<Task>2</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-12-20T16:51:59.000000000Z" />
<EventRecordID>252729</EventRecordID>
<Channel>Application</Channel>
<Computer>optimus.com</Computer>
<Security />
</System>
- <EventData>
<Data>C:\Program Files\Microsoft\Exchange Server\V14\UnifiedMessaging\voicemail\4d4b108e-a592-4eae-8aeb-0089baa4b537.txt</Data>
<Data>Microsoft.Exchange.UM.UMCore.SmtpSubmissionException: Submission to the Hub Transport server failed. The operation will be retried. --->
System.IO.IOException: Unable to read data from the transport connection:
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because
connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) --- End of inner exception stack trace --- at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) at Microsoft.Exchange.Net.ExSmtpClient.SmtpSslStream.Handshake() at Microsoft.Exchange.Net.ExSmtpClient.SmtpTalk.StartTls() at Microsoft.Exchange.Net.ExSmtpClient.SmtpClient.Submit(Boolean disableDelayedAck) at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.SubmitMessage(MessageItem message, String senderAddress, String recipientAddress, OutboundConversionOptions submissionConversionOptions, InternalExchangeServer smtpServer) at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.SubmitMessage(MessageItem message, String senderAddress, String recipientAddress, OutboundConversionOptions submissionConversionOptions, String requestId) --- End of inner exception stack trace --- Server stack trace: at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.HandleTransientSmtpFailure(Exception e, InternalExchangeServer smtpServer, String recipientAddress) at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.SubmitMessage(MessageItem message, String senderAddress, String recipientAddress, OutboundConversionOptions submissionConversionOptions, String requestId) at Microsoft.Exchange.UM.UMCore.SmtpSubmitStage.InternalDoSynchronousWork() at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs) at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase) at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData) at Microsoft.Exchange.UM.UMCore.SynchronousPipelineStageBase.SynchronousWorkDelegate.EndInvoke(IAsyncResult result) at Microsoft.Exchange.UM.UMCore.SynchronousPipelineStageBase.EndSynchronousWork(IAsyncResult r)</Data>
</EventData>
</Event>
In the above example, uninstalling the anti-virus software and rebooting the UM server helped. Apparently, there were recent anti-virus definitions applied to the UM server that may cause this.