Dela via

Troubleshooting issues where the hybrid migration endpoint cannot be created

  • Artikel

This article is meant to help you troubleshoot scenarios where the connection from Office365 to your hybrid migration server cannot be established: it can  be about the first time when you attempt to create the migration endpoint, but it also applies to cases where the endpoint was created a while ago and now you cannot connect anymore to the migration server and none of the mailboxes can be migrated.

It is also good to know that the steps mentioned here will not necessarily deliver you the solution, but for sure they will help you in isolating the issue.

Introduction to remote move requests

I will take this opportunity to discuss about the process of remote moves and about the factors that matter in a hybrid scenario, where you would like to move your mailboxes to the cloud.

After a hybrid setup is deployed, most of the customers will create the migration endpoint in Office365 Exchange Admin Center, as a first step in the migration process.

The interesting part is that if you managed to successfully run the hybrid wizard, you would expect  to have the same success when creating the migration endpoint. This is a wrong expectation, because in some cases, for example due to firewall configuration (URL filtering or IP restriction), your attempt to create a migration endpoint on Office365 side can fail.

It is worth mentioning that with the new Hybrid Configuration Wizard you will get the benefit of having the “remote move” migration endpoint automatically created into Office365 EAC. But like I said above, due to some configuration issues, even the new HCW might be unable to create the endpoint on your behalf.

Soon we will simplify this process for our customers, especially when they just need to perform a remote move migration to the cloud, by giving them the option to deploy a simplified hybrid configuration. For example, in the case where you do not need free-busy or centralized mailflow features , you can choose only the options that you are interested in.

Before starting the migration, you must ensure that your on-premises Exchange servers meet the prerequisites. Here I want to mention that you need to first enable the MRSproxy on each of your CAS servers, as this endpoint is responsible for the accepting the requests for remote moves and to proxy them to the servers that are running Mailbox Replication Service.

The ability of a Client Access server to accept these MRS requests is disabled by default. To allow the Client Access server to accept incoming move requests, you must enable the MRS Proxy endpoint. You can refer to below article for more details:

https://technet.microsoft.com/en-us/library/dn155787(v=exchg.150).aspx

 

The Mailbox Replication Service  is the service responsible for cross-forest mailbox moves and remote move migrations between your on-premises Exchange organization and Exchange Online:

  • in Exchange 2010 the MRS service is running on the CAS server.
  • in Exchange 2013, MRS is running on the Mailbox server role; during cross-forest and remote move migrations, a Client Access server acts as a proxy for incoming move requests for the Mailbox server.
  • in Exchange 2016 we have a new architecture, where the Mailbox server integrates both CAS and Mailbox roles, hence the MRS service is running on the Mailbox server and the CAS role acts first like a proxy, as in Exchange 2013 case.

Creating migration endpoint in Office365

In Office365 EAC, you are basically provided with 2 options  to create a “Remove Move” migration endpoint:

a) First you can use Autodiscover and mention below parameters:

- the email address of an on-premises mailbox: this will be used by Autodiscover service to detect the server running the MRSproxy endpoint.

-the local Exchange administrator credentials: these will be used to connect remotely to your hybrid server and to authorize the incoming migration requests.

You can see an example in the next picture :

 

autodiscover_endpoint1

b) If Autodiscover can’t detect your migration server, you can input manually the external hostname of your server, as in below example:

 

failed_endpoint1

 

Below diagram explains how the cloud MRSproxy is connecting  to a server from your organization, in order to migrate a mailbox:

 

In above example we have an environment with multiple CAS servers and a Load Balancer in front of them:

  •  Exchange 2013 CAS server, that has MRSproxy enabled and acts like a proxy for the MRS requests.
  •  Exchange 2013 Mailbox Server, that runs the Mailbox Replication Service.
  • Exchange 2013 multi-role server, where we have both MRSproxy and Mailbox Replication Service on a single place.

When we try to establish a connection from the cloud with the on-premises MRSproxy endpoint, the request is passing through the firewall, from here it is forwarded to the load balancer and this can choose one of the 2 CAS servers as a destination for the incoming MRS request.

Configuration requirements for remote move requests

Firewall considerations:

Let's  imagine an environment configured like in the mentioned example. Here we can have a lot of factors that can prevent the incoming MRS calls reaching the destination server.

If you are not aware about the way your firewall is configured or you are having a third-party company that is managing it, it is strongly recommended to double check with the responsible team that the firewall meets below pre-requisites:

  • Your firewall must allow the incoming connections on URL containing /ews/mrsproxy.svc, in case it is using URL filtering, or if possible, you can allow all the incoming traffic on port 443.
  • If you are using an IP filtering configuration in the firewall, you should make sure that you have the updated Exhange Online IP list in the firewall configuration; you can find IP ranges list for Exchange Online in below article:

https://support.office.com/en-us/article/Office-365-URLs-and-IP-address-ranges-8548a211-3fe7-47cb-abb1-355ea5aa88a2?ui=en-US&rs=en-US&ad=US#BKMK_EXO

  • In case you want to be informed every time when we update this list, you can choose the option “ Subscribe via RSS " from above url.
  • If your firewall requires pre-authentication or you are using a solution that is performing SSL offloading in front of CAS servers, please note that these configurations are not supported in remote moves scenarios, due to the fact the on-premises MRS expects to receive the incoming traffic over port 443 with SSL encryption and it refuses the connection if these conditions are not met.
  • If you are using a TMG server, make sure that you created a rule to publish EWS service, in order for this service to be accessible from outside the network.

Load Balancer considerations:

  • When you have multiple Exchange 2010 Client Access Servers in the organization with a Load Balancers in front of them, it is strongly recommended that the load balancer to maintain the session affinity(or IP persistence), more precisely all the migration requests for specific mailbox should be processed by the same CAS server.
  • If you have Exchange 2010 with a such miss-configured load balancer, you will most likely experience slowness during the remote moves, due to a lot of transient errors like this ones: The remote endpoint no longer recognizes this sequence. This is most likely due to an abort on the remote endpoint. The value of wsrm:Identifier is not a known Sequence identifier. The reliable session was faulted" or "Couldn't switch the mailbox into Sync Source mode/SourceMailboxAlreadyBeingMovedTransientException".
  •  Exchange Server 2013 and Exchange Server 2016 do not require session affinity for the load balancing layer. This means that regardless of where a connection originates, it always connects, either directly or by proxy, to the Mailbox server that has an active copy of the database where that user’s mailbox is located.
  • Some of the existing Load Balancers might not support this configuration and are providing only the option to forward the incoming requests based on the CAS servers availability/loading. In this scenario, the suggested option would be to bypass the Load Balancer during the migration, to better isolate the issue.

Troubleshooting steps

If you managed to understand the process behind the migration endpoint creation and how MRS works with remote moves, the troubleshooting process will become much easier and you will have the answers to these very common questions:

Why is Office365 unable to detect my migration server?

What are the firewall requirements for remote move migrations?

How can I make sure that the firewall is not blocking the migration requests?

What is the server from my organization that should process the incoming MRSproxy requests?

Where do I see the incoming requests for my MRSproxy endpoint?

In these scenarios we should agree that the cause of the issue lies on the on-premises environment, where the communication between cloud MRSproxy  and local MRSproxy service is failing somewhere locally.

 

These are the most relevant steps that we should perform on-premises during the troubleshooting phase:

  • check the EWS configuration on each CAS server
  • test MRS health on the on-premises Exchange Servers
  • browse to the MRSproxy external URL
  • use Test-MigrationServerAvailability from Exchange Online
  • check the IIS logs on CAS servers
  • check HttpProxy and HTTPERR  logs from CAS servers

The order of the steps might vary from one case to another, so it depends on you to choose the order in which you want to perform them.

Once you managed to complete all these steps , you will most likely be able to identify the cause of the issue.

1. Check the EWS configuration on each CAS server

Ensure that the MRS proxy endpoint is enabled on every CAS server that will be used in the migration process.

Collect below output from local Exchange Management Shell:

Get-WebServicesVirtualDirectory|fl ExternalAuthenticationMethods,Externalurl,MRSproxyEnabled,Server

The output can be similar to below example:

ExternalAuthenticationMethods : {Basic, Ntlm, WindowsIntegrated, WSSecurity, OAuth}

ExternalUrl                   : https://mail.pronicsi.com/EWS/Exchange.asmx

MRSProxyEnabled               : True

Server                        : EX2015

Here is important to check that Windows authentication and Basic authentication are enabled and that you have the correct ExternalUrl populated on each CAS server.

 

2. Confirm that the local MRSproxy service works fine

  • Check first that the MSExchangeMailboxReplication.exe service is running on the migration servers.
  • Perform some local move requests to confirm that MRS service works fine.
  • You can check if Microsoft Exchange Mailbox Replication service is consistent on your Mailbox/ CAS servers by using the command “ Test-MRSHealth ” . Please refer to below article for more details:

https://technet.microsoft.com/en-us/library/ee332325%28v=exchg.160%29.aspx?f=255&MSPPError=-2147217396

3. Browse to the MRSproxy external URL

  • This can help you to confirm that your MRS proxy URL is reachable from outside the network, by accessing it from an internet browser.
  • Since MRSproxy endpoint is running under Exhange Web Services, it is expected that all the MRS request will be encapsulated in EWS calls, so the URL should have this format: https://mail.contoso.com/ews/mrsproxy.svc
  • The middle portion “mail.contoso.com” is the external URL for your EWS service.
  • If MRSproxy service is accessible from outside the network, you should receive a credential prompt.
  • If no credential prompt occurs, this indicates that your EWS service is not accessible externally and the firewall could block the MRS incoming requests, so you need to double check the firewall configuration and see if you can find any relevant logs in the firewall.

 

4. Test-MigrationServerAvailability

This test is very useful because you can see more details about why the connection with your on-prem server cannot be created and you can use the received error message or the HTTP response code to investigate the issue further.

You need to connect to Exchange Online from PowerShell and test the connection with the migration server( for remote moves), using below commands:

$Cred=Get-Credential (input the local Exchange Admin credentials)

Test-MigrationServerAvailability -ExchangeRemoteMove -Autodiscover -EmailAddress user@contoso.com -Credentials $Cred

If you want to test the connection without Autodiscover, you can use below command:

Test-MigrationServerAvailability -ExchangeRemoteMove -RemoteServer mail.contoso.com -Credentials(Get-Credential)

If both of the tests failed, you might have to check the firewall logs for any incoming MRS request, to see if these are not blocked or if they are allowed, to see what is the destination CAS server.

 

 5.Check the IIS logs from CAS servers

If you are sure that the firewall is not the problem, you can see further if the incoming MRS calls have reached out to correct destination, by checking the IIS logs on the CAS servers.

Depending on Exchange Server versions and the roles installed, the CAS server can process the request directly or it may forward it to a Mailbox server, that hosts the mailboxes that you want to migrate.

If you have multiple servers in the organization, some of them might have important services disabled or not enough resources assigned; these are usually called "passive" servers. If the migration request will somehow reach out to one of these servers, we can expect to have a failure.

Therefore, to avoid these doubts, you can check the IIS logs on the destination CAS servers and you will clarify if the migration calls are present there or not.

The location of the IIS logs is by default C:\inetpub\logs\LogFiles and in these logs we should search for entries that contain EWS/mrsproxy.svc as destination URL.

Note that by default the IIS logs are stamped in UTC time format, regardless the time that you set on your server.

Every call to MRS service should be generating some IIS logs. So you should try either to browse to the MRSproxy external URL or to use the command Test-MigrationServerAvailability to reproduce the issue and in a few minutes you can check the IIS logs, to find any entries for EWS/mrsproxy.svc.

If no IIS entry can be found on the CAS servers that are meant to be the destination servers or if the migration requests can be seen on the passive/wrong CAS server logs , this indicates clearly that you should reconfigure the firewall rules or the load balancers, if you are using any.

Please be aware that each incoming request for the on-premises MRSproxy will generate several entries/calls on the IIS logs, so if you will see multiple lines with mrsproxy.svc, consider it normal.

In the next picture we ran Test-MigrationServerAvailability from Exchange Online PowerShell, so we can see an example with the IIS log entry generated by a failed connection to MRSproxy:

test-migrationserver

When checking the IIS logs in the hybrid server, we see below entry:

2016-09-15 08:20:18 10.2.0.5 POST /EWS/mrsproxy.svc&CorrelationID=<empty>;&ClientId=NF0C4KC30EGFQUUPAZ7QW&cafeReqId=13bae44a-b1c7-402f-aa2f-763aee410ffa; 443 - 132.245.35.172 - - 401 1 2148074254 156

From this log entry, we can all agree that this is not a firewall problem, since our request reached to the correct destination, but rather an authentication problem on the CAS server, where we got a 401 HTTP response. In my simulation, the issue occurred due to invalid credentials of the local Exchange admin account used for testing the MRS connection,

Below is an example of an IIS log generated by a successful connection:

2016-09-15 08:23:57 10.2.0.5 POST /EWS/mrsproxy.svc &CorrelationID=<empty>;&ClientId=SR0WSDR7B0IXI4TCOLJG&cafeReqId=d26343b6-8790-4ccc-954d-936aed081d88; 443 pronicsi\nicsi 132.245.35.172 - - 200 0 0 343

In this log you can also see the admin account used to establish the remote connection with MRS, along with the HTTP response 200, meaning that our request was accepted by the server.

If you receive a different HTTP response error during this test, you might have to deal with a wrong configuration on IIS side. Like I said before, the MRS proxy endpoint is configured under EWS, which is actually a virtual directory in IIS.

You might find useful below article, for identifying possible causes for various HTTP status codes in IIS:

https://support.microsoft.com/en-us/kb/943891

 

6. Check HttpProxy and HTTPERR logs from CAS servers

If you are running  Exchange Server 2013 or/and Exchange Server 2016 and you saw the MRS requests on the IIS logs from CAS servers, but you are not sure about where the issue might be, you can extract more logs from other places on the CAS server.

HTTP proxy logs for EWS can be found at the following path :

C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Ews

In this log we can see where the CAS server is going to proxy the MRS request, it can be either another CAS server or to a stand-alone Mailbox server. You need to find the entries for /EWS/mrsproxy.svc, like in below example:

HTTPproxylog

 HTTP error logs that are located to below path:

C:\Windows\System32\LogFiles\HTTPERR

This is an example of some MRS related calls from HTTP error logs :

httpERR

 

I hope you found this article useful in troubleshooting issues related to hybrid migration endpoints and I am looking forward to  receiving any feedback from your side.

Comments

  • Anonymous
    September 19, 2016
    Thanksvery interesting
    • Anonymous
      November 23, 2016
      Thanks by your help. In my case was resolved checking the IIS log and using another Exchange Admin Account.
  • Anonymous
    December 16, 2016
    Very informative, but one thing I'm not sure on and that I've not been able to find a good answer for is the best practice guidance for the on-premise administrator account. Is this just required for the creation of the endpoint (in which case just providing my own admin credentials is sufficient) or are these credentials used for every migration (in which case, presumably this should be a dedicated service account with a non-expiring password)?
    • Anonymous
      December 16, 2016
      Hi Joe,When speaking above remote move requests(hybrid migration), we can manage them in 2 ways:1) via migration batches, so in this scenario you need to first create the migration endpoint using your local Exchange admin account.2) via individual move requests started from Exchange Online PowerShell, where you need to use for every move request the credentials of the local admin account(or you can store them in a PowerShell variable).So in case of the migration batches, you don't need to use the local Exchange account credentials for every new batch, as these credentials will be stored on the migration endpoint settings. However, in both scenarios, if you want that, you can use a dedicated account with a non-expiring password.
      • Anonymous
        December 16, 2016
        That's great Nicu, thanks!
  • Anonymous
    January 12, 2017
    Excellent post, great information to have all in one place
  • Anonymous
    March 14, 2017
    Hi all,This is a great article - maybe worth adding our situation (as its probably not unique).IIS Logs showed an 503 server unavailable error when attempting to create an MRS endpoint.The issue turned out to be simple. The email address used as the test user when creating the endpoint had an email address associated that was not an accepted domain for Office 365 tenant. This was not the primary email address but a long forgotten and unused domain. Once this was removed from BOTH the test user and the associated MRS endpoint administrator the MRS was created without any issues. Original error was: Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the server 'mrs.removed.com' could not be completed. ---> Microsoft.Exchange.MailboxReplicationService.RemoteTransientException: The call to 'https://mrs.removed.com/EWS/mrsproxy.svc' failed. Error details: The HTTP service located at https://mrs.removed.com/EWS/mrsproxy.svc is unavailable. This could be because the service is too busy or because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later. --> The remote server returned an error: (503) Server Unavailable..
    • Anonymous
      March 14, 2017
      Hi Andrew,Thanks a lot for sharing with us your scenario. I am pretty sure it can help other people who will run into a similar issue.During the troubleshooting of the issues where hybrid migration endpoint cannot be created we can interact with a variety of root causes, but in most of the cases, the steps from this article can help to isolate the issue.
  • Anonymous
    April 20, 2017
    Very helpful ! Thank for your patience and passion.
  • Anonymous
    April 29, 2017
    The comment has been removed
    • Anonymous
      May 01, 2017
      The comment has been removed
      • Anonymous
        May 23, 2017
        Thanks a lot, work for me... just changed EWS authentication in IIS enabled basic and windows..
  • Anonymous
    June 30, 2017
    Hello thank you, this worked for me. my scenario as follow below.2016 exch in DAG x2 server. there was an dead server object in 2016 CAS. so I found in https, http logs, that request are hitting that server, cleaned dead server object in AD. rebooted both mail server. cross checked all virtual directories, found EWS virtual directory was wrong on secondary CAS server. assigned SSL certificate on secondary server. after all this changes final reboot performed , BINGO ! I was prompted to enter a FQDN name and site name, etc. thanks againSolomon
    • Anonymous
      June 30, 2017
      Hi Solomon,I am very happy that it worked for you as well! It seems that your scenario was a challenging one, but you did a very good job!
  • Anonymous
    August 25, 2017
    Thanks!! In my case it was this: RunspaceId : 5c6063f8-42b6-4651-ad7e-f9dbc9b1a00fResult : FailedMessage : The ExchangeRemote endpoint settings could not be determined from the autodiscover response. No MRSProxy was found running at 'autodiscover.domain.com'.'https://autodiscover.domain.com/EWS/mrsproxy.svc' failed. Error details: The HTTP requestwas forbidden with client authentication scheme 'Negotiate'. --> The remote server returned anerror: (403) Forbidden.. --->This was due to not having basic auth enabled on the EWS vDir. enabled, waited and now we are cooking. no iis reset.
  • Anonymous
    October 26, 2017
    hi, Very nice article thank you. I would like to ask if a trusted certificate is needed in order to create the migration endpoint as i can't find any other reasons why o365 can't connect with my migration mail server. iis logs record nothing when microsoft tries to create the migration endpoint from the portal. I am also wondering if autodiscover should be included in the ssl certificate. This hybrid configuration will end when all mailboxes are uploaded so i suppose the ssl certificate won't be necessary afterwards. I may even get a trial certificate in order to do my migration.thank you in advance
    • Anonymous
      October 26, 2017
      Hello Kostas,In order to configure the hybrid setup(HCW) or for creating the migration endpoint, you will need a valid certificate from a CA provider.Just for migration purpose, Autodiscover is not really mandatory, as you can configure the endpoint with manual settings(as mentioned in this article), but it would be nice to have it included in certificate SANs.
  • Anonymous
    November 01, 2017
    Really helpful!!
  • Anonymous
    November 11, 2017
    Error: MigrationTransientException: The call to ‎'https://mail.test.com/EWS/mrsproxy.svc‎' failed because no service was listening on the specified endpoint. Error details: There was no endpoint listening at https://mail.test.com/EWS/mrsproxy.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. --> Unable to connect to the remote server --> A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond X.X.X.X:443 --> The call to ‎'https://mail.test.com/EWS/mrsproxy.svc‎' failed because no service was listening on the specified endpoint. Error details: There was no endpoint listening at https://mail.test.com/EWS/mrsproxy.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. --> Unable to connect to the remote server --> A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond X.X.X.X:443 --> There was no endpoint listening at https://mail.test.com/EWS/mrsproxy.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. --> Unable to connect to the remote server --> A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond X.X.X.X:443 Report:
    • Anonymous
      November 12, 2017
      Can someone please help on this
      • Anonymous
        November 14, 2017
        Hello,As per the error message, it can be related to a configuration issue with your firewall or port 443 being blocked, as the MRS incoming connections are stopped before reaching to your CAS server.Please follow my recommendation about firewall configuration, along with the troubleshooting steps mentioned here, to isolate the issue faster.
  • Anonymous
    February 18, 2018
    The comment has been removed
    • Anonymous
      February 27, 2018
      The comment has been removed