Share via

Cloud search service applications fails to crawl and throws 403 forbidden due to outbound network proxy

  • Article

Recently we received couple of questions around failures while crawling content (Onpremise 2016, 2013, 2010 farms) from Cloud Search Service Application when there is an Outbound proxy server in the network. All content sources experience errors regarding Azure plugin and fails to successfully submit content.This post is focused around common error and remediation for Cloud Search Service Application (Cloud Hybrid Search, CSSA) proxy settings.

Below are some of the extracts from ULS logs.

Exception details

mssearch.exe (0x29CC) 0x1084 SharePoint Server Search Crawler:Azure Plugin ayg2o High AzureServiceProxy caught Exception: *** System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it XX.XX.XX.XXX:443 at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult) at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)

--- End of inner exception stack trace ---

at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context) at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar) --- End of inner exception stack trace --- --- End of inner exception stack trace --- at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) at Microsoft.Office.Server.Search.AzureSearchService.AzureServiceProxy.SubmitDocumentsRequest(HttpClient client, HttpRequestMessage batchRequest, Int32 numDocs) at Microsoft.Office.Server.Search.AzureSearchService.AzureServiceProxy.SubmitDocuments(String azureServiceLocation, String authRealm, String SPOServiceTenantID, String SearchContentService_ContentFarmId, String portalURL, String testId, String farmLabel, String encryptionCert, Boolean allowUnencryptedSubmit, sSubmitDocument[] documents, sDocumentResult[]& results, sAzureRequestInfo& RequestInfo) ---> (Inner Exception #0) System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it XX.XX.XX.XXX:443 at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult) at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception) --- End of inner exception stack trace --- at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context) at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar) --- End of inner exception stack trace ---<--- ***

Or you may see the below exception.

AzureServiceProxy::Getcerts caught AggregateException: Unable to connect to the remote server. AzureServiceProxy::GetCerts: Failed to get encryption certificates from AzureServiceProxy:: GetAzureTenantInfo caught Aggregate Exception: Unable to connect to the remote server, unable to get Service Properties, submit is blocked. AzureServiceProxy caught Exception: *** Microsoft.Office.Server.Search.AzureSearchService.AzureException: AzurePlugin was not able to get Tenant Info from configuration server. AzureServiceProxy::SubmitDocuments: No encryption certificates found and unencrypted submit is not allowed. AzureServiceProxy::GetCerts caught AggregateException: Unable to connect to the remote server AzureServiceProxy::GetCerts: Failed to get encryption certificates from cert server * for realm *, documents will be send unencrypted (if unecrypted submit is allowed)AzureServiceProxy::GetAzureTenantInfo caught AggregateException: Unable to connect to the remote server, unable to get ServiceProperties, submit is blocked AzureServiceProxy caught Exception: *** Microsoft.Office.Server.Search.AzureSearchService.AzureException: AzurePlugin was not able to get Tenant Info from configuration server at Microsoft.Office.Server.Search.AzureSearchService.AzureServiceProxy.GetAzureTenantInfo(String portalURL, String realm, String& returnPropertyValue, String propertyName)at Microsoft.Office.Server.Search.AzureSearchService.AzureServiceProxy.SubmitDocuments(String azureServiceLocation, String authRealm, String SPOServiceTenantID, String SearchContentService_ContentFarmId, String portalURL, String testId, String encryptionCert, Boolean allowUnencryptedSubmit, sSubmitDocument[] documents, sDocumentResult[]& results, sAzureRequestInfo& RequestInfo) **

This typically happens if Cloud Search Service Application farm is on a network that has Outbound proxy server for internet access or no internet access available from the SharePoint crawl servers. To confirm, its recommended to grab a Network trace preferably fiddler trace in context of the account running msssearch15 while crawl is running.

To fix the issue, perform the following actions

1.Ensure that servers hosting Sharepoint Cloud Search Service Application farm has access listed in these articles.(Blog, TechNet). A good test is also to access the urls mentioned in the blog with account running the search service in the farm.

  1. You also need to ensure that account running the search service (msssearch, noderunner accounts) (for crawl and Query federation scenarios to work) in the Cloud Search Service Application farm have unrestricted outbound internet access. If not, set system level proxy settings for these accounts with netsh. Below are the steps.
  2. Make a note of the accounts (account running the search service) either from windows services or taskmanager.
  3. Log in to a server in the farm (preferably crawl server) in the Cloud Search Service Application farm. Run the below commands

Open Internet explorer and validate proxy server has been set correctly and validate exclusions list if any.

Open a command prompt and type in netsh winhttp show proxy. (Make a note of the output to ensure what has been set currently and if you need to revert.

netsh winhttp import proxy source=ie

  1. If you cannot log in to the server with the account running search service, you can launch an admin command prompt and manually update the proxy settings. If the command prompt fails to launch with the search service account and correct credentials ensure that the account has log in locally rights on the server.  Below are the steps.
  2. Launch a command prompt with the account running the search service account.
  3. At the command prompt, type netsh winhttp set proxy proxyservername:portnumber.

 

Watch out this space for more such hybrid troubleshooting guidelines coming shortly.!!