Odd issue around configuring a dedicated WFE for SharePoint crawl

We had an interesting issue recently where the SharePoint services would not start up and the configuration cache would not get populated. The event logs and ULS were filled with errors about Invalid Uri.

Error details:

Exception information:
    Exception type: UriFormatException
    Exception message: Invalid URI: The format of the URI could not be determined.

And the call stack of the error looks something like this,

[UriFormatException: Invalid URI: The format of the URI could not be determined.]
   System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind) +7204799
   Microsoft.SharePoint.Administration.SPAutoSerializingObject.DeserializeBasicObject(XmlElement xmlValue, Boolean isMerge, Boolean bResolveMissingTypes) +1848
   Microsoft.SharePoint.Administration.SPAutoSerializingObject.DeserializeBasicObject(XmlElement xmlValue, Boolean isMerge, Boolean bResolveMissingTypes) +3124
   Microsoft.SharePoint.Administration.SPAutoSerializingObject.DeserializeBasicObject(XmlElement xmlValue, Boolean isMerge, Boolean bResolveMissingTypes) +2926
   Microsoft.SharePoint.Administration.SPAutoSerializingObject.SetStateXml(XmlNodeList childNodes, Boolean isMerge, Boolean bResolveMissingTypes) +685
   Microsoft.SharePoint.Administration.SPAutoSerializingObject.SetStateXml(XmlDocument value, Boolean bResolveMissingTypes) +85
   Microsoft.SharePoint.Administration.SPPersistedObject.Initialize(ISPPersistedStoreProvider persistedStoreProvider, Guid id, Guid parentId, String name, SPObjectStatus status, Int64 version, XmlDocument state) +27172224
   Microsoft.SharePoint.Administration.SPConfigurationDatabase.GetObject(Guid id, Guid parentId, Guid type, String name, SPObjectStatus status, Byte[] versionBuffer, String xml) +347
   Microsoft.SharePoint.Administration.SPConfigurationDatabase.FetchObject(Guid id) +1396
   Microsoft.SharePoint.Administration.SPConfigurationDatabase.GetObject(Guid id, Boolean checkInMemoryCache, Boolean checkFileSystemCache) +406
   Microsoft.SharePoint.Administration.SPConfigurationDatabase.Microsoft.SharePoint.Administration.ISPPersistedStoreProvider.GetObject(Guid id) +59
   Microsoft.SharePoint.Administration.SPPersistedObjectCollection`1.get_Item(Guid objId) +50
   Microsoft.SharePoint.Administration.<GetEnumeratorImpl>d__0.MoveNext() +239
   Microsoft.SharePoint.Administration.Enumerator`1.MoveNext() +23
   Microsoft.SharePoint.Administration.SPWebApplication.AcquireAlternateUrlCache(SPFarm farm) +402
   Microsoft.SharePoint.Administration.SPWebApplication.Lookup(SPFarm farm, Uri requestUri, Boolean fallbackToHttpContext, SPAlternateUrl& alternateUrl, SPSiteLookupInfo& hostHeaderSiteInfo, Boolean& lookupRequiredContext) +237
   Microsoft.SharePoint.ApplicationRuntime.SPRequestModule.EnsureInitialize(HttpRequest request) +646
   Microsoft.SharePoint.ApplicationRuntime.SPRequestModule.BeginRequestHandler(Object oSender, EventArgs ea) +164
   System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +80
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +171

We initially isolated it to a corruption in the URL within the XML for the Web Application properties. But when we looked through the XML, we did not find anything corrupt in the URLs. This was strange!

Finally capturing a dump of the process and investigating further, I found that the property that was used in the above callstack was m_SiteDataServers.

The value of this property was set to just the servername without the protocol information. i.e. it was set to "wfe1" instead of "https://wfe1" OR "https://wfe2".

NOTE: The script to configure the dedicated frontend for crawling uses the command "$webApp.SiteDataServers.Add($zoneUrl, $listOfUri);". This command will not accept just the servername, but seems like we have an entry point somewhere to send it through.