SharePoint 2013/2016 RSS Viewer not working with external sites when through a proxy

 

Recently I worked on a case for a Fortune 500 company that was having problems to render the RSS Viewer Web Part in their SharePoint 2013 farm. They still have a SharePoint 2010 farm in the same network environment and this farm was not having problems rendering the RSS Viewer. The constraint was this: access to outside sites requires going through a proxy. Default proxy configuration is a tricky thing. It can be done in several ways: DHCP settings, DNS settings, registry settings, Internet Explorer (for the user logged in only) or for .NET applications via configuration. In production environment is always a good idea to inform the default proxy explicitly since auto-discovery may take a few seconds the first time the process try it for an endpoint. So, initially we configured their proxy explicitly by adding this in the web.config file of the SharePoint application. So, for the local domain ‘contoso.com’, proxy address ‘**https://proxy.contoso.com:8080**’ this is the configuration:

   <system.net>
     <defaultProxy>
         <proxy
             usesystemdefault="false"
             autoDetect="false"
             proxyaddress="https://proxy.contoso.com:8080"
             bypassonlocal="true"
         />
       <bypasslist>
         <add address="[a-zA-Z0-9\.]+\.contoso\.com$" />
       </bypasslist>
      </defaultProxy>
   </system.net>

This configuration is better placed one line below <configuration> or one line above </configuration>. Another important thing concerning bypass list: one should never use * at the beginning of an exclusion item. System.Net uses RegEx to check whether the address is in exclusion or not and will fail if wildcard is used (specially if it starts with '*' which is a no-no in .NET RegEx. The regex equivalent to * is ‘ [a-zA-Z0-9\.\-]+ ’.

They added the suggested configuration and analyzing the dump file we could confirm the proxy configuration was applied. Still we were seeing the error below in the ULS logs:

DATE TIME    w3wp.exe (0x15B4)    0x4878    SharePoint Portal Server    Web Parts    8imh    High    RssWebPart: Exception handed to HandleRuntimeException.HandleException System.Net.Sockets.SocketException (0x80004005): No such host is known     at System.Net.Dns.GetAddrInfo(String name)     at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)     at System.Net.Dns.GetHostAddresses(String hostNameOrAddress)     at Microsoft.SharePoint.Utilities.CommonUtility.IsIntranetAddress(String hostName)     at Microsoft.SharePoint.Utilities.SPWebRequest.PreventIntranetCalls(Uri requestUri, SPContext context, String sharePointMarkValue, String appContextLoggingValue)     at Microsoft.SharePoint.Utilities.SPWebRequest.SafeCreate(Uri requestUri, SPContext context, Type callingType, String callingInstanceTag)     at Microsoft.SharePoint.WebControls.XmlUrlDataSource.FetchData(String requestUrl)     at Microsoft.SharePoint.WebControls.BaseXmlDataSource.Execute(String request)     at Microsoft.SharePoint.WebControls.BaseXmlDataSource.GetXmlDocument()     at Microsoft.SharePoint.WebControls.SingleDataSource.GetXPathNavigatorInternal()     at Microsoft.SharePoint.WebControls.SingleDataSource.GetXPathNavigator()     at Microsoft.SharePoint.WebPartPages.DataFormWebPart.GetXPathNavigator(String viewPath)     at Microsoft.SharePoint.WebPartPages.DataFormWebPart.PrepareAndPerformTransform(Boolean bDeferExecuteTransform)

 

We captured a dump file and could confirm that .NET was indeed applying the proxy configuration. We did some more troubleshooting and we tested in a lab farm. Things worked like a charm for us. Also the SharePoint 2010 farm worked as expected. We noticed that the last public method called before failure was this one:

System.Net.Dns.GetHostAddresses(String hostNameOrAddress)

And this was odd, since proxy resolution will first try to request the page via the proxy server and then if, and only if, it fails it will try DNS resolution. Checking the source code (you may achieve the same result with reflection) we were able to find out the culprit: if SPFarm.Properties.DisableIntranetCalls or SPFarm.Properties.DisableIntranetCallsFromApps (if it is an App application) is set to true, SharePoint will use DNS to verify whether the address is intranet or internet. As the address cannot be resolved because the rss feed site is not in the DNS and can only be resolved via proxy which would be the next step after this, SharePoint throws the exception you see in the ULS log entry above.

So, how to resolve this issue? The solution is very counter-intuitive: set DisableIntranetCalls and DisableIntranetCallsFromApps to false and it will work. Then you may ask yourself, if the call is extranet why enabling internet calls will resolve the issue? The answer is the fact that if these properties are set to false the call to resolve the address via DNS will not be called so there will be no exception and the retrieval of the page will occur without a problem.

Is this change introducing security risk to SharePoint? Not at all. This property is rather to force people to use better web parts to list a feed of a local site.

This PowerShell script will disable the undesired DNS search if you are facing this issue:

 

 # These commands will prevent SharePoint from trying to resolve the RSS name host in DNS which may cause problem with proxies 
 # 
  Add-PSSnapin Microsoft.SharePoint.PowerShell
 
  $farm = Get-SPFarm
  $farm.Properties.DisableIntranetCalls = $false
  $farm.Properties.DisableIntranetCallsFromApps = $false
  $farm.Update()

 

How to verify if you are facing this problem in your environment. Run this in PowerShell (change the URL for the RSS feed to test):

 $url = New-Object System.Uri "https://blogs.msdn.microsoft.com/rodneyviana/feed/"
 [System.Net.Dns]::GetHostAddresses($url.DnsSafeHost)

 

If it throws an exception, then the RSS web part will not render unless a proxy is set and the DisableIntranetCalls* properties are set to false.

Special thanks to Tony DeVere from Distributed Services team for his invaluable help in this case.

Additional information on Proxy discovery

 

The .Net Framework proxy detecting/loading, is using the following setting combination
https://msdn.microsoft.com/en-us/library/sa91de1e.aspx
<proxy
autoDetect="true|false|unspecified"
bypassonlocal="true|false|unspecified"
proxyaddress="uriString"
scriptLocation="uriString"
usesystemdefault="true|false|unspecified "
/>

 

By default, autoDetect is set as unspecified, which equals to enabled. It means the proxy is automatically detected. The proxy auto detection follows this white paper for design.
https://msdn.microsoft.com/en-us/magazine/cc300743.aspx
“Automatic configuration is a two-step process. First, the user agent (Internet Explorer or the .NET Framework) tries to discover the URL of a configuration script. Typically, the configuration script is a JavaScript file that contains logic to determine whether to use a proxy and what proxy to use for a given target resource. The protocol for automatic detection is documented at Web Proxy Auto-Discovery Protocol. The user agent first tries a DHCPINFORM message. The reply to this message contains the URL of the configuration script. If that doesn't work, a DNS lookup for a well-known alias, WPAD, is issued. Then the configuration script's location is determined as https://<machine>/wpad.dat where the machine name is the result of the DNS lookup.

The second step is to download the configuration script from the URL and run the script. The specification of what the configuration script looks like is at Navigator Proxy Auto-Config File Format. And then parse and run the proxy auto configuration (PAC) script.”
This detection implementation is expensive, take time and also require you have either DHCP server or DNS set up correctly. If you are using a middle tier server type of client, for performance consideration, we will need to disable it to save the time spent on proxy detection routine.

The 2nd proxy detection is using IE proxy. It is controlled through setting, usesystemdefault="true|false|unspecified "
usesystemdefault -- >  Specifies whether to use Internet Explorer proxy settings. If set to true, subsequent attributes will override Internet Explorer proxy settings. The default value is unspecified, which equals to true.
But this setting will not work well for a service. A service typically doesn’t have any IE profile with proxy setting. Creating a IE profile for a service account is like a hacking. Usesystemdefault setting is more designed for a log on user type of application instead of service. IE proxy connection is implemented through wininet.dll, but wininet.dll is not supported in service.
https://support.microsoft.com/kb/238425
INFO: WinInet Not Supported for Use in Services
So, we will discourage using the IE proxy setting and creating an IE profile for a service account to load proxy in .Net as it will make the proxy loading behavior hard to be predicted if a service has some user impersonation implemented.

For the middle tier or service type of client app, we can set the proxy use the following choice:
bypassonlocal="true|false|unspecified"
proxyaddress="uriString"
scriptLocation="uriString"
We can using proxy address to well specify the proxy information combined with proxy bypass rule. If you have to use the proxy auto configuration (PAC) script rule, you can use the choice in scriptLocation. This way, we can well control the proxy loading behavior to avoid delay on Proxy detection or unpredictable result for using impersonated account to load IE proxy setting.