Share via

SharePoint 2016: The local farm is not accessible after CU installation

  • Article

Problem

I had completed Windows Server 2012 R2 patches and SharePoint 2016 CU installations on all farm servers and was preparing to perform the b2b upgrade (psconfiui) using my farm administrator account. Launching an elevated SharePoint Management Shell (SMS) on the server hosting Central Administration (CA Server), I saw this message appear in the shell just after it started:

The local farm is not accessible.  Cmdlets with FeatureDependency are not registered.

I then executed "Get-SPShellAdmin" but experienced this error message: 

Object reference not set to an instance of an object.

I was familiar with this message appearing when using an account that was not a member of the SharePoint_Shell_Access group on the farm configuration database.  However, I had previously used this same account the previous week, on this same server, to execute various SharePoint commandlets in an elevated SMS; and I was not aware that there had been any permissions re-provisioning of this account since then. This problem was preventing performing b2b upgrade (psconfigui) from within an elevated SMS on the CA server.  I began troubleshooting.

Troubleshooting

  1. Restart SharePoint Management Shell
    • Still experienced initial message.  Restarted again but still experienced message.
  2. In SQL Server Management Studio, check my administrator account database user mappings
    • Granted db_owner and SharePoint_Shell_Access role memberships to farm configuration and content databases.
  3. Launch Central Administration on the CA Server
    • Started successfully
  4. Start elevated SMS on other farm servers
    • These started successfully; message did not display.  
    • Cmdlets completed successfully on all other farm servers except for the CA server.
    • ==>> Issue localized to farm CA server only.
  5. Reboot CA Server and then re-launch SMS
    • Still experienced message
  6. Check server logs on CA server

    • Saw these errors repeating:

      7024 System Critical Event Service Control Manager The SharePoint Timer Service service terminated with the following service-specific error: Unspecified error.

      and

      7031 System Critical Event Service Control Manager The SharePoint Timer Service service terminated unexpectedly.  It has done this X times...

      and

      3 Error Application Error System.ServiceModel 4.0.0.0
      WebHost failed to process a request.
       Sender Information: System.ServiceModel.Activation.HostedHttpRequestAsyncResult/31485076
       Exception: System.ServiceModel.ServiceActivationException: The service '/SecurityTokenServiceApplication/securitytoken.svc' cannot be activated due to an exception during compilation.  

      and

      8306 Application Error SharePoint Foundation
      An exception occurred when trying to issue security token: The requested service, 'http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc/actas' could not be activated. See the server's diagnostic trace logs for more information..

  7. Check Security Token Service Application
    • Started
  8. Check SharePoint Timer Service
    • Stopped.  Started the timer service.  It started.  
  9. Launched elevated SMS: 
    • Still saw original message.  
  10. Check timer service again:
    • Started. 
  11. Rebooted server and then checked timer service: 
    • Started.
  12. Check Application Pools in IIS
    • All started.  Restarted SecurityTokenServiceApplicationPool, and then tried launching elevated SMS again: still saw message. Repeated: same result.
  13. Check web applications in IIS
    • All Started.  Restarted SharePoint Web Services, and then tried launching elevated SMS: still saw message.  Repeated: same result.
  14. Check server logs
    • Continued to see 7031 System Error Service Control Manager.
  15. Check SharePoint Timer Service
    • Running.  Viewed service Properties dialog: Service status was running, but, while watching the dialog, saw the status change to Stopped for less than a second and then back to Running. This suggested that perhaps the timer service was trying to access or launch something and these attempts were failing.
    • Suspected permissions issue caused perhaps by CU installation.  This in turn suggested running PSConfig to fix SharePoint resource permission issues.
    • Was unable to launch PSConfig from SMS == >> launch DOS version
  16. Launch PSConfigUI from shortcut
    • Launched and completed successfully. 
  17. Check CA Server Upgrade Status in farm Central Administration
    • Succeeded
  18. Launch elevated SharePoint Management Shell
    • The initial message no longer appeared.  
    • Get-SPShellAdmin completed successfully. 

Summary

Still unclear what caused the problem.  Initial guess is that CU installation affected timer service permissions or the permissions of SharePoint resources that the timer service needs to interact with, and that this in turn caused the timer service to repeatedly stop and restart.

In the future, if repeated timer restarts are observed after CU installation (but before PSConfig is run), launch PSConfigUI from the shortcut or from an elevated DOS shell.

References

  • tbd

Notes

  • tbd