A Deeper Look: Patching SharePoint 2013 Farms
So you need to patch a SharePoint farm and you’re unsure about the best way of doing so & the risks this might involve. This post broadly covers the best way of updating SharePoint farms, looks at typical problems you could see when it happens, and tries to answer some common questions about the process.
SharePoint farms do need updating occasionally even if it’s just to remain on a supported build (assuming you ever want our help with it), so it’s worth knowing a bit about how updates work.
It’s a complicated process but I’ve tried to touch on all the most important things to keep in mind at least.
So how do I Update SharePoint then?
It’s pretty simple really; for each server in the SharePoint farm you have to install the binaries 1st, which can be done in parallel. Once each server is binary patched, you need to run the SharePoint Configuration Wizard (PSConfigGUI is the process name, or PSConfig – the command-line equivalent) to finish the job on each server, one-by-one. PSConfig will also upgrade the databases as part of the final upgrade process too.
This SharePoint update process is broken down in more detail in a previous post here. The three important points about the update process for SharePoint are:
- Patching is broadly speaking a two-part process: update binaries on all servers, then update database schemas.
- What’s not mentioned is that per web-front-end, each IIS site binaries need updating too which also happens post-binary install - “install application content” it’s called.
- Without updating all databases, SharePoint is running in “compatibility mode” which isn’t supported.
- We don’t fully test SharePoint while it’s running compatibility mode for databases. Things might break, and the only way to guarantee that they don’t is to let SharePoint finish the upgrade process for all databases in farm.
- It is not possible to patch a single SharePoint 2013 farm (or 2010/2007) without at some point having all servers in the farm offline.
- Yes, there may be hacks to get around this but they are just hacks – SharePoint in a half-baked state is not something I’d want to run my production sites on.
- If you do need 100% uptime for SharePoint during the patch process, there are ways of achieving this.
So, assuming you have just one SharePoint farm without the option of sending users to another one temporarily, you need to find a big maintenance window to allow your only farm to upgrade (see “SharePoint 2013 Update Installations are Slow!” below).
SharePoint Update Blockers
One of the few things that’ll cause a normal SharePoint admin to reconsider their job prospects & general life choices is a failed SharePoint upgrade, in production anyway. It’s a scary thing indeed and will leave your farm is the half-baked state of doom where nothing’s guaranteed to work properly until the upgrade blockers are fixed and the upgrade then starts + finishes successfully this time, on every server.
I could (but probably won’t) write a book on this subject, but for now I’ll start with some guidance on where to look for the failure. You need to be clear what failed:
- Update binary installation failure.
- Version incompatibility error.
- Installation error or failure (“product: Something SharePoint could not be installed. Error code 1603. Additional information is available in the log file something.LOG”).
- PSConfig/SharePoint Configuration Wizard critical error.
- Application content install (rare).
- SharePoint services provisioning failure.
- Farm/databases upgrade error.
- Service applications.
- Content database.
My aim here is to make it clear where the upgrade could fail and to give some initial pointers on where to start troubleshooting if it does happen.
Binary Installation Failure
This is where the binaries haven’t made it onto the server for one reason or another. This could happen for one of two reasons; either the installer rejects the patch because you’ve not got what it expects already installed or the upgrade fails midway through. In either case, I’ll let my fellow blogger navigate you through these troubles - https://blogs.technet.com/b/arudell/archive/2012/12/05/sharepoint-installer-troubleshooting.aspx
One small note from me; I’ve resolved unexpectedly failing installations by running “Get-SPProduct –Local” before as it basically forces the local machine to refresh the packages installed.
PSConfig/Configuration Wizard Upgrade Failures
PSConfig (and the GUI equivalent, “SharePoint 2013 Products Configuration Wizard”) does a lot of important things during an upgrade operation, so there’s plenty of places it might fail. You need to figure out where the upgrade is failing if it does from the upgrade log-file; any errors (and successes) are logged into “PSCDiagnostics_date_time.log” files in your SharePoint logging directory – these files are key to figuring out what went wrong. Once you’ve fixed the original problem, you then need to run it again until it finishes.
More on PSConfig here - https://technet.microsoft.com/en-us/library/Cc263093(v=Office.14).aspx
Install Application Content
I’ll not cover application content installation errors here in detail because they are rare indeed, but that said, if you’re getting MissingMethodException with SharePoint it could be because this stage was either not run or it failed.
Try and fix application content mismatches with “psconfig.exe -cmd applicationcontent -install”.
Services Provisioning Error
Generally about the only thing that fails here is the SPAdmin service not starting. I blogged about this previously but the gist is, the revocation check on the binary certificates can fail thus causing the service to not start in time. Do your SharePoint servers have line-of-sight to the internet? No? This is probably why. That’s not the only reason; group policies and all sorts of other reason can hold-up this process.
Try and fix with “psconfig.exe -cmd services -install".
Databases Upgrade Failures (Most Common)
So finally, the new binaries will need to upgrade the databases attached to the farm. Depending on what was patched this may be lots of changes or not, but certainly all databases will be looked at.
Not just that though, the definition of each objects in the farm may need upgrading too, and this can be where things go wrong. If you have run SharePoint in a supported way you almost certainly won’t ever see anything go wrong here but there are some things that can cause the database upgrades to fail. If this is happening, this article should help - https://technet.microsoft.com/en-us/library/Cc262967.aspx
Here’s an example; in the PSCDiagnostics log-file, searching from the bottom upwards (to find the most recent error) we eventually come across the rather vague error “Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException was thrown”:
This is the “I’m done upgrading the server, lets upgrade the databases” task, and it’s breaking for some reason or other. Let’s find out why; looking in the logs folder we’ll see an upgrade “error” log:
It turns out this particular upgrade failed because the content database in question is in read-only mode, as it’s a replica in an AlwaysOn cluster on the instance SharePoint is looking at (another server is the primary, which does have read/write access). Solution: disconnect content DB and re-run the wizard/PSConfig.
SharePoint Updates FAQ
The most common questions I hear about SharePoint upgrades are below. Let me know if I’ve missed anything.
SharePoint 2013 Update Installations are Slow! Why?
Agreed. This is for a bunch of reasons; some avoidable, and some not. SharePoint isn’t just one product for one thing; it’s actually several products lumped under the umbrella name of SharePoint Server so a patch may update 10-20 products that would otherwise have to be done one-by-one. Yes, the binary installation alone can take 5+ hours to complete per server (although you can update binaries in parallel) depending on how many parts of SharePoint need patching.
There are ways of speeding up the installation to some extent though, at least the binary update part as is well documented here and it definitely does reduce the install time. That said, we’ve seen a very small number of cases where disabling the services this way have caused the “install application content” phase to fail, causing all sorts of errors due to the mismatch of binaries in IIS virtual directories & what’s installed globally.
What Can I Do If The Upgrade Fails?
That depends on where the upgrade fails to some extent. Above I’ve tried to give some guidance on what to do depending on where it dies.
Why is the Upgrade Process So Complicated?
SharePoint is a complicated system, in short. It’s a web-platform, a database engine of sorts, an apps framework, a service-engine (search/user-profiles/etc), and lots more. There’s just lots of moving parts to it, and that’s before we talk about customisations. Complicated things are not easy to update, even assuming they’re used as recommended.
How to Make SharePoint Farm Patching Easier?
Good question. I actually blogged about this too not long ago. My best advice is to have another SharePoint farm so SharePoint can be patched without any downtime; that way there’s no immediate pressure to get everything done ASAP as there’s no service outage. There are other options though; take a look at my previous post on it.
Wrap-up
That’s it for now on SharePoint patching. If any areas are of particular interest, I’ll consider writing more on the subject.
Cheers,
Sam Betts
Comments
- Anonymous
April 05, 2017
Hello Sam,I ran PsConfig on server multiple times withoput issues but we had a 3rd code deployment on farm and after which we tried to run this command "PSCONFIG -cmd upgrade -inpace b2b -force -cmdd applicationcontent -install -cmd installfeatures" and this command took more time than expected however when checking log file i saw continous entries of "SyncUpgradeTimerJob: sleeping for 10 seconds" i waited for 2hrs and then i closed powershell window and terminated process and then restarted with this command "PSCONFIG.EXE -cmd upgrade -inplace b2b -wait -force"I would really like to know why these "SyncUpgradeTimerJob: sleeping for 10 seconds" entries are logged on logs and what should be my action.Awaiting for your response !