Partager via

Experience of upgrading Terabytes(TB) of MOSS2007 farm to SP2

  • Article

Few weeks ago I had visited a customer regarding the MOSS2007 SP2 upgrade. Their total size of the data in MOSS was in Terabytes and at the same time it was a business critical platform where multiple teams were depended on the SharePoint for their daily work. After observing the farm, I came to know that there were lots of customizations deployed in the farm and also we had a challenge that customer’s IT department didn’t know completely what all components, custom code, files and configuration changes are deployed in the farm. Considering these challenges in mind, it was absolutely required that we plan the SP2 upgrade very carefully. We did the dry run in staging environment which had a closest environment to production and it went successfully. However, while doing the upgrade in production environment, we did face some runtime problems and we had to resolve them well within the time limit as every single min was precious. Here is how our plan was and I have also documented at which point we faced the issues and how we resolved them. I am putting this complete exercise here so that someone in a similar situation, can benefit from this post.

Servers in farm

  • WFE1
  • WFE2
  • APP Server1(CA Hosted)
  • APP Server2(Indexing)

To begin with, I had referenced the following articles which talk about how you should plan the steps for the upgrade.

https://technet.microsoft.com/en-us/library/cc263467(office.12).aspx 

https://blogs.architectingconnectedsystems.com/blogs/cjg/archive/2009/05/10/How-to-upgrade-to-SharePoint-2007-SP2-_2D00_-Step-by-Step.aspx

Here is how we went

Task 1 – Check disk space on the servers

Task 2 – Backup the databases (truncate and backup)

    1. Create a backup folder (“D:\Backups”) & ensure that you have enough disk space to save all your backups to this location (add the size of each database to determine how much you will need)
    2. Run the following commands TWICE for each database (this will shrink, backup and truncate your database and log files):
      • WSS_Content*
      • WSS_Search*
      • SharePoint_Config
      • SharedServices*

use WSS_Content

dbcc shrinkfile ('WSS_Content') dbcc shrinkfile ('WSS_Content_log') go

backup database WSS_Content to disk = 'D:\backups\wss_content.bak' go backup log WSS_Content to disk = 'D:\backups\wss_content.bak' go

dbcc shrinkfile ('WSS_Content') dbcc shrinkfile ('WSS_Content_log') go

Task 3 – Evaluate Database Size

Make sure you are in the supported boundary and in case you need to shift any site collection in any additional Database please do so.

Task 4 – Prepare to move

Large farm optimization

In very large server farms, installing a software update with the content databases attached can result in too much in too much downtime. If there are multiple sites or many Web servers, then to minimize the downtime required to upgrade, we recommend that you perform the additional steps of running the Stsadm preparetomove operation from the command prompt and then detach the content databases.

For best performance with the upgrade operations, use four or five front-end Web servers per database server. You must configure alternate access mappings on these temporary front-end Web servers to match the original servers. If the alternate access mappings are not identical, the content databases might be upgraded with the wrong URLs within their site content. This results in the inaccurate display of certain pages; you must contact Microsoft Product Services to correct the problem.

To prepare to move the content databases

1. In Windows Server 2003, open a Command Prompt window and navigate to the %COMMONPROGRAMFILES%\Microsoft Shared\Web server extensions\12\Bin directory, which usually contains the Stsadm command-line tool.

2. Before you detach the databases, you must run the Stsadm preparetomove operation to ensure that the content database is included in the membership and profile synchronization after you reattach the content database. At the command prompt, type the following command:

stsadm -o preparetomove -contentDB <database_server:database_name>

Note: If you do not run this operation before you detach the content database, then the membership and profile information in the content database is static and is not synchronized after upgrade.

Task 5 – Detach the content databases

We had created a batch files with detach command for all the content databases. Config databases and SSP databases were remained attached to the farm.

stsadm -o deletecontentdb -url https:// computername –databasenameContentDatabaseName

Task 6 – Backup important files

We had taken an exact synch backup of all the servers in the farm on a separate datacenter so we don’t miss any file on the servers.

Task 7 – Disconnect users from the server farm

Disconnect users from the server farm by stopping the World Wide Web Publishing Service (W3SVC) on all Web servers.

clip_image001Note: This manual step is done as a precaution to ensure that the service is fully stopped.

Stop IIS

o Open a command prompt, run “iisreset /stop”

Task 8 - Download and install the appropriate Windows SharePoint Services 3.0 software update for all servers in your server farm.

We installed in the order, App Server1, App Server2, WFE1 & WFE2

Task 9 - Download and install the appropriate Office SharePoint Server 2007 software update for all servers in your server farm.

We installed in the order, App Server1, App Server2, WFE1 & WFE2

Note: You can run the SharePoint Products and Technologies Configuration Wizard after you install the update on Windows SharePoint Services and again after you install the update on Office SharePoint Server. Or, you can run the wizard once after you install all the packages.

Task 10 – PSConfig

1. At the end of the software update installation, the SharePoint Products and Technologies Configuration Wizard starts.

clip_image001[1]Note: If the wizard does not start automatically, click Start, point to All Programs, point to Administrative Tools, and then click SharePoint Products and Technologies Configuration Wizard.

2. On the SharePoint Products and Technologies Configuration Wizard Welcome page, click Next.

3. In the dialog box that notifies you that some services might have to be restarted during configuration, click Yes.

4. On the Completing the SharePoint Products and Technologies Configuration Wizard page, click Next.

5. When the dialog box about installation in a server farm opens, do not click OK. Instead, leave each server with the following dialog box displayed: You must run Setup to install new binary files for every server in your server farm. If you have multiple servers in your server farm, run Setup and the configuration wizard on the other servers now, and then return to this server and click OK to continue.

6. When the dialog box from the previous step is displayed on all the application servers and Web servers in the server farm, use one Web server that hosts the Central Administration Web site to finalize the installation.

clip_image001[2]Note: Microsoft recommends that you first install the software update on an application server that is hosting the Central Administration Web site, then on the other application servers, and finally on the front-end Web servers. If you are hosting your Central Administration Web site on a front-end Web server, it is recommended that you first install the software update on the front-end Web server that is hosting the Central Administration Web site, then on the application servers, and finally on the remaining front-end Web servers.

7. On the server you selected in the previous step, click OK.

8. In the Configuration Successful dialog box, click Finish.

9. After you finish updating one Web server that hosts the Central Administration Web site, follow the procedures in the Verify update completion and success section in this article to ensure that the software update installation on this one Web server was successful.

10. Continue updating the remaining computers in the server farm, one at a time, by clicking OK in the dialog box.

Now this is the step where we faced some runtime issues, We began with App Server1(CA), and PSConfig failed with an error, “The B2B upgrader timer job failed”. An exception of type Microsoft.SharePoint.PostSetupConfiguration.PostSetupConfigurationTaskException was thrown.

We did the initial internet search for this exception but no any articles and resolutions seem to be matching to our situation and then we figured out by analyzing upgrade logs in detail that one of our SSP DBs was in “Read only” & PSConfig was failing at that DB. We made it “Read-Write” and PSConfig went passed that DB but failed again with the same exception and then same way by looking into upgrade logs we figured out that there was one more Content DB which was in “Read only”. We made that DB also in “Read-Write” & finally third time PSConfig finished successfully. Big Reliefclip_image002

Second, we ran PSConfig in App Server2 and it finished successfully.

Third, We started on WFE1 and again boom, it kept on running for more than half an hour at one step in middle and continuously it was writing a message in upgrade log file, “SyncUpgradeTimerJob: sleeping for 10 seconds”. So we monitored this message for 5-10 min and then decided to kill PSConfig wizard from task manager. For this exception we found a good resolution steps from,

https://geekswithblogs.net/SharePointCherie/archive/2009/12/22/error-in-upgrade-log-syncupgradetimerjob-sleeping-for-10-seconds.aspx

Set the following services to manual startup and shut them off:
World wide web service
Microsoft Single Sign-on Service
Windows Sharepoint Services Search

Restart spadmin and sptimer services

If needed: Command line with C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\BIN>psconfig.exe -cmd upgrade -inplace b2b -wait –force
[We didn’t want to do this step & wanted to start PSConfig from start menu]

But when we tried to Re-Run PSConfig wizard, we got a message that another instance is already running and we suddenly felt that we are in a half-baked situation where we couldn’t run this wizard successfully and now we are not able to start it again. however, later on I realized that this wizard is handled by timer service and we waited for 5-10 min so timer job finishes the default execution cycle of every 5 min and then we could start PSConfig wizard. And at the end, It Finished successfully

Fourth, we started the PSConfig on WFE2 and same issue that we faced in WFE1. Did the similar steps of WFE1 and it finished successfully.

Finally after so much of tense moments, we managed to finish PSCofig successfully in all the machines.

Task 11 – Verify Install

1. Open the upgrade.log file (in 12 hive LOGS directory)

  • Look for “Finished upgrading SPFarm Name=<Name of Configuration Database>”
  • Look for “In-place upgrade session finishes. Root object = SPFarm=<Name of Configuration Database>, recursive = True. 0 errors and 0 warnings encountered.”

2. If the above entries DO NOT exist, look for all instances of logs under the files for each servers

  • “fail”
  • “error”

3. Check version number on:

  • Owssvr.dll (in 12 hive isapi directory) should be “12.0.6421.1000”
  • Registry “HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\12.0”
  • Central administration

4. Check version of the SharePoint databases:

  • Run the following sql command on each database:
  • select * from versions order by timestamp desc
  • You should get “12.0.0.6421”

5. On Central Administration, click “Operations”

  • Click “Servers In Farm”
  • The version for the farm and servers should be “12.0.0.6421”

Task 12 – Attach the content DB

First, We need to attach the content DB which contains the root site collection and since that DB was in GBs, commad took around three hours to attach it to the farm and gave us a “Successful” message. there after we had to attach remianing hunderds of DBs and it went fairly fast for these DB than compared to first root content DB. at early morning we finsihed successful upgrade of the MOSS2007 farm to SP2 from SP1.

one small site was orphaned and that we managed to restore to a different location by using SP-Export.

To attach the content database at the command prompt

  • To attach the database, open a Command Prompt window and type the following at the command prompt:

    stsadm -o addcontentdb -url <https://backupservername:port> -databasename <ContentDBName> -databaseserver <NewPrincipalServer>

Task 13 – Check for SharePoint 2010 readiness

1. Run the following command:

stsadm –o preupgradecheck

2. Review the PreUpgradeCheck-*.htm file in the 12 hive logs directory (it should open in a browser window)

3. You should watch out for the following items:

  • The above command should be run on all Web Front end servers to ensure they are identical
  • You should review the Site Definition information for any non “Internal” site definitions, these will need to have an upgrade definition file. Your developers will need to build this file for SP 2010
  • If you have language packs installed, you will need to install the latest version when SP 2010 comes out
  • Look for any referenced and missing features. Either install them or delete the references to them
  • Depending on the type of upgrade to SP 2010 you do, you many need to plan for URL changes in your sites
  • Review the Lists that have more than the recommends number of items. These could slow the migration process to SP 2010. Consider removing the list or deleting items to shrink the list size
  • Review any Custom Field types that have been added to your Farm. CAML is not used in SP2010 and each of them will need to be re-developed with XSLT in mind.
  • If you are running on 32 bit OS and Server 2003, you will need to start planning for migration to a 64bit server 2008 environment to run SP 2010

Task 14 : MOSS SP2 causes the license to turn in to trial, which is a known bug.

https://support.microsoft.com/kb/971620

SP2 Package Installation steps

Steps to Install Windows SharePoint Services 3.0 SP2

Download Windows SharePoint Services 3.0 SP2 from https://www.microsoft.com/downloads/details.aspx?FamilyId=79BADA82-C13F-44C1-BDC1-D0447337051B&displaylang=en

Step 1: Right click EXE and click Run as Administrator. You only have to do this, if you are running this EXE on Windows Server 2008 Operating System

Make sure you have download correct version of executable i.e. 32 bit for 32-bit OS and 64-bit for 64-bit OS.

clip_image002

Step 2: Accept license agreement and Click Continue

clip_image004

Step 3: Windows SharePoint Services 3.0 Installation will begin.

clip_image006

Step 4: Click NEXT to start the WSS 3.0 SP2 wizard.

clip_image008

Step 5: Click YES to restart the services.

clip_image010

Step 6: Click NEXT to start the configuration wizard.

clip_image012

Step 7: You will be prompted to install WSS 3.0 SP2 on other SharePoint Servers in the Farm. Now logged into other SharePoint Servers and Install WSS 3.0 SP2 and run configuration wizard and leave the configuration wizard at this stage. Repeat the same step for all other SharePoint Servers. Once you are done with all other SharePoint Servers then go back to the server, where you have first started to install WSS 3.0 SP2 and Click "OK".

clip_image014

Step 8: Now configuration wizard will start to update SharePoint databases.

clip_image016

Step 9: Configuration wizard is installed successfully. Now go back to other SharePoint Servers in the farm and complete the WSS 3.0 SP 2 upgrade.

clip_image018

Steps to Install Microsoft Office SharePoint Server 2007 SP2

Download MOSS 2007 SP2 from

https://www.microsoft.com/downloads/details.aspx?FamilyId=B7816D90-5FC6-4347-89B0-A80DEB27A082&displaylang=en

Step 1: Right click and run MOSS 2007 SP2 EXE as Administrator, if you are using Windows Server 2008 OS

clip_image020

Step 2: Accept the license agreement and Click Continue.

clip_image022

Step 3: Installation package will start and check for any previous update i.e. WSS 3.0 SP2

clip_image024

Step 4: Once update detection is completed, MOSS 2007 SP2 installation will start. Installation of MOSS 2007 will take at least 20 minutes

clip_image026

Step 5: SharePoint configuration wizard will automatically start. Click NEXT to Continue.

clip_image028

Step 6: Click Yes to continue.

clip_image030

Step 7: Click Next to complete the wizard

clip_image032

Step 8: At this step, go to other SharePoint Servers in the farm and install MOSS 2007 SP2 and leave the installation at the same stage. Once you are done with other SharePoint servers, then come back to the server where you have started MOSS 2007 SP2 upgrade process and Click OK to continue.

clip_image034

Step 9: SharePoint databases will be updated at this stage.

clip_image036

Step 10: SharePoint Configuration Wizard is successfully completed. Now logged into other SharePoint Servers and complete the upgrade process.

clip_image038

Step 11: Now you can open Central Administration to check the version number of MOSS 2007 is updated. SP2 version number is 12.0.0.6421

clip_image040

Step 12: confirm in the registry location whether all servers have been upgraded HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office\12.0

Step 13: Check under the Databases in SQL for tables versions under content db and config db whether it has been upgraded.

Comments

  • Anonymous
    April 10, 2011
    Nicely written Tejas :)

  • Anonymous
    April 10, 2011
    Great Post tejas

  • Anonymous
    April 12, 2011
    Thanks Manoj, Sreejith. Hope it helps someone in a similar situation!