Share via


SharePoint 2010: a spreadsheet method for quickly resolving missing non-upgrade-blocking dependencies

Introduction

Resolving the various missing dependencies that surface in upgrade reports, like those generated by Test-SPContentDatabase, can be tedious, especially when there these are scattered about the site collection in no particular order and there are many hundreds of them to resolve.  You could create a script that loops through all libraries and lists in in all webs in the site collection, but this would require a number of logic expressions to test each list and library and verify that it is the right one. Using spreadsheet methods makes this much less tedious and much more efficient and accurate and doesn't require expertise in PowerShell development.  Using a spreadsheet, you can quickly create as many simple PowerShell statements as are needed to resolve, say, a missing assembly or web part reference; and then executing them all is as simple as copy and paste.  This posting walks you through a few examples of that process.

Each of these examples employ a 2013 development farm.  This farm needn't be anything complicated.  It can't be a stand-alone farm, but it can be just a single-server farm, where development instances of SQL Server 2012 and SharePoint Server 2013 are co-hosted on the same server.  This development farm will not be used to serve production content but to support SQL queries and API interaction for development and testing purposes only.  The copy of the content database mounted to this farm is not the copy destined for production.  This development and testing farm is only used to develop the PowerShell scripts that will then be run against the copy of the content database destined for production and can be terminated after completing the upgrade.  Since only the API is used for production content database interaction, this ensures that the copy of the content database destined for production remains supportable.

Each example begins by showing you a typical missing dependency error that can appear in the report generated by Test-SPContentDatabase.  It then shows how the information in that error message can be used to interrogate the content database in order to obtain additional information on the dependency needed to resolve it.   Then the SQL script is shown that it will be executed to obtain the necessary information, and then the results of this SQL script are shown.  The example then presents the analytical arguments used to develop appropriate resolutions for each result, and the spreadsheet method for building the resolution.

Example 1: MissingFeature

In this example, we will walk through the process of resolving a number of missing features that appear to be of unknown origin, meaning that they appear in the report without any name.  This example was taken from an actual upgrade experience.  These missing feature dependencies can still be resolved even though they are missing a Name attribute. Listed below are the five features.  The Remedy column was left out.

Category Error UpgradeBlocking Message
MissingFeature TRUE FALSE Database [WSS_Content] has reference(s) to a missing feature: Id = [50e31660-5eaa-4691-9f44-04552b127ad5].
MissingFeature TRUE FALSE Database [WSS_Content] has reference(s) to a missing feature: Id = [3a465ae2-6114-418b-be8d-081557b79517].
MissingFeature TRUE FALSE Database [WSS_Content] has reference(s) to a missing feature: Id = [8c42c1c3-f444-449c-a1fc-307eb4170e70].
MissingFeature TRUE FALSE Database [WSS_Content] has reference(s) to a missing feature: Id = [2a05cbcb-6b59-4d52-b601-a923fa2f0d21].
MissingFeature TRUE FALSE Database [WSS_Content] has reference(s) to a missing feature: Id = [89ab5aa0-28d1-4700-9db7-bb8a9c672131].

The critical item of information for each of these report entries is the "Id", which is the feature ID.  This information enables us to query the content database's Features table to find out which site collections and webs the feature reference is associated with.  Here is the SQL script that needs to be executed:

USE WSS_Content
SELECT SiteID, WebID, FeatureID
FROM Features WITH (NoLOCK)
WHERE FeatureID IN 
('50e31660-5eaa-4691-9f44-04552b127ad5',
'3a465ae2-6114-418b-be8d-081557b79517',
'8c42c1c3-f444-449c-a1fc-307eb4170e70',
'2a05cbcb-6b59-4d52-b601-a923fa2f0d21',
'89ab5aa0-28d1-4700-9db7-bb8a9c672131')

Executing this script returns this listing:

SiteID WebID FeatureID
D39E1F1E-B2F2-4210-98F5-734272A884B6 00000000-0000-0000-0000-000000000000 50E31660-5EAA-4691-9F44-04552B127AD5
D39E1F1E-B2F2-4210-98F5-734272A884B6 00000000-0000-0000-0000-000000000000 3A465AE2-6114-418B-BE8D-081557B79517
D39E1F1E-B2F2-4210-98F5-734272A884B6 00000000-0000-0000-0000-000000000000 8C42C1C3-F444-449C-A1FC-307EB4170E70
D39E1F1E-B2F2-4210-98F5-734272A884B6 00000000-0000-0000-0000-000000000000 2A05CBCB-6B59-4D52-B601-A923FA2F0D21
D39E1F1E-B2F2-4210-98F5-734272A884B6 00000000-0000-0000-0000-000000000000 89AB5AA0-28D1-4700-9DB7-BB8A9C672131
DA918BC1-F7B5-4F55-8E5D-800F4A51699A 00000000-0000-0000-0000-000000000000 50E31660-5EAA-4691-9F44-04552B127AD5
DA918BC1-F7B5-4F55-8E5D-800F4A51699A 00000000-0000-0000-0000-000000000000 3A465AE2-6114-418B-BE8D-081557B79517
DA918BC1-F7B5-4F55-8E5D-800F4A51699A 00000000-0000-0000-0000-000000000000 8C42C1C3-F444-449C-A1FC-307EB4170E70
DA918BC1-F7B5-4F55-8E5D-800F4A51699A 00000000-0000-0000-0000-000000000000 2A05CBCB-6B59-4D52-B601-A923FA2F0D21
DA918BC1-F7B5-4F55-8E5D-800F4A51699A 00000000-0000-0000-0000-000000000000 89AB5AA0-28D1-4700-9DB7-BB8A9C672131
B5008741-96ED-4B7F-B690-946B1228EF49 00000000-0000-0000-0000-000000000000 50E31660-5EAA-4691-9F44-04552B127AD5
B5008741-96ED-4B7F-B690-946B1228EF49 00000000-0000-0000-0000-000000000000 3A465AE2-6114-418B-BE8D-081557B79517
B5008741-96ED-4B7F-B690-946B1228EF49 00000000-0000-0000-0000-000000000000 8C42C1C3-F444-449C-A1FC-307EB4170E70
B5008741-96ED-4B7F-B690-946B1228EF49 00000000-0000-0000-0000-000000000000 2A05CBCB-6B59-4D52-B601-A923FA2F0D21
B5008741-96ED-4B7F-B690-946B1228EF49 00000000-0000-0000-0000-000000000000 89AB5AA0-28D1-4700-9DB7-BB8A9C672131
41161F28-172E-4A77-8CA9-A9D97B392BEB 00000000-0000-0000-0000-000000000000 50E31660-5EAA-4691-9F44-04552B127AD5
41161F28-172E-4A77-8CA9-A9D97B392BEB 00000000-0000-0000-0000-000000000000 3A465AE2-6114-418B-BE8D-081557B79517
41161F28-172E-4A77-8CA9-A9D97B392BEB 00000000-0000-0000-0000-000000000000 8C42C1C3-F444-449C-A1FC-307EB4170E70
41161F28-172E-4A77-8CA9-A9D97B392BEB 00000000-0000-0000-0000-000000000000 2A05CBCB-6B59-4D52-B601-A923FA2F0D21
41161F28-172E-4A77-8CA9-A9D97B392BEB 00000000-0000-0000-0000-000000000000 89AB5AA0-28D1-4700-9DB7-BB8A9C672131
5A2055D4-37C9-4C67-93B3-DBE9C6D89BB3 00000000-0000-0000-0000-000000000000 50E31660-5EAA-4691-9F44-04552B127AD5
5A2055D4-37C9-4C67-93B3-DBE9C6D89BB3 00000000-0000-0000-0000-000000000000 3A465AE2-6114-418B-BE8D-081557B79517
5A2055D4-37C9-4C67-93B3-DBE9C6D89BB3 00000000-0000-0000-0000-000000000000 8C42C1C3-F444-449C-A1FC-307EB4170E70
5A2055D4-37C9-4C67-93B3-DBE9C6D89BB3 00000000-0000-0000-0000-000000000000 2A05CBCB-6B59-4D52-B601-A923FA2F0D21
5A2055D4-37C9-4C67-93B3-DBE9C6D89BB3 00000000-0000-0000-0000-000000000000 89AB5AA0-28D1-4700-9DB7-BB8A9C672131
1E3A4808-79C7-4818-8703-F987CF144CA8 00000000-0000-0000-0000-000000000000 50E31660-5EAA-4691-9F44-04552B127AD5
1E3A4808-79C7-4818-8703-F987CF144CA8 00000000-0000-0000-0000-000000000000 3A465AE2-6114-418B-BE8D-081557B79517
1E3A4808-79C7-4818-8703-F987CF144CA8 00000000-0000-0000-0000-000000000000 8C42C1C3-F444-449C-A1FC-307EB4170E70
1E3A4808-79C7-4818-8703-F987CF144CA8 00000000-0000-0000-0000-000000000000 2A05CBCB-6B59-4D52-B601-A923FA2F0D21
1E3A4808-79C7-4818-8703-F987CF144CA8 00000000-0000-0000-0000-000000000000 89AB5AA0-28D1-4700-9DB7-BB8A9C672131

30 results were returned, which corresponds to the same set of five missing feature dependencies appearing for each of the six site collections of this website. A WebID of all "0" indicates that the feature has site collection scope and not web scope.  This is a critical distinction because it determines which object Features collection (spSite.Features or spWeb.Features)  contains the feature.  These well-formatted and columnar results lends themselves to spreadsheet analysis.  Leave the SQL query window open as it will be useful in a few moments to execute the query again as a quick verification measure.

Now, copy the results from the SQL query window and paste directly into a spreadsheet.  Add headers to make things easier to track and identify.  Then, in the fourth column over, next to the first row of data (row 2) enter something similar to the following:

=CONCAT("$Site=Get-SPSite -Identity """,A2,""";$Site.Features.Remove(""",C2,""", $True)")

Once you have verified that the cell references are correct, click on the cell handle and drag to the bottom of the data.  Instantly, a set of 30, accurate PowerShell statements are generated.  Now, just select this range of cells, copy them (CTRL+C), and then paste them into an elevated SharePoint PowerShell window.  They will all be instantly executed.Now re-execute the SQL query.  No results were returned.  Next, re-run Test-SPContentDatabase, and then review the report.  Of the five MissingFeature problems, four no longer appeared but one remained:

Category Error UpgradeBlocking Message
MissingFeature TRUE FALSE Database [WSS_Content] has reference(s) to a missing feature: Id = [2a05cbcb-6b59-4d52-b601-a923fa2f0d21].

A search on the ID returns the Amrein Engineering Discussion Rollup Web Part.  This missing feature dependency persists, even though it has been removed from the Features table, likely (from previous experience) because it is associated with a list or library,  This can be checked by querying the AllLists table, using this SQL script:

USE WSS_Content
SELECT tp_SiteID, tp_FeatureID, tp_WebID, tp_Title, tp_BaseType, AllWebs.FullUrl
FROM AllLists WITH (NoLOCK)
    LEFT JOIN AllWebs ON AllWebs.ID = AllLists.tp_WebID
WHERE tp_FeatureID IN ('2a05cbcb-6b59-4d52-b601-a923fa2f0d21')

Executing this SQL script returns the following:

tp_SiteID tp_FeatureID tp_WebID tp_Title tp_BaseType AllWebs.FullUrl
41161F28-172E-4A77-8CA9-A9D97B392BEB 2A05CBCB-6B59-4D52-B601-A923FA2F0D21 4C39BCD0-6AE1-423D-99A1-78EB4FE3B5FF Tags 0
41161F28-172E-4A77-8CA9-A9D97B392BEB 2A05CBCB-6B59-4D52-B601-A923FA2F0D21 4C39BCD0-6AE1-423D-99A1-78EB4FE3B5FF TITLE1 FORUM 0
41161F28-172E-4A77-8CA9-A9D97B392BEB 2A05CBCB-6B59-4D52-B601-A923FA2F0D21 4C39BCD0-6AE1-423D-99A1-78EB4FE3B5FF Title2 Forum 0
41161F28-172E-4A77-8CA9-A9D97B392BEB 2A05CBCB-6B59-4D52-B601-A923FA2F0D21 4C39BCD0-6AE1-423D-99A1-78EB4FE3B5FF Title3 Forum 0
41161F28-172E-4A77-8CA9-A9D97B392BEB 2A05CBCB-6B59-4D52-B601-A923FA2F0D21 4C39BCD0-6AE1-423D-99A1-78EB4FE3B5FF Title4  Forum 0
41161F28-172E-4A77-8CA9-A9D97B392BEB 2A05CBCB-6B59-4D52-B601-A923FA2F0D21 4C39BCD0-6AE1-423D-99A1-78EB4FE3B5FF Title5 Forum 0
41161F28-172E-4A77-8CA9-A9D97B392BEB 2A05CBCB-6B59-4D52-B601-A923FA2F0D21 4C39BCD0-6AE1-423D-99A1-78EB4FE3B5FF Title6 Forum 0
41161F28-172E-4A77-8CA9-A9D97B392BEB 2A05CBCB-6B59-4D52-B601-A923FA2F0D21 4C39BCD0-6AE1-423D-99A1-78EB4FE3B5FF Title7 Forum 0
41161F28-172E-4A77-8CA9-A9D97B392BEB 2A05CBCB-6B59-4D52-B601-A923FA2F0D21 4C39BCD0-6AE1-423D-99A1-78EB4FE3B5FF Title8 Forum 0
41161F28-172E-4A77-8CA9-A9D97B392BEB 2A05CBCB-6B59-4D52-B601-A923FA2F0D21 B03FBC6B-2ED6-42B5-A7C4-DC512A3A868F Title9  Board 0 web1
41161F28-172E-4A77-8CA9-A9D97B392BEB 2A05CBCB-6B59-4D52-B601-A923FA2F0D21 B03FBC6B-2ED6-42B5-A7C4-DC512A3A868F Tags 0 web1
41161F28-172E-4A77-8CA9-A9D97B392BEB 2A05CBCB-6B59-4D52-B601-A923FA2F0D21 82A304BA-1013-4F4F-BD61-E8709DCE8A55 Discussion Board Plus 0 web2

A basetype of "1" indicates a document library, and "0" indicates a standard list. A NULL value in the FullUrl column indicates that the resource is located in the root web. Navigating to each of these lists, I found that they had not been used for a long time and contained little or no content and thus could be deleted.  

To remove these via spreadsheet, copy the SQL results into a spreadsheet.  Then, in the next column over, enter the following into the second row cell:

=CONCAT("(Get-SPSite -Identity """,A2,""").openweb(""/",F2,""").lists[""",D2,"""].Delete()")

Then drag the cell down to accurately and efficiently create the 12 delete statements needed.  Select and copy (CTRL+C) these, and then past them into an elevated SharePoint Management Shell.  They'll execute immediately.  Afterwards, re-executing Test-SPContentDatabase to generate the upgrade report no longer returned the last remaining missing feature problem, indicating that the missing feature problems lacking any name were resolved.

Example 2: MissingWebpart

First, let's review the report generated by Test-SPContentDatabase and look at the missing web part error surfaced in this report:

Category Error UpgradeBlocking Message
MissingWebPart TRUE FALSE WebPart class [c7843aae-4c86-8206-0125-d00117cb461c] (class [Bamboo.WebParts.AlertPlus] from assembly [Bamboo.AlertPlus, Version=1.0.0.0, Culture=neutral, PublicKeyToken=2cc91efae2d531be]) is referenced [8] times in the database [WSS_Content], but is not installed on the current farm. Please install any feature/solution which contains this web part.

The key piece of information from this error entry is the ID of the web part.  The name and number of instances will also be useful for corroboration purposes.  This ID is the ID associated with the type of web part.  It's not the ID of the actual instance of the web part that appears on a web page.  Extracting just the relevant information, we have:

Name ID Instances
Bamboo.WebParts.AlertPlus c7843aae-4c86-8206-0125-d00117cb461c  8

Next, interrogate the content database's AllWebParts and AllDocs tables for all references to this particular type of web part.  The AllWebParts table lists the type of web parts and where they are used (i.e., what pages they are used in) and the AllDocs table lists the actual instances of these webparts, i.e., their instance IDs.  Here is the SQL script:

USE WSS_Content
SELECT SiteID, WebID, ListID, DirName, LeafName, tp_WebPartTypeId, tp_WebPartIDProperty, tp_DisplayName
FROM AllDocs WITH (NOLOCK) 
INNER JOIN AllWebParts ON AllDocs.Id = AllWebParts.tp_PageUrlID
WHERE AllWebParts.tp_WebPartTypeID IN
    ('c7843aae-4c86-8206-0125-d00117cb461c')

Executing this SQL script in a new query window obtains these results:

#  WebID SiteID ListID DirName LeafeName tp_WebPartIDProperty
1 20086E89-4341-4716-9E53-21B9D70F535C 41161F28-172E-4A77-8CA9-A9D97B392BEB A0ACDBC9-A631-4423-9BBB-06A9FD34382C web1/web11/TestPages AlertTest.aspx NULL
2 20086E89-4341-4716-9E53-21B9D70F535C 41161F28-172E-4A77-8CA9-A9D97B392BEB FE38AA5C-B10A-4846-BA50-0B2C0617E636 web1/web11/SitePages AlertPage.aspx g_ad2d018b_426f_4c74_a25c_d32262d4cab6
3 20086E89-4341-4716-9E53-21B9D70F535C 41161F28-172E-4A77-8CA9-A9D97B392BEB FE38AA5C-B10A-4846-BA50-0B2C0617E636 web1/web11/SitePages AlertPage.aspx g_ad2d018b_426f_4c74_a25c_d32262d4cab6
4 20086E89-4341-4716-9E53-21B9D70F535C 41161F28-172E-4A77-8CA9-A9D97B392BEB FE38AA5C-B10A-4846-BA50-0B2C0617E636 web1/web11/SitePages AlertPage.aspx g_ad2d018b_426f_4c74_a25c_d32262d4cab6
5 4C39BCD0-6AE1-423D-99A1-78EB4FE3B5FF 41161F28-172E-4A77-8CA9-A9D97B392BEB 7449E9AD-6CBE-4F00-8B97-D455F3877577 SitePages test1.aspx NULL
6 57C5C4D4-A421-4B36-8FA0-2F1B491D2E27 41161F28-172E-4A77-8CA9-A9D97B392BEB FE265830-11B2-416B-8D68-E426C52A54FF web2/web21/SitePages Home.aspx g_356c9a0c_2222_4da3_8524_fb372385f68f
7 57C5C4D4-A421-4B36-8FA0-2F1B491D2E27 41161F28-172E-4A77-8CA9-A9D97B392BEB FE265830-11B2-416B-8D68-E426C52A54FF web2/web21/SitePages Home.aspx g_356c9a0c_2222_4da3_8524_fb372385f68f
8 FB68983E-3B9B-446C-ABB8-4EFC8D308A19 41161F28-172E-4A77-8CA9-A9D97B392BEB 2A0C72C0-99E9-411C-8F4A-C83A3E8A5508 web3/web31/Pages Workflows.aspx NULL

The number of results returned, "8", matches what was reported in the upgrade report.  All of the results involved web pages.  One of the results appears to show that several instances of the web part were used on the same page.  While this is certainly possible, it is also possible that these several versions have to do with the web part appearing not on the same page but on multiple versions of the page.

There weren't many instances returned, so let's navigate to each of the pages involved and discover what's going on.  Here are the results of the analysis:

  • web1/web11/TestPages/AlertTest.aspx: this web page did not have the webpart on the current page version.  In fact, this web page only had one version and that was the one that was displayed, so it was unclear why SharePoint found a reference to the web part on this page,.  The NULL webpartidproperty value seems to suggest something, but it's not clear that this needs to be researched in depth since this was a test page only.  Given the low value of the page, I opted to delete the page, rather than researching it further.
  • Web1/web11/SitePages/AlertPage.aspx: here is an instance where the web part is associated with multiple versions of the page. Here again, since this was a testing page only, it wasn't clear that developing the script for extracting each version and removing the web part from each version was worth the effort.  Instead, given the low value of the page, I opted to simply delete the page.
  • SitesPages/TestWPP.aspx: the web part could not be found on this page.  The NULL webpartidproperty value seems to offer a clue, but again, its a low priority page, and so I opted to delete the page.
  • web2/web21/SitePages/Home.aspx: this page also had multiple web part references, but could not find it on an earlier version of the page.  Reviewing the page, I found not original content and it only featured list web parts pointing to content elsewhere.  Thus I opted to renove this page as removing it would not affect existing content and the page could be easily rebuilt.
  • web3/web31/Pages/workflow.aspx: this was an odd one.  I could not find the page in the Pages document library, which in fact contained nothing at all.  Nor could I find the page in 1st or 2nd stage recycling.  Now, there is a "Workflow.aspx" file usually located at /15/TEMPLATES/LAYOUTS, or "/_layouts/15/Workflow.aspx".  So, my thought was that perhaps the site owner or a designer had mistakenly copied the file into the Pages document library and had worked on it for a bit and then deleted.  Or something else was going on.  Since there was not actual Workflow.aspx file in the Pages document library, I couldn't delete it using normal methods.  Nor could I delete and recreate the Pages library, which is a special library created by activating the web's Publishing feature.  I discovered that deleting this library causes the entire web to become inaccessible.  Thankfully, this was only a development copy I was working with.  In the end, I opted to remove the web (which thankfully was still available through PowerShell).

Creating a looping script that systematically looks at each document library in each web of the site collection would be too much for the needs of this upgrade.  There are only a few missing web part instances, and these instances were scattered in various document libraries and sites.  Since the syntax for the script that needed to be run is the same in some instances, the spreadsheet method could be used to quickly and efficiently and accurate develop all of the needed statements.

To do this, I first copied all the results returned from executing the SQL query into a spreadsheet.  Then, using the CONCAT function, I built the PowerShell script for deleting a page like so:

=CONCAT("$web=Get-SPWeb -Identity """,A2,""" -Site """,B2,""" ;$list=$web.lists | where {$_.id -eq """,C2,""" };$list.Title;$item=$list.items | Where {$_.File.Name -eq """, E2,""" };$item.Name;$item.Delete()")

Note that this statement effectively "walks" the SharePoint object model down to the item.  See my note below for further discussion on this.  Anyway, I then selected this cell and then dragged it down the spreadsheet to create individual statements for each page that needed to be deleted.  Some error entries did not need to be resolved since they referred to multiple web part instances on the same page and the page's previous versions.  This was the case for items 2, 3 and 4 above.  I therefore executed the statement once to remove all three error message.  Repeat for items 6 and 7. Not shown above is how to build the statement for removing websites, which usesRemove-SPWeb; but its the same process.

Second, after all statements have been generated, I just selected them, copied them all (CTRL+C), and then pasted them (CTRL+V) into an elevated SharePoint Management Shell.

After checking to ensure that they all successfully completed, I re-execute Test-SPContentDatabase to verify that the this missing web part dependency had been fully resolved.

Summary

This posting has presented several examples that employed a spreadsheet-based method for quickly, efficiently and accurately drafting the PowerShell statements needed for resolving missing dependency problems surfaced in upgrade reports.  The spreadsheet method leverages the nicely formatted and columnar listings of missing dependency information returned from executing SQL scripts against the content database.  Using the spreadsheet method, you can draft hundreds of PowerShell statements in seconds (e.g., see reference 3), and then execute them all in less than a minute.  This method does not require expertise in PowerShell.

References

  1. Test-SPContentDatabase
  2. Remove-SPWeb
  3. SharePoint 2010: Case Study on How to Remove Project-related Missing Dependencies when Upgrading to 2013

Notes

  • I've also employed spreadsheet methods for other missing dependencies.  For example, when resolving Microsoft Project-related missing dependencies, I needed to resolve nearly three hundred missing assembly references (see reference 3).  Building the first commandlet took a few minutes, but then building the other 293 took less than 10 seconds, and executing them all took less than a minute.

  • I executed the SQL script against a copy of the content database mounted to a development farm.  This copy is not the copy that was actually upgraded, but is the copy used for data extraction purposes.  This copy is only used to build the PowerShell statements that will later be executed on the production copy.

  • For example 1, instead of JOINing AllWebs on AllLists you could JOIN AllDocs on AllLists to get more detailed information.  Here's the SQL script:

           USE WSS_Content
            SELECT AllLists.tp_FeatureID, AllLists.tp_WebID, AllLists.tp_ID, 
                               AllLists.tp_Title, AllLists.tp_BaseType, AllDocs.ID,
                              AllDocs.DirName,AllDocs.LeafName
            FROM AllLists WITH (NoLOCK)
                LEFT JOIN AllDocs ON AllDocs.ID=AllLists.tp_RootFolder
            WHERE tp_FeatureID IN ('2a05cbcb-6b59-4d52-b601-a923fa2f0d21')
    
  • tbd