Share via


SharePoint 2010 Troubleshooting: Resolve Missing PDF Forms Dependencies when Upgrading to SharePoint 2013

Introduction

Missing PDF Share Forms dependencies may be found when upgrading a production SharePoint 2010 site collection to SharePoint 2013, where the the PDF Share pdfforms.services.wsp and pdfforms.designer.wsp farm solutions are installed to the 2010 farm but not to the  2013 farm. The site collection in this study is hosted in a dedicated content database that in turn is mounted to a dedicated web application. The need for resolving these PDF Share Forms missing dependencies is due to the fact that the 2010 version of PDF Forms solutions cannot be directly upgraded to 2013 on account of these solutions being substantially revised, including changes to solution names and GUIDs. Three types of missing dependencies were found:

  • MissingFeatures
  • MissingSetupFiles
  • MissingAssemblies

Methods for developing resolutions for each of these will be presented here. This case study presumes that:

  • All SQL queries are engaged against a full backup copy of the production content database that has been restored to a development farm;
  • The full backup used here to developing the resolutions will not be used for the actual production upgrade but is used for testing purposes only;
  • All SQL queries are performed on the test instance of the content database and no SQL queries are ever performed against that copy of the content database that will be used for the actual upgrade

This posting begins at the point where: 1) a copy of the full backup has been restored to the development SharePoint 2013 farm but not yet mounted to that farm; and 2) Test-SPContentDatabase has been executed against this test instance of the content database and the resultant report saved to a CSV file.

Results of Test-SPContentDatabase

Summarized below are the missing PDFShare-related dependencies found in the report generated by Test-SPContentDatabase.  

Category Error UpgradeBlocking Message
 01 MissingFeature TRUE FALSE Database [WSS_Content_ORG] has reference(s) to a missing feature: Name = [PDF Share Forms Designer Enterprise Feature], Id = [4fb212cc-8ff3-4c28-af79-09ee1d8fbace], Description = [], Install Location = [PDFForms.Designer_PDFFormsDesignerFeature]
 02 MissingFeature TRUE FALSE Database [WSS_Content_ORG] has reference(s) to a missing feature: Name = [PDF Share Forms Enterprise Core Feature], Id = [a1962e07-22a4-4dde-a2e4-c1986150e35c], Description = [], Install Location = [PDFForms.Services_PDFFormsCoreFeature].
 03 MissingSetupFile TRUE FALSE File [Features\ PDFForms.Designer_PDFFormsDesignerFeature\ PdfTemplatesListDefinition\repair.aspx] is referenced [15] times in the database [WSS_Content_ORG], but is not installed on the current farm. Please install any feature/solution which contains this file.
 04 MissingSetupFile TRUE FALSE File [Features\ PDFForms.Designer_PDFFormsDesignerFeature\ PdfTemplatesListDefinition\Upload.aspx] is referenced [15] times in the database [WSS_Content_ORG], but is not installed on the current farm. Please install any feature/solution which contains this file.
 05 MissingSetupFile TRUE FALSE File [Features\ PDFForms.Services_PDFFormsCoreFeature\ _PDFShareFormsInternal\ErrorTemplate.html] is referenced [1] times in the database [WSS_Content_ORG], but is not installed on the current farm. Please install any feature/solution which contains this file.
 06 MissingSetupFile TRUE FALSE File [Features\ PDFForms.Services_PDFFormsCoreFeature\ _PDFShareFormsInternal\ErrorTemplate.pdf] is referenced [1] times in the database [WSS_Content_ORG], but is not installed on the current farm. Please install any feature/solution which contains this file.
 07 MissingSetupFile TRUE FALSE File [Features\  PDFForms.Services_PDFFormsCoreFeature\ _PDFShareFormsInternal\SuccessTemplate.html] is referenced [1] times in the database [WSS_Content_ORG], but is not installed on the current farm. Please install any feature/solution which contains this file.
 08 MissingSetupFile TRUE FALSE File [Features\ PDFForms.Services_PDFFormsCoreFeature\ _PDFShareFormsInternal\SuccessTemplate.pdf] is referenced [1] times in the database [WSS_Content_ORG], but is not installed on the current farm. Please install any feature/solution which contains this file.
 09 MissingSetupFile TRUE FALSE File [Features\ PDFForms.Services_PDFFormsCoreFeature\ FormsList\FormsList.webpart] is referenced [1] times in the database [WSS_Content_ORG], but is not installed on the current farm. Please install any feature/solution which contains this file.
 10 MissingSetupFile TRUE FALSE File [Features\ PDFForms.Services_PDFFormsCoreFeature\ RenderForm\RenderForm.webpart] is referenced [1] times in the database [WSS_Content_ORG], but is not installed on the current farm. Please install any feature/solution which contains this file.
 11 MissingSetupFile TRUE FALSE File [Features\ PDFForms.Services_PDFFormsCoreFeature\ SubmissionHistory\SubmissionHistory.webpart] is referenced [1] times in the database [WSS_Content_ORG], but is not installed on the current farm. Please install any feature/solution which contains this file.
12 MissingSetupFile TRUE FALSE File [Layouts\ PDFForms.Designer\ TemplatesLibraryAllItems.aspx] is referenced [15] times in the database [WSS_Content_ORG], but is not installed on the current farm. Please install any feature/solution which contains this file.
 13 MissingAssembly TRUE FALSE Assembly [PDFForms.Designer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=ec1de91b62c9319e] is referenced in the database [WSS_Content_ORG], but is not installed on the current farm. Please install any feature/solution which contains this assembly.
14 MissingAssembly TRUE FALSE Assembly [PDFForms.Services, Version=1.0.0.0, Culture=neutral, PublicKeyToken=ec1de91b62c9319e] is referenced in the database [WSS_Content_ORG], but is not installed on the current farm. Please install any feature/solution which contains this assembly.

Each of these missing PDFShare-related dependencies will be resolved. NOTE: spaces were introduced into the setup file paths shown in the listing above to enable the table to render completely in the space available.

Resolving Missing Features

  1. Review the report generated by Test-SPContentDatabase.

  2. Document relevant information for each of the missing features listed in this report. For example:

     #  Feature Name Feature ID
     01 PDF Share Forms Designer Enterprise Feature a1962e07-22a4-4dde-a2e4-c1986150e35c
     02 PDF Share Forms Enterprise Core Feature 4fb212cc-8ff3-4c28-af79-09ee1d8fbace
  3. On the 2013 farm's SQL Server, in SQL Server Management Studio, open a new query window and execute the following SQL script:

    USE WSS_Content_ORG

    SELECT SiteID,WebID,FeatureID

    FROM Features WITH (NoLOCK)

    WHERE FeatureID IN ('4fb212cc-8ff3-4c28-af79-09ee1d8fbace', 'a1962e07-22a4-4dde-a2e4-c1986150e35c')

    This will return something like the following:

    SiteID WebID FeatureID
    C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF 00000000-0000-0000-0000-000000000000 A1962E07-22A4-4DDE-A2E4-C1986150E35C
    C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF CA9840EF-DDE8-4E65-A915-16E8BE1D4C39 4FB212CC-8FF3-4C28-AF79-09EE1D8FBACE
    C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF C4127B51-B19A-4DDC-8591-295DAA3646CD 4FB212CC-8FF3-4C28-AF79-09EE1D8FBACE
    C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF 16A5028A-B295-4E52-9556-2F6FBF12ECE7 4FB212CC-8FF3-4C28-AF79-09EE1D8FBACE
    C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF 19571030-4327-444F-9931-444FED7F1E1B 4FB212CC-8FF3-4C28-AF79-09EE1D8FBACE
    C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF EE7B9533-FEC3-4A64-855C-4D432750C94C 4FB212CC-8FF3-4C28-AF79-09EE1D8FBACE
    C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF 3341EC72-FF39-4922-A940-5621AB59746C 4FB212CC-8FF3-4C28-AF79-09EE1D8FBACE
    C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF 66C864FC-A29D-441A-9566-A4C08FA3F0E6 4FB212CC-8FF3-4C28-AF79-09EE1D8FBACE
    C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF 080FCB70-C7D4-4254-8B48-AF6FD6A7BFB1 4FB212CC-8FF3-4C28-AF79-09EE1D8FBACE
    C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF 2E0C6F2F-7F36-4C65-9403-B2E35A4ABDF3 4FB212CC-8FF3-4C28-AF79-09EE1D8FBACE
    C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF 3704D664-DE6B-463C-96F2-BBA21EAF439E 4FB212CC-8FF3-4C28-AF79-09EE1D8FBACE
    C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF 906F4132-2D2E-416B-983A-D4BA8B508F0D 4FB212CC-8FF3-4C28-AF79-09EE1D8FBACE
    C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF FF3C3E82-F991-4B57-AE6B-DC844065D525 4FB212CC-8FF3-4C28-AF79-09EE1D8FBACE
  4. Mount the target content database to the 2013 development farm.

  5. On the 2013 development farm server, in an elevated SharePoint Management Shell, execute the following SharePoint PowerShell script:

    $SiteID = "C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF"

    $Site=Get-SPSite -Identity "C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF"

    $Site.Features.Remove("A1962E07-22A4-4DDE-A2E4-C1986150E35C",$True)

    $FeatureID = "4FB212CC-8FF3-4C28-AF79-09EE1D8FBACE"

    (Get-SPWeb -Identity "CCA9840EF-DDE8-4E65-A915-16E8BE1D4C39" -Site $siteID).Features.Remove($FeatureID,$True)

    (Get-SPWeb -Identity "C4127B51-B19A-4DDC-8591-295DAA3646CD" -Site $siteID).Features.Remove($FeatureID,$True)

    (Get-SPWeb -Identity "16A5028A-B295-4E52-9556-2F6FBF12ECE7" -Site $siteID).Features.Remove($FeatureID,$True)

    (Get-SPWeb -Identity "19571030-4327-444F-9931-444FED7F1E1B" -Site $siteID).Features.Remove($FeatureID,$True)

    (Get-SPWeb -Identity "EE7B9533-FEC3-4A64-855C-4D432750C94C" -Site $siteID).Features.Remove($FeatureID,$True)

    (Get-SPWeb -Identity "3341EC72-FF39-4922-A940-5621AB59746C" -Site $siteID).Features.Remove($FeatureID,$True)

    (Get-SPWeb -Identity "66C864FC-A29D-441A-9566-A4C08FA3F0E6" -Site $siteID).Features.Remove($FeatureID,$True)

    (Get-SPWeb -Identity "080FCB70-C7D4-4254-8B48-AF6FD6A7BFB1" -Site $siteID).Features.Remove($FeatureID,$True)

    (Get-SPWeb -Identity "2E0C6F2F-7F36-4C65-9403-B2E35A4ABDF3" -Site $siteID).Features.Remove($FeatureID,$True)

    (Get-SPWeb -Identity "3704D664-DE6B-463C-96F2-BBA21EAF439E" -Site $siteID).Features.Remove($FeatureID,$True)

    (Get-SPWeb -Identity "906F4132-2D2E-416B-983A-D4BA8B508F0D" -Site $siteID).Features.Remove($FeatureID,$True)

    (Get-SPWeb -Identity "FF3C3E82-F991-4B57-AE6B-DC844065D525" -Site $siteID).Features.Remove($FeatureID,$True)

  6. If the SSMS query window is still open, re-execute the query to verify that nothing is returned. If nothing is returned from this SQL query, this is a pretty good indication that these orphaned features have been resolved. For the definitive verification, perform another Test-SPContentDatabase to verify that the two missing features no longer appear in the report.

Resolving Missing Setup Files

  1. Review the report generated by Test-SPContentDatabase.

  2. Document each of the missing setup files listed in this report. For example:

     # SetupPath
     03 Features\PDFForms.Designer_PDFFormsDesignerFeature\PdfTemplatesListDefinition\repair.aspx
     04 Features\PDFForms.Designer_PDFFormsDesignerFeature\PdfTemplatesListDefinition\Upload.aspx
     05 Features\PDFForms.Services_PDFFormsCoreFeature\_PDFShareFormsInternal\ErrorTemplate.html
     06 Features\PDFForms.Services_PDFFormsCoreFeature\_PDFShareFormsInternal\ErrorTemplate.pdf
     07 Features\PDFForms.Services_PDFFormsCoreFeature\_PDFShareFormsInternal\SuccessTemplate.html
     08 Features\PDFForms.Services_PDFFormsCoreFeature\_PDFShareFormsInternal\SuccessTemplate.pdf
     09 Features\PDFForms.Services_PDFFormsCoreFeature\FormsList\FormsList.webpart
     10 Features\PDFForms.Services_PDFFormsCoreFeature\RenderForm\RenderForm.webpart
     11 Features\PDFForms.Services_PDFFormsCoreFeature\SubmissionHistory\SubmissionHistory.webpart
  3. We'll begin with item 05; items 03 and 04 will be engaged later.

  4. On the 2013 farm's SQL Server, in SQL Server Management Studio, open a new query window and execute the following SQL script:

    USE WSS_Content_ORG

    SELECT id AS SetupFileID, SiteID, WebID, DirName, LeafName, ListId

    FROM AllDocs WITH (NOLOCK)

    WHERE SetupPath = 'Features\PDFForms.Services_PDFFormsCoreFeature\PDFShareFormsInternal\ErrorTemplate.html'

    This will return something like this

    SetupFileID SiteID WebID DirName LeafName ListId
    1DB0B49A-B49A-429D-9E0B-EB278E85FF5D C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF 66C864FC-A29D-441A-9566-A4C08FA3F0E6 _PDFShareFormsInternal ErrorTemplate.html NULL

    Only the first three items are needed, but the DirName and Leafname provide helpful context.

  5. Mount the target content database to the 2013 development farm.

  6. In an elevated SharePoint Management Shell, execute the following script:

    $setupFileID = "1DB0B49A-B49A-429D-9E0B-EB278E85FF5D"

    $SiteID = "C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF"

    $WebID = "66C864FC-A29D-441A-9566-A4C08FA3F0E6"

    $site = Get-SPSite -Identity $siteID

    $Web = Get-SPWeb -Identity $WebID -Site $SiteID

    $file = $web.GetFile([GUID]$setupFileID)

    $filelocation = "{0}{1}" -f ($site.WebApplication.Url).TrimEnd("/"), $file.ServerRelativeUrl

    Write-Host "Found file location:" $filelocation

    $file.Delete()

    $web.dispose()

  7. Back in the Query window, re-execute the script to verify that the setup file entry in the AllDocs table has been removed. This will be a pretty good indication that this particular missing setup file has been resolved.

  8. Repeat steps 3 - 5 above for the next setup file and continue until items 05 through 11 have been resolved. 

  9. Now remove items 03 and 04.  In the same shell, execute the following commandlets:

    $site =Get-SPSite -Identity "C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF"

    #execute this method for the root web:

    $site.openweb("/").Lists["PDF Forms Templates"].Delete()

    #and then repeat this one for each of the other 14 webs:

    $site.openweb("/web01").Lists["PDF Forms Templates"].Delete()

    Items 03 and 04 occur 15 times each. Through review of STSADM's EnumAllWebs report, these 15 occurrences were identified with the PDF Forms Templates document library that is created when activating the web scope PDF Share Forms Designer Enterprise Feature.  De-activating this feature does not remove these document libraries.  Due to the re-architecture of the PDF Share Forms solutions, the name of the document library created by this feature has changed in the 2016 version of the solution and thus the old libraries should be removed.  Removing the document library also removes the reference to the setup file.  These document libraries cannot be removed pre-upgrade since they are still being used in production.  Therefore, they are removed during the upgrade process.  Resolving the missing setup files associated with the PDF Forms Templates document library (items 03 - 11) also results in resolving the missing assembly (item 01).

  10. To verify that all missing setup file problems have been successfully resolved, dismount the content database and execute Test-SPContentDatabase again.

Resolving Missing Assemblies

  1. Review the report generated by Test-SPContentDatabase.

  2. Document each of the missing assemblies listed in this report (focusing on PDFShare-related for this posting). For example:

     #  Assembly Name
     02 PDFForms.Services, Version=1.0.0.0, Culture=neutral, PublicKeyToken=ec1de91b62c9319e

    Remember that the first missing assembly, item 01, was already resolved, though indirectly, by removing all of the PDF Forms Templates document libraries, as was done previously.

  3. On the 2013 farm's SQL Server, in SQL Server Management Studio, open a new query window and execute the following SQL script: TEST

    USE WSS_Content_ORG

    Select Id, SiteID, WebID, HostType, hostId

    FROM EventReceivers WITH (NOLOCK)

    WHERE Assembly = 'PDFForms.Services, Version=1.0.0.0, Culture=neutral, PublicKeyToken=ec1de91b62c9319e'

    This may return a lengthy table of entries. Just the first few will be listed here:

    id SiteID WebID HostType HostID
    4DDADC9C-9A70-4484-8993-1B2F7B76D20E C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF CA9840EF-DDE8-4E65-A915-16E8BE1D4C39 2 6EB4A915-117F-4B22-A6A3-4BEE88C13944
    F78D3647-5B37-4467-8A52-ADACCE2E7BF7 C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF CA9840EF-DDE8-4E65-A915-16E8BE1D4C39 2 6EB4A915-117F-4B22-A6A3-4BEE88C13944
    11E24DFD-A053-42C5-A52E-3503BF3E0BA7 C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF CA9840EF-DDE8-4E65-A915-16E8BE1D4C39 2 6EB4A915-117F-4B22-A6A3-4BEE88C13944
    742E9539-8FC2-4408-B20B-F1337AA635B1 C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF CA9840EF-DDE8-4E65-A915-16E8BE1D4C39 2 6EB4A915-117F-4B22-A6A3-4BEE88C13944
     etc  ...  ...  ...  ...

    The HostType of "2" indicates that all of these event receivers are attached to lists, where a list in SharePoint terms can be a standard list or a document library. A full enumeration of Host Types is provided in one of the references.

  4. In an elevated SharePoint Management Shell, execute the following script to remove all of the event receiver instances attached to these lists:

    $SiteID = "C1A41F57-9B36-4F02-A8DB-1CB6AE2555BF"

    $WebID = "CA9840EF-DDE8-4E65-A915-16E8BE1D4C39"

    $hostID = "6EB4A915-117F-4B22-A6A3-4BEE88C13944"

    $ER1 = "4DDADC9C-9A70-4484-8993-1B2F7B76D20E"

    $ER2 = "F78D3647-5B37-4467-8A52-ADACCE2E7BF7"

    $ER3 = "11E24DFD-A053-42C5-A52E-3503BF3E0BA7"

    $ER4 = "742E9539-8FC2-4408-B20B-F1337AA635B1"

    $web = Get-SPWeb -Identity $webID -Site $siteID

    $list = $web.lists | ?{$_.id -eq $hostID}

    ($list.EventReceivers | ?{$_.id -eq $ER1}).delete()

    ($list.EventReceivers | ?{$_.id -eq $ER2}).delete()

    ($list.EventReceivers | ?{$_.id -eq $ER3}).delete()

    ($list.EventReceivers | ?{$_.id -eq $ER4}).delete()

    $web.dispose()

  5. To verify that the event receiver instances were removed, execute this script in the same shell:

    $list.eventreceivers.ID

    and then review the list to verify that none of the IDs appear that were removed

  6. Repeat step 4 for each group of HostIDs that are listed.

  7. When you are done, re-execute the SQL script in step 3: no results should be returned. This is a pretty good indication that the missing assembly problem has been fully resolved. For definitive verification, re-execute Test-SPContentDatabase.

Summary

The process presented in this case study was used for an actual upgrade of a production content database and completely and successfully resolved all of the missing dependencies associated with the PDF Share Forms.  This same approach is used to resolve missing dependencies associated with other farm solutions.

References

Notes

  • Example syntax of using Test-SPContentDatabase and saving its output to a separate file:

    Test-SPContentDatabase -name "WSS_Content_ORG" -webapplication "http://APPSVR:1234/" | Export-CSV -Path "D:\Temp\TEST.CSV"

  • A WebID of "00000000-0000-0000-0000-000000000000" returned from queries of the Features table indicates that the specific feature having this WebID is not activated at the web-scope but at the site collection-scope.

  • An instance of a feature object does not have a delete or remove method that enables the object to delete itself out of existence. To remove an orphaned feature entry, it has to be removed from the Features collection of the web or site collection, and this is accomplished using the Features collection's Remove method. To discover all of the methods and properties of a Features collection, use the Get-Member method.

  • The SQL script used for resolving missing assemblies only returns the absolute minimum information needed to resolve them. If you want additional information regarding the missing assembly, such as the name of the website, the relative URL to the site, the name of the list, etc, you'll need to query multiple tables for that information, and not just one. Here's a SQL script that JOINS EventReceivers, AllWebs and AllLists to get that information:

    USE WSS_Content_ORG

    SELECT EventReceivers.Id AS EventReceiversID, EventReceivers.SiteID AS EventReceiversSiteID, EventReceivers.WebID AS EventReceiversWebID, AllWebs.Id AS AllWebsID, AllWebs.Title AS AllWebsTitle, AllWebs.FullUrl AS ServerRelativeURL, EventReceivers.HostType AS EventReceiversHostType, EventReceivers.hostId AS EventReceiversHostID, AllLists.tp_ID AS AllListsID,AllLists.tp_Title AS AllListsTitle

    FROM EventReceivers WITH (NOLOCK)

         LEFT JOIN (

               AllLists LEFT JOIN AllWebs ON AllLists.tp_WebId=AllWebs.Id

          ) ON EventReceivers.hostId=AllLists.tp_ID

    WHERE EventReceivers.Assembly='PDFForms.Services, Version=1.0.0.0, Culture=neutral, PublicKeyToken=ec1de91b62c9319e'

  • SQL queries were only used on a test instance of the content database mounted to a 2013 development farm; not on the copy of the 2010 content database that was eventually upgraded to the 2013 production farm.  This ensures that the copy of the content database actually upgraded to production remains in a supported state.

  • Some additional missing dependencies were resolved administratively, by removing lists, document libraries and sites on the 2010 production site, using administrative interfaces, before upgrading the site. This was accomplished by working with the site collection administrator to identify lists, document libraries and sites that could be removed without impact to user needs. For example, review of lists, document libraries and sites having PDF Share Forms features activated were found to have not ever been used or had no contents or were no longer being used. Such items could be deleted without impact to users, and this reduced the scope work required for resolving the missing PDF Share Forms dependencies.

  • The tasks discussed in this case study were part of an overall effort to upgrade the site collection from a production 2010 farm to a production 2016 farm.  The PDF Share Forms missing dependencies were removed on the 2013 development farm during the upgrade process.  The 2010 version of the PDF Share Forms solutions were installed on the 2010 farm.  No PDF Share Forms solutions were installed to the intermediary development 2013 farm nor on the development 2016 farm.  The 2016 version of PDF Share Forms solutions were installed to the production 2016 farm.

  • The overall upgrade process went like so:

    production 2010 >> development 2013 >> development 2016 >> production 2016

  • The Enumerate All Webs report is generated using STSADM, like so:

    CD C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\BIN\

    echo ^ > "D:\EnumAllWebs.xml"

    stsadm -o enumallwebs -databasename "WSS_Content_ORG" -includefeatures -includeeventreceivers -includesetupfiles >> "D:\EnumAllWebs.xml"

    This report lists site collections, webs, features etc in a hierarchical format, in XML, greatly facilitating simply text searches to find items by name, ID, etc.