RBS Garbage Collection Settings and Rationale

I recently got a question on RBS Garbage Collection settings and their usage. So I decided to write this blog post describing the different parts of GC and the associated settings.

 

RBS Maintainer does garbage collection (GC) in 3 phases:

1. Reference Scan (RS) - Look through the application tables and find blobs that are no longer referenced by the application. The list of registered RBS columns is used for this purpose. BlobIds must not be stored in any place other than the registered columns. The blobs that are no longer referenced by the application are marked to be deleted.

2. Delete Propagation (DP) - Blobs marked for deletion are actually deleted from the blob store. There is a gap between when the blobs get marked for deletion and when they are actually deleted. This gap duration can be configured using the "garbage_collection_time_window" config item and defaults to 30 days. The reason for having this GC time window is to allow restoring old backups of the RBS database. Backups as old as the time window (e.g. 30 days) can be restored and all blobs that were referenced by that database are guaranteed to be present in the blob store. If we had deleted the blobs immediately, restoring an old backup will lead to dangling pointers (some blobs referenced by the application are not present in the blob store). Having this gap ensures that if an old backup of the database is restored, blobs that were referenced by the application at the time the backup was taken (but deleted by the application later) are still present in the blob store. For this reason, this config item must be set to the SLA time period for backup/restore.

3. Orphan Cleanup (OC) - All blobs in the blob store are enumerated and we compute the list of blobs that are present in the blob store but are not known to RBS. These blobs are "orphans" and can be caused due to aborted transactions, application misbehavior or other failures. Orphan blobs created before the GC time window are deleted from the blob store.

In addition to the GC time window setting, there are 2 more config items related to GC: "delete_scan_period" is the time period for running one scan of RS and DP phases of GC. After one pass of RS and DP is completed, attempting to run them again within this time period will just skip RS and DP and do nothing. Similarly, "orphan_scan_period" is the period for the OC phase of GC. These are also 30 days by default.

These settings can be set by calling rbs_sp_set_config_value. The format for these config items is: 'days n' where n is a positive number. For testing purposes, it can also be set to sub-day durations using the format 'time hh:mm:ss'. A smalldatetime field is used internally, so the precision of this setting is 1 minute. It can also be set to 0 using 'time 00:00:00' .

The actual work of GC is done by the RBS Maintainer application. The maintainer is a console application that takes command line parameters such as the connection string to the database and the phases of GC to execute. This can be run from any machine that has access to the DB and the blob store(s). It can also be run from multiple machines simultaneously. You can schedule it using your favorite scheduler e.g. Windows Task Scheduler.

Maintainer also takes an optional parameter to limit the amount of time it is run. Here is a scenario showing how this time limit can be used in conjunction with the scan_period settings above for a GC schedule:

· Maintenance window in production environment is 2 hours every day from 2 AM to 4 AM.

· A complete GC pass takes 6-10 hours to run.

· Customer wants to run one GC pass every week (7 days).

Solution:

· Schedule a Maintainer.exe task to run every day at 2 AM, and include the command line option " -TimeLimit 120" - this will stop maintainer after running for 2 hours, even if GC pass is not complete. The time limit is specified in minutes.

· On the database, set the RBS config values of delete_scan_period and orphan_scan_period to 7 days.

This way, Maintainer.exe will run on Mon-Thu (assuming 7 hours run time) and then do nothing on Fri-Sun instead of starting a new GC scan. Next Monday, a new GC scan is started.

For testing purposes, you can do the following to verify that blobs are getting deleted from the blob store:

1. Set garbage_collection_time_window and delete_scan_period to 'time 00:00:00'

2. Delete BlobIds from the application table

3. Run Maintainer.exe, specifying RS and DP phases

Blobs should get deleted from the blob store at this point.

 

- Pradeep

Comments

• Anonymous
  April 08, 2009
  What are the recommendations for the maintainer account?  An existing service account?  A dedicated account?  A local admin?

• Anonymous
  April 08, 2009
  It is preferable to have a dedicated service account for RBS Maintainer. This account does not need local admin / SQL Admin / DBA privileges. It only needs to have access to the DB with rbs_maintainer role and RW access to the blob stores.

• Anonymous
  January 11, 2010
  The comment has been removed

• Anonymous
  January 13, 2010
  RBS Maintainer runs as a seperate executable that must be scheduled outside of SharePoint's control. It's on a per-database level (for SharePoint this would be a content database) so any number of providers can be installed in a db and only one maintainer process needs to be run. The maintainer runs against RBS and the application's metadata tables to determine which blobs require removal. Once a blob requires deletion, a specific deletion request is sent to the provider for that individual blob. The provider only needs to implement the provider interface (of which one of the methods is DeleteBlob) and does not have any knowledge about the application or even the RBS metadata tables. There's some information in the blog and in the examples on the codeplex site (http://www.codeplex.com/sqlrbs) but this doesn't comprehensively cover the interaction of SharePoint and RBS. As O14 and SQL Server 2008 R2 move towards release you can expect more information to become available, in the meantime feel free to ask any questions you may have. Thanks Mike

• Anonymous
  January 18, 2010
  I have successfully implemented the CodePlex RBS Provider for SharePoint 2010 (Beta).  I'm playing with the Maintainer application now to understand how the different commands are used to clean up the orphans. So to test a complete "uninstall" of an RBS provider, I migrated all binary content back inline to the database and disabled RBS in all SharePoint content databases.  I also used the information above to set my garbage_collection_time_window and delete_scan_period to "00:00:00". It seems to me that I should now be able to run the Maintainer and it should clean up (delete) all of the old BLOBs from my BLOB store.  But that doesn't seem to be the case.  All of the old (irrelevent) BLOBs are just haging out in my BLOB store after running all garbage collection phases using Maintainer. Am I missing something or am I trying to do something that the Maintainer was not meant for?

• Anonymous
  January 19, 2010
  It's hard to debug what's happening without more information, but here's some things to try. You can check the SharePoint content database all_doc_streams table to see if there's still RBS ID entries in the table. SharePoint has a multi-level delete where the initial delete moves the file to a first level recycle bin accessible to the user, then if the user clears that it moves to the administrator's recycle bin. Both these need to be cleared for the blob to be truly deleted from the content database. Only once the blob is not in the SharePoint content db will the GC consider it for removal. It sounds like you have the time windows and scan periods set correctly. The maintainer logs would be helpful if you still have issues.

• Anonymous
  January 19, 2010
  Hi Russ, You have followed the right steps. Can you give us the console output of Maintainer.exe? Another place to look at what happened is to see the contents of views mssqlrbs.rbs_history and mssqlrbs.rbs_counters. In addition, there is an internal table that is not meant to be used by applications/tools/utilities (because we reserve the right to change it in the future, and only DBO has access to it), but can be used for troubleshooting: mssqlrbs_resources.rbs_internal_deleted_blobs. This table has the list of blobs that Reference Scan found to be no longer used by the application (e.g. SharePoint). The Delete Propagation phase goes through this list of blobs and deletes them from the blob store as well as from this table. If you still cannot find where the issue is, you can give us access to the database and we will take a look to find out what happened. Do post back to let us know what the issue was. Thanks, Pradeep.

• Anonymous
  February 12, 2010
  Pradeep, I appreciate your willingness to help.  I'm hoping to get something to you soon.  Unfortuntately, as you may have guessed, I ran out of "free" time to continue R&D on this.  So unfortunately, at the moment I don't have time to gather the Maintainer console output.  But I'll get it eventually. (Didn't want to leave you hanging on this comment post.) But I still need more information on this topic.  Is there any documentation on Maintainer anywhere?  There are several commands and switches and I can't seem to find them documented anywhere except for the contents of this blog post and the Maintainer syntax help. Best Regards, Russ

• Anonymous
  February 12, 2010
  That is all the public documentation about RBS Maintainer that is available as of now. Some more will be added with the SQL Server 2008 R2 release in a few months from now. Feel free to ask questions on this blog for anything you want more explanation on.

• Anonymous
  April 13, 2010
  Pradeep, Do you know what the  configurations in rbs_internal_config mean? Best Regards, Lee April 14, 2010 9:17 AM

• Anonymous
  April 19, 2010
  rbs_internal_config is an internal table and is not supposed to be used by applications/customers. It contains config items that are publicly exposed, items to be used by support personnel for troubleshooting  as well as some internal ones used for storing state or other information for the working of RBS. Users/applications are only allowed to use the view rbs_config that shows the publicly exposed config items.

• Anonymous
  September 26, 2010
  Hi I'm using sharepoint 2010 and RBS. The Database is clustered. In this case, do i need to use Database agent instead of maintainer? Thanks.

• Anonymous
  January 03, 2011
  The comment has been removed