Reduce the size of test data in VS 2010 by avoiding publishing deployment binaries into TFS
Over the past few weeks, we have heard a spate of complaints about TFS database being bloated with test data and growing in size rapidly with time. Running tests regularly generates different kinds of data in your TFS database, primary categories being:
- Deployment binaries - the pdbs and dlls that you use to deploy during your test run are all stored in TFS each time you kick off a test run. Typically, this data is needed only if user downloads the test result's trx file , associated test binaries and attempts to "rerun" the test in the same test result
- Test attachment data from diagnostic data adapters - Diagnostic data adapters like intellitrace, video recording, event logs etc. collect data during your test execution and store this data in the test run if you file bugs during the test run. This data is usually heavy in manual test runs since it is uncommon for automated test runs to capture all of this data each time a set of tests is run
- Code coverage files - The code coverage data diagnostic adapter when used in automated runs, generates a fairly heavy coverage file and also uploads all the pdb files of your app under test since the coverage file needs those. This can tend to be fairly heavy
After analyzing several customer TFS databases, we realized that 1 was contributing to a significant chunk of the data being accumulated in the TFS database - anywhere from 30% to 80% of all test data. Most users did not however want these binaries in TFS since the rerun operation was very rarely done. Given this, we released a QFE last week, that would stop uploading these deployment binaries into TFS every time a test run is published into TFS: Team Foundation Server 2010 Service Pack 1: Reduce the size of the test data saved to the TFS database
More details on the specifics of the fix are in the KB article from where you can also download the hotfix by simply typing out your email ID where you will receive the hotfix link. Sweet! I tried this with my mail id and here is the link I was sent.
To deal with the remaining test data though, we recommend running the Test Attachment Cleaner to routinely clean up the TFS database of unwanted or stale data. The tool is recommended to be run on the application tier of your TFS and has several modes that it can be executed in - see the description here for more details on the tool. The tool generates a whole lot of database records when you clean up your database and we recently learnt of a couple of SQL bugs that would cause these deleted records to persist as ghosted records instead of getting cleaned up automatically. To address this, SQL shipped 2 fixes in their Cumulative update package 10 for SQL Server 2008 R2 - it is recommended you install this update on your TFS before running the test attachment cleaner. In addition, we are also working on reducing (either in space or the number of db records) other kind of test data published to TFS.
In the meantime, if you are experiencing growth in your test data size in TFS, please download and install this hotfix and send your comments my way :-)
Comments
- Anonymous
November 01, 2011
Does SQL 2008 R2 SP1 has this issue?[Anu] Yes Alex - I believe you need this on top of the SP1 for SQL 2008 as well. This will be ported to the SP1 in the next cumulative release cycle - Anonymous
November 03, 2011
The comment has been removed - Anonymous
November 07, 2011
The comment has been removed - Anonymous
December 13, 2011
Cumulative update package can be installed only to instances not having the SP1. Update requires SQL Server 2008 R2 version 10.50.1600.1 and SP1 is 10.50.2500.0. So how this fix can be installed to instances with SP1, if the fix is not included in the SP1? - Anonymous
January 04, 2012
Hi Anu,Do you know what specific 2 fixes are in the Cumulative update package 10 for SQL Server 2008 R2 that helps address this problem?I found one KB fix mentioned below, but what is the other KB?VSTS bug number: 791684;791691KB article number: 2622823 Description: FIX: Increasing values of the ghost_record_count and version_ghost_record_count columns in SQL Server 2008 R2I can be reached at lanky_tx @ h o t m a i l.c o m[Anu] Mike - this is a single KB with 2 fixes - the ones you mention - Anonymous
January 13, 2012
Hi Anu,I've run the Test Attachment Cleaner & the logs tell me it has deleted lots of stuff, however when I look at the database I see no change in its size, nor in the size of the problem tables. From SQL Server management studio I ran a shrink job but it reported that only 5GB can be freed-up - the cleaner tool deleted ~35GB.Am I missing something here? Does the database now need to be re-indexed or something similar to free up the space? Any tips would be much appreciated.[Anu] Hi Marcus - can you please raise this as an issue via our CSS so we can help you debug the problem? If you dont have CSS support accessible, you can file a bug on connect so we can connect back with you to help. ThanksThanks