Freigeben über

Migrating a large collection from on premises TFS to Azure hosted service (VSTS)

  • Artikel

 

The TFS Database Import Service provides a way for Team Foundation Server (TFS) customers to complete a high-fidelity migration into Visual Studio Team Services (VSTS). It works by importing a collection in its entirety into a brand new VSTS account.

The Import Service has been used by large number of customers, small-to-large, to successfully migrate to VSTS. This includes using it internally at Microsoft to move older TFS instances into VSTS. Helping us achieve our larger goal of using one engineering system – more formerly called the 1ES effort. As you can imagine, there are lots of exceptionally large collections within at Microsoft. We thought we could share our experiences from a recent migration of a collection that was around 10.5 TB in total size. Yep, that TB for Terabytes 😊.

Please refer to my other blogs for details about the process, tool, common commands etc. here. You can also learn more about the Import Service by visiting https://aka.ms/tfsimport.

Below are some statistics on the size of the collection we imported. It’s truly a big one!

 

Description Numbers
Full database size 10578 GB
Database meta data size 583 GB
Number of users 2800
Team Projects 5
Work Items Count 19.2 Million
Test runs Count 1.3 Million
Builds Count 28.7 Thousand

 

Migration time:

Migrating the collection including all the data within the collection took ~92 hours. Please note that the migration time may differ based on the data/content size, content type, network speed etc.

Performance:

For most of companies/team’s performance of the service is crucial factor to make the change. I am glad to share that in our testing after migrating this large collection, the work items, builds and other information was accessible in similar of less time than on-premises collection.

What if I need help

Please reach out to migration team in Microsoft (vstsdataimport@microsoft.com)  to discuss/plan the migration plan or if you have any queries.

Tips & Tricks:

If your on-premises SQL server does not have in-bound & out-bound internet collection you can host the SQL database to an azure hosted SQL virtual machine (VM). It’s recommended to host the VM in the same region where you are planning to host your VSTS account, data transfer rate from the same region will be much better.

If you need to transfer collection backup to Azure hosted VM, plan to divide the backup file in multiple files (I divided backup to 24 files) and use power of parallelism to transfer files).

Below are details to speed-up the process:

  1. If your collection is not hosted on a sever which allows inbound internet traffic, you will have to restore the collection on an internet inbound/outbound enabled server virtual machine. To reduce overhead of network latency, It's  recommenced that you host the server/virtual machine in the same region as your planned VSTS account. Please refer to migration guide for the setup options.

 

  1. Collection database Backup: Divide collection backup into multiple files (16 to 24 files) and use MAXTRANSFERSIZE hint. This approach reduced total backup duration by 75% for our collection database.

 

Example:

BACKUP DATABASE [Tfs_DefaultCollection] TO

DISK = N'H:\MSSQL\BAK\Tfs_DefaultCollection_1.bak',

DISK = N'H:\MSSQL\BAK\Tfs_DefaultCollection_2.bak',

DISK = N'H:\MSSQL\BAK\Tfs_DefaultCollection_3.bak',

DISK = N'H:\MSSQL\BAK\Tfs_DefaultCollection_4.bak',

DISK = N'H:\MSSQL\BAK\Tfs_DefaultCollection_5.bak',

.

.

.

.

DISK = N'H:\MSSQL\BAK\Tfs_DefaultCollection_23.bak',

DISK = N'H:\MSSQL\BAK\Tfs_DefaultCollection_24.bak'

 

WITH NOFORMAT, NOINIT,  NAME = N'Tfs_DefaultCollection-Full Database Backup',

SKIP, NOREWIND, NOUNLOAD, COMPRESSION, MAXTRANSFERSIZE = 4194304, STATS = 5

GO

 

  1. Use AZCopy to transfer backup to Azure blob/ VM: You can get optimal performance for database files copy to Azure blob/VM using AZCopy utility.

 

  1. Initiate multiple instance of AZCopy simultaneously: Running multiple instances of AZCopy in parallel (for different files upload) would help speed-up the upload/download files to/from blob storage. You will have to use /z switch to run parallel instances of AZCopy

 

Example: AzCopy /Source:H:\MSSQL\BAK\ /Dest:https://<storage account>.blob.core.windows.net/tfsMigration /SourceKey:"<SAS>" /pattern:"File.bak" /S /z:<unique folder for azcopy journal file>

 

Summary:

In addition to small collections you can migrate large collections to VSTS and can take advantage of hosted service.

 

Special thanks to Rogan Ferguson for review & contribution.