Moving Large Files Across the WAN using BITS
In most data center environments there's a requirement to move large amounts of data from one location to another. This could be from one server within a site to another at the same site or between two geographically distributed data centers. While the first scenario is relatively simple the second is a bit more challenging and forces you to contend with inherent WAN issues such as latency (timeouts).
To address this requirement the MPSC looked at several pre-existing technologies used for massive file distribution systems such as the Windows Update services and discovered the common component of BITS (Background Intelligent Transfer Service). BITS enables applications and powershell access for asynchronously transfering large files while preserving the the responsiveness of other network applications. And provided the ability to restart file transfers in the event of a network outage/latency or even if the system reboots.
A primer for using BITS can be found in the MSDN site https://msdn.microsoft.com/en-us/library/aa362708(VS.85).aspx however there are two critical pieces, the server and the client. In our case we needed the server to be highly available to a large number of customers. And for the client we decided on the use of powershell for easier automation through the use of powershell.
Using IIS for receiving files via BITS:
- On Windows Server 2008, use the Server Manager to install the BITS Server Extensions feature. From Server Manager, click Features in the left pane. In the Add Features Wizard, check BITS Server Extensions. Note that the IIS 6 Management Compatibility roles must be installed.
- To use the IIS user interface to configure the virtual directory for BITS uploads, select Properties from the context menu of the virtual directory. Press the BITS Extension tab to access the BITS properties page (the BITS server extension ISAPI must be installed). Change the default property values as appropriate and enable the directory for uploads; you cannot upload files until the directory is enabled. For details on setting the property values, see the online Help.
Using Powershell to upload files:
Simple sample powershell script to upload a file -
Import-Module BitsTransfer
#Ask for user authentication credentials for accessing the server
$c = Get-Credential
Start-BitsTransfer -source SourceFilePathAndName -Destination ServerURLandPathAndFullFilename -transfertype upload -Credential $c
#To get more information about other cmdlets execute the following:
#get-help start-bitransfer -examples | more
NOTE: The powershell CmdLet for BITS changed from the Windows 7 Beta to the Windows 7 RC. Primarily the modiule changed from Filetransfer to BitsTransfer.
Jeffrey Amels - MPSC Technical Operations Lead