About Bulk Import and Bulk Export Operations
Microsoft SQL Server 2005 supports bulk exporting data from a SQL Server table and for bulk importing data into a SQL Server table or nonpartitioned view. The following basic methods are available.
| Method | Description | Imports data | Exports data |
|---|---|---|---|
A command-line utility (Bcp.exe) that bulk exports and bulk imports data and generates format files. |
Yes |
Yes |
|
A Transact-SQL statement that imports data directly from a data file into a database table or nonpartitioned view. |
Yes |
No |
|
A Transact-SQL statement that uses the OPENROWSET bulk rowset provider to bulk import data into a SQL Server table by specifying the OPENROWSET(BULK…) function to select data in an INSERT statement. |
Yes |
No |
In-Process vs. Out-of-Process Operation
The BULK INSERT statement and OPENROWSET(BULK) function execute in-process with SQL Server, sharing the same memory address space. Because the data files are opened by a SQL Server process, data is not copied between client process and SQL Server processes. For security considerations when importing data by using BULK INSERT or INSERT ... SELECT * FROM OPENROWSET(BULK...), see Importing Bulk Data by Using BULK INSERT or OPENROWSET(BULK...).
In contrast, the bcp utility runs out-of-process. To move data across process memory spaces, bcp must use inter-process data marshaling. Inter-process data marshaling**is the process of converting parameters of a method call into a stream of bytes. This can add significant load to the processor. However, because both bcp parses the data and convert data into native storage format in the client process, they can offload parsing and data conversion from the SQL Server process. Therefore, if you have a CPU constraint, you may achieve better bulk-import performance on a computer that has more than one CPU or on different computers, by using bcp instead of by using BULK INSERT or INSERT ... SELECT * FROM OPENROWSET(BULK).
Format Files
The bcp utility, BULK INSERT, and INSERT ... SELECT * FROM OPENROWSET(BULK...) all support the use of a specialized format file that stores format information for each field in a data file. A format file might also contain information about the corresponding SQL Server table. The format file can be used to provide all the format information that is required to bulk export data from and bulk import data to an instance of SQL Server.
Format files provide a flexible way to interpret data as it is in the data file during import, and also to format data in the data file during export. This flexibility eliminates the need to write special-purpose code to interpret the data or reformat the data to the specific requirements of SQL Server or the external application. For example, if you are bulk exporting data to be loaded into an application that requires comma-separated values, you can use a format file to insert commas as field terminators in the exported data.
SQL Server 2005 supports two kinds of format files: XML format files and non-XML format files. Non-XML format files are supported by earlier versions of SQL Server; XML format files are new in SQL Server 2005.
The bcp utility is the only tool that can generate a format file. For more information, see Creating a Format File. For more information about format files, see Format Files for Importing or Exporting Data.
Note
In cases when a format file is not supplied during a bulk export or import operations, the user can choose to override the default formatting at the command line.
The Query Processor and Bulk Import
To bulk import data into an instance of SQL Server, the bcp utility, BULK INSERT statement, and INSERT ... SELECT * FROM OPENROWSET(BULK...) statement all work with the query processor.
All three methods convert the data in a data file into OLE DB rowsets. But the conversion method varies, as follows:
- The bcp utility reads the data file and sends a TDS stream to the SQL Server Bulk Copy Program (BCP) API, which converts the data into OLE DB rowsets.
- BULK INSERT and the OPENROWSET bulk rowset provider both convert a file data directly into an OLE DB rowset.
The OLE DB rowsets are inserted into the target table by the query processor, which plans and optimizes each operation.
Performance Considerations
Performance considerations can also be significant when large amounts of data are being imported. In some cases, performance can be improved by changing how a bulk-import or bulk-export operation handles one or more of the following:
- Batch switches
- Constraint checking of CHECK constraints
- How bulk transactions are logged. This is relevant for databases that typically use the full recovery model.
- Ordering exported data
- Parallel data importing
- Table locking
- Trigger execution
For more information, see Optimizing Bulk Import Performance.
Note
No special optimization techniques exist for bulk-export operations. These operations simply select the data from the source table by using a SELECT statement.
See Also
Concepts
Basic Guidelines for Bulk Importing Data
Data Formats for Importing or Exporting Data
Scenarios for Bulk Importing and Exporting Data
Single SQL Statement Processing
Other Resources
Performing Bulk Load of XML Data (SQLXML 4.0)
SQL Server Integration Services
Performing Bulk Copy Operations
bcp Utility
BULK INSERT (Transact-SQL)
Format Files for Importing or Exporting Data
OPENROWSET (Transact-SQL)