Configure Salesforce in a copy activity
This article outlines how to use the copy activity in data pipeline to copy data from and to Salesforce.
Supported configuration
For the configuration of each tab under copy activity, go to the following sections respectively.
General
Refer to the General settings guidance to configure the General settings tab.
Source
The following properties are supported for Salesforce under the Source tab of a copy activity.
The following properties are required:
- Connection: Select a Salesforce connection from the connection list. If no connection exists, then create a new Salesforce connection by selecting New.
- Use query: Select from Object API, Report, or SOQL Query.
If you select Object API:
- Object API: Specify the Salesforce object name to retrieve data from. Select the name from the drop-down list.
If you select Report:
Report ID: Specify the ID of the Salesforce report to retrieve data from. It isn't supported in destination. There are limitations when you use reports.
If you select SOQL Query:
SOQL Query: Use the custom query to read data. You can only use Salesforce Object Query Language (SOQL) query with limitations Understanding Bulk API 2.0 Query. If you don't specify SOQL query, all the data of the Salesforce object specified in Object API or Report ID will be retrieved.
Under Advanced, you can specify the following fields:
- Include deleted objects: Specify whether to query the existing records (unselected), or query all records including the deleted ones (selected).
- Additional columns: Add additional data columns to store source files' relative path or static value. Expression is supported for the latter.
Destination
The following properties are supported for Salesforce under the Destination tab of a copy activity.
The following properties are required:
- Connection: Select a Salesforce connection from the connection list. If no connection exists, then create a new Salesforce connection by selecting New.
- Object API: Specify the Salesforce object name to write data to. Select the name from the drop-down list.
Under Advanced, you can specify the following fields:
Write behavior: Specify the write behavior for the operation. Allowed values are Insert and Upsert. You can choose a behavior from the drop-down list.
If you select Upsert:
External ID field: The name of the external ID field for the upsert operation. The specified field must be defined as External ID Field in the Salesforce object. It can't have NULL values in the corresponding input data.
Ignore null values: Specify whether to ignore NULL values from input data during a write operation.
- When it is selected: Leave the data in the destination object unchanged when you do an upsert or update operation. Insert a defined default value when you do an insert operation.
- When it is unselected: Update the data in the destination object to NULL when you do an upsert or update operation. Insert a NULL value when you do an insert operation.
Write batch size: Specify the row count of data written to Salesforce in each batch. Suggest set this value from 10,000 to 200,000. Too few rows in each batch reduces copy performance. Too many rows in each batch may cause API timeout.
Max concurrent connections: The upper limit of concurrent connections established to the data store during the activity run. Specify a value only when you want to limit concurrent connections.
Mapping
For Mapping tab configuration, see Configure your mappings under mapping tab.
Settings
For Settings tab configuration, go to Configure your other settings under settings tab.
Table summary
The following tables contain more information about the copy activity in Salesforce.
Source information
| Name | Description | Value | Required | JSON script property |
|---|---|---|---|---|
| Connection | Your connection to the source data store. | <your Salesforce connection> | Yes | connection |
| Connection type | Your connection type. Select Salesforce objects. | SalesforceObject | Yes | / |
| Use query | The way to read data from Salesforce. | • Object API • Report • SOQL Query |
Yes | / |
| For Object API | ||||
| Object API | Specify the Salesforce object name to retrieve data from. | < your object name > | Yes | objectApiName |
| For Report | ||||
| Report ID | The ID of the Salesforce report to retrieve data from. It isn't supported in destination. There are limitations when you use reports. | <your report ID> | Yes | reportId |
| For SOQL Query | ||||
| SOQL Query | Use the custom query to read data. You can only use Salesforce Object Query Language (SOQL) query with limitations Understanding Bulk API 2.0 Query. If you don't specify SOQL query, all the data of the Salesforce object specified in Object API or Report ID will be retrieved. | < your SOQL query > | Yes | query |
| Include deleted objects | Indicates whether to query the existing records, or query all records including the deleted ones. | selected or unselected (default) | No | includeDeletedObjects: true or false (default) |
| Additional columns | Add additional data columns to store source files' relative path or static value. Expression is supported for the latter. | • Name • Value |
No | additionalColumns: • name • value |
Destination information
| Name | Description | Value | Required | JSON script property |
|---|---|---|---|---|
| Connection | Your connection to the destination data store. | <your Salesforce connection> | Yes | connection |
| Object API | Specify the Salesforce object name to write data to. | < your object name > | Yes | objectApiName |
| Write behavior | The write behavior for the operation. Allowed values are Insert and Upsert. You can choose a behavior from the drop-down list. | • Insert • Upsert |
No (default is Insert) | writeBehavior: insert upsert |
| External ID field | The name of the external ID field for the upsert operation. The specified field must be defined as External ID Field in the Salesforce object. It can't have NULL values in the corresponding input data. | < your external ID field > | Yes for "Upsert" | externalIdFieldName |
| Ignore null values | Indicates whether to ignore NULL values from input data during a write operation. | selected or unselected (default) | No | ignoreNullValues: true or false (default) |
| Write batch size | The row count of data written to Salesforce in each batch. Suggest set this value from 10,000 to 200,000. Too few rows in each batch reduces copy performance. Too many rows in each batch may cause API timeout. | <number of rows> (integer) |
No (default is 100,000) | writeBatchSize |
| Max concurrent connections | The upper limit of concurrent connections established to the data store during the activity run. Specify a value only when you want to limit concurrent connections. | <max concurrent connections> | No | maxConcurrentConnections |
Salesforce Bulk API 2.0 Limits
We use Salesforce Bulk API 2.0 to query and ingest data. In Bulk API 2.0, batches are created for you automatically. You can submit up to 15,000 batches per rolling 24-hour period. If batches exceed the limit, you encounter failures.
In Bulk API 2.0, only ingest jobs consume batches. Query jobs don't. For details, see How Requests Are Processed in the Bulk API 2.0 Developer Guide.
For more information, see the General Limits section in Salesforce developer limits.