Preparing to Query for the Change Data

In the control flow of an Integration Services package that performs an incremental load of change data, the third and final task is to prepare to query for the change data and add a Data Flow task.

Note

The second task for the control flow is to ensure that the change data for the selected interval is ready. For more information about this task, see Determining Whether the Change Data Is Ready. For a description of the overall process of designing the control flow, see Improving Incremental Loads with Change Data Capture.

To see complete, end-to-end samples that demonstrate the use of change data capture in packages, see the Integration Services samples on Codeplex.

Design Considerations

To retrieve the change data, you will call a Transact-SQL table-valued function that accepts the endpoints of the interval as input parameters and returns change data for the specified interval. A source component in the data flow calls this function. For information about this source component, see Retrieving and Understanding the Change Data.

The most frequently used Integration Services source components, including the OLE DB source, the ADO source, and the ADO NET source, cannot derive parameter information for a table-valued function. Therefore, most sources cannot call a parameterized function directly.

You have two design options for passing the input parameters to the function:

• Assemble the parameterized query as a string. You can use a Script task or an Execute SQL task to assemble a dynamic SQL string with parameter values hard-coded into the string. Then, you can store this string in a package variable and use it to set the SqlCommand property of a source component. This approach succeeds because the source component no longer requires parameter information.

  Note

  A precompiled script incurs less overhead than an Execute SQL task.

• Use a parameterized wrapper. Alternatively, you can create a parameterized stored procedure as a wrapper that calls the parameterized table-valued function. This approach succeeds because a source component can successfully derive parameter information for a stored procedure.

This topic uses the first design option and assembles a parameterized query as a string.

Preparing the Query

Before you can concatenate the values of the input parameters into a single query string, you have to set up the package variables that the query needs.

To set up package variables

• In Business Intelligence Development Studio, in the Variables window, create a variable with a string data type to hold the query string returned by the Execute SQL Task.

  This example uses the variable name, SqlDataQuery.

With the package variable created, you can use either a Script task or Execute SQL task to concatenate the values of the input parameters. The following two procedures describe how to configure these components.

To use a Script task to concatenate the query string

1. On the Control Flow tab, add a Script task to the package after the For Loop container and connect the For Loop container to the task.

   Note

   This procedure assumes that the package performs an incremental load from a single table. If the package loads from multiple tables and has a parent package with multiple child packages, this task would be added as the first component to each child package. For more information, see Performing an Incremental Load of Multiple Tables.

2. In the Script Task Editor, on the Script page, select the following options:

   1. For ReadOnlyVariables, select User::DataReady, User::ExtractStartTime, and User::ExtractEndTime from the.

   2. For ReadWriteVariables, select User::SqlDataQuery from the list.

3. In the Script Task Editor, on the Script page, click Edit Script to open the script development environment.

4. In the Main procedure, enter one of the following code segments:

   • If you are programming in C#, enter the following lines of code:

         int dataReady;
         System.DateTime extractStartTime;
         System.DateTime extractEndTime;
         string sqlDataQuery;
     
         dataReady = (int)Dts.Variables["DataReady"].Value;
         extractStartTime = (System.DateTime)Dts.Variables["ExtractStartTime"].Value;
         extractEndTime = (System.DateTime)Dts.Variables["ExtractEndTime"].Value;
     
         if (dataReady == 2)
           {
           sqlDataQuery = "SELECT * FROM CDCSample.uf_Customer('" + string.Format("{0:yyyy-MM-dd hh:mm:ss}", extractStartTime) + "', '" + string.Format("{0:yyyy-MM-dd hh:mm:ss}", extractEndTime) + "')";
           }
         else
           {
           sqlDataQuery = "SELECT * FROM CDCSample.uf_Customer(null" + ", '" + string.Format("{0:yyyy-MM-dd hh:mm:ss}", extractEndTime) + "')";
           }
     
         Dts.Variables["SqlDataQuery"].Value = sqlDataQuery;
     

     - or -

   • If you are programming in Visual Basic, enter the following lines of code:

         Dim dataReady As Integer
         Dim extractStartTime As Date
         Dim extractEndTime As Date
         Dim sqlDataQuery As String
     
         dataReady = CType(Dts.Variables("DataReady").Value, Integer)
         extractStartTime = CType(Dts.Variables("ExtractStartTime").Value, Date)
         extractEndTime = CType(Dts.Variables("ExtractEndTime").Value, Date)
     
         If dataReady = 2 Then
           sqlDataQuery = "SELECT * FROM CDCSample.uf_Customer('" & _
               String.Format("{0:yyyy-MM-dd hh:mm:ss}", extractStartTime) & _
               "', '" & _
               String.Format("{0:yyyy-MM-dd hh:mm:ss}", extractEndTime) & _
               "')"
         Else
           sqlDataQuery = "SELECT * FROM CDCSample.uf_Customer(null" & _
               ", '" & _
               String.Format("{0:yyyy-MM-dd hh:mm:ss}", extractEndTime) & _
               "')"
         End If
     
         Dts.Variables("SqlDataQuery").Value = sqlDataQuery
     

5. Leave the default line of code which returns DtsExecResult.Success from the execution of the script.

6. Close the script development environment and the Script Task Editor.

To use an Execute SQL task to concatenate the query string

1. On the Control Flow tab, add an Execute SQL task to the package after the For Loop container and connect the For Loop container to this task.

   Note

   This procedure assumes that the package performs an incremental load from a single table. If the package loads from multiple tables and has a parent package with multiple child packages, this task would be added as the first component to each child package. For more information, see Performing an Incremental Load of Multiple Tables.

2. In the Execute SQL Task Editor, on the General page, select the following options:

   1. For ResultSet, select Single row.

   2. Configure a valid connection to the source database.

   3. For SQLSourceType, select Direct input.

   4. For SQLStatement, enter the following SQL statement:

      declare @ExtractStartTime datetime,
      @ExtractEndTime datetime, 
      @DataReady int
      
      select @DataReady = ?, 
      @ExtractStartTime = ?, 
      @ExtractEndTime = ?
      
      if @DataReady = 2
      select N'select * from CDCSample.uf_Customer'
      + N'('''+ convert(nvarchar(30),@ExtractStartTime,120)
      + ''', '''
      + convert(nvarchar(30),@ExtractEndTime,120) + ''') ' 
      as SqlDataQuery
      else
      select N'select * from CDCSample.uf_Customer'
      + N'(null, '''
      + convert(nvarchar(30),@ExtractEndTime,120)
      + ''') '
      as SqlDataQuery
      

      Note

      The else clause in this sample generates a query for the initial load of change data by passing a null value for the starting date and time. This sample does not address the scenario in which changes that were made before change data capture was enabled also have to be uploaded to the data warehouse.

3. On the Parameter Mapping page of the Execute SQL Task Editor, do the following mapping:

   1. Map the DataReady variable to parameter 0.

   2. Map the ExtractStartTime variable to parameter 1.

   3. Map the ExtractEndTime variable to parameter 2.

4. On the Result Set page of the Execute SQL Task Editor, map the Result Name to the SqlDataQuery variable.

   The Result Name is the name of the single column that is returned, SqlDataQuery.

The previous procedures configure a task that prepares a query string with hard-coded string values for the input parameters. The following code is an example of such a query string:

select * from CDCSample. uf_Customer('2007-06-11 14:21:58', '2007-06-12 14:21:58')

Adding a Data Flow Task

The last step in designing the control flow for the package is to add a Data Flow task.

To add a Data Flow task and complete the control flow

• On the Control Flow tab, add a Data Flow task and connect the task that concatenated the query string.

Next Step

After you prepare the query string and configure the Data Flow task, the next step is create the table-valued function that will retrieve the change data from the database.

Next topic:Creating the Function to Retrieve the Change Data

Stay Up to Date with Integration Services For the latest downloads, articles, samples, and videos from Microsoft, as well as selected solutions from the community, visit the Integration Services page on MSDN or TechNet: Visit the Integration Services page on MSDN Visit the Integration Services page on TechNet For automatic notification of these updates, subscribe to the RSS feeds available on the page.