Staging the Data
This topic discusses the fourth step in the process of importing custom data into the Data Warehouse. For the previous step, see Preparing the Data Warehouse.
The main task in staging the data involves extracting the pertinent information from the data source. This information is formatted for direct import into the Data Warehouse schema structure.
Source, Process, and Destination
The following table summarizes the source, process, and destination for the data.
Source data (existing log file or commerce object) |
Processing |
Destination (new staged data file) |
|---|---|---|
Title (string) |
Copy |
UserTitle (string) |
Fullname (string) |
Parse First Name |
FirstName (string) |
FullName (string) |
Parse Last Name |
LastName (string) |
Email_address (string) |
Copy |
Email (string) |
City (string) |
Copy |
City (string) |
State (string) |
Copy |
State (string) |
Fav_Flavor_Code (Integer) |
Convert to ULONG |
Fav_Flavor_Code (ULONG) |
As the input log file is processed, keep track of the number of entries. This will help simplify the last step in the import process.
For the running example, the data in the staged data file is grouped by user. The final step requires a function that can extract the next data value from the staged data file. No effort is made in this example to optimize the processing.
The data is now staged and ready for import. For the final step, see Executing the Custom Import Task.