Compare Look up transformation over Truncate load in case of Incremental load
While populating data warehouse, we have to do incremental load, and to do so, there are multiple ways to do that.
One is old traditional way of truncating the data in destination and loading it with the fresh data from source, and then move it to data warehouse. But this has many consequences. But our discussion will move around the comparisons between the two ways
Second way is to use lookup and check which all rows are present in destination, and perform necessary insert and updates. This will utilize the memory comparisons and minimum amount of writes will be performed
In traditional method a lot of disk writes are preformed, whereas in case of look up very few writes will happen. This improves the performance multifold
I will create two packages one utilizing the first method, and the other utilizing the power of second method, and then we will compare the performance