Dynamic editing using foreach loop "Variable does not contain valid data object" and a dynamic query data flow error
Hi! I'm currently trying to create an SSIS package that dynamically acquire columns from an azure SQL database and transform it. This is my current setup:
Populate column list has the following query:
SELECT COLUMN_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'PerformanceRating'
AND COLUMN_NAME LIKE 'Performance_Rating%'
AND COLUMN_NAME NOT LIKE '%20XX%';
It takes the headers that contain performance rating data in their column and stores it in a object variable called ColumnList (Below is variables lista nd output of columnList
Transform Obj to Str transforms ColumnList from an object class into a string so that DynamicSQLQuery variable can be used:
This is the dynamic SQL query:
"SELECT Employee_Number, " + @[Performance_Rating::ColumnListStr] + " FROM PerformanceRating WHERE " +
REPLACE(REPLACE(@[Performance_Rating::ColumnListStr], ",", " IS NOT NULL OR "), " IS NOT NULL OR ", " IS NOT NULL")
And this is the output which is correct:
The foreach loop is contains a data flow component that houses an OLE DB Source and Derived column (for now) OLE DB Source uses the DynamicSQLQuery to acquire all rows for the columns with rating data and the derived column has substring to make 2 new columns for each rating column and splits the number and letter (e.g. a performance rating of A2 gets split to 2 columns, 1 containing A and the other the number 2.) I currently have 2 issues. The first is the foreach loop, where this error is shown:
These are my current configurations:
The second issue is to do with the Data flow, where using the DynamicSQLQuery looks like this:
and I'm unable to run the foreach loop because of this error (Even though the DynamicSQLQuery is only completed when I begin running the package)
I have tried many solutions with these 2 issues to no avail, any ideas on how to solve them?