Select transformation output schema is empty after renaming columns

Question

I've followed the example from the link:

https://learn.microsoft.com/en-us/azure/data-factory/tutorial-data-flow-dynamic-columns

It's exactly what I need for my project, but unfortunately, no matter what I do, the output schema of the select transformation remains empty.

Data Preview of the select transformation (and all subsequent transformations) shows that columns are, in fact, renamed as expected.

I could map drifted columns, but that transformation maps only renamed columns, and the rest I'd need to manually map which is not ideal when dealing with a lot of non-drifted columns, but ideally, I shouldn't even need to do this.

Is there a reason why this happens, and what would be the solution (preferably not a hack-y one :D)?

Answer 1

Hi AdnanT ,

Welcome to Microsoft Q&A platform and thanks for posting your query here.

It seems you are trying to perform dynamic mapping in dataflow but it is not working giving empty schema in the output.

The official documentation tutorial which you have shared is suggesting to use cached sink. Could you please confirm if sink2 in your solution is cached sink ?

Since you are getting empty schema in output, kindly check if there are any null values in the source data. If there are null values, try replacing them with default values or removing them.

You can try using map drifted option in data preview tab and manually map the non-drifted columns once.

Hope that helps . Kindly let us know how it goes. Thankyou