Select transformation output schema is empty after renaming columns

AdnanT 0

I've followed the example from the link:

https://learn.microsoft.com/en-us/azure/data-factory/tutorial-data-flow-dynamic-columns

It's exactly what I need for my project, but unfortunately, no matter what I do, the output schema of the select transformation remains empty.

User's image

Data Preview of the select transformation (and all subsequent transformations) shows that columns are, in fact, renamed as expected.

I could map drifted columns, but that transformation maps only renamed columns, and the rest I'd need to manually map which is not ideal when dealing with a lot of non-drifted columns, but ideally, I shouldn't even need to do this.

Is there a reason why this happens, and what would be the solution (preferably not a hack-y one :D)?

AdnanT 0 Reputation points

2024-11-20T16:29:26.23+00:00

This happens in both Data Factory and Synapse
AnnuKumari-MSFT 33,976 Reputation points Microsoft Employee

2024-11-24T06:31:24.9933333+00:00

Hi AdnanT ,

Welcome to Microsoft Q&A platform and thanks for posting your query here.

It seems you are trying to perform dynamic mapping in dataflow but it is not working giving empty schema in the output.

The official documentation tutorial which you have shared is suggesting to use cached sink. Could you please confirm if sink2 in your solution is cached sink ?

Since you are getting empty schema in output, kindly check if there are any null values in the source data. If there are null values, try replacing them with default values or removing them.

You can try using map drifted option in data preview tab and manually map the non-drifted columns once.

Hope that helps . Kindly let us know how it goes. Thankyou
AdnanT 0 Reputation points

2024-12-04T11:26:51.68+00:00

Hi,

sink2 is cached sink.

Null values shouldn't affect the output schema of a transformation.

From my OP:

I could map drifted columns, but that transformation maps only renamed columns, and the rest I'd need to manually map which is not ideal when dealing with a lot of non-drifted columns, but ideally, I shouldn't even need to do this.
AnnuKumari-MSFT 33,976 Reputation points Microsoft Employee

2024-12-20T06:44:27.0466667+00:00

AdnanT ,

Did you try to preview data after 'refreshing from source' ? If not, kindly try .
AnnuKumari-MSFT 33,976 Reputation points Microsoft Employee

2024-12-24T06:40:50.8266667+00:00

Hi AdnanT ,

We haven't heard anything from your side on the updates regarding issue resolution. Thankyou

Share via

Select transformation output schema is empty after renaming columns

Your answer