Columns disappear when I extract data from dataverse using synapse

Question

I am trying to extract data from dataverse source, but when I execute the pipeline, the output file does not contain some columns, I thought it was my fetchXML query but when I executed it using external tools the columns are shown, so :

• It is not the FetchXML Query
• The columns exist

What could be the reason?

Answer 1

Check if the Synapse pipeline has any transformations or mappings that are altering the schema. Sometimes during data ingestion, pipelines might drop or rename columns if they are not properly configured.

If the schema inference does not detect all the columns or there is a mismatch between data types, it may omit some columns. You can try manually specifying the schema or using schema mapping in your dataflow.

Data masking or security policies in Dataverse could also prevent some columns from being extracted under certain conditions. Ensure that there are no row-level security (RLS) or column-level security (CLS) policies in place that restrict certain columns from being visible during extraction.