I am experiencing an issue where the file-sink of the Copy Data command ( SnowflakeExportCopyCommand ) is producing incomplete files when uploading data from Snowflake to Azure Blob Storage in our Azure Data Factory pipeline. Observations: The number of rows read from Snowflake matches the number of rows written to Azure Blob Storage, as indicated in the copy details. However, when multiple files are generated using the COPY command, the resulting Parquet files in Azure storage have incorrect sizes and row counts. I have explicitly set the following Snowflake copy options: SINGLE=TRUE and MAX_FILE_SIZE= 900000000 but the issue persists. Has anyone encountered similar behavior, and are there any known solutions or workarounds? Would appreciate any insights into possible causes or additional configurations that might resolve this.

Incomplete Files from Copy Data Command in Azure Data Factory pipeline when uploading data from Snowflake

Susan Rakers 0

I am experiencing an issue where the file-sink of the Copy Data command (SnowflakeExportCopyCommand) is producing incomplete files when uploading data from Snowflake to Azure Blob Storage in our Azure Data Factory pipeline.

Observations:

The number of rows read from Snowflake matches the number of rows written to Azure Blob Storage, as indicated in the copy details.
However, when multiple files are generated using the COPY command, the resulting Parquet files in Azure storage have incorrect sizes and row counts.
I have explicitly set the following Snowflake copy options: SINGLE=TRUE and MAX_FILE_SIZE=900000000 but the issue persists.

Has anyone encountered similar behavior, and are there any known solutions or workarounds?

Would appreciate any insights into possible causes or additional configurations that might resolve this.

Share via

Incomplete Files from Copy Data Command in Azure Data Factory pipeline when uploading data from Snowflake

Your answer