Azure Data Factory pipeline fails randomly with error: "Specified argument was out of the range of valid values"

Question

Hey,

Currently I'm struggling with ADF pipeline that fails pretty randomly. It loads multiple tables from sql database to data lake in parquet format. There are 3 retries for copy data activity, sometimes it succeeds on one of the retries, sometimes not. And sometimes it just works with no errors. These are the two errors I get:

What these errors refer to? Is there a way to fix it? Is it due to the data I want to store in parquet format?
For this I'm using Self Hosted Integration Runtime as a source of computing power. It matches all the requirements for SH-IR.

Answer 1

Hi Sylwester Jedrys ,

Welcome to Microsoft Q&A platform and thanks for posting your query here.

I understand that you are facing this error "Specified argument was out of the range of valid values" while trying to load data from SQL to ADLS (Parquet file)

This might be occurring due to mismatch between the source data and expected sink data type. Mostly arises on the column having datetime format. Kindly try using FormatDateTime() function to convert the datetime column into certain desired format before loading it in sink.

Additionally, For the error "IndexOutofRange" , Kindly check if there are any restrictions on the maximum file size for the target Azure Blob Storage container.

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou