copy data gives error NullReferenceException: Object reference not set to an instance of an object

MarkDG 0

I'm copying from Azure Blob Storage .csv files into Snowflake, and receiving the error ErrorCode=SessionLogFailToWriteLog,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Write log is failed with error message: 'System.NullReferenceException: Object reference not set to an instance of an object.

I know this is trying to write a null where a value is expected. However as it is logging the copy operation of a file I'd expect there to be a log. It happens on different files each day and only started last week on an operation that's been running fine for months.

However the blob is present with data in it.

I can see that there is no log file for the run ID in the copy activity log location. I also note the output run table that records skipped rows is missing an entry. Therefore I can't see any reason why it has failed to write to Snowflake and log that it did. Unless there's a data issue throwing the copy activity off, something I can look for in the file itself, an error in ADF or an error transporting to blob, I don't know.

Any ideas? The origin is Oracle OBIA (which could be the issue that cascades through ADF. There are 51 files written through this run every day.

Many thanks

Mark

Chandra Boorla 7,490 Reputation points Microsoft Vendor

2025-02-04T02:10:23.1266667+00:00
@MarkDG

Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!

The error you are encountering, "NullReferenceException: Object reference not set to an instance of an object", is a common issue in programming that usually indicates that your code is trying to use an object that hasn't been initialized. In the context of your Azure Data Factory pipeline copying data from Azure Blob Storage to Snowflake, several areas could be contributing to this error.

Here are some steps and considerations to help you diagnose and potentially resolve the issue:

Check Data and Schema - Ensure that the data in your .csv files is consistent with the schema expected by Snowflake. If there are changes in the source data that don't match the expected schema, it might cause issues during the copy operation. Check for any null values in columns that are not nullable in the target Snowflake table.

Review Pipeline Configuration - Double-check the configuration settings in your ADF pipeline. Make sure all required properties are set and that there are no misconfigurations in your dataset or linked service definitions. Ensure that the dataset schema in ADF matches the schema of the Snowflake table.

Inspect Logging and Monitoring - Investigate the ADF activity logs to see if there are more detailed error messages that could provide further insight. Check if there have been any changes to the logging configuration that might affect how logs are written or stored.

Investigate Recent Changes - Since this issue started recently, consider any changes that might have been made to the pipeline, source data, or Snowflake configuration. If Oracle OBIA is involved, ensure there are no upstream changes affecting data output.

Test with Smaller Data Set - Try running the pipeline with a smaller subset of data to see if you can isolate the issue to specific files or data content. This can help identify if there’s a particular file or set of data that's causing the failure.

Examine Error Handling and Retry Policies - Review any error handling or retry policies in place for the ADF pipeline. Ensure that they are appropriately configured to handle transient errors and log failures correctly.

By systematically going through these steps, you should be able to identify the root cause of the NullReferenceException and address it appropriately.

Please refer to the links provided below, as they might offer some insights that could help you address your question.

NullReferenceException Class

Fault tolerance of copy activity in Azure Data Factory and Synapse Analytics pipelines

Session log in a Copy activity

Data consistency verification in copy activity

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.
MarkDG 0 Reputation points

2025-02-04T17:38:18.2133333+00:00

Hi, thanks Chandra. I think here the only relevant area is error handling and retry policies. Although the database (OBIA) is flaky at best, the error is randomly on different tables. That could still be OBIA. This morning ADF failed on a different table/csv file in blob.

However, later in the day, ADF ran fine with no changes other than clicking 'rerun' on the pipeline. So this is either an ADF bug or a timing issue.

The initial job that wrote to blob had finished over 10 minutes before ADF ran. As it works some times and not others, with different tables affected, yet files in the blob appear correct when re-run, it's either a bug or potentially a connection that isn't being dropped.

I can see that on the days it fails it runs for an extra 40 minutes (nearly 50 minutes in total) for an 8-9 minute job, so it's clearly timing out. There's no throughput on the errored run ID and no error log. But it has found the file amongst the 50 others and tried to start up, but failed.

Share via

copy data gives error NullReferenceException: Object reference not set to an instance of an object

Your answer