ADF Data Flow Mapping shows 0 rows written to Dataverse sink

Noah Gilbert 26

Running a simple set of pipelines with data flow mappings to Dataverse sinks. Data flow runs successfully, but Rows Written through interface consistently shows 0 rows. I can see the rows were written by going to the table in Dataverse.

phemanth 11,465 Reputation points Microsoft Vendor

2024-09-27T10:50:45.9833333+00:00
@Noah Gilbert

Thanks for reaching out to Microsoft Q&A

you’re encountering a known issue where the “Rows Written” metric in Azure Data Factory (ADF) shows 0, even though the data is successfully written to the Dataverse sink.

This can happen due to certain limitations or configurations in ADF.

Here are a few things you can check:

Known Limitations: There are known limitations with some sink types where the “Rows Written” metric might not be accurately reported.

For example, this issue is known to occur with Delta sinks.

Sink Transformation Settings: Ensure that your sink transformation settings are correctly configured. Sometimes, specific settings or schema projections might need to be overridden.

Error Handling: Check if there are any errors being logged that might affect the row count reporting. You can set the sink error row handling to “Continue on Error” to ensure that errors don’t stop the data flow.

Timeout Settings: If the operation times out while writing data, it might not report the rows correctly. Increasing the timeout value in the sink transformation settings can help.

If you’ve checked these and the issue persists, please share your data flow configuration and sink settings screenshots if possible
Noah Gilbert 26 Reputation points

2024-09-27T19:41:39.58+00:00

Screenshots below:
Note: df ran successfully, 0 Rows written, Stages shows "-" for rows
Not a inline table
No options to configure error handling to continue on error
Noah Gilbert 26 Reputation points

2024-09-27T19:44:21.4733333+00:00

Attempted to comment but comment isnt showing up despite no error showing, so trying again.
phemanth 11,465 Reputation points Microsoft Vendor

2024-10-01T07:37:14.9133333+00:00
@Noah Gilbert

Verify Data in Dataverse: The most critical step is to directly check the Dataverse target table. Even though the data flow reports 0 rows written, the data might have actually been written. This would indicate a problem with the pipeline's reporting, not the Dataverse sink itself.

Examine the Data Flow: Review the data flow mapping carefully. Look for potential issues such as:

Filtering: Does the data flow contain any filters that might be excluding all rows?

Transformation Errors: Are there any transformations that might be causing rows to be dropped or modified in unexpected ways? Check for error messages in the data flow execution logs.

Data Types: Mismatches in data types between the source and the Dataverse target could cause rows to be rejected.

Sink Settings: Ensure the Dataverse sink is correctly configured. Double-check the table name, connection details, and any other sink-specific settings.

Check Data Flow Logs: Investigate the complete logs for the data flow run. They often contain more detailed information about errors or warnings that might not be visible in the summary view.

Test with a Smaller Dataset: Create a smaller, simpler test dataset to run through the data flow. This can help isolate whether the issue is specific to the size or complexity of your original data.

Recreate the Data Flow: If all else fails, consider creating a new data flow from scratch. This can help rule out any subtle corruption or issues within the existing data flow definition.

Dataverse Specific Considerations: While unlikely given the success status, consider:

Permissions: Verify the Power Platform user running the data flow has adequate permissions to write to the Dataverse target table.

Dataverse Limits: Are there any Dataverse limits (e.g., concurrent operations) that may have impacted the write operation?

if the above steps didn't resolve we will further investigate and reach our internal team to resolve the issue
Noah Gilbert 26 Reputation points

2024-10-01T15:42:07.5233333+00:00

The rows are being written, I've checked the actual dataverse table. Agree this is an issue with either ADF or the Dataverse sink (or their integration). Kind of shocking how bad 2 MS products work together in this case, the 0 rows written bug is hardly the only issue I've run into trying to perform a basic data source to Dataverse load using mapping data flows.

Since the rows are being written, I think we can rule out permissions, data type mismatches, etc. I'm assuming the fundamental issue is that the pipeline 'rows written' is derived from the sink providing that information, and the dataverse sink is for some reason unable to transmit the number of rows written back to the pipeline. Unrelated, I have issues with the dataverse sink when it comes to upserts, complaining that 'no insert method was specified' despite me not enabling inserts on the sink (only upserts), and several other problems.
phemanth 11,465 Reputation points Microsoft Vendor

2024-10-07T02:07:49.17+00:00

@Noah Gilbert We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.
phemanth 11,465 Reputation points Microsoft Vendor

2024-10-10T08:17:09.12+00:00

@Noah Gilbert Please add Alter Row transformation before Sink transformation with the below condition upsert If , set to true() :
Noah Gilbert 26 Reputation points

2024-10-10T16:21:14.5133333+00:00

Hi phemanth,

Unfortunately I already have an Upsert if true() Alter Row transformation and no other alter row conditions.

If MS is troubleshooting can I make a simple request. Can someone over there put together a simple data flow mapping with upserts only (alter row upsert if true(), key defined, etc.) and a dataverse sink and verify whether the Rows Written shows up correctly?
phemanth 11,465 Reputation points Microsoft Vendor

2024-10-18T06:38:10.1866667+00:00

@Noah Gilbert I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us? In case if you don't have a support plan please let us know here.
Noah Gilbert 26 Reputation points

2024-10-18T20:27:58.07+00:00
@phemanth thanks for the information.

Just to confirm, you are using a data flow mapping to push to a dataverse sink and it correctly reports the number of rows written?
phemanth 11,465 Reputation points Microsoft Vendor

2024-10-21T08:43:23.5466667+00:00

@Noah Gilbert

Thank you
As it is known issue with Azure Data Factory (ADF) where the "Rows Written" metric may not accurately report the number of rows written to a Dataverse sink, even if the data is successfully written.

so please consider filing a support ticket

1 answer

Pinaki Ghatak 4,840 Reputation points Microsoft Employee

2024-09-27T18:36:57.54+00:00

Hello @Noah Gilbert

It's possible that the metrics for the sink are not being captured correctly. If a sink has zero rows written, it will not show up in metrics. Existence can be verified using the contains function. For example, contains(activity('dataflowActivity').output.runStatus.metrics, 'sink1') will check whether any rows were written to sink1.

To get the number of rows written to a sink named 'sink1' in an activity named 'dataflowActivity', use @activity('dataflowActivity').output.runStatus.metrics.sink1.rowsWritten.

To get the number of rows read from a source named 'source1' that was used in that sink, use @activity('dataflowActivity').output.runStatus.metrics.sink1.sources.source1.rowsRead.

If you are still having issues, it may be helpful to check the debug logs for any errors or warnings related to the sink. You can also try running the data flow from a pipeline and checking the output in the pipeline run details to see if the rows are being written correctly.

I hope that this response has addressed your query and helped you overcome your challenges. If so, please mark this response as Answered. This will not only acknowledge our efforts, but also assist other community members who may be looking for similar solutions.
Please sign in to rate this answer.
Noah Gilbert 26 Reputation points

2024-09-27T19:36:24.07+00:00

Hi Pinaki,

I am running the data flow via a pipeline. Once it completes (and shows success) I go to the Data Flow details under the Monitor tab. This is where it displays: Rows written: 0.

Is there a known issue with using Dataverse as a sink? Can you confirm that writing to the Dataverse as a sink in a data flow mapping correctly shows rows written?

Thanks

Noah Gilbert 26 Reputation points

2024-09-30T19:09:41.3466667+00:00

Appears that Q&A is also buggy, I cannot provide feedback on your answer above, "Submit" button does not work.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

ADF Data Flow Mapping shows 0 rows written to Dataverse sink

1 answer

Your answer