Copy Data Activity with Upsert Operation to MongoDB Not Updating Records

Ramakrishna Vaitla 0 Reputation points
2025-03-06T20:28:21.6766667+00:00

I am trying to use Azure Data Factory's Copy Data Activity to copy data from a CSV file into a MongoDB collection using the upsert operation. However, I am facing an issue where the pipeline run indicates that 4 rows were written, but no records are being updated in MongoDB.

Here is my scenario:

Input CSV:

_id,name,gender,age
67c88f641b4dd8cef2342e9d,Sam,M,25
67c88f641b4dd8cef2342e9d,Tam,F,23
,Dam,M,45
,Cam,F,54

Observations:

  1. When I map _id as required for upsert, I expect the record with _id = 67c88f641b4dd8cef2342e9d to be updated (based on the second row of the CSV). However, this does not happen.
  2. If I leave _id unmapped, it creates new records for each row in the CSV, which is essentially an insert operation and not an upsert.
  3. I enabled data consistency verification and logging, but I do not see any useful logs or errors indicating what went wrong.

Sink Mapping Configuration:

Below is the mapping configuration I used for the sink:

{
"mappings": [
    {
        "source": {
            "name": "_id",
            "type": "String"
        },
        "sink": {
            "path": "$['_id']"
        }
    },
    {
        "source": {
            "name": "Name",
            "type": "String"
        },
        "sink": {
            "path": "$['Name']"
        }
    },
    {
        "source": {
            "name": "gender",
            "type": "String"
        },
        "sink": {
            "path": "$['gender']"
        }
    },
    {
        "source": {
            "name": "age",
            "type": "int16"
        },
        "sink": {
            "path": "$['age']"
        }
    }
]
}

Questions:

  1. Why is the upsert operation not updating the existing record in MongoDB?
  2. Are there any additional configurations required to make the upsert work correctly in the Copy Data Activity?
  3. How can I debug this further since enabling logging and data consistency verification did not provide meaningful insights?

Any guidance or suggestions would be greatly appreciated!

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,323 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.