Copy Data Activity with Upsert Operation to MongoDB Not Updating Records
Ramakrishna Vaitla
0
Reputation points
I am trying to use Azure Data Factory's Copy Data Activity to copy data from a CSV file into a MongoDB collection using the upsert operation. However, I am facing an issue where the pipeline run indicates that 4 rows were written, but no records are being updated in MongoDB.
Here is my scenario:
Input CSV:
_id,name,gender,age
67c88f641b4dd8cef2342e9d,Sam,M,25
67c88f641b4dd8cef2342e9d,Tam,F,23
,Dam,M,45
,Cam,F,54
Observations:
- When I map
_id
as required for upsert, I expect the record with_id = 67c88f641b4dd8cef2342e9d
to be updated (based on the second row of the CSV). However, this does not happen. - If I leave
_id
unmapped, it creates new records for each row in the CSV, which is essentially an insert operation and not an upsert. - I enabled data consistency verification and logging, but I do not see any useful logs or errors indicating what went wrong.
Sink Mapping Configuration:
Below is the mapping configuration I used for the sink:
{
"mappings": [
{
"source": {
"name": "_id",
"type": "String"
},
"sink": {
"path": "$['_id']"
}
},
{
"source": {
"name": "Name",
"type": "String"
},
"sink": {
"path": "$['Name']"
}
},
{
"source": {
"name": "gender",
"type": "String"
},
"sink": {
"path": "$['gender']"
}
},
{
"source": {
"name": "age",
"type": "int16"
},
"sink": {
"path": "$['age']"
}
}
]
}
Questions:
- Why is the upsert operation not updating the existing record in MongoDB?
- Are there any additional configurations required to make the upsert work correctly in the Copy Data Activity?
- How can I debug this further since enabling logging and data consistency verification did not provide meaningful insights?
Any guidance or suggestions would be greatly appreciated!
Sign in to answer