Desktop data migration tool is unable to export large data set

Question

Hi,

I am using Cosmos DB desktop data migration tool to export cosmos container data into JSON file, with 10K records tool works fine, meaning all records were exported but with 50K+ records for another container when tried to export it couldn't export complete data, no error displayed

below is the configuration I'm using, please help why large data is not being exported completely.

{
  "Source": "cosmos-nosql",
  "SourceSettings": {   
    "ConnectionString" :"<ConnectionString>",
    "Database": "DH-PRESENTATION-DB",
    "Container":"DPOMPOPdfEvent",
    "PartitionKeyPath": "/id"
  },
  "Sink": "json",
  "SinkSettings": {    
    "FilePath" : "C:\\Users\\mhs25\\Downloads\\qa-db-migrated-data\\qa2\\DPOMPOPdfEvent.json"
  }
}

Answer 1

Hello Md Khalid Hussain,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that your desktop data migration tool is unable to export large data set.

Beyond general troubleshooting steps and the issue might not be due to timeouts, Request Units (RUs), batch sizes, or outdated software as @NIKHILA NETHIKUNTA stated. Though, these are possible causes, you can also look into system-level constraints such as file system limits, disk space, or memory usage, which might be causing the silent failure. Also, the desktop tool might have any hard limits on JSON exports and try alternative export methods if the tool is unreliable for large datasets.

1. The Cosmos DB Desktop Data Migration Tool runs on your local machine, meaning it relies on system memory (RAM) and disk space. Therefore, start by open Task Manager and monitor memory usage while running the export. If it consumes excessive memory and crashes silently, try running the tool on a machine with higher RAM or splitting the export into smaller batches.
2. Verify Disk Space and File Size Limitations to ensure you have sufficient disk space in C:\Users\mhs25\Downloads\qa-db-migrated-data\qa2\ You can try changing the FilePath to another drive (D:\, E:\) and see if the issue persists.
3. The JSON file might be too large to handle in one go so, instead of exporting directly to JSON, try exporting to CSV or another format and convert it later if needed.
4. You can use the Azure CLI instead, if the desktop tool has limitations, try using the Azure CLI for exporting:

      az cosmosdb sql container export --account-name <your_account> \
      --database-name DH-PRESENTATION-DB \
      --container-name DPOMPOPdfEvent \
      --file-path "C:\Users\mhs25\Downloads\DPOMPOPdfEvent.json"
   

   This method might be more reliable and provide better error handling.
5. Also, if the tool is failing silently, logs might have clues. Locate and check the migration tool logs in the installation folder or use Event Viewer to check for related application errors.
6. Finally, if the migration tool is unable to handle 50K+ records at once, batch processing may be required. Use a query to limit the number of records exported per run:

      "Query": "SELECT * FROM c OFFSET 0 LIMIT 10000"
   

   Run multiple exports with different offset values.

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

---

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.