How to send 15000 json data to api 1 by 1 in adf pipeline without using data flow and lookup activity?

Question

I can not use lookup activity as it loops only for 5000 records at a time, so what are the possible ways we can send 15000 JSON data to web API, without using Dataflow and lookup activity?

Answer 1

Hi @gurudas puthal
Since you can't use Lookup or Data Flow, you can try the following approach using ForEach and Web Activity in Azure Data Factory:

Solution:

1. Use Get Metadata Activity: Get the JSON file from Azure Blob Storage.
2. Use a Web Activity to read the file: This will load the JSON content.
3. Use a ForEach Activity: To loop through each record (split the JSON array).
4. Inside ForEach, use Web Activity: To send each record one by one to the API.

Main Points to Think About:

• If the JSON file is large, you may need to paginate or batch the records to avoid performance issues.
• Ensure the API can handle multiple requests efficiently.
• If required, consider using Azure Functions or Logic Apps for more flexibility.

I hope this helps. Let me know if you need more details.