Issues Handling Paginated Data in My Azure Function for a Test API

Question

I’m building a small project that helps students learn to work with APIs by using real-world examples. I decided to use Free Test API as it provides dummy product data that’s easy to work with.

Here’s what I’m trying to do:

1. Fetch paginated product data from this API: https://freetestapi.com/api/v1/products?page=1&limit=10
2. Process and display the data in a web app hosted on Azure Static Web Apps.
3. Use an Azure Function to manage the backend logic, such as fetching and caching API data.

It works fine when I fetch small datasets (e.g., 10 products at a time). However, when I increase the pagination limit or fetch multiple pages in quick succession, I run into two challenges:

1. The response times become inconsistent, sometimes taking significantly longer.
2. My Azure Function logs show spikes in execution time, even when I cache results to minimize API calls.

What I’ve Already Tried:

• Reduced the pagination limit to process smaller chunks of data at a time.
• Cached API responses in-memory using the Azure Function's state.
• Added retry logic in case the API throttles requests.

However, while the API itself supports pagination, I’m unsure how to optimize my Azure Function to handle large data sets more efficiently. My current approach feels inefficient, and I suspect I might be overloading the function or missing an opportunity to use Azure's tools better.

Steps to Reproduce:

1. Set up an Azure Function that fetches paginated data from: https://freetestapi.com/api/v1/products?page={page}&limit={limit}
2. Fetch data for 10 pages (limit = 10).
3. Log the execution time and observe spikes when handling more pages.

What I Need Help With:

1. Are there better ways to handle paginated data in Azure Functions without running into performance issues?
2. Would Durable Functions be a better fit for handling larger tasks like these?
3. Should I use a queue to process requests asynchronously instead of fetching all data in a single run?

I’d appreciate any advice or guidance on how to make this process smoother!

Answer 1

Hello @Kishore

Here are some suggestions that might help:

1. Consider using Durable Functions: Durable Functions are designed to handle long-running, stateful workflows. They can help you manage the state of your function and handle retries and failures more gracefully. You might find that using Durable Functions makes it easier to handle large data sets more efficiently.
2. Use a queue to process requests asynchronously: If you're fetching all data in a single run, you might be overloading your function. Consider using a queue to process requests asynchronously. This can help you manage the load on your function and ensure that it doesn't become overwhelmed.
3. Optimize your code: Make sure that your code is optimized for performance. For example, you might be able to use parallel processing to fetch data more quickly. You might also be able to optimize your caching strategy to minimize the number of API calls you need to make.
4. Monitor your function: Make sure that you're monitoring your function's performance and looking for ways to optimize it. Use tools like Application Insights to track performance metrics and identify areas for improvement.

---

I hope these suggestions helps. If so, please mark this response as Answered. This will not only acknowledge our efforts, but also assist other community members who may be looking for similar solutions.