Using Rest API Pagination Not Working as Expected

Sai Kiran 0

I am using the REST API to retrieve data with a pagination rule. It works successfully for the first page, but continuous pagination isn't functioning as expected. I have attached the sample response and pagination rule parsing. Could you please help me figure out how to implement continuous pagination?

User's image

[ { "odata.metadata": "https://test.sharepoint.com/sites/test/_api/$metadata#SP.ListData.QuotationsListItems&$select=Id,DisciplinestoQuoteId", "odata.nextLink": "https://test.sharepoint.com/sites/test/_api/web/lists/GetByTitle(%27Job%20Records%27)/items?%24skiptoken=Paged%3dTRUE%26p_ID%3d882&%24select=Id%2cDisciplinestoQuoteId&%24format=JSON", "value": [] } ]

Chandra Boorla 7,260 Reputation points Microsoft Vendor

2025-01-07T19:01:54.8366667+00:00

Hi @Sai Kiran

Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!

Thank you for sharing the details and the sample response. I understand you are facing an issue with continuous pagination while using a REST API, where only the first page of data is being retrieved successfully, and subsequent pages are not being fetched as expected.

For further information, please refer to the documents listed below. They may provide valuable insights.

https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support

https://www.youtube.com/watch?v=pd0JN_8rC3w

Here are some similar threads which you can refer for some useful insights.

https://learn.microsoft.com/en-us/answers/questions/1062199/rest-api-pagination-help-in-azure-data-factory
https://learn.microsoft.com/en-us/answers/questions/606235/azure-data-factory-pagination-rule-(for-rest-api)
[https://learn.microsoft.com/en-us/answers/questions/1662059/how-to-use-next-link-for-pagination-in-adf-(i-have](https://learn.microsoft.com/en-us/answers/questions/1662059/how-to-use-next-link-for-pagination-in-adf-(i-have)

I hope this information helps. Please do let us know if you have any further queries.

Thank you.
Sai Kiran 0 Reputation points

2025-01-08T07:08:50.68+00:00

@Chandra Boorla
I have Tried the Solution provided and None them worked and I have took the nextLink column name from Mapping using Import Schema and applied the same value in the Pagination but it didn't worked , attaching the screenshots for reference.Please help me with the solution Next Link in ImportSchema in Mapping: Same applied in the Pagination:
Chandra Boorla 7,260 Reputation points Microsoft Vendor

2025-01-08T18:05:19.51+00:00

@Sai Kiran

Apologies for the inconvenience.

Please refer to the below mentioned similar thread that might help you in getting some useful insights:

https://learn.microsoft.com/en-us/answers/questions/732455/pagination-in-azure-adf

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

I hope this information helps.
Sai Kiran 0 Reputation points

2025-01-09T06:05:32.7433333+00:00

@Chandra Boorla According to our sample data what is the absoluteUrl value we need to pass , aleady we have passed the absolute URL in different Format's but it didn't worked out, please help us to how we need to pass the AbsoluteUrl according to our Scenario. You can refer the Screenshots attached in above comments.
Chandra Boorla 7,260 Reputation points Microsoft Vendor

2025-01-10T15:00:25.47+00:00
@Sai Kiran

To implement continuous pagination using the REST API in Azure Data Factory, follow these steps to configure and troubleshoot the pagination rules:

Key Correction - You should not be setting the "AbsoluteUrl" to $.odata.nextLink in the initial configuration. The "AbsoluteUrl" is the base URL for your API call. The odata.nextLink is used dynamically during the pagination process.

Correct Implementation -

Initial API Call Configuration:

AbsoluteUrl - This should be your base API endpoint, without any $skiptoken or other pagination parameters. For example - https://test.sharepoint.com/sites/test/_api/web/lists/GetByTitle('Job Records')/items?$select=Id,DisciplinestoQuoteId&$format=JSON

Pagination Rule - This is where you extract the odata.nextLink. This part is correct: "NextLink": "$.odata.nextLink" (or similar, depending on your tool's syntax)

Pagination Logic (Internal to your tool):

After the first API call, your tool should - Extract the value of odata.nextLink from the response. For the next API call, use the entire extracted odata.nextLink value as the URL. Do not try to construct the URL yourself.

This process repeats - extract odata.nextLink, use it for the next call, until there is no odata.nextLink.

End Condition - The correct "EndCondition" is to check if odata.nextLink is null or empty. This is also correct: "EndCondition": "$.odata.nextLink": "Empty" (or similar, depending on your tool's syntax).

Why your previous approach didn't work - By setting "AbsoluteUrl": "$.odata.nextLink", you were telling your tool to use the next link from the very first response as the base URL for all subsequent requests. This is incorrect because the initial response's odata.nextLink is only valid for the second page.

In summary - The "AbsoluteUrl" is for the initial call. The odata.nextLink is used dynamically during the pagination process to construct the URLs for subsequent calls. Don't try to manage the URL construction yourself; let your tool handle it using the extracted odata.nextLink values.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

I hope this information helps.

Thank you.
Chandra Boorla 7,260 Reputation points Microsoft Vendor

2025-01-13T06:05:40.6533333+00:00

@Sai Kiran

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Chandra Boorla 7,260 Reputation points Microsoft Vendor

2025-01-15T18:07:42.3566667+00:00

@Sai Kiran

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.

Share via

Using Rest API Pagination Not Working as Expected

Your answer