ADF Copy data activity pagination with continuation token with GET request
I am building ETL pipeline in Microsoft Fabric, ADF service to load data from API, but I am having difficulties implementing pagination rules with continuation token. I know that, according to Azure documentation, it should be possible to use this approach - next request’s header = property value in current response body.
The structure I would like to achieve:
make a GET request to https://test.net/abc/tableName?size=1000 using pipeline Web activity, and get a continuationToken from response body as an output
store it in Set Variable activity
- use Copy activity to make a GET request to https://test.net/abc/tableName?size=1000&seekOperation=Next with header = continuationToken from response body of Web activity (stored in variable).
- Make another GET requests to retrieve all data until continuationToken is non-existent in response body, each time adding token to header.
I am trying pagination rules, so I set something like this, but it is not working:
"paginationRules": {
"supportRFC5988": "true",
"Headers.ContinuationToken": {
"value": "$.continuationToken",
"type": "Expression"
},
"EndCondition:body.continuationToken": "NonExist"
}
Error I am getting is something like this:
Failure happened on 'Source' side. ErrorCode=RestInvalidPaginationRule,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Invalid PaginationRule, RuleKey='body.continuationToken', RuleValue='NonExist',Source=Microsoft.DataTransfer.ClientLibrary,'
Also, I don't know how to make it do GET request first time in copy activity, using value stored in variable, and then retrieve it from response body of this request iteratively.
In additional headers I added Token: @variables('continuationToken'), but how to make it work iteratively?
I would be really grateful if someone can help me with constructing this logic. As for now, it looks like this: