API Token expires in mid of Pipeline run

Kunal Kumar Sinha 171

Hi Team,

My API token expires within 90 min of it's creation and I use this to copy data from API. But my pipeline runs for more than 90 mins and as a result the token expires and pipeline fails. Can you please let me know how it can be resolved ?

Thanks,
Kunal

HimanshuSinha-msft 19,476 Reputation points Microsoft Employee

2020-12-09T21:38:42.033+00:00

Hello @Kunal Kumar Sinha ,
Thanks for the ask and using the forum .

Is this a publically exposed API , if yes please do let us know and we can give iot a try . I do not see any setting on the actvity where we can refresh the token but still I can check with the team and let you know .

Thanks
Himanshu
Kunal Kumar Sinha 171 Reputation points

2020-12-10T06:30:27.377+00:00

Hi @HimanshuSinha-msft ,

It's not a public exposed API, can you please suggest some work around to avoid such situation? I use the token to fetch data by passing one value at a time in request body under for each activity and there are around 2k such values that I'm doing lookup and passing in foreach copy activity but the pipeline fails as the token expires in 90 mins.

Thanks,
Kunal
HimanshuSinha-msft 19,476 Reputation points Microsoft Employee

2020-12-10T21:29:02.233+00:00

Hello @Kunal Kumar Sinha ,

Thanks for sharing the pipeline , I think you can work around this , by creating more then 1 lookup and FE loop and running all of them in parallel . The lookup should look for different set of data eg LU1 looks for id - 1-500 , LU2 - 501-1000 and so on .

Let me know if this helps .

Thanks
Himanshu .
Kunal Kumar Sinha 171 Reputation points

2020-12-11T05:05:47.733+00:00

Hi @HimanshuSinha-msft ,

I'll try that and update the results, Is there a way in which i can generate a token once it expires and use that newly generated token in copy activity ?

Thanks,
Kunal
HimanshuSinha-msft 19,476 Reputation points Microsoft Employee

2020-12-17T22:48:06.83+00:00

Hello ,
We have not heard back from you on this and was just following up .
Incase if you have resolution , request you to share the same here , so that other community members can benefit from that .
Thanks
Himanshu
Kunal Kumar Sinha 171 Reputation points

2020-12-18T05:32:05.03+00:00

Hi @HimanshuSinha-msft ,

Is it possible to generate the token via ADF and store it in key vault, and once the token expires in 90 mins, I shall have one wait activity which will trigger after 90 min and fetches new token and over writes the existing token in vault ?

I tried this option but I'm not able to find a work around on how to write in key vault, can you help me on this ?

Thanks,
Kunal
Kunal Kumar Sinha 171 Reputation points

2020-12-21T07:40:11.99+00:00

Hi @HimanshuSinha-msft ,

Can you please provide your input on this ?

Thanks,
Kunal
Kunal Kumar Sinha 171 Reputation points

2020-12-22T05:39:36.527+00:00

Hi @HimanshuSinha-msft ,

Can you please help & provide your input on this ?

Thanks,
Kunal
HimanshuSinha-msft 19,476 Reputation points Microsoft Employee

2020-12-23T19:54:14.29+00:00

Hello @Kunal Kumar Sinha ,

Apoloziges for not able to reply sooner .
In ADF we do have a connector to update KV , what you will have to do is use the REST API for KV . I think you will have to use this https://learn.microsoft.com/en-us/rest/api/keyvault/setsecret/setsecret from inside a web activity .

Hope this helps .

Thanks
Himanshu
Saurabh Sharma 23,826 Reputation points Microsoft Employee

2020-12-29T18:57:20.373+00:00

@Kunal Kumar Sinha Please let me know if you find above reply useful or you have any other questions.
Kunal Kumar Sinha 171 Reputation points

2020-12-30T08:19:39.473+00:00

Hi @Saurabh Sharma ,

I did try out that option but it didn't reduce the pipeline run time, I need to pass 2043 values to for each activity and for each value the copy activity makes an API call to fetch the data. The value passed in for each copy activity is passed in the body part of copy activity. I'm trying to figure out a way in which multiple values from lookup activity can be passed at once in the body of copy activity inside the foreach activity, can you please help me out to find a way around this.

Thanks,
Kunal
Kunal Kumar Sinha 171 Reputation points

2021-01-06T08:30:28.4+00:00

Hi @Saurabh Sharma @HimanshuSinha-msft ,

I tried to pass the values as an array but it's not working. The copy activity is failing stating The property 'CompanyIdentifiers' cannot contain text. When i manually tried to pass one value it got succeeded. I've attached both request body along with pipeline code.
please help me on this. Thanks

Pipeline name - PL_Api_Refinitiv_Variable_Test_copy1
ADF Name - acemwpdatradf01
Region - North Europe

53963-pl-api-refinitiv-variable-test-copy1.txt
53965-successrequestbody.txt
53955-failedrequestbody.txt
Saurabh Sharma 23,826 Reputation points Microsoft Employee

2021-01-13T18:11:11.313+00:00

@Kunal Kumar Sinha I suggest if you have a support plan you may file a support ticket, else could you please send an email
to AzCommunity[at]microsoft[dot]com with the below details,
so that we can create a one-time-free support ticket for you to work closely on this matter.
Thread URL:
Subscription ID:
Subject:Attention :Saurabh
Please let me know once you have done the same.

Share via

API Token expires in mid of Pipeline run

Your answer