How to create a link service in Azure Data Factory for Rest API which has OAuth grant type as password type of authentication

Kukade, Atharva 0

We need to call Rest API which has OAuth grant type as password type of authentication in a loop to store response in CSV. I know we could use Copy Activity but we are unable to create link service with mentioned authentication.

We tried to do simple web call in a loop and append response in variable and then copy variable to file. However in this case variable exceeds size.

Any help is appreciated.

phemanth 12,575 Reputation points Microsoft Vendor

2024-12-20T19:22:29.9166667+00:00

@Kukade, Atharva We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Kukade, Atharva 0 Reputation points

2024-12-23T07:03:40.21+00:00

Facing very weird issue.
When I use token from web1 activity to another web activity, the call works fine. But when I pass it to copy activity it fails with below error,

Failure happened on 'Source' side. ErrorCode=RestResourceReadFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Fail to read from REST resource.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.Http.HttpRequestException,Message=An error occurred while sending the request.,Source=mscorlib,''Type=System.Net.WebException,Message=The underlying connection was closed: An unexpected error occurred on a send.,Source=System,''Type=System.IO.IOException,Message=Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.,Source=System,''Type=System.Net.Sockets.SocketException,Message=An existing connection was forcibly closed by the remote host,Source=System,'

Could it be firewall issue?
Kukade, Atharva 0 Reputation points

2024-12-23T07:06:32.37+00:00

Facing very weird issue. Call works fine when I use token from web1 activity to web2 activity. But gives me below error when trying to work in copyActivity Failure happened on 'Source' side. ErrorCode=RestResourceReadFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Fail to read from REST resource.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.Http.HttpRequestException,Message=An error occurred while sending the request.,Source=mscorlib,''Type=System.Net.WebException,Message=The underlying connection was closed: An unexpected error occurred on a send.,Source=System,''Type=System.IO.IOException,Message=Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.,Source=System,''Type=System.Net.Sockets.SocketException,Message=An existing connection was forcibly closed by the remote host,Source=System,'
phemanth 12,575 Reputation points Microsoft Vendor

2024-12-23T18:16:20.11+00:00
@Kukade, Atharva

Thanks for details

The error message you're seeing suggests that the connection is being forcibly closed by the remote host, which could indeed be related to network issues such as a firewall blocking the connection.

Steps to troubleshoot and potentially resolve this issue:

Ensure that your firewall settings allow outbound connections to the REST API endpoint. You might need to whitelist the API's IP address or domain.

Double-check that the token is being passed correctly in the Copy Activity. Ensure that the token is not expired and is formatted correctly in the Authorization header.

Sometimes, increasing the timeout settings for the Copy Activity can help if the API response is taking longer than expected.

Verify that there are no network issues or restrictions on the server hosting the REST API. This includes checking for any rate limiting or IP blocking.

Use tools like Postman to manually test the API calls with the token to ensure that the issue is not with the API itself.

Implement retry logic in your pipeline to handle transient network issues. This can help in cases where the connection is intermittently failing.

I hope the above steps will resolve the issue, please do let us know if issue persists. Thank you

1 answer

phemanth 12,575 Reputation points Microsoft Vendor

2024-12-20T17:26:41.1433333+00:00
@Kukade, Atharva

Thanks for the question and using MS Q&A platform.

To create a linked service in Azure Data Factory for a REST API with OAuth password grant type authentication, you can follow these steps:

Create a Linked Service:

Go to your Azure Data Factory instance.

Navigate to the "Manage" tab and select "Linked Services".

Click on "New" and choose "REST" as the type of linked service.

Configure the linked service with the necessary details, including the base URL of your REST API.

Configure OAuth Authentication:

In the linked service configuration, set the authenticationType to Basic.

Provide the username and password for the OAuth password grant type.

Use Web Activity for Token Retrieval:

Since Azure Data Factory does not natively support OAuth password grant type directly in the linked service, you can use a Web Activity to first retrieve the OAuth token.

Create a Web Activity to call the token endpoint of your OAuth server. This activity should include the necessary parameters (username, password, client_id, client_secret, etc.) to obtain the token.

Pass Token to Copy Activity:

Use the output of the Web Activity to pass the token to subsequent activities.

In your Copy Activity, set the Authorization header to Bearer <token> using the token obtained from the Web Activity.

Loop and Store Responses:

Use a ForEach activity to loop through your API calls.

Within the loop, use the Copy Activity to store the responses in a CSV file. Ensure you handle pagination or multiple API calls as needed.

I hope the above steps will resolve the issue, please do let us know if issue persists. Thank you
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to create a link service in Azure Data Factory for Rest API which has OAuth grant type as password type of authentication

1 answer

Your answer