Azure ML | Support for ADLS Gen2 Datastore with SAS Token Authentication

Pål Hannus 0

Hello,

I’m currently working on an application where we’re connecting various data sources—such as file shares and ADLS Gen2—to Azure Machine Learning. While we can create file-share datastores with SAS token authentication, I noticed that this option isn’t available for ADLS Gen2 datastores in Azure ML. Interestingly, this type of SAS-based authentication seems to be supported in Azure Databricks. Could you clarify if there’s a particular reason for this limitation in Azure ML? Enabling SAS token support for ADLS Gen2 would simplify our integration significantly.

Our use case involves utilizing ADLS Gen2 for blob storage, but we don’t have direct access to the ADLS resource via service principals. Instead, the data lake is primarily accessed through an API, and we are only allowed to generate SAS tokens. Would it be possible to create custom datastores in Azure ML that communicate with the data lake via the API?

Additionally, is it feasible to configure datastores using non-root-level SAS tokens? For instance, if a SAS token provides access only to a specific directory within a blob (e.g., root/some/path/), can we use it to create a datastore scoped to that directory?

Thank you for your assistance!

santoshkc 9,905 Reputation points Microsoft Vendor

2024-11-15T13:27:52.4633333+00:00

Hi @Pål Hannus,

Thank you for reaching out to Microsoft Q&A forum!

Azure Machine Learning does not currently support SAS token authentication for ADLS Gen2 datastores, likely due to its preference for service principal or managed identity approaches for enhanced security and management. While Azure Databricks supports SAS tokens, Azure ML's design prioritizes different authentication models. To work around this, you can create custom datastores that interact with the data lake via an API. Additionally, while the feasibility of using non-root-level SAS tokens is not explicitly documented, it should be possible to scope a datastore to a specific directory if the SAS token permissions align with the required configuration. For a more seamless solution, you can raise a feature request via the Azure Feedback Forum to advocate for SAS token support in ADLS Gen2 datastores.

I hope you understand. Thank you.
santoshkc 9,905 Reputation points Microsoft Vendor

2024-11-18T14:38:32.96+00:00

Hi @Pål Hannus,

Following up to see if the given response was helpful. Thank you.
santoshkc 9,905 Reputation points Microsoft Vendor

2024-11-19T13:25:07.4733333+00:00

Hi @Pål Hannus,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Thank you.
Pål Hannus 0 Reputation points

2024-11-19T13:34:53.9033333+00:00
Hello @santoshkc ,

Thank you for your response and clarification regarding the limitations of SAS token authentication for ADLS Gen2 in Azure Machine Learning.

I am a bit curious about creating custom datastores that interact with ADLS Gen2 via the API. Could you provide further details on the process of setting up a custom datastore in Azure ML? I have not been able to find documentation on this.

All help would be appreciated. I am specifically interested in:

The steps to configure a custom datastore with SAS token-based authentication.

Any example code or references to implement such a solution.

How to integrate this with Azure ML pipelines for data access inference jobs.

Thanks again for your help!

1 answer

Sina Salam 12,816 Reputation points

2024-11-15T11:59:00.43+00:00
Hello Pål Hannus,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you are in need to create a data store that support Azure ML | Support for ADLS Gen2 Datastore with SAS Token Authentication.

This might be complex than expected, but I will give you best solution and links to read further for more steps and instructions, also utilize the additional resources by the right side of this page.

Therefore,

Currently, Azure Machine Learning does not support SAS token authentication for ADLS Gen2 datastores. - https://knowledge.informatica.com/s/article/FAQ-What-is-Shared-Access-Signature-Token-in-ADLS-Gen2 This limitation is likely due to security and management considerations, as SAS tokens can be less secure compared to other authentication methods like service principals or managed identities. Azure Databricks, on the other hand, does support SAS tokens for ADLS Gen2, which might be due to different security models and use cases. - https://github.com/easonlai/sas_access_to_adls_databricks

While Azure ML does not natively support SAS tokens for ADLS Gen2, you can potentially create a custom solution. One approach could be to use Azure Functions or Azure Logic Apps to act as an intermediary, generating SAS tokens and accessing the data via the API. This would involve setting up a service that handles the authentication and data access, then connecting this service to your Azure ML workspace.

Yes, it is feasible to configure datastores using non-root-level SAS tokens. If your SAS token provides access to a specific directory within a blob, you can scope your datastore to that directory. When creating the datastore, you would specify the path within the blob storage that the SAS token has access to. This allows you to limit the scope of access to just the necessary data. - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-access-data-interactive?view=azureml-api-2

This links below cover all detailed steps on setting up a custom solution or any other specific aspect of your integration:

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-datastore?view=azureml-api-2

https://learn.microsoft.com/en-us/samples/azure/azure-quickstart-templates/machine-learning-datastore-create-adls-gen2

https://reintech.io/blog/developing-with-azure-data-lake-storage

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.
Please sign in to rate this answer.
Pål Hannus 0 Reputation points

2024-11-19T14:33:44.1033333+00:00

Hello @Sina Salam ,

Thank you for your detailed response and for clarifying the current limitations in Azure ML regarding SAS token support for ADLS Gen2 datastores.

I’m particularly interested in the custom solution you mentioned. Specifically, you mentioned using Azure Functions or Logic Apps as intermediaries to handle authentication and data access. Could you elaborate on how this integration would work with Azure ML? For instance:

How would the datastore in Azure ML connect to such an intermediary service?

Azure ML datastores typically have fixed formats (unless I am mistaken) that map directly to the underlying storage. Could you provide an example of how a datastore using this custom intermediary solution might be configured in Azure ML?

Or would this be something separate from a datastore?

Thank you again for your time and guidance!

Sina Salam 12,816 Reputation points

2024-11-19T16:12:52.04+00:00

Hello Pål Hannus,

You're welcome and thank you for your feedback. It's good to know that the answer was helpful.

Your new questions would have been a new thread to reduce lengthy contents.

However, to address your questions regarding the custom solution using Azure Functions or Logic Apps as intermediaries for enabling SAS token authentication with ADLS Gen2 in Azure ML.

NOTE: By practice, the below information are to:

Protect your Azure Function using Azure Active Directory (AAD) authentication to prevent unauthorized access.

Integrate these steps within your ML pipelines for efficient and secure data handling.

Your Question 1:

You can set up an Azure Function or Logic App to generate and return SAS tokens. Azure ML can then call this service to obtain a SAS token and use it to access ADLS Gen2 storage directly within your scripts. By doing the followings:

Create an Azure Function** to generate a SAS token: This function will authenticate with ADLS Gen2 and generate a SAS token for the specified path.

Call the Azure Function from your Azure ML script using HTTP, the sample Azure Function code will be similar to:
import os import datetime from azure.storage.filedatalake import DataLakeServiceClient import azure.functions as func def generate_sas_token(): account_name = os.getenv("STORAGE_ACCOUNT_NAME") account_key = os.getenv("STORAGE_ACCOUNT_KEY") file_system_name = "my-file-system" directory_path = "my-directory" # Initialize the DataLakeServiceClient service_client = DataLakeServiceClient( account_url=f"https://{account_name}.dfs.core.windows.net", credential={"account_name": account_name, "account_key": account_key} ) # Generate a SAS token valid for 1 hour sas_token = service_client.generate_file_system_sas( file_system_name=file_system_name, permission="r", expiry=datetime.datetime.utcnow() + datetime.timedelta(hours=1) ) return sas_token def main(req: func.HttpRequest) -> func.HttpResponse: try: sas_token = generate_sas_token() return func.HttpResponse(sas_token, status_code=200) except Exception as e: return func.HttpResponse(f"Error: {str(e)}", status_code=500)

Check out this link for more review - https://learn.microsoft.com/en-us/azure/azure-functions/functions-reference-python

Your Question 2:

Azure ML datastores can't directly use an intermediary like this. Instead, bypass the datastore and access data dynamically in your ML scripts. The Python Code sample to Access Data you look like the below:

import requests from azure.storage.filedatalake import DataLakeServiceClient def get_sas_token(): # Call the Azure Function to get the SAS token url = "https://<your-function-app>.azurewebsites.net/api/GenerateSASToken" response = requests.get(url) response.raise_for_status() return response.text def read_data_with_sas(sas_token, file_path): service_client = DataLakeServiceClient(account_url="https://<storage_account_name>.dfs.core.windows.net", credential=sas_token) file_system_client = service_client.get_file_system_client(file_system="my-file-system") file_client = file_system_client.get_file_client(file_path) return file_client.download_file().readall() sas_token = get_sas_token() data = read_data_with_sas(sas_token, "my-directory/my-file.csv") print(data)

You might like to read more in this link - https://learn.microsoft.com/en-us/python/api/overview/azure/storage-datalake-readme

Your Question 3:

Yes, this approach works separately from a datastore. Instead of relying on Azure ML's datastore abstraction, you handle the connection and data transfer manually within your script or pipeline. The benefit of this method is flexibility—enabling you to leverage SAS tokens dynamically, even though they are not natively supported in datastores. So, if you prefer to use Azure ML's Dataset class, you can directly pass the SAS token in your script as part of the URL like the following:

from azureml.core import Dataset sas_url = "https://<storage_account_name>.dfs.core.windows.net/<container>/<path>?<sas_token>" dataset = Dataset.Tabular.from_delimited_files(path=sas_url) df = dataset.to_pandas_dataframe() print(df.head())

This approach allows seamless integration within Azure ML without needing a datastore. You can also read more on https://learn.microsoft.com/en-us/azure/machine-learning/how-to-create-register-datasets

Thank you for the time taken to review these answers, kindly let's support the community by close up the thread here by upvoting and accept it as an answer.

Thank you.

Pål Hannus 0 Reputation points

2024-11-20T08:18:54.22+00:00

Hi Sina Salam,

Thank you for the detailed response, it was very helpful! I will do some investigation but I think it may be exactly what I was looking for.

Is there any reason why such a solution would not work on Azure ML Compute Clusters and video files (roughly 2gb)? I guess it would also work with custom APIs that serve files, correct?

Thanks again.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Azure ML | Support for ADLS Gen2 Datastore with SAS Token Authentication

1 answer

Your answer