How to use Azure Spark serverless compute in AML Compute?

Puneet 0

We want to use Azure Serverless compute from our Azure python sdk.

cpu_cluster = AmlCompute(
                name=self.cluster_name,
                type="amlcompute",
                size=self.compute_size,  
                min_instances=self.min_instances,  
                max_instances=self.max_instances,  
                idle_time_before_scale_down=self.idle_time_before_scale_down,  
                tier=self.tier
            )

Saideep Anchuri 415 Reputation points Microsoft Vendor

2024-12-02T13:56:20.97+00:00

Hi Puneet

Welcome to Microsoft Q&A Forum, thank you for posting your query here!

When using Azure Serverless compute in Azure Machine Learning (AML) with the Python SDK, there is no need to create a compute cluster as you would with AmlCompute. you can submit your jobs directly to serverless compute.

kindly refer the below document: https://learn.microsoft.com/en-us/azure/machine-learning/apache-spark-azure-ml-concepts?view=azureml-api-2

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-serverless-compute?view=azureml-api-2&tabs=python

Thank You.
Saideep Anchuri 415 Reputation points Microsoft Vendor

2024-12-03T08:02:14.82+00:00

Hi Puneet

Following up to see if the given response was helpful.

Thank You.

Puneet 0

@Saideep Anchuri thanks for giving me your time..

We are still facing challenges in accessing the Pandas DataFrame on our standard compute environment, as the current feature set object requires Spark compute, which is unavailable in our setup.

df = transactions_featureset.to_spark_dataframe()

# Prerequsites: 
# (a) azureml-featurestore sdk is required 
# (b) needs to be run from a spark environment like azureml managed spark notebook
from azureml.featurestore import FeatureStoreClient
from azure.ai.ml.identity import AzureMLOnBehalfOfCredential

featurestore_subscription_id = ''
featurestore_resource_group_name = ''
featurestore_name = ''

# lookup featurestore
featurestore = FeatureStoreClient(
  credential = AzureMLOnBehalfOfCredential(),
  subscription_id = featurestore_subscription_id,
  resource_group_name = featurestore_resource_group_name,
  name = featurestore_name
)

# lookup featureset
transactions_featureset = featurestore.feature_sets.get('segment_predictive_performance_view', '1')

# get access to the feature data
df = transactions_featureset.to_spark_dataframe()

df.show()

Share via

How to use Azure Spark serverless compute in AML Compute?

Your answer