Is it possible to get pandas dataframe from azure feature store client? or can we only get spark dataframe while running the job on serverless spark compute?

Hargurjeet Singh 0

I made DSL pipeline to train ML models in AML studio, which runs on standard compute "Standard DS3_v2"

We also set up Azure Feature Store.

But while getting feature data using below code , I am getting error - "RuntimeError: Fail to get spark session, please check if spark environment is set up." when using to_spark_dataframe() method.

feature_store = FeatureStoreClient(
            credential = credential,
            subscription_id = feature_store_subscription_id,
            resource_group_name = feature_store_resource_group_name,
            name = feature_store_name
        )


feature_set = feature_store.feature_sets.get("Feature_store_name", '1')

df = feature_set.to_spark_dataframe()

Is there a way to not use spark dataframe and spark serverless compute to get data from azure feature store? Can we use standard compute to run this?

santoshkc 10,870 Reputation points Microsoft Vendor

2024-11-29T14:09:10.92+00:00
Hi @Hargurjeet Singh,

Thank you for reaching out to Microsoft Q&A forum!

To retrieve data from Azure Feature Store without using Spark, you can use the get_offline_features() method, which returns a Pandas DataFrame by default and does not require a Spark environment.

Example:

df = feature_set.get_offline_features() print(df.head())

This approach will allow you to use standard compute like "Standard DS3_v2" without needing Spark or serverless Spark compute. I hope this helps! Thank you.

Puneet 0

@santoshkc

im getting

'FeatureSet' object has no attribute 'get_offline_features'

code:

# feature store client
#!pip install azureml-featurestore==1.0.0
from azureml.featurestore import FeatureStoreClient
from azure.ai.ml.identity import AzureMLOnBehalfOfCredential
# from azureml.featurestore.contracts import QueryResultFormat

featurestore = FeatureStoreClient(
    credential=AzureMLOnBehalfOfCredential(),
    subscription_id=featurestore_subscription_id,
    resource_group_name=featurestore_resource_group_name,
    name=featurestore_name,
)
transactions_featureset = featurestore.feature_sets.get('segment_predictive_performance_view', '1')
df = transactions_featureset.get_offline_features()
print(df.head())

santoshkc 10,870 Reputation points Microsoft Vendor

2024-12-02T13:21:26.77+00:00

Hi @Hargurjeet Singh,

Thank you for your follow-up query.

This could be due to using an older version of the SDK or because the method is not part of the API you are working with. To resolve this, I recommend upgrading to the latest version of the AzureML Feature Store SDK by running pip install --upgrade azureml-featurestore. Additionally, try using the to_pandas_dataframe() method instead of get_offline_features(), as this is typically used to retrieve data in Pandas format and does not require a Spark environment.

If this still doesn’t work, I suggest you to raise the support request through the Azure portal.

Thank you.
Hargurjeet Singh 0 Reputation points

2024-12-03T05:33:08.1866667+00:00

hi @santoshkc,

I upgraded the package to its latest version, but the methods (to_pandas_dataframe()) you had mentioned are not available. If possible can you please share any source/documentation where these might be listed because I am not able to find anything related to these.

Thanks
Hargurjeet Singh 0 Reputation points

2024-12-03T06:55:05.33+00:00

hi @santoshkc ,

I upgraded the libraries to the latest version, tried using above methods but I am getting error that the attribute doesn't exist.

Could you please share any source/documentation where the to_pandas_dataframe() method is being used.

Thanks
santoshkc 10,870 Reputation points Microsoft Vendor

2024-12-03T14:29:25.2833333+00:00
Hi @Hargurjeet Singh,

Sorry for the trouble you are facing. I recommend reporting this issue to the Azure ML Services support team. They will be able to investigate the issue further and provide a more targeted solution. You can report the issue by following these steps:

Go to the Azure portal and navigate to your ML Services resource.

Click on the "Support + troubleshooting" tab.

Fill out the required information, including a detailed description of the issue and any steps you have taken to troubleshoot it.

Submit the support request.

The Azure ML Services support team will review your request and provide assistance as soon as possible. Thank you.

1 answer

Azar 23,960 Reputation points MVP

2024-11-29T13:03:00.41+00:00

Hi there Hargurjeet Singh

Thanks for using QandA platform

the error iindicates that the to_spark_dataframe() method is trying to use a Spark session, but it's not available in your environment (since you're using a standard compute instance instead of Spark-based compute).

I guess cannot directly bypass Spark to use Pandas DataFrames with the Azure Feature Store API in the current setup. To use Pandas, you would need to convert the Spark DataFrame to a Pandas DataFrame, but this may only work for smaller datasets. For larger data, Spark compute is a must.

If this helps kindly accept the answer thanks much.
Please sign in to rate this answer.
Azar 23,960 Reputation points MVP

2024-11-29T13:18:07.07+00:00

Hi there, checking if this answer helped, if it did kindly accept the answer thanks much.

Puneet 0 Reputation points

2024-12-02T05:15:17.81+00:00

hi @Azar "Could you please provide a code snippet on how to convert a Spark DataFrame to a pandas DataFrame? If we proceed with this conversion, i think Spark compute still be required because first object result is spark.Additionally, since we do not have access to Spark compute and need to run this on a Standard DS3_v2 machine, how can we achieve this?"

Hargurjeet Singh 0 Reputation points

2024-12-02T07:04:27.0333333+00:00

hi, my feature store data extraction is part of a dsl model training pipeline which is running on standard compute (as mentioned initially).

The code I had shared initially will also be a part of that pipeline.

Is there any other way to get data apart from spark dataframe format?

feature_set.to_spark_dataframe()
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Is it possible to get pandas dataframe from azure feature store client? or can we only get spark dataframe while running the job on serverless spark compute?

1 answer

Your answer