Azure Machine Learning serverless cores occupied but not used

Question

Hi,

I have been testing the usage of spark serverless with azure ml notebooks for a couple of days now, and I just noticed that my workspace has 300 dedicated cores, of which 128 are being used by serverless:

The thing is, this number didn't decrease since yesterday, even though I haven't been running any notebooks/jobs on top of spark serverless; but if I try to do so, the number would increase depending on the cores the notebook uses then it goes down after execution is done; so it just seems that something is occupying these cores but I can't find anything on the workspace side:

Is there a way to deallocate these cores so that they can be used by other notebooks/jobs?

Thanks.

Answer 1

Hi @Abdelkhalek Hamdi,

It looks like your Azure Machine Learning Spark Serverless instance is holding onto 128 cores even when there are no active jobs. This could be due to idle Spark sessions that haven’t been properly deallocated. Check for any active Spark sessions in your workspace by running a CLI command to list them.

If sessions are still running, try stopping them manually or deleting the compute instance associated with Spark Serverless. You can also restart the Azure ML workspace compute cluster to force a reset. Additionally, check for any resource locks in the Azure Portal that might be preventing deallocation. If the issue persists, review logs in Azure Monitor to identify any hanging job.

Thank you.

Answer 2

That seems to fix the issue, thanks @santoshkc