Azure online endpoint is Scaling taking long time

Tran Hong Thu (DPS.VI.DTS) 0

I have a project to run an AI model using an online endpoint as a backend service, the endpoint is configured (manually set in the portal) to be auto-scale based on the number of requests.

Expect the endpoint to scale up between 1 to 2 minutes, like other services such as virtual machine scale set, etc...

But with the ML online endpoint, scaling takes a long time, about 12-18 minutes.

Do you have suggestions for speeding up the scaling time?

Saideep Anchuri 2,110 Reputation points Microsoft Vendor

2025-02-13T09:31:26.59+00:00
Hi Tran Hong Thu (DPS.VI.DTS)

Welcome to Microsoft Q&A Forum, thank you for posting your query here!

Scaling times for Azure Machine Learning online endpoints can vary due to several factors, including the complexity of the model, the size of the resources being allocated, and the current load on the system. Scaling an online endpoint takes at least 5 minutes. For faster scaling, you can use a compute cluster for a batch endpoint, which allows for a scale-down time of less than a minute. Alternatively, you can use an AKS (Azure Kubernetes Service) cluster, which automatically adjusts based on incoming traffic, offering a more reactive scaling solution.

az aks update \ --resource-group <yourResourceGroup> \ --name <yourAKSCluster> \ --enable-cluster-autoscaler \ --min-count <minNodeCount> \ --max-count <maxNodeCount>

To increase nodes based on workload azure-cli

Kindly refer below link:

kubernetes-online-endpoints

how-to-attach-kubernetes

machine-learning-reference

Thank You.
Tran Hong Thu (DPS.VI.DTS) 0 Reputation points

2025-02-13T09:57:55.4866667+00:00

Thanks @Saideep Anchuri for replying,

for requesting from the provider and business so we don't change to another service such as AKS, VM scaleset, batch endpoint (needs to be near realtime of the response)

My source code and AI model are very simple, I used to try to deploy it on a VM scaleset and the scaling time is only 1 - 2 minutes, and AKS is about 5 minutes.

Do you have suggestions for configuration or tips & tricks for me to reduce it to 5 minutes as you mentioned?
Saideep Anchuri 2,110 Reputation points Microsoft Vendor

2025-02-13T10:42:00.4066667+00:00

Hi Tran Hong Thu (DPS.VI.DTS)

You can adjust the Cool down (minutes) (can be reduced to 1 min and responsible Scaling out) and Duration (minimum 5 minutes for metrics monitoring) from and experiment with other metrics like CPU Utilization and connection per second etc. to adjust your scale out time. You can select custom auto scale option then click on "add a rule" to select endpoint-based metrics.

Kindly refer below screenshot:

Thank You.

Share via

Azure online endpoint is Scaling taking long time

Your answer