How to fix an endpoint that is perpetually stuck updating

Question

I have a batch endpoint that I was adding a new deployment to, and the Provisioning state of the endpoint is stuck on updating.

This means that I cannot take any other action concerning this endpoint, be it another deployment or deleting the endpoint, as it clashes with the updating process it is stuck on.

There were 3 successful deployments prior to this broken state.

In the same workspace, another endpoint has broken with the Provisioning state of the endpoint set to succeeded but the deployment Provisioning state is stuck on Creating.

I'd like to end these actions so that I can update the endpoints & continue to use, or in the worst case delete them so there aren't broken endpoints hanging around.

Answer 1

Hi there CB

Thanks for using QandA platform

you can try a few steps. First, check for any active operations that might be causing the endpoint to be stuck. If there are no ongoing tasks, you can attempt to cancel any stuck operations using the Azure CLI with the az ml endpoint operation cancel command. Also, verify if there are any resource quota issues that could be preventing the update.

If the endpoint remains unresponsive, you can try force-deleting it using the az ml endpoint delete command.

If this helps kindly accept the response thanks much.

Answer 2

The issue was in the AzureML backend, after trying many front end fixes, support liaised with the developers in order to manually force an update to the state of the end points.

I didn't get emailed about these comments, so did not know they had been made.

Answer 3

Hi Gary Cowan !

We have noticed that you rated an answer as not helpful. We appreciate your feedback and are committed to improving your experience with the Q&A.

Sorry for the inconvenience.

Re-attached commands for force update/cancel of batch endpoint

#deleting batch endpoint
az ml batch-endpoint delete --name <endpointname> --resource-group <rgname> -- workspace-name <workspace name> --no-wait True

#force update
az ml batch-endpoint update --name <endpointname> --resource-group <rgname> -- workspace-name <workspace name> --no-wait True

But please note that endpoint provisioning also fails if the computes does not have sufficient memory or syntax issues in scoring script or corrupt environments

We should check in endpoint logs for actual issue and act accordingly.

Reference

Troubleshoot Batch endpoint

If you find this answer useful, please upvote for this answer.

Thank you.