I initiated a run on Azure ML that was expected to take approximately 10 days. The run details are as follows: VM Type: Standard-NC12s-v3-LOW (2 GPUs), Low Priority Instances: 1 machine Framework: Python, Detectron2 After a few days, the following error occurred: UserError: Operation returned an invalid status code 'Unauthorized' . Issue Details: This error is not visible in either the user_logs or the system_logs . Both logs appear normal and do not show any signs of an issue. The run does not make any Azure API calls and is a straightforward "local" training process. After encountering the error, I was able to restart the run and resume training without any issues, which suggests the error might originate from Azure's side. Request: Could you please investigate the root cause of this error? Are there any known issues with the runtime environment that could lead to such Unauthorized errors? Are there additional logs or diagnostic tools I can use to gain further insights? Thank you for your support!I initiated a run on Azure ML that was expected to take approximately 10 days. The run details are as follows: VM Type: Standard-NC12s-v3-LOW (2 GPUs), Low Priority Instances: 1 machine Framework: Python, Detectron2 After a few days, the following error occurred: UserError: Operation returned an invalid status code 'Unauthorized' . Issue Details: This error is not visible in either the user_logs or the system_logs . Both logs appear normal and do not show any signs of an issue. The run does not make any Azure API calls and is a straightforward "local" training process. After encountering the error, I was able to restart the run and resume training without any issues, which suggests the error might originate from Azure's side. Request: Are there any known issues with the runtime environment that could lead to such Unauthorized errors? Are there additional logs or diagnostic tools I can use to gain further insights? Thank you for your support! Best regards, Marc

Error in Azure ML Run: UserError: Operation returned an invalid status code 'Unauthorized'

Marc Eichenberger 0

I initiated a run on Azure ML that was expected to take approximately 10 days. The run details are as follows:

After a few days, the following error occurred: UserError: Operation returned an invalid status code 'Unauthorized'.

Issue Details:

This error is not visible in either the user_logs or the system_logs. Both logs appear normal and do not show any signs of an issue.
The run does not make any Azure API calls and is a straightforward "local" training process.
After encountering the error, I was able to restart the run and resume training without any issues, which suggests the error might originate from Azure's side.

Request:

Could you please investigate the root cause of this error?
Are there any known issues with the runtime environment that could lead to such Unauthorized errors?
Are there additional logs or diagnostic tools I can use to gain further insights?

Thank you for your support!I initiated a run on Azure ML that was expected to take approximately 10 days. The run details are as follows:

After a few days, the following error occurred:
UserError: Operation returned an invalid status code 'Unauthorized'.

Issue Details:

This error is not visible in either the user_logs or the system_logs. Both logs appear normal and do not show any signs of an issue.
The run does not make any Azure API calls and is a straightforward "local" training process.
After encountering the error, I was able to restart the run and resume training without any issues, which suggests the error might originate from Azure's side.

Request:

Are there any known issues with the runtime environment that could lead to such Unauthorized errors?
Are there additional logs or diagnostic tools I can use to gain further insights?

Thank you for your support!

Best regards,
Marc

romungi-MSFT 47,211 Reputation points Microsoft Employee

2024-11-19T12:07:09.23+00:00

@Marc Eichenberger AFAIK this error could occur in your case since the training time seems to be too long and the job might have timed out and the subsequent retries might have failed to authenticate. I think this should be captured in the logs, since you are unable to spot the obvious reason, you can try to create a support case and check for any other issues.

Is there any reason to use a long training time? Can this be decreased or could you use compute that can complete this faster?
Marc Eichenberger 0 Reputation points

2024-11-19T12:38:19.2066667+00:00

Thank you for the quick response. The logs don’t show any errors, which makes me think this isn’t a “typical” issue. Specifically, it shows "UserError:", and errors related to Python logic usually appear differently in the portal.

I could likely speed up the training by using more GPUs. However, I wasn’t aware that Azure ML might have such limitations..
romungi-MSFT 47,211 Reputation points Microsoft Employee

2024-11-20T07:23:12.0133333+00:00

@Marc Eichenberger Yes, please try to complete the training sooner and check if it goes through. Thanks!!