Intermittent Mount configuration error

I'm getting this error in my new Azure batch workloads. This error does not appear every time though. It is like 25% of the time my pool creates a new node this error happens. The only solution I have now is to re-image the node but that is not what I want to be doing.

NOTES:

I'm using a custom image from Ubuntu 24.04
The mount configuration is using blobfuse2 to mount a blob container.

Thanks,

1 answer

Markapuram Sudheer Reddy 990 Reputation points Microsoft Vendor

2025-02-19T22:54:31.1233333+00:00

Hi Arthur Francisco Araujo Fernandes,

Thank You for reaching out to Microsoft Q&A forum.

The error you are facing like out of memory can be due to high memory usage by running applications or insufficient memory allocation.

Ensure that the image you configured does not have any unnecessary services or applications running that could consume memory.

Ensure that the VM size you are using in your Azure Batch pool has sufficient memory and CPU resources.

Use the commands like top, htop, or free -m to check memory usage.

If you have any queries, please do let us know, we will help you.
Please sign in to rate this answer.
Arthur Francisco Araujo Fernandes 0 Reputation points

2025-02-19T23:17:54.8066667+00:00

Thanks for the comment @Markapuram Sudheer Reddy . I will recreate my custom image and see if that solves the issue. But that issue happens even BEFORE my task starts, so I don't get why the OOM error. Also, what bothers me more is that it is intermittent. I.E. I have a test job that ran 80% of the time yesterday. The fails happened, most of the time when imaging the nodes. The test job is not supposed to cause any OOM issues. When I ran it locally it uses at most 50G and the SKU has 160G of memory.

Another piece that bothers me is that the error in the portal does NOT match the bfuse.log. This is what was inside the bfuse.log

2025-02-18T23:39:36,664296465+00:00 - INFO: Setting account name, key and SAS URL for /mnt/batch/tasks/fsmounts/bfuseconnect/dmuconnection.yaml 2025-02-18T23:39:36,666878498+00:00 - INFO: Downloading: https://packages.microsoft.com/keys/microsoft.asc as microsoft.asc 2025-02-18T23:39:36,721643129+00:00 - INFO: Downloading: https://packages.microsoft.com/config/ubuntu/24.04/prod.list as /etc/apt/sources.list.d/microsoft-prod.list gpg: no valid OpenPGP data found. gpg: dearmoring failed: File exists

Markapuram Sudheer Reddy 990 Reputation points Microsoft Vendor

2025-02-20T21:51:15.8733333+00:00

Hi Arthur Francisco Araujo Fernandes,

As you stated above that, even before the task starts, sometimes you are facing OOM issue, as it can also be due to Job preparation tasks or start task which runs on each node consuming memory during node setup. https://learn.microsoft.com/en-us/azure/batch/batch-job-prep-release#job-preparation-task

Check quotas and limits of batch service. https://docs.azure.cn/en-us/batch/batch-quota-limit

https://learn.microsoft.com/en-us/azure/batch/batch-quota-limit#pool-size-limits

As you are looking between the discrepancy in the error logs as the Azure portal often displays summarized error information, provides us a general overview of the issue whereas bfuse.log shows detailed logs generated by the bfuse process. The bfuse.log focuses mainly on bfuse's operations, such as setting up file system mounts or downloading packages.

The GPG error logs which suggests above states that the system is unable to validate the downloaded package lists. Ensure that your custom image is having the necessary GPG keys, and that the system is configured to properly validate packages.

If possible, try to recreate with a custom image or if it is convenient to use Azure Marketplace image.

If you have any queries, please do let us know, we will help you.

Arthur Francisco Araujo Fernandes 0 Reputation points

2025-02-21T20:20:09.1+00:00

I don't have any job preparation task, the only preparation task that runs is from MS that installs blobfuse2. I didn't install/configure blobfuse2 since MS explicitly says to not install any Azure Extension when creating a custom image. https://learn.microsoft.com/en-us/azure/batch/batch-sig-images

Diving into the uploaded logs this is the full log error:

■■20250221T164509.170Z■00000000-0000-0000-0000-000000000000■ERROR■agent.mount.filesystems.basefilesystem■basefilesystem.py■run_cmd_persist_output_async■71■16719■MainThread■123136443162752■Mount command failed with exit code: 2, output: 2025-02-21T16:45:08,992119172+00:00 - INFO: Setting account name, key and SAS URL for /mnt/batch/tasks/fsmounts/bfuseconnect/dmuconnection.yaml 2025-02-21T16:45:08,994454408+00:00 - INFO: Downloading: https://packages.microsoft.com/keys/microsoft.asc as microsoft.asc 2025-02-21T16:45:09,086694242+00:00 - INFO: Downloading: https://packages.microsoft.com/config/ubuntu/24.04/prod.list as /etc/apt/sources.list.d/microsoft-prod.list gpg: no valid OpenPGP data found. gpg: dearmoring failed: File exists ■■20250221T164509.170Z■00000000-0000-0000-0000-000000000000■ERROR■agent.task.prep■prep.py■_perform_mounts_async■326■16719■MainThread■123136443162752■Error while performing mount□Traceback (most recent call last):□ File "agent/task/prep.py", line 324, in _perform_mounts_async□ File "agent/mount/mountmanager.py", line 71, in mount_async□ File "agent/mount/filesystems/blobfuse.py", line 55, in mount_async□ File "agent/mount/filesystems/blobfuse.py", line 88, in _install_and_mount_blobfuse_async□common.exceptions.MountConfigurationException: System error (out of memory, cannot fork, no more loop devices)

The error is with MS code when mounting the blob storage. However, it does not make sense to me that it fails from time to time but still runs correctly for some nodes. Seems like the node fails to automatically set the account and create the connection.yaml file. The OOM error happens when the agent attempts to trace back, which is probably a poor nested function call leading to the OOM error. But I don't have any control over Azure functions!

Markapuram Sudheer Reddy 990 Reputation points Microsoft Vendor

2025-02-21T20:33:28.67+00:00

Hi Arthur Francisco Araujo Fernandes,

I just wanted to check with you regarding the review of above response. If you have any queries, please share here, we will help you. If you are facing errors, please share the logs, to investigate further.

Arthur Francisco Araujo Fernandes 0 Reputation points

2025-02-21T20:36:45.9633333+00:00

@Markapuram Sudheer Reddy see my answer above.

Markapuram Sudheer Reddy 990 Reputation points Microsoft Vendor

2025-02-24T17:16:06.89+00:00

Hi Arthur Francisco Araujo Fernandes,

Based on above logs as unable to mount with Azure Batch pool node, as a mount configuration fails, can you please try once manually on the compute node which is failing with mount process by following the steps provided in the documentation.

https://learn.microsoft.com/en-us/azure/batch/virtual-file-mount?tabs=linux#troubleshoot-mount-issues

https://learn.microsoft.com/en-us/rest/api/batchservice/compute-node/get?view=rest-batchservice-2024-07-01&tabs=HTTP#computenodeerror

Please check network settings for a compute node -https://learn.microsoft.com/en-us/azure/batch/batch-virtual-network

If you have any queries, please do let us know, we will help you.

Markapuram Sudheer Reddy 990 Reputation points Microsoft Vendor

2025-02-25T22:05:01.8866667+00:00

Hi Arthur Francisco Araujo Fernandes,

I just wanted to follow up regarding above response to your query.

If you have any questions, please do let us know, we will help you.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Intermittent Mount configuration error

1 answer

Your answer