Intermittent Mount configuration error

I'm getting this error in my new Azure batch workloads. This error does not appear every time though. It is like 25% of the time my pool creates a new node this error happens. The only solution I have now is to re-image the node but that is not what I want to be doing.

NOTES:

I'm using a custom image from Ubuntu 24.04
The mount configuration is using blobfuse2 to mount a blob container.

Thanks,

Markapuram Sudheer Reddy 830 Reputation points Microsoft Vendor

2025-02-19T22:54:31.1233333+00:00

Hi Arthur Francisco Araujo Fernandes,

Thank You for reaching out to Microsoft Q&A forum.

The error you are facing like out of memory can be due to high memory usage by running applications or insufficient memory allocation.

Ensure that the image you configured does not have any unnecessary services or applications running that could consume memory.

Ensure that the VM size you are using in your Azure Batch pool has sufficient memory and CPU resources.

Use the commands like top, htop, or free -m to check memory usage.

If you have any queries, please do let us know, we will help you.
Arthur Francisco Araujo Fernandes 0 Reputation points

2025-02-19T23:17:54.8066667+00:00
Thanks for the comment @Markapuram Sudheer Reddy . I will recreate my custom image and see if that solves the issue. But that issue happens even BEFORE my task starts, so I don't get why the OOM error. Also, what bothers me more is that it is intermittent. I.E. I have a test job that ran 80% of the time yesterday. The fails happened, most of the time when imaging the nodes. The test job is not supposed to cause any OOM issues. When I ran it locally it uses at most 50G and the SKU has 160G of memory.

Another piece that bothers me is that the error in the portal does NOT match the bfuse.log. This is what was inside the bfuse.log

2025-02-18T23:39:36,664296465+00:00 - INFO: Setting account name, key and SAS URL for /mnt/batch/tasks/fsmounts/bfuseconnect/dmuconnection.yaml 2025-02-18T23:39:36,666878498+00:00 - INFO: Downloading: https://packages.microsoft.com/keys/microsoft.asc as microsoft.asc 2025-02-18T23:39:36,721643129+00:00 - INFO: Downloading: https://packages.microsoft.com/config/ubuntu/24.04/prod.list as /etc/apt/sources.list.d/microsoft-prod.list gpg: no valid OpenPGP data found. gpg: dearmoring failed: File exists
Markapuram Sudheer Reddy 830 Reputation points Microsoft Vendor

2025-02-20T21:51:15.8733333+00:00

Hi Arthur Francisco Araujo Fernandes,

As you stated above that, even before the task starts, sometimes you are facing OOM issue, as it can also be due to Job preparation tasks or start task which runs on each node consuming memory during node setup. https://learn.microsoft.com/en-us/azure/batch/batch-job-prep-release#job-preparation-task

Check quotas and limits of batch service. https://docs.azure.cn/en-us/batch/batch-quota-limit

https://learn.microsoft.com/en-us/azure/batch/batch-quota-limit#pool-size-limits

As you are looking between the discrepancy in the error logs as the Azure portal often displays summarized error information, provides us a general overview of the issue whereas bfuse.log shows detailed logs generated by the bfuse process. The bfuse.log focuses mainly on bfuse's operations, such as setting up file system mounts or downloading packages.

The GPG error logs which suggests above states that the system is unable to validate the downloaded package lists. Ensure that your custom image is having the necessary GPG keys, and that the system is configured to properly validate packages.

If possible, try to recreate with a custom image or if it is convenient to use Azure Marketplace image.

If you have any queries, please do let us know, we will help you.

Share via

Intermittent Mount configuration error

Your answer