Problem with BlobTrigger in a self hosted k8s runtime using Managed Identities

Question

I am trying to setup managed identities so that a BlobTrigger function can access a storage container. The blob trigger function is hosted in a k8s runtime and not directly by Azure Functions. I have followed the docs as far as I can go but the pod is failing to spin up because of an HTTP 403 error.

The error in the pod logs:

A host error has occurred during startup operation '278cb85c-6dfa-4bb3-97e2-1c5e149f03b8'.
 Azure.RequestFailedException: This request is not authorized to perform this operation using this permission.
 RequestId:fd5da6ed-301e-001a-38d8-18f05d000000
 Time:2024-10-07T16:43:39.1764188Z
 Status: 403 (This request is not authorized to perform this operation using this permission.)
 ErrorCode: AuthorizationPermissionMismatch
 Content:
 <?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using this permission.
 RequestId:fd5da6ed-301e-001a-38d8-18f05d000000
 Time:2024-10-07T16:43:39.1764188Z</Message></Error>
 Headers:
 Server: Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
 x-ms-request-id: fd5da6ed-301e-001a-38d8-18f05d000000
 x-ms-client-request-id: 5a6c6d04-8483-4183-ba9f-e38e2ed13765
 x-ms-version: 2021-06-08
 x-ms-error-code: AuthorizationPermissionMismatch
 Date: Mon, 07 Oct 2024 16:43:39 GMT
 Content-Length: 279
 Content-Type: application/xml
 at Azure.Storage.Blobs.ServiceRestClient.GetPropertiesAsync(Nullable`1 timeout, CancellationToken cancellationToken)
 at Azure.Storage.Blobs.BlobServiceClient.GetPropertiesInternal(Boolean async, CancellationToken cancellationToken)
 at Azure.Storage.Blobs.BlobServiceClient.GetPropertiesAsync(CancellationToken cancellationToken)
 at Microsoft.Azure.WebJobs.Extensions.Storage.Blobs.Listeners.BlobLogListener.EnableLoggingAsync(BlobServiceClient blobClient, CancellationToken cancellationToken)
 at Microsoft.Azure.WebJobs.Extensions.Storage.Blobs.Listeners.BlobLogListener.CreateAsync(BlobServiceClient blobClient, IWebJobsExceptionHandler exceptionHandler, ILogger`1 logger, CancellationToken cancellationToken)
 at Microsoft.Azure.WebJobs.Extensions.Storage.Blobs.Listeners.PollLogsStrategy.RegisterAsync(BlobServiceClient blobServiceClient, BlobContainerClient container, ITriggerExecutor`1 triggerExecutor, CancellationToken cancellationToken)
 at Microsoft.Azure.WebJobs.Extensions.Storage.Blobs.Listeners.ScanBlobScanLogHybridPollingStrategy.RegisterAsync(BlobServiceClient blobServiceClient, BlobContainerClient container, ITriggerExecutor`1 triggerExecutor, CancellationToken cancellationToken)
 at Microsoft.Azure.WebJobs.Extensions.Storage.Blobs.Listeners.BlobListenerFactory.RegisterWithSharedBlobListenerAsync(String hostId, SharedBlobListener sharedBlobListener, BlobServiceClient blobClient, BlobTriggerQueueWriter blobTriggerQueueWriter, CancellationToken cancellationToken)
 at Microsoft.Azure.WebJobs.Extensions.Storage.Blobs.Listeners.BlobListenerFactory.CreateAsync(CancellationToken cancellationToken)
 at Microsoft.Azure.WebJobs.Host.Indexers.FunctionIndexer.ListenerFactory.CreateAsync(CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Indexers\FunctionIndexer.cs:line 489
 at Microsoft.Azure.WebJobs.Host.Listeners.HostListenerFactory.CreateAsync(CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Listeners\HostListenerFactory.cs:line 64
 at Microsoft.Azure.WebJobs.Host.Listeners.ListenerFactoryListener.StartAsyncCore(CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Listeners\ListenerFactoryListener.cs:line 45
 at Microsoft.Azure.WebJobs.Host.Listeners.ShutdownListener.StartAsync(CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Listeners\ShutdownListener.cs:line 29
 at Microsoft.Azure.WebJobs.JobHost.StartAsyncCore(CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\JobHost.cs:line 99
 at Microsoft.Azure.WebJobs.Script.ScriptHost.StartAsyncCore(CancellationToken cancellationToken) in /src/azure-functions-host/src/WebJobs.Script/Host/ScriptHost.cs:line 264
 at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
 at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService.UnsynchronizedStartHostAsync(ScriptHostStartupOperation activeOperation, Int32 attemptCount, JobHostStartupMode startupMode) in /src/azure-functions-host/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs:line 376

Here is my .csproj file:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net7.0</TargetFramework>
    <AzureFunctionsVersion>v4</AzureFunctionsVersion>
    <EnforceCodeStyleInBuild>true</EnforceCodeStyleInBuild>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage.Blobs" Version="5.0.1" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage.Queues" Version="5.0.1" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="4.1.3" />
    <PackageReference Include="Azure.Identity" Version="1.5.0" />
    <PackageReference Include="Microsoft.ApplicationInsights" Version="2.18.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.ApplicationInsights" Version="2.18.0" />
  </ItemGrou
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
  </ItemGroup>
  <ItemGroup>
    <Reference Include="netstandard, Version=2.1.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51">
      <SpecificVersion>True</SpecificVersion>
    </Reference>
</Project>

Here is the Run function definition:

		[FunctionName("TelemetryBlobTrigger")]
		public void Run(
			[BlobTrigger("telemetry/telem_{fileNameSuffix}.tel", Connection = "TelemetryStorage")] Stream blobStream,
			[Blob("telemetry")] BlobContainerClient blobContainerClient,
			string fileNameSuffix,
			IDictionary<string, string> metadata,
			ILogger _
		)
		{
			TelemetryExporter exporter = new(telemetryClient, blobStream, fileNameSuffix, metadata);
			exporter.ReadAndTrack();
			MarkBlobAsProcessed($"telem_{fileNameSuffix}.tel", blobContainerClient);
		}

The live deployment manifest env definitions are below, (the TelemetryStorage__managedIdentityResourceId can be swapped for TelemetryStorage__clientId resulting in the same error in the log):

        - name: TelemetryStorage__blobServiceUri
          value: 'https://<storageContainer>.blob.core.windows.net'
        - name: TelemetryStorage__managedIdentityResourceId
          value: >-
            /subscriptions/<subscriptionId>/resourcegroups/<rg-group-id>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<managedIdentity>
        - name: TelemetryStorage__queueServiceUri
          value: 'https://<storageContainer>.queue.core.windows.net'
        - name: TelemetryStorage__credential
          value: managedIdentity

        - name: AzureWebJobsStorage__credential
          value: workloadIdentity
        - name: AzureWebJobsStorage__queueServiceUri
          value: 'https://<storageContainer>.queue.core.windows.net'
        - name: AzureWebJobsStorage__blobServiceUri
          value: 'https://<storageContainer>.blob.core.windows.net'
        - name: AzureFunctionsJobHost__functions__0
          value: TelemetryBlobTrigger

        - name: FUNCTIONS_WORKER_RUNTIME
          value: dotnet

The managedIdentity is listed under Storage Blob Data Owner and Storage Queue Data Contributor roles in the storage account. It seems like the environment variable defined managedIdentityResourceId or clientId are not being applied in the credentials of the service client but I don't know how to prove that because the local debug setup doesn't make use of it from what I can see from online examples. Am I missing a configuration or something that is needed here?

Answer 1

Hello Jon Burggraaf,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you are having challenges with BlobTrigger in a self hosted k8s runtime using Managed Identities.

Let me say, thank you for detailed information.

The error indicated that the managed identity does not have the necessary permissions to perform the requested operation on the storage account.

Firstly, the TelemetryStorage__managedIdentityResourceId and TelemetryStorage__clientId are correctly pointing to the managed identity's resource ID and client ID, respectively.

Secondly, check your connection string, it should be in the format:

   TelemetryStorage__blobServiceUri: 'https://<storageContainer>.blob.core.windows.net'
   TelemetryStorage__queueServiceUri: 'https://<storageContainer>.queue.core.windows.net'
   TelemetryStorage__credential: managedIdentity

Thirdly you mentioned that the managed identity is listed under Storage Blob Data Owner and Storage Queue Data Contributor roles. You will need to double-check these assignments in the Azure portal to confirm they are correctly applied to the storage account. Also, make sure that the managed identity is correctly configured in your Kubernetes cluster. This includes setting up the Azure AD workload identity and ensuring that the Kubernetes service account is correctly mapped to the Azure managed identity.

Lastly, let's not overlook network restriction or firewall rules that might be blocking access to the storage account from your Kubernetes cluster. You can read from this link: https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-blob-trigger for detailed setup and configuration steps.

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

---

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.