Event Subscription Storage Blob create/delete not triggered via azure cli, azcopy or python SDK

Ayriss, Barnaby (The Bridge) 0

I've set up a Service Bus queue and an Event Subscription to monitor a directory in my Blob storage, to log the event in the service bus queue, whenever a blob is created, or deleted. This system works as intended when I manually upload files using the web browser portal. However the event subscription does not trigger when I upload or delete files using the Azure cli, AzCopy cli, or the Python SDK functions.

Is this functionality intended? and if so how can I have my event subscription detect files being uploaded though these methods? otherwise can you inform me of other programatic way of uploading files that will trigger the event subscriber to put the event in my service bus queue?

I have also tired using the REST api to PUT files on the blob store but could not get this to work with our storage account repeatedly getting "PublicAccessNotPermitted" error codes, even when using various company proxies and SAS tokens, so I do not believe this avenue will work for my use case.

Vinod Kumar Reddy Chilupuri 1,125 Reputation points Microsoft Vendor

2024-11-01T13:33:07.52+00:00

Hi Ayriss, Barnaby (The Bridge),

Welcome to Microsoft Q&A, thanks for posting your query.

The issue you are facing is blob events are not triggering notifications in your service bus when uploading files through programmatic methods like CLI, AzCopy or Python SDK. And working perfectly if you upload or delete files manually.

Here are few steps helps you to solve your issue.

Event grid encryption:

Verify that your Event Grid subscription is configured to capture BlobCreated and BlobDeleted events. Sometimes, event subscriptions can have limitations to specific storage access methods. Re-create the subscription can help you to check whether you are connected to the events you want.

Check Storage Account Setting:

Check your storage account is configured to support all the access types. Event Grid should capture events from different clients like SDK and CLI, but sometimes account settings such as IP restrictions, Network Security Groups (NSGs) or private endpoints can sometimes block the event handling. Verify that there are no network restrictions that might interfere with the delivery of events.

Upload workarounds with azure functions:

Consider implementing an Azure Function with a Blob Trigger as an alternative to Event Grid. This Blob Trigger can effectively monitor changes, such as creation or deletion of blobs in your storage account, regardless of how the uploads are performed. You can configure the function to send a message to your Service Bus queue whenever it detects a blob event. This approach ensures that you capture all blob modifications without relying solely on Event Grid.

Debugging Rest API Access:

To resolve the PublicAccessNotPermitted errors when using the REST API, ensure that your SAS token or credentials have the necessary permissions (such as create, write, and delete) for the container. Additionally, check that your storage account is configured to permit REST API access; this may include settings related to public access or specific IP allow lists, especially in restricted network environments.

Alternative Programmatic Solutions:

If issues with the REST API continue, consider using Azure Storage SDKs (for languages like Python or .NET) with a Managed Identity or a connection string. This approach can often circumvent network restrictions encountered with direct REST API calls, while still allowing for programmatic access to your storage resources.

Reference:

https://learn.microsoft.com/en-us/azure/event-grid/event-schema-blob-storage?tabs=cloud-event-schema

https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-blob-trigger?tabs=csharp

https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview

Please let us know if you have any further queries. I’m happy to assist you further.
Vinod Kumar Reddy Chilupuri 1,125 Reputation points Microsoft Vendor

2024-11-04T10:47:48.1366667+00:00

Hello Ayriss, Barnaby (The Bridge),
Just checking to see that the below answer helped you to solve your query. Please don't forget to Accept the answer and Up-vote for the answer which helped you, which might be beneficial to other community members reading this thread. If you have any further query do let us know.
Vinod Kumar Reddy Chilupuri 1,125 Reputation points Microsoft Vendor

2024-11-05T02:41:03.8266667+00:00

Hello Ayriss, Barnaby (The Bridge),
Just checking to see that the below answer helped you to solve your query. Please don't forget to Accept the answer and Up-vote for the answer which helped you, which might be beneficial to other community members reading this thread. If you have any further query do let us know.
Ayriss, Barnaby (The Bridge) 0 Reputation points

2024-11-08T12:17:27.31+00:00

Hello Vinod Kumar Reddy Chilupuri, I did not manage to fix my Event Subscriber problem. Instead I decided to implement a different solution for now, running the code at set intervals instead of when a blob change is detected. But I will likely return to this problem in the future as we would very much like to have the file detection functionality.

1 answer

Sina Salam 12,816

Hello Ayriss, Barnaby (The Bridge),

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you are having issues with the event subscription does not trigger when I upload or delete files using the Azure cli, AzCopy cli, or the Python SDK functions, except when you uploaded manually.

This is an unexpected behavior with your event subscription, and you forgot to share your code, error or logs with your question.

However, the possible causes of these issues could be due to your Subscription Configuration, check if your Event Grid subscription is correctly configured to listen to all blob events, not just those triggered by the portal. Also, your permissions level, you will need the identity used by the CLI, AzCopy, or SDK to have the necessary permissions to trigger events. Because different authentication methods might have different permissions.

About programmatic uploads that trigger Event Subscriptions, you have used all methods, but yet to affirm your code and detail error logs.

About REST API and “PublicAccessNotPermitted” Error, you do not need to give up. The “PublicAccessNotPermitted” error typically indicates that the storage account or container does not allow public access. You can resolve them by:

Make sure the container’s access level is set to allow the required operations. az storage container set-permission --name <container_name> --public-access blob
Generate a SAS token with the necessary permissions and use it in your REST API calls. az storage container generate-sas --name <container_name> --permissions rwdl --expiry <expiry_date>
Make sure your network and proxy settings allow the necessary traffic to Azure Blob Storage.

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

Ayriss, Barnaby (The Bridge) 0

Hello @Sina Salam Thank you for your response.

There are no error logs to share as no errors are occurring, just a lack of events. Below is the python code I have been using to upload and delete blobs, for testing. I have also run the cli commands manually with the same outcome.

The storage account is Gen2 storage. If you wish for further info on this please tell me which CLI command I should use.

I made my Event Subscription though the portal, and in my filters I have set subject starts with /blobServices/default/containers/container_name/blobs/directory_name and have all 10 available event types selected "Blob Created, Blob Deleted, Directory Created, Directory Deleted, Blob Renamed, Directory Renamed, Blob Tier Changed, Blob Inventory Completed, Async Operation Initiated, Lifecycle Policy Completed" but when running the below code no events are triggered other than actions through the portal, I also do not see any messages in the service bus queue nor updates in the event subscription metrics graph.

How can I check the permissions of the identities with regards to triggering event subscriptions?

I have also now looked into the storage account activity logs. I can see the Event subscription attaching itself whenever I update it but uploading the code via the SDK or CLI illicit no logged activity.

I will also retry using the REST API in the mean time.

Thank you for your help.

Python code to upload and delete a blob to Azure blob store. Login to azure cli before running code.


# %% --------------------------------------------------------------------------
# Imports and Constants
# -----------------------------------------------------------------------------
#region Imports_and_Constants
# Standard Library
import os
from pathlib import Path
from psutil._common import bytes2human
from datetime import datetime, timedelta, timezone
import logging
from pprint import PrettyPrinter
import subprocess
from subprocess import CompletedProcess, CalledProcessError
from typing import Dict
# 3rd Party
from azure.identity import DefaultAzureCredential, AzureCliCredential
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, BlobPrefix, BlobProperties
from azure.servicebus import ServiceBusClient, ServiceBusReceivedMessage, ServiceBusMessage, ServiceBusSender
import azure.core.exceptions as Azexp

# Custom



# Constants & Globals
NO_PROXY = { "http":"", "https":"" }

pp = PrettyPrinter(indent=1)
logging.basicConfig(level="DEBUG")
log = logging.getLogger(__name__)
# Notes


# %% --------------------------------------------------------------------------
# Azure utils
# -----------------------------------------------------------------------------
#region Azure Utils
def get_BlobServiceClient(account_url:str, credentials:str|Dict[str, str]|None=None, proxies:Dict[str,str]=None )->BlobServiceClient:
    """Get standard blob service client for using this library.
    Can use None/DefaultAzureCredential if azure cli has already been logged in. \"$ az login\" on runners \"$ az login --identity\"
    
    src: https://learn.microsoft.com/en-us/python/api/azure-storage-blob/azure.storage.blob.blobserviceclient?view=azure-python
    proxy docs: https://learn.microsoft.com/en-us/azure/developer/python/sdk/azure-sdk-configure-proxy?tabs=cmd#custom-configuration
    
    Args:
        account_url (str): Storage account URL, e.g. \"https://<storage-account-name>.blob.core.windows.net\"
        credentials (str | Dict[str, str] | None, optional): Credentails, If already logged in using the azure cli, then give None and DefaultAzureCredential will be used. Defaults to None.
        proxies (Dict[str,str] | None, optional): A dictionary with the keys, \"http\" and/or \"https\" and their respective proxy URLS. If not given will use default/environment proxy settings. If usernames or passwords may contain special characters use urllib.parse.quote() to encode them properly. 
    Returns:
        BlobServiceClient: BlobServiceClient object.
    """
    if not credentials:
        credentials = DefaultAzureCredential() # Assumes terminal/client is already logged in
        
    client = BlobServiceClient(
        account_url=account_url,
        credential=credentials,
        proxies=proxies)
    return client

def blob_exists(client_blobsvc: BlobServiceClient, container_name:str, pathblob:str)->bool:
    """Return if a blob exists.

    Args:
        client_blobsvc (BlobServiceClient): Blob service client
        container_name (str): Container name e.g. \"container1\"
        pathblob (str): Path in container to blob.

    Returns:
        bool: True if blob exists.
    """
    client_blob = client_blobsvc.get_blob_client(container=container_name, blob=pathblob)
    output = client_blob.exists()
    return output


def blob_upload_from_file(client_blobsvc:BlobServiceClient, container_name:str, pathfile_src:Path|str, pathblob_dest:Path|str, overwrite=True)->None:
    """Upload a blob to Azure blob store.
    Add \"/\" to pathfile_dest to indicate parent directories. Parent directorise will automatically be created.

    src: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-upload-python
    src2: https://github.com/Azure-Samples/AzureStorageSnippets/blob/master/blobs/howto/python/blob-devguide-py/blob-devguide-upload.py
    
    Args:
        client_blobsvc (BlobServiceClient): Blob service client
        container_name (str): Container name e.g. \"container1\"
        pathfile_src (Path|str): File path to file to upload.
        pathfile_dest (Path|str): Path representing where to upload the file to excluding the container. Add \"/\" delimiteres to place in directory, parent fdirectories will be automatically created. e.g. \"dir0/dir1/filename.txt\"
        overwrite (bool, optional): Whether to overwrite the file if it exsits. Defaults to True.
    Raises:
        azure.core.exceptions.ResourceExistsError, ErrorCode:BlobAlreadyExists: When overwrite is False and blob already exists 
    """
    # Can add / in name to specify subdirectory
    client_container = client_blobsvc.get_container_client(container=container_name)
    file_size = bytes2human(os.path.getsize(pathfile_src))
    log.debug("Uploading file: {}, {} -> {}/{}".format(file_size, str(pathfile_src), container_name, str(pathblob_dest)))
    dt_start = datetime.now()
    with open(pathfile_src, mode="rb") as f:
        client_blob = client_container.upload_blob(name=str(pathblob_dest), data=f, overwrite=overwrite)
    log.debug("Upload complete: {}, {} -> {}/{}".format(file_size,(datetime.now()-dt_start),container_name,str(pathblob_dest)))
    return None

def blob_upload_from_file_cli(storage_account_name:str, container_name:str, pathfile_src:Path, pathblob_dest:str )->CompletedProcess:
    """Upload file using az cli commands.
    Assumes az cli is already logged in. Run `az login` in terminal before running this code`
    Args:
        storage_account_name (str): Name of Azure blob storage account.
        container_name (str): Name of container within sotrage account.
        pathfile_src (Path): Path to file to upload.
        pathblob_dest (str): Path to location file should be uploaded to.

    Returns:
        CompletedProcess: Completed process of subroutine, with output captured.
    """
    log.debug("AZ cli uploading: {} -> {}/{}".format(str(pathfile_src), container_name, str(pathblob_dest)))
    dt_start = datetime.now()
    cmd = "no_proxy=\"*\" ;NO_PROXY=\"*\" ;az storage blob upload --account-name \"{}\" --container-name \"{}\" --auth-mode login --overwrite --file \"{}\" --name \"{}\"".format(storage_account_name, container_name, str(pathfile_src), str(pathblob_dest))
    cp = subprocess.run(cmd, shell=True, capture_output=True)
    if "ERROR" in cp.stderr.decode("utf-8"):
        log.warning("File upload may have failed Check following outputs.\nstdout:\n{}\nstderr:\n{}".format(cp.stdout.decode("utf-8"),cp.stderr.decode("utf-8")))
    else:
        log.debug("AZ cli upload complete - {} file:{} \nstdout:\n{}\nstderr:\n{}".format(
            datetime.now()-dt_start, str(pathfile_src), cp.stdout.decode("utf-8"), cp.stderr.decode("utf-8")))
    return cp

def blob_delete(client_blobsvc: BlobServiceClient, container_name:str, pathblob:str):
    """Delete given blob.
    Blob must exists, and if directory blob then must be empty.
    Use blob_exists() function to check if blob exists first.
    Args:
        client_blobsvc (BlobServiceClient): 
        container_name (str): Container name e.g. \"container1\"
        pathblob (str): Path to blob/full blob name e.g. \"blob.txt\" or \"dir0/blob.txt\"
        
    Raises:
        azure.core.exceptions.ResourceNotFoundError, ErrorCode:BlobNotFound: When given blob file does not exist in given container e.g. \"fakefile.txt\".
        azure.core.exceptions.HttpResponseError, ErrorCode:InvalidUri: When given path does not exist in given container e.g. \"fakedir/\".
        azure.core.exceptions.ResourceExistsError, ErrorCode:DirectoryIsNotEmpty: When directory blob is not empty.
    """
    log.debug("Deleting blob: {}/{}".format(container_name, pathblob))
    client_blob = client_blobsvc.get_blob_client(container=container_name, blob=pathblob)
    client_blob.delete_blob()

#endregion 


def main():
    storage_account_name = "storage_account_name"
    az_storage_url = "https://{}.blob.core.windows.net".format(storage_account_name)
    container_name = "container_name"
    pathblobdir_upload_test = "/dir/upload_test.txt"
    client_blobsvc = get_BlobServiceClient(az_storage_url, AzureCliCredential(), NO_PROXY)
    
    # Make test file
    pathfile_test = Path(__file__).parent/"upload_temp.txt"
    pathfile_test.parent.mkdir(parents=True, exist_ok=True)
    with open(pathfile_test,"w",encoding="utf-8") as f:
        f.write("This is a test upload file")
    # Step though code an watch event subscription/service bus queue for events.
    # Test if "Microsoft.Storage.BlobCreated" event is triggered on each upload.
    
    # Delete test file if it exists before testing
    if blob_exists(client_blobsvc, container_name, pathblobdir_upload_test):
        blob_delete(client_blobsvc, container_name, pathblobdir_upload_test)
    # Upload file to blob store using Python SDk
    blob_upload_from_file(client_blobsvc, container_name, pathfile_test, pathblobdir_upload_test, True)
    blob_delete(client_blobsvc, container_name, pathblobdir_upload_test)
    # Upload via cli
    blob_upload_from_file_cli(storage_account_name, container_name, pathfile_test, pathblobdir_upload_test)
    blob_delete(client_blobsvc, container_name, pathblobdir_upload_test)
    
    pathfile_test.unlink()
    return None




if __name__ == "__main__":
    main()
else:
    pass

Ayriss, Barnaby (The Bridge) 0 Reputation points

2024-11-01T13:07:21.65+00:00

Hello @Sina Salam

I accidentally posted my response to your comment as it own comment instead of as response to your comment.

Many Thanks

Barnaby
Ayriss, Barnaby (The Bridge) 0 Reputation points

2024-11-01T13:11:56.8366667+00:00

Hello @Sina Salam

I tired to respond to your answer and provide my code as requested, but my comment was deleted for a violation of the code of conduct. I will rewrite my answer and hopefully it is not deleted again.
Sina Salam 12,816 Reputation points

2024-11-01T14:49:18.0166667+00:00

Hello Ayriss, Barnaby (The Bridge),

Thank you for your feedback.

Yes! I saw deleted one, and I have restored it. After careful examining the content, since there are no errors, focusing on permissions, diagnostic logging, and verifying Event Grid filters and metrics should help pinpoint the issue.

Success.
Ayriss, Barnaby (The Bridge) 0 Reputation points

2024-11-08T12:20:32.9666667+00:00

I'm trying to complete the "Did this solve your problem?" section to close the issue but the submit button does nothing.
Sumarigo-MSFT 47,106 Reputation points Microsoft Employee

2024-11-19T06:00:12.21+00:00

@Ayriss, Barnaby (The Bridge) Apologies for the delay response! I would like to work closer on this issue, Can you please share the details asked in the private message

Share via

Event Subscription Storage Blob create/delete not triggered via azure cli, azcopy or python SDK

1 answer

Your answer