message delivery failed with errorCode=RequestEntityTooLarge although only 100KB

Question

I have a trigger that subscribes to changes in Cosmos DB and pushes the document to Azure Event Grid. I am getting FailedEvents.

From the diagnostic log, it is clear that the reason is PayloadTooLarge, although the message is only about 150KB. (Event Grid is supposed to support messages up to 1MB).

outcome=PayloadTooLarge, latencyInMs=86, id=8e10d942-42f5-435c-a416-49af181c936e, outputEventSystemId=e549e75e-dc5d-484f-a839-67ef9fd3210a, state=Filtered, deliveryTime=11/4/2024 7:51:11 AM, deliveryCount=0, probationCount=0, deliverySchema=EventGridEvent, trackedSystemTopicState=, eventSubscriptionDeliverySchema=EventGridEvent, outputEventFields=InputEvent| EventSubscriptionId| DeliveryTime| State| Id| LastHttpStatusCode| DeliverySchema| LastDeliveryAttemptTime| SystemId, outputEventFieldCount=, dedicatedQueueDeliveryQueueId=, requestExpiration=1/1/0001 12:00:00 AM, delivered=False id=eqpt-1111, inputEventSystemId=5d72a537-62ec-4bcd-b24a-3433a53cec68 publishTime=11/4/2024 7:51:11 AM, eventTime=11/4/2024 7:51:11 AM, eventType=update, deliveryTime=11/4/2024 7:51:11 AM, filteringState=FilteredWithRpc, inputSchema=EventGridEvent, publisher=XXX-EVENT-TOPIC.SOUTHEASTASIA-1.EVENTGRID.AZURE.NET, size=131852, subject=cosmos/update, inputEventFields=Id| PublishTime| SerializedBody| EventType| Topic| Subject| FilteringHashCode| SystemId| Publisher| FilteringTopic| TopicCategory| DataVersion| MetadataVersion| InputSchema| EventTime| FilteringPolicy, inputEventFieldCount=16, type=HttpPost, subType=AzureDataFactory, supportsBatching=False, aadIntegration=False, managedIdentityType=None, urlPath=https://xxx.svc.datafactory.azure.com:4443/triggerevent/XXTrigger/triggerCosmosDbChange, deliveryResponse=PayloadTooLarge, errorCode=RequestEntityTooLarge, HttpRequestMessage: httpVersion=1.1, HttpResponseMessage: HttpVersion=1.1, StatusCode=RequestEntityTooLarge(RequestEntityTooLarge), StatusDescription=Payload Too Large, IsRedirected=False, RedirectUrl=,

The code is in Azure Function and uses output binding to send messages to Azure Event Grid.

@app.event_grid_output(
    arg_name="outputEvent",
    topic_endpoint_uri="EventGridTopicEndpointUri",  # in application settings
    topic_key_setting="EventGridTopicKeySetting",  # in application settings
)                     
def cosmosdb_trigger1(
    docs: func.DocumentList, 
    outputEvent: func.Out[func.EventGridOutputEvent]
):
    logging.info('Python CosmosDB triggered.')
    if docs:
        for document in docs:
            json_obj = document.to_json()
            logging.info(f"Content: {json_obj[:1000]}")
            logging.info(f"len:{len(json_obj)}")
            event_message = func.EventGridOutputEvent(
                id= document['id'],
                data= json_obj,
                subject="cosmos/update",
                event_type="update",
                event_time=datetime.datetime.utcnow(),
                data_version="1.0"
            )
            outputEvent.set(event_message)
            logging.info(f'sent {event_message}')
    else:
        logging.info("No documents found in the trigger.")

Answer 1

Hi @Hobday, Colin I have tested the behavior locally and I could not replicate this issue. I could successfully send an event of size ~68kb from my Azure function to the Event Grid topic. As you pointed out, the default limit of Event size is 1MB. The only limitation I see on the documentation that has a 32kb limit is for an MQTT topic which says MQTTv5 total size of all user properties has to be 32Kb.

Unless your custom topic is routing the events to an MQTT topic end point, I don't see why you would see this error for a data size greater than 32Kb.

One alternate approach I would recommend to try is to use the EventGridPublisherClient in the Function App instead of output binding. You can find the reference code in the following GitHub repository

If you continue to see the same behavior, please reach out to me through the comments below for a deeper investigation on this issue.