DeliveryFailedEvent from Azure Event Grid to Synapse trigger with errorCode=RequestEntityTooLarge

Question

I have an event-driven flow where

1. changes in Comsos DB triggers an Azure Function
2. the Azure Function pushes each changed document to a topic in Azure Event Grid
3. a Synapse trigger is triggered and pushes the changed document to a Synapse Pipeline

Parts 1 and 2 seem to be working, but in Azure Event Grid, I am getting DeliveryFailedEvents.

Part 3 is not working when the documents is 150KB.

From the diagnostic logs in AegDeliveryFailureLogs table, it is clear that the reason is PayloadTooLarge, although the message is only about 150KB.

outcome=PayloadTooLarge, latencyInMs=86, id=8e10d942-42f5-435c-a416-49af181c936e, outputEventSystemId=e549e75e-dc5d-484f-a839-67ef9fd3210a, state=Filtered, deliveryTime=11/4/2024 7:51:11 AM, deliveryCount=0, probationCount=0, deliverySchema=EventGridEvent, trackedSystemTopicState=, eventSubscriptionDeliverySchema=EventGridEvent, outputEventFields=InputEvent| EventSubscriptionId| DeliveryTime| State| Id| LastHttpStatusCode| DeliverySchema| LastDeliveryAttemptTime| SystemId, outputEventFieldCount=, dedicatedQueueDeliveryQueueId=, requestExpiration=1/1/0001 12:00:00 AM, delivered=False id=eqpt-1111, inputEventSystemId=5d72a537-62ec-4bcd-b24a-3433a53cec68 publishTime=11/4/2024 7:51:11 AM, eventTime=11/4/2024 7:51:11 AM, eventType=update, deliveryTime=11/4/2024 7:51:11 AM, filteringState=FilteredWithRpc, inputSchema=EventGridEvent, publisher=XXX-EVENT-TOPIC.SOUTHEASTASIA-1.EVENTGRID.AZURE.NET, size=131852, subject=cosmos/update, inputEventFields=Id| PublishTime| SerializedBody| EventType| Topic| Subject| FilteringHashCode| SystemId| Publisher| FilteringTopic| TopicCategory| DataVersion| MetadataVersion| InputSchema| EventTime| FilteringPolicy, inputEventFieldCount=16, type=HttpPost, subType=AzureDataFactory, supportsBatching=False, aadIntegration=False, managedIdentityType=None, urlPath=https://xxx.svc.datafactory.azure.com:4443/triggerevent/XXTrigger/triggerCosmosDbChange, deliveryResponse=PayloadTooLarge, errorCode=RequestEntityTooLarge, HttpRequestMessage: httpVersion=1.1, HttpResponseMessage: HttpVersion=1.1, StatusCode=RequestEntityTooLarge(RequestEntityTooLarge), StatusDescription=Payload Too Large, IsRedirected=False, RedirectUrl=,

I have found that it works when the document is <32KB, i.e. the Synapse pipeline is trggered.

@app.event_grid_output(
    arg_name="outputEvent",
    topic_endpoint_uri="EventGridTopicEndpointUri",  # in application settings
    topic_key_setting="EventGridTopicKeySetting",  # in application settings
)                     
def cosmosdb_trigger1(
    docs: func.DocumentList, 
    outputEvent: func.Out[func.EventGridOutputEvent]
):
    logging.info('Python CosmosDB triggered.')
    if docs:
        for document in docs:
            now_time = datetime.datetime.utcnow() 
            event_data = {
                'id':  now_time.isoformat() + '-' + document['id'], 
                'data': document.to_json(),
                'subject':"cosmos/update",
                'event_type': "update",
                'event_time': now_time.isoformat(),
                'data_version': "1.0"                
            }
            event_data_str = json.dumps(event_data)
            event_data_size = len(event_data_str.encode('utf-8'))
            logging.info(f"Event data size: {event_data_size} bytes Content: {event_data_str[:1000]}")

            event_data.pop('event_time')
            outputEvent.set(func.EventGridOutputEvent(**event_data, event_time=now_time))
    else:
        logging.info("No documents found in the trigger.")

Why is the Synapse trigger not running when the payload is >32 KB?