DeliveryFailedEvent from Azure Event Grid to Synapse trigger with errorCode=RequestEntityTooLarge
Hobday, Colin
40
Reputation points
I have an event-driven flow where
- changes in Comsos DB triggers an Azure Function
- the Azure Function pushes each changed document to a topic in Azure Event Grid
- a Synapse trigger is triggered and pushes the changed document to a Synapse Pipeline
Parts 1 and 2 seem to be working, but in Azure Event Grid, I am getting DeliveryFailedEvents.
Part 3 is not working when the documents is 150KB.
From the diagnostic logs in AegDeliveryFailureLogs table, it is clear that the reason is PayloadTooLarge, although the message is only about 150KB.
outcome=PayloadTooLarge, latencyInMs=86, id=8e10d942-42f5-435c-a416-49af181c936e, outputEventSystemId=e549e75e-dc5d-484f-a839-67ef9fd3210a, state=Filtered, deliveryTime=11/4/2024 7:51:11 AM, deliveryCount=0, probationCount=0, deliverySchema=EventGridEvent, trackedSystemTopicState=, eventSubscriptionDeliverySchema=EventGridEvent, outputEventFields=InputEvent| EventSubscriptionId| DeliveryTime| State| Id| LastHttpStatusCode| DeliverySchema| LastDeliveryAttemptTime| SystemId, outputEventFieldCount=, dedicatedQueueDeliveryQueueId=, requestExpiration=1/1/0001 12:00:00 AM, delivered=False id=eqpt-1111, inputEventSystemId=5d72a537-62ec-4bcd-b24a-3433a53cec68 publishTime=11/4/2024 7:51:11 AM, eventTime=11/4/2024 7:51:11 AM, eventType=update, deliveryTime=11/4/2024 7:51:11 AM, filteringState=FilteredWithRpc, inputSchema=EventGridEvent, publisher=XXX-EVENT-TOPIC.SOUTHEASTASIA-1.EVENTGRID.AZURE.NET, size=131852, subject=cosmos/update, inputEventFields=Id| PublishTime| SerializedBody| EventType| Topic| Subject| FilteringHashCode| SystemId| Publisher| FilteringTopic| TopicCategory| DataVersion| MetadataVersion| InputSchema| EventTime| FilteringPolicy, inputEventFieldCount=16, type=HttpPost, subType=AzureDataFactory, supportsBatching=False, aadIntegration=False, managedIdentityType=None, urlPath=https://xxx.svc.datafactory.azure.com:4443/triggerevent/XXTrigger/triggerCosmosDbChange, deliveryResponse=PayloadTooLarge, errorCode=RequestEntityTooLarge, HttpRequestMessage: httpVersion=1.1, HttpResponseMessage: HttpVersion=1.1, StatusCode=RequestEntityTooLarge(RequestEntityTooLarge), StatusDescription=Payload Too Large, IsRedirected=False, RedirectUrl=,
I have found that it works when the document is <32KB, i.e. the Synapse pipeline is trggered.
@app.event_grid_output(
arg_name="outputEvent",
topic_endpoint_uri="EventGridTopicEndpointUri", # in application settings
topic_key_setting="EventGridTopicKeySetting", # in application settings
)
def cosmosdb_trigger1(
docs: func.DocumentList,
outputEvent: func.Out[func.EventGridOutputEvent]
):
logging.info('Python CosmosDB triggered.')
if docs:
for document in docs:
now_time = datetime.datetime.utcnow()
event_data = {
'id': now_time.isoformat() + '-' + document['id'],
'data': document.to_json(),
'subject':"cosmos/update",
'event_type': "update",
'event_time': now_time.isoformat(),
'data_version': "1.0"
}
event_data_str = json.dumps(event_data)
event_data_size = len(event_data_str.encode('utf-8'))
logging.info(f"Event data size: {event_data_size} bytes Content: {event_data_str[:1000]}")
event_data.pop('event_time')
outputEvent.set(func.EventGridOutputEvent(**event_data, event_time=now_time))
else:
logging.info("No documents found in the trigger.")
Why is the Synapse trigger not running when the payload is >32 KB?
Sign in to answer