Changing ADF Trigger changes Advanced Filters in Container Blob EventGrid

Question

Hi,

I am using SFTP in a Storage Container set up with an EventGrid that triggers an ADF pipeline when a new file is uploaded to the container.

My event grid subscription is set to the filters values you can see in the first attached image. With the filters below my pipeline runs fine.

Then, if I change the setting of my trigger in ADF (see the 2nd picture attached) the filter values (3rd picture) change on their own without warning. This breaks my flow and the triggers don't work anymore. What is going on?

This happens when I change "Ignore empty blobs" to "Yes" and the new filter values are:

which break my overall pipeline and the triggers don't run anymore, this happens whether I use the Preview of ADF or not.

Answer 1

HI Luis Chaves Rodriguez - Thanks for reaching out over Q&A Forum.

When it comes to SFTP storage events, ADF doesn't add that by default and need to be defined explicitly. This is documented in the below link:

If you're working with Secure File Transfer Protocol (SFTP) storage events, you need to specify the SFTP Data API under the filtering section too.

https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger?tabs=data-factory#storage-event-trigger-considerations

If you edit, it is like a re-deployment of trigger which overrides back to default setting i.e. no SFTP related API. Unfortunately, it is by design behavior.

Hope that helps!

Please let me know if there are any further queries/concerns, will be glad to assist.

---

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

@Luis Chaves Rodriguez

Welcome to the Microsoft Q&A and thank you for posting your questions here

Your concerns about the fragility of the SFTP + Event Grid + ADF Trigger setup are valid, especially given the behavior you've encountered when modifying trigger settings. Here are some suggestions and alternatives to consider:

• If your current setup is working well, you might want to avoid changing the "Ignore empty blobs" setting unless absolutely necessary. This will help maintain the integrity of your Event Grid filters.
• If you do need to change the trigger settings, be prepared to manually reconfigure the Event Grid filters afterward. Keep a record of the necessary filter settings so you can quickly restore them if needed.
• Consider using Azure Logic Apps as an alternative to ADF triggers. Logic Apps can listen to Event Grid events and can be configured to handle SFTP events more flexibly. This might provide a more stable integration without the fragility you’re experiencing.
• You could create an Azure Function that listens to Event Grid events. This function can then trigger your ADF pipeline programmatically. This approach gives you more control over the event handling and can be designed to accommodate any specific logic you need.
• Instead of relying solely on Event Grid, you could implement a polling mechanism that checks for new files in the SFTP location at regular intervals. This would eliminate the dependency on Event Grid and its filters but may introduce some latency.
• Implement monitoring and alerting for your ADF pipelines. This way, if a pipeline fails due to a trigger issue, you can quickly respond and fix the configuration.

Please let me know if there are any further queries/concerns