Changing ADF Trigger changes Advanced Filters in Container Blob EventGrid

Luis Chaves Rodriguez 10

Hi,

I am using SFTP in a Storage Container set up with an EventGrid that triggers an ADF pipeline when a new file is uploaded to the container.

My event grid subscription is set to the filters values you can see in the first attached image. With the filters below my pipeline runs fine. Screenshot 2024-12-04 at 14.53.25

Then, if I change the setting of my trigger in ADF (see the 2nd picture attached) the filter values (3rd picture) change on their own without warning. This breaks my flow and the triggers don't work anymore. What is going on? Screenshot 2024-12-04 at 14.54.01

This happens when I change "Ignore empty blobs" to "Yes" and the new filter values are:
Screenshot 2024-12-04 at 15.06.21

which break my overall pipeline and the triggers don't run anymore, this happens whether I use the Preview of ADF or not.

phemanth 13,150 Reputation points Microsoft Vendor

2024-12-06T18:37:05.7033333+00:00

@Luis Chaves Rodriguez We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Luis Chaves Rodriguez 10 Reputation points

2024-12-30T23:04:50.14+00:00
Hey @phemanth , thanks for checking in, I do and I don't

I did add the SFTP Data API filters to the EventGrid subscriptions. I added SFTPCommit and SFTPSetStat.

I noticed some of the files being sent by my client were being uploaded atomically rather than directly. I found this .csv.filepart file that was being created with SFTPCreate, then SFTPWrite and finally SFTPCommit before being renamed via SFTPRename to the final .csv file. My client is using WinSCP to upload the files.

It was all pretty confusing. I thought my EventGrid was working properly, catching the last SFTPSetStat on the CSV file.

See below a screenshot of the StorageBlogLogs at the end of the SFTP process with these .filepart files:

I am confused about several things:

Why is there an SFTPRemove operation trying to delete the CSV file?

I imagine is to delete it in case it exists?

The atomic upload (with filepart) only seems to be active with bigger files (these are all very small either way). There is one file (143 kB) where the file gets uploaded directly (no filepart files) but for the bigger one.

I imagine this correspond to a setting in WinSCP, do you know which one?

The last step SFTPSetStat is failing, returning a 403 error. Why? In an EventGrid Subscription, if the operation that triggers a trigger returns a non-200 error, does it still work?

I was initially adding SFTPSetStat as a filter because in the StorageBlobLogs, that operation was the last one to take place on the .csv file. I have now added the SFTPRename to the data.api filter, would it work that way even if the file in question is the filepart file? For instance this is my Subject Filters in my event grid subscription. I am not sure the "Subject" equates to the file in the SFTP request that triggers it.

Thank you for your attention in advance

2 answers

Amrinder Singh 5,555 Reputation points Microsoft Employee

2024-12-04T15:58:19.2333333+00:00

HI Luis Chaves Rodriguez - Thanks for reaching out over Q&A Forum.

When it comes to SFTP storage events, ADF doesn't add that by default and need to be defined explicitly. This is documented in the below link:

If you're working with Secure File Transfer Protocol (SFTP) storage events, you need to specify the SFTP Data API under the filtering section too.

https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger?tabs=data-factory#storage-event-trigger-considerations

If you edit, it is like a re-deployment of trigger which overrides back to default setting i.e. no SFTP related API. Unfortunately, it is by design behavior.

Hope that helps!

Please let me know if there are any further queries/concerns, will be glad to assist.

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.

1 person found this answer helpful.
Luis Chaves Rodriguez 10 Reputation points

2024-12-04T16:03:34.79+00:00

Thank you for your reply! What am I meant to do then? Am I not using EventGrid + ADF Triggers properly? Or am I just meant to not touch the "ignore empty blobs" setting of the ADF trigger?

Luis Chaves Rodriguez 10 Reputation points

2024-12-04T17:11:00.5133333+00:00

this setup with SFTP+EventGrid+ADF Trigger seems fragile by design, what alternative is there?

Luis Chaves Rodriguez 10 Reputation points

2024-12-04T17:11:40.6966667+00:00

This setup (SFTP + EventGrid + ADF Trigger) seems fragile by design, what alternative is there?

Amrinder Singh 5,555 Reputation points Microsoft Employee

2024-12-10T16:12:11.1133333+00:00

Hi @Luis Chaves Rodriguez - Thanks for the response and I understand your concerns.

From the snippets you seem to be editing this for ignore empty blobs however you won't need to edit that again and again. Once you have decided whether you need to keep that or not, you can add the SFTP APIs thereafter and shouldn't affect thereafter.

Is there any particular reason for updating the trigger condition on regular/continuous basis?

Luis Chaves Rodriguez 10 Reputation points

2024-12-30T23:26:07.2433333+00:00

Hey, indeed, I am not updating those manually often. It just surprised me that I changed something in the Trigger UI and the default behaviour was overwritten the manually set up config. Are there any docs on what SFTP APIs to include in the filters? I haven't found any despite SFTP being quite a common tool for data ingestion
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
phemanth 13,150 Reputation points Microsoft Vendor

2024-12-05T20:47:56.9033333+00:00
@Luis Chaves Rodriguez

Welcome to the Microsoft Q&A and thank you for posting your questions here

Your concerns about the fragility of the SFTP + Event Grid + ADF Trigger setup are valid, especially given the behavior you've encountered when modifying trigger settings. Here are some suggestions and alternatives to consider:

If your current setup is working well, you might want to avoid changing the "Ignore empty blobs" setting unless absolutely necessary. This will help maintain the integrity of your Event Grid filters.

If you do need to change the trigger settings, be prepared to manually reconfigure the Event Grid filters afterward. Keep a record of the necessary filter settings so you can quickly restore them if needed.

Consider using Azure Logic Apps as an alternative to ADF triggers. Logic Apps can listen to Event Grid events and can be configured to handle SFTP events more flexibly. This might provide a more stable integration without the fragility you’re experiencing.

You could create an Azure Function that listens to Event Grid events. This function can then trigger your ADF pipeline programmatically. This approach gives you more control over the event handling and can be designed to accommodate any specific logic you need.

Instead of relying solely on Event Grid, you could implement a polling mechanism that checks for new files in the SFTP location at regular intervals. This would eliminate the dependency on Event Grid and its filters but may introduce some latency.

Implement monitoring and alerting for your ADF pipelines. This way, if a pipeline fails due to a trigger issue, you can quickly respond and fix the configuration.

Please let me know if there are any further queries/concerns
Please sign in to rate this answer.

1 person found this answer helpful.
phemanth 13,150 Reputation points Microsoft Vendor

2024-12-10T15:29:03.9+00:00

@Luis Chaves Rodriguez Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

phemanth 13,150 Reputation points Microsoft Vendor

2024-12-31T18:27:26.4666667+00:00

I understand your situation with the settings being overwritten. When working with SFTP in Azure Data Factory (ADF) and EventGrid, it's essential to include specific SFTP APIs in your filters to ensure the triggers work correctly. Here are some key SFTP APIs you should include:

SFTPCreate: This API is triggered when a new file is created.

SFTPWrite: This API is used when data is written to a file.

SFTPCommit: This API is triggered when a file write operation is committed.

SFTPRename: This API is used when a file is renamed.

SFTPSetStat: This API is triggered when file attributes are set or modified.

These APIs help capture the necessary events for your ADF pipeline to trigger correctly. You can find more detailed information and guidance in the Azure Data Factory documentation

Luis Chaves Rodriguez 10 Reputation points

2025-01-06T08:28:59.2833333+00:00

Hi @phemanth I don't understand I surely should not add all those filters. I am aware about these operations and I am tracking all of them via the Monitoring tools/logs in my container. I have read lots of Microsoft documentation by this point and I still can't get it to work.

I know that part of my issue is that the client has this WinSCP setting enabled (https://learn.microsoft.com/en-us/azure/storage/blobs/secure-file-transfer-protocol-known-issues#client-settings) but I still feel like I should be able to work around that. Why is that an issue in the first place?

My event grid webhooks are still not triggered despite me setting the following data.api filters: CopyBlob, PutBlob, PutBlockList, FlushWithClose, DeleteBlob, CreateFile, DeleteFile, sftpcommit, sftpsetstat, sftprename. These work fine for files that are not broken down into .fileparts but doesn't for those where the .filepart breakdown happens. This is what happens at the end of a file upload:

I also have my filter to only work with .csv files because I want the triggers to only happen with .csv files. I don't understand why despite setting SftpSetStat in the data.api filter the trigger is not "reacting" to this event.

I appreciate your assistance in advance.

phemanth 13,150 Reputation points Microsoft Vendor

2025-01-09T05:22:43.04+00:00

@phemanth

The issue with the WinSCP setting you mentioned is known to cause problems with Event Grid triggers. This setting can lead to multiple partial file uploads (e.g., .filepart files), which might not trigger the events as expected

Review WinSCP Settings: Ensure that the WinSCP client settings are configured correctly to avoid creating .filepart files. You can refer to the WinSCP documentation for guidance on the correct settings

Event Grid Filters: Double-check your Event Grid filters to ensure they are set up correctly. Since you are already tracking multiple events, make sure that the filters are not too restrictive and are capturing all necessary events, including SftpSetStat.

Webhook Validation: Ensure that your webhook endpoint is correctly handling the validation events from Event Grid. Sometimes, issues with webhook validation can prevent events from being triggered

Testing with Different Clients: If possible, test the file uploads using a different SFTP client to see if the issue persists. This can help determine if the problem is specific to WinSCP.

Luis Chaves Rodriguez 10 Reputation points

2025-01-17T08:10:07.3733333+00:00

@phemanth based on your expertise what would be the correct EventGrid Filters? As you can see in my previous replies I have SFTPSetStat enabled as well as SFTPRename, SFTPCommit, SFTPCreate

Luis Chaves Rodriguez 10 Reputation points

2025-01-22T14:40:49.26+00:00

Hi @phemanth, trying to follow up, why are my filters not reacting to the .filepart operations? I have set up everything appropriately.

My client modified their WinSCP settings to disable the .filepart creations but I am still receiving .filepart files, it's really weird! I need help please

phemanth 13,150 Reputation points Microsoft Vendor

2025-01-23T10:12:18.6133333+00:00

@Luis Chaves Rodriguez

sure, Let's try to address your concerns:

Correct Event Grid Filters: Based on your setup, you should include the following filters to capture all relevant SFTP operations:

SFTPCreate

SFTPWrite

SFTPCommit

SFTPRename

SFTPSetStat

These filters should cover the creation, writing, committing, renaming, and setting attributes of files. Ensure that these filters are correctly configured in your Event Grid subscription.

.filepart Operations: If your filters are not reacting to .filepart operations, it might be due to the way these operations are being handled by WinSCP. Even though your client has disabled the .filepart creation, it's possible that some settings or residual configurations are still causing these files to be created.

Troubleshooting Steps:

Double-check WinSCP Settings: Ensure that all settings related to temporary file creation are disabled. Sometimes, changes might not take effect immediately, or there might be other settings influencing the behavior.

Event Grid Logs: Check the Event Grid logs to see if the events related to .filepart files are being captured. This can help identify if the issue is with the event generation or the filtering.

Test with Different SFTP Client: If possible, test the file uploads using a different SFTP client to see if the issue persists. This can help determine if the problem is specific to WinSCP.

Alternative Solutions:

Azure Logic Apps: Consider using Azure Logic Apps to handle SFTP events more flexibly. Logic Apps can listen to Event Grid events and provide more robust handling of different scenarios.

Azure Functions: Create an Azure Function to listen to Event Grid events and trigger your ADF pipeline programmatically. This approach gives you more control over event handling.

Polling Mechanism: Implement a polling mechanism to check for new files in the SFTP location at regular intervals. This can eliminate the dependency on Event Grid and its filters but may introduce some latency.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Changing ADF Trigger changes Advanced Filters in Container Blob EventGrid

2 answers

Your answer