Hi Melan Peiris - Thanks for reaching out over Q&A Forum.
Ideally that shouldn't be the case. When using SFTP, you may want to limit public access through configuration of a firewall, virtual network, or private endpoint. These settings are enforced at the application layer, which means they aren't specific to SFTP and will impact connectivity to all Azure Storage Endpoints. So, even the SFTP calls should also tend to get blocked.
I'll test this once however it would request you to test it again just in case there is some regression in applying the changes and also enable the logging to see what is the IP being passed for calls.