SQL Server Integration Services and Azure Data Lake Store Connection Manager
Is it possible a SQL Server Integration Services (SSIS) package can use the Azure Data Lake Store Connection Manager to connect to an Azure Data Lake Storage Gen2 ?? From what I read the article below refers only for Gen 1 but and about the Gen 2? Link…
Azure data lake folder structure
Hi, the link provided in the below thread is not working, any way to find the information or a new URL/link: FAQs About Organizing a Data…
Unable to connect the ADLS to snowflake
I am not able to connect the ADLS to snowflake. I requested azure to create the connection but I don't see the user that starts with Snowflake... when I create a role in IAM.
Copy Activity very slow for Azure data factory
Hi All, we are having issue where copy activity from Azure Table storage to data factory is taking over 50mins on daily basis. it used to take 15-20mins until a week back. What's the source of the slow copy activity? Azure Table storage What's the sink…
How to overwrite a table from SQL Server to SQL Dedicated Pool using Copy Data Activity in Azure Synapse Analytics
I am trying to use the Copy Data Activity in Azure Synapse Analytics to copy data from SQL Server to SQL Dedicated Pool. However, I want to ensure that every time I run the Copy Data Activity, the table in SQL Dedicated Pool is overwritten. How can I…
can abac be applied to a data lake?
Can abac be applied to a data lake storage account or does it have to be rbac with potential ACL?
Slow performance reading parquet files from Data Lake Gen2
I'm using Azure Data Lake Gen2 to store data in parquet file format. I have split the data using partitions by year, month and day to benefit from the filtering functionality. I'm reading the data from python directly (no Spark or Synapse here) using the…
I got this error when debug my copy data on azure df 7cf2da46-b65b-4098-b8ec-d9c7858c190e
after connecting my Azure data factory to MsSqL, I tried to copy my data into azure data factory and I got this error 7cf2da46-b65b-4098-b8ec-d9c7858c190e
Need to use service principal in datalake connection string instead of storage key
Hello Team, I hope you are doing well. Currently in function app customer using connection string which is consist of storage key but now he wants to use service principal in connection string please let me know how to format connection string: current…
Can't create a datalake storage account Gen2 when creating an azure synapse resource
When trying to create a new data lake gen2 storage account (while creating a azure Synapse resource), I keep on getting an error message that the account name can't be verified and that I need to try again. I've waited for a couple of hours and have…
How to improve speed of data transfer of RO_MOUNT Data asset in Azure ML Job?
Is there a reason for this slow transfer speed during job run? This slowdown is causing GPU utilization to tank from 85-90% (if files are on disk during test runs) to 12-15% during actual job training with complete data being transferred through…
Dealing with sensitive data in azure
I am ingesting data RaaS and APIs for other source systems, which must be accessed through API and HTTPS or HTTP. ADF does not encrypt the data, and data masking using a dataflow for huge amounts of data is hard. We are dealing with a lot of sensitive…
Lost access to data storage containers
I created a storage account last week verticaltbr. I have created 2 storage containers and uploaded the contents. But today I can't access these contents on Azure portal and Storage Explorer. I can, on the other hand, create new containers but can't…
Azure storage Adls blob container set read only
I have storage account with hierarchical namespace enabled. Now I have container "test" where I have uploaded one file, I need this file to be read only so that no one can update it. Now problem is "test" container inherit rbac role…
Azure data factory and secure access
What are the secure options available to prevent ADF and ADLS from cyber threats and other security vulnerabilities besides accessing the services through private link and express routes and using vnet and private endpoints? Please suggest a holistic…
Got error code 403 when try to access my Synapse workspace created within a virtual network
I got a 403 error when access my Synapse Workspace created in a managed virtual network within a jumphost inside my virtual network. This is the network settings of my Synapse Analytics workspace My bicep files have help me set up some private…
Cost of ADLS Gen2 while enabling Az synpase link for Cosmos database
Hi Team, I am exploring the Azure synapse link for Azure cosmos database, but when I enable the Azure synapse link then it require the azure synapse workspace to be created along with Azure data lake storage Gen2. So, below are my queries around it. …
Data is missing from management API using service principle. ex: Advisor Recommendations
I have a requirement to load datasets from management.Azure.com (Management API), for example: Advisor Recommendations, Resources I have created a service principal in app registration and have been able to obtain the access token. However, when loading…
Restore ADLS Gen2 (HNS) Account / Containers
Hello, I am using an ADLS Gen2 / HNS storage account or data lake. I have enabled blob versioning and soft delete. That is helpful to recover deleted blobs. However I want to selectively restore my storage account as well as container to a point in…