How to fix nested split issue in Azure data factory Pipeline expression builder?
Hi there, I'm trying to make use of hierarchical namespaces for Azure data lake storage gen2. When I'm dynamically setting the file structure in the Pipeline expression builder in the ADF, the following error is returned: Cannot fit unknown into the…
ErrorCode=AdlsGen2OperationFailed...Failure happened on 'Sink' side.
Getting the below error while copying the data to ADLS from SAP Hana, however this error is random to tables failing daily. When I try load each of the failed tables from the same source and with the same file name then there is no issue and it works as…
Monitor AutoResolve Integration Runtime in Azure Data Factory
Hi all I need to monitor the Integration Runtime performance for a pipeline that uses stored procedure activity to load data from ADLS to Snowflake. The stored procedure in this case resides in snowflake itself and it’s called from Snowflake Tenant to…
I am integrating the dataflow activity which removes duplicates from the csv file to the existing pipeline i have. I need a help to configure the parameters.
I created a dataflow activity which removes duplicates from the source files( csv formatted files). So now i integrated this dataflow activity to existing pipeline i have. but i am not able to parametrize the source files. It would be helpful if i get…
How to write the query on Azure to get the information of particular SPN linked with all the folders present in containers..
Hi, As there are multiple folders are preset into Azure storage containers, each folders of containers linked with some SPN. I am looking for the script to create to find, which SPN associate with which all folders?
Azure Data Factory: No data when trying to preview data source
Azure Data Factory: No data when trying to preview data source. Data is present at source and even when we minimize the browser to 30% we are able to see data in data preview but when we are doing normal data preview we are unable to see any data in all…
ADF/ Synapse Activity times out after 12 hours
Hello, I have a Notebook activity in one of my pipelines that is expected to run longer than 12 hours. Below we configured the timeout to the max value, 7 days. But the activity would still timeout after 12 hours. Is there a way for the activity…
D365 Outbound Marketing Data sync to AdLS Gen2 cannot be synced inside a folder structure but only on conatiner.
I wanted to copy Market Outbound Data from CRM like email sent, delivered, Email clicked, Email Opened , etc from the marketing campaign to ADLS Gen 2 using Data Factory but seems like I cannot use ADF to do that as they are calculated live. Also, I…
Stream RealTime data from an On Prem Microsoft Server to a Azure Data Lake.
There is a need for Real Time data to be collected from a On Prem SQL Server. This means ADF is not suitable because it only does batch. Azure Streaming Analytics only connects to Azure SQL Server and not on prem. Has anybody else encountered this and…
Why are access tier options greyed out for Azure Data Lake Gen 2 storage?
When attempting to change the access tier of a file stored in Azure Data Lake Gen 2, all the access tier options are greyed out. What could be causing this issue and what are the possible solutions?
generate SAS tokens from serviceprincipal credentials
I am working on to create java client that generates sas tokens for the given service principal credentials. I am taking a reference from…
Unable to create a directory named space character using ADLS Gen2 REST API
I am unable to create a directory in "Azure Data Lake Storage Gen2" that is named as just a single space character, despite reviewing the documentation and there being no indication this is a disallowed name. My primary access to "Azure…
How to Automate Large .SAV File to Parquet Conversion in Azure?
I'm using Azure ADLS as our primary storage, Azure Data Factory (ADF) for data transformations, and Power BI for reporting and visualization. I have a large .SAV file (200-300 MB, containing 2-4 million rows) stored in Azure Data Lake Storage (ADLS). To…
Read AppServiceConsoleLogs and push to Blob Storage container/table
Hello Team, we have a webapp python Django based. We are generating custom logs for user actions using code below. Settings.py: LOGGING = { "version": 1, "disable_existing_loggers": False, "formatters": { …
Change the Azure synapse workspace default storage account.
Hi Team, We created an Azure Synapse workspace with a default storage account associated with it. However, due to various reasons, we are not using the storage account tied to the Synapse workspace. Instead, we have created another ADLS Gen2 storage…
how to fix this error The request content was invalid and could not be deserialized: Required property 'name' not found in 'sku'
getting error while synapse workspace deployment
Preview features like AllowDataSharingInHeroRegion and AllowDataSharing are in pending status since a month
Hi team, Azure subscription preview features are like AllowDataSharingInHeroRegion and AllowDataSharing are in pending status since a month. We tried to unregister and re-register but still no luck. Could you please help us on this? Thanks, and…
How to extract ProcessInstanceHistory object records using new Salesforce connector from ADF
We are trying to extract the data from Salesforce for the ProcessInstanceHistory object through the new Salesforce connector. There is a limitation for the new connector as it doesn't allow inner SOQL query for Bulk API. We have tried using REST API but…
Can i recover blob file from Azure Storage blob in container, if soft delete is not is turned on
ive accidentaly deleted nessecery file (.xlsx extension), is there any way to recover it?
Ingress and Egress cost in azure
I have gone through below thread: https://learn.microsoft.com/en-us/answers/questions/892660/will-the-ingress-egress-costs-of-a-azure-storage-a but still confused and clear about the ingress and egress costs of ADLS, and what parameters it takes to…