Introduction
One of the biggest challenges organizations face today is that it's difficult to understand who your customers are and what is important to them. Customer data is often scattered across multiple data silos. For example, the transaction system used by your organization's retail website might operate independently from the point-of-sale system used at your retail stores. Additionally, your organization's support and call centers might use a separate application designed for call center scenarios. With customer data scattered across different systems, how can you gain a clear picture of who that customer is?
Dynamics 365 Customer Insights - Data helps your organization unlock insights to build a deeper understanding of your customers. Customer Insights - Data ingests data from all your different data sources and unifies them into a single customer profile. From within the customer profile, you can track data across different demographics, and identify trends based on key data you want to track.
Gaining a 360-degree view of your customers starts by ingesting the specific data that you need to work with across all your organizations' different data silos.
These sources might include:
Transactional data such as point-of-sale systems
Observational data such as product testing application
Behavioral sources such as Customer Service, Sales, etc., applications
Any data source where customer-related data is stored
Extract, transform, load (ETL)
Before applications that work with data from different sources, such as Customer Insights - Data can use it. The data needs to be collected and refined into something that the application can consume. This process is critical to ensuring that the data is used appropriately. This process is referred to as ETL, which stands for Extract, Transform, Load.
As the name states, there are three steps that make up the ETL process and enable data to be integrated from source to destination.
Data extraction: Extracts raw data from a data source such as legacy systems, cloud environments, CRM/ERP applications, data warehouses, etc.
Data transformation: Improves data quality and accessibility through processes such as data cleansing, standardizations, sorting, etc.
Data loading: Loads the data into its new location.
Depending on the data ingestion method that you're going to be using, ETL might be done as part of the ingestion process, or prior to ingesting the data. As we examine the different ingestion methods available, we look at when ETL should be done.
Ingesting data
Data ingested into Customer Insights - Data is stored in a data set. A data set is a table that contains the data you want to use. It might be profile-related data coming from a CRM application, or purchases coming from a Point of Sale (PoS) system. Data sets are stored in data sources, which are organizational units that help organize your data sets and make it easier to group/find them. For example, you might define a data source called "eCommerce" to ingest data from an ecommerce application. The ecommerce application contains both customer profile and customer purchase data. A data set exists for each of them in the data source.
Depending on the volume of data you want to ingest, and where the data is located, you need to decide which ingestion method to use.
Customer Insights - Data provides multiple data source options to choose from:
Azure Data Lake - Used when you want to connect to an Azure Data Lake Storage Gen 2 Account. For more information, see Azure Data Lake.
Azure Data Lake Delta tables - Used when you want to attach Delta tables stored in Azure Data Lake storage optimized for efficient processing on incremental data changes. For more information, see Azure Data Lake tables.
Microsoft Dataverse - Used when you want to connect to data sets in the Dataverse data lake. For more information, see Dataverse.
Microsoft Power Query - Used when you want to connect to data such as Microsoft Dataverse, Azure Blobs, OData sources, etc. For more information, see Power Query Connectors.
Throughout the remaining units, we examine each of these options in more detail.