Performance scenarios
This unit presents performance scenarios when you migrate data.
Optimize data migration performance by using correct data entity and configuration
Loading large volumes of data by using the Data management framework takes longer than the defined cutover time. You can migrate data only during the defined cutover window.
Patterns
One way to optimize data migration is to use the correct entities for migration. You should use entities with fewer data sources so that they’re lighter.
For scenarios with a high volume of data, use entities that support multithreading. CustomersV3 seems to be the entity that’s used most frequently, but it doesn’t support multithreading. Alternatives to the CustomersV3 entity are Customer definition and Customer details, which should be used in high volume scenarios because they support multithreading.
- Use entities that support multithreading, such as Customer definition (CustCustomerBase) and Customer details (CustCustomerDetailsV2).
- Disable change tracking for data migration.
- Use bulk import by enabling set-based processing (not all entities support set-based processing).
- Review data migration system and entity configurations.
- Review the upcoming changes and new entities.
Anti-patterns
Avoid the following anti-patterns when you’re optimizing data migration performance by using correct data entity and configuration:
- Enabling change tracking
- Using a composite data entity
- Turning on X++ business validations
- Using SQL to migrate data
Optimize data migration performance through bundling and iterative performance testing
During performance testing the customer notices that loading a large volume of data in batch mode (Import in batch) is taking the same amount of time as the asynch import process (Import now). The business determines it cannot complete the data migration activities within the defined cutover time.
Patterns
Use the following approaches when optimizing data migration performance through bundling and iterative performance testing:
- Use Import in batch to take advantage of multithreading capabilities.
- Use Import threshold record count and Import task count to use batch bundling.
- Determine the appropriate number of threads, threshold record count, import task count, and so on, based on data size and number of available batch threads.
- Collect and compare information regarding each test because data migration performance testing is an iterative process. This approach will help you determine the optimal configuration for a specific entity.
- Review Optimize data migration for finance and operations apps.
Anti-patterns
Avoid the following anti-patterns when optimizing data migration performance through bundling and iterative performance testing:
- Importing large datasets interactively
- Starting without a plan for data migration performance testing and tuning
- Using production for data migration testing
- Only planning for a single iteration data migration dry run
Warm up
Data loading slows down after the customer has imported a large volume of data into the initial set of legal entities. Performance keeps deteriorating for the same data entity for subsequent data loads in other legal entities.
Patterns
Use the following approaches when warming up the system for subsequent data loads in other legal entities:
- Plan to clean up the history by using staging clean-up, which is available on the Data management workspace.
- Review the number of records in each file and investigate if smaller files will improve overall performance by allowing SQL to select a new query plan.
Anti-patterns
Avoid the following anti-patterns when warming up the system for subsequent data loads in other legal entities:
- Lacking a review of performance telemetry, such as indexes, query plans, and more
- Using a tier-1 environment for data migration, because:
- Contact emails and more will be stripped.
- The architecture of a cloud-hosted tier 1 environment and a tier 2+ environment is different and therefore performance is not comparable.
Optimize the number sequence approach
The customer is importing an exceptionally large volume of journal balances. The journal name is defined as In connection with balance where, whenever a voucher is balanced, a new voucher number is assigned.
For more information, see Importing vouchers by using the General journal entity. Use the Set-based processing setting; the General Journal entity uses Set-based processing by default. Make sure that this option is turned on.
Consider the following factors for this scenario:
- The General journal entity uses the Set-based processing setting. When you use that setting on the General journal entity, you’ll need to provide the voucher number in the imported file. Every transaction in the General journal is assigned the voucher number that’s provided in the imported file, even if the voucher isn’t balanced.
- With set-based processing, if you want to use the voucher numbers from the system and the journal name that’s used for imports, set the Number allocation at posting option to Yes.
- When you don't use the Set-based processing setting, you don’t need to provide a voucher number in the imported file. The voucher numbers are created during import, based on the setup of the journal name (One voucher only, In connection of balance, and so on). For example, if the journal name is defined as In connection with balance, then the first line will receive a new default voucher number. Then, the system will evaluate the line to determine whether the debits equal the credits. If an offset account exists on the line, the next line that’s imported will receive a new voucher number. If no offset account exists, the system will evaluate whether the debits equal the credits as each new line is imported.
- The Auto-generated setting for the Voucher number field isn't supported.
Patterns
Use the following best practices when you’re optimizing the number sequence approach:
- Where possible include the necessary number sequence values directly in the import file. After the import is complete update the next number in the number sequence so it aligns with the data that has been migrated.
- If number sequences are required ensure that Preallocation is enabled and the value of Quantity of numbers is set to an appropriate value
Anti-pattern
When you’re optimizing the number sequence approach, don’t use a continuous number sequence unless it’s a legal requirement.
Keep data migration separate from dual-write
The customer uses many different Dynamics 365 products. They plan to use the dual-write infrastructure to integrate data from Microsoft Dataverse to finance and operations apps and vice versa. For example, the customer will need to migrate the Accounts data into Dataverse, and that same Accounts data will need to move as Customers data into Dynamics 365 Finance at go-live.
Patterns
Consider the following information when you’re keeping data migration separate from dual-write:
- Dual-write infrastructure is designed to integrate data in near real-time between Dataverse and finance and operations apps.
- You can use Initial Write/Initial Sync to synchronize only low-volume setup and reference data.
- Initial Write/Initial Sync isn’t a replacement for large-volume master or transactions data migration.
- Dual-write infrastructure uses OData integration through data entities in finance and operations apps. It’s not suitable for data migration scenarios where bulk data import is needed.
- Customers should migrate large volumes of data separately in Dataverse and finance and operations apps before enabling dual-write.
Anti-patterns
Avoid the following anti-patterns when keeping data migration separate from dual-write:
- Using OData for data migration
- Using dual-write as a data migration technology
Use the Recurring Integrations Scheduler (RIS) for automating data movement
The business has a substantial number of legal entities (for example, over 50) in Dynamics 365. The customer has created folders for each legal entity (on-premises). The customer extracts data from the legacy application and then generates one file per data entity, for each legal entity, which is then placed in the corresponding legal entity folder. The customer has considered creating a data project and manually selecting the data file for each legal entity, but it’s a tedious process and prone to error.
Patterns
Follow these best practices when using the Recurring Integrations Scheduler (RIS) for automating data movement:
- Set up input folders for each legal entity.
- Use data projects or the package template option within RIS to create data packages (zip file) with data and meta data.
- Set up a recurring job in RIS and manage the RIS configuration (import/export).
- Alternatively, look at the data task automation functionality.
Anti-pattern
When using the Recurring Integrations Scheduler (RIS) for automating data movement, don’t develop X++ code to manage the files.
Follow performance due diligence in the environment
The customer is experiencing performance issues during data migration into finance and operations apps.
Patterns
Use these best practices when following performance due diligence in the environment:
- Make sure that recurring imports/exports are limited. Having a high recurrence rate can affect system performance.
- Disable deployment job-pushing data to the Entity Store Database (AXDW) and Bring Your Own Database (BYOD) jobs while you are performing large imports.
- Ensure that batch jobs are spaced out enough to allow time for finishing before the next job starts in the queue.
- Ensure that data migration packages don’t include V1 and V2 entities for the same system table. If a batch job includes an import of both, it can put extra/unnecessary strain on the tables.
- Ensure the system is not used for other purposes during data migration.
- Disable external automation, such as Microsoft Power Automate, Microsoft Logic Apps, and so on, during data migration.
Anti-patterns
Avoid these anti-patterns when you’re following performance due diligence in the environment:
- Setting a high recurrence rate on imports and exports
- Including V1 and V2 entities for the same system table