How materialized views work in Azure Data Explorer regarding cache policy?

Question

Hey everyone,

I am trying to implement some materialized views within my ADX cluster. I need to backfill the data for the last year and I understood that I must increase my hot cache policy for it to be able to. I have two questions regarding that: is it possible to change the hot cache policy for the sake of the materialized view, and then change it again to a lesser value? Would that impact the materialized view? Also, what happens with the aggregated data in the materialized view once the data used for the aggregation moves into cold cache, does that impact it somehow?

Answer 1

Hi @Marcos Olmedilla
Thank you for using Microsoft Q&A platform and thanks for posting your question here.

Changing Hot Cache Policy for Materialized Views

Yes, you can change the hot cache policy for the sake of the materialized view and then change it again to a lesser value. However, it's essential to understand the implications of doing so.

When you increase the hot cache policy, ADX will store more data in the hot cache, which can improve query performance. If you then decrease the hot cache policy, ADX will evict less frequently accessed data from the hot cache, which can lead to slower query performance.

Regarding materialized views, changing the hot cache policy will not directly impact the materialized view itself. However, it may affect the performance of queries that rely on the materialized view.

Aggregated Data in Materialized Views and Cold Cache

When the data used for aggregation in a materialized view moves into cold cache, it can impact the materialized view in the following ways:

1. Query Performance: If the aggregated data is stored in cold cache, queries that rely on the materialized view may experience slower performance.
2. Data Freshness: If the data used for aggregation is no longer in hot cache, the materialized view may not reflect the latest changes to the underlying data.
3. Materialized View Maintenance: ADX will continue to maintain the materialized view, even if the underlying data is in cold cache. However, the maintenance process may be slower due to the colder storage tier.

To mitigate these effects, you can consider the following strategies:

• Increase the hot cache policy: Temporarily increase the hot cache policy to ensure that the aggregated data remains in hot cache for a longer period.
• Use a larger cache: Consider increasing the cache size to accommodate more data in hot cache.
• Optimize materialized view maintenance: Schedule materialized view maintenance during periods of low query activity to minimize performance impacts.

For more information can refer the below articles.

https://learn.microsoft.com/en-us/kusto/management/materialized-views/materialized-view-policies?view=microsoft-fabric https://learn.microsoft.com/en-us/kusto/management/cache-policy?view=azure-data-explorer&preserve-view=true

https://learn.microsoft.com/en-us/kusto/management/alter-materialized-view-cache-policy-command?view=microsoft-fabric

I hope this helps. Please let me know if you have any questions!