Troubleshoot data loss in Azure Managed Redis (preview)
This article discusses how to diagnose actual or perceived data losses that might occur in Azure Managed Redis (preview).
Note
Several of the troubleshooting steps in this guide include instructions to run Redis commands and monitor various performance metrics. For more information and instructions, see the articles in the Additional information section.
Partial loss of keys
Azure Managed Redis doesn't randomly delete keys after they've been stored in memory. However, it does remove keys in response to expiration policies, eviction policies, and to explicit key-deletion commands. You can run these commands through the CLI.
Keys that have been written to the primary node in an Azure Managed Redis instance also might not be available on a replica right away. Data is replicated from the primary to the replica in an asynchronous and non-blocking manner.
If you find that keys have disappeared from your cache, check the following possible causes:
| Cause | Description |
|---|---|
| Key expiration | Keys are removed because of time-outs set on them. |
| Key eviction | Keys are removed under memory pressure. |
| Key deletion | Keys are removed by explicit delete commands. |
| Async replication | Keys are not available on a replica because of data-replication delays. |
Key expiration
Azure Managed Redis removes a key automatically if the key is assigned a time-out and that period has passed. For more information about Redis key expiration, see the EXPIRE command documentation. Time-out values also can be set by using the SET, SETEX, GETSET, and other *STORE commands.
To get stats on how many keys have expired, use the INFO command. The Stats section shows the total number of expired keys. The Keyspace section provides more information about the number of keys with time-outs and the average time-out value.
# Stats
expired_keys:46583
# Keyspace
db0:keys=3450,expires=2,avg_ttl=91861015336
You can also look at diagnostic metrics for your cache, to see if there's a correlation between when the key went missing and a spike in expired keys. See the Appendix of Debugging Redis Keyspace Misses for information about using keyspace notifications or MONITOR to debug these types of issues.
Key eviction
Azure Managed Redis requires memory space to store data. It purges keys to free up available memory when necessary. When the used_memory or used_memory_rss values in the INFO command approach the configured maxmemory setting, Azure Managed Redis starts evicting keys from memory based on cache policy.
You can monitor the number of evicted keys by using the INFO command:
# Stats
evicted_keys:13224
You can also look at diagnostic metrics for your cache, to see if there's a correlation between when the key went missing and a spike in evicted keys. See the Appendix of Debugging Redis Keyspace Misses for information about using keyspace notifications or MONITOR to debug these types of issues.
Key deletion
Redis clients can issue the DEL or HDEL command to explicitly remove keys from Azure Managed Redis. You can track the number of delete operations by using the INFO command. If DEL or HDEL commands have been called, they'll be listed in the Commandstats section.
# Commandstats
cmdstat_del:calls=2,usec=90,usec_per_call=45.00
cmdstat_hdel:calls=1,usec=47,usec_per_call=47.00
Async replication
Any Azure Managed Redis instance with high availability enabled is configured with a primary node and at least one replica. Data is copied from the primary to a replica asynchronously by using a background process. The redis.io website describes how Redis data replication works in general. For scenarios where clients write to Redis frequently, partial data loss can occur because replication is not guaranteed to be instantaneous. For example, if the primary goes down after a client writes a key to it, but before the background process has a chance to send that key to the replica, the key is lost when the replica takes over as the new primary.
Major or complete loss of keys
If most or all keys have disappeared from your cache, check the following possible causes:
| Cause | Description |
|---|---|
| Key flushing | Keys have been purged manually. |
| Redis instance failure | The Redis server is unavailable. |
Key flushing
Clients can call the FLUSHDB or FLUSHALL command to remove all keys from the Redis instance. To find out whether keys have been flushed, use the INFO command. The Commandstats section shows whether either FLUSH command has been called:
# Commandstats
cmdstat_flushall:calls=2,usec=112,usec_per_call=56.00
cmdstat_flushdb:calls=1,usec=110,usec_per_call=52.00
Redis instance failure
Redis is an in-memory data store. Data is kept on the physical or virtual machines (VM) that host the Redis cache. Azure Managed Redis caches offer high resiliency against data loss by providing Zone Resilient caches by default. When the primary shard in such a cache fails, the replica shard takes over to serve data automatically. These VMs are located on separate domains for faults and updates, to minimize the chance of both becoming unavailable simultaneously. If a major data center outage happens, however, the VMs might still go down together. Your data will be lost in these rare cases.
Consider using Redis data persistence and geo-replication to improve protection of your data against these infrastructure failures.
Additional information
These articles provide more information on avoiding data loss: