Monitor and Report for Data Deduplication
Applies To: Windows Storage Server 2012, Windows Server 2012 R2, Windows Server 2012
Administrators can query the progress of a deduplication job, view the achieved space savings on the volume, and view the status of the deduplication process by using the Get-DedupStatus and Get-DedupVolume Windows PowerShell cmdlets.
Here is an example of Get-DedupStatus output:
PS C:\> Get-DedupStatus
FreeSpace SavedSpace OptimizedFiles InPolicyFiles Volume
-------------- ---------- -------------- ------------- ------
280.26 GB 529.94 GB 36124 36125 X:
151.26 GB 84.19 GB 43017 43017 Z:
Here is an example of Get-DedupStatus verbose output for drive X: in the previous example by using the |fl pipe:
PS C:\> Get-DedupStatus |fl
Volume : X:
VolumeId : \\?\Volume{1234567890}\
Capacity : 465.76 GB
FreeSpace : 280.26 GB
UsedSpace : 185.5 GB
UnoptimizedSize : 715.44 GB
SavedSpace : 529.94 GB
SavingsRate : 74 %
OptimizedFilesCount : 36124
OptimizedFilesSize : 696.46 GB
OptimizedFilesSavingsRate : 76 %
InPolicyFilesCount : 36125
InPolicyFilesSize : 704.44 GB
LastOptimizationTime : 5/23/2012 2:45:09 PM
LastOptimizationResult : 0x00000000
LastOptimizationResultMessage : The operation completed successfully.
LastGarbageCollectionTime : 5/19/2012 2:45:51 AM
LastGarbageCollectionResult : 0x00000000
LastGarbageCollectionResultMessage : The operation completed successfully.
LastScrubbingTime : 5/19/2012 3:45:21 AM
LastScrubbingResult : 0x00000000
LastScrubbingResultMessage : The operation completed successfully.
Here is an example of Get-DedupVolume output:
PS C:\> Get-DedupVolume
Enabled SavedSpace SavingsRate Volume
------- ---------- ----------- ------
True 529.94 GB 74 % X:
Here is an example of Get-DedupVolume verbose output for drive X: in the previous example by using the |fl pipe:
PS C:\> Get-DedupVolume |fl
Volume : X:
VolumeId : \\?\Volume{1234567890}\
Enabled : True
DataAccessEnabled : True
Capacity : 465.76 GB
FreeSpace : 280.26 GB
UsedSpace : 185.5 GB
UnoptimizedSize : 715.44 GB
SavedSpace : 529.94 GB
SavingsRate : 74 %
MinimumFileAgeDays : 0
MinimumFileSize : 32768
NoCompress : False
ExcludeFolder :
ExcludeFileType :
NoCompressionFileType : {aac, aif, aiff, asf...}
ChunkRedundancyThreshold : 100
Verify : False
In addition to the status of a drive or volume, you can get detailed data about how deduplication processed the data by running the Get-DedupMetadata cmdlet:
PS C:\> Get-DedupMetadata
Volume : X:
VolumeId : \\?\Volume{1234567890}\
StoreId : {FDF037AE-4AAC-440F-84DF-A94A3D82DF7C}
DataChunkCount : 3314369
DataContainerCount : 173
DataChunkAverageSize : 52.48 KB
DataChunkMedianSize : 0 B
DataStoreUncompactedFreespace : 0 B
StreamMapChunkCount : 36575
StreamMapContainerCount : 16
StreamMapAverageDataChunkCount :
StreamMapMedianDataChunkCount :
StreamMapMaxDataChunkCount :
HotspotChunkCount : 10561
HotspotContainerCount : 1
HotspotMedianReferenceCount :
CorruptionLogEntryCount : 0
TotalChunkStoreSize : 167.2 G
The Get-DedupMetadata cmdlet shows the number of chunks in the chunk store, the number of StreamMaps, and the hotspot counts. (Hotspots are the most frequently-referenced chunks.)
Here is an example of a status report generated by Get-DedupSchedule:
PS C:\> Get-DedupSchedule
Enabled Type StartTime Days Name
------- ---- --------- ---- ----
True Optimization BackgroundOptimization
True GarbageCollection 2:45 AM Saturday WeeklyGarbageCollection
True Scrubbing 3:45 AM Saturday WeeklyScrubbing
Here is an example of Get-DedupSchedule verbose output in the previous example by using the |fl pipe:
PS C:\> Get-DedupSchedule |fl
Name : BackgroundOptimization
Enabled : True
Type : Optimization
Days :
Start :
DurationHours :
StopWhenSystemBusy : True
Memory : 25 %
Priority : Low
ScheduledTask : \Microsoft\Windows\Deduplication\BackgroundOptimization
Full :
ReadOnly :
Name : WeeklyGarbageCollection
Enabled : True
Type : GarbageCollection
Days : Saturday
Start : 3/22/2012 2:45:00 AM
DurationHours : 48
StopWhenSystemBusy : True
Memory : 50 %
Priority : Normal
ScheduledTask : \Microsoft\Windows\Deduplication\WeeklyGarbageCollection
Full : False
ReadOnly : False
Name : WeeklyScrubbing
Enabled : True
Type : Scrubbing
Days : Saturday
Start : 3/22/2012 3:45:00 AM
DurationHours : 0
StopWhenSystemBusy : True
Memory : 50 %
Priority : Normal
ScheduledTask : \Microsoft\Windows\Deduplication\WeeklyScrubbing
Full : False
ReadOnly : False
A note about File Explorer properties
When you use File Explorer to view the properties of a deduplicated file, you will see that Size shows the logical size of the file, and Size on Disk shows the true physical allocation to that file. Size on Disk is less than the actual file size because deduplication has moved the contents of the file to a common chunk store and replaced the original file with an NTFS reparse point stub and metadata. Use the Measure-DedupFileMetadata Windows PowerShell cmdlet to determine how much space could be freed if you deleted particular files or folders.
How do I tell if deduplication is keeping up with the rate of incoming data?
For most deployments, the default settings will be appropriate to process existing data and keep up with the incoming data. In some environments, you may want to monitor deduplication if additional jobs need to be scheduled. The easiest way is to use the Get-DedupStatus Windows PowerShell cmdlet to monitor the number of optimized files compared to the number of in-policy files. This enables you to see if all in-policy files are processed, for example:
PS X:\> Get-DedupStatus
FreeSpace SavedSpace OptimizedFiles InPolicyFiles Volume
--------- ---------- -------------- ------------- ------
222.14 GB 528.93 GB 36130 36126 X:
151.26 GB 84.19 GB 43017 43017 Z:
If the number of in-policy files is continuously rising faster than the number of optimized files, you might consider running additional jobs.
Monitoring the event log can also be helpful to understand deduplication events and status. To view deduplication events, in File Explorer, navigate to Applications and Services Logs, click Microsoft, click Windows, and then click Deduplication.
If the value LastOptimizationResult = 0x00000000 appears in the Get-DedupStatus |fl Windows PowerShell results, the entire dataset was processed by the previous optimization job.