NTFS Misreports Free Space (Part 3)
It’s been a while since my last post on this topic, and I wanted to take some time to update everyone on a cool new feature in Windows Server 2012 R2 and Windows 8.1. Today we declare part 1 and part 2 of this blog as obsolete - at least for Windows Server 2012 R2 and Windows 8.1 users.
The latest fsutil.exe now allows for the creation of an allocation report which summarizes how all of your disk space is being used by NTFS. This new fsutil.exe functionality is implemented though some new file system controls that only exist on Windows Server 2012 R2 and Windows 8.1, so the binary is not portable to previous versions of Windows.
USAGE: fsutil volume allocationreport X:
X: is the drive letter of an NTFS volume on your system.
Allocation Report
The allocation report gives a summary of total reserved, free, and allocated clusters. Reserved clusters are clusters that NTFS reserves just in case it needs to allocate space for a critical operation (like expanding a compressed file or extending the $MFT). If you’re experiencing insufficient disk space errors on a volume that has plenty of free space, the issue could be caused by opening many compressed NTFS files at the same time. Please refer to Understanding Ntfs Compression for more information on how to troubleshoot this.
Allocation report:
Total clusters : 244100351 (999835037696 bytes)
Free clusters : 232507563 (952350978048 bytes)
Reserved clusters : 18352 (75169792 bytes)
Total allocated : 47484059648 bytes
System Files
If you suspect that there’s something you can’t see that’s taking up disk space, check the System Files section to see how much disk space is used by the system. In this example, I have 884,703,232 bytes in use by NTFS metadata, and the breakdown of each system file’s usage is outlined below. For details on each system file type, refer to https://blogs.technet.com/b/askcore/archive/2009/12/30/ntfs-metafiles.aspx.
System files : Count: 29. Total allocated: 884703232 bytes.
$Mft : File ID 0x0001000000000000. Total allocated: 238063616 bytes.
$MftMirr : File ID 0x0001000000000001. Total allocated: 4096 bytes.
$LogFile : File ID 0x0002000000000002. Total allocated: 67108864 bytes.
$Volume : File ID 0x0003000000000003. Total allocated: 0 bytes.
$AttrDef : File ID 0x0004000000000004. Total allocated: 4096 bytes.
Root folder : File ID 0x0005000000000005. Total allocated: 8192 bytes.
$Bitmap : File ID 0x0006000000000006. Total allocated: 30515200 bytes.
$Boot : File ID 0x0007000000000007. Total allocated: 8192 bytes.
$BadClus : File ID 0x0008000000000008. Total allocated: 0 bytes.
$Secure : File ID 0x0009000000000009. Total allocated: 1855488 bytes.
$UpCase : File ID 0x000a00000000000a. Total allocated: 131072 bytes.
$Extend : File ID 0x000b00000000000b. Total allocated: 0 bytes.
$ObjId : File ID 0x0001000000000019. Total allocated: 24576 bytes.
$Quota : File ID 0x0001000000000018. Total allocated: 0 bytes.
$Reparse : File ID 0x000100000000001a. Total allocated: 786432 bytes.
$UsnJrnl : File ID 0x0002000000012f66. Total allocated: 34144256 bytes.
$RmMetadata : File ID 0x000100000000001b. Total allocated: 0 bytes.
$Repair : File ID 0x000100000000001c. Total allocated: 94371840 bytes.
$Txf : File ID 0x000100000000001e. Total allocated: 4096 bytes.
$TxfLog : File ID 0x000100000000001d. Total allocated: 4096 bytes.
$Tops : File ID 0x000100000000001f. Total allocated: 396623872 bytes.
$TxfLog.blf : File ID 0x0001000000000020. Total allocated: 65536 bytes.
Other system files : Count: 4. Total allocated: 0 bytes.
Other system files under $Txf folder:
Count : 1
Total allocated : 8192 bytes.
Other system files under $TxfLog folder:
Count : 2
Total allocated : 20971520 bytes.
System Volume Information
If the usage in System Volume Information is higher than expected, the issue is likely to be caused by storage of diff areas for VSS volume shadow copies. Deleting the volume shadow copies with VSSAdmin or Diskshadow will return the free space. System Volume Information is also the home of the chunk store used by NTFS deduplication.
System Volume Information : Total allocated: 5366915072 bytes.
Files : Count: 18. Total allocated: 5366882304 bytes.
Folders : Count: 7. Total allocated: 32768 bytes.
User Folders
It costs something to maintain the folder structure of a volume, and the user folders section summarizes the overall cost. Within this section is also a summary of how many NTFS compressed folders exist. As you can see below, I have 145 folders with a compressed attribute flag but the total number of compressed bytes is zero. I puzzled over the idea of zero compressed bytes until I discovered that this measurement is of how many compressed bytes exist in the context of folder indexes, and indexes are never compressed. Only user data streams are compressed natively by NTFS.
User folders : Count: 23101. Total allocated: 77889536 bytes.
Default streams : 4689
Allocated : 4689
Total allocated : 77885440 bytes.
Named streams : 7
Allocated : 0
Total allocated : 0 bytes.
Local metadata streams : 95566
Allocated : 1
Total allocated : 4096 bytes.
Within these folders there are:
Compressed : 145
Total allocated : 0 bytes
Total size : 0 bytes.
Savings : 0.00 %
Sparse : 0
Total allocated : 0 bytes
Total size : 0 bytes.
Savings : 0.00 %
Encrypted : 0
Total allocated : 0 bytes
With named streams : 7
Compressed : 0
Sparse : 0
Encrypted : 0
With no allocation : 18412
User Files
In the user files section, we have a total of all user files and the compression statistics to show how much space is being saved by native NTFS compression. There is also a nice summary of alternate named stream usage (ANS). ANS allocations do not show up in DIR or Explorer, so this is a quick and easy way to see exactly how your named streams are affecting overall disk usage. On my volume, I had 3115 files with named streams and zero bytes were allocated. This seems to be another paradox, but there’s a logical explanation for what’s happening. If a file has a named stream and the stream size is small enough for it to be resident, then the stream lives in the file’s MFT record (which is accounted in this report as part of $Mft : File ID 0x0001000000000000. Total allocated: 238063616 bytes.).
User files : Count: 94128. Total allocated: 41154551808 bytes.
Default streams : 94128
Allocated : 72123
Total allocated : 41087229952 bytes.
Named streams : 4637
Allocated : 4562
Total allocated : 66740224 bytes.
Local metadata streams : 333248
Allocated : 142
Total allocated : 581632 bytes.
Within these files there are:
Compressed : 2006
Total allocated : 374972416 bytes
Total size : 816416626 bytes.
Savings : 54.07 %
Sparse : 1519
Total allocated : 1572864 bytes
Total size : 273374082 bytes.
Savings : 99.42 %
Encrypted : 0
Total allocated : 0 bytes
With named streams : 3115
Compressed : 0
Sparse : 0
Encrypted : 0
With no allocation : 20485
As you can see, this new functionality in fsutil makes it easier and faster to determine what is using space on an NTFS volume.