Building the next generation file system for Windows: ReFS

Artikkeli
01/16/2012

We wanted to continue our dialog about data storage by talking about the next generation file system being introduced in Windows 8. Today, NTFS is the most widely used, advanced, and feature rich file system in broad use. But when you’re reimagining Windows, as we are for Windows 8, we don’t rest on past successes, and so with Windows 8 we are also introducing a newly engineered file system. ReFS, (which stands for Resilient File System), is built on the foundations of NTFS, so it maintains crucial compatibility while at the same time it has been architected and engineered for a new generation of storage technologies and scenarios. In Windows 8, ReFS will be introduced only as part of Windows Server 8, which is the same approach we have used for each and every file system introduction. Of course at the application level, ReFS stored data will be accessible from clients just as NTFS data would be. As you read this, let’s not forget that NTFS is by far the industry’s leading technology for file systems on PCs.

This detailed architectural post was authored by Surendra Verma, a development manager on our Storage and File System team, though, as with every feature, a lot of folks contributed. We have also used the FAQ approach again in this post.
--Steven

PS: Don't forget to track us on @buildwindows8 where we were providing some updates from CES.

In this blog post I’d like to talk about a new file system for Windows. This file system, which we call ReFS, has been designed from the ground up to meet a broad set of customer requirements, both today’s and tomorrow’s, for all the different ways that Windows is deployed.

The key goals of ReFS are:

Maintain a high degree of compatibility with a subset of NTFS features that are widely adopted while deprecating others that provide limited value at the cost of system complexity and footprint.
Verify and auto-correct data. Data can get corrupted due to a number of reasons and therefore must be verified and, when possible, corrected automatically. Metadata must not be written in place to avoid the possibility of “torn writes,” which we will talk about in more detail below.
Optimize for extreme scale. Use scalable structures for everything. Don’t assume that disk-checking algorithms, in particular, can scale to the size of the entire file system.
Never take the file system offline. Assume that in the event of corruptions, it is advantageous to isolate the fault while allowing access to the rest of the volume. This is done while salvaging the maximum amount of data possible, all done live.
Provide a full end-to-end resiliency architecture when used in conjunction with the Storage Spaces feature, which was co-designed and built in conjunction with ReFS.

The key features of ReFS are as follows (note that some of these features are provided in conjunction with Storage Spaces).

Metadata integrity with checksums
Integrity streams providing optional user data integrity
Allocate on write transactional model for robust disk updates (also known as copy on write)
Large volume, file and directory sizes
Storage pooling and virtualization makes file system creation and management easy
Data striping for performance (bandwidth can be managed) and redundancy for fault tolerance
Disk scrubbing for protection against latent disk errors
Resiliency to corruptions with "salvage" for maximum volume availability in all cases
Shared storage pools across machines for additional failure tolerance and load balancing

In addition, ReFS inherits the features and semantics from NTFS including BitLocker encryption, access-control lists for security, USN journal, change notifications, symbolic links, junction points, mount points, reparse points, volume snapshots, file IDs, and oplocks.

And of course, data stored on ReFS is accessible through the same file access APIs on clients that are used on any operating system that can access today’s NTFS volumes.

Key design attributes and features

Our design attributes are closely related to our goals. As we go through these attributes, keep in mind the history of producing file systems used by hundreds of millions of devices scaling from the smallest footprint machines to the largest data centers, from the smallest storage format to the largest multi-spindle format, from solid state storage to the largest drives and storage systems available. Yet at the same time, Windows file systems are accessed by the widest array of application and system software anywhere. ReFS takes that learning and builds on it. We didn’t start from scratch, but reimagined it where it made sense and built on the right parts of NTFS where that made sense. Above all, we are delivering this in a pragmatic manner consistent with the delivery of a major file system—something only Microsoft has done at this scale.

Code reuse and compatibility

When we look at the file system API, this is the area where compatibility is the most critical and technically, the most challenging. Rewriting the code that implements file system semantics would not lead to the right level of compatibility and the issues introduced would be highly dependent on application code, call timing, and hardware. Therefore in building ReFS, we reused the code responsible for implementing the Windows file system semantics. This code implements the file system interface (read, write, open, close, change notification, etc.), maintains in-memory file and volume state, enforces security, and maintains memory caching and synchronization for file data. This reuse ensures a high degree of compatibility with the features of NTFS that we’re carrying forward.

Underneath this reused portion, the NTFS version of the code-base uses a newly architected engine that implements on-disk structures such as the Master File Table (MFT) to represent files and directories. ReFS combines this reused code with a brand-new engine, where a significant portion of the innovation behind ReFS lies. Graphically, it looks like this:

NTFS.SYS = NTFS upper layer API/semantics engine / NTFS on-disk store engine; ReFS.SYS = Upper layer engine inherited from NTFS / New on-disk store engine

Reliable and scalable on-disk structures

On-disk structures and their manipulation are handled by the on-disk storage engine. This exposes a generic key-value interface, which the layer above leverages to implement files, directories, etc. For its own implementation, the storage engine uses B+ trees exclusively. In fact, we utilize B+ trees as the single common on-disk structure to represent all information on the disk. Trees can be embedded within other trees (a child tree’s root is stored within the row of a parent tree). On the disk, trees can be very large and multi-level or really compact with just a few keys and embedded in another structure. This ensures extreme scalability up and down for all aspects of the file system. Having a single structure significantly simplifies the system and reduces code. The new engine interface includes the notion of “tables” that are enumerable sets of key-value pairs. Most tables have a unique ID (called the object ID) by which they can be referenced. A special object table indexes all such tables in the system.

Now, let’s look at how the common file system abstractions are constructed using tables.

File structures

As shown in the diagram above, directories are represented as tables. Because we implement tables using B+ trees, directories can scale efficiently, becoming very large. Files are implemented as tables embedded within a row of the parent directory, itself a table (represented as File Metadata in the diagram above). The rows within the File Metadata table represent the various file attributes. The file data extent locations are represented by an embedded stream table, which is a table of offset mappings (and, optionally, checksums). This means that the files and directories can be very large without a performance impact, eclipsing the limitations found in NTFS.

As expected, other global structures within the file system such ACLs (Access Control Lists) are represented as tables rooted within the object table.

All disk space allocation is managed by a hierarchical allocator, which represents free space by tables of free space ranges. For scalability, there are three such tables – the large, medium and small allocators. These differ in the granularity of space they manage: for example, a medium allocator manages medium-sized chunks allocated from the large allocator. This makes disk allocation algorithms scale very well, and allows us the benefit of naturally collocating related metadata for better performance. The roots of these allocators as well as that of the object table are reachable from a well-known location on the disk. Some tables have allocators that are private to them, reducing contention and encouraging better allocation locality.

Apart from global system metadata tables, the entries in the object table refer to directories, since files are embedded within directories.

Robust disk update strategy

Updating the disk reliably and efficiently is one of the most important and challenging aspects of a file system design. We spent a lot of time evaluating various approaches. One of the approaches we considered and rejected was to implement a log structured file system. This approach is unsuitable for the type of general-purpose file system required by Windows. NTFS relies on a journal of transactions to ensure consistency on the disk. That approach updates metadata in-place on the disk and uses a journal on the side to keep track of changes that can be rolled back on errors and during recovery from a power loss. One of the benefits of this approach is that it maintains the metadata layout in place, which can be advantageous for read performance. The main disadvantages of a journaling system are that writes can get randomized and, more importantly, the act of updating the disk can corrupt previously written metadata if power is lost at the time of the write, a problem commonly known as torn write.

To maximize reliability and eliminate torn writes, we chose an allocate-on-write approach that never updates metadata in-place, but rather writes it to a different location in an atomic fashion. In some ways this borrows from a very old notion of “shadow paging” that is used to reliably update structures on the disk. Transactions are built on top of this allocate-on-write approach. Since the upper layer of ReFS is derived from NTFS, the new transaction model seamlessly leverages failure recovery logic already present, which has been tested and stabilized over many releases.

ReFS allocates metadata in a way that allows writes to be combined for related parts (for example, stream allocation, file attributes, file names, and directory pages) in fewer, larger I/Os, which is great for both spinning media and flash. At the same time a measure of read contiguity is maintained. The hierarchical allocation scheme is leveraged heavily here.

We perform significant testing where power is withdrawn from the system while the system is under extreme stress, and once the system is back up, all structures are examined for correctness. This testing is the ultimate measure of our success. We have achieved an unprecedented level of robustness in this test for Microsoft file systems. We believe this is industry-leading and fulfills our key design goals.

Resiliency to disk corruptions

As mentioned previously, one of our design goals was to detect and correct corruption. This not only ensures data integrity, but also improves system availability and online operation. Thus, all ReFS metadata is check-summed at the level of a B+ tree page, and the checksum is stored independently from the page itself. This allows us to detect all forms of disk corruption, including lost and misdirected writes and bit rot (degradation of data on the media). In addition, we have added an option where the contents of a file are check-summed as well. When this option, known as “integrity streams,” is enabled, ReFS always writes the file changes to a location different from the original one. This allocate-on-write technique ensures that pre-existing data is not lost due to the new write. The checksum update is done atomically with the data write, so that if power is lost during the write, we always have a consistently verifiable version of the file available whereby corruptions can be detected authoritatively.

We blogged about Storage Spaces a couple of weeks ago. We designed ReFS and Storage Spaces to complement each other, as two components of a complete storage system. We are making Storage Spaces available for NTFS (and client PCs) because there is great utility in that; the architectural layering supports this client-side approach while we adapt ReFS for usage on clients so that ultimately you’ll be able to use ReFS across both clients and servers.

In addition to improved performance, Storage Spaces protects data from partial and complete disk failures by maintaining copies on multiple disks. On read failures, Storage Spaces is able to read alternate copies, and on write failures (as well as complete media loss on read/write) it is able to reallocate data transparently. Many failures don’t involve media failure, but happen due to data corruptions, or lost and misdirected writes.

These are exactly the failures that ReFS can detect using checksums. Once ReFS detects such a failure, it interfaces with Storage Spaces to read all available copies of data and chooses the correct one based on checksum validation. It then tells Storage Spaces to fix the bad copies based on the good copies. All of this happens transparently from the point of view of the application. If ReFS is not running on top of a mirrored Storage Space, then it has no means to automatically repair the corruption. In that case it will simply log an event indicating that corruption was detected and fail the read if it is for file data. I’ll talk more about the impact of this on metadata later.

Checksums (64-bit) are always turned on for ReFS metadata, and assuming that the volume is hosted on a mirrored Storage Space, automatic correction is also always turned on. All integrity streams (see below) are protected in the same way. This creates an end-to-end high integrity solution for the customer, where relatively unreliable storage can be made highly reliable.

Integrity streams

Integrity streams protect file content against all forms of data corruption. Although this feature is valuable for many scenarios, it is not appropriate for some. For example, some applications prefer to manage their file storage carefully and rely on a particular file layout on the disk. Since integrity streams reallocate blocks every time file content is changed, the file layout is too unpredictable for these applications. Database systems are excellent examples of this. Such applications also typically maintain their own checksums of file content and are able to verify and correct data by direct interaction with Storage Spaces APIs.

For those cases where a particular file layout is required, we provide mechanisms and APIs to control this setting at various levels of granularity.

At the most basic level, integrity is an attribute of a file (FILE_ATTRIBUTE_INTEGRITY_STREAM). It is also an attribute of a directory. When present in a directory, it is inherited by all files and directories created inside the directory. For convenience, you can use the “format” command to specify this for the root directory of a volume at format time. Setting it on the root ensures that it propagates by default to every file and directory on the volume. For example:

 D:\>format /fs:refs /q /i:enable <volume>

D:\>format /fs:refs /q /i:disable <volume>

By default, when the /i switch is not specified, the behavior that the system chooses depends on whether the volume resides on a mirrored space. On a mirrored space, integrity is enabled because we expect the benefits to significantly outweigh the costs. Applications can always override this programmatically for individual files.

Battling “bit rot”

As we described earlier, the combination of ReFS and Storage Spaces provides a high degree of data resiliency in the presence of disk corruptions and storage failures. A form of data loss that is harder to detect and deal with happens due to “bit rot,” where parts of the disk develop corruptions over time that go largely undetected since those parts are not read frequently. By the time they are read and detected, the alternate copies may have also been corrupted or lost due to other failures.

In order to deal with bit rot, we have added a system task that periodically scrubs all metadata and Integrity Stream data on a ReFS volume residing on a mirrored Storage Space. Scrubbing involves reading all the redundant copies and validating their correctness using the ReFS checksums. If checksums mismatch, bad copies are fixed using good ones.

The file attribute FILE_ATTRIBUTE_NO_SCRUB_DATA indicates that the scrubber should skip the file. This attribute is useful for those applications that maintain their own integrity information, when the application developer wants tighter control over when and how those files are scrubbed.

The Integrity.exe command line tool is a powerful way to manage the integrity and scrubbing policies.

When all else fails…continued volume availability

We expect many customers to use ReFS in conjunction with mirrored Storage Spaces, in which case corruptions will be automatically and transparently fixed. But there are cases, admittedly rare, when even a volume on a mirrored space can get corrupted – for example faulty system memory can corrupt data, which can then find its way to the disk and corrupt all redundant copies. In addition, some customers may not choose to use a mirrored storage space underneath ReFS.

For these cases where the volume gets corrupted, ReFS implements “salvage,” a feature that removes the corrupt data from the namespace on a live volume. The intention behind this feature is to ensure that non-repairable corruption does not adversely affect the availability of good data. If, for example, a single file in a directory were to become corrupt and could not be automatically repaired, ReFS will remove that file from the file system namespace while salvaging the rest of the volume. This operation can typically be completed in under a second.

Normally, the file system cannot open or delete a corrupt file, making it impossible for an administrator to respond. But because ReFS can still salvage the corrupt data, the administrator is able to recover that file from a backup or have the application re-create it without taking the file system offline. This key innovation ensures that we do not need to run an expensive offline disk checking and correcting tool, and allows for very large data volumes to be deployed without risking large offline periods due to corruption.

A clean fit into the Windows storage stack

We knew we had to design for maximum flexibility and compatibility. We designed ReFS to plug into the storage stack just like another file system, to maximize compatibility with the other layers around it. For example, it can seamlessly leverage BitLocker encryption, Access Control Lists for security, USN journal, change notifications, symbolic links, junction points, mount points, reparse points, volume snapshots, file IDs, and oplocks. We expect most file system filters to work seamlessly with ReFS with little or no modification. Our testing bore this out; for example, we were able to validate the functionality of the existing Forefront antivirus solution.

Some filters that depend on the NTFS physical format will need greater modification. We run an extensive compatibility program where we test our file systems with third-party antivirus, backup, and other such software. We are doing the same with ReFS and will work with our key partners to address any incompatibilities that we discover. This is something we have done before and is not unique to ReFS.

An aspect of flexibility worth noting is that although ReFS and Storage Spaces work well together, they are designed to run independently of each other. This provides maximum deployment flexibility for both components without unnecessarily limiting each other. Or said another way, there are reliability and performance tradeoffs that can be made in choosing a complete storage solution, including deploying ReFS with underlying storage from our partners.

With Storage Spaces, a storage pool can be shared by multiple machines and the virtual disks can seamlessly transition between them, providing additional resiliency to failures. Because of the way we have architected the system, ReFS can seamlessly take advantage of this.

Usage

We have tested ReFS using a sophisticated and vast set of tens of thousands of tests that have been developed over two decades for NTFS. These tests simulate and exceed the requirements of the deployments we expect in terms of stress on the system, failures such as power loss, scalability, and performance. Therefore, ReFS is ready to be deployment-tested in a managed environment. Being the first version of a major file system, we do suggest just a bit of caution. We do not characterize ReFS in Windows 8 as a “beta” feature. It will be a production-ready release when Windows 8 comes out of beta, with the caveat that nothing is more important than the reliability of data. So, unlike any other aspect of a system, this is one where a conservative approach to initial deployment and testing is mandatory.

With this in mind, we will implement ReFS in a staged evolution of the feature: first as a storage system for Windows Server, then as storage for clients, and then ultimately as a boot volume. This is the same approach we have used with new file systems in the past.

Initially, our primary test focus will be running ReFS as a file server. We expect customers to benefit from using it as a file server, especially on a mirrored Storage Space. We also plan to work with our storage partners to integrate it with their storage solutions.

Conclusion

Along with Storage Spaces, ReFS forms the foundation of storage on Windows for the next decade or more. We believe this significantly advances our state of the art for storage. Together, Storage Spaces and ReFS have been architected with headroom to innovate further, and we expect that we will see ReFS as the next massively deployed file system.

-- Surendra

FAQ:

Q) Why is it named ReFS?

ReFS stands for Resilient File System. Although it is designed to be better in many dimensions, resiliency stands out as one of its most prominent features.

Q) What are the capacity limits of ReFS?

The table below shows the capacity limits of the on-disk format. Other concerns may determine some practical limits, such as the system configuration (for example, the amount of memory), limits set by various system components, as well as time taken to populate data sets, backup times, etc.

Attribute	Limit based on the on-disk format
Maximum size of a single file	2^64-1 bytes
Maximum size of a single volume	Format supports 2^78 bytes with 16KB cluster size (2^64 * 16 * 2^10). Windows stack addressing allows 2^64 bytes
Maximum number of files in a directory	2^64
Maximum number of directories in a volume	2^64
Maximum file name length	32K 255 unicode characters (for compatibility this was made consistent with NTFS for the RTM product)
Maximum path length	32K
Maximum size of any storage pool	4 PB
Maximum number of storage pools in a system	No limit
Maximum number of spaces in a storage pool	No limit

Q) Can I convert data between NTFS and ReFS?

In Windows 8 there is no way to convert data in place. Data can be copied. This was an intentional design decision given the size of data sets that we see today and how impractical it would be to do this conversion in place, in addition to the likely change in architected approach before and after conversion.

Q) Can I boot from ReFS in Windows Server 8?

No, this is not implemented or supported.

Q) Can ReFS be used on removable media or drives?

No, this is not implemented or supported.

Q) What semantics or features of NTFS are no longer supported on ReFS?

The NTFS features we have chosen to not support in ReFS are: named streams, object IDs, short names, compression, file level encryption (EFS), user data transactions, sparse, hard-links, extended attributes, and quotas.

Q) What about parity spaces and ReFS?

ReFS is supported on the fault resiliency options provided by Storage Spaces. In Windows Server 8, automatic data correction is implemented for mirrored spaces only.

Q) Is clustering supported?

Failover clustering is supported, whereby individual volumes can failover across machines. In addition, shared storage pools in a cluster are supported.

Q) What about RAID? How do I use ReFS capabilities of striping, mirroring, or other forms of RAID? Does ReFS deliver the read performance needed for video, for example?

ReFS leverages the data redundancy capabilities of Storage Spaces, which include striped mirrors and parity. The read performance of ReFS is expected to be similar to that of NTFS, with which it shares a lot of the relevant code. It will be great at streaming data.

Q) How come ReFS does not have deduplication, second level caching between DRAM & storage, and writable snapshots?

ReFS does not itself offer deduplication. One side effect of its familiar, pluggable, file system architecture is that other deduplication products will be able to plug into ReFS the same way they do with NTFS.

ReFS does not explicitly implement a second-level cache, but customers can use third-party solutions for this.

ReFS and VSS work together to provide snapshots in a manner consistent with NTFS in Windows environments. For now, they don’t support writable snapshots or snapshots larger than 64TB.

Comments

Anonymous
January 16, 2012
Nice.. I'm really curious... The change of filesystem is a milestone in the windows developing...
Anonymous
January 16, 2012
This sounds interesting (and I could update my TechNet Wiki article about Windows file systems): is there any kind of documentation available yet?
Anonymous
January 16, 2012
The comment has been removed
Anonymous
January 16, 2012
oh thats really great to change filesystem...i think this step give more advetage for every user and will let all poeple like windows 8 more http://www.full-windows8.com/
Anonymous
January 16, 2012
Though initially this will only ship with Windows 8 Server, when can we expect it with Windows 8 and can Windows 8 access a ReFS volume from a Windows 8 Server?
Anonymous
January 16, 2012
Are things like Compression (which I use a fair amount) and Quotas likely to be implemented in the future?
Anonymous
January 16, 2012
Please add support for HFS+ and ext4 in Windows 8. Thank you.
Anonymous
January 16, 2012
WinFS 2.0? Just saying... en.wikipedia.org/.../WinFS
Anonymous
January 16, 2012
Very nice. A few questions:

You say that while ReFS does not provide 2nd level caching, but third parties can provide that feature. What API are they using, and is that caching protected from errors? It seems a little odd to have 3rd party code in the middle of a very critical part of a filesystem.
You said that correction is only available on mirrored spaces not parity spaces. Is that just for ReFS, and that storage spaces is autocorrecting errors on disk that ReFS finds, or only errors that are found doing a parity space scrub, or not at all?
What management tools are being created to administer these new features?

Anonymous
January 16, 2012
I have a few questions:

Will Windows 7 be able to read drives formatted with ReFS?
Do you intend for ReFS to eventually repace NTFS on everything, with the Windows Server 8 version being similar to a pilot program?
Will this be like the transition from FAT32 to NTFS where some operating systems can't read drives formatted by newer versions of Windows?
Is this the rumored "Protogon" file system that you are now calling ReFS?

Anonymous
January 16, 2012
I want that on my desktop machine too!!! I'm also not a fan of the 100 different Windows versions (Home/Premium/Professional/Ultimate) The ultimate thing would be, if there is only an ultimate version (and one server version)
Anonymous
January 16, 2012
The one thing I was looking for: "Maximum file name length 32K unicode characters". Solves the problem with the current limit of 255 characters. Very good job!
Anonymous
January 16, 2012
The comment has been removed
Anonymous
January 16, 2012
@WindowsVista567 All answers are in the article, a quick read would reveal most answers either directly or indirectly:

Will Windows 7 be able to read drives formatted with ReFS? Yes
Do you intend for ReFS to eventually repace NTFS on everything, with the Windows Server 8 version being similar to a pilot program? Yes
Will this be like the transition from FAT32 to NTFS where some operating systems can't read drives formatted by newer versions of Windows? If the system can read NTFS then it can read ReFS
Is this the rumored "Protogon" file system that you are now calling ReFS? Probably

Anonymous
January 16, 2012
TheKernel, where are you reading that REFS will succeed NTFS? I don't believe that's stated either directly or indirectly, clearly the inability to boot from RTFS or use it for external storage strictly prevents it from replacing NTFS right now, and they make no statements whether either limitation will be changed. Without that it would be clearly impossible for it to replace NTFS.
Anonymous
January 16, 2012
@Rand "With this in mind, we will implement ReFS in a staged evolution of the feature: first as a storage system for Windows Server, then as storage for clients, and then ultimately as a boot volume. This is the same approach we have used with new file systems in the past."
Anonymous
January 16, 2012
@Cakes - if you want an OS that supports Linux filesystems and file formats, then go use Linux.
Anonymous
January 16, 2012
First of all: THANK YOU VERY MUCH for this very interesting article. No marketing blah blah, just the interesting stuff. After reading the article two questions remain:

You write that you found Log Structured Filesystems to be "... unsuitable for the type of general-purpose file system required by Windows.". Do you plan to publish your results? I couldn't find any papers regarding this problem. With the Microsoft's scale, technology and battery of tests I'm sure you could give the file system community valuable insights.
Why didn't you store the file data right in the tree? I'm looking forward to see real world benchmarks and detailed design decisions of this file system. Thanks a lot!

Anonymous
January 16, 2012
I'm confused. If a storage pool can only be 2^52, yet a volume can be 2^78... where are those extra bytes going to be stored? How can you have a volume that is larger than the pool that presumably supports it? Also, I second pmbAustin's question: We curently use Quotas, Compression and hard-links; are those going to be implemented? We've saved 100s of GBs every day with our usage of hard links.
Anonymous
January 16, 2012
@WindowsVista567 @TheKernel No, I don't think windows 7 will read ReFS unless they release a driver for it (I doubt that happeneing). If a computer can read NTFS has nothing to do with reading ReFS. The API used to access the files are the same, but not how they are stored on disk (Think of it as the implementation of the API). It's the same problem as a Microsoft operating system reading a linux drive. It needs file system drivers to be loaded. But overall, this combined with Storage Spaces sounds like ZFS for windows. And that makes me happy.
Anonymous
January 16, 2012
@TheKernel Thanks.
Anonymous
January 16, 2012
@JohannesB I'm not sure it will. NTFS already supports 256 char file names, but you have to use alternate versions of all the win32 file APIs in order to take advantage of it. The default versions don't support super long paths because of the buffer overflow potential in huge numbers of badly written programs. The only 3rd party userspace app I've seen which (partially) supported it is IBM Rational Clearcase (source control app), which will write >256 paths when copying files from the server to your computer. It's check for changes and update routine however barfed on them. (This might be fixed in a newer version than I've used; my employer has deprecated CC, so I might never have gotten the latest and greatest.)
Anonymous
January 16, 2012
The comment has been removed
Anonymous
January 16, 2012
Matt, thanks for the quick reply. I guess I don't understand exactly what parity is supposed to do. If I lose a disk completely, I can replace the dead disk, and the parity will then be recalculated across the space, correct? if that's true, why doesn't the same logic apply to a slab? I can just as easily reconstruct the correct data for a slab, and then correct the bad data, and maybe not use that sector on the disk again, etc... If it doesn't do this, then if I get a bad sector that corrupts a slab, the slab still has good data because of the parity calculation, so a read works fine. But how do I clear the known fault? Do I have to get a bunch of new disks, create a new space, and then copy from the old space to the new space, and then go back and reclaim the old space's storage? In other words, I am trying to understand how as an administrator I deal with a disk that has has some sectors that are bad on it.
Anonymous
January 16, 2012
"The NTFS features we have chosen to not support in ReFS are: named streams" - does this impact the way files downloaded from the Internet get marked as Blocked (ie Zone.Identifier alternate data streams) on an ReFS file system? Regards, Jason
Anonymous
January 16, 2012
Will you be releasing the source to ReFS? Filesystem readability on different operating systems is a major issue these days and opening the source under a non-restrictive license would help tremendously.
Anonymous
January 16, 2012
Will this be an open filesystem I could read/write to from other OSses like OSX Linux and BSD? That is the only thing I care for at the moment since otherwise it just throws up too many hurdles in my daily use.
Anonymous
January 16, 2012
While your at it, can you replace the kernel too.
Anonymous
January 16, 2012
@Matt Garson [MSFT] I was afraid of that. It looks like the FAT32 problem all over again.
Anonymous
January 16, 2012
Finally a new file system. hopefully this one will be faster and more secure than NTFS. and when will Windows 8 be able to boot from ReFS? if Windows is going to have a new file system, i want to use it now!
Anonymous
January 16, 2012
"ReFS, (which stands for Resilient File System), is built on the foundations of NTFS, so it maintains crucial compatibility..." Very good. "This file system, which we call ReFS, has been designed from the ground up..." ...Wait, what? Clearly it's not designed from the ground up if it's built on the foundations of NTFS maintaining compatibility. What are you trying to say here?
Anonymous
January 16, 2012
The comment has been removed
Anonymous
January 16, 2012
So sad this is not coming to the client now. :-) Some comparison between NTFS <--> SS (client) and ReFS <--> SS (server) would have made it more interesting. Although it's not considered or planned to be competing technologies, it would have created some excitement for folks planning to deploy ReFS and SS for file server scenarios.
Anonymous
January 16, 2012
@WindowsVista567 Would you mind clarifying what you mean by the "FAT32 problem"? It sounds like you've got some past experience in this area and I'd like to understand better before responding.
Anonymous
January 16, 2012
The comment has been removed
Anonymous
January 16, 2012
The comment has been removed
Anonymous
January 16, 2012
I'm really disappointed why ReFS doesn't support second level caching/hybrid storage using SSD.
Anonymous
January 16, 2012
Great news. I run Win 7 Ultimate as a home server. Will there be a Foundation edition of Win 8 Server? It may be time to switch from a client SKU to Foundation for the home file server when Win 8 Server is released.
Anonymous
January 16, 2012
@JohannesB, @Daniel Neely Regarding path length limits, the following MSDN article provides excellent information which applies to ReFS just like it does to NTFS. msdn.microsoft.com/.../aa365247(v=vs.85).aspx @deiruch Regarding log structured filesystems - We want to optimize reads very carefully since they far outnumber writes and they affect response time more directly. This is better done by careful placement. We do want to reliably update metadata and checksum protected user data and for that we do write them to new locations like a log structured filesystem does. Ultimately, the difference can be narrowed down to allocation and defrag policies. Regarding storing file data directly in the tree - there is a tradeoff here. Storing data in the tree gets access to the part of the file that fits within the leaf page in the same I/O as the original lookup, but the rest of the data is on other nodes which causes extra IOs (similar to when the whole file is stored separately). It may also increase the depth of the tree. This is exactly the type of issue that makes designing filesystems challenging.
Anonymous
January 16, 2012
@ZipZapRap - "Consumers" generally don't read technical blogs about how products are built so we have chosen to focus on the details of building and engineering the product -- "behind the scenes". This isn't really a place where we are marketing the product to consumers. Since we've received feedback (and tried to respond to it) that we have not focused enough on the more foundational elements of Windows, we did a few posts on the file system. What would you like to hear more about? THere are lots of ways to potentially classify posts, but we are working to be balanced. We've done about 40 posts on engineering details with about 25 of them focused on "end-user" details (though still throuhg an engineering lens) and about 15 focused on what I think of as "core" or "foundational" topics. It is not clear how to categorize some topics though like boot and setup, which can be viewed through either lens. Your feedback is welcome as to where you think we should best go.
Anonymous
January 16, 2012
but it's not going to do anything about the miserable performance when accessing files, is it? It really is ridiculous how slow operations that touch lots of files are when compared to non-Windows filesystems.
Anonymous
January 16, 2012
When is this coming to WHS? I am most excited about this, was sad it was scrapped from WHS 2011. Will you ever support a raid-6 like scheme to support multiple drive failure for read only data?
Anonymous
January 16, 2012
The comment has been removed
Anonymous
January 16, 2012
It's great to see that you even reimagine the filesysteme in Windows 8! But when can i expect ReFS as a storage filesysteme for clients? You said it will come later, because of caution. Thats fully understandable, but i think professional users can't wait to use it on the client too! Is there a chance to get it in the client after a Service Pack release (SP1 or SP2) or do i have to wait future Windows releases?
Anonymous
January 16, 2012
No sparse and hard-links? No-no-no, we need these. At least hard-links are a must.
Anonymous
January 16, 2012
Hi, I still do not uderstand a lot of jargon said above..so pardon me. *. On Laptops and handhelds and tablets, the raid like behaviour (striping/mirroring) cannot be had as easily. how do they work on a single drive. *. on a Laptop.. ReFS can only be had on the second partion which is not a boot volume. no external connected 2tb drives, no boot volume... etc *. is there any SSD related improvements *. Is there any performance level improvements over NTFS? *. Please add a little support to recognise other Operating systems Names and preserve boot menu/order etc
Anonymous
January 16, 2012
@ thechuckesstart i completely agree - i want a storage pool with raid 6 / dual parity single parity just isnt good enough when you have 10+ disks!
Anonymous
January 16, 2012
Nice! It's a shame there probably won't be a Server Core edition priced cheaply enough to be used for a home NAS. >does this impact the way files downloaded from the Internet get marked as Blocked (ie Zone.Identifier alternate data streams) on an ReFS file system? >but it's not going to do anything about the miserable performance when accessing files, is it? I would like to read your answer to that, too.
Anonymous
January 16, 2012
Just like the storage pools, this is another interesting feature. The only drawback i see is, as has been pointed out by others, the lack of some features that are heavily used (in my corporate environment at least), like quotas, hard links (junctions) and compression. Although i can quite happily live without compression on the file system (given deduplication is in and the data is appropriate), i can not quite get around losing the other two. Are these specs final? Joao
Anonymous
January 16, 2012
"nothing is more important than the reliability of data" I agree. You need to release the spec for this so that the data is accessible from any OS.
Anonymous
January 16, 2012
Am I correct in understanding that when using Integrity streams just the changed blocks will be written out to disk and checksummed?
Anonymous
January 16, 2012
While you are at it, will you please finally get rid of that stupid 260 character limit for paths. NTFS supports much longer paths, and has for ages, but the shell is still limited to 260 characters. Come on, this is 2012, not 1995. Honestly, the first time I ran into that limit (and when doing programming, that happens easily!) I thought it was some kind of joke, but alas not.
Anonymous
January 16, 2012
What about performance of this file system?
Anonymous
January 16, 2012
I have found info about performance info Thanks
Anonymous
January 16, 2012
very nice article. removing the chkdsk outage windows will be nice
Anonymous
January 16, 2012
Still NTFS for all the consumers... ReFS only for server version... = goodbye Microsoft... Office maybe will remain for some years... it depends on the next versions (for windows and for Mac). Too much "touch" features and no other advancements in desktop features = few years left, even for Office...
Anonymous
January 16, 2012
nice post! a little dissapointed abt not able to boot using that, still will wait
Anonymous
January 16, 2012
> Q) How come ReFS does not have deduplication, second level caching between DRAM & storage, and writable snapshots? > ReFS does not itself offer deduplication. One side effect of its familiar, pluggable, file system architecture is that other deduplication products will be able to plug into ReFS the same way they do with NTFS. But in Windows 8 server dev build deduplication exists. It will be removed in beta version or not? Or it will be available only on NTFS volumes? Sorry if comment appears twice.
Anonymous
January 16, 2012
You should really consider to get this in the client versions fast. I would like to use this with Storage Spaces, but I'm not buying a server version for my home server. So I will wait with buying Windows 8 until it's working ... By the way: Will all versions of Windows 8 eventually get full featured Storage Spaces and ReFS? And you really should get the auto correction working with parity, and implement multiple parity in Storage Spaces.
Anonymous
January 16, 2012
@Asbjørn, as pointed out by Surendra Verma above, win32-unicode does support 32k path lengths. However the existing non-unicode APIs can't safely be changed to support longer paths because it would result in buffer overflows for any program that has the maximum length hardcoded at compile time.
Anonymous
January 16, 2012
nice post! a little dissapointed about the performance trade-offs
Anonymous
January 16, 2012
One other question, that some have alluded to in previous comments: will this new file system be supported in Windows 8 Home Server? I currently use a Synology DiskStation because all the Windows Home Server implementations are quite lousy. In this day and age more and more people are getting servers in their home this is a major oversight and problem. NASs are very popular and they all pretty much run Linux. These home servers usually have multiple disks that have arrays that are auto-expanding. My Synology has an iTunes server, a photo server, a web file server, works as a TimeMachine backup point, supports AD and ACLS, can record webcams, even recording only when there is motion in the frame. There are a ton of other features. Then I have a netbook hooked to my TV to provide me with access to all sorts of entertainment. So I now have two devices running all the time in my house and neither can be used in both scenarios. We really need a version of Windows that can be used to fill both of these roles, coupled with hardware that is very low power such that it can be on all the time and still be relatively 'green'. I can't believe that I need to run Linux in my home to satisfy reasonable consumer needs when I'm otherwise a 100% Windows shop, even waiting patiently for a convertible tablet/netbook running Windows 8. I think it would be quite a service to the community if you talked about your plans for Windows 8 Home Server and how it can be used in the role that I've addressed above.
Anonymous
January 16, 2012
The comment has been removed
Anonymous
January 16, 2012
"The NTFS features we have chosen to not support in ReFS are: named streams, object IDs, short names, compression, file level encryption (EFS), user data transactions, sparse, hard-links, extended attributes, and quotas." By "user data transactions", I'm assuming you mean Transactional NTFS (TxF)? This has seriously been discontinued? it was highly touted back when Vista came out. This is a highly breaking change to existing code and throughly pulls the rug... Ouch! Please don't release something so half baked and unfinished!
Anonymous
January 16, 2012
The comment has been removed
Anonymous
January 16, 2012
Only one word! "YEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEAH!"
Anonymous
January 16, 2012
Hi Microsoft! Please don't drop the support for hardlinks! We use them DAILY to deduplicate hard disks. They are vital! If you really remove them I have to switch to Linux and ext4 as filesystem. Roland
Anonymous
January 16, 2012
@those who miss hardlinks: You don't need hardlinks when you have full softlinks available. (Since Vista) Anyway, I'd like too ask which features are hardkilled. (i.e.: Are dropped permanently, right now it is bit uncertain) @ShaneM : Just wait for beta and you'll see how it'll work out for you. (Am too waiting with judgement for beta)
Anonymous
January 16, 2012
Also, I'd like to see evidence for NTFS is "slow". So far didn't see any tests, just assertions.... (And AFAIK no SSD/HDD review mentioned anything about this supposed "problem")
Anonymous
January 16, 2012
All you people complaining about the (currently) missing features: NTFS is not going away. You won't have to use ReFS if it doesn't meet your needs, just continue doing what you are doing.
Anonymous
January 16, 2012
@Daniel Neely: Yes, they do, but only via convoluted workarounds. To quote from MSDN (msdn.microsoft.com/.../aa365247%28v=vs.85%29.aspx): "In the Windows API (with some exceptions discussed in the following paragraphs), the maximum length for a path is MAX_PATH, which is defined as 260 characters. [...] These prefixes are not used as part of the path itself. They indicate that the path should be passed to the system with minimal modification, which means that you cannot use forward slashes to represent path separators, or a period to represent the current directory, or double dots to represent the parent directory. Because you cannot use the "?" prefix with a relative path, relative paths are always limited to a total of MAX_PATH characters." So basically, it is theoretically supported, but only just, and many things won't work. This is what needs to change. Windows needs to support longer paths by default and everywhere, and then use a compatibility shim for programs that don't work. It is especially embarrassing that Explorer is still limited to 260 characters. I don't want these limits forced on me, if I don't use any programs that have problems with long paths.
Anonymous
January 16, 2012
Another proprietary and closed file system! Just what we needed!
Anonymous
January 16, 2012
I have just posted about this on my blog: www.nicklowe.org/.../pulling-the-rug-microsofts-refs Mapped to file system attribute flags, the following are not going to be supported: FILE_NAMED_STREAMS The file system supports named streams. FILE_SUPPORTS_OBJECT_IDS The file system supports object identifiers. FILE_FILE_COMPRESSION The file system supports file-based compression. FILE_SUPPORTS_ENCRYPTION The file system supports the Encrypted File System (EFS). FILE_SUPPORTS_TRANSACTIONS The file system supports transactions. FILE_SUPPORTS_SPARSE_FILES The file system supports sparse files. FILE_SUPPORTS_HARD_LINKS The file system supports hard links. FILE_SUPPORTS_EXTENDED_ATTRIBUTES The file system supports extended attributes. FILE_VOLUME_QUOTAS The file system supports per-user quotas. This is an incredible list of features that are not going to be supported in ReFS, at least in its initial implementation. It is likely to have huge compatibility implications and to break many of the existing applications that are in use today.
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
@ Steven Sinofsky, some very important issues: 1-what will be done to improve performance? 2-what will be done to reduce the fragmentation of files? 3-What is the forecast for refs on Windows 8 client? 4-Why not stop with the SKUs? Stop, this only makes the Microsoft lose customers. Copying Apple. Love the Windows, but it needs to improve.
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
Here is a summary of the facts regarding ReFS: helgeklein.com/.../new-file-system-refs-in-windows-8-quick-facts
Anonymous
January 17, 2012
I guess when ReFS is brought to the Windows client or when ReFS supports booting, many limitations will be addressed. Few examples: The Distributed Link Tracking service in NT depends on Object IDs. The servicing stack depends on hard links. IE adds a named stream to every file it downloads. MS can't possibly take away the Encrypting File System feature out of Windows. Or compression or sparse files or quotas or transactions?
Anonymous
January 17, 2012
@xpclient - File system transactions (TxF) are also extensively used: "As for platform stability, this is achieved by some of the ways that Microsoft is using TxF in its own technologies. There are three core features in Windows Vista and Windows Server "Longhorn" that now make use of Transactional NTFS: Windows Update, System Restore, and Task Scheduler. All of these use TxF to write files to the file system within the scope of a transaction in order to handle rollback/commit in case of any exceptions, such as a system reboot due to a loss of power. If you’ve ever experienced a loss of power right in the middle of Windows Update, you know this can leave your system dead and in need of a fresh installation. But now that Windows Update uses TxF, if this same situation occurs on your Windows Vista system, the computer can recover by rolling back the transaction when the system reboots. By adopting TxF internally, Microsoft has helped make its own operating system more stable." msdn.microsoft.com/.../cc163388.aspx
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
@Surendra Verma [MSFT]: Does ReFS have automatic sparse files? e.g. the file extents table looks like it can check if an entry contains a block of all 0's during checksumming and decide not to store it.
Anonymous
January 17, 2012
No more file locking of open files, please?? If you're going to break backwards compatibility, really do it on that one. I want to be able to delete a file that is currently open by another process and have things work like Linux. Your current semantics of locking open files makes windows more costly to support in a heterogeneous environment! Just do it! Promise us you will!!!
Anonymous
January 17, 2012
Dropping Hardlinks and Named Streams really hurts. Hardlinks are the core of any incremental backup aka timemachine, and symbolic links can not be used as a fill in here. I am the author of a little freeware tool, which does incremental backup and this really smashes any 'Delorean Copy' aka timemachine like backups. See schinagl.priv.at/.../ln.html On the other hand we heavily use named streams, to provide checksums to arbitrary files, and give them kind of 'free xml based attributes' in this alternate named stream. How do we transport this without streams? When do you ship ReFS for the first time? Can we have this in a normal windows8 beta(not server) just to try out file system features? Maybe you rethink your decision with respect to hardlinks and named streams, but I guess it is too late for the first version To be honest I am little bit disappointed. BR Hermann
Anonymous
January 17, 2012
For those that are doing backup operations, are there any reasons you're not using shadow copies to perform these functions?
Anonymous
January 17, 2012
My only request though I know it will go unheard: ReFS should be made available to Linux in some shape or form where the fiasco that has been NTFS drivers should go away. This should NEVER have been an issue. Please do something about it. You don't even have to GPL the code, make a proprietary but 100% compatible kernel module.
Anonymous
January 17, 2012
Would there be any downside for me using Windows 8 Server as my desktop OS?
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
@Matt Garson [MSFT] "FAT32 problem" is probably not the best way to phrase my statement. I am referring to the fact that no Windows 9x operating system can read NTFS drives. Not even Windows ME, which was released after Windows 2000, can read NTFS drives. It might be better to call it the "NTFS problem," but NTFS is superior to FAT32 and I consider the problem to be related to the fact that Windows 9x must use a FAT file system to store data, and that disks often have to be specially formatted to work with Windows 9x computers. In the same way, once ReFS is expanded and becomes more common, we may see a point where there are disk compatibility problems on certain computers because those computers cannot read the file system of some drives.
Anonymous
January 17, 2012
Building the next generation kernel for Windows: MinWin would be also a good post and your work on MinWin!
Anonymous
January 17, 2012
@Asbjørn 17 Jan 2012 7:27 AM: Shim every single application which used ANSI version of api or which compiles with Unicode support yet hardocodes max length? Remember, in headers many functions are compile time defined either to A or W versions, so many programms use W version yet they have original limit. I don't think it would be efficient use of space... However so far no such limitation is mentioned for new CreateFile2 (msdn.microsoft.com/.../hh449422(VS.85).aspx).
Anonymous
January 17, 2012
Thanks for all your comments and feedback so far. More responses below. @Bastien92, @Sinnfrei, @Andre, @xpclient, Regarding when ReFS will be available on the client, we’re following a deliberate approach where we evolve it in a staged manner starting with Server first, then on the Client and ultimately on the boot volume. We expect NTFS to be supported and available with all of its features. We do like to hear feedback about features, so that is highly appreciated. @ravloony, Regarding accessing ReFS from OS’s other than Windows 8 Server – the data is accessible via a network file share over both SMB and NFS protocols that are very widely adopted. You would have to expose the ReFS volume as a file share, which is then visible to the other OS as a folder to work on. @Lewis “Am I correct in understanding that when using Integrity streams just the changed blocks will be written out to disk and checksummed?” Yes that is correct. @Alex “What about performance of this file system?” We believe that it will meet the performance requirements of general file server scenarios. We would be interested in learning about the performance of your specific scenario, as such feedback is very valuable to us. @FZB “very nice article. removing the chkdsk outage windows will be nice” Thank you. @xpclient “Is there a performance overhead of enabling integrity streams?” There is a small CPU cost in computing checksums. Other than that, there is an additional cost to store the updated checksums with the new data. We expect these costs to be acceptable given the benefit provided by checksums.
Anonymous
January 17, 2012
@Surendra Verma [MSFT] So ReFS will be in the Client in the RTM release? I am confused, your statement is not clear for me.
Anonymous
January 17, 2012
@Mike I'm sorry I wasn't clearer. I meant to say that we will evolve across releases of Windows. This gives us a chance to get valuable feedback.
Anonymous
January 17, 2012
Would windows 8 have multiple desktops support like ubuntu and mac?
Anonymous
January 17, 2012
@Surendra Verma [MSFT] Ahh.thx I get it!
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
... So those filenames can now be long like on Unix based OSs? and in Windows 9, I assume we'll be allowed to boot off this OS? how does ReFS handle fragmentation?
Anonymous
January 17, 2012
You guys really have no clue how happy I am. this is like an amazing Christmas present. you guys rawk.
Anonymous
January 17, 2012
girlyngeek.blogspot.com/.../microsoft-returns-customers-to-past.html my comparison of features of ReFS with other filesystems.
Anonymous
January 17, 2012
WinFS never was a filesystem tho bro... anywho, I thought up this idea lastnight, here it is. hopefully you'll implement this one too (; Microsoft, you should give up on constantly indexing the HDD/SDDs and instead save advanced metadate in teh MFT, and when a search is performed, you simply consult the MFT, *** you have to anyway. and it'd free u[ resources as well.
Anonymous
January 17, 2012
@Willem We don't have device and cloud storage unified like that because bandwidth just isn't there yet.
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
Whoa. Disturbance in the force... A girl has arrived... Hide your motherboards boys!
Anonymous
January 17, 2012
No-boot, no-sparse, no-hardlinks, no-EFS, no-tx, NO GO! The maximum path length has been 30K Unicode chars in NTFS since 1992, and NTFS was available on workstation and server AND bootable from day one in the very first NT beta. Where were you Surendra? :-) Maybe 5 years from now when your telemetry tells you no one uses ReFS (mostly because the forgotten features breaks far more apps than you thought, including many not released yet) we'll get a proper "reimagined" NTFS 8 with most of the touted advantages of ReFS. THAT happened before...
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
I'm really surprised to see that compression is not going to be implemented on the ground of "complexity and limited value". I just don't understand the complexity argument. I agree that the compression algorithm in NTFS was too slow for today's HDD (especially with interleaving properties of Pool Spaces), but then, there are newer algorithms out there which are much faster (a few hundred of MB/s per core !) and compress better. In fact, a compressor which decodes very fast will even improve HDD read performance as perceived by the application. And since this is a classical bottleneck, such property will have a positive impact on user experience....
Anonymous
January 17, 2012
For Windows 8 news I have created an aggregator that you guys can follow: win8tabs.com/windows-8-news-reviews
Anonymous
January 17, 2012
The article made it clear that ReFS and NTFS do not require Storage Spaces (although there are benefits to deploying them together), but I am wondering if the reverse is true. Does a storage space care if it is formatted NTFS/ReFS, or does it function at the block-level only? In other words, will a storage space (whether simple or resilient) be seen as a literal virtual hard drive that can be formatted with any file system? Can you format a storage space with FAT32, ext3, or any other arbitrary file system that a 3rd party program running on Windows was written to create/handle? or must you absolutely use either NTFS or ReFS if you are using storage spaces?
Anonymous
January 17, 2012
I'm amazed to see this as MS came up with WinFX couple of years back.
Anonymous
January 17, 2012
@Mubeen M WinFX ran on top of NTFS, it was never a file system.
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
Hello. Will this filesystem finally support API's to split large files into two or more parts at arbitrary offsets without copying? I know that in the context of existing file format, it may not make sense or be very appealing. But it can be very appealing for developers of new file formats and editing applications.
Anonymous
January 17, 2012
Very informative. Thanks for sharing. www.sevenit.com/Software-Solutions.php
Anonymous
January 17, 2012
@Surendra Verma [MSFT] Thanks for the reply. Let me push on this a little bit more if I might. I understand that the filesystem can only repair corruption detected in a space that is mirrored. But can ReFS report up to the administrator when corruption is detected in a parity volume, even if it can't be repaired? For example, if the file with the corrupted extent was a video file, an error may be only slightly noticeable on playback, so you wouldn't necessarily want to remove the file. If the file was an executable, the error would be fatal, but the administrator might be able to restore it from install media or a download, or from another system. If the detection of the corruption happens relatively quickly, then it may be possible to recover the good file from backup. All of these examples make me think there is significant value in checksumming all file in all spaces, not just mirrored ones. If you could support this and alert administrators when corruption is detected for action, that would be a lot better than just ignoring the problem or not allowing checksumming at all. Thank you for your posts - I am a longtime unix admin, but I appreciate the scale and GUI management utilities that windows has always done better in. If you can deliver this kind of functiionality and performance with robustness, I think you will win quite a few accounts from oracle and other platforms. Mike S
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
It allows 32K characters in file path as opposed to just 256 in NTFS............Great . . .Congrats!
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
@Sunil Jadhav: As i have understood there is no reason to be hapy about that because there is a limitation build into the Windows Explorer because also NTFS can handle such a long file path.
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
The comment has been removed
Anonymous
January 17, 2012
That "File Structures" picture reminds me of Windows Azure. Any plan to merge them, with a clever Network stack?
Anonymous
January 17, 2012
So we are being shown a new FS that we won't be able to use on Windows 8 Client? when do we get to use it? if not in Win 8 are there any NTFS improvments in the mean time? Can you do a post on kernel improvments please.
Anonymous
January 17, 2012
Error checking and correction needs to placed at the drive level and not the File system level. This is because drive technology can change making File System error correction pointless and obsolete (a file system that mirrors data on a drive that provides internal deduplication). The error checking needs to be dependent on the technology of the hardware.
Anonymous
January 17, 2012
Wow ! This sounds great. Finally something that did wake me up about Windows 8. But i think there must be updates to all other Windows systems still used; 2000, XP, Vista and Windows 7. They must be able to read the new filesystem as well. I mention the older versions of Windows because people and corporations do still use them. If there should be a smooth upgrade from the old Windows versions to Windows 8 or later i can think of a scenario where the upgrade goes step by step. If You do as You think i guess many users and corporations will run in to problem with incompability between different Windows versions. That is not very smart. Don't say it is impossible to upgrade the older operatingsystems, because i can read filesystems in Linux, Mac and more in my old Windows 2000 and XP. If they can, then Microsoft can.
Anonymous
January 17, 2012
Should add that it is software installed that makes it possible to read from Linux , Mac and so on. in my old Windows.
Anonymous
January 18, 2012
"In Windows 8, you cannot boot from a space. As an alternative, you can continue to use dynamic volumes for booting." blogs.msdn.com/.../virtualizing-storage-for-scale-resiliency-and-efficiency.aspx Can I install Windows 8 on a dynamic disk?
Anonymous
January 18, 2012
The comment has been removed
Anonymous
January 18, 2012
The comment has been removed
Anonymous
January 18, 2012
The comment has been removed
Anonymous
January 18, 2012
No hardlinks, eh? Any idea how that will impact WinSxS (As far as I know side by side is built around hard links? (Sorry if this is a double post, I cannot see my first attempt in the thread)
Anonymous
January 18, 2012
New FS? Looks like NILFS, BtrFS, or ZFS. This is NOT suggesting most users would use ReFS. Most users would stick with NTFS, and enterprise users with many hard drives and clusters to manage would find ReFS useful. The boot loader won't even know how to access ReFS at first. I do wish Microsoft would cite that this technology has been in use for a number of years in other systems, and all these ideas can be found in other file systems (BtrFS for instance).
Anonymous
January 18, 2012
@Mike S Regarding checksums on non-mirrored Storage Spaces – ReFS supports this. In fact integrity can be enabled even when not running on Storage Spaces. When integrity is enabled and we detect a checksum mismatch, we log an event and, by default, fail the read operation. You can change this behavior for a file if you don’t want the read to fail (the event will still be logged so you’ll know about the corruption). @Misplaced Error Checking Our architecture allows running ReFS directly on disks (i.e., without Storage Spaces). If you know that your storage does a good job of mirroring, deduplication, etc., then you have that choice. ReFS will still employ techniques like not writing metadata in place, using checksums on metadata (and optionally on user data) for end-to-end robustness to guard against firmware and hardware faults. @Oti5 @Alvaro Yes you can boot from dynamic volumes. More details to follow. @Jason We have improved NTFS disk checking and availability quite significantly in Windows 8, along with making it more resilient on SATA drives that are very common on clients. For more details you can watch this BUILD talk starting at 25 minutes channel9.msdn.com/.../SAC-446T
Anonymous
January 18, 2012
The comment has been removed
Anonymous
January 18, 2012
The comment has been removed
Anonymous
January 18, 2012
The headline is misleading "Building the next generation file system for Windows: ReFS". Should be "Building the next generation file system for Windows Server: ReFS". Does it still get fragmented with times with ReFS?
Anonymous
January 18, 2012
Does ReFS provide support for "secure deletion"? For example, will one be able to SecureDelete("c:/CitiAcctXYZ.dat") and rely on the filesystem doing whatever is necessary to wipe/eradicate the data associated with that file (file contents, file name, file size, etc)? I think such a feature "needs" to be built into modern file systems and perhaps also passed down to the drivers that would know about wear leveling, etc.
Anonymous
January 18, 2012
Is there any kind of specification/documentation available yet? If not, are there any plans to release those in near future?
Anonymous
January 18, 2012
Is this file system going to be RAM hungry like ZFS?
Anonymous
January 18, 2012
@MSFT: Impressive that you engage in direct consumer discussions. That's great to see. Reminds me of the pre-Vista blog days where every feature was discussed on some Microsoft blog and the people responsible for a feature would discuss and explain their reasoning. As Steven Sinofsky is already discussing offtopic stuff: I'm running the dev preview on my main machine since it came out and I'm quite happy with it. I even got accustomed to the new start menu. If I had a single wish it would be this: Give me a "Never combine, hide labels" option for the taskbar icons.
Anonymous
January 18, 2012
Realmente isto é um marco na tecnologia Microsoft. Temos que acompanhar este processo e entender como o mesmo será migrado para as versões do Windows 2008 e Windows 7.
Anonymous
January 19, 2012
The comment has been removed
Anonymous
January 19, 2012
The comment has been removed
Anonymous
January 19, 2012
Looks promising--should improve performance!
Anonymous
January 19, 2012
The comment has been removed
Anonymous
January 19, 2012
The comment has been removed
Anonymous
January 19, 2012
The comment has been removed
Anonymous
January 19, 2012
No quotas? Is this a joke? Filesystem without quotas is good for desktops or pendrives but not for [storage] servers. You're kidding, right?
Anonymous
January 19, 2012
The comment has been removed
Anonymous
January 19, 2012
How about displaying folder-size in explorer just like file-size is shown instantly?
Anonymous
January 19, 2012
Our legal team working very hard to find some of our patents in ReFS to sue you guys. Happy New Year.
Anonymous
January 19, 2012
Please implement multi desktop support in windows 8 Because desktop is the most used feature and it usaully gets crowded so multi desktop like ubuntu and mac would be very usful
Anonymous
January 19, 2012
The comment has been removed
Anonymous
January 19, 2012
It's time for a new post, maybe about multitasking, multimonitor or other user related topic (to mix it up a little). At least clos comment here. The latest comments are beggining to derail from OT to simply trolling. (im still waiting for a post about dynamic disk booting, in the future) Alvaro:)
Anonymous
January 20, 2012
The comment has been removed
Anonymous
January 20, 2012
The comment has been removed
Anonymous
January 21, 2012
I'd like to thank everyone for their comments and feedback. As we stated in the blog, we believe we've advanced the state of the art for Microsoft file systems in multiple dimensions, even though we currently don't support some of the features of NTFS. We have a lot of headroom for innovation in the design, and we will evolve it in stages. As usual we'll look for feedback on functionality as we do that.
Anonymous
January 21, 2012
The comment has been removed
Anonymous
January 21, 2012
The comment has been removed
Anonymous
January 22, 2012
The comment has been removed

Jaa