Can I really use JBOD storage with Exchange 2010?
I just did an Exchange 2010 architectural design session for a university in Ohio and this was a key question they asked me. The answer is yes, you truly can leverage JBOD (Just a Bunch Of Disks) storage with large TB SATA drives for Exchange 2010 now. Microsoft IT has deployed Exchange 2010 with JBOD storage and we are benchmarked as some of the heaviest mail profiles available. This proves using JBOD storage with Exchange 2010 can handle even the highest of mail loads/stress and IOPS per user (.11+).
Why would I consider JBOD storage for Exchange 2010?
Storage is going to be your largest expense in deploying Exchange 2010. If you can reduce your storage costs by anywhere from 90%+ to 40%, maintain the same performance, and significantly grow your mail quota per user it becomes a no brainer. This is the same conclusion MS IT came to. They also did not need to grow their storage staff to maintain JBOD.
How is this possible?
Cheap SATA and JBOD storage are possible since, with Exchange 2010, the product team managed to significantly increase and optimize database performance which resulted in reducing the IOPS per user by another 70% over Exchange 2007. This translates to the use of cheaper storage can be leveraged for equivalent performance of higher cost storage used with Exchange 2007.
For example, a 7200RPM 1TB SATA drive will provide an estimated 75-80 IOPS per spindle. This now becomes a viable spindle type for Exchange 2010 with the additional 70% IOPS/mailbox reduction. If you add in our new DAG availability, where we replicate and failover at the database layer, JBOD becomes a viable option as well.
What do I have to have in place in order to use JBOD storage?
The recommended DAG architecture for JBOD storage is a minimum of 3 mailbox server nodes with 3 database copies per database. You dedicate one spindle per database and associated transaction logs. If a spindle fails, since it is non-RAIDed, it doesn’t matter since you have TWO other good copies of the database to failover to on two other dedicated spindles.
What does a sample JBOD storage look like?
This is just a sample architecture for reference. You could use larger SATA drives now such as 2TB SATA drives to lower your spindle count. If you look closely you will see, 3 copies of any given database spread across 3 servers.
Are there tools I can use to validate my JBOD storage?
Yes, first thing I would recommend would be to use the Mailbox Calculator to level set that JBOD is a good fit for your organization. Note: Make sure you specify 3 or more database copies or the calculator will not recommend JBOD as a viable option. The nice part about the calculator is you can play with different SATA spindles (7200 RPM, 3.5”, 1TB, 2TB, etc) to arrive at different results.
Sample Mailbox calculator data showing JBOD is a fit:
Once you have purchased JBOD storage, I would leverage the following stress tools:
JetStress
LoadGen
to benchmark your storage and hardware. Grab those tools here.
Is there a storage matrix I can use to reference?
I found this useful storage matrix put together from the product team:
Exchange 2010 Storage Guidance |
Stand Alone |
Database Availability Group: 2 nodes, 2 Database copies |
Database Availability Group: 3+ nodes, 3+ Database copies |
Storage Type |
|||
Direct Attached Storage (DAS) |
Supported |
Supported |
Supported |
Storage Area Network (SAN): iSCSI |
Supported. Best Practice = Do not share physical disks backing Exchange data with other applications. |
Supported. Best Practice = Do not share physical disks backing Exchange data with other applications. |
Supported. Best Practice = Do not share physical disks backing Exchange data with other applications. |
Storage Area Network (SAN): Fibre Channel (FC) |
Supported. Best Practice = Do not share physical disks backing Exchange data with other applications. |
Supported. Best Practice = Do not share physical disks backing Exchange data with other applications. Best Practice = Do not place both database copies on the same physical spindles. |
Supported. Best Practice = Do not share physical disks backing Exchange data with other applications. Best Practice = Do not place both database copies on the same physical spindles. |
Network Attached Storage (NAS): SMB |
Not Supported |
Not Supported |
Not Supported |
Physical Disk Type |
|||
SATA |
Supported, requires battery backed caching array controller for data integrity |
Supported, requires battery backed caching array controller for data integrity |
Supported, requires battery backed caching array controller for data integrity |
SAS |
Supported |
Supported |
Supported |
FC/FATA |
Supported |
Supported |
Supported |
SSD (Flash Disk) |
Supported |
Supported |
Supported |
Physical Disk Write Caching (enabled) |
Not Supported |
Not Supported |
Not Supported |
Storage RAID |
RAID recommended |
RAID recommended |
RAID optional |
EDB Volume |
RAID5/6, RAID10, RAID1 |
RAID5/6, RAID10, RAID1 |
JBOD, RAID5/6, RAID10, RAID1 |
Log Volume |
RAID1, RAID10 |
RAID1, RAID10 |
JBOD, RAID1, RAID10 |
Disk Array RAID Stripe Size (kb) |
256KB |
256KB |
256KB |
Storage Array Cache Settings |
75% Write Cache, 25% Read Cache (with Battery Backed Cache) |
75% Write Cache, 25% Read Cache (with Battery Backed Cache) |
75% Write Cache, 25% Read Cache (with Battery Backed Cache) |
Database/Log file placement |
|||
Database/Log Isolation |
Best Practice (for recoverability) = separate database file (.edb) and logs from same Database on to different volumes backed by different physical disks |
Database file (.edb) and logs from same Database can share same volume and same physical disk. |
Database file (.edb) and logs from same Database can share same volume and same physical disk. This is a best practice for JBOD/RAID'less storage scenario where one or more volumes store the edb and log files backed by the same physical disk. |
Database Files/Volume |
Based on backup methodology |
Based on backup methodology |
RAID = based on backup methodology, JBOD = one DB file/volume is recommended |
Log Streams/Volume |
Based on backup methodology |
Based on backup methodology |
RAID = based on backup methodology, JBOD = one log stream/volume is recommended |
Windows Disk Type |
|||
Basic Disk |
Recommended |
Recommended |
Recommended |
Dynamic Disk |
Supported |
Supported |
Supported |
Partition Type |
|||
GUID Partition Table (GPT) |
Recommended |
Recommended |
Recommended |
Master Boot Record (MBR) |
Supported |
Supported |
Supported |
Partition Alignment |
Windows 2008 Default: 1MB |
Windows 2008 Default: 1MB |
Windows 2008 Default: 1MB |
Volume Path |
Drive Letter or Mount Point (mount point host volume must be RAIDed) |
Drive Letter or Mount Point (mount point host volume must be RAIDed) |
Drive Letter or Mount Point (mount point host volume must be RAIDed) |
File System |
NTFS support only |
NTFS support only |
NTFS support only |
NTFS Defragmentation |
Not required, not recommended |
Not required, not recommended |
Not required, not recommended |
NTFS Allocation Unit Size |
64KB for both edb and log volumes |
64KB for both edb and log volumes |
64KB for both edb and log volumes |
NTFS Compression |
Not Supported for Exchange Database files |
Not Supported for Exchange Database files |
Not Supported for Exchange Database files |
NTFS Encrypted File System (EFS) |
Not Supported for Exchange Database files |
Not Supported for Exchange Database files |
Not Supported for Exchange Database files |
Windows Bitlocker (volume encryption) |
Supported for all Exchange database and log files |
Supported for all Exchange database and log files |
Supported for all Exchange database and log files |
More on storage planning here and here.
DAG samples here.