Determine the True amount of WhiteSpace in an Exchange Database
So we come across this from time to time so I thought it would be interesting to talk about… besides I needed something to show that I still cared about my blog.
So the question is…
“If I add up all the space used by mailboxes, why doesn’t roughly match the size of the database? And what does that number represent in the 1221 event I receive?”
First looking at the 1221 event:
Event: 1221
Source: MSExchangeIS Public
Type: Information
Category: General
Description:
The database "<storage_group>\<mailbox_store> (<server_name>)" has
<nnn> megabytes of free space after online defragmentation has
terminated.
The number presented is derived from the number of free pages that are available at the database root, the messages table, folders, and the attachments table. Statistically over the years we know these tables represent nearly 90% of the space used in the database.
To understand the reasoning behind this, a quicker primer on space management in the database is needed. Here are the some aspects to keep in mind:
· The fundamental space allocation is a 4k page.
· A series of 16 consecutive pages is the amount of space that is allocated to a table when space is requested. (not just one page at a time)
· A page cannot store data from two different tables or belong from two different trees.
There are two levels of space management that occurs in the database. One is at the global database level (space available to all tables in the database store in ranges of 16 consecutive pages) and at the B+tree level. If you are unfamiliar with the concepts of B+Trees, it may be easier to think of them as a Table. Tables maintain the state of pages they own and do not free up pages to the overall database until 16 consecutive block of pages is freed up. The reason for this is efficiency… by reusing pages that are physically adjacent to existing pages we minimize the effort necessary to perform read and write operations. So by holding onto pages we ensure that a larger majority of pages are physically adjacent.
In a database, we could literally have thousands of tables (at least one for each folder in every mailbox). As I mentioned, we know statistically the messages and attachment tables represent 90% of the space used in the database. Likewise they would have the also have the greatest percentage of free space, or commonly referred to as white space in the database.
In order to arrive at a calculation for the amount of available space an individual table possesses, we have to a lot of work interrogating the space usage information of the table. If we were to do this for thousands of tables, it would take incredible amount of time to perform this overall calculation. Given the greatest majority of space statistically exists at the root, in the messages table, and attachments table and the amount of time it takes to calculate space, we only take consideration this information when logging the 1221 event.
So there is also a lot of other stuff hidden away in the database that mailboxes are not charged for. For example there are search folders, indexes, and system tables that allow Exchange to operate. In addition, we have already stated that a page cannot belong to multiple tables, so if a page does have free space, it can only store new records for that table and further more, it will only contain records meet the criteria to be stored on that page. If you stored every record beginning with the letter “A” on one page and every record beginning with “B” on a second page, there is no reason a record beginning with “A” should appear on the page containing all the “B” values. Therefore there can be space available on a individual page that also isn’t taken take into consideration when calculating the 1221 event since we only add up the empty/free pages in the messages and attachment tables.
Online Defragmentation takes aim at “compressing” records to fewer pages in a particular table. For example, if a table owned 100 pages, each half full, online defrag will attempt to free approximately half of the pages. When 16 contiguous pages are freed, the space is released back to the database itself to be used by other tables in the database. It is only the pages that are freed up in the messages and attachments tables that eventually get reported.
If you want to take a more in-depth look at the space used in the database, then a space dump using ESEUTIL is necessary. For example (note the output is truncated for brevity):
C:\Program Files\Exchsrvr\MDBDATA>..\bin\eseutil /ms priv1.edb
Microsoft(R) Exchange Server Database Utilities
Version 6.5
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating FILE DUMP mode...
Database: priv1.edb
****************************** SLV SPACE DUMP ******************************
Chunk Free Res Del Com |------------ Used ------------|
============================================================================
512 110 0 0 402 *************************
1024 0 512 0 0 ********************************
1536 512 0 0 0
2048 512 0 0 0
2560 512 0 0 0
3072 512 0 0 0
3584 512 0 0 0
4096 512 0 0 0
4608 118 0 0 394 ************************
5120 0 0 0 512 ********************************
5632 0 0 0 512 ********************************
6144 0 0 0 512 ********************************
6656 0 0 0 512 ********************************
7168 0 0 0 512 ********************************
7680 0 0 0 512 ********************************
8192 238 0 0 274 *****************
============================================================================
TOTALS:
Free: 3538
Reserved: 512
Deleted: 0
Committed: 4142
Unknown: 0
-------------
8192
****************************************************************************
******************************** SPACE DUMP ***********************************
Name Type ObjidFDP PgnoFDP PriExt Owned Available
===============================================================================
priv1.edb Db 1 1 256-m 8960 116
<SLV Avail Map> SLV 6 33 32-m 32 29
<SLV Owner Map> SLV 7 65 32-m 80 3
1-122 Tbl 75 301 8-s 8 3
MsgFolderIndex7 Idx 77 302 1-s 1 0
MsgFolderIndexPtagDel Idx 80 305 1-s 1 0
MsgFolderIndexURLComp Idx 79 304 1-s 1 0
RuleMsgFolderIndex Idx 78 303 1-s 1 0
1-23 Tbl 61 236 2-m 7778 16
<Long Values> LV 222 237 1-m 7691 8
1-24 Tbl 63 257 8-s 8 3
MsgFolderIndex7 Idx 65 258 1-s 1 0
MsgFolderIndexPtagDel Idx 68 261 1-s 1 0
MsgFolderIndexURLComp Idx 67 260 1-s 1 0
RuleMsgFolderIndex Idx 66 259 1-s 1 0
1-33 Tbl 336 8821 8-m 14 1
<Long Values> LV 342 8826 1-m 5 2
?T668f-T6654+Q3f88 Idx 354 8827 1-s 1 0
MsgFolderIndex7 Idx 338 8822 1-s 1 0
MsgFolderIndexPtagDel Idx 341 8825 1-s 1 0
MsgFolderIndexURLComp Idx 340 8824 1-s 1 0
RuleMsgFolderIndex Idx 339 8823 1-s 1 0
Folders Tbl 8 97 9-m 100 10
<Long Values> LV 105 243 1-s 1 0
*T668f+Q6749+S3001+Q6 Idx 60 232 1-s 1 0
?T668f+Q6749+S3001+Q6 Idx 59 109 1-m 9 0
Folders Fid to Pfid I Idx 17 108 1-m 9 2
FoldersIndex10 Idx 13 102 1-s 1 0
FoldersIndex13 Idx 14 103 1-s 1 0
FoldersIndex5 Idx 9 98 1-m 5 1
FoldersIndex6 Idx 10 99 1-s 1 0
FoldersIndex7 Idx 11 100 1-m 7 0
FoldersIndex8 Idx 12 101 1-s 1 0
Hashed URL Name Index Idx 15 104 1-m 5 0
ScopeFIDs DeleteTime Idx 16 105 1-s 1 0
Mailbox Tbl 21 140 2-m 10 2
MailboxIndex2 Idx 22 141 1-s 1 0
MailboxIndex3 Idx 23 160 1-s 1 0
Msg Tbl 19 112 2-m 194 63
<Long Values> LV 106 359 1-m 14 0
-------------------------------------------------------------------------------
253
First Lets take a look at this row. There is two columns, the owned and available. The owned value is the number of pages in the database that contains *some* data. The next value, available, represents the number of free pages available at the database root level that can be distributed to tables as they need space to grow.
******************************** SPACE DUMP ***********************************
Name Type ObjidFDP PgnoFDP PriExt Owned Available
===============================================================================
priv1.edb Db 1 1 256-m 8960 116
Next lets look at the attachments table. You see below two rows, one called 1-23 which is the primary B+tree and then a <Long Values> which contains records or fragments or records that are too large to be stored in the primary table are stored. The value under the owned column for 1-23 (7778) represents the total number of pages owned by this table. This encompasses the long values and any other secondary indexes that could be in use by this table. The next value is the available pages (16) that can be reused by this individual table (but not by indexes or Long Value tree). Of the 7778 pages in use by this table, the long value tree is occupying 7691 of them and has 8 pages available.
******************************** SPACE DUMP ***********************************
Name Type ObjidFDP PgnoFDP PriExt Owned Available
===============================================================================
1-23 Tbl 61 236 2-m 7778 16
<Long Values> LV 222 237 1-m 7691 8
Next lets take a peek at a standard folder in use by someone in the database. The folder is 1-33. (Internally we represent tables as numbers.) This would typically represent someone’s INBOX folder or some other folder the user may have created. Here we have a primary entry, 1-33, that is the table itself. Listed under this row are associated Long Value and indexes to this table. As you can see, overall this table occupies 14 pages with 1 page available to the primary table. The Long Values owns 5 of the total 14 pages owned by this table and has 2 free pages while the table entry only has one. The thing to keep in mind is the Owned value represents ALL pages in use by the table, indexes and LV trees and the Available only represents the number of pages available to that individual index/LV/Table and is not cumulative.
******************************** SPACE DUMP ***********************************
Name Type ObjidFDP PgnoFDP PriExt Owned Available
===============================================================================
1-33 Tbl 336 8821 8-m 14 1
<Long Values> LV 342 8826 1-m 5 2
?T668f-T6654+Q3f88 Idx 354 8827 1-s 1 0
MsgFolderIndex7 Idx 338 8822 1-s 1 0
MsgFolderIndexPtagDel Idx 341 8825 1-s 1 0
MsgFolderIndexURLComp Idx 340 8824 1-s 1 0
RuleMsgFolderIndex Idx 339 8823 1-s 1 0
At the end of the dump is the a line similar to the following. It contains a summation of the total number of pages that are available throughout all the tables. By taking this value and multiplying it by 4096, you arrive at the true amount of whitespace in the database.
-------------------------------------------------------------------------------
253
So the summation of all the mailboxes do match because the size of the database because while only 100k of information total is stored in a mailbox, the tables that actually store the data could occupy a larger amount of space depending upon the density of the pages at the time and if pages have been freed up to the database root. I am not saying this is the answer to every discrepancy, but in the majority of cases this has proven to be the case.
I hope this was informative and if there is something I need to drill down on further, please let me know in your feedback.
Comments
Anonymous
April 09, 2004
Great article, but can you go some deeper in the process of Online maintenance and the online defragmentation ? As far as I can read, the maintenance is something hardcoded which can not be tuned in any way. But this process has influence on the amount of whitespace. When the store is busy, maintenance is stopped from working. So I understand from this that if I have a busy store, than probabely I have more whitespace than what is reported, if reported as the maintenance can take days/weeks to finish. Can you enlighten this ? TIA.Anonymous
April 15, 2004
Erik, I started a post to help understand IS maintenance... hopefully it will answer your questions... and others as well.Anonymous
June 10, 2004
Hi Jeremy,
Great article, it might actually explain a situation we've been seeing. In our case we have 5x DB in a storage group (SG). One of our storage groups is starting to run low on free disk space so as a temporary workaround (untill our next outage where we can take the databaes offline to defrag) we've been moving mailboxes to other SG's to free up whitespace.
But we've been noticing that even though we've moved mailbox data and created a few GB of white space per DB the amount of free disk space on our SG is still decreasing. Sooo.. if I understand this article correctly the DB could still be gowing as other tables (not the ones used to generate 1221 events) are still growing. Please let me know if this sounds right to you.
Many thanks for the informative article.
M@Anonymous
June 10, 2004
Things to consider regarding your situation:
1) How long do you have Mailbox retention set for (default 30days after being deleted)?
2) Is ISMaintenance / online defrag completeing a full pass? If not the mailbox space may not be getting freed properly.
Offline defrag is the only way to ensure the space is completely freed up.Anonymous
December 22, 2008
PingBack from http://telnet25.wordpress.com/2008/12/22/offline-defrag-exchange-maintenance/Anonymous
January 20, 2009
PingBack from http://www.hilpers.com/456001-exchange-2003-informationsspeicher-ueberwachenAnonymous
January 21, 2009
PingBack from http://www.keyongtech.com/1031644-determining-percent-utilized-of-aAnonymous
May 25, 2009
PingBack from http://restartis.wordpress.com/2009/05/26/does-exchange-2007always-use-whitespace-available-in-a-database/