Understanding Online Maintenance, Offline Defrag and their interrelationship with the Deleted Items Retention Period

[Today’s post comes to us courtesy Nandan Sheth.]

By default, an SBS 2003 install limits the Exchange database size limit to 16 GB. This was a cause of grief for a lot of users. So, with Service Pack 2 for SBS 2003, the default limit was increased to 18 GB, with the provision of using registry keys to bump this limit up to 75 GB. However, we occasionally come across users who have databases that are over 75 GB. The only option available once you have hit this limit is to run an offline defrag on the database, and hope that doing so shrinks the database to a manageable, usable size limit. Running a defrag is documented on KB 328804. The procedure is pretty straight forward, and pretty much always succeeds. However, there may be times when you see the defrag complete, and no space being freed up. Before we get into why you don’t see free space after a successful defrag, let’s take a quick look at what a defrag does.

The Exchange databases are organized in the form of pages. Up until Exchange 2003, each “page” was 4 KB in size. When data is committed into the databases, the pages are sequentially filled. When information is deleted by users, the page that holds this data is removed. But the space freed on the database is not automatically reclaimed. This reclamation is completed when a defrag identifies and fills the free space.

Online Maintenance: Also known as Online Defragmentation, Online Maintenance is run every day. On a default install, the window for Online Maintenance is set from 1:00 AM to 6:00 AM everyday. When the system starts an online maintenance cycle, it scans the database, and reports how much free space is available in the form of reclaimable empty pages. This information is made available through the event viewer. When the scan is complete, Event ID 1221 is generated in the event logs, which reports the total reclaimable space available. Once we know how much free space is available, we can run an offline defrag, recover this free space, and make it available to the file system.

Online Maintenance can be configured from the properties of the database on the Exchange System Manager:

Clicking on the Customize button will bring up a box where you can configure the time:

You can use this box to schedule the scan for your online maintenance cycle.

Retention Period: The retention period specifies how long Exchange will keep items that users have deleted. Upon deleting an item, Exchange marks the item for complete removal based on the retention period. The default retention period is set to 30 days:

The retention period can be modified to suit your requirements.

Events logged during an online maintenance cycle: When the server completes an online maintenance cycle, you should see the following events logged in the application logs:

Date: Computer_Date
Time: Computer_Time
Type: Information
User: N/A
Computer: Computer_Name
Source: MSExchangeIs Mailbox Store
Category: General
Event ID: 1206
Description: Starting cleanup of items past retention date for Item Recovery on database “First Storage Group\Mailbox Store (Computer_Name)”.

Date: Computer_Date
Time: Computer_Time
Type: Information
User: N/A
Computer: Computer_Name
Source: MSExchangeIs Mailbox Store
Category: General
Event ID: 1207
Description: Cleanup of items past retention date for Item Recovery is complete for database “First Storage Group\Mailbox Store (Computer_Name)”.

Start: 0 items; 0 Kbytes
End: 0 items; 0Kbytes

Date: Computer_Date
Time: Computer_Time
Type: Information
User: N/A
Computer: Computer_Name
Source: MSExchangeIs Mailbox Store
Category: General
Event ID: 9531
Description: Starting cleanup of deleted mailboxes that are past the retention date on database “First Storage Group\Mailbox Store (Computer_Name)”.

Date: Computer_Date
Time: Computer_Time
Type: Information
User: N/A
Computer: Computer_Name
Source: MSExchangeIs Mailbox Store
Category: General
Event ID: 9535
Description: Cleanup of deleted mailboxes that are past the retention date is finished on database “First Storage Group\Mailbox Store (Computer_Name)”.

0 deleted mailboxes (0 KB) have been removed.
0 deleted mailboxes (0 KB) have been retained.

Date: Computer_Date
Time: Computer_Time
Type: Information
User: N/A
Computer: Computer_Name
Source: ESE
Category: Online Defragmentation
Event ID: 700
Description: Information Store (PID_for_Store.exe) First Storage Group: Online defragmentation is beginning a full pass on database ‘C:\Program Files\Exchsrvr\mdbdata\priv1.edb’.

Date: Computer_Date
Time: Computer_Time
Type: Information
User: N/A
Computer: Computer_Name
Source: ESE
Category: Online Defragmentation
Event ID: 701
Description: Information Store (PID_for_Store.exe) First Storage Group: Online defragmentation has completed a full pass on database ‘C:\Program Files\Exchsrvr\mdbdata\priv1.edb’.

Date: Computer_Date
Time: Computer_Time
Type: Information
User: N/A
Computer: Computer_Name
Source: MSExchangeIs Mailbox Store
Category: General
Event ID: 1221
Description: The database “First Storage Group\Mailbox Store (Computer Name) has 0 megabytes of free space after online defragmentation terminated.

It is recommended you monitor these logs to determine if you need an offline defrag of the Exchange databases.

What to do if an offline defrag does not free up the expected disk space?

If, after running an offline defrag on the database, we still don’t see space being freed up, then we have items that are in the retention period, and are not allowing the system to reclaim the expected space. In this case, the first thing to do is make sure that the users have not deleted anything they might want to recover. Once we are sure that there is nothing the users need to recover, follow the procedure below:

1. Set the system to run online maintenance at all times. Use the customize button under the maintenance interval section on the database tab of the properties for the Mailbox store.

2. From the limits tab, set both retention options to 0.

3. Dismount the Mailbox store and then mount it.

4. Within 15 minutes of mounting the store, the system will start an online maintenance cycle.

5. Once this completes, dismount the store and run an offline defrag on the database.

Note: Please remember to change the settings for the retention period back to default or a value higher than zero to make sure you don't lose important items that have been deleted accidentally, after the maintenance is over and the expected disk space has been freed.

Retention period negatively effecting an offline defrag is documented on KB 258951. Please note that even though the article was written for Exchange 5.5, the concept remains the same.

For a deeper understanding of the Exchange Storage Architecture, you can refer to the Exchange 2003 technical reference. This information is available here.