Content Conversion Issue with Exchange & Entourage
Lately a lot of our professional and premier customers are running into the 'content conversion' issue due to mailbox moves they perform for Entourage users. I will be talking about that in this blog.
Issue:
After you move an Entourage 2004 user's mailbox on server side using 'Move Mailbox' wizard in Exchange 2003 Server, you see performance issues on your Exchange Servers, most prominent of them is high number of disk Input/Output operations. This happens due to content conversion (MAPI to MIME) which takes place on server in real time, when those moved mailbox users access their mailbox content using Entourage.
KB Article:
This has been a known issue since long for other MIME clients like POP3 & IMAP (Microsoft Outlook is one of them) but its being exposed now more than ever by Entourage - the most popular MIME based Exchange client, which uses WebDAV (HTTP) for Exchange mailbox access. Now is a good time to read the KB article on this issue to properly familiarize yourself. I will then try to emphasize important parts in this blog.
329067 Information about performance issues that occur when you use POP3, IMAP, or WebDAV clients, such as Entourage, after you move mailboxes in Exchange Server 2003 or in Exchange 2000 Server
Details:
Let's go thru some important points and analyze them side by side.
1. Exchange server stores your mailbox content in a streaming store (.stm) file (a file to store content in MIME format) if you are an Entourage user, which is a MIME client. So let's say, John Doe got a new mailbox on an Exchange 2003 Server in his company, he is an Entourage 2004 for Mac user. He configured an Exchange account in Entourage (he specified a front-end Exchange Server name in his account settings) and connected to it, started using it for e-mail exchange with internal and external people. Exchange 2003 Server will keep all his mailbox contents in the 'stm' file.
2. Now one fine day Exchange Server Administrator decided to move John's mailbox to another database or server (as part of mailbox maintenance or planning for new server/storage, etc.). As per KB 329067 when an Entourage user's mailbox is moved on server side using 'Move Mailbox' wizard, Exchange Server performs that move by converting the content from MIME to MAPI and after that conversion the mailbox content is stored in an Exchange database (.edb) file (a file to store content in MAPI format). This is the store file format which Exchange uses by default for MAPI clients (like Microsoft Outlook 2003). This content conversion as a result of mailbox move is 'by design', i.e. that's how the 'Move Mailbox' feature was designed to work in Exchange Server. So after the move all items in John's mailbox have been converted into MAPI format.
3. John is still using Entourage 2004 and is totally unaware of the move. As he connects to a front-end server, he is not concerned about what's going on with his mailbox on back-end server. This is the beauty of connecting to a front-end. He continues to use Entourage and may see some performance lag especially when he works with his e-mails in Inbox, which has large attachments (like if John is a Graphic Designer and sends/receives large image files). This is due to the fact that every single item in John's mailbox, which went thru the 'mailbox move' operation, is now in MAPI format. And Entourage (a MIME client) does not recognize MAPI format, so Exchange Server does a content conversion 'on the fly' for Entourage and provides it a converted copy of that item in MIME format, which is later used by Entourage user. This happens on demand for a message or item when user access that in Entourage. The conversion is not persistent, so it will happen again if the same item is accessed again by Entourage user.
4. The 'content conversion' happens on Exchange server in memory (RAM) and a temporary folder on hard drive, which results in lots of disk I/O operations. So for instance, if you moved 50 Entourage users and all of them have large items in their Inboxes, the 'content conversion' for them could cause a lot of disk I/O ops. High number of unexpected disk I/O ops can impact your server's performance. It can also muster the capability to take down your IIS Server (remember Entourage is a WebDAV client and connects to your mailbox thru OWA Virtual Directories under 'Default Website' in IIS, mainly 'Exchange' & 'Public') which is also discussed in KB 329067.
5. As a result of 'content conversion', Exchange Server Administrator noticed a sharp decrease in server performance. Main indicator will be high disk I/O ops, which can easily be caught and logged thru PerfMon (see 'Exchange Troubleshooting Assistant') or Performance Monitor Wizard. If you have Outlook/MAPI clients connecting to their mailboxes on the same server, then they may start seeing RPC delays (KB: 839862).
Recommendations:
Let's talk about some ways to address this issue on client and server side.
Client Side:
First let me clearly state that there is nothing to fix in Entourage to resolve this issue, unless we change the very nature of Entourage from being a MIME client to being a MAPI client. Still there are some steps you can take to alleviate this issue.
1. Organize your Inbox: Work to make sure you have least possible number of items in your Inbox. Both number and size of items are important here. Entourage downloads the full message along with the attachment, thus messages with attachments really matter here. They are going to cause a lot of 'content conversion' cycles on server side if you access them in your Inbox in Entourage. See if you can move them to a different folder, which you can create at the same level as Inbox under the main mailbox tree. This has been the number one reason which aggravates the 'content conversion' issue and contributes most to server performance degradation.
2. Sub-folders under Inbox: Please refrain from creating any sub-folders under Inbox. Some users keep their most important messages in sub-folders they create under Inbox or archive important messages there for easy access. If you have many messages in such sub-folders and regularly access them in Entourage, that will have almost the same effect as having lots of messages in Inbox. Reason being Entourage prioritize the synching of a folder based on your clicks and if you frequently access mail in these folders then that activity will cause Entourage to bump the sync priority of those folders, resulting in more 'content conversion' on server side. Thus Entourage can get really bogged down by all that synching of messages in Inbox and its sub-folders. Move all such sub-folders of Inbox to the root of your mailbox, at the same level as your Inbox is. Custom folders at the root of mailbox get lower priority in Entourage's built-in Sync Algorithm (keep in mind you can't modify the sync algorithm) and thus Entourage will sync them only when an item changes in those folders.
3. Organize your mailbox: Please delete the items you don't need anymore. Archive old items which you don't access anymore but may still need it. It would be better to archive them by moving to local folder set (Folders on My Computer), which will free up space in your Exchange mailbox as well. In case you want to have access to them at all times, like using Outlook Web Access while traveling then move them to a custom folder created under the main root of the mailbox.
4. Outbox & Drafts Folders: Make sure there are no messages stuck in Outbox folder, which you may have tried to send earlier and they weren't delivered due to some issue. Also make sure you don't have any large sized messages saved in Drafts folder. When you send a large size message which is larger than the allowed message size limit set on Exchange Server, then Entourage will give you a warning about that and will move the message to your Drafts folder, letting you edit that message and resend later. Sometimes users don't realize that those messages will be synched back to Drafts folder in your mailbox on Exchange Server silently in the background, that sync back and forth may also add to performance issues created by 'content conversion'.
A couple of notes as well:
a. Entourage does not re-sync all mailbox content after the mailbox move on server side. Content conversion will only happen if you will access an old message which was there in your mailbox prior to mailbox move and thus got converted from MIME to MAPI due to move on sever side.
b. All new content in Entourage, i.e. mail coming after mailbox move will be in MIME format, thus there will be no content conversion when you access that mail in Entourage. This is the main reason why adverse effects of 'content conversion' issue on server's performance subside after some time.
Server Side:
Please refer to KB 329067 for all server side recommendations, I recently edited that article to make sure it's up to date with correct information. Some main points to consider are:
1. IIS on front-end (or back-end) server may also go down due to 'content conversion' issue. All connections from Entourage clients to Exchange Server go thru IIS as mentioned above.
2. The Reg Key helps IIS maintain its memory better and sustain the communication pipe between Entourage clients and Exchange Server.
3. We have seen better results with higher values for this Reg Key, like 500 to 800 MB mostly, in some cases maximum value of 1 GB helped us bring the situation under control. It all depends on how your environment is setup, number of servers and clients.
4. Always set proper mailbox and message size limits on Exchange Server for your users. There is a higher probability that this issue will hit you more and in worst shape if you don't have such limits in place.
5. When the issue subsides after some time, please reset the Reg Key value to 128 MB (default) or you can simply remove it.
6. For 'Disk Storage Planning and Recommendations', please review this article on TechNet. It's written mainly for Exchange 2007 but will be helpful for all versions of Exchange and is particularly helpful in realizing the utilization of disk resources by 'content conversion' process.
Can This Issue Be Totally Avoided?
Yes, it can be totally avoided. I worked with my peers in Exchange Admin Support Team and one of them came up with this procedure. You will need 2 servers to work on this, one is the current mailbox server (Server1) where mailboxes of your Entourage users are located currently, other is the new or existing server (Server2) to which you will move mailboxes of your Entourage users. This procedure is not meant for the scenario where you are moving mailboxes from one database to other on the same server.
1. Ensure that only Entourage clients exist in the mailbox store you want to move
2. Stop 'SMTP Service' on both the original (Server1) and new server (Server2)
3. Dismount the stores on both servers
4. Run 'eseutil /mh' command on store's '.edb' and '.stm' files, verify that these files indicate the 'Attribute State = Clean Shutdown'
5. Copy the database files offline to the new server (Server2)
6. Ensure that on the new server (Server2) a blank mailbox store is mounted
7. Delete the stores on the original server (Server1)
8. Mount blank stores on the original server (Server1)
9. Perform a move mailbox of all mailboxes on this server (Server1) to the new blank stores on the new server (Server2)
10. Dismount the stores on the new server (Server2)
11. Replace store files on Server2 with the store files from the original server (Server1)
12. Mount the stores on Server2 with these database files and ensure client access by Entourage users
13. Start 'SMTP Service' on both servers
Note: Please test this procedure before using it with your production servers. If you need help, contact PSS at Microsoft and request to talk to an Exchange Admin Support Engineer.
I hope this information will be helpful for all the Exchange Server Administrators out there who have Entourage users in their organization. Feel free to post comments and ask questions if something is not clear here. Cheers!