Understanding of Managed Folder Assistant with retention policies
Exchange includes a number of mailbox assistants to perform automated processing on the data held in mailboxes. Exchange 2010 or Exchange 2013 uses different types of assistants:
Event-based assistants These respond to events as they occur. For example, the resource booking assistant monitors requests to book rooms and lets users know whether the room is unavailable.
Throttle-based assistants These operate all the time to process data and are subject to throttling to ensure that their workload does not affect the responsiveness of a server. The Managed Folder Assistant is the best known of these assistants.
Other-known assistants are the following:
Calendar assistant Checks mailboxes for incoming meeting requests and processes the requests according to mailbox settings
Resource booking assistant Processes meeting requests for room and equipment mailboxes
Scheduling assistant Scans attendee calendars for suitable meeting slots and suggests them to meeting organizers
Junk email options assistant Copies the safelist data from user mailboxes to their Active Directory accounts so the anti-spam agents can use it
Managed folder assistant Processes retention policies for managed folders and applies retention policy tags to items in mailboxes
Exchange uses a work cycle model to define how its assistants perform work while remaining active in the background to accomplish goals over a set period. For example, you might define that the Managed Folder Assistant must process every mailbox on a server at least once in 24 hours. Given the constraints determined in a work cycle, the assistant will work out how much work it has to do over the period and create an internal schedule to do the work on a phased basis so that a smooth and predictable demand is generated on the server rather than the peaks created by the older model. The assistant monitors system conditions on an ongoing basis to make any required adjustments to perform well. If the server is under high demand for a period due to user activity, the assistant can back off its work and then speed up when server load drops.
The Get-MailboxServer and Set-MailboxServer cmdlets retrieve and set work cycle information for assistant processes. First, find out what the current work cycles are for the assistants that run on a multirole server:
Get-MailboxServer –Identity ExServer2 | Format-List *WorkCycle*
The majority of the assistant processes process the data for which they are responsible at least once daily because the work cycle period is set to one day. In effect, the goal for the assistant is to process all relevant data for all mailboxes on the server in a 24-hour period, assuming that system resources allow it and the assistants are not throttled back to allow more important processes to proceed. The checkpoint for an assistant tells it how often it should check for new objects that need to be processed, such as looking to see whether new mailboxes have been added to the databases on a server. In most cases, the checkpoint is equivalent to the work cycle period. It makes sense to have the same work cycle settings applied to all Mailbox servers in the organization. This is easily done by running the Get-MailboxServer command to fetch a collection of all Mailbox servers and then using Set-MailboxServer to set the properties.
After you apply a retention policy to a mailbox, you can either wait for the next scheduled run of the MFA or start it manually so that the new policy is applied immediately. When the MFA runs, it performs the following tasks:
It applies the tags specified in retention policies to the mailboxes covered by these policies and stamps the items in the various folders covered by the policies with the appropriate tag name and expiration date.
If a policy defines a retention or expiry period for items, it stamps a Messaging Application Programming Interface (MAPI) property (ElcMoveDate) on the items, indicating the date and time from which the retention period will start. A future run of the MFA can then use this date and time to calculate when to delete an item or mark it as expired.
It locates items in folders that are past their expiration date and takes whatever action is defined in the policy (delete, move to Archive).
The MFA works on a work cycle basis: it assesses the expected workload in terms of the number of mailboxes it has to process and then spreads out its processing across the complete window. For example, if 600 mailboxes are to be processed over three hours, the MFA creates its own internal schedule to process 200 mailboxes per hour or roughly three mailboxes per minute. In addition, a checkpoint is defined for the work cycle, at which time the MFA looks for new mailboxes that should be added to its list for processing. The default values for the work cycle and checkpoint are both 1 day, so the MFA attempts to process every mailbox in its list daily and checks for new mailboxes daily. The MFA logs event 9017
when it begins to process mailboxes, but the more important event to look for is 9025, logged when the MFA has had to skip a mailbox for some reason, such as when the mailbox is being moved to another database.
Overall, the work cycle mechanism makes more effective use of server resources in an easy and relaxed manner throughout the day and doesn’t create potential spikes in demand.
You might want to run the MFA immediately, perhaps to apply a policy to a group of users for the first time. Forcing immediate execution for a selected mailbox is useful when you start to apply policies to mailboxes and want to gauge the effect of the policy by examining the contents of a known mailbox. This might be easier than asking users what happened to items in their mailboxes. To force processing for a selected mailbox, specify its name with the –Identity parameter:
Start-ManagedFolderAssistant –Identity 'John'
Note: We should avoid running Start-managedFolderAssistant against bulk of users during the production hours, which may lead to some performance issues on the mailbox server.
The time required for the MFA to complete its run depends on the number of mailboxes and the number of items to which it has to apply retention policies. A run on a small server that hosts a few hundred mailboxes will complete in a couple of minutes unless the mailboxes hold thousands of items. However, processing 6,000 mailboxes, each of which holds an average of 25,000 items, could take several hours, especially if the server is loaded with other tasks or the policies cause a heavy I/O load because many items are permanently removed or moved from primary to archive mailboxes. You should monitor the first runs of the MFA on a server to gauge the scope of the activity and how long a normal run takes to complete.
After the MFA has applied a new policy to a mailbox, the next time the user connects to the mailbox with a client that supports retention policies, user will see that retention tags are shown on items and the retention policy options are visible. Another important point that you should understand is that if you apply a retention policy that contains a default policy tag, the MFA stamps the default tag on every item in the mailbox. This action forces Outlook to download the complete contents of the mailbox the next time the client connects and synchronizes with Exchange. Such a massive synchronization has the potential to flood a network and keep clients fully occupied for a long time. Including a default archive tag in a policy does not have the same effect because the MFA does not stamp every item with this tag.
User notified with retention policies
The first evidence users see that their mailbox has been assigned a retention policy is when retention information appears when they look at messages. This information is based on the tag stamped on an item by the MFA. Thirty days or so before an item’s retention period expires, Outlook and Outlook Web App begin to inform users that they might want to take action to preserve the item; otherwise, the MFA will process it again and delete it or move it into the archive, depending on the action required by policy. These warnings are visible when a message is opened or shown in the message preview shows how Outlook Web App advises that a message has five days before it expires as the result of the retention policy tag placed on the Inbox. The user now has the choice either to take action or to let the message expire, in which case the MFA will process whatever action is defined in the tag.
Outlook Web App display warning that an item will expire after certain specific deadline
Users have two options. First, they can move the item to a different folder and so remove it from the influence of the retention policy tag that applies to Inbox items. After it is moved, the item is governed by the default policy tag defined in the retention policy that applies to the mailbox, if one exists, or by an explicit tag that is applied to the folder and therefore inherited by all items that are added to the folder. If neither of these conditions exists, the item is left untagged and is therefore not subject to processing by the MFA
The second option is to apply a personal tag to the item. Users can choose from any of the personal tags defined in the retention policy applied to their mailbox by right-clicking an item and then selecting the personal tag to apply.
You won’t see the user interface for retention policies unless a policy is applied to your mailbox and the MFA has processed the mailbox to apply the retention policy. As part of this process, the MFA creates a hidden folder-associated item (FAI) in the user mailbox that clients use to populate the retention tag picker. If the policy is subsequently updated with a new retention or archive tag, the new tag will not be visible to clients until after the MFA next processes the mailbox.
After a personal tag has been applied to an item, the item is no longer subject to the provisions of either the folder policy or the default policy because an explicit tag always takes precedence over a tag placed on a folder. The personal tag also remains with the item if it is moved to another folder or into the personal archive. If users want to impose a different retention policy on the item, they must replace the existing tag with a new personal tag.
Hidden working of MFA
When the MFA first processes a mailbox, it creates a hidden FAI item of type IPM.Configuration.MRM in the Associated Contents table of the Inbox folder. The item stores MRM configuration in XML format in a PR_ROAMING_XMLSTREAM property. (PR = property; Roaming means that the information is available wherever the client connects, and XMLSTREAM indicates the type.) The MFA then updates the item any time a change occurs in the retention policy assigned to the mailbox, such as when a new personal tag is added to the policy. If a new retention policy is assigned, the MFA updates the item with details of that policy.
Exchange uses FAIs as a means to hold configuration and other data it needs to store in a mailbox but does not want to reveal to users when they run clients such as Outlook. The item the MFA creates holds details of the retention policy that has been assigned to the mailbox, including details of the retention tags the client needs to display in its user interface. If the item does not exist in the mailbox, a client remains unaware that a retention policy is in force. This is why the MFA has to process a mailbox before the client user interface is populated with details of the policy. In addition to the tags provisioned through the retention policy, the item also holds information about any personal tags the user has selected for his own use through Outlook Web App Options, you cannot see this information with a normal client, but you can with the MFCMAPI utility by opening a mailbox that has been assigned a retention policy, opening the Inbox folder, and then opening the associated contents table before finally identifying the MRM item shows the details of the MRM configuration item as exposed by MFCMAPI, and you can clearly see details of the retention tags specified in the policy in the text box paste it in notepad and review the tags information.
The MFA uses a number of MAPI properties for mailbox folders and items during its processing. These properties are:
PR_START_DATE_ETC For a folder, this property holds the GUID of the retention tag that administers a folder. Every tag is assigned a GUID (stored logically in the GUID property and visible with EMS). This GUID is stamped into this property so the MFA knows which retention tag, action, and period applies to the folder. For an item, the property holds a composite value containing the default retention period plus the start time for the retention period. By adding the retention period to the start time, the MFA determines the expiry date.
PR_RETENTION_PERIOD This is the number of days items should be retained in a folder. When a personal tag is applied to an item, this property is set. However, if the property does not exist, the item (or subfolder) inherits the retention period from its parent folder.
PR_RETENTION_DATE This is the calculated date when an item’s retention period expires. Clients display this information to users. When clients work in online mode (such as in Outlook Web App), Exchange takes care of calculating this value, otherwise clients that work in offline mode perform the calculation.PR_RETENTION_FLAGS For a folder, this flag indicates whether the retention tag is inherited from the parent folder. If the user sets an explicit tag on a folder, the value is nonzero.
PR_POLICY_TAG This exists only for items and contains a binary encoded value pointing to the policy tag that governs the item.
PR_ARCHIVE_TAG This exists only for items and points to the archive tag that governs an item.
PR_ARCHIVE_PERIOD This exists only when an item has been stamped with an explicit archive tag to contain the number of days in the archive retention period.
PR_ARCHIVE_DATE This contains the date when an item will be archived.
You can view these properties through MFCMAPI shows the value of the PR_RETENTION_DATE property for an item.
Viewing the PR_RETENTION_DATE property for an item
Retention date calculation
Date calculation is a very important part of the work the MFA does when it processes items. The MFA has to understand the date that should be used to calculate the age of the item (when the item first appears in the mailbox) and either the date when the item will expire or the date when the MFA has to take the action the retention policy requires (deleting the item or moving it into the archive).
Given the nature of email, most items enter a mailbox when they are delivered in the Inbox or are sent and stored in Sent Items. Items often stay in these folders for much of their lifetime and, if they do stay, will probably come under the control of either a folder tag (if defined for the folder) or a default tag (that applies to untagged items in the mailbox). The exception, of course, occurs when a user explicitly applies a personal tag to an item in the Inbox or Sent Items.
When the MFA runs, it examines items in the mailbox and determines what processing is required. Assume that an Inbox item exists that was delivered on 1 April 2013 and that a folder tag that requires items to be deleted and allow recovery after 30 days is applied to the Inbox. MFA stamps this item with a start date of 1 April 2013 by updating the PR_START_DATE_ETC property. It then calculates the expiration date by adding 30 days to the start date; 1 May 2013 is the result. This date is then stamped on the item by updating the PR_RETENTION_DATE property. On or after 1 May 2013, the MFA returns to the item and discovers that its retention period has expired. (If you view the item by using Outlook before the MFA returns, you’ll see the item marked as Expired.) When the MFA processes an expired item, it takes the retention action defined in the tag. In this case, it moves the item into the Recoverable Items folder, where it will remain until the retention period for the mailbox expires, at which time the MFA will remove the item permanently from the database.
This is the simplest kind of processing the MFA is called on to perform, but it happens for a large percentage of items because many users leave messages in the Inbox and Sent Items folders. People who let items accumulate in these folders are often called pilers because they create piles of messages and then rely on client search facilities to locate specific items when required.
Broadly speaking, the other kind of user behavior is represented by the filers, or people who move items from the Inbox and Sent Items into more appropriate folders in which the items form collections that represent important categories of work (or play) as viewed by the user. Or indeed the user attempts to keep the Inbox and other folders under some form of control by deleting unwanted items on a regular basis.
Comments
- Anonymous
January 01, 2003
Hi Jeff,Managed folder workcycle is throttled from Exchange 2010 and it won't use intensive resource unless we run the start-Managedfolderassistants manually against all the mailboxes on the server. Please avoid running the start-Managedfolderassistants during production hours. Managed Folder Assistant creates its own internal schedule to process all the mailboxes on the server. For example, if 600 mailboxes are to be processed over three hours, the MFA creates its own internal schedule to process 200 mailboxes per hour or roughly three mailboxes per minute. - Anonymous
July 16, 2014
Super Article, great information..!! - Anonymous
August 18, 2014
Great Article !!! - Anonymous
November 20, 2014
Thanks - Anonymous
February 05, 2015
Very nice article, this helped me a lot when designing these policies.
Only question is, there seems to be a little detail missing about the Managed Folder Assistant work cycle. All documentation suggests that it will spread its work over the entire work cycle duration, but when I applied a policy to 600 mailboxes, it processed them all within two hours time. (Which caused a noticeable performance impact.)
Does the Managed Folder Assistant ignore the schedule for the first cycle? - Anonymous
February 12, 2015
wow fantastic write up and throughly enjoyed!!!! Cleared up everything i wanted to know and so useful..,. thanks for posting it really helped!!! - Anonymous
April 24, 2015
My experience shows that the MFA actually just runs once every 24 hours. Basically it runs when you first boot up the server and then once every 24 hours after that. It does all of the mailboxes at once, not spread out throughout the day. So now I make sure to reboot our nodes an hour apart so that they at least aren't all processing at the same time. You might want to double-check and make sure that it works the way you intend.
This is Exchange 2013 CU7. - Anonymous
June 25, 2015
Great article! very thorough! - Anonymous
August 31, 2015
Thanks for sharing...very elaborated and made simple! - Anonymous
September 09, 2015
Great Information. :) - Anonymous
October 23, 2015
Thanks! Great article! - Anonymous
November 12, 2015
How long does it take for Retention comment to update with new message expiry date once User applies a personal tag?