Condividi tramite

The mysterious Outlook Outbox

  • Articolo

Understand why messages get “stuck in the Outbox folder” in an Exchange environment.

This is a taste to whet your appetite. For the complete story, open the attached .PDF file.

(Today's post is courtesy of Scott Bradley, Principal Escalation Engineer, Office Technical Support.)


A trip back in time—the evolution of Outlook

A thorough understanding of mail submission/delivery starts with a trip back in time. Years ago, before Outlook was born, Microsoft defined some standards for messaging applications. These standards are commonly referred to as MAPI (Messaging Application Programming Interface), and detailed documentation can still be found on MSDN. At the root of the MAPI world is this idea of three main messaging components:

  1. A place to actually STORE the messages—the MAPI MESSAGE STORE
  2. A method to SEND the messages—the MAPI TRANSPORT
  3. A place to store and look up ADDRESSES—the MAPI ADDRESS BOOK

Most people are familiar with the common stores and address books in the Exchange/Outlook world. We have the mailbox and Global Address List (GAL) from Exchange, and things like the PST/OST file and Offline Address Book or Contacts Folder (which is abstracted into an address book by the Outlook Address Book service) from the Outlook side. But this middle part of MAPI, the transport layer, is less understood.

In non-Exchange scenarios, the transports are much easier to understand. Consider the POP/SMTP setup. You use the POP protocol to receive mail, and SMTP to send mail. The protocols are easy to read and understand. You can look at a network trace and see the commands and the mail “going over the wire,” etc. In the Exchange scenario, things are more complicated.

Along with this notion of the MAPI TRANSPORT being responsible for actually sending mail, MAPI includes the idea of a “spooler” which is something that manages the transports. It loads them up, checks for mail that needs to be sent and actually calls into the transport’s code to send the mail. You can think of it as the “driver” of mail delivery. Until Outlook 2002, the spooler was implemented in a completely separate executable. So when you ran Outlook (or any other MAPI application), you would also see a process named MAPISP32.EXE in your task list. The spooler ran independently of Outlook. When it came time to send mail, MAPISP32.EXE “woke up” and did the work of driving the mail.

In Outlook 2002 the design changed, and the spooler code was moved into the Outlook executable itself. This design still exists today and has some interesting implications. Understanding these implications means you really have to embrace the fact that Outlook IS the spooler. Anything that you expected to happen with the spooler, now only happens when Outlook is running.

It’s easy to see the result of this design when you send mail from another application when Outlook is not running. Consider this scenario:

  1. Outlook is installed but not running on your computer.
  2. On the desktop, you right-click a file and choose “Send to Mail Recipient.”
  3. Outlook comes to life and gives you a message to work on.
  4. You type a recipient name and a message, and hit Send.
  5. Later you shut down your computer and go home.
  6. The next morning, the recipient is mad because they never got the mail.

What happens in this case is that immediately AFTER you hit send in step 4, Outlook is done, so it closes. Now Outlook is not running, and therefore no spooler is running, and so nothing is going to send your mail. The next time you launch Outlook, you will see the mail in the Outbox. Assuming Outlook stays open long enough for a spooling cycle to happen (default is 15 minutes), the mail will be sent successfully.

One easy way to conceptualize the spooler in Outlook is to think of it as “everything in the Send Receive Groups” dialog. This dialog roughly maps to the things the Spooler code does and how Outlook manages all the options.


  
As an aside, look closely at the second screenshot. This is from a standard cached mode configuration of Outlook. You will notice that almost all the options are gray and unavailable. This is because the Exchange account is NOT included in this group in a standard cached mode configuration. Note the red X over the account on the left, and the empty checkbox at the top. In cached mode, you do NOT typically want the send/receive group settings to apply. Cached mode has its own logic for synchronizing and sending mail, and it is notification based, not scheduled like the send/receive groups.

Finally, consider the impact of the cached mode feature that came along for Outlook 2003. Cached mode is many things, but one specific part of the cached mode design is that mail delivery uses this spooler process in a more intricate way, and uses it for ALL cached mode sends. Again, it works independent of the settings in the Send/Receive Groups, and it has switches and configuration choices that add complexity.

So there are four key takeaways to start our understanding of Outbox problems:

  1. MAPI Transports are the way that Outlook “sends mail”
  2. There is this notion of a spooler that is the key to managing and driving the sending mail cycle
  3. Outlook.EXE *IS* the spooler ever since Outlook 2002
  4. The cache mode feature drastically changed the importance of the spooler

 

Mail delivery in Exchange

Up until now, I have described the general MAPI ideas around mail delivery and how Outlook implements these options. Now we need to understand some specifics in the Exchange Server configurations. Again, some history is needed to help understand how we got to where we are now.

In early versions of Outlook/Exchange, users almost always used Outlook in the “online” mode of operation. You interacted with Mail and Addresses directly from the Exchange Mailbox. This is in contrast to today’s world where most people use cached mode. In cached mode, you work with data and addresses from a LOCAL data store (your OST file), and then synchronize your data to the mailbox. Note that SENDING mail is not the same as synchronizing. The Outbox folder is never part of synchronization. Mail submission and delivery is different from synchronizing.

There is a fundamental difference in mail submission/delivery between online and cached mode. Understanding this difference is one big key in your ability to grasp the entire Outbox story. For this blog entry, I am going to define two different methods to deliver mail:

  1. Transport Send (this is the spooler method that I described in detail above)
  2. Store Send (this is the second method that ONLY is used in the Exchange Online scenario)

Since we know that in cached mode you work from your OST and not online, you can see that all cached mode mail delivery must happen using the Transport Send method. If you are running in the old online profile (which is common even today on terminal server configurations), then mail delivery is typically done using the Store Send method.

So why do we care which method is used? Mainly because a Store Send does NOT require any of the spooler process to happen. When the Store Send method is used, the message is delivered by the server with no further client interaction. In the “send to mail recipient” scenario that I described in the previous section, if the Outlook profile is configured for online mode, that mail *IS* sent successfully. Since no spooler is needed to deliver the mail, it is delivered immediately by Exchange, whether Outlook is running or not. There are other differences “under the hood” for each method. Appendix C in the attached .PDF complete version of this blog has a table that summarizes the differences.

 

How Outlook chooses which delivery method to use

A good way to conceptually think about the Outlook decision process is to think in terms of the Outbox folder itself. When a message is created in the Outbox folder of the OST/PST, Outlook and MAPI hook up all the plumbing so that mail submission happens through the Transport Send method. When a message is created in the Outbox folder of the online mailbox, Outlook and MAPI hook up the plumbing so that mail submission happens through the Store Send method. So using this concept, if you are in cached mode you should expect a Transport Send, and if you are in online mode, you should expect a Store Send.

Additionally, even if you are in ONLINE mode and would typically be using the Store Send method, there are options and configurations that will prevent the use of the Store Send. The most common example is the presence of an additional transport. So for example, if you have an online configuration and you ADD an SMTP/POP account, you are no longer eligible for delivery via the Store Send method. Likewise if you add a meeting service transport like the LiveMeeting service, you must now send mail using the Transport Send method. Any additional transport will “turn off” the ability to do Store Send with Exchange. Finally there are a couple of Outlook configuration options that will “turn off” the Store Send in online mode. The primary one is the option to save replies with their original mail. So to summarize, you can only take advantage of the Store Send method if:

  1. You have Exchange Server.
  2. You are configured for online mode and not cached mode.
  3. You have no other transports in the profile.
  4. You are not using the “save replies with original” option.

The formal term for the “save replies with original” is this setting in the options dialog:

 

The NEEDS_SPOOLER flag

This option gives a good example of an important “extra” trigger in the mail delivery process. Even if you are using the Store Send method, where the spooler (Outlook) is not required, there are scenarios where the mail will require the spooler. The “save replies with original” feature requires some processing by the spooler. So even in a Store Send scenario (online mode, no extra transports, etc.), Outlook will add a special flag to the message during submission that tells Exchange not to deliver it immediately. This flag is called the NEEDS_SPOOLER flag.

The process looks like this from a non-technical point of view:

  1. Outlook’s logic runs through some tests and determines that this message will need the spooler for some reason.
  2. When Exchange accepts the message for submission, it looks for the “this message NEEDS the spooler” trigger.
  3. If there is no trigger, then Exchange sends the mail itself and you have a Store Send.
  4. If the trigger is there that says “this message NEEDS the spooler,” then Exchange does NOT deliver the message and waits on another command from the spooler before sending.
  5. When the spooler cycle happens, it sends the message up to Exchange to “Send the message” and you have a Transport Send.

So all the criteria are tested for each method and in the end, each message either gets the trigger for “needs the spooler” added or not.

Notice that if Outlook calculates that a Transport Send is needed, there is some amount of time between message submission and message delivery. In a Store Send, there is only the one submission, but in the Transport Send, you have the message submission, and then at some later time, the command to do the Transport Send coming from Outlook itself. There is a choice in the Outlook options screen called “Send Immediately when connected” and the option is on by default. When this option is on, Outlook will kick of a spooler cycle and do a Transport Send immediately after sending the message if a spooler cycle is needed. If the option to “Send Immediately when connected” is turned off, then the message will only be sent according to the defined send/receive settings. Since the send/receive settings, by default, do not include the Exchange Account for cached mode, it can be pretty easy to have a send/receive definition that you do not expect. So it’s important to examine the send/receive settings and make sure the expected definitions exist for how often you want to send mail. Below is the screenshot showing the “Send Immediately” option.


 
As a summary of this section, here is the way Outlook thinks about how a message gets submitted and delivered:

  1. Is this new message I am sending based in an online Outbox folder or an OST/PST folder? If it’s online, route the submission through the Exchange “Store Send code.” If it is OST/PST, route the submission though the “Transport Send” code.
  2. After submitting the message, do a bunch of checks to see if either this is a straight Transport Send, or one of the special case Store Sends that “need the spooler.” If either of these is true, then kick off a spooler cycle to do the Transport Send.
  3. If this was an OST/PST based send, then do the work to move the item from the Outbox folder to the Sent Items folder, and send any NDRs that have occurred.

So we’ve unlocked some of the Outlook Outbox mysteries, but there are more! Open the complete paper attached to this blog. You’ll learn all about:

  • Submission state
  • The Outbox
  • Synchronization
  • The spooler cycle
  • Address types
  • Non-delivery reports
  • Troubleshooting issues, including
    • Transport logging
    • Synchronization issues folder messages
    • Registry settings
    • MFCMAPI

The Mysterious Outlook Outbox_The Whole Story.pdf

Comments

  • Anonymous
    January 01, 2003
    thank you indeed for the information

  • Anonymous
    January 01, 2003
    @Barbara - Yes, true for Outlook 2013.

  • Anonymous
    January 01, 2003
    Hi Christian, It is true that sometimes warning and informational events appear in the sync issues folder and that can create confusion. Microsoft is considering ways to improve this experience, but nothing is officially ready for release at this time. Thanks for the question.

  • Anonymous
    September 04, 2013
    Thanks for posting this. Very useful! A bit off-topic but related: when will we finally be able to disable the "Sync Issues" folder functionallity(or hide the folder)? This causes a lot of headaches for admins in the real world. Even if you tell users to ignore it...it produces loads of support calls. Thanks Christian Schindler

  • Anonymous
    February 19, 2014
    This is really good stuff thank you. I found it looking for an answer to the "Synchronization of some deletions failed." error. I am getting hundreds of these after an exchange migration and my support team cannot resolve (it doesn't stop anything is just annoying having to keep deleting them - would much rather could be stopped). However in the Troubleshooting section it says: "Normal error for this case. Really just a warning that either there was no message to delete server side, or the deletion will be done async and Exchange tells Outlook that it doesn’t know if it worked yet or not. 12:30:49 [0-130] This is just the error code for the above error." Which implies that I just have to live with it? Is there anything I can do (either server or client side) that would clear this out of the system and stop these messages landing in my Sync Issues folder? Hope you can help Tim

  • Anonymous
    February 19, 2014
    While clicking SEND from any folder other than Outbox (and not clicking Outbox to see the message status) is an excellent solution for many unexplained fail-to-send-from Outbox problems, here is a better one: Create a dummy email to a dummy address e.g. "Space_holder@gmail.com." Open Outbox. The email will indicate read. Any emails created afterwords will not revert from SEND to READ. You can open Outbox and click SEND/RECEIVE with impunity. Only the dummy placeholder email will be stuck..

  • Anonymous
    March 03, 2014
    @Tim, The "some deletions failed" issue is on our list of things to make better. So while I cannot announce anything specific for you, maybe check back here once in a while, and I will post something when i can give an official plan.

  • Anonymous
    March 05, 2014
    Hey all, thanks for this article, very useful :)

    Does this explain why when using the Word (or other Office app) "send as email attachment' feature the emails don't send until outlook is started fully? My experience is that emails just end up in the outbox and don't get sent until the full client is ran.

    Cheers,

    Gavin

  • Anonymous
    May 07, 2014
    Is this still valid and true for Outlook 2013? We are evaluating cached vs online mode and this would be a deciding factor.

  • Anonymous
    May 08, 2014
    @Gavin - Yes, if outlook isn't already running when you send from Word, then it is launched, mail is queued up, and outlook exits. There is no time for a spooler cycle to happen, so the mail is not sent until the next time outlook is "really running"

  • Anonymous
    June 16, 2014
    Thank you for a good article:-)

    Just solved a long and diffcult issue with e-mails stuck in outboox - turned out to be a bad dlink router, who worked perfect in all other scenarios. Behavior here was that both send/recieve worked after starting Outlook, but sending e-mails would be stuck in outbox after about 5-7 minutes of inactivity.

    This case started with the usual suspects - AddOns in Outlook, OffCat scan to get rid of any sync issues, actually reinstalling both OS and Office, verifiying that issue appeared only on this computer and not in OWA / other Networks.

    Lesson learned - test the computer in a different network environment to eliminate firewall/router/isp issues before sinking alot of hours into Outlook/Office365 troubleshooting:-)

  • Anonymous
    June 19, 2014
    @Vidar - Great find. I have been thinking/working on a way to help surface those type of issues. Outlook does a lot of work to recognize and handle network issues, but sometimes the timeout or problem is just too far down the stack and outlook/mapi never gets the message. To outlook, the Rops to the server just stop coming back. The move to Mapi/Http (removing the RPC layer) in Outlook 2013 SP1 should help some in this area.

  • Anonymous
    August 03, 2014
    Accidentally hit send and Inmediatelly powered off my laptop. Chequed with other PC and seems unwanted mail was not sent. I use Outlook for several Gmail accounts. How can I delete it? Where is it located? Do not see an outbox folder in this account.
    Thanks for your help.

  • Anonymous
    October 11, 2014
    The information you have posted is very useful. The sites you have referred was good. Thanks for sharing...

    SEO

  • Anonymous
    October 19, 2014
    I have read your blog it is very helpful for me. I want to say thanks to you. I have bookmark your site for future updates.
    Auto Forum

  • Anonymous
    October 24, 2014
    The mysterious Outlook Outbox - Office IT Pro Blog - Site Home - TechNet Blogs

  • Anonymous
    December 10, 2014
    Hi! I have 3 mail accounts set up in my Outlook. 2 are personal (1 POP, 1 IMAP) and one is with my employer (Microsoft Exchange). I noticed that each and every message I send from my personal mailboxes go through the Microsoft Exchange Outbox of my emplyer's supplied account..... Does it mean that all my personal messages are sent though my employer's server?????

  • Anonymous
    January 19, 2015
    Hi, thanks a lot for the info! But there`s a tricky one: my outlook sends the letters but they dont move to sent folder, instead they get stuck in the outbox, any ideas how to fix that?? Any comments will be highly appreciated!

  • Anonymous
    January 19, 2015
    I am not going to read this article, I just wanted to find out how to see the outbox not some long winded explanation of how they changed outlooks implementation. i don't care if they are spooling, polling etc... I don't care, i want to see what was attempted to be sent but did not go out, if it is an "outbox" i want to see it!! Call it what ever you want I WANT TO SEE IT!

    I will never use office 365! no cloud based corporately managed storage for my data and there is no way I will pay a fee for SAAS.

  • Anonymous
    April 04, 2015
    Great stuff, many thanks!

    When one considers thousands and thousands of inquiries for solution re. “Mail Stuck in Outbox” in many forums, then everybody should first read this perfect article. My personal experience deals with (by mistake) deleting the "Send Immediately When Connected" flag after years of use in Outlook 2010 and then searching forums for a solution. You can find there everything:- from advises to reinstall Outlook, reinstall Windows (!?- coming from MS support “gurus”), going back from IMAP to POP type servers up to the abandoning Windows and going to Apple …

    We all tend to use more and more applications, without understanding how they work. Then by simple change of some settings things are suddenly not working as before. So, this article is a must for every User of MS Outlook- no doubts!
    ________________________________________________________

  • Anonymous
    June 03, 2015
    Thank you for sharing this post here!!!!

  • Anonymous
    June 12, 2015
    BUT I would think with the thousands of "problems with Outlook not sending my emails" reported, that there would be a patch for small-time users who want to use it "like Outlook Express" , i.e. right-click file - send - fill in the address and off it goes. OR

    1. perhaps REMOVE the option to ' "right-click", SEND' unless outlook is open.
    OR
    2. DO open Outlook TO SEND IT, then close.

    Make those choices easily accessible to user.

    It's been 10+ YEARS that I've been seeing this frustration from users with Outlook, so many went to Outlook Express, but with IT gone, we're back to a 10 year old problem.

    From their standpoint, the FREE program THAT WORKED has been replaced by an expensive program THAT DOESN'T WORK.

    Re:"In Outlook 2002 the design changed, and the spooler code was moved into the Outlook executable itself. Understanding these implications means you really have to embrace the fact that Outlook IS the spooler. Anything that you expected to happen with the spooler, now only happens when Outlook is running."

  • Anonymous
    July 08, 2015
    This is a collection of the top Microsoft Support solutions for the most common issues IT Pros may face

  • Anonymous
    September 08, 2015
    Great article, thanks for creating it.