Compartilhar via

Outlook 2007 RTM Docs - Notification Based Indexing Support

  • Artigo

[This is now documented here: https://msdn.microsoft.com/en-us/library/bb821096.aspx, https://msdn.microsoft.com/en-us/library/bb821097.aspx, https://msdn.microsoft.com/en-us/library/bb821099.aspx, https://msdn.microsoft.com/en-us/library/bb821100.aspx]

(Now that Outlook 2007 is available I'm reposting some of the articles from the Outlook 2007 Beta Documentation series. Some of the articles are virtually unchanged, others are completely different to reflect the differences between Beta and RTM. This article is the updated version of the first half of Outlook 2007 Beta Documentation - Notification Based Indexing Support.)

New flag that identifies stores as using Pusher model when indexing

This flag is set by pusher stores.

Definition

 #define STORE_PUSHER_OK ((ULONG) 0x00800000)

If the store provider sets this flag the MAPI PH won’t crawl the store and the store is responsible to push any changes through notifications to the indexer to have messages indexed.

Usage

This flag can be retrieved by getting the property PR_STORE_SUPPORT_MASK from the store.

MAPI URLs

Every time a folder, message or an attachment is to be indexed a unique Unicode URL has to be generated and sent to the indexer. This URL will later be used to identify which object to be indexed in the MAPI PH that runs in the Search Protocol Host. Store providers are responsible for generating these URLs.

MAPI URLs have the following format:

Mapi://SID/StoreDisplayName ($HashNumber)/StoreType/FolderNameA/…/FolderNameN/[EntryIDEncoded[/at=AttachIDEncoded:FileName]]

Parameters

SID

Current user’s SID

HashNumber

DWORD in hex calculated based on the store entry id or the store mapping signature. This value will be stored in the registry and will be used later to identify the store in the MAPI PH.

This number has to be calculated in a way that minimizes collisions between other stores. See below for the algorithm Outlook uses to calculate the hash.

StoreType

Number that identifies the type of the store that contains the object to be indexed. Here are the possible values:

0 – Default store

1 – Delegate store (used for delegate items cached locally)

2 – Public Folders (used for public folder favorites)

Note that if the store is being crawled instead of pushed the value used is none of the above but instead it will be the character X.

FolderNameA/…/FolderNameN

The path from the root of the IPM_SUBTREE to the folder or message. For instance, a message in the “Family” folder under “Inbox” will have Inbox/Family for this parameter.

EntryIDEncoded

MAPI Entry ID for the item encoded as a Unicode string. See below for how it gets encoded. Note that when viewed as text, this encoded entry ID will appear as random Hangul characters or boxes depending on available fonts.

AttachIDEncoded

Attachment ID encoded as a Unicode string.

FileName

Attachment file name as it appears in the message.

Here are some examples of MAPI URLs for a folder, message and attachment respectively:

mapi://S-1-5-21-2127521184-1604012920-1887927527-71418/Mailbox – Some User ($be19928f)/2/Office

mapi://S-1-5-21-2127521184-1604012920-1887927527-71418/Mailbox – Some User ($484efb89)/0/Calendar/곯가가가걍걝걌곌겷걢곒갑겛개가검걟곔걙곾걤곂갠가

mapi://S-1-5-21-2127521184-1604012920-1887927527-71418/Mailbox – Some User ($484efb89)/0/Inbox/곯가가가걍걝걌곌겷걢곒갑겛개가검걟곔걙곾간곷갦가/at=겅걋각가:somefile.txt

Characters that get encoded if they are in the store or folder display name

Here is the table that shows which characters we encode:

     %    ->    %25, 
    /    ->    %2F 
    \    ->    %5C 
    *    ->    %2A 
    ?    ->    %3F

Algorithm to calculate the store hash

Note that if a store has PR_MAPPING_SIGNATURE in the global profile section then that’s what we will hash instead of the store entry id. For all other stores we will hash the Entry ID.

 DWORD ComputeStoreHash(ULONG cbStoreEID, LPENTRYID pbStoreEID, LPCWSTR pwzFileName)
{
    DWORD  dwHash = 0;
    ULONG  cdw    = 0;
    DWORD* pdw    = NULL;
    ULONG  cb     = 0;
    BYTE*  pb     = NULL;
    ULONG  i      = 0;
 
    // Get the Store Entry ID
    // pbStoreEID is a pointer to the Entry ID
    // cbStoreEID is the size in bytes of the Entry ID
    pdw = (DWORD*)pbStoreEID;
    cdw = cbStoreEID / sizeof(DWORD);
 
    for (i = 0; i < cdw; i++)
    {
        dwHash = (dwHash << 5) + dwHash + *pdw++;
    }
 
    pb = (BYTE *)pdw;
    cb = cbStoreEID % sizeof(DWORD);
 
    for (i = 0; i < cb; i++)
    {
        dwHash = (dwHash << 5) + dwHash + *pb++;
    }
 
    // You may want to also include the store file name in the hash calculation
    // Get store FileName
    // pwzFileName is a NULL terminated string with the path and filename of the store
    if (pwzFileName)
    {
        while (*pwzFileName)
        {
            dwHash = (dwHash << 5) + dwHash + *pwzFileName++;
        }
    }

    // dwHash now contains the hash to be used. It should be written in hex when building the URL.
    return dwHash;
}// ComputeStoreHash

Algorithm to encode the Entry ID and the attachment ID

The goal of this algorithm is to generate a compact representation of the Entry ID or attachment ID.

 const WORD kwBaseOffset = 0xAC00;  // Hangul char range (AC00-D7AF)
LPWSTR EncodeID(ULONG cbEID, LPENTRYID rgbID)
{
    ULONG   i = 0;
    LPWSTR  pwzDst = NULL;
    LPBYTE  pbSrc = NULL;
    LPWSTR  pwzIDEncoded = NULL;

    // rgbID is the item Entry ID or the attachment ID
    // cbID is the size in bytes of rgbID

    // Allocate memory for pwzIDEncoded
    pwzIDEncoded = new WCHAR[cbEID];
    if (!pwzIDEncoded) return NULL;

    for (i = 0, pbSrc = (LPBYTE)rgbID, pwzDst = pwzIDEncoded;
        i < cbEID;
        i++, pbSrc++, pwzDst++)
    {
        *pwzDst = (WCHAR) (*pbSrc + kwBaseOffset);
    }

    // Ensure NULL terminated
    *pwzDst = L'\0';

    // pwzIDEncoded now contains the entry ID encoded.
    return pwzIDEncoded;
}// EncodeID

New notification to notify the MAPI PH of the process pushing MAPI URLs to be indexed

A new MAPI notification type has been introduced to facilitate shutdown scenarios for pusher stores. These stores will have to persist what has to be pushed since it may not be able to index everything before a shutdown occurs. When a store provider is a pusher it should send the following notification so that the MAPI PH knows which process it should watch for a given store. This way if the process is shut down or crashes the MAPI PH will immediately detect that and close (stop indexing) the store it opened.

 #define fnevIndexing        ((ULONG) 0x00010000) 

/* Indexing notifications (used for FTE related communications) */ 
/* Shares EXTENDED_NOTIFICATION to pass structures below,       */ 
/* but NOTIFICATION type will be fnevIndexing                   */ 

// Stores that are pusher enabled (PR_SUPPORT_MASK contains STORE_PUSHER_OK) 
// are required to send notifications regarding the process that is pushing. 
#define INDEXING_SEARCH_OWNER        ((ULONG) 0x00000001) 

typedef struct _INDEX_SEARCH_PUSHER_PROCESS 
{ 
    DWORD dwPID; /* PID for process pushing */ 
} INDEX_SEARCH_PUSHER_PROCESS;

BLOB structure associated with each URL when indexing

When pushing URLs to be indexed pusher stores should also create a blob with some information that will be used by the MAPI PH. This blob will be associated with each URL and will be sent when the URL is pushed to the indexer.

The format of the blob is the following:

 DWORD  dwVersion
DWORD  dwFlags
ULONG  cbProfileName
WCHAR  wszProfileName   
ULONG  cbProviderItemID
WCHAR  wszProviderItemID

Note that these values have to be written to the blob in the order shown above. The provider item ID should only be sent for folders to prevent opening extra folders to get this information.

Parameters

dwVersion

This is the version of the data being sent. Currently this value is 1.

dwFlags

Reserved for future use. Currently this should be 0.

cbProfileName

Size of the profile name in bytes. This information will be useful for the MAPI PH to know which profile to use when indexing the item.

wszProfileName

Null terminated Unicode string with the profile name.

cbProviderItemID

Size of the Provider Item ID in bytes

wszProviderItemID

Null terminated Unicode string with the provider item ID that uniquely identifies the item in the store.

Comments

  • Anonymous
    March 26, 2007
    Anyone have a vbScript code sample to derive the EntryID and StoreID of an Outlook item from the ItemURL?   I am looking to use ADO from Script to search the WDS index for Outlook items and then display the results in a grid for user review/selection and then display the Outlook Item from the user selection... So I will need the EntryID and StoreID to get the Outlook Item.

  • Anonymous
    October 15, 2007
    For those looking for C# code for ComputeStoreHash and EncodeID -- take a look here: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=810581&SiteID=1

  • Roy
  • Anonymous
    January 15, 2008
    I am still not clear how does this notification-based indexing works. Can anyone please clarify some of my confusions :)
  1. If a store registers itself in the registry to let MAPI PH know that it needs to be indexed, and also expose the STORE_PUSHER_OK bit in the STORE_SUPPORT_MASK, does it mean that the MAPI PH will register itself through the normal store notification mechanism, such as through IMsgStore::Advise(). So, the pusher store is responsible to send out indexing-notification to the MAPI PH?
  2. On the other hand, if a store is registered to be indexed, but does not expose the STORE_PUSHER_OK bit, then MAPI PH will just crawl the store on its own? There is nothing special or extra that the non-pusher store needs to be to get its contents indexed by the MAPI PH?
  3. In either scenarios, how does this affect the Instant Search behavior? Does the store need to do something extra to support the Instant Search feature?
  4. Finally, if a store does not register itself to be indexed at all, then does it mean that the content in the store will never be indexed? Again, how does that affect the Instant Search feature? Any information or thoughts are appreciated! Hai

  • Anonymous
    January 23, 2008
    "Every time a folder, message or an attachment is to be indexed a unique Unicode URL has to be generated and sent to the indexer." Exactly how does the store provider "send" the URL to the indexer? Through MAPI notification? Or through some interface exposed by the Indexer? Where can I find more information about this topic? Please help! Hai

  • Anonymous
    January 25, 2008
    Hai - the PH should be indexing your store if it's in the profile and you're not setting STORE_PUSHER_OK. If it's not, you might want to open a support case so we can investigate.

  • Anonymous
    January 28, 2008
    Steve, thank you very much for the response :-) Currently, the store provider does NOT expose the STORE_PUSHER_OK bit, and also registers itself in the registry to be crawled by the PH at "[HKCU]SoftwareMicrosoftWindowsWindows SearchPreferences". I can see that the PH does actually index the store, because the "Word-Wheeling" does return some results at some time. However, the problem is that the search result is not reliable. Sometimes no result is returned at all, and sometimes partial result is returned. I never know if a search does find all the results. To make it worse, Outlook never shows the warning message "your search result may be incomplete...". By not registering the store for indexing, I lose the wonderful "Word-Wheeling" feature, but the search result looks much more stable and reliable. Steve, should I unregister the store for indexing? Does this mean that "Word-Wheeling" feature is not supported by a non-pusher store? Again, any information is appreciated! Hai

  • Anonymous
    January 28, 2008
    When your store isn't indexed, Outlook falls back on the old "Search Folder"/restriction style searches. That's why the results seem more reliable and why you don't get "Word-Wheeling". Really - you should open a case to investigate why the PH is having trouble indexing your store.

  • Anonymous
    March 12, 2008
    When using "Advanced Find" in Outlook 2007, does it also use the index built by the SearchProtocolHost? When the store is registered for indexing, and thus Instant Search is enabled for the store, it looks that the Advanced Find will also take advantage of the index and get results much faster. On the other hand, when the store is NOT registered for indexing, the Instant Search is not available, and also the Advanced Find will fall back to the old way of slow search.

  • Anonymous
    September 28, 2009
    The comment has been removed