How to Manage Exchange 2010 Message Queues
Queues
· Get-Queue
· Resume-Queue
· Retry-Queue
· Suspend-Queue
Messages
· Get-Message
· Remove-Message
· Resume-Message
· Suspend-Message
· Export-Messages
GUI is BAD! Cmd line good!
So the First thing we get a SCOM alert mentioning a Queue is over its threshold on ServerX.
1. In Exchange Shell do Get-Queue –Server <servername> and see what other queues may be affected.
Get-Queue –Server <ServerName>
[PS] C:\scripts>get-Queue -Server Lab-E2k10CSHT01
Identity DeliveryType Status MessageCount NextHopDomain
-------- ------------ ------ ------------ -------------
LAB-E2k10CSHT01\4 SmtpRelay... Retry 2196 adsite2
LAB-E2k10CSHT01\Submission Undefined Ready 0 Submission
2. Notice the NextHopDomain, this will show the next location the message is to be routed. The adsite2 is for a mailbox in the remote ADSite2. This can be another Queue or a SMTP Domain.
3. Let’s look at this closer, add an “|FL” to the command and see what information is added. The Key ones are the LastError, Status, LastRetryTime, NextretryTime. The lastError in this example not only gives the SMTP code "421 4.2.1 Unable to connect." But also elaborates to the cause
[PS] C:\scripts>get-Queue -Server Lab-E2k10CSHT01 |fl
RunspaceId : 59f16a72-c038-4c79-9c48-d1b14d5411bf
DeliveryType : SmtpRelayToRemoteAdSite
NextHopDomain : adsite2
TlsDomain :
NextHopConnector : 2c2bba82-cdb3-460a-82b3-6816d7e7fb70
Status : Retry
MessageCount : 2196
LastError : 451 4.4.0 Primary target IP address responded with: "421 4.2.1 Unable to connect." Attempted faillover to alternate host, but that did not succeed. Either there are no alternate hosts, or delivery failed to all alternate hosts.
LastRetryTime : 4/22/2011 8:57:34 AM
NextRetryTime : 4/22/2011 9:07:34 AM
DeferredMessageCount : 0
QueueIdentity : LAB-E2k10CSHT01\4
Identity : LAB-E2k10CSHT01\4
IsValid : True
RunspaceId : 59f16a72-c038-4c79-9c48-d1b14d5411bf
DeliveryType : Undefined
NextHopDomain : Submission
TlsDomain :
NextHopConnector : 00000000-0000-0000-0000-000000000000
Status : Ready
MessageCount : 0
LastError :
LastRetryTime :
NextRetryTime :
DeferredMessageCount : 0
QueueIdentity : LAB-E2k10CSHT01\Submission
Identity : LAB-E2k10CSHT01\Submission
4. Note: You could alternately use get-Queue –Identity “<Queue>” |fl
[PS] C:\scripts>get-Queue -Identity "LAB-E2k10CSHT01\4" |fl
RunspaceId : 59f16a72-c038-4c79-9c48-d1b14d5411bf
DeliveryType : SmtpRelayToRemoteAdSite
NextHopDomain : adsite2
TlsDomain :
NextHopConnector : 2c2bba82-cdb3-460a-82b3-6816d7e7fb70
Status : Retry
MessageCount : 2196
LastError : 451 4.4.0 Primary target IP address responded with: "421 4.2.1 Unable to connect." Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts, or delivery failed to all alternate hosts.
LastRetryTime : 4/22/2011 8:57:34 AM
NextRetryTime : 4/22/2011 9:07:34 AM
DeferredMessageCount : 0
QueueIdentity : LAB-E2k10CSHT01\4
Identity : LAB-E2k10CSHT01\4
IsValid : True
5. Ok, what do I do? Well, that depends on the error. In this case, I just need to start the Hub Transport in my Remote Site. That LastError message is the key to troubleshooting the reason for the Queue backup.
a. Remote Queues will all be SMTP Based. SMTP based Queues use the SMTP Error Codes. *See the SMTP Troubleshooting See https://support.microsoft.com/kb/256321/
i. Note: These will vary in appearance, name and number.
b. Mailbox or Local Submission queues check Database for Errors/warning
c. “Submission” will always be AV or an Agent/Rule triggered on Submission Event.
d. Poison Message Queue – These are messages that have caused the Transport Service to crash.
e. Shadow Queues will always be that something is preventing the Q-Discard from being received / sent.
f. Unreachable Queue – anytime there is not a route to the destination. Check Config to these recipients.
6. Now, here is some helpful “-Filter” to help with looking at Queues. Note Get-Queue only gets the queue on the local server. You will have to use Get-ExchangeServer | Where { $_.ServerRole –like “*transport*”} or Get-ExchangeServer | where {$_.isHubTransportServer -eq $true } or something similar to look at all Server Queues.
a. To get all the queues with MessageCount Greater Than 100
Get-queue –Filter {MessageCount –gt 200}
b. To get all the queues with LastError not equal to $Null
Get-Queue –Filter {LastError –ne $Null}
c. And the list can go on… Common Comparison Operators
Operator |
Name |
Description |
-eq |
Equality |
Tests whether values are the same. Other languages may use = or == to test for equality. |
-ne |
Not Equal |
Tests for inequality. Other languages may use <> or != to test for inequality. |
-gt |
Greater Than |
Tests whether one value is larger than another. Other languages may use the > character. |
-lt |
Less Than |
Tests whether one value is smaller than another. Other languages may use the < character. |
-ge |
Greater Than or Equal To |
Tests whether a value is larger than or equal to another value. Similar to >= in VBScript and other languages. |
-le |
Less Than or Equal To |
Tests whether a value is smaller than or equal to another value. Similar to <= in VBScript and other languages. |
7. Ok, I found the Queue and I want to Export the messages…how? Well, I am glad you asked. Use the Export-message command. Note it is best to Suspend the Queue or Message before exporting.
a. Now, there are several options.
i. Export a Specific Message
Export-Message -Identity LAB-E2k10CSHT01\4 -Path "C:\Tailspintoys\export.eml"
ii. Export ALL message from a Specific Queue, this gets a little more involved. Requires a ForEach loop and some logic for the name of the files exported is highlighted.
Get-Message -Queue "LAB-E2k10CSHT01\4" | ForEach-Object
{
$Temp="C:\tailspintoys Export\"+$_.InternetMessageID+".eml";
$Temp=$Temp.Replace("<","_");
$Temp=$Temp.Replace(">","_");
Export-Message $_.Identity | AssembleMessage -Path $Temp
}
iii. Export ALL Message from a Server
Get-Message -Server "LAB-E2k10CSHT01" | ForEach-Object
{
$Temp="C:\tailspintoys Export\"+$_.InternetMessageID+".eml";
$Temp=$Temp.Replace("<","_");
$Temp=$Temp.Replace(">","_");
Export-Message $_.Identity | AssembleMessage -Path $Temp
}
iv. Now, you can add filter to the above say you only wanted to get-messages from senders in the Tailspintoys domain. You could add -Filter {FromAddress -like "@Tailspintoys.com"} to the above after Get-Message.
v. To Replay these messages on any Exchange 2010 Hub Server, drop into the “Replay” Directory.
8. Ok, how do I delete them? With and Without an NDR….well the Remove message does both! Only difference in the command –WithNDR $True or $False
a. Remove-Message -Filter {property -operator "value"} -WithNDR <$true | $false>
Remove-Message -Filter {Subject -eq "Win Big"} -WithNDR $false will Turf the message withOUT NDRs being sent.
Remove-Message -Filter {Subject -eq "Win Big"} -WithNDR $True will Turf the message with an NDR. Since the default is True. You could remove the –WithNDR $True
b. The –Filter is the same that you used with the Get-Message.
Appendix I
· DeliveryType The delivery type for this queue as defined by transport. The delivery type must be one of the following values:
o DNSConnectorDelivery
o NonSMTPGatewayDelivery
o SmartHostConnectorDelivery
o SmtpRelayWithinAdSitetoEdge
o MapiDelivery
o SmtpRelayWithinAdSite
o SmtpRelaytoRemoteAdSite
o SmtpRelaytoTiRg
o Undefined
o Unreachable
· Identity The queue identity in the form of Server\destination, where destination is a remote domain, Mailbox server, or persistent queue name.
· LastError A text string of the last error recorded for a queue.
· LastRetryTime The time when a connection was last tried for this queue.
· MessageCount The number of items in the queue.
· NextHopConnector The GUID of the connector that was used to create the queue.
· NextHopDomain The next hop domain of the queue, specified as a remote SMTP domain, a server name, the name of an Active Directory site, or a message database (MDB) identifier.
· NextRetryTime The time when a connection will next be tried for this queue.
· Status The status of the queue. Queue status options are Active, Ready, Retry, or Suspended.
Queue identity formats
Queue identity format |
Usage |
Server\QueueJetID (Int64) |
The complete, unique identity for a delivery queue. |
\QueueJetID |
The identity of a queue on the local server. The server name is omitted. Therefore, the local server is implied. |
Server\* |
Any queue on the specified server. |
Server\NextHopDomain |
A queue on the specified server holding messages for delivery to a specific remote domain. |
\NextHopDomain |
A queue holding messages destined for a specific domain and located on the local server. |
Server\Poison |
The poison message queue located on the specified server. |
Server\Submission |
The queue that contains items waiting to be processed by the categorizer. |
Server\Unreachable |
The queue that contains items that can't be routed and located on the specified server. |
Message Filters
The Filter parameter requires an expression that specifies the property value criteria for the messages that you want to remove. The expression includes a property name followed by a comparison operator and value. The following message properties are valid criteria for the Filter parameter:
· DateReceived The date that the message was received.
· ExpirationTime The time that a message will expire.
· FromAddress The SMTP address of the sender of a message. This value is taken from MAIL FROM in the message envelope.
· Identity An integer that represents a particular message and an optional server and queue identity.
· InternetMessageId The value of the Message-ID header field. This property is expressed as a GUID followed by the SMTP address of the sending server, as in this example: 67D7543D6103DC4FBEBA6BC7205DACABA61231@exchange.contoso.com.
· LastError A text string of the last error recorded for a message.
· MessageSourceName A text string of the name of the component that submitted this message to the queue.
· Queue The identity of the queue that holds the message. Enter the queue identity in the form of Server\destination, where destination is a remote domain, Mailbox server, or persistent queue name.
· RetryCount The number of times that delivery of a message to a destination was tried.
· SCL The spam confidence level (SCL) of the message. Valid SCL entries are integers 0 through 9. An empty SCL property value indicates that the message hasn't been processed by the Content Filter agent.
· Size The size of a message.
· SourceIP The IP address of the external server that submitted the message to the Exchange organization.
· Status The current message status. Message status options are Active, Retry, Suspended, PendingSuspend, and PendingRemove.
· Subject A text string that represents the e-mail subject. The value is taken from the Subject header field.
You can specify multiple filter criteria by using the and comparison operator. Property values that aren't expressed as a single integer must be enclosed in quotation marks (").
Comparison Operators.
TOPIC
about_Comparison_Operators
SHORT DESCRIPTION
Describes the operators that compare values in Windows PowerShell.
LONG DESCRIPTION
Comparison operators let you specify conditions for comparing values and
finding values that match specified patterns. To use a comparison operator,
specify the values that you want to compare together with an operator that
separates these values.
By default, all comparison operators are case-insensitive. To make a
comparison operator case-sensitive, precede the operator name with a "c".
For example, the case-sensitive version of "-eq" is "-ceq". To make the
case-insensitivity explicit, precede the operator with an "i". For example,
the explicitly case-insensitive version of "-eq" is "ieq".
All comparison operators except the containment operators
(-contains, -notcontains) and type operators (-is, -isnot) return a Boolean
value when the input to the operator (the value on the left side of the
operator) is a single value (a scalar). When the input is a collection of
values, the containment operators and the type operators return any
matching values. If there are no matches in a collection, these operators
do not return anything. The containment operators and type operators always
return a Boolean value.
Windows PowerShell supports the following comparison operators.
-eq
Description: Equal to. Includes an identical value.
Example:
C:\PS> "abc", "def" -eq "abc"
abc
-ne
Description: Not equal to. Includes a different value.
Example:
C:\PS> "abc", "def" -ne "abc"
def
-gt
Description: Greater-than.
Example:
C:\PS> 8 -gt 6
True
-ge
Description: Greater-than or equal to.
Example:
C:\PS> 8 -ge 8
True
-lt
Description: Less-than.
Example:
C:\PS> 8 -lt 6
False
-le
Description: Less-than or equal to.
Example:
C:\PS> 6 -le 8
True
-like
Description: Match using the wildcard character (*).
Example:
C:\PS> "Windows PowerShell" -like "*shell"
True
-notlike
Description: Does not match using the wildcard character (*).
Example:
C:\PS> "Windows PowerShell" -notlike "*shell"
False
-match
Description: Matches a string using regular expressions.
When the input is scalar, it populates the
$Matches automatic variable.
Example:
C:\PS> "Sunday" -match "sun"
True
C:\PS> $matches
Name Value
---- -----
0 sun
-notmatch
Description: Does not match a string. Uses regular expressions.
When the input is scalar, it populates the $Matches
automatic variable.
Example:
C:\PS> "Sunday" -notmatch "sun"
False
C:\PS> $matches
Name Value
---- -----
0 sun
-contains
Description: Containment operator. Tells whether a single test value appears
in a set of reference values. Returns TRUE only when the test value exactly
matches at least one of the reference values. Contains uses reference equality
and returns a Boolean value.
Syntax:
<Reference-values> -contains <Test-value>
Examples:
C:\PS> "abc", "def" -contains "def"
True
C:\PS> "Windows", "PowerShell" -contains "Shell"
False #Not an exact match
# Does the list of computers in $domainServers
# include $thisComputer?
# -------------------------------------------
C:\PS> $domainServers -contains $thisComputer
True
-notcontains
Description: Containment operator. Tells whether a single (scalar) test
value appears in a set of reference values. Returns TRUE when the test
value is not an exact match for any of the reference values. Always
returns a Boolean value.
Syntax:
<Reference-values> -notcontains <Test-value>
Examples:
C:\PS> "Windows", "PowerShell" -notcontains "Shell"
True #Not an exact match
# Get cmdlet parameters, but exclude common parameters
function get-parms ($cmdlet)
{
$common = "Verbose", "Debug", "WarningAction", "WarningVariable", `
"ErrorAction", "ErrorVariable", "OutVariable", "OutBuffer"
$allparms = (get-command $cmdlet).parametersets | foreach {$_.parameters} | `
foreach {$_.name} | sort-object | get-unique
$allparms | where {$common -notcontains $_ }
}
# Find unapproved verbs in the functions in my module
# -------------------------------------------
C:\PS> $approvedVerbs = get-verb | foreach {$_.verb}
C:\PS> $myVerbs = get-command -module MyModule | foreach {$_.verb}
C:\PS> $myVerbs | where {$approvedVerbs -notcontains $_}
ForEach
Sort
Tee
Where
-replace
Description: Replace operator. Changes the specified elements of a value.
Example:
C:\PS> "Get-Process" -replace "Get", "Stop"
Stop-Process
# Change all .GIF file name extension to .JPG
C:\PS> dir *.gif | foreach {$_ -replace ".gif", ".jpg"}
Equality Operators
The equality operators (-eq, -ne) return a value of TRUE or the matches
when one or more of the input values is identical to the specified
pattern. The entire pattern must match an entire value.
The following examples show the effect of the equal to operator:
C:PS> 1,2,3 -eq 2
2
C:PS> "PowerShell" -eq "Shell"
False
C:PS> "Windows", "PowerShell" -eq "Shell"
C:PS>
C:\PS> "abc", "def", "123" -eq "def"
def
Containment Operators
The containment operators (-contains and -notcontains) are similar to the
equality operators. However, the containment operators always return a
Boolean value, even when the input is a collection.
Also, unlike the equality operators, the containment operators return a
value as soon as they detect the first match. The equality operators
evaluate all input and then return all the matches in the collection.
The following examples show the effect of the -contains operator:
C:PS> 1,2,3 -contains 2
True
C:PS> "PowerShell" -contains "Shell"
False
C:PS> "Windows", "PowerShell" -contains "Shell"
False
C:\PS> "abc", "def", "123" -contains "def"
True
C:\PS> "true", "blue", "six" -contains "true"
True
The following example shows how the containment operators differ from the
equal to operator. The containment operators return a value of TRUE on the
first match.
C:\PS> 1,2,3,4,5,4,3,2,1 -eq 2
2
2
C:\PS> 1,2,3,4,5,4,3,2,1 -contains 2
True
In a very large collection, the -contains operator returns results
quicker than the equal to operator.
Match Operators
The match operators (-match and -notmatch) find elements that match or
do not match a specified pattern using regular expressions.
The syntax is:
<string[]> -match <regular-expression>
<string[]> -notmatch <regular-expression>
The following examples show some uses of the -match operator:
C:\PS> "Windows", "PowerShell" -match ".shell"
PowerShell
C:\PS> (get-command get-member -syntax) -match "-view"
True
C:\PS> (get-command get-member -syntax) -notmatch "-path"
True
C:\PS> (get-content servers.txt) -match "^Server\d\d"
Server01
Server02
The match operators search only in strings. They cannot search in arrays
of integers or other objects.
The -match and -notmatch operators populate the $Matches automatic
variable when the input (the left-side argument) to the operator
is a single scalar object. When the input is scalar, the -match and
-notmatch operators return a Boolean value and set the value of the
$Matches automatic variable to the matched components of the argument.
If the input is a collection, the -match and -notmatch operators return
the matching members of that collection, but the operator does not
populate the $Matches variable.
For example, the following command submits a collection of strings to
the -match operator. The -match operator returns the items in the collection
that match. It does not populate the $Matches automatic variable.
C:\PS> "Sunday", "Monday", "Tuesday" -match "sun"
Sunday
C:\PS> $matches
C:\PS>
In contrast, the following command submits a single string to the
-match operator. The -match operator returns a Boolean value and
populates the $Matches automatic variable.
C:\PS> "Sunday" -match "sun"
True
C:\PS> $matches
Name Value
---- -----
0 Sun
The -notmatch operator populates the $Matches automatic variable when
the input is scalar and the result is False, that it, when it detects
a match.
C:\PS> "Sunday" -notmatch "rain"
True
C:\PS> $matches
C:\PS>
C:\PS> "Sunday" -notmatch "day"
False
C:\PS> $matches
C:\PS>
Name Value
---- -----
0 day
Replace Operator
The -replace operator replaces all or part of a value with the specified
value using regular expressions. You can use the -replace operator for
many administrative tasks, such as renaming files. For example, the
following command changes the file name extensions of all .gif files
to .jpg:
Get-ChildItem | Rename-Item -NewName { $_ -replace '.gif$','.jpg$' }
The syntax of the -replace operator is as follows, where the <original>
placeholder represents the characters to be replaced, and the
<substitute> placeholder represents the characters that will replace
them:
<input> <operator> <original>, <substitute>
By default, the -replace operator is case-insensitive. To make it case
sensitive, use -creplace. To make it explicitly case-insensitive, use
-ireplace. Consider the following examples:
C:\PS> "book" -replace "B", "C"
Cook
C:\PS> "book" -ireplace "B", "C"
Cook
C:\PS> "book" -creplace "B", "C"
book
Bitwise Operators
Windows PowerShell supports the standard bitwise operators, including
bitwise-AND (-bAnd), the inclusive and exclusive bitwise-OR operators
(-bOr and -bXor), and bitwise-NOT (-bNot). Beginning in Windows
PowerShell 2.0, all bitwise operators work with 64-bit integers.
Windows PowerShell supports the following bitwise operators.
Operator Description Example
-------- ---------------------- -------------------
-bAnd Bitwise AND C:\PS> 10 -band 3
2
-bOr Bitwise OR (inclusive) C:\PS> 10 -bor 3
11
-bXor Bitwise OR (exclusive) C:\PS> 10 -bxor 3
9
-bNot Bitwise NOT C:\PS> -bNot 10
-11
Bitwise operators act on the binary format of a value. For example, the
bit structure for the number 10 is 00001010 (based on 1 byte), and the
bit structure for the number 3 is 00000011. When you use a bitwise
operator to compare 10 to 3, the individual bits in each byte are
compared.
In a bitwise AND operation, the resulting bit is set to 1 only when both
input bits are 1.
1010 (10)
0011 ( 3)
-------------- bAND
0010 ( 2)
In a bitwise OR (inclusive) operation, the resulting bit is set to 1
when either or both input bits are 1. The resulting bit is set to 0 only
when both input bits are set to 0.
1010 (10)
0011 ( 3)
-------------- bOR (inclusive)
1011 (11)
In a bitwise OR (exclusive) operation, the resulting bit is set to 1 only
when one input bit is 1.
1010 (10)
0011 ( 3)
-------------- bXOR (exclusive)
1001 ( 9)
The bitwise NOT operator is a unary operator that produces the binary
complement of the value. A bit of 1 is set to 0 and a bit of 0 is set
to 1.
For example, the binary complement of 0 is -1, the maximum unsigned
integer (0xffffffff), and the binary complement of -1 is 0.
C:\PS> -bNOT 10
-11
0000 0000 0000 1010 (10)
------------------------- bNOT
1111 1111 1111 0101 (-11, xfffffff5)
Comments
Anonymous
September 27, 2013
Hi I have a serious situation on my hands. I made a local move request of 500 mailboxes from one DB to another. all of a sudden all those user mailboxes are not receiving emails because they get stuck in the "unreachable domain" queue. ive restarted services, restarted the exchange servers and still no luck. im really lost and don't know what to do anymore. pleae helpAnonymous
September 07, 2015
what happened after that ??Anonymous
September 16, 2015
Probably he got fired....poor guy :(