Large Tenant and PowerShell I/O timeout
So did you ever run a large office365 Tenant with million of mailboxes and you wanted to run a report about devices stat for example and it keep failing?
Did you ever face a timeout error when you run a long report “The I/O operation has been aborted because of either a thread exit or an application request”?
It’s not quite easy with the network latency and the exchange throttling policy which is hardly implemented at the office365 to have a long running session opened with exchange especially that some commands consume more time like Get-mobileDeviceStatistics
So how can we bypass this limitation?
Let’s start with a sample code to get the device statistics for all users in one tenant.
| PowerShell |
#Retrieve the list of mailboxes $MbxList=(Get-Mailbox -resultsize unlimited).alias #Loop in the list and retrieve the device’s information $file = "e:\activesync\office365$((get-date).tostring("yyyyMMdd")).csv" ForEach($mbx in $MbxList) { $deviceinfo = Get-mobileDeviceStatistics -mailbox $mbx | select @{name="mailbox"; expression={$mbx}},devicetype, devicemodel, devicefriendlyname, deviceos, lastsuccesssync If ($deviceinfo -ne $null) { If (Test-Path $file){ $mbx.alias + “,” + ($deviceinfo | ConvertTo-Csv)[2] | Out-File $file -Append } Else{
}
} } |
With over 50k mailboxes we might hit the limitations and receive the famous error
Starting a command on the remote server failed with the following error message : The I/O operation has been aborted because of either a thread exit or an application request. For more
So how to bypass the limitations?
A first analysis will show that we are running two commands Get-mailbox and Get-mobileDeviceStatistics so why not we run each command in a separate session.
In stage 1 we starts with Get-Mailbox and send the results to a csv file on the disk and once the command finish we start stage 2 with a new PowerShell session and run the second command for Get-mobileDeviceStatistics .
Looks promising but again with 50k or more of mailboxes the Get-mobileDeviceStatistics will throw the same error as it last for a long time.
What if we have the list of mailboxes less than 1000 mailbox? cool that will work fine, but is it possible to load 1000 entries only from the csv file?
What’s really brilliant is that you can load some lines from any csv file using the ‘-skip’ option in combine with the ‘-First’ so we load from 0..1000 and close the file, open again and load from 1000…2000.
| PowerShell |
$Offset=0; $PageSize=1000; $mbxMax=(import-csv users.csv).count $file = "office365$((get-date).tostring("yyyyMMdd")).csv" do{ #Load the list of users from a csv file limited with the pageSize 1000 starting from the line $Offset + 1 $mbxlist=@(import-csv users.csv|select-object -skip $Offset -First $PageSize) "Start at offset $($Offset) till $($Offset+$PageSize)" ForEach($mbx in $mbxlist) {
{
{ $mbx.alias + ”,” + ($deviceinfo | ConvertTo-Csv)[2] | Out-File $file –Append
}
{
}
} "------------------------------" #Increase the start point for the next chunk $Offset+=$PageSize } while($offset -lt $mbxMax) |
$PageSize variable here defines the number of items will processed in each run
That will split the 50k mailboxes into multiple chunks and process each chunk in a separate ForEach loop.
But we still run all commands in the same session, what we really need to add is to run each chunk in a new session and aggregate all the results.
This is the time to write a small function for office365 remote connection so that we close the session and reopen a new one without any interaction.
| PowerShell |
| Function New-O365ExchangeSession(){param( [parameter(mandatory=$true)]$365master)#close any old remote sessionGet-PSSession | Remove-PSSession -Confirm:$false#start a new office 365 remote session$365session = New-PSSession -ConfigurationName "Microsoft.Exchange" -ConnectionUri 'https://ps.outlook.com/powershell' -Credential $365master -Authentication Basic -AllowRedirection$office365 = Import-PSSession $365session} |
The last thing is to call the remote connection function inside the foreach loop so we run each loop in a separate remote session
Finally we reached the objective; Enjoy 
Final Code |
| Function New-O365ExchangeSession(){param( [parameter(mandatory=$true)]$365master)#close any old remote sessionGet-PSSession | Remove-PSSession -Confirm:$false#start a new office 365 remote session$365session = New-PSSession -ConfigurationName "Microsoft.Exchange" -ConnectionUri 'https://ps.outlook.com/powershell' -Credential $365master -Authentication Basic
-AllowRedirection$office365 = Import-PSSession $365session}
#Main
$Offset=0; $PageSize=1000; $mbxMax=(import-csv users.csv).count $file = "office365$((get-date).tostring("yyyyMMdd")).csv" $365master = get-credential ashour@mylab.onmicrosoft.com New-O365ExchangeSession $365master # call the office365 remote connection function do{ #Load the list of users from a csv file limited with the pageSize 1000 starting from the line $Offset + 1 $mbxlist=@(import-csv users.csv|select-object -skip $Offset -First $PageSize) "Start at offset $($Offset) till $($Offset+$PageSize)" ForEach($mbx in $mbxlist) {
{
{ $mbx.alias + ”,” + ($deviceinfo | ConvertTo-Csv)[2] | Out-File $file –Append
}
{
}
} "------------------------------" #Increase the start point for the next chunk $Offset+=$PageSize #Call the office365 remote session function to close the current one and open a new session New-O365ExchangeSession $365master } while($offset -lt $mbxMax) |
Disclaimer The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.
Comments
- Anonymous
May 25, 2014
Thanks Ahmed for this wonderful script , I used it with a tenant that has more than 500 thousands users , and it works as it should ! - Anonymous
June 22, 2014
Great - Anonymous
June 23, 2014
Amazing :) - Anonymous
June 29, 2014
Excellent as usual Ahmed.Would it be possible to post it on Exchange team blog ? - Anonymous
September 08, 2014
Hello,Well it still does not work for me.I'm getting the same error when it reaches 4500-6500 users and it's asking me for credentials again.Earlier when i didn't found you blog. I tried to use 'if' and close old session when users counter reached (-like "*000") and opening new one. But i still have same issues.I even tried with other settings in New-PSSessionOption but it didn't help as well.Any ideas what could be wrong ? - Anonymous
September 10, 2014
Nice script Ahmed - will save some admins a lot of time. - Anonymous
June 25, 2015
This is good stuff. we need more sessions