Speeding up the Get-MailboxStatistics cmdlet for ~19K mailboxes.

Question

Greetings,
While this is partially a PowerShell scripting question, I am posting it in the Exchange 2010 forum because the issues I believe center around the Get-MailboxStatistics itself, and the speed of my scripts in Exchange 2010 (possibly
due to the additional overhead in remote PowerShelling).
In my Exchange 2010 system we have ~19,000 mailboxes spread accross multiple DAG nodes, and ever since we upgraded from Exchange 2007 to Exchange 2010, gathering all of the mailboxes and then gathering their statistics takes almost
twice as long. For example a script that used to take ~45 minutes in Exchange 2007, takes about an hour and a ½.
The issue I am running into when clocking core aspects of a mailbox data gathering scripts is that the Get-MailboxStatistics seems to be taking an excessively long period of time, and I am hoping someone can help me figure out a
way to speed up the process.
For example this is a boiled down script I created, where I ripped out a ton of other things and just focused on the Get-Mailbox and Get-MailboxStatistics commands:
$BaseOU
=
"Customers"
# Capture the date and time in a variable using the "Fri 11/01/2010 6:00 AM" format.
$DateTime
=
Get-Date
-Format
"ddd MM/dd/yyyy h:mm tt"
# Select a single domain controller to use for all the queries (to avoid mid AD replication inconsistencies)
from the environment variable LOGONSERVER - this ensures the variable will always be dynamically updated.
$DomainController
= ($Env:LogonServer).Substring(2)
# Set the loop count to 0 so it can be used to track the percentage of completion.
$LoopCount
= 0
# Start tracking the time this script takes to run.
$StopWatch1
=
New-Object
System.Diagnostics.Stopwatch
$StopWatch1.Start()
# Get the mailbox info for all IHS customer mailboxes.the storage limit is Prohibit send or mailbox disabled
Write-Host
-ForegroundColor
Green
"Beginning mailbox gathering. In a short while a progress bar will appear."
$GatheredMailboxes
=
Get-Mailbox
-ResultSize:Unlimited
-OrganizationalUnit
"ADDomain.com/$BaseOU"
-DomainController
$DomainController |
Select Identity,DisplayName,ProhibitSendQuota
Write-Host
-ForegroundColor
Green
"Mailbox data gathering is complete."
$StopWatch1.Stop()
$StopWatch2
=
New-Object
System.Diagnostics.Stopwatch
$StopWatch2.Start()
Foreach ($Mailbox
in
$GatheredMailboxes) {
# Show a status bar for progress while the mailbox data is collected.
$PercentComplete
= [Math]::Round(($LoopCount++
$GatheredMailboxes.Count
* 100),1)
$CurrentMBDisplay
=
$Mailbox.DisplayName
Write-Progress
-Activity
"Mailbox Data Gathering in Progress"
-PercentComplete
$PercentComplete
`
-Status
"$PercentComplete% Complete"
-CurrentOperation
"Current Mailbox: $CurrentMBDisplay"
#Get the mailbox statistics for each mailbox gathered above.
$MailboxStats
=
Get-MailboxStatistics
$Mailbox.Identity |
Select StorageLimitStatus,TotalItemSize
# Proceed only if the the mailbox statistics show the storage limit is Prohibit Send or Mailbox Disabled.
# Write-Host "Stats for"$Mailbox.DisplayName"are Limit ="$MailboxStats.StorageLimitStatus"and Size ="$MailboxStats.TotalItemSize.Value.ToMB()"MB."
# Calculate the amount of time the script took to run and write the information to the screen.
$StopWatch2.Stop()
$ElapsedTime
=
$StopWatch1.Elapsed
Write-Host
"he mailbox gathering took"
$ElapsedTime.Hours
"hours,"
$ElapsedTime.Minutes
"minutes, and"
$ElapsedTime.Seconds
`
"seconds to run."
$ElapsedTime
=
$StopWatch2.Elapsed
Write-Host
"The foreach loop took"
$ElapsedTime.Hours
"hours,"
$ElapsedTime.Minutes
"minutes, and"
$ElapsedTime.Seconds
`
"seconds to run."
Using the two stop clocks, I was able to see that the Get-Mailbox of all mailboxes took ~9 minutes. That isn’t lightning fast, but it isn’t unreasonable.
The issue comes in where the Foreach loop with the Get-MailboxStatistics took ~53 minutes, and I am sure some of the mailbox data was cached on the servers from my various tests so it would probably take even longer with a cold
run.
I did some digging around and I really couldn’t find anything on how to speed up the Get-MailboxStatistics, and the only thing I found was this link:
http://social.technet.microsoft.com/Forums/en-US/winserverpowershell/thread/9ceefddd-7a59-44ec-8fc0-8de244acb58b
However I am not clear on how moving the Get-MailboxStatistics into the Get-Mailbox syntax (which is odd to me in general) would speed things up if I still have to have a foreach loop to process the data a little bit and add the
users to a datatable. That discussion also made think the foreach loop itself is slowing things down but unclear as to how/why if that is true. 
Can someone help share some ideas on how to speed up this process? I think there are some other things I could try but I can’t think of them.
Thank you in advance.

Answer 1

I think it's impossible to speed up the Get-MailboxStatistics when it is being called for each and every mailbox individually.
I read somewhere in other posts people were having better performance by calling the cmdlet against an entire database or server so I gave it a shot with this code:
$DAGS = "EXCHDAG1"
# Start tracking the time this script takes to run.
$StopWatch = New-Object System.Diagnostics.Stopwatch
$StopWatch.Start()
$MailboxStatistics = New-Object System.Data.DataTable “MailboxStatistics”
$MailboxStatistics.Columns.Add("TotalitemSize",[String]) | Out-Null
$MailboxStatistics.Columns.Add("ItemCount",[String]) | Out-Null
$MailboxStatistics.Columns.Add("LastLogonTime",[String]) | Out-Null
$MailboxStatistics.Columns.Add("LastLogoffTime",[String]) | Out-Null
$MailboxStatistics.Columns.Add("MailboxGUID",[String]) | Out-Null
$MailboxStatistics.PrimaryKey = $MailboxStatistics.Columns["MailboxGUID"]
ForEach ($DAGServer in (Get-DatabaseAvailabilityGroup $DAGS).Servers) {
ForEach ($MailboxStats in (Get-MailboxStatistics -Server $DAGServer.Name | Where {$_.DisconnectDate -eq $Null})) {
$NewMBXStatsDTRow = $MailboxStatistics.NewRow()
$NewMBXStatsDTRow.TotalitemSize = $MailboxStats.TotalItemSize
$NewMBXStatsDTRow.ItemCount = $MailboxStats.ItemCount
$NewMBXStatsDTRow.LastLogonTime = $MailboxStats.LastLogonTime
$NewMBXStatsDTRow.LastLogoffTime = $MailboxStats.LastLogoffTime
$NewMBXStatsDTRow.MailboxGUID = $MailboxStats.MailboxGuid.ToString()
$MailboxStatistics.Rows.Add($NewMBXStatsDTRow)
$StopWatch.Stop()
$ElapsedTime = $StopWatch.Elapsed
Write-Host "The script took" $ElapsedTime.Hours "hours," $ElapsedTime.Minutes "minutes, and" $ElapsedTime.Seconds `
"seconds to run."
Here are the results in speed:
The script took 0 hours, 3 minutes, and 13 seconds to run.
So yeah... ~3 minutes versus ~1 hour, I would say that's an improvement.
Now I will go back to my script and as I process each mailbox I will pull it's statistics information out of the DataTable using its GUID with:
If ($MailboxStats = $MailboxStatistics.Rows.Find($Mailbox.MailboxGUID)) {
# Insert mailbox statistics processing here using the $MailboxStats variable with "." extensions.
} Else {
# Mailbox statistics weren't found so go grab them individually as a backup mechanism for scenarios when a user's mailbox got moved out of the DAG to a non-DAG database for whatever reason.
It's a little silly that I have to extract the information out of each DAG server and put it in an in-memory table just to speed this process up, but clearly there is overhead with the Get-MailboxStatistics cmdlet and grabbing more mailboxes at once helps
negate this issue.
I'm going to mark my own response as an answer because I don't think anyone else is going to come up with something better than what I put together.