Exchange 2010 Slow Database Mount Time - Almost 1 Hour

Question

Hello -
We have 2 DAG nodes on the production site running EX2010 SP3 RU6
In preparation for Windows patching I moved the active copy of DB2 active from EX2 to EX1, this took approx 20 seconds as per usual
Patching happened, and EX2 got a reboot
Later that evening I moved the active copy of DB2 back from EX1 to EX2
The move took 59 minutes and 32 seconds
During which time the DB status showed as mounting on EX2 and mailboxes where inaccessible [luckily this was out of hours]
The DB is 185 GB with 4 GB of logs at that time
There weren't ESE events related to playing transaction logs
There have been ESE 629 events in the log for a while showing... "While attempting to move to the next or previous node in a B-Tree, the database skipped over 25583 non-visible nodes in 73 pages... The database may benefit from widening the online maintenance
window..." but these warnings show up really quickly on our site even with a new database
We also have a problem that emerged approx 2 weeks ago where CPU runs at 100% for prolonged periods [mainly store.exe for 10/20/30 mins] and people get slow response times in Outlook, it's running on 4 x vCPU's and 18 GB RAM
Could the really slow mount time be related to DB corruption?
Thanks

Answer 1

Thanks for the feedback
This is not related to a dirty shutdown
According to Get-MailboxDatabaseCopyStatus etc all DB copies are healthy
I tested the DB activation again last night
Mailbox Database 2 is normally active on EX2
So we made the copy on EX1 active [server is on the same network] and this took 3 secs
When it came to making the same DB active on EX2 again it took 1 hr 46 mins 34 seconds!
We ran the task from EMC and the GUI did not hang, all the while it kept counting the hours, minutes & seconds
During this time mailboxes were unavailable, and the DB status showed as Mounting in EMC
From the Event Log, when ESE event 326 is finally logged there are some timings…
 Internal Timing Sequence: [1] 0.000, [2] 0.000, [3] 0.031, [4] 0.000, [5] 0.000, [6] 0.094, [7] 0.000, [8] 6386.578,
[9] 0.016, [10] 0.000, [11] 0.000, [12] 0.047.
And step 8 [whatever that is] took 6386 seconds while everything else flew
Of the 2 other Mailbox DB’s… DB1 took 7 seconds to activate on EX2 and DB3 took 32 minutes 41 seconds
Any ideas what is going on?
Thanks
Log Name:      Application
Source:        MSExchangeRepl
Date:          12/08/2014 21:02:02
Event ID:      2090
Task Category: Action
Level:         Information
Keywords:      Classic
User:          N/A
Computer:     
EX2.company.com
Description:
Database: Mailbox Database 2\EX2
Mailbox server:
EX2.company.com
A failover or switchover occurred. Database Mailbox Database 2\EX2 will be
mounted with no data loss.
Log Name:      Application
Source:        MSExchangeIS Mailbox Store
Date:          12/08/2014 21:02:05
Event ID:      9539
Task Category: General
Level:         Information
Keywords:      Classic
User:          N/A
Computer:     
EX2.company.com
Description:
The Microsoft Exchange Information Store database "c21e43b5-feb8-4bde-a993-aca830cd71a3: /o=Company X/ou=Exchange
Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=xxx.company.com/cn=Microsoft
Private MDB" was stopped.
Log Name:      Application
Source:        MSExchangeIS Mailbox Store
Date:          12/08/2014 21:02:05
Event ID:      1000
Task Category: General
Level:         Information
Keywords:      Classic
User:          N/A
Computer:     
EX2.company.com
Description:
Attempting to start the Information Store "Mailbox Database 2".
Log Name:      Application
Source:        ESE
Date:          12/08/2014 21:02:05
Event ID:      102
Task Category: General
Level:         Information
Keywords:      Classic
User:          N/A
Computer:     
EX2.company.com
Description:
Information Store (4308) Mailbox Database 2:  The database engine (14.03.0162.0000) is starting a new instance (4).
Log Name:      Application
Source:        ESE
Date:          12/08/2014 21:02:06
Event ID:      105
Task Category: General
Level:         Information
Keywords:      Classic
User:          N/A
Computer:     
EX2.company.com
Description:
Information Store (4308) Mailbox Database 2: The database engine started a new instance (4). (Time=0 seconds)
Internal Timing Sequence: [1] 0.000, [2] 0.000, [3] 0.109, [4] 0.079, [5] 0.000, [6] 0.000, [7] 0.000, [8] 0.000, [9] 0.219, [10] 0.000, [11] 0.000.
Log Name:      Application
Source:        ESE
Date:          12/08/2014 22:48:33
Event ID:      326
Task Category: General
Level:         Information
Keywords:      Classic
User:          N/A
Computer:     
EX2.company.com
Description:
Information Store (4308) Mailbox Database 2: The database engine attached a database (4, X:\Database\Mailbox Database 2\Mailbox Database 2.edb). (Time=6386 seconds)
Internal Timing Sequence: [1] 0.000, [2] 0.000, [3] 0.031, [4] 0.000, [5] 0.000, [6] 0.094, [7] 0.000, [8] 6386.578, [9] 0.016, [10] 0.000, [11] 0.000, [12] 0.047.
Saved Cache: 1