Windows 2008 R2 SP1 - Exchange 2010 SP2, RU7, 3 node DAG - Possible Port Exhaustion?

Question

Hey TechNet
So today at about 10:00 am the exchange server dropped all connectivity (except for ping) and was not accessible by RDP and I could not connect to the service control panel etc. (from a remote machine). In addition I could not log on to the console, when
I pressed ctrl+alt+del - nothing happened. (This is running on VMware and I was not able to to use the VMware tools to restart the server either) it had to be "reset" in the console.
I found numerous errors in the event viewer as you might suspect, namely that the server could not locate any domain controllers, global catalogs etc. Of course all the mailbox databases failed over to the other dag member and no one noticed (so
that worked well).
In checking some settings, I have determined that the servers memory page file is not set to RAM+10MB, but it was never set anyways, in addition
static RPC ports are not set. The client was doing mailbox moves today (I am going to confirm with the client again to make sure)
My question to you is this: will not setting static RPC ports cause port exhaustion? and if so can you provide a link that says as much?
Thanks,
Robert
Robert

Answer 1

Rhoderick,
Thanks for the response. I reviewed several different links regarding this especially some of the ones related to memory exhaustion I didn't see RADAR events in the event log. Here is a short list of some of the events that I have seen in the event viewer.
ESE, 490 (Time Occurred: 9:47 am), several instances.  
Information Store (5524) db-7: An attempt to open the file "V:\Program Files\Microsoft\Exchange Server\V14\Mailbox\database\E0E.chk" for read / write access failed with system error 1450 (0x000005aa):
"Insufficient system resources exist to complete the requested service. ".  The open file operation will fail with
error -1011 (0xfffffc0d).
ESE 906 (Time Occurred 10:00 am), several instances.
Information Store (5524) A significant portion of the database buffer cache has been written out to the system paging file.  This may result in severe performance degradation.
See help link for complete details of possible causes.
Resident cache has fallen by 78083 buffers (or 38%) in the last 118 seconds.
Current Total Percent Resident: 16% (124338 of 762960 buffers)
MSExchangeREPL 2170 (Time occurred, 9:51) several instances.
A slow file IO operation was encountered on file 'V:\Program Files\Microsoft\Exchange Server\V14\Mailbox\database7\E0E000BB30C.log' for copy 'db-7\CBEXCH1'. The observed latency was 9737.6786
ms while performing a 'MissingFileNotification'. This may indicate an overloaded system or a storage-related problem.
MSExchagneADaccess 2070, (Time Occurred, 9:56 am) several instances.
Process Microsoft.Exchange.EdgeSyncSvc.exe (PID=2864).  Exchange Active Directory Provider lost contact with domain controller dc.contoso.local.  Error was 0x51 (ServerDown) (Active
directory response: The LDAP server is unavailable.).  Exchange Active Directory Provider will attempt to reconnect with this domain controller when it is reachable. 
MSExchangeADaccess 2122, (Time Occurred 9:57) Several Instances.
Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=164). Error 0x8007267c occurred when DNS was queried for the service location (SRV) resource record used to locate a domain controller for domain
contoso.local
The query was for the SRV record for _ldap._tcp.dc._dc.contoso.local
For information about correcting this problem, Type in the command line:
hh tcpip.chm::/sag_DNS_tro_dcLocator_messageA.htm
While I do understand that the to really nail this down we would need to setup a data capture set using as you suggested perfmon, and probably (prodump, netmon, etc) I was trying to get the client an answer.
I do have 2 perfmon collectors from last week and the week before,
to be clear these captures were not taken when issues occurred; they show some memory pressure on 1 or 2 counters with some high spikes, but in most cases all the averages (with the exception of 2 counters) are completely below the thresholds.
All captures were done for 4 hours @ 15 secs.
Page Reads/Sec, has an average of 19.433 with a max of 909.51, There are about 10 spikes over 100
Pages Input/Sec, Shows an Average of 132.981, Max, 6,695 (I realize the max is bad), I see 13 Instance's of this counter spiking above 1,000.
Pages Output/Sec, Shows An average of 38.113, Max, 5,221 (I realize the max is bad here), I see about 8 instances of spikes over 1,000.
Systemcache Resident Bytes remains stable with very minor increases over the entire capture.
Pool non paged bytes, is stable across the entire capture as well (Stable, meaning straight line)
Working Set for Store.exe, stable, straight line almost no changes.
Working set for Total, Stable, Straight line almost no changes.
% Committed Bytes in use is at 60%, stable across the whole capture.
This server does not have its page file set to RAM+10MB its set to Dynamically grow. My plan (before yesterday) was to set the page file correctly, then recapture and see if there were any differences. My problem that I have experienced here is that none
of the counters seem to correlate, or if they do its not really a complete correlation that tells me what's going on.
Thanks for your help here!!
Robert
Robert