Troubleshoot connectivity issues Outlook Anywhere - Exchange 2013

Question

Hi there,
As part of our Exchange 2010 -> 2013 migration we've transitioned CAS to Exchange 2013 2 weeks ago. Some 50 mailboxes have been moved to exchange 2013. Moving mailboxes for everyone is scheduled the 2nd week of july . Because our current version of Outlook
is 2007 (migration to 2013 is due thissummer) we've configured NTLM authentication for OA.
Exchange setup: 8 Multirole (CAS/MBX) virtual (VMware) Servers: each 4 cores, 24 Gb memory (reserved) : Windows 2012 SP1, Exchange 2013 SP1 (15.0.847.4030)
Right now we're facing client connectivity issue's: Outlook Anywhere clients are continiously losing connection with exchange, some people (outllook 2007/exc. 2007) report every minute or worse.... Moving the mailboxes of affected people results in
less problems: Sometimes no disconnects for 10 to 30 minutes, then reconnects every minute for some time.
We've already set the timeout for the oa-pool in our network proxy (riverbed steelapp) to 20 minutes and the minimum keep alive on the 2013 servers to 120 seconds which improved Oultook 2013 clients; before I experienced reconnects every minute, after every
10 to 30 minutes (with periods of reconnects every minute)
testconnectivity.microsoft.com gives positive results (apart from a nspi warning about server side encryption)
testing with rpcping according to
http://blogs.technet.com/b/exchange/archive/2008/06/20/3405633.aspx gives some interesting results:
I've tested all (8) CAS 2013 servers over ports 6001, 6002 and 6003; each 100 rpcpings with a for loop: every response was either about 500 ms or about 21 Seconds ?!?
How can we furher troubleshoot the reason of the long reply time. Eventlogs (and SCOM with exchange 2013 MP) show no relevant events or alerts.
Part of the output of a rpcping
 RPCPing v6.0. Copyright (C) Microsoft Corporation, 2002-2006
 RPCPing set Activity ID:  {59b56c7f-af5d-4836-b701-92070f674de6}
 Completed 1 calls in 452 ms
2 T/S or 452.000 ms/T
 RPCPing v6.0. Copyright (C) Microsoft Corporation, 2002-2006
 RPCPing set Activity ID:  {1197cd5e-c79d-4659-b598-3134c335b103}
 Completed 1 calls in 468 ms
2 T/S or 468.000 ms/T
 RPCPing v6.0. Copyright (C) Microsoft Corporation, 2002-2006
 RPCPing set Activity ID:  {0cbaef91-ec96-402e-aa00-4913e2be1c51}
 Completed 1 calls in 483 ms
2 T/S or 483.000 ms/T
 RPCPing v6.0. Copyright (C) Microsoft Corporation, 2002-2006
 RPCPing set Activity ID:  {525717e5-441b-4a8e-8398-dc86d38852c7}
 Completed 1 calls in 21450 ms
0 T/S or 21450.000 ms/T
 RPCPing v6.0. Copyright (C) Microsoft Corporation, 2002-2006
 RPCPing set Activity ID:  {408d806d-ed5a-4f96-8c3c-2446a1d48ad8}
 Completed 1 calls in 21497 ms
0 T/S or 21497.000 ms/T
 RPCPing v6.0. Copyright (C) Microsoft Corporation, 2002-2006
 RPCPing set Activity ID:  {3b441a9f-7606-4106-850f-fccb7c0f1bb1}
 Completed 1 calls in 21497 ms
0 T/S or 21497.000 ms/T
 RPCPing v6.0. Copyright (C) Microsoft Corporation, 2002-2006
 RPCPing set Activity ID:  {bf994811-8528-433f-b532-f29d347fce5b}
 Completed 1 calls in 21590 ms
0 T/S or 21590.000 ms/T
 RPCPing v6.0. Copyright (C) Microsoft Corporation, 2002-2006
 RPCPing set Activity ID:  {ddb5248b-82aa-4586-b2f7-9c04c9922034}
 Completed 1 calls in 577 ms
1 T/S or 577.000 ms/T
Summary of all servers (test this morning)
Server
Port
# >20.0001
Server1
6001
32/100
Server1
6002
27/100
Server1
6004
0/100
Server2
6001
47/100
Server2
6002
0/100
Server2
6004
37/100
Server3
6001
0/100
Server3
6002
0/100
Server3
6004
41/100
Server4
6001
0/100
Server4
6002
29/100
Server4
6004
42/100
Server5
6001
69/100
Server5
6002
48/100
Server5
6004
69/100
Server6
6001
0/100
Server6
6002
0/100
Server6
6004
1/100
Server7
6001
0/100
Server7
6002
1/100
Server7
6004
1/100
Server8
6001
0/100
Server8
6002
0/100
Server8
6004
0/100
I've repeated above test this afternoon: All test resulted in about 40-60 (of 100) replies >20 seconds

Answer 1

We've changed a timeout setting in the Steelapp virtual server (old: 10 sec; new: disabled) the connections between Outlook and Exchange are, when established, very stable (almost no failed request anymore)
However there still existst a conectivity issue:
The 8 Exchange 2013 servers are placed in 2 different Active Directory sites (4 servers in each site) and I have found that a cross site rpcping consequently takes more than 20 seconds (with the loadbalancer bypassed ! !) where a rpcping on the
same AD-site takes 200-300 miliseconds...
rpcping -t ncacn_http -o RpcProxy=Host-in-site-A -P "user,domain,password" -H 2 -F 3 -a connect -u 10 -v 3 -s RpcProxy=Host-in-site-B -I "user,domain,password" -e 6001 => 20+ seconds
rpcping -t ncacn_http -o RpcProxy=Host-in-site-A -P "user,domain,password" -H 2 -F 3 -a connect -u 10 -v 3 -s RpcProxy=Host-in-site-A -I "user,domain,password" -e 6001 => 200 miliseconds
rpcping -t ncacn_http -o RpcProxy=Host-in-site-B -P "user,domain,password" -H 2 -F 3 -a connect -u 10 -v 3 -s RpcProxy=Host-in-site-B -I "user,domain,password" -e 6001 => 200 miliseconds
rpcping -t ncacn_http -o RpcProxy=Host-in-site-B -P "user,domain,password" -H 2 -F 3 -a connect -u 10 -v 3 -s RpcProxy=Host-in-site-A -I "user,domain,password" -e 6001 => 20+ seconds
The same tests with our Exchange 2010 CAS and MBX (NO multirole) shows fast (300 ms.) with every combination. The servers are both on the same networks in each site
We've already started talking with the network guy's: There should be no rules between both networks.
OWA, Autodiscover, EAS all work fine.
How or where to troubleshoot this slow response between two AD site's ??