Connection timeout depending on how many client JVMs (but as many requests)

Question

Hello,
sorry for the cryptic title, I hit the length limit.
Here is my problem. I'm asking first for an explanation, or for ways to investigate it. Afterwards I'll need a solution but as I suspect the problem is involved, and involves several tiers, I will probably open dedicated topics instead.
Short Story*: a web service server and 10 test client JVMs work fine - but a single test client simulating 10 virtual users experiences ConnectException.
What can I log or probe to understand why?
Long Story*:
I have an application that I used to simulate load on a WebService server. The app simulates a variable number of clients of these WebServices, each virtual client running in its own thread, and sequentially calling the same set of WebServices in a fixed order, and looping.
I have, on some occasions, met various problems with the server product (under development and validation), or the test client (that may not have had the same level of attention and quality in the design as the product), some of you may remember some of my asking here and there. But we achieved a fair level of reliability.
However, today, after a few weeks' change to the server(1), we load-tested it using the test client simulating 10 concurrent clients. After a few minutes, we repeatedly witnessed a fair number (1% of the requests) of java.net.ConnectException "connection timed out" raised at the client's end. I immediately suspected the server to not honor connection requests timely, and went on monitoring the server's queues, listeners,etc.
As nothing seemed to highlight a problem on this side, a colleague suggested that we instead launch 10 client JVMs (on the same single test client machine), each one simulating one single client. I blatantly qualified his suggestion of "hopeless random try" but let him try his way.
Dang! He repeatedly demonstrated that the 10 client JVMs do not undergo the ConnectException, even though they simulate the same number of concurrent requests (well, 10 at most), at the same thoughput - and we observe the same level of server CPU, and same functional results.
I respectfully apologized for being a pretentious moron (2), but he hasn't proposed any satisfactory explanation so far, and I'm clueless too. So we haven't been able to make the "single JVM load-test client" work. We need to make this one work, for various reasons.
There may be some server-side tweaks (worker thread pools, waiting queues,... for the record it's a Glassfish server) that I may tune, but as the "regular" situation of 10 client VMs is handled appropriately, I deem the server-side setting are correct.
So I'm primarily interested in monitoring the client host and JVM(s) to spot the differences between the two scenarios. But there may be some server-side indicators as well (queue size, keep-alive pool) that may help the investigations - and if need be, I will happily tune the server for the benefit of the "test client investigating".
One track I'd like to investigate, is the way HTTP connections are reused: we're using a JAX-WS client querying a Glassfish server, and both seem to agree on using HTTP1.1 (I checked HTTP packets with tcpmon), which supposedly "reuses connections" (is that the same as "keeps TCP socket open and reuses them"?). Could it be that the 10 threads in the multi-client simulator have a bottleneck of using a shared HTTP connection(s)? I don't know how to log or monitor this:
- I did a couple of netstat dumps during the tests on the (Windows) client test machine, and it exhibits only a few hundreds of sockets in TIME_WAIT state, and a few dozens of ESTABLISHED ones.
- I don't have the numbers for the server, though.
I don't know how to investigate that further. Any clues?
I may exhibit a poor turnaround time in this thread, as the test platform is used for real tests, but I do care about the topic, so I attach shiny little dukes.
Thank you for your time (well if you've read this to the end you at least deserve a thanks!) (3)
J.
(1) and after switching test platform (room & network); lesson one: "Never mix test platform changes and other changes in the same iteration; regression-test the new test platform with the former software version"
(2) lesson two: "When you don't know, don't tell a colleague he doesn't know"
(3) lesson three: "Short posts get better attention, are better understood, and more happily replied to"... (sigh)

Answer 1

Hello ejp,
I not only missed your points, but missed your latest post as well :o)
I'm discovering it.
You've missed the point. Problems that only shows up with multiple clients in the same JVM but not with multiple JVMs don't correspond to real-world situations any more than the configuration does so there is little in even investigating them, and no point whatsoever in trying to fix them.
(...) It's not 'hopeless', but you have to be constantly aware that it can lead to false positives and false negatives.Fixing them woud enable me to continue load tests with a cheaper test platform. I cannot afford 10, 100 or 1000 client machines. I know you're talking about JVMs, but 10 or 100 client JVMs running on a single host is not a real-world situation either, no more than "all clients are on the same LAN". I have to do away with approximations, keeping aware as completely as possible of the differences it introduces.
This is not true for HTTP connection caching: this favors the server as much as the client, so I can't infer that it's a favorable case overall.Again you've missed the point. It can only happen inside a single JVM. So it may present you with false positives.I'm still not sure I got the point now:
-> For the server, wich are the differences between both client configurations (single-JVM vs 10 JVMs)?
- - Does a new TCP connection request from the same JVM involve less work if one is already established?
If not, and if each virtual client uses its own TCP connection, there is no cheating, and I don't need to worry about this level of bias.
- - Does an HTTP request that comes over an established TCP socket require less server work than an HTTP request that reqests a new TCP socket?
Probably (in the second case the server has to accept the TCP request and probably pool it, and assign a reader thread, etc...). So I do need to worry about this bias. One way to avoid it is to manage to have the virtual clients not using each other's HTTP connection.
And so on...
Is it possible to build a comprehensive list of the differences incurred by the single-JVM configuration?
Then for each of them, I can try to evaluate ways to alleviate the bias.
- If I managed to avoid or alleviate each and every bias, I would be back to a realistic simulation.
- If there remained only the biases that handicap the server (such as, the server crashes under a 15 clients load, whereas in reality it could sustain 25), then I could be conservative and announce "the server will handle 15 clients gracefully" to the salespeople that will establish the SLAs. I understand that I may never be able to prove that the server actually scales up to 25 concurrent clients, but as long as the conservative SLA matches the market target that will do.
- If I'm left with biases that do favor the server, then I can't rely on the test platform and I'll reluctantly but resignedly try to have a more realistic platform funded (the more likely outcome is to be asked to provide probabilities and disclaimers in the test reports).
Thanks for your help so far, and please bear with my stubborn but polite request to try to identify further the differences ad causes.
J.
Edited by: jduprez on Dec 14, 2009 5:56 PM
Edited by: jduprez on Dec 14, 2009 5:57 PM