Performance impact of going via Proxy seems high?

Question

Just wondered if anyone had tested the impact of a client app. that joins the cluster directly, versus one that goes via a Proxy?
I've been testing retrievals of Position objects (roughly 300 bytes each) on 4 server cluster (2 Xeon Quads in each). The Positions cache is a distributed cache and the objects are stored in POF format. For queries that return few positions (i.e. 10s/100s), the performance seems pretty comparable, whether you go via the Proxy or not. However, as I query more Positions, the Proxy figures get much slower (50-90%), and seem much more volatile.
For example, here's the timings for queries without the proxy:
Total Positions          Time(ms)
24598.....68
38579.....294
97781.....385
106151.....449
107317.....433
107518.....436
And here's the same queries via the Proxy (Proxy running on the same servers, same LAN, as is the client):
Total Positions          Time(ms)
24598.....290
38579.....421
97781.....754
106151.....591
107317.....787
107518.....604
As you can see, the figures are much slower in places, and also much more volatile (look at the last two rows, for example.) CPU stats are low, and I've configured plenty of RAM in the JVMs for both the cache servers and proxies. I've also got plenty of service threads configured on both the Cache Service and the Proxy service (although I'm only running one test at a time, so this shouldn't be a factor anyhow.)
I've looked through the "Best Practice" section of the Docs for proxies, and I'm following all the hints given in it. Still, I'm very surprised by how much of a hit the Proxy seems to be introducing. Is this normal? Or is there something I can try to improve the situation?
Cheers,
Steve

Answer 1

Hi Wei,
Thanks for the update.
I've re-run the tests, and the results are pretty much the same. If I go via the proxy, the time increases by about a factor of 2. For smaller result sets (but still 25k entries), that can be as high as 4. Here's the data from a re-run:
With Proxy:
Total Books Total Positions Indexed Time(ms)
5 24598 true 179
10 38579 true 422
50 97781 true 880
100 106151 true 770
200 107317 true 802
300 107518 true 841
Without:
Total Books Total Positions Indexed Time(ms)
5 24598 true 67
10 38579 true 296
50 97781 true 391
100 106151 true 436
200 107317 true 415
300 107518 true 439
I actually run a whole variety of tests, these results are just a sub-set. I have indexes/queries (using auto-generated ReflectionExtractors, as I use CohQL/QueryHelper) and select Positions using "where" clauses against Book, AggregationUnit and Instrument - all attributes of this one object type. I also do a "key().type = 'XXX'" call in the where clause also.
General format of all the code is ("book" type selections shown here):
// Create a Filter to select Positions for the supplied Book.
Filter filter = QueryHelper.createFilter("key().type = 'TRADING' and bookId in " + bookList);
// Create an aggregator to extract the data required from the Position.
ReducerAggregator aggregator = new ReducerAggregator(new MultiExtractor("getBookId,getPositionType"));
// Get the Positions and iterate through them.
Map positionsMap = (Map) cache.aggregate(filter, aggregator);
Set<Map.Entry<TradingPositionKey,List>> positions = (Set<Map.Entry<TradingPositionKey,List>>)positionsMap.entrySet();
for(Map.Entry<TradingPositionKey,List> e : positions) {
If, as you suggest, these timings match Oracle's own internal testing, then I guess I'll just have to accept it. Still, was a little higher than what I thought it would be. I expected the proxy to simply act as a "pipe" between my client and the grid and not really do much, adding nothing more than a few milli-seconds overhead at most.
Proxy, client and cache server are all using POF, BTW, all reading it from the same pof-config file.
Cheers,
Steve