How to diagnose performance problems?

Question

Hi all,
I'm trying to run some basic performance tests of our app with Coherence, and I'm getting some pretty miserable results. I'm obviously missing something very basic about the configuration, but I can't figure out what it is.
For our test app, I'm using 3 machines, all Mac Pros running OSX 10.5. Each JVM is configured with 1G RAM and is running in server mode. We're connected on a gigabit LAN (I ran the datagram test, we get about 108Mb/sec but with high packet loss, around 10-12%). Two of the nodes are storage enabled, the other runs as a client performing an approximation of one of the operations we perform routinely.
The test consists of running a single operation many times. We're developing a card processing system, so I create a bunch of cards (10,000 by default), each of which has about 10 related objects in different caches. The objects are serialised using POF. I write all these to the caches, then perform a single operation for each card. The operation consists of about 14 reads and 6 writes. I can do this about 6-6.5 times per second, for a total of ~120 Coherence ops/sec, which seems extremely low.
During the test, the network tops out at about 1.2M/s, and the CPU at about 35% on the storage nodes and almost unnoticeable on the client. There's clearly something blocking somewhere, but I can't find what it is. The client uses a thread pool of 100 threads and has 100 threads assigned to its distributed service, and the storage nodes also have 100 threads assigned to their distributed services. I also tried giving more threads to the invocation service (since we use VersionedPut a lot), but that didn't seem to help either. I've also created the required indexes.
Any help in diagnosing what the problem would be greatly appreciated. I've attached the config used for the two types of node below.
Cheers,
Colin
Config for storage nodes:
-Dtangosol.coherence.wka=10.1.1.2
-Dtangosol.coherence.override=%HOME%/coherence.override.xml
-Dtangosol.coherence.distributed.localstorage=true
-Dtangosol.coherence.wkaport=8088
-Dtangosol.coherence.distributed.threads=100
<cache-config>
<caching-scheme-mapping>
<cache-mapping>
<cache-name>*</cache-name>
<scheme-name>distributed-scheme</scheme-name>
</cache-mapping>
</caching-scheme-mapping>
<caching-schemes>
<distributed-scheme>
<scheme-name>distributed-scheme</scheme-name>
<service-name>DistributedCache</service-name>
<backing-map-scheme>
<local-scheme>
<scheme-ref>distributed-backing-map</scheme-ref>
</local-scheme>
</backing-map-scheme>
<autostart>true</autostart>
<serializer>
<class-name>runtime.util.protobuf.ProtobufPofContext</class-name>
</serializer>
<backup-count>1</backup-count>
</distributed-scheme>
<local-scheme>
<scheme-name>distributed-backing-map</scheme-name>
<listener>
<class-scheme>
<class-name>service.data.manager.coherence.impl.utilfile.CoherenceBackingMapListener</class-name>
<init-params>
<init-param>
<param-type>com.tangosol.net.BackingMapManagerContext</param-type>
<param-value>{manager-context}</param-value>
</init-param>
<init-param>
<param-type>java.lang.String</param-type>
<param-value>{cache-name}</param-value>
</init-param>
</init-params>
</class-scheme>
</listener>
</local-scheme>
<invocation-scheme>
<service-name>InvocationService</service-name>
<autostart>true</autostart>
</invocation-scheme>
</caching-schemes>
</cache-config>
Config for client node:
-Dtangosol.coherence.wka=10.1.1.2
-Dtangosol.coherence.wkaport=8088
-Dtangosol.coherence.distributed.localstorage=false
<cache-config>
<caching-scheme-mapping>
<cache-mapping>
<cache-name>*</cache-name>
<scheme-name>distributed-scheme</scheme-name>
</cache-mapping>
</caching-scheme-mapping>
<caching-schemes>
<distributed-scheme>
<scheme-name>distributed-scheme</scheme-name>
<service-name>DistributedCache</service-name>
<thread-count>100</thread-count>
<backing-map-scheme>
<local-scheme>
<scheme-ref>distributed-backing-map</scheme-ref>
</local-scheme>
</backing-map-scheme>
<autostart>true</autostart>
<backup-count>1</backup-count>
</distributed-scheme>
<local-scheme>
<scheme-name>distributed-backing-map</scheme-name>
</local-scheme>
<invocation-scheme>
<service-name>InvocationService</service-name>
<autostart>true</autostart>
</invocation-scheme>
</caching-schemes>
</cache-config>

Answer 1

Hi David,
Thanks for the quick response. I'm currently using an executor with 100 threads on the client. I thought that might not be enough since the CPU on the client hardly moves at all, but bumping that up to 1000 didn't change things. If I run the same code with a single Coherence instance in my local machine I get around 110/sec, so I suspect that somewhere in the transport layer something doesn't have enough threads.
The code is somewhat complicated because we have an abstraction layer on top of Coherence. I'll try to post the relevant parts below:
The main loop is pretty straightforward. This is a Runnable that I post to the client executor:
public void run()
Card card = read(dataManager, Card.class, "cardNumber", cardNumber);
assert card != null;
Group group = read(dataManager, Group.class, "id", card.getGroupId());
assert group != null;
Account account = read(dataManager, Account.class, "groupId", group.getId());
assert account != null;
User user = read(dataManager, User.class, "groupId", group.getId());
assert user != null;
ClientUser clientUser = read(dataManager, ClientUser.class, "userId", user.getId());
HoldLog holdLog = createHoldLog(group, account);
account.getCurrentHolds().add(holdLog);
account.setUpdated(now());
write(dataManager, HoldLog.class, holdLog);
ClientUser clientUser2 = read(dataManager, ClientUser.class, "userId", user.getId());
write(dataManager, Account.class, account);
NetworkMessage networkMessage = createNetworkMessage(message, card, group, holdLog);
write(dataManager, NetworkMessage.class, networkMessage);
read does a bit of juggling with our abstraction layer, basically this just gets a ValueExtractor for a named field and then creates an EqualsFilter with it:
private static <V extends Identifiable> V read(DataManager dataManager,
Class<V> valueClass,
String keyField,
Object keyValue)
DataMap<Identifiable> dataMap = dataManager.getDataMap(valueClass.getSimpleName(), valueClass);
FilterFactory filterFactory = dataManager.getFilterFactory();
ValueExtractorFactory extractorFactory = dataManager.getValueExtractorFactory();
Filter<Identifiable> filter = filterFactory.equals(extractorFactory.fieldExtractor(valueClass, keyField), keyValue);
Set<Map.Entry<GUID, Identifiable>> entries = dataMap.entrySet(filter);
if (entries.isEmpty())
return null;
assert entries.size() == 1 : "Expected single entry, found " + entries.size() + " for " + valueClass.getSimpleName();
return valueClass.cast(entries.iterator().next().getValue());
write is trivial:
private static <V extends Identifiable> void write(DataManager dataManager, Class<V> valueClass, V value)
DataMap<Identifiable> dataMap = dataManager.getDataMap(valueClass.getSimpleName(), valueClass);
dataMap.put(value.getId(), value);
And entrySet directly wraps the Coherence call:
public Set<Entry<GUID, V>> entrySet(Filter<V> filter)
validateFilter(filter);
return wrapEntrySet(dataMap.entrySet(((CoherenceFilterAdapter<V>) filter).getCoherenceFilter()));
This just returns a Map.EntrySet implementation that decodes the binary encoded objects.
I'm not really sure how much more code to post - what I have is really just a thin layer over Coherence, and I'm reasonably sure that the main problem isn't there since the CPU hardly moves on the client. I suspect thread starvation at some point, but I'm not sure where to look.
Thanks,
Colin