How many of these objects should I be able to insert per second?

Question

I'm inserting these objects using default (not POF) serialization with putAll(myMap). I receive about 4000 new quotes per second to put in the cache. I try coalescing them to various degrees but my other apps are still slowing down when these inserts are taking place. The applications are listening to the cache where these inserts are going using CQCs. The apps may also be doing get()s on the cache. What is the ideal size for the putAll? If I chop up myMap into batches of 100 or 200 objects then it increases the responsiveness of other apps but slows down the overall time to complete the putAll. Maybe I need a different cache topology? Currently I have 3 storage enabled cluster nodes and 3 proxy nodes. The quotes go to a distributed-scheme cache. I have tried both having the quote inserting app use Extend and becoming a TCMP cluster member. Similar issues either way.
Thanks,
Andrew
import java.io.Serializable;
public class Quote implements Serializable {
    public char type;
    public String symbol;
    public char exch;
    public float bid = 0;
    public float ask = 0;
    public int bidSize = 0;
    public int askSize = 0;
    public int hour = 0;
    public int minute = 0;
    public int second = 0;
    public float last = 0;
    public long volume = 0;
    public char fastMarket; //askSource for NBBO
    public long sequence = 0;
    public int lastTradeSize = 0;
    public String toString() {
        return "type='" + type + "'\tsymbol='" + symbol + "'\texch='" + exch + "'\tbid=" +
                bid + "\task=" + ask +
                "\tsize=" + bidSize + "x" + askSize + "\tlast=" + lastTradeSize + " @ " + last +
                "\tvolume=" + volume + "\t" +
                hour + ":" + (minute<10?"0":"") + minute + ":" + (second<10?"0":"") + second + "\tsequence=" + sequence;
    public boolean equals(Object object) {
        if (this == object) {
            return true;
        if ( !(object instanceof Quote) ) {
            return false;
        final Quote other = (Quote)object;
        if (!(symbol == null ? other.symbol == null : symbol.equals(other.symbol))) {
            return false;
        if (exch != other.exch) {
            return false;
        return true;
    public int hashCode() {
        final int PRIME = 37;
        int result = 1;
        result = PRIME * result + ((symbol == null) ? 0 : symbol.hashCode());
        result = PRIME * result + (int)exch;
        return result;
    public Object clone() throws CloneNotSupportedException {
        Quote q = new Quote();
        q.type=this.type;
        q.symbol=this.symbol;
        q.exch=this.exch;
        q.bid=this.bid;
        q.ask = this.ask;
        q.bidSize = this.bidSize;
        q.askSize = this.askSize;
        q.hour = this.hour;
        q.minute = this.minute;
        q.second = this.second;
        q.last = this.last;
        q.volume = this.volume;
        q.fastMarket = this.fastMarket;
        q.sequence = this.sequence;
        q.lastTradeSize = this.lastTradeSize;
        return q;
}

Answer 1

Well, firstly, I surprised you are using "float" objects in a financial object, but that's a different debate... :)
Second, why aren't you using pof? Much more compact from my testing; better performance too.
I've inserted similar objects (but with BigDecimal for the numeric types) and seen insert rates in the 30-40,000 / second (single machine, one node). Obviously you take a whack when you start the second node (backup's being maintained, plus that node is probably on a separate server, so you are introducing network latency.) Still, I would have thought 10-20,000/second would be easily doable.
What are the thread counts on the service's you are using?; I've found this to be quite a choke point on high-throughput caches. What stats are you getting back from JMX for the Coherence components?; what stats from the server (CPU, Memory, swap, etc)?; What spec of machines are you using? Which JVM are you using? How is the JVM configured? What's are the GC times looking like? Are you CQC queries using indexes? Are your get()'s using indexes, or just using keys? Have you instrumented your own code to get some stats from it? Are you doing excessive logging? So many variables here... Very difficult to say what the problem is with so little info./insight into your system.
Also, maybe look at using a multi-threaded "feeder" client program for your trades. That's what I do (as well as upping the thread-count on the cache service thread) and it seems to run fine (with smaller batch sizes per thread, say 50.) We "push" as well as fully "process" trades (into Positions) at a rate of about 7-10,000 / sec on a 4 server set-up (two cache storage nodes / server; two proxies / server.) Machines are dual socket, quad-core 3GHz Xeons. The clients use CQC and get()'s, similar to your set-up.
Steve