Performance degradation on multi-processor computer

Question

I saw couple similar topics but they are not the same. The issue is that .NET app is not so affected and Java app. See below. Thank you.
There is strange picture is observed here. The same test, the same data - the more CPUs computer has, the slower test results:
1) Powerful 8 CPUs server with 1 CPU assigned to process via "Set Affinity" in Task Manager: 1578 ms
2) Powerful 8 CPUs server with 2 CPUs assigned to process via "Set Affinity" in Task Manager: 1656 ms
3) Powerful 8 CPUs server with no adjusments via Task Manager - all 8 CPUs are enabled: 3469 ms
4) Those tests 1-3 were on powerful server. On my less powerful laptop with single core set in BIOS I get: 921 ms (!!!) How come?
The test has 2 active threads. First one puts 5,000,000 updates into queue. Second thread at the same time pops updates from our queue. Source is provided.
With more sophisticated test that I cannot provide I have performance difference much more noticable. That test has more synchronize calls.
The only 2 explanations that I can think of are:
1) Necessity to synchronize CPU caches on all possible CPUs. But why would computer need to synch all 8 CPUs if only 2 threads are using that data - meaning at most only 2 CPUs are active.
2) Windows Task Scheduler switches current CPU for given activ thread all the time. But why would it does so if it makes more powerful server only worse?
I hope there is some good explanation for this behavior. And the most important - there is a solution for our server to really beat my laptop performance counter.
By the way the same type of sample in .NET v.2.0 (also provided) with x64 binary runs 546 ms on my laptop and 781 and 813 ms on server with affinity to 1 and to 1-8 CPUs respectively. It makes me think this issue is JVM related. Please, comment.
== ThreadBoundaryQueue.java file ==
import java.text.NumberFormat;
import java.util.ArrayDeque;
public class ThreadBoundaryQueue
    private static final int COUNT = 5*1000*1000;
    private static final int delayBeforeStart = 10;
    private static void print(String text)
                System.out.println(Thread.currentThread().getName() + ": " + text);
    public static void main(String[] args) throws Exception
                SysInfo.dump();
                print("Preparing data for test...");
        Long events[] = new Long[COUNT];
        for(int i = 0; i < COUNT; ++i)
            events[i] = new Long(i/87);
        final Object notifier = new Object();
        final boolean finished[] = new boolean[1];
        SimpleEventQueueWithArrayDeque queue = new SimpleEventQueueWithArrayDeque(new IEventDispatcher<Long>()
            long sum_;
            int count_;
            @Override
            public void dispatchEvent(Long event)
                for(long l = event.longValue() * 50; l > 0; l /=17)
                    sum_ += l;
                if(++count_ == COUNT)
                    synchronized (notifier)
                       finished[0] = true;
                       notifier.notify();
                       print("some dummy sum is " + sum_);
        queue.startDispatching();
        print("Test starts in "+delayBeforeStart+" seconds");
        print("================================");
        Thread.sleep(delayBeforeStart*1000);
        print("Started...");
        TimeCounter sendingTime = new TimeCounter();
        TimeCounter processingTime = new TimeCounter();
        for(int i = 0; i < COUNT; ++i)
            queue.postEvent(events);
sendingTime.dump("sending of " + COUNT + " events");
synchronized (notifier)
while(!finished[0])
notifier.wait(1);
processingTime.dump("processing of " + COUNT + " events");
print("...Done");
queue.stopDispatching();
private static class SysInfo
public static void dump()
for(String p : new String[]{
"os.name",
"os.version",
"os.arch",
"sun.arch.data.model",
"java.runtime.version",
"java.vm.name",
}) print(p+"="+System.getProperty(p));
print("CPU count=" + Runtime.getRuntime().availableProcessors() + "\n");
private static class TimeCounter
private long start_;
public TimeCounter()
start_ = System.currentTimeMillis();
public void dump(String name)
print( name
+ " took, milliseconds: "
+ NumberFormat.getIntegerInstance().format(System.currentTimeMillis()-start_));
private interface IEventDispatcher<EventType>
void dispatchEvent(EventType event);
private static class SimpleEventQueueWithArrayDeque implements Runnable
private ArrayDeque<Long> buffer_;
private Thread thread_;
private IEventDispatcher<Long> dispatcher_;
public SimpleEventQueueWithArrayDeque(IEventDispatcher<Long> dispatcher)
buffer_ = new ArrayDeque<Long>(65535);
dispatcher_ = dispatcher;
public void postEvent(Long event)
synchronized (buffer_)
buffer_.addLast(event);
buffer_.notify();
@Override
public void run()
try
Long event;
while(!Thread.interrupted())
synchronized (buffer_)
event = buffer_.poll();
while(event == null)
buffer_.wait(300);
event = buffer_.poll();
dispatcher_.dispatchEvent(event);
catch(InterruptedException ex)
public synchronized void startDispatching()
if(thread_ == null)
thread_ = new Thread(this, "queue dispatcher");
thread_.setDaemon(false);
thread_.start();
public synchronized void stopDispatching()
if(thread_ != null)
thread_.interrupt();
try
thread_.join();
catch(Exception ex){}
thread_ = null;

Answer 1

No problem. We are pretty sure in what we see. We repeated a test. So, new test makes 5 rounds of test with 5,000,000 updates. And then it makes another 20 rounds that are counted. Results with our process having "Real-time priority" in both cases:
8 CPUs on 8 CPUs server: 40.5 seconds
1 CPU on the same 8 CPU server: 17,9 seconds
Just to add more confidence in results we made the same with our PRODUCTION system that has 4 heavy processes on 8 CPUs server. We set affinity for each process to have 2 of 8 CPUs. It made heavy loaded system to take around half less CPU time (each of 4 processes with a lot of "synchronized" calls). How to explain this and how to configure our system with that knowledge? And why Windows and/or JVM does not do this for us?
Code is below. Thank you
import java.text.NumberFormat;
import java.util.ArrayDeque;
public class ThreadBoundaryQueue {
    private static final int COUNT = 5*1000*1000;
    private static final int delayBeforeStart = 15;
    private static Long Events_[];
    private static void print(String text) {
        System.out.println(Thread.currentThread().getName() + ": " + text);
    public static void main(String[] args) throws Exception
        SysInfo.dump();
        final Object notifier = new Object();
        final boolean finished[] = new boolean[1];
        IEventDispatcher<Long> d = new IEventDispatcher<Long>() {
            long sum_;
            int count_;
            @Override
            public void dispatchEvent(Long event) {
                for(long l = event.longValue() * 50; l > 0; l /=17) {
                    sum_ += l;
                if(++count_ == COUNT) {
                    synchronized (notifier) {
                        finished[0] = true;
                        print("some dummy sum is " + sum_);
                        sum_ = 0;
                        count_ = 0;
                        notifier.notify();
        print("Preparing data for tests...");
        Events_ = new Long[COUNT];
        for(int i = 0; i < COUNT; ++i) {
            Events_ = new Long(i/87);
print("Test starts in "+delayBeforeStart+" seconds");
print("================================");
Thread.sleep(delayBeforeStart*1000);
print("BEGIN Warmup...");
for(int i = 0; i < 5; ++i) {
print("Test " + (i+1) + "...");
test(new SimpleEventQueueWithArrayDeque(d), notifier, finished);
print("END Warmup...");
TimeCounter allTime = new TimeCounter();
for(int i = 0; i < 20; ++i) {
print("Test " + (i+1) + "...");
test(new SimpleEventQueueWithArrayDeque(d), notifier, finished);
print("...Done");
allTime.dump("ALL TESTS after warm up");
private static void test(SimpleEventQueueWithArrayDeque queue, Object notifier, boolean[] finished) throws Exception {
synchronized (notifier) {
finished[0] = false;
queue.startDispatching();
print("Started...");
TimeCounter sendingTime = new TimeCounter();
TimeCounter processingTime = new TimeCounter();
for(int i = 0; i < COUNT; ++i) {
queue.postEvent(Events_[i]);
sendingTime.dump("sending of " + COUNT + " events");
synchronized (notifier) {
while(!finished[0]) {
notifier.wait(100);
processingTime.dump("processing of " + COUNT + " events");
queue.stopDispatching();
private static class SysInfo {
public static void dump() {
for(String p : new String[]{"os.name", "os.version", "os.arch",
"sun.arch.data.model", "java.runtime.version", "java.vm.name",})
print(p+"="+System.getProperty(p));
print("CPU count=" + Runtime.getRuntime().availableProcessors() + "\n");
private static class TimeCounter {
private long start_;
public TimeCounter() {
start_ = System.currentTimeMillis();
public void dump(String name) {
print( name
+ " took, milliseconds: "
+ NumberFormat.getIntegerInstance().format(System.currentTimeMillis()-start_));
private interface IEventDispatcher<EventType> {
void dispatchEvent(EventType event);
private static class SimpleEventQueueWithArrayDeque implements Runnable {
private ArrayDeque<Long> buffer_;
private Thread thread_;
private IEventDispatcher<Long> dispatcher_;
public SimpleEventQueueWithArrayDeque(IEventDispatcher<Long> dispatcher) {
buffer_ = new ArrayDeque<Long>(65535);
dispatcher_ = dispatcher;
public void postEvent(Long event) throws Exception {
synchronized(buffer_) {
buffer_.addLast(event);
buffer_.notifyAll();
@Override
public void run() {
try {
Long event;
while(!Thread.interrupted()) {
synchronized(buffer_) {
event = buffer_.poll();
while(event == null) {
buffer_.wait(100);
event = buffer_.poll();
dispatcher_.dispatchEvent(event);
catch(InterruptedException ex){}
public synchronized void startDispatching() {
if(thread_ == null) {
thread_ = new Thread(this, "queue dispatcher");
thread_.setDaemon(false);
thread_.start();
public synchronized void stopDispatching() {
if(thread_ != null) {
thread_.interrupt();
try {
thread_.join();
catch(Exception ex){}
thread_ = null;