Slave termination error

Question

Hi Henk,
I apologize for the lack of adequate information.
The following is the message on the logfile.
Jun 25, 2014  interval        i/o   MB/sec   bytes   read     resp     read    write     resp     resp queue  cpu%  cpu%
                             rate  1024**2     i/o    pct     time     resp     resp      max   stddev depth sys+u   sys
13:31:42.977        31     135.00    14.68  114056  47.56   24.651   51.483    0.320  371.895   42.754   3.4  94.4  10.3
13:31:42.990        32     140.00    15.45  115694  47.57   24.563   51.288    0.314  371.895   42.357   3.5  94.4  10.3
13:31:43.008        33     152.00    16.79  115809  47.89   24.733   51.248    0.360  371.895   42.472   3.9  94.4  10.3
13:31:43.571 Debug_cmds(); stdout 13:31:43.570 home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre
13:31:43.572 Debug_cmds(); stdout 13:31:43.572 home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64
13:31:43.572 Debug_cmds(); stdout 13:31:43.572 ==> where: Vdb.Jstack.main(Jstack.java:398)
13:31:43.572 Debug_cmds(); stdout 13:31:43.572 ==> where: Vdb.Jstack.main(Jstack.java:401)
13:31:43.572 Debug_cmds(); stdout 13:31:43.572 jstack: null
13:31:43.572 Debug_cmds(); stdout 13:31:43.572 jstack: null
13:31:43.573 Debug_cmds(); stdout 13:31:43.572 execute(): ps -ef
13:31:43.609 Debug_cmds(); stdout 13:31:43.609 program: ../jre/bin/java -classpath ../jre/lib/rt.jar:../jre/lib/jsse.jar:../jre/lib/jce.jar:mail.jar:framework.jar -djava.library.path=. framework.frameworkmanager
13:31:43.609 Debug_cmds(); stdout 13:31:43.609 program: ../jre/bin/java -classpath ../jre/lib/rt.jar:../jre/lib/jsse.jar:../jre/lib/jce.jar:mail.jar:framework.jar -djava.library.path=. framework.frameworkmanager
13:31:43.609 Debug_cmds(); stdout 13:31:43.609 program: ../jre/bin/java -classpath .:popup.jar:../gui.jar popup.communicator ajsgyqkj=71244
13:31:43.610 Debug_cmds(); stdout 13:31:43.610 program: java -client -xmx512m -xms64m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.vdbmain -m 8 -f /var/log/validator-logs/vdbench-config.140625.132608 -o /var/log/validator-logs/vdbench-output.140625.132608 -e 1800
13:31:43.611 Debug_cmds(); stdout 13:31:43.611 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570
13:31:43.611 Debug_cmds(); stdout 13:31:43.611 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570
13:31:43.611 Debug_cmds(); stdout 13:31:43.611 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570
13:31:43.611 Debug_cmds(); stdout 13:31:43.611 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570
13:31:43.611 Debug_cmds(); stdout 13:31:43.611 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570
13:31:43.612 Debug_cmds(); stdout 13:31:43.612 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570
13:31:43.612 Debug_cmds(); stdout 13:31:43.612 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570
13:31:43.612 Debug_cmds(); stdout 13:31:43.612 program: java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570
13:31:43.612 Debug_cmds(); stdout 13:31:43.612
13:31:43.612 Debug_cmds(); stdout 13:31:43.612 execute(): null 3268
13:31:43.615 Debug_cmds(); stdout 13:31:43.614 Jstatus for 'other' 3268 ../jre/bin/java -classpath ../jre/lib/rt.jar:../jre/lib/jsse.jar:../jre/lib/jce.jar:mail.jar:framework.jar -djava.library.path=. framework.frameworkmanager
13:31:43.615 Debug_cmds(); stdout 13:31:43.615 stderr: null: Command not found.
13:31:43.615 Debug_cmds(); stdout 13:31:43.615
13:31:43.615 Debug_cmds(); stdout 13:31:43.615 execute(): null 3523
13:31:43.617 Debug_cmds(); stdout 13:31:43.617 Jstatus for 'other' 3523 ../jre/bin/java -classpath ../jre/lib/rt.jar:../jre/lib/jsse.jar:../jre/lib/jce.jar:mail.jar:framework.jar -djava.library.path=. framework.frameworkmanager
13:31:43.617 Debug_cmds(); stdout 13:31:43.617 stderr: null: Command not found.
13:31:43.617 Debug_cmds(); stdout 13:31:43.617
13:31:43.617 Debug_cmds(); stdout 13:31:43.617 execute(): null 3527
13:31:43.619 Debug_cmds(); stdout 13:31:43.619 Jstatus for 'other' 3527 ../jre/bin/java -classpath .:popup.jar:../gui.jar popup.communicator ajsgyqkj=71244
13:31:43.619 Debug_cmds(); stdout 13:31:43.619 stderr: null: Command not found.
13:31:43.619 Debug_cmds(); stdout 13:31:43.619
13:31:43.619 Debug_cmds(); stdout 13:31:43.619 execute(): null 4278
13:31:43.621 Debug_cmds(); stdout 13:31:43.621 Jstatus for 'other' 4278 java -client -xmx512m -xms64m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.vdbmain -m 8 -f /var/log/validator-logs/vdbench-config.140625.132608 -o /var/log/validator-logs/vdbench-output.140625.132608 -e 1800
13:31:43.622 Debug_cmds(); stdout 13:31:43.621 stderr: null: Command not found.
13:31:43.622 Debug_cmds(); stdout 13:31:43.622
13:31:43.622 Debug_cmds(); stdout 13:31:43.622 execute(): null 4412
13:31:43.624 Debug_cmds(); stdout 13:31:43.624 Jstatus for 'other' 4412 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570
13:31:43.624 Debug_cmds(); stdout 13:31:43.624 stderr: null: Command not found.
13:31:43.624 Debug_cmds(); stdout 13:31:43.624
13:31:43.624 Debug_cmds(); stdout 13:31:43.624 execute(): null 4413
13:31:43.626 Debug_cmds(); stdout 13:31:43.626 Jstatus for 'other' 4413 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570
13:31:43.627 Debug_cmds(); stdout 13:31:43.626 stderr: null: Command not found.
13:31:43.627 Debug_cmds(); stdout 13:31:43.627
13:31:43.627 Debug_cmds(); stdout 13:31:43.627 execute(): null 4414
13:31:43.629 Debug_cmds(); stdout 13:31:43.629 Jstatus for 'other' 4414 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570
13:31:43.629 Debug_cmds(); stdout 13:31:43.629 stderr: null: Command not found.
13:31:43.629 Debug_cmds(); stdout 13:31:43.629
13:31:43.629 Debug_cmds(); stdout 13:31:43.629 execute(): null 4416
13:31:44.634 Debug_cmds(); stdout 13:31:44.634 Jstatus for 'other' 4416 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570
13:31:44.634 Debug_cmds(); stdout 13:31:44.634 stderr: null: Command not found.
13:31:44.634 Debug_cmds(); stdout 13:31:44.634
13:31:44.635 Debug_cmds(); stdout 13:31:44.634 execute(): null 4418
13:31:44.638 Debug_cmds(); stdout 13:31:44.638 Jstatus for 'other' 4418 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570
13:31:44.638 Debug_cmds(); stdout 13:31:44.638 stderr: null: Command not found.
13:31:44.638 Debug_cmds(); stdout 13:31:44.638
13:31:44.639 Debug_cmds(); stdout 13:31:44.638 execute(): null 4419
13:31:44.642 Debug_cmds(); stdout 13:31:44.641 Jstatus for 'other' 4419 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570
13:31:44.642 Debug_cmds(); stdout 13:31:44.642 stderr: null: Command not found.
13:31:44.642 Debug_cmds(); stdout 13:31:44.642
13:31:44.642 Debug_cmds(); stdout 13:31:44.642 execute(): null 4424
13:31:44.645 Debug_cmds(); stdout 13:31:44.645 Jstatus for 'other' 4424 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570
13:31:44.645 Debug_cmds(); stdout 13:31:44.645 stderr: null: Command not found.
13:31:44.645 Debug_cmds(); stdout 13:31:44.645
13:31:44.646 Debug_cmds(); stdout 13:31:44.645 execute(): null 4425
13:31:45.649 Debug_cmds(); stdout 13:31:45.649 Jstatus for 'other' 4425 java -client -xmx1024m -xms128m -cp /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar vdb.slavejvm slavejvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570
13:31:45.649 Debug_cmds(); stdout 13:31:45.649 stderr: null: Command not found.
13:31:45.649 Debug_cmds(); stdout 13:31:45.649 Jstack completed successfully
13:31:45.655 Memory total Java heap:   69.562 MB; Free:   52.830 MB; Used:   16.732 MB;
13:31:45.655 VdbCount statistics for Heartbeat:
13:31:45.655 Java heap:   69.563 /   52.830 /   16.732
13:31:45.655 Class                               added   cloned   serial   finals     left
13:31:45.656 VdbObject.finalize() not active!
13:31:45.656 tg: java.lang.ThreadGroup[name=main,maxpri=10]
13:31:45.656 tg.get_name: main
13:31:45.656 tg.activeCount: 35
13:31:45.661 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.661 tl[]:         Thread[Vdbmain,5,main]
13:31:45.661 tl[].isalive: true
13:31:45.661 Threadname:   Vdbmain
13:31:45.661 dumpOneStack: java.lang.Thread -2
13:31:45.661 dumpOneStack: java.lang.Thread 338
13:31:45.661 dumpOneStack: Vdb.common 376
13:31:45.661 dumpOneStack: Vdb.common 365
13:31:45.662 dumpOneStack: Vdb.Vdbmain 800
13:31:45.662 dumpOneStack: Vdb.Vdbmain 599
13:31:45.662 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.662 tl[]:         Thread[SlaveStarter localhost-10-140625-13.26.18.930,5,main]
13:31:45.662 tl[].isalive: true
13:31:45.662 Threadname:   SlaveStarter localhost-10-140625-13.26.18.930
13:31:45.662 dumpOneStack: java.lang.Object -2
13:31:45.662 dumpOneStack: java.lang.Object 503
13:31:45.662 dumpOneStack: java.lang.UNIXProcess 210
13:31:45.662 dumpOneStack: Utils.OS_cmd 319
13:31:45.662 dumpOneStack: Utils.OS_cmd 250
13:31:45.662 dumpOneStack: Vdb.SlaveStarter 220
13:31:45.663 dumpOneStack: Vdb.SlaveStarter 169
13:31:45.663 dumpOneStack: Vdb.SlaveStarter 68
13:31:45.663 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.663 tl[]:         Thread[SlaveStarter localhost-11-140625-13.26.18.930,5,main]
13:31:45.663 tl[].isalive: true
13:31:45.663 Threadname:   SlaveStarter localhost-11-140625-13.26.18.930
13:31:45.663 dumpOneStack: java.lang.Object -2
13:31:45.663 dumpOneStack: java.lang.Object 503
13:31:45.663 dumpOneStack: java.lang.UNIXProcess 210
13:31:45.663 dumpOneStack: Utils.OS_cmd 319
13:31:45.663 dumpOneStack: Utils.OS_cmd 250
13:31:45.664 dumpOneStack: Vdb.SlaveStarter 220
13:31:45.664 dumpOneStack: Vdb.SlaveStarter 169
13:31:45.664 dumpOneStack: Vdb.SlaveStarter 68
13:31:45.664 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.664 tl[]:         Thread[SlaveStarter localhost-12-140625-13.26.18.930,5,main]
13:31:45.664 tl[].isalive: true
13:31:45.664 Threadname:   SlaveStarter localhost-12-140625-13.26.18.930
13:31:45.664 dumpOneStack: java.lang.Object -2
13:31:45.664 dumpOneStack: java.lang.Object 503
13:31:45.664 dumpOneStack: java.lang.UNIXProcess 210
13:31:45.664 dumpOneStack: Utils.OS_cmd 319
13:31:45.664 dumpOneStack: Utils.OS_cmd 250
13:31:45.664 dumpOneStack: Vdb.SlaveStarter 220
13:31:45.665 dumpOneStack: Vdb.SlaveStarter 169
13:31:45.665 dumpOneStack: Vdb.SlaveStarter 68
13:31:45.665 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.665 tl[]:         Thread[SlaveStarter localhost-13-140625-13.26.18.930,5,main]
13:31:45.665 tl[].isalive: true
13:31:45.665 Threadname:   SlaveStarter localhost-13-140625-13.26.18.930
13:31:45.665 dumpOneStack: java.lang.Object -2
13:31:45.665 dumpOneStack: java.lang.Object 503
13:31:45.665 dumpOneStack: java.lang.UNIXProcess 210
13:31:45.665 dumpOneStack: Utils.OS_cmd 319
13:31:45.665 dumpOneStack: Utils.OS_cmd 250
13:31:45.666 dumpOneStack: Vdb.SlaveStarter 220
13:31:45.666 dumpOneStack: Vdb.SlaveStarter 169
13:31:45.666 dumpOneStack: Vdb.SlaveStarter 68
13:31:45.666 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.666 tl[]:         Thread[SlaveStarter localhost-14-140625-13.26.18.930,5,main]
13:31:45.666 tl[].isalive: true
13:31:45.666 Threadname:   SlaveStarter localhost-14-140625-13.26.18.930
13:31:45.666 dumpOneStack: java.lang.Object -2
13:31:45.666 dumpOneStack: java.lang.Object 503
13:31:45.666 dumpOneStack: java.lang.UNIXProcess 210
13:31:45.666 dumpOneStack: Utils.OS_cmd 319
13:31:45.666 dumpOneStack: Utils.OS_cmd 250
13:31:45.667 dumpOneStack: Vdb.SlaveStarter 220
13:31:45.667 dumpOneStack: Vdb.SlaveStarter 169
13:31:45.667 dumpOneStack: Vdb.SlaveStarter 68
13:31:45.667 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.667 tl[]:         Thread[SlaveStarter localhost-15-140625-13.26.18.930,5,main]
13:31:45.667 tl[].isalive: true
13:31:45.667 Threadname:   SlaveStarter localhost-15-140625-13.26.18.930
13:31:45.667 dumpOneStack: java.lang.Object -2
13:31:45.667 dumpOneStack: java.lang.Object 503
13:31:45.667 dumpOneStack: java.lang.UNIXProcess 210
13:31:45.667 dumpOneStack: Utils.OS_cmd 319
13:31:45.667 dumpOneStack: Utils.OS_cmd 250
13:31:45.668 dumpOneStack: Vdb.SlaveStarter 220
13:31:45.668 dumpOneStack: Vdb.SlaveStarter 169
13:31:45.668 dumpOneStack: Vdb.SlaveStarter 68
13:31:45.668 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.668 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570,5,main]
13:31:45.668 tl[].isalive: true
13:31:45.668 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570
13:31:45.668 dumpOneStack: java.io.FileInputStream -2
13:31:45.668 dumpOneStack: java.io.FileInputStream 242
13:31:45.668 dumpOneStack: java.io.BufferedInputStream 273
13:31:45.668 dumpOneStack: java.io.BufferedInputStream 334
13:31:45.669 dumpOneStack: sun.nio.cs.StreamDecoder 283
13:31:45.669 dumpOneStack: sun.nio.cs.StreamDecoder 325
13:31:45.669 dumpOneStack: sun.nio.cs.StreamDecoder 177
13:31:45.669 dumpOneStack: java.io.InputStreamReader 184
13:31:45.669 dumpOneStack: java.io.BufferedReader 154
13:31:45.669 dumpOneStack: java.io.BufferedReader 317
13:31:45.669 dumpOneStack: java.io.BufferedReader 382
13:31:45.669 dumpOneStack: Utils.Get_cmd_stream 679
13:31:45.669 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.669 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570,5,main]
13:31:45.669 tl[].isalive: true
13:31:45.669 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570
13:31:45.669 dumpOneStack: java.io.FileInputStream -2
13:31:45.670 dumpOneStack: java.io.FileInputStream 242
13:31:45.670 dumpOneStack: java.io.BufferedInputStream 273
13:31:45.670 dumpOneStack: java.io.BufferedInputStream 334
13:31:45.670 dumpOneStack: sun.nio.cs.StreamDecoder 283
13:31:45.670 dumpOneStack: sun.nio.cs.StreamDecoder 325
13:31:45.670 dumpOneStack: sun.nio.cs.StreamDecoder 177
13:31:45.670 dumpOneStack: java.io.InputStreamReader 184
13:31:45.670 dumpOneStack: java.io.BufferedReader 154
13:31:45.670 dumpOneStack: java.io.BufferedReader 317
13:31:45.670 dumpOneStack: java.io.BufferedReader 382
13:31:45.670 dumpOneStack: Utils.Get_cmd_stream 679
13:31:45.670 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.670 tl[]:         Thread[SlaveStarter localhost-16-140625-13.26.18.930,5,main]
13:31:45.670 tl[].isalive: true
13:31:45.671 Threadname:   SlaveStarter localhost-16-140625-13.26.18.930
13:31:45.671 dumpOneStack: java.lang.Object -2
13:31:45.671 dumpOneStack: java.lang.Object 503
13:31:45.671 dumpOneStack: java.lang.UNIXProcess 210
13:31:45.671 dumpOneStack: Utils.OS_cmd 319
13:31:45.671 dumpOneStack: Utils.OS_cmd 250
13:31:45.671 dumpOneStack: Vdb.SlaveStarter 220
13:31:45.671 dumpOneStack: Vdb.SlaveStarter 169
13:31:45.671 dumpOneStack: Vdb.SlaveStarter 68
13:31:45.671 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.671 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570,5,main]
13:31:45.671 tl[].isalive: true
13:31:45.672 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570
13:31:45.672 dumpOneStack: java.io.FileInputStream -2
13:31:45.672 dumpOneStack: java.io.FileInputStream 242
13:31:45.672 dumpOneStack: java.io.BufferedInputStream 273
13:31:45.672 dumpOneStack: java.io.BufferedInputStream 334
13:31:45.672 dumpOneStack: sun.nio.cs.StreamDecoder 283
13:31:45.672 dumpOneStack: sun.nio.cs.StreamDecoder 325
13:31:45.672 dumpOneStack: sun.nio.cs.StreamDecoder 177
13:31:45.672 dumpOneStack: java.io.InputStreamReader 184
13:31:45.672 dumpOneStack: java.io.BufferedReader 154
13:31:45.672 dumpOneStack: java.io.BufferedReader 317
13:31:45.672 dumpOneStack: java.io.BufferedReader 382
13:31:45.673 dumpOneStack: Utils.Get_cmd_stream 679
13:31:45.673 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.673 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570,5,main]
13:31:45.673 tl[].isalive: true
13:31:45.673 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570
13:31:45.673 dumpOneStack: java.io.FileInputStream -2
13:31:45.673 dumpOneStack: java.io.FileInputStream 242
13:31:45.673 dumpOneStack: java.io.BufferedInputStream 273
13:31:45.673 dumpOneStack: java.io.BufferedInputStream 334
13:31:45.673 dumpOneStack: sun.nio.cs.StreamDecoder 283
13:31:45.673 dumpOneStack: sun.nio.cs.StreamDecoder 325
13:31:45.673 dumpOneStack: sun.nio.cs.StreamDecoder 177
13:31:45.673 dumpOneStack: java.io.InputStreamReader 184
13:31:45.673 dumpOneStack: java.io.BufferedReader 154
13:31:45.674 dumpOneStack: java.io.BufferedReader 317
13:31:45.674 dumpOneStack: java.io.BufferedReader 382
13:31:45.674 dumpOneStack: Utils.Get_cmd_stream 679
13:31:45.674 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.674 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570,5,main]
13:31:45.674 tl[].isalive: true
13:31:45.674 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570
13:31:45.674 dumpOneStack: java.io.FileInputStream -2
13:31:45.674 dumpOneStack: java.io.FileInputStream 242
13:31:45.674 dumpOneStack: java.io.BufferedInputStream 273
13:31:45.674 dumpOneStack: java.io.BufferedInputStream 334
13:31:45.675 dumpOneStack: sun.nio.cs.StreamDecoder 283
13:31:45.675 dumpOneStack: sun.nio.cs.StreamDecoder 325
13:31:45.675 dumpOneStack: sun.nio.cs.StreamDecoder 177
13:31:45.675 dumpOneStack: java.io.InputStreamReader 184
13:31:45.675 dumpOneStack: java.io.BufferedReader 154
13:31:45.675 dumpOneStack: java.io.BufferedReader 317
13:31:45.675 dumpOneStack: java.io.BufferedReader 382
13:31:45.675 dumpOneStack: Utils.Get_cmd_stream 679
13:31:45.675 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.675 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570,5,main]
13:31:45.675 tl[].isalive: true
13:31:45.675 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570
13:31:45.675 dumpOneStack: java.io.FileInputStream -2
13:31:45.675 dumpOneStack: java.io.FileInputStream 242
13:31:45.675 dumpOneStack: java.io.BufferedInputStream 273
13:31:45.675 dumpOneStack: java.io.BufferedInputStream 334
13:31:45.675 dumpOneStack: sun.nio.cs.StreamDecoder 283
13:31:45.676 dumpOneStack: sun.nio.cs.StreamDecoder 325
13:31:45.676 dumpOneStack: sun.nio.cs.StreamDecoder 177
13:31:45.676 dumpOneStack: java.io.InputStreamReader 184
13:31:45.676 dumpOneStack: java.io.BufferedReader 154
13:31:45.676 dumpOneStack: java.io.BufferedReader 317
13:31:45.676 dumpOneStack: java.io.BufferedReader 382
13:31:45.676 dumpOneStack: Utils.Get_cmd_stream 679
13:31:45.676 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.676 tl[]:         Thread[SlaveStarter localhost-17-140625-13.26.18.930,5,main]
13:31:45.676 tl[].isalive: true
13:31:45.676 Threadname:   SlaveStarter localhost-17-140625-13.26.18.930
13:31:45.676 dumpOneStack: java.lang.Object -2
13:31:45.676 dumpOneStack: java.lang.Object 503
13:31:45.676 dumpOneStack: java.lang.UNIXProcess 210
13:31:45.676 dumpOneStack: Utils.OS_cmd 319
13:31:45.676 dumpOneStack: Utils.OS_cmd 250
13:31:45.676 dumpOneStack: Vdb.SlaveStarter 220
13:31:45.676 dumpOneStack: Vdb.SlaveStarter 169
13:31:45.677 dumpOneStack: Vdb.SlaveStarter 68
13:31:45.677 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.677 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570,5,main]
13:31:45.677 tl[].isalive: true
13:31:45.677 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570
13:31:45.677 dumpOneStack: java.io.FileInputStream -2
13:31:45.677 dumpOneStack: java.io.FileInputStream 242
13:31:45.677 dumpOneStack: java.io.BufferedInputStream 273
13:31:45.677 dumpOneStack: java.io.BufferedInputStream 334
13:31:45.677 dumpOneStack: sun.nio.cs.StreamDecoder 283
13:31:45.677 dumpOneStack: sun.nio.cs.StreamDecoder 325
13:31:45.677 dumpOneStack: sun.nio.cs.StreamDecoder 177
13:31:45.677 dumpOneStack: java.io.InputStreamReader 184
13:31:45.677 dumpOneStack: java.io.BufferedReader 154
13:31:45.677 dumpOneStack: java.io.BufferedReader 317
13:31:45.678 dumpOneStack: java.io.BufferedReader 382
13:31:45.678 dumpOneStack: Utils.Get_cmd_stream 679
13:31:45.678 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.678 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570,5,main]
13:31:45.678 tl[].isalive: true
13:31:45.678 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570
13:31:45.678 dumpOneStack: java.io.FileInputStream -2
13:31:45.678 dumpOneStack: java.io.FileInputStream 242
13:31:45.678 dumpOneStack: java.io.BufferedInputStream 273
13:31:45.678 dumpOneStack: java.io.BufferedInputStream 334
13:31:45.678 dumpOneStack: sun.nio.cs.StreamDecoder 283
13:31:45.678 dumpOneStack: sun.nio.cs.StreamDecoder 325
13:31:45.678 dumpOneStack: sun.nio.cs.StreamDecoder 177
13:31:45.678 dumpOneStack: java.io.InputStreamReader 184
13:31:45.678 dumpOneStack: java.io.BufferedReader 154
13:31:45.678 dumpOneStack: java.io.BufferedReader 317
13:31:45.679 dumpOneStack: java.io.BufferedReader 382
13:31:45.679 dumpOneStack: Utils.Get_cmd_stream 679
13:31:45.679 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.679 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570,5,main]
13:31:45.679 tl[].isalive: true
13:31:45.679 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570
13:31:45.679 dumpOneStack: java.io.FileInputStream -2
13:31:45.679 dumpOneStack: java.io.FileInputStream 242
13:31:45.679 dumpOneStack: java.io.BufferedInputStream 273
13:31:45.679 dumpOneStack: java.io.BufferedInputStream 334
13:31:45.679 dumpOneStack: sun.nio.cs.StreamDecoder 283
13:31:45.679 dumpOneStack: sun.nio.cs.StreamDecoder 325
13:31:45.679 dumpOneStack: sun.nio.cs.StreamDecoder 177
13:31:45.679 dumpOneStack: java.io.InputStreamReader 184
13:31:45.679 dumpOneStack: java.io.BufferedReader 154
13:31:45.679 dumpOneStack: java.io.BufferedReader 317
13:31:45.680 dumpOneStack: java.io.BufferedReader 382
13:31:45.680 dumpOneStack: Utils.Get_cmd_stream 679
13:31:45.680 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.680 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570,5,main]
13:31:45.680 tl[].isalive: true
13:31:45.680 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570
13:31:45.680 dumpOneStack: java.io.FileInputStream -2
13:31:45.680 dumpOneStack: java.io.FileInputStream 242
13:31:45.680 dumpOneStack: java.io.BufferedInputStream 273
13:31:45.680 dumpOneStack: java.io.BufferedInputStream 334
13:31:45.680 dumpOneStack: sun.nio.cs.StreamDecoder 283
13:31:45.680 dumpOneStack: sun.nio.cs.StreamDecoder 325
13:31:45.680 dumpOneStack: sun.nio.cs.StreamDecoder 177
13:31:45.680 dumpOneStack: java.io.InputStreamReader 184
13:31:45.680 dumpOneStack: java.io.BufferedReader 154
13:31:45.680 dumpOneStack: java.io.BufferedReader 317
13:31:45.680 dumpOneStack: java.io.BufferedReader 382
13:31:45.681 dumpOneStack: Utils.Get_cmd_stream 679
13:31:45.681 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.681 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570,5,main]
13:31:45.681 tl[].isalive: true
13:31:45.681 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570
13:31:45.681 dumpOneStack: java.io.FileInputStream -2
13:31:45.681 dumpOneStack: java.io.FileInputStream 242
13:31:45.681 dumpOneStack: java.io.BufferedInputStream 273
13:31:45.681 dumpOneStack: java.io.BufferedInputStream 334
13:31:45.681 dumpOneStack: sun.nio.cs.StreamDecoder 283
13:31:45.681 dumpOneStack: sun.nio.cs.StreamDecoder 325
13:31:45.681 dumpOneStack: sun.nio.cs.StreamDecoder 177
13:31:45.681 dumpOneStack: java.io.InputStreamReader 184
13:31:45.681 dumpOneStack: java.io.BufferedReader 154
13:31:45.681 dumpOneStack: java.io.BufferedReader 317
13:31:45.681 dumpOneStack: java.io.BufferedReader 382
13:31:45.681 dumpOneStack: Utils.Get_cmd_stream 679
13:31:45.682 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.682 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570,5,main]
13:31:45.682 tl[].isalive: true
13:31:45.682 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570
13:31:45.682 dumpOneStack: java.io.FileInputStream -2
13:31:45.682 dumpOneStack: java.io.FileInputStream 242
13:31:45.682 dumpOneStack: java.io.BufferedInputStream 273
13:31:45.682 dumpOneStack: java.io.BufferedInputStream 334
13:31:45.682 dumpOneStack: sun.nio.cs.StreamDecoder 283
13:31:45.682 dumpOneStack: sun.nio.cs.StreamDecoder 325
13:31:45.682 dumpOneStack: sun.nio.cs.StreamDecoder 177
13:31:45.682 dumpOneStack: java.io.InputStreamReader 184
13:31:45.682 dumpOneStack: java.io.BufferedReader 154
13:31:45.682 dumpOneStack: java.io.BufferedReader 317
13:31:45.682 dumpOneStack: java.io.BufferedReader 382
13:31:45.682 dumpOneStack: Utils.Get_cmd_stream 679
13:31:45.683 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.683 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570,5,main]
13:31:45.683 tl[].isalive: true
13:31:45.683 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570
13:31:45.683 dumpOneStack: java.io.FileInputStream -2
13:31:45.683 dumpOneStack: java.io.FileInputStream 242
13:31:45.683 dumpOneStack: java.io.BufferedInputStream 273
13:31:45.683 dumpOneStack: java.io.BufferedInputStream 334
13:31:45.683 dumpOneStack: sun.nio.cs.StreamDecoder 283
13:31:45.683 dumpOneStack: sun.nio.cs.StreamDecoder 325
13:31:45.683 dumpOneStack: sun.nio.cs.StreamDecoder 177
13:31:45.683 dumpOneStack: java.io.InputStreamReader 184
13:31:45.683 dumpOneStack: java.io.BufferedReader 154
13:31:45.683 dumpOneStack: java.io.BufferedReader 317
13:31:45.683 dumpOneStack: java.io.BufferedReader 382
13:31:45.683 dumpOneStack: Utils.Get_cmd_stream 679
13:31:45.684 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.684 tl[]:         Thread[Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570,5,main]
13:31:45.684 tl[].isalive: true
13:31:45.684 Threadname:   Get_cmd_stream stderr /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570
13:31:45.684 dumpOneStack: java.io.FileInputStream -2
13:31:45.684 dumpOneStack: java.io.FileInputStream 242
13:31:45.684 dumpOneStack: java.io.BufferedInputStream 273
13:31:45.684 dumpOneStack: java.io.BufferedInputStream 334
13:31:45.684 dumpOneStack: sun.nio.cs.StreamDecoder 283
13:31:45.684 dumpOneStack: sun.nio.cs.StreamDecoder 325
13:31:45.684 dumpOneStack: sun.nio.cs.StreamDecoder 177
13:31:45.684 dumpOneStack: java.io.InputStreamReader 184
13:31:45.684 dumpOneStack: java.io.BufferedReader 154
13:31:45.684 dumpOneStack: java.io.BufferedReader 317
13:31:45.684 dumpOneStack: java.io.BufferedReader 382
13:31:45.684 dumpOneStack: Utils.Get_cmd_stream 679
13:31:45.685 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.685 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570,5,main]
13:31:45.685 tl[].isalive: true
13:31:45.685 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570
13:31:45.685 dumpOneStack: java.io.FileInputStream -2
13:31:45.685 dumpOneStack: java.io.FileInputStream 242
13:31:45.685 dumpOneStack: java.io.BufferedInputStream 273
13:31:45.685 dumpOneStack: java.io.BufferedInputStream 334
13:31:45.685 dumpOneStack: sun.nio.cs.StreamDecoder 283
13:31:45.685 dumpOneStack: sun.nio.cs.StreamDecoder 325
13:31:45.685 dumpOneStack: sun.nio.cs.StreamDecoder 177
13:31:45.685 dumpOneStack: java.io.InputStreamReader 184
13:31:45.685 dumpOneStack: java.io.BufferedReader 154
13:31:45.685 dumpOneStack: java.io.BufferedReader 317
13:31:45.685 dumpOneStack: java.io.BufferedReader 382
13:31:45.686 dumpOneStack: Utils.Get_cmd_stream 679
13:31:45.686 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.686 tl[]:         Thread[Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570,5,main]
13:31:45.686 tl[].isalive: true
13:31:45.686 Threadname:   Get_cmd_stream stdout /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570
13:31:45.686 dumpOneStack: java.io.FileInputStream -2
13:31:45.686 dumpOneStack: java.io.FileInputStream 242
13:31:45.686 dumpOneStack: java.io.BufferedInputStream 273
13:31:45.686 dumpOneStack: java.io.BufferedInputStream 334
13:31:45.686 dumpOneStack: sun.nio.cs.StreamDecoder 283
13:31:45.686 dumpOneStack: sun.nio.cs.StreamDecoder 325
13:31:45.686 dumpOneStack: sun.nio.cs.StreamDecoder 177
13:31:45.686 dumpOneStack: java.io.InputStreamReader 184
13:31:45.686 dumpOneStack: java.io.BufferedReader 154
13:31:45.686 dumpOneStack: java.io.BufferedReader 317
13:31:45.686 dumpOneStack: java.io.BufferedReader 382
13:31:45.687 dumpOneStack: Utils.Get_cmd_stream 679
13:31:45.687 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.687 tl[]:         Thread[SlaveOnMaster,10,main]
13:31:45.687 tl[].isalive: true
13:31:45.687 Threadname:   SlaveOnMaster
13:31:45.687 dumpOneStack: java.net.SocketInputStream -2
13:31:45.687 dumpOneStack: java.net.SocketInputStream 150
13:31:45.687 dumpOneStack: java.net.SocketInputStream 121
13:31:45.687 dumpOneStack: java.net.SocketInputStream 203
13:31:45.687 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266
13:31:45.687 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559
13:31:45.687 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569
13:31:45.687 dumpOneStack: java.io.ObjectInputStream 1315
13:31:45.687 dumpOneStack: java.io.ObjectInputStream 369
13:31:45.687 dumpOneStack: Vdb.SlaveSocket 148
13:31:45.688 dumpOneStack: Vdb.SlaveOnMaster 118
13:31:45.688 dumpOneStack: Vdb.SlaveOnMaster 60
13:31:45.688 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.688 tl[]:         Thread[SlaveOnMaster,10,main]
13:31:45.688 tl[].isalive: true
13:31:45.688 Threadname:   SlaveOnMaster
13:31:45.688 dumpOneStack: java.net.SocketInputStream -2
13:31:45.688 dumpOneStack: java.net.SocketInputStream 150
13:31:45.688 dumpOneStack: java.net.SocketInputStream 121
13:31:45.688 dumpOneStack: java.net.SocketInputStream 203
13:31:45.688 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266
13:31:45.688 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559
13:31:45.688 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569
13:31:45.688 dumpOneStack: java.io.ObjectInputStream 1315
13:31:45.688 dumpOneStack: java.io.ObjectInputStream 369
13:31:45.688 dumpOneStack: Vdb.SlaveSocket 148
13:31:45.689 dumpOneStack: Vdb.SlaveOnMaster 118
13:31:45.689 dumpOneStack: Vdb.SlaveOnMaster 60
13:31:45.689 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.689 tl[]:         Thread[SlaveOnMaster,10,main]
13:31:45.689 tl[].isalive: true
13:31:45.689 Threadname:   SlaveOnMaster
13:31:45.689 dumpOneStack: java.net.SocketInputStream -2
13:31:45.689 dumpOneStack: java.net.SocketInputStream 150
13:31:45.689 dumpOneStack: java.net.SocketInputStream 121
13:31:45.689 dumpOneStack: java.net.SocketInputStream 203
13:31:45.689 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266
13:31:45.689 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559
13:31:45.689 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569
13:31:45.689 dumpOneStack: java.io.ObjectInputStream 1315
13:31:45.689 dumpOneStack: java.io.ObjectInputStream 369
13:31:45.689 dumpOneStack: Vdb.SlaveSocket 148
13:31:45.690 dumpOneStack: Vdb.SlaveOnMaster 118
13:31:45.690 dumpOneStack: Vdb.SlaveOnMaster 60
13:31:45.690 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.690 tl[]:         Thread[SlaveOnMaster,10,main]
13:31:45.690 tl[].isalive: true
13:31:45.690 Threadname:   SlaveOnMaster
13:31:45.690 dumpOneStack: java.net.SocketInputStream -2
13:31:45.690 dumpOneStack: java.net.SocketInputStream 150
13:31:45.690 dumpOneStack: java.net.SocketInputStream 121
13:31:45.690 dumpOneStack: java.net.SocketInputStream 203
13:31:45.690 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266
13:31:45.690 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559
13:31:45.690 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569
13:31:45.690 dumpOneStack: java.io.ObjectInputStream 1315
13:31:45.690 dumpOneStack: java.io.ObjectInputStream 369
13:31:45.690 dumpOneStack: Vdb.SlaveSocket 148
13:31:45.691 dumpOneStack: Vdb.SlaveOnMaster 118
13:31:45.691 dumpOneStack: Vdb.SlaveOnMaster 60
13:31:45.691 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.691 tl[]:         Thread[SlaveOnMaster,10,main]
13:31:45.691 tl[].isalive: true
13:31:45.691 Threadname:   SlaveOnMaster
13:31:45.691 dumpOneStack: java.net.SocketInputStream -2
13:31:45.691 dumpOneStack: java.net.SocketInputStream 150
13:31:45.691 dumpOneStack: java.net.SocketInputStream 121
13:31:45.691 dumpOneStack: java.net.SocketInputStream 203
13:31:45.691 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266
13:31:45.691 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559
13:31:45.691 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569
13:31:45.691 dumpOneStack: java.io.ObjectInputStream 1315
13:31:45.691 dumpOneStack: java.io.ObjectInputStream 369
13:31:45.692 dumpOneStack: Vdb.SlaveSocket 148
13:31:45.692 dumpOneStack: Vdb.SlaveOnMaster 118
13:31:45.692 dumpOneStack: Vdb.SlaveOnMaster 60
13:31:45.692 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.692 tl[]:         Thread[SlaveOnMaster,10,main]
13:31:45.692 tl[].isalive: true
13:31:45.692 Threadname:   SlaveOnMaster
13:31:45.692 dumpOneStack: java.net.SocketInputStream -2
13:31:45.692 dumpOneStack: java.net.SocketInputStream 150
13:31:45.692 dumpOneStack: java.net.SocketInputStream 121
13:31:45.692 dumpOneStack: java.net.SocketInputStream 203
13:31:45.692 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266
13:31:45.692 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559
13:31:45.692 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569
13:31:45.692 dumpOneStack: java.io.ObjectInputStream 1315
13:31:45.693 dumpOneStack: java.io.ObjectInputStream 369
13:31:45.693 dumpOneStack: Vdb.SlaveSocket 148
13:31:45.693 dumpOneStack: Vdb.SlaveOnMaster 118
13:31:45.693 dumpOneStack: Vdb.SlaveOnMaster 60
13:31:45.693 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.693 tl[]:         Thread[SlaveOnMaster,10,main]
13:31:45.693 tl[].isalive: true
13:31:45.693 Threadname:   SlaveOnMaster
13:31:45.693 dumpOneStack: java.net.SocketInputStream -2
13:31:45.693 dumpOneStack: java.net.SocketInputStream 150
13:31:45.693 dumpOneStack: java.net.SocketInputStream 121
13:31:45.693 dumpOneStack: java.net.SocketInputStream 203
13:31:45.693 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266
13:31:45.693 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559
13:31:45.693 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569
13:31:45.693 dumpOneStack: java.io.ObjectInputStream 1315
13:31:45.694 dumpOneStack: java.io.ObjectInputStream 369
13:31:45.694 dumpOneStack: Vdb.SlaveSocket 148
13:31:45.694 dumpOneStack: Vdb.SlaveOnMaster 118
13:31:45.694 dumpOneStack: Vdb.SlaveOnMaster 60
13:31:45.694 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.694 tl[]:         Thread[SlaveOnMaster,10,main]
13:31:45.694 tl[].isalive: true
13:31:45.694 Threadname:   SlaveOnMaster
13:31:45.694 dumpOneStack: java.net.SocketInputStream -2
13:31:45.694 dumpOneStack: java.net.SocketInputStream 150
13:31:45.694 dumpOneStack: java.net.SocketInputStream 121
13:31:45.694 dumpOneStack: java.net.SocketInputStream 203
13:31:45.694 dumpOneStack: java.io.ObjectInputStream$PeekInputStream 2266
13:31:45.694 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2559
13:31:45.694 dumpOneStack: java.io.ObjectInputStream$BlockDataInputStream 2569
13:31:45.695 dumpOneStack: java.io.ObjectInputStream 1315
13:31:45.695 dumpOneStack: java.io.ObjectInputStream 369
13:31:45.695 dumpOneStack: Vdb.SlaveSocket 148
13:31:45.695 dumpOneStack: Vdb.SlaveOnMaster 118
13:31:45.695 dumpOneStack: Vdb.SlaveOnMaster 60
13:31:45.695 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.695 tl[]:         Thread[Slave HeartBeat,5,main]
13:31:45.695 tl[].isalive: true
13:31:45.695 Threadname:   Slave HeartBeat
13:31:45.695 dumpOneStack: java.lang.Thread 1567
13:31:45.695 dumpOneStack: Vdb.common 1523
13:31:45.695 dumpOneStack: Vdb.common 1517
13:31:45.695 dumpOneStack: Vdb.HeartBeat 97
13:31:45.695 tl            [Ljava.lang.Thread;@8a7d8b0
13:31:45.695 tl[]:         Thread[Reporter,10,main]
13:31:45.695 tl[].isalive: true
13:31:45.695 Threadname:   Reporter
13:31:45.696 dumpOneStack: java.lang.Thread -2
13:31:45.696 dumpOneStack: java.lang.Thread 338
13:31:45.696 dumpOneStack: Vdb.common 376
13:31:45.696 dumpOneStack: Vdb.common 365
13:31:45.696 dumpOneStack: Vdb.common 668
13:31:45.696 dumpOneStack: Vdb.Reporter 122
13:31:45.696 killCommand: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570
13:31:45.696 killCommand: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570
13:31:45.696 killCommand: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570
13:31:45.696 killCommand: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570
13:31:45.696
13:31:45.696 Heartbeat monitor: One or more slaves did not respond
13:31:45.697
java.lang.RuntimeException: Heartbeat monitor: One or more slaves did not respond
  at Vdb.common.failure(common.java:307)
  at Vdb.HeartBeat.run(HeartBeat.java:98)
The vdbench parameter file used is as follows:
sd=sd1,lun=/dev/sdl,openflags=o_direct
sd=sd2,lun=/dev/sdj,openflags=o_direct
sd=sd3,lun=/dev/sdk,openflags=o_direct
sd=sd4,lun=/dev/sdh,openflags=o_direct
sd=sd5,lun=/dev/sdi,openflags=o_direct
sd=sd6,lun=/dev/sdf,openflags=o_direct
sd=sd7,lun=/dev/sdg,openflags=o_direct
sd=sd8,lun=/dev/sdd,openflags=o_direct
sd=sd9,lun=/dev/sde,openflags=o_direct
sd=sd10,lun=/dev/sdb,openflags=o_direct
sd=sd11,lun=/dev/sdc,openflags=o_direct
wd=wd1,sd=sd1,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random
wd=wd2,sd=sd2,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random
wd=wd3,sd=sd3,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random
wd=wd4,sd=sd4,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random
wd=wd5,sd=sd5,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random
wd=wd6,sd=sd6,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random
wd=wd7,sd=sd7,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random
wd=wd8,sd=sd8,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random
wd=wd9,sd=sd9,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random
wd=wd10,sd=sd10,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random
wd=wd11,sd=sd11,xfersize=(16k,25,32k,25,128k,25,256k,25),seekpct=random
rd=rd1,wd=wd*,forthreads=16,forrdpct=50,interval=5,iorate=max,pause=0,elapsed=1800
This test would pass if there are no other unconfigured drives in the server other than the ones contained in the sd* defined on this parameter file. The test would fail if there are other unconfigured drives. Could you please share your thoughts?
Thanks,
Priyanka

Answer 1

<pre>
13:26:18.958 Vdbench distribution: vdbench503rc11
13:26:18.959
13:26:18.973 *
13:26:18.973 * This beta version 'vdbench503rc11' was built on Fri April 08  9:48:23 MDT 2011.
13:26:18.973 * which is more than 90 days ago.
13:26:18.973 * It is recommended that you look for a newer beta or possibly
13:26:18.974 * a newer GA version.
13:26:18.974 * Of course, it is preferred you download a newer beta version
13:26:18.974 * if available to help test the latest code.
13:26:18.974 *
13:26:18.976 input argument scanned: '-m8'
13:26:18.995 input argument scanned: '-f/var/log/validator-logs/vdbench-config.140625.132608'
13:26:18.996 input argument scanned: '-o/var/log/validator-logs/vdbench-output.140625.132608'
13:26:18.996 input argument scanned: '-e1800'
13:26:19.000 java.vendor                   Oracle Corporation
13:26:19.000 java.home                     /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre
13:26:19.000 java.vm.specification.version 1.7
13:26:19.000 java.vm.version               23.2-b09
13:26:19.000 java.vm.vendor                Oracle Corporation
13:26:19.001 java.specification.version    1.7
13:26:19.001 java.class.version            51.0
13:26:19.001 user.name                     root
13:26:19.001 user.dir                      /
13:26:19.001 java.class.path               /opt/vdbench/:/opt/vdbench/classes:/opt/vdbench/vdbench.jar:/opt/vdbench/jaramiko.jar
13:26:19.001 os.name                       Linux
13:26:19.001 os.arch                       amd64
13:26:19.001 os.version                    2.6.32-358.el6.x86_64
13:26:19.001 sun.arch.data.model           64
13:26:19.203 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.204 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.204 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.205 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.205 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.206 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.206 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.207 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.207 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.208 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.208 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.225 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.225 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.226 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.226 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.227 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.227 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.228 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.228 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.229 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.229 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:26:19.230 will_we_use_waiter: rd1 false; rc=6; req:  999988; iorate= 999988
13:28:52.947 Starting slave: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-11-140625-13.26.18.930 -l localhost-1 -p 5570  
13:28:52.947 Starting slave: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-12-140625-13.26.18.930 -l localhost-2 -p 5570  
13:28:52.947 Starting slave: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-10-140625-13.26.18.930 -l localhost-0 -p 5570  
13:28:52.947 Starting slave: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-13-140625-13.26.18.930 -l localhost-3 -p 5570  
13:28:52.948 Starting slave: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-14-140625-13.26.18.930 -l localhost-4 -p 5570  
13:28:52.948 Starting slave: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-15-140625-13.26.18.930 -l localhost-5 -p 5570  
13:28:52.949 Starting slave: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-16-140625-13.26.18.930 -l localhost-6 -p 5570  
13:28:52.950 Starting slave: /opt/vdbench/vdbench SlaveJvm -m localhost -n localhost-17-140625-13.26.18.930 -l localhost-7 -p 5570  
13:28:53.524 Slave localhost-0 connected
13:28:53.524 Slave localhost-5 connected
13:28:53.524 Slave localhost-7 connected
13:28:53.524 Slave localhost-6 connected
13:28:53.524 Slave localhost-2 connected
13:28:53.524 Slave localhost-3 connected
13:28:53.524 Slave localhost-4 connected
13:28:53.525 Slave localhost-1 connected
13:28:53.548 All slaves are now connected
13:28:54.671 sd=sd1,lun=/dev/sdl lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)
13:28:54.671 sd=sd2,lun=/dev/sdj lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)
13:28:54.671 sd=sd3,lun=/dev/sdk lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)
13:28:54.671 sd=sd4,lun=/dev/sdh lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)
13:28:54.672 sd=sd5,lun=/dev/sdi lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)
13:28:54.672 sd=sd6,lun=/dev/sdf lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)
13:28:54.672 sd=sd7,lun=/dev/sdg lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)
13:28:54.672 sd=sd8,lun=/dev/sdd lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)
13:28:54.672 sd=sd9,lun=/dev/sde lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)
13:28:54.673 sd=sd10,lun=/dev/sdb lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)
13:28:54.673 sd=sd11,lun=/dev/sdc lun size: 53687091200 bytes; 50.0000 GB (1024**3); 53.6871 GB (1000**3)
13:28:54.678 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.678 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.679 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.680 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.681 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.682 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.683 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.684 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.684 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.685 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.686 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.687 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.688 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.689 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.690 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.691 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.692 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.692 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.693 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.693 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.694 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.695 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.696 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.698 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.699 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.700 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.701 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.702 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.702 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.703 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.703 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.704 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.705 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.706 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.707 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.708 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.709 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.710 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.711 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.712 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.712 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.713 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.713 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.714 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.715 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.715 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.716 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.717 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.718 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.719 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.720 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.721 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.721 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.722 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.723 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.724 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.724 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.725 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.725 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.726 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.726 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.727 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.727 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.728 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.728 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.729 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.729 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.730 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.730 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.731 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.731 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.732 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.732 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.733 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.733 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.734 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.735 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.735 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.736 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.736 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.737 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.737 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.738 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.738 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.739 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.740 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.740 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.741 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
Link to Run Definitions:         <A HREF="#_215181933">rd1 For loops: threads=16.0 rdpct=50.0</A>
13:28:54.772 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.772 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.773 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.773 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.774 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.774 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.775 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.775 ++Interarrival time: wd=wd1 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.776 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.776 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.777 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.777 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.778 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.778 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.779 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.779 ++Interarrival time: wd=wd2 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.780 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.780 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.781 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.781 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.782 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.782 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.783 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.783 ++Interarrival time: wd=wd3 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.784 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.784 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.785 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.785 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.786 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.792 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.792 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.793 ++Interarrival time: wd=wd4 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.793 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.794 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.794 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.795 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.795 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.795 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.796 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.796 ++Interarrival time: wd=wd5 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.797 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.797 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.798 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.798 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.798 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.799 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.799 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.800 ++Interarrival time: wd=wd6 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.800 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.801 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.801 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.802 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.802 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.803 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.803 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.804 ++Interarrival time: wd=wd7 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.804 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.805 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.805 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.805 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.806 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.806 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.807 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.807 ++Interarrival time: wd=wd8 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.808 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.808 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.808 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.809 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.809 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.810 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.810 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.810 ++Interarrival time: wd=wd9 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.811 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.811 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.812 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.812 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.813 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.813 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.813 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.814 ++Interarrival time: wd=wd10 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.814 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.814 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.815 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.815 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.815 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.816 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.816 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.816 ++Interarrival time: wd=wd11 arrival=88 rdiorate=999988 target_rate=999988 skew=1.14, wgiorate=0 wgprio=0 iorate=11364
13:28:54.817 Sending to localhost-0  wd=wd1,sd=sd1,lun=/dev/sdl
13:28:54.817 Sending to localhost-0  wd=wd2,sd=sd2,lun=/dev/sdj
13:28:54.817 Sending to localhost-0  wd=wd3,sd=sd3,lun=/dev/sdk
13:28:54.817 Sending to localhost-0  wd=wd4,sd=sd4,lun=/dev/sdh
13:28:54.817 Sending to localhost-0  wd=wd5,sd=sd5,lun=/dev/sdi
13:28:54.817 Sending to localhost-0  wd=wd6,sd=sd6,lun=/dev/sdf
13:28:54.817 Sending to localhost-0  wd=wd7,sd=sd7,lun=/dev/sdg
13:28:54.817 Sending to localhost-0  wd=wd8,sd=sd8,lun=/dev/sdd
13:28:54.817 Sending to localhost-0  wd=wd9,sd=sd9,lun=/dev/sde
13:28:54.817 Sending to localhost-0  wd=wd10,sd=sd10,lun=/dev/sdb
13:28:54.817 Sending to localhost-0  wd=wd11,sd=sd11,lun=/dev/sdc
13:28:54.817 Sending to localhost-1  wd=wd1,sd=sd1,lun=/dev/sdl
13:28:54.817 Sending to localhost-1  wd=wd2,sd=sd2,lun=/dev/sdj
13:28:54.817 Sending to localhost-1  wd=wd3,sd=sd3,lun=/dev/sdk
13:28:54.817 Sending to localhost-1  wd=wd4,sd=sd4,lun=/dev/sdh
13:28:54.818 Sending to localhost-1  wd=wd5,sd=sd5,lun=/dev/sdi
13:28:54.818 Sending to localhost-1  wd=wd6,sd=sd6,lun=/dev/sdf
13:28:54.818 Sending to localhost-1  wd=wd7,sd=sd7,lun=/dev/sdg
13:28:54.818 Sending to localhost-1  wd=wd8,sd=sd8,lun=/dev/sdd
13:28:54.818 Sending to localhost-1  wd=wd9,sd=sd9,lun=/dev/sde
13:28:54.818 Sending to localhost-1  wd=wd10,sd=sd10,lun=/dev/sdb
13:28:54.818 Sending to localhost-1  wd=wd11,sd=sd11,lun=/dev/sdc
13:28:54.818 Sending to localhost-2  wd=wd1,sd=sd1,lun=/dev/sdl
13:28:54.818 Sending to localhost-2  wd=wd2,sd=sd2,lun=/dev/sdj
13:28:54.818 Sending to localhost-2  wd=wd3,sd=sd3,lun=/dev/sdk
13:28:54.818 Sending to localhost-2  wd=wd4,sd=sd4,lun=/dev/sdh
13:28:54.818 Sending to localhost-2  wd=wd5,sd=sd5,lun=/dev/sdi
13:28:54.818 Sending to localhost-2  wd=wd6,sd=sd6,lun=/dev/sdf
13:28:54.818 Sending to localhost-2  wd=wd7,sd=sd7,lun=/dev/sdg
13:28:54.818 Sending to localhost-2  wd=wd8,sd=sd8,lun=/dev/sdd
13:28:54.818 Sending to localhost-2  wd=wd9,sd=sd9,lun=/dev/sde
13:28:54.819 Sending to localhost-2  wd=wd10,sd=sd10,lun=/dev/sdb
13:28:54.819 Sending to localhost-2  wd=wd11,sd=sd11,lun=/dev/sdc
13:28:54.819 Sending to localhost-3  wd=wd1,sd=sd1,lun=/dev/sdl
13:28:54.819 Sending to localhost-3  wd=wd2,sd=sd2,lun=/dev/sdj
13:28:54.819 Sending to localhost-3  wd=wd3,sd=sd3,lun=/dev/sdk
13:28:54.819 Sending to localhost-3  wd=wd4,sd=sd4,lun=/dev/sdh
13:28:54.819 Sending to localhost-3  wd=wd5,sd=sd5,lun=/dev/sdi
13:28:54.819 Sending to localhost-3  wd=wd6,sd=sd6,lun=/dev/sdf
13:28:54.819 Sending to localhost-3  wd=wd7,sd=sd7,lun=/dev/sdg
13:28:54.819 Sending to localhost-3  wd=wd8,sd=sd8,lun=/dev/sdd
13:28:54.819 Sending to localhost-3  wd=wd9,sd=sd9,lun=/dev/sde
13:28:54.819 Sending to localhost-3  wd=wd10,sd=sd10,lun=/dev/sdb
13:28:54.819 Sending to localhost-3  wd=wd11,sd=sd11,lun=/dev/sdc
13:28:54.819 Sending to localhost-4  wd=wd1,sd=sd1,lun=/dev/sdl
13:28:54.819 Sending to localhost-4  wd=wd2,sd=sd2,lun=/dev/sdj
13:28:54.819 Sending to localhost-4  wd=wd3,sd=sd3,lun=/dev/sdk
13:28:54.820 Sending to localhost-4  wd=wd4,sd=sd4,lun=/dev/sdh
13:28:54.820 Sending to localhost-4  wd=wd5,sd=sd5,lun=/dev/sdi
13:28:54.820 Sending to localhost-4  wd=wd6,sd=sd6,lun=/dev/sdf
13:28:54.820 Sending to localhost-4  wd=wd7,sd=sd7,lun=/dev/sdg
13:28:54.820 Sending to localhost-4  wd=wd8,sd=sd8,lun=/dev/sdd
13:28:54.820 Sending to localhost-4  wd=wd9,sd=sd9,lun=/dev/sde
13:28:54.820 Sending to localhost-4  wd=wd10,sd=sd10,lun=/dev/sdb
13:28:54.820 Sending to localhost-4  wd=wd11,sd=sd11,lun=/dev/sdc
13:28:54.820 Sending to localhost-5  wd=wd1,sd=sd1,lun=/dev/sdl
13:28:54.820 Sending to localhost-5  wd=wd2,sd=sd2,lun=/dev/sdj
13:28:54.820 Sending to localhost-5  wd=wd3,sd=sd3,lun=/dev/sdk
13:28:54.820 Sending to localhost-5  wd=wd4,sd=sd4,lun=/dev/sdh
13:28:54.820 Sending to localhost-5  wd=wd5,sd=sd5,lun=/dev/sdi
13:28:54.820 Sending to localhost-5  wd=wd6,sd=sd6,lun=/dev/sdf
13:28:54.820 Sending to localhost-5  wd=wd7,sd=sd7,lun=/dev/sdg
13:28:54.820 Sending to localhost-5  wd=wd8,sd=sd8,lun=/dev/sdd
13:28:54.821 Sending to localhost-5  wd=wd9,sd=sd9,lun=/dev/sde
13:28:54.821 Sending to localhost-5  wd=wd10,sd=sd10,lun=/dev/sdb
13:28:54.821 Sending to localhost-5  wd=wd11,sd=sd11,lun=/dev/sdc
13:28:54.821 Sending to localhost-6  wd=wd1,sd=sd1,lun=/dev/sdl
13:28:54.821 Sending to localhost-6  wd=wd2,sd=sd2,lun=/dev/sdj
13:28:54.821 Sending to localhost-6  wd=wd3,sd=sd3,lun=/dev/sdk
13:28:54.821 Sending to localhost-6  wd=wd4,sd=sd4,lun=/dev/sdh
13:28:54.821 Sending to localhost-6  wd=wd5,sd=sd5,lun=/dev/sdi
13:28:54.821 Sending to localhost-6  wd=wd6,sd=sd6,lun=/dev/sdf
13:28:54.821 Sending to localhost-6  wd=wd7,sd=sd7,lun=/dev/sdg
13:28:54.821 Sending to localhost-6  wd=wd8,sd=sd8,lun=/dev/sdd
13:28:54.821 Sending to localhost-6  wd=wd9,sd=sd9,lun=/dev/sde
13:28:54.821 Sending to localhost-6  wd=wd10,sd=sd10,lun=/dev/sdb
13:28:54.821 Sending to localhost-6  wd=wd11,sd=sd11,lun=/dev/sdc
13:28:54.822 Sending to localhost-7  wd=wd1,sd=sd1,lun=/dev/sdl
13:28:54.822 Sending to localhost-7  wd=wd2,sd=sd2,lun=/dev/sdj
13:28:54.822 Sending to localhost-7  wd=wd3,sd=sd3,lun=/dev/sdk
13:28:54.822 Sending to localhost-7  wd=wd4,sd=sd4,lun=/dev/sdh
13:28:54.822 Sending to localhost-7  wd=wd5,sd=sd5,lun=/dev/sdi
13:28:54.822 Sending to localhost-7  wd=wd6,sd=sd6,lun=/dev/sdf
13:28:54.822 Sending to localhost-7  wd=wd7,sd=sd7,lun=/dev/sdg
13:28:54.822 Sending to localhost-7  wd=wd8,sd=sd8,lun=/dev/sdd
13:28:54.822 Sending to localhost-7  wd=wd9,sd=sd9,lun=/dev/sde
13:28:54.822 Sending to localhost-7  wd=wd10,sd=sd10,lun=/dev/sdb
13:28:54.822 Sending to localhost-7  wd=wd11,sd=sd11,lun=/dev/sdc
13:28:55.039 SlaveList.printWorkForSlaves() *
13:28:55.064 slv=localhost-0 rd=rd1    wd=wd1    sd=sd1    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.065 slv=localhost-0 rd=rd1    wd=wd2    sd=sd2    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.065 slv=localhost-0 rd=rd1    wd=wd3    sd=sd3    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.066 slv=localhost-0 rd=rd1    wd=wd4    sd=sd4    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.067 slv=localhost-0 rd=rd1    wd=wd5    sd=sd5    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.067 slv=localhost-0 rd=rd1    wd=wd6    sd=sd6    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.068 slv=localhost-0 rd=rd1    wd=wd7    sd=sd7    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.069 slv=localhost-0 rd=rd1    wd=wd8    sd=sd8    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.070 slv=localhost-0 rd=rd1    wd=wd9    sd=sd9    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.070 slv=localhost-0 rd=rd1    wd=wd10   sd=sd10   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.071 slv=localhost-0 rd=rd1    wd=wd11   sd=sd11   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.075 slv=localhost-1 rd=rd1    wd=wd1    sd=sd1    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.076 slv=localhost-1 rd=rd1    wd=wd2    sd=sd2    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.076 slv=localhost-1 rd=rd1    wd=wd3    sd=sd3    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.077 slv=localhost-1 rd=rd1    wd=wd4    sd=sd4    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.078 slv=localhost-1 rd=rd1    wd=wd5    sd=sd5    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.078 slv=localhost-1 rd=rd1    wd=wd6    sd=sd6    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.079 slv=localhost-1 rd=rd1    wd=wd7    sd=sd7    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.080 slv=localhost-1 rd=rd1    wd=wd8    sd=sd8    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.080 slv=localhost-1 rd=rd1    wd=wd9    sd=sd9    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.081 slv=localhost-1 rd=rd1    wd=wd10   sd=sd10   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.095 slv=localhost-1 rd=rd1    wd=wd11   sd=sd11   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.096 slv=localhost-2 rd=rd1    wd=wd1    sd=sd1    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.097 slv=localhost-2 rd=rd1    wd=wd2    sd=sd2    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.097 slv=localhost-2 rd=rd1    wd=wd3    sd=sd3    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.098 slv=localhost-2 rd=rd1    wd=wd4    sd=sd4    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.098 slv=localhost-2 rd=rd1    wd=wd5    sd=sd5    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.099 slv=localhost-2 rd=rd1    wd=wd6    sd=sd6    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.100 slv=localhost-2 rd=rd1    wd=wd7    sd=sd7    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.100 slv=localhost-2 rd=rd1    wd=wd8    sd=sd8    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.101 slv=localhost-2 rd=rd1    wd=wd9    sd=sd9    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.102 slv=localhost-2 rd=rd1    wd=wd10   sd=sd10   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.102 slv=localhost-2 rd=rd1    wd=wd11   sd=sd11   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.103 slv=localhost-3 rd=rd1    wd=wd1    sd=sd1    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.104 slv=localhost-3 rd=rd1    wd=wd2    sd=sd2    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.105 slv=localhost-3 rd=rd1    wd=wd3    sd=sd3    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.105 slv=localhost-3 rd=rd1    wd=wd4    sd=sd4    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.106 slv=localhost-3 rd=rd1    wd=wd5    sd=sd5    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.107 slv=localhost-3 rd=rd1    wd=wd6    sd=sd6    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.108 slv=localhost-3 rd=rd1    wd=wd7    sd=sd7    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.108 slv=localhost-3 rd=rd1    wd=wd8    sd=sd8    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.109 slv=localhost-3 rd=rd1    wd=wd9    sd=sd9    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.110 slv=localhost-3 rd=rd1    wd=wd10   sd=sd10   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.110 slv=localhost-3 rd=rd1    wd=wd11   sd=sd11   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.111 slv=localhost-4 rd=rd1    wd=wd1    sd=sd1    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.112 slv=localhost-4 rd=rd1    wd=wd2    sd=sd2    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.112 slv=localhost-4 rd=rd1    wd=wd3    sd=sd3    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.113 slv=localhost-4 rd=rd1    wd=wd4    sd=sd4    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.114 slv=localhost-4 rd=rd1    wd=wd5    sd=sd5    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.114 slv=localhost-4 rd=rd1    wd=wd6    sd=sd6    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.115 slv=localhost-4 rd=rd1    wd=wd7    sd=sd7    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.116 slv=localhost-4 rd=rd1    wd=wd8    sd=sd8    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.116 slv=localhost-4 rd=rd1    wd=wd9    sd=sd9    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.117 slv=localhost-4 rd=rd1    wd=wd10   sd=sd10   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.118 slv=localhost-4 rd=rd1    wd=wd11   sd=sd11   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.118 slv=localhost-5 rd=rd1    wd=wd1    sd=sd1    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.119 slv=localhost-5 rd=rd1    wd=wd2    sd=sd2    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.120 slv=localhost-5 rd=rd1    wd=wd3    sd=sd3    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.120 slv=localhost-5 rd=rd1    wd=wd4    sd=sd4    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.121 slv=localhost-5 rd=rd1    wd=wd5    sd=sd5    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.122 slv=localhost-5 rd=rd1    wd=wd6    sd=sd6    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.122 slv=localhost-5 rd=rd1    wd=wd7    sd=sd7    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.123 slv=localhost-5 rd=rd1    wd=wd8    sd=sd8    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.124 slv=localhost-5 rd=rd1    wd=wd9    sd=sd9    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.124 slv=localhost-5 rd=rd1    wd=wd10   sd=sd10   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.125 slv=localhost-5 rd=rd1    wd=wd11   sd=sd11   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.125 slv=localhost-6 rd=rd1    wd=wd1    sd=sd1    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.126 slv=localhost-6 rd=rd1    wd=wd2    sd=sd2    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.127 slv=localhost-6 rd=rd1    wd=wd3    sd=sd3    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.128 slv=localhost-6 rd=rd1    wd=wd4    sd=sd4    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.128 slv=localhost-6 rd=rd1    wd=wd5    sd=sd5    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.129 slv=localhost-6 rd=rd1    wd=wd6    sd=sd6    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.130 slv=localhost-6 rd=rd1    wd=wd7    sd=sd7    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.130 slv=localhost-6 rd=rd1    wd=wd8    sd=sd8    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.131 slv=localhost-6 rd=rd1    wd=wd9    sd=sd9    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.132 slv=localhost-6 rd=rd1    wd=wd10   sd=sd10   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.132 slv=localhost-6 rd=rd1    wd=wd11   sd=sd11   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.144 slv=localhost-7 rd=rd1    wd=wd1    sd=sd1    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.145 slv=localhost-7 rd=rd1    wd=wd2    sd=sd2    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.146 slv=localhost-7 rd=rd1    wd=wd3    sd=sd3    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.147 slv=localhost-7 rd=rd1    wd=wd4    sd=sd4    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.147 slv=localhost-7 rd=rd1    wd=wd5    sd=sd5    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.148 slv=localhost-7 rd=rd1    wd=wd6    sd=sd6    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.149 slv=localhost-7 rd=rd1    wd=wd7    sd=sd7    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.149 slv=localhost-7 rd=rd1    wd=wd8    sd=sd8    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.150 slv=localhost-7 rd=rd1    wd=wd9    sd=sd9    rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.151 slv=localhost-7 rd=rd1    wd=wd10   sd=sd10   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.151 slv=localhost-7 rd=rd1    wd=wd11   sd=sd11   rd= 50 sk=100 sw= 1.14 rh=  0 hi=0 th=2
13:28:55.151
13:28:55.152 Waiting for synchronization of all slaves
13:28:55.653 Synchronization of all slaves complete
13:28:56.003 Starting RD=rd1; I/O rate: Uncontrolled MAX; elapsed=1800; For loops: threads=16.0 rdpct=50.0
<a name="_215181933"></a><i><b>13:28:56.013 Starting RD=rd1; I/O rate: Uncontrolled MAX; elapsed=1800; For loops: threads=16.0 rdpct=50.0</b></i>
Jun 25, 2014  interval        i/o   MB/sec   bytes   read     resp     read    write     resp     resp queue  cpu%  cpu%
                             rate  1024**2     i/o    pct     time     resp     resp      max   stddev depth sys+u   sys
13:29:01.179         1    3465.60   366.26  110817  49.38   48.609   98.276    0.165 1075.965  100.053 173.8  16.4   2.1
13:29:05.542
13:29:05.542 Message from slave localhost-0:
13:29:05.542 New messages found on /var/adm/messages. Do they belong to you?
13:29:05.542 /var/log/messages: Jun 25 13:29:04 svt-smx9dri-002 kernel: megasas: Found FW in FAULT state, will reset adapter.
13:29:05.542 /var/log/messages: Jun 25 13:29:04 svt-smx9dri-002 kernel: megaraid_sas: resetting fusion adapter.
13:29:05.543
13:29:06.099         2    1202.80   124.35  108403  49.57   74.091  149.305    0.166 1239.084  136.329 174.8   2.6   0.5
13:29:11.083         3       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000 176.0   1.1   0.2
13:31:23.556 HeartBeat.checkHeartBeat(): slave localhost-0 has not responded for 125 seconds.
13:31:42.042 HeartBeat.checkHeartBeat(): slave localhost-1 has not responded for 125 seconds.
13:31:42.042 HeartBeat.checkHeartBeat(): slave localhost-2 has not responded for 125 seconds.
13:31:42.042 HeartBeat.checkHeartBeat(): slave localhost-3 has not responded for 125 seconds.
13:31:42.042 HeartBeat.checkHeartBeat(): slave localhost-4 has not responded for 125 seconds.
13:31:42.042 HeartBeat.checkHeartBeat(): slave localhost-5 has not responded for 125 seconds.
13:31:42.042 HeartBeat.checkHeartBeat(): slave localhost-6 has not responded for 125 seconds.
13:31:42.043 HeartBeat.checkHeartBeat(): slave localhost-7 has not responded for 125 seconds.
13:31:42.043 Start/end command: executing '/opt/vdbench/vdbench jstack'
13:31:42.043 execute(): /opt/vdbench/vdbench jstack
13:31:42.066         4       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000 177.5   1.1   0.2
13:31:42.099         5       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000  88.0   0.3   0.1
13:31:42.154         6       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000 132.0   0.5   0.3
13:31:42.277         7       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000 176.0   0.2   0.1
13:31:42.362         8       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000  89.6   0.2   0.0
13:31:42.456         9       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000 199.5   0.2   0.0
13:31:42.520 Debug_cmds(); stderr 13:31:42.519 Jstack positional parameter: 'jstack'
13:31:42.525        10       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000  89.6   NaN   NaN
13:31:42.568        11       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000 110.0   NaN   NaN
13:31:42.600        12       0.00     0.00       0   0.00    0.000    0.000    0.000    0.000    0.000 153.9   NaN   NaN
13:31:42.625        13      48.80     5.15  110592  52.87 37672.016 71255.465    0.146 158989.885 67579.972 1175.5   NaN   NaN
13:31:42.649        14      35.40     3.68  109041  53.11 21497.768 40479.680    0.181 158826.936 54393.776 443.7   0.2   0.0
13:31:42.673        15      95.20    10.43  114860  50.21 9012.154 17948.704    0.190 159358.884 36695.871 423.5   0.1   0.0
13:31:42.698        16       2.00     0.16   81920  80.00   41.933   52.397    0.078  176.845   53.803  22.0   NaN   NaN
13:31:42.724        17      48.40     5.19  112453  48.35 12460.927 25773.729    0.145 158989.060 42715.660 378.0  81.3  12.5
13:31:42.744        18      11.40     1.40  128772  49.12 11148.681 22695.195    0.323 158873.193 40875.328   0.3  85.7  14.3
13:31:42.764        19      28.00     3.32  124284  50.71 2286.839 4509.065    0.200 158448.763 18867.068   0.8  90.0   0.0
13:31:42.783        20      11.80     1.23  109411  52.54   26.111   49.329    0.406  163.620   39.186   0.4  80.0   0.0
13:31:42.792
13:31:42.792 Message from slave localhost-0:
13:31:42.792 New messages found on /var/adm/messages. Do they belong to you?
13:31:42.793 /var/log/messages: Jun 25 13:29:10 svt-smx9dri-002 kernel: megasas: Waiting for FW to come to ready state
13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: INFO: task jbd2/sda2-8:696 blocked for more than 120 seconds.
13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: jbd2/sda2-8   D 0000000000000000     0   696      2 0x00000000
13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880276235c20 0000000000000046 ffff880276235bb0 ffffffff8108105d
13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880276235b90 ffffffff81012b69 ffff88027643f938 ffff88027643f848
13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880276681ab8 ffff880276235fd8 000000000000fb88 ffff880276681ab8
13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: Call Trace:
13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8108105d>] ? del_timer+0x7d/0xe0
13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81012b69>] ? read_tsc+0x9/0x20
13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff810a1aa9>] ? ktime_get_ts+0xa9/0xe0
13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811b5f60>] ? sync_buffer+0x0/0x50
13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8150de73>] io_schedule+0x73/0xc0
13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811b5fa0>] sync_buffer+0x40/0x50
13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8150e82f>] __wait_on_bit+0x5f/0x90
13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811b5f60>] ? sync_buffer+0x0/0x50
13:31:42.793 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8150e8d8>] out_of_line_wait_on_bit+0x78/0x90
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096cc0>] ? wake_bit_function+0x0/0x50
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811b5f56>] __wait_on_buffer+0x26/0x30
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa02240a6>] jbd2_journal_commit_transaction+0xa76/0x14b0 [jbd2]
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81081a3b>] ? try_to_del_timer_sync+0x7b/0xe0
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0229f38>] kjournald2+0xb8/0x220 [jbd2]
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096c80>] ? autoremove_wake_function+0x0/0x40
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0229e80>] ? kjournald2+0x0/0x220 [jbd2]
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096916>] kthread+0x96/0xa0
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096880>] ? kthread+0x0/0xa0
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: INFO: task flush-8:0:800 blocked for more than 120 seconds.
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: flush-8:0     D 000000000000000a     0   800      2 0x00000080
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff88027a5c1590 0000000000000046 ffff88027a5c1560 ffffffff813791cb
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: 0000000000000003 0000000000000086 ffff88027a5c1540 ffffffff8108105d
13:31:42.794 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880273f665f8 ffff88027a5c1fd8 000000000000fb88 ffff880273f665f8
13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: Call Trace:
13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff813791cb>] ? scsi_request_fn+0xdb/0x7a0
13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8108105d>] ? del_timer+0x7d/0xe0
13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096f6e>] ? prepare_to_wait+0x4e/0x80
13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa022305d>] do_get_write_access+0x29d/0x520 [jbd2]
13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096cc0>] ? wake_bit_function+0x0/0x50
13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0223431>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa026e708>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa02705b2>] ext4_mb_mark_diskspace_used+0xf2/0x300 [ext4]
13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa02771e1>] ext4_mb_new_blocks+0x311/0x5b0 [ext4]
13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0269feb>] ? ext4_ext_find_extent+0x2ab/0x310 [ext4]
13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa026c170>] ext4_ext_get_blocks+0x4f0/0x14b0 [ext4]
13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811b62d0>] ? end_buffer_async_write+0x0/0x190
13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa024b270>] ? noalloc_get_block_write+0x0/0x60 [ext4]
13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa024b270>] ? noalloc_get_block_write+0x0/0x60 [ext4]
13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811b75d0>] ? block_write_full_page_endio+0xe0/0x120
13:31:42.795 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa024ab89>] ext4_get_blocks+0xf9/0x2a0 [ext4]
13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8112ef45>] ? pagevec_lookup_tag+0x25/0x40
13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa024f201>] mpage_da_map_and_submit+0xa1/0x470 [ext4]
13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa0222455>] ? jbd2_journal_start+0xb5/0x100 [jbd2]
13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffffa024fe1e>] ext4_da_writepages+0x2ee/0x620 [ext4]
13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8112e0f1>] do_writepages+0x21/0x40
13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811ac6ed>] writeback_single_inode+0xdd/0x290
13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811acafe>] writeback_sb_inodes+0xce/0x180
13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811acc5b>] writeback_inodes_wb+0xab/0x1b0
13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811acffb>] wb_writeback+0x29b/0x3f0
13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8150d6e0>] ? thread_return+0x4e/0x76e
13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81081ac2>] ? del_timer_sync+0x22/0x30
13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811ad2e9>] wb_do_writeback+0x199/0x240
13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff811ad3f3>] bdi_writeback_task+0x63/0x1b0
13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096b47>] ? bit_waitqueue+0x17/0xd0
13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8113ca40>] ? bdi_start_fn+0x0/0x100
13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8113cac6>] bdi_start_fn+0x86/0x100
13:31:42.796 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8113ca40>] ? bdi_start_fn+0x0/0x100
13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096916>] kthread+0x96/0xa0
13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81096880>] ? kthread+0x0/0xa0
13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: INFO: task jbd2/sda3-8:1948 blocked for more than 120 seconds.
13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: jbd2/sda3-8   D 0000000000000000     0  1948      2 0x00000080
13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880273605b30 0000000000000046 0000000000000000 ffff880273605af4
13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff880200000000 ffff88027fc24180 ffff8800282d6700 0000000000000400
13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: ffff8802735465f8 ffff880273605fd8 000000000000fb88 ffff8802735465f8
13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: Call Trace:
13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81119cd0>] ? sync_page+0x0/0x50
13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff8150de73>] io_schedule+0x73/0xc0
13:31:42.797 /var/log/messages: Jun 25 13:31:30 svt-smx9dri-002 kernel: [<ffffffff81119d0d>] sync_page+0x3d/0x50
13:31:42.797 /var/log/message