Poor disk io thourghtput on OVM and guest VM

Question

We are using DELL R720 with H710 raid controller.
A wired problem about the disk io was discovered during tests.
We found that the disk throught on VM server(OVS repo) and VM guest are only about 1/3 of the physical server, around only 50M/s.
We are tring to compare disk io write with simple dd instruction on following enviroments with same hardware
The test cases are:
(H710, 2T 7200rpm * 8 configured with RAID10 as two virtual disks, first for ovm, second for OVS repo)
1. Centos 6 installed directly
2. OVM 3.2.1, sda
3. OVM 3.2.1, /dev/mapper/36848f690ec834b0018df77d30704a452 (used by OVS repo)
4. Guest vm Oracal Linux 6.0 on OVM 3.2.1,pvm (using local physical disk as OVS repository)
The result is:
1.Centos 6 installed directly
dd if=/dev/zero of=~/a.out bs=16k count=20000 oflag=direct
20000+0 records in
20000+0 records out
327680000 bytes (328 MB) copied, 1.85022 s, 177 MB/s
2. OVM 3.2.1, sda
dd if=/dev/zero of=~/a.out bs=16k count=20000 oflag=direct
20000+0 records in
20000+0 records out
327680000 bytes (328 MB) copied, 2.5659 seconds, 128 MB/s
3.OVM 3.2.1, /dev/mapper/36848f690ec834b0018df77d30704a452
dd if=/dev/zero of=/OVS/Repositories/0004fb00000300000102bf00d2adeb8a/a.out bs=16k count=10000 oflag=direct
10000+0 records in
10000+0 records out
163840000 bytes (164 MB) copied, 3.00933 seconds, 54.4 MB/s
4. Guest vm Oracal Linux 6.0 on OVM 3.2.1,pvm
dd if=/dev/zero of=~/a.out bs=16k count=20000 oflag=direct
20000+0 records in
20000+0 records out
327680000 bytes (328 MB) copied, 7.60928 s, 43.1 MB/s
We can see that on pure Centos 6 we got the reasonable disk io, on OVM sda the score is slower but still acceptable.
But on OVM disk and guest VM, we saw a big drop.
An interesting thing is : I have 3 guest vms on this vm server, when I run a same dd command samultaniously on 3 guest vms, I can still
get the same score. Adding them together we get a 120+ MB/s throughput.
I guess the reason might be that OVS put a io threshold on both OVS and each vms on it to make sure no single vm can drain up all disk io
in order to preserve some to other vms ?
I plan to use mongo db on the guest vm if the io is close to the real machine, could anyone please help me improve the disk io ?
thanks a lot!
Edited by: user12945979 on 2013-5-1 上午9:21
Edited by: user12945979 on 2013-5-3 上午5:10
Edited by: user12945979 on 2013-5-3 上午5:14

Answer 1

Thanks for your advice,
I followed played below tests against physical server(Centos6) and Oracal vm guest server(OL6,pvm).
fio -filename=/srv/test.out -direct=1 -rw=randwrite -bs=4k -size=2g -numjobs=8 -runtime=60 -group_reporting -name=test
fio -filename=/srv/test.out -direct=1 -rw=randread -bs=4k -size=2g -numjobs=8 -runtime=60 -group_reporting -name=test
fio -filename=/srv/test.out -direct=1 -rw=randrw -bs=4k -size=2g -numjobs=8 -runtime=60 -group_reporting -name=test
fio -filename=/srv/test.out -direct=1 -rw=read -bs=4k -size=2g -numjobs=8 -runtime=60 -group_reporting -name=test
fio -filename=/srv/test.out -direct=1 -rw=write -bs=4k -size=2g -numjobs=8 -runtime=60 -group_reporting -name=test
fio -filename=/srv/test.out -direct=1 -rw=read -bs=4k -size=2g -numjobs=8 -runtime=60 -group_reporting -name=test
fio version is 2.0.13
The results shows random read on VM guest matches about 60% of physical server, and random write about 90%.
But sequential read only counts about 42% and write 35%.
I am not sure if the big drop on sequential read/write performance is reasonable. Maybe I did something wrong ?
                         Physical     VM guest
randwrite(4k,60s)     write iops     2385          2114
               bw(k)          9542          8456
randread(4k,60s)     read iops     1455          838
               br          5822          3352
randrw(4k,60s)          read/write iops     791/794          569/565
               b r/w           3167/3176     2278/2260
write(4k,60s)          write iops     14465          5114
               bw(k)          57864          20457
read(4k,60s)          read iops     48870          20985
               br           195483          83940
Edited by: user12945979 on 2013-5-3 上午5:16