[SOLVED] Encryption I/O is very slow on Atom 330

Question

EDIT: Initially titled "Everyday I/O on software RAID1 is very slow"
I recently upgraded my home server from Ubuntu (Hardy, I think) to Arch, coinciding with an disk update (2 * 500GB to 2 * 2TB).
However, I seem to have quite bad performance problems during "standard" usage of the server. Read or write speeds, even on large (2G) files, never exceed 17/18 MB/s (according to 'iostat -m 1').
This causes issues when using the server, for example, watching a video mounted over SSHFS. When the array is written to, it causes the video to pause for a moment, as though all read I/O stops for the ~2 seconds the array is being written to (at the aforementioned 17MB/s). I noticed this initially when I was transferring the files from the old drive and running digest checks (sha1sum) were taking far longer than I expected on my new installation, so I checked iostat which showed me the 17MB/s speed during the sha1sum checks.
This doesn't seem to be an issue with the drives themselves, nor the controllers they are attached to, as when I was creating the array, or if I force a disk to be rebuilt, it read/writes at 80MB/s (per drive and simultaneously), which is what I would expect from these drives (SATA II, 5400RPM).
I currently have it set up as such:
2 partitions per drive, set to raid_autodetect which are md0 and md1 respectively.
md0 is /boot (ext2), md1 is an encrypted volume
Encrypted volume contains an LVM with a 1G swap partition and the rest of the volume is assigned to / (ext3).
Could this setup be the root cause of my issues?
I've never had any performance issues before with encryption (even on my 1.6Ghz tablet which runs at 600Mhz with cpufreq scaling) so it doesn't strike me as it would be an issue for my Dual Core 1.6Ghz/2G RAM server (a Zotac ION-ITX-A for lower power consumption). However, it is possible that a difference in encryption methods may account for it being an issue here as I use the defaults, which may be "paranoid"..
I don't know much about LVM, so I am concerned the mere size of the LVM '/' volume may be the cause of my issue.
I'm happy to provide any extra information or run any tests to help diagnose this issue as it is quite the show-stopper for this server!
Last edited by Xyem (2010-10-20 10:41:09)

Answer 1

Something just occurred to me.. I have just over 90M of free space in /boot which I can play with, so I just ran a quick test (while watching 'iostat')...
sudo dd if=/dev/zero of=/boot/test.zero bs=1M count=90
75+0 records in
75+0 records out
78643200 bytes (79 MB) copied, 0.541392 s, 145 MB/s
But 'iostat' didn't show anything. Then it hit me, caching!
sync
/dev/md0 gets written to at 80M/s! So this narrows it down it either being an encryption slowdown or a LVM issue. I can't imagine LVM would slow it down to 17M/s so I am going to format and set this thing up without encryption and see how it goes then
EDIT: I just found this page (http://www.shimari.com/dm-crypt-on-raid … benchmarks) which seems to match the poor performance I am seeing and points the finger squarely at dm-crypt. What is worrying is encryption seems to come with a 75% speed penalty, which is really making me rethink full disk encryption on all my computers (server, desktops, laptops, tablet and carputer)!
EDIT2: Not quite sure on the etiquette in this case, in regards to whether I should edit this post or have made a new one but the cause of this has been identified.
The server has a Atom 330 CPU and after several tests (and confirmed on 2 other machines with the same CPU) it has been discovered that despite being dual core 1.6Ghz cores, the encryption performance on it is absolutely terrible. Changing the cipher to aes-ecb-null results in 28M/s but on a desktop machine (non-Atom), that cipher results in zero performance drop (80M/s raw and encrypted) and the default cipher (aes-cbc-essiv:sha256) only drops to 55M/s. Hope this helps someone else. Off to go and reformat my server and carputer as unencrypted.
Last edited by Xyem (2010-10-17 16:46:32)