Ridiculously slow mdadm reshape / grow

Question

I had a software RAID5 setup with mdadm and my onboard RAID-controller. It was a 4x1TB RAID5 giving me a total of 3TB. Yesterday I added one more 1TB disk with these commands
mdadm --add /dev/md0 /dev/sdf1
mdadm --grow /dev/md0 --raid-devices=5 --backupfile=/mnt/temp/backupfileraid5
In the begining the speed was around 13MB/s but today when I checked the progress, this is what I saw:
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdf1[4] sde1[0] sdc1[3] sdb1[2] sdd1[1]
2930279424 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
[>....................] reshape = 3.6% (35387136/976759808) finish=58349.5min speed=267K/sec
unused devices: <none>
# cat /proc/sys/dev/raid/speed_limit_min
30000
And I don't read or write to the array, so that is not the problem.
Does anyone have any creative ideas how to increase the reshape speed? :S

Answer 1

I am using
5 2 TB drives
24gb ddr3
x58m board
Xeon 5550 CPU
This is what I am getting with nearly the same storage setup
root@xxxxxxxf(/etc) $ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdf1[5] sdb1[0] sde1[4] sdd1[2] sdc1[1]
      5589036288 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
      [========>............]  reshape = 44.7% (832884860/1863012096) finish=706.3min speed=24306K/sec
Here is the settings I used to do it. You might want to use slightly smaller or larger stripe cache_size depending how much ram you have.
echo 6144 >> /sys/block/md3/md/stripe_cache_size 
echo 75000 >> /sys/block/md3/md/sync_speed_min
echo 225000 >> /sys/block/md3/md/sync_speed_max
* The tweak will go away after a reboot unles syou add it to a boot up file. My reshape was going to take 3600 minutes I made the change and it dropped to 1100 minutes *
The speeds are pretty safe min/max
Adjust the cache in 1024 increments start at 3072 and go up  use top and watch for sillyness and back it off a bit if it is getting hammered
It has already gone up while typing this.
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdf1[5] sdb1[0] sde1[4] sdd1[2] sdc1[1]
      5589036288 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
      [=========>...........]  reshape = 45.2% (843672896/1863012096) finish=728.1min speed=23331K/sec
Hope this helps
2G33K4U