Poor iSCSI Performance over 10GbE connection.

Question

Hi Solaris Folks,
I'v a setup of two SunFire X4540 connected over a 10GbE link (a Cisco Catalyst 4507 in the middle) using Sun's Dual 10 GbE NIC (X1027A-Z) on both machines.
The plan is to provide the 2nd machine a ZFS volume via iSCSI from the 1st machine. The 1st machine has a ZPool wiht 22x 2-way mirror vdevs and a locale
FileBench test shows good results.
I've now configured the 1st machine to share a ZFS volume via iSCSI and on the 2nd machine I used that target as a raw-device for a new ZPool. If I now run
FileBench againt the "iSCSI-ZPool" I get results which are far away from the local benchmarks and far away from really using the 10GbE link (most times the
bandwidth is around 1,5 GBit/s).
Did a couple of performance tuning which I found in the Sun Blogs and Wikis but without any luck to increase the performance significantly.
Anybody out there who tried the same (iSCSI over 10GbE)? Any experience with iSCSI performance? Suggestions?
Here is an overview about the changes I made:
iSCSI Initiator only:
tcp-nodelay=1; in /kernel/drv/iscsi.conf
iSCSI Targer only:
Configured a TPGT so that only the 10GbE connection can be used.
On both machines:
tcp_recv_hiwat 400000, tcp_xmit_hiwat 400000, tcp_max_buf 2097152, tcp_cwnd_max 2097152 on /dev/tcp via ndd
soft-lso-enable = 1; and accept_jumbo=1; within nxge.conf
MTU on both NICs is set to 9194. Jumbo Frames are also configured on the Cisco switch ports.
Regards,
Lemmy

Answer 1

The answer could be .. don't use ZFS volumes
I have an SSD and a simple Seagate disk. Both on SATA. Same capacity of 74.53GB
If I use those disks to create a pool and make volumes on the pool and then
use the shareiscsi property to create a target (and there is plenty of documentation
describing that as the easy way to do it ) the performance
is dreadful.
If on the other hand I use iscsitadm to make a target out of the devices of two disks using commands like this
for example
iscsitadm create target --type raw -b /dev/dsk/c4t0d0
iscsitadm create target --type raw -b /dev/dsk/c4t1d0
and then on the initiator create the pool then the performance is entirely different!
A bit more detail:
For example I built a target out of a zfs volume then on the initiator I wrote to the pool using zfs send/recv with about 3.2GB. It took 26 minutes 24 sec - pathetic.
I almost gave up waiting!
So i exported the these useless zfs pools, removed the static-configs etc and cleared the zfs property etc and started again.
If the target is instead built "manually" using iscsitadm e.g. "iscsitadm create target --type raw -b /dev/dsk/c4t0d0" (dsk or rdsk it doesn't seem to make much difference)
the same write test takes just 3 min 12sec.
So in this example Solaris 10 10/09 (x86) the performance degrades by a factor of 8 if you use zfs volumes. Nice idea ... easy to use but ... terrible overhead.
At least for SATA disks.
Incidentally for a SATA SSD disk the factor is not so bad (5 min vs 3 min) . However an ordinary disk like ST3808110AS .. 8 times slower with zfs volumes.
No idea why .. but easily reproducible.
So .. what do you see if you don't use zfs volumes but instead use the disk device name itself (whole disk remember - not a slice)
For target and initiator I was using Solaris 10 10/09 (x86) on a v20z with 3rd party eSATA card. SunOS kernel patched recently to 5.10 Generic_142901-10 on the target.