Cache Flushes Solaris10 StorageTek D280 and NFS and ZFS

Question

I encounter complains from users, who are connected via nfs, to Sun Solaris10 server.
The server is connected via Fibre to a Storage Tek d280.
The performance on the server is okay.
However, on the , via nfs connected , clients, the performance is poor.
I found this document, and want to try to disable the cache flushes on the server.
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Cache_Flushes
However, I rather want to have the Storage Tek D280 acting as a nice zfs storage device, rather than tweaking the operating system.
But I can not find any document on how to configure the behavior of the cache flushes on this device.
Is there someone who know how to setup this storage tek D280 box correctly to ingnore the Cache Flush commands generated by the NFS ?
Kind regards,

Answer 1

806760 wrote:
Thanks for the response.
I don't know how the D280 internally has been setup. It should use raid 5. That's about the only thing I know about it.
It is under control of an ICT department.
However, the effect, or if the D280 is poor configured, does it only affect the NFS clients connected to the Solaris Server ?
I have ruled out the network configuration. This is a 1Gb connection. And for diagnose I tryed with a different switch, and direct connection.
But it did not influence the poor performance of the client, using NFS.
As a test, I just extract a tar file with a big amount of empty files.
This goes over 25 times slower on the clients, than on the server.
I have installed about 8 of those systems, but none is performing so bad.
Since everything on all systems is about the same configuration, the only things which are out of my control, is the network, and the san.
I tryed to test the network, but I don't see any problems with that.
So in my mind, the only thing left, would be the san device.
Searching on this topic, I found some explanations, about the zfs with nfs, which works poorly, due to the nfs is committing a regular synchronous write (NFS commit) However I don't like to do this.
I also can not find any description on how to configure a D280.
It would be nice, if you could provide some settings which has to be set in a D280.
The configuration is two cluster nodes, and two clients.
The cluster node mainly task is to provide the nfs shares.
The clients and servers are in one 19" rack.
The San, I don't know where it is.
It has a 2Gb fibre coupling. ( On the server side there are 4Gb Emulex HBA's installed )
Kind regards,If a tar file extracts 25 times faster on the server then it does over the network, yet both times the data is being written to the SAN LUNs on the D280, the problem is the network.
That tar file extracts slower across the network for two reasons: bandwidth and latency.
There's only so much data you can stuff through a gigE network. Your single 1 gigE link can handle about 100 MB/sec read and 100 MB/sec write combined - total. For all users. That may be part of your performance problem, because the configuration LUN layout of that D280 would have to be really, REALLY bad for it to be unable handle that relatively small amount of IO. You CAN test the performance of the LUNs being presented to your server - just use your favorite benchmarking tool to do various reads from the "/dev/rdsk/...." device files that make up your filesystem(s). Just make doggone sure you ONLY do reads - if you write to those LUNs your filesystem(s) will be corrupted. Something like "dd if=/dev/rdsk/... of=/dev/null bs=1024k count=10000" will tell you how fast that one LUN can stream data - but it won't tell you how many IO ops/sec the LUN can support as you'd need to do random small reads to do that. Any halfway-decently configured D280 LUN should be able to stream data at a constant 200 MB/sec while you're reading from it.
And even if the bandwidth were much higher, you still have to deal with the additional latency of having to do all communications across your network. No matter how fat the pipe is, it still takes more time to send data across the network and have to wait for a reply. What do your ping times look like between client and server? And even with that added latency, there are some things you can do on your hosts. Increase your TCP buffer sizes, mount your filesystems on your Linux clients with the "rsize=32768,wsize=32768,intr,noatime" options, and maybe use NFSv3 instead of NFSv4 - make sure you change both the server and client settings to be sure. And work with your network admins to get jumbo frames enabled. Moving more data per packet is a good way to address latency because you wind up having to wait for a response much fewer times.