4GB HBAs on a 66mhz 64-bit PCI-X Bus

Question

I have some older servers that are having issues and I believe the bus speed might be the cause. I'm actually a storage guy, not a UNIX admin. I'm trying to identify why I have some slow-drain devices on my switches and I believe the Bus speed on some of our older Sun servers might be the issue.
Slow-drains are quite annoying, as they drive response times up across my fabric. When we run backups, I can see my response times go way up across all servers. Basically my fabric is waiting on these servers to send their acknowledgement that they received frames. No issues with Switches, Arrays, or Cables.
Here's the main server that seems to be giving issues
Server1
Sunfire V890
4 Emulex LP11002 4gb HBAs - Plugged into 66mhz 64-bit PCI-X slots (all the same bus)
Data is on 80 - 15k FC Disks
Here are 2 other servers which also give issues.
Server2
Sunfire v890
2 Emulex LP10000 2gb HBAs - Plugged into 33mhz PCI-X Slots
Data is on 15 - 15k FC Disks
Server3
e2900
4 Emulex LP11002 4GB HBAs - Plugged into 100mhz 64-bit PCI-X slots (all the same bus)
Data is on 100 - 15k FC disks
When I run backups on any of these servers (server3 isn't that bad) I see response times increase on every server SAN attached. I've done extensive research on my array and the array is not the problem. I constantly get latency alerts on my switch (host not responding quick enough and sucking up all the buffer credits)
So when this issue happens, Server1 is pulling around 530megabytes/s (~265 tx, ~265rx) through those HBAs. That is close to the upper-limit of a 64-bit 66mhz PCI-X bus.
When I see that issue happen on Server2, it's pulling around 220MB/s (110 tx, 110 rx) through those HBAs. That's close to the upper-limit of a 33 mhz PCI-X bus.
So I guess what I want to know is... Has anyone had issues before by using HBAs that are a higher speed than the Bus can handle? Would this be expected? Am I missing something, or am I completely wrong here? I thought about dropping these HBAs down to 2GB or 1Gb to see if it fixes the issue. I was using 4 paths to my backup array and dropped it down to 2 (reducing the bandwidth to the backup array by 50%) and that seemed to help a bit.

Answer 1

Remember that by definition, the bus is an interrupt driven mechanism, and if each HBA is attempting an operation, it gets the bus for a time slice to the exclusion of the other HBAs. Sounds like your servers are trying to pump a lot of data through one bus, and having a lot of contention doing so.
You should see if the servers have a split bus structure. For example, some servers have a bus for PCI slots 0-3, then another separate bus for slots 4-7, etc. You'd have to dig out the specs to determine this. Placing your HBAs across these separate buses might eliminate the contention you are seeing (for example, HBA0 and 1 would go into slots 0, 1, then HBA2 and 3 would go into slots 4,5).