Scsi offline or reservation conflict errors afer swapping HBAs

Question

We swapped out all of our EMC branded Emulex LP10000 HBAs and replaced them with the Sun branded Emulex LP11000 (SG-XPCI1FC-EM4-Z). We now receive a steady stream of SCSI errors in /var/ad/messages similar to the following:
Nov 8 13:37:29 fhtulsaps201 scsi: [ID 107833 kern.warning] WARNING: /ssm@0,0/pci@19,600000/SUNW,emlxs@1/fp@0,0/ssd@w50060482d52dd0b7,16 (ssd27):
Nov 8 13:37:29 fhtulsaps201 offline or reservation conflict
These devices are target devices from a remote server and are kept in sync using EMCs RecoverPoint appliances. They are normally in a read write disabled state. I do not have access to SRDF but expect I would get the same errors since SRDF also puts the target devices in the same read write disabled state.
As you can see the messages file fills up rather quickly and is basically unusable.
root@fhtulsaps201# wc -l /var/adm/messages
208634 /var/adm/messages
root@fhtulsaps201# ls -l /var/adm/messages*
-rw-r--r-- 1 root root 28340369 Nov 8 13:52 /var/adm/messages
-rw-r--r-- 1 root root 94960043 Nov 7 03:07 /var/adm/messages.0
-rw-r--r-- 1 root root 94643625 Oct 31 03:08 /var/adm/messages.1
-rw-r--r-- 1 root root 113367276 Oct 24 03:07 /var/adm/messages.2
-rw-r--r-- 1 root root 118930213 Oct 17 03:08 /var/adm/messages.3
root@fhtulsaps201#
As a part of the HBA upgrade we applied several kernel patches so at this point I'm not sure if the new HBAs are to blame or one of the patches.
The old HBAs never produced these errors.
I would like to know it there's a way to reduce the logging output from the scsi driver.
Thanks for any help.

Answer 1

We see the same error, though not nearly in such abundance, on our Clariion systems when they are using "explicit trespass" mode. In this mode the "passive" path is completely inaccessible until a special control message is sent to it. Veritas sends that special control message when the primary path has failed (EMC's PowerPath does the same thing).
I know EMC recommends this because you should never end up with I/O going to the wrong service processor accidentally (a trespass) because it simply won't service such I/O's unless explicitly told to do so (gee, I wonder where the name explicit trespass came from? :)
I'd guess your RecoverPoint devices behave similarly.
The relevant code appears to this this bit from http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/io/scsi/targets/sd.c#9589:
   9570           /*
   9571            * If the target is not yet ready here (defined by a TUR
   9572            * failure), invalidate the geometry and print an 'offline'
   9573            * message. This is a legacy message, as the state of the
   9574            * target is not actually changed to SD_STATE_OFFLINE.
   9575            *
   9576            * If the TUR fails for EACCES (Reservation Conflict),
   9577            * SD_RESERVED_BY_OTHERS will be returned to indicate
   9578            * reservation conflict. If the TUR fails for other
   9579            * reasons, SD_NOT_READY_VALID will be returned.
   9580            */I'm not a SCSI expert by any stretch of the imagination, but it seems to me that either Solaris is misinterpreting the "EACCES" error code or EMC is mis-using it. Could be that there's no good code in the SCSI protocol to say "Yea, I'm here, but I won't talk to you right now" except this one.
This exists in the "sd_ready_and_valid" function which appears to be called any time I/O is attempted to the device if the geometry of the device is not yet known (which it never will be for these standby devices).
So, in summary, if your system can "see" a SCSI LUN which responds to a Test Unit Ready (TUR) with "EACCES" then you're going to be seeing these messages. I think your only hope it to check your systems carefully and find out what process is trying to do I/O to them and fix that process.
Best of luck!