IPMP / ARP Cache oddity - Solaris to Windows comm errors

Question

First - a qualification - I'm not an Solaris admin, so feel free to call me out for any blatant errors..
I've got several Solaris 10 servers that are having intermittent network communication issues with Windows 2003 servers on the same subnet. All Solaris boxes are using two NICs and IPMP for their connections to the "primary" network. For example, one server (hostname bugbear) has two adapters ce0 and ce9:
# ifconfig -a
ce0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 16.106.64.227 netmask fffff800 broadcast 16.106.71.255
        groupname shared0
ce9: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
        inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
        groupname shared0
# cat /etc/hostname.ce0   
bugbear netmask + broadcast + group shared0 up
# cat /etc/hostname.ce9
group shared0 upIf I look at the arp cache, (almost) every other server it talks to shows up twice - an entry for each of the NIC devices:
# arp -a
Net to Media Table: IPv4
Device   IP Address               Mask      Flags      Phys Addr
ce9    win2k3box.city.acme.com 255.255.255.255 o        00:50:56:ae:13:58
ce0    win2k3box.city.acme.com 255.255.255.255 o        00:50:56:ae:13:58I guess that's as expected - again, I'm not a Solaris expert.
Finally, here's the issue I've found. For any other UNIX/Solaris hosts listed in the arp cache, my server can ping/FTP/whatever to the other - no problem. Sometimes the ce0 interface is listed first in the output for the target server, sometimes the ce9 interface is listed first. Doesn't matter - my server can talk to the other just fine.
HOWEVER - for Windows servers, the order of entry in the arp cache seems to have relevance. If the ce9 interface is listed first, I can't ping, can't FTP, can't connect to the other server. If the ce0 interface is listed first, then everything works fine.
Eventually the arp cache entries age out and get deleted. Usually then my server can talk to the Windows server again. I also found that if I delete both arp cache entries for the target Windows server, my server can talk to it again (arp cache gets rebuilt with the ce0 interface). If left alone, the behavior seems to be that my server can talk to a Windows target for an hour or two, then it can't for another hour or two, then the cycle repeats.
NOTE - The mac address in the arp cache is not getting poisoned, as with the Broadcom windows driver issue that hit some folks (myself included). All the mac addresses for the servers in the arp cache are consistent as best I can tell, so that's not the problem.
Any input/suggestions greatly appreciated.

Answer 1

Here are the codes:
In the initializing stage,
     try {
          this.serialPort = (SerialPort) portId.open("SimpleReadApp", 2000);
     } catch (PortInUseException e) {
         e.printStackTrace();
     this.serialPort.notifyOnDataAvailable(false);
     try {
        this.serialPort.setSerialPortParams(9600, SerialPort.DATABITS_8, SerialPort.STOPBITS_1, SerialPort.PARITY_NONE);
        this.serialPort.setFlowControlMode(SerialPort.FLOWCONTROL_NONE);
        this.serialPort.enableReceiveTimeout(10);
//this.serialPort.disableReceiveTimeout();
//this.serialPort.enableReceiveThreshold(0);
        this.serialPort.enableReceiveThreshold(1*1024*1024);
        if (this.serialPort.getInputStream() != null)
                this.inputStream = this.serialPort.getInputStream();
        else
                System.out.print("can not get inpustream!\n");
      } catch (Exception e) {
              e.printStackTrace();
      }Then in the reading stage, we have:
    byte[] readBuffer = new byte[1 * 1024 * 10];
    while (true) {
          try {
               int numBytes = inputStream.read(readBuffer);
               System.out.print("There are " + numBytes + " bytes having been read.");
          } catch (Exception e) {
              e.printStackTrace();
          }The above codes do not have any problem using COMM API 2.0 on Windows. Or COMM API 3.0 on Solaris. But, on RedHat, it can run out of memory very quickly.
Edited by: EJP on 13/05/2011 11:27: added { code } tags. Please use them.