Cluster 3.2 Interconnect Weirdness

Question

I am installing Cluster 3.2 with Solaris 10 update 3 on a V210 and V240 and I am having an interconnect problem.
I have bge2 and bge3 on each node cabled back to back via a cross-over cable. I am able to plumb up the interfaces, assign a test IP and ping back and forth with no problem so I know the cables and ports are OK on each server.
After putting bge2 and bge3 down/unplumbing on each node I run scinstall from one of the nodes everything goes well until this point:
Cluster Creation
Log file - /var/cluster/logs/install/scinstall.log.blah
Checking installation status ... done
The Sun Cluster software is installed on "marge".
The Sun Cluster software is installed on "homer".
Started sccheck on "marge".
Started sccheck on "homer".
sccheck completed with no errors or warnings for "marge".
sccheck completed with no errors or warnings for "homer".
Configuring "homer" ... done
Rebooting "homer" ...
The node Homer boots into cluster mode but clintr status shows this:
=== Cluster Transport Paths ===
Endpoint1 Endpoint2 Status
clintr show:
--- Transport Adapters for homer ---
Transport Adapter: bge2
State: Disabled
Transport Type: dlpi
device_name: bge
device_instance: 2
lazy_free: 1
dlpi_heartbeat_timeout: 10000
dlpi_heartbeat_quantum: 1000
nw_bandwidth: 80
bandwidth: 70
Transport Adapter: bge3
State: Disabled
Transport Type: dlpi
device_name: bge
device_instance: 3
lazy_free: 1
dlpi_heartbeat_timeout: 10000
dlpi_heartbeat_quantum: 1000
nw_bandwidth: 80
bandwidth: 70
I am guessing that Marge will not reboot into the cluster because she cannot talk to Homer over the interconnects after he reboots.
I'm not using switches in my scinstall config, just good old network interfaces and crossover cables.
fconfig -a on Homer shows bge0 and bge1 as my public interfaces in IPMP configuration and clprivnet0 using the default 172.16 address range. bge2 and bge3 are nowhere to be found in the listing.
The physical ports on Homer are not lit anymore, almost like something is missing to plumb up the interface.
Do I need an /etc/hostname.bge2 and /etc/hostname.bge3 on each host with 'up' in it? Do I have to manually assign bge2 and bge3 to clprivnet0 (and if so , how)?
I'm missing something subtle here, any insight would be most appreciated.

Answer 1

Did a rebuild with current patches, did not run JASS or an in-house hardening script on the nodes and I get the same result.
I think I might have a bug issue with the bge driver or something.
On the first node before running scinstall all nic lights are lit.
After that first node reboots from scinstall the nic lights stay lit until right before the CMM messages begin appearing on the console. At that point the interconnect lights go out and stay out physically.
No errors were detected with sccheck or in the install logs for the cluster.
I tried a rebuild of the cluster nodes using a switch (ProCurve 2626) for the interconnects rather than an ethernet cable or cross-over cable.
I have a hme interface in my V210 and V240 and I am going to use that for one of the interconnects to see if it matters.
Basically at this point it is definitely not something physical (bad cable, bad switch port, etc) but something in the cluster configuration from scinstall that is not digging the interconnects and keeping the cluster nodes from conversing.
Since the cluster isn't working anyway I can do a clintr enable node:port,switch@port and see that the ports and switch ports show as enabled by clintr status does not show an interconnect active and the physical ports are not lit.
I do see references to bgeX/0 unregistered in /var/adm/messages but I haven't found information as to what this means or what to do about it exactly yet.
Closest thing so far is this:
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6453203