WLC 5508 in HA pair (7.4.121.0) sudden reload

Question

I have a pair of WLC 5508 in HA pair running version 7.4.121.0, last week I have two sudden reload on my active WLC. Here's the error from my syslog server on the first sudden reload. The second reload has almost the same logs.
10.x.x.234 - active
10.x.x.237 - standby
2014-01-30 17:52:20 Local0.Error 10.x.x.237 WLC-HA01: *rmgrMain: Jan 30 17:52:24.498: #RMGR-3-RED_HEARTBEAT_TMOUT: rmgr_main.c:242 rmgrTmoHeartbeat: Recved GW ping count 6 phyMgr ping count 0.
2014-01-30 17:52:20 Local0.Emerg 10.x.x.237 WLC-HA01: *rmgrMain: Jan 30 17:52:24.555: #RMGR-0-RED_HA_RELOAD: rmgr_utils.c:198 System reboot: reason: category Sanity check object Self
2014-01-30 17:52:21 Local0.Emerg 10.x.x.234 WLC-HA01: *rmgrMain: Jan 30 17:52:24.989: #RMGR-0-RED_HA_RELOAD: rmgr_utils.c:188 System reboot: reason: category Peer reload req object Peer
2014-01-30 17:52:21 Local0.Alert 10.x.x.234 WLC-HA01: *dtlArpTask: Jan 30 17:52:25.106: #DTL-1-IP_CONFLICT_DETECTED: dtl_net.c:4857 Network device with mac addr 7c:ad:74:8d:6b:0f using IP address of local interface
Cisco TAC recommends to disable monitoring the default gateway.
--> config redundancy management-gateway-failover disable
I was wondering if someone has the issue with what I have.
Second issue I have is when it fails over to the standby WLC, I do get a web-auth certificate error from the WLC when clients login. This only happens after a sudden reload. If I do a redundancy force-switchover during maintenance window, the certificate error doesn't show up. To fix the certificate error I have to bounce both WLCs one after the other.
Thanks in advance.

Answer 1

Hi,
I exeprienced a reload problem in standby WLC, with HA in release 7.6.100.0.
I use a dedicated VLAN to transport the redundancy sync and info, 'cause the two WLCs are in different buildings.
The standby WLC reload continuously 'cause it doesn,t find the default gateway.
(Cisco Controller-Standby) >show redundancy summary
            Redundancy Mode = SSO ENABLED
                Local State = STANDBY HOT
                 Peer State = ACTIVE
                       Unit = Secondary - HA SKU (Inherited AP License Count = 500)
                    Unit ID = 00:06:F6:DB:E3:E0
           Redundancy State = SSO (Both AP and Client SSO)
               Mobility MAC = 58:8D:09:CD:81:C0
Management Gateway Failover = ENABLED (Management GW failover would be operational in few moments)
Average Redundancy Peer Reachability Latency = 621 usecs
Average Management Gateway Reachability Latency = 0 usecs
Redundancy Management IP Address................. 40.231.36.6
Peer Redundancy Management IP Address............ 40.231.36.5
Redundancy Port IP Address....................... 169.254.36.6
Peer Redundancy Port IP Address.................. 169.254.36.5
Rebooting as default GW is not reachable from Standby Controller
Restarting system. Reason: Default Gateway is not reachable ..
The problem is that the WLC tries to ping the DGW using the primary IP management address belonging to the active WLC, so we have duplicated IP problem, ARP problem and so on .....
The standby WLC should use the redundancy managemet address to ping the default gateway, instead the primary IP management address!!!!!!
So the workaround is the CLI command :
config redundancy management-gateway-failover disable
on the primary WLC, via console or in SSH.
When the standby will reload it will inherit the config from the active primary WLC
(Cisco Controller-Standby) >show redundancy summary   
            Redundancy Mode = SSO ENABLED
                Local State = STANDBY HOT
                 Peer State = ACTIVE
                       Unit = Secondary - HA SKU (Inherited AP License Count = 500)
                    Unit ID = 00:06:F6:DB:E3:E0
           Redundancy State = SSO (Both AP and Client SSO)
               Mobility MAC = 58:8D:09:CD:81:C0
Management Gateway Failover = ENABLED (Management GW failover is disabled as it is DISABLED on the Peer)
Average Redundancy Peer Reachability Latency = 666 usecs
Average Management Gateway Reachability Latency = 0 usecs
Redundancy Management IP Address................. 40.231.36.6
Peer Redundancy Management IP Address............ 40.231.36.5
Redundancy Port IP Address....................... 169.254.36.6
Peer Redundancy Port IP Address.................. 169.254.36.5
The workaround works in my experience.