*** WARNING *** Upgrade to 5.0 may require isolating LWAPs from the WLC

Question

Before you run into the problem that we ran into when one of our larger customers upgraded from 4.2 to 5.0 and the result was that the controllers got caught in a loop during the upgrade and took the entire WLAN down.
Please be advised that Cisco recommends that you ISOLATE all lightweight access points from their controller during the upgrade of the controller from 4.2 to 5.0.
In part, this is due to a bug in 4.2.xx that starts upgrading the LWAP software images before the controller finishes updating/rebooting. This can cause interruption to the LWAPs as they upgrade their software from the controller when it reboots.
I have also heard of some situations where the LWAPs get corrupted to such a degree where you may have to out and physically access/reset the LWAPs that were interrupted during the upgrade process. I believe the worst-possible-case for this would be limited to no more than 10 LWAPs per controller
that can upgrade at a time)
In our case, the result was that the upgrade process hung up - with some LWAPs stuck, and all other LWAPs unable to pass traffic through the upgraded (ver 5.0) controller because they were at the previous revision (4.2).
In our case, this resulted in a system-twide WLAN outage.
As part of the call into Cisco TAC, the engineer said, in effect, 'why didn't you ignore the release notes instructions and isolate your controller from the LWAPs during the upgrade?'.
Our response was, we would have done so if it had been in the release notes (which it is not). So, after much cajoling by us, TAC finally agreed that it might be a good idea to include this additional step in the release notes.
Please reference the bug which requests an update to the 5.0 release note installation instructions: CSCso02420
TAC is aware of this problem and it is my understanding that there is discussion at Cisco regarding a maintenance release for 4.2 (since 4.2 is where the problem actually lies). The maintenance release would be applied to 4.2 controllers PRIOR to upgrading them to 5.0.
In the meantime, if you are still anxious to upgrade to 5.0, one workaround is to isolate the LWAPs from the controller either by:
1) Physically disconnecting the controller from the network during the upgrade and reconnecting it after its upgrade/reboot is complete.
or
2) Temporarily disabling the VLAN that carries the LWAP traffic to/from the LWAPs/Controller and re-enabling the VLAN after the upgrade/reboot of the controller.
I hope that this is helpful for anyone out there who is about to upgrade to 5.0.
I hope your upgrade goes smoother than our customer's did.
- John
(Please remember to rate helpful posts)

Answer 1

FYI - I noticed these messages in the message log on a 4.0.179.11 controller, sourced from 4.2 controllers.
Fri Mar 14 17:14:36 2008 [ERROR] ethoip.c 106: Eth-over-IP pkt too short; pkt size=82, expected min=66
Fri Mar 14 17:14:36 2008 [ERROR] mm_listen.c 4157: Mobility packet IP-message IP mismatch: pkt: 172.20.48.68 msg: 215.42.227.2
Fri Mar 14 17:14:36 2008Fri Mar 14 17:14:33 2008 [ERROR] ethoip.c 305: ethoipSocketTask: ethoipRecvPkt returned error
Fri Mar 14 17:14:33 2008 [ERROR] ethoip.c 106: Eth-over-IP pkt too short; pkt size=82, expected min=66
Fri Mar 14 17:14:32 2008 [ERROR] ethoip.c 305: ethoipSocketTask: ethoipRecvPkt returned error
Fri Mar 14 17:14:32 2008 [ERROR] ethoip.c 106: Eth-over-IP pkt too short; pkt size=82, expected min=66
Fri Mar 14 17:14:32 2008 [ERROR] mm_listen.c 4157: Mobility packet IP-message IP mismatch: pkt: 172.20.48.6 msg: 215.42.227.24
Fri Mar 14 17:14:32 2008Fri Mar 14 17:14:29 2008 [ERROR] ethoip.c 305: ethoipSocketTask: ethoipRecvPkt returned error
Fri Mar 14 17:14:29 2008 [ERROR] ethoip.c 106: Eth-over-IP pkt too short; pkt size=82, expected min=66
Fri Mar 14 17:14:26 2008 [ERROR] ethoip.c 305: ethoipSocketTask: ethoipRecvPkt returned error
Fri Mar 14 17:14:26 2008 [ERROR] ethoip.c 106: Eth-over-IP pkt too short; pkt size=82, expected min=66
Fri Mar 14 17:14:23 2008 [ERROR] ethoip.c 305: ethoipSocketTask: ethoipRecvPkt returned error
Fri Mar 14 17:14:23 2008 [ERROR] ethoip.c 106: Eth-over-IP pkt too short; pkt size=82, expected min=66
Fri Mar 14 17:14:22 2008 [ERROR] ethoip.c 305: ethoipSocketTask: ethoipRecvPkt returned error
Fri Mar 14 17:14:22 2008 [ERROR] ethoip.c 106: Eth-over-IP pkt too short; pkt size=82, expected min=66
Fri Mar 14 17:14:19 2008 [ERROR] ethoip.c 305: ethoipSocketTask: ethoipRecvPkt returned error
Fri Mar 14 17:14:19 2008 [ERROR] ethoip.c 106: Eth-over-IP pkt too short; pkt size=82, expected min=66
Fri Mar 14 17:14:16 2008 [ERROR] ethoip.c 305: ethoipSocketTask: ethoipRecvPkt returned error
Fri Mar 14 17:14:16 2008 [ERROR] ethoip.c 106: Eth-over-IP pkt too short; pkt size=82, expected min=66
Fri Mar 14 17:14:13 2008 [ERROR] mm_listen.c 4157: Mobility packet IP-message IP mismatch: pkt: 172.20.48.4 msg: 215.42.227.24