Cisco APs not updating after WLC-update

Question

Hello everyone,
I need to update my 5508 WLCs to a newer software, to support new AP-models.
Started with AIR-CT5500-K9-1-7-0-0-FUS.aes and AIR-CT5500-K9-7-0-240-0.aes, everything worked fine. Pre-Downloaded the newer Image to the APs, restarted the WLCs and everything was ok.
Now I tried to update to 7.6.100.0 as well as 7.4.121.0. Both Versions should support my APs, but it doesn't work at all.
Any ideas are highly appreciated.
If you need further output, just let me know.
Regards,
Manuel
These are some informations about the environment, AP-info and logging after "upgrade" to 7.4.121.0, controller-information after downgrading again...
AP#  sh ver
Cisco IOS Software, C1240 Software (C1240-K9W8-M), Version 12.4(23c)JA7, RELEASE SOFTWARE (fc1)
ROM: Bootstrap program is C1240 boot loader BOOTLDR: C1240 Boot Loader (C1240-BOOT-M) Version 12.4(13d)JA, RELEASE SOFTWARE (fc2)
AP uptime is 1 minute System returned to ROM by power-on System image file is "flash:/c1240-k9w8-mx.124-23c.JA7/c1240-k9w8-mx.124-23c.JA7"
cisco AIR-LAP1242AG-E-K9  (PowerPCElvis) processor (revision A0) with 27638K/5120K bytes of memory.
Processor board ID FCZ1545812F
PowerPCElvis CPU at 262Mhz, revision number 0x0950
Last reset from power-on LWAPP image version 7.0.240.0
1 FastEthernet interface
2 802.11 Radio(s)
32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 70:CA:9B:07:86:B8
Part Number                          : 73-10256-07
PCA Assembly Number                  : 800-26918-06
PCA Revision Number                  : A0
PCB Serial Number                    : FOC15402NP4
Top Assembly Part Number            : 800-29152-03
Top Assembly Serial Number          : FCZ1545812F
Top Revision Number                  : A0
Product/Model Number                : AIR-LAP1242AG-E-K9
Configuration register is 0xF
AP#dir
Directory of flash:/
2  -rwx      89311  Jan 18 2014 20:41:00 +00:00  event.log
3  drwx          64  Jan 18 2014 20:43:21 +00:00  update
5  drwx        256  Jan 18 2014 20:40:55 +00:00  c1240-k9w8-mx.124-23c.JA7
4  -rwx        6168  Nov 2 2011 23:32:18 +00:00  private-multiple-fs
7  -rwx        395  Mar 1 2002 00:00:05 +00:00  env_vars
15740928 bytes total (8772096 bytes free)
AP#dir
Directory of flash:/c1240-k9w8-mx.124-23c.JA7/
9  -rwx      131328  Jan 18 2014 20:39:46 +00:00  7101.img
10  -rwx        292  Jan 18 2014 20:39:46 +00:00  info
11  -rwx    4642714  Jan 18 2014 20:40:55 +00:00  c1240-k9w8-mx.124-23c.JA7
15  -rwx      131328  Jan 18 2014 20:40:56 +00:00  6701.img
#sh logging --> see attached file
CONTROLLER (unfortunately after downgrading it again):
(Cisco Controller) >show sysinfo
Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 7.0.240.0
Bootloader Version............................... 1.0.16
Field Recovery Image Version..................... 7.0.112.21
Firmware Version................................. FPGA 1.7, Env 1.8, USB console 2.2
Build Type....................................... DATA + WPS
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
IP Address....................................... WLC-IP
Last Reset....................................... Software reset
System Up Time................................... 0 days 0 hrs 26 mins 3 secs
System Timezone Location......................... (GMT +1:00) Amsterdam, Berlin, Rome, Vienna Current Boot
License Level....................... base
Current Boot License Type........................ Permanent
Next Boot License Level.......................... base
Next Boot License Type........................... Permanent
Configured Country............................... DE  - Germany
State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 2
Number of Active Clients......................... 5
Burned-in MAC Address............................ 1C:DF:0F:C6:D8:80
Power Supply 1................................... Present, OK
Power Supply 2................................... Absent
Maximum number of APs supported.................. 150
(Cisco Controller) >show boot
Primary Boot Image............................... 7.4.121.0
Backup Boot Image................................ 7.0.240.0 (default) (active)
(Cisco Controller) >show ap bundle primary
Primary AP Image        Size
ap1g2                  9576
ap3g1                  6684
ap3g2                  11208
ap801                  5192
ap802                  5232
c1100                  3096
c1130                  4972
c1140                  4992
c1200                  3364
c1240                  4812
c1250                  5512
c1310                  3136
c1520                  6412
c3201                  4324
c602i                  3716
(Cisco Controller) >show ap bundle secondary
Secondary AP Image      Size
ap3g1                  6684
ap801                  5192
ap802                  5232
c1100                  3096
c1130                  4972
c1140                  4992
c1200                  3364
c1240                  4812
c1250                  5512
c1310                  3136
c1520                  6412
c3201                  4324
c602i                  3716
Nachricht geändert durch Manuel Sporleder

Answer 1

Hi Scott,
I am not trying to pre-download it anymore, since this doesn't work at all.
If I just restart the controller, the APs are downloading the image telling me "everything is fine", are rebooted and that stated with the old image again.
This is what you can see in the attached log-file:
*Mar  1 00:00:05.873: soap_prepare_new_image_crash: mini ios flash:/c1240-rcvk9w8-mx/c1240-rcvk9w8-mx
*Mar  1 00:00:06.242: %SOAP_FIPS-2-SELF_TEST_IOS_SUCCESS: IOS crypto FIPS self test passed
*Mar  1 00:00:07.662: %SOAP_FIPS-2-SELF_TEST_RAD_SUCCESS: RADIO crypto FIPS self test passed on interface Dot11Radio 0
*Mar  1 00:00:09.054: %SOAP_FIPS-2-SELF_TEST_RAD_SUCCESS: RADIO crypto FIPS self test passed on interface Dot11Radio 1
*Mar  1 00:00:09.152: %LWAPP-3-CLIENTEVENTLOG: Read and initialized AP event log (contains, 1024 messages)
*Mar  1 00:00:09.181:  status of voice_diag_test from WLC is false
*Mar  1 00:00:11.381: %LINK-3-UPDOWN: Interface FastEthernet0, changed state to up
*Mar  1 00:00:11.440: %SYS-5-RESTART: System restarted
*Mar  1 00:00:11.441: %SNMP-5-COLDSTART: SNMP agent on host AP is undergoing a cold start
*Nov  2 23:31:59.107: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
*Nov  2 23:31:59.108: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Nov  2 23:31:59.929: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0, changed state to up
*Nov  2 23:32:00.107: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
*Nov  2 23:32:00.107: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*Nov  2 23:32:18.102: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY
*Nov  2 23:32:18.163: bsnUnlockDevice: not bring radio up: radio 1 is in admin disable state
*Nov  2 23:32:18.345: %SSH-5-ENABLED: SSH 2.0 has been enabled
*Nov  2 23:32:18.759:  status of voice_diag_test from WLC is false
*Nov  2 23:32:18.847: Logging LWAPP message to 255.255.255.255.
*Nov  2 23:32:33.181: %CDP_PD-4-POWER_OK: Full power - NEGOTIATED inline power source
*Nov  2 23:32:33.247: %LINK-3-UPDOWN: Interface Dot11Radio0, changed state to up
*Nov  2 23:32:34.212: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Nov  2 23:32:34.213: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 255.255.255.255 started - CLI initiated
*Jan 20 20:32:44.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: WLC-1-IP peer_port: 5246
*Jan 20 20:32:44.001: %CAPWAP-5-CHANGED: CAPWAP changed state to
*Jan 20 20:32:45.479: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: WLC-1-IP peer_port: 5246
*Jan 20 20:32:45.480: %CAPWAP-5-SENDJOIN: sending Join Request to WLC-1-IP
*Jan 20 20:32:45.481: %CAPWAP-5-CHANGED: CAPWAP changed state to JOIN
*Jan 20 20:32:45.483: %CAPWAP-3-ERRORLOG: Invalid event 10 & state 5 combination.
*Jan 20 20:32:45.483: %CAPWAP-3-ERRORLOG: CAPWAP SM handler: Failed to process message type 10 state 5.
*Jan 20 20:32:45.483: %CAPWAP-3-ERRORLOG: Failed to handle capwap control message from controller
*Jan 20 20:32:45.484: %CAPWAP-3-ERRORLOG: Failed to process encrypted capwap packet from WLC-1-IPperform archive download capwap:/c1240 tar file
*Jan 20 20:32:45.494: %CAPWAP-5-AP_IMG_DWNLD: Required image not found on AP. Downloading image from Controller.
*Jan 20 20:32:45.499: %CAPWAP-5-CHANGED: CAPWAP changed state to IMAGE
*Jan 20 20:33:58.755: %DTLS-3-BAD_RECORD: Erroneous record received from WLC-1-IP: Duplicate (replayed) record
*Jan 20 20:33:59.315: image upgrade successfully, system is now reloading
This happens again and again and again...
Regards, Manuel