Updated WLC has strange error log and AP's not joining

Question

Hi we recently updated all of our WLC's to 7.098 and it all went smoothly, controllers rebooted and AP's updated their firmware and rebooted OK.
One WLC (4402) which was working fine since the update now has no AP's associated. The AP's were all configured to run in HREAP mode and are on remote sites within our WAN. I have checked that all policies and ports are still open (none have changed anyway) but the AP's can not join with the contoller.
The log from an AP trying to join with the WLC.
*Mar  1 00:15:24.966: %CAPWAP-3-ERRORLOG: Did not get log server settings from DHCP.
*Mar  1 00:15:34.991: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Jan 12 02:17:56.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.96.4.17 peer_port: 5246
*Jan 12 02:17:56.000: %CAPWAP-5-CHANGED: CAPWAP changed state to
*Jan 12 02:18:17.447: %CDP_PD-2-POWER_LOW: All radios disabled - NON_CISCO-NO_CDP_RECEIVED  (0000.0000.0000)
*Jan 12 02:18:25.999: DTLS_CLIENT_ERROR: ../dtls/dtls_connection_db.c:2013 Max retransmission count reached!
*Jan 12 02:18:25.999: %DTLS-3-HANDSHAKE_RETRANSMIT: Max retransmit count for 10.96.4.17 is reached.
*Jan 12 02:18:56.000: %DTLS-5-SEND_ALERT: Send WARNING : Close notify Alert to 10.96.4.17:5246
*Jan 12 02:18:56.000: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY
*Jan 12 02:18:56.001: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY
*Jan 12 02:19:06.006: %CAPWAP-3-ERRORLOG: Go join a lwapp controller
*Jan 12 02:19:06.006: %LWAPP-3-CLIENTERRORLOG: Set Transport AddressCalled
*Jan 12 02:19:06.014: %LWAPP-5-CHANGED: CAPWAP changed state to JOIN
*Jan 12 02:19:11.013: %LWAPP-3-CLIENTERRORLOG: Join Timer: did not recieve join response (controller - ceo-wlc-01)
The logs on WLC show as below.
*emWeb: Jan 12 13:14:13.629: %AAA-5-AAA_AUTH_ADMIN_USER: aaa.c:1289 Authentication succeeded for admin user 'adann'
*spamReceiveTask: Jan 12 13:14:12.919: %LWAPP-6-CAPWAP_SUPP_VER: spam_lrad.c:1440 Discarding discovery request in LWAPP from AP 00:21:a0:81:a4:10 supporting CAPWAP
*spamReceiveTask: Jan 12 13:14:11.543: %LWAPP-6-CAPWAP_SUPP_VER: spam_lrad.c:1440 Discarding discovery request in LWAPP from AP 00:21:a0:81:8f:a0 supporting CAPWAP
*spamReceiveTask: Jan 12 13:14:11.395: %LWAPP-6-CAPWAP_SUPP_VER: spam_lrad.c:1440 Discarding discovery request in LWAPP from AP 00:24:14:ff:f1:70 supporting CAPWAP
*emWeb: Jan 12 13:14:10.731: %AAA-5-AAA_AUTH_ADMIN_USER: aaa.c:1289 Authentication succeeded for admin user 'adann'[...It occurred 2 times/sec!.]
*emWeb: Jan 12 13:14:09.459: %AAA-5-AAA_AUTH_ADMIN_USER: aaa.c:1289 Authentication succeeded for admin user 'adann'
*spamReceiveTask: Jan 12 13:14:09.457: %LWAPP-6-CAPWAP_SUPP_VER: spam_lrad.c:1440 Discarding discovery request in LWAPP from AP 00:24:14:ff:ec:00 supporting CAPWAP
Any suggestions would be appreciated!
Tony

Answer 1

Strange. Something must have happened that you didn't notice.
From the logs, it looks like only lwapp requests are arriving at the WLC. And WLC discards them cause it knows the AP can also do capwap so it's waiting for the capwap join packet.
As next step, I'd take a look at network traffic. Mostly close to the WLC where we want to know if we are receiving capwap discovery/join from the AP or not.
Nicolas