Old famous problem - AP LAP1142N can't join the controller - 4402, please participate, let's make a good guide!

Question

Hello folks,
I really feel sorry for bringing up this discussion again. I wouldn't dare to ask this question if I find someone's clear cut suggestion/solution or an overview giving a detailed step-by-step procedure. People just suggest jumping through so many  hoops like resetting the AP or converting it back to standalone mode and then back to LWAPP.
Hence I have so many questions and hope that we can make a good guide covering all possible problems.
1) AP was originally running a standalone image.  I booted it into a so-called ROMMON or AP mode (ESC is the right key to make it boot into this mode)
I found a recovery image in its flash - c1140-rcvk9w8-mx. I made the AP boot from it by using "set" command and I see that it start booting using this recovery image. Here goes the question. Do all AP settings matter ? E.g. when I run "set" command from AP I see the following:
ap: set
?=
DEFAULT_ROUTER=10.0.0.1
Default_router=10.9.99.1
ENABLE_BREAK=yes
IP_ADDR=10.0.0.1
IP_AddR=10.9.99.9
MANUAL_BOOT=no
NETMASK=255.255.255.224
NEW_IMAGE=yes
PWR_INJECTOR_DETECT=0016.c7fa.b394
RELOAD_REASON=9
ROM_PERSISTENT_UTC=1014941470
TERMLINES=0
netmask=255.255.255.0
2) How would do something like "write erase" or even recover the enable password while being in AP mode? Do I really need to do it ? What I see next makes me believe there's something with the AP configuration (particularly SSH) that prevents an AP join WLC.
3) The AP is powered on, connected to the switchport on the same L2 VLAN where WLC management interface. Then it boots and gets an IP address from the DHCP server located on the other switch.
*Mar  1 00:00:08.695: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY
*Mar  1 00:00:08.705: %CDP_PD-2-POWER_LOW: All radios disabled - AC_ADAPTOR  (0000.0000.0000)
*Mar  1 00:00:09.629: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Mar  1 00:00:17.534: %DHCP-6-ADDRESS_ASSIGN: Interface GigabitEthernet0 assigned DHCP address 192.168.1.122, mask 255.255.255.0, hostname AP2
Here comes the question, why do I see this on the console (pay attention at "transport input ssh" line)? Does it have anything to do  with an error for DTLS ?
*Apr 12 12:44:21.034: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY
*Apr 12 12:44:31.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 192.168.1.141 peer_port: 5246
*Apr 12 12:44:55.000: DTLS_CLIENT_ERROR: ../dtls/dtls_connection_db.c:1924 Max retransmission count reached!
*Apr 12 12:44:55.000: %DTLS-3-HANDSHAKE_RETRANSMIT: Max retransmit count for 192.168.1.141 is reached.
transport input ssh
^
% Invalid input detected at '^' marker.
4) Do I have to connect the controller ap-manager interface to the network or I can rely on the AP find the WLC via its management interface. I have never worked with 4400 series controllers. Just started with 5500 and they don't have the ap-manager interface.
Cisco guide says: "The management interface is also used for layer two communications between the WLC and access points", so I can safely assume that it is enough. Moreover I can ping the AP from the WLC while connected to WLC via SSH and its management interface.
5) And finally, what's wrong with the discovery? This is what I see on the controller while debugging capwap packets:
(Cisco Controller) debug>*spamReceiveTask: Apr 12 12:53:52.253: <<<<  Start of CAPWAP Packet  >>>>
*spamReceiveTask: Apr 12 12:53:52.253: CAPWAP Control mesg Recd from 192.168.1.122, Port 57046
*spamReceiveTask: Apr 12 12:53:52.253:          HLEN 4,   Radio ID 0,    WBID 1
*spamReceiveTask: Apr 12 12:53:52.253:          Msg Type   :   CAPWAP_DISCOVERY_REQUEST
*spamReceiveTask: Apr 12 12:53:52.253:          Msg Length : 29
*spamReceiveTask: Apr 12 12:53:52.253:          Msg SeqNum : 0
*spamReceiveTask: Apr 12 12:53:52.253:
*spamReceiveTask: Apr 12 12:53:52.253:   Type : CAPWAP_MSGELE_DISCOVERY_TYPE, Length 1
*spamReceiveTask: Apr 12 12:53:52.253:          Discovery Type : CAPWAP_DISCOVERY_TYPE_UNKNOWN
*spamReceiveTask: Apr 12 12:53:52.253:
*spamReceiveTask: Apr 12 12:53:52.253:   Type : CAPWAP_MSGELE_WTP_FRAME_TUNNEL, Length 1
*spamReceiveTask: Apr 12 12:53:52.253:          WTP Frame Tunnel Mode : NATIVE_FRAME_TUNNEL_MODE
*spamReceiveTask: Apr 12 12:53:52.253:
*spamReceiveTask: Apr 12 12:53:52.253:   Type : CAPWAP_MSGELE_WTP_MAC_TYPE, Length 1
*spamReceiveTask: Apr 12 12:53:52.253:          WTP Mac Type  : SPLIT_MAC
*spamReceiveTask: Apr 12 12:53:52.253:
*spamReceiveTask: Apr 12 12:53:52.253:   Type : CAPWAP_MSGELE_VENDOR_SPECIFIC_PAYLOAD, Length 10
*spamReceiveTask: Apr 12 12:53:52.253:          Vendor Identifier  : 0x00409600
*spamReceiveTask: Apr 12 12:53:52.254:
What discovery mode are we in? L2 or L3 ?

Answer 1

(Cisco Controller) >show sysinfo
Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 7.0.230.0
RTOS Version..................................... 7.0.230.0
Bootloader Version............................... 4.0.219.0
Emergency Image Version.......................... N/A
Build Type....................................... DATA + WPS
System Name...................................... Cisco_8b:83:03
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.14179.1.1.4.3
IP Address....................................... 192.168.1.140
System Up Time................................... 0 days 19 hrs 9 mins 20 secs
System Timezone Location.........................
Configured Country............................... CA  - Canada
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +41 C
--More-- or (q)uit
State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 1
Number of Active Clients......................... 0
Burned-in MAC Address............................ 00:23:5E:8B:83:00
Crypto Accelerator 1............................. Absent
Crypto Accelerator 2............................. Absent
Power Supply 1................................... Absent
Power Supply 2................................... Present, OK
Maximum number of APs supported.................. 12
"Show inventory" on the AP doesn't give any output. Giving you the "show version" instead omitting some legal stuff
AP2>sh ver
Cisco IOS Software, C1140 Software (C1140-RCVK9W8-M), Version 12.4(18a)JA, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by Cisco Systems, Inc.
Compiled Fri 21-Nov-08 01:28 by prod_rel_team
ROM: Bootstrap program is C1140 boot loader
BOOTLDR: C1140 Boot Loader (C1140-BOOT-M) Version 12.4(18a)JA, RELEASE SOFTWARE (fc4)
AP2 uptime is 1 hour, 42 minutes
System returned to ROM by reload
System image file is "flash:/c1140-rcvk9w8-mx/c1140-rcvk9w8-mx"
Last reload reason:
cisco AIR-LAP1142N-N-K9    (PowerPC405ex) processor (revision A0) with 98294K/32768K bytes of memory.
Processor board ID FTX1329S9NB
PowerPC405ex CPU at 586Mhz, revision number 0x147E
Last reset from reload
LWAPP image version 3.0.51.0
1 Gigabit Ethernet interface
32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 00:22:BD:18:ED:66
Part Number                          : 73-11451-06
PCA Assembly Number                  : 800-30554-03
PCA Revision Number                  : A0
PCB Serial Number                    : FOC13282UKM
Top Assembly Part Number             : 800-31273-01
Top Assembly Serial Number           : FTX1329S9NB
Top Revision Number                  : A0
Product/Model Number                 : AIR-LAP1142N-N-K9
Configuration register is 0xF
And the full boot process on WAP:
IOS Bootloader - Starting system.
Xmodem file system is available.
DDR values used from system serial eeprom.
WRDTR,CLKTR: 0x84000800, 0x40000000
RQDC, RFDC : 0x80000038, 0x0000020a
PCIE0: link is up.
PCIE0: VC0 is active
PCIE1: link is up.
PCIE1: VC0 is active
PCIEx: initialization done
flashfs[0]: 149 files, 8 directories
flashfs[0]: 0 orphaned files, 0 orphaned directories
flashfs[0]: Total bytes: 32385024
flashfs[0]: Bytes used: 7901696
flashfs[0]: Bytes available: 24483328
flashfs[0]: flashfs fsck took 16 seconds.
Reading cookie from system serial eeprom...Done
Base Ethernet MAC address: 00:22:bd:18:ed:66
Ethernet speed is 100 Mb - FULL duplex
Loading "flash:/c1140-rcvk9w8-mx/c1140-rcvk9w8-mx"...#############################################################################################################################################################################################################################
File "flash:/c1140-rcvk9w8-mx/c1140-rcvk9w8-mx" uncompressed and installed, entry point: 0x4000
executing...
enet halted
Cisco IOS Software, C1140 Software (C1140-RCVK9W8-M), Version 12.4(18a)JA, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by Cisco Systems, Inc.
Compiled Fri 21-Nov-08 01:28 by prod_rel_team
Image text-base: 0x00004000, data-base: 0x00430000
Proceeding with system init
Proceeding to unmask interrupts
Initializing flashfs...
flashfs[1]: 149 files, 8 directories
flashfs[1]: 0 orphaned files, 0 orphaned directories
flashfs[1]: Total bytes: 32385024
flashfs[1]: Bytes used: 7901696
flashfs[1]: Bytes available: 24483328
flashfs[1]: flashfs fsck took 4 seconds.
flashfs[1]: Initialization complete....done Initializing flashfs.
Ethernet speed is 100 Mb - FULL duplex
cisco AIR-LAP1142N-N-K9    (PowerPC405ex) processor (revision A0) with 98294K/32768K bytes of memory.
Processor board ID FTX1329S9NB
PowerPC405ex CPU at 586Mhz, revision number 0x147E
Last reset from power-on
LWAPP image version 3.0.51.0
1 Gigabit Ethernet interface
32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 00:22:BD:18:ED:66
Part Number                          : 73-11451-06
PCA Assembly Number                  : 800-30554-03
PCA Revision Number                  : A0
PCB Serial Number                    : FOC13282UKM
Top Assembly Part Number             : 800-31273-01
Top Assembly Serial Number           : FTX1329S9NB
Top Revision Number                  : A0
Product/Model Number                 : AIR-LAP1142N-N-K9
% Please define a domain-name first.
ip ssh version 2
^
% Invalid input detected at '^' marker.
transport input ssh
^
% Invalid input detected at '^' marker.
aaa new-model
^
% Invalid input detected at '^' marker.
aaa authentication login default local
^
% Invalid input detected at '^' marker.
login authentication default
^
% Invalid input detected at '^' marker.
transport input ssh
^
% Invalid input detected at '^' marker.
RS
Press RETURN to get started!
SI IDB null
RSSI IDB null
*Mar  1 00:00:05.992: *** CRASH_LOG = YES
Base Ethernet MAC address: 00:22:BD:18:ED:66
*Mar  1 00:00:06.203: %LWAPP-3-CLIENTEVENTLOG: Read and initialized AP event log (contains, 1024 messages)
*Mar  1 00:00:08.251: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Mar  1 00:00:08.292: %SYS-5-RESTART: System restarted --
Cisco IOS Software, C1140 Software (C1140-RCVK9W8-M), Version 12.4(18a)JA, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by Cisco Systems, Inc.
Compiled Fri 21-Nov-08 01:28 by prod_rel_team
*Mar  1 00:00:08.318: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY
*Mar  1 00:00:08.327: %CDP_PD-2-POWER_LOW: All radios disabled - AC_ADAPTOR  (0000.0000.0000)
*Mar  1 00:00:09.251: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Mar  1 00:00:17.157: %DHCP-6-ADDRESS_ASSIGN: Interface GigabitEthernet0 assigned DHCP address 192.168.1.123, mask 255.255.255.0, hostname AP2
logging origin-id string AP:0022.bd18.ed66
^
% Invalid input detected at '^' marker.
logging 255.255.255.255
^
% Invalid input detected at '^' marker.
logging trap 3
^
% Invalid input detected at '^' marker.
*Mar  1 00:00:27.230: %CAPWAP-3-ERRORLOG: Not sending discovery request AP does not have an Ip !!
*Mar  1 00:00:27.343: Logging LWAPP message to 255.255.255.255.
Translating "CISCO-LWAPP-CONTROLLER"...domain server (192.168.1.40)
*Mar  1 00:00:38.267: %CAPWAP-3-ERRORLOG: Did not get log server settings from DHCP.
*Mar  1 00:00:39.267: %CAPWAP-3-ERRORLOG: Could Not resolve CISCO-LWAPP-CONTROLLER
*Apr 12 15:39:49.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 192.168.1.141 peer_port: 5246