Wireless APs going down randomly

Question

Hi All
We have a situation where wireless access points are just going down randomly on our network. The situation is as follows:
We have 2 Cisco 5508 WLCs (licensed for 250 APs each, with more licenses en-route), both running software version 7.2.103.0. Then we have a large-ish number (±300) of Cisco AIR-CAP3502I-E-K9 APs connecting to these two controllers. The APs are either connected to Cisco WS-C2960S switches, or in "older" areas of the network they are connected to WS-C2950 switches. We do not have PoE switches installed yet, so all APs are powered by power injectors (AIR-PWRINJ4=).
Anyhat, all the access points are in FlexConnect (previously HREAP) mode, and they all connect via trunk ports to the switches in order to get the VLAN per SSID working. Both controllers are in the same mobility group, and both are in the DNS as "cisco-capwap-controller" as they are load balancing. The APs are installed about 25m apart in most areas. There is one installation where two APs are about 10m apart, but this is due to the shape of the building in which they are installed. We have no autonomous APs deployed at all...
The problem is that some access points work for about a day, and then they just disassociate with whichever controller they were associated to, and then they just fall off the network completely. Both WLCs, the NCS, and the network monitoring system (WhatsUp Gold) show the affected APs as physically unreachable. Looking at the trunk port on the switch, it shows that the port is up and the line protocol is up, input rate is 0 bits/sec, and output rate is up to 70,000bits/sec. There are no input or output errors on the switch port. Here is the output from one such switch port:
FastEthernet0/19 is up, line protocol is up (connected)
  Hardware is Fast Ethernet, address is dc7b.9439.da13 (bia dc7b.9439.da13)
  Description: Link to NC_K22-AP1
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:50:20, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 71000 bits/sec, 91 packets/sec
     977358 packets input, 80661413 bytes, 0 no buffer
     Received 29925 broadcasts (16038 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 16038 multicast, 0 pause input
     0 input packets with dribble condition detected
     37567889 packets output, 3624164243 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Connecting a console cable to an affected AP is fruitless, as the AP appears to be unresponsive while it is "down". Rebooting the AP, then connecting a console cable works, but then you've lost your syslogs. Looking at the Ethernet interface on the AP, it appears as if the interface is picking up a lot of errors, especially CRC errors, which would indicate a physical cable problem.
I got our cabling contractor out to do a cable test again, and it passes every time. We have them test both the Systimax and the Molex cable installations of affected APs, and all is OK. They use a Fluke MicroScanner² cable tester which certifies the Cat 6 cable from patch lead to flylead. No problems there, and I trust that the cable is 100% as tested.
So I swap the power injector of an affected AP (let's call it AP A) with an AP that is not affected by this (let's call this one AP B), and AP A goes down again after a day, or sometimes only a couple of hours. AP B remains operational as usual. If I swap AP A out with a new AP (AP C), then AP C starts showing the same symptoms after about a day, and then it also just goes down. When "suspect" AP A is installed somewhere else, then it works and stays operational for, well, 58 days now. Changing the switch/switch port also makes no difference.
This happens to one of the APs that are part of the "10m apart deployment", but it also happens to another AP that is completely on its own in a building far away from any other wireless source, so I cannot think that this could be the problem. I happens to an AP that is less than 100m away from its closest WLC, and it happens to an AP that is more than 350km away from its closest controller. All the other APs deployed in our remote campus (350km fibre link) are 100% OK. It happens to APs connected to the older WS-C2950 switches as well as the brand new WS-C2960S switches.
I am at wits end with this one. Where could the problem be? I trust the "Big Green" company that does our cable installations, and I trust the equipment they use. I also trust both Systimax and Molex to be very good products. And I do trust Cisco, as this is the only network equipment we deploy.
Where to from here?
Regards
Connie

Answer 1

This issue has now been resolved. Details can be found here: http://www.cisco.com/en/US/partner/ts/fn/635/fn63537.html.
Here is the short version:
Problem Description:
Calibration between DRAM and processor on AIR-CAP350x and AIR-LAP126x results in Access Points (APs) failing to boot properly. One symptom includes a white LED and no console output.
Background:
The root cause is the DDR memory calibration process that was performed on the Access Points (APs) at the time of manufacture. The fix recalibrates the existing memory hardware, so it will operate more reliably.
Workaround/Solution:
The fix for this issue is now publicly available. Go to www.cisco.com to obtain Access Point (AP) IOS code version: 15.2(2)JA or later and/or WLC software version: 7.0.235.3 (or later 7.0 release), 7.2.111.3 (or later 7.2 release) or 7.3.101.0 (or later 7.3 release).
As of June 2012, new AP3500 and AP1260 products with version ID (VID) V02 are free of this issue.