CAPWAP APs drop off the 7500 controller

I have a multiple 7500 flex controller deployed with over 2000 APs each on them and I notice that APs occasionally drop off.  When I find these APs I am able to telnet to them and I have found a fix for getting them back on the controller, but I want to know why this happens and if there is a way to avoid the problem.
Observed:
The APs have telnet enabled so I can get to the CLI.  Once in I do a dir command and see that there is little to no memory available (512 bytes to 0 bytes) in the flash memory.  I see that there are 5 large log files, file names are in the commands below.  When I do a show logging command I see the following over and over again
*Oct  3 20:31:44.102: %CAPWAP-3-ERRORLOG: Certificate verification failed!
*Oct  3 20:31:44.102: DTLS_CLIENT_ERROR: ../capwap/base_capwap/capwap/base_capwap_wtp_dtls.c:447 Certificate verified failed!
*Oct  3 20:31:44.102: %DTLS-5-SEND_ALERT: Send FATAL : Bad certificate Alert to 10.128.5.5:5246
*Oct  3 20:31:44.102: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.128.5.5:5246
*Oct  3 20:31:44.103: %CAPWAP-3-ERRORLOG: Invalid event 38 & state 3 combination.
*Oct  3 20:32:48.999: %CAPWAP-3-ERRORLOG: Selected MWAR 'tc-cl-wlc01'(index 0).
*Oct  3 20:32:48.999: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Oct  3 20:31:44.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.128.5.5 peer_port: 5246
*Oct  3 20:31:44.125: CRYPTO_PKI: New CRL Not Valid - expired (router time not synched to CA?)
*Oct  3 20:31:44.125:  CRL expires: 05:29:39 UTC Mar 3 2012
*Oct  3 20:31:44.125:  Router time: 20:31:44 UTC Oct 3 2013
*Oct  3 20:31:44.125: %PKI-4-CRLINSERTFAIL: Trustpoint "Trustpool2" unknown (error 1804:E_VALIDITY : validity period start later than end)Peer certificate verification failed 0059
To resolve:
The working theory is that the flash gets filled up with log files and is unable to download the certificate from the controller during the join process.  I delete the logs with the commands below and then do a wr mem and a copy run start and then reload.  This will fix the problem every time.
delete /force flash:ap_log_r0_0.log
delete /force flash:ap_log_r1_0.log
delete /force flash:ap_log_r0_1.log
delete /force flash:ap_log_r0_2.log
delete /force flash:ap_log_r1_1.log
delete /force flash:ap_log_r1_2.log
Other info
- currently running an engineering code of 7.3.113.12 on one 7500 and 7.4.110 on another, both seem to be having this issue.  I do not have this issue on a 5508 running 7.5 code.  Currently getting 7.4 vetted for deployment.
Good luck with this one

I have seen this issue, but only with older model access points and it doesn't have to be flexconnect and it doesn't matter what WLC code version your running.  Problamatic access points, I always check the flash to verify if there are logs or not, and do delete them in order to get the AP back up.  Again, I have only seen this with older non-802.11n access points.
Thanks,
Scott
Help out other by using the rating system and marking answered questions as "Answered"

Similar Messages

  • Discoverer Report whenever we create a new customer , the information is dropping off the Property manager module

    Dear Friends,
    whenever we create a new customer , the information is dropping off the Property manager module.
    Can you please help as its a very critical business issue now.
    Thanks a lot in advance.
    Kind Regards
    Siva Charan

    Dear Friends,
    whenever we create a new customer , the information is dropping off the Property manager module.
    Can you please help as its a very critical business issue now.
    Thanks a lot in advance.
    Kind Regards
    Siva Charan

  • HP Color LaserJet 1500L - drops off the 802.11n network

    My HP Color LaserJet 1500L drops off the network (it just stops printing). This happens when it is plugged into the usb bus of my new 802.11n base station or directly into the usb bus of a computer. The only way to fix it seems to be to plug it directly into a computer and unplug the usb plug and replug it, or to add the printer again. This problem has occurred in the past, but over time, and after various upgrades to the operating systems and hardware, etc, the situation tends to change. Before I installed the 802.11n base station the printer was staying in contact with my computers, but now it is misbehaving again. I doubt there is a real solution out there now, because it happened before and I never did discover a direct fix, but I thought the problem to be worth posting. This problem does not happen with my Canon PhotoPrinter iP6000S, which sticks to my network like glue whereever I plug it in (airport express, airport extreme, direct to computer, etc.) The HP loses contact with either of my computers in exactly the same way.
    Mac Pro Intel Core 2 Duo, and iMac Intel Core Duo   Mac OS X (10.4.8)   airport express, airport extreme 802.11n, cable modem
    iMac Intel Core Duo   Mac OS X (10.4.8)  

    To use the OS X standard network protocols (Windows/IP printing), you must have either a postscript printer (the driver in that case is just a ppd for the OS X postscript output) OR, a driver designed to work with CUPS (CUPS driver). HP doesn't provide CUPS drivers for OS X (who knows why, since they do for linux).
    You need to install foomatic and ESP ghostscript:
    http://www.linuxprinting.org/macosx/foomatic/
    then, install the foo2oak driver:
    http://www.linuxprinting.org/showprinter.cgi?recnum=HP-Color_LaserJet1500

  • Image dropping off the screen

    I am new to Photoshop (CC).  I have been using some tools successfully, but now when I try to use the Healing, Tool, Patch, Type and some others, the image drops off the screen as soon as I place the tool onto the screen. Then I cannot work on the image.  It just started today. Amy ideas???

    Usually any issue thats have to do with the displaying an image or any part of photoshop for that matter, is due to an out dated video driver. Try updating it and see if that fixes the issue.
    If the image is just too large to display, you can hold the ctrl key and the 0 on the numeric panel to zoom out to the full size of that image.

  • Officejet 4620 drops off the network after the first operation

    HP 4620 drops off the network soon after one operation is performed. I have to turn it off and on again to bring it on line and then it repeats. The message is " unable to communicate with printer".  I have run all the diognostics and they say your printer is operating normally.  I have down loaded the latest drivers. Running Win 7 (64 bit). My only clue is that I have a new router, Netgear.

    Hi LennieZup
    I understand you can print over the network one time and then you have to turn the printer off and on again to continue to print.
    I will be happy to assist you with this.
    It may not be the printer, it may be your router's inability to wake the printer.
    Disconnect the power cable on the router for 10 seconds, then reconnect the power cable again. You might need to check to see how many devices can be setup on the router at the same time. Depending on each router.
    I would recommend updating the printers firmware. HP Officejet Firmware.
    I have included a document for Printer Does Not Maintain Wireless Connection. Try Step six: Assign your HP product a static IP address.
    If you need more assistance just let me know.
    Please click “Accept as Solution ” if you feel my post solved your issue, it will help others find the solution.
    Click the “Kudos Thumbs Up" on the right to say “Thanks” for helping!
    Gemini02
    I work on behalf of HP

  • TS3899 I have an iPhone 5 and it shows on the Mail Icon that I have over 3,000 emails to read but the number continues to increase even when I have read the email.  I thought  once you have read an email it drops off the count of "new emails"

    I have an iPhone 5 and it shows on the Mail Icon that I have over 3,000 emails to read but the number continues to increase even when I have read the email.  I thought  once you have read an email it drops off the count of "new emails"

    The received emails should show in two locations:
    1. In the All Inboxes tab of Mail.
    2. The Inbox for each mail account,
    If your emails are not showing in these locations try quitting the mail app and then reset your iPhone.
    Go to the Home screen and double click the Home button. That will reveal the row of recently used apps at the bottom of the screen. Tap and hold on the app in question until it wiggles and displays a minus sign. Tap the minus sign to actually quit the app. Then tap anywhere on the screen above that bottom row to return the screen to normal.
    To reset your device. Press and hold the Home and Sleep buttons simultaneously ignoring the red slider should one appear until the Apple logo appears. Let go of the buttons and let the device restart. See if that fixes your problem.

  • Machines 'dropping off' the render queue

    Hi guys,
         We have Windows 2008 server as our centralized storage.  All the render machines have the shared folders mapped correctly with all the permissions set.  We are using After Effects CS5.  What we are finding is that after a few comps have been rendered in a multi-comp project, some will drop off the render queue.  What happens is that the render window list disappears and the watch window pops up and keeps cycling as if it's 'lost' the project is supposed to render.  Usually, if we reboot or restart the render engine and go back to the watch folder, it "might" pick up the project and carry on rendering.  All the machines are on Windows 7 64-bit.
    Any suggestions?
    Jason

    This is a known state, but not directly AE's fault - one of your machines is not able to access footage items within a specific limit and this effectively also shuts down other machines because they, too, go into this waiting forever state. Ultimately it boils down to too many machines trying to access the same files or directories and clogging up your network. The only way to avoid it is to ramp up your network bandwidth or e.g. instead of working with compound movie clips, work with image sequences.
    Mylenium

  • My fix for the AE dropping off the network in client mode

    Turn off any encryption.
    I have the linksys/vonage/wireless G router, which supports WPA. The AE will sometimes join the network with WPA but it takes anywhere from 1-5 minutes for it to do so. When finally on the network it shows up on the Airport manager list and in iTunes. The most playtime I could get out of it was 15 mins, before it would drop from the network. The light on the AE would be solid green, but I couldn't ping it. The only solution was to power cycle the AE. Note this is dropping w/out running the microwave, cordless phone, verifying my network channel is different from my neighbors, etc.
    I tried fiddling with WEP but the only way I could get that working was to specify the full key in hex, appearantly Apple and Linksys don't agree on the way to take a passphrase and turn it into a key.
    So I dropped to the unencrypted network, disabled the SSID broadcast and restricted wireless access to my machines. Works great now. :/ If I run the microwave and the cordless phone at the same time I can get Airtunes to drop, but it comes right back when the microwaving is done.
    Very frustrating, and the part that is the most irritatating is that my mac laptop and iMac can get on the network just fine.
    Still deciding if I want to keep it this way or say nevermind and return the AE.

    I found the solution after many many many attemps.
    It's not a MBP problem, it's your wireless router.
    1- Set your BEACON between 20 and 50. D-LINK router for example use 100 has default value. It's too long.
    2- Change your channel. D-link default is 6. Use whatever channel. I use 10.
    Hope you enjoy!

  • Why does my ipad drop off the Internet frequently?

    Why does my ipad drop ff the Internet frequently? It will be going fine then just won't connect

    Could be interference from something near you like a cordless phone, etc.

  • I have a number of playlists configured to play songs I haven't heard in a while. They then drop off the playlist once they've played. When I use my iPhone, however, the songs stay on the iPhone version of the playlist. Why aren't they syncing properly?

    I don't like Apple's algorithm that constantly pushes the same songs at me, so I have a couple of playlists configured for songs I haven't listened to in a month, two months, etc. This has worked fine for years. Since I've gotten my iPhone 4S, however, things are goofy.  The songs are marked as played and fall off of the playlist on iTunes, but they stay on the playlist on the iPhone, which means they come up again and again in the shuffle of that playlist. Even if I sync the iPhone the songs are still on the version of the playlist on the phone and I can't figure out a way to "refresh" that playlist and get rid of them. I know this is my own particular quirk about my music, but it's worked with every other iPod I've had. I don't know why the iPhone is holding on to a different version of the playlist than the one on iTunes. Is there a setting I'm missing somewhere?
    My iTunes is on a MacBook running Snow Leopard.

    Lynn,
    I had a very similar issue (same playlist).  After several frustrating hours of various combinations I was able to sync by changing the options to "manuall sync.." and then "dragging and dropping" the needed playlist from the iTunes Playlist list to the iPhone button (on my itunes setup/view that was from the far left column list of all of my  playlist in iTunes to the far upper right "iPhone" button).  To sum up "checking" the playlist and syncing didnt work, "drag and droping" the playlist did... at least for now... then I unselected the "manually manage" in the options and re-synced and all playlist were synced correctly.  Good Luck.

  • My ipad keeps dropping off the internet unless it is beside the router.

    I have tried in a friend's house but it does the same there, so it is an ipad problem not my router.
    Tried turning everything off etc but no good.

    Hello there, phillippa272.
    Good job with testing on another network. The following Knowledge Base article offers some standard troubleshooting steps as well as some additional information, that you should find helpful:
    iOS: Troubleshooting Wi-Fi networks and connections
    http://support.apple.com/kb/ts1398
    Thanks for reaching out to Apple Support Communities.
    Cheers,
    Pedro.

  • My Ipad dropped off the sofa and now I cannot get the screen to appear - it sometimes shows up bright yellow and green.  What can I do to reset? or whatever?

    I cannot get the screen to appear - can see if it's on or off.  There are times when the screen is bright bright green and yellow then disappears.  Can I reset it?  if so how?

    It will cost you approximately US$219.00 to US$299.00 depending on the model.
    Apple will replace the entire iPad; they don't repair.
    http://support.apple.com/kb/index?page=servicefaq&geo=United_States&product=ipad

  • I am having problems using Firefox with Ebay in the last 10 days. The advanced search link is dropping off the page and there is no way to access it now.

    I have been using Firefox and Ebay for at least 12 years with no problems whatsoever. But in the last 10 days I have been having trouble with the Advanced Search link disappearing from the page on Ebay. If I do a search from the Ebay home page it will show the results, but the Advanced Search link (in blue letters) which always used to appear to the right of the Search button is gone. If I click Search again at this point I can see the Advanced Search link appear for a millisecond, but then disappear just as quickly. Also, if I refresh the page the same thing happens. I tried to duplicate this problem with Internment Explorer and it didn't occur, so it appears to be Firefox that is causing this issue. Of course when I contacted Ebay they told me my problem was that I wasn't using Internment Explorer! I love Firefox and do not want to switch!

    You can try to clear the cache and cookies to see if that makes a difference:
    <h3>Clearing Cache</h3>
    #Go to Firefox > History > Clear recent history or (if no Firefox button is shown) go to Tools > Clear recent history.
    #Under "Time range to clear", select "Everything".
    #Now, click the arrow next to Details to toggle the Details list active.
    #From the details list, check ''Cache'' and uncheck everything else.
    #Now click the ''Clear now'' button.
    <h3>Clearing Cookies</h3>
    #Go to Firefox > Tools > Privacy > Remove Individual Cookies
    #Find the site (ebay.com) that's having problems and clear all cookies.
    #Restart Firefox
    Further information can be found in the [[Clear your cache, history and other personal information in Firefox]] article.

  • Since updating my system to Lion my HP MFP scanner has dropped off the map and I cannot print anymore. Any suggestions?

    Since upgrading my operating system to Lion my HP mfp/scanner has become almost unusable.  I understand an update  is coming.  Any suggestions about what I can do with what I have until then?

    As to what you can do until HP puts out an updated driver for your printer to use Lion, about your only recourse is to delete Lion and reinstall Snow Leopard or purchase a new printer for which a Lion compatible printer driver already exists.  As for your current printer, there is no such thing as a workaround when a compatible driver for a given OS is not available.
    I am assuming in the above comments that you have already visited the HP Printer Suuport site and found that HP has NOT yet released a Lion compatible driver for your printer.  If you have not done this yet, please proceed to do so and that should solve your problem.
    Hope this helps
    Message was edited by: Radiation Mac

  • Why does my mac mini server keep dropping off the network?

    I have Verizon FiOS and use its router/wireless to connect all macbooks, macbook pros, etc. connect to each other and the snow leopard server, which runs the printers and time machine for one of the laptops.  I'm not using any other heavy duty server apps. I just want the mac mini to stay on the network and do its job.  Invariably when I go down stairs to the mac mini and ethernet to the router to share screen, I find some update that needs to be installed, which apparently is why the mac mini has gone on strike.  I usually can see the problem when I go to connect to server and see the other macbooks, but not the mac mini.  Then after much coaxing, I'm able to get the server back on line for a few days, until the next software update.  The server is set to auto restart, but why is it balking at rejoining the network?
    mac mini is running snow leopard, as are three of the four macbooks. router is an actiontec model.  wireless network is WPA2 and mac mini is ethernet connected to the router.

    I've just experienced this problem when using a Marantz SR4001 receiver between my Mac Mini and HD TV. I noticed that the problem didn't happen all the time and I have found a workaround solution.
    The problem is not a fault, but seems to be caused by the receiver defaulting the HDMI video properties. It's an old unit so I'm thinking that 720p is a default resolution for the receiver.
    I can make the problem happen if the receiver is turned on before the TV. Seems that without an output connection the receiver is telling the mac that it expects 720p.
    If I turn on the TV before the receiver, the TV seems to tell the receiver that it is expecting 1080p 25fps and the receiver obliges and the Mac retains those display properties.
    So it appears that the video output device dictates the output properties back to the receiver and the receiver in turn back to the Mac. In absence of an output device the receiver defaults to 720p.
    Solution: Turn on the TV before the receiver.

Maybe you are looking for

  • Message no. TPM_TRAC1209 (Charge / Tax from 30.10.2009 was fixed (test run)

    Good morning people. We are having the following issue when running T-code TBB1. We created 2 additional movements for Money Market products. If we DON´T include these 2 movements in Money Market contract and then run TBB1 we don´t have any issue, bu

  • Cannot select buttons in FileMaker after Mountain Lion Install

    Because the installer claimed I'd tinkered with my partitions (I haven't, unless accessing BootCamp from Parallels counts), I had to do a clean install of Mountain Lion. That in itself was a pain because I could not successfully create an installer o

  • Charge overhead based on Item

    We are on 11.5.10. We are using average costing. Towards freight cost, we defined overhead elements. This is working Okay. For some items, we do not want freight cost to be added to the item cost. What options are available to control overhead elemen

  • Motion keeps on crashing after startup on MBP 2,33

    there have been a couple of post of motion crashing after launching - I thought the new Apple Software Update 1.1 would fix it, but now I had to realize that is only for mbp 2,2 or 2,4 - so what do I do with my mbp 2,33? Is there hope out there that

  • 6720c unwanted E-mail setup dialog

    I'm using my 6720c roaming on foreign networks, no problem except the phone from time to time displays an E-mail Setup dialog: "Allow application to use network to send or receive data?". I already have email accounts set up on the phone, but I can't