High CPU due to dispatch unit in cisco ASA 5540
Hi Any suggestion help
High CPU due to dispatch unit in cisco ASA 5540
ciscoasa# sh processes cpu-usage
PC Thread 5Sec 1Min 5Min Process
0805520c ad5afdf8 0.0% 0.0% 0.0% block_diag
081a8d34 ad5afa08 82.6% 82.1% 82.3% Dispatch Unit
083b6c05 ad5af618 0.0% 0.0% 0.0% CF OIR
08a60aa0 ad5af420 0.0% 0.0% 0.0% lina_int
08069f06 ad5aee38 0.0% 0.0% 0.0% Reload Control Thread
08072196 ad5aec40 0.0% 0.0% 0.0% aaa
08c76f3d ad5aea48 0.0% 0.0% 0.0% UserFromCert Thread
080a6f36 ad5ae658 0.0% 0.0% 0.0% CMGR Server Process
080a7445 ad5ae460 0.0% 0.0% 0.0% CMGR Timer Process
081a815c ad5ada88 0.0% 0.0% 0.0% dbgtrace
0844d75c ad5ad2a8 0.0% 0.0% 0.0% 557mcfix
0844d57e ad5ad0b0 0.0% 0.0% 0.0% 557statspoll
08c76f3d ad5abef8 0.0% 0.0% 0.0% netfs_thread_init
09319755 ad5ab520 0.0% 0.0% 0.0% Chunk Manager
088e3f0e ad5ab328 0.0% 0.0% 0.0% PIX Garbage Collector
088d72d4 ad5ab130 0.0% 0.0% 0.0% IP Address Assign
08ab1cd6 ad5aaf38 0.0% 0.0% 0.0% QoS Support Module
08953cbf ad5aad40 0.0% 0.0% 0.0% Client Update Task
093698fa ad5aab48 0.0% 0.0% 0.0% Checkheaps
08ab6205 ad5aa560 0.0% 0.0% 0.0% Quack process
08b0dd52 ad5aa368 0.0% 0.0% 0.0% Session Manager
08c227d5 ad5a9f78 0.0% 0.0% 0.0% uauth
08bbf615 ad5a9d80 0.0% 0.0% 0.0% Uauth_Proxy
08bf5cbe ad5a9798 0.0% 0.0% 0.0% SSL
08c20766 ad5a95a0 0.0% 0.0% 0.0% SMTP
081c0b4a ad5a93a8 0.0% 0.0% 0.0% Logger
08c19908 ad5a91b0 0.0% 0.0% 0.0% Syslog Retry Thread
08c1346e ad5a8fb8 0.0% 0.0% 0.0% Thread Logger
08e47c82 ad5a81f0 0.0% 0.0% 0.0% vpnlb_thread
08f0f055 ad5a7a10 0.0% 0.0% 0.0% pci_nt_bridge
0827a43d ad5a7620 0.0% 0.0% 0.0% TLS Proxy Inspector
08b279f3 ad5a7428 0.0% 0.0% 0.0% emweb/cifs_timer
086a0217 ad5a7230 0.0% 0.0% 0.0% netfs_mount_handler
08535408 ad5a7038 0.0% 0.0% 0.0% arp_timer
0853d18c ad5a6e40 0.0% 0.0% 0.0% arp_forward_thread
085ad295 ad5a6c48 0.0% 0.0% 0.0% Lic TMR
08c257b1 ad5a6a50 0.0% 0.0% 0.0% tcp_fast
08c28910 ad5a6858 0.0% 0.0% 0.0% tcp_slow
08c53f79 ad5a6660 0.0% 0.0% 0.0% udp_timer
080fe008 ad5a6468 0.0% 0.0% 0.0% CTCP Timer process
08df6853 ad5a6270 0.0% 0.0% 0.0% L2TP data daemon
08df7623 ad5a6078 0.0% 0.0% 0.0% L2TP mgmt daemon
08de39b8 ad5a5e80 0.0% 0.0% 0.0% ppp_timer_thread
08e48157 ad5a5c88 0.0% 0.0% 0.0% vpnlb_timer_thread
081153ff ad5a5a90 0.0% 0.0% 0.0% IPsec message handler
081296cc ad5a5898 0.0% 0.0% 0.0% CTM message handler
089b2bd9 ad5a56a0 0.0% 0.0% 0.0% NAT security-level reconfiguration
08ae1ba8 ad5a54a8 0.0% 0.0% 0.0% ICMP event handler
I want exact troubleshooting.
(1) Steps to follow.
(2) Required configuration
(3) Any good suggestions
(4) Any Tool to troubleshoot.
Suggestions are welcome
Hello,
NMS is probably not the right community to t/s this. You probably want to move this to Security group (Security > Firewalling).
In the meanwhile, i have some details to share for you to check, though i am not a security/ASA expert.
The Dispatch Unit is a process that continually runs on single-core ASAs (models 5505, 5510, 5520, 5540, 5550). The Dispatch Unit takes packets off of the interface driver and passes them to the ASA SoftNP for further processing; it also performs the reverse process.
To determine if the Dispatch Unit process is utilizing the majority of the CPU time, use the command show cpu usage and show process cpu-usage sorted non-zero
show cpu usage (and show cpu usage detail) will show the usage of the ASA CPU cores:
ASA# show cpu usage
CPU utilization for 5 seconds = 0%; 1 minute: 1%; 5 minutes: 0%
show process cpu-usage sorted non-zero will display a sorted list of processes that are using the CPU usage.
In the example below, the Dispatch Unit process has used 50 percent of the CPU for the last 5 seconds:
ASA# show process cpu-usage sorted non-zero
0x0827e731 0xc85c5bf4 50.5% 50.4% 50.3% Dispatch Unit
0x0888d0dc 0xc85b76b4 2.3% 5.3% 5.5% esw_stats
0x090b0155 0xc859ae40 1.5% 0.4% 0.1% ssh
0x0878d2de 0xc85b22c8 0.1% 0.1% 0.1% ARP Thread
0x088c8ad5 0xc85b1268 0.1% 0.1% 0.1% MFIB
0x08cdd5cc 0xc85b4fd0 0.1% 0.1% 0.1% update_cpu_usage
If Dispatch Unit is listed as a top consumer of CPU usage, then use this document to narrow down what might be causing the Dispatch Unit process to be so active.
Most cases of high CPU utilization occur because the Dispatch Unit process is high. Common causes of high utilization include:
Oversubscription
Routing loops
Host with a high number of connections
Excessive system logs
Unequal traffic distribution
More t/s details can be shared by the ASA members from the community.
HTH
-Thanks
Vinod
Similar Messages
-
Trying to use DS 6.2 w/ Cisco ASA 5540 for VPN Auth
Hello all,
I'm trying to connect our Cisco ASA 5540 with LDAP authentication to our DSEE 6.2 directory. The authentication is failing and this line in the debug output from the firewall is really getting to me: "No results returned for iPlanet global password policy".
Their authentication process is two-steps.. It binds with a service account, searches on the "naming attribute" (in our case uid), grabs the DN of the user, and unbinds. With step 2, it binds to the directory with the DN it found when searching, and the password the user supplied. If the second bind is successful, then the firewall lets them on the VPN.
When the firewall binds with the service account, it successfully finds the user's DN and disconnects, so I know my ACI is working correctly there. It just seems to fail when trying to re-bind with the user's DN...
We opened a TAC case with Cisco, and this is their response:
The DN configured on the security appliance to access a Sun directory server must be able to access the default password policy on that server. We recommend using the directory administrator, or a user with directory administrator privileges, as the DN. Alternatively, you can place an ACI on the default password policy.
I refuse to let a poorly written application or appliance bind as cn=Directory Administrator!
I tried putting an ACI on the default password policy located at cn=Password Policy,cn=config , but that doesn't seem to make any difference to the ASA.. My best guess is that it's looking somewhere else for the password policy... did it used to be located elsewhere in iPlanet? Has anyone made this work before with a Cisco ASA?My network admin and I ended up solving this problem by sheer dumb luck. In the ASA config, you tell it what kind of LDAP server it's connecting to. In one set of docs, it had the available options as microsoft, sun, or generic. In another set of docs, we found that openldap was also an acceptable option.
I'm guessing the ASA is thinking the "sun" option is connecting to the old Netscape Directory Server. Changing the "server type" to openldap made it work immediately. It also does not look like it's trying to look at the LDAP server's password policy now either. -
4500 High CPU due to "K2CpuMan"
I am facing high CPU on one of my 4507R-E Chassis. I am having "WS-X45-SUP6-E" SUP and all of my line cards are "WS-X4648-RJ45V-E".
My current IOS is : 12.2(54)SG1
I have followed the below document.
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-4000-series-switches/65591-cat4500-high-cpu.html
Till the time I have not run any CPU SPAN nor I have taken the debug outputs which I cannot do due to sensitivity of application running over the servers terminated over that switch to avoid any millisecond downtime.
I am pasting the rest of the output. Please suggest what can be the possible cause as we are not using DAI, DHCP Snooping or IGMP.
Please suggest if someone can suggest in this regard
------------------ SHOW PROC CPU ------------------
CPU utilization for five seconds: 81%/1%; one minute: 63%; five minutes: 61%
60 5656124073704026290 152 54.15% 55.05% 54.58% 0 Cat4k Mgmt LoPri
------------------ SHOW PLATFORM HEALTH ------------------
%CPU %CPU RunTimeMax Priority Average %CPU Total
Target Actual Target Actual Fg Bg 5Sec Min Hour CPU
K5CpuMan Review 30.00 60.02 30 34 100 500 62 74 62 32140:33
K2CpuMan Review The process that performs software packet forwarding If you see high CPU utilization due to this process, investigate the packets that hit the CPU with use of the show platform cpu packet statistics command.
---------------------SHOW PLATFORM CPU PACKET STATISTICS-----------------------------
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
Unknown 0 0 0 0 0
Esmp 0 0 0 0 0
Input ACL fwd(snooping) 2715401932 1734 1920 1703 1667
ACL fwd(snooping): Packets that are processed by the DHCP snooping, dynamic ARP inspection, or IGMP snoopingI have got the root cause. I took the CPU Span and right after that I came to know there was a server which was consistently sending Multicast Packets causing high CPU. After disabling the related service on server CPU got normal.
-
High receive discards on Sub-Interfaces in Cisco ASA.
Hello Everyone,
Over the past few weeks Solarwinds is reporting high receive discards on two of our subinterfaces created on Cisco ASA. No errors are observed on other subinterfaces. I checked the trunk port interface on the switch for any errors but found none. These errors are visible only under subinterface. What could be the issue?
RegardsI have the same problem too.
I have Cisco ASA 5515 with the next version:
Cisco Adaptive Security Appliance Software Version 9.1(4)
My interface configuration is the next:
PortChannel5 made with Interface GigabitEthernet 0/2 + Interface GigabitEthernet 0/3
Subinterfaces in PortChannel5
Nagios Graphs shows:
- many input discards in virtual subinterfaces
- many output discards in interface Gi0/2 and Gi0/3
- PortChannel5 output discards is the sum of discards in interface Gi0/2 and Gi0/3
if I run the snmpwalk command against the ASA the following results were obtained:
Interface description
[user@FIREWALL01 ~]$ snmpwalk -v 2c -c XXXXXXX 10.255.16.1 | grep ifDescr
IF-MIB::ifDescr.2 = STRING: Adaptive Security Appliance 'asa_mgmt_plane' interface
IF-MIB::ifDescr.3 = STRING: Adaptive Security Appliance 'Internet' interface
IF-MIB::ifDescr.4 = STRING: Adaptive Security Appliance 'LAN_MPLS' interface
IF-MIB::ifDescr.5 = STRING: Adaptive Security Appliance 'GigabitEthernet0/2' interface
IF-MIB::ifDescr.6 = STRING: Adaptive Security Appliance 'GigabitEthernet0/3' interface
IF-MIB::ifDescr.7 = STRING: Adaptive Security Appliance 'stateifha' interface
IF-MIB::ifDescr.8 = STRING: Adaptive Security Appliance 'statelink' interface
IF-MIB::ifDescr.9 = STRING: Adaptive Security Appliance 'Internal-Data0/1' interface
IF-MIB::ifDescr.10 = STRING: Adaptive Security Appliance 'cplane' interface
IF-MIB::ifDescr.11 = STRING: Adaptive Security Appliance 'mgmt_plane_int_tap' interface
IF-MIB::ifDescr.12 = STRING: Adaptive Security Appliance 'management' interface
IF-MIB::ifDescr.13 = STRING: Adaptive Security Appliance 'Virtual254' interface
IF-MIB::ifDescr.14 = STRING: Adaptive Security Appliance 'Port-channel5' interface
IF-MIB::ifDescr.15 = STRING: Adaptive Security Appliance 'VLAN_USGLB_OOB' interface
IF-MIB::ifDescr.16 = STRING: Adaptive Security Appliance 'VLAN_USGLBHSTHYP_MGNT' interface
IF-MIB::ifDescr.17 = STRING: Adaptive Security Appliance 'VLAN_USGLBVRM_OM' interface
IF-MIB::ifDescr.18 = STRING: Adaptive Security Appliance 'VLAN_USGLBVRM_MGNTOM' interface
IF-MIB::ifDescr.19 = STRING: Adaptive Security Appliance 'VLAN_USGLBVRM_MGNT' interface
IF-MIB::ifDescr.20 = STRING: Adaptive Security Appliance 'VLAN_USGLBVRM_SRVF' interface
IF-MIB::ifDescr.21 = STRING: Adaptive Security Appliance 'VLAN_USGLBVRM_SRVB' interface
IF-MIB::ifDescr.22 = STRING: Adaptive Security Appliance 'VLAN_USGLB_DMZ' interface
Input discards
[user@FIREWALL01 ~]$ snmpwalk -v 2c -c xxxxxxxxxx 10.255.16.1 | grep ifInDiscards
IF-MIB::ifInDiscards.2 = Counter32: 0
IF-MIB::ifInDiscards.3 = Counter32: 0
IF-MIB::ifInDiscards.4 = Counter32: 0
IF-MIB::ifInDiscards.5 = Counter32: 0
IF-MIB::ifInDiscards.6 = Counter32: 0
IF-MIB::ifInDiscards.7 = Counter32: 0
IF-MIB::ifInDiscards.8 = Counter32: 0
IF-MIB::ifInDiscards.9 = Counter32: 0
IF-MIB::ifInDiscards.10 = Counter32: 0
IF-MIB::ifInDiscards.11 = Counter32: 0
IF-MIB::ifInDiscards.12 = Counter32: 0
IF-MIB::ifInDiscards.13 = Counter32: 0
IF-MIB::ifInDiscards.14 = Counter32: 0
IF-MIB::ifInDiscards.15 = Counter32: 12481926
IF-MIB::ifInDiscards.16 = Counter32: 9927941
IF-MIB::ifInDiscards.17 = Counter32: 134120211
IF-MIB::ifInDiscards.18 = Counter32: 124695686
IF-MIB::ifInDiscards.19 = Counter32: 27081148
IF-MIB::ifInDiscards.20 = Counter32: 2941537222
IF-MIB::ifInDiscards.21 = Counter32: 32714719
IF-MIB::ifInDiscards.22 = Counter32: 4008856
Output discards
[user@FIREWALL01 ~]$ snmpwalk -v 2c -c xxxxxxxxxxxx 10.255.16.1 | grep ifOutDiscards
IF-MIB::ifOutDiscards.2 = Counter32: 0
IF-MIB::ifOutDiscards.3 = Counter32: 0
IF-MIB::ifOutDiscards.4 = Counter32: 0
IF-MIB::ifOutDiscards.5 = Counter32: 3635696
IF-MIB::ifOutDiscards.6 = Counter32: 119099
IF-MIB::ifOutDiscards.7 = Counter32: 0
IF-MIB::ifOutDiscards.8 = Counter32: 0
IF-MIB::ifOutDiscards.9 = Counter32: 0
IF-MIB::ifOutDiscards.10 = Counter32: 0
IF-MIB::ifOutDiscards.11 = Counter32: 0
IF-MIB::ifOutDiscards.12 = Counter32: 0
IF-MIB::ifOutDiscards.13 = Counter32: 0
IF-MIB::ifOutDiscards.14 = Counter32: 3754795
IF-MIB::ifOutDiscards.15 = Counter32: 0
IF-MIB::ifOutDiscards.16 = Counter32: 0
IF-MIB::ifOutDiscards.17 = Counter32: 0
IF-MIB::ifOutDiscards.18 = Counter32: 0
IF-MIB::ifOutDiscards.19 = Counter32: 0
IF-MIB::ifOutDiscards.20 = Counter32: 0
IF-MIB::ifOutDiscards.21 = Counter32: 0
IF-MIB::ifOutDiscards.22 = Counter32: 0
Output discards may be normals, but I don't understand input discards in virtual subinterfaces of PortChannel5
By the other hand, show interface command in subinterfaces don't show error or discards packets
FIREWALL01/pri/act# sh interface VLAN_USGLBVRM_SRVB detail
Interface Port-channel5.1020 "VLAN_USGLBVRM_SRVB", is up, line protocol is up
Hardware is EtherChannel/LACP, BW 2000 Mbps, DLY 10 usec
VLAN identifier 1020
Description: VLAN_USGLBVRM_SRVB
MAC address 6073.5c69.0917, MTU 1500
IP address 10.255.19.65, subnet mask 255.255.255.192
Traffic Statistics for "VLAN_USGLBVRM_SRVB":
42067433644 packets input, 45125599467459 bytes
28153119062 packets output, 8866514693262 bytes
32715765 packets dropped
Control Point Interface States:
Interface number is 21
Interface config status is active
Interface state is active
Control Point Vlan1020 States:
Interface vlan config status is active
Interface vlan state is UP
FIREWALL01/pri/act# sh interface VLAN_USGLBVRM_SRVF detail
Interface Port-channel5.1019 "VLAN_USGLBVRM_SRVF", is up, line protocol is up
Hardware is EtherChannel/LACP, BW 2000 Mbps, DLY 10 usec
VLAN identifier 1019
Description: VLAN_USGLBVRM_SRVF
MAC address 6073.5c69.0917, MTU 1500
IP address 10.255.19.1, subnet mask 255.255.255.192
Traffic Statistics for "VLAN_USGLBVRM_SRVF":
30475814698 packets input, 14615432248013 bytes
27472348465 packets output, 20872697455933 bytes
2941588838 packets dropped
Control Point Interface States:
Interface number is 20
Interface config status is active
Interface state is active
Control Point Vlan1019 States:
Interface vlan config status is active
Interface vlan state is UP
FIREWALL01/pri/act#
Can anyone explain why so many input errors appear in the subinterfaces?
Thanks in advance! -
Iptables command "translated" Cisco ASA 5540 Ver 9.0(1)
I would like to have these commands on our Firewall to avoid at least several students to use this service. Can someone help me to translate this? It's apparently working great if I will use an Linux box or another firewall compatible with iptables.
Thanks in advance.
Hermano
iptables -I INPUT -s hotspotshield.com -j REJECT
iptables -I INPUT -s hotspotshield.net -j REJECT
iptables -I INPUT -s anchorfree.com -j REJECT
iptables -I INPUT -s anchorfree.net -j REJECT
iptables -I INPUT -s openvpn.net -j REJECT
iptables -I OUTPUT -d hotspotshield.com -j REJECT
iptables -I OUTPUT -d hotspotshield.net -j REJECT
iptables -I OUTPUT -d anchorfree.com -j REJECT
iptables -I OUTPUT -d anchorfree.net -j REJECT
iptables -I OUTPUT -d openvpn.net -j REJECTCheck the following link, it should help you out.
http://www.packetpros.com/2012/08/url-filter-on-asa.html -
Cisco ASA 5540 Syslog logging everything
Hello. I am trying to log every connection (Build, deny, etc)
But for some reason I don't see them sh log
Any ideas?
asa1# show run log
logging enable
logging timestamp
logging emblem
logging asdm-buffer-size 512
logging console debugging
logging monitor debugging
logging buffered debugging
logging trap debugging
logging asdm debugging
logging from-address ***********
logging recipient-address ************* level errors
logging recipient-address ********** level errors
logging queue 0
logging host LOG 192.168.168.2 format emblem
logging debug-trace
logging permit-hostdownRahul; I fix the problem. I had every logging output enabled (logging console debugging,logging monitor debugging,logging buffered debugging) and the ASA was loosing the abilitity to log. I disable them and now everyhing is logged.
Thanks -
Dispatch Unit - High Cpu Usage
Hi,
ASA5510 8.2.5(50)
The Dispatch unit process is contantly having high cpu usage for last 10 hours.
Things checked:
1. show proc cpu-usage
2. show perf
It seems fine . Output attached
3. Show interfaces for error
No error, overruns, underrruns on interfaces
4. show traffic
Total cumulative through put on approx 4 Mbps.
drop rate max is 3 pkts /sec randomly and rare occurance on some interfaces
5. Connections and Xlate seem normal.
approx 1100.
Counters were reset 1 hour before the data was collected.
# sh cpu usage
CPU utilization for 5 seconds = 39%; 1 minute: 38%; 5 minutes: 44%
# sh processes cpu-usage sorted
PC Thread 5Sec 1Min 5Min Process
081aadc4 a79aff7c 35.7% 37.5% 42.5% Dispatch Unit
0853f89e a79a0b68 0.4% 0.2% 0.2% ARP Thread
# show perfmon
PERFMON STATS: Current Average
Xlates 0/s 0/s
Connections 21/s 32/s
TCP Conns 17/s 28/s
UDP Conns 1/s 1/s
URL Access 0/s 0/s
URL Server Req 0/s 0/s
TCP Fixup 0/s 0/s
TCP Intercept Established Conns 0/s 0/s
TCP Intercept Attempts 0/s 0/s
sh interface e0/0 | inc overrun
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
fw01/act# sh interface e0/1 | inc overrun
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
# sh conn all
1135 in use, 8777 most used
# sh xlate count
112 in use, 265 most used
# show asp drop frame
No route to host (no-route) 870
Flow is denied by configured rule (acl-drop) 103915
First TCP packet not SYN (tcp-not-syn) 1317
Bad TCP checksum (bad-tcp-cksum) 2
TCP failed 3 way handshake (tcp-3whs-failed) 6695
TCP RST/FIN out of order (tcp-rstfin-ooo) 4025
TCP packet SEQ past window (tcp-seq-past-win) 13
TCP Out-of-Order packet buffer full (tcp-buffer-full) 1949
TCP Out-of-Order packet buffer timeout (tcp-buffer-timeout) 600
TCP RST/SYN in window (tcp-rst-syn-in-win) 5
TCP dup of packet in Out-of-Order queue (tcp-dup-in-queue) 617
TCP packet failed PAWS test (tcp-paws-fail) 1248
IPSEC tunnel is down (ipsec-tun-down) 2
Slowpath security checks failed (sp-security-failed) 1699
DNS Inspect id not matched (inspect-dns-id-not-matched) 4
FP L2 rule drop (l2_acl) 15436
Dropped pending packets in a closed socket (np-socket-closed) 2
Please let us know what reason can be there for high cpu usage by Dispatch unit under current statistics?
What else should be checked to ensure cpu usage comes down?
Regards,
Gurjit Singh
Network Engineer
Spooster IT Services.Hi Gurjar,
r u getting the below mentioned syslog messages?
Flow is denied by configured rule (acl-drop) 103915
106023, 106100, 106004
TCP Out-of-Order packet buffer full (tcp-buffer-full) 1949
TCP Out-of-Order packet buffer full:
This counter is incremented and the packet is dropped when appliance receives an
out-of-order TCP packet on a connection and there is no buffer space to store this packet.
Typically TCP packets are put into order on connections that are inspected by the
appliance or when packets are sent to SSM for inspection. There is a default queue size
and when packets in excess of this default queue size are received they will be dropped.
Recommendations:
On ASA platforms the queue size could be increased using queue-limit configuration
under tcp-map.
Similarly you need to check many reasons for the asp drop logs that you have captured and you need to monitor how much it is increasing and the difference.......
but 40 % CPU utilization is a okay kind of thing and you do not need to worry if that happens only during peak hours ans it is not increasing drastically more and more.
http://www.cisco.com/c/en/us/td/docs/security/asa/asa82/command/reference/cmd_ref/s2.html#wp1435096
Regards
Karthik -
Getting High cpu in s720 due to snmp(Prime)
Hi Team,
We are getting high cpu Spike in our all distribution switches in many locations.
Also we found our Cisco prime infrastructure causing the high cpu.but the spike only for 20 to 30 seconds.
If we deny the PI there will be no CPU spike..
Is there any troubleshooting or configuration need to done for this issue..but we are getting in all our sites due to this in different cpu process I'd ..
Affected sup like 720 but in core we are not getting that issue with same sup and ios.
Kindly suggest any action plan regarding the high cpu due to the High cpu due to snmp..
Regards,
ArunHi Afroj,
PFB
i have attached the EEM logs during high snmp cpu(605).
Please expertise
Process 605: SNMP ENGINE
Stack segment 0x528E3084 - 0x528E5F64
FP: 0x528E5E88, RA: 0x413DFB00
FP: 0x528E5EB8, RA: 0x41200E60
Process 605: SNMP ENGINE
Stack segment 0x528E3084 - 0x528E5F64
FP: 0x528E5E88, RA: 0x413DFB00
FP: 0x528E5EB8, RA: 0x41200E60
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 10.237.102.34 162 10.251.1.1 52465 0 0 0 0
Queues: output 0
input 0 (drops 0, max 0, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 10.238.53.198 162 10.251.1.1 53296 0 0 0 0
Queues: output 0
input 0 (drops 0, max 0, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 10.238.53.200 162 10.251.1.1 61585 0 0 0 0
Queues: output 0
input 0 (drops 0, max 0, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 10.238.53.202 162 10.251.1.1 58228 0 0 0 0
Queues: output 0
input 0 (drops 0, max 0, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 10.242.225.134 162 10.251.1.1 52865 0 0 0 0
Queues: output 0
input 0 (drops 0, max 0, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 10.242.225.136 162 10.251.1.1 50720 0 0 0 0
Queues: output 0
input 0 (drops 0, max 0, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 209.51.49.10 162 10.251.1.1 51376 0 0 0 0
Queues: output 0
input 0 (drops 0, max 0, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 209.51.49.21 162 10.251.1.1 49787 0 0 0 0
Queues: output 0
input 0 (drops 0, max 0, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 209.51.49.25 162 10.251.1.1 58862 0 0 0 0
Queues: output 0
input 0 (drops 0, max 0, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 209.51.49.3 162 10.251.1.1 56352 0 0 0 0
Queues: output 0
input 0 (drops 0, max 0, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 209.51.49.40 162 10.251.1.1 53080 0 0 0 0
Queues: output 0
input 0 (drops 0, max 0, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 209.51.49.41 162 10.251.1.1 59354 0 0 0 0
Queues: output 0
input 0 (drops 0, max 0, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 209.51.49.42 162 10.251.1.1 64147 0 0 0 0
Queues: output 0
input 0 (drops 0, max 0, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 10.251.1.14 56112 10.251.1.1 2228 0 0 211 0
Queues: output 0
input 0 (drops 0, max 50, highwater 1)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 --listen-- 10.251.1.66 1645 0 0 11 0
Queues: output 0
input 0 (drops 0, max 50, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 --listen-- 10.251.1.66 1646 0 0 11 0
Queues: output 0
input 0 (drops 0, max 50, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 10.184.66.171 67 10.251.9.225 67 0 0 2211 0
Queues: output 0
input 0 (drops 0, max 50, highwater 22)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 10.238.53.198 1342 10.251.1.1 161 0 0 1001 0
Queues: output 0
input 0 (drops 0, max 50, highwater 28)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 --listen-- 10.251.1.66 162 0 0 1011 0
Queues: output 0
input 0 (drops 0, max 50, highwater 2)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 --listen-- 10.251.1.66 55325 0 0 1011 0
Queues: output 0
input 0 (drops 0, max 50, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17(v6) --listen-- --any-- 161 0 0 20001 0
Queues: output 0
input 0 (drops 0, max 50, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17(v6) --listen-- --any-- 162 0 0 20011 0
Queues: output 0
input 0 (drops 0, max 50, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17(v6) --listen-- --any-- 61700 0 0 20001 0
Queues: output 0
input 0 (drops 0, max 50, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 --listen-- 10.251.1.66 123 0 0 1 0
Queues: output 0
input 0 (drops 0, max 50, highwater 2)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 10.237.100.198 514 10.251.1.1 57434 0 0 400241 0
Queues: output 0
input 0 (drops 0, max 50, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 10.237.204.214 514 10.251.1.1 50817 0 0 400241 0
Queues: output 0
input 0 (drops 0, max 50, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 10.238.60.34 514 10.251.1.1 51858 0 0 400241 0
Queues: output 0
input 0 (drops 0, max 50, highwater 0)
Proto Remote Port Local Port In Out Stat TTY OutputIF
17 10.243.115.105 162 10.251.1.1 52505 0 0 0 0
Queues: output 0
input 0 (drops 0, max 0, highwater 0)
11:50:13.680 IST Wed Jan 8 2014
Process 605: SNMP ENGINE
Stack segment 0x528E3084 - 0x528E5F64
FP: 0x528E5E88, RA: 0x413DFB00
FP: 0x528E5EB8, RA: 0x41200E60
Process 605: SNMP ENGINE
Stack segment 0x528E3084 - 0x528E5F64
FP: 0x528E5E88, RA: 0x413DFB00
FP: 0x528E5EB8, RA: 0x41200E60
CPU utilization for five seconds: 40%/7%; one minute: 23%; five minutes: 16%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
169 12896 12026 1072 23.10% 9.12% 2.93% 2 SSH Process
11 103780680 268103677 387 2.95% 2.77% 2.70% 0 ARP Input
323 59149684 230740289 256 1.67% 1.50% 1.40% 0 IP Input
215 112 632 177 1.67% 0.13% 0.02% 3 Virtual Exec
412 35386816 13614064 2599 0.63% 0.69% 0.69% 0 CEF: IPv4 proces
214 12 47 255 0.39% 0.03% 0.00% 0 EEM Callback Thr
608 23656 4174 5667 0.23% 0.02% 0.00% 0 SNMP Traps
268 1711956 38291258 44 0.23% 0.21% 0.22% 0 cpf_process_tpQ
85 11661952 28435848 410 0.23% 0.28% 0.29% 0 ARP HA
317 3057232 14577406 209 0.15% 0.17% 0.16% 0 CDP Protocol
350 1365761024332778 0 0.15% 0.17% 0.16% 0 Ethernet Msec Ti
2 9760 1621549 6 0.15% 0.01% 0.00% 0 Load Meter
601 9567588 11060330 865 0.15% 0.20% 0.21% 0 DHCPD Receive
455 159172 114655346 1 0.07% 0.04% 0.02% 0 OSPF-1 Hello
168 20 6694 2 0.07% 0.00% 0.00% 0 TACACS+
50 804004 8351522 96 0.07% 0.09% 0.11% 0 Per-Second Jobs
454 59648 252056393 0 0.07% 0.03% 0.02% 0 RADIUS
325 1268180 11957057 106 0.07% 0.02% 0.00% 0 ADJ resolve proc
456 9988 34907 286 0.07% 0.03% 0.01% 0 TPLUS
293 56980 16209877 3 0.07% 0.01% 0.00% 0 ACE Config Prop
267 6486544 1768422 3667 0.07% 0.07% 0.07% 0 Compute load avg
577 516 47525 10 0.07% 0.00% 0.00% 0 EEM ED SNMP
23 74331892 646724056 114 0.00% 0.90% 0.83% 0 IPC Seat Manager
22 0 1 0 0.00% 0.00% 0.00% 0 IPC Process leve
25 0 1 0 0.00% 0.00% 0.00% 0 IPC Stdby Update
24 0 1 0 0.00% 0.00% 0.00% 0 IPC Session Serv
21 7584 8101413 0 0.00% 0.00% 0.00% 0 IPC Deferred Por
28 0 2 0 0.00% 0.00% 0.00% 0 Dialer event
29 0 12 0 0.00% 0.00% 0.00% 0 ifIndex Receive
30 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun
20 232492 8101397 28 0.00% 0.04% 0.05% 0 IPC Periodic Tim
26 176 810789 0 0.00% 0.00% 0.00% 0 Compute SRP rate
33 0 1 0 0.00% 0.00% 0.00% 0 VSL LTL PACL ICC
27 0 2 0 0.00% 0.00% 0.00% 0 DDR Timers
35 280 2529 110 0.00% 0.00% 0.00% 0 rf proxy rp agen
19 0 1 0 0.00% 0.00% 0.00% 0 NTI Example Proc
37 4 3 1333 0.00% 0.00% 0.00% 0 client_entity_se
38 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect
39 0 1 0 0.00% 0.00% 0.00% 0 Connection Mgr
40 4 40 100 0.00% 0.00% 0.00% 0 Snmp ICC Process
41 60 262 229 0.00% 0.00% 0.00% 0 Cat6k SNMP
42 8 239 33 0.00% 0.00% 0.00% 0 Cat6k SNMP Trap
31 0 1 0 0.00% 0.00% 0.00% 0 Crash writer
32 1182260 6068061 194 0.00% 0.01% 0.00% 0 EnvMon
45 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd
46 70304 1415029 49 0.00% 0.00% 0.00% 0 Net Background
18 2224 135036 16 0.00% 0.00% 0.00% 0 IPC Dynamic Cach
48 212 8653 24 0.00% 0.00% 0.00% 0 Logger
49 24752 8101358 3 0.00% 0.00% 0.00% 0 TTY Background
34 25544 8101346 3 0.00% 0.00% 0.00% 0 GraphIt
36 0 41 0 0.00% 0.00% 0.00% 0 rf proxy message
52 28 209 133 0.00% 0.00% 0.00% 0 Cat6k NTI ICC pr
53 152 511 297 0.00% 0.00% 0.00% 0 IF-MGR control p
43 0 1 0 0.00% 0.00% 0.00% 0 ARP Snoop
55 60 583 102 0.00% 0.00% 0.00% 0 Env action
56 0 2 0 0.00% 0.00% 0.00% 0 Flash ICC Server
57 0 13514 0 0.00% 0.00% 0.00% 0 foundation
58 0 1 0 0.00% 0.00% 0.00% 0 VS Redirect EOBC
59 3450908 116847868 29 0.00% 0.04% 0.04% 0 VSIBC process
60 8 80 100 0.00% 0.00% 0.00% 0 ICC Nego
61 0 13522 0 0.00% 0.00% 0.00% 0 itasca
62 124 67 1850 0.00% 0.00% 0.00% 0 EOBC Monitor Pro
63 2543040 128884246 19 0.00% 0.03% 0.02% 0 Net Input
64 12 45 266 0.00% 0.00% 0.00% 0 draco-oir-proces
65 0 1 0 0.00% 0.00% 0.00% 0 EnergyWise incre
66 0 1 0 0.00% 0.00% 0.00% 0 K+ RED MAMBA pr
67 0 1 0 0.00% 0.00% 0.00% 0 K+ RED MAMBA Adj
68 0 9 0 0.00% 0.00% 0.00% 0 MWAM CONFIG ON S
44 11928 8101398 1 0.00% 0.00% 0.00% 0 Dynamic ARP Insp
47 0 3 0 0.00% 0.00% 0.00% 0 IDB Work
71 0 1 0 0.00% 0.00% 0.00% 0 PF Sync process
72 0 1 0 0.00% 0.00% 0.00% 0 PF SCP process
73 0 8 0 0.00% 0.00% 0.00% 0 RPC rf_rfmib_not
74 12 11 1090 0.00% 0.00% 0.00% 0 RPC Sync
75 0 11 0 0.00% 0.00% 0.00% 0 SCP Multicast
76 0 2 0 0.00% 0.00% 0.00% 0 SCP Generic Unso
77 16 487 32 0.00% 0.00% 0.00% 0 TCAM API Process
17 0 1 0 0.00% 0.00% 0.00% 0 IFS Agent Manage
79 7608 8101392 0 0.00% 0.00% 0.00% 0 VS CHASSIS Info
80 0 16 0 0.00% 0.00% 0.00% 0 VS ICC request p
81 0 2 0 0.00% 0.00% 0.00% 0 cpf_msg_holdq_pr
82 336 1211 277 0.00% 0.00% 0.00% 0 cpf_msg_rcvq_pro
16 336 4338 77 0.00% 0.00% 0.00% 0 EEM ED Syslog
51 6060004 221751 27328 0.00% 0.07% 0.05% 0 Per-minute Jobs
14 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager
54 36 524 68 0.00% 0.00% 0.00% 0 IF-MGR event pro
87 0 2 0 0.00% 0.00% 0.00% 0 IPSEC HA Mgr
88 14092 135040 104 0.00% 0.00% 0.00% 0 RF CWAN HA Proce
89 0 3 0 0.00% 0.00% 0.00% 0 CWAN HA VLAN SYN
90 0 7 0 0.00% 0.00% 0.00% 0 CWAN IF EVENT HA
91 0 3 0 0.00% 0.00% 0.00% 0 CWAN APS HA Proc
69 12 386 31 0.00% 0.00% 0.00% 0 svclc
93 0 5 0 0.00% 0.00% 0.00% 0 DHCP Snooping HA
94 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Dest
95 0 1 0 0.00% 0.00% 0.00% 0 FM HA Sync
96 26576 8102313 3 0.00% 0.00% 0.00% 0 Heartbeat Proces
97 4 108 37 0.00% 0.00% 0.00% 0 Const ICC reques
98 0 1 0 0.00% 0.00% 0.00% 0 Const ICC event
99 0 1 0 0.00% 0.00% 0.00% 0 Const ICC except
100 0 1 0 0.00% 0.00% 0.00% 0 ICC Local Reques
101 16660 246024 67 0.00% 0.00% 0.00% 0 ICC Async mcast
102 0 1 0 0.00% 0.00% 0.00% 0 ICC_REQ Debug Pr
103 16 148 108 0.00% 0.00% 0.00% 0 ICC FC Process
104 0 1 0 0.00% 0.00% 0.00% 0 ICC FC Test Send
105 0 1 0 0.00% 0.00% 0.00% 0 ICC FC Test Low
106 0 1 0 0.00% 0.00% 0.00% 0 L3 Manager HA
107 0 1 0 0.00% 0.00% 0.00% 0 FWSM HA Sync
108 0 1 0 0.00% 0.00% 0.00% 0 QoS HA CHKPT
109 0 5 0 0.00% 0.00% 0.00% 0 CWRP SPA TSM HA
110 4 4 1000 0.00% 0.00% 0.00% 0 Switch IP Host T
111 0 3 0 0.00% 0.00% 0.00% 0 RPC virtual_mode
112 0 3 0 0.00% 0.00% 0.00% 0 RPC vs_sp_print_
113 0 3 0 0.00% 0.00% 0.00% 0 RPC vs_rp2sp_rpc
114 4 20 200 0.00% 0.00% 0.00% 0 IP Admission HA
115 0 1 0 0.00% 0.00% 0.00% 0 Mistral Reset Pr
116 0 1 0 0.00% 0.00% 0.00% 0 ECC correctable
117 300 44 6818 0.00% 0.00% 0.00% 0 PF_Init Process
118 0 1 0 0.00% 0.00% 0.00% 0 K+ RED MAMBA STA
119 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
120 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
121 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
122 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
70 0 21 0 0.00% 0.00% 0.00% 0 PF IPC ICC Init
78 0 3 0 0.00% 0.00% 0.00% 0 RPC vs_prov_lc-R
125 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
126 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
127 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
128 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
129 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
130 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
131 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
132 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
133 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
134 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
135 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
136 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
137 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
138 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
139 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
83 0 1 0 0.00% 0.00% 0.00% 0 CHKPT Test clien
141 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
142 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
143 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
144 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
145 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
146 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
147 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
148 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
149 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
84 7804 68315 114 0.00% 0.00% 0.00% 0 DHCP Snooping
151 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
86 0 2 0 0.00% 0.00% 0.00% 0 IKE HA Mgr
153 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
154 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
155 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
92 0 3 0 0.00% 0.00% 0.00% 0 CWAN SRP HA Proc
157 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
158 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
159 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
160 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
161 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
162 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
163 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
164 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
165 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
166 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
167 0 35 0 0.00% 0.00% 0.00% 0 KEYSTORE HA IPC
123 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
13 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit
170 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
171 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
172 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
173 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
174 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
175 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
176 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
177 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
178 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
179 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
180 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
181 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
182 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
183 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
184 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
185 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
186 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
187 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
188 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
189 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
190 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
191 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
192 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
193 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
194 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
195 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
196 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
197 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
198 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
199 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
200 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
201 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
202 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
203 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
204 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
205 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
206 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
207 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
208 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
209 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
210 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
211 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
212 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
213 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
12 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT
10 0 2 0 0.00% 0.00% 0.00% 0 Timers
217 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
218 0 196 0 0.00% 0.00% 0.00% 0 SNMP Timers
219 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
220 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
221 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
222 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
223 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
224 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
225 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
9 448 1689 265 0.00% 0.00% 0.00% 0 Pool Manager
227 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
228 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
229 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
230 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
231 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
124 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
140 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
235 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
236 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
237 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
238 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
239 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
240 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
241 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
242 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
243 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
244 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
245 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
246 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
247 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
248 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
249 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
250 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
251 4 1 4000 0.00% 0.00% 0.00% 0 IPC ISSU Version
252 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
253 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
254 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
255 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
256 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
257 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
258 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
259 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
260 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
261 0 1 0 0.00% 0.00% 0.00% 0 Cat6k NTI RP Exa
262 0 3 0 0.00% 0.00% 0.00% 0 RPC rpc-slave-11
263 16 126 126 0.00% 0.00% 0.00% 0 PF_Split Sync Pr
264 36 233 154 0.00% 0.00% 0.00% 0 RPC pf-split-rp
265 40 104 384 0.00% 0.00% 0.00% 0 RPC idprom-MP
266 0 1 0 0.00% 0.00% 0.00% 0 BACK CHECK
150 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
8 33163948 1904341 17414 0.00% 0.31% 0.37% 0 Check heaps
269 0 13 0 0.00% 0.00% 0.00% 0 IPC Apps Task
271 12 13 923 0.00% 0.00% 0.00% 0 RTTYS Process
272 0 8 0 0.00% 0.00% 0.00% 0 RFS server proce
273 0 35 0 0.00% 0.00% 0.00% 0 OBFL Cfg Dispatc
274 8 2254 3 0.00% 0.00% 0.00% 0 PF Clock Process
275 0 1 0 0.00% 0.00% 0.00% 0 PPP IP Add Route
276 16 16330 0 0.00% 0.00% 0.00% 0 Transport Port A
277 0 15 0 0.00% 0.00% 0.00% 0 AggMgr Process
278 48 59125 0 0.00% 0.00% 0.00% 0 SFF8472
279 0 1 0 0.00% 0.00% 0.00% 0 const_dump_info_
280 0 1 0 0.00% 0.00% 0.00% 0 POS APS Event Pr
281 0 1 0 0.00% 0.00% 0.00% 0 DSX3MIB ll handl
282 0 1 0 0.00% 0.00% 0.00% 0 FPD Management P
283 0 1 0 0.00% 0.00% 0.00% 0 FPD Action Proce
284 16 253 63 0.00% 0.00% 0.00% 0 Remote Console P
285 0 1 0 0.00% 0.00% 0.00% 0 DIAG-RP-PROC
286 0 1 0 0.00% 0.00% 0.00% 0 DiagErrorLogProc
287 0 3 0 0.00% 0.00% 0.00% 0 Remote CWAN OIR
152 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
289 0 1 0 0.00% 0.00% 0.00% 0 SCP async: LCP#5
290 0 2 0 0.00% 0.00% 0.00% 0 CWAN ACD tasks
291 0 4 0 0.00% 0.00% 0.00% 0 ACE_REQUESTS han
292 61004 252151322 0 0.00% 0.04% 0.02% 0 ACE Tunnel Task
156 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
294 0 3 0 0.00% 0.00% 0.00% 0 ACE VRF Task
295 13552 8101376 1 0.00% 0.00% 0.00% 0 CWAN CHOCX PROCE
296 0 1 0 0.00% 0.00% 0.00% 0 HAL Mailbox
297 22672 8101370 2 0.00% 0.00% 0.00% 0 CWAN CHDSX PROCE
298 0 1 0 0.00% 0.00% 0.00% 0 CT3 Mailbox
299 0 1 0 0.00% 0.00% 0.00% 0 CE3 Mailbox
15 244 647 377 0.00% 0.00% 0.00% 0 Entity MIB API
301 0 1 0 0.00% 0.00% 0.00% 0 CWAN OIR Handler
302 4 7 571 0.00% 0.00% 0.00% 0 Earl NDE Task
7 0 1 0 0.00% 0.00% 0.00% 0 PF Redun ICC Req
304 1619456 4378957 369 0.00% 0.01% 0.00% 0 esw_vlan_stat_pr
6 0 1 0 0.00% 0.00% 0.00% 0 PF Redun ICC Req
306 0 1 0 0.00% 0.00% 0.00% 0 CWRP SPA ATM OIR
307 0 1 0 0.00% 0.00% 0.00% 0 ACE event
308 0 1 0 0.00% 0.00% 0.00% 0 ACE HAPI
309 2128 540126 3 0.00% 0.00% 0.00% 0 ACE policy loade
310 40 191 209 0.00% 0.00% 0.00% 0 CMFI RP process
311 132 3822 34 0.00% 0.00% 0.00% 0 AAA Server
312 88 3276 26 0.00% 0.00% 0.00% 0 AAA ACCT Proc
313 0 1 0 0.00% 0.00% 0.00% 0 ACCT Periodic Pr
314 0 1 0 0.00% 0.00% 0.00% 0 AAA System Acct
315 0 1 0 0.00% 0.00% 0.00% 0 Auth-proxy AAA B
316 24 27027 0 0.00% 0.00% 0.00% 0 IP Admin SM Proc
5 0 10 0 0.00% 0.00% 0.00% 0 IPC ISSU Dispatc
319 0 3 0 0.00% 0.00% 0.00% 0 BFD HA
320 44 4952 8 0.00% 0.00% 0.00% 0 CDP EXPORT PROCE
321 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R
322 0 2 0 0.00% 0.00% 0.00% 0 CEF switching ba
4 0 150 0 0.00% 0.00% 0.00% 0 Retransmission o
324 0 1 0 0.00% 0.00% 0.00% 0 ICMP event handl
232 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
326 0 1 0 0.00% 0.00% 0.00% 0 PPCP RP Stats Ba
327 0 1 0 0.00% 0.00% 0.00% 0 IPv6 Echo event
328 0 2 0 0.00% 0.00% 0.00% 0 PASVC create VA
330 0 3 0 0.00% 0.00% 0.00% 0 HDLC HA
331 0 2 0 0.00% 0.00% 0.00% 0 Spanning Tree
332 0 2 0 0.00% 0.00% 0.00% 0 SpanTree Msg
333 8 5 1600 0.00% 0.00% 0.00% 0 EAPoUDP Process
334 0 2 0 0.00% 0.00% 0.00% 0 KRB5 AAA
335 0 3 0 0.00% 0.00% 0.00% 0 CTS CORE
336 24 381 62 0.00% 0.00% 0.00% 0 SXP CORE
337 0 1 0 0.00% 0.00% 0.00% 0 LSP Tunnel FRR
338 0 3 0 0.00% 0.00% 0.00% 0 MPLS Auto-Tunnel
339 0 1 0 0.00% 0.00% 0.00% 0 O-UNI Client Msg
340 8 266 30 0.00% 0.00% 0.00% 0 SSM connection m
341 4 4 1000 0.00% 0.00% 0.00% 0 cv6 per prefix a
342 0 2 0 0.00% 0.00% 0.00% 0 REP Topology cha
343 0 1 0 0.00% 0.00% 0.00% 0 AUTH POLICY Fram
344 52 255 203 0.00% 0.00% 0.00% 0 Auth Manager
345 0 1 0 0.00% 0.00% 0.00% 0 CMD HANDLER
346 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Mgr Proces
347 0 1 0 0.00% 0.00% 0.00% 0 EAP Framework
348 0 2 0 0.00% 0.00% 0.00% 0 Ethernet CFM
349 7040 48484291 0 0.00% 0.00% 0.00% 0 Ethernet Timer C
3 316 905 349 0.00% 0.00% 0.00% 1 SSH Process
351 0 2 0 0.00% 0.00% 0.00% 0 Ethernet LMI
352 0 1 0 0.00% 0.00% 0.00% 0 MAB Framework
353 244 690 353 0.00% 0.00% 0.00% 0 SW VLAN RP
354 0 9 0 0.00% 0.00% 0.00% 0 const ipfib per-
355 0 2 0 0.00% 0.00% 0.00% 0 CFMPAL-RP Proces
356 0 2 0 0.00% 0.00% 0.00% 0 CFMPAL RPC Proce
357 8 7 1142 0.00% 0.00% 0.00% 0 Const Rate Limit
358 0 1 0 0.00% 0.00% 0.00% 0 802.1x Webauth F
360 0 2 0 0.00% 0.00% 0.00% 0 Ethernet OAM Pro
361 0 1 0 0.00% 0.00% 0.00% 0 DMLP Counter
362 388 1200 323 0.00% 0.00% 0.00% 0 HWIF QoS Process
363 0 5 0 0.00% 0.00% 0.00% 0 MLS FIB RP
364 12 47 255 0.00% 0.00% 0.00% 0 mcast_rp_icc Pro
365 0 2 0 0.00% 0.00% 0.00% 0 QOS Stats Export
366 5543336 4050397 1368 0.00% 0.06% 0.05% 0 QOS Stats Gather
367 0 1 0 0.00% 0.00% 0.00% 0 satvs_interf_pro
368 8 264 30 0.00% 0.00% 0.00% 0 Switch IP Host T
369 0 1 0 0.00% 0.00% 0.00% 0 SSS Manager
370 0 1 0 0.00% 0.00% 0.00% 0 SSS Feature Mana
371 9872 31671669 0 0.00% 0.00% 0.00% 0 SSS Feature Time
372 0 1 0 0.00% 0.00% 0.00% 0 AC Switch
373 0 3 0 0.00% 0.00% 0.00% 0 ATM HA
374 15156 162270 93 0.00% 0.00% 0.00% 0 CEF background p
375 0 1 0 0.00% 0.00% 0.00% 0 IP IRDP
376 269560 141933 1899 0.00% 0.00% 0.00% 0 IP Background
377 44808 135215 331 0.00% 0.00% 0.00% 0 IP RIB Update
378 0 1 0 0.00% 0.00% 0.00% 0 IP Traceroute
379 0 30 0 0.00% 0.00% 0.00% 0 Socket Timers
380 0 3 0 0.00% 0.00% 0.00% 0 FR HA
381 4 4 1000 0.00% 0.00% 0.00% 0 GLBP HA
382 224 29448 7 0.00% 0.00% 0.00% 0 TCP Timer
383 712 3708 192 0.00% 0.00% 0.00% 0 TCP Protocols
384 0 27049 0 0.00% 0.00% 0.00% 0 HTTP CORE
385 0 4 0 0.00% 0.00% 0.00% 0 HSRP HA
386 0 6 0 0.00% 0.00% 0.00% 0 MRIB RP Proxy
387 0 9 0 0.00% 0.00% 0.00% 0 PIM HA
388 0 3 0 0.00% 0.00% 0.00% 0 XDR background p
389 278160 298487 931 0.00% 0.00% 0.00% 0 XDR mcast
390 17463136 4115749 4243 0.00% 0.22% 0.23% 0 IPC LC Message H
391 0 3 0 0.00% 0.00% 0.00% 0 XDR RP Ping Back
392 185652 143551 1293 0.00% 0.00% 0.00% 0 XDR RP backgroun
393 0 1 0 0.00% 0.00% 0.00% 0 XDR RP Test Back
394 0 1 0 0.00% 0.00% 0.00% 0 RETRY_REPOPULATE
395 0 3 0 0.00% 0.00% 0.00% 0 LSD HA Proc
396 0 11 0 0.00% 0.00% 0.00% 0 PPP HA
397 0 3 0 0.00% 0.00% 0.00% 0 PPP IPC flow con
398 0 1 0 0.00% 0.00% 0.00% 0 SLB CF process
399 0 4 0 0.00% 0.00% 0.00% 0 VRRP HA
400 0 2 0 0.00% 0.00% 0.00% 0 CTS DATAPATH
401 0 1 0 0.00% 0.00% 0.00% 0 ac_atm_state_eve
402 0 1 0 0.00% 0.00% 0.00% 0 Inspect Timer
403 0 4 0 0.00% 0.00% 0.00% 0 IPv6 ACL RP Proc
404 0 1 0 0.00% 0.00% 0.00% 0 MFI Comm RP Proc
405 0 10 0 0.00% 0.00% 0.00% 0 MPLS ACL IPC Bac
406 0 16 0 0.00% 0.00% 0.00% 0 LDP HA
407 0 3 0 0.00% 0.00% 0.00% 0 MPLS VPN HA Clie
408 0 4 0 0.00% 0.00% 0.00% 0 RSVP HA Services
409 0 6 0 0.00% 0.00% 0.00% 0 TSPTUN HA
410 0 4 0 0.00% 0.00% 0.00% 0 Inter Chassis Pr
411 0 2 0 0.00% 0.00% 0.00% 0 LFDp Input Proc
1 19280 100248 192 0.00% 0.00% 0.00% 0 Chunk Manager
413 0 4 0 0.00% 0.00% 0.00% 0 ADJ background
414 0 1 0 0.00% 0.00% 0.00% 0 Cv6 RP Process
415 4 10 400 0.00% 0.00% 0.00% 0 mfib-const-rp Pr
416 0 3 0 0.00% 0.00% 0.00% 0 AAA HA INT
417 0 1 0 0.00% 0.00% 0.00% 0 Critical Auth
418 0 1 0 0.00% 0.00% 0.00% 0 COPS
419 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant
420 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant
421 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant
422 0 3 0 0.00% 0.00% 0.00% 0 ECFM HA IPC flow
423 4525688 8507921 531 0.00% 0.01% 0.01% 0 L3 Manager
424 0 6 0 0.00% 0.00% 0.00% 0 L3 Manager Commo
425 84556 10616336 7 0.00% 0.00% 0.00% 0 FM core
426 0 1 0 0.00% 0.00% 0.00% 0 Probe Input
427 0 2 0 0.00% 0.00% 0.00% 0 Netflow XDR Mana
428 0 2 0 0.00% 0.00% 0.00% 0 RARP Input
429 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall
430 0 2 0 0.00% 0.00% 0.00% 0 X.25 Background
431 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana
432 36 67566 0 0.00% 0.00% 0.00% 0 DHCPD Timer
433 3164500 4053966 780 0.00% 0.04% 0.05% 0 HIDDEN VLAN Proc
434 0 1 0 0.00% 0.00% 0.00% 0 CWAN LTL manager
435 0 1 0 0.00% 0.00% 0.00% 0 CWAN-RP SCP Proc
436 0 14 0 0.00% 0.00% 0.00% 0 CWAN-RP SCP Inpu
437 0 14 0 0.00% 0.00% 0.00% 0 IPHC RP Process
438 0 6 0 0.00% 0.00% 0.00% 0 L2TRACE SERVER
439 0 1 0 0.00% 0.00% 0.00% 0 MLSM-CC Process
440 0 2 0 0.00% 0.00% 0.00% 0 MWAM_CONSOLE Tim
441 0 2 0 0.00% 0.00% 0.00% 0 CWPA QoS
442 0 2 0 0.00% 0.00% 0.00% 0 Const TCAM Mgr R
443 0 2 0 0.00% 0.00% 0.00% 0 Dual active proc
444 8 2 4000 0.00% 0.00% 0.00% 0 BFD Splitb
445 0 1 0 0.00% 0.00% 0.00% 0 ATM HA AC
446 4 2 2000 0.00% 0.00% 0.00% 0 CMFI VPN HA Proc
447 0 1 0 0.00% 0.00% 0.00% 0 Crypto PAS Proc
448 0 1 0 0.00% 0.00% 0.00% 0 Crypto HW Proc
449 0 1 0 0.00% 0.00% 0.00% 0 EPM MAIN PROCESS
450 0 1 0 0.00% 0.00% 0.00% 0 EM Background Pr
451 16 132 121 0.00% 0.00% 0.00% 0 LOCAL AAA
452 0 2 0 0.00% 0.00% 0.00% 0 ENABLE AAA
453 0 2 0 0.00% 0.00% 0.00% 0 LINE AAA
233 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
226 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Version
288 0 1 0 0.00% 0.00% 0.00% 0 SEA_SCP_PRC
457 8 7 1142 0.00% 0.00% 0.00% 0 CEF RP IPC Backg
458 0 1 0 0.00% 0.00% 0.00% 0 CWAN OIR IPC Rea
459 0 2 0 0.00% 0.00% 0.00% 0 Crypto Support
460 4 52 76 0.00% 0.00% 0.00% 0 MPLS Auto Mesh P
461 0 12 0 0.00% 0.00% 0.00% 0 crypto engine pr
462 0 1 0 0.00% 0.00% 0.00% 0 encrypt proc
463 0 7 0 0.00% 0.00% 0.00% 0 Crypto CA
464 0 1 0 0.00% 0.00% 0.00% 0 Crypto PKI-CRL
465 0 1 0 0.00% 0.00% 0.00% 0 Crypto SSL
466 0 35 0 0.00% 0.00% 0.00% 0 Crypto ACL
467 0 1 0 0.00% 0.00% 0.00% 0 Crypto INT
468 0 2 0 0.00% 0.00% 0.00% 0 Crypto IKE Dispa
469 0 3 0 0.00% 0.00% 0.00% 0 Crypto IKMP
470 3844 405097 9 0.00% 0.00% 0.00% 0 IPSEC key engine
471 0 1 0 0.00% 0.00% 0.00% 0 IPSEC manual key
472 0 1 0 0.00% 0.00% -
Hi all,
I'm having high CPU usage with one of my Cisco 3845.
It works as an IP-IP Gateway and the CPU is quite high when the total number of calls only around 100-200 calls.
I check the CPU usage with "show process cpu sort" and it looks like there are some "hidden" processes that consuming CPU.
For example, 41% is total CPU, 25% is due to interrups, so CPU utilization on process level = 41 - 25 = 16%.
But as showed below, processes don't consume that much CPU, only around 7% ???
Please help to advise on this case. Any help is highly appreciated..
Thank you.
3845-GW#show process cpu sort | ex 0.00% 0.00% 0.00%
CPU utilization for five seconds: 41%/25%; one minute: 46%; five minutes: 47%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
382 6619708 1473171 4493 1.59% 1.81% 1.92% 0 CCSIP_SPI_CONTRO
141 4228940 10181955 415 1.35% 1.51% 1.57% 0 IP Input
65 2450824 163102 15026 1.19% 1.16% 1.17% 0 Per-Second Jobs
370 2702292 3709512 728 0.87% 0.88% 0.88% 0 VOIP_RTCP
224 321680 245640 1309 0.47% 0.49% 0.50% 0 AFW_application_
112 93940 18093506 5 0.39% 0.31% 0.32% 0 Ethernet Msec Ti
384 1058280 1553567 681 0.23% 0.28% 0.30% 0 CCSIP_UDP_SOCKET
2 18148 32905 551 0.07% 0.03% 0.02% 0 Load Meter
137 35644 4657843 7 0.07% 0.04% 0.05% 0 IPAM Manager
189 206392 267959 770 0.07% 0.05% 0.07% 0 TCP Protocols
30 30792 198554 155 0.07% 0.01% 0.00% 0 ARP Input
368 145456 176151 825 0.07% 0.04% 0.05% 0 CC-API_VCM
28 9628 32759 293 0.00% 0.01% 0.00% 0 Environmental mo
48 221352 37922 5837 0.00% 0.11% 0.11% 0 Net Background
63 16728 32924 508 0.00% 0.01% 0.00% 0 Compute load avg
64 72080 2781 25918 0.00% 0.01% 0.00% 0 Per-minute Jobs
6 371644 29792 12474 0.00% 0.14% 0.12% 0 Check heaps
176 12216 240288 50 0.00% 0.01% 0.00% 0 CEF: IPv4 proces
284 36416 4929826 7 0.00% 0.02% 0.01% 0 MMON MENG
307 12168 806151 15 0.00% 0.01% 0.00% 0 Atheros LED Ctro
335 35300 19755 1786 0.00% 3.16% 1.00% 708 Virtual Exec
3845-GW#sh int g0/0
GigabitEthernet0/0 is up, line protocol is up
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is XON, input flow-control is XON
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/2/56803 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1551000 bits/sec, 5751 packets/sec
5 minute output rate 4207000 bits/sec, 7643 packets/sec
925128804 packets input, 939078510 bytes, 0 no buffer
Received 62732 broadcasts (0 IP multicasts)
0 runts, 0 giants, 2 throttles
2 input errors, 0 CRC, 0 frame, 2 overrun, 0 ignored
0 watchdog, 3763438515 multicast, 0 pause input
1472816545 packets output, 3214770103 bytes, 0 underruns
0 output errors, 2067720191 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 2281155551 late collision, 0 deferred
2 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
3845-GW#sh int g0/1
GigabitEthernet0/1 is up, line protocol is up
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is XON, input flow-control is XON
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/30335 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1684000 bits/sec, 7697 packets/sec
5 minute output rate 3372000 bits/sec, 5632 packets/sec
1484558664 packets input, 2383177786 bytes, 0 no buffer
Received 208998 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
2 input errors, 0 CRC, 0 frame, 2 overrun, 0 ignored
0 watchdog, 3060386282 multicast, 0 pause input
903478941 packets output, 2814588854 bytes, 0 underruns
0 output errors, 2910776303 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 4157448025 late collision, 0 deferred
2 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped outHas this been something that just recently started happening, or have you had this issue for a while? Have you installed any new programs recently?
You may want to download Glary Utilities, which is a free software(they will ask you if you want to go Pro, just say no, the free version works very well). There is a module for startup manager. You can go in and disable stuff that starts with the computer. I would advise unchecking adobe, java, quicktime, printers, etc. Anything that doesn't REALLY need to start with the computer. The nice thing with Glary is that you can restart the computer, and if you find that you need one of the programs to start with windows, you can go back in and enable it again.
The Celeron 925 processor in your computer is a decent entry level processor, but if there are too many programs running in the background, it can bog down quick. I would also recommend downloading and running Malwarebytes Anti-malware, to be sure that there is nothing malicous running in the background.
Qosmio X875 i7-3630QM, 32GB RAM, OCZ SSD Qosmio X505 i7-920XM, PM55, 16GB RAM, OCZ SSD
Satellite Pro L350 T9900, GM45, 8GB RAM , Intel 320 SSD (my baby) Satellite L655 i7-620M, HM55, 8GB RAM, Intel 710 SSD (travel system) -
High CPU usage CTM message handler (i think)
Hi
I recently reboot my asa 5520, I was trying to remove webvpn listening from my outside nic, even though it wasn't configured !
Since then I am getting these messages
Jan 21 2013 12:00:59: %ASA-4-402127: CRYPTO: The ASA is skipping the writing of latest Crypto Archive File as the maximum # of files ( 2 ) allowed have been written to < disk0:/crypto_archive >. Please archive & remove files from < disk0:/crypto_archive > if you want more Crypto Archive Files saved
Jan 21 2013 12:00:59: %ASA-4-402124: CRYPTO: The ASA hardware accelerator encountered an error (HWErrAddr= 0x0, Core= 226, HwErrCode= 32, IstatReg= 0x2, PciErrReg= 0x0, CoreErrStat= 0x8, CoreErrAddr= 0x48804F90, Doorbell Size[0]= 1024, DoorBell Outstanding[0]= 0, Doorbell Size[1]= 0, DoorBell Outstanding[1]= 0, SWReset= 42418851)
Jan 21 2013 12:00:59: %ASA-4-402127: CRYPTO: The ASA is skipping the writing of latest Crypto Archive File as the maximum # of files ( 2 ) allowed have been written to < disk0:/crypto_archive >. Please archive & remove files from < disk0:/crypto_archive > if you want more Crypto Archive Files saved
Jan 21 2013 12:01:03: %ASA-4-402124: CRYPTO: The ASA hardware accelerator encountered an error (HWErrAddr= 0x0, Core= 226, HwErrCode= 32, IstatReg= 0x2, PciErrReg= 0x0, CoreErrStat= 0x8, CoreErrAddr= 0x48804F90, Doorbell Size[0]= 1024, DoorBell Outstanding[0]= 0, Doorbell Size[1]= 0, DoorBell Outstanding[1]= 0, SWReset= 42418852)
I have removed the archive files but it still keeps complaining !
show processes cpu-usage shows
0x08158ce1 0x6ddb69f4 57.1% 57.1% 57.1% CTM message handler
when i reloaded the asa I used the fast-reload option
I can't seem to find anything about this process or maybe how to kill it or ...
Cisco Adaptive Security Appliance Software Version 8.4(1)
Device Manager Version 6.4(1)
show processes cpu-hog
Process: Dispatch Unit, NUMHOG: 18030, MAXHOG: 49, LASTHOG: 2
LASTHOG At: 12:10:30 AEDT Jan 21 2013
PC: 0x081ecc51 (suspend)
Call stack: 0x081ecc51 0x080680cc
Process: Dispatch Unit, PROC_PC_TOTAL: 6027722, MAXHOG: 268, LASTHOG: 3
LASTHOG At: 12:10:39 AEDT Jan 21 2013
PC: 0x081ececa (suspend)
Process: Dispatch Unit, NUMHOG: 6026714, MAXHOG: 268, LASTHOG: 3
LASTHOG At: 12:10:39 AEDT Jan 21 2013
PC: 0x081ececa (suspend)
Call stack: 0x081ececa 0x080680cc
Process: CTM message handler, NUMHOG: 28781099, MAXHOG: 328, LASTHOG: 128
LASTHOG At: 12:10:40 AEDT Jan 21 2013
PC: 0x08158ce1 (suspend)
Call stack: 0x080680cc
Process: CTM message handler, PROC_PC_TOTAL: 42421460, MAXHOG: 329, LASTHOG: 128
LASTHOG At: 12:10:40 AEDT Jan 21 2013
PC: 0x08158ce1 (suspend)
I was planning to do another reload without the fast reload option but I am not sure that will solve the problem !
AlexSounds like a bug to me, i would open a TAC case to get it further troubleshot.
-
Seeing ASA 5540 with High CPU Utilization
I have a remote site customer with a Cisco ASA 5540 running SSLVPN (Anyconnect)(8.03). It currently only serves about 450 SSLVPN clients. Since last friday, they've seen the CPU utilization go up to high 90% while only serving 400+ remote users. Please help us troubleshoot the root cause of the CPU high utilization on Cisco ASA 5540.
This doc is a good starting point:
http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a008009491c.shtml -
Switch High cpu Catalyst 4500 Version 12.2(53)SG5
I need help trying to understand the following.
%CPU %CPU RunTimeMax Priority Average %CPU Total
Target Actual Target Actual Fg Bg 5Sec Min Hour CPU
GalChassisVp-review 3.00 0.10 10 45 100 500 0 0 0 701:22
K2AclCamMan hw stats 3.00 1.48 10 5 100 500 0 0 0 3265:20
K2AclCamMan kx stats 1.00 0.10 10 5 100 500 0 0 0 1917:38
K2L2 Address Table R 2.00 72.93 12 5 100 500 95 87 66 458521:03
K2PortMan Review 3.00 4.73 15 11 100 500 5 5 4 28199:47
K2Fib Consistency Ch 1.00 0.19 5 2 100 500 0 0 0 1082:02
K2FibPbr route map r 2.00 0.21 20 5 100 500 0 0 0 2031:50
K2QosDblMan Rate DBL 2.00 0.11 7 1 100 500 0 0 0 662:26
K2 VlanStatsMan Revi 2.00 0.39 15 2 100 500 0 0 0 2062:11
K2PacketBufMonitor-P 3.00 2.76 10 1 100 500 2 2 2 13968:10
RkiosPortMan Port Re 2.00 0.30 12 29 100 500 0 0 0 1744:23
RkiosIpPbr IrmPort R 2.00 0.22 10 3 100 500 0 0 0 342:05
Grandprix 1-1 Stub R 2.00 0.17 15 5 100 500 0 0 0 1244:55
Grandprix 1-2 Stub R 2.00 0.14 15 5 100 500 0 0 0 866:57
Grandprix 1-3 Stub R 2.00 0.13 15 5 100 500 0 0 0 840:36
Grandprix 1-4 Stub R 2.00 0.16 15 5 100 500 0 0 0 846:55
Grandprix 1-5 Stub R 2.00 0.14 15 5 100 500 0 0 0 855:31
Grandprix 1-6 Stub R 2.00 0.15 15 124 100 500 0 0 0 1051:18
Grandprix 1-1 Stats 2.00 0.64 4 2 100 500 0 0 0 3151:58
Grandprix 1-2 Stats 2.00 0.48 4 2 100 500 0 0 0 675:30
Grandprix 1-3 Stats 2.00 0.20 4 2 100 500 0 0 0 567:34
Grandprix 1-6 Stats 2.00 0.35 4 2 100 500 0 0 0 2072:59
%CPU Totals 241.20 101.19
Allocation ceiling Current allocation
kbytes % in use kbytes % in use
Linecard 1's Store 258.00 62% 161.12 100%
TSM objects ------------------ ------------------
PacketInfoItem 859.37 0% 0.64 0%
VbufNodes1600 55.50 0% 5.20 0%
VbufNodes400 288.00 0% 1.12 50%
PacketBufRaw 25219.50 100% 25219.50 100%
PacketBufRawJumbo 184.29 100% 184.29 100%
Packet 2079.42 0% 54.10 0%
PimPhyports 1054.68 5% 54.84 100%
PimPorts 906.25 11% 105.12 100%
PimModules 162.00 0% 0.63 100%
PimChassis 38.37 6% 2.39 100%
EbmVlans 5088.00 1% 67.07 100%
EbmPorts 344.00 11% 38.63 100%
IrmVrfs 315.00 0% 2.46 50%
IrmFibEntries 12288.00 0% 0.75 6%
IrmMfibEntryMemMan 6656.00 0% 0.10 100%
Acl 1536.00 0% 1.87 100%
Ace24 9215.85 0% 4.92 100%
Ace48 15359.76 0% 0.35 100%
AclListNode 256.00 0% 0.30 100%
AclClassifierActionL 1152.00 0% 1.40 87%
CommandTables 48.00 22% 10.59 100%
K2FibVrfs 152.00 0% 1.18 50%
K2TxPacketInfo 256.00 0% 0.25 0%
Packets Dropped In Hardware By CPU Subport (txQueueNotAvail)
CPU Subport TxQueue 0 TxQueue 1 TxQueue 2 TxQueue 3
1 0 0 0 822
2 0 7538177 0 0
RkiosSysPacketMan:
Packet allocation failures: 0
Packet Buffer(Software Common) allocation failures: 0
Packet Buffer(Software ESMP) allocation failures: 0
Packet Buffer(Software EOBC) allocation failures: 0
Packet Buffer(Software SupToSup) allocation failures: 0
IOS Packet Buffer Wrapper allocation failures: 0
Packets Dropped In Processing Overall
Total 5 sec avg 1 min avg 5 min avg 1 hour avg
881 0 0 0 0
Packets Dropped In Processing by CPU event
Event Total 5 sec avg 1 min avg 5 min avg 1 hour avg
SA Miss 874 0 0 0 0
Packets Dropped In Processing by Priority
Priority Total 5 sec avg 1 min avg 5 min avg 1 hour avg
Normal 7 0 0 0 0
Medium 881 0 0 0 0
Packets Dropped In Processing by Reason
Reason Total 5 sec avg 1 min avg 5 min avg 1 hour avg
SrcAddrTableFilt 87 0 0 0 0
STPDrop 771 0 0 0 0
L2DstDrop 21 0 0 0 0
NoDstPorts 2 0 0 0 0
Total packet queues 16
Packets Received by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
L2/L3Control 7269264609 236 268 214 205
Host Learning 3617387874 38 49 38 53
L3 Fwd Low 21851501 0 0 0 0
L2 Fwd Low 6786759 0 0 0 0
L3 Rx High 1582 0 0 0 0
L3 Rx Low 10306 0 0 0 0
Packets Dropped by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
L2/L3Control 73992 0 0 0 0
Host Learning 7561249 0 0 0 0
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
L2/L3Control 7269264609 236 268 214 205
PacketBufRaw 25219.50 100% 25219.50 100%
PacketBufRawJumbo 184.29 100% 184.29 100%
Both of these seem a bit high don't you think?
If you have any idea as to why this is happening. Or is this normal?
ThanksHi,
This could be related to L2 issues like a loop, see that the K2L2 Address Table R process has a target of 2.00% and is currently running at 72.93%, which is way over the expected target, this can be an indication of host/mac flapping issues. Also the host learning queue seems very high and the L2/L3 control.
You can enable mac notification mac move command to see if on the logs are any flapping issues. Also below is a link which is very helpful when dealing with high cpu issues on 4500.
http://www.cisco.com/c/en/us/support/docs/switches/catalyst-4000-series-switches/65591-cat4500-high-cpu.html
Hope this helps. -
Could you explain please reason first ACL,log unreach.
No log key in ACL, switch has a lot of VLAN with ACL in.
Packets Received by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
L2/L3Control 236758309 36 34 35 34
Host Learning 6011600 0 0 0 0
L3 Fwd Low 2424793938 2 0 1 8
L2 Fwd Low 201893895 34 33 36 43
L3 Rx Low 34737942 19 3 3 5
RPF Failure 494739 0 0 0 0
ACL log, unreach 27408733812 2770 13291 11964 6553
ACL sw processing 520 0 0 0 0
Packets Dropped by Packet Queue
Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg
L2/L3Control 1509428 0 0 0 0
Host Learning 24358337 0 0 0 0
L3 Fwd Low 32625 0 0 0 0
L2 Fwd Low 29248805 0 0 0 0
L3 Rx Low 19362 0 0 0 0
ACL log, unreach 4465098816 5 410 345 339
Thank youLooks to be like some issue iwth your ACL.
To help isolate the issue, I would request you to answer the following
questions:
1. Since how long the device has been running with high Cpu?
2. Have you made any hardware/software or configuration changes on these
devices which lead to high CPU utilization?
To isolate the issue, I would request you to send me the following:
1. Show proc cpu sorted | ex 0.00 (3-4 times)
2. Show proc cpu history
3. Show platform cpu packet statistics all
4. Show platform health
Additionally I would also like you to:
#debug platform packet all receive buffer
Wait 30 seconds and capture the command below:
#show platform cpu packet buffered (Twice)
#undebug all
Note: All these outputs should be taken while the CPU indicates high.
Please see a very handy document on handling high CPU on 4500 switches
http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note0918
6a00804cef15.shtml
HTH -
High CPU Utilization on ASA 5540
I have a remote site customer with a Cisco ASA 5540 running SSLVPN (Anyconnect)(8.03). It currently only serves about 450 SSLVPN clients. Since last friday, they've seen the CPU utilization go up to high 90% while only serving 400+ remote users. I saw some high cpu utilization bugs, but none looked to be relevant. Any ideas on how I can find the root cause of the CPU high utilization?
Hi rlortiz,
I ran into this issue as well on an ASA 5540 with only about 150 users. In the case if you are using large modulus operations including large key size certificates and a higher Diffie-Hellman group, it will cause for high processing.
Since the default method of processing these operations is software-based, it will cause higher CPU usage and also slower SSL/IPsec connection establishment.
If this is the scenario for you, use hardware-based processing by using the following configuration:
"crypto engine large-mod-accel" -
Cisco ASA 5505 Failover issue..
Hi,
I am having two firewalls (cisco ASA 5505) which is configured as active/standby Mode.It was running smoothly for more than an year,but last week the secondary firewall got failed and It made my whole network down.then I just removed the connectivity of the secondary firewall and run only the primary one.when I login by console i found out that the failover has been disabled .So again I connected to the Network and enabled the firewall.After a couple of days same issue happen.This time I take down the Secondary firewall erased the Flash.Reloaded the IOS image.Configured the failover and connected to the primary for the replication of configs.It found out the Active Mate.Replicated the configs and got synced...But after sync the same thing happened,The whole network gone down .I juz done the same thing removed the secondary firewall.Network came up.I feel there is some thing with failover thing ,but couldnt fin out :( .And the firewalls are in Router Mode.Please find the logs...
Secondary Firewall While Sync..
cisco-asa(config)# sh failover
Failover On
Failover unit Secondary
Failover LAN Interface: e0/7 Vlan3 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 4 of 23 maximum
Version: Ours 8.2(5), Mate 8.2(5)
Last Failover at: 06:01:10 GMT Apr 29 2015
This host: Secondary - Sync Config
Active time: 55 (sec)
slot 0: ASA5505 hw/sw rev (1.0/8.2(5)) status (Up Sys)
Interface outside (27.251.167.246): No Link (Waiting)
Interface inside (10.11.0.20): No Link (Waiting)
Interface mgmt (10.11.200.21): No Link (Waiting)
slot 1: empty
Other host: Primary - Active
Active time: 177303 (sec)
slot 0: ASA5505 hw/sw rev (1.0/8.2(5)) status (Up Sys)
Interface outside (27.251.167.247): Unknown (Waiting)
Interface inside (10.11.0.21): Unknown (Waiting)
Interface mgmt (10.11.200.22): Unknown (Waiting)
slot 1: empty
=======================================================================================
Secondary Firewall Just after Sync ,Active (primary Firewall got rebootted)
cisco-asa# sh failover
Failover On
Failover unit Secondary
Failover LAN Interface: e0/7 Vlan3 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 4 of 23 maximum
Version: Ours 8.2(5), Mate Unknown
Last Failover at: 06:06:12 GMT Apr 29 2015
This host: Secondary - Active
Active time: 44 (sec)
slot 0: ASA5505 hw/sw rev (1.0/8.2(5)) status (Up Sys)
Interface outside (27.251.167.246): Normal (Waiting)
Interface inside (10.11.0.20): No Link (Waiting)
Interface mgmt (10.11.200.21): No Link (Waiting)
slot 1: empty
Other host: Primary - Not Detected
Active time: 0 (sec)
slot 0: empty
Interface outside (27.251.167.247): Unknown (Waiting)
Interface inside (10.11.0.21): Unknown (Waiting)
Interface mgmt (10.11.200.22): Unknown (Waiting)
slot 1: empty
==========================================================================================
After Active firewall got rebootted failover off,whole network gone down.
cisco-asa# sh failover
Failover Off
Failover unit Secondary
Failover LAN Interface: e0/7 Vlan3 (up)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 4 of 23 maximum
===========================================================================================
Primary Firewall after rebootting
cisco-asa# sh failover
Failover On
Failover unit Primary
Failover LAN Interface: e0/7 Vlan3 (Failed - No Switchover)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 4 of 23 maximum
Version: Ours 8.2(5), Mate Unknown
Last Failover at: 06:17:29 GMT Apr 29 2015
This host: Primary - Active
Active time: 24707 (sec)
slot 0: ASA5505 hw/sw rev (1.0/8.2(5)) status (Up Sys)
Interface outside (27.251.167.246): Normal (Waiting)
Interface inside (10.11.0.20): Normal (Waiting)
Interface mgmt (10.11.200.21): Normal (Waiting)
slot 1: empty
Other host: Secondary - Failed
Active time: 0 (sec)
slot 0: empty
Interface outside (27.251.167.247): Unknown (Waiting)
Interface inside (10.11.0.21): Unknown (Waiting)
Interface mgmt (10.11.200.22): Unknown (Waiting)
slot 1: empty
cisco-asa# sh failover history
==========================================================================
From State To State Reason
==========================================================================
06:16:43 GMT Apr 29 2015
Not Detected Negotiation No Error
06:17:29 GMT Apr 29 2015
Negotiation Just Active No Active unit found
06:17:29 GMT Apr 29 2015
Just Active Active Drain No Active unit found
06:17:29 GMT Apr 29 2015
Active Drain Active Applying Config No Active unit found
06:17:29 GMT Apr 29 2015
Active Applying Config Active Config Applied No Active unit found
06:17:29 GMT Apr 29 2015
Active Config Applied Active No Active unit found
==========================================================================
cisco-asa#
cisco-asa# sh failover state
State Last Failure Reason Date/Time
This host - Primary
Active None
Other host - Secondary
Failed Comm Failure 06:17:43 GMT Apr 29 2015
====Configuration State===
====Communication State===
==================================================================================
Secondary Firewall
cisc-asa# sh failover h
==========================================================================
From State To State Reason
==========================================================================
06:16:32 GMT Apr 29 2015
Not Detected Negotiation No Error
06:17:05 GMT Apr 29 2015
Negotiation Disabled Set by the config command
==========================================================================
cisco-asa# sh failover
Failover Off
Failover unit Secondary
Failover LAN Interface: e0/7 Vlan3 (down)
Unit Poll frequency 1 seconds, holdtime 15 seconds
Interface Poll frequency 5 seconds, holdtime 25 seconds
Interface Policy 1
Monitored Interfaces 4 of 23 maximum
ecs-pune-fw-01# sh failover h
==========================================================================
From State To State Reason
==========================================================================
06:16:32 GMT Apr 29 2015
Not Detected Negotiation No Error
06:17:05 GMT Apr 29 2015
Negotiation Disabled Set by the config command
==========================================================================
cisco-asa# sh failover state
State Last Failure Reason Date/Time
This host - Secondary
Disabled None
Other host - Primary
Not Detected None
====Configuration State===
====Communication State===
Thanks...
Maybe you are looking for
-
How can I get my InDesign app to update now that it has changed to 'canceling'?
I am new to Creative Cloud. This is just another example of why I feel so much discontent. I just touched the app to get it to stop. Well, it DID! The second app just changed to 'update'. This old lady needs someone to help me. Please. This is just
-
How to design a template without Template Builder in Word 2007?
How can i design a template in Word 2007 without Template Builder, since i am using Vista that does not support BI Publisher Desktop. Is it possible to desgin a template in Open Office ?
-
Why do we have to activate individually all third party apps!!! There should be as easier way!
Why do we have to activate individually all third party apps!!! There should be as easier way! Apple TV is not Apple anymore...
-
Indd CS3 Hyperlinks in placed Indd not available
Create indd file (a) and create a hyperlink to a URL in it. Create indd file (b) and place file (a) in it, like you would a peice of art. The hyperlink is not active and does not carry over when a PDF is made. We create large books with many ads in t
-
Lightroom 2 Crashed - Please Help!
Hi, Lightroom 2 Crashed and caused a Question Mark (?) to appear on all the files I have loaded since Lightroom Crashed. For the past three (3) frustrating days I can not process any RAW files because Lightroom will not see the location of the files