6880 - VSS - C6880-X-16P10G
Ciao,
in order to create a VSS cluster is it mandatory to put the C6880-X-16P10G line card in performance mode ?
BR,
Luciano
Hi
I was not able to find doc against the abover module, but found under config guide for 15.1SY that recommendation is to put them in performance mode as below for VSL links
http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/15-1SY/config_guide/sup2T/15_1_sy_swcg_2T/virtual_switching_systems.html#pgfId-1053927
Note • When using the ports on a switching module that can operate in oversubscribed mode as VSL links, you must operate the ports in performance mode, not in oversubscription mode. Enter the no hw-module switch x slot y oversubscription port-group num command when configuring the switching module. If you enter the no hw-module switch switch_number slot slot_number oversubscription command to configure non-oversubscription mode (performance mode), then only ports 1, 5, 9, and 13 are configurable; the other ports on the module are disabled.
Port-groups are independent of each other and one or more port-groups can operate in non-oversubscribed mode for VSL with the unused ports administratively shutdown, while the others can still operate in oversubscribed mode.
AAAnd for 6880 I see the below as 2:1 oversubcribed
Ovversubsription 16P 10G at 2:1 in oversubscription mode (default) for base module and port cards 8P 10G at 1:1 in performance mode (configurable) for base module and port cards
Hope this helps.
Ho
Similar Messages
-
6880 in VSS Mode and OSPF maximum paths
Hi Folks,
I have an issue observed in testing. We have implemented a VSS solution on the 6880-X-LE. We have two MEC L3 PO's upstream and thus two OSPF path's for our default route. After creating an event in the lab to put the boxes in Active - Active (pulled VSL links off SW1) and then go through the process of restoring the VSS (put VSL links back in) I noticed my additional path did not restore.
From the restoration perspective, it went as follows:
Sw1 - Active SW2-standby - pull VSL links
VSL PO goes down
SW1 - Active SW2 - Active (sub second traffic impact)
SW1 - enters recovery mode SW2 - Active
VSL links back in - VSL PO up
SW1 - reboots and comes up standby SW2 - Active
Routing table now only shows entry for PO200 and no routes can go out PO100.
6880#sh ip route
<cut>
0.0.0.0/0 [110/1000] via 192.168.0.253, 00:38:34, Port-channel200
1.0.0.0/24 is subnetted, 2 subnets
O 1.1.1.0 [110/151] via 10.86.50.253, 00:31:55, Port-channel200
O 1.1.2.0 [110/151] via 10.86.50.253, 00:31:55, Port-channel200
192.168.0.0/16 is variably subnetted, 59 subnets, 8 masks
O 192.168.1.0/25 [110/150] via 1192.168.0.253, 00:31:55, Port-channel200
O 192.168.1.128/25 [110/350] via 192.168.0.253, 00:31:55, Port-channel200
O 192.168..3.0/25 [110/350] via 192.168.0.253, 00:31:55, Port-channel200
6880#sh ip ospf ne
Neighbor ID Pri State Dead Time Address Interface
192.168.0.253 0 FULL/ - 00:00:31 192..168.50.253 Port-channel200
192.168.0.251 0 FULL/ - 00:00:32 192.168.50.251 Port-channel100
Can anyone help?
Thanks,
RashHi Reza,
I can't post the configuration. However, I can tell you that I have a TAC case open with Cisco and they agree with me that this behavior requires further investigation. So the "PO" in question disappears seems to match when I get the error related to the a port-member not being compatible:
Jan 26 15:18:17 est: %EC-SW2_STBY-5-CANNOT_BUNDLE2: Te2/3/14 is not compatible with Te1/5/14 and will be suspended (speed of Te2/3/14 is 10G, Te1/5/14 is 1000M)
So what happens is that in PO100 you'll see only one bundled port and the other is suspended. This causes OSPF to withdraw the PO from the routing table (hypothesis but reproducible).
In our trial production launch, I performed the same test above redundancy force-switchover and you'll see both entries in the routing-table. Once the other modules come up and I get a suspended port-member that particular PO disappears from the routing table.
If you shut/no shut that PO, the PO returns as an available path in the routing table. In production we are not using GLC-T (as in the lab) but are using GLC-SX-MM. The bug that puts this member into a suspended state falls under the this bug id:
CSCur17071
What I missed initially is the fact, that the routing table removes that PO from being an available path even though it still has an active port-member.
Cheers,
Rash -
Issues with deploying 6880-X in VSS Mode
Hi Everyone,
I wanted to query the support community to see if folks are encountering issues in deploying the 6880-X in VSS mode. We first looked at this soluition when it was running 15.1(2) SY2 as the recommened GA code. We ran into several issues:
1) VSL Link Timeout - Workaround:modify LMP timer or upgrade to 15.1(2)SY4
2) Port-member goes into suspendstate when using 1000BASE-SX - shut/no shut suspended interface
We ran into this and some other issues even with the eFSU upgrade process. So my question is have you run into similiar difficulties in running the 6880 in VSS mode. Did you go ahead and deploy? If so did you deploy with 15.1(2)SY4 (new Cisco GA code) or did you live with the state you are in?
Were there ways that you mitigated these issues?
Thanks,
PatnaikHi Reza,
Here you go. Just so you know I resolved the suspended port-member by OIR the optic that was displayed as a 10G interface.
lab6880#sh run inter te1/3/14
Building configuration...
Current configuration : 1546 bytes
interface TenGigabitEthernet1/3/14
no switchport
mtu 9216
no ip address
no ip proxy-arp
platform qos queue-mode mode-dscp
wrr-queue bandwidth 100 150 125 150 0 0 100
wrr-queue queue-limit 20 40 10 10 0 0 5
wrr-queue random-detect min-threshold 1 50 60 80 100
wrr-queue random-detect min-threshold 2 70 100 100 100
wrr-queue random-detect min-threshold 3 70 100 100 100
wrr-queue cos-map 4 1 3
wrr-queue cos-map 5 1 4
wrr-queue dscp-map 1 1 1 2 3 4 5 6 7 8
wrr-queue dscp-map 1 1 9 11 13 15 16 17 19 21
wrr-queue dscp-map 1 1 23 25 27 29 31 33 35 37
wrr-queue dscp-map 1 1 39 41 42 43 44 45 47 49
wrr-queue dscp-map 1 1 50 51 52 53 54 55 57 58
wrr-queue dscp-map 1 1 59 60 61 62 63
wrr-queue dscp-map 1 2 0
wrr-queue dscp-map 2 1 10 12 14
wrr-queue dscp-map 3 1 18 20 22
wrr-queue dscp-map 3 2 26 28 30
wrr-queue dscp-map 4 1 24 32 34 36 38
wrr-queue dscp-map 7 1 48 56
rcv-queue dscp-map 1 1 1 2 3 4 5 6 7 8
rcv-queue dscp-map 1 1 9 11 13 15 17 19 21 23
rcv-queue dscp-map 1 1 25 27 29 31 33 35 37 39
rcv-queue dscp-map 1 1 41 42 43 44 45 47 49 50
rcv-queue dscp-map 1 1 51 52 53 54 55 57 58 59
rcv-queue dscp-map 1 1 60 61 62 63
rcv-queue dscp-map 1 2 0
rcv-queue dscp-map 1 3 16
rcv-queue dscp-map 2 1 10 12 14
rcv-queue dscp-map 3 1 18 20 22
rcv-queue dscp-map 3 2 26 28 30
rcv-queue dscp-map 4 1 24 32 34 36 38
rcv-queue dscp-map 7 1 48 56
macro description UPLINK
arp timeout 300
channel-group 100 mode active
end
lab6880#sh run inter te2/5/14
Building configuration...
Current configuration : 1546 bytes
interface TenGigabitEthernet2/5/14
no switchport
mtu 9216
no ip address
no ip proxy-arp
platform qos queue-mode mode-dscp
wrr-queue bandwidth 100 150 125 150 0 0 100
wrr-queue queue-limit 20 40 10 10 0 0 5
wrr-queue random-detect min-threshold 1 50 60 80 100
wrr-queue random-detect min-threshold 2 70 100 100 100
wrr-queue random-detect min-threshold 3 70 100 100 100
wrr-queue cos-map 4 1 3
wrr-queue cos-map 5 1 4
wrr-queue dscp-map 1 1 1 2 3 4 5 6 7 8
wrr-queue dscp-map 1 1 9 11 13 15 16 17 19 21
wrr-queue dscp-map 1 1 23 25 27 29 31 33 35 37
wrr-queue dscp-map 1 1 39 41 42 43 44 45 47 49
wrr-queue dscp-map 1 1 50 51 52 53 54 55 57 58
wrr-queue dscp-map 1 1 59 60 61 62 63
wrr-queue dscp-map 1 2 0
wrr-queue dscp-map 2 1 10 12 14
wrr-queue dscp-map 3 1 18 20 22
wrr-queue dscp-map 3 2 26 28 30
wrr-queue dscp-map 4 1 24 32 34 36 38
wrr-queue dscp-map 7 1 48 56
rcv-queue dscp-map 1 1 1 2 3 4 5 6 7 8
rcv-queue dscp-map 1 1 9 11 13 15 17 19 21 23
rcv-queue dscp-map 1 1 25 27 29 31 33 35 37 39
rcv-queue dscp-map 1 1 41 42 43 44 45 47 49 50
rcv-queue dscp-map 1 1 51 52 53 54 55 57 58 59
rcv-queue dscp-map 1 1 60 61 62 63
rcv-queue dscp-map 1 2 0
rcv-queue dscp-map 1 3 16
rcv-queue dscp-map 2 1 10 12 14
rcv-queue dscp-map 3 1 18 20 22
rcv-queue dscp-map 3 2 26 28 30
rcv-queue dscp-map 4 1 24 32 34 36 38
rcv-queue dscp-map 7 1 48 56
macro description UPLINK
arp timeout 300
channel-group 100 mode active
end
interface Port-channel100
no switchport
dampening 20
mtu 9216
ip address 172.16.2.250 255.255.255.254
ip pim sparse-dense-mode
ip ospf network point-to-point
logging event link-status
logging event nfas-status
logging event bundle-status
load-interval 30
carrier-delay msec 0 -
6880-X VSS dual-active detection with Enhanced PAgP
Hello,
I plan to use Enhanced PAgP for dual-active detection for a VSS consisting of 6880-X.
Has anybody tested it?
What about the 6800IA switch - does it have a similar feature that can help the active switch in the VSS to find out if the former standby has become also active?
Thanks, kind regards,
Eduard Gheorghiuhey, i ran this up in the lab, the FEX itself runs dual active detection without the need for PAGP.
see below info:
switch#show fex dual-active
FEX dual-active detection enabled: Yes
Channel Group 101 (FEX 101)
Splitbrain detection capable: Yes
switch 1 member port state:
Te1/5/2 - Interface Up. state: bound
Te1/5/3 - Interface Up. state: bound
switch 2 member port state:
Te2/5/2 - Interface Up. state: bound
Te2/5/3 - Interface Up. state: bound
Channel Group 102 (FEX 102)
Splitbrain detection capable: Yes
switch 1 member port state:
Te1/5/4 - Interface Up. state: bound
switch 2 member port state:
Te2/5/4 - Interface Up. state: bound
switch#show switch virtual dual-active summary
Pagp dual-active detection enabled: Yes
Fast-hello dual-active detection enabled: Yes
FEX dual-active detection enabled: Yes
No interfaces excluded from shutdown in recovery mode
In dual-active recovery mode: No
switch#
switch#show fex
FEX FEX FEX FEX
Number Description State Model Serial
101 FEX0101 online C6800IA-48FPD FOC1740Y33Y
102 FEX0102 online C6800IA-48FPD FOC1741S58T -
Hello, can someone tell me where i can find a document how to upgrade a cisco 6880 running vss
I copied the ios to bootdisk, but when I change the boot system and reload my standby peer de standby peer starts up in rommon mode. I need the correct parameters for the boot command for a 6880. On the cisco site I can only find how to upgrade a 6500 with vss,but I need the documentation how to upgrade a cisco 6880. Can someone please help me to find this . Thank you.
There are several methods. Both methods has downtime. One method is FSU/eFSU, which I will NOT go into, is an "automated" method of upgrade. The automation, if it works, include booting each chassis one at a time.
The second method works well for me. Before I begin, it is very, very important that you check the MD5 hash value of the IOS file you've downloaded against the MD5 hash value found in the Cisco website.
1. Copy the IOS to the sup-bootdisk:.
2. Copy the IOS to the slavesup-bootdisk:
3. Change the boot variable string to point to the IOS of your choice. Additionally, you can specify a second boot variable string to point to your OLD IOS. This is very handy/important if your chassis attempts to boot the new IOS you've copied and failed. It is important to remove the old boot variable string before changing.
4. Ensure your config-registry is 0x2102.
5. Save the config first and copy the startup-config to an external media: CF or TFTP
6. Reboot the chassis.
Hope this helps. -
Hi Guys,
I have a head scratcher that I'm dealing with. I have 2x (2 month) 6880-x switches configured with vss with 2 vsl links on each of its sup cards and switch 1 is the active switch on the vss. Both 6880x chassis also have 2 additional 10g modules each. I am dealing with 2 issues in this setup.
The first issue is, i have a 3750x with a 10g module thats port channeled to the vss with one 10g link going to each of the chassis. Within the first 3 days it started dropping the link that's plugged to switch1, then comes back on right away. It does it randomly throughout the day. I have changed out the optics on both ends, changed the cable, and also changed the module on the 3750x and I still get the same issue. The strange part is that when switch2 on the vss becomes the active switch, the problem goes away. I did notice this before changing out the optics, cable, module on the 3750x.
The second issue is the one I'm been trying to figure out along with cisco TAC. This started about a month after bring the vss online. When switch1 was the active switch in the vss, every couple of days vsl links drop one at a time, eventually killing the vss and putting the standby unit into recovery mode because of the dual active detection. once I reboot the standby switch, the vss comes back up normally. it did this a couple of times until I decided to force switch2 to be the active switch on the vss. when switch2 became active, the vss was stable for about a month then the vsl links died again, and the system failed over to switch1. after looking at the logs with cisco tac, we see that the vsl links stop responding which causes the failover, but up until now we still can't determine what is causing the vsl links to fail. cisco tac said that maybe the vsl links were being overloaded but we have been monitoring the bandwidth utilization on the vsl links and they never go beyond 1% utilization. The last suggestion by cisco tac was to add another vsl link but through another module other than the sup. This was done a couple of days ago so now i'm waiting to see if the vsl links fail again and since switch1 is the active switch on the vss, i'm having to deal with the first issue above with the 3750x.
I've included some log entries from the dropped port-channel member and also logs for when the vsl links fail.
Logs for 3750x port-channel member drops.
*Jul 5 05:27:10 PDT: %LINEPROTO-SW1-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/2/5, changed state to down
*Jul 5 05:27:10 PDT: %LINK-SW1-3-UPDOWN: Interface TenGigabitEthernet2/2/5, changed state to down
*Jul 5 05:27:10 PDT: %LINK-SW1-3-UPDOWN: Interface TenGigabitEthernet2/2/5, changed state to up
*Jul 5 05:28:10 PDT: %LINEPROTO-SW1-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/2/5, changed state to up
*Jul 6 11:28:52 PDT: %LINEPROTO-SW1-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/2/5, changed state to down
*Jul 6 11:28:52 PDT: %LINK-SW1-3-UPDOWN: Interface TenGigabitEthernet2/2/5, changed state to down
*Jul 6 11:28:53 PDT: %LINK-SW1-3-UPDOWN: Interface TenGigabitEthernet2/2/5, changed state to up
*Jul 6 11:29:53 PDT: %LINEPROTO-SW1-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/2/5, changed state to up
*Jul 7 06:27:25 PDT: %LINEPROTO-SW1-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/2/5, changed state to down
*Jul 7 06:27:25 PDT: %LINK-SW1-3-UPDOWN: Interface TenGigabitEthernet2/2/5, changed state to down
*Jul 7 06:27:26 PDT: %LINK-SW1-3-UPDOWN: Interface TenGigabitEthernet2/2/5, changed state to up
*Jul 7 06:27:33 PDT: %LINEPROTO-SW1-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/2/5, changed state to up
*Jul 7 06:33:43 PDT: %LINEPROTO-SW1-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/2/5, changed state to down
*Jul 7 06:33:43 PDT: %LINK-SW1-3-UPDOWN: Interface TenGigabitEthernet2/2/5, changed state to down
*Jul 7 06:33:43 PDT: %LINK-SW1-3-UPDOWN: Interface TenGigabitEthernet2/2/5, changed state to up
*Jul 7 06:34:43 PDT: %LINEPROTO-SW1-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/2/5, changed state to up
*Jul 7 07:38:35 PDT: %LINEPROTO-SW1-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/2/5, changed state to down
*Jul 7 07:38:35 PDT: %LINK-SW1-3-UPDOWN: Interface TenGigabitEthernet2/2/5, changed state to down
*Jul 7 07:38:36 PDT: %LINK-SW1-3-UPDOWN: Interface TenGigabitEthernet2/2/5, changed state to up
log for VSL link failures
*Jun 27 23:59:25 PDT: %VSLP-SW2-3-VSLP_LMP_FAIL_REASON: Te2/5/15: Link down
*Jun 27 23:59:25 PDT: %VSL-SW2-5-VSL_CNTRL_LINK: New VSL Control Link 2/5/16
*Jun 27 23:59:25 PDT: %LINEPROTO-SW2-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/5/15, changed state to down
*Jun 27 23:59:25 PDT: %LINEPROTO-SW2-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/5/15, changed state to down
*Jun 27 23:59:25 PDT: %LINK-SW2-3-UPDOWN: Interface TenGigabitEthernet2/5/15, changed state to down
*Jun 27 23:59:25 PDT: %VSLP-SW2-3-VSLP_LMP_FAIL_REASON: Te2/5/16: Link down
*Jun 27 23:59:25 PDT: %VSLP-SW2-2-VSL_DOWN: Last VSL interface Te2/5/16 went down
*Jun 27 23:59:25 PDT: %VSLP-SW2-2-VSL_DOWN: All VSL links went down while switch is in ACTIVE role
*Jun 27 23:59:25 PDT: %LINEPROTO-SW2-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/5/16, changed state to down
*Jun 27 23:59:25 PDT: %LINEPROTO-SW2-5-UPDOWN: Line protocol on Interface Port-channel2, changed state to down
*Jun 27 23:59:25 PDT: %LINK-SW2-3-UPDOWN: Interface Port-channel2, changed state to down
*Jun 27 23:59:25 PDT: %LINK-SW2-3-UPDOWN: Interface TenGigabitEthernet2/5/16, changed state to down
*Jun 27 23:59:25 PDT: %LINEPROTO-SW2-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to down
*Jun 27 23:59:25 PDT: %LINK-SW2-3-UPDOWN: Interface Port-channel1, changed state to down
*Jun 27 23:59:25 PDT: %OIR-SW2-6-INSREM: Switch 1 Physical Slot 5 - Module Type LINE_CARD removed
*Jun 27 23:59:25 PDT: %OSPF-SW2-5-ADJCHG: Process 1, Nbr 10.253.0.3 on TenGigabitEthernet1/1/1 from FULL to DOWN, Neighbor Down: Interface down or detached
*Jun 27 23:59:25 PDT: %LINEPROTO-SW2-5-UPDOWN: Line protocol on Interface Port-channel24, changed state to down
*Jun 27 23:59:25 PDT: %LINK-SW2-3-UPDOWN: Interface Port-channel24, changed state to down
*Jun 27 23:59:26 PDT: %OIR-SW2-6-INSREM: Switch 1 Physical Slot 1 - Module Type LINE_CARD removed
*Jun 27 23:59:26 PDT: %LINK-SW2-3-UPDOWN: Interface Port-channel17, changed state to down
*Jun 27 23:59:26 PDT: %PFREDUN-SW2-6-ACTIVE: Standby processor removed or reloaded, changing to Simplex mode
*Jun 27 23:59:26 PDT: %OIR-SW2-6-INSREM: Switch 1 Physical Slot 2 - Module Type LINE_CARD removed
*Jun 27 23:59:27 PDT: %LINK-SW2-3-UPDOWN: Interface TenGigabitEthernet1/5/14, changed state to down
*Jun 27 23:59:27 PDT: %LINK-SW2-3-UPDOWN: Interface TenGigabitEthernet1/5/15, changed state to down
*Jun 27 23:59:27 PDT: %LINK-SW2-3-UPDOWN: Interface TenGigabitEthernet1/5/16, changed state to down
*Jun 27 23:59:27 PDT: %LINEPROTO-SW2-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/5/14, changed state to down
*Jun 27 23:59:27 PDT: %LINEPROTO-SW2-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/5/16, changed state to down
*Jun 27 23:59:27 PDT: %LINK-SW2-3-UPDOWN: Interface TenGigabitEthernet1/1/1, changed state to down
*Jun 27 23:59:27 PDT: %LINK-SW2-3-UPDOWN: Interface TenGigabitEthernet1/1/2, changed state to down
*Jun 27 23:59:27 PDT: %LINK-SW2-3-UPDOWN: Interface TenGigabitEthernet1/1/3, changed state to down
*Jun 27 23:59:27 PDT: %LINEPROTO-SW2-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/1/1, changed state to down
*Jun 27 23:59:27 PDT: %LINEPROTO-SW2-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/1/2, changed state to down
*Jun 27 23:59:27 PDT: %LINEPROTO-SW2-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/1/3, changed state to down
Press RETURN to get started!
*Jun 28 07:03:33.421: %USBFLASH-SW2_STBY-5-CHANGE: bootdisk has been inserted!
*Jun 28 07:03:53.297: %OIR-SW2_STBY-6-INSPS: Power supply inserted in slot 1
*Jun 28 07:03:53.301: %C6KPWR-SW2_STBY-4-PSOK: power supply 1 turned on.
*Jun 28 07:04:26.093: %FABRIC-SW2_STBY-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 5 became active.
*Jun 28 07:04:48.497: %DIAG-SW2_STBY-6-RUN_MINIMUM: Switch 2 Module 5: Running Minimal Diagnostics...
*Jun 28 07:04:48.497: %CONST_DIAG-SW2_STBY-6-DIAG_PORT_SKIPPED: Module 5 port 15 is skipped in TestLoopback due to: the port is used as a VSL link.
*Jun 28 07:04:48.497: %CONST_DIAG-SW2_STBY-6-DIAG_PORT_SKIPPED: Module 5 port 16 is skipped in TestLoopback due to: the port is used as a VSL link.
*Jun 28 07:04:55.049: %CONST_DIAG-SW2_STBY-6-DIAG_PORT_SKIPPED: Module 5 port 15 is skipped in TestFexModeLoopback due to: the port is used as a VSL link.
*Jun 28 07:04:55.049: %CONST_DIAG-SW2_STBY-6-DIAG_PORT_SKIPPED: Module 5 port 16 is skipped in TestFexModeLoopback due to: the port is used as a VSL link.
*Jun 28 07:05:00.165: %CONST_DIAG-SW2_STBY-6-DIAG_PORT_SKIPPED: Module 5 port 15 is skipped in TestL2CTSLoopback due to: the port is used as a VSL link.
*Jun 28 07:05:00.165: %CONST_DIAG-SW2_STBY-6-DIAG_PORT_SKIPPED: Module 5 port 16 is skipped in TestL2CTSLoopback due to: the port is used as a VSL link.
*Jun 28 07:05:06.049: %CONST_DIAG-SW2_STBY-6-DIAG_PORT_SKIPPED: Module 5 port 15 is skipped in TestL3CTSLoopback due to: the port is used as a VSL link.
*Jun 28 07:05:06.049: %CONST_DIAG-SW2_STBY-6-DIAG_PORT_SKIPPED: Module 5 port 16 is skipped in TestL3CTSLoopback due to: the port is used as a VSL link.
*Jun 28 07:05:23.309: %DIAG-SW2_STBY-6-DIAG_OK: Switch 2 Module 5: Passed Online Diagnostics
*Jun 28 00:05:38 PDT: %SYS-SW2_STBY-6-CLOCKUPDATE: System clock has been updated from 00:05:38 PDT Sat Jun 28 2014 to 00:05:38 PDT Sat Jun 28 2014, configured from console by console.
*Jun 28 00:05:38 PDT: %SYS-SW2_STBY-6-CLOCKUPDATE: System clock has been updated from 00:05:38 PDT Sat Jun 28 2014 to 00:05:38 PDT Sat Jun 28 2014, configured from console by console.
*Jun 28 00:05:38 PDT: %SSH-SW2_STBY-5-DISABLED: SSH 2.0 has been disabled
*Jun 28 00:05:54 PDT: %SYS-SW2_STBY-5-RESTART: System restarted --
Cisco IOS Software, c6880x Software (c6880x-ADVENTERPRISEK9-M), Version 15.1(2)SY2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Wed 26-Feb-14 15:30 by prod_rel_team
*Jun 28 00:05:54 PDT: %SSH-SW2_STBY-5-ENABLED: SSH 2.0 has been enabled
*Jun 28 00:05:54 PDT: %SYS-SW2_STBY-3-LOGGER_FLUSHED: System was paused for 00:03:01 to ensure console debugging output.
*Jun 28 00:05:56 PDT: %C6KENV-SW2_STBY-4-LOWER_SLOT_EMPTY: The lower adjacent slot of module 5 might be empty. Airdam must be installed in that slot to be NEBS compliant
*Jun 28 00:07:36 PDT: %DIAG-SW2_STBY-6-RUN_MINIMUM: Switch 2 Module 1: Running Minimal Diagnostics...
*Jun 28 00:07:37 PDT: %SYS-SW2_STBY-3-LOGGER_FLUSHED: System was paused for 00:01:29 to ensure console debugging output.
*Jun 28 00:07:41 PDT: %DIAG-SW2_STBY-6-RUN_MINIMUM: Switch 2 Module 2: Running Minimal Diagnostics...
*Jun 28 00:08:10 PDT: %DIAG-SW2_STBY-6-DIAG_OK: Switch 2 Module 1: Passed Online Diagnostics
*Jun 28 00:08:22 PDT: %EC-SW2_STBY-5-CANNOT_BUNDLE2: Te2/1/10 is not compatible with Te1/2/10 and will be suspended (Operational flow control send of Te2/1/10 is off, Te1/2/10 is on)
*Jun 28 00:08:31 PDT: %EC-SW2_STBY-5-COMPATIBLE: Te2/1/10 is compatible with port-channel members
*Jun 28 00:08:45 PDT: %DIAG-SW2_STBY-6-DIAG_OK: Switch 2 Module 2: Passed Online Diagnostics
*Jun 28 00:08:46 PDT: %C6KENV-SW2_STBY-4-HIGHER_SLOT_EMPTY: The higher adjacent slot of module 2 might be empty. Airdam must be installed in that slot to be NEBS compliant
ELDC1C1-AG01-1 line 0
************************ W A R N I N G ***************************
* THIS IS A PRIVATE COMPUTER SYSTEM, AND FOR AUTHORIZED USE ONLY.*
* THIS SYSTEM IS MONITORED, AND ANY UNAUTHORIZED USE MAY BE *
* SUBJECT TO CRIMINAL PROSECUTION. *
* IF YOU ARE NOT AUTHORIZED, LOG OUT IMMEDIATELY!!!!! *
so according to the logs, it seems like the switch was reloaded or something of the nature but even the cisco tac said that it wasnt the case but couldnt determine whats causing it either. Tac went through the config on the vss and has said that its configured correctly.
Maybe someone else has experienced this issue or if someone can point out something that I can look at... sorry for the very long post.
thanksI suppose, we have the same problem. Only when Switch 1 is the active switch of the 6880-X VSS system, the connection to the FEX-stack breaks suddenly after running fine for some days and the FEX-stack (3 x 6800-IA) get only ready again when I reset the whole VSS-system, both VSS-switches and the FEX-stack.
All access ports at the FEX can't connect to our infrastructure (DHCP,DNS...). The two connected TenG FEX-stack uplinks sometimes get down and sometimes only the uplink to the active parent switch get down. The SYST LED of FEX 1 lighting amber.
When parent switch 2 is the active unit the issue never occured.
6880-X running with 15.2(1)SY, 6800-IA running with 15.2(3)E.
Cisco TAC is investigating that issue, but till now without any result.
When the issue occurs, the recorded logging messages are:
Syslog 6880-X
Apr 7 07:42:43.182: %PLATFORM_RPC-3-MSG_THROTTLED: RPC Msg Dropped by throttle mechanism: type 3, class 21, max_msg 32, total throttled 1274 (FEX-101)
Apr 7 07:45:43.181: %PLATFORM_RPC-3-MSG_THROTTLED: RPC Msg Dropped by throttle mechanism: type 3, class 21, max_msg 32, total throttled 1276 (FEX-101)
Syslog 6800-IA:
Apr 7 07:57:43.182: %PLATFORM_RPC-3-MSG_THROTTLED: RPC Msg Dropped by throttle mechanism: type 3, class 21, max_msg 32, total throttled 1284
-Traceback= 5198C4z 21413F8z 1C3B35Cz 1C3C6FCz 1C3CA00z 2654CC0z 26509BCz
Apr 7 08:00:43.181: %PLATFORM_RPC-3-MSG_THROTTLED: RPC Msg Dropped by throttle mechanism: type 3, class 21, max_msg 32, total throttled 1286
-Traceback= 5198C4z 21413F8z 1C3B35Cz 1C3C6FCz 1C3CA00z 2654CC0z 26509BCz -
4500-X vs 6880-X for Core/Distribution
Hi,
We are in the process of looking at upgrading our eight year old Cisco network to something newer.
Currently we have a 6500 for core/distribution with about 40+ 1 gig fibre links running out to access layer switches. (3560 and 3560X)
The whole design is routed with the links from core/distribtion to access all being layer 3. The intent was to dual attach access layer switches to 6500s in two buildings but this was never actually implemented.
The new design uses 3650 switches at the access layer, and a recommendation to use the 6880-X for core/distribution. This time around key buildings will be dual attached and I expect we will keep the routed design. 9396PX will be introduced at the datacenter layer.
It looks to be like the 6880-X is a pretty limited unit. No real path to 40 or 100 gig, no redundant supervisors, just a big box of 10 Gig ports.
I am assuming the cost on a 6807 chassis is just going to be too high, although 40 G and 100G are either available today or on the roadmap.
I am trying to figure out why two 4500-X with Enterprise software in VSS mode isn't a better fit. The port count is good at 80 (between the two), it has more bandwidth, features like MACSEC and SGT are on the develpment radar (if not already implemented), and two of these in VSS means I can perform a firmware update with minimal down time.
I plan to implement ISE in 2015 so having a platform that is going to work with ISE is really important to me.
Thanks
MurrayWow, amazing that noone answered such a real world question. I had like 2 similar cases within the last 2 months!!!
I normaly really like 4500X, it´s a powefull 1U 10G unit, and I myself am a fan of VSS, so - I agree with you, that should be a first choice! (btw - 6880 also supports VSS)
When to use 6880X? Well, I had a case recently with 45 3750X stacks in the core, so 80 ports in total wouldnt even cover a port-channel per stack, so - I used 6880X. How do I see this switch? As a moderatly priced almost-Nexus alternative for Campus with 10G with a VSS support.
To sum it up - yeah, I´d use 4500X if I were you...
Cheers! -
6800 ia upgrade loop on 6880-X w 15.1SY
Have received our first couple of 6800ia switches and tried to get them to work on a 6880-X in VSS. All works fine until the 6800IA concludes it has an older image then the one included in 15.1SY(2) that is on the 6880-X.
The feature set on the 6880-X is Advanced Enterprice
This is what I see on the 6880-X
Router#show fex
FEX FEX FEX FEX
Number Description State Model Serial
101 FEX0101 img_dnld C6800IA-48TD FOC1742S2L4
121 FEX0121 img_dnld C6800IA-48TD FOC1742S2KK
Router#
Router#
*May 6 06:30:33.528: %SATMGR-SW1-5-IMAGE_VERSION_MISMATCH: Image version mismatch detected for FEX 101: image download initiated. FEX will reboot on completion.
This is what I see on both 6880IA
FEX-101#
Could not buffer tarfile...using multiple downloads
examining image...
%Error opening tftp://192.1.1.1/c6800ia-universalk9-mz.150-2.EX5.bin (Timed out)
%Error opening flash:update/info (No such file or directory)
ERROR: Image is not a valid IOS image archive.
I can not find any documentation on how the upgrade works on CCO. Is the ip address 192.1.1.1 a default loopback on the parent switch or do I need to create it? Been dealing with FEX on Nexus switches before (which is pretty similar) but never had to config anything else then fex association on the parent switch ports and it have worked.I have a case open and instructions on how to unpack the FEX image from the 6880-X image and tftp it over but the fex refuse to boot on the manually extracted file as well. Just updated the case.
You can try this as well and see if any one else is lucky. I suspect everyone is running the same 6880-X version since there is only one available for download.
I used the 688-X as tftp-server and connected a cable between mgmt0 on the 6880 and the EOB ethernet port (top left corner) of the 6800ia. I guess one can forget using USB...
####### Instruction start
Copy image via TFTP and boot 6800IA in case the IA-image is corrupted
SETUP TFTP SERVER :
1. a. If you have a Lab TFTP Server :
Connect 6800IA TFTP Port (left hand topmost port) to the TFTP Server .
b. If you don’t have a TFTP Server :
i. Using USB Stick/compact-flash copy the image to bootdisk.
Internally image available at :
/release/150/bin/150-2.EX5/.3DES (Please refer to the compatibility chart above for right image)
bash-3.2$ ls /release/150/bin/150-2.EX5/.3DES/c6800ia*
/release/150/bin/150-2.EX5/.3DES/c6800ia-universalk9-mz.150-2.EX5 <<<< Can be used directly
/release/150/bin/150-2.EX5/.3DES/c6800ia-universalk9-tar.150-2.EX5 <<<< Will need to extract as below.
OR
1. Get the image present on the controller by executing following on Controller
archive tar /xtract microcode:FEX bootdisk:
IA-PARENT#archive tar /xtract microcode:FEX bootdisk:
c6800ia-universalk9-mz.150-2.5.10.MK1D/ (directory)
extracting c6800ia-universalk9-mz.150-2.5.10.MK1D/info (821 bytes)
c6800ia-universalk9-mz.150-2.5.10.MK1D/html/ (directory)
extracting c6800ia-universalk9-mz.150-2.5.10.MK1D/html/foo.html (0 bytes)
extracting c6800ia-universalk9-mz.150-2.5.10.MK1D/c6800ia-universalk9-mz.150-2.5.10.MK1D.bin (15414400 bytes)
extracting c6800ia-universalk9-mz.150-2.5.10.MK1D/dc_default_profiles.txt (71241 bytes)
extracting info (118 bytes)
2. Identify the FEX image to use : It will have a .bin suffix in the directory created above.
IA-PARENT#dir bootdisk:c6800ia-universalk9-mz.150-2.5.10.MK1D
Directory of bootdisk:/c6800ia-universalk9-mz.150-2.5.10.MK1D/
23 -rw- 821 Apr 10 2014 13:54:22 +05:00 info
24 drw- 0 Apr 10 2014 13:54:24 +05:00 html
32 -rw- 15414400 Apr 10 2014 13:54:52 +05:00 c6800ia-universalk9-mz.150-2.5.10.MK1D.bin <<<<<====== FEX IMAGE
33 -rw- 71241 Apr 10 2014 13:54:52 +05:00 dc_default_profiles.txt
1981050880 bytes total (348753920 bytes free)
3. Copy the FEX image to the bootdisk: and "RENAME" the file . It should NOT MATCH the directory name or the FEX image name.
IA-PARENT#copy bootdisk:c6800ia-universalk9-mz.150-2.5.10.MK1D/c6800ia-universalk9-mz.150-2.5.10.MK1D.bin bootdisk:c6800ia-universalk9-mz.150-2.5.10.MK1
Destination filename [c6800ia-universalk9-mz.150-2.5.10.MK1]?
Copy in progress...CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
15414400 bytes copied in 7.488 secs (2058547 bytes/sec)
IA-PARENT#
ii. Make the Controller Box as the TFTP Server like this :-
tftp-server bootdisk:c6800ia-universalk9-mz.150-2.5.10.MK1 <<<====RENAMED FILE
iii.Connect the IA TFTP Port with an Interface on Controller Box.
2.
a.Setup the IP ADDRESS and GATEWAY on the 6800IA
switch: IP_ADDR=8.10.8.19/255.255.0.0 (Based on the connection to tftp network)
switch: DEFAULT_ROUTER=8.10.0.1 (IP address of the gateway to reach tftp-server or
IP address of the controller interface)
b. If using Controller as TFTP-SERVER
Setup the IP ADDRESS on the Controller interface
This is same as the DEFAULT_ROUTER configured on 6800IA
4. Check the reachability of the tftp server.
switch: ping 202.153.144.25
ping 202.153.144.25 with 32 bytes of data ...
Host 202.153.144.25 is alive.
5. Copy the image to the 6800IA :
switch: copy tftp://x.x.x.x/<imagename> flash:<imagename>
6. Set the boot variable :
switch: BOOT=flash:/<image-name>
Example image-name = c6800ia-universalk9-mz.150-2.EX4.bin
7. Check if the auto-boot is enabled, if not :
switch: MANUAL_BOOT=no
8. Boot the 6800IA
switch: boot -
Hi Board,
i´ve search across the board, technet and symantec sites but did not found a hint about my problem.
we drive a 2 node DAG (Location1-Ex1-mb1
Location2-exc1-mb1), on SP2 RU4 patchlevel with 40 Databases.
Since some time the backup of one - and only one DB - is failing with these events, logged on the Mailboxserver on which the passive DB is hosted.
Log Name: Application
Source: MSExchangeRepl
Date: 28.09.2012 00:37:17
Event ID: 2112
Task Category: Exchange VSS Writer
Level: Error
Keywords: Classic
User: N/A
Computer: Location1-Exc1-MB1
Description: The Microsoft Exchange Replication service VSS Writer instance 1ab7d204-609a-4aea-b0a7-70afb0db38de failed with error code 80070020 when preparing for a backup of database 'DB012'.
Followed by
Log Name: Application
Source: MSExchangeRepl
Date:
01.10.2012 03:33:06
Event ID: 2024
Task Category: Exchange VSS Writer
Level: Error
Keywords: Classic
User:
N/A
Computer: Location1-Exc1-MB1
Description:
The Microsoft Exchange Replication service VSS Writer (Instance 42916d80-36c1-4f73-86d0-596d30226349) failed with error 80070020 when preparing for a backup.
The backup Application - Symantec Backup Exec 2010 R3 – states, this error
Snapshot provider error (0xE000FED1): A failure occurred querying the Writer status.
Check the Windows Event Viewer for details.
Writer Name: Exchange Server, Writer ID: {76FE1AC4-15F7-4BCD-987E-8E1ACB462FB7}, Last error: The VSS Writer failed, but the operation can be retried (0x800423f3), State: Stable (1).
Symatec suggests within http://www.symantec.com/business/support/index?page=content&id=TECH184095
to restart the MS Exchange Replication Service – BUT the mentioned eventID
8229 isn´t present on any of the both Mailboxservers.
The affected Database is active on Location2-Exc1-Mb1 Server and in an overall healthy state. I found during my research, that below Location2-Exc1-Mb1 Server, there are not removed shadow copies present!
This confuses me, since all Backups are normally taken from the passive copy of a Database.
So my questions to the board are:
* Does anyone is facing similar issues?
* Can someone explain why snapshots are present on the Mailboxserver hosting the Active Database, whilst the errors are logged on the passive one?
- * Does someone know the conditions, why shadows copies remain and
aren´t removed in a proper manner?
What can cause the circumstance, that only 1 DB is facing such issues?
Any suggestion is welcome!
BR
MarkusHi Lenora,
I´ve encreases VSS / Exchange Backup Log levels to expert, before starting
those things i´ve all tried now:
- Backup from passive DB (forced within Symantec Backup Exec)
- Backup from active DB (forced within Symantec Backup Exec)
- Backup from passive DB without GRT enabled (forced within Symantec Backup Exec)
- Backup from active DB without GRT enabled(forced within Symantec Backup Exec)
All those attempts failed.
But brought some more details - the backup against the active DB states, that there is still a backup in progress and therefore this backup is cancelled by VSS.
The Solution was, that i´ve needed to restart the Exchange Replication Service on the Mailbox Server hosting the passive DB.
Backups are working again on all DBs!
THX for your replys.
Best regards
Markus -
Windows Server 2012 VSS error "The system writer is missing"
Hello All,
I have a HP server that I have just installed Server 2012 standard, SQL Server 2012 and Sharepoint 2013.
When I try to take bare metal backup I get the error: "The system writer is missing", I have tried the backup as the local admin and domain admin.
The backup will only work if I decide to back up C:, once I select "Bare Metal" or "System State" the backup fails.
I have tried this solution: http://technet.microsoft.com/en-us/library/cc734021.aspx but have not had any luck yet.
When I run the command "vssadmin list writers" I can not see "System Writer" listed.
How can I add/turn on the System Writer?Hi,
The system writer fails due to the fact that the trusted installer & system accounts are missing permissions to files in the directory %windir%\winsxs\filemaps\.
To resolve this issue, please run the following commands from an administrative elevated Command Window:
CD c:\windows\system32
Takeown /f %windir%\winsxs\filemaps\* /a
icacls %windir%\winsxs\filemaps\*.* /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps\*.* /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps\*.* /grant "BUILTIN\Users:(RX)"
For more information, please also refer to the following links:
Discussion Forum FAQ: VSS System Writer is missing.
http://social.technet.microsoft.com/Forums/en-US/winserverfiles/thread/0b1eb10a-d666-4f5e-b657-c93b1d754a7f
Failure during Windows Server 2008 System State Backup & System Writer Missing
http://blogs.technet.com/b/chrad/archive/2009/11/18/failure-during-windows-server-2008-system-state-backup-system-writer-missing.aspx
Regards,
Arthur Li
TechNet Community Support -
Issues with CWLMS 3.2 RME 4.3.1 cannot fetch vlan.dat off a VSS and 4510.
Hi,
I am having a major headache with RME collecting the vlan.dat from a VSS and 4510, the device and credentials work fine, however when archiving the config i get partial success due to vlan failing. You can see in the IC_Services log that it attempts to TFTP the .dat file off which it fails with, i believe vlan fetch is only supported by SSH or telnet
when you do a CDA test both devices pass on everything..managed to log on and please see below, tftp on one device works fine but from the 4510 or vss still fails
GRA_CHUB_CR_01#copy cat4000_flash: tftp:
Source filename []? vlan.dat
Address or name of remote host []? 172.20.220.10
Destination filename [vlan.dat]?
%Error opening tftp://172.20.220.10/vlan.dat (Timed out)
GRA_CHUB_CR_01#
DAR_R002_AS_01#
DAR_R002_AS_01#copy flash:vlan.dat tftp:
Address or name of remote host []? 172.20.220.10
Destination filename [vlan.dat]?
616 bytes copied in 0.009 secs (68444 bytes/sec)
DAR_R002_AS_01# -
A Problem with VSS in the 6509
Hello,
I have a problem with VSS in the 6500, when the switch 1 is active,probably a few months will switchover to switch 2 and switch 1 reload.
config:
switch virtual domain 1
switch mode virtual
switch 1 priority 110
switch 2 priority 110
interface Port-channel1
description To 6509-B-VSS
no switchport
no ip address
switch virtual link 1
mls qos trust cos
no mls qos channel-consistency
interface Port-channel2
description 6509-A-VSS
no switchport
no ip address
switch virtual link 2
mls qos trust cos
no mls qos channel-consistency
interface GigabitEthernet1/4/20
no switchport
no ip address
dual-active fast-hello
interface TenGigabitEthernet1/5/4
description To 6509-B-VSS
no switchport
no ip address
mls qos trust cos
channel-group 1 mode on
interface TenGigabitEthernet1/5/5
description To 6509-B-VSS
no switchport
no ip address
mls qos trust cos
channel-group 1 mode on
interface GigabitEthernet2/4/20
no switchport
no ip address
dual-active fast-hello
interface TenGigabitEthernet2/5/4
description To 6509-A-VSS
no switchport
no ip address
mls qos trust cos
channel-group 2 mode on
interface TenGigabitEthernet2/5/5
description To 6509-A-VSS
no switchport
no ip address
mls qos trust cos
channel-group 2 mode on
show switch virtual redundancy
My Switch Id = 2
Peer Switch Id = 1
Last switchover reason = active unit removed
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Switch 2 Slot 5 Processor Information :
Current Software state = ACTIVE
Uptime in current state = 5 hours, 24 minutes
Image Version = Cisco IOS Software, s72033_rp Software (s72033_rp-IPBASEK9-M), Version 12.2(33)SXJ5, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Thu 31-Jan-13 14:30 by prod_rel_team
BOOT = sup-bootdisk:/s72033-ipbasek9-mz.122-33.SXJ5.bin,12;
Configuration register = 0x2102
Fabric State = ACTIVE
Control Plane State = ACTIVE
Switch 1 Slot 5 Processor Information :
Current Software state = STANDBY HOT (switchover target)
Uptime in current state = 5 hours, 18 minutes
Image Version = Cisco IOS Software, s72033_rp Software (s72033_rp-IPBASEK9-M), Version 12.2(33)SXJ5, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Thu 31-Jan-13 14:30 by prod_rel_team
BOOT = sup-bootdisk:/s72033-ipbasek9-mz.122-33.SXJ5.bin,12;
Configuration register = 0x2102
Fabric State = ACTIVE
Control Plane State = STANDBY
show log
Jun 21 11:20:52.742: %VSL-SW2_SPSTBY-5-VSL_CNTRL_LINK: New VSL Control Link Te2/5/4
Jun 21 11:20:52.786: %VSLP-SW2_SPSTBY-3-VSLP_LMP_FAIL_REASON: Te2/5/4: Link down
Jun 21 11:20:52.786: %VSLP-SW2_SPSTBY-2-VSL_DOWN: Last VSL interface Te2/5/4 went down
Jun 21 11:20:52.790: %VSLP-SW2_SPSTBY-2-VSL_DOWN: All VSL links went down while switch is in Standby role
Jun 21 11:20:52.790: %DUAL_ACTIVE-SW2_SPSTBY-1-VSL_DOWN: VSL is down - switchover, or possible dual-active situation has occurred
Jun 21 11:20:53.622: %SYS-SW2_SPSTBY-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
Jun 21 11:20:54.678: %C6KPWR-SP-4-PSOK: power supply 1 turned on.
Jun 21 11:20:54.682: %C6KPWR-SP-4-PSOK: power supply 2 turned on.
Jun 21 11:20:54.738: %SATVS_IBC-SW2_SP-5-VSL_DOWN_SCP_DROP: VSL inactive - dropping cached SCP packet: (SA/DA:0x4/0x4, SSAP/DSAP:0x19/0x0, OP/SEQ:0x320/0x8FD9, SIG/INFO:0x1/0x502, eSA:0000.0500.0000)
Jun 21 11:20:56.490: %SATVS_IBC-5-VSL_DOWN_SCP_DROP: VSL inactive - dropping cached SCP packet: (SA/DA:0x14/0x4, SSAP/DSAP:0x18/0x0, OP/SEQ:0x19/0x98, SIG/INFO:0x1/0x502, eSA:0000.1500.0000)
Jun 21 11:21:02.125: %VSDA-SW2_SP-3-LINK_DOWN: Interface Gi2/4/20 is no longer dual-active detection capable
Jun 21 11:23:27.719: %VSLP-SW2_SP-5-RRP_ROLE_RESOLVED: Role resolved as ACTIVE by VSLP
Jun 21 11:23:27.719: %VSL-SW2_SP-5-VSL_CNTRL_LINK: New VSL Control Link Te2/5/4
Jun 21 11:23:27.799: %VSLP-SW2_SP-5-VSL_UP: Ready for control traffic
Jun 21 11:24:43.346: %PFINIT-SW2_SP-5-CONFIG_SYNC: Sync'ing the startup configuration to the standby Router.
Jun 21 11:25:35.304: %VSLP-SW2_SP-5-VSL_UP: Ready for data traffic
Jun 21 11:25:14.946: %C6KPWR-SW1_SPSTBY-4-PSOK: power supply 1 turned on.
Jun 21 11:25:14.950: %C6KPWR-SW1_SPSTBY-4-PSOK: power supply 2 turned on.
Jun 21 11:25:16.975: %FABRIC-SW1_SPSTBY-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 5.
Jun 21 11:25:16.975: %FABRIC-SW1_SPSTBY-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 5 became active.
Jun 21 11:26:13.738: %SYS-SW1_SPSTBY-5-RESTART: System restarted --
Jun 21 11:26:20.446: %SYS-SW1_SPSTBY-3-LOGGER_FLUSHED: System was paused for 00:01:47 to ensure console debugging output.
Jun 21 11:26:32.056: %RF-SW2_SP-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)
Jun 21 11:27:56.962: %SYS-SW1_SPSTBY-3-LOGGER_FLUSHED: System was paused for 00:01:24 to ensure console debugging output.
Jun 21 11:27:53.970: %SYS-DFC4-5-RESTART: System restarted --
Jun 21 11:28:34.222: %VSDA-SW1_SPSTBY-5-LINK_UP: Interface Gi1/4/20 is now dual-active detection capable
Jun 21 11:28:36.236: %VSDA-SW2_SP-5-LINK_UP: Interface Gi2/4/20 is now dual-active detection capable
Best Regardhi,
Information of Last System Crash - SP
Writing crashinfo to bootflash:crashinfo_SP_20140621-192052-TW 00 07 00 00 00 00 00 00 00 0F 00 00 00 00 00 00 10 D6 0A 06 FD 37 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00
Jun 21 11:19:43.723: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.723: SW1_SP: IPC: Message 504CA554 timed out waiting for Ack
Jun 21 11:19:43.723: SW1_SP: IPC: MSG: ptr: 0x504CA554, flags: 0x24100, retries: 8, seq: 0x315E1ED, refcount: 1, rpc_result = 0x0, data_buffer = 0x504104BC, header = 0x8A453C8, data = 0x8A453E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57837, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AF38, lo: 0x8A453E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 5E 0A 06 01 01 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 08 00 00
Jun 21 11:19:43.723: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504CA554, flags: 0x24100, retries: 8, seq: 0x315E1ED, refcount: 1, rpc_result = 0x0, data_buffer = 0x504104BC, header = 0x8A453C8, data = 0x8A453E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57837, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AF38, lo: 0x8A453E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 5E 0A 06 01 01 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 08 00 00
Jun 21 11:19:43.727: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.727: SW1_SP: IPC: Message 4528FE20 timed out waiting for Ack
Jun 21 11:19:43.727: SW1_SP: IPC: MSG: ptr: 0x4528FE20, flags: 0x24100, retries: 8, seq: 0x315E1EE, refcount: 1, rpc_result = 0x0, data_buffer = 0x4520B23C, header = 0x8C884C8, data = 0x8C884E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57838, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AF39, lo: 0x8C884E8 || DATA: 2C D5 00 01 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 87 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 40 0A 06 0A 15 01 04 00 00 00 87 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00
Jun 21 11:19:43.727: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x4528FE20, flags: 0x24100, retries: 8, seq: 0x315E1EE, refcount: 1, rpc_result = 0x0, data_buffer = 0x4520B23C, header = 0x8C884C8, data = 0x8C884E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57838, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AF39, lo: 0x8C884E8 || DATA: 2C D5 00 01 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 87 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 40 0A 06 0A 15 01 04 00 00 00 87 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00
Jun 21 11:19:43.727: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.727: SW1_SP: IPC: Message 4529E380 timed out waiting for Ack
Jun 21 11:19:43.727: SW1_SP: IPC: MSG: ptr: 0x4529E380, flags: 0x24100, retries: 8, seq: 0x315E1EF, refcount: 1, rpc_result = 0x0, data_buffer = 0x50418920, header = 0x8A93DC8, data = 0x8A93DE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57839, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AF3A, lo: 0x8A93DE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 BC 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 12 00 00 00 00 00 00 08 9E 0A 06 9A 0B 01 04 00 00 00 BC 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 16 00 00
Jun 21 11:19:43.727: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x4529E380, flags: 0x24100, retries: 8, seq: 0x315E1EF, refcount: 1, rpc_result = 0x0, data_buffer = 0x50418920, header = 0x8A93DC8, data = 0x8A93DE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57839, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AF3A, lo: 0x8A93DE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 BC 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 12 00 00 00 00 00 00 08 9E 0A 06 9A 0B 01 04 00 00 00 BC 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 16 00 00
Jun 21 11:19:43.731: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.731: SW1_SP: IPC: Message 45238524 timed out waiting for Ack
Jun 21 11:19:43.731: SW1_SP: IPC: MSG: ptr: 0x45238524, flags: 0x24100, retries: 8, seq: 0x315E1F0, refcount: 1, rpc_result = 0x0, data_buffer = 0x50372694, header = 0x8468FC8, data = 0x8468FE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57840, sz: 1332, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AF3B, lo: 0x8468FE8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 05 00 00 00 00 00 00 03 74 0A 06 FD 4C 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00
Jun 21 11:19:43.731: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x45238524, flags: 0x24100, retries: 8, seq: 0x315E1F0, refcount: 1, rpc_result = 0x0, data_buffer = 0x50372694, header = 0x8468FC8, data = 0x8468FE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57840, sz: 1332, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AF3B, lo: 0x8468FE8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 05 00 00 00 00 00 00 03 74 0A 06 FD 4C 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00
Jun 21 11:19:43.731: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.731: SW1_SP: IPC: Message 4524C89C timed out waiting for Ack
Jun 21 11:19:43.731: SW1_SP: IPC: MSG: ptr: 0x4524C89C, flags: 0x24100, retries: 6, seq: 0x315E1F1, refcount: 1, rpc_result = 0x0, data_buffer = 0x451B45EC, header = 0x894FCC8, data = 0x894FCE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57841, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFE9, lo: 0x894FCE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00 00 00 00 00 01 CF 0A 06 01 06 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00
Jun 21 11:19:43.731: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x4524C89C, flags: 0x24100, retries: 6, seq: 0x315E1F1, refcount: 1, rpc_result = 0x0, data_buffer = 0x451B45EC, header = 0x894FCC8, data = 0x894FCE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57841, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFE9, lo: 0x894FCE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00 00 00 00 00 01 CF 0A 06 01 06 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 01 00 00
Jun 21 11:19:43.731: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.731: SW1_SP: IPC: Message 45268348 timed out waiting for Ack
Jun 21 11:19:43.731: SW1_SP: IPC: MSG: ptr: 0x45268348, flags: 0x24100, retries: 6, seq: 0x315E1F2, refcount: 1, rpc_result = 0x0, data_buffer = 0x50462254, header = 0x8D4EFC8, data = 0x8D4EFE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57842, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFEA, lo: 0x8D4EFE8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 86 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 38 00 00 00 00 00 00 26 AB 0A 06 05 41 01 04 00 00 00 86 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 19 00 00
Jun 21 11:19:43.731: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x45268348, flags: 0x24100, retries: 6, seq: 0x315E1F2, refcount: 1, rpc_result = 0x0, data_buffer = 0x50462254, header = 0x8D4EFC8, data = 0x8D4EFE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57842, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFEA, lo: 0x8D4EFE8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 86 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 38 00 00 00 00 00 00 26 AB 0A 06 05 41 01 04 00 00 00 86 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 19 00 00
Jun 21 11:19:43.735: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.735: SW1_SP: IPC: Message 45281D58 timed out waiting for Ack
Jun 21 11:19:43.735: SW1_SP: IPC: MSG: ptr: 0x45281D58, flags: 0x24100, retries: 6, seq: 0x315E1F3, refcount: 1, rpc_result = 0x0, data_buffer = 0x45201860, header = 0x8C2CEC8, data = 0x8C2CEE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57843, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFEB, lo: 0x8C2CEE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 B9 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00 00 00 00 00 04 24 0A 06 97 0D 01 04 00 00 00 BB 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00
Jun 21 11:19:43.735: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x45281D58, flags: 0x24100, retries: 6, seq: 0x315E1F3, refcount: 1, rpc_result = 0x0, data_buffer = 0x45201860, header = 0x8C2CEC8, data = 0x8C2CEE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57843, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFEB, lo: 0x8C2CEE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 B9 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00 00 00 00 00 04 24 0A 06 97 0D 01 04 00 00 00 BB 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00
Jun 21 11:19:43.735: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.735: SW1_SP: IPC: Message 504A2D18 timed out waiting for Ack
Jun 21 11:19:43.735: SW1_SP: IPC: MSG: ptr: 0x504A2D18, flags: 0x24100, retries: 6, seq: 0x315E1F4, refcount: 1, rpc_result = 0x0, data_buffer = 0x451F3A88, header = 0x8BA92C8, data = 0x8BA92E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57844, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFEC, lo: 0x8BA92E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 05 00 00 00 00 00 00 01 48 0A 06 FD 2F 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00
Jun 21 11:19:43.735: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504A2D18, flags: 0x24100, retries: 6, seq: 0x315E1F4, refcount: 1, rpc_result = 0x0, data_buffer = 0x451F3A88, header = 0x8BA92C8, data = 0x8BA92E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57844, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFEC, lo: 0x8BA92E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 05 00 00 00 00 00 00 01 48 0A 06 FD 2F 01 04 00 00 00 DD 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00
Jun 21 11:19:43.735: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.735: SW1_SP: IPC: Message 504AA128 timed out waiting for Ack
Jun 21 11:19:43.735: SW1_SP: IPC: MSG: ptr: 0x504AA128, flags: 0x24100, retries: 6, seq: 0x315E1F5, refcount: 1, rpc_result = 0x0, data_buffer = 0x45228364, header = 0x8D9C8C8, data = 0x8D9C8E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57845, sz: 568, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFED, lo: 0x8D9C8E8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 02 26 00 0D 00 10 02 20 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 08 00 00 00 00 00 00 02 3D 0A 06 FE 6C 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 2D 00 00
Jun 21 11:19:43.739: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504AA128, flags: 0x24100, retries: 6, seq: 0x315E1F5, refcount: 1, rpc_result = 0x0, data_buffer = 0x45228364, header = 0x8D9C8C8, data = 0x8D9C8E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57845, sz: 568, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56AFED, lo: 0x8D9C8E8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 02 26 00 0D 00 10 02 20 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 08 00 00 00 00 00 00 02 3D 0A 06 FE 6C 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 2D 00 00
Jun 21 11:19:43.739: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.739: SW1_SP: IPC: Message 504C15B4 timed out waiting for Ack
Jun 21 11:19:43.739: SW1_SP: IPC: MSG: ptr: 0x504C15B4, flags: 0x24100, retries: 4, seq: 0x315E1F6, refcount: 1, rpc_result = 0x0, data_buffer = 0x503C193C, header = 0x87593C8, data = 0x87593E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57846, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B08B, lo: 0x87593E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 3B 00 00 00 00 00 00 10 A7 0A 06 01 06 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 01 3B 00 00
Jun 21 11:19:43.739: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504C15B4, flags: 0x24100, retries: 4, seq: 0x315E1F6, refcount: 1, rpc_result = 0x0, data_buffer = 0x503C193C, header = 0x87593C8, data = 0x87593E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57846, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B08B, lo: 0x87593E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 3B 00 00 00 00 00 00 10 A7 0A 06 01 06 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 01 3B 00 00
Jun 21 11:19:43.739: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.739: SW1_SP: IPC: Message 45273050 timed out waiting for Ack
Jun 21 11:19:43.739: SW1_SP: IPC: MSG: ptr: 0x45273050, flags: 0x24100, retries: 4, seq: 0x315E1F7, refcount: 1, rpc_result = 0x0, data_buffer = 0x5044348C, header = 0x8C29BC8, data = 0x8C29BE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57847, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B08C, lo: 0x8C29BE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 89 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 02 00 00 00 00 00 00 02 24 0A 06 0C 18 01 04 00 00 00 98 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00
Jun 21 11:19:43.739: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x45273050, flags: 0x24100, retries: 4, seq: 0x315E1F7, refcount: 1, rpc_result = 0x0, data_buffer = 0x5044348C, header = 0x8C29BC8, data = 0x8C29BE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57847, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B08C, lo: 0x8C29BE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 89 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 02 00 00 00 00 00 00 02 24 0A 06 0C 18 01 04 00 00 00 98 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00
Jun 21 11:19:43.743: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.743: SW1_SP: IPC: Message 452A33E4 timed out waiting for Ack
Jun 21 11:19:43.743: SW1_SP: IPC: MSG: ptr: 0x452A33E4, flags: 0x24100, retries: 4, seq: 0x315E1F8, refcount: 1, rpc_result = 0x0, data_buffer = 0x50432830, header = 0x8B8A5C8, data = 0x8B8A5E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57848, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B08D, lo: 0x8B8A5E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 C4 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00 00 00 00 00 04 24 0A 06 A2 0B 01 04 00 00 00 C5 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00
Jun 21 11:19:43.743: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x452A33E4, flags: 0x24100, retries: 4, seq: 0x315E1F8, refcount: 1, rpc_result = 0x0, data_buffer = 0x50432830, header = 0x8B8A5C8, data = 0x8B8A5E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57848, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B08D, lo: 0x8B8A5E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 C4 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00 00 00 00 00 04 24 0A 06 A2 0B 01 04 00 00 00 C5 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00
Jun 21 11:19:43.743: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.743: SW1_SP: IPC: Message 4523FFA4 timed out waiting for Ack
Jun 21 11:19:43.743: SW1_SP: IPC: MSG: ptr: 0x4523FFA4, flags: 0x24100, retries: 4, seq: 0x315E1F9, refcount: 1, rpc_result = 0x0, data_buffer = 0x5036F47C, header = 0x844B3C8, data = 0x844B3E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57849, sz: 772, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B08E, lo: 0x844B3E8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 02 F2 00 0D 00 16 02 EC 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 04 00 00 00 00 00 00 01 46 0A 06 FE 5A 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 04 00 00
Jun 21 11:19:43.743: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x4523FFA4, flags: 0x24100, retries: 4, seq: 0x315E1F9, refcount: 1, rpc_result = 0x0, data_buffer = 0x5036F47C, header = 0x844B3C8, data = 0x844B3E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57849, sz: 772, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B08E, lo: 0x844B3E8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 02 F2 00 0D 00 16 02 EC 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 04 00 00 00 00 00 00 01 46 0A 06 FE 5A 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 04 00 00
Jun 21 11:19:43.743: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.743: SW1_SP: IPC: Message 45279200 timed out waiting for Ack
Jun 21 11:19:43.743: SW1_SP: IPC: MSG: ptr: 0x45279200, flags: 0x24100, retries: 2, seq: 0x315E1FA, refcount: 1, rpc_result = 0x0, data_buffer = 0x5039C3B0, header = 0x85F65C8, data = 0x85F65E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57850, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B123, lo: 0x85F65E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00 00 00 00 00 01 CF 0A 06 01 06 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 02 00 00
Jun 21 11:19:43.747: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x45279200, flags: 0x24100, retries: 2, seq: 0x315E1FA, refcount: 1, rpc_result = 0x0, data_buffer = 0x5039C3B0, header = 0x85F65C8, data = 0x85F65E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57850, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B123, lo: 0x85F65E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00 00 00 00 00 01 CF 0A 06 01 06 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 02 00 00
Jun 21 11:19:43.747: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.747: SW1_SP: IPC: Message 504D3544 timed out waiting for Ack
Jun 21 11:19:43.747: SW1_SP: IPC: MSG: ptr: 0x504D3544, flags: 0x24100, retries: 2, seq: 0x315E1FB, refcount: 1, rpc_result = 0x0, data_buffer = 0x5044AE34, header = 0x8C71FC8, data = 0x8C71FE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57851, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B124, lo: 0x8C71FE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 88 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 04 00 00 00 00 00 00 04 48 0A 06 0B 24 01 04 00 00 00 89 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 04 00 00
Jun 21 11:19:43.747: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504D3544, flags: 0x24100, retries: 2, seq: 0x315E1FB, refcount: 1, rpc_result = 0x0, data_buffer = 0x5044AE34, header = 0x8C71FC8, data = 0x8C71FE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57851, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B124, lo: 0x8C71FE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 88 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 04 00 00 00 00 00 00 04 48 0A 06 0B 24 01 04 00 00 00 89 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 04 00 00
Jun 21 11:19:43.747: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.747: SW1_SP: IPC: Message 504D5940 timed out waiting for Ack
Jun 21 11:19:43.747: SW1_SP: IPC: MSG: ptr: 0x504D5940, flags: 0x24100, retries: 2, seq: 0x315E1FC, refcount: 1, rpc_result = 0x0, data_buffer = 0x4521415C, header = 0x8CDD4C8, data = 0x8CDD4E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57852, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B125, lo: 0x8CDD4E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 BE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00 00 00 00 00 0E 6D 0A 06 9C 5C 01 04 00 00 00 BF 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 08 00 00
Jun 21 11:19:43.747: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504D5940, flags: 0x24100, retries: 2, seq: 0x315E1FC, refcount: 1, rpc_result = 0x0, data_buffer = 0x4521415C, header = 0x8CDD4C8, data = 0x8CDD4E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57852, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B125, lo: 0x8CDD4E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 BE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00 00 00 00 00 0E 6D 0A 06 9C 5C 01 04 00 00 00 BF 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 08 00 00
Jun 21 11:19:43.751: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.751: SW1_SP: IPC: Message 452989EC timed out waiting for Ack
Jun 21 11:19:43.751: SW1_SP: IPC: MSG: ptr: 0x452989EC, flags: 0x24100, retries: 2, seq: 0x315E1FD, refcount: 1, rpc_result = 0x0, data_buffer = 0x45172660, header = 0x86DCEC8, data = 0x86DCEE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57853, sz: 1060, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B126, lo: 0x86DCEE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 07 00 00 00 00 00 00 03 3E 0A 06 FE 1D 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 03 BF 00 00
Jun 21 11:19:43.751: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x452989EC, flags: 0x24100, retries: 2, seq: 0x315E1FD, refcount: 1, rpc_result = 0x0, data_buffer = 0x45172660, header = 0x86DCEC8, data = 0x86DCEE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57853, sz: 1060, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B126, lo: 0x86DCEE8 || DATA: 2C D5 00 00 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 07 00 00 00 00 00 00 03 3E 0A 06 FE 1D 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 03 BF 00 00
Jun 21 11:19:43.751: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.751: SW1_SP: IPC: Message 504C9DAC timed out waiting for Ack
Jun 21 11:19:43.751: SW1_SP: IPC: MSG: ptr: 0x504C9DAC, flags: 0x24108, retries: 0, seq: 0x315E1FE, refcount: 1, rpc_result = 0x0, data_buffer = 0x451D3748, header = 0x8A772C8, data = 0x8A772E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57854, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B1D4, lo: 0x8A772E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00 00 00 00 00 01 CF 0A 06 01 06 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 97 00 00
Jun 21 11:19:43.751: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504C9DAC, flags: 0x24108, retries: 0, seq: 0x315E1FE, refcount: 1, rpc_result = 0x0, data_buffer = 0x451D3748, header = 0x8A772C8, data = 0x8A772E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57854, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B1D4, lo: 0x8A772E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00 00 00 00 00 01 CF 0A 06 01 06 01 04 00 00 00 3F 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 97 00 00
Jun 21 11:19:43.751: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.751: SW1_SP: IPC: Message 4525926C timed out waiting for Ack
Jun 21 11:19:43.751: SW1_SP: IPC: MSG: ptr: 0x4525926C, flags: 0x24108, retries: 0, seq: 0x315E1FF, refcount: 1, rpc_result = 0x0, data_buffer = 0x50405568, header = 0x89DD1C8, data = 0x89DD1E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57855, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B1D5, lo: 0x89DD1E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 88 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00 00 00 00 00 01 80 0A 06 0B 24 01 04 00 00 00 8A 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 02 00 00
Jun 21 11:19:43.751: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x4525926C, flags: 0x24108, retries: 0, seq: 0x315E1FF, refcount: 1, rpc_result = 0x0, data_buffer = 0x50405568, header = 0x89DD1C8, data = 0x89DD1E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57855, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B1D5, lo: 0x89DD1E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 88 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 06 00 00 00 00 00 00 01 80 0A 06 0B 24 01 04 00 00 00 8A 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 02 00 00
Jun 21 11:19:43.755: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.755: SW1_SP: IPC: Message 452512E0 timed out waiting for Ack
Jun 21 11:19:43.755: SW1_SP: IPC: MSG: ptr: 0x452512E0, flags: 0x24108, retries: 0, seq: 0x315E200, refcount: 1, rpc_result = 0x0, data_buffer = 0x451AF00C, header = 0x891CCC8, data = 0x891CCE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57856, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B1D6, lo: 0x891CCE8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 C0 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0B 00 00 00 00 00 00 04 B2 0A 06 9E 0B 01 04 00 00 00 C3 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 17 00 00
Jun 21 11:19:43.755: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x452512E0, flags: 0x24108, retries: 0, seq: 0x315E200, refcount: 1, rpc_result = 0x0, data_buffer = 0x451AF00C, header = 0x891CCC8, data = 0x891CCE8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57856, sz: 1008, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B1D6, lo: 0x891CCE8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 03 E0 00 0D 00 1D 03 DA 01 04 00 00 00 C0 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0B 00 00 00 00 00 00 04 B2 0A 06 9E 0B 01 04 00 00 00 C3 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 17 00 00
Jun 21 11:19:43.755: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:43.755: SW1_SP: IPC: Message 504A9C90 timed out waiting for Ack
Jun 21 11:19:43.755: SW1_SP: IPC: MSG: ptr: 0x504A9C90, flags: 0x24108, retries: 0, seq: 0x315E201, refcount: 1, rpc_result = 0x0, data_buffer = 0x45158028, header = 0x85E22C8, data = 0x85E22E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57857, sz: 736, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B1D7, lo: 0x85E22E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 02 D0 00 0D 00 15 02 CA 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00 00 00 00 00 03 4E 0A 06 FE 4C 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 01 C2 00 00
Jun 21 11:19:43.755: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504A9C90, flags: 0x24108, retries: 0, seq: 0x315E201, refcount: 1, rpc_result = 0x0, data_buffer = 0x45158028, header = 0x85E22C8, data = 0x85E22E8 || HDR: src: 0x10000, dst: 0x315001B, index: 0, seq: 57857, sz: 736, type: 882, flags: 0x400, ext_flags: 0x0, hi: 0xA56B1D7, lo: 0x85E22E8 || DATA: 2C D5 00 02 0E 0B 00 00 00 00 00 20 00 00 02 D0 00 0D 00 15 02 CA 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 00 0A 00 00 00 00 00 00 03 4E 0A 06 FE 4C 01 04 00 00 00 DE 00 00 00 00 00 00 00 07 00 00 00 00 00 00 01 C2 00 00
Jun 21 11:19:43.755: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:19:52.687: %CPU_MONITOR-SW1_SP-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 120 seconds [21/1]
Jun 21 11:20:22.687: %CPU_MONITOR-SW1_SP-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 150 seconds [21/1]
Jun 21 11:20:33.175: SW1_SP: IPC: Message 504F68E8 timed out waiting for Ack
Jun 21 11:20:33.175: SW1_SP: IPC: MSG: ptr: 0x504F68E8, flags: 0x34101, retries: 21, seq: 0x315AABD, refcount: 2, rpc_result = 0x0, data_buffer = 0x4517147C, header = 0x86D24C8, data = 0x86D24E8 || HDR: src: 0x10000, dst: 0x315000E, index: 1, seq: 43709, sz: 80, type: 1, flags: 0x1404, ext_flags: 0x0, hi: 0xA56AE85, lo: 0x86D24E8 || DATA: 00 00 00 15 00 00 00 00 00 00 07 D1 00 00 00 0A 00 00 00 0C 00 00 00 0F 00 00 00 05 00 00 00 0A 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Jun 21 11:20:33.175: SW1_SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x504F68E8, flags: 0x34101, retries: 21, seq: 0x315AABD, refcount: 2, rpc_result = 0x0, data_buffer = 0x4517147C, header = 0x86D24C8, data = 0x86D24E8 || HDR: src: 0x10000, dst: 0x315000E, index: 1, seq: 43709, sz: 80, type: 1, flags: 0x1404, ext_flags: 0x0, hi: 0xA56AE85, lo: 0x86D24E8 || DATA: 00 00 00 15 00 00 00 00 00 00 07 D1 00 00 00 0A 00 00 00 0C 00 00 00 0F 00 00 00 05 00 00 00 0A 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Jun 21 11:20:33.175: SW1_SP: -Traceback= 4042B17C 404543D0 40454508 408A5DE4 4044FF98 408AAFE8 408AAFD4
Jun 21 11:20:33.179: %C6K_PROCMIB-SW1_SP-3-IPC_TRANSMIT_FAIL: Failed to send process statistics update : error code = timeout
-Traceback= 40BF9E18 40BF9E68 40BFA070 40BFA2D4 408AAFE8 408AAFD4
Jun 21 11:20:52.687: %CPU_MONITOR-SW1_SP-3-TIMED_OUT: CPU_MONITOR messages have failed, resetting system [21/1] -
Hi-
Looked thru the thread related to my questions and found nothing that applies yet.
Example:
http://social.technet.microsoft.com/Forums/en-US/4e4472b9-8e6f-4fdc-a287-1cc8c0f6de09/windows-2008-r2-backup-fails-with-event-521-error-code-2155348001?forum=windowsbackup
http://social.technet.microsoft.com/Forums/en-US/e8c910f7-f0dd-4613-beae-b98894fb6f23/windows-server-2008-std-backup-fails-constantly-with-error-code-2155348001?forum=windowsbackup
Here are details of the issues....
Backup - OS error - system image:
External disk - USB
Mon - Fri @ 6PM.
Windows SystemState
OS(C:)
Last backup failure...
Start time: 8/11/2014 6:01:32 PM
End time: 8/12/2014 4:51:29 AM
Duration: 10:49:56.4414784
[Critical error WINIMAGE_UNABLE_VSS] Unable to create VSS snapshot
- Additional information: Refer to the log for more details
- Troubleshooting link:
http://kb.backupassist.com/errormap.php?BA2512
The backup operation stopped before completing.
Detailed error: ERROR - The shared restore point operation failed with error (0x81000101)
The creation of a shadow copy has timed out. Try this operation again.
Log of files successfully backed up:
C:\Windows\Logs\WindowsServerBackup\Backup-12-08-2014_04-41-00.log
Log of files for which backup failed:
C:\Windows\Logs\WindowsServerBackup\Backup_Error-12-08-2014_04-41-00.log
NOTE:
The errors logs are not there?
Windows Backup timed-out before the shared protection point was created.
ERROR - The shared restore point operation failed with error (0x81000101)
The creation of a shadow copy has timed out. Try this operation again.
Event Viewer Log:
Sequence of Events...
1.
The Block Level Backup Engine service has successfully started.
2.
Backup operation that started...
Log Name: Application
Source: Microsoft-Windows-Backup
Date: 8/12/2014 4:51:06 AM
Event ID: 521
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: mdnfile.Auxiant.local
Description:
The backup operation that started at '?2014?-?08?-?12T09:41:00.492000000Z' has failed because the
Volume Shadow Copy Service operation to create a shadow copy of the volumes being backed up failed
with following error code '2155348001'.
Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.
--> Suggestion are to check Virus scanner or consult DPM forum...
3.
OS Job failed with errors...( Backup Assist is 3rd party backup - wrapper round Wbadmin/Win Backup)
Log Name: Application
Source: BackupAssist
Date: 8/12/2014 4:51:51 AM
Event ID: 5634
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: mdnfile.Auxiant.local
Description:
Backup job OS backup failed with errors.
Critical error: Unable to create VSS snapshot
Destination: External disk
Start time: 8/11/2014 6:01:32 PM
End time: 8/12/2014 4:51:29 AM
Duration: 10:49:56.4414784
Destination: External disk
Start time: 8/11/2014 6:01:32 PM
End time: 8/12/2014 4:51:29 AM
Duration: 10:49:56.4414784
4.
Backup Assist reports via Win backup - Snapshot in progress
Log Name: Application
Source: BackupAssist
Date: 8/12/2014 4:52:20 AM
Event ID: 5634
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: mdnfile.Auxiant.local
Description:
Backup job Data backup failed with errors.
Ultra critical error: Volume Shadow Copy Error 0x80042316 - snapshot set in progress
Destination: Local directory
Start time: 8/12/2014 4:52:10 AM
End time: 8/12/2014 4:52:12 AM
Duration: 00:00:02.7300000
5.
The Block Level Backup Engine service has stopped.
6.
Later...we tried manually stopping VSS and SWPRV services - only VSS had the issue.
WBengine was not running...
C:\>sc stop swprv
SERVICE_NAME: swprv
TYPE : 10 WIN32_OWN_PROCESS
STATE : 3 STOP_PENDING
(STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0
C:\>sc query swprv
SERVICE_NAME: swprv
TYPE : 10 WIN32_OWN_PROCESS
STATE : 1 STOPPED
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0
C:\>sc query wbengine
SERVICE_NAME: wbengine
TYPE : 10 WIN32_OWN_PROCESS
STATE : 1 STOPPED
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0
C:\>sc query vss
SERVICE_NAME: vss
TYPE : 10 WIN32_OWN_PROCESS
STATE : 3 STOP_PENDING
(STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)
WIN32_EXIT_CODE : 0 (0x0)
SERVICE_EXIT_CODE : 0 (0x0)
CHECKPOINT : 0x0
WAIT_HINT : 0x0
C:\>sc stop vss
[SC] ControlService FAILED 1061:
The service cannot accept control messages at this time.
Log Name: Application
Source: VSS
Date: 8/12/2014 9:20:33 AM
Event ID: 8225
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: mdnfile.Auxiant.local
Description:
The VSS service is shutting down due to shutdown event from the Service Control Manager.
7.
VSS still having issues here...not stopping!
Log Name: Application
Source: VSS
Date: 8/12/2014 9:42:33 AM
Event ID: 11
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: mdnfile.Auxiant.local
Description:
Volume Shadow Copy Service information:
The COM Server with CLSID {e579ab5f-1cc4-44b4-bed9-de0991ff0623} and name IVssCoordinatorEx2 cannot be started.
Most likely the CPU is under heavy load. [0x80080005, Server execution failed]
8.
Not sure why this is started again?
Plus a DCOM timeout...
Log Name: Application
Source: Microsoft-Windows-Backup
Date: 8/12/2014 9:31:13 AM
Event ID: 753
Task Category: None
Level: Information
Keywords:
User: SYSTEM
Computer: mdnfile.Auxiant.local
Description:
The Block Level Backup Engine service has successfully started.
Event Xml:
Log Name: System
Source: Microsoft-Windows-DistributedCOM
Date: 8/12/2014 9:31:43 AM
Event ID: 10010
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: mdnfile.Auxiant.local
Description:
The server {E579AB5F-1CC4-44B4-BED9-DE0991FF0623} did not register with DCOM within the required timeout.
9.
Lots more errors about VSS...repeating Event ID's 11 and 8193...
Log Name: Application
Source: VSS
Date: 8/12/2014 9:42:33 AM
Event ID: 8193
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: mdnfile.Auxiant.local
Description:
Volume Shadow Copy Service error: Unexpected error calling routine CoCreateInstance.
hr = 0x80080005, Server execution failed
. Operation:
Instantiating VSS server
10.
Then, finally stopping again...
Log Name: Application
Source: Microsoft-Windows-Backup
Date: 8/12/2014 9:51:13 AM
Event ID: 754
Task Category: None
Level: Information
Keywords:
User: SYSTEM
Computer: mdnfile.Auxiant.local
Description:
The Block Level Backup Engine service has stopped.
11.
And still repeating 11 & 8193 at this point...as we write up this post.
12.
This was suggested by the Backup Assist - software vendor....
A.)
...for 0x81080101:
"ERROR - The shared restore point operation failed with error (0x81000101)
The creation of a shadow copy has timed out. Try again this operation."
If you receive the above error in your report, we have recently found a registry key which
contains the timeout value set by Microsoft for VSS and found that you can edit this to extend it.
To get to the registry key, open Regedit (Start > Run > Regedit) and browse to:
"HKLM\Software\\Microsoft\\Windows NT\\CurrentVersion\\SPP\\CreateTimeout".
Change value to 1200000 (2*10*60*1000 = 20 mins).
The default for the timeout period is 10 mins so hopefully doubling this will give enough time to
complete the snapshot.
If this isn't present, you'll need to manually enter the registry key. It's a DWORD 32-bit key.
--> NOTE: HAve not done this yet.
B.)
And for 0x80080005...
http://support.microsoft.com/kb/870655/en-gb
Our server does have high CPU load at times...
13.
Another post from technet mentions system volume size - here is our layout below:
A.)
OS/system Disk...
DISKPART> list disk
Disk ### Status Size Free Dyn Gpt
Disk 0 Online 278 GB 0 B (System Vol = C: Drive - OS w/ Programs and batch jobs)
Disk 1 Online 2791 GB 0 B * (Data drive - DFS, rsync,etc. run on D: Drive)
Disk 2 Online 1863 GB 0 B (USB 2/3.0 - Enclosure w/ SATA disk = 2GB formatted w/ NTFS.)
DISKPART> detail disk
DELL PERC H710P SCSI Disk Device
Disk ID: CA0DEEA5
Type : RAID
Status : Online
Path : 1
Target : 0
LUN ID : 0
Location Path : PCIROOT(0)#PCI(0100)#PCI(0000)#RAID(P01T00L00)
Current Read-only State : No
Read-only : No
Boot Disk : Yes
Pagefile Disk : Yes
Hibernation File Disk : No
Crashdump Disk : Yes
Clustered Disk : No
Volume ### Ltr Label Fs Type Size Status Info
Volume 1 RECOVERY NTFS Partition 3072 MB Healthy System
Volume 2 C OS NTFS Partition 275 GB Healthy Boot
B.)
Backup disk...
ST320006 41AS USB Device
Disk ID: 801AF48C
Type : USB
Status : Online
Path : 0
Target : 0
LUN ID : 0
Location Path : UNAVAILABLE
Current Read-only State : No
Read-only : No
Boot Disk : No
Pagefile Disk : No
Hibernation File Disk : No
Crashdump Disk : No
Clustered Disk : No
Volume ### Ltr Label Fs Type Size Status Info
Volume 4 F BACKUPS NTFS Partition 1863 GB Healthy
C.)
NOTE:
Also on disk 0(system Disk) - we have a small OEM partition = 39MB and a Recovery Partition = 3GB.
D.)
Issues with WBAdmin now...
C:\>wbadmin get disks
wbadmin 1.0 - Backup command-line tool
(C) Copyright 2004 Microsoft Corp.
ERROR - A Volume Shadow Copy Service operation error has
occurred: (0x80042302)
A Volume Shadow Copy Service component encountered an unexpected error.
Check the Application event log for more information.
14.
Ran "Check-Disk" -no errors...
C:\>chkdsk
The type of the file system is NTFS.
The volume is in use by another process. Chkdsk
might report errors when no corruption is present.
Volume label is OS.
WARNING! F parameter not specified.
Running CHKDSK in read-only mode.
CHKDSK is verifying files (stage 1 of 3)...
116224 file records processed.
File verification completed.
654 large file records processed.
0 bad file records processed.
0 EA records processed.
76 reparse records processed.
CHKDSK is verifying indexes (stage 2 of 3)...
156676 index entries processed.
Index verification completed.
0 unindexed files scanned.
0 unindexed files recovered.
CHKDSK is verifying security descriptors (stage 3 of 3)...
116224 file SDs/SIDs processed.
Security descriptor verification completed.
20227 data files processed.
CHKDSK is verifying Usn Journal...
35628352 USN bytes processed.
Usn Journal verification completed.
Windows has checked the file system and found no problems.
289233919 KB total disk space.
100241492 KB in 85477 files.
57380 KB in 20228 indexes.
0 KB in bad sectors.
226627 KB in use by the system.
65536 KB occupied by the log file.
188708420 KB available on disk.
4096 bytes in each allocation unit.
72308479 total allocation units on disk.
47177105 allocation units available on disk.
15.
There are no backup error logs here - only .etl files...
C:\Windows\Logs\WindowsServerBackup\
Wbadmin.1.etl = 9:42AM on 8/12
Wbadmin.2.etl = 4:51AM on 8/12
Per windows Win troubleshooting:
http://technet.microsoft.com/en-us/library/ee692290(WS.10).aspx
http://blogs.technet.com/b/filecab/archive/2009/09/16/diagnosing-failures-in-windows-server-backup-part-1-vss-spp-errors.aspx
--> Related to Windows backup and not VSS?
2155348001 - 0x80780021 - Windows Backup timed-out before the shared protection point was created.
Should we run the "FIXVSS08.BAT" script - restarts services and re-registers DLL's - does this ever help?
Will this server need a reboot since VSS is in a state and we can't even test backups?
Is there a tool we can use to debug VSS or troubleshoot or OS backups further?
MSReport/Diag, VSS debugging, or another way of helping fix our OS(SystemState/C: drive) backup issues?
Any help is greatly appreciated - thx.
-SP
Scot PichelmanHi Scot,
Sometimes a third party backup application will change the default VSS provider adding VSS writers which causes Windows Server Backup using an incorrect component in doing backup, that's why I suggest to delete it for a test. Anyway you can reinstall it
after testing.
And as you said a high-load CPU could also be the cause of the issue so you may need to first rule hardware effective out.
If issue still exists, please try the steps below as well:
1. Please check DCOM Server Process Launcher service is started.
2. If not, try to change RPC service to run as SYSTEM account. To do this please modify the following key:
Note: Backup the key before edit incase it cause any issue.
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Servic es\RpcSs\
Find "ObjectName" and change the value to LocalSystem.
Reboot to see if DCOM back to work and if Windows Server Backup starts working. If not, please provide the result if there is any new error occurs.
If DCOM still not work, please compare the content of the following key with a working server:
HKEY_LOCAL_MACHINE\Software\Microsoft\OLE
Edit if there is any difference and make sure you created a backup of the key before editing.
If you have any feedback on our support, please send to [email protected] -
We have two Physical Windows 2012 R2 Storage Edition configured as DFS.
Server B is read only and nobody can access it, all the clients conntect to server A.
On both server we have activated VSS
We have 6 TB of shared data on the server.
One share is very large 3TB and its staging area is 1 TB large.
In DFS event log I see 4202 error for this share.
VSS are scheduled at 10 am and 15 pm and stored on a different disk partition 1,5 TB large.
When VSS start and for an hour long server A:
- stop responding to user interfaces
- windows 2003, 2003 R2, xp client can't connect anymore to the shares
- windows 2008, vista or higher access to share is very slow
Any ideas?Hi,
Sorry for the delay.
If you can access the shared folder using IP address, you could enable NetBIOS over TCP/IP on server to see if the issue still exists.
For more detailed information, please see:
Error : No network provider accepted the given network path" when trying to access server via UNC
https://social.technet.microsoft.com/Forums/windowsserver/en-US/5c21fd13-41d7-452e-a9c7-8dc58d12c5b5/error-no-network-provider-accepted-the-given-network-path-when-trying-to-access-server-via-unc?forum=winserverfiles
Best Regards,
Mandy
Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Subscriber Support, contact [email protected] -
DPM 2012 Backing up a VM on Server 2012 Hyper-V CSV Host - Not Working with Hardware VSS
Hi All,
I'm trying to backup a VM on a 2012 Cluster. I can do it by using the system VSS provider, but when I try to use the hardware provider, (Dell equalogic) it doesn't work. DPM will sit for a while trying and then report a retryable VSS error.
The only error I'm seeing on the Host is the following:
Event ID 12297Volume Shadow Copy Service error: The I/O writes cannot be flushed during the shadow copy creation period on volume \\?\Volume{3312155e-569a-42f3-ab3a-baff892a2681}\. The volume index in the shadow copy set is 0. Error details: Open[0x00000000, The operation completed successfully.
], Flush[0x80042313, The shadow copy provider timed out while flushing data to the volume being shadow copied. This is probably due to excessive activity on the volume. Try again later when the volume is not being used so heavily.
], Release[0x00000000, The operation completed successfully.
], OnRun[0x00000000, The operation completed successfully.
Operation:
Executing Asynchronous Operation
Context:
Current State: DoSnapshotSet
I don't know where to go from here - There is no activity on the CSV (this is the only VM on it, and both the CSV and VM were created specifically for testing this issue
Does anyone have any ideas? I'm desperate.
Update:
Ok, so I can Take DPM out of the picture. Trying to do a snapshot from the Dell Auto-Snapshot manager, I get the same errors. But I also get a bit more information:
Started at 3:02:47 PM
Gathering Information...
Phase 1: Checking pre-requisites... (3:02:47 PM)
Phase 2: Initializing Smart Copy Operation (3:02:47 PM)
Adding components from cluster node SB-BLADE01
Adding components from cluster node SB-BLADE04
Adding components from cluster node SB-BLADE02
Retrieving writer information
Phase 3: Adding Components and Volumes (3:02:52 PM)
Adding components to the Smart Copy Set
Adding volumes to the Smart Copy Set
Phase 4: Creating Smart Copy (3:02:52 PM)
Creating Smart Copy Set
An error occurred:
An error occurred during phase: Creating Smart Copy
Exception from HRESULT: 0x80042313.
Creating Smart Copy Set
An error occurred:
An error occurred during phase: Creating Smart Copy
Exception from HRESULT: 0x80042313.
An error occurred:
Writer 'Microsoft Hyper-V VSS Writer' reported an error: 'VSS_WS_FAILED_AT_FREEZE'. Check the application component to verify it is in a valid state for the operation.
An error occurred:
One or more errors occurred during the operation. Check the detailed progress updates for details.
An error occurred:
Smart Copy creation failed.
Source: Creating Smart Copy Set
An error occurred:
An error occurred during phase: Creating Smart Copy
Exception from HRESULT: 0x80042313.
An error occurred:
Writer 'Microsoft Hyper-V VSS Writer' reported an error: 'VSS_WS_FAILED_AT_FREEZE'. Check the application component to verify it is in a valid state for the operation.
An error occurred:
One or more errors occurred during the operation. Check the detailed progress updates for details.
Error: VSS can no longer flush I/O writes.
Thanks,
JohnI had a similar issue with an environment that had previously been working with the Dell HIT configured correctly. As we added a third node to the cluster I began seeing this problem.
In my case I had the HIT volume max sessions per volume at 6 and maximum sessions per volume slice set to 2 and the CSV was using a LUN/Volume on the SAN that was split across 2 members.
When the backup takes place and Dell HIT is configured to use SAN snapshots the vss-control iSCSI target is used which in my case exceeded my limits for maximum connections per volume as I'm using 2 paths per Hyper-V node with MPIO (this is my
current theory).
Once I'd modified these settings I could then back up the VHD's on that CSV again.
Hope this helps.
Maybe you are looking for
-
How and Where Do I install downloadable Plugins for Shake 4.1?
I've looked on this Forum and can't find the answer to my question above. There was one answer that I found, but I wasn't sure it applied to my issue. I have downloaded some free plug ins for Shake and have no idea how to insert into the app so that
-
Unable to view tables in Oracle BI Answers
Hi, I am able to view my subject area in Oracle BI Answers, but the tables aren't displaying. Also the tabs arent functioning. I was able to view the data in my repository and there is no error. What might be the problem for this? We uninstalled obie
-
Need for iPad/PC/Mac networking at hospitals-
Need for networking at hospitals… Our doctor (plus an IT guy for the hospital) is the go-to guy for the MDs that use Island hospital and want to review test results and documentation on each patient on the network. All imaging systems are now digital
-
Stupid, simple question...I hope
When I want to play the songs in my library, I go to edit and hit select all. All my songs get highlighted but not checked, therefore, only the first song plays. Do I really need to physically check every one of the songs in my library in order to ge
-
Compensate for file path changes in dynamically loaded HTML?
I'm using DWCS3 to work on a site which has the HTML inside a "Content" folder. The site's programmer uses AJAX to load the HTML into ASPX files which are in the root folder. So, although the HTML files are in the Content folder, all of their relativ