6509E VSS w/Quad-Sup 720 - Traffic Forwarding Issue

Question

Hello all,
I ran into an interesting problem yesterday evening while performing a quad-sup upgrade to an existing dual-sup 6509E VSS architecture. Unfortunately, the expiration of my maintenance window forced me to roll back the changes before I had the opportunity to truly discover the root issue, however, I wanted to post my experience on Support Forums with the hope that someone else has run into it before.
The customer I was working with has an existing dual-sup (720) environment operating in VSS mode. Both sup's were installed an operational in slot 5 of each chassis. The switch ID's had been assigned as 1 and 2, and the VSL was comprised of the two 10GE interfaces connected between each of the sup modules (1/5/4-2/5/4 and 1/5/5-2/5/5). The customer wanted to increase their intra-chassis redundancy by installing two additional sup 720 modules in slot 6 of each chassis, and adding these 10GE interfaces to the VSL.
The overall upgrade went very well. I began by physically installing an additional sup 720 in slot 6 of the VSS standby chassis, and the sup module came up into RPR-Warm status (which is the desired state). I then physically installed an additional sup 720 in the VSS active chassis, and it too came up into RPR-Warm status. After reviewing the VSS redundancy state, I determined that everything looked good, and proceeded to add the additional 10GE links (1/6/4-2/6/4 and 1/6/5-2/6/5) into the VSL configuration. I then physically connected the additional VSL links (but shifted them around as to achieve diversity between the chassis (1/5/4-2/5/4, 1/5/5-2/6/5, 1/6/4-2/6/4, 1/6/5-2/5/5). I then reviewed the VSL link state and EtherChannel state, and everything looked great.
At that point, we were pretty much done with the switch configuration, and proceeded to verify that server/network resources were not affected. I found that there seemed to be an issue sending traffic across the VSL. For example, some physical servers connected to switch ID 1 were having a difficult time communicating to physical servers connected to switch ID 2 (this customer is still working on dual-homing hosts). I looked at all of the obvious things - physical VSL interfaces, EtherChannel status, etc., and everything looked good. I then proceeded to remove physical VSL interfaces from the configuration, eliminating those that were added that evening (e.g., all interfaces except the original 1/5/4-2/5/4 were removed) - nothing. At this point, I was 10 minutes from the expiration of the maintenance window and was forced to roll back. It wasn't until the newly added sup's were physically removed from the chassis that the issue subsided. That said, I was able to conclude that the VSL wasn't the problem at all, but perhaps the installation of the sup's was??
All of the sup 720's were running 12.2(33)SXI6 code, which I had verified to be a safe harbor version (although there are known bugs, none appear to be related to this issue).
Has anyone run into this issue before? Any thoughts would be appreciated!

Answer 1

You bring up VSS outside of the current environemnt. Create all vlans with a spanning tree priority higher than the existing.
Create all the vlan interfaces and leave them shutdown.
Connect the existing core to the new VSS core. Make sure that all vlans are properly crossing the trunk between old and new cores. Depending on how you're routing is set up, you might need to create a vlan to use for routing updates only.
At this point, you can start changing spanning tree priorities to move the root of the vlans to the vss. Once these have been moved, you can start to manipulate hsrp.
Whichever switch has the backup interface, shut those interfaces down. After this is done, shut the vlan inteface on the old core and no shut on the vss. Since vss doesn't use hsrp, it's hard to manipulate the vss since you want the vlan interface to be the previous standby ip.
Flipping the vlan intefaces shouldn't cause any issues. I've done this method several times in large hospitals with no issues.
Once you have the vss running, you can move the access switches 1 link at a time. Make sure you're running rapid pvst.
The method I've used for this is to create a port channel on both the access switch and the vss. On the vss, you can assign the interfaces into the port channel immediately. On the access switch, disconnect one of the interfaces that goes to the old core. Add the disconnected inteface into the port channel then plug it back in. It should come up as the only member of the etherchannel. After you verify this link is up properly, perform the same with the second uplink on the access switch. When you plug it in, it should join the channel and you should be fine. With rapid pvst, nothing should be noticeable when the links block and unblock.