Fabric Interconnect Uplink port channeling...

Hi,
We setup port channel of two 10G uplink interfaces on each Fabric Interconnect. Everything is running as expected.
However we noticed that under 'Port-Channel # (Fabric #)' setting on LAN tap, 'Admin Speed' was set to '10Gbps'.
Should we leave this configuration as is or match to the maximum bandwidth of port Channel we set up (20Gbps)?
I only see four options on each port channel configuration, 1 Gbps, 10Gbps, 20Gbps and 40Gbps so I'm not sure if we should change this setting.
Thank you

Hello,
You do not need to change the speed ( admin ). 20 Gbps configuration is applicable for certain server models.
-------- NXOS output -----------------
FI-A(nxos)# sh run int port-channel 1
interface port-channel1
description U: Uplink
switchport mode trunk
speed 10000
FI-A(nxos)# show int port-channel 1
port-channel1 is up
Hardware: Port-Channel, address: 547f.ee67.e0fb (bia 547f.ee67.e0fb)
Description: U: Uplink
MTU 1500 bytes, BW 20000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is trunk
full-duplex, 10 Gb/s
FI-A(nxos)# sh int bri
Port-channel VLAN Type Mode   Status Reason                    Speed Protocol
Interface
Po1          1     eth trunk up      none                       a-10G(D) lacp
There is cosmetic defect where we list the operational speed as 10 Gbps instead of 20Gbps
FI-A /eth-uplink/fabric # show port-channel detail
Port Channel:
    Port Channel Id: 1
    Name: FIA
    Admin State: Enabled
    Oper State: Up
   Speed: 10 Gbps
    Oper Speed: 10 Gbps
    State Reason:
    flow control policy: default
HTH
Padma

Similar Messages

Fabric Interconnect uplink failure

Hi, FI-A has 10GE uplink to LAN switch-A. Let's call this UplinkA FI-B has 10GE uplink to LAN switch-B. Let's call this UplinkB If UplinkA fails, but the server ports between FI-A and IOM-A is still up. Will traffic be sent to FI-A and then black hole there? I tried this in our setup and it seems to be the case. Then I try to apply Network control policy to the vnic.. In the policy, i specify {"CDP=disabled", Action on uplink fail=link down"} But it still did not work, when UplinkA is down, my ESX still see it's VMNIC as up and continue to send packet to FI-A and blackhole there. (1) Is Network Control policy required to detect FI uplink failure to work properly? (2) How do i see the operation status of the Palo vnic, i.e, whether it is up or down? Can't seem to find a place to see the status. Did i miss out anything? THanks Eng Wee

Hi Ferreira, Thanks for the reply. The uplink LANSwitch is actually a VSS. If i connect the two FI together using some 10GE ports, a layer 2 loop will be formed unless i can etherchannel (Multichassis Etherchannel) the uplink of FI-A and FI-B when connecting to the VSS. I think MEC is not supported in the FI. Alternate solution would be to have each FI to dual home to the LANSwitch. Each FI then creates a port-channel to the LANSwitch. In this way, there will be no L2 loop and no single point of failure. Unfortunately, my client only purchase 1x10GE uplink on each FI. How does end-host mode solve this problem? WHen UplinkA is down, vnic will go down? Rgds Eng Wee

Fabric Interconnect Uplinks to Nexus 7710

Can someone please help suggest how many Fabric Interconnect switches I can uplink to pair of Nexus 7710's.

in most deployment, Fabric Interconnect is put as a pair for the redundancy. They are either vPCed to the Nexus switches or straight-through up to non-Nexus switches.
i guess we would like to see more background of your questions in term of traffic requirement, what kind of line cards used on the N7710 etc.
Regards,
Michael

UCS fabric Interconnect storage ports

Hi experts,
Is it possible in FI to connect storage switches directly to fixed FCOE ports rather than having FC expansion module.
Thanks,
Mazhar

Sorry think I replyed in error to your question. Was thinking strictly FC. If you are doing FCOE, well that is ethernet. Think you would need to go appliance port to plug straight into device without a switch between. Shouldn't need anything special.
Craig

PIN GROUP, Port Channel

Hi, Cisco Gurus:
Need your authoritative take on the following: Ethernet and FC is all End-Host Mode.
1. If client is using FC Port Channel on all of the FC Uplinks for the 2 Fabric Interconnects with a SINGLE VSAN on each FI, MUST THEY USE SAN PIN GROUP? Currently, they are pinning the vHBA template to this SAN Pin Group. vHBA-Fabric A pins to FC-PIN-GROUP-A and vice-versa,
2.
If FC Port Channel is being used as in 1, from WHICH UCSM option should the VSAN be created?
Should VSAN be STILL created under the Individual Fabric A and B as before?
Or
It have to be Global VSAN or Dual VSAN outside the Fabric A or Fabric B?
3.If client is using Ethernet Port Channel for each Fabric Interconnet or each fabric, must they use LAN PIN GROUP?
Currently, they are pinning the vNIC template to this Pin Group (Fabric A & B). vNICA-Fabric A pins to LAN-PIN-GROUP-A and vice-versa.
Something which I have SELDOM SEE before.
As usual, I know it is easy meat for Cisco gurus. Thanks, PAL.
So Long.
SiM

All VSANs in the screenshot above are Global. Notice they're not created under the Fabric A/ B section.
Unlike "VLANs", which are almost always Global and normally accessible from each Fabric Interconnect, VSANs are different. Best practice for storage has keeping each SAN Fabric separate. Keeping this in mind, does it make more sense to create VSANs as global or under their respective Fabric interconnects? (Under the separate Fabric Inteconnects of course. The only use case for using Global VSANs is as Craig pointed out - when there is a single Storage switch northbound that both Fabric Interconnects uplink to.
Where this gets confusing is when you're assigning vHBA's to a VSAN.
Ex.
Let's say, as your customer has done, they've created their two VSAN 10 and 20 as "Global". For comparison sake I've also created VSAN 100 just under Fabric-A and VSAN 200 just under Fabric-B. Their VSANs would looks like:
Now, when creating an vHBA under your service profile, you need to assign it to a VSAN. Take a look:
So depending which Fabric (A/B) they want the vHBA connected to the options will change. You can see that by creating the VSANs 10 and 20 as "Global" they appear when either Fabric is selected - Conversly you can see the two 100 and 200 I created under each Fabric, only appear when that fabric is selected.
So what? Well if your upstream SAN switches only use one of the respective VSANs configured and only connect to one Fabric Interconnect, then you can potentially "blackhole" your storage traffic by assigning it to the wrong Fabric Interconnect. Creating the VSANs only under the appropriate Fabric Interconnect prevents this possibility.
SAN Port-Channels have no bearing on this. All a SAN Port Channel does is make mulitple FC Uplinks appear as one - which requires SAN Port Channels configured upstream also. Leaving the FC Uplinks as individual just lets UCS dynamically pin virtual interfaces across the FC uplinks in a round-robin fashion.
In your case you have no need for Pin-Groups either. As Jeremy pointed out, unless you have a specific need for them, it's safer to not use them.
Regards,
Robert

MDS configuration for port channel

I've tried to set up the new FC port channel upload to a pair of MDS 9124s, but as I don't know enough about the MDS side I can't get the link to come up. Are there any references available anywhere that tell you blow by blow exactly how to configure the MDS side of the port channel? Or perhaps a sample working config?
Thanks

Simon,
A few items:
1) You are on 1.4.1i Balboa code
2) FI's are in FC switch mode as MDS 9124's don't support F-port channel (NPV/NPIV)
3) It's a lot easier the first time if you have matched speed sfp+ all around. Although, with the correct configs, unmatched speeds can be made to work.
If the FIs are in switch mode and your MDS is running a minimum 3.3 here are some configs that may help:
MDS side interfaces 4/11 and 4/12 are plugged into the UCS fc ports. First setup and ensure ISLs are working properly between UCS and MDS, then configure the port-channels
interface fc4/11
switchport rate-mode dedicated
switchport mode E
switchport trunk mode auto
no shutdown
interface fc4/12
switchport rate-mode dedicated
switchport mode E
switchport trunk mode auto
no shutdown
The UCS GUI Equipment -> Fabric Interconnects -> FI-A -> Set FC Switching Mode. This will cause both FI's to reboot into FC switch mode. This is VERY DISRUPTIVE, both FI's will reboot.
On the UCS CLI you should see this on the UCS fc ports connected to the MDS
cae-sj-ca3-A(nxos)# show running-config interface fc 2/1-2
!Command: show running-config interface fc2/1-2
!Time: Wed Oct 20 16:49:39 2010
version 4.2(1)N1(1.4)
interface fc2/1
switchport mode E
no shutdown
interface fc2/2
switchport mode E
no shutdown
Until you have VSAN trunks enabled, make sure the ports on the UCS and MDS are in the same VSAN. VSAN 1 for example.
Once you have working ISLs, then proceed to turn them into port-channels and enable VSAN trunking. Start with the MDS
Create a channel group, it should look like this
cae-sj-9506-1# show run interface port-channel 3
!Command: show running-config interface port-channel 3
!Time: Wed Oct 20 17:06:01 2010
version 5.0(1a)
interface port-channel 3
channel mode active
switchport mode E
switchport rate-mode dedicated
switchport trunk mode auto <-- VSAN trunking
Enable VSAN trunking on the MDS ISL interfaces, it'll look like this:
cae-sj-9506-1# show run interface fc 4/11-12
!Command: show running-config interface fc4/11-12
!Time: Wed Oct 20 17:07:05 2010
version 5.0(1a)
interface fc4/11
switchport rate-mode dedicated
switchport mode E
switchport trunk mode auto
no shutdown
interface fc4/12
switchport rate-mode dedicated
switchport mode E
switchport trunk mode auto
no shutdown
Add the channel group information to the MDS ISL interfaces:
Enter configuration commands, one per line. End with CNTL/Z.
cae-sj-9506-1(config)# interface fc 4/11-12
cae-sj-9506-1(config-if)# channel-group 3 force
fc4/11 fc4/12 added to port-channel 3 and disabled
please do the same operation on the switch at the other end of the port-channel,
then do "no shutdown" at both ends to bring it up
cae-sj-9506-1(config-if)# show run interface fc 4/11-12
!Command: show running-config interface fc4/11-12
!Time: Wed Oct 20 17:07:39 2010
version 5.0(1a)
interface fc4/11
switchport rate-mode dedicated
switchport mode E
switchport trunk mode auto
channel-group 3 force
no shutdown
interface fc4/12
switchport rate-mode dedicated
switchport mode E
switchport trunk mode auto
channel-group 3 force
no shutdown
Create the SAN port channel on the UCS side
SAN -> SAN Cloud -> Fabric A -> FC Port Channels -> Create Port Channel
After you create the SAN port channel in UCS, make sure and enable it. I also bounce the MDS port channel at this point.
If all is correct, in a few agonizing minutes, the port-channel will be formed and passing data.
cae-sj-ca3-A(nxos)# show interface san-port-channel 1
san-port-channel 1 is trunking
    Hardware is Fibre Channel
    Port WWN is 24:01:00:0d:ec:d3:5d:c0
    Admin port mode is E, trunk mode is on
    snmp link state traps are enabled
    Port mode is TE
    Port vsan is 1
    Speed is 8 Gbps
    Trunk vsans (admin allowed and active) (1,10,26,50,66,100-101,103,123,222,24
0)
    Trunk vsans (up)                       (1,10,50,100,103)
    Trunk vsans (isolated)                 (26,66,101,123,222,240)
    Trunk vsans (initializing)             ()
    5 minute input rate 2312 bits/sec, 289 bytes/sec, 2 frames/sec
    5 minute output rate 1440 bits/sec, 180 bytes/sec, 2 frames/sec
      2669 frames input, 194760 bytes
        0 discards, 0 errors
        0 CRC, 0 unknown class
        0 too long, 0 too short
      2677 frames output, 158316 bytes
        0 discards, 0 errors
    0 input OLS, 1 LRR, 0 NOS, 0 loop inits
Again, make sure you have the ISLs up an running first before configuring the port-channels. It makes troubleshooting much easier.
Let me know if you need any help.

Creating san-port-channel on 6248 using CLI

I can create the san port channel using UCS manager. But I would like to know the syntax using CLI. So far I was able to figure out the following. One thing missing was moving the port channel from the default VSAN(1) to a different VSAN i.e VSAN 10 in my case. I have looked at the CLI guide, but have not found the commands syntax. Appreciate if anyone can post the syntax
### Create Port Channel ###
scope fc-uplink
scope fabric a
create port-channel 1
enable
set name port-channel-1
set adminspeed auto
commit-buffer
end
### Add interfaces to Port Channel ###
scope fc-uplink
scope fabric a
scope port-channel 1
create member-port 1 29
exit
create member-port 1 30
commit-buffer
end

Thanx Wdey. That's exactly what I was looking for. Here is the syntax which works for me.
scope fc-uplink
scope fabric a
create port-channel 1
enable
set name port-channel-1
set adminspeed auto
commit-buffer
end
scope fc-uplink
scope fabric a
enter port-channel 1
enable
enter member-port 1 29
enable
exit
enter member-port 1 30
enable
exit
set adminspeed auto
exit
enter vsan VSAN10 10 10
set fc-zoning disabled
set fcoe-vlan 10
set id 10
enter member-port-channel a 1
exit
localize
exit
end
commit-buffer

Multiple vsan traffic over single port-channel

Hi -
Scenario - 2 interface uplink (port-channel - Po10 ) from NetApp FAS-A to N5548-A & B. Po10 is currently configured with vPC10 and vFC10 at N5k end. single vfc currently mapped with a single vsan (vfc10 with vsan 1011).
Q - Is it possible to make the Port-channel to pass multiple vsan (vsan 1011 & 1012). If yes, then how (over same vfc or by separate vfc on same port-channel)
Subhankar

This router’s capability is only limited and dependent on the services that your ISP has given or allowed for you to use. I think it really has to be a one is to one configuration, not only with this router because I haven’t noticed any router that has this feature so far. This is really another idea for Linksys can work on.

Port-channel Problem between Fabric Interconnect and N7K vPC

Dear all,
I have a problem with Port-channel Uplink between Fabric Interconnect with N7K using vPC
This is my network topology for UCS Deployment
In N7K I has configured vPC for red link and green link, at Fabric Interconnect A I has configured Port-Channel with member is Port 1 and Port 2, uplink is red link. At Fabric Interconnect B, I has configured Port-Channel with member is Port 1 and Port 2, uplink is green link.
The show interface port-channel on N7K is good, every port-channel is up and have all member. But At Fabric Interconnnect, when I see on UCS Manager, the status of Port-Channel on Fabic A and Fabric B is fault with Additional Info: No operational member. Although all link is link up and I has status of Port-Channel is enable on UCS Manager. When I see the Properties of Port 1, Port 2 on Port-channel, I see the membership status is : individual. This mean port-channel is not up and no membership in this configuration. I want to using port-channel for load balance and plus more bandwidth for uplink to 20Gig. I don't understand why ?
Please help me resolve this problem, I has send the capture screen of UCS Manager when I show status of Port-channel and Port-member in port-channel in attach items.
Anyone can help me to resolve this, thanks you very much. Please reference attach items for more detail about fault.
Thanks,
Trung.

Thanks Matthew very much,
I has resolved this problem. The reason of problem is miss match protocol of port-channel between N7K and Fabric Interconnect. The Fabric Interconnect always use LACP protocol, but N7K using Port-channel mode on, that why the port-channel failed. I has configured LACP for port-channel in N7K, it has resolved the problems.
Thanks,
Trung.

Fabric Interconnect To MDS 9148 FC Port Channel/Trunks

Is it possible to create both a Port Channel and Trunk on a Cisco MDS 9148? Yes correct?
Or can I only trunk between the 2 MDS switches and not the MDS to Fabric Interconnect?
These FC links would be connected to a 2248XP Fabric Interconnect carrying multiple VSANs from the MDS to the Fabric Interconnect.
Thanks

Hello,
Yes, you can have trunk links grouped into port channel ( F port-channel trunking ) between FI and MDS.
http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/2.0/b_UCSM_GUI_Configuration_Guide_2_0_chapter_0101.html
Padma

Port-channel L2 problem with Fabric Interconnect and Nexus 7010

Hi,
i using port-channel from both fabric interconnect to N7k with 3 cables per Fabric Interconnect.
but, my problem is when i creating port-channel, Fabric Interconnect don't support mode ON dan rate-mode share in Interface 10G Nexus 7010.
I was trying :
1. I using non dedicated port in Nexus 7010.
      - rate-mode share
      - channel-group 1 mode active
      - switchport mode trunk
when i using this option, the port-channel in Nexus 7010 was suspended
2. I using non dedicated port in Nexus 7010
     - rate-mode share
     - channel group 1 mode on
     - switchport mode trunk
when i using this option, the port-channel in Nexus 7010 was came up, but in Fabric interconnect was failed.
3. I using dedicated port in Nexus 7010
     - rate-mode share
     - channel group 1 mode active
     - switchport mode trunk
when i using this option, the port-channel in Nexus 7010 was suspended
4. I using dedicated port in Nexus 7010
     - rate-mode dedicated
     - channel group 1 mode active
     - switchport mode trunk
when i using this option, the port-channel in Nexus 7010 was came up and running well.
but, the problem is my costumer do not want using a dedicated rate-mode. if i using dedicated mode the only available port is 8 interfaces instead of 32 ports. i want to using rate-mode share in nexus 7010.
is there any way to configuring port-channel using mode on in fabric interconnect ? i was trying using CLI to create port-channel in Fabric interconect but i cannot configure the channel group protocol.
i attach the topology of N7K with Fabric interconnect.
regards,
Berwin H

Hi Manish,
the issue was solved, i was fix it last week.
the solution is:
i enable the license grace-priode (since my license is Enterprise so cannot create VDC) then i create a VDC (ex: VDC 2) so i allocate the interface on all module
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-qformat:yes;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin-top:0cm;
mso-para-margin-right:0cm;
mso-para-margin-bottom:10.0pt;
mso-para-margin-left:0cm;
line-height:115%;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:"Times New Roman";
mso-fareast-theme-font:minor-fareast;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
N7K-M132XP-12 to VDC 2. after that i delete VDC 2 then all interface back to VDC 1 (default vdc). then i enable the rate-mode share in dedicated port and bundle into port-channel and its working.
i dont know why it must move to VDC first then it will working, maybe cisco can explain the reasons.
So here the result of my port-channel :
SVRN7KFARM-HO-01# show port-channel summary
Flags: D - Down        P - Up in port-channel (members)
        I - Individual H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        S - Switched    R - Routed
        U - Up (port-channel)
Group Port-       Type     Protocol Member Ports
      Channel
1     Po1(SU)     Eth      LACP      Eth1/1(P)    Eth1/2(P)    Eth1/3(P)
                                     Eth1/4(P)    Eth1/25(P)
2     Po2(SU)     Eth      LACP      Eth1/9(P)    Eth1/10(P)   Eth1/11(P)
                                     Eth1/12(P)   Eth1/26(P)
3     Po3(SU)     Eth      LACP      Eth1/17(P)   Eth1/18(P)
4     Po4(SU)     Eth      NONE      Eth10/32(P) Eth10/34(P) Eth10/35(P)
                                     Eth10/36(P)
Thanks.
Berwin H

No link between IOM port 2/2/3 and fabric interconnect B

Hello
I am trying to remove an uplink from an IOM to the fabric interconnect as its not required. I have tried to reset the I/O module and re-acknowledge the chassis with no luck, it is still shown in the GUI with the error message as in the subject. We are currently running 2.0(1s). The kit isn't in production yet.
Thanks for any help

Yes i agress thats a pretty heavy approach to clean it. No if you left it there as it is, there would have been no operational issues. It would have just left this fault in the UCSM.
./Abhinav

Appliance port on 6248 Fabric Interconnects

Is it possible to setup a storage device with a CIFS share on the 6248's through a port defined as an appliance port? The GUI guide says this:
"Appliance ports are only used to connect fabric interconnects to directly attached NFS storage."
But what would be the difference if it were a CIFS share rather than an NFS mount?
I'm trying to setup a storage unit that does both NFS and CIFS, but the backup software that I want to point to it through the FI's needs to have it as a CIFS share due to being a Windows platform.
Any and all comments welcome.
Thanks,
Chad

iSCSI, CIFS and NFS targets are all valid uses for the appliance port.
All the appliance port does is allow MAC learning on the interface.
Regards,
Robert

- No link between IOM port 3/2/2 and fabric interconnect B:1/18

Specifically I'm trying to find out what the 3/2/2 means in - No link between IOM port 3/2/2 and fabric interconnect B:1/18

This from the fault reference guide:
"No link between IOM port [chassisId]/[slotId]/[portId] and fabric interconnect [switchId]:[peerSlotId]/[peerPortId]"
http://www.cisco.com/en/US/docs/unified_computing/ucs/ts/faults/reference/UCS_SEMs.html#wp1306388

Fabric Port Channel

Hello,
Once in production, when adding cables in a Fabric PortChannel between an IOM and a FI, ¿is it needed to re-ack the chassis? (so that, we'll have downtime of the whole chassis). Or rather, as it is a Port-Channel the cables are detected automatically, so that it is not needed to re-ack, and so there's no downtime at all...which one is the correct statement?
Thanks,

Hello,
Let us consider following scenario on adding or removing links to existing deployment.
Initial deployment
UCSM 2.0, FI 62XX ,2208 IOM and 2 physical links with port channel configuration ( 2 link discovery policy )
Change
1) If additional links are added and if they need to be utilized and part of port channel, we need to re-ack the chassis.
2) If a link goes down,traffic is distributed across other links in port channel and does not require re-ack. Parameters like bandwidth, VIF ports that are calculated based on initial configuration remains the same.
3) If we physically remove the additional links between IOM / FI, re-acknowledment would create port-channel with original two links.Failure to re-ack will create over subscription ( bandwidth, available VIF ports ) of the network links.Re-ack is not mandatory but it creates appropriate configuration that matches required design criteria.
Caution
Adding or removing links from a fabric port channel is disruptive and may affect the available amount of VIF namespace.
http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/2.0/b_UCSM_GUI_Configuration_Guide_2_0_chapter_0101.html#concept_5D3D88341BFB43468B62B5A77876C67B
Padma

Fabric Interconnect Uplink port channeling...

Similar Messages

Maybe you are looking for