Jumbo MTU on Nexus 2232 FEX

Hi,
This is related to bug CSCum09831 - jumbo MTU does not work on 2232 FEXes unless it is configured globally. Currently we have several qos groups with jumbo MTU enabled on only one of them and this setup works fine on 2148 FEXes but not on the 2232.
Now I want to enable Jumbo MTU on all qos groups but I'm not sure that this will solve the problem, any suggestions would be highly appreciated.
Thanks,
Adrian

Hi Adrian,
I checked the BUG and for 22XX Fex the Jumbo Frame configuration has to be done globally. This is part of the design for these series of Fex.
So i would suggest to enable Jumbo frames globally and see if the issue gets resolved. Else you can reach out to me i will be more than happy to assist.
Regards,
Seemab

Similar Messages

Nexus 2232 - sfpInvalid

I get sfpInvalid when inserting a FET-10G in a Nexus 2232. I've inserted several FET's in several ports, same error on all ports. Output;
nx-d12-01# sh int eth115/1/5 transceiver details
Ethernet115/1/5
    transceiver is present
    type is Fabric Extender Transceiver
    name is CISCO-JDSU
    part number is PLRXPL-VC-S43-CG
    revision is B
    serial number is JUS1708B94M
    nominal bitrate is 10300 MBit/sec
    Link length supported for 50/125um OM3 fiber is 100 m
    Link length supported for 62.5/125um fiber is 10 m
    cisco id is --
    cisco extended id number is 4
           SFP Detail Diagnostics Information (internal calibration)
                Current              Alarms                  Warnings
                Measurement     High        Low         High          Low
Temperature   31.91 C        75.00 C      5.00 C     70.00 C       10.00 C
Voltage        3.30 V         3.63 V      2.97 V      3.46 V        3.13 V
Current        0.00 mA       10.00 mA     2.59 mA     8.50 mA       3.00 mA
Tx Power        N/A           2.69 dBm -11.30 dBm   -1.30 dBm     -7.30 dBm
Rx Power        N/A     --    2.99 dBm -13.97 dBm   -1.00 dBm     -9.91 dBm
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
The port is configured default (no config on port);
Port          Name               Status    Vlan      Duplex Speed   Type
Eth115/1/5    --                 sfpInvali 1         full    10G     --

Hi Leo,
Apologies in advance if I'm teaching you to suck eggs in this explanation.
I guess technically and logically this is an interface on the parent Nexus switch, but in this case we can deduce it to be a physical interface on a FEX. Here we know the parent switch is a Nexus 5K as we can see the NX-OS version 5.2(1)N1(4) from the sh run int eth115/1/1 command output that was included. The N designation in the NX-OS version number is that given to the Nexus 5K platform.
nx..(config-if)# do sh run int eth115/1/1
!Command: show running-config interface Ethernet115/1/1
!Time: Thu Jul 4 11:22:47 2013
version 5.2(1)N1(4)
interface Ethernet115/1/1
The Nexus 5K built in ports are numbered Eth1/1-32 and eth2/1-16 (N5548) or Eth1/1-48, eth2/1-16, eth3/1-16 and eth4/1-16 (N5596) so it's not one of those.
For the host interfaces on the FEX, the number is in the form eth X/Y/Z where X is the number assigned to the FEX in the parent switch configuration, Y is always 1, and Z is the number of the physical interface on the FEX. In the case of the Nexus 2232 this would obviously be in the range 1-32.
So what we can see from the output the OP has included is that the FET has been inserted in host interface 1 and 5 of the FEX that is assigned the number 115 in the parent switch.
Regards

Odd power supply fault callhome emails on Cisco 2232 FEXs?

We recently enabled smtp callhome for our ucs pod a few days ago. Since then, we've received more than a dozen email bursts about power supply failures in each of the four Cisco Nexus 2232TP FEXs in our pod. In each case, we get a "Recovered" email within a minute or two, and we're not seeing power supply failure emails for any other devices within the UCS pod.
Here's a sample email burst from this morning, showing the subject lines within my email inbox:
System Notification from atl01-ucsf01a-A - environment:major - 2012-10-23 06:01:27 GMT-00:00 Recovered : Power supply 2 in fex 6 operability: inoperable
System Notification from atl01-ucsf01a-A - environment:major - 2012-10-23 06:01:27 GMT-00:00 Recovered : Power supply 2 in fex 6 operability: inoperable
System Notification from atl01-ucsf01a-A - environment:major - 2012-10-23 06:01:27 GMT-00:00 Recovered : Power supply 2 in fex 6 operability: inoperable
System Notification from atl01-ucsf01a-A - environment:major - 2012-10-23 06:01:27 GMT-00:00 Recovered : Power supply 2 in fex 6 power: error
System Notification from atl01-ucsf01a-A - environment:major - 2012-10-23 06:01:27 GMT-00:00 Recovered : Power supply 2 in fex 6 power: error
System Notification from atl01-ucsf01a-A - environment:major - 2012-10-23 06:01:27 GMT-00:00 Recovered : Power supply 2 in fex 6 power: error
System Notification from atl01-ucsf01a-A - environment:major - 2012-10-23 06:00:58 GMT-00:00 Power supply 2 in fex 6 operability: inoperable
System Notification from atl01-ucsf01a-A - environment:major - 2012-10-23 06:00:58 GMT-00:00 Power supply 2 in fex 6 operability: inoperable
System Notification from atl01-ucsf01a-A - environment:major - 2012-10-23 06:00:58 GMT-00:00 Power supply 2 in fex 6 operability: inoperable
System Notification from atl01-ucsf01a-A - environment:major - 2012-10-23 06:00:58 GMT-00:00 Power supply 2 in fex 6 power: error
System Notification from atl01-ucsf01a-A - environment:major - 2012-10-23 06:00:58 GMT-00:00 Power supply 2 in fex 6 power: error
If we were seeing this from just one fex, I'd suspect a hardware issue. If any of the other UCS devices (rackmount/blade servers, fabric interconnects, etc) reported similar issues, I'd suspect a power feed problem within the DC.
But this has me baffled. If it helps, the UCS pod is running firmware 2.0.3c
Cheers,
Paul

Thanks Paul,
It is known ( cosmetic ) issue with power supply model.
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtz84683
The above is resolved in recent version of Nexus 5K switch software. New defect to port the fix to UCSM has been submitted today
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCuc87547
It will take a while to get published on web site.
Even though they are cosmetic, I would suggest you open a TAC SR with UCSM, FEX tech support and above bug IDs so that we can track the resolution.
HTH
Padma

Nexus 2232 - 10gig only

Looking for a quick answer here. Someone is telling me they are able to purchase a 1gig Copper SFP tranciever and use that in a Nexus 2232. As in, connecting a 1gig connection into a Nexus 2232. I was under the impression that a N2232 was 10gig only
Am I wrong?
Thanks
Jeremy

Hey Jeremy,
Yes, the Nexus 2232PP can operate also as 1Gbps and the GLC-T transceiver is listed as compatible in the Cisco Nexus 2232PP matrix.
http://www.cisco.com/c/en/us/td/docs/interfaces_modules/transceiver_modules/compatibility/matrix/OL_6981.html
Regards.
Richard

Nexus 5548 fex interface configuration in vPC scenario

Hello Forum Team!
I have a pair of Nexus 5548UP configured as a vPC domain active-standby couple with six Nexus 2248TP Fex's in a dual-homed scenario (each fex is connected to each 5548UP). At the same time, both 5548UP are dual-homed to a pair of VSS Catalyst 6509.
I have a couple of questions regarding performance and configuration:
1. When configuring a fex interface, I've noticed that if the configuration is not the same or replicated on both 5548UP, the fex interface won't come up; is this normal behaviour? For now I am replicating manually the configuration on both 5548UP (not confyg-sync).
2. When performing ICMP tests from the 2248TP to the VSS, the response time is around 1ms to 2ms; from the 5548UP to the VSS pair is approximately 0.5ms. Is that normal delay response time from the 2248TP? I was expecting less response time from the 2248TP (the fex's have 20 Gigs of uplink bandwidth to the 5548UP pair and the 5548UP pair have 40 gig's of bandwidth to the VSS pair).
Thanks in advanced team!

Hi,
1. When configuring a fex interface, I've noticed that if the configuration is not the same or replicated on both 5548UP, the fex interface won't come up; is this normal behaviour?
Yes, that is a normal behavior. Since yours FEXes are connected to BOTH 5ks, you have to configure both 5Ks in order for the FEX to come up. If you connect your FEX physically to only one 5k, then you only configurethat one 5k, but in your case you need to configure both 5ks exactly the same.
2. When performing ICMP tests from the 2248TP to the VSS, the response time is around 1ms to 2ms; from the 5548UP to the VSS pair is approximately 0.5ms. Is that normal delay response time from the 2248TP? I was expecting less response time from the 2248TP (the fex's have 20 Gigs of uplink bandwidth to the 5548UP pair and the 5548UP pair have 40 gig's of bandwidth to the VSS pair).
I have never tested this, but I would think since the 2k needs to to 5k for every packet including talking to another host on the same 2k, the delay would slightly be higher then when you ping directly from the 5k.
HTH

Enable jumbo frame on Nexus switch

I read following phrase in configuration guide of Nexus 5020
The Cisco Nexus 5000 Series switch is a Layer 2 device. This means it does not fragment frames. As a result,
the switch cannot have two ports in the same Layer 2 domain with different maximum transmission units
(MTUs). A per-physical Ethernet interface MTU is not supported. Instead, the MTU is set according to the
QoS classes. You modify the MTU by setting Class and Policy maps.
When you show the interface settings, a default MTU of 1500 is displayed for physical Ethernet interfaces
and a receive data field size of 2112 is displayed for Fibre Channel interfaces
Has anybody configured jumbo frame with above mentioned way ?

Yes, it looks something like this:
policy-map jumbo
class class-default
mtu 9216
policy-map class-default
class class-default
mtu 9216
system qos
service-policy class-default

Nexus 2K FEX dual-homed to two Nexus 5K Configuration

Hi.
I am trying to look for Nexus 5K config when the nexus2K FEX is dual-homed into it.
What will be the configuration on both 5K especiall the pinning?
Assuming one N2K 10G port is connected to N5K-1 and another N2K 10G port connected to N5K-2?
N5Ks are also configured as dual-sided vpc with two N7K.
thanks

N5k-1 ====vpc peer==== N5k-2
\ E1/10 E1/10 /
\ Fex-100 /
N5k-1 & N5k-2 config:
===============
Interface po100
switchport mode fex-fabric
fex associat 100
vpc 100
int e1/10
switchport mode fex-fabric
channel-group 100

Nexus 2232 and IBM 10 Gig CNA compatability

Has anyone got a good doc on software/hardware compatability for Nexus, specifically 2232, and IBM 10 Gig CNAs? Im struggling with some Unix guys, I can use this particular pair of 2232s all day, but when I link a new 770 VIO....nada. We bypass the 2232 and link to the 5596 and it links up immediately. I suspect the IBM CNA and compatability but need the proof. Stumbling google I didn't see anything but I don't speak Unix very well either. We're running 2232PPs on 5596s at 5.1(3)N2(1b). Any help is appreciated.

hi Robert,
thanks a lot for your reply.
here is the uplink port profile config:
port-profile type ethernet HostUplink
vmware port-group
switchport mode trunk
switchport trunk native vlan 1
switchport trunk allowed vlan 1-3967,4048-4093
channel-group auto mode on mac-pinning
no shutdown
system vlan 803,832,900-901
state enabled
803 is mgmt for both esx and vsm, 900 is control, 901 is packet.
my NC553i driver version is different from what you suggested, it is 2.102.518.0, any reason why you suggest to use the version you specified?
I have been trying to contact HP support, to be honest, not saying overall, but from my experience, they are much less helpful then Cisco support, I will keep trying.
thanks
ming

Nexus 5548UP / FEX Connectivity doubt

Hello Everybody,
Some weeks ago, we decided to do a connectivity test with our nexus equipment. The topology was fairly simple, only one Nexus 5548UP, no vPC domain configured. 2 Nexus 2248TP Fabric Extenders connected to the 5548UP. Finally, a server, doing NIC teaming, ACTIVE-ACTIVE in a PortChannel config, one link to each Nexus 2248TP Fabric Extender. Topology attached below.
My understanding was that the FEXs were remote line cards, so what we are doing here is like adding 2 linecards to a moludar switch like a 6500 and creating a PortChannel using 1 link to 1 linecard, and 1 link to another linecard.
To my surprise, this was not allowed.
I tried to find and explanation why this could not be configured, but nothing found.
Anyone has an idea why this could be happening?
thanks in advance,
Fernando

Hello Everybody,
Some weeks ago, we decided to do a connectivity test with our nexus equipment. The topology was fairly simple, only one Nexus 5548UP, no vPC domain configured. 2 Nexus 2248TP Fabric Extenders connected to the 5548UP. Finally, a server, doing NIC teaming, ACTIVE-ACTIVE in a PortChannel config, one link to each Nexus 2248TP Fabric Extender. Topology attached below.
My understanding was that the FEXs were remote line cards, so what we are doing here is like adding 2 linecards to a moludar switch like a 6500 and creating a PortChannel using 1 link to 1 linecard, and 1 link to another linecard.
To my surprise, this was not allowed.
I tried to find and explanation why this could not be configured, but nothing found.
Anyone has an idea why this could be happening?
thanks in advance,
Fernando

Nexus 5500, FEX 2248T, ESX

v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin-top:0cm;
mso-para-margin-right:0cm;
mso-para-margin-bottom:10.0pt;
mso-para-margin-left:0cm;
line-height:115%;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
Hi Lucien,
I have a configuration in place according to the attached file net_design_n5k.vsd. Today I tried to configure port-channel interfaces between the FEX (in this case FEX 100) and attached ESX hosts.
When defining a VPC to the port-channel interface first by using stmt “vpc 203”, I received the error-msg below when entering the command “channel-group 203 ….” to the physical interfaces.
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin-top:0cm;
mso-para-margin-right:0cm;
mso-para-margin-bottom:10.0pt;
mso-para-margin-left:0cm;
line-height:115%;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
When defining the physical interface first with the stmt “channel-group 203 ….” I was not able to add the stmt vpc 203 to the channel interface.
The devices are running the latest NX-OS image according to the following screen-shots:
Any hints or tips are very much appreciated.
Best regards
Roman

Allright. Regarding ESX channels: Maybe a better option is to setup VST-mode and loadbalancing based on the virtual port ID on the vSwitch. Then attach the vNIC's to different port-groups (you will have one port group per VLAN). With this setup you don't need to configure a channel on the ESX but you still have a active/active design due to the fact, that the port-groups are distributed to the uplink-NIC's on your ESX host (all vNIC's in port group 1 will be pinned to uplink NIC 1, all vNIC's in port group 2 to the uplink NIC 2, vNIC's in port group 3 to uplink NIC 1 etc. In case of a uplink failure the port group to uplink mapping will recalculate and the port groups uses the remaining uplink NIC's).

Converting Nexus 2000 FEX from Single Homed to Multihomed

Greetings!
I have a 2248TP FEX and two 5548UP Nexi. Yesterday, one of the 5548s experienced an issue that caused it to go offline. The FEX is single-homed to that 5548, and so everything connected to it went offline as well.
I want to multihome the FEX to both 5548s to avoid future outages to the FEX should one 5548 go offline. However, I am aware that both 5548s must have identical configuration for the FEX. Will multihoming it cause the FEX to stop operating because 5548 #2 will come up with no configuration for the FEX? I do not have the 5548s doing any configuration sharing, they are separately configured.
I am on NXOS version 6.0(2)N2(4).
Thank you!
Henry

Hi Henry,
Before you do anything, you need to configure your 5548 switches with VPC. You need to have one or 2 connections between the 5548s and configure them as VPC peer-link. You also need to configure the out of band management port on each switch as your VPC peer keep-alive.
Once all the VPC configs are complete, you can connect your FEXs to both 5ks and create a VPC for each one.
Please refer to this document for config example:
http://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf
HTH

Nexus 2232 TM uplink ports are they only for uplink use?

We have upgraded our N5K to support the new 2232TM, but now we face the issue that the 8 SFP+ ports are not visible in our config:
Nexus5010_1# sh fex 103 det
FEX: 103 Description: 2KSRV02   state: Online
FEX version: 5.0(3)N2(2a) [Switch version: 5.0(3)N2(2a)]
FEX Interim version: 5.0(3)N2(2a)
Switch Interim version: 5.0(3)N2(2a)
Extender Model: N2K-C2232TM-10GE, Extender Serial: FOC15262P7Q
Part No: 73-13626-03
Card Id: 164, Mac Addr: 30:e4:db:66:12:02, Num Macs: 64
Module Sw Gen: 21 [Switch Sw Gen: 21]
post level: complete
pinning-mode: static    Max-links: 1
Fabric port for control traffic: Eth1/9
Fabric interface state:
    Po103 - Interface Up. State: Active
    Eth1/9 - Interface Up. State: Active
    Eth1/10 - Interface Up. State: Active
Fex Port        State Fabric Port
       Eth103/1/1 Down       Po103
       Eth103/1/2 Down       Po103
       Eth103/1/3 Down       Po103
       Eth103/1/4 Down       Po103
       Eth103/1/5 Down       Po103
       Eth103/1/6 Down       Po103
       Eth103/1/7 Down       Po103
       Eth103/1/8 Down       Po103
       Eth103/1/9 Down       Po103
      Eth103/1/10 Down       Po103
      Eth103/1/11 Down       Po103
      Eth103/1/12 Down       Po103
      Eth103/1/13 Down       Po103
      Eth103/1/14 Down       Po103
      Eth103/1/15 Down       Po103
      Eth103/1/16 Down       Po103
      Eth103/1/17 Down       Po103
      Eth103/1/18 Down       Po103
      Eth103/1/19 Down       Po103
      Eth103/1/20 Down       Po103
      Eth103/1/21 Down       Po103
      Eth103/1/22 Down       Po103
      Eth103/1/23 Down       Po103
      Eth103/1/24 Down       Po103
      Eth103/1/25 Down       Po103
      Eth103/1/26 Down       Po103
      Eth103/1/27 Down       Po103
      Eth103/1/28 Down       Po103
      Eth103/1/29 Down       Po103
      Eth103/1/30 Down       Po103
      Eth103/1/31 Down       Po103
      Eth103/1/32 Down       Po103
They are not visible: neither with the "show fex" neither with the "show int status", we cannot configure them....?
Do you think it's possible to access these ports by another way and configure them to allow some hosts to connect on it or is their purpose exclusively UPLINK?
best regards,
Eric Gillis

Do you mean the Fabric interfaces? If yes, you cannot use those for any host connectivities.
HTH,
jerry

6200 Fabric Interconnect licensing with 2232 FEX

I have searched and searched but can't find the document to answer this. I know that every port on a 6248 or 6296 needs to be licensed to work whether it's Ethernet, FC, or FCoE.
We know it's supported and we want to connect a pair of 2232PP FEX up to our 6200 FI for C-series server connectivity.
When connecting a 2232PP FEX up to a 6200 FI, there are a potential of 8 uplink ports from the FEX to the FI. Do all 8 of those ports need to be licensed to work? Do any of the new 32 ports that show up once the FEX is connected need to be licensed as well?
Thanks!
Chris

Hello Chris,
The ports on FI connecting to FEX ( Server ports ) needs to have licenses. The ports on FEX connecting to hosts ( rack servers - backplane ports ) is not counted for licenses.
You can use all of those back plane ports without worrying about licenses.
HTH
Padma

Nexus 56128P FEX Limit

how many FEXs in a dual homed setup does a 56128P support in a layer 2 configuration? Is this 48 or 24 - verified Maximum States 48??

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5600/sw/verified_scalability/702N11/b_N5600_Verified_Scalability_702N11/b_N6000_Verified_Scalability_700N11_chapter_01.html
Maximum FEXs per Switch
24
24
Maximum FEXs dual-homed to a vPC Switch Pair
24
24

IPv4 Backbone Baby Jumbo MTU size - any adverse effect?

Hello.
Is there any complication or adverse effect, setting IPv4/MPLS backbone GE link MTU size bigger than the Ethernet De facto of 1500, in the ASR9000 IOX-XR 4.3.x? What would be practical size? 4470, 8192? We have customers request to set 1800 but too odd value. What is the value the ASR9K (Trident LC with RSP, not 440) and LCs supports?
Much appreciate for your support.
Thanks,
Intopian.

That is correct Won, there are other advantages also as you noted.
For routing protocols:
OSPF uses it to fill its DBD's do both sides need to ahve the same MTU size,
same applies to ISIS.
RIP doesnt really care, although it can pack more updates in a single packet
BGP, using TCP, will use the MTU to adjust its MSS value so BGP benefits from this also big time.
As for delay on voip, yes theoretically, if the link speed is low enough, serialization delay of a large mtu packet may induce jitter on the high priority voip packet. But with 1500 bytes, the "fragment" size should be adjusted at 750k (so half T1 speed) only. With the majority circuits operating now at 1G, 10G etc, this fragment size requirement is almost no longer there as the serialization delay is so low at these speeds.
Other then very low end CPE's or very old devices that dont like larger MTU's (eg very old PA's in the 7200) there is really no worry anymore increasing the MTU size.
I dont think there is a formal doc on this, but I am sure if google it, you can find recommendations from other folks
on what they have done/seen etc.
regards
xander
Xander Thuijs CCIE #6775
Principal Engineer
ASR9000, CRS, NCS6000 & IOS-XR

Jumbo MTU on Nexus 2232 FEX

Similar Messages

Maybe you are looking for