FCoE and FC

Hi all
Perhaps the community could settle an argument that is stopping everyone here from doing any real work.
Like a lot of shops we're tossing around FCOE and trying to weigh up the pros and cons. This has moved up the agenda as we're now about to fit out a new data centre and want something that's good for 5-7 years.
The stumbling block we've hit is should we follow what seems to be the trend and go with a pair of Nexus 5000 in the top of each rack for the 'first hop' or wait for FCOE blades to come out for the 9513?
The 5000 plan gets everything in and cabled up from day one and we're good if FCOE takes off in the next 12 months. The wait-for-MDS plan is attractive to the bean counters but could leave us unable to react to server connectivity demands in the short term.
What's the opinion round the virtual water cooler?

DMX 4's. You have soo much money it doesn't matter.. :)
My major stumbling block is hundreds of Enterprise Storage ports all at 4 Gpbs. 95% of our SAN connected servers are HP C class blades and they are fairly hefty systems and relatively new.
I have seen no mention at all of CNA's for any of our current infrastructure. I also have core/edge and NPV/NPIV solutions and so far, nothing Nexus wise is worthwhile. But, give it a couple of years and that might change.
I am sure my infrastructure is similar to most organisations. Blades running virtual servers seems to be rampant in the data centre. Thats really what has bemused me with Brocade. They created their own 8 Gbps HBA's. Has anyone bought one? Does anyone use chassis servers anymore?
I just spent huge dollars replacing our aging FC switches because accountants say so. I also have huge dollars invested in native fibre channel DR solutions across data centres. What can the FCoE do for me right now?
The really big question is: Do I want to hand over my nice neat well designed SAN to our network admin?
Stephen

Similar Messages

FCoE and multiple Nexus 5000s and a 7000 core

Hi
I have a customer who is looking at four Nexus 5020s to start with and more in the future uplinked to a Nexus 7000 core.
Am I right in thinking that whilst data traffic will be able to reach hosts ocnnected to a different Nexus 5002 via the 7000 core FCoE traffic will not ?
If so what is teh recommended way of rolling out pods of 5020s for VMware servers with converged network adaptors so that they can all access the SAN ?
Regards
Pat

Don't use corecenter for fan speed control. Instead, use the BIOS:
1) Open Corecenter from the administrator account or a user account that has admin privileges. Click on the top center logo which should open the fan speed control window. There should be two items, CoolnQuiet and User/Manual or something like. Put it in manual and move the slider all the way to the right. Ignore the CoolnQuiet, it's misnamed here - MSI's corecenter does not control CoolnQuiet. What MSI is calling CoolnQuiet is just fan speed control. Anyway, put it in manual. Close Corecenter and you should never have to open it again unless you want to monitor fan RPM's.
2) Next, reboot and enter the BIOS. Go to the H/W Monitor settings. Turn on Smart CPU Fan Speed. Set to 40 C +/- 1. This will allow the motherboard to control the fan speed. On the other hand, if you want your fan always at maximum, then disable Smart Fan Speen control. Don't worry about the Smart NB Fan speed, leave it disabled.
Also, keep in mind that when you first open Corecenter, the immediate fan RPM's reported are not correct. It takes it a few seconds to get the readings. So wait until Corecenter minimizes itself to the system tray, then open it from there, and you will see the correct fan RPM's.

Nexus 7000, 2000, FCOE and Fabric Path

Hello,
I have a couple of design questions that I am hoping some of you can help me with.
I am working on a Dual DC Upgrade. It is pretty standard design, customer requires a L2 extension between the DC for Vmotion etc. Customer would like to leverage certain features of the Nexus product suite, including:
Trust Sec
VDC
VPC
High Bandwidth Scalability
Unified I/O
As always cost is a major issue and consolidation is encouraged where possible. I have worked on a couple of Nexus designs in the past and have levergaed the 7000, 5000, 2000 and 1000 in the DC.
The feedback that I am getting back from Customer seems to be mirrored in Cisco's technology roadmap. This relates specifically to the features supported in the Nexus 7000 and Nexus 5000.
Many large enterprise Customers ask the question of why they need to have the 7000 and 5000 in their topologies as many of the features they need are supported in both platforms and their environments will never scale to meet such a modular, tiered design.
I have a few specific questions that I am hoping can be answered:
The Nexus 7000 only supports the 2000 on the M series I/O Modules; can FCOE be implemented on a 2000 connected to a 7000 using the M series I/O Module?
Is the F Series I/O Module the only I/O Module that supports FCOE?
Are there any plans to introduce the native FC support on the Nexus 7000?
Are there any plans to introduce full fabric support (230 Gbps) to the M series I/O module?
Are there any plans to introduce Fabric path to the M series I/O module?
Are there any plans to introduce L3 support to the F series I/O Module?
Is the entire 2000 series allocated to a single VDC or can individual 2000 series ports be allocated to a VDC?
Is Trust Sec only support on multi hop DCI links when using the ASR on EoMPLS pwire?
Are there any plans to inroduce Trust Sec and VDC to the Nexus 5500?
Thanks,
Colm

Hello Allan
The only IO card which cannot co-exist with other cards in the same VDC is F2 due to specific hardware realisation.
All other cards can be mixed.
Regarding the Fabric versions - Fabric-2 gives much bigger throughoutput in comparing with Fabric-1
So in order to get full speed from F2/M2 modules you will need Fab-2 modules.
Fab2 modules won't give any advantages to M1/F1 modules.
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/data_sheet_c78-685394.html
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/prodcut_bulletin_c25-688075.html
HTH,
Alex

Nexus 5500 VoIP cos 3 and FCoE

Hi
I have a requirment to do QoS for a DC that includes some VoIP as well FCoE with CNAs in the servers. I want to use COS 3 for the VoIP signalling and drop it in the same input and output quues to the servers as FCoE.
so we have
class-map type qos class-fcoe
class-map type queuing class-fcoe
match qos-group 1
which is the default for FCoE and I have 50% bandwidth assoicated with it.
policy-map type queuing
     class type queuing class-fcoe
       bandwidth percent 50
     class type queing someother class etc
The issue is the "class-fcoe" is a default class and there doesnt seem to be anywhere I can edit it to add the VOIP control traffic to the same class or can I just add it here:
class-map type queuing match any class-fcoe
match qos-group 1
match cos 3
Thanks
Pat

Hmm
From the UCS 8 SRND
For instance, voice signaling traffic with L3 DSCP value of CS3 is mapped to L2 CoS value of 3 by Nexus 1000V. All Fibre Channel over Ethernet (FCoE) traffic is marked with L2 CoS value of 3 by Cisco UCS. When voice signaling and FCoE traffic enter the Cisco UCS 6100 Fabric Interconnect Switch, both will carry a CoS value of 3. In this situation voice signaling traffic will share queues and scheduling with the Fibre Channel priority class and will be given lossless behavior. (Fibre Channel priority class for CoS 3 in the UCS 6100 Fabric Interconnect Switch does not imply that the class cannot be shared with other types of traffic.)
This seems ti pmply you can run Call SIgnalling COS 3 in the same queueu as FCOE COS 3. I kow its talking about a 6100 fabric switch but thats very similar to a 5500.
Could I not do something like:
class-map type qos match-all voice-signal-global
   match-cos 3
policy-map type qos classify-global
   class voice-signal-global
      set qos-group 1
   class class-fcoe
      set qos-group 1
But then I cant see how to link this where we set the queuing bandwidth percentages as this uses the "class-fcoe"
policy-map type queuing global-fcoe-queuing-in
   class type queuing class-fcoe
      bandwidth percent 50
Could I create a class to replace "class-fcoe" and jsut match on qos-group 1 to define whats in it ?
Regards
Pat

Fcoe for ethernet AND san

I'm having a bit of a brain fart....
I see that you can create an "unified" uplink from the 6248s to an upstream Nexus 5k. Can I use this single uplink for both VSAN and VLAN traffic, or si the best practice to separare them into an Ethernet uplink and a FCOE San uplink?
Thanks in advance.

I'm not sure of any immediate disadvantages. A lot of design related inputs to separate LAN and SAN are related to pacing the integrating of the two networks.
If your already running FCoE and your not migrating from a pure LAN and Fibre channel environment to FCoE - then I'd run both on a unified link.
Please rate helpful posts.

UCS/Nexus and FCoE Question

Hi, All:
Can one connect an FCoE target directly to a Fabric Interconnect or a Nexus 5xxx FCF when in NPV mode and not connected to an FC switch for fabric services and still be able to have the initiator see the target and be able to provision storage accordingly?
So, the scenario would be the following:
a rack server with a CNA connected to a Nexus 5xxx FCF in NPV mode
OR
a blade server with a CNA connected to a UCS Fabric Interconnect (which is also an FCF) in NPV mode
AND
In either of the cases above, there is NO CONNECTION between the FCF and an FC switch.
Will I be able to have the CNA (FCoE initiator) see the Strorage array (FCoE target)?
Thank you

Ganesh, your answers make sense, and given my understanding of FCoE and the FLOGI and PLOGI processes, I want to agree with you. BUT I have had a person who I believe to be very knowledgeable in UCS (former Cisco solutions engineer) tell me that you CAN connect an FCoE storage array directly to an FI FCoE port while the FI is in NPV mode and be able to provision storage, even if the FI is not connected to an FC switch. To me, that doesnt make sense because, in such a case, there is no appliance to provide FC services to support the FIP FLOGI and PLOGI processes - and therefore, how would an E_Node (VN_Port) be able to log into the fabric, get an FC-ID and FPMA, register with the name server and engage in peer VN_Port capability discovery? How would a server CNA even know that there is a storage array that is reachable without the PLOGI process?

Can I configure FCoE between UCS Fiber Interconnect and Nexus 5k?

Hello,
I have 2 UCS with 2 pair of Fiber interconnect 6120XP and they are conencted to Nexus 5k. Currently 10Gig Ethernet and FC are used for connectvity. I have 8 port FC module in Nexus 5k. Storage is connected to Nexus 5k over FC. I would like to change the FI to Nexus 5K connectvity and configurtation with FCoE so that I can get few more free FC ports for stoarge connecting. Could you please help me whether it is possible with Nexus OS version 5 and how do we do this?
Thanks.

What storage are you using and is it capable of running a CNA?
If so, one configuration I have used to reduce ports out of the storage and reduce the need for a high density of FC ports is to run FCoE from Storage to 5k, then trad FC from 5k to FI.
With the 5k acting as a bridge between FCoE and FC you don't hit the single-hop limitations that currently exist within the protocol.
Sent from Cisco Technical Support iPhone App

Nexus 5548 UP and FCoE an brocade

Hi,
I need help.
Architecture :
I have got two Nexus 5548UP .
The nexus are linked to 2 brocade 200e.
Esxi are conrected to Nexus 2248PP, Fcoe and Data >>> interface 101/1/9
Problem:
I try configured Fcoe to Esxi 5.0 but the interface vfc 1 still down.
My configuration :
fex 101
pinning max-links 1
description "FEX101-N2232PP-S12"
fcoe
fex 102
pinning max-links 1
description "FEX102-N2232PP-Sxx"
fcoe
slot 1
port 31-32 type fc
vlan 2900
fcoe vsan 2900
vsan database
vsan 2900
feature npv
interface fc1/31
no shutdown
interface fc1/32
interface Ethernet101/1/9
descri TO-ESXI
switchport mode trunk
vfc1 is down (Error disabled)
    Bound interface is Ethernet101/1/9
    Hardware is Ethernet
    Port WWN is 20:00:54:7f:ee:b6:4b:bf
    Admin port mode is F, trunk mode is on
    snmp link state traps are enabled
    Port vsan is 2900
    1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
    1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
      0 frames input, 0 bytes
        0 discards, 0 errors
      0 frames output, 0 bytes
        0 discards, 0 errors
    last clearing of "show interface" counters never
Help me,
Cdlt,

Hi,
I had same configuration on both N5K.
I have removed the configuration Fcoe on second N5k.
Interface vfc1 seems to be Ok.
But i have another problem, On esxi the Emulex One command plug-in says ethernet's interfaces is up.
But Vcenter sees them down !!
Emulex One command :
In the distibuted Vswitch :
But LLDP seems Ok :

An Alternative to FCoE??

Has anyone heard of the IO virtualization approach?
It accomplishes the exact same thing with regard to IO consolidation, ie the reduction of cables and the consolidation of NICs and HBAs into a single entity. But it does NOT unify the fabrics or get into any complex FCoE control and data plane requirements. No worries about single or multihop FCoE, no complex FCF bridges to configure,

Hey, Robert:
I do, actually, but I was afraid of violating Cisco's NetPro Forum rules by specifying a particular vendor and their solution. But if its OK, the company I am thinking about is Xsigo. I'm only mentioning them because I would like to hear the opinions of those who know about their IO consolidation approach and who also know about FCoE and can discuss both of them.
I think it would be beneficial to have a discussion about the merits of their solution - or lack thereof - from a pure engineering perspective. No marketing, no company loyalty, etc. I do know you work for Cisco, so I will consider that, too.
It's a different approach to IO consolidation from FCoE. The Xsigo solution does not offer fabric unification. It offers IO consolidation, centralized management of IO, MR-IOV (or something along those lines) and a robust set of tools to manage and maintain a virtualized environment. It also makes provisioning and deploying of server IO resources much simpler -- or so it seems. I cant say for sure because I am still learning about it.
But in the end, the goal is not a converged fabric. However, since fabric unification - at this point in the technology's development - is limited to the server edge (ie, ToR), the two solutions are comparable, since the Xsigo approach also presents both Ethernet and FC ports at the ToR.
They have some white papers which can be found here:
http://www.xsigo.com/resource_center/resource_center/index.php

Ask the Expert: FSPF Concepts and Troubleshooting in Cisco SAN Environments

            With Upinder Sujlana
Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn and ask questions about FSPF, VSAN interaction, load balancing, and troubleshooting with Upinder Sujlana.
According to the FC-SW-2 standard, Fabric Shortest Path First (FSPF) is a link state path selection protocol. FSPF keeps track of the links on all switches in the fabric and associates a cost with each link. FSPF tracks the state of links on all switches in the fabric, associates a cost with each link in its database, and then chooses the path with a minimal cost. The cost associated with an interface can be administratively changed to implement the FSPF route selection. Upinder will discuss Cisco's implementation of FSPF.
Upinder Sujlana is a customer support engineer for Cisco's SAN TAC team based in San Jose, CA. He has worked in the TAC for the past five years with a focus on WAN technologies (L2TP, T1, T3, SCE 2K, 8K) and data center technologies such as MDS; Cisco Nexus 7000, 5000, and 2000; FCoE; and FC. Prior to joining the TAC, Upinder was a Java client-side programmer for an NMS startup company and then transitioned to network testing for a cloud company. He holds a master's degree in electrical engineering from Santa Clara University and has CCIE certification (no. 37318) in routing and switching. These days he is enthusiastic about Python programming.
Remember to use the rating system to let Upinder know if you have received an adequate response.
Upinder might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation in Data Center community, sub-community, Storage Networking discussion forum shortly after the event. This event lasts through March 14, 2014. Visit this forum often to view responses to your questions and the questions of other community members.

Hi Evan,
You can use my favorite command as below to find out the cost and check what path traffic will take. Here is a example :
switch1# show fspf internal route vsan 2
FSPF Unicast Routes
VSAN     Number          Dest Domain          Route Cost          Next hops
1                   0x01(1)                    1000                  fc1/2
1                   0xEF(239)                  1000                  fc1/1
1                   0xED(238)                  2000                  fc1/1
                                                                     fc1/2
This shows the total cost of all links.
The next hop (238) has two interfaces. This indicates that both paths will be used during load sharing. Up to sixteen paths can be used by FSPF with a Cisco MDS 9000 Family switch.
http://www.cisco.com/en/US/products/ps5989/prod_troubleshooting_guide_chapter09186a008067a306.html#wp126591
HTH,
~upinder

HP and Cisco

We have a client with a Nexus 5000 at the ToR. They want to start migrating to a blade server environment for their new cloud infrastructure.
How does an HP blade server move FCoE traffic from a server CNA to the ToR? I need a definitive answer.
I posted this question on HPs forums, but they are horrifically slow.
Thanks

ok I see.
If the goal is to leverage the CNAs you have solutions :
1 - use passthrough modules in the c7000 and send all the wires to a Nexus 2232 at the top of the rack.
2 - use HP FlexFabric modules, it's some kind of "internal" FCoE and the module splits the I/O at the chassis level, on one side in Ethernet, and the other in FC
3 - there are rumours about a Nexus 4000 for HP coming soon. But it's only rumours.
Technically the only one that currently put FCoE at the top of the rack is the first one.

Reg: UTA ports FCoE, 10GbE flow control settings

Could someone shed some light on the flow control settings for UTA ports, below is the scenario.I am working on the configutation of 4 node 3220 custer. We are using this 4 node cluster for FC and NFS traffic. each UTA port on the node used to serve both FcoE and NFS traffic( UTA port logically acting as FC and 10GbE ports) . However, the flow control is set to FULL on the 10GbE logical port to serve the NFS traffic. I think this flow control settings at 10GbE logical port level doesn't effect the FC traffic flow of the UTA port. the flow control settings would be effective on 10GE logical ports only.. Please confirm this... NetApp recommend to set the flow control to NONE and upper layer protocalls can take care of flow control( as per Best practise). If I want to change the flow control setting of 10GbE logical port, does it effect the data flow of logical FC0E port aswell?Now the actual issues is we are using the controllers mostly to serve FC trafic and we are experienceing Performance issues on all the nodes. current ONTAP version on the controller is 8.2p5we are hitting the latency of 50-80ms, I am trying to track the IO on the controllers, IO is not really high when we had the performace issues, But some times disk utilizations and cpu utlization hitting 80%. Performace manager doesn't have the granularity at aggregate level to track what LUN/volumes are casuing the high utlization at aggregate level..I am using the coomands sysstat -M, sysstat -x, netstat , statit commands mainly. However, I am not sure on any other internal porcesses are causing the high disk uitlizations...Please share your ideas on this..Oh.. another point this.. These controllers are flexpod config.....

Hi,We haven't worked on flow control settings.. you mean you are able to Disable the flow control on CNA cards?We have upgraded these filers to 8.2.3P3. at the moment filers are working fine, no performance issues.. However I wanted to disable the flow control on these ports, if possble. But NetApp docs tell that we cann't disable the flow control on the CNA cards. https://library.netapp.com/ecmdocs/ECMP1368834/html/GUID-31241255-4914-46FF-BDB5-A02DCCEABCCA.html Regards,Chandra

Maximum Distance for FCOE on Nexus 6004

Hi,
We have a fabric composed by 2 Nexus 6004. The links between the 2 Nx 6004 are configured with vE_ports.
What is the maximum length for the links between the 2 Nexus 6004 ?
Many thanks in advance.
Nicolas.

I found the following, which is most likely not up to date
Q. Is the Nexus 6004 capable of long distance FCoE and will this functionality work at FCS?
The Nexus 6004 hardware allows FCoE distances up to 100km with 10GbE interface, 30km with 40GbE interface between two Nexus 6004 devices. However at FCS, the FCoE distance will be limited to 300m. Future software release will provide CLI tuning options to support longer distance.
•300m FCoE at FCS (optical transceiver distance limitation)
10-km FCoE for 10 GE and 40 GE ports with global QoS policy (roadmap)

ISCSI vs. FCoE

Hi:
Does anyone here have an informed opinion (probably through practical experience) regarding the iSCSI vs. FCoE speed debate going on? Which method of I/O consolidation and fabric unification is "better"? Of course each technology has its advantages and disadvantages...
I know that Cisco has invested a lot in FCoE and they have gotten large technology partners, like EMC, to join their caravan.
Please dont give me an explanation of each technology and a link to some resource. I know what each technology does and how it works. I am looking for informed opinion regarding performance.
Thanks

I think that in theory, provided there is no contention, there will be little if any real world performance difference between the two. - If in both cases the hosts have enough CPU/buffers/10 gig connectivity, there's no congestion in the network etc., the SAN has similarly no resource issues processing whichever protocol.
However, I think that in practice, FCoE is the better engineered protocol, just because the perfect world of the above is unlikely, and when congestion/contention occurs, FCoE will handle it gracefully.
I'm resisting linking you to lots of white papers on the relevant standards
Simon

Using FCoE connection to non Cisco switches

Hello,
does anyone know what port configuration needs to be configured on a Nexus switch that is connected to say Brocade switch or any other vendor that supports FCoE. I have created VLAN to VSAN mapping, i assume next step is to create a vfc device ? Since this is not connection to an initiator or a target but another FCoE capable switch, how do i need to configure this vfc ? Any tips ?
Thanks

Hi,
Looks like from the document that you referenced that this switch runs in NPV mode for FCoE:
FCoE features
Fibre Channel over Ethernet (FCoE*)
FIP & FCoE packets are all forwarded when DCB is configured
* FCoE frames as defined by T11 Committee
So, I would configure the Cisco Nexus switch for FCoE and NPIV
Configure the Nexus 5000 for FCoE and NPIV
There are several procedures that are required in order to configure the Nexus 5000 for FCoE and NPIV:
    Enable Feature FCoE
    Enable N Port Identifier Virtualization (NPIV) on the Nexus 5000
    Enable Nexus 5000 Quality of Service (QoS) for FCoE
    Enable Link Layer Discovery Protocol (LLDP)
    Configure VLAN
    Configure VSAN
    Map the VLAN to VSAN
Example:
feature lldp
feature fcoe
feature npiv
interface vfc130
bind interface Ethernet1/30
switchport mode F
no shutdown
interface Ethernet1/30
switchport mode trunk
switchport trunk allowed vlan 1,100
spanning-tree port type edge trunk
vsan database
vsan 100 interface vfc130
vlan 100
fcoe vsan 100
Example of QOS:
system qos
service-policy type qos input fcoe-default-in-policy
service-policy type queuing input fcoe-default-in-policy
service-policy type queuing output fcoe-default-out-policy
service-policy type network-qos fcoe-default-nq-policy
Best regards,
Jim

FCoE and FC

Similar Messages

Maybe you are looking for