Monitoring traffic between UCS fabric extenders and fabric interconnect?

What is the best way to monitor the traffic between UCS fabric extenders (chassis) and fabric interconnect? Specifically I am looking for parameters to keep an eye on to determine when we may need to move from 2 links per fabric extender to 4 to handle greater IO needs.
Thanks.
- Klaus

One way you could monitor usage is by looking at the interface stats just as you would for any switch uplink.
Connect to the Cluster CLI
connect NXOS
Look at the input/output rate of your Server Uplink interfaces.
In my example I'm using fixed ports 1 through 4 on my FI's connecting to the IOM of the Chassis.
UCS-A(nxos)# show int eth1/1-4 | include rate
30 seconds input rate 17114328 bits/sec, 1089 packets/sec
30 seconds output rate 8777736 bits/sec, 693 packets/sec
    input rate 5.07 Mbps, 198 pps; output rate 1.03 Mbps, 99 pps
30 seconds input rate 2376 bits/sec, 0 packets/sec
30 seconds output rate 1584 bits/sec, 2 packets/sec
    input rate 1.58 Kbps, 0 pps; output rate 1.58 Kbps, 3 pps
30 seconds input rate 2376 bits/sec, 0 packets/sec
30 seconds output rate 31680 bits/sec, 20 packets/sec
    input rate 1.58 Kbps, 0 pps; output rate 30.89 Kbps, 18 pps
30 seconds input rate 2376 bits/sec, 0 packets/sec
30 seconds output rate 1584 bits/sec, 1 packets/sec
    input rate 1.58 Kbps, 0 pps; output rate 1.58 Kbps, 3 pps
If you notice your two links pushing near 10G consistently, it might be time to add another 2 link.
Other than this you can use SNMP to log the stats and look based on daily/weekly/monthly usage.
Robert

Similar Messages

Possible to segment traffic between 2 interfaces? And other questions...

I would like to set my G5 up as a server utilizing a second connection and to keep traffic seperated between this server connection and my regular internet connection (would be wireless). I'm pretty sure this alone is fairly straightforward and can be accomplished by setting up the new interface and moving it down to the bottom of the connection list with wireless at the top. That should keep all non-specific traffic from flowing out the ethernet/server connection - I think.
If the above works the way I stated then I would also want to firewall ONLY the ethernet/server connection (the wireless has it's own hardware firewall). AND - this is the tricky part - I also want to add a fake interface that has a fake IP and bind that to the "real" ethernet/server connection. The reason for that is because I need a static IP to bind the service to. I know if the connection list thing works to flow the traffic that if I had an external router on the server connection, this wouldn't be needed. I'd already have a fake IP to bind to and I wouldn't have to run the firewall on the Mac. But I don't and I'd rather not have to buy one.
So can this be done through the network/sharing preferance panes? If so, are there any "gotchas" I should be aware of? If not, is there any software tool out there that would make setting this up easier/faster? I'm not opposed to doing it all via command line, but I'm a bit rusty with my linux/unix admin knowledge. Plus I'm not 100% certain how to set all that up command line wise without screwing up OS X!
Thanks.

I'm not sure I fully understand what you are attempting to accomplish. Lets see if I have the general idea.
You have a single G5, that you want to use as both your desktop machine and also to provided specific services, such as web, email, etc.
You have some type of hardware firewall/security appliance.
You have some type of wireless access point.
You don't seem to have any type of router or switch in your configuration.
You want all of your server based traffic to be sent and received on it's own Ethernet port. You want your personal Internet traffic to be sent and received on your wireless connection.
So my questions are:
Where is the server traffic going to, coming from? Who is accessing the server, is it users on the Internet, or just computers on your own LAN (which you didn't mention).
If your server is to allow data from or send to the Internet, then you need to have a way to route the traffic there. Do you have more then one method to access the Internet, or will all traffic, both personal and server being going though the same Internet access pipe?
If it is all going through the same pipe, and you only have the single computer, I don't understand why you wish to segment the traffic.
If on the other hand you have multiple computers on your LAN. then segmenting traffic may make sense. This would allow access to your server and keep your LAN well secure.
Anyway, to get to specifics, you'll need to use the terminal app to bind specific services to specific IP's and ports on your Mac. You will also need to manually configure the firewall to be able to select specific connection ports and bindings. However, while I think it can be done, I'm not sure it makes a great deal of sense.
I would be more inclined to suggest a router or switch that can provide VLAN support, or a router that provides true DMZ support, would be a good way to go.
Anyway, a little more info would be helpful.
Oh and if I have this totally worng in what I think your doing.. My mistake.
Tom N.

What is the difference between UCS-SD-16G and UCS-SD-16G= ?

Hi,
I have some Cisco B200 m3 blades and will be installing some 16GB SD cards in them. The documentation says that I need UCS-SD-16G cards, but I received cards that say UCS-SD-16G= . Does anyone know what, if any, difference the '=' makes? Are they the same card? I have to make a large purchase of these cards and want to make sure these are the correct type before ordering. Thanks

Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
I believe the "=" suffix indicates "spare" part.

ImplementationOpt Design[Opt 31-67] Problem for AXI bus between data mover ip and axi interconnect

I'm using vivado 2014.4 and win7 64bit for my zynq design. Previously, the design is good. I made some revisions. Then I came across the problem.
If the synthesis strategies option is flow_runtime_optimatized(Vivadio synthesis 2014), everything works
If the synthesis strategies option is default options (the synthesis settings are shown in the picture), Synthesis is still fine, but the implementation failed. Some of the errror msg is shown below.
The error message shows there are unconnected pins on axi_interconnect_1. The connections between AXI master from my own IP(it is the axi port from a DataMover IP) and axi_interconnected is shown in the attached picture.
The synthesis schematic is also checked. The connections of the axi_interconnect have some pins unconnected as shown in the picture (interconnect_schematic_synth.PNG). The connections of my IP are good, but it misses some pins (like _arready, _rvalid, _ruser, _bid...). The master AXI port in my port is from the data mover IP. By default, the data mover IP doesn't have these missed pins. The AXI port on my IP declares these missed pins, but actually they are not connected to any inside my IP.
Btw, previously, the project works well in my design. But now it doesn't.
I also check the connections of the axi_interconnect when synthesis strategies option is flow_runtime_optimatized, the schematic show all pins are connected.
Please help. Thx.
Sam
ImplementationOpt Design[Opt 31-67] Problem: A LUT1 cell in the design is missing a connection on input pin I0, which is used by the LUT equation. This pin has either been left unconnected in the design or the connection was removed due to the trimming of unused logic. The LUT cell name is: design_top_level_i/Zynq_Processing_System/axi_interconnect_1/s00_couplers/auto_pc/inst/gen_axi4_axi3.axi3_conv_inst/USE_READ.USE_SPLIT_R.read_addr_inst/size_mask_q[3]_i_1__0.
[Opt 31-67] Problem: A LUT2 cell in the design is missing a connection on input pin I0, which is used by the LUT equation. This pin has either been left unconnected in the design or the connection was removed due to the trimming of unused logic. The LUT cell name is: design_top_level_i/Zynq_Processing_System/axi_interconnect_1/s00_couplers/auto_pc/inst/gen_axi4_axi3.axi3_conv_inst/USE_READ.USE_SPLIT_R.read_addr_inst/access_is_incr_q_i_1__0.
[Opt 31-67] Problem: A LUT2 cell in the design is missing a connection on input pin I1, which is used by the LUT equation. This pin has either been left unconnected in the design or the connection was removed due to the trimming of unused logic. The LUT cell name is: design_top_level_i/Zynq_Processing_System/axi_interconnect_1/s00_couplers/auto_pc/inst/gen_axi4_axi3.axi3_conv_inst/USE_READ.USE_SPLIT_R.read_addr_inst/access_is_incr_q_i_1__0.
[Opt 31-67] Problem: A LUT2 cell in the design is missing a connection on input pin I0, which is used by the LUT equation. This pin has either been left unconnected in the design or the connection was removed due to the trimming of unused logic. The LUT cell name is: design_top_level_i/Zynq_Processing_System/axi_interconnect_1/s00_couplers/auto_pc/inst/gen_axi4_axi3.axi3_conv_inst/USE_READ.USE_SPLIT_R.read_addr_inst/command_ongoing_i_2__0.
[Opt 31-67] Problem: A LUT2 cell in the design is missing a connection on input pin I0, which is used by the LUT equation. This pin has either been left unconnected in the design or the connection was removed due to the trimming of unused logic. The LUT cell name is: design_top_level_i/Zynq_Processing_System/axi_interconnect_1/s00_couplers/auto_pc/inst/gen_axi4_axi3.axi3_conv_inst/USE_READ.USE_SPLIT_R.read_addr_inst/size_mask_q[1]_i_1__0.
[Opt 31-67] Problem:

Hi Muzaffer
THx for your reply.
I tried implementation with opt_design option -directive Explore, it didn't work.
Aslo I disabled the opt_design and enabled phys_opt_design, it still has the same error in the implementation.
I would try to delete the pins related _AR_(like _arready, etc. ) and _R_ (like _rdata, _rid, etc.) pins in the AXI4 port in my IP. The data mover ip doesn't contains thes pins. I want to see whether it works this way.
Hopefully, the new vivado version will help.

Fabric Extender and Blade server connectivity(Cabling)

Hi,
There is lot of information available about how a Fabric extender connects to the Fabric interconnect. e.g. SFP+ ports, cabling etc.
I am bit a confused, how does connectivity happen between the blade server and fabric extender inside the chassis.
The information available only gives us idea about the possible network cards in the blade servers.
I want to understand, how actual connectivity is done between the blade servers and fabric extenders.
Is it through some cables or anything else? Please help.
Thanks,
Sushant

Hi Sushant
This has been discussed before
https://supportforums.cisco.com/discussion/12130401/vnic-bandwidth-question
https://supportforums.cisco.com/discussion/12207581/m81kr-vic-1240-and-vic-1280
https://supportforums.cisco.com/discussion/11897266/vic1240-and-2208xp
https://supportforums.cisco.com/discussion/11500981/vic-1280-vs-vic-1240-port-expander
https://supportforums.cisco.com/discussion/11900386/10g-between-adpater-and-blades-vic12401280
https://supportforums.cisco.com/discussion/12321671/ucs-enabledisable-dce
Cheers
Walter.

Two Fabric redundancy and storage flapping

Hello!
I have a fairly new Nexus 5548 implementation, using the Nexus for stricly storage. I have two 5548s for two different Fabrics, for redundancy. They are two seperate fabrics, and the Nexus are not stacked so they are managed individually. When I have both Nexus online my VMware side starts flapping, and loosing storage which causes my ESXi hosts to lock up and VMs to go unresponsive. This does not happen 100% of the time, but it happens intermitentaly and sometimes is catestrophic to datacenter services. When I shut down one of the Nexus switches, storage comes back and everything is healthy.
All hosts are connected via 4GB FC (supposedly HP cant do 8GB without problems)
5/6 of my hosts are on HP c7000 enclosures via 10gb FlexFabric switches, the rest via UCS
Netapp clustered pair is the target. When the ESXi hosts loose storage, they are still flogi'd in to the storage and fabric
ESXi 5.0 w/newest (and correct) drivers. VMWare tech support sees no problems, other than the "storage is getting pulled from the host"
Newest firmware on everything HP & UCS
Nexus running 5.0(3)N2(2a)
Using per-iniator zoning
Fabric-A and Fabric-B are different VSANs
Any ideas? Do I have a design flaw in my fabric? HP, Cisco, Netapp, and VMware all pretty much have no clue. So this forum is a shot in the dark. Thanks for ANY ideas you guys can provide

could it be some kind of trespassing on the storage array ? LUNs start balancing between the controllers and eventually the host times out. Have you tried different failover (NMP) policy in ESXi ?

Hyper-V Fabric and Fabric Management Cluster

Dear All...........Would appreciate understanding of following:
What is the difference between Fabric Cluster and Fabric Management Cluster?
What virtual machines should be part of Fabric Cluster and Fabric Management Cluster respectively?
Thanks in advance.

It would be helpful if you provided a source from which you pulled those terms. They could mean just about anything without any context.
But I will take a stab at it based upon work I do with the Microsoft Private Cloud Fast Track program. In this reference architecture, a private cloud is being built. To manage the private cloud, a Hyper-V cluster is created and all the System
Center components required to manage the private cloud are installed on this cluster. This cluster is called the Fabric Management Cluster because from it all components of the Fabric, servers, storage, networking, VMs, policies, libraries, etc. are
managed.
Once that Fabric Management Cluster is established, any number of Fabric Clusters can be configured. For example, the Private Cloud could be configured for a business, and multiple business units have virtual systems deployed. The Fabric Management
manager could decide to build one large Fabric Cluster on which to deploy the various business unit machines, or the manager could create separate clouds, each consisting of its own Fabric Cluster. Each separate cloud could be dedicated to the use of
a single business unit.
Again, all conjecture on my part because you have not provided any context.
. : | : . : | : . tim

- No link between IOM port 3/2/2 and fabric interconnect B:1/18

Specifically I'm trying to find out what the 3/2/2 means in - No link between IOM port 3/2/2 and fabric interconnect B:1/18

This from the fault reference guide:
"No link between IOM port [chassisId]/[slotId]/[portId] and fabric interconnect [switchId]:[peerSlotId]/[peerPortId]"
http://www.cisco.com/en/US/docs/unified_computing/ucs/ts/faults/reference/UCS_SEMs.html#wp1306388

Fabric Manager and UCS

I am having some problems getting Fabric Manager 5.0(4d) to authenticate to UCS switches (6120XP's at code level 4.2(1)). We are using TACACS+ for the authentication and the UCS switches are configured within TACACS. In Fabric Manager we get the status message of "Unknown User or Password (Server,Client). Anyone run into this?

Hello,
you will need to setup snmp user/password on UCS 6120 with the same snmp user and password as the switches
setup snmp for UCS
1.     UCS-A# scope monitoring
2.     UCS-A/monitoring# enable snmp
3.     UCS-A/monitoring# set snmp community public
4.     UCS-A/monitoring# commit-buffer
5.     UCS-A/monitoring# create snmp-user admin
6.     UCS-A/monitoring/snmp-user# set aes-128 no
7.     UCS-A/monitoring/snmp-user# set auth md5
8.     UCS-A/monitoring/snmp-user# set password (same admin password as the
seed switch)
9.     UCS-A/monitoring/snmp-user# commit-buffer
After setup snmp/password on UCS, click re-discover button on FM or better remove fabric and then re-discover fabric

Nexus 1000v and fabric extenders

Would be possible in near future to connect fabric extenders like n2k-c2232tm to nexus 1000v virtual switch?
Regards,
Vice

Hi vlacmanov,
It would depend on the current hardware setup that you have. please feel free to send me an email [email protected] so we can discuss this further. hope to hear from you soon!

FC Port Channel between UCS and MDS.

Hi All,
I am new to Cisco Fabric Concepts. In my enviornment i have a F port channel(8 Port Group) created on MDS 9513 switch and this Fport
channel is connected to Cisco UCS 6296 FI. The Cisco UCS Blade servers are connected to the Fabric Interconnect .
On MDS NPIV is enabled.
Can anyone explain the below questions.
1. Why do we create an F port Channel Group and connect it to the UCS FI? Is this something similar to Brocade Edge to AG Switch Connectivity.
2. How to configure F port Channel Group in MDS . Can anyone explain with an example.
3. Do we need to make any Configuration on UCS FI ports for server connectivity and Channel port Connectivity if yes what are the steps required to
do the same.Does the WWPN shows up in the FLOGI Database if the connectivity and configuration looks good in UCS FI and MDS.
4. What happens when a VSAN on MDS switch is added to the Port Channel.
Thanks and Regards,
Santosh surya

Look at my remarks in
https://supportforums.cisco.com/discussion/12468266/fc-port-channels-between-mds-and-ucs-fi-best-practice
1. Why do we create an F port Channel Group and connect it to the UCS FI? Is this something similar to Brocade Edge to AG Switch Connectivity.
F port channel is proprietary; therefore any such F port channel between UCS FI and Brocade doesn't work.
2. How to configure F port Channel Group in MDS . Can anyone explain with an example.
see eg.
https://supportforums.cisco.com/sites/default/files/legacy/9/9/2/53299-UCS_1-4-1_F-port_channel-trunk-v1.pdf
3. Do we need to make any Configuration on UCS FI ports for server connectivity and Channel port Connectivity if yes what are the steps required to
see eg.
https://supportforums.cisco.com/sites/default/files/legacy/9/9/2/53299-UCS_1-4-1_F-port_channel-trunk-v1.pdf
Does the WWPN shows up in the FLOGI Database if the connectivity and configuration looks good in UCS FI and MDS.
flogi database is on the MDS, not FI; there are however UCS CLI commands, like "show npv ...."
4. What happens when a VSAN on MDS switch is added to the Port Channel.
If its not created on UCS, it will just not become the status "up"

Port-Channel issue between UCS FI and MDS 9222i switch

Hi
I have a problem between UCS FI and MDS switch port-channel. When MDS-A is powered down the port-channel fails but UCS blade vHBA does not detect the failure of the port-chanel on UCS-FI and leaves the vHBA online. However, if there is no port-channel between FI-->MDS it works fine.
UCS version
System version: 2.0(2q)
FI - Cisco UCS 6248 Series Fabric Interconnect ("O2 32X10GE/Modular Universal Platform Supervisor")
Software
BIOS:      version 3.5.0
loader:    version N/A
kickstart: version 5.0(3)N2(2.02q)
system:    version 5.0(3)N2(2.02q)
power-seq: Module 1: version v1.0
             Module 3: version v2.0
uC:        version v1.2.0.1
SFP uC:    Module 1: v1.0.0.0
MDS 9222i
Software
BIOS:      version 1.0.19
loader:    version N/A
kickstart: version 5.0(8)
system:    version 5.0(8)
Here is the config from MDS switch
Interface Vsan   Admin Admin   Status          SFP    Oper Oper   Port
                  Mode   Trunk                          Mode Speed Channel
                         Mode                                 (Gbps)
fc1/1      103    auto   on      trunking         swl    TF      4    10
fc1/2      103    auto   on      trunking         swl    TF      4    10
fc1/9      103    auto   on      trunking         swl    TF      4    10
fc1/10     103    auto   on      trunking         swl    TF      4    10
This is from FI.
Interface Vsan   Admin Admin   Status          SFP    Oper Oper   Port
                  Mode   Trunk                          Mode Speed Channel
                         Mode                                 (Gbps)
fc1/29     103    NP     on      trunking         swl    TNP     4    103
fc1/30     103    NP     on      trunking         swl    TNP     4    103
fc1/31     103    NP     on      trunking         swl    TNP     4    103
fc1/32     103    NP     on      trunking         swl    TNP     4    103
Any thoughts on this?

Sultan,
This is a recently found issue and is fixed in UCSM 2.0.3a version .
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCua88227
which got duped to CSCtz21585
It happens only when following conditions are met
FI in End host mode
FC uplinks are configured for portchannel + trunking
Certain link event failures ( such abrupt power loss by upstream MDS switch )
Padma

Is FC analyser and Fabric analyser the same thing?

Folks,
Is FC analyser and fabric analyser in MDS series switches the same thing. I am confussed. Also,
Does the FC analyser has a Test based ( Etherreal) and GUI based interface? has anyone installed the GUI interface? which one is better?

The Cisco "Fabric Analyzer" that is included with the MDS is a tool for monitoring control traffic on the switch - in other words, anything that passes through the Supervisor module. It has both interfaces, or you can export the traffic to a file and look at it later with another GUI (ethereal) if you like.
You can also use a SPAN port to monitor all traffic on a specific port. Perhaps this is what you are referring to as "FC Analyzer". Typically you would use a PAA to capture the traffic using ethereal or maybe a hardware protocol analyzer. ($$$!)
So, to answer your question - No, they are not the same thing (if my assumption about your description is correct).
Josh

Configuring Nexus 5k for SAN switching between UCS 6100s and NetApp storage

We are trying to get our UCS environment setup to do SAN boot to LUNs on our NetApp storage. We have a pair of Nexus 5Ks with the Enterprise / SAN license on them and the 6-port FC module installed. What needs to be done to configure the Nexus 5000 to operate as the SAN switch between the storage target and the UCS environment?

I'm still not seeing the LUN on the NetApp device from my service profile. Here are the outputs from the two
commands you referenced here along with a few other commands if they help at all.
Port fc2/1 is the conneciton to the UCS 6100 with FCID 0x640004 being the vHBA in my Server profile.
Port fc2/5 is the NetApp target. I have the LUN masked to the vHBA port name 20:00:00:25:b5:01:00:ff
I have just the wwpn from the vHBA in my server profile and the wwpn of the NetApp target port zoned together. I'm not seeing any FC traffic at the NetApp though from looking at the statistics. Do I need to include something else in my zoning?
Again, any assistance would be appreciated. This is obviously our first venture into FC...

Nexus 7000 - Fabric Failure and VOQ

I have been doing some research on the Nexus 7k and from what i am reading the following occurs:
1. Fabric Module Failure - Causes all traffic sent across that fabric modules crossbar to be lost
2. VOQ - protects against lack of buffer availability on the egress interface
Neither of these provide reliable transmission over the crossbar or acknowledgement of data crossing the crossbar fabric.
So my question is, if i have storage traffic (unicast based FCIP) that is crossing the fabric when a fabric module fails, is my understanding correct, that those frames are lost on the portion of the fabric that is controlled by the failed fabric module?
Even though the main fabric itself is intact for other traffic, this still means that I have loss in what is supposed to be a system built for zero-loss to support storage traffic.
Am i way off here or is this accurate.
Thanks.

Thanks for the response. From what i have read the control plane and data plane are completely isolated in the nexus 7k. The supervisor modules control the control plane and the central arbiter and the fabric modules handle the VOQ and the xbar communication.
It works like this as i understand it:
1. packet arrives at the ingress of a line card and is passed on the port asic
2. port asic does its thing and forwards the packet to the replication engine
3. rep engine passes the packet onto the L2 and L3 Forwarding engines - they do their dance and pass the packet on to the fabric engine
4. Fabric Engine and VOQ mgmr consults the central arbiters to get credits to send traffic on the fabric
5. Central Arbiter checks the egress line card to ensure buffer space is available. If its available it grants credit to the fabric engine and VOQ engine to send the packet on the fabric.
The fabric crossbar is BW is determined by the amount of fabric modules installed - 1 FM = 23Gbs x 2. When 2 or more FM are installed to create more Fabric BW, they forwarding across the fabric for unicast traffic acts like a Etherchannel and performs some sort of hashing algorithm to send the packet across the fabric.
Lets say you have a 9216Byte packet and 3 Fabric modules installed. From what i am reading the packet would be broken up into 4 packets, around 2304 Bytes each (i think they might be 2460 can't recall), and passed across the fabric.
So you have 1 large packet, fragmented across the fabric cards, sent to the destination IO card.
While in transit, lets say one of the fabric Modules in the LB group dies. my understanding is the traffic on the trace goes with it.
The traffic is lost in this case since there is no acknowledgement of traffic sent across the fabric. I would think in a high bandwidth situtation this could be a lot of traffic, considering the speeds we are talking about here.
Is this a possibility or am i missing some redundancy here that will protect the traffic that would be lossed crossing the fabric?
Is this the case on the 65k as well for traffic crossing the fabric?
Thanks in advance.
Mike

Monitoring traffic between UCS fabric extenders and fabric interconnect?

Similar Messages

Maybe you are looking for