Networking Best Practices - Connecting Two Switches

Question

Connecting two switches together is an easy task, which makes it so frustrating when it doesn’t work. Here we will outline a basic scenario of connecting two switches and achieving connectivity. In these scenarios we will be using commands and settings that will work for most modern PowerConnect switches. However this does not cover all possible scenarios and the commands may differ slightly from switch to switch.
For instance, in most cases you can use General or Trunk mode when connecting two switches. However, on the PowerConnect 62xx series switches, you must use General mode if you want to allow management traffic onto the switch over the PVID.  If you use Trunk mode, you will not have the default VLAN on those ports.  The ports will only allow tagged traffic.
For more details on the difference between Access, General, and Trunk modes, follow this link.
http://en.community.dell.com/support-forums/network-switches/f/866/p/19445142/20089157.aspx#20089157
It is always a good idea to have the user and CLI guide for your switch, to reference any possible changes in command syntax.
http://support.dell.com/support/edocs/network/
Layer 2
Layer 2 switches operate at the data link layer of the OSI model. Layer 2 is responsible for error checking and transmitting data across the physical media. MAC addressing sources and destination protocols are layer 2 protocols. Layer 2 switches use the MAC address of data packets to determine where those packets should go. It learns the MAC addresses of all devices and creates a segment/forwarding table.
When a switch receives a frame with a destination address that isn't in its forwarding table, the switch forwards the frame to all other ports. If the destination machine responds to the server, the switch will listen to the reply and learn which port the destination machine is attached to. It then adds that MAC address to the forwarding table.
The Dell PowerConnect Layer 2 switches have ports that all operate in VLAN 1 by default. If it is acceptable to have all traffic on the same broadcast domain, then you can simply leave the default alone, connect the two switches and traffic will flow.
 If you do not want all traffic on the same broadcast domain, then we need to look at adding additional broadcast domains through the use of VLANs.
 We will use 3 VLANs for the following scenario.
VLAN 1=Management
VLAN 2=Client
VLAN 3=Server
To create these VLANs we do the following commands (VLAN 1 is already created by default)
console(config)# vlan database
console(config-vlan)# VLAN 2
console(config-vlan)# VLAN 3
console(config-vlan)# exit
We can then name the VLANs to help keep things organized.
console(config)# interface vlan 2
console(config-vlan)# name Client
console(config-vlan)# exit
console(config)# interface vlan 3
console(config-vlan)# name Server
console(config-vlan)# exit
Once we have the VLANs created we can place a device in that VLAN by placing the port it plugs into, in access mode for the specific VLAN.
So we have a workstation on port e2 we want to be placed in VLAN 2, we would issue the following commands.
console(config)# interface ethernet 1/e2
console(config-if)# switchport mode access
console(config-if)# switchport access vlan 2
console(config-if)# exit
The next port plugs into a server on port e3 we want on VLAN 3, we would issue these commands.
console(config)# interface ethernet 1/e3
console(config-if)# switchport mode access
console(config-if)# switchport access vlan 3
console(config-if)# exit
For the ports connecting the two switches together, we place the ports in trunk mode and specify the native VLAN and allowed VLANs.
For the port e1 that connect the two switches to each other would be configured like this.
console(config)# interface ethernet 1/e1
console(config-if)# switchport mode general
console(config-if)# switchport general allowed vlan add 2,3 tagged
console(config-if)# switchport general pvid 1
console(config-if)# exit
Once these VLANs and port settings are made on both switches. A server connected to switch A on VLAN 3 should be able to communicate with another Server connected to switch B that is also in VLAN 3.  Without the use of a router the devices in VLAN 3 will not be able to communicate with devices that are outside of their broadcast domain (i.e. VLAN 2 devices could not reach VLAN 3 devices)
Layer 3 + Layer 2
 Until recently, routers were the only devices capable of layer 3 protocols. Switches capable of routing are now available and in widespread use. In most cases we will connect our layer 2 switches to a Layer 3 capable switch to perform our routing for us.
 On the layer 3 switches we will use the same VLANs and setup that we did with the layer 2 switches.  Then we will add to the configuration.
 We can assign an IP address to each switch with the following command.
Switch A
console(config)#ip address 172.16.1.1 255.255.255.0
Switch B
console(config)#ip address 172.16.2.1 255.255.255.0
Then we will enable routing only on Switch A
console(config)# ip routing
Switch A we assign an IP address to VLAN 2 and enabling routing on the VLAN.
console(config)# interface vlan 2
console(config-if-vlan2)# Routing
console(config-if-vlan2)# ip address 172.16.20.1 255.255.255.0
console(config-if-vlan2)# exit
Switch A we assign an IP address to VLAN 3 and enabling routing on the VLAN.
console(config)# interface vlan 3
console(config-if-vlan2)# Routing
console(config-if-vlan2)# ip address 172.16.30.1 255.255.255.0
console(config-if-vlan2)# exit
On both switch A and switch B we will keep things simple and use interface 1/e1 for the connection between each switch. Setting both switches 1/e1 to general mode, allowing the additional VLAN 2,3, and keeping the PVID of 1.
console(config)# interface ethernet 1/e1
console(config-if)# switchport mode general
console(config-if)# switchport general allowed vlan add 2,3 tagged
console(config-if)# switchport general pvid 1
console(config-if)# exit
We will have one client computer connect to switch A on port 1/e2 and one client connect to switch B on port 1/e2. These ports will be in access mode for VLAN 2, and the config should look like this on both switches.
console(config)# interface ethernet 1/e2
console(config-if)# switchport mode access
console(config-if)# switchport access vlan 2
console(config-if)# exit
We will have another client computer connect to switch A on port 1/e3 and one client connect to switch B on port 1/e3. These ports will be in access mode for VLAN 3, and the config should look like this on both switches.
console(config)# interface ethernet 1/e3
console(config-if)# switchport mode access
console(config-if)# switchport access vlan 3
console(config-if)# exit
On Clients connected to Switch A we will assign an IP address and gateway based on the VLAN they are in access mode for.
Client connected to access port for VLAN 2.
IP Address:172.16.20.11
Default Gateway:172.16.20.1
Client connected to access port for VLAN 3.
IP Address:172.16.30.11
Default Gateway:172.16.30.1
On Clients connected to Switch B we will assign an IP address and gateway based on the VLAN they are in access mode for.
Client connected to access port for VLAN 2.
IP Address:172.16.20.12
Default Gateway:172.16.20.1
Client connected to access port for VLAN 3.
IP Address:172.16.30.12
Default Gateway:172.16.30.1
External Connection
At some point we may want traffic to have an external connection. To do this we can create a new VLAN for our point to point connection from Switch A to our router. We will use VLAN 7 for this and assign an IP address.
console(config)# vlan database
console(config-vlan)# VLAN 7
console(config-vlan)# exit
console(config)# interface vlan 7
console(config-vlan)# name WAN
console(config-if-vlan2)# Routing
console(config-if-vlan2)# ip address 10.10.10.2 255.255.255.0
console(config-if-vlan2)# exit
On our router we will assign an IP address of 10.10.10.1
Then place the port connecting the switch and router into access mode for VLAN 7.  In this case we use port e4.
 console(config)# interface ethernet 1/e4
console(config-if)# switchport mode access
console(config-if)# switchport access vlan 7
console(config-if)# exit
We will then need to put in a default route with the next hop as the router IP address.  This allows the switch to know where to route traffic not destined for VLANs 2, 3, or 7.
console(config)#ip route 0.0.0.0 0.0.0.0 10.10.10.1
Next on the router we’ll need to add a route back so the router knows about the networks attached to switch A.  Generally adding a static route on most routers is done with the following command: 
ip route {Network} {Wildcard Mask} {Next Hop-IP}
In our case here are the 2 static routes we could use.
Ip route 172.16.20.0 0.0.0.255 10.10.10.2
Ip route 172.16.30.0 0.0.0.255 10.10.10.2
The routing that we enabled on Switch A will enable traffic from the other VLANs to traverse over port 1/e4 to the router, connecting us to external traffic. The routes we added to the router allow the traffic to flow back to the switch over port 1/e4.
Layer 3 + Layer 3
In some situations we have two switches, each setup to route for its own broadcast domain, which we want to connect together. In this situation we no longer have a need to use Trunk or General mode between the switches. Instead we can create a common VLAN that will be used for the connection between the two switches.
To create this VLAN we will run the following commands on both switch A and B
console(config)# vlan database
console(config-vlan)# vlan 6
console(config-vlan)# exit
console(config)# interface vlan 6
console(config-vlan)# name Connection
console(config-vlan)# exit
On switch A we assign an IP address to VLAN 6 and enable routing on the VLAN.
console(config)# interface vlan 6
console(config-if-vlan2)# Routing
console(config-if-vlan2)# ip address 172.16.60.1 255.255.255.0
console(config-if-vlan2)# exit
On switch B we assign an IP address to VLAN 6 and enable routing on the VLAN.
console(config)# interface vlan 6
console(config-if-vlan2)# Routing
console(config-if-vlan2)# ip address 172.16.60.2 255.255.255.0
console(config-if-vlan2)# exit
On both switch A and B we place the connecting ports into Access mode for VLAN 6.
console(config)# interface ethernet 1/e1
console(config-if)# switchport mode access
console(config-if)# switchport access vlan 6
console(config-if)# exit
We then need to make some changes to switch B now that it is layer 3 and not layer 2 and has its own broadcast domain.
We will enable routing on Switch B
console(config)# ip routing
What used to be VLAN 2 and 3 will now be VLAN 4 and 5 for our separate broadcast domains.
Switch B we assign an IP address to VLAN 4 and enabling routing on the VLAN.
console(config)# interface vlan 4
console(config-if-vlan2)# Routing
console(config-if-vlan2)# ip address 172.16.40.1 255.255.255.0
console(config-if-vlan2)# exit
Switch B we assign an IP address to VLAN 5 and enabling routing on the VLAN.
console(config)# interface vlan 5
console(config-if-vlan2)# Routing
console(config-if-vlan2)# ip address 172.16.50.1 255.255.255.0
console(config-if-vlan2)# exit
On Clients connected to Switch B we will assign an IP address and gateway based on the VLAN they are in access mode for.
Client connected to access port for VLAN 4.
IP Address:172.16.40.11
Default Gateway:172.16.40.1
Client connected to access port for VLAN 5.
IP Address:172.16.50.11
Default Gateway:172.16.50.1
The end result should look like this.
 Troubleshooting
If we are having issues with connectivity, we may need to place some static routes in place to help traffic to the next hop in the network.
On switch A we configure a static route to help traffic to the next hop in the network, which is the router.
console(config)# ip route 0.0.0.0 0.0.0.0 10.10.10.1
The external router will also need a path defined back to all networks/VLANs.
To check the status of a port we can use the command. Show interfaces detail, this will help us see the port status. For example to check the status of port 48, we would run this command.
console# show interfaces detail ethernet 1/g48
 To check routing paths:
console# show ip route
The IP address of the network for each VLAN should be listed as C – Connected. Then also a path or default route to your upstream router.
We can use basic ping commands from a client to help test where connectivity is dropping off at. By doing this we can narrow down where in the network to start troubleshooting.
-Ping from client to default gateway, being the VLAN the client is in access mode for. If this fails then we may need to double check our client settings making sure the proper IP and gateway are being used.
-Ping from client to the ip address of the switch the client plugs into. If this fails we may not have VLAN routing enabled on the VLAN the client is in.
-Ping from client to another client on same VLAN, same switch. If this fails we need to check on client settings, IP address and gateway.
-ping from client to another client on different VLAN, same switch. If this fails we need to double check the VLAN routing commands are in place.
-ping from client to the ip address of the next switch in the network. If this fails then check Trunk port configuration from switch to switch, ensuring the VLAN is added to the Trunk port.
-ping from client to another client on same VLAN, different switch. If this fails, check Trunk port settings.
-ping from client to another client on different VLAN, different switch. If this fails then check trunk settings and VLAN routing configuration.

Answer 1

Derek,
I tried to draw my prefered setup for this network configuration.
I would create a Team with the two 1 GBit NICs and use it for Domain, DNS, Backup and any SystemCenter Agents.
I would also Team the two 10 GBit NICs and than assign it to a Hyper-V Switch for the VMs. In Windows Server 2012 it is posible to create vNICs for the Management OS that use this Hyper-V Switch (Converged Network Design). I would create two vNICs SMB1
and SMB2 to use them for Cluster and Livemigration traffic with SMB Multichannel. If your storage system supports SMB Multichannel you can also use both as storage NICs (but this depends wich vendor you have).
Hope this helps.
Grüße/Regards Carsten Rachfahl | MVP Virtual Machine | MCT | MCITP | MCSA | CCA | Husband and Papa |
www.hyper-v-server.de | First German Gold Virtualisation Kompetenz Partner ---- If my answer is helpful please mark it as answer or press the green arrow.