Multiple Vlans with multiple Internet connections using PBR

Question

Hello all,
I'm trying to wrap my head around this configuration and not having a lot of success.  I have several Vlans 3,6,71,72,160, and 180.  I have two internet connections, Internet1 is connected to an ASA5510 and Internet2 is connected to a Meraki MX80.  I'm using two 4506 switches on my backbone trunked to 3750 switches that my clients connect to.  None of these switches have IP Services and my 4506 supervisor does not have an Enterprise license. However I do have one 3750 100Mbit switch with IP Services so I'm using that to do my PBR.  All my routing is currently being done on the 4506 switches and all Internet traffic is going to the ASA.  What I would like to do is force vlan160 and vlan180 through the Meraki as their Internet connection and the rest of the Vlans go through the ASA.  I'm thinking about trunking my vlans from the 4506 to the 3750 (the one with IP Services) and use policy based routing from there to force vlan160 and vlan180 to the Meraki.  But in order to do this I think I would have to move my routing onto the 3750 switch but since that is only 100Mbits I'm thinking this is going to choke my network down and defeat the purpose of the 4506 backbones.  Any suggestions or alternate ways to achieve my goal?
Appreciate any help you guys can send my way.
Matt

Answer 1

Matthew
What is the speed of the connection from the 4500 to the ASA and what is the combined speeds of the internet connections ?
You definitely don't want to do all the inter vlan routing on the 3750. You could connect it up as shown in your diagram but leave all the routing between vlans on the 4500s. Then you -
1) connect the 3750 to the 4500 using a L3 point to point link
2) connect the 3750 to the ASA using a L3 point to point link
3) do PBR on the 3750 interface connected to the 4500 for traffic coming from the 4500.
If the 4500 supervisor/IOS version doesn't support routed links on that end just use an access port in a dedicated vlan ie. no other ports in the vlan and create a new SVI for it.
You would need to update your routing to reflect the next hop on the ASA, Meraki, 3750 and the 4500.
Disadvantages are -
1) you only have fast ethernet ports on the 3750 so if the combined internet speed is greater than that then it will be a bottleneck.
2) it is a single point of failure ie. if it is lost all internet via both connections is lost.
The alternative would be to not have the 3750 in the path but connected to the 4500 via a trunk link and then route just vlan 160 and 180 on the 3750 ie. move their SVI(s) onto the 3750. Then the 3750 could have a direct connection to the Meraki device and point the default route that way (no PBR needed). The trunk would only allow those specific vlans on it.  This would mean a failure of the 3750 would not mean ASA internet lost but it would mean loss of connectivity for the two vlans routed on the 3750.
You would need to add routes to the Meraki for return traffic plus routes on the 3750 and 4500 for inter vlan routing.
The main disadvantages here are -
1) inter vlan routing between the vlans routed on the 4500s and the vlans on the 3750 will be limited by the 100Mbps connection. However you could use an etherchannel trunk so you could get greater overall throughput and some redundancy
2) more importantly though i suspect you are running HSRP between the 4500s for the client vlans and moving the SVIs onto the 3750 means a single point of failure for those vlans. 
Personally i would tend towwards option 1) because of the SVI HSRP issue and perhaps because there may be a lot of inter vlan traffic and even with an etherchannel it would be too much. 
But, single point of failure issues aside, a lot does depend on internet bandwidth in option 1) vs inter vlan traffic in option 2).
So it's a tradeoff and personally i don't think either are ideal  so i'll have another think on this in the morning to see if there is anything more obvious that i have missed or maybe someone else will add to the post.
Jon