DMVPN with 270 spokes

Hi
We have a DMVPN network with about 270 spokes (831) and 2 hubs (3825). We run EIGRP in all branch of the network.
We had experience many problems so far, but one of them seems to be related with the amount of traffic that is generated when some of our hub went down for about 15 to 30 seconds. All of our spokes start blinking on EIGRP but our VPN still up. Here is some of our messages that we have on Hub side (peer goodbye receive, stuck in active, hold timer expire). Our EIGRP timer are 5/15 but we tried 60/180, but we had the same issue.
We have a fiber 5Mb on one hub and the other hub have 10Mb fiber link.
Does anyone heard someting about that ?
Thank your very much for your help.

This problem may happen because of various reasons like one of the neighbors reseting the other due to routes stuck in active, some router reporting a k-value mismatch, etc. Following links may help you
http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a00800949ab.shtml
http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080093feb.shtml

Similar Messages

DMVPN Hub and Spoke behind NAT device

Hi All,
I have seen many documents stating about DMVPN Hub behind NAT or DMVPN Spoke behind NAT.
But My case i involve in both situation.
1) HUB have a Load Balancer (2 WAN Link) ISP A & B
2) Spoke have Load Balancer (2 WAN Link) ISP A & B
Now the requirement is Spoke ISP A Tunnel to HUB ISP A. Spoke ISP B tunnel to HUB ISP B
So total of two DMVPN tunnel from spoke to hub, and i will use EIGRP and PBR to select path.
As I know at HUB site, LB must do Static NAT for HUB router IP, so spoke will point to it as tunnel destination address. At spoke LB, i will do policy route to reach HUB ISP A IP via Spoke ISP A link, HUB ISP B IP via Spoke ISP B link.
HUB and Spoke have to create 2 tunnel with two different network ID but using same source interface.
The Tunnel destination IP at spoke router is not directly belongs to HUB router. Its hold by HUB LB , and forwarded to HUB router by Static NAT.
Any problem will face with this setup? Any guide?
Sample config at HUB.
interface Tunnel0
bandwidth 1000
ip address 172.16.1.1 255.255.255.0
ip mtu 1440
ip nhrp authentication cisco123
ip nhrp map multicast dynamic
ip nhrp network-id 1
ip nhrp holdtime 600
delay 1000
tunnel source FastEthernet0/0
tunnel mode gre multipoint
tunnel key 0
tunnel protection ipsec profile cisco
interface Tunnel1
bandwidth 1000
ip address 172.17.1.1 255.255.255.0
ip mtu 1440
ip nhrp authentication cisco123
ip nhrp map multicast dynamic
ip nhrp network-id 2
ip nhrp holdtime 600
delay 1000
tunnel source FastEthernet0/0
tunnel mode gre multipoint
tunnel key 1
tunnel protection ipsec profile cisco
Spoke Config
interface Tunnel0
bandwidth 1000
ip address 172.16.1.2 255.255.255.0
ip mtu 1440
ip nhrp authentication cisco123
ip nhrp map 172.16.1.1 199.1.1.1
ip nhrp network-id 1
ip nhrp holdtime 300
ip nhrp nhs 172.16.1.1
delay 1000
tunnel source FastEthernet0/0
tunnel destination 199.1.1.1
tunnel key 0
tunnel protection ipsec profile cisco
interface Tunnel1
bandwidth 1000
ip address 172.17.1.2 255.255.255.0
ip mtu 1440
ip nhrp authentication cisco123
ip nhrp map 172.17.1.1 200.1.1.1
ip nhrp network-id 2
ip nhrp holdtime 300
ip nhrp nhs 172.17.1.1
delay 1500
tunnel source FastEthernet0/0
tunnel destination 200.1.1.1
tunnel key 1
tunnel protection ipsec profile cisco

Hi Marcin,
thanks for your reply. The NAT was set up in a way it was/is just to simulate the spoke to be behind NAT device.
About AH and ESP, you are correct there... this was actually my issue. I should have used pure ESP. At the end, TAC actually assisted me with this. Before I called TAC, i did notice the following. ISAKMP traffic was NATed to 3.3.3.3, as expected. Anything after that, did not work and it has to with NAT and AH. Traffic was no longer NATed so the hub, saw the traffic come from 2.2.2.2 rather than 3.3.3.3, you can also see that in the error message you have pointed out. I also saw it in my packet captures. That caught my eye and i started troubleshooting it. I did not understand that AH can't be NATed, Below is TAC's explanation. All is good now. Thanks
. Essentially, it comes down to the fact that AH will encapsulate the entire IP packet (hence why it is the outermost header) with the exception of a few mutable fields, including the DSCP/ToS, ECN, flags, fragment offset, TTL, and the header checksum. Since the source/destination IP addresses & port numbers are actually protected by the AH integrity checking, this means that a device performing a NAT operation on the packet will alter these IP header fields and effectively cause the hub router to drop the packet due to AH failure.
Conversely, ESP traffic is able to properly traverse NAT because it doesn't include the IP header addresses & ports in its integrity check. In addition, ESP doesn't need to be the outermost header of the packet in order to work, which is why devices will attach an outer UDP/4500 header on the traffic going over NAT."

HUB with 3 spokes redesign with IPsec

Hello
Our Current network represent HUB with 3 spokes using frame relay as a wan connection that's indeed costly, I suggested to buy a DSL link with 512 k on each site and Connect these 3 branches to H.O using IPSEC,I need help to redesign the network from a 2 z according the attached diagram
Thanks

Ali,
This design/what you are trying to do is quite normal, I have a few questions:-
1) What speed is the connection from the HQ to the internet, it would ideally be 1.5mbs or higher?
2) Would internet access from the remote sites be required to pass thru the HQ?
3) Are running VoIP - you need to define a QoS policy
4) Are the remote sites going to have static Internet IP's or Dynamic?
5) Will the HQ have a static IP internet address? This is a must.
6) Are you currently running any dynamic routing protocols? if so you need to have GRE tunnels.
HTH>

Dual cloud dual hub single tier dmvpn with backup service provider

Hi,
I have a design issue with a WAN network. I have decided to use dual cloud dual hub single tier DMVPN topology (ref. to http://www.cisco.com/application/pdf/en/us/guest/netsol/ns171/c649/ccmigration_09186a008075ea98.pdf - "Dynamic Multipoint VPN (DMVPN) Design Guide"). I have tested in lab 2 hubs and 3 spokes, applying the mentioned technology. Everything is OK, when the primary hub fails, there is only 1-3 seconds loss (3 pings).
The problem is that each spoke and hub will have 2 service providers for WAN - primary and backup. I am still wondering which design is better and more stable to implement - using more DMVPN clouds (for the backup service provider network) or creating static IPSEC GRE tunnels in the backup links?
Is there a guide for this case?
What is the best practice in this case?
Thanks in advance,
Mladen

Dynamic spoke-to-spoke requires your spoke routers to have mGRE tunnel interfaces. If you ever have a spoke which sources 2 tunnels from the same physical interface, you have a problem: how to resolve which tunnel is an incoming NHRP request for?
My DMVPN is a bit different in that the crypto is GETVPN on the physical interface. There is a crypto-map applied to the physical interface and it has 2 entries which correspond to the GETVPN crypto-groups for each tunnel.
I resolved this issue by making one of the 2 tunnels on each spoke router mGRE and the 2nd one point to point. the mGRE tunnel is preferred as primary (we use eBGP through the tunnel, so routes received through the mGRE tunnel are local-pref'd high and we AS path prepend routes advertised out the point-to-point tunnel)
I haven't gone back and tested what happens when you have a spoke which has 2 tunnels sourced from the same interface and another spoke with 2 tunnels sourced from the same interface or from 2 different physical interfaces. The concern is that you may get a situation where one router uses Tunnel 2 for dynamic spoke-to-spoke tunneling, and the other uses Tunnel1, and that the dynamic tunnel setup fails because the crypto map cannot properly decide which crypto group to use for the incoming traffic on the router where 2 tunnels use the same physical interface.

DMVPN security per spoke

Hello,
I currently have a DMVPN configuration with an isakmp key that is used for all spokes. Is there a way to create a key per spoke or another method of configuring security per spoke so if someone leaves the company I can remove config from the HUB and they can no longer connect?
Thank you,
Scott

If your spokes have dynamic IP addresses, you are out of luck with PSKs. In these scenarios, using certificates is the way to go. If your spokes have fixed IPs, you can configure the PSKs individually, but you lose spoke-to-spoke communication.

DMVPN with SLB

Hello,
I was just wondering if any improvment has been done to DMVPN in Server Load Balancing configuration (DMVPN HUBs are behind an IOS SLB and all the spokes pointing to the single virtual IP). Apparently this is the most scalable solution going beyond 1000 spokes. The only problem with it is it did not support the spoke-spoke DMVPN. Mike Sullenberger's presentation at the mini Networkers in Toronto in 2007 stated that this missing piece is in development by Cisco.
Thanks

This feature is still not available but possibly is in work and will be available shortly in some future release. As of now there is no support for spoke to spoke DMVPN.

Dual-DMVPN with Dual Hubs

Are there any routing issues when using mGRE interfaces on spokes. I need spoke-to-spoke connectivity. Obviously if I opt for p-pGRE interfaces then traffic from spoke-to-spoke will have to go via one of the hubs.
I understand there was a limitation in IOS whereby mGRE interfaces on spokes prevented it from learning many routes via the hub.
tia
Ajaz

Since no one has answered yet, I'll give you the practical answer.
You'll have issues with IPSec and static routing. "DMVPN" itself probably wouldn't have an issue, but it would depend on IPSec and routing to work.
It is easier, by far, to put in a second router. And when you factor in your time to try to make it work (and it may not work), the second router is less expensive.
Rob

DMVPN with IPSec

Hi
I set the IPSec on the DMVPN on HUb and Spokes as follow bellow, But I get the Error on All DMVPN end points.
I am using CSR1000v on a rack rentals site.
R18(config-if)#
*Jan 2 02:18:24.458: %ACE-3-TRANSERR: IOSXE-ESP(11): IKEA trans 0x63; opcode 0x60; param 0x2F; error 0x5; retry cnt 0
*Jan 2 02:18:24.459: %ACE-3-TRANSERR: IOSXE-ESP(11): IKEA trans 0x65; opcode 0x60; param 0x30; error 0x5; retry cnt 0
R18(config-if)#
EIGRP does not come up. But removing The IPsec profile from the Tunn1l 100 interfaces brings the EIGRP up and DMVPN works fine.
Any suggestions ?
The configs are:
R18
crypto isakmp policy 18
encr aes 192
hash sha256
authentication pre-share
group 5
crypto isakmp key DmvPn!23 address 89.211.116.16
crypto isakmp key DmvPn!23 address 89.211.117.17
crypto ipsec transform-set TRANS_SET esp-aes esp-sha-hmac
mode transport
crypto ipsec profile CRY_PROFILE
set transform-set TRANS_SET
int tu 100
tunnel protection ipsec profile CRY_PROFILE
R16
crypto isakmp policy 16
encr aes 192
hash sha256
authentication pre-share
group 5
crypto isakmp key DmvPn!23 address 202.4.180.0
crypto ipsec transform-set TRANS_SET esp-aes esp-sha-hmac
mode transport
crypto ipsec profile CRY_PROFILE
set transform-set TRANS_SET
int tu 100
tunnel protection ipsec profile CRY_PROFILE
R17
crypto isakmp policy 17
encr aes 192
hash sha256
authentication pre-share
group 5
crypto isakmp key DmvPn!23 address 202.4.180.0
crypto ipsec transform-set TRANS_SET esp-aes esp-sha-hmac
mode transport
crypto ipsec profile CRY_PROFIL
crypto ipsec profile CRY_PROFILE
set transform-set TRANS_SET
int tu 100
tunnel protection ipsec profile CRY_PROFILE
R18
interface Tunnel100
ip address 172.100.123.18 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication NHRPKEY
ip nhrp map multicast dynamic
ip nhrp network-id 123
ip nhrp holdtime 300
ip tcp adjust-mss 1360
tunnel source 202.4.180.0
tunnel mode gre multipoint
tunnel key 123
tunnel protection ipsec profile CRY_PROFILE
R16
interface Tunnel100
ip address 172.100.123.16 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication NHRPKEY
ip nhrp map 172.100.123.18 202.4.180.0
ip nhrp map multicast 202.4.180.0
ip nhrp nhs 172.100.123.18
ip nhrp network-id 123
ip nhrp holdtime 300
ip tcp adjust-mss 1360
tunnel source 89.211.116.16
tunnel mode gre multipoint
tunnel key 123
tunnel protection ipsec profile CRY_PROFIL
R17
interface Tunnel100
ip address 172.100.123.17 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication NHRPKEY
ip nhrp map 172.100.123.18 202.4.180.0
ip nhrp map multicast 202.4.180.0
ip nhrp nhs 172.100.123.18
ip nhrp network-id 123
ip nhrp holdtime 300
ip tcp adjust-mss 1360
tunnel source 89.211.117.17
tunnel mode gre multipoint
tunnel key 123
tunnel protection ipsec profile CRY_PROFIL

Thanks guys for your inputs. I had it solved. The issue is with CSR 1000v platform is that it somehow does not like AH-SHA-HMAC so I changed to using ESP-SHA-HMAC instead.
Thanks & Regards,
Saleh

Running DMVPN Hub and Spoke on same router?

My client has a project in which traffic flow is hierarchial in nature. Using DMVPNs, the design is for a "center" router to be a DMVPN spoke to the cloud above it, and a DMVPN Hub to the cloud below it. I have tried to lab this up, but no success. I initially build the center router as a DMVPN spoke to teh upper cloud and all is well. As soon as I had the second tunnel config (as the DMVPN hub to the lower cloud), the first tunnel goes down and my EIGRP flaps. Im running EIGRP across the DMVPN tunnels. The two DMVPN clouds are using different network IDs and are running separate EIGRP routing instances.
I can post configs if desired - just wanted to see if anyone is doing this or knows whether it is possible.
Jeff

Hi,
I know it is possible using two DMVPN clouds, but it seems that you need DMVPN phase 3 in this situation. This is suitable for the hierarchical model you want. Take a look at the following link
http://www.cisco.com/en/US/partner/prod/collateral/iosswrel/ps6537/ps6586/ps6660/ps6808/prod_white_paper0900aecd8055c34e_ps6658_Products_White_Paper.html
Hope this helps.

DMVPN With EIGRP - MTU adaptation

Hi all,
I am managing a Hub and Spoke network with more than 1000 Spokes. The initial MTU of the tunnel interface we've configured is 1400 (on both hubs and Spokes).
I am facing now some issue with customers that have a WAN line with a lower MTU: ping [tunnel_IP] size 1400 doesn't give any reply.
The problem is that:Two routers with MTU of 1400, but in the path between them there is apparently a hop with a smaller MTU. Hello and neighbor-ship is working (I assume that Hello are smaller than 1400)
Route from Spoke are received on Hub (displayed in sh ip route ei) but the route from Hubs are not received by the Spoke. I assume that's due to the fact that there are not a lot of route advertised by the spoke (so packet size lower) and more advertised by the hub reaching MTU size....
I did some tests with these customer Spokes connecting them to another hub router used for R&D, adapting the MTU on both size to a lower value: working !
I have now 2 solutions
- Ask all my concerned customers to contact their ISP to check if they can increase MTU or find another gateway for my router... Most of my customer doesn't have any IT people and won't understand at all what I want...
- Modify MTU on my router: modify in on the Spoke... but do I need to reconfigure it on the hub to avoid neighbor flapping... So I need to reconfigure all my 1000+ Spokes
So my question is: is there a way to adapt MTU size of the tunnel interface depending on the destination.
I tried to find a solution like a route map that will modify MTU when matching an ACL that contains my impacted Spokes but it's not possible to adapt MTU with a set in a route-map...
Any idea on your side ? workaround ?
Best regards !

Great! I got this working. Not sure what the issue was. The thing is, I was using GNS3 and I had a router with a switch card acting as both SP clouds. When I tried configuring the route tracking, it wasn't behaving. I since switched it from the router-switch to a router connected to 2 of the dumb switches and route tracking started behaving properly. The failover and everything works perfectly now.
I also cleaned up some of the tunnel config so I don't know if that's actually what fixed it.
One more question though. I just discovered that the customer has no data license so there's no clean automatic failover using floating static routes. Can I get this clean failover using VRFs without route tracking or will I still need it?

Dmvpn with Redundancy

I am setting up a backup dmvpn.
What is the best approach to this design whiile using eigrp
one tunnel with load balance or two tunnels while changing the metric on the SPOKES

Hi,
You can refer the below mentioned document to find a best fit solution for your redundant model.
http://www.cisco.com/c/en/us/support/docs/security-vpn/ipsec-negotiation-ike-protocols/41940-dmvpn.html
Regards
Karthik

IPv6 WAN Adresses on DMVPN Hubs and Spokes

Hi,
I have one question about IPv6 and DMVPN. Let's assume we have a DMVPN Hub Site running IPv6 IP addresses on the WAN Interfaces and Spoke Sites running either IPv4 or IPv6 IP adresses on their WAN Interfaces.
In IPv4 only Networks, the Tunnel Konfiguration looked like that:
interface Tunnel 1
<...>
ip nhrp map multicast <IPv4 NMBA>
ip nhrp map 10.0.10.1 <IPv4 NMBA>
There are IPv6 nhrp commands as well, but they can only map an IPv6 Address to an IPv4 NMBA:
interface Tunnel1
<...>
ipv6 nhrp map multicast <IPv4 NMBA>
ipv6 nhrp map <IPv6 Tunnel Adress> <IPv4 NMBA>
I'm wondering how you would configure DMVPN for a network using IPv6 on their WAN Interfaces ??
Best regards,
Thomas

Thomas,
VTI1(config-if)#ip nhrp map 1.2.3.4 ? A.B.C.D IP NBMA address A.B.C.D IP mask of destination X:X:X:X::X IPv6 NBMA address
and
VTI1(config-if)#ipv6 nhrp map 2001:db8::1/128 ? A.B.C.D IPv4 NBMA address X:X:X:X::X IPv6 NBMA address
I _seem_to_remember_ this was added in 15.2T but can't find it back.
M.

DMVPN: Too many spoke to spoke tunnels created!

Hi all,
We have a customer wiht a DMVPN network phase 2. There are around 40 spokes. What I can see on all the spokes is between 10 and 15 IPsec spoke to spoke tunnels which is really strange as most of the spoke locations should only have between 1 to 4 spoke to spoke tunnels maximum.
My question is what could cause the spoke routers to create an IPsec spoke to spoke tunnel? Some weired applications?
IOS version: 12.4(15)T11 Platform: Cisco 1812.
Best regards,
Laurent

Hi,
But there may be an apllication that trigger this Spoke to spoke tunnels?
I cannot in the inspect sessions the IP of the destination spokes. So it looks it is a quick trigger, then inspect session time ud og IPsec SAs stayded for the defaut life time.
Regards,
Laurent

DMVPN with mgre and IKEv2

Hello,
Wondering if there is a good document with some examples on DMVPN using IKE v2.
Thanks!

Jeffrey,
(it's around 1AM for me some please forgive typos / ambiguities).
With mGRE you have one logical interfaces to apply features to.
I.e. everything that was applied to the tunnel interface applies to all the sessions.
A typical problem is QoS; Per-tunnel QoS solves SOME of those problems, but we can do so much more if we use VAs.
If you forgive a shameless plug I can demonstrate with a document I wrote for a specific use case:
http://www.cisco.com/en/US/products/ps12922/products_tech_note09186a0080bf9d4e.shtml
In this document I demonstrate how to apply different features based on particular IKE identity using local attributes.
mGRE/DMVPN is still a valid choice, if you want to have A policy applied to most of your sessions, having multiple VAs, allows you to have flexability to apply policies as needed.
As a note DMVPN does mix with IKEv2. It's Flex/Multiple VAs which are adding the benefit.
M.

IPSec for Redundant DMVPN with VRF

Hi.
I have been labbing up a solution using DMVPN and VRF, similar to that described in the blog post here. It works very well, however when I try to extend the concept to a redundant hub, it breaks with IPSec. If I remove the tunnel protection, it works fine.
Does anyone have any ideas about providing IPSec protection to multiple DMVPN tunnels for VRFs to a redundant Hub?
Thanks.
Client config (no IPSec):
interface Tunnel10
ip vrf forwarding Staff
ip address 10.254.254.23 255.255.255.0
no ip redirects
ip mtu 1416
ip nhrp authentication MFS
ip nhrp map multicast 172.16.1.1
ip nhrp map 10.254.254.1 172.16.1.1
ip nhrp map 10.254.254.3 172.16.1.3
ip nhrp map multicast 172.16.1.3
ip nhrp network-id 10
ip nhrp holdtime 600
ip nhrp nhs 10.254.254.1
ip nhrp nhs 10.254.254.3
ip tcp adjust-mss 1360
tunnel source FastEthernet0/0
tunnel mode gre multipoint
tunnel key 10
interface Tunnel20
ip vrf forwarding Clients
ip address 10.254.253.23 255.255.255.0
no ip redirects
ip mtu 1416
ip nhrp authentication MFSC
ip nhrp map 10.254.253.1 172.16.1.1
ip nhrp map multicast 172.16.1.1
ip nhrp map multicast 172.16.1.3
ip nhrp map 10.254.253.3 172.16.1.3
ip nhrp network-id 20
ip nhrp holdtime 600
ip nhrp nhs 10.254.253.1
ip nhrp nhs 10.254.253.3
ip tcp adjust-mss 1360
tunnel source FastEthernet0/0
tunnel mode gre multipoint
tunnel key 20
Hub 1:
interface Tunnel10
ip vrf forwarding Staff
ip address 10.254.254.1 255.255.255.0
no ip redirects
ip mtu 1416
ip nhrp authentication MFS
ip nhrp map multicast dynamic
ip nhrp network-id 10
ip nhrp holdtime 360
ip tcp adjust-mss 1360
tunnel source FastEthernet0/0
tunnel mode gre multipoint
tunnel key 10
interface Tunnel20
ip vrf forwarding Clients
ip address 10.254.253.1 255.255.255.0
no ip redirects
ip mtu 1416
ip nhrp authentication MFSC
ip nhrp map multicast dynamic
ip nhrp network-id 20
ip nhrp holdtime 360
ip tcp adjust-mss 1360
tunnel source FastEthernet0/0
tunnel mode gre multipoint
tunnel key 20
Hub 2:
interface Tunnel10
ip vrf forwarding Staff
ip address 10.254.254.3 255.255.255.0
no ip redirects
ip mtu 1416
ip nhrp authentication MFS
ip nhrp map multicast dynamic
ip nhrp network-id 10
ip nhrp holdtime 360
ip nhrp server-only
ip tcp adjust-mss 1360
tunnel source FastEthernet0/0
tunnel mode gre multipoint
tunnel key 10
interface Tunnel20
ip vrf forwarding Clients
ip address 10.254.253.3 255.255.255.0
no ip redirects
ip mtu 1416
ip nhrp authentication MFSC
ip nhrp map multicast dynamic
ip nhrp network-id 20
ip nhrp holdtime 360
ip tcp adjust-mss 1360
tunnel source FastEthernet0/0 tunnel mode gre multipoint
tunnel key 20

Under the Hub you have to add
HUB1
interface Tunnel10
ip nhrp map 10.254.254.1
ip nhrp map multicast < ip add of FastEthernet0/0 for HUB2>
HUB2
interface Tunnel10
ip nhrp map 10.254.254.3
ip nhrp map multicast < ip add of FastEthernet0/0 for HUB1>
The same thing for the other tunnel interfaces

DMVPN with 270 spokes

Similar Messages

Maybe you are looking for