L2L VPN Issue - one subnet not reachable

Question

Hi Folks,
I have a strange issue with a new VPN connection and would appreciate any help.
I have a pair of Cisco asa 5540s configured as a failover pair (code version 8.2(5)).   
I have recently added 2 new L2L VPNs - both these VPNs are sourced from the same interface on my ASA (called isp), and both are to the same customer, but they terminate on different firewalls on the cusomter end, and encrypt traffic from different customer subnets.    There's a basic network diagram attached.
VPN 1 - is for traffic from the customer subnet 10.2.1.0/24.    Devices in this subnet should be able to access 2 subnets on my network - DMZ 211 (192.168.211.0./24) and DMZ 144 (192.168.144.0/24).    This VPN works correctly.
VPN 2 - is for traffic from the customer subnet 192.168.1.0/24.    Devices in  this subnet should be able to access the same 2 subnets on my network - DMZ 211  (192.168.211.0./24) and DMZ 144 (192.168.144.0/24).    This VPN is not working correctly - the customer can access DMZ 144, but not DMZ 211.
There are isakmp and ipsec SAs for both VPNs.    I've noticed that the packets encaps/decaps counter does not increment when the customer sends test traffic to DMZ 211.  This counter does increment when they send test traffic to DMZ144.   I can also see traffic sent to DMZ 144 from the customer subnet 192.168.1.0/24 in packet captures on the DMZ 144 interface of the ASA.   I cannot see similar traffic in captures on the DMZ211 interface (although I can see traffic sent to DMZ211 if it is sourced from 10.2.1.0/24 - ie when it uses VPN1)
Nat exemption is configured for both 192.168.1.0/24 and 10.2.1.0/24.
There is a route to both customer subnets via the same next hop.
There is nothing in the logs toindicate that traffic from 192.168.1.0/24 is being dropped
I suspect that this may be an issue on the customer end, but I'd like to be able to prove that.   Specifically, I would really like to be able to capture traffic destined to DMZ 211 on the isp interface of the firewall after it has been decrypted - I don't know if this can be done however, and I haven'treally found a good way to prove or disprove that VPN traffic from 192.168.1.0/24 to DMZ211 is arriving at the isp interface of my ASA, and to show what's happening to that traffic after it arrives.
Here is the relevant vpn configuration:
crypto map MY_CRYPTO_MAP 90 match address VPN_2
crypto map MY_CRYPTO_MAP 90 set peer 217.154.147.221
crypto map MY_CRYPTO_MAP 90 set transform-set 3dessha
crypto map MY_CRYPTO_MAP 90 set security-association lifetime seconds 86400
crypto map MY_CRYPTO_MAP 100 match address VPN_1
crypto map MY_CRYPTO_MAP 100 set peer 193.108.169.48
crypto map MY_CRYPTO_MAP 100 set transform-set 3dessha
crypto map MY_CRYPTO_MAP 100 set security-association lifetime seconds 86400
crypto map MY_CRYPTO_MAP interface isp
ASA# sh access-list VPN_2
access-list VPN_2; 6 elements; name hash: 0xa902d2f4
access-list VPN_2 line 1 extended permit ip object-group VPN_2_NETS 192.168.1.0 255.255.255.0 0x56c7fb8f
  access-list VPN_2 line 1 extended permit ip 192.168.144.0 255.255.255.0 192.168.1.0 255.255.255.0 (hitcnt=45) 0x93b6dc21
  access-list VPN_2 line 1 extended permit ip 192.168.211.0 255.255.255.0 192.168.1.0 255.255.255.0 (hitcnt=6) 0x0abf7bb9
  access-list VPN_2 line 1 extended permit ip host 192.168.146.29 192.168.1.0 255.255.255.0 (hitcnt=8) 0xcc48a56e
ASA# sh access-list VPN_1
access-list VPN_1; 3 elements; name hash: 0x30168cce
access-list VPN_1 line 1 extended permit ip 192.168.144.0 255.255.252.0 10.2.1.0 255.255.255.0 (hitcnt=6) 0x61759554
access-list VPN_1 line 2 extended permit ip 192.168.211.0 255.255.255.0 10.2.1.0 255.255.255.0 (hitcnt=3) 0xa602c97c
access-list VPN_1 line 3 extended permit ip host 192.168.146.29 10.2.1.0 255.255.255.0 (hitcnt=0) 0x7b9f32e3
nat (dmz144) 0 access-list nonatdmz144
nat (dmz211) 0 access-list nonatdmz211
ASA# sh access-list nonatdmz144
access-list nonatdmz144; 5 elements; name hash: 0xbf28538e
access-list nonatdmz144 line 1 extended permit ip 192.168.144.0 255.255.255.0 192.168.0.0 255.255.0.0 (hitcnt=0) 0x20121683
access-list nonatdmz144 line 2 extended permit ip 192.168.144.0 255.255.255.0 172.28.2.0 255.255.254.0 (hitcnt=0) 0xbc8ab4f1
access-list nonatdmz144 line 3 extended permit ip 192.168.144.0 255.255.255.0 194.97.141.160 255.255.255.224 (hitcnt=0) 0xce869e1e
access-list nonatdmz144 line 4 extended permit ip 192.168.144.0 255.255.255.0 172.30.0.0 255.255.240.0 (hitcnt=0) 0xd3ec5035
access-list nonatdmz144 line 5 extended permit ip 192.168.144.0 255.255.255.0 10.2.1.0 255.255.255.0 (hitcnt=0) 0x4c9cc781
ASA# sh access-list nonatdmz211 | in 192.168\.1\.
access-list nonatdmz1 line 3 extended permit ip 192.168.211.0 255.255.255.0 192.168.1.0 255.255.255.0 (hitcnt=0) 0x2bbfcfdd
ASA# sh access-list nonatdmz211 | in 10.2.1.
access-list nonatdmz1 line 4 extended permit ip 192.168.211.0 255.255.255.0 10.2.1.0 255.255.255.0 (hitcnt=0) 0x8a836d91
route isp 192.168.1.0 255.255.255.0 137.191.234.33 1
route isp 10.2.1.0 255.255.255.0 137.191.234.33 1
Thanks in advance to anyone who gets this far!

Answer 1

Darragh
Clearing the counters was a good idea. If the counter is not incrementing and if ping from the remote side is not causing the VPN to come up it certainly confirms that something is not working right.
It might be interesting to wait till the SAs time out and go inactive and then test again with the ping from the remote subnet that is not working. Turn on debug for ISAKMP and see if there is any attempt to negotiate. Especially if you do not receive any attempt to initiate ISAKMP from then then that would be one way to show that there is a problem on the remote side.
Certainly the ASA does have the ability to do packet capture. I have used that capability and it can be quite helpful. I have not tried to do a capture on the outside interface for incoming VPN traffic and so am not sure whether you would be capturing the encrypted packet or the de-encrypted packet. You can configure an access list to identify traffic to capture and I guess that you could write an access list that included both the peer addresses as source and destination to capture the encrypted traffic and entries that were the un-encrypted source and destination subnets to capture traffic after de-encryption.
HTH
Rick