How do I troubleshoot a site to site vpn connection?

Question

I have a site to site vpn connection setup to a client site that functions fine except for 2 ip addresses on the client are not responding.
They insist the problem is at our end but I don't know how to troubleshoot it. The access rules are there for both the problematic IP addresses exactly the same as all the others which work fine.
The connection is over port 21.
Can anyone point me in the right direction to find the fault?
Thanks
Graham

Answer 1

Thanks for the ongoing help Abhishek. Here are the results you requested:
Result of the command: "packet-tracer input inside tcp 78.129.151.9 21 172.16.157.164 21 detailed"
Phase: 1
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   0.0.0.0         0.0.0.0         Outside
Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group allow in interface Inside
access-list allow extended permit ip any any
Additional Information:
Forward Flow based lookup yields rule:
in  id=0xab8da840, priority=12, domain=permit, deny=false
hits=23185385, user_data=0xa8acab80, cs_id=0x0, flags=0x0, protocol=0
src ip=0.0.0.0, mask=0.0.0.0, port=0
dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Phase: 3
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in  id=0xab87da98, priority=0, domain=inspect-ip-options, deny=true
hits=26999122, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip=0.0.0.0, mask=0.0.0.0, port=0
dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Phase: 4
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0xac19fc90, priority=70, domain=encrypt, deny=false
hits=5, user_data=0x7c0e604, cs_id=0xab83dc88, reverse, flags=0x0, protocol=0
src ip=78.129.151.9, mask=255.255.255.255, port=0
dst ip=172.16.157.164, mask=255.255.255.255, port=0, dscp=0x0
Phase: 5
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group allow out interface Outside
access-list allow extended permit ip any any
Additional Information:
Forward Flow based lookup yields rule:
out id=0xab8da4b8, priority=12, domain=permit, deny=false
hits=23344429, user_data=0xa8acac00, cs_id=0x0, flags=0x0, protocol=0
src ip=0.0.0.0, mask=0.0.0.0, port=0
dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Phase: 6
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in  id=0xac19fbb0, priority=69, domain=ipsec-tunnel-flow, deny=false
hits=5, user_data=0x7c1089c, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip=172.16.157.164, mask=255.255.255.255, port=0
dst ip=78.129.151.9, mask=255.255.255.255, port=0, dscp=0x0
Phase: 7
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in  id=0xab831ea0, priority=0, domain=inspect-ip-options, deny=true
hits=23303214, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip=0.0.0.0, mask=0.0.0.0, port=0
dst ip=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
Phase: 8
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 27682039, packet dispatched to next module
Module information for forward flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_adjacency
snp_fp_encrypt
snp_fp_fragment
snp_ifc_stat
Module information for reverse flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_ipsec_tunnel_flow
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat
Result:
input-interface: Inside
input-status: up
input-line-status: up
output-interface: Outside
output-status: up
output-line-status: up
Action: allow
Result of the command: "sh crypto isa sa"
1   IKE Peer: 94.128.3.13
    Type    : L2L             Role    : initiator
    Rekey   : no              State   : MM_ACTIVE
Result of the command: "sh crypto ipsec sa peer 94.128.3.130"
peer address: 94.128.3.130
    Crypto map tag: VPNPEER, seq num: 40, local addr: 87.117.213.66
      access-list HK extended permit ip host 78.129.151.9 host 172.16.158.72
      local ident (addr/mask/prot/port): (78.129.151.9/255.255.255.255/0/0)
      remote ident (addr/mask/prot/port): (172.16.158.72/255.255.255.255/0/0)
      current_peer: 94.128.3.130
      #pkts encaps: 134528, #pkts encrypt: 134528, #pkts digest: 134528
      #pkts decaps: 134434, #pkts decrypt: 134434, #pkts verify: 134434
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 134528, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #send errors: 0, #recv errors: 0
      local crypto endpt.: 87.117.213.66, remote crypto endpt.: 94.128.3.130
      path mtu 1500, ipsec overhead 58, media mtu 1500
      current outbound spi: DC42A606
      current inbound spi : 8D93AAC5
    inbound esp sas:
      spi: 0x8D93AAC5 (2375264965)
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, }
         slot: 0, conn_id: 20058112, crypto-map: VPNPEER
         sa timing: remaining key lifetime (kB/sec): (1564526/2674)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0xDC42A606 (3695355398)
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, }
         slot: 0, conn_id: 20058112, crypto-map: VPNPEER
         sa timing: remaining key lifetime (kB/sec): (1565777/2674)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001
    Crypto map tag: VPNPEER, seq num: 40, local addr: 87.117.213.66
      access-list HK extended permit ip host 78.129.151.9 host 172.16.158.11
      local ident (addr/mask/prot/port): (78.129.151.9/255.255.255.255/0/0)
      remote ident (addr/mask/prot/port): (172.16.158.11/255.255.255.255/0/0)
      current_peer: 94.128.3.130
      #pkts encaps: 48577, #pkts encrypt: 48577, #pkts digest: 48577
      #pkts decaps: 55798, #pkts decrypt: 55798, #pkts verify: 55798
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 48577, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #send errors: 0, #recv errors: 0
      local crypto endpt.: 87.117.213.66, remote crypto endpt.: 94.128.3.130
      path mtu 1500, ipsec overhead 58, media mtu 1500
      current outbound spi: F6D08308
      current inbound spi : 8DF816EB
    inbound esp sas:
      spi: 0x8DF816EB (2381846251)
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, }
         slot: 0, conn_id: 20058112, crypto-map: VPNPEER
         sa timing: remaining key lifetime (kB/sec): (1565999/2672)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x000001FF
    outbound esp sas:
      spi: 0xF6D08308 (4140860168)
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, }
         slot: 0, conn_id: 20058112, crypto-map: VPNPEER
         sa timing: remaining key lifetime (kB/sec): (1565999/2672)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001
    Crypto map tag: VPNPEER, seq num: 40, local addr: 87.117.213.66
      access-list HK extended permit ip host 78.129.151.9 host 172.16.157.164
      local ident (addr/mask/prot/port): (78.129.151.9/255.255.255.255/0/0)
      remote ident (addr/mask/prot/port): (172.16.157.164/255.255.255.255/0/0)
      current_peer: 94.128.3.130
      #pkts encaps: 33383, #pkts encrypt: 33383, #pkts digest: 33383
      #pkts decaps: 62020, #pkts decrypt: 62020, #pkts verify: 62020
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 33383, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #send errors: 0, #recv errors: 0
      local crypto endpt.: 87.117.213.66, remote crypto endpt.: 94.128.3.130
      path mtu 1500, ipsec overhead 58, media mtu 1500
      current outbound spi: 4F1C0C7B
      current inbound spi : E7AD7104
    inbound esp sas:
      spi: 0xE7AD7104 (3886903556)
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, }
         slot: 0, conn_id: 20058112, crypto-map: VPNPEER
         sa timing: remaining key lifetime (kB/sec): (1554196/1167)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0x4F1C0C7B (1327238267)
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, }
         slot: 0, conn_id: 20058112, crypto-map: VPNPEER
         sa timing: remaining key lifetime (kB/sec): (1565786/1167)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001
    Crypto map tag: VPNPEER, seq num: 40, local addr: 87.117.213.66
      access-list HK extended permit ip host 78.129.151.9 host 172.16.158.71
      local ident (addr/mask/prot/port): (78.129.151.9/255.255.255.255/0/0)
      remote ident (addr/mask/prot/port): (172.16.158.71/255.255.255.255/0/0)
      current_peer: 94.128.3.130
      #pkts encaps: 37285, #pkts encrypt: 37285, #pkts digest: 37285
      #pkts decaps: 65179, #pkts decrypt: 65179, #pkts verify: 65179
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 37285, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #send errors: 0, #recv errors: 0
      local crypto endpt.: 87.117.213.66, remote crypto endpt.: 94.128.3.130
      path mtu 1500, ipsec overhead 58, media mtu 1500
      current outbound spi: A4FF525F
      current inbound spi : 25C04F5C
    inbound esp sas:
      spi: 0x25C04F5C (633360220)
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, }
         slot: 0, conn_id: 20058112, crypto-map: VPNPEER
         sa timing: remaining key lifetime (kB/sec): (1490798/1167)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0xA4FF525F (2768196191)
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, }
         slot: 0, conn_id: 20058112, crypto-map: VPNPEER
         sa timing: remaining key lifetime (kB/sec): (1564601/1167)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001
    Crypto map tag: VPNPEER, seq num: 40, local addr: 87.117.213.66
      access-list HK extended permit ip host 78.129.151.9 host 172.16.158.73
      local ident (addr/mask/prot/port): (78.129.151.9/255.255.255.255/0/0)
      remote ident (addr/mask/prot/port): (172.16.158.73/255.255.255.255/0/0)
      current_peer: 94.128.3.130
      #pkts encaps: 2173002, #pkts encrypt: 2173002, #pkts digest: 2173002
      #pkts decaps: 4166416, #pkts decrypt: 4166416, #pkts verify: 4166416
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 2173002, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #send errors: 0, #recv errors: 0
      local crypto endpt.: 87.117.213.66, remote crypto endpt.: 94.128.3.130
      path mtu 1500, ipsec overhead 58, media mtu 1500
      current outbound spi: B5DAC5BA
      current inbound spi : 8B01255A
    inbound esp sas:
      spi: 0x8B01255A (2332108122)
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, }
         slot: 0, conn_id: 20058112, crypto-map: VPNPEER
         sa timing: remaining key lifetime (kB/sec): (934511/1915)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0xB5DAC5BA (3051013562)
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, }
         slot: 0, conn_id: 20058112, crypto-map: VPNPEER
         sa timing: remaining key lifetime (kB/sec): (1556432/1915)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001
    Crypto map tag: VPNPEER, seq num: 40, local addr: 87.117.213.66
      access-list HK extended permit ip host 78.129.151.9 host 172.17.167.10
      local ident (addr/mask/prot/port): (78.129.151.9/255.255.255.255/0/0)
      remote ident (addr/mask/prot/port): (172.17.167.10/255.255.255.255/0/0)
      current_peer: 94.128.3.130
      #pkts encaps: 289506, #pkts encrypt: 289506, #pkts digest: 289506
      #pkts decaps: 424145, #pkts decrypt: 424145, #pkts verify: 424145
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 289506, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #send errors: 0, #recv errors: 0
      local crypto endpt.: 87.117.213.66, remote crypto endpt.: 94.128.3.130
      path mtu 1500, ipsec overhead 58, media mtu 1500
      current outbound spi: 66956330
      current inbound spi : 55173067
    inbound esp sas:
      spi: 0x55173067 (1427583079)
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, }
         slot: 0, conn_id: 20058112, crypto-map: VPNPEER
         sa timing: remaining key lifetime (kB/sec): (1552267/2682)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0x66956330 (1721066288)
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, }
         slot: 0, conn_id: 20058112, crypto-map: VPNPEER
         sa timing: remaining key lifetime (kB/sec): (1565576/2682)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001