Split tunnel works... but only for one IP

Question

Hi All,
Dealing with a really frustrating problem. Our setup, roughly, is as follows:
- We have a remote access VPN that users connect to with any connect; in turn, they are assigned a local LAN address: 10.1.11.192-10.1.11.200
- We have a site-to-site VPN that connects to Amazon AWS to access 10.0.249.0 and other subnets, and now certain hosts on the Amazon *public* network (e.g, 54.1.2.3). This is done via a split tunnel.
What we're seeing is this:
- Users connect to the VPN and are assigned one of the addresses above. Let's use 10.1.11.192 for this example.
- They can then access anything in the 10.0.249.0 subnet (via the split tunnel) just fine. This goes across both ASA devices.
- They can then access anything in the public Amazon network (via the split tunnel) just fine. This should only use the remote access ASA.
So, it seemed like everything was working. When connected to the VPN, Amazon hosts in both 10.x.x.x networks and public IPs that I had specifically tunneled (we plan to transition these to a VPC soon) were accessible, and access happened via the remote access VPN IP (i.e, when connecting to 54.1.2.3, it showed the user being logged in from the Cisco's gateway's IP address, as opposed to the local client IP).
Now, here's where things got weird: *public* tunneled hosts at Amazon only works with the first address in the pool, 10.1.11.192. No other addresses work. 10.0.249.x is always available, regardless of assigned IP. 54.x.y.z is only available with .192.
I've used the same computer with different assigned IPs (10.1.11.193-10.1.11.200), and none work. I've connected using different computers.. they work if assigned .192, but not any other addresses. Other users report the same issue.
TCP handshaking is failing
I'll use our IRC server (and sometimes ssh server) for testing. I can see my client laptop with a SYN_SENT on that specific connection. I can see the IRC server with a SYN_RECV, and the ASA shows a SYN timeout after 30 seconds. Thus, it looks as though packets from the IRC server can't make their way back through the ASA to my client laptop.
I suspect this has something to do with dynamic vs. static NAT, etc., but I've fiddled with every setting I can, and coming up blank.
I'm also baffled as to why .192 works, but no other addresses do.
I've attached our configuration, minus keys and passwords and certain IPs/hostnames. It's a little bit ugly because it has some poor attempts at fixing this, things I'll probably remove after it works, but.. Could it have anything to do with TCP sequence randomization?
Thank you in advance for any help.

Answer 1

Hi Jouni! Thank you so much for your quick reply. Mine was delayed because I've continued to fight the ASA this afternoon! Apologies for my verbosity, always found it better to include too much info than too little!
To clarify, ideally, a VPN client that connects to this ASA (10.1.11.5) should have three possible destinations:
- The internet at large - this should not go through the ASA, but should go through the client's own gateway, bypassing this VPN entirely. This works.
- My servers in an Amazon VPC (10.0.249.x and some others) - this should come in to this ASA (10.1.11.5), which in turn is already successfully routed to another ASA (10.1.11.4 in the config). This works, too.
- My servers at Amazon AWS that are on the public internet (example below: 107.22.xxx.yyy). This should only involve the one ASA in question (10.1.11.5). This is where I'm running into issues.
The split tunnel, thus, includes the networks 10.0.249.x and specific public AWS hosts like 107.22.xxx.yyy, etc. I want 10.0.249.x to go to Amazon via the other ASA 10.1.11.4 (again, this works) and I want 107.22.xxx.yyy, etc. to go to Amazon via *this* ASA (10.1.11.5). Basically, it's this last part that's causing problems: it works only one in very, very narrow situation: when the client is assigned the address 10.1.11.192. If a user logs in and is assigned any other address, they won't be able to access the "public" AWS servers.
I began by taking your advice and moving the IP assignment to an entirely different subnet: 10.1.12.100-10.1.12.200. I then spent a decent chunk of the afternoon adjusting NAT rules (and removing ACLs, per your suggestion). The only brief success I had was when I had about a bajillion NAT rules, and somehow I made it on to the IRC server! Oddly, my connection used the ASA's address itself, rather than the ASA gateway's address. Unfortunately, I stopped being able to reach 10.0.249.x at the time :/ I tried to adjust further, but cannot restore/recall what the NAT rules were.
I'm miffed because 10.1.11.192 works, and *nothing* else does. There is nothing special about this address in the config - I've tried everything I can to figure out why this one address is different. I've also looked in places that *shouldn't* matter because this bypasses the VPC entirely (AWS security groups, AWS iptables rules, VPC options, etc.)
Again, the behavior of 10.1.11.193-200 is a partial TCP handshake: the local computer sees SYN_SENT, and the remote server has SYN_RECV, but no connection is ultimately established. This is why I think it could be a NAT issue, but I'm obviously a bit lost here Here's a sample connection attempt, with DNS traffic removed:
2013-10-17 18:29:09.100 [DEBUG] Message from Host: 10.1.11.5 - Message - %ASA-6-302013: Built inbound TCP connection 43606 for outside:10.1.11.193/56626 (10.1.11.193/56626)(LOCAL\justinsTestMac) to outside:107.22.xxx.yyy/6667 (107.22.xxx.yyy/6667) (justinsTestMac) (pid:25912)2013-10-17 18:29:39.129 [DEBUG] Message from Host: 10.1.11.5 - Message - %ASA-6-302014: Teardown TCP connection 43606 for outside:10.1.11.193/56626(LOCAL\justinsTestMac) to outside:107.22.xxx.yyy/6667 duration 0:00:30 bytes 0 SYN Timeout (justinsTestMac) (pid:25912)2013-10-17 18:29:45.293 [DEBUG] Message from Host: 10.1.11.5 - Message - %ASA-6-302013: Built inbound TCP connection 43612 for outside:10.1.11.193/56626 (10.1.11.193/56626)(LOCAL\justinsTestMac) to outside:107.22.xxx.yyy/6667 (107.22.xxx.yyy/6667) (justinsTestMac) (pid:25912)102 (justinsTestMac) (pid:25912)2013-10-17 18:30:15.322 [DEBUG] Message from Host: 10.1.11.5 - Message - %ASA-6-302014: Teardown TCP connection 43612 for outside:10.1.11.193/56626(LOCAL\justinsTestMac) to outside:107.22.xxx.yyy/6667 duration 0:00:30 bytes 0 SYN Timeout (justinsTestMac) (pid:25912)2013-10-17 18:30:17.976 [DEBUG] Message from Host: 10.1.11.5 - Message - %ASA-6-302013: Built inbound TCP connection 43617 for outside:10.1.11.193/56626 (10.1.11.193/56626)(LOCAL\justinsTestMac) to outside:107.22.xxx.yyy/6667 (107.22.xxx.yyy/6667) (justinsTestMac) (pid:25912)2013-10-17 18:30:48.400 [DEBUG] Message from Host: 10.1.11.5 - Message - %ASA-6-302014: Teardown TCP connection 43617 for outside:10.1.11.193/56626(LOCAL\justinsTestMac) to outside:107.22.xxx.yyy/6667 duration 0:00:30 bytes 0 SYN Timeout (justinsTestMac) (pid:25912)
Thanks for any insight you have!