ACE VIP OK HTTP, NOK other TCP port

Question

Hi,
we are having issues in configuring load balancing for a TCP port. For HTTP it's working without issues and we have the ACE also balancing for other TCP ports.
Here goes the relevant config:
probe http PROBE-HTTP
  interval 5
  passdetect interval 2
  passdetect count 1
  request method get url /idc/
  expect status 200 200
probe tcp PROBE-TCP
  port 4444
  interval 5
  passdetect interval 10
rserver host PRD1
  ip address 10.10.10.1
  inservice
rserver host PRD2
  ip address 10.10.10.2
  inservice
serverfarm host SF-HTTP
  probe PROBE-HTTP
  rserver PRD1 80
    inservice
  rserver PRD2 80
    inservice
serverfarm host SF-TCP
  probe PROBE-TCP
  rserver PRD1 4444
    inservice
  rserver PRD2 4444
    inservice
sticky ip-netmask 255.255.255.255 address source SC-IP-PRD-HTTP
  timeout 10
  serverfarm SF-HTTP
class-map match-all NAT-VIP-HTTP
  2 match virtual-address 10.10.35.1 any
class-map match-all NAT-VIP-TCP
  2 match virtual-address 10.10.35.1 tcp eq 4444
policy-map type loadbalance first-match LB-VIP-HTTP
  class class-default
    sticky-serverfarm SC-IP-PRD-HTTP
    insert-http x-forward header-value "%is"
policy-map type loadbalance first-match LB-NAT-VIP-TCP
  class class-default
    serverfarm SF-TCP
policy-map multi-match POLICY-RSERVER-VIP
  class NAT-VIP-TCP
    loadbalance vip inservice
    loadbalance policy LB-NAT-VIP-TCP
    loadbalance vip icmp-reply active
    nat dynamic 1 vlan 200
  class NAT-VIP-HTTP
    loadbalance vip inservice
    loadbalance policy LB-VIP-HTTP
    loadbalance vip icmp-reply active
    nat dynamic 1 vlan 200
interface vlan 200
  description SERVER-SIDE
  ip address 10.10.14.2 255.255.255.0
  alias 10.10.14.1 255.255.255.0
  peer ip address 10.10.14.3 255.255.255.0
  access-group input EVERYONE
  nat-pool 1 10.10.4.6 10.10.4.6 netmask 255.255.255.255 pat
  service-policy input AllowICMP
  service-policy input POLICY-RSERVER-VIP
  no shutdown
The probe are OK, but nothing seems to get to the VIP:
ACE/CTX# show probe PROBE-TCP
probe       : PROBE-TCP
type        : TCP
state       : ACTIVE
   port      : 4444    address     : 0.0.0.0         addr type  : -
   interval  : 5       pass intvl  : 10              pass count : 3
   fail count: 3       recv timeout: 10
                       --------------------- probe results --------------------
   probe association   probed-address  probes     failed     passed     health
   ------------------- ---------------+----------+----------+----------+-------
   serverfarm  : SF-TCP
     real      : PRD1[4444]
                       10.10.10.1     8853       1          8852       SUCCESS
     real      : PRD2[4444]
                       10.10.10.2     8853       1          8852       SUCCESS
ACE/CTX# show serverfarm SF-TCP detail
serverfarm     : SF-TCP, type: HOST
total rservers : 2
active rservers: 2
description    : -
state          : ACTIVE
predictor      : ROUNDROBIN
failaction     : -
back-inservice    : 0
partial-threshold : 0
num times failover       : 0
num times back inservice : 1
total conn-dropcount : 0
Probe(s) :
    PROBE-TCP,  type = TCP
                                                ----------connections-----------
       real                  weight state        current    total      failures
   ---+---------------------+------+------------+----------+----------+---------
   rserver: PRD1
       10.10.10.1:4444      8      OPERATIONAL  0          0          0
         max-conns            : -         , out-of-rotation count : -
         min-conns            : -
         conn-rate-limit      : -         , out-of-rotation count : -
         bandwidth-rate-limit : -         , out-of-rotation count : -
         retcode out-of-rotation count : -
         load value           : 0
   rserver: PRD2
       10.10.10.2:4444      8      OPERATIONAL  0          0          0
         max-conns            : -         , out-of-rotation count : -
         min-conns            : -
         conn-rate-limit      : -         , out-of-rotation count : -
         bandwidth-rate-limit : -         , out-of-rotation count : -
         retcode out-of-rotation count : -
         load value           : 0
ACE/CTX# show service-policy POLICY-RSERVER-VIP
Status     : ACTIVE
Interface: vlan 1 200
  service-policy: POLICY-RSERVER-VIP
    class: NAT-VIP-TCP
      nat:
        nat dynamic 1 vlan 200
        curr conns       : 0         , hit count        : 0
        dropped conns    : 0
        client pkt count : 0         , client byte count: 0
        server pkt count : 0         , server byte count: 0
        conn-rate-limit      : 0         , drop-count : 0
        bandwidth-rate-limit : 0         , drop-count : 0
      loadbalance:
        L7 loadbalance policy: LB-NAT-VIP-TCP
        VIP ICMP Reply       : ENABLED-WHEN-ACTIVE
        VIP State: INSERVICE
        curr conns       : 0         , hit count        : 0
        dropped conns    : 0
        client pkt count : 0         , client byte count: 0
        server pkt count : 0         , server byte count: 0
        conn-rate-limit      : 0         , drop-count : 0
        bandwidth-rate-limit : 0         , drop-count : 0
      compression:
        bytes_in  : 0
        bytes_out : 0
I see a lot of this messages in the logging of the ACE:
show logging | i 4444
22:02:52 : %ACE-6-302023: Teardown TCP connection 0x18b6 for vlan200:10.10.14.2/26768 to vlan200:10.10.10.2/4444 duration 0:00:00 bytes 1051 TCP FINs
22:02:55 : %ACE-6-302022: Built TCP connection 0x14dc for vlan200:10.10.14.2/30318 (10.10.10.1/30318) to vlan200:10.10.10.1/4444 (10.10.14.2/4444)
22:02:55 : %ACE-6-302023: Teardown TCP connection 0x14dc for vlan200:10.10.14.2/30318 to vlan200:10.10.10.1/4444 duration 0:00:00 bytes 1103 TCP FINs
22:02:57 : %ACE-6-302022: Built TCP connection 0xc6c for vlan200:10.10.14.2/26784 (10.10.10.2/26784) to vlan200:10.10.10.2/4444 (10.10.14.2/4444)
22:02:57 : %ACE-6-302023: Teardown TCP connection 0xc6c for vlan200:10.10.14.2/26784 to vlan200:10.10.10.2/4444 duration 0:00:00 bytes 1103 TCP FINs
22:03:02 : %ACE-6-302022: Built TCP connection 0x151a for vlan200:10.10.14.2/26800 (10.10.10.2/26800) to vlan200:10.10.10.2/4444 (10.10.14.2/4444)
show logging | i 4444
22:02:52 : %ACE-6-302023: Teardown TCP connection 0x18b6 for vlan200:10.10.14.2/26768 to vlan200:10.10.10.2/4444 duration 0:00:00 bytes 1051 TCP FINs
22:02:55 : %ACE-6-302022: Built TCP connection 0x14dc for vlan200:10.10.14.2/30318 (10.10.10.1/30318) to vlan200:10.10.10.1/4444 (10.10.14.2/4444)
22:02:55 : %ACE-6-302023: Teardown TCP connection 0x14dc for vlan200:10.10.14.2/30318 to vlan200:10.10.10.1/4444 duration 0:00:00 bytes 1103 TCP FINs
22:02:57 : %ACE-6-302022: Built TCP connection 0xc6c for vlan200:10.10.14.2/26784 (10.10.10.2/26784) to vlan200:10.10.10.2/4444 (10.10.14.2/4444)
22:02:57 : %ACE-6-302023: Teardown TCP connection 0xc6c for vlan200:10.10.14.2/26784 to vlan200:10.10.10.2/4444 duration 0:00:00 bytes 1103 TCP FINs
22:03:02 : %ACE-6-302022: Built TCP connection 0x151a for vlan200:10.10.14.2/26800 (10.10.10.2/26800) to vlan200:10.10.10.2/4444 (10.10.14.2/4444)
The client request it's going trough an ASA, in the ASA side I see that the TCP connection it' half-open with SAaB flags. It seems that the VIP never replies with SYN+ACK to the ASA...
Thank you.
Best regards

Answer 1

Hi Norberto,
The log messages you are getting are most probably the probe connections and not a failure, looking to them you will see your ACE is establishing TCP connection on 4444 then it will teardown the connection with FIN which is expected since you are using TCP keepalives.
I would recommend to go back and define the problem exactly, what are you exteriancing when you try to telnet on port 4444 toward the VIP from the client?
Run sniffing software on the client, the server and enable capture on ACE and ASA will give you exact idea what you are experiencing.
Note: The ASA and the ACE has great capture feature which will show you exactly the packet flows.
Note: Since you are applying NAT on the client requests, you should see the NATed IP address on the server capture.
Note: With L4 load balancing the ACE is not spoofing the clients' request, it just forward the SYN, SYN+ACK and ACK between the server and the client.
Let me know if you have any other questions.
Best regards,
Ahmad