ACE Rst Packets

Hello Everyone,
I have ACE10 Module in my switch core 6509, my context "Proxy" was criated for balance connections to Forefront TMG Servers, this balance needs original client IP Address connections end to end in the solution.
My problem is: The clients are complaining of slowness connection to the internet, i captured the traffic in the ace capture feature and i see some RST packets and severals checksum error packets in pcap file.
The topology is:
Client -> ACE VIP VLAN 81 -> RSERVERS VLAN 80
Vlan 80 is in L2 mode(no interface vlan in the switch core 6509, route occurs through the ace appliance).
The IP address 10.96.200.6 is the gw for rservers.
system:    Version A2(3.4) [build 3.0(0)A2(3.4)]
system image file: [LCP] disk0:c6ace-t1k9-mz.A2_3_4.bin
rserver host PANFPRXP301A
ip address 10.96.200.11
inservice
rserver host PANFPRXP301B
ip address 10.96.200.12
inservice
sticky ip-netmask 255.255.255.255 address source STICKY-SF-PANPROXY
replicate sticky
serverfarm SF-PAN-PROXY
interface vlan 80
ip address 10.96.200.4 255.255.255.0
alias 10.96.200.6 255.255.255.0
peer ip address 10.96.200.5 255.255.255.0
no normalization
no icmp-guard
access-group input all-access
access-group output all-access
service-policy input ACCESS
no shutdown
interface vlan 81
ip address 10.96.201.4 255.255.255.0
alias 10.96.201.6 255.255.255.0
peer ip address 10.96.201.5 255.255.255.0
no normalization
no icmp-guard
access-group input all-access
access-group output all-access
service-policy input ACCESS
service-policy input INTVLAN80
no shutdown
policy-map multi-match INTVLAN80
class VIP-SF-PANPROXY
    loadbalance vip inservice
    loadbalance policy SLB-SF-PANPROXY
    loadbalance vip icmp-reply active primary-inservice
    appl-parameter http advanced-options PARAMETER-HTTP
Logs
====================================================================
Aug 15 2012 10:24:09 : %ACE-6-302023: Teardown TCP connection 0xb9fec for vlan81
:10.93.15.69/1439 (10.93.15.69/1439) to vlan80:10.96.201.10/8080 (10.96.200.12/8
080) duration 0:01:28 bytes 13741 TCP FINs
Aug 15 2012 10:24:09 : %ACE-6-302022: Built TCP connection 0x1121b8 for vlan81:1
0.93.15.69/1443 (10.93.15.69/1443) to vlan80:10.96.201.10/8080 (10.96.200.12/808
0)
Aug 15 2012 10:24:10 : %ACE-6-302022: Built TCP connection 0xc400b for vlan81:10
.93.7.69/4863 (10.93.7.69/4863) to vlan80:10.96.201.10/8080 (10.96.200.11/8080)
Aug 15 2012 10:24:10 : %ACE-6-302022: Built TCP connection 0xc676f for vlan81:10
.93.15.29/2173 (10.93.15.29/2173) to vlan80:10.96.201.10/8080 (10.96.200.12/8080
Aug 15 2012 10:24:10 : %ACE-6-302022: Built TCP connection 0xc3621 for vlan81:10
.93.7.84/54169 (10.93.7.84/54169) to vlan80:10.96.201.10/8080 (10.96.200.11/8080
Aug 15 2012 10:24:10 : %ACE-6-302025: Teardown UDP connection 0x110764 for vlan8
0:10.96.200.11/32230 (10.96.200.11/32230) to vlan81:172.17.2.35/53 (172.17.2.35/
53) duration 0:00:11 bytes 126 Idle Timeout
Aug 15 2012 10:24:10 : %ACE-6-302023: Teardown TCP connection 0x111c70 for vlan8
1:10.93.15.69/1441 (10.93.15.69/1441) to vlan80:10.96.201.10/8080 (10.96.200.12/
8080) duration 0:00:02 bytes 1759 TCP FINs
Aug 15 2012 10:24:10 : %ACE-6-302022: Built TCP connection 0x5fc51 for vlan81:10
.93.7.69/4864 (10.93.7.69/4864) to vlan80:10.96.201.10/8080 (10.96.200.11/8080)
Aug 15 2012 10:24:11 : %ACE-6-302022: Built TCP connection 0xc5282 for vlan81:10
.93.5.157/1522 (10.93.5.157/1522) to vlan80:10.96.201.10/8080 (10.96.200.11/8080
Aug 15 2012 10:24:11 : %ACE-6-302022: Built TCP connection 0x10e7a2 for vlan81:1
0.93.15.29/2174 (10.93.15.29/2174) to vlan80:10.96.201.10/8080 (10.96.200.12/808
0)
Aug 15 2012 10:24:11 : %ACE-6-302023: Teardown TCP connection 0x102c48 for vlan8
1:10.84.34.23/1130 (10.84.34.23/1130) to vlan80:10.96.201.10/8080 (10.96.200.12/
====================================================================
If needed, i can send the pcap file for analyse.
Tks a Lot.
Rafael

Hi Rafael,
Are RST's coming from ACE? What if you access the server directly? If you could raise a TAC case we would do in-depth analysis of the problem.
Regards,
Siva

Similar Messages

Does ACE send a RST packet when it reach inactivity timeout?

Hi experts
I have some questions about ace's behavier.
1st one is, Does ACE send a RST packet when it reach to inactivity timeout?
2nd, Does half-closed timeout works properly with "no normalization"?
3rd, How does ACE treat the packets there is no flows in conn table? Drop or forwarding?
Thanks

Hi Kilsoo,
1st one is, Does ACE send a RST packet when it reach to inactivity timeout?
----yes, the ACE is going to send a RST if the client or server tries to do something over a connection that was already timed out
3rd, How does ACE treat the packets there is no flows in conn table? Drop or forwarding?
drops the connection
Let me do some research for your second question
Cesar R
ANS Team

ACE RST problem

Hi ,
We can not solve the following situation.
The client has a normal tcp connection to server via ACE. if network interrupt occured (link up-down ) the client send SYN packet with same source port number what was used in the previously session between them. The ACE send the SYN to server but the server respond ACK packet only and not SYN,ACK packet because the TCP session is live for server. The client send the rst packet after syn but the ACE drops it.
The show conn shows the in and out sessions which were originaly betwen client and server.
Can ACE solve this situation ?
Regards,

hi !
Thanks the ideas. We tried them.
The output the supposed command
Lajos-ACE/Admin# sho np 1 me-stats "-stcp" | i dow
Segs outside window: 0
Connection shutdown FIN: 0
Connection shutdown RST: 0
We disabled the normalization without results.
The idle timeout does not help because the ACE
feels that client and server continue the old session. !!!!
the show conn output shwos the following while the client send the SYN and RST and the server send the ACK only.
8 2 in TCP 73 10.46.2.2:12346 192.168.37.221:1072 ESTAB
[ idle time : 00:00:01, byte count : 2049 ]
[ elapsed time: 00:12:41, packet count: 41 ]
90 2 out TCP 75 192.168.37.217:1072 10.46.2.2:12346 ESTAB
[ conn in reuse pool : FALSE]
[ idle time : 00:00:01, byte count : 2319 ]
[ elapsed time: 00:12:41, packet count: 46 ]
My opinion the ACE try to make a new ,second connection before SYN . The RST packet resets the second session and the first session unchanged. ( but the idle timer is not increasing )The server respond in the frisst session.
Unfortunetly the client uses the same source and destination TCP ports in every session. :-)
Regards,

A connection was abortively closed after one of the peers sent an rst packet

Hello,
i have a dvr on netwoek that is wok fine. i tried to publish it over internet.
i have tmg with two wan connections(load Balancing) and two internal networks.
i create a server application rule and and but dvr protcols on it.
when i try to open it from outside it's not working. on tmg log it's give me this error:
A connection was closed because no SYN/ACK reply was received from the server.
also with others dvr i get this error:
a connection was abortively closed after one of the peers sent an rst packet
i tired to read all post on forums but i didn't get a solution for it.
please not that the network rule from internal to external is route and the publish rule is set to make the request is from local host.
so whats the problem?
thanks.

Hi,
Thank you for your post here.
As far as i know, if you would like to publish server located in internal, you need to set the relationship between internal and external as NAT.
Best Regards
Quan Gu

Terminating established connections with TCP RST packet

Hi,
I'm making a small application for our campus. The idea is to block certain connections from outside of our network to hosts in our network. I'm analyzing the connections using jpcap, this API also has a send method that sends packets, I thought that I can terminate the connections by sending RST packet to the source but it doesn't work, connections don't terminate. Obviously I don't get any error message from the host where I'm sending it to. I think that problem might be in sequence number or something like that. For now I set the sequence number of RST packet to (acknowledgment number from the last packet that comes from the outside host+1) is this where I'm going wrong?
Cheers.

This isn't really a Java question, although I'm sure ejp will have some good advice.
I suggest you look at the relevant TCP RFCs.

Edge Server send RST packet to Client

Hi all,
I'm meeting an issue, please help me!
I'm setting up a testing LAB. After I deployed Edge Server, everything may be fine. But Client connects to Edge server, after TLS handshake, the server send RST packet to
Client. Please refer picture below.
I used CA built on Domain Controller server to assign Cert to internal and external interface of Edge server. I know I should use a public CA on Internet to assign Cert to external interface, but I'm setting LAB for testing, so I used internal CA. And my
domain internal and external are the same (e.g: internal is edge.sip96x2.com and external is access.sip96x2.com). From Client, I installed Root CA Cert downloaded from CA on Domain Controller. Client from external doesn't
have DNS server, instead of using Hosts file, the Host file includes:
"100.20.252.12 access.sip96x2.com"
I don't know what is information need to show here, if you required any information, please let me know, thanks so much!

To work with your Lync Client from External over the edge, the Lync Client has to reach
Access Edge, Audio/Video Edge and Web Edge IP.
To login to your Lync Edge you can use the lync Manual Configuration access.sip96x2.com:443.
You should use the host fqdn for internal Connection and the three needed External FQDN for the edge.
To use a private CA ist allways possible for a Lab.
http://ocsguy.com/2010/11/21/deploying-an-edge-server-with-lync/
regards Holger Technical Specialist UC

SNMP Ace client packets

Hi All,
I am doing an snmpwalk on our ACE using the following oid:
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin:0cm;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:10.0pt;
font-family:"Times New Roman","serif";}
1.3.6.1.4.1.9.9.161.1.4.2.1.9
The problem is that on some vips after doing an snmp walk I am receiving 0 for bandwidth utilisation.
When I scan the device I see there is bandwidth usage.
Below is output form snmpwalk and the device itself.
SNMP-Walk
1.3.6.1.4.1.9.9.161.1.4.2.1.2.2.222 : Counter: 0
sh service policy CM-Rebranding-888-http
class: CM-Rebranding-888-http
     VIP Address:    Protocol: Port:
     10.x.x.x      tcp        eq    80
      loadbalance:
        L7 loadbalance policy: PM-Rebranding-888-http
        VIP Route Metric     : 77
        VIP Route Advertise : DISABLED
        VIP ICMP Reply       : DISABLED
        VIP State: INSERVICE
        curr conns       : 3374      , hit count        : 8113708
        dropped conns    : 82195
        client pkt count : 186343165 , client byte count: 17308888870
       server pkt count : 292836401 , server byte count: 362759465286
        conn-rate-limit      : -         , drop-count : -
        bandwidth-rate-limit : -         , drop-count : -
        L7 Loadbalance policy : PM-Rebranding-888-http
          class/match : class-default
             LB action: :
               sticky group: Rebranding-888-http
                  primary serverfarm: SF-Rebranding-888-http
                    state: UP
                  backup serverfarm : -
            hit count        : 8113703
            dropped conns    : 0
        Parameter-map(s):
          Rebranding-888-http-Idle
It looks like a bug to me.
Any help would be appreciated in understanding this issue.
If anyone has encounterd this issue and overcome it please let me know.
Thanks.
Jack.

Jack
Probably easiest if we can set it up in the lab and test it. Would you be willing to share your config ? Or maybe open a tac case and I can take a look at it. Which version of s/w ?
Matthew

ACE ignores RST

Hi,
Please look over the attached files.
I need the teardown the session manually ( clear conn )
In spite of RST packet, the session is still reaminig ESTABLISHED state to client. The other way is closed before of FIN.
Where is the problem? wrong RST, ACE bug ?
Regards,

Karoly,
By any chance have you disabled normalization on your interfaces? The reason I ask is when I follow the tcp stream I see the 3way handshake, data, and the fin from the server and reset from client. This normally should close out the connection but after the reset I see an Ack From the .2 device. If you have disabled normalization this new Ack would build a new flow built from 100.1.1.2 to 100.1.1.7 and since the two endpoints think the connection is closed they will not try to close it out. This new connection will take 1 hour to idle out since from the ACE perspective it is just the server initiating a new connection to the server. With normalization enabled the ACE would just drop the last ack and no new connection would be built.
Hope this helps some.
Regards
Jim

CSM RST issues after SYN packet

Environment:
A couple of CSMs in a campus manage costumer's WAP browsing service. A VIP virtualizes WAP1 and WAP2 service on different tcp, udp port and CSM balances it to WAP gateway proxies.
WAP gateway's proxies initiate new connection to internet passing through CSM.
HTTP sessions are intercepted and balanced to transparent proxies to provide enrichment.
NAT is implemented for all traffic that goes out to CSM.
Other flows are managed by this CSM but they aren't involved in the reset issues.
Behavior:
Costumer sets up connection with his WAP gateway. WAP gateway initiates connection to internet properly and flow is properly balanced to transparent proxies.
Transparent proxy also initiates new connection to internet.
Sometime CSM sends RST to transparent proxies and they send to all other elements a 502 bad gateway error.
RST packet is sent in two different cases.
1. RST after a few SYN packets, 30 second between first and last SYN.
2. RST immediately after the first SYN packet from transparent proxies.
My ideas:
I putted a test WEB server on the Client VLAN of CSM to leave out other network elements or internet problems
The second issue probably is a sell-out of some resources. Looking âLB Rjct: no cl NAT portâ counter on CSM's tech-support it increases. Probably one IP of NAT isn't enough anymore.
No ideas for the first issue.
Do you have any idea?
Thanks in advance.
Roberta

when you say, RST after a few SYN, does it mean the 3-way handshake never completes ?
So, the server never responds with a SYN/ACK ??
30 sec is the pending timeout on the CSM.
That's the time we allow the tcp 3-way handshake to complete.
You can increase this timeout with the command 'pending ' under the vserver.
You can verify if this is a pending timeout issue w/ the command :
sho mod csm 3 tech proc 1 | i Pending
Gilles.

ACE - Connection Reset

Hello All,
I have a strange issue but I'm not sure it is content switch related in any way.
A group of hosts talk to two servers connected behind a content switch via a VIP.
Some dev are complaining about a high level of discarded / reset connections.
From the trace we ran you can see some RST,ACK packets in Wireshark but no RST packet prior to that last RST,ACK packet sent by the ACE module to the clients.
Did anybody come across the same kind of situation?
Regards,
Thibault.

Is there a chance that you are running code A2 (3.2)? You may be hitting a bug that I have found within my environment as well. CSCti88248.
CSCti88248—When the ACE is waiting to reassemble client packets, it may reset TCP-based client connections if all the following conditions exist:
–ACE is configured with a Layer 7 load-balancing policy where the ACE proxies the client-side TCP connection before making a load-balancing decision
–Client-side connection experiences packet loss
–The TCP TX racing messages (data) counter in the output of the show np n me-stats -stcp is incrementing
This problem can also occur with secure (SSL) terminated connections. Workaround: Configure an empty connection parameter map and add it to a multi-match policy map under the class map that is configured for the VIP experiencing the problem. For example:
parameter-map type connection TCPReassembly
policy-map multi-match MultiMatch_PolicyMap
   class HTTP_VIP_80
      loadbalance vip inservice
      loadbalance policy L7_HTTP_PolicyMap
      loadbalance vip icmp-reply active
      connection advanced-options TCPReassembly
Regards

ACE: 1/2 open connections and reset

Is there anyway to force the ACE to send a RST packet for a connection that it doesn't have in its state table? i.e. upon a 2nd failure from the secondary ACE back to the primary where existing connections do not replicate back to the primary.
EDIT: What I'm trying to do is to send a RST to a TCP flow that the client has open but the ACE is not aware of it. I'm trying to do that because I have these long lived connections that do not get replicated back upon a preemptive recovery. So the client sits there waiting forever to timeout. It would be helpful to have the ACE send a RST for a 1/2 open connection instead of just dropping it.

Casy,
You can apply a nat policy to the server vlan only, so traffic will only be nated when the connection comes from the server vlan.
If you don't want to nat all traffic, you can use a class-map that only matches a specific destination ip.
If you need further detail let me know.
Gilles.

ACE Serverform Error

I have ACE 4710 with c4710ace-mz.A3_2_2 Image.
Everything is working fine but I am getting followin failure Error for the severform related to my Proxy Server.
ACE02/Rack2# show serverfarm SF_BCPR
serverfarm     : SF_BCPR, type: HOST
total rservers : 2
                                                ----------connections-----------
       real                  weight state        current    total      failures
   ---+---------------------+------+------------+----------+----------+---------
   rserver: RS_BCPR01
       192.168.0.103:0       8      OPERATIONAL 2661       5599360    4207
   rserver: RS_BCPR02
       192.168.0.104:0       8      OPERATIONAL 986        9991646    5324
I have checked the CPU and Memory load on the ACE there is no high load. I am concern about this failure what is the reason for this failure.

In general, the counter is incremented the following reason.
1) SYN timeout
2) RST received
3) Internal exception
Please monitor the following output. If Drop counter is incremented
when failures counter is incremented, ACE drops packets due to
internal exception.
show np 1 me-stats -sicm
show np 1 me-stats -socm
show np 1 me-stats -shttp
If Drop counter is not incremented, failures counter maybe incremented
by syn timeout or rst received. In that case, to isolate the source of the
problem, you will have to get capture trace on both client and server side.
Regards,
Yuji

ACE 30, dropped conns counter incorrect number

We have host in our network which tests reachability of ACE's VIP address at regular intervals. The test sequence consists of 4 TCP packets (SYN, SYN-ACK, FIN-ACK, RST-ACK; see picture attached) and causes incrementation of "dropped conns" counter in show service-policy output.
ACE30# sh service-policy XYZ detail | inc drop
        dropped conns    : 266812
        conn-rate-limit      : 0         , drop-count : 0
        bandwidth-rate-limit : 0         , drop-count : 0
                     dropped conns: 238177
            dropped conns    : 7
ACE30# sh service-policy XYZ detail | inc drop
        dropped conns    : 266813
        conn-rate-limit      : 0         , drop-count : 0
        bandwidth-rate-limit : 0         , drop-count : 0
                     dropped conns: 238178
            dropped conns    : 7
Is this normal behavior of ACE? Is there a way how to get rid of the dropped cons counter incrementation.
Petr

Hi Kanwal,
When I set "no normalization" problem is solved. Disadvantage of this appoach is that by this command all trafic on interface is affected.
I've also tried to tune timeout for embrionic connection.
When I had set it to 0, dropped conns counter stopped to increase. Client which sends those "SYN,FIN" packets ends communication after 30 seconds using RST. This cause that connection ends and dropped conns counter does not increase.
Unfortunately for some reason sometimes happens that client doesn't send this final RST packet. This cause that number of active connection increases ...
ACE30-hto2/TEST-WEBAPP# sh service-policy XYZ | inc conn
curr conns : 9 , hit count : 2279841
dropped conns : 385467
conns per second : 0
conn-rate-limit : - , drop-count : -
ACE30-hto2/TEST-WEBAPP# sh service-policy XYZ | inc conn
curr conns : 22 , hit count : 2283653
dropped conns : 385467
conns per second : 0
conn-rate-limit : - , drop-count : -
When I set timeout to 120, those "non RST" connections are cleared but of course dropped conns counter increases ...
I guess I will try to reconfigure the probe.
Kanwal, thanks for your suggestions!
Kind regards
Petr

ACE: VIP Out of service, Still accepts TCP connections

Hi Guys. I am looking at a issue with an ACE. SW is 3.0(0)A3(2.6)
We have a setup where most of it appears fine. It detect the loss of rservers, probes fail, the VIP stops responding to Pings, but it still accept TCP connections, even though there is nothing behind to accept them.
The question is, is this correct behaviour? and if so is there any reference I can look at to confirm?
While this behaviour is inconvenient for us. I can see why it may actually be correct.
Thanks,
Paul.

Hello Paul,
This is expected behaviour for L7 LB connections:
The 'down' VIP will reply to SYN requests, but will then send a RST packet.
This is because the ACE doesn't know what to do with the L7 connection until it has been build up. Only when the L7 connection is 'open' we notice that all the vserver which could serve this request are down.
So it is correct and expected, but not exactly desired. It's just a side effect of the design. So far I do not known of any plans to change this behaviour. However similar limitations have been addressed in the past, like: CSCsq17137.
Hope this helps, Peter

Ace module dropping assymetric layer 2 connections

Hi we had a situation in where the ACE would randomly drop certain tcp connections, and all ICMP packets from a certain windows server. The server in question was using Transmit Load Balancing with Fault Tolerance.
The server has one Nic connected to Access switch1, and the other nic connected to Access switch2. Each access switch connects up to a pair of 6509's, which is active on Core1 on both switches.
I am guessing If the server sends on Nic 2, core1 knows it came in on the downstream trunk port to Switch2, it must reply to these packets based on the teamed mac of the layer 3 address(no idea who is arping for the destination - the ace?), and send them back out the downstream trunk port to switch1. The ace module is in transparent mode. When contacting a server on the other side of the ace, the ace drop packets that came from the second nic - and I am wondering how it "knows" that the return path is out of different downstream port. Does it share some kind of layer 2 RPF check with the 6500 ?
Please note there is no routing involved here. The destination server is just on another vlan on the same subnet, on the other side of the ace.

Bryan,
As long as the server replies back to the ACE the client should only be commmunicating with the VIP address in either of your two examples.
In your first example the flow will look like this.
client > VIP after the ACE client > rserver
the reply would be
rserver > client after the ACE VIP > rserver
In your second example using client nat it will look like this
Client > VIP After ACE Natpool > rserver.
the reply would be
rserver > Nat-pool after ACE VIP > client.
The ACE by default will always nat the vip to the server ip unless you use the command "transparent" under the serverfarm. When using this command we send the packet to the MAC address of the server leaving the destination IP of the VIP. The server would need to have the VIP address configured under the loopback interface.
Regards
Jim

ACE Rst Packets

Similar Messages

Maybe you are looking for