ACE SM sending a RST prematurely
I have run into an odd situation where the ACE sends a TCP reset for the public leg of a TCP session prematurely. The setup looks something like this.
Internet-->FWSM(context)-->ACE-->rservers
client upload NAT exemption routed
For VIP
<--RST
The client reports that this seems to happen most frequently on slow connection with a fair amount of packet loss. I cannot make any such claim but it is their theory. A capture from the ACE doesn't reveal anything obvious. The ACE sends ACK with Seq-1 Ack- 51777 Win29852 at timestamp 0.641248. It then follows with a RST with Seq -1 Ack - 51777 Win29852 at timestamp 0.641405 with no apparent reason why. The last packet sent by the rserver prior to this termination had Seq - 1 Ack - 51778 Win 63480 with a timestamp of 0.615157.
I don't really have much else to go on right now. I'm still digging through show commands and show-tech in the hopes that I find something. Does anyone out there have any thoughts?
Thanks,
Dave
Hi David,
For these kind of issues, the best approach would be opening a TAC case to have it investigated further.
The traffic capture you got and a showtech from the ACE would be a good start, but even better if you could get a capture showing both the client and server sides of the connection and a showtech before and after the test.
Daniel
Similar Messages
-
ACE not creating session to rserver (sending a RST)
Having a ACE-Deployed for loadbalancing web-requests which are coming from a reverse-proxy. The session persistency is based on the x-forwarded-for HTTP-header entry.
The situation works fine but in certain situations it looks like the ACE (172.16.3.200) is sending a RST shortly after an ACK in direction of the reverse-proxy (172.16.2.10).
Investigating this RST shows me that ACE is not creating a session towards to the real-server, meaning session from reverse-proxy to ACE is there but session from ACE to real-server doesn’t get created (no SYN sent from ACE).
Example:
(1) 11:20:07.677541 src:172.16.2.10 dst:172.16.3.200 proto:TCP info: 38776 > http (SYN)
(2) 11:20:07.677891 src:172.16.3.200 dst:172.16.2.10 proto:TCP info: http > 38776 (SYN, ACK)
(3) 11:20:07.677920 src:172.16.2.10 dst:172.16.3.200 proto:TCP info: 38776 > http (ACK)
(4) 11:20:07.677979 src:172.16.2.10 dst:172.16.3.200 proto:HTTP info: GET /media/global/stylesheets/class.css?v=0.20 HTTP/1.1
(5) 11:20:07.678553 src:172.16.3.200 dst:172.16.2.10 proto:TCP info: http > 38776 (ACK)
(6) 11:20:07.678553 src:172.16.3.200 dst:172.16.2.10 proto:TCP info: http > 38776 (RST, ACK)
Normally, for every session from the reverse-proxy to ACE, ACE creates a session to the real-server. In this particular trace, ACE only creates the incoming one but not the outgoing to the real-server. The real-server is alive at this time, requests just some milliseconds before and after packet four (4) are processed to the same real-server correctly.
Normalization is disabled and we’re running in routed mode.
Any idea why ACE itself doesn’t creates this new session ?I just verified "show stats http" and there is a zero (0) for max parslen errors and static parse errros, so we should be fine on the length and on the value we're expecting.
Here the relevant snippets from the configuration.
sticky http-header X-Forwarded-For STICKY_HTTP-HEADER
timeout 180
serverfarm SF_FRONTEND
class-map type http loadbalance match-all CM_STICKY_HTTP-HEADER
2 match http header X-Forwarded-For header-value ".*"
class-map match-any CM_VIP_FRONTEND
description VIP for FRONTEND
5 match virtual-address 172.16.3.200 tcp eq www
policy-map type loadbalance first-match PM_LB_FRONTEND
class CM_STICKY_HTTP-HEADER
sticky-serverfarm STICKY_HTTP-HEADER
class class-default
serverfarm SF_FRONTEND
I would love to share the broken capture with you (see attached). -
Does ACE send a RST packet when it reach inactivity timeout?
Hi experts
I have some questions about ace's behavier.
1st one is, Does ACE send a RST packet when it reach to inactivity timeout?
2nd, Does half-closed timeout works properly with "no normalization"?
3rd, How does ACE treat the packets there is no flows in conn table? Drop or forwarding?
ThanksHi Kilsoo,
1st one is, Does ACE send a RST packet when it reach to inactivity timeout?
----yes, the ACE is going to send a RST if the client or server tries to do something over a connection that was already timed out
3rd, How does ACE treat the packets there is no flows in conn table? Drop or forwarding?
drops the connection
Let me do some research for your second question
Cesar R
ANS Team -
ACE: Request Method: Options = RST
I am in need of troubleshooting a situation where a user wants to access the load balanced web application (J2EE) with an application called "portal drive".
From my understanding this application utilizes the WebDAV extension and is more or less WebDAV client with "explorer" functionalities.
Application team now complains about following symptoms:
- The Portal Drive client connects to the rservers directly just fine.
- PortalDrive client won't connect using the VIP.
Therefore i sniffed the setup from my client to the VIP and i can see following behavior.
After the initial TCP setup the first HTTP Request used is "Method: OPTIONS". After this request the ACE sends and RST,ACK and drops the connection. This happens every time i connect using the VIP.
So now my approach to find the root cause circles around the two following scenarios.
A: ACE does not support a first request with Method OPTION.
or
B: After the initial request trough the VIP address the server maybe has to be SNAT'ed into the client side vlan.
I hope you guys have an idea how to fix this.
Thanks for reading
RobleHi Gilles,
we switched of the whole http inspection a while backed due to a previous issue with the PROPFIND option, you might remember that.
I was not aware that the Inspection of WebDAV is now officially supported. So therefore the inspection is unlikely the root cause if this issue.
What about the SNAT thingy, could that be a valid reason?
Roble -
Hi,
Although we haven't change the configuration we started experiencing following error:
2009.01.29 at 15:34:30:364: Thread-40: B2B - (DEBUG) oracle.tip.adapter.b2b.msgproc.Request:preTransmit Create Event Table row for Message Retries
2009.01.29 at 15:34:30:364: Thread-40: B2B - (DEBUG) oracle.tip.adapter.b2b.msgproc.Request:preTransmit timeToAck = Thu Jan 01 03:00:00 CET 1970
2009.01.29 at 15:34:30:365: Thread-40: B2B - (DEBUG) oracle.tip.adapter.b2b.msgproc.DbAccess:DbAccess:insertEvtTblRow Enter
2009.01.29 at 15:34:30:365: Thread-40: B2B - (DEBUG) oracle.tip.adapter.b2b.msgproc.DbAccess:DbAccess:insertEvtTblRow Event Type = 2
2009.01.29 at 15:34:30:366: Thread-40: B2B - (DEBUG) oracle.tip.adapter.b2b.msgproc.DbAccess:DbAccess:insertEvtTblRow EventId = 12
2009.01.29 at 15:34:30:366: Thread-40: B2B - (DEBUG) oracle.tip.adapter.b2b.msgproc.DbAccess:DbAccess:insertEvtTblRow Id = 33156:5:0:137
2009.01.29 at 15:34:30:369: Thread-40: B2B - (DEBUG) oracle.tip.adapter.b2b.msgproc.DbAccess:DbAccess:insertEvtTblRow Exit
2009.01.29 at 15:34:30:372: Thread-40: B2B - (DEBUG) oracle.tip.adapter.b2b.msgproc.Request:preTransmit Current TimeStamp isThu Jan 29 15:34:30 CET 2009
2009.01.29 at 15:34:30:372: Thread-40: B2B - (DEBUG) oracle.tip.adapter.b2b.msgproc.Request:preTransmit Retry shall happen at Thu Jan 29 17:34:30 CET 2009
2009.01.29 at 15:34:30:372: Thread-40: B2B - (DEBUG) oracle.tip.adapter.b2b.msgproc.Request:preTransmit business transaction info name null revision null
2009.01.29 at 15:34:30:372: Thread-40: B2B - (DEBUG) oracle.tip.adapter.b2b.msgproc.Request:preTransmit Exit
2009.01.29 at 15:34:30:372: Thread-40: B2B - (DEBUG) DBContext commit: Enter
2009.01.29 at 15:34:30:374: Thread-40: B2B - (DEBUG) DBContext commit: Transaction.commit()
2009.01.29 at 15:34:30:374: Thread-40: B2B - (DEBUG) DBContext commit: Leave
2009.01.29 at 15:34:30:375: Thread-40: B2B - (DEBUG) oracle.tip.adapter.b2b.msgproc.Request:outgoingRequestPostColab Calling Send to transmit the message
2009.01.29 at 15:34:30:375: Thread-40: B2B - (DEBUG) Protocol Name: HTTPS
2009.01.29 at 15:34:30:375: Thread-40: B2B - (DEBUG) Version Name: 1.1
2009.01.29 at 15:34:30:375: Thread-40: B2B - (DEBUG) Endpoint: https://example.test.com:8443/ibis/servlet/IBISHTTPUploadServlet/as2_me_in
2009.01.29 at 15:34:30:375: Thread-40: B2B - (DEBUG) using SSL
2009.01.29 at 15:34:30:375: Thread-40: B2B - (DEBUG) oracle.tip.adapter.b2b.transport.TransportInterface:send URL: HTTPS://EXAMPLE.TEST.COM:8443/IBIS/SERVLET/IBISHTTPUPLOADSERVLET/AS2_ME_IN
2009.01.29 at 15:34:30:376: Thread-40: B2B - (DEBUG) oracle.tip.adapter.b2b.transport.TransportInterface:send TO Endpoint: 502 https:/example.test.com:8443/ibis/servlet/IBISHTTPUploadServlet/as2_me_in
2009.01.29 at 15:34:30:376: Thread-40: B2B - (DEBUG)
Protocol = HTTPS
Version = 1.1
Transport Header
Content-Transfer-Encoding:binary
Message-ID:<33156:5:0:137@as2me>
MIME-version:1.0
ACTION_NAME:PROCESS
From:as2Fideltronik
Disposition-Notification-To:[email protected]
AS2-To:RUT_AS2
User-Agent:AS2 Server
Date:Thu, 29 Jan 2009 14:34:30 GMT
DOCTYPE_NAME:PROCESS
FROM_PARTY:as2me
DOCTYPE_REVISION:1
TO_PARTY:TP_AS2
AS2-From:as2me
AS2-Version:1.1
Content-Disposition:attachment; filename=7.2
Content-Type:application/XML; name=7.2
Parameters
-- listing properties --
http.sender.timeout=0
2009.01.29 at 15:34:30:464: Thread-40: B2B - (DEBUG) scheme null userName null realm null
2009.01.29 at 15:34:45:178: B2BStarter thread: Deployment - (DEBUG) Query Configurations null Lifecycle status Active exclude design true
2009.01.29 at 15:34:45:178: B2BStarter thread: BusinessLogicLayer - (DEBUG) Authorization disabled. UserBootstrapped:false, useAuthorization:true,
2009.01.29 at 15:37:20:021: Thread-40: B2B - (WARNING)
Message Transmission Transport Exception
Transport Error Code is OTA-HTTP-SEND-1000
StackTrace oracle.tip.transport.TransportException: [IPT_HttpSendError] HTTP encounters send error :.
at oracle.tip.transport.TransportException.create(TransportException.java:91)
at oracle.tip.transport.basic.HTTPSender.createTransportResponse(HTTPSender.java:754)
at oracle.tip.transport.basic.HTTPSender.send(HTTPSender.java:598)
at oracle.tip.transport.b2b.B2BTransport.send(B2BTransport.java:311)
at oracle.tip.adapter.b2b.transport.TransportInterface.send(TransportInterface.java:1013)
at oracle.tip.adapter.b2b.msgproc.Request.outgoingRequestPostColab(Request.java:1754)
at oracle.tip.adapter.b2b.msgproc.Request.outgoingRequest(Request.java:972)
at oracle.tip.adapter.b2b.engine.Engine.processOutgoingMessage(Engine.java:1166)
at oracle.tip.adapter.b2b.xmlgwIntg.XMLGWIntegration.raiseOutboundMessage(XMLGWIntegration.java:168)
at oracle.tip.adapter.b2b.xmlgwIntg.Outbound.onMessage(Outbound.java:297)
at oracle.jms.AQjmsListenerWorker.dispatchOneMsg(AQjmsListenerWorker.java:316)
at oracle.jms.AQjmsListenerWorker.run(AQjmsListenerWorker.java:129)
at java.lang.Thread.run(Thread.java:534)
Caused by: java.io.EOFException: Premature EOF encountered
at HTTPClient.StreamDemultiplexor.read(StreamDemultiplexor.java:283)
at HTTPClient.RespInputStream.read(RespInputStream.java:157)
at HTTPClient.RespInputStream.read(RespInputStream.java:116)
at HTTPClient.Response.readResponseHeaders(Response.java:997)
at HTTPClient.Response.getHeaders(Response.java:713)
at HTTPClient.Response.getStatusCode(Response.java:262)
at HTTPClient.RetryModule.responsePhase1Handler(RetryModule.java:83)
at HTTPClient.HTTPResponse.handleResponse(HTTPResponse.java:739)
at HTTPClient.HTTPResponse.getStatusCode(HTTPResponse.java:196)
at oracle.tip.transport.basic.HTTPSender.createTransportResponse(HTTPSender.java:721)
... 11 more
2009.01.29 at 15:37:20:022: Thread-40: B2B - (DEBUG) oracle.tip.adapter.b2b.transport.TransportInterface:send Error in sending message
2009.01.29 at 15:37:20:022: Thread-40: B2B - (INFORMATION) oracle.tip.adapter.b2b.msgproc.Request:outgoingRequestPostColab Request Message Transmission failed
2009.01.29 at 15:37:20:022: Thread-40: B2B - (DEBUG) DBContext beginTransaction: Enter
2009.01.29 at 15:37:20:022: Thread-40: B2B - (DEBUG) DBContext beginTransaction: Transaction.begin()
2009.01.29 at 15:37:20:022: Thread-40: B2B - (DEBUG) DBContext beginTransaction: Leave
2009.01.29 at 15:37:20:022: Thread-40: B2B - (DEBUG) oracle.tip.adapter.b2b.msgproc.Request:outgoingRequestPostColab [IPT_HttpSendError] HTTP encounters send error :.
Premature EOF encountered
2009.01.29 at 15:37:20:023: Thread-40: BusinessLogicLayer - (DEBUG) Authorization disabled. UserBootstrapped:false, useAuthorization:true, configType:null
2009.01.29 at 15:37:20:023: Thread-40: BusinessLogicLayer - (DEBUG) Push Stack: updateBusinessMessage
2009.01.29 at 15:37:20:026: Thread-40: BusinessLogicLayer - (DEBUG) Pop Stack: updateBusinessMessage
2009.01.29 at 15:37:20:026: Thread-40: B2B - (DEBUG) DBContext commit: Enter
2009.01.29 at 15:37:20:027: Thread-40: B2B - (DEBUG) DBContext commit: Transaction.commit()
2009.01.29 at 15:37:20:027: Thread-40: B2B - (DEBUG) DBContext commit: Leave
2009.01.29 at 15:37:20:028: Thread-40: B2B - (DEBUG) oracle.tip.adapter.b2b.msgproc.Request:outgoingRequest Exit
2009.01.29 at 15:37:20:028: Thread-40: B2B - (INFORMATION) oracle.tip.adapter.b2b.engine.Engine:processOutgoingMessage:
***** REQUEST MESSAGE *****
Exchange Protocol: AS2 Version 1.1
Transport Protocol: HTTPS
Unique Message ID: <33156:5:0:137@as2me>
Trading Partner: TEST_AS2
Message Signed: No
Payload encrypted: No
Attachment: None
***** REQUEST MESSAGE *****
2009.01.29 at 15:37:20:028: Thread-40: B2B - (INFORMATION) oracle.tip.adapter.b2b.engine.Engine:processOutgoingMessage Exit
2009.01.29 at 15:37:20:028: Thread-40: B2B - (DEBUG) : Thu Jan 29 15:37:20 CET 2009 Exit Outbound onMessage
2009.01.29 at 15:37:20:030: Thread-40: B2B - (DEBUG) DBContext commit: Enter
2009.01.29 at 15:37:20:030: Thread-40: B2B - (DEBUG) DBContext commit: Leave
2009.01.29 at 15:37:20:031: Thread-40: B2B - (DEBUG) : Thu Jan 29 15:37:20 CET 2009 Available memory Leave Outbound onMessage: 35505240
2009.01.29 at 15:37:20:031: Thread-40: B2B - (DEBUG) : Thu Jan 29 15:37:20 CET 2009 Leave Outbound onMessage
I assume that this might be caused by applying the latest MLR patch or due to some changes on the TP side.
Please adviseHi Suhas,
Yes, no matter the size of the XML file, it always sends it 6 times. Error occurs only at the end of the process, that is after 6th time the file has been sent. We tested it many times with new messages and each time the results are identical. There is no trace in b2b.log that the file is being sent again, transport log however shows 6 http send entries each time.
Also, between the start of sending the message and the error occurring after 2-3 minutes (after 6th message is sent) the message is not removed from the ECX outbound queue (we are getting the XML from E-Bussines). It is removed after the error. It cannot manually dequeued during that time.
Thank you
Kamil, -
Can ACE 4710 send ICMP-dest-unreachable?
Dear Community!
We have previously configured an ACE context for implementing redundant corporate DNS service and now testing a transparent ACE context and HA configuration.One virtual-IP is configured for UDP/53, listening for DNS requests. Behind the VIP, there are 3 DNS server. The next step of our testing process, we have shut down all real-server instance behind the virtual-IP while inspecting DNS clients behaviour. Besides the DNS clients requesting the virtual-IP DNS service need ICMP-destination-unreachable packet to switchover the secondary DNS server.
Can ACE 4710 send ICMP-dest-unreachable?
Thanks in advance!
Regards,
Belabacsi
from HungaryUnfortunately the 4710 does not send icmp unreachable when a vserver is down.
If you have backup dns service, you can configure it on ace itself.
Gilles. -
ACE 4710 send Connection:Close when should be Keep-Alive
After user request to front end http to 10.85.10.4 (default 80) after a port redirect and action list header rewrite
header rewrite request host header-value http://10[.]85[.]10[.]4 replace http://10.85.10.67:84/jde/E1Menu.maf%1
I see the request go out (wireshark) to the back-end javaserver but in the Connection it's close not keepalive:
GET /jde/E1Menu.maf HTTP/1.1
Connection: Close
Host:10.85.10.67:84
After the get from the ACE the jserver replies with the JDE login screen but the ACE ignores it?Try by enabling persistence rebalance in an http parameter-map.
Also your rewrite rule is wrong, you've been mistaken regarding the role of the Host field I guess. What you try to configure in your config is a URL rewrite but it's not supported by the ACE. -
ACE: 1/2 open connections and reset
Is there anyway to force the ACE to send a RST packet for a connection that it doesn't have in its state table? i.e. upon a 2nd failure from the secondary ACE back to the primary where existing connections do not replicate back to the primary.
EDIT: What I'm trying to do is to send a RST to a TCP flow that the client has open but the ACE is not aware of it. I'm trying to do that because I have these long lived connections that do not get replicated back upon a preemptive recovery. So the client sits there waiting forever to timeout. It would be helpful to have the ACE send a RST for a 1/2 open connection instead of just dropping it.Casy,
You can apply a nat policy to the server vlan only, so traffic will only be nated when the connection comes from the server vlan.
If you don't want to nat all traffic, you can use a class-map that only matches a specific destination ip.
If you need further detail let me know.
Gilles. -
Server 2008 (R2) TCP/IP Stack sending RST to close short lived sessions
Hi,
I'm having an issue with some vendor software, but it appears to be more closely related to the way the TCP/IP stack is handling session shutdown. I'd like to know what this feature is called, any available documentation, and ideally how to disable it.
Basically, what appears to happen, is Server 2008 is sending a Rst, Ack to terminate a short lived connection, instead of entering the standard TCP shutdown (Using FIN flags). This appears to be an attempt to avoid having short lived sessions sit in a
TIME_WAIT state, as I can see long TCP connections properly being shutdown.
I realize the benefits of what this is trying to accomplish, however, the software in question is making HTTP calls, and the server being rather basic, is sending HTTP responses without content-length or transfer encoding: chunked, which means the only
way to tell the server is done sending content is for the connection to close. However, it appears that the stack is interpreting this type of Tcp shutdown as in error, and generating annoying alerts within the application that is monitoring the close state.
Does windows have a way to disable this stack feature. I've confirmed the chimney offload doesn't appear to be in use, so this is an effect of the Windows stack itself. I don't have control of the software on either end, but do have a bug open with the vendor,
I'm more interested in a possible workaround for the short term.
** Entire connection lasts ~1 second
Internet Protocol, Src: 172.25.149.231 (172.25.149.231), Dst: 172.25.147.172 (172.25.147.172)
Transmission Control Protocol, Src Port: 49740 (49740), Dst Port: 8089 (8089), Seq: 0, Len: 0
Flags: 0x02 (SYN)
Internet Protocol, Src: 172.25.147.172 (172.25.147.172), Dst: 172.25.149.231 (172.25.149.231)
Transmission Control Protocol, Src Port: 8089 (8089), Dst Port: 49740 (49740), Seq: 0, Ack: 1, Len: 0
Flags: 0x12 (SYN, ACK)
Internet Protocol, Src: 172.25.149.231 (172.25.149.231), Dst: 172.25.147.172 (172.25.147.172)
Transmission Control Protocol, Src Port: 49740 (49740), Dst Port: 8089 (8089), Seq: 1, Ack: 1, Len: 0
Flags: 0x10 (ACK)
Internet Protocol, Src: 172.25.147.172 (172.25.147.172), Dst: 172.25.149.231 (172.25.149.231)
Transmission Control Protocol, Src Port: 8089 (8089), Dst Port: 49740 (49740), Seq: 1, Ack: 2921, Len: 0
Flags: 0x10 (ACK)
Internet Protocol, Src: 172.25.149.231 (172.25.149.231), Dst: 172.25.147.172 (172.25.147.172)
Transmission Control Protocol, Src Port: 49740 (49740), Dst Port: 8089 (8089), Seq: 2921, Ack: 1, Len: 1234
Flags: 0x18 (PSH, ACK)
Internet Protocol, Src: 172.25.147.172 (172.25.147.172), Dst: 172.25.149.231 (172.25.149.231)
Transmission Control Protocol, Src Port: 8089 (8089), Dst Port: 49740 (49740), Seq: 1, Ack: 4155, Len: 57
Flags: 0x18 (PSH, ACK)
Internet Protocol, Src: 172.25.149.231 (172.25.149.231), Dst: 172.25.147.172 (172.25.147.172)
Transmission Control Protocol, Src Port: 49740 (49740), Dst Port: 8089 (8089), Seq: 4155, Ack: 58, Len: 0
Flags: 0x10 (ACK)
Internet Protocol, Src: 172.25.147.172 (172.25.147.172), Dst: 172.25.149.231 (172.25.149.231)
Transmission Control Protocol, Src Port: 8089 (8089), Dst Port: 49740 (49740), Seq: 58, Ack: 4155, Len: 1024
Flags: 0x18 (PSH, ACK)
Internet Protocol, Src: 172.25.147.172 (172.25.147.172), Dst: 172.25.149.231 (172.25.149.231)
Transmission Control Protocol, Src Port: 8089 (8089), Dst Port: 49740 (49740), Seq: 1082, Ack: 4155, Len: 1460
Flags: 0x18 (PSH, ACK)
Internet Protocol, Src: 172.25.149.231 (172.25.149.231), Dst: 172.25.147.172 (172.25.147.172)
Transmission Control Protocol, Src Port: 49740 (49740), Dst Port: 8089 (8089), Seq: 4155, Ack: 2542, Len: 0
Flags: 0x10 (ACK)
Internet Protocol, Src: 172.25.147.172 (172.25.147.172), Dst: 172.25.149.231 (172.25.149.231)
Transmission Control Protocol, Src Port: 8089 (8089), Dst Port: 49740 (49740), Seq: 2542, Ack: 4155, Len: 1460
Flags: 0x18 (PSH, ACK)
Internet Protocol, Src: 172.25.147.172 (172.25.147.172), Dst: 172.25.149.231 (172.25.149.231)
Transmission Control Protocol, Src Port: 8089 (8089), Dst Port: 49740 (49740), Seq: 4002, Ack: 4155, Len: 1460
Flags: 0x18 (PSH, ACK)
Internet Protocol, Src: 172.25.147.172 (172.25.147.172), Dst: 172.25.149.231 (172.25.149.231)
Transmission Control Protocol, Src Port: 8089 (8089), Dst Port: 49740 (49740), Seq: 5462, Ack: 4155, Len: 1460
Flags: 0x18 (PSH, ACK)
Internet Protocol, Src: 172.25.147.172 (172.25.147.172), Dst: 172.25.149.231 (172.25.149.231)
Transmission Control Protocol, Src Port: 8089 (8089), Dst Port: 49740 (49740), Seq: 6922, Ack: 4155, Len: 1081
Flags: 0x18 (PSH, ACK)
Internet Protocol, Src: 172.25.149.231 (172.25.149.231), Dst: 172.25.147.172 (172.25.147.172)
Transmission Control Protocol, Src Port: 49740 (49740), Dst Port: 8089 (8089), Seq: 4155, Ack: 8003, Len: 0
Flags: 0x10 (ACK)
Internet Protocol, Src: 172.25.147.172 (172.25.147.172), Dst: 172.25.149.231 (172.25.149.231)
Transmission Control Protocol, Src Port: 8089 (8089), Dst Port: 49740 (49740), Seq: 8003, Ack: 4155, Len: 1460
Flags: 0x18 (PSH, ACK)
Internet Protocol, Src: 172.25.147.172 (172.25.147.172), Dst: 172.25.149.231 (172.25.149.231)
Transmission Control Protocol, Src Port: 8089 (8089), Dst Port: 49740 (49740), Seq: 9463, Ack: 4155, Len: 108
Flags: 0x18 (PSH, ACK)
Internet Protocol, Src: 172.25.149.231 (172.25.149.231), Dst: 172.25.147.172 (172.25.147.172)
Transmission Control Protocol, Src Port: 49740 (49740), Dst Port: 8089 (8089), Seq: 4155, Ack: 9571, Len: 0
Flags: 0x10 (ACK)
Internet Protocol, Src: 172.25.147.172 (172.25.147.172), Dst: 172.25.149.231 (172.25.149.231)
Transmission Control Protocol, Src Port: 8089 (8089), Dst Port: 49740 (49740), Seq: 9571, Ack: 4155, Len: 0
Flags: 0x14 (RST, ACK)Rick, this did not help as it's the approach i've already taken in testing. Albeit, I did disable the chimney offload on the NIC drivers instead of the windows options, but in the wireshark captures I'm still seeing the same behavior.
Humand, I don't think you're issue is the same as mine, I beleive these RST, ACK's are in response to a NORMAL connection shutdown, and being interpretted as errors be the partner stack. I haven't seen this cause premature tcp shutdown, which would be you're
dropped RDP connections.
TCP Global Parameters
Receive-Side Scaling State : disabled
Chimney Offload State : disabled
NetDMA State : enabled
Direct Cache Acess (DCA) : disabled
Receive Window Auto-Tuning Level : disabled
Add-On Congestion Control Provider : ctcp
ECN Capability : disabled
RFC 1323 Timestamps : disabled -
Hi ,
We can not solve the following situation.
The client has a normal tcp connection to server via ACE. if network interrupt occured (link up-down ) the client send SYN packet with same source port number what was used in the previously session between them. The ACE send the SYN to server but the server respond ACK packet only and not SYN,ACK packet because the TCP session is live for server. The client send the rst packet after syn but the ACE drops it.
The show conn shows the in and out sessions which were originaly betwen client and server.
Can ACE solve this situation ?
Regards,hi !
Thanks the ideas. We tried them.
The output the supposed command
Lajos-ACE/Admin# sho np 1 me-stats "-stcp" | i dow
Segs outside window: 0
Connection shutdown FIN: 0
Connection shutdown RST: 0
We disabled the normalization without results.
The idle timeout does not help because the ACE
feels that client and server continue the old session. !!!!
the show conn output shwos the following while the client send the SYN and RST and the server send the ACK only.
8 2 in TCP 73 10.46.2.2:12346 192.168.37.221:1072 ESTAB
[ idle time : 00:00:01, byte count : 2049 ]
[ elapsed time: 00:12:41, packet count: 41 ]
90 2 out TCP 75 192.168.37.217:1072 10.46.2.2:12346 ESTAB
[ conn in reuse pool : FALSE]
[ idle time : 00:00:01, byte count : 2319 ]
[ elapsed time: 00:12:41, packet count: 46 ]
My opinion the ACE try to make a new ,second connection before SYN . The RST packet resets the second session and the first session unchanged. ( but the idle timer is not increasing )The server respond in the frisst session.
Unfortunetly the client uses the same source and destination TCP ports in every session. :-)
Regards, -
I have an ACE context sending TCP resets. The configuration is the same as another ACE in a different data center, and in the other data center it is working. I'm doing end-to-end SSL (SSL termination and initiation), and PCAP traces show the ACE sending the reset both to client and server. "show stats loadbalance" shows layer 7 rejections, but the layer 7 policy being matched is 'match http url .*'. Any ideas would be welcome.
Hi There,
In case everything looks good on the captures, meaning the SSL handshake and all that then perhaps you may consider to take a look of this bug and perhaps apply the workaround:
CSCtx92484
—During a Layer 7 file transfer is terminated after transferring approximately 16 kB of data. Workaround: Configure an HTTP parameter map and set the content-maxparse-length and header-maxparse-length to larger values. For example:
parameter-map type http PM-HTTP
persistence-rebalance
set header-maxparse-length 65535
set content-maxparse-length 65535
Hope this helps
Jorge -
I recently replaced a Local Director with a CSS 11500 (v 8.2). I have an application that uses port 80 to send SOAP heartbeats at 1 minute intervals to a web server to maintain state. For some reason the CSS randomly decides to send RST to the client even though the backend service is active. In other words the the web server is not sending a RST. Is this an issue with flows? Load balancing schema? I did not have this issue with the Local Director.
no. This is not possible.
Gilles. -
ACE-TCP RST, ACE initiated RST
Hello members,
What is the command to identify if ACE is initiating TCP RST?
Is there a way to overcome or change timers/window/ect, if ACE do initiate TCP RST?
Thanks in advance members.Thanks Giles.
Very useful command
show np 1 me-stats "-s tcp"
I'll look at the capture file that was done couple of days ago.
I'm attaching the show output, just collected. Any clues will be appriciated.
switch/001_snippedcus#
switch/001_snippedcus# show np 1 me-stats "-s tcp"
TCP Statistics: (Current)
TCP RX messages received: 0x50bc5841
TCP RX unknown messages: 0
TCP RX racing messages (fin): 6815
TCP RX racing messages (forward): 199102
TCP RX racing messages (conn create): 170
TCP TX messages received: 0x41fa25a8
TCP TX Hi Priority messages received: 14002856
TCP TX unknown messages: 0
TCP TX racing messages (connect): 7
TCP TX racing messages (data): 1884954
TCP TX racing messages (proxy): 8066793
Reproxy message received: 0x073533fd
Data messages received: 0x2a741cdc
TCP connect message received: 0x02639a8a
Ack trigger message received: 3036
Unproxy req. message received: 0x07d7f2ce
Unproxy rsp. message received: 0x07cacdc4
TCP accepted msgs sent: 0x01343370
TCP connected msgs sent: 0x0263408b
Conn_ctrl msgs sent: 5029091
Buffer alloc failed: 0
Invalid msg ring id: 1218
Start retrans timer: 0x1f175a0c
Start ackdelay timer: 0x171d7346
Start persist timer: 300659
Start timewait timer: 0
Delete act timer: 0x16ee4b8f
Delete rtp timer: 0x1e3ee16d
Connections unproxying: 0x07cacfa5
Connections unproxying canceled by TCP: 833690
Connections unproxying canceled by app: 156752
Connections unproxying immediate reproxy 0x0315f058
Connections unproxying flush retransq: 0x07c2788a
Connections unproxying flush inputq: 0x02167109
Connections unproxied: 0x04829af2
Connections reproxied: 0x039a94a0
Drop reproxy msg queue full: 2916
Drop control msg: 7972
Drops due to FastTX queue full: 0
Drops due to Fastpath queue full: 0
Drops due to HTTP queue full: 134145
Drops due to SSL queue full: 0
Drops due to AI queue full: 0
ACK past SEQ: 36504
Unproxy rsp post failed: 207
Drops due to invalid proxy id: 0
(Context ALL Statistics)
Handshakes completed: 0x0395731f
Handshakes failed: 150512
Packets received to app: 0x2ad1fc92
Packets sent to network: 0x57f27343
Segs outside window: 524277
Dup ACKs received: 0
Dup ACK limit met: 0
Malformed TCP options: 0
Reassemble segs: 806787
Nagled data segs: 0
Retransmitted data segs: 0x065e16c4
Round-trip timeouts: 10595624
Round-trip timeout limit met: 385691
Persist timeouts: 70351
..... see attachment, as this exceeds char limits -
ACE - Policy map bound to multiple interface
Hello,
I have a policy map bound to multiple VLAN interfaces. The policy is pretty standard, any traffic hitting the VIP is load balanced.
Now, is it ok to assign the same policy map / VIP to to multiple VLAN interfaces on a virtual context?
I addition, I should add that one of the clients hitting the vip are the servers configured in the serverfarm of the context.
Basically the requirement here is that the rservers are client and server at the same time.
The problem I have is that when one of the servers send an HTTP request to the VIP, the ACE module reset the connection. I can see the dropped conns counter increasing as i generate requests to the ACE.
Rdgs,
Thibault.Thibault,
the RESET is probably comming from the server.
If the server sends a SYN to the VIP, the packet is nated and forwarded to another server which sees a packet coming from a neighbor server (not ACE) and sends the SYN/ACK directly to the client(rserver).
This one is expecting a packet from the VIP and not the server itself and sends a RST.
You need to enable client nat for server opening connections to the vip.
Gilles. -
Sticky session reset by ¿ACE or real server?
Hello team.
I am looking for hints to debug cookie-based sessions that are failing to work across my ACE. Basically, the user types http://10.150.3.130/iwsupport, and that shoud be distributed across a farm of servers hidden behind the ACE.The servers set a cookie PHPSESSID=<value> when this URL is requested.
The customer tells me that he thinks that the problem arises when he requests access to the VIP with the POST command (please see the attached wireshark capture, line 52). His browser receives the following message:
Based on the original requirements, I configured the ACE, whose related section of the configuration is the following:
sticky http-cookie PHPSESSID STICKY_SERVERS
timeout 720
serverfarm TEST_SERVERFARM
replicate sticky
class-map type http loadbalance match-all iwsupport
match http url /iwsupport.*
policy-map type loadbalance http first-match TEST_POLICY
class iwsupport
sticky-serverfarm STICKY_SERVERS
class class-default
serverfarm TEST_SERVERFARM
class-map match-all VIP-130
match virtual-address 10.150.3.130 tcp eq www
policy-map multi-match CLIENT_VIPS
class VIP-130
loadbalance vip inservice
loadbalance policy TEST_POLICY
loadbalance vip icmp-reply active
I would appreciate your hints to get session information, debugs, or whatever it could be useful in order to see why this is not working properly.
Thank you very much in advance
Rogelio Alvez
ArgentinaHi Rogello,
Do you see on server itself if POST request sent by client reached server or not? And if yes what did server reply? If you don't see POST request on the server then most probably it is the ACE which is sending the RST.
the outputs suggested by Jorge should help us and of course the suggested changes.
The changes will ensure that ACE parses upto 65535 bytes which is to ensure that ACE doesn't drop connection because it couldn't read which it was told to because it was way too far in the packet. By default ACE parses up to 4096 bytes.
Regarding persistence rebalance, When the first HTTP request comes in, the ACE will match the request to a layer-7 class-map and load balance it to one of the servers within the serverfarm associated with that class-map. The ACE will then also match all subsequent requests on the same TCP connection to a layer 7 class-map. If the subsequent request matches the same layer 7 class-map as the previous request, then it will be sent to the same server as the previous request. If it matches a different layer 7 class-map, then it will be load balanced to one of the servers within the serverfarm of the newly matched layer-7 class-map according to the serverfarm’s predictor.
I doubt this will make any difference since without rebalance the traffic would be sent to the same server which i guess is not a problem here.
switch/Admin(config-parammap-http)# parsing non-strict--->This is a valid command and should work fine.
For allocating resources you can go to resource class and use limit resource command to allocate resources.
You can send the data at [email protected] Also, it would be good to have 2-3 instances of outputs while you do testing so that we can see the difference if any fail counter is increasing.
Regards,
Kanwal
Maybe you are looking for
-
Addition of a new column in Table Maintenance Generator
Hi All, I have a requirement in which I have to include a new field in Table Maintenance Generator . This field is not present in the table but its data will be fetched from another table . Pls help me in this regard. I have little idea of Table cont
-
Lightroom 2.7 plug-in for Photoshop CS5?
Is there a Lightroom plugin for photoshop so I don't have to open both programs in order to use Lightroom? If so, how do I find and install it?
-
Numpy package installs in the $HOME
I wanted to compile numpy with Atlas support, so I downloaded the package and edited the PKGBUILD adding the lib names: # $Id: PKGBUILD 169025 2012-10-17 10:20:51Z allan $ # Maintainer: Jan de Groot <[email protected]> # Contributor: Douglas Soares d
-
Pda graph redraws every time when cursor is moved
Hi, I use PDA Module 8.2 and Windows Mobile 5 OS on PDA. I have a problem that every time I move cursor on graph control plot is being redrawn, so it realy annoying when you have 5000 or more points and you have to wait a second while plot is being r
-
Second install is always a trial version?
Good morning, I have bought the CC photographers package (Photoshop and Lightroom). Installing on my Mac was no problem, but when I installed the package on my Windows laptop I only see a trial version. I cannot find the license code for input in the