ACE Drop (Dest nat fail):
Hi All,
I'm using ACE module A2(2.4)
I'm trying to use parameter server-conn reuse, but clients get sometimes statuscode 503.
A#1/Test1# show np 1 me-stats "-socm -v"
OCM Statistics: (Current)
Errors: 0 0
Connection create received: 231121503 1142
LB dest decision received: 365473159 1473
Nat app fixup recieved: 0 0
Connection unproxy received: 52997475 393
Connection reproxy received: 51249279 375
IPCP received: 83227 2
ACK trigger received: 52733008 390
TCP connected received 218498529 1065
Unknown message received: 0 0
Drop [LB dest decision fail]: 29392 0
Drop [invalid ifid] 0 0
Drop [Out of buffers]: 0 0
Dest decision transmitted: 248735645 1174
TCP connect transmitted: 212827881 828
ACK trigger transmitted: 12 0
IPCP transmitted: 83227 2
NAT[static mapped]: 0 0
NAT[static real]: 0 0
NAT[xlate alloc fail]: 0 0
NAT[xlate real hit]: 0 0
NAT[xlate mapped hit]: 0 0
NAT[invalid xlate]: 0 0
NAT[dump xlate]: 0 0
NAT[xlate release failed]: 0 0
NAT Pool Alloc [fail]: 0 0
NAT Pool Alloc [addr]: 0 0
NAT Pool Alloc [addr/port]: 33689970 81
NAT Pool Free [addr]: 0 0
NAT Pool Free [addr/port]: 33689214 88
NAT Pool Free [orphan IP]: 0 0
Reuse retrieve link update conn invalid 0 0
Reuse retrieve link update conn not on r 0 0
Reuse retrieve success but conn invalid: 0 0
Drop [Next Hop queue full]: 0 0
Reuse retrieve miss: 845627 3
OCM Packet count (Hi & Lo): 976499360 4850
Packet forward received: 4343180 10
NAF Error [no route or unresolved adjace 0 0
NAF Error [nat resp fail]: 0 0
UDP Chaser received: 10406 0
(Context 1 Statistics)
Drop [out of connections]: 0 0
Drop [out of proxies]: 0 0
Drop [out of ssl]: 0 0
Drop [mac lookup fail]: 0 0
Drop [route lookup fail]: 0 0
Drop [nat fail] 0 0
Drop [ip sanity check fail] 0 0
Drop [acl deny]: 0 0
Drop [redundant connection]: 0 0
Connection inserted: 862670 3
Packet message transmitted: 6409302 230
Reuse conns retrieved: 6390611 238
Drop [Reproxy fail]: 171 0
Drop [dest nat fail]: 58286 2
The last counter is increasing. What does it mean? Can this be the problem?
I do not get 503 in the retcode map of the servers.
Regards
Mats
Hi Mats,
I find it very strange that the ACE is sending a 503 message back to the client, because, in case of issues, it normally just resets the connection. With that in mind, we should also investigate the server itself. This is not trivial, so, you should open a TAC case.
Let me just explain the meaning of the "Drop [dest nat fail]" counter. It will be incremented if, after a connection has been natted, one of the servers tries to open a new connection against the natted IP and port. This shouldn't happen unless you are using a protocol composed of several connections (for example, FTP)
Regards
Daniel
Similar Messages
-
ACE 4710 - DM initialization failed
When trying to get to the device manager GUI on my ACE 4710 I get to the login screen. On entering credentials I am given an error
"DM initialization failed (Failed to import ACE configuration: Device discovery failed: unknown). Contact your technical support team."
I have tried "dm reload" but I am still getting the error.
Any help greatfully appreciated.You are probably hitting CSCsv95366. This is fixed in A3(2.2).
You can get the details about this bug at
http://tools.cisco.com/Support/BugToolKit/action.do?hdnAction=searchBugs
HTH
Syed Iftekhar Ahmed -
Have a MacMini that is dropping NFS Share connections, Using cifs://servername/path however the NFS Share is dropped everyday without fail. Is there a better way to setup this share? I've got 5 other Macs that use the same NFS Share and none of them are dropping the shared connection. This is on OSX 10.9.x Maverick.
cifs:// is not NFS cifs:// is effectively SMB i.e. Windows filesharing, NFS is Unix filesharing, furthermore cifs:// is a way of forcing Mavericks to use SMB version 1 protocol. Normally you would use smb:// which will default to trying to use SMB version 2 protocol.
Saying all that, SMB version 1, SMB version 2, and NFS should all normally work fine. It is possible that the server may be configured to disconnect idle users although one would expect this to happen with other users as well. It is perhaps more likely that your problem Mavericks Mac maybe going to sleep periodically, while asleep the connection to the server maybe being lost. Apple's own server software has specific cleverness to allow clients to sleep and resume their connection when they wake.
Therefore it might be worth checking your Energy Saver settings on this Mac and disabling computer sleep. -
ACE dropped conns with New Vip
I have been load balancing our mail servers for quite sometime without an issue however I have been using a dynamic Nat statement. This however causes our mail team to have problems with logging. I then created a whole new vlan and ace context for the mail servers to use. This is where my dilemma is.
I now have dropped connections going to my vip but only from one server which is our Anti-span / Antivirus server which filters the mail from the internet and then passes it on to these other mail servers.
I can send mail just fine if I don't use the VIP I created.
Also if I use a Nat statement the mail sends fine but obviously I don't want to use that anymore.
The only thing I see that the ACE is not doing is closing the connections. So if every five minutes I do a clear conn all, I won't get any dropped connections for at least 10 to 15 minutes but I am not going to be doing this. Right now I have a server with a script that logs into the ace and then clears the connection but this is a band aid problem.
Here is my config. This is the only thing on this context. All 6 of my other contexts do not have this issue.
access-list ALL line 10 extended permit ip any any
access-list ALL line 18 extended permit icmp any any
probe smtp SMTP_Probe
interval 15
passdetect interval 30
expect status 210 250
parameter-map type connection TCP_Mail_TO
slowstart
set timeout inactivity 2
set tcp timeout half-closed 15
set tcp ack-delay 300
tcp-options timestamp allow
rserver host hub2
ip address *.*.*.*.*.*
inservice
serverfarm host Mail_Hub_Servers_SF
probe SMTP_Probe
rserver hub2 25
inservice
class-map match-all Mail_Hub_VIP
2 match virtual-address *.*.*.*.*.* tcp eq smtp
class-map type management match-any Remote_Management
2 match protocol http any
3 match protocol icmp any
4 match protocol telnet any
5 match protocol ssh any
policy-map type management first-match rmt_mgt_policy
class Remote_Management
permit
policy-map type loadbalance first-match Mail_Hub_VIP-l7slb
class class-default
serverfarm Mail_Hub_Servers_SF
policy-map multi-match int7
class Mail_Hub_VIP
loadbalance vip inservice
loadbalance policy Mail_Hub_VIP-l7slb
loadbalance vip icmp-reply active
loadbalance vip advertise active
connection advanced-options TCP_Mail_TO
access-group input ALL
interface vlan 108
ip address *.*.*.*.
alias *.*.*.*
peer ip address *.*.*.*.
no normalization
no icmp-guard
service-policy input rmt_mgt_policy
service-policy input int7
no shutdown
ip route 0.0.0.0 0.0.0.0 *.*.*.*I would like to avoid trying routed mode for this just right now because we haven't had a good experience in routed mode here. I can try creating a new context in routed mode because I cannot experiment with production mail. Also I have this scenario working fine on 3 other contexts with 0 Connections being dropped. The other thing is I am not dropping all connections its dropping about 2-8%. of the connections. I have been playing around with connection limits.
Interface: vlan 108
service-policy: int7
class: Mail_Hub_VIP
loadbalance:
L7 loadbalance policy: Mail_Hub_VIP-l7slb
VIP Route Metric : 77
VIP Route Advertise : ENABLED-WHEN-ACTIVE
VIP ICMP Reply : ENABLED-WHEN-ACTIVE
VIP State: INSERVICE
curr conns : 1 , hit count : 12052
dropped conns : 839
client pkt count : 385190 , client byte count: 375718706
server pkt count : 133814 , server byte count: 11089648
conn-rate-limit : 50 , drop-count : 0
bandwidth-rate-limit : - , drop-count : -
Parameter-map(s):
TCP_Mail_TO -
ACE dropped conns problem (Bridged mode)
Dear all,
I configured an ACE in bridged mode (inside vlan: 2012, outside vlan: 2021) and I apply the L4 policy on the 2 VLAN interface to loadbalance HTTP incoming request (Virtual IP: 172.22.22.130).
interface vlan 2112
bridge-group 1
access-group input BPDU-Allow
service-policy input POLICY-LB-HMC-2112
no shutdown
interface vlan 2122
bridge-group 1
access-group input BPDU-Allow
service-policy input POLICY-LB-HMC-2112
no shutdown
But I need also that some other server connected to the same vlan 2112 and having to send HTTP request on the same VIP but this failed and I get dropped conns.
Can anyone helps?
Regards
AbdelazizHi Olivier,
This below the full config, and my need is to make a server in the inside VLAN 2112 (172.22.22.121) to open HTTPS connexion on the VIP (172.22.22.130 for rserver .131 & .132). Trafic from the outside is working well.
Thanx,
Abdealziz
Generating configuration....
access-list BPDU-Allow ethertype permit bpdu
probe tcp HTTPS
port 443
interval 15
passdetect interval 15
passdetect count 1
probe icmp PING
interval 5
rserver host CASHUB131
ip address 172.22.22.131
inservice
rserver host CASHUB132
ip address 172.22.22.132
inservice
serverfarm host SFARM-EXCAS130
probe HTTPS
rserver CASHUB131
inservice
rserver CASHUB132
inservice
parameter-map type connection TCP_IDLE_30min
set timeout inactivity 1800
class-map match-all CLASS-L4-VIP-EXCAS130
2 match virtual-address 172.22.22.130 any
class-map type management match-any REMOTE-ACCESS
description management ACE
10 match protocol telnet any
20 match protocol ssh any
30 match protocol icmp any
31 match protocol https any
32 match protocol snmp any
policy-map type management first-match REMOTE-MGT
class REMOTE-ACCESS
permit
policy-map type loadbalance first-match POLICY-L7-VIP-EXCAS130
class class-default
serverfarm SFARM-EXCAS130
policy-map multi-match POLICY-LB-HMC-2112
class CLASS-L4-VIP-EXCAS130
loadbalance vip inservice
loadbalance policy POLICY-L7-VIP-EXCAS130
loadbalance vip icmp-reply
connection advanced-options TCP_IDLE_30min
interface vlan 2112
bridge-group 1
access-group input BPDU-Allow
service-policy input POLICY-LB-HMC-2112
no shutdown
interface vlan 2122
bridge-group 1
access-group input BPDU-Allow
service-policy input POLICY-LB-HMC-2112
no shutdown
interface bvi 1
ip address 172.22.22.250 255.255.255.0
peer ip address 172.22.22.251 255.255.255.0
no shutdown
ip route 0.0.0.0 0.0.0.0 172.22.22.254 -
I have been trying to load balance ftp through the ACE SM. Active FTP works fine. Passive FTP fails every time and I cannot for the life of me figure out why. The context is routed and I have included the relevant config for the FTP farm. Can anyone advise?
serverfarm host 52-FTP
failaction purge
probe PING
rserver FTP1
inservice
class-map match-all class-52-VIP-FTP
2 match virtual-address x.x.x.52 tcp eq ftp
policy-map type loadbalance first-match policy-52-LB
class class-default
serverfarm 52-FTP
policy-map multi-match policy-inbound-vlan-665
class class-49-VIP
loadbalance vip inservice
loadbalance policy policy-49-LB
loadbalance vip icmp-reply
class class-50-VIP
loadbalance vip inservice
loadbalance policy policy-50
loadbalance vip icmp-reply
class class-52-VIP-FTP
loadbalance vip inservice
loadbalance policy policy-52-LB
loadbalance vip icmp-reply
inspect ftp
interface vlan 665
description inside
ip address 192.168.1.254 255.255.255.0
no normalization
access-group input any-allow
service-policy input policy-REMOTE-MGMT-ALLOW
service-policy input policy-inbound-vlan-665
no shutdownDavid,
It appears you are implementing the ACE in one-arm mode, which will require source nat for client traffic unless your FTP servers have a DFGW of the ACE..
With that said, inorder to statefully load balance PASV FTP you will require some additional lines of configuration since the server will negotate with the client a random high port to connect on you need to create a match any vip (see in RED)
class-map match-any FTP-VIP
2 match virtual-address 10.10.10.10 tcp eq ftp
class-map match-all FTP-NAT
2 match virtual-address 10.10.10.10 any <----required for the passive ftp data connection
policy-map type loadbalance first-match LB_FTP
class class-default
serverfarm REAL_SERVERS
policy-map multi-match VIPS
class FTP-VIP
loadbalance vip inservice
loadbalance policy LB_FTP
loadbalance vip icmp-reply active
inspect ftp
class FTP-NAT
nat dynamic 10 vlan 172 <----- added in order to client nat both the ftp control and data channel
Note, if you add the nat statement you will need to define a nat pool 10 under your vlan 665 interface ( you can just PAT connections and use a single IP or PAT to the VIP address). For example:
interface vlan 665
description inside
ip address 192.168.1.254 255.255.255.0
no normalization
access-group input any-allow
service-policy input policy-REMOTE-MGMT-ALLOW
service-policy input policy-inbound-vlan-665
nat-pool 10 192.168.1.10 192.168.1.10 netmask 255.255.255.0 pat
no shutdown
HTH. -
ACE 4710 Appliance: init: failed to initialize modlock_init(): No such file or directo
Hi,
I have ACE 4710 Appliance, but it is failed and giving following error while login at console.....
I am suspecting hardware issue..most probably with harddrive.... Please let me know if it can be recoverable of only replacement is the solution..
switch login: init: failed to initialize modlock_init(): No such file or directo ry
eth2: ERROR while getting interface flags: No such device
perform_sysmgr_offline: unable to move MTS to MTS_STATE_OFFLINE: Invalid argumen t (error-id 0x801E0016).
init: failed to initialize modlock_init(): No such file or directory
eth2: ERROR while getting interface flags: No such device
perform_sysmgr_offline: unable to move MTS to MTS_STATE_OFFLINE: Invalid argumen t (error-id 0x801E0016).
init: failed to initialize modlock_init(): No such file or directory
eth2: ERROR while getting interface flags: No such device
perform_sysmgr_offline: unable to move MTS to MTS_STATE_OFFLINE: Invalid argumen t (error-id 0x801E0016).
/isan/sbin/sysmgr: symbol lookup error: /isan/lib/libutils.so: undefined symbol: tftp_callback_fn
Regards
NadeemHi,
I RMAed the appliace, i think it was hardware failure which casue this issue.
If some one face this issue please let me know...Thanks!
Regards
Nad -
On editing clips in iMovie10, the drop down menus fail to appear
After an upload to iMovies V10, I tried to edit a clip by using "Enhance" & "Adjust" buttons on the top right side of the edit screen. But, the drop down menu (item 3 in Help for stailisation, etc) didn't appear. The icon changed colour to blue and the image in the top section (like a post card image) advanced slightly in the same frame. On pressing "Enhance", there was only a little one off bit of sound delivered. I tried the following remedies
1) Opened iMovies10 in Admin profile, but same problem;
2) Moved iMovies10 to trash & emptied Trash;&
3) Re-loaded iMovies10.
It made no difference. I'm also using another iMac loaded with V10 with no problems with the drop down menu not appearing. I have spoken to 4 people in Applecare under a case number. They have been supplied with screen shots from both iMacs for illustration purposes.The problem was to be elevated to Apple's Engineering unit for a possible solution. Meanwhile, if anyone has a solution in the community, I would be most apprecoative of any useful ideas to arrive at a fully functioning IMovie10 app. I have not tried to add transitions yet as I can't even do the stabilsation routine!Updated iMovie on 21NOV13 with version 10.0.1. It failed to eliminate the bug mentioned above. Basically means that iMovie continues to be inoperable for me.
-
Secondary ACE dropping connections
Let me start with: I have a failover set of ACE-4710's running FW 4.1. This had occurred on previos versions of the firmware as well.
I have a application (Oracle Forms Based) with a persistent connection required. On LB1 the service runs without issue, as soon as I try to run it on LB2 the connection drops after 30 seconds or so... Any ideas, and further troubleshooting I can do? The configs on both devices seem to be in sync if I do a sh ft group brief on both devices and there are no sync errors.
Thanks in advance.
MattMatt,
Unfortunately nothing in the config jumps out to be an issue. Could you get a sniffer capture from the ACE that has the problem to see who is resetting the connection after 30 seconds?
If you do not have a sniffer available you could try using the ACE capture utility.
You first need to create an ACL to match a test client IP destined to the vip address.
access-list test extended permit ip host client host vip
replace the client IP with the word "client" and the vip you are hitting for "vip"
Once the ACL is built to match the interesting traffric you can build the capture.
you can name the capture anything you want the sample below uses the name "name"
from the #prompt
capture name interface vlan 1000 access-list test
capture name start
Have the cleint hit the vip. You should see the capture data scroll accross the screen at this point. Once you experience the failure you can stop the capture.
capture name stop.
If you need to poll this off in a format that you can open with Wireshark you can issue the command
copy capture name disk0: name
This will create a file on disk0: with the name you give it. From there you can ftp or tftp this file off and open it with wireshark. If you want to send this I will be happy to look at it.
Regards
Jim -
ACE Running S-NAT with IIs 6 logging
Have a customer that is running IIS 6, and needs to source nat traffic from ACE module to those servers. From what I can see, IIS 6 does not support/use “x-forwarded-for” header natively, and require an add-on module. I found only one option (
http://www.winfrasoft.com/X-Forwarded-For.htm#Pricing). Does anyone know of another option/approach to accomplish the same thing? PBR not a viable option, BTW.You can use any name you want in the field you add within http headers in the client's requests, but you'll always have to add a specific field.
-
Hello
I had pix+CSM on 6500. I've changed it to new ACE module on 6500.
I've made loadbalancing which was done on CSM. Now i wanted to connect dmz which was connected to pix and make static DNAT.
I used configuration guide/examples from: http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/v3.00_A1/configuration/security/guide/nat.html
I need to make static DNAT, but i can't figure how it works. There are many errors in this document including incorrect (old?) syntax (for example: nat static 192.0.0.0 255.0.0.0 80 vlan 101)
I analyzed three examples at the and of this document. My questions:
1. how do i choose if it's source or destination NAT ?
2. do i always apply service-policy to vlan interface which receives packets which should be natted ?
3. What is class-map(it's ACL) choosing ? Incoming traffic which destination address should be changed ?
4. is in command: "nat static A netmask netmaskA vlan B" A is outside ip address before translation to inside address ?
5. Could anybody give me a simple example of static DNAT ? (or any links?)
ThanxDestination nat is equivalent to loadbalancing to one server.
I would therefore configure a vip being the inbound destination address, and a rserver which would be the outbound nated destination ip address.
Then create a policy-map to link the 2 together and apply the policy-map to the incoming vlan, or you can apply it globally.
For the reverse connections, where you then need to nat the source ip back to the 'VIP' you use the static nat config that you have found in the document.
By the way, I don't see anything wrong with it.
Those commands are in A1 and also the new A2 release.
ACE is really a loadbalancer with some firewall features and not the opposite.
This is why pure nating functions are not straightfoward to configure.
Gilles. -
ACE: dropped conns due to header insert
My LB is dropping connections on port 443 when I have "insert-http source header-value "%is" configured. Other ports such as 80, or 8080 are working. The config is the same for all ports.
class-map match-any Service_VIP_Class
4 match virtual-address 1.1.1.1 tcp eq https
policy-map type loadbalance first-match Service_L7_Policy
class class-default
serverfarm Service_Serverfarm
insert-http source header-value "%is"
policy-map multi-match Service_LB_Policy
class Service_VIP_Class
loadbalance vip inservice
loadbalance policy Service_L7_Policy
loadbalance vip icmp-reply active
loadbalance vip advertise active
I see dropped conns on the service policy. When I remove the header insertion config, it connects ok.
Please help!There is no way any device (including ACE) can open an https packet to insert anything.
Only exception:
You offload ssl using server keys and certs.Then make changes to the decrypted packet.
Syed -
Hi,
We have a VIP for an FTP service where we do not wish to lose the Client (Source) IP for auditing purposes. So we don't source NAT and force the return traffic back to the ACE with PBR. However the return flow still bypasses the ACE VIP and straight back to the client, diagram below. Is anyone aware of a technique where I can force the ACE to connect the return traffic to the incoming flow?Hi Mark,
Check and compare the config with the example config.
http://docwiki.cisco.com/wiki/FTP_Load_Balancing_on_ACE_in_One-Arm_Mode_Configuration_Example
Avoid the NAT part of the config. Also make sure you are using something like below : ( which is there in the above Doc )
ACE-1/onearm(config)# policy-map multi-match client-vips
ACE-1/onearm(config-pmap)# class slb-vip
ACE-1/onearm(config-pmap-c)# loadbalance vip inservice
ACE-1/onearm(config-pmap-c)# loadbalance policy slb
ACE-1/onearm(config-pmap-c)# inspect ftp <<<<<<< This will make difference
Hope that helps.
regards,
Ajay Kumar -
Idle Oracle DB connection through ACE dropped after 1 hour
Hi folks,
I'm looking for some ideas how to troubleshoot a problem we're having with an Oracle App.
What we are finding is that when a request takes more than an hour for the Oracle DB to process the connection is being dropped.
When wireshark is used at DB server interface we see nothing for an hour and then a single packet RST,ACK with the source identified as the App Server.
We have an App server farm that is behind an ACE module that is in bridging mode. The DB server is on another VLAN so the path the traffic takes is from the App Svr through the ACE from the back to the frontside vlan, through the 6506's MSFC to be routed to the DB server.
Path like this:
Appserver|-->VL203-->|ACE|-->VL202-->|L3 Switch|-->VL200-->|DB Server
If we move the App server to Vlan 202 in front of the ACE the process carries on to completion (after 75 mins).
Is there anything in the ACE settings that could cause the connection to be dropped after an hour for traffic that should simply be being bridged through?
Any suggestions as to where to look next would be appreciated.
TIA
ZacOK Gilles, I'll look at that in the morning. However, this is where it gets interesting.
We have DB servers on two other VLANs routed by the same switch. The connections to those DB servers don't get cut off after an hour (In the connection path I outlined swap VLAN 200 for VLAN 50 or VLAN 205) One of them is even behind the ACE in a different server farm.
Zac -
Kit kat update leads to dropped calls an failed text messages?
I've had a total of 3 replacement devices due to this sad part is verizon has nothing to say about this problem, I heard at other phone companies they will swatch your phone with another of your choice of equal value. In the mean time I pay $200 to get all my calls dropped an have to sit an wait 10 minutes to see that my text failed.
Hello Plude
Start with the article below to troubleshoot issues with the cellular connection. It will walk you through checking for carrier updates and also restoring your iPhone as well.
iPhone: Troubleshooting a cellular data connection
http://support.apple.com/kb/ts3780
Regards,
-Norm G.
Maybe you are looking for
-
Hi, I had a working SSIS package deployment methodoligy (double clicking on the manifest file) and suddently it's not working. I'm running SSIS as administrator and get the error below. TITLE: Package Installation Wizard You need to specify the full
-
Pass iView property as parameter to JSP in PAR file.
Hi All, I have a custom created PAR file in which there is a JSP page there is a variable which I need to provide to users as a configurable property of the iView. So that when a user configures a value to that defined property to the iView, my JSP f
-
Preemptive forms getting in the way!
i want to name the album artist for an album "Ministry Of Sound" however when i type Mi it recognises another artist "MIA" and capitalises the m and i so when i continue to type i get "MInistry Of Sound". any ideas? i don't want to turn it off becaus
-
Why do my loops slow down after many iterations
My Vi seems to freeze up or slow down after several iterations of the loop. This is an Analog Input Capture VI. Attachments: Final_Main_VI_with_working_amp.vi 456 KB
-
Hi all, Windows 2008 R2 SP1 Two HP NICs has been configured to use teaming When one NIC fails, the whole team was down for a while. So, anything we need to check on the switch config? Thank you!