Cisco ace Load balancer not maintaining session persistence

Hi All,
We have observed from the IIS logs on the internal webservers that loadbalancer is not maintaining session persistence for two specific request for the internal servers.
https://123.xyz.com/Webresource.axd
https://123.xyz.com/ScriptResource.axd
Error
Webresource.axd : 500
Scriptresource.axd: 404
Session persistence is maintained for all other requests hitting loadbalancer.
Issue is observerd on hits for these two specified components. WebResource.axd and ScriptResource.axd are Http Handlers used by ASP.NET and Ajax to add client-side scripting to the outgoing web page.
For e.g /WebResource.axd d=t2GXfySdqWmJ-lZSI0KVbw2&t=634868473645172160 is valid for server 1 and return 200 response but the same request is seen on few other servers where the response is 404 even though load balancer cookie is same. This means that if the request for the both the axd contains a valid decrypter and it connects to the right server then the response seen is 200.
The url passed by the user contains d and t parameters when are unique for each user session.
Solution tried:
Accessed website via another VIP without http redirect rule but could not see difference.
Tried to match machine key across all servers : Failed . Could see the ‘d’ value different for each server.
Load balancer VIP :
x.x.x.x
redirect: http > https
SSL Offload : ON
Poool:
WEB1
WEB2
WEB3
WEB4
WEB5
All servers listening on port 80
sticky config:
sticky ihttp-cookie cookie1 vip-1.1.1.1-80-stickyfarm
cookie insert browser-expire
replicate sticky
serverfarm vip-1.1.1.1_80
sticky http-cookie cookie1 vip-farm:1.1.1.1:443
cookie insert browser-expire
replicate sticky
serverfarm farm:1.1.1.1:443
Has anyone else come across similar issue?
Can you plese check if there is any config on cisco ace that will ensure that session persistence is maintained for these 2 requests.
Thank you for all the help.
regards,
Sangram

Hello Sangram,
We would need simultanous packet traces before and after the ACE to get to the root cause of this issue so I would recommend that you open a cisco tac case for more in depth troubleshooing of this issue.
Joel Lamousnery
CCIE R&S - 36768
Engineer, Customer Support
Technical Services

Similar Messages

Two isp load balancing on cisco ACE(load balancer)

I don't know much about load balancer(ACE).
Is this is possible to load balance two isp's on load balancer (ACE). If so, how i can do so , any configuration example, or cisco document.

Wrong forum, post in "Datacenter". You can move your posting with the Actions panel on the right.

Ask the Expert: Configuration and Troubleshooting the Cisco Application Control Engine (ACE) load balancer

With Ajay Kumar and Telmo Pereira
Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn and ask questions about configuration and troubleshooting the Cisco Application Control Engine (ACE) load balancer with Cisco expert Ajay Kumar and Telmo Pereira. The Cisco ACE Application Control Engine Module for Cisco Catalyst 6500 Series Switches and Cisco 7600 Series Routers is a next-generation load-balancing and application-delivery solution. A member of the Cisco family of Data Center 3.0 solutions, the module: Helps ensure business continuity by increasing application availability Improves business productivity by accelerating application and server performance Reduces data center power, space, and cooling needs through a virtualized architecture Helps lower operational costs associated with application provisioning and scaling
Ajay Kumar is a customer support engineer in the Cisco Technical Assistance Center in Brussels, covering content delivery network technologies including Cisco Application Control Engine, Cisco Wide Area Application Services, Cisco Content Switching Module, Cisco Content Services Switches, and others. He has been with Cisco for more than four years, working with major customers to help resolve their issues related to content products. He holds DCASI and VCP certifications.
Telmo Pereira is a customer support engineer in the Cisco Technical Assistance Center in Brussels, where he covers all Cisco content delivery network technologies including Cisco Application Control Engine (ACE), Cisco Wide Area Application Services (WAAS), and Digital Media Suite. He has worked with multiple customers around the globe, helping them solve interesting and often highly complex issues. Pereira has worked in the networking field for more than 7 years. He holds a computer science degree as well as multiple certifications including CCNP, DCASI, DCUCI, and VCP
Remember to use the rating system to let Ajay know if you have received an adequate response.
Ajay and Telmo might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Data Center sub-community discussion forum Application Networking shortly after the event.
This event lasts through July 26, 2013. Visit this forum often to view responses to your questions and the questions of other community members.

Hello Krzysztof,
Another set of good/interesting questions posted. Thanks!
I will try to clarify your doubts.
In the output below both resources (proxy-connections and ssl-connections rate) are configured with a min percentage of resources (column Min), while 'Max' is set to equal to the min.
ACE/Context# show resource usage
                                                     Allocation
        Resource         Current       Peak        Min        Max       Denied
-- outputs omitted for brevity --
proxy-connections             0      16358      16358      16358      17872
ssl-connections rate          0        626        626        626      23204
Most columns are self explanatory, 'Current' is current usage, 'Peak' is the maximum value reached, and the most important counter to monitor 'Denied' represents the amount of packets denied/dropped due to exceeding the configured limits.
On the resources themselves, Proxy-connections is simply the amount of proxied connections, in other words all connections handled at layer 7 (SSL connections are proxied, as are any connections with layer 7 load balance policies, or inspection).
So in this particular case for the proxy-connections we see that Peak is equal to the Max allocated, and as we have denies we can conclude that you have surpassed the limits for this resource. We see there were 17872 connections dropped due to that.
ssl-connections rate should be read in the same manner, however all values for this resource are in bytes/s, except for Denied counter, that is simply the amount of packets that were dropped due to exceeding this resource.
For your particular tests you have allocated a min percentage and set max equal to min, this way you make sure that this context will not use any other additional resources.
If you had set the max to unlimited during resource allocation, ACE would be allowed to use additional resources on top of those guaranteed, if those resources were available.
This might sound a great idea, but resource planning on ACE should be done carefully to avoid any sort of oversubscription, specially if you have business critical contexts.
We have a good reference for ACE resource planning that contains also description of all resources (this will help to understand the output better):
http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/v3.00_A2/configuration/virtualization/guide/config.html#wp1008224
1) When a resource is utilized to its maximum limit, the ACE denies additional requests made by any context for that resource. In other words, the action is to Drop. ACE should in theory silently drop (No RST is sent back to the client). So unless we changed something on the code, this is what you should see.
To give more context, seeing resets with SSL connections is not necessarily synonym of drops. As it is usual to see them during normal transactions.
For instance Microsoft servers are usually ungracefully terminating SSL connections with RESET. Also when there is renegotiation during an SSL transaction you may see RESETS, but this will pass unnoticed for end users.
2) ACE will simply drop/ignore new connections when we reach the maximum amount of proxied connections for that context. Exisiting connections will continue there.
As ACE doesn't respond back, client would simply retransmit, and if he is lucky maybe in the next attempt he will be able to establish the connection.
To overcome the denies, you will definitely have to increase the resource allocation. This of course, assuming you are not reaching any physical limit of the box.
As mentioned setting max as unlimited might work for you, assuming there are a lot of unused resources on the box.
3) If a new connection comes in with a sticky value, that matches the sticky entry of a real server, which is already in MAXCONNS state, then both the ACE module/appliance should reject the connection and that sticky entry would be removed.
The client would at that point reestablish a new connection and ACE would associate a new sticky entry with the flow for a new RSERVER after the loadbalancing decision.
I hope this makes things clearer! Uff...
Regards,
Telmo

ACE - Load Balance insert cookie method for https

I am trying to load balance between 2 web servers using the cookie insert method by ACE for achieving the session persistence. The servers are not inserting any cookie. It works fine for the http connections but when trying with https connection it is not working.
Can anyone help me with this please.
Is it that ACE cookie insert method of session persistence will not work with https connections.

Hi,
1. for https you can use src ip as sticky (mega proxy problem).
2. you can terminate ssl connection on ace (ssl between client and ace only, between ace and server it's clear) and you can use any L7 sticky (for example cookie)
3. if you need ssl terminate up to real server, you can first terminate ssl between client and ace on ace, then use L7 sticky and after then terminate second ssl to real server.
in other words, if you don't decrypt ssl on ace, you can use only L2/3 data for sticky (or ssl id for ssl v2.0)
martin

Is it possible to use UCS Blade Servers in ACE Load Balancing

Hi all ,
Is it possible to use UCS Blade Servers in ACE Load Balancing ?? Please note that UCS Blade Servers are not connected directly to 6500 Switch where ACE Module installed .i am expecting a good suggestion from whether ACE or Switching Expert
Thanks in advance
Sanjeevi

There is nothing that would prevent you from loadbalancing the applications that run on UCS servers. ACE can loadbalance applications that are directly L2 attached (bridged or routed mode) or even servers that are multiple hops L3 hops away using one-armed mode with source nat. The key to this is that the return traffic from the server needs to make it back to the ACE.

ACE load balancing and testing using soapUI

Hey, I am trying to crowd source a solution for this problem.
A client is testing using soapUI to an application that is being load balanced via ACE. There are two webservers behind the VIP servicing the client request. When client tests, requests are timing out per the soapUI log. A packet capture was taken and it clearly shows that ACE is not forwarding the HTTP data back to the client. When client tests by bypassing the ACE load balancer, it works fine. But, there are other clients from other applications that are making successful connection to the load balanced application via the VIP.
Question, is there any thing unique with making HTTP/XML based requests using soapUI? LB configuration is shown below:
class-map match-all EAI_PWS_9083
2 match virtual-address 10.5.68.29 tcp eq 9083
serverfarm host EAI_PWS_9083
description WebSphere Porduction
failaction purge
probe tcp9083
rserver ESSWSPAPP01 9083
    inservice
rserver ESSWSPAPP02 9083
    inservice
policy-map type loadbalance first-match L7_POLICY_EAI_PWS_9083
class class-default
    serverfarm EAI_PWS_9083
policy-map multi-match L4SLBPOLICY
class EAI_PWS_9083
    loadbalance vip inservice
    loadbalance policy L7_POLICY_EAI_PWS_9083
    loadbalance vip icmp-reply active
    appl-parameter http advanced-options CASE_PARAM
parameter-map type http CASE_PARAM
case-insensitive

Hi,
Your configuration looks fine. I am not familiar with soapUI but if it is like a normal TCP connection followed by HTTP requests, i don't see why this shouldn't work.
Do you know if there is a difference while using soapUI and normal request using browser?
Regards,
Kanwal

ACE - Load Balance SMB?

Can the ACE load balance SMB?
Server 1 DNS is msserver1
Server 2 DNS is msserver2
VIP DNS is msserver
Can the ACE replace the server name (or IP address) in a tree connect query with the actual real server name that is chosen for the request?

Hi , If I understood you correctly and you're looking for intelligent way to loadbalance NetBios/Samba - I'm afraid there is no such functionality on ACE, we can only do simple L4 loadbalancing for such sessions and can't change anything.

Load balancing and HA for persistence chat pool

Hi,
I have 12000 users for lync 2013 . plan to have 2 persistence chat servers in a pool , have below queries regarding the same.
1. which type of load balancing will work for persistence chat pool.... DNS or HLB? if its HLB then is it mandatory to use HLB , agenda is to minimize the cost.
2. there is already 2 HLB for FE pool , one for external service and 1 for internal service.... can i use the same HLB for persistence chat pool?
3. does persistence chat pool have HA bydefault like FE and Edge pool?

1) DNS LB is fine: You can see them doing it here:http://technet.microsoft.com/en-us/library/jj205391.aspx
2) I suppose you could, but I'd suggest sticking with DNS as it's easy, supported, and it works.
3) Yes. I believe so.
Please remember, if you see a post that helped you please click "Vote As Helpful" and if it answered your question please click "Mark As Answer".
SWC Unified Communications

Need help with ACE Load Balancing Base on URL pattern

This is the first time for me trying to configure something like this on the ACE load balancer. I need help configuring a load balancing policy base on URL pattern. URL https://ineedhelp.com base on /willuhelpme and /imlost
Key: ineedhelp_key
cert: ineedhelp_cert
serverfarmA
serverA 10.1.1.1 443
serverfarmB
serverB 10.1.1.2 443
ineedhelp.com/willuhelpme-------serverfarmA
ineedhelp.ocm/imlost---------------serverfarmB

This is the first time for me trying to configure something like this on the ACE load balancer. I need help configuring a load balancing policy base on URL pattern. URL https://ineedhelp.com base on /willuhelpme and /imlost
Key: ineedhelp_key
cert: ineedhelp_cert
serverfarmA
serverA 10.1.1.1 443
serverfarmB
serverB 10.1.1.2 443
ineedhelp.com/willuhelpme-------serverfarmA
ineedhelp.ocm/imlost---------------serverfarmB

Ace load balancing, inservice/no inservice serverfarms

I've started working with an ACE load balancer and came across something that just didn't add up to me. I can pull and put servers in and out of rotation without a problem however when working with a serverfarm or a group of servers I have to pull each one individually and can't find a way to remove say the entire serverfarm via one command. Does anyone know of a way to put a serverfarm 'inservice' or set it to 'no inservice' that would make it easier for large groups of servers needing to be adjusted.
Sorry if this isn't the write forum for this kind of question. Please feel free to move it if needed.

Hello Chris,
There is no toggle to set every rserver under a serverfarm out of service. You can only take a single rserver out of service at a global level, or under a serverfarm inividually.
One thing to think about - bringing down all of the servers would be the same as removing the serverfarm from under the policy map type loadbalance since it would effectively bring the vip down.
Regards,
Chris Higgins

RV042 Load Balancing not working correctly?

We have an RV042 on firmware version 1.3.13.02 and 2 ISPs:
WAN1 = Telepacific T1
1.5Mbps down and 1.5Mbps up
WAN2 = AT&T U-Verse
12Mbps down and 5Mbps up
I have it set to Load Balance, Primary WAN = WAN2
Network Service Detection enabled, only pinging the Remote Host of 4.2.2.2 and set to Generate Log
Bandwidth is set to:
WAN1 = 1000Kbps upstream & downstream
WAN2 = 5040Kbps upstream & 12000Kbps downstream
It seems to pick WAN1 a lot of the time. Do I have something setup wrong?

Ok, so what do you recommend? To setup binding for ports important to direct over say… WAN1 (my fast access) shall I also keep the bandwidth management set up as is, example:
For me it’s confusing because this config appears to be clear to me that has to control traffic the way I want .
Thanks for your response.
H Aragon
De: jasbryan
Enviado el: lunes, 20 de febrero de 2012 03:18 p.m.
Para: HECTOR MANUEL ARAGON
Asunto: - Re: RV042 Load Balancing not working correctly?
Home
Re: RV042 Load Balancing not working correctly?
created by jasbryan in Small Business Routers - View the full discussion

TACACS and Cisco ACE Load Balancers authentication ?

Is there a need to have user accounts locally on the Cisco ACE Load Balancers as well as the User accounts on TACACS where it is being authenticated ?
Many thanks
Florrie

Yes.
http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_services/ace_appliances/vA3_1_0/configuration/security/guide/aaa.html#wpmkr1517596

Session replication in oc4j load balancing not working ..

Hi All,
I have windows 2000 machine. I have installed 2 instances of oc4j running on ports 8888 and 8889. I started the loadbalancer.jar in the first instance, started the first oc4j instance and then started the second oc4j instance. I have a common application deployed on both instance1 and instance2 and that is nothing but out famous SessionServlet.
If I access this servlet using http://localhost:80/app/servlet/SessionServlet then I am getting a count as 1 . My loadbalancer that is started from first oc4j instance(running on port 8888) is showing that the request is routed to the first instance. I stopped my first instance1 and then again from the same browser/session/client if I access the same servlet using http://localhost:80/app/servlet/SessionServlet then still I am seeing the count as 1 instead of 2 . At this point my loadbalancer is showing that the request is routed to the second oc4j instance(running on port 8889) since first instance is stopped. So why am I seeing the count as 1 instead as 2.
Also,
1. Is it enough that we start the loadbalancer.jar in the first oc4j instance. What about the loadbalancer.jar in the second oc4j instance ?
2. We all know that Apache HTTP Server runs on port 80. But since I didnt[i]Long postings are being truncated to ~1 kB at this time.

thank you debu, I have one doubt . In the clustering/load-balancing documentation at metalink(doc id: 151717.1) it is said that in point 4b that we should add the tag <cluster-config /> to orion-web.xml file but this file will be created only after the web application is deployed and it is accessed atlest for one time. So is it that we should first deploy the web application and then access it for atlest one time then stop the server and add this tag .. or is there any other way workaround ?

ACE Load Balancing Configuration For NATed User Traffic

Hello,
I am currently working on a requirement where the shared application services will be hosted in DC and these services will be accessed by multiple (thousands) users from different corporates/customers. The user traffic will be hidden behind customer's proxy servers or firewalls so the load balancer (ACE modules) services hosted in DC will not be able to see requests coming in from induvidual users IP addresses.
In this scenario what are options of load balancing are available in Lyer3/4 and Layer7 ?
Thanks in advance for your help.
Sanjay

Hi Sanjay,
In a set up where all users are coming from behind a proxy, all users will be loadbalanced to same server thus overloading it. This is when you are doing standard L3/L4 LB.
In the situation of proxies, for HTTP applications you shall use L7 LB and use information(cookie) in HTTP client request or server response. The ace will use this information to stick the user to same server for persistence. If a client comes with no cookie it will be loadbalanced according to the predictor method configured. Below is the link for L7 configuration example and other TS steps you can take while configuring L7 policies on ACE. For more informatin i would suggest reading ACE user guide too.
http://docwiki.cisco.com/wiki/Cisco_Application_Control_Engine_(ACE)_Troubleshooting_Guide_-_Troubleshooting_Layer_7_Load_Balancing
If you have any questions please feel free to ask.
Regards,
Kanwal

ACE LOAD BALANCER - secure tls renegotiation

I have a cisco ace loadbalancer and a server farm behind it.
We have implemented sll-to-ssl termination, but we are facing certain problems with opera browser and android mobiles.
On both we get "The server does not support secure TLS renegoriation...."
Running the following: openssl s_client -connect aaa.bbb.ccc.ddd:443
On the load balancer we get:
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 1024 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol : TLSv1
    Cipher    : AES256-SHA
    Session-ID:
    Session-ID-ctx:
    Master-Key: xxxxxxxxx
    Key-Arg   : None
    Krb5 Principal: None
    Start Time: 1323349587
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
On one of the servers from the farm we get:
ew, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: yyyyyyy
    Session-ID-ctx:
    Master-Key: xxxxxxxx
    Key-Arg   : None
    Krb5 Principal: None
    Start Time: 1323349689
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
Is there any connection to our problem with this outputs ?
Does anyone have any idea on how to solve this problem ?
Thanks in advance

Hi Thanassis,
TLS renegotiation was disabled in all Cisco devices due to a vulnerability of the protocol. Check
http://www.cisco.com/en/US/products/products_security_advisory09186a0080b01d1d.shtml for more details
Since the renegotiation was disabled for security reasons, there is no way to enable it back, so you should rather be looking for a way to force your browsers not to require this option to be enabled. I would suggest you to contact the Opera support team.
Regards
Daniel

Cisco ace Load balancer not maintaining session persistence

Similar Messages

Maybe you are looking for