Ask the Expert: Configuration and Troubleshooting the Cisco Application Control Engine (ACE) load balancer

Question

With Ajay Kumar and Telmo Pereira 
Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn and ask questions about configuration and troubleshooting the Cisco Application Control Engine (ACE) load balancer with Cisco expert Ajay Kumar and Telmo Pereira. The Cisco ACE Application Control Engine Module for Cisco Catalyst 6500 Series Switches and Cisco 7600 Series Routers is a next-generation load-balancing and application-delivery solution. A member of the Cisco family of Data Center 3.0 solutions, the module: Helps ensure business continuity by increasing application availability Improves business productivity by accelerating application and server performance Reduces data center power, space, and cooling needs through a virtualized architecture Helps lower operational costs associated with application provisioning and scaling
Ajay Kumar  is a customer support engineer in the Cisco Technical Assistance Center in Brussels, covering content delivery network technologies including Cisco Application Control Engine, Cisco Wide Area Application Services, Cisco Content Switching Module, Cisco Content Services Switches, and others. He has been with Cisco for more than four years, working with major customers to help resolve their issues related to content products. He holds DCASI and VCP certifications. 
Telmo Pereira is a customer support engineer in the Cisco Technical Assistance Center in Brussels, where he covers all Cisco content delivery network technologies including Cisco Application Control Engine (ACE), Cisco Wide Area Application Services (WAAS), and Digital Media Suite. He has worked with multiple customers around the globe, helping them solve interesting and often highly complex issues. Pereira has worked in the networking field for more than 7 years. He holds a computer science degree as well as multiple certifications including CCNP, DCASI, DCUCI, and VCP
Remember to use the rating system to let Ajay know if you have received an adequate response.
Ajay and Telmo might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Data Center sub-community discussion forum Application Networking shortly after the event.
This event lasts through July 26, 2013. Visit this forum often to view responses to your questions and the questions of other community members.

Answer 1

Hello Krzysztof,
Another set of good/interesting questions posted. Thanks! 
I will try to clarify your doubts.
In the output below both resources (proxy-connections and ssl-connections rate) are configured with a min percentage of resources (column Min), while 'Max' is set to equal to the min.
ACE/Context# show resource usage
                                                     Allocation
        Resource         Current       Peak        Min        Max       Denied
-- outputs omitted for brevity --
  proxy-connections             0      16358      16358      16358      17872
  ssl-connections rate          0        626        626        626      23204
Most columns are self explanatory, 'Current' is current usage, 'Peak' is the maximum value reached, and the most important counter to monitor 'Denied' represents the amount of packets denied/dropped due to exceeding the configured limits.
On the resources themselves, Proxy-connections is simply the amount of proxied connections, in other words all connections handled at layer 7 (SSL connections are proxied, as are any connections with layer 7 load balance policies, or inspection).
So in this particular case for the proxy-connections we see that Peak is equal to the Max allocated, and as we have denies we can conclude that you have surpassed the limits for this resource. We see there were 17872 connections dropped due to that.
ssl-connections rate should be read in the same manner, however all values for this resource are in bytes/s, except for Denied counter, that is simply the amount of packets that were dropped due to exceeding this resource. 
For your particular tests you have allocated a min percentage and set max equal to min, this way you make sure that this context will not use any other additional resources.
If you had set the max to unlimited during resource allocation, ACE would be allowed to use additional resources on top of those guaranteed, if those resources were available.
This might sound a great idea, but resource planning on ACE should be done carefully to avoid any sort of oversubscription, specially if you have business critical contexts.
We have a good reference for ACE resource planning that contains also description of all resources (this will help to understand the output better):
http://www.cisco.com/en/US/docs/interfaces_modules/services_modules/ace/v3.00_A2/configuration/virtualization/guide/config.html#wp1008224
1) When a resource is utilized to its maximum limit, the ACE denies additional requests made by any context for that resource. In other words, the action is to Drop. ACE  should in theory silently drop (No RST is sent back to the client). So unless we changed something on the code, this is what you should see.
To give more context, seeing resets with SSL connections is not necessarily synonym of drops. As it is usual to see them during normal transactions.
For instance Microsoft servers are usually ungracefully terminating SSL connections with RESET. Also when there is renegotiation during an SSL transaction you may see RESETS, but this will pass unnoticed for end users. 
2)  ACE will simply drop/ignore new connections when we reach the maximum amount of proxied connections for that context. Exisiting connections will continue there.
As ACE doesn't respond back, client would simply retransmit, and if he is lucky maybe in the next attempt he will be able to establish the connection.
To overcome the denies, you will definitely have to increase the resource allocation. This of course, assuming you are not reaching any physical limit of the box.
As mentioned setting max as unlimited might work for you, assuming there are a lot of unused resources on the box.
3)  If a new connection comes in with a sticky value, that matches the sticky entry of a real server, which is already in MAXCONNS state, then both the ACE module/appliance should reject the connection and that sticky entry would be removed.
The client would at that point reestablish a new connection and ACE would associate a new sticky entry with the flow for a new RSERVER after the loadbalancing decision.
I hope this makes things clearer! Uff...
Regards,
Telmo