Cisco CSS 11503 Arrowpoint/Load Balance question

Question

I am troubleshooting an issue with my 11503.  I am running version 07.40.0.04. I have it configured as follows:
  content upcadtoa-rule
    add service cadtoa-wls1-e0
    add service cadtoa-wls1-e1
    add service cadtoa-wls2-e0
    add service cadtoa-wls2-e1
    add service cadtoa-wls3-e0
    add service cadtoa-wls3-e1
    add service cadtoa-wls4-e0
    add service cadtoa-wls4-e1
    add service cadtoa-wls5-e0
    add service cadtoa-wls5-e1
    add service cadtoa-wls6-e0
    add service cadtoa-wls6-e1
    arrowpoint-cookie expiration 00:00:15:00
    protocol tcp
    port 8001
    advanced-balance arrowpoint-cookie
    redundant-index 2
    vip address 172.30.194.195 range 2
    arrowpoint-cookie name TOA
    active
However, the load-balancing across the servers does not seem to be doing much balancing.  One of those servers is getting hit with 5 times as much traffic as another and another server is lucky to get a connection at all.  With the cookie expiration set, one would think that this would all balance out over time.
I just came across this information from Cisco and I am wondering if it is relevant:
If you configure a balance or advanced-balance method on a content rule that requires the TCP protocol for Layer 5 (L5) spoofing, you should configure a default URL string, such as url "/*". The addition of the URL string forces the content rule to become an L5 rule and ensures L5 load balancing or stickiness. If you do not configure a default URL string, unexpected results can occur.
In the following configuration example, if you configure a Layer 3 (L3) content rule with an L5 balance method, the CSS performs L5 load balancing, but will reject UDP packets.
content testing
vip address 192.168.128.131
add service s1
balance url
active
The balance url method is an L5 load-balancing method in which the CSS must spoof the connection and examine the HTTP GET content request to perform load balancing. The CSS rejects the UDP packet sent to this rule because a UDP connection cannot be L5. Though the CSS allows this rule configuration, its expected behavior would be more clear if you promote the rule to L5 by configuring the url "/*" command.
In the next example, if you configure an L3 content rule with an L5 advanced-balance method, L5 stickiness will not work as expected.
content testing
vip address 192.168.128.131
add service s1
advanced-balance arrowpoint-cookie
active
The advanced-balance arrowpoint-cookie method causes the CSS to spoof the connection, however, the CSS still marks it as an L3 rule. Thus, the CSS does not insert the generated cookie and the rule defaults to L3 stickiness (sticky-srcip). You must configure a URL like url "/*" to promote this rule to L5, ensuring that L5 stickiness works as expected.
Thanks in advance for any help you can give.  The thing is not down, it is just balancing strangely causing application performance issues.
James

Answer 1

Hey James,
You will need to suspend the content rule in order to add the url statement.  This will cause a quick downtime until the content rule is activated again.  I have shown below the commands to add the statement.  Perhaps you can create your commands in a Notepad file, then paste them all in so they execute quickly to minimize your downtime:
  content MY-SITE
    vip address 10.201.130.140
    port 80
    protocol tcp
    add service MY-SERVER
    active
CSS11503# config t
CSS11503(config)# owner TEST
CSS11503(config-owner[TEST])# content MY-SITE
CSS11503(config-owner-content[TEST-MY-SITE])# url "/*"
%% Attribute may not be modified on active rule
CSS11503(config-owner-content[TEST-MY-SITE])# suspend
CSS11503(config-owner-content[TEST-MY-SITE])# url "/*"
CSS11503(config-owner-content[TEST-MY-SITE])# active
CSS11503(config-owner-content[TEST-MY-SITE])# exit
CSS11503(config-owner[TEST])# exit
CSS11503(config)# exit
CSS11503# show run
  content MY-SITE
    vip address 10.201.130.140
    add service MY-SERVER
    port 80
    protocol tcp
   url "/*"       <--------
    active
Hope this helps,
Sean