CSS SSL L5 balancing

Hello
I have four servers that I want to load balance based on a URL both HTTP and HTTPS. Two are tomcat and two are IIS and I would like to use something like /jsp/* and /aspx/*. I can get the http L5 rules setup just fine but when I try and use port 443 with a layer 5 content rule I get nothing. The show flows command shows the external ip, the vip but 0.0.0.0 for the NAT IP. Is it possible to do what I'm trying to do?
my config is
service iis1
ip address 10.0.0.1
active
service iis2
ip address 10.0.0.2
active
service tomcat1
ip address 10.0.0.3
active
service tomcat2
ip address 10.0.0.4
active
owner test
content iis
vip address 10.1.1.1
url "/aspx/*"
advanced-balance arrowpoint-cookie
add service iis1
add service iis2
protocol tcp
port 80
active
content iis_ssl
vip address 10.1.1.1
url "/aspx/*"
advanced-balance ssl
application ssl
add service iis1
add service iis2
protocol tcp
port 443
active
Thanks in advance
Justin

Thanks for the response Giles. I've been working on doing that and I think I have it working but the problem now is that we have some apps that look to make sure the conversation is secure and redirect if not. With the SSL module, it doesn't look like the servers will ever see whether or not the user is connecting via HTTPS. Is there any way around that?

Similar Messages

SSL Load Balancing (Java applets problem)

Hi,
I have implemented loadbalancing of two webservers using CSS 11503.
Servers are containing SSL pages which need to be loadbalanced.
I have tried configuring the content rule for ssl using
port 443
application ssl
advanced-balance ssl
commands.The site opens properly using the VIP address (checked the SSL certificate issued by one of the servers. ) and I am able to see all the TABS on the same.The real problem starts when I am trying to upload a file to the server using the VIP address.
The moment I try to upload a file the session shifts to the second server and prompts for new certificate issued by the second server.
One more thing which I would like to mention here is that while uploading the file, JAVA is used.
i.e.the files are being uploaded using JAVA applets.
If anybody has encountered this kind of problem kindly suggest on the same.
Would appreciate if you send the solution on [email protected]
Any configs needed plz contact me on the above e-mail ID.
Thanks,
Pankaj P.

HI Pankaj,
depending on your Java applett it might be possible that the applett opens a new connection. therefore depending on your total configuration it might be possible taht another server is used.Even worse if you do SSL-offloading the applett might tell the user to do http instead of https. I suggest that you check with a sniffertrace what is happening:
1) is there a new connection setup while the applett runs
2) is it again http or https and if https is it a new https session which will be again balanced not depending on the original https session.
Hope that helps
regards,
Joerg

Random failures to CSS doing https balancing.

So I have a cluster of about 10 machines behind a 11503, each server is setup like
service server-1
ip address 192.168.10.171
port 443
string cluster01
keepalive type script ap-kal-httplist "192.168.10.171 /webct/about.jsp"
keepalive frequency 15
active
and clustered in a service via
content ssl-rule
balance leastconn
protocol tcp
port 443
advanced-balance sticky-srcip-dstport
vip address 192.168.200.19
add service server-1
add service server-2
add service server-3
add service server-9
add service server-10
active
I am not currently doing ssl termination, just balancing.
Ok, so recently the load has started to rise (it is an e-learning application for a university and it's finals time) and now I see a scenario where random users are unable to connect to the https://elearningapp.somedomain.ca URL, while the person sitting next to them (both physically and IP-wise) connects fine. It is only a percentage of users who see this, seemingly no correlation between them, and if I reset the css it goes away for a while.

You'll need to collect some info.
First, capture a sniffer trace on one of the host showing the problem.
Check if the client gets a response to the SYN.
Check if the client can ping the CSS.
Then verify that the SYN comes to the CSS.
[capture a sniffer trace in front of CSS].
Then use 'sho flows x.x.x.x' to see if a flow is created.
Verify if the SYN is forwarded to a server.
Could be the server not responding.
What version do you run ?
Gilles.

How many CSS SSL certificates needed?

From reading the CSS SSL Configuration Guide, it seems that one certificate is needed for each virtual SSL server (or VIP), regardless of how many servers are being load-balanced behind that VIP, but that is not made very clear. Also, it appears that a separate certificate is required for each virtual SSL server. Can someone please confirm or correct this for me? Thank You.

A quick (I hope) follow-up question on this...
Given multiple domain names being load-balanced by a CSS with a single SSL module, would I need different key and cert associations? I am thinking of something like this:
ssl associate rsakey prodkey prodkey.pem
ssl associate cert prodcert prodcert.pem
ssl associate dhparam proddh proddh.pem
ssl associate rsakey intkey intkey.pem
ssl associate cert intcert intcert.pem
ssl associate dhparam intdh intdh.pem

Using a single CSS to load balance multiple services

Is it possible to use a single CSS to load balance 3 different services (server farm) ? That mean the CSS need to advertise 3 VIP
I'm thinking of two scenarios:
1 - configure the CSS to use 4 interfaces: 1 to public, 3 to private (each interface will plug-in to a different vlan/server farm)
2 - configure the CSS to use 2 interfaces: 1 to public, 1 to private (all 3 server farms are in the same vlan)
Will both scenarios work ?
Thanks
--Phillip.

Hi Phillip,
both scenarios will work. One CSS can certainly manage more than 3 services! You can even use just one VIP for all traffic, then just create the proper rules to send specific traffic to the corresponding service(s). No need for 3 VIPs.
Regards
-juerg

CSS/SSL termination - cypher negotiation Q

Hi everyone
question regarding SSL termination on CSS/SSL module.
I have several several cyphers in my ssl-proxy list,
What is the algorithm to choose the cypher ?
I may assume that CSS and browser negotiate it during SSL session establishing.
The testing shows that same browser gets different cyphers when it hits
different CSSs (cyphers are in the same order in proxy-lists on CSSs)
Thanks
Alex

Alex,
it's not really an algorithm.
The browser selects the first cipher that matches its requirements in the list presented by the server/CSS.
The CSS builds a list in the order of weight.
If you did not specify any weight, the list can be random depending in which order you entered the command.
I would say, if you want a specific cipher to be selected, use a highest weight for this cipher.
Gilles.

CSS 11503 Load Balancing Verification

Alright, so I have toiled long and hard to get this right. I think I have the config down but I am unsure on how to verify how this load balancing is working.
Here is the Content Config that I am speaking of:
content cad-rule
    add service wls1-e0
    add service wls1-e1
    add service wls2-e0
    add service wls2-e1
    add service wls3-e0
    add service wls3-e1
    add service wls4-e0
    add service wls4-e1
    add service wls5-e0
    add service wls5-e1
    add service wls6-e0
    add service wls6-e1
    arrowpoint-cookie expiration 00:00:15:00
    advanced-balance arrowpoint-cookie
    redundant-index 2
    vip address 172.30.194.195 range 2
    arrowpoint-cookie name TOQ
    protocol tcp
    port 8001
    url "/*"
    active
Each service in the rule above is configured as follows:
service wls1-e1
port 8001
protocol tcp
strin ags001-e1
ip address 172.30.193.81
keepalive type http
keepalive uri "/cad/index.html"
redundant-index 12
keepalive frequency 20
keepalive maxfailure 10
keepalive retryperiod 2
active
I am using the advanced arrowpoint cookies because I need some stickiness here. Straight round-robin would not have done what I needed it to do.
Now, when I go to my show summary, this is what I see for this rule:
                 cad-rule    Master   wls1-e0 84274
                                            wls1-e1 13144
                                            wls2-e0 96884
                                            wls2-e1 26374
                                            wls3-e0 71145
                                            wls3-e1 16592
                                            wls4-e0 76403
                                            wls4-e1 8657
                                            wls5-e0 118623
                                            wls5-e1 22760
                                            wls6-e0 30836
                                            wls6-e1 20464
The far right column indicates the services hits. I originally had the E1's suspended and activated them later on. So if this was true round robin, all the E0's should have the same number of service hits and all the E1's should have the same number of service hits. But as you can see, the wls5 server is getting hit the most while the wls6 server is sitting there twiddling its thumbs.
Now understanding how the arrowpoint cookies do their load balancing (inserting a cooking into the flow and then timing out after 15 mins as configured above) I would not expect a 1:1 ratio of load balancing between servers. But the distribution above seems rather extreme.
Does anyone have any suggestions on how to both A) verify that this is the right config and B) suggest to my boss that this is working the way it should be working?
Thanks!
James

Hi James,
There are several reasons of the uneven load balancing that you are seeing (based on the show summary). First
of all, the CSS is configured to do stickiness (advance-balance).
With arrowpoint-cookies (for HTTP only) method for stickiness, only the requests coming with the same cookie
are going to get stuck to the same server, since the cookie is
lost when the browser is closed (or based on the expiration), then the stickiness is going to be session
based and if the same client open a new session is going to be load balanced.
Is important to understand that when using stickiness, no real even load balancing is
going to happen since we are sticking new flows to the same server; even when layer 5 stickiness would
permit more even balancing than layer 3 stickiness (source IP based).
Also consider that the "show summary" is a command to see the hits (requests) being balanced to an specific
server, this is a good command to see the load balancing, anyway since the CSS balance
connections (flows), a persistent connection could have a lot of requests, so all those requests are
always going to the same server (incrementing the amount of hits in the counter) while a non-persistent
connection would be just one request (refer to HTTP persistence).
Also keep in mind that if a service is take out for maintenance, or is added to the load balancing later
than another, or if goes down for a period of time, then the CSS will be balancing among the remaining alive
servers. When you add the server again, the another servers are going to have connections
already established, so since the CSS is doing round robin, the server last added will
never have the same amount of connections (nor hits) that the other ones, because while one could
have 55 for example, the new one will have it first connection, and when the first one
gets the 56, the another will get the second, and so on.
Please let me know if this makes any sense.
Diego M

Newly Occuring CSS SSL Issue in Chrome, FF10, IE9 with L5 rules; 3 second delay, loss of L5 stickyness

We recently started suffering an issue with our CSS11501S-K9 units not performing URL stickiness on our SSL wrapped L5 rules. I've spent dozens of manhours working on the problem, and have quite a bit of information to report, including a solution. There is a high probability that anybody who uses SSL to an L5 rule on a CSS unit will become affected by this problem over the next few weeks/months as users update their browsers with new SSL patches.
We hadn't made any changes to our config in months, and eliminated hardware problems by testing a second unit.
Here are the exact symptoms we saw:
Browsers affected: Firefox 10, Chrome, IE9, others (and some earlier versions of IE depending on patch levels)
Browsers not affected: FireFox 3.5, w3m 0.5.2, curl7.19.7
Impact 1: For SSL Rules backed by L5 rules, the initial response to the first request would be 3 seconds. Further requests on the same TCP connection would not be delayed
Impact 2: L5 rules being accessed via SSL would nolonger perform any URL based stickiness. Accessing the same rule skipping SSL, would work fine
I focused on the 3 second delay, since that was a new issue and was easier to debug than monitoring multiple servers to see if stickiness was broken. This is what I found when a client tries to connect to an SSL rule that ultimately is routed to a L5 HTTP rule:
1. Client/CSS perform initial TLS handshake, crypto cyphers determined (nearly instantly)
2. Client sends HTTP 1.1 request for resource (nearly instantly)
3. 3 seconds of no traffic in our out of the CSS related to this request
4. CSS opens an HTTP connection to backend webserver, backend webserver responds (nearly instantly)
5. The CSS seems to route to the backend server using the balance method (round-robin) instead of the advanced-balance method (url)
6. Response is sent to the client with the resource (nearly instantly)
7. Future requests sent from the browser on the same TCP connection have no delay, but the advanced-balance continues to be ignored
The 3 seconds is quite an exact figure (within a few milliseconds) and appears to be entirely happening inside of the CSS unit itself, since it does not connect to the backend server until after the 3 seconds elapse. 3 seconds smelled like some sort of internal timeout set in the CSS unit after it gives up waiting for something.
Looking at the packets from affected browsers I discovered that the GET /foobar HTTP/1.1 request was being broken into two separate TLSv1 application messages, the first was 24 bytes and the second was 400 bytes. Decrypting these messages I found the first message was a
G
and the second message was:
ET /foobar HTTP/1.1
This essentially splits the initial request the client is sending into two pieces. This confuses wireshark so much, it doesn't decode this as a HTTP request, and just decodes it as "continuation or non-HTTP traffic".
On the working browsers I saw only one TLSv1 application message, decrypting it I saw:
GET /foobar HTTP/1.1
(obviously I'm simplifying the contents of the request, there were lots of headers and stuff)
I am aware that the CSS can't handle L5 rules appropriately if they get fragmented, so I suspected this was the problem. I pulled a packet trace from a few years ago, and at that time confirmed we never saw a double TLSv1 application messages before.
A number of openssl vulnerabilities were recently fixed: http://www.ubuntu.com/usn/usn-1357-1
and browsers may have been recently updated to fix some of these issues, changing the way they encode their traffic.
Solution:
Our ssl config looked something like this:
ssl-proxy-list SSL_ACCEL
ssl-server 10 vip address XX.XX.XX.XX
ssl-server 10 rsakey XXXX
ssl-server 10 cipher rsa-with-3des-ede-cbc-sha XX.XX.XX.XX 80
ssl-server 10 cipher rsa-with-rc4-128-sha XX.XX.XX.XX 80
ssl-server 10 cipher rsa-with-rc4-128-md5 XX.XX.XX.XX 80
ssl-server 10 unclean-shutdown
ssl-server 10 rsacert XXXXXX
Removing:
ssl-server 10 cipher rsa-with-3des-ede-cbc-sha XX.XX.XX.XX 80
Solves the problem. After that's removed, the browsers will nolonger fragment the first character of their request into a separate TLSv1 message. The 3 second delay goes away, and L5 stickiness is fixed. The "CBC" in the cyper refers to Cypher-Block-Chaining (a great article here:
http://en.wikipedia.org/wiki/Cipher-block_chaining), and breaking the payload into multiple packages may have been an attempt to initialize the IV for encryption -- although I'm really just guessing, I stopped researching once I verified this solution was acceptable.
This issue became serious enough for us to notice first on Monday Feb 13th 2012. We believe a number of our large customers distributed workstation updates over the weekend. The customers affected were using IE7, although my personal IE7 test workstation did not appear to be affected. It's quite possible our customers were going through an SSL proxy. I suspect as more people upgrade their browsers, this will become a more serious issue for CSS users, and I hope this saves somebody a huge headache and problems with their production environment.
-Joe

Hi Joe,
That's a very good analysis you did.
As you already suspected, the issue comes from the TLS record fragmentation feature that was introduced in the latest browser versions to overcome a SSL vulnerability (http://www.kb.cert.org/vuls/id/864643). Unfortunately, similar issues are happening with multiple products.
For CSS, the bug tracking this issue is CSCtx68270. The development team is actively working on a fix for it, which should be available (in an interim software release, so to get it you wil have to go through TAC) in the next couple of weeks
In the meantime, as workaround, you can configure the CSS to use only RC4 cyphers (which is what you were suggesting also). These are not affected by the vulnerability, so, browsers don't apply the record fragmentation when they are in use. This workaround has been tested by several customers already, and the results seem to be very positive.
Regards
Daniel

CSS 11501 Load Balancing with X-forwarded-for

Hi,
We have a pair of CSS 11501,
Currently it is using source ip for load balancing and 5 servers as backend , however we have users loggin in using http and based on its source IP (ISP PROXY) , it is forwarded to SERVER A.
However, we have a SSL page and when the client switches over to SSL , it is forwarded to SERVER B/C/D/E based on its source IP ( REAL CLIENT IP) .
This will cause the user to be terminated as the 5 servers are independent and not running in a cluster.
Is there any way that we can use the X-Forwarded-For address to load balance so that when users loging , they are sent to SERVER A (Based on X-Forwarded-For Header IP which translate to REAL CLIENT IP).
This way we are able to also send it back to the same server when it uses SSL.
I believe that we should be able to load balance using X-Forwarded-For IP or to rewrite the X-Forwarded-For IP into client source IP
Regards

Hi,
Unfortunately CSS does not support X-Forwarded-For, and even if CSS supports that, this wont work if you are not using SSL termination.
One option that you can use here, is using SSL termination, so you can manage the SSL traffic on HTTP on the CSS, in this way you can use the same HTTP content rule which is the one currently working.
In summary, you will have an SSL content rule that will decrypt the traffic, and this one will use the same content rule that already exist for HTTP, in case that the server is the one doing the redirect to SSL, but this is something that requires testing since depending on the redirect behavior we might have a redirect loop, but without details it is kind of hard to confirm that you will face this with this option.
Another option, which is less complex, is to use a portless content rule, so this content rule will match port 443 and 80 at the same time, and using sticky or balance based on source IP, you will get the same result with less config. The downside is the troubleshooting, but in this way you will have what you want.
content HTTP-HTTPS
    vip address 10.198.44.70
    advanced-balance sticky-srcip
    add service server1
    add service server2
    add service server3
    add service server4
    add service server5
    protocol tcp
    active
Here the content rule is not looking for the destination port, it is just looking for the source IP, and HTTP and HTTPS will end all the time on the same server.
Thanks,
Rodrigo

CSS SSL

I have a CSS11501 and the decision has been made to load the certificates on the servers instead of using the load balancer ssl module. Is this possible? The ssl termination point will be the servers instead of the css. I don't feel that this is the best way to go, but mgmt does. Can someone please point me in the right direction.
Thanks!

As Jeramy mentioned the configuration you have provided will work. However, the services do not require the "port 443" NAT rule to be hardset(services will inherit the port defined within the content rule), the keep-alive check for the services you created are using the default ICMP check, and what would be the reason for the group rule? Do you wish to perform internal load balancing with this rule?
The group rule will SNAT all client requests to appear as the 192.168.20.4 VIP address. Even though the CSS does not support the X-Forwarded-For HTTP option you can accomplish the same thing and be able to hit your VIP internally while preserving the client IP addresses by using ACLs on the CSS.
- Jason

CSS - SSL Stickiness

Gilles,
Could you please advice the CSS content configured with stickiness SSL ID and balance method round robin is recommended configuration or not.Are there are any issues with SSL stickiness with the browsers i.e IE .
Note:- I am not using SSL Module in the CSS.
Thanks in advance...

There are two issues
Some versions of IE (5.0, 5.5 --check http://support.microsoft.com/directory/article.asp?ID=KB;EN-US;Q265369) will
cause the client to change its SSL ID every 2 minutes and this will break
stickyness with application ssl and advanced balance SSL as this is layer 5
stickyness based on SSL session ID. A sniffer trace from the client will
show the ID field change.
You have to be aware that SSL stickiness will only work with SSL v3,
because it comes with the session ID not encrypted. SSL v2 comes with the session ID encrypted and you can't do stickyness
based on that version.So your appliaction servers must be using SSL v3, if you want to use SSL ID based stickiness.
Hope it helps
Syed Iftekhar Ahmed

SSL load balancing requirements

Hi all
I'm fairly new to CSS so apologies if this is an aobvious question...
Do I need an ssl module to load balance https requests using certificates on the servers themselves, simple round robin, two web servers, setup
regards
nigel

Nigel,
you only need and ssl module if you want the css to decrypt the traffic in order to make loadbalancing decision based on the http data contained in the encrypted ssl data.
For example, if you want to check the url or do stickyness based on a cookie.
All this info is encrypted in HTTPS, so this is when you need the ssl module.
If you don't need this feature, you can simpply consider https as normal tcp traffic and loadbalanced based on the tcp destination port.
Gilles.

CSS SSL Proxy - how can I write the original source address in http header

I'm replacing some BigIP's with CSS11500's that are configured to do front/backend ssl proxying in a one-armed configuration. The BigIP's write the original source IP address as a http header value when the traffic is sent to the application, and the application uses the IP to match against an application ACL. How can I do the same in the CSS.
thanks,
Brian

here is what you can insert with the SSL module :
http://www.cisco.com/en/US/products/hw/contnetw/ps792/products_configuration_guide_chapter09186a0080292a76.html#wp1027619
Gilles.

Webdispatcher SSL load balance server mismatch errors

We are setting up a webdispatcher to access an Enterprise Portal with multiple instances. Currently it is working but we are having to overide host mismatches. in webdispacther log we see
[Thr 4856] Mon Mar 07 11:38:02 2011
[Thr 4856] MatchTargetName("aaa.mycompany.com", "CN=bbb.mycompany.com, OU=xxx, O=ooo, L=ccc, SP=sss, C=US") FAILS
[Thr 4856] SSL NI-sock: local=##.21.13.137:50746 peer=##.21.13.131:51001
[Thr 4856] <<- ERROR: SapSSLSessionStart(sssl_hdl=0000000008565100)==SSSLERR_SERVER_CERT_MISMATCH
The Portal instances are on
aaa.mycompany.com
bbb.mycompany.com
Currently have a CA approved certificate for each server installed in the portal. Dispatcher on aaa uses aaa cert, dispatcher on bbb uses bbb cert.
Message server is on aaa, but it will load balance and place you on either instance.
have following related parameters
wdisp/ssl_encrypt = 2
wdisp/ssl_auth = 2
wdisp/ssl_cred = C:\usr\sap\XXX\W00\sec\XXX.pse
wdisp/ssl_certhost = aaa.mycompany.com
wdisp/ssl_ignore_host_mismatch = TRUE
C:\usr\sap\XXX\W00\sec\XXX.pse has ssl cert of both aaa and bbb servers.
All seems to be working, as users are load balancing. They are not getting certificate mismatches in their browser anymore. We are getting the SSSLERR_SERVER_CERT_MISMATCH errors, but the messages do not seem to cause an issue since we have wdisp/ssl_ignore_host_mismatch set.
Can we eliminate those mismatch errors instead of masking the problem with wdisp/ssl_ignore_host_mismatch?
Should each portal instance have their own ssl cert, or is there a way to use one cert such as the aaa.mycompany.com cert on each portal instance? It seems like that might eliminate the mismatch errors. However, what happens when you go directly to the bbb.mycompany.com portal instance? there is a certificate error if you specify aaa's and you go to bbb. I was wondering if the wdisp/ssl_auth and wdisp/ssl_certhost are valid in the portal system so that each server uses the aaa server and certificate. I could not tell if this parameter is valid for java-only portal systems.
Thanks for your help.
Edited by: Fett Patrick on Mar 7, 2011 8:35 PM

Thank you Martin for your prompt reply. Can you clarify please, can we use the wdisp/ssl_certhost parameter in the instance profiles of the portal instances? I wasn't sure if that is only valid for webdispatchers or can also be used in abap/java systems?
We orginally had the aaa server certificate listed for each dispatcher in the portal under ssl provider runtime server identity. That caused a browser "certificate error" when accessing the bbb server. So we then installed an ssl certificate for bbb for its dispatcher. We could then go to either server with no browser "certificate mismatch" error.
Then when we added the webdispatcher, we started getting the server mismatch errors at the webdispatcher level. If the wdisp/ssl_certhost can be used in the portal profiles, then that would hopefully resolve direct access or via web dispatcher aceess mismatches. I.E. only the aaa ssl certificate would be used and parameters would be set at both the webdispatcher and portal profiles
Thanks, Pat.

CSS SSL renewal problem

While renewing the ssl certification in CSS everything went fine while installation but after that when i checked with the following command
sh ssl associate rsakey | grep url(dont want to mention name)
i can see the previous as well as the new both key as associated and says yes
while the new should show yes and old should be no
same it is showing for cert
can anyone help me to sort out with this problem what it can be
Thanks in advance

Sagar,
Have you performed the "no ssl associate rsakey" and the "no ssl associate cert"?
After that, perform the "clear ssl file " and "clear ssl file rsakey "
HTH
Dave

CSS SSL L5 balancing

Similar Messages

Maybe you are looking for