Random failures to CSS doing https balancing.

So I have a cluster of about 10 machines behind a 11503, each server is setup like
service server-1
ip address 192.168.10.171
port 443
string cluster01
keepalive type script ap-kal-httplist "192.168.10.171 /webct/about.jsp"
keepalive frequency 15
active
and clustered in a service via
content ssl-rule
balance leastconn
protocol tcp
port 443
advanced-balance sticky-srcip-dstport
vip address 192.168.200.19
add service server-1
add service server-2
add service server-3
add service server-9
add service server-10
active
I am not currently doing ssl termination, just balancing.
Ok, so recently the load has started to rise (it is an e-learning application for a university and it's finals time) and now I see a scenario where random users are unable to connect to the https://elearningapp.somedomain.ca URL, while the person sitting next to them (both physically and IP-wise) connects fine. It is only a percentage of users who see this, seemingly no correlation between them, and if I reset the css it goes away for a while.

You'll need to collect some info.
First, capture a sniffer trace on one of the host showing the problem.
Check if the client gets a response to the SYN.
Check if the client can ping the CSS.
Then verify that the SYN comes to the CSS.
[capture a sniffer trace in front of CSS].
Then use 'sho flows x.x.x.x' to see if a flow is created.
Verify if the SYN is forwarded to a server.
Could be the server not responding.
What version do you run ?
Gilles.

Similar Messages

CSS 11503 Load Balancing Verification

Alright, so I have toiled long and hard to get this right. I think I have the config down but I am unsure on how to verify how this load balancing is working.
Here is the Content Config that I am speaking of:
content cad-rule
    add service wls1-e0
    add service wls1-e1
    add service wls2-e0
    add service wls2-e1
    add service wls3-e0
    add service wls3-e1
    add service wls4-e0
    add service wls4-e1
    add service wls5-e0
    add service wls5-e1
    add service wls6-e0
    add service wls6-e1
    arrowpoint-cookie expiration 00:00:15:00
    advanced-balance arrowpoint-cookie
    redundant-index 2
    vip address 172.30.194.195 range 2
    arrowpoint-cookie name TOQ
    protocol tcp
    port 8001
    url "/*"
    active
Each service in the rule above is configured as follows:
service wls1-e1
port 8001
protocol tcp
strin ags001-e1
ip address 172.30.193.81
keepalive type http
keepalive uri "/cad/index.html"
redundant-index 12
keepalive frequency 20
keepalive maxfailure 10
keepalive retryperiod 2
active
I am using the advanced arrowpoint cookies because I need some stickiness here. Straight round-robin would not have done what I needed it to do.
Now, when I go to my show summary, this is what I see for this rule:
                 cad-rule    Master   wls1-e0 84274
                                            wls1-e1 13144
                                            wls2-e0 96884
                                            wls2-e1 26374
                                            wls3-e0 71145
                                            wls3-e1 16592
                                            wls4-e0 76403
                                            wls4-e1 8657
                                            wls5-e0 118623
                                            wls5-e1 22760
                                            wls6-e0 30836
                                            wls6-e1 20464
The far right column indicates the services hits. I originally had the E1's suspended and activated them later on. So if this was true round robin, all the E0's should have the same number of service hits and all the E1's should have the same number of service hits. But as you can see, the wls5 server is getting hit the most while the wls6 server is sitting there twiddling its thumbs.
Now understanding how the arrowpoint cookies do their load balancing (inserting a cooking into the flow and then timing out after 15 mins as configured above) I would not expect a 1:1 ratio of load balancing between servers. But the distribution above seems rather extreme.
Does anyone have any suggestions on how to both A) verify that this is the right config and B) suggest to my boss that this is working the way it should be working?
Thanks!
James

Hi James,
There are several reasons of the uneven load balancing that you are seeing (based on the show summary). First
of all, the CSS is configured to do stickiness (advance-balance).
With arrowpoint-cookies (for HTTP only) method for stickiness, only the requests coming with the same cookie
are going to get stuck to the same server, since the cookie is
lost when the browser is closed (or based on the expiration), then the stickiness is going to be session
based and if the same client open a new session is going to be load balanced.
Is important to understand that when using stickiness, no real even load balancing is
going to happen since we are sticking new flows to the same server; even when layer 5 stickiness would
permit more even balancing than layer 3 stickiness (source IP based).
Also consider that the "show summary" is a command to see the hits (requests) being balanced to an specific
server, this is a good command to see the load balancing, anyway since the CSS balance
connections (flows), a persistent connection could have a lot of requests, so all those requests are
always going to the same server (incrementing the amount of hits in the counter) while a non-persistent
connection would be just one request (refer to HTTP persistence).
Also keep in mind that if a service is take out for maintenance, or is added to the load balancing later
than another, or if goes down for a period of time, then the CSS will be balancing among the remaining alive
servers. When you add the server again, the another servers are going to have connections
already established, so since the CSS is doing round robin, the server last added will
never have the same amount of connections (nor hits) that the other ones, because while one could
have 55 for example, the new one will have it first connection, and when the first one
gets the 56, the another will get the second, and so on.
Please let me know if this makes any sense.
Diego M

CSS 11501 Load Balancing with X-forwarded-for

Hi,
We have a pair of CSS 11501,
Currently it is using source ip for load balancing and 5 servers as backend , however we have users loggin in using http and based on its source IP (ISP PROXY) , it is forwarded to SERVER A.
However, we have a SSL page and when the client switches over to SSL , it is forwarded to SERVER B/C/D/E based on its source IP ( REAL CLIENT IP) .
This will cause the user to be terminated as the 5 servers are independent and not running in a cluster.
Is there any way that we can use the X-Forwarded-For address to load balance so that when users loging , they are sent to SERVER A (Based on X-Forwarded-For Header IP which translate to REAL CLIENT IP).
This way we are able to also send it back to the same server when it uses SSL.
I believe that we should be able to load balance using X-Forwarded-For IP or to rewrite the X-Forwarded-For IP into client source IP
Regards

Hi,
Unfortunately CSS does not support X-Forwarded-For, and even if CSS supports that, this wont work if you are not using SSL termination.
One option that you can use here, is using SSL termination, so you can manage the SSL traffic on HTTP on the CSS, in this way you can use the same HTTP content rule which is the one currently working.
In summary, you will have an SSL content rule that will decrypt the traffic, and this one will use the same content rule that already exist for HTTP, in case that the server is the one doing the redirect to SSL, but this is something that requires testing since depending on the redirect behavior we might have a redirect loop, but without details it is kind of hard to confirm that you will face this with this option.
Another option, which is less complex, is to use a portless content rule, so this content rule will match port 443 and 80 at the same time, and using sticky or balance based on source IP, you will get the same result with less config. The downside is the troubleshooting, but in this way you will have what you want.
content HTTP-HTTPS
    vip address 10.198.44.70
    advanced-balance sticky-srcip
    add service server1
    add service server2
    add service server3
    add service server4
    add service server5
    protocol tcp
    active
Here the content rule is not looking for the destination port, it is just looking for the source IP, and HTTP and HTTPS will end all the time on the same server.
Thanks,
Rodrigo

Dreamweaver is randomly deleting my .css file and then crashing...need help ASAP!

I use Dreamweaver CS 5.5 at my job. I just built a new website for the company using it and for a while, it was deleting the css file completely while in dreamweaver, but it had the little * next to the title like it hadn't been saved. i would usually try to close it without saving or crashing it which is normally does after this happens. So far, the css files have not been totally deleted...yet. I'm not waiting around for that to happen before I try to fix it. I've searched google and the forums and have found nothing or any related topics. I had my computer completely reformatted and that seemed to have fixed it for a short time. Now I'm on another web project and it's doing it again, but this site is bigger and more important than the last, and I don't have as much time to work on it. Has anyone come across this bug at all? I need to get this fixed ASAP...I would really hate to lose my files. I tend to hit Ctrl+S instinctively these days and i'd hate to do that while it's been deleted. not to mention the downtime of restarting Dreamweaver only to have it do it again 5 minutes later.
Computer specs:
Windows 7 Pro SP1 64-Bit
Dreamweaver CS 5.5
24GB RAM
intel Xeon X5650 @ 2.67GHz/2.66GHz (2 processors - 24 cores)
Nvidia GeForce GTX 570
Please let me know if there are any solutions to this. Thank you!
-Travys

Hmmmm.... When file content randomly disappears, it's usually a symptom of something much deeper -- like a nasty Trojan/Virus, a failing hard drive, bad CPU or corrupt user account.
You can try the usual DW solutions below, but I think you might want your IT guys/gals to run some diagnostics on your computer.
#1 Ensure your CSS & HTML files contain valid code. Use the W3C tools below.
http://jigsaw.w3.org/css-validator/
http://validator.w3.org/
# 2 When DW starts acting weird, the first thing to try is Deleting Corrupted Cache in DW
http://forums.adobe.com/thread/494811
#3 If that doesn't help, try Restore Preferences
http://helpx.adobe.com/dreamweaver/kb/restore-preferences-dreamweaver-cs4-cs5.html
#4 If all else fails, use the CS Cleaner Tools below followed by a software re-install.
http://helpx.adobe.com/creative-suite/kb/cs5-cleaner-tool-installation-problems.html
Good luck!
Nancy O.

Random Failure to power up.

I have bootcamp installed and I run XP and Mac osx. Occasionally, my computer will either just freeze or shutdown. This generally only happens in XP, but when I go to turn the computer back on, the dvd drive whizzes but the computer will not give the starting sound. At this point, it just refuses to boot up, and the power button flashes three times in sequence continuously. I am not sure what is causing this, but if I boot up the computer without the power plugged in, I can sort of force it to boot up. I just bought the computer Last month, so it shouldn't be having any serious issues.

Welcome to the Forums!
Assuming you have an MBP, the power button does not flash (no light there). So, either the latch light (aka power LED) is flashing, or you have a desktop Mac.
Either way, your RAM is the culprit. The 3 flash-pause-repeat sequence indicates "marginal RAM" (vs. flashing once per second, which means bad RAM or no RAM installed).
See this page:
http://docs.info.apple.com/article.html?artnum=303363
So, your first step is to remove and re-seat your RAM - it may have become loose. The 'random' failure to start up would also be caused by this. This step solved the problem for the last poster with this same problem, see this thread.
These instructions will show you how to access the MBP's RAM:
http://docs.info.apple.com/article.html?artnum=303491
If that does not allow you to boot normally, try booting with only one RAM chip inserted, then the other (i.e. test each chip independently). If a chip is bad, contact AppleCare.
Hope this helps...

Using a single CSS to load balance multiple services

Is it possible to use a single CSS to load balance 3 different services (server farm) ? That mean the CSS need to advertise 3 VIP
I'm thinking of two scenarios:
1 - configure the CSS to use 4 interfaces: 1 to public, 3 to private (each interface will plug-in to a different vlan/server farm)
2 - configure the CSS to use 2 interfaces: 1 to public, 1 to private (all 3 server farms are in the same vlan)
Will both scenarios work ?
Thanks
--Phillip.

Hi Phillip,
both scenarios will work. One CSS can certainly manage more than 3 services! You can even use just one VIP for all traffic, then just create the proper rules to send specific traffic to the corresponding service(s). No need for 3 VIPs.
Regards
-juerg

Error in the filename response vaule when doing Http to file scenario

Hi,
   I am doing Http to File Scenario. I am trying to retrieve the source filename in the structure field "filename". I have defined a UDF for that and It worked well for file to file scenario.
Now for Http to file scenario if i want to get the filename, I am getting "null" value in the response.
I am using Http Client test tool for testing the scenario.
My output is
<?xml version="1.0" encoding="UTF-8" ?>
- <ns0:FileTarget_MT xmlns:ns0="http://soacity.net/HTTPtoFile">
<MsgId>DD74CE4AE01AC6F19C0900199912BCA0</MsgId>
<filename>nullnull</filename>
<Body>hello world.</Body>
</ns0:FileTarget_MT>
I got nullnull because i am trying to retrieve Source directory and filename.
Desired Out put
<?xml version="1.0" encoding="UTF-8" ?>
- <ns0:FileTarget_MT xmlns:ns0="http://soacity.net/HTTPtoFile">
<MsgId>DD74CE4AE01AC6F19C0900199912BCA0</MsgId>
<filename>/FileServer/Sender/test.xml</filename>
<Body>hello world.</Body>
</ns0:FileTarget_MT>
For File receiver i am using FTP Parameters.
I got the correct output for file to file scenario because i had placed the file in the source directory, so that XI can read the source directory and file name. but now for Http to file scenario i am using the Http client tool, where in i have an option to paste the payload directly or upload the file.
Any Inputs would be appreciated.
Regards,
Varun

Hi,
I Have defined the structure as
<?xml version="1.0" encoding="UTF-8"?>
<ns0:MT_Source_FILE xmlns:ns0="http://soacity.net/HTTPtoFile">
   <Message>
      <Id/>
      <Head>
         <Id/>
         <Filename/>
      </Head>
      <Body/>
   </Message>
</ns0:MT_Source_FILE>
My requirement is
"The HTTP adapter would recieve a HTTP document. It has two main tags: head and body; there are two options : either specify the filename in <head> and data in <body> or you send the full message as payload into <body><message>.... </message></body>"
So How do I proceed in Defining the objects.
Regards,
Varun

Clearing document does not balance to zero

Hello to the community,
maybe someone ever been through this issue before:
We had to regenerate documents (via RFSEPA01 program) for a GL account which was switched to open items incorrectly years ago and for which account balance and drilldown did not tie. After running the program all tables seemed to be aligned as FAGLB03, FAGLL03, FBL3N and F-03 were all in sync.
The issue comes when a clearing document was created for one of these documents and now it is still sitting in BSIS table although it was cleared. The clearing document does not balance out to zero but the user was able to post it through F-03 but cannot reverse it through FBRA we get message:
Balance of the line items is not zero F5136.
The solution would be to delete this record from the BSIS table as account balance and details match but not FAGLL03, FBL3N and F-03. I ran report TFC_COMPARE_VZ for the company code and nothing was found.
Any idea?
PS- I have been through a bunch of OSS notes and could not find anything as this is not related to payment card neither to + or - signs.
Kindest regards,
Gen
Here are the details of the clearing document created:

Hello Geneviee,
Thank you for your reply, please return the message when you are ready for me to logon and I will try and deal with it as soon as possible.
Kind Regards,
Martin

SAP P & L Report does not balance to Balance sheet profit

Dear support,
I have a client that we have just taken over and they have a strange problem with their financial reports. The profit on the P & L Report does not balance to the profit on the balance sheet? The trial balance still balances so I am not sure whee the difference is coming from??
It has not balanced since the first take on balance were brought int othe system. Is it possible that some account with balances were moved from a P & L draw to a Balance sheet draw in the chart of accounts? If so how do I find the problem account?
Regards
Clayton

I resolved this issue myself.

Does the balance of a iTunes Gift Card that is "loaded" to my iTunes ? App Store account have an expiry date ?

Does the balance of a iTunes Gift Card that is "loaded" to my iTunes Store / App Store account have an expiry date ?
I know that the actual card/s have an expiry date, but when I load the card value into my iTunes Store / App Store account, will this balance have an expiry date ?
I can't find an answer.

UPDATE
Last night, approx.11:00PM I sent a "Express Lane" enquiry regarding my question.
I received the following e-mail back at 11:30PM:
I have read through your email and understand that you would like to know if there is any expiry time for Store Credit on your account. My name is ******* and I thank you for letting me help you with this.
That is a very good question *******, and the answer is once on your account there is no time frame required for you to spend it. It does not collect interest, but it will certainly be there for you, ready to be used when you find something that interests you.
Thank you for choosing iTunes as your source of entertainment, *******. If you have any more questions feel free to contact me as I am only an email away. Have a great day!
I have removed the names and highlighted the answer to the question I asked the forum.

CSS 11501 http load balancing

Hi,
i have configured to load balancing the http traffic to 2 servers, servers have the ip address 10.10.50.100 and 10.10.50.101 resp and the vip is 10.10.46.10
iam not able to access the http through the vip, can some one help on this
am i required to the nating, below is the connectivity
User -->SW->ASA->CSS->SW->server1 & server2
iam not able to access the server through vip
Please help
Thanks
Ravi
Ravi

What is the default gateway defined on Servers?
Is it CSS circuit IP or the ASA?
How is CSS conected? Are there diff vlans connected to servers and ASA? Or CSS is connected to the Switch in one arm mode?
You need to make sure that the return traffic from Servers should pass through the CSS.
Syed

CSS 11500 Re-balance services after service failure

We have 3 servers in our server farm and under normal circumstances wish to balance them unevenly such that the 3rd server only gets a small proportion of the traffic. The other two should be equally balanced. (Say 45, 45, 10).
When one of the prime servers (1st or 2nd) fails we wish to distribute the load across the remaining 2 evenly.
We are currently considering a cmd-sched script which will monitor the status of all three servers and reset the weights in the case of a server failure.
Is there a better way, preferably without a cmd-sched script?

Andrew,
This is messy but it should work. Configure multiple services for each server. Then use keepalive scripts to make some of the server3 services show as down if both the other servers are up, but if one of server1 and server2 is down, then all the server3 services show as up. You need to end up with the same number of reals configured for each physical box and vary the scripts so that most of the server3 services are down when the other 2 servers are OK.
Peter

Oracle CSS does not start up

Hi guys,
After installing Oracle 10g on Solaris 9, I could not get the database to start up on the web browser. I was wondering is the problem is due to the error below. If yes, what steps should I take to remedy it?
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'other'..
Operation successful.
Oracle Cluster Registry for cluster has been initialized
Usage: /etc/init.d/init.cssd start
Usage: /etc/init.d/init.cssd start
cat: cannot open /css/admin/inittab_local
Adding to inittab
/export/home/app/oracle/product/10.1.0/db_1/bin/localconfig:
/usr/bin/cp: not found
Sep 6 16:50:06 unknown root: [ID 702911 user.alert] (Oracle CSSD will
be run out of init)
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
Giving up: Oracle CSS stack appears NOT to be running.
Oracle CSS service would not start as installed
Automatic Storage Management(ASM) cannot be used until Oracle CSS
service is started
Thanx!

Thanks for replying. My database does not work with ASM.
I've reinstalled Oracle10g on Solaris as I believed that I made some errors for the previous versions.
Now, for this current Oracle, I can access the database using SQL and dbconsole service to start the EM. However I can't access the EM's database through the web browser after the first reboot.
i am able to use the EM using the path
http://unknown:5500/em
Note: My hostname is unknown as i had used DHCP during installation.
But once after reboot, the following error msgs appear at the startup and i could not access the EM:
Oracle CSSD being stopped
Shutting down CRS daemon
Shutting down EVM daemon
Failure 3 at CSS context initialization
Oracle CSSD will be run out of init
My qn is if it is my unknown hostname that cause the problem? Or is there any other soln pertaining to the error msg.

CSS 11500 Load balancing

Hello,
We have a CSS 11503 with the following partial config
==================
service 10.10.10.221-1724
ip address 10.10.10.1
keepalive type tcp
port 1724
keepalive port 1724
active
service 10.10.10.222-1724
ip address 10.10.10.1
keepalive type tcp
keepalive port 1724
port 1724
string string1
active
content 10.10.10.1-80-website
    vip address 10.10.10.1
    no persistent
    advanced-balance arrowpoint-cookie
    add service 10.10.10.221-1724
    add service 10.10.10.222-1724
    port 80
    protocol tcp
    url "/*"
    active
============================
There is connectivity from CSS to both IP's, 10.10.10.221 and 10.10.10.222. Problem we face is as following:
A client can hit web site on both servers by going to http://10.10.10.221:1724 and http://10.10.10.222:1724.
With service started on 10.10.10.221 and 10.10.10.222, a client PC can hit website by using http://10.10.10.1.
With step 2 above, connection count increasing on "service 10.10.10.221-1724" service.
There is no activty on "service 10.10.10.222-1724"
When we stop services on 10.10.10.221, client can no longer access web site using http://10.10.10.1. In this situation, connection counter on "service 10.10.10.222-1724" increases with each attempt to access web site but the page on client machine times out.
With service stopped on 10.10.10.221, client can access web site using server IP, http://10.10.10.222:1724
Restarting service on 10.10.10.221 makes access to website usig http://10.10.10.1, load balancer IP.
When capturing packets using wireshark, we see that the client machine sends re-transmission on "HTTP Get" and evantually times out.
With behavior above, it is clear that the server at 10.10.10.222 is active. What we cannot understand is why web site is inaccessible thru load balancer using http://10.10.10.1.
Please help.
Thanks,
Paresh.

Hi Paresh,
To troubleshoot this, I would recommend doing a traffic capture on the server vlan to see what is really happening with the connection.
One thing worth checking would be comparing the routing configured on both servers. If the traffic back from the server towards the client is not going through the CSS, the connection would fail, with the exact symptoms you are describing.
Regards
Daniel

Random failure with "hdiutil create"

Hi everyone
I'm working on a Shell script supposed to build disk image files with hdiutil. The problem is that it randomly fails. Here is the output :
+hdiutil: create failed - erreur 49168+
That's all I get when using
+hdiutil create -format UDBZ -volname "SFML-1.6-dev-macosx" -srcfolder "/Users/me/sfml/trunk/dist/SFML-1.5-dev-macosx" "$/Users/me/sfml/trunk/dist/SFML-1.6-dev-macosx.dmg" > /dev/null+
When dropping "> /dev/null" and adding -debug, I noticed these lines before the end :
+2009-07-24 18:31:40.620 diskimages-helper\[25526:1603] deleting /Users/lucas/sfml/trunk/dist/SFML-1.6-dev-macosx.dmg+
+2009-07-24 18:31:40.620 diskimages-helper\[25526:1603] DIHelperNBI performOperation: returning 49168+
+DIHLDiskImageCreate() returned 49168+
+2009-07-24 18:31:40.842 diskimages-helper\[25526:10b] DIHelper dealloc.+
+2009-07-24 18:31:40.843 diskimages-helper\[25526:10b] -DIHelperAgentMaster terminateUIAgentConnection.+
+hdiutil: create failed - erreur 49168+
I don't understand why it would try to delete the dmg file I'm trying to make...
Here is the full log is you want to have look at it : http://pagesperso-orange.fr/c.sobecki/ceylo/data/hdiutil-failure-log.txt
All I was able to find on this error is here : http://lists.apple.com/archives/Carbon-dev/2007/Aug/msg00357.html
Where it says :
+1) Some of the files you're trying to archive are being used exclusively by another process. A couple of people figured that their Anti-Virus software (Norton AV) was to blame.+
+2) Disk space is low, and the hdiutil tool cannot perform its internal caching.+
+3) A file system error which is not detected by Disk Utility prevents proper accessing some crucial files. Repairing the disk using "Disk Warrior" did help.+
+4) Files which were put into the trash can but were not deleted yet cause problems when tried to being put into the disk image.+
1) I'm not using an antivirus
2) I've about 70 GB remaining
3) But why would it be random ? It's always the same files that are being packed.
4) I've no file in my trash.
Does anyone have an idea why it fails ?
Thanks
Ceylo

Ceylo wrote:
Hi everyone
I'm working on a Shell script supposed to build disk image files with hdiutil. The problem is that it randomly fails. Here is the output :
+hdiutil: create failed - erreur 49168+
Where it says :
+1) Some of the files you're trying to archive are being used exclusively by another process. A couple of people figured that their Anti-Virus software (Norton AV) was to blame.+
1) I'm not using an antivirus
That's not the only reason, it's just one example. If the file is locked because it's being used by another process, that might explain the randomness of what is happening.
Try running
lsof
and seeing if any of the files in question are open.

Random failures to CSS doing https balancing.

Similar Messages

Maybe you are looking for