Apache POST flex2gateway never closes or times out, reaches max child processes

We have been trying to pass an external PCI scan, and noticed some server lockups after starting a scan. We are scanning a couple hundred IP addresses, which all resolve to the same servers. The scans are actively looking for vulnerabilities on the box, and one of which is flash remoting. When we look at the apache /server-status page, it shows a ton of long running flex2gateway processes. For instance:
22-4
4466
0/3817/3817
W
4.07
163840
0
0.0
57.76
57.76
x.x.x.101
WebNode2.ambassador.int
POST /flex2gateway/http HTTP/1.1
As you can see, this POST request has been running for 163840 seconds, or nearly two days. Since it seems these POST requests never complete, even though the client has long since disconnected, they simply stack up until the server's max number of child processes has been reached, effectively killing our webserver.
When I try to restart the clustered coldfusion instances one at a time, these POST requests do not die off.
If I stop both clustered CF instances, the requests complete (or get killed).
If I reload or restart apache, the requests are gone as well.
strace gives me nothing useful:
[root@WebNode1 ~]# strace -p 34025
Process 34025 attached - interrupt to quit
read(185,
pstack gives a little more, but nothing that looks obvious to me:
[root@WebNode1 ~]# pstack -p 34025
Usage: pstack <process-id>
[root@WebNode1 ~]# pstack 34025
#0 0x00007fdd40444740 in __read_nocancel () from /lib64/libpthread.so.0
#1 0x00007fdd33efe2e6 in jk_tcp_socket_recvfull () from /opt/coldfusion10/config/wsconfig/1/mod_jk.so
#2 0x00007fdd33f1b68d in ajp_connection_tcp_get_message () from /opt/coldfusion10/config/wsconfig/1/mod_jk.so
#3 0x00007fdd33f1ceea in ajp_get_reply () from /opt/coldfusion10/config/wsconfig/1/mod_jk.so
#4 0x00007fdd33f20308 in ajp_service () from /opt/coldfusion10/config/wsconfig/1/mod_jk.so
#5 0x00007fdd33ef8f5d in jk_handler () from /opt/coldfusion10/config/wsconfig/1/mod_jk.so
#6 0x00007fdd41b92cd0 in ap_run_handler ()
#7 0x00007fdd41b9658e in ap_invoke_handler ()
#8 0x00007fdd41ba1c50 in ap_process_request ()
#9 0x00007fdd41b9eac8 in ?? ()
#10 0x00007fdd41b9a7d8 in ap_run_process_connection ()
#11 0x00007fdd41ba6ad7 in ?? ()
#12 0x00007fdd41ba6dea in ?? ()
#13 0x00007fdd41ba7a6c in ap_mpm_run ()
#14 0x00007fdd41b7e9b0 in main ()
I dont know what that tells us exactly, but I'm leaning toward the hangup between apache and tomcat.
Any suggestions on where how to troubleshoot this issue?

On a test server, I have removed the wildcard from the uriworkermap.properties file, so it now only matches "/flex2gateway" and "/flex2gateway/". Unfortunately I'm still seeing the occasional hung apache worker.
Anyone have any leads on this issue? I don't mind doing the research, I'v just exhausted the limits of my Google Fu.
Apache Server Status for 10.10.10.205
Server Version: Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips mod_wsgi/3.2 Python/2.6.6 mod_jk/1.2.32 mod_perl/2.0.4 Perl/v5.10.1
Server Built: Oct 16 2014 14:48:21
Current Time: Monday, 10-Nov-2014 16:49:22 EST
Restart Time: Monday, 10-Nov-2014 15:25:16 EST
Parent Server Generation: 0
Server uptime: 1 hour 24 minutes 6 seconds
Total accesses: 5313 - Total Traffic: 98.4 MB
CPU Usage: u3.97 s1.26 cu0 cs0 - .104% CPU load
1.05 requests/sec - 20.0 kB/second - 19.0 kB/request
15 requests currently being processed, 11 idle workers
WWWWWWW_W_W_W__W__W__WW_W_...................................... ................................................................ ................................................................ ................................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
Srv
PID
Acc
M
CPU
SS
Req
Conn
Child
Slot
Client
VHost
Request
0-0
8727
0/12/12
W
0.03
4572
0
0.0
0.05
0.05
10.10.2.201
qc.company.int
POST /flex2gateway HTTP/1.1
1-0
8728
0/11/11
W
0.03
4358
0
0.0
0.18
0.18
10.10.2.201
qc.company.int
POST /flex2gateway HTTP/1.1
2-0
8729
0/38/38
W
0.04
3910
0
0.0
1.11
1.11
10.10.2.201
qc.company.int
POST /flex2gateway HTTP/1.1
3-0
8730
0/27/27
W
0.03
4064
0
0.0
0.79
0.79
10.10.2.201
qc.company.int
POST /flex2gateway HTTP/1.1
4-0
8731
0/16/16
W
0.03
4354
0
0.0
0.12
0.12
10.10.2.201
qc.company.int
POST /flex2gateway HTTP/1.1
5-0
8732
0/7/7
W
0.02
4564
0
0.0
0.02
0.02
10.10.2.201
qc.company.int
POST /flex2gateway HTTP/1.1
6-0
8733
0/8/8
W
0.02
4673
0
0.0
0.01
0.01
10.10.2.201
qc.company.int
POST /flex2gateway HTTP/1.1
7-0
8734
0/386/386
0.37
4
0
0.0
6.49
6.49
10.10.2.212
www.company.qc
GET /marketingpages/images/login_over.jpg HTTP/1.1
8-0
9422
0/10/10
W
0.02
4564
0
0.0
0.04
0.04
10.10.2.201
qc.company.int
POST /flex2gateway HTTP/1.1
9-0
10112
0/393/393
0.37
6
0
0.0
14.59
14.59
10.10.2.212
www.company.qc
GET /marketingpages/images/box_onesource.jpg HTTP/1.1
10-0
10468
0/321/321
W
0.32
846
0
0.0
4.42
4.42
10.10.2.212
qc.company.int
POST /flex2gateway HTTP/1.1
11-0
10470
0/398/398
0.38
6
0
0.0
12.80
12.80
10.10.2.212
www.company.qc
GET /marketingpages/images/home_eco.jpg HTTP/1.1
12-0
10471
0/340/340
W
0.32
837
0
0.0
4.99
4.99
10.10.2.212
qc.company.int
POST /flex2gateway/ HTTP/1.1
13-0
10544
0/404/404
0.34
6
0
0.0
5.21
5.21
10.10.2.212
www.company.qc
GET /marketingpages/images/box_top.jpg HTTP/1.1
14-0
10592
0/353/353
0.40
6
12
0.0
14.10
14.10
10.10.2.212
www.company.qc
GET /?login HTTP/1.1
15-0
10648
0/296/296
W
0.31
800
0
0.0
3.82
3.82
10.10.2.212
qc.company.int
POST /flex2gateway/ HTTP/1.1
16-0
12382
0/339/339
0.33
6
0
0.0
2.85
2.85
10.10.2.212
www.company.qc
GET /marketingpages/images/logo_sourceone.jpg HTTP/1.1
17-0
12387
0/336/336
0.34
6
0
0.0
5.06
5.06
10.10.2.212
www.company.qc
GET /marketingpages/images/logo_onesource.jpg HTTP/1.1
18-0
12388
0/265/265
W
0.25
839
0
0.0
2.87
2.87
10.10.2.212
qc.company.int
POST /flex2gateway/ HTTP/1.1
19-0
12389
0/323/323
0.31
0
0
0.0
4.82
4.82
10.10.2.212
www.company.qc
GET /marketingpages/lib/dimming.js HTTP/1.1
20-0
12390
0/336/336
0.31
4
0
0.0
5.24
5.24
10.10.2.212
www.company.qc
GET /marketingpages/lib/superfish.js HTTP/1.1
21-0
12391
0/289/289
W
0.27
805
0
0.0
2.49
2.49
10.10.2.212
qc.company.int
POST /flex2gateway/ HTTP/1.1
22-0
12392
0/281/281
W
0.27
831
0
0.0
3.17
3.17
10.10.2.212
qc.company.int
POST /flex2gateway HTTP/1.1
23-0
14750
0/41/41
0.04
6
0
0.0
0.92
0.92
10.10.2.212
www.company.qc
GET /marketingpages/images/close.jpg HTTP/1.1
24-0
14751
0/43/43
W
0.04
0
0
0.0
1.21
1.21
10.10.2.36
qc.company.int
GET /server-status HTTP/1.1
25-0
14752
0/40/40
0.04
6
0
0.0
0.96
0.96
10.10.2.212
www.company.qc
GET /marketingpages/images/box_sourceone.jpg HTTP/1.1

Similar Messages

Data Transfer Process (DTP) - Messages Never Received and Times Out

Hello,
I'm having an Issue with DTPs at the moment. First I'll give you a little background on the system
Formerly BW 3.1, upgraded recently to BI 7.0 (Support Pack 15). Most of our Data Flows are still under 3.x style, but many are being migrated or recreated under BI 7.0 standards.
I am able to successfully load data using DTPs however on many, and seemingly random, occasions the message for individual data packages under "Processing -> Updating InfoProvider" never arrive, and thus the DTP never completes and eventually times out.
Under BW 3.x it was a simple matter of changing the QM status to Red and then "Update Manually" on the data package that did not receive the message. This would reprocess and complete the data package, allowing you to then turn the QM status Green and move on.
Under BI 7.0 I do not seem to have any options for the Data Packages regardless of QM status. I have not found a way to resolve this situation and in every instance had to delete the request from the InfoProvider and try again. It seems like a rather large design flaw, so I'm hoping I'm missing something.
Any suggestions would be greatly appreciated.
Thanks,
Matt

Three things to be noted:
1. Dont use the std. settings of 50000 records. Change it to 10000 records in DTP.
2. Since you are not getting till Updating tab, I foresee something wrong with routine..start/end/transformation. Check there how much time it is taking.
3. I dont think we have any option as you mentioned for QM. We need to delete the req and start the DTP again.
Thanks..
Shambhu

Broadcaster times-out due to dialog process

Hi !
I'm trying to set up a broadcasting for a report with a large amount of records, and since when I execute it in dialog it times out, broadcasting it in the background seemed a great idea.
The thing is that, despite I schedule it to run in a background process, it still creates a dialog process to fetch data wich times out and therefore the background process (the document creation and email sendig) ends correctly sending a 'No Data Available' report or with an error: com.sap.ip.bi.base.exception.BIBase.RuntimeException.
The parameters for the report are being set in the General Precalculation tab, while there is a Filter Navigation tab which allows you to filer specific characteristics. Which is the difference here ?
Regards,
Santiago

Sonal, thanks for the quick answer.
Let me see if I get this right.
The Precalculation tab allows to filter before precalculation, in order to fetch de query from precalc. cache instead of direct db access. The data excluded from filters in precalc. tab will not be available in the query.
On the other hand, Filter tab sets filter values for characteristics contained in the query. Is this like the right column of the filter tab of Query Designer ?
Let me know if this is right.
Regards,
Santiago

HT200160 Using a D-Link DNS-320 NAS. No problem connecting with afp protocol from a MBP OS10.8. MBP OS10.9 hangs up; shows the volumes, but never connects or time out. "Finder Not Responding." What gives?

Similar problem on a Mac Mini running OS10.9 Mavericks. Note: Several weeks ago I posted a comment about Console displaying error messages when my MBP running 10.9 was connected to the DNS-320, but the 10.8 machine doesn't display those same messages. Might be related.

I had the same problem with a D-link DNS-320.
The following worked for me (Manual back up):
Set Time Machine backups off to allow manual back-up.
Create one alias of your NAS share in the desktop. (I don’t know why, I first had to create the share alias in a external USB drive and then move the alias to the desktop cause mavericks didn’t allow me to create the alias directly in the desktop.)
After using the “back up now” option in the time machine menu, just wait for a couple of seconds and double click your time machine share alias. This will mount the share and time machine back up starts immediately.
It seems as if double click initialize the communication and then the back up works normally.
I hope this helps.
Jcarlos

No time out at Datasocket read

I read two or more Datasocket items in a FOR loop. The time out flag is set and the time out is 500ms. The problem is that I never get an time out for an item although the item is not written from anywhere. What's the problem ?

semmar1,
This was a design decision that was made for DataSocket a long time ago. If you read from a dynamic data item that has not yet been written to, you will get the default data value for that item. That is, you will get either a FALSE, 0, or "" as a value. The timeout is there as more of a way to know if the network is down or not.
Randy Hoskin
Applications Engineer
National Instruments
http://www.ni.com/ask

Time Out for a syncronus BPEL process

Hi All,
Need your help with regards to the following:
From a parent BPEL process we need to invoke a synchronous BPEL Process .This(child) could take anything between 20-30 seconds for processing a request.
Now the requirement is to time out the parent BPEL process if the BPEL process it is calling takes more than 25 seconds for processing a request.
If( time taken to get from partner process >25 sec)
time out
Else
Move along with the rest of the process.
Approaches considered:
1) Pick activity :
Fallout: the pick activity would work only when you are calling a asynchronous BPEL process.
2) Having the invoke activity in a scope and adding an on Alarm branch for the scope.
Fallout: The BPEL process manager executes all the other activities even after exceeding the maximum time the process is set for timing out.
That is if we consider the below scenario
If( time taken to get from partner process >25 sec)
time out
Else
move along with the rest of the process.
Instead of executing the if block( onAlarm) it is executing the else block.
Request you to suggests approaches to suffice this requirement.

Hi Sai Madalli,
An approach to it, would be use the syncMaxWaitTime with time that you need, its a configuration per domain.
http://download-west.oracle.com/docs/cd/B31017_01/integrate.1013/b28981/events.htm#BABGGADB
http://download-east.oracle.com/docs/cd/B31017_01/integrate.1013/b28981/app_trblshoot.htm
Regards

How to avoid Time out issues in Datapump?

Hi All,
Iam loading one of our schema from stage to test server using datapump expdp and impdp.Its size is around 332GB.
My Oracle server instance is on unix server rwlq52l1 and iam connecting to oracle from my client instance(rwxq04l1).
iam running the expdp and impdp command from Oracle client using the below commands.
expdp pa_venky/********@qdssih30 schemas=EVPO directory=PA_IMPORT_DUMP dumpfile=EVPO_Test.dmp CONTENT=all include=table
impdp pa_venky/********@qdsrih30 schemas=EVPO directory=PA_IMPORT_DUMP dumpfile=EVPO_Test.dmp CONTENT=all include=table table_exists_action=replace
Here export is completed and import is struck at below index building.After some time iam seeing below time out in log files
Processing object type SCHEMA_EXPORT/TABLE/INDEX/INDEX.
Error:-
VERSION INFORMATION:
TNS for Linux: Version 11.1.0.7.0 - Production
Unix Domain Socket IPC NT Protocol Adaptor for Linux: Version 11.1.0.7.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.1.0.7.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 11.1.0.7.0 - Production
Time: 13-JAN-2012 12:34:31
Tracing not turned on.
Tns error struct:
ns main err code: 12535
TNS-12535: TNS:operation timed out
ns secondary err code: 12560
nt main err code: 505
TNS-00505: Operation timed out
nt secondary err code: 110
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=170.217.82.86)(PORT=65069))
The above ip address is my unix client system(rwxq04l1) ip.
How to see oracle client system port number?
Please suggest me how to avoid this time out issues.Seems this time out is between oracle server and client.
Thanks,
Venkat Vadlamudi.

Don't run from the client ... run from the server
or
if running from a client use the built-in DBMS_DATAPUMP package's API.
http://www.morganslibrary.org/reference/pkgs/dbms_datapump.html

Server.app can't load alerts and throws out error getAndLockContext: flock(servermgr_info) FATAL time out

Hi everybody,
I did a working migration of my Lion Server stating all services has been migrated successfully!
All my used services are running fine (dns, mail, iCal, dhcp, web, VPN, adressbook, profilemanger, sharing, opendirectory) but using server.app causes an error while server.app seems to load the alerts section without success throwing out some message windows and those messages in my log console
servermgrd[36677] [36677] error in getAndLockContext: flock(servermgr_info) FATAL time out
servermgrd[24195] [24195] process will force-quit to avoid deadlock
com.apple.launchd[1] (com.apple.servermgrd[24195]) Exited with exit code: 1
Servermgrd process goes up to 100% and you have to quit server.app to calm it down.
Anyone got a similar error or knows any workaround to abandon this error?
Thanks and regards
Eldrik

RB jp's solution (move the /var/log/eventmonitor folder) solved my problem completely.
Not only did Server.app suddenly become response again, the spinner trying to access alerts stopped spinning.
Best of all, the servermgrd daemon stopped crashing every two minutes! There were messages in system.log that read:
error in getAndLockContext: flock(servermgr_info) FATAL time out
server servermgrd[38070]: [38070] process will force-quit to avoid deadlock
Those are now gone. Server is happy again.

Displaying "Your Session is going to time out , Extend or Close" dialog ?

Hi!
we have a requirement to display a warning message to users just (1 minute) before the session is going to time out, and ask the user if he wants to extend the session or close the session( in which case we close the browser window). We have come up with a probable solution but would like to know what other people are doing/will do in similar scenarios.
Here is what we came up with:
1. On every page load we set/update a cookie which is meant to track the session time out. And we have a javascript timer that counts down user's inactivity. If any PPR occurs on the page, then we plan to add a hook to the PPR javascript calls so that we can update the time out cookie not only on page load/refreash but also on every PPR call made to the server.
We have a worry that we are doing everything on the client side solely depending on javascript.Is our approach in the right direction? Is there any better way to achieve the same? Are there any pitfalls/security issues involved in our design?
2. I have read in some articles/forum threads that we can poll the server on a regualr interval to find out the time remaining for the session to expire and once we get time to last minute we can display a javascript popup to the user. But I have a doubt that wouldn't polling itself refresh the user activity on the server and reset the session timeout to normal on every poll? In that case wouldn't we always get the same time-remaining info for every poll? Have I mis-understood some thing here? can some one explain how to implement the session time out based on polling to the server?
3. Any new ideas are welcome!
Thanks and Regards,
Samba

Hi,
well, if you use the af:poll command then the session will never expire because from a server point of view the user frequently sends a request. So if you prefer this option then you will have to expire the session programmatically calling invalidate() on the session
javaScript is the only option I can imagine that works without polling.
Frank

9 times out of 10 I have to force-quit to get out of my g-mail. This only happens only on my i-Mac, never on my Macbook pro. Can anybody help ?

MAX OS X - version 10.7.5 :
For some weeks now, I have to force-quit 9 times out of 10 to get out of my e-mail account (g-mail). This only happens on my i-Mac, never with my other devices (Macbook Pro, i.Phone, i-Pad).
Does anybody know what could be the reason for this ?
Thanks

OK, you have iMovie 08, which is also known is iMovie Version 7.1.4. It has been a few years since i used this version, so this is based on memory.
The only components you must have in the Library/QuickTime folder are the AppleIntermediateCodec.component and the AppleMPEG2Codec.component.
All the others could potentially be causing problems. I would drag the DivX components, the Flip4Mac components, and the Google Camera components to the Desktop temporarily, so nothing is left in this folder except the two Apple components.
Then restart your Mac and start iMovie. If it works, yay!
If you still need these other components for some other app, you can probably get updated DivX components for free at Perian.org.
You can get updated Flip4Mac components by running WMV Player and updating. It should prompt you to update automatically.
I am not sure about the Google components, but if you are running something that needs them, it should prompt you to install the current versions.
If none of that works, post back.

Konqueror 4.3.2 frequently times out vs. Firefox never does

Hi,
I'm on a fully (daily) Arch 32b/KDE 4.3.2 PC.
I've noted that Konqueror frequently times out (error message "Socket operation timed out") whereas Firefox NEVER does that (same sites). I've topped up the socket read (System Settings, Connection Preference) to 20 Sec (was 15 originally) but this didn't help.
Besides I notice that Firefox is much faster rendering various sites.
Please Advise!
Thanks

The Chrome bookmark issue is internal with Google since the Android team and Chrome team are seprate and is being addressed. There is a great app that I'm using called Chromemarks which comes in free and pay versions that sync your Chrome bookmarks and allow you to access (and for the pay version edit) them. Highly recommend this app since it looks like this issue is months old.
I know I returned my Ideapad for another, but I'm trying to stick around and help with small issues best I can and share any info I've gotten. Desipite this set back I'm still a Lenovo fanboy and hope to one day own a fully functioning Lenovo tablet (with all the bugs worked out). Highly recommend you guys also turn to some of the larger Android communities for help, some issues aren't Lenovo issues, but issues with the actual OS shared by all Androids (though nothing as major as the current Ideapad K1 issues). This inclues small app crashes, the bookmark syncing and being able to access all of your internal storage on your PC. Most of the communities are still mostly focused on phones, but because of the popularity of the tablets are beginning to shift gears.

HT204266 My App Store keeps struggling to load up. Requests frequently time out and searches just don't do anything. If it does load up a search, the iPad closes down App Store, returns to home screen and, upon re-opening, the search has been deleted. Hel

I've had my iPad (retina display) since last year. Over the last few weeks the App Store struggles to load on occasion and times out completely. When searches do rarely complete, the App Store closes down and returns my iPad to the home screen and, when I open the App Store again, it has just reset the store, forcing it to load all over again. I've tried closing all running background applications, with no difference being made. Any help would be greatly appreciated.

1. Double-click the Home button to reveal Task Bar
2. Hold the App Store app down for a second or two until you see the minus sign
3. Tap the minus sign to close app
4. Re-launch the app and test
Note: If necessary close all apps in the Task Bar

New ipad mini retina cannot select "never" for time out period. It makes the product useless for such things as navigation and frustrating for reading

Must add the never option to the time out period or the product is useless for navigation and other purposes where continual use is required.

This should be an option in the General settings on your iPad. The option is called Auto-Lock. There should be the following options: 2 minutes, 5 minutes, 10 minutes, 15 minutes, and Never.
If you select Never, that should stop the screen from going to sleep automatically. If the Never option is not present, you more than likely have a Exchange account synced with the device that does not allow that option. If that account is disabled, that option should show.
 Cheers, Tom

ITunes tells me there is an update (5.1) but it never gets any further than trying to connect to the iPad server. I've tried updating directly from my iPad and it too times out saying it cannot connect to any server. My regular Apple updates work fine.

iTunes tells me there is an iOS update (5.1) but it never gets any further than trying to connect to the iPad server. I've tried updating directly from my iPad and it too times out saying it cannot connect to any server. My regular Apple updates work fine.

I haven't updated yet. Apple's download servers have been too busy. Did you know you can update via wifi instead of connection to your computer?
Here's a description of the update. Not a lot of changes for iPad 1/2.
iOS 5.1 Software Update
http://support.apple.com/kb/DL1504
You can wait a while & update at your convenience.
 Cheers, Tom

Why does browser never close and unresponsive all the time horrible experience

it never closes repeatedly crashes takes for ever to load when I try to close it is unresponsive wont open attachments horrible browser never responds what a joke no phone to contact live person takes forever if it decides to even open

There are two places to check:
1. In the Summary pane for your iPad make sure that this box is not checked;
2. In the iTunes app you may wish to go to Preferences/Devices and check this box;

Apache POST flex2gateway never closes or times out, reaches max child processes

Similar Messages

Maybe you are looking for