OEM Grid generating too many core files

Hi all,
Database: Oracle9i Enterprise Edition Release 9.2.0.6.0 - 64bit Production
OEM (OMS and Agent): 10.2.0.5
OS: Solaris 10 (SPARC 64bit)
I have a weird problem concerning the agent, each an every time I start the agent, the cdump directory will be filled with hundreds of core files (66 MB each), thus feeling up the whole filesystem where the cdump directory resides. I'm not sure why this is happening. Is this a bug? Has anyone experienced this before?
Thanks in advance for your help.
Regards,
Durbanite - South Africa

Hi again,
This is the content of the alert log:
bash-3.00$ tail alert_clpr1.log
ORA-07445: exception encountered: core dump [00000001002258A4] [SIGBUS] [Invalid address alignment] [0x60708090A0B0002] [] []
Fri Jul 17 10:01:11 2009
Errors in file /udd001/app/oracle/admin/clpr/udump/clpr1_ora_27740.trc:
ORA-07445: exception encountered: core dump [00000001002258A4] [SIGBUS] [Invalid address alignment] [0x60708090A0B0002] [] []
bash-3.00$ tail /udd001/app/oracle/admin/clpr/udump/clpr1_ora_27839.trc
FD6C6EF9:00005D6A 265 0 10280 1 0x0000000000000109
F3FED575:00005992 266 0 10280 1 0x000000000000010A
FD6E1007:00005D6B 266 0 10280 1 0x000000000000010A
F40260C6:00005994 267 0 10280 1 0x000000000000010B
F40735E8:00005995 268 0 10280 1 0x000000000000010C
F40C0992:00005997 269 0 10280 1 0x000000000000010D
F40F9C50:00005999 270 0 10280 1 0x000000000000010E
KSTDUMP: End of in-memory trace dump
ssexhd: crashing the process...
Shadow_Core_Dump = PARTIAL
I think I might need to contact Oracle Supprt for this one, I'll start by using the ORA-07445 tool on metalink.
Thanks and regards,
Durbanited

Similar Messages

Getting java.io.IOException: Too many open files+ClassNotFoundException

Dear All
We have a web application deployed on Rational Application Developer 6.0 (Operating System is Windows 2000 professional) our users are randomly getting java.io.IOException: Too many open files+ClassNotFoundException when they click on some servlet or jsp link, but they are getting this error randomly for example when they click some link they may get these exceptions but refreshing page or clicking once again on same link executes servlet successfully.If anyone could help on this topic we will be grateful
Thanks

I think this these two exceptions are occuring in differrent environment
java.io.IOException is occuring under heavy load to web server its stack trace is as follows:
JSPG0225E: An error occurred at line: 2 in the statically included file: /SessionCheck.jsp
JSPG0093E: Generated servlet error from file: /Admin/AdminInsuranceCertificates.jsp
E:\WebSphere_6\AppServer\profiles\AUSECert\temp\centraNode04\server1\AUSECert\Vero.war\Admin\_AdminInsuranceCertificates.java:259: cannot access com.bplus.natmar.LoginDetails
bad class file: E:\WebSphere_6\AppServer\java\jre\lib\core.jar(java/io/Writer.class)
unable to access file: E:\WebSphere_6\AppServer\profiles\AUSECert\installedApps\centraNode04Cell\AUSECert.ear\Vero.war\WEB-INF\classes\com\bplus\natmar\LoginDetails.class (Too many open files)
Please remove or make sure it appears in the correct subdirectory of the classpath.
(source unavailable)
1 error
]: com.ibm.ws.jsp.JspCoreException: JSPG0049E: /Admin/AdminInsuranceCertificates.jsp failed to compile :
this error always occurs in reference to logindetails class this clas is used to make a session check on different roles in our project for e.g., user having end user role should not be able to log in as a user having admin role
we have included a sessioncheck.jsp in our every jsp page in this jsp we have simply used logindetails class as useBean and called its getresource() method
above stacktrace is from our live application server
while testing same project on our local system we are not getting too many open files exception but we are getting following ClassNotFoundException
[11/30/05 17:11:42:797 EST] 0000004a SystemOut O SELECT count(*) as NoofRecs FROM resourcerolebindings WHERE ResourceName = 'mainEdit.jsp' and IsEndUser=1
[11/30/05 17:12:50:891 EST] 000001eb SystemOut O SELECT count(*) as NoofRecs FROM resourcerolebindings WHERE ResourceName = 'InsuranceCertificates.jsp' and IsEndUser=1
[11/30/05 17:17:40:828 EST] 0000008d SystemOut O AppURL is: http://www.VeroECert.com/Vero/indexU.jsp
[11/30/05 17:17:58:141 EST] 0000008b SystemOut O SELECT count(*) as NoofRecs FROM resourcerolebindings WHERE ResourceName = 'InsuranceCertificates.jsp' and IsEndUser=1
[11/30/05 17:20:41:703 EST] 00000034 ServletWrappe E SRVE0026E: [Servlet Error]-[com.servlet.UserHelpServlet]: java.lang.ClassNotFoundException: com.servlet.UserHelpServlet
     at com.ibm.ws.classloader.CompoundClassLoader.findClass(CompoundClassLoader.java(Compiled Code))
     at com.ibm.ws.classloader.CompoundClassLoader.loadClass(CompoundClassLoader.java(Compiled Code))
     at java.lang.ClassLoader.loadClass(ClassLoader.java(Compiled Code))
     at java.beans.Beans.instantiate(Beans.java:202)
     at java.beans.Beans.instantiate(Beans.java:63)
     at com.ibm.ws.webcontainer.servlet.ServletWrapper$3.run(ServletWrapper.java:1384)
     at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java(Compiled Code))
     at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadServlet(ServletWrapper.java(Compiled Code))
     at com.ibm.ws.webcontainer.servlet.ServletWrapper.initialize(ServletWrapper.java:1312)
     at com.ibm.wsspi.webcontainer.extension.WebExtensionProcessor.createServletWrapper(WebExtensionProcessor.java:84)
     at com.ibm.ws.webcontainer.extension.InvokerExtensionProcessor.handleRequest(InvokerExtensionProcessor.java:238)
     at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:2841)
     at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:220)
     at com.ibm.ws.webcontainer.VirtualHost.handleRequest(VirtualHost.java:204)
     at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java(Compiled Code))
     at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java(Compiled Code))
     at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java(Compiled Code))
     at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java(Compiled Code))
     at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java(Compiled Code))
     at com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java(Compiled Code))
     at com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java(Compiled Code))
     at com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java(Compiled Code))
     at com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java(Compiled Code))
     at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled Code))
this error is occuring on concurrent clicks suppose three users click on same button then it is possible that one user gets correct output while other users experience this exception
Also one more point we are not using web.xml for calling servlets we are directly calling it giving its full path.
Thanks

LGWR generated too many logs.

Hi. I need your help.
My LGWR processor generated too many logs in trace file. Like..
WAIT #0: nam='rdbms ipc message' ela= 295 timeout=85 p2=0 p3=0 obj#=-1 tim=5887168973687
WAIT #0: nam='rdbms ipc message' ela= 490 timeout=85 p2=0 p3=0 obj#=-1 tim=5887168974217
WAIT #0: nam='log file parallel write' ela= 125 files=2 blocks=4 requests=2 obj#=-1 tim=5887168974443
WAIT #0: nam='rdbms ipc message' ela= 268 timeout=85 p2=0 p3=0 obj#=-1 tim=5887168974750
WAIT #0: nam='rdbms ipc message' ela= 394 timeout=84 p2=0 p3=0 obj#=-1 tim=5887168975184
WAIT #0: nam='log file parallel write' ela= 102 files=2 blocks=4 requests=2 obj#=-1 tim=5887168975384
WAIT #0: nam='rdbms ipc message' ela= 362 timeout=84 p2=0 p3=0 obj#=-1 tim=5887168975785
WAIT #0: nam='rdbms ipc message' ela= 464 timeout=84 p2=0 p3=0 obj#=-1 tim=5887168976291
WAIT #0: nam='log file parallel write' ela= 137 files=2 blocks=4 requests=2 obj#=-1 tim=5887168976519
WAIT #0: nam='rdbms ipc message' ela= 273 timeout=84 p2=0 p3=0 obj#=-1 tim=5887168976830
and trace file size is 30 GB. ;(
I checked average wait time 'log file parallel write' and it is '0'.
What is problem..? Somebody know about this issue??
Please help me.
dbms version : 10.2.0.4.0
OS : SUN sparc
글 수정: 82Star

82Star wrote:
There is no errors in alertlog.
But LGWR is still generating that messages in trace file. (About 6 lines per sec)
Is it normal?No, it's not normal. Is lgwr the only process doing this ? If so it looks as if someone may have used oradebug or dbms_monitor to enable extend tracing on the log writer process.
Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
Author: <b><em>Oracle Core</em></b>

Intermittent too many open files error and Invalid TLV error

Post Author: jam2008
CA Forum: General
I'm writing this up in the hopes of saving someone else a couple of days of hair-pulling...
Environment: Crystal Reports XI Enterprise / also runtime via Accpac ERP 5.4
Invalid TLV error in Accpac
"too many open files" error in event.log file
Situation:
Invalid TLV error occurs seemingly randomly on report created in CR Professional 11. Several days of troubleshooting finally lead to the following diagnosis:
This error occurs in a report that contains MORE THAN 1 bitmap image.
The error only shows up after 20 or more reports have been generated sequentially, WITHOUT CLOSING the application that is calling the report. In our case the Invoice Report dialog within Accpac. This same error occurred in a custom 3rd party VB.NET app that also called the report through an Accpac API.

after getting this message you need to do 2 things:
1. delete the current workspace because it contains some bad data in one the config files - failure to delete the workspace will result the error message to appear even if trying to upload a single file.
2. add to DTR files in groups - no more than 500 in a single add.

Too many open files in system cause database goes down

Hello experts I am very worry because of the following problems. I really hope you can help me.
some server features
OS: Suse Linux Enterprise 10
RAM: 32 GB
CPU: intel QUAD-CORE
DB: There is 3 instances RAC databases (version 11.1.0.7) in the same host.
Problem: The database instances begin to report Error message: Linux-x86_64 Error: 23: Too many open files in system
and here you are other error messages:
ORA-27505: IPC error destroying a port
ORA-27300: OS system dependent operation:close failed with status: 9
ORA-27301: OS failure message: Bad file descriptor
ORA-27302: failure occurred at: skgxpdelpt1
ORA-01115: IO error reading block from file 105 (block # 18845)
ORA-01110: data file 105: '+DATOS/dac/datafile/auditoria.519.738586803'
ORA-15081: failed to submit an I/O operation to a disk
At the same time I search into the /var/log/messages as root user and I the error notice me the same problem:
Feb 7 11:03:58 bls3-1-1 syslog-ng[3346]: Cannot open file /var/log/mail.err for
writing (Too many open files in system)
Feb 7 11:04:56 bls3-1-1 kernel: VFS: file-max limit 131072 reached
Feb 7 11:05:05 bls3-1-1 kernel: oracle[12766]: segfault at fffffffffffffff0 rip
0000000007c76323 rsp 00007fff466dc780 error 4
I think I get clear about the cause, maybe I need to increase the fs.file-max kernel parameter but I do not know how to set a good value. Here you are my sysctl.conf file and the limits.conf file:
sysctl.conf
kernel.shmall = 2097152
kernel.shmmax = 17179869184
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6553600
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 4194304
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 4194304
limits.conf
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

process limit
bcm@bcm-laptop:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

WL10 Compiler executable.exec error "too many open files" deploying ear

When I try to deploy an ear containing a web module and some ejb modules, I obtain this error:
<Info> <J2EE Deployment SPI> <BEA-260121> <Initiating deploy operation for application, MB-ADM_EAR [archive: /wl/wlments/MB-ADM_EAR/MB-ADM_EAR.ear], to Cluster1 .>
Task 1 initiated: [Deployer:149026]deploy application MB-ADM_EAR on Cluster1.
Task 1 failed: [Deployer:149026]deploy application MB-ADM_EAR on Cluster1.
Target state: deploy failed on Cluster Cluster1
java.io.IOException: Compiler failed executable.exec:
/wl/servers/MS1/cache/EJBCompilerCache/-1dj0waj53cbu8/it/apps/ejbs/core/ExSvc_167qnt_Impl.java:17: error while writing it.apps.ejbs.core.ExSvc_167qnt_Impl: /wl/servers/MS1/cache/EJBCompilerCache/-1dj0waj53cbu8/it/apps/ejbs/core/ExSvc_167qnt_Impl.class (Too many open files)If i split the ear in two parts, web in one ear and ejbs in another ear, deploy is succesfull.
Do you have any idea of what is happening?
Below the environment specifications:
JVM Version: jrockit_150_11
JVM Heap: 512
Web Logic: 10.0.1.0
Server: Hewlett Packard DL585 G2
OS: Linux / 2.6.5-7.287.3.PTF.345489.1-smp
Thank you, bye,
Marco

Hello Marco.
When you try to deploy an EAR weblogic server at the time of deployment unjar's it and compiles the files and so on. Every Operating system has a limit on how many number of files can be opened by a process. If your EAR is big then the number of files which WLS will unjar will also be large hence you hit the limit. By splitting your ear into 2, you are splitting wls task into smaller parts, which means the number of files it unjars at a time is less.
The following note tells what needs to be done to avert this issue.
http://download.oracle.com/docs/cd/E12839_01/doc.1111/e14772/weblogic_server_issues.htm#CHDGFFHD

"Too many open files" - Mac OS X 10.7.5

I'm getting errors about "Too many open files" when trying to use Berkeley DB on Mac OS X 10.7.5.
The BerkeleyDB site mentions this, but it goes back to 2003 and refers to a file which doesn’t exist in OS X 10.7.5:
http://docs.oracle.com/cd/E17076_03/html/installation/build_unix_macosx.html
I tried ulimit, but it doesn't seem to have any affect:
$ sudo ulimit -n 1024 1024
Password:
$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited
Any suggestions?
Thanks,
James.

sudo sysctl -w kern.maxfiles=32768
sudo sysctl -w kern.maxprocperuid=16384
seemed to work.
Created file /etc/sysctl.conf containing:
kern.maxfiles=32768
kern.maxprocperuid=16384
so that the setting would survive a reboot.
Hopefully that permanently fixes it.

DrawLine(...) method generates too many in[] instances

My appplication performs very intensive painting (some kind of real time charts). It often invokes drawLine(...) method of Graphics class. Under profiler I have found out that it instantiate too many int[] (about 300000 every few seconds). Though garbage collector cleans these auxilary instances well, I don't like when it runs every time. How can I write my own drawLine method, which doesn't generate so many auxilary instances? By the way I need to draw only vertical or horizontal lines.

Are you talking about drawing a grid?
If so, 300000 sounds / few seconds sounds like there
is something wrong.
Suppose your screen is 1000x1000 and you are drawing
lines every 10 pixels apart. That is 100 lines in
both directions for 200 lines.
200 lines and 4 ints per call is 800. Assume 100000 /
second, then that means you are calling repaint 125
times / second?
I have about 40 - 50 JFrames, each of them contains panel, which performs painting. Repaint invokes 10-50 times every second for each JFrame. Each repaint draws 1200 lines every time it is invoked. And int[] instances are created inside Swing version of drawLine(...) method, at any case Borland Optimizeit 5.0 says so.
If anything, I would make a temporary line object:
private Line2D line = new Line2D.Double();
public void paint(Graphics _g) {
super.paint(_g);
Graphics2D g = (Graphics2D) _g;
for each horizontal and vertical line:
line.setLine(x1, y1, x2, y2);
g.draw(line);
I've tried your code. Thanks a lot, but it generates too many int[] instances too:(.
>
Also, make sure you are not drawing lines outside of
the area that needs to be repainted. That is, only
draw a line if it would intersect the viewable
rectangle of the panel, or even better if it
intersects the getClipBounds() of the graphics
object.

Java.io.IOException: Too many open files while deploying in soa 11g

hi all,
I am getting a strange error while deploying any composite .. it's a hello world kinda composite but while i am trying to deploy it i am getting "java.io.IOException: Too many open files" while deployment.. i have tried to deploy it in 2-3 ways but all of them resulted in the same error..bouncing the soa server might be an option but can someone give an insight as why it is happening and can it be resolved without restarting the server..
Thanks

yes..so this problem is with unix only ..coz i previously worked in Windows ..never got this problem..

STARTING DATABASE : PROBLEM OF Linux Error: 23: Too many open files in syst

Hi everybody,
I am running an RMAN script and get this error,
9> @/u01/app/oracle/admin/devpose/backup/configuration.rcv
RMAN> ###################################################################
2> # Configuration file used to set Rman policies.
3> #
4> ###################################################################
5>
6> CONFIGURE DEFAULT DEVICE TYPE TO DISK;
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of configure command at 08/26/2009 20:03:30
RMAN-06403: could not obtain a fully authorized session
ORA-01034: ORACLE not available
RMAN> CONFIGURE RETENTION POLICY TO REDUNDANCY 1;
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of configure command at 08/26/2009 20:03:30
RMAN-06403: could not obtain a fully authorized session
ORA-01034: ORACLE not available
RMAN> #CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;
2> CONFIGURE DEVICE TYPE DISK PARALLELISM 2;
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of configure command at 08/26/2009 20:03:30
RMAN-06403: could not obtain a fully authorized session
ORA-01034: ORACLE not available
RMAN>
RMAN> CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '/u01/app/oracle/backup/db/ora_df%t_s%s_s%p';
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of configure command at 08/26/2009 20:03:30
RMAN-06403: could not obtain a fully authorized session
ORA-01034: ORACLE not available
But this problem is understandable, as the database is not running. The main problem why database is not running, I have found the reason but do not understand how to solve the problem.
Since, the database was not running, I tried to startup the database, I then came across the following which is my problem (Why so many files are open? Linux OS error says too many files open. See below,
SQL> conn /as sysdba
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 419430400 bytes
Fixed Size 779516 bytes
Variable Size 258743044 bytes
Database Buffers 159383552 bytes
Redo Buffers 524288 bytes
Database mounted.
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/u01/app/oracle/oradata/devpose/redo02.log'
ORA-27041: unable to open file
Linux Error: 23: Too many open files in system
Can anybody has run into such problem and guide me to a solution, please?
Thanks

Hi,
yes, this DB was functioning o.k. this configuration script was part of RMAN daily backup.
Last night the backup failed. So, when I opened "Failed job" in the EM, I saw this type of messages.
That was the starting point. Gradually, I tried to narrow down on to the actual problem and found the findings as I have posted.
One way of sovling problem, I thought that, all these processes I would kill and then try to open the database, it might startup. However, that wouldnot lead me in ensuring this won't occur again.
That's why I am trying to understand why it should open, so many processes (why spawn so many .flb files?) Any thoughts you have around this?
I will try to restart the OS as the last resort.
Thanks for your help and suggestions.
Regards,

"java.io.IOException: Too many open files" in LinuX

Hi Developers,
* I am continiously running and processing more than 2000 XML files by using SAX and DOM.....
* My process is as follows,
- Converting the XML file as Document object by DOM....
- And that DOM will be used while creating log file report, that log file will be created after executing all XML files..
* After processing approx 1000 files, it throws *"java.io.IOException: Too many open files" in LinuX system* ....
* I have googled more and more in all sites including sun forum also, but they are telling only to increase the system config by ULIMIT in linux....If i increase that its executing well without exception........
* My question is, Is it possible to do it by JAVA code itself or any other VM arguments like -Xms512m and -Xmx512m.....
* Please let me know , if you have any idea.....
Thanks And Regards,
JavaImran

Doh! I forgot to post my little code sample...
package forums.crap;
import java.io.*;
import java.util.*;
public class TooManyFileHandles
private static final int HOW_MANY = 8*1024;
public static void main(String[] args) {
    List<PrintWriter> writers = new ArrayList<PrintWriter>(HOW_MANY);
    try {
      try {
        for (int i=1; i<=HOW_MANY; i++ ) {
          writers.add(new PrintWriter("file"+i+".txt"));
      } finally {
        for (PrintWriter w : writers) {
          if(w!=null)w.close();
    } catch (Exception e) {
      e.printStackTrace();
}... and the problem still isn't OOME ;-)
Cheers. Keith.

WLS 10.3.5 on RHEL 5.4, SocketException: Too many open files

Hi
I'm running Weblogic server 10.3.5 on Red Hat Enterprise Linux Server release 5.4 (Tikanga), with Java jdk1.6.0_27.
My order handling application, when receiving client orders, needs to make outbound SOAP calls to fulfill the order. During a performance test, we got following errors:
####<Feb 10, 2012 2:28:41 PM ICT> <Critical> <Server> <KKMOMAPP2> <KKMOMPE2> <DynamicListenThread[Default]> <<WLS Kernel>> <> <> <1328858921806> <BEA-002616> <Failed to listen on channel "Default" on 172.24.106.81:4095, failure count: 1, failing for 0 seconds, java.net.SocketException: Too many open files>
I monitored the java process of this application, when the "Too many open files" error happened, it had 1388 open file descriptors, among which 655 were sockets.
I also monitored the total open file descriptors of the weblogic user account, the count was around 6300 during this error.
These numbers are far smaller than the file limits configured on OS:
- Under weblogic account, ulimit -n shows 65536
- /proc/sys/fs/file-max shows 772591
- Following lines are already in /etc/security/limits.conf
weblogic soft nofile 65536
weblogic hard nofile 65536
weblogic soft nproc 16384
weblogic hard nproc 16384
I did another test using a simple java program to open large number of sockets under weblogic account. It has no problem to open 15,000 sockets. It seems the file descriptor limit is indeed quite high, but for some reasons, the Weblogic process fails even when it has merely 1388 open files. Are there other Linux or Weblogic parameters I should tune? Or anything else I missed?
Thank you very much
Ning

Hi All,
Any help on this issue ?
Thank you,
Ram

Java.util.zip.ZipException: Too many open files on Linux

Hi,
We have web application running on Caucho's resin server on jdk 1.5.0_11 and Red hat Linux. We are noticing that java process is running out of file handles within 24-30 hours. We have file limit of 5000 which it consumes in 24 hours throwing 'java.util.zip.ZipException: Too many open files'.
I have made sure all sorts of file handles are closed from application point of view. Here is the snapshot of lsof (list of file handles) from java process. The following list keeps growing until it runs out of limit. Do you have tips/suggestions on how to mitigate this problem (considering we dont want to increase ulimit for this process)? Also, can you make out any thing more from the description of file handles like, are they unclosed POP3 connections or URL connection to external sites?
java 7156 resin 120u IPv4 34930051 UDP localhost.localdomain:59693
java 7156 resin 121u IPv4 34927823 UDP localhost.localdomain:59663
java 7156 resin 122u IPv4 34931861 UDP localhost.localdomain:59739
java 7156 resin 123u IPv4 34932023 UDP localhost.localdomain:59745
java 7156 resin 124u IPv4 34930054 UDP localhost.localdomain:59700
java 7156 resin 125u IPv4 34927826 UDP localhost.localdomain:59665
java 7156 resin 126u IPv4 34927829 UDP localhost.localdomain:59666
java 7156 resin 127u IPv4 34930057 UDP localhost.localdomain:59703
java 7156 resin 128u IPv4 34930713 UDP localhost.localdomain:59727
java 7156 resin 129u IPv4 34930716 UDP localhost.localdomain:59730
java 7156 resin 130u IPv4 34932238 UDP localhost.localdomain:59789
java 7156 resin 131u IPv4 34932026 UDP localhost.localdomain:59749
java 7156 resin 132u IPv4 34932221 UDP localhost.localdomain:59770
java 7156 resin 133u IPv4 34932224 UDP localhost.localdomain:59775
java 7156 resin 134u IPv4 34932029 UDP localhost.localdomain:59753
java 7156 resin 135u IPv4 34932032 UDP localhost.localdomain:59754
java 7156 resin 138u IPv4 34932035 UDP localhost.localdomain:59760
java 7156 resin 139u IPv4 34932038 UDP localhost.localdomain:59763
java 7156 resin 140u IPv4 34932227 UDP localhost.localdomain:59780
java 7156 resin 141u IPv4 34932230 UDP localhost.localdomain:59781
java 7156 resin 144u IPv4 34932234 UDP localhost.localdomain:59786
java 7156 resin 146u IPv4 34932241 UDP localhost.localdomain:59792
java 7156 resin 147u IPv4 34932247 UDP localhost.localdomain:59802

Finally we resolved this issue. It was oracle driver which had some compatibility issue, we upgraded our Oracle client driver to newer version, and this fixed the problem. Base line, there was nothing wrong with application code, code was doing good resource clean up, but oracle driver was leaking handles per every connection.

Runtime.exec - Too Many Open Files

System version : Red Hat Enterprise Linux 2.4.21-47.ELsmp AS release 3 (Taroon Update 8)
JRE version : 1.6.0-b105
Important : the commands described below are launched from a Web application : Apache Tomcat 6.0.10
Hello,
I'm facing a problem already known, but appearantly never really solved ??!! ;)
When I invoke many system commands with the 'Runtime.exec(...)' method, there are open files that are not released (I can see them with the "lsof" system command) .
At the end, the unavoidable "too many open files" Exception.
The lauched commands are "ssh ... " commands.
In the topics relating to this problem, the solution is always to close all Streams / threads and to explicitely invoke the method "Process.destroy()".
My problem is that this is what I do ! And I can't do more...
Here is the code :
       Runtime rt = Runtime.getRuntime();
       Process process = rt.exec("ssh ...");
        // ProcessStreamHolder extends Thread and reads from the InputStream given in constructor...
        ProcessStreamHolder errorStream = new ProcessStreamHolder(process.getErrorStream());
        ProcessStreamHolder outputStream = new ProcessStreamHolder(process.getInputStream());
        errorStream.start();
        outputStream.start();
        exitValue = process.waitFor();
        try {
            errorStream.interrupt();
        } catch (RuntimeException e) {
            logger.warn("...");
        try {
            outputStream.interrupt();
        } catch (RuntimeException e) {
            logger.warn("...");
        try {
            process.getInputStream().close();
        } catch (RuntimeException e) {
            logger.warn("...");
        try {
            process.getOutputStream().close();
        } catch (RuntimeException e) {
            logger.warn("...");
        try {
            process.getErrorStream().close();
        } catch (RuntimeException e) {
            logger.warn("...");
        process.destroy();Does someone know if my code is wrong or if there's a workaround for me ?
Thanks by advance !
Richard.

Don't interrupt those threads. Close the output stream first, then wait for the process to exit, then both threads reading the stdout and stderr of the process should get EOFs, so they should exit naturally, and incidentally close the streams themselves.

"Too many open files" Exception on "tapestry-framework-4.1.1.jar"

When a browser attempts accessing to my webwork, the server opens a certain number of file descriptors to "tapestry-framework-4.1.1.jar" file and don't release them for a while.
Below is the output from "lsof | grep tapestry":
java 26735 root mem REG 253,0 62415 2425040 /usr/local/apache-tomcat-5.5.20/my_webwork/WEB-INF/lib/tapestry-portlet-4.1.1.jar
java 26735 root mem REG 253,0 2280602 2425039 /usr/local/apache-tomcat-5.5.20/my_webwork/WEB-INF/lib/tapestry-framework-4.1.1.jar
java 26735 root mem REG 253,0 320546 2425036 /usr/local/apache-tomcat-5.5.20/my_webwork/WEB-INF/lib/tapestry-contrib-4.1.1.jar
java 26735 root mem REG 253,0 49564 2424979 /usr/local/apache-tomcat-5.5.20/my_webwork/WEB-INF/lib/tapestry-annotations-4.1.1.jar
java 26735 root 28r REG 253,0 2280602 2425039 /usr/local/apache-tomcat-5.5.20/my_webwork/WEB-INF/lib/tapestry-framework-4.1.1.jar
java 26735 root 29r REG 253,0 2280602 2425039 /usr/local/apache-tomcat-5.5.20/my_webwork/WEB-INF/lib/tapestry-framework-4.1.1.jar
java 26735 root 30r REG 253,0 2280602 2425039 /usr/local/apache-tomcat-5.5.20/my_webwork/WEB-INF/lib/tapestry-framework-4.1.1.jar
These unknown references are sometimes released automatically, but sometimes not.
And I get "Too many open files" exception after using my application for a few hours.
The number of the unknown references increases as I access to my webwork or just hit on "F5" key on my browser to reload it.
I tried different types of browsers to see if I could see any differences in consequence, and in fact it differed by the browser I used.
When viewed by Internet Explorer it increased by 3 for every access.
On the other hand it increased by 7 for each attempt when accessed by FireFox.
I have already tried optimizing the max number of file discriptors, and it solved the "Too many open files" exception.
But stil I'm wondering who actually is opening "tapestry-framework-4.1.1.jar" this many.
Could anyone figure out what is going on?
Thanks in advance.
The following is my environmental version info:
- Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
- Java: 1.5.0_11
- Tomcat: 5.5.20
- Tapestry: 4.1.1

Hi,
Cause might The server got an exception while trying to accept client connections. It will try to backoff to aid recovery.
The OS limit for the number of open file descriptor (FD limit) needs to be increased. Tune OS parameters that might help the server to accept more client connections (e.g. TCP accept back log).
http://e-docs.bea.com/wls/docs90/messages/Server.html#BEA-002616
Regards,
Prasanna Yalam

OEM Grid generating too many core files

Similar Messages

Maybe you are looking for