Too many open files in system cause database goes down

Hello experts I am very worry because of the following problems. I really hope you can help me.
some server features
OS: Suse Linux Enterprise 10
RAM: 32 GB
CPU: intel QUAD-CORE
DB: There is 3 instances RAC databases (version 11.1.0.7) in the same host.
Problem: The database instances begin to report Error message: Linux-x86_64 Error: 23: Too many open files in system
and here you are other error messages:
ORA-27505: IPC error destroying a port
ORA-27300: OS system dependent operation:close failed with status: 9
ORA-27301: OS failure message: Bad file descriptor
ORA-27302: failure occurred at: skgxpdelpt1
ORA-01115: IO error reading block from file 105 (block # 18845)
ORA-01110: data file 105: '+DATOS/dac/datafile/auditoria.519.738586803'
ORA-15081: failed to submit an I/O operation to a disk
At the same time I search into the /var/log/messages as root user and I the error notice me the same problem:
Feb 7 11:03:58 bls3-1-1 syslog-ng[3346]: Cannot open file /var/log/mail.err for
writing (Too many open files in system)
Feb 7 11:04:56 bls3-1-1 kernel: VFS: file-max limit 131072 reached
Feb 7 11:05:05 bls3-1-1 kernel: oracle[12766]: segfault at fffffffffffffff0 rip
0000000007c76323 rsp 00007fff466dc780 error 4
I think I get clear about the cause, maybe I need to increase the fs.file-max kernel parameter but I do not know how to set a good value. Here you are my sysctl.conf file and the limits.conf file:
sysctl.conf
kernel.shmall = 2097152
kernel.shmmax = 17179869184
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6553600
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 4194304
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 4194304
limits.conf
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

process limit
bcm@bcm-laptop:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Similar Messages

Linux Error: 23: Too many open files in system

I crashed the my oracle instance with the following error:
Tue Feb 13 22:15:16 2001
Errors in file /home/oracle/product/8.1.6/admin/v2qa1/bdump/lgwr_14175.trc:
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/db03/v2qa1/system/log/redo02.log'
ORA-27041: unable to open file
Linux Error: 23: Too many open files in system
Additional information: 2
LGWR: terminating instance due to error 313
Instance terminated by LGWR, pid = 14175
Is the number of open files adjustable?
Why am I opening files?
Could the fact that our java stored procedures are trying, unsuccessfully, be leaving files open?
Thanks - Craig
null

increasing the oracle users ulimit to unlimited fixed that error for me. Type
ulimit -a
and look at the open files parameter. See your linux doc's to increase it to unlimited (will probably be increased from 1024 to 4096).

STARTING DATABASE : PROBLEM OF Linux Error: 23: Too many open files in syst

Hi everybody,
I am running an RMAN script and get this error,
9> @/u01/app/oracle/admin/devpose/backup/configuration.rcv
RMAN> ###################################################################
2> # Configuration file used to set Rman policies.
3> #
4> ###################################################################
5>
6> CONFIGURE DEFAULT DEVICE TYPE TO DISK;
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of configure command at 08/26/2009 20:03:30
RMAN-06403: could not obtain a fully authorized session
ORA-01034: ORACLE not available
RMAN> CONFIGURE RETENTION POLICY TO REDUNDANCY 1;
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of configure command at 08/26/2009 20:03:30
RMAN-06403: could not obtain a fully authorized session
ORA-01034: ORACLE not available
RMAN> #CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;
2> CONFIGURE DEVICE TYPE DISK PARALLELISM 2;
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of configure command at 08/26/2009 20:03:30
RMAN-06403: could not obtain a fully authorized session
ORA-01034: ORACLE not available
RMAN>
RMAN> CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '/u01/app/oracle/backup/db/ora_df%t_s%s_s%p';
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of configure command at 08/26/2009 20:03:30
RMAN-06403: could not obtain a fully authorized session
ORA-01034: ORACLE not available
But this problem is understandable, as the database is not running. The main problem why database is not running, I have found the reason but do not understand how to solve the problem.
Since, the database was not running, I tried to startup the database, I then came across the following which is my problem (Why so many files are open? Linux OS error says too many files open. See below,
SQL> conn /as sysdba
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 419430400 bytes
Fixed Size 779516 bytes
Variable Size 258743044 bytes
Database Buffers 159383552 bytes
Redo Buffers 524288 bytes
Database mounted.
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/u01/app/oracle/oradata/devpose/redo02.log'
ORA-27041: unable to open file
Linux Error: 23: Too many open files in system
Can anybody has run into such problem and guide me to a solution, please?
Thanks

Hi,
yes, this DB was functioning o.k. this configuration script was part of RMAN daily backup.
Last night the backup failed. So, when I opened "Failed job" in the EM, I saw this type of messages.
That was the starting point. Gradually, I tried to narrow down on to the actual problem and found the findings as I have posted.
One way of sovling problem, I thought that, all these processes I would kill and then try to open the database, it might startup. However, that wouldnot lead me in ensuring this won't occur again.
That's why I am trying to understand why it should open, so many processes (why spawn so many .flb files?) Any thoughts you have around this?
I will try to restart the OS as the last resort.
Thanks for your help and suggestions.
Regards,

"java.io.IOException: Too many open files" in LinuX

Hi Developers,
* I am continiously running and processing more than 2000 XML files by using SAX and DOM.....
* My process is as follows,
- Converting the XML file as Document object by DOM....
- And that DOM will be used while creating log file report, that log file will be created after executing all XML files..
* After processing approx 1000 files, it throws *"java.io.IOException: Too many open files" in LinuX system* ....
* I have googled more and more in all sites including sun forum also, but they are telling only to increase the system config by ULIMIT in linux....If i increase that its executing well without exception........
* My question is, Is it possible to do it by JAVA code itself or any other VM arguments like -Xms512m and -Xmx512m.....
* Please let me know , if you have any idea.....
Thanks And Regards,
JavaImran

Doh! I forgot to post my little code sample...
package forums.crap;
import java.io.*;
import java.util.*;
public class TooManyFileHandles
private static final int HOW_MANY = 8*1024;
public static void main(String[] args) {
    List<PrintWriter> writers = new ArrayList<PrintWriter>(HOW_MANY);
    try {
      try {
        for (int i=1; i<=HOW_MANY; i++ ) {
          writers.add(new PrintWriter("file"+i+".txt"));
      } finally {
        for (PrintWriter w : writers) {
          if(w!=null)w.close();
    } catch (Exception e) {
      e.printStackTrace();
}... and the problem still isn't OOME ;-)
Cheers. Keith.

Runtime.exec - Too Many Open Files

System version : Red Hat Enterprise Linux 2.4.21-47.ELsmp AS release 3 (Taroon Update 8)
JRE version : 1.6.0-b105
Important : the commands described below are launched from a Web application : Apache Tomcat 6.0.10
Hello,
I'm facing a problem already known, but appearantly never really solved ??!! ;)
When I invoke many system commands with the 'Runtime.exec(...)' method, there are open files that are not released (I can see them with the "lsof" system command) .
At the end, the unavoidable "too many open files" Exception.
The lauched commands are "ssh ... " commands.
In the topics relating to this problem, the solution is always to close all Streams / threads and to explicitely invoke the method "Process.destroy()".
My problem is that this is what I do ! And I can't do more...
Here is the code :
       Runtime rt = Runtime.getRuntime();
       Process process = rt.exec("ssh ...");
        // ProcessStreamHolder extends Thread and reads from the InputStream given in constructor...
        ProcessStreamHolder errorStream = new ProcessStreamHolder(process.getErrorStream());
        ProcessStreamHolder outputStream = new ProcessStreamHolder(process.getInputStream());
        errorStream.start();
        outputStream.start();
        exitValue = process.waitFor();
        try {
            errorStream.interrupt();
        } catch (RuntimeException e) {
            logger.warn("...");
        try {
            outputStream.interrupt();
        } catch (RuntimeException e) {
            logger.warn("...");
        try {
            process.getInputStream().close();
        } catch (RuntimeException e) {
            logger.warn("...");
        try {
            process.getOutputStream().close();
        } catch (RuntimeException e) {
            logger.warn("...");
        try {
            process.getErrorStream().close();
        } catch (RuntimeException e) {
            logger.warn("...");
        process.destroy();Does someone know if my code is wrong or if there's a workaround for me ?
Thanks by advance !
Richard.

Don't interrupt those threads. Close the output stream first, then wait for the process to exit, then both threads reading the stdout and stderr of the process should get EOFs, so they should exit naturally, and incidentally close the streams themselves.

"Too many open files" Exception on "tapestry-framework-4.1.1.jar"

When a browser attempts accessing to my webwork, the server opens a certain number of file descriptors to "tapestry-framework-4.1.1.jar" file and don't release them for a while.
Below is the output from "lsof | grep tapestry":
java 26735 root mem REG 253,0 62415 2425040 /usr/local/apache-tomcat-5.5.20/my_webwork/WEB-INF/lib/tapestry-portlet-4.1.1.jar
java 26735 root mem REG 253,0 2280602 2425039 /usr/local/apache-tomcat-5.5.20/my_webwork/WEB-INF/lib/tapestry-framework-4.1.1.jar
java 26735 root mem REG 253,0 320546 2425036 /usr/local/apache-tomcat-5.5.20/my_webwork/WEB-INF/lib/tapestry-contrib-4.1.1.jar
java 26735 root mem REG 253,0 49564 2424979 /usr/local/apache-tomcat-5.5.20/my_webwork/WEB-INF/lib/tapestry-annotations-4.1.1.jar
java 26735 root 28r REG 253,0 2280602 2425039 /usr/local/apache-tomcat-5.5.20/my_webwork/WEB-INF/lib/tapestry-framework-4.1.1.jar
java 26735 root 29r REG 253,0 2280602 2425039 /usr/local/apache-tomcat-5.5.20/my_webwork/WEB-INF/lib/tapestry-framework-4.1.1.jar
java 26735 root 30r REG 253,0 2280602 2425039 /usr/local/apache-tomcat-5.5.20/my_webwork/WEB-INF/lib/tapestry-framework-4.1.1.jar
These unknown references are sometimes released automatically, but sometimes not.
And I get "Too many open files" exception after using my application for a few hours.
The number of the unknown references increases as I access to my webwork or just hit on "F5" key on my browser to reload it.
I tried different types of browsers to see if I could see any differences in consequence, and in fact it differed by the browser I used.
When viewed by Internet Explorer it increased by 3 for every access.
On the other hand it increased by 7 for each attempt when accessed by FireFox.
I have already tried optimizing the max number of file discriptors, and it solved the "Too many open files" exception.
But stil I'm wondering who actually is opening "tapestry-framework-4.1.1.jar" this many.
Could anyone figure out what is going on?
Thanks in advance.
The following is my environmental version info:
- Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
- Java: 1.5.0_11
- Tomcat: 5.5.20
- Tapestry: 4.1.1

Hi,
Cause might The server got an exception while trying to accept client connections. It will try to backoff to aid recovery.
The OS limit for the number of open file descriptor (FD limit) needs to be increased. Tune OS parameters that might help the server to accept more client connections (e.g. TCP accept back log).
http://e-docs.bea.com/wls/docs90/messages/Server.html#BEA-002616
Regards,
Prasanna Yalam

RE: (forte-users) [UNIX] "Too many open files" 3.0.M2question

Actually, the stuff works in interpreted mode.
It's only when having the server partition compiled that this happen.
j-p
-----Message d'origine-----
De: Adamek, Zenon [mailto:ZAdamekpurolator.com]
Date: lundi 25 septembre 2000 17:13
À: 'Jean-Paul.Gabriellisema.fr'
Cc: Forte-userslists.xpedior.com
Objet: RE: (forte-users) [UNIX] "Too many open files" 3.0.M2 question
see Technote 10981
-----Original Message-----
From: Jean-Paul Gabrielli [SMTP:Jean-Paul.Gabriellisema.fr]
Sent: Monday, September 25, 2000 11:02 AM
To: zeForte-users
Subject: (forte-users) [UNIX] "Too many open files" 3.0.M2 question
Hi,
running a server partition that reads a configuration file,
and apparently doen't close it after, I have that exception:
SYSTEM ERROR: System Error: Too many open files, opening '....'with mode
'r'
Class: qqos_FileResourceException
1) Is there such a limit, or does this rely only on the OS one ?
2) How is this error not trapped, as I only got itinteractively, whereas
my server log does a exception trap/segmentation fault,
thanlks
j-p
For the archives, go to: http://lists.xpedior.com/forte-users and use
the login: forte and the password: archive. To unsubscribe,send in a new
email the word: 'Unsubscribe' to: forte-users-requestlists.xpedior.com

Hi Jean-Paul,
As described in the Technote 10981 some Forte programs (Nodemanager and
router) handle correct the high-file descriptor-use problem. It is possible
that Forte interpreter do it correct too.
Zenon
-----Original Message-----
From: Jean-Paul Gabrielli [SMTP:Jean-Paul.Gabriellisema.fr]
Sent: Monday, September 25, 2000 12:11 PM
To: Adamek, Zenon
Cc: Forte-userslists.xpedior.com
Subject: RE: (forte-users) [UNIX] "Too many open files" 3.0.M2
question
Actually, the stuff works in interpreted mode.
It's only when having the server partition compiled that this happen.
j-p
-----Message d'origine-----
De: Adamek, Zenon [mailto:ZAdamekpurolator.com]
Date: lundi 25 septembre 2000 17:13
À: 'Jean-Paul.Gabriellisema.fr'
Cc: Forte-userslists.xpedior.com
Objet: RE: (forte-users) [UNIX] "Too many open files" 3.0.M2 question
see Technote 10981
-----Original Message-----
From: Jean-Paul Gabrielli [SMTP:Jean-Paul.Gabriellisema.fr]
Sent: Monday, September 25, 2000 11:02 AM
To: zeForte-users
Subject: (forte-users) [UNIX] "Too many open files" 3.0.M2 question
Hi,
running a server partition that reads a configuration file,
and apparently doen't close it after, I have that exception:
SYSTEM ERROR: System Error: Too many open files, opening '....'with mode
'r'
Class: qqos_FileResourceException
1) Is there such a limit, or does this rely only on the OS one ?
2) How is this error not trapped, as I only got itinteractively, whereas
my server log does a exception trap/segmentation fault,
thanlks
j-p
For the archives, go to: http://lists.xpedior.com/forte-users and use
the login: forte and the password: archive. To unsubscribe,send in a new
email the word: 'Unsubscribe' to:
forte-users-requestlists.xpedior.com
>
For the archives, go to: http://lists.xpedior.com/forte-users and use
the login: forte and the password: archive. To unsubscribe, send in a new
email the word: 'Unsubscribe' to: forte-users-requestlists.xpedior.com

WL10 Compiler executable.exec error "too many open files" deploying ear

When I try to deploy an ear containing a web module and some ejb modules, I obtain this error:
<Info> <J2EE Deployment SPI> <BEA-260121> <Initiating deploy operation for application, MB-ADM_EAR [archive: /wl/wlments/MB-ADM_EAR/MB-ADM_EAR.ear], to Cluster1 .>
Task 1 initiated: [Deployer:149026]deploy application MB-ADM_EAR on Cluster1.
Task 1 failed: [Deployer:149026]deploy application MB-ADM_EAR on Cluster1.
Target state: deploy failed on Cluster Cluster1
java.io.IOException: Compiler failed executable.exec:
/wl/servers/MS1/cache/EJBCompilerCache/-1dj0waj53cbu8/it/apps/ejbs/core/ExSvc_167qnt_Impl.java:17: error while writing it.apps.ejbs.core.ExSvc_167qnt_Impl: /wl/servers/MS1/cache/EJBCompilerCache/-1dj0waj53cbu8/it/apps/ejbs/core/ExSvc_167qnt_Impl.class (Too many open files)If i split the ear in two parts, web in one ear and ejbs in another ear, deploy is succesfull.
Do you have any idea of what is happening?
Below the environment specifications:
JVM Version: jrockit_150_11
JVM Heap: 512
Web Logic: 10.0.1.0
Server: Hewlett Packard DL585 G2
OS: Linux / 2.6.5-7.287.3.PTF.345489.1-smp
Thank you, bye,
Marco

Hello Marco.
When you try to deploy an EAR weblogic server at the time of deployment unjar's it and compiles the files and so on. Every Operating system has a limit on how many number of files can be opened by a process. If your EAR is big then the number of files which WLS will unjar will also be large hence you hit the limit. By splitting your ear into 2, you are splitting wls task into smaller parts, which means the number of files it unjars at a time is less.
The following note tells what needs to be done to avert this issue.
http://download.oracle.com/docs/cd/E12839_01/doc.1111/e14772/weblogic_server_issues.htm#CHDGFFHD

WLS 92MP1: Application Poller issue Too many open files

Hi,
We have a wls92mp1 domain on linux AS4(64bit) with Sun jdk 1.5.0_14. It contains only Admin server where we have deployed the application. Over a period of time the server start showing up below message in the logs. We have not deployed the application from autodeploy directory. And the file "/home/userid/wls92/etg/servers/userid_a/cache/.app_poller_lastrun " is available in the location, still it throws FileNotFoundException.
<Error> <Application Poller> <BEA-149411> <I/O exception encountered java.io.FileNotFoundException: /home/userid/wls92/etg/servers/userid_a/cache/.a
pp_poller_lastrun (Too many open files).
java.io.FileNotFoundException: /home/userid/wls92/etg/servers/userid_a/cache/.app_poller_lastrun (Too many open files)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
at java.io.FileWriter.<init>(FileWriter.java:73)
at weblogic.management.deploy.GenericAppPoller.setLastRunFileMap(GenericAppPoller.java:423)
Any help regarding this would be highly appreciated.
Thanks.

Hi,
By above seeing in error, this error code (BEA-149411) describe the following
149411: I/O exception encountered {0}.
L10n Package: weblogic.management.deploy.internal
I18n Package: weblogic.management.deploy.internal
Subsystem: Application Poller
Severity: Error
Stack Trace: true
Message Detail: An I/O exception denotes a failure to perform a read/write operation on the application files.
Cause: An I/O exception can occur during file read/write while deploying an application.
Action: Take corrective action based on the exception message details.
i think it helps u.
-abhi

Getting java.io.IOException: Too many open files+ClassNotFoundException

Dear All
We have a web application deployed on Rational Application Developer 6.0 (Operating System is Windows 2000 professional) our users are randomly getting java.io.IOException: Too many open files+ClassNotFoundException when they click on some servlet or jsp link, but they are getting this error randomly for example when they click some link they may get these exceptions but refreshing page or clicking once again on same link executes servlet successfully.If anyone could help on this topic we will be grateful
Thanks

I think this these two exceptions are occuring in differrent environment
java.io.IOException is occuring under heavy load to web server its stack trace is as follows:
JSPG0225E: An error occurred at line: 2 in the statically included file: /SessionCheck.jsp
JSPG0093E: Generated servlet error from file: /Admin/AdminInsuranceCertificates.jsp
E:\WebSphere_6\AppServer\profiles\AUSECert\temp\centraNode04\server1\AUSECert\Vero.war\Admin\_AdminInsuranceCertificates.java:259: cannot access com.bplus.natmar.LoginDetails
bad class file: E:\WebSphere_6\AppServer\java\jre\lib\core.jar(java/io/Writer.class)
unable to access file: E:\WebSphere_6\AppServer\profiles\AUSECert\installedApps\centraNode04Cell\AUSECert.ear\Vero.war\WEB-INF\classes\com\bplus\natmar\LoginDetails.class (Too many open files)
Please remove or make sure it appears in the correct subdirectory of the classpath.
(source unavailable)
1 error
]: com.ibm.ws.jsp.JspCoreException: JSPG0049E: /Admin/AdminInsuranceCertificates.jsp failed to compile :
this error always occurs in reference to logindetails class this clas is used to make a session check on different roles in our project for e.g., user having end user role should not be able to log in as a user having admin role
we have included a sessioncheck.jsp in our every jsp page in this jsp we have simply used logindetails class as useBean and called its getresource() method
above stacktrace is from our live application server
while testing same project on our local system we are not getting too many open files exception but we are getting following ClassNotFoundException
[11/30/05 17:11:42:797 EST] 0000004a SystemOut O SELECT count(*) as NoofRecs FROM resourcerolebindings WHERE ResourceName = 'mainEdit.jsp' and IsEndUser=1
[11/30/05 17:12:50:891 EST] 000001eb SystemOut O SELECT count(*) as NoofRecs FROM resourcerolebindings WHERE ResourceName = 'InsuranceCertificates.jsp' and IsEndUser=1
[11/30/05 17:17:40:828 EST] 0000008d SystemOut O AppURL is: http://www.VeroECert.com/Vero/indexU.jsp
[11/30/05 17:17:58:141 EST] 0000008b SystemOut O SELECT count(*) as NoofRecs FROM resourcerolebindings WHERE ResourceName = 'InsuranceCertificates.jsp' and IsEndUser=1
[11/30/05 17:20:41:703 EST] 00000034 ServletWrappe E SRVE0026E: [Servlet Error]-[com.servlet.UserHelpServlet]: java.lang.ClassNotFoundException: com.servlet.UserHelpServlet
     at com.ibm.ws.classloader.CompoundClassLoader.findClass(CompoundClassLoader.java(Compiled Code))
     at com.ibm.ws.classloader.CompoundClassLoader.loadClass(CompoundClassLoader.java(Compiled Code))
     at java.lang.ClassLoader.loadClass(ClassLoader.java(Compiled Code))
     at java.beans.Beans.instantiate(Beans.java:202)
     at java.beans.Beans.instantiate(Beans.java:63)
     at com.ibm.ws.webcontainer.servlet.ServletWrapper$3.run(ServletWrapper.java:1384)
     at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java(Compiled Code))
     at com.ibm.ws.webcontainer.servlet.ServletWrapper.loadServlet(ServletWrapper.java(Compiled Code))
     at com.ibm.ws.webcontainer.servlet.ServletWrapper.initialize(ServletWrapper.java:1312)
     at com.ibm.wsspi.webcontainer.extension.WebExtensionProcessor.createServletWrapper(WebExtensionProcessor.java:84)
     at com.ibm.ws.webcontainer.extension.InvokerExtensionProcessor.handleRequest(InvokerExtensionProcessor.java:238)
     at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:2841)
     at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:220)
     at com.ibm.ws.webcontainer.VirtualHost.handleRequest(VirtualHost.java:204)
     at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java(Compiled Code))
     at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java(Compiled Code))
     at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java(Compiled Code))
     at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java(Compiled Code))
     at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java(Compiled Code))
     at com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java(Compiled Code))
     at com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java(Compiled Code))
     at com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java(Compiled Code))
     at com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java(Compiled Code))
     at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled Code))
this error is occuring on concurrent clicks suppose three users click on same button then it is possible that one user gets correct output while other users experience this exception
Also one more point we are not using web.xml for calling servlets we are directly calling it giving its full path.
Thanks

Berkeley DB: Too Many Open Files

I've installed BerkelyDB 6.0.3 on Mavericks but when I try to use it I get errors about there being too many open files:
Apr-10-14 11:20:45 [Worker_10001] BerkeleyDB-CRT-ENV-ERROR (1) in HASH Delay , on file /Applications/assp/delaydb.bdb : BDB:Too many open files
Apr-10-14 11:20:45 [Worker_10001] BerkeleyDB-ENV-ERROR Delay: - BDB:Too many open files
Apr-10-14 11:20:45 [Worker_10001] Delaydb database error: BerkeleyDB-ENV-ERROR Delay: - BDB:Too many open files
Does anyone know how to (permanently) increase these limits in Mavericks?
Thanks,
James.

sudo sysctl -w kern.maxfiles=32768
sudo sysctl -w kern.maxprocperuid=16384
seemed to work.
Created file /etc/sysctl.conf containing:
kern.maxfiles=32768
kern.maxprocperuid=16384
so that the setting would survive a reboot.
Hopefully that permanently fixes it.

Open Directory logging? (Too many open files)

I recently had a request from another admin to up the logging output of my OD servers. So I opened WGM, opened the OLCGlobalConfig section and changed the olcLogLevel to Filter. At which point my OD server stopped allowing edits and updates to the server. And started reporting "too many open files" in the system.log. So I guess I have 2 questions.
1) What is the proper method for increasing the log level of an OD server? The other admin is looking for query data and queries per second data. So if there is some other way of getting that data I'm all for it.
2) Is there any solution for the "too many open files" if I change the olcLogLevel in the future?
Thanks,
Derek

Explore the odutil command. There are 7 levels of logging available. error is the default. You may want to try notice to see if that gives you the information you want.
sudo odutil set log notice

What have "Too many open Files" to do with FIFOs?

Hi folks.
I've just finished a middleware service for my company, that receives files via a TCP/IP connection and stores them into some cache-directory. An external program gets called, consumes the files from the cache directory and puts a result-file there, which itself gets sent back to the client over TCP/IP.
After that's done, the cache file (and everything leftover) gets deleted.
The middleware-server is multithreaded and creates a new thread for each request connection.
These threads are supposed to die when the request is done.
All works fine, cache files get deleted, threads die when they should, the files get consumed by the external program as expected and so on.
BUT (there's always a butt;) to migrate from an older solution, the old data gets fed into the new system, creating about 5 to 8 requests a second.
After a time of about 20-30 minutes, the service drops out with "IOException: Too many open files" on the very line where the external program gets called.
I sweeped through my code, seeking to close even the most unlikely stream, that gets opened (even the outputstreams of the external process ;) but the problem stays.
Things I thought about:
- It's the external program: unlikely since the lsof-command (shows the "list of open files" on Linux) says that the open files belong to java processes. Having a closer look at the list, I see a large amount of "FIFO" entries that gets bigger, plus an (almost) constant amount of "normal" open file handles.
So perhaps the handles get opened (and not closed) somehwere else and the external program is just the drop that makes the cask flood over.
- Must be a file handle that's not closed: I find only the "FIFO" entries to grow. Yet I don't really know what that means. I just think it's something different than a "normal" file handle, but maybe I'm wrong.
- Must be a socket connection that's not closed: at least the client that sends requests to the middleware service closes the connection properly, and I am, well, quite sure that my code does it as well, but who knows? How can I be sure?
That was a long description, most of which will be skipped by you. To boil it down to some questions:
1.) What do the "FIFO" entries of the lsof-command under Linux really mean ?
2.) How can I make damn sure that every socket, stream, filehandle etc. pp. is closed when the worker thread dies?
Answers will be thanked a lot.
Tom

Thanks for the quick replies.
@BIJ001:
ls -l /proc/<PID>/fdGives the same information as lsof does, namely a slowly but steadily growing amount of pipes
fuserDoesn't output anything at all
Do you make exec calls? Are you really sure stdout and stderr are consumed/closed?Well, the external program is called by
Process p = Runtime.getRuntime().exec(commandLine);and the stdout and stderr are consumed by two classes that subclass Thread (named showOutput) that do nothing but prepending the corresponding outputs with "OUT:" and "ERR" and putting them into a log.
Are they closed? I hope so: I call the showOutput's halt method, that should eventually close the handles.
@sjasja:
Sounds like a pipe.Thought so, too ;)
Do you have the waitFor() in there?Mentioning the waitFor():
my code looks more like:
try {
     p = Runtime.getRuntime.exec(...);
     outshow = new showOutput(p.getInputStream(), "OUT").start;
     errshow = new showOutput(p.getErrorStream(), "ERR").start;
     p.waitFor();
} catch (InterruptedException e) {
     //can't wait for process?
     //better go to sleep some.
     log.info("Can't wait for process! Going to sleep 10sec.");
     try{ Thread.sleep(10000); } catch (InterruptedException ignoreMe) {}
} finally {
     if (outShow!=null) outShow.halt();
     if (errShow!=null) errShow.halt();
/**within the class showOutput:*/
/**This method gets called by showOutput's halt:*/
public void notifyOfHalt() {
     log.debug("Registered a notification to halt");
     try {
          myReader.close(); //is initialized to read from the given InputStream
     } catch (IOException ignoreMe) {}
}Seems as if the both of you are quite sure that the pipes are actually created by the exec command and not closed afterwards.
Would you deem it unlikely that most of the handles are opened somewhere else and the exec command is just the final one that crashes the prog?
That's what I thought.
Thanks for your time
Tom

Suddenly getting "Too Many Open File" error

Dear All,
I have listener program which have been working well for the past few months. Suddenly I start to get "Too Many Open File" error. What can be solution? Is it I need to increase the file descriptors or any other solution? Thank you.

Dear Ejp,
Attached below is my codes. I have remove some of the fields for db just to make the code a bit more easier to read. Where do you think is leaking?
public class commServer {
public static void main(String[] args) {
try {
               final ServerSocket serverSocketConn = new ServerSocket(8888);
                    while (true)
                              try
                         Socket socketConn1 = serverSocketConn.accept();
new Thread(new ConnectionHandler(socketConn1)).start();
                              catch(Exception e)
                                             System.out.println(e.toString());
catch (Exception e)
System.out.println(e.toString());
//System.exit(0);
class ConnectionHandler implements Runnable {
private Socket receivedSocketConn1;
DateFormat dateFormat = new SimpleDateFormat("dd/MM/yyyy HH:mm:ss");
DateFormat formatter = new SimpleDateFormat("EEE, dd MMM yyyy");
DateFormat inDf=new SimpleDateFormat("ddMMyyHHmmss");
DateFormat outDf=new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
ConnectionHandler(Socket receivedSocketConn1) {
this.receivedSocketConn1=receivedSocketConn1;
//@Override
public void run() {
Connection dbconn = null;
BufferedWriter w = null;
BufferedReader r = null;
try {
PrintStream out = System.out;
     BufferedWriter fout = null;
w = new BufferedWriter(new OutputStreamWriter(receivedSocketConn1.getOutputStream()));
r = new BufferedReader(new InputStreamReader(receivedSocketConn1.getInputStream()));
int m = 0, count=0;
String line="";
String n="";
w.write("$PA\n");
w.flush();
while ((m=r.read()) != -1)
Date dateIn = new Date();
     n = n + (char) m;
     int i = n.indexOf("GET");
                              if(i != -1) {
                                   break;
     if (m==35)
     String ori = n;
     String noCheckSum = n.substring(0,(n.length()-4));
               int addExist = n.indexOf("@");
String[] slave = null;
if(addExist!=-1)
     slave = noCheckSum.split("@");
          n = slave[0];
          //System.out.println(" slave : "+slave.length);
               else
                    n = noCheckSum;
     String[] result = n.split(",");
     Statement stmt = null;
     w.write("$PA\n");
w.flush();
               int count1 = 0;
Date date = Calendar.getInstance().getTime();
String today = formatter.format(date);
                    String filename= "MyDataFile"+today+".txt";
boolean append = true;
FileWriter fw = null;
try
fw = new FileWriter(filename,append);
     fw.write(ori+" "+dateFormat.format(dateIn)+"\n");//appends the string to the file
     Date dateOut = new Date();
     fw.write("$PA"+" "+dateFormat.format(dateOut)+"\n");//appends the string to the file
catch (IOException ex)
                    //ex.printStackTrace(new PrintWriter(sWriter));
                    System.out.println("MyError:IOException has been caught in in the file operation");
                    ex.printStackTrace();
                    finally
               try
     if ( fw != null )
          fw.close();
     else
          System.out.println("MyError:fw is null in finally close");
                    catch(IOException ex){
                    System.out.println("MyError:IOException has been caught in fw is null in finally close");
                    ex.printStackTrace();
     try
                         String deviceID=result[3].trim();
                         String dateTime=result[4].trim();
                         String[] result2 = result[10].split("'");
                         String gpsDate = result2[1].trim().substring(0,6);
                         String gpsTime = result2[1].trim().substring(6,12);
                         String gpsLat = result2[1].trim().substring(13,20);
                         String latitude = result2[1].trim().substring(13,20).substring(0,2)+"."+result2[1].trim().substring(13,20).substring(2,7);
                         String gpsLong = result2[1].trim().substring(21,29);
                         String longitude = result2[1].trim().substring(21,29).substring(0,3)+"."+result2[1].trim().substring(21,29).substring(3,8);
                         String speed = result2[1].trim().substring(30,33);
                         String course = result2[1].trim().substring(33,36);
               String dateTimer = null;
                         try
                         Date inDate=null;
                         inDf.setTimeZone(TimeZone.getTimeZone("UTC"));
                         inDate=inDf.parse(dateTime);
                         outDf.setTimeZone(TimeZone.getTimeZone("Asia/Kuala_Lumpur"));
                                   dateTimer=outDf.format(inDate);
                         catch(ParseException ex)
                         System.out.println("MyError:Parse Error has been caught for date parse close");
          ex.printStackTrace();
                    dbconn = DriverManager.getConnection("jdbc:mysql://192.168.1.155:3306/db1?"+"user=db1&password=test1");
               stmt = dbconn.createStatement();
               String selectQuery2 = "Select * from tripData Where deviceID='"+ deviceID +"' and dateTimer>'"+dateTimer+"' Order By dateTimer Desc Limit 1";
               ResultSet rs2 = stmt.executeQuery(selectQuery2);
               String updateQuery = "";
               if(rs2.next())
                    String previousLatitude="";
                    String previousLongitude="";
                    String previousSpeed="";
                    previousLatitude = rs2.getString("latitude");
                    previousLongitude = rs2.getString("longitude");
                    previousSpeed = rs2.getString("speed");
                    updateQuery = "UPDATE device SET " +
                                   "latitude='" + previousLatitude +
                                   "',longitude='" + previousLongitude +
                                   "',speed='" + previousSpeed +
                                   "' WHERE serialNumber='" + deviceID + "'";
               else
               updateQuery = "UPDATE device SET " +
                                   "latitude='" + latitude +
                                   "',longitude='" + longitude +
                                   "',speed='" + speed +
                                   "',course='" + course +
                                   "',dateTimer='" + dateTimer +
                                   "' WHERE serialNumber='" + deviceID + "'";
                    count = stmt.executeUpdate(updateQuery);
               String insertQuery = "INSERT INTO tripData" +
                                   "(latitude,longitude,speed,course,dateTimer,deviceID)" + " VALUES ('" +
                                   latitude + "','" + longitude + "','" + speed + "','" + course + "','" + dateTimer + "','" + deviceID + "' )";
     count = stmt.executeUpdate(insertQuery);
          if(addExist!=-1)
          for(int iSlave=1; iSlave<slave.length ; iSlave++)
                                        String[] slaveDetails = slave[iSlave].split(",",-1);
                                        String slaveEventType = slaveDetails[0];
                                        String slaveGroup = slaveDetails[1];
                                        String slaveUnitID = slaveDetails[2];
                                        String slaveBattLevel = slaveDetails[3];
                                        String slaveSwitchStat = slaveDetails[4];
                                        String slaveTempHumid = slaveDetails[5];
                                   String slaveTemp="",slaveHumid="";
                                   if(slaveTempHumid.length()>0)
                                   slaveHumid = slaveTempHumid.substring(5,7);
                                             if(slaveTempHumid.charAt(0)=='P')
                                             slaveTemp = "+"+slaveTempHumid.substring(1,3)+"."+slaveTempHumid.substring(3,5);
                                             else if(slaveTempHumid.charAt(0)=='M')
                                             slaveTemp = "-"+slaveTempHumid.substring(1,3)+"."+slaveTempHumid.substring(3,5);
                                   slaveBattLevel = slaveBattLevel.trim().substring(0,2)+"."+slaveBattLevel.trim().substring(2,3);
                                        String insertQuery2 = "INSERT INTO tripDataSlave" +
                                        "(dateTimer,deviceID,slaveUnitID,slaveEventType,slaveGroup,slaveBattLevel,slaveSwitchStat,slaveTemp,slaveHumidity)" + " VALUES ('" +
                                        dateTimer + "','" + deviceID + "','" + slaveUnitID + "','" + slaveEventType + "','" + slaveGroup + "','" + slaveBattLevel + "','" + slaveSwitchStat + "','" + slaveTemp + "','" + slaveHumid + "')";
          count = stmt.executeUpdate(insertQuery2);
               catch (SQLException ex)
System.out.println("MyError:Error : "+ex);
                    finally
                    try
                         if ( stmt != null )
                         stmt.close();
                         else
                              System.out.println("MyError:stmt is null in finally close");
                    catch(SQLException ex){
                    System.out.println("MyError:SQLException has been caught for stmt close");
ex.printStackTrace();
                    try
                         if ( dbconn != null )
                         dbconn.close();
                         else
                         System.out.println("MyError:dbconn is null in finally close");
                    catch(SQLException ex){
                    System.out.println("MyError:SQLException has been caught for dbconn close");
ex.printStackTrace();
     n="";
catch (IOException ex)
System.out.println("MyError:IOException has been caught in in the main first try");
ex.printStackTrace();
finally
try
     if ( w != null )
          w.close();
     else
          System.out.println("MyError:w is null in finally close");
catch(IOException ex){
System.out.println("MyError:IOException has been caught in w in finally close");
ex.printStackTrace();
}

DirectoryService[67]: socket(PF_ROUTE) failed: Too many open files

Hello!
On MacOSX 10.4.10 which is an OD master, my log are filed with this:
DirectoryService[67]: socket(PF_ROUTE) failed: Too many open files
It happens at exactly every hours 44 minutes 18 seconds 161 times. At the same time, it makes a lot dns request for "kerberos-master.udp.XXXXXXXXX.COM IN SRV +"
The server works fine, but there's probably a cronjob that go crazy and i would like to know why it's happening.
Thanks a lot!
Fred

Hi,
Cause might The server got an exception while trying to accept client connections. It will try to backoff to aid recovery.
The OS limit for the number of open file descriptor (FD limit) needs to be increased. Tune OS parameters that might help the server to accept more client connections (e.g. TCP accept back log).
http://e-docs.bea.com/wls/docs90/messages/Server.html#BEA-002616
Regards,
Prasanna Yalam

Too many open files in system cause database goes down

Similar Messages

Maybe you are looking for