Hung jvm

We are running into strange situation where the jvm seems hung and doesn't respond to 'kill -3' either, and the java process consumes very high amount of CPU (100%).
This happens very rarely; eventually the java process times-out and when it is re-run on a different machine it runs successfully ...
$ java -version
java version "1.6.0_06"
Java(TM) SE Runtime Environment (build 1.6.0_06-b02)
Java HotSpot(TM) Server VM (build 10.0-b22, mixed mode)
$ uname -ar
Linux 2.6.9-55.ELsmp #1 SMP Fri Apr 20 16:36:54 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
Looks like lots of threads in JVM are hung in a call to Mutex::lock_without_safepoint_check; when I ran jstack over the hung java process with
$ jstack -F -m <pid>
I saw:
----------------- 22175 -----------------
0xffffe410 ????????
0x00000002 ????????
0x064e5344 ZN5Mutex28lockwithout_safepoint_checkEv + 0x44
0x06557f57 _ZN20SafepointSynchronize5blockEP10JavaThread + 0xd7
0x064fc204 _ZN2os5sleepEP6Threadxb + 0x244
0x063d5d20 JVM_Sleep + 0x320
0xf4d863aa * java.lang.Thread.sleep(long) bci:0 (Interpreted frame)
0xf4d7ef0d * org.apache.hadoop.mapred.Task$1.run() bci:31 line:301 (Interpreted frame)
0xf4d7f3e9 * java.lang.Thread.run() bci:11 line:619 (Interpreted frame)
----------------- 22178 -----------------
0xffffe410 ????????
0x00000002 ????????
0x064e5344 ZN5Mutex28lockwithout_safepoint_checkEv + 0x44
0x06557f57 _ZN20SafepointSynchronize5blockEP10JavaThread + 0xd7
0x065b79f4 ZN10JavaThread44checksafepoint_and_suspend_for_native_transEPS_ + 0x2b4
0x06398e9c jni_SetByteArrayRegion + 0x7c
0xd034b55b Java_java_net_SocketInputStream_socketRead0 + 0xfb
0xf4d863aa * java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) bci:0 (Interpreted frame)
0xf4d7ede9 * java.net.SocketInputStream.read(byte[], int, int) bci:84 line:129 (Interpreted frame)
0xf4d7ede9 * java.io.BufferedInputStream.fill() bci:175 line:218 (Interpreted frame)
0xf4d7ef0d * java.io.BufferedInputStream.read1(byte[], int, int) bci:44 line:258 (Interpreted frame)
0xf4d7ede9 * java.io.BufferedInputStream.read(byte[], int, int) bci:49 line:317 (Interpreted frame)
0xf4d7ede9 * sun.net.www.http.HttpClient.parseHTTPHeader(sun.net.www.MessageHeader, sun.net.ProgressSource, sun.net.www.protocol.http.HttpURLConnection) bci:51 line:687 (Interpreted frame)
0xf4d7ece1 * sun.net.www.http.HttpClient.parseHTTP(sun.net.www.MessageHeader, sun.net.ProgressSource, sun.net.www.protocol.http.HttpURLConnection) bci:30 line:632 (Interpreted frame)
0xf4d7ece1 * sun.net.www.protocol.http.HttpURLConnection.getInputStream() bci:290 line:1000 (Interpreted frame)
0xf4d7eda7 * org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(java.net.URLConnection, int, int) bci:71 line:1230 (Interpreted frame)
0xf4d7eda7 * org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputLocation, org.apache.hadoop.fs.Path) bci:54 line:1068 (Interpreted frame)
----------------- 22182 -----------------
0xffffe410 ????????0x00000002 ????????
0x064e5344 ZN5Mutex28lockwithout_safepoint_checkEv + 0x440x06557f57 _ZN20SafepointSynchronize5blockEP10JavaThread + 0xd7
0x06594250 _ZN13ObjectMonitor5enterEP6Thread + 0x2a0
0x0659253a ZN18ObjectSynchronizer10slowenterE6HandleP9BasicLockP6Thread + 0x6a
0x06365bde _ZN18InterpreterRuntime12monitorenterEP10JavaThreadP15BasicObjectLock + 0x6e
0xf4d8f263 * org.apache.log4j.Category.callAppenders(org.apache.log4j.spi.LoggingEvent) bci:13 line:202 (Interpreted frame)
0xf4d7ef0d * org.apache.log4j.Category.forcedLog(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) bci:14 line:388 (Interpreted frame)
0xf4d7ef0d * org.apache.log4j.Category.log(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) bci:34 line:853 (Interpreted frame)
0xf4d7ef0d * org.apache.commons.logging.impl.Log4JLogger.info(java.lang.Object) bci:21 line:133 (Interpreted frame)
0xf4d7f3e9 * org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputLocation, org.apache.hadoop.fs.Path) bci:311 line:1114 (Interpreted frame)
0xf4d7eda7 * org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputLocation) bci:181 line:976 (Interpreted frame)
----------------- 22185 -----------------
0xffffe410 ????????
0x00000002 ????????
0x064e5344 ZN5Mutex28lockwithout_safepoint_checkEv + 0x44
0x06557f57 _ZN20SafepointSynchronize5blockEP10JavaThread + 0xd7
0x063639cc _ZN18InterpreterRuntime8newarrayEP10JavaThread9BasicTypei + 0xfc
0xf4d8eb97 * java.lang.AbstractStringBuilder.<init>(int) bci:6 line:45 (Interpreted frame)
0xf4d7ef0d * java.lang.StringBuffer.<init>() bci:3 line:79 (Interpreted frame)
0xf4d7ef0d * java.text.NumberFormat.format(long) bci:6 line:280 (Interpreted frame)
0xf4d7eda7 * org.apache.hadoop.mapred.JobID.toStringWOPrefix() bci:30 line:107 (Interpreted frame)
0xf4d7eda7 * org.apache.hadoop.mapred.TaskID.toStringWOPrefix() bci:13 line:135 (Interpreted frame)
0xf4d7eda7 * org.apache.hadoop.mapred.TaskAttemptID.toStringWOPrefix() bci:13 line:124 (Interpreted frame)
0xf4d7eda7 * org.apache.hadoop.mapred.TaskAttemptID.toString() bci:20 line:118 (Interpreted frame)
0xf4d7eda7 * java.lang.String.valueOf(java.lang.Object) bci:10 line:2827 (Interpreted frame)
0xf4d7eda7 * java.lang.StringBuilder.append(java.lang.Object) bci:2 line:115 (Interpreted frame)
0xf4d7eda7 * org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputLocation, org.apache.hadoop.fs.Path) bci:372 line:1119 (Interpreted frame)
0xf4d7eda7 * org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputLocation) bci:181 line:976 (Interpreted frame)
0xf4d7ee2b * org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run() bci:93 line:925 (Interpreted frame)
----------------- 22176 -----------------
0xffffe410 ????????
0x00000ffc ????????
0x0639c149 jni_GetDirectBufferAddress + 0x79
0xcf9fdf90 Java_org_apache_hadoop_io_compress_lzo_LzoDecompressor_decompressBytesDirect + 0xd0
0xf4d863aa * org.apache.hadoop.io.compress.lzo.LzoDecompressor.decompressBytesDirect(int) bci:0 (Interpreted frame)
0xf4d7ede9 * org.apache.hadoop.io.compress.lzo.LzoDecompressor.decompress(byte[], int, int) bci:116 line:273 (Interpreted frame)
0xf4d7f2c5 * org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(byte[], int, int) bci:49 line:75 (Interpreted frame)
0xf4d7ede9 * org.apache.hadoop.io.compress.DecompressorStream.read(byte[], int, int) bci:39 line:74 (Interpreted frame)
0xf4d7ede9 * org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputLocation, org.apache.hadoop.fs.Path) bci:549 line:1141 (Interpreted frame)
0xf4d7eda7 * org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputLocation) bci:181 line:976 (Interpreted frame)
0xf4d7ee2b * org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run() bci:93 line:925 (Interpreted frame)
----------------- 22191 -----------------
0xffffe410 ????????
0x00000002 ????????
0x064e5344 ZN5Mutex28lockwithout_safepoint_checkEv + 0x44
0x06557f57 _ZN20SafepointSynchronize5blockEP10JavaThread + 0xd7
0x065b79f4 ZN10JavaThread44checksafepoint_and_suspend_for_native_transEPS_ + 0x2b4
0x06398e9c jni_SetByteArrayRegion + 0x7c
0xd034b55b Java_java_net_SocketInputStream_socketRead0 + 0xfb
0xf4d863aa * java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) bci:0 (Interpreted frame)
0xf4d7ede9 * java.net.SocketInputStream.read(byte[], int, int) bci:84 line:129 (Interpreted frame)
0xf4d7ede9 * java.io.BufferedInputStream.fill() bci:175 line:218 (Interpreted frame)
0xf4d7ef0d * java.io.BufferedInputStream.read1(byte[], int, int) bci:44 line:258 (Interpreted frame)
0xf4d7ede9 * java.io.BufferedInputStream.read(byte[], int, int) bci:49 line:317 (Interpreted frame)
0xf4d7ede9 * sun.net.www.http.HttpClient.parseHTTPHeader(sun.net.www.MessageHeader, sun.net.ProgressSource, sun.net.www.protocol.h
ttp.HttpURLConnection) bci:51 line:687 (Interpreted frame)
----------------- 22192 -----------------
0xffffe410 ????????
0x00000002 ????????
0x064e5344 ZN5Mutex28lockwithout_safepoint_checkEv + 0x44
0x06557f57 _ZN20SafepointSynchronize5blockEP10JavaThread + 0xd7
0x06367f2b ZN18InterpreterRuntime12atsafepointEP10JavaThread + 0x15b
0xf4d84604 * sun.net.www.MessageHeader.remove(java.lang.String) bci:115 line:306 (Interpreted frame)
0xf4d7ef0d * sun.net.www.protocol.http.HttpURLConnection.getInputStream() bci:1006 line:1159 (Interpreted frame)
0xf4d7eda7 * org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStream(java.net.URLConnection, int, int) bci
:71 line:1230 (Interpreted frame)
0xf4d7eda7 * org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(org.apache.hadoop.mapred.ReduceTask$Red
uceCopier$MapOutputLocation, org.apache.hadoop.fs.Path) bci:54 line:1068 (Interpreted frame)
0xf4d7eda7 * org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(org.apache.hadoop.mapred.ReduceTask$Reduc
eCopier$MapOutputLocation) bci:181 line:976 (Interpreted frame)
0xf4d7ee2b * org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run() bci:93 line:925 (Interpreted frame)
----------------- 22190 -----------------
0xffffe410 ????????
0x00000002 ????????
0x064e5344 ZN5Mutex28lockwithout_safepoint_checkEv + 0x44
0x06557f57 _ZN20SafepointSynchronize5blockEP10JavaThread + 0xd7
0x0639bfd5 lookupDirectBufferClasses + 0x245
0x0639d5d2 Z29initializeDirectBufferSupportP7JNIEnvP10JavaThread + 0x192
0x0639c149 jni_GetDirectBufferAddress + 0x79
0xcf9fdf90 Java_org_apache_hadoop_io_compress_lzo_LzoDecompressor_decompressBytesDirect + 0xd0
0xf4d863aa * org.apache.hadoop.io.compress.lzo.LzoDecompressor.decompressBytesDirect(int) bci:0 (Interpreted frame)
0xf4d7ede9 * org.apache.hadoop.io.compress.lzo.LzoDecompressor.decompress(byte[], int, int) bci:116 line:273 (Interpreted frame)
0xf4d7f2c5 * org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(byte[], int, int) bci:49 line:75 (Interpreted frame)
0xf4d7ede9 * org.apache.hadoop.io.compress.DecompressorStream.read(byte[], int, int) bci:39 line:74 (Interpreted frame)
0xf4d7ede9 * org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputLocation, org.apache.hadoop.fs.Path) bci:549 line:1141 (Interpreted frame)
0xf4d7eda7 * org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputLocation) bci:181 line:976 (Interpreted frame)
0xf4d7ee2b * org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run() bci:93 line:925 (Interpreted frame)
Thoughts?
Some links to similar bugs wrt SafepointSynchronize::block -> Mutex::lock_without_safepoint_check:
http://bugs.sun.com/bugdatabase/view_bug.do;:WuuT?bug_id=4472904
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4639387
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4681171
and:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6601725
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6317397
PS: We do have some JNI code...

I just saw many postings on this type of problem in different apps, e.g., MySQL, Oracle, and JBoss. I also wonder what the root cause is.
I have an application that have multiple threads fetching data from a number of urls. And the problem happened randomly that once in a while it would just hang and not proceed. I did notice the application starts a large number of worker threads. And when this problem was noticed, there was only one worker thread left in this state. Your stack trace shows the native Mutex function name but mine doesn't (and I don't know why!)
Not sure what to do but like to follow this thread to see what comes out of this.
Thanks,
Student

Similar Messages

  • JVM 1.4.2_06-b03 version has got a problem??

    Hi all,
    I am currently using the JVM version 1.4.2_06-b03 in Fedora Core 2.It gave me some problem in server socket program.The Problem is as follows ,
    I wrote a single threaded server program,which connects to database and does some insertion and selection from the database.
    While testing,i wrote a CLIENT program which which will PUMP the server socket with DATA,every two seconds.
    I ran the same program in JVM version 1.3.1_15-b01 and i was checking the logs.
    After three hours or four, the 1.4 version program got hung. The Socket was still listening, as i was able to telnet to the port.
    The socket was not taking any input and i was not able to KILL the JAVA PROCESS also.
    But the same PROGRAM was running fine in JDK 1.3.1_15 version.
    Is there any problem in that version of JVM?? If yes, which version was the bug solved and which is the stable version in JVM 1.4 ??
    There was no issues with the Database server as it was working fine for other programs.
    Please if any one could help me let me know as soon as possible
    Regards
    Vickraman V.P

    Thanks BIJO .i will try out with the version you have specified.and can u explain how to check in bug parade???
    Regards,
    Vickraman V.P

  • ADF Application + OAS 10.1.3, how many OC4J and JVM ?

    Hi all,
    We are running in Production with ADF (BC+Faces) 10.1.3 in OAS 10.1.3, with maximum 300++ users.
    (but not nessarily all submit request on same time)
    Intermittently (about two times) a week, there is time where user get hung and all request take forever to get responded.
    We watch during that time that DB load is not high, and OAS CPU 99.9 %. We have 12 GB RAM installed on OAS.
    Then user have to restat the OPMN to make system responsive again.
    Currently we run on one OC4J instance with 1 JVM, with 3GB heap.
    My question is :
    1) Will it help if we add 1 more JVM to the OC4J ?
    2) WIll it affect the Application behaviour ? (if running in 2 JVM compared to in 1 JVM like now) ?
    Thank you very much,
    xtanto

    Xtanto,
    yes you will need to retest it.
    Why?
    Because it is highly possible session will be 'transported' between JVM's
    There is a chance your application will not deploy/start at all on distributed JVM's, this because ALL objects stored on the session need to be serializable.
    When this is the case, the application will deploy/start on multiple JVM's
    But this doesn't resolve issues around developers who put random stuff in the session which is a bad practice in general.
    Session behavior will need to be tested.
    -Anton

  • Running multiple weblogic servers on the same JVM

    Is there a way to make Weblogic 7.0 and Weblogic 5.1 run on the same JVM? If so
    what are the pros and cons?

    I hope you also checked the IPCKEY (and all other identifiers of that kind)
    in your ubbconfig file: they should all be different.
    We have similar configurations running without any major difficulty.
    One catch: very heavy IPC activity (due to errors in the application,
    never because of heavy user load) in one environment sometimes overloads
    the OS, resulting in hung applications in the other environments.
    Regards
    Frank
    "Manoj SASIDHARAN" <[email protected]> wrote:
    >
    Hi Troy,
    Here are a few tips:
    1. Check the APPDIR values in both environments. They should point to
    the appropriate
    directories containing the Tuxedo server binaries for each environment.
    Analyse
    the values of APPDIR environment variable and also inside the UBBCONFIG
    file.
    2. Check the WSL address. Both environments should have different ports
    for listening
    to requests.
    3. Check the WSNADDR environment variable at the client side. They should
    point
    to the appropriate WSL addresses.
    HTH
    regards
    MS

  • Hung queries, and Oracle thin driver query timeout implementation

    We are seeing occasional instances of stuck threads, where the code is in Oracle thin XA driver code waiting for a socket read for a response from the database. Usually only one thread at a time. The connection never terminates and we have to shut down WebLogic to clear it. The DBAs tell us that all connections to the 11g RAC cluster members from our hosts are idle, with no hung or long-running transactions.
    This happens every few days across two WebLogic 10.3.5 clustered instances in our QA environment, usually not the same time on two servers (one had four incidents, the other had only one, in the last 15 or so days). Sometimes it has resulted in a hung server, as the driver is holding a lock that blocks other threads (today we had numerous threads block in a TX rollback).
    I'm guessing our WebLogic instance somehow is not getting the connection close from the remote host. BTW, there is no firewall between us and the DB. I have not got any strong suspect for the cause (although we are running Linux on a VMWare 4.0 VM, which always worries me).
    Ordinarily I'd either ask the application to set a query timeout, or set the query timeout parameter for the connection pools as a workaround (we are using a MultiDatasource), so the transaction would at least abort and the application can handle it. However, while the Oracle driver does support java.sql.Statement.setQueryTimeout(), some brief investigation on my part leads me to believe the timeout is implemented on the server side and not the client side - so that if it is indeed the case that WebLogic does not see the close on the connection, it also would never see the timeout.
    Two questions:
    1. Is my suspicion about server-side implementation of timeout for Oracle 11g correct?
    2. If so, is there some property I can set for the driver that will implement a socket timeout? Is this "safe"?
    Thanks for any help!
    [ Oracle WebLogic 10.3.5, HotSpot 64 bit JVM 1.6.0_29 on RHEL 5.6 on VMWare; Oracle Thin XA driver bundled with WebLogic; Oracle RAC 11g (three-node cluster)]
    Edited by: SteveElkind on Dec 3, 2012 5:25 AM

    Thanks, Joe.
    My investigations were heading in this direction (e.g., http://docs.oracle.com/cd/B28359_01/java.111/b31224/apxtblsh.htm#CHDBBDDA). However, for WebLogic, is it as simple as adding the following property in the Datasource Connection properties edit box in the WebLogic console?
    oracle.net.READ_TIMEOUT=300000
    or do I have to edit the connection URL?
    I've tried this, and the Datasource restarts OK after the change, but I have no way right now to check whether it actually "works".
    (we have some long-running queries; five minutes seems to be a "safe" limita for now)

  • Verify and re-install jvm if necessary

    I'm attempting to install jvm in my 10.2.0.2 database on an aix 5.2 server. I ran the dbca and told it to install jvm and intermedia, but it just hung at 11%. So I cancelled it after about 45 minutes.
    I found reference of the following selects to verify jvm has been installed:
    sys@DMRDEV> select version, status, comp_id from dba_registry;
    VERSION STATUS COMP_ID
    10.2.0.2.0 VALID XDB
    10.2.0.2.0 INVALID ORDIM
    10.2.0.2.0 VALID EM
    10.2.0.2.0 VALID CATALOG
    10.2.0.2.0 VALID CATPROC
    10.2.0.2.0 VALID JAVAVM
    6 rows selected.
    sys@DMRDEV> select dbms_java.longname('foo') from dual;
    DBMS_JAVA.LONGNAME('FOO')
    foo
    sys@DMRDEV> select count(*) from dba_objects
    2 WHERE object_type LIKE 'JAVA%';
    COUNT(*)
    15568
    sys@DMRDEV> select owner, count(*)
    2 from dba_objects
    3 where object_type like 'JAVA%'
    4 group by owner;
    OWNER COUNT(*)
    SYS 14114
    ORDSYS 1454
    sys@DMRDEV> SELECT dbms_java.longname(name)
    2 FROM obj$
    3 WHERE type# = 29 and status != 1;
    DBMS_JAVA.LONGNAME(NAME)
    oracle/ord/media/annotator/utils/MADataInputStream
    oracle/ord/media/annotator/utils/Preferences
    312 rows returned.
    The fact that I cancelled the install and the 312 records returned by the last select make me think it didn't install correctly. I have searched for about 30 minutes now trying to figure out how to un-install or repair the jvm install, but can't find anything. Anyone have a link to some documentation on this? I think my initial problem might have been my JAVA_HOME, LD_LIBRARY_PATH, and PATH variables not pointing to an up to date java directory.

    disregard. i just restored and recovered from the backup that I took before messing it up.
    for further reference if anyone else has the same problem I had to manually reload jvm and then install Oracle Text manually. The problem came up because this was a 9.2 database that didn't have either component installed when it was upgraded to 10g. I used metalink note 276554.1 for the jvm portion and 280713.1 for the Oracle Text part. For the jvm portion I only ran the part that installs jvm.

  • Extremely slow performance with ojdbc6.jar on IBM JVMs (Java 6)

    We are consistently seeing slow performance (easily demonstrable by the simplest of test cases) while using ojdbc6.jar on IBM JDKs. Pefrormance is normal when we simply opt for ojdbc14.jar under the same JDK, same java program.
    Works well
    =======
    JRockit, ojdbc6.jar
    JRockit, ojdbc14.jar
    IBM JDK, ojdbc14.jar
    A lot slower
    =======
    IBM JDK, ojdbc6.jar
    All we had to do was to write a simple JDBC access program and measure time taken around
    Connection conn = DriverManager.getConnection(jdbcURL, dbProps);
    The first 3 combinations mentioned above return comparable numbers. The 4th combination takes 3-4 times as much.
    We initially saw this during performance test of our software on WebSphere (also supported on Weblogic). Later was able to eliminate WebSphere and Weblogic as the problem areas and narrow it down to the JVM. Reproduce-able on development machines.
    We have been using Oracle/WLS since BEA used to market 'Tengah' servers. This is the worst we have seen as far as the qaulity of support staff goes. We first level support seems clueless. We filed this case and the support comes back tells us that IBM JDKs are not supported for WLS! Oh, we have been running WLS/AIX/IBM JVM combination only for 5+ years.

    A quick solution to test this would be to do the following:
    mv /dev/random /dev/random.bk
    ln /dev/urandom /dev/random
    Note: The quick solution will disappear after a reboot. You could add it to rc.local to automatically make this change, but I wouldn't recommend it.
    The following is from [http://www.usn-it.de/index.php/2009/02/20/oracle-11g-jdbc-driver-hangs-blocked-by-devrandom-entropy-pool-empty/]:
    "Oracle 11g JDBC driver hangs blocked by /dev/random – entropy pool empty
    On a headless (=without console) network server, the 11g JDBC driver used for (java) application connect may cause trouble. In my case, it refused to connect to the DB without any error, trace or log entry. It simply hung. After several hours, it connected one time, and freezed again. Remote debugging done by the development clarified that it locks after calling SeedGenerator() and SecureRandom().
    Reason: The JDBC 11g needs about 40 bytes of secure random numbers, gathered from /dev/random, to encrypt its connect string.
    But public-available “man 4 random” says:
    When read, the /dev/random device will only return random bytes within the estimated number of bits of noise in the entropy pool. /dev/random should be suitable for uses that need very high quality randomness such as one-time pad or key generation. When the entropy pool is empty, reads from /dev/random will block until additional environmental noise is gathered.
    So far so good, now the question arises: Why does this mystic “entropy pool” runs out of gas?
    The answer is as simple as unsatisfying: because too less entropy “noise” was generated by the system. You can check the “filling level” (maybe zero?) of your pool and the overall size of the pool (usually 4096) by issuing
    cat /proc/sys/kernel/random/entropy_avail
    cat /proc/sys/kernel/random/poolsize
    Hint: /dev/random will deliver one new random number as soon as the pool has reached more than 64 entropy units.
    So why does my box not generate more entropy noise?
    Because only few drivers will fill the entropy pool, first of all keyboard and mouse. Sounds very useful on a server in a datacenter, isn’t it? Some block device and network drivers seem to do so as well, and I have read from guys on the net changing their network card and driver to enjoy this “feature”! But let’s stop ranting, /dev/random is simply made for high security randomness, and if it can’t make sure that randomness is as good as possible in this deterministic world, it stops. Intelligent people have created /dev/urandom for that, like “man 4 random” clearly states:
    A read from the /dev/urandom device will not block waiting for more entropy. As a result, if there is not sufficient entropy in the entropy pool, the returned values are theoretically vulnerable to a cryptographic attack on the algorithms used by the driver. Knowledge of how to do this is not available in the current non-classified literature, but it is theoretically possible that such an attack may exist. If this is a concern in your application, use /dev/random instead.
    Now let’s get back on our JDBC problem. Oracle JDBC 11g seems to use /dev/random by default, which causes usually no trouble on clients running with console access by a user, because his/her unpredictable :) actions will keep the entropy pool well-fed. But to make it usable on a headless server with a latently empty entropy pool, you should do several things, in descending security order (without warranty):
    1. Involve an audio entroy daemon like AED to gather noise from your datacenter with an open microphone, maybe combine it with a webcam noise collector like VED. Other sources are talking about “Cryptographic Randomness from Air Turbulence in Disk devices“. :)
    2. Use the Entropy Gathering Daemon to collect weaker entropy from randomness of userspace programs.
    3. Talk your JDBC into using /dev/urandom instead:
    -Djava.security.egd=file:///dev/urandom"

  • System Hung frequently Generates Core file

    Hi All,
    One of my workstation is getting hung frequently when i kill that session it generates core file and the details are as follows.
    Hardware : Sun Blade 2500
    Memory: 2 GB
    Operating System : Solaris 8 2/04
    Patch Version : Generic_117350-12
    I found following error in core file:
    XtToolkitError.XtToolkitError
    CancelDrag
    typeConversionError.noConverter
    typeConversionError.noConverter
    files.so/usr/lib/locale/C/LC_CTYPE/textboundary.so.1
    override
    rter registered for 'Pixel' to 'SelectColor' conversion.
    No type converter registered for 'Pixel' to 'SelectColor' conversion.
    ListKbdCancel
    Cp9{
    override
    No type converter registered for '%s' to '%s' conversion.
    No type converter registered for '%s' to '%s' conversion.
    Please help me to fix this issue.
    Thanks & Regards,
    Ramana

    Hi,
    These files are getting generated because your JVM is getting crashed frequently.
    When JVM crash, JVM captures the state of the JVM at the time of crash and these files generate.
    Please study your std_server log very carefully and you will find some lines like "Out of Memory" kind of error.
    Now if you can stop the crash of the JVM, these files wont generate any more.
    I have faced the same issue and Fix it using the following ways.
    1)  Upgrade your JVM to the latest version so that better memory management will be in place.
    2) If still these files getting generated, you can tune the new version of the JVM.
    With Regards,
    Saurabh

  • Strange JVM pauses

    Running jrockit-R26.4.0-jdk1.5.0_06 on RHEL4U3 amd64 boxes. 64 bit version.
    We get fairly regular hangs where the JVM is unresponsive. Sometimes they last for a few seconds, sometimes for several minutes. While the JVM is hung I can't get a thread dump.
    I run the jrcmd to print threads and it just hangs until the JVM comes back, then prints a stack trace.
    I'm not sure how to debug this. I've noticed that if I do an strace of the JVM, I'll normally see something like:
    futex(0x5fc388, FUTEX_WAKE, 1) = 1
    rt_sigsuspend(~[USR1 RTMIN RT_1] <unfinished ...>
    --- SIGUSR1 (User defined signal 1) @ 0 (0) ---
    <... rt_sigsuspend resumed> ) = -1 EINTR (Interrupted system call)
    rt_sigreturn(0xa) = -1 EINTR (Interrupted system call)
    futex(0x5fc388, FUTEX_WAKE, 1) = 1
    rt_sigreturn(0x7fbfe06c40) = -1 EINTR (Interrupted system call)
    repeated at semi-regular intervals.
    During one of the pauses I just see:
    futex(0x5fc388, FUTEX_WAKE, 1)
    for the entire time of the pause.

    I checked sar and no swapping is happening:
    12:00:01 AM pswpin/s pswpout/s
    12:10:01 AM 0.00 0.00
    12:20:01 AM 0.00 0.00
    12:30:01 AM 0.00 0.00
    12:40:01 AM 0.00 0.00
    12:50:01 AM 0.00 0.00
    01:00:01 AM 0.00 0.00
    01:10:01 AM 0.00 0.00
    01:20:01 AM 0.06 0.00
    01:30:01 AM 0.00 0.00
    01:40:01 AM 0.00 0.00
    01:50:01 AM 0.00 0.00
    02:00:01 AM 0.00 0.00
    02:10:01 AM 0.00 0.00
    02:20:01 AM 0.00 0.00
    02:30:02 AM 0.00 0.00
    02:40:01 AM 0.00 0.00
    02:50:01 AM 0.00 0.00
    03:00:01 AM 0.00 0.00
    03:10:01 AM 0.00 0.00
    03:20:01 AM 0.00 0.00
    03:30:01 AM 0.00 0.00
    03:40:01 AM 0.00 0.00
    03:50:02 AM 0.00 0.00
    04:00:01 AM 0.00 0.00
    04:10:01 AM 0.00 0.00
    04:20:01 AM 0.00 0.00
    04:30:01 AM 0.00 0.00
    04:40:01 AM 0.00 0.00
    04:50:01 AM 0.00 0.00
    05:00:01 AM 0.00 0.00
    05:10:01 AM 0.00 0.00
    05:20:01 AM 0.00 0.00
    05:30:01 AM 0.00 0.00
    05:40:01 AM 0.00 0.00
    05:50:01 AM 0.00 0.00
    06:00:01 AM 0.00 0.00
    06:10:01 AM 0.00 0.00
    06:20:01 AM 0.00 0.00
    06:30:01 AM 0.00 0.00
    06:40:01 AM 0.00 0.00
    06:50:01 AM 0.00 0.00
    07:00:01 AM 0.00 0.00
    07:10:01 AM 0.00 0.00
    07:20:01 AM 0.00 0.00
    07:30:01 AM 0.00 0.00
    07:40:01 AM 0.00 0.00
    07:50:01 AM 0.00 0.00
    08:00:01 AM 0.00 0.00
    08:10:01 AM 0.00 0.00
    08:20:01 AM 0.00 0.00
    08:20:01 AM pswpin/s pswpout/s
    08:30:01 AM 0.00 0.00
    08:40:01 AM 0.00 0.00
    08:50:01 AM 0.00 0.00
    09:00:01 AM 0.00 0.00
    09:10:01 AM 0.00 0.00
    09:20:01 AM 0.00 0.00
    09:30:01 AM 0.00 0.00
    09:40:01 AM 0.00 0.00
    09:50:01 AM 0.00 0.00
    10:00:01 AM 0.00 0.00
    10:10:01 AM 0.00 0.00
    10:20:01 AM 0.00 0.00

  • Hung application and rejected memory lead detector

    I had two problems that may be related with JRockit.
    1. After a few days, the application stopped processing input data:
    - The application receives input using a server sockit on tcp port 2000
    from which it would still accept a connection. Then the tcp receive queue
    just started building, indicating that the application was not removing data
    from the receive queue.
    - The application rejected connections on the other server sockit it
    advertises - tcp 5000.
    - It also seemed to have disconnected from mysql, but I couldn't be sure
    of that.
    2. To rule out a memory leak, I started up Management Console's wonderful
    Memory Leak detector from a remote machine. I use Management Console and
    Memory Leak detector in this configuration alot so I'm familiar with it.
    The memory leak detector was refused. The console output from the
    application's server follows.
    The application is running on JRockit 1.4.2_05 on SuSE 8.2.
    I'm more concerned and baffled by the application's stalling. Do you have
    any problem isolation ideas? Does the Memory Leak detector rejection
    provides some clue?
    Thanks
    -bash: syntax error near unexpected token `newline'
    tmx@tmx-ccdb:/>
    tmx@tmx-ccdb:/> tmx@tmx-ccdb:/> jrockit.license.LicenseException:
    java.lang.reflect.InvocationTargetException
    -bash: jrockit.license.LicenseException:: Permission denied
    at
    jrockit.memleak.MemLeakFactory.getInstance()Ljrockit.memleak.MemoryLeakDetector;(Unknown
    Source)
    tmx@tmx-ccdb:/> at
    jrockit.memleak.MemLeakFactory.createMemLeakDetector()Ljrockit.memleak.MemoryLeakDetector;(Unknown
    Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at
    jrockit.memleak.MemLeakFactory.getInstance()Ljrockit.memleak.MemoryLeakDetector;(Unknown
    Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at
    com.jrockit.management.rmp.RmpQueries$5.doCommand(Lcom.jrockit.management.rmp.RmpServer;Ljava.lang.String;)Z(RmpQueries.java:184)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at
    com.jrockit.management.rmp.RmpQueries.executeRmpCommand(Lcom.jrockit.common.rmp.RmpPacket;Lcom.jrockit.management.rmp.RmpServer;)Ljava.lang.Boolean;(RmpQueries.java:1127)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at
    com.jrockit.management.rmp.RmpServer.handleCommand(Lcom.jrockit.common.rmp.RmpPacket;)V(RmpServer.java:324)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at
    com.jrockit.management.rmp.RmpServer.runConnectionLoop()V(RmpServer.java:356)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at
    com.jrockit.management.rmp.RmpServer.run()V(RmpServer.java:163)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at java.lang.Thread.run()V(Unknown Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at
    java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> Caused by: java.lang.reflect.InvocationTargetException
    -bash: Caused: command not found
    at
    com.jrockit.management.rmp.RmpQueries$5.doCommand(Lcom.jrockit.management.rmp.RmpServer;Ljava.lang.String;)Z(RmpQueries.java:184)
    tmx@tmx-ccdb:/> at
    jrockit.memleak.MemLeakFactory.createMemLeakDetector()Ljrockit.memleak.MemoryLeakDetector;(Unknown
    Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at
    jrockit.reflect.NativeMethodInvoker.invoke0(Ljava.lang.Object;ILjava.lang.Object;[Ljava.lang.Object;)Ljava.lang.Object;(Unknown
    Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/>         at
    jrockit.reflect.NativeMethodInvoker.invoke(Ljava.lang.Object;[Ljava.lang.Object;)Ljava.lang.Object;(Unknown
    Source)
    -bash: syntax error near unexpected token `('
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/>         at
    java.lang.reflect.Method.invoke(Ljava.lang.Object;[Ljava.lang.Object;I)Ljava.lang.Object;(Unknown
    Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/>         at
    com.jrockit.management.rmp.RmpQueries$5.doCommand(Lcom.jrockit.management.rmp.RmpServer;Ljava.lang.String;)Z(RmpQueries.java:184)
    tmx@tmx-ccdb:/>         ... 9 more
    tmx@tmx-ccdb:/> Caused by: jrockit.license.LicenseException:
    jrockit.license.LicenseException: License not valid: license has expired
    -bash: Caused: command not found
    tmx@tmx-ccdb:/>         at
    com.jrockit.management.rmp.RmpServer.handleCommand(Lcom.jrockit.common.rmp.RmpPacket;)V(RmpServer.java:324)
    tmx@tmx-ccdb:/>         at
    jrockit.license.LicenseManager.initialize0([Ljava.lang.String;)V(Unknown
    Source)
    -bash: syntax error near unexpected token `('
            at
    jrockit.reflect.NativeMethodInvoker.invoke0(Ljava.lang.Object;ILjava.lang.Object;[Ljava.lang.Object;)Ljava.lang.Object;(Unknown
    Source)
    tmx@tmx-ccdb:/>         at
    jrockit.license.LicenseManager.initialize()V(Unknown Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/>         at
    jrockit.memleak.MemLeak.getInstance()Ljrockit.memleak.MemLeak;(Unknown
    Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/>         at
    com.jrockit.management.rmp.RmpServer.runConnectionLoop()V(RmpServer.java:356)
    tmx@tmx-ccdb:/>         ... 12 more
    tmx@tmx-ccdb:/> Caused by: jrockit.license.LicenseException: License not
    valid: license has expired
    -bash: Caused: command not found
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/>         at
    com.jrockit.management.rmp.RmpServer.run()V(RmpServer.java:163)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/>         ... 15 more
    tmx@tmx-ccdb:/>
    tmx@tmx-ccdb:/> tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>
    -bash: syntax error near unexpected token `newline'
    tmx@tmx-ccdb:/>         at java.lang.Thread.run()V(Unknown Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/>         at
    java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> -bash: syntax error near unexpected token `newline'
    tmx@tmx-ccdb:/> tmx@tmx-ccdb:/> jrockit.license.LicenseException: > java.lang.reflect.InvocationTargetException> -bash: jrockit.license.LicenseException:: Permission denied>         at > jrockit.memleak.MemLeakFactory.getInstance()Ljrockit.memleak.MemoryLeakDetector;(Unknown > Source)>         at > com.jrockit.management.rmp.RmpQueries$5.doCommand(Lcom.jrockit.management.rmp.RmpServer;Ljava.lang.String;)Z(RmpQueries.java:184)> tmx@tmx-ccdb:/>         at > jrockit.memleak.MemLeakFactory.createMemLeakDetector()Ljrockit.memleak.MemoryLeakDetector;(Unknown > Source)> -bash: syntax error near unexpected token `('-bash: syntax error near unexpected token `('-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/>         at jrockit.reflect.NativeMethodInvoker.invoke(Ljava.lang.Object;[Ljava.lang.Object;)Ljava.lang.Object;(Unknown Source)-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/>         at com.jrockit.management.rmp.RmpQueries.executeRmpCommand(Lcom.jrockit.common.rmp.RmpPacket;Lcom.jrockit.management.rmp.RmpServer;)Ljava.lang.Boolean;(RmpQueries.java:1127)-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/>         at java.lang.reflect.Method.invoke(Ljava.lang.Object;[Ljava.lang.Object;I)Ljava.lang.Object;(Unknown Source)-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/>         at com.jrockit.management.rmp.RmpServer.handleCommand(Lcom.jrockit.common.rmp.RmpPacket;)V(RmpServer.java:324)tmx@tmx-ccdb:/>         ... 9 moretmx@tmx-ccdb:/> Caused by: jrockit.license.LicenseException: jrockit.license.LicenseException: License not valid: license has expired-bash: Caused: command not foundtmx@tmx-ccdb:/>         at jrockit.license.LicenseManager.initialize0([Ljava.lang.String;)V(Unknown Source)-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/>         at jrockit.license.LicenseManager.initialize()V(Unknown Source)-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/>         at jrockit.memleak.MemLeak.getInstance()Ljrockit.memleak.MemLeak;(Unknown Source)-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/>         ... 12 moretmx@tmx-ccdb:/> Caused by: jrockit.license.LicenseException: License not valid: license has expired-bash: Caused: command not foundtmx@tmx-ccdb:/>         ... 15 moretmx@tmx-ccdb:/>-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         at jrockit.memleak.MemLeakFactory.getInstance()Ljrockit.memleak.MemoryLeakDetector;(Unknown Source)-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/>         at > com.jrockit.management.rmp.RmpQueries$5.doCommand(Lcom.jrockit.management.rmp.RmpServer;Ljava.lang.String;)Z(RmpQueries.java:184)> -bash: syntax error near unexpected token `('-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         at com.jrockit.management.rmp.RmpQueries.executeRmpCommand(Lcom.jrockit.common.rmp.RmpPacket;Lcom.jrockit.management.rmp.RmpServer;)Ljava.lang.Boolean;(RmpQueries.java:1127)-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/>         at > com.jrockit.management.rmp.RmpServer.handleCommand(Lcom.jrockit.common.rmp.RmpPacket;)V(RmpServer.java:324)> -bash: syntax error near unexpected token `('-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/>         at jrockit.reflect.NativeMethodInvoker.invoke0(Ljava.lang.Object;ILjava.lang.Object;[Ljava.lang.Object;)Ljava.lang.Object;(Unknown Source)-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         at com.jrockit.management.rmp.RmpServer.runConnectionLoop()V(RmpServer.java:356)-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/>         at > com.jrockit.management.rmp.RmpServer.run()V(RmpServer.java:163)> -bash: syntax error near unexpected token `('-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         at java.lang.Thread.run()V(Unknown Source)-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/>         at > java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)> -bash: syntax error near unexpected token `('-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/> tmx@tmx-ccdb:/> Caused by: java.lang.reflect.InvocationTargetException-bash: Caused: Permission deniedtmx@tmx-ccdb:/> -bash: Caused: command not found-bash: -bash:: command not foundtmx@tmx-ccdb:/>         at jrockit.memleak.MemLeak.getInstance()Ljrockit.memleak.MemLeak;(Unknown Source)-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/>         ... 12 moretmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         at jrockit.reflect.NativeMethodInvoker.invoke0(Ljava.lang.Object;ILjava.lang.Object;[Ljava.lang.Object;)Ljava.lang.Object;(Unknown Source)-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/>         at > jrockit.reflect.NativeMethodInvoker.invoke(Ljava.lang.Object;[Ljava.lang.Object;)Ljava.lang.Object;(Unknown > Source)> -bash: syntax error near unexpected token `('-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/>tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         at java.lang.reflect.Method.invoke(Ljava.lang.Object;[Ljava.lang.Object;I)Ljava.lang.Object;(Unknown Source)-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/>         ... 9 more> tmx@tmx-ccdb:/> Caused by: jrockit.license.LicenseException: > jrockit.license.LicenseException: License not valid: license has expired> -bash: Caused: command not found> tmx@tmx-ccdb:/>         at > jrockit.license.LicenseManager.initialize0([Ljava.lang.String;)V(Unknown > Source)> -bash: syntax error near unexpected token `('-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         at jrockit.license.LicenseManager.initialize()V(Unknown Source)-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/>         at > jrockit.memleak.MemLeak.getInstance()Ljrockit.memleak.MemLeak;(Unknown > Source)> -bash: syntax error near unexpected token `('-bash: syntax error near unexpected token `('tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         ... 12 more-bash: ...: Permission deniedtmx@tmx-ccdb:/> tmx@tmx-ccdb:/> Caused by: jrockit.license.LicenseException: License not valid: license has expired-bash: Caused: Permission deniedtmx@tmx-ccdb:/> -bash: Caused: command not found-bash: -bash:: command not foundtmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         ... 15 more-bash: ...: Permission deniedtmx@tmx-ccdb:/> tmx@tmx-ccdb:/>-bash: syntax error near unexpected token `newline'tmx@tmx-ccdb:/>-- Jeff                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 

    one addition clue...
    When the application was hung, the Management Console could not connect to
    the JVM. I verified that -Xmanagement was one of the switches using SuSE's
    ps -ax command. I had connected a few days earlier successfully.
    So the JVM was not accepting management connections.
    "Jeff" <[email protected]> wrote in message
    news:[email protected]...
    I had two problems that may be related with JRockit.
    1. After a few days, the application stopped processing input data:
    - The application receives input using a server sockit on tcp port 2000
    from which it would still accept a connection. Then the tcp receive queue
    just started building, indicating that the application was not removing
    data from the receive queue.
    - The application rejected connections on the other server sockit it
    advertises - tcp 5000.
    - It also seemed to have disconnected from mysql, but I couldn't be
    sure of that.
    2. To rule out a memory leak, I started up Management Console's wonderful
    Memory Leak detector from a remote machine. I use Management Console and
    Memory Leak detector in this configuration alot so I'm familiar with it.
    The memory leak detector was refused. The console output from the
    application's server follows.
    The application is running on JRockit 1.4.2_05 on SuSE 8.2.
    I'm more concerned and baffled by the application's stalling. Do you have
    any problem isolation ideas? Does the Memory Leak detector rejection
    provides some clue?
    Thanks
    -bash: syntax error near unexpected token `newline'
    tmx@tmx-ccdb:/>
    tmx@tmx-ccdb:/> tmx@tmx-ccdb:/> jrockit.license.LicenseException:
    java.lang.reflect.InvocationTargetException
    -bash: jrockit.license.LicenseException:: Permission denied
    at
    jrockit.memleak.MemLeakFactory.getInstance()Ljrockit.memleak.MemoryLeakDetector;(Unknown
    Source)
    tmx@tmx-ccdb:/> at
    jrockit.memleak.MemLeakFactory.createMemLeakDetector()Ljrockit.memleak.MemoryLeakDetector;(Unknown
    Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at
    jrockit.memleak.MemLeakFactory.getInstance()Ljrockit.memleak.MemoryLeakDetector;(Unknown
    Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at
    com.jrockit.management.rmp.RmpQueries$5.doCommand(Lcom.jrockit.management.rmp.RmpServer;Ljava.lang.String;)Z(RmpQueries.java:184)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at
    com.jrockit.management.rmp.RmpQueries.executeRmpCommand(Lcom.jrockit.common.rmp.RmpPacket;Lcom.jrockit.management.rmp.RmpServer;)Ljava.lang.Boolean;(RmpQueries.java:1127)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at
    com.jrockit.management.rmp.RmpServer.handleCommand(Lcom.jrockit.common.rmp.RmpPacket;)V(RmpServer.java:324)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at
    com.jrockit.management.rmp.RmpServer.runConnectionLoop()V(RmpServer.java:356)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at
    com.jrockit.management.rmp.RmpServer.run()V(RmpServer.java:163)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at java.lang.Thread.run()V(Unknown Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at
    java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> Caused by: java.lang.reflect.InvocationTargetException
    -bash: Caused: command not found
    at
    com.jrockit.management.rmp.RmpQueries$5.doCommand(Lcom.jrockit.management.rmp.RmpServer;Ljava.lang.String;)Z(RmpQueries.java:184)
    tmx@tmx-ccdb:/> at
    jrockit.memleak.MemLeakFactory.createMemLeakDetector()Ljrockit.memleak.MemoryLeakDetector;(Unknown
    Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> at
    jrockit.reflect.NativeMethodInvoker.invoke0(Ljava.lang.Object;ILjava.lang.Object;[Ljava.lang.Object;)Ljava.lang.Object;(Unknown
    Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/>         at
    jrockit.reflect.NativeMethodInvoker.invoke(Ljava.lang.Object;[Ljava.lang.Object;)Ljava.lang.Object;(Unknown
    Source)
    -bash: syntax error near unexpected token `('
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/>         at
    java.lang.reflect.Method.invoke(Ljava.lang.Object;[Ljava.lang.Object;I)Ljava.lang.Object;(Unknown
    Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/>         at
    com.jrockit.management.rmp.RmpQueries$5.doCommand(Lcom.jrockit.management.rmp.RmpServer;Ljava.lang.String;)Z(RmpQueries.java:184)
    tmx@tmx-ccdb:/>         ... 9 more
    tmx@tmx-ccdb:/> Caused by: jrockit.license.LicenseException:
    jrockit.license.LicenseException: License not valid: license has expired
    -bash: Caused: command not found
    tmx@tmx-ccdb:/>         at
    com.jrockit.management.rmp.RmpServer.handleCommand(Lcom.jrockit.common.rmp.RmpPacket;)V(RmpServer.java:324)
    tmx@tmx-ccdb:/>         at
    jrockit.license.LicenseManager.initialize0([Ljava.lang.String;)V(Unknown
    Source)
    -bash: syntax error near unexpected token `('
    at
    jrockit.reflect.NativeMethodInvoker.invoke0(Ljava.lang.Object;ILjava.lang.Object;[Ljava.lang.Object;)Ljava.lang.Object;(Unknown
    Source)
    tmx@tmx-ccdb:/>         at
    jrockit.license.LicenseManager.initialize()V(Unknown Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/>         at
    jrockit.memleak.MemLeak.getInstance()Ljrockit.memleak.MemLeak;(Unknown
    Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/>         at
    com.jrockit.management.rmp.RmpServer.runConnectionLoop()V(RmpServer.java:356)
    tmx@tmx-ccdb:/>         ... 12 more
    tmx@tmx-ccdb:/> Caused by: jrockit.license.LicenseException: License not
    valid: license has expired
    -bash: Caused: command not found
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/>         at
    com.jrockit.management.rmp.RmpServer.run()V(RmpServer.java:163)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/>         ... 15 more
    tmx@tmx-ccdb:/>
    tmx@tmx-ccdb:/> tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>
    -bash: syntax error near unexpected token `newline'
    tmx@tmx-ccdb:/>         at java.lang.Thread.run()V(Unknown Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/>         at
    java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
    -bash: syntax error near unexpected token `('
    tmx@tmx-ccdb:/> -bash: syntax error near unexpected token `newline'
    tmx@tmx-ccdb:/> tmx@tmx-ccdb:/> jrockit.license.LicenseException: >> java.lang.reflect.InvocationTargetException>> -bash: jrockit.license.LicenseException:: Permission denied>>         at >> jrockit.memleak.MemLeakFactory.getInstance()Ljrockit.memleak.MemoryLeakDetector;(Unknown >> Source)>>         at >> com.jrockit.management.rmp.RmpQueries$5.doCommand(Lcom.jrockit.management.rmp.RmpServer;Ljava.lang.String;)Z(RmpQueries.java:184)>> tmx@tmx-ccdb:/>         at >> jrockit.memleak.MemLeakFactory.createMemLeakDetector()Ljrockit.memleak.MemoryLeakDetector;(Unknown >> Source)>> -bash: syntax error near unexpected token `('> -bash: syntax error near unexpected token `('> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/>         at > jrockit.reflect.NativeMethodInvoker.invoke(Ljava.lang.Object;[Ljava.lang.Object;)Ljava.lang.Object;(Unknown > Source)> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/>         at > com.jrockit.management.rmp.RmpQueries.executeRmpCommand(Lcom.jrockit.common.rmp.RmpPacket;Lcom.jrockit.management.rmp.RmpServer;)Ljava.lang.Boolean;(RmpQueries.java:1127)> -bash: syntax error near unexpected token `('>>> tmx@tmx-ccdb:/>         at > java.lang.reflect.Method.invoke(Ljava.lang.Object;[Ljava.lang.Object;I)Ljava.lang.Object;(Unknown > Source)> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/>         at > com.jrockit.management.rmp.RmpServer.handleCommand(Lcom.jrockit.common.rmp.RmpPacket;)V(RmpServer.java:324)> tmx@tmx-ccdb:/>         ... 9 more> tmx@tmx-ccdb:/> Caused by: jrockit.license.LicenseException: > jrockit.license.LicenseException: License not valid: license has expired> -bash: Caused: command not found> tmx@tmx-ccdb:/>         at > jrockit.license.LicenseManager.initialize0([Ljava.lang.String;)V(Unknown > Source)> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/>         at > jrockit.license.LicenseManager.initialize()V(Unknown Source)> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/>         at > jrockit.memleak.MemLeak.getInstance()Ljrockit.memleak.MemLeak;(Unknown > Source)> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/>         ... 12 more> tmx@tmx-ccdb:/> Caused by: jrockit.license.LicenseException: License not > valid: license has expired> -bash: Caused: command not found> tmx@tmx-ccdb:/>         ... 15 more> tmx@tmx-ccdb:/>> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         at > jrockit.memleak.MemLeakFactory.getInstance()Ljrockit.memleak.MemoryLeakDetector;(Unknown > Source)> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/> -bash: syntax error near unexpected token `('>> tmx@tmx-ccdb:/>         at >> com.jrockit.management.rmp.RmpQueries$5.doCommand(Lcom.jrockit.management.rmp.RmpServer;Ljava.lang.String;)Z(RmpQueries.java:184)>> -bash: syntax error near unexpected token `('> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         at > com.jrockit.management.rmp.RmpQueries.executeRmpCommand(Lcom.jrockit.common.rmp.RmpPacket;Lcom.jrockit.management.rmp.RmpServer;)Ljava.lang.Boolean;(RmpQueries.java:1127)> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/> -bash: syntax error near unexpected token `('>> tmx@tmx-ccdb:/>         at >> com.jrockit.management.rmp.RmpServer.handleCommand(Lcom.jrockit.common.rmp.RmpPacket;)V(RmpServer.java:324)>> -bash: syntax error near unexpected token `('> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/>         at > jrockit.reflect.NativeMethodInvoker.invoke0(Ljava.lang.Object;ILjava.lang.Object;[Ljava.lang.Object;)Ljava.lang.Object;(Unknown > Source)> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         at > com.jrockit.management.rmp.RmpServer.runConnectionLoop()V(RmpServer.java:356)> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/> -bash: syntax error near unexpected token `('>> tmx@tmx-ccdb:/>         at >> com.jrockit.management.rmp.RmpServer.run()V(RmpServer.java:163)>> -bash: syntax error near unexpected token `('> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         at java.lang.Thread.run()V(Unknown > Source)> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/> -bash: syntax error near unexpected token `('>> tmx@tmx-ccdb:/>         at >> java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)>> -bash: syntax error near unexpected token `('> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/> tmx@tmx-ccdb:/> Caused by: > java.lang.reflect.InvocationTargetException> -bash: Caused: Permission denied> tmx@tmx-ccdb:/> -bash: Caused: command not found> -bash: -bash:: command not found> tmx@tmx-ccdb:/>         at > jrockit.memleak.MemLeak.getInstance()Ljrockit.memleak.MemLeak;(Unknown > Source)> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/>         ... 12 more> tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         at > jrockit.reflect.NativeMethodInvoker.invoke0(Ljava.lang.Object;ILjava.lang.Object;[Ljava.lang.Object;)Ljava.lang.Object;(Unknown > Source)> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/> -bash: syntax error near unexpected token `('>> tmx@tmx-ccdb:/>         at >> jrockit.reflect.NativeMethodInvoker.invoke(Ljava.lang.Object;[Ljava.lang.Object;)Ljava.lang.Object;(Unknown >> Source)>> -bash: syntax error near unexpected token `('> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/>> tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         at > java.lang.reflect.Method.invoke(Ljava.lang.Object;[Ljava.lang.Object;I)Ljava.lang.Object;(Unknown > Source)> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/> -bash: syntax error near unexpected token `('>> tmx@tmx-ccdb:/>         ... 9 more>> tmx@tmx-ccdb:/> Caused by: jrockit.license.LicenseException: >> jrockit.license.LicenseException: License not valid: license has expired>> -bash: Caused: command not found>> tmx@tmx-ccdb:/>         at >> jrockit.license.LicenseManager.initialize0([Ljava.lang.String;)V(Unknown >> Source)>> -bash: syntax error near unexpected token `('> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         at > jrockit.license.LicenseManager.initialize()V(Unknown Source)> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/> -bash: syntax error near unexpected token `('>> tmx@tmx-ccdb:/>         at >> jrockit.memleak.MemLeak.getInstance()Ljrockit.memleak.MemLeak;(Unknown >> Source)>> -bash: syntax error near unexpected token `('> -bash: syntax error near unexpected token `('> tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         ... 12 more> -bash: ...: Permission denied> tmx@tmx-ccdb:/> tmx@tmx-ccdb:/> Caused by: > jrockit.license.LicenseException: License not valid: license has expired> -bash: Caused: Permission denied> tmx@tmx-ccdb:/> -bash: Caused: command not found> -bash: -bash:: command not found> tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>         ... 15 more> -bash: ...: Permission denied> tmx@tmx-ccdb:/> tmx@tmx-ccdb:/>> -bash: syntax error near unexpected token `newline'> tmx@tmx-ccdb:/>>>>>>>>>> -- > Jeff                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             

  • JVM spawning mysterious child process of itself using Runtime.exec()

    Hello, I'm not sure if this is how this is supposed to work but I have a java application that monitors legacy c programs and after a period of time (its intermittent), I'll see a duplicate jvm process running the same classpath, classname as a child of the java application monitor. This behaviour can be reproduced with the following simple class running on either solaris 9 or 10 using 1.6.0_03-b05:
    public class Monitor {
    Process procss;
    public Monitor() {
    try {
    Runtime runtime = Runtime.getRuntime();
    for (int i = 0; i < 10000; i++) {
    System.out.println("execing command ls -l.");
    procss = runtime.exec("ls -l");
    procss.waitFor();
    catch (Exception e) {
    e.printStackTrace();
    public static void main(String[] args) {
    new Monitor();
    Using java -classpath ./ Monitor to run it. While this is running, at intermittent times doing a ps -ef you will see a duplicate jvm running whose parent process is the one that was started on the command line. Ie:
    UID PID PPID etc
    user 17434 10706 .... java -classpath ./ Monitor (the one I put running)
    user 27771 17434 .....java -classpath ./ Monitor (intermittently started)
    in another window I'll run the following shell script that will output the processes when a duplicate java process gets started as they don't seem to run very long (on my production system they will occasionally get hung up until I manually kill them):
    #!/usr/bin/ksh
    while ((1 == 1))
    do
    ps -ef | grep "Monitor" | grep -v grep > /tmp/test.out
    VAL=`cat /tmp/test.out | wc -l`
    if (($VAL != 1))
    then
    echo "Duplicate java process started"
    cat /tmp/test.out
    fi
    done
    It takes roughly 30 seconds before I start to see duplicate jvms starting to run. The concern is that is the new jvm instance running the Monitor class? Eventually on my production system the real application will have a child or 2 linger indefinetly, and threads will be deadlocked. Once I kill these child java processes, everything is back to normal. This doesn't seem to occur with the above java class but will show the duplicate child jvm's start to run after a bit.

    This is true for Solaris and Linux. Sun's implementation does a fork. A lot of people who have very large memory java applications wish there was a way to create a process from Java that doesn't involve copying the parent process. As far as I know your stuck.
    A workaround: Use jms, rmi, sockets, or files to communicate with a low memory footprint java application whose sole purpose is to spawn child processes.

  • Sun Convergence hung

    Hi,
    I have 2 instances of Sun Convergence. Both instances run well, but Application (or Application Server) sometimes hung.
    If it happen I have blank login screen, server has no load, can't login or do something.
    I'm still watching with jconsole JVM behaviour and there are nothing strange (resources, connection, threads, memory - JVM looks normal). I register in server.log and jvm.log some jave exception. My Application Sevrer was tuned with Sun Convergence manual.
    I wonder if some other people (Sun Convergence administrators) have noticed some similar.
    Irek

    You could try to disable the file cache in the AS server :
    http://docs.sun.com/app/docs/doc/820-4343/abege?a=view
    Cheers, Ben

  • How do I run multiple java apps in one JVM to reduce memory use?

    Hi all,
    I saw an article either on the web or in a magazine not too long ago about how to "detect" if the app is already running, and if so, it hands off the new instance to the already running JVM, which then creates a thread to run the Java app in. As it turns out, my app will be used in an ASP environment, through Citrix. We may have as many as 50 to 100 users running the same app, each with their own unique user ID, but all using the same one server to run it on. Each instance eats up 25MB of memory right now. So the question is if anybody knows of a URL or an app like this that can handle the process of running the same (or even different Java) apps in one JVM as separate threads, instead of requring several instances of the JVM to run? I know this article presented a fully working example, and I believe I know enough to do it but I wanted ot use the article as a reference to make sure it is done right. I know that each app basically would use the same one "launcher" program that would on first launch "listen" to a port, as well as send a message through the port to see if an existing launcher was running. If it does, it hands off the Java app to be run to the existing luancher application and shuts down the 2nd launching app. By using this method, the JVM eats up its normal memory, but each Java app only consumes its necessary memory as well and doesn't use up more JVM instance memory.
    Thanks.

    <pre>
    import java.util.Properties;
    import java.io.FileInputStream;
    import java.io.IOException;
    import java.lang.reflect.Method;
    import java.lang.reflect.InvocationTargetException;
    import java.util.Enumeration;
    import java.util.NoSuchElementException;
    public class RunProg implements Runnable, Cloneable
    private String iProg;
    private String iArgs[];
    public static void main(String args[])
    new RunProg().main();
    // first step is to start main program itself
    private void main()
    Properties properties = System.getProperties();
    try
    properties.load(new FileInputStream("RunProg.properties"));
    catch(IOException e)
    System.setProperties(properties);
    int i = 0;
    System.out.println("enter main, activeCount=" + Thread.activeCount());
    while(true)
    String program = properties.getProperty("Prog" + i);
    if(program == null)
    break;
    StringTokenizer st = new StringTokenizer(program);
    String[] args = new String[st.countTokens() - 1];
    try
    RunProg rp = (RunProg)this.clone();
    rp.iProg = st.nextToken();
    for(int j = 0; st.hasMoreTokens(); j++)
         args[j] = st.nextToken();
    rp.iArgs = args;
    Thread th = new Thread(rp);
    th.setName("prog" + i + "=" + program);
    th.start();
    System.out.println("prog" + i + "=" + program + ", started");
    catch(CloneNotSupportedException e)
    System.out.println("prog" + i + "=" + program + ", can't start");
    i++;
         System.out.println("end of main, activeCount=" + Thread.activeCount());
    // next step is to start all others one by one
    public void run()
    try
    Class c = Class.forName(iProg);
    Class p[] = new Class[1];
    p[0] = String[].class;
    Method m = c.getMethod("main", p);
    Object o[] = new Object[1];
    o[0] = iArgs;
    m.invoke(null, o);
    catch(ClassNotFoundException e)
    System.out.println(iProg + "ClassNotFoundException");
    catch(NoSuchMethodException e)
    System.out.println(iProg + "NoSuchMethodException");
    catch(InvocationTargetException e)
    System.out.println(iProg + "NoSuchMethodException");
    catch(IllegalAccessException e)
    System.out.println(iProg + "NoSuchMethodException");
    System.out.println(Thread.currentThread().getName() + ", ended");
    System.out.println("exit run, activeCount=" + Thread.activeCount());
    // setup SecurityManager to disable method System.exit()
    public RunProg()
         SecurityManager sm = new mySecurityManager();
         System.setSecurityManager(sm);
    // inner-class to disable method System.exit()
    protected class mySecurityManager extends SecurityManager
         public void checkExit(int status)
              super.checkExit(status);
              Thread.currentThread().stop();
              throw new SecurityException();
    * inner-class to analyze StringTokenizer. This class is enhanced to check double Quotation marks
    protected class StringTokenizer implements Enumeration
    private int currentPosition;
    private int maxPosition;
    private String str;
    private String delimiters;
    private boolean retTokens;
    * Constructs a string tokenizer for the specified string. All
    * characters in the <code>delim</code> argument are the delimiters
    * for separating tokens.
    * <p>
    * If the <code>returnTokens</code> flag is <code>true</code>, then
    * the delimiter characters are also returned as tokens. Each
    * delimiter is returned as a string of length one. If the flag is
    * <code>false</code>, the delimiter characters are skipped and only
    * serve as separators between tokens.
    * @param str a string to be parsed.
    * @param delim the delimiters.
    * @param returnTokens flag indicating whether to return the delimiters
    * as tokens.
    public StringTokenizer(String str, String delim, boolean returnTokens)
    currentPosition = 0;
    this.str = str;
    maxPosition = str.length();
    delimiters = delim;
    retTokens = returnTokens;
    * Constructs a string tokenizer for the specified string. The
    * characters in the <code>delim</code> argument are the delimiters
    * for separating tokens. Delimiter characters themselves will not
    * be treated as tokens.
    * @param str a string to be parsed.
    * @param delim the delimiters.
    public StringTokenizer(String str, String delim)
    this(str, delim, false);
    * Constructs a string tokenizer for the specified string. The
    * tokenizer uses the default delimiter set, which is
    * <code>"&#92;t&#92;n&#92;r&#92;f"</code>: the space character, the tab
    * character, the newline character, the carriage-return character,
    * and the form-feed character. Delimiter characters themselves will
    * not be treated as tokens.
    * @param str a string to be parsed.
    public StringTokenizer(String str)
    this(str, " \t\n\r\f", false);
    * Skips delimiters.
    protected void skipDelimiters()
    while(!retTokens &&
    (currentPosition < maxPosition) &&
    (delimiters.indexOf(str.charAt(currentPosition)) >= 0))
    currentPosition++;
    * Tests if there are more tokens available from this tokenizer's string.
    * If this method returns <tt>true</tt>, then a subsequent call to
    * <tt>nextToken</tt> with no argument will successfully return a token.
    * @return <code>true</code> if and only if there is at least one token
    * in the string after the current position; <code>false</code>
    * otherwise.
    public boolean hasMoreTokens()
    skipDelimiters();
    return(currentPosition < maxPosition);
    * Returns the next token from this string tokenizer.
    * @return the next token from this string tokenizer.
    * @exception NoSuchElementException if there are no more tokens in this
    * tokenizer's string.
    public String nextToken()
    skipDelimiters();
    if(currentPosition >= maxPosition)
    throw new NoSuchElementException();
    int start = currentPosition;
    boolean inQuotation = false;
    while((currentPosition < maxPosition) &&
    (delimiters.indexOf(str.charAt(currentPosition)) < 0 || inQuotation))
    if(str.charAt(currentPosition) == '"')
    inQuotation = !inQuotation;
    currentPosition++;
    if(retTokens && (start == currentPosition) &&
    (delimiters.indexOf(str.charAt(currentPosition)) >= 0))
    currentPosition++;
    String s = str.substring(start, currentPosition);
    if(s.charAt(0) == '"')
    s = s.substring(1);
    if(s.charAt(s.length() - 1) == '"')
    s = s.substring(0, s.length() - 1);
    return s;
    * Returns the next token in this string tokenizer's string. First,
    * the set of characters considered to be delimiters by this
    * <tt>StringTokenizer</tt> object is changed to be the characters in
    * the string <tt>delim</tt>. Then the next token in the string
    * after the current position is returned. The current position is
    * advanced beyond the recognized token. The new delimiter set
    * remains the default after this call.
    * @param delim the new delimiters.
    * @return the next token, after switching to the new delimiter set.
    * @exception NoSuchElementException if there are no more tokens in this
    * tokenizer's string.
    public String nextToken(String delim)
    delimiters = delim;
    return nextToken();
    * Returns the same value as the <code>hasMoreTokens</code>
    * method. It exists so that this class can implement the
    * <code>Enumeration</code> interface.
    * @return <code>true</code> if there are more tokens;
    * <code>false</code> otherwise.
    * @see java.util.Enumeration
    * @see java.util.StringTokenizer#hasMoreTokens()
    public boolean hasMoreElements()
    return hasMoreTokens();
    * Returns the same value as the <code>nextToken</code> method,
    * except that its declared return value is <code>Object</code> rather than
    * <code>String</code>. It exists so that this class can implement the
    * <code>Enumeration</code> interface.
    * @return the next token in the string.
    * @exception NoSuchElementException if there are no more tokens in this
    * tokenizer's string.
    * @see java.util.Enumeration
    * @see java.util.StringTokenizer#nextToken()
    public Object nextElement()
    return nextToken();
    * Calculates the number of times that this tokenizer's
    * <code>nextToken</code> method can be called before it generates an
    * exception. The current position is not advanced.
    * @return the number of tokens remaining in the string using the current
    * delimiter set.
    * @see java.util.StringTokenizer#nextToken()
    public int countTokens()
    int count = 0;
    int currpos = currentPosition;
    while(currpos < maxPosition)
    * This is just skipDelimiters(); but it does not affect
    * currentPosition.
    while(!retTokens &&
    (currpos < maxPosition) &&
    (delimiters.indexOf(str.charAt(currpos)) >= 0))
    currpos++;
    if(currpos >= maxPosition)
    break;
    int start = currpos;
    boolean inQuotation = false;
    while((currpos < maxPosition) &&
    (delimiters.indexOf(str.charAt(currpos)) < 0 || inQuotation))
    if(str.charAt(currpos) == '"')
    inQuotation = !inQuotation;
    currpos++;
    if(retTokens && (start == currpos) &&
    (delimiters.indexOf(str.charAt(currpos)) >= 0))
    currpos++;
    count++;
    return count;
    </pre>
    RunProg.properties like this:
    Prog1=GetEnv 47838 837489 892374 839274
    Prog0=GetEnv "djkfds dfkljsd" dsklfj

  • Communication between multiple JVMs

    We have a Java toolkit that is shipped as a JAR file. The toolkit is ported from a C++ DLL running on Windows. Therefore, in both instances (Java and C++), we can't control who loads us or when.
    I need to communicate between different JVMs running on the same machine. The communication is very simple: "Is this user logged on in your JVM?" I send a string to the other JVM and I get back a boolean. I don't need to worry about crossing machine boundaries. Also, I'm not expecting to have a huge number of JVMs running. Maybe 3 or 4 could be likely. However, the solution does need to scale in case there are more than that. I'm not setting a limit on the number of JVMs either.
    The C++ code handled this situation very easily and elegantly. It created a named system semaphore (mutex) whenever a user logged on. The name of the mutex was the username. So, if there were multiple instances of the DLL running in separate processes (EXEs), we could easily tell if this user was logged on in another instance. We'd try to create the system semaphore - it would fail saying the name already exists. Therefore, we'd know the user was already logged on. The named system semaphore provided the means for a machine-global list - which is exactly what we wanted. It also had this extra benefit: if the process terminates normally or abnormally, the system semaphore is removed from memory. This means: the application is terminated, the user is no longer logged on, and we can relog this user on.
    Therefore, I have 2 requirements:
    1) A machine-global list where we can place a string. Keep in mind, it doesn't absolutely have to be a machine-global list. A suitable means to talk to other JVMs is acceptable too.
    2) If the process exits normally or abnormally, the string(s) get removed (for this JVM) from the list. Abnormal termination is the more important one to focus on because lots of people of varying skill levels use our toolkit. Abnormal terminations can be common.
    The first thought is to store these in a file. That solves #1, but not #2. I've seen the JIPC package. However, I'm not too crazy about requiring 3rd party developers to start up another program (JIPC) before they start up their application. As I said, we're just a toolkit so we can't control when or who loads us. It's not totally out of the question, but I'd prefer something else.
    I have a fairly involved solution that involves sockets. The first JVM creates a ServerSocket on a specific port and becomes the server. Subsequent JVMs also try to create the ServerSocket on the same port. They get a BindException because the ServerSocket already exists, so they know they're clients. Then, they create a client socket and talk to the server that way. This gets a little hairy when the server goes away. The clients will scramble to become the server and then all the other clients need to reconnect to the new server.
    This proposed solution sounds like it will address both requirements. However, I'm looking for something simpler. I'm asking this forum for help in case there's an easier way to do this. I don't have the breadth of experience with Java yet to know if there's a simpler way to fix this. If I have to go with the socket solution, I will. I just didn't want to overlook something simple that is already built into Java.
    Thanks for any tips or suggestions

    Thanks for the response.
    FileLock. We still have to target JDK 1.3 so we can't use FileLocks (at this point)
    JNI: That's an interesting idea. I suspect many people are using our software on Windows. Therefore, we could probably fix it in Windows the same as in the C++ code. If they're not on Windows, we could use the Sockets approach.
    I also had another idea: how about hashing the username string into some integer (or long) value. Then use the hashed value to lock some other resource: like the port number passed to ServerSocket. I know ServerSocket only accepts 0 - 0xFFFF so this obviously won't work. But is there some other system-wide thing we could lock given an integral value?

  • TS1363 I was upgrading iPhone OS from 5.1 to 5.1.1 on Win 7 using iTunes 10.6.1.7.  The iPhone hung.  It is no longer visible by iTunes on Win 7

    While upgrading iPhone OS from 5.1 to 5.1.1 on Win 7 using iTunes 10.6.1.7, the iPhone hung.  I could not get iTunes to find it and Win 7 does not see it.
    When I connect the iPhone to the USB, Win 7 issues the welcome sound, but that is it.  The same happens when I disconnect iPhone.
    The Devise Manager has the Apple Mobile Device USB Driver correct without any problem.  I updated this driver called usbaapl64.
    The iPhone is still hung.
    I restarted the iPod service
    I restarted the Apple Mobile Device Service
    I tried to empty and delete the temp directory in C:\users\myname\AppData\Local\ but could not
    The iTunes I am using is the latest version
    I updated the Apple Mobile Device Driver as stated above correctly
    The iPhone is still hung and is not recognized by Win 7 or iTunes, so I cannot restore the OS to the previous working version
    What else can I do to restore the previous working OS on my iPhone 4

    I was connecting the iPhone 4 to a USB 3.0 port.  I was told by Apple Express Support (at US$29.99) not to use USB 3.0 and to use USB 2.1 instead.
    If you are connecting your iPhone to the computer using a USB 3.0 port, disconnect it immediately.  iPhones will lock up if you try to update while  using a connection through a USB 3.0 port.
    Disconnect the iPhone from the computer
    Go to Device Manager and find the connected USB port.  Click the mouse right button and update the driver
    Shut the iPhone by pressing the sleep button and the home button concurrently for some time
    Move the USB cable to a USB 2.1 port or older without the iPhone.  
    Restart iTunes.
    Connect the iPhone
    Your computer should recognize the phone and reloads the correct USB 2.1 driver (Apple Mobile Device)
    Then iTunes should recognize the iPhone and starts reloading the firmware.
    When that is complete it may ask if you wish to restore, naturally you say YES
    iTunes will restore the iOS
    Using iTunes, restore the latest backup to get your data back.
    When all of the above is completed.  Start enjoying your iPhone.
    If the above does not work.  Throw your iPhone out of the window and get a replacement Android phone.

Maybe you are looking for

  • IChat freezes, please help

    Just today, my macbook is starting to hate me. I turn it on, go into ichat, i connect, i can see my buddies, but i get the spinning wheel, and it NEVER goes away. I cant even force quit ( i click, but it doesnt). And then i noticed that screen captur

  • Can't Remove HD

    I hope someone can help with this. I have a Powermac G5 with 2 HD's in the top and bottom bays. I can't remove the bottom drive because it's really wedged in there. I can't remove the top drive first because of the bottom drive. I've tried to wiggle

  • Ipod 4g wont sync to intunes.

    why wont my ipod touch 4g sync to i tunes. i plug my i pod into the usb port and the sync process starrts. step1, 2, 3, are ok but then itunes freezes on step 4 and i have to ctrl alt del task manager shutdown i tunes and try again.

  • Power supply with geforce 5950 ultra 256

    my computer when im playing sometimes tells me that the video card its not recivin enough power and its going to lower the performance of the video card, i need some help with this i have a power supply of 400 watts processor pentium 4 2.8 motherboar

  • Settings on iPhone to avoid charges when travelling question about carrier

    Hi Want to check setting  on iPhone to avoid charges when travelling on iPhone 5 . I have googled but still wanted to check as I only want to use wifi and with air plane mode I actually recovered a text message which led me to think I was not fully d