JVM Experts - Challenging IBM JVM segfault

We have a 100% java web application (running on Tomcat 5) that is occasionally causing a JVM core dump on a single machine. (Redhat Linux
ES3)
SIGSEGV received in ?? at 0xffffffff in ??. Processing terminated.Full core dump is here:
http://office.bensoft.com/ibmjvm/javacore.20060519.122632.20212.txt
This is a baffling problem that only occurs on a single machine (1 out of 7). Due to client 'policies' I am not privy to the hardware, but I do know it's a dual Xeon with hyperthreading enabled (appears as 4
processors)
We cannot reliably replicate the problem. IT recommends that we 'code around' the issue in the application, however, I am convinced this has nothing to do with the application which has never exhibited a segfault on 6 other machines, some under much heavier load.
The problem has never occured on an identical software stack. Running on a Compaq - single CPU Celeron (Coppermine) box.
IT is very inflexible and insists on using only RHES3 binaries. They are unwilling to bend the rules (at least for us). In other words, we cannot upgrade to the latest IBM JVM release, we cannot use a Sun JVM, and we cannot use a different JVM that is not distributed through a RedHat ES3 channel.
The ultimate goal would be to identify the problem and get Redhat to update the kernel, lib, or JVM which would be downloaded via RHN.
Application stats:
- 100% java web app (no JNI, no system calls)
- Using JasperReports for PDF, JFreeChart for GANTT charts
- Running on Tomcat 5 (RHES3 tomcat5-5.0.28-2jpp_5rh)
- DB is MySQL 4.0.13 (only part that is not RHES3, don't ask :)
Here is the part of the core dump that is really interesting:
1HPTHDDETAILS Current Thread Details
NULL           ----------------------
NULL
3HPREGISTERS    Register Values
NULL            ---------------
3HPREGVALUES     EAX : 00000000, EBX : B2B89EEC, ECX : B1467270
3HPREGVALUES     EDX : B42A5EE8, ESI : B44E9520, EDI : B2B89AC0
3HPREGVALUES     EBP : B35DEB48, ESP : B2B89BA0, EIP : FFFFFFFF
3HPREGVALUES     EFLAGS : 00010292
NULL
3HPNATIVESTACK Native Stack of "http-(ip)-8080-Processor3" PID 1802
NULL            -------------------------
3HPSTACKLINE     FFFFFFFF
3HPSTACKLINE     0
3HPSTACKLINE     0In each case:
- EIP is always FFFFFFFF
- EAX is always 00000000
- the value FFFFFFFF(eip) is always at the top of the stack.
- Under cxia32142-20050609, this problem would occur and the value of
EDX was always B42A5EE8.
- After upgrading to cxia32142-20050929 (via RHN) the problem still
occurs but the value of EDX is now B42A8EE8.
- The Processor that the runaway thread is on changes every time.
I can provide objdump segments of any shared library or binary listed.
A ThinkGeek gift certificate is waiting for anyone who can conclusively solve this problem!!!
Thanks,
Dave Bennett
Bensoft Inc.

Which OS will you be installing on? It's likely that you'll be fine, since you'll be able to configure the primary JVM via update-alternatives.
This is how we configure all of ours:
Install IBM JDK into /opt/
Point installers explicitly to the IBM directory.
Keep the default JDK installed for all other functions.

Similar Messages

IBM's and SAP's

Hi Experts,
So, IBM is having their own specification for SOA and SAP is also having its own specification for SOA. What is the main difference between IBM's and SAP's specifications and their implementations side. IBM has its own stack for SOA is SAP has its own stack for SOA?
Thanks and Regards
Viswanath

Hi Vishwanath,
Nice and interesting query .
Go through this link.
http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/index.jsp?topic=/com.ibm.iea.wsapiw/wsapiw/5.0/IBMandSAP_Intro2SAP_tech/P03_SOA_IBM_ESA_SAP_Khirallah/player.html
It doesn't provide all the information but is quite useful.
Regards,
Piyush

Bechmarking i5 OS with ERP

Hi
We are presently running SAP ECC 6.0 with DBM on i5 OS with Power6 machine. Over these years we were facing lots of performance issues and investment towards i5 OS is also on the higher side. Presently we are at a stage of upgrading our existing machine to Power7. In this process we have searched for the SAP ERP 6.0 benchmark results on i5 OS with power 7 processor. Surprisingly it was found that all power 7 machines are benchmarked with AIX / Linux as operating system and DB2 for Unix as database. Is it that SAP gives better performance on Unix platforms than i5 OS. In addition we have observed that IBM also put lots of efforts in unix to add lot of features like compression, Memory management to better the ERP performance.
Experts pls provide your comments. Your comments are valuable for us to take a decision whether to migrate to Unix platform.

Thank you very much for the detailed reply.
A1:
I am sorry to hear about performance problems on your IBM system, and i do hope they could be resolved to your satisfaction..
Our experience from more than 3500 live SAP systems worldwide running at IBM i customers, is that the great majority of them runs smoothly with no major problems.
Occasional problems do occur , like on other DB/OS platforms, and are usually resolved quickly and competently by the joint IBM SAP support team , or by local BPs with professional SAP-IBM i skills.
When reporting problems, it is important the provide precise OS/DB data information ( e.g. DB/400) to ensure fast and correct problem assignment to the right SAP expert support team.
SAP is a powerful , but also a challenging application, and requires specific skills and knowledge on any OS/DB platform to ensure best performance and operation.
R1:
We presently have around 1000 users using SAP and the configuration of our landscape is Central instance 7 core with 93GB RAM and 2 TB HDD. Which is very huge according to all the experts who has seen the system. Administration cost if you compare . Windows and Unix platform has less administration cost. Simple example is the Online reorg. We are presently shutting down the system for carrying out online reorg as recommended by IBM. AIX has the feature of Online automatic reorg. Every month we carry out an entire system backup by shutting down the system. In AIX and Windows we have the facility to take OS partition also online backup. We are using BRMS for backing up iSeries. Network card teaming in iSeries is not like other OS, where in if one network card goes down other takes over. It is just configured with Virtual IP. There are more customer base for AIX than iSeries because of which IBM as well as SAP is more concentrating on this platform. In our experience there are lots of issues which SAP also has seen for the first time in our environment. I feel it would be better to be in open platform rather than a proprietary OS. Any advantage are we getting by saying the DB is integrated with OS? Lots of page faults appear in the system even after providing 93 GB RAM. Getting IBM i Skilled person is also difficult today compared to Unix. IN SAPTeched's all sessions talks about Windows and Unix not iSeries. If we ask queries no one is aware. If SAP gives equal importance to i5 OS then the person who is delivering the session should be aware. Any services like near zero downtime service which was launched last year was first delivered for Windows and Unix customers. iSeries road map was project as not clear.
A2:
IBM assumes equal SAP performance on AIX and IBM i , as both OSs are running on identical POWER hardware platform. Consequently, the IBM SAP sizing and configuration guidelines are identical for AIX and IBM i , and SAP solution landscapes are designed identically for AIX and IBM i. Also, IBM does run SAP benchmarks on both POWER AIX ( for SAP SD ), and on POWER IBM i ( for SAP BW ). But for economic reasons IBM does not run both benchmarks on both OSs.
R2:
I do not agree with your statement that SAP performance is same on both AIX and IBM i. Both OS architecture is entirely different. Given a normal PC if we install Windows 2000 or Windows XP or Windows 2003 all will have different performance matrix. Neither in TPC nor in SAP benchmarks I could see i5 OS.
A3:
As commented earlier by IBM Rochester , both functions, DB compression, as well as AME, have their pros and cons. They do provide advantages in specific usage scenarios , but also bring along their drawbacks, like e.g. additional system management , higher demand for hardware, or slower response times, in certain situations. Please see SAP Note 980067 on the effects of DB compression , just as an example.
The IBM i DB2 development team is working to provide alternative enhancements, other than compression, e.g. to better manage variable length and LOB data, which is where most of the compression would take place.
While IBM i does not provide DB compression or AME, it does offer advantages in other important areas , like native integration of DB2 into the IBM i operating system, its unique "Single Level Storage" concept for low-effort DB management, or its unique "Subsystem" feature , which allows perfect self managing CPU and memory virtualization for SAP workload - just to name some key benefits. Our existing SAP-IBM i customer base widely confirms these benefits in the feedback we receive in regular user group meetings and in numerous SAP-IBM i customer references across the world.
R3:
I have read through the document that if you enable compression there will be increase in CPU and memory usage by 10 %. But if the overall CPU and memory usage is less by migrating to AIX there itself there will be an initial advantage. After that we will slowly introduce compression.
Apart from all these if an expert from IBM competency center can analyze our existing system and help us to identify this huge usage of resources that would be really helpful. We still haven't got any concrete answer from any of the experts . That is the reason why we are looking for a change.
Edited by: Harish Madhavadas on Aug 4, 2011 6:14 PM
Edited by: Harish Madhavadas on Aug 4, 2011 6:15 PM
Edited by: Harish Madhavadas on Aug 4, 2011 6:56 PM

Problem in payrolls it not showing some of the pay rolls-- ?

Hi experts,
The IBM employess Alistar hired on 9/28/2010 and srinu hired 6/24/10 haven't shown up in their C-Cure system.
Please let me know

Alistar is kung fu fighting
srinu is kung fu fighting
actually, everybody's [Kung fu fighting|http://www.youtube.com/watch?v=jhUkGIsKvn0], yaaaah!

Pls answer this!

Hi,
I have a doubt..!
if we are using "this" will this improve performance
or degrades performance.. Should we use "this" or not?
eg .. part of code....
Class A{
String BB;
A(){
this.BB="";

Hi,
The usage of THIS will degrade the performance. I do not know exactly why, but I know that referencing variables from within is best done without using THIS. This has been told to me by a performance expert of IBM that I consulted once to increase the performance of my (web)aplication.
regards,
Jeroen.

How to configure DHCP in Hyper-V

Hi
Recently I have created 2 Hyper V nodes in Windows 8. One is windows server 2012 and second one is windows 8.
My Question is :
How can I configure DHCP in server 2012 that can provide IP address to windows 8 . I have made win server 2012 a domain controller and also installed DHCP role.
PLZ HELP!!!!!!!!

As Ryan has said in his reply, there's nothing different to setting up DHCP in a Hyper-V VM as there is on a physical host.
If you've not done this in 2012 (R2) before then you may be missing the notification in Server Manager after the install notifying you that there is post installation tasks for DHCP to complete.
I suggest you read the information at the link Alexey posted as there information to take you through step by step.
The link again:
http://blogs.technet.com/b/keithmayer/archive/2012/11/26/configuring-dhcp-in-windows-server-2012-early-experts-challenge-exam-70-410.aspx
Direct step links:
Authorize a DHCP Server
Create and configure scopes
Configure a DHCP reservation
Configure DHCP options
Configure a Client and Server for PXE Boot (BOOTP)
Configure IPv4 DHCP Relay Agent
Configure IPv6 DHCP Relay Agent
Regards,
Steve
My System Center Blog

Problem with IBM JVM or the Thread

Hi All,
I have a class which implements Runnable. The purpose of the program is to read a file and count the number of lines present in that.
I am using the IBM JVM. The logic runs like this..
For each 2048 bytes read , it spawns a new thread with this object and each such new thread counts the no. of newline chars present. Once the thread is finished with its 2048 bytes it upddates an instance variable , count, to that value , updates the boolean done flag to "true" and stops. Once the whole file is read(thereby creating filesize/2048 threads ) , it keeps waiting for the threads to finish and then adds all the counts.
Now the problem is that for some reason these threads seem to hang. There are no static vars present.
Pls help.
Tx

Help yourself first.
If this is "Windows", try typing Ctrl-Break in your window (Type Ctrl-\ if you're on Windows). This will produce a thread dump with all of your threads.
See which ones are running, and what they're doing.
Did you remember to either make your threads daemons (setDaemon(true)), or explicitly exit the run() method of the threads?

Anyone had problems with BouncyCastle on AIX using IBM's JVM 1.3.0?

I'm having an issue on AIX with IBM's JVM 1.3.0; the same code and configuration has worked on other OSs and JVMs, including the IBM JVM on Linux and Win2K. The JVM is barfing with a ClassCastException before even the first line of my code is being executed. The rather cryptic (no pun intended) error message is:
Exception in thread "main" java.lang.ClassCastException: org.bouncycastle.jce.X509Principal
The only reference I've seen on this is here:
http://groups.google.com/groups?q=java.lang.ClassCastException:+org.bouncycastle.jce.X509Principal&hl=en&safe=off&rnum=1&selm=8s3l7u%249ks1%40webint.na.informix.com
Anyone else seen this? Any AIX/IBM JVM 1.3.0/BouncyCastle users out there?
TIA,
Matthew
Matthew T. Adams [[email protected]]
Software Engineer, Architecture Group
www.highwire.com

You have the same problem that I was very annoyed.
Remove your IBMJCEfw.jar from %JAVA_HOME%/jre/lib/ext directory.
Anyway , I believe this message and exception is not appropriate ,
or at least unkind one.
I hope the ClassCastException will be handled and another message
and Exception thrown such as "UnverifiedProviderException" in
future release of JCE.
# I'll submit a bug-report later.
UKAI Hiroshi

Use IBM JVM in presense of 64 and 32 bit

Hello,
I am running WIN7 and have both the 32 and the 64 bit IBM JVM installed on it, the 64 bit IBM JVM being the system JVM. The latter is mandatory for various applications.
I am experiencing problems with the Java plugin of Firefox (version 31.5.0). The plugin is listed as "always activate" in the Add-ons Manager, and when I look into pluginreg.dat, it looks like the correct (i.e. 32 bit) version:
npjp2.dll|$
C:\Program Files (x86)\IBM\Java70\jre\bin\plugin2\npjp2.dll|$
10.71.0.0|$
1414303728000|0|0|0|$
Next Generation Java Plug-in 10.71.0 for Mozilla browsers|$
IBM Developer Kit for Windows,Java,1.7.0|$
Nevertheless, Java content in an application does not show up. When I click on the little Lego brick left of the web address after I clicked on the link requesting the Java-generated content, it says "IBM Developer Kit for Windows is enabled, Continue allowing" but nothing happens, even when I click on the "Continue allowing" button. What is supposed to happen is that a new window opens displaying the Java-generated content. The web-site is allowed to open new windows, and does so for non-Java content, so it cannot be just popup-blocking which prevents the Java content from showing up.
The last thing various support entries here pointed me to is the "Configure Java" panel. I ran it both for the 64 bit and 32 bit JVM. For both, when I try to enable "Mozilla family" for "Default java for browsers" under the "Advanced" tab, the setting does not save. Which perhaps seems correct for the 64 bit JVM, but not for the 32 bit JVM. So I cannot verify whether this setting would help. On the security tab, the site is listed on the exception site list, so this shouldn't be a problem.
Any help or ideas are greatly appreciated!
Regards+thanks, Irene

Hello,
We do have 64 and 32 bit heteregeneous landscapes with Windows 2000 32 bit x86 and Windows 2003 64 bit ia64.
There are no problems for transports between 32 bit and 64 bit systems.
Problems can happen with differences between the différent operating systems.
We had, for instance, an issue with some scripts (bat files) because Microsoft had the very strange idea to change the syntax of the DIR command between win 2000 and win 2003. If your script need to work in the same way in both OS, you have to detect the OS and have the 2 syntaxes in the script...
We also have systems with a 64 bit CI and 32 bit dialog instances.
Here, the problems may be with memory hungry programs running OK on the 64 bit system and short dumping on 32 bit app servers
We had also a bad surprise with Win 2003 IA64 servers when we discovered that some SAP software is not available on this platform. Exemples : Adobe Services or TREX.

Challenge: Different ways to crash the JVM.

This is a challenge to find different ways to crash the JVM with code written in Java.
For starters: Mis-use of the Unsafe class.
Field field = Unsafe.class.getDeclaredField("theUnsafe");
field.setAccessible(true);
Unsafe unsafe = (Unsafe) field.get(null);
unsafe.getObject(null, 0L); // or many variations on methods in this class.

Although I presume you mean the latest JVM (6u11), I can reproduce a crash with 1.5.0_05 fairly frequently using the SQL tool "squirrel" ( [http://squirrel-sql.sourceforge.net/]) during startup (while the splash screen is visible):
# An unexpected error has been detected by HotSpot Virtual Machine:
# SIGSEGV (0xb) at pc=0xb7a02c41, pid=9019, tid=2651868080
# Java VM: Java HotSpot(TM) Server VM (1.5.0_05-b05 mixed mode)
# Problematic frame:
# V [libjvm.so+0x39ec41]The crash is not reproducible on 6u10 (haven't tried it with 6u11), so I don't know how relevant this tidbit actually is but would be glad to check into it with a few other JVMs. For what it's worth, the above happens on RH Enterprise Linux (2.6.9 kernel, smt, Xeon CPU).

IBM J2SE 1.3.0 JVM

Anyone using the IBM J2SE 1.3.0 JVM with weblogic?
According to some reports (Volano Report) It is one of the fastest java
implementations.
/\/\i|<e
Mike Kincer
Solutions Developer/Engineer
Atlas Commerce "ebusiness evolved"
Office phone: +1-607-741-9988
mailto:[email protected] [http://www.atlascommerce.com]

Re: ramifications of using an unsupported config ... basically you are on
your own if you run into any problems! :-)
Srikant.
denis krizanovic wrote:
Hi,
We're in the performance tuning phase of our project and we notice that
the IBM jre's are about 130% faster than the Sun jre's. Based on the
VolanoMark [a server-side jvm benchmark].
We also notice that BEA doesn't "certify" against this jre. Does anyone
know why? or what the ramifications are of using this jre?
regards,
dk---
Srikant, [email protected], http://weblogic.bea.com/, etc.

IBM JVM and WebLogic

I consider using IBM JVM 1.2.2 with WebLogic 5.1. Has anyone tried this
before?
Are there any suggestions about performance and realibility of such
configuration?
Thanks in advance
Wojtek

I am pretty sure that it is not certified. The platform support page for
WebLogic Server is located at:
http://www.weblogic.com/platforms/index.html
This page has the latest and greatest information on the platforms that
WebLogic is certified and supported on.
Thanks,
Michael
Michael Girdley
BEA Systems Inc
"Wojciech Ozimek" <[email protected]> wrote in message
news:[email protected]..
I consider using IBM JVM 1.2.2 with WebLogic 5.1. Has anyone tried this
before?
Are there any suggestions about performance and realibility of such
configuration?
Thanks in advance
Wojtek

Extremely slow performance with ojdbc6.jar on IBM JVMs (Java 6)

We are consistently seeing slow performance (easily demonstrable by the simplest of test cases) while using ojdbc6.jar on IBM JDKs. Pefrormance is normal when we simply opt for ojdbc14.jar under the same JDK, same java program.
Works well
=======
JRockit, ojdbc6.jar
JRockit, ojdbc14.jar
IBM JDK, ojdbc14.jar
A lot slower
=======
IBM JDK, ojdbc6.jar
All we had to do was to write a simple JDBC access program and measure time taken around
Connection conn = DriverManager.getConnection(jdbcURL, dbProps);
The first 3 combinations mentioned above return comparable numbers. The 4th combination takes 3-4 times as much.
We initially saw this during performance test of our software on WebSphere (also supported on Weblogic). Later was able to eliminate WebSphere and Weblogic as the problem areas and narrow it down to the JVM. Reproduce-able on development machines.
We have been using Oracle/WLS since BEA used to market 'Tengah' servers. This is the worst we have seen as far as the qaulity of support staff goes. We first level support seems clueless. We filed this case and the support comes back tells us that IBM JDKs are not supported for WLS! Oh, we have been running WLS/AIX/IBM JVM combination only for 5+ years.

A quick solution to test this would be to do the following:
mv /dev/random /dev/random.bk
ln /dev/urandom /dev/random
Note: The quick solution will disappear after a reboot. You could add it to rc.local to automatically make this change, but I wouldn't recommend it.
The following is from [http://www.usn-it.de/index.php/2009/02/20/oracle-11g-jdbc-driver-hangs-blocked-by-devrandom-entropy-pool-empty/]:
"Oracle 11g JDBC driver hangs blocked by /dev/random – entropy pool empty
On a headless (=without console) network server, the 11g JDBC driver used for (java) application connect may cause trouble. In my case, it refused to connect to the DB without any error, trace or log entry. It simply hung. After several hours, it connected one time, and freezed again. Remote debugging done by the development clarified that it locks after calling SeedGenerator() and SecureRandom().
Reason: The JDBC 11g needs about 40 bytes of secure random numbers, gathered from /dev/random, to encrypt its connect string.
But public-available “man 4 random” says:
When read, the /dev/random device will only return random bytes within the estimated number of bits of noise in the entropy pool. /dev/random should be suitable for uses that need very high quality randomness such as one-time pad or key generation. When the entropy pool is empty, reads from /dev/random will block until additional environmental noise is gathered.
So far so good, now the question arises: Why does this mystic “entropy pool” runs out of gas?
The answer is as simple as unsatisfying: because too less entropy “noise” was generated by the system. You can check the “filling level” (maybe zero?) of your pool and the overall size of the pool (usually 4096) by issuing
cat /proc/sys/kernel/random/entropy_avail
cat /proc/sys/kernel/random/poolsize
Hint: /dev/random will deliver one new random number as soon as the pool has reached more than 64 entropy units.
So why does my box not generate more entropy noise?
Because only few drivers will fill the entropy pool, first of all keyboard and mouse. Sounds very useful on a server in a datacenter, isn’t it? Some block device and network drivers seem to do so as well, and I have read from guys on the net changing their network card and driver to enjoy this “feature”! But let’s stop ranting, /dev/random is simply made for high security randomness, and if it can’t make sure that randomness is as good as possible in this deterministic world, it stops. Intelligent people have created /dev/urandom for that, like “man 4 random” clearly states:
A read from the /dev/urandom device will not block waiting for more entropy. As a result, if there is not sufficient entropy in the entropy pool, the returned values are theoretically vulnerable to a cryptographic attack on the algorithms used by the driver. Knowledge of how to do this is not available in the current non-classified literature, but it is theoretically possible that such an attack may exist. If this is a concern in your application, use /dev/random instead.
Now let’s get back on our JDBC problem. Oracle JDBC 11g seems to use /dev/random by default, which causes usually no trouble on clients running with console access by a user, because his/her unpredictable :) actions will keep the entropy pool well-fed. But to make it usable on a headless server with a latently empty entropy pool, you should do several things, in descending security order (without warranty):
1. Involve an audio entroy daemon like AED to gather noise from your datacenter with an open microphone, maybe combine it with a webcam noise collector like VED. Other sources are talking about “Cryptographic Randomness from Air Turbulence in Disk devices“. :)
2. Use the Entropy Gathering Daemon to collect weaker entropy from randomness of userspace programs.
3. Talk your JDBC into using /dev/urandom instead:
-Djava.security.egd=file:///dev/urandom"

Help! XSQL with IBM JVM and Websphere

We are attempting to use XML/XSQL using IBM's JVM 1.2.2 and IBM Websphere 3.5.2 - when we attempt to execute a simple XSQL statement, we are receiving a Dr. Watson message stating that the JVM has an access violation.
The XSQL queries have been successfully run under IIS using Sun's JVM.
Any suggestions would be greatly appreciated - we are under the gun to get this working.
Thanks in advance!

Steve Muench kindly provided me with the answer to this problem:
"We have repeatedly experienced problems with IBM's JIT compiler when using our Java-based XML components. They work fine with other Java VM's as you've seen, so IBM must have a bug in their JIT compiler.
Disabling the JIT compiler has worked around the problem for 100% of the customers that has reported the trouble with IBM's JDK."
We tried disabling the JIT compilier, and everything worked fine after that.
Thanks, Steve!

Help with IBM Websphere JVM for Opera Mini

Hi,
I just downloaded oper mini. When I try to run it, it say "Please ensure IBM Websphere Micro JVM is installed". Can someone help me with this? Thanks.
Post relates to: Centro (Verizon)

you can download that software at this link.
http://mytreo.net/downloads/ibm-websphere-everyplace-java-virtual-machine,1313.html
Post relates to: Treo 680 (Unlocked GSM)

JVM Experts - Challenging IBM JVM segfault

Similar Messages

Maybe you are looking for