Core dump during malloc
Hi,
In one of my C applications, malloc is dumping core even if the pointer was free'd earlier (this is the behavior for both 32-bit and 64-bit).
So I re-built the application with -lmapmalloc, and it worked fine.
Can any one suggest whether it is ok to link the application with libmapmalloc or not.
Thanks.
This is almost always cause by overwriting areas that you shouldn't be. Preload watchmalloc (man watchmalloc) and see where it blows up.
-r
Similar Messages
-
I get a core dump while making a call to malloc.
Owing to waht reasons can malloc give a coredump?Your application has corrupted the malloc heap data structures.
Use dbx' access checking feature, or use watchmalloc(3MALLOC), or libumem, or "Electric Fence", or ... to
find out where your application corrupts malloc's data structures. -
Core dump in malloc - Signal BUS
When I did performence test for my application, core dump happened serveral times, below list two stack back trace
t@29 (l@29) terminated by signal BUS (invalid address alignment)
0xffffffff7a04f68c: t_delete+0x006c: stx %l2, [%g5 + 0x10]
dbx: warning: can't find file "/local/vendor/R24.2/ACE/ACE_wrappers/ace/.shobj/TP_Reactor.o"
dbx: warning: see `help finding-files'
dbx: warning: can't find file "/local/vendor/R24.2/ACE/ACE_wrappers/ace/.shobj/Reactor.o"
dbx: warning: can't find file "/local/vendor/R24.2/ACE/ACE_wrappers/ace/.shobj/Thread_Adapter.o"
dbx: warning: can't find file "/local/vendor/R24.2/ACE/ACE_wrappers/ace/.shobj/Base_Thread_Adapter.o"
(dbx) where
current thread: t@29
=>[1] t_delete(0x1025badb0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xffffffff7a04f68c
[2] mallocunlocked(0x108, 0x0, 0x10dc40, 0xffffffff7a0a177c, 0xfffffc00, 0xffffffff), at 0xffffffff7a04ecc8
[3] malloc(0x108, 0xffffffff73d030b4, 0x10dc40, 0xffffffff7a0a177c, 0xfffffc00, 0xffffffff), at 0xffffffff7a04eaf4
[4] operator new(0x108, 0xffffffff73d030b4, 0x10dc40, 0xffffffff7a0a177c, 0x0, 0x0), at 0xffffffff7df50274
[5] MAS_TCPMessageFactory::makeNewMessage(0x102310230, 0x0, 0x310e48, 0xffffffff7c640b1c, 0x1020048f8, 0x0), at 0x101cf3ac4
[6] MAS_MessageFactory::getMessage(0x102310230, 0x0, 0x8, 0x0, 0xffffffff726eb040, 0x1020048f8), at 0x101d08f1c
[7] MAS_ProtocolLayerMgr::createMessage(0x10230fbb0, 0x1026fd230, 0x102310230, 0x0, 0x0, 0xffffffff726eb140), at 0x101ccbda4
[8] MAS_TCPSession::handle_input(0x1025b4b80, 0x1026fd230, 0x0, 0x1020048f8, 0x10230fbb0, 0x12800), at 0x101cf6ef8
[9] ACE_TP_Reactor::dispatch_socket_event(0x1023102b0, 0xffffffff726fb870, 0xffffffff726fb870, 0x0, 0x1, 0x0), at 0xffffffff7c6ee264
[10] ACE_TP_Reactor::handle_socket_events(0x1023102b0, 0xffffffff726fb97c, 0xffffffff726fba68, 0x0, 0x0, 0x0), at 0xffffffff7c6ed804
[11] ACE_TP_Reactor::dispatch_i(0x1023102b0, 0x0, 0xffffffff726fba68, 0x0, 0x0, 0x0), at 0xffffffff7c6ed2b4
[12] ACE_TP_Reactor::handle_events(0x1023102b0, 0x0, 0x3, 0x2, 0x102229220, 0xffffffff7adfe330), at 0xffffffff7c6ec7f0
[13] ACE_Reactor::run_reactor_event_loop(0x1022fb870, 0x0, 0x319144, 0x101cc6638, 0x10230fbb0, 0x1022291f0), at 0xffffffff7c6c0118
[14] initReactor(0x0, 0x1022fb870, 0xffffffff7cb82e18, 0x1020048f8, 0xffffffff7f0c2ed8, 0x0), at 0x101ceb07c
[15] ACE_Thread_Adapter::invoke_i(0x10232b1c0, 0x10232aee0, 0x0, 0x0, 0x0, 0x0), at 0xffffffff7c67106c
[16] ACE_Thread_Adapter::invoke(0x10232b1c0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xffffffff7c670f14
[17] ace_thread_adapter(0x10232b1c0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xffffffff7c531f88
(dbx)
Another one
t@47 (l@47) terminated by signal BUS (invalid address alignment)
0xffffffff7a14ec9c: mallocunlocked+0x018c: ldx [%l0], %g4
dbx: warning: can't find file "/local/vendor/R24.2/ACE/ACE_wrappers/ace/.shobj/T
ask.o"
dbx: warning: see `help finding-files'
dbx: warning: can't find file "/local/vendor/R24.2/ACE/ACE_wrappers/ace/.shobj/T
hread_Adapter.o"
dbx: warning: can't find file "/local/vendor/R24.2/ACE/ACE_wrappers/ace/.shobj/B
ase_Thread_Adapter.o"
(dbx) where
current thread: t@47
=>[1] mallocunlocked(0x100, 0x0, 0x10dc40, 0xffffffff7a1a177c, 0x0, 0x0), at 0
xffffffff7a14ec9c
[2] malloc(0x100, 0x18c314, 0x0, 0x3fff9e35, 0xfffffc00, 0xffffffff), at 0xfff
fffff7a14eaf4
[3] operator new(0x100, 0x821, 0x7fff0000, 0x2, 0xffffffff703fb7b0, 0x0), at 0
xffffffff7df50274
[4] std::vector<std::basic_string<char,std::char_traits<char>,std::allocator<c
har> >,std::allocator<std::basic_string<char,std::char_traits<char>,std::allocat
or<char> > > >::__insert_aux(0xffffffff703fb930, 0x0, 0xffffffff703fb908, 0x0, 0
xffffffff703fb948, 0x20), at 0x101bd8178
[5] MAS_ProtocolLayerMgr::getAllSessions(0xffffffff703fba20, 0x102208ec0, 0xff
ffffff703fb920, 0x102209018, 0x32808c, 0xffffffff703fb950), at 0x101bf7b88
[6] MAS_ProtocolLayerMgr::audit(0x102208ec0, 0x9496d, 0x8, 0x4364bd37, 0x9496d
, 0x9496d), at 0x101bf7c70
[7] MAS_BasicActiveObject::svc(0x102208ec0, 0x97063, 0xffffffff703fbb50, 0x3e8
, 0x8, 0x0), at 0x101c2667c
[8] ACE_Task_Base::svc_run(0x102208ec0, 0x1, 0x0, 0x0, 0x0, 0x0), at 0xfffffff
f7c93be04
[9] ACE_Thread_Adapter::invoke_i(0x10222b420, 0x102224760, 0x0, 0x0, 0x0, 0x0)
, at 0xffffffff7c67106c
[10] ACE_Thread_Adapter::invoke(0x10222b420, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xff
ffffff7c670f14
[11] ace_thread_adapter(0x10222b420, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xffffffff7c
531f88
(dbx) quit
Could any one share some light for me?
thanksFound the root cause. It was the heap corruption that lead to the coredump. watchmalloc is a good malloc debug library. It helped me to find the bug.
thanks for your help -
ORA-07445: exception encountered: core dump during database open
Hi,
after running OracleXE for 1 year on my notebook, I am running into a strange problem. During startup of the database ( opening the DB ) I get the following error:
ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [__VInfreq__kmmlrl+6329] [PC:0x2D7DF01] [ADDR:0x8] [UNABLE_TO_READ] []
I then tried to reinstall the DB but the problem persists. I have no clues what influences this error.
Any ideas welcomeThe machine is a notebook which is remotely administrated. The admin says there has not been a major software update.
The notebook has 2 GB of main memory and 4 GB of free disk space.
The database was up and running since 1 year. The first time I ran into this problem, was when I dropped a user from database. But that for sure was not the reason, as I totally removed oraclexe and reinstalled it from scratch and the problem persists.
Interestingly the installation doesn't fail, but the final startup, so it might be one of the last packages being installed which produces the error.
I'm looking forward to find some more details. -
Hotspot core dumping during JVM garbage collection ?
We have an application which calls a 3rd party supplied server API which has recently been upgraded to use Java 1.5
We are getting the following error reported by our client application. The application is also now running Java 1.5 but references many classes in jar files which would have quite old code in.
The supplier of the API has stated that the problem requires us to recompile all our jar files using v 1.5 ( including things like jconnect and jms ?!?!? ). This sounds like a bit of a cop-out to me, not to mention being impossible since we don't have the source for things like jconnect.
I suspect that there is a garbage collection problem at the bottom of all this, but I'm not sure how I can "prove" this, nor do I currently have any real clue as to how to fix any GC problem that may exist.
The application is supposed to wait for a message on a MQSeries queue and then transforms it via Xalan XSLT and sends it to a Server application. I've tried playing around with heap sizes etc but that just seems to make it worse. If I leave it at the settings that the previous version used then the client at least manages to process a couple of messages before core dumping. There doesn't seem to be a consistent trigger event to cause the core dump ( it's not a message arriving on a queue for example ) but it does seem to be fairly consistent timewise, i.e. after
Any ideas gratefully accepted.
Here's a logfile excerpt from my applications showing the Hotspot error message :
=====================================================================================
08-Jul-2008 10:01:05 Waiting for messages from COLT.BBFS
08-Jul-2008 10:02:05 Waiting for messages from COLT.BBFS
08-Jul-2008 10:03:05 Waiting for messages from COLT.BBFS
405.815: [GC [PSYoungGen: 17331K->9244K(37632K)] 111702K->103615K(192128K), 0.1615910 secs]
405.977: [Full GC#
# An unexpected error has been detected by HotSpot Virtual Machine:
# SIGBUS (0xa) at pc=0xfe141348, pid=2600, tid=8
# Java VM: Java HotSpot(TM) Server VM (1.5.0_03-b07 mixed mode)
# Problematic frame:
# V [libjvm.so+0x141348]
# An error report file with more information is saved as hs_err_pid2600.log
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
=====================================================================================
The logfile referred to in the error message contains the following.
=====================================================================================
# An unexpected error has been detected by HotSpot Virtual Machine:
# SIGBUS (0xa) at pc=0xfe141348, pid=2600, tid=8
# Java VM: Java HotSpot(TM) Server VM (1.5.0_03-b07 mixed mode)
# Problematic frame:
# V [libjvm.so+0x141348]
--------------- T H R E A D ---------------
Current thread (0x001484d8): VMThread [id=8]
siginfo:si_signo=10, si_errno=0, si_code=1, si_addr=0x0000080f
Registers:
O0=0x00487588 O1=0xfe7d6454 O2=0x000079b0 O3=0x00007800
O4=0x00008868 O5=0x00147d48 O6=0xf8781460 O7=0xfe0f7938
G1=0xe52aaae8 G2=0x00000003 G3=0x00000003 G4=0x001484d8
G5=0xf8781d98 G6=0x00000002 G7=0xf8781d98 Y=0x805683e2
PC=0xfe141348 nPC=0xfe14134c
Top of Stack: (sp=0xf8781460)
0xf8781460: fe786000 00c6ba20 fe10013c e54bab68
0xf8781470: 0000080f e54bab6c 0000080f 00000004
0xf8781480: 00487588 00000134 e54bab6c 00000004
0xf8781490: 00000001 00000000 f87814c0 fe17aef4
0xf87814a0: fe6485b4 fe7d899c 001484d8 0011da40
0xf87814b0: 00148988 00148c10 00148d7c f8781880
0xf87814c0: 007c3389 007c3c0e 00000f87 00008868
0xf87814d0: 00008800 00487588 fe141310 fe7d6454
Instructions: (pc=0xfe141348)
0xfe141338: ec 06 c0 1a 80 a5 a0 00 22 40 00 0a ba 07 60 01
0xfe141348: f2 05 a0 00 ae 0e 60 03 80 a5 e0 03 22 40 00 05
Stack: [0xf8702000,0xf8781d98), sp=0xf8781460, free space=509k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x141348]
V [libjvm.so+0x17aefc]
V [libjvm.so+0x2d557c]
V [libjvm.so+0x300ef8]
V [libjvm.so+0x301e84]
V [libjvm.so+0x2ff950]
V [libjvm.so+0x29df30]
V [libjvm.so+0x362b44]
V [libjvm.so+0x6436f0]
VM_Operation (0xe03012b0): parallel gc system gc, mode: safepoint, requested by thread 0x0031bca0
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x00b2c028 JavaThread "Thread-4" [_thread_in_native, id=85]
0x007f5048 JavaThread "Thread-0" [_thread_blocked, id=84]
0x00c27cf0 JavaThread "Notification Delivery" [_thread_blocked, id=81]
0x0026fa08 JavaThread "RMI LeaseChecker" daemon [_thread_blocked, id=73]
0x00821048 JavaThread "RMI RenewClean-[162.11.2.32:44425]" daemon [_thread_blocked, id=70]
0x0031bca0 JavaThread "GC Daemon" daemon [_thread_blocked, id=67]
0x00cd5d28 JavaThread "RMI Reaper" [_thread_blocked, id=66]
0x003c9300 JavaThread "Timer-0" daemon [_thread_blocked, id=65]
0x00929fe0 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=64]
0x0089bf18 JavaThread "SeedGenerator Thread" daemon [_thread_blocked, id=42]
0x00c47248 JavaThread "Pool thread #7" daemon [_thread_blocked, id=38]
0x00c466a0 JavaThread "Pool thread #6" daemon [_thread_blocked, id=37]
0x00311850 JavaThread "Pool thread #5" daemon [_thread_blocked, id=36]
0x00287a40 JavaThread "Pool thread #4" daemon [_thread_blocked, id=35]
0x00286e98 JavaThread "Pool thread #3" daemon [_thread_blocked, id=34]
0x00c134b0 JavaThread "Pool thread #2" daemon [_thread_blocked, id=33]
0x00ad09e0 JavaThread "Pool thread #1" daemon [_thread_blocked, id=32]
0x00286cd8 JavaThread "PoolThreadManager" daemon [_thread_blocked, id=31]
0x00c129e0 JavaThread "Channel Reaper" daemon [_thread_blocked, id=30]
0x00c669e8 JavaThread "ORB Daemon Thread" daemon [_thread_blocked, id=29]
0x00b10170 JavaThread "Worker for ServerProtocol: (iiop) /0.0.0.0:20168" daemon [_thread_blocked, id=22]
0x008a17e0 JavaThread "Syn~ Client" daemon [_thread_blocked, id=21]
0x003dc378 JavaThread "PoolScavenger0" daemon [_thread_blocked, id=20]
0x0015a928 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=15]
0x00159880 JavaThread "CompilerThread1" daemon [_thread_blocked, id=14]
0x00158a18 JavaThread "CompilerThread0" daemon [_thread_blocked, id=13]
0x00157b98 JavaThread "AdapterThread" daemon [_thread_blocked, id=12]
0x00156dc8 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=11]
0x0014ccd8 JavaThread "Finalizer" daemon [_thread_blocked, id=10]
0x0014ad90 JavaThread "Reference Handler" daemon [_thread_blocked, id=9]
0x00038238 JavaThread "main" [_thread_in_native, id=1]
Other Threads:
=>0x001484d8 VMThread [id=8]
0x0015c3b0 WatcherThread [id=16]
VM state:at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00037728/0x00037758] Threads_lock - owner thread: 0x001484d8
[0x00033650/0x00037ba8] Heap_lock - owner thread: 0x0031bca0
Heap
PSYoungGen total 37632K, used 9244K [0xf2eb0000, 0xf7ba0000, 0xf8400000)
eden space 28352K, 0% used [0xf2eb0000,0xf2eb0000,0xf4a60000)
from space 9280K, 99% used [0xf4a60000,0xf5367080,0xf5370000)
to space 25216K, 0% used [0xf6300000,0xf6300000,0xf7ba0000)
PSOldGen total 154496K, used 94371K [0xe8400000, 0xf1ae0000, 0xf2eb0000)
object space 154496K, 61% used [0xe8400000,0xee028e78,0xf1ae0000)
PSPermGen total 35584K, used 18260K [0xe4400000, 0xe66c0000, 0xe8400000)
object space 35584K, 51% used [0xe4400000,0xe55d5158,0xe66c0000)
Dynamic libraries:
0x00010000 /dsdvlp/java/jvm/jdk1.5.0_03/bin/java
0xff350000 /usr/lib/libthread.so.1
0xff340000 /usr/lib/libdl.so.1
0xff200000 /usr/lib/libc.so.1
0xff390000 /usr/platform/SUNW,Sun-Fire-880/lib/libc_psr.so.1
0xfe000000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/server/libjvm.so
0xff1e0000 /usr/lib/libsocket.so.1
0xff2d0000 /usr/lib/libsched.so.1
0xff1b0000 /usr/lib/libCrun.so.1
0xff160000 /usr/lib/libm.so.1
0xff080000 /usr/lib/libnsl.so.1
0xff060000 /usr/lib/libmp.so.2
0xff030000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/native_threads/libhpi.so
0xfdfc0000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/libverify.so
0xfdf80000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/libjava.so
0xfdf50000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/libzip.so
0xfb7e0000 /usr/lib/locale/en_GB.ISO8859-1/en_GB.ISO8859-1.so.2
0xe4190000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/libnet.so
0xe3bd0000 /dsdvlp/lib/5/libSolarisNatives.so
0xe3e90000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/librmi.so
VM Arguments:
jvm_args: -Djava.ext.dirs=/dsdvlp/java/tmijar/firs7/lib/cli:/dsdvlp/java/tmijar/firs7/lib/cli/ext:/dsdvlp/java/tmijar/firs7/lib/cmn/OpenORB:/dsdvlp/java/tmijar/firs7/lib/cmn/OpenORB/ext:/dsdvlp/java/tmijar/firs7/lib/cmn:/dsdvlp/java/tmijar/firs7/lib/cmn/ext:/dsdvlp/java/tmijar/firs7/daemonlib -Duser.dir=/dsdvlp/java/tmijar/firs7 -Dopenorb.config=file:/dsdvlp/java/tmijar/firs7/configs/OpenORB/config/SynOpenORB.xml -Dopenorb.home=file:/dsdvlp/java/tmijar/firs7/configs/OpenORB -Dcom.coexis.syn.general.orbbinding=com.coexis.syn.general.orbbinding.openorb.OpenORBBinding_1_4 -Dsun.rmi.dgc.client.gcInterval=360000 -Dsun.rmi.dgc.server.gcInterval=360000000 -Xms32m -Xmx256m -Dcom.coexis.syn.clientcommandsconfiglocation=file://localhost//dsdvlp/java/tmijar/firs7/configs/clientcommands.xml -Dcom.coexis.syn.clientconfiglocation=file://localhost//dsdvlp/java/tmijar/firs7/configs/fsbbtd_client.xml -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
java_command: com.coexis.syn.mqmessaging.daemon.RunDaemon -p /dsdvlp/bin/5/lndsfsd_fsbbtd.properties start
Environment Variables:
JAVA_HOME=/dsdvlp/java/jvm/jdk150
CLASSPATH=.:/dsdvlp/java/jar/jconnect520.jar:/dsdvlp/java/jar/vbjapp340.jar:/dsdvlp/java/jar/vbjorb340.jar:/dsdvlp/java/jar/javax_jndi120.jar
PATH=/usr/local/etc:/usr/lang:/usr/openwin/bin:/usr/ucb:/bin:/usr/etc:/usr/local/5/bin:/dsdvlp/bin/5:/dsdvlp/bin/4:/home/app/sybase/5/bin:/home/app/sybase/5/localscripts:/home/app/sybase/5/sqr:/home/app/lang:/home/app/lang/SC2.0.1:/usr/ccs/bin:/usr/local/opt/Acrobat3/bin:/dsdvlp/bin:.
LD_LIBRARY_PATH=/dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/server:/dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc:/dsdvlp/java/jvm/jdk1.5.0_03/jre/../lib/sparc:/usr/lib:/usr/openwin/lib:/usr/local/5/lib:/dsdvlp/lib/5:/dstest/lib/5:/home/app/sybase/5/lib:/dstest/cats/sun4/lib:/tmitest/Opus/opus/lib
SHELL=/bin/csh
DISPLAY=CLI00184.mfil.local:1.0
OS=5
--------------- S Y S T E M ---------------
OS: Solaris 8 2/02 s28s_u7wos_08a SPARC
Copyright 2002 Sun Microsystems, Inc. All Rights Reserved.
Assembled 18 December 2001
uname:SunOS 5.8 Generic_117350-20 sun4u (T1 libthread)
rlimit: STACK 8192k, CORE 9216k, NOFILE 4096, AS infinity
load average:2.24 2.67 2.68
CPU:total 4 has_v8, has_v9, has_vis1, has_vis2, is_ultra3
Memory: 8k page, physical 8388608k(166384k free)
vm_info: Java HotSpot(TM) Server VM (1.5.0_03-b07) for solaris-sparc, built on Apr 13 2005 03:31:26 by unknown with unknown Workshop:0x550The very first suggestion I have is to move your VM to a more recent update of 1.5.0.
It looks like you are crashing with 5.0u3, and I'm pretty sure 5.0u16 is available. You
don't want to waste your time chasing a bug that's already been fixed. -
11gr2 crsd core dump during failover or start attempt on second node
Hi,
I installed 11gr2 with ASM on one node (solaris SPARC). Then I added another node to this cluster (via addNode.sh script).
Than got strange error: If my first node is up, second node is started fine and run well. If I shutdown first node - crsd on second node dump to core and fails to restart. I get the same error if I try to start second node when the first one is down.
In the crsd.log I see the following:
[ clsdmt][2]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=mskbkp2DBG_CRSD))
2010-03-03 17:31:35.330: [ clsdmt][2]PID for the Process [18669], connkey 1
2010-03-03 17:31:35.331: [ clsdmt][2]Creating PID [18669] file for home /u01/grid/11.2.0 host mskbkp2 bin crs to /u01/grid/11
.2.0/crs/init/
2010-03-03 17:31:35.331: [ clsdmt][2]Writing PID [18669] to the file [u01/grid/11.2.0/crs/init/mskbkp2.pid]
2010-03-03 17:31:35.925: [ default][1] CRS Daemon Starting
2010-03-03 17:31:35.933: [ default][1] ENV Logging level for Module: AGENT 1
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: AGFW 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLSFRAME 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLSVER 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLUCLS 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: COMMCRS 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: COMMNS 0
2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSAPP 0
2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCCL 0
2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCEVT 0
2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCOMM 1
2010-03-03 17:31:35.936: [ CRSD][1] ENV Debug Level(CRSD): 50
2010-03-03 17:31:35.936: [ CRSD][1] ENV Logging level for Module: CRSD 50
2010-03-03 17:31:35.937: [ CRSD][1] ENV Debug Level(CRSEVT): 0
2010-03-03 17:31:35.937: [ CRSD][1] ENV Logging level for Module: CRSEVT 0
2010-03-03 17:31:35.937: [ CRSD][1] ENV Debug Level(CRSMAIN): 1
2010-03-03 17:31:35.937: [ CRSD][1] ENV Logging level for Module: CRSMAIN 1
2010-03-03 17:31:35.937: [ CRSD][1] ENV Debug Level(CRSOCR): 0
2010-03-03 17:31:35.937: [ CRSD][1] ENV Logging level for Module: CRSOCR 0
2010-03-03 17:31:35.939: [ CRSD][1] ENV Debug Level(CRSPE): 0
2010-03-03 17:31:35.939: [ CRSD][1] ENV Logging level for Module: CRSPE 0
2010-03-03 17:31:35.939: [ CRSD][1] ENV Debug Level(CRSPLACE): 0
2010-03-03 17:31:35.939: [ CRSD][1] ENV Logging level for Module: CRSPLACE 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Debug Level(CRSRES): 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Logging level for Module: CRSRES 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Debug Level(CRSRPT): 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Logging level for Module: CRSRPT 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Debug Level(CRSRTI): 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Logging level for Module: CRSRTI 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Debug Level(CRSSE): 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Logging level for Module: CRSSE 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Debug Level(CRSSEC): 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Logging level for Module: CRSSEC 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Debug Level(CRSSHARED): 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Logging level for Module: CRSSHARED 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(CRSTIMER): 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Logging level for Module: CRSTIMER 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(CRSUI): 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Logging level for Module: CRSUI 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(CSSCLNT): 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Logging level for Module: CSSCLNT 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(OCRAPI): 1
2010-03-03 17:31:35.942: [ CRSD][1] ENV Logging level for Module: OCRAPI 1
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(OCRASM): 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Logging level for Module: OCRASM 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Debug Level(OCRCAC): 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Logging level for Module: OCRCAC 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Debug Level(OCRCLI): 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Logging level for Module: OCRCLI 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Debug Level(OCRMAS): 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Logging level for Module: OCRMAS 1
2010-03-03 17:31:35.944: [ CRSD][1] ENV Debug Level(OCRMSG): 1
2010-03-03 17:31:35.944: [ CRSD][1] ENV Logging level for Module: OCRMSG 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(OCROSD): 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Logging level for Module: OCROSD 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(OCRRAW): 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Logging level for Module: OCRRAW 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(OCRSRV): 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Logging level for Module: OCRSRV 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(OCRUTL): 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Logging level for Module: OCRUTL 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(SuiteTes): 1
2010-03-03 17:31:35.946: [ CRSD][1] ENV Logging level for Module: SuiteTes 1
2010-03-03 17:31:35.946: [ CRSD][1] ENV Debug Level(UiServer): 0
2010-03-03 17:31:35.946: [ CRSD][1] ENV Logging level for Module: UiServer 0
2010-03-03 17:31:35.946: [ CRSMAIN][1] Checking the OCR device
2010-03-03 17:31:35.948: [ CRSMAIN][1] Connecting to the CSS Daemon
2010-03-03 17:31:35.976: [ CRSMAIN][1] Initializing OCR
2010-03-03 17:31:35.981: [ OCRAPI][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
2010-03-03 17:31:35.981: [ OCRAPI][1]Check namebufs
2010-03-03 17:31:35.981: [ OCRAPI][1]Finished checking namebufs
2010-03-03 17:31:35.982: [ GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsinet.c : 3232
], original from [clsss.c : 5026]
2010-03-03 17:31:36.036: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
2010-03-03 17:31:36.037: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
2010-03-03 17:31:36.059: [ GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 680
], original from [clsss.c : 5026]
2010-03-03 17:31:36.067: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
fatal if both fail
2010-03-03 17:31:36.068: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
f 2: file wallet (LSKP-FSW)
2010-03-03 17:31:36.068: [ GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:36.069: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
2: file wallet (LSKP-FSW) OK
2010-03-03 17:31:36.069: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
f 2: OLR wallet (LSKP-CLSW-OLR)
[ CLWAL][1]clsw_Initialize: OLR initlevel [30000]
2010-03-03 17:31:36.080: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
2: OLR wallet (LSKP-CLSW-OLR) OK
2010-03-03 17:31:36.081: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
providers - fatal if all fail)
2010-03-03 17:31:36.081: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:36.152: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:36.152: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:36.152: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:36.172: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
providers - fatal if all fail)
2010-03-03 17:31:36.172: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:36.238: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:36.239: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:36.239: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:36.239: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=0
2010-03-03 17:31:36.541: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
(0)
2010-03-03 17:31:36.552: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
s (0)
2010-03-03 17:31:36.771: [ OCRRAW][1]proprioo: for disk 0 (+DR2_BIN), id match (1), total id sets, (1) need recover (0), my v
otes (0), total votes (0), commit_lsn (9), lsn (9)
2010-03-03 17:31:36.771: [ OCRRAW][1]proprioo: my id set: (833490748, 1028247821, 0, 0, 0)
2010-03-03 17:31:36.772: [ OCRRAW][1]proprioo: 1st set: (833490748, 1028247821, 0, 0, 0)
2010-03-03 17:31:36.772: [ OCRRAW][1]proprioo: 2nd set: (0, 0, 0, 0, 0)
2010-03-03 17:31:36.830: [ OCRSRV][1]th_init: Successfully retrieved CSS misscount [31].
2010-03-03 17:31:36.830: [ OCRSRV][1]th_init: Successfully query CLSS mode [3].
[ OCRMAS][20]th_calc_av:5': Rturn persisted AV [186646784] [11.2.0.1.0]
2010-03-03 17:31:36.920: [ OCRSRV][20]th_not_master_change: Master change callback not registered
2010-03-03 17:31:36.920: [ OCRMAS][20]th_master:12: I AM THE NEW OCR MASTER at incar 1. Node Number 2
2010-03-03 17:31:37.134: [ OCRASM][20]proprasmo: ASM cache size is [5MB]
2010-03-03 17:31:37.142: [ OCRASM][20]proprasmo: ASM cache [5MB] enabled for disk group [DR2_BIN].
2010-03-03 17:31:37.155: [ OCRRAW][20]proprioo: for disk 0 (+DR2_BIN), id match (1), total id sets, (1) need recover (0), my
votes (0), total votes (0), commit_lsn (9), lsn (9)
2010-03-03 17:31:37.155: [ OCRRAW][20]proprioo: my id set: (833490748, 1028247821, 0, 0, 0)
2010-03-03 17:31:37.155: [ OCRRAW][20]proprioo: 1st set: (833490748, 1028247821, 0, 0, 0)
2010-03-03 17:31:37.155: [ OCRRAW][20]proprioo: 2nd set: (0, 0, 0, 0, 0)
2010-03-03 17:31:37.214: [ OCRMAS][20]proath_master:18: Spawned connection mgr thread
2010-03-03 17:31:37.214: [ OCRMAS][20]proath_master:20: Spawned upgrade thread
2010-03-03 17:31:37.214: [ OCRMAS][20]th_master:19.1: Wake up upgrade thread
2010-03-03 17:31:37.216: [ OCRSRV][1]th_snap_local_spawn: Inside snap local spawn. host is [mskbkp2]
2010-03-03 17:31:37.219: [ CRSMAIN][1] Running as user: root
2010-03-03 17:31:37.219: [ CRSMAIN][1] CRSD running as the Privileged user
2010-03-03 17:31:37.219: [ CLSVER][1] Static Version 11.2.0.1.0
2010-03-03 17:31:37.226: [ OCRMAS][20]th_master:1': Recvd pubdata event from node [2]
2010-03-03 17:31:37.227: [ OCRMAS][20]th_master:2': Recvd pubdata event for self. Do nothing.
2010-03-03 17:31:37.227: [ CLSVER][1] Daemon version: 11.2.0.1.0 Software version: 11.2.0.1.0
2010-03-03 17:31:37.231: [ CLSVER][1] Active Version from OCR:11.2.0.1.0
2010-03-03 17:31:37.232: [ CLSVER][1] Active Version and Software Version are same
2010-03-03 17:31:37.232: [ CLSVER][1] Active Version changed to 11.2.0.1.0
2010-03-03 17:31:37.232: [ OCRSRV][1]th_reg_master_change: Master change callback registered
2010-03-03 17:31:37.232: [ OCRAPI][1]a_reg_master_change: Registered master change callback
2010-03-03 17:31:37.232: [ OCRSRV][1]th_not_master_change: Invoking master change callback. Master [2] Inc [1]
2010-03-03 17:31:37.232: [ OCRAPI][1]a_reg_master_change: Notified master change
2010-03-03 17:31:37.232: [ CRSMAIN][1] CAA Node Group Pri Data size: 128
2010-03-03 17:31:37.233: [ CRSMAIN][1] CAA Node Group Pub Data size: 128
2010-03-03 17:31:37.247: [ CRSMAIN][1] Getting private data of booted nodes
2010-03-03 17:31:37.247: [ CRSMAIN][1] Checking for booted param on nodenum: 2
2010-03-03 17:31:37.306: [ CLSE][1]clse_get_auth_loc: Returning default authloc: /u01/grid/11.2.0/auth/crs/mskbkp2
2010-03-03 17:31:37.306: [ CRSMAIN][1] Using Authorizer location: /u01/grid/11.2.0/auth/crs/mskbkp2
2010-03-03 17:31:37.314: [ OCRSRV][23]th_upgrade: Starting upgrade calculation
2010-03-03 17:31:37.364: [ CLSCLU][1]clsclu_init: rc 0
2010-03-03 17:31:37.381: [ OCRSRV][23]th_upgrade:10.1 AV [186646784]. State [11]. Already upgraded.Updated global data to the
crs version group. Return [0]
2010-03-03 17:31:37.385: [ CRSMAIN][1] Initializing RTI
2010-03-03 17:31:37.433: [ CRSMAIN][1] Initializing ResouceStateListener
2010-03-03 17:31:37.433: [CRSTIMER][37] Timer Thread Starting.
2010-03-03 17:31:37.433: [ CRSMAIN][1] Initializing EVMMgr
2010-03-03 17:31:37.446: [ CRSMAIN][1] Initializing ResourceMap Map
2010-03-03 17:31:37.461: [ CRSMAIN][1] Subscribing to EVM events for apps
2010-03-03 17:31:37.504: [ CRSMAIN][1] CRSD locked during state recovery, please wait.
2010-03-03 17:31:37.516: [ CRSMAIN][1] CRSD recovered, unlocked.
2010-03-03 17:31:37.525: [ default][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
2010-03-03 17:31:37.525: [ default][1]Check namebufs
2010-03-03 17:31:37.525: [ default][1]Finished checking namebufs
2010-03-03 17:31:37.526: [ GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsinet.c : 3232
], original from [clsss.c : 5026]
2010-03-03 17:31:37.569: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
2010-03-03 17:31:37.569: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
2010-03-03 17:31:37.587: [ GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 680
], original from [clsss.c : 5026]
2010-03-03 17:31:37.595: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
fatal if both fail
2010-03-03 17:31:37.595: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
f 2: file wallet (LSKP-FSW)
2010-03-03 17:31:37.596: [ GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:37.596: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
2: file wallet (LSKP-FSW) OK
2010-03-03 17:31:37.596: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
f 2: OLR wallet (LSKP-CLSW-OLR)
[ CLWAL][1]clsw_Initialize: OLR initlevel [30000]
2010-03-03 17:31:37.607: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
2: OLR wallet (LSKP-CLSW-OLR) OK
2010-03-03 17:31:37.607: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
providers - fatal if all fail)
2010-03-03 17:31:37.607: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:37.673: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:37.673: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:37.673: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:37.690: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
providers - fatal if all fail)
2010-03-03 17:31:37.690: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:37.754: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:37.754: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:37.754: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:37.755: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=0
2010-03-03 17:31:37.806: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
(0)
2010-03-03 17:31:37.817: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
s (0)
2010-03-03 17:31:37.822: [ CRSMAIN][1] CRSD listening on 10 style E2E port (ADDRESS=(PROTOCOL=tcp)(HOST=172.31.25.112)(PORT=38
983))
2010-03-03 17:31:37.835: [ CRSMAIN][1] Starting Threads
2010-03-03 17:31:37.858: [ CLSE][1]clse_get_auth_loc: Returning default authloc: /u01/grid/11.2.0/auth/crs/mskbkp2
2010-03-03 17:31:37.858: [ CRSD][1] AuthLoc /u01/grid/11.2.0/auth/crs/mskbkp2
2010-03-03 17:31:37.859: [ CRSD][1] PE active version: 11.2.0.1.0
2010-03-03 17:31:37.859: [ CRSD][1] PE Engine: NEW
2010-03-03 17:31:37.859: [ CRSD][1] Using OCR batch ops : ENABLED
2010-03-03 17:31:37.860: [ CRSMAIN][1] Initializing Node Down Monitor
2010-03-03 17:31:37.860: [ CRSMAIN][1] CRS Daemon Started.
2010-03-03 17:31:37.860: [ CRSD][1] Connecting to the CSS Daemon
2010-03-03 17:31:37.861: [ CRSD][1] Local CSS Node Number is: 2
2010-03-03 17:31:37.863: [ CRSD][1] Local Css Node Name is: mskbkp2
2010-03-03 17:31:37.863: [ CRSD][1] CRSDPersonality initialized
2010-03-03 17:31:37.864: [ CRSMAIN][1] Process member data: CRSD:mskbkp2
2010-03-03 17:31:37.864: [ CRSD][1][F-ALGO] getIpcPath returning (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
2010-03-03 17:31:37.865: [CLSFRAME][1] Inited lsf context 102b3f670
2010-03-03 17:31:37.865: [CLSFRAME][1] Initing CLS Framework messaging
2010-03-03 17:31:37.869: [ CRSD][1][F-ALGO] getIpcPath returning (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
2010-03-03 17:31:37.873: [UiServer][1] UI Comms initalize() 1
2010-03-03 17:31:37.873: [CLSFRAME][1] New Framework state: 2
2010-03-03 17:31:37.873: [CLSFRAME][1] M2M is starting...
2010-03-03 17:31:37.873: [ CRSCCL][1]clsCclInit called by process: 18669
2010-03-03 17:31:37.885: [ CRSCCL][1]USING CLSC ============
2010-03-03 17:31:37.895: [ default][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
2010-03-03 17:31:37.895: [ default][1]Check namebufs
2010-03-03 17:31:37.895: [ default][1]Finished checking namebufs
2010-03-03 17:31:37.950: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
2010-03-03 17:31:37.951: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
fatal if both fail
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
f 2: file wallet (LSKP-FSW)
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
2: file wallet (LSKP-FSW) OK
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
f 2: OLR wallet (LSKP-CLSW-OLR)
[ CLWAL][1]clsw_Initialize: OLR initlevel [70000]
2010-03-03 17:31:37.980: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
2: OLR wallet (LSKP-CLSW-OLR) OK
2010-03-03 17:31:37.980: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
providers - fatal if all fail)
2010-03-03 17:31:37.980: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:38.049: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:38.049: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:38.049: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:38.068: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
providers - fatal if all fail)
2010-03-03 17:31:38.068: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:38.134: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:38.135: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:38.135: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:38.135: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=3
2010-03-03 17:31:38.184: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
(0)
2010-03-03 17:31:38.194: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
s (0)
2010-03-03 17:31:38.200: [ CRSCCL][1]Listening endpoint created sucessfully @ (ADDRESS=(PROTOCOL=tcp)(DEV=54)(HOST=172.31.25.
112)(PORT=38984)).con = 10359a0d0
2010-03-03 17:31:38.209: [ CRSCCL][48]CSS Group Registration complete.
2010-03-03 17:31:38.213: [ CRSCCL][48]cclGetMemberData called
2010-03-03 17:31:38.215: [ CRSCCL][48]Obtained first membership map.
2010-03-03 17:31:38.215: [ CRSCCL][48]Dumping member data ------------------
2010-03-03 17:31:38.215: [ CRSCCL][48]Member (2, 603412550) on node port=.
2010-03-03 17:31:38.216: [ CRSCCL][48]Done ------------------
2010-03-03 17:31:38.216: [ CRSCCL][48]Waiting for reconfigs
2010-03-03 17:31:38.216: [ CRSCCL][49]cclCommunicationHandler started.
2010-03-03 17:31:38.220: [ CRSCOMM][1] Ipc: m_pClscCtx=1020c4850m_pUgblm=1035b2a50
2010-03-03 17:31:38.220: [ CRSCOMM][1] Ipc: Starting send thread
2010-03-03 17:31:38.220: [ CRSCOMM][1] IpcL: Listener instantiated for: (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
2010-03-03 17:31:38.221: [ CRSCOMM][52] Ipc: sendWork thread started.
2010-03-03 17:31:38.222: [ CRSCOMM][1] IpcL: Listener started listening.
2010-03-03 17:31:38.223: [ CRSCOMM][53] IpcL: thread started listening
2010-03-03 17:31:38.223: [CLSFRAME][1] Starting thread model named: AgfwProxySrvTM
2010-03-03 17:31:38.224: [CLSFRAME][1] Starting thread model named: OcrModuleTM
2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: PolicyEngineTM
2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: SharedThreadTM
2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: UiServerTM
2010-03-03 17:31:38.225: [CLSFRAME][1] New Framework state: 3
2010-03-03 17:31:38.227: [ CRSRPT][62] Enabled
2010-03-03 17:31:38.228: [ CRSPE][61] PE Role|State Update: old role [INVALID] new [INVALID]; old state [Not yet initialized
] new [Enabling: waiting for role]
2010-03-03 17:31:38.229: [ CRSSE][62] Master Change Event; New Master Node ID:2 This Node's ID:2
2010-03-03 17:31:38.230: [ CRSPE][61] PE Role|State Update: old role [INVALID] new [MASTER]; old state [Enabling: waiting fo
r role] new [Configuring]
2010-03-03 17:31:38.230: [ CRSPE][61] PE MASTER NAME: mskbkp2
2010-03-03 17:31:38.230: [ CRSPE][61] Starting to read configuration
2010-03-03 17:31:38.260: [ CRSPE][61] Reading (2) servers
2010-03-03 17:31:38.459: [ CRSPE][61] DM: set global config version to: 150
2010-03-03 17:31:38.459: [ CRSPE][61] DM: set pool freeze timeout to: 60000
2010-03-03 17:31:38.459: [ CRSPE][61] DM: Set event seq number to: 13900000
2010-03-03 17:31:38.459: [ CRSPE][61] DM: Set threshold event seq number to: 13980000
2010-03-03 17:31:38.460: [ CRSPE][61] Sent request to write event sequence number 14000000 to repository
2010-03-03 17:31:38.483: [ CRSPE][61] Wrote new event sequence to repository
2010-03-03 17:31:38.568: [ CRSPE][61] Reading (15) types
2010-03-03 17:31:38.593: [ CRSPE][61] Reading (3) server pools
2010-03-03 17:31:38.624: [ CRSPE][61] Reading (21) resources
2010-03-03 17:31:39.987: [ CRSPE][61] Finished reading configuration. Parsing...
2010-03-03 17:31:39.988: [ CRSPE][61] Parsing resource types...
2010-03-03 17:31:40.030: [ CRSD][61] Initializing the config version for type ora.asm.type to: 1
2010-03-03 17:31:40.035: [ CRSD][61] Initializing the config version for type ora.cluster_resource.type to: 1
2010-03-03 17:31:40.040: [ CRSD][61] Initializing the config version for type ora.cluster_vip.type to: 1
2010-03-03 17:31:40.044: [ CRSD][61] Initializing the config version for type ora.cluster_vip_net1.type to: 1
2010-03-03 17:31:40.048: [ CRSD][61] Dump State Starting ...
2010-03-03 17:31:40.048: [ CRSD][61] State Dump for RTILock
2010-03-03 17:31:40.048: [ CRSD][61] Lock State List is busy, skipping ..
2010-03-03 17:31:40.048: [ CRSD][61] State Dump for Timer
2010-03-03 17:31:40.049: [ CRSD][61] Timer map size=0
2010-03-03 17:31:40.049: [ CRSPE][61] Dumping PE Data Model...:DM has [0 resources][0 types][0 servers][0 spools]
------------- RESOURCES:
------------- TYPES:
------------- SERVERS:
------------- SERVER POOLS:
2010-03-03 17:31:40.049: [ CRSPE][61] Dumping ICE contents...:ICE operation count: 0
2010-03-03 17:31:40.049: [ CRSD][61] Dump State Done.
I guess that there is some thing wrong in configuration, but cannot find out what.
Any help would be appreciated.
ThanksHi,
Please check your disk attributes and permission of OCR/Voting and other ASM devices. The disk attribute should be changed to be shared among all nodes of cluster. It happened with us in 10.2.0.4 where disk was not shared and we were able to start crs from only one node at a time so please check disk attributes. Please see blog keyurmakwanacrs.blogspot.com for AIX which we faced. Not surle whether you've similar problem or not. We had 10.2.0.4 clusterware.
thanks,
Keyur -
Hello All,
We are developing a multithreaded application on
SunOS 5.8 using Forte C++ compiler.
#0 0xff350a9c in malloc_unlocked () from /usr/lib/libmalloc.so.1
#1 0xff350940 in malloc () from /usr/lib/libmalloc.so.1
#2 0xff068514 in __1cDOSLMAPSubscriberJgetValues6kMIpnHldapmod__v_ (this=0xfc0fd8f8, idx=11, out=0x28253f48)
#3 0xff060b64 in __1cDOSLNaddSubscriber6Mrn0AMAPSubscriber__n0AFError__ (this=0x139f2454, newSubs=@0xfc0fd8f8)
#4 0xff20d784 in __1cMDPMD_AdpAPR3PdoPutMbxProfile6M_v_ (this=0x139f2418)
Let me know what might be the reason for this coredump.
Thanks
VenkatTake a look at the following threads for some ideas.
http://forum.sun.com/thread.jspa?threadID=2514
http://forum.sun.com/thread.jspa?threadID=25174
http://forum.sun.com/thread.jspa?threadID=25077
If none of these help, please provide more information about your program and the compiler you are using.
Is the program multi-threaded?
Are you using the default libCstd, or the option STLport, or some other version of the C++ Standard LIbrary?
Run the command "CC-V" to find out the compiler version and patch level. -
New / malloc / threads core dump
Hi,
I'm trying to port a ftp client from Linux to Solaris. The client runs fine on both Linux and on *BSD.
I'm using the Sun Studio 8 C++ compiler on Solaris 9 4/04 x86.
The program crashes in the below class function:
-- CUT --
FILELIST CServer::ObtainFilelist(bool use_jump) {
FILELIST fl_temp, fl_temp1 = NULL, fl_new, fl_start = NULL;
pthread_mutex_lock(&(this->filelist_lock));
fl_temp = this->actual_filelist;
while (fl_temp) {
fl_new = new(FILELIST);
fl_new->next = NULL;
fl_new->magic = fl_temp->magic;
fl_new->is_marked = FALSE;
fl_new->name = new(char[strlen(fl_temp->name) + 1]);
if (!fl_temp1)
fl_start = fl_new;
else
fl_temp1->next = fl_new;
fl_temp1 = fl_new;
fl_temp = fl_temp->next;
pthread_mutex_unlock(&(this->filelist_lock));
*use_jump = this->prefs.use_jump;
return (fl_start);
-- CUT --
When running the program in dbx I get:
-- CUT --
t@2 (l@2) signal SEGV (no mapping at the fault address) in smalloc at 0xd2f20b09 0xd2f20b09: smalloc+0x0078: movl 0x00000008(%eax),%ecx
Current function is CServer::ObtainFilelist
1815 und bookfl_new->name = new(char[strlen(fl_temp->name) + 1]);
(dbx) where
current thread: t@2
[1] _smalloc(0x8), at 0xd2f20b09
[2] mallocunlocked(0x8), at 0xd2f20d66
[3] malloc(0x5), at 0xd2f20b57
[4] operator new(0x5), at 0xd2ff5d8c
=>[5] CServer::ObtainFilelist(this = 0x839d328, use_jump = 0xd2d8df33), line 1815 in "server.cc"
[6] CDisplayHandler::UpdateServerFilelist(this = 0x80c53a0, server = 0x839d328, window = 0x80eb600), line 1692 in "displayupdate.cc"
[7] CDisplayHandler::HandleServerMessage(this = 0x80c53a0, msg = 2, magic = 1, data = (nil)), line 4596 in "displayhandler.cc"
[8] CDisplayHandler::Loop(this = 0x80c53a0), line 4841 in "displayhandler.cc"
[9] DetachDisplayHandler(dummy = (nil)), line 302 in "main.cc"
[10] thrsetup(0xd2e30200), at 0xd2e54513
[11] lwpstart(), at 0xd2e54790
-- CUT --
I tried to replace the "fl_new->name = new(char[strlen(fl_temp->name) + 1]);" with "fl_new-name new char [255]" with success. fl_temp->name is accessible, I tried to output the contents before doing the new with success.
FILELIST is a struct and FILELIST.name is a char*.
Any help would be really appreciated.martin@sara# CC -V
CC: Sun C++ 5.5 2003/03/12
martin@sara# uname -a
SunOS sara 5.9 Generic_112234-12 i86pc i386 i86pc
Regards,
Martin AnderssonThe stack trace seems to show a core dump from malloc after being asked to allocate 5 bytes. (The operator new in the standard library calls malloc to allocate storage.)
If your code was compiled optimized and without -g, the stack trace is not always reliable, however.
But assuming the trace is accurate, the crash is probably due to a corrupted heap. An invalid free() or delete can corrupt the heap data structures, and the the crash occurs at at later time, often far removed from the offending location. "Invalid" means
- using the wrong form of delete
-deleting an address that was not provided by the matching new operator
-freeing an address that was not provided by malloc/calloc/realloc
-freeing or deletting the same block more than once
The heap (and anything else in the program) can also become corrupted by storing through an invalid pointer or an uninitialized pointer. "Invalid" means the object pointed to no longer exists.
The heap can also become corrupted if you store beyond the bounds of allocated storage.
The classic error is this:
char* p = new char[strlen(msg)]; // forgot about trailing null
strcpy(p, msg); // write one byte too many
If you can't see any of these errors in the souce code (and theyare hard to find), try running under dbx with RTC (run-time checking) enabled, or use some memory debugger. -
Netscape Directory Server core dumps
Hi,
I have NDS version 5.1 running on Solaris 10. My software is querying the LDAP at around 10-15 requests/sec. The LDAP memory is runnig at around 50M. (I have around 900 entries in the LDAP tree). At some random instance of time, the LDAP memory shoots up from 50M to 3.2 GB when it goes out of memory and then restarts. My software has a LDAP connection pool to it and one of the threads goes OutOfmemory after the error.
The LDAP does a core dump during the process. From the analysis of the core dump, it seems that it is trying to access some memory which is not part of the LDAP process map.
bash-3.00$ mdb core
mdb: failed to initialize /lib/libc_db.so.1: libthread_db call failed unexpectedly
mdb: warning: debugger will only be able to examine raw LWPs
Loading modules: [ libc.so.1 libuutil.so.1 ld.so.1 ]
::stacklibc_psr.so.1`memcpy+0x234(87c00038, 6b646174, 74776f6a, 8, 6b646174, 0)
libslapd.so`slapi_dup_control+0x94(53db678, 2ce38f8, fc93ff2c, 0, 2ce38f8,2ce38f8)
0x2f3b4(7e55c80, 2f30150, 2ce6bb8, 4c7fde8, 2f30150, ffffffff)
0xfef17d18(2f60164, fc940000, 0, 0, fef17c74, 1)
0xff2c8a20(0, 0, 0, 0, 0, 0)
::statusdebugging core file of ns-slapd (32-bit) from XXXXXX
initial argv:XXXXXXXX
threading model: multi-threaded using native lwps
status: process terminated by SIGSEGV (Segmentation Fault)
>
The failure is at address 0xfee8063c.
0xfee8063c::dislibc_psr.so.1`memcpy+0x20c: 0x2480012
libc_psr.so.1`memcpy+0x210: nop
libc_psr.so.1`memcpy+0x214: ldub [%i1], %o2
libc_psr.so.1`memcpy+0x218: stb %o2, [%i0]
libc_psr.so.1`memcpy+0x21c: add %i1, 1, %i1
libc_psr.so.1`memcpy+0x220: subcc %i3, 1, %i3
libc_psr.so.1`memcpy+0x224: 0x184ffffc
libc_psr.so.1`memcpy+0x228: add %i0, 1, %i0
libc_psr.so.1`memcpy+0x22c: ba +0x60 <libc_psr.so.1`memcpy+0x28c>
libc_psr.so.1`memcpy+0x230: nop
libc_psr.so.1`memcpy+0x234: ld [%i1], %o2
libc_psr.so.1`memcpy+0x238: st %o2, [%i0]
libc_psr.so.1`memcpy+0x23c: add %i1, 4, %i1
libc_psr.so.1`memcpy+0x240: subcc %i3, 4, %i3
// Looking for register i1
::regs%g0 = 0x00000000 %l0 = 0x87c00038
%g1 = 0xff37c840 libsh.so`_shi_sysAllocPool+0xd4 %l1 = 0xff16c2bc
%g2 = 0x00000000 %l2 = 0x07970778
%g3 = 0x0000036c %l3 = 0x05410000
%g4 = 0x00000001 %l4 = 0xff165a34
%g5 = 0x87c00038 %l5 = 0x00000000
%g6 = 0x00000000 %l6 = 0x01000000
%g7 = 0xfd06ca00 %l7 = 0x00000004
%o0 = 0x87c00038 %i0 = 0x87c00038
%o1 = 0x74776f72 %i1 = 0x6b646174
%o2 = 0xefe4617c %i2 = 0x74776f6a
%o3 = 0x00000002 %i3 = 0x00000008
%o4 = 0x00000002 %i4 = 0x6b646174
%o5 = 0x053d9c28 %i5 = 0x00000000
%o6 = 0xfc93fe00 %i6 = 0xfc93fe60
%o7 = 0xff0c0c44 libslapd.so`slapi_ch_malloc+0x64 %i7 = 0xff0e8e78
libslapd.so`slapi_dup_control+0x94
%psr = 0xfe401004 impl=0xf ver=0xe icc=nZvc
ec=0 ef=4096 pil=0 s=0 ps=0 et=0 cwp=0x4
%y = 0x00000002
%pc = 0xfee8063c libc_psr.so.1`memcpy+0x234
%npc = 0xfee80640 libc_psr.so.1`memcpy+0x238
%sp = 0xfc93fe00
0x6b646174::dismdb: failed to read instruction at 6b646174: no mapping for address
>
The register "i1"'s content was memory address 0x6b646174 which is not part of process address space.
Has anyone experienced similar problem before? I was trying to find the same in the forum but without any luck. This problem is happening quiet regularly.Seeems like content of register i1 got missed.. here is the content..
%l2 = 0x07970778
%l3 = 0x05410000
%l4 = 0xff165a34
%l5 = 0x00000000
%l6 = 0x01000000
%l7 = 0x00000004
%i0 = 0x87c00038
%i1 = 0x6b646174
%i2 = 0x74776f6a
%i3 = 0x00000008
%i4 = 0x6b646174
%i5 = 0x00000000
%i6 = 0xfc93fe60 -
Solaris 10, Ruby on Rails, Core Dumps
I am not very savvy with Solaris administration and we are having issues with configuring our system to run Ruby on Rails applications. We opted to use the Coolstack framework for our environment, thankfully without many hiccups. That said, our applications load but core dump during the second page load.
At this point, I'm not sure if it's our recent switch to Coolstack that is causing the problem. We've also worked to get a Mongrel cluster up and running, so that could also be the culprit.
What I'm looking for really is advice on how to pin down what the error is. I've learned about pstack to deal with the core dump file, but I'd have a better time understanding Chinese than that output. I also read about gdb and dbx, neither of which appear to be installed on our system. A strings on the file, another recommendation I found, spits out a bunch of gibberish.
How can I move forward in discovering the problem?I'm having the exact same problem with my Solaris server. I can't seem to get any form of Rails running on the system, and pages generated by Rails crash after a few page loads (not always the same number).
Did you have any luck tracking down the issue? I've asked around in a number of places and come up empty-handed every time. -
Core dump on boot during Yosemite installation
I have downloaded Yosemite from the App Store. All appears fine. When I run the installer, may MBP restarts for the installation but on rebooting it goes repeatedly into core dumps at the boot step. This can be interrupted using Cmd-R. When in Recovery Mode, resetting the Startup Disk will cure the failing reboots but then the installation has simply stopped and the computer boots into 10.9.5.
I have repeated this multiple times. I have 25 GB free disk space.
I expect that a clean install with cure this but I hate to have to do that.
TimI have the same problem.
After upgrading my MBpro from Lion to Yosemite installation finishes correctly and I can work in Yosemite.
But the next boot will result in repeated core dumps.
Restored Lion, deleted startup objects and 3rd Party drivers, tried again but the same problem.
Has the clean install helped ? -
Is there an way to get the value of a local string variable from core dump?
Hi All,
Please help me on this, How to extract the string variable content from the core file.
During core file analysis using dbx, it puts value as CLASS against string local variables.
But I need to know the exact value during the core dump.
I'm able to print other than string variables.
Is this possbile using dbx or is there any other tool can help me on this.
Thanks in advance, please find the code details below:
here is my core details:
(dbx) where -h
current thread: t@74
[1] __lwp_kill(0x0, 0x6, 0x217b8, 0x7f950b94, 0x2, 0x7f0fb944), at 0x7f99fb84
[2] raise(0x6, 0x0, 0x7f0fb958, 0x7f6a5434, 0x1, 0x7f0fb9a4), at 0x7f950b9c
[3] abort(0x7f6bf0f8, 0x4, 0x7f6bf0b8, 0x15360, 0x1, 0x7f6bf0f8), at 0x7f936d10
[4] exregister(0x7f6bf0fc, 0x4, 0x7f6bf0b8, 0x15360, 0x1, 0x7f6a5048), at 0x7f6a504c
[5] exregister(0x7f6bf0f8, 0x7f6a505c, 0x7f6bf0f8, 0x7f6bb080, 0x0, 0x1), at 0x7f6a5060
[6] exregister(0x7f0fbb28, 0x66bd8, 0xe4fd0, 0x7f6bb080, 0x7f6ba780, 0x1), at 0x7f6a4fd8
[7] 0x7f6a5edc(0x7f6bae64, 0x0, 0x66bd8, 0x14cb8, 0x14928, 0x0), at 0x7f6a5edc
=>[8] AreaManager::commitCombine(this = 0xe5234, parsedSetName = CLASS), line 369 in "AreaManager.cc"
[9] TaskDispatcher::combineParsedSet(parsedSetName = CLASS), line 113 in "TaskDispatcher.cc"
[10] CombineExecutor::run(this = 0x1312d8), line 159 in "CombineExecutor.cc"
[11] thread_body(threadObj = 0x1312d8), line 35 in "ThreadUtils.cc"
(dbx)
(dbx) dump
parsedSetName = CLASS
parsedSetCombinedName = CLASS
parserSetBuiltName = CLASS
this = 0xe5234
(dbx)
(dbx) print parsedSetName
dbx: cannot access address 0x7f6690f4
(dbx)
BR//SahooWhen I was running in a higher version, it gives follwing error:
A loadobject was found with an unexpected checksum value.
When using the same core dump binary, the result are different than the locally build binary
This is the error it pointing now:
if ((ptr = (CHAR *) malloc((unsigned int) size)) == NULL)
(dbx) proc -map
Loadobject mappings for current core file:
0x00010000 /usr/omc/1.8.0.0.56/bin/PmLoader
0x7fb80000 /usr/lib/libaio.so.1
0x7fb30000 /usr/lib/libm.so.1
0x7fbfa000 /usr/lib/libdl.so.1
is being filtered by: /usr/lib/ld.so.1
Warning: checksum in file(a5bd) doesn't match image(a5bf)
0x7faf0000 /usr/lib/libelf.so.1
Warning: checksum in file(38a7) doesn't match image(38a9)
0x7fad0000 /usr/lib/libsocket.so.1
0x7fa00000 /usr/lib/libnsl.so.1
0x7f900000 /usr/lib/libc.so.1
is being filtered by: /usr/platform/SUNW,Ultra-80/lib/libc_psr.so.1
Warning: checksum in file(3b8c) doesn't match image(3ec6)
0x7f890000 /usr/informix/lib/esql/libthsql.so
0x7f830000 /usr/informix/lib/libthasf.so
0x7f7b0000 /usr/informix/lib/esql/libthgen.so
0x7f770000 /usr/informix/lib/esql/libthos.so
0x7f700000 /usr/informix/lib/esql/libifgls.so
0x7f9e0000 /usr/informix/lib/esql/libifglx.so
0x7f6d0000 /usr/lib/libthread.so.1
0x7f500000 /usr/lib/libCstd.so.1
Warning: checksum in file(1fc4) doesn't match image(3f67)
0x7f6a0000 /usr/lib/libCrun.so.1
Warning: checksum in file(1a8d) doesn't match image(b1ea)
0x7fba0000 /usr/lib/libw.so.1
is being filtered by: libc.so.1
0x7f4e0000 /usr/lib/libmp.so.2
0x7f4a0000 /usr/lib/cpu/sparcv8plus/libCstd_isa.so.1
Warning: checksum in file(0fc3) doesn't match image(27d6)
0x7f9d0000 /usr/platform/sun4u-us3/lib/libc_psr.so.1
0x7f410000 /usr/lib/nss_files.so.1
0x7fbb0000 /usr/lib/ld.so.1 [LM_ID_LDSO]
Warning: checksum in file(f2f2) doesn't match image(2a40)
dbx ./PmLoader ./core
For information about new features see `help changes'
To remove this message, put `dbxenv suppress_startup_message 7.5' in your .dbxrc
Reading PmLoader
core file header read successfully
Reading ld.so.1
Reading libaio.so.1
Reading libm.so.1
Reading libdl.so.1
Reading libelf.so.1
Reading libsocket.so.1
Reading libnsl.so.1
Reading libc.so.1
Reading libthsql.so
Reading libthasf.so
Reading libthgen.so
Reading libthos.so
Reading libifgls.so
Reading libifglx.so
Reading libthread.so.1
Reading libCstd.so.1
Reading libCrun.so.1
Reading libw.so.1
Reading libmp.so.2
Reading libCstd_isa.so.1
Reading libc_psr.so.1
Reading nss_files.so.1
WARNING!!
A loadobject was found with an unexpected checksum value.
See `help core mismatch' for details, and run `proc -map'
to see what checksum values were expected and found.
dbx: warning: Some symbolic information might be incorrect.
t@1 (l@1) terminated by signal BUS (invalid address alignment)
0x7f947974: t_delete+0x0068: clr [%o1 + 8]
Current function is Malloc
40 if ((ptr = (CHAR *) malloc((unsigned int) size)) == NULL)
(dbx)
(dbx) where -h
current thread: t@1
[1] t_delete(0x17c980, 0x470, 0x1ef778, 0x7f9bc000, 0x0, 0x0), at 0x7f947974
[2] mallocunlocked(0x414, 0x0, 0x0, 0x7f9bc000, 0x0, 0x0), at 0x7f947004
[3] malloc(0x414, 0xebae0, 0x8, 0xffbfc9a8, 0xffbfc9a0, 0x1a3791), at 0x7f946e38
=>[4] Malloc(size = 1044U), line 40 in "Malloc.c"
[5] Drain_Socket(sc = 0x1ab100, fd = 10, block = 0xffbfce44), line 1648 in "ipc_sockio.c"
[6] Sockio_Refresh(instance = 0x1ab100, fd = 10U, readable = 1, writeable = 0, block = 0xffbfce44), line 3465 in "ipc_sockio.c"
[7] do_Refresh(cid = 10U, readable = 1, writeable = 0, block = 0xffbfce44), line 1238 in "ipc_IO.c"
[8] IO_Refresh(block = 0, file_descriptor = -1), line 685 in "ipc_IO.c"
[9] ipc_receive_message(gid = 0xffbfcf88, reply = 0xffbfd080, buffer = 0x11ae08 "", len = 0xffbfcf34, block = 0), line 313 in "ipc_services.c"
[10] IPCProcessor::receiveMessage(this = 0x1172d0, channelID = 2U, pReplyChannelID = 0xffbfd080), line 312 in "IPCProcessor.cc"
[11] IPCProcessor::mainLoop(this = 0x1172d0, pHandler = 0xf9c28), line 203 in "IPCProcessor.cc"
[12] IPCService::mainLoop(pMediator = 0x1172d0, pHandler = 0xf9c28), line 51 in "IPCService.cc"
[13] main(argc = 3, argv = 0xffbfe1f4), line 91 in "PmLoader.cc"
I got another instance of core dump in same m/c
Edited by: chittusun on Nov 5, 2008 5:19 AM -
Investigation on JVM crash without core dump (by JIT compiler)
Hi, All
I posted "JVM crash without core dump due to CompilerThread1" couple months ago.
http://forum.java.sun.com/thread.jspa?threadID=5253434
I would like show what we found, and ask couple questions
1) The reason of JVM crash is "CompilerThread0/1 wanted to allocate more native memory and eventually exceeded the limit of 32-bit Linux (on Redhat, the limit of VIRT is 3G)
2) After we lower the heap size (to reduce process size) and added process size monitor (track process size every minute) and JIT compilation log (-XX:+PrintCompilation),
We found sometimes JVM process jumped more than 800M when compiling one method. with the following log
Total time for which application threads were stopped: 0.1997400 seconds
5828 xxx.xxx.SomeClass::someMethod (1507 bytes)
5828 COMPILE SKIPPED: out of nodes during split (not retryable)
549002.449: [GC [PSYoungGen: 517897K->28390K(551296K)] 942855K->453348K(1229952K), 0.0726350 secs]
if we saw "COMPILE SKIPPED: out of nodes during split (not retryable)", the process size of JVM (VIRT/RES) always jumped 800M~1000M, and sometimes the memory get reverted in 30mins to couple hours, and sometimes it lasted forever. (so before we have larger footprint, this jump will kill JVM, right now, if the jump lasts forever, any more allocation on top of it also can kill jvm (much rare))
3) This is definitely a bug of JVM, because
it only happened on server mode, not client mode
it is random, that method can be compiled success on other JVM or next restart (we have 20 JVMs)
the method is not that complicate, like 100 lines, (bigger method gets compiled success)
this only happen on PROD environment, we can't reproduce it locally or QA (the method is always compiled success)
It mostly like under some condition (maybe node space is not enough), to compile that method will trigger JVM trying to allocate much more native memory.
4) We are going to disable this method by ".hotspot_compiler" to fix it, right now we are using lower footprint, JVM dies rarely.
+To understand more about this, I have some questions about JIT compiling+
I saw same method are compiled more than once from JIT log, like the method caused our problem, it only happens on 2nd time compilation.
Is it that JIT compiler will recompile the method with deeper optimization level some time, and more optimization it uses, the more memory it requires?
(like gcc has -O2 -O3)
Thanks!
NeoToday, one JVM crashed again, with
{Heap before gc invocations=4188:
PSYoungGen total 540864K, used 487958K [0x8ba50000, 0xb1250000, 0xb1250000)
eden space 467328K, 100% used [0x8ba50000,0xa82b0000,0xa82b0000)
from space 73536K, 28% used [0xa82b0000,0xa96d5850,0xaca80000)
to space 71680K, 0% used [0xacc50000,0xacc50000,0xb1250000)
ParOldGen total 1024000K, used 493477K [0x4d250000, 0x8ba50000, 0x8ba50000)
object space 1024000K, 48% used [0x4d250000,0x6b439610,0x8ba50000)
PSPermGen total 101760K, used 101240K [0x2d250000, 0x335b0000, 0x4d250000)
object space 101760K, 99% used [0x2d250000,0x3352e1f8,0x335b0000)
405290.711: [GC [PSYoungGen: 487958K->17397K(544192K)] 981435K->518248K(1568192K), 0.0769350 secs]
Heap after gc invocations=4188:
PSYoungGen total 544192K, used 17397K [0x8ba50000, 0xb1250000, 0xb1250000)
eden space 472512K, 0% used [0x8ba50000,0x8ba50000,0xa87c0000)
from space 71680K, 24% used [0xacc50000,0xadd4d480,0xb1250000)
to space 70208K, 0% used [0xa87c0000,0xa87c0000,0xacc50000)
ParOldGen total 1024000K, used 500851K [0x4d250000, 0x8ba50000, 0x8ba50000)
object space 1024000K, 48% used [0x4d250000,0x6bb6cf70,0x8ba50000)
PSPermGen total 101760K, used 101240K [0x2d250000, 0x335b0000, 0x4d250000)
object space 101760K, 99% used [0x2d250000,0x3352e1f8,0x335b0000)
Total time for which application threads were stopped: 0.0779620 seconds
Exception in thread "CompilerThread1" java.lang.OutOfMemoryError: requested 4522768 bytes for Chunk::new. Out of swap space? -
I am working on Solaris 7 (newest patches installed) using Sun Workshop 6. My multithreaded application runs well for a while but eventually it core dumps.
t@4 (l@1) terminated by signal SEGV (no mapping at the fault address)
The point in my code where it core dumps differs but the top of the stack always ends in
malloc(0x5fa, 0x0, 0xfea0b5c0, 0xfea0b540, 0x31, 0xff26db58)
mallocunlocked(0x0, 0x600, 0x10b9f0, 0x10bab8, 0x0, 0xff0b3f00)
Has anyone an idea?
Thanx in advance
BoerniI have a core dump, my pstack is:
----------------- lwp# 1 / thread# 1 --------------------
fe74251c realfree (3655b468, fe7c2858, fe7bc008, 1e1cc0, 36373539,
1e7f38) + 70
fe742e20 cleanfree (0, fe7bc008, fe7c27cc, fe7c284c, fe7c27e0, 0) + 58
fe741f54 mallocunlocked (1e, 0, fe7bc008, 20, 1f, 0) + f0
fe741e48 malloc (1e, 1a9f00, fe7c3a54, fe7bfc78, 339, fef53cf4) + 20
00041604 ???????? (1f0928, 48052560, ffbef450, 3c, 7400012c, 1acb78)
00042e3c ???????? (1f0918, 1ac740, 1ac718, 1ac6f0, 1ac6c8, 196c74)
00036c88 ???????? (0, 0, 0, 0, 0, 0)
----------------- lwp# 2 / thread# 2 --------------------
fe79f4b0 _signotifywait (fef1c000, ff3de79c, feee028c, 40000000,
feee1508, ff3de0c0) + 8
fef01c90 thr_yield (0, 0, 0, 0, 0, 0) + 8c
----------------- lwp# 3 --------------------------------
fe79d054 doorreturn (4, fef1c000, fef1d678, 3, fef1c000, 1) + 10
feefa380 lwpstart (fe685d98, 0, 6000, ffbeeeac, 0, 0) + 18
fef01c90 thr_yield (0, 0, 0, 0, 0, 0) + 8c
----------------- lwp# 4 --------------------------------
fef09300 private___lwp_cond_wait (fe20bd98, fef1cd74, fef1c000, 3,
fef1c000, 1) + 8
feefa380 lwpstart (fe20bd98, 0, 0, 0, 0, 0) + 18
fef01c90 thr_yield (0, 0, 0, 0, 0, 0) + 8c
-------------------------- thread# 3 --------------------
feefd9e0 reapwait (fef20988, 1e8fc, 0, fef1c000, 0, 0) + 38
feefd738 _reaper (fef1ce08, fef22710, fef20988, fef1cde0, 1, fe400000)
+ 38
fef0b11c threadstart (0, 0, 0, 0, 0, 0) + 40
Help!!!
Cesar Jorge -
Core dump/hang problem solaris 8
Hello ,
I am using the following configuration, to build my application
SunOS 5.8 Generic_108528-12 sun4u sparc
SUNW,Ultra-5_10
64 bit machine with CC 5.0
While excuting the program i get core dump/hang in cout , following is the stack looking from dbx
[1] mutexadaptive_lock(0xffffffff7c322000, 0xffffffff7c331670, 0x4c00, 0x1000, 0xfffeffff, 0x1), at 0xffffffff7c20daac
[2] cmutexlock(0x10017eca0, 0x20, 0x20, 0x10002f66c, 0x0, 0x0), at 0xffffffff7c20d870
[3] std::locale::__install(0x10, 0x10017ec90, 0x0, 0xffffffff7fffe7e0, 0x0, 0x0), at 0x10002f66c
[4] std::locale::__make_explicit(0x100047a90, 0x0, 0x0, 0x10017ec90, 0x100176570, 0x100170068), at 0x10002f514
[5] std::basic_filebuf<char,std::char_traits<char> >::overflow(0x10016f620, 0x73, 0x10016f620, 0x0, 0x0, 0x0), at 0x10004759c
[6] std::basic_filebuf<char,std::char_traits<char> >::sync(0x10016f620, 0x10016f620, 0x0, 0x0, 0x0, 0x0), at 0x1000497c8
[7] std::basic_ostream<char,std::char_traits<char> >::sentry::~sentry(0xffffffff7fffeb30, 0xffffffff7fffec50, 0x73, 0x73, 0x73, 0x0), at 0x10002c0bc
[8] std::operator<<(0x10016f7f0, 0x10016f800, 0x10016f7f0, 0x73, 0x0, 0x100063cf0), at 0x10002b70c
[9] usage(0x10016f900, 0x1001706d8, 0x1001706c0, 0x10016f930, 0x10016f910, 0x10016f888), at 0x10001d1cc
[10] main(0x1, 0xffffffff7ffff5d8, 0xffffffff7ffff5e8, 0x0, 0x0, 0x100000000), at 0x10001ad34
Any ideas regarding this,
Thanks and Regards
AmolHi,
I didnt compile with -g option, but i got to the root of problem. In the program I was just doing cout , so it was very simple, but I was linking libpthread library with the program , also was giving -mt flag during compilation. When I removed this both, the program disn't dump core and nither did it hang.
But the same thing worked with solaris 2.8 64 bit CC 5.0 with patch 108528-10 .
So I just can make out the difference as to why this this is happening
Regards
Amol
Maybe you are looking for
-
VISA file write works on 32 bit WIN 7 but not 64 bit
I've got an application that has worked for years on several XP machines. It works on WIN 7 32 bit as well. It does NOT work on WIN 7 64 bit. I download a file as a binary, break it into 4096 bit packets and send each packet via VISA write to a UUT.
-
I cannot sign a text to pdf the option is not activated
how do i sign a text into the pdf document? it's not working, the option is clickable though
-
Hi all, I have this forms declared inside a jsp. </FORM> <FORM name="temp" METHOD="GET" ACTION="basket.jsp"> <INPUT name="i" value="23" TYPE="hidden"> <INPUT name="special" value="" TYPE="hidden"> </FORM>Then inside the <% tag i want to access the va
-
Dynamic Dialog Praticipant step
Hello experts, The OOTB cq comes with a "dialog participant step" and "dynamic participant step". I am looking for "dynamic dialog participant step". Any ideas of how to implement it ? Rama.
-
Need to know BW Roles and how are they used
Hi, While activating Business content, I see some Roles( for example ( Sales Manager SAP_BW_0ROLE_007 etc) and not really sure what exactly they are. I have been able to activate content ( Cubes, queries etc) and expose them over the Enterprise por