Pstack on Solaris 9
Hello,
Is there a way to filter out the "pstack" command so that it does not print the stack output of every thread of my multithreaded application but only the one thread that crashes?
My multithreaded application runs on Solaris 9.
Thanks,
Greg
Hey there -
CSA is not supported on Solaris 9 currently.
I have seen messages from customers who receive an error message during install.
Hope this helps,
peter
Similar Messages
-
Pstack showing "??????" in place of function name
I am using solaris 5.10 kernel patch Generic_127127-11
When I do pstack on a core file it shows following output
----------------- lwp# 11 --------------------------------
001b8d88 ???????? (14c0fb90, 14c50c70, 8, fe3fb9b4, 0, 0)
00199964 ???????? (0, 148db224, 148db224, 0, c, 0)
00196b30 ???????? (ffffffff, 449d3, 7a2dc2, 36, fe3fba9c, 0)
00176eec ???????? (12a8b4a0, 1, 176e10, 0, 0, 0)
001d7210 ???????? (0, 0, 1300678, 0, 0, 0)
001da740 ???????? (12797c0, 899028, 897800, ffffc2f7, f423f, fff0bdc0)
001a7ec0 ???????? (ffffffff, 0, 0, 0, 0, 0)
001a7e20 ???????? (0, 0, 0, 0, 0, 0)
000e3c68 ???????? (117d5c0, fe3fbdb8, 0, 117bea0, fe3fbd8c, 13880)
000caab4 ???????? (117bd50, ca3a4, 471e2c, fe3fbd98, feef1818, 0)
00471ed8 ???????? (117bd50, 471ea0, 6203b0, ff000000, fec64a00, 0)
00485674 ???????? (117b0b0, 485624, 0, 89cc00, 0, fec64a00)
004855e4 ???????? (117b0b0, 906d48, 0, 0, fec64a00, 1)
Is there any specific reason for "??????" in place of function names ?
Is is possible that core file is not written completely ?
If yes, what can be done to make core file be written completely ?You are asking on a wrong forum: if you really hit pstack problem, there's no one here to help you, as pstack is Solaris utility. It does not belong to Sun Studio. It's better to post to one of OpenSolaris forums, for example, this one: http://www.opensolaris.org/jive/forum.jspa?forumID=9
However, in this case I guess I can answer your question: ??? instead of function names appear when pstack can't map address of particular PC (0x001b8d88 for example) to a symbol.
Symbols are located in two separate sections of executable/shared library: .symtab and .dynsym. The former is used to (this is very rough description) find names when linking object (*.o) files and is not needed at run time. .dynsym is used at run time to find symbols referenced by name, which are usually in the other library (like "printf" symbol, for example).
pstack can use both .symtab and .dynsym, provided that they can be found. .symtab is not present in memory image of the process from which core file is generated, so pstack needs original executable file to be available to read .symtab (you can ask the system to pull symtab from files at the time core is generated using coreadm -I defaultsymtab+ command; see coreadm(1M)). Also, many build scripts remove symbol table from final executable (by using strip(1), for example).
In terms of C++ (again, rough description), all functions end up in .symtab and only non-local functions are present both in .symtab and .dynsym. So there are fewer chances for a static function to have a name in a.out than for an extern function.
To summarize, pstack can print ??? instead of symbol name because either (or combination) of
- absence of original executable (NB: it should be at the same place; core file may contain relative path to executable!);
- stripped executable and ??? symbols are all local (static) functions;
- mismatch of some kind (core file refers to a.out, but a.out there is different so addresses does not make sense anymore);
- missing information in core file (see coreadm(1M)),
- corrupted core file (theoretically, if core was corrupted during write, it should have PF_SUNW_FAILURE flag in pflag core output, but I have no personal experience here). -
What is the problem in my Thread Dump's output?
Dear all,
Below is my Thread Dump output. I can not figure out what and where is the problem of this Thread Dump's output. Could some one please give me some hints?
Thanks alot
Tu
----------------Thread Dump's Output-----------------
Full thread dump Java HotSpot(TM) Client VM (1.5.0_07-b03 mixed mode, sharing):
"TP-Monitor" daemon prio=1 tid=0xb0b14490 nid=0xfaa in Object.wait()
[0xb08fe000..0xb08fee40]
at java.lang.Object.wait(Native Method)
- waiting on <0x89433f70> (a
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
- locked <0x89433f70> (a
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at java.lang.Thread.run(Thread.java:595)
"TP-Processor4" daemon prio=1 tid=0xb0b13730 nid=0xfa9 runnable
[0xb097e000..0xb097efc0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked <0x89438110> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:293)
at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:647)
at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:857)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
"TP-Processor3" daemon prio=1 tid=0xb0b0f6b8 nid=0xfa8 in
Object.wait() [0xb09fe000..0xb09fef40]
at java.lang.Object.wait(Native Method)
- waiting on <0x89434250> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89434250> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"TP-Processor2" daemon prio=1 tid=0xb0b0fec8 nid=0xfa7 in
Object.wait() [0xb0a7e000..0xb0a7f0c0]
at java.lang.Object.wait(Native Method)
- waiting on <0x894342e8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x894342e8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"TP-Processor1" daemon prio=1 tid=0xb0b14978 nid=0xfa6 in
Object.wait() [0xb0afe000..0xb0aff040]
at java.lang.Object.wait(Native Method)
- waiting on <0x89434380> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89434380> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Monitor" prio=1 tid=0x08609b18 nid=0xfa5 in Object.wait()
[0xb0cc9000..0xb0cc91c0]
at java.lang.Object.wait(Native Method)
- waiting on <0x89378950> (a
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
- locked <0x89378950> (a
org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor25" daemon prio=1 tid=0x08608ce0 nid=0xfa4
runnable [0xb0d48000..0xb0d49140]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked <0x89010c58> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:407)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor24" daemon prio=1 tid=0x086085f8 nid=0xfa3 in
Object.wait() [0xb0dc8000..0xb0dc8ec0]
at java.lang.Object.wait(Native Method)
- waiting on <0x89378ab8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89378ab8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor23" daemon prio=1 tid=0x08606dd0 nid=0xfa2 in
Object.wait() [0xb0e48000..0xb0e48e40]
at java.lang.Object.wait(Native Method)
- waiting on <0x89378b50> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89378b50> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor22" daemon prio=1 tid=0x08605ed0 nid=0xfa1 in
Object.wait() [0xb0ec8000..0xb0ec8fc0]
at java.lang.Object.wait(Native Method)
- waiting on <0x89378be8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89378be8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor21" daemon prio=1 tid=0x08604fd0 nid=0xfa0 in
Object.wait() [0xb0f48000..0xb0f48f40]
at java.lang.Object.wait(Native Method)
- waiting on <0x89378c80> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89378c80> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor20" daemon prio=1 tid=0x086040f0 nid=0xf9f in
Object.wait() [0xb0fc8000..0xb0fc90c0]
at java.lang.Object.wait(Native Method)
- waiting on <0x89378d18> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89378d18> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor19" daemon prio=1 tid=0x08432538 nid=0xf9e in
Object.wait() [0xb1048000..0xb1049040]
at java.lang.Object.wait(Native Method)
- waiting on <0x89378db0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89378db0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor18" daemon prio=1 tid=0x08431638 nid=0xf9d in
Object.wait() [0xb10c9000..0xb10c91c0]
at java.lang.Object.wait(Native Method)
- waiting on <0x89378e48> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89378e48> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor17" daemon prio=1 tid=0x08430738 nid=0xf9c in
Object.wait() [0xb1148000..0xb1149140]
at java.lang.Object.wait(Native Method)
- waiting on <0x89378ee0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89378ee0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor16" daemon prio=1 tid=0x0842f838 nid=0xf9b in
Object.wait() [0xb11c8000..0xb11c8ec0]
at java.lang.Object.wait(Native Method)
- waiting on <0x89378f78> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89378f78> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor15" daemon prio=1 tid=0x0842e938 nid=0xf9a in
Object.wait() [0xb1248000..0xb1248e40]
at java.lang.Object.wait(Native Method)
- waiting on <0x89379010> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89379010> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor14" daemon prio=1 tid=0x0842da38 nid=0xf99 in
Object.wait() [0xb12c8000..0xb12c8fc0]
at java.lang.Object.wait(Native Method)
- waiting on <0x893790a8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x893790a8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor13" daemon prio=1 tid=0x0842cb38 nid=0xf98 in
Object.wait() [0xb1348000..0xb1348f40]
at java.lang.Object.wait(Native Method)
- waiting on <0x89379140> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89379140> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor12" daemon prio=1 tid=0x0842bc50 nid=0xf97 in
Object.wait() [0xb13c8000..0xb13c90c0]
at java.lang.Object.wait(Native Method)
- waiting on <0x893791d8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x893791d8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor11" daemon prio=1 tid=0x084f75d0 nid=0xf96 in
Object.wait() [0xb1448000..0xb1449040]
at java.lang.Object.wait(Native Method)
- waiting on <0x89379270> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89379270> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor10" daemon prio=1 tid=0x084f66d0 nid=0xf95 in
Object.wait() [0xb14c9000..0xb14c91c0]
at java.lang.Object.wait(Native Method)
- waiting on <0x89379308> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89379308> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor9" daemon prio=1 tid=0x084f57d0 nid=0xf94 in
Object.wait() [0xb1548000..0xb1549140]
at java.lang.Object.wait(Native Method)
- waiting on <0x893793a0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x893793a0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor8" daemon prio=1 tid=0x08488970 nid=0xf93 in
Object.wait() [0xb15c8000..0xb15c8ec0]
at java.lang.Object.wait(Native Method)
- waiting on <0x89379438> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89379438> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor7" daemon prio=1 tid=0x08487a70 nid=0xf92 in
Object.wait() [0xb1648000..0xb1648e40]
at java.lang.Object.wait(Native Method)
- waiting on <0x893794d0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x893794d0> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor6" daemon prio=1 tid=0x08486b90 nid=0xf91 in
Object.wait() [0xb16c8000..0xb16c8fc0]
at java.lang.Object.wait(Native Method)
- waiting on <0x89379568> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89379568> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor5" daemon prio=1 tid=0x084e3e30 nid=0xf90 in
Object.wait() [0xb1748000..0xb1748f40]
at java.lang.Object.wait(Native Method)
- waiting on <0x89379600> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89379600> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor4" daemon prio=1 tid=0x084e2f90 nid=0xf8f in
Object.wait() [0xb17c8000..0xb17c90c0]
at java.lang.Object.wait(Native Method)
- waiting on <0x89379698> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89379698> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor3" daemon prio=1 tid=0x084e2158 nid=0xf8e in
Object.wait() [0xb1848000..0xb1849040]
at java.lang.Object.wait(Native Method)
- waiting on <0x89379730> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89379730> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor2" daemon prio=1 tid=0x085e74d0 nid=0xf8d in
Object.wait() [0xb18c9000..0xb18c91c0]
at java.lang.Object.wait(Native Method)
- waiting on <0x893797c8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x893797c8> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor1" daemon prio=1 tid=0x085e6638 nid=0xf8c in
Object.wait() [0xb1948000..0xb1949140]
at java.lang.Object.wait(Native Method)
- waiting on <0x89379860> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:474)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x89379860> (a
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:595)
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=1
tid=0x083719f8 nid=0xf8b waiting on condition [0xb1a85000..0xb1a85ec0]
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1547)
at java.lang.Thread.run(Thread.java:595)
"Low Memory Detector" daemon prio=1 tid=0x080a44f8 nid=0xf7f runnable
[0x00000000..0x00000000]
"CompilerThread0" daemon prio=1 tid=0x080a2f98 nid=0xf7e waiting on
condition [0x00000000..0xb21b2928]
"Signal Dispatcher" daemon prio=1 tid=0x080a2068 nid=0xf7d waiting on
condition [0x00000000..0x00000000]
"Finalizer" daemon prio=1 tid=0x0809c540 nid=0xf7c in Object.wait()
[0xb24b2000..0xb24b3040]
at java.lang.Object.wait(Native Method)
- waiting on <0x88f01de0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x88f01de0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=1 tid=0x0809a6b0 nid=0xf7b in
Object.wait() [0xb2533000..0xb25331c0]
at java.lang.Object.wait(Native Method)
- waiting on <0x88f01e60> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:474)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x88f01e60> (a java.lang.ref.Reference$Lock)
"main" prio=1 tid=0x0805d1c8 nid=0xf79 runnable [0xbfbd9000..0xbfbda4f8]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked <0x8946f690> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:388)
at org.apache.catalina.startup.Catalina.await(Catalina.java:615)
at org.apache.catalina.startup.Catalina.start(Catalina.java:575)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:294)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:432)
"VM Thread" prio=1 tid=0x08097b58 nid=0xf7a runnable
"VM Periodic Task Thread" prio=1 tid=0x080a5998 nid=0xf80 waiting on conditionThanks alot for your reply. It is a web application
using java and tomcat server. And the problem is that
sometime when I click on a button or a link than the
CPU goes to 100% and it hangs but normally it works
smoothly without any problem.This often indicates a busy retry loop somewhere. You need to acquire the thread dump when the problem occurs, but be warned that depending on the problem it may not be possible to obtain a Java-level thread dump. In that case you need to try and take an OS level thread dump - eg using pstack on solaris/linux (and there's some win32 tool as well if I recall correctly). -
Load a java program core in dbx
Hi Guys,
I have a core of a Java program which is writen in a mix mode means that at higher level we have Java code then for native call we have JINI calls. So we have a core of this application and we are not able to see the code written in Java. Also do we have any other debugger which can be used to debug this core. Below is the snapshot of the stack we have
bash-3.00$ /users/SUNWspro12/bin/dbx
(dbx) source pathmaps
(dbx) debug -f java core
(dbx) where
Here JINI code is visible
[15] 0xf9c0ba3c(0xf27dd058, 0xb7, 0x0, 0xc, 0xf1a20000, 0xfc67f450), at 0xf9c0ba3c
[16] 0xf9c05c64(0xf27dd058, 0xb6, 0xfc67f5bc, 0xf9c146e0, 0xc9db8, 0xfc67f4d8), at 0xf9c05c64
[17] 0xf9c05c64(0xf27dd058, 0xb6, 0xfc67f63c, 0xf9c144d0, 0xf9c0aae0, 0xfc67f558), at 0xf9c05c64
[18] 0xf9c05c64(0xfc67f8c8, 0xfc67f77c, 0xfc67f7d4, 0xf9c144d0, 0xc9db8, 0xfc67f5e0), at 0xf9c05c64
[19] 0xf9c00118(0xfc67f6c8, 0xfc67f8d0, 0xa, 0xf5c2a110, 0xf9c0aae0, 0xfc67f7dc), at 0xf9c00118
[20] JavaCalls::call_helper(0xfc67f8c8, 0xfc67f77c, 0xfc67f7d4, 0xc9db8, 0xc9db8, 0xfc67f9c0), at 0xfecc8448
[21] jni_invoke_nonstatic(0xb24ac, 0x0, 0xca380, 0xff005eb0, 0xb24a0, 0xfc67f8ac), at 0xfecdb1f8
[22] jni_CallVoidMethod(0xc9e4c, 0xfc67fa24, 0xb24a0, 0xff0037c8, 0xcaab8, 0xfc67f9c0), at 0xfed95410
[23] Java_java_lang_ref_Finalizer_invokeFinalizeMethod(0xc9e4c, 0xfc67f9a8, 0xfc67fa24, 0xf9c147f0, 0xc9db8, 0x0), at 0xfe9de928
[24] 0xf9c0ba3c(0xf27dd058, 0xb8, 0xfc67fa28, 0x4, 0xf27dc3b0, 0xfc67f9c0), at 0xf9c0ba3c
[25] 0xf9c05c64(0xf27dc3b0, 0xb7, 0xfc67fb24, 0xf9c147f0, 0xc9db8, 0xfc67fa50), at 0xf9c05c64
[26] 0xf9c05c64(0xf27dc3b0, 0xb8, 0xfc67fba4, 0xf9c14690, 0xf581ea70, 0xfc67fac8), at 0xf9c05c64
[27] 0xf9c05c64(0xfc67fe90, 0xfc67fcf8, 0xfc67fdb0, 0xf9c147f0, 0xc9db8, 0xfc67fb48), at 0xf9c05c64
[28] 0xf9c00118(0xfc67fc30, 0xfc67fe98, 0xa, 0xf5850078, 0xf9c0aae0, 0xfc67fdb8), at 0xf9c00118
[29] JavaCalls::call_helper(0xfc67fe90, 0xfc67fcf8, 0xfc67fdb0, 0xc9db8, 0xc9db8, 0xfc67fd08), at 0xfecc8448
[30] JavaCalls::call_virtual(0xfeff0000, 0xca360, 0xfc67fda4, 0xfc67fda0, 0xfc67fdb0, 0xc9db8), at 0xfecd6de4
[31] JavaCalls::call_virtual(0xfc67fe90, 0xfc67fe8c, 0xfc67fe84, 0xfc67fe7c, 0xfc67fe74, 0xc9db8), at 0xfecd6c70
[32] thread_entry(0xc9db8, 0xc9db8, 0x8cb70, 0xca360, 0x319568, 0xfecd69fc), at 0xfecd6bf8
[33] JavaThread::run(0xc9db8, 0xffffffe2, 0xff00f7d8, 0xffff8000, 0x0, 0x0), at 0xfecd6a24
[34] _start(0xc9db8, 0xfc680000, 0x0, 0x0, 0xfecc6ba8, 0x1), at 0xfecc6cdc
How to check the varible in this code frame as I am getting this error when I try to check state of some variable
(dbx) frame 15
0xf9c0ba3c: call %o0
(dbx) print _rep
dbx: warning: unknown language, 'c' assumed
dbx: "_rep" is not defined in the scope `java`(unknown)`
dbx: see `help scope' for details
Any pointer in this regard will be a great help.Java debugging requires an active agent to interface with the debugger.
Since a corefile is not active it cannot run an agent, so no Java corefile
debugging with dbx.
There exist lighter weight "agents" that 'pstack' on Solaris for example uses
but we haven't looked into using it. I recall from early experimentation
that it was supposed to help with corefiles too but while pstack
on newer Solarises can detect Java frames in a live process it still
misses them with VM coredumps. -
We have core dump of our application on Solaris 2.6.
When we investigate the core file with pstack command on Solaris 2.8 we do not see function names:
pstack corecore 'core' of 4352: FillServer -v -c 1 -s ssl=15 -n 3 -i 3 -l /db/prod/eq/gems/iris/releas
pstack: warning: librtld_db failed to initialize; symbols from shared libraries will not be available
006d7f84 ???????? (ef368c84, 824, 0, d, 0, fffffff1)
00258e30 ???????? (3133372c335e2d36, 3035363038312e34, 3739353831392c33, 5e34363632353135, 2e35343538363832, 2c335e302c335e30)
I made some test to understand situation with pstack command.
These are results
core: Solaris 6 ->Solaris 2.6(by pid -- ok):Solaris 8/host1(??? symbols):Solaris 8/host2(??? symbols)
core: Solaris 8/Moscow ->Solaris 2.6(n/a):Solaris 8/host1(ok):Solaris 8/host2(??? symbols)
core: Solaris 8/London ->Solaris 2.6(n/a):Solaris 8/host1(??? symbols):Solaris 8/host2(ok)
So the fact is that we can not see symbols even between the versions of Solaris 8.
Always when I saw ??? symbols I saw warining "pstack: warning: librtld_db failed to initialize; symbols from shared libraries will not be available"
Solaris 8/host2:
uname -a
SunOS host2 5.8 Generic_108528-07 sun4u sparc SUNW,Sun-Fire
Solaris 8/host1:
uname -a
SunOS host1 5.8 Generic_108528-13 sun4u sparc SUNW,Sun-Fire-880
Help, please.Make sure you have installed the latest recommended patches for Solaris 8 (and especially the ones around pam, native LDAP and nscd).
Regards,
Ludovic. -
Error in reading core on Solaris 2.8 using pstack command
We are unable to read core file on Solaris 2.8 using pstack command. We are getting warning that librtld_db failed to initialize; symbols from shared libraries will not be available.
Man page of pstack says that such error can occur if a core file from one operating system release is examined on a different operating system release.
But we are trying to read the core on the same machine on which it occurred.
Does any one know what could be reason and what are the alternative to read core in such situation?Dear Gopinath,
Thanks for the response.
The problem occurred again.
In fact now it is repeatable.
We have not installed any Solaris patches after original OS installation. We have checked the core using pstack immediately after it was dumped.
We have tried reading it using dbx but it is also giving same error.
Here is the output of dbx
dbx: could not initialize rtld_db.so -- verify that corefile was generated on a compatible machine
Note: We compile our program on Solaris 2.6 as 32 bit and then install it on Solatis 2.8.
We have tried reading the core using dbx on both Solaris 2.6 as well as Solaris 2.8.
We know that this must be problem in our program (some memory violation) and has nothing to do with 2.8 or 2.6.
But inability to analyze core is happering the debugging.
Thanks
Dhananjay -
Problem with JNI in a Solaris shared library
Hi,
I have a problem.
I have used JNI in my .dll on Windows & .so on Solaris . I have also used the same JNI code in a .exe on Windows & an executable on Solaris. 3 out of the above 4 combinations work & 1 doesen't. When I use JNI from a .so on Solaris 8(with/without the J2SE recommended patch), the JVM seems to have some exception & dumps core when I invoke CreateJavaVM(). See Below - I've included the output on the screen & the output from 'pstack core' :
Unexpected Signal : 11 occurred at PC=0xFED29D44
Function=[Unknown. Nearest: JVM_GetMethodIxExceptionIndexes+0x1994]
Library=/usr/java/jre/lib/sparc/libjvm.so
Current Java thread:
Dynamic libraries:
0x10000 SblSim
0xff280000 /usr/lib/libnsl.so.1
0xfec00000 /usr/java/jre/lib/sparc/libjvm.so
0xff370000 /usr/lib/libpthread.so.1
0xff350000 /usr/lib/librt.so.1
0xff260000 /usr/lib/libsocket.so.1
0xff390000 /usr/lib/libdl.so.1
0xff230000 /usr/lib/libCrun.so.1
0xff200000 /usr/lib/libm.so.1
0xff340000 /usr/lib/libw.so.1
0xff1b0000 /usr/lib/libthread.so.1
0xff080000 /usr/lib/libc.so.1
0xff180000 /usr/lib/libmp.so.2
0xff160000 /usr/lib/libaio.so.1
0xff1f0000 /usr/platform/SUNW,Sun-Blade-1000/lib/libc_psr.so.1
0xfebd0000 /usr/j2se/jre/lib/sparc/native_threads/libhpi.so
0xfeba0000 /usr/j2se/jre/lib/sparc/libverify.so
0xfeb60000 /usr/j2se/jre/lib/sparc/libjava.so
0xfeb40000 /usr/j2se/jre/lib/sparc/libzip.so
0xfe920000 /usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.2
0xf1400000 ./libaicd.so
0xfc480000 ./libmttoolkit.so
0xfdfe0000 ./libmttlogger.so
0xfdfb0000 /usr/j2se/jre/lib/sparc/libnet.so
Local Time = Tue Nov 12 09:51:37 2002
Elapsed Time = 29
# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002D5 01
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
# Java VM: Java HotSpot(TM) Client VM (1.4.0_01-b03 mixed mode)
# An error report file has been saved as hs_err_pid3120.log.
# Please refer to the file for further information.
Abort(coredump)
Below is the output of 'pstack core'
$ pstack core
core 'core' of 3120: SblSim libaicd.so 127.0.0.1 9999
----------------- lwp# 1 / thread# 1 --------------------
ff1c9764 __sigprocmask (ff1cbf60, 0, 0, 565f8, ff1de000, 0) + 8
ff1be110 _sigon (565f8, ff1e5930, 6, ffbedba4, 565f8, 6) + d0
ff1c1150 thrpkill (0, 1, 6, ff1de000, 1, ff140450) + f8
ff0cb900 raise (6, 0, 0, ffffffff, ff1403bc, 4) + 40
ff0b58ec abort (ff13c000, ffbedcf8, 0, fffffff8, 4, ffbedd19) + 100
feee2700 ???????? (1, fef98e50, ffbedd98, 0, feffc130, feee1258)
feee12c8 ???????? (60650, b, fed29d44, ffbeeab8, b, 0)
fede77c0 ???????? (fed29d44, ffbeeab8, ffbee800, fed29d44, fed03744, 0)
ff1cb824 __sighndlr (b, ffbeeab8, ffbee800, fede6f38, 5669c, 5668c) + c
ff1c84d8 sigacthandler (b, 565f8, 0, 0, 0, ff1de000) + 708
--- called from signal handler with signal 11 (SIGSEGV) ---
fed29d44 ???????? (0, 60650, 0, 0, 0, 0)
fed2d7d4 ???????? (606e4, 0, 0, ffbeed00, 0, 0)
f172e818 __1cHJNIEnv_JNewObject6MpnH_jclass_pnK_jmethodID_E_pnI_jobject__ (606e4, 0, 0, ffbeed70, 39, 107f90) + 70
f172dbf8 __1cGJEvent2t5B6M_v_ (107f60, f1a877f4, db088, ff0c1c44, 107f48, 0) + 80
f16f2864 __1cGCEvent2t6Mii_v_ (107f60, 0, 1, ff13c000, 107f48, 0) + 7c
f16e62e8 __1cVCISCHelperThreadMutex2t6M_v_ (107f50, 0, 560ac, 30f68, fefe4000, 0) + c0
f16ddc34 __1cLinitGlobals6F_v_ (0, 1, 19a, f1a198ec, f1a198ec, 0) + 5c
f16de4e4 CreateISCDriverInstance (0, 5cc40, 0, 57608, 565e8, 0) + 554
00034f08 __1cJRunSblSim6F_v_ (1, ffbef322, ffbef32c, ff13fc70, 100d4, 0) + 320
00035588 main (4, ffbef1dc, ffbef1f0, 55c00, 0, 0) + 1a8
00018460 _start (0, 0, 0, 0, 0, 0) + 108
----------------- lwp# 2 / thread# 2 --------------------
ff11e958 _signotifywait (ff1de000, 59, 0, 0, 0, 0) + 8
ff1c2030 thr_yield (0, 0, 0, 0, 0, 0) + 8c
----------------- lwp# 3 --------------------------------
ff1c9774 private___lwp_cond_wait (ff075d70, ff1ded9c, ff1de000, 0, 0, 4) + 8
ff11c554 doorreturn (ff075cb0, ff1ba740, 0, 0, 0, 0) + 68
----------------- lwp# 4 / thread# 4 --------------------
ff11efbc ___lwp_cond_wait (60048, 60030, fe901ad0, 0, 0, fe901ac8) + 8
ff116610 lwpcond_timedwait (1cb8c88, 3dd1157a, fe901ba4, 60030, 60048, 1) + 98
fecfc7d8 ???????? (0, 1, 0, 581b8, feffa26c, 0)
fecfcfd4 ???????? (ff00a15c, feffa4b8, feffa4b4, 0, 0, 0)
fecfccc0 ???????? (d3980, 0, 0, 0, 0, 0)
fecfcbd0 ???????? (d3980, ff1df688, 1, 1, ff1de000, 0)
ff1cb728 threadstart (d3980, 0, 0, 0, 0, 0) + 40
----------------- lwp# 5 --------------------------------
ff11c4fc doorreturn (4, ff1df688, ff1df6a0, 3, ff1de000, 1) + 10
ff1ba740 lwpstart (fea95d70, 0, 6000, ff075b74, 0, 0) + 18
ff1c2030 thr_yield (0, 0, 0, 0, 0, 0) + 8c
----------------- lwp# 6 / thread# 5 --------------------
ff11efbc ___lwp_cond_wait (d5720, d5708, 0, fed10a40, 0, 0) + 8
fed110d4 ???????? (d56c8, 0, 5000, 5210, d4d38, 4d24)
fed10824 ???????? (fdf8172c, 0, 0, d4d38, d4d38, 30dec8)
fed103b0 ???????? (d4dcc, fdf81824, 0, 0, 0, 0)
fa40cc94 ???????? (f2000380, b6, fdf818a4, ffffffff, 0, fdf817c0)
fa405da0 ???????? (f2000380, b6, fdf818a8, fa415444, 0, fdf81840)
fa405da0 ???????? (fdf81938, 0, 0, fa4157a4, 2f4210, fdf818d8)
fa400438 ???????? (fdf819c0, fdf81c08, a, f6050b68, fa40bc8c, fdf81b50)
fecfe2f8 ???????? (fdf81c00, fdf81a88, fdf81b48, d4d38, d4d38, fdf81a9c)
fed0ff90 ???????? (fefe4000, d5298, fdf81b3c, fdf81b38, fdf81b48, d4d38)
fed0fdf0 ???????? (fdf81c00, fdf81bfc, fdf81bf0, fdf81be8, fdf81be0, d4d38)
fed0fd78 ???????? (d4d38, d4d38, ac9a0, d5298, 2d4488, fed0fa60)
fed0fa88 ???????? (d4d38, 0, 0, 0, 0, 0)
fecfcbd0 ???????? (d4d38, ff1df688, 1, 1, ff1de000, 0)
ff1cb728 threadstart (d4d38, 0, 0, 0, 0, 0) + 40
----------------- lwp# 7 / thread# 6 --------------------
ff11efbc ___lwp_cond_wait (d7040, d7028, 0, 1, 0, 0) + 8
fed110d4 ???????? (d6fe8, 0, 5000, 5210, d6658, 4d24)
fed10824 ???????? (fa38169c, 0, 0, d6658, d6658, 30dec8)
fed103b0 ???????? (d66ec, fa38179c, 0, 0, 0, 0)
fa40cc94 ???????? (f2000490, b6, fa3817a0, fa414050, 0, fa381730)
fa405da0 ???????? (f2000478, b6, fa3818b4, fa415444, 0, fa3817d8)
fa405c4c ???????? (f2000478, b6, fa381934, fa415444, d6658, fa381858)
fa405c4c ???????? (fa381938, 0, 0, fa38197c, 2f4210, fa3818d8)
fa400438 ???????? (fa3819c0, fa381c08, a, f60520b8, fa40bc8c, fa381b50)
fecfe2f8 ???????? (fa381c00, fa381a88, fa381b48, d6658, d6658, fa381a9c)
fed0ff90 ???????? (fefe4000, d6bb8, fa381b3c, fa381b38, fa381b48, d6658)
fed0fdf0 ???????? (fa381c00, fa381bfc, fa381bf0, fa381be8, fa381be0, d6658)
fed0fd78 ???????? (d6658, d6658, ac9e0, d6bb8, 2d4488, fed0fa60)
fed0fa88 ???????? (d6658, 0, 0, 0, 0, 0)
fecfcbd0 ???????? (d6658, ff1df688, 1, 1, ff1de000, 0)
ff1cb728 threadstart (d6658, 0, 0, 0, 0, 0) + 40
----------------- lwp# 8 / thread# 7 --------------------
ff11d194 _poll (0, 0, 0, fa281a28, ff13f1b4, fa281a28) + 8
ff1cb134 select (0, a, 0, ff1de000, fa281e14, 0) + 34
fed32afc ???????? (d8040, 0, a, 0, 0, 0)
fed328b4 ???????? (d8040, 0, 0, 0, 0, 0)
fecfcbd0 ???????? (d8040, ff1df688, 1, 1, ff1de000, 0)
ff1cb728 threadstart (d8040, 0, 0, 0, 0, 0) + 40
----------------- lwp# 9 / thread# 8 --------------------
ff11efbc ___lwp_cond_wait (5f998, 5f980, 2e7aa0, ff1de000, fa181e14, 0) + 8
fecfc634 ???????? (5f950, 0, 0, 0, feffa26c, 0)
fed3371c ???????? (3c00, 3f44, 0, 0, 0, 0)
fecfcbd0 ???????? (d81a8, ff1df688, 1, 1, ff1de000, 0)
ff1cb728 threadstart (d81a8, 0, 0, 0, 0, 0) + 40
----------------- lwp# 10 / thread# 9 --------------------
ff11f008 lwpsema_wait (f1f81e30, ff1de000, 0, f1f81d70, ff3e2628, 0) + c
ff1b94c0 _swtch (f1f81d70, ffffffff, ff1de000, 5, 1000, 0) + 158
ff1bcc64 sema_wait (ff020fe0, ff1e98e4, 5000, 10000, 534d, 1) + bc
fed345c8 ???????? (0, 6, b, a, 1, 0)
fed34254 ???????? (ff002ae4, ff00a5a0, ff00a5d4, dab90, 2d4488, fed0fa60)
fed0fa88 ???????? (da630, 0, 0, 0, 0, 0)
fecfcbd0 ???????? (da630, ff1df688, 1, 1, ff1de000, 0)
ff1cb728 threadstart (da630, 0, 0, 0, 0, 0) + 40
----------------- lwp# 11 / thread# 10 --------------------
ff11efbc ___lwp_cond_wait (604e8, 604d0, 2e7aa0, feffa26c, 106c40, 0) + 8
fecfc88c ???????? (604a0, 0, 0, dbbd0, feffa26c, 0)
fed349ac ???????? (db670, db670, dbfd0, dbbd0, 2d4488, fed0fa60)
fed0fa88 ???????? (db670, 0, 0, 0, 0, 0)
fecfcbd0 ???????? (db670, ff1df688, 1, 1, ff1de000, 0)
ff1cb728 threadstart (db670, 0, 0, 0, 0, 0) + 40
-------------------------- thread# 3 --------------------
ff1bddbc reapwait (ff1e29e0, 20520, 0, ff1de000, 0, 0) + 38
ff1bdb14 _reaper (ff1dee30, ff1e4740, ff1e29e0, ff1dee08, 1, fe400000) + 38
ff1cb728 threadstart (0, 0, 0, 0, 0, 0) + 40This is an update to the original message -
I have tried both HotSpot & Server JVMs but still see the
same behavior. I'm using 1.4 .
I'd like to provide more information to those who can help
me. I have a feeling there might be something wrong in my
CC options. Thus you will find below what my CC statement
looks like for one of the files in my project & for the
linking of the final shared library. I also have
provided the output of ldd on my shared library libaicd.so
& for comparison the output of ldd on libjvm.so as that
might provide some usefull information just incase
something looks suspecious & might ring a bell.
/opt/SUNWspro.cc6/bin/CC -c -KPIC -g -DDEBUG=DEBUG -
D_DEBUG -mt -DSOLARIS -DSTD_STREAM -
I/vob1/qwsrc/fw/include -I.-I/vob4/source/idl -
I/vob4/source/mttoolkit -I/vob4/source/mttoolkit/tools -
I/vob4/source/integrations/siebel/aicd -
I/vob4/source/include -I/vob1/qwsrc/fw/include -
I/vob4/source/source -I/vob4/source/mttoolkit -
I/vob1/external/platform_neutral/siebel/scapi/include -
I/external/sparc/Sun/jdk/1.3.1/include -
I/external/sparc/Sun/jdk/1.3.1/include/solaris -
I/vob4/source/idl -I/vob4/source/mttoolkit -
I/vob4/source/mttoolkit/tools -
I/vob4/source/mttoolkit/vnet -
I/vob4/source/mttoolkit/bool -DUNIX -
D_USE_MTTServerToolkit -D_JAVA_ WIAcceptTimeOut.cpp -
o /space/users/janak/buildoutput/sparc/debug/obj/integratio s/siebel/aicd/WIAcceptTimeOut.o
/opt/SUNWspro.cc6/bin/CC -G -
o /space/users/janak/buildoutput/sparc/debug/lib/fw/libaicd.so
/space/us
ers/janak/buildoutput/sparc/debug/obj/integrations/siebel/ai
cd/AgentService.o /space/users/janak/buildoutput/sparc/debug
/obj/integrations/siebel/aicd/AgentServiceCmds.o /space/user
s/janak/buildoutput/sparc/debug/obj/integrations/siebel/aicd
/AgentServiceEvents.o /space/users/janak/buildoutput/sparc/d
ebug/obj/integrations/siebel/aicd/AgentServiceICAReq.o /spac
e/users/janak/buildoutput/sparc/debug/obj/integrations/siebe
l/aicd/AgentServiceICEvents.o /space/users/janak/buildoutput
/sparc/debug/obj/integrations/siebel/aicd/AgentServiceUtilit
ies.o /space/users/janak/buildoutput/sparc/debug/obj/integra
tions/siebel/aicd/AvMc.o /space/users/janak/buildoutput/spar
c/debug/obj/integrations/siebel/aicd/InternationalStrings.o
/space/users/janak/buildoutput/sparc/debug/obj/integrations/
siebel/aicd/JEvent.o /space/users/janak/buildoutput/sparc/de
bug/obj/integrations/siebel/aicd/JMultipleEvents.o /space/us
ers/janak/buildoutput/sparc/debug/obj/integrations/siebel/ai
cd/JNIObject.o /space/users/janak/buildoutput/sparc/debug/ob
j/integrations/siebel/aicd/JObject.o /space/users/janak/buil
doutput/sparc/debug/obj/integrations/siebel/aicd/JSemaphore.
o /space/users/janak/buildoutput/sparc/debug/obj/integration
s/siebel/aicd/JWorkerThread.o /space/users/janak/buildoutput
/sparc/debug/obj/integrations/siebel/aicd/MultipleEvents.o /
space/users/janak/buildoutput/sparc/debug/obj/integrations/s
iebel/aicd/Param.o /space/users/janak/buildoutput/sparc/debu
g/obj/integrations/siebel/aicd/ParamQueue.o /space/users/jan
ak/buildoutput/sparc/debug/obj/integrations/siebel/aicd/RefC
ount.o /space/users/janak/buildoutput/sparc/debug/obj/integr
ations/siebel/aicd/SiebelDriver.o /space/users/janak/buildou
tput/sparc/debug/obj/integrations/siebel/aicd/SiebelEvent.o
/space/users/janak/buildoutput/sparc/debug/obj/integrations/
siebel/aicd/SiebelWorkItem.o /space/users/janak/buildoutput/
sparc/debug/obj/integrations/siebel/aicd/SingleEvent.o /spac
e/users/janak/buildoutput/sparc/debug/obj/integrations/siebe
l/aicd/SmartPtr.o /space/users/janak/buildoutput/sparc/debug
/obj/integrations/siebel/aicd/StdAfx.o /space/users/janak/bu
ildoutput/sparc/debug/obj/integrations/siebel/aicd/ThreadPoo
l.o /space/users/janak/buildoutput/sparc/debug/obj/integrati
ons/siebel/aicd/Utilities.o /space/users/janak/buildoutput/s
parc/debug/obj/integrations/siebel/aicd/WIAcceptTimeOut.o -
lnsl -lc -lCstd -
L/space/users/janak/buildoutput/sparc/debug/lib/cti -
lmttoolkit -
L/space/users/janak/buildoutput/sparc/debug/lib/fw -
lmttlogger -
L/space/users/janak/buildoutput/sparc/debug/lib/fw -
lqwlicmtt -
L/external/sparc/Sun/jdk/1.3.1/jre/lib/sparc/server -
Bdynamic -ljvm -mt -lpthread -lposix4
$ ldd /usr/j2se/jre/lib/sparc/libjvm.so
libCrun.so.1 => /usr/lib/libCrun.so.1
libdl.so.1 => /usr/lib/libdl.so.1
libthread.so.1 => /usr/lib/libthread.so.1
libsocket.so.1 => /usr/lib/libsocket.so.1
libnsl.so.1 => /usr/lib/libnsl.so.1
libm.so.1 => /usr/lib/libm.so.1
libw.so.1 => /usr/lib/libw.so.1
libc.so.1 => /usr/lib/libc.so.1
libmp.so.2 => /usr/lib/libmp.so.2
/usr/platform/SUNW,Sun-Blade-1000/lib/libc_psr.so.1
$ ldd libaicd.so
libnsl.so.1 => /usr/lib/libnsl.so.1
libc.so.1 => /usr/lib/libc.so.1
libmttoolkit.so => ./libmttoolkit.so
libmttlogger.so => ./libmttlogger.so
libjvm.so => /usr/java/jre/lib/sparc/libjvm.so
libpthread.so.1 => /usr/lib/libpthread.so.1
librt.so.1 => /usr/lib/librt.so.1
libdl.so.1 => /usr/lib/libdl.so.1
libmp.so.2 => /usr/lib/libmp.so.2
libCrun.so.1 => /usr/lib/libCrun.so.1
libthread.so.1 => /usr/lib/libthread.so.1
libsocket.so.1 => /usr/lib/libsocket.so.1
libm.so.1 => /usr/lib/libm.so.1
libaio.so.1 => /usr/lib/libaio.so.1
libw.so.1 => /usr/lib/libw.so.1
/usr/platform/SUNW,Sun-Blade-1000/lib/libc_psr.so.1 -
How to understand the strange parameter values shown by 'pstack core'
I encounterred a problem that led to a core dump. I use 'pstack core' to shown the calling stack.
However I could hardly understand some of the parameter values below:
core 'core.150108' of 5016: cmcfun
ff1e1370 __1cLComEndPointFsendv6MpnFiovec_i_i_ (c, ffbfedf8, 2, 2, c, 0) + 4
Why did the 'this' pointer of the instance of class ComEndPoint become 0xc? I don't think it is a valid value of pointer. Does anybody have any idea about this problem?
ff1e5b00 __1cYComTransaction_Icallback6FpnPpdo_transaction_pcii_v_ (6ab90, 0, 0, 0, ffbff1b8, 0) + dc
According to the source code, the 3rd parameter in the function below, which value is 0x66, was not changed in the scope of the function. However it became 0x0 when it was used as the 4th parameter to call the function above. What't the matter?
ff1528e8 pdo_dispatch (6ab90, ad, 66, 0, 0, 0) + dc
I heard that Sun C++ uses 'register windows' to pass the parameters, i0~i5 are used for the first 6 ones. Somebody told me that what I saw in the pstack might not be creditable because the registers might be changed within the function. Is that so? Would anybody plz tell me more details?
Thanks in advance.
Edited by: DirtyBear on Feb 15, 2008 1:17 AMRegister windows are a property of the SPARC computer architecture, not a property of a compiler. SPARC has multiple sets of registers. The function call instruction advances the "register window" to reveal the next set of registers, if there is one. If not, the registers are spilled to the stack to free up a window. The function return instruction moves the window back. (It's a purely hardware feature.) Eight of the registers in adjacent windows overlap, the origin of the "window" term. Registers %o0 thorugh %o7 in the caller become registers %i0 through %i7 in the called function.
On SPARC and x64, some function arguments are passed in registers, depending on how many arguments there are and their types. (The x86 architecture has no registers available for passing arguments.) On entry to a function, the parameters passed in registers might be saved on the stack, but do not necessarily need to be saved on SPARC. The Solaris ABI requires that space for the registers be allocated, but the space is not necessarily used.
In a core dump, the actual registers, particularly for functions other than the current one, are not available. The pstack operation doesn't know how many parameters a function has or what type they are. It shows the contents of the first 6 words on the stack that would hold arguments, if there were arguments, and if they were actually saved on the stack.
In optimized code on any platform, the stack area for a variable or argument is not kept up to date. Computations are kept in registers to the extent possible, and saved in memory only when necessary.
On SPARC in particular, the argument values shown in a stack trace need not be accurate. -
How to fix aSun Studio 9 bug running on a Solaris 10 ?
Hi all
I have Sun Studio 9 installed on a Sun Solaris 10 and when I try to run the debug tools I receive this Issue
(dbx) cd /users/****/ISMD_V10_integration/MMI/IHM/SCHEMA/bin
(dbx) debug /users/***/ISMD_V10_integration/MMI/IHM/SCHEMA/bin/dbtool.Solaris
Reading dbtool.Solaris
Reading ld.so.1
dbx: internal error: signal SIGSEGV (no mapping at the fault address)
dbx's coredump will appear in /tmpSo I tried to install this Patch [1186876|http://sunsolve.sun.com/search/document.do?assetkey=1-21-118676] .
This is what I received
# patchadd 118676-02
Validating patches...
Loading patches installed on the system...
Done!
Loading patches requested to install.
Cannot open pkginfo file /users/spiteri/118676-02/SUNWxcu4t/pkginfo
Done!
Checking patches that you specified for installation.
Done!
The following requested patches will not be installed because
they have been made obsolete by other patches already
installed on the system or by patches you have specified for installation.
0 All packages from patch 118676-02 are patched by higher revision patches.
No patches to install.More over when I checked if the patch exist I have this
# showrev -p | grep 118676
Patch: 118676-03 Obsoletes: Requires: Incompatibles: Packages: SUNWsprot, SUNWxcu4tThen I checked the DBX version
# dbx
For information about new features see `help changes'
To remove this message, put `dbxenv suppress_startup_message 7.3' in your .dbxrc
(dbx) version
Machine hardware: sun4u
OS version: 5.10
Processor type: sparc
Hardware: SUNW,Sun-Fire-280R
The following components are installed on your system:
Sun Studio 9
Sun Studio 9 C Compiler
Sun Studio 9 C++ Compiler
Sun Studio 9 Tools.h++ 7.1
Sun Studio 9 C++ Standard 64-bit Class Library
Sun Studio 9 Garbage Collector
Sun Studio 9 Fortran 95
Sun Studio 9 Debugging Tools (including dbx)
Sun Studio 9 Debugger GUI
Sun Studio 9 Performance Tools (including collect, ...)
Sun Studio 9 X-Designer
Sun Studio 9 VIM editor
Sun Studio 9 XEmacs editor
Sun Studio 9 Native Connector Tool
Sun Studio 9 LockLint
Sun Studio 9 Building Software (including dmake)
Sun Studio 9 Documentation Set
version of "/opt/SUNWspro/bin/../prod/bin/../../bin/cc": Sun C 5.6 2004/07/15
version of "/opt/SUNWspro/bin/../prod/bin/../../bin/CC": Sun C++ 5.6 2004/07/15
version of "/opt/SUNWspro/bin/../prod/bin/../../bin/f90": Sun Fortran 95 8.0 2004/07/15
version of "/opt/SUNWspro/bin/../prod/bin/../../bin/dbx": Sun Dbx Debugger 7.3 2004/07/15
version of "/opt/SUNWspro/bin/../prod/bin/../../bin/analyzer": Sun Performance Analyzer 7.3 2004/07/15
version of "/opt/SUNWspro/bin/../prod/bin/../../bin/dmake": Sun Distributed Make 7.5 2004/07/15And to conclude I tried this
pstack core | c++filt
core 'core' of 15779: /opt/SUNWspro/bin/../prod/bin/sparcv9/dbx -g sync,stdio -c dbxenv supp
ffffffff7e8d3894 _lwp_kill (6, 0, ffffffff7e8b5dd4, ffffffffffffffff, ffffffff7e9ec000, 0) + 8
ffffffff7e84a5ec abort (1, 1b8, 0, 1a1b20, 0, 0) + 118
00000001000b4204 ???????? (1005fd, 100400, a, 100400, 10060b000, 100512e30)
ffffffff7e8d2798 __sighndlr (a, ffffffff7fffb7d0, ffffffff7fffb4f0, 1000b3f30, 0, 9) + c
ffffffff7e8c6a04 call_user_handler (ffffffff7e700200, ffffffff7e700200, ffffffff7fffb4f0, 8, 0, 0) + 3e0
ffffffff7db19b30 Pgetauxval (1012eae48, 7e3, ffffffff7f736c60, ffffffff7f61c5bc, 6f723c636861723e, 7e3) + 38
ffffffff7db0b12c Pbrandname (1012eae48, ffffffff7fffbfc0, 400, 0, ffffffff7f736c60, 821) + 8
ffffffff7db185a0 ps_pbrandname (1012eae48, ffffffff7fffbfc0, 400, 1001afee0, ffffffff7d806000, 1) + c
ffffffff7d701bfc _rd_reset32 (7e0, ffffffff7d806000, ffffffff7d806000, 100691860, ffffffff7d806000, 1) + 488
ffffffff7d700df0 rd_reset (1011f0540, 0, 1052ac, ffffffff7e8c88f4, 1, 0) + a8
ffffffff7d700eb8 rd_new (1012eae48, ffffffff7cf00160, 1025a6b40, 0, ffffffff7d806000, ffffffff7d806000) + a8
00000001003146e0 void RtldAgent::open(Pcs*,const ps_prochandle*) (101789570, 10062a180, 1012eae48, 100558, 1, ffffffff7d700e10) + 68
00000001001acbec ???????? (1012aec40, 0, 100691cf0, 10053dd33, 10053d000, ffffffff7fffc6e0)
00000001001addd4 Proc*ProcMgr::start(Target*,bool,bool,bool) (100512e30, 0, 100627a50, 0, 1012aec40, 10053e18e) + 92c
0000000100171bb8 int do_run(Interp*,bool,Level_e) (100400, 10053b000, 100400, 100512e30, 1, 100400) + 19c
00000001002abf28 ???????? (10056b477, 1025a7890, 10062b638, 0, 100642750, 10068ec70)
00000001002aac18 int pdksh_execute(Interp*,op*,int) (10062b5f0, 100643740, 0, 10062b5f0, 103084180, 101339a10) + 1c0
000000010029746c int pdksh_shell(Interp*,Source*) (10062b5f0, 102703bd0, 3, 100569df9, 100569e00, 1000b3c10) + 4c0
0000000100296eac int pdksh_command(Interp*,const char*) (ffffffffffffffff, 0, 1, 1, 10062b5f0, 10066bb40) + 88
00000001001523e0 void MyServantDbx::ksh_cmd(int,unsigned,const char*) (1006604e0, 0, 0, 1025a7714, 10052a, 10060c3f0) + 98
00000001002d82e4 bool Dispatcher::dispatch(ProtoReceiver*,MsgRcv*) (1002c1b4c, 1006604e0, ffffffff7fffd278, 1002c1b4c, 100000, 0) + 100
00000001002d4968 void Messenger::handle_message_help(AuthStyle,MsgRcv&) (1006604e0, 0, ffffffff7fffd278, 10056f358, 10056f000, 1002c8498) + a4
00000001002d46dc void Messenger::handle_message(AuthStyle) (1006604e0, 0, 0, ffffffff7fffd278, 10052b0d0, 100000) + dc
00000001002c87ac void Servant::cb_message(int,unsigned short,void*) (b, 10052b0d0, 1006604e0, 1003e3000, 1003e3, 100000) + 8c
00000001002c3ed4 void CBInfo::dispatch(int,unsigned short) (100625bc0, 10056f, 1003e1687, b, 100400, 1) + ac
00000001002c4304 void Notifier::invoke(int,unsigned short,CbData*) (10065f0b0, b, 1, 100625bc0, 1002c8721, 1002c3e28) + 18
00000001002c4b88 void NotifierDirect::dispatch_help(int,unsigned short,CBInfo*,bool*) (10065f0b0, b, 100400, 100625bc0, ffffffff7fffda2f, 1000000000000) + c4
00000001002c51ac bool NotifierPoll::dispatch(bool*) (10065f0b0, ffffffff7fffda2f, ffffffff7fffd630, 18, 1003e1d7c, ffffffff7fffd770) + 27c
00000001002c4d0c void NotifierDirect::loop(bool*) (10065f0b0, ffffffff7fffda2f, 1003e1, 10056eeb8, 100000, 1003e1b89) + 9c
000000010014fd0c ???????? (10062b5f0, ffffffff7fffdb1c, 100400, 10052a5b3, 100512e30, 10052a)
000000010029e874 ???????? (10062b5f0, 100400, 10056a69d, 0, 10066bb40, 1)
000000010029c888 int yylex(Interp*,int) (10062b5f0, 9, 0, 0, 10062b830, ffffffff7fffdc5c) + 220
000000010029a3fc int yyparse(Interp*) (10062b5f0, 100400, ffffffff7fffddd8, 0, 0, 1) + 18
000000010029c5c4 op*compile(Interp*,Source*) (10062b5f0, 10066bb40, 64, 100506000, 10062b798, 10066bbc8) + 28
00000001002972a8 int pdksh_shell(Interp*,Source*) (10062b5f0, 10066bb40, 100400, 100569df9, 100569e00, 1000b3c10) + 2fc
00000001000b817c void main_cmd_loop(Interp*) (10062b5f0, 100400, 100507c50, 1005e52e8, 100555400, 100507) + a8
00000001000b91ac main (100400, 100506, 100507c50, 10062b5f0, 100512e30, 1) + b8c
00000001000b1e3c _start (0, 0, 0, 0, 0, 0) + 17cSo can someone help me please ??You are seeing 6477975, which is fixed in dbx patch 117564-07 (which is for sparc) and 117565-07 (for x86).
However, if you are already on Solaris 10, I'd recommend to upgrade to Sun Studio 12, which is free, has many improvements and works much better with Solaris 10. If you, for any reason, can't upgrade compilers to Studio 12, you can still use Sun Studio 12 IDE plus debugger and compilers from Studio 9. It is easy to install several versions of Sun Studio on a machine, you just need to give different base directories to the installer. -
JVM 1.3.1_06 Solaris 8 (sparc) multiple jvm's crash at the same time
Hi All,
I had multiple jvms crash all at the same time today morning, i was wondering if anyone had any similar issues with 1.3.1_06:
Only a few jvms left an hs_errpid.log: here is the one with the most info:
An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : 10 occurred at PC=0xff02f8f8
Function name=VerifyClassCodes
Library=/www/gmgwh/weblogic/jdk/j2sdk1_3_1_06/jre/lib/sparc/libjava.so
Current Java thread:
at java.lang.Shutdown.halt(Native Method)
at java.lang.Shutdown.exit(Shutdown.java:209)
at java.lang.Terminator$1.handle(Terminator.java:33)
at sun.misc.Signal$1.run(Signal.java:193)
at java.lang.Thread.run(Thread.java:479)
Dynamic libraries:
0x10000 /www/gmgwh/weblogic/jdk/j2sdk1_3_1_06/jre/bin/../bin/sparc/native_threads/java
0xff350000 /usr/lib/libthread.so.1
0xff340000 /usr/lib/libdl.so.1
0xff200000 /usr/lib/libc.so.1
0xff330000 /usr/platform/SUNW,Ultra-Enterprise/lib/libc_psr.so.1
0xfe400000 /www/gmgwh/weblogic/jdk/j2sdk1_3_1_06/jre/lib/sparc/hotspot/libjvm.so
0xff2d0000 /usr/lib/libCrun.so.1
0xff1e0000 /usr/lib/libsocket.so.1
0xff100000 /usr/lib/libnsl.so.1
0xff0d0000 /usr/lib/libm.so.1
0xff1c0000 /usr/lib/libmp.so.2
0xff090000 /www/gmgwh/weblogic/jdk/j2sdk1_3_1_06/jre/lib/sparc/native_threads/libhpi.so
0xff060000 /www/gmgwh/weblogic/jdk/j2sdk1_3_1_06/jre/lib/sparc/libverify.so
0xff020000 /www/gmgwh/weblogic/jdk/j2sdk1_3_1_06/jre/lib/sparc/libjava.so
0xfe7d0000 /www/gmgwh/weblogic/jdk/j2sdk1_3_1_06/jre/lib/sparc/libzip.so
0xfe3e0000 /usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.2
0xfe390000 /www/gmgwh/weblogic/jdk/j2sdk1_3_1_06/jre/lib/sparc/libnet.so
0xfe230000 /usr/lib/nss_files.so.1
0xfe160000 /www/gmgwh/weblogic/6.1/sp4/lib/solaris/libmuxer.so
0xfd0e0000 /usr/ucblib/libucb.so.1
0xfd090000 /usr/lib/libresolv.so.2
0xfd050000 /usr/lib/libelf.so.1
0xfafd0000 /app/oracle/product/8.1.7.4/lib/libocijdbc8.so
0xe1800000 /app/oracle/product/8.1.7.4/lib/libclntsh.so.8.0
0xfafb0000 /app/oracle/product/8.1.7.4/lib/libwtc8.so
0xfaf90000 /usr/lib/libsched.so.1
0xfaee0000 /usr/lib/libaio.so.1
Local Time = Wed Jan 19 09:09:31 2005
Elapsed Time = 22367
# The exception above was detected in native code outside the VM
# Java VM: Java HotSpot(TM) Client VM (1.3.1_06-b01 mixed mode)
#hi there
i would direct my efforts to search for the culprit on who generated the signal (either a INT, TERM or HUP). The crash occurred is just but another side-effect. The VM received a shutdown signal and shld exit. Just that while exiting, it crashed.
There is this bug
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4904881
Which is pretty rare. I am not sure if you are encountering this bug but if u have a core, a pstack of the core should show > 2 threads with doing the signal shutdown. -
JCaps 5.1.3 Sun Solaris CPU performance issue
Folks,
We are experiencing a serious CPU performance issue on our Solaris server with HL7 projects deployed.
The projects consist of the sample HL7 inbound and outbound projects with an additional service sending to a batch local file external for writing journals.
The performance issue occurs when there is volume of data in the queues/topics. As we continue to deploy additional HL7 projects (usually about 6 interfaces), the CPU increases until it reached 100%.
This sanapshot is prstat when no date is transmitting through the interfaces (One inbound - one outbound):
B PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
15598 jre 379M 177M sleep 59 0 2:49:11 3.1% eManager/74
21549 phs 1174M 1037M sleep 59 0 14:49:00 2.5% is_dm_phs/113
23090 phs 3456K 3136K cpu1 59 0 0:00:01 0.4% prstat/1
23102 phs 3792K 3496K sleep 59 0 0:00:00 0.2% prstat/1
21550 phs 46M 35M sleep 59 0 0:13:27 0.1% stcms.exe/3
1272 noaccess 209M 95M sleep 59 0 0:26:30 0.1% java/25
11733 jre 420M 212M sleep 59 0 1:35:40 0.1% java/34
131 root 4368K 2480K sleep 59 0 0:02:10 0.1% nscd/30
23094 phs 3064K 2168K sleep 59 0 0:00:00 0.1% bash/1
This sanapshot is prstat when data is transmitting through the interfaces(One inbound - one outbound):
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
21549 phs 1174M 1037M cpu1 20 0 14:51:20 88% is_dm_phs/113
15598 jre 379M 181M sleep 59 0 2:49:18 1.3% eManager/74
21550 phs 46M 35M sleep 49 0 0:13:29 1.2% stcms.exe/3
23090 phs 3456K 3128K cpu3 49 0 0:00:03 0.4% prstat/1
1272 noaccess 209M 95M sleep 59 0 0:26:30 0.1% java/25
11733 jre 420M 212M sleep 59 0 1:35:40 0.1% java/34
21546 phs 118M 904K sleep 59 0 0:01:21 0.1% isprocmgr_dm_ph/13
This sanapshot is prstat -L when data is transmitting through the interfaces (One inbound - one outbound):
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/LWPID
21549 phs 1174M 1037M cpu1 41 0 0:00:45 22% is_dm_phs/13971
21549 phs 1174M 1037M sleep 51 0 3:31:06 21% is_dm_phs/1394
21549 phs 1174M 1037M run 51 0 3:14:16 20% is_dm_phs/1296
21549 phs 1174M 1037M sleep 52 0 3:14:13 19% is_dm_phs/1380
15598 jre 379M 181M sleep 50 0 1:49:57 3.1% eManager/4
21549 phs 1174M 1037M sleep 59 0 0:15:36 1.7% is_dm_phs/4
21550 phs 46M 35M sleep 59 0 0:10:52 1.0% stcms.exe/1
21549 phs 1174M 1037M sleep 59 0 0:10:45 0.9% is_dm_phs/6
15598 jre 379M 181M sleep 54 0 0:33:35 0.3% eManager/35
21549 phs 1174M 1037M sleep 59 0 0:03:34 0.3% is_dm_phs/5
21550 phs 46M 35M sleep 59 0 0:02:37 0.2% stcms.exe/2
21549 phs 1174M 1037M sleep 59 0 0:02:17 0.2% is_dm_phs/3
21549 phs 1174M 1037M sleep 59 0 0:02:17 0.2% is_dm_phs/2
Solaris 10 server details:
CPU's (4x900 Sparc III+)
4096 MB RAM
SunOS testican 5.9 Generic_118558-39 sun4u sparc SUNW,Sun-Fire-880
Disk: 6 internal Fujitsu 72GBs
swapspace on the server:
total: 4305272k bytes allocated + 349048k reserved = 4654320k used, 10190536k available
My sysadmin has run statistics (iostat, vmstat, psig, pmap, pfind, pstack, mpstat, etc.) - and has reported that the server is performing fine - with the exception of the CPU. It also looked like the swap space was not being utilized.
We have increased the MaxPerm value to 512, and increased the heapsize on isprocmgr_dm_phs to -Xmx2048m, and increased the heapsize on the domain to 2048 per KB 103824
We have also added the -d64 value (specific to Solaris) per the Deployment Guide.
We increased the value of Maximum Pool size in the JMS clients to 128 - per the deployment Guide.
We increased the swapspace on the server to 10Gb:
total: 4305272k bytes allocated + 349048k reserved = 4654320k used, 10190536k available
We have modified the tcpip and kernal parameters per the Sun Administration server 8.2 performance tuning guide:
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
open files (-n) 8192
pipe size (512 bytes, -p) 10
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 29995
virtual memory (kbytes, -v) unlimited
None of these modificatons appear to increase performance.
Any help is appreciated.
Thanks
Rich...Hi,
I noticed this behavior with the Alert + SNMP Agents installed but not configured. In this situation, the SNMP agent generates traps for all events, leading to high CPU using, even when nothing was processed. Are you in a similar case?
Regards -
Weblogic managed server crash and automatic restart on Solaris SPARC platform
hi Everyone,
We are facing a problem a problem with Weblogic managed server where the managed server crashes and automatically restarts (via nodemanager) on Solaris SPARC platform.
The system info is as below:
OS: Oracle Solaris 11.1 SPARC
Copyright (c) 1983, 2012, Oracle and/or its affiliates. All rights reserved.
Assembled 19 September 2012
uname:SunOS 5.11 11.1 sun4v
(T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 65536, AS infinity
load average:1.04 0.52 0.32
CPU:total 4 v9, popc, vis1, vis2, vis3, blk_init, cbcond, sun4v, niagara_plus
Memory: 8k page, physical 16777216k(5930008k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (23.25-b01) for solaris-sparc JRE (1.7.0_25-b11), built on May 15 2013 04:30:17 by "" with Sun Studio 12u1
The pstack output of analyzing the core file is as below (for the particular thread, I can attach the entire output of pstack if required):
--- called from signal handler with signal 11 (SIGSEGV) ---
ffffffff7ec1a7c4 memcpy%sun4v-hwcap3 (103d34720, 10a7e7710, 108ae1060, 6300, 0, ffffffff5f66394b) + 990
ffffffff36102fa4 ZIP_GetEntry (103d34720, ffffffff636f66a4, bd, 103d347f0, a26f1c, 2f) + e4
ffffffff36103440 Java_java_util_zip_ZipFile_getEntry (2b, ffffffff636f66a4, 103d34720, ffffffff636f6b70, 2b, 1) + a8
ffffffff7586ac1c * *java/util/zip/ZipFile.getEntry(J[BZ)J [compiled]
ffffffff75b61b58 * *java/util/zip/ZipFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry; [compiled] +39 (line 606)
ffffffff75b61b58 * *java/util/jar/JarFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry;+3 (line 452)
ffffffff75b61b58 * *weblogic/utils/classloaders/ZipClassFinder.getSource(Ljava/lang/String;)Lweblogic/utils/classloaders/Source;+36 (line 75)
ffffffff75b6eb84 * *weblogic/utils/classloaders/JarClassFinder.getSource(Ljava/lang/String;)Lweblogic/utils/classloaders/Source; [compiled] +6 (line 100)
ffffffff75a40938 * *weblogic/utils/classloaders/AbstractClassFinder.getClassSource(Ljava/lang/String;)Lweblogic/utils/classloaders/Source; [compiled] +74 (line 58)
ffffffff75976c88 * *weblogic/utils/classloaders/MultiClassFinder.getClassSource(Ljava/lang/String;)Lweblogic/utils/classloaders/Source; [compiled] +30 (line 114)
ffffffff75976c88 * *weblogic/utils/classloaders/MultiClassFinder.getClassSource(Ljava/lang/String;)Lweblogic/utils/classloaders/Source; [compiled] +30 (line 114)
ffffffff75976c88 * *weblogic/utils/classloaders/MultiClassFinder.getClassSource(Ljava/lang/String;)Lweblogic/utils/classloaders/Source; [compiled] +30 (line 114)
ffffffff75806d10 * weblogic/application/utils/CompositeWebAppFinder.getClassSource(Ljava/lang/String;)Lweblogic/utils/classloaders/Source;+5 (line 176)
ffffffff75976c88 * *weblogic/utils/classloaders/MultiClassFinder.getClassSource(Ljava/lang/String;)Lweblogic/utils/classloaders/Source; [compiled] +30 (line 114)
ffffffff75976c88 * *weblogic/utils/classloaders/MultiClassFinder.getClassSource(Ljava/lang/String;)Lweblogic/utils/classloaders/Source; [compiled] +30 (line 114)
ffffffff75806d10 * weblogic/utils/classloaders/CodeGenClassFinder.getClassSource(Ljava/lang/String;)Lweblogic/utils/classloaders/Source;+43 (line 48)
ffffffff75806d10 * weblogic/utils/classloaders/GenericClassLoader.findLocalClass(Ljava/lang/String;)Ljava/lang/Class;+87 (line 571)
ffffffff75806d10 * weblogic/utils/classloaders/GenericClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class;+117 (line 515)
ffffffff75806d10 * weblogic/utils/classloaders/ChangeAwareClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class;+60 (line 121)
ffffffff75a465c8 * *java/lang/ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; [compiled] +71 (line 836)
ffffffff75db42c0 * *java/lang/ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; [compiled] +4 (line 714)
ffffffff75806d10 * weblogic/utils/classloaders/GenericClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+2 (line 358)
ffffffff75806d10 * weblogic/utils/classloaders/ChangeAwareClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+27 (line 83)
ffffffff7580024c * StubRoutines (1)
ffffffff3691f44c __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (ffffffff636f7f98, 10234f000, 2, c, 2, 10) + 2f4
ffffffff36e546c0 __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_pnGSymbol_63pnGThread__v_ (ffffffff636f7f98, 1013e7748, 10234f000, 10011bf30, 100119ec0, 772d4ec08) + 1b8
ffffffff3725bf0c __1cQSystemDictionaryTload_instance_class6FpnGSymbol_nGHandle_pnGThread__nTinstanceKlassHandle__ (105cb6100, 1013e7978, 10234f000, 102b6e500, 0, 1013e7748) + 494
ffffffff3725a028 __1cQSystemDictionarybEresolve_instance_class_or_null6FpnGSymbol_nGHandle_3pnGThread__pnMklassOopDesc__ (105cb6100, 19dc00, 70, 10234f000, 70, 1013e7700) + ae0
ffffffff37257800 __1cQSystemDictionaryPresolve_or_fail6FpnGSymbol_nGHandle_3bpnGThread__pnMklassOopDesc__ (105cb6100, 1013e76f0, 1013e76f8, 1, 10234f000, 1) + 290
ffffffff368ba9dc __1cTconstantPoolOopDescNklass_at_impl6FnSconstantPoolHandle_ipnGThread__pnMklassOopDesc__ (10201b6e0, 1013e7978, 10234f000, 10201b6e0, 783cfb098, ffffffff37588000) + 264
ffffffff36937380 __1cSInterpreterRuntimeE_new6FpnKJavaThread_pnTconstantPoolOopDesc_i_v_ (10234f000, 77548b730, 49, 10201b6e0, 1013e76e8, 10234f000) + b8
ffffffff7581ab58 * com/jdedwards/base/logging/log4j/JdeLogger.warn(ILjava/lang/String;Ljava/lang/String;Ljava/lang/Object;Ljava/lang/Throwable;)V+14
ffffffff75807bcc * com/jdedwards/system/security/proxy/SecurityServerProxyImpl.install(Ljava/util/Hashtable;)Lcom/jdedwards/system/security/proxy/SecurityServerProxyImpl;+30
ffffffff75806d10 * com/jdedwards/system/security/proxy/SecurityServerProxyFactory.create(Ljava/util/Hashtable;)Lcom/jdedwards/system/security/SecurityServer;+1
ffffffff75807898 * com/jdedwards/system/security/SecurityServerInstance.getInstance()Lcom/jdedwards/system/security/SecurityServer;+27
ffffffff75806d10 * com/jdedwards/mgmt/security/UserManager.checkPassword(Ljava/lang/String;Ljava/lang/String;)Z+59
ffffffff75806b24 * com/jdedwards/mgmt/security/UserManagerImpl.checkPassword(Ljava/lang/String;Ljava/lang/String;)Z+5
ffffffff75806b24 * com/jdedwards/mgmt/security/SCFLoginModule.login()Z+481
The output of the hs_err_pid1127.log file is as below:
# A fatal error has been detected by the Java Runtime Environment:
# SIGSEGV (0xb) at pc=0xffffffff7ec1a544, pid=1127, tid=20
# JRE version: 7.0_25-b11
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode solaris-sparc compressed oops)
# Problematic frame:
# C [libc.so.1+0x11a544] memcpy%sun4v-hwcap3+0x710
# Core dump written. Default location: /slot/ems9437/appmgr/Oracle/Middleware/user_projects/domains/smc_domain/core or core.1127
# If you would like to submit a bug report, please visit:
# http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
--------------- T H R E A D ---------------
Current thread (0x000000010294d000): JavaThread "[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_in_native, id=20, stack(0xffffffff62f00000,0xffffffff63000000)]
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0xffffffff5f6ce000
I ran the suspect code which i believed was causing the issue as a standalone Java program using the same Java executable and it worked fine without any crash.
Appreciate any inputs / discussion / feedback / thoughts on this topic.
Good day.I just wanted to throw in one more thought. I have also seen that there is a restart=true parameter. Maybe if we set this to true, and just issue the command to shut down the server in the script, the managed server will restart on its own.
Would greatly appreciate any advice! -
WLS6.1 hanging on Solaris- and SIGQUIT doesn't work
Hi,
We are experiencing a problem that looks like a deadlock when our server is under load (using WLS6.1SP2, Solaris 8 and the 1.3.1 JVM supplied with WLS). After processing many requests, the server hangs and CPU usage drops to 0%. Unfortunately the standard means of getting a thread dump (sending a SIGQUIT with kill -3 or Ctrl-\) does nothing. The problem also occurs if WebLogic is running under debug (using
-Xdebug -Xnoagent -Djava.compiler=none -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5555) but alas the debugger hangs when trying to pause the server after this deadlock). It appears as though the server is not accepting any requests (including administrative requests from the console) but this doesn't seem to be due to thread starvation as CPU usage is 0%.
The only output we can get is from truss, which seems to indicate no activity at all until the SIGQUIT is received. It seems to register the SIGQUIT but it looks as though it is not being processed.
Does anyone have any ideas?
Kevin.We found a solution to this problem, relating to the threading model that the JVM
uses.
Have a look at http://java.sun.com/docs/hotspot/threads/threads.html for more
details.
The default threading model on Solaris 8 is many-to-many (threads to LWPs) with
thread-based synchronisation. After playing with various threading models, we
found that the best was one-to-one with the alternate thread library. This is
the default provided by Solaris 9, but you can also tell Java to use this with
older versions of Solaris by putting the following line in your start command
prior to executing WebLogic:
export LD_LIBRARY_PATH=/usr/lib/lwp:$LD_LIBRARY_PATH
Apologies to anyone who was looking for an answer to this beforehand: I should
have replied to this newsgroup back in September!
Kevin Thomas
J2EE Consultant
LogicaCMG
Charlie Therit <[email protected]> wrote:
Kevin,
If you are not able to capture a JVM thread dump, then the next best
thing would be to capture several "/usr/proc/bin/pstack pid" C-level
thread dumps. This information may enable BEA Support to help suggest
potential work arounds. Depending upon the data in the pstack output,
you may also wish to open a support case with Sun.
Sincerely,
Charlie Therit
Developer Relations Engineer
BEA Support
Kevin Thomas wrote:
Hi,
We are experiencing a problem that looks like a deadlock when our serveris under load (using WLS6.1SP2, Solaris 8 and the 1.3.1 JVM supplied
with WLS). After processing many requests, the server hangs and CPU
usage drops to 0%. Unfortunately the standard means of getting a thread
dump (sending a SIGQUIT with kill -3 or Ctrl-\) does nothing. The problem
also occurs if WebLogic is running under debug (using
-Xdebug -Xnoagent -Djava.compiler=none -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5555)but alas the debugger hangs when trying to pause the server after this
deadlock). It appears as though the server is not accepting any requests
(including administrative requests from the console) but this doesn't
seem to be due to thread starvation as CPU usage is 0%.
The only output we can get is from truss, which seems to indicate noactivity at all until the SIGQUIT is received. It seems to register
the SIGQUIT but it looks as though it is not being processed.
Does anyone have any ideas?
Kevin. -
Non interactive zlogin eats up all CPU on Solaris 10 u4
Hi,
We are having trouble with a zlogin command that is started via a script from inittab on a Solaris 10 u4 system.
When a zlogin command is started from the script this zlogin will within 10-15 seconds start spinning and consume 97-98% of the CPU. The zlogin command never ends, and the startup is stuck there.
Running a pstat during the zlogin shows the following picture:
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
1912 root 6576K 3560K run 100 - 0:22:30 97% zlogin/1
1489 root 3544K 3184K cpu0 100 - 0:00:15 0.9% prstat/1
159 root 3880K 3080K sleep 100 - 0:00:04 0.3% picld/11
ZONEID NPROC SWAP RSS MEMORY TIME CPU ZONE
0 89 509M 548M 27% 0:23:07 100% global
1 22 26M 40M 1.9% 0:00:05 0.0% zone01
Total: 111 processes, 362 lwps, load averages: 4.25, 4.23, 3.49A ptree of the zlogin command
# ptree 1912
993 /sbin/sh /opt/sz/sysadmin/bin/set_remote_status.sh -b
1912 zlogin -l root zone01 svcs -a |grep svc:/milestone/multi-user-server:def
1913 sh -c svcs -a |grep svc:/milestone/multi-user-server:default
1919 grep svc:/milestone/multi-user-server:default
1920 svcs -aA truss of the zlogin shows that it is spinning in a poll/fstat loop.
# truss -p 1912
pollsys(0xFFBFE6E0, 3, 0x00000000, 0x00000000) = 1
fstat(4, 0xFFBFC5F8) = 0
pollsys(0xFFBFE6E0, 3, 0x00000000, 0x00000000) = 1
fstat(4, 0xFFBFC5F8) = 0
pollsys(0xFFBFE6E0, 3, 0x00000000, 0x00000000) = 1
fstat(4, 0xFFBFC5F8) = 0
Received signal #18, SIGCLD [caught]
siginfo: SIGCLD CLD_EXITED pid=1913 status=0x0000
lwp_sigmask(SIG_SETMASK, 0x00020000, 0x00000000) = 0xFFBFFEFF [0x0000FFFF]
waitid(P_PID, 1913, 0xFFBFBC38, WEXITED|WTRAPPED|WNOHANG|WNOWAIT) = 0
setcontext(0xFFBFBC20)
pollsys(0xFFBFE6E0, 2, 0xFFBFC678, 0x00000000) = 1
read(5, " o f f l i n e ".., 8192) = 65
write(1, " o f f l i n e ".., 65) = 65
pollsys(0xFFBFE6E0, 2, 0xFFBFC678, 0x00000000) = 0
waitid(P_PID, 1913, 0xFFBFE618, WEXITED|WTRAPPED) = 0
_exit(0)When running the process under truss the zlogin runs to end, probably because the child processes now get time for doing their work.
A pstack of the zlogin process shows something similar
# pstack 1912
1912: zlogin -l root zone01 svcs -a |grep svc:/milestone/multi-user-server:de
ff2465c8 __pollsys (ffbfe6e0, 3, 0, 0, 0, 0) + 8
ff1e1b34 poll (ffbfe6e0, 3, ffffffff, 0, 0, 0) + 7c
000128b4 ???????? (3, 4, 5, 7, 1, 126400)
00013c30 ???????? (ffbff47b, 12400, ffbff476, 127450, 1428f8, 2)
00014460 main (4, 1, 1428f8, 15400, 15400, 0) + 728
00011a24 _start (0, 0, 0, 0, 0, 0) + 108A pfiles out the zlogin show the following:
# pfiles 1912
1912: zlogin -l root zone01 svcs -a |grep svc:/milestone/multi-user-server:de
Current rlimit: 256 file descriptors
0: S_IFREG mode:0555 dev:85,10 ino:47996 uid:0 gid:0 size:64656
O_RDONLY|O_LARGEFILE
/opt/sz/sysadmin/bin/set_remote_status.sh
1: S_IFIFO mode:0000 dev:308,0 ino:1377 uid:0 gid:0 size:0
O_RDWR
3: S_IFIFO mode:0000 dev:308,0 ino:1378 uid:0 gid:0 size:0
O_RDWR
4: S_IFIFO mode:0000 dev:308,0 ino:1378 uid:0 gid:0 size:8192
O_RDWR
5: S_IFIFO mode:0000 dev:308,0 ino:1379 uid:0 gid:0 size:0
O_RDWR
6: S_IFIFO mode:0000 dev:308,0 ino:1379 uid:0 gid:0 size:0
O_RDWR
7: S_IFIFO mode:0000 dev:308,0 ino:1380 uid:0 gid:0 size:0
O_RDWR
8: S_IFIFO mode:0000 dev:308,0 ino:1380 uid:0 gid:0 size:0
O_RDWRSo do anybody know a workaround for this behavior, or any other clues?Found the reason. The default scheduling class was set to RT. Probably explained why zlogin was allowed to eat up all CPU resources
-
Xsun (high CPU utilization) on Solaris 10 Sparc
hi
i have sun blade 1500 and am running solaris 10 on it. the machine is a 2 CPU (750Mhz) 4GB Sparc with the lastest cluster patch.
The Xsun process is alway at 50% util and the windowmaker (wmaker) is at 27%.
The Xsun is alway using all available CPU and the machine is really slow.. any help on what patch will fix the Xsun process ? any operation on the machine increases the Xsun's CPU util.
thx
SriramHi sridhar,
can i know which platform you are using....
if it is solaris,
can you paste the details of the prstat -L -p wlpid 1 1 ---> which gives the light wieght pid threads
and also (pstact wlspid) for lwpid to process id mapping
or you can follow these steps for finding which thread is causing the hight cpu utilization
1.find the highest usage lwpid in prstat output
2.find the lwpid in pstack output and get the matching thread number
3.convert the thread number to hexadecimal
4.find the hexadecimal thread number in the server thread dump (nid= xxx)
5.determine what thread was doing to cause the high cpu usage
you can find similar way if it is linux..
ps u -Lp wlspid and thread dump
Thank you,
Bob
Edited by: Bob on Sep 21, 2010 10:18 AM
Edited by: Bob on Sep 21, 2010 10:24 AM
Maybe you are looking for
-
How to refresh interal table in Webdynpro ABAP
Hi all, I want to delete/free/refresh interal table in webdynpro abap. One of the form gave solution to pass blank internal table? I cleared node and itab in webdynpro, but internally SAP doesn't delete the context node and itab values. Please help m
-
Webapp - where to store user files?
Hi All, My app is packaged in a WAR file, which gets expanded by servlet container during deployment. The app operates with folders containing user data (files uploaded by users, email templates, etc). Where should I create these user data folders -
-
Hi Gurus, We schedule background jobs to convert Planned order to a Production order using program PPBICO40. This program/job is generating batch input sessions, weekly we go to SM35 and delete these sessions manually. This is becoming a weekly maint
-
Can Crystal Reports reports run unattended?
Can Crystal Reports run in an unattended mode as in: 1) Being run by a batch process to produce a file for mailing 2) Being run within the scope of an NUnit unit test?
-
I had my ipad mini replaced and everything got restored except my contacts