Performance Degradated Possibly due to CPU Time

Hi Gurus,
There is a utility in our application with which we can upload an excel sheet containing data and schedule the timing of the job, now when the job is executed, each row in the excel sheet leads to dml operations on multiple tables finally leading to generation of a transaction no. Now at the start around 100-120 transaction nos were generated which goes down drastically to around 30-35 after 6-7 hours. AWR report at the two instances shows that CPU time has decreased considerably in the 2nd case.
I would like you experts to check the awr reports and suggest me the probable reason for the decrease in performance.
Brief AWR Report When Performance was OK
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 2151 14-Dec-10 16:32:57 26 3.7
End Snap: 2152 14-Dec-10 17:31:04 40 16.7
Elapsed: 58.13 (mins)
DB Time: 55.37 (mins)
Cache Sizes
~~~~~~~~~~~ Begin End
Buffer Cache: 436M 444M Std Block Size: 8K
Shared Pool Size: 120M 120M Log Buffer: 6,968K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 27,541.56 1,747.07
Logical reads: 49,830.97 3,160.97
Block changes: 181.79 11.53
Physical reads: 1,270.12 80.57
Physical writes: 2.81 0.18
User calls: 119.95 7.61
Parses: 200.94 12.75
Hard parses: 29.29 1.86
Sorts: 91.80 5.82
Logons: 0.03 0.00
Executes: 457.16 29.00
Transactions: 15.76
% Blocks changed per Read: 0.36 Recursive Call %: 96.36
Rollback per transaction %: 0.01 Rows per Sort: 270.64
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 97.45 In-memory Sort %: 100.00
Library Hit %: 90.18 Soft Parse %: 85.42
Execute to Parse %: 56.05 Latch Hit %: 100.00
Parse CPU to Parse Elapsd %: 98.04 % Non-Parse CPU: 94.98
Shared Pool Statistics Begin End
Memory Usage %: 72.65 84.55
% SQL with executions>1: 71.49 75.08
% Memory for SQL w/exec>1: 84.79 85.25
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time Wait Class
CPU time 2,541 76.5
db file scattered read 284,992 410 1 12.3 User I/O
log file parallel write 31,188 145 5 4.4 System I/O
TCP Socket (KGAS) 24 131 5459 3.9 Network
log file sync 8,617 46 5 1.4 Commit
Time Model Statistics DB/Inst: ABCTEST/abctest Snaps: 2151-2152
-> Total time in database user-calls (DB Time): 3322.4s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 3,176.8 95.6
DB CPU 2,541.1 76.5
PL/SQL execution elapsed time 288.5 8.7
parse time elapsed 278.7 8.4
hard parse elapsed time 254.6 7.7
PL/SQL compilation elapsed time 28.9 .9
failed parse elapsed time 4.9 .1
hard parse (sharing criteria) elapsed time 1.3 .0
sequence load elapsed time 1.1 .0
repeated bind elapsed time 1.1 .0
connection management call elapsed time 0.7 .0
hard parse (bind mismatch) elapsed time 0.3 .0
DB time 3,322.4 N/A
background elapsed time 197.1 N/A
background cpu time 5.6 N/A
Wait Class DB/Inst: ABCTEST/abctest Snaps: 2151-2152
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc
Avg
%Time Total Wait wait Waits
Wait Class Waits -outs Time (s) (ms) /txn
User I/O 292,720 .0 427 1 5.3
System I/O 37,408 .0 190 5 0.7
Network 272,062 .0 132 0 4.9
Commit 8,617 .0 46 5 0.2
Configuration 4 .0 2 593 0.0
Application 3,212 .0 0 0 0.1
Other 280 .4 0 0 0.0
Concurrency 247 .0 0 0 0.0
Wait Events DB/Inst: ABCTEST/abctest Snaps: 2151-2152
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
db file scattered read 284,992 .0 410 1 5.2
log file parallel write 31,188 .0 145 5 0.6
TCP Socket (KGAS) 24 .0 131 5459 0.0
log file sync 8,617 .0 46 5 0.2
db file parallel write 4,215 .0 29 7 0.1
db file sequential read 7,634 .0 16 2 0.1
control file parallel write 1,202 .0 16 13 0.0
Streams AQ: enqueue blocked 1 .0 2 2055 0.0
control file sequential read 795 .0 1 1 0.0
Data file init write 48 .0 0 9 0.0
SQL*Net message to client 266,802 .0 0 0 4.9
log file switch completion 3 .0 0 106 0.0
SQL*Net break/reset to clien 3,212 .0 0 0 0.1
SQL*Net more data to client 4,789 .0 0 0 0.1
direct path write 23 .0 0 3 0.0
rdbms ipc reply 67 .0 0 1 0.0
kksfbc child completion 1 100.0 0 47 0.0
latch: shared pool 213 .0 0 0 0.0
latch: library cache 26 .0 0 1 0.0
log file single write 4 .0 0 7 0.0
log file sequential read 4 .0 0 5 0.0
db file single write 3 .0 0 5 0.0
os thread startup 3 .0 0 4 0.0
enq: JS - queue lock 4 .0 0 3 0.0
LGWR wait for redo copy 207 .0 0 0 0.0
library cache pin 1 .0 0 6 0.0
SQL*Net more data from clien 447 .0 0 0 0.0
library cache load lock 1 .0 0 2 0.0
latch: cache buffers chains 1 .0 0 0 0.0
latch: row cache objects 1 .0 0 0 0.0
direct path read 20 .0 0 0 0.0
latch free 1 .0 0 0 0.0
cursor: mutex S 1 .0 0 0 0.0
SQL*Net message from client 266,789 .0 64,143 240 4.9
Streams AQ: qmn slave idle w 124 .0 3,488 28127 0.0
Streams AQ: qmn coordinator 257 51.4 3,488 13571 0.0
virtual circuit status 116 100.0 3,480 29999 0.0
Streams AQ: waiting for time 5 60.0 745 148902 0.0
jobq slave wait 52 96.2 155 2987 0.0
PL/SQL lock timer 16 100.0 16 995 0.0
class slave wait 1 100.0 5 4995 0.0
Background Wait Events DB/Inst: ABCTEST/abctest Snaps: 2151-2152
-> ordered by wait time desc, waits desc (idle events last)
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
log file parallel write 31,188 .0 145 5 0.6
db file parallel write 4,215 .0 29 7 0.1
control file parallel write 1,193 .0 16 13 0.0
Streams AQ: enqueue blocked 1 .0 2 2055 0.0
control file sequential read 691 .0 0 1 0.0
db file sequential read 66 .0 0 5 0.0
direct path write 23 .0 0 3 0.0
log file single write 4 .0 0 7 0.0
log file sequential read 4 .0 0 5 0.0
events in waitclass Other 211 .0 0 0 0.0
os thread startup 3 .0 0 4 0.0
db file scattered read 1 .0 0 13 0.0
latch: shared pool 5 .0 0 0 0.0
direct path read 20 .0 0 0 0.0
latch: library cache 1 .0 0 0 0.0
rdbms ipc message 34,411 32.3 30,621 890 0.6
Streams AQ: qmn slave idle w 124 .0 3,488 28127 0.0
Streams AQ: qmn coordinator 257 51.4 3,488 13571 0.0
pmon timer 1,235 100.0 3,486 2822 0.0
smon timer 19 47.4 3,460 182099 0.0
Streams AQ: waiting for time 5 60.0 745 148902 0.0
class slave wait 1 100.0 5 4995 0.0
Operating System Statistics DB/Inst: ABCTEST/abctest Snaps: 2151-2152
Statistic Total
AVG_BUSY_TIME 81,951
AVG_IDLE_TIME 266,698
AVG_SYS_TIME 10,482
AVG_USER_TIME 71,389
BUSY_TIME 328,163
IDLE_TIME 1,067,144
SYS_TIME 42,281
USER_TIME 285,882
RSRC_MGR_CPU_WAIT_TIME 0
VM_IN_BYTES 1,625,600,000
VM_OUT_BYTES 145,162,240
PHYSICAL_MEMORY_BYTES 3,755,851,776
NUM_CPUS 4
NUM_CPU_CORES 1
Brief AWR Report When Performance* Deteriorated.
Snap Id Snap Time Sessions Curs/Sess
Begin Snap: 2168 15-Dec-10 08:31:05 32 18.4
End Snap: 2169 15-Dec-10 09:30:56 32 18.3
Elapsed: 59.85 (mins)
DB Time: 17.97 (mins)
Cache Sizes
~~~~~~~~~~~ Begin End
Buffer Cache: 448M 448M Std Block Size: 8K
Shared Pool Size: 116M 116M Log Buffer: 6,968K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 10,503.58 1,792.02
Logical reads: 17,583.21 2,999.87
Block changes: 68.60 11.70
Physical reads: 472.37 80.59
Physical writes: 1.54 0.26
User calls: 39.12 6.67
Parses: 53.32 9.10
Hard parses: 7.99 1.36
Sorts: 13.84 2.36
Logons: 0.00 0.00
Executes: 130.30 22.23
Transactions: 5.86
% Blocks changed per Read: 0.39 Recursive Call %: 94.39
Rollback per transaction %: 0.00 Rows per Sort: 691.64
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 97.31 In-memory Sort %: 100.00
Library Hit %: 92.41 Soft Parse %: 85.02
Execute to Parse %: 59.08 Latch Hit %: 100.00
Parse CPU to Parse Elapsd %: 100.28 % Non-Parse CPU: 95.35
Shared Pool Statistics Begin End
Memory Usage %: 88.40 88.48
% SQL with executions>1: 76.15 80.48
% Memory for SQL w/exec>1: 86.82 88.85
Top 5 Timed Events Avg %Total
~~~~~~~~~~~~~~~~~~ wait Call
Event Waits Time (s) (ms) Time Wait Class
CPU time 918 85.1
db file scattered read 113,003 127 1 11.7 User I/O
log file parallel write 11,978 52 4 4.8 System I/O
db file parallel write 3,089 16 5 1.4 System I/O
control file parallel write 1,217 15 13 1.4 System I/O
Time Model Statistics DB/Inst: ABCTEST/abctest Snaps: 2168-2169
-> Total time in database user-calls (DB Time): 1078.1s
-> Statistics including the word "background" measure background process
time, and so do not contribute to the DB time statistic
-> Ordered by % or DB time desc, Statistic name
Statistic Name Time (s) % of DB Time
sql execute elapsed time 1,032.1 95.7
DB CPU 917.6 85.1
parse time elapsed 71.8 6.7
hard parse elapsed time 52.4 4.9
PL/SQL execution elapsed time 7.2 .7
PL/SQL compilation elapsed time 6.2 .6
failed parse elapsed time 1.8 .2
sequence load elapsed time 0.4 .0
repeated bind elapsed time 0.3 .0
connection management call elapsed time 0.1 .0
hard parse (sharing criteria) elapsed time 0.0 .0
hard parse (bind mismatch) elapsed time 0.0 .0
DB time 1,078.1 N/A
background elapsed time 89.4 N/A
background cpu time 6.4 N/A
Wait Class DB/Inst: ABCTEST/abctest Snaps: 2168-2169
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc
Avg
%Time Total Wait wait Waits
Wait Class Waits -outs Time (s) (ms) /txn
User I/O 122,810 .0 133 1 5.8
System I/O 17,013 .0 83 5 0.8
Commit 3,129 .0 14 5 0.1
Network 90,186 .0 0 0 4.3
Configuration 2 .0 0 63 0.0
Application 1,120 .0 0 0 0.1
Other 112 .0 0 0 0.0
Concurrency 2 .0 0 6 0.0
Wait Events DB/Inst: ABCTEST/abctest Snaps: 2168-2169
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
db file scattered read 113,003 .0 127 1 5.4
log file parallel write 11,978 .0 52 4 0.6
db file parallel write 3,089 .0 16 5 0.1
control file parallel write 1,217 .0 15 13 0.1
log file sync 3,129 .0 14 5 0.1
db file sequential read 9,753 .0 6 1 0.5
control file sequential read 725 .0 0 0 0.0
Data file init write 32 .0 0 7 0.0
SQL*Net message to client 88,906 .0 0 0 4.2
log file switch completion 2 .0 0 63 0.0
SQL*Net break/reset to clien 1,120 .0 0 0 0.1
rdbms ipc reply 4 .0 0 8 0.0
direct path write 10 .0 0 3 0.0
SQL*Net more data to client 1,120 .0 0 0 0.1
db file single write 2 .0 0 6 0.0
os thread startup 2 .0 0 6 0.0
log file single write 2 .0 0 4 0.0
log file sequential read 2 .0 0 3 0.0
SQL*Net more data from clien 160 .0 0 0 0.0
LGWR wait for redo copy 108 .0 0 0 0.0
direct path read 10 .0 0 0 0.0
SQL*Net message from client 88,906 .0 55,500 624 4.2
virtual circuit status 120 100.0 3,588 29900 0.0
Streams AQ: qmn slave idle w 127 .0 3,550 27949 0.0
Streams AQ: qmn coordinator 260 51.2 3,550 13652 0.0
class slave wait 2 100.0 10 4994 0.0
SGA: MMAN sleep for componen 9 22.2 0 4 0.0
Background Wait Events DB/Inst: ABCTEST/abctest Snaps: 2168-2169
-> ordered by wait time desc, waits desc (idle events last)
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
log file parallel write 11,978 .0 52 4 0.6
db file parallel write 3,089 .0 16 5 0.1
control file parallel write 1,211 .0 15 13 0.1
db file scattered read 175 .0 0 1 0.0
control file sequential read 33 .0 0 2 0.0
db file sequential read 53 .0 0 1 0.0
direct path write 10 .0 0 3 0.0
os thread startup 2 .0 0 6 0.0
log file single write 2 .0 0 4 0.0
log file sequential read 2 .0 0 3 0.0
events in waitclass Other 108 .0 0 0 0.0
direct path read 10 .0 0 0 0.0
rdbms ipc message 19,991 57.4 31,320 1567 0.9
pmon timer 1,208 100.0 3,590 2972 0.1
Streams AQ: qmn slave idle w 127 .0 3,550 27949 0.0
Streams AQ: qmn coordinator 260 51.2 3,550 13652 0.0
smon timer 12 100.0 3,302 275149 0.0
SGA: MMAN sleep for componen 9 22.2 0 4 0.0
Operating System Statistics DB/Inst: ABCTEST/abctest Snaps: 2168-2169
Statistic Total
AVG_BUSY_TIME 30,152
AVG_IDLE_TIME 328,781
AVG_SYS_TIME 4,312
AVG_USER_TIME 25,757
BUSY_TIME 120,981
IDLE_TIME 1,315,433
SYS_TIME 17,612
USER_TIME 103,369
RSRC_MGR_CPU_WAIT_TIME 0
VM_IN_BYTES 353,361,920
VM_OUT_BYTES 163,041,280
PHYSICAL_MEMORY_BYTES 3,755,851,776
NUM_CPUS 4
NUM_CPU_CORES 1
Request you to help me.
Thanks in Advance,
Rajesh

Hi CKPT,
Thanks for your reply.
The main finding that I have got from addm report (in both the cases i.e when performance was good initially vis a vis when performance deteriorated is the same -
FINDING 1: 100% impact (3234 seconds)
Significant virtual memory paging was detected on the host operating system.
RECOMMENDATION 1: Host Configuration, 100% benefit (3234 seconds)
ACTION: Host operating system was experiencing significant paging but no
particular root cause could be detected. Investigate processes that
do not belong to this instance running on the host that are consuming
significant amount of virtual memory. Also consider adding more
physical memory to the host.
I still am unable to find out the reasons ... pls help.
Thanks
Rajesh

Similar Messages

Transaction aborted (possibly due to transaction time out) - Question

This one is driving me insane!
Bellow you will find the code, the descriptor stuff and the exception. The code belongs to a CMP EJB using CMT (transactions). This gets called from a session bean. Please help.
Here is the code:
public Object ejbCreate( String id )
     throws CreateException {
     s_logger.debug( "ejbCreate()" ); // this prits
     //setId( id );
     return null;
public void ejbPostCreate( String id )
     throws CreateException {
     s_logger.debug( "ejbPostCreate()" ); // this also prints
here is descriptor stuff:
<container-transaction>
<method>
<ejb-name>AgentBean</ejb-name>
<method-name>*</method-name>
</method>
<trans-attribute>Required</trans-attribute>
</container-transaction>
here is the !@#$ exception (sorry been tracing this one for hours):
RemoteException occurred in server thread; nested exception is:
java.rmi.RemoteException: Transaction aborted (possibly due to transaction time out).; nested exception is: javax.transaction.RollbackException: Transaction marked for rollback; nested exception is:
javax.transaction.RollbackException: Transaction marked for rollback
java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
java.rmi.RemoteException: Transaction aborted (possibly due to transaction time out).; nested exception is: javax.transaction.RollbackException: Transaction marked for rollback; nested exception is:
javax.transaction.RollbackException: Transaction marked for rollback
java.rmi.RemoteException: Transaction aborted (possibly due to transaction time out).; nested exception is: javax.transaction.RollbackException: Transaction marked for rollback; nested exception is:
javax.transaction.RollbackException: Transaction marked for rollback
javax.transaction.RollbackException: Transaction marked for rollback
<<no stack trace available>>

It usually means that you're doing something which creates a loop in your locking. Something like setting a value in each instance from an update method of one of the instances. Your posted code doesn't seem to be the trouble, but I could be missing something.
Success,
S.

Performance issue possibly due to wrong parameters??

Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit.
We have a program that runs every two weeks to process 3million records in our Oracle 10g database. Processing normally takes about 6 hours. With no change to the program (which is a java client program), the processing time has gotten longer and longer over the last few weeks. The processor on the database server was upgraded to iTanium and during this upgrade the databases were striped to fix a read/write issue that occurred due to poor configuration (it wasn't using multiple channels for processing so everything ran super slow) when the server hardware was upgraded. Since the last upgrade to the processor, we've noticed many errors being generated on the server when our program runs (java null reference errors). these errors never occurred before the upgrade. The upgrade was in July. In August we noticed the beginning of a degradation in performance - the process went from 6 hours to 10 hours. This month, it is taking 20 hours. Next month I fear it will be 40 hours.
The program launches multiple sessions that work at once doing an update against the same table. The last time it ran it started at 6AM and by 1PM it was only 11% done. I looked at the session stats and saw the top 5 wait events:
SQL*Net message from client-->
totalwaits=11,997
timewaited=2,070,070
avgwait=172
enq: TX - row lock contention-->
totalwaits=587
timewaited 65,614
avgwait=111
timeouts=180
latch:cache buffers chains-->
totalwaits=933
timewaited=1,815
avgwait=2
db file sequential read-->
totalwaits=1,426
timewaited=1,519
avgwait=1
log file sync-->
totalwaits=1,422
totalwaited=2594
avgwait=2
It looks like all of these values are way too high and I'm wondering what parameters we could change on the database/server side that might improve performance in these areas.
I read that increasing the INITRANS VALUE TO SOMETHING GREATER THAN ONE WOULD HELP WITH THE ROW LOCK CONTENTION AND TIMEOUTS.
I also read that changing DB_CACHE_ADVICE OFF WOULD HELP WITH THE CACHE BUFFER CHAINS ISSUE.
Are these viable solutions? Changing the program is not an option right now. Any help is greatly appreciated.

Rakesh jayappa wrote:
Hi,
sorry i am not getting your question, i am guessing your question, you can reduce the log file sync by
COMMIT WRITE BATCH;
COMMIT WRITE IMMEDIATE;
or
The disks or I/O subsystems where the redologs are placed may be too busy.
- Reduce other I/O activity on the disks containing the redo logs, or use dedicated disks.
- Move the redo logs to faster disks or a faster I/O subsystem.
- Move the redolog files from RAID 5 devices. RAID 5 is not efficient for writes.
- Alternate redo logs on different disks to minimize the effect of the archiver on the log writer.
Kind Regards,
Rakesh JayappaMy point is that even if you eliminate it completely you have only eliminated a tiny fraction of the total wait time -- it may look like low hanging fruit, but it's a very small piece of fruit indeed.

Performance Improvement possible due to replacing delete-command?

Dear Specialists,
has anybody of you an idea how i still could improve the performance of the following part of a program?
I was thinking if it could be helpful to avoid the deletion at the end of the pasted coding somehow...
DATA gt_knb1 TYPE HASHED TABLE OF t_knb1 WITH UNIQUE KEY bukrs akont.
IF s_bukrs[] IS INITIAL.
    SELECT akont COUNT(*) AS count FROM knb1
      INTO CORRESPONDING FIELDS OF TABLE gt_knb1        "#EC CI_NOFIRST
      GROUP BY akont.                                   "#EC CI_NOWHERE
ELSE.
    SELECT bukrs akont COUNT(*) FROM knb1 INTO TABLE gt_knb1"#EC CI_NOFIRST
      WHERE bukrs IN s_bukrs
      GROUP BY bukrs akont.                             "#EC CI_NOWHERE
ENDIF.
DELETE gt_knb1 WHERE bukrs IN s_bukrs AND akont = space.
Thanks a lot in advance
Best regards
Carsten

Manu D'Haeyer wrote:
Hi,
>
>...The ugliest part for me anyway is this
INTO CORRESPONDING FIELDS OF
... I think a dynamic internal table could help so that gt_knb1 fits exactly to the fetched rows.
>
> kr,
> m.
Hello all,
please note, the "CORRESPONDING FIELDS" is not so relevant nowadays. Few milliseconds are not going to change a picture.
Refer to this thread:
into corresponding fields of table VERSUS into table
Please stop recommending it in EACH AND EVERY thread appearing in this forum.
The correct answer is given by Volker above.
And by the way, are you really sure that your problem is in the DELETE statement? I would actually suspect the SELECT to be a time-consumer.
Regards,
Yuri

Performance degradation in pl/sql parsing

We are trying to use xml pl/sql parser and noticed performance degradation as we run multiple times. We zeroed into the following clause:
doc := xmlparser.getDocument(p);
The first time the procedure is run the elapsed time at sqlplus is something like .45sec, but as we run repeatedly in the same session the elapsed time keeps on increasing by .02 seconds. If we log out and start fresh, we start again from .45sec.
We noticed similar degradation with
p := xmlparser.newParser;
but we got around by making the 'p' variable as package variable, initializing it once and using the same for all invocations.
Any suggestions?
Thank you.

Can I enhance the PL/SQL code for better performance ? Probably you can enhance it.
or, is this OK to take so long to process these many rows? It should take a few minutes, not several hours.
But please provide some more details like your database version etc.
I suggest to TRACE the session that executes the PL/SQL code, with WAIT events, so you'll see where and on what time is spent, you'll identify your 'problem statements very quickly' (after you or your DBA have TKPROF'ed the trace file).
SQL> alter session set events '10046 trace name context forever, level 12';
SQL> execute your PL/SQL code here
SQL> exitWill give you a .trc file in your udump directory on the server.
http://www.oracle-base.com/articles/10g/SQLTrace10046TrcsessAndTkprof10g.php
Also this informative thread can give you more ideas:
HOW TO: Post a SQL statement tuning request - template posting
as well as doing a search on 10046 at AskTom, http://asktom.oracle.com will give you more examples.
and reading Oracle's Performance Tuning Guide: http://www.oracle.com/pls/db102/to_toc?pathname=server.102%2Fb14211%2Ftoc.htm&remark=portal+%28Getting+Started%29

Please help me figure out why this thread's performance degrades over time

Hello.
The code below is from a Runnable that I've tested inside a Thread operating on a TreePath array of size 1500; the array 'clonedArrayB' is this TreePath array. The code is designed to create a more stepped version of setSelectionPaths(TreePath[]) from class JTree.
The performance decrease of the thread is very rapid; this is discernible simply from viewing the println speed. When it gets to about 1400 TreePaths added to the JTree selection, it's running at roughly 1/10 the speed it started running at.
I know there's no problem with maintaining a set of selected paths of that size inside a JTree. I also know that the thread stops when it should. So it must be some operation I'm performing inside the brief piece of code shown below that is causing the performance degradation.
Does anyone have any idea what could be causing the slowdown?
Many thanks for your help. Apologies if you would have liked an SSCCE, but I very much doubt it's necessary for this. Either you can see the problem or you can't. And sadly I can't x:'o(
int indexA = 0;
public void run() {
     // Prevent use of / Pause scanner
     try {
          scannerLock.acquire();
     } catch (InterruptedException exc) {
          Gecko.logException("Scanner lock could not be acquired by expansion thread", exc);
     while (!autoExpansionComplete) {
          while (indexA < clonedArrayA.length) {
               int markerA = indexA + 10;
               for (int a = indexA; a < markerA && a < clonedArrayA.length; a++) {
                    pluginTreeA.addSelectionPath(clonedArrayA[a]);
               indexA = markerA;
               System.out.println(indexA + "," + clonedArrayA.length);
                    if (autoExpansionComplete) {
                         break;
               stop();
};

Well, since I've had no responses, I tried to think of other ways to speed the code up.
I'd already made nearly every tweak I know. The only additional thing I could think of was to use addSelectionPaths(TreePath[]) on a subarray of the cloned array, instead of addSelectionPath(TreePath) on the cloned array's elements, since obviously it would be fewer method calls. It has sped things up an awful lot (my new code is shown below - I've left in some things I chopped out above, so you can see exactly what I see). The problem is though, obviously an increase in initial velocity doesn't solve the problem of deceleration occurring, if you get me.
// Clone the selection arrays to non-volatile arrays for better access
// speeds
final TreePath[] clonedArrayA = selectionPathsArrayA.clone();
final TreePath[] clonedArrayB = selectionPathsArrayB.clone();
// Create a new runnable to perform the selection task
Runnable selectionExpander = new Runnable() {
     /** Position within cloned array A */
     int indexA = 0;
     /** Position within cloned array B */
     int indexB = 0;
     /** Length of subarray grabbed from cloned array A */
     int lengthA;
     /** Length of subarray grabbed from cloned array B */
     int lengthB;
     /** Subarray destination */
     private TreePath[] subarray = new TreePath[100];
     public void stop() {
          autoExpansionComplete = true;
          automatedSelection = false;
          scannerLock.release();
      * Grabs 10 blocks of each selection paths array at a time, adding
      * these to the tree's current selection and then moving to the next
      * cycle
     public void run() {
          // Prevent use of / Pause scanner
          try {
               scannerLock.acquire();
          } catch (InterruptedException exc) {
               Gecko.logException("Scanner lock could not be acquired by expansion thread", exc);
          while (!autoExpansionComplete) {
               while (indexA < clonedArrayA.length || indexB < clonedArrayB.length) {
                    // Set subarray lengths
                    lengthA = subarray.length;
                    lengthB = subarray.length;
                    // If subarray length is greater than the number of
                    // remaining indices in the source array, set length to
                    // the number of remaining indices
                    lengthA = indexA + lengthA > clonedArrayA.length ? clonedArrayA.length - indexA : lengthA;
                    lengthB = indexB + lengthB > clonedArrayB.length ? clonedArrayB.length - indexB : lengthB;
                    // Create subarrays and add TreePath elements to trees'
                    // selections
                    System.arraycopy(clonedArrayA, indexA, subarray, 0, lengthA);
                    pluginTreeA.addSelectionPaths(subarray);
                    System.arraycopy(clonedArrayB, indexB, subarray, 0, lengthB);
                    pluginTreeB.addSelectionPaths(subarray);
                    // Remember the latest index reached in source arrays
                    indexA += lengthA;
                    indexB += lengthB;
                    System.out.println(indexA + "," + clonedArrayA.length);
                    System.out.println(indexB + "," + clonedArrayB.length);
                    if (autoExpansionComplete) {
                         break;
               stop();
// Create and start new thread to manage the selection task runner
selector = new Thread(selectionExpander);
selector.start();I really can't think what could be causing the slowdown. I've done everythng I can think of to increase the velocity, such as cloning the source arrays since they're volatile and access could be slightly slower as a result.
Nothing I try gets rid of the slowdown effect though :(
- Dave

Query performance difference -- CPU time difference on two servers

Two SQL servers, server A & B, Hardware information as below:
Server A: Clustered. Two physical processors (4 cores). RAM 64GB, SQL Server max memory: 58GB.
Server B: Standalone. One physical processor (4 cores). RAM 16GB, SQL Server max memory: 10GB.
The two databases on A & B are identical (using backup & restore method), same query execution CPU time are different, it is slower on the cluster server with more CPU & Mmeory. I tried many times and all same result, I also tried run DBCC DROPCLEANBUFFERS
& DBCC FREEPROCCACHE before running the query.
When I run the query, on Server A (Slow one), CPU usage is at 20% ~30%. RAM usage is 14%. It is new set up environment so just me using it now.
When I run the query on Server B (Fast one), CPU usage is 40%, RAM usage is 85%.
Both two servers have SAN connected storage.
Query execution plan is exactly same on two serers.
Could someone give me some advise how to troubleshoot this problem? Any suggestion is appreciated a lot! Thanks a lot!
FYI, Below are the statistics when running same query in same DB on two servers:
Server A CPU & IO Statistics:
SQL Server parse and compile time:
CPU time = 109 ms, elapsed time = 297 ms.
(1005301 row(s) affected)
Table 'PODT'. Scan count 1, logical reads 55184, physical reads 3, read-ahead reads 55180, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'MRICLDEFENT'. Scan count 0, logical reads 71898, physical reads 3, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'POHD'. Scan count 1, logical reads 103154, physical reads 3, read-ahead reads 103150, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
SQL Server Execution Times:
CPU time = 7769 ms, elapsed time = 16693 ms.
Server B CPU & IO Statistics:
SQL Server parse and compile time:
CPU time = 32 ms, elapsed time = 39 ms.
(1005301 row(s) affected)
Table 'PODT'. Scan count 1, logical reads 55184, physical reads 3, read-ahead reads 55180, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'MRICLDEFENT'. Scan count 0, logical reads 71898, physical reads 3, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'POHD'. Scan count 1, logical reads 103154, physical reads 3, read-ahead reads 103150, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
SQL Server Execution Times:
CPU time = 3931 ms, elapsed time = 14387 ms.

hi, Vivian_Vivian
i just want to ask you
which sql version do you use？
have you ever tried xevent before？？
if you use the sql2k8 or latter you can try xevent out to find somgthing clues about your cpu time problem
i provide you the script，pls post your output in the xevent here so that we could judge what the head of the problem is
USE [master]
GO
CREATE EVENT SESSION [TrackSQLWait] ON SERVER
ADD EVENT sqlserver.sql_statement_starting (
ACTION ( sqlserver.session_id, sqlserver.database_id,sqlserver.sql_text ,package0.collect_system_time)
WHERE ( sqlserver.database_id = 7 ) ), --★Do
ADD EVENT sqlserver.sql_statement_completed (
ACTION ( sqlserver.session_id, sqlserver.database_id, sqlserver.sql_text ,package0.collect_system_time)
WHERE ( sqlserver.database_id = 7 ) ), --★Do
ADD EVENT sqlserver.error_reported (
ACTION ( sqlserver.session_id, sqlserver.database_id, sqlserver.sql_text ,package0.collect_system_time)
WHERE ( sqlserver.database_id = 7 ) ), --★Do
ADD EVENT sqlos.wait_info (
ACTION ( sqlserver.database_id, sqlserver.session_id, sqlserver.sql_text,sqlserver.plan_handle ,package0.collect_system_time)
WHERE
( duration > 1000
AND ( ( wait_type > 31 -- Waits for latches and important wait resources (not locks)
-- that have exceeded 10 seconds.
AND ( ( wait_type > 47
AND wait_type < 54
OR wait_type < 38
OR ( wait_type > 63
AND wait_type < 70
OR ( wait_type > 96
AND wait_type < 100
OR ( wait_type = 107 )
OR ( wait_type = 113 )
OR ( wait_type > 174
AND wait_type < 179
OR ( wait_type = 186 )
OR ( wait_type = 207 )
OR ( wait_type = 269 )
OR ( wait_type = 283 )
OR ( wait_type = 284 )
OR ( duration > 30000 -- Waits for locks that have exceeded 30 secs.
AND wait_type < 22
ADD TARGET package0.asynchronous_file_target ( SET filename = 'E:\ExtendedEvent\TrackSQLWait.xel' ,
metadatafile = 'E:\ExtendedEvent\TrackSQLWait.xem' )
WITH ( MAX_MEMORY = 4 MB ,
EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS )
GO
ALTER EVENT SESSION [TrackSQLWait]
ON SERVER
STATE=START
-- Query the Event data from the Target.
SELECT event_data.value('(event/@name)[1]', 'varchar(50)') AS [event_name] ,
event_data.value('(event/data[@name="source_database_id"]/value)[1]',
'int') AS [source_database_id] ,
OBJECT_NAME(event_data.value('(event/data[@name="object_id"]/value)[1]',
'int')) AS [object] ,
event_data.value('(event/data[@name="object_type"]/value)[1]',
'varchar(60)') AS [object_type] ,
event_data.value('(event/data[@name="state"]/text)[1]', 'varchar(50)') AS [state] ,
event_data.value('(event/data[@name="duration"]/value)[1]', 'bigint') AS [duration] ,
event_data.value('(event/data[@name="cpu"]/value)[1]', 'bigint') AS [cpu] ,
event_data.value('(event/data[@name="reads"]/value)[1]', 'bigint') AS [reads] ,
event_data.value('(event/data[@name="writes"]/value)[1]', 'bigint') AS [writes] ,
event_data.value('(event/data[@name="error"]/value)[1]', 'bigint') AS [error] ,
event_data.value('(event/data[@name="severity"]/value)[1]', 'int') AS [severity] ,
event_data.value('(event/data[@name="user_defined"]/value)[1]',
'varchar(5)') AS [user_defined] ,
event_data.value('(event/data[@name="message"]/value)[1]',
'varchar(max)') AS [message] ,
event_data.value('(event/data[@name="wait_type"]/text)[1]',
'varchar(250)') AS wait_typetype ,
event_data.value('(event/data[@name="wait_type"]/value)[1]',
'varchar(250)') AS wait_typevalue ,
event_data.value('(event/data[@name="max_duration"]/value)[1]',
'bigint') AS max_duration ,
event_data.value('(event/data[@name="completed_count"]/value)[1]',
'bigint') AS [completed_count] ,
event_data.value('(event/action[@name="plan_handle"]/value)[1]',
'varchar(max)') AS [plan_handle] ,
DATEADD(hh, DATEDIFF(hh, GETUTCDATE(), CURRENT_TIMESTAMP),
event_data.value('(event/action[@name="collect_system_time"]/text)[1]',
'datetime2')) AS [system_time] ,
event_data.value('(event/action[@name="session_id"]/value)[1]',
'bigint') AS [session_id] ,
event_data.value('(event/action[@name="sql_text"]/value)[1]',
'varchar(max)') AS [sql_text]
FROM ( SELECT CAST(event_data AS XML) AS event_data
FROM sys.fn_xe_file_target_read_file('E:\ExtendedEvent\TrackSQLWait_*.xel',
'E:\ExtendedEvent\TrackSQLWait_*.xem',
NULL, NULL)
) AS tab
WHERE event_data.value('(event/data[@name="duration"]/value)[1]', 'bigint') > 100
AND event_data.value('(event/@name)[1]', 'varchar(50)') LIKE 'wait_info'
ORDER BY [system_time]
ALTER EVENT SESSION [TrackSQLWait]
ON SERVER
STATE=STOP
GO
DROP EVENT SESSION [TrackSQLWait] ON SERVER
please replacing the related parameter to fit for your situation
thanks

Performance Degradation - High fetches and Prses

Hello,
My analysis on a particular job trace file drew my attention towards:
1) High rate of Parses instead of Bind variables usage.
2) High fetches and poor number/ low number of rows being processed
Please let me kno as to how the performance degradation can be minimised, Perhaps the high number of SQL* Net Client wait events may be due to multiple fetches and transactions with the client.
EXPLAIN PLAN FOR SELECT /*+ FIRST_ROWS (1) */ * FROM SAPNXP.INOB
WHERE MANDT = :A0
AND KLART = :A1
AND OBTAB = :A2
AND OBJEK LIKE :A3 AND ROWNUM <= :A4;
call     count       cpu    elapsed       disk      query    current        rows
Parse      119      0.00       0.00          0          0          0           0
Execute    239      0.16       0.13          0          0          0           0
Fetch      239   2069.31    2127.88          0   13738804          0           0
total      597   2069.47    2128.01          0   13738804          0           0
PLAN_TABLE_OUTPUT
Plan hash value: 1235313998
| Id | Operation                    | Name   | Rows | Bytes | Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT             |        |     2 |   268 |     1   (0)| 00:00:01 |
|* 1 | COUNT STOPKEY               |        |       |       |            |          |
|* 2 |   TABLE ACCESS BY INDEX ROWID| INOB   |     2 |   268 |     1   (0)| 00:00:01 |
|* 3 |    INDEX SKIP SCAN           | INOB~2 | 7514 |       |     1   (0)| 00:00:01 |
Predicate Information (identified by operation id):
   1 - filter(ROWNUM<=TO_NUMBER(:A4))
   2 - filter("OBJEK" LIKE :A3 AND "KLART"=:A1)
   3 - access("MANDT"=:A0 AND "OBTAB"=:A2)
       filter("OBTAB"=:A2)
18 rows selected.
SQL> SELECT INDEX_NAME,TABLE_NAME,COLUMN_NAME FROM DBA_IND_COLUMNS WHERE INDEX_OWNER='SAPNXP' AND INDEX_NAME='INOB~2';
INDEX_NAME      TABLE_NAME                     COLUMN_NAME
INOB~2          INOB                           MANDT
INOB~2          INOB                           CLINT
INOB~2          INOB                           OBTAB
Is it possible to Maximise the rows/fetch
call     count       cpu    elapsed       disk      query    current        rows
Parse      163      0.03       0.00          0          0          0           0
Execute    163      0.01       0.03          0          0          0           0
Fetch   174899     55.26      59.14          0    1387649          0     4718932
total   175225     55.30      59.19          0    1387649          0     4718932
Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 27
Rows     Row Source Operation
28952 TABLE ACCESS BY INDEX ROWID EDIDC (cr=8505 pr=0 pw=0 time=202797 us)
28952   INDEX RANGE SCAN EDIDC~1 (cr=1457 pr=0 pw=0 time=29112 us)(object id 202995)
Elapsed times include waiting on following events:
Event waited on                             Times   Max. Wait Total Waited
----------------------------------------   Waited ---------- ------------
SQL*Net message to client                  174899        0.00          0.16
SQL*Net more data to client                155767        0.01          5.69
SQL*Net message from client                174899        0.11        208.21
latch: cache buffers chains                     2        0.00          0.00
latch free                                      4        0.00          0.00
********************************************************************************

user4566776 wrote:
My analysis on a particular job trace file drew my attention towards:
1) High rate of Parses instead of Bind variables usage.
But if you look at the text you are using bind variables.
The first query is executed 239 times - which matches the 239 fetches. You cut off some of the useful information from the tkprof output, but the figures show that you're executing more than once per parse call. The time is CPU time spent using a bad execution plan to find no data -- this looks like a bad choice of index, possibly a side effect of the first_rows(1) hint.
2) High fetches and poor number/ low number of rows being processedThe second query is doing a lot of fetches because in 163 executions it is fetching 4.7 million rows at roughly 25 rows per fetch. You might improve performance a little by increasing the array fetch size - but probably not by more than a factor of 2.
You'll notice that even though you record 163 parse calls for the second statement the number of " Misses in library cache during parse" is zero - so the parse calls are pretty irrelevant, the cursor is being re-used.
Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk
To post code, statspack/AWR report, execution plans or trace files, start and end the section with the tag {noformat}{noformat} (lowercase, curly brackets, no spaces) so that the text appears in fixed format.
"Science is more than a body of knowledge; it is a way of thinking"
Carl Sagan

Performance problem - mutexes with multi-cpu machine

Hi!
My company is developing multi-threaded server program for
multi-cpu machine which communicates with Oracle database on separate
machine. We use Solaris 7, Workshop 5 CC and pthreads API.
We tested our program on 4 CPU E4500 with 2 CPU E420 Oracle server.
We upgraded E4500 from 4 to 8 CPU and to our surprise instead of
performance improvement we got performance degradation ( 8 CPU runs
about 5% slower than 4 CPU ).
After a long investigation we found out that under stress load most of the
time our performs lwpmutex related operation.
With truss -c statistics was 160 secs in mutex operations and
about 2 secs was read/write in oracle client side library.
Here is output of truss for example:
19989 29075/5: 374.0468 0.0080 lwp_mutex_lock(0x7F2F3F60 = 0
19990 29075/31: 374.0466 0.0006 lwp_mutex_wakeup(0x7F2F3F60) = 0
19991 29075/5: 374.0474 0.0006 lwp_mutex_wakeup(0x7F2F3F60) = 0
19992 29075/30: 374.0474 0.0071 lwp_mutex_lock(0x7F2F3F60) = 0
19993 29075/30: 374.0484 0.0010 lwp_mutex_wakeup(0x7F2F3F60) = 0
19994 29075/31: 374.0483 0.0017 lwp_mutex_lock(0x7F2F3F60) = 0
19995 29075/5: 374.0492 0.0018 lwp_mutex_lock(0x7F2F3F60) = 0
19996 29075/31: 374.0491 0.0008 lwp_mutex_wakeup(0x7F2F3F60) = 0
19997 29075/5: 374.0499 0.0007 lwp_mutex_wakeup(0x7F2F3F60) = 0
19998 29075/30: 374.0499 0.0015 lwp_mutex_lock(0x7F2F3F60) = 0
19999 29075/5: 374.0507 0.0008 lwp_mutex_lock(0x7F2F3F60) = 0
20000 29075/30: 374.0507 0.0008 lwp_mutex_wakeup(0x7F2F3F60) = 0
20001 29075/5: 374.0535 0.0028 lwp_mutex_wakeup(0x7F2F3F60) = 0
20002 29075/30: 374.0537 0.0030 lwp_mutex_wakeup(0x7F2F3F60) = 0
20003 29075/31: 374.0537 0.0046 lwp_mutex_lock(0x7F2F3F60) = 0
20004 29075/5: 374.0547 0.0012 lwp_mutex_lock(0x7F2F3F60) = 0
20005 29075/31: 374.0546 0.0009 lwp_mutex_wakeup(0x7F2F3F60) = 0
20006 29075/5: 374.0554 0.0007 lwp_mutex_wakeup(0x7F2F3F60) = 0
20007 29075/30: 374.0557 0.0020 lwp_mutex_lock(0x7F2F3F60) = 0
20008 29075/31: 374.0555 0.0009 lwp_mutex_wakeup(0x7F2F3F60) = 0
20009 29075/5: 374.0564 0.0010 lwp_mutex_lock(0x7F2F3F60) = 0
20010 29075/30: 374.0564 0.0007 lwp_mutex_wakeup(0x7F2F3F60) = 0
20011 29075/5: 374.0572 0.0008 lwp_mutex_wakeup(0x7F2F3F60) = 0
20012 29075/28: 374.0574 0.0170 lwp_mutex_lock(0x7F2F3F60) = 0
20013 29075/31: 374.0575 0.0020 lwp_mutex_wake(0x7F2F3F60) = 0
We have a several question:
1. We always get the same mutex address : 0x7F2F3F60 even with different
binaries. It looks that all threads wait on one and magic
mutex. Why?
2. We read in article on unixinsider.com that on Solaris when mutex is
unlocked all the threads waiting on this mutex are waked up. It also looks so
from truss output. What is solution for this problem? unixinsider.com
recommends native Solaris read-write lock with all threads as writers?
Is there any other solution? Should in improve performance?
3. We heard that Solaris 8 has better pthreads implementation using a
one-level threading model, where threads are one-to-one with
lwp, rather than the two-level model that is used in the
standard libthread implementation, where user-level threads are
multiplexed over possibly fewer lwps. Are mutexes in this library
behave in "Solaris 7" way or do it put thread to sleep when it unlocks
the mutex? Is it possible to use this library on Solaris 7?
4. Is there plug - in solution like mtmalloc or hoard for new/delete that change
pthread mutexes?
Thank you in advance for your help,
Alexander Indenbaum

<pre>
Hi!
My company is developing multi-threaded server program for
multi-cpu machine which communicates with Oracle database on separate
machine. We use Solaris 7, Workshop 5 CC and pthreads API.
We tested our program on 4 CPU E4500 with 2 CPU E420 Oracle server.
We upgraded E4500 from 4 to 8 CPU and to our surprise instead of
performance improvement we got performance degradation ( 8 CPU runs
about 5% slower than 4 CPU ).
After a long investigation we found out that under stress load most of the
time our performs lwpmutex related operation.
With truss -c statistics was 160 secs in mutex operations and
about 2 secs was read/write in oracle client side library.
Here is output of truss for example:
19989 29075/5: 374.0468 0.0080 lwp_mutex_lock(0x7F2F3F60) = 0
19990 29075/31: 374.0466 0.0006 lwp_mutex_wakeup(0x7F2F3F60) = 0
19991 29075/5: 374.0474 0.0006 lwp_mutex_wakeup(0x7F2F3F60) = 0
19992 29075/30: 374.0474 0.0071 lwp_mutex_lock(0x7F2F3F60) = 0
19993 29075/30: 374.0484 0.0010 lwp_mutex_wakeup(0x7F2F3F60) = 0
19994 29075/31: 374.0483 0.0017 lwp_mutex_lock(0x7F2F3F60) = 0
19995 29075/5: 374.0492 0.0018 lwp_mutex_lock(0x7F2F3F60) = 0
19996 29075/31: 374.0491 0.0008 lwp_mutex_wakeup(0x7F2F3F60) = 0
19997 29075/5: 374.0499 0.0007 lwp_mutex_wakeup(0x7F2F3F60) = 0
19998 29075/30: 374.0499 0.0015 lwp_mutex_lock(0x7F2F3F60) = 0
19999 29075/5: 374.0507 0.0008 lwp_mutex_lock(0x7F2F3F60) = 0
20000 29075/30: 374.0507 0.0008 lwp_mutex_wakeup(0x7F2F3F60) = 0
20001 29075/5: 374.0535 0.0028 lwp_mutex_wakeup(0x7F2F3F60) = 0
20002 29075/30: 374.0537 0.0030 lwp_mutex_wakeup(0x7F2F3F60) = 0
20003 29075/31: 374.0537 0.0046 lwp_mutex_lock(0x7F2F3F60) = 0
20004 29075/5: 374.0547 0.0012 lwp_mutex_lock(0x7F2F3F60) = 0
20005 29075/31: 374.0546 0.0009 lwp_mutex_wakeup(0x7F2F3F60) = 0
20006 29075/5: 374.0554 0.0007 lwp_mutex_wakeup(0x7F2F3F60) = 0
20007 29075/30: 374.0557 0.0020 lwp_mutex_lock(0x7F2F3F60) = 0
20008 29075/31: 374.0555 0.0009 lwp_mutex_wakeup(0x7F2F3F60) = 0
20009 29075/5: 374.0564 0.0010 lwp_mutex_lock(0x7F2F3F60) = 0
20010 29075/30: 374.0564 0.0007 lwp_mutex_wakeup(0x7F2F3F60) = 0
20011 29075/5: 374.0572 0.0008 lwp_mutex_wakeup(0x7F2F3F60) = 0
20012 29075/28: 374.0574 0.0170 lwp_mutex_lock(0x7F2F3F60) = 0
20013 29075/31: 374.0575 0.0020 lwp_mutex_wakeup(0x7F2F3F60) = 0
We have a several question:
1. We always get the same mutex address : 0x7F2F3F60 even with different
binaries. It looks that all threads wait on one and magic
mutex. Why?
2. We read in article on unixinsider.com that on Solaris when mutex is
unlocked all the threads waiting on this mutex are waked up. It also looks so
from truss output. What is solution for this problem? unixinsider.com
recommends native Solaris read-write lock with all threads as writers?
Is there any other solution? Should in improve performance?
3. We heard that Solaris 8 has better pthreads implementation using a
one-level threading model, where threads are one-to-one with
lwp, rather than the two-level model that is used in the
standard libthread implementation, where user-level threads are
multiplexed over possibly fewer lwps. Are mutexes in this library
behave in "Solaris 7" way or do it put thread to sleep when it unlocks
the mutex? Is it possible to use this library on Solaris 7?
4. Is there plug - in solution like mtmalloc or hoard for new/delete that change
pthread mutexes?
Thank you in advance for your help,
Alexander Indenbaum
</pre>

Performance degradation with -g compiler option

Hello
Our mearurement of simple program compiled with and without -g option shows big performance difference.
Machine:
SunOS xxxxx 5.10 Generic_137137-09 sun4u sparc SUNW,Sun-Fire-V250
Compiler:
CC: Sun C++ 5.9 SunOS_sparc Patch 124863-08 2008/10/16
#include "time.h"
#include <iostream>
int main(int argc, char ** argv)
   for (int i = 0 ; i < 60000; i++)
       int *mass = new int[60000];
       for (int j=0; j < 10000; j++) {
           mass[j] = j;
       delete []mass;
   return 0;
}Compilation and execution with -g:
CC -g -o test_malloc_deb.x test_malloc.c
ptime test_malloc_deb.xreal 10.682
user 10.388
sys 0.023
Without -g:
CC -o test_malloc.x test_malloc.c
ptime test_malloc.xreal 2.446
user 2.378
sys 0.018
As you can see performance degradation of "-g" is about 4 times.
Our product is compiled with -g option and before shipment it is stripped using 'strip' utility.
This will give us possibility to open customer core files using non-stripped exe.
But our tests shows that stripping does not give performance of executable compiled without '-g'.
So we are losing performance by using this compilation method.
Is it expected behavior of compiler?
Is there any way to have -g option "on" and not lose performance?

In your original compile you don't use any optimisation flags, which tells the compiler to do minimal optimisation - you're basically telling the compiler that you are not interested in performance. Adding -g to this requests that you want maximal debug. So the compiler does even less optimisation, in order that the generated code more closely resembles the original source.
If you are interested in debug, then -g with no optimisation flags gives you the most debuggable code.
If you are interested in optimised code with debug, then try -O -g (or some other level of optimisation). The code will still be debuggable - you'll be able to map disassembly to lines of source, but some things may not be accessible.
If you are using C++, then -g will in SS12 switch off front-end inlining, so again you'll get some performance hit. So use -g0 to get inlining and debug.
HTH,
Darryl.

Performance Degradation with EJBs

I have a small J2EE application that consists of a Session EJB calling 3 Entity EJBs that access the database. It is a simple Order capture application. The 3 Entity beans are called Orders, OrderItems and Inventory.
A transaction consists of inserting a record into the order table, inserting 5 records into the orderitems table and updating the quantity field in the inventory table for each order item in an order. With this transaction I observe performance degradation as the transactions per second decreases dramatically within 5 minutes of running.
When I modify the transaction to insert a single record into the orderitems table I do not observe performance degradation. The only difference in this transaction is we go through the for loop 1 time as opposed to 5 times. The code is exactly the same as in the previous case with 5 items per order.
Therefore I believe the problem is a performance degradation on Entity EJBs that
get invoked in a loop.
I am using OC4J 10.1.3.3.
I am using CMP (Container Managed Persistence) and CMT (Container Managed Transactions). The Entity EJBs were all generated by Oracle JDeveloper.
EJB version being used is 2.1.

One thing to consider it downloading and using the Oracle AD4J utility to see if it can help you identify any possible bottlenecks, on the application server or the database.
AD4J can be used to monitor/profile/trace applications in real time with no instrumentation required on the application. Just install it into the container and go. It can even trace a request from the app server down into the database and show you the situation is down there (it needs a db agent installed to do that).
Overview:
http://www.oracle.com/technology/products/oem/pdf/wp_productionappdiagnostics.pdf
Download:
http://www.oracle.com/technology/software/products/oem/htdocs/jade.html
Install/Config Guide:
http://download.oracle.com/docs/cd/B16240_01/doc/install.102/e11085/toc.htm
Usage Scenarios:
http://www.oracle.com/technology/products/oem/pdf/oraclead4j_usagescenarios.pdf

Performance degradation of Weblogic 5.1 sp 6 bundled with Peoplesoft 8.1.2:

Recenly we have upgraded from Peoplesoft 7 to Peoplesoft 8.1.2
Peoplesoft 8.1.2 bundled with Peopletools(Web based front end ) for the first
time and Weblogic 5.1 sp6.
There is performance degradation of the weblogic 5.1 sp 6 (on Windows 2000 ) when
the no of users increases to 80. The Weblogic is becoming 100% CPU bound. Besides
the weblogic wont even shutdown completely when trying to shutdown.
Peoplesoft customer support advised to upgrade to Weblogic 5.1 sp 9 but sp 9 wont
support 128 bit encription which Peoplesoft 8.1.2 application need. Peoplesoft
8.1.3 will be supporting 128 bit encription after some 3 months. We have to get
along with the above mentioned configuration (Peoplesoft 8.1.2 with Weblogic 5.1
sp 9) in the mean time.
Any of you had such an experience ? Please let me know if there is a solution
or workaround.
Thanks in advance.
Mani

There shouldn't be any reason that 5.1 SP9 wouldn't support 128 bit
encryption. If that's the issue, you should post in the security
newsgroup or contact [email protected]
-- Rob
Mani Ayyalas wrote:
Recenly we have upgraded from Peoplesoft 7 to Peoplesoft 8.1.2
Peoplesoft 8.1.2 bundled with Peopletools(Web based front end ) for the first
time and Weblogic 5.1 sp6.
There is performance degradation of the weblogic 5.1 sp 6 (on Windows 2000 ) when
the no of users increases to 80. The Weblogic is becoming 100% CPU bound. Besides
the weblogic wont even shutdown completely when trying to shutdown.
Peoplesoft customer support advised to upgrade to Weblogic 5.1 sp 9 but sp 9 wont
support 128 bit encription which Peoplesoft 8.1.2 application need. Peoplesoft
8.1.3 will be supporting 128 bit encription after some 3 months. We have to get
along with the above mentioned configuration (Peoplesoft 8.1.2 with Weblogic 5.1
sp 9) in the mean time.
Any of you had such an experience ? Please let me know if there is a solution
or workaround.
Thanks in advance.
Mani

Performance degradation encountered while running BOE in clustered set up

Problem Statement:
We have a clustered BOE set up in Production with 2 CMS servers (named boe01 and boe02) . Mantenix application (Standard J2EE application in a clustered set up) points to these BOE services hosted on virtual machines to generate reports. As soon as BOE services on both boe01 and boe02 are up and running , performance degradation is observed i.e (response times varies from 7sec to 30sec) .
The same set up works fine when BOE services on boe02 is turned off i.e only boe01 is up and running.No drastic variation is noticed.
BOE Details : SAP BusinessObjects environment XIR2 SP3 running on Windows 2003 Servers.(Virtual machines)
Possible Problem Areas as per our analysis
1) Node 2 Virtual Machine Issue:
This currently being part of the Production infrastructure, any problem assessment testing is not possible.
2) BOE Configuration Issue
Comparison report to check the build between BOE 01 and BOE 02 - Support team has confirmed no major installation differences apart from a minor Operating System setting difference.Question being is there some configuration/setting that we are missing ?
3) Possible BOE Cluster Issue:
Tests in staging environment ( with a similar clustered BOE setup ) have proved inconclusive.
We require your help in
- Root cause Analysis for this problem.
- Any troubleshooting action henceforth.
Another observation from our Weblogic support engineers for the above set up which may or may not be related to the problem is mentioned below.
When the services on BOE_2 are shutdown and we try to fetch a particular report from BOE_1 (Which is running), the following WARNING/ERROR comes up:-
07/09/2011 10:22:26 AM EST> <WARN> <com.crystaldecisions.celib.trace.d.if(Unknown Source)> - getUnmanagedService(): svc=BlockingReportSourceRepository,spec=aps<BOE_1> ,cluster:@BOE_OLTP, kind:cacheserver, name:<BOE_2>.cacheserver.cacheserver, queryString:null, m_replaceable:true,uri=osca:iiop://<BOE_1>;SI_SESSIONID=299466JqxiPSPUTef8huXO
com.crystaldecisions.thirdparty.org.omg.CORBA.TRANSIENT: attempt to establish connection failed: java.net.ConnectException: Connection timed out: connect minor code: 0x4f4f0001 completed: No
     at com.crystaldecisions.thirdparty.com.ooc.OCI.IIOP.Connector_impl.connect(Connector_impl.java:150)
     at com.crystaldecisions.thirdparty.com.ooc.OB.GIOPClient.createTransport(GIOPClient.java:233)
     at com.crystaldecisions.thirdparty.com.ooc.OB.GIOPClientWorkersPool.next(GIOPClientWorkersPool.java:122)
     at com.crystaldecisions.thirdparty.com.ooc.OB.GIOPClient.getWorker(GIOPClient.java:105)
     at com.crystaldecisions.thirdparty.com.ooc.OB.GIOPClient.startDowncall(GIOPClient.java:409)
     at com.crystaldecisions.thirdparty.com.ooc.OB.Downcall.preMarshalBase(Downcall.java:181)
     at com.crystaldecisions.thirdparty.com.ooc.OB.Downcall.preMarshal(Downcall.java:298)
     at com.crystaldecisions.thirdparty.com.ooc.OB.DowncallStub.preMarshal(DowncallStub.java:250)
     at com.crystaldecisions.thirdparty.com.ooc.OB.DowncallStub.setupRequest(DowncallStub.java:530)
     at com.crystaldecisions.thirdparty.com.ooc.CORBA.Delegate.request(Delegate.java:556)
     at com.crystaldecisions.thirdparty.org.omg.CORBA.portable.ObjectImpl._request(ObjectImpl.java:118)
     at com.crystaldecisions.enterprise.ocaframework.idl.ImplServ._OSCAFactoryStub.getServices(_OSCAFactoryStub.java:806)
     at com.crystaldecisions.enterprise.ocaframework.ServiceMgr.do(Unknown Source)
     at com.crystaldecisions.enterprise.ocaframework.ServiceMgr.a(Unknown Source)
     at com.crystaldecisions.enterprise.ocaframework.ServiceMgr.getUnmanagedService(Unknown Source)
     at com.crystaldecisions.enterprise.ocaframework.AbstractStubHelper.getService(Unknown Source)
     at com.crystaldecisions.enterprise.ocaframework.e.do(Unknown Source)
     at com.crystaldecisions.enterprise.ocaframework.o.try(Unknown Source)
     at com.crystaldecisions.enterprise.ocaframework.o.a(Unknown Source)
     at com.crystaldecisions.enterprise.ocaframework.o.a(Unknown Source)
     at com.crystaldecisions.enterprise.ocaframework.p.a(Unknown Source)
     at com.crystaldecisions.enterprise.ocaframework.ServiceMgr.getManagedService(Unknown Source)
     at com.crystaldecisions.sdk.occa.managedreports.ps.internal.a$a.getService(Unknown Source)
     at com.crystaldecisions.enterprise.ocaframework.e.do(Unknown Source)
     at com.crystaldecisions.enterprise.ocaframework.o.try(Unknown Source)
     at com.crystaldecisions.enterprise.ocaframework.o.a(Unknown Source)
     at com.crystaldecisions.enterprise.ocaframework.o.a(Unknown Source)
     at com.crystaldecisions.enterprise.ocaframework.p.a(Unknown Source)
We see the above warning coming 2 or 3 times before the request is processed and then we see the report. We have checked our config's for the cluster but didn't find anything concrete.
Is this a normal behavior of the software or can we optimize it?
Any assistance that you can provide would be great

Rahul,
I have exactly the same problem running BO 3.1 SP3 in a 2 machine cluster on AIX. Exact same full install on both machines. When I take down one of the machines the performance is much better.
An example of the problem now is that when i run the command ./ccm.sh -display -username administrator -password xxx on the either box when they are both up and running, I sometimes receive a timeout error (over 15mins)
If I run SQLplus direct on the boxes to the CMS DB then the response is instant. Tnspings of course shows no problems
When I bring down one of the machines and run the command ./ccm.sh -display again then this brings back results in less than a minute...
I am baffled as to the problem so was wondering if you found anything from your end
Cheers
Chris

Performance degradation using Jolt ASP Connectivity for TUXEDO

We have a customer that uses Jolt ASP Connectivity for TUXEDO and is suffering
from a severe performance degradation over time.
Initial response times are fine (1 s.), but they tend to increase to 3 minutes
after some time (well, eh, a day or so).
Data:
- TUXEDO 7.1
- Jolt 1.2.1
- Relatively recent rolling patch installed (so no there are probably no JSH performance
issues and memory leaks as fixed in earlier patches)
The ULOG shows that during the night the JSH instances notice a timeout on behalf
of the client connection and do a forced shutdown of the client:
040911.csu013.cs.kadaster.nl!JSH.234333.1.-2: JOLT_CAT:1185: "INFO: Userid:
[ZZ_Webpol], Clientid: [AP_WEBSRV3] timed out due to inactivity"
040911.csu013.cs.kadaster.nl!JSH.234333.1.-2: JOLT_CAT:1198: "WARN: Forced
shutdown of client; user name 'ZZ_Webpol'; client name 'AP_WEBSRV3'"
This happens every 10 minutes as per configuration of the JSL (-T flag).
The customer "solved" the problem for the time being by increasing the connection
pool size on the IIS web server.
However, they didn't find a "smoking gun" - no definite cause for the problem.
So, it is debatable whether their "solution" suffices.
It is my suspicion the problem might be located in the Jolt ASP classes running
on the IIS.
Maybe the connection pool somehow loses connections over time, causing subsequent
users having to queue before they get served (although an exception should be
raised if no connections are available).
However, there's no documentation on the functioning of the connection pool for
Jolt ASP.
My questions:
1) What's the algorithm used for managing connections with Jolt ASP for TUXEDO?
2) If connections are terminated by a JSH, will a new connection be established
from the web server automatically? (this is especially interesting, because the
connection policy can be configured in the JSL CLOPT, but there's no info on how
this should be handled/configured by Jolt ASP connectivity for TUXEDO)
Regards,
Winfried Scheulderman

Hi,
For ASP connectivity I would suggest looking at the .Net client facility provided in Tuxedo 9.1 and later.
Regards,
Todd Little
Oracle Tuxedo Chief Architect

Performance degradation of Weblogic 5.1 sp 6 when used with Peoplesoft 8

Recenly we have upgraded from Peoplesoft 7 to Peoplesoft 8.
There is performance degradation of the weblogic 5.1 sp 6 (on Windows 2000 ) when
the no of users increases to 2000. Besides the weblogic wont even shutdown completely
when trying to shutdown.
Weblogic customer support advised to upgrade to sp 8 but sp 8 wont support 128
bit encription which peoplesoft 8 need.
Any of you had such an experience ? Please let me know if there is a solution
or workaround.
Thanks in advance.
Mani

There shouldn't be any reason that 5.1 SP9 wouldn't support 128 bit
encryption. If that's the issue, you should post in the security
newsgroup or contact [email protected]
-- Rob
Mani Ayyalas wrote:
Recenly we have upgraded from Peoplesoft 7 to Peoplesoft 8.1.2
Peoplesoft 8.1.2 bundled with Peopletools(Web based front end ) for the first
time and Weblogic 5.1 sp6.
There is performance degradation of the weblogic 5.1 sp 6 (on Windows 2000 ) when
the no of users increases to 80. The Weblogic is becoming 100% CPU bound. Besides
the weblogic wont even shutdown completely when trying to shutdown.
Peoplesoft customer support advised to upgrade to Weblogic 5.1 sp 9 but sp 9 wont
support 128 bit encription which Peoplesoft 8.1.2 application need. Peoplesoft
8.1.3 will be supporting 128 bit encription after some 3 months. We have to get
along with the above mentioned configuration (Peoplesoft 8.1.2 with Weblogic 5.1
sp 9) in the mean time.
Any of you had such an experience ? Please let me know if there is a solution
or workaround.
Thanks in advance.
Mani

Performance Degradated Possibly due to CPU Time

Similar Messages

Maybe you are looking for