Database Hung
Hi,
During running GLG report ,DB server load bumped up to 50. I can't connect thru sqlplus.Application page were down. Then I killed smon and pmon to bring instance down ,reboot the server and started db & apps.
Does any one had such situation? If yes please suggest fix.
Linux AS 4.4
Oracle 10.2.0.3
Application 11.5.10.2
One thing I noticed load and IO (40) was crasy during crash.What area I need to focus?
Do the following.
If you can able to get the GL report SQL statement then enable sql tracing run it, and check the trace file.
AFAIK there might be some locking issue when you run the GL report, try to run off peak time and check does db hung?
Edit: Sorry for IM Speak, now it is corrected.
Edited by: Mohammed Taj on May 10, 2009 7:17 PM
Similar Messages
-
ROW CACHE ENQUEUE LOCK/ibrary cache load lock leads to database hung
(lowercase, curly brackets, no spaces)
We faced database hung on 3 node 11i erp 9i rac database.
We saw the library cache load lock timed out events reported in alert log.
Then few ora-600 and later ROW CACHE ENQUEUE LOCK timed out event. Eventually database was hung and we had to bounce the services .
we created support sr 7845542.992 for RCA.
The support says to increase shared pool size to avoid shared pool fragmentation and avoid reload ,additionaly to upgrade to 10g database.
I am not covinced adding additional pool size would solve this or upgrade to 10 .furthermore even 10g has such issues reported.
I saw couple of bugs mentioned such issue can happen due deadlock of session holding latches .
kindly let me know your view on issue
If required i can attach statspack for more information. (lowercase, curly brackets, no spaces)Many Thanks, i was keen to have your update .
There are 8 cpus on each node . Reloads very high during time period ,but normally there are not high reloads.
Statspack details for 3 nodes
STATSPACK report for
DB Name DB Id Instance Inst Num Release Cluster Host
PROD 21184234 PROD1 1 9.2.0.8.0 YES npi-or-db-p-
11.npi.corp
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 149817 30-Oct-09 13:00:09 574 #########
End Snap: 149837 30-Oct-09 14:00:17 602 #########
Elapsed: 60.13 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 8,192M Std Block Size: 8K
Shared Pool Size: 1,024M Log Buffer: 10,240K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 122,414.93 11,449.13
Logical reads: 69,550.76 6,504.89
Block changes: 928.41 86.83
Physical reads: 196.24 18.35
Physical writes: 28.65 2.68
User calls: 343.97 32.17
Parses: 558.61 52.25
Hard parses: 43.48 4.07
Sorts: 467.24 43.70
Logons: 0.63 0.06
Executes: 2,046.99 191.45
Transactions: 10.69
% Blocks changed per Read: 1.33 Recursive Call %: 97.59
Rollback per transaction %: 5.07 Rows per Sort: 15.85
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 99.72 In-memory Sort %: 100.00
Library Hit %: 96.79 Soft Parse %: 92.22
Execute to Parse %: 72.71 Latch Hit %: 99.77
Parse CPU to Parse Elapsd %: 60.10 % Non-Parse CPU: 78.07
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
db file sequential read 249,234 0 1,537 6 6.5
db file scattered read 61,776 0 769 12 1.6
row cache lock 780,098 10 566 1 20.2
library cache lock 697,849 157 432 1 18.1
latch free 127,926 4,715 387 3 3.3
global cache cr request 370,770 3,091 309 1 9.6
PL/SQL lock timer 59 58 112 1903 0.0
wait for scn from all nodes 303,572 18 103 0 7.9
library cache pin 26,231 2 100 4 0.7
global cache null to x 17,717 716 92 5 0.5
buffer busy waits 5,388 18 74 14 0.1
db file parallel read 5,245 0 69 13 0.1
log file sync 20,407 29 66 3 0.5
enqueue 52,200 70 60 1 1.4
buffer busy global CR 4,845 33 55 11 0.1
CGS wait for IPC msg 412,512 407,106 50 0 10.7
ksxr poll remote instances 1,279,565 483,046 48 0 33.2
log file parallel write 160,040 0 42 0 4.1
library cache load lock 1,491 2 29 20 0.0
global cache open x 19,507 344 28 1 0.5
buffer busy global cache 957 0 22 23 0.0
global cache s to x 16,516 180 20 1 0.4
db file parallel write 11,120 0 12 1 0.3
log file sequential read 618 0 11 18 0.0
DFS lock handle 23,768 0 10 0 0.6
control file sequential read 8,563 0 4 0 0.2
KJC: Wait for msg sends to c 1,549 57 4 3 0.0
lock escalate retry 76 76 4 52 0.0
SQL*Net break/reset to clien 12,546 0 3 0 0.3
SQL*Net more data to client 85,773 0 3 0 2.2
control file parallel write 1,265 0 2 1 0.0
global cache null to s 648 23 1 2 0.0
global cache busy 200 0 1 5 0.0
global cache open s 1,493 28 1 1 0.0
log file switch completion 12 0 1 61 0.0
PX Deq Credit: send blkd 161 70 1 4 0.0
kksfbc child completion 119 118 1 5 0.0
PX Deq: reap credit 5,948 5,456 0 0 0.2
PX Deq: Execute Reply 83 29 0 3 0.0
process startup 8 0 0 25 0.0
LGWR wait for redo copy 992 12 0 0 0.0
IPC send completion sync 450 450 0 0 0.0
PX Deq: Parse Reply 100 28 0 1 0.0
undo segment extension 10,380 10,372 0 0 0.3
PX Deq: Join ACK 146 65 0 1 0.0
buffer deadlock 222 221 0 0 0.0
async disk IO 1,179 0 0 0 0.0
wait list latch free 2 0 0 16 0.0
PX Deq: Msg Fragment 112 28 0 0 0.0
Library Cache Activity for DB: PROD Instance: PROD1 Snaps: 149817 -149837
->"Pct Misses" should be very low
Get Pct Pin Pct Invali-
Namespace Requests Miss Requests Miss Reloads dations
BODY 116,007 1.1 133,347 19.9 24,338 0
CLUSTER 4,224 0.6 5,131 1.0 0 0
INDEX 15,048 24.1 13,798 26.4 2 0
JAVA DATA 82 0.0 692 39.6 136 0
JAVA RESOURCE 66 39.4 206 25.2 12 0
PIPE 1,140 0.5 1,160 0.5 0 0
SQL AREA 1,197,908 12.6 13,517,660 1.5 111,833 73
TABLE/PROCEDURE 3,847,439 0.8 4,230,265 7.9 142,200 0
TRIGGER 8,444 2.4 8,657 18.5 1,274 0
GES Lock GES Pin GES Pin GES Inval GES Invali-
Namespace Requests Requests Releases Requests dations
BODY 1 1,234 1,258 985 0
CLUSTER 3,222 25 25 25 0
INDEX 13,792 3,641 3,631 3,629 0
JAVA DATA 0 0 0 0 0
JAVA RESOURCE 0 26 25 0 0
PIPE 0 0 0 0 0
SQL AREA 0 0 0 0 0
TABLE/PROCEDURE 857,137 13,130 13,264 10,762 0
TRIGGER 0 200 202 200 0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
STATSPACK report for
DB Name DB Id Instance Inst Num Release Cluster Host
PROD 21184234 PROD2 2 9.2.0.8.0 YES npi-or-db-p-
12.npi.corp
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 149847 30-Oct-09 14:00:05 493 #########
End Snap: 149857 30-Oct-09 15:00:02 432 #########
Elapsed: 59.95 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 8,192M Std Block Size: 8K
Shared Pool Size: 1,024M Log Buffer: 10,240K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 71,853.44 32,058.65
Logical reads: 273,904.84 122,207.36
Block changes: 889.13 396.70
Physical reads: 40.40 18.03
Physical writes: 20.97 9.35
User calls: 153.74 68.60
Parses: 66.19 29.53
Hard parses: 2.66 1.19
Sorts: 25.70 11.47
Logons: 0.16 0.07
Executes: 726.41 324.10
Transactions: 2.24
% Blocks changed per Read: 0.32 Recursive Call %: 92.41
Rollback per transaction %: 4.84 Rows per Sort: 193.55
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 99.99
Buffer Hit %: 99.99 In-memory Sort %: 100.00
Library Hit %: 99.35 Soft Parse %: 95.97
Execute to Parse %: 90.89 Latch Hit %: 99.99
Parse CPU to Parse Elapsd %: 36.55 % Non-Parse CPU: 98.28
Wait Events for DB: PROD Instance: PROD2 Snaps: 149847 -149857
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
enqueue 65,823 33,667 90,459 1374 8.2
row cache lock 38,996 560 1,795 46 4.8
PX Deq Credit: send blkd 522 499 1,223 2344 0.1
PX Deq: Parse Reply 466 416 987 2117 0.1
db file sequential read 50,130 0 421 8 6.2
library cache lock 78,842 172 210 3 9.8
db file scattered read 6,904 0 152 22 0.9
global cache cr request 84,801 575 113 1 10.5
latch free 8,096 736 65 8 1.0
log file sync 5,676 27 41 7 0.7
wait for scn from all nodes 18,891 10 24 1 2.3
CGS wait for IPC msg 394,678 392,142 21 0 49.0
library cache pin 1,339 0 17 13 0.2
global cache null to x 2,145 48 16 8 0.3
global cache s to x 3,242 32 16 5 0.4
buffer busy waits 366 10 15 40 0.0
ksxr poll remote instances 70,990 31,295 14 0 8.8
db file parallel read 359 0 11 31 0.0
global cache open x 2,708 55 10 4 0.3
async disk IO 3,474 0 8 2 0.4
global cache open s 3,470 10 6 2 0.4
log file parallel write 13,076 0 5 0 1.6
global cache busy 58 40 5 90 0.0
PL/SQL lock timer 1 1 5 4877 0.0
DFS lock handle 3,362 0 5 1 0.4
log file sequential read 412 0 4 10 0.1
db file parallel write 2,774 0 3 1 0.3
library cache load lock 59 0 3 58 0.0
buffer busy global CR 722 0 3 4 0.1
control file sequential read 6,398 0 3 0 0.8
SQL*Net break/reset to clien 16,078 0 2 0 2.0
name-service call wait 26 0 2 67 0.0
control file parallel write 1,248 0 2 1 0.2
process startup 24 0 1 49 0.0
KJC: Wait for msg sends to c 3,491 4 1 0 0.4
SQL*Net more data to client 23,724 0 1 0 2.9
buffer busy global cache 23 0 0 19 0.0
global cache null to s 114 0 0 4 0.0
PX Deq: reap credit 5,646 5,509 0 0 0.7
log file switch completion 4 0 0 58 0.0
lock escalate retry 54 54 0 1 0.0
IPC send completion sync 119 118 0 0 0.0
direct path read 2,820 0 0 0 0.3
direct path read (lob) 3,632 0 0 0 0.5
PX Deq: Join ACK 88 37 0 0 0.0
direct path write 2,470 0 0 0 0.3
kksfbc child completion 6 6 0 6 0.0
buffer deadlock 3 3 0 11 0.0
global cache quiesce wait 4 4 0 8 0.0
Library Cache Activity for DB: PROD Instance: PROD2 Snaps: 149847 -149857
->"Pct Misses" should be very low
Get Pct Pin Pct Invali-
Namespace Requests Miss Requests Miss Reloads dations
BODY 27,353 0.5 28,091 6.5 1,643 0
CLUSTER 203 1.0 269 1.5 0 0
INDEX 526 9.9 271 19.9 0 0
JAVA DATA 18 0.0 120 6.7 4 0
JAVA RESOURCE 20 45.0 56 26.8 3 0
JAVA SOURCE 1 100.0 1 100.0 0 0
PIPE 999 0.4 1,043 0.4 0 0
SQL AREA 131,793 7.6 3,406,577 0.4 7,012 0
TABLE/PROCEDURE 926,987 0.2 1,907,993 1.0 8,845 0
TRIGGER 1,519 0.1 1,532 4.9 69 0
GES Lock GES Pin GES Pin GES Inval GES Invali-
Namespace Requests Requests Releases Requests dations
BODY 1 129 277 117 0
CLUSTER 168 2 2 2 0
INDEX 271 52 56 52 0
JAVA DATA 0 0 0 0 0
JAVA RESOURCE 0 9 6 0 0
JAVA SOURCE 0 1 1 1 0
PIPE 0 0 0 0 0
SQL AREA 0 0 0 0 0
TABLE/PROCEDURE 89,523 764 868 460 0
TRIGGER 0 2 14 2 0
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DB Name DB Id Instance Inst Num Release Cluster Host
PROD 21184234 PROD3 3 9.2.0.8.0 YES npi-or-db-p-
13.npi.corp
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 149808 30-Oct-09 14:00:00 31 #########
End Snap: 149809 30-Oct-09 15:00:02 34 11,831.4
Elapsed: 60.03 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 8,192M Std Block Size: 8K
Shared Pool Size: 1,024M Log Buffer: 10,240K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 1,518.14 36,700.35
Logical reads: 1,333.43 32,235.02
Block changes: 5.09 123.01
Physical reads: 54.31 1,312.88
Physical writes: 3.91 94.44
User calls: 1.46 35.40
Parses: 2.24 54.21
Hard parses: 0.04 0.93
Sorts: 0.84 20.28
Logons: 0.06 1.45
Executes: 3.11 75.23
Transactions: 0.04
% Blocks changed per Read: 0.38 Recursive Call %: 94.31
Rollback per transaction %: 45.64 Rows per Sort: 215.97
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.99 Redo NoWait %: 100.00
Buffer Hit %: 96.21 In-memory Sort %: 100.00
Library Hit %: 99.07 Soft Parse %: 98.29
Execute to Parse %: 27.94 Latch Hit %: 99.98
Parse CPU to Parse Elapsd %: 69.88 % Non-Parse CPU: 97.92
Wait Events for DB: PROD Instance: PROD3 Snaps: 149808 -149809
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
enqueue 19,510 7,472 15,509 795 130.9
PX Deq: Parse Reply 1,152 1,071 2,577 2237 7.7
row cache lock 2,202 518 1,579 717 14.8
db file scattered read 31,556 0 354 11 211.8
db file sequential read 17,272 0 67 4 115.9
db file parallel read 1,722 0 34 20 11.6
global cache cr request 53,754 91 32 1 360.8
wait for scn from all nodes 1,897 13 10 5 12.7
CGS wait for IPC msg 403,358 401,478 10 0 2,707.1
DFS lock handle 4,753 0 8 2 31.9
direct path read 1,248 0 6 5 8.4
PX Deq: Execute Reply 110 38 6 51 0.7
global cache open s 160 10 5 31 1.1
control file sequential read 6,442 0 3 0 43.2
name-service call wait 26 0 2 78 0.2
latch free 129 109 2 13 0.9
KJC: Wait for msg sends to c 153 24 1 9 1.0
control file parallel write 1,245 0 1 1 8.4
buffer busy waits 199 0 1 6 1.3
process startup 20 0 1 44 0.1
global cache null to x 74 2 1 9 0.5
global cache null to s 19 0 1 29 0.1
global cache open x 268 1 1 2 1.8
library cache lock 1,150 0 0 0 7.7
PX Deq: Join ACK 129 48 0 3 0.9
log file parallel write 1,157 0 0 0 7.8
async disk IO 219 0 0 1 1.5
direct path write 1,024 0 0 0 6.9
ksxr poll remote instances 6,740 4,595 0 0 45.2
PX Deq: reap credit 6,580 6,511 0 0 44.2
buffer busy global CR 73 0 0 2 0.5
log file sequential read 11 0 0 10 0.1
log file sync 100 0 0 1 0.7
global cache s to x 282 2 0 0 1.9
db file parallel write 95 0 0 1 0.6
library cache pin 142 0 0 0 1.0
SQL*Net break/reset to clien 28 0 0 1 0.2
IPC send completion sync 81 81 0 0 0.5
PX Deq: Signal ACK 32 14 0 1 0.2
PX Deq Credit: send blkd 3 1 0 7 0.0
SQL*Net more data to client 841 0 0 0 5.6
PX Deq: Msg Fragment 37 17 0 0 0.2
log file single write 4 0 0 1 0.0
db file single write 1 0 0 1 0.0
SQL*Net message from client 4,213 0 13,673 3246 28.3
gcs remote message 214,784 75,745 7,016 33 1,441.5
wakeup time manager 233 233 6,812 29237 1.6
PX Idle Wait 2,338 2,294 5,686 2432 15.7
PX Deq: Execution Msg 2,151 1,979 4,796 2229 14.4
Library Cache Activity for DB: PROD Instance: PROD3 Snaps: 149808 -149809
->"Pct Misses" should be very low
Get Pct Pin Pct Invali-
Namespace Requests Miss Requests Miss Reloads dations
BODY 1,290 0.0 1,290 0.0 0 0
CLUSTER 18 0.0 8 0.0 0 0
SQL AREA 4,893 2.0 36,371 0.5 2 0
TABLE/PROCEDURE 1,555 3.9 3,834 4.9 71 0
TRIGGER 286 0.0 286 0.0 0 0
GES Lock GES Pin GES Pin GES Inval GES Invali-
Namespace Requests Requests Releases Requests dations
BODY 1 0 0 0 0
CLUSTER 4 0 0 0 0
SQL AREA 0 0 0 0 0
TABLE/PROCEDURE 863 224 42 42 0
TRIGGER 0 0 0 0 0
------------------------------------------------------------- -
All possible causes of database hung
Hi,
Can someone please tell me what are all the possible causes of database hung?
Thanks.
CreemsAlready discussed threads.
http://forums.oracle.com/forums/search.jspa?threadID=&q=database+hang&objID=f61&dateRange=lastyear&userID=&numResults=15
Regards,
Sabdar Syed. -
Compiled trigger causes database hung
Hi there,
I compile some triggers in our database and they cause our database hung, how can I do ? any hint? ThxThat is ugly - calling a remote proc from inside a trigger. And the wrong thing to do in most cases.
A trigger is there to protect the integrity of the data and transaction. A remote proc call can fail for numerous reasons. Network issues. Remote db is in restricted mode/down. Etc. These will cause the trigger to fail. The trigger failure will cause the business transaction to fail.
And I'm pretty sure that no business transaction and validation and processing logic dictates that it must fail because of sonmething like a network problem if it can be prevented.
If this is a replication issue, you should consider using Oracle's built-in replication instead.
Anything else - I would rather have the trigger inserting instructions (into a table) for a background (DBMS_JOB) process to pick up and execute. -
Database Hung problem in Oracle9i Enterprise Edition Release 9.2.0.7.0
Hello,
I am using the Oracle9i Enterprise Edition Release 9.2.0.7.0. Our database suddenly get hung and we need to give a Hard reset.
Following are errors are in listener log.
TNS-12571: TNS:packet writer failure
27-OCT-2008 00:33:00 * service_died * nm45 * 12571
TNS-12571: TNS:packet writer failure
27-OCT-2008 00:33:13 * CONNECT_DATA=(SERVICE_NAME=<Service>)(SID=NM45)(SDU=2048)(CID=(PROGRAM=<EXE>)(HOST=<Host>(USER=<User>))) * (ADDRESS=(PROTOCOL=tcp)(HOST=<IP>)(PORT=<Port>)) * establish * <DB> * 12560
TNS-12500: TNS:listener failed to start a dedicated server process
TNS-12537: TNS:connection closed
TNS-12560: TNS:protocol adapter error
TNS-00507: Connection closed
32-bit Windows Error: 109: Unknown error
Error in Oracle trace:
ORACLE V9.2.0.7.0 - Production vsnsta=0
vsnsql=12 vsnxtr=3
Windows 2000 Version 5.2 Service Pack 1, CPU type 586
Oracle9i Enterprise Edition Release 9.2.0.7.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.7.0 - Production
Windows 2000 Version 5.2 Service Pack 1, CPU type 586
Instance name: <SID>
Redo thread mounted by this instance: 1
Oracle process number: 30
Windows thread id: 1172, image: ORACLE.EXE
*** 2008-10-27 00:54:04.300
async error encountered when answering new connection:
NS Primary Error: TNS-12560: TNS:protocol adapter error
*** 2008-10-27 01:14:38.175
async error encountered when answering new connection:
NS Primary Error: TNS-12560: TNS:protocol adapter error
async error encountered when answering new connection:
NS Primary Error: TNS-12560: TNS:protocol adapter error
*** 2008-10-27 01:22:48.894
async error encountered when answering new connection:
NS Primary Error: TNS-12560: TNS:protocol adapter error
async error encountered when answering new connection:
NS Primary Error: TNS-12560: TNS:protocol adapter error
*** 2008-10-27 01:39:21.800
async error encountered when answering new connection:
NS Primary Error: TNS-12560: TNS:protocol adapter error
*** 2008-10-27 01:50:40.910
async error encountered when answering new connection:
NS Primary Error: TNS-12560: TNS:protocol adapter error
*** 2008-10-27 02:39:51.285
async error encountered when answering new connection:
NS Primary Error: TNS-12560: TNS:protocol adapter error
What could be the reason.
Regards
HemantHello Hemant,
Thanks for the response. Following error are in sqlnet.log. We are using the static id.
Fatal NI connect error 12560, connecting to:
(LOCAL=NO)
VERSION INFORMATION:
TNS for 32-bit Windows: Version 9.2.0.7.0 - Production
Oracle Bequeath NT Protocol Adapter for 32-bit Windows: Version 9.2.0.7.0 - Production
Windows NT TCP/IP NT Protocol Adapter for 32-bit Windows: Version 9.2.0.7.0 - Production
Time: 27-OCT-2008 00:33:13
Tracing not turned on.
Tns error struct:
ns main err code: 12560
TNS-12560: TNS:protocol adapter error
ns secondary err code: 0
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
Regards
Hemant. -
Primary RAC database hung on sqlplus as sysdba, dgmgrl and rman target
I need urgent help here, our qa environment is two nodes RAC primary, and two nodes standby.
Standby environment is ok, so far works, but no entries added to alert log.
however on primary rac, i cannot sqlplus to the db, and rman target hung, and dgmgrl hung.
I tried to srvctl stop database -o abort, it just hung there.
please help me to get that resolved.
Thanks in advance.982335 wrote:
I need urgent help here, our qa environment is two nodes RAC primary, and two nodes standby.
Standby environment is ok, so far works, but no entries added to alert log.
however on primary rac, i cannot sqlplus to the db, and rman target hung, and dgmgrl hung.
I tried to srvctl stop database -o abort, it just hung there.
please help me to get that resolved.
Thanks in advance.check the primary alert log for error and paste here.
also do network connectivity test....e.g
lsnrctl status
from a client system do:
ping primary_ip
tnsping primary_tns
always make sure Abort and killing OS process is your last option!
Tobi -
Icloud transition upgrade mail database hung up
Just installed Lion and doing the icloud transition. "upgrade mail database" has been running for an hour. Something is wrong. What should I do? I am sending this from another computer.
I am having the same problem, but I can't figure out how to force quit the mail application and if I try to restart the computer it tells me that the mail application overode the restart and it won't let me restart the computer. Any suggestions?
-
Report with multiplying columns and WITH clause
Hello
I saw sth strange in my report. I assume that it could happens very often.
I have report with few columns which two of them ar most complicated (many joins subqueries aggreagations on joined values etc.) These two columns (i.e C3,C4) should be multiplied (C3*C4).
When i do pure report without multiplying only columns C1,C2,C3,C4 everything is ok - duration about 15 sec. but... when I put next column on report which multiply these columns (in Answers C5=C3*C4)
I wait 3-4 minutes and my database hungs :(. After investigation I saw that in first case to databese goes pure "SELECT" statement it means:
"Select ... as C1, ... as C2, max(...) as C3, sum(xxx)... C4 from yyy,sss,ttt WHERE aaa"
but in second case BI uses WITH clause it means:
WITH SAWITH0 AS
( Select ... as C1, ... as C2, max(...) as C3, sum(xxx)... C4 from yyy,sss,ttt WHERE aaa )
SELECT SAWITH0.C1 as C1,
SAWITH0.C2 as C2,
SAWITH0.C3 as C3,
SAWITH0.C4 as C4,
SAWITH0.C3*SAWITH0.C4 as C5 FROM SSS
and this statement is long runninq query and kills my database :(.
I checked that SQL like this:
Select ... as C1, ... as C2, max(...) as C3, sum(xxx)... C4, max(...)*sum(xxx)... As C5 from yyy,sss,ttt WHERE aaa" -
runs few times faster than that above
I know that I can do this multiply in business model layer but sometimes users can multiply(or other operations) on columns in reports without my knowledge and it kills my db :(. Where is bug? Why SQLs with WITH clause takes so much db time?
Thank you for each kind of helpWITH clause or Subquery Factoring allows the set of data to be reused multiple times within the SQL. Oracle will usually materialize the data into a temporary table (you will see it if you take an explain plan of the SQL).
I would be surprised if it was the actual WITH clause that was causing the performance issue, however you can test this by turning the WITH clause feature off. Go to the Physical model, right mouse click on your Database > Properties > Features Tab, scroll down to WITH_CLAUSE_SUPPORTED and switch it off.
I'd be interested to know if you do see actual improvement.
Good Luck. -
Randomly hang in Crystal Reports for Visual Studio 2005
after a certain number of prints (PrintMyDoc) the application hangs on the line of code shown below
the void is PrintMyDoc
private ConnectionInfo CreateConnectionInfo(string User, string Password, string Server, string Database)
ConnectionInfo objConn = new ConnectionInfo();
objConn.UserID = User;
objConn.Password = Password;
objConn.ServerName = Server;
objConn.DatabaseName = Database;
return objConn;
private void SetDBLogonForReport(ConnectionInfo connectionInfo, ReportDocument reportDocument)
try
CrystalDecisions.CrystalReports.Engine.Database db = reportDocument.Database; // HUNGS HERE !!!! CrystalDecisions.CrystalReports.Engine.Tables tables = db.Tables;
foreach (CrystalDecisions.CrystalReports.Engine.Table table in tables)
TableLogOnInfo tblInfo = new TableLogOnInfo();
tblInfo.ConnectionInfo = connectionInfo;
table.ApplyLogOnInfo(tblInfo);
foreach (CrystalDecisions.CrystalReports.Engine.ReportDocument subrep in reportDocument.Subreports)
foreach (CrystalDecisions.CrystalReports.Engine.Table table in subrep.Database.Tables)
TableLogOnInfo tblInfo = new TableLogOnInfo();
tblInfo.ConnectionInfo = connectionInfo;
table.ApplyLogOnInfo(tblInfo);
catch (Exception ex)
MessageBox.Show(ex.Message);
the objects are disposed as shown in the lines here below.
protected override void Dispose(bool disposing)
try
if (!(crystalReportViewer.ReportSource == null))
((IDisposable)(crystalReportViewer.ReportSource)).Dispose();
crystalReportViewer.ReportSource = null;
try
objReport.Close();
objReport.Dispose();
objReport = null;
catch (Exception)
crystalReportViewer.Dispose();
crystalReportViewer = null;
catch
if (disposing && (components != null))
components.Dispose();
base.Dispose(disposing);
public void PrintMyDoc()
SqlConnection conn = OpenConnection();
if (conn != null)
if (System.IO.File.Exists(reportName))
ReportDocument myReport = new ReportDocument();
myReport.FileName = reportName;
myReport.PrintOptions.PrinterName = Stampante;
ConnectionInfo objConn = CreateConnectionInfo (Utente, Password, conn.DataSource, conn.Database);
SetDBLogonForReport(objConn, myReport); // <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
this.crystalReportViewer.SelectionFormula = SelectionFormula;
if (Preview)
this.crystalReportViewer.ReportSource = myReport;
this.crystalReportViewer.RefreshReport();
else
myReport.RecordSelectionFormula = SelectionFormula;
myReport.Refresh();
myReport.PrintToPrinter(Numcopie , false, 0, 0);
objReport = myReport;
myReport = null;
this.Close();
conn.Dispose();
else
MessageBox.Show("Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
Hope someone can help me asap
thanksHow did you come to the conclusion that it is a problem with the UFL? If it is a problem with the UFL then there is little that we can do to troubleshoot it because the UFL is developed outside of Crystal. The code can do anything. You'd need to follow up with the person who wrote the UFL.
If you are still unsure of the cause, then I suggest you start testing with other reports. Use ones with no database connection as a control group, then use reports that DON'T include the UFL.
After that, you could capture a stack dump of your application and look at the results. See if you can find where a problem library file is. One thing to note, we won't look at your stack dump here. That is beyond the help we offer on forums.
You could also look at restructuring the portion of your application that handles Crystal Reports. Instead of calling Crystal within your application, you could spawn a new process that does your reporting then shuts down. It will help to isolate the reporting to its own application space that is only run when a report is requested. It doesn't sound like reporting is a large portion of this application if you run 20 reports a day and it runs for 10 hours straight. -
In Noarchivelog mode "Thread 1 cannot allocate new log, sequence 3298"
Hi,My database mode is noarchivelog but i see my alert.log "Thread 1 cannot allocate new log, sequence 3298"
what it`s mean?
I have enough disk space.This is very similar to the your still open thread cannot allocate new log, sequence
Better protocol would surely be to acknowledge the references given then, and why you think they do or do not apply.
Please get one of these threads marked as answered and continue on the other.
Please also indicate:
1) is your database hung.
2) Output form this query:
select log_mode from v$database;
3) Database Version:
4) Are there any other messages in alert log.
5) In sqlplus connected as SYSDBA show output from
archive log list
Rgds - bigdelboy. -
Io_getevents timed out 600 sec
We have one server with OEL5.5 and two oracle databases 10.2.0.4. We are using nfs for the databases. There were no errors about a long time.
Both of these databases hung on weekend after an automatic restart with the same error for 12 hours, no connection possible:
Fri May 20 21:18:39 2011
Completed: ALTER DATABASE OPEN
Fri May 20 21:29:13 2011
WARNING:io_getevents timed out 600 sec
Fri May 20 21:29:14 2011
WARNING:io_getevents timed out 600 sec
Fri May 20 21:39:13 2011
WARNING:io_getevents timed out 600 sec
Fri May 20 21:39:14 2011
WARNING:io_getevents timed out 600 sec
Fri May 20 21:49:13 2011
WARNING:io_getevents timed out 600 sec
Fri May 20 21:49:14 2011
After 12 hours both databases are online back again without doing anything.
In my opinion (and with the help of metalink) i think this is os- and not db-related, but there an no errors at all on os level (/var/log/messages). Ideas?
Thank you!Then it's really strange. How about a problem with the network, if the network was involved? If there was some network congestion it may not necessarily show up as an error in the NFS logs. Are you managing the network? From my experience, If you have a fancy network and i.e Cisco equipment, administrators often like to play around on weekends and evenings or during other system maintenance windows, not necessarily informing since it is not supposed to interfere. Of course the problem could also be with your systems or the NFS server you mentioned. I would leave the problem for now and monitor the next system restart to see if it happens again. If it does, then you have more options to analyze and troubleshoot, e.g. power reset.
Edited by: Dude on Jun 7, 2011 2:31 AM -
Our production OLTP (10.1.0.3) has two archivelog destinations, one of which is on a remote physical standby. This afternoon as ARC1 was finishing writing a log to the standby destination, the standby server suffered a hardware emergency and rebooted itself.
When the standby came back I restarted replication and the archivelog it had started applying before the reboot continued, finished, and then logfile application stopped. Investigation showed that the primary had stopped shipping archivelogs over to the remote destination. In fact, all archival activity had stopped on the primary.
I was able to force log switches, and checkpoints were being reported in the alert log, but no archivelogs were being created. log_archive_min_succeed_dest is set to 1, so this behavior was blatantly illegal. Stranger yet, both ARC0 and ARC1 were still sitting in the process list. No errors or trace files were ever reported in the alert log. I added some redo logs as a temporary fix and to give the netops people time to bring the website down gracefully, and then I bounced the database.
The database hung while dismounting. I noticed ARC1 hanging out there in the process list so I killed it, after which the shutdown proceeded normally. Upon startup, the missing archivelogs were created during instance recovery and both ARCH processes resumed normal activities.
It's a little frightening that an essential service like ARCH could die without any errors being reported, and if I had not just happened to be poking around the redo logs could easily have filled up and brought the website to its knees. Has anyone else ever witnessed such bizarre behavior? My best guess is that the hardware event on the standby "did something" to ARC1 and shut it down, but I don't see how ARC0 could have been affected by that as well.Since you can see the panel, the web client process is running on remote computer. The "X" and error message mean the communication between the web client and the server process has problem. This may be the networking problem, or the configuration problem.
Take a look at this KB. Do you get the same error message? Try to ping the server machine from the web client machine.
http://digital.ni.com/public.nsf/allkb/DFF87FABEE7393B586256945007961FA?OpenDocument
Also refer to this KB.
http://digital.ni.com/public.nsf/websearch/93D8E67CDB4F930686256F990067B3ED?OpenDocument
Is there a firewall on your computer?
By the way, you can develop both server process and client process on the same machine and test the communication. But you can't run the web client on the same machine, because Lookout webclient player can't be installed with Lookout dev/runtime server at the same time. So, the web client has to be running on remote machine.
The web server can be on the same machine with Lookout server.
Ryan Shi
National Instruments -
Hello.
I have a question about online redo logs.
Assume that I have two small redo log groups in my database. I am processing a big batch load consisting of many inserts, but no checkpoing. The first redo log group gets full, it's switched to the second one. The second one gets full too and it need to be switched to the first. No commit until this time, so (I guess) the data has not been saved to datafiles.
Is that possible? Will the database hung ?
If not the first redo log group gets overwritten, doesn't it? What if, after the commit, the database crashes. I will not be able to restore the operations from the first redo group...?Depends,
If you are running in archive log mode then when you fill the first redo log group it is copied to the archvielog destination, then when the second is filled its copied and we switch back to the first if the archiver hasn't completed you will get a short 'hang' whillst the archiver completes then the transaction will proceed.
if in no archivelog mode you will just switch back and forth between the two log files.
You may be confused because you think that the datablocks in the buffer cache and datafiles are not updated untill the transaction is completed this is not true, the assumption is that most transactions will be completed and commited for a usefull discussion of the concepts see.
Re: basic concepts -
ORA-00257: archiver error. Connect internal only
Hello everyone,
On 10gR2, if we face the above error, the system would hang for some time. During this time no one will be able to connect. SYSDBA session will hang and wont give you SQL prompt quickly.
I want to know if there is any way to remove this time gap.
thanksHello,
The ORA-00257 is likely due to a lack of space which prevent generating Archived redologs:
ORA-00257: archiver error. Connect internal only, until freed.
Cause: The archiver process received an error while trying to archive a redo log. If the problem
is not resolved soon, the database will stop executing transactions. The most likely cause of this
message is the destination device is out of space to store the redo log file.
Action: Check archiver trace file for a detailed description of the problem. Also verify that the
device specified in the initialization parameter ARCHIVE_LOG_DEST is set up properly for archiving.So check for a File System full and the use of the Flash Recovery Area (FRA).
The following Note from MOS explains how to check the FRA and how to free space in it:
How to Resolve ORA-00257: Archiver is Stuck Error in 10g? [ID 278308.1]
The database hungs for some time even after clearing the archive log destination. And it is this time that I want to reduce.I think that the best way is to monitore the Archive destination so that you can clear the space before the Database hang.
If you use EM Database Control, you may set some Alert Notification. More over, you have the specific Archive Full Metric:
http://download.oracle.com/docs/cd/B19306_01/em.102/b25986/oracle_database.htm#sthref540
Hope this help.
Best regards,
Jean-Valentin
Edited by: Lubiez Jean-Valentin on Jul 24, 2011 10:46 PM -
Has any one out there encountered the ksbsrv error :-
'no startup aknowledgement from forked process after 3 mins'
on a windows RAC configuration using 9.2.0.5?
We have had this error on one of our 4 nodes. This rendered the entire database unusable through the application, but the 3 unaffected nodes were accessble through sqplus(beq access?)
The problem was rectified by restarting the effected node, which had high CPU despite the fact that the rdbms was unuasable..we suspect an orphaned process......
We are looking at possible causes as changes in OS, and DB init parameters as causes, as well as poor coding..anything else we should be looking at?
Any other information gratefully received..
ThanksHave you found a reason for this error.
I'm not using RAC, but occasionally we receive this error
and have to reboot the dataase.
Oracle TAR and research netted, that we should make sure the listener
log never over 200mg. So I've setup a routine to clean out the
listener log while the db is still running 24x7.
oct 3 and again today the database hung with this error and listener.log
was well within the 200mg limits.
Maybe you are looking for
-
Problem for sign in my Apple mini ipad
I have problem of sign in for Skype in Apple iPad mini, if i go other device Skype is working properly ,please help me Sree
-
I had my last ipod stolen, when I got the 5th Gen I restored from my old ipod from itunes and now there is a bunch of crap on there that I don't want, but the songs only show up on the ipod so I can not delete them off of the ipod from itunes. Any q
-
Re: Reason Of Rejection field in the Sales Order screen in IC Web client
Hi guys WE have implemented CRM 6.0 & ECC 6.0 Our call center guys login to IC Webclient to place Standard Orders in the CRM system which replicates to ECC. Now as most of us know that the Order cannot be rejected once a Delivery Document is created
-
I'm having a book printed. I tried to upload a .pdf of InDesign using photos I edited in iPhoto 09. The printer's "preflight" system said the resolution on the photos was too low--that they'd be fuzzy. They recommend a minimum of 150 dpi. How in
-
Upgrading to new Hard Drives on XW4400 Workstation
Trying to upgrade to higher capacity hard drives (Seagate/300Gb & 500Gb) on my XW4400 workstation is NO-GO. Took original drives out, put in new drives - System (Windows7/64) cannot find my new drives when install is attempted. F10 Setup (I believe