[代发帖子]10.2.0.4.0版本rac 报IPC Send timeout detected.Sender 错误问题
以下为代网友发帖，内容为
环境：db：10.2.0.4.0 system：aix 6.1
请各位大师帮忙确定是什么原因导致的
node1
Wed Aug 22 09:59:17 2012
IPC Send timeout detected.Sender: ospid 25821502
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 09:59:17 2012
IPC Send timeout detected.Sender: ospid 24838548
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 09:59:18 2012
IPC Send timeout detected.Sender: ospid 18743888
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 09:59:18 2012
IPC Send timeout detected.Sender: ospid 8586514
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 09:59:19 2012
IPC Send timeout detected.Sender: ospid 21365006
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 09:59:20 2012
IPC Send timeout detected.Sender: ospid 18088896
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 09:59:20 2012
IPC Send timeout detected.Sender: ospid 15467098
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 09:59:20 2012
Wed Aug 22 09:59:44 2012
IPC Send timeout detected.Sender: ospid 9635304
Receiver: inst 2 binc 1412232544 ospid 25362610
Wed Aug 22 09:59:44 2012
IPC Send timeout detected.Sender: ospid 1704160
Receiver: inst 2 binc 1412232544 ospid 25362610
Wed Aug 22 09:59:44 2012
IPC Send timeout detected.Sender: ospid 2687916
Receiver: inst 2 binc 1412232544 ospid 25362610
Wed Aug 22 09:59:44 2012
IPC Send timeout detected.Sender: ospid 1836306
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 09:59:45 2012
IPC Send timeout detected.Sender: ospid 11273456
Wed Aug 22 10:00:03 2012
IPC Send timeout detected.Sender: ospid 4982212
Receiver: inst 2 binc 1412232544 ospid 25362610
Wed Aug 22 10:00:03 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 629
Wed Aug 22 10:00:03 2012
IPC Send timeout detected.Sender: ospid 15270298
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 10:00:03 2012
IPC Send timeout to 1.5 inc 28 for msg type 36 from opid 374
Wed Aug 22 10:00:03 2012
IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 619
Wed Aug 22 10:00:03 2012
IPC Send timeout detected.Sender: ospid 16712906
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 10:00:03 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 626
Wed Aug 22 10:00:03 2012
IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 376
Wed Aug 22 10:00:03 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 260
Wed Aug 22 10:00:03 2012
IPC Send timeout detected.Sender: ospid 12125594
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 10:00:04 2012
IPC Send timeout detected.Sender: ospid 22086174
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 10:00:04 2012
IPC Send timeout detected.Sender: ospid 8979076
Receiver: inst 2 binc 1412232544 ospid 25362610
Wed Aug 22 10:00:04 2012
IPC Send timeout detected.Sender: ospid 23004052
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 10:00:04 2012
IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 640
Wed Aug 22 10:00:04 2012
IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 114
Wed Aug 22 10:00:04 2012
IPC Send timeout to 1.5 inc 28 for msg type 36 from opid 642
Wed Aug 22 10:00:04 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 644
Wed Aug 22 10:00:04 2012
IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 228
Wed Aug 22 10:00:04 2012
IPC Send timeout detected.Sender: ospid 13436000
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 10:00:05 2012
IPC Send timeout detected.Sender: ospid 4850956
Receiver: inst 2 binc 1412232544 ospid 25362610
Wed Aug 22 10:00:05 2012
IPC Send timeout detected.Sender: ospid 21627618
Receiver: inst 2 binc 1412232544 ospid 25362610
Wed Aug 22 10:00:05 2012
IPC Send timeout to 1.1 inc 28 for msg type 36 from opid 637
Wed Aug 22 10:00:05 2012
IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 648
Wed Aug 22 10:00:05 2012
IPC Send timeout to 1.1 inc 28 for msg type 36 from opid 628
Wed Aug 22 10:00:05 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 647
Wed Aug 22 10:00:05 2012
IPC Send timeout detected.Sender: ospid 23724892
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 10:00:05 2012
IPC Send timeout detected.Sender: ospid 14353416
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 10:00:05 2012
IPC Send timeout detected.Sender: ospid 15270628
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 10:00:06 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 634
Wed Aug 22 10:00:06 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 235
Wed Aug 22 10:00:06 2012
IPC Send timeout detected.Sender: ospid 10749150
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 10:00:06 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 72
Wed Aug 22 10:00:06 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 654
Wed Aug 22 10:00:06 2012
IPC Send timeout detected.Sender: ospid 16123646
Receiver: inst 2 binc 1412232544 ospid 25362610
Wed Aug 22 10:00:06 2012
IPC Send timeout detected.Sender: ospid 19005890
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 10:00:06 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 655
Wed Aug 22 10:00:06 2012
IPC Send timeout detected.Sender: ospid 1311244
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 10:00:06 2012
IPC Send timeout to 1.5 inc 28 for msg type 36 from opid 658
Wed Aug 22 10:00:07 2012
IPC Send timeout detected.Sender: ospid 14091918
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 10:00:07 2012
IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 424
Wed Aug 22 10:00:07 2012
IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 631
Wed Aug 22 10:00:07 2012
IPC Send timeout detected.Sender: ospid 14876874
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 10:00:07 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 661
Wed Aug 22 10:00:07 2012
IPC Send timeout detected.Sender: ospid 18613208
Receiver: inst 2 binc 1412232506 ospid 27066946
Wed Aug 22 10:00:07 2012
IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 664
Wed Aug 22 10:00:07 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 603
Wed Aug 22 10:00:07 2012
IPC Send timeout detected.Sender: ospid 2818720
Receiver: inst 2 binc 1412232544 ospid 25362610
Wed Aug 22 10:03:54 2012
IPC Send timeout to 1.1 inc 28 for msg type 36 from opid 1219
Wed Aug 22 10:04:05 2012
IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 2256
Wed Aug 22 10:04:08 2012
IPC Send timeout to 1.1 inc 28 for msg type 36 from opid 1908
Wed Aug 22 10:04:10 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 1476
Wed Aug 22 10:04:12 2012
IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 1653
Wed Aug 22 10:04:12 2012
IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 1266
Wed Aug 22 10:04:13 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 1345
Wed Aug 22 10:04:13 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 1202
Wed Aug 22 10:04:13 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 1203
Wed Aug 22 10:04:13 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 1239
Wed Aug 22 10:04:13 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 1237
Wed Aug 22 10:04:13 2012
IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 1207
Wed Aug 22 10:04:13 2012
IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 1197
Wed Aug 22 10:04:13 2012
IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 1313
Wed Aug 22 10:04:13 2012
IPC Send timeout to 1.5 inc 28 for msg type 32 from opid 1274
Wed Aug 22 10:04:13 2012
IPC Send timeout to 1.1 inc 28 for msg type 32 from opid 1140
Wed Aug 22 10:04:13 2012
IPC Send timeout to 1.1 inc Wed Aug 22 10:25:55 2012Node2:
Wed Aug 22 09:47:57 2012
Error 2056 trapped in 2PC on transaction 20.16.21521114. Cleaning up.
Error stack returned to user:
ORA-02056: 2PC: k2lpre: bad two-phase command number 0 from coord
Wed Aug 22 09:48:28 2012
Error 2056 trapped in 2PC on transaction 78.44.4484418. Cleaning up.
Error stack returned to user:
ORA-02056: 2PC: k2lpre: bad two-phase command number 0 from coord
Error 2056 trapped in 2PC on transaction 17.42.11081684. Cleaning up.
Error stack returned to user:
ORA-02056: 2PC: k2lpre: bad two-phase command number 0 from coord
Wed Aug 22 09:48:42 2012
Error 2056 trapped in 2PC on transaction 75.15.10618335. Cleaning up.
Error stack returned to user:
ORA-02056: 2PC: k2lpre: bad two-phase command number 0 from coord
Wed Aug 22 09:48:58 2012
Error 2056 trapped in 2PC on transaction 62.28.20411900. Cleaning up.
Error stack returned to user:
ORA-02056: 2PC: k2lpre: bad two-phase command number 0 from coord
Wed Aug 22 09:49:04 2012
Error 2056 trapped in 2PC on transaction 81.34.7287924. Cleaning up.
Error stack returned to user:
ORA-02056: 2PC: k2lpre: bad two-phase command number 0 from coord
Wed Aug 22 09:50:01 2012
Thread 2 advanced to log sequence 53611 (LGWR switch)
Current log# 5 seq# 53611 mem# 0: /dev/rredo05_rzdb
Wed Aug 22 09:50:34 2012
Error 2056 trapped in 2PC on transaction 71.15.14657575. Cleaning up.
Error stack returned to user:
ORA-02056: 2PC: k2lpre: bad two-phase command number 0 from coord
Wed Aug 22 09:50:46 2012
Error 2056 trapped in 2PC on transaction 84.12.11181366. Cleaning up.
Error stack returned to user:
ORA-02056: 2PC: k2lpre: bad two-phase command number 0 from coord
Wed Aug 22 09:50:48 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/udump/crmrz2_ora_23856588.trc:
ORA-00600: internal error code, arguments: [ksqcmi:res2], [], [], [], [], [], [], []
Wed Aug 22 09:50:49 2012
Trace dumping is performing id=[cdmp_20120822095049]
Wed Aug 22 09:59:16 2012
IPC Send timeout detected. Receiver ospid 27066946
Receiver is waiting for a latch dumping latch state for receiver -17356
Wed Aug 22 09:59:16 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms4_25362610.trc:
Wed Aug 22 09:59:38 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms0_27066946.trc:
Wed Aug 22 09:59:38 2012
Trace dumping is performing id=[cdmp_20120822095938]
Wed Aug 22 10:02:09 2012
kkjcre1p: unable to spawn jobq slave process
Wed Aug 22 10:02:09 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_cjq0_25100728.trc:
Wed Aug 22 10:02:32 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lmon_25493740.trc:
ORA-29740: evicted by member 0, group incarnation 30
Wed Aug 22 10:02:32 2012
LMON: terminating instance due to error 29740
Wed Aug 22 10:02:32 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms2_24904066.trc:
ORA-29740: evicted by member , group incarnation
Wed Aug 22 10:02:32 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms6_25626088.trc:
ORA-29740: evicted by member , group incarnation
Wed Aug 22 10:02:32 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms3_25559280.trc:
ORA-29740: evicted by member , group incarnation
Wed Aug 22 10:02:32 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms5_22873874.trc:
ORA-29740: evicted by member , group incarnation
Wed Aug 22 10:02:32 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms1_24905100.trc:
ORA-29740: evicted by member , group incarnation
Wed Aug 22 10:02:32 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_arc1_26543176.trc:
ORA-29740: evicted by member , group incarnation
Wed Aug 22 10:02:32 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_arc1_26543176.trc:
ORA-29740: evicted by member , group incarnation
Wed Aug 22 10:02:32 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_arc1_26543176.trc:
ORA-29740: evicted by member , group incarnation
Wed Aug 22 10:02:32 2012
Master background archival failure: 29740
Wed Aug 22 10:02:32 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lgwr_26477376.trc:
ORA-29740: evicted by member , group incarnation
Wed Aug 22 10:02:32 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lck0_22677374.trc:
ORA-29740: evicted by member , group incarnation
Wed Aug 22 10:02:32 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lmd0_27132530.trc:
ORA-29740: evicted by member , group incarnation
Wed Aug 22 10:02:32 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_pmon_25494950.trc:
ORA-29740: evicted by member , group incarnation
Wed Aug 22 10:02:33 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_mman_21235492.trc:
ORA-29740: evicted by member , group incarnation
Wed Aug 22 10:02:33 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_arc0_25036272.trc:
ORA-29740: evicted by member , group incarnation
Wed Aug 22 10:02:33 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_arc0_25036272.trc:
ORA-29740: evicted by member , group incarnation
Wed Aug 22 10:02:33 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_arc0_25036272.trc:
ORA-29740: evicted by member , group incarnation
Wed Aug 22 10:02:33 2012
Master background archival failure: 29740
Wed Aug 22 10:02:33 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_smon_22743004.trc:
ORA-29740: evicted by member , group incarnation
Wed Aug 22 10:02:34 2012
Shutting down instance (abort)
License high water mark = 1722
Wed Aug 22 10:02:38 2012
Instance terminated by LMON, pid = 25493740
Wed Aug 22 10:02:39 2012
Instance terminated by USER, pid = 22873880
Wed Aug 22 10:03:09 2012
Starting ORACLE instance (normal)
sskgpgetexecname failed to get name
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 en10 192.100.100.0 configured from OCR for use as a cluster interconnect
Interface type 1 en9 134.224.106.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
processes = 3000
sessions = 3305
resource_limit = TRUE
event =
sga_max_size = 51539607552
__shared_pool_size = 4227858432
__large_pool_size = 16777216
__java_pool_size = 33554432
__streams_pool_size = 0
spfile = /dev/rspfile_rzdb
sga_target = 51539607552
control_files = /dev/rcntrl01_rzdb, /dev/rcntrl02_rzdb, /dev/rcntrl03_rzdb
db_block_size = 8192
__db_cache_size = 46707769344
db_keep_cache_size = 536870912
compatible = 10.2.0.3.0
log_archive_dest_1 = location=/arch02
db_files = 1000
db_file_multiblock_read_count= 16
cluster_database = TRUE
cluster_database_instances= 2
thread = 2
instance_number = 2
undo_management = AUTO
undo_tablespace = UNDOTBS2
undo_retention = 900
recyclebin = OFF
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=crmrzXDB)
local_listener = (ADDRESS = (PROTOCOL =TCP)(HOST = 134.224.106.126)(PORT = 1521))
remote_listener = LISTENERS_CRMRZ
utl_file_dir = /arch01
job_queue_processes = 20
cursor_sharing = SIMILAR
background_dump_dest = /oracle/opt/product/10.2.0/db/admin/crmrz/bdump
user_dump_dest = /oracle/opt/product/10.2.0/db/admin/crmrz/udump
core_dump_dest = /oracle/opt/product/10.2.0/db/admin/crmrz/cdump
audit_file_dest = /oracle/opt/product/10.2.0/db/admin/crmrz/adump
db_name = crmrz
open_cursors = 1500
pga_aggregate_target = 10726932480
Cluster communication is configured to use the following interface(s) for this instance
192.100.100.2
Wed Aug 22 10:03:11 2012
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=24379686
DIAG started with pid=3, OS id=22743022
PSP0 started with pid=4, OS id=24773700
LMON started with pid=5, OS id=25429322
LMD0 started with pid=6, OS id=25494130
LMS0 started with pid=7, OS id=24904068
LMS1 started with pid=8, OS id=9437246
LMS2 started with pid=9, OS id=13304862
LMS3 started with pid=10, OS id=24969498
LMS4 started with pid=11, OS id=26936330
LMS5 started with pid=12, OS id=26935868
LMS6 started with pid=13, OS id=25167108
MMAN started with pid=14, OS id=27132604
DBW0 started with pid=15, OS id=26739208
DBW1 started with pid=16, OS id=23790482
DBW2 started with pid=17, OS id=25626104
DBW3 started with pid=18, OS id=25494956
LGWR started with pid=19, OS id=26673758
CKPT started with pid=20, OS id=26542676
SMON started with pid=21, OS id=20383626
RECO started with pid=22, OS id=26477394
CJQ0 started with pid=23, OS id=26411882
MMON started with pid=24, OS id=26346430
Wed Aug 22 10:03:14 2012
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=25, OS id=25624582
Wed Aug 22 10:03:14 2012
starting up 1 shared server(s) ...
Wed Aug 22 10:03:19 2012
lmon registered with NM - instance id 2 (internal mem no 1)
Wed Aug 22 10:03:26 2012
Reconfiguration started (old inc 0, new inc 33)
List of nodes:
0 1
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* domain 0 valid = 0 according to instance 0
Wed Aug 22 10:03:26 2012
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Wed Aug 22 10:03:26 2012
LMS 4: 0 GCS shadows cancelled, 0 closed
Wed Aug 22 10:03:26 2012
LMS 6: 0 GCS shadows cancelled, 0 closed
Wed Aug 22 10:03:26 2012
LMS 2: 0 GCS shadows cancelled, 0 closed
Wed Aug 22 10:03:26 2012
LMS 1: 0 GCS shadows cancelled, 0 closed
Wed Aug 22 10:03:26 2012
LMS 5: 0 GCS shadows cancelled, 0 closed
Wed Aug 22 10:03:26 2012
LMS 3: 0 GCS shadows cancelled, 0 closed
Wed Aug 22 10:03:26 2012
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Wed Aug 22 10:03:28 2012
LMS 5: 0 GCS shadows traversed, 0 replayed
Wed Aug 22 10:03:28 2012
LMS 6: 0 GCS shadows traversed, 0 replayed
Wed Aug 22 10:03:28 2012
LMS 4: 0 GCS shadows traversed, 0 replayed
Wed Aug 22 10:03:28 2012
LMS 2: 0 GCS shadows traversed, 0 replayed
Wed Aug 22 10:03:28 2012
LMS 1: 0 GCS shadows traversed, 0 replayed
Wed Aug 22 10:03:28 2012
LMS 3: 0 GCS shadows traversed, 0 replayed
Wed Aug 22 10:03:28 2012
LMS 0: 0 GCS shadows traversed, 0 replayed
Wed Aug 22 10:03:28 2012
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=28, OS id=26215276
Wed Aug 22 10:03:33 2012
ALTER DATABASE MOUNT
Wed Aug 22 10:03:37 2012
Setting recovery target incarnation to 1
Wed Aug 22 10:03:37 2012
Successful mount of redo thread 2, with mount id 3477695295
Wed Aug 22 10:03:37 2012
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Completed: ALTER DATABASE MOUNT
Wed Aug 22 10:03:38 2012
ALTER DATABASE OPEN
Picked broadcast on commit scheme to generate SCNs
Wed Aug 22 10:03:59 2012
Trace dumping is performing id=[cdmp_20120822100344]
IPC Send timeout detected. Receiver ospid 24904068
Wed Aug 22 10:04:03 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms0_24904068.trc:
Wed Aug 22 10:04:59 2012
Trace dumping is performing id=[cdmp_20120822100458]
Wed Aug 22 10:05:19 2012
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=32, OS id=26739814
Wed Aug 22 10:05:19 2012
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=31, OS id=24445226
Wed Aug 22 10:05:20 2012
IPC Send timeout detected. Receiver ospid 24904068
Wed Aug 22 10:05:20 2012
Errors in file /oracle/opt/product/10.2.0/db/admin/crmrz/bdump/crmrz2_lms0_24904068.trc:
Wed Aug 22 10:05:21 2012
Thread 2 opened at log sequence 53612
Current log# 6 seq# 53612 mem# 0: /dev/rredo06_rzdb
Wed Aug 22 10:05:21 2012
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Wed Aug 22 10:05:21 2012
ARC0: Becoming the heartbeat ARCH
Wed Aug 22 10:05:21 2012
Successful open of redo thread 2
Wed Aug 22 10:05:21 2012
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Aug 22 10:05:21 2012
SMON: enabling cache recovery
Wed Aug 22 10:06:31 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Aug 22 10:06:32 2012
Successfully onlined Undo Tablespace 4.
Wed Aug 22 10:06:32 2012
SMON: enabling tx recovery
Wed Aug 22 10:06:32 2012
Database Characterset is ZHS16GBK
Opening with internal Resource Manager plan
where NUMA PG = 1, CPUs = 28
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=42, OS id=25887794
Wed Aug 22 10:06:43 2012
Completed: ALTER DATABASE OPEN
Wed Aug 22 10:07:00 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Aug 22 10:07:13 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Aug 22 10:07:25 2012
WARNING: inbound connection timed out (ORA-3136)
Wed Aug 22 10:07:31 2012
Trace dumping is performing id=[cdmp_20120822100731]
Wed Aug 22 10:09:20 2012
Trace dumping is performing id=[cdmp_20120822100732]
Wed Aug 22 10:25:54 2012
2012-08-22 10:01:23.571: [ CRSEVT][11527]32CAAMonitorHandler :: 0:Could not join /oracle/opt/product/10.2.0/crs/bin/racgwrap(check)
category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child
2012-08-22 10:01:31.473: [ CRSEVT][11527]32CAAMonitorHandler :: 0:Action Script /oracle/opt/product/10.2.0/crs/bin/racgwrap(check) timed out for ora.crmrz01.vip! (timeout=60)
2012-08-22 10:01:31.474: [ CRSAPP][11527]32CheckResource error for ora.crmrz01.vip error code = -2
2012-08-22 10:03:04.553: [ CRSEVT][11530]32CAAMonitorHandler :: 0:Could not join /oracle/opt/product/10.2.0/crs/bin/racgwrap(check)
category: 1234, operation: scls_process_join, loc: childcrash, OS error: 0, other: Abnormal termination of the child
2012-08-22 10:03:04.553: [ CRSEVT][11530]32CAAMonitorHandler :: 0:Action Script /oracle/opt/product/10.2.0/crs/bin/racgwrap(check) timed out for ora.crmrz01.vip! (timeout=60)
2012-08-22 10:03:04.554: [ CRSAPP][11530]32CheckResource error for ora.crmrz01.vip error code = -2
2012-08-22 10:03:39.055: [ COMMCRS][3866]Authorization failed, network error
2012-08-22 10:03:39.138: [ OCRSRV][3866]th_select_answer: Failure in answer. clsc ret [3]
2012-08-22 10:03:39.630: [ COMMCRS][3866]Authorization failed, network error
2012-08-22 10:03:39.814: [ OCRSRV][3866]th_select_answer: Failure in answer. clsc ret [3]
2012-08-22 10:04:13.506: [ COMMCRS][3866]Authorization failed, network error
2012-08-22 10:04:13.522: [ OCRSRV][3866]th_select_answer: Failure in answer. clsc ret [3]
2012-08-22 10:04:13.610: [ COMMCRS][3866]Authorization failed, network error
2012-08-22 10:04:13.618: [ OCRSRV][3866]th_select_answer: Failure in answer. clsc ret [3]
2012-08-22 10:04:44.509: [ COMMCRS][3866]Authorization failed, network error
2012-08-22 10:04:44.525: [ OCRSRV][3866]th_select_answer: Failure in answer. clsc ret [3]
2012-08-22 10:04:44.898: [ COMMCRS][3866]Authorization failed, network error
2012-08-22 10:04:44.915: [ OCRSRV][3866]th_select_answer: Failure in answer. clsc ret [3]
2012-08-22 10:04:58.959: [ COMMCRS][3866]Authorization failed, network error
2012-08-22 10:04:58.963: [ OCRSRV][3866]th_select_answer: Failure in answer. clsc ret [3]
2012-08-22 10:04:59.047: [ COMMCRS][3866]Authorization failed, network 2012-08-22 10:12:25.310: [ CRSRES][11569]32startRunnable: setting CLI values
crs 日志network error 基本上可以确定当时的网络是存在问题
Similar Messages
-
OD Failing - mach message send error: (ipc/send)
We are seeing our master OD server failing, usually overnight.
When it does so the mail server continues to accept mails but then reports that the users are not on the mail server and bounces them back. In addition all network users are natually unable to log into the network. This continues till the server appears to lock up and we have to power cycle.
We have recently upgraded from 10.3.8 to 10.4.4 in an attempt to fix this but it appears the problem is still there. The machine uses both network interfaces, I have tried to get it to only use one but LDAP fails when I do.
Errors of note are:
2006-02-08 01:05:50 EST - mach message send error: (ipc/send) invalid destination port
in
/Library/Logs/DirectoryService/DirectoryService.error.log
then
2006-02-28 01:34:29 EST - Network transition occurred.
in /Library/Logs/DirectoryService/DirectoryService.server.log
and then:
Feb 28 03:17:16 stingray DirectoryService[74]: could not load the replica file.
in /var/log/system.log (we have no replica so this is to be expected.
This is really causing us grief, has anyone else come across this error?
G5 XServe Mac OS X (10.4.4) used for mail, authentication and DNSJust had a look in the Apply Operating System step :
1. the option "apply operating system from a captured image" is checked , the Image package is "Windows 8.1 Enterprise en-US", the Image is "1 - Windows 8.1 Enterprise"
2. The "use an unattended or sysprep answer file for a custom installation" is NOT checked...
So... Getting closer to a solution? I've already F8'd the PE boot and saw that the unattend.txt does exist under c:\windows\panther\unattend\unattend.xml - also filled in with our values which we specified in the "apply windows settings" +
"apply network settings" (both following right after a "BCDBoot Fix" step I created for this UEFI device [command : bcdboot c:\windows /s c: /f UEFI /l en-us].
My questions :
the package option : what should I specify as this?
the file name option : can I just use that "c:\windows\panther\unattend\unattend.xml" ?
/dimi -
Active session Spike on Oracle RAC 11G R2 on HP UX
Dear Experts,
We need urgent help please, as we are facing very low performance in production database.
We are having oracle 11G RAC on HP Unix environment. Following is the ADDM report. Kindly check and please help me to figure it out the issue and resolve it at earliest.
---------Instance 1---------------
ADDM Report for Task 'TASK_36650'
Analysis Period
AWR snapshot range from 11634 to 11636.
Time period starts at 21-JUL-13 07.00.03 PM
Time period ends at 21-JUL-13 09.00.49 PM
Analysis Target
Database 'MCMSDRAC' with DB ID 2894940361.
Database version 11.2.0.1.0.
ADDM performed an analysis of instance mcmsdrac1, numbered 1 and hosted at
mcmsdbl1.
Activity During the Analysis Period
Total database time was 38466 seconds.
The average number of active sessions was 5.31.
Summary of Findings
Description Active Sessions Recommendations
Percent of Activity
1 CPU Usage 1.44 | 27.08 1
2 Interconnect Latency .07 | 1.33 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Findings and Recommendations
Finding 1: CPU Usage
Impact is 1.44 active sessions, 27.08% of total activity.
Host CPU was a bottleneck and the instance was consuming 99% of the host CPU.
All wait times will be inflated by wait for CPU.
Host CPU consumption was 99%.
Recommendation 1: Host Configuration
Estimated benefit is 1.44 active sessions, 27.08% of total activity.
Action
Consider adding more CPUs to the host or adding instances serving the
database on other hosts.
Action
Session CPU consumption was throttled by the Oracle Resource Manager.
Consider revising the resource plan that was active during the analysis
period.
Finding 2: Interconnect Latency
Impact is .07 active sessions, 1.33% of total activity.
Higher than expected latency of the cluster interconnect was responsible for
significant database time on this instance.
The instance was consuming 110 kilo bits per second of interconnect bandwidth.
20% of this interconnect bandwidth was used for global cache messaging, 21%
for parallel query messaging and 7% for database lock management.
The average latency for 8K interconnect messages was 42153 microseconds.
The instance is using the private interconnect device "lan2" with IP address
172.16.200.71 and source "Oracle Cluster Repository".
The device "lan2" was used for 100% of interconnect traffic and experienced 0
send or receive errors during the analysis period.
Recommendation 1: Host Configuration
Estimated benefit is .07 active sessions, 1.33% of total activity.
Action
Investigate cause of high network interconnect latency between database
instances. Oracle's recommended solution is to use a high speed
dedicated network.
Action
Check the configuration of the cluster interconnect. Check OS setup like
adapter setting, firmware and driver release. Check that the OS's socket
receive buffers are large enough to store an entire multiblock read. The
value of parameter "db_file_multiblock_read_count" may be decreased as a
workaround.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Additional Information
Miscellaneous Information
Wait class "Application" was not consuming significant database time.
Wait class "Cluster" was not consuming significant database time.
Wait class "Commit" was not consuming significant database time.
Wait class "Concurrency" was not consuming significant database time.
Wait class "Configuration" was not consuming significant database time.
Wait class "Network" was not consuming significant database time.
Wait class "User I/O" was not consuming significant database time.
Session connect and disconnect calls were not consuming significant database
time.
Hard parsing of SQL statements was not consuming significant database time.
The database's maintenance windows were active during 100% of the analysis
period.
----------------Instance 2 --------------------
ADDM Report for Task 'TASK_36652'
Analysis Period
AWR snapshot range from 11634 to 11636.
Time period starts at 21-JUL-13 07.00.03 PM
Time period ends at 21-JUL-13 09.00.49 PM
Analysis Target
Database 'MCMSDRAC' with DB ID 2894940361.
Database version 11.2.0.1.0.
ADDM performed an analysis of instance mcmsdrac2, numbered 2 and hosted at
mcmsdbl2.
Activity During the Analysis Period
Total database time was 2898 seconds.
The average number of active sessions was .4.
Summary of Findings
Description Active Sessions Recommendations
Percent of Activity
1 Top SQL Statements .11 | 27.65 5
2 Interconnect Latency .1 | 24.15 1
3 Shared Pool Latches .09 | 22.42 1
4 PL/SQL Execution .06 | 14.39 2
5 Unusual "Other" Wait Event .03 | 8.73 4
6 Unusual "Other" Wait Event .03 | 6.42 3
7 Unusual "Other" Wait Event .03 | 6.29 6
8 Hard Parse .02 | 5.5 0
9 Soft Parse .02 | 3.86 2
10 Unusual "Other" Wait Event .01 | 3.75 4
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Findings and Recommendations
Finding 1: Top SQL Statements
Impact is .11 active sessions, 27.65% of total activity.
SQL statements consuming significant database time were found. These
statements offer a good opportunity for performance improvement.
Recommendation 1: SQL Tuning
Estimated benefit is .05 active sessions, 12.88% of total activity.
Action
Investigate the PL/SQL statement with SQL_ID "d1s02myktu19h" for
possible performance improvements. You can supplement the information
given here with an ASH report for this SQL_ID.
Related Object
SQL statement with SQL_ID d1s02myktu19h.
begin dbms_utility.validate(:1,:2,:3,:4); end;
Rationale
The SQL Tuning Advisor cannot operate on PL/SQL statements.
Rationale
Database time for this SQL was divided as follows: 13% for SQL
execution, 2% for parsing, 85% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "d1s02myktu19h" was executed 48 times and had
an average elapsed time of 7 seconds.
Rationale
Waiting for event "library cache pin" in wait class "Concurrency"
accounted for 70% of the database time spent in processing the SQL
statement with SQL_ID "d1s02myktu19h".
Rationale
Top level calls to execute the PL/SQL statement with SQL_ID
"63wt8yna5umd6" are responsible for 100% of the database time spent on
the PL/SQL statement with SQL_ID "d1s02myktu19h".
Related Object
SQL statement with SQL_ID 63wt8yna5umd6.
begin DBMS_UTILITY.COMPILE_SCHEMA( 'TPAUSER', FALSE ); end;
Recommendation 2: SQL Tuning
Estimated benefit is .02 active sessions, 4.55% of total activity.
Action
Run SQL Tuning Advisor on the SELECT statement with SQL_ID
"fk3bh3t41101x".
Related Object
SQL statement with SQL_ID fk3bh3t41101x.
SELECT MEM.MEMBER_CODE ,MEM.E_NAME,Pol.Policy_no
,pol.date_from,pol.date_to,POL.E_NAME,MEM.SEX,(SYSDATE-MEM.BIRTH_DATE
) AGE,POL.SCHEME_NO FROM TPAUSER.MEMBERS MEM,TPAUSER.POLICY POL WHERE
POL.QUOTATION_NO=MEM.QUOTATION_NO AND POL.BRANCH_CODE=MEM.BRANCH_CODE
and endt_no=(select max(endt_no) from tpauser.members mm where
mm.member_code=mem.member_code AND mm.QUOTATION_NO=MEM.QUOTATION_NO)
and member_code like '%' || nvl(:1,null) ||'%' ORDER BY MEMBER_CODE
Rationale
The SQL spent 92% of its database time on CPU, I/O and Cluster waits.
This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 100% for SQL
execution, 0% for parsing, 0% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "fk3bh3t41101x" was executed 14 times and had
an average elapsed time of 4.9 seconds.
Rationale
At least one execution of the statement ran in parallel.
Recommendation 3: SQL Tuning
Estimated benefit is .02 active sessions, 3.79% of total activity.
Action
Run SQL Tuning Advisor on the SELECT statement with SQL_ID
"7mhjbjg9ntqf5".
Related Object
SQL statement with SQL_ID 7mhjbjg9ntqf5.
SELECT SUM(CNT) FROM (SELECT COUNT(PROC_CODE) CNT FROM
TPAUSER.TORBINY_PROCEDURE WHERE BRANCH_CODE = :B6 AND QUOTATION_NO =
:B5 AND CLASS_NO = :B4 AND OPTION_NO = :B3 AND PR_EFFECTIVE_DATE<=
:B2 AND PROC_CODE = :B1 UNION SELECT COUNT(MED_CODE) CNT FROM
TPAUSER.TORBINY_MEDICINE WHERE BRANCH_CODE = :B6 AND QUOTATION_NO =
:B5 AND CLASS_NO = :B4 AND OPTION_NO = :B3 AND M_EFFECTIVE_DATE<= :B2
AND MED_CODE = :B1 UNION SELECT COUNT(LAB_CODE) CNT FROM
TPAUSER.TORBINY_LAB WHERE BRANCH_CODE = :B6 AND QUOTATION_NO = :B5
AND CLASS_NO = :B4 AND OPTION_NO = :B3 AND L_EFFECTIVE_DATE<= :B2 AND
LAB_CODE = :B1 )
Rationale
The SQL spent 100% of its database time on CPU, I/O and Cluster waits.
This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 0% for SQL execution,
0% for parsing, 100% for PL/SQL execution and 0% for Java execution.
Rationale
SQL statement with SQL_ID "7mhjbjg9ntqf5" was executed 31 times and had
an average elapsed time of 3.4 seconds.
Rationale
Top level calls to execute the SELECT statement with SQL_ID
"a11nzdnd91gsg" are responsible for 100% of the database time spent on
the SELECT statement with SQL_ID "7mhjbjg9ntqf5".
Related Object
SQL statement with SQL_ID a11nzdnd91gsg.
SELECT POLICY_NO,SCHEME_NO FROM TPAUSER.POLICY WHERE QUOTATION_NO
=:B1
Recommendation 4: SQL Tuning
Estimated benefit is .01 active sessions, 3.03% of total activity.
Action
Investigate the SELECT statement with SQL_ID "4uqs4jt7aca5s" for
possible performance improvements. You can supplement the information
given here with an ASH report for this SQL_ID.
Related Object
SQL statement with SQL_ID 4uqs4jt7aca5s.
SELECT DISTINCT USER_ID FROM GV$SESSION, USERS WHERE UPPER (USERNAME)
= UPPER (USER_ID) AND USERS.APPROVAL_CLAIM='VC' AND USER_ID=:B1
Rationale
The SQL spent only 0% of its database time on CPU, I/O and Cluster
waits. Therefore, the SQL Tuning Advisor is not applicable in this case.
Look at performance data for the SQL to find potential improvements.
Rationale
Database time for this SQL was divided as follows: 100% for SQL
execution, 0% for parsing, 0% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "4uqs4jt7aca5s" was executed 261 times and had
an average elapsed time of 0.35 seconds.
Rationale
At least one execution of the statement ran in parallel.
Rationale
Top level calls to execute the PL/SQL statement with SQL_ID
"91vt043t78460" are responsible for 100% of the database time spent on
the SELECT statement with SQL_ID "4uqs4jt7aca5s".
Related Object
SQL statement with SQL_ID 91vt043t78460.
begin TPAUSER.RECEIVE_NEW_FAX_APRROVAL(:V00001,:V00002,:V00003,:V0000
4); end;
Recommendation 5: SQL Tuning
Estimated benefit is .01 active sessions, 3.03% of total activity.
Action
Run SQL Tuning Advisor on the SELECT statement with SQL_ID
"7kt28fkc0yn5f".
Related Object
SQL statement with SQL_ID 7kt28fkc0yn5f.
SELECT COUNT(*) FROM TPAUSER.APPROVAL_MASTER WHERE APPROVAL_STATUS IS
NULL AND (UPPER(CODED) = UPPER(:B1 ) OR UPPER(PROCESSED_BY) =
UPPER(:B1 ))
Rationale
The SQL spent 100% of its database time on CPU, I/O and Cluster waits.
This part of database time may be improved by the SQL Tuning Advisor.
Rationale
Database time for this SQL was divided as follows: 100% for SQL
execution, 0% for parsing, 0% for PL/SQL execution and 0% for Java
execution.
Rationale
SQL statement with SQL_ID "7kt28fkc0yn5f" was executed 1034 times and
had an average elapsed time of 0.063 seconds.
Rationale
Top level calls to execute the PL/SQL statement with SQL_ID
"91vt043t78460" are responsible for 100% of the database time spent on
the SELECT statement with SQL_ID "7kt28fkc0yn5f".
Related Object
SQL statement with SQL_ID 91vt043t78460.
begin TPAUSER.RECEIVE_NEW_FAX_APRROVAL(:V00001,:V00002,:V00003,:V0000
4); end;
Finding 2: Interconnect Latency
Impact is .1 active sessions, 24.15% of total activity.
Higher than expected latency of the cluster interconnect was responsible for
significant database time on this instance.
The instance was consuming 128 kilo bits per second of interconnect bandwidth.
17% of this interconnect bandwidth was used for global cache messaging, 6% for
parallel query messaging and 8% for database lock management.
The average latency for 8K interconnect messages was 41863 microseconds.
The instance is using the private interconnect device "lan2" with IP address
172.16.200.72 and source "Oracle Cluster Repository".
The device "lan2" was used for 100% of interconnect traffic and experienced 0
send or receive errors during the analysis period.
Recommendation 1: Host Configuration
Estimated benefit is .1 active sessions, 24.15% of total activity.
Action
Investigate cause of high network interconnect latency between database
instances. Oracle's recommended solution is to use a high speed
dedicated network.
Action
Check the configuration of the cluster interconnect. Check OS setup like
adapter setting, firmware and driver release. Check that the OS's socket
receive buffers are large enough to store an entire multiblock read. The
value of parameter "db_file_multiblock_read_count" may be decreased as a
workaround.
Symptoms That Led to the Finding:
Inter-instance messaging was consuming significant database time on this
instance.
Impact is .06 active sessions, 14.23% of total activity.
Wait class "Cluster" was consuming significant database time.
Impact is .06 active sessions, 14.23% of total activity.
Finding 3: Shared Pool Latches
Impact is .09 active sessions, 22.42% of total activity.
Contention for latches related to the shared pool was consuming significant
database time.
Waits for "library cache lock" amounted to 5% of database time.
Waits for "library cache pin" amounted to 17% of database time.
Recommendation 1: Application Analysis
Estimated benefit is .09 active sessions, 22.42% of total activity.
Action
Investigate the cause for latch contention using the given blocking
sessions or modules.
Rationale
The session with ID 17 and serial number 15595 in instance number 1 was
the blocking session responsible for 34% of this recommendation's
benefit.
Symptoms That Led to the Finding:
Wait class "Concurrency" was consuming significant database time.
Impact is .1 active sessions, 24.96% of total activity.
Finding 4: PL/SQL Execution
Impact is .06 active sessions, 14.39% of total activity.
PL/SQL execution consumed significant database time.
Recommendation 1: SQL Tuning
Estimated benefit is .05 active sessions, 12.5% of total activity.
Action
Tune the entry point PL/SQL "SYS.DBMS_UTILITY.COMPILE_SCHEMA" of type
"PACKAGE" and ID 6019. Refer to the PL/SQL documentation for addition
information.
Rationale
318 seconds spent in executing PL/SQL "SYS.DBMS_UTILITY.VALIDATE#2" of
type "PACKAGE" and ID 6019.
Recommendation 2: SQL Tuning
Estimated benefit is .01 active sessions, 1.89% of total activity.
Action
Tune the entry point PL/SQL
"SYSMAN.EMD_MAINTENANCE.EXECUTE_EM_DBMS_JOB_PROCS" of type "PACKAGE" and
ID 68654. Refer to the PL/SQL documentation for addition information.
Finding 5: Unusual "Other" Wait Event
Impact is .03 active sessions, 8.73% of total activity.
Wait event "DFS lock handle" in wait class "Other" was consuming significant
database time.
Recommendation 1: Application Analysis
Estimated benefit is .03 active sessions, 8.73% of total activity.
Action
Investigate the cause for high "DFS lock handle" waits. Refer to
Oracle's "Database Reference" for the description of this wait event.
Recommendation 2: Application Analysis
Estimated benefit is .03 active sessions, 8.27% of total activity.
Action
Investigate the cause for high "DFS lock handle" waits in Service
"mcmsdrac".
Recommendation 3: Application Analysis
Estimated benefit is .02 active sessions, 5.05% of total activity.
Action
Investigate the cause for high "DFS lock handle" waits in Module "TOAD
9.7.2.5".
Recommendation 4: Application Analysis
Estimated benefit is .01 active sessions, 3.21% of total activity.
Action
Investigate the cause for high "DFS lock handle" waits in Module
"toad.exe".
Symptoms That Led to the Finding:
Wait class "Other" was consuming significant database time.
Impact is .15 active sessions, 38.29% of total activity.
Finding 6: Unusual "Other" Wait Event
Impact is .03 active sessions, 6.42% of total activity.
Wait event "reliable message" in wait class "Other" was consuming significant
database time.
Recommendation 1: Application Analysis
Estimated benefit is .03 active sessions, 6.42% of total activity.
Action
Investigate the cause for high "reliable message" waits. Refer to
Oracle's "Database Reference" for the description of this wait event.
Recommendation 2: Application Analysis
Estimated benefit is .03 active sessions, 6.42% of total activity.
Action
Investigate the cause for high "reliable message" waits in Service
"mcmsdrac".
Recommendation 3: Application Analysis
Estimated benefit is .02 active sessions, 4.13% of total activity.
Action
Investigate the cause for high "reliable message" waits in Module "TOAD
9.7.2.5".
Symptoms That Led to the Finding:
Wait class "Other" was consuming significant database time.
Impact is .15 active sessions, 38.29% of total activity.
Finding 7: Unusual "Other" Wait Event
Impact is .03 active sessions, 6.29% of total activity.
Wait event "enq: PS - contention" in wait class "Other" was consuming
significant database time.
Recommendation 1: Application Analysis
Estimated benefit is .03 active sessions, 6.29% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits. Refer to
Oracle's "Database Reference" for the description of this wait event.
Recommendation 2: Application Analysis
Estimated benefit is .02 active sessions, 6.02% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits in Service
"mcmsdrac".
Recommendation 3: Application Analysis
Estimated benefit is .02 active sessions, 4.93% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits with
P1,P2,P3 ("name|mode, instance, slave ID") values "1347616774", "1" and
"3599" respectively.
Recommendation 4: Application Analysis
Estimated benefit is .01 active sessions, 2.74% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits in Module
"Inbox Reader_92.exe".
Recommendation 5: Application Analysis
Estimated benefit is .01 active sessions, 2.74% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits in Module
"TOAD 9.7.2.5".
Recommendation 6: Application Analysis
Estimated benefit is .01 active sessions, 1.37% of total activity.
Action
Investigate the cause for high "enq: PS - contention" waits with
P1,P2,P3 ("name|mode, instance, slave ID") values "1347616774", "1" and
"3598" respectively.
Symptoms That Led to the Finding:
Wait class "Other" was consuming significant database time.
Impact is .15 active sessions, 38.29% of total activity.
Finding 8: Hard Parse
Impact is .02 active sessions, 5.5% of total activity.
Hard parsing of SQL statements was consuming significant database time.
Hard parses due to cursor environment mismatch were not consuming significant
database time.
Hard parsing SQL statements that encountered parse errors was not consuming
significant database time.
Hard parses due to literal usage and cursor invalidation were not consuming
significant database time.
The Oracle instance memory (SGA and PGA) was adequately sized.
No recommendations are available.
Symptoms That Led to the Finding:
Contention for latches related to the shared pool was consuming
significant database time.
Impact is .09 active sessions, 22.42% of total activity.
Wait class "Concurrency" was consuming significant database time.
Impact is .1 active sessions, 24.96% of total activity.
Finding 9: Soft Parse
Impact is .02 active sessions, 3.86% of total activity.
Soft parsing of SQL statements was consuming significant database time.
Recommendation 1: Application Analysis
Estimated benefit is .02 active sessions, 3.86% of total activity.
Action
Investigate application logic to keep open the frequently used cursors.
Note that cursors are closed by both cursor close calls and session
disconnects.
Recommendation 2: Database Configuration
Estimated benefit is .02 active sessions, 3.86% of total activity.
Action
Consider increasing the session cursor cache size by increasing the
value of parameter "session_cached_cursors".
Rationale
The value of parameter "session_cached_cursors" was "100" during the
analysis period.
Symptoms That Led to the Finding:
Contention for latches related to the shared pool was consuming
significant database time.
Impact is .09 active sessions, 22.42% of total activity.
Wait class "Concurrency" was consuming significant database time.
Impact is .1 active sessions, 24.96% of total activity.
Finding 10: Unusual "Other" Wait Event
Impact is .01 active sessions, 3.75% of total activity.
Wait event "IPC send completion sync" in wait class "Other" was consuming
significant database time.
Recommendation 1: Application Analysis
Estimated benefit is .01 active sessions, 3.75% of total activity.
Action
Investigate the cause for high "IPC send completion sync" waits. Refer
to Oracle's "Database Reference" for the description of this wait event.
Recommendation 2: Application Analysis
Estimated benefit is .01 active sessions, 3.75% of total activity.
Action
Investigate the cause for high "IPC send completion sync" waits with P1
("send count") value "1".
Recommendation 3: Application Analysis
Estimated benefit is .01 active sessions, 2.59% of total activity.
Action
Investigate the cause for high "IPC send completion sync" waits in
Service "mcmsdrac".
Recommendation 4: Application Analysis
Estimated benefit is .01 active sessions, 1.73% of total activity.
Action
Investigate the cause for high "IPC send completion sync" waits in
Module "TOAD 9.7.2.5".
Symptoms That Led to the Finding:
Wait class "Other" was consuming significant database time.
Impact is .15 active sessions, 38.29% of total activity.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Additional Information
Miscellaneous Information
Wait class "Application" was not consuming significant database time.
Wait class "Commit" was not consuming significant database time.
Wait class "Configuration" was not consuming significant database time.
CPU was not a bottleneck for the instance.
Wait class "Network" was not consuming significant database time.
Wait class "User I/O" was not consuming significant database time.
Session connect and disconnect calls were not consuming significant database
time.
The database's maintenance windows were active during 100% of the analysis
period.
Please help.Hello experts...
Please do the needful... It's really very urgent.
Thanks,
Syed -
Here's is the contents of v$system_event..
Is this
EVENT TOTAL_WAITS TIME_WAITED AVERAGE_WAIT
enq: TX - index contention 40564851 214701526 5.29
enq: TX - row lock contention 188846 12454614 65.95
enq: SQ - contention 141971 70568 0.5
cause for concern?
EVENT TOTAL_WAITS TIME_WAITED AVERAGE_WAIT
SQL*Net message to client 6015051449 607254 0
SQL*Net message from client 6015048542 178177969892 29.62
gcs remote message 2948555287 2633481757 0.89
CGS wait for IPC msg 1517805027 634397 0
db file sequential read 1500615188 816364485 0.54
ges remote message 1247679701 1407300224 1.13
gc cr multi block request 778432813 9913464 0.01
gc current block 2-way 747852637 38030616 0.05
db file scattered read 709428365 460939295 0.65
rdbms ipc message 708473316 37650068633 53.14
gc buffer busy acquire 671285134 1033621285 1.54
PX Deq: reap credit 667784615 484449 0
gcs log flush sync 592376026 171712257 0.29
gc cr block 2-way 530861847 19607062 0.04
library cache pin 437937120 15126237 0.03
log file sync 379523272 797193932 2.1
DIAG idle wait 359607166 2822108755 7.85
log file parallel write 351225436 259263769 0.74
LNS ASYNC end of log 350170653 1398410516 3.99
LNS wait on SENDREQ 321652621 3209301 0.01
PX qref latch 297396661 94308 0
read by other session 289140108 148440270 0.51
buffer deadlock 163505781 983055 0.01
gc current block busy 119223825 467716658 3.92
PX Deq: Table Q Normal 117332841 23574867 0.2
ksxr poll remote instances 110480324 90333 0
buffer busy waits 106938153 19933900 0.19
direct path read 93429599 108427028 1.16
SQL*Net more data from client 86471785 23026529 0.27
gc current grant busy 84978157 28215346 0.33
control file sequential read 82646297 23694583 0.29
PX Deq Credit: send blkd 78641669 9569299 0.12
latch: cache buffers chains 74218671 690277 0.01
gc current grant 2-way 72557796 1920419 0.03
library cache: mutex X 71106697 75993 0
DFS lock handle 70722498 2716407 0.04
gc cr grant 2-way 64558237 1633004 0.03
PX Deq: Execution Msg 61706261 314222076 5.09
gc cr block busy 61469863 119850802 1.95
library cache lock 52428649 3773354 0.07
PX Deq: Slave Session Stats 48040224 1886805 0.04
db file parallel read 46415188 118467902 2.55
IPC send completion sync 46250594 965101 0.02
enq: TX - index contention 40564851 214701526 5.29
PX Deq: Execute Reply 39689685 17243721 0.43
gc buffer busy release 36976909 242714774 6.56
SQL*Net more data to client 36627952 44167 0
PX Deq: Msg Fragment 30501244 343397 0.01
rdbms ipc reply 29725302 1352370 0.05
RMAN backup & recovery I/O 28824547 37722662 1.31
reliable message 27892263 3082134 0.11
PX Idle Wait 27356097 4651277341 170.03
ASM file metadata operation 25098749 8850323 0.35
gc object scan 22705857 7485 0
db file parallel write 19896252 52152606 2.62
latch: ges resource hash list 19336183 427451 0.02
enq: PS - contention 19143961 707455 0.04
PX Deq: Parse Reply 19093356 895799 0.05
gc cr disk read 17816846 448909 0.03
ASM background timer 16101806 1383957874 85.95
PX Deq: Slave Join Frag 16044789 233149 0.01
wait for unread message on broadcast channel 15056320 1413552546 93.88
cursor: mutex X 13435193 24140 0
KJC: Wait for msg sends to complete 13268497 11397 0
PX Deq: Signal ACK RSG 13214824 101941 0.01
KSV master wait 13206286 4235645 0.32
direct path read temp 12617694 5487608 0.43
PX Deq Credit: need buffer 11675868 879967 0.08
row cache lock 11536185 398216 0.03
PX Deq Credit: Session Stats 9480862 78910 0.01
SQL*Net message to dblink 9312894 1538 0
SQL*Net message from dblink 9312894 6279631 0.67
control file parallel write 7760982 11854435 1.53
pmon timer 7558889 1412576090 186.88
PX Deq: Join ACK 7548816 498931 0.07
gc current multi block request 6035173 155898 0.03
PING 5706961 1413230267 247.63
enq: XR - database force logging 4662671 198813 0.04
class slave wait 4561877 7097429006 1555.81
Streams AQ: waiting for messages in the queue 4495828 1543411682 343.3
SQL*Net more data from dblink 3696582 444575 0.12
LGWR wait for redo copy 3655353 17840 0
log file sequential read 3387305 6610414 1.95
Log archive I/O 2990486 276772 0.09
SQL*Net break/reset to client 2971976 2385935 0.8
direct path write temp 2839390 2522114 0.89
Space Manager: slave idle wait 2827526 1412987186 499.73
latch: shared pool 2808517 298150 0.11
latch: gc element 2421717 24688 0.01
SGA: MMAN sleep for component shrink 2336447 2458094 1.05
latch: enqueue hash chains 2279645 15435 0.01
latch free 2089418 78732 0.04
gc current split 2044784 1864009 0.91
PX Deq: Signal ACK EXT 1976164 19263 0.01
enq: FB - contention 1473469 61036 0.04
cursor: pin S wait on X 1313129 1464789 1.12
SQL*Net more data to dblink 1232891 986 0
Streams AQ: RAC qmn coordinator idle wait 1211300 788 0
enq: HW - contention 1175390 2077008 1.77
latch: session allocation 1167768 21883 0.02
Streams AQ: qmn coordinator idle wait 1144699 1412546634 1233.99
Streams AQ: qmn slave idle wait 1031585 2227183681 2158.99
lock deadlock retry 962937 4698 0
enq: CF - contention 956154 609647 0.64
latch: cache buffers lru chain 902764 37552 0.04
latch: object queue header operation 817911 27717 0.03
global enqueue expand wait 768633 654105 0.85
Data file init write 756191 329758 0.44
latch: gcs resource hash 647021 4147 0.01
local write wait 603007 286191 0.47
latch: row cache objects 599358 6453 0.01
ges lmd/lmses to freeze in rcfg - mrcvr 481759 156345 0.32
shared server idle wait 471190 1413238589 2999.3
enq: RF - DG Broker Current File ID 469833 23209 0.05
smon timer 432383 1411851085 3265.28
SGA: allocation forcing component growth 363333 379008 1.04
gc current retry 341104 1121252 3.29
enq: RF - synch: DG Broker metadata 319143 588290 1.84
enq: PG - contention 313659 14830 0.05
enq: TT - contention 260134 11207172 43.08
enq: KO - fast object checkpoint 236745 820808 3.47
dispatcher timer 236637 1413242481 5972.2
direct path write 231382 191008 0.83
cursor: pin S 229011 394 0
Streams AQ: waiting for time management or cleanup tasks 199981 1413148548 7066.41
enq: TX - row lock contention 188846 12454614 65.95
enq: TX - allocate ITL entry 153703 54252 0.35
enq: SQ - contention 141971 70568 0.5
ksdxexeother 141885 56 0
latch: redo allocation 138912 1858 0.01
recovery area: computing applied logs 126415 45925 0.36
gc current block congested 126318 21768 0.17
resmgr:cpu quantum 123074 151384 1.23
jobq slave wait 120678 35574221 294.79
Datapump dump file I/O 90431 9127 0.1
ges inquiry response 89402 4041 0.05
os thread startup 83809 222586 2.66
cr request retry 80062 71896 0.9
PX Deq: Table Q Sample 79665 133402 1.67
gc cr block congested 79026 14792 0.19
gc cr failure 77521 25019 0.32
enq: WF - contention 73983 825388 11.16
enq: TQ - TM contention 72871 3319 0.05
lock escalate retry 65714 1574 0.02
buffer exterminate 59775 64919 1.09
fbar timer 47136 1413183353 29980.98
log file switch completion 46911 452097 9.64
recovery area: computing obsolete files 45699 8547 0.19
enq: US - contention 40401 8805 0.22
enq: TM - contention 39149 5435032 138.83
library cache load lock 36311 382575 10.54
kjbdrmcvtq lmon drm quiesce: ping completion 31668 47443 1.5
enq: TD - KTF dump entries 31468 1424 0.05
enq: RO - fast object reuse 28422 31772 1.12
parallel recovery slave wait for change 27558 3163 0.11
name-service call wait 23694 181533 7.66
control file single write 22375 1624 0.07
kksfbc child completion 21239 106926 5.03
PX Deq: Table Q qref 19325 245 0.01
enq: TX - contention 18805 113253 6.02
latch: messages 17203 181 0.01
enq: RS - prevent file delete 16913 1013 0.06
enq: RS - prevent aging list update 15682 642 0.04
PX Deq: Table Q Get Keys 14322 42935 3
gc current grant congested 14292 2192 0.15
cursor: mutex S 13285 8 0
log file single write 13164 5371 0.41
latch: undo global data 12649 178 0.01
kksfbc research 11894 12680 1.07
parallel recovery slave idle wait 11193 5872 0.52
wait list latch free 11026 11794 1.07
enq: CT - state 11001 417 0.04
latch: checkpoint queue latch 10526 132 0.01
enq: PE - contention 10506 1139 0.11
ARCH wait on SENDREQ 9957 216480 21.74
gc cr grant congested 9465 1584 0.17
wait for scn ack 9377 3155 0.34
enq: TA - contention 8856 324 0.04
log buffer space 8777 89323 10.18
enq: TK - Auto Task Serialization 8542 343 0.04
enq: DR - contention 7842 323 0.04
process diagnostic dump 7707 2072 0.27
JOX Jit Process Sleep 7612 11286431 1482.72
enq: TC - contention 7357 340817 46.33
ges global resource directory to be frozen 7140 12299 1.72
enq: CO - master slave det 6850 312 0.05
enq: JS - job run lock - synchronize 6704 397 0.06
gcs drm freeze in enter server mode 6542 40742 6.23
enq: TS - contention 5959 89332 14.99
ARCH wait for archivelog lock 5600 36 0.01
PX Nsq: PQ load info query 5377 104798 19.49
db file single write 5373 3452 0.64
gc remaster 5315 50625 9.52
latch: parallel query alloc buffer 4939 1906 0.39
enq: TO - contention 4799 143 0.03
enq: AF - task serialization 4395 161 0.04
enq: PI - contention 4251 163 0.04
ges2 LMON to wake up LMD - mrcvr 4210 28 0.01
enq: DL - contention 3889 239 0.06
kjctssqmg: quick message send wait 3408 22 0.01
LNS wait on DETACH 3275 741 0.23
ksfd: async disk IO 3274 1 0
LNS wait on ATTACH 3273 51940 15.87
ARCH wait on DETACH 3231 714 0.22
ARCH wait on ATTACH 3226 43238 13.4
enq: BR - file shrink 2787 116 0.04
write complete waits 2631 1070 0.41
enq: MD - contention 2596 67 0.03
enq: WL - contention 2198 266518 121.25
single-task message 2098 25896 12.34
enq: OD - Serializing DDLs 2054 66 0.03
resmgr:internal state change 2001 14735 7.36
ARCH wait on c/f tx acquire 2 1751 175230 100.07
enq: WR - contention 1636 69 0.04
latch: cache buffer handles 1610 29 0.02
statement suspended, wait error to be cleared 1497 22626 15.11
Streams AQ: qmn coordinator waiting for slave to start 1214 678966 559.28
enq: PD - contention 1182 33 0.03
JS kgl get object wait 1096 4922 4.49
undo segment extension 1070 10065 9.41
PL/SQL lock timer 949 8739819 9209.5
enq: AE - lock 937 28 0.03
LGWR-LNS wait on channel 832 913 1.1
ges DFS hang analysis phase 2 acks 816 495 0.61
latch: redo writing 729 9 0.01
gc quiesce 665 564 0.85
enq: JS - queue lock 482 2111 4.38
PX Deq: Test for credit 442 13 0.03
enq: SS - contention 386 274 0.71
recovery area: computing dropped files 328 1400 4.27
recovery area: computing backed up files 328 496 1.51
ksdxexeotherwait 279 10592 37.97
log switch/archive 250 137570 550.28
gc domain validation 223 39964 179.21
auto-sqltune: wait graph update 195 96514 494.95
wait for a undo record 170 1214 7.14
parallel recovery coord send blocked 168 4 0.02
enq: JS - wdw op 168 3741 22.27
enq: KT - contention 165 5 0.03
switch logfile command 156 6290 40.32
gcs resource directory to be unfrozen 149 12839 86.17
Data Guard Broker Wait 139 10906 78.46
enq: SK - contention 129 4 0.03
enq: JS - job recov lock 128 4 0.03
gc cr block lost 125 6772 54.17
virtual circuit wait 122 3 0.03
ges LMON to get to FTDONE 100 187 1.87
enq: CU - contention 80 242 3.02
enq: JQ - contention 78 7 0.09
cursor: pin X 73 83 1.14
parallel recovery coord wait for reply 70 510 7.29
PX Deq: Txn Recovery Start 67 3436 51.29
SQL*Net break/reset to dblink 60 0 0
gc current block lost 57 2869 50.33
ges LMD suspend for testing event 51 709 13.89
inactive session 46 4550 98.91
recovery read 45 5 0.11
JS kill job wait 41 3548 86.53
enq: AS - service activation 40 1 0.03
enq: TL - contention 35 2 0.05
enq: UL - contention 34 524 15.42
gcs enter server mode 33 1559 47.24
wait for stopper event to be increased 30 218 7.27
enq: TQ - DDL contention 24 300 12.52
enq: MR - contention 21 1 0.03
ges reconfiguration to start 20 54 2.72
ges enter server mode 20 502 25.08
buffer latch 18 1337 74.26
enq: SR - contention 18 1 0.05
Streams: RAC waiting for inter instance ack 18 3748 208.21
enq: PR - contention 17 46 2.72
kupp process wait 16 166 10.39
checkpoint completed 15 3678 245.19
PX Deque wait 14 68 4.87
enq: BF - allocation contention 14 1 0.08
enq: XL - fault extent map 14 51 3.66
enq: FU - contention 14 17 1.18
enq: TH - metric threshold evaluation 13 114 8.78
enq: MW - contention 12 0 0.04
enq: DD - contention 10 0 0.04
process terminate 8 41 5.08
ges cgs registration 8 151 18.9
buffer resize 7 0 0
ktm: instance recovery 7 698 99.66
LNS wait on LGWR 6 0 0
ASM background starting 6 381 63.43
gc cr block 3-way 5 0 0.08
enq: PV - syncstart 5 9 1.74
Global transaction acquire instance locks 4 4 1.09
enq: RS - read alert level 4 0 0.04
LGWR wait on LNS 3 0 0
gc recovery 3 540 179.85
Streams AQ: enqueue blocked on low memory 3 544 181.2
DBWR range invalidation sync 3 17 5.83
enq: DM - contention 3 0 0.03
enq: RF - FSFO Observer Heartbeat 3 0 0.03
enq: JS - q mem clnup lck 3 0 0
DG Broker configuration file I/O 2 0 0
enq: RC - Result Cache: Contention 2 493 246.6
enq: KM - contention 2 0 0.03
enq: RT - contention 2 0 0.04
instance state change 2 0 0.12
kkdlgon 2 10 5.11
enq: TQ - INI contention 2 292 146.07
enq: JS - contention 2 0 0
ARCH wait for netserver start 1 400 400.02
log file switch (checkpoint incomplete) 1 3 3.44
JS coord start wait 1 50 50.09
ges lmd and pmon to attach 1 1 1.26
wait for tmc2 to complete 1 3 3.03
control file heartbeat 1 400 400.02
enq: SW - contention 1 0 0.04
enq: PW - perwarm status in dbw0 1 0 0.09
enq: FS - contention 1 0 0.04
enq: XR - quiesce database 1 0 0.04
enq: RS - write alert level 1 0 0.02
enq: CN - race with init 1 0 0.03
enq: FE - contention 1 4 3.77
Wait for shrink lock2 1 10 10.03
enq: IA - contention 1 0 0.02
enq: RF - atomicity 1 0 0.05
enq: RF - synchronization: aifo master 1 0 0.02
enq: RF - RF - Database Automatic Disable 1 0 0.06
enq: WP - contention 1 0 0.02
enq: TB - SQL Tuning Base Cache Load 1 0 0.05
enq: JS - evt notify 1 0 0.02Edited by: steffi on Mar 20, 2011 12:21 AM
Edited by: steffi on Mar 20, 2011 8:18 AM
Edited by: steffi on Mar 20, 2011 8:19 AMText can be formatted by tagging the beginning and end of the block of text with the code ta
\Formatted text goes here.
\Example:
This is formatted.When cutting and pasting text such as execution plans, excerpts from AWR reports, etc, it will maintain spacing and formatting, and make for much easier reading.
As to the content, well, dumping the contents of v$system_event is pretty close to useless.
As to the first three events you listed, 'enq: TX - index contention', 'enq: TX - row lock contention', 'enq: SQ - contention', well, all of those are easily tunable.
First, for 'enq: SQ - contention', check your sequences. Do you have any NOCACHE sequences? Or sequences with small caches?
As for 'enq: TX - row lock contention', well that's fairly self-explanatory. You have multiple sessions trying to lock the same row in the same table at the same time.
Last, 'enq: TX - index contention', that's non-row level contention on an index. For example, if you have a unique index, insert a row w/ column value 1, but don't commit, then try to insert that same value from another session.
But, before you do any of that, I think you need to clearly understand where the bottlenecks are. Try taking some AWR snapshots, about 5 minutes apart, when you're having performance problems. Look at the AWR report for that 5 minute period. In particular, look at your Top 5 timed events.
Hope that helps,
-Mark -
Dear all,
My version is 11.2.0.2.5 one of my rac instance crashes with message ORA-00240: control file enqueue held for more than 120 seconds. Received an instance abort message from instance 1.
here are the contents of alert log file
IPC Send timeout detected. Receiver ospid 27423 [[email protected] (LMON)]
2013-03-22 22:30:05.644000 -07:00
Errors in file /u01/app/oracle/diag/rdbms/lfgoimdb/LFGoimdb2/trace/LFGoimdb2_lmon_27423.trc:
2013-03-22 22:31:08.734000 -07:00
Errors in file /u01/app/oracle/diag/rdbms/lfgoimdb/LFGoimdb2/trace/LFGoimdb2_arc2_27691.trc (incident=15905):
ORA-00240: control file enqueue held for more than 120 seconds
Incident details in: /u01/app/oracle/diag/rdbms/lfgoimdb/LFGoimdb2/incident/incdir_15905/LFGoimdb2_arc2_27691_i15905.trc
2013-03-22 22:31:13.409000 -07:00
Received an instance abort message from instance 1
Please check instance 1 alert and LMON trace files for detail.
LMS0 (ospid: 27427): terminating the instance due to error 481
System state dump requested by (instance=2, osid=27427 (LMS0)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/lfgoimdb/LFGoimdb2/trace/LFGoimdb2_diag_27413.trc
2013-03-22 22:31:18.376000 -07:00
Dumping diagnostic data in directory=[cdmp_20130322223113], requested by (instance=2, osid=27427 (LMS0)), summary=[abnormal instance termination].
ORA-1092 : opitsk aborting process
Instance terminated by LMS0, pid = 27427Thanks for reply,
My redo logs size is default 50mb.There is currently no load on the system since we are not using this environment for time being.The log switches are averaged to be 8 per day.I think Increasing the size of redo will further cause the problems since the archiver may again hold lock for more time.
Since there is no dedicated connection between the nodes and storage ,So increasing the hardware and network configuration is only solution to this? Or I am still missing something...
As far as configuration is considered i cannot add more resources to this environment.How can I solve this issue? -
Hi,
We are facing RAC Interconnect performance problems.
Oracle Version: Oracle 9i RAC (9.2.0.7)
Operating system: SunOS 5.8
SQL> SELECT b1.inst_id, b2.value "RECEIVED",
b1.value "RECEIVE TIME",
((b1.value / b2.value) * 10) "AVG RECEIVE TIME (ms)"
FROM gv$sysstat b1, gv$sysstat b2
WHERE b1.name = 'global cache cr block receive time'
AND b2.name = 'global cache cr blocks received'
AND b1.inst_id = b2.inst_id;
INST_ID RECEIVED RECEIVE TIME AVG RECEIVE TIME (ms)
1 323849 172359 5.32220263
2 675806 94537 1.39887778
After database restart average time increases for Instance 1 and instance 2 remains similar.
Application performance degrades, restart database solves the issue. This is critical application and can not have frequent downtimes for restart.
What specific points should I check to find out to improve interconnect performance?
Thanks
Dilip Patel.Hi,
Configurations:
Node: 1
Hardware Model: Sun-Fire-V890
OS: SunOS 5.8
Release: Generic_117350-53
CPU: 16 sparcv9 cpu(s) running at 1200 MHz
Memory: 40.0GB
Node: 2
Hardware Model: Sun-Fire-V890
OS: SunOS 5.8
Release: Generic_117350-53
CPU: 16 sparcv9 cpu(s) running at 1200 MHz
Memory: 40.0GB
CPU Utilization on Node 1 is never exceeded 40%.
CPU Utilization on Node 2 is between 20% to 30%.
Application load is more Node 1 compared to Node 2.
I can observer wait event "global cache cr request" in top 5 wait events on most of the statspack report. Application faces degrade performacne after few days of restart database. No major changes done on application recently.
Statapack report for Node 1:
DB Name DB Id Instance Inst Num Release Cluster Host
XXXX 2753907139 xxxx1 1 9.2.0.7.0 YES xxxxx
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: 61688 17-Feb-09 09:10:06 253 299.4
End Snap: 61698 17-Feb-09 10:10:06 285 271.6
Elapsed: 60.00 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 2,048M Std Block Size: 8K
Shared Pool Size: 384M Log Buffer: 2,048K
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: 102,034.92 4,824.60
Logical reads: 60,920.35 2,880.55
Block changes: 986.07 46.63
Physical reads: 1,981.12 93.67
Physical writes: 28.30 1.34
User calls: 2,651.63 125.38
Parses: 500.89 23.68
Hard parses: 21.44 1.01
Sorts: 66.91 3.16
Logons: 3.69 0.17
Executes: 553.34 26.16
Transactions: 21.15
% Blocks changed per Read: 1.62 Recursive Call %: 22.21
Rollback per transaction %: 2.90 Rows per Sort: 7.44
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 99.99 Redo NoWait %: 100.00
Buffer Hit %: 96.75 In-memory Sort %: 100.00
Library Hit %: 98.30 Soft Parse %: 95.72
Execute to Parse %: 9.48 Latch Hit %: 99.37
Parse CPU to Parse Elapsd %: 90.03 % Non-Parse CPU: 92.97
Shared Pool Statistics Begin End
Memory Usage %: 94.23 94.93
% SQL with executions>1: 74.96 74.66
% Memory for SQL w/exec>1: 82.93 72.26
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
db file sequential read 1,080,532 13,191 40.94
CPU time 10,183 31.60
db file scattered read 456,075 3,977 12.34
wait for unread message on broadcast channel 4,195 2,770 8.60
global cache cr request 1,633,056 873 2.71
Cluster Statistics for DB: EPIP Instance: epip1 Snaps: 61688 -61698
Global Cache Service - Workload Characteristics
Ave global cache get time (ms): 0.8
Ave global cache convert time (ms): 1.1
Ave build time for CR block (ms): 0.1
Ave flush time for CR block (ms): 0.2
Ave send time for CR block (ms): 0.3
Ave time to process CR block request (ms): 0.6
Ave receive time for CR block (ms): 4.4
Ave pin time for current block (ms): 0.2
Ave flush time for current block (ms): 0.0
Ave send time for current block (ms): 0.3
Ave time to process current block request (ms): 0.5
Ave receive time for current block (ms): 2.6
Global cache hit ratio: 3.9
Ratio of current block defers: 0.0
% of messages sent for buffer gets: 3.7
% of remote buffer gets: 0.3
Ratio of I/O for coherence: 1.1
Ratio of local vs remote work: 10.9
Ratio of fusion vs physical writes: 0.0
Global Enqueue Service Statistics
Ave global lock get time (ms): 0.1
Ave global lock convert time (ms): 0.0
Ratio of global lock gets vs global lock releases: 1.0
GCS and GES Messaging statistics
Ave message sent queue time (ms): 0.4
Ave message sent queue time on ksxp (ms): 1.8
Ave message received queue time (ms): 0.2
Ave GCS message process time (ms): 0.1
Ave GES message process time (ms): 0.0
% of direct sent messages: 8.0
% of indirect sent messages: 49.4
% of flow controlled messages: 42.6
GES Statistics for DB: EPIP Instance: epip1 Snaps: 61688 -61698
Statistic Total per Second per Trans
dynamically allocated gcs resourc 0 0.0 0.0
dynamically allocated gcs shadows 0 0.0 0.0
flow control messages received 0 0.0 0.0
flow control messages sent 0 0.0 0.0
gcs ast xid 0 0.0 0.0
gcs blocked converts 2,830 0.8 0.0
gcs blocked cr converts 7,677 2.1 0.1
gcs compatible basts 5 0.0 0.0
gcs compatible cr basts (global) 142 0.0 0.0
gcs compatible cr basts (local) 142,678 39.6 1.9
gcs cr basts to PIs 0 0.0 0.0
gcs cr serve without current lock 0 0.0 0.0
gcs error msgs 0 0.0 0.0
gcs flush pi msgs 798 0.2 0.0
gcs forward cr to pinged instance 0 0.0 0.0
gcs immediate (compatible) conver 9,296 2.6 0.1
gcs immediate (null) converts 52,460 14.6 0.7
gcs immediate cr (compatible) con 752,507 209.0 9.9
gcs immediate cr (null) converts 4,047,959 1,124.4 53.2
gcs msgs process time(ms) 153,618 42.7 2.0
gcs msgs received 2,287,640 635.5 30.0
gcs out-of-order msgs 0 0.0 0.0
gcs pings refused 70,099 19.5 0.9
gcs queued converts 0 0.0 0.0
gcs recovery claim msgs 0 0.0 0.0
gcs refuse xid 1 0.0 0.0
gcs retry convert request 0 0.0 0.0
gcs side channel msgs actual 40,400 11.2 0.5
gcs side channel msgs logical 4,039,700 1,122.1 53.1
gcs write notification msgs 46 0.0 0.0
gcs write request msgs 972 0.3 0.0
gcs writes refused 4 0.0 0.0
ges msgs process time(ms) 2,713 0.8 0.0
ges msgs received 73,687 20.5 1.0
global posts dropped 0 0.0 0.0
global posts queue time 0 0.0 0.0
global posts queued 0 0.0 0.0
global posts requested 0 0.0 0.0
global posts sent 0 0.0 0.0
implicit batch messages received 288,801 80.2 3.8
implicit batch messages sent 622,610 172.9 8.2
lmd msg send time(ms) 2,148 0.6 0.0
lms(s) msg send time(ms) 1 0.0 0.0
messages flow controlled 3,473,393 964.8 45.6
messages received actual 765,292 212.6 10.1
messages received logical 2,360,972 655.8 31.0
messages sent directly 654,760 181.9 8.6
messages sent indirectly 4,027,924 1,118.9 52.9
msgs causing lmd to send msgs 33,481 9.3 0.4
msgs causing lms(s) to send msgs 13,220 3.7 0.2
msgs received queue time (ms) 379,304 105.4 5.0
msgs received queued 2,359,723 655.5 31.0
msgs sent queue time (ms) 1,514,305 420.6 19.9
msgs sent queue time on ksxp (ms) 4,349,174 1,208.1 57.1
msgs sent queued 4,032,426 1,120.1 53.0
msgs sent queued on ksxp 2,415,381 670.9 31.7
GES Statistics for DB: EPIP Instance: epip1 Snaps: 61688 -61698
Statistic Total per Second per Trans
process batch messages received 278,174 77.3 3.7
process batch messages sent 913,611 253.8 12.0
Wait Events for DB: EPIP Instance: epip1 Snaps: 61688 -61698
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
db file sequential read 1,080,532 0 13,191 12 14.2
db file scattered read 456,075 0 3,977 9 6.0
wait for unread message on b 4,195 1,838 2,770 660 0.1
global cache cr request 1,633,056 8,417 873 1 21.4
db file parallel write 8,243 0 260 32 0.1
buffer busy waits 16,811 0 168 10 0.2
log file parallel write 187,783 0 158 1 2.5
log file sync 75,143 0 147 2 1.0
buffer busy global CR 9,713 0 102 10 0.1
global cache open x 31,157 1,230 50 2 0.4
enqueue 58,261 14 45 1 0.8
latch free 33,398 7,610 44 1 0.4
direct path read (lob) 9,925 0 36 4 0.1
library cache pin 8,777 1 34 4 0.1
SQL*Net break/reset to clien 82,982 0 32 0 1.1
log file sequential read 409 0 31 75 0.0
log switch/archive 3 3 29 9770 0.0
SQL*Net more data to client 201,538 0 16 0 2.6
global cache open s 8,585 342 14 2 0.1
global cache s to x 11,098 148 11 1 0.1
control file sequential read 6,845 0 8 1 0.1
db file parallel read 1,569 0 7 4 0.0
log file switch completion 35 0 7 194 0.0
row cache lock 15,780 0 6 0 0.2
process startup 69 0 6 82 0.0
global cache null to x 1,759 48 6 3 0.0
direct path write (lob) 685 0 5 7 0.0
DFS lock handle 8,713 0 3 0 0.1
control file parallel write 1,350 0 2 2 0.0
wait for master scn 1,194 0 1 1 0.0
CGS wait for IPC msg 30,830 30,715 1 0 0.4
global cache busy 14 1 1 75 0.0
ksxr poll remote instances 30,997 12,692 1 0 0.4
direct path read 752 0 0 1 0.0
switch logfile command 3 0 0 148 0.0
log file single write 24 0 0 13 0.0
library cache lock 668 0 0 0 0.0
KJC: Wait for msg sends to c 1,161 0 0 0 0.0
buffer busy global cache 26 0 0 6 0.0
IPC send completion sync 261 260 0 0 0.0
PX Deq: reap credit 3,477 3,440 0 0 0.0
LGWR wait for redo copy 1,751 0 0 0 0.0
async disk IO 1,059 0 0 0 0.0
direct path write 298 0 0 0 0.0
slave TJ process wait 1 1 0 18 0.0
PX Deq: Execute Reply 3 1 0 3 0.0
PX Deq: Join ACK 8 4 0 1 0.0
global cache null to s 8 0 0 1 0.0
ges inquiry response 16 0 0 0 0.0
Wait Events for DB: EPIP Instance: epip1 Snaps: 61688 -61698
-> s - second
-> cs - centisecond - 100th of a second
-> ms - millisecond - 1000th of a second
-> us - microsecond - 1000000th of a second
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
PX Deq: Parse Reply 6 2 0 1 0.0
PX Deq Credit: send blkd 2 1 0 0 0.0
PX Deq: Signal ACK 3 1 0 0 0.0
library cache load lock 1 0 0 0 0.0
buffer deadlock 6 6 0 0 0.0
lock escalate retry 4 4 0 0 0.0
SQL*Net message from client 9,470,867 0 643,285 68 124.4
queue messages 42,829 41,144 42,888 1001 0.6
wakeup time manager 601 600 16,751 27872 0.0
gcs remote message 795,414 120,163 13,606 17 10.4
jobq slave wait 2,546 2,462 7,375 2897 0.0
PX Idle Wait 2,895 2,841 7,021 2425 0.0
virtual circuit status 120 120 3,513 29273 0.0
ges remote message 142,306 69,912 3,504 25 1.9
SQL*Net more data from clien 206,559 0 19 0 2.7
SQL*Net message to client 9,470,903 0 14 0 124.4
PX Deq: Execution Msg 313 103 2 7 0.0
Background Wait Events for DB: EPIP Instance: epip1 Snaps: 61688 -61698
-> ordered by wait time desc, waits desc (idle events last)
Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
db file parallel write 8,243 0 260 32 0.1
log file parallel write 187,797 0 158 1 2.5
log file sequential read 316 0 22 70 0.0
enqueue 56,204 0 15 0 0.7
control file sequential read 5,694 0 6 1 0.1
DFS lock handle 8,682 0 3 0 0.1
db file sequential read 276 0 2 8 0.0
control file parallel write 1,334 0 2 2 0.0
wait for master scn 1,194 0 1 1 0.0
CGS wait for IPC msg 30,830 30,714 1 0 0.4
ksxr poll remote instances 30,972 12,681 1 0 0.4
latch free 356 54 1 2 0.0
direct path read 752 0 0 1 0.0
log file single write 24 0 0 13 0.0
LGWR wait for redo copy 1,751 0 0 0 0.0
async disk IO 812 0 0 0 0.0
global cache cr request 69 0 0 1 0.0
row cache lock 45 0 0 1 0.0
direct path write 298 0 0 0 0.0
library cache pin 29 0 0 1 0.0
rdbms ipc reply 29 0 0 0 0.0
buffer busy waits 10 0 0 0 0.0
library cache lock 2 0 0 0 0.0
global cache open x 2 0 0 0 0.0
rdbms ipc message 179,764 36,258 29,215 163 2.4
gcs remote message 795,409 120,169 13,605 17 10.4
pmon timer 1,388 1,388 3,508 2527 0.0
ges remote message 142,295 69,912 3,504 25 1.9
smon timer 414 0 3,463 8366 0.0
------------------------------------------------------------- -
Query performance on RAC is a lot slower than single instance
I simply followed the steps provided by oracle to install a rac db of 2 nodes.
The performce on Insertion (java, thin ojdbc) is pretty much the same compared to a single instance on NFS
However the performance on the select query is very slow compared to single instance.
I have tried using different methods for the storage configuration (asm with raw, ocfs2) but the performance is still slow.
When I shut down one instance, leaving only one instance up, the query performance is very fast (as fast as one single instance)
I am using rhel5 64 bit (16G of physical memory) and oracle 11.1.0.6 with patchset 11.1.0.7
Could someone help me how to debug this problem?
Thanks,
Chau
Edited by: user638637 on Aug 6, 2009 8:31 AMtop 5 timed foreground events:
DB CPU: times 943(s), %db time (47.5%)
cursor.pin S wait on X: wait(13940), time (321s), avg wait(23ms), %db time (16.15%)
direct path read (95,436), times (288), avg watie (3ms), %db ime (14.51%)
IPC send completion sync: wait(546,712), times(149s), avg wait (0), %db time (7.49%)
gc cr multi block request: waits (7574), teims (78) avg wait (10 ms), %db time (4.0)
another thing i see is the "avg global cache cr block flush time (ms): is 37.6 msThe DB CPU Oracle metric is the amount of CPU time (in microseconds) spent on database user-level calls.
You should check your sql statement from report and tuning them.
- Check from Execute Plan.
- If not index, determine to use index.
SQL> set autot trace explain
SQL> sql statement;
cursor: pin S wait on X.
A session waits on this event when requesting a mutex for sharable operationsrelated to pins (such as executing a cursor), but the mutex cannot be granted becauseit is being held exclusively by another session (which is most likely parsing the cursor).
use variable SQL , avoid dynamic sql
http://blog.tanelpoder.com/2008/08/03/library-cache-latches-gone-in-oracle-11g/
check about memory MEMORY_TARGET initialization parameter.
By the way you have high "DB CPU"(47.5%), you should tune about your sql statement (check sql in report and tune)
Good Luck -
一套2节点的10.2.0.4的RAC ,不定期出现一节点（节点1）无法逐出未自动重启，导致整个RAC被HANG住了。故障发生时被踢出节点OS无法登录控制台查看系统资源使用情况，也没有该节点相关数据库日志，只有另一节点的。是不是也得从五台山请个小狮子回来镇一下呢。还是请刘大帮忙分析一下其中原因。节点2的相关日志已发往您的邮箱。
OS: REDHAT 4
相关日志：
节点1的系统日志：
Oct 31 15:30:28 bf01 sshd(pam_unix)[3356]: session closed for user mocha
Oct 31 15:30:45 bf01 kernel: Losing some ticks... checking if CPU frequency changed.
Oct 31 15:31:08 bf01 kernel: warning: many lost ticks.
Oct 31 15:31:08 bf01 kernel: Your time source seems to be instable or some driver is hogging interupts
Oct 31 15:31:08 bf01 kernel: rip __do_softirq+0x4d/0xd0 <----------之后便无响应，冷启动该节点
Oct 31 15:58:56 bf01 syslogd 1.4.1: restart.
Oct 31 15:58:56 bf01 syslog: syslogd startup succeeded
Oct 31 15:58:56 bf01 kernel: klogd 1.4.1, log source = /proc/kmsg started.
BF02的数据库日志：
Wed Oct 31 15:17:15 2012
Thread 2 advanced to log sequence 104758 (LGWR switch)
Current log# 6 seq# 104758 mem# 0: /dev/raw/raw23
Wed Oct 31 15:36:30 2012
IPC Send timeout detected.Sender: ospid 20539
Receiver: inst 1 binc 429417533 ospid 20263
Wed Oct 31 15:36:31 2012
IPC Send timeout detected.Sender: ospid 20982
Receiver: inst 1 binc 429417533 ospid 20263
Wed Oct 31 15:36:50 2012
IPC Send timeout detected.Sender: ospid 10334
Receiver: inst 1 binc 429417533 ospid 20263
Wed Oct 31 15:36:58 2012
IPC Send timeout to 0.0 inc 4 for msg type 12 from opid 60
Wed Oct 31 15:36:58 2012
Communications reconfiguration: instance_number 1
Wed Oct 31 15:37:02 2012
IPC Send timeout to 0.0 inc 4 for msg type 8 from opid 14
Wed Oct 31 15:37:22 2012
IPC Send timeout to 0.0 inc 4 for msg type 16 from opid 160
Wed Oct 31 15:37:26 2012
IPC Send timeout detected.Sender: ospid 31899
Receiver: inst 1 binc 429417533 ospid 20263
Wed Oct 31 15:37:32 2012
Trace dumping is performing id=[cdmp_20121031153701]
Wed Oct 31 15:37:35 2012
IPC Send timeout detected.Sender: ospid 20549
Receiver: inst 1 binc 429417533 ospid 20263
Wed Oct 31 15:37:58 2012
IPC Send timeout to 0.0 inc 4 for msg type 16 from opid 507
Wed Oct 31 15:38:00 2012
IPC Send timeout to 0.0 inc 4 for msg type 8 from opid 19
Wed Oct 31 15:38:11 2012
IPC Send timeout detected.Sender: ospid 20543
Receiver: inst 1 binc 429417533 ospid 20263
Wed Oct 31 15:38:18 2012
Trace dumping is performing id=[cdmp_20121031153701]
Wed Oct 31 15:38:41 2012
IPC Send timeout detected.Sender: ospid 12157
Receiver: inst 1 binc 429417533 ospid 20263
Wed Oct 31 15:38:43 2012
IPC Send timeout to 0.0 inc 4 for msg type 16 from opid 16
Wed Oct 31 15:38:48 2012
Evicting instance 1 from cluster
Wed Oct 31 15:39:12 2012
IPC Send timeout to 0.0 inc 4 for msg type 16 from opid 145
Wed Oct 31 15:39:29 2012
Waiting for instances to leave:
1
Wed Oct 31 15:39:38 2012
IPC Send timeout detected.Sender: ospid 15417
Receiver: inst 1 binc 429417533 ospid 20263
Wed Oct 31 15:39:49 2012
Waiting for instances to leave:
1
后面就一直在等待节点1踢出，持续了将近10多分钟。使用crs_stat -t 查看也无响应，RAC似乎被HANG住，因此后面手动重启节点1后恢复正常。cat /etc/issues
看一下
Oct 31 15:31:08 bf01 kernel: Your time source seems to be instable or some driver is hogging interupts
Oct 31 15:31:08 bf01 kernel: rip __do_softirq+0x4d/0xd0 <----------之后便无响应，冷启动该节点
Oct 31 15:58:56 bf01 syslogd 1.4.1: restart.可能是Linux的一个 BUG;
Configuration Feedback
Bug No 4593892
Summary repeated warnings in dmesg about lost ticks (it 80129/bz 139376)
Affects usually seen on x86-64, both Intel and Opteron based systems
Symptom dmesg will have errors like: warning: many lost ticks. Your time source seems to be instable or some driver is hogging interrupts rip __do_softirq+0x4d/0xd0
Solution/Workaround Fixed in U5
安装Linux 4 Update U5 修改该问题
改造Oracle中文技术生态环境！
*<font color="red" size="2" face="courier">如果觉得本回复有意义，请点击本条回复右手边的Correct按钮，谢谢！</font>*
如何在OTN中文技术论坛提一个问题？
论坛礼仪需知及学习oracle的方法论
Maclean Liu
Oracle Database Administrator
Oracle Certified 10g/11g Master
www.askmaclean.com
请把需要上传的文件发给我
有问题请去OTN中文论坛开个帖子 我会回复 地址：http://www.otncn.org
如果需要发送附件，可以直接发邮件到 [email protected] -
Global Resource Directory frozen oracle rac database
I have 2 node oracle cluster on x86_64 solaris 10 system and i am getting the following error while starting asm instance on node2.
Here are the error in the log file.
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
IPC Send timeout detected. Sender: ospid 2388 [[email protected] (PING)]
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
I have disable the firewalls on both the nodes.
Can any one please help me on this.Without four digit version info?
No way, Jose
Sybrand Bakker
Senior Oracle DBA -
Hi all,
I am searching for doc on Oracle 10G on RAC.Please send me the related link.
Thanks in advance
VipinLook into this URL for Oracle Documentations on Oracle 10g RAC.
http://www.oracle.com/pls/db102/portal.portal_db?selected=3
Fore more related links, take a look at the following link.
http://syedracdba.wordpress.com/2008/03/24/list-of-useful-links-for-oracle-rac/
Regards,
Sabdar Syed, -
I need help on IPC-VMC how to config.
Hi ,
System SRM 5.5, ECC 6.0 and AS IPC VMC 7.0.
I am working first time on IPC, Send me configuration document or steps IPC VMC. I would be highly appreciated.
I think IPC has to configure in SM53. But I donu2019t know which one has to config first and how link SRM conditions and ECC conditions. I really need your help. Thank in advance.
Regards,
John.Hi Srikanth,
I am sorry for delay. This has given some information, thanks a lot for your help. In my scenario we are using master record for every code (Material and Services). My doubt is, Shell we create a Pricing conditions and Tax procedures in SRM and assign them into Master record?. If yes how can I maintain combination of Plant/Vendor/Material in SRM?.
Why means, It is Indian Project, weather this IPC will work for this country or not?. Thatu2019s why I am thinking alternative solution also. If you have any solutions please help to address this issue. Any how I want to configure ECS by using BADI, BAPI, Developments or anything for this issue. I hope the best will get some solutions.
Regards,
John. -
Dear All
Can any one help me in stallaing Oracle DB with RAC, and send me a manual or site that will help step-by-step installation.
i want to intsall it with the following specifications.
1) the server details and memory
x3650, Xeon Quad X5355 120W 2.33GHz/13333Mhz/2x4MB L2, 2x1GB Chk,o/Bay
Intel Xeon Quad Core Processor Model X5355 120W 2.33Ghz/1333 MHz/8MB L2
2GB (2x1GB) PC2-5300 CL5 ECCDDR2 Chipkill FBDIMM Memory kit ( add 2 GB RAM for each)
2xIBM 146 GB 15 K 3.5 HS SAS HDD ( add HD 146 GB for eaxh oracle Database server)
IBM MegaRAID 8480 SAS Adapter
xseries 835 Redundant Power option
with SAN Storage .
2) operating system Windows 2003 64-bit.
3) its a 2 node RAC.
4) storage or file system to use OCFS2.
5) ORacle Version ORacle 10gR2.
with m y best wishes.Mrashaideh,is Link mentioned in following thread not helped?
RAC 10G With SAN storage -
When there is archiving done to the recovery area, there is "rdbms ipc message" wait event spike.
What to conclude from it?
Wait Class Wait Event P1 Text P1 P2 Text P2 P3 Text P3 Wait Time (ms)
System I/O log file parallel write files 2 blocks 4096 requests 2 10
Idle rdbms ipc message timeout 150 0 0 90
System I/O log file parallel write files 2 blocks 4096 requests 2 10
Idle rdbms ipc message timeout 161 0 0 100
System I/O log file parallel write files 2 blocks 4096 requests 2 10
Idle rdbms ipc message timeout 181 0 0 190
System I/O log file parallel write files 2 blocks 4096 requests 4 10
Idle rdbms ipc message timeout 191 0 0 90
System I/O log file parallel write files 2 blocks 4096 requests 2 10
Idle rdbms ipc message timeout 201 0 0 90
ThanksYou can use the following details
Wait Class Wait Event P1 Text P1 P2 Wait Time(ms)
System I/O log file parallel write files 2 4096 10
Idle rdbms ipc message timeout 150 0 90
System I/O log file parallel write files 2 4096 10
Idle rdbms ipc message timeout 161 0 100
System I/O log file parallel write files 2 4096 10
Idle rdbms ipc message timeout 181 0 190
System I/O log file parallel write files 2 4096 10
Idle rdbms ipc message timeout 191 0 90
System I/O log file parallel write files 2 4096 10
Idle rdbms ipc message timeout 201 0 90 -
Cleint tnsnames.ora for connecting to RAC instance
Dear all,
DB:10.2.0.1
o/s:redhat 5.6
2node rac cluster
I am getting ERROR: ORA-12170: TNS:Connect timeout occurred when trying to connect to rac instance.the timeout frequency is very less once in 10 times.
cleint tnsname.ora
RACTEST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.2.25)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.2.27)(PORT = 1521))
(FAILOVER= TRUE)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = RACTEST)
RegardsTry to connect straight to the nodes. One node at the time. You can add new tnsnames entries for all RAC nodes so you can connect only one node not the whole RAC.
example:
RACTEST1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.2.25)(PORT = 1521))
(CONNECT_DATA =
(SERVER = dedicated)
(SERVICE_NAME = RACTEST)
(INSTANCE_NAME = RACTEST1)
RACTEST2 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.2.27)(PORT = 1521))
(CONNECT_DATA =
(SERVER = dedicated)
(SERVICE_NAME = RACTEST)
(INSTANCE_NAME = RACTEST2)
If both nodes are working then problem is probably in network / firewall.
Usually this kind of problems comes because one node's database instance is somehow broken. And if that is the problem correction is restart failing instance.
Edited by: Simo Kemppinen on Jul 15, 2012 12:58 PM
Edited by: Simo Kemppinen on Jul 15, 2012 12:59 PM -
Automatic Parallelism causes Merge statement to take longer.
We have a problem in a new project as part of the ETL load into the Oracle datawarehouse we perform a merge statement to update rows in a global temporary table then load
the results into a permanant table, when testing with automatic parallel execution enabled the plan changes and the merge never finishes and consumes vast amounts of resources.
The database version is:-
Database version :11.2.0.3
OS: redhat 64bit
three node rac 20 cores per node
when executing serially the query response is typically similar to the following:
MERGE /*+ gather_plan_statistics no_parallel */ INTO T_GTTCHARGEVALUES USING
(SELECT
CASTACCOUNTID,
CHARGESCHEME,
MAX(CUMULATIVEVALUE) AS MAXMONTHVALUE,
MAX(CUMULATIVECOUNT) AS MAXMONTHCOUNT
FROM
V_CACHARGESALL
WHERE
CHARGEDATE >= TRUNC(TO_DATE(:B1,'YYYY-MM-DD'),'MM')
AND CHARGEDATE < TO_DATE(:B1,'YYYY-MM-DD')
GROUP BY
CASTACCOUNTID,
CHARGESCHEME
HAVING MAX(CUMULATIVECOUNT) IS NOT NULL ) MTOTAL
ON
(T_GTTCHARGEVALUES.CASTACCOUNTID=MTOTAL.CASTACCOUNTID AND
T_GTTCHARGEVALUES.CHARGESCHEME=MTOTAL.CHARGESCHEME)
WHEN MATCHED
THEN UPDATE SET
CUMULATIVEVALUE=CUMULATIVEVALUE+MTOTAL.MAXMONTHVALUE ,
CUMULATIVECOUNT=CUMULATIVECOUNT+MTOTAL.MAXMONTHCOUNT;
1448340 rows merged.
select * from table(dbms_xplan.display_cursor(null,null,'ALLSTATS LAST'));
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | OMem | 1Mem | Used-Mem |
| 0 | MERGE STATEMENT | | 1 | | 0 |00:03:08.43 | 2095K| 186K| | | |
| 1 | MERGE | T_GTTCHARGEVALUES | 1 | | 0 |00:03:08.43 | 2095K| 186K| | | |
| 2 | VIEW | | 1 | | 1448K|00:02:53.14 | 619K| 177K| | | |
|* 3 | HASH JOIN | | 1 | 1 | 1448K|00:02:52.70 | 619K| 177K| 812K| 812K| 1218K (0)|
| 4 | VIEW | | 1 | 1 | 203 |00:02:51.26 | 608K| 177K| | | |
|* 5 | FILTER | | 1 | | 203 |00:02:51.26 | 608K| 177K| | | |
| 6 | SORT GROUP BY | | 1 | 1 | 480 |00:02:51.26 | 608K| 177K| 73728 | 73728 | |
|* 7 | FILTER | | 1 | | 21M|00:02:56.04 | 608K| 177K| | | |
| 8 | PARTITION RANGE ITERATOR| | 1 | 392K| 21M|00:02:51.32 | 608K| 177K| | | |
|* 9 | TABLE ACCESS FULL | T_CACHARGES | 24 | 392K| 21M|00:02:47.48 | 608K| 177K| | | |
| 10 | TABLE ACCESS FULL | T_GTTCHARGEVALUES | 1 | 1451K| 1451K|00:00:00.48 | 10980 | 0 | | | |
Predicate Information (identified by operation id):
3 - access("T_GTTCHARGEVALUES"."CASTACCOUNTID"="MTOTAL"."CASTACCOUNTID" AND "T_GTTCHARGEVALUES"."CHARGESCHEME"="MTOTAL"."CHARGESCHEME")
5 - filter(MAX("CUMULATIVECOUNT") IS NOT NULL)
7 - filter(TRUNC(TO_DATE(:B1,'YYYY-MM-DD'),'fmmm')<TO_DATE(:B1,'YYYY-MM-DD'))
9 - filter(("LOGICALLYDELETED"=0 AND "CHARGEDATE">=TRUNC(TO_DATE(:B1,'YYYY-MM-DD'),'fmmm') AND "CHARGEDATE"<TO_DATE(:B1,'YYYY-MM-DD')))removing the no_parallel hint results in the following, (this is pulled from the sql monitoring report and editied to remove the lines relating to individual parallel servers)
I understand that the query is considered for parallel execution due to the estimated length of time it will run for and although the degree of parallleism seems excessive
it is the default maximum for the server configuration, what we are tryig to understand is which statistics could be inacurate or missing and could cause this kind of problem.
In this case we can add the no_parallel hint in the etl package as a workaround but would really liek to identify the root cause to avoid similar problems elsewhere.
SQL Monitoring Report
SQL Text
MERGE INTO T_GTTCHARGEVALUES USING (SELECT CASTACCOUNTID, CHARGESCHEME, MAX(CUMULATIVEVALUE) AS MAXMONTHVALUE,
MAX(CUMULATIVECOUNT) AS MAXMONTHCOUNT FROM V_CACHARGESALL WHERE CHARGEDATE >= TRUNC(TO_DATE(:B1,'YYYY-MM-DD'),'MM')
AND CHARGEDATE < to_date(:B1,'YYYY-MM-DD')
GROUP BY CASTACCOUNTID, CHARGESCHEME HAVING MAX(CUMULATIVECOUNT) IS NOT NULL ) MTOTAL
ON (T_GTTCHARGEVALUES.CASTACCOUNTID=MTOTAL.CASTACCOUNTID AND
T_GTTCHARGEVALUES.CHARGESCHEME=MTOTAL.CHARGESCHEME) WHEN MATCHED THEN UPDATE SET
CUMULATIVEVALUE=CUMULATIVEVALUE+MTOTAL.MAXMONTHVALUE ,
CUMULATIVECOUNT=CUMULATIVECOUNT+MTOTAL.MAXMONTHCOUNT
Error: ORA-1013
ORA-01013: user requested cancel of current operation
Global Information
Status : DONE (ERROR)
Instance ID : 1
Session : XXXX(2815:12369)
SQL ID : 70kzttjbyyspt
SQL Execution ID : 16777216
Execution Started : 04/27/2012 09:43:27
First Refresh Time : 04/27/2012 09:43:27
Last Refresh Time : 04/27/2012 09:48:43
Duration : 316s
Module/Action : SQL*Plus/-
Service : SYS$USERS
Program : sqlplus@XXXX (TNS V1-V3)
Binds
========================================================================================================================
| Name | Position | Type | Value |
========================================================================================================================
| :B1 | 1 | VARCHAR2(32) | 2012-04-25 |
========================================================================================================================
Global Stats
====================================================================================================================
| Elapsed | Queuing | Cpu | IO | Application | Concurrency | Cluster | Other | Buffer | Read | Read |
| Time(s) | Time(s) | Time(s) | Waits(s) | Waits(s) | Waits(s) | Waits(s) | Waits(s) | Gets | Reqs | Bytes |
====================================================================================================================
| 7555 | 0.00 | 4290 | 2812 | 0.08 | 27 | 183 | 243 | 3M | 294K | 7GB |
====================================================================================================================
SQL Plan Monitoring Details (Plan Hash Value=323941584)
==========================================================================================================================================================================================================
| Id | Operation | Name | Rows | Cost | Time | Start | Execs | Rows | Read | Read | Mem | Activity | Activity Detail |
| | | | (Estim) | | Active(s) | Active | | (Actual) | Reqs | Bytes | (Max) | (%) | (# samples) |
==========================================================================================================================================================================================================
| 0 | MERGE STATEMENT | | | | | | 1 | | | | | | |
| 1 | MERGE | T_GTTCHARGEVALUES | | | | | 1 | | | | | | |
| 2 | PX COORDINATOR | | | | 57 | +1 | 481 | 0 | 317 | 5MB | | 4.05 | latch: shared pool (40) |
| | | | | | | | | | | | | | os thread startup (17) |
| | | | | | | | | | | | | | Cpu (7) |
| | | | | | | | | | | | | | DFS lock handle (36) |
| | | | | | | | | | | | | | SGA: allocation forcing component growth (14) |
| | | | | | | | | | | | | | latch: parallel query alloc buffer (200) |
| 3 | PX SEND QC (RANDOM) | :TQ10003 | 1 | 19054 | | | | | | | | | |
| 4 | VIEW | | | | | | | | | | | | |
| 5 | FILTER | | | | | | | | | | | | |
| 6 | SORT GROUP BY | | 1 | 19054 | | | | | | | | | |
| 7 | PX RECEIVE | | 1 | 19054 | | | | | | | | | |
| 8 | PX SEND HASH | :TQ10002 | 1 | 19054 | | | 240 | | | | | | |
| 9 | SORT GROUP BY | | 1 | 19054 | 246 | +70 | 240 | 0 | | | 228M | 49.32 | Cpu (3821) |
| 10 | FILTER | | | | 245 | +71 | 240 | 3G | | | | 0.08 | Cpu (6) |
| 11 | HASH JOIN | | 1 | 19054 | 259 | +57 | 240 | 3G | | | 276M | 4.31 | Cpu (334) |
| 12 | PX RECEIVE | | 1M | 5 | 259 | +57 | 240 | 1M | | | | 0.04 | Cpu (3) |
| 13 | PX SEND HASH | :TQ10000 | 1M | 5 | 6 | +56 | 240 | 1M | | | | 0.01 | Cpu (1) |
| 14 | PX BLOCK ITERATOR | | 1M | 5 | 6 | +56 | 240 | 1M | | | | 0.03 | Cpu (1) |
| | | | | | | | | | | | | | PX Deq: reap credit (1) |
| 15 | TABLE ACCESS FULL | T_GTTCHARGEVALUES | 1M | 5 | 7 | +55 | 5486 | 1M | 5487 | 86MB | | 2.31 | gc cr grant 2-way (3) |
| | | | | | | | | | | | | | gc current block lost (7) |
| | | | | | | | | | | | | | Cpu (7) |
| | | | | | | | | | | | | | db file sequential read (162) |
| 16 | PX RECEIVE | | 78M | 19047 | 255 | +61 | 240 | 801K | | | | 0.03 | IPC send completion sync (2) |
| 17 | PX SEND HASH | :TQ10001 | 78M | 19047 | 250 | +66 | 240 | 3M | | | | 0.06 | Cpu (5) |
| 18 | PX BLOCK ITERATOR | | 78M | 19047 | 250 | +66 | 240 | 4M | | | | | |
| 19 | TABLE ACCESS FULL | T_CACHARGES | 78M | 19047 | 254 | +62 | 1016 | 4M | 288K | 6GB | | 37.69 | gc buffer busy acquire (104) |
| | | | | | | | | | | | | | gc cr block 2-way (1) |
| | | | | | | | | | | | | | gc cr block lost (9) |
| | | | | | | | | | | | | | gc cr grant 2-way (14) |
| | | | | | | | | | | | | | gc cr multi block request (1) |
| | | | | | | | | | | | | | gc current block 2-way (3) |
| | | | | | | | | | | | | | gc current block 3-way (2) |
| | | | | | | | | | | | | | gc current block busy (1) |
| | | | | | | | | | | | | | gc current grant busy (2) |
| | | | | | | | | | | | | | Cpu (58) |
| | | | | | | | | | | | | | latch: gc element (1) |
| | | | | | | | | | | | | | db file parallel read (26) |
| | | | | | | | | | | | | | db file scattered read (207) |
| | | | | | | | | | | | | | db file sequential read (2433) |
| | | | | | | | | | | | | | direct path read (1) |
| | | | | | | | | | | | | | read by other session (57) |
==========================================================================================================================================================================================================
Parallel Execution Details (DOP=240 , Servers Allocated=480)
Instances : 3chris_c wrote:
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads | OMem | 1Mem | Used-Mem |
|* 9 | TABLE ACCESS FULL | T_CACHARGES | 24 | 392K| 21M|00:02:47.48 | 608K| 177K| | | |
Based on the discrepancy between the estimated number of rows and the actual, and the below posted bind value of 2012-04-25 i'd first be checking if the statistics on T_CACHARGES are up to date.
As a reference
http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:4399338600346902127
So that would be my first avenue of exploration.
Cheers,
Maybe you are looking for
-
Open file in folder with command line program
I have a file, which is in native format for a given program [1] I have installed. This program I can start from Terminal.app and within the program I can open the file. The question is if I can have associated with this specific filetype this progra
-
N95 not seen by nokia products
My problem is that no nokia software can see my phone I use winxp sp3 up to date neither my phone, nor my wife's one are seen by nsu or pc suite whereas windows sees them i've tried on different computer with the same conclusion with fw or wo, with a
-
Hi all, In PO, there is field "Outline agreement" originally set to mandatory field. I'd like to set it optional temporarily to delete some items that is no contract in some PO. Therefore I go to the IMG Purchase Order > Define Screen Layout at Docum
-
Websites loaded only with question marks.
Some websites are loaded only with question marks after I updated to Firefox 4.0.1. I have attached a screenshot in the link below. [http://img64.imageshack.us/img64/2462/wmarks.png http://img64.imageshack.us/img64/2462/wmarks.png]
-
When will new Intel Graphics Driver Graphics be released for X61
Intel has released a new driver version 6.14.10.5002 for gm965 - any word when lenovo will release their version? Can Intel's driver be used directly?