11gr2 crsd core dump during failover or start attempt on second node
Hi,
I installed 11gr2 with ASM on one node (solaris SPARC). Then I added another node to this cluster (via addNode.sh script).
Than got strange error: If my first node is up, second node is started fine and run well. If I shutdown first node - crsd on second node dump to core and fails to restart. I get the same error if I try to start second node when the first one is down.
In the crsd.log I see the following:
[ clsdmt][2]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=mskbkp2DBG_CRSD))
2010-03-03 17:31:35.330: [ clsdmt][2]PID for the Process [18669], connkey 1
2010-03-03 17:31:35.331: [ clsdmt][2]Creating PID [18669] file for home /u01/grid/11.2.0 host mskbkp2 bin crs to /u01/grid/11
.2.0/crs/init/
2010-03-03 17:31:35.331: [ clsdmt][2]Writing PID [18669] to the file [u01/grid/11.2.0/crs/init/mskbkp2.pid]
2010-03-03 17:31:35.925: [ default][1] CRS Daemon Starting
2010-03-03 17:31:35.933: [ default][1] ENV Logging level for Module: AGENT 1
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: AGFW 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLSFRAME 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLSVER 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: CLUCLS 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: COMMCRS 0
2010-03-03 17:31:35.934: [ default][1] ENV Logging level for Module: COMMNS 0
2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSAPP 0
2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCCL 0
2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCEVT 0
2010-03-03 17:31:35.936: [ default][1] ENV Logging level for Module: CRSCOMM 1
2010-03-03 17:31:35.936: [ CRSD][1] ENV Debug Level(CRSD): 50
2010-03-03 17:31:35.936: [ CRSD][1] ENV Logging level for Module: CRSD 50
2010-03-03 17:31:35.937: [ CRSD][1] ENV Debug Level(CRSEVT): 0
2010-03-03 17:31:35.937: [ CRSD][1] ENV Logging level for Module: CRSEVT 0
2010-03-03 17:31:35.937: [ CRSD][1] ENV Debug Level(CRSMAIN): 1
2010-03-03 17:31:35.937: [ CRSD][1] ENV Logging level for Module: CRSMAIN 1
2010-03-03 17:31:35.937: [ CRSD][1] ENV Debug Level(CRSOCR): 0
2010-03-03 17:31:35.937: [ CRSD][1] ENV Logging level for Module: CRSOCR 0
2010-03-03 17:31:35.939: [ CRSD][1] ENV Debug Level(CRSPE): 0
2010-03-03 17:31:35.939: [ CRSD][1] ENV Logging level for Module: CRSPE 0
2010-03-03 17:31:35.939: [ CRSD][1] ENV Debug Level(CRSPLACE): 0
2010-03-03 17:31:35.939: [ CRSD][1] ENV Logging level for Module: CRSPLACE 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Debug Level(CRSRES): 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Logging level for Module: CRSRES 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Debug Level(CRSRPT): 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Logging level for Module: CRSRPT 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Debug Level(CRSRTI): 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Logging level for Module: CRSRTI 0
2010-03-03 17:31:35.940: [ CRSD][1] ENV Debug Level(CRSSE): 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Logging level for Module: CRSSE 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Debug Level(CRSSEC): 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Logging level for Module: CRSSEC 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Debug Level(CRSSHARED): 0
2010-03-03 17:31:35.941: [ CRSD][1] ENV Logging level for Module: CRSSHARED 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(CRSTIMER): 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Logging level for Module: CRSTIMER 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(CRSUI): 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Logging level for Module: CRSUI 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(CSSCLNT): 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Logging level for Module: CSSCLNT 0
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(OCRAPI): 1
2010-03-03 17:31:35.942: [ CRSD][1] ENV Logging level for Module: OCRAPI 1
2010-03-03 17:31:35.942: [ CRSD][1] ENV Debug Level(OCRASM): 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Logging level for Module: OCRASM 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Debug Level(OCRCAC): 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Logging level for Module: OCRCAC 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Debug Level(OCRCLI): 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Logging level for Module: OCRCLI 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Debug Level(OCRMAS): 1
2010-03-03 17:31:35.943: [ CRSD][1] ENV Logging level for Module: OCRMAS 1
2010-03-03 17:31:35.944: [ CRSD][1] ENV Debug Level(OCRMSG): 1
2010-03-03 17:31:35.944: [ CRSD][1] ENV Logging level for Module: OCRMSG 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(OCROSD): 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Logging level for Module: OCROSD 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(OCRRAW): 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Logging level for Module: OCRRAW 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(OCRSRV): 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Logging level for Module: OCRSRV 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(OCRUTL): 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Logging level for Module: OCRUTL 1
2010-03-03 17:31:35.945: [ CRSD][1] ENV Debug Level(SuiteTes): 1
2010-03-03 17:31:35.946: [ CRSD][1] ENV Logging level for Module: SuiteTes 1
2010-03-03 17:31:35.946: [ CRSD][1] ENV Debug Level(UiServer): 0
2010-03-03 17:31:35.946: [ CRSD][1] ENV Logging level for Module: UiServer 0
2010-03-03 17:31:35.946: [ CRSMAIN][1] Checking the OCR device
2010-03-03 17:31:35.948: [ CRSMAIN][1] Connecting to the CSS Daemon
2010-03-03 17:31:35.976: [ CRSMAIN][1] Initializing OCR
2010-03-03 17:31:35.981: [ OCRAPI][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
2010-03-03 17:31:35.981: [ OCRAPI][1]Check namebufs
2010-03-03 17:31:35.981: [ OCRAPI][1]Finished checking namebufs
2010-03-03 17:31:35.982: [ GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsinet.c : 3232
], original from [clsss.c : 5026]
2010-03-03 17:31:36.036: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
2010-03-03 17:31:36.037: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
2010-03-03 17:31:36.059: [ GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 680
], original from [clsss.c : 5026]
2010-03-03 17:31:36.067: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
fatal if both fail
2010-03-03 17:31:36.068: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
f 2: file wallet (LSKP-FSW)
2010-03-03 17:31:36.068: [ GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:36.069: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
2: file wallet (LSKP-FSW) OK
2010-03-03 17:31:36.069: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
f 2: OLR wallet (LSKP-CLSW-OLR)
[ CLWAL][1]clsw_Initialize: OLR initlevel [30000]
2010-03-03 17:31:36.080: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
2: OLR wallet (LSKP-CLSW-OLR) OK
2010-03-03 17:31:36.081: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
providers - fatal if all fail)
2010-03-03 17:31:36.081: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:36.152: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:36.152: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:36.152: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:36.172: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
providers - fatal if all fail)
2010-03-03 17:31:36.172: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:36.238: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:36.239: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:36.239: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:36.239: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=0
2010-03-03 17:31:36.541: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
(0)
2010-03-03 17:31:36.552: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
s (0)
2010-03-03 17:31:36.771: [ OCRRAW][1]proprioo: for disk 0 (+DR2_BIN), id match (1), total id sets, (1) need recover (0), my v
otes (0), total votes (0), commit_lsn (9), lsn (9)
2010-03-03 17:31:36.771: [ OCRRAW][1]proprioo: my id set: (833490748, 1028247821, 0, 0, 0)
2010-03-03 17:31:36.772: [ OCRRAW][1]proprioo: 1st set: (833490748, 1028247821, 0, 0, 0)
2010-03-03 17:31:36.772: [ OCRRAW][1]proprioo: 2nd set: (0, 0, 0, 0, 0)
2010-03-03 17:31:36.830: [ OCRSRV][1]th_init: Successfully retrieved CSS misscount [31].
2010-03-03 17:31:36.830: [ OCRSRV][1]th_init: Successfully query CLSS mode [3].
[ OCRMAS][20]th_calc_av:5': Rturn persisted AV [186646784] [11.2.0.1.0]
2010-03-03 17:31:36.920: [ OCRSRV][20]th_not_master_change: Master change callback not registered
2010-03-03 17:31:36.920: [ OCRMAS][20]th_master:12: I AM THE NEW OCR MASTER at incar 1. Node Number 2
2010-03-03 17:31:37.134: [ OCRASM][20]proprasmo: ASM cache size is [5MB]
2010-03-03 17:31:37.142: [ OCRASM][20]proprasmo: ASM cache [5MB] enabled for disk group [DR2_BIN].
2010-03-03 17:31:37.155: [ OCRRAW][20]proprioo: for disk 0 (+DR2_BIN), id match (1), total id sets, (1) need recover (0), my
votes (0), total votes (0), commit_lsn (9), lsn (9)
2010-03-03 17:31:37.155: [ OCRRAW][20]proprioo: my id set: (833490748, 1028247821, 0, 0, 0)
2010-03-03 17:31:37.155: [ OCRRAW][20]proprioo: 1st set: (833490748, 1028247821, 0, 0, 0)
2010-03-03 17:31:37.155: [ OCRRAW][20]proprioo: 2nd set: (0, 0, 0, 0, 0)
2010-03-03 17:31:37.214: [ OCRMAS][20]proath_master:18: Spawned connection mgr thread
2010-03-03 17:31:37.214: [ OCRMAS][20]proath_master:20: Spawned upgrade thread
2010-03-03 17:31:37.214: [ OCRMAS][20]th_master:19.1: Wake up upgrade thread
2010-03-03 17:31:37.216: [ OCRSRV][1]th_snap_local_spawn: Inside snap local spawn. host is [mskbkp2]
2010-03-03 17:31:37.219: [ CRSMAIN][1] Running as user: root
2010-03-03 17:31:37.219: [ CRSMAIN][1] CRSD running as the Privileged user
2010-03-03 17:31:37.219: [ CLSVER][1] Static Version 11.2.0.1.0
2010-03-03 17:31:37.226: [ OCRMAS][20]th_master:1': Recvd pubdata event from node [2]
2010-03-03 17:31:37.227: [ OCRMAS][20]th_master:2': Recvd pubdata event for self. Do nothing.
2010-03-03 17:31:37.227: [ CLSVER][1] Daemon version: 11.2.0.1.0 Software version: 11.2.0.1.0
2010-03-03 17:31:37.231: [ CLSVER][1] Active Version from OCR:11.2.0.1.0
2010-03-03 17:31:37.232: [ CLSVER][1] Active Version and Software Version are same
2010-03-03 17:31:37.232: [ CLSVER][1] Active Version changed to 11.2.0.1.0
2010-03-03 17:31:37.232: [ OCRSRV][1]th_reg_master_change: Master change callback registered
2010-03-03 17:31:37.232: [ OCRAPI][1]a_reg_master_change: Registered master change callback
2010-03-03 17:31:37.232: [ OCRSRV][1]th_not_master_change: Invoking master change callback. Master [2] Inc [1]
2010-03-03 17:31:37.232: [ OCRAPI][1]a_reg_master_change: Notified master change
2010-03-03 17:31:37.232: [ CRSMAIN][1] CAA Node Group Pri Data size: 128
2010-03-03 17:31:37.233: [ CRSMAIN][1] CAA Node Group Pub Data size: 128
2010-03-03 17:31:37.247: [ CRSMAIN][1] Getting private data of booted nodes
2010-03-03 17:31:37.247: [ CRSMAIN][1] Checking for booted param on nodenum: 2
2010-03-03 17:31:37.306: [ CLSE][1]clse_get_auth_loc: Returning default authloc: /u01/grid/11.2.0/auth/crs/mskbkp2
2010-03-03 17:31:37.306: [ CRSMAIN][1] Using Authorizer location: /u01/grid/11.2.0/auth/crs/mskbkp2
2010-03-03 17:31:37.314: [ OCRSRV][23]th_upgrade: Starting upgrade calculation
2010-03-03 17:31:37.364: [ CLSCLU][1]clsclu_init: rc 0
2010-03-03 17:31:37.381: [ OCRSRV][23]th_upgrade:10.1 AV [186646784]. State [11]. Already upgraded.Updated global data to the
crs version group. Return [0]
2010-03-03 17:31:37.385: [ CRSMAIN][1] Initializing RTI
2010-03-03 17:31:37.433: [ CRSMAIN][1] Initializing ResouceStateListener
2010-03-03 17:31:37.433: [CRSTIMER][37] Timer Thread Starting.
2010-03-03 17:31:37.433: [ CRSMAIN][1] Initializing EVMMgr
2010-03-03 17:31:37.446: [ CRSMAIN][1] Initializing ResourceMap Map
2010-03-03 17:31:37.461: [ CRSMAIN][1] Subscribing to EVM events for apps
2010-03-03 17:31:37.504: [ CRSMAIN][1] CRSD locked during state recovery, please wait.
2010-03-03 17:31:37.516: [ CRSMAIN][1] CRSD recovered, unlocked.
2010-03-03 17:31:37.525: [ default][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
2010-03-03 17:31:37.525: [ default][1]Check namebufs
2010-03-03 17:31:37.525: [ default][1]Finished checking namebufs
2010-03-03 17:31:37.526: [ GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsinet.c : 3232
], original from [clsss.c : 5026]
2010-03-03 17:31:37.569: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
2010-03-03 17:31:37.569: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
2010-03-03 17:31:37.587: [ GIPC][1] gipcCheckInitialization: possible incompatible non-threaded init from [clsgpnp0.c : 680
], original from [clsss.c : 5026]
2010-03-03 17:31:37.595: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
fatal if both fail
2010-03-03 17:31:37.595: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
f 2: file wallet (LSKP-FSW)
2010-03-03 17:31:37.596: [ GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:37.596: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
2: file wallet (LSKP-FSW) OK
2010-03-03 17:31:37.596: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
f 2: OLR wallet (LSKP-CLSW-OLR)
[ CLWAL][1]clsw_Initialize: OLR initlevel [30000]
2010-03-03 17:31:37.607: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
2: OLR wallet (LSKP-CLSW-OLR) OK
2010-03-03 17:31:37.607: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
providers - fatal if all fail)
2010-03-03 17:31:37.607: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:37.673: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:37.673: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:37.673: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:37.690: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
providers - fatal if all fail)
2010-03-03 17:31:37.690: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:37.754: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:37.754: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:37.754: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:37.755: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=0
2010-03-03 17:31:37.806: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
(0)
2010-03-03 17:31:37.817: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
s (0)
2010-03-03 17:31:37.822: [ CRSMAIN][1] CRSD listening on 10 style E2E port (ADDRESS=(PROTOCOL=tcp)(HOST=172.31.25.112)(PORT=38
983))
2010-03-03 17:31:37.835: [ CRSMAIN][1] Starting Threads
2010-03-03 17:31:37.858: [ CLSE][1]clse_get_auth_loc: Returning default authloc: /u01/grid/11.2.0/auth/crs/mskbkp2
2010-03-03 17:31:37.858: [ CRSD][1] AuthLoc /u01/grid/11.2.0/auth/crs/mskbkp2
2010-03-03 17:31:37.859: [ CRSD][1] PE active version: 11.2.0.1.0
2010-03-03 17:31:37.859: [ CRSD][1] PE Engine: NEW
2010-03-03 17:31:37.859: [ CRSD][1] Using OCR batch ops : ENABLED
2010-03-03 17:31:37.860: [ CRSMAIN][1] Initializing Node Down Monitor
2010-03-03 17:31:37.860: [ CRSMAIN][1] CRS Daemon Started.
2010-03-03 17:31:37.860: [ CRSD][1] Connecting to the CSS Daemon
2010-03-03 17:31:37.861: [ CRSD][1] Local CSS Node Number is: 2
2010-03-03 17:31:37.863: [ CRSD][1] Local Css Node Name is: mskbkp2
2010-03-03 17:31:37.863: [ CRSD][1] CRSDPersonality initialized
2010-03-03 17:31:37.864: [ CRSMAIN][1] Process member data: CRSD:mskbkp2
2010-03-03 17:31:37.864: [ CRSD][1][F-ALGO] getIpcPath returning (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
2010-03-03 17:31:37.865: [CLSFRAME][1] Inited lsf context 102b3f670
2010-03-03 17:31:37.865: [CLSFRAME][1] Initing CLS Framework messaging
2010-03-03 17:31:37.869: [ CRSD][1][F-ALGO] getIpcPath returning (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
2010-03-03 17:31:37.873: [UiServer][1] UI Comms initalize() 1
2010-03-03 17:31:37.873: [CLSFRAME][1] New Framework state: 2
2010-03-03 17:31:37.873: [CLSFRAME][1] M2M is starting...
2010-03-03 17:31:37.873: [ CRSCCL][1]clsCclInit called by process: 18669
2010-03-03 17:31:37.885: [ CRSCCL][1]USING CLSC ============
2010-03-03 17:31:37.895: [ default][1]clsu_get_private_ip_addr: Calling clsu_get_private_ip_addresses to get first private ip
2010-03-03 17:31:37.895: [ default][1]Check namebufs
2010-03-03 17:31:37.895: [ default][1]Finished checking namebufs
2010-03-03 17:31:37.950: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:405] gpnp tracelevel 3, component tracelevel 0
2010-03-03 17:31:37.951: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:535] '/u01/grid/11.2.0' in effect as GPnP home base.
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3867] Init gpnp local security key providers (2)
fatal if both fail
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3870] Init gpnp local security key proveders 1 o
f 2: file wallet (LSKP-FSW)
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:398] Using FS Wallet Location : /u01/grid/11.2.0/
gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3892] Init gpnp local security key provider 1 of
2: file wallet (LSKP-FSW) OK
2010-03-03 17:31:37.970: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3898] Init gpnp local security key proveders 2 o
f 2: OLR wallet (LSKP-CLSW-OLR)
[ CLWAL][1]clsw_Initialize: OLR initlevel [70000]
2010-03-03 17:31:37.980: [ GPnP][1]clsgpnp_InitCKProviders: [at clsgpnp0.c:3921] Init gpnp local security key provider 2 of
2: OLR wallet (LSKP-CLSW-OLR) OK
2010-03-03 17:31:37.980: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;7. (2
providers - fatal if all fail)
2010-03-03 17:31:37.980: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:38.049: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:38.049: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:38.049: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:38.068: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1952] <Get gpnp security keys (wallet) for id:1,typ;4. (2
providers - fatal if all fail)
2010-03-03 17:31:38.068: [ GPnP][1]clsgpnpkwf_getWalletPath: [at clsgpnpkwf.c:501] req_id=1 ck_prov_id=1 wallet path: /u01/
grid/11.2.0/gpnp/mskbkp2/wallets/peer/
2010-03-03 17:31:38.134: [ GPnP][1]clsgpnpwu_walletfopen: [at clsgpnpwu.c:496] Opened SSO wallet: '/u01/grid/11.2.0/gpnp/ms
kbkp2/wallets/peer/cwallet.sso'
2010-03-03 17:31:38.135: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1968] Result: (0) CLSGPNP_OK. Get gpnp wallet - provider 1
of 2 (LSKP-FSW(1))
2010-03-03 17:31:38.135: [ GPnP][1]clsgpnp_getCK: [at clsgpnp0.c:1982] Got gpnp security keys (wallet).>
2010-03-03 17:31:38.135: [ GPnP][1]clsgpnp_Init: [at clsgpnp0.c:840] GPnP client pid=18669, tl=3, f=3
2010-03-03 17:31:38.184: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1735], ret gipcretSuccess
(0)
2010-03-03 17:31:38.194: [GIPCXCPT][1] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSucces
s (0)
2010-03-03 17:31:38.200: [ CRSCCL][1]Listening endpoint created sucessfully @ (ADDRESS=(PROTOCOL=tcp)(DEV=54)(HOST=172.31.25.
112)(PORT=38984)).con = 10359a0d0
2010-03-03 17:31:38.209: [ CRSCCL][48]CSS Group Registration complete.
2010-03-03 17:31:38.213: [ CRSCCL][48]cclGetMemberData called
2010-03-03 17:31:38.215: [ CRSCCL][48]Obtained first membership map.
2010-03-03 17:31:38.215: [ CRSCCL][48]Dumping member data ------------------
2010-03-03 17:31:38.215: [ CRSCCL][48]Member (2, 603412550) on node port=.
2010-03-03 17:31:38.216: [ CRSCCL][48]Done ------------------
2010-03-03 17:31:38.216: [ CRSCCL][48]Waiting for reconfigs
2010-03-03 17:31:38.216: [ CRSCCL][49]cclCommunicationHandler started.
2010-03-03 17:31:38.220: [ CRSCOMM][1] Ipc: m_pClscCtx=1020c4850m_pUgblm=1035b2a50
2010-03-03 17:31:38.220: [ CRSCOMM][1] Ipc: Starting send thread
2010-03-03 17:31:38.220: [ CRSCOMM][1] IpcL: Listener instantiated for: (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11))
2010-03-03 17:31:38.221: [ CRSCOMM][52] Ipc: sendWork thread started.
2010-03-03 17:31:38.222: [ CRSCOMM][1] IpcL: Listener started listening.
2010-03-03 17:31:38.223: [ CRSCOMM][53] IpcL: thread started listening
2010-03-03 17:31:38.223: [CLSFRAME][1] Starting thread model named: AgfwProxySrvTM
2010-03-03 17:31:38.224: [CLSFRAME][1] Starting thread model named: OcrModuleTM
2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: PolicyEngineTM
2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: SharedThreadTM
2010-03-03 17:31:38.225: [CLSFRAME][1] Starting thread model named: UiServerTM
2010-03-03 17:31:38.225: [CLSFRAME][1] New Framework state: 3
2010-03-03 17:31:38.227: [ CRSRPT][62] Enabled
2010-03-03 17:31:38.228: [ CRSPE][61] PE Role|State Update: old role [INVALID] new [INVALID]; old state [Not yet initialized
] new [Enabling: waiting for role]
2010-03-03 17:31:38.229: [ CRSSE][62] Master Change Event; New Master Node ID:2 This Node's ID:2
2010-03-03 17:31:38.230: [ CRSPE][61] PE Role|State Update: old role [INVALID] new [MASTER]; old state [Enabling: waiting fo
r role] new [Configuring]
2010-03-03 17:31:38.230: [ CRSPE][61] PE MASTER NAME: mskbkp2
2010-03-03 17:31:38.230: [ CRSPE][61] Starting to read configuration
2010-03-03 17:31:38.260: [ CRSPE][61] Reading (2) servers
2010-03-03 17:31:38.459: [ CRSPE][61] DM: set global config version to: 150
2010-03-03 17:31:38.459: [ CRSPE][61] DM: set pool freeze timeout to: 60000
2010-03-03 17:31:38.459: [ CRSPE][61] DM: Set event seq number to: 13900000
2010-03-03 17:31:38.459: [ CRSPE][61] DM: Set threshold event seq number to: 13980000
2010-03-03 17:31:38.460: [ CRSPE][61] Sent request to write event sequence number 14000000 to repository
2010-03-03 17:31:38.483: [ CRSPE][61] Wrote new event sequence to repository
2010-03-03 17:31:38.568: [ CRSPE][61] Reading (15) types
2010-03-03 17:31:38.593: [ CRSPE][61] Reading (3) server pools
2010-03-03 17:31:38.624: [ CRSPE][61] Reading (21) resources
2010-03-03 17:31:39.987: [ CRSPE][61] Finished reading configuration. Parsing...
2010-03-03 17:31:39.988: [ CRSPE][61] Parsing resource types...
2010-03-03 17:31:40.030: [ CRSD][61] Initializing the config version for type ora.asm.type to: 1
2010-03-03 17:31:40.035: [ CRSD][61] Initializing the config version for type ora.cluster_resource.type to: 1
2010-03-03 17:31:40.040: [ CRSD][61] Initializing the config version for type ora.cluster_vip.type to: 1
2010-03-03 17:31:40.044: [ CRSD][61] Initializing the config version for type ora.cluster_vip_net1.type to: 1
2010-03-03 17:31:40.048: [ CRSD][61] Dump State Starting ...
2010-03-03 17:31:40.048: [ CRSD][61] State Dump for RTILock
2010-03-03 17:31:40.048: [ CRSD][61] Lock State List is busy, skipping ..
2010-03-03 17:31:40.048: [ CRSD][61] State Dump for Timer
2010-03-03 17:31:40.049: [ CRSD][61] Timer map size=0
2010-03-03 17:31:40.049: [ CRSPE][61] Dumping PE Data Model...:DM has [0 resources][0 types][0 servers][0 spools]
------------- RESOURCES:
------------- TYPES:
------------- SERVERS:
------------- SERVER POOLS:
2010-03-03 17:31:40.049: [ CRSPE][61] Dumping ICE contents...:ICE operation count: 0
2010-03-03 17:31:40.049: [ CRSD][61] Dump State Done.
I guess that there is some thing wrong in configuration, but cannot find out what.
Any help would be appreciated.
Thanks
Hi,
Please check your disk attributes and permission of OCR/Voting and other ASM devices. The disk attribute should be changed to be shared among all nodes of cluster. It happened with us in 10.2.0.4 where disk was not shared and we were able to start crs from only one node at a time so please check disk attributes. Please see blog keyurmakwanacrs.blogspot.com for AIX which we faced. Not surle whether you've similar problem or not. We had 10.2.0.4 clusterware.
thanks,
Keyur
Similar Messages
-
RAC, ASM failed to start up on second node , ORA-03113: end-of-file on comm
i'm installing an RAC with 2 nodes on top of ASM
when creating ASM Diskgroup , it failed and reported error CRS-0215 failed to start asm on node2
Oracle 10.2.0.1
linux CentOs 4.x
u01/app/oracle/product/10.2.0/db_1/bin/dbca -progress_only -configureASM -templateName NO_VALUE -gdbName NO -sid NO -emConf
iguration NONE -diskList /dev/raw/raw2,/dev/raw/raw3 -diskGroupName DATA -datafileJarLocation /u01/app/oracle/product/10.2.0/db_
1/assistants/dbca/templates -responseFile NO_VALUE -nodeinfo node1,node2 -obfuscatedPasswords true -oratabLocation /u01/app/o
racle/product/10.2.0/db_1/install/oratab -asmSysPassword 05dbb0be38ecf8cca822cf3cf99e675448 -redundancy EXTERNA
[oracle@node2 bin]$ ./crs_stat -t -v
Name Type R/RA F/FT Target State Host
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE node1
ora....E1.lsnr application 0/5 0/0 ONLINE ONLINE node1
ora.node1.gsd application 0/5 0/0 ONLINE ONLINE node1
ora.node1.ons application 0/3 0/0 ONLINE ONLINE node1
ora.node1.vip application 0/0 0/0 ONLINE ONLINE node1
ora....SM2.asm application 0/5 0/0 OFFLINE OFFLINE
ora....E2.lsnr application 0/5 0/0 ONLINE ONLINE node2
ora.node2.gsd application 0/5 0/0 ONLINE ONLINE node2
ora.node2.ons application 0/3 0/0 ONLINE ONLINE node2
ora.node2.vip application 0/0 0/0 ONLINE ONLINE node2
i checked the status , asm is able to start on both nodes if not at the same time ,
when trying to start the second node , with srvctl or sqlplus , each give the error 03113
can anyone suggest me of how to bring up both instances ,
thanks~
[oracle@node2 bin]$ srvctl stop asm -n node1
[oracle@node2 bin]$ srvctl start asm -n node1
[oracle@node2 bin]$ srvctl start asm -n node2
PRKS-1009 : Failed to start ASM instance "+ASM2" on node "node2", [PRKS-1009 : Failed to start ASM instance "+ASM2" on node "node2", [node2:ora.node2.ASM2.asm:
node2:ora.node2.ASM2.asm:SQL*Plus: Release 10.2.0.1.0 - Production on Wed May 27 16:14:50 2009
node2:ora.node2.ASM2.asm:
node2:ora.node2.ASM2.asm:Copyright (c) 1982, 2005, Oracle. All rights reserved.
node2:ora.node2.ASM2.asm:
node2:ora.node2.ASM2.asm:Enter user-name: Connected to an idle instance.
node2:ora.node2.ASM2.asm:
node2:ora.node2.ASM2.asm:SQL> ORA-03113: end-of-file on communication channel
node2:ora.node2.ASM2.asm:SQL> Disconnected
node2:ora.node2.ASM2.asm:
[code/]
Edited by: zs_hzh on May 27, 2009 1:25 AMIs it possible to start ASM on second node with SQL*Plus in NOMOUNT state?
-
Hotspot core dumping during JVM garbage collection ?
We have an application which calls a 3rd party supplied server API which has recently been upgraded to use Java 1.5
We are getting the following error reported by our client application. The application is also now running Java 1.5 but references many classes in jar files which would have quite old code in.
The supplier of the API has stated that the problem requires us to recompile all our jar files using v 1.5 ( including things like jconnect and jms ?!?!? ). This sounds like a bit of a cop-out to me, not to mention being impossible since we don't have the source for things like jconnect.
I suspect that there is a garbage collection problem at the bottom of all this, but I'm not sure how I can "prove" this, nor do I currently have any real clue as to how to fix any GC problem that may exist.
The application is supposed to wait for a message on a MQSeries queue and then transforms it via Xalan XSLT and sends it to a Server application. I've tried playing around with heap sizes etc but that just seems to make it worse. If I leave it at the settings that the previous version used then the client at least manages to process a couple of messages before core dumping. There doesn't seem to be a consistent trigger event to cause the core dump ( it's not a message arriving on a queue for example ) but it does seem to be fairly consistent timewise, i.e. after
Any ideas gratefully accepted.
Here's a logfile excerpt from my applications showing the Hotspot error message :
=====================================================================================
08-Jul-2008 10:01:05 Waiting for messages from COLT.BBFS
08-Jul-2008 10:02:05 Waiting for messages from COLT.BBFS
08-Jul-2008 10:03:05 Waiting for messages from COLT.BBFS
405.815: [GC [PSYoungGen: 17331K->9244K(37632K)] 111702K->103615K(192128K), 0.1615910 secs]
405.977: [Full GC#
# An unexpected error has been detected by HotSpot Virtual Machine:
# SIGBUS (0xa) at pc=0xfe141348, pid=2600, tid=8
# Java VM: Java HotSpot(TM) Server VM (1.5.0_03-b07 mixed mode)
# Problematic frame:
# V [libjvm.so+0x141348]
# An error report file with more information is saved as hs_err_pid2600.log
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
=====================================================================================
The logfile referred to in the error message contains the following.
=====================================================================================
# An unexpected error has been detected by HotSpot Virtual Machine:
# SIGBUS (0xa) at pc=0xfe141348, pid=2600, tid=8
# Java VM: Java HotSpot(TM) Server VM (1.5.0_03-b07 mixed mode)
# Problematic frame:
# V [libjvm.so+0x141348]
--------------- T H R E A D ---------------
Current thread (0x001484d8): VMThread [id=8]
siginfo:si_signo=10, si_errno=0, si_code=1, si_addr=0x0000080f
Registers:
O0=0x00487588 O1=0xfe7d6454 O2=0x000079b0 O3=0x00007800
O4=0x00008868 O5=0x00147d48 O6=0xf8781460 O7=0xfe0f7938
G1=0xe52aaae8 G2=0x00000003 G3=0x00000003 G4=0x001484d8
G5=0xf8781d98 G6=0x00000002 G7=0xf8781d98 Y=0x805683e2
PC=0xfe141348 nPC=0xfe14134c
Top of Stack: (sp=0xf8781460)
0xf8781460: fe786000 00c6ba20 fe10013c e54bab68
0xf8781470: 0000080f e54bab6c 0000080f 00000004
0xf8781480: 00487588 00000134 e54bab6c 00000004
0xf8781490: 00000001 00000000 f87814c0 fe17aef4
0xf87814a0: fe6485b4 fe7d899c 001484d8 0011da40
0xf87814b0: 00148988 00148c10 00148d7c f8781880
0xf87814c0: 007c3389 007c3c0e 00000f87 00008868
0xf87814d0: 00008800 00487588 fe141310 fe7d6454
Instructions: (pc=0xfe141348)
0xfe141338: ec 06 c0 1a 80 a5 a0 00 22 40 00 0a ba 07 60 01
0xfe141348: f2 05 a0 00 ae 0e 60 03 80 a5 e0 03 22 40 00 05
Stack: [0xf8702000,0xf8781d98), sp=0xf8781460, free space=509k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x141348]
V [libjvm.so+0x17aefc]
V [libjvm.so+0x2d557c]
V [libjvm.so+0x300ef8]
V [libjvm.so+0x301e84]
V [libjvm.so+0x2ff950]
V [libjvm.so+0x29df30]
V [libjvm.so+0x362b44]
V [libjvm.so+0x6436f0]
VM_Operation (0xe03012b0): parallel gc system gc, mode: safepoint, requested by thread 0x0031bca0
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x00b2c028 JavaThread "Thread-4" [_thread_in_native, id=85]
0x007f5048 JavaThread "Thread-0" [_thread_blocked, id=84]
0x00c27cf0 JavaThread "Notification Delivery" [_thread_blocked, id=81]
0x0026fa08 JavaThread "RMI LeaseChecker" daemon [_thread_blocked, id=73]
0x00821048 JavaThread "RMI RenewClean-[162.11.2.32:44425]" daemon [_thread_blocked, id=70]
0x0031bca0 JavaThread "GC Daemon" daemon [_thread_blocked, id=67]
0x00cd5d28 JavaThread "RMI Reaper" [_thread_blocked, id=66]
0x003c9300 JavaThread "Timer-0" daemon [_thread_blocked, id=65]
0x00929fe0 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=64]
0x0089bf18 JavaThread "SeedGenerator Thread" daemon [_thread_blocked, id=42]
0x00c47248 JavaThread "Pool thread #7" daemon [_thread_blocked, id=38]
0x00c466a0 JavaThread "Pool thread #6" daemon [_thread_blocked, id=37]
0x00311850 JavaThread "Pool thread #5" daemon [_thread_blocked, id=36]
0x00287a40 JavaThread "Pool thread #4" daemon [_thread_blocked, id=35]
0x00286e98 JavaThread "Pool thread #3" daemon [_thread_blocked, id=34]
0x00c134b0 JavaThread "Pool thread #2" daemon [_thread_blocked, id=33]
0x00ad09e0 JavaThread "Pool thread #1" daemon [_thread_blocked, id=32]
0x00286cd8 JavaThread "PoolThreadManager" daemon [_thread_blocked, id=31]
0x00c129e0 JavaThread "Channel Reaper" daemon [_thread_blocked, id=30]
0x00c669e8 JavaThread "ORB Daemon Thread" daemon [_thread_blocked, id=29]
0x00b10170 JavaThread "Worker for ServerProtocol: (iiop) /0.0.0.0:20168" daemon [_thread_blocked, id=22]
0x008a17e0 JavaThread "Syn~ Client" daemon [_thread_blocked, id=21]
0x003dc378 JavaThread "PoolScavenger0" daemon [_thread_blocked, id=20]
0x0015a928 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=15]
0x00159880 JavaThread "CompilerThread1" daemon [_thread_blocked, id=14]
0x00158a18 JavaThread "CompilerThread0" daemon [_thread_blocked, id=13]
0x00157b98 JavaThread "AdapterThread" daemon [_thread_blocked, id=12]
0x00156dc8 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=11]
0x0014ccd8 JavaThread "Finalizer" daemon [_thread_blocked, id=10]
0x0014ad90 JavaThread "Reference Handler" daemon [_thread_blocked, id=9]
0x00038238 JavaThread "main" [_thread_in_native, id=1]
Other Threads:
=>0x001484d8 VMThread [id=8]
0x0015c3b0 WatcherThread [id=16]
VM state:at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00037728/0x00037758] Threads_lock - owner thread: 0x001484d8
[0x00033650/0x00037ba8] Heap_lock - owner thread: 0x0031bca0
Heap
PSYoungGen total 37632K, used 9244K [0xf2eb0000, 0xf7ba0000, 0xf8400000)
eden space 28352K, 0% used [0xf2eb0000,0xf2eb0000,0xf4a60000)
from space 9280K, 99% used [0xf4a60000,0xf5367080,0xf5370000)
to space 25216K, 0% used [0xf6300000,0xf6300000,0xf7ba0000)
PSOldGen total 154496K, used 94371K [0xe8400000, 0xf1ae0000, 0xf2eb0000)
object space 154496K, 61% used [0xe8400000,0xee028e78,0xf1ae0000)
PSPermGen total 35584K, used 18260K [0xe4400000, 0xe66c0000, 0xe8400000)
object space 35584K, 51% used [0xe4400000,0xe55d5158,0xe66c0000)
Dynamic libraries:
0x00010000 /dsdvlp/java/jvm/jdk1.5.0_03/bin/java
0xff350000 /usr/lib/libthread.so.1
0xff340000 /usr/lib/libdl.so.1
0xff200000 /usr/lib/libc.so.1
0xff390000 /usr/platform/SUNW,Sun-Fire-880/lib/libc_psr.so.1
0xfe000000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/server/libjvm.so
0xff1e0000 /usr/lib/libsocket.so.1
0xff2d0000 /usr/lib/libsched.so.1
0xff1b0000 /usr/lib/libCrun.so.1
0xff160000 /usr/lib/libm.so.1
0xff080000 /usr/lib/libnsl.so.1
0xff060000 /usr/lib/libmp.so.2
0xff030000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/native_threads/libhpi.so
0xfdfc0000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/libverify.so
0xfdf80000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/libjava.so
0xfdf50000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/libzip.so
0xfb7e0000 /usr/lib/locale/en_GB.ISO8859-1/en_GB.ISO8859-1.so.2
0xe4190000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/libnet.so
0xe3bd0000 /dsdvlp/lib/5/libSolarisNatives.so
0xe3e90000 /dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/librmi.so
VM Arguments:
jvm_args: -Djava.ext.dirs=/dsdvlp/java/tmijar/firs7/lib/cli:/dsdvlp/java/tmijar/firs7/lib/cli/ext:/dsdvlp/java/tmijar/firs7/lib/cmn/OpenORB:/dsdvlp/java/tmijar/firs7/lib/cmn/OpenORB/ext:/dsdvlp/java/tmijar/firs7/lib/cmn:/dsdvlp/java/tmijar/firs7/lib/cmn/ext:/dsdvlp/java/tmijar/firs7/daemonlib -Duser.dir=/dsdvlp/java/tmijar/firs7 -Dopenorb.config=file:/dsdvlp/java/tmijar/firs7/configs/OpenORB/config/SynOpenORB.xml -Dopenorb.home=file:/dsdvlp/java/tmijar/firs7/configs/OpenORB -Dcom.coexis.syn.general.orbbinding=com.coexis.syn.general.orbbinding.openorb.OpenORBBinding_1_4 -Dsun.rmi.dgc.client.gcInterval=360000 -Dsun.rmi.dgc.server.gcInterval=360000000 -Xms32m -Xmx256m -Dcom.coexis.syn.clientcommandsconfiglocation=file://localhost//dsdvlp/java/tmijar/firs7/configs/clientcommands.xml -Dcom.coexis.syn.clientconfiglocation=file://localhost//dsdvlp/java/tmijar/firs7/configs/fsbbtd_client.xml -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
java_command: com.coexis.syn.mqmessaging.daemon.RunDaemon -p /dsdvlp/bin/5/lndsfsd_fsbbtd.properties start
Environment Variables:
JAVA_HOME=/dsdvlp/java/jvm/jdk150
CLASSPATH=.:/dsdvlp/java/jar/jconnect520.jar:/dsdvlp/java/jar/vbjapp340.jar:/dsdvlp/java/jar/vbjorb340.jar:/dsdvlp/java/jar/javax_jndi120.jar
PATH=/usr/local/etc:/usr/lang:/usr/openwin/bin:/usr/ucb:/bin:/usr/etc:/usr/local/5/bin:/dsdvlp/bin/5:/dsdvlp/bin/4:/home/app/sybase/5/bin:/home/app/sybase/5/localscripts:/home/app/sybase/5/sqr:/home/app/lang:/home/app/lang/SC2.0.1:/usr/ccs/bin:/usr/local/opt/Acrobat3/bin:/dsdvlp/bin:.
LD_LIBRARY_PATH=/dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc/server:/dsdvlp/java/jvm/jdk1.5.0_03/jre/lib/sparc:/dsdvlp/java/jvm/jdk1.5.0_03/jre/../lib/sparc:/usr/lib:/usr/openwin/lib:/usr/local/5/lib:/dsdvlp/lib/5:/dstest/lib/5:/home/app/sybase/5/lib:/dstest/cats/sun4/lib:/tmitest/Opus/opus/lib
SHELL=/bin/csh
DISPLAY=CLI00184.mfil.local:1.0
OS=5
--------------- S Y S T E M ---------------
OS: Solaris 8 2/02 s28s_u7wos_08a SPARC
Copyright 2002 Sun Microsystems, Inc. All Rights Reserved.
Assembled 18 December 2001
uname:SunOS 5.8 Generic_117350-20 sun4u (T1 libthread)
rlimit: STACK 8192k, CORE 9216k, NOFILE 4096, AS infinity
load average:2.24 2.67 2.68
CPU:total 4 has_v8, has_v9, has_vis1, has_vis2, is_ultra3
Memory: 8k page, physical 8388608k(166384k free)
vm_info: Java HotSpot(TM) Server VM (1.5.0_03-b07) for solaris-sparc, built on Apr 13 2005 03:31:26 by unknown with unknown Workshop:0x550The very first suggestion I have is to move your VM to a more recent update of 1.5.0.
It looks like you are crashing with 5.0u3, and I'm pretty sure 5.0u16 is available. You
don't want to waste your time chasing a bug that's already been fixed. -
ORA-07445: exception encountered: core dump during database open
Hi,
after running OracleXE for 1 year on my notebook, I am running into a strange problem. During startup of the database ( opening the DB ) I get the following error:
ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [__VInfreq__kmmlrl+6329] [PC:0x2D7DF01] [ADDR:0x8] [UNABLE_TO_READ] []
I then tried to reinstall the DB but the problem persists. I have no clues what influences this error.
Any ideas welcomeThe machine is a notebook which is remotely administrated. The admin says there has not been a major software update.
The notebook has 2 GB of main memory and 4 GB of free disk space.
The database was up and running since 1 year. The first time I ran into this problem, was when I dropped a user from database. But that for sure was not the reason, as I totally removed oraclexe and reinstalled it from scratch and the problem persists.
Interestingly the installation doesn't fail, but the final startup, so it might be one of the last packages being installed which produces the error.
I'm looking forward to find some more details. -
Core dump while trying to access attributes in a node using SAX
I'm running the 10g xdk with Solaris 8 and was able to get the SAX examples working. However when I tried to add some code to the example code in order to access/print out attributes, I get a core dump.
code snippet:
void MyHandler::startElementNS ( oratext* qname, oratext* local, oratext* ns_URI, NodeListRef<xmlnode>* attrs_refp)
printf( " Element: %s\n", qname);
if (attrs_refp)
for (int i=0; i<attrs_refp->getLength(); i++) <-- core dumps with getLength() call
xmlnode* node = attrs_ref->item(i);
Am I not accessing the attributes correctly? Is there an easier way to do this that I have overlooked?I have very similar code for the parser and ran in to two problems. I am currently using 10.1.0.2.0 on Windows.
Problem 1:
Can the original poster of the following code tell me what you have under the ... section? I am trying to iterate through the attributes in attrs_refp
void MyHandler::startElementNS ( oratext* qname, oratext* local, oratext* ns_URI, NodeListRef<xmlnode>* attrs_refp)
printf( " Element: %s\n", qname);
if (attrs_refp)
for (int i=0; i<attrs_refp->getLength(); i++) <-- core dumps with getLength() call
xmlnode* node = attrs_ref->item(i);
In the ... section, I have the following line:
oratext* a = node->getname();
However, the compiler gives the following error:
error C2027: use of undefined type 'xmlnode'
Anyone have any clues as to why xmlnode is undefined?
Problem 2:
If I comment out the line oratext* a = node->getname();, I can successfully compile my program but I get a link error stating that both the methods item() and getLength() are unresolved external symbols. This lead me to believe these methods are not implemented in the Oracle XDK. user457758 seems to have gotten past this stage as he is getting a run time error so the compilation appears to be successful. I assume user457758 is using the latest 10.1.0.3.0 available on Solaris instead of 10.1.0.2.0 that I am currently using on Windows. Did anyone experience similar compilation errors?
Thanks for any help/info in advance. -
11gR2 RAC install fail when running root.sh script on second node
I get the errors:
ORA-15018: diskgroup cannot be created
ORA-15072: command requires at least 2 regular failure groups, discovered only 0
ORA-15080: synchronous I/O operation to a disk failed
[main] [ 2012-04-10 16:44:12.564 EDT ] [UsmcaLogger.logException:175] oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: ORA-15018: diskgroup cannot be created
ORA-15072: command requires at least 2 regular failure groups, discovered only 0
ORA-15080: synchronous I/O operation to a disk failed
I have tried the fix solutions from metalink note, but did not fix issue
11GR2 GRID INFRASTRUCTURE INSTALLATION FAILS WHEN RUNNING ROOT.SH ON NODE 2 OF RAC USING ASMLIB [ID 1059847.1Hi,
it looks like, that your "shared device" you are using is not really shared.
The second node does "create an ASM diskgroup" and create OCR and Voting disks. If this indeed would be a shared device, he should have recognized, that your disk is shared.
So as a result your VMware configuration must be wrong, and the disk you presented as shared disk is not really shared.
Which VMWare version did you use? It will not work correctly with the workstation or player edition, since shared disks are only really working with the server version.
If you indeed using the server, could you paste your vm configurations?
Furthermore I recommend using Virtual Box. There is a nice how-to:
http://www.oracle-base.com/articles/11g/OracleDB11gR2RACInstallationOnOEL5UsingVirtualBox.php
Sebastian -
Hi,
In one of my C applications, malloc is dumping core even if the pointer was free'd earlier (this is the behavior for both 32-bit and 64-bit).
So I re-built the application with -lmapmalloc, and it worked fine.
Can any one suggest whether it is ok to link the application with libmapmalloc or not.
Thanks.This is almost always cause by overwriting areas that you shouldn't be. Preload watchmalloc (man watchmalloc) and see where it blows up.
-r -
essbase generating core dump when ever I am starting ..this is rpm 11.1.2.3.000..
if I delete the core dump and tries to start , again it generated the core dump..
any help will be greatly appreciated.
thanksYou didn't say which OS it is running on, is this a new install?
So if you run ./startMaxl.sh you get a segmentation error as well, sometimes there is an issue with profile, variables to generate sgementation errors.
Cheers
John -
Netscape Directory Server core dumps
Hi,
I have NDS version 5.1 running on Solaris 10. My software is querying the LDAP at around 10-15 requests/sec. The LDAP memory is runnig at around 50M. (I have around 900 entries in the LDAP tree). At some random instance of time, the LDAP memory shoots up from 50M to 3.2 GB when it goes out of memory and then restarts. My software has a LDAP connection pool to it and one of the threads goes OutOfmemory after the error.
The LDAP does a core dump during the process. From the analysis of the core dump, it seems that it is trying to access some memory which is not part of the LDAP process map.
bash-3.00$ mdb core
mdb: failed to initialize /lib/libc_db.so.1: libthread_db call failed unexpectedly
mdb: warning: debugger will only be able to examine raw LWPs
Loading modules: [ libc.so.1 libuutil.so.1 ld.so.1 ]
::stacklibc_psr.so.1`memcpy+0x234(87c00038, 6b646174, 74776f6a, 8, 6b646174, 0)
libslapd.so`slapi_dup_control+0x94(53db678, 2ce38f8, fc93ff2c, 0, 2ce38f8,2ce38f8)
0x2f3b4(7e55c80, 2f30150, 2ce6bb8, 4c7fde8, 2f30150, ffffffff)
0xfef17d18(2f60164, fc940000, 0, 0, fef17c74, 1)
0xff2c8a20(0, 0, 0, 0, 0, 0)
::statusdebugging core file of ns-slapd (32-bit) from XXXXXX
initial argv:XXXXXXXX
threading model: multi-threaded using native lwps
status: process terminated by SIGSEGV (Segmentation Fault)
>
The failure is at address 0xfee8063c.
0xfee8063c::dislibc_psr.so.1`memcpy+0x20c: 0x2480012
libc_psr.so.1`memcpy+0x210: nop
libc_psr.so.1`memcpy+0x214: ldub [%i1], %o2
libc_psr.so.1`memcpy+0x218: stb %o2, [%i0]
libc_psr.so.1`memcpy+0x21c: add %i1, 1, %i1
libc_psr.so.1`memcpy+0x220: subcc %i3, 1, %i3
libc_psr.so.1`memcpy+0x224: 0x184ffffc
libc_psr.so.1`memcpy+0x228: add %i0, 1, %i0
libc_psr.so.1`memcpy+0x22c: ba +0x60 <libc_psr.so.1`memcpy+0x28c>
libc_psr.so.1`memcpy+0x230: nop
libc_psr.so.1`memcpy+0x234: ld [%i1], %o2
libc_psr.so.1`memcpy+0x238: st %o2, [%i0]
libc_psr.so.1`memcpy+0x23c: add %i1, 4, %i1
libc_psr.so.1`memcpy+0x240: subcc %i3, 4, %i3
// Looking for register i1
::regs%g0 = 0x00000000 %l0 = 0x87c00038
%g1 = 0xff37c840 libsh.so`_shi_sysAllocPool+0xd4 %l1 = 0xff16c2bc
%g2 = 0x00000000 %l2 = 0x07970778
%g3 = 0x0000036c %l3 = 0x05410000
%g4 = 0x00000001 %l4 = 0xff165a34
%g5 = 0x87c00038 %l5 = 0x00000000
%g6 = 0x00000000 %l6 = 0x01000000
%g7 = 0xfd06ca00 %l7 = 0x00000004
%o0 = 0x87c00038 %i0 = 0x87c00038
%o1 = 0x74776f72 %i1 = 0x6b646174
%o2 = 0xefe4617c %i2 = 0x74776f6a
%o3 = 0x00000002 %i3 = 0x00000008
%o4 = 0x00000002 %i4 = 0x6b646174
%o5 = 0x053d9c28 %i5 = 0x00000000
%o6 = 0xfc93fe00 %i6 = 0xfc93fe60
%o7 = 0xff0c0c44 libslapd.so`slapi_ch_malloc+0x64 %i7 = 0xff0e8e78
libslapd.so`slapi_dup_control+0x94
%psr = 0xfe401004 impl=0xf ver=0xe icc=nZvc
ec=0 ef=4096 pil=0 s=0 ps=0 et=0 cwp=0x4
%y = 0x00000002
%pc = 0xfee8063c libc_psr.so.1`memcpy+0x234
%npc = 0xfee80640 libc_psr.so.1`memcpy+0x238
%sp = 0xfc93fe00
0x6b646174::dismdb: failed to read instruction at 6b646174: no mapping for address
>
The register "i1"'s content was memory address 0x6b646174 which is not part of process address space.
Has anyone experienced similar problem before? I was trying to find the same in the forum but without any luck. This problem is happening quiet regularly.Seeems like content of register i1 got missed.. here is the content..
%l2 = 0x07970778
%l3 = 0x05410000
%l4 = 0xff165a34
%l5 = 0x00000000
%l6 = 0x01000000
%l7 = 0x00000004
%i0 = 0x87c00038
%i1 = 0x6b646174
%i2 = 0x74776f6a
%i3 = 0x00000008
%i4 = 0x6b646174
%i5 = 0x00000000
%i6 = 0xfc93fe60 -
Solaris 10, Ruby on Rails, Core Dumps
I am not very savvy with Solaris administration and we are having issues with configuring our system to run Ruby on Rails applications. We opted to use the Coolstack framework for our environment, thankfully without many hiccups. That said, our applications load but core dump during the second page load.
At this point, I'm not sure if it's our recent switch to Coolstack that is causing the problem. We've also worked to get a Mongrel cluster up and running, so that could also be the culprit.
What I'm looking for really is advice on how to pin down what the error is. I've learned about pstack to deal with the core dump file, but I'd have a better time understanding Chinese than that output. I also read about gdb and dbx, neither of which appear to be installed on our system. A strings on the file, another recommendation I found, spits out a bunch of gibberish.
How can I move forward in discovering the problem?I'm having the exact same problem with my Solaris server. I can't seem to get any form of Rails running on the system, and pages generated by Rails crash after a few page loads (not always the same number).
Did you have any luck tracking down the issue? I've asked around in a number of places and come up empty-handed every time. -
Timed out waiting for the CRS stack to start on Second node.
Hi,
I am trying to setup 2 node 11gR2 RAC in vmware. I face the Timed out waiting for the CRS stack to start error on second node while running root.sh.I checked the cluster log files located in /u01/app/11.2.0/grid/log/node1/alertnode2.log it shows as mentioned below.But when i logged into ASM instance and checked the diskgroup and it is in mount state on both nodes,So i am really confused why i ended up with error in 2nd node on running root.sh. Can anyone tell me how to correct this error or any other checking needs to be done?
/u01/app/11.2.0/grid/log/node1/alertnode2.log
[ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
2014-01-21 21:40:10.314
[crsd(25309)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
2014-01-21 21:40:11.217
[ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
2014-01-21 21:40:12.422
[crsd(25335)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
2014-01-21 21:40:13.327
[ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
2014-01-21 21:40:14.514
[crsd(25356)]CRS-1006:The OCR location +DATA is inaccessible. Details in /u01/app/11.2.0/grid/log/node2/crsd/crsd.log.
2014-01-21 21:40:15.439
[ohasd(23261)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
2014-01-21 21:40:15.440
[ohasd(23261)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
root.sh output
[root@node2 ~]# /u01/app/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2014-01-21 21:37:55: Parsing the host name
2014-01-21 21:37:55: Checking for super user privileges
2014-01-21 21:37:55: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_p
arams
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node 11gdb, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'node2'
CRS-2676: Start of 'ora.mdnsd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'node2'
CRS-2676: Start of 'ora.gipcd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'node2'
CRS-2676: Start of 'ora.gpnpd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node2'
CRS-2676: Start of 'ora.cssdmonitor' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'node2'
CRS-2672: Attempting to start 'ora.diskmon' on 'node2'
CRS-2676: Start of 'ora.diskmon' on 'node2' succeeded
CRS-2676: Start of 'ora.cssd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'node2'
CRS-2676: Start of 'ora.ctssd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'node2'
CRS-2676: Start of 'ora.drivers.acfs' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'node2'
CRS-2676: Start of 'ora.asm' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'node2'
CRS-2676: Start of 'ora.crsd' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'node2'
CRS-2676: Start of 'ora.evmd' on 'node2' succeeded
Timed out waiting for the CRS stack to start.
Regards,
007Hi,
Are you trying to install on Vmware Workstation or Vmware Server?
If you are using vmware server add below line in your .vmx file for both node
scsi1.sharedBus = "VIRTUAL"
First check which scsi serial number are you using as above I have selected scsi1 serial during disk addition, -
Cannot start BI server, cannot start cluster successfully, core dumped
Hi
I have the following configuration. The four machines are virtualized containers on solaris sparc
machine 1 - PS, java host, web logic
machine 2 - PS, java host, web logic
machine 3 - bi server, primary cluster controller, scheduler
machine 4 - bi server, secondary cluster controller, scheduler
However i dont need the scheduler and i have not configured them completely
I was able to successfully test on compete stack with no clustering and used only machines 1 and 3. I was able to login to the sample URL and see some graphs.
The problem started once i was trying the clustering. I looked at the admin docs and the installation and deployment guides and followed the instructions and carefully did the configuration.
I face multiple problems.
In single stack some time the BI server was randoly stopping itself for no reason after starting it. it was very random that i used to feel happy if it does not stop.
During clustering i was not able to start the cluster successfully at all. I did not want to worry about scheduler to start with and even the PS. I just wanted to connect to the BI server using the primary ccs and using the admin tool in the windows machine.
Today morning i tried to start the BI server and then there was a core dump in both the machines and then i am not even able to start the server at all. There was nothing in the log files and i dont know what to do.
The log files before the core dump as follows.
server log file
2009-07-30 18:18:47
[36007] Loading repository /software/oraclebi/server/Repository/samplesales.rpd.
2009-07-30 18:18:49
[14055] Loading subject area: Sample Sales ...
2009-07-30 18:18:49
[14056] Finished loading subject area: Sample Sales.
2009-07-30 18:18:49
[58002] Query Cache loaded with 7 entries from saved cache files.
2009-07-30 18:18:49
[43030] : Oracle BI Server (64-bit) started. Version: 10.1.3.4.1.090414.1900.
2009-07-30 18:19:11
[43031] : Oracle BI Server shutdown.
2009-07-30 18:19:12
[58001] Query Cache Statistics -- Hits:0(0%), Qualified Misses:0(0%), Unqualified Misses:0(0%).
2009-07-30 18:19:12
[14058] Unloaded all subject areas.
2009-07-30 18:25:25
[36007] Loading repository /software/oraclebi/server/Repository/samplesales.rpd.
2009-07-30 18:25:27
[14055] Loading subject area: Sample Sales ...
2009-07-30 18:25:27
[14056] Finished loading subject area: Sample Sales.
2009-07-30 18:25:27
[58002] Query Cache loaded with 7 entries from saved cache files.
2009-07-30 18:25:27
[43030] : Oracle BI Server (64-bit) started. Version: 10.1.3.4.1.090414.1900.
2009-07-30 18:26:25
[43071] A connection with Cluster Controller maa-zone4:9706 was established.
2009-07-30 18:30:08
[nQSError: 12002] Socket communication error at call=recv: (Number=131) Connection reset by peer
2009-07-30 18:30:08
[nQSError: 12002] Socket communication error at call=recv: (Number=131) Connection reset by peer
2009-07-30 20:57:24
[36007] Loading repository /software/oraclebi/server/Repository/samplesales.rpd.
2009-07-30 20:57:26
[14055] Loading subject area: Sample Sales ...
2009-07-30 20:57:26
[14056] Finished loading subject area: Sample Sales.
2009-07-30 20:57:26
[58002] Query Cache loaded with 7 entries from saved cache files.
2009-07-30 20:57:26
[43030] : Oracle BI Server (64-bit) started. Version: 10.1.3.4.1.090414.1900.
2009-07-30 20:58:39
[43031] : Oracle BI Server shutdown.
2009-07-30 20:58:39
[58001] Query Cache Statistics -- Hits:0(0%), Qualified Misses:0(0%), Unqualified Misses:0(0%).
2009-07-30 20:58:39
[14058] Unloaded all subject areas.
2009-07-30 21:01:40
[36007] Loading repository /software/oraclebi/server/Repository/samplesales.rpd.
2009-07-30 21:01:42
[14055] Loading subject area: Sample Sales ...
2009-07-30 21:01:42
[14056] Finished loading subject area: Sample Sales.
2009-07-30 21:01:42
[58002] Query Cache loaded with 7 entries from saved cache files.
2009-07-30 21:01:42
[43030] : Oracle BI Server (64-bit) started. Version: 10.1.3.4.1.090414.1900.
2009-07-30 21:08:51
[43071] A connection with Cluster Controller maa-zone4:9706 was established.
2009-07-30 21:10:31
[43065] The connection with Cluster Controller maa-zone4:9706 was lost.
2009-07-30 21:12:59
[43071] A connection with Cluster Controller maa-zone4:9706 was established.
:x
$ ls
cluster controller log file
2009-07-30 21:12:59
[71008] Cluster Controller started. Version: 10.1.3.4.1.090414.1900
2009-07-30 21:12:59
[71021] An attempt to establish a connection with Oracle BI Server maa-zone03:9703 failed.
2009-07-30 21:12:59
[71020] A connection with Oracle BI Server maa-zone4:9703 was established.
2009-07-30 21:12:59
[71010] Oracle BI Server maa-zone4:9703 has transitioned to ONLINE state.
2009-07-30 21:13:14
[71050] This Cluster Controller is promoted to Controlling because there has been no recent contact from
the Primary Cluster Controller.
2009-07-30 21:13:14
[71071] An attempt to establish a connection with Oracle BI Scheduler maa-zone03:9705 failed.
2009-07-30 21:13:14
[71068] Oracle BI Scheduler maa-zone03:9705 has transitioned to OFFLINE state.
2009-07-30 21:13:14
[71071] An attempt to establish a connection with Oracle BI Scheduler maa-zone4:9705 failed.
2009-07-30 21:13:14
[71068] Oracle BI Scheduler maa-zone4:9705 has transitioned to OFFLINE state.
2009-07-30 23:10:34
[71019] The connection with Oracle BI Server maa-zone4:9703 was lost.
2009-07-30 23:10:34
[71011] Oracle BI Server maa-zone4:9703 has transitioned to OFFLINE state.
2009-07-30 23:10:39
[71021] An attempt to establish a connection with Oracle BI Server maa-zone4:9703 failed.
your help is appreciated.
thanks
chaitanya kadaruall the four machines are four containers in a solaris server.
The problem was with the hosts file, there were not entries matching the ipaddress and the hotname.
After i fixed it now i have a stable BI servers and cluster controllers.
Now i configured the presentation services to connect to the BI servers but i am getting the following exception.
Type: Error
Severity: 42
Time: Thu Aug 6 13:04:02 2009
File: project/webcomm/socket.cpp Line: 892
Properties: ThreadID-6
Location:
saw.rpc.client.impl.verify
saw.rpc.client.impl.initFixedRPC
saw.rpc.client.cluster
saw.threads
An error occurred during execution of "connect". Connection refused [Socket:15]
Any idea?
thanks -
Core dump when starting extract, no error messages
Hello,
This is a basic one way replication, Data Pump and Replicat start but Extract stops at once.
In ggserr.log there's only a message: EXTRACT EORATE02 starting, nothing else. In ggsci it can be seen that it's stopped.
CHECKPARAMS didn't give the message "Parameters processed successfully", neither did it give error messages
when starting the extract.
There are no old Oracle processes which could prevent Extract from starting.
There's EORATE02.pce in dirpcs, although EORATE02 is not running. If I delete it and try to start the extract,
the extract will immediately stop.
A core file has been created in GG_HOME when trying to start the extract. I haven't been able to find out how to read it.
Source and target are on Aix 6100-06. The source database is Oracle 10.1.0.5, target database is Oracle 11.2.0.3.
The installation files are V32435-01.zip for db 10.1.0.5 (ggs_AIX_ppc_ora10.1_64bit.tar), V32437-01.zip
for db 11.2.0.3 (ggs_AIX_ppc_ora11g_64bit.tar). This is GoldenGate 11.2.1.0.1.
The db:s are on Asm.
There's only 1 table in source, 1 in target, they are identical and empty.
There's enough space in the OS.
If I run the extract in OS (extract paramfile dirprm/eorate02.prm), it ends like this without more information:
TRANLOGOPTIONS ASMUSER ggs_owner@HOSTNAME:1521/+ASM , ASMPASSWORD *********
Abort(coredump)
This happens in both ways of extract described below.
(1)
EXTRACT EORATE02
USERID ggs_owner, PASSWORD [PASSWORD]
EXTTRAIL ./dirdat/data02/te
TRANLOGOPTIONS ASMUSER ggs_owner@HOSTNAME:1521/+ASM , ASMPASSWORD [PASSWORD]
TABLE USER.myobjects2;
(2)
EXTRACT EORATE02
USERID ggs_owner, PASSWORD [PASSWORD]
EXTTRAIL ./dirdat/data02/te
tranlogoptions COMPLETEARCHIVEDLOGONLY ALTARCHIVELOGDEST /backup/arch_log/abcd/ABCD/archivelog
TRANLOGOPTIONS ASMUSER ggs_owner@HOSTNAME:1521/+ASM , ASMPASSWORD [PASSWORD]
TABLE USER.myobjects2;
Here's more information on this setup and commands which have been used:
add extract EORATE02 tranlog begin now, threads 1
add exttrail ./dirdat/data02/te, extract EORATE02, megabytes 5
EXTRACT EPMPTE02
PASSTHRU
RMTHOST HOSTNAME, MGRPORT 13447
RMTTRAIL /u01/app/oracle/gg11210/for_11g/dirdat/data02/tr
TABLE USER.myobjects2;
ADD EXTRACT EPMPTE02 EXTTRAILSOURCE ./dirdat/data02/te
ADD RMTTRAIL /u01/app/oracle/gg11210/for_db11/dirdat/data02/tr EXTRACT EPMPTE02, MEGABYTES 5
Target
REPLICAT RORATE02
USERID ggs_owner, PASSWORD [PASSWORD]
HANDLECOLLISIONS
ASSUMETARGETDEFS
DISCARDFILE ./dirrpt/RORATE02.DSC, PURGE
MAP USER.MYOBJECTS2, TARGET USER.MYOBJECTS2;
add replicat RORATE02, exttrail /u01/app/oracle/gg11210/for_11g/dirdat/data02/tr
add checkpointtable ggs_owner.ggschkpt
Grants
Source:
grant connect, resource to ggs_owner;
grant select any dictionary, select any table to ggs_owner;
grant create table, alter any table to ggs_owner;
grant flashback any table to ggs_owner;
grant execute on dbms_flashback to ggs_owner;
grant execute on utl_file to ggs_owner;
grant create session, alter session to ggs_owner;
Target:
grant create session, alter session, resource, connect, select any dictionary to ggs_owner;
grant select any table, insert any table, update any table, delete any table to ggs_owner;
grant create table, alter any table, lock any table to ggs_owner;
Source
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
ggsci add trandata USER.myobjects2
mgr.prm
--GoldenGate Manager parameter file
PORT 13346
Target
GLOBALS
CHECKPOINTTABLE ggs_owner.ggschkpt
mgr.prm
--GoldenGate Manager parameter file
PORT 13447
Thanks for reading this far. Possible comments will be much appreciated!
Best regards
Edited by: user11323660 on 14.6.2012 6:39
Edited by: user11323660 on 14.6.2012 6:41Hello,
Could you wrap your process definitions in proper code brackets so its easier to understand.
Also I'm not sure what your problem is other then you are receiving core dumps. I'm happy to help.
Two small comment though regarding your ASM config
1 . Dont not use ggs_owner as the asm user
TRANLOGOPTIONS ASMUSER ggs_owner@HOSTNAME:1521/+ASM , ASMPASSWORD *********2. Create a TNSNAMES entry for your ASM instance and tehn use the following:
TRANLOGOPTIONS ASMUSER SYS@ASM, ASMPASSWORD *****Also here is an example of an extract process that I've setup on RAC database with ASM
EXTRACT EWFMMTLP
EXTTRAIL ./dirdat/ew
USERID GGATE@MY_DB, PASSWORD ggatehcc
DDLOPTIONS ADDTRANDATA RETRYOP RETRYDELAY 20 MAXRETRIES 5 REPORT
TRANLOGOPTIONS EXCLUDEUSER ggate
TRANLOGOPTIONS ASMUSER SYS@ASM, ASMPASSWORD ****
TRANLOGOPTIONS ALTARCHIVELOGDEST /u02/oracle/fast_recovery_area/MY_DB/archivelog
TRANLOGOPTIONS PURGEORPHANEDTRANSACTIONS
STATOPTIONS RESETREPORTSTATS
REPORT AT 00:01
REPORTROLLOVER AT 00:01
REPORTCOUNT EVERY 60 SECONDS, RATELet me know if that helps
Jan S. -
Xsan core dumping on volume start.
Hey guys,
I reciently upgraded to 10.4.7 (in hopes of fixing a kernel memory leak when hammering the san actualy) and now the xsan will not mount at all. I have applied all patches including the new xserver patch, and no love. When I try to start the volume it core dumps, but indicates that it started. However, I can not mount anything to it. The results of /Library/Filesystems/Xsan/data/CYBERSAN/log/cvlog when I try to start the volume is below.
Any help or suggestions greatly apreciated.
Thanks very much.
Most sincerely,
-- James
The first issue seems to be this one here:
[0630 15:46:27] 0x1886400 (*FATAL*) PANIC: /Library/Filesystems/Xsan/bin/fsm "alsplaytree_insert: CvSplayInsert(sblk) failed!" file alloc.c, line 447
Complete log of the "start" process is below.
[0630 15:46:19.570614] 0x1801000 (Debug) sigwait handler starting
[0630 15:46:19] 0xa000ed98 (Info) Server Revision 2.7.200 Build 92.1 Built for Darwin 8.0 Created on Fri Apr 14 20:01:22 PDT 2006
[0630 15:46:19] 0xa000ed98 (Info)
Configuration:
DiskTypes-4
Disks-4
StripeGroups-2
ForceStripeAlignment-1
MaxConnections-75
ThreadPoolSize-128
StripeAlignSize-256
FsBlockSize-8192
BufferCacheSize-32M
InodeCacheSize-8192
RestoreJournal-Disabled
RestoreJournalDir-None
[0630 15:46:19] 0xa000ed98 (Info) Self (10.0.1.202) IP address is 10.0.1.202 .
[0630 15:46:19.583710] 0xa000ed98 (Debug) No fsports file - port range enforcement disabled.
[0630 15:46:19] 0xa000ed98 (Info) Listening on TCP socket 10.0.1.202:49543
[0630 15:46:19] 0xa000ed98 (Info) Node [0] [10.0.1.202:49543] File System Manager Login.
[0630 15:46:19] 0xa000ed98 (Info) Service standing by on host '10.0.1.202:49543'.
[0630 15:46:20.629890] 0xa000ed98 (Debug) Standby service - NSS ping from 10.0.1.21:52043.
[0630 15:46:20.630535] 0xa000ed98 (Debug) FOUsurpCheck: read ARB info (pass 1): host (10.0.1.202:49489) conns 999999 age 1151707516.00 secs his delta 0.00 secs my delta 0.00 secs.
[0630 15:46:20.630634] 0xa000ed98 (Debug) FOUsurpCheck: polling ARB block to check for active peer (pass 1).
[0630 15:46:21.631149] 0xa000ed98 (Debug) FOUsurpCheck: read ARB info (pass 2): host (10.0.1.202:49489) conns 999999 age 1151707516.00 secs his delta 0.00 secs my delta 1.00 secs.
[0630 15:46:21.631211] 0xa000ed98 (Debug) FOUsurpCheck: ARB is already mine.
[0630 15:46:21] 0xa000ed98 (Info) Branding Arbitration Block (attempt 1) votes 2.
[0630 15:46:23.634395] 0xa000ed98 (Debug) Cannot find fail over script [/Library/Filesystems/Xsan/bin/cvfail.10.0.1.202] - looking for generic script.
[0630 15:46:23] 0xa000ed98 (Info) Launching fail over script [/Library/Filesystems/Xsan/bin/cvfail 10.0.1.202 49543 CYBERSAN]
[0630 15:46:24.756291] 0xa000ed98 (Debug) Starting journal log recovery.
[0630 15:46:24.966758] 0xa000ed98 (Debug) Completed journal log recovery.
[0630 15:46:24.967322] 0xa000ed98 (Debug) Inodeinit_postactivation: FsStatus 0x3, Brl_ResyncState 1
[0630 15:46:24] 0x1889c00 (Info) FSM Alloc: Loading Stripe Group "Meta". 372.58 GB.
[0630 15:46:24] 0x188b200 (Info) FSM Alloc: Loading Stripe Group "VideoStorage". 5.46 TB.
[0630 15:46:25] 0x1889c00 (Info) FSM Alloc: Stripe Group "Meta" active.
[0630 15:46:27] 0x188b200 (Info) FSM Alloc: free blocks 4700392 with 947200 blocks reserved for client delayed buffers.
[0630 15:46:27] 0x188b200 (Info) FSM Alloc: reserved blocks are product of MaxConnections(75) and MaxMBPerClientReserve(100).
[0630 15:46:27] 0x188b200 (Info) FSM Alloc: Stripe Group "VideoStorage" active.
[0630 15:46:27] 0xa000ed98 (Info) File System Service 'CYBERSAN[0]' now active on host '10.0.1.202:49543'.
[0630 15:46:27.598917] 0xa000ed98 (Debug) Node [1] [10.0.1.202:49547] connected.
[0630 15:46:27] 0x1886400 (*FATAL*) PANIC: /Library/Filesystems/Xsan/bin/fsm "alsplaytree_insert: CvSplayInsert(sblk) failed!" file alloc.c, line 447
[0630 15:46:27] 0x1886400 (*FATAL*) PANIC: wait 3 secs for journal to flush
[0630 15:46:27.609425] 0xa000ed98 (Debug) Active service - NSS ping from 10.0.1.184:52477.
[0630 15:46:27.609472] 0xa000ed98 (Debug) Listener_thread: flushing journal.
[0630 15:46:27.609492] 0xa000ed98 (Debug) Listener_thread: journal flush complete.
[0630 15:46:27.609519] 0xa000ed98 (Debug) FSM I/O SUMMARY writes total/20.
[0630 15:46:27.609535] 0xa000ed98 (Debug) FSM I/O SUMMARY writes journal/1 sb/0 buf/2 abm/0.
[0630 15:46:27.609555] 0xa000ed98 (Debug) FSM I/O SUMMARY writes inode/1 ganged/2 (12.50%).
[0630 15:46:27.609600] 0xa000ed98 (Debug) FSM wait SUMMARY inode pool expand waits/0.
[0630 15:46:27.609615] 0xa000ed98 (Debug) FSM wait SUMMARY journal waits/0.
[0630 15:46:27.609630] 0xa000ed98 (Debug) FSM wait SUMMARY journal bytes used avg/156501 max/158208.
[0630 15:46:27.609644] 0xa000ed98 (Debug) FSM wait SUMMARY free buffer waits/0.
[0630 15:46:27.609657] 0xa000ed98 (Debug) FSM wait SUMMARY free inode waits/0.
[0630 15:46:27.609736] 0xa000ed98 (Debug) FSM wait SUMMARY revokes/0 avg/0 min/0 max/0.
[0630 15:46:27.609756] 0xa000ed98 (Debug) FSM threads SUMMARY max busy hi-prio/0 lo-prio/0.
[0630 15:46:27.609770] 0xa000ed98 (Debug) FSM threads SUMMARY max busy dmig/0 events/0.
[0630 15:46:27.609784] 0xa000ed98 (Debug) FSM msg queue SUMMARY hi-prio now/0 min/0 max/0.
[0630 15:46:27.609798] 0xa000ed98 (Debug) FSM msg queue SUMMARY lo-prio now/0 min/0 max/0.
[0630 15:46:27.609813] 0xa000ed98 (Debug) FSM msg queue SUMMARY dmig now/0 min/0 max/0.
[0630 15:46:27.609827] 0xa000ed98 (Debug) FSM msg queue SUMMARY events now/0 min/0 max/0.
[0630 15:46:27.609842] 0xa000ed98 (Debug) FSM cache SUMMARY inode lookups/11 misses/7 hits/36.36%.
[0630 15:46:27.609857] 0xa000ed98 (Debug) FSM cache SUMMARY free incore inodes now/8191 min/8189 max/8192.
[0630 15:46:27.609875] 0xa000ed98 (Debug) FSM cache SUMMARY buffer lookups/21 misses/10 hits/52.38%.
[0630 15:46:27.609890] 0xa000ed98 (Debug) FSM cache SUMMARY free buffers now/4096 min/1 max/4096.
[0630 15:46:27.609904] 0xa000ed98 (Debug) FSM cache SUMMARY attrs now/0 min/0 max/0.
[0630 15:46:27.609919] 0xa000ed98 (Debug) FSM extent SUMMARY extent lookups/0 misses/0 hits/0.00%.
[0630 15:46:27.609934] 0xa000ed98 (Debug) FSM extent SUMMARY hint tries/0 misses/0 hits/0.00%.
[0630 15:46:27.609989] 0xa000ed98 (Debug) SG SUMMARY Meta space total/372.58 GB free/371.90 GB (99.82%)
[0630 15:46:27.610011] 0xa000ed98 (Debug) SG SUMMARY Meta space minfree/371.90 GB (99.82%) maxfree/371.90 GB (99.82%)
[0630 15:46:27.610027] 0xa000ed98 (Debug) SG SUMMARY Meta alloc extent cnt/0 avgsize/0.00 B.
[0630 15:46:27.610045] 0xa000ed98 (Debug) SG SUMMARY VideoStorage space total/5.46 TB free/35.91 GB (0.64%)
[0630 15:46:27.610065] 0xa000ed98 (Debug) SG SUMMARY VideoStorage space minfree/35.86 GB (0.64%) maxfree/35.90 GB (0.64%)
[0630 15:46:27.610082] 0xa000ed98 (Debug) SG SUMMARY VideoStorage alloc extent cnt/0 avgsize/0.00 B.
[0630 15:46:27.610102] 0xa000ed98 (Debug) PIO HiPriWr SUMMARY Briana2_META cnt/1 maxq/0.
[0630 15:46:27.610119] 0xa000ed98 (Debug) PIO HiPriWr SUMMARY Briana2_META avg/482 min/482 max/482.
[0630 15:46:27.610135] 0xa000ed98 (Debug) PIO HiPriWr SUMMARY Briana2_META sysavg/480 sysmin/480 sysmax/480.
[0630 15:46:27.610152] 0xa000ed98 (Debug) PIO HiPriWr SUMMARY Briana2_META avglen/2560 minlen/2560 maxlen/2560.
[0630 15:46:27.610168] 0xa000ed98 (Debug) PIO Read SUMMARY Briana2_META cnt/127 maxq/5.
[0630 15:46:27.610184] 0xa000ed98 (Debug) PIO Read SUMMARY Briana2_META avg/16471 min/345 max/84712.
[0630 15:46:27.610200] 0xa000ed98 (Debug) PIO Read SUMMARY Briana2_META sysavg/7001 sysmin/326 sysmax/23029.
[0630 15:46:27.610285] 0xa000ed98 (Debug) PIO Read SUMMARY Briana2_META avglen/970760 minlen/512 maxlen/2097152.
[0630 15:46:27.610302] 0xa000ed98 (Debug) PIO Write SUMMARY Briana2_META cnt/19 maxq/1.
[0630 15:46:27.610342] 0xa000ed98 (Debug) PIO Write SUMMARY Briana2_META avg/1634 min/303 max/21173.
[0630 15:46:27.610360] 0xa000ed98 (Debug) PIO Write SUMMARY Briana2_META sysavg/1015 sysmin/290 sysmax/11011.
[0630 15:46:27.610376] 0xa000ed98 (Debug) PIO Write SUMMARY Briana2_META avglen/5362 minlen/512 maxlen/8192.
[0630 15:46:27] 0x1886400 (*FATAL*) PANIC: aborting threads now.On mdc open terminal and goto;
/Library/Filesystems/Xsan/bin
run as root
cvfsck -j <selectyourvolume>
to make sure journal is written out to volume
then run
cvfsck <selectyourvolume> until no errors are reported
you can run it with -n option the first time to make sure you do only a readonly check (no modification, hence the FS will not be fixed) -
I want to know why display core dump when arch starts!
I want to know why display core dump when arch starts!
2008. 06 iso!
u can see display coredump in starting......GAH, you silly people!
Now I'm going to have to go off and write a program that selects a random string from a file to show on startup!
(I'll probably fill the file with all the Arch snapshot names)
EDIT: I should mention that you're silly because you managed to get me following a random compulsion.
EDIT AGAIN: Done! I now get a random snapshot name in that spot whenever my system boots. I can post the script if anyone wants it, it's three lines.
EDIT ONE LAST TIME: Oh wait, three lines and a extra file full of snapshot names.
Last edited by arew264 (2008-07-19 02:35:05)
Maybe you are looking for
-
Can't connect PHP/MySQL 404 error
Wow, the learning curve is killing me. Yet another challenge. I'm using Dreamweaver CC and this tutorial http://www.adobe.com/devnet/dreamweaver/articles/setup_php.html I'm at this step: Open comments.php. You must have a PHP page open in the Documen
-
Is it OK to connect the i-pod to the computer speakers???
Can anyone tell me if it is alright to connect my 5th generation i-pod to the computer speakers as I don't have a sound system.My friend connected his i-pod mini to the PC speakers and it seems though as if it has damaged his earphones. IBM Windows
-
Hi Gurus, I have 2 questions. I'm working on BI 7 and I have migrated a DataSource from 3.5 into BI 7. After migration, it's showing unequal sign (M version is different from A version). When I checked the BI Content I found the DataSource(InfoSource
-
Error in connecting with Netweaver2004s Preview
I'm a new guy in this area. After some days working , I finally installed NW04s preview correctly on windows2003. Now I've installed .Net2003 , PDK for .net2003 , Protal Runtime and SAP .NET Connector 2.0. According to tutorials of Portal Development
-
HT1363 Hi, I have an ipod Classic. I tried all the steps, but still the error exists
I am not able to reset my Ipod Classic. Please help