Can't start cluster, 2 node 3.3 cluster lost 2 quorum disks

Hi,
I have a 2 node cluster 1 one iscsi quorum disk, I was in the middle of migrating the quorum device to another iscsu disk, when the servers lost contact with the disks(iscsi targe problem), so the 2 cluster nodes where left with no quorum, because of the 2 quorum devices 3 votes are needed, I only have 2 votes from the 2 cluster nodes.
iscsi disks are back online, but the cluster/quorum isn't able to get hold of them.
May 11 11:21:59 vmcluster1 genunix: [ID 965873 kern.notice] NOTICE: CMM: Node vmcluster2 (nodeid = 1) with votecount = 1 added.
May 11 11:21:59 vmcluster1 genunix: [ID 965873 kern.notice] NOTICE: CMM: Node vmcluster1 (nodeid = 2) with votecount = 1 added.
May 11 11:22:04 vmcluster1 genunix: [ID 832830 kern.warning] WARNING: CMM: Open failed for quorum device /dev/did/rdsk/d1s2 with error 2.
May 11 11:22:10 vmcluster1 genunix: [ID 832830 kern.warning] WARNING: CMM: Open failed for quorum device /dev/did/rdsk/d2s2 with error 2.
May 11 11:22:14 vmcluster1 genunix: [ID 884114 kern.notice] NOTICE: clcomm: Adapter e1000g2 constructed
May 11 11:22:15 vmcluster1 genunix: [ID 884114 kern.notice] NOTICE: clcomm: Adapter e1000g1 constructed
May 11 11:22:15 vmcluster1 genunix: [ID 843983 kern.notice] NOTICE: CMM: Node vmcluster1: attempting to join cluster.
May 11 11:22:15 vmcluster1 e1000g: [ID 801725 kern.info] NOTICE: pci8086,100e - e1000g[2] : link up, 1000 Mbps, full duplex
May 11 11:22:16 vmcluster1 e1000g: [ID 801725 kern.info] NOTICE: pci8086,100e - e1000g[1] : link up, 1000 Mbps, full duplex
May 11 11:23:20 vmcluster1 genunix: [ID 832830 kern.warning] WARNING: CMM: Open failed for quorum device /dev/did/rdsk/d1s2 with error 2.
May 11 11:23:25 vmcluster1 genunix: [ID 832830 kern.warning] WARNING: CMM: Open failed for quorum device /dev/did/rdsk/d2s2 with error 2.
May 11 11:23:25 vmcluster1 genunix: [ID 980942 kern.notice] NOTICE: CMM: Cluster doesn't have operational quorum yet; waiting for quorum.
Looks like the server thinks the ID of the disks have changed:
[root@vmcluster1:/]# scdidadm -L (05-11 11:27)
1 vmcluster1:/dev/rdsk/c3t5d0 /dev/did/rdsk/d1
1 vmcluster2:/dev/rdsk/c3t5d0 /dev/did/rdsk/d1
2 vmcluster1:/dev/rdsk/c3t4d0 /dev/did/rdsk/d2
2 vmcluster2:/dev/rdsk/c3t4d0 /dev/did/rdsk/d2
3 vmcluster2:/dev/rdsk/c1t0d0 /dev/did/rdsk/d3
4 vmcluster1:/dev/rdsk/c1t0d0 /dev/did/rdsk/d4
5 vmcluster2:/dev/rdsk/c3t6d0 /dev/did/rdsk/d5
5 vmcluster1:/dev/rdsk/c3t6d0 /dev/did/rdsk/d5
6 vmcluster2:/dev/rdsk/c1t1d0 /dev/did/rdsk/d6
7 vmcluster1:/dev/rdsk/c1t1d0 /dev/did/rdsk/d7
[root@vmcluster1:/]# scdidadm -r (05-11 11:27)
scdidadm: Device ID "vmcluster1:/dev/rdsk/c3t5d0" does not match physical device ID for "d1".
Warning: Device "vmcluster1:/dev/rdsk/c3t5d0" might have been replaced.
scdidadm: Device ID "vmcluster1:/dev/rdsk/c3t4d0" does not match physical device ID for "d2".
Warning: Device "vmcluster1:/dev/rdsk/c3t4d0" might have been replaced.
scdidadm: Device ID "vmcluster1:/dev/rdsk/c3t6d0" does not match physical device ID for "d5".
Warning: Device "vmcluster1:/dev/rdsk/c3t6d0" might have been replaced.
scdidadm: Could not save DID instance list to file.
scdidadm: File /etc/cluster/ccr/global/did_instances exists.
Disks are ok, and accesible from format
[root@vmcluster1:/]# echo | format (05-11 11:28)
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c1t0d0 <DEFAULT cyl 8351 alt 2 hd 255 sec 63>
/pci@0,0/pci8086,2829@d/disk@0,0
1. c1t1d0 <DEFAULT cyl 1020 alt 2 hd 64 sec 32>
/pci@0,0/pci8086,2829@d/disk@1,0
2. c3t4d0 <IET-VIRTUAL-DISK-0-1.00GB>
/iscsi/[email protected]%3Astorage.lun10001,0
3. c3t5d0 <DEFAULT cyl 497 alt 2 hd 64 sec 32>
/iscsi/[email protected]%3Astorage.lun20001,1
4. c3t6d0 <DEFAULT cyl 496 alt 2 hd 64 sec 32>
/iscsi/[email protected]%3Astorage.lun30001,2
Is there a way to remove a quorum device without the cluster online?
Or is there another alternative?, try and fix the did problem ?
Thanks!

This is the primary reason that you have one and only one quorum device. There are many failure modes that result in your cluster not starting. Looks like your only option is to hand edit the CCR. If this is a production cluster, please log a service desk ticket for the full procedure. If it's just a development cluster and you are happy to take a risk, the basic outline is (IIRC):
1. Boot nodes into non-cluster mode
2. Edit /etc/cluster/ccr/global/infrastructure and either remove the cluster.quorum_devices.* entries or set the votecount to 0
3. cd /etc/cluster/ccr/global
4. Run /usr/cluster/lib/sc/ccradm replace -i infrastructure infrastructure
5. Reboot back into cluster mode
6. Add one new quorum disk
You may need to run one or more of:
# cldev refresh
# cldev check
# cldev clean
# cldev populate
to get the right DID entries between steps 5 and 6.
Tim
---

Similar Messages

How can i start cluster stack ?

Dears,,
I am using Oracle Enterprise Linux 32-bit
and trying to install Oracle Clusterware 10g
While trying to use ocfs2console to Configure Oracle Cluster File System (OCFS2)
Error appear to me as following:
Could not start cluster stack error.This must be resolved before any OCFS2 filesystem can be mounted.
How can solve this please?
thanks & regards,,

M o h a m m e d wrote:
While trying to use ocfs2console to Configure Oracle Cluster File System (OCFS2)
Error appear to me as following:
Could not start cluster stack error.This must be resolved before any OCFS2 filesystem can be mounted.What is the contents of the file +/etc/ocfs2/cluster.conf+ ? Does the same file (same contents) exist on all nodes in the cluster?
What is the contents of the file +/etc/fstab+ ? Is the ocfs2 device configured to be mounted on a mount point? Does this mount point exist? Is this (config and mount point) identical across all cluster nodes?
Lastly, what happens when you issue the following commands (on any cluster node) as root:
+/etc/init.d/o2cb load+
+/etc/init.d/o2cb online+
+/etc/init.d/ocfs2 start+

Oracle RAC 10g: ASM can't start in one node.

Hi all,
I succesfully installed RAC in Enterprise Linux. Node1 starts correctly (ASM and DB) but Node2 doesn´t. This is the error in Node2 when I try to startup de ASM instance:
SQL> startup
ASM instance started
Total System Global Area 92274688 bytes
Fixed Size 1217884 bytes
Variable Size 65890980 bytes
ASM Cache 25165824 bytes
ORA-15032: not all alterations performed
ORA-15130: diskgroup "DG1" is being dismounted
ORA-15066: offlining disk "VOL1" may result in a data loss
Any ideas?
Thanks!

Hi, you will must check the configuration of your shared storage, maybe the problem is there, you can verify the some log information in the clusterware layer, and you can find more information in the bdump directory of ASM Instance.
Please paste here the alert logfile of ASM Instance.
Luck.
Have a good day.

Can't start http

Hello all
i've installed oracle9i (9.2.0.1.0) and oracle9ias and oracle9ifs (9.0.2)
on a sun60 with Solrais 8.
After installing, i run ifsdeployear script then emctl start (in fact, i restart em).
Then i launch the node, launch the dc, start the dc.
Here i can start the node but i can't start the http node.
Thanks for any help

for information, i start ifs from oracle9ias web interface.

Can not start messaging server resource group in cluster 3.2

Hi all,
Please help in the following issue.
I am not able to start resource group (msg-rg) and following is the error:
ms1@root# clrg online -M -e msg-rg
clrg: (C748634) Resource group msg-rg failed to start on chosen node and might fail over to other node(s)
clrg: (C135343) No primary node could be found for resource group msg-rg; it remains offline
scstat output (remove some for brief description)
-- Device Group Servers --
Device Group Primary Secondary
Device group servers: SJMS ms1 ms2
-- Device Group Status --
Device Group Status
Device group status: SJMS Online
-- Resource Groups and Resources --
Group Name Resources
Resources: msg-rg mail msg-hasp-rs msg-rs
-- Resources --
Resource Name Node Name State Status Message
Resource: mail ms1 Offline Offline - LogicalHostname offline.
Resource: mail ms2 Offline Offline - LogicalHostname offline.
Resource: msg-hasp-rs ms1 Offline Offline
Resource: msg-hasp-rs ms2 Offline Offline
Resource: msg-rs ms1 Offline Offline - Stop Succeeded
Resource: msg-rs ms2 Offline Offline - Stop Succeeded
Following is the from /var/adm/messages (remove some for brief description)
Sep 26 12:25:19 ms1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <ims_svc_start> for resource <msg-rs>, resou
rce group <msg-rg>, node <ms1>, timeout <300> seconds
Sep 26 12:25:19 ms1 Cluster.RGM.rgmd: [ID 784560 daemon.notice] resource msg-rs status on node ms1 change to R_FM_UNKNOWN
Sep 26 12:25:19 ms1 Cluster.RGM.rgmd: [ID 922363 daemon.notice] resource msg-rs status msg on node ms1 change to <Starting>
Sep 26 12:25:19 ms1 Cluster.RGM.rgmd: [ID 751138 daemon.notice] 47 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/r
gm/rt/hafoip/hafoip_monitor_start>:tag=<msg-rg.mail.7>: Calling security_clnt_connect(..., host=<ms1>, sec_type {0:WEAK, 1:ST
RONG, 2:DES} =<1>, ...)
Sep 26 12:25:19 ms1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/sun/comms/msg
scha/bin/imssvc_start>:tag=<msg-rg.msg-rs.0>: Calling security_clnt_connect(..., host=<ms1>, sec_type {0:WEAK, 1:STRONG, 2:
DES} =<1>, ...)
Sep 26 12:25:19 ms1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_monitor_start> completed successfully for reso
urce <mail>, resource group <msg-rg>, node <ms1>, time used: 0% of timeout <300 seconds>
Sep 26 12:25:19 ms1 Cluster.RGM.rgmd: [ID 443746 daemon.notice] resource mail state on node ms1 change to R_ONLINE
Sep 26 12:26:53 ms1 Cluster.PMF.pmfd: [ID 887656 daemon.notice] Process: tag="msg-rg,msg-rs,1.svc", cmd="/bin/sh -c /opt/sun/
comms/messaging64/bin/start-msg watcher", Failed to stay up.
Sep 26 12:26:55 ms1 Cluster.RGM.rgmd: [ID 784560 daemon.notice] resource msg-rs status on node ms1 change to R_FM_ONLINE
Sep 26 12:26:55 ms1 Cluster.RGM.rgmd: [ID 922363 daemon.notice] resource msg-rs status msg on node ms1 change to <Start succe
eded.>
Sep 26 12:26:55 ms1 Cluster.PMF.pmfd: [ID 819736 daemon.notice] PMF is restarting process that died: tag=msg-rg,msg-rs,1.svc,
cmd_path=/bin/sh -c /opt/sun/comms/messaging64/bin/start-msg watcher, max_retries=0, num_retries=0
Sep 26 12:27:25 ms1 SC[SUNW.ims:7.0,msg-rg,msg-rs,ims_svc_start]: [ID 141062 daemon.error] Failed to connect to host 192.168.
0.250 and port 27442: Connection refused.
Sep 26 12:29:55 ms1 last message repeated 6 times
Sep 26 12:30:26 ms1 Cluster.RGM.rgmd: [ID 764140 daemon.error] Method <ims_svc_start> on resource <msg-rs>, resource group <m
sg-rg>, node <ms1>: Timeout.
Sep 26 12:30:26 ms1 Cluster.RGM.rgmd: [ID 443746 daemon.notice] resource msg-rs state on node ms1 change to R_START_FAILED
Sep 26 12:30:26 ms1 Cluster.RGM.rgmd: [ID 529407 daemon.notice] resource group msg-rg state on node ms1 change to RG_PENDING_
OFF_START_FAILED
Sep 26 12:30:26 ms1 Cluster.RGM.rgmd: [ID 784560 daemon.notice] resource msg-rs status on node ms1 change to R_FM_FAULTED
Sep 26 12:30:26 ms1 Cluster.RGM.rgmd: [ID 443746 daemon.notice] resource msg-rs state on node ms1 change to R_STOPPING
S

I got the mistake in adding /etc/hosts. I pasted the area here for any person who can notice if they encountered same problem or same mistake.
it should be following format:
192.168.0.250 mail.test.com mail msg-lcreate logical hostname as follow:
clrslh create -g msg-rg msg-lNotice qfe0:1
# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
eri0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.0.240 netmask ffffff00 broadcast 192.168.0.255
groupname sc_ipmp0
ether 0:3:ba:29:8a:ac
eri0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 >index 2
inet 192.168.0.242 netmask ffffff00 broadcast 192.168.0.255
qfe0: flags=9040842<BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 192.168.0.243 netmask ffffff00 broadcast 192.168.0.255
groupname sc_ipmp0
ether 0:3:ba:22:d4:36
qfe0:1: flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500 index 3
inet 192.168.0.250 netmask ffffff00 broadcast 192.168.0.255
qfe2: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 5
inet 172.16.0.129 netmask ffffff80 broadcast 172.16.0.255
ether 0:3:ba:22:d4:38
qfe3: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 4
inet 172.16.1.1 netmask ffffff80 broadcast 172.16.1.127
ether 0:3:ba:22:d4:39
clprivnet0: flags=1009843<UP,BROADCAST,RUNNING,MULTICAST,MULTI_BCAST,PRIVATE,IPv4> mtu 1500 >index 6
inet 172.16.4.1 netmask fffffe00 broadcast 172.16.5.255
ether 0:0:0:0:0:1Now I am able to plumb logical hostname ip. messaging resource group is able to swing over between nodes and resource group is able to go online (before creating messaging server resource (msg-rs).
after creating messaging server resource, use following command to start message resource group:
ms1@root #clrg online -eM msg-rgI used the following command to create message resource (msg-rs)
clrs create -g msg-rg -t SUNW.ims -x IMS_serverroot=/opt/sun/comms/messaging64 -y >Resource_dependencies=msg-l,msg-hasp-rs msg-rsBut still having problem in starting resource group after adding msg-rs
Please advise where I went wrong..
Thanks.

Can I create different Coherence nodes in the same cluster with defferent?

Can I create different Coherence nodes in the same cluster with defferent cache-config.xml file ?
Can a cache be distributed in these deffirent nodes?

Yes. You can create different Coherence nodes in the same cluster with defferent cache-config.xml files as long as you use the same tangosol-coherence.xml file and the same tangosol-coherence-override.xml file. But you cannot store the cache data in the different nodes (started with different cache-config file). In other word, a node only create caches in their own's modes which are started with the same cache-config.xml file.
See the following demo:
I start a cache server using the cache config file examples-cache-server.xml. Then I start a storage-disabled cache console (cache client) using the cache config file coherence-cache-config.xml. Both of them using the same tangosol-coherence.xml file and the same tangosol-coherence-override.xml file.
The cache server uses a cache service PartitionedPofCache. But the client side is using the Distributedcache service. The cluster address is same 224.3.5.2.
The cluster name is also samme. They know each other.
D:\coherence\lib>D:\examples\java\bin\run-cache-server.cmd
D:\coherence\lib>D:\examples\java\bin\run-cache-server.cmd
The system cannot find the file D:\coherence.
The system cannot find the file C:\Oracle\Middleware\jdk160_11.
2009-12-22 12:09:31.400/4.987 Oracle Coherence 3.5.2/463 <Info> (thread=main, member=n/a): Loaded operational configurat
ion from resource "jar:file:/D:/coherence/lib/coherence.jar!/tangosol-coherence.xml"
2009-12-22 12:09:31.450/5.037 Oracle Coherence 3.5.2/463 <Info> (thread=main, member=n/a): Loaded operational overrides
from resource "jar:file:/D:/coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2009-12-22 12:09:31.470/5.057 Oracle Coherence 3.5.2/463 <D5> (thread=main, member=n/a): Optional configuration override
"/tangosol-coherence-override.xml" is not specified
2009-12-22 12:09:31.540/5.127 Oracle Coherence 3.5.2/463 <D5> (thread=main, member=n/a): Optional configuration override
"/custom-mbeans.xml" is not specified
Oracle Coherence Version 3.5.2/463
Grid Edition: Development mode
Copyright (c) 2000, 2009, Oracle and/or its affiliates. All rights reserved.
2009-12-22 12:09:33.864/7.451 Oracle Coherence GE 3.5.2/463 <Info> (thread=main, member=n/a): Loaded cache configuration
from "file:/D:/examples/java/resource/config/examples-cache-config.xml"
2009-12-22 12:09:39.983/13.570 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=n/a): Service Cluster joined t
he cluster with senior service member n/a
2009-12-22 12:09:43.187/16.774 Oracle Coherence GE 3.5.2/463 <Info> (thread=Cluster, member=n/a): Created a new cluster
"cluster:0xD3FB" with Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Locatio
n=process:144, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=1, SocketCount=1) UID=0xC0A8085000
000125B75D888C60501F98
2009-12-22 12:09:43.508/17.095 Oracle Coherence GE 3.5.2/463 <D5> (thread=Invocation:Management, member=1): Service Mana
gement joined the cluster with senior service member 1
2009-12-22 12:09:46.582/20.169 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache:PartitionedPofCache, member=1
): Service PartitionedPofCache joined the cluster with senior service member 1
2009-12-22 12:09:46.672/20.259 Oracle Coherence GE 3.5.2/463 <Info> (thread=DistributedCache:PartitionedPofCache, member
=1): Loading POF configuration from resource "file:/D:/examples/java/resource/config/examples-pof-config.xml"
2009-12-22 12:09:46.702/20.289 Oracle Coherence GE 3.5.2/463 <Info> (thread=DistributedCache:PartitionedPofCache, member
=1): Loading POF configuration from resource "jar:file:/D:/coherence/lib/coherence.jar!/coherence-pof-config.xml"
2009-12-22 12:09:47.734/21.321 Oracle Coherence GE 3.5.2/463 <Info> (thread=main, member=1): Started DefaultCacheServer.
SafeCluster: Name=cluster:0xD3FB
Group{Address=224.3.5.2, Port=35463, TTL=4}
MasterMemberSet
ThisMember=Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=process
:144, Role=CoherenceServer)
OldestMember=Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=proce
ss:144, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=1, BitSetCount=2
    Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=process:144, Rol
e=CoherenceServer)
RecycleMillis=120000
RecycleSet=MemberSet(Size=0, BitSetCount=0
Services
TcpRing{TcpSocketAccepter{State=STATE_OPEN, ServerSocket=192.168.8.80:8088}, Connections=[]}
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.5, OldestMemberId=1}
InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
DistributedCache{Name=PartitionedPofCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCo
unt=1, AssignedPartitions=257, BackupPartitions=0}
2009-12-22 12:12:29.737/183.324 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=2, Timestamp=20
09-12-22 12:12:29.541, Address=192.168.8.80:8089, MachineId=24656, Location=process:1188, Role=CoherenceConsole) joined
Cluster with senior member 1
2009-12-22 12:12:30.498/184.085 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 2 joined Service M
anagement with senior member 1
2009-12-22 12:12:31.860/185.447 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): TcpRing: connecting to me
mber 2 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/192.168.8.80,port=8089,localport=2463]}
2009-12-22 12:12:51.338/204.925 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 2 joined Service D
istributedCache with senior member 2The following command starts a cache client.
D:\coherence\bin>coherence.cmd
D:\coherence\bin>coherence.cmd
** Starting storage disabled console **
java version "1.6.0_11"
Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) Server VM (build 11.0-b16, mixed mode)
2009-12-22 12:12:21.054/3.425 Oracle Coherence 3.5.2/463 <Info> (thread=main, member=n/a): Loaded operational configurat
ion from resource "jar:file:/D:/coherence/lib/coherence.jar!/tangosol-coherence.xml"
2009-12-22 12:12:21.355/3.726 Oracle Coherence 3.5.2/463 <Info> (thread=main, member=n/a): Loaded operational overrides
from resource "jar:file:/D:/coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2009-12-22 12:12:21.365/3.736 Oracle Coherence 3.5.2/463 <D5> (thread=main, member=n/a): Optional configuration override
"/tangosol-coherence-override.xml" is not specified
2009-12-22 12:12:21.415/3.786 Oracle Coherence 3.5.2/463 <D5> (thread=main, member=n/a): Optional configuration override
"/custom-mbeans.xml" is not specified
Oracle Coherence Version 3.5.2/463
Grid Edition: Development mode
Copyright (c) 2000, 2009, Oracle and/or its affiliates. All rights reserved.
2009-12-22 12:12:29.316/11.687 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=n/a): Service Cluster joined t
he cluster with senior service member n/a
2009-12-22 12:12:29.356/11.727 Oracle Coherence GE 3.5.2/463 <Info> (thread=Cluster, member=n/a): Failed to satisfy the
variance: allowed=16, actual=20
2009-12-22 12:12:29.356/11.727 Oracle Coherence GE 3.5.2/463 <Info> (thread=Cluster, member=n/a): Increasing allowable v
ariance to 17
2009-12-22 12:12:29.807/12.178 Oracle Coherence GE 3.5.2/463 <Info> (thread=Cluster, member=n/a): This Member(Id=2, Time
stamp=2009-12-22 12:12:29.541, Address=192.168.8.80:8089, MachineId=24656, Location=process:1188, Role=CoherenceConsole,
Edition=Grid Edition, Mode=Development, CpuCount=1, SocketCount=1) joined cluster "cluster:0xD3FB" with senior Member(I
d=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=process:144, Role=CoherenceS
erver, Edition=Grid Edition, Mode=Development, CpuCount=1, SocketCount=1)
2009-12-22 12:12:29.977/12.348 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=n/a): Member 1 joined Service
Management with senior member 1
2009-12-22 12:12:29.977/12.348 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=n/a): Member 1 joined Service
PartitionedPofCache with senior member 1
2009-12-22 12:12:30.578/12.949 Oracle Coherence GE 3.5.2/463 <D5> (thread=Invocation:Management, member=2): Service Mana
gement joined the cluster with senior service member 1
SafeCluster: Name=cluster:0xD3FB
Group{Address=224.3.5.2, Port=35463, TTL=4}
MasterMemberSet
ThisMember=Member(Id=2, Timestamp=2009-12-22 12:12:29.541, Address=192.168.8.80:8089, MachineId=24656, Location=proces
s:1188, Role=CoherenceConsole)
OldestMember=Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=proce
ss:144, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=2, BitSetCount=2
    Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=process:144, Rol
e=CoherenceServer)
    Member(Id=2, Timestamp=2009-12-22 12:12:29.541, Address=192.168.8.80:8089, MachineId=24656, Location=process:1188, R
ole=CoherenceConsole)
RecycleMillis=120000
RecycleSet=MemberSet(Size=0, BitSetCount=0
Services
TcpRing{TcpSocketAccepter{State=STATE_OPEN, ServerSocket=192.168.8.80:8089}, Connections=[]}
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.5, OldestMemberId=1}
InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
Map (?):
2009-12-22 12:12:49.505/31.906 Oracle Coherence GE 3.5.2/463 <Info> (thread=main, member=2): Loaded cache configuration
from "jar:file:/D:/coherence/lib/coherence.jar!/coherence-cache-config.xml"
2009-12-22 12:12:51.358/33.729 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache, member=2): Service Distribut
edCache joined the cluster with senior service member 2
<distributed-scheme>

<scheme-name>example-distributed</scheme-name>
<service-name>DistributedCache</service-name>
<backing-map-scheme>
    <local-scheme>
      <scheme-ref>example-binary-backing-map</scheme-ref>
    </local-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>But when I try to store data into cache from the client side, it report error message: it's staorage-disabled. It shows that this cache console cannot store the data in the existing cache server because then using different cache config files.
Map (ca3): cache ca2
<distributed-scheme>

<scheme-name>example-distributed</scheme-name>
<service-name>DistributedCache</service-name>
<backing-map-scheme>
    <local-scheme>
      <scheme-ref>example-binary-backing-map</scheme-ref>
    </local-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
Map (ca2): put 1 one
2009-12-22 14:00:04.999/6467.370 Oracle Coherence GE 3.5.2/463 <Error> (thread=main, member=2):
java.lang.RuntimeException: Storage is not configured
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.onMissing
Storage(DistributedCache.CDB:9)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.ensureReq
uestTarget(DistributedCache.CDB:34)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.put(Distr
ibutedCache.CDB:22)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.put(Distr
ibutedCache.CDB:1)
        at com.tangosol.util.ConverterCollections$ConverterMap.put(ConverterCollections.java:1541)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$ViewMap.put(Distrib
utedCache.CDB:1)
        at com.tangosol.coherence.component.util.SafeNamedCache.put(SafeNamedCache.CDB:1)
        at com.tangosol.coherence.component.application.console.Coherence.processCommand(Coherence.CDB:581)
        at com.tangosol.coherence.component.application.console.Coherence.run(Coherence.CDB:39)
        at com.tangosol.coherence.component.application.console.Coherence.main(Coherence.CDB:3)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.tangosol.net.CacheFactory.main(CacheFactory.java:1400)

Cannot start a WSFC node with -ForceQuorum in a cluster that's lost quorum

Hi,
I've got a really simple setup: a two node healthy cluster constisting of SRV1 and SRV2. Current Vote is 1 for SRV1 and 0 for SRV2. To simulate a lost node (and in this case cluster losing quorum) I remove SRV1 from the network. Failover Cluster Manager
(FCM) on SRV1 pretty instantly reports the status of the nodes as:
SRV1 - UP
SRV2 - DOWN
Fine. On SRV2 however, nothing happens in FCM for some time. After about a minute, FCM loses contact with the cluster. When I try to reconnect FCM to the local node (SRV2), I get the following error:
Node 'SRV2' is in the process of being started. The remote server has been paused or is in the process of being started.
Waiting does not help - the problem persists. I then resort to PowerShell and "Start-ClusterNode -ForceQuorum". It responds with State=Joining. But the node is never started. Cannot connect to it in FCM. And any other PowerShell command (e.g. Get-ClusterNode)
returns "The remote server has been paused or is in the process of being started".
What am I doing wrong? How can I manually force a node to start in a cluster that's lost quorum?
Kindly,
Fredrik

Hi Fredrik,
Could you clarify how you “remove SRV1” did you use the FCM or unplug-in the network? If you use FCM the cluster resource will move two SRV2 automatically and you needn’t to
force the SRV 2 up, but why you say the cluster resource is:
SRV1 - UP
SRV2 – DOWN
Does the SRV1 has “removed” right? if it is mistype and the scenario is
SRV1 – DOWN, it must your cluster may have some incorrect configuration, please run the cluster validation first then post the warning and error section. With two node witness we need to use Node and Disk Majority quorum mode please confirm
you have choose the correct witness mode.
The related KB:
Appendix B: Additional Information About Quorum Modes
https://technet.microsoft.com/en-us/library/cc770830(v=ws.10).aspx
Overview and Requirements for a Two-Node Failover Cluster
https://technet.microsoft.com/en-us/library/cc772544(v=ws.10).aspx
I’m glad to be of help to you!
Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Support, contact [email protected]

SQL Cluster services is not start in secound node.

SQL Cluster services is not start in secound node.

Start or Stop the Cluster Service on a Cluster Node
https://technet.microsoft.com/en-us/library/cc771742.aspx
Ahsan Kabir Please remember to click Mark as Answer and Vote as Helpful on posts that help you. This can be beneficial to other community members reading the thread. http://www.aktechforum.blogspot.com/

Can't start service on one of 4 nodes

Dear All,
We have a RAC system with 4 nodes. One of the services is supposed to run on all the 4 nodes. However, we see it running on 2 nodes only. When trying to bring it up on one of the nodes, we got the following error:
srvctl start service -d ABC -s ABC_S -i ABC1
PRKP-1061 : Can not find a service member to start for service ABC_S.
Any idea?
Edited by: user10454776 on Jan 23, 2009 7:08 AM

Hello,
Are you using spfile or pfile for these 2 nodes or your RAC? Verify init.ora file on both the nodes
and see and add service names to the existing list
init.ora
*.service_names= ....
what is the output of following command?
$> crs_stat -t
You can add services to this nodes similar to other nodes using srvctl, make sure no white space between instances and configure it fit your needs
srvctl add service -d abc -s abc_s -r abc1,abc2,abc3 -a abc4
or
srvctl add service -d ABC -s ABC_S -r ABC1,ABC2,ABC3 -a ABC4Regards

Can we start Node server as a windows service?

Hi,
I am starting admin server, managed server as windows service. I've also started node manager as windows service on a node machine. But, now if i want to start the node server, I am starting it from windows console. Instead of this, can i start the node server as windows service? If yes, please give the details.
Thanks

When you also want the WebLogic Server as a service - Re: Installing Weblogic Admin Server as service - service won't start

Starting cluster ocfs2: Failed,Checking cluster ocfs2 is offline

Checking cluster ocfs2 is offline. And when I online the ocfs2 ,display the following error message,who can tell me what's wrong with the ocfs2? thanks!!!!
# /etc/init.d/o2cb status
Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking cluster ocfs2: Offline
# /etc/init.d/o2cb force-reload
Unmounting ocfs2_dlmfs filesystem: OK
Unloading module "ocfs2_dlmfs": OK
Unmounting configfs filesystem: OK
Unloading module "configfs": OK
Loading module "configfs": OK
Mounting configfs filesystem at /config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting cluster ocfs2: Failed
Cluster ocfs2 created
o2cb_ctl: Configuration error discovered while populating cluster ocfs2. None of its nodes were considered local. A node is considered local when its node name in the configuration maches this machine's host name.
Stopping cluster ocfs2: OK
# /etc/init.d/o2cb online ocfs2
Starting cluster ocfs2: Failed
Cluster ocfs2 created
o2cb_ctl: Configuration error discovered while populating cluster ocfs2. None of its nodes were considered local. A node is considered local when its node name in the configuration maches this machine's host name.
Stopping cluster ocfs2: OK
ocfs2_hb_ctl: Unable to access cluster service while starting heartbeat
mount.ocfs2: Error when attempting to run /sbin/ocfs2_hb_ctl: "Operation not permitted"
: Could not mount /dev/sdb1
o2cb_ctl: Unable to access cluster service while creating node

http://unirac.in/qtoa/122/o2cb_ctl-unable-access-cluster-service-while-creating-node
TODO:
- Quit ocfs2console
- Stop the service
- Remove file /etc/ocfs2/cluster.conf
- Restart ocfs2console
- Configure the nodes again
[root@rac1 ~]# /etc/init.d/ocfs2 stop
Stopping Oracle Cluster File System (OCFS2) [ OK ]
[root@rac1 ~]# /etc/init.d/o2cb offline ocfs2
[root@rac1 ~]# /etc/init.d/o2cb unload
Unmounting ocfs2_dlmfs filesystem: OK
Unloading module "ocfs2_dlmfs": OK
Unmounting configfs filesystem: OK
Unloading module "configfs": OK
[root@rac1 ~]# rm -f /etc/ocfs2/cluster.conf
[root@rac1 ~]# /usr/sbin/ocfs2console &

Checking cluster ocfs2 is offline,Starting cluster ocfs2: Failed

Checking cluster ocfs2 is offline. And when I online the ocfs2 ,display the following error message,who can tell me what's wrong with the ocfs2? thanks!!!!
# /etc/init.d/o2cb status
Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking cluster ocfs2: Offline
# /etc/init.d/o2cb force-reload
Unmounting ocfs2_dlmfs filesystem: OK
Unloading module "ocfs2_dlmfs": OK
Unmounting configfs filesystem: OK
Unloading module "configfs": OK
Loading module "configfs": OK
Mounting configfs filesystem at /config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting cluster ocfs2: Failed
Cluster ocfs2 created
o2cb_ctl: Configuration error discovered while populating cluster ocfs2. None of its nodes were considered local. A node is considered local when its node name in the configuration maches this machine's host name.
Stopping cluster ocfs2: OK
# /etc/init.d/o2cb online ocfs2
Starting cluster ocfs2: Failed
Cluster ocfs2 created
o2cb_ctl: Configuration error discovered while populating cluster ocfs2. None of its nodes were considered local. A node is considered local when its node name in the configuration maches this machine's host name.
Stopping cluster ocfs2: OK
ocfs2_hb_ctl: Unable to access cluster service while starting heartbeat
mount.ocfs2: Error when attempting to run /sbin/ocfs2_hb_ctl: "Operation not permitted"
: Could not mount /dev/sdb1
o2cb_ctl: Unable to access cluster service while creating node

Hi
Getting exactly the same errors when trying to implement OCFS2 on a VMWARE solution
[root@raclinux1 init.d]# ./o2cb status
Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking cluster ocfs2: Offline
[root@raclinux1 init.d]# ./o2cb online ocfs2
Starting cluster ocfs2: Failed
Cluster ocfs2 created
o2cb_ctl: Configuration error discovered while populating cluster ocfs2. None of its nodes were considered local. A node is considered local when its node name in the configuration maches this machine's host name.
Stopping cluster ocfs2: OK
And from ocfs2console we get the following errors when trying to add our node to the configuration
o2cb_ctl: Unable to access cluster service while creating node
Could not add node raclinux1
Any one got it to work on VMWARE?
PMcM

Can't start J2EE Engine

Hi.
I can't start my local J2EE Engine after a workplace SP11 installation.
OS: Windows 2000 Pro.
JRE: 1.4.2_07
Below you see the developer trace that is displayed when I try to start the J2EE Engine.
// -Fredrik
trc file: "C:\usr\sap\J2E\JC00\work\dev_jcontrol", trc level: 1, release: "640"
node name   : jcontrol
pid         : 1868
system name : J2E
system nr. : 00
started at : Wed Apr 27 10:43:48 2005
arguments   :
    arg[00] : C:\usr\sap\J2E\JC00\j2ee\os_libs\jcontrol.exe
    arg[01] : pf=C:\usr\sap\J2E\SYS\profile\J2E_JC00_wkx4590
[Thr 1844] Wed Apr 27 10:43:48 2005
[Thr 1844] *** ERROR => Invalid property value [box.number/J2EJC00wkx4590] [jstartxx.c   808]
[Thr 1844] *** ERROR => Invalid property value [system.id/0] [jstartxx.c   808]
JStartupReadInstanceProperties: read instance properties [C:\usr\sap\J2E\JC00\j2ee\cluster\instance.properties;C:\usr\sap\J2E\JC00\SDM\program\config\sdm_jstartup.properties]
-> ms host    : wkx4590
-> ms port    : 3601
-> OS libs    : C:\usr\sap\J2E\JC00\j2ee\os_libs
-> Admin URL :
-> run mode   : NORMAL
-> run action : NONE
-> enabled    : yes
Used property files
-> files [00] : C:\usr\sap\J2E\JC00\j2ee\cluster\instance.properties
-> files [01] : C:\usr\sap\J2E\JC00\SDM\program\config\sdm_jstartup.properties
Instance properties
-> ms host    : wkx4590
-> ms port    : 3601
-> os libs    : C:\usr\sap\J2E\JC00\j2ee\os_libs
-> admin URL :
-> run mode   : NORMAL
-> run action : NONE
-> enabled    : yes
Bootstrap nodes
-> [00] bootstrap            : C:\usr\sap\J2E\JC00\j2ee\cluster\instance.properties
-> [01] bootstrap_ID4431400 : C:\usr\sap\J2E\JC00\j2ee\cluster\instance.properties
-> [02] bootstrap_ID4431450 : C:\usr\sap\J2E\JC00\j2ee\cluster\instance.properties
Worker nodes
-> [00] ID4431400            : C:\usr\sap\J2E\JC00\j2ee\cluster\instance.properties
-> [01] ID4431450            : C:\usr\sap\J2E\JC00\j2ee\cluster\instance.properties
-> [02] sdm                  : C:\usr\sap\J2E\JC00\SDM\program\config\sdm_jstartup.properties
[Thr 1844] *** ERROR => can't adjust privileges (rc=1300) [jstartnt.c   1058]
[Thr 1844] *** ERROR => can't set privilege (rc=-1) [jstartnt.c   963]
[Thr 1844] JControlCloseProgram: started (exitcode = -1)
[Thr 1844] JControlCloseProgram: good bye... (exitcode=-1)

Hi Fred,
have a look at this discussion that solved exactly the same problem as yours: Re: Web AS J2E instance 0 Installation problem.
The solution most probably will be that you change the value of the jstartup/trimming_properties property in your instance profile from "standard" to "off".
Good luck!
Message was edited by: Ivaylo Ivanov

Can't start ASM,help me

Help I can't start ASM on rhel4(32bit) + oracle 10G 10.2 RAC
Wed Jun 28 17:10:40 2006
Error: KGXGN polling error (15)
Wed Jun 28 17:10:40 2006
Errors in file /oracle/product/10.2.0/db/admin/+ASM/bdump/+asm1_lmon_24375.trc:
ORA-29702: error occurred in Cluster Group Service operation
LMON: terminating instance due to error 29702
Wed Jun 28 17:10:40 2006
Dump system state for local instance only
System State dumped to trace file /oracle/product/10.2.0/db/admin/+ASM/bdump/+asm1_diag_24363.trc
Wed Jun 28 17:10:40 2006
Trace dumping is performing id=[cdmp_20060628171040]
Wed Jun 28 17:10:41 2006
Shutting down instance (abort)
License high water mark = 2
Wed Jun 28 17:10:45 2006
Instance terminated by LMON, pid = 24375
Wed Jun 28 17:10:46 2006
Instance terminated by USER, pid = 29540
Wed Jun 28 17:10:47 2006
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
WARNING: No cluster interconnect has been specified. Depending on
the communication driver configured Oracle cluster traffic
may be directed to the public interface of this machine.
Oracle recommends that RAC clustered databases be configured
with a private interconnect for enhanced security and
performance.
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_1 parameter default value as /oracle/product/10.2.0/db/dbs/arch
Autotune of undo retention is turned off.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
large_pool_size = 12582912
spfile = /dev/raw/raw3
instance_type = asm
cluster_database = TRUE
instance_number = 1
remote_login_passwordfile= EXCLUSIVE
background_dump_dest = /oracle/product/10.2.0/db/admin/+ASM/bdump
user_dump_dest = /oracle/product/10.2.0/db/admin/+ASM/udump
core_dump_dest = /oracle/product/10.2.0/db/admin/+ASM/cdump
asm_diskgroups =
Cluster communication is configured to use the following interface(s) for this instance
203.144.222.81
Wed Jun 28 17:10:48 2006
cluster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=29735
DIAG started with pid=3, OS id=29737
PSP0 started with pid=4, OS id=29739
LMON started with pid=5, OS id=29741
LMD0 started with pid=6, OS id=29743
LMS0 started with pid=7, OS id=29745
MMAN started with pid=8, OS id=29755
DBW0 started with pid=9, OS id=29757
LGWR started with pid=10, OS id=29759
CKPT started with pid=11, OS id=29761
SMON started with pid=12, OS id=29763
RBAL started with pid=13, OS id=29765
GMON started with pid=14, OS id=29767
Wed Jun 28 17:10:48 2006
lmon registered with NM - instance id 1 (internal mem no 0)
Wed Jun 28 17:12:59 2006
Error: KGXGN polling error (15)
Wed Jun 28 17:12:59 2006
Errors in file /oracle/product/10.2.0/db/admin/+ASM/bdump/+asm1_lmon_29741.trc:
ORA-29702: error occurred in Cluster Group Service operation
LMON: terminating instance due to error 29702
Wed Jun 28 17:13:01 2006
Shutting down instance (abort)
License high water mark = 1
Wed Jun 28 17:13:03 2006
Instance terminated by LMON, pid = 29741
and trace file
/oracle/product/10.2.0/db/admin/+ASM/bdump/+asm1_lmon_29741.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
ORACLE_HOME = /oracle/product/10.2.0/db
System name: Linux
Node name: aadb01
Release: 2.6.9-34.ELsmp
Version: #1 SMP Fri Feb 24 16:54:53 EST 2006
Machine: i686
Instance name: +ASM1
Redo thread mounted by this instance: 0 <none>
Oracle process number: 5
Unix process pid: 29741, image: oracle@aadb01 (LMON)
*** SERVICE NAME:() 2006-06-28 17:10:48.174
*** SESSION ID:(46.1) 2006-06-28 17:10:48.174
GES resources 1890 pool 4
GES enqueues 2510
GES IPC: Receivers 2 Senders 2
GES IPC: Buffers Receive 1000 Send (i:830 b:830) Reserve 81
GES IPC: Msg Size Regular 388 Batch 4096
Batching factor: enqueue replay 99, ack 110
Batching factor: cache replay 71 size per lock 56
kjxggin: receive buffer size = 32768
*** 2006-06-28 17:10:48.509
kjxgmrcfg: Reconfiguration started, reason 1
kjxgmcs: Setting state to 0 0.
*** 2006-06-28 17:10:48.509
Name Service frozen
kjxgmcs: Setting state to 0 1.
kjfcpiora: published my fusion master weight 5622
kjfcpiora: publish my flogb 9
*** 2006-06-28 17:11:53.883
=====================================================
kjxgmpoll: CGS state (0 1) start 0x635d41bb cur 0x672b6877 diff 0x3ce26bc
=====================================================
Group name: +ASM
Member id: 0
Cached KGXGN event: 0
Group State:
State: 0 1
Reconfig started start-tm 0x635d4173 tmout period 0x7270e00
Commited Map: 0 1
New Map: 0 1
KGXGN Map: 0 1
Master node: 0
Memcnt 0 Rcvcnt 0
Substate Proposal: false
Inc Proposal:
incarn 1 memcnt 2 master 0
proposal true matched false
map: 0 1
Master Inc State:
incarn 1 memcnt 2 agrees 1 flag 0xa
wmap: 1
nmap: 0 1
ubmap:
Substate Handler Execution State
substate 0 status done
substate 1 status done
substate 2 status done
substate 3 status done
substate 4 status done
substate 5 status done
substate 6 status done
*** 2006-06-28 17:12:59.243
=====================================================
kjxgmpoll: CGS state (0 1) start 0x635d41bb cur 0x6af9582a diff 0x79c166f
=====================================================
Group name: +ASM
Member id: 0
Cached KGXGN event: 0
Group State:
State: 0 1
Reconfig started start-tm 0x635d4173 tmout period 0x7270e00
Commited Map: 0 1
New Map: 0 1
KGXGN Map: 0 1
Master node: 0
Memcnt 0 Rcvcnt 0
Substate Proposal: false
Inc Proposal:
incarn 1 memcnt 2 master 0
proposal true matched false
map: 0 1
Master Inc State:
incarn 1 memcnt 2 agrees 1 flag 0xa
wmap: 1
nmap: 0 1
ubmap:
Substate Handler Execution State
substate 0 status done
substate 1 status done
substate 2 status done
substate 3 status done
substate 4 status done
substate 5 status done
substate 6 status done
kjxgmpoll: terminate the CGS reconfig.
Error: KGXGN polling error (15)
error 29702 detected in background process
ORA-29702: error occurred in Cluster Group Service operation
ksuitm: waiting for [5] seconds before killing DIAG

Modify the classpath as follows.
set JAVA_HOME=C:\JDK1.3.1
set ANT_HOME=C:\jakarta-tomcat
set TOMCAT_HOME=C:\jakarta-tomcat
set CLASSPATH=%CLASSPATH%;C:\jdk1.3.1\lib\tools.jar;JAVA_HOME\lib;
Modify the Path as follows
set PATH = %PATH%;%JAVA_HOME%\bin;%TOMCAT_HOME%\bin

Can't start manager server on other machine from admin console

hi, m fredzh
i have weblogic 10.3.2 where i created a cluster named cluster1 on production mode
the machines and servers list as follow:
cluster1 192.168.1.100
--machine1
--server1 192.168.1.100
--machine2
-server2 192.168.1.101
i create nodemanager on machine2, and set the type as ssl, the listener address 192.168.1.101,
then login on 192.168.1.101, and run wlst.sh , nmEnroll and then start startNodemanaged.sh ..
start server2 on the admin console(on 192.168.1.100), failed and the log as follow:
can anbody help me, thank you..
==============================================================================
<Critical> <WebLogicServer> <localhost.localdomain> <managerServer2> <main> <<WLS Kernel>> <> <> <1286788326928> <BEA-000386> <Server subsystem failed. Reason: java.lang.AssertionError: java.lang.reflect.InvocationTargetException
java.lang.AssertionError: java.lang.reflect.InvocationTargetException
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy._invokeServiceMethod(DescriptorManager.java:175)
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy.decrypt(DescriptorManager.java:192)
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl.decrypt(DescriptorManager.java:114)
at weblogic.descriptor.internal.AbstractDescriptorBean._decrypt(AbstractDescriptorBean.java:991)
at weblogic.management.configuration.SecurityConfigurationMBeanImpl.getCredential(SecurityConfigurationMBeanImpl.java:709)
at weblogic.security.internal.ServerPrincipalValidatorImpl.getSecret(ServerPrincipalValidatorImpl.java:88)
at weblogic.security.internal.ServerPrincipalValidatorImpl.sign(ServerPrincipalValidatorImpl.java:67)
at weblogic.security.service.PrivilegedActions$SignPrincipalAction.run(PrivilegedActions.java:62)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.security.service.SecurityServiceManager.createServerID(SecurityServiceManager.java:1098)
at weblogic.security.service.SecurityServiceManager.getServerID(SecurityServiceManager.java:1111)
at weblogic.security.service.SecurityServiceManager.sendASToWire(SecurityServiceManager.java:602)
at weblogic.server.channels.ChannelService.resetQOS(ChannelService.java:284)
at weblogic.server.channels.ChannelService.start(ChannelService.java:250)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused By: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy._invokeServiceMethod(DescriptorManager.java:173)
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy.decrypt(DescriptorManager.java:192)
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl.decrypt(DescriptorManager.java:114)
at weblogic.descriptor.internal.AbstractDescriptorBean._decrypt(AbstractDescriptorBean.java:991)
at weblogic.management.configuration.SecurityConfigurationMBeanImpl.getCredential(SecurityConfigurationMBeanImpl.java:709)
at weblogic.security.internal.ServerPrincipalValidatorImpl.getSecret(ServerPrincipalValidatorImpl.java:88)
at weblogic.security.internal.ServerPrincipalValidatorImpl.sign(ServerPrincipalValidatorImpl.java:67)
at weblogic.security.service.PrivilegedActions$SignPrincipalAction.run(PrivilegedActions.java:62)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.security.service.SecurityServiceManager.createServerID(SecurityServiceManager.java:1098)
at weblogic.security.service.SecurityServiceManager.getServerID(SecurityServiceManager.java:1111)
at weblogic.security.service.SecurityServiceManager.sendASToWire(SecurityServiceManager.java:602)
at weblogic.server.channels.ChannelService.resetQOS(ChannelService.java:284)
at weblogic.server.channels.ChannelService.start(ChannelService.java:250)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused By: weblogic.security.internal.encryption.EncryptionServiceException: com.rsa.jsafe.JSAFE_PaddingException: Could not perform unpadding: invalid pad byte.
at weblogic.security.internal.encryption.JSafeEncryptionServiceImpl.decryptBytes(JSafeEncryptionServiceImpl.java:125)
at weblogic.security.internal.encryption.JSafeEncryptionServiceImpl.decryptString(JSafeEncryptionServiceImpl.java:173)
at weblogic.security.internal.encryption.ClearOrEncryptedService.decrypt(ClearOrEncryptedService.java:96)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy._invokeServiceMethod(DescriptorManager.java:173)
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy.decrypt(DescriptorManager.java:192)
at weblogic.descriptor.DescriptorManager$SecurityServiceImpl.decrypt(DescriptorManager.java:114)
at weblogic.descriptor.internal.AbstractDescriptorBean._decrypt(AbstractDescriptorBean.java:991)
at weblogic.management.configuration.SecurityConfigurationMBeanImpl.getCredential(SecurityConfigurationMBeanImpl.java:709)
at weblogic.security.internal.ServerPrincipalValidatorImpl.getSecret(ServerPrincipalValidatorImpl.java:88)
at weblogic.security.internal.ServerPrincipalValidatorImpl.sign(ServerPrincipalValidatorImpl.java:67)
at weblogic.security.service.PrivilegedActions$SignPrincipalAction.run(PrivilegedActions.java:62)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.security.service.SecurityServiceManager.createServerID(SecurityServiceManager.java:1098)
at weblogic.security.service.SecurityServiceManager.getServerID(SecurityServiceManager.java:1111)
at weblogic.security.service.SecurityServiceManager.sendASToWire(SecurityServiceManager.java:602)
at weblogic.server.channels.ChannelService.resetQOS(ChannelService.java:284)
at weblogic.server.channels.ChannelService.start(ChannelService.java:250)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Edited by: fredzh on 2010-10-12 下午6:54

hi,Jay
i followed the steps as follow:
1、change prod mode to dev
2、modified the config.xml entrys on the all two servers:
<node-manager-password-encrypted>test</node-manager-password-encrypted>
<credential-encrypted>test</credential-encrypted>
<embedded-ldap>
<name>cluster_domain</name>
<credential-encrypted>test</credential-encrypted>
</embedded-ldap>
3、enrolled the node, then started the two adminservers, started server2's nodemanager..server2 can be started from the cluster admin console.
4、change the dev mode to prod mode on admin console, it failed throw error:
"in production mode, it's not allowed to set a clear text value to the property: CredentialEncrypted of SecurityConfigurationMBean"
i checked the config.xml ,the entrys i had changed were not aotomatically encrypted,may i manually encryp the clear text value? ..follow is the config.xml :
<?xml version='1.0' encoding='UTF-8'?>
<domain xmlns="http://xmlns.oracle.com/weblogic/domain" xmlns:sec="http://xmlns.oracle.com/weblogic/security" xmlns:wls="http://xmlns.oracle.com/weblogic/security/wls" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/security/xacml http://xmlns.oracle.com/weblogic/security/xacml/1.0/xacml.xsd http://xmlns.oracle.com/weblogic/security/providers/passwordvalidator http://xmlns.oracle.com/weblogic/security/providers/passwordvalidator/1.0/passwordvalidator.xsd http://xmlns.oracle.com/weblogic/domain http://xmlns.oracle.com/weblogic/1.0/domain.xsd http://xmlns.oracle.com/weblogic/security http://xmlns.oracle.com/weblogic/1.0/security.xsd http://xmlns.oracle.com/weblogic/security/wls http://xmlns.oracle.com/weblogic/security/wls/1.0/wls.xsd">
<name>cluster_domain</name>
<domain-version>10.3.2.0</domain-version>
<security-configuration>
<name>cluster_domain</name>
<realm>
<sec:authentication-provider xsi:type="wls:default-authenticatorType"></sec:authentication-provider>
<sec:authentication-provider xsi:type="wls:default-identity-asserterType">
<sec:active-type>AuthenticatedUser</sec:active-type>
</sec:authentication-provider>
<sec:role-mapper xmlns:xac="http://xmlns.oracle.com/weblogic/security/xacml" xsi:type="xac:xacml-role-mapperType"></sec:role-mapper>
<sec:authorizer xmlns:xac="http://xmlns.oracle.com/weblogic/security/xacml" xsi:type="xac:xacml-authorizerType"></sec:authorizer>
<sec:adjudicator xsi:type="wls:default-adjudicatorType"></sec:adjudicator>
<sec:credential-mapper xsi:type="wls:default-credential-mapperType"></sec:credential-mapper>
<sec:cert-path-provider xsi:type="wls:web-logic-cert-path-providerType"></sec:cert-path-provider>
<sec:cert-path-builder>WebLogicCertPathProvider</sec:cert-path-builder>
<sec:name>myrealm</sec:name>
<sec:password-validator xmlns:pas="http://xmlns.oracle.com/weblogic/security/providers/passwordvalidator" xsi:type="pas:system-password-validatorType">
<sec:name>SystemPasswordValidator</sec:name>
<pas:min-password-length>8</pas:min-password-length>
<pas:min-numeric-or-special-characters>1</pas:min-numeric-or-special-characters>
</sec:password-validator>
</realm>
<default-realm>myrealm</default-realm>
<anonymous-admin-lookup-enabled>false</anonymous-admin-lookup-enabled>
<credential-encrypted>test</credential-encrypted>
<web-app-files-case-insensitive>false</web-app-files-case-insensitive>
<compatibility-connection-filters-enabled>false</compatibility-connection-filters-enabled>
<node-manager-username>weblogic</node-manager-username>
<node-manager-password-encrypted>test</node-manager-password-encrypted>
<principal-equals-case-insensitive>false</principal-equals-case-insensitive>
<principal-equals-compare-dn-and-guid>false</principal-equals-compare-dn-and-guid>
<downgrade-untrusted-principals>false</downgrade-untrusted-principals>
<enforce-strict-url-pattern>true</enforce-strict-url-pattern>
<cross-domain-security-enabled>false</cross-domain-security-enabled>
</security-configuration>
<jta>
<security-interop-mode>default</security-interop-mode>
</jta>
<server>
<name>AdminServer</name>
<ssl>
<hostname-verifier xsi:nil="true"></hostname-verifier>
<hostname-verification-ignored>true</hostname-verification-ignored>
<export-key-lifespan>500</export-key-lifespan>
<client-certificate-enforced>false</client-certificate-enforced>
<two-way-ssl-enabled>false</two-way-ssl-enabled>
<ssl-rejection-logging-enabled>true</ssl-rejection-logging-enabled>
<inbound-certificate-validation>BuiltinSSLValidationOnly</inbound-certificate-validation>
<outbound-certificate-validation>BuiltinSSLValidationOnly</outbound-certificate-validation>
<allow-unencrypted-null-cipher>false</allow-unencrypted-null-cipher>
<use-server-certs>false</use-server-certs>
</ssl>
<listen-port>7110</listen-port>
<listen-address></listen-address>
</server>
<server>
<name>server2</name>
<machine>machine2</machine>
<listen-port>7110</listen-port>
<listen-address>192.168.1.101</listen-address>
<server-start>
<java-vendor>Sun</java-vendor>
<java-home>/opt/jdk1.6.0_21</java-home>
<bea-home>/root/Oracle/Middleware</bea-home>
<root-directory>/root/Oracle/Middleware/user_projects/domains/cluster_domain</root-directory>
<username>weblogic</username>
<password-encrypted>{AES}xhSI75AxXtXdMrCXRgA2uY5pfcxUGE9X4YXTjfe1XEY=</password-encrypted>
</server-start>
</server>
<embedded-ldap>
<name>cluster_domain</name>
<credential-encrypted>test</credential-encrypted>
</embedded-ldap>
<configuration-version>10.3.2.0</configuration-version>
<machine xsi:type="unix-machineType">
<name>machine2</name>
<node-manager>
<nm-type>SSL</nm-type>
<listen-address>192.168.1.101</listen-address>
<listen-port>5556</listen-port>
<debug-enabled>false</debug-enabled>
</node-manager>
</machine>
<admin-server-name>AdminServer</admin-server-name>
</domain>
Edited by: fredzh on 2010-10-13 下午7:57

Can't start cluster, 2 node 3.3 cluster lost 2 quorum disks

Similar Messages

Maybe you are looking for