Second node of 2 node cluster rebooted on it's own

Hello,
I hope that I am submitting this question correctly. Please let me know if I have done something wrong. I would like to explain what happened in my production environment last night during a maintenance window.
The infrastructure team had discovered that the first node was experiencing some multipath issues to the remote ASM disk, so, in an effort to resolve those issues we took the following actions. I turned off the listener on the first node and shut down the instance. I then verified that all connections to the first node had stopped and that they had moved to the second node (This is the way that our application vendor wanted us to set up RAC - automatic failover). I also stopped ASM, CRS and HAS on the first node for the maintenance.
While the first node was down, something unusual happened with the connection to the ASM disk (both nodes use the same disk for the database) wherein the connection was lost. Once the connection was set back to right, which only took a few minutes, I looked at the alert logs and saw that the instance had started a crash recovery and applied redo from the last logs, which was perfect. But I've been asked this morning to try and discover why the server rebooted. Is there a cluster log that I can reference that might explain why that happened? Is it a failsafe action for an Oracle server (running on CentOS 6) to restart and avoid any issues that might arise from only having one instance up and losing connection to the ASM disk?
We are using Oracle 11.2.0.3.4, which is the recommended version as per our vendor. Please let me know if you require any more information. Please also keep in mind that I am only in my second year as a Database Administrator, working for a small company, so I may not know the answer to every question you ask me right away, but will do my best to answer them. I did graduate from a local college with an Oracle certificate, but I have not yet completed any of the exams to have the Oracle certifications, though.

I should also have mentioned that we are using Oracle Standard Edition and so do not have access to Enterprise Manager.

Similar Messages

Node does not join cluster upon reboot

Hi Guys,
I have two servers [Sun Fire X4170] clustered together using Solaris cluster 3.3 for Oracle Database. They are connected to a shared storage which is Dell Equallogic [iSCSI]. Lately, I have ran into a weird kind of a problem where as both nodes come up fine and join the cluster upon reboot; however, when I reboot one of nodes then any of them does not join cluster and shows following errors:
This is happening on both the nodes [if I reboot only one node at a time]. But if I reboot both the nodes at the same time then they successfully join the cluster and everything runs fine.
Below is the output from one node which I rebooted and it did not join the cluster and puked out following errors. The other node is running fine will all the services.
In order to get out of this situation, I have to reboot both the nodes together.
# dmesg output #
Apr 23 17:37:03 srvhqon11 ixgbe: [ID 611667 kern.info] NOTICE: ixgbe2: link down
Apr 23 17:37:12 srvhqon11 iscsi: [ID 933263 kern.notice] NOTICE: iscsi connection(5) unable to connect to target SENDTARGETS_DISCOVERY
Apr 23 17:37:12 srvhqon11 iscsi: [ID 114404 kern.notice] NOTICE: iscsi discovery failure - SendTargets (010.010.017.104)
Apr 23 17:37:13 srvhqon11 iscsi: [ID 240218 kern.notice] NOTICE: iscsi session(9) iqn.2001-05.com.equallogic:0-8a0906-96cf73708-ef30000005e50a1b-sblprdbk online
Apr 23 17:37:13 srvhqon11 scsi: [ID 583861 kern.info] sd11 at scsi_vhci0: unit-address g6090a0887073cf961b0ae505000030ef: g6090a0887073cf961b0ae505000030ef
Apr 23 17:37:13 srvhqon11 genunix: [ID 936769 kern.info] sd11 is /scsi_vhci/disk@g6090a0887073cf961b0ae505000030ef
Apr 23 17:37:13 srvhqon11 scsi: [ID 243001 kern.info] /scsi_vhci (scsi_vhci0):
Apr 23 17:37:13 srvhqon11 /scsi_vhci/disk@g6090a0887073cf961b0ae505000030ef (sd11): Command failed to complete (3) on path iscsi0/[email protected]:0-8a0906-96cf73708-ef30000005e50a1b-sblprdbk0001,0
Apr 23 17:46:54 srvhqon11 svc.startd[11]: [ID 122153 daemon.warning] svc:/network/iscsi/initiator:default: Method or service exit timed out. Killing contract 41.
Apr 23 17:46:54 srvhqon11 svc.startd[11]: [ID 636263 daemon.warning] svc:/network/iscsi/initiator:default: Method "/lib/svc/method/iscsid start" failed due to signal KILL.
Apr 23 17:46:54 srvhqon11 svc.startd[11]: [ID 748625 daemon.error] network/iscsi/initiator:default failed repeatedly: transitioned to maintenance (see 'svcs -xv' for details)
Apr 24 14:50:16 srvhqon11 svc.startd[11]: [ID 694882 daemon.notice] instance svc:/system/console-login:default exited with status 1
root@srvhqon11 # svcs -xv
svc:/system/cluster/loaddid:default (Oracle Solaris Cluster loaddid)
State: offline since Tue Apr 23 17:46:54 2013
Reason: Start method is running.
See: http://sun.com/msg/SMF-8000-C4
See: /var/svc/log/system-cluster-loaddid:default.log
Impact: 49 dependent services are not running:
svc:/system/cluster/bootcluster:default
svc:/system/cluster/cl_execd:default
svc:/system/cluster/zc_cmd_log_replay:default
svc:/system/cluster/sc_zc_member:default
svc:/system/cluster/sc_rtreg_server:default
svc:/system/cluster/sc_ifconfig_server:default
svc:/system/cluster/initdid:default
svc:/system/cluster/globaldevices:default
svc:/system/cluster/gdevsync:default
svc:/milestone/multi-user:default
svc:/system/boot-config:default
svc:/system/cluster/cl-svc-enable:default
svc:/milestone/multi-user-server:default
svc:/application/autoreg:default
svc:/system/basicreg:default
svc:/system/zones:default
svc:/system/cluster/sc_zones:default
svc:/system/cluster/scprivipd:default
svc:/system/cluster/cl-svc-cluster-milestone:default
svc:/system/cluster/sc_svtag:default
svc:/system/cluster/sckeysync:default
svc:/system/cluster/rpc-fed:default
svc:/system/cluster/rgm-starter:default
svc:/application/management/common-agent-container-1:default
svc:/system/cluster/scsymon-srv:default
svc:/system/cluster/sc_syncsa_server:default
svc:/system/cluster/scslmclean:default
svc:/system/cluster/cznetd:default
svc:/system/cluster/scdpm:default
svc:/system/cluster/rpc-pmf:default
svc:/system/cluster/pnm:default
svc:/system/cluster/sc_pnm_proxy_server:default
svc:/system/cluster/cl-event:default
svc:/system/cluster/cl-eventlog:default
svc:/system/cluster/cl-ccra:default
svc:/system/cluster/ql_upgrade:default
svc:/system/cluster/mountgfs:default
svc:/system/cluster/clusterdata:default
svc:/system/cluster/ql_rgm:default
svc:/system/cluster/scqdm:default
svc:/application/stosreg:default
svc:/application/sthwreg:default
svc:/application/graphical-login/cde-login:default
svc:/application/cde-printinfo:default
svc:/system/cluster/scvxinstall:default
svc:/system/cluster/sc_failfast:default
svc:/system/cluster/clexecd:default
svc:/system/cluster/sc_pmmd:default
svc:/system/cluster/clevent_listenerd:default
svc:/application/print/server:default (LP print server)
State: disabled since Tue Apr 23 17:36:44 2013
Reason: Disabled by an administrator.
See: http://sun.com/msg/SMF-8000-05
See: man -M /usr/share/man -s 1M lpsched
Impact: 2 dependent services are not running:
svc:/application/print/rfc1179:default
svc:/application/print/ipp-listener:default
svc:/network/iscsi/initiator:default (?)
State: maintenance since Tue Apr 23 17:46:54 2013
Reason: Restarting too quickly.
See: http://sun.com/msg/SMF-8000-L5
See: /var/svc/log/network-iscsi-initiator:default.log
Impact: This service is not running.
######## Cluster Status from working node ############
root@srvhqon10 # cluster status
=== Cluster Nodes ===
--- Node Status ---
Node Name Status
srvhqon10 Online
srvhqon11 Offline
=== Cluster Transport Paths ===
Endpoint1 Endpoint2 Status
srvhqon10:igb3 srvhqon11:igb3 faulted
srvhqon10:igb2 srvhqon11:igb2 faulted
=== Cluster Quorum ===
--- Quorum Votes Summary from (latest node reconfiguration) ---
Needed Present Possible
2 2 3
--- Quorum Votes by Node (current status) ---
Node Name Present Possible Status
srvhqon10 1 1 Online
srvhqon11 0 1 Offline
--- Quorum Votes by Device (current status) ---
Device Name Present Possible Status
d2 1 1 Online
=== Cluster Device Groups ===
--- Device Group Status ---
Device Group Name Primary Secondary Status
--- Spare, Inactive, and In Transition Nodes ---
Device Group Name Spare Nodes Inactive Nodes In Transistion Nodes
--- Multi-owner Device Group Status ---
Device Group Name Node Name Status
=== Cluster Resource Groups ===
Group Name Node Name Suspended State
ora-rg srvhqon10 No Online
srvhqon11 No Offline
nfs-rg srvhqon10 No Online
srvhqon11 No Offline
backup-rg srvhqon10 No Online
srvhqon11 No Offline
=== Cluster Resources ===
Resource Name Node Name State Status Message
ora-listener srvhqon10 Online Online
srvhqon11 Offline Offline
ora-server srvhqon10 Online Online
srvhqon11 Offline Offline
ora-stor srvhqon10 Online Online
srvhqon11 Offline Offline
ora-lh srvhqon10 Online Online - LogicalHostname online.
srvhqon11 Offline Offline
nfs-rs srvhqon10 Online Online - Service is online.
srvhqon11 Offline Offline
nfs-stor-rs srvhqon10 Online Online
srvhqon11 Offline Offline
nfs-lh-rs srvhqon10 Online Online - LogicalHostname online.
srvhqon11 Offline Offline
backup-stor srvhqon10 Online Online
srvhqon11 Offline Offline
cluster: (C383355) No response from daemon on node "srvhqon11".
=== Cluster DID Devices ===
Device Instance Node Status
/dev/did/rdsk/d1 srvhqon10 Ok
/dev/did/rdsk/d2 srvhqon10 Ok
srvhqon11 Unknown
/dev/did/rdsk/d3 srvhqon10 Ok
srvhqon11 Unknown
/dev/did/rdsk/d4 srvhqon10 Ok
/dev/did/rdsk/d5 srvhqon10 Fail
srvhqon11 Unknown
/dev/did/rdsk/d6 srvhqon11 Unknown
/dev/did/rdsk/d7 srvhqon11 Unknown
/dev/did/rdsk/d8 srvhqon10 Ok
srvhqon11 Unknown
/dev/did/rdsk/d9 srvhqon10 Ok
srvhqon11 Unknown
=== Zone Clusters ===
--- Zone Cluster Status ---
Name Node Name Zone HostName Status Zone Status
Regards.

check if your global devices are mounted properly
#cat /etc/mnttab | grep -i global
check if proper entries are there on both systems
#cat /etc/vfstab | grep -i global
give output for quoram devices .
#scstat -q
or
#clquorum list -v
also check why your scsi initiator service is going offline unexpectedly
#vi /var/svc/log/network-iscsi-initiator:default.log

Node can not join cluster after RAC HA Testing

Dear forum,
We are performing RAC failover tests according to document "RAC System Test Plan Outline 11gR2, Version 2.0". In testcase #14 - Interconnect network failure (11.2.0.2 an higher), we have disabled private interconnect network of node node1 (OCR Master).
Then - as expected - node node2 was evicted. Now, after enabling private interconnect network on node node1, i want to start CRS again on node2. However, node does not join cluster with messages:
2012-03-15 14:12:35.138: [ CSSD][1113114944]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2012-03-15 14:12:35.371: [ CSSD][1109961024]clssnmvDHBValidateNCopy: node 1, node1, has a disk HB, but no network HB, DHB has rcfg 226493542, wrtcnt, 2301201, LATS 5535614, lastSeqNo 2301198, uniqueness 1331804892, timestamp 1331817153/13040714
2012-03-15 14:12:35.479: [ CSSD][1100884288]clssnmvDHBValidateNCopy: node 1, node1, has a disk HB, but no network HB, DHB has rcfg 226493542, wrtcnt, 2301202, LATS 5535724, lastSeqNo 2301199, uniqueness 1331804892, timestamp 1331817154/13041024
2012-03-15 14:12:35.675: [ CSSD][1080801600]clssnmvDHBValidateNCopy: node 1, node1, has a disk HB, but no network HB, DHB has rcfg 226493542, wrtcnt, 2301203, LATS 5535924, lastSeqNo 2301200, uniqueness 1331804892, timestamp 1331817154/13041364
Rebooting node2 did not help. Node1 which was online all the time (although private interconnect interface was unplugged for a few minutes and then plugged back in). I suppose that if we reboot node2, the problem will disappear. But there should be solution, which keeps availability requirements.
Setup:
2 Nodes (OEL5U7, UEK)
2 Storages
Network bonding via Linux bonding
GI 11.2.0.3.1
RDBMS 11.1.0.7.10
Any ideas?
Regards,
Martin

I have found a solution myself:
[root@node1 trace]# echo -eth3 > /sys/class/net/bond1/bonding/slaves
[root@node1 trace]# echo -eth1 > /sys/class/net/bond1/bonding/slaves
[root@node1 trace]# echo +eth1 > /sys/class/net/bond1/bonding/slaves
[root@node1 trace]# echo +eth3 > /sys/class/net/bond1/bonding/slaves
Now node2 is automatically joining the cluster.
Regards,
martin

How to add a new instance to a 2 node SQL 2008 R2 Cluster under the same networkname

Hello All,
I'm fairly new to sql and have just deployed a new 2 node SQL 2008 R2 cluster.
During installation, I created a networkname SQLCLU and a named instance I01.
I now would like to add another Instance (I02) under the same networkname.
What is the proper procedure to do this?
I don't believe that I should do this using the action: "New SQL Server failover cluster" from the setup menu, but rather using the action "New Installation or add features to an existing installation"
With the second option, I'm however not sure how I should make this instance clustered..
Should I also execute the "New Installation or add features to an existing installation" action on the 2nd node in the cluster?
Many thanks for your advice!
Filip

You cannot use the same network name for 2 instances. You need to use a different network name for the second instance I02
A failover cluster instance contains:
A combination of one or more disks in a Microsoft Cluster Service (MSCS) cluster group, also known as a resource group. Each resource group can contain at most one instance of SQL Server.
A network name for the failover cluster instance.
One or more IP addresses assigned to the failover cluster instance.
One instance of SQL Server that includes SQL Server, SQL Server Agent, the Full-text Search (FTS) service, and Replication. You can install a failover cluster with SQL Server only, Analysis Services
only, or SQL Server and Analysis Services
http://msdn.microsoft.com/en-in/library/ms179410(v=sql.105).aspx

Installing 11gR1 CRS and receive error when executing root.sh on 2nd node of 2 node cluster

This is the error from the execution of root.sh on 2nd node of 2 node RAC cluster:
dhzusbx98: /u01/app/crs # ./root.sh
WARNING: directory '/u01/app' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
Setting the permissions on OCR backup directory
Setting up Network socket directories
PROT-1: Failed to initialize ocrconfig
Failed to upgrade Oracle Cluster Registry configuration
dhzusbx98: /u01/app/crs #
This is what I find in the following log file: /u01/app/crs/log/dhzusbx98/client\ocrconfig_6422692.log
Oracle Database 11g CRS Release 11.1.0.6.0 - Production Copyright 1996, 2007 Ora                                                                                                             cle. All rights reserved.
2015-02-11 11:21:07.384: [ OCRCONF][1]ocrconfig starts...
2015-02-11 11:21:07.388: [ OCRCONF][1]Upgrading OCR data
2015-02-11 11:21:07.455: [ OCROSD][1]utopen:7:failed to open OCR file/disk /dev                                                                                                             /lnkocrd01CC3 , errno=16, os err string=Device busy
2015-02-11 11:21:07.465: [ OCRRAW][1]proprinit: Could not open raw device
2015-02-11 11:21:07.465: [ default][1]a_init:7!: Backend init unsuccessful : [26                                                                                                             ]
2015-02-11 11:21:07.467: [ OCRCONF][1]Exporting OCR data to [OCRUPGRADEFILE]
2015-02-11 11:21:07.467: [ OCRAPI][1]a_init:7!: Backend init unsuccessful : [33                                                                                                             ]
2015-02-11 11:21:07.467: [ OCRCONF][1]There was no previous version of OCR. erro                                                                                                             r:[PROC-33: Oracle Cluster Registry is not configured]
2015-02-11 11:21:07.524: [ OCROSD][1]utopen:7:failed to open OCR file/disk /dev                                                                                                             /lnkocrd01CC3 , errno=16, os err string=Device busy
2015-02-11 11:21:07.524: [ OCRRAW][1]proprinit: Could not open raw device
2015-02-11 11:21:07.524: [ default][1]a_init:7!: Backend init unsuccessful : [26                                                                                                             ]
2015-02-11 11:21:07.560: [ OCROSD][1]utopen:7:failed to open OCR file/disk /dev                                                                                                             /lnkocrd01CC3 , errno=16, os err string=Device busy
2015-02-11 11:21:07.560: [ OCRRAW][1]proprinit: Could not open raw device
2015-02-11 11:21:07.560: [ OCRAPI][1]a_init:6b!: Backend init unsuccessful : [2                                                                                                             6]
2015-02-11 11:21:07.560: [ OCRCONF][1]Failed to initialized OCR context. error:[                                                                                                             PROC-26: Error while accessing the physical storage Operating System error [Devi                                                                                                             ce busy] [16]]
2015-02-11 11:21:07.560: [ OCRCONF][1]Exiting [status=failed]...
dhzusbx98: /u01/app/crs/log/dhzusbx98/client #
dhzusbx98: /u01/app/crs/log/dhzusbx98/client #
dhzusbx98: /u01/app/crs/log/dhzusbx98/client #
dhzusbx98: /u01/app/crs/log/dhzusbx98/client # set -o vi
dhzusbx98: /u01/app/crs/log/dhzusbx98/client # cat ocrconfig_6422692.log
Oracle Database 11g CRS Release 11.1.0.6.0 - Production Copyright 1996, 2007 Oracle. All rights reserved.
2015-02-11 11:21:07.384: [ OCRCONF][1]ocrconfig starts...
2015-02-11 11:21:07.388: [ OCRCONF][1]Upgrading OCR data
2015-02-11 11:21:07.455: [ OCROSD][1]utopen:7:failed to open OCR file/disk /dev/lnkocrd01CC3 , errno=16, os err string=Device busy
2015-02-11 11:21:07.465: [ OCRRAW][1]proprinit: Could not open raw device
2015-02-11 11:21:07.465: [ default][1]a_init:7!: Backend init unsuccessful : [26]
2015-02-11 11:21:07.467: [ OCRCONF][1]Exporting OCR data to [OCRUPGRADEFILE]
2015-02-11 11:21:07.467: [ OCRAPI][1]a_init:7!: Backend init unsuccessful : [33]
2015-02-11 11:21:07.467: [ OCRCONF][1]There was no previous version of OCR. error:[PROC-33: Oracle Cluster Registry is not configured]
2015-02-11 11:21:07.524: [ OCROSD][1]utopen:7:failed to open OCR file/disk /dev/lnkocrd01CC3 , errno=16, os err string=Device busy
2015-02-11 11:21:07.524: [ OCRRAW][1]proprinit: Could not open raw device
2015-02-11 11:21:07.524: [ default][1]a_init:7!: Backend init unsuccessful : [26]
2015-02-11 11:21:07.560: [ OCROSD][1]utopen:7:failed to open OCR file/disk /dev/lnkocrd01CC3 , errno=16, os err string=Device busy
2015-02-11 11:21:07.560: [ OCRRAW][1]proprinit: Could not open raw device
2015-02-11 11:21:07.560: [ OCRAPI][1]a_init:6b!: Backend init unsuccessful : [26]
2015-02-11 11:21:07.560: [ OCRCONF][1]Failed to initialized OCR context. error:[PROC-26: Error while accessing the physical storage Operating System error [Device busy] [16]]
2015-02-11 11:21:07.560: [ OCRCONF][1]Exiting [status=failed]...
dhzusbx98: /u01/app/crs/log/dhzusbx98/client # cat /etc/oracle/ocr.loc
ocrconfig_loc=/dev/lnkocrd01CC3
local_only=FALSE
Can I just retry the root.sh on the 2nd node? or, do I have to do some sort of cleanup and then retry? Please advise.... Thanks, Adolfo

Are you using raw disk devices for Oracle 11g RAC? You might want to move to ASM. For starters, your version, 11.1 is no longer fully supported by Oracle Corp. You'll need to be on 11.2.0.3 or higher to be supported. The 11.2.0.3 version loses Extended Support in August I think. The 11.2.0.4 version loses Extended Support in January. All of this means your production Oracle RAC database should be on the 12c version by January. And guess what? Oracle 12c RAC does not support raw disk devices.
Since you're starting with this cluster fresh, why not aim for a configuration that will allow you to go forth into the foreseeable future without major configuration changes?
Cheers,
Brian

Can I create different Coherence nodes in the same cluster with defferent?

Can I create different Coherence nodes in the same cluster with defferent cache-config.xml file ?
Can a cache be distributed in these deffirent nodes?

Yes. You can create different Coherence nodes in the same cluster with defferent cache-config.xml files as long as you use the same tangosol-coherence.xml file and the same tangosol-coherence-override.xml file. But you cannot store the cache data in the different nodes (started with different cache-config file). In other word, a node only create caches in their own's modes which are started with the same cache-config.xml file.
See the following demo:
I start a cache server using the cache config file examples-cache-server.xml. Then I start a storage-disabled cache console (cache client) using the cache config file coherence-cache-config.xml. Both of them using the same tangosol-coherence.xml file and the same tangosol-coherence-override.xml file.
The cache server uses a cache service PartitionedPofCache. But the client side is using the Distributedcache service. The cluster address is same 224.3.5.2.
The cluster name is also samme. They know each other.
D:\coherence\lib>D:\examples\java\bin\run-cache-server.cmd
D:\coherence\lib>D:\examples\java\bin\run-cache-server.cmd
The system cannot find the file D:\coherence.
The system cannot find the file C:\Oracle\Middleware\jdk160_11.
2009-12-22 12:09:31.400/4.987 Oracle Coherence 3.5.2/463 <Info> (thread=main, member=n/a): Loaded operational configurat
ion from resource "jar:file:/D:/coherence/lib/coherence.jar!/tangosol-coherence.xml"
2009-12-22 12:09:31.450/5.037 Oracle Coherence 3.5.2/463 <Info> (thread=main, member=n/a): Loaded operational overrides
from resource "jar:file:/D:/coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2009-12-22 12:09:31.470/5.057 Oracle Coherence 3.5.2/463 <D5> (thread=main, member=n/a): Optional configuration override
"/tangosol-coherence-override.xml" is not specified
2009-12-22 12:09:31.540/5.127 Oracle Coherence 3.5.2/463 <D5> (thread=main, member=n/a): Optional configuration override
"/custom-mbeans.xml" is not specified
Oracle Coherence Version 3.5.2/463
Grid Edition: Development mode
Copyright (c) 2000, 2009, Oracle and/or its affiliates. All rights reserved.
2009-12-22 12:09:33.864/7.451 Oracle Coherence GE 3.5.2/463 <Info> (thread=main, member=n/a): Loaded cache configuration
from "file:/D:/examples/java/resource/config/examples-cache-config.xml"
2009-12-22 12:09:39.983/13.570 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=n/a): Service Cluster joined t
he cluster with senior service member n/a
2009-12-22 12:09:43.187/16.774 Oracle Coherence GE 3.5.2/463 <Info> (thread=Cluster, member=n/a): Created a new cluster
"cluster:0xD3FB" with Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Locatio
n=process:144, Role=CoherenceServer, Edition=Grid Edition, Mode=Development, CpuCount=1, SocketCount=1) UID=0xC0A8085000
000125B75D888C60501F98
2009-12-22 12:09:43.508/17.095 Oracle Coherence GE 3.5.2/463 <D5> (thread=Invocation:Management, member=1): Service Mana
gement joined the cluster with senior service member 1
2009-12-22 12:09:46.582/20.169 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache:PartitionedPofCache, member=1
): Service PartitionedPofCache joined the cluster with senior service member 1
2009-12-22 12:09:46.672/20.259 Oracle Coherence GE 3.5.2/463 <Info> (thread=DistributedCache:PartitionedPofCache, member
=1): Loading POF configuration from resource "file:/D:/examples/java/resource/config/examples-pof-config.xml"
2009-12-22 12:09:46.702/20.289 Oracle Coherence GE 3.5.2/463 <Info> (thread=DistributedCache:PartitionedPofCache, member
=1): Loading POF configuration from resource "jar:file:/D:/coherence/lib/coherence.jar!/coherence-pof-config.xml"
2009-12-22 12:09:47.734/21.321 Oracle Coherence GE 3.5.2/463 <Info> (thread=main, member=1): Started DefaultCacheServer.
SafeCluster: Name=cluster:0xD3FB
Group{Address=224.3.5.2, Port=35463, TTL=4}
MasterMemberSet
ThisMember=Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=process
:144, Role=CoherenceServer)
OldestMember=Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=proce
ss:144, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=1, BitSetCount=2
    Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=process:144, Rol
e=CoherenceServer)
RecycleMillis=120000
RecycleSet=MemberSet(Size=0, BitSetCount=0
Services
TcpRing{TcpSocketAccepter{State=STATE_OPEN, ServerSocket=192.168.8.80:8088}, Connections=[]}
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.5, OldestMemberId=1}
InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
DistributedCache{Name=PartitionedPofCache, State=(SERVICE_STARTED), LocalStorage=enabled, PartitionCount=257, BackupCo
unt=1, AssignedPartitions=257, BackupPartitions=0}
2009-12-22 12:12:29.737/183.324 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member(Id=2, Timestamp=20
09-12-22 12:12:29.541, Address=192.168.8.80:8089, MachineId=24656, Location=process:1188, Role=CoherenceConsole) joined
Cluster with senior member 1
2009-12-22 12:12:30.498/184.085 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 2 joined Service M
anagement with senior member 1
2009-12-22 12:12:31.860/185.447 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): TcpRing: connecting to me
mber 2 using TcpSocket{State=STATE_OPEN, Socket=Socket[addr=/192.168.8.80,port=8089,localport=2463]}
2009-12-22 12:12:51.338/204.925 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=1): Member 2 joined Service D
istributedCache with senior member 2The following command starts a cache client.
D:\coherence\bin>coherence.cmd
D:\coherence\bin>coherence.cmd
** Starting storage disabled console **
java version "1.6.0_11"
Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) Server VM (build 11.0-b16, mixed mode)
2009-12-22 12:12:21.054/3.425 Oracle Coherence 3.5.2/463 <Info> (thread=main, member=n/a): Loaded operational configurat
ion from resource "jar:file:/D:/coherence/lib/coherence.jar!/tangosol-coherence.xml"
2009-12-22 12:12:21.355/3.726 Oracle Coherence 3.5.2/463 <Info> (thread=main, member=n/a): Loaded operational overrides
from resource "jar:file:/D:/coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2009-12-22 12:12:21.365/3.736 Oracle Coherence 3.5.2/463 <D5> (thread=main, member=n/a): Optional configuration override
"/tangosol-coherence-override.xml" is not specified
2009-12-22 12:12:21.415/3.786 Oracle Coherence 3.5.2/463 <D5> (thread=main, member=n/a): Optional configuration override
"/custom-mbeans.xml" is not specified
Oracle Coherence Version 3.5.2/463
Grid Edition: Development mode
Copyright (c) 2000, 2009, Oracle and/or its affiliates. All rights reserved.
2009-12-22 12:12:29.316/11.687 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=n/a): Service Cluster joined t
he cluster with senior service member n/a
2009-12-22 12:12:29.356/11.727 Oracle Coherence GE 3.5.2/463 <Info> (thread=Cluster, member=n/a): Failed to satisfy the
variance: allowed=16, actual=20
2009-12-22 12:12:29.356/11.727 Oracle Coherence GE 3.5.2/463 <Info> (thread=Cluster, member=n/a): Increasing allowable v
ariance to 17
2009-12-22 12:12:29.807/12.178 Oracle Coherence GE 3.5.2/463 <Info> (thread=Cluster, member=n/a): This Member(Id=2, Time
stamp=2009-12-22 12:12:29.541, Address=192.168.8.80:8089, MachineId=24656, Location=process:1188, Role=CoherenceConsole,
Edition=Grid Edition, Mode=Development, CpuCount=1, SocketCount=1) joined cluster "cluster:0xD3FB" with senior Member(I
d=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=process:144, Role=CoherenceS
erver, Edition=Grid Edition, Mode=Development, CpuCount=1, SocketCount=1)
2009-12-22 12:12:29.977/12.348 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=n/a): Member 1 joined Service
Management with senior member 1
2009-12-22 12:12:29.977/12.348 Oracle Coherence GE 3.5.2/463 <D5> (thread=Cluster, member=n/a): Member 1 joined Service
PartitionedPofCache with senior member 1
2009-12-22 12:12:30.578/12.949 Oracle Coherence GE 3.5.2/463 <D5> (thread=Invocation:Management, member=2): Service Mana
gement joined the cluster with senior service member 1
SafeCluster: Name=cluster:0xD3FB
Group{Address=224.3.5.2, Port=35463, TTL=4}
MasterMemberSet
ThisMember=Member(Id=2, Timestamp=2009-12-22 12:12:29.541, Address=192.168.8.80:8089, MachineId=24656, Location=proces
s:1188, Role=CoherenceConsole)
OldestMember=Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=proce
ss:144, Role=CoherenceServer)
ActualMemberSet=MemberSet(Size=2, BitSetCount=2
    Member(Id=1, Timestamp=2009-12-22 12:09:38.06, Address=192.168.8.80:8088, MachineId=24656, Location=process:144, Rol
e=CoherenceServer)
    Member(Id=2, Timestamp=2009-12-22 12:12:29.541, Address=192.168.8.80:8089, MachineId=24656, Location=process:1188, R
ole=CoherenceConsole)
RecycleMillis=120000
RecycleSet=MemberSet(Size=0, BitSetCount=0
Services
TcpRing{TcpSocketAccepter{State=STATE_OPEN, ServerSocket=192.168.8.80:8089}, Connections=[]}
ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.5, OldestMemberId=1}
InvocationService{Name=Management, State=(SERVICE_STARTED), Id=1, Version=3.1, OldestMemberId=1}
Map (?):
2009-12-22 12:12:49.505/31.906 Oracle Coherence GE 3.5.2/463 <Info> (thread=main, member=2): Loaded cache configuration
from "jar:file:/D:/coherence/lib/coherence.jar!/coherence-cache-config.xml"
2009-12-22 12:12:51.358/33.729 Oracle Coherence GE 3.5.2/463 <D5> (thread=DistributedCache, member=2): Service Distribut
edCache joined the cluster with senior service member 2
<distributed-scheme>

<scheme-name>example-distributed</scheme-name>
<service-name>DistributedCache</service-name>
<backing-map-scheme>
    <local-scheme>
      <scheme-ref>example-binary-backing-map</scheme-ref>
    </local-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>But when I try to store data into cache from the client side, it report error message: it's staorage-disabled. It shows that this cache console cannot store the data in the existing cache server because then using different cache config files.
Map (ca3): cache ca2
<distributed-scheme>

<scheme-name>example-distributed</scheme-name>
<service-name>DistributedCache</service-name>
<backing-map-scheme>
    <local-scheme>
      <scheme-ref>example-binary-backing-map</scheme-ref>
    </local-scheme>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
Map (ca2): put 1 one
2009-12-22 14:00:04.999/6467.370 Oracle Coherence GE 3.5.2/463 <Error> (thread=main, member=2):
java.lang.RuntimeException: Storage is not configured
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.onMissing
Storage(DistributedCache.CDB:9)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.ensureReq
uestTarget(DistributedCache.CDB:34)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.put(Distr
ibutedCache.CDB:22)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$BinaryMap.put(Distr
ibutedCache.CDB:1)
        at com.tangosol.util.ConverterCollections$ConverterMap.put(ConverterCollections.java:1541)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.DistributedCache$ViewMap.put(Distrib
utedCache.CDB:1)
        at com.tangosol.coherence.component.util.SafeNamedCache.put(SafeNamedCache.CDB:1)
        at com.tangosol.coherence.component.application.console.Coherence.processCommand(Coherence.CDB:581)
        at com.tangosol.coherence.component.application.console.Coherence.run(Coherence.CDB:39)
        at com.tangosol.coherence.component.application.console.Coherence.main(Coherence.CDB:3)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.tangosol.net.CacheFactory.main(CacheFactory.java:1400)

Will HANA node supports cluster (Initiator node and Eexecutor node)

Hi Gurus,
In vertica DB supports below cluster concepts, Will HANA supports
Initiator node and Eexecutor node
1. client submits query to vertica cluster the node to which the client is connected is called the initiator node
2. Initiator node analyses query and distributes query plan to executor nodes
3. Executor nodes run query on local data set and return partial result to initator node
Regards,
Manoj.

Hello,
as Vitaliy wrote - you will not find exactly identical equivalent in HANA database but also HANA database can offer similar functions.
In case you want to spread workload across several machines you should check:
SAP HANA Database Server Installation Guide (page 13 - 5.2 Distributed System; page 35 - Adding a Host to the System):
https://service.sap.com/~sapidb/011000358700000604572011
Note: Master nameserver can logcally failover to other node. This can be seen as high availability.
In case you are interested in high availability on instance level - you might want to look as well at following guide:
SAP HANA Database Scaleout with Standby Host:
http://help.sap.com/hana/hana1_imdb_scale_en.pdf
Tomas

How to monitor the performances of VMs & Hyper-v Cluster host node running on SCVMM Cluster.

hello...,
How to monitor the performances of VMs & Hyper-v Cluster hosts node running on SCVMM Cluster from SCOM so that we can
Identify the highest utilized(CPU and Memory ) VM on that from cluster hyper-v host.
Identify the lowest utilized (CPU and Memory )Hyper-v Host in the Cluster .
After identifies VMs and Hyper-v cluster host on SCVMM , so that we can proceed to do migrate the Highest Utilized VM to Lowest Utilized
Hyper-v cluster host.
To identified and implement above ,what are the things I need to do or configured on SCOM.
Thanks
RICHA KM

hello...,
How to monitor the performances of VMs & Hyper-v Cluster hosts node running on SCVMM Cluster from SCOM
so that we can
Identify the highest utilized(CPU and Memory ) VM on that from cluster hyper-v
host.
Identify the lowest utilized (CPU and Memory )Hyper-v Host in the Cluster .
After identifies VMs and Hyper-v cluster host on SCVMM , so that we can proceed
to do migrate the Highest Utilized VM to Lowest Utilized Hyper-v cluster host.
To identified and implement above ,what are MPs i need to installed on SCOM for implementing
this.
Thanks
RICHA KM

Methods of Performing Maintenance on a Two Node Exchange 2007 CCR Cluster

What methods do you have for performing maintenance on a two node Exchange 2007 CCR cluster? This is what I use:Via the EMS (pre-maintenance steps)On the passive node: Get-StorageGroup | SuspendGroupCopyCheck Event ViewerOn the passive node, check replication status: Get-StorageGroupCopyStatusOn the active node, check replication status: Get-StorageGroupCopyStatusCheck Outlook is still working for usersGet-ClusterMailboxServerStatus -identity CLUSTERNAMEVia the EMS (post-maintenance steps)On the passive node, resume replication: Get-StorageGroup | Resume-StorageGroupCopyOn the passive node, check replication status: Get-StorageGroupCopyStatusUseful ArticlesApplying Exchange 2007 Rollups to Clustered Mailbox Servers:https://technet.microsoft.com/en-us/library/bb885043(v=exchg.80).aspxMove a Clustered Mailbox Server in a CCR environment:...
This topic first appeared in the Spiceworks Community

I thought the virtual IP address by virtue of being 'virtual' should not change? :)
I guess it makes more sense to add the Virtual IPs. You can even add multiple hosts (with different IPs).
Regards
Farrukh

Maximum number of nodes in a Weblogic cluster on RedHat Linux?

Is there a limitation of the number of nodes in a weblogic cluster
          running under RedHat Linux?
          Can I start with 5 nodes and in a year scale up to 500 or 5000 nodes?
          Thanks!
          Ralf.


Ralf,
          > Is there a limitation of the number of nodes in a weblogic cluster
          > running under RedHat Linux?
          A realistic limit, of course.
          > Can I start with 5 nodes and in a year scale up to 500 or 5000 nodes?
          If your app is completely stateless, then it can scale to 40 maybe 80
          servers.
          The problem is that stateless apps typically manage state that sits behind
          them, and there's basically no database in the world that can handle the
          load that 40 servers can put on it. Depending on the app, you can easily
          saturate 4 database CPUs per 1 app server CPU, but usually the factor is
          closer to 1:1, and with agressive caching in the app tier even less.
          Things like stateful session bean replication and HTTP session replication
          in a cluster ... well, YMMV ... but I would hypothesize that it won't scale
          up anywhere close to 40 servers under load.
          Peace,
          Cameron Purdy
          Tangosol, Inc.
          http://www.tangosol.com/coherence.jsp
          Tangosol Coherence: Clustered Replicated Cache for Weblogic
          "Ralf Reddin" <[email protected]> wrote in message
          news:[email protected]..
          >

Help required for temporary removing node 2 from the cluster

Hi Support,
There are some hardware issues on node 2 of 11g RAC cluster and vendor going to fix it;
We want to make node 2 not available until they fix it;
The user will continue with node 1 of the cluster;
Following are the current status of the site and I'm considering to perform the folowing task
while the folllowing command from node 1, i got the following output
D:\srvctl status database –d EPVAL
Instance epval1 is running on node corskmaps001
Instance epval2 is running on node corskmaps001
D:\srvctl status database –d EPPROD
Instance epprod1 is running on node corskmaps001
Instance epprod2 is running on node corskmaps001
I made the following steps in chorological order; I appreciate if someone can correct me if I made any mistake or add/amend/update as per requirement.
Assuming corskmaps002 is our passive mode which will be taken away by HP for hardware checking;
srvctl stop instance –d EPPROD -i EPPROD2
srvctl stop instance –d EPVAL -i EPVAL2
srvctl stop listerner_corskmaps002 –n corskmaps002
srvctl stop nodeapps -n corskmaps002
Do we require stopping service also once we stop the instance? Following are the commands if your answer is yes;
srvctl stop service -d EPROD -i EPPROD2
srvctl stop service -d EPVAL -i EPVAL2
srvctl disable service -d EPPROD -s EPPROD2
srvctl disable service -d EPVAL -s EPVAL2
I appreciate help
Thanks

Hi Bjoern,
I agree with you, very valid point, thanks for that;
Just for clarification, do you thing the sequence should be like this?
********disable command to start with***********
srvctl disable service -d EPPROD -s EPPROD2
srvctl disable service -d EPVAL -s EPVAL2
*********than instance stop commad**********
srvctl stop instance –d EPPROD -i EPPROD2
srvctl stop instance –d EPVAL -i EPVAL2
************than listener stopping command********
srvctl stop listerner_corskmaps002 –n corskmaps002
*************than node stopping command*********
srvctl stop nodeapps -n corskmaps002
*********do we also require stopping of the service ?>***************************
Do we require stopping service also once we stop the instance? Following are the commands if your answer is yes;
srvctl stop service -d EPROD -i EPPROD2
srvctl stop service -d EPVAL -i EPVAL2
Can you please eleborate further
Then, you also might want to add '-o transactional' or "transactional local" to the
instance stop command so that running transactions are not affected by this.
Thanks

Converting 8 node to 4 node caused CLUSTER waits

Hi,
We had an 8 node 10.2.0.3 cluster which was working fine. For some reasons, we reinstalled the cluster with 4 nodes (fresh OSs, but everything was same). we mounted the previous asm diskgroups with the new cluster. Everything seems to be working fine, except we have lots of waits on cluster class, mostly gc cr block busy. Does anyone have any idea about that?
thanks

AWR says a few sql statements causing the cluster waits, but these are not new queries. On the 8 node cluster we had the same sql queries, but we rarely had CLUSTER waits.Not Sure about nodes to 4 nodes.
But I think we need to investigate and improve from that query before.
check Top 5 Timed Events, Wait Events, objects, indexes , gather stats, block size and etc...
Good Luck

Cannot perform add node procedure for non-cluster Oracle homes

After deleting one of nodes,tried to reinstall the node by addNode.sh.
Got this error in Installer Window "*Cannot perform add node procedure for non-cluster Oracle homes*"
Any idea?

what is your oracle version and operating system name and version?
check the output of cluvfy.sh
runcluvfy.sh stage -pre crsinst -n node1,node2,.... -verbose
refer the link:-
http://download.oracle.com/docs/cd/B28359_01/rac.111/b28254/adddelunix.htm#CEGBACAH

Additional Node License for existing cluster

Hi,
I am having two MCS servers having call manager version 8.5 installed in a single cluster. Now I have bought one new MCS server with the same version call manager and I want to add this node in the same cluster but I am having only two node license installed in my call manager cluster and that is already being used.
Please help me how can I can get the license to add this additional node in the cluster. If I have to order something then please share the part code which I have to order.
Thanks,
Shaheen Sidana

Nodes are free with version 8.5, however you need to order the free part number and receive a PAK, contact your Cisco reseller to submit the order or you can try sending and email to [email protected] to see if they can assist.
HTH,
Chris

How to fill or bind data using Value Node in Tree Node

Hi Gurus,
Can anybody help me on how to fill data or bind data using Value Node in Tree Node View. I know how to create Tree Node but not able to show value on the UI in Tree View.
Can u please let if anybody has done it?
Thanks in advance.
Madhusudan

continued...
TRY.
          lv_child = me->node_factory->get_proxy(
                    iv_bo = lv_value_node
                    iv_parent_proxy = me
                    iv_proxy_type = 'ZL_CLASS_CN02' ).
          lv_child->is_leaf = 'X'.
          APPEND lv_child TO rt_children.
        CATCH cx_sy_move_cast_error cx_sy_ref_is_initial.
      ENDTRY.
In the above code iv_bo , lv_value_node will be the actual object of the second node or leaf node here, which will have the same structure of parent node along with data. After/before this, you would need to build table and refresh in do-prepare_output of IMPL class.In the above code iv_bo , lv_value_node will be the actual object of the second node or leaf node here, which will have the same structure of parent node along with data. After/before this, you would need to buid table and refresh in do-prepare_output of IMPL class.
ztyped_context->resultlist->build_table( ).
IF ztyped_context->resultlist->node_tab IS INITIAL.
    ztyped_context->resultlist->refresh( ).
ENDIF.
Also the EH_ONEXPAND has to be implemented and event handled in DO_HANDLE_EVENT. But this expand event has to be delegated to context node directly as CL_BSP_WD_CONTEXT_NODE_TREE will already have the implementation.
ztyped_context->resultlist->expand_node( lv_event->row_key ).
Where in result list is the node ZL_CLASS_CN00.
After typing the whole content , i found this blog :). There are few things i have written more that in the blog. /people/poonam.assudani/blog/2009/06/24/create-a-tree-view-display-in-crm-web-ui
Regards,
Karthik

Second node of 2 node cluster rebooted on it's own

Similar Messages

Maybe you are looking for