Multiple Senior Cluster Members?

Hi Guys,
We've had a few nodes kicked out of one of our production clusters all with messages similar to this:
ERROR 2008-04-21 18:17:05.753 Oracle Coherence GE 3.3.1/389p1 <Error> (thread=Cluster, member=29): Received cluster heartbeat from the senior Member(Id=2, Timestamp=2008-04-18 11:07:21.948, Address=172.21.205.151:8089, MachineId=29847, Location=process:17367@trulxfw0006,member:trulxfw0006-2) that does not contain this Member(Id=29, Timestamp=2008-04-18 11:07:34.491, Address=172.21.205.148:8092, MachineId=29844, Location=process:17098@trulxfw0003,member:trulxfw0003-5); stopping cluster service.
DEBUG 2008-04-21 18:17:05.753 Oracle Coherence GE 3.3.1/389p1 <D5> (thread=Cluster, member=29): Service Cluster left the cluster
The logs on the senior member (2) are interesting though:
DEBUG 2008-04-21 18:17:05.601 Oracle Coherence GE 3.3.1/389p1 <D5> (thread=Cluster, member=2): Member 29 left service Management with senior member 2
DEBUG 2008-04-21 18:17:05.602 Oracle Coherence GE 3.3.1/389p1 <D5> (thread=Cluster, member=2): Member 29 left service WriteQueueSync with senior member 3
DEBUG 2008-04-21 18:17:05.602 Oracle Coherence GE 3.3.1/389p1 <D5> (thread=Cluster, member=2): Member 29 left service WriteQueueAsync with senior member 3
DEBUG 2008-04-21 18:17:05.603 Oracle Coherence GE 3.3.1/389p1 <D5> (thread=Cluster, member=2): Member 29 left service DistributedCache with senior member 3
DEBUG 2008-04-21 18:17:05.603 Oracle Coherence GE 3.3.1/389p1 <D5> (thread=Cluster, member=2): Member 29 left service ServiceControl with senior member 3
DEBUG 2008-04-21 18:17:05.604 Oracle Coherence GE 3.3.1/389p1 <D5> (thread=Cluster, member=2): Member 29 left service InvocationService with senior member 2
DEBUG 2008-04-21 18:17:05.607 Oracle Coherence GE 3.3.1/389p1 <D5> (thread=Cluster, member=2): Member 29 left Cluster with senior member 2
I wasn't aware that there could be multiple senior members. Is this indicative of something bad going on?
Thanks, Paul
PS Metalink is not behaving so I can't raise it there.

Hi Jon,
I've done some more digging into the logs for the whole cluster.
Of the five nodes that left the cluster, three have the same reason:
trulxfw0002/180-primary-0.log.1:ERROR 2008-04-21 17:05:19.763 Oracle Coherence GE 3.3.1/389p1 <Error> (thread=Cluster, member=34): This node appears to have partially lost the connectivity: it receives responses from MemberSet(Size=2, BitSetCount=2, ids=[8, 32]) which communicate with Member(Id=22, Timestamp=2008-04-18 11:07:29.911, Address=172.21.205.149:8092, MachineId=29845, Location=process:11858@trulxfw0004,member:trulxfw0004-5), but is not responding directly to this member; that could mean that either requests are not coming out or responses are not coming in; stopping cluster service.
trulxfw0003/180-primary-7.log.3:ERROR 2008-04-21 00:40:02.153 Oracle Coherence GE 3.3.1/389p1 <Error> (thread=Cluster, member=28): This node appears to have partially lost the connectivity: it receives responses from MemberSet(Size=2, BitSetCount=3, ids=[35, 38]) which communicate with Member(Id=5, Timestamp=2008-04-18 11:07:21.992, Address=172.21.205.151:8090, MachineId=29847, Location=process:17351@trulxfw0006,member:trulxfw0006-1), but is not responding directly to this member; that could mean that either requests are not coming out or responses are not coming in; stopping cluster service.
trulxfw0006/180-primary-0.log.6:ERROR 2008-04-18 23:13:33.896 Oracle Coherence GE 3.3.1/389p1 <Error> (thread=Cluster, member=1): This node appears to have partially lost the connectivity: it receives responses from MemberSet(Size=2, BitSetCount=3, ids=[14, 42]) which communicate with Member(Id=28, Timestamp=2008-04-18 11:07:34.381, Address=172.21.205.148:8091, MachineId=29844, Location=process:17152@trulxfw0003,member:trulxfw0003-7), but is not responding directly to this member; that could mean that either requests are not coming out or responses are not coming in; stopping cluster service.
Member 29 had been marked as paused during its lifetime by various other nodes, however there are not frequent at the time of eviction:
grep -R "member:trulxfw0003-5" * | grep "failed to respond" | grep "18:17"
grep -R "member:trulxfw0003-5" * | grep "failed to respond" | grep "18:16"
trulxfw0004/180-primary-6.log.8:DEBUG 2008-04-18 18:16:26.337 Oracle Coherence GE 3.3.1/389p1 <D6> (thread=PacketPublisher, member=23): Member(Id=29, Timestamp=2008-04-18 11:07:34.491, Address=172.21.205.148:8092, MachineId=29844, Location=process:17098@trulxfw0003,member:trulxfw0003-5) has failed to respond to 17 packets; declaring this member as paused.
trulxfw0005/180-primary-1.log.7:DEBUG 2008-04-21 18:16:37.315 Oracle Coherence GE 3.3.1/389p1 <D6> (thread=PacketPublisher, member=10): Member(Id=29, Timestamp=2008-04-18 11:07:34.491, Address=172.21.205.148:8092, MachineId=29844, Location=process:17098@trulxfw0003,member:trulxfw0003-5) has failed to respond to 17 packets; declaring this member as paused.
[xflow@lonrs00342 machines]$ grep -R "member:trulxfw0003-5" * | grep "failed to respond" | grep "18:15"
trulxfw0002/180-primary-7.log.8:DEBUG 2008-04-18 18:15:44.161 Oracle Coherence GE 3.3.1/389p1 <D6> (thread=PacketPublisher, member=37): Member(Id=29, Timestamp=2008-04-18 11:07:34.491, Address=172.21.205.148:8092, MachineId=29844, Location=process:17098@trulxfw0003,member:trulxfw0003-5) has failed to respond to 17 packets; declaring this member as paused.
trulxfw0006/180-primary-7.log.9:DEBUG 2008-04-18 18:15:51.477 Oracle Coherence GE 3.3.1/389p1 <D6> (thread=PacketPublisher, member=4): Member(Id=29, Timestamp=2008-04-18 11:07:34.491, Address=172.21.205.148:8092, MachineId=29844, Location=process:17098@trulxfw0003,member:trulxfw0003-5) has failed to respond to 17 packets; declaring this member as paused.
grep -R "member:trulxfw0003-5" * | grep "failed to respond" | grep "18:14"
trulxfw0002/180-primary-0.log.41:DEBUG 2008-04-18 22:18:14.220 Oracle Coherence GE 3.3.1/389p1 <D6> (thread=PacketPublisher, member=34): Member(Id=29, Timestamp=2008-04-18 11:07:34.491, Address=172.21.205.148:8092, MachineId=29844, Location=process:17098@trulxfw0003,member:trulxfw0003-5) has failed to respond to 17 packets; declaring this member as paused.
[xflow@lonrs00342 machines]$
grep -R "member:trulxfw0003-5" * | grep "failed to respond" | grep "18:13"
trulxfw0002/180-primary-0.log.43:DEBUG 2008-04-18 18:13:41.083 Oracle Coherence GE 3.3.1/389p1 <D6> (thread=PacketPublisher, member=34): Member(Id=29, Timestamp=2008-04-18 11:07:34.491, Address=172.21.205.148:8092, MachineId=29844, Location=process:17098@trulxfw0003,member:trulxfw0003-5) has failed to respond to 17 packets; declaring this member as paused.
How closely does a paused declaration correlate with a vote for eviction?
The last node which was kicked out had a similar pattern to trulxfw0003-5 above, however it had significantly more paused declaration messages in the minutes preceding eviction.
We have a listener on the Cluster service which detects the local node leaving the cluster and kills itself. Is this a good idea, or is it safer to let Coherence sort itself out?
Thanks, Paul

Similar Messages

Possible to have multiple cluster members in the same JVM (for unit tests)?

Hi,
I'm wondering if it's possible to simulate multiple cluster members inside of a single JVM. I'm looking to unit test my code and it would be great to write cases for various boundary conditions.
This could certainly be done with multiple JVMs but would be more difficult to run in something like cruise control etc.
TIA,
Danny

Hi Danny,
I do not know how to accomplish what you are asking for and run several Coherence nodes in a single JVM, but I have written a JUnit base class that starts a configurable number of nodes (each in its own JVM) and provides some other useful methods for unit/integration testing with Coherence, such as method to clear all caches or to load test data from the CSV file.
It is still a work in progress and you might need to tweak things a bit to make it work in your environment, but it will at least give you a head start.
Shoot me an email if you are interested and I'll send it to you (you can find my email in my profile).
Regards,
Aleks

What is behind the scenes of "Automatic Discovery of Cluster Members?"

Do any of you know the basic mechenism ( or concept ) of authomatic discovery of cluster members?
And how can we be confident that an application deployed anywhere on the network (LAN,WAN, or the Internet) can join a cluster?
Sorry for this clumsy question, but I realy need to know this.
Thanks in advance!
Scott

Hi Scott,
Coherence uses a protocol called TCMP which is described in the following Wiki article:
http://wiki.tangosol.com/display/COH32UG/Network+Protocols
By default, multicast is used to discover if a cluster is already running that the new member may join.
In order to test multicast in your environment, you need to run the multicast test:
http://wiki.tangosol.com/display/COH32UG/Multicast+Test
Some environments don't allow multicast or some switches don't handle multicast very well, in which case you'd need to run using WKA (Well Known Address):
http://wiki.tangosol.com/display/COH32UG/well-known-addresses
Regards,
Jon Hall.
p.s. once you've confirmed that clustering is taking place (either using the default of multicast or using WKA), then you should run the datagram test to test your network's performance:
http://wiki.tangosol.com/display/COH32UG/Datagram+Test
Also, check out the production checklist as this has lots of really good information in it that will be useful going forward:
http://wiki.tangosol.com/display/COH32UG/Production+Checklist

Determining Cluster Members Using a NamedCache

Is it possible to get the cluster members using a particular NamedCache?
Is it possible to add a listener which listens for members getting/returning (ensure/release) a particular NamedCache?
Is there a concept of groups within a cluster membership? Some way of both indicating what a particular node will be doing (say if you wanted to have a "nodata" member which held no objects in its partition and a "data" member which did hold the objects and could have the same configuration file, only somehow give it the parameter to indicate which an instance would be) as well as identify it from the cluster info (get all cluster members of the "data" type)?
--Tim

1) It's possible to get all cluster members that are
using (or providing storage for) a given cache
service:Excellent. Just what I was looking for.
More generally, starting with Coherence 3.2, there
are number of <aThis is what we need. I used storage enabled as an example, but we want to have roles for cluster members, so this is the better solution. We're currently on 3.0, but I guess this is a good impetus to upgrade.
--Tim

Query - Sharing single Input/output FRS across multiple CMS Cluster

Hi,
I have two BOE XI R2 SP3 servers on windows 2003r2 x64 which are with one BOE clustered, sharing file repository (FRS) on common NAS storage folders. Objective is to vertically scale BOE servers by adding more clusters on these two BOE servers. I wish to add few more clusters by adding CMS through CCM wizard option pointing to separate CMS db sachems.
Can multiple CMS cluster share same FRS component with separate folder structure OR do I have to duplicate each BOE component per cluster basis.
Regards,
G.

Hi
Thanks for your response.
Hardware allocated for BOE is only 2 servers, and design is to maintain 3 separate environments( One change should not affect other). We wish to share Production, Testing and support BOE environments on same hardware.
Only objective of maintaining multiple BOE cluster on same hardware is to provide non inter dependent BOE service to various user groups.
Suggest needful
Regards,

IPlanet Plug-in does not recognize all cluster members

          We are running 7 WLS 6.1 SP2 server instances on two Solaris 2.8 machines. (3 on
          machine A , 4 on machine B). The admin server is also running on Machine A.
          All the cluster members appear to join the cluster correctly on startup, at least
          as far as the WLS console shows. No logged errors.
          We are using the IPlanet 6.0, sp4 with the Weblogic NSAPI plug-in to load-balance
          across the cluster.
          However the Plug-in will only recognize the server instances on machine A or machine
          B during any one life of the IPlanet. We can never get the plug-in to recognize
          all instances at one time, only on one machine or the other. We can see this from
          the results of a __WebLogicBridgeConfig.
          Using same configuration WLS configuration, run where machine A and B are domains
          on the same physical machine uses both A and B servers.
          We suspect this has to do with a multicast problem, running MulticastTest between
          machines A and B fail to receive messages.
          My questions are,
          A) Why would we be seeing all server instances as having successfully joined the
          cluster, if multicast is failing?
          B) How would multicast have any effect on how the NSAPI plug-in sees the cluster
          members?
          C) Why would the cluster instances seen by the NSAPI plug-in alternate between
          machine only A or B, after each shutdown and start-up ?
          WLS 6.1 SP2 on Solaris 2.8, IPlanet 6.0 sp4.


I am seeing a similar problem. I have three machines - each with four managed servers running on them. All 12 servers in the cluster are are specified in the obj.conf for NSAPI. Only eight of the servers get used, however. The wierd part is that machine1 contains managed servers 1, 4, 7, and 10. Machine 2 contains managed servers 2, 5, 8, and 11. Machine 3 contains managed servers 3, 6, 9, and 12. In the obj.conf, we list the machines in ourder 1 to 12 for the cluster. For some reason the NSAPI is skipping managed servers 3, 6, 9, and 12. I have checked both the connectivity for the ip and the port. The fire wall seems fine.
          Did you ever get a response or figure out what was causing this problem.
          Thanks,
          Mark

Any way to communicate a message to all cluster members?

Is there any way I can communicate a message to all cluster members?
          In my case I do some data caching within each cluster member. I'm trying to
          implement a mechanism that will enable me to have every server flush their
          cache. I was hoping that I could send the request to a servlet (on any
          machine) that in-turn would send a 'multicast' message to all app servers
          that would request a cache refresh.
          Any ideas?
          Thanks.
          Marko.


JMS topics are great for multi-server synchronization.
          Marko Milicevic <[email protected]> wrote in message
          news:39dba2e7$[email protected]..
          > Is there any way I can communicate a message to all cluster members?
          > In my case I do some data caching within each cluster member. I'm trying
          to
          > implement a mechanism that will enable me to have every server flush their
          > cache. I was hoping that I could send the request to a servlet (on any
          > machine) that in-turn would send a 'multicast' message to all app servers
          > that would request a cache refresh.
          >
          > Any ideas?
          >
          > Thanks.
          >
          > Marko.
          > .
          >
          >

Configure weight for cluster members ?

Hai everyone,
I have installed weblogic 10.3.4 and configured a cluster with two members.
Now i want to do load balancing & failover for my cluster and also want to set the weight for each cluster members.
please guys help me to resolve this one...
Regards,
JagadesH L.

General cluster documentation can be found here: http://download.oracle.com/docs/cd/E17904_01/web.1111/e13709/toc.htm
The information concerning load balancing (and the description of the various algorithms) can be found here: http://download.oracle.com/docs/cd/E17904_01/web.1111/e13709/load_balancing.htm#CHDGFIBD

EPMDimensionOverride - Multiple Properties and Members

Hello Experts,
I am having trouble getting an =EPMDimensionOverride formula to work for multiple Properties and Members. Currently we override our EMPLOYEE Dimension based on the COST_CENTER property. The Formula is =EPMDimensionOverride("000","EMPLOYEE",A3) where A3 equals COST_CENTER = 1234.
I want add an additional filter to Exclude a couple members from the GROUP property. The Members are "Hourly" and "Temp"
I have tried a couple versions including =EPMDimensionOverride("000","EMPLOYEE",A3 & GROUP <> Hourly, Temp) and =EPMDimensionOverride("000","EMPLOYEE",A3; GROUP <> Hourly, Temp)
Any help is appreciated.
Thanks,
Sean

Hi Vadim,
I am also having the same issue, but in my case i need to get all the members with the entity=&C4, in C4 i have my page axis entity dimension value is there.
I am dynamically filtering the members with the help of property Entity here. But at the same time i need to have the immediate parent values of these members also along with these members, which have the property entity = BLDG.
But I couldn't able to execute it, rather i am getting the below error #VALUE! , any suggestions to make this possible. I am in EPM SP 18 .Net 4
Please find the Screen shot below for your reference.
Thanks,
Siva

How to have multiple default dimension members

Hi There
I am using AS2008, when editing a role, under the dimension data tab, under advanced, you can define a default dimension member.
However i need my default dimension member to be members, i have tried using a attribute set to true for many members and creating a hierarchy.
But neither works, it seems to randomly pick one Key for a set of members and you cannot use a hierarchy to define members.
How can i define multiple default members ? what i really want is all members except one.
Thanx

We're going to need some more details about exactly what you did because it really sounds like you're making a mistake. This extra attribute should work.
The only situation I can think of where it might not work as intended is if your new attribute is a property of the state attribute... when you slice by the state attribute, it will override the default member on the new attribute.
What are your attribute relationships in that dimension? What do you get when you run the following MDX queries?
with
member test1 as [Geography].[State].DefaultMember.UniqueName
member test2 as [Geography].[NewAttribute].DefaultMember.UniqueName
select {test1,test2} on 0
from YourCube
select {} on 0,
[Geography].[NewAttribute].[NewAttribute].Members
*[Geography].[State].[State].Members on 1
from YourCube
As for the calculated member [Geography].[State].[All].[West], you can add to that to the MDX script (the Calculations tab in the cube designer) and it will be available to all sessions.http://artisconsulting.com/Blog/GregGalloway

Resource plan not migrating to all targeted cluster members

Hi,
We got an environment setup like:
Physical machine 1: AdminServer, OSB1, WLS1
Physical machine 2: OSB2, WLS2
Where we have put OSB[1,2] into a cluster and WLS[1,2] into a cluster. The servers are running WLS 10.3.3.4.
The problem is that: When creating a new connectionFactory for the DBAdapter, the plan.xml is only updated on 1 machine (not both of them). In my setup, it is always physical machine 1 that gets the updated plan.xml
I looked at the target for the deployment, and it is set for AdminServer and OSBCluster (meaning, OSB1 and OSB2). And when going into the "Monitoring -> Outbound Connection Pools", I can also see that only the AdminServer and OSB1 has an outbound connection pool.
To fix this, I've had to use SCP from OSB1 -> OSB2 for the plan.xml and then doing a redeploy of the DBAdapter. Then, I see an outbound connection from OSB2.
Have I miss understood something? Shouldn't the configurations (plan.xml) be replicated across all targeted members, in case one servers fails?
Best Regards,
Mathias

hello,
          and what about ServerAffinity (to be set on the ConnectionFactory-level)? Should be "false". Antoher explanation is that your cluster is out of sync (check, if multicast has been set correctly).
          load balancing and server affinity for dd
          regards,
          makiey

How to setup cluster members on different os and communicate between them ?

Hi ,
I was wondering how to set up a cluster environment with members say c1 and c2, where c1 will be on windows and c2 will be on solaris(sparc) platforms with Sun Java System App Server 9.1 and then to control and communicate effectively. We have a Sun Java Syst. Web Server 7.0 as the web server.
Any help will be highly appreciated.
Enthusiast.

Could you explain the rationale behind this configuration? We've had inquiries in the past about mixing Solaris/SPARC and Solaris/X64 in clusters but never this diversity of operating environment.
I'm not entirely sure you need a cluster per se. Forgive my ignorance, but doesn't SJS App Server have some sort of build-in clustering capability? If so, why do you need, in this limited circumstance, a cluster framework? You wouldn't be able to have a common file system (endian issues, etc) any way.
Regards,
Tim
---

Multiple Series Cluster Barchart locks up HTMLDB_PUBLIC_USER DB sessions

I am trying out cluster barcharts. I had a cluster barchart with two series. If I have no ORDER BY clause on the SELECT statements, the chart displays, but the bars are not clustered together correctly. I want all the bars from series #1 together and the bars from series #2 together ...
When I put an ORDER BY clause on the SELECTs in the two series, no chart is rendered. The chart region is blank. The problem is that the session connected to the database locks up/freezes. I refresh the page and now I have two sessions connected to the database stuck. When I refresh enough times we are unable to run HTML DB. The sessions run until they fill up TEMP tablespace. (I do not know why. The table they are running against only has 9 rows.) I had to go into the database with SQL*Plus and kill the sessions. Once these HTML DB session were killed, I was able to connect to HTML DB.
Questions
1) Is this a known problem with cluster barcharts?
2) How do I cluster the bars for each series together?
Mike

Hi,
if you remove "encryption mode ciphers aes-ccm tkip" from the radio interface does it help?
it should remain like this:
interface Dot11Radio0
no ip address
no ip route-cache
encryption vlan 1 mode ciphers aes-ccm tkip
ssid WLAN_Corporate
ssid WLAN_HartKitGuest
HTH,
Tiago
If this helps you and/or answers your question please mark the question as "answered" and/or rate it, so other users can easily find it.

Multiple error cluster lines

Hallo LabVIEWers,
trying to make a good programming style, I implement error cluster inputs
and outputs on each of my VIs very thoroughly.
Now then, if I use eg the VI Server to call my VIs (call by reference node)
I have two error cluster outputs in my data flow, one from the Call by
reference and one from my own VI.
Now my question: How should I handle these two flows, is there a way to or
them together ? Should I ignore one of them ? Should I keep them seperate ?
Wiring the CallByReference output cluster to the own VI's input doesnt't
work because of cycle dataflow dependancies (of course)! What's a good
LabVIEW style and what are your suggestions ??
Thank you
Oliver

Under the Time&Dialog palette
Merge Errors
Lets you merge error I/O clusters from different functions. The function
first looks for errors among error in 1, error in 2 and error in 3, then
through error array in. The first error found is reported. If no errors are
found, the function then looks for warnings and returns the first warning
found. If no error warning is found, the function returns no error.
Click the parameters for more information.
If you're in LV5 you'll just have to implement a similar function.
Oliver Friedrich wrote in message
news:[email protected]..
> Hallo LabVIEWers,
>
> trying to make a good programming style, I implement error cluster inputs
>
and outputs on each of my VIs very thoroughly.
> Now then, if I use eg the VI Server to call my VIs (call by reference
node)
> I have two error cluster outputs in my data flow, one from the Call by
> reference and one from my own VI.
>
> Now my question: How should I handle these two flows, is there a way to or
> them together ? Should I ignore one of them ? Should I keep them seperate
> Wiring the CallByReference output cluster to the own VI's input doesnt't
> work because of cycle dataflow dependancies (of course)! What's a good
> LabVIEW style and what are your suggestions ??
>
> Thank you
>
> Oliver
>
>

Multiple devices & family members

Ok, I am new to the Apple family and am confused on some items. I have an Iphone and Imac, my kids have Itouches, and ipods, my husband is addicted to the ipad! So, Does each person need to set up their own apple id? We have all been using mine. I just want one main music library that we can all share from, but want them to have their own apps and things for their own devices. What is my best bet to accomplish this?

Read this through and it will give you some ideas how to configure your Apple devices for a family.
http://www.macstories.net/stories/ios-5-icloud-tips-sharing-an-apple-id-with-you r-family/

Multiple Senior Cluster Members?

Similar Messages

Maybe you are looking for