Server Pool Master - 2 node HA setup - Need suggestion

I run OVM 2.2.1 in HA since 1 year.
Have 2 node. Guests failover when Utility Server,Virtual Machine Server crash.
But have problem with Server Pool Master.... ( My Server Pool Master it installed on my Node 1)
When my Server Pool Master crash, All HA guest move and restart on my second server. But because i can have only one Server Pool Master i can't graphically manage my cluster and my guest anymore.
When my primary server come back it doesn't resume it master role.
The only way to take back my console, is to restart my second server.
Note: I have a cluster URL to access to my console, however, But Oracle Express have no replication option then the BD of my second server is always empty.
What is you solution ?
How have you fix this situation ?
What the trick ?
Thanks
David

I have no answer on why your serverpool master does not resume its role when it restarts but I suggest you learn and use the XM commands on your oracle VM server to manage your guests.
To get a quick list of xm commands you can type "xm help" on your console. For more details and info run "man xm"
When your serverpool master comes back up, run the xm migrate command to move your guests from node2 to node1. When this completes, you can now reboot node2 with no issues and as I understand it, the serverpool master resumes its role after node2 is rebooted.

Similar Messages

Changing the Server Pool Master

I need to remove one of the servers of a three node cluster, but before I remove that node I need to remove it as the Server Pool Master and make another node the server pool master. When I go to edit a server, the check box is there, but it’s disable. How do I enable this or accomplish this switch?

It's actually pretty simple, if both the old and new Server Pool Master servers are online. First, make sure that you're not running any VMs on either the old (existing) or new (planned) Pool Master servers. You also need to be sure that all your VMs are either Powered Off or Running, i.e. none being migrated or suspended.
Then, here are the steps:
1. Go to the Servers tab and delete the NEW Pool Master from the pool.
2. Edit the EXISTING Pool Master and replace it's details with those of the NEW Pool Master, i.e. IP Address/Hostname. If the new pool master is also a utility server, you must ensure that you select that option in the Edit screen.
3. Click OK
4. Click the Server Pool tab and select your server pool. Then click the Restore button to update the status of all your servers/VMs in the pool.
5. Manually restart the agent on the NEW Pool Master. Login to the Dom0 of the new pool master and run:
# service ovs-agent stop
# service ovs-agent start
You now have the new Server Pool Master running and the old one has actually been removed from the pool. You can add it back as a normal server using the default process in the GUI.
I know there are guys inside Oracle working on an official whitepaper on Reassigning the Server Pool Master role in situations where the old/new servers are not online, so you should keep a look out on the Virtualization OTN site.

OVM-2020 Server pool master can not be set to maintenance mode

Hi Guys.
I have 2 server in ha mode
server 1 (10.99.99.161) with oracle Manager installed (server pool master, utility server, vm server)
Server 2 (10.99.99.161) without Oracle Manager installed (utility server, vmserver)
HA enable with server1 and server 2
10.99.99.165 is the virtual server pool master
1 San connected with server 1 and server 2.
Without problem I can connect on https://10.99.99.165:4443/OVS without problem.
All seems work, but :
1 - I I try to set in maintenance mode the server 1, the VM Manager say me:
OVM-2020 Server pool master (10.99.99.162) can not be set to maintenance mode, please use server pool master failover policy to change its role, then try again.
What I shuld do ?
2 - If I turn off the server 1 the server pool master is attribuite to server 2, but I'm unable to connect on https://10.99.99.165:4443/OVS webpage. It is correct, it is a bug, or I'm doing something wrong ? Must I upgrate something ?
Many Thanks
Luca

user8857532 wrote:
1 - I I try to set in maintenance mode the server 1, the VM Manager say me:
What I shuld do ?You need to live migrate all your guests off the current Master server, then issue service ovs-agent stop on the command-line. This will cause the pool mastery to switch to another server. Once that's done, you can start the agent agent.
2 - If I turn off the server 1 the server pool master is attribuite to server 2, but I'm unable to connect on https://10.99.99.165:4443/OVS webpage. It is correct, it is a bug, or I'm doing something wrong ? Must I upgrate something ?
You need to ensure the VM that's running the Oracle VM Manager software is still running. Live Migrate it first (preferred option) and flag it HA-enabled so that it is automatically restarted in the case of a server failure.

OVM-2008 The Server Pool Master has been registered with some other pool

I am getting this error, while creating my first ever server-pool.
Check prerequisites to create server pool (First-Pool) failed: (OVM-2008 The Server Pool Master (172.16.0.37) has been registered with some other pool, and can not register it again.)
Any idea what can be going wrong?
regards, Yora

The only time I've seen this is when the server really has been registered with another pool. You need to clean out the local ovs-agent database at */etc/ovs-agent/db* and restart the agent.

Non-server pool master in VM Manager

So I have two vm hosts that are not HA. The first is the Server Pool Master and everything is working well there. On the second host machine (the non-master) I put my template in the seed_pool folder and I can use xm to start the template and everything is fine, but I can not seem to get Oracle VM Manager to import the Virtual Machine Templates from the second host. Any ideas on what I did wrong?
Thanks,
Wesley

Hi Wesley,
all servers in a server pool (even if not enabled for HA) have shared repositories (this was changed with OVM 2.2, before that there was the concept of local repositories, but it does not seem to be existing anymore with OVS 2.2). These can be listed with
/opt/ovs-agent-latest/utils/repos.py -l
These are the folders the agents looks into the seed_pool to import the machines.
(note repos.py does not have an option for local repository).
If the repository is not listed here, where you put you seed_pool, then it is clear why you cannot import the template.
Regards
Sebastian

OVM 3.0.1 : Server Pool Master Error

am starting to see sudden errors on the console. All the vm's went offline including the server icon showing RED and in status Running (Error)
The event shows : Server Pool Master Missing Event / Server Pool Master Error.
Can anyone help me how to fix this issue? This just happened all of a sudden.

Ok I found the problem...although the locking from the rescan job seemed to disappear or timeout overnight.
It was actually the virtual machine that I had under that host that was in error. All I had to do was acknowledge the event that the virtual machine was powered off (selecting the vm, going to events, then acknowledge all). It's really annoying that VM 3 has this function because you are forced to acknowledge the event before you can continue with other activities.
It would be nice if there was maybe a way to only have that enabled for specific events. Another reason that this might have happened, was that I did not have HA enabled on the VM, so it wasn't migrated to my other server when I did the reboot.

Set Server Pool Master to Maintenance Mode

Hi all,
when trying to set a server pool master to maintenance, I got this: "OVM-2020 Server pool master (vm-13) can not be set to maintenance mode, please use server pool master failover policy to change its role, then try again."
I have dug through the help and docs, but can not find a method for dynamically reassigning the server pool master role.
This document: http://download.oracle.com/docs/cd/E15458_01/doc.22/e15441/server.htm#CCHIEBCE , just says "You must first reassign the Server Pool Master role to another server in the server pool.", without describing how.
This document: http://download.oracle.com/docs/cd/E15458_01/doc.22/e15441/site.htm#insertedID4 , states "You can also dynamically change the Oracle VM Server which acts as the Server Pool Master without causing any outages. See Section 3.4.1, "Editing Server Pool".", which just links back to the same section I am already in.
Now, I can just migrate all my machines and then stop ovs-agent, but I am not always the one managing this, and having a nice way through the web interface would be great.
Please just tell me I am blind and point me to the section in the manuals which tells me how to do it, or is this a feature which got dropped before release?
Thanks.

I have got the same problem, I want to switch the role of the server pool master to another server in the same pool. But after reading these answers, I have absolutly no clue how to do it!
We have a productiv environment, so any ideas like shutting down or stopping VMs are not acceptable.
So here is the situation:
"ovs#1" is hosting all VMs at the moment - Server type: Utility Server,Virtual Machine Server
"ovs#2" is free of any VMs - Server type: Server Pool Master,Utility Server,Virtual Machine Server
I want to switch the Server Pool Master from "ovs#2" to "ovs#1" without any downtime. How does this work? Thanks!
Specifications:
- Oracle VM Manager 2.2.0
- Oracle VM server release 2.2.1
Edited by: user11932329 on 15.09.2010 04:50

Server Pool Master fails and cannot fail over to another VM Server

Dear All,
Oracle VM 2.2.2
I have 2 VM Servers connect to Storage 6140 Array and on VM Manager I enable HA on the server pool, then on all virtual machines.
- VM Server 1 has role as Server Pool Master, Utility Server, Virtual Machine Server and has virtual machines running
- VM Server 2 has role as Utility Server, Virtual Machine Server and has virtual machines running.
I try to shutdown the VM Server 1 act as Server Pool Master but I don't see Server Pool Master fail over to another VM Server 2 and also status become to Unreachable both of 2 Servers.
Especially, All virtual machines cannot be accessible.
Please kindly give advice for this.
Thanks and regards,
Heng

Thanks Avi, I'll find and read that document. And thanks also for elaborating about the Utility Server.
After reading the followups to my original question, I tried to think of possible server "layouts" in a HA environment.
1) "N" servers in the pool, one of them is Pool Master, Utility Server AND VM Guests Server at the same time. Maybe this will be the preferred server for smaller, quicker VMs.
2) "N" servers in the pool, one is Pool Master AND Utility Server, but has no VM guests running on it
3) "N" servers in the pool, one is the Pool Master, another one is the Utility Server (none of them has VMs running on them), and finally a number of VM Guest servers
Let's take case 1. If the Pool Master & Utility server fails, given that it has VM guests running on it as well, I understand from your explanation that I'll be ANYWAY able to manually "live migrate" the guests somewhere else, using VM Manager. Is this correct?
If it's correct, then it's just a question of how much money I want to spend to have dedicated servers for different tasks, JUST FOR BETTER PERFORMANCES REASONS. Do you agree? And especially: do YOU have dedicated Pool Masters (just to figure out your "real" approach to the problem :-) )
I feel that I still miss something, the picture is not completely clear to me. The fact is, that I'm now testing on my new bladesystem, but for now I put up one single blade. Testing HA will be the next step. I was just trying to get a few things sorted out in advance, but there is still something that I'm missing, as I was saying...
Looking forward to your next reply, thanx again
Rob

Logic Pro + Node + Studio Setup, Need Advice, Please Help.

Hello everyone.
Ive just recently bought Logic 7.1 for my Quad PowerMac G5, and Im loving it!
Im going to build a studio in my garden (yup, im no pro). And I was thinking of purchasing a PowerBook for the studio and offload all processing and storage to my Quad machine (located in my study room in the house).
Im also considering purchasing another PowerMac G5 as for my server, and I would like to setup the server to also process Logic data and storage. Is this possible? Would i need to buy Xsan?
Another question, can I use more than 1 PowerMac G5 for Logic Pro? Also, how do I setup a Node? And in order for the Node to work do I need to purchase more Logic Pro USB Keys?
You may be wondering why am I going to purchase a PowerBook for the studio, well the answer is, i don't feel its safe to have a very expensive machine in studio, don't want it to get damaged by environment such as damage or theft, plus i spend alot of time studying.
One more thing, I have a PowerBook G4 and IMac G5, can i also setup Nodes on these machines aswell?
Any input would be brilliant. Also, your suggestions on studio setup would be also welcome.
Thanks in advance.
Haresh
Quad PowerMac G5 Mac OS X (10.4.4)

You won't need Xsan. I would test things out before getting too far into the idea. Your host should really be the strongest computer, all 3rd party, EXS, etc..plugins are not nodeable. You can use more than 1 node but as I see it your bottleneck would be the PBook. You can use regular file sharing just fine for this. You will need 1 USB key for each computer running Logic, the Node app requires no key. Therefore, you can run Logic on your book and 3 nodes with 1 key if you like. I would encourage you to use an iMac G5 as opposed to a Pbook if you really want to flex Logic muscle, which from the sounds of it, you intend to. iMac G5s are even cheaper than the Pbooks. Another thing to consider is some people have had no luck setting up Nodes-I've never had problems with it but there are some out there, maybe they're weird, maybe I am-but don't be 100% reliant on Nodes all the time until you have proven that in practice.

Server 2012 - Problem with multiple instances of RDS looking to the same license server. Server Pools requiring the wrong servers.

Hi All,
I have two domains, prod.local and test.local. On prod.local, I have a license server living on the connection broker. I have created a two way trust between the domains so that the RDS instance on test.local can use the license server on prod.local.
When I first setup the RDS instance on test.local and point it to use the license server on prod.local everything appears to work, all servers recognize the license server.
However when I log off and back on, and click the RDS dashboard in test.local, the Remote Desktop Services overview states I need to add every server from prod.local to the server pool. This is bothersome because the Remote Desktop Services on test.local
is still functioning, but I am no longer able to manage it. When I add the servers from prod.local to the server pool I am displayed the setup for prod.local instead of test.local
Is there a way around this? Can I use the same RDS license server key multiple times?
tldr: Using a different domain's license server causes my RDS instance in Server Manager to display the wrong RDS farm.

Hi Nathan,
Thank you for posting in Windows Server forum.
Please check the required ports must be opened on firewall.
- To issue RDS Per User CALs to users in other domains, there must be a two-way trust between the domains, and the license server must be a member of the Terminal Server License Servers group in those domains.
- To restrict the issuance of RDS CALs, you can add RDS Host Servers into Terminal Server Computers group on RDS Licensing servers.
- Configure RDS licensing server on all RDS Host Servers in each domain/forest. You can do it through RDS host configuration snap-in or through a group policy.
- Add administrators group of each domain/forest in the local administrators of RDS licensing server. This way, you’ll not get a prompt to enter your credentials when you’ll open RDS host configuration snap-ins in trusted domains/forests.
More information:
Best practices for setting up Remote Desktop Licensing (Terminal Server Licensing) across Active Directory Domains/Forests or Workgroup
http://support.microsoft.com/kb/2473823
Hope it helps!
Thanks.
Dharmesh Solanki

Creation of server pool in dhcp environment fails

Hi,
I am new to Oracle VM, please help me with my aim of setting up an "as simple to configure as possible" virtual machine. I chose the "all-in-one" approach, mentioned in the documentation, where server-pool-master, utility-server, vm-server are all on the same machine.
Now, I have a problem, which consists in the dhcp based network environment in my workgroup. How can work with fixed IP's in order to configure my server pool?
Can the server pool virtual IP be the same, as the IP of
Server Pool Master
Utility Server
Virtual Machine Server
Many thanks in advance

user8768362 wrote:
Do i understand you correctly, that it is possible to configure the Oracle VM server and the pool and the utility server and the vm guests on one machine?Yes.
Do i understand you correctly, that i should assign the pool the very same physical IP of the Oracle VM server host?In the case of a single server, the pool doesn't need an IP address.

Full server pool crashes when adding new iSCSI server

Hi,
we have a Pool Group with 2 machines (1 Server Pool Master + Server Virtual Machine + Utility Server and another 1 Server Virtual Machine). Both have a iSCSI Shared Disk which builds /OVS partition.
This is working, we can use High Availability, Migrate guests etc.
But when adding new Server Virtual Machines to the pool (with guests running), current machines in the Pools get restarted.
My question is, can server virtual machines be "hot added" to the pool group while guests are running?
Thanks and regards,
Marc

Hi,
hosts file seems to be correct.
Messages logs during the crash time are the following:
Node vmserver15 = Server Pool Master, Utility Master and Server Virtual Machine
Dec 10 12:40:02 vmserver15 kernel: vlan500: port 3(vif6.0) entering disabled state
Dec 10 12:40:02 vmserver15 kernel: device vif6.0 left promiscuous mode
Dec 10 12:40:02 vmserver15 kernel: type=1700 audit(1260445202.434:16): dev=vif6.0 prom=0 old_prom=256 auid=4294967295 ses=4294967295
Dec 10 12:40:02 vmserver15 kernel: vlan500: port 3(vif6.0) entering disabled state
Dec 10 12:40:02 vmserver15 kernel: loop10: dropped 10114 extents
Dec 10 12:40:03 vmserver15 udhcpc: udhcp client (v0.9.8) started
Dec 10 12:40:03 vmserver15 udhcpc: Lease of 193.109.175.25 obtained, lease time 172800
Dec 10 12:40:04 vmserver15 kernel: device vif7.0 entered promiscuous mode
Dec 10 12:40:04 vmserver15 kernel: type=1700 audit(1260445204.774:17): dev=vif7.0 prom=256 old_prom=0 auid=4294967295 ses=4294967295
Dec 10 12:40:04 vmserver15 kernel: vlan500: topology change detected, propagating
Dec 10 12:40:04 vmserver15 kernel: vlan500: port 3(vif7.0) entering forwarding state
Dec 10 12:40:05 vmserver15 kernel: loop10: fast redirect
Dec 10 12:40:06 vmserver15 kernel: blkback: ring-ref 770, event-channel 9, protocol 1 (x86_32-abi)
Dec 10 12:53:35 vmserver15 kernel: o2net: no longer connected to node vmserver10.pic.es (num 1) at 193.109.174.110:7777
Dec 10 12:53:36 vmserver15 kernel: (4989,0):o2hb_do_disk_heartbeat:776 ERROR: Device "sdb1": another node is heartbeating in our slot!
Dec 10 12:53:37 vmserver15 kernel: o2net: accepted connection from node vmserver10.pic.es (num 1) at 193.109.174.110:7777
Dec 10 12:53:38 vmserver15 kernel: (4989,0):o2hb_do_disk_heartbeat:776 ERROR: Device "sdb1": another node is heartbeating in our slot!
Dec 10 12:53:50 vmserver15 last message repeated 6 times
Dec 10 12:53:51 vmserver15 kernel: o2net: no longer connected to node vmserver10.pic.es (num 1) at 193.109.174.110:7777
Dec 10 12:53:52 vmserver15 kernel: (4989,0):o2hb_do_disk_heartbeat:776 ERROR: Device "sdb1": another node is heartbeating in our slot!
Dec 10 12:53:53 vmserver15 kernel: o2net: accepted connection from node vmserver10.pic.es (num 1) at 193.109.174.110:7777
Dec 10 12:53:53 vmserver15 kernel: (5638,0):dlm_send_remote_convert_request:393 ERROR: dlm status = DLM_IVLOCKID+
Dec 10 12:53:53 vmserver15 kernel: (5638,0):dlmconvert_remote:327 ERROR: dlm status = DLM_IVLOCKID+
Dec 10 12:53:53 vmserver15 kernel: (5638,0):ocfs2_cluster_lock:1206 ERROR: DLM error DLM_IVLOCKID while calling dlmlock on resource M000000000000000001050c00000000: bad lockid+
Dec 10 12:53:53 vmserver15 kernel: (5638,0):ocfs2_inode_lock_full:2064 ERROR: status = -22+
Dec 10 12:53:53 vmserver15 kernel: (5638,0):ocfs2_inode_lock_atime:2193 ERROR: status = -22+
Dec 10 12:53:53 vmserver15 kernel: (5638,0):__ocfs2_file_aio_read:2434 ERROR: status = -22+
Dec 10 12:53:53 vmserver15 kernel: (5638,0):dlm_send_remote_convert_request:393 ERROR: dlm status = DLM_IVLOCKID+
Dec 10 12:53:53 vmserver15 kernel: (5638,0):dlmconvert_remote:327 ERROR: dlm status = DLM_IVLOCKID+
Dec 10 12:53:53 vmserver15 kernel: (5638,0):ocfs2_cluster_lock:1206 ERROR: DLM error DLM_IVLOCKID while calling dlmlock on resource M000000000000000001050c00000000: bad lockid+
Dec 10 12:53:53 vmserver15 kernel: (5638,0):ocfs2_inode_lock_full:2064 ERROR: status = -22+
Dec 10 12:53:53 vmserver15 kernel: (5638,0):ocfs2_write_begin:1845 ERROR: status = -22+
Dec 10 12:53:53 vmserver15 kernel: (5638,0):ocfs2_file_buffered_write:2016 ERROR: status = -22+
Dec 10 12:53:53 vmserver15 kernel: (5638,0):__ocfs2_file_aio_write:2173 ERROR: status = -22+
Dec 10 12:53:58 vmserver15 kernel: (7923,0):dlm_send_remote_convert_request:393 ERROR: dlm status = DLM_IVLOCKID+
Dec 10 12:53:58 vmserver15 kernel: (7923,0):dlmconvert_remote:327 ERROR: dlm status = DLM_IVLOCKID+
Dec 10 12:53:58 vmserver15 kernel: (7923,0):ocfs2_cluster_lock:1206 ERROR: DLM error DLM_IVLOCKID while calling dlmlock on resource M000000000000000000020744c1370e: bad lockid+
Dec 10 12:53:58 vmserver15 kernel: (7923,0):ocfs2_inode_lock_full:2064 ERROR: status = -22+
Dec 10 12:53:58 vmserver15 kernel: (7923,0):ocfs2_reserve_suballoc_bits:449 ERROR: status = -22+
Dec 10 12:53:58 vmserver15 kernel: (7923,0):ocfs2_reserve_cluster_bitmap_bits:682 ERROR: status = -22+
Dec 10 12:53:58 vmserver15 kernel: (7923,0):ocfs2_local_alloc_reserve_for_window:930 ERROR: status = -22+
Dec 10 12:53:58 vmserver15 kernel: (7923,0):ocfs2_local_alloc_slide_window:1063 ERROR: status = -22+
Dec 10 12:53:58 vmserver15 kernel: (7923,0):ocfs2_reserve_local_alloc_bits:537 ERROR: status = -22+
Dec 10 12:53:58 vmserver15 kernel: (7923,0):__ocfs2_reserve_clusters:725 ERROR: status = -22+
Dec 10 12:53:58 vmserver15 kernel: (7923,0):ocfs2_lock_allocators:677 ERROR: status = -22+
Dec 10 12:53:58 vmserver15 kernel: (7923,0):ocfs2_write_begin_nolock:1751 ERROR: status = -22+
Dec 10 12:53:58 vmserver15 kernel: (7923,0):ocfs2_write_begin:1861 ERROR: status = -22+
Dec 10 12:53:58 vmserver15 kernel: (7923,0):ocfs2_file_buffered_write:2016 ERROR: status = -22+
Dec 10 12:53:58 vmserver15 kernel: (7923,0):__ocfs2_file_aio_write:2173 ERROR: status = -22+
Dec 10 12:53:58 vmserver15 kernel: loop: Write error at byte offset 37644512256, length 4096.+
. <the above bold and cursive text is repetead few times>
Dec 10 12:58:29 vmserver15 kernel: (5638,3):dlmconvert_remote:327 ERROR: dlm status = DLM_IVLOCKID
Dec 10 12:58:29 vmserver15 kernel: (5638,3):ocfs2_cluster_lock:1206 ERROR: DLM error DLM_IVLOCKID while calling dlmlock on resource M000000000000000001050c00000000: bad lockid
Dec 10 12:58:29 vmserver15 kernel: (5638,3):ocfs2_inode_lock_full:2064 ERROR: status = -22
Dec 10 12:58:29 vmserver15 kernel: (5638,3):ocfs2_write_begin:1845 ERROR: status = -22
Dec 10 12:58:29 vmserver15 kernel: (5638,3):ocfs2_file_buffered_write:2016 ERROR: status = -22
Dec 10 12:58:29 vmserver15 kernel: (5638,3):__ocfs2_file_aio_write:2173 ERROR: status = -22
Dec 10 13:01:16 vmserver15 syslogd 1.4.1: restart.
Node vmserver10: Virtual Server Machine
Dec 10 12:53:35 vmserver10 kernel: o2net: no longer connected to node vmserver15.pic.es (num 0) at 193.109.174.115:7777
Dec 10 12:53:35 vmserver10 kernel: (5029,0):ocfs2_dlm_eviction_cb:98 device (8,17): dlm has evicted node 0
Dec 10 12:53:35 vmserver10 kernel: (20996,0):dlm_get_lock_resource:844 E3FE9E5767CA457FA697980EB637E93B:M000000000000000000022044c1370e: at least one node (0) to recover before lock mastery can begin
Dec 10 12:53:36 vmserver10 kernel: (5344,4):dlm_get_lock_resource:844 E3FE9E5767CA457FA697980EB637E93B:$RECOVERY: at least one node (0) to recover before lock mastery can begin
Dec 10 12:53:36 vmserver10 kernel: (5344,4):dlm_get_lock_resource:878 E3FE9E5767CA457FA697980EB637E93B: recovery map is not empty, but must master $RECOVERY lock now
Dec 10 12:53:36 vmserver10 kernel: (5344,4):dlm_do_recovery:524 (5344) Node 1 is the Recovery Master for the Dead Node 0 for Domain E3FE9E5767CA457FA697980EB637E93B
Dec 10 12:53:36 vmserver10 kernel: (20996,0):ocfs2_replay_journal:1183 Recovering node 0 from slot 0 on device (8,17)
Dec 10 12:53:37 vmserver10 kernel: o2net: connected to node vmserver15.pic.es (num 0) at 193.109.174.115:7777
Dec 10 12:53:38 vmserver10 kernel: (8672,1):dlm_get_lock_resource:844 ovm:$RECOVERY: at least one node (0) to recover before lock mastery can begin
Dec 10 12:53:38 vmserver10 kernel: (8672,1):dlm_get_lock_resource:878 ovm: recovery map is not empty, but must master $RECOVERY lock now
Dec 10 12:53:38 vmserver10 kernel: (8672,1):dlm_do_recovery:524 (8672) Node 1 is the Recovery Master for the Dead Node 0 for Domain ovm
Dec 10 12:53:40 vmserver10 kernel: kjournald starting. Commit interval 5 seconds
Dec 10 12:53:51 vmserver10 kernel: o2net: no longer connected to node vmserver15.pic.es (num 0) at 193.109.174.115:7777
Dec 10 12:53:53 vmserver10 kernel: o2net: connected to node vmserver15.pic.es (num 0) at 193.109.174.115:7777
Dec 10 12:53:53 vmserver10 kernel: (3761,0):dlm_convert_lock_handler:489 ERROR: did not find lock to convert on grant queue! cookie=0:92+
Dec 10 12:53:53 vmserver10 kernel: lockres: M000000000000000001050c0000000, owner=1, state=0+
Dec 10 12:53:53 vmserver10 kernel:   last used: 0, refcnt: 3, on purge list: no+
Dec 10 12:53:53 vmserver10 kernel:   on dirty list: no, on reco list: no, migrating pending: no+
Dec 10 12:53:53 vmserver10 kernel:   inflight locks: 0, asts reserved: 0+
*Dec 10 12:53:53 vmserver10 kernel:   refmap nodes: [ ], inflight=0*+
Dec 10 12:53:53 vmserver10 kernel:   granted queue:+
Dec 10 12:53:53 vmserver10 kernel:     type=5, conv=-1, node=1, cookie=1:243, ref=2, ast=(empty=y,pend=n), bast=(empty=y,pend=n), pending=(conv=n,lock=n,cancel=n,unlock=n)+
Dec 10 12:53:53 vmserver10 kernel:   converting queue:+
Dec 10 12:53:53 vmserver10 kernel:   blocked queue:+
. <the above bold and cursive text is repetead few times>
Dec 10 12:57:18 vmserver10 modprobe: FATAL: Module ocfs2_stackglue not found.
Dec 10 12:57:18 vmserver10 kernel: (3761,0):dlm_convert_lock_handler:489 ERROR: did not find lock to convert on grant queue! cookie=0:92+
Dec 10 12:57:18 vmserver10 kernel: lockres: M000000000000000001050c0000000, owner=1, state=0+
Dec 10 12:57:18 vmserver10 kernel:   last used: 0, refcnt: 3, on purge list: no+
Dec 10 12:57:18 vmserver10 kernel:   on dirty list: no, on reco list: no, migrating pending: no+
Dec 10 12:57:18 vmserver10 kernel:   inflight locks: 0, asts reserved: 0+
*Dec 10 12:57:18 vmserver10 kernel:   refmap nodes: [ ], inflight=0*+
Dec 10 12:57:18 vmserver10 kernel:   granted queue:+
Dec 10 12:57:18 vmserver10 kernel:     type=5, conv=-1, node=1, cookie=1:243, ref=2, ast=(empty=y,pend=n), bast=(empty=y,pend=n), pending=(conv=n,lock=n,cancel=n,unlock=n)+
Dec 10 12:57:18 vmserver10 kernel:   converting queue:+
Dec 10 12:57:18 vmserver10 kernel:   blocked queue:+
. <the above bold and cursive text is repetead few times>
Dec 10 12:58:32 vmserver10 kernel: (3761,0):dlm_unlock_lock_handler:511 ERROR: failed to find lock to unlock! cookie=0:1849
Dec 10 12:58:33 vmserver10 modprobe: FATAL: Module ocfs2_stackglue not found.
Dec 10 12:59:02 vmserver10 kernel: o2net: connection to node vmserver15.pic.es (num 0) at 193.109.174.115:7777 has been idle for 30.0 seconds, shutting it down.
Dec 10 12:59:02 vmserver10 kernel: (0,0):o2net_idle_timer:1503 here are some times that might help debug the situation: (tmr 1260446312.830107 now 1260446342.828243 dr 1260446312.830066 adv 1260446312.830319:1260446312.830320 func (b9f5fd13:506) 1260446312.830109:1260446312.830303)
Dec 10 12:59:02 vmserver10 kernel: o2net: no longer connected to node vmserver15.pic.es (num 0) at 193.109.174.115:7777
Dec 10 12:59:32 vmserver10 kernel: (3761,0):o2net_connect_expired:1664 ERROR: no connection established with node 0 after 30.0 seconds, giving up and returning errors.
Dec 10 13:01:42 vmserver10 kernel: o2net: connected to node vmserver15.pic.es (num 0) at 193.109.174.115:7777
Dec 10 13:01:45 vmserver10 kernel: ocfs2_dlm: Node 0 joins domain E3FE9E5767CA457FA697980EB637E93B
Dec 10 13:01:45 vmserver10 kernel: ocfs2_dlm: Nodes in domain ("E3FE9E5767CA457FA697980EB637E93B"): 0 1
Dec 10 13:01:51 vmserver10 kernel: o2net: accepted connection from node vmserver16.pic.es (num 2) at 193.109.174.116:7777
Dec 10 13:01:56 vmserver10 kernel: ocfs2_dlm: Node 2 joins domain E3FE9E5767CA457FA697980EB637E93B
Dec 10 13:01:56 vmserver10 kernel: ocfs2_dlm: Nodes in domain ("E3FE9E5767CA457FA697980EB637E93B"): 0 1 2
Dec 10 13:09:05 vmserver10 modprobe: FATAL: Module ocfs2_stackglue not found.
Dec 10 13:16:45 vmserver10 kernel: o2net: connection to node vmserver16.pic.es (num 2) at 193.109.174.116:7777 has been idle for 30.0 seconds, shutting it down.
Dec 10 13:16:45 vmserver10 kernel: (0,0):o2net_idle_timer:1503 here are some times that might help debug the situation: (tmr 1260447375.655426 now 1260447405.655712 dr 1260447375.655413 adv 1260447375.655427:1260447375.655427 func (b9f5fd13:503) 1260446516.75600:1260446516.75608)
Dec 10 13:16:45 vmserver10 kernel: o2net: no longer connected to node vmserver16.pic.es (num 2) at 193.109.174.116:7777
Dec 10 13:17:15 vmserver10 kernel: (3761,0):o2net_connect_expired:1664 ERROR: no connection established with node 2 after 30.0 seconds, giving up and returning errors.
Dec 10 13:17:19 vmserver10 kernel: (5029,0):ocfs2_dlm_eviction_cb:98 device (8,17): dlm has evicted node 2
Dec 10 13:17:20 vmserver10 kernel: (3761,0):ocfs2_dlm_eviction_cb:98 device (8,17): dlm has evicted node 2
Dec 10 13:29:05 vmserver10 kernel: o2net: no longer connected to node vmserver15.pic.es (num 0) at 193.109.174.115:7777
Dec 10 13:29:05 vmserver10 kernel: (5029,0):ocfs2_dlm_eviction_cb:98 device (8,17): dlm has evicted node 0
Dec 10 13:29:06 vmserver10 kernel: (5344,4):dlm_get_lock_resource:844 E3FE9E5767CA457FA697980EB637E93B:$RECOVERY: at least one node (0) to recover before lock mastery can begin
Dec 10 13:29:06 vmserver10 kernel: (5344,4):dlm_get_lock_resource:878 E3FE9E5767CA457FA697980EB637E93B: recovery map is not empty, but must master $RECOVERY lock now
Dec 10 13:29:06 vmserver10 kernel: (5344,4):dlm_do_recovery:524 (5344) Node 1 is the Recovery Master for the Dead Node 0 for Domain E3FE9E5767CA457FA697980EB637E93B
Dec 10 13:29:06 vmserver10 kernel: (28412,0):ocfs2_replay_journal:1183 Recovering node 0 from slot 0 on device (8,17)
Dec 10 13:29:09 vmserver10 kernel: kjournald starting. Commit interval 5 seconds
Dec 10 13:29:16 vmserver10 kernel: o2net: accepted connection from node vmserver16.pic.es (num 2) at 193.109.174.116:7777
Dec 10 13:29:20 vmserver10 kernel: ocfs2_dlm: Node 2 joins domain E3FE9E5767CA457FA697980EB637E93B
Dec 10 13:29:20 vmserver10 kernel: ocfs2_dlm: Nodes in domain ("E3FE9E5767CA457FA697980EB637E93B"): 1 2
Dec 10 13:32:08 vmserver10 kernel: o2net: connected to node vmserver15.pic.es (num 0) at 193.109.174.115:7777
Dec 10 13:32:11 vmserver10 kernel: ocfs2_dlm: Node 0 joins domain E3FE9E5767CA457FA697980EB637E93B
Dec 10 13:32:11 vmserver10 kernel: ocfs2_dlm: Nodes in domain ("E3FE9E5767CA457FA697980EB637E93B"): 0 1 2
Dec 10 13:36:10 vmserver10 shutdown[28681]: shutting down for system reboot+
I will investigate what seems to be going on and post it here.
Thanks for your help.
Edited by: Marc Caubet on 11-Dec-2009 02:05
Edited by: Marc Caubet on 11-Dec-2009 02:07

Added server not listed in Server pool (not even on the list of servers)

Hello VM Gurus,
I'm playing with Oracle VM and I've encountered problem I can not solve of find existing solution.
I'm playing with Oracle VM version 2.1.5.
I have 3 psychical machines (let say A,B,C). On every machine I have installed WMWare workstation. I've created all together 5 VMWare virtual machines.
On Physical machine A I have 1 VMWare virtual machine named A1. This is used as Virtual Machine Server.
On Physical machine B I have 2 VMWare virtual machine named B1 and B2. B1 is used as Oracle VM Manager. B2 is used as Virtual Machine Server (also Server Pool Master and Utility Server)
On Physical machine C I have 2 VMWare virtual machine named C1 and C2. Both are used as Virtual Machine Serves.
1. I've Created Pool of servers (HA set up) and first added server B2.
2. After that I've added to pool A1 without any problems
3. After that I've added to pool C1 without any problems.
3. After that I've tried to add to pool C2. Process (wizard "Add server") finishes without any problems. But server does not appear on the list of servers in pool. It does not exist On a list of servers neither. If I try to add server again I got error OVS-2005 saying that this server is part of another Pool.
If I stop agent on C2 , delete folder /etc/ovs-agent/db and start agent, I can again add server, but results are same (it does not appear on list of servers. Another try to add server results in OVS-2005 error).
If done also this process:
1. I've stopped agent on C2, deleted folder /etc/ovs-agent/db and starte agent
2. I've removed C1 from pool
3. I've added C2 to pool without any problems
4. I've tried to add c1 to Pool. Same problems as in the past with C2 (not appearing on the list of servers).
I went though agent logs, VM manager logs. I ca not spot any problems. I've connected to database with SQL plus and there was no evidence of failed server there (so data is really not written to repository).
I run out of ideas. What could be wrong? where to look whyt is problem. How to diagnose problem. I believe that Oracle VM somehow treat this two VMWare servers as one, but I can not prove this.
Somebody else had this problem? Any hint what to diagnose further? I understand my configuration is a bit stupid, but I'm trying to get familiar with Oracle VM.
Any hint will be appreciated.
Regards
Darko

user465397 wrote:
I have 3 psychical machines (let say A,B,C). On every machine I have installed WMWare workstation. I've created all together 5 VMWare virtual machines.This is a really bad idea, btw -- if you have three physical machines, you should install Oracle VM directly on each machine. Or use one machine as the NFS server and Oracle VM Manager and the other two as Oracle VM Servers. Trying to run a Type 1 Hypervisor inside a Type 2 hypervisor is fraught with peril. I'm surprised you actually got this far, to be honest. :)

Server pool design

If a server pool with a single x86 server plays the role of Server pool master, utility server and also VM server, how are the resources divided for these three functions to be performed simultaneously. For instance, VM deployment being done in another pool, along with file transmission for a VM created in a domU of the server. How are the CPU cycles divided and any hard partitioning of the CPUs and hard disk or memory for the 3 functions?

I am sorry you feel uncomfortable about the license terms. Being a software company owner I know the difficulty of figuring out the license restrictions of various software components. However that is exactly I provide ExpressionFlow example code with creative commons attribution license. The creative commons licenses do not restrict where the example code can be used and the terms are rather widely known. Shoud example code have no license terms, you would not know how you would be allowed to use or share the code. Now you have it black-on-white.
Human-readable summary of the Creative Commons Attribution 3.0 Unported license:
You are free:
to Share — to copy, distribute and transmit the work
to Remix — to adapt the work
Under the following conditions:
Attribution —
You
must attribute the work in the manner specified by the author or
licensor (but not in any way that suggests that they endorse you or
your use of the work).
With the understanding that:
Waiver
Any of the above conditions can be waived if you get permission from the copyright holder.
Other Rights
In no way are any of the following rights affected by the license:
Your fair dealing or fair use rights;
The author's moral rights;
Rights other persons may have either in the work itself or in how the work is used, such as publicity or privacy rights.
Notice
— For any reuse or distribution, you must make clear to others the
license terms of this work. The best way to do this is with a link to
this web page.
Tomi Maila

Pool Master server failover issue in Oracle VM 2.2.1

Hello All , We are new to oracle VM world. Sorry about detailed explanation.
Our current configuration is, server1-poolmaster/utility/VM server & Server2-utility/VM server
We have guest-VM running on both servers, and serverpool-VIP is configured properly. Below is our OVS-version.
#rpm -qa | grep -i ovs
oracle-logos-4.9.17-7ovs
enterprise-linux-ovs-5-1.0
ovs-release-2.2-1.0
ovs-utils-1.0-34
kernel-ovs-2.6.18-128.2.1.4.25.el5
ovs-agent-2.3-42
When we tested HA failover(shutting down server1), it work fine as expected. Pool master moved from server1 to server2, and guest VM restarted on server-2(which was running on server1 earlier).
Now-- Pool master is server2.
When we shutdown server2 now, pool master is not migrated to server1 & the guest-VMs(running on server 2) all went to power-off mode & serverpool is "inactive status'.
Found below error in server1's /var/log/messages.. It seems like some 'dead-lock situation, and the serverpool-VIP is not moved from server2 to server1, until server2 came up online". Why is it so? The expected result should be "pool-master" & serverpool-VIP should moved to server1, but it didnt.
Anyone experienced this? Any help/ input is appreciated.
log file from server1's /var/log/ovs-agent/ovs_remaster.log
2011-01-14 01:47:56 INFO=> run(): release_master_dlm_lock ...
2011-01-14 01:48:02 INFO=> run(): release_master_dlm_lock ...
2011-01-14 01:48:08 INFO=> run(): release_master_dlm_lock ...
2011-01-14 01:48:14 INFO=> run(): release_master_dlm_lock ...
2011-01-14 01:48:20 INFO=> run(): release_master_dlm_lock ...
2011-01-14 01:48:26 INFO=> run(): release_master_dlm_lock ...
***** At this time its waiting to release the server pool-VIP on server 2
*** Once server2 came online, serverpool-VIP released and taken by server1***
2011-01-14 01:54:11 INFO=> cluster_get_next_master: => {"status": "SUCC", "value": "10.24.60.41"}
2011-01-14 01:54:11 INFO=> run(): cluster_get_next_master: => {"status": "SUCC", "value": "10.24.60.41"}
2011-01-14 01:54:13 INFO=> run(): clusterm_setup_master_env: => {"status": "SUCC"}
2011-01-14 01:54:20 INFO=> run(): i am the new master. vip=10.24.60.45
truncated logs from server1's /var/log/messages
Jan 14 01:46:40 fwblade1 kernel: ocfs2_dlm: Node 1 leaves domain 70FFE4CF84634F5DB61BEA66E04693A7
Jan 14 01:46:40 fwblade1 kernel: ocfs2_dlm: Nodes in domain ("70FFE4CF84634F5DB61BEA66E04693A7"): 0
Jan 14 01:47:59 fwblade1 kernel: ocfs2_dlm: Node 1 leaves domain ovm
Jan 14 01:47:59 fwblade1 kernel: ocfs2_dlm: Nodes in domain ("ovm"): 0
Jan 14 01:48:55 fwblade1 kernel: o2net: connection to node fwblade2.wg.kns.com (num 1) at 10.24.60.42:7777 has been idle for 30.0 second
s, shutting it down.
Jan 14 01:48:55 fwblade1 kernel: (0,0):o2net_idle_timer:1503 here are some times that might help debug the situation: (tmr 1294987705.66
5702 now 1294987735.663612 dr 1294987705.665695 adv 1294987705.665724:1294987705.665725 func (53ed487f:505) 1294987705.665424:1294987705
.665428)
Jan 14 01:48:55 fwblade1 kernel: o2net: no longer connected to node fwblade2.wg.kns.com (num 1) at 10.24.60.42:7777
Jan 14 01:48:55 fwblade1 kernel: (5190,0):dlm_send_remote_lock_request:333 ERROR: status = -112
Jan 14 01:48:55 fwblade1 kernel: (5186,2):dlm_send_remote_lock_request:333 ERROR: status = -107
Jan 14 01:48:55 fwblade1 kernel: (5190,0):dlm_send_remote_lock_request:333 ERROR: status = -107
Jan 14 01:48:55 fwblade1 kernel: (5186,2):dlm_send_remote_lock_request:333 ERROR: status = -107
** the above messages is repeated till server1 came online ***
Jan 14 01:48:57 fwblade1 kernel: (4694,2):dlm_drop_lockres_ref:2211 ERROR: status = -107
Jan 14 01:48:57 fwblade1 kernel: (4694,2):dlm_purge_lockres:206 ERROR: status = -107
Jan 14 01:48:57 fwblade1 kernel: (4694,2):dlm_drop_lockres_ref:2211 ERROR: status = -107
Jan 14 01:48:57 fwblade1 kernel: (4694,2):dlm_purge_lockres:206 ERROR: status = -107
Jan 14 01:49:30 fwblade1 kernel: (4651,0):ocfs2_dlm_eviction_cb:98 device (253,0): dlm has evicted node 1
Jan 14 01:49:30 fwblade1 kernel: (32373,0):dlm_get_lock_resource:844 78CD07B6D4C34CEAB756BF56E6D9C561:M00000000000000000002182aa14db5: a
t least one node (1) to recover before lock mastery can begin
** Still no sign of server1 taking up the serverpool-VIP, all the guest-VM are still power-off status***
Jan 14 01:49:35 fwblade1 kernel: (4695,0):dlm_get_lock_resource:844 78CD07B6D4C34CEAB756BF56E6D9C561:$RECOVERY: at least one node (1) to
recover before lock mastery can begin
Jan 14 01:49:35 fwblade1 kernel: (4695,0):dlm_get_lock_resource:878 78CD07B6D4C34CEAB756BF56E6D9C561: recovery map is not empty, but mus
t master $RECOVERY lock now
Jan 14 01:49:35 fwblade1 kernel: (4695,0):dlm_do_recovery:524 (4695) Node 0 is the Recovery Master for the Dead Node 1 for Domain 78CD07
B6D4C34CEAB756BF56E6D9C561
** still no luck.. all guest VM are down***
Jan 14 01:53:59 fwblade1 kernel: (5186,1):dlm_send_remote_lock_request:333 ERROR: status = -92
Jan 14 01:53:59 fwblade1 kernel: (5190,10):dlm_send_remote_lock_request:333 ERROR: status = -92
Jan 14 01:53:59 fwblade1 kernel: (5186,1):dlm_send_remote_lock_request:333 ERROR: status = -92
Jan 14 01:53:59 fwblade1 kernel: (5190,10):dlm_send_remote_lock_request:333 ERROR: status = -92
Jan 14 01:53:59 fwblade1 kernel: (5186,1):dlm_send_remote_lock_request:333 ERROR: status = -92
Jan 14 01:54:00 fwblade1 kernel: (5190,10):dlm_send_remote_lock_request:333 ERROR: status = -92
Jan 14 01:54:00 fwblade1 kernel: (5186,1):dlm_send_remote_lock_request:333 ERROR: status = -92
Jan 14 01:54:00 fwblade1 kernel: ocfs2_dlm: Node 1 joins domain 78CD07B6D4C34CEAB756BF56E6D9C561
Jan 14 01:54:00 fwblade1 kernel: ocfs2_dlm: Nodes in domain ("78CD07B6D4C34CEAB756BF56E6D9C561"): 0 1
Jan 14 01:54:00 fwblade1 kernel: (5190,10):dlmlock_remote:269 ERROR: dlm status = DLM_IVLOCKID
Jan 14 01:54:00 fwblade1 kernel: (5190,10):dlmlock:747 ERROR: dlm status = DLM_IVLOCKID
Jan 14 01:54:00 fwblade1 kernel: (5190,10):ocfs2_lock_create:997 ERROR: DLM error DLM_IVLOCKID while calling dlmlock on resource F000000
000000000000a50dd279960c: bad lockid
Jan 14 01:54:00 fwblade1 kernel: (5190,10):ocfs2_file_lock:1584 ERROR: status = -22
Jan 14 01:54:00 fwblade1 kernel: (5190,10):ocfs2_do_flock:79 ERROR: status = -22
Jan 14 01:54:00 fwblade1 kernel: (5186,1):dlmlock_remote:269 ERROR: dlm status = DLM_IVLOCKID
Jan 14 01:54:00 fwblade1 kernel: (5186,1):dlmlock:747 ERROR: dlm status = DLM_IVLOCKID
Jan 14 01:54:00 fwblade1 kernel: (5186,1):ocfs2_lock_create:997 ERROR: DLM error DLM_IVLOCKID while calling dlmlock on resource F0000000
00000000000a50dd279960c: bad lockid
Jan 14 01:54:00 fwblade1 kernel: (5186,1):ocfs2_file_lock:1584 ERROR: status = -22
Jan 14 01:54:00 fwblade1 kernel: (5186,1):ocfs2_do_flock:79 ERROR: status = -22
Jan 14 01:54:05 fwblade1 kernel: ocfs2_dlm: Node 1 joins domain 70FFE4CF84634F5DB61BEA66E04693A7
Jan 14 01:54:05 fwblade1 kernel: ocfs2_dlm: Nodes in domain ("70FFE4CF84634F5DB61BEA66E04693A7"): 0 1
** Now server2 came online(old pool-master) and server-pool-VIP is moved to server1.** All guest-VM are restarted on SERVER2 itself.
Thanks
Prakash

You might be running into a OCFS2 bug. Check the bug list for bug 1099
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1099
also related to this subject might be bug 1095 and 1080. You might want to check with the OCFS2 guys at Oracle and participate in resolving this bug. Not sure this is the case however I think this is a good starting point.
Please keep us posted.
Regards,
Johan Louwers.

Server Pool Master - 2 node HA setup - Need suggestion

Similar Messages

Maybe you are looking for