RAC 9i to 11g 3 node cluster upgradation.

Hi,
I have been given the requirement to migrate a 9i RAC 3 node database( 9i cluster) to 11g R2 RAC database iam looking for a document or steps to do this activity. any help on this is highly appreciated.
should I upgrade the cluster first and then upgrade each node one by one or I should upgrade the cluster and upgrade all the home in one go ?
Thanks,
Satish Abraham.J

hi,
you may want to take look this first:
Upgrade 9i to 11gR2

Similar Messages

  • Upgrade oracle 9i RAC to Oracle 11g RAC

    Hi all,
    db version:oracle 9.2.0.5 RAC
    OS:Linux
    My project team and client is planning to upgrade oracle 9i RAC to Oracle 11g RAC.
    please suggest any MOS notes any links.
    thanks,
    Visu.
    Edited by: visu996253 on Apr 8, 2013 5:30 AM

    db version:oracle 9.2.0.5 RAC
    OS:Linux
    My project team and client is planning to upgrade oracle 9i RAC to Oracle 11g RAC.
    please suggest any MOS notes any links.Master Note For Oracle Database Upgrades and Migrations [ID 1152016.1]
    http://www.oracle.com/technetwork/products/upgrade/index-088044.html
    785351.1 - Oracle 11g Upgrade Companion
    http://download.oracle.com/docs/cd/E11882_01/server.112/e17222.pdf - Oracle® Database Upgrade Guide 11g Release 2 (11.2) E10819-02
    Edited by: KR10822864 on Apr 8, 2013 5:52 AM
    added more info.

  • Easy way to upgrade a one-node cluster?

    Hi all...
    I have a one-node cluster running SQL Server.  I would like to upgrade the O/S from W2008R2 to W2012R2.  Unfortunately, the Compatibility report tells me that it won't upgrade because it is a member of a cluster.
    I understand that if it is a member of a two+ node cluster that it won't upgrade, but I don't understand why I can't do it on a single-node cluster. 
    Is there a work-around for this?  I don't want to start redirecting clients to a different cluster name / ip address.
    Thanks!

    Why even have a one-node cluster for anything but testing purposes?
    Why not create another server with the name of your cluster and set it up to serve out the SQL database.  Or, if you want to do it properly, get two nodes and make the cluster and keep the cluster name the same.  Those are two work arounds. 
    There is no upgrade in place for what you want to do.
    . : | : . : | : . tim

  • Oracle RAC 11g R2. Node pinned or unpinned?

    Hi all, I'm working with Oracle Clusterware & RAC 11 g R2.
    As is my first time, I really don't understand what is the meanning of pinning or unpinning a node.
    Can anyone help me please?
    Thanks in advance!!

    From 11.2 RAC deployment guide,
    Pinning a node means that the association of a node name with a node number is fixed. If a node is not pinned, its node number may change if the lease expires while it is down. The lease of a pinned node never expires.Since your installation is a clean installation ( no previous installation done) , you don't need to pin the nodes, it would be done by oracle clusterware.
    HTH
    Aman....

  • What are best practices for rolling out cluster upgrade?

    Hello,
    I am looking for your input on the approaches to implement Production RAC upgrade without having a spare instance of RAC servers. We have a 2-node database RAC 11.1 on Production. We are planning to upgrade to 11.2 but only have a single database that the pre-Production integration can be verified on. Our concern is that the integration may behave differently on RAC vs. the signle database instance. How everybody else approaches this problem?

    you want to test RAC upgrade on NON RAC database. If you ask me that is a risk but it depends on may things
    Application configuration - If your application is configured for RAC, FAN etc. you cannot test it on non RAC systems
    Cluster upgrade - If your standalone database is RAC one node you can probably test your cluster upgrade there. If you have non RAC database then you will not be able to test cluster upgrade or CRS
    Database upgrade - There are differences when you upgrade RAC vs non RAC database which you will not be able to test
    I think the best way for you is to convert your standalone database to RAC one node database and test it. that will take you close to multi node RAC

  • Query on adding new node to existing 2 node cluster

    Hi experts
    current environment
    Oracle VM 2 Node RAC cluster
    Oracle 11G 11.2.0.3.0
    Oracle 11g rel 2 GI
    on current 2 node cluster we have GI and RAC db configured
    Nodes: vmorarac1,vmorarac2
    we shutdown vmorarac1 to clone it to vmorarac5
    on new node I have changed the hostname to vmorarac5
    In /etc/sysconfig/network-scripts/ifcfg-eth0 change ip to new ip and same for /etc/sysconfig/network-scripts/ifcfg-eth1
    152.144.199.210,152.144.199.211
    mad echanges to /etc/hosts on vmorarac1/2 for 2 new IP address assigned to vmorarac5
    152.144.199.171 vmorarac1.pbi.global.pvt vmorarac1
    192.168.2.30 vmorarac1-priv.pbi.global.pvt vmorarac1-priv
    152.144.199.184 vmorarac1-vip.pbi.global.pvt vmorarac1-vip
    152.144.199.172 vmorarac2.pbi.global.pvt vmorarac2
    192.168.2.31 vmorarac2-priv.pbi.global.pvt vmorarac2-priv
    152.144.199.185 vmorarac2-vip.pbi.global.pvt vmorarac2-vip
    152.144.199.210 vmorarac2.pbi.global.pvt vmorarac2
    192.168.2.32 vmorarac2-priv.pbi.global.pvt vmorarac2-priv
    152.144.199.211 vmorarac2-vip.pbi.global.pvt vmorarac2-vip
    152.144.199.201 vmorarac-scan.pbi.global.pvt
    Query is is that all ok to reboot the new node to reflect changes as below and change the hostname to vmorarac2
    152.144.199.210 vmorarac2.pbi.global.pvt vmorarac2
    192.168.2.32 vmorarac2-priv.pbi.global.pvt vmorarac2-priv
    152.144.199.211 vmorarac2-vip.pbi.global.pvt vmorarac2-vip
    hostname change in /etc/sysconfig/network
    please let me know
    On new node vmorarac5, there is alreay software for RAC DB and GUI cloned from vmorarac1
    addnode.sh is used to add existing node, but as part of pre-requisite configuratoin is there any config step missing
    thanks

    The only thing you want "cloned" is the base OS install... The GI home directory, the oraInventory and a whole lot of other stuff is configured at addNode time or cluster install time. I would say that proceeding with your current plans will corrupt your clusterware. You really need to STOP LISTENING to the SA's and START reading the documentation on how to add a node to a cluster. Simply changing the hostname IS NOT a supported "feature".
    Start with a CLEAN OS. Configure shared devices, networks, /etc/hosts, memory and packages.
    Follow the instructions for addNode.
    Doing this any other way has the potential to corrupt your entire cluster.

  • GI installation on a single-node cluster error.

    Hello, I am trying to install GI on a single-node cluster (Solaris 10 / Sparc) but the root.sh script fails with the following error (this is not a GI installation for a Standalone Server :
    root@selvac./dev/ASM/OCRVTD_DG # /app/oracle/grid/11.2/root.sh
    Running Oracle 11g root script...
    The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME= /app/oracle/grid/11.2
    Enter the full pathname of the local bin directory: [usr/local/bin]:
    Copying dbhome to /usr/local/bin ...
    Copying oraenv to /usr/local/bin ...
    Copying coraenv to /usr/local/bin ...
    Creating /var/opt/oracle/oratab file...
    Entries will be added to the /var/opt/oracle/oratab file as needed by
    Database Configuration Assistant when a database is created
    Finished running generic part of root script.
    Now product-specific root actions will be performed.
    Using configuration parameter file: /app/oracle/grid/11.2/crs/install/crsconfig_params
    Creating trace directory
    LOCAL ADD MODE
    Creating OCR keys for user 'root', privgrp 'root'..
    Operation successful.
    OLR initialization - successful
    root wallet
    root wallet cert
    root cert export
    peer wallet
    profile reader wallet
    pa wallet
    peer wallet keys
    pa wallet keys
    peer cert request
    pa cert request
    peer cert
    pa cert
    peer root cert TP
    profile reader root cert TP
    pa root cert TP
    peer pa cert TP
    pa peer cert TP
    profile reader pa cert TP
    profile reader peer cert TP
    peer user cert
    pa user cert
    Adding daemon to inittab
    ACFS-9200: Supported
    ACFS-9300: ADVM/ACFS distribution files found.
    ACFS-9312: Existing ADVM/ACFS installation detected.
    ACFS-9314: Removing previous ADVM/ACFS installation.
    ACFS-9315: Previous ADVM/ACFS components successfully removed.
    ACFS-9307: Installing requested ADVM/ACFS software.
    ACFS-9308: Loading installed ADVM/ACFS drivers.
    ACFS-9327: Verifying ADVM/ACFS devices.
    ACFS-9309: ADVM/ACFS installation correctness verified.
    CRS-2672: Attempting to start 'ora.mdnsd' on 'selvac'
    CRS-2676: Start of 'ora.mdnsd' on 'selvac' succeeded
    CRS-2672: Attempting to start 'ora.gpnpd' on 'selvac'
    CRS-2676: Start of 'ora.gpnpd' on 'selvac' succeeded
    CRS-2672: Attempting to start 'ora.cssdmonitor' on 'selvac'
    CRS-2672: Attempting to start 'ora.gipcd' on 'selvac'
    CRS-2676: Start of 'ora.cssdmonitor' on 'selvac' succeeded
    CRS-2676: Start of 'ora.gipcd' on 'selvac' succeeded
    CRS-2672: Attempting to start 'ora.cssd' on 'selvac'
    CRS-2672: Attempting to start 'ora.diskmon' on 'selvac'
    CRS-2676: Start of 'ora.diskmon' on 'selvac' succeeded
    CRS-2676: Start of 'ora.cssd' on 'selvac' succeeded
    ASM created and started successfully.
    Disk Group OCRVTD_DG created successfully.
    The ora.asm resource is not ONLINE
    Did not succssfully configure and start ASM at /app/oracle/grid/11.2/crs/install/crsconfig_lib.pm line 6465.
    /app/oracle/grid/11.2/perl/bin/perl -I/app/oracle/grid/11.2/perl/lib -I/app/oracle/grid/11.2/crs/install /app/oracle/grid/11.2/crs/install/rootcrs.pl execution failed
    I also found the "PRVF-5150: Path OCRL:DISK1 is not a valid path on all nodes" error but as I have read it is a bug I Ignored it. But...
    I think my ASM_DG OCR and voting is ok, accessible by grid user and 660. It seems ASM does not start or does not start in time.
    Any help is wellcome.
    Thanks in advance.

    Thanks a lot for the hint. I had already checked this doc. but I think it is not the problem. Actually de error ora.asm is not online is not correct. After failing root.sh, ora.asm is ONLINE:
    root@selvac./app/oracle/grid/11.2/bin # ./crsctl check resource ora.asm -init
    root@selvac./app/oracle/grid/11.2/bin # ./crsctl stat resource ora.asm -init
    NAME=ora.asm
    TYPE=ora.asm.type
    TARGET=ONLINE
    STATE=ONLINE on selvac
    The last part of the /app/oracle/grid/11.2/cfgtoollogs/crsconfig/rootcrs_selvac.log file reads :
    >
    ASM created and started successfully.
    Disk Group OCRVTD_DG created successfully.
    End Command output2011-04-14 13:24:16: Executing cmd: /app/oracle/grid/11.2/bin/crsctl check resource ora.asm -init
    2011-04-14 13:24:17: Executing cmd: /app/oracle/grid/11.2/bin/crsctl status resource ora.asm -init
    2011-04-14 13:24:17: Command output:
    NAME=ora.asm
    TYPE=ora.asm.type
    TARGET=ONLINE
    STATE=OFFLINE
    End Command output2011-04-14 13:24:17: Checking the status of ora.asm
    2011-04-14 13:24:22: Executing cmd: /app/oracle/grid/11.2/bin/crsctl status resource ora.asm -init
    2011-04-14 13:24:22: Command output:
    NAME=ora.asm
    TYPE=ora.asm.type
    TARGET=ONLINE
    STATE=OFFLINE
    End Command output2011-04-14 13:24:22: Checking the status of ora.asm
    2011-04-14 13:24:27: Executing cmd: /app/oracle/grid/11.2/bin/crsctl status resource ora.asm -init
    2011-04-14 13:24:28: Command output:
    NAME=ora.asm
    TYPE=ora.asm.type
    TARGET=ONLINE
    STATE=OFFLINE
    End Command output2011-04-14 13:24:28: Checking the status of ora.asm
    2011-04-14 13:24:33: Executing cmd: /app/oracle/grid/11.2/bin/crsctl status resource ora.asm -init
    2011-04-14 13:24:33: Command output:
    NAME=ora.asm
    TYPE=ora.asm.type
    TARGET=ONLINE
    STATE=OFFLINE
    End Command output2011-04-14 13:24:33: Checking the status of ora.asm
    2011-04-14 13:24:38: Executing cmd: /app/oracle/grid/11.2/bin/crsctl status resource ora.asm -init
    2011-04-14 13:24:38: Command output:
    NAME=ora.asm
    TYPE=ora.asm.type
    TARGET=ONLINE
    STATE=OFFLINE
    End Command output2011-04-14 13:24:38: Checking the status of ora.asm
    2011-04-14 13:24:43: Executing cmd: /app/oracle/grid/11.2/bin/crsctl status resource ora.asm -init
    2011-04-14 13:24:43: Command output:
    NAME=ora.asm
    TYPE=ora.asm.type
    TARGET=ONLINE
    STATE=OFFLINE
    End Command output2011-04-14 13:24:43: Checking the status of ora.asm
    2011-04-14 13:24:48: Executing cmd: /app/oracle/grid/11.2/bin/crsctl status resource ora.asm -init
    2011-04-14 13:24:49: Command output:
    NAME=ora.asm
    TYPE=ora.asm.type
    TARGET=ONLINE
    STATE=OFFLINE
    End Command output2011-04-14 13:24:49: Checking the status of ora.asm
    2011-04-14 13:24:54: Executing cmd: /app/oracle/grid/11.2/bin/crsctl status resource ora.asm -init
    2011-04-14 13:24:54: Command output:
    NAME=ora.asm
    TYPE=ora.asm.type
    TARGET=ONLINE
    STATE=OFFLINE
    End Command output2011-04-14 13:24:54: Checking the status of ora.asm
    2011-04-14 13:24:59: Executing cmd: /app/oracle/grid/11.2/bin/crsctl status resource ora.asm -init
    2011-04-14 13:24:59: Command output:
    NAME=ora.asm
    TYPE=ora.asm.type
    TARGET=ONLINE
    STATE=OFFLINE
    End Command output2011-04-14 13:24:59: Checking the status of ora.asm
    2011-04-14 13:25:04: Executing cmd: /app/oracle/grid/11.2/bin/crsctl status resource ora.asm -init
    2011-04-14 13:25:04: Command output:
    NAME=ora.asm
    TYPE=ora.asm.type
    TARGET=ONLINE
    STATE=OFFLINE
    End Command output2011-04-14 13:25:04: Checking the status of ora.asm
    2011-04-14 13:25:09: The ora.asm resource is not ONLINE
    2011-04-14 13:25:09: Running as user grid: /app/oracle/grid/11.2/bin/cluutil -ckpt -oraclebase /app/grid -writeckpt -name ROOTCRS_BOOTCFG -state FAIL
    2011-04-14 13:25:09: s_run_as_user2: Running /bin/su grid -c ' /app/oracle/grid/11.2/bin/cluutil -ckpt -oraclebase /app/grid -writeckpt -name ROOTCRS_BOOTCFG -state FAIL '
    2011-04-14 13:25:10: Removing file /var/tmp/mbahSaGPn
    2011-04-14 13:25:10: Successfully removed file: /var/tmp/mbahSaGPn
    2011-04-14 13:25:10: /bin/su successfully executed
    2011-04-14 13:25:10: Succeeded in writing the checkpoint:'ROOTCRS_BOOTCFG' with status:FAIL
    2011-04-14 13:25:10: ###### Begin DIE Stack Trace ######
    2011-04-14 13:25:10: Package File Line Calling
    2011-04-14 13:25:10: --------------- -------------------- ---- ----------
    2011-04-14 13:25:10: 1: main rootcrs.pl 322 crsconfig_lib::dietrap
    2011-04-14 13:25:10: 2: crsconfig_lib crsconfig_lib.pm 6465 main::__ANON__
    2011-04-14 13:25:10: 3: crsconfig_lib crsconfig_lib.pm 6390 crsconfig_lib::perform_initial_config
    2011-04-14 13:25:10: 4: main rootcrs.pl 671 crsconfig_lib::perform_init_config
    2011-04-14 13:25:10: ####### End DIE Stack Trace #######
    2011-04-14 13:25:10: 'ROOTCRS_BOOTCFG' checkpoint has failed
    So this must be a bug. During root.sh execution ora.asm is OFFLINE but after failing it is ONLINE. It maight be a question of waiting/repeating or timeout as I see the "Checking the status of ora.asm" command is repeated several times during root.sh, but not enough perhaps. Now root.sh is failed, installation halted but ASM is ONLINE.
    Any other Idea?
    Thanks again.

  • Hyper-V - 2 node cluster goes down if one server shutsdown

    Hi all,
    I built a 2 node cluster with tiered storage and then I started doing some tests:
    * Drain one node and all the VMs moved to the other node, perfect
    * shutdown the drained node.
    The entire cluster crashed!!! The remaining node is trying to re-connect to the iSCSI SAN without success.
    * I booted the drained node. And it would not re-connect to the iSCSI SAN either. I had to force the reconnect in the iscsi control panel to make it re connect.
    So why would shutting down one node kill the cluster ? Sure it was the node that had the tiered pool online, but even then, isn't failover cluster supposed to put that one back and working on the other node ?
    Why did the active node lose the iSCSI connection too ? It had VMs running on it prior to the shutdown of the other node. My DC that was running on that other node is also now un available, can't ping it or anything.
    So what did I miss in the configuration of the cluster ? I followed the msdn 2 node hyper-v cluster doc.
    I am really worried atm since I had over the past 3 months a ton of issues with hyper-v going from using tiered storage, shutting down nodes, MAC address on the VMs and the hosts,... I thought that after hyper-v 2008, Microsoft had really made some progress
    with Hyper-V but I truly regret not going with VMWare again this time around.
    That cluster was supposed to go into homologation phase tomorrow at the datacenter but now I am unsure if I ll ever be able to trust it to work.
    The SAN is an MD3200i which is reported as Hyper-V ready.
    Any hint on where I have gone wrong would be appreciated.
    Regards,
    Edit: even from the host with powershell I was not able to shutdown the DC and reboot it clean. Said the integration services were not reachable... it is a 2012 R2 servers...
    Edit2: One of my VM is gone ! Can't even find the file on the disks either locally on the hosts or on the SAN. WTF!!!

    Actually comes across very reasonable. And I think you are right. I tend to compare Hyper-V to vSphere with vCenter included. I have not seen nor used VMM. Also true that Storage Pools and iSCSI is not Hyper-V, but to me it comes as a package just as much
    as ESXi 'comes' with it.
    As for burning personnal hours and money on books, I have, just as much as I go to conferences when I can and can afford it. And the only thing I would envy you, is the fact you have your colleagues to bounce idea of / lay on if necessary.
    As for the few hundred box for the management suite. I believe the stack you speak of would actually cost my current company about 14k$, that is not a few hundred box. That is pretty much the cost of one of the 2 SANs.  By 14k$, I mean that
    we have 6 servers with 2 sockets each, running a lot of VMs, which means Datacenter licences which list price is 3.6k$. I am not even including the CALs. Or am I mistaken on the licensing ?
    If VMWare, I would be going with essentials which, at the same server perimeter, would be 30% less expensive. We don't really need the full blown one at our level.
    I am also locked by hardware that was ordered by my predecessor which do not provide the service we need them for. (I blame the vendor on that one, my predecessor was not an infrastructure guy). As for storage, if you are referring to SMB3 and using
    a failover cluster to provide the disks to the hyper-v hosts, I agree. I am just not too sure on the technology yet and went more for safety until I can test it thoroughly in the dev environment.
    I also hope Microsoft will add an easier way to set the media type on disks as well as allow for more than just SSD and HDD or even allow us to define our own.
    I actually fooled the system this time around because SSD are too expensive and too high a failure rate compared to 15kRPM (yes the performance are lovely) at the moment. So I made the 15kRPMs into SSD.
    For the remote management issue, i meant that actually, at this time, I have to disable the domain firewall to be able to manage the hyper-v 2012 R2. I tried hvremote, adding all the necessary rules, etc... What I did for 2012 worked perfectly fine and I
    had full controls. 2012 R2 does not. I have another thread on this in the forum and I ll come back to it as soon as I can.
    I don't really care for the no interface thingy, I enjoy Powershell and scripting fine :) Does a lot of automation for me :) I am used to scripting anyway and it is faster to reproduce steps that way. You do it once, and you got something you can apply with
    little changes to everything.
    I joined the company in November last year and I got dropped a full stack to upgrade in 6 months while maintaining the current one. Encountering the problem now is better and is, fundamentally, good, though it is time consumming.
    By full stack I mean:
    * Help the dev re-design the apps so it handles load better. Get out of Windows 2003 and migrate to 2012 and validate all the applications on 2012 as well as improve security.
    * Implementing a monitoring system for the infrastructure and the applications.
    * Upgrade SQL Server 2005 to, in this case 2012 Standard (no choice and enterprise not in the price range of the company). Converge our current test and prod environments. Optimize all the queries... And naturally validate the applications.
    * Upgrade the certificate authorities so they are available on all sites. Haven't scratched that one yet.
    * Design a fully site redundant architecture so that if a site goes dark we have no impact. If there is partial failure on one site, no issue either, and so on XD. I wish I had AG :)
    * Implement a single windows domain on all the sites, that will be a relief :) Running 4 domains and about 10 different workgroups atm
    * Upgrade the firewalls, switches, servers hardware, ... Implement the necessary networks for virtualization and improved security rules... (Don't buy SonicWalls, had a worse headache with them than Hyper-V :) )
    * Migrate the users from old domain to new one. Windows 8 does not have the user migration tool anymore :(
    * Plan the DR tests and processes :)
    * Gotta get Hyper-V replica working as well as backups.
    * And naturally make all the documentation so that if I happen to get under a bus, anyone in the company can just follow the documentation in case of an emergency.
    And I am sure I am still missing some part of the environment no one knows about at some places :) Found a new network today XD.
    As for the mixed VM management, I mean some that are in HA on the cluster and others that don't need and must not failover. Am sure there is a way to configure them in the failover cluster so they don't failover. I just need to spend the 15 minutes looking
    into it :)
    Since I have had that SAN, I spent about 5 days on the phone with Dell support. That iSCSI issue is not the first one I got :( But in the production environment, that issue has disappeared. So at this time, i have switched working on other issues. And with
    the information you provided, I'll be able to know if one of the node of the cluster lose the connectivity which will help avoid this issue in the future.
    And it is a very good and interesting challenge. I am just running in way to many issues doing a 10 year old system upgrade :) I was expecting to have an easier time with Microsoft Virtualization than I've had.
    Oh and just for the fun, I also had VM disappear, no more file, no nothing, just poof. Have not figured how that happened yet. Had the backup so not much of an issue, but still, it went poof when the original issue happened.
    Thanks for the time and the tips :)

  • How to install and configure APEX listener on a 2 node cluster WLS

    So I've installed the apex listener on a 2 node cluster WLS and can get the Apex login screen to pop only when node 1 is up and node 2 is down.
    I'm going through a F5 - round robin and the login page does not pop when on node 2.
    I've followed the Oracle Apex Listener install doc: Oracle® Application Express Listener Installation and Configuration Guide
    Release 2.0 Part Number E25066-08
    Also followed the following blog exactly and compared it the install doc: http://www.snapdba.com/2013/05/configuring-the-apex-listener-2-0-1-on-weblogic-12c/
    Not much literature on what to do for a cluster multi node WLS configuration.
    What do I need to deploy onto node 2? I assume the images but what else?
    WLS: 10.3.6.0 - 2 linux servers (Linux 2.6.39-200.24.1.el6uek.x86_64 #1 SMP Sat Jun 23 02:39:07 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux)
    APEX: 4.2.2
    APEX Listener: 2.0.3.221.10.13
    DATABASE: 11.2.0.3 RAC (only one database)

    Ok so I went and copied the image directory to both WLS servers in the correct directory that was dictated when creating the image war (i.war). Seems to work.
    Thx,
    Russ

  • SOA 11.1.1.5 two Node Cluster BAM not starting on the second Machine

    Hi,
    I have installed Oracle SOA 11.1.1.5 in a 2 Node Cluster. Both SOA Managed Servers are coming up fine. BAM Managed Server installed on the primary HOST also comes up fine. But my second BAM Managed Server is not starting. I don't see anything in the logs and on the Console the status is showing as Starting but it never starts. I can access the BAM Console for the one installed on the primary HOST but when giving the URL for the second HOST it doesn't come up as the I don't see an option the server is running.
    I have 2 machines for this cluster setup
    On machine one with IP Address: *192.168.2.2* all below servers come-up fine
    Admin
    BAM1
    SOA1
    On machine two with IP Address: *192.168.3.3* have the below servers and only the SOA is coming up fine BAM not starting
    BAM2 not starting
    SOA2 coming up fine also
    After giving user/password this is what I see in the terminal window and it never moves looks like getting stuck:
    <Mar 8, 2012 12:31:40 PM EST> <Info> <Security> <BEA-090065> <Getting boot ident
    ity from user.>
    Enter username to boot WebLogic server:weblogic
    Enter password to boot WebLogic server:
    <Mar 8, 2012 12:32:58 PM EST> <Info> <Management> <BEA-141107> <Version: WebLogi
    c Server 10.3.5.0  Fri Apr 1 20:20:06 PDT 2011 1398638 >
    <Mar 8, 2012 12:33:01 PM EST> <Notice> <WebLogicServer> <BEA-000365> <Server sta
    te changed to STARTING>
    <Mar 8, 2012 12:33:01 PM EST> <Info> <WorkManager> <BEA-002900> <Initializing se
    lf-tuning thread pool>
    <Mar 8, 2012 12:33:01 PM EST> <Notice> <LoggingService> <BEA-320400> <The log fi
    le C:\oracle\Middleware\user_projects\domains\EP_SOA_DOMAIN\servers\BAM2\logs\BA
    M2.log will be rotated. Reopen the log file if tailing has stopped. This can hap
    pen on some platforms like Windows.>
    <Mar 8, 2012 12:33:01 PM EST> <Notice> <LoggingService> <BEA-320401> <The log fi
    le has been rotated to C:\oracle\Middleware\user_projects\domains\EP_SOA_DOMAIN\
    servers\BAM2\logs\BAM2.log00001. Log messages will continue to be logged in C:\o
    racle\Middleware\user_projects\domains\EP_SOA_DOMAIN\servers\BAM2\logs\BAM2.log.
    >
    <Mar 8, 2012 12:33:01 PM EST> <Notice> <Log Management> <BEA-170019> <The server
    log file C:\oracle\Middleware\user_projects\domains\EP_SOA_DOMAIN\servers\BAM2\
    logs\BAM2.log is opened. All server side log events will be written to this file
    .>
    <Mar 8, 2012 12:33:09 PM EST> <Notice> <Security> <BEA-090082> <Security initial
    izing using security realm myrealm.>
    <Mar 8, 2012 12:33:15 PM EST> <Notice> <WebLogicServer> <BEA-000365> <Server sta
    te changed to STANDBY>
    <Mar 8, 2012 12:33:15 PM EST> <Notice> <WebLogicServer> <BEA-000365> <Server sta
    te changed to STARTING>
    <Mar 8, 2012 12:33:26 PM EST> <Warning> <J2EE> <BEA-160140> <Unresolved optional
    package references (in META-INF/MANIFEST.MF): [Extension-Name: oracle.applcore.
    model, Specification-Version: 0.1, Implementation-Version: 11.1.1.0.0, reference
    d from: C:\oracle\Middleware\user_projects\domains\EP_SOA_DOMAIN\servers\BAM2\tm
    p\_WL_user\usermessagingserver\wehyzl]. Make sure the referenced optional packag
    e has been deployed as a library.>
    <Mar 8, 2012 12:33:28 PM EST> <Warning> <EJB> <BEA-015036> <In the EJB MessageRe
    ceiverBean(Application: usermessagingserver, EJBComponent: sdpmessagingengine.ja
    r), the value for the activation config property 'connectionFactoryJNDIName' wil
    l be ignored since it is overridden with the value 'OraSDPM/QueueConnectionFacto
    ry';>
    <Mar 8, 2012 12:33:32 PM EST> <Warning> <EJB> <BEA-012035> <The Remote interface
    method: 'public abstract void oracle.ucs.userprefs.DefaultFactStore.setBusiness
    FactValue(java.lang.String,java.lang.String,java.lang.Object) throws oracle.ucs.
    userprefs.UserPrefsException' in EJB 'UserPrefsDefaultFactStore' contains a para
    meter of type: 'java.lang.Object' which is not Serializable. Though the EJB 'Use
    rPrefsDefaultFactStore' has call-by-reference set to false, this parameter is no
    t Serializable and hence will be passed by reference. A parameter can be passed
    using call-by-value only if the parameter type is Serializable.>
    <Mar 8, 2012 12:33:32 PM EST> <Warning> <EJB> <BEA-015036> <In the EJB MessageDi
    spatcherBean(Application: usermessagingserver, EJBComponent: sdpmessagingclient-
    ejb-parlayx.jar), the value for the activation config property 'connectionFactor
    yJNDIName' will be ignored since it is overridden with the value 'OraSDPM/QueueC
    onnectionFactory';>
    <Mar 8, 2012 12:33:37 PM EST> <Warning> <Connector> <BEA-190110> <Resource Adapt
    er is calling BootstrapContext.createTimer() and allocating a Timer Thread that
    is not managed by WebLogic Server. This may adversely impact the performance/ope
    ration of WebLogic Server.>
    <Mar 8, 2012 12:33:37 PM EST> <Warning> <EJB> <BEA-015036> <In the EJB DriverDis
    patcherBean(Application: usermessagingdriver-email, EJBComponent: sdpmessagingdr
    iver-dispatcher-ejb.jar), the value for the activation config property 'connecti
    onFactoryJNDIName' will be ignored since it is overridden with the value 'OraSDP
    M/QueueConnectionFactory';>
    <Mar 8, 2012 12:33:43 PM EST> <Warning> <Munger> <BEA-2156203> <A version attrib
    ute was not found in element ejb-jar in the deployment descriptor in C:\oracle\M
    iddleware\user_projects\domains\EP_SOA_DOMAIN\servers\BAM2\tmp\_WL_user\oracle-b
    am_11.1.1\ert5kb\oracle-bam-adc-ejb.jar/META-INF/ejb-jar.xml. A version attribut
    e is required, but this version of the Weblogic Server will assume that the JEE5
    is used. Future versions of the Weblogic Server will reject descriptors that do
    not specify the JEE version.>
    <Mar 8, 2012 12:33:44 PM EST> <Warning> <Munger> <BEA-2156203> <A version attrib
    ute was not found in element ejb-jar in the deployment descriptor in C:\oracle\M
    iddleware\user_projects\domains\EP_SOA_DOMAIN\servers\BAM2\tmp\_WL_user\oracle-b
    am_11.1.1\ert5kb\oracle-bam-ems-ejb.jar/META-INF/ejb-jar.xml. A version attribut
    e is required, but this version of the Weblogic Server will assume that the JEE5
    is used. Future versions of the Weblogic Server will reject descriptors that do
    not specify the JEE version.>
    <Mar 8, 2012 12:33:44 PM EST> <Warning> <Munger> <BEA-2156203> <A version attrib
    ute was not found in element ejb-jar in the deployment descriptor in C:\oracle\M
    iddleware\user_projects\domains\EP_SOA_DOMAIN\servers\BAM2\tmp\_WL_user\oracle-b
    am_11.1.1\ert5kb\oracle-bam-eventengine-ejb.jar/META-INF/ejb-jar.xml. A version
    attribute is required, but this version of the Weblogic Server will assume that
    the JEE5 is used. Future versions of the Weblogic Server will reject descriptors
    that do not specify the JEE version.>
    <Mar 8, 2012 12:33:44 PM EST> <Warning> <Munger> <BEA-2156203> <A version attrib
    ute was not found in element ejb-jar in the deployment descriptor in C:\oracle\M
    iddleware\user_projects\domains\EP_SOA_DOMAIN\servers\BAM2\tmp\_WL_user\oracle-b
    am_11.1.1\ert5kb\oracle-bam-reportcache-ejb.jar/META-INF/ejb-jar.xml. A version
    attribute is required, but this version of the Weblogic Server will assume that
    the JEE5 is used. Future versions of the Weblogic Server will reject descriptors
    that do not specify the JEE version.>
    <Mar 8, 2012 12:33:52 PM EST> <Warning> <oracle.sdp.messaging.driver.base> <SDP-
    26024> <Registration of driver Email-Driver did not complete; will retry periodi
    cally until messaging server responds.>This is what I see in the diagnostic log
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-014021> <The EJB 'EMSServerBean(Application: oracle-bam#11.1.1, EJBComponent: oracle-bam-ems-ejb.jar)' has been successfully deployed. The following remote interfaces have been bound into JNDI with the specified JNDI names:>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-014022> <******** oracle.bam.ems.ejb.EMSServerRemote is bound with JNDI name:EMSServerBean#oracle.bam.ems.ejb.EMSServerRemote ********>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-010009> <EJB Deployed EJB with JNDI name oracle-bam#11_1_1oracle-bam-ems-ejb_jarEMSServerBean_Home.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-149060> <Module oracle-bam-ems-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-149059> <Module oracle-bam-eventengine-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-014021> <The EJB 'EventEngineServerBean(Application: oracle-bam#11.1.1, EJBComponent: oracle-bam-eventengine-ejb.jar)' has been successfully deployed. The following remote interfaces have been bound into JNDI with the specified JNDI names:>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-014022> <******** oracle.bam.eventengine.ejb.EventEngineServerRemote is bound with JNDI name:EventEngineServerBean#oracle.bam.eventengine.ejb.EventEngineServerRemote ********>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-010009> <EJB Deployed EJB with JNDI name oracle-bam#11_1_1oracle-bam-eventengine-ejb_jarEventEngineServerBean_Home.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-149060> <Module oracle-bam-eventengine-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-149059> <Module oracle-bam-reportcache-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-014021> <The EJB 'ReportCacheServerBean(Application: oracle-bam#11.1.1, EJBComponent: oracle-bam-reportcache-ejb.jar)' has been successfully deployed. The following remote interfaces have been bound into JNDI with the specified JNDI names:>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-014022> <******** oracle.bam.reportcache.ejb.ReportCacheServerRemote is bound with JNDI name:ReportCacheServerBean#oracle.bam.reportcache.ejb.ReportCacheServerRemote ********>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-010009> <EJB Deployed EJB with JNDI name oracle-bam#11_1_1oracle-bam-reportcache-ejb_jarReportCacheServerBean_Home.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-149060> <Module oracle-bam-reportcache-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-149059> <Module oracle-bam-statuslistener-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-010009> <EJB Deployed EJB with JNDI name StatusListenerEJB.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-149060> <Module oracle-bam-statuslistener-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-149059> <Module sdpmessagingclient-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-010009> <EJB Deployed EJB with JNDI name MessagingClientBAM.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-010223> <EJB Deployed Message Driven Bean: MessageDispatcherBean>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-149060> <Module sdpmessagingclient-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <JDBC> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <OracleSystemUser> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048859> <BEA-001516> <Connection Pool "BAMDataSource" connected to Database: "Oracle", Version: "Oracle Database 11g Express Edition Release 11.2.0.2.0 - Production".>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <JDBC> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <OracleSystemUser> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048859> <BEA-001517> <Connection Pool "BAMDataSource" using Driver: "Oracle JDBC driver", Version: "11.2.0.2.0".>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Common> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <OracleSystemUser> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048859> <BEA-000628> <Created "1" resources for pool "BAMDataSource", out of which "1" are available and "0" are unavailable.> And this is what I see in the log
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048125> <BEA-149059> <Module oracle-bam-adc-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048359> <BEA-014021> <The EJB 'BamAdcServerBean(Application: oracle-bam#11.1.1, EJBComponent: oracle-bam-adc-ejb.jar)' has been successfully deployed. The following remote interfaces have been bound into JNDI with the specified JNDI names:>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048359> <BEA-014022> <******** oracle.bam.adc.ejb.BamAdcServerRemote is bound with JNDI name:BamAdcServerBean#oracle.bam.adc.ejb.BamAdcServerRemote ********>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048359> <BEA-010009> <EJB Deployed EJB with JNDI name oracle-bam#11_1_1oracle-bam-adc-ejb_jarBamAdcServerBean_Home.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048359> <BEA-149060> <Module oracle-bam-adc-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048359> <BEA-149059> <Module oracle-bam-ems-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-014021> <The EJB 'EMSServerBean(Application: oracle-bam#11.1.1, EJBComponent: oracle-bam-ems-ejb.jar)' has been successfully deployed. The following remote interfaces have been bound into JNDI with the specified JNDI names:>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-014022> <******** oracle.bam.ems.ejb.EMSServerRemote is bound with JNDI name:EMSServerBean#oracle.bam.ems.ejb.EMSServerRemote ********>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-010009> <EJB Deployed EJB with JNDI name oracle-bam#11_1_1oracle-bam-ems-ejb_jarEMSServerBean_Home.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-149060> <Module oracle-bam-ems-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-149059> <Module oracle-bam-eventengine-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-014021> <The EJB 'EventEngineServerBean(Application: oracle-bam#11.1.1, EJBComponent: oracle-bam-eventengine-ejb.jar)' has been successfully deployed. The following remote interfaces have been bound into JNDI with the specified JNDI names:>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-014022> <******** oracle.bam.eventengine.ejb.EventEngineServerRemote is bound with JNDI name:EventEngineServerBean#oracle.bam.eventengine.ejb.EventEngineServerRemote ********>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-010009> <EJB Deployed EJB with JNDI name oracle-bam#11_1_1oracle-bam-eventengine-ejb_jarEventEngineServerBean_Home.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-149060> <Module oracle-bam-eventengine-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-149059> <Module oracle-bam-reportcache-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-014021> <The EJB 'ReportCacheServerBean(Application: oracle-bam#11.1.1, EJBComponent: oracle-bam-reportcache-ejb.jar)' has been successfully deployed. The following remote interfaces have been bound into JNDI with the specified JNDI names:>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-014022> <******** oracle.bam.reportcache.ejb.ReportCacheServerRemote is bound with JNDI name:ReportCacheServerBean#oracle.bam.reportcache.ejb.ReportCacheServerRemote ********>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-010009> <EJB Deployed EJB with JNDI name oracle-bam#11_1_1oracle-bam-reportcache-ejb_jarReportCacheServerBean_Home.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-149060> <Module oracle-bam-reportcache-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-149059> <Module oracle-bam-statuslistener-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-010009> <EJB Deployed EJB with JNDI name StatusListenerEJB.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-149060> <Module oracle-bam-statuslistener-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-149059> <Module sdpmessagingclient-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-010009> <EJB Deployed EJB with JNDI name MessagingClientBAM.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-010223> <EJB Deployed Message Driven Bean: MessageDispatcherBean>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-149060> <Module sdpmessagingclient-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <JDBC> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <OracleSystemUser> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048859> <BEA-001516> <Connection Pool "BAMDataSource" connected to Database: "Oracle", Version: "Oracle Database 11g Express Edition Release 11.2.0.2.0 - Production".>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <JDBC> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <OracleSystemUser> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048859> <BEA-001517> <Connection Pool "BAMDataSource" using Driver: "Oracle JDBC driver", Version: "11.2.0.2.0".>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Common> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <OracleSystemUser> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048859> <BEA-000628> <Created "1" resources for pool "BAMDataSource", out of which "1" are available and "0" are unavailable.> Any help is appreciated.
    Thanks

    Hi,
    I have installed Oracle SOA 11.1.1.5 in a 2 Node Cluster. Both SOA Managed Servers are coming up fine. BAM Managed Server installed on the primary HOST also comes up fine. But my second BAM Managed Server is not starting. I don't see anything in the logs and on the Console the status is showing as Starting but it never starts. I can access the BAM Console for the one installed on the primary HOST but when giving the URL for the second HOST it doesn't come up as the I don't see an option the server is running.
    I have 2 machines for this cluster setup
    On machine one with IP Address: *192.168.2.2* all below servers come-up fine
    Admin
    BAM1
    SOA1
    On machine two with IP Address: *192.168.3.3* have the below servers and only the SOA is coming up fine BAM not starting
    BAM2 not starting
    SOA2 coming up fine also
    After giving user/password this is what I see in the terminal window and it never moves looks like getting stuck:
    <Mar 8, 2012 12:31:40 PM EST> <Info> <Security> <BEA-090065> <Getting boot ident
    ity from user.>
    Enter username to boot WebLogic server:weblogic
    Enter password to boot WebLogic server:
    <Mar 8, 2012 12:32:58 PM EST> <Info> <Management> <BEA-141107> <Version: WebLogi
    c Server 10.3.5.0  Fri Apr 1 20:20:06 PDT 2011 1398638 >
    <Mar 8, 2012 12:33:01 PM EST> <Notice> <WebLogicServer> <BEA-000365> <Server sta
    te changed to STARTING>
    <Mar 8, 2012 12:33:01 PM EST> <Info> <WorkManager> <BEA-002900> <Initializing se
    lf-tuning thread pool>
    <Mar 8, 2012 12:33:01 PM EST> <Notice> <LoggingService> <BEA-320400> <The log fi
    le C:\oracle\Middleware\user_projects\domains\EP_SOA_DOMAIN\servers\BAM2\logs\BA
    M2.log will be rotated. Reopen the log file if tailing has stopped. This can hap
    pen on some platforms like Windows.>
    <Mar 8, 2012 12:33:01 PM EST> <Notice> <LoggingService> <BEA-320401> <The log fi
    le has been rotated to C:\oracle\Middleware\user_projects\domains\EP_SOA_DOMAIN\
    servers\BAM2\logs\BAM2.log00001. Log messages will continue to be logged in C:\o
    racle\Middleware\user_projects\domains\EP_SOA_DOMAIN\servers\BAM2\logs\BAM2.log.
    >
    <Mar 8, 2012 12:33:01 PM EST> <Notice> <Log Management> <BEA-170019> <The server
    log file C:\oracle\Middleware\user_projects\domains\EP_SOA_DOMAIN\servers\BAM2\
    logs\BAM2.log is opened. All server side log events will be written to this file
    .>
    <Mar 8, 2012 12:33:09 PM EST> <Notice> <Security> <BEA-090082> <Security initial
    izing using security realm myrealm.>
    <Mar 8, 2012 12:33:15 PM EST> <Notice> <WebLogicServer> <BEA-000365> <Server sta
    te changed to STANDBY>
    <Mar 8, 2012 12:33:15 PM EST> <Notice> <WebLogicServer> <BEA-000365> <Server sta
    te changed to STARTING>
    <Mar 8, 2012 12:33:26 PM EST> <Warning> <J2EE> <BEA-160140> <Unresolved optional
    package references (in META-INF/MANIFEST.MF): [Extension-Name: oracle.applcore.
    model, Specification-Version: 0.1, Implementation-Version: 11.1.1.0.0, reference
    d from: C:\oracle\Middleware\user_projects\domains\EP_SOA_DOMAIN\servers\BAM2\tm
    p\_WL_user\usermessagingserver\wehyzl]. Make sure the referenced optional packag
    e has been deployed as a library.>
    <Mar 8, 2012 12:33:28 PM EST> <Warning> <EJB> <BEA-015036> <In the EJB MessageRe
    ceiverBean(Application: usermessagingserver, EJBComponent: sdpmessagingengine.ja
    r), the value for the activation config property 'connectionFactoryJNDIName' wil
    l be ignored since it is overridden with the value 'OraSDPM/QueueConnectionFacto
    ry';>
    <Mar 8, 2012 12:33:32 PM EST> <Warning> <EJB> <BEA-012035> <The Remote interface
    method: 'public abstract void oracle.ucs.userprefs.DefaultFactStore.setBusiness
    FactValue(java.lang.String,java.lang.String,java.lang.Object) throws oracle.ucs.
    userprefs.UserPrefsException' in EJB 'UserPrefsDefaultFactStore' contains a para
    meter of type: 'java.lang.Object' which is not Serializable. Though the EJB 'Use
    rPrefsDefaultFactStore' has call-by-reference set to false, this parameter is no
    t Serializable and hence will be passed by reference. A parameter can be passed
    using call-by-value only if the parameter type is Serializable.>
    <Mar 8, 2012 12:33:32 PM EST> <Warning> <EJB> <BEA-015036> <In the EJB MessageDi
    spatcherBean(Application: usermessagingserver, EJBComponent: sdpmessagingclient-
    ejb-parlayx.jar), the value for the activation config property 'connectionFactor
    yJNDIName' will be ignored since it is overridden with the value 'OraSDPM/QueueC
    onnectionFactory';>
    <Mar 8, 2012 12:33:37 PM EST> <Warning> <Connector> <BEA-190110> <Resource Adapt
    er is calling BootstrapContext.createTimer() and allocating a Timer Thread that
    is not managed by WebLogic Server. This may adversely impact the performance/ope
    ration of WebLogic Server.>
    <Mar 8, 2012 12:33:37 PM EST> <Warning> <EJB> <BEA-015036> <In the EJB DriverDis
    patcherBean(Application: usermessagingdriver-email, EJBComponent: sdpmessagingdr
    iver-dispatcher-ejb.jar), the value for the activation config property 'connecti
    onFactoryJNDIName' will be ignored since it is overridden with the value 'OraSDP
    M/QueueConnectionFactory';>
    <Mar 8, 2012 12:33:43 PM EST> <Warning> <Munger> <BEA-2156203> <A version attrib
    ute was not found in element ejb-jar in the deployment descriptor in C:\oracle\M
    iddleware\user_projects\domains\EP_SOA_DOMAIN\servers\BAM2\tmp\_WL_user\oracle-b
    am_11.1.1\ert5kb\oracle-bam-adc-ejb.jar/META-INF/ejb-jar.xml. A version attribut
    e is required, but this version of the Weblogic Server will assume that the JEE5
    is used. Future versions of the Weblogic Server will reject descriptors that do
    not specify the JEE version.>
    <Mar 8, 2012 12:33:44 PM EST> <Warning> <Munger> <BEA-2156203> <A version attrib
    ute was not found in element ejb-jar in the deployment descriptor in C:\oracle\M
    iddleware\user_projects\domains\EP_SOA_DOMAIN\servers\BAM2\tmp\_WL_user\oracle-b
    am_11.1.1\ert5kb\oracle-bam-ems-ejb.jar/META-INF/ejb-jar.xml. A version attribut
    e is required, but this version of the Weblogic Server will assume that the JEE5
    is used. Future versions of the Weblogic Server will reject descriptors that do
    not specify the JEE version.>
    <Mar 8, 2012 12:33:44 PM EST> <Warning> <Munger> <BEA-2156203> <A version attrib
    ute was not found in element ejb-jar in the deployment descriptor in C:\oracle\M
    iddleware\user_projects\domains\EP_SOA_DOMAIN\servers\BAM2\tmp\_WL_user\oracle-b
    am_11.1.1\ert5kb\oracle-bam-eventengine-ejb.jar/META-INF/ejb-jar.xml. A version
    attribute is required, but this version of the Weblogic Server will assume that
    the JEE5 is used. Future versions of the Weblogic Server will reject descriptors
    that do not specify the JEE version.>
    <Mar 8, 2012 12:33:44 PM EST> <Warning> <Munger> <BEA-2156203> <A version attrib
    ute was not found in element ejb-jar in the deployment descriptor in C:\oracle\M
    iddleware\user_projects\domains\EP_SOA_DOMAIN\servers\BAM2\tmp\_WL_user\oracle-b
    am_11.1.1\ert5kb\oracle-bam-reportcache-ejb.jar/META-INF/ejb-jar.xml. A version
    attribute is required, but this version of the Weblogic Server will assume that
    the JEE5 is used. Future versions of the Weblogic Server will reject descriptors
    that do not specify the JEE version.>
    <Mar 8, 2012 12:33:52 PM EST> <Warning> <oracle.sdp.messaging.driver.base> <SDP-
    26024> <Registration of driver Email-Driver did not complete; will retry periodi
    cally until messaging server responds.>This is what I see in the diagnostic log
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-014021> <The EJB 'EMSServerBean(Application: oracle-bam#11.1.1, EJBComponent: oracle-bam-ems-ejb.jar)' has been successfully deployed. The following remote interfaces have been bound into JNDI with the specified JNDI names:>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-014022> <******** oracle.bam.ems.ejb.EMSServerRemote is bound with JNDI name:EMSServerBean#oracle.bam.ems.ejb.EMSServerRemote ********>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-010009> <EJB Deployed EJB with JNDI name oracle-bam#11_1_1oracle-bam-ems-ejb_jarEMSServerBean_Home.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-149060> <Module oracle-bam-ems-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-149059> <Module oracle-bam-eventengine-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-014021> <The EJB 'EventEngineServerBean(Application: oracle-bam#11.1.1, EJBComponent: oracle-bam-eventengine-ejb.jar)' has been successfully deployed. The following remote interfaces have been bound into JNDI with the specified JNDI names:>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-014022> <******** oracle.bam.eventengine.ejb.EventEngineServerRemote is bound with JNDI name:EventEngineServerBean#oracle.bam.eventengine.ejb.EventEngineServerRemote ********>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-010009> <EJB Deployed EJB with JNDI name oracle-bam#11_1_1oracle-bam-eventengine-ejb_jarEventEngineServerBean_Home.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-149060> <Module oracle-bam-eventengine-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-149059> <Module oracle-bam-reportcache-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-014021> <The EJB 'ReportCacheServerBean(Application: oracle-bam#11.1.1, EJBComponent: oracle-bam-reportcache-ejb.jar)' has been successfully deployed. The following remote interfaces have been bound into JNDI with the specified JNDI names:>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-014022> <******** oracle.bam.reportcache.ejb.ReportCacheServerRemote is bound with JNDI name:ReportCacheServerBean#oracle.bam.reportcache.ejb.ReportCacheServerRemote ********>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-010009> <EJB Deployed EJB with JNDI name oracle-bam#11_1_1oracle-bam-reportcache-ejb_jarReportCacheServerBean_Home.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-149060> <Module oracle-bam-reportcache-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-149059> <Module oracle-bam-statuslistener-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-010009> <EJB Deployed EJB with JNDI name StatusListenerEJB.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-149060> <Module oracle-bam-statuslistener-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-149059> <Module sdpmessagingclient-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-010009> <EJB Deployed EJB with JNDI name MessagingClientBAM.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-010223> <EJB Deployed Message Driven Bean: MessageDispatcherBean>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-149060> <Module sdpmessagingclient-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <JDBC> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <OracleSystemUser> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048859> <BEA-001516> <Connection Pool "BAMDataSource" connected to Database: "Oracle", Version: "Oracle Database 11g Express Edition Release 11.2.0.2.0 - Production".>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <JDBC> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <OracleSystemUser> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048859> <BEA-001517> <Connection Pool "BAMDataSource" using Driver: "Oracle JDBC driver", Version: "11.2.0.2.0".>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Common> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <OracleSystemUser> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048859> <BEA-000628> <Created "1" resources for pool "BAMDataSource", out of which "1" are available and "0" are unavailable.> And this is what I see in the log
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048125> <BEA-149059> <Module oracle-bam-adc-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048359> <BEA-014021> <The EJB 'BamAdcServerBean(Application: oracle-bam#11.1.1, EJBComponent: oracle-bam-adc-ejb.jar)' has been successfully deployed. The following remote interfaces have been bound into JNDI with the specified JNDI names:>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048359> <BEA-014022> <******** oracle.bam.adc.ejb.BamAdcServerRemote is bound with JNDI name:BamAdcServerBean#oracle.bam.adc.ejb.BamAdcServerRemote ********>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048359> <BEA-010009> <EJB Deployed EJB with JNDI name oracle-bam#11_1_1oracle-bam-adc-ejb_jarBamAdcServerBean_Home.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048359> <BEA-149060> <Module oracle-bam-adc-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048359> <BEA-149059> <Module oracle-bam-ems-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-014021> <The EJB 'EMSServerBean(Application: oracle-bam#11.1.1, EJBComponent: oracle-bam-ems-ejb.jar)' has been successfully deployed. The following remote interfaces have been bound into JNDI with the specified JNDI names:>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-014022> <******** oracle.bam.ems.ejb.EMSServerRemote is bound with JNDI name:EMSServerBean#oracle.bam.ems.ejb.EMSServerRemote ********>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-010009> <EJB Deployed EJB with JNDI name oracle-bam#11_1_1oracle-bam-ems-ejb_jarEMSServerBean_Home.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-149060> <Module oracle-bam-ems-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048390> <BEA-149059> <Module oracle-bam-eventengine-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-014021> <The EJB 'EventEngineServerBean(Application: oracle-bam#11.1.1, EJBComponent: oracle-bam-eventengine-ejb.jar)' has been successfully deployed. The following remote interfaces have been bound into JNDI with the specified JNDI names:>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-014022> <******** oracle.bam.eventengine.ejb.EventEngineServerRemote is bound with JNDI name:EventEngineServerBean#oracle.bam.eventengine.ejb.EventEngineServerRemote ********>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-010009> <EJB Deployed EJB with JNDI name oracle-bam#11_1_1oracle-bam-eventengine-ejb_jarEventEngineServerBean_Home.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-149060> <Module oracle-bam-eventengine-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048437> <BEA-149059> <Module oracle-bam-reportcache-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-014021> <The EJB 'ReportCacheServerBean(Application: oracle-bam#11.1.1, EJBComponent: oracle-bam-reportcache-ejb.jar)' has been successfully deployed. The following remote interfaces have been bound into JNDI with the specified JNDI names:>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-014022> <******** oracle.bam.reportcache.ejb.ReportCacheServerRemote is bound with JNDI name:ReportCacheServerBean#oracle.bam.reportcache.ejb.ReportCacheServerRemote ********>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-010009> <EJB Deployed EJB with JNDI name oracle-bam#11_1_1oracle-bam-reportcache-ejb_jarReportCacheServerBean_Home.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-149060> <Module oracle-bam-reportcache-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048484> <BEA-149059> <Module oracle-bam-statuslistener-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-010009> <EJB Deployed EJB with JNDI name StatusListenerEJB.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-149060> <Module oracle-bam-statuslistener-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-149059> <Module sdpmessagingclient-ejb.jar of application oracle-bam [Version=11.1.1] is transitioning from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-010009> <EJB Deployed EJB with JNDI name MessagingClientBAM.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <EJB> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-010223> <EJB Deployed Message Driven Bean: MessageDispatcherBean>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Deployer> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048500> <BEA-149060> <Module sdpmessagingclient-ejb.jar of application oracle-bam [Version=11.1.1] successfully transitioned from STATE_PREPARED to STATE_ADMIN on server BAM2.>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <JDBC> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <OracleSystemUser> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048859> <BEA-001516> <Connection Pool "BAMDataSource" connected to Database: "Oracle", Version: "Oracle Database 11g Express Edition Release 11.2.0.2.0 - Production".>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <JDBC> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <OracleSystemUser> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048859> <BEA-001517> <Connection Pool "BAMDataSource" using Driver: "Oracle JDBC driver", Version: "11.2.0.2.0".>
    ####<Mar 8, 2012 12:34:08 PM EST> <Info> <Common> <humzalpt> <BAM2> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <OracleSystemUser> <> <0000JNnNotlFg4WFLzrI8A1FMEpP000001> <1331228048859> <BEA-000628> <Created "1" resources for pool "BAMDataSource", out of which "1" are available and "0" are unavailable.> Any help is appreciated.
    Thanks

  • Unable to bring up ASM on 2nd node of a 2-node Cluster

    Having a very wierd problem on a 2-node cluster. I can only bring up on ASM instance at a time. If i bring up the second, it hangs. This is what the second (hung) instance puts in the alert log:
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Picked latch-free SCN scheme 3
    Using LOG_ARCHIVE_DEST_1 parameter default value as /ORAUTL/oraasm/product/ASM/dbs/arch
    Autotune of undo retention is turned off.
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    ksdpec: called for event 13740 prior to event group initialization
    Starting up ORACLE RDBMS Version: 10.2.0.3.0.
    System parameters with non-default values:
    large_pool_size = 12582912
    instance_type = asm
    cluster_interconnects = 192.168.0.12
    cluster_database = TRUE
    instance_number = 2
    remote_login_passwordfile= EXCLUSIVE
    background_dump_dest = /ORAUTL/oraasm/admin/+ASM2/bdump
    user_dump_dest = /ORAUTL/oraasm/admin/+ASM2/udump
    core_dump_dest = /ORAUTL/oraasm/admin/+ASM2/cdump
    pga_aggregate_target = 0
    Cluster communication is configured to use the following interface(s) for this instance
    192.168.0.12
    Fri Nov 21 21:10:48 2008
    cluster interconnect IPC version:Oracle UDP/IP (generic)
    IPC Vendor 1 proto 2
    PMON started with pid=2, OS id=5428
    DIAG started with pid=3, OS id=5430
    PSP0 started with pid=4, OS id=5432
    LMON started with pid=5, OS id=5434
    LMD0 started with pid=6, OS id=5436
    LMS0 started with pid=7, OS id=5438
    MMAN started with pid=8, OS id=5442
    DBW0 started with pid=9, OS id=5444
    LGWR started with pid=10, OS id=5446
    CKPT started with pid=11, OS id=5448
    SMON started with pid=12, OS id=5458
    RBAL started with pid=13, OS id=5475
    GMON started with pid=14, OS id=5487
    Fri Nov 21 21:10:49 2008
    lmon registered with NM - instance id 2 (internal mem no 1)
    Fri Nov 21 21:10:49 2008
    Reconfiguration started (old inc 0, new inc 2)
    ASM instance
    List of nodes:
    0 1
    Global Resource Directory frozen
    Communication channels reestablished
    After this it hangs. i've checked everything. CRS is fine.
    I suspect its the kernel revision. This is a cluster of two v890's. Kernel rev is 127127-11. Anyone seen this issue ?
    thanks

    Responses in-line:
    Have you got any issue reported from Lock Monitor's (LMON) ? (those messages are in the alert.log are summaries of the reconfiguration event.
    No issues that I have seen. I see trc files on both nodes for lmon, but neither contain errors.Do you have any post issues on the date that issue began (something with Reconfiguration started) ?
    This is a new build. Its going to be a DR environment (Dataguard Physical Standby), so we've never managed to get ASM up yet.Do you have any other errors on the second node on the date the issue appears (some ORA-27041 or other messages) errors?
    No errors at all.What is the result of a crs_stat -t ?
    HA Resource Target State
    ora.vzdfwsdbp01.LISTENER_VZDFWSDBP01.lsnr ONLINE ONLINE on vzdfwsdbp01
    ora.vzdfwsdbp01.gsd ONLINE ONLINE on vzdfwsdbp01
    ora.vzdfwsdbp01.ons ONLINE ONLINE on vzdfwsdbp01
    ora.vzdfwsdbp01.vip ONLINE ONLINE on vzdfwsdbp01
    ora.vzdfwsdbp02.LISTENER_VZDFWSDBP02.lsnr ONLINE ONLINE on vzdfwsdbp02
    ora.vzdfwsdbp02.gsd ONLINE ONLINE on vzdfwsdbp02
    ora.vzdfwsdbp02.ons ONLINE ONLINE on vzdfwsdbp02
    ora.vzdfwsdbp02.vip ONLINE ONLINE on vzdfwsdbp02
    ASM isn't registered with CRS/OCR yet. I did add it at one time, but it didnt seem to make any difference.What is the release of your installation 10.2.0.4? Otherwise control if you can upgrade CRS, ASM and your RDBMS to that release.
    CRS, ASM and Oracle will be 10.2.0.3Can't go to 10.2.0.4 yet as primary site is at 10.2.0.3 on a live system.
    Can you please tell us what is the OS / Hardware in use?
    Solaris 10, Sun v890$ uname -a
    SunOS dbp02 5.10 Generic_127127-11 sun4u sparc SUNW,Sun-Fire-V890
    What is the result of that on the second node:
    even a startup nomount hangs on second node.connect sqlplus / as sysdba;
    startup nomount
    desc v$asmdiskgroup;
    select name, mount from v$diskgroup;
    In the case that no group is mounted do
    alter database mount diskgroup 'your diskgroupname';
    What is the result of that?
    thanks
    -toby

  • How to clone a 2-node ASM RAC db to a 3-node ASM RAC db by RMAN?

    We need to clone a 2-node ASM RAC db to a 3-node ASM RAC db by RMAN. I would think this will be about the same as a regular single db procedure by RMAN mostly. However, in 3 node RAC, there will be additional instance thread in redo (thread 1,2, and 3). Questions will be
    1. Should I use CRSCTL to add new instance (3rd)? -- I guess yes
    2. Should I create additional online redo group for thread 3? -- I guess yes
    3. Should I create addition undo tablespace? -- I guess yes
    Can any body provide a procedure of cloning the DB?
    Thanks!

    Not just easy as next, next, next
    version depenedencies and etc.
    [from Oracle DOC|http://download.oracle.com/docs/cd/B19306_01/rac.102/b14197/adddelunix.htm#BEICADHD]
    AND
    [Doc ID:      270512.1: Adding a Node to a 10g RAC Cluster|https://metalink.oracle.com/CSP/ui/flash.html#tab=KBHome%28page=KBHome&id=%28%29%29,%28page=KBNavigator&id=%28bmDocID=270512.1&from=BOOKMARK&bmDocType=BULLETIN&bmDocDsrc=KB&bmDocTitle=%3Cb%3EAdding%3C/b%3E%20a%20%3Cb%3ENode%3C/b%3E%20to%20a%2010g%20%3Cb%3ERAC%3C/b%3E%20Cluster&viewingMode=1143%29%29]

  • 2 node cluster windows 2008 R2 cluster won't form

    Hi,
    I'm trying to build a 2 node cluster and having no luck at all.   My validation works great but when I go to create the cluster it fails on the forming cluster section after about 2 minutes.   I seem to have everything in order and can't
    figure out the problem.   Here is what is in the cluster log.
    000010b0.00001108::2010/08/26-20:43:54.067 INFO  [CHANNEL 192.168.210.236:~59824~] graceful close, status (of previous failure, may not indicate problem) ERROR_SUCCESS(0)
    000010b0.00001108::2010/08/26-20:43:54.067 ERR   [SM] Caught GracefulClose(1226)' because of 'channel to remote endpoint 192.168.210.236:~59824~ is closed' authenticating.
    000010b0.000012d8::2010/08/26-20:43:54.067 INFO  [SV] Authentication and authorization were successful
    000010b0.000012d8::2010/08/26-20:43:54.067 DBG   [SM] SrvCtxt initialized with package Kerberos, MaxTokenSize = 12000, RequiredCtxAttrib = 133122, HandShakeTimeout = 30000
    000010b0.00001108::2010/08/26-20:43:54.067 DBG   [SM] Handling auth handshake posted by thread id 4824
    000010b0.00001108::2010/08/26-20:43:54.067 DBG   [CHANNEL 192.168.210.236:~59824~]/recv: Attempting to perform I/O on closed stream.
    000010b0.00001108::2010/08/26-20:43:54.067 DBG   [CHANNEL 192.168.210.236:~59824~] Not closing handle because it is invalid.
    000010b0.00001108::2010/08/26-20:43:54.067 INFO  [CHANNEL 192.168.210.236:~59824~] graceful close, status (of previous failure, may not indicate problem) ERROR_SUCCESS(0)
    000010b0.00001108::2010/08/26-20:43:54.067 ERR   [SM] Caught GracefulClose(1226)' because of 'channel to remote endpoint 192.168.210.236:~59824~ is closed' authenticating.
    000010b0.000012d8::2010/08/26-20:43:54.067 DBG   [CHANNEL 192.168.210.236:~59824~] Not closing handle because it is invalid.
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO  [CS] Service Stopping...
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO  [CORE] Node quorum state is 'Not yet formed or joined a cluster'. Form/join status with other nodes is as follows:
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO  [NODE] Node 1: Farthest reported progress joining with node iQCXSQL2 (id 2) is: Authenticate Initial Connection at time 2010/08/26-20:43:54.004: status 802 [SV] Authentication failed
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO  [CORE] Node 1: Proposed View is <ViewChanged joiners=() downers=(1) newView=101() oldView=001(1) joiner=false form=false/>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO  [RGP] Node 1: Stable_`0 => Opening`1
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO  <class mscs::detail::ConsensusMessage>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <senderId> 1 </senderId>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <bestEpochSeen> 0 </bestEpochSeen>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <lastStableView> 001(1) </lastStableView>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <proposedView> 101() </proposedView>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <stage> Opening`1 </stage>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <inStage> () </inStage>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <inStagePrev> () </inStagePrev>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <joiners> () </joiners>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <trimmedNodes> (1) </trimmedNodes>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <innerScreen> () </innerScreen>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <pruningResult> () </pruningResult>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <matrix>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO      <ConnectivityMatrix>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO        <row id='0'00000000000000000000000000000000/>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO      </ConnectivityMatrix>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    </matrix>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <gemBlob> <counted_ptr p = 'nullptr'/> </gemBlob>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO  </class mscs::detail::ConsensusMessage>
    000010b0.000006fc::2010/08/26-20:44:52.906 INFO  [RGP] Node 1: Timer Tick Started
    000010b0.0000106c::2010/08/26-20:44:52.906 DBG   [NODE] Node 1: eating message sent to the dead node 2
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO  [RGP] Node 1: Just sent farewell regroup message
    000010b0.000010b8::2010/08/26-20:44:52.906 DBG   [NODE] Node 1: just about to send a message of size 0 to 1
    000010b0.000010b8::2010/08/26-20:44:52.906 DBG   [NODE] Node 1: message to node 1 sent
    000010b0.000010b8::2010/08/26-20:44:52.906 DBG   [NODE] Node 1: just about to send a message of size 361 to 1
    000010b0.000010b8::2010/08/26-20:44:52.906 DBG   [NODE] Node 1: message to node 1 sent
    000010b0.000006fc::2010/08/26-20:44:53.218 INFO  [RGP] Node 1: Timer Tick Started
    000010b0.0000106c::2010/08/26-20:44:53.420 INFO  [DM]: Shutting down, so unloading the cluster database.
    000010b0.0000106c::2010/08/26-20:44:53.420 INFO  [DM] Shutting down, so unloading the cluster database (waitForLock: true).
    000010b0.0000106c::2010/08/26-20:44:53.420 DBG   [DM] Unloading Hive, Key \Registry\Machine\Cluster, discardCurrentChanges true
    000010b0.0000106c::2010/08/26-20:44:53.420 INFO  [CS] Service Stopped...
    000010b0.0000106c::2010/08/26-20:44:53.420 INFO  [CS] About to exit service...
    Just to make sure I have given both nodes every permission under the sun.  I have tried creating the cluster computer in AD beforehand and disabling the computer and I have tried letting clustering services create the computer to no avail.  
    Please help!

    Here is the complete log Edwin.   Thank you.   
    000010b0.0000121c::2010/08/26-20:41:52.722 INFO  -----------------------------+ LOG BEGIN +-----------------------------
    000010b0.0000121c::2010/08/26-20:41:52.722 INFO  [CS] Starting clussvc as a service
    000010b0.0000121c::2010/08/26-20:41:52.722 INFO  [CS] cluster service logging level is 5
    000010b0.000012d0::2010/08/26-20:41:52.738 INFO  [CS] Creating cluster node <vector len='1'>
    000010b0.000012d0::2010/08/26-20:41:52.738 INFO      <item>ClusSvc</item>
    000010b0.000012d0::2010/08/26-20:41:52.738 INFO  </vector>
    000010b0.00000fd8::2010/08/26-20:41:52.738 INFO  [StartupConfig]: Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.738 INFO  [StartupConfig]: Local node name: iQCXSQL1.
    000010b0.00000fd8::2010/08/26-20:41:52.738 INFO  [StartupConfig]: Initializing NetFT adapter management information.
    000010b0.00000fd8::2010/08/26-20:41:52.738 INFO  [NETFT] Initializing NetFT driver manager
    000010b0.00000fd8::2010/08/26-20:41:52.738 INFO  [CORE] Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.738 INFO  [GEM] Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.738 INFO  [MM] Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.738 INFO  [MRR] Node 1: Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.738 INFO  [Causal] Node 1: Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.738 INFO  [EM] Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.738 INFO  [GUM] Node 1: Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.738 INFO  [DM] Node 1: Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.800 INFO  [CLFS] Initializing CLFS logger.
    000010b0.00000fd8::2010/08/26-20:41:52.800 DBG   [CLFS] Largest written record has size 131072 bytes
    000010b0.00000fd8::2010/08/26-20:41:52.800 DBG   [CLFS] Deleting old marshalling area.
    000010b0.00000fd8::2010/08/26-20:41:52.800 DBG   [CLFS] Marshaling area resized
    000010b0.00000fd8::2010/08/26-20:41:52.800 INFO  [TM] Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.800 INFO  [JPM] Node 1: Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.800 INFO  [VER]: Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.800 INFO  [CCM] Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.800 INFO  [FTI] Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.800 INFO  [ReM] Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.800 INFO  [HM]: Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.800 INFO  [NETFT] Starting NetFT eventing for FTI
    000010b0.00000fd8::2010/08/26-20:41:52.816 DBG   [NETFTAPI] received NsiInitialNotification
    000010b0.00000fd8::2010/08/26-20:41:52.816 DBG   [NETFTAPI] received NsiInitialNotification
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO  [NETFT] FTI NetFT event handler ready for events.
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO  [IM] Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO  [NM] Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO  [NSM] Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO  [CM] Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO  [RCM] Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO  [QUORUM] Node 1: Initializing.
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO  [STARTUP] Node 1: startup config
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO  <StartupConfig localNodeId='1' testMode='false' newCluster='true'
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO      <ClusterName> iqcxcluster </ClusterName>
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO      <PaxosTag> 0:0:0 </PaxosTag>
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO      <ConfiguredNodes> (1 2) </ConfiguredNodes>
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO      <NodeWeights> (1 2) </NodeWeights>
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO      <Nodes>
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO          <Node name='iQCXSQL1' id='1'/>
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO          <Node name='iQCXSQL2' id='2'/>
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO      </Nodes>
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO      <LocalHiveFile> C:\Windows\Cluster\CLUSDB </LocalHiveFile>
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO      <LocalHiveKey> Cluster </LocalHiveKey>
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO      <ParametersKey> System\\CurrentControlSet\\Services\ClusSvc\Parameters </ParametersKey>
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO      <Virtual IPv6> fe80::c14f:8b1f:d265:9b42%22:~3343~ </Virtual IPv6>
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO      <Virtual IPv4> 169.254.1.247:~3343~ </Virtual IPv4>
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO      <NoVersionCheck> false </NoVersionCheck>
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO      <Cookie> 3529ace1-67e4-4107-8940-c725883fab51 </Cookie>
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO      <ClusterGuid> d4a97f31-178f-4726-993c-c0c5ff8407dc </ClusterGuid>
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO      <RetainNetFTConnSecurity> false </RetainNetFTConnSecurity>
    000010b0.00000fd8::2010/08/26-20:41:52.816 INFO  </StartupConfig>
    000010b0.000012d0::2010/08/26-20:41:52.816 INFO  [CS] Node object creation completed.
    000010b0.000012d0::2010/08/26-20:41:52.816 INFO  [CS] Starting cluster node
    000010b0.000012d0::2010/08/26-20:41:52.816 INFO  [CS] Cluster Service started
    000010b0.000012d0::2010/08/26-20:41:52.816 INFO  [CS] The current time is 2010/08/26-16:41:52.816
    000010b0.000012d0::2010/08/26-20:41:52.816 INFO  [CS] Using virtual IPv6 Address fe80::c14f:8b1f:d265:9b42%22:~3343~
    000010b0.000012d0::2010/08/26-20:41:52.816 INFO  [CS] Using virtual IPv4 Address 169.254.1.247:~3343~
    000010b0.000012d0::2010/08/26-20:41:52.816 INFO  [CS] Max Licensed nodes is set 16
    000010b0.000012d0::2010/08/26-20:41:52.816 INFO  [NETFT] Disabling IP autoconfiguration on the NetFT adapter.
    000010b0.000012d0::2010/08/26-20:41:52.816 INFO  [NETFT] Disabling DHCP on the NetFT adapter.
    000010b0.000012d0::2010/08/26-20:41:52.816 DBG   [NETFT] Disabling DHCP on NetFT interface name ethernet_2.
    000010b0.000012d0::2010/08/26-20:41:52.816 INFO  [CS] Starting DM
    000010b0.000012d0::2010/08/26-20:41:52.816 INFO  [DM] Node 1: Reading quorum config
    000010b0.000012d0::2010/08/26-20:41:52.816 INFO  [DM] Node 1: new cluster. generating new hive
    000010b0.000012d0::2010/08/26-20:41:52.816 DBG   [DM] Unloading Hive, Key \Registry\Machine\Cluster, discardCurrentChanges true
    000010b0.000012d8::2010/08/26-20:41:52.816 DBG   [NETFTAPI] received NsiAddInstance  for 169.254.1.247
    000010b0.000010a0::2010/08/26-20:41:52.816 INFO  [CS] Disabling connection security.
    000010b0.000012d0::2010/08/26-20:41:52.816 INFO  [DM] Key \Registry\Machine\Cluster does not appear to be loaded (status STATUS_OBJECT_NAME_NOT_FOUND(c0000034))
    000010b0.000012d0::2010/08/26-20:41:52.816 DBG   [DM] Unloading Hive, Key \Registry\Machine\Cluster, discardCurrentChanges true
    000010b0.000012d0::2010/08/26-20:41:52.816 INFO  [DM] Key \Registry\Machine\Cluster does not appear to be loaded (status STATUS_OBJECT_NAME_NOT_FOUND(c0000034))
    000010b0.000012d0::2010/08/26-20:41:52.816 DBG   [DM] Creating Hive, Key Cluster, FilePath C:\Windows\Cluster\CLUSDB
    000010b0.000010a0::2010/08/26-20:41:52.816 INFO  [NETFT] Adding connection security permit filters for the NetFT adapter.
    000010b0.000010a0::2010/08/26-20:41:52.816 INFO  [NETFT] Adding FWPM filter Failover Cluster Permit Inbound Transport IPv4
    000010b0.000010a0::2010/08/26-20:41:52.831 INFO  [NETFT] Adding FWPM filter Failover Cluster Permit Inbound Transport IPv6
    000010b0.000010a0::2010/08/26-20:41:52.831 INFO  [NETFT] Adding FWPM filter Failover Cluster Permit Outbound Transport IPv4
    000010b0.000010a0::2010/08/26-20:41:52.831 INFO  [NETFT] Adding FWPM filter Failover Cluster Permit Outbound Transport IPv6
    000010b0.000010a0::2010/08/26-20:41:52.831 INFO  [NETFT] Adding FWPM filter Failover Cluster Permit ALE Accept IPv4 Universal
    000010b0.000010a0::2010/08/26-20:41:52.831 INFO  [NETFT] Adding FWPM filter Failover Cluster Permit ALE Accept IPv6 Universal
    000010b0.000010a0::2010/08/26-20:41:52.831 INFO  [NETFT] Adding FWPM filter Failover Cluster Permit ALE Accept IPv4 LIPS
    000010b0.000010a0::2010/08/26-20:41:52.831 INFO  [NETFT] Adding FWPM filter Failover Cluster Permit ALE Accept IPv6 LIPS
    000010b0.000010a0::2010/08/26-20:41:52.831 DBG   [CS] Done with connection security
    000010b0.000012d0::2010/08/26-20:41:52.847 DBG   [DM] Saving temp Hive
    000010b0.000012d0::2010/08/26-20:41:52.847 DBG   [DM] Unloading temp Hive
    000010b0.000012d0::2010/08/26-20:41:52.847 DBG   [DM] Deleting temp Hive
    000010b0.000012d0::2010/08/26-20:41:52.847 INFO  [DM] Loading Hive, Key Cluster, FilePath C:\Windows\Cluster\CLUSDB
    000010b0.000012d0::2010/08/26-20:41:52.878 INFO  [DM] Paxos Tag Read from Hive: 0:0:0
    000010b0.000012d0::2010/08/26-20:41:52.878 DBG   [DM] Beginning transaction
    000010b0.000012d0::2010/08/26-20:41:52.878 INFO  [QUORUM] unique node id for new node 1 is 00000000-0000-0000-0000-000000000001
    000010b0.000012d0::2010/08/26-20:41:52.878 INFO  [QUORUM] unique node id for new node 2 is 00000000-0000-0000-0000-000000000002
    000010b0.000012d0::2010/08/26-20:41:52.878 DBG   [CLFS] Deleting old marshalling area.
    000010b0.000012d0::2010/08/26-20:41:52.878 DBG   [CLFS] Marshaling area resized
    000010b0.000012d0::2010/08/26-20:41:52.878 DBG   [CLFS] Logger wrote 2820 bytes
    000010b0.000012d0::2010/08/26-20:41:52.878 DBG   [DM] Committed transaction 0:0:1
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [DM] Populating StartupConfig data from local hive.
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [DM] Node 1: startup configuration loaded:
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  <StartupConfig localNodeId='1' testMode='false' newCluster='true'
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <ClusterName> iqcxcluster </ClusterName>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <PaxosTag> 0:0:1 </PaxosTag>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <ConfiguredNodes> (1 2) </ConfiguredNodes>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <NodeWeights> (1 2) </NodeWeights>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <Nodes>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO          <Node name='iQCXSQL1' id='1'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO          <Node name='iQCXSQL2' id='2'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      </Nodes>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <LocalHiveFile> C:\Windows\Cluster\CLUSDB </LocalHiveFile>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <LocalHiveKey> Cluster </LocalHiveKey>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <ParametersKey> System\\CurrentControlSet\\Services\ClusSvc\Parameters </ParametersKey>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <Virtual IPv6> fe80::c14f:8b1f:d265:9b42%22:~3343~ </Virtual IPv6>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <Virtual IPv4> 169.254.1.247:~3343~ </Virtual IPv4>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <NoVersionCheck> false </NoVersionCheck>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <Cookie> 3529ace1-67e4-4107-8940-c725883fab51 </Cookie>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <ClusterGuid> d4a97f31-178f-4726-993c-c0c5ff8407dc </ClusterGuid>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <RetainNetFTConnSecurity> false </RetainNetFTConnSecurity>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  </StartupConfig>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [CS] Starting MM
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [MM] Starting.
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [GEM] Node 1: Resume (newView = 001(1), previousView = 000())
    000010b0.000012d0::2010/08/26-20:41:52.894 DBG   [GEM] Node 1: SendNextMessage
    000010b0.000012d0::2010/08/26-20:41:52.894 DBG   [GEM] Node 1: No more messages in queue
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [StartupConfig] Writing new StartupConfig entries to ObjectManager
    000010b0.000012d0::2010/08/26-20:41:52.894 DBG   [StartupConfig] Writing node iQCXSQL2 (id = 2) to ObjectManager
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [QUORUM] unique node id for new node 2 is 00000000-0000-0000-0000-000000000002
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [MQ-iQCXSQL2] Starting
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [MQ-iQCXSQL2] Clearing 0 unsent and 0 unacknowledged messages.
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  <ObjectManager>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO    <ROOT id='.root' name='.root'>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <GEMA id='GemAgent' name='GemAgent1'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <MRRA id='MrrAgent1' name='MrrAgent1'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <CAUS id='CausalAgent' name='CausalAgent'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <EVTM id='EventManager' name='EventManager'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <GUMA id='GumAgent1' name='GumAgent1'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <JPMA id='JpmAgent' name='JpmAgent'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <IMSC id='[IM-1] ' name='[IM-1] '/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <RPCS id='[RPC-1] ' name='[RPC-1] '/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <RCMA id='RcmAgent' name='RcmAgent'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <QRMA id='QuorumAgent1' name='QuorumAgent1'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <DCMA id='DcmAgent' name='DcmAgent'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <VSSR id='Cluster VSS Writer' name='Cluster VSS Writer'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <CLUS id='iqcxcluster' name='iqcxcluster'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <DEAD id='.Dead' name='.Dead'>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO        <NODE id='2' name='iQCXSQL2'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      </DEAD>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <LIVE id='.Live' name='.Live'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <JOIN id='.Joiners' name='.Joiners'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO      <MBRA id='MembershipAgent' name='MembershipAgent'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO    </ROOT>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  </ObjectManager>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [MM] Node 1: got a network stream to 1. Creating new NodeObject
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [QUORUM] unique node id for new node 1 is 00000000-0000-0000-0000-000000000001
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [MQ-iQCXSQL1] Starting
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [MQ-iQCXSQL1] Clearing 0 unsent and 0 unacknowledged messages.
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [NODE] Node 1: n1 node object adding stream
    000010b0.000012d0::2010/08/26-20:41:52.894 DBG   [NODE] Node 1: n1 node object got a channel
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [NODE] Node 1: Highest version with n1 = Major 0 Minor 0 Upgrade 0 ClusterVersion 0x00000000, lowest = Major 0 Minor 0 Upgrade 0 ClusterVersion 0x00000000
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [NODE] Node 1: Done processing new stream to n1.
    000010b0.00000e50::2010/08/26-20:41:52.894 INFO  [PULLER iQCXSQL1] Just about to start reading from <LoopbackStream name='iQCXSQL1'/>
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [CS] Starting QM
    000010b0.000012d0::2010/08/26-20:41:52.894 DBG   [NETFT] Halt Notification set
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [CS] Starting TM
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [NETFT] Starting NetFT eventing for TM
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [NETFT] TM NetFT event handler ready for events.
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [CS] Starting IM
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [NETFT] Starting NetFT eventing for IM
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [NETFT] IM NetFT event handler ready for events.
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [CS] Starting NM
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [CS] Starting api server
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [API] Api Onlined
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [API] UseProtseq ncalrpc => 0
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [API] UseProtseq ncadg_ip_tcp => 0
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [API] InqBindings => 0
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [API] RpcEpRegister => 0
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [API] RpcServerRegister => 0
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [API] Initializing
    000010b0.000012d0::2010/08/26-20:41:52.894 INFO  [API] Adding Mgmt API SPNs to node's Computer Object
    000010b0.000012d0::2010/08/26-20:41:52.972 INFO  [API] Successfully added Mgmt API SPNs to node's Computer Object
    000010b0.000012d0::2010/08/26-20:41:52.972 ERR   [API] DmQueryString failed to retrieve the security   descriptor status 2, default security descriptor will be used for authorizing client connections
    000010b0.000012d0::2010/08/26-20:41:52.972 INFO  [API] Successfully built the default cluster SD.
    000010b0.000012d0::2010/08/26-20:41:52.972 INFO  [CS] Bringing API Online
    000010b0.000012d0::2010/08/26-20:41:52.972 INFO  [CORE] Node 1: New View is <ViewChanged joiners=(1) downers=() newView=001(1) oldView=000() joiner=true form=true/> (Start Dispatch)
    000010b0.000012d0::2010/08/26-20:41:52.972 INFO  [MRR] Node 1: Process view 001(1)
    000010b0.000012d0::2010/08/26-20:41:52.972 INFO  [RCM] rcm::RcmAgent::ProcessView: Ignoring view change (won't attempt to move orphaned groups), since I'm not in RCM view yet.
    000010b0.000012d0::2010/08/26-20:41:52.972 DBG   [GEM] Node 1: message added to send queue
    000010b0.000012d0::2010/08/26-20:41:52.972 DBG   [GEM] Node 1: SendNextMessage
    000010b0.000012d0::2010/08/26-20:41:52.972 DBG   [GEM] Node 1: delivering message 1:1
    000010b0.000012d0::2010/08/26-20:41:52.972 INFO  [CORE] EmitReadyForState for node 1
    000010b0.000012d0::2010/08/26-20:41:52.972 INFO  [CORE] Issuing NeedStateViewChanged((1))
    000010b0.000012d0::2010/08/26-20:41:52.972 DBG   [GEM] Node 1: ProcessAck from node 1 (already acked ())
    000010b0.000012d0::2010/08/26-20:41:52.972 DBG   [GEM] Node 1: SendNextMessage
    000010b0.000012d0::2010/08/26-20:41:52.972 DBG   [GEM] Node 1: No more messages in queue
    000010b0.000012d0::2010/08/26-20:41:52.972 DBG   [NODE] Node 1: eating message sent to the dead node 2
    000010b0.000012d0::2010/08/26-20:41:52.972 INFO  [QUORUM] Node 1: I am the form/join coordinator (offline and lowest number node). Launching worker to request quorum config from other nodes
    000010b0.000012d0::2010/08/26-20:41:52.972 DBG   [MRR] Node 1: insert request record for RID 1
    000010b0.000012d0::2010/08/26-20:41:52.972 DBG   [MRR] SendRequest(<message action='qrm/request-info' GemId='0' target='QRMA' sender='-1'mrr='1'/>)
    000010b0.000012d0::2010/08/26-20:41:52.972 DBG   [CORE] Node 1: sending qrm/request-info to QRMA at iQCXSQL1
    000010b0.000010b8::2010/08/26-20:41:52.972 DBG   [NODE] Node 1: just about to send a message of size 109 to 1
    000010b0.000012d0::2010/08/26-20:41:52.972 INFO  [CORE] Node 1: New View is <ViewChanged joiners=(1) downers=() newView=001(1) oldView=000() joiner=true form=true/> (Stop Dispatch)
    000010b0.000012d0::2010/08/26-20:41:52.972 INFO  [CS] Starting HM Server
    000010b0.000010b8::2010/08/26-20:41:52.972 DBG   [NODE] Node 1: message to node 1 sent
    000010b0.000012d0::2010/08/26-20:41:52.972 INFO  [NODE] Node 1: New join with n2: stage: 'Listen for Connection'
    000010b0.000012d0::2010/08/26-20:41:52.972 INFO  [CS] Starting JPM (attempting connections to other cluster nodes)
    000010b0.000012d0::2010/08/26-20:41:52.972 DBG   [JPM] Node 1: Dead node mentioned iQCXSQL1
    000010b0.000012d0::2010/08/26-20:41:52.972 DBG   [JPM] Node 1: Dead node mentioned iQCXSQL2
    000010b0.000012d0::2010/08/26-20:41:52.972 INFO  [CS] Subscribing for VSS requests
    000010b0.000012d0::2010/08/26-20:41:52.972 INFO  [VSS] Initializing Cluster VSS Writer at the command of the VSS framework.
    000010b0.00000e50::2010/08/26-20:41:52.972 INFO  [QUORUM] Node 1: received request for quorum info. Replying with quorum state <QuorumConfig tag='0:0:1' set='(1 2)' weights='(1 2)'/>
    000010b0.00000e50::2010/08/26-20:41:52.972 DBG   [CORE] Node 1: sending mrr/reply to MRRA at iQCXSQL1
    000010b0.000010b8::2010/08/26-20:41:52.972 DBG   [NODE] Node 1: just about to send a message of size 0 to 1
    000010b0.000010b8::2010/08/26-20:41:52.972 DBG   [NODE] Node 1: message to node 1 sent
    000010b0.000012d8::2010/08/26-20:41:52.972 DBG   [JPM] Node 1: Building contact data for node iQCXSQL2.
    000010b0.000010b8::2010/08/26-20:41:52.972 DBG   [NODE] Node 1: just about to send a message of size 278 to 1
    000010b0.000010b8::2010/08/26-20:41:52.972 DBG   [NODE] Node 1: message to node 1 sent
    000010b0.000012d8::2010/08/26-20:41:52.972 INFO  [JPM] Node 1: Scheduling DNS query for node name iQCXSQL2
    000010b0.000012d8::2010/08/26-20:41:52.972 DBG   [JPM] Node 1: no valid contact endpoints yet for node iQCXSQL2.
    000010b0.00000e50::2010/08/26-20:41:52.972 DBG   [MRR] Node 1: processing reply <message action='mrr/reply' GemId='0' target='MRRA' sender='1'mrr='1'/>
    000010b0.00000e50::2010/08/26-20:41:52.972 DBG   [MRR] Node 1: Request RID 1 is completed
    000010b0.0000119c::2010/08/26-20:41:52.972 INFO  [QUORUM] Node 1: setting next best epoch to 0
    000010b0.0000119c::2010/08/26-20:41:52.972 DBG   [QC] Calculating quorum: Configured nodes count = 2, Node weights count = 2, Quorum witness configured = false, Intersection count = 1
    000010b0.0000119c::2010/08/26-20:41:52.972 INFO  [QUORUM] Node 1: Coordinator: no quorum yet. do nothing
    000010b0.000012d8::2010/08/26-20:41:52.972 INFO  [JPM] Node 1: Found possible contact address 192.168.210.236:~0~ for node iQCXSQL2 via DNS.
    000010b0.000012d0::2010/08/26-20:41:52.987 INFO  [CS] Starting the Watchdog Monitor
    000010b0.000012d0::2010/08/26-20:41:52.987 INFO  [WM] Monitoring the kernel-mode watchdog.
    000010b0.000012d0::2010/08/26-20:41:52.987 INFO  [CS] Reporting to SCM that cluster service has started.
    000010b0.000012d8::2010/08/26-20:41:53.019 INFO  [NM] Received request from client address ::1.
    000010b0.000012d8::2010/08/26-20:41:53.019 DBG   [API] Authenticated client--Client: IQCX\iqcxsql1 Interface: b97db8b2-4c63-11cf-bff6-08002be23f2f Server:
    [email protected] Level: RPC_C_AUTHN_LEVEL_PKT_PRIVACY Service: RPC_C_AUTHN_WINNT Protocol Sequence: ncacn_ip_tcp Client Address: ::1 Network Option: .
    000010b0.0000119c::2010/08/26-20:41:53.986 DBG   [JPM] Node 1: contacts size for node iQCXSQL2 is 1, current index 0
    000010b0.0000119c::2010/08/26-20:41:53.986 DBG   [JPM] Node 1: Trying to connect to node iQCXSQL2 (IP: 192.168.210.236:~0~)
    000010b0.0000119c::2010/08/26-20:41:53.986 DBG   [HM] Trying to connect to iQCXSQL2 at 192.168.210.236:~3343~
    000010b0.000012d8::2010/08/26-20:41:53.986 INFO  [CONNECT] 192.168.210.236:~3343~: Established connection to remote endpoint 192.168.210.236:~3343~.
    000010b0.000012d8::2010/08/26-20:41:53.986 INFO  [NODE] Node 1: New join with n2: stage: 'Attempt Initial Connection'
    000010b0.000012d8::2010/08/26-20:41:53.986 INFO  [SV] Securing route from (192.168.210.239:~49538~) to remote iQCXSQL2 (192.168.210.236:~3343~).
    000010b0.000012d8::2010/08/26-20:41:53.986 INFO  [SV] Got a new outgoing stream to iQCXSQL2 at 192.168.210.236:~3343~
    000010b0.000012d8::2010/08/26-20:41:53.986 DBG   [SM] Joiner: Initialized with SPN = iQCXSQL2, Package = Kerberos, RequiredCtxAttrib = 83990, HandShakeTimeout = 30000
    000010b0.000013f0::2010/08/26-20:41:53.986 DBG   [SM] Handling auth handshake posted by thread id 4824
    000010b0.000013f0::2010/08/26-20:41:53.986 DBG   [SM] Joiner: ISC returned status = 590610 output Blob size 1388
    000010b0.000013f0::2010/08/26-20:41:53.986 DBG   [SM] Joiner: Received SSPI blob from the Sponsor of size 96
    000010b0.000013f0::2010/08/26-20:41:53.986 DBG   [SM] Joiner: ISC returned status = 590610 output Blob size 1388
    000010b0.000013f0::2010/08/26-20:41:53.986 DBG   [SM] Joiner: Received SSPI blob from the Sponsor of size 96
    000010b0.000013f0::2010/08/26-20:41:53.986 DBG   [SM] Joiner: ISC returned status = -2146893022 output Blob size 12000
    000010b0.000012d8::2010/08/26-20:41:53.986 INFO  [NODE] Node 1: New join with n2: stage: 'Authenticate Initial Connection' status HrError(0x80090322) reason: '[SV] Authentication failed'
    000010b0.000012d8::2010/08/26-20:41:53.986 DBG   [CHANNEL 192.168.210.236:~3343~] Close().
    000010b0.000012d8::2010/08/26-20:41:53.986 WARN  cxl::ConnectWorker::operator (): HrError(0x80090322)' because of '[SV] Authentication or Authorization Failed'
    000010b0.00000ea8::2010/08/26-20:41:54.048 INFO  [ACCEPT] 0.0.0.0:~3343~: Accepted inbound connection from remote endpoint 192.168.210.236:~59820~.
    000010b0.000012d8::2010/08/26-20:41:54.048 INFO  [SV] Securing route from (192.168.210.239:~3343~) to remote  (192.168.210.236:~59820~).
    000010b0.000012d8::2010/08/26-20:41:54.048 INFO  [SV] Got a new incoming stream from 192.168.210.236:~59820~
    000010b0.000012d8::2010/08/26-20:41:54.048 DBG   [SM] SrvCtxt initialized with package Kerberos, MaxTokenSize = 12000, RequiredCtxAttrib = 165910, HandShakeTimeout = 30000
    000010b0.00001108::2010/08/26-20:41:54.048 DBG   [SM] Handling auth handshake posted by thread id 4824
    000010b0.00001108::2010/08/26-20:41:54.048 DBG   [SM] Sponsor: Received SSPI blob from the Joiner of size 1411
    000010b0.00001108::2010/08/26-20:41:54.048 DBG   [SM] Sponsor: SSPI ASC returned status = 590610
    000010b0.00001108::2010/08/26-20:41:54.048 DBG   [SM] Sponsor: Sending SSPI blob of size 96 to Joiner
    000010b0.00001108::2010/08/26-20:41:54.048 DBG   [SM] Sponsor: Received SSPI blob from the Joiner of size 1411
    000010b0.00001108::2010/08/26-20:41:54.048 DBG   [SM] Sponsor: SSPI ASC returned status = 590610
    000010b0.00001108::2010/08/26-20:41:54.048 DBG   [SM] Sponsor: Sending SSPI blob of size 96 to Joiner
    000010b0.00001108::2010/08/26-20:41:54.048 DBG   [SM] Sponsor: Received SSPI blob from the Joiner of size 12000
    000010b0.00001108::2010/08/26-20:41:54.048 DBG   [SM] Sponsor: SSPI ASC returned status = 590610
    000010b0.00001108::2010/08/26-20:41:54.048 DBG   [SM] Sponsor: Sending SSPI blob of size 115 to Joiner
    000010b0.00001108::2010/08/26-20:41:54.048 DBG   [CHANNEL 192.168.210.236:~59820~] Close().
    000010b0.00001108::2010/08/26-20:41:54.048 INFO  [CHANNEL 192.168.210.236:~59820~] graceful close, status (of previous failure, may not indicate problem) ERROR_SUCCESS(0)
    000010b0.00001108::2010/08/26-20:41:54.048 ERR   [SM] Caught GracefulClose(1226)' because of 'channel to remote endpoint 192.168.210.236:~59820~ is closed' authenticating.
    000010b0.000012d8::2010/08/26-20:41:54.048 INFO  [SV] Authentication and authorization were successful
    000010b0.000012d8::2010/08/26-20:41:54.048 DBG   [SM] SrvCtxt initialized with package Kerberos, MaxTokenSize = 12000, RequiredCtxAttrib = 133122, HandShakeTimeout = 30000
    000010b0.00001108::2010/08/26-20:41:54.048 DBG   [SM] Handling auth handshake posted by thread id 4824
    000010b0.00001108::2010/08/26-20:41:54.048 DBG   [CHANNEL 192.168.210.236:~59820~]/recv: Attempting to perform I/O on closed stream.
    000010b0.00001108::2010/08/26-20:41:54.048 DBG   [CHANNEL 192.168.210.236:~59820~] Not closing handle because it is invalid.
    000010b0.00001108::2010/08/26-20:41:54.048 INFO  [CHANNEL 192.168.210.236:~59820~] graceful close, status (of previous failure, may not indicate problem) ERROR_SUCCESS(0)
    000010b0.00001108::2010/08/26-20:41:54.048 ERR   [SM] Caught GracefulClose(1226)' because of 'channel to remote endpoint 192.168.210.236:~59820~ is closed' authenticating.
    000010b0.000012d8::2010/08/26-20:41:54.048 DBG   [CHANNEL 192.168.210.236:~59820~] Not closing handle because it is invalid.
    000010b0.00001310::2010/08/26-20:42:53.995 DBG   [JPM] Node 1: contacts size for node iQCXSQL2 is 1, current index 0
    000010b0.00001310::2010/08/26-20:42:53.995 DBG   [JPM] Node 1: Trying to connect to node iQCXSQL2 (IP: 192.168.210.236:~0~)
    000010b0.00001310::2010/08/26-20:42:53.995 DBG   [HM] Trying to connect to iQCXSQL2 at 192.168.210.236:~3343~
    000010b0.00001310::2010/08/26-20:42:53.995 INFO  [CONNECT] 192.168.210.236:~3343~: Established connection to remote endpoint 192.168.210.236:~3343~.
    000010b0.00001310::2010/08/26-20:42:53.995 INFO  [SV] Securing route from (192.168.210.239:~49561~) to remote iQCXSQL2 (192.168.210.236:~3343~).
    000010b0.00001310::2010/08/26-20:42:53.995 INFO  [SV] Got a new outgoing stream to iQCXSQL2 at 192.168.210.236:~3343~
    000010b0.00001310::2010/08/26-20:42:53.995 DBG   [SM] Joiner: Initialized with SPN = iQCXSQL2, Package = Kerberos, RequiredCtxAttrib = 83990, HandShakeTimeout = 30000
    000010b0.000013f0::2010/08/26-20:42:53.995 DBG   [SM] Handling auth handshake posted by thread id 4880
    000010b0.000013f0::2010/08/26-20:42:53.995 DBG   [SM] Joiner: ISC returned status = 590610 output Blob size 1388
    000010b0.000013f0::2010/08/26-20:42:53.995 DBG   [SM] Joiner: Received SSPI blob from the Sponsor of size 96
    000010b0.000013f0::2010/08/26-20:42:53.995 DBG   [SM] Joiner: ISC returned status = 590610 output Blob size 1388
    000010b0.000013f0::2010/08/26-20:42:53.995 DBG   [SM] Joiner: Received SSPI blob from the Sponsor of size 96
    000010b0.000013f0::2010/08/26-20:42:53.995 DBG   [SM] Joiner: ISC returned status = -2146893022 output Blob size 12000
    000010b0.00001310::2010/08/26-20:42:53.995 INFO  [NODE] Node 1: New join with n2: stage: 'Authenticate Initial Connection' status HrError(0x80090322) reason: '[SV] Authentication failed'
    000010b0.00001310::2010/08/26-20:42:53.995 DBG   [CHANNEL 192.168.210.236:~3343~] Close().
    000010b0.00001310::2010/08/26-20:42:53.995 WARN  cxl::ConnectWorker::operator (): HrError(0x80090322)' because of '[SV] Authentication or Authorization Failed'
    000010b0.00000ea8::2010/08/26-20:42:54.057 INFO  [ACCEPT] 0.0.0.0:~3343~: Accepted inbound connection from remote endpoint 192.168.210.236:~59822~.
    000010b0.00001310::2010/08/26-20:42:54.057 INFO  [SV] Securing route from (192.168.210.239:~3343~) to remote  (192.168.210.236:~59822~).
    000010b0.00001310::2010/08/26-20:42:54.057 INFO  [SV] Got a new incoming stream from 192.168.210.236:~59822~
    000010b0.00001310::2010/08/26-20:42:54.057 DBG   [SM] SrvCtxt initialized with package Kerberos, MaxTokenSize = 12000, RequiredCtxAttrib = 165910, HandShakeTimeout = 30000
    000010b0.00001108::2010/08/26-20:42:54.057 DBG   [SM] Handling auth handshake posted by thread id 4880
    000010b0.00001108::2010/08/26-20:42:54.057 DBG   [SM] Sponsor: Received SSPI blob from the Joiner of size 1411
    000010b0.00001108::2010/08/26-20:42:54.057 DBG   [SM] Sponsor: SSPI ASC returned status = 590610
    000010b0.00001108::2010/08/26-20:42:54.057 DBG   [SM] Sponsor: Sending SSPI blob of size 96 to Joiner
    000010b0.00001108::2010/08/26-20:42:54.057 DBG   [SM] Sponsor: Received SSPI blob from the Joiner of size 1411
    000010b0.00001108::2010/08/26-20:42:54.057 DBG   [SM] Sponsor: SSPI ASC returned status = 590610
    000010b0.00001108::2010/08/26-20:42:54.057 DBG   [SM] Sponsor: Sending SSPI blob of size 96 to Joiner
    000010b0.00001108::2010/08/26-20:42:54.057 DBG   [SM] Sponsor: Received SSPI blob from the Joiner of size 12000
    000010b0.00001108::2010/08/26-20:42:54.057 DBG   [SM] Sponsor: SSPI ASC returned status = 590610
    000010b0.00001108::2010/08/26-20:42:54.057 DBG   [SM] Sponsor: Sending SSPI blob of size 115 to Joiner
    000010b0.00001108::2010/08/26-20:42:54.057 DBG   [CHANNEL 192.168.210.236:~59822~] Close().
    000010b0.00001108::2010/08/26-20:42:54.057 INFO  [CHANNEL 192.168.210.236:~59822~] graceful close, status (of previous failure, may not indicate problem) ERROR_SUCCESS(0)
    000010b0.00001108::2010/08/26-20:42:54.057 ERR   [SM] Caught GracefulClose(1226)' because of 'channel to remote endpoint 192.168.210.236:~59822~ is closed' authenticating.
    000010b0.00001310::2010/08/26-20:42:54.057 INFO  [SV] Authentication and authorization were successful
    000010b0.00001310::2010/08/26-20:42:54.057 DBG   [SM] SrvCtxt initialized with package Kerberos, MaxTokenSize = 12000, RequiredCtxAttrib = 133122, HandShakeTimeout = 30000
    000010b0.00001108::2010/08/26-20:42:54.057 DBG   [SM] Handling auth handshake posted by thread id 4880
    000010b0.00001108::2010/08/26-20:42:54.057 DBG   [CHANNEL 192.168.210.236:~59822~]/recv: Attempting to perform I/O on closed stream.
    000010b0.00001108::2010/08/26-20:42:54.057 DBG   [CHANNEL 192.168.210.236:~59822~] Not closing handle because it is invalid.
    000010b0.00001108::2010/08/26-20:42:54.057 INFO  [CHANNEL 192.168.210.236:~59822~] graceful close, status (of previous failure, may not indicate problem) ERROR_SUCCESS(0)
    000010b0.00001108::2010/08/26-20:42:54.057 ERR   [SM] Caught GracefulClose(1226)' because of 'channel to remote endpoint 192.168.210.236:~59822~ is closed' authenticating.
    000010b0.00001310::2010/08/26-20:42:54.057 DBG   [CHANNEL 192.168.210.236:~59822~] Not closing handle because it is invalid.
    000010b0.000012d8::2010/08/26-20:43:54.004 DBG   [JPM] Node 1: contacts size for node iQCXSQL2 is 1, current index 0
    000010b0.000012d8::2010/08/26-20:43:54.004 DBG   [JPM] Node 1: Trying to connect to node iQCXSQL2 (IP: 192.168.210.236:~0~)
    000010b0.000012d8::2010/08/26-20:43:54.004 DBG   [HM] Trying to connect to iQCXSQL2 at 192.168.210.236:~3343~
    000010b0.000012d8::2010/08/26-20:43:54.004 INFO  [CONNECT] 192.168.210.236:~3343~: Established connection to remote endpoint 192.168.210.236:~3343~.
    000010b0.000012d8::2010/08/26-20:43:54.004 INFO  [SV] Securing route from (192.168.210.239:~49563~) to remote iQCXSQL2 (192.168.210.236:~3343~).
    000010b0.000012d8::2010/08/26-20:43:54.004 INFO  [SV] Got a new outgoing stream to iQCXSQL2 at 192.168.210.236:~3343~
    000010b0.000012d8::2010/08/26-20:43:54.004 DBG   [SM] Joiner: Initialized with SPN = iQCXSQL2, Package = Kerberos, RequiredCtxAttrib = 83990, HandShakeTimeout = 30000
    000010b0.000013f0::2010/08/26-20:43:54.004 DBG   [SM] Handling auth handshake posted by thread id 4824
    000010b0.000013f0::2010/08/26-20:43:54.004 DBG   [SM] Joiner: ISC returned status = 590610 output Blob size 1388
    000010b0.000013f0::2010/08/26-20:43:54.004 DBG   [SM] Joiner: Received SSPI blob from the Sponsor of size 96
    000010b0.000013f0::2010/08/26-20:43:54.004 DBG   [SM] Joiner: ISC returned status = 590610 output Blob size 1388
    000010b0.000013f0::2010/08/26-20:43:54.004 DBG   [SM] Joiner: Received SSPI blob from the Sponsor of size 96
    000010b0.000013f0::2010/08/26-20:43:54.004 DBG   [SM] Joiner: ISC returned status = -2146893022 output Blob size 12000
    000010b0.000012d8::2010/08/26-20:43:54.004 INFO  [NODE] Node 1: New join with n2: stage: 'Authenticate Initial Connection' status HrError(0x80090322) reason: '[SV] Authentication failed'
    000010b0.000012d8::2010/08/26-20:43:54.004 DBG   [CHANNEL 192.168.210.236:~3343~] Close().
    000010b0.000012d8::2010/08/26-20:43:54.004 WARN  cxl::ConnectWorker::operator (): HrError(0x80090322)' because of '[SV] Authentication or Authorization Failed'
    000010b0.00000ea8::2010/08/26-20:43:54.067 INFO  [ACCEPT] 0.0.0.0:~3343~: Accepted inbound connection from remote endpoint 192.168.210.236:~59824~.
    000010b0.000012d8::2010/08/26-20:43:54.067 INFO  [SV] Securing route from (192.168.210.239:~3343~) to remote  (192.168.210.236:~59824~).
    000010b0.000012d8::2010/08/26-20:43:54.067 INFO  [SV] Got a new incoming stream from 192.168.210.236:~59824~
    000010b0.000012d8::2010/08/26-20:43:54.067 DBG   [SM] SrvCtxt initialized with package Kerberos, MaxTokenSize = 12000, RequiredCtxAttrib = 165910, HandShakeTimeout = 30000
    000010b0.00001108::2010/08/26-20:43:54.067 DBG   [SM] Handling auth handshake posted by thread id 4824
    000010b0.00001108::2010/08/26-20:43:54.067 DBG   [SM] Sponsor: Received SSPI blob from the Joiner of size 1411
    000010b0.00001108::2010/08/26-20:43:54.067 DBG   [SM] Sponsor: SSPI ASC returned status = 590610
    000010b0.00001108::2010/08/26-20:43:54.067 DBG   [SM] Sponsor: Sending SSPI blob of size 96 to Joiner
    000010b0.00001108::2010/08/26-20:43:54.067 DBG   [SM] Sponsor: Received SSPI blob from the Joiner of size 1411
    000010b0.00001108::2010/08/26-20:43:54.067 DBG   [SM] Sponsor: SSPI ASC returned status = 590610
    000010b0.00001108::2010/08/26-20:43:54.067 DBG   [SM] Sponsor: Sending SSPI blob of size 96 to Joiner
    000010b0.00001108::2010/08/26-20:43:54.067 DBG   [SM] Sponsor: Received SSPI blob from the Joiner of size 12000
    000010b0.00001108::2010/08/26-20:43:54.067 DBG   [SM] Sponsor: SSPI ASC returned status = 590610
    000010b0.00001108::2010/08/26-20:43:54.067 DBG   [SM] Sponsor: Sending SSPI blob of size 115 to Joiner
    000010b0.00001108::2010/08/26-20:43:54.067 DBG   [CHANNEL 192.168.210.236:~59824~] Close().
    000010b0.00001108::2010/08/26-20:43:54.067 INFO  [CHANNEL 192.168.210.236:~59824~] graceful close, status (of previous failure, may not indicate problem) ERROR_SUCCESS(0)
    000010b0.00001108::2010/08/26-20:43:54.067 ERR   [SM] Caught GracefulClose(1226)' because of 'channel to remote endpoint 192.168.210.236:~59824~ is closed' authenticating.
    000010b0.000012d8::2010/08/26-20:43:54.067 INFO  [SV] Authentication and authorization were successful
    000010b0.000012d8::2010/08/26-20:43:54.067 DBG   [SM] SrvCtxt initialized with package Kerberos, MaxTokenSize = 12000, RequiredCtxAttrib = 133122, HandShakeTimeout = 30000
    000010b0.00001108::2010/08/26-20:43:54.067 DBG   [SM] Handling auth handshake posted by thread id 4824
    000010b0.00001108::2010/08/26-20:43:54.067 DBG   [CHANNEL 192.168.210.236:~59824~]/recv: Attempting to perform I/O on closed stream.
    000010b0.00001108::2010/08/26-20:43:54.067 DBG   [CHANNEL 192.168.210.236:~59824~] Not closing handle because it is invalid.
    000010b0.00001108::2010/08/26-20:43:54.067 INFO  [CHANNEL 192.168.210.236:~59824~] graceful close, status (of previous failure, may not indicate problem) ERROR_SUCCESS(0)
    000010b0.00001108::2010/08/26-20:43:54.067 ERR   [SM] Caught GracefulClose(1226)' because of 'channel to remote endpoint 192.168.210.236:~59824~ is closed' authenticating.
    000010b0.000012d8::2010/08/26-20:43:54.067 DBG   [CHANNEL 192.168.210.236:~59824~] Not closing handle because it is invalid.
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO  [CS] Service Stopping...
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO  [CORE] Node quorum state is 'Not yet formed or joined a cluster'. Form/join status with other nodes is as follows:
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO  [NODE] Node 1: Farthest reported progress joining with node iQCXSQL2 (id 2) is: Authenticate Initial Connection at time 2010/08/26-20:43:54.004: status 802 [SV] Authentication failed
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO  [CORE] Node 1: Proposed View is <ViewChanged joiners=() downers=(1) newView=101() oldView=001(1) joiner=false form=false/>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO  [RGP] Node 1: Stable_`0 => Opening`1
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO  <class mscs::detail::ConsensusMessage>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <senderId> 1 </senderId>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <bestEpochSeen> 0 </bestEpochSeen>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <lastStableView> 001(1) </lastStableView>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <proposedView> 101() </proposedView>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <stage> Opening`1 </stage>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <inStage> () </inStage>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <inStagePrev> () </inStagePrev>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <joiners> () </joiners>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <trimmedNodes> (1) </trimmedNodes>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <innerScreen> () </innerScreen>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <pruningResult> () </pruningResult>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <matrix>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO      <ConnectivityMatrix>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO        <row id='0'00000000000000000000000000000000/>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO      </ConnectivityMatrix>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    </matrix>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO    <gemBlob> <counted_ptr p = 'nullptr'/> </gemBlob>
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO  </class mscs::detail::ConsensusMessage>
    000010b0.000006fc::2010/08/26-20:44:52.906 INFO  [RGP] Node 1: Timer Tick Started
    000010b0.0000106c::2010/08/26-20:44:52.906 DBG   [NODE] Node 1: eating message sent to the dead node 2
    000010b0.0000106c::2010/08/26-20:44:52.906 INFO  [RGP] Node 1: Just sent farewell regroup message
    000010b0.000010b8::2010/08/26-20:44:52.906 DBG   [NODE] Node 1: just about to send a message of size 0 to 1
    000010b0.000010b8::2010/08/26-20:44:52.906 DBG   [NODE] Node 1: message to node 1 sent
    000010b0.000010b8::2010/08/26-20:44:52.906 DBG   [NODE] Node 1: just about to send a message of size 361 to 1
    000010b0.000010b8::2010/08/26-20:44:52.906 DBG   [NODE] Node 1: message to node 1 sent
    000010b0.000006fc::2010/08/26-20:44:53.218 INFO  [RGP] Node 1: Timer Tick Started
    000010b0.0000106c::2010/08/26-20:44:53.420 INFO  [DM]: Shutting down, so unloading the cluster database.
    000010b0.0000106c::2010/08/26-20:44:53.420 INFO  [DM] Shutting down, so unloading the cluster database (waitForLock: true).
    000010b0.0000106c::2010/08/26-20:44:53.420 DBG   [DM] Unloading Hive, Key \Registry\Machine\Cluster, discardCurrentChanges true
    000010b0.0000106c::2010/08/26-20:44:53.420 INFO  [CS] Service Stopped...
    000010b0.0000106c::2010/08/26-20:44:53.420 INFO  [CS] About to exit service...

  • Clone multi-node RAC and ASM to single node

    Hi everyone.
    I need clone system with 3 application server and 2 oracle database RAC and ASM to single-node. The operating system is RHEL 5.
    I see some metalink notes, but we can't found nothing. I find notes with multi-node to single-node, but nothing with RAC to non-RAC.
    The eBS version is 11.5.10.2 and database version is 10.2.0.3
    Is possible this clone?
    Thanks very much.

    Hi User;
    Please follow below and see its helpful:
    EBS R12 with RAC and non-RAC
    Re: RAC to single instance ebs R12
    Re: Clone Oracle Apps 11.5.10.2 RacDB to Non-RAC DB
    Re: CLONING R12 RAC to NON RAC CLONING giving error RMAN-05517 temporary file
    [b]Migrating the DB-Tier (DB and CM) to Two node non RAC cluster[/b]
    Also check:
    http://www.oracle.com/technology/pub/articles/chan_sing2rac_install.html
    Regard
    Helios

  • How to install Enterprise manager on two node cluster

    Hi,
    I am new to Oracle working on 10g
    can anyone post a document or link for step by step installation of oracel 10g enterprise manager on a two node cluster?
    Thanks in advance.

    userArshad wrote:
    Thanks Hans.
    To be specific I have to install Database control .that means I need to use the GUI .
    Also I could see database control and DBconsole service on on 1st node but I cannot find that in 2nd node.That is the normal configuration - you use the DB Control from node 1 and it gets the information using the agent on Node 2. The DB Console then displays the information for both nodes.
    Please note that the DB Control will place a relatively heavy load on each machine, which is why we want to restrict our load to the agent. It is for that reason that we prefer to use the Grid Control whenever possible - it provides the same mechanism and screens, but the JEE platform is on a separate machine and all you have on the RAC nodes is the agent.
    But if you really want to run the DB Console from each node (again, not normally needed) you need to use emca to configure it on the second node. Metalink note 863631.1 should help you with that.

Maybe you are looking for

  • SEVERE: java.io.FileNotFoundException: ./config/jps-config.xml (No such file or directory)

    I am installing OIM/OAM in a cluster configuration using release 11.1.1.7. I looked at the EDG here: http://docs.oracle.com/cd/E28280_01/core.1111/e12035/toc.htm And its pointing me to the 11.1.1.5 version http://docs.oracle.com/cd/E21764_01/core.111

  • SAP Versions for B1i

    Hi all, I tried to add new SAP versions (e.g. ERP2004, 2005) to the drop down list in landscape configurator in control center but they are not visible. I added new modified xml files ( copies of R3.47.200.xml) to setup/com.sap.b1i.repository\SysType

  • Colliding Plug-in Versions

    I have a multi-frame JSP app that uses the left frame to load a Java Applet(with a JTree) using the <Object> or <Embed> tags that specify the 1.4.1_01 plugin version to use. Once the applet is loaded and a selection is made form the JTree, the main f

  • How to pick up sid, serial# for a concurrent request

    I am trying to find a sid, serial# for a concurrent request; any suggestions/scripts? Thanks in advance,

  • Same transaction variant runs differently on different servers

    Hi experts, We have created a transaction variant for MK01. Our production system consists of 3 servers. Everything is ok when we use this transaction variant on the main server, but there is a dump when the transaction variant is run on another serv