Locks after migrate to RAC 2 nodes.

After migrating a stand alone database to RAC two-nodes (11.2.0.4) we are facing lock in database.
It occurs always on the same tables. I asked developers if commit was issued after the update command and they confirmed commit was set after the update statement.
Is there something we have to configure in database side to eliminate these locks?
We have just 3 active sessions in this database, so there is no high-concurrent DML (initrans parameter)
Unless you can show what does this question has to do with Oracle 12c and multitenancy please mark it ANSWERED and repost it in the General Database forum.
General Database Discussions
When you repost you need to SHOW us what you did, not just tell us. Post the actual commands executed.
What do you mean by 'migrating'? What 'same tables' are you talking about? What 'after the update command' are you talking about?

Similar Messages

Folders and Files Locked After Migrating

I replaced my beloved iMac G5 with a new 27" Intel iMac. I migrated my files over to the new Mac using Migration Assistant. Now, all [or all that I've tried] of my Folders and Files are locked so I can't rename them or add new files to a folder without first Getting Info, then unlocking, typing in my PW and changing the Privileges to Read & Write for everyone. I'm the Administrator -- and the only account on the Mac. My account shows that I have both Read & Write Privileges, but that doesn't seem to be enough for the new Mac.
Any suggestions? It's rather time consuming to have to reset the Privileges every time I try to save a file to a new Folder.
Thanks.

Thanks again. That was helpful. I wish the Genius at the Apple Store had bothered to explain that there would be two accounts.
If I can trouble you a bit more. While I understand that I shouldn't merge the two accounts, there are now some files under the "new" account that I need to have in the "migrated" account. What's the best way to move them? I tried copying them on to an External HD, but when I tried to copy them in to the Merged account I get nothing but Error -36 [and everything stops b/c of .ds_store files. I've been looking at posts about .ds_store files and it seems that lots of people are having problems with them and that there doesn't seem to be an easy fix. So much for the Intuitive thinking of just copying files from one drive to another.] The External is Mac Journaled -- I reformatted it yesterday just to be sure.

Excessive log switching after upgrading to RAC + DG

Hi everyone, I'm not sure if this should go in the RAC or the Data Guard section, seeing how I'm not sure which one is the potential culprit, but here goes:
I recently assisted in upgrading a database from Solaris 9i to HP-UX 10gR2 (10.2.0.4) using RAC and Data Guard (2 nodes). However after upgrading we noticed that there is now an increase in the redo log switches. We currently have 4 logfile groups for each thread sized at 100MB. Previously when the database was just a single instance, such a configuration was sufficient as we were only seeing 4 log switches per hour, max 8 at peak times. After the migration, this spiked up to around 40-60 log switches per hour, sometimes hitting 80.
The userbase/load of the database hasn't changed since the upgrade, we just migrated the database. Is this normal? Should I have to increase the redo logs after migrating to RAC + DG? I've implemented DG before as well as RAC, but not together so I'm not sure if this would be a factor, however I'm not totally sure if that line of thinking is correct. Any help would be much appreciated.

Previously when the database was just a single instance, such a configuration was sufficient as we were only seeing 4 log switches per hour, max 8 at peak times. After the migration, this spiked up to around 40-60 log switches per hour, sometimes hitting 80. Are all the ArchiveLogs actually 100MB each ? If Oracle has to switch logs more frequently, it doesn't necessarily mean that more Redo is being generated. Check the volum of Redo. (e.g. from v$sysstat, from StatsPack, from AWR, from V$ARCHIVED_LOG or by just listing the ArchiveLog file sizes)
Hemant K Chitale

Migration from Single Node to RAC Node

Hi,
We are planning to migrate the Database from Single-Node to RAC Node. Are there any checklist list to be considered regarding Performance, System and Database ( or any other topic), before migrating to RAC.
Thanks

If it is already the same version, you will need to focus any "single-threadedness" of the application like sequences that are mandatory sequential, "semaphore-style" locking etc...
My experience is that going RAC and in particular to ASM can give a performance boost. If it is the same version (11gR2 for example) then any statistics-related performance issues will still be there.

Domain migration of Oracle RAC nodes on windows 2008

Hello masters,
We are planning to migrate servers from old domain to a new domain due to company name change. That is, our DB servers named like db-node-x.old-domain.com will be migrated to db-node-x.new-domain.com. The servers are all on Windows 2008 R2 and Oracle RAC 11g2. As dba, i have no idea about what will happen to DBs. Please let me know any possible solutions?
Thanks in advance.

user2033016 wrote:
Hi Everyone,
I installed Oracle RAC 11g on Windows 2008 64 bit, The installation was successfull crs_state -t executed successfully.
Bt the machine got crashed ,when executed a select query with 100 sessions( For performance evaluation using TG). A blue screen is appeared on two nodes. So i am not able to evaluate the performance.
Can you please tell any suggesstions to solve this issue? Is our hardware is not sufficient? Or any software settings has to be done?. I tried most of hte technique ,bt the problem still exists.
The following machine configuration is used for Oracle RAC setup
Intel Core 2 Duo 2.66 GHz,
Ram :4GB.
OS : windows 2008 Server 64 Bit
Oracle RAC , Clusterware 11g R2 64 Bit.i guess you have installed/configured RAC using VMware. i guess, the problem is with your RAM size. how much disk space and RAM size you have allocated for each machine?
which documentation you have followed?

Objects in transports are not locked after DB migration

Hi all,
we had an OS\DB migration to Linux\Oracle.
The transports were Not released prior to the migration.
after migration, when changing an Object the system asks for a new transport allthough there a transport allready exists with the locked object.
this way , we have the same object in 2 different transports - and the system displays the object is locked in both.
please advise.
many thanks.
Orit
Moderator message: not directly related to ABAP development, please have a look in the SAP Netweaver or Software Logistics forums.
Edited by: Thomas Zloch on Nov 25, 2011 1:57 PM

Characteristic are locked after data load; try loading and than check it.
Hope it helps.
Regards

Unable to bring up crs on rac node 2 after server reboot

Hi,
We have a 2 node rac architecture. We are only able to bring up Node 1 on the cluster, whereas node 2 is failing. Here are some points::
1. After the server reboot, node 2 crs crs/resources weren't starting up apart from OHAS.
2. We again stopped both the CRS and tried bringing up CRS on node 2 initially and succeeded. But now node 1 wasn't coming up.
3. Again brought down both nodes' CRS and tried bringing up CRS on node1 and succeded but asm wasn't showing the Diskgroups. So we changed pfile to include asm_diskstring from ORCL* to /dev/oracleasm/disks and we could lsdg in asm now.So started all the instances from node 1 now. Apart from this, again node 2 CRS wasn't starting. From alertlog I saw "CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds;". But we were able to query voting disks initially. What has gone wrong now??
./crsctl status res -t -init
NAME           TARGET STATE        SERVER                   STATE_DETAILS
Cluster Resources
ora.asm
      1        OFFLINE OFFLINE
ora.cluster_interconnect.haip
      1        ONLINE OFFLINE
ora.crf
      1        ONLINE ONLINE       kusmnd0r
ora.crsd
      1        ONLINE OFFLINE
ora.cssd
      1        ONLINE OFFLINE                               STARTING
ora.cssdmonitor
      1        ONLINE ONLINE       kusmnd0r
ora.ctssd
      1        ONLINE OFFLINE
ora.diskmon
      1        OFFLINE OFFLINE
ora.evmd
      1        ONLINE OFFLINE
ora.gipcd
      1        ONLINE ONLINE       kusmnd0r
ora.gpnpd
      1        ONLINE ONLINE       kusmnd0r
ora.mdnsd
      1        ONLINE ONLINE       kusmnd0r
This is the history of activitites. Kindly someone throw light on this please.
Thanks,
Anirban.

It is on a raw device.
Healthy node::
ls -ltrh /dev/vote*
brw-rw---- 1 crsdwqa dbadwqa 120, 1057 Nov 6 11:32 /dev/vote3
brw-rw---- 1 crsdwqa dbadwqa 120, 1025 Nov 6 11:32 /dev/vote1
brw-rw---- 1 crsdwqa dbadwqa 120, 1041 Nov 6 11:32 /dev/vote2
Affected Node::
ls -ltrh /dev/vote*
brw-rw-r-- 1 crsdwqa dbadwqa 120, 1025 Nov 4 12:06 /dev/vote1
brw-rw-r-- 1 crsdwqa dbadwqa 120, 1041 Nov 4 12:06 /dev/vote2
brw-rw-r-- 1 crsdwqa dbadwqa 120, 1057 Nov 5 04:42 /dev/vote3
Regards,
Anirban.

RAC node restarting!

hi
one of our RAC environment keep restarting.
i've disable the init.cssd, init.crs, init.evmd in the /etc/inittab in order to check the logs.
this is the situation:
crsd.log:
2009-02-04 00:09:00.118: [ COMMCRS][9]clsc_connect: (8000000100318640) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_node1_loud))
2009-02-04 00:09:00.132: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9
2009-02-04 00:09:00.134: [ CRSRTI][1]32CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2009-02-04 00:09:08.016: [    CRSD][1]32Daemon Version: 10.2.0.2.0 Active Version: 10.2.0.2.0
2009-02-04 00:09:08.016: [    CRSD][1]32Active Version and Software Version are same
2009-02-04 00:09:08.017: [ CRSMAIN][1]32Initializing OCR
2009-02-04 00:09:08.037: [ OCRRAW][1]proprioo: for disk 0 (/dev/rdsk/ora_ocr_raw), id match (1), my id set (752560621,1028247821) total id sets (1), 1st set
(752560621,1028247821), 2nd set (0,0) my votes (2), total votes (2)
2009-02-04 00:09:08.140: [ CSSCLNT][24]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying
ocssd.log:
[    CSSD]2009-02-03 21:52:08.651 [9] >USER: clssnmHandleUpdate: NODE 1 (node1l) IS ACTIVE MEMBER OF CLUSTER
[    CSSD]2009-02-03 21:52:08.651 [9] >TRACE: clssnmHandleUpdate: diskTimeout set to (200000)ms
[    CSSD]2009-02-03 21:52:08.651 [16] >TRACE: clssnmWaitForAcks: done, msg type(15)
[    CSSD]2009-02-03 21:52:08.651 [16] >TRACE: clssnmDoSyncUpdate: Sync Complete!
[    CSSD]2009-02-03 21:52:08.722 [1] >USER: NMEVENT_SUSPEND [00][00][00][00]
[    CSSD]2009-02-03 21:52:08.724 [17] >TRACE: clssgmReconfigThread: started for reconfig (1)
[    CSSD]2009-02-03 21:52:08.749 [17] >USER: NMEVENT_RECONFIG [00][00][00][02]
[    CSSD]2009-02-03 21:52:08.749 [17] >TRACE: clssgmEstablishConnections: 1 nodes in cluster incarn 1
[    CSSD]2009-02-03 21:52:08.751 [13] >TRACE: clssgmPeerListener: connects done (1/1)
[    CSSD]2009-02-03 21:52:08.752 [17] >TRACE: clssgmEstablishMasterNode: MASTER for 1 is node(1) birth(1)
[    CSSD]2009-02-03 21:52:08.752 [17] >TRACE: clssgmChangeMasterNode: requeued 0 RPCs
[    CSSD]2009-02-03 21:52:08.752 [17] >TRACE: clssgmMasterCMSync: Synchronizing group/lock status
[    CSSD]2009-02-03 21:52:08.752 [17] >TRACE: clssgmMasterSendDBDone: group/lock status synchronization complete
[    CSSD]CLSS-3000: reconfiguration successful, incarnation 1 with 1 nodes
[    CSSD]CLSS-3001: local node number 1, master node number 1
[    CSSD]2009-02-03 21:52:08.753 [17] >TRACE: clssgmReconfigThread: completed for reconfig(1), with status(1)
[    CSSD]2009-02-03 21:52:08.863 [10] >TRACE: clssgmClientConnectMsg: Connect from con(80000001008fd2a0) proc(8000000100ae26a8) pid() proto(10:2:1:1)
[    CSSD]2009-02-03 21:52:08.864 [10] >TRACE: clssgmClientConnectMsg: Connect from con(8000000100ae0128) proc(8000000100ae2a10) pid() proto(10:2:1:1) from con(8000000100aa32c0) proc(8000000100aa5b90) pid() proto(10:2:1:1)
alertlog:
[cssd(2535)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node1 .
2009-02-03 23:55:20.821
[cssd(2575)]CRS-1605:CSSD voting file is online: /dev/rdsk/ora_voting_raw. Detai ls in /work/crs/product/10.2/crs/log/lourmel/cssd/ocssd.log.
2009-02-03 23:55:28.376
evmd.log:
Oracle Database 10g CRS Release 10.2.0.2.0 Production Copyright 1996, 2004, Oracle. All rights reserved
2009-02-04 00:08:58.331: [    EVMD][1]32EVMD waiting for CSS to be ready err = 3
2009-02-04 00:08:59.939: [ COMMCRS][9]clsc_connect: (800000010007d658) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_node1_loud))
2009-02-04 00:08:59.946: [ CSSCLNT][1]clsssInitNative: connect failed, rc 9
2009-02-04 00:08:59.948: [    EVMD][1]32EVMD waiting for CSS to be ready err = 3
2009-02-04 00:09:07.596: [ CSSCLNT][1]clssgsGroupJoin: CSS has not reached fatal mode.Registration is not yet safe. Retrying
syslog:
Feb 4 00:08:41 lourmel syslog: Oracle Cluster Ready Services starting up automatically.
Feb 4 00:08:45 lourmel sfd[2153]: starting the daemon.
Feb 4 00:08:45 lourmel su: + tty?? root-orac
Feb 4 00:08:45 lourmel krsd[2152]: Delay time is 300 seconds
Feb 4 00:08:43 lourmel syslog: Oracle Cluster Ready Services starting up automatically.
Feb 4 00:08:52 lourmel above message repeats 2 times
Feb 4 00:08:52 lourmel syslog: Cluster Ready Services completed waiting on dependencies.
Feb 4 00:08:53 lourmel syslog: Running CRSD with TZ =
when i checked(befor the restart) the command crs_stat i got the message:
ORA-0184: Cannot communicate wirh CRS
crsctl check crs gives us:
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM
as i said befor, the machine always restarting
anyone have an idea?? please

Dear All,
I recently upgrade the Few RAC setups with Oracle 10g Patchset 3 (10.2.0.4) on Linux Servers
In one of the RAC setup, found servers are rebooting daily. The same setup was working fine and problem started only after applying the Patchset. Checked all the logs and Found nothing relevant.
Then i checked the things which added with this Patchset.
The Most interesting found , Oracle Added a New Daemon- oprocd.
# ps -efl | grep oprocd
4 S root 6440 6063 0 -40 - - 2114 - Mar03 ? 00:00:00 /opt/oracle/product/10.2.0/crs/bin/oprocd.bin run -t 1000 -m 500 -hsi 5:10:50:75:90 -f
These are Interesting Points about above line
1.This Process is running by root user
2. With Highest Priority -40
3. Probing every Seconds (t 1000)
4. waiting CPU response for 500 Milliseconds ( -m 500 means margin time is 500 Milli Seconds)
5. Process status is Fatal (-f)
Now I am concluding these points- This daemon will probe cpu every second and wait for response within 500 Mill seconds. If in the 500 Milli second not getting any response from the cpu, will assume the CPU is hang and try to Reboot the Machine. The OPERATING SYSTEM will not get enough time to write the system logs and server reboots.
So the solution is increase the Margin time for 500 Milli second to 10 seconds.
These are following steps to increase the Margin time.
Please Remember- The Modification process need Downtime and You need to stop cluster service in all member nodes.
1. Stop The CRS Process
#crsctl stop crs
#<CRS_HOME>/bin/oprocd stop
2. Ensure that Clusterware stack is down and not running
#ps -ef |egrep "crsd.bin|ocssd.bin|evmd.bin|oprocd"
This should return no processes.
3. From one node of the cluster, change the value of the "diagwait" parameter to 13 by issuing the command as root:
#crsctl set css diagwait 13 -force
4. Check if diagwait is successfully set.
#crsctl get css diagwait
5. Restart the Oracle Clusterware on all the nodes by executing:
#crsctl start crs
(Note- If facing any problem to restarting the CRS services, ASM and Database, You can reboot the Nodes.The Cluster and Database will come automatically due to init startup scripts.)
6. The oprocd daemon process will show with -m 10000
# ps -efl| grep oprocd
# 4 S root 6440 6063 0 -40 - - 2114 - Feb02 ? 00:00:00 /opt/oracle/product/10.2.0/crs/bin/oprocd.bin run -t 1000 -m 10000 -hsi 5:10:50:75:90 -f
Rollback Procedure-
If You need to unset oprocd value due any reason
#crsctl unset css diagwait
I am confident, The abnormal RAC Node restart problem will solve with this workaround.
Regards,
Sumit
Bangalore,India

JDBC read stuck if RAC node goes down

We did several tests with Java applications against our RAC DB and face a hanging application if we power off the RAC node that executes the current (long) running query.
We can see that the application receives HA-events via UCP:
2015-01-22 13:02:11 | r-thread-1 | WARN | o.ucp.jdbc.oracle.ONSDatabaseFailoverEvent    | NO timezone in HA event
However, the application started a query before and the query is not aborted with an exception. A Thread dump after about 7 minutes shows that the application is hanging in a socket read call:
"pool-1-thread-1" #32 prio=5 os_prio=0 tid=0x00007fedf45b2000 nid=0xbc4 runnable [0x00007fee00cd3000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:150)
    at java.net.SocketInputStream.read(SocketInputStream.java:121)
    at oracle.net.ns.Packet.receive(Packet.java:283)
    at oracle.net.ns.DataPacket.receive(DataPacket.java:103)
    at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:230)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:175)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:100)
    at oracle.net.ns.NetInputStream.read(NetInputStream.java:85)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
    at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1122)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1099)
    at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288)
    at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
    at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
    at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
    at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:863)
    at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1153)
    at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1275)
    at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3576)
    at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3620)
    - locked <0x00000000c0ddcb20> (a oracle.jdbc.driver.T4CConnection)
    at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491)
    at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:703)
The expected behaviour would be that a running query is aborted with an exception. (BTW: This happens if the service is taken down with "shutdown immediate". All ok for this case.)
We consider to implement custom ONS listeners [1], but we actually expect that UCP would handle such situations or lets us register strategies/callbacks for certain events.
Our config:
Oracle Enterprise 11.2.0.4.0 with RAC
ons.jar 12.1.0.1
ojdbc6.jar 11.2.0.2
ucp.jar 12.1.0.1
Server JRE 1.8.0_25
Any hints appreciated.
[1] http://docs.oracle.com/cd/E11882_01/java.112/e16548/apxracfan.htm#JJDBC28945

You're concept isn't right:
http://docs.oracle.com/cd/E11882_01/server.112/e25494/restart.htm#ADMIN13178
Overview of Fast Application Notification
FAN is a notification mechanism that Oracle Restart can use to notify other processes about configuration changes that include service status changes, such as UP or DOWN events. FAN provides the ability to immediately terminate inflight transaction when an instance or server fails. Integrated Oracle clients receive the events and respond. Applications can respond either by propagating the error to the user or by resubmitting the transactions and masking the error from the application user. When a DOWN event occurs, integrated clients immediately clean up connections to the terminated database. When an UP event occurs, the clients create new connections to the new primary database instance.
Also, take a look at these docs: http://docs.oracle.com/cd/E11882_01/java.112/e12265/rac.htm#JJUCP08100 ; and https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=890204623685515&id=566573.1&_afrWindowMode=0&_adf.ctrl-s…
And make a test, execute a query that took about 1 minute and after you executed, just power down the node where it is executing, to see if it will retrieve the results.
Regards.

Can't see EJB objects in Application Navigator after migration

Hello
I can't see any objects in Application Navigator after migration from previous version of JDeveloper (9.2.0.5), in System Navigator view I can see all sources. Has somebody any idea what the reason is?
Tom

In the application navigator you should see one node per EJB, when you stand on it you'll see the various source files in the structure window.

RAC node outage causes SOA Suite 10.1.3.4 BPEL failure

Using weblogic 9.2 and the SOA Suite 10.1.3.4. We use a 10g Oracle RAC ( 2 nodes ); the WL cluster has a multi data source of 2 pools, each pool pointing to a single node in the rac, each pool deployed to the cluster, and the multi data source in load-balancing mode.
So the other night, one of the db nodes had a hardware failure ( ironically, with a remote monitoring / management card ). Annoying, but it should not have caused the BPEL servers to be in "FAILED NOT RESTARTABLE" status the next morning.
Jun 9, 2009 12:10:07 AM EDT> <Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool "esbaqds2": Io exception: The Network Adapter could not establish the connection>
SEVERE: Destroying JMSDequeuer failed
oracle.jms.AQjmsException: Connection has been administratively destroyed. Reconnect.
at oracle.jms.AQjmsSession.preClose(AQjmsSession.java:980)
at oracle.jms.AQjmsObject.close(AQjmsObject.java:409)
at oracle.jms.AQjmsSession.close(AQjmsSession.java:1020)
at oracle.tip.esb.server.dispatch.JMSDequeuer.destroy(JMSDequeuer.java:419)
at oracle.tip.esb.server.dispatch.JMSDequeuer.destroyWithoutUnsubscribing(JMSDequeuer.java:395)
at oracle.tip.esb.server.dispatch.JMSDequeuer.dequeue(JMSDequeuer.java:175)
at oracle.tip.esb.server.dispatch.agent.ESBWork.process(ESBWork.java:174)
at oracle.tip.esb.server.dispatch.agent.ESBWork.run(ESBWork.java:132)
at weblogic.connector.security.layer.WorkImpl.runIt(WorkImpl.java:108)
at weblogic.connector.security.layer.WorkImpl.run(WorkImpl.java:44)
at weblogic.connector.work.WorkRequest.run(WorkRequest.java:93)
at weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
followed by a 2 GB log file containing 1.3 million iterations of the following within the next 10 minutes before the managed servers failed.
java.lang.NullPointerException
at java.lang.String.<init>(String.java:144)
at oracle.tip.esb.server.dispatch.JMSDequeuer.dequeue(JMSDequeuer.java:168)
at oracle.tip.esb.server.dispatch.agent.ESBWork.process(ESBWork.java:174)
at oracle.tip.esb.server.dispatch.agent.ESBWork.run(ESBWork.java:132)
at weblogic.connector.security.layer.WorkImpl.runIt(WorkImpl.java:108)
at weblogic.connector.security.layer.WorkImpl.run(WorkImpl.java:44)
at weblogic.connector.work.WorkRequest.run(WorkRequest.java:93)
at weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
Both managed instances of the BPEL cluster failed, even though the 1st node of the Oracle RAC was still available.
Our 10.3 cluster, also using multi data sources to the same RAC for the OSB components, simply went on about its business using the remaining rac node pool.
Seems to be a single point of failure...

We haven't changed the JDBC connection string yet, but we did run a test in the same environment while Oracle support considers the situation.
For the test, we simply shutdown one node of the RAC and watched to see what happens. Within the space of a minute, the JDBC "Failed Reserve Request Count" was increasing by thousands on every refresh of the screen. We restarted the RAC node after 5 minutes, by which time the "Failed Reserve Request Count" was over 190,000
The 2 BPEL managed servers remained in Running status and each created a 660 MB log file within that 5 minutes. In the original outage, the nodes were down for about 15 minutes. Most of the logging is being generated from within the oracle.tip.esb classes, not by the weblogic classes. It looks like that once the pool pointing to the downed RAC node becomes disabled, the Oracle BPEL code is still trying to use it even though the multi-source JNDI is the published lookup:
INFO: JMSDequeuer::createConnection - AQ Topics
java.sql.SQLException: weblogic.common.ResourceException:
esbaqds(esbaqds2): Pool esbaqds2 is disabled, cannot allocate resources to applications..
esbaqds(esbaqds1): Pool esbaqds1 is disabled, cannot allocate resources to applications..
at weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUtil.java:250)
at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:348)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:364)
at oracle.tip.esb.server.dispatch.JMSDequeuer.createAQConnection(JMSDequeuer.java:559)
at oracle.tip.esb.server.dispatch.JMSDequeuer.dequeue(JMSDequeuer.java:159)
at oracle.tip.esb.server.dispatch.agent.ESBWork.process(ESBWork.java:174)
at oracle.tip.esb.server.dispatch.agent.ESBWork.run(ESBWork.java:132)
at weblogic.connector.security.layer.WorkImpl.runIt(WorkImpl.java:108)
at weblogic.connector.security.layer.WorkImpl.run(WorkImpl.java:44)
at weblogic.connector.work.WorkRequest.run(WorkRequest.java:93)
at weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
Caused by: weblogic.common.ResourceException:
esbaqds(esbaqds2): Pool esbaqds2 is disabled, cannot allocate resources to applications..
esbaqds(esbaqds1): Pool esbaqds1 is disabled, cannot allocate resources to applications..
at weblogic.jdbc.common.internal.MultiPool.searchLoadBalance(MultiPool.java:331)
at weblogic.jdbc.common.internal.MultiPool.findPool(MultiPool.java:202)
at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:77)
at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:346)
... 11 more
SEVERE: Failed to process deferred message
oracle.tip.esb.server.dispatch.QueueHandlerException: Error creating "weblogic.common.ResourceException: No good connections available."
at oracle.tip.esb.server.dispatch.JMSDequeuer.createAQConnection(JMSDequeuer.java:661)
at oracle.tip.esb.server.dispatch.JMSDequeuer.dequeue(JMSDequeuer.java:159)
at oracle.tip.esb.server.dispatch.agent.ESBWork.process(ESBWork.java:174)
at oracle.tip.esb.server.dispatch.agent.ESBWork.run(ESBWork.java:132)
at weblogic.connector.security.layer.WorkImpl.runIt(WorkImpl.java:108)
at weblogic.connector.security.layer.WorkImpl.run(WorkImpl.java:44)
at weblogic.connector.work.WorkRequest.run(WorkRequest.java:93)
at weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

Exception while failing over to 2nd RAC Node

We are using Weblogic 10.3.4. Our setup is that we have a Web Application (A tapestry front end Web UI) and EJb 2.1 back-end talking to the Oracle database. The EJB’s are CMP. Our product always was just stand alone and it wasn’t until this release we needed to make it work with RAC. To get this to work we followed the model of having a Multidatasource with datasources pointing to our RAC nodes. We have two types of datasources that we use persistent and non-persistent. And we are using the Oracle thin driver – non-XA for RAC Service Instances, supporting global transactions.
When we do failover to the 2nd node we get a nasty exception in our GUI but after logging out and logging back it we are fine.
My question is that I assumed I shouldn't have to restart our web-application and it should have stayed up ?? Or is there something wrong with our setup ?
Thanks,
Ian

Showing us the exception and/or the error messages at the server might help...
Note that failing over does not save any ongoing connection or transaction that
had been to the dead RAC node... Does your web-app get-use-close JDBC
connections on a per-user-invoke basis, or does it hold onto connections?
Joe

Found the errors in CSSD logs of RAC node

Found the below error in CSSD logs in One of RAC nodes from 5:15 to 5:18 PM, after this the error got disappeared. Could anyone please have an idea what could be the reason of this error.
Also, at that time we didn't find any errors in the alert log.
[    CSSD]2009-07-19 17:15:51.048 [3600] >TRACE: Authorization failed (112bd2a70), timed out, start 17:13:51.041, duration 120009
[    CSSD]2009-07-19 17:15:51.048 [3600] >TRACE: Authorization prepare time: 2 ms
[    CSSD]2009-07-19 17:15:51.233 [3086] >TRACE: clssgmClientConnectMsg: Connect from con(112b67930) proc(112b680b0) pid(1049540) proto(10:2:1:1)
[    CSSD]2009-07-19 17:15:51.268 [3600] >TRACE: Authorization failed (112bd4a10), timed out, start 17:13:51.268, duration 120003
[    CSSD]2009-07-19 17:15:51.268 [3600] >TRACE: Authorization prepare time: 3 ms
[    CSSD]2009-07-19 17:15:52.544 [3086] >TRACE: clssgmClientConnectMsg: Connect from con(112b67930) proc(112b680b0) pid(786918) proto(10:2:1:1)
[    CSSD]2009-07-19 17:15:53.297 [3600] >TRACE: Authorization failed (112c38af0), timed out, start 17:13:53.290, duration 120009
[    CSSD]2009-07-19 17:15:53.297 [3600] >TRACE: Authorization prepare time: 3 ms
[    CSSD]2009-07-19 17:15:53.317 [3600] >TRACE: Authorization failed (112d356f0), timed out, start 17:13:53.320, duration 120000
[    CSSD]2009-07-19 17:15:53.317 [3600] >TRACE: Authorization prepare time: 2 ms
[    CSSD]2009-07-19 17:16:02.342 [3086] >TRACE: clssgmClientConnectMsg: Connect from con(112b932b0) proc(112b67d10) pid(1336252) proto(10:2:1:1)
[    CSSD]2009-07-19 17:16:02.977 [3600] >TRACE: Authorization failed (112d04f70), timed out, start 17:14:02.978, duration 120001
[    CSSD]2009-07-19 17:16:02.977 [3600] >TRACE: Authorization prepare time: 2 ms
[    CSSD]2009-07-19 17:16:03.007 [3600] >TRACE: Authorization failed (112d38210), timed out, start 17:14:03.006, duration 120002
[    CSSD]2009-07-19 17:16:03.007 [3600] >TRACE: Authorization prepare time: 2 ms
[    CSSD]2009-07-19 17:16:10.447 [3600] >TRACE: Authorization failed (112bd7e30), timed out, start 17:14:10.441, duration 120007
[    CSSD]2009-07-19 17:16:10.447 [3600] >TRACE: Authorization prepare time: 2 ms
[    CSSD]2009-07-19 17:16:10.847 [3600] >TRACE: Authorization failed (112d3ee70), timed out, start 17:14:10.840, duration 120008
[    CSSD]2009-07-19 17:16:10.847 [3600] >TRACE: Authorization prepare time: 2 ms
Thanks,
Mahi

Check the metalink note:
6996694-OCSSD.BIN CONSUMING 100% CPU AND ASM/DB HANGING

Just got a new imac and my iwork software wont work. I have the serial numbers but after migrating from my old mac running OSX tiger it won't even give me that opportunity. I do not have any discs as this was downloaded. Help

Just got a new imac and my iwork software wont work. I have the serial numbers but after migrating from my old mac running OSX tiger it won't even give me that opportunity. I do not have any discs as this was downloaded. Help

Hi, hopefully you guys can help me out, I just
purchased a G5 iMac to replace my aging dell desktop,
and now i'm more or less completely os x dependant
(I've had an ibook for a few months now). Anyway, I
have a few questions:
1. Does anyone know of a (free) mail notifier tool,
that will alert me when I recieve new pop3 mail? I
used to use AIM for this in windows
The built in email program does sound an alert for new messages.
2. Anyone know of a good (free, again) IRC client?
iChat works well.
3. I noticed earlier that my screen was flickering,
it seems to have subsided for now, but is that normal
to encounter in a new display? It wasn't really bad
flicker, but I could see it.
No idea.
4. I leave my desktops on 24/7x365, will that be a
problem with this iMac? Display shuts off of course
after 30 mins
You could set the iMac to go to sleep after non use. My Macs are
always sleeping when not in use, waking them up takes seconds.
Unlike windoze, never knew if it would be locked up or not.
5. Should I get the extended warranty? I'm usually
against them, but I am expecting this computer to
last me at least 3 years (for $1300 it better!)
before I upgrade, as I got that much out of a Dell
and from what I understand, Mac's do not age nearly
as fast as Windows pc's.
It is well worth the cost. Never know what might happen. Also
it is good insurance for future upgrades. I traded in my G4 tower
purchased two years ago for 75% of what I paid for it. Apple care
transfers, giving the buyer the remainder of your warranty.
Thank you for any help,
-Evan

Issues while Add / Delete RAC Node in Oracle 10g R2

Hi,
I have an requirement to add a New Node in the existing 2 Node RAC at Production, where 1 Node is Active & other one is passive due to licence issue & cannot keep both the nodes as active. Due to performance issues (Memory , CPU Cores ..etc) we are adding another new node.
Right now we are planning to add a 3rd database node making the new node as active and current active one as passive which is a swap & later on after final observation delete and decommission the current passive node.
This activity is checked at the Dev database with the same infrastructure (OS + Memory ..etc) but want to check what is the best approach (or) challenges we face during the RAC Node Addition / Deletion
RAC DB Version : 10.2.o.4
OS Version : RHEL 5.8
(1) Is the approach is right one , First Adding the node & later on delete
(2) If the approach is the correct , what would be the behavious of the 3rd node in means of active (or) passive
(3) We have taken RMAN backup , OS backup , CRS , ORACLE_HOME , ASM_Home backup , OCR & VD.
(4) Could you please give detail steps for adding / deleting node in 10g R2.
(5) Are they any known bugs to us with the DB release (or) OS while performing this activity.
Since this is a production machine we want to more proactive . Please correct or add any thing i am missing out ...
With Thanks,
Rakesh

Hello Rakesh,
Please follow the following steps.
Node Addition Steps
1. Install and configure OS and hardware for new node.
2. Add Oracle Clusterware to the new node.
3. Configure ONS for the new node.
4. Add ASM home to the new node.
5. Add Databse home to the new node.
6. Add a listener to the new node.
7. Add ASM instance to the New Node.
8. Add a database instance to the new node.
Details of steps
1. run cluvfy to verify whether New node is ready for addition or not.
     $ cluvfy stage -pre crsinst -n node2
2. from node1, execute
          $/u01/app/crs11g/oui/bin/addNode.sh
3. Specify node2 vip address and follow instructions.
4. In the last of installtion it may through an wornig and will ask to click on YES. click on YES
5. from node1,
          /u01/app/crs11g/bin/racgons add_config node2:6200
6. from Node1,set ORACLE_HOME=ASM_HOME and then execute addNode.sh from $ASM_HOME/oui/bin and Follow instrusctions.
7. From node1, set ORACLE_HOME=DB_HOME and then
     /u01/app/oracle/product/11.1.0/db_1/oui/bin/addNode.sh
     and Follow instructions.
8. from node2 start NETCA and configure listener for new node. While configuring Listener select the name of new node.
9. from node1 start dbca from ASM Home to configure ASM instance for new node.
10. Again from node1 start dbca from DB Home to add DB instance
Node deletion Steps
1. Delete the Database instance on the node to be deleted.
2. Clean up the ASM instance.
3. Remove the listener from the node to be deleted.
4. Remove the node from the database.
5. Remove the node from ASM.
6. Remove ONS configuration from the node to be deleted.
7. Remove the node from the clusterware
Details of Steps
1. Remove database Instance of node2
     Dbca -> instance Management -> delete instance -> password for sys -> select node -> finish.
2. Stop asm for node2 from any nodes.
     $srvctl stop asm –n node2
3. Remove asm for node2
     $ srvctl remove asm -n node2
4. Remove Listener from Node2 using NETCA.
5. From Node2:
          ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=node2" -local
6. From Node2, start runinstaller from Oracle_DB_Home/oui/bin, and remove "DB_HOME"
     $ ./runinstaller
     On the WELCOME Screen -> Deinstall product -> Select dbhome name (OraDb10g_Home1) -> Remove
7. From Node1:
          ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=node1"
8. From Node2, set Oracle_Home to asm_1 and then fire:
          ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=node2" -local
9. From Node2, start OUI and deinstall ASM Home.
10. From Node1, Set ORACLE_HOME= /u01/app/oracle/product/11.1.0/asm_1
11. From Node1: from /u01/app/oracle/product/11.1.0/asm_1/oui/bin, start OUI
          ./runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=node1"
12. From Node2: as a root user (#) execute rootdelete.sh from /u01/app/crs11g/install
     # /u01/app/crs11g/install/rootdelete.sh
13.From Node-1 first find out the node numbers
     # /u01/app/crs11g/bin/olsnodes -n
     output : node1 1
          node2 2
14. From Node-1 as a root user (#):
     # /u01/app/crs11g/install/rootdeletenode.sh node2[Node_Name] 2[node_no]
     output:
          CRS nodeapps are deleted successfully
          clscfg: EXISTING configuration version 4 detected.
          clscfg: version 4 is 11 Release 1.
          Node deletion operation successful.
          'node2' deleted successfully
15. From Node2 set ORACLE_HOME=CRS_HOME and then execute
     $$ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=node2" CRS=TRUE -local
16. ./runInstaller and remove CRS_HOME
17. From Node-1:
     $ /u01/app/crs11g/oui/bin/runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES=node1" CRS=TRUE
18. check node is deleted from ./crs_stat -t

Locks after migrate to RAC 2 nodes.

Similar Messages

Maybe you are looking for